Standard Transformer Pipelines

- Overview

The standard transformer pipeline turns input text into numbers, processes context through repeating layers of attention and feed-forward networks, and projects the final results back into readable words or probabilities. 

1. Token Embeddings and Positional Encoding:

• Tokenization: Splits raw text into small pieces called tokens (words, parts of words, or characters).
• Embedding Layer: Converts each token into a dense vector of numbers that captures basic semantic meaning.
• Positional Encoding: Adds extra numbers to the vectors so the model knows the order and position of words in a sentence.

2. Stacked Attention and Feed-Forward Blocks

• Multi-Head Self-Attention: Allows each token to look at and relate to all other tokens in the sequence to understand context.
• Feed-Forward Networks: Processes the outputs of the attention layer through standard linear and non-linear transformations.
• Stacking and Residuals: Repeats these attention and feed-forward blocks many times, using shortcut connections (residual connections) and layer normalization to keep data flowing smoothly.

3. Output Projection Layer

• Linear Projection: Takes the final hidden state vectors and scales them up to match the size of the entire vocabulary.
• Softmax / Decoding: Converts these raw scores into probabilities for the next likely token or classification label.