A clean, modular implementation of the Transformer architecture from the "Attention is All You Need" paper. This project provides a full Encoder-Decoder pipeline for sequence-to-sequence tasks.
- Multi-Head Attention: Captures complex dependencies across different sequence positions using parallel attention heads.
- Position-Wise Feed Forward: Transforms attention outputs through linear layers and ReLU activations.
- Positional Encoding: Injects sequence order information using sine and cosine functions.
- Encoder: A stack of self-attention and point-wise feed-forward layers.
- Decoder: Implements masked self-attention and cross-attention for target sequence generation.
The implementation supports a standard training pipeline:
-
Loss: nn.CrossEntropyLoss (ignores padding index 0).
-
Optimizer: Adam with specific betas (0.9, 0.98) and epsilon (1e-9).
-
Input: Tokenized source and target sequences.
-
Output: Predicted token probabilities for the next sequence element.
Ensure you have PyTorch installed:
pip install torch torchvision torchaudio