Train decoder-only language models from scratch with PyTorch, featuring RoPE, RMSNorm, SwiGLU, and SDPA

A complete implementation of a Decoder-Only Transformer (GPT-style) built using PyTorch, without relying on high-level abstractions. This implementation includes all core components: token embeddings, positional embeddings, multi-head self-attention, feedforward networks, causal masking, and output logits generation.

Decoder-only GPT-style Transformer for autoregressive language modeling with BPE tokenization, supporting greedy, temperature, top-k, and nucleus sampling

From-scratch byte-level language-model experiments in an inspectable C/C++ systems stack with a C11 core, C++17 harnesses, and optional LibTorch training.