Revise DeepSeek Sparse Attention page with new content structure and enhanced focus

- Updated the tutorial content to emphasize the key innovation of DeepSeek Sparse Attention (DSA) and its efficiency in long-context processing.
- Reorganized sections for clarity, introducing a "How It Works" and "Results" segment to better explain the model's mechanics and performance.
- Removed the previous tutorial markdown file and integrated interactive technical cards to improve user engagement and understanding.
- Adjusted the page title and introductory text to reflect the new focus on DSA's breakthroughs and community contributions.

Author: vukrosic

…og/deepseek-sparse-attention/tutorial.md …deepseek-sparse-attention/explanation.mdapp/blog/deepseek-sparse-attention/tutorial.md renamed to app/blog/deepseek-sparse-attention/explanation.md app/blog/deepseek-sparse-attention/tutorial.md renamed to app/blog/deepseek-sparse-attention/explanation.md

@@ -148,4 +148,95 @@ The key takeaways are:
 *   **Decoupled design works:** Using a small, specialized indexer to guide the main attention mechanism is an effective and efficient strategy.
 *   **Careful training is key:** A multi-stage process of warming up the indexer and then adapting the full model is crucial for success.
 
-While this model is labeled "experimental," it points toward a future where we can routinely process entire books, legal archives, and code repositories without breaking the bank. The era of truly long-context AI is getting much, much closer.
+While this model is labeled "experimental," it points toward a future where we can routinely process entire books, legal archives, and code repositories without breaking the bank. The era of truly long-context AI is getting much, much closer.
+
+
+---
+
+
+Of course. Let's break down the paper "DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention" step by step.
+
+### High-Level Summary
+
+The paper introduces an experimental model, **DeepSeek-V3.2-Exp**, which is a more efficient version of its predecessor, **DeepSeek-V3.1-Terminus**. The core problem with large language models is that the computational cost of the attention mechanism grows quadratically with the length of the input sequence (O(L²)), making very long contexts (like a whole book) extremely expensive to process.
+
+The solution presented is **DeepSeek Sparse Attention (DSA)**, a new attention mechanism that reduces this complexity to be nearly linear (O(Lk), where k is a small constant). They achieve this without a significant drop in performance, making long-context processing much faster and cheaper.
+
+---
+
+### Step 1: The Problem - The Cost of Long Contexts
+
+Standard self-attention, the core of the Transformer architecture, requires every token in a sequence to "attend to" (or compare itself with) every single token that came before it.
+
+*   If you have a sequence of **L** tokens, the 10th token looks at the first 9 tokens.
+*   The 100th token looks at the first 99 tokens.
+*   The 100,000th token looks at the first 99,999 tokens.
+
+This results in a total number of computations proportional to L², which becomes computationally infeasible and very expensive for long sequences (e.g., L = 128,000).
+
+### Step 2: The Solution - DeepSeek Sparse Attention (DSA)
+
+DSA is the key innovation. Instead of having every token look at *all* previous tokens, it intelligently selects only a small, fixed number (`k`) of the most relevant previous tokens to look at. This is a two-part process.
+
+#### Part A: The "Lightning Indexer" (The Scout)
+
+This is a very small, fast component whose only job is to quickly figure out which previous tokens are most important for the current token.
+
+*   For a current query token (`h_t`), the indexer calculates an "index score" (`I_t,s`) for every preceding token (`h_s`).
+*   This score represents the predicted relevance of token `s` to token `t`.
+*   As described in **Equation (1)**, this calculation is designed to be extremely fast. It uses a small number of heads and can even run in low-precision FP8 format, making it much cheaper than full attention.
+*   Think of it as a "scout" that quickly scans the entire history and flags the most promising locations.
+
+#### Part B: Fine-grained Token Selection & Sparse Attention (The Main Operation)
+
+Once the Lightning Indexer has calculated scores for all preceding tokens, this mechanism kicks in.
+
+*   It simply picks the **top-k** highest scores. For this model, `k` is set to **2048**.
+*   The main attention mechanism then operates *only* on the key-value pairs of these 2048 selected tokens.
+*   Instead of calculating attention over L tokens, it now only calculates it over `k` tokens. This dramatically reduces the complexity from O(L²) to O(L * k). Since `k` is a fixed number and much smaller than `L`, the cost grows linearly with the sequence length, not quadratically.
+
+**Figure 1** in the paper visualizes this. The input (`h_t`) is split. One path goes to the Lightning Indexer to get the scores. The other path goes to the main attention module. The indexer's output is used by a "Top-k Selector" to filter the key-value pairs that the main attention module is allowed to see.
+
+
+
+### Step 3: The Training Process - How to Teach the Model to be Sparse
+
+They couldn't just switch on DSA in a pre-trained model and expect it to work. They used a careful, multi-stage training process, starting from an already powerful model (DeepSeek-V3.1-Terminus).
+
+#### Stage 1: Dense Warm-up (Teaching the Scout)
+
+The first step was to train just the Lightning Indexer.
+*   **Goal:** Teach the indexer to find the same tokens that the full, dense attention mechanism would find important.
+*   **Method:** They froze the main model parameters and kept the standard (dense) attention active. They then trained the indexer to mimic the attention patterns of the main model.
+*   **Loss Function (Equation 3):** They used a KL-divergence loss, which essentially measures how different two probability distributions are. The goal was to minimize the difference between the indexer's scores and the actual attention scores from the main model.
+*   This stage was very short (1000 steps), just to get the indexer properly initialized.
+
+#### Stage 2: Sparse Training (Adapting the Whole System)
+
+Now, they activate the full DSA mechanism, including the top-k selection.
+*   **Goal:** Adapt the entire model (both the main part and the indexer) to work effectively with this new sparse attention pattern.
+*   **Method:** The model now only "sees" the 2048 tokens selected by the indexer.
+    *   The **main model** is trained on the standard language modeling task (predicting the next token).
+    *   The **Lightning Indexer** continues to be trained to align with the main attention distribution, but now only on the set of selected tokens (as shown in **Equation 4**).
+*   This was the main training phase, running for 15,000 steps on a massive amount of data (943.7 billion tokens).
+
+#### Stage 3: Post-Training (Fine-tuning and Alignment)
+
+After the model learned to use sparse attention, they fine-tuned it for specific tasks like coding, math, and following instructions. Crucially, they used the **exact same data and methods** as they did for the non-sparse DeepSeek-V3.1-Terminus. This ensures a fair comparison of the models' capabilities, isolating the impact of adding DSA.
+
+### Step 4: The Results - The Payoff
+
+The paper evaluates the new model on two fronts: capabilities and efficiency.
+
+#### Capabilities (Table 1 & Figure 2)
+
+*   **Performance:** DeepSeek-V3.2-Exp performs **almost identically** to its dense predecessor, DeepSeek-V3.1-Terminus. There is no significant drop in quality on benchmarks for math, coding, and general knowledge.
+*   **Training Stability:** The training curves in Figure 2 show that the sparse model learns just as steadily during Reinforcement Learning (RL) fine-tuning as the dense model. This proves that DSA is a stable architecture.
+
+#### Efficiency (Figure 3)
+
+This is the main victory. The graphs show the cost per million tokens during inference.
+*   **Prefilling (Processing the prompt):** As the input context gets longer (moving right on the x-axis), the cost for the old model (blue line) skyrockets. The cost for the new sparse model (orange line) grows much, much slower.
+*   **Decoding (Generating the response):** The same pattern holds. The cost of generating a new token is significantly lower with the sparse model when the context is long, as it doesn't need to re-scan the entire history with expensive, dense attention.
+
+In summary, they successfully traded a tiny, almost negligible amount of model performance for a massive improvement in computational efficiency for long-context tasks.