diff --git a/.gitignore b/.gitignore index 0345b3b1..bc705032 100644 --- a/.gitignore +++ b/.gitignore @@ -195,3 +195,4 @@ Qwen3/ # Telegram bot logs telegram_bot_logs/ +experiment_results/ diff --git a/blog/ARTICLE.md b/blog/ARTICLE.md new file mode 100644 index 00000000..c23872c0 --- /dev/null +++ b/blog/ARTICLE.md @@ -0,0 +1,195 @@ +# ThinkBooster: unlock your LLM's Pro reasoning mode by changing one URL + +*Plus a benchmark that finally tells you what the extra accuracy costs.* + +*Read the paper on [arXiv](https://arxiv.org/abs/2606.06915).* + +> _[Figure: hero — the endpoint gateway diagram. Upload `blog/endpoint_hero.png` (1392×889, 149 KB; exported from `images/endpoint.pdf`).]_ + +--- + +Every frontier model now ships a "thinking" mode. Ask o1, R1, or Qwen3 a hard question and it spends more compute before answering, and usually does better for it. Test-time compute scaling is the name for that move: you pay more at answer time, not training time, to get a better answer. + +The catch is that "spend more compute" is not one method. You can sample ten answers and keep the best. You can search a tree of reasoning steps and prune the weak branches. You can let the model run until it is confident, then stop. You can add compute only on the steps where it hesitates. These cost very different amounts, and most papers don't report the cost. So the question every practitioner actually has, *for the accuracy I need, which method is cheapest?*, has no clean answer. + +ThinkBooster is built to answer it. + +## What ThinkBooster is + +ThinkBooster is an open-source (MIT) framework for test-time compute scaling. It puts nine scaling strategies and four scorer families behind one API, measures accuracy and compute together, and ships as an OpenAI-compatible proxy with a visual debugger. + +The one design idea worth knowing: strategy and scorer are separate. The strategy decides the shape of the search, such as how many samples to draw, whether to follow a single line or branch into a tree, and when to stop. The scorer decides which partial answer looks best. ThinkBooster ships four kinds of scorer: + +- a **process reward model**: a separate critic model trained to grade reasoning steps; +- the model's own **confidence**, read straight off its token probabilities, with no extra model to run; +- an **LLM-as-judge**: ask a model to grade the work; +- **ReProbe**: a light supervised probe on the model's internal states (our own work; more on it below). + +Any strategy pairs with any scorer through a registry, so you can ask for "beam search with a reward model" or "best-of-N with confidence" without writing glue code. That registry is also the extension point: add a strategy or scorer once as a subclass, and the gateway, the benchmark, and the debugger all pick it up. Generation runs on vLLM underneath, so the same code goes from one GPU to multi-node. + +> _[Figure: the nine strategies (`blog/fig_strategies.png`)]_ + +## The hook, in one line of code + +The proxy is the part most people touch first. ThinkBooster sits in front of your model as an OpenAI-compatible endpoint, and the strategy and scorer live in the URL: + +``` +base_url = "/v1/beam_search/prm" +``` + +Point your existing OpenAI client at that, and the same model now runs beam search scored by a reward model. Change it to `.../v1/offline_bon/entropy` and you have swapped the entire scaling method, with no other edit to your code. Agents, copilots, and enterprise stacks already speak OpenAI, so you add reasoning scaling by editing a string, and you keep the compute budget in your hands. + +## Does it actually work? + +Three results from the benchmark are worth pulling out, because each one cuts against the obvious intuition. + +**Confidence can beat a trained reward model, and confidence is free.** On HumanEval+, pairing the MUR strategy (which spends extra compute only on uncertain steps) with a plain entropy signal takes Qwen3-8B from 79.3 to 88.8, the best coding score in the study and higher than any reward-model setup. Entropy is read off the tokens the model already produced, so it adds almost nothing, while a reward model is a second network you have to run. Against the most accurate reward-model setup, beam search with a PRM, confidence is both more accurate and under a quarter of the compute. The scope is narrow: this is a coding result. On math, the cheap signal does not win; there the strongest results come from a PRM, or from self-consistency on the hardest sets. + +> _[Figure: Qwen3-8B on HumanEval+ — accuracy vs. compute (`blog/fig_qwen3_humaneval.png`)]_ + +**Spending more does not reliably buy more.** With the cheap scorers, beam search trails plain best-of-N and even self-consistency, despite searching much harder. Its one strong configuration, beam search with a reward model, reaches the top math accuracy on several of the benchmarks, such as OlympiadBench and Gaokao, though best-of-N with the same reward model ties or beats it on the hardest sets. And it costs 17 to 24 times more than best-of-N to get there, often for a tie or a single point. For most budgets, best-of-N with a reward model sits at the better accuracy-per-FLOP point. + +> _[Figure: Qwen2.5-Math — accuracy vs. compute, aggregate over MATH-500, OlympiadBench, Gaokao (`blog/fig_qwen25_aggregate.png`)]_ + +**A drop-in change can improve real engineering output.** On GPT-OSS-120B writing CUDA kernels, adding best-of-N with a reward model raised end-to-end correctness from 26 to 30 and cut syntax errors by five points. Compilation rate dipped by one point, consistent with the reward model favoring more ambitious kernels that are likelier to fail to compile, a trade the correctness gain pays for. + +> _[Figure: GPT-OSS-120B results (`blog/fig_gpt_oss.png`)]_ + +None of this shows up in an accuracy column on its own. Because ThinkBooster's benchmark logs TFLOPs and tokens next to accuracy, you can see the trade and pick the point you can afford. + +## ReProbe, on a model the paper never tested + +The internal-state scorer above, ReProbe, is its own line of work: *ReProbe: Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Models*. Instead of a full reward model or a single confidence number, it trains a small probe to read a model's hidden states and predict whether a reasoning step is on track. No other framework in our comparison offers it as a first-class scorer, and because ThinkBooster keeps the model backend swappable, we can point it at models the original paper never benchmarked. + +We did that with K2-Think-V2, the reasoning model from MBZUAI and G42. With no changes to the model, ReProbe-guided best-of-N inside ThinkBooster lifts K2-Think-V2's MBPP+ pass rate over plain single-shot decoding, on both the base tests and the harder plus set. + +> _[Figure: K2-Think-V2 + ReProbe best-of-N vs single-shot on MBPP+ base (`blog/fig_k2_base.png`)]_ + +> _[Figure: K2-Think-V2 + ReProbe best-of-N vs single-shot on MBPP+ plus (`blog/fig_k2_plus.png`)]_ + +The point it makes is the one that matters here: a new model and a research-grade scorer drop into the same framework and the same benchmark with no special-casing. + +## Which one should you use? + +A rough rule of thumb from the numbers above: + +- **Closed API, no access to logits:** best-of-N, majority voting, or extended thinking. +- **Self-hosted and cost-sensitive:** best-of-N or MUR with a confidence scorer, which is free to compute and needs no second model. +- **Want top accuracy on math and can pay for it:** beam search with a reward model, though best-of-N with the same model is often as good for far less. + +## How it compares to OptiLLM and the rest + +ThinkBooster is not the only tool here, and the closest one is good. OptiLLM is an OpenAI-compatible proxy with a wide menu of inference techniques, and by a loose count it covers more tricks than ThinkBooster does. Our framework table uses a strict rule, counting genuine search, scoring, and decoding methods and excluding prompt-engineering scaffolds, and by that rule it is nine strategies to OptiLLM's seven. ThinkBooster's nine families span more than twenty recent methods, and OptiLLM likewise bundles many prompt and inference techniques. We are not claiming more tricks. + +What no other framework offers all at once: + +- uncertainty and ReProbe-style internal-state scoring as first-class, swappable scorer families, built on LM-Polygraph, our group's uncertainty library; +- first-class process-reward-model scoring and FLOPs-level compute accounting, which OptiLLM does not foreground; +- a benchmark that reports accuracy and compute together, with bundled, judged datasets across math, coding, and science; +- a visual debugger for reasoning trajectories. + +The other frameworks are narrower. LLM Reasoners is a strong modular research library, but it has no OpenAI endpoint, no native vLLM backend, and no joint performance–compute benchmark. The rest are narrower in scope, each oriented around a specific algorithm, a reproduction recipe, or visualization. The cells in our comparison are our own assessment rather than an audited score, but the gaps in uncertainty scoring and compute accounting are real. + +> _[Figure: framework comparison (`blog/fig_ttc_frameworks.png`)]_ + +## See why a trajectory wins: the visual debugger + +A scaling run makes decisions you never normally see: which candidates it sampled, what each one scored, which branch it kept, where it stopped. ThinkBooster ships an interactive debugger that opens all of it in the browser, and it is live: open the [demo](http://demo-thinkbooster.nlpresearch.group), pick a provider, model, strategy, and scorer, type in a problem, and run it. The run comes back as a reasoning timeline: every step with its score, the candidates the strategy considered, and which one was selected versus pruned. Click a step and the inspector shows the full candidate text and the number behind the decision. + +> _[Figure: the debugger — run config, strategy comparison, reasoning timeline, and step inspector in one place (`blog/fig_debugger_a.png`)]_ + +The trajectory tree shows the whole search at once. The highlighted path is the trajectory that was selected; the other branches were explored and dropped. It is the quickest way to see how beam search or best-of-N spends its compute and where it walks away from a dead end. + +> _[Figure: the trajectory tree — selected path in orange, pruned branches in gray (`blog/fig_debugger_b.png`)]_ + +You can also run two strategies on the same problem under the same compute budget and compare them side by side, which turns "which method is better" from an argument into something you can watch. + +## Try it in five minutes + +Install: + +```bash +pip install thinkbooster +``` + +**Option 1 — run it as a service and call the endpoint.** Stand the service up locally (the [local service guide](https://github.com/IINemo/thinkbooster/blob/main/docs/service/running_locally.md) covers building it and calling it as an endpoint), then point any OpenAI client at it. The strategy and scorer live in the URL, so the rest of your code does not change: + +```python +from openai import OpenAI + +client = OpenAI( + base_url="/v1/beam_search/prm", + api_key="", +) +response = client.chat.completions.create( + model="Qwen/Qwen3-30B-A3B", + messages=[{"role": "user", "content": + "Find the number of ordered pairs (x, y) of " + "positive integers satisfying x + 2y = 2xy."}], + extra_body={"max_tokens": 8192, "tts_beam_size": 4}, +) +print(response.choices[0].message.content) +``` + +**Option 2 — import the pieces as a library.** Compose a model, a step generator, and a strategy yourself, then run a question through it: + +```python +import os + +from lm_polygraph.utils.generation_parameters import GenerationParameters +from thinkbooster.models.blackboxmodel_with_streaming import BlackboxModelWithStreaming +from thinkbooster.generators.api import StepCandidateGeneratorThroughAPI +from thinkbooster.strategies.strategy_self_consistency import StrategySelfConsistency + +# 1. A model behind any OpenAI-compatible endpoint (OpenRouter here) +model = BlackboxModelWithStreaming( + openai_api_key=os.environ["OPENROUTER_API_KEY"], + model_path="openai/gpt-4o-mini", + base_url="https://openrouter.ai/api/v1", + supports_logprobs=True, + generation_parameters=GenerationParameters( + max_new_tokens=2048, temperature=0.7, top_p=0.8, + ), +) + +# 2. A step generator over that model +generator = StepCandidateGeneratorThroughAPI( + model=model, max_new_tokens=2048, temperature=0.7, top_p=0.8, +) + +# 3. A strategy: self-consistency over 8 sampled paths +strategy = StrategySelfConsistency( + step_generator=generator, num_paths=8, data_name="math", +) + +# 4. Run it on a question +request = [ + {"role": "system", "content": + "Please reason step by step, and put your final answer within \\boxed{}."}, + {"role": "user", "content": "What is 2^10 + 3^5?"}, +] +results = strategy.generate_trajectories_batch( + requests=[request], sample_indices=[0], +) +print(results[0]["trajectory"]) +print("Answer:", results[0]["extracted_answer"]) +``` + +From there you can run the compute-aware benchmark with one command to get accuracy and TFLOPs side by side, or open the visual debugger from the section above at `localhost:8001/debugger`. + +## Limitations + +The dynamic, confidence-driven strategies (MUR, DeepConf, uncertainty CoT) need logits or hidden states, so they want open-weight or self-hosted models. Against a closed API you get the black-box subset: best-of-N, majority voting, extended thinking, and LLM-as-judge scoring. Splitting native, unstructured "thinking" traces into clean steps is still hard, which can affect step-level scoring. And the evidence so far covers math, coding, and science QA; compute is reported as theoretical TFLOPs and tokens, with wall-clock logged but not yet studied. + +## Why you should try it + +Test-time compute scaling is here to stay; the useful question is which method, at what cost. ThinkBooster lets you measure that and ship the answer by changing a URL. If you research reasoning, you get a reproducible, compute-aware benchmark and scorers, including uncertainty and ReProbe, that you will not find elsewhere. If you build with LLMs, you get a drop-in proxy with the budget in your control. + +You can try it in your browser right now, no install needed: [the live demo](http://demo-thinkbooster.nlpresearch.group), or watch the [screencast](https://www.youtube.com/watch?v=7Idrcs_4kfQ) first. To run it locally: + +```bash +pip install thinkbooster +``` + +Code and docs are on [GitHub](https://github.com/IINemo/thinkbooster); the paper is on [arXiv](https://arxiv.org/abs/2606.06915). + diff --git a/blog/FIGURES.md b/blog/FIGURES.md new file mode 100644 index 00000000..024401c2 --- /dev/null +++ b/blog/FIGURES.md @@ -0,0 +1,66 @@ +# Article figures — how they were made + +Medium only accepts raster images, so every figure in `ARTICLE.md` is exported to PNG +here. Recipes below so they are reproducible. + +## Plots (rasterized from the paper's PDF figures) + +```bash +IMG="../_ACL_2026__ThinkBooster_demo (1)/images" +pdftoppm -png -r 200 -singlefile "$IMG/endpoint.pdf" endpoint_hero +pdftoppm -png -r 220 -singlefile "$IMG/qwen3_humaneval_ratio.pdf" fig_qwen3_humaneval +pdftoppm -png -r 220 -singlefile "$IMG/qwen25_aggregate_ratio.pdf" fig_qwen25_aggregate +``` + +| PNG | Article slot | +|-----|--------------| +| `endpoint_hero.png` | hero | +| `fig_qwen3_humaneval.png` | §4, after the "confidence beats the reward model" paragraph | +| `fig_qwen25_aggregate.png` | §4, after the "spending more" paragraph | + +## Tables (rebuilt as styled HTML, screenshotted) + +The academic LaTeX tables don't read well in a blog, so they're rebuilt as clean HTML +(`fig_gpt_oss.html`, `fig_ttc_frameworks.html`) and screenshotted with headless Chrome at +2x for crisp text. Edit the HTML and re-run: + +```bash +CHROME="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" +"$CHROME" --headless --disable-gpu --hide-scrollbars --force-device-scale-factor=2 \ + --default-background-color=FFFFFFFF --screenshot=fig_gpt_oss.png \ + --window-size=1000,348 "file://$PWD/fig_gpt_oss.html" +"$CHROME" --headless --disable-gpu --hide-scrollbars --force-device-scale-factor=2 \ + --default-background-color=FFFFFFFF --screenshot=fig_ttc_frameworks.png \ + --window-size=1240,660 "file://$PWD/fig_ttc_frameworks.html" +"$CHROME" --headless --disable-gpu --hide-scrollbars --force-device-scale-factor=2 \ + --default-background-color=FFFFFFFF --screenshot=fig_strategies.png \ + --window-size=1080,628 "file://$PWD/fig_strategies.html" +``` + +| PNG | Article slot | +|-----|--------------| +| `fig_gpt_oss.png` | §4, after the "drop-in change / CUDA kernels" paragraph | +| `fig_ttc_frameworks.png` | §6, framework comparison | +| `fig_strategies.png` | §2, the nine strategies | +| `fig_k2_base.png` | ReProbe section — K2-Think-V2 on MBPP+ base (source `fig_k2_base.html`, window 660x352) | +| `fig_k2_plus.png` | ReProbe section — K2-Think-V2 on MBPP+ plus (source `fig_k2_plus.html`, window 660x352) | + +## Debugger figures (from the paper's composited PDFs) + +```bash +IMG="../_ACL_2026__ThinkBooster_demo (1)/images" +# Render full pages (1700x2200 at 200 dpi), then crop with the paper's LaTeX trim +# values (trim=L B R T in pt). At 200 dpi, 1pt = 200/72 px; Pillow box = (L, T, W-R, H-B). +pdftoppm -png -r 200 -singlefile "$IMG/main-1-new.pdf" /tmp/m1 +pdftoppm -png -r 200 -singlefile "$IMG/main-2-new.pdf" /tmp/m2 +# fig_debugger_a <- main-1-new, trim 75 105 75 116 (paper uses T=123; eased to 116 for a top margin) +# fig_debugger_b <- main-2-new, trim 120 360 120 40 (paper's tree crop; drops the border + empty space) +``` + +| PNG | View | +|-----|------| +| `fig_debugger_a.png` | the debugger: run config, strategy comparison, reasoning timeline, step inspector | +| `fig_debugger_b.png` | trajectory tree (selected path in orange) | + +The individual raw screenshots `images/demo-config.png` / `demo-result.png` / +`demo-treeview.png` are alternatives if you prefer one view per image. diff --git a/blog/LINKEDIN.md b/blog/LINKEDIN.md new file mode 100644 index 00000000..32856f9f --- /dev/null +++ b/blog/LINKEDIN.md @@ -0,0 +1,44 @@ +# ThinkBooster — LinkedIn post + +Ready to paste. The first two lines are the hook (LinkedIn hides the rest behind "see more"). +Fill in the Medium URL once it's published; optionally @-tag your co-authors and MBZUAI / ETH Zürich. + +--- + +ThinkBooster was accepted to the ACL 2026 System Demonstration Track. + +It turns any LLM into its own "Pro reasoning mode" by changing one URL, and it finally tells you what the extra accuracy costs. + +Every frontier model can "think longer" at inference now. But there are many ways to spend that test-time compute: sample and rerank, search a tree of reasoning steps, stop when the model is confident, add compute only on the hard steps. They cost wildly different amounts, and almost no one reports the cost. So the question that actually matters, "for the accuracy I need, which method is cheapest?", has had no clean answer. + +ThinkBooster is built to answer it. It is an open-source framework that: + +• puts 9 test-time scaling strategies and 4 scorer families behind one API +• measures accuracy and compute (TFLOPs + tokens) together, on bundled math, coding, and science benchmarks +• ships as an OpenAI-compatible proxy: point your client at /v1// and the same model gets test-time scaling, with no change to your app +• includes a visual debugger that shows why a reasoning trajectory was selected + +Two results that cut against intuition: + +→ On HumanEval+, a free uncertainty signal beat a trained reward model, taking Qwen3-8B from 79.3 to 88.8. +→ Dropping best-of-N with a reward model into GPT-OSS-120B raised its CUDA-kernel correctness from 26 to 30. + +Built by MBZUAI, ETH Zürich, Imperial College London, NUS, and collaborators. + +📄 Paper (arXiv): https://arxiv.org/abs/2606.06915 +📝 Deep dive (Medium): [add Medium URL once published] +🕹️ Live demo: http://demo-thinkbooster.nlpresearch.group +💻 Code (GitHub): https://github.com/IINemo/thinkbooster + +#LLM #AI #NLP #MachineLearning #ACL2026 #Reasoning #OpenSource + +--- + +## Notes / before posting + +- **Medium URL**: replace the placeholder once the article is live. +- **Tag people**: on LinkedIn, @-mention co-authors and MBZUAI / ETH Zürich / Imperial / NUS for reach. +- **Emoji**: the link icons (📄 📝 🕹️ 💻) and bullets are standard LinkedIn formatting; strip them if you want it fully plain. +- **Demo link is http-only** (ephemeral pod) — keep the pod up while the post is circulating, or swap to the GitHub/paper links if it's down. +- **First comment trick**: LinkedIn down-ranks posts with outbound links. Consider moving the four links into the first *comment* and ending the post body with "Links in the comments." for better reach. +- **Length**: ~280 words. Fine for LinkedIn; trim the feature bullets to 2 if you want it shorter. diff --git a/blog/PUBLISH_CHECKLIST.md b/blog/PUBLISH_CHECKLIST.md new file mode 100644 index 00000000..31857d70 --- /dev/null +++ b/blog/PUBLISH_CHECKLIST.md @@ -0,0 +1,20 @@ +# Pre-publish checklist — ThinkBooster article + +Internal notes for `ARTICLE.md`. Do not paste into Medium/LinkedIn. + +## Before publishing +- [ ] **Figures/tables** — insert at the marked `> _[Figure: ...]_` spots; export `tab:gpt_oss` and `tab:ttc_frameworks` as images. +- [ ] **Live demo** — linked in the CTA (`http://demo-thinkbooster.nlpresearch.group`, verified up 2026-06-30). It is http-only and runs on an ephemeral RunPod pod, so keep the pod up through launch; serving https on the custom domain would be more robust. +- [ ] **K2 + ReProbe** — the MBPP+ number is single-seed (matched N=1 single-shot vs N=4 best-of-N, seed 42, 378-problem split; baseline = SLURM job 2651567, posted to PR #257). Replace with a multi-seed mean if available before launch. +- [ ] **OptiLLM** — the article no longer prints a specific OptiLLM count (softened to "many prompt and inference techniques" after codex review), so no number to verify. Optional: confirm their current count if you want to add one back. + +## Verified this round (codex review, 2026-06-30) +- Service route names in the snippets match the live service: strategies `self_consistency` / `offline_bon` / `online_bon` / `beam_search`, scorers `entropy` / `perplexity` / `sequence_prob` / `prm`. (Fixed an invalid `best_of_n/confidence` route to `offline_bon/entropy`.) +- The `running_locally.md` link resolves on `main` (PR #259 merged). +- Library snippet (`BlackboxModelWithStreaming` / `StepCandidateGeneratorThroughAPI` / `StrategySelfConsistency` / `generate_trajectories_batch`) matches the current API. +- Numbers checked against the paper tables: HumanEval+ 79.3→88.8, CUDA 26→30 / −5pp syntax / 65→64 compile, beam+PRM 17–24× vs best-of-N+PRM. Math claim scoped (PRM/self-consistency win, not entropy); "visual debugger" reframed (ReasonGraph also has one); "first tool" and "20-plus for OptiLLM" superlatives removed. +- arXiv link (`2606.06915`) is live; the screencast resolves to the YouTube video. + +## Notes +- `tts_metadata` field names: only verify if the extended metadata example is ever added (it is not currently in the article). +- ReProbe vs UHead: the public name is **ReProbe** (deliberate); the saved experiment branch/class still says `uhead`/`step_reasoning`. diff --git a/blog/STRUCTURE.md b/blog/STRUCTURE.md new file mode 100644 index 00000000..d1b10e52 --- /dev/null +++ b/blog/STRUCTURE.md @@ -0,0 +1,229 @@ +# ThinkBooster article — proposed structure (Medium first, LinkedIn cut after) + +> **Full draft now written to [`ARTICLE.md`](ARTICLE.md).** Added a dedicated section, +> "ReProbe, on a model the paper never tested," presenting the K2-Think-V2 (MBZUAI + G42) +> + ReProbe best-of-N result (MBPP+ 66.1→66.9 plus / 80.7→81.7 base), framed as a +> model-agnostic-generality point, not an accuracy headline (single-seed; scaling up). +> ReProbe = *Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States +> of Large Language Models* — the group's own scorer, also one of the four scorer families. + +Draft plan, revised after an independent fact-check pass. Not committed. Source of truth +for facts: the ACL 2026 paper (`_ACL_2026__ThinkBooster_demo (1)/latex/main.tex`), the +repo README, and `tables/ttc_frameworks.tex`. Numbers cross-checked against +`tables/qwen3_full_results.tex`, `qwen25_full_results.tex`, `gpt_oss.tex`. + +--- + +## The thesis (one sentence) + +Your model can already think longer; the open question is *which* way of spending that +test-time compute is cheapest for the accuracy you need — and ThinkBooster is the first +tool that lets you measure the answer and ship it by changing one URL. + +This is the spine. Every section either sets it up, proves it, or acts on it. If a +paragraph doesn't serve the thesis, it gets cut. + +## Audience + +Two readers at once: the researcher (wants a reproducible, compute-aware benchmark and the +uncertainty scorers) and the builder (wants a drop-in proxy with a compute budget). The +piece speaks to the builder first, with enough rigor to keep the researcher. + +## Title options + +1. *Your model can think longer. ThinkBooster helps you decide how.* +2. *Test-time compute, without the guesswork.* +3. *One URL turns any LLM into its own "Pro reasoning mode."* + +Lead candidate: #1 as title, #3 as subtitle. + +--- + +## Section-by-section + +### 0. Title, subtitle, hero image +- Hero: the framework diagram (`images/thinkbooster.png`) or the gateway diagram + (`images/endpoint.pdf`). + +### 1. The problem, in one scene (~160 words) +- Thinking models (o1, R1, Qwen3, GPT-OSS) made one idea mainstream: spend more compute at + inference, get better answers. **Add one plain primer sentence:** test-time compute + scaling means paying more at answer time, not training time, to get a better answer. +- But there are many ways to spend it — sample and rerank, search a tree of steps, stop when + the model is confident, add compute only on the hard steps. Each lives in its own repo, is + measured its own way, and most never report what it costs. +- So the question a practitioner actually has has no clean answer: *for the accuracy I need, + which method is cheapest?* +- Land the thesis. One line: ThinkBooster is built to answer that, and to let you switch the + answer on without rewriting your app. +- Asset: hero diagram. + +### 2. What ThinkBooster is (~210 words) +- Plain definition, no adjectives: an open-source (MIT) framework that puts nine test-time + scaling strategies and four scorer families behind one API, measures accuracy and compute + together, and ships as an OpenAI-compatible proxy plus a visual debugger. +- The one design idea worth explaining: strategy and scorer are separate. The *strategy* + decides the shape of the search (how many samples, tree or line, when to stop). The + *scorer* decides which partial answer looks best. ThinkBooster ships four kinds, and naming + them sets up the results section: + - a process reward model (a separate critic model trained to grade reasoning steps), + - the model's own uncertainty / confidence (read off its token probabilities, no extra model), + - an LLM-as-judge, + - ReProbes (a light probe on the model's internal states; the group's own work). +- You mix and match any strategy with any scorer. The four system pieces: library, benchmark, + gateway, debugger. +- Asset: strategy table (the nine strategies). + +### 3. The hook, in one line of code (~110 words + one annotated line) +- The drop-in proxy. Point an OpenAI client's `base_url` at `.../v1//` and + the same model now does test-time scaling. No other code changes. +- Show only the annotated URL line here (the *concept*); the full runnable snippet lives in + §6 so we don't print the same code twice. +- Why it matters: agents, copilots, and enterprise stacks already speak OpenAI. You add + reasoning scaling by changing a string, and you keep the compute budget in your hands. +- Asset: gateway diagram (`images/endpoint.pdf`). + +### 4. Does it actually work? (the results that surprised us) (~280 words) +Write this as three short **paragraphs** with plain topic sentences — not a bolded-bullet +triad (that pattern is a known AI tell, and three parallel one-liners is rule-of-three). + +- **Uncertainty as a near-free scorer (coding).** On HumanEval+, the MUR strategy with a plain + entropy signal takes Qwen3-8B from 79.3 to 88.8 — the best coding number in the study, and + higher than any reward-model setup. The point is the scorer's cost: entropy is read off the + tokens the model already produced, so it adds almost nothing, whereas a reward model is a + second network you have to run. Against the *most accurate* reward-model setup (beam search + + PRM) MUR+entropy is both better and under a quarter of the compute. State the honest + scope: on math a real PRM still wins, so this is a coding finding, not a universal one. +- **Spending more does not reliably buy more.** With the cheap scorers, beam search + underperforms plain Best-of-N and even self-consistency, despite searching far harder. Its + one strong configuration, beam search + PRM, does take the top math accuracy — but it costs + 17–24× more than Best-of-N + PRM to get there, often for a tie or a point or two. + Best-of-N + PRM is the better accuracy-per-FLOP point for most budgets. (Do **not** say + "beam loses to BoN" and "17–24×" in the same breath — they are different comparisons.) +- **A real engineering payoff.** On GPT-OSS-120B generating CUDA kernels, dropping in + Best-of-N + PRM raised end-to-end correctness from 26 to 30 and cut syntax errors by 5 + points. Add the honest caveat the table shows: compilation rate dips a hair (65→64) because + the reward model favors more ambitious kernels that are likelier to fail to compile — a + trade the correctness gain pays for. +- Closing line that ties to the thesis: none of this is visible from an accuracy column + alone; because the benchmark logs TFLOPs and tokens next to accuracy, you can see the trade + and pick the point you can afford. +- Asset: the accuracy-vs-compute plot (`qwen3_humaneval_ratio.pdf`, + `qwen25_aggregate_ratio.pdf`) and the GPT-OSS results table. + +### 5. Which one should you use? (~90 words, a shareable decision box) +A three-line rule of thumb straight off the results and the white/black-box split: +- Closed API, no access to logits: Best-of-N, majority voting, or extended thinking. +- Self-hosted and cost-sensitive: MUR or Best-of-N with an uncertainty scorer (free to + compute, no second model). +- Need the most accuracy and can pay for it: beam search + PRM. +This is likely the most-shared element and it directly serves the thesis. Set it as a boxed +callout. + +### 6. How it compares — vs OptiLLM and the rest (~260 words + table) +- Honest opening: there are good tools here. OptiLLM is the closest — an OpenAI-compatible + proxy with a wide menu of inference techniques. On raw count of tricks it has more than us; + say so. +- The fairness note, kept *next to the table*: ThinkBooster's own paper claims its nine + strategies "cover more than twenty" recent methods, and OptiLLM's full menu is "20+" too — + so under a loose count both are 20+. Under a strict rule (genuine search/scoring/decoding + methods, excluding prompt scaffolds) it is 9 vs 7. Use that symmetry; don't print "9 vs 7" + next to an unexplained "20+". +- What we add that none of them ship: + - uncertainty / confidence as a first-class, swappable scorer family (built on + LM-Polygraph) — narrow wording on purpose (a rival ships entropy decoding; the claim is + "first-class swappable family," not "nobody else touches confidence"), + - process-reward-model scoring and FLOPs-level compute accounting (OptiLLM has neither), + - a joint performance–compute benchmark with bundled, judged datasets (5 math, 3 coding, + 1 science), + - a visual debugger for reasoning trajectories. +- Trim the per-competitor tour to the two names a reader knows (LLM Reasoners, OptiLLM); let + the table carry OpenR / search-and-learn / TreeQuest / ReasonGraph. +- Honesty caveat in the caption: the cells are our own assessment, not an audited score. + (Keep this — it is more candid than the paper itself, and it is the best anti-spin move.) +- Asset: the `ttc_frameworks` comparison table. + +### 7. Try it in five minutes (usage + code) (~230 words + snippets) +- Install: `pip install thinkbooster`. +- As a proxy: the full `base_url`-swap snippet with `extra_body` budget controls (use the + exact model id from the README, `Qwen/Qwen3-30B-A3B`). +- As a library: import a strategy and run it (for researchers). +- Run the compute-aware benchmark: the `run_tts_eval.py` command that prints accuracy and + TFLOPs. +- Open the visual debugger at `localhost:8001/debugger`. +- Optional: the response-metadata example (per-step scores, token count, TFLOPs) — flag for + verification against the shipped API before publishing. +- Asset: debugger screenshot (`images/demo-treeview.png` or `demo-result.png`). + +### 8. The honest limitations (~140 words) +- White-box strategies (MUR, DeepConf, uncertainty CoT) need logits or hidden states, so they + want open-weight or self-hosted models. Closed APIs get the black-box subset (Best-of-N, + majority voting, extended thinking, LLM-as-judge). +- Step-boundary detection is still hard for native, unstructured thinking traces. +- Scope so far: math, coding, science QA. Compute is theoretical TFLOPs plus tokens; + wall-clock is logged but not yet studied. +- Why this section exists: it is true, it sets correct expectations, and it is what keeps the + piece from reading as a pitch. + +### 9. Why you should try it / call to action (~130 words) +- Restate the thesis: the question is not whether to scale test-time compute but which way + and at what cost — and ThinkBooster lets you answer that and ship the answer with a URL + change. +- Who it is for, one line each: researchers (reproducible compute-aware benchmark, + uncertainty scorers) and builders (drop-in proxy, budget control). +- Trim to two asks: `pip install thinkbooster` and try the live demo. Links to GitHub / + arXiv / paper go inline, not as a four-item list. + +--- + +## Figure → section map (the user inserts these) +- §0 hero — `thinkbooster.png` or `endpoint.pdf` +- §2 — `tab:tts_strategies` (nine strategies) or `thinkbooster_library.pdf` +- §3 — `endpoint.pdf` (gateway) +- §4 — `qwen3_humaneval_ratio.pdf` + `qwen25_aggregate_ratio.pdf` + GPT-OSS table +- §6 — `ttc_frameworks` table +- §7 — `demo-treeview.png` / `demo-result.png` / `demo-config.png` +- (spare: `beam_search.pdf` to illustrate a strategy in §2 or §4) + +## Length +- Medium: ~2,000 words, an 8–9 minute read. Comprehensive but not padded. If it runs long, + cut the per-competitor one-liners in §6 first. +- LinkedIn 3-minute cut (~300 words): keep §1 hook, one line of §2, the §4 HumanEval result, + the §3 URL line, the §5 decision box, and the §9 CTA. Drop the comparison detail and + limitations (link to the Medium piece for those). + +## Voice guardrails (so it does not read as AI-generated) +- Sentence-case headings. Sparse bold. Few em dashes (commas, periods, parentheses instead). +- Banned words: delve, leverage, seamless, robust, powerful, state-of-the-art, pivotal, + underscore, showcase, testament, crucial, realm, landscape (figurative). +- Banned openers: sentence-initial "Additionally,", "Moreover,", "Furthermore," (the top + AI-vocab tell). +- Banned shapes: "not just X but Y" / "X is not Y, it's Z" parallelisms; rule-of-three + padding; bolded-header-colon bullet lists (write §4 as paragraphs); `-ing` tails that + editorialize ("highlighting its importance"). +- Concrete nouns and real numbers over adjectives. Plain "is/are", not "serves as". State + affiliations flat ("LM-Polygraph, our group's UQ library"), not as self-importance + ("the group's research edge / lineage"). +- One clear argument, not a feature list with a bow on top. + +## Claims to keep exactly (verified against the tables) +- HumanEval+ 79.3 → 88.8 (+9.5), MUR + entropy, Qwen3-8B. +- GPT-OSS-120B CUDA: correctness 26 → 30 (+4pp), syntax errors −5pp, compilation 65 → 64. +- Beam+PRM vs BoN+PRM on Qwen2.5 math = 17–24× compute, beam ties/wins on accuracy. +- Nine strategies, four scorer families. base_url `.../v1//`. MIT license. + +## Claims to fix or qualify (the two a competitor would unpick) +- Beam-search §4: do not weld "loses to BoN" to "17–24×". Separate the cheap-scorer + underperformance from the beam+PRM cost premium. +- Uncertainty §4: "fraction of the compute" is true only vs the *strongest* PRM setup + (beam+PRM). The durable claim is the near-zero *scorer* overhead, not that the strategy is + globally cheaper (vs cheap BoN+PRM it is ~6× more). Keep it coding-scoped. +- §6: "only framework with uncertainty scorers" → "only one with uncertainty as a + first-class, swappable scorer family." + +## Open checks before publishing (facts to confirm) +- Is the live demo URL up? (reviewers flagged it down during review) +- `tts_metadata` field names match the shipped service? +- OptiLLM's current technique count (so the loose "20+" is accurate). +- Title wording: paper `\title` says "Seamless Test-Time Scaling"; body says "test-time + compute scaling" — pick one. diff --git a/blog/endpoint_hero.png b/blog/endpoint_hero.png new file mode 100644 index 00000000..893b181f Binary files /dev/null and b/blog/endpoint_hero.png differ diff --git a/blog/fig_debugger_a.png b/blog/fig_debugger_a.png new file mode 100644 index 00000000..9d72d9b9 Binary files /dev/null and b/blog/fig_debugger_a.png differ diff --git a/blog/fig_debugger_b.png b/blog/fig_debugger_b.png new file mode 100644 index 00000000..b8442058 Binary files /dev/null and b/blog/fig_debugger_b.png differ diff --git a/blog/fig_gpt_oss.html b/blog/fig_gpt_oss.html new file mode 100644 index 00000000..4853fa61 --- /dev/null +++ b/blog/fig_gpt_oss.html @@ -0,0 +1,36 @@ +
+ + + + + + + + + + + + + + +
GPT-OSS-120B with a PRM scorer, across QA, coding, and CUDA-kernel generation. Best per column in bold. Dropping in best-of-N lifts CUDA-kernel correctness from 26 to 30 and cuts syntax errors by five points; compilation dips one point as the reward model favors more ambitious kernels.
KernelBench
StrategyGPQA-DiamondHumanEval+MBPP+SyntaxCompilationCorrectness
Raw CoT70.383.576.082.065.026.0
Offline Best-of-N72.285.478.887.064.030.0
Beam Search73.287.878.4———
+
diff --git a/blog/fig_gpt_oss.png b/blog/fig_gpt_oss.png new file mode 100644 index 00000000..f9d4a79a Binary files /dev/null and b/blog/fig_gpt_oss.png differ diff --git a/blog/fig_k2_base.html b/blog/fig_k2_base.html new file mode 100644 index 00000000..06fc5b3d --- /dev/null +++ b/blog/fig_k2_base.html @@ -0,0 +1,31 @@ +
+

K2-Think-V2 on MBPP+ (base tests)

+ + + + + + + + + +
ReProbe-guided offline best-of-N (N=4) vs single-shot CoT, medium thinking budget. MBPP+ base tests (EvalPlus, 378 problems). pass@1, single seed. Green = gain over the baseline.
MethodNpass@1 (base)
Single-shot CoT (baseline)180.7%305 / 378
Offline best-of-N + ReProbe481.7%309 / 378+1.0 pp
+
diff --git a/blog/fig_k2_base.png b/blog/fig_k2_base.png new file mode 100644 index 00000000..f765a8c1 Binary files /dev/null and b/blog/fig_k2_base.png differ diff --git a/blog/fig_k2_plus.html b/blog/fig_k2_plus.html new file mode 100644 index 00000000..85b50cff --- /dev/null +++ b/blog/fig_k2_plus.html @@ -0,0 +1,31 @@ +
+

K2-Think-V2 on MBPP+ (plus tests)

+ + + + + + + + + +
ReProbe-guided offline best-of-N (N=4) vs single-shot CoT, medium thinking budget. MBPP+ plus tests (378 problems; base + extra auto-generated tests, a stricter check). pass@1, single seed. Green = gain over the baseline.
MethodNpass@1 (plus)
Single-shot CoT (baseline)166.1%250 / 378
Offline best-of-N + ReProbe466.9%253 / 378+0.8 pp
+
diff --git a/blog/fig_k2_plus.png b/blog/fig_k2_plus.png new file mode 100644 index 00000000..b403bc1b Binary files /dev/null and b/blog/fig_k2_plus.png differ diff --git a/blog/fig_qwen25_aggregate.png b/blog/fig_qwen25_aggregate.png new file mode 100644 index 00000000..f8ca7b5f Binary files /dev/null and b/blog/fig_qwen25_aggregate.png differ diff --git a/blog/fig_qwen3_humaneval.png b/blog/fig_qwen3_humaneval.png new file mode 100644 index 00000000..aa01fdf9 Binary files /dev/null and b/blog/fig_qwen3_humaneval.png differ diff --git a/blog/fig_strategies.html b/blog/fig_strategies.html new file mode 100644 index 00000000..71b5ba79 --- /dev/null +++ b/blog/fig_strategies.html @@ -0,0 +1,41 @@ +
+ + + + + + + + + + + + + + + + +
The nine test-time-scaling strategies in ThinkBooster. + Online strategies score while the model reasons; offline ones score after a full answer. + White-box strategies read logits or hidden states; black-box ones need only the generated text. Any strategy pairs with any scorer.
StrategyWhat it doesModeAccessPrefill
Best-of-NSample N full solutions, keep the best one by the scorerOfflineBlack-box✗
Majority votingSample N solutions, take the majority answer (self-consistency)OfflineBlack-box✗
Beam search (ToT)Explore a tree of reasoning steps, prune low-scoring branchesOnlineBlack-box✓
Extended thinkingForce a longer chain of thought within a budgetOnlineBlack-box✓
MURSpend extra compute only on the steps the model is unsure aboutOnlineWhite-box✓
DeepConf (online)Steer generation toward high-confidence tokensOnlineWhite-box✓
DeepConf (offline)Rerank sampled solutions by the model's own confidenceOfflineWhite-box✗
Phi-decodingForesight sampling with adaptive pruning on an uncertainty signalOnlineWhite-box✓
Uncertainty-CoTBranch into extra trajectories when the model is uncertainOnlineWhite-box✓
+
diff --git a/blog/fig_strategies.png b/blog/fig_strategies.png new file mode 100644 index 00000000..4c43cecb Binary files /dev/null and b/blog/fig_strategies.png differ diff --git a/blog/fig_ttc_frameworks.html b/blog/fig_ttc_frameworks.html new file mode 100644 index 00000000..7695fe9e --- /dev/null +++ b/blog/fig_ttc_frameworks.html @@ -0,0 +1,43 @@ +
+ + + + + + + + + + + + + + + + + + + +
ThinkBooster vs. other test-time-scaling frameworks. + ✓ supported   ◐ partial   ✗ not supported. + "API" = any OpenAI-compatible HTTP endpoint. Counts include genuine search / scoring / decoding methods only (prompt-engineering scaffolds excluded). This is our own assessment, not an audited score.
FeatureThinkBoosterOptiLLMLLM ReasonersOpenRS&LTreeQuestReasonGraph
Strategy breadth✓ (9)◐ (7)◐ (5)◐ (5)✗ (3)✗ (1)✗ (0)
Scorer-family breadth✓ (4)◐ (2)◐ (2)✗ (1)✗ (1)◐ (1)✗ (0)
Supports methods up to2026202620242024202420262023
Uncertainty-based scorers✓✗✗✗✗✗✗
Joint perf–compute benchmark✓◐◐◐◐✗✗
Bundled math benchmarks5002100
Bundled coding benchmarks3000000
BackendsvLLM+HF+APIAPI+HF+MLXHF+SGLang+APIvLLM+HFvLLMagnosticAPI
OpenAI-compatible gateway✓✓✗✗✗✗✗
Visual debugger✓✗◐◐✗✗✓
Modular architecture✓◐◐◐✗◐◐
+
diff --git a/blog/fig_ttc_frameworks.png b/blog/fig_ttc_frameworks.png new file mode 100644 index 00000000..4c9db588 Binary files /dev/null and b/blog/fig_ttc_frameworks.png differ