Kingy AI
Subscribe, 2.1M subscribersWork with Kingy
AI News

s1: Simple Test-time Scaling – Paper Summary

February 7, 2025 AI News
DeepSeek’s Reinforcement Learning Breakthrough: Redefining the AI Arms Race and Nvidia’s Edge article image

This summary covers the February 3, 2025 version 2 of s1: Simple Test-Time Scaling, embedded below. The paper introduces s1K, fine-tunes the existing Qwen2.5-32B-Instruct model to produce s1-32B, and studies inference-time budget forcing. Results below are the authors’ reported experiments, not a Kingy training or benchmark run. The arXiv link now serves a later revision; the figures here refer to the embedded version 2.

2501.19393v2Download

Modern large language models (LLMs) exhibit remarkable aptitude across myriad tasks, from arithmetic and geometry to biology and quantum theory. Yet, many frontiers remain poorly understood, particularly the notion of test-time scaling. The notion is straightforward, but its impact is profound: at inference, one can allocate more computational steps (or “thinking tokens”) to potentially boost a model’s final accuracy. But how does one control those steps effectively? How large can they get before saturating performance? And might that same approach drive performance upward on exceptionally difficult problems, or does it buckle when context windows max out?

The paper combines 1,000 curated questions with teacher-generated reasoning traces and answers. Its budget forcing method controls generated thinking tokens at inference by suppressing or inserting end-of-thinking delimiters. Evaluation uses AIME24, MATH500 and GPQA Diamond. These are specific benchmark results, not a guarantee that longer generation improves every answer.

test-time compute

The reported gains illustrate test-time scaling in this experimental setting. Budget forcing appends “Wait” when continuing a generation or inserts an end-of-thinking delimiter when shortening it. “Wait” is text added to the model context, not a wall-clock pause. The generated reasoning can include errors and repetition; the paper measures final-answer accuracy rather than establishing that every reasoning step is correct.

Genesis of s1K: Quality, Difficulty, Diversity

Fundamental to the entire enterprise is a 1,000-sample dataset, named s1K. Hunting for an eclectic mix that covers mathematics (geometry, real analysis, number theory, partial differential equations, etc.) and scientific topics (biology, quantum theory, electromagnetic fields, computer science, and more), the authors set out three guiding data principles:

  1. High Quality: The authors remove formatting problems, ASCII diagrams and questions that depend on missing images. Gemini 2.0 Flash Thinking generates reasoning traces and answers. Model-based grading and benchmark-specific selection help curate the data; they do not certify that every generated solution is correct.
  2. High Difficulty: For the main pool, the authors remove questions answered correctly by either Qwen2.5-7B-Instruct or Qwen2.5-32B-Instruct. An initial set of 384 high-quality questions is exempt from that further filtering, so this is not a rule applied to every final example.
  3. Diversity: The main text describes 50 domains, while Appendix B’s Table 5 labels its distribution as 51 domains. This version contains that discrepancy. The selection aims for coverage across mathematics and science; it does not guarantee coverage of every task.

The authors start with 59,029 questions drawn from 16 sources, including MATH (Hendrycks et al., 2021) and AGIEval (Zhong et al., 2023). The paper also lists OlympiadBench, OmniMath, SciEval and USACO among its inputs. Gemini generation failures reduce the pool to 54,116; quality filtering leaves 51,581. Difficulty filtering and domain-aware selection then produce the 1,000-example s1K set, with the 384-example exception described above.

Long teacher-generated reasoning traces are one selection signal. The paper uses traces from Gemini 2.0 Flash Thinking, not Qwen-generated traces, for s1K training. Appendix B describes benchmark-specific choices, including MATH traces longer than 5,600 tokens. The appendices show model-generated examples in probability, geometry and astronomy; a long trace is a heuristic for difficulty, not proof of a valid solution.

The Pillars of Test-Time Scaling

The central theme is test-time scaling, or how to systematically enlarge the model’s iterative reasoning at inference to amplify accuracy. One can do this in a “sequential” manner, letting the LLM refine its internal chain of thought step by step, or “parallel,” generating many solutions simultaneously and then aggregating them (e.g., by majority vote).

The paper classifies extant or novel methods to scale test-time compute:

  1. Sequential Methods
    • Budget Forcing: The authors’ contribution. This directly manipulates the upper or lower bound on “thinking tokens” by forcibly inserting either end-of-thinking tokens or extra statements like “Wait.” If the LLM tries to end early, they re-append “Wait,” forcing it to reflect further. Conversely, if they want fewer steps, they insert the end-of-thinking delimiter quickly.
    • Step-Conditional Control: Prompts ask for an upper bound on high-level steps, such as “up to” a specified number. The paper’s token-budget proxy assumes 100 tokens per step. A prompt does not enforce an exact number of steps.
    • Token-Conditional Control: The user specifies “Use no more than 1,000 tokens to think,” or some similar directive.
    • Class-Conditional Control: The user indicates “use short thinking” or “use long thinking.”
  2. Parallel Methods
    • Majority Voting: Sample multiple solutions in parallel, then choose the most frequent final answer.
    • Best-of-N Selection: Sample multiple solutions, then pick the final answer that a reward model ranks highest.
    • Tree Search Variants: Such as REBASE (Wu et al., 2024b) or Monte Carlo Tree Search. Often a process-level reward model is used to prune or refine partial solutions.

Budget forcing itself requires neither a reward model nor a search tree. The authors give an example in which continuing generation with “Wait” helps the model correct an answer. That example supports the possibility of self-correction under the tested intervention, not a guarantee that additional reasoning is useful or correct.

test time scaling

Evaluative Framework

To measure the utility and reliability of these scaling methods, the paper proposes three metrics:

  1. Control: The paper measures the proportion of generations whose thinking-token use falls within the requested bounds. This is a benchmark-level measure, not a guarantee of exactly 4,000 tokens or a production latency limit.
  2. Scaling: The paper measures how accuracy changes with additional thinking tokens. Positive scaling in one setting does not imply that doubling tokens always improves accuracy.
  3. Performance: The maximum accuracy observed among the evaluated budgets and settings, rather than an absolute upper bound on the model’s possible accuracy.

Plots relate average thinking tokens to benchmark accuracy. The finite context window limits sequential scaling. In Section 6.2, prompting for up to 512 steps causes responses to exceed the context window on 12 of the 30 AIME24 questions, reducing performance. Repetition and diminishing gains are separate limitations. The usable limit depends on the underlying model and the tokens needed for its answer.

Results: The Emergence of s1-32B

The authors supervised-fine-tune the existing Qwen2.5-32B-Instruct model on s1K; they do not train a 32B model from scratch. They report 26 minutes on 16 NVIDIA H100 GPUs, approximately seven GPU-hours. Training uses reasoning traces and answers, excluding question tokens from the loss. The paper gives the following settings; the reported runtime does not include the cost of base-model pretraining or all teacher generation.

  • A learning rate of 1e−5.
  • Five epochs in total (so about 315 gradient steps).
  • Bfloat16 precision.
  • AdamW optimizer (Loshchilov & Hutter, 2019) with 0.9 / 0.95 betas.

Table 1 reports s1-32B results both without and with budget forcing, alongside Qwen2.5-32B-Instruct, QwQ-32B-Preview and DeepSeek-R1-Distill-Qwen-32B. The fine-tuning data counts differ, but those counts alone are not a comparison of total development cost or base-model training. The following values belong to the paper’s version 2 experiments.

  • AIME24: 50.0% without budget forcing and 56.7% with it, compared with 26.7% for the base Qwen2.5-32B-Instruct model in Table 1.
  • MATH500: 92.6% without budget forcing and 93.0% with it; the base model records 84.0%.
  • GPQA Diamond: 56.6% without budget forcing and 59.6% with it; the base model records 49.0%.

GPQA Diamond contains 198 graduate-level biology, chemistry and physics multiple-choice questions; it is not an open-ended benchmark. Version 2 reports the 69.7% domain-expert figure by citing OpenAI, 2024, rather than conducting a new human evaluation. Table 1 lists DeepSeek-R1-Distill-Qwen-32B, trained on 800,000 reasoning samples, at 72.6% on AIME24, 94.3% on MATH500 and 62.1% on GPQA Diamond. s1-32B’s smaller fine-tuning set is an experimental result, not evidence that it matches that model on every benchmark.

Budget Forcing in Action

One central experiment is a demonstration of budget forcing. The authors systematically run s1-32B on AIME24 while forcing minimum and maximum token budgets. For example:

  1. Minimum Bound (“Wait”)
    When a generation reaches the end-of-thinking delimiter too early, the authors suppress it and append “Wait” to continue generation. The extrapolation experiment varies the number of interventions, including 2x, 4x and 6x. AIME24 rises from 50.0% to 56.7% before gains flatten. The counts refer to interventions, not exact multiples of reasoning length.
  2. Maximum Bound
    At a token limit, the method inserts the end-of-thinking delimiter and prompts the model to produce its final answer. Table 3 records 100% control for budget forcing under the evaluated bounds. This constrains thinking-token generation in that experiment, not every component of runtime or answer length.

Table 1’s MATH500 comparison is 92.6% without budget forcing versus 93.0% with it. GPQA Diamond changes from 56.6% to 59.6%. The results vary by benchmark, and additional compute eventually gives diminishing returns.

The underlying model’s finite context window leaves less room for an answer as the generated trace grows. The paper documents repetition when extending generation and the 12-of-30 context-window failures in its 512-step comparison. The study does not fix a single thinking-to-answer allocation for all requests.

Comparison to Other Scaling Methods

The paper also examines alternative ways to harness more test-time compute:

  1. Parallel Approaches
    Section 6.2 augments the sequentially scaled s1-32B model with majority voting and REBASE, aggregating 16 trajectories. Both parallel methods build on s1-32B in this comparison. REBASE uses a process reward model initialized from LLaMA-34B and adds a reward-model forward pass at each step. Figure 7 excludes that extra reward-model compute from its thinking-token axis; REBASE scales better than the compared sequential setting there.
  2. Token-/Step-Conditional Control
    The authors test token bounds and prompts requesting “up to” a number of steps. The model may make a few steps verbose or many steps brief. Table 3 reports control of 40% for token-conditional and 60% for step-conditional prompting, compared with 100% for budget forcing. Prompted step counts are not exact enforced reasoning units.
  3. Rejection Sampling
    The method samples at temperature 1 until a generation fits a token bound. The authors observe inverse scaling and many rejected samples. Figure 6’s caption lists an average of 655 tries for its shortest-budget end, but contains five threshold labels and six try counts. That internal caption mismatch should not be treated as a complete unambiguous threshold-to-count mapping.

Table 3 reports budget forcing at 100% control, a scaling value of 15 and a maximum AIME24 performance of 56.7% under that comparison. Other tested methods score lower there. Section 6.2 separately shows REBASE complementing sequential scaling at larger budgets, with additional reward-model compute. Neither comparison proves one method is best for all tasks or budgets.

Further Investigations and Future Directions

While the authors celebrate test-time scaling’s potential, they identify limitations:

  1. Flattening Gains: In the reported AIME24 extrapolation, performance reaches 56.7% and further interventions flatten out. This is a result of those runs, not a universal accuracy ceiling or proof that every extra trace contains deeper reasoning.
  2. Context Window Restrictions: The underlying context window limits how far sequential generation can be extended. Section 6.2 proposes parallel scaling as one way to complement it. External scratchpads or streaming memory would be separate design proposals, not systems evaluated in this study.
  3. Risk of Loops or Repetitions: The paper observes unproductive repetition and suggests varying continuation strings, frequency penalties or temperature as possible improvements. Their effectiveness would require further evaluation.

The authors identify reinforcement-learning-trained reasoning models as a future direction for testing budget forcing and extrapolation. They also evaluate a process-reward-model search approach through REBASE. These are distinct from claiming that s1-32B itself was trained with reinforcement learning.

The paper proposes Control, Scaling and Performance as metrics for comparing test-time-scaling methods. Reporting them together helps distinguish adherence to a budget from accuracy and further scaling.

s1K Data Ablations

Table 2 compares data-selection strategies under its stated setup. s1K records 50.0% on AIME24, 93.0% on MATH500 and 57.6% on GPQA Diamond. Random 1,000-example selection records 36.7%, 90.6% and 52.0%; diverse selection records 26.7%, 91.2% and 54.6%. The full roughly 59,000-example set records 53.3%, 92.8% and 58.1%. The ablation uses different settings from Table 1’s budget-forcing comparison.

Broader Comparisons and the Road Ahead

This paper engages with multiple parallel efforts:

  • OpenAI’s o1 Family (OpenAI, 2024). Version 2 discusses it as a contemporary reasoning-model comparison. This historical summary does not establish current access or a new model ranking.
  • DeepSeek-R1 (DeepSeek-AI et al., 2025; https://arxiv.org/abs/2501.12948). The paper cites its reinforcement-learning approach and compares a distilled Qwen-32B model in Table 1. Those are different models and should not be conflated.
  • QwQ-32B-Preview (Qwen Team, 2024). This is the model listed in version 2’s comparison, not a claim about every later QwQ release.
  • Sky-T1 (NovaSky Team, 2025). The paper’s related-work discussion reports a $450 training-cost example. That attributed figure is not s1’s cost and is not a complete cost audit of teacher or base-model development.
  • Bespoke-Stratos (Bespoke Labs, 2025). The paper discusses it among contemporary reasoning-distillation efforts, separately from its own s1K experiments.

The paper calls s1-32B sample-efficient because it achieves these reported results after fine-tuning on 1,000 examples. It connects the result to LIMA’s Superficial Alignment Hypothesis. That connection is a research interpretation, not proof that a small dataset always unlocks every reasoning capability.

As an editorial implication, carefully selected task-specific data may be worth studying alongside larger training sets. The paper’s ablation supports that question in its settings; it does not establish a general rule that 1,000 examples beat tens of thousands on any domain.

Scientifically Open Collaboration

The authors describe releasing code, model weights and data, and acknowledge Stanford’s Marlowe GPU cluster. Appendix A shows that greedy-decoding outputs can change with batching, tensor parallelism or continuation in vLLM; a seed alone does not resolve all such differences. This summary reviews the reported reproducibility limits without claiming to have reproduced training or inference.

Conclusion: Taming Complexity with Simple Tools

The study shows how a curated fine-tuning set and a simple inference intervention can improve the authors’ measured benchmark results. It does not establish an optimal reasoning length for an arbitrary query. A longer generated trace can correct an answer, repeat an error or run into the context limit.

Parallel sampling and process-reward-model search offer other ways to spend inference compute, with their own overhead. External memory or scratchpads would be additional research designs. The paper’s experiments do not demonstrate unbounded generation, a 100k-token system or reliable eventual self-correction.

Ultimately, the study’s contribution weaves together:

  • A 1,000-example curated dataset, s1K, with teacher-generated reasoning traces and answers.
  • A fine-tuned model, s1-32B, evaluated on three benchmarks under the paper’s stated settings.
  • A comparison framework for sequential and parallel test-time scaling, plus budget forcing for constraining or extending generated thinking tokens.

This synergy of fine-grained data curation, short but potent training, and robust inference manipulations stands as a powerful statement. It invites anyone fascinated by large language models, mathematical or scientific AI breakthroughs, or new directions in open-source collaboration to explore further.

References to Linkable Sources Cited

  • DeepSeek-AI et al. (2025). DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. https://arxiv.org/abs/2501.12948. Reference cited in s1 version 2.
  • Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., & Mirhoseini, A. (2024). Large language monkeys: Scaling inference compute with repeated sampling. https://arxiv.org/abs/2407.21787.
  • Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., Phang, J., Reynolds, L., Tang, E., Thite, A., Wang, B., Wang, K., & Zou, A. (2021). A framework for few-shot language model evaluation. https://doi.org/10.5281/zenodo.5371628.
  • Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021). Measuring mathematical problem solving with the math dataset. https://arxiv.org/abs/2103.03874.
  • Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. (2023). AGIEval: A human-centric benchmark for evaluating foundation models. https://arxiv.org/abs/2304.06364.
  • Wu, Y., Sun, Z., Li, S., Welleck, S., & Yang, Y. (2024b). Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. https://arxiv.org/abs/2408.00724.
  • Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with pagedattention. https://arxiv.org/abs/2309.06180.
  • Kapfer, C., Stine, K., Narasimhan, B., Mentzel, C., & Candes, E. (January 2025). Marlowe: Stanford’s gpu-based computational instrument. https://doi.org/10.5281/zenodo.14751899.
  • Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. https://arxiv.org/abs/1711.05101.
  • OpenAI (September 2024). Learning to reason with LLMs. https://openai.com/index/learning-to-reason-with-llms/. Historical reference cited by the paper.
  • Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., & Bowman, S. R. (2023). Gpqa: A graduate-level google-proof q&a benchmark. https://arxiv.org/abs/2311.12022.
  • Google (December 2024). Gemini 2.0 Flash Thinking mode (gemini-2.0-flash-thinking-exp-1219). https://cloud.google.com/vertex-ai/generative-ai/docs/thinking-mode. Teacher endpoint named in the February 2025 paper.
  • DeepSeek Team (November 2024). Historical R1 announcement cited in the version 2 bibliography: https://x.com/deepseek_ai/status/1859200141355536422. The January 2025 R1 paper is listed separately above.
  • Bespoke Labs (2025). Bespoke-Stratos: The unreasonable effectiveness of reasoning distillation. https://hf.co/bespokelabs/Bespoke-Stratos-32B. Historical related-work reference from the paper.

Related Guides

Recent Launches

Latest News

Continue Reading