Quick Facts
📍 In One Sentence
Chain-of-thought prompting asks the model to work through intermediate steps before answering, which improves accuracy on multi-step reasoning at the cost of more tokens and slower responses.
💬 In Plain Terms
It is telling the model to show its work. Made to reason out loud, it catches its own mistakes partway through instead of blurting a confident guess — which is why it helps on maths and logic and does nothing for looking up a fact.
- 1Introduced by: Wei et al. (2022), Google Research (Brain Team) — NeurIPS 2022 paper
- 2Key result: Zero-shot CoT ("Let's think step by step") raised InstructGPT (text-davinci-002) accuracy on MultiArith from 17.7% to 78.7% (Kojima et al., 2022)
- 3Trigger phrase: "Let's think step by step" (zero-shot) or worked examples (few-shot)
- 42026 evolution: Built-in reasoning modes (OpenAI GPT-6 Astra, Claude adaptive thinking) now automate CoT internally
- 5Cost impact: Prompt-level CoT = more output tokens; reasoning modes = separate thinking token budget
- 6Best models for prompt-level CoT: Models without a built-in reasoning mode, such as the Llama 4 Scout instruct release and many small local models
What Chain-of-Thought Prompting Is
Chain-of-thought prompting asks the model to reason step by step before giving a final conclusion. Instead of returning only "the answer," the model writes out intermediate calculations, logical steps, or explanations.
You can trigger this behavior by instructions like "think step by step," "show your reasoning," or by providing worked examples where the reasoning is explicit. The result is a written trace you can read and check, although it does not always reflect how the model actually reached its answer (Turpin et al., 2023).
Why Chain-of-Thought Prompting Matters
Chain-of-thought prompting matters because it makes model behavior more transparent on tasks that involve multi-step reasoning. When you see each step, you can spot misinterpretations, missing assumptions, or arithmetic errors.
This is especially valuable in domains like analytics, planning, and troubleshooting. Instead of a single opaque output, you get a narrative that can be checked, corrected, or reused as documentation.
🔍 Works with local models
Chain-of-thought gains depend on model size: Wei et al. (2022) found that it only helped models of around 100B parameters or more, and that smaller models often wrote fluent but illogical steps. Results vary by model and task, so test local models on your own prompts with Ollama or LM Studio.
When Chain-of-Thought Helps (and When It Doesn't)
Chain-of-thought prompting helps most on tasks that naturally break into clear steps, but it is not necessary for every prompt. It shines wherever the path is as important as the destination.
Chain-of-Thought reasoning is what lets a tool-calling agent stay reliable across multi-step tasks. For a local agent stack that gives a CoT-capable model the ability to query databases and edit files, see Local AI Agents With MCP.
Good use cases include:
- Math and quantitative reasoning problems.
- Multi-step logical puzzles or decision analyses.
- Root-cause analysis, incident postmortems, and trade-off discussions.
- Planning tasks where the sequence of actions must be explicit.
For simple classification, quick copywriting, or short factual answers, chain-of-thought often adds verbosity without much extra value. In sensitive domains, you may also want to keep reasoning internal and show only the final answer to end users.
Example: Without vs With Chain of Thought
The difference becomes clear when you compare a direct-answer prompt with one that explicitly asks for reasoning. Here is a simple decision example.
Bad Prompt
"Which project should we prioritize next quarter?"
Good Prompt
"You are a product operations manager. We have three candidate projects for next quarter. Use chain-of-thought reasoning to decide which project to prioritize. 1) List the decision criteria you will use (for example revenue impact, risk, alignment with strategy). 2) Evaluate each project against these criteria step by step. 3) Make a clear recommendation and justify it in 3–5 sentences. At the end, provide a short final answer starting with `Recommendation:` on a separate line."
In the "good" version, the model explains how it chose its criteria, how each project scores, and then states a recommendation you can challenge or accept.
How to Write Effective Chain-of-Thought Prompts
To write effective chain-of-thought prompts, you should define the structure of the reasoning and the structure of the final answer. Vague requests like "explain more" are less reliable than concrete instructions.
A practical pattern is:
- Tell the model its role (for example "You are a senior data analyst.").
- Specify that it should think step by step or use chain-of-thought.
- Define the sections of reasoning you expect (for example assumptions, calculations, comparison, conclusion).
- Ask for a short, clearly marked final answer at the end so you can use it quickly.
This separates the detailed reasoning from the concise output, which is helpful when you integrate the result into other tools or reports.
Chain-of-Thought Prompting in PromptQuorum
PromptQuorum is a multi-model AI dispatch tool where you can apply chain-of-thought prompting consistently across different models. You write one structured chain-of-thought prompt and send it to several providers in parallel.
In PromptQuorum, you can:
- Combine chain-of-thought instructions with reasoning-focused frameworks such as TRACE or APE so that thinking steps are explicitly labeled.
- Compare how different models handle the same reasoning task and inspect their step-by-step traces side by side.
- Save chain-of-thought prompts as templates for recurring analyses, incident reviews, or strategic decisions.
This turns chain-of-thought prompting from a one-off trick into a repeatable part of your decision-making process.
How to Use Chain-of-Thought (CoT) Prompting
- 1For logic, reasoning, or debugging tasks, ask the model to 'think step by step' before answering. Instead of 'What is the bug?', ask 'Trace the execution step by step, then identify the bug.'
- 2Provide a worked example showing step-by-step reasoning. Don't just describe it—show the model what step-by-step reasoning looks like. Example: 'First, I check the function signature... Then, I trace the first call with input X...'
- 3Use explicit prompts like 'Let's think step by step' or 'First, identify... Then...' These trigger more deliberate reasoning in the model.
- 4For complex problems, ask the model to trace intermediate outputs. Example: 'Trace the execution of this function for input 5. Show the value of each variable after each line.'
- 5Combine CoT with verifiable outputs: ask the model to show its work so you can audit it. 'Explain your reasoning at each step. If you make a mistake, I should be able to spot it from your shown work.'
Math Example: Revenue Calculation
Without CoT, a model might give a single final answer. With CoT, the model shows calculations step by step.
Without CoT:
"A customer buys 50 units at $15 each, but gets a 10% discount. What do they pay?"
Model: "$675"
With CoT:
"A customer buys 50 units at $15 each, but gets a 10% discount. Work through this step by step: 1) Calculate the subtotal. 2) Calculate the discount amount. 3) Subtract the discount from the subtotal to get the final price."
Model: "1) Subtotal = 50 × $15 = $750. 2) Discount = 10% of $750 = $75. 3) Final price = $750 − $75 = $675."
Both give the same answer, but the CoT version lets you see the math and catch errors (e.g., if someone miscalculates 10% of $750).
CoT vs Built-In Reasoning Models (2026)
In 2026, reasoning-capable models—Claude Opus 5.5, OpenAI's reasoning models (for example GPT-6 Astra) and Gemini 3.1 Pro thinking—have built-in reasoning modes that internalize chain-of-thought automatically. You usually do not need to add "think step by step" instructions on these models.
When to use prompt-level CoT: Models without a built-in reasoning mode (for example the Llama 4 Scout instruct release), local LLMs, models where thinking is switched off or set to a low effort, or when you want to avoid the extra cost of thinking tokens.
When to use built-in reasoning modes: Hard tasks on frontier models, where the built-in mode usually beats a plain prompt (compare both on your own task), math-heavy tasks, complex analysis. Claude and OpenAI bill reasoning (thinking) tokens as output tokens, so cost grows with how long the model thinks. Test your CoT prompts in Anthropic's Console or OpenAI's Playground before deploying to production.
Approach | Best For | Cost | Transparency | Models |
|---|---|---|---|---|
| Prompt-level CoT ("think step by step") | Small models, local LLMs, cost-sensitive tasks | Increases output tokens | Full: visible steps in output | Llama 4 Scout instruct, small local models |
| Claude adaptive thinking (Opus 5.5, Sonnet 5.5) | Complex analysis | Thinking tokens billed as output tokens | Summarized thinking via API | Claude Opus 5.5, Claude Sonnet 5.5 |
| OpenAI GPT-6 Astra | Hardest problems (math, coding, competition) | Reasoning tokens billed as output tokens | Raw reasoning hidden; summaries available | OpenAI GPT-6 Astra |
| Gemini 3.1 Pro thinking | Google Cloud integration, Gemini ecosystem | Thinking tokens billed as output tokens | thinking_level parameter (LOW, MEDIUM, HIGH) | Gemini 3.1 Pro |
| DeepSeek-R1 | Open-weights option, self-hosted reasoning | Visible reasoning streamed as output text | Full: inline CoT in output | DeepSeek-R1 |
💡 Pro Tip
If you're building for cost, use prompt-level CoT on smaller models. If you're building for accuracy on hard problems, use GPT-6 Astra or Claude adaptive thinking and let the model handle reasoning internally.
Chain-of-Thought Variants and Extensions
Beyond the basic "think step by step" pattern, researchers have developed several CoT variants, each optimized for different problem types.
- Zero-shot CoT (Kojima et al., 2022): Add "Let's think step by step" with no examples. Helps mainly on larger models and is the simplest to implement. In the paper it raised InstructGPT (text-davinci-002) accuracy on MultiArith from 17.7% to 78.7% and on GSM8K from 10.4% to 40.7%.
- Few-shot CoT (Wei et al., 2022): Show a few worked examples where the reasoning is explicit, then ask the model to apply the same pattern to a new problem. More reliable than zero-shot but requires writing examples by hand. With eight exemplars, a 540B-parameter model set a new state of the art on the GSM8K math benchmark, ahead of fine-tuned GPT-3 with a verifier.
- Automatic CoT (Zhang et al., 2022): Generates the worked examples itself by sampling diverse questions and answering them with the zero-shot cue. On ten reasoning benchmarks with GPT-3 it matched or exceeded hand-written demonstrations.
- Self-consistency (Wang et al., 2023): Sample several independent CoT reasoning paths, then take a majority vote on the final answer. The paper reports gains over standard CoT of +17.9% on GSM8K, +11.0% on SVAMP, +12.2% on AQuA, +6.4% on StrategyQA and +3.9% on ARC-Challenge.
- Tree of Thoughts (Yao et al., 2023): Instead of one linear chain, explore several reasoning branches and prune poor ones. On the Game of 24 task, GPT-4 with CoT prompting solved 4% of tasks, versus 74% with Tree of Thoughts. Use it for planning, search and puzzle-style problems.
- ReAct (Yao et al., 2023): Interleave reasoning with external actions (call APIs, search databases, run code) and feed the results into the next reasoning step. Best for real-world tasks that need live data or verification.
Model Comparison: How Models Handle CoT Prompting (2026)
Model | Prompt-Level CoT | Built-In Reasoning | Best Use Case | Cost (approx.) |
|---|---|---|---|---|
| Claude Opus 5.5 | Usually not needed | Adaptive thinking | Deep analysis | Higher (input + output + thinking tokens) |
| Claude Sonnet 5.5 | Usually not needed | Adaptive thinking | Balanced accuracy/cost | Medium |
| Claude Haiku 5.5 | Usually not needed | Adaptive thinking | Fast, cost-effective reasoning | Low |
| OpenAI GPT-6 Astra | Usually not needed | Effort levels (low, medium, high, xhigh, max) | Competition-level problems | Reasoning tokens billed as output (see pricing) |
| GPT-6 Luna | Optional | Adjustable reasoning effort | Budget-conscious deployment | Efficiency-focused (see pricing) |
| Gemini 3.1 Pro | Works | Thinking levels (thinking_level param) | Google Cloud integration | Medium-high |
| Gemini Flash | Optional | Adjustable thinking levels | Fast responses | Low |
| DeepSeek-R1 | Usually not needed | Inline reasoning in output | Open-weights, self-hosted | MIT-licensed weights; hardware costs apply |
| Llama 4 | Recommended | None in the base instruct releases | Local deployment, privacy | Self-hosted (compute-dependent) |
Frequently Asked Questions
Does chain-of-thought work on all models?
Not equally well. Wei et al. (2022) found that chain-of-thought prompting only gave gains with models of around 100B parameters or more, and that smaller models often wrote fluent but illogical steps. Results vary by model and task, so test small or local models on your own prompts. On frontier models with built-in reasoning (Claude Opus 5.5, GPT-6 Astra), the built-in mode often makes prompt-level CoT unnecessary.
Does chain-of-thought increase cost?
Yes. Prompt-level CoT increases output tokens because the model writes out its reasoning before the final answer. Built-in reasoning modes (Claude adaptive thinking, OpenAI GPT-6 Astra) also spend thinking tokens, which Anthropic and OpenAI bill as output tokens. Test both on your use case to compare cost against accuracy.
When should I use few-shot CoT instead of zero-shot?
Use zero-shot CoT first—it's the simpler option. Move to few-shot (typically 2–5 examples) if zero-shot is unreliable or if your domain requires specific reasoning patterns (e.g., financial analysis with standard line-item structure).
Can I combine chain-of-thought with structured output (JSON)?
Yes. You can ask the model to show its reasoning in plain text first, then output a JSON object with the final answer. Combine instructions: "Think step by step. Then output your result as valid JSON."
What's the difference between chain-of-thought and tree-of-thought?
Chain-of-thought is a linear sequence: step 1 → step 2 → ... → conclusion. Tree-of-thought explores multiple branches (alternative reasoning paths) and prunes weaker ones before arriving at the answer. Tree-of-thought can help on harder search-style problems but costs more (requires multiple model calls).
Does OpenAI GPT-6 Astra require chain-of-thought prompting?
Usually not. OpenAI's reasoning models, such as GPT-6 Astra, have built-in reasoning that activates automatically. You typically do not need to add "think step by step" instructions. Just give GPT-6 Astra the problem and set the effort level (low/medium/high/xhigh/max) to control how much reasoning budget to spend.
Can I audit the reasoning of built-in reasoning models?
Yes, but it depends on the model. Claude can return thinking blocks through the API, summarized depending on the model and settings. OpenAI does not expose raw reasoning tokens and offers summaries instead. For reasoning you can read in the output, use prompt-level CoT or an open model that streams its reasoning, such as DeepSeek-R1. Written reasoning is a useful check, but it is not proof of how the model actually arrived at its answer.
Is chain-of-thought prompting suitable for real-time applications?
Prompt-level CoT adds latency (more output tokens = slower generation). For real-time use cases, either use smaller models with minimal reasoning, or use stream endpoints to show tokens as they arrive. Built-in reasoning modes may add even more latency; benchmark your specific use case.
What is the difference between chain-of-thought and prompt chaining?
Chain-of-thought asks one model to reason step by step inside a single prompt. Prompt chaining splits a task into several separate prompts, where each output feeds the next prompt. You can combine them: use chain-of-thought inside one step of a chain when that step needs multi-step reasoning.
Sources & Further Reading
- Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022). "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." NeurIPS 2022. arXiv:2201.11903
- Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). "Large Language Models are Zero-Shot Reasoners." NeurIPS 2022. arXiv:2205.11916
- Wang, X., Wei, J., Schuurmans, D., et al. (2023). "Self-Consistency Improves Chain of Thought Reasoning in Language Models." ICLR 2023. arXiv:2203.11171
- Zhang, Z., Zhang, A., Li, M., & Smola, A. (2022). "Automatic Chain of Thought Prompting in Large Language Models." arXiv preprint. arXiv:2210.03493
- Yao, S., Yu, D., Zhao, J., et al. (2023). "Tree of Thoughts: Deliberate Problem Solving with Large Language Models." NeurIPS 2023. arXiv:2305.10601
- Yao, S., Zhao, J., Yu, D., et al. (2023). "ReAct: Synergizing Reasoning and Acting in Language Models." ICLR 2023. arXiv:2210.03629
- Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting." NeurIPS 2023. arXiv:2305.04388
- Anthropic. Claude API documentation: Thinking and Extended thinking.
- OpenAI. API documentation: Reasoning models.
