Skip to main content
PromptQuorumBuilt for humans. Structured for AI.
Home/Prompt Engineering
Techniques

Chain-of-Thought Prompting: Make AI Show Its Reasoning

·13 min readBy · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI

Chain-of-thought (CoT) prompting asks an AI model to show its reasoning step by step before giving a final answer. It improves accuracy on math, logic and multi-step tasks, and matters most on models without a built-in reasoning mode.

Key Takeaways

  • Chain-of-thought prompting asks models to show reasoning steps before giving a final answer, improving accuracy on math, logic, and multi-step tasks.
  • Zero-shot CoT ("think step by step") is the simplest option and helps mainly on larger models. Few-shot CoT (with worked examples) is more reliable.
  • In 2026, reasoning-capable models such as Claude Opus 5.5 and OpenAI's reasoning models (for example GPT-6 Astra) have built-in reasoning modes that usually make prompt-level CoT unnecessary — you typically don't need to say "think step by step" on these models.
  • CoT increases output tokens and therefore cost. Built-in reasoning modes also spend thinking tokens, which Anthropic and OpenAI bill as output tokens.
  • Use prompt-level CoT on models without a built-in reasoning mode, such as many small and locally run models, for low-cost reasoning. On frontier models, the built-in reasoning mode usually gives higher accuracy on hard tasks than a plain prompt; compare both on your own task.
  • CoT is most valuable for math, logic, planning, and root-cause analysis. Skip it for simple classification, short answers, and quick copywriting.

Chain-of-thought prompting is a technique where you explicitly ask the model to show its reasoning steps instead of jumping straight to the final answer. This makes complex decisions easier to audit, debug, and improve over time.

Chain-of-Thought Prompting: Make AI Show Its Reasoning

Quick Facts

📍 In One Sentence

Chain-of-thought prompting asks the model to work through intermediate steps before answering, which improves accuracy on multi-step reasoning at the cost of more tokens and slower responses.

💬 In Plain Terms

It is telling the model to show its work. Made to reason out loud, it catches its own mistakes partway through instead of blurting a confident guess — which is why it helps on maths and logic and does nothing for looking up a fact.

  1. 1
    Introduced by: Wei et al. (2022), Google Research (Brain Team) — NeurIPS 2022 paper
  2. 2
    Key result: Zero-shot CoT ("Let's think step by step") raised InstructGPT (text-davinci-002) accuracy on MultiArith from 17.7% to 78.7% (Kojima et al., 2022)
  3. 3
    Trigger phrase: "Let's think step by step" (zero-shot) or worked examples (few-shot)
  4. 4
    2026 evolution: Built-in reasoning modes (OpenAI GPT-6 Astra, Claude adaptive thinking) now automate CoT internally
  5. 5
    Cost impact: Prompt-level CoT = more output tokens; reasoning modes = separate thinking token budget
  6. 6
    Best models for prompt-level CoT: Models without a built-in reasoning mode, such as the Llama 4 Scout instruct release and many small local models

What Chain-of-Thought Prompting Is

Chain-of-thought prompting asks the model to reason step by step before giving a final conclusion. Instead of returning only "the answer," the model writes out intermediate calculations, logical steps, or explanations.

You can trigger this behavior by instructions like "think step by step," "show your reasoning," or by providing worked examples where the reasoning is explicit. The result is a written trace you can read and check, although it does not always reflect how the model actually reached its answer (Turpin et al., 2023).

Why Chain-of-Thought Prompting Matters

Chain-of-thought prompting matters because it makes model behavior more transparent on tasks that involve multi-step reasoning. When you see each step, you can spot misinterpretations, missing assumptions, or arithmetic errors.

This is especially valuable in domains like analytics, planning, and troubleshooting. Instead of a single opaque output, you get a narrative that can be checked, corrected, or reused as documentation.

🔍 Works with local models

Chain-of-thought gains depend on model size: Wei et al. (2022) found that it only helped models of around 100B parameters or more, and that smaller models often wrote fluent but illogical steps. Results vary by model and task, so test local models on your own prompts with Ollama or LM Studio.

When Chain-of-Thought Helps (and When It Doesn't)

Chain-of-thought prompting helps most on tasks that naturally break into clear steps, but it is not necessary for every prompt. It shines wherever the path is as important as the destination.

Chain-of-Thought reasoning is what lets a tool-calling agent stay reliable across multi-step tasks. For a local agent stack that gives a CoT-capable model the ability to query databases and edit files, see Local AI Agents With MCP.

Good use cases include:

  • Math and quantitative reasoning problems.
  • Multi-step logical puzzles or decision analyses.
  • Root-cause analysis, incident postmortems, and trade-off discussions.
  • Planning tasks where the sequence of actions must be explicit.

For simple classification, quick copywriting, or short factual answers, chain-of-thought often adds verbosity without much extra value. In sensitive domains, you may also want to keep reasoning internal and show only the final answer to end users.

Example: Without vs With Chain of Thought

The difference becomes clear when you compare a direct-answer prompt with one that explicitly asks for reasoning. Here is a simple decision example.

Bad Prompt

"Which project should we prioritize next quarter?"

Good Prompt

"You are a product operations manager. We have three candidate projects for next quarter. Use chain-of-thought reasoning to decide which project to prioritize. 1) List the decision criteria you will use (for example revenue impact, risk, alignment with strategy). 2) Evaluate each project against these criteria step by step. 3) Make a clear recommendation and justify it in 3–5 sentences. At the end, provide a short final answer starting with `Recommendation:` on a separate line."

In the "good" version, the model explains how it chose its criteria, how each project scores, and then states a recommendation you can challenge or accept.

How to Write Effective Chain-of-Thought Prompts

To write effective chain-of-thought prompts, you should define the structure of the reasoning and the structure of the final answer. Vague requests like "explain more" are less reliable than concrete instructions.

A practical pattern is:

  • Tell the model its role (for example "You are a senior data analyst.").
  • Specify that it should think step by step or use chain-of-thought.
  • Define the sections of reasoning you expect (for example assumptions, calculations, comparison, conclusion).
  • Ask for a short, clearly marked final answer at the end so you can use it quickly.

This separates the detailed reasoning from the concise output, which is helpful when you integrate the result into other tools or reports.

Chain-of-Thought Prompting in PromptQuorum

PromptQuorum is a multi-model AI dispatch tool where you can apply chain-of-thought prompting consistently across different models. You write one structured chain-of-thought prompt and send it to several providers in parallel.

In PromptQuorum, you can:

  • Combine chain-of-thought instructions with reasoning-focused frameworks such as TRACE or APE so that thinking steps are explicitly labeled.
  • Compare how different models handle the same reasoning task and inspect their step-by-step traces side by side.
  • Save chain-of-thought prompts as templates for recurring analyses, incident reviews, or strategic decisions.

This turns chain-of-thought prompting from a one-off trick into a repeatable part of your decision-making process.

How to Use Chain-of-Thought (CoT) Prompting

  1. 1
    For logic, reasoning, or debugging tasks, ask the model to 'think step by step' before answering. Instead of 'What is the bug?', ask 'Trace the execution step by step, then identify the bug.'
  2. 2
    Provide a worked example showing step-by-step reasoning. Don't just describe it—show the model what step-by-step reasoning looks like. Example: 'First, I check the function signature... Then, I trace the first call with input X...'
  3. 3
    Use explicit prompts like 'Let's think step by step' or 'First, identify... Then...' These trigger more deliberate reasoning in the model.
  4. 4
    For complex problems, ask the model to trace intermediate outputs. Example: 'Trace the execution of this function for input 5. Show the value of each variable after each line.'
  5. 5
    Combine CoT with verifiable outputs: ask the model to show its work so you can audit it. 'Explain your reasoning at each step. If you make a mistake, I should be able to spot it from your shown work.'

Math Example: Revenue Calculation

Without CoT, a model might give a single final answer. With CoT, the model shows calculations step by step.

Without CoT:

"A customer buys 50 units at $15 each, but gets a 10% discount. What do they pay?"

Model: "$675"

With CoT:

"A customer buys 50 units at $15 each, but gets a 10% discount. Work through this step by step: 1) Calculate the subtotal. 2) Calculate the discount amount. 3) Subtract the discount from the subtotal to get the final price."

Model: "1) Subtotal = 50 × $15 = $750. 2) Discount = 10% of $750 = $75. 3) Final price = $750 − $75 = $675."

Both give the same answer, but the CoT version lets you see the math and catch errors (e.g., if someone miscalculates 10% of $750).

CoT vs Built-In Reasoning Models (2026)

In 2026, reasoning-capable models—Claude Opus 5.5, OpenAI's reasoning models (for example GPT-6 Astra) and Gemini 3.1 Pro thinking—have built-in reasoning modes that internalize chain-of-thought automatically. You usually do not need to add "think step by step" instructions on these models.

When to use prompt-level CoT: Models without a built-in reasoning mode (for example the Llama 4 Scout instruct release), local LLMs, models where thinking is switched off or set to a low effort, or when you want to avoid the extra cost of thinking tokens.

When to use built-in reasoning modes: Hard tasks on frontier models, where the built-in mode usually beats a plain prompt (compare both on your own task), math-heavy tasks, complex analysis. Claude and OpenAI bill reasoning (thinking) tokens as output tokens, so cost grows with how long the model thinks. Test your CoT prompts in Anthropic's Console or OpenAI's Playground before deploying to production.

Approach
Best For
Cost
Transparency
Models
Prompt-level CoT ("think step by step")Small models, local LLMs, cost-sensitive tasksIncreases output tokensFull: visible steps in outputLlama 4 Scout instruct, small local models
Claude adaptive thinking (Opus 5.5, Sonnet 5.5)Complex analysisThinking tokens billed as output tokensSummarized thinking via APIClaude Opus 5.5, Claude Sonnet 5.5
OpenAI GPT-6 AstraHardest problems (math, coding, competition)Reasoning tokens billed as output tokensRaw reasoning hidden; summaries availableOpenAI GPT-6 Astra
Gemini 3.1 Pro thinkingGoogle Cloud integration, Gemini ecosystemThinking tokens billed as output tokensthinking_level parameter (LOW, MEDIUM, HIGH)Gemini 3.1 Pro
DeepSeek-R1Open-weights option, self-hosted reasoningVisible reasoning streamed as output textFull: inline CoT in outputDeepSeek-R1

💡 Pro Tip

If you're building for cost, use prompt-level CoT on smaller models. If you're building for accuracy on hard problems, use GPT-6 Astra or Claude adaptive thinking and let the model handle reasoning internally.

Chain-of-Thought Variants and Extensions

Beyond the basic "think step by step" pattern, researchers have developed several CoT variants, each optimized for different problem types.

  • Zero-shot CoT (Kojima et al., 2022): Add "Let's think step by step" with no examples. Helps mainly on larger models and is the simplest to implement. In the paper it raised InstructGPT (text-davinci-002) accuracy on MultiArith from 17.7% to 78.7% and on GSM8K from 10.4% to 40.7%.
  • Few-shot CoT (Wei et al., 2022): Show a few worked examples where the reasoning is explicit, then ask the model to apply the same pattern to a new problem. More reliable than zero-shot but requires writing examples by hand. With eight exemplars, a 540B-parameter model set a new state of the art on the GSM8K math benchmark, ahead of fine-tuned GPT-3 with a verifier.
  • Automatic CoT (Zhang et al., 2022): Generates the worked examples itself by sampling diverse questions and answering them with the zero-shot cue. On ten reasoning benchmarks with GPT-3 it matched or exceeded hand-written demonstrations.
  • Self-consistency (Wang et al., 2023): Sample several independent CoT reasoning paths, then take a majority vote on the final answer. The paper reports gains over standard CoT of +17.9% on GSM8K, +11.0% on SVAMP, +12.2% on AQuA, +6.4% on StrategyQA and +3.9% on ARC-Challenge.
  • Tree of Thoughts (Yao et al., 2023): Instead of one linear chain, explore several reasoning branches and prune poor ones. On the Game of 24 task, GPT-4 with CoT prompting solved 4% of tasks, versus 74% with Tree of Thoughts. Use it for planning, search and puzzle-style problems.
  • ReAct (Yao et al., 2023): Interleave reasoning with external actions (call APIs, search databases, run code) and feed the results into the next reasoning step. Best for real-world tasks that need live data or verification.

Model Comparison: How Models Handle CoT Prompting (2026)

Model
Prompt-Level CoT
Built-In Reasoning
Best Use Case
Cost (approx.)
Claude Opus 5.5Usually not neededAdaptive thinkingDeep analysisHigher (input + output + thinking tokens)
Claude Sonnet 5.5Usually not neededAdaptive thinkingBalanced accuracy/costMedium
Claude Haiku 5.5Usually not neededAdaptive thinkingFast, cost-effective reasoningLow
OpenAI GPT-6 AstraUsually not neededEffort levels (low, medium, high, xhigh, max)Competition-level problemsReasoning tokens billed as output (see pricing)
GPT-6 LunaOptionalAdjustable reasoning effortBudget-conscious deploymentEfficiency-focused (see pricing)
Gemini 3.1 ProWorksThinking levels (thinking_level param)Google Cloud integrationMedium-high
Gemini FlashOptionalAdjustable thinking levelsFast responsesLow
DeepSeek-R1Usually not neededInline reasoning in outputOpen-weights, self-hostedMIT-licensed weights; hardware costs apply
Llama 4RecommendedNone in the base instruct releasesLocal deployment, privacySelf-hosted (compute-dependent)

Frequently Asked Questions

Does chain-of-thought work on all models?

Not equally well. Wei et al. (2022) found that chain-of-thought prompting only gave gains with models of around 100B parameters or more, and that smaller models often wrote fluent but illogical steps. Results vary by model and task, so test small or local models on your own prompts. On frontier models with built-in reasoning (Claude Opus 5.5, GPT-6 Astra), the built-in mode often makes prompt-level CoT unnecessary.

Does chain-of-thought increase cost?

Yes. Prompt-level CoT increases output tokens because the model writes out its reasoning before the final answer. Built-in reasoning modes (Claude adaptive thinking, OpenAI GPT-6 Astra) also spend thinking tokens, which Anthropic and OpenAI bill as output tokens. Test both on your use case to compare cost against accuracy.

When should I use few-shot CoT instead of zero-shot?

Use zero-shot CoT first—it's the simpler option. Move to few-shot (typically 2–5 examples) if zero-shot is unreliable or if your domain requires specific reasoning patterns (e.g., financial analysis with standard line-item structure).

Can I combine chain-of-thought with structured output (JSON)?

Yes. You can ask the model to show its reasoning in plain text first, then output a JSON object with the final answer. Combine instructions: "Think step by step. Then output your result as valid JSON."

What's the difference between chain-of-thought and tree-of-thought?

Chain-of-thought is a linear sequence: step 1 → step 2 → ... → conclusion. Tree-of-thought explores multiple branches (alternative reasoning paths) and prunes weaker ones before arriving at the answer. Tree-of-thought can help on harder search-style problems but costs more (requires multiple model calls).

Does OpenAI GPT-6 Astra require chain-of-thought prompting?

Usually not. OpenAI's reasoning models, such as GPT-6 Astra, have built-in reasoning that activates automatically. You typically do not need to add "think step by step" instructions. Just give GPT-6 Astra the problem and set the effort level (low/medium/high/xhigh/max) to control how much reasoning budget to spend.

Can I audit the reasoning of built-in reasoning models?

Yes, but it depends on the model. Claude can return thinking blocks through the API, summarized depending on the model and settings. OpenAI does not expose raw reasoning tokens and offers summaries instead. For reasoning you can read in the output, use prompt-level CoT or an open model that streams its reasoning, such as DeepSeek-R1. Written reasoning is a useful check, but it is not proof of how the model actually arrived at its answer.

Is chain-of-thought prompting suitable for real-time applications?

Prompt-level CoT adds latency (more output tokens = slower generation). For real-time use cases, either use smaller models with minimal reasoning, or use stream endpoints to show tokens as they arrive. Built-in reasoning modes may add even more latency; benchmark your specific use case.

What is the difference between chain-of-thought and prompt chaining?

Chain-of-thought asks one model to reason step by step inside a single prompt. Prompt chaining splits a task into several separate prompts, where each output feeds the next prompt. You can combine them: use chain-of-thought inside one step of a chain when that step needs multi-step reasoning.

Sources & Further Reading

  • Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022). "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." NeurIPS 2022. arXiv:2201.11903
  • Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). "Large Language Models are Zero-Shot Reasoners." NeurIPS 2022. arXiv:2205.11916
  • Wang, X., Wei, J., Schuurmans, D., et al. (2023). "Self-Consistency Improves Chain of Thought Reasoning in Language Models." ICLR 2023. arXiv:2203.11171
  • Zhang, Z., Zhang, A., Li, M., & Smola, A. (2022). "Automatic Chain of Thought Prompting in Large Language Models." arXiv preprint. arXiv:2210.03493
  • Yao, S., Yu, D., Zhao, J., et al. (2023). "Tree of Thoughts: Deliberate Problem Solving with Large Language Models." NeurIPS 2023. arXiv:2305.10601
  • Yao, S., Zhao, J., Yu, D., et al. (2023). "ReAct: Synergizing Reasoning and Acting in Language Models." ICLR 2023. arXiv:2210.03629
  • Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting." NeurIPS 2023. arXiv:2305.04388
  • Anthropic. Claude API documentation: Thinking and Extended thinking.
  • OpenAI. API documentation: Reasoning models.

Apply these techniques with a local LLM or your own API keys — PromptQuorum works with any backend.

Try PromptQuorum free →

← Back to Prompt Engineering

MCP