Skip to main content
PromptQuorumBuilt for humans. Structured for AI.
Home/Prompt Engineering
Techniques

Self-Consistency Prompting: Generate Multiple Answers, Pick the One That's Right

·12 min readBy · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI

Self-consistency prompting asks the model the same question several times at a temperature above zero, then picks the majority answer. In the original paper, 40 samples raised PaLM-540B on GSM8K from 56.5% to 74.4%. Cost: one full response per sample.

Key Takeaways

  • Self-consistency samples 5-20 answers to the same prompt and picks the majority answer, using agreement between attempts as a reliability signal.
  • The technique was introduced by Wang et al. (2023). The paper reports PaLM-540B on GSM8K rising from 56.5% (greedy chain-of-thought) to 74.4% (self-consistency, 40 sampled paths).
  • Use higher temperature (0.7-1.0) to generate diverse reasoning paths — low temperature produces identical outputs, defeating the purpose.
  • Best for: math, logic, multi-step analysis, and any task where single runs are observably unstable.
  • Trade-off: 5-20× more tokens consumed per task — use only when accuracy matters more than cost.
  • PromptQuorum extends this concept across models: instead of sampling one model multiple times, dispatch to multiple models and compare consensus.
  • Self-consistency catches sampling instability; multi-model consensus catches model-specific biases. Combining both covers more failure modes.

Self-consistency prompting samples several independent reasoning paths for the same question and selects the answer that appears most often. Instead of trusting one answer that may be wrong, you ask several times at a higher temperature and let majority voting decide.

Self-Consistency Prompting: Generate Multiple Answers, Pick the One That's Right

⚡ Quick Facts

  • ·Technique: Generate 5-20 independent reasoning paths for the same question, then select the most frequent answer via majority voting.
  • ·Paper: Wang et al. (2023), "Self-Consistency Improves Chain of Thought Reasoning in Language Models," ICLR 2023.
  • ·Key result (paper-reported): PaLM-540B on GSM8K rose from 56.5% (greedy chain-of-thought) to 74.4% (self-consistency, 40 sampled paths), a gain of 17.9 points.
  • ·Temperature requirement: Must be above 0 (temperature=0 produces identical outputs, defeating the purpose); the paper sampled at T=0.5-0.7.
  • ·Sample count: The paper suggests starting with 5 or 10 paths, since performance saturates quickly.
  • ·Cost: 5-20× more tokens per task; justified only for high-stakes reasoning where accuracy matters more than cost.

What Self-Consistency Prompting Is

📍 In One Sentence

Self-consistency prompting runs the same question several times at a non-zero temperature and takes the answer that appears most often, using agreement across independent attempts as a proxy for correctness.

💬 In Plain Terms

Ask the same question five times and go with the answer that keeps coming back. A model can talk itself into a wrong answer once; it is much less likely to make the same wrong turn five times in a row.

Self-consistency prompting means sampling several independent answers to the same prompt and selecting the most consistent conclusion. Rather than one chain of thought, you get multiple, potentially different chains.

The idea is simple: if the model reasons in several different ways and most paths converge on the same answer, that answer is more trustworthy than a single run. If the paths disagree, you know the problem is ambiguous or difficult and needs closer review.

Self-consistency was introduced by Wang et al. (ICLR 2023) and reported accuracy gains on arithmetic and commonsense reasoning benchmarks. The technique leverages a fundamental principle from statistics: a consensus of many independent estimates is more reliable than a single estimate.

Why Self-Consistency Prompting Matters

Self-consistency prompting matters because language models can be unstable on hard reasoning tasks—small changes in sampling can flip the answer. By looking at a set of attempts instead of one, you reduce the impact of any single hallucination or mistake.

  • Math and logic puzzles.
  • Multi-step analytical questions.
  • Decisions with subtle trade-offs where small reasoning slips change the outcome.
  • Any domain-specific reasoning where single-pass accuracy is below 90% (rule of thumb).

🔍 Pro Tip

You don't need to manually compare 10 outputs. Add a final aggregation step: paste all N answers into a new prompt and ask "These are 10 answers to the same question. Which answer appears most frequently? State the consensus answer and your confidence level." The model can do the counting for you; check its tally on important questions.

What the Paper Reports

Wang et al. (2023) compared self-consistency (40 sampled reasoning paths per question, majority vote) with greedy chain-of-thought decoding on arithmetic benchmarks. The table shows the paper-reported PaLM-540B results; we did not reproduce them.

Gains are largest where chain-of-thought alone is weakest (GSM8K +17.9, AQuA +12.5 points) and small where it is already strong (AddSub +1.8). The abstract also reports +6.4 on StrategyQA and +3.9 on ARC-challenge. These are 2022-era models and settings, so they do not predict gains on your task or on current models. The paper notes that self-consistency costs more computation and suggests starting with 5 or 10 paths, since performance saturates quickly.

Benchmark
CoT (greedy)
Self-consistency
Gain
GSM8K56.5%74.4%+17.9
AQuA35.8%48.3%+12.5
SVAMP79.0%86.6%+7.6
ASDiv74.0%81.9%+7.9
MultiArith94.7%99.3%+4.6
AddSub91.9%93.7%+1.8

🔍 Did You Know

On GSM8K, PaLM-540B rose from 56.5% (greedy chain-of-thought) to 74.4% (self-consistency) in the paper, with no fine-tuning and no extra training data: only sampling and voting. Figures are paper-reported, not our own measurements.

How Self-Consistency Prompting Works in Practice

In practice, self-consistency prompting follows a two-phase pattern: generate diverse answers, then aggregate them. You keep the task prompt the same but allow randomness so the model explores different reasoning paths.

A typical flow:

  1. 1
    Use a reasoning-style prompt (often with chain-of-thought instructions) and set temperature to 0.7-1.0 so the model produces varied explanations. Temperature controls randomness: 0 = deterministic (same answer every time), 1.0 = maximum diversity.
  2. 2
    Run the same prompt multiple times (for example 5–20) and collect all final answers. Each run should be independent — different temperature samples, not cached results.
  3. 3
    Aggregate: count which answer appears most frequently, or cluster similar answers. Use the majority-vote answer as your final result.
  4. 4
    Optionally, ask the model to reconcile disagreements: "These are 10 answers to the same question. Which appears most frequently? Any reasons for disagreement?" This adds confidence metadata.

Self-Consistency vs Multi-Model Consensus

Self-consistency samples the SAME model multiple times. Multi-model consensus samples DIFFERENT models once each. Both apply the same principle — majority voting over diverse reasoning paths — but they catch different failure modes.

PromptQuorum enables multi-model consensus natively — dispatch one prompt to multiple models and compare. For critical decisions, combine both: run self-consistency within your primary model AND check the consensus answer against a second model.

Approach
How It Works
What It Catches
Blind Spots
Self-consistency (single model)Same prompt, same model, 5-20 runs at T=0.7+Sampling instability, random errorsSystematic model bias (same bias in every sample)
Multi-model consensusSame prompt, different models, 1 run eachModel-specific biases, architectural blind spotsAll models may share the same training data gap
Combined (broadest coverage)Multiple models × multiple samples eachBoth random errors AND systematic biasesCost: N models × M samples = N×M API calls

When to Use Self-Consistency Prompting

You should use self-consistency prompting when the cost of a wrong answer is high and the task involves non-trivial reasoning. It trades compute and latency for better robustness.

Good candidates include:

  • Analytical questions driving business or technical decisions.
  • Complex coding tasks where logical mistakes are expensive.
  • Educational or exam-style reasoning where intermediate steps matter.
  • Any workflow where you have already observed that single runs are unstable.
  • Math problems, logic puzzles, research synthesis, financial analysis.
Technique
Samples
Cost
Best For
Accuracy Gain
Single answer (baseline)11×Simple tasks, low stakes—
Chain-of-thought11 call, longer outputMath, logic, step-by-stepBaseline in the paper
Self-consistency5-205-20×Hard reasoning, high stakesUp to +17.9 pts on GSM8K (paper)
Multi-model consensus3-5 models3-5×Catching model-specific biasesTask-dependent
Both combined5 × 3 models15×Broadest failure-mode coverageTask-dependent

⚠️ Warning

Self-consistency at temperature 0 is useless — every sample produces the identical output. Set temperature above 0; the paper sampled at T=0.5-0.7, and 0.7 is a common starting point. Temperature 0 is a frequent implementation mistake.

Common Mistakes With Self-Consistency Prompting

Here are the pitfalls that undermine self-consistency and how to avoid them:

  • Using temperature 0 (deterministic mode). Why it hurts: Every sample is identical. Voting on 10 identical answers tells you nothing. Fix: Set temperature to 0.7-1.0 to generate diverse reasoning paths.
  • Using self-consistency for simple factual questions. Why it hurts: "What is the capital of France?" produces "Paris" every time. You spent 10× the tokens for no accuracy gain. Fix: Reserve self-consistency for tasks where single-run accuracy is observably below 90% (rule of thumb).
  • Generating too few samples (2-3). Why it hurts: With 2 samples that disagree, you have no tiebreaker. With 3, a 2-1 split gives weak consensus. Fix: Use at least 5 samples; the paper suggests starting with 5 or 10 paths because most of the gain arrives early.
  • Voting on the full response text instead of the final answer. Why it hurts: Two responses may reach the same answer via completely different reasoning paths. Text comparison says they're different; answer comparison says they agree. Fix: Extract only the final answer (require "Answer: X" format) and vote on that.

Self-Consistency Prompting in PromptQuorum

PromptQuorum is a multi-model AI dispatch tool that naturally complements self-consistency prompting by letting you generate and compare multiple answers easily. You can treat "multiple runs from one model" and "multiple models on one prompt" as two layers of consistency checks.

With PromptQuorum, you can:

  • Reuse a reasoning-focused framework (such as TRACE or APE) and run it several times per model to collect diverse chains of thought.
  • Run the same reasoning prompt across several models side by side to see whether they converge on the same answer.
  • Save self-consistency workflows as templates, so your team can repeatedly apply "sample multiple times, then aggregate" without designing the pattern from scratch.

How to Use Self-Consistency Prompting

  1. 1
    For complex reasoning tasks, generate multiple outputs (5–10) from the same prompt with different random seeds. Ask the model the same question 5 times. You'll get 5 different responses.
  2. 2
    Analyze the outputs to find consistent patterns (the 'consensus'). If 4 of 5 responses agree on an answer, that agreement is your confidence signal. If all 5 disagree, the task is ambiguous or the prompt needs refinement.
  3. 3
    Use Self-Consistency to flag possible hallucinations in research and knowledge tasks. If asking 'What is the capital of France?' and 3 responses say 'Paris' while 2 say 'Lyon,' the majority (Paris) is your candidate answer. If the answers scatter across many different cities, treat the model as unreliable on that question. Agreement is a warning signal, not proof: a model can repeat the same wrong answer every time.
  4. 4
    Set Temperature (T) higher (0.7–1.0) to encourage diverse outputs. Lower temperatures (T = 0) produce the same deterministic output every time, defeating the purpose. Self-Consistency needs variation to find consensus.
  5. 5
    Implement self-consistency in production pipelines where cost allows. Running 5–10× more generations is expensive, but for high-stakes analysis such as research synthesis, the consensus signal can justify the cost. Medical, legal and financial decisions still need review by a qualified professional; majority voting does not replace it.

Sources

Frequently Asked Questions

What is self-consistency prompting?

Self-consistency prompting is a technique where you generate multiple independent answers to the same question — each with its own reasoning path — and then select the answer that appears most frequently. Instead of trusting one AI response, you trust the consensus of many. It was introduced by Wang et al. (2023), who reported accuracy gains on arithmetic and commonsense reasoning benchmarks.

How many samples do I need for self-consistency?

For most tasks, start with 5-10 samples. The original paper notes that performance saturates quickly as paths are added and suggests starting with 5 or 10; every extra sample costs a full extra response. Start with 5; increase to 10-20 only for high-stakes decisions.

Does self-consistency work on simple tasks?

Not meaningfully. For factual lookups, simple classification, or short-form writing, a single answer is almost always sufficient and much cheaper. As a rule of thumb (our heuristic, not a figure from the paper), self-consistency adds value mainly on tasks where the model's single-pass accuracy is below ~90% — typically math, logic puzzles, multi-step analysis, and complex reasoning.

What temperature should I use for self-consistency?

Set temperature to 0.7-1.0. The technique requires diverse reasoning paths — if temperature is 0 (deterministic), every sample produces the identical output and voting is meaningless. Higher temperature creates the variation that makes majority voting informative.

How much more does self-consistency cost?

Roughly 5-20× more tokens per task, since you generate 5-20 complete responses instead of one. For a response that costs $0.01, self-consistency at 10 samples costs $0.10. This can be justified for high-stakes analysis but is wasteful for routine tasks. For medical, legal or financial decisions, a qualified professional must review the result; this article is general information, not professional advice.

Is self-consistency the same as "best-of-N" sampling?

Similar but not identical. Best-of-N generates N responses and selects the best one (often by a quality scorer). Self-consistency generates N reasoning paths and selects the most common ANSWER — the voting is on the conclusion, not on quality. Self-consistency doesn't need a quality scorer; it uses agreement as the signal.

Can I use self-consistency with chain-of-thought prompting?

Yes — this is the combination used in the original paper. Each of your N samples uses chain-of-thought reasoning, producing a full reasoning trace plus a final answer. You then vote on the final answers across all N traces. The reasoning paths may differ, but if most reach the same conclusion, that conclusion is more likely to be correct, though never guaranteed.

How does PromptQuorum relate to self-consistency?

PromptQuorum applies the same consensus principle across different models instead of within one model. Instead of asking the same model 10 times, you ask 5 different models once each and compare their answers. Where they agree, confidence is high. Where they disagree, the claim needs verification. This catches model-specific biases that single-model self-consistency cannot detect.

Apply these techniques with a local LLM or your own API keys — PromptQuorum works with any backend.

Try PromptQuorum free →

← Back to Prompt Engineering

MCP