What Self-Consistency Prompting Is
📍 In One Sentence
Self-consistency prompting runs the same question several times at a non-zero temperature and takes the answer that appears most often, using agreement across independent attempts as a proxy for correctness.
💬 In Plain Terms
Ask the same question five times and go with the answer that keeps coming back. A model can talk itself into a wrong answer once; it is much less likely to make the same wrong turn five times in a row.
Self-consistency prompting means sampling several independent answers to the same prompt and selecting the most consistent conclusion. Rather than one chain of thought, you get multiple, potentially different chains.
The idea is simple: if the model reasons in several different ways and most paths converge on the same answer, that answer is more trustworthy than a single run. If the paths disagree, you know the problem is ambiguous or difficult and needs closer review.
Self-consistency was introduced by Wang et al. (ICLR 2023) and reported accuracy gains on arithmetic and commonsense reasoning benchmarks. The technique leverages a fundamental principle from statistics: a consensus of many independent estimates is more reliable than a single estimate.
Why Self-Consistency Prompting Matters
Self-consistency prompting matters because language models can be unstable on hard reasoning tasks—small changes in sampling can flip the answer. By looking at a set of attempts instead of one, you reduce the impact of any single hallucination or mistake.
- Math and logic puzzles.
- Multi-step analytical questions.
- Decisions with subtle trade-offs where small reasoning slips change the outcome.
- Any domain-specific reasoning where single-pass accuracy is below 90% (rule of thumb).
🔍 Pro Tip
You don't need to manually compare 10 outputs. Add a final aggregation step: paste all N answers into a new prompt and ask "These are 10 answers to the same question. Which answer appears most frequently? State the consensus answer and your confidence level." The model can do the counting for you; check its tally on important questions.
What the Paper Reports
Wang et al. (2023) compared self-consistency (40 sampled reasoning paths per question, majority vote) with greedy chain-of-thought decoding on arithmetic benchmarks. The table shows the paper-reported PaLM-540B results; we did not reproduce them.
Gains are largest where chain-of-thought alone is weakest (GSM8K +17.9, AQuA +12.5 points) and small where it is already strong (AddSub +1.8). The abstract also reports +6.4 on StrategyQA and +3.9 on ARC-challenge. These are 2022-era models and settings, so they do not predict gains on your task or on current models. The paper notes that self-consistency costs more computation and suggests starting with 5 or 10 paths, since performance saturates quickly.
Benchmark | CoT (greedy) | Self-consistency | Gain |
|---|---|---|---|
| GSM8K | 56.5% | 74.4% | +17.9 |
| AQuA | 35.8% | 48.3% | +12.5 |
| SVAMP | 79.0% | 86.6% | +7.6 |
| ASDiv | 74.0% | 81.9% | +7.9 |
| MultiArith | 94.7% | 99.3% | +4.6 |
| AddSub | 91.9% | 93.7% | +1.8 |
🔍 Did You Know
On GSM8K, PaLM-540B rose from 56.5% (greedy chain-of-thought) to 74.4% (self-consistency) in the paper, with no fine-tuning and no extra training data: only sampling and voting. Figures are paper-reported, not our own measurements.
How Self-Consistency Prompting Works in Practice
In practice, self-consistency prompting follows a two-phase pattern: generate diverse answers, then aggregate them. You keep the task prompt the same but allow randomness so the model explores different reasoning paths.
A typical flow:
- 1Use a reasoning-style prompt (often with chain-of-thought instructions) and set temperature to 0.7-1.0 so the model produces varied explanations. Temperature controls randomness: 0 = deterministic (same answer every time), 1.0 = maximum diversity.
- 2Run the same prompt multiple times (for example 5–20) and collect all final answers. Each run should be independent — different temperature samples, not cached results.
- 3Aggregate: count which answer appears most frequently, or cluster similar answers. Use the majority-vote answer as your final result.
- 4Optionally, ask the model to reconcile disagreements: "These are 10 answers to the same question. Which appears most frequently? Any reasons for disagreement?" This adds confidence metadata.
Self-Consistency vs Multi-Model Consensus
Self-consistency samples the SAME model multiple times. Multi-model consensus samples DIFFERENT models once each. Both apply the same principle — majority voting over diverse reasoning paths — but they catch different failure modes.
PromptQuorum enables multi-model consensus natively — dispatch one prompt to multiple models and compare. For critical decisions, combine both: run self-consistency within your primary model AND check the consensus answer against a second model.
Approach | How It Works | What It Catches | Blind Spots |
|---|---|---|---|
| Self-consistency (single model) | Same prompt, same model, 5-20 runs at T=0.7+ | Sampling instability, random errors | Systematic model bias (same bias in every sample) |
| Multi-model consensus | Same prompt, different models, 1 run each | Model-specific biases, architectural blind spots | All models may share the same training data gap |
| Combined (broadest coverage) | Multiple models × multiple samples each | Both random errors AND systematic biases | Cost: N models × M samples = N×M API calls |
When to Use Self-Consistency Prompting
You should use self-consistency prompting when the cost of a wrong answer is high and the task involves non-trivial reasoning. It trades compute and latency for better robustness.
Good candidates include:
- Analytical questions driving business or technical decisions.
- Complex coding tasks where logical mistakes are expensive.
- Educational or exam-style reasoning where intermediate steps matter.
- Any workflow where you have already observed that single runs are unstable.
- Math problems, logic puzzles, research synthesis, financial analysis.
Technique | Samples | Cost | Best For | Accuracy Gain |
|---|---|---|---|---|
| Single answer (baseline) | 1 | 1× | Simple tasks, low stakes | — |
| Chain-of-thought | 1 | 1 call, longer output | Math, logic, step-by-step | Baseline in the paper |
| Self-consistency | 5-20 | 5-20× | Hard reasoning, high stakes | Up to +17.9 pts on GSM8K (paper) |
| Multi-model consensus | 3-5 models | 3-5× | Catching model-specific biases | Task-dependent |
| Both combined | 5 × 3 models | 15× | Broadest failure-mode coverage | Task-dependent |
⚠️ Warning
Self-consistency at temperature 0 is useless — every sample produces the identical output. Set temperature above 0; the paper sampled at T=0.5-0.7, and 0.7 is a common starting point. Temperature 0 is a frequent implementation mistake.
Common Mistakes With Self-Consistency Prompting
Here are the pitfalls that undermine self-consistency and how to avoid them:
- Using temperature 0 (deterministic mode). Why it hurts: Every sample is identical. Voting on 10 identical answers tells you nothing. Fix: Set temperature to 0.7-1.0 to generate diverse reasoning paths.
- Using self-consistency for simple factual questions. Why it hurts: "What is the capital of France?" produces "Paris" every time. You spent 10× the tokens for no accuracy gain. Fix: Reserve self-consistency for tasks where single-run accuracy is observably below 90% (rule of thumb).
- Generating too few samples (2-3). Why it hurts: With 2 samples that disagree, you have no tiebreaker. With 3, a 2-1 split gives weak consensus. Fix: Use at least 5 samples; the paper suggests starting with 5 or 10 paths because most of the gain arrives early.
- Voting on the full response text instead of the final answer. Why it hurts: Two responses may reach the same answer via completely different reasoning paths. Text comparison says they're different; answer comparison says they agree. Fix: Extract only the final answer (require "Answer: X" format) and vote on that.
Self-Consistency Prompting in PromptQuorum
PromptQuorum is a multi-model AI dispatch tool that naturally complements self-consistency prompting by letting you generate and compare multiple answers easily. You can treat "multiple runs from one model" and "multiple models on one prompt" as two layers of consistency checks.
With PromptQuorum, you can:
- Reuse a reasoning-focused framework (such as TRACE or APE) and run it several times per model to collect diverse chains of thought.
- Run the same reasoning prompt across several models side by side to see whether they converge on the same answer.
- Save self-consistency workflows as templates, so your team can repeatedly apply "sample multiple times, then aggregate" without designing the pattern from scratch.
How to Use Self-Consistency Prompting
- 1For complex reasoning tasks, generate multiple outputs (5–10) from the same prompt with different random seeds. Ask the model the same question 5 times. You'll get 5 different responses.
- 2Analyze the outputs to find consistent patterns (the 'consensus'). If 4 of 5 responses agree on an answer, that agreement is your confidence signal. If all 5 disagree, the task is ambiguous or the prompt needs refinement.
- 3Use Self-Consistency to flag possible hallucinations in research and knowledge tasks. If asking 'What is the capital of France?' and 3 responses say 'Paris' while 2 say 'Lyon,' the majority (Paris) is your candidate answer. If the answers scatter across many different cities, treat the model as unreliable on that question. Agreement is a warning signal, not proof: a model can repeat the same wrong answer every time.
- 4Set Temperature (T) higher (0.7–1.0) to encourage diverse outputs. Lower temperatures (T = 0) produce the same deterministic output every time, defeating the purpose. Self-Consistency needs variation to find consensus.
- 5Implement self-consistency in production pipelines where cost allows. Running 5–10× more generations is expensive, but for high-stakes analysis such as research synthesis, the consensus signal can justify the cost. Medical, legal and financial decisions still need review by a qualified professional; majority voting does not replace it.
Sources
- Wang et al. (2023). "Self-Consistency Improves Chain of Thought Reasoning in Language Models." ICLR 2023. arXiv:2203.11171 — the foundational paper introducing self-consistency with majority voting over reasoning paths
- Wei et al. (2022). "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." NeurIPS 2022. arXiv:2201.11903 — the chain-of-thought paper that self-consistency builds upon
- Brown et al. (2020). "Language Models are Few-Shot Learners." NeurIPS 2020. arXiv:2005.14165 — foundational work on in-context learning that enables both CoT and self-consistency
Frequently Asked Questions
What is self-consistency prompting?
Self-consistency prompting is a technique where you generate multiple independent answers to the same question — each with its own reasoning path — and then select the answer that appears most frequently. Instead of trusting one AI response, you trust the consensus of many. It was introduced by Wang et al. (2023), who reported accuracy gains on arithmetic and commonsense reasoning benchmarks.
How many samples do I need for self-consistency?
For most tasks, start with 5-10 samples. The original paper notes that performance saturates quickly as paths are added and suggests starting with 5 or 10; every extra sample costs a full extra response. Start with 5; increase to 10-20 only for high-stakes decisions.
Does self-consistency work on simple tasks?
Not meaningfully. For factual lookups, simple classification, or short-form writing, a single answer is almost always sufficient and much cheaper. As a rule of thumb (our heuristic, not a figure from the paper), self-consistency adds value mainly on tasks where the model's single-pass accuracy is below ~90% — typically math, logic puzzles, multi-step analysis, and complex reasoning.
What temperature should I use for self-consistency?
Set temperature to 0.7-1.0. The technique requires diverse reasoning paths — if temperature is 0 (deterministic), every sample produces the identical output and voting is meaningless. Higher temperature creates the variation that makes majority voting informative.
How much more does self-consistency cost?
Roughly 5-20× more tokens per task, since you generate 5-20 complete responses instead of one. For a response that costs $0.01, self-consistency at 10 samples costs $0.10. This can be justified for high-stakes analysis but is wasteful for routine tasks. For medical, legal or financial decisions, a qualified professional must review the result; this article is general information, not professional advice.
Is self-consistency the same as "best-of-N" sampling?
Similar but not identical. Best-of-N generates N responses and selects the best one (often by a quality scorer). Self-consistency generates N reasoning paths and selects the most common ANSWER — the voting is on the conclusion, not on quality. Self-consistency doesn't need a quality scorer; it uses agreement as the signal.
Can I use self-consistency with chain-of-thought prompting?
Yes — this is the combination used in the original paper. Each of your N samples uses chain-of-thought reasoning, producing a full reasoning trace plus a final answer. You then vote on the final answers across all N traces. The reasoning paths may differ, but if most reach the same conclusion, that conclusion is more likely to be correct, though never guaranteed.
How does PromptQuorum relate to self-consistency?
PromptQuorum applies the same consensus principle across different models instead of within one model. Instead of asking the same model 10 times, you ask 5 different models once each and compare their answers. Where they agree, confidence is high. Where they disagree, the claim needs verification. This catches model-specific biases that single-model self-consistency cannot detect.
