Before paying for more models to vote on the same question, establish that the additional members correct existing errors without introducing too many shared mistakes. Different brands and more responses do not supply that evidence. For tasks with verifiable answers, compare paired errors and complete costs. When evidence is missing, rules are ambiguous or consequences are substantial, improving the information and adjudication process is usually more useful than adding similar votes.
Define the vote before choosing the panel size
This article concerns combining several model outputs for one input, such as deciding whether an enterprise knowledge passage supports an answer. It differs from selecting one model for a request: enterprise model routing allocates resources, while voting aggregates results. They can be combined, but savings from routing do not establish a quality benefit from voting.
Freeze the answer unit first. With just “supported” and “unsupported” labels, three responses allow a direct majority decision. With paragraphs of advice, similar wording does not establish agreement on conditions, amounts or applicable versions. Break the output into checkable facts or conclusions and define how uncertainty, refusals, parsing failures and timeouts are handled. Do not discard difficult-to-count responses after seeing the results.
Self-consistency research samples several reasoning paths from one model and aggregates their final answers, improving performance on particular arithmetic and commonsense tasks.[1] This adds candidate paths rather than external facts; the paper also discusses the conditions associated with a fixed answer set. It is a candidate method for enterprise evaluation, but a majority saying “supported” is not itself a probability of support.
Also freeze data access, source versions and the judgment rubric. If one model receives the full document and another only a summary, their difference includes an input change. When comparing different evidence-gathering methods is intentional, make those paths explicit components of the alternatives and record them separately. Otherwise, better data preparation can be incorrectly credited entirely to panel size.
Research supports experiments with combinations, alongside counterexamples
Du and colleagues' ICML 2024 study had multiple model instances propose answers and discuss them over several rounds, observing improvements in reasoning and factual performance on their tested tasks.[2] A comparison by Smit and colleagues in the same year found that debate strategies did not consistently outperform baselines such as self-consistency, with results sensitive to parameters and protocol choices.[3] Together, these findings support task-specific comparison rather than assuming debate must help or cannot help.
Kim and colleagues' ICML 2025 study examined multiple model collections and found correlated errors. On one of its leaderboard datasets, pairs chose the same wrong answer about 60% of the time conditional on both being wrong.[4] The denominator is questions both models answered incorrectly, not all questions, and the figure is not a universal correlation coefficient for enterprise models. It motivates inspecting shared failures, not pricing a particular panel using that percentage.
On August 26, 2026, Amazon Science researchers described aggregation that models dependencies between judges as well as individual reliability, with comparisons on three binary tasks.[5] This offers a richer alternative to equal-weight voting, but estimating relationships requires enough data. Those results do not eliminate an enterprise's need for independent reference judgments.
None of this establishes that a specific enterprise should procure three model providers today. Tasks, versions, prompts and evidence sources change the ways models fail together. The next step is to translate “more opinions” into incremental corrections on the same questions, while also counting previously correct answers that the combination makes wrong. That produces a testable investment rationale.
A 1,000-item ledger: what does the majority actually correct?
Construct 1,000 knowledge-passage support judgments, each with a binary reference label checked against the source material. All three candidates return valid labels within the deadline. Let A, B and C denote models; 1 means matching the reference and 0 means disagreeing. The bit order is always ABC.
The eight outcome groups contain: 111, 800 items; 110, 40; 101, 35; 011, 30; 100, 25; 010, 20; 001, 15; and 000, 35. Their sum is 800+40+35+30+25+20+15+35=1,000. A is correct on 900 items, B on 890 and C on 880. The majority is correct when all three or any two are correct: 800+40+35+30=905 items.
Relative to A, the panel corrects 30 items in group 011 but corrupts 25 previously correct answers in group 100. The net gain is five correct answers, taking accuracy from 90% to 90.5%. Showing only “30 additional corrections” hides the 25 regressions. Showing only each member's overall score hides both directions of change. Keep corrected, damaged and unchanged outcomes in the acceptance record.
Of A's 100 errors, B is also wrong on the 15 items in group 001 and the 35 in group 000: 50 in total. B's error rate conditional on A being wrong is therefore 50%, whereas B's overall error rate is only 11%. In this input set, their failures do not spread like independent draws. This conditional proportion helps diagnose the panel, but does not establish shared training data as the cause. Commonly difficult questions or missing evidence are alternative explanations.
At least one model is correct on 965 items, but the majority is correct on only 905. The 60-item gap shows why having a correct candidate requires a reliable selection mechanism before it becomes a delivery benefit. The 965 answers one could choose after seeing the reference describe the available candidate set, not a deliverable accuracy commitment.
Why independence and unanimous acceptance cannot simply be assumed
Temporarily assume independent model errors with fixed probabilities of 0.10, 0.11 and 0.12. The probability of an incorrect majority is then 0.10×0.11+0.10×0.12+0.11×0.12−2×0.10×0.11×0.12=0.03356. Subtracting twice the all-wrong probability corrects the first three terms, which count that outcome three times when it should count once.
This independent model predicts approximately 33.56 majority errors per 1,000 items; the constructed ledger contains 95. Correct arithmetic does not establish independence. Entering individual accuracy figures into a formula and obtaining an attractive ensemble score produces only a conditional calculation. Paired outputs are needed to assess whether the task resembles those assumptions. A mixture of easy and difficult questions also calls for stratified inspection.
Another policy delivers only unanimous results and sends disagreements to people. Under this example's binary-label and complete-response assumptions, both 111 and 000 are unanimous: 835 items, including 35 errors. The error share within delivered results is 35÷835≈4.19%, with 165 items referred for review. The delivered set becomes cleaner, but automated coverage falls. Its denominator differs from the 9.5% error rate of majority voting over the entire set.
This simplification does not hold for multiple-choice or free-text outputs: three wrong models can give three different answers. Production evaluation must also count unsupported answers wrongly approved and supported answers wrongly rejected. The eight correctness groups alone cannot identify those two rates. If false approval has greater consequences, give it a separate acceptance condition rather than allowing many correct rejections to dilute it.
Put corrections, calls and review in the same investment ledger
Extend the example into a budget scenario with 20,000 similar inputs per month and unchanged error proportions. Assume A costs CNY 0.04 per call, B and C each cost CNY 0.03, aggregation adds CNY 0.005, and additional panel maintenance costs CNY 1,200 per month. These are scenario budget assumptions. A alone costs CNY 800 in calls; three-model processing costs 20,000×0.105=CNY 2,100. Including added maintenance, the extra expenditure is CNY 2,500.
Scaling the sample proportions gives 2,000 monthly errors for A and 1,900 for the majority, a net reduction of 100. At CNY 20 of average rework per error, A's calls plus rework total CNY 40,800. The majority costs 2,100+1,200+1,900×20=CNY 41,300, or CNY 500 more. This temporarily treats other handling costs for correct results as equal and assumes errors are found and reworked. A real ledger must add undiscovered errors and distinguish severity.
Under these assumptions, each net avoided error must prevent more than 2,500÷100=CNY 25 of loss to cover the additional expense. At CNY 60 avoided per error, net benefit becomes 100×60−2,500=CNY 3,500. Halving traffic while holding quality proportions fixed reduces extra calling fees to CNY 650, but maintenance remains CNY 1,200. Only 50 errors are avoided, raising the break-even amount to CNY 37. Fixed expenditure makes modest correction gains harder to justify at low volume.
The unanimous-only policy also needs a staffing calculation: 20,000×16.5%=3,300 reviews, at two minutes each, require 110 hours. A team with 40 available hours can process only 1,200, leaving 2,100 backlogged. Paying CNY 80 per hour would cost CNY 8,800 for all 110 hours, but review quality and deadlines still require verification. Funding does not automatically supply available, qualified people.
When errors involve unacceptable unauthorized actions or business commitments, expected monetary loss cannot replace hard controls. First satisfy boundaries prohibiting automatic release, then compare costs among eligible paths. The budget ledger informs investment; it is not permission to trade away any risk at a low price.
Keep one model, add voting, or change the evidence
Keeping one model is appropriate when existing quality meets the use case, errors predominantly reflect missing material, or extra members reproduce the same failures. Fixing retrieval coverage, document versions and field validation may help more directly than additional calls. Retaining the current design still requires audits and an error ledger; declining to add models does not mean ending verification.
Independent initial answers followed by voting fit tasks with normalizable outputs, demonstrated complementarity, and affordable additional latency and cost. Have each model answer the same frozen input independently before applying the predefined aggregation. If the second model sees the first conclusion, do not treat their votes as independent observations. For multi-round discussion, preserve before-and-after outputs to identify both corrections and cases where persuasion turns a correct answer wrong.
Different evidence is useful where rules and sources can be checked externally. Use deterministic calculations for numerical totals, approved master data for model applicability, original conditions for citation support, and a responsible owner to adjudicate disputed definitions. The investment may involve rules, better material or specialist review rather than another model. Different brands reading the same defective source cannot manufacture missing facts.
Weighted or dependence-aware aggregation is a fourth candidate, but fit weights on development data and compare on independent samples. Increasing the weight of whichever model was right after inspecting acceptance answers, then reporting performance on those same answers, supplies no launch evidence. Complex aggregation earns its extra cost only when it continues to improve on both simple majority voting and the single-model baseline, with manageable maintenance responsibility.
Three failure classes and how to retest them
Shared source errors: all models receive an outdated document and may unanimously accept an expired condition. Audits must sample unanimous approvals, not just disagreement queues, and check source versions and effective dates. Correct the evidence path, then rerun the original error, neighboring versions and normal cases. Switching providers while retaining the obsolete material does not repair the failure.
A deteriorating member: a model may perform worse on a new language or product line and outvote a correct minority. Inspect both “baseline right, panel wrong” and the reverse by slice, alongside missing responses, timeouts and label parsing. Restrict panel coverage, replace a member or restore the single-model path as appropriate. Retest on fresh independent samples rather than merely deleting the questions that exposed the issue.
Agreement produced by negotiation: after discussion, all members may adopt the same argument while losing an initially correct objection. Retain initial answers, changes by round and cited evidence; inspect correct-to-incorrect trajectories. Possible corrections include limiting mutual influence, retaining minority opinions for review, or adding source checks. Rerun independent-answer and discussion conditions on the same questions and examine net change rather than final agreement alone.
Also test whether service failures silently alter the voting rule. If one of three models times out, define in advance whether to wait, fall back or send the item to a person. One vote among two remaining responses is not a three-vote majority. Account for costs, end-to-end deadlines and delivery status across complete requests, including missing calls, to avoid evaluating only surviving results.
Give procurement and operations a concrete panel acceptance record
The first group of fields covers task and inputs: answer unit, allowed labels, source version, business slices, consequences, and timeout or uncertainty handling. The second covers the panel: model and prompt versions, visibility of initial answers, aggregation rule and member-absence policy. Each field should point to a reproducible case or configuration record, not simply say “multi-model cross-checking.”
The third group records evidence: single-model baseline, net panel corrections, correct-to-incorrect changes, unanimous errors, disagreements, false approvals by label, and sample counts per slice. Reference labels need checkable sources and explicit rules. Separate development from final acceptance, grouping near-duplicate versions of one document together. Mark a slice with insufficient evidence as unconfirmed rather than substituting an overall score.
The fourth covers resources and decisions: monthly volume, actual calls and retries, added maintenance, review minutes, available staff hours, end-to-end deadlines and exit conditions. Proceed only when evidence supports net correction, critical failures are controlled, staffing is sufficient and the complete ledger supports investment. Improve evidence first if shared mistakes dominate; retain the simpler path if maintenance consumes the benefit.
Assign an owner and a review date to panel changes. Recheck paired errors and review queues after model, rubric, input-distribution or source changes, and preserve comparison records when reverting. Enterprises should procure verifiable incremental correction and delivery capacity, not a growing number of agreeing votes.
References
- [1] Wang et al. — Self-Consistency Improves Chain of Thought Reasoning in Language Models (ICLR 2023; arXiv v4)
- [2] Du et al. — Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024)
- [3] Smit et al. — Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs (ICML 2024)
- [4] Kim et al. — Correlated Errors in Large Language Models (ICML 2025)
- [5] Balasubramanian and Podkopaev — When LLM judges agree, should we believe them? (Amazon Science, 2026-08-26)
