As AI moves from recognizing well-defined objects to judging complaints, business priorities and answer adequacy, enterprises need more than a larger supply of labels: they need executable decision rules. This is an enduring governance issue, not a claim of recent news. Our recommendation is to identify the cause of disagreement before deciding whether to remove it. Labels made uniform through adjudication cannot establish that the original judgments were independent and consistent.
Buy a reproducible judgment process, not just a label count
For a measurement task, a rule can specify that a dimension outside tolerance is unacceptable. In a language task, however, “the customer is very dissatisfied” and “this case requires immediate escalation” are different labels. The former describes an expression; the latter depends on operational responsibility and escalation conditions. Combining them lets conscientious annotators answer different questions.
Artstein and Poesio distinguish agreement from validity: observers may agree and still be wrong.[1] Plank’s position paper argues that some language-label variation reflects subjectivity or multiple plausible interpretations rather than removable noise.[2] These studies do not establish that every enterprise dataset should retain all disagreement. They do expose a procurement distinction between receiving a final label and receiving evidence of a dependable annotation process.
That distinction changes budgets. Quality checks can address careless entry. When essential context is absent, more annotators may merely reproduce guesses. When business policy is unresolved, further model training cannot take over the policy owner’s responsibility. A small annotation pilot can reveal these differences before a large fixed-price labeling order is placed.
Separate the task, visible evidence and action label
Before the pilot, freeze a short label contract: the unit being judged, the business decision time, permitted input fields, whether categories are mutually exclusive, how insufficient information is represented, who adjudicates and what operation will use the label. Provide positive, negative and boundary examples for every category, rather than definitions alone.
Consider this constructed ticket: “Yesterday you said it would arrive today. It still has not arrived, and production is waiting.” The text supplies evidence of a delivery complaint. Deciding whether it represents an urgent line-stoppage event also requires the actual production status, alternative inventory and an approved escalation rule. Two annotators asked to establish a stoppage from text alone may disagree because necessary facts are invisible, not because one lacks competence.
A possible design separates whether the text expresses a delivery complaint from whether approved policy requires escalation. The first field records textual evidence; the second identifies required operational fields and remains pending when those fields are missing. This is an author-proposed workflow, not a statement about an existing IDENIFE product feature.
Annotation information must also be related to information available at prediction time. If the final disposition is used as supervision, assess whether the initial input can reasonably predict it. Reviewers with the complete subsequent history cannot set an unqualified certainty requirement for a model acting at initial submission. For related data controls, see evaluation independence and version acceptance for training datasets.
Explain agreement with a cross-tabulation, not a universal cutoff
Construct a set of 100 records independently labeled by two people as “escalate” or “do not escalate.” They both choose no escalation for 90 records and escalation for two. On four records only annotator A chooses escalation; on four only B does. Raw agreement is 92%. Yet eight of the ten records flagged by at least one person are disputed—the region that may matter most operationally.
For Cohen’s κ, each annotator selects escalation on 6% of records. Agreement expected from their marginal proportions is 0.06×0.06 + 0.94×0.94 = 0.8872. Thus κ = (0.92 − 0.8872)/(1 − 0.8872), approximately 0.291. This hand-calculated example illustrates distribution effects; it is not field evidence. A low coefficient alone does not identify an incompetent annotator, and a high raw agreement rate does not establish reliability for the critical class.[1]
Report the original cross-tabulation, class counts, number of independent judgments, missing judgments and exclusion rules, together with disagreements that change business actions. Multiple annotators, ordinal scores, multilabel tasks and missing ratings require an appropriate metric and, where relevant, distance definition. Do not transfer a binary coefficient mechanically or use one generic cutoff for all procurement decisions.
Sampling changes interpretation too. Oversampling boundary cases helps diagnosis but alters the meaning of the overall agreement rate. Report a production-representative evaluation sample separately from a deliberately difficult challenge set. Several near-identical tickets from one customer are not fully independent evidence.
Route disagreement to four different remedies
The first route addresses execution errors: missed negations, accidental selections or evidence attached to the wrong record. Check the underlying material, correct the label and retain the original judgment, reason and responsible reviewer. Examine whether the same error recurs. Reviewing only disputed records will miss errors shared by both annotators.
The second addresses insufficient information: absent attachments, missing temporal context or unestablished facts. Add information that the task legitimately permits and obtain new independent judgments. If production cannot obtain that information, narrow the task or retain an indeterminate state. Do not quietly supply evaluation-only fields unavailable online.
The third addresses policy conflict: departments interpret urgency, ownership or exceptions differently. The accountable policy owner must resolve the operational choice, publish a new guide and relabel the affected scope. Adjudication establishes an actionable policy; it does not necessarily make the earlier minority interpretation factually wrong.
The fourth addresses legitimate alternatives: a text supports multiple readings or the task measures preferences. Possible responses include retaining several acceptable answers, a label distribution or role-specific judgments, then selecting an evaluation method suited to the use. Basile and colleagues analyze disagreement arising from annotators, data and context, supporting diagnosis before aggregation.[3] The four-route enterprise workflow proposed here is an editorial recommendation, not a substitute for domain review.
Adjudicated uniformity is not an independent retest
Once an owner changes all disputed labels to one answer, the dataset can look perfectly consistent. That measures the adjudication outcome, not whether other people can apply the new guide independently. Keep the pre-adjudication judgments, select fresh cases not used in the rule discussions and relabel independently under the same information conditions. Compare like-for-like case groups across rounds.
Showing a model suggestion before annotation may also cause several people to accept the same error. On a review subset, hide suggestions and compare independent with assisted annotation on time, errors and consequential disagreement. Labels produced largely by accepting model suggestions are not equivalent evidence to independent human evaluation.
A versioned delivery should include the task and guide versions, input snapshot, raw judgments, evidence locations, adjudication records, final labels and unresolved cases. Retain only the identities and access needed when records contain personal evaluations or sensitive business material. After a substantive guide change, old evaluation scores and new labels must not be joined into an unqualified improvement trend.
Decide whether to expand, narrow or pause
Expansion is justified when major causes of disagreement are understood, new cases can be handled under the rules, consequential errors meet pre-agreed requirements and the cost per usable label is affordable. Include pilot work, review, adjudication, relabeling and version maintenance rather than counting only the first labeling action.
Narrowing is appropriate when some categories can be judged reliably while others depend on missing information. A model may handle the evidenced categories while people or an information-gathering process handle the rest. Report the resulting reduction in automation coverage. Removing difficult cases to improve a score cannot support a claim of full production coverage.
Pause expansion when policy ownership is unresolved, essential evidence cannot be obtained or adjudication backlog keeps growing. The next investment should follow the diagnosis: acquire information, simplify categories, resolve policy or switch to decision support. Annotation agreement helps reveal whether a task is executable. The delivery goal remains a bounded, reviewable business judgment.