Review Annotation Disagreements Before Adding More Labels

As of 2026-10-23 · The AZET team. We wrote this small design exercise for fun and reference. All messages and reviewer responses are invented. They are not a measured study, a training dataset from customers, or a demonstration of an AZET capability.

Two fictional reviewers label the same note: “The paper lanterns are lovely, but could we make them smaller?” One chooses Praise. The other chooses Request. Both can point to words in the message. Counting the disagreement is easy; explaining it requires inspecting the labeling rules.

Before adding more examples, decide what the labels are supposed to capture. This exercise uses a handful of invented craft-club messages to show how an AI-assisted review can organize that decision without pretending to be an independent judge.

Give the labeling task one purpose

Suppose the club wants to identify messages that need a response. Its first draft offers Praise, Request, and Other, with one label allowed per message. It says nothing about messages that contain both appreciation and a request.

The lantern note exposes that gap. If the purpose is response routing, the request may be the relevant part. If the purpose is studying expressions of appreciation, the praise may matter more. Neither purpose follows automatically from the label names.

Write the task purpose at the top of the guide. Then define each label in terms of that purpose. For this fictional routing exercise, propose Request whenever a message explicitly asks for an action or answer, even if it also contains praise. Treat that as a rule to review, not as a discovery about natural language.

Keep each reviewer’s evidence

Use separate fields for the selected label and the phrase supporting it. Do not overwrite the first review when a second person disagrees.

Message	Review A	Review B	Question for the guide
Lovely lanterns; could they be smaller?	Praise: lovely	Request: could they be smaller	Which part takes priority?
Those lanterns again	Other	Praise	Is positive meaning actually stated?
Please use the small version	Request	Request	What does version refer to?

The third row shows agreement with a remaining uncertainty. Both reviewers can identify a request while the requested object is unclear. Agreement on the label does not establish that the message is ready for action.

The second row lacks enough context to support Praise under the proposed rules. Record that limitation rather than inventing a friendly tone for the sender.

Ask AI to organize the disagreement

Use a prompt that limits the model's authority:

Review these fictional messages, label definitions, and two sets of annotations. Group disagreements into ambiguous label rules, missing message context, and apparent departures from an explicit rule. Cite the exact wording behind each finding. Keep unresolved cases unresolved. Suggest questions for the guide owner; do not infer sender emotions or treat majority agreement as proof.

Check the groups against the original records. The first lantern example belongs under ambiguous label rules until the priority rule has been adopted. After adoption, the same pair of labels would be evaluated against a different guide version.

That distinction matters when revisiting old work. Do not mark a reviewer wrong for failing to apply a rule that did not exist when they annotated the message.

Revise the guide with a boundary example

Add the proposed priority rule and include two examples: one message combining praise with an explicit request, and one containing praise alone. Explain why the labels differ in terms of the task purpose.

Also add an uncertainty instruction. If the words do not support any defined label, reviewers should use the guide's unresolved path or request more context. Do not force them to reconstruct a conversation they cannot see.

Now ask reviewers to annotate a fresh, comparable message independently. For instance: “The blue cards look good. Could we add one blank card?” Keep the original answers hidden until both have finished. Any resulting agreement is a local observation about that example, not proof that the entire guide works.

Finish with a revision record

Record the original ambiguity, the rule change, the reason for choosing it, and which earlier items need another look. A modest guide with a clear revision history is easier to inspect than a large collection of unexplained corrections.

Avoid reporting a score unless you have defined what is counted and what the score means. For this tiny fictional set, plain descriptions are enough: one priority gap, one unsupported interpretation, and one unresolved reference.

If you later use real messages, obtain the necessary permission and remove unrelated personal details. This exercise needs only invented text. Its useful output is a better specified labeling task, not a claim about a population or a model's performance.

FAQ

Can the model cast a deciding vote?

A generated label is another interpretation. It does not settle the task definition. The guide owner must decide what distinction the labels are meant to express, then review examples under that rule.

Should every mixed message receive several labels?

Only if that fits the task. Multiple labels might preserve useful information, but would change this single-label exercise. Make that design decision explicit before comparing annotations.

What if reviewers agree for different reasons?

Keep the evidence phrases and inspect them. Matching labels can conceal incompatible interpretations. If the reasons imply different rules, the guide may still need clarification.

Product context and next step

The AZET introduction, dated October 3, 2026, described an assistant and an early-access waitlist. It does not document an annotation service or establish today's product status.

For the AZET waitlist, visit azet.io and read the information provided there.