19/09 edition: an acceptance corpus for classifying requests

A label is not a consequence-free decision

An SME wants to separate billing, technical support and contract-change requests. The question is not whether a model recognises clean examples in a demonstration. It is whether it can route the service's actual requests without concealing ambiguity inside a convenient category.

We propose starting with advisory routing. The responsible person retains the final decision and sees the text supporting the label. Only after a documented trial should limited automation be discussed. This case excludes hiring, credit and diagnosis; the acceptance design belongs only to the selected support queue.

Evaluation references, not a mandatory recipe

The NIST Playbook's MEASURE 2.1 recommends documenting test sets, metrics and TEVV tools. OWASP LLM09 recommends human verification for critical or sensitive content. The AI RMF is voluntary. Corpus selection and the following criteria are Velqory proposals, not a universal NIST requirement.

Curate disagreement before evaluating the model

Our proposal starts with synthetic examples or data whose use has been validated by the responsible owner. Each case receives an expected category, supporting passage and indication of whether clarification is needed. Two service specialists label it without seeing the model's answer; a third resolves disagreement and records why.

Do not force artificial consensus. A message combining a technical error and an invoice dispute may need two routes or human contact. If the service rule itself is incomplete, mark the case as a rule awaiting decision. The classifier should not be tasked with resolving organisational uncertainty through a confident label.

The case an average would conceal

Hypothetical scenario: many simple billing requests are classified correctly, but a message involving a contract change reaches technical support. A favourable average does not answer the acceptance question for that route. We propose showing errors by category and separately identifying cases that should stop for clarification.

Reserve examples for acceptance rather than repeatedly tuning the prompt against them. Include short messages, local terminology, missing attachments and multiple requests. Human review must allow selection of 'insufficient information' without automatically counting it as reviewer failure. Record corrections and their rationale as well.

A narrow authorisation supported by cases

The matrix proposes internal conditions, not measured results. The support owner selects in advance the errors that block automatic routing. If a critical case fails, the team should not dilute it with more easy examples; correct the workflow or retain human confirmation for that category.

After acceptance, retain the ability to correct

Retain the corpus version, adjudication rules, evaluated configuration and case-by-case results. Our proposal permits automation only for categories with explicitly recorded acceptance; new labels and out-of-scope messages return to the human queue. A model or rule change requires deciding which cases to rerun.

In a small Portuguese team, one person may fill two roles, but the conflict should be visible and offset through additional review of blocking cases. We report no accuracy rates or statistically representative sample. This corpus supports a bounded operational decision; it does not demonstrate universal quality or remove the need to follow emerging errors.