29/09 edition: validate the answer before turning it into action

The boundary is the answer's destination

An SME wants to turn support conversations into notes for its internal system. The answer arrives in a structured format with a problem description. The relevant question is not merely whether the text is correct: it is what the application will do with each field.

We propose drawing the complete workflow from the received answer to display and any record update. A description should remain a description. An identifier should be checked as an identifier. The model may suggest an operation, but should not decide that the destination is authorised to perform it.

What OWASP distinguishes

OWASP LLM05 identifies risks when outputs pass without adequate validation to destinations such as shells, browsers or SQL queries. LLM06 separately addresses excessive authority and recommends downstream authorisation. NIST's AI RMF is voluntary. These documents do not imply that every generated answer is exploitable.

Separate structure, content and authority

Our design starts with a closed structure: expected fields, types and lengths defined by the application. Extra or missing values cause a readable rejection or review rather than silent adaptation inventing the user's intent. Accepting structure does not authorise content.

Each destination then receives its own rule. Displayed notes should be treated as text or pass through an explicit permitted-formatting policy. Record updates use application-defined operations and verified parameters. File paths, commands and unrestricted queries do not belong to this use case and remain outside its contract.

The case valid outside but unsafe inside

Hypothetical scenario: the answer matches the structure but contains active markup and a link to an unapproved destination. The trial should introduce harmless synthetic markers and observe the final result. We do not need a real exploit to check whether content stopped being treated as text.

Include an overlong note, another customer's identifier and an unrecognised operation. The test team preserves the synthetic raw answer, validation decision and observed destination effect. An on-screen error does not prove that no update occurred: also inspect the record the application tried to change.

Accept by destination, not by impression

This original matrix defines different evidence for display and writing. The application owner approves the field contract; security checks boundary cases. A refused field should not automatically be resubmitted through a less restrictive path. Include that behaviour in the trial as well.

After rejection, preserve useful work

We propose a review queue where a person safely sees the refused note, reason and fields requiring correction. Reusing text does not inherit a refused operation. Correction produces a new request subject to the same checks and the scope of the submitting identity.

In a Portuguese SME, authorisation may remain manual for initial writes even after display has passed its trial. Record that distinction instead of declaring the entire system validated. Review this plan when the renderer, field contract or write destination changes. We report no executed tests, guaranteed effectiveness or internal experience.