AI in September: a multi-provider gate separating capability, security, and continuity

The signal and the facts that change the decision

The closing days of August brought three different signals: a DeepMind double-blind evaluation pilot, an evaluation incident described by OpenAI, and containment measures published by Anthropic. None independently decides which AI system should enter an organisation's production environment.

DeepMind's pilot tests evaluation integrity; the OpenAI and Anthropic accounts address containment and behaviour in test environments. All three are provider or partner materials and should be attributed, with internal results retained as an independent layer.

NIST's AI RMF and TEVV require context, measurement, and continuous management; approval is a system decision, not a brand decision. Mixing provider claims with buyer-owned results creates a meaningless average. Comparing models under different configurations or ignoring tools, permissions, and data can select the best benchmark and the worst operating system.

Decision, limits, and stop condition

For September, use a gate with four separate scores: task quality, behaviour under abuse and failure, surrounding-system controls, and contractual continuity. A high score does not offset a disqualifying condition in another dimension; the decision should bind to the tested version and configuration.

Mixing provider claims with buyer-owned results creates a meaningless average. Comparing models under different configurations or ignoring tools, permissions, and data can select the best benchmark and the worst operating system.

Execution and proof of completion

Execution should start by fixing one task, dataset, version, prompt, tools, limits, cost, and metrics consistently across candidates. Then, running normal, tail, abuse, outage, and authority-exceeding cases in an isolated environment. Closure requires testing export, substitution, and rollback and publishing the decision with conditions, validity, and reevaluation triggers.

The gate closes when a second team reproduces scores and failures, every disqualifier is resolved or rejected, the exit plan passes, and approval identifies the artefact, configuration, scope, owner, and expiry date.

• fixing one task, dataset, version, prompt, tools, limits, cost, and metrics consistently across candidates.
• running normal, tail, abuse, outage, and authority-exceeding cases in an isolated environment.
• testing export, substitution, and rollback and publishing the decision with conditions, validity, and reevaluation triggers.
• The gate closes when a second team reproduces scores and failures, every disqualifier is resolved or rejected, the exit plan passes, and approval identifies the artefact, configuration, scope, owner, and expiry date.