Not high-risk, still accountable: an AI that routes motor claims, end to end

A motor claims team wants an AI assistant to take first notice of loss in the app's chat and route each claim: pay it on the fast track, send it to the handlers' queue, or refer it to the fraud team. The assistant is not high-risk under the EU AI Act. That does not leave it unregulated, and the supervisor will still ask the same questions: what does this work owe, which claims may the assistant route on its own, who decides the rest, which provider does it depend on, and what happened when that provider failed?

We followed that work from start to finish and wrote it up as a case study, with every call it makes and every answer it gets. This post is the short version: why work outside the high-risk list still owes so much, and the three moments in the story where a person, not the AI, had the last word.

Example Insurance is fictional, and so are its people, its model provider and its claims. What Deliverd does in the story is what it does: an integration test runs the case study's story against a real database, from the same facts, on every change. Nothing here is legal advice.

Not high-risk is not unregulated

A motor claims assistant is not in Annex III of the EU AI Act (Regulation (EU) 2024/1689). Its insurance entry, point 5(c), covers risk assessment and pricing for life and health insurance, and the Commission's draft guidelines on classifying high-risk systems (19 May 2026) say that motor insurance is not covered by it, nor is a system that checks whether a claim is valid or how much to pay. The guidelines are still a draft, and the classification is the insurer's to make: it records its call on the work's profile.

Being outside that list still leaves a lot:

• The AI Act reaches it twice. Claimants chat with the assistant, so they must be told they are dealing with an AI system unless that is obvious (Art. 50(1)), from 2 August 2026. And a referral to fraud investigators has to rest on the claim's facts, not on who the claimant is: assessing a person's risk of offending from profiling alone is prohibited (Art. 5(1)(d)).
• DORA (Regulation (EU) 2022/2554) has applied to insurers since 17 January 2025. The model provider's contract goes in the register of information (Art. 28(3)), and when the provider fails, the incident is classified and, if major, reported (Arts. 18–19). Claims handling is a critical or important function at this insurer, which adds testing at least yearly (Art. 24(6)) and an exit strategy from the provider (Art. 28(8)).
• Under the GDPR, a decision based solely on automated processing may not rest on health data unless the claimant has given explicit consent or there is a substantial public interest, with suitable safeguards (Art. 22(4)), and injury claims carry health data.
• EIOPA's Opinion on AI governance and risk management (EIOPA-BoS-25-360, 6 August 2025) says what supervisors expect of AI in insurance that is neither prohibited nor high-risk: fairness, data governance, record keeping, explainability, human oversight, and accuracy and robustness. It sets no new requirements; it reads the existing ones for AI.

The insurer writes its own framework

No built-in pack covers the EIOPA Opinion, so Example Insurance writes its own. An agent drafts it from the Opinion, mapping each expectation onto the controls Deliverd already knows, and adds the one the mapping needs that no common control covers: a fairness review of who gets fast-tracked and who gets referred. A draft holds nothing to anything, and the agent's attempts to publish it or adopt it are refused. Compliance reads it, publishes it, and adopts it alongside the EU AI Act and DORA packs.

A framework pack maps a framework's requirements onto Deliverd's common controls as a starting point for your own review. It is not legal advice and does not establish that any use complies with the framework. Whether a requirement applies, and whether the mapped controls meet it, is for your organisation to decide.

24 controls, and one article of the AI Act

The profile says what kind of work this is: medium risk, personal data and health data in injury claims, claimants chatting with it, in production on a critical or important function, and an AI that acts under a policy and asks a person where the policy says to. From that profile and the three packs, Deliverd works out what the work owes: 24 controls, each with the requirement behind it. Of those, 6 are met by what the profile already records, 16 need evidence, and 2 need a person's approval.

Of the EU AI Act, only Art. 50 reaches this work; most of the list comes from DORA and from the insurer's own reading of the Opinion. Nobody wrote the list, and nobody has to keep it up to date.

The first turn: the injuries the eval missed

The actuarial function sets one bar for every claims model in the organisation, once, and any governed task without a bar of its own is judged against it:

• route_correct >= 0.9: whether the assistant chose the route a senior handler chose, over 400 historic claims.
• injury_detected >= 0.99: whether the assistant flagged an injury, scored only on the 120 claims that report one.

No API key can set or move that bar. Whatever submits the results must not also decide what passes.

Both are Langfuse scores. Claims operations connects the insurer's Langfuse project once, as a person, and the build names the experiment it ran. It holds no Langfuse key:

deliverd governance evidence $TASK --langfuse motor-triage-v3 --control model_evaluation

374 of 400 claims were routed as the handlers routed them, a route_correct of 0.935: comfortably over the bar. But 117 of 120 injuries were flagged, an injury_detected of 0.975, and the gate stays closed. The task says why:

injury_detected was 0.975; the threshold is >= 0.99.

Three missed injuries sounds small. It matters because of the policy below: a small claim is paid on the fast track only when it reports no injury, and it is the assistant that says whether a claim does. The bar on injuries is what makes that rule safe to write. A bar on routing accuracy alone would have let this release through.

The team rewrites the chat's questions about injuries, and motor-triage-v4 meets the bar: 377 of 400 routed as the handlers did, a route_correct of 0.9425, and 120 of 120 injuries flagged. The other 15 pieces of evidence are documents people write and send the same way: the risk assessment, the data protection impact assessment, the fairness review, the exit plan from the model provider and the rest. Compliance approves the release, the head of motor claims approves going live, and the gate opens.

The second turn: a policy says what the assistant may do

The assistant's authority is a policy, not a prompt. Claims operations registers what it does as an action, claims.route, with the claim, the route, the amount and whether an injury is reported, and writes the policy against it from their own key:

deliverd actions push --file claims.actions.json
deliverd policy push --file claims-routing.policy.json --activate

An agent's key is refused both. A policy decides when an agent needs a person, so an agent that could write one would decide that for itself. The first rule that matches decides:

1. The assistant never declines a claim.
2. A referral to the special investigations unit is the unit's call.
3. A small claim with no injury is paid on the fast track.
4. Any other fast-track claim needs a senior handler.
5. Standard handling goes to the handlers' queue.

Before each route, the assistant asks, and the policy in force answers:

The senior handler approves the injury claim: "Soft-tissue injury, GP's note attached. Fast track, with the injury handled as its own claim." The SIU lead turns the referral down: "The second key is with the finance company, as the claimant said. Standard handling." The assistant routes that claim to standard handling instead, which the policy allows. Each permission is used once, and no key decides an approval, the assistant's included.

Keeping referrals with the fraud team also keeps a person between the assistant and any suspicion of fraud: the assistant proposes a referral from the claim's facts, and the unit decides it, with its reason on the record.

The third turn: the provider goes down

At 07:10 the assistant's calls to its model provider start failing. The agent reports an incident against the AI system at high severity, which closes the gate on the work. First notice moves to the phone line, and handlers triage claims by hand under the continuity plan.

An agent cannot classify an incident: one that could call an outage minor would stop its own reporting clock. At 09:30 the head of ICT risk classifies it from the facts. Critical services were affected, and two criteria met their thresholds, how long the service was down and how many Member States it reached, so it is major (Commission Delegated Regulation (EU) 2024/1772, Art. 8). The initial notification is due by 13:30: 4 hours from classification, which comes before 24 hours from detection (Commission Delegated Regulation (EU) 2025/301, Art. 5).

The agent drafts the report, and it goes to compliance to sign off. Only once it is approved can the head of ICT risk record it as filed, with the authority's reference, IR-2026-0117. Deliverd works out each later deadline, the intermediate report within 72 hours and the final report a month after the last intermediate one, and files nothing itself.

The register of information had already said how much rested on this provider: two of the insurer's three AI systems, both in production, one of them supporting a critical function. The provider restores service after 30 hours, the head of ICT risk resolves the incident, and the gate opens again.

What the supervisor gets

One evidence pack for the work, with a sha256 digest so a copy can be checked against the one that left. It holds the AI system and its contract with the provider, each control with the requirements behind it as the insurer mapped them, the bar and who set it, each Langfuse run judged against it, each approval with who was asked and who decided, and the incident: its classification, its deadlines, each report, who signed it off and when it was filed. Each route the assistant asked for, with the rule that decided and who approved, is on its gate and in the organisation's audit trail. The pack assists audit; it does not certify compliance with any law, and says so in its own text.

Read the whole thing

The case study has every step: the profile, the insurer's framework paragraph by paragraph, all 24 controls with the requirements behind each, both Langfuse runs, the policy in full, every claim, and the incident's criteria. Compliance Mode is in Business and Enterprise, and policies for an agent's actions are in Business and Enterprise; framework packs, an organisation's own frameworks and the ICT register, which this story uses, are in Enterprise. How policies decide is in the policy docs, and how evidence, bars and the gate work is in the Compliance Mode guide. For the same story at a bank, where the model changes twice, read the credit decisioning case study.

Deliverd is human-in-the-loop infrastructure for AI agents.