Inside the limit, and still not shipped: an AI credit assistant, end to end

A lending team wants an AI assistant to assess consumer loan applications and recommend approve, refer or decline to a credit officer, who decides. Before it goes live, risk will ask three questions: what does this work owe, what was it tested on, and who approved it? The day the model changes, they will ask all three again.

We followed that work from start to finish and wrote it up as a case study, with every command the build runs and every answer it gets. This post is the short version: why credit work owes so much, and the two moments in the story where a release was held back that a simpler setup would have let through.

Example Bank is fictional, and so are its people. What Deliverd does in the story is what it does: an integration test runs the case study's story against a real database, from the same facts, on every change. Nothing here is legal advice.

Why credit work owes so much

An assistant that assesses consumer loan applications evaluates the creditworthiness of natural persons, which Annex III, point 5(b) of the EU AI Act (Regulation (EU) 2024/1689) lists as high-risk. Credit scoring profiles people, so the Act's exemption for systems that pose no significant risk is not open to it (Art. 6(3)). The AI Act is not the only law that reaches it:

• DORA (Regulation (EU) 2022/2554) has applied since 17 January 2025. The assistant is an ICT-supported function: it is identified and classified, its changes go through change management, it is tested, and its model provider goes in the register of information.
• The Consumer Credit Directive ((EU) 2023/2225) applies from 20 November 2026. Where a creditworthiness assessment involves automated processing, the consumer may ask for a person to intervene: to explain the assessment, to hear their view, and to review the assessment and the decision (Art. 18(8)).
• The AI Act's high-risk obligations apply to Annex III systems from 2 December 2027, the date set by Regulation (EU) 2026/1744, the AI Omnibus. A system already in service by then is caught once its design changes significantly (Art. 111(2)).
• Under the GDPR, a credit score the lender draws on strongly can itself be an automated decision (Art. 22; CJEU, C-634/21, SCHUFA, 7 December 2023).

Whether a system is high-risk is the bank's legal classification to make, and it records its call on the work's profile. It also adopts two framework packs, the EU AI Act and DORA, so the work is held to both now rather than in December 2027. A framework pack maps a framework's requirements onto Deliverd's common controls as a starting point for your own review. It is not legal advice and does not establish that any use complies with the framework. Whether a requirement applies, and whether the mapped controls meet it, is for your organisation to decide.

20 controls nobody typed

The profile says what kind of work this is: high risk, personal data, in production, a critical or important function under DORA, and an AI that recommends while a person decides. From that profile and the two packs, Deliverd works out what the work owes: 20 controls, each with the requirement behind it. Of those, 6 are met by what the profile already records, 12 need evidence, and 2 need a person's approval.

Nobody wrote that list, and nobody has to keep it up to date. Change the profile or adopt another pack, and the list changes with it.

A person sets the bar

Before any result counts, the head of model risk sets what results must meet, on the task's page:

• The regression eval, over 200 historic applications: pass_rate >= 0.95 and decision_agreement >= 0.9. The second is a promptfoo score: how closely each recommendation matched the decision the credit officers actually took.
• The red team, promptfoo's, against the assistant's own purpose: attack_success_rate <= 0.02, and no failed result at high severity or above.

No API key can set or move that bar, the build agent's included. Whatever submits the results must not also decide what passes.

Release 1

The build runs in CI with an agent's key. It records the release first: the model version, and the digests of the prompt and guardrail files, which never leave the runner. Then it sends promptfoo's own reports, each naming the release it tested:

deliverd governance release $SYSTEM --prompt prompts/ --guardrails guardrails/ --model-version 2026-09
deliverd governance evidence $TASK --report redteam.json --format promptfoo --control resilience_testing \
  --type red_team --prompt prompts/ --guardrails guardrails/ --model-version 2026-09
deliverd governance check $TASK   # exits 1 while the gate is closed

197 of 200 applications passed the eval, with a decision_agreement of 0.94, and 1 of 120 attacks got through the red team, at low severity. Both meet the bar. The documents people write, one for each of the other 10 evidence controls, go in the same way: the risk assessment, the data protection impact assessment, the exit plan from the model provider and the rest.

That still does not open the gate. What is left is people's: compliance approves the release, and the head of consumer lending approves going live. The developer who built the release asks for both approvals and cannot give either. Named as the only approver of their own release, they are refused.

The first turn: inside the limit, and still stopped

Weeks later the provider ships model version 2026-10, with the prompt and guardrails unchanged. The build records the new release, and Deliverd treats it as one: the work is reassessed, its gate is closed until a person completes the reassessment, and the evaluation, resilience testing, change record and compliance approval of release 1 stop counting.

The new eval meets the bar. The red team does not. This time 2 of 120 attacks got through, an attack_success_rate of 0.0167, inside the ceiling on the rate. But one of them gave out another applicant's income when asked for it by name, a high-severity failure, and the bar fails any run with one. The task says why the gate is closed:

1 failed result at high severity or above.

A bar on the rate alone would have let this release ship. One applicant's income handed to another is not a rate problem. The rule that caught it was the head of model risk's, set before the run, and the pipeline that sent the run could not have changed it.

The second turn: the job that finished late

The team scopes applicant data to the application in hand, as guardrails v5, and the build records that release too, which opens a second reassessment. Both runs of the fix meet the bar.

Then the model update's red team, re-run on a slow runner, finishes after the fix. It is the newest run on its control, and it does not count, however it scored: it names the guardrails it tested, and the system now runs different ones. The task says exactly how they differ:

guardrails sha256:c08ef511eaad… where the system now runs sha256:b1938a3c8f3b…

"The newest run decides" is only safe when the newest run tested the release that ships. A result names its release, and one that names another release stays on the record and out of the decision.

The change record is sent again, compliance approves the release again, and the head of model risk completes both reassessments with a note of what was checked. The gate opens.

What the auditor gets

One evidence pack for the work, with a sha256 digest so a copy can be checked against the one that left. It holds the AI system and its release fingerprint, each control and the requirements behind it, the bar each scored control was judged against and who set it, every run with the release it tested, each approval with who was asked and who decided, and the audit timeline. The pack assists audit; it does not certify compliance with any law, and says so in its own text.

Read the whole thing

The case study has every step: the profile, the full list of controls with the requirements behind each, every command the build runs, and every run's figures. Compliance Mode is in Business and Enterprise; framework packs and reassessment on a model change, which this story uses, are in Enterprise. How evidence, thresholds and the gate work is in the Compliance Mode guide.

Deliverd is human-in-the-loop infrastructure for AI agents.