Case study · credit decisioning
An AI credit assistant, from the register to the evidence pack
You run consumer lending at a bank, and your team wants an AI assistant to assess loan applications and recommend a decision to a credit officer. Before it goes live, risk will ask what it owes, what it was tested on and who approved it — and the day the model changes, they will ask again. This is that story, step by step, with every command the build runs and every answer it gets.
Example Bank is fictional, and so are its people: their addresses are on bank.example, a domain reserved for examples. What Deliverd does in the story is what it does — an integration test runs this page's story against a real database, from the same facts, on every change. Nothing here is legal advice.
Why this work owes so much
An assistant that assesses consumer loan applications evaluates the creditworthiness of natural persons, which Annex III, point 5(b) of the EU AI Act (Regulation (EU) 2024/1689) lists as high-risk; only systems used to detect financial fraud are excepted. Credit scoring profiles people, so the Act's exemption for systems that pose no significant risk is not open to it (Art. 6(3)). Whether a system is high-risk is the bank's legal classification to make, and the profile records the bank's call.
- DORA (Regulation (EU) 2022/2554) has applied since 17 January 2025. The assistant is an ICT-supported function to identify and classify (Art. 8), its changes go through change management (Art. 9(4)(e)), it is tested (Arts. 24–25), and its model provider is a third party in the register of information (Art. 28(3)).
- The Consumer Credit Directive ((EU) 2023/2225) applies from 20 November 2026. Where a creditworthiness assessment involves automated processing, the consumer may ask for a person to intervene: to explain the assessment, to hear their view, and to review the assessment and the decision (Art. 18(8)). A refused applicant is told of that right (Art. 18(9)).
- The AI Act's high-risk obligations apply to Annex III systems from 2 December 2027, the date set by Regulation (EU) 2026/1744 (the AI Omnibus); a system already in service by then is caught once its design changes significantly (Art. 111(2)). A deployer then assigns oversight to people with the competence and authority for it (Art. 26(2)), keeps the system's logs for at least six months (Art. 26(6)), tells applicants a high-risk system assists the decision (Art. 26(11)), and assesses the impact on fundamental rights before first use — Art. 27(1) names deployers of point 5(b) systems expressly. An applicant refused on the strength of its output can ask what part it played, where no other EU law already gives them that (Art. 86).
- Under the GDPR, a credit score the lender draws on strongly can itself be an automated decision (Art. 22; CJEU, C-634/21, SCHUFA, 7 December 2023). The EBA's guidelines on loan origination expect a bank to understand, document and back-test its credit models, and to be able to override them (EBA/GL/2020/06, paras. 53–55).
Example Bank adopts two framework packs, the EU AI Act and DORA, so the work is held to both now rather than in December 2027. A framework pack maps a framework's requirements onto Deliverd's common controls as a starting point for your own review. It is not legal advice and does not establish that any use complies with the framework. Whether a requirement applies, and whether the mapped controls meet it, is for your organisation to decide.
The assistant, and what the work owes
The credit team registers the assistant once, as an AI system with its provider, model and version (register_ai_system, or in the app). The work that uses it — assessing applications — is a governed task, and its profile says what kind of work it is:
| On the profile | Recorded as |
|---|---|
| Purpose | Assess consumer loan applications against the bank's credit policy and recommend approve, refer or decline, with the reasons, for a credit officer who decides. |
| Risk | High |
| Data | Confidential, with personal data |
| Reaches people | Yes: decisions reach applicants |
| In production | Yes, and a critical or important function — the bank's classification under DORA |
| What the AI does | Recommends; a credit officer decides |
| Oversight | Approval before action |
| Owners | Head of Consumer Lending, and the credit modelling team |
| Jurisdiction | EU |
With both packs adopted, that profile owes these 20 controls. Six are met by what the profile already records, twelve need evidence and two need a person's approval. Nobody wrote this list: Deliverd works it out from the profile and the packs, and the integration test holds the page to it.
| Control | Met by | Required by |
|---|---|---|
| Purpose documented | Recorded on the profile | EU AI Act Art. 11, DORA Art. 8 |
| AI system identified | Recorded on the profile | EU AI Act Art. 11, DORA Art. 8 |
| Business owner assigned | Recorded on the profile | EU AI Act Art. 26, DORA Art. 5 |
| Data classified | Recorded on the profile | EU AI Act Art. 10, DORA Art. 8 |
| Human oversight defined | Recorded on the profile | EU AI Act Art. 14, EU AI Act Art. 26 |
| Technical owner assigned | Recorded on the profile | DORA Art. 5 |
| Risk assessment | Evidence | EU AI Act Art. 9, EU AI Act Art. 27, DORA Art. 6 |
| Privacy review | Evidence | The profile: personal data |
| Security review | Evidence | EU AI Act Art. 15, DORA Art. 9, DORA Art. 25 |
| Model evaluation | Evidence | EU AI Act Art. 15, DORA Art. 25 |
| Transparency to affected people | Evidence | EU AI Act Art. 13, EU AI Act Art. 50 |
| Human review recorded | Evidence | EU AI Act Art. 14 |
| Compliance approval | A person's approval | EU AI Act Art. 27 |
| Deployment approval | A person's approval | The profile: production use |
| Resilience testing | Evidence | DORA Art. 24, DORA Art. 24(6), DORA Art. 25 |
| ICT change recorded | Evidence | DORA Art. 9 |
| Incident process defined | Evidence | DORA Art. 10, DORA Art. 17, DORA Art. 19 |
| Business continuity | Evidence | DORA Art. 11 |
| ICT third party assessed | Evidence | DORA Art. 28(3), DORA Art. 28(4), DORA Art. 30, DORA Art. 30(3) |
| Exit strategy | Evidence | DORA Art. 28(8), DORA Art. 30(3) |
A person sets the bar
Before any result counts, the head of model risk sets what results must meet, on the task's page. No key can set it — the build agent's own attempt is refused — because whatever submits the results must not also decide what passes.
| Control | The bar |
|---|---|
| Model evaluation | pass_rate >= 0.95 · decision_agreement >= 0.9 |
| Resilience testing | attack_success_rate <= 0.02 · no failed result at high severity or above |
decision_agreement is a promptfoo score: how closely each recommendation matched the decision the credit officers actually took on that historic application, from 0 to 1, over 200 of them. The red team is promptfoo's, against the assistant's own purpose.
Release 1: the build sends the evidence
The build runs in CI with an agent's key. It records the release first — the model version, and the digests of the prompt and guardrail files, which never leave the runner — then sends each run's own report, naming the same release. Results that name another release do not count.
# Record the release this build is made from: digests of the files, never the text
deliverd governance release $SYSTEM \
--prompt prompts/ --guardrails guardrails/ --model-version 2026-09
# The regression eval and the red team, as promptfoo writes them
promptfoo eval -c eval.yaml -o eval.json
promptfoo redteam run -c redteam.yaml && promptfoo export eval latest -o redteam.json
# Send each, naming the release it tested
deliverd governance evidence $TASK --report eval.json --format promptfoo --control model_evaluation \
--prompt prompts/ --guardrails guardrails/ --model-version 2026-09
deliverd governance evidence $TASK --report redteam.json --format promptfoo --control resilience_testing \
--type red_team --prompt prompts/ --guardrails guardrails/ --model-version 2026-09
deliverd governance check $TASK # exits 1 while the gate is closed| Run | What Deliverd recorded | Against the bar |
|---|---|---|
| Credit assessment regression, release 1 | 197 of 200 passed · pass_rate 0.985 · decision_agreement 0.94 | Meets the bar |
| Credit assistant red team, release 1 | 1 of 120 attacks got through · attack_success_rate 0.0083 | Meets the bar |
Recording a first fingerprint reopens nothing. The other ten pieces of evidence are documents people write, sent the same way with deliverd governance evidence $TASK --type … --title … --control …, or added on the task's page:
| Control | Evidence |
|---|---|
| Risk assessment | Risk assessment and fundamental rights impact assessment |
| Privacy review | Data protection impact assessment |
| Security review | Security review of the assistant and what it connects to |
| Transparency to affected people | Applicant notice: AI supports the assessment, and how to ask for a person to review it |
| Human review recorded | Credit officer review procedure, with a sample of reviewed recommendations |
| ICT change recorded | Change record for the release |
| Incident process defined | Incident procedure for the assistant, DORA classification included |
| Business continuity | Continuity plan: assessment by hand while the assistant is unavailable |
| ICT third party assessed | Due diligence on the model provider |
| Exit strategy | Exit plan from the model provider |
Every evidence control is met, and the gate is still closed: what is left is people's — the compliance approval and the deployment approval.
People approve, and the developer cannot
The developer asks for both approvals (request_control_approval). Named as the only approver of their own release, they are refused; named alongside compliance, they are left out; and at the decision, they are refused again. Compliance approves the release, the head of consumer lending approves going live, and deliverd governance check $TASK exits 0, so the deploy step runs. An agent never decides an approval at all.
The model updates
Weeks later the provider ships model version 2026-10, with the prompt and guardrails unchanged. The build records the new release, and Deliverd treats it as one: the task is reassessed, its gate closed until a person completes the reassessment, and the evaluation, resilience testing, change record and compliance approval of release 1 stop counting.
deliverd governance release $SYSTEM \
--prompt prompts/ --guardrails guardrails/ --model-version 2026-10| Run | What Deliverd recorded | Against the bar |
|---|---|---|
| Credit assessment regression, model update | 198 of 200 passed · pass_rate 0.99 · decision_agreement 0.95 | Meets the bar |
| Credit assistant red team, model update | 2 of 120 attacks got through · attack_success_rate 0.0167 · high: another applicant's income, asked for by name | Fails: 1 failed result at high severity or above. |
Two attacks in 120 got through, inside the ceiling on the rate. One of them gave out another applicant's income when asked for it by name — a high-severity failure — and the bar fails any run with one. The gate stays closed, and the task says why: 1 failed result at high severity or above.
The fix, and the job that finished late
The team scopes applicant data to the application in hand — guardrails v5 — and the build records that release too, which opens a second reassessment. Both runs of the fix meet the bar.
| Run | What Deliverd recorded | Against the bar |
|---|---|---|
| Credit assessment regression, guardrails v5 | 198 of 200 passed · pass_rate 0.99 · decision_agreement 0.95 | Meets the bar |
| Credit assistant red team, guardrails v5 | 1 of 120 attacks got through · attack_success_rate 0.0083 | Meets the bar |
Then the model update's red team, re-run on a slow runner, finishes after the fix. It names the guardrails it tested, so it does not count, however it scored, and the task says how its release differs: guardrails sha256:c08ef511eaad… where the system now runs sha256:b1938a3c8f3b…. The change record is sent again, compliance approves the release again, and the head of model risk completes both reassessments with what was checked. The gate opens.
What the auditor gets
One evidence pack for the work (get_evidence_pack, or Evidence pack on the task's page), with a sha256 digest so a copy can be checked against the one that left:
- The AI system and its release fingerprint, and the profile the work was assessed on.
- Each control, how it was met, and the requirements behind it.
- The bar each scored control was judged against, and who set it.
- Every run with its results, the release it tested, and — for the late red team — why it did not count.
- Each approval with who was asked and who decided, and each reassessment with what changed and the note that closed it.
- The audit timeline of the work.
The pack assists audit; it does not certify compliance with any law, and says so in its own text. Compliance Mode is in Business and Enterprise; framework packs and reassessment on a model change, which this story uses, are in Enterprise.