Langfuse experiments as release evidence

Langfuse was the first eval tool we connected, for a simple reason. It is open source, it is widely self-hosted in the EU, and its experiments already carry the scores a release is judged on. Many of the teams asking us about DORA keep their traces in a Langfuse they run themselves.

This post covers what the connector reads, what it leaves alone, and how a build step uses it. Why we built the integrations this way is its own post.

Connecting it

An owner or an administrator connects Langfuse on Admin → Integrations, under Evaluation tools. Choose Langfuse Cloud by region (EU, US, Japan or HIPAA) or Self-hosted with its https address, then paste the project's public and secret keys.

The keys are tried against Langfuse before anything is kept, and the card names the project they opened, so a typo shows up when you press Connect rather than at the first build. The secret key is sealed. Only a person, in the app, can connect or disconnect. No API key can.

A self-hosted Langfuse has to be reachable over https from the internet. Deliverd will not call a private address.

Pulling an experiment

From a build step, name the experiment and the controls it speaks to:

npx deliverd governance evidence "$TASK" --langfuse credit-memo-v12 --control model_evaluation

Deliverd reads the newest experiment with that name from the last 90 days (--since looks further back), or the exact one if you give its id. From the TypeScript SDK:

import { Deliverd } from "@deliverd/sdk";

const deliverd = new Deliverd(); // DELIVERD_API_KEY from the environment
const task = await deliverd.tasks.get(process.env.TASK_ID!);

await task.submitEvidence({
  type: "model_evaluation",
  title: "Langfuse: credit-memo-v12",
  controls: ["model_evaluation"],
  pull: { source: "langfuse", experiment: "credit-memo-v12" },
});

const gate = await task.checkGovernanceGate();
if (!gate.allowed) throw new Error(gate.reason);

On a governed task in the app, Pull from Langfuse does the same thing for a person.

What is read

It uses Langfuse's v4 public API:

• the project the keys belong to, so links open in the right place;
• the newest experiment with the name, and the scores on the experiment itself;
• each item's trace, and its item and trace scores, 100 at a time and up to 2,000 items.

If the experiment is larger than that, the evidence says the read was partial instead of passing off part of the run as all of it.

What is never read: an item's input, output or metadata. Those are where personal data and the model's answers live, and the evidence needs neither.

How the scores map

Each Langfuse score becomes one result under its own name, so a person writes the threshold on what Langfuse calls it:

• a numeric score becomes its mean;
• a boolean score becomes the share of items it was true for;
• a categorical score becomes a count for each category (tone: rude).

A boolean score that was false on an item is listed as a failure, and so is an item whose run failed. Each one links to its trace in Langfuse, so a reviewer can go from "failed" to the exact trace in one click. The evidence links to the experiment.

Text scores are left out, since there is nothing to set a bar on.

When something is wrong

Every refusal says why:

• no Langfuse connected, or no experiment with that name: 404;
• keys Langfuse no longer accepts: 409 connection_refused;
• Langfuse unreachable: 502 upstream_unavailable.

A build step sees the reason and can stop on it, rather than recording an empty run.

One note on versions

The connector uses Langfuse's experiments API, not the older dataset-run and scores endpoints, which Langfuse Cloud is retiring. Nothing changes for you when they go.

Deliverd is human-in-the-loop infrastructure for AI agents. Connecting Langfuse is covered in Integrations, and evidence and thresholds in the Compliance Mode guide.