Braintrust experiments as release evidence
14 October 2026 · Deliverd Engineering Team · 4 min read
Braintrust is one of four eval tools Deliverd pulls from. It follows the same rules as Langfuse and LangSmith: a person connects it, the key is sealed, and the build step asking for evidence never holds it. Why we built it that way is its own post.
Two things about Braintrust shaped this connector more than the others: where its data lives, and how its API returns rows.
No region to choose
Braintrust separates its app from its data plane, the API that holds your experiments. The data plane can be hosted in the US or the EU, or run in your own cloud. You do not pick one when you connect. The key is sent to Braintrust's login, the same call Braintrust's own SDKs make, and the login says where your organisation's data plane is.
Because your key will be sent to that address, Deliverd checks it before using it, the same way it checks an address a person types: plain https, no credentials, no path, and not a private address. If the data plane is in your own cloud, it has to be reachable over https from the internet.
Connecting it
An owner or an administrator connects Braintrust on Admin → Integrations, under Evaluation tools. Paste a service token from your organisation's settings in Braintrust, and name the project your experiments are in. If the key opens several organisations, name the organisation too. The key is tried before anything is kept, and the card names the organisation and project it opened.
A pull reads that project's experiments and no others.
Pulling an experiment
npx deliverd governance evidence "$TASK" --braintrust credit-memo-v12 --control model_evaluation
Name the experiment, or give its id. The newest experiment with that name from the last 90 days is read, and that includes a rerun: Braintrust names a second run of credit-memo-v12 something like credit-memo-v12-db3e9cae. A longer name never matches by accident, so credit-memo-v1 does not find credit-memo-v12.
The SDKs take the same pull, { source: "braintrust", experiment: "credit-memo-v12" }, and Pull from Braintrust on a governed task does it for a person.
What is read, and what is not
The scores come from Braintrust's own summary of the experiment: each score's mean over the cases, and its metrics.
For failures, Deliverd needs to know which cases errored. Braintrust's row endpoint, /fetch, always returns each row's input and output, and there is no way to ask it not to. So the connector does not use it. It sends a BTQL query that selects only each span's id, whether it is a root, and whether it has an error. Braintrust applies that projection on its own side, so a case's input, output and error text never leave Braintrust.
Push and pull, one threshold
Before this connector existed, a pipeline could already send the report braintrust eval --jsonl prints. The pull connector maps Braintrust's summary with the same function the push reader uses. So a threshold such as accuracy >= 0.9, written while you were sending reports, is met in exactly the same way when you switch to pulling.
Cases are the experiment's root spans. Each one that errored is listed as a failure, with its permalink in Braintrust. A low score is not called a failure: the bar is a person's, in the thresholds. A scorer that errored is not called a failure either. Only a case that errored is.
If the experiment has more rows than one pull reads, the evidence says that only the first cases were checked for failures. The scores are still Braintrust's own, over every case.
Deliverd is human-in-the-loop infrastructure for AI agents. Connecting Braintrust is covered in Integrations, and evidence and thresholds in the Compliance Mode guide.
