LangSmith experiments as release evidence

If your evals run through LangSmith's evaluate(), the scores a release should be judged on already exist. Each evaluator has scored every run, and LangSmith has aggregated them. What LangSmith does not record is who decided those numbers were good enough to ship.

LangSmith is one of four eval tools Deliverd pulls from, built on the same rules as Langfuse. A person connects it, the key is sealed, and the build step asking for evidence never holds it. Why we built it that way is its own post.

Connecting it

An owner or an administrator connects LangSmith on Admin → Integrations, under Evaluation tools. Choose a LangSmith Cloud region (US, EU, APAC or US on AWS) or Self-hosted with its https address. Self-hosted needs version 0.16 or later and has to be reachable over https from the internet.

Paste a service key for the workspace your experiments are in. If the key spans several workspaces, give the workspace ID too. Every pull then reads that workspace and no other, even if the key could open more. The key is tried before anything is kept, the card names the workspace it opened, and a stored key is only ever sent to the address it was connected for.

Pulling an experiment

LangSmith gives each evaluate() run a name made from the prefix you passed and a suffix of its own. You can name either one. credit-memo-v12 finds credit-memo-v12-1a2b3c4d, and the newest match from the last 90 days is read. A longer name is never matched by mistake: credit-memo-v1 does not find credit-memo-v12.

From a build step:

npx deliverd governance evidence "$TASK" --langsmith credit-memo-v12 --control model_evaluation

Or from the Python SDK, in the same script that ran the evaluation:

from deliverd import Deliverd

deliverd = Deliverd()  # DELIVERD_API_KEY from the environment
task = deliverd.tasks.get(task_id)

task.submit_evidence(
    type="model_evaluation",
    title="LangSmith: credit-memo-v12",
    controls=["model_evaluation"],
    pull={"source": "langsmith", "experiment": "credit-memo-v12"},
)

gate = task.check_governance_gate()
if not gate.allowed:
    raise SystemExit(gate.reason)

Pull from LangSmith on a governed task does the same for a person, in the app.

What is read

• the workspace the key reads, for the links;
• the newest matching experiment, with LangSmith's own statistics for each evaluator over every run, including summary evaluators;
• the experiment's root runs, up to 2,000, asking LangSmith for each run's id, trace and status only.

Never a run's inputs, outputs, metadata or error text. No feedback comment or correction either, because a comment is where a reviewer pastes the output they were unhappy with.

How the scores map

Each evaluator becomes one result under its own key: the mean for a numeric score, or a count for each label. A run that failed to complete is listed as a failure, with a link to it in LangSmith, and the evidence links to the experiment's compare view.

One deliberate choice: a low score is not called a failure. LangSmith's statistics do not say which way a metric points. toxicity: 0 is a pass, and correctness: 0 is not. Deciding which is which is a person's job, in the thresholds: correctness >= 0.9, toxicity <= 0.05. The connector reports. It does not judge.

When something is wrong

No connection or no matching experiment comes back as 404. A key LangSmith no longer accepts comes back as 409 connection_refused. LangSmith being unreachable comes back as 502 upstream_unavailable. A self-hosted install older than 0.16 is refused when someone tries to connect it, with the version it needs.

Deliverd is human-in-the-loop infrastructure for AI agents. Connecting LangSmith is covered in Integrations, and evidence and thresholds in the Compliance Mode guide.