W&B Weave evaluations as release evidence

Weave's Evaluation already does the hard part. It runs your model over a dataset, scores every row with your scorers, and keeps the whole run, call by call, in Weave. What evaluate() returns at the end is a summary: one number for each thing that was measured.

That summary is the evidence a release decision rests on. Weave is the fourth eval tool Deliverd pulls from, on the same rules as Langfuse, LangSmith and Braintrust: a person connects it, the key is sealed, and the build step asking for evidence never holds it. Why we built it that way is its own post.

Connecting it

An owner or an administrator connects Weave on Admin → Integrations, under Evaluation tools. Paste a W&B API key (a team service account's is best) and name the project your evaluations are logged to, as team/project.

Leave the deployment blank for W&B's cloud. For a Dedicated or self-managed deployment, give its address, which has to be reachable over https from the internet. The key is tried before anything is kept, and the card names the team and project it opened. Only a person, in the app, can connect or disconnect it.

Pulling an evaluation

Name the evaluation and the controls it speaks to:

npx deliverd governance evidence "$TASK" --weave credit-memo-v12 --control model_evaluation

The name can be the one your Evaluation object was given, or the name of a run. The newest finished run from the last 90 days is read. A run that is still going, or one that failed outright, is skipped.

From the Python SDK, in the job that gates the release:

from deliverd import Deliverd

deliverd = Deliverd()  # DELIVERD_API_KEY from the environment
task = deliverd.tasks.get(task_id)

task.submit_evidence(
    type="model_evaluation",
    title="Weave: credit-memo-v12",
    controls=["model_evaluation"],
    pull={"source": "weave", "experiment": "credit-memo-v12"},
)

gate = task.check_governance_gate()
if not gate.allowed:
    raise SystemExit(gate.reason)

Pull from Weave on a governed task does the same thing for a person.

What is read

W&B answers in two places, and the connector reads both:

• W&B's API, to check that the key is accepted and to find the team and project;
• Weave's trace server, for the newest finished run of the evaluation, selecting its id, name, time and what evaluate() returned;
• the number of cases the run had, and how many failed, as counts rather than rows;
• each failed case's id and time, so it can be linked.

What is never read: a case's inputs or outputs, or the text of an exception, which can repeat them. Weave also keeps its own summary of the model's outputs inside the evaluation's result. That is left out too.

How the scores map

Every number evaluate() returned becomes a result named by its path, so a threshold is written on exactly what Weave reported:

• exact_match.correct.true_fraction for a boolean scorer;
• groundedness.score.mean for a numeric one;
• model_latency.mean, which Weave adds to every evaluation.

A person might set exact_match.correct.true_fraction >= 0.9 and model_latency.mean <= 2.5 on the control. A case whose model or scorer raised is listed as a failure, with a link to it in Weave. A low score is not called a failure. That judgement is the person's, in the thresholds.

An evaluation with no scorers is refused with a sentence saying so. It is not recorded as an empty run.

Or send the summary yourself

If you would rather not connect W&B, the pipeline that ran the evaluation can send the summary instead. Deliverd then holds no W&B key at all:

summary, call = asyncio.run(evaluation.evaluate.call(evaluation, model))

task.submit_evidence(
    type="model_evaluation",
    title="Weave: credit-memo-v12",
    controls=["model_evaluation"],
    report={
        "format": "weave",
        "content": json.dumps({"summary": summary, "url": call.ui_url, "id": call.id}),
    },
)

A pulled run and a sent summary are mapped by the same function, so a threshold written on one is met in the same way by the other. The difference is failures: a summary has aggregates and no rows, so only a pull can list the cases that failed.

Deliverd is human-in-the-loop infrastructure for AI agents. Connecting Weave is covered in Integrations, and evidence and thresholds in the Compliance Mode guide.