Integrations and webhooks

Integrations lists everything Deliverd connects to, for everyone, with whether each is connected here. Organisation → Integrations & webhooks is the administrator's half of the same list, drawn as the same tiles: the integrations set up once for the organisation (the Slack and Teams apps, Hoplite, the evaluation tools, webhooks and sign-in). Open a tile from either list and an administrator finds its install or keys on that integration's page. Organisation → Integrations & webhooks → Webhooks is where you add endpoints. Owners and admins can open both tabs.

Approve from chat#

With the Slack or Microsoft Teams app connected, each approver gets a request as a direct message, never a channel post, with Approve and Decline buttons and a link to the full request. Every copy of the message updates once the request is decided, expires or is withdrawn, wherever that happened. Email still goes out as well, so chat is never the only route to anybody.

1. 1

   Slack

   Choose Add to Slack on the Slack card under Chat and email and finish the install in Slack. Approvers are matched to their Slack account by email address. You can add more than one workspace. To disconnect a workspace, remove the app in Slack: Deliverd deletes that workspace's token and member links as soon as Slack tells it. What the app may do and what it keeps are on the Slack app page.

2. 2

   Microsoft Teams

   Choose Download the app package and upload it in the Teams admin centre or the Microsoft 365 admin centre. The same package adds the Deliverd agent to Microsoft 365 Copilot Chat and Deliverd's tools and skills to Copilot Cowork. Each approver then adds the app from their own chat; the card shows how many have. A Teams app can only message somebody who has added it.

Either card says Coming soon when the app has not been configured on this deployment.

Evaluation tools: Langfuse#

With Compliance Mode, connect your organisation's Langfuse project under Evaluation so evidence can be pulled from it: a build step, a pipeline or a person names an experiment, and Deliverd reads its scores with your keys. Whoever asks for the evidence, an agent included, never holds the keys.

1. 1

   Connect

   Choose where Langfuse runs: Langfuse Cloud by region, or Self-hosted with its https address. A self-hosted Langfuse must be reachable over https from the internet. Paste a project's public and secret keys from its settings in Langfuse, then Connect. The keys are tried against Langfuse before anything is kept, and the card names the project they opened.

2. 2

   Pull

   On a governed task, Pull from Langfuse names the experiment and the controls it speaks to. From a build step: deliverd governance evidence $TASK --langfuse <experiment> --control model_evaluation, or pull on the evidence API.

3. 3

   Change or disconnect

   Change keeps the stored secret if you leave it blank. Disconnect removes the keys; nothing can be pulled until somebody connects again.

Each Langfuse score becomes one result under its own name, so a person sets thresholds on what Langfuse calls it. Only the scores and a link to each trace come across, never the items' inputs or outputs. The secret key is sealed, and only an owner or an administrator connects or disconnects, in the app; no API key can.

Evaluation tools: LangSmith#

Connect LangSmith the same way, under Evaluation, so evidence can be pulled from your LangSmith experiments too. Deliverd reads them with your key, in the workspace you connected, and whoever asks never holds the key.

1. 1

   Connect

   Choose where LangSmith runs: a LangSmith Cloud region, or Self-hosted with its https address (0.16 or later, reachable over https from the internet). Paste a service key for the workspace your experiments are in. A key that spans several workspaces needs the Workspace ID too, from Settings → Workspaces in LangSmith. The key is tried before anything is kept, and the card names the workspace it opened.

2. 2

   Pull

   On a governed task, Pull from LangSmith names the experiment — its name, or the prefix evaluate() was given — and the controls it speaks to. From a build step: pull on the evidence API with "source": "langsmith".

3. 3

   Change or disconnect

   Change keeps the stored key if you leave it blank, for the same address. Disconnect removes it.

Each evaluator becomes one result under its own key, from LangSmith's own statistics over every run, and each run that failed is listed with a link to it. A low score is not called a failure — set thresholds for that. Runs are read by id and status only, never their inputs or outputs, and no feedback comment comes across.

Evaluation tools: Braintrust#

Connect Braintrust under Evaluation to pull from your Braintrust experiments. Deliverd reads them with your key, in the project you connected, and whoever asks never holds the key.

1. 1

   Connect

   Paste a service token from your organisation's settings in Braintrust and name the project your experiments are in. A key that opens several organisations needs the Organisation too. There is no region to choose: Braintrust says where your data is — hosted in the US or the EU, or in your own cloud, which must be reachable over https from the internet. The key is tried before anything is kept, and the card names the organisation and project it opened.

2. 2

   Pull

   On a governed task, Pull from Braintrust names the experiment and the controls it speaks to. From a build step: pull on the evidence API with source set to braintrust.

3. 3

   Change or disconnect

   Change keeps the stored key if you leave it blank — to move to another project, say. Disconnect removes it.

Each score becomes one result under its own name, Braintrust's mean over the cases — the same as a report sent with braintrust eval --jsonl, so one threshold meets either. Each case that errored is listed with a link to it; a low score is not called a failure. Only ids and whether a case errored are read for that, never a case's inputs or outputs.

Evaluation tools: Weave#

Connect Weave, Weights & Biases' evaluation tool, under Evaluation to pull from your Weave evaluations. Deliverd reads them with your W&B key, in the project you connected, and whoever asks never holds the key.

1. 1

   Connect

   Paste a W&B API key — a team service account's is best — and name the project your evaluations are logged to as team/project. Leave Deployment blank for W&B's cloud; for a Dedicated or self-managed deployment, give its address, which must be reachable over https from the internet. The key is tried before anything is kept, and the card names the team and project it opened.

2. 2

   Pull

   On a governed task, Pull from Weave names the evaluation — the name its Evaluation was given, or its run's — and the controls it speaks to. The newest finished run is read. From a build step: pull on the evidence API with source set to weave.

3. 3

   Change or disconnect

   Change keeps the stored key if you leave it blank, for the same deployment. Disconnect removes it.

Every number the evaluation returned becomes a result by its path — exact_match.correct.true_fraction — the same as a report of the dict evaluate() returns, so one threshold meets either. Weave's own summary of the model's outputs is left out. Each case whose model or scorer raised is listed with a link to it; a low score is not called a failure. Cases are read by id and time only, never their inputs or outputs.

Email and browser notifications#

• Email. Approvers are emailed each request. For a low or medium-risk request that no ethics rule flagged, the email carries a link that opens the request without signing in; the approver still presses Approve or Reject on the page. High and critical risk, flagged requests, and organisations that require single sign-on get a plain link behind sign-in.
• Browser notifications. Each person turns them on for their own browser or phone in their settings; on an iPhone, only once the app is added to the home screen. Tapping one opens the request as the email would. See Your inbox and notifications.

What each person is notified about, and how, is theirs to choose in Your settings, except for the events that may not be muted.

Webhooks#

A webhook sends an HTTPS POST to an address you choose whenever something happens. Deliveries are queued and retried on their own, so a publish never waits on your endpoint or fails because of it.

Add an endpoint

1. 1

   Choose a format

   Standard JSON is the signed envelope, for anything you have built. Slack posts a native message to a channel: paste the URL of a Slack Incoming Webhook. Microsoft Teams posts an Adaptive Card: paste the URL from the Workflows app's *Post to a channel when a webhook request is received* template. Discord posts an embed with a button: paste the URL of a channel webhook from *Server Settings → Integrations → Webhooks*.

2. 2

   Paste the URL

   It must start with https:// and be on the public internet; addresses inside a private network are refused.

3. 3

   Choose events

   Tick the events you want, or leave them all unticked to receive every event, including any added later.

4. 4

   Copy the signing secret

   It is shown once, when the endpoint is created, and cannot be retrieved afterwards. If you lose it, remove the endpoint and add it again.

Events

Individual verdicts and replies are recorded in the audit log rather than sent; a channel hears when the answer is in. Actions a gate rule allowed are counted, not sent: only a refusal, gate.denied, is.

Signing

Every delivery carries a Deliverd-Signature header: a timestamp and an HMAC-SHA256 of the timestamp and the raw body, keyed with the endpoint's secret. Check it on every delivery, because the address is all anyone needs to send convincing fakes. Slack and Teams ignore it. The developer docs show how to verify it: Webhooks and callbacks.

Retries and pausing

• Any 2xx answer counts as delivered. Answer quickly and do the work afterwards: a request times out after ten seconds. Redirects are not followed.
• Anything else is tried again with growing gaps, five attempts in all, except a 4xx other than 408 or 429, which would fail the same way again.
• After twenty failures in a row the endpoint is paused and the page says why. Fix the endpoint, then choose Resume. You can also pause an endpoint yourself.

Testing

Each endpoint has Send test: choose an event and it is posted at once, signed like any other, marked test: true, with obviously fake ids. The answer your endpoint gave appears beside the button. A test is never retried and never counts toward the failures that pause an endpoint. Recent deliveries, at the foot of the page, shows the last 20 attempts and what came back.

Posting events to a Slack or Teams channel this way is separate from the chat apps above, which message approvers directly.