Policy as code

Gate policy can be written in the app, and it can live in a repository beside the code it governs: reviewed in a pull request, tested, checked against real traffic, and put in force on merge. Three pieces, each useful on its own.

Replay past decisions#

Before a rule change goes live, see what it would have done. Every gated request from the last 30 days (7 to 90) is evaluated twice: under the rules in force, and with the proposal in its policy's place, beside every other policy. You get how many decisions change, which way, and the requests a person rejected that the proposal would let through without asking.

• In the app: Replay past decisions on any draft in the policy builder.
• From a terminal: deliverd policy replay --file deliverd/policies.
• Over REST: POST /api/v1/policies/replay with a stored version (policyId, version) or a file (name, rules).

It compares two evaluations of the same request, not the proposal with what was recorded at the time, so a change in the count is a change your edit makes. Ethics checks are not replayed; they can only send an allowed request to a person.

Policy files with tests#

One file per policy, JSON or YAML, with the test cases that say what its rules must do. Each case names an action, its input, and an expect of an effect and, optionally, the rule that must decide it (null for none). Cases run through the same schema validation and evaluation a real request meets, beside every other policy in force.

# deliverd/policies/refunds.yaml
name: Refunds
rules:
  - name: Small refunds go through
    priority: 20
    effect: allow
    match:
      actions: [finance.refund]
      input:
        amount: { lte: 50 }
tests:
  - name: a small refund is allowed
    action: finance.refund
    input: { customer_id: cus_1, amount: 20, currency: GBP }
    expect: { effect: allow, rule: Small refunds go through }
  - name: a large one goes to a person
    action: finance.refund
    input: { customer_id: cus_1, amount: 900, currency: GBP }
    expect: { effect: require_human }

deliverd policy test runs a file or a directory of them and exits 1 on a failure. Over REST, POST /api/v1/policies/test takes the file's name, rules and tests and returns each case passed or failed with the reason.

deliverd policy test --file deliverd/policies

In CI#

Store an owner's or administrator's API token as a repository secret. On a pull request, run the tests and the replay, leaving a summary on the run, and fail when a test fails or a request a person rejected would now be allowed (failOn: loosened or changed is stricter). On merge, push each file as the next version with source naming the repository and commit, and put it in force; the app then marks the policy Managed in Git and warns anyone editing it there. skipUnchanged stores nothing for a policy the merge did not change.

With the CLI (0.19.1 or later), set the repository variable DELIVERD_URL to your Deliverd address and the secret DELIVERD_TOKEN, then add:

# .github/workflows/deliverd-policy.yml
name: Deliverd policy
on:
  pull_request:
    paths: ["deliverd/policies/**"]
  push:
    branches: [main]
    paths: ["deliverd/policies/**"]

jobs:
  check:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    env:
      DELIVERD_URL: ${{ vars.DELIVERD_URL }}
      DELIVERD_TOKEN: ${{ secrets.DELIVERD_TOKEN }}
    steps:
      - uses: actions/checkout@v4
      - run: npx -y deliverd@latest policy test --file deliverd/policies --markdown "$GITHUB_STEP_SUMMARY"
      - run: npx -y deliverd@latest policy replay --file deliverd/policies --markdown "$GITHUB_STEP_SUMMARY"

  push:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    env:
      DELIVERD_URL: ${{ vars.DELIVERD_URL }}
      DELIVERD_TOKEN: ${{ secrets.DELIVERD_TOKEN }}
    steps:
      - uses: actions/checkout@v4
      - run: npx -y deliverd@latest policy push --file deliverd/policies --activate --git --skip-unchanged

What will surprise you#

• Replay and test write nothing, and both need an owner's or administrator's key. Agents are refused: knowing which inputs a rule lets through is the search space for getting past it.
• --skip-unchanged stores nothing when the version in force already says the same, so a merge that did not touch a policy adds no version to it.
• A version pushed from Git can still be replaced in the app in an emergency. The next merge replaces it again, which is why the app says so.
• There is no Terraform provider yet. The CLI and the API cover the same ground from any pipeline.