PromptCanary CI Quality Gates

Docs / CI quality gates

Block risky prompt and model changes before they ship.

PromptCanary can run as a synchronous quality gate in CI. A passing monitor exits cleanly. A failing monitor returns a non-zero exit code so the workflow stops before rollout.

Back to docs Create API key

Setup

Three-step path

Create a workspace API key with runs:trigger scope from /dashboard/api.

Choose a monitor id or stable slug to represent the workflow you want to gate.

Call the CI quality-gate path from GitHub Actions or any other CI runner and let the returned exit code decide the job.

Exit codes

exit 0Monitor passed — deploy continues

exit 1Monitor failed — workflow stops

exit 2Configuration error

GitHub Actions

Public action syntax

name: PromptCanary gate

on:
  pull_request:

jobs:
  quality-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: promptcanary/action@v1
        with:
          monitor: support-bot
          api-key: ${{ secrets.PROMPTCANARY_KEY }}

The action interface is now shaped for a standalone public GitHub Action, with `monitor` and `api-key` as the only required inputs for the hosted PromptCanary app.

Operator signal

What the same monitor surfaces when output drifts

CI-triggered quality-gate runs stop the job without sending notifications. The scheduled version of the same monitor can still post a diff-rich Slack alert like this when the behavior later regresses in production.

Slack alert

#ai-incidents

Live diff

PromptCanary10:42 AM

⚠ Support bot checkout classifier

Regression detected — 1 of 2 test cases failed quorum.

Endpoint: https://api.example.com/support-bot

Response diff (last passing → latest failing)

- "intent": "refund_request"
- "intent": "general_support"
- "priority": null

View monitor in PromptCanary

Local dev

Use the same action from this repo

name: PromptCanary gate (local repo)

on:
  workflow_dispatch:

jobs:
  quality-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./
        with:
          monitor: support-bot
          api-key: ${{ secrets.PROMPTCANARY_KEY }}
          base-url: http://127.0.0.1:3000

This repository keeps the action at the repo root in `action.yml`, so local verification uses `uses: ./` with an explicit `base-url` override.

Direct API

Call the quality gate without the CLI

curl -X POST \
  -H "Authorization: Bearer pc_..." \
  "https://app.promptcanary.dev/api/v1/monitors/support-bot/runs?trigger=ci"

`?trigger=ci` tags the run as a CI quality-gate check.
Monitor lookup accepts either the monitor id or the stable monitor slug.
CI-tagged runs are stored in history, but they do not overwrite scheduler state or fire alert notifications.