Create the first monitor you would actually trust in production.
PromptCanary works best when the first monitor is narrow, objective, and tied directly to customer or revenue risk. The goal is not broad coverage on day one. The goal is one canary check with signal.
From signup to first check
Start at promptcanary.dev/register, create your workspace, and land in the dashboard without touching infrastructure.
Start with the template that matches the job, paste your endpoint URL, add an API key only if the endpoint requires auth, and keep the first monitor tied to one real production risk.
Start simple with valid_json or one required keyword, run the monitor once, and confirm the diff and alerting surface make sense before scheduling it.
Choose the template that matches the first risk
PromptCanary now starts setup with four one-click templates so the first monitor does not begin as a blank form.
Use this when a workflow expects structured JSON from an OpenAI-compatible endpoint. It starts with valid_json, json_schema, and latency checks.
Use this for support or triage bots that must keep specific phrases present, avoid known failure phrases, and stay within a latency threshold.
Use this for summarisation flows where the response must stay concise and still mention key facts or terms from the source text.
Use this for extraction flows that feed downstream systems and need stable required fields, not just roughly correct prose.
How to keep the initial signal clean
- Use one prompt and one or two objective assertions that cannot lie.
- Prefer JSON validity, schema checks, required keywords, or latency thresholds before subjective scoring.
- Start with the same endpoint and auth pattern you already use in production so the first signal is real.
- Keep fixtures synthetic and non-sensitive so retained artifacts stay safe to inspect.
Good starting point for a first monitor
This is intentionally simple. The first monitor should prove that PromptCanary can catch one concrete regression before you ask it to score nuance.
All supported assertion types
Each monitor can have up to 10 assertions per test case. Combine objective checks for the strongest signal.
valid_json
Passes if the response body is parseable JSON. Use this as the first check for any workflow that expects structured output.
json_schema: { "type": "object", "required": ["intent"] }
Validates the response body against a JSON Schema definition. Fails if the schema does not match or if the body is not valid JSON.
contains: "refund"
Passes if the response text includes the specified string. Case-sensitive. Good for required keywords.
not_contains: "I cannot"
Passes if the response text does not include the specified string. Use to catch known failure phrases.
regex: "^\{"
Passes if the response matches the regular expression pattern. Supports optional flags.
word_count: min 10, max 200
Passes if the response word count falls within the specified range. Either bound is optional.
latency: <= 12000ms
Passes if the time from request to first response byte does not exceed the threshold. Always included — default is 12 000 ms.
semantic_similarity: >= 0.82
Compares the response against a reference answer and fails when the semantic distance exceeds the threshold you set.