# LLM feature launch checklist

**Feature:** {{feature_name}}
**Owner:** {{owner}}
**Target launch:** {{launch_date}}
**Rollout plan:** {{rollout_plan}} (e.g. internal only, then 5% of accounts, then everyone)

Copy this file into the feature's folder or the launch ticket. Every item is either checked, or marked `n/a` with a one-line reason. Items marked **(blocker)** must be done before any external user sees the feature.

## 1. Scope and ownership

- [ ] **(blocker)** One sentence describes what the feature does and what it must never do.
- [ ] A named owner will be on the hook for quality after launch, not only for shipping.
- [ ] We wrote down which decisions the model makes alone and which a person confirms.
- [ ] We know what the feature does when the model is unavailable (fallback UI, queued retry, or feature hidden).

## 2. Data and privacy

- [ ] **(blocker)** We listed every data field sent to the model provider and confirmed each is allowed by our privacy policy and customer contracts.
- [ ] **(blocker)** The provider, plan and region are approved; data retention and training-use terms for that plan are documented.
- [ ] Fields not needed for the task are removed before the call (names, emails, IDs).
- [ ] Prompts, inputs and outputs that we log have a retention period and access controls.
- [ ] The provider is listed as a subprocessor where required.

## 3. Prompt and model

- [ ] Prompts live in version control and follow our prompt style guide.
- [ ] The model ID is pinned in config, not hardcoded in several places, and we know the provider's deprecation date for it if one is announced.
- [ ] `max_tokens` (or equivalent) and request timeouts are set deliberately.
- [ ] Machine-read output uses structured output or tool calling with a schema, and the code validates it before use.
- [ ] Code handles: refusals, truncated output (stop reason for length), empty output, and invalid JSON.

## 4. Evaluation

- [ ] **(blocker)** An eval set exists with at least the common cases, known edge cases, and inputs in every language we support.
- [ ] **(blocker)** Pass criteria were agreed before the final eval run, and the current version meets them.
- [ ] A person reviewed a random sample of real-looking outputs, not only the eval scores.
- [ ] Evals run in CI when prompts or the model change.
- [ ] We have a baseline to compare against (previous version, simpler heuristic, or no feature).

## 5. Safety and abuse

- [ ] **(blocker)** Prompt injection is tested: instructions hidden in user input, uploaded files, retrieved documents and tool results.
- [ ] **(blocker)** If the model can call tools or take actions, each tool has the least permission it needs, and destructive or costly actions require confirmation.
- [ ] Output shown to users is escaped or sanitized for the place it is rendered (HTML, Markdown links, images).
- [ ] The system prompt contains nothing that would hurt us if a user extracted it.
- [ ] We know how the feature behaves with harmful, abusive or off-topic requests, and the behavior matches our content policy.
- [ ] Users can report a bad output, and reports reach the owner.

## 6. Cost and capacity

- [ ] We estimated cost per request and per active user, and finance or the budget owner has seen it.
- [ ] Per-user and global rate limits protect against loops and abuse.
- [ ] Spend alerts are set with the provider and in our own monitoring.
- [ ] We checked our provider rate limits against expected peak traffic.

## 7. Observability

- [ ] Each request logs: feature name, model ID, prompt version, latency, input and output token counts, stop reason, and error type.
- [ ] Dashboards show error rate, latency percentiles, cost per day, and user feedback rate.
- [ ] Alerts exist for error rate and cost spikes, routed to the owner's on-call.

## 8. User experience and disclosure

- [ ] Users can tell the content is AI-generated where that matters (customer-facing text, decisions about them).
- [ ] The UI sets expectations: what the feature is good at, and that it can be wrong.
- [ ] Users can edit, undo or reject AI output before it takes effect.
- [ ] Loading and streaming states are handled; long waits show progress.
- [ ] Accessibility: generated content is readable by screen readers and keyboard users can reach every control.

## 9. Launch and rollback

- [ ] **(blocker)** A feature flag or kill switch can turn the feature off without a deploy, and we tested it.
- [ ] Rollout starts small, with a named person watching the dashboards for the first {{watch_period}}.
- [ ] Support has a short brief: what the feature does, known limitations, how to escalate.
- [ ] Incident runbook links to the AI incident postmortem template.

## 10. After launch

- [ ] {{review_after_launch}} after launch: review real outputs, feedback, cost and incidents; add failures to the eval set.
- [ ] Re-run evals before any model or prompt change, and before the pinned model's deprecation date.

## Sign-off

| Role | Name | Date |
|---|---|---|
| Feature owner | | |
| Engineering reviewer | | |
| Security or privacy reviewer | | |
| Product | | |