Pre-launch checklist for shipping an LLM-powered feature
A launch readiness checklist for LLM features covering scope, data and privacy, prompt and model config, evals, injection and tool safety, cost limits, observability, user disclosure, kill switch and post-launch review, with blockers marked.
File in Product by AISkillHub Library
What it does
LLM features fail in ways normal launch reviews do not look for: a prompt injection through an uploaded file, a provider plan that retains customer data, a retry loop that costs a month of budget in a weekend, a model deprecation nobody tracked. This checklist covers those gaps in one page, with the items that must be done before external users see the feature marked as **blockers**.
**Sections**
1. **Scope and ownership**: one sentence on what it must never do, a named quality owner, and fallback behavior when the model is unavailable. 2. **Data and privacy**: every field sent to the provider listed and approved, plan retention and training terms documented, unnecessary fields removed. 3. **Prompt and model**: prompts in version control, model ID pinned in config, token limits and timeouts set, schema-validated output, handling for refusals, truncation and invalid JSON. 4. **Evaluation**: eval set with edge cases and all supported languages, pass criteria agreed before the final run, human sample review, evals in CI, a baseline. 5. **Safety and abuse**: injection tested through every input channel, least-privilege tools with confirmation for destructive actions, output escaping, report-a-problem path. 6. **Cost and capacity**: per-user cost estimate, rate limits, spend alerts, provider limits checked against peak. 7. **Observability**: per-request logging of model, prompt version, tokens, latency and stop reason; dashboards and alerts. 8. **User experience and disclosure**: AI content labeled where it matters, edit or undo before it takes effect, accessible output. 9. **Launch and rollback**: tested kill switch, small rollout, support brief. 10. **After launch**: scheduled review of real outputs and re-running evals before model changes.
**Variables to fill**
`{{feature_name}}`, `{{owner}}`, `{{launch_date}}`, `{{rollout_plan}}`, `{{watch_period}}` (e.g. "48 hours") and `{{review_after_launch}}` (e.g. "Two weeks").
**What it will not do**
It does not replace your security review or legal sign-off, and it does not tell you what thresholds your evals should hit. Those depend on the feature. Mark items `n/a` with a reason rather than deleting them, so reviewers can see they were considered.
How to use it
1. Copy `docs/llm-feature-launch-checklist.md` into the feature's folder or paste it into the launch ticket. 2. Fill the `{{...}}` fields at the top and in sections 9 and 10. 3. Work through it with the feature owner, an engineer, and whoever handles security or privacy. Every item is either checked or marked `n/a` with a one-line reason. 4. Do not start an external rollout until every **(blocker)** item is checked. 5. Collect sign-offs in the table at the end. 6. Keep the completed file with the feature's docs and revisit it at the post-launch review and before any model change.
Files
Save each file at the path shown, relative to your project root.