Before you ship an LLM feature: a checklist
The demo works, the team is pleased, and someone asks when it can go live. Before it does, it’s worth running through a short list. None of this is exotic, but each item is something that has bitten teams after launch.
The checklist
Figure 1 · A pre-launch checklist
| Area | The question | Done when |
|---|---|---|
| Quality | How good is it on real inputs? | A fixed test set exists and a pass bar is agreed |
| Safety | What if someone feeds it hostile text? | Permissions are minimal and risky actions need approval |
| Privacy | What personal data does it touch? | You know where prompts and logs are stored and for how long |
| Cost | What does one request cost, and what is the cap? | There is a per-request estimate and a limit on runaway use |
| Failure | What happens when the model is slow or wrong? | Timeouts, fallbacks and a human handover are in place |
| Monitoring | How will you know it’s getting worse? | Logs, user feedback and a regular review exist |
| Ownership | Who is responsible after launch? | A named person owns it |
The ones that get skipped
Cost. OWASP’s 2025 list for LLM applications includes unbounded consumption, which covers runaway API costs as well as denial of service. One long loop or one abusive user can run up a bill. Set limits before launch, not after the first surprise.
Quality. “It looked fine in the demo” isn’t a test. Use real inputs, including awkward ones, and keep the set so you can re-run it after every change. The spreadsheet method is enough to start.
Safety and privacy. Read the prompt injection note and, if you handle personal data in India, the DPDP note.
Launch small
Release to a limited group first, watch real use for a week or two, and fix what you find. A narrow launch with good monitoring beats a wide one you can’t see into. If you’d like a second pair of eyes on a feature before it ships, that’s part of what we do in our projects.