What I'd Check Before Putting an LLM Feature in a Government System

Getting a language model to do something impressive in a demo is easy now. Point it at a folder of PDFs, write a decent prompt, and you can have a room full of nodding heads by the afternoon.

Putting that feature into a government system is a different project. The data rules are stricter, the users are busy, and some of them will act on what it tells them. Below is roughly the list I’d go through before writing much code. It isn’t complete, and your agency’s own AI and security policies come first.

Before picking a model

Write the task down in one sentence. “Summarize incoming incident reports so the duty officer can triage faster” is a task; “use AI on our documents” still needs work.

Then work out what a wrong answer costs. A clumsy draft email gets fixed and forgotten. A wrong eligibility determination is serious, and a feature like that needs a person to review the output before anything happens.

It’s also worth asking whether you need a model at all. Better search, a rules engine, or a redesigned form will sometimes do the job with much less risk.

Data and hosting

• Classify everything the feature touches, including prompts and outputs. If the inputs are sensitive, assume the outputs are too.
• Find out where a model is allowed to run. For many agencies that means a FedRAMP authorized service or a model deployed inside an environment you already control.
• Ask the provider whether they keep or train on your inputs, and get the answer in writing.
• Make retrieval respect existing permissions. If a user can’t open a document, the model shouldn’t quote it to them.

Evaluation

Prototypes are usually weakest here. Before you tune prompts, build a set of test cases from real examples. A few dozen genuine cases will teach you more than hundreds of made-up ones. For each case, write down what a good answer looks like in terms a program manager would recognize.

Run the evaluation whenever the prompt, the model, or the retrieval setup changes. A provider updating their model counts, even if you didn’t touch anything. Read the failures one at a time, too, since an average score can look fine while a whole category of answers is consistently wrong.

Security

• Treat retrieved documents and user input as untrusted. Prompt injection is a real attack, and a document can contain instructions aimed at the model.
• Give the model the narrowest permissions that work. If it can call tools, require a person to confirm anything that changes data.
• Log prompts, retrieved context, and outputs where policy allows, so you can trace a bad answer later.
• Put the feature through your normal security review and authorization process instead of leaving it off to the side as an experiment.

Running it day to day

Estimate costs from realistic usage, since the pilot’s handful of queries will understate them. Decide what the application does when the model service is down. Give users an easy way to flag a bad answer, and make sure someone reads the flags.

Also plan for the model you tested against to be retired. Providers do retire models, and you’ll need to run your evaluations again when that happens.

Governance

Check the feature against your agency’s AI policy, and against the NIST AI Risk Management Framework if your organization uses it. Tell users clearly when they’re looking at generated content, and give the feature a named owner so someone stays responsible for it. Most of this is ordinary engineering discipline, applied to a component that won’t always give the same answer twice.