Using an LLM to Generate and Edit Config Files

Configuration is an unusually good fit for a language model, and for a reason that has nothing to do with the model being clever. Config files have a schema, that schema is usually machine-readable, and the result can be checked by a program before anyone runs it. You get a proposal you can mechanically reject.

That property is rarer than it sounds. Most generated code is only verifiable by running it, which means the check is expensive and late. A CI pipeline definition, a Kubernetes manifest, a Terraform block or a bundler config can be validated in milliseconds against a schema you already have. The model becomes a fast draft generator inside a loop you control.

Why config is a good target

Three things line up. The output is small, so it fits comfortably in a context window and in a diff a human can read. The format is constrained, so there is a finite set of valid keys rather than an open-ended program. And the failure is usually loud: an invalid manifest is rejected by the API server, an unknown key in a strict parser throws.

There is also a lot of it in training data. Public repositories are full of CI workflows, Docker Compose files and Nginx blocks, so a model has seen thousands of examples of the shape you are asking for. That is why the output looks so convincing, and it is also the root of the main failure mode discussed below.

The economics work too. A config edit is a short prompt and a short completion, so per-run cost is low enough that iterating three times to get it right is still cheaper than reading documentation for twenty minutes. That calculus does not hold for large code generation, where a wrong answer costs you a review cycle.

Supply the schema, do not describe it

The single largest quality improvement comes from putting the actual schema in the prompt rather than naming the tool and hoping. If a JSON Schema exists, paste the relevant subset. If the tool ships a schema file, read it and include it.

Describing the schema in prose invites the model to fill gaps from memory, and memory is an average of every version of that tool it has seen. Supplying the schema converts the task from recall into transformation, which is where models are strongest and most reliable.

Where no schema exists, the next best input is a known-good example from your own repository. Two or three real files establish the version, the conventions and the house style far more reliably than instructions do. This is the same reasoning behind supplying real table definitions in LLM-assisted schema design rather than describing your data model.

Trim what you supply. A 4,000-line schema for a tool where you need six keys wastes context and dilutes attention. Extract the relevant definitions, and if you are generating the same kind of config repeatedly, keep that extract as a reusable prompt fragment.

Constrain the output format

Ask for the config alone, in a fenced block, with no commentary. Prose wrapped around the output is the most common reason a generation cannot be piped straight into a validator, and it is trivially avoidable with an explicit instruction and a small parsing step that extracts the block.

Where the provider supports it, structured output modes give you a stronger guarantee than an instruction does, because the decoder is constrained rather than merely asked. That works cleanly for JSON and for anything you can express as JSON and serialise afterwards. Structured outputs and JSON mode covers what those modes actually enforce.

YAML deserves a specific warning. It is not JSON with fewer brackets, and models generate YAML with subtly wrong indentation, unquoted strings that parse as booleans, and the classic Norway problem where the country code becomes false. Generate JSON and convert, or parse and re-serialise with a real YAML library before comparing anything.

Validate mechanically before you apply

Treat every generation as untrusted input and run it through a gate. Parse it, validate it against the schema, and only then show it to a human or write it to disk. If any step fails, feed the error back and regenerate rather than fixing it by hand.

const draft = extractBlock(completion)
const parsed = parseYamlOrJson(draft)            // throws on syntax errors
const result = validator.validate(parsed)        // schema check

if (!result.valid) {
  return retryWith(result.errors.map((e) => e.message).join('\n'))
}
await writeAndDiff(parsed)

Most tools bring their own validator, which is better than a generic one because it encodes semantics a schema cannot. Use kubectl apply --dry-run=server, terraform validate, nginx -t, docker compose config, or the linter your CI provider ships. These catch cross-field constraints that a structural check misses.

Feeding the validator error back into the next attempt is what makes the loop converge. An untouched retry re-rolls the same dice; a retry carrying the specific rejected key becomes a correction. The cost arithmetic behind that difference is in the hidden cost of retries.

The plausible-but-wrong key

This is the failure mode that matters, and it is specific to config. The model produces a key that is correctly formatted, sits in the right place, has a sensible name and a sensible value, and does not exist in the version of the tool you are running.

It happens because the model has seen the key in a different tool, an older release, a fork, or a blog post proposing it. The output is not random; it is an average over sources, and averages contain keys from all of them. A related failure is a real key whose default changed between versions, so the generated value silently means something different than intended.

Strict validation catches most of these, which is exactly why the validation step is not optional. But many config parsers ignore unknown keys by design, so the file loads, the tool runs, and the setting you thought you applied does nothing. That is worse than an error, because there is no signal at all.

The defence is to prefer strict parsing modes where they exist, to diff generated config against the previous version rather than reading it whole, and to verify the effect rather than the file. Confirm the timeout actually changed by observing behaviour, not by reading the key you asked for. The general mechanism behind this class of error is covered in why LLMs hallucinate.

Edits beat rewrites

Asking for a modified version of an existing file usually returns the whole file, and the whole file usually contains changes you did not request: reordered keys, dropped comments, normalised quoting, a version pin quietly bumped. Reviewing that diff is more work than making the edit yourself.

Two things help. Ask for the change as a patch or as the specific block to replace, so the diff is minimal by construction. And always review the actual diff rather than the output, because your eye slides over an unchanged-looking file in a way it does not over three highlighted lines.

Comments are worth protecting explicitly. Config comments carry the reason a setting exists, which is precisely the information nobody can reconstruct later. If your pipeline round-trips YAML through a parser that drops comments, you have built a comment-destroying machine.

Where this fits in a pipeline

The useful deployment is not a chat window. It is a step that takes a request, the current file and the schema, produces a candidate, validates it, and opens a pull request with the diff. Everything that follows is your existing review process, which already knows how to handle config changes.

Keep the model out of the apply path. Generation and validation can be automatic; applying to production should go through whatever gate a human-written change goes through. Config is exactly the category where a plausible mistake is deployed instantly and discovered slowly, so the argument for a person in the loop is stronger here than in most places — see human-in-the-loop design for where to put the gate.

The decision rule is simple. If you have a schema and a validator, generation is worth automating. If you have neither, you are trading twenty minutes of reading documentation for an unbounded debugging session, and the model is the wrong tool. The same test applies to the adjacent case of generating migration scripts, where the verification step is harder and the stakes are higher.