Logging LLM Requests Safely Without Building a Liability

The reflex when an LLM feature misbehaves is to log everything — full request body, full response, headers, the lot. It works, in the sense that you can reproduce anything afterwards. It also quietly converts your observability stack into a store of customer data, pasted credentials and whatever else happened to be in the context window that day.

The useful position is not to log nothing. It is that the fields which make a log line diagnostic are almost entirely separate from the fields that make it a liability, and you can keep the first without accumulating the second.

What actually makes a log line diagnostic

Start from the questions you ask when something goes wrong. Was it slow, and slow where? Was the output truncated? Did it retry, and how many times? Which model version served it, and whose budget did it come out of?

Every one of those is answerable from metadata. A trace or request identifier, the resolved model string including any version suffix, timestamps for enqueue, first token and completion, prompt and completion token counts with cached tokens broken out separately, the finish reason, the provider status and error code, the retry attempt number, the names of any tools invoked, and an opaque tenant or user identifier.

That set answers most production incidents without storing a word the user wrote, and it aggregates well, which bodies never do — token counts and finish reasons roll up into the dashboards that catch a regression before anyone files a ticket. Agent observability metrics covers which are worth alerting on.

Prompt bodies are where the liability lives

The body is also the only part whose contents you do not control. Users paste stack traces containing internal hostnames, support tickets containing customer names and addresses, rows from a production database they were debugging, and occasionally an entire environment file. Your prompt template did not ask for any of it and your logging layer will store all of it.

Once it lands in a log store it inherits that store's properties: cross-region replication, retention defaults measured in months, index snapshots, backups, and often a third-party vendor with its own subprocessors. A data classification decision has been made by infrastructure rather than by anyone who thought about it. Handling PII in prompts covers how personal data gets there.

Redact at the boundary, before the write

Redaction belongs in the client or proxy that makes the outbound call, not in a processor further down the log pipeline. Downstream redaction means the raw value genuinely existed in every buffer, agent and queue between the two points, and each of those is a place it can be persisted or crash-dumped.

A gateway or proxy in front of the provider is the natural single place to do this, because it is the one component every call already passes through, and it gives you a consistent policy across every service and SDK in the estate. Proxying LLM traffic covers the shape.

Prefer an allowlist to a denylist when the request is structured. If your prompt is assembled from named fields, log the fields you have classified as safe and drop the rest, rather than trying to enumerate every pattern that might be sensitive. Denylists fail silently on the case you did not think of, and the case you did not think of is the one that matters. Redacting secrets from prompts covers the credential-shaped subset.

Sample by outcome, not uniformly

You do not need every body. You need a handful of representative ones, and you need them disproportionately from the requests that failed.

A policy that works well in practice: capture bodies for every schema-validation failure, every provider error, and every request that exhausted its retries, plus a small percentage of successes so you have a baseline to diff against. Uniform sampling gives you a large corpus dominated by the boring case.

Retention is a decision, not a default

The single most valuable structural change is to stop treating metadata and bodies as one stream. They have completely different useful lifetimes.

Token counts, latencies and error codes stay valuable for months — they are what you use to spot cost drift, model regressions after a version change, and slow degradation in completion rates. Bodies are useful for as long as someone is actively debugging, which is measured in days.

Write them to separate destinations with separate retention, and make the deletion real. Expiry on the primary index is not deletion if the same records sit in nightly backups, a warehouse copy fed by a connector, and a snapshot in object storage. Enumerate every copy before you claim a retention window to anyone.

Access is a separate control from storage

Retention limits how long data exists. It says nothing about who can read it while it does, and in most teams the log platform is the one system where everybody has read access because that is how incident response works.

Putting bodies in their own stream gives you somewhere to attach a tighter role. Fewer people, granted deliberately, with reads recorded. That last part matters more than it sounds: an audit trail on who read what turns an unbounded exposure into a bounded one you can describe accurately if you are ever asked. Agent audit logging covers building that trail.

When you genuinely need the full request

Sometimes the sampled, truncated version is not enough and you need the exact bytes that produced a failure. Handle that as an explicit, time-boxed mode rather than the standing default — a per-request debug flag, set by an operator or a support flow the customer has consented to, that turns on full capture for one tenant for a fixed window. Mark those records so you can purge them later without hunting.

The discipline that makes all of this hold is a single question applied per field: what question does this answer? If nobody can name one, it does not go in the log. Start with metadata always on and a long retention, bodies sampled on failure, truncated, redacted at the proxy and expired within a week — then add fields when a real incident proves you needed one. Replaying agent traces covers getting reproducibility from metadata alone.