Generating Regular Expressions With an LLM, Carefully

Regular expressions are close to an ideal use case for a language model. They are dense, hard to write from memory, easy to test, and the cost of a wrong one is usually a failing test rather than a lost afternoon. Most developers who use a model for regex report it as the single highest hit-rate task they give it.

They are also a trap in a specific way. A regex that is nearly right looks identical to a regex that is exactly right, because neither is readable at a glance. You cannot review a pattern the way you review a function. The only meaningful review is running it against inputs, which means the discipline is entirely about test cases rather than about reading.

Describe the target with examples, not adjectives

Asking for a pattern that matches an email address, a phone number or a valid identifier produces the internet average of that pattern, which is a pattern from a blog post that may or may not match your data. The word does not pin down the target.

Instead, supply examples. Five or six strings that must match and five or six that must not, drawn from real data rather than invented. The negative examples do more work than the positive ones, because they are what constrain the pattern from being too permissive, and a too-permissive regex is the failure that reaches production.

Pull the examples from the actual corpus. Grep the log file, sample the column, take the real header lines. Invented examples share your assumptions about the data, and your assumptions about the data are precisely what is wrong when a regex fails in production. The same argument applies to sampling real rows in LLM-based data extraction.

State the surrounding conditions too. Whether the pattern will be anchored, whether it runs against a whole line or a whole file, whether the input can contain newlines, and whether matching is case sensitive. Each of these changes the correct answer and none is inferable from the examples alone.

Always demand test cases in the same response

Make the deliverable a pattern plus a runnable test table, never a pattern alone. This costs almost nothing in tokens and changes what you receive, because generating the tests forces the model to commit to a specific interpretation of your request that you can then check.

const cases: [string, boolean][] = [
  ['2026-08-30', true],
  ['2026-8-30', false],
  ['2026-13-01', false],
  ['x2026-08-30', false],
]

for (const [input, expected] of cases) {
  const got = pattern.test(input)
  if (got !== expected) throw new Error(input + ': got ' + got)
}

Then read the test table before you read the pattern. Disagreements between the tests and what you meant are visible in seconds, whereas the same disagreement encoded in the pattern is invisible. If the model asserted that a case should fail and you expected it to pass, you have found the specification gap without ever parsing the regex.

Add your own cases on top, particularly the empty string, a very long string, a string containing only the delimiter, and one with leading or trailing whitespace. These four catch a large share of real regex bugs and models rarely generate them unprompted. Keeping the table in the repository turns it into a regression test for the day someone tweaks the pattern — the general practice is covered in regression testing generated output.

Catastrophic backtracking is the dangerous failure

Most wrong regexes fail loudly. One category fails by consuming CPU until the process dies, and it is the one you have to actively look for because the pattern usually passes every functional test you wrote.

The trigger is nested or adjacent quantifiers that can match the same input in exponentially many ways: a group with a quantifier inside another quantifier, or two adjacent variable-length pieces that overlap in what they accept. On a matching input the engine finds an answer immediately. On a slightly-off non-matching input it explores every combination before giving up, and the time doubles with each additional character.

Backtracking engines are the norm — PCRE, Java, .NET, JavaScript and Python all use one. Go and Rust use RE2-style engines with linear-time guarantees and simply refuse to compile the constructs that cause the problem, which is a real advantage if you have the choice.

The practical defences: prefer a specific character class to a dot wherever you can, avoid a quantified group containing another quantifier, anchor the pattern so failure is decided early, and test with a long non-matching string of the shape the pattern almost accepts. If a hundred-character input takes measurable time, the pattern is unsafe on user input. Where the engine supports possessive quantifiers or atomic groups, they eliminate the class entirely; where it does not, rewrite. If the pattern will ever see input from outside your system, add a timeout or a length cap regardless.

Dialects differ more than you expect

A model trained on the whole internet has seen every regex dialect and does not reliably distinguish them. Ask for a pattern without naming the engine and you will get a plausible blend.

The recurring mismatches are worth memorising. Lookbehind exists in JavaScript, PCRE and .NET but not in Go or older Python builds, and variable-length lookbehind is rarer still. Named groups use different syntax across engines. POSIX classes are widely but not universally supported. Unicode property escapes need an explicit flag in JavaScript and are absent in some engines. And grep, sed and awk have their own basic and extended flavours where escaping rules for parentheses and braces invert.

So name the engine and the version in the prompt, and say where the pattern will run: browser JavaScript, Node, Python, a Postgres query, a Kubernetes admission rule, a log-collector config. Then verify by compiling it in that engine rather than trusting that it looks fine.

Watch the escaping layer too. A regex embedded in a JSON config, a YAML file or a shell command passes through an extra round of escaping, and a pattern that is correct in isolation becomes wrong once serialised. Test the pattern as it will actually be loaded, not as it appears in the chat window.

Ask for the pattern to be explained back

A useful and cheap step is requesting a component-by-component breakdown alongside the pattern. Not because the explanation is authoritative, but because it is checkable against your intent in a way the pattern is not.

When the explanation says a group matches an optional protocol prefix and you never wanted a protocol, you have found the mismatch without decoding anything. The explanation is a translation back into the language you specified in, which makes round-tripping possible.

Verbose or extended mode is worth requesting where the engine supports it, since a commented multi-line pattern survives being maintained by someone who did not write it. A dense one-liner in a codebase is a pattern nobody will ever safely change, which means the next fix will be a rewrite. This is the same reasoning as demanding readable generated code anywhere else — see generating tests with LLMs for the parallel case.

When a parser beats a regex

The most valuable thing a model can tell you is that you should not use a regex, and it will not tell you unless you ask. Add the question explicitly: is a regex the right tool here, or is there a parser.

The clear cases are nested or recursive structures — HTML, JSON, XML, balanced brackets, source code. These are not regular languages and a regex cannot match them correctly, only approximately, and the approximation fails on the inputs that matter. Every mature ecosystem has a parser for these, and the parser is faster to use and impossible to get subtly wrong in the same way.

The other cases are formats with a specification and a library: URLs, email addresses, dates, semantic versions, CSV, IP addresses and CIDR ranges. A URL parser handles internationalised domains, percent encoding and default ports. A regex you generated this morning handles the seven examples you supplied. Use the library and reserve regex for the genuinely ad hoc: a log line format specific to your application, a filename convention, a marker in a text file.

The dividing line is whether the format has a grammar someone else has already implemented. If it does, the regex is technical debt with a plausible appearance. If it does not, generate one, test it hard, and check it does not backtrack. And when the task is really classification rather than extraction, a model call may be a better fit than either — that trade-off is in using an LLM for classification.

The short version

Supply real matching and non-matching examples rather than a description. Name the engine. Require a test table in the same response and read it before the pattern. Add the empty string, the long string and the whitespace cases yourself. Time the pattern against a long near-miss input to rule out backtracking. Keep the tests in the repository. And ask whether a parser exists before accepting any of it.