Emacs LLM Setup: A Model Endpoint Without the Mouse

Emacs users want the same thing from a model that they want from everything else: a buffer, a keybinding and no mouse. That is achievable, but the packages in this space each make a different assumption about where the text you care about lives, and picking the one that disagrees with how you work means fighting it every day.

The connection itself is the easy part. Every mainstream Emacs LLM package will talk to an OpenAI-compatible endpoint given a base URL, an API key and a model identifier. The decisions worth thinking about are the workflow shape and how you keep the key out of a dotfiles repository you push publicly.

The packages come in three shapes

The first shape is the chat buffer. You open a dedicated buffer, type, send, and the reply is appended. The buffer is a normal Emacs buffer, so it saves, searches, and yanks like anything else. This is the closest analogue to a web chat and the easiest to reason about, which is why most people start here.

The second shape is the in-place transformer. You mark a region, invoke a command, and the region is replaced or annotated with the result. There is no conversation and no history — one input, one output, applied where the point is. For rewriting a docstring or converting a block of JSON into a struct, this is far faster than copying text into a chat.

The third shape is a client library rather than a user interface: a package that exposes a function you call from your own elisp, with a callback for the response. If you want a command that summarises the current Magit diff into a commit message, this is the layer you build it on. Many of the buffer-based packages are themselves built on one of these.

Most people end up with two installed — a chat buffer for open-ended work, an in-place command for edits — and that is a reasonable place to land rather than a failure to choose.

Configure the endpoint, not the vendor

Whatever the package, the configuration reduces to the same three values. The base URL should be the root of the OpenAI-compatible surface, ending at the version segment. Almost every client appends the operation path itself, so including /chat/completions in the base URL produces a duplicated path and a 404 that reads like an authentication failure. Our explainer on what OpenAI compatibility actually covers goes through the rest of the surface.

The model identifier is forwarded verbatim. There is no fuzzy matching, so a name your provider does not publish comes back as an error rather than resolving to something close. Read it from the model list route rather than from memory.

Several Emacs packages have a notion of a named backend, which is worth using even if you only have one. A backend gives the endpoint a label you can switch between interactively, so trying a second provider is a command rather than an edit to your init file and a restart.

(setq my-llm-base-url "https://api.example.com/v1"
      my-llm-model    "kimi-k3")

Get the key out of your init file

Emacs has a good answer to this and it is older than the problem: auth-source. Put the credential in ~/.authinfo.gpg, which is encrypted at rest and decrypted on demand, and read it with a function rather than a literal.

machine api.example.com login apikey password cozy_xxxxxxxx

(defun my-llm-key ()
  (auth-source-pick-first-password
   :host "api.example.com" :user "apikey"))

Most packages accept a function where they accept a string, and calling it lazily matters: it means the key is fetched when a request is made rather than sitting in a variable for the lifetime of the session, and it means a stale value is fixed by editing the file rather than restarting Emacs.

If you prefer an environment variable, remember that an Emacs started from a desktop launcher or as a daemon at login does not inherit a shell profile. The variable that works when you launch from a terminal will be empty in the daemon, and the resulting error looks like a rejected key rather than a missing one. That single difference accounts for a large share of confused reports.

Buffer workflows versus region workflows

The choice between these is not aesthetic — it changes what the model sees. A chat buffer sends a conversation: every previous turn is part of the request, so context accumulates and so does the cost of each subsequent message. That is the right shape for exploring a problem, and the wrong shape for the fortieth mechanical rewrite of the day.

A region command sends exactly the region, plus whatever system prompt you configured. It is cheap, repeatable and stateless. The failure mode is the opposite one: the model has no idea what the surrounding file does, so it invents plausible names for things that already exist three lines above the region.

The practical middle ground is a region command that also sends a small amount of surrounding context — the enclosing function, the file path, the current major mode — assembled by your own elisp. That is where the client-library packages earn their place. It is also worth understanding how a context window is actually consumed before you start attaching whole buffers to every request, because a long chat session in a large file grows faster than people expect.

Streaming, and why Emacs makes it awkward

Without streaming, a long answer means a frozen-looking editor followed by a wall of text. With streaming, tokens land in the buffer as they arrive and you can read while it writes, which changes how usable the whole thing feels.

Emacs is single-threaded, so streaming has to be done with an asynchronous process filter rather than a blocking read. Any package that gets this right will use url-retrieve asynchronously or shell out to curl and attach a process filter; any package that gets it wrong will lock the editor until the response completes. If your Emacs stops responding to input during a request, that is the diagnosis, and it is not something you can configure away.

The other buffering culprit sits between you and the provider. A corporate proxy or a gateway that does not flush chunk by chunk will hold the whole response and deliver it at the end, and the client cannot tell the difference between that and a slow model. The mechanics of the wire format are in the server-sent events walkthrough, and handling dropped streams covers what to do when a long generation dies halfway.

Keybindings that do not fight Emacs

Do not scatter commands across the global map. Bind a prefix and hang everything off it, so the whole feature is discoverable through one key and does not collide with a mode you install next year. The reserved user prefix is C-c followed by a letter, and that is the right place for it.

(global-set-key (kbd "C-c l l") #'my-llm-chat)
(global-set-key (kbd "C-c l r") #'my-llm-rewrite-region)
(global-set-key (kbd "C-c l c") #'my-llm-commit-message)

Keep the region command and the chat command distinct rather than making one command guess from whether a region is active. Guessing is fine until the day you have a stale mark set and a rewrite silently replaces half a file. Version control is the only real undo for that, and it is worth having the edit be an explicit choice.

If you use a modal editing layer, put these on the local leader for programming modes rather than globally. They are editing commands and they belong next to your other editing commands.

Verifying and debugging

Test in order. First a one-sentence chat request, which confirms the base URL, key and model name together. Then a region rewrite, which confirms your prompt assembly. Then a long request, which confirms streaming actually streams rather than arriving in one lump.

When something fails, reproduce it outside Emacs before touching your configuration. A plain curl with the same three values tells you immediately whether the problem is the provider or the editor, and the curl recipes cover the exact shapes to try. If curl works and Emacs does not, look at the *Messages* buffer and, for url-based packages, enable the HTTP debug output so you can see the real status code rather than a generic failure.

If you run several tools against the same provider, keep one credential across all of them rather than minting a key per editor — one key across your tooling makes rotation a single action instead of an archaeology exercise.

The takeaway

Pick a chat buffer package and an in-place command, put the key in ~/.authinfo.gpg behind a function, and confirm streaming works before you judge the experience. Bind everything under one C-c prefix, keep chat and region commands separate, and when it breaks, reproduce with curl before you edit your init file.