Using LangChain With Any OpenAI-Compatible Endpoint

LangChain talks to any OpenAI-compatible endpoint through the same chat model class it uses for OpenAI itself. You change a base URL and a model name and requests go somewhere else. The connection is not the interesting part.

What is interesting is that several LangChain abstractions above the chat model make assumptions about the provider underneath — about structured output enforcement, about tool call formats, about token counting — and those assumptions fail quietly rather than loudly. This is a guide to the two lines that connect it and the six things that break afterwards.

The connection

Use the OpenAI chat integration and override the base URL. The class reads the key from an environment variable by default, or takes it explicitly:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="kimi-k3",
    base_url="https://api.example.com/v1",
    api_key="your-key",
    temperature=0,
    timeout=60,
    max_retries=0,
)

print(llm.invoke("Summarise this repo layout in two sentences.").content)

The base URL ends at /v1. LangChain appends the path itself, so including /chat/completions produces a 404 that looks like an authentication problem. This is the single most common setup failure and it is worth checking first every time.

Set max_retries=0 deliberately. The underlying OpenAI SDK retries by default, and a retry policy you did not choose is a retry policy you cannot see in your metrics. Put retries in one place you control, as described in handling rate limits and retries, so a brief upstream failure does not multiply into sustained load.

Tool calling is where compatibility actually gets tested

Binding tools works the same way as with OpenAI. LangChain converts a function signature or a Pydantic model into a JSON schema and sends it in the tools array:

from pydantic import BaseModel, Field

class SearchDocs(BaseModel):
    """Search the internal documentation."""
    query: str = Field(description="The search query")
    limit: int = Field(default=5, description="Max results")

model_with_tools = llm.bind_tools([SearchDocs])
result = model_with_tools.invoke("Find the docs on session expiry")
print(result.tool_calls)

Whether this works depends entirely on the endpoint honouring the OpenAI tool call contract, and support varies. The failure modes are recognisable: the model describes the tool it would call in prose instead of emitting a tool_calls array, arguments come back as a JSON string that does not parse, or parallel tool calls are collapsed into one.

Test this before you build an agent on it. Bind one trivial tool, invoke with a prompt that obviously requires it, and inspect the raw tool_calls field. If that does not work end to end, nothing above it will. The characteristics that make a model good at this are covered in choosing a model for tool calling, and the underlying protocol in how tool calling works.

Structured output has three implementations

LangChain's structured output helper can be backed by native JSON schema enforcement, by a tool call, or by prompting plus parsing. Which one you get depends on the provider and on the method you request, and they have very different reliability.

Native enforcement, where the server constrains generation to your schema, is the strongest — invalid JSON becomes impossible rather than unlikely. It also depends on the endpoint implementing that feature, and many OpenAI-compatible endpoints accept the parameter and ignore it. An ignored parameter fails silently; you get valid-looking output most of the time and a parse error under load.

The defensive pattern is to assume you are in the prompt-and-parse case: validate every response against the schema yourself, and have a retry path for when validation fails. If you then discover native enforcement works, you have lost nothing. More on the trade-offs in structured outputs and JSON mode.

Streaming, callbacks and the token count problem

Streaming works through the same interface, yielding message chunks you accumulate. The transport-level failure modes — buffering proxies, middleware that consumes the stream, missing usage data — are not LangChain's and are covered in streaming and server-sent events.

The LangChain-specific issue is usage metadata. Callback handlers and cost-tracking utilities were built around OpenAI's response shape and its published prices. Against a different provider you will often get token counts that are absent, partial while streaming, or present but priced against the wrong table.

Treat any built-in cost readout as decorative unless you have verified it against your own billing. Log the raw usage block from responses and do the arithmetic yourself with the prices you actually pay. Token counting utilities that use OpenAI's tokeniser are also wrong for other model families, sometimes by twenty percent or more — see how tokenisers differ across models before you build a budget on an estimate.

Caching, embeddings and the rest of the stack

The chat model is only one integration. If you use LangChain for embeddings, that is a separate client with its own base URL, and many chat-only endpoints do not serve an embeddings route at all. Point the embeddings model at whatever actually provides it rather than assuming one endpoint serves both.

Prompt caching is another place where the abstraction hides a provider difference. Some backends cache implicitly on a shared prefix, some require an explicit marker, and some do neither. LangChain does not normalise this for you, so a chain designed around a stable system prompt may or may not get the discount. Verify by watching for a cached-tokens field in the response rather than by reading documentation. The mechanics are in how prompt caching works.

LangChain's own in-memory and SQLite caches are a different thing entirely — exact-match response caching on your side, which is useful for deterministic replays in tests and does nothing for a chain whose prompts vary.

When LangChain is the wrong tool

If your application makes one chat call and parses the result, the abstraction is overhead: a dependency tree, a layer of indirection when debugging, and a version upgrade cadence to keep up with. Call the endpoint with the OpenAI SDK or plain HTTP and keep the code readable. The OpenAI Python SDK setup takes about the same time.

LangChain earns its place when you genuinely use the ecosystem — multiple retriever backends, document loaders you would otherwise write, or a graph-shaped agent runtime. It is worth being honest about which of those you are actually using, because the cost of the abstraction is paid on every debugging session.

A verification sequence

Before building anything on a new endpoint, run four checks in order. A plain invoke returning text confirms the base URL and key. A streamed call confirms transport. A bound tool with an inspected tool_calls field confirms the agent path. A structured output call with a deliberately awkward schema confirms whether enforcement is real or theatre.

Each takes a minute and each failure points at a different layer. Doing them in that order means the first thing that breaks tells you where the problem is, rather than surfacing three abstractions later as an agent that silently stops calling tools.