OpenAI Python SDK Setup for Any Compatible Endpoint

The official OpenAI Python client is the most widely supported way to talk to any endpoint that speaks the chat completions shape. It is not tied to OpenAI as a company — it is tied to a request and response format that most providers now implement.

Which means the useful version of this guide is not "how do I install it". It is: which client defaults will surprise you, what to set explicitly, and which errors you should be catching by class rather than by string.

Constructing the client

Set the base URL and the key on the client rather than relying on ambient environment variables. Implicit configuration is convenient until you need two clients in one process, at which point it becomes a bug you cannot see.

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["LLM_BASE_URL"],   # ends at /v1
    api_key=os.environ["LLM_API_KEY"],
    timeout=30.0,
    max_retries=0,
)

resp = client.chat.completions.create(
    model="kimi-k3",
    messages=[
        {"role": "system", "content": "Answer in one paragraph."},
        {"role": "user", "content": "What does a KV cache do?"},
    ],
)
print(resp.choices[0].message.content)
print(resp.usage)

The base URL ends at /v1. The SDK appends /chat/completions itself, so including it yields a 404 that most people first read as an auth problem. If a key that worked yesterday returns 404 today, check the URL before the key.

The model string is passed through verbatim. There is no fallback and no fuzzy matching — an alias your endpoint does not recognise is an error, not a substitution. That is a feature; it means you always know which weights answered.

The two defaults worth overriding

The client retries twice by default and allows ten minutes per request. Both are reasonable for a script and wrong for a service.

A ten-minute timeout means a hung upstream connection occupies a worker for ten minutes. Under load that is how one slow provider takes down an unrelated endpoint in the same process. Set a timeout that reflects what your caller will actually wait for — thirty seconds for an interactive path, longer only where something genuinely needs it.

The retry default is subtler. Two automatic retries on 429 and 5xx are invisible in your own metrics: your dashboard shows one request, your bill shows three, and your latency percentiles include silent backoff. Set max_retries=0 and implement retries in one place you control and can observe. The reasoning, and what a sane backoff looks like, is in rate limits and retries.

Per-request overrides exist too, which is the right tool when one call genuinely needs longer than the rest — a large summarisation job on an otherwise interactive client, for instance.

Catch errors by class

The SDK raises a typed hierarchy rather than a single generic exception, and using it is the difference between a retry loop that works and one that hammers a dead endpoint.

from openai import (
    APIConnectionError, APITimeoutError,
    RateLimitError, AuthenticationError,
    BadRequestError, APIStatusError,
)

try:
    resp = client.chat.completions.create(model=MODEL, messages=msgs)
except RateLimitError as e:
    # Retryable. Honour Retry-After if present.
    retry_after = e.response.headers.get("retry-after")
except (APIConnectionError, APITimeoutError):
    # Retryable, network side.
    pass
except (AuthenticationError, BadRequestError):
    # Not retryable. Fix the request or the key.
    raise
except APIStatusError as e:
    # Everything else with an HTTP status: inspect e.status_code.
    raise

The rule is that 4xx other than 429 means your request is wrong and retrying changes nothing, while 429, 5xx and connection failures are worth another attempt. Branching on the exception class is more portable than branching on the error code string in the body, because those strings differ between providers even when the status codes agree. The error code reference covers what each status usually means, and debugging LLM API errors covers the ambiguous ones.

Async, and when it matters

There is an async client with an identical surface. Use it when you are fanning out — evaluating a prompt against fifty fixtures, embedding a corpus, running shadow traffic against two providers at once.

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url=BASE_URL, api_key=KEY, max_retries=0)

async def ask(q):
    r = await client.chat.completions.create(
        model="kimi-k3",
        messages=[{"role": "user", "content": q}],
    )
    return r.choices[0].message.content

async def main(questions):
    sem = asyncio.Semaphore(8)
    async def guarded(q):
        async with sem:
            return await ask(q)
    return await asyncio.gather(*(guarded(q) for q in questions))

The semaphore is not optional. Firing five hundred concurrent requests at a rate-limited endpoint produces five hundred 429s, and if you left retries on, fifteen hundred requests. Bound the concurrency to something below your rate limit and raise it only when you have measured the headroom.

Streaming

Set stream=True and iterate. The SDK handles the server-sent event framing, so you receive parsed chunk objects rather than raw data: lines:

with client.chat.completions.create(
    model="kimi-k3", messages=msgs, stream=True,
) as stream:
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            print(delta, end="", flush=True)

Two details. The delta is often None — on the first chunk carrying only a role, and on the last carrying only a finish reason — so guard before concatenating. And usage is generally absent from a stream unless you ask for it; if you bill or budget on tokens, verify you actually receive it. More on the transport in streaming in Python.

Use the context manager form. It closes the underlying connection if you break out of the loop early, which matters because an abandoned stream that stays open keeps generating and keeps costing.

Tool calls and structured output

Both are just request fields, so both depend on the endpoint rather than the SDK. Send the tools array, read message.tool_calls from the response, and check that arguments parse as JSON before you act on them — they arrive as a string, not an object, and a truncated response produces a string that will not parse.

The failure to watch for on a new endpoint is silent degradation: the request is accepted, no error is returned, and the model answers in prose describing the function it would have called. That is not an SDK bug. It means tool calls are not being negotiated, and the fix is a different model or a different endpoint. Test with one trivial tool before building an agent on it — the same check described in how tool calling works.

The same caution applies to response_format. Unknown parameters are usually ignored rather than rejected, so an endpoint that does not implement schema enforcement accepts your request and silently gives you unconstrained output. Validate every response against your schema regardless — see structured outputs and JSON mode.

A short checklist

Set base URL, key, timeout and max_retries=0 explicitly on one shared client. Catch by exception class, not by error string. Bound concurrency with a semaphore. Use the streaming context manager. Log resp.usage from day one, because every cost question later needs numbers you did not collect.

Then verify in order: a plain call, a streamed call, a tool call. Each one that fails points at a different layer, and running them in that sequence means you never debug three at once.