OpenAI Node SDK Setup Against a Custom Base URL

The Node client is a thin wrapper over fetch with types, retries and stream parsing attached. Pointing it at a non-OpenAI endpoint is one constructor argument. The things that go wrong afterwards are almost all environmental: where the code runs, what cancels it, and what buffers it.

This guide assumes you are calling an endpoint that serves the chat completions shape and that you want the code to survive a real deployment rather than a local script run.

One client module, constructed explicitly

// lib/ai.ts
import OpenAI from 'openai'

export const client = new OpenAI({
  baseURL: process.env.LLM_BASE_URL,   // ends at /v1
  apiKey: process.env.LLM_API_KEY,
  timeout: 30_000,
  maxRetries: 0,
})

Export one instance rather than constructing per call. It reuses connections, and it means the base URL appears exactly once in your codebase — so changing providers is a one-line edit rather than a search across handlers.

The base URL ends at /v1; the SDK appends the route. A doubled path is the most common first-run failure and it surfaces as a 404 that reads like an authentication problem.

Never construct this client in browser code. The SDK deliberately refuses to run in a browser unless you set a flag whose name tells you it is dangerous, because doing so ships your API key to every visitor. If a browser needs model output, put a route handler in front of it.

Override the retry and timeout defaults

The client retries failed requests automatically and allows a generous request timeout. Both defaults exist to make a first script work, and both are wrong in a service.

Automatic retries are invisible in your own instrumentation. One logged request can be three billed requests, and the backoff time hides inside your latency numbers. Set maxRetries: 0 and put retry logic somewhere you can see it fire, honouring Retry-After when the response carries it.

A long timeout is worse in Node than in most runtimes because a hung request holds an event loop slot and a serverless invocation you are paying for by the second. Pick a timeout that matches what your caller will wait for, and override it per request for the rare call that legitimately runs long.

Cancellation is not optional

Every request method accepts an abort signal in its options. Use it, and thread the incoming request's signal through:

export async function POST(req: Request) {
  const { messages } = await req.json()
  const completion = await client.chat.completions.create(
    { model: 'kimi-k3', messages },
    { signal: req.signal },
  )
  return Response.json({ text: completion.choices[0].message.content })
}

When a user closes the tab, the browser aborts its fetch. Without propagation your handler keeps waiting, the provider keeps generating, and you pay in full for output nobody will read. On a chat interface where people abandon long answers routinely, this is a real and entirely avoidable line item.

The same applies to your own timeouts and to any race between two providers. Whichever loses, abort it — a fallback that leaves the original request running doubles your cost on every failover, which is a quiet contributor to the hidden cost of retries.

Streaming and the runtimes that break it

Set stream: true and the returned object is an async iterable of parsed chunks:

const stream = await client.chat.completions.create(
  { model: 'kimi-k3', messages, stream: true },
  { signal: req.signal },
)

for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content
  if (delta) process.stdout.write(delta)
}

Note the optional chaining. Chunks arrive that carry only a role or only a finish reason, and indexing into them unguarded is the most common runtime error in otherwise working streaming code.

Breaking out of the loop early leaves the underlying connection open unless you close it, so abort the request rather than just returning from the function. The SDK exposes a controller for this, and the abort signal above covers the common case.

Then there is deployment. Streaming that works locally routinely arrives all at once in production because something in the path buffers — a reverse proxy, compression middleware, a serverless runtime that collects the whole response, or your own logging middleware reading the body. Diagnose with curl -N against the deployed route before touching application code. The full list is in streaming and server-sent events, and the client side is covered in streaming in TypeScript.

Typed errors, and what to do with each

The SDK exports error classes rather than making you parse strings. Branch on those:

import OpenAI from 'openai'

try {
  await client.chat.completions.create({ model, messages })
} catch (err) {
  if (err instanceof OpenAI.APIConnectionTimeoutError) { /* retry */ }
  else if (err instanceof OpenAI.RateLimitError) { /* back off */ }
  else if (err instanceof OpenAI.AuthenticationError) { /* fix the key */ }
  else if (err instanceof OpenAI.BadRequestError) { /* fix the request */ }
  else if (err instanceof OpenAI.APIError) { console.error(err.status, err.message) }
  else throw err
}

Class checks are portable across providers in a way that error code strings are not. Two endpoints will agree on HTTP 429 and disagree on the code inside the body, so keying off status first and the code only as a refinement survives a provider change. What each status usually means is in the LLM API error codes reference.

One case deserves separate handling: a request that returns 200, starts streaming, and then fails partway. Your user already has partial text on screen. Decide deliberately whether to keep it marked incomplete or discard it — silently leaving truncated output is the one option that is always wrong, because the reader cannot tell.

Types, tools and structured output

The SDK ships TypeScript types for request and response shapes, which is worth using rather than casting. Type your messages array as the SDK's message parameter type so a malformed role or a missing content field fails at compile time instead of as a 400 in production.

Tool calls are a request field, so whether they work is a property of the endpoint. Send the tools array, read message.tool_calls, and parse arguments — it is a JSON string, not an object, and a truncated response yields a string that will not parse. Wrap that parse in a try block; a model returning malformed arguments should be a handled path, not a crash.

The silent failure to watch for is a model that answers in prose describing the function it would call. That means tool calls are not being negotiated at all. Test with one trivial tool before you build a loop on it — see choosing a model for tool calling. The same silent-ignore risk applies to response_format, so validate structured responses against your schema regardless of what the endpoint claims to support.

A setup order that isolates failures

Make one non-streaming call from a server route and log usage. That proves base URL, key and model alias together, and gives you the token numbers every later cost question needs.

Then stream the same route and verify with curl -N before involving any client framework. Then add abort propagation and confirm that closing the tab actually stops generation. Then add tools, one at a time, inspecting the raw tool call each time.

Four steps, each with one plausible failure cause. The alternative is a chat feature that shows nothing and four layers that could be responsible.