LlamaIndex Setup on a Custom OpenAI-Compatible Endpoint

The failure that costs people the most time with LlamaIndex is not a connection error. It is a working pipeline that quietly sends half its traffic to OpenAI because the embedding model was never reconfigured.

LlamaIndex has two model slots, not one. The LLM answers questions; the embedding model turns documents and queries into vectors. They are configured separately, they usually live behind different endpoints, and setting one does not set the other. Everything else in this guide is downstream of that fact.

Set both models explicitly

LlamaIndex reads defaults from a global settings object. Anything you do not set falls back to an OpenAI default, which requires an OpenAI key and produces a confusing authentication error at the point where you build the index rather than where you configured it.

For a chat endpoint that serves the OpenAI chat completions shape but is not OpenAI, use the OpenAI-like integration rather than the OpenAI one. It exists precisely because the strict OpenAI class validates model names against a list it knows, and a custom alias fails that check:

from llama_index.core import Settings
from llama_index.llms.openai_like import OpenAILike

Settings.llm = OpenAILike(
    model="kimi-k3",
    api_base="https://api.example.com/v1",
    api_key="your-key",
    is_chat_model=True,
    context_window=200000,
    timeout=60,
)

Two of those arguments matter more than they look. is_chat_model=True routes requests to the chat completions path rather than the legacy completions path — omit it and you get a 404 against an endpoint that only serves chat. And context_window is what LlamaIndex uses to decide how many retrieved chunks fit into a prompt, so a wrong value causes either wasteful truncation or an upstream overflow error mid-query.

The embedding model is a separate decision

Set it explicitly even if you think you have. If your chat provider does not serve an embeddings route — many do not — point this at whatever does, or run it locally:

from llama_index.embeddings.huggingface import HuggingFaceEmbedding

Settings.embed_model = HuggingFaceEmbedding(
    model_name="BAAI/bge-small-en-v1.5"
)

A local embedding model is often the right answer for a document set that fits on one machine. Embedding is a cheap, parallel, non-interactive workload, and running it locally removes a per-document API cost and a network dependency from your ingestion pipeline entirely.

The constraint to respect is that an index is bound to the embedding model that built it. Change the model, and every stored vector is meaningless against new queries — not wrong in an obvious way, just quietly bad retrieval. Record the model name alongside the index and re-embed from scratch when it changes. There is no incremental path. The reasoning behind vector similarity is covered in how embeddings actually work.

A minimal working pipeline

With both slots set, the standard ingestion and query path works unchanged:

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex

docs = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(docs)

engine = index.as_query_engine(similarity_top_k=5)
print(engine.query("How does session expiry work?"))

Behind those four lines: documents are split into chunks, each chunk is embedded, the vectors go into an in-memory store, and at query time the query is embedded, the nearest chunks are retrieved, and they are stuffed into a prompt with your question. Knowing that shape is what lets you debug it, because every retrieval problem is a problem in one of those steps.

The in-memory store is fine for prototyping and wrong for anything you restart. Persist the index to disk or use a real vector store before you have re-embedded the same corpus for the tenth time.

Chunking decides retrieval quality

Chunk size is the parameter people tune last and should tune first. Too small and a chunk lacks the context to be interpretable on its own; too large and a single retrieved chunk burns budget on mostly irrelevant text while diluting the part that mattered.

The default is a reasonable starting point for prose and usually wrong for code and for structured documents. Code wants chunks that respect function boundaries; a reference table wants to stay whole. LlamaIndex ships several node parsers for this, and picking one that matches your document type is worth more than any amount of prompt tuning downstream.

Overlap exists so that a sentence spanning a boundary is not lost to both chunks. A modest overlap is cheap insurance; a large one duplicates content across chunks, which inflates the index and makes your top-k results near-identical.

Whether you should be chunking at all is a real question now that several models ship very large context windows. RAG versus long context covers the trade: retrieval costs engineering and adds a failure mode, long context costs tokens per query and degrades in the middle of very long inputs.

Watch the token spend in ingestion and query separately

These are two different cost curves and conflating them makes budgets useless. Ingestion is a one-off per document — embedding cost, paid once, proportional to corpus size. Query cost is per request and proportional to similarity_top_k multiplied by chunk size, plus your prompt and the answer.

A top-k of ten with large chunks means every trivial question sends a lot of tokens. Start at three to five and raise it only if you can show that answers improve. Adding a reranking step over a larger candidate set often beats simply raising top-k, because it lets you retrieve broadly and send narrowly.

Log the token usage per query from day one. The counts are the input to any sensible model choice, and the arithmetic in context window cost trade-offs only works with real numbers from your own corpus.

Debugging bad answers

When a query returns something wrong, find out which stage failed before touching the prompt. Inspect the retrieved source nodes on the response object: if the right chunk is not in there, the problem is retrieval — chunking, embedding model, or top-k — and no prompt change will fix it.

If the right chunk is present and the answer is still wrong, it is a generation problem. That is when you look at the prompt template, the model, and whether the relevant text is buried in the middle of a long context where models attend to it least, a well-documented effect described in the lost in the middle problem.

Enable callback-based tracing while developing so you can see the actual prompt sent upstream, not the template. A surprising share of RAG bugs turn out to be a chunk of boilerplate — a navigation menu, a licence header — that got embedded and now dominates retrieval for every query.

A setup checklist

Set the LLM with an explicit base URL, model alias, chat flag and context window. Set the embedding model explicitly, even to a local one. Record which embedding model built each index. Persist the index. Start with a small top-k and a chunk size chosen for your document type. Then verify with one query whose correct answer you already know, and inspect the retrieved nodes rather than just the answer.

If the pipeline works but feels heavy for what you are doing, it may be. A handful of documents that fit comfortably in a modern context window do not need an index at all, and models with very large context windows have made that threshold considerably higher than it was.