Which Embedding Model Should You Use? API vs Self-Hosted in 2026

Choosing an embedding model looks like the easy part of building retrieval. You pick the one everyone uses, pass a string, get an array of floats back, and move on to the interesting problem. The model name sits in one line of configuration and feels like something you can change on a Tuesday.

It is not. That one line fixes the width of every vector you will ever store, the size of your index, the shape of your database schema, and the bill for changing your mind. Embedding is the cheapest part of a retrieval pipeline to run and one of the most expensive to redo. This is a guide to the four things that actually decide the choice — dimensions, hardware, price and migration — and to why the top of the leaderboard is almost never the right answer.

What You Are Actually Choosing

An embedding model turns text into a fixed-length vector of floats, where distance approximates meaning. Every model on the market does that. What separates them is four properties, and only one of them is quality.

Output dimension — how many floats come back. This is your schema. It sets storage, index size, memory pressure and comparison cost, and it is the hardest property to change later.

Context length — how much text fits in one call. OpenAI’s v3 models accept 8,192 tokens. Voyage’s current models accept 32,000. Cohere’s embed-v4.0 accepts 128,000. Long context sounds strictly better, but it mostly changes how much chunking you have to do, and chunking is where retrieval quality is usually won or lost anyway.

Where it runs — a hosted API, or a model you download and serve. This is a data-residency and dependency decision before it is a cost one.

Retrieval quality on your data — not on a benchmark. More on that in a moment.

There is a fifth property that almost nobody checks: what the model knows. OpenAI documents plainly that text-embedding-3-large and text-embedding-3-small have no knowledge of events after September 2021. For most semantic similarity work that is irrelevant, but if your corpus is full of product names and terminology invented since then, the model is embedding unfamiliar tokens.

The Leaderboard Is Not the Answer

MTEB — the Massive Text Embedding Benchmark, later expanded into MMTEB for multilingual coverage — is the standard scoreboard. It is genuinely useful, and it is routinely misread as a ranking of which model you should use.

Here is why. The table below is a snapshot, not a current ranking — these are multilingual scores as reported by the Qwen team in the Qwen3-Embedding-0.6B model card, taken from the MTEB leaderboard on 24 May 2025. Read it as a lesson, not as today’s board:

Two things jump out. In that snapshot the default choice of most teams — OpenAI’s flagship embedding model — sat at the bottom of the table, below an open 0.6B model you can run yourself. And the 0.6B model scored about six points behind the 8B one, at a fraction of the serving cost.

Now the caveats, because they matter more than the ranking. These are vendor-reported numbers from one snapshot of one benchmark, and both have moved since: MTEB now spans over a thousand tasks across more than a thousand languages and has grown to cover image and audio as well. Whatever the board says today, it will not be these numbers. The multilingual mean also averages retrieval, classification, clustering, bitext mining and more, while your application does exactly one of those. It says nothing about your domain, your language pair, your latency budget, your dimensions or your licence.

The lesson survives the staleness, and it is the only part worth keeping: the best-known model is not automatically the best model, and a small open one can be competitive. Use the leaderboard to build a shortlist of three or four candidates, and go and read the live board rather than trusting a table in a blog post. Then settle it the only way that counts: take a hundred real queries, mark which chunk should come back for each, and measure. That evaluation set is a more valuable asset than the model you pick with it, because it survives every future migration.

Dimensions Are a Storage Bill

This is the part that gets discovered in production.

A vector is not free to keep. In Postgres with pgvector, a vector column stores 4 bytes per dimension plus 8 bytes of overhead, and halfvec stores 2 bytes per dimension plus 8. Multiply that out and the model choice becomes a line item:

That is raw column storage, before index and row overhead. The index is the bigger number in practice, and it wants to sit in memory.

Then there is a hard wall. pgvector indexes vector columns up to 2,000 dimensions. A 3,072-dimension embedding from text-embedding-3-large cannot be put behind a plain HNSW or IVFFlat index at all — you need halfvec, quantization, or a narrower vector. This is one of the mechanisms I covered in detail in do you need a vector database, and it is the point where the two decisions meet: the model you choose can disqualify the storage engine you already run.

The good news is that you are usually allowed to ask for fewer dimensions. OpenAI documents that its v3 models are trained with Matryoshka representation learning, which packs the most important information into the leading dimensions of the vector so the tail can be cut off. Qwen lists the same capability on its embedding models, and Voyage exposes an equivalent output_dimension parameter. In OpenAI’s API it is called dimensions:

from openai import OpenAI

client = OpenAI()

# 3,072 dimensions by default — too wide for a pgvector index.
# Ask for 1,024 and it fits, with a modest quality trade.
response = client.embeddings.create(
    model="text-embedding-3-large",
    input="How do I rotate a service account key?",
    dimensions=1024,
)

The headline result from OpenAI’s own documentation is worth stating precisely: on MTEB, a text-embedding-3-large embedding shortened to 256 dimensions still outperforms an unshortened text-embedding-ada-002 embedding at 1,536. Six times narrower, still better. If you are storing default-width vectors because that is what the example code did, you are very likely paying for dimensions you would not miss.

Voyage goes further and lets you request the output data type — int8, uint8, or bit-packed binary, where the returned array is one eighth the length. That is the same idea as quantizing a language model, applied to the output vector instead of the weights.

Does It Need a GPU?

This question comes up constantly and is answered badly, usually by people who have only ever run generative models.

An embedding model is an encoder. One forward pass over the input, a pooling step, done. There is no autoregressive loop, no KV cache growing per token, no sampling. A 0.6B embedding model and a 0.6B chat model are not remotely the same workload. And because every input in a batch is independent, embedding parallelises almost perfectly — throughput scales with batch size until you run out of memory.

That changes the answer depending on which side of the pipeline you are on.

Query time embeds one short string per request. A CPU handles this comfortably. Sentence Transformers explicitly suggests moving small embedding models to CPU in latency-sensitive local setups, so they do not contend for VRAM and compute with a generative model on the same GPU.

Index time embeds millions of chunks. Here a GPU is about finishing today instead of over the weekend. It is a scheduling decision, not an architectural one — and it is usually a rented box for a few hours, not a permanent line item.

If you go the CPU route, do not run raw PyTorch. Sentence Transformers supports three backends — PyTorch, ONNX and OpenVINO — and provides helpers to export and quantize models to int8 for CPU inference. Its own decision guidance runs roughly: on GPU, use Flash Attention if the model supports it; on CPU, decide whether a small accuracy loss is acceptable, then take OpenVINO on Intel hardware and ONNX elsewhere.

from sentence_transformers import SentenceTransformer

# int8-quantized ONNX build, exported once and reused.
model = SentenceTransformer(
    "sentence-transformers/all-MiniLM-L6-v2",
    backend="onnx",
    model_kwargs={"file_name": "onnx/model_qint8_avx512_vnni.onnx"},
)

embeddings = model.encode(chunks, batch_size=64)

For a serving path rather than a script, Hugging Face’s Text Embeddings Inference ships container images for both GPU and CPU, which turns “run an embedding model” into a deployment rather than a Python process someone owns.

The extreme end of the CPU argument is static embedding models — attention-free, dramatically faster, notably weaker. They exist, they are documented, and for high-volume low-stakes similarity they are a real option worth benchmarking before you assume you need hardware.

What the API Actually Costs

Hosted embedding is cheap in a way that is easy to get wrong in both directions.

OpenAI’s published prices:

The pages-per-dollar column is OpenAI’s own, assuming roughly 800 tokens per page. Note the third row: ada-002 is five times the price of text-embedding-3-small and scores lower on MTEB — 61.0% against 62.3% in OpenAI’s table. If it is still in your code, it is there because nobody revisited it.

Now split the cost by phase, because the two behave completely differently.

Queries are free in practice. A search query is maybe fifteen tokens. A million queries is fifteen million tokens, which is thirty cents with the small model. You will never see this on an invoice.

Indexing is the bill. Five billion tokens — a serious corpus — is about $100 with text-embedding-3-small and about $650 with text-embedding-3-large. Still not much. Which is exactly why the next section matters, because that number is not paid once.

Against that, self-hosting is not free either: a GPU instance for the indexing window, engineering time, a container to keep patched, and a model file to version. The honest crossover is rarely about the token price. It is about whether your text is allowed to leave your infrastructure, whether you need to pin a model version that a vendor cannot deprecate under you, and whether you already run inference hardware for something else. If none of those apply, the API is very hard to beat at these prices.

The Migration Nobody Prices

Here is the trap.

Vectors from two different models are not comparable. Not “less accurate” — meaningless. Different dimensionality, different geometry, different training objective. There is no conversion. So switching model means re-embedding every chunk you have ever stored, and rebuilding every index over them.

That is why the indexing cost above is the number to plan around. It is not a one-time setup fee; it is the price of every future opinion change about your embedding model. At $650 per full pass the money is trivial, but the operational choreography is not — you cannot serve half-migrated results, because a query embedded with the new model against documents embedded with the old one returns noise.

The pattern that works is a shadow index:

1. Add a second vector column, or a second collection, alongside the live one.
2. Backfill it in the background at whatever rate your budget and rate limits allow.
3. Run your evaluation set against both. Compare on your queries, not on a leaderboard.
4. Cut reads over behind a flag, keep the old column for a rollback window, then drop it.

Two things make that migration cheaper before you ever need it. Store the model name and dimension count alongside every vector, so a partially migrated table is legible rather than a mystery. And keep the raw chunk text, not just its vector — if you have to re-derive the source text from somewhere else to re-embed, the migration goes from a background job to a project. Both cost nothing on day one, and both are the difference between a weekend and a quarter later on. The same instinct applies to how you chunk in the first place, which I get into in context engineering.

The Bottom Line

Start with text-embedding-3-small at 1,536 dimensions, or a small open model like Qwen3-Embedding-0.6B if the text cannot leave your infrastructure. Both are cheap, both are fast, both are good enough to build the rest of the pipeline against.

Then do the four things that actually matter. Build an evaluation set of real queries with known-correct answers, because it outlives every model you will use. Ask for fewer dimensions than the default and measure what you lose — often nothing you can detect, at half the storage. Assume you will change models, and store the model name and the raw text so that day is a background job. And check the leaderboard for a shortlist, never for a decision.

The model is the easy call. The vector width is the schema, and the schema is what you are stuck with.