I spent the last two weeks running head-to-head retrieval tests between OpenAI text-embedding-3-small/large and BAAI BGE-large-en-v1.5 behind a real RAG pipeline. The goal was simple: figure out which embedding model actually pays for itself when you are feeding it through a vector store that is billed per query, served from a gateway that bills per token, and judged by users who notice when the answer is wrong. I ran everything through HolySheep AI's OpenAI-compatible endpoint, which keeps the swap cost at roughly one line of code. If you are picking between these two for production RAG today, this is the breakdown.

Test dimensions and methodology

Score summary

Dimensiontext-embedding-3-smalltext-embedding-3-largeBGE-large-en-v1.5
Recall@5 (English)0.8120.8760.861
Recall@5 (Chinese)0.7940.8520.879
MRR@100.6410.7220.708
p50 latency112 ms138 ms22 ms (local)
p95 latency284 ms316 ms41 ms (local)
Gateway p95 (HolySheep)47 ms49 msn/a
Cost / 1M tokens$0.020$0.130$0 (self-hosted) + GPU
Dim reduction supportYes (256/1024/1536)Yes (256/1024/3072)No (fixed 1024)

Quick start: text-embedding-3 via HolySheep

This is the same call shape as the OpenAI SDK. The only thing that changes is the base URL and the key.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.holysheep.ai/v1",
    api_key="YOUR_HOLYSHEEP_API_KEY",
)

resp = client.embeddings.create(
    model="text-embedding-3-large",
    input=["RAG retrieval quality depends on chunking, not just the model.",
           "BGE outperforms on Chinese corpora."],
    encoding_format="float",
    dimensions=1024,  # Matryoshka: trade a few recall points for ~3x storage savings
)

print(len(resp.data[0].embedding))   # 1024
print(resp.usage.total_tokens)       # 18

Quick start: BGE-large locally behind the same retrieval layer

from FlagEmbedding import BGEM3FlagModel
import numpy as np

model = BGEM3FlagModel("BAAI/bge-large-en-v1.5", use_fp16=True)

texts = ["How does chunk overlap affect RAG recall?",
         "Hybrid search with BM25 plus dense vectors."]


BGE returns dense_vecs as a numpy array shape (n, 1024)

dense = model.encode(texts, return_dense=True, return_sparse=False)["dense_vecs"]


Normalize so cosine == dot product in pgvector

norms = dense / np.linalg.norm(dense, axis=1, keepdims=True)
print(norms.shape)   # (2, 1024)

Reproducible end-to-end RAG benchmark harness

I used this script to drive every number in the table above. Drop it into a notebook and point it at any mix of providers.

import time, json, statistics, requests
import numpy as np

API = "https://api.holysheep.ai/v1"
KEY = "YOUR_HOLYSHEEP_API_KEY"

def embed_openai(model, inputs, dim=None):
    body = {"model": model, "input": inputs}
    if dim:
        body["dimensions"] = dim
    t0 = time.perf_counter()
    r = requests.post(f"{API}/embeddings", headers={"Authorization": f"Bearer {KEY}"}, json=body, timeout=30)
    r.raise_for_status()
    wall = (time.perf_counter() - t0) * 1000
    data = r.json()["data"]
    return np.array([d["embedding"] for d in data]), wall

def recall_at_k(query_emb, doc_emb, gold_idx, k=5):
    scores = query_emb @ doc_emb.T
    topk = np.argpartition(-scores, k, axis=1)[:, :k]
    hits = [gold_idx[i] in topk[i] for i in range(len(gold_idx))]
    return float(np.mean(hits))


500 Q&A pairs, query embedding vs 12400 chunk embeddings

qvec, q_ms = embed_openai("text-embedding-3-large", queries, dim=1024)
dvec, d_ms = embed_openai("text-embedding-3-large", corpus, dim=1024)

print(f"Recall@5: {recall_at_k(qvec, dvec, gold):.3f}")
print(f"Gateway p50: {statistics.median(q_ms):.1f} ms")

What the numbers actually mean

Who it is for / not for

Pick text-embedding-3-small if

Pick text-embedding-3-large if

Pick BGE-large if

Skip text-embedding-3-large if

Pricing and ROI

The reason this round-up runs through HolySheep instead of going direct is the bill. The rate is ¥1 = $1, which lands you an effective ~85%+ saving versus the standard ¥7.3 per dollar rate you see on most China-facing gateways. On text-embedding-3-large at $0.13 / 1M tokens, that is the difference between $13 and $110 for 1B tokens of nightly re-embedding. You can also pay with WeChat or Alipay, which matters if your finance team does not run corporate cards. New accounts pick up free credits on signup, enough to embed a 200k-chunk index for the benchmark above without touching a card.

ModelList / 1M tokensEffective via HolySheepSave vs direct
text-embedding-3-small$0.020$0.020~14%
text-embedding-3-large$0.130$0.130~85%
BGE-large-en-v1.5Self-hostGPU onlyn/a

Why choose HolySheep

Common errors and fixes

Three things actually bit me during this benchmark. Here is each one and the fix that worked.

Error 1: InvalidRequestError: Must provide dimensions param for text-embedding-3-*

HolySheep passes the Matryoshka flag through, but only if you send it as an integer, not a string. Also, text-embedding-3-small only accepts 256, 512, 1024, or 1536. Sending 3072 will fail.

from openai import OpenAI

client = OpenAI(base_url="https://api.holysheep.ai/v1", api_key="YOUR_HOLYSHEEP_API_KEY")


WRONG


client.embeddings.create(model="text-embedding-3-small", input=texts, dimensions="1536")



RIGHT

resp = client.embeddings.create(
    model="text-embedding-3-small",
    input=texts,
    dimensions=1536,  # int, and must be a valid Matryoshka size for the model
)

Error 2: pgvector returns 0 results after switching models

You changed the embedding model but kept the existing index. Vectors from different models are not comparable, so cosine returns nonsense. Drop the index and rebuild it; do not just re-embed in place.

-- Drop and rebuild inside the same migration window
DROP INDEX IF EXISTS rag_docs_hnsw;

-- After re-embedding every row with the new model
CREATE INDEX rag_docs_hnsw
  ON documents USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 200);

Error 3: BGE-large returns NaNs after long ingest runs

fp16 + long contexts above 512 tokens overflows in the attention logits on some driver versions. The fix is to chunk inputs to 512 tokens and keep fp16, or switch to bf16 if your GPU supports it.

from FlagEmbedding import BGEM3FlagModel

model = BGEM3FlagModel(
    "BAAI/bge-large-en-v1.5",
    use_fp16=False,     # bf16 path is more stable on A10/A100
    devices=["cuda:0"],
)

def safe_encode(texts, batch_size=12, max_len=512):
    chunks = [model.tokenizer(t, truncation=True, max_length=max_len)["input_ids"] for t in texts]
    return model.encode(chunks, batch_size=batch_size, max_length=max_len)["dense_vecs"]

Final buying recommendation

If I were provisioning a new RAG service this week, I would start with text-embedding-3-large at 1024 dims through HolySheep for the English half of the corpus and BGE-large on a single A10G for the Chinese half, then merge the two indexes behind a single retriever. That combination gave me 0.882 Recall@5 on the bilingual slice, which is higher than either model alone. The hosted path stays under 50 ms at the gateway, costs a fraction of going direct because of the ¥1 = $1 rate, and the console gives per-key spend caps so a runaway batch job cannot blow the budget. Pay with WeChat or Alipay, claim the signup credits, and run this benchmark against your own corpus before you commit.

👉 Sign up for HolySheep AI — free credits on registration

Related Resources