A few days ago, I wrote about what 10 million documents teach you the hard way. The principles. The discipline. The moment when “it works” stops being good enough.

This is the sequel. The part where you take those 60 million documents and try to make them searchable by meaning.

Because keyword search stops being useful at this scale. Users don’t always know the right term. They don’t always spell it the same way. They’re looking for concepts, not strings. And the only way to bridge that gap is embeddings, turning every document into a dense numerical vector that captures what it’s about, not just what words it contains.

Simple idea. Brutal execution.

The napkin math that changes everything

At 1,000 documents, embeddings are a solved problem. Call OpenAI. Get vectors back. Store them. Done. You don’t even think about it.

At 60 million documents, that napkin math starts looking different.

OpenAI’s embedding API charges per token. Even their latest budget model, text-embedding-3-small, runs about $0.02 per million tokens. For 60 million documents with full text, we’re talking roughly 90 billion tokens. That’s $1,800 for a single embedding pass. And if your model improves, or you want a different dimension, or you realize your text preprocessing was wrong? Pay that $1,800 again. Every time.

That’s when the question shifts from “which API should I call” to “should I be calling an API at all?”

Why the L40S, not the H100

When most people think “GPU for AI,” they think H100. It’s the king of training. But for embedding 60 million documents, the NVIDIA L40S is a strategically better choice. Here’s why.

48GB of GDDR6 memory. Embedding at scale requires loading a large transformer model (like BGE-M3) and processing massive batches to keep the GPU saturated. The L40S has enough VRAM to hold the model comfortably while leaving headroom for the large input/output buffers that high-throughput batching demands. It offers more memory than the standard A100 (40GB variant) and is far more accessible than the 80GB H100s, which are often overkill for inference workloads.

Native FP8 via the Transformer Engine. The L40S is built on Ada Lovelace architecture, which includes a dedicated Transformer Engine that automatically recasts computations into FP8 precision. For embeddings, which don’t require the extreme precision of scientific simulations, FP8 provides a roughly 2x speedup over FP16 without losing meaningful accuracy. Older cards like the A100 lack native FP8 support entirely.

18,176 CUDA cores. More than the H100’s 14,592. The L40S is a parallel processing monster built for inference throughput, not training. It’s designed to chew through millions of small inference requests simultaneously, which is exactly what embedding is: millions of short texts, each producing a 1024-dimensional vector.

Economics and availability. H100s are often on backorder or carry a premium tax on cloud providers. The L40S uses standard PCIe slots and doesn’t require specialized cooling like SXM-based H100s, making it far easier to rent bare metal quickly. Rentals run $0.63-$0.99/hour on platforms like RunPod or CloudRift, often 35-50% cheaper than comparable A100 or H100 time.

The math becomes… rent three L40S GPUs at $0.87/hour each, run them for a few days, and embed your entire corpus for a fraction of what any API would charge. And if you need to re-embed? Run it again. Same cost. No API pricing anxiety.

API vs. self-hosted: the real comparison

Here's how the economics actually stack up for 90 billion tokens:

The API cost has dropped significantly since I first estimated it. But the “API anxiety” remains. If you need to tweak your chunking strategy, switch from title+abstract to full text, or change models entirely, you pay the full amount again every time. With rented silicon, you own the compute cycles for as long as you need them.

And there are advantages beyond cost.

• Zero “token tax”: the longer you keep the GPU saturated, the lower your effective cost per token drops.
• Predictable performance: no fluctuation based on global API demand.
• Data privacy: with a reverse SSH tunnel, your documents never leave your controlled environment, which matters for enterprise and sensitive datasets.

For anyone doing the long-term math: a new L40S retails for approximately $8,000-$9,000. If your business requires constant re-indexing or real-time embeddings for millions of users, the card pays for itself in about five full runs of a 60 million document corpus.

The architecture nobody draws on a whiteboard

Here’s what the tutorials don’t show you. When your GPU is rented in a data center in Europe and your database is in a container on a different server, there is no clean localhost connection. There’s no shared network. There’s no “just point it at the database.”

What actually works… reverse SSH tunnels.

Each GPU server establishes an SSH tunnel back to the database host, forwarding a local port to PostgreSQL. The embedding script runs on the GPU, calls the embedding model over localhost (fast, no network hop), and writes results through the tunnel to the database.

It’s not elegant. It’s not what you’d put in an architecture diagram to impress anyone. But it works, and the separation turns out to be an advantage: the GPU does what it’s good at (matrix math), and the database does what it’s good at (ACID writes), and the SSH tunnel is just plumbing between them.

The real bottleneck isn’t where you think

Here’s the thing that surprised me most. The GPU is fast. Embarrassingly fast. An L40S can embed a batch of 512 documents in under two seconds. The embedding model is not the bottleneck.

The database is the bottleneck.

Each cycle requires three round-trips through the SSH tunnel:

1. Claim a batch of unclaimed documents (UPDATE with FOR UPDATE SKIP LOCKED)
2. Write the embeddings back (batch INSERT with ON CONFLICT)
3. Update the document status (mark as complete)

At a batch size of 64, each cycle takes about 12 seconds. The GPU spends 0.5 seconds computing and 11.5 seconds waiting for database round-trips. That’s 95% idle time on hardware you’re paying by the hour for.

The fix was almost embarrassingly simple: increase the batch size from 64 to 512. Same three round-trips, eight times more documents per trip. Throughput jumped 4x overnight. Not from better hardware. Not from a smarter algorithm. From changing a single integer.

The second optimization was removing an ORDER BY published_at DESC from the claim query. It felt responsible when I wrote it, process newest documents first. But at 59 million unclaimed rows, that ORDER BY was forcing PostgreSQL to sort tens of millions of rows before selecting 512 of them. Removing it let the database grab any available batch instantly.

Combined with pipelining, embedding the next batch while the previous one writes to the database, we went from 300 documents per minute per GPU to over 1,400.

A 4x improvement from a configuration change and a modest architectural tweak.

The index that was secretly killing us

Then we found the real monster.

We were using pgvector with an HNSW index on the embedding column. HNSW (Hierarchical Navigable Small World) is the standard index type for approximate nearest neighbor search on vectors. It’s what makes similarity queries fast at query time.

What nobody emphasizes enough: HNSW indexes are maintained on every INSERT. Every time you write a new embedding, PostgreSQL has to find the nearest neighbors in the existing graph and update connections. That’s an O(log n) operation that gets progressively slower as the graph grows.

At 1.25 million rows, our HNSW index was already 11 GB. The table data itself was 8 GB. The index was larger than the data it was indexing, and it was being rebuilt incrementally on every single write.

We dropped it.

DROP INDEX idx_doc_embeddings_1024_hnsw;

One line. Throughput immediately jumped another 2.5x.

Here’s the full optimization journey:

From 1,000 documents per minute to 10,300. A 10x improvement. None of it came from better GPUs. All of it came from removing things that were in the way.

The plan is simple: finish the bulk load without the index, then rebuild HNSW as a single operation afterward. Building the index once on 60 million vectors will take hours, but it only happens once, and it produces a far more efficient graph than incremental maintenance ever could.

This is one of those lessons that seems obvious in retrospect…

do not maintain a search index during a bulk load.

But when you’re building an application incrementally, adding documents a few at a time, the index feels natural. It’s only when you cross into batch-ETL territory that it becomes a boat anchor.

Concurrent workers and the art of not stepping on yourself

With three GPUs writing to the same database, you need coordination. Two workers can’t embed the same document. Two workers can’t claim the same batch.

PostgreSQL has a beautiful primitive for this: FOR UPDATE SKIP LOCKED. Each worker grabs a batch of unclaimed documents and atomically marks them as in-progress. If another worker tries to grab the same rows, it skips them and takes the next available ones. No distributed locks. No message queues. No coordination service.

Just a SQL clause that does exactly what you need.

UPDATE documents SET embedding_status = 1

WHERE id IN (
    SELECT id FROM documents
    WHERE embedding_status = 0
    LIMIT 512
    FOR UPDATE SKIP LOCKED
) RETURNING id, title, abstract, full_text

This one pattern — claim, process, mark complete — scales horizontally without any of the workers knowing about each other. Add a fourth GPU? It just works. Remove one? The others don’t notice. It’s the kind of design that makes you wonder why distributed job queues exist.

The SDK that fought back

One detour worth mentioning. When I first tried connecting to the GPU embedding server from the main application, I used the OpenAI SDK. It’s the standard. Every embedding API speaks the same protocol.

Except the SDK adds its own URL path segments. It transforms your endpoint into something like /openai/deployments/model-name/embeddings, which is an Azure-specific convention that no other embedding server understands. The open-source Text Embeddings Inference server expects plain /v1/embeddings.

After too long debugging why perfectly good HTTP requests were returning 404s, I threw out the SDK and wrote a raw HttpClient wrapper. Twelve lines of code. No path mangling. No surprises.

Sometimes the “standard” tool is the wrong tool.

What the numbers actually look like

After optimization, here’s where we landed:

For context, the same job through OpenAI’s API would have cost 5-6x more and taken just as long due to rate limits. And we’d have no ability to re-run it cheaply if our preprocessing improved.

The meta-lesson

Every layer of this project has taught me the same thing in a different accent…

at scale, the expensive part is never what you think it is.

At 10,000 documents, the expensive part of embeddings is the model. The intelligence. The quality of the vectors.

At 60 million, the expensive part is the plumbing. The batch size. The round-trip latency. The claim query. The write throughput. The SSH tunnel keepalive configuration. A 4x speedup came from changing an integer and removing a sort clause, not from a better model or better hardware.

This keeps happening. Search performance wasn’t limited by the ranking algorithm, it was limited by scanning too many candidate rows before ranking. Import speed wasn’t limited by parsing, it was limited by thread pool starvation in the job scheduler. And embedding throughput wasn’t limited by the GPU, it was limited by how many documents we packed into each database round-trip.

The glamorous part is always the model, the algorithm, the architecture. The leverage is always in the plumbing.

What I’d do differently

If I were starting over…

1. Start with self-hosted embeddings from day one. The API is fine for prototyping, but it creates a dependency on pricing that scales linearly with your ambition. Own the inference. The models are open-source. The hardware is rentable.
2. Design the batch size for the write path, not the model. I sized batches for what the embedding model could handle comfortably. I should have sized them for what the database connection could amortize.
3. Drop all non-essential indexes before bulk loading. This one cost me days of unnecessary runtime. HNSW indexes are designed for query-time performance, not write-time performance. Maintain them during normal operations. Drop them during bulk ETL. Rebuild once at the end. The resulting index will actually be higher quality because it's built with full knowledge of the data distribution.
4. Separate the embedding pipeline from the application entirely. Embedding is a batch ETL job, not a feature of the web application. It should run on its own infrastructure, on its own schedule, with its own monitoring. Trying to coordinate it through the application’s job scheduler (Hangfire, Sidekiq, Celery, whatever) introduces thread pool contention, memory pressure, and operational coupling that you don’t need.
5. Track embedding status in the embedding table, not the document table. We used an embedding_status column on the documents table to coordinate workers. It works, but it couples document state with embedding state. A cleaner design: just check whether an embedding row exists. LEFT JOIN ... WHERE embedding.id IS NULL. No status column to manage. No state machine to debug.

The current state

As I write this, three L40S GPUs are grinding through 59 million remaining documents at about 4,000 per minute. Zero errors. No human intervention needed. The scripts recover from connection drops automatically, coordinate through the database without talking to each other, and log their progress so I can check in whenever I want.

It’s not glamorous work. There’s no demo to show. No UI to screenshot. It’s infrastructure, the kind that only gets noticed when it breaks.

But when those 60 million vectors are in place, every document in the system becomes searchable by meaning. A researcher looking for studies about “cardiovascular mortality in elderly populations” will find papers titled “Heart Disease Death Rates Among Seniors”, papers that share zero keywords with the query but answer exactly the right question.

That’s the promise of embeddings. And the gap between the promise and the reality is about 10 days of GPU time, 900 dollars, and a batch size of 512.

A series about building CiteLynq, a citation verification platform.

• In Part 1, we covered the principles are the foundation that keeps your debt from spiraling.
• In Part 3, we’ll tackle the final boss: Hybrid Search Retrieval. Once you have the data and the embeddings, how do you actually get the right answer back out?

Scale doesn’t just change the size of your database; it changes the rules of the game. See you in Part 3.