Vector search has moved from exotic to table stakes. pgvector is bundled into Amazon RDS and Aurora, Google Cloud SQL, Azure Database for PostgreSQL, and countless self-managed clusters — and the benchmark posts keep coming: 10,000 queries per second! 50x faster than brute force! The New Stack recently published a sharp critique of the genre — “The reason your pgvector benchmark is lying to you” — and anyone evaluating vector search for a real workload should read it before choosing a database.

That warning matters in Houston, where teams are building retrieval-augmented generation (RAG) pipelines over well logs, seismic attribute libraries, clinical notes at Texas Medical Center institutions, and Port of Houston shipping documentation. A benchmark built on toy data will send you down the wrong path — and you won’t find out until production. Here are the traps, and how to benchmark pgvector honestly.

Trap 1: Toy Datasets

Benchmarks over 10,000 or even 100,000 vectors say almost nothing about a 50-million-row production table. Approximate nearest neighbor (ANN) indexes shine exactly when brute force stops being viable — usually somewhere past a few million vectors at 768 to 1536 dimensions. If a benchmark doesn’t state dataset size, dimensionality, and the distance metric, it isn’t comparable to your workload. Test with your own data at production scale, or at least a representative slice of it.

Trap 2: Speed Without Recall

HNSW and IVFFlat are approximate indexes. They trade a little accuracy for a lot of speed — and the speed number is meaningless without the accuracy number. The metric that matters is recall@k: of the true k nearest neighbors, how many does the index actually return? A 1 ms query that returns 60% of the true top-10 is not a win; in a RAG pipeline it is a hallucination generator.

Trap 3: The Wrong Index — or None at All

Posting “we scanned 5M vectors in 40ms” usually means the benchmark ran a sequential scan and called it a day — that number does not generalize. And when an index is used, defaults matter: an IVFFlat index created with the default lists on a large table, or an HNSW index built with default m and ef_construction, can silently underperform. pgvector’s documentation is explicit about when each index type makes sense.

Trap 4: Single-Client Latency

RAG applications are concurrent applications. A single-threaded latency number from EXPLAIN ANALYZE ignores connection pool saturation, WAL and vacuum overhead, and CPU contention. Measure throughput under realistic concurrency — 20, 50, 100 parallel queries — and watch p95 and p99, not just the median.

Trap 5: The Warm Cache Mirage

Running the same query ten times means the second run hits shared_buffers, not disk. Real workloads have working sets that do not fit in memory. Run with a cold cache, size your shared_buffers honestly, and report the cache state alongside the numbers.

Benchmark Like You Mean It

Start with a schema that mirrors production:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE documents (
  id          bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  chunk_text  text NOT NULL,
  embedding   vector(1536)          -- e.g. text-embedding-3-large
);

-- HNSW: a good default for most RAG workloads
CREATE INDEX ON documents
  USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

Then measure recall against an exact baseline. Run the exact query with sequential scans enabled, and the approximate query with the index forced:

-- Exact baseline (brute force)
SET LOCAL enable_seqscan = on;
SELECT id FROM documents
ORDER BY embedding <=> $1
LIMIT 10;

-- Approximate via HNSW
SET LOCAL enable_seqscan = off;
SET LOCAL hnsw.ef_search = 40;
SELECT id FROM documents
ORDER BY embedding <=> $1
LIMIT 10;

For a sample of real query vectors, recall@10 = |exact ∩ approximate| / 10 — the fraction of the true nearest neighbors the index returns. If that number dips below roughly 95% at your target ef_search, raise it (at some latency cost) until recall and p95 both satisfy your SLO.

Finally, verify the plan is actually using the index:

EXPLAIN (ANALYZE, BUFFERS)
SELECT id FROM documents
ORDER BY embedding <=> $1
LIMIT 10;

You want to see an Index Scan using documents_embedding_idx — not a Sort over a sequential scan.

The Houston Takeaway

The ecosystem is moving fast — this week alone, Tiger Data launched a PostgreSQL extension aimed at AI agents, and managed Postgres vendors keep shipping vector features into every major cloud. That is good news: pgvector is a credible, low-operational-overhead home for Houston’s RAG workloads, keeping embeddings next to the relational data they describe. Just don’t pick it — or reject it — on someone else’s benchmark. Build the honest test, measure recall and p99 under load, and let your own data decide.