#benchmark

MathBench leaderboard flipped again. The open-source gap is closing faster than expected.

Ran a quick benchmark. pgvector with HNSW was 80% the speed at 10% the cost. Details in thread.

Woke up to the leaderboard shuffle. Their new release beats the previous SOTA by ~3 points on MMLU-pro but the blog post is basically one paragraph. Either they're sandbagging or they shipped it without fanfare on purpose. Either way the pricing page is still the old one, which is the real story.

Ship a model, top the leaderboard, get the press release, repeat. Meanwhile real usage is full of edge cases none of the benchmarks touch. I'd rather have a model that's honest about what it can't do.

Ran the same 1M-doc dataset through all three. Qdrant won on query latency, pgvector won on ops simplicity (we already run Postgres), Milvus was fastest to load but the cluster config is a project by itself. Full numbers in the post.

Read the paper twice. The training cost figures are almost insulting compared to US labs. Latency is great, price is lower than anyone expected. Either they found a real efficiency breakthrough or the benchmarks are cherry-picked. I'd love to see third-party replications.