We built the tool we couldn't find.

PeerLM was born out of frustration. We were building AI-powered products and needed to know which LLM could deliver the most realistic, high-quality output for our specific use cases. The existing approach — flipping between chat interfaces, running ad-hoc prompts, and making gut-feel decisions — wasn't cutting it.

So we built a testing ground that ran real prompts across multiple models, anonymized and shuffled the responses, and asked evaluators to rank them blind. Production Monitors later added pairwise position swaps and cross-provider judge pools that exclude the vendors under test. Those controls make comparisons more resistant to name, position, and vendor bias.

The results were eye-opening. Models we assumed were the best often weren't. Cheaper models frequently outperformed premium ones for specific tasks. And the evidence was auditable — something we could present to leadership and defend.

We realized every team building with LLMs was facing the same problem. That internal tool became PeerLM: a production monitoring and blind evaluation platform that turns model selection from guesswork into reviewable evidence.

Today, PeerLM supports 200+ models across 15+ providers, reads production traffic from six tracing tools, and gates every prompt and model change on evidence from your own traffic.