Everything You Need to Know
How blind evaluation works, what Monitors deliver, what credits cost, and why teams switch from manual testing.
What is blind comparative ranking?
PeerLM anonymizes all model responses, shuffles the order, and asks evaluator models to rank them best-to-worst. The evaluator never knows which model produced which response — eliminating name bias and positional bias. Rankings map back to source models as normalized scores.
How is PeerLM different from testing models manually?
Manual testing means one person, one prompt, one afternoon — and a decision nobody can reproduce. PeerLM runs blind evaluations across dozens of scenarios, personas, and criteria simultaneously. You get statistically meaningful data instead of anecdotes, and audit-ready reports instead of Slack threads.
What models can I evaluate?
200+ models across OpenAI, Anthropic, Google, Meta, Mistral, and more — via OpenRouter and Groq integrations. Evaluate any combination against each other. Model capabilities sync automatically so you always have access to the latest releases.
How do eval credits work?
Credits use a multiplier system based on model tier: Standard and Advanced = 1x, Premium = 2x, Frontier = 3x. Both generator and evaluator models consume credits. Formula: SUM(all model multipliers) × system prompts × test prompts × samples. Cache hits consume 0 credits, so re-running the same evaluation costs nothing. Free plans get 200 signup credits with pay-as-you-go at $0.20/credit. Pro plans include 1,000 credits/month with overage at $0.10/credit. All PAYG and overage charges have a configurable monthly spending cap ($20 default for Free, $100 for Pro) — runs are blocked when the cap is reached, so you're always in control of costs.
What happens when I subscribe to a plan?
Your workspace is activated immediately. All evaluations and data are retained — nothing resets. Pro plans unlock 1,000 monthly credits, Premium models, API & MCP access, and custom evaluation criteria. Start running evals right away. Reach out to sales@peerlm.com if you have any questions.
What are Monitors?
A Monitor connects PeerLM to a production use case and accumulates comparison Runs against live traffic. Each Run samples real prompts, replays candidates on the same inputs, and judges quality retained vs your control alongside cost. The Monitor rolls those Runs into a living switch / route / hold verdict. One-off authored evaluations stay under Suites and Runs; Monitors are for continuous comparison from production traffic. Available on all plans; comparison Runs consume credits from your existing balance or pay-as-you-go.
What is a living verdict?
The living verdict is the Monitor's recommendation — switch, route, or hold — refreshed by every comparison Run. It surfaces quality retained vs control, projected savings, and Switch Confidence (SCS), a 0–100 composite of quality parity, latency, cost savings, sample power, and judge calibration. Results break down by prompt cluster so you can see where a candidate excels or struggles. Export routing recommendations as JSON, YAML, LiteLLM config, Portkey config, or a code snippet. Receipts live on the Run; the verdict lives on the Monitor.
What log sources does PeerLM connect to?
Langfuse, Helicone, Cloudflare AI Gateway, and any OpenTelemetry-compatible source — plus file upload and PeerLM's SDK (TypeScript, Python, Go, Ruby) for in-app logging with configurable sampling. Connect Sources on each Monitor, then start a comparison Run from that traffic.
What triggers an Auto-Run?
Three events can start another comparison Run automatically: a model release (when a provider ships a new version), a price change (when model pricing changes), and quality drift (when statistical divergence in your traffic exceeds a threshold). Triggers include a 24-hour debounce, configurable credit caps per run, and monthly run limits to keep costs predictable. Configure on the Monitor → Schedule.
What are system prompts?
System prompts (personas) let you test models in context — as a customer support agent, code reviewer, creative writer, etc. Each system prompt defines the role and instructions given to the model. PeerLM breaks down results by system prompt so you see which model wins for each use case.
What's the difference between JSON output and text output?
JSON mode returns structured arrays (e.g., '3 one-liner jokes') where each item is scored independently — giving granular data. Text mode evaluates free-form prose as a whole. JSON mode is recommended for comparative evaluations as it produces more reliable rankings.
Can I share results with my team?
Yes. All paid plans include shareable reports — a public link anyone can view without an account. Reports include rankings, per-persona breakdowns, and response samples. Export as CSV or JSON for further analysis.
How does response caching work?
PeerLM hashes the combination of model, persona, and topic content. Matching cached responses are reused at zero cost. Edit any part of the prompt and the cache auto-invalidates. Re-running evaluations is free — you only pay for new generations.
What is deterministic mode?
PeerLM sets temperature to 0 and uses fixed seeds where supported. It checks each model's capability registry first to prevent silent failures. Parameters used are logged for audit purposes — fully reproducible results.
Do you store my API keys?
Model calls use PeerLM-managed keys by default — no setup needed. For BYOK (Bring Your Own Key) on Enterprise plans, keys are encrypted with AES-256 at rest and never logged or exposed.
What is the PeerLM MCP server?
The PeerLM MCP (Model Context Protocol) server lets you run evaluations directly from Claude Desktop, Cursor, or any MCP-compatible client. Create prompts, configure suites, trigger runs, and check results — all without leaving your IDE. Available on Pro and Enterprise plans. Set it up with: npx -y @peerlm/mcp
Does PeerLM have an API?
Yes. The REST API lets you manage suites, create system prompts and test prompts, trigger evaluation runs, and retrieve results programmatically. Available on Pro and Enterprise plans. Generate an API key from Settings > API Keys.
Can I cancel anytime?
Yes. Month-to-month, no long-term commitment. Cancel or downgrade anytime from billing settings. You keep access through the end of your billing period.
Still have questions?
Contact support