How We Pick the Right Model
for Every Job in Audria

Benchmarks Answer Someone Else's Question

At Audria, we try not to assume. We work from first principles. Before a number goes into a decision, we want to know where it came from, what it measures, and what it means for our workloads.

A leaderboard can't tell you how a model handles a long transcript, how reliably it calls your tools, or how it holds up under the specific mix of requests your product throws at it. Those are different questions, and the gap between them is where most model choices quietly go wrong. This is a write-up of how we worked through that gap. It took four steps, and each one existed because the step before it failed in a specific, useful way: try something, find where it breaks, work out why, and let that define what comes next. The failures are as much the point as the results.

A large share of what Audria does runs through a model: analyzing recordings, answering in chat, handling voice, clustering topics overnight, drafting suggestions. Some of these are simple, single-shot prompts. Others are agentic workflows that chain several tool calls to get something done. They don't ask the same things of a model. Voice has to be fast. Nightly clustering can take its time. Some prompts are simple; others need a frontier model.

So the question was never "which model is best." It was narrower: for each workflow, which model is smart enough, fast enough, and efficient enough at once. That means taking a few unglamorous properties seriously. These are only some of what we track, but they set the shape.

• Speed has two parts: how fast the first token appears, and how long the whole answer takes. What counts as "fast enough" is completely different for a live voice reply than for a background job no one is waiting on.
• Throughput matters too. An answer that starts quickly but then trickles out word by word still feels broken.
• Reliability and intelligence sit under all of it. A model that's excellent but down even a small fraction of the time quietly fails that same fraction of requests. For an always-on assistant, that isn't something you can wave away, so how stable a provider is counts as a real signal.
• Prompt caching, which we care about more than most people expect. Much of what we send repeats from call to call. When a provider caches those tokens instead of reprocessing them, the request gets cheaper and faster at the same time.

Most of what follows is really about measuring properties like these honestly, and weighing them against how capable the model actually is.

Start With Existing Benchmarks, Carefully

The obvious first move is to reuse the benchmarks that already exist. So we did, but carefully, not by trusting a single ranking.

We combined two kinds of data. From Artificial Analysis we took benchmark scores, the intelligence side. From OpenRouter we took live operational stats, the reliability and cost side. Then we filtered: current, in-use text models only, nothing dated or retired. Each had to clear a tool-call success rate of 85% (tool use is central to how Audria works) and sit under a firm price ceiling. That gave us a working set of 113 models: clean, current, and worth evaluating properly.

Turning 113 models into per-workflow choices took two scores per model, because different workflows value different things.

The Intelligence Side: Scored Per Task

We don't use a single "intelligence score." A model's usefulness depends on the task you point it at. One that's strong at long-context reasoning but weak at tool use is a good pick for one workflow and a bad pick for another. So we weigh the underlying benchmarks by what each workflow actually relies on. The same model earns a different number depending on the job. That's why we call it a Task Intelligence score.

The benchmarks we blend, and what each tells us:

• AA-Omniscience: knowledge coverage, and how often a model is confidently wrong. A confident wrong answer is worse than a slow one, so this carries weight.
• AA-LCR (Long-Context Reasoning): reasoning over long inputs. Matters wherever a workflow works across long transcripts and accumulated context.
• AA-Briefcase: structured reasoning on realistic professional tasks.
• GDPval: performance on real, economically valuable work, graded against expert output.
• τ-bench: multi-step tool use in a realistic setting; our best public proxy for driving tools correctly across turns.

Each benchmark is converted to a common 0 to 10 scale first, since they're natively on very different ones. Omniscience is an index, Briefcase and GDPval are rating-style, the rest are percentages. When a benchmark is missing for a model, we rescale the weights over the ones present rather than invent a value.

For example, one specific workflow had to reason over a lot of accumulated context while staying broadly knowledgeable and grounded. So the weights leaned that way:

To see it end to end, take a model most people will recognize: OpenAI's GPT-5.4 Mini. Its raw benchmarks live on wildly different scales, and one is even negative. We normalize each against the whole field, then apply the weights.

That negative Omniscience index isn't a low score in absolute terms; it's simply mid-range once you see where every other model falls. Apply the weights (0.30 × 9.4 + 0.25 × 5.6 + 0.25 × 6.5 + 0.20 × 5.6) and GPT-5.4 Mini lands at about 7.0 out of 10 for this workflow. Run the same procedure across all 113 models and you get the full Task Intelligence column, ranked for this specific job. For a different workflow the priorities shift, tool reliability and not hallucinating matter more than long-document reasoning, so the same benchmarks get different weights and the ranking reshuffles.

The Logistic Side: Viable in Production?

Intelligence is only half the picture. The logistic score captures whether a model is operationally good enough to ship, and we built it the same way. It's drawn entirely from OpenRouter's live stats: throughput, end-to-end latency, tool-call error rate, structured-output error rate, cache-hit rate, and uptime. Each is normalized to 0 to 10, then weighted per workflow to match what that workflow cares about, exactly as we did for intelligence. A tool-heavy workflow leans its weights toward error rates; a latency-critical one leans toward speed.

The Two Plots

With both scores in hand for every model, we built two interactive plots, each answering a different question. Both are live below, scored for one of our workflows: hover any point, search to highlight a model you know, and drag the quality threshold. Every dot is one of the models that survived our screen.

Plot 1, Intelligence vs Logistic. Every model is a point: vertical axis is Task Intelligence (/10), horizontal axis is the logistic score (/10). Dashed lines mark the medians, splitting the field into quadrants; top-right is strong on both. Point color encodes blended cost. Top-right and cheaply-colored is what you want. This shows the raw trade-off between being capable and being operationally solid, and makes it obvious which models are strong on one axis but quietly weak on the other.

Plot 2, Balanced quality vs cost. The vertical axis collapses both scores into one balanced-quality number: the minimum of the two percentile ranks. The horizontal axis is blended cost. Up and to the left is best value; the line is the value frontier. The "Pareto picks only" toggle isolates the models nothing else beats on both quality and price. Prefer everything in one place? Open the full explorer in its own tab.

Underneath both plots sits a three-way Pareto filter across intelligence, cost, and operational quality at once. A model survives only if nothing else beats it on all three: smarter and cheaper and more reliable. Everything else is strictly dominated and dropped. What the filter keeps are the genuine trade-offs: a little pricier but noticeably smarter, or a little weaker but much cheaper. Those are the choices worth debating per workflow. Filtering three ways, instead of one, is what stops a cheap-but-weak or smart-but-fragile model from surviving on a single flattering axis.

What This Gave Us, and Where It Strained

A real result, not a warm-up. From 113 current models, the scoring and the Pareto filter produced a data-grounded first cut of 10 to 15 models per prompt, each ranked on task-appropriate intelligence, weighed against real operational stats, and placed against the model we already ran. We could argue the trade-offs with numbers instead of impressions.

The problem showed up when we pressed on what those benchmark numbers actually were. Every benchmark score is an average over someone else's tasks. It can't tell you the input-to-reasoning-to-output ratio you'll see on your own prompts, whether a model caches the way your workload needs, how its newer versions behave, or how it reasons through your specific requests. The shortlist was solid on paper. Paper was the problem. We were missing real behavior on real Audria work, and that defined step two.

Try to Borrow Live Usage Data (It Broke Twice)

We needed real behavior, not benchmark averages. OpenRouter publishes recent usage stats, so the plan was to pull input:reasoning:output ratios and caching behavior from them, and to check successor models, assuming newer means better.

It broke twice. First, the successors. For many, benchmark scores were lower than the models they replaced. One case stuck with us:

Second, and more fundamental: the OpenRouter stats are weekly averages, shaped by how the crowd happens to use each model. They're volume-weighted across whoever calls a model that week, for whatever purpose. So a model used mostly for short, easy tasks can post flattering numbers, while a genuinely stronger model, reached for on hard, high-volume work, looks worse precisely because it's doing harder jobs for more people. The metric partly rewards easy usage, not model quality. Two models on the same page aren't being measured on the same thing.

No usable data, but two firm conclusions. Aggregate, crowd-sourced usage tells you little about a specific workload. And you can't reconstruct a model's behavior on your tasks from other people's traffic. If you want your numbers, you have to generate them. We dropped the approach, and it pointed straight at the next step.

Measure on a Small Slice of Real Work

We ran our own prompts, starting small on purpose. We pulled a small dataset per prompt from production, easy-to-moderate cases representative of routine load, and ran the shortlist on it. We added a few frontier models to see the ceiling next to the cheaper options. Every result was graded by Claude Sonnet 5 as the judge, so all models were scored the same way on the same cases.

With a consistent judge on real cases, the shortlist dropped to 7 models across all prompts combined. And running real cases forced two distinctions we'd been treating too simply.

The first was prompt caching. We had to separate two kinds that behave differently and suit different workloads. Since a large, predictable share of every Audria request is stable, which one fits best has real cost and latency consequences, and the answer isn't the same across workflows.

The second was where the model is served. OpenRouter is an aggregator: one key, many models, automatic fallback if a provider goes down. Strong on reach and resilience. But it sits between us and the provider, and we grew less willing to trust caching quality and consistency through that middle layer than straight from a first-party provider. Going direct can mean more predictable caching and cleaner cost. Rather than settle it from theory, we set up the next step to test it.

The limit of this step was known going in: easy-to-moderate cases only. A model that shines on simple work can still fail on hard cases. Seven models that pass an easy test aren't seven models we'd commit the product to. The next step needed data that looked like everything Audria sees.

Test on the Full Distribution

This step had to decide, which meant an apples-to-apples test on the real distribution of Audria's work. We built a 370-conversation dataset from real production recordings of our team.

A clarification on "hard": we don't mean difficult or exotic data. We mean the full mixture. Easy, moderate, and genuinely tricky cases, in the proportions they actually occur in production. That's what makes it the best test we have. A model can't score well by being good at only one end of the range; it has to handle the real spread.

Alongside the seven, we added a set of high-end frontier models for premium users, held to a stricter tool-call reliability bar. Every model ran all 370 scenes, judged again by Claude Sonnet 5, scored as a profile (correctness, tool use, reasoning quality, handling of ambiguity, and efficiency), not a single number.

Evaluating How Models Are Run, Not Just Which

Here we stopped evaluating models in isolation and started evaluating how they're run. Execution tier was the first lever. The same model can be served on different tiers, and they trade cost against speed. Standard is the default. Batch and flex are cheaper but slower or deferred, which is fine for work no one is waiting on and wrong for anything live. So the same model carries different economics depending on the lane you put each workflow in.

Provider was the second. Running direct-versus-aggregator side by side, the experiment we'd promised ourselves in step three, showed per workflow where the aggregator's breadth and failover were worth it, and where going straight to a provider won on caching quality, consistency, and control.

We ran the final analysis across the 370-scene scores, the efficiency profiles, and the execution and provider choices, and arrived at a short final list: the model best suited to each workflow, given how it behaves on representative data, not the highest-scoring model overall. And alongside each model, the way we run it: which caching, which provider, which service tier.

The other result wasn't a model. It was the setup itself. New models ship constantly, so the list is never final; there will always be additions to test and more experiments to run. What this process left us with is a structure we can rerun whenever that happens. We know which parameters actually matter, how to weight them for a given workflow, and how to fold a new model into our agentic workflows as and when it's needed.

What the Four Steps Established

• Benchmarks are a starting point, not a decision. They narrowed 113 models to a shortlist. They never picked the winner.
• Newer isn't automatically better. Several successor models scored below their predecessors and threw errors a frontier model shouldn't. Upgrading by default carries real risk.
• You can't borrow your numbers. Public usage stats are weekly, crowd-shaped averages that partly reflect task difficulty. The behavior that decides quality and cost only shows up when you run your own prompts on your own data.
• Fit matters more than tier. On many tasks the gap between frontier and non-frontier models was small, and on some a leaner non-frontier model did better. Paying frontier prices everywhere adds cost without a matching gain.
• More reasoning isn't always good. Many tasks don't need reasoning. Forcing it adds latency, can cause overthinking, and often doesn't improve the answer. Match reasoning depth to the task.
• Ratios and prompt caching drive behavior more than a headline benchmark score does, and picking the right kind of caching is part of the work.
• How you run a model matters as much as which one you pick. Standard, batch, and flex tiers, plus direct-versus-aggregator, change cost and reliability without changing the model at all.

Where This Leaves Us

The aim was never to crown the single strongest model. It was to give each workflow the model that fits it: responsive enough, reliable enough, efficient enough for that specific job, based on how models behave on representative data, not on external rankings.

We worked through it in four steps because each failure isolated what the next one had to test. Benchmarks gave a shortlist but not real behavior. Borrowed usage data didn't transfer. Easy cases didn't cover the hard end. The full distribution finally allowed a fair comparison.

The method is the part that lasts. The next time the model landscape shifts, and it will, the same process runs again.