I run an AI product called WebSeek.ai that lets the user pick between Claude, GPT and Gemini for the same job. That wasn't a benchmarking exercise — it was a product decision that turned into the most useful thing I've learned about building with models, because it forced me to handle all three in production rather than reading comparisons of them.
Here's what that actually taught me. Note what this post deliberately doesn't contain: benchmark tables and version numbers. Both are stale within weeks, and neither predicted anything useful about my own workload.
The differences that actually mattered
Prompts are not portable
This was the biggest surprise. A prompt tuned carefully for one provider does not simply transfer. Same instruction, same input, materially different output — one follows an edge-case rule closely, another generalises it, a third quietly reformats. Nothing is broken; they were shaped differently.
The practical consequence is that "we'll just swap the model later" is more expensive than it sounds. Swapping the API call is trivial. Re-validating every prompt against your real cases is the actual work.
Failure modes differ more than quality does
On a task any of them can do, the ceiling is close enough that arguing about it is mostly wasted time. What differs is how each fails when pushed past what it can do — one hedges, one confidently invents, one over-formats. For production, knowing the failure mode is worth more than knowing the benchmark score, because the failure mode is what your users will actually meet.
Latency and cost shape the product more than capability
Beyond a certain point the interesting question isn't "which is smartest" but "which is fast enough that the feature feels alive, at a per-call cost that survives success." A feature that's brilliant and takes eleven seconds is a feature people stop using. This is where the smaller, faster tiers earn their place, and where a lot of teams over-buy.
How I choose now
Per feature, not per project — and in this order:
- Does the cheapest tier that could plausibly work, work? Test that first. On a surprising share of real tasks — classification, extraction, routing, short rewrites — it does, at a fraction of the cost and latency. Teams routinely reach for the largest model out of caution and pay for it forever.
- What does failure cost here? A wrong product blurb is an edit. A wrong refund policy is a support ticket and a broken promise. High-stakes paths get the more capable tier and a retrieval layer, not a bigger prompt.
- Is latency in the user's way? If the person is waiting, speed is a feature. If it's a background job, it isn't.
- Only then, capability. Genuinely hard reasoning, long documents, multi-step work — that's the frontier tier, and it's worth it there.
Build so the answer can change
The single most useful architectural decision: don't let a provider's SDK spread through your codebase. Put model calls behind one interface of your own. Keep prompts in version control as data, not buried in application logic. Keep a small evaluation set — twenty real inputs with expected outputs is enough to start — so a switch becomes a measurement rather than a debate.
Do that and changing providers is an afternoon. Skip it and you've locked yourself into a vendor as thoroughly as any SaaS platform, which is the thing custom code was supposed to avoid. It's the same argument as owning your stack, applied one layer up.
What this means if you're commissioning the work
Before any of this matters, though, check that the feature is one worth building — most proposed AI features aren't, and the model question is the last one to ask, not the first.
If a developer tells you a specific model is the answer for your product, ask what they tested it against. The right answer is almost always "it depends on the feature, and here's how we'll find out cheaply" — because the model that's best today is a fact with a short shelf life, and the architecture that lets you change your mind is the part that lasts.
I'm Divyansh Sood. I build AI features and products — and I run all three of these providers in production in my own. If you're deciding what to build on, email me the use case and I'll tell you what I'd actually pick for it, and why.