I wanted to find the best model for my Librarian. So I sent the same research job to several capable models and compared them. Reasonable enough. The problem was that my test was so picky about the shape of the answer that a perfectly useful answer could still fail.

Then I did the dumb part: I kept trying more models. The error was basically the same every time, and the bill kept moving. I burned about $15 learning that model variety cannot fix a busted test.

When everybody fails the same way, check the test

If different models all hit the same formatting error, broken data pipe, permission problem, or impossible rule, that is a big clue. The system around the model may be broken. Paying for another opinion will not repair it.

Hitting retry with a different logo is still just hitting retry.

Make one cheap run work first

Now one small test has to work before I unleash a group of agents. I still check the hard stuff: required IDs, valid fields, source links, and permission limits. I do not fail a useful research job because one model used a different heading.

Once that first run passes, then it makes sense to compare models on the things I care about: did it find good sources, cover the subject, show its uncertainty, and produce something testable?

Two matching failures, then stop

If the same system failure happens twice, that lane stops spending money. Save the evidence, diagnose it once, and work on something else until the plumbing is fixed.

  • Data did not arrive? Fix the pipe. Do not pay a model to send the same answer again.
  • Parser rejected it? Test the parser with a saved example before buying another answer.
  • Website is unavailable? Mark it blocked and move to another source.
  • People can reasonably disagree? Mark it uncertain. Do not keep tightening the rules until nothing can pass.

Why I still use more than one model

Different models are still useful. One may find a source another missed or poke a hole in a strategy everybody else liked. But six AIs agreeing is not proof. The code, data, and backtest still have to hold up.

I did not learn which model was smartest. I learned to make the system notice when I built the wrong test.