Key Info

OpenRouter tested Jev, a lightweight decision model by typesafeai, against popular LLMs on judging and classification tasks using Ori Eval. Jev was more than 5x faster than the next-fastest alternative and matched the accuracy of full-size LLMs, coming within a handful of cases across all five models.

Highlights

  • Jev is designed to break tasks into small yes/no and multiple-choice decisions, enabling fast, low-cost answers.
  • In OpenRouter's evaluation, it delivered a >5x speedup over the next-fastest LLM.
  • Accuracy was comparable to popular classification LLMs: all five models finished within a few cases of one another.
  • The result suggests decision models can be fast enough to run on every message or tool call without giving up correctness on this type of task.