A hands-on benchmark of the new hosted "system one" classification model (transcribed as "Jeff"/"Jev") against classic few-shot machine learning. On AG News, TREC, 20 Newsgroups and Clinc-150, it beats SetFit trained on eight examples per class by roughly 9 points and costs about a penny and a half per validation run, but open Qwen models wrapped in a "Decider"-style token-matching setup get competitive or better locally. The conclusion: don't think of it as a classification model — it's a classifier plus a little bit of quick reasoning, and its best use may be as a labeler that bootstraps or audits your own models.

• SetFit works by turning a handful of labeled examples into many pairs — agreeing pairs labeled 1, disagreeing pairs labeled 0 — then back-propagating a cosine-similarity loss into a shared embedding model so the embedding space itself separates the classes.
• With only eight training examples per class, SetFit using a near-decade-old cheap embedding model beat plain sentence-transformer and Qwen3-embedding pipelines with logistic regression.
• The hosted model still beat SetFit by about 9% on AG News and performed around the 90% mark on TREC, and it handled Clinc-150's 150 classes where SetFit effectively fails because the pair-generation blows up.
• Supplying written descriptions of the labels gives the hosted model a measurable accuracy boost.
• Even if you won't run it in production, you can point it at an existing dataset and ask for examples where it confidently disagrees with the label — in the speaker's experience it's often right, making it a good bad-label detector.
• The open-source "Decider" approach — run a Qwen model normally and read which output tokens best match the candidate labels — is competitive with the hosted model on standard classification tasks and runs locally.
• On BoolQ (500 questions) the hosted model hit about 93%, well above raw Qwen models; but using it to generate labels and then fine-tuning Qwen raised 0.8B from 73% to 87%, 2B from 67% to 88%, and 4B from 84% to 91%.
• The limits are real: you can't fine-tune a hosted model, you're sending data to a third party, the API went down during testing, and non-English slang can give mixed results you can't correct.

Eight examples per class, then the API wins anyway 6:56–8:00

This is the benchmark payoff: SetFit on eight examples per class beats every embedding-plus-logistic-regression baseline, and the hosted model still beats SetFit by about 9%.

The state of the art method from 5 to 7 years ago for few-shot learning, that is being outperformed by a mere API call that is costing me a penny and a half on this validation data set.

Use the API to label, then fine-tune your own model 13:19–14:07

Concrete numbers show distillation closing most of the gap — 73% to 87%, 67% to 88%, 84% to 91% — which is the practical escape hatch for anyone who can't send data to a third party.

Even if you have a data set where you have examples but no labels, you can actually point Jeff to the data set to generate those labels and then fine-tune your own system after the fact.

It's fast and the API is so cheap that it makes pennies look expensive.

Jeff can do classification, but the kind of model that it represents, it's really best not to think of it as a classification model.

You might be able to calibrate it in hindsight, but you're definitely not going to be able to fine-tune it because it's hosted somewhere else.