Jev for Python engineers

Jev is a classifier, not a generative LLM—takes data + multiple-choice questions, returns answers + confidence scores

Structured JSON output, not free text generation

Vercel AI SDK for Python now includes experimental evaluate() API to access Jev directly

Install via 'uv add ai', requires AI_GATEWAY_API_KEY

Three question types: ChoiceQuestion (pick one), ScoreQuestion (rate on scale), NoulQuestion (probability estimate)

Author's tests: Jev classifies 'Python or English' input better than hand-trained classifier, but still misses edge cases like 'what's' + ' up'

Jev can't generate text natively; forcing it to choose letter-by-letter or word-by-word produces mostly incorrect output

AST-based approach (generating abstract syntax tree one node at a time) yielded syntactically valid but semantically incorrect Python code

Use case: narrow decision-making tasks where multiple-choice framing fits the problem

Jev model: classifier-as-LLM that answers multiple-choice questions and returns confidence scores

API returns structured JSON (ChoiceQuestion, ScoreQuestion, NoulQuestion) instead of free text

Vercel AI SDK for Python ships experimental evaluate() API to call Jev

Installation: uv add ai; requires AI_GATEWAY_API_KEY

Use case 1: Python vs English detection in a REPL—Jev outperformed hand-rolled classifier but still has gaps (e.g., misclassified 'what's + up')

Use case 2: AST-based code generation—Jev can generate syntactically valid Python by choosing tree nodes at each step, but output remains mostly incorrect

Author's conclusion: Jev excels at narrow classification, struggles when forced to behave like a text-generative LLM

Post is tutorial/engineering deep-dive from Vercel, not independent benchmarking