Jev’s Vision Test: 30 Images, All Mine

🔍 Read the full analysis: Jev’s Vision Test: 30 Images, All Mine on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on office and shipping supplies

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

TypeSafe released Jev 1.14 on October 14, adding image input and a new bounding-box answer type to its decision model, while keeping both capabilities behind explicit opt-in headers because the company says they are not ready for production use. In a report published the next day, Thorsten Meyer tested Jev 1.14 and 1.13 on 30 featured images from his website; he reported strong results for product photos, but the available source ends before giving findings for news and essay images.

Jev is designed to answer structured questions rather than write prose: it can return a yes-or-no probability, choose from a list, or assign a score, with confidence information. Before this release, those questions were text-only. TypeSafe says image support can now be used with jev-1.13.0 as well as jev-1.14.0, and requires the header TypeSafe-Danger-Zone-Images: true to activate it. The new box response type, which returns one bounding box and a confidence score for a described target, is limited to Jev 1.14.

Meyer tested three yes-or-no questions against each image: whether it matched its own headline, a randomly selected other headline, and an unrelated headline about repotting a houseplant. The test covered 10 product roundups, 15 news posts and five personal essays. The 10 product images also received a multiple-choice question asking which product each showed. Meyer says he ran both model versions, using about 60 requests in total.

For the product images, Meyer reports that Jev 1.14 gave each image’s own headline a score of 0.65 to 0.89, with a median of 0.76. Randomly paired headlines scored no higher than 0.18, with a median of 0.06, while the unrelated plant headline stayed at or below 0.10. Jev 1.14 identified nine of the 10 products correctly. The miss was a photo showing a mesh Wi-Fi unit beside a router with antennas, which the model labeled a Wi-Fi 7 router with confidence 0.60. Meyer says every correct product identification had confidence of at least 0.77. Jev 1.13 identified all 10 products, though two answers had confidence below 0.5.

A Test of Image-Based Decisions

The change extends Jev from judging text to evaluating visual inputs inside the same structured-question format. That could let developers test image classification, check whether a picture fits a description, or locate a specified object without asking the model to produce open-ended prose. The bounding-box output adds a way to return a location as well as a decision.

Meyer’s test offers an early, small-scale example rather than a general performance assessment. The results he reports suggest the model could distinguish relevant product imagery from unrelated headlines in this particular set. But the one product-identification error, the differing confidence patterns across versions, and the incomplete source findings for the other image categories limit what can be concluded. TypeSafe’s own warning that the features are still jagged makes the distinction between experimentation and production use especially relevant for developers.

Opt-In Features in Jev 1.14

TypeSafe describes Jev as a model for structured decisions, with answers constrained to defined formats and questions. Meyer wrote in September about the model as part of what he called “System One” AI and later outlined 24 possible uses. The October release is the first change he describes that lets those structured questions refer to images.

Images are supplied through an assets map: a reference string in the request is replaced with the corresponding image wherever that reference appears. TypeSafe’s documented limits in Meyer’s report are PNG or JPEG still images, 256 to 1,568 pixels per side, up to 16 MiB per file and 32 MiB per request. The report says images are billed as ordinary input tokens at $0.042 per million input tokens, with output tokens free; base64 encoding increases file size by about one-third and counts toward the request limit. TypeSafe recommends multipart uploads, and the Python and JavaScript SDKs handle that format.

The release also adds an opt-in long-context setting that raises stated limits from 64,000 tokens per request and 32,000 per question to 262,000 for each, while TypeSafe advises against the larger context because reliability declines on long inputs. Jev 1.14 is available as jev-preview; TypeSafe said jev-latest would move to it about a week after release. The company also described the model’s improvements as broad, with multilingual performance expected to benefit most, and shared internal estimates rather than public benchmark scores.

“jev-1.13.0 was multimodal all along.”

— TypeSafe, describing the earlier model in its release post as reported by Meyer

Results Beyond Product Photos

The source material provided for this report cuts off immediately after the heading “News images: Jev was right, my images weren’t.” It does not include Meyer’s results for the 15 news images or five essays, so their scores and the conclusions behind that heading cannot be verified from the available text. The test is also limited to one publisher’s 30 images and does not establish how Jev will perform across other image types, users or tasks.

TypeSafe has not vouched for the features in production use. The supplied material does not provide an independent benchmark, a detailed evaluation of the bounding-box feature, or a full comparison of model versions across all image categories. It also does not specify the precise timing of the planned jev-latest update beyond roughly one week after the October 14 release.

Production Readiness and Rollout

Developers can try the image and bounding-box capabilities by sending the required opt-in headers, while TypeSafe’s warning means production suitability remains unconfirmed. The company advises users to evaluate the model themselves. Meyer’s report says TypeSafe’s internal comparisons use open-weight models and may be biased toward Jev, so they should not be treated as an independent performance measure.

TypeSafe said jev-latest was expected to move to Jev 1.14 around October 21, about a week after release. The company also gave deprecation notice periods of 60 days for Jev 1.13.0 and 30 days for Jev 1.14.0, while saying a formal policy was still to come. More complete test results, the formal deprecation policy and evidence from broader evaluations would help clarify how reliable the new image features are and where they can be used safely.

Key Questions

What changed in Jev 1.14?

TypeSafe added opt-in image input and a new box answer type that returns one bounding box and a confidence score. Image support also works with Jev 1.13, according to the company.

What did Meyer’s test find for product images?

Meyer reported that Jev 1.14 matched product images to their own headlines with scores from 0.65 to 0.89, and correctly identified nine of 10 products. Jev 1.13 identified all 10, though two answers had confidence below 0.5.

Are the new features ready for production?

TypeSafe said it could not yet vouch for the features in production use and placed them behind explicit opt-in headers. The company advises users to evaluate them for their own applications.

What image files does the reported API support?

Meyer’s report lists PNG and JPEG still images, with each side between 256 and 1,568 pixels. It gives limits of 16 MiB per file and 32 MiB per request; WebP and HEIC images need conversion first.

Are results for news images and essays available?

Not in the source material provided here. It lists 15 news posts and five essays in the test, but cuts off before giving results for those groups.

Source: ThorstenMeyerAI.com