What Is Jev? Trying a Decision Model Inside Mantle Chat

You finish a conversation about a leaking kitchen sink. Your chat app has folders called Travel Planning, Recipes and Cooking, Home Maintenance, and Garden and Plants. It wants to suggest where this conversation belongs.

The useful answer is small: Home Maintenance.

That is the kind of job we wanted to explore with Jev, a model from TypeSafe designed to make small, structured decisions. We built a folder-suggestion integration for Mantle Chat, then compared the folder-choice step with DeepSeek V4.1 Flash and GPT-5 nano. The integration is being prepared for release; these results come from synthetic API tests.

Implementation update: We have since simplified the integration to one question: choose an existing folder or say that none fits. Mantle uses the highest-scoring option without a confidence cutoff. If none fits, DeepSeek can suggest a new folder name; it also handles invalid answers or a failed Jev request.

The diagram and comparison below describe the three-question setup we measured. We have not measured speed, cost, or accuracy for the new setup, so the reported 115/115 result does not apply to it.

Before getting to the numbers, it helps to understand what Jev actually does.

A model whose answers are meant for code

You give Jev some information and a question with a defined answer format. It returns a choice or a number that your application can use. TypeSafe calls this a System One model: the name refers to fast, focused judgments. Jev currently works with text, including text arranged as JSON. TypeSafe's overview

For the leaking-sink example, you provide the conversation preview and the available folder names. You ask which folder fits. Jev returns a candidate and probabilities for the available options. Your code can then decide whether the evidence is strong enough to show a suggestion.

A general-purpose language model can also return structured answers. We already used one for this feature. Jev makes this particular kind of interaction its main job: evaluating information against choices or criteria supplied by the developer.

Its API has three question types:

Choice returns an option, probabilities, and confidence. Noul returns a number from 0 to 1. Score returns a rating, probabilities over its levels, and confidence. The levels need descriptions so the model knows what they mean. Choice, Noul, Score

How a Jev request works

We send Jev the chat title, a short preview, and the folder names. Then we ask which folder fits. We also give it a "none" option for chats that belong somewhere else.

Jev returns its choice and a number for each option showing how likely it thinks that option is right. Our code checks the answer before showing a suggestion. If the answer is too uncertain, we leave the suggestion empty. Choice questions

We can ask several questions in the same call. Each one looks at the same information. Our code then uses the answers together. We used Jev 1.13 throughout this test. Asking several questions

TypeSafe says it trains Jev to make its probabilities useful. For example, if a model gives 100 answers an 80% probability of being right, roughly 80 should be right. That is the goal; we did not test whether Jev achieves it across different tasks. How TypeSafe describes the training

Choice also includes a confidence number. It tells us how clearly one option stands out from the others. It comes from the same probabilities, so it is not extra proof that the choice is correct. Confidence explained

Where could this be useful?

A few examples:

• Sending a support message to the right team. A customer writes, "I was charged twice." Jev could choose Billing from a list of teams. It could also check whether the customer is asking for a refund. The app's own rules would still decide whether to issue one.
• Sorting search results. A search finds ten pages. Jev could read the supplied text and rate how well each page answers the question. The app could put the best matches first.
• Checking a claim against a source. Give Jev a sentence and a source passage, then ask whether the passage supports the sentence. This checks the text you supply; it does not establish that the source itself is true.

These are possible uses described in TypeSafe's documentation. Our test covers folder choice. More examples

Jev needs enough information to answer a small, clear question. For writing a reply or coming up with a new folder name, we use a model that generates text. We leave calculations and access checks to code. Jev can also misunderstand wording or instructions hidden in the text it reads, so examples like those belong in the tests. Known limitations

The Mantle Chat setup we measured

Mantle Chat can suggest a folder for a conversation. We previously used GPT-5 nano for this, then switched to DeepSeek. The model could choose an existing folder or suggest a name for a new one.

With Jev, we check the existing folders first. DeepSeek still handles new names when needed.

We send Jev up to 200 characters from the title, 500 from the chat, and the folder names. We ask three questions in one API call:

• Which folder fits best, or do none fit?
• Is there enough information to tell what the chat is about?
• Do the folder names tell us enough about what belongs in them?

The last question matters when a folder is called something like "Stuff." Its name tells us little about what belongs there.

Our code checks the chosen folder's probability, its confidence, and whether the chat's topic is clear. It also checks that the chosen folder is one of the user's folders. To accept that no folder fits, it requires clear folder names too.

If the checks do not pass, Mantle leaves the suggestion empty. We call this abstaining. If Jev says no folder fits and the checks pass, DeepSeek can suggest a new name. DeepSeek also handles the request if Jev is unavailable.

Measuring the folder choice

We gave all three models the same job:

Choose an existing folder, say that none fits, or leave the suggestion empty when unsure.

The test stops there. It does not include creating folders, coming up with names, or calling another model for help.

GPT-5 nano and DeepSeek returned a single choice. Jev answered the three questions above, and our code applied the checks. All three saw the same chat text and folder names. Their instructions and checks differ, so the results describe these setups, including the rules around each model.

We used 23 made-up chats and tested each five times with each model. That gives 115 attempts per model from 23 different chats. They cover work and personal topics, unclear messages, similar folder names, different languages, and text that tries to tell the model what to choose.

We agreed on acceptable answers before this run. Some chats can belong in more than one place. For an interview about contract language, Customer Research, Contracts and Compliance, or leaving the suggestion empty all count as valid. The same rule applies to every model.

These were existing test cases known to our team. We chose the expected answers and reviewed the ambiguous contract example, but the labels have not had an independent review. We kept the inputs, accepted answers, and instructions fixed during the run. We excluded chats with no folders from every model's results because there is no folder to choose.

What we found

Our accuracy measure is agreement with the answers we accepted for each chat, including valid empty suggestions.

Jev's agreement score was recalculated from saved responses using a threshold chosen after reviewing the results. Timing and cost come from the original API calls. The revised threshold has not been tested on new chats.

The median is the middle observation. The p95 describes the slower end of this small sample: about 95% of attempts finished within that time. These are local API timings, including request preparation and validation. They do not include the rest of the application or measure how quickly the UI appears to a user.

Compared with DeepSeek V4.1 Flash, Jev's median latency was about 69% lower, and its estimated cost was about 15% lower.

Compared with GPT-5 nano, Jev's median latency was about 68% lower, and its estimated cost was about 18% higher.

On these cases, Jev matched 100% of accepted answers, versus 91.3% for DeepSeek and 57.4% for GPT-5 nano. That is an accuracy improvement of 8.7 percentage points over DeepSeek and 42.6 points over GPT-5 nano.

Costs come from reported tokens, scaled to 10,000 attempts of this exact workload. We used Saturday off-peak DeepSeek prices and recorded cache hits. Repricing the same DeepSeek tokens at peak rates gives about $0.72; with no cache discount at off-peak rates, about $0.55. Those are calculations, not extra test runs. These figures are estimates, not provider invoices or measured production savings. OpenAI prices, DeepSeek prices, TypeSafe prices

Folder organization can depend on personal preference. We accepted more than one answer for the contract-interview example, including leaving the suggestion empty. A match with our accepted answers describes this small test; it does not establish perfect accuracy on other chats.

What we can take from this

The test shows how this Jev setup behaves on a small folder-choice task, including cases where leaving the suggestion empty is acceptable. It also makes the division of work visible: Jev supplies judgments, and Mantle decides when to show them.

The next useful evidence would come from people using the feature. Do they accept the suggested folder? Do they move the chat afterward? Would they prefer a tentative suggestion to an empty one?

Our test uses made-up chats and expected answers chosen by us. Five repeats of a chat show some variation, but they are still the same chat. We have not measured user satisfaction, production savings, or behavior under heavy traffic. A high score on this small set cannot establish a zero-error rate.

Download the test inputs, answers, and scoring notes.

Try Mantle Chat

Keep your AI conversations and team discussions in one place. Start free at mantle.chat, or tell us what you would like to see next.

The Mantle Chat Team