Skip to content
All Jev use cases

Prompt pack

Jev as a judge prompts

Prompts that turn Jev into an answer judge: pick the better of two chatbot replies, score helpfulness, correctness and length, and check an answer against the facts or a source passage. Each prompt was run on real Jev as a judge against human ratings from public datasets, and the agreement rate is listed next to it.

Jev as a judge is for replies you already have. It can pick the better reply, score helpfulness from 0 to 4, check a factual answer, or say if a passage contains the answer. Use Jev as a judge when you rank replies or check a source. It does not write the reply.

The weak line in Jev as a judge is Pick the better answer, or call a tie: agreement with human labels was 69% on 45 examples. The plain pick, with no tie, was 93% on 40 examples. Agreement is the share of examples where Jev as a judge matched the human label. When Jev as a judge is unsure, send the pair to a person. Run Jev as a judge on your own pairs before you lock a threshold. This site is unofficial. TypeSafe does not run it. You call Jev with your own key.

How we tested these Jev as a judge prompts

Model
typesafe/jev-1.14-20261014
Tested on
Sent as
Decisions API request (state + questions) → typesafe/jev-1.14, served by TypeSafe

We ran these tests ourselves. TheJevAI is an unofficial, independent site and is not affiliated with TypeSafe.

Free Jev as a judge prompts

Copy a request, put your own data where the placeholders in double braces are, and send it. To send it to TypeSafe's API, add "model": "jev-1.14.0" to the request body.

Download the free prompts (JSON)

Jev as a judge: Pick the better of two answers

This Jev as a judge prompt compares two replies to the same request and says which one a careful human judge would prefer.

Request body
{
  "state": {
    "user_request": "{{user_request}}",
    "response_a": "{{response_a}}",
    "response_b": "{{response_b}}"
  },
  "questions": {
    "better_response": {
      "type": "choice",
      "instructions": "Which response would a careful expert prefer for `user_request`? First look for mistakes in each response: wrong facts, wrong answers, broken code, or instructions it ignored. Then compare how fully and directly each one does what was asked. Length and the order of the responses do not matter.",
      "criteria": {
        "response_a": "`response_a` handles the request better.",
        "response_b": "`response_b` handles the request better."
      }
    }
  }
}

Put the user's message in `user_request` and the two replies in `response_a` and `response_b`. `choice` names the winner and `probabilities` shows how close the call was; treat a winner under about 0.7 as a toss-up. Jev agreed with the human majority on 37 of 40 pairs. For comparison, the MT-Bench authors report that two human experts agreed on 81% of first-turn calls like these. Only 6 of the 40 pairs were math or coding questions, so for those, run the code or check the numbers as well.

93% agreement (37 of 40): pairs where Jev as a judge matched preferred replies, ties left out

Dataset: MT-Bench human judgments (LMSYS)License: CC BY 4.0

Jev as a judge: Score how helpful an answer is

Run this Jev as a judge prompt to rate a reply from 0 to 4 for how well it does what the user asked.

83% agreement (33 of 40): rounded helpfulness within 1 point, Jev as a judge, 0 to 4Dataset: HelpSteer2 (NVIDIA)

Jev as a judge: Check whether an answer is true

Use this Jev as a judge prompt to say whether an answer to a factual question is correct or repeats a falsehood.

93% agreement (37 of 40): true or false, Jev as a judge yes at 0.5 upDataset: TruthfulQA

Jev as a judge: Check whether a passage answers the question

Use Jev as a judge to say whether a source passage contains the answer, so a RAG app knows when to answer or say it does not know.

88% agreement (35 of 40): answerable question passage pairs, Jev as a judge yes from 0.5Dataset: SQuAD 2.0

Jev as a judge FAQ