Jev as a judge: Pick the better of two answers
This Jev as a judge prompt compares two replies to the same request and says which one a careful human judge would prefer.
{
"state": {
"user_request": "{{user_request}}",
"response_a": "{{response_a}}",
"response_b": "{{response_b}}"
},
"questions": {
"better_response": {
"type": "choice",
"instructions": "Which response would a careful expert prefer for `user_request`? First look for mistakes in each response: wrong facts, wrong answers, broken code, or instructions it ignored. Then compare how fully and directly each one does what was asked. Length and the order of the responses do not matter.",
"criteria": {
"response_a": "`response_a` handles the request better.",
"response_b": "`response_b` handles the request better."
}
}
}
}Put the user's message in `user_request` and the two replies in `response_a` and `response_b`. `choice` names the winner and `probabilities` shows how close the call was; treat a winner under about 0.7 as a toss-up. Jev agreed with the human majority on 37 of 40 pairs. For comparison, the MT-Bench authors report that two human experts agreed on 81% of first-turn calls like these. Only 6 of the 40 pairs were math or coding questions, so for those, run the code or check the numbers as well.
93% agreement (37 of 40): pairs where Jev as a judge matched preferred replies, ties left out
Dataset: MT-Bench human judgments (LMSYS)License: CC BY 4.0