Jev essay grading: Flag a wrong short answer
This Jev essay grading prompt says whether a student answer is wrong or misses the point, compared with the answer key.
{
"state": {
"question": "{{question}}",
"reference_answer": "{{reference_answer}}",
"student_answer": "{{student_answer}}"
},
"questions": {
"wrong": {
"type": "noul",
"instructions": "Is `student_answer` wrong, judged against `reference_answer` as an answer to `question`? Wrong means it gets the main point wrong, misses all the key points, or answers something else. An answer that is incomplete but partly right is not wrong."
}
}
}Fill in `question`, `reference_answer` (your answer key or marking notes) and `student_answer`. A yes means the answer is wrong or misses the point; an answer that is incomplete but partly right should come back no. On tutor-graded university networking answers Jev agreed 34 times out of 40. Use it to put the likely wrong answers at the top of a teacher's pile; the marks stay with the teacher. Telling full marks from partial credit did not work well enough for us to include: a three-way grade we tested matched the tutors only 51 times out of 90 across two versions.
85% agreement (34 of 40): yes means incorrect, partial counts correct, Jev essay grading at 0.5
Dataset: Short Answer Feedback (SAF), communication networks, EnglishLicense: CC BY 4.0