Bar-Ilan university pilots AI-driven code exam that checks student explanations

Two years ago researchers at Bar-Ilan University noticed that automatic grading in an operating-systems course returned perfect scores for almost every student, yet no one trusted that the submissions reflected genuine understanding. The solution they developed does not block AI tools; it simply verifies that the student can explain what the code does and why.

Prof. David Sarna, a faculty member in the computer-science and artificial-intelligence department, says the system was built in three weeks using Claude Code. The development team also includes Shahar Shagay, the teaching assistant who led the implementation, and Dr. Yossi Ben Tzion, a learning-technology specialist who oversaw design and deployment. About 260 students were examined in a pilot during the last semester.

During a 16-minute exam the student enters a dedicated room and sits at a locked workstation running in kiosk mode, which automatically retrieves the code the student submitted. The screen is split: the code appears on one side, while an AI examiner chat appears on the other. Typically three questions are asked, and each time the relevant lines in the code are highlighted. The questions are not prepared in advance; the instructor defines topics and the expected knowledge level, and the system generates dozens of templates that the instructor approves. A template becomes a personal question only when it matches a specific student’s code.

A quarter of the grade is based on the quality of the code itself, and three quarters on the depth of understanding and control the student demonstrates in the conversation. Three AI agents participate in the assessment: one evaluates the code without seeing the exam, a second reads the conversation without seeing the code, and a third aggregates the findings. At the end the student receives an email with the grade, a breakdown of answers and recommendations for further study. Students who fail may retake the exam with a new set of questions.

Students initially resisted. Dvir, one of the examinees, describes pressure in the first exam, but the joint preparation forced them to examine the code in depth. “In hindsight, the very preparation made us understand the material much better,” he says. The exam felt less like a test and more like explaining to a peer, and the system improved over the semester by fixing bugs and refining answer evaluation. Bar-Ilan now plans to expand the model to mathematics courses.