Digitalization of VERA or how we aced the test

21 June 2023

Today we would like to talk about one of our most recent successes. We were tasked to do an upgrade and enhancement that involved Django, Vue.js, beautiful plots and some fascinating mathematics.

The project consisted of three main steps. First we had to beef up security, bring it up to modern standards and pass a penetration test. Second, building upon the first step, increase automated test coverage and migrate from a single user per school to a multi user per school system, where each teacher gets their own individual account. Finally, beautiful reports!

But, in order to better explain the project we first have to introduce you to VERA.

VERA (VERgleichsArbeiten in der Schule) is a joint project between all 16 German federal states. It is aimed at conducting and analyzing annual exams in schools with the ultimate goal to improve education of the country's school children.

For a single test run, the VERA process loosely goes as follows:

the test is generated by a central research institute by producing tasks which are grouped in various test booklets
specific test booklets are distributed to the respective schools
pupils participate in the test
answer sheets are collected and evaluated
schools receive information about their performance

Each German federal state is responsible for the process execution for all of the schools in their state. We were commissioned by the state of Bavaria to enhance the web application used to collect and evaluate test results in their state. Our job was to make the gathering, monitoring and analysing of exam data easier.

We didn't have to start from scratch though. There was an existing solution built on Django already in place, which we found absolutely delightful.

The Backend

Our first task in this project was to upgrade the runtime from Python 2 to Python 3, as well as all the Python dependencies. Because Python 3 is not backward compatible with Python 2 this involved adapting the existing code to work with the new runtime.

We then had to upgrade Django and while doing this we took the liberty to automate some of the manual management tasks, streamlining the overall workflow for the admin team. While on it, we improved other aspects of the Django application. For instance, we reduced the size of the vera results data 12 fold, added unit tests for major functionality using pytest, and introduced support for multiple users per school, enhancing the overall security of the application.

The Frontend

With the backend revitalized, we turned our attention to the frontend and reimplemented the data input functionality for the teachers as a modern Vue.js-based single page application.

Our main foci for the frontend application were usability, reliability and consistency. The motto we worked with was “No data gets lost” and this guided all design decisions. Every single answer given by every single pupil had to be accounted for. This had to be the case even if the responsible teacher took a break before hitting the save button and came back to find their computer fried. They should be able to pick up where they left off on another machine.

We also put special attention to data validation and in the case of missing or incorrect data the teachers are notified and guided with instant hints without loss of what was already entered.

Data Analysis

Once all data has been input into the system, an admin can stop the input functionality for the test and initiate the test assessment calculation. The calculation contains two parts:

Rasch model-based statistical analysis to determine competence levels of each pupil
classical statistical analysis (for example correct answer rates calculation) on different levels - class, school, all schools of the same school type

Rasch-model based statistical analysis is performed by an already existing external system. The input data for that calculation is generated by the Django application on demand and the results are fed back into the Django application. These results along with the results from the classical statistical analysis, form the base for the online VERA assessment reports.

Reporting

We used Plotly in the Vue.js application to generate the reports. The Django backend performs all the needed calculations and provides ready to use data to the frontend via an API.

As mentioned above, classical statistical analysis is performed on different levels - class, school, and school type. Let’s consider two reports for classes in two different schools of the same type. They would contain the same result on the school type level. So the calculation could be performed once, saved and the result could be looked up for all of the respective reports. Following this approach and caching some of the reusable values we not only optimized resource usage, but also sped up the views such that they appear almost instant to the user.

The new online reports are superior to the legacy PDF when it comes to both speed of updating and responsiveness. The responsive design removes the need to print them out - the first step into the sustainable paperless future.

But speed and appearance are not the only benefit of the new reporting structure. VERA results are sensitive information and we introduced robust permission management to secure them. And last but not least is the ability to access online reports for historical data in one place by a single click of a mouse.

Summary

The current project is about to be wrapped up and looking back, we can't help but appreciate the great foundation Django provides for web applications. It’s been powering the system for years and it will certainly continue to do so. With the fast modern frontend and user friendly input and report functionality we hope teachers would enjoy using the system and embrace the paperless option. And for the admins, they would appreciate automation in place of manual tasks, saving their time, freeing up mental capacity and preventing human error.

We are grateful to the Bavarian State Office of Education for entrusting us with such an important project. In particular we want to thank the entire team from “Referat Vergleichsarbeiten”, working with them was a very efficient and pleasant experience which we would happily repeat at any time.

Note: No real data is used in the provided screenshots.