Automated evaluation systems to enhance exam quality and reduce test anxiety

Exam paper quality

The importance of exam quality in education has been well-documented in the literature. Studies, including those summarized in Table 1, investigate key aspects of exam paper evaluation, from question design to the cognitive demands placed on students. Research shows that high-quality exams, characterized by balanced cognitive challenges and clear formulations, can significantly enhance learning outcomes (Nguyen & Habók, 2023). However, few studies explicitly connect exam quality with student well-being, particularly with respect to stress and anxiety.

This research highlights the need to expand existing studies by focusing not only on how exam quality affects learning outcomes but also on how it contributes to student stress. By integrating exam evaluation with psychological factors like test anxiety, we can develop a more holistic understanding of how assessment impacts student performance and mental health.

Previous research has established that test quality significantly influences students’ cognitive and emotional responses, drawing on cognitive load theory (Sweller, 1988) and test anxiety theory (Sarason, 1984). Cognitive load theory emphasizes the role of well-structured assessments in reducing unnecessary mental burden, enabling students to focus on problem-solving rather than deciphering poorly written questions. Test anxiety theory further highlights the interplay between unclear assessment criteria and heightened anxiety, underscoring the need for precise and transparent exam design to promote a fair testing environment. These theories provide a foundation for examining how automated systems can enhance exam quality and indirectly alleviate student anxiety by creating fairer and clearer assessments.

Academic stress and test anxiety

Academic stress, particularly test anxiety, is a prevalent issue in higher education, affecting a significant portion of the student population (Pascoe, Hetrick & Parker, 2020). Table 2 provides a summary of research exploring the various factors contributing to academic stress, including cognitive load, exam clarity, and question complexity. Studies reveal that poor exam design can exacerbate stress levels, leading to diminished academic performance and negative psychological outcomes (Trigueros, Aguilar-Parra & Cangas, 2020).

While much of the literature focuses on academic stress as an isolated issue, there is limited research on how specific exam design elements contribute directly to test anxiety. This study extends current research by exploring the link between exam paper quality and test anxiety, offering insights into how well-designed exams can alleviate stress.

Automated evaluation systems

The advent of automated evaluation systems in education introduces new possibilities for improving both the consistency and fairness of assessments. Previous research has primarily focused on how these systems enhance grading efficiency and reduce workload for educators (Nguyen & Habók, 2023). However, little attention has been given to how these systems might also reduce stress for students by providing objective, bias-free evaluations.

Automated systems not only minimize human error in grading but also ensure uniformity in assessments, which can help reduce the anxiety that students often experience from perceived grading inconsistencies (Pascoe, Hetrick & Parker, 2020). For educators, these systems free up time by automating routine grading tasks, allowing them to focus on providing more meaningful feedback. This dual impact on both educators and students highlights the potential for these systems to be more than just efficiency tools—they are integral to creating a more supportive learning environment that addresses both academic and psychological needs.

Automated evaluation systems in education

The growing complexity of educational assessment, coupled with the increased number of students in higher education, has led to a rising demand for automated evaluation systems. These systems provide several advantages, including consistency in grading, reduction of human errors, and efficiency in managing large-scale evaluations (Nguyen & Habók, 2023). Automated systems have been shown to significantly reduce the workload for evaluators, particularly when handling repetitive tasks like grading multiple-choice or true/false questions, allowing them to focus more on creative and critical-thinking assessments (Mate & Weidenhofer, 2022).

Studies have demonstrated the effectiveness of machine learning models, such as Naive Bayes and SVM, in automating the evaluation process. For instance, Nguyen & Habók (2023) achieved 90.2% accuracy using Naive Bayes, while Mate & Weidenhofer (2022) achieved 88.5% with an SVM approach. However, these models primarily focus on evaluating multiple-choice questions, leaving a gap in evaluating more complex, open-ended responses.

By addressing this research gap, our proposed automated evaluation system introduces an integrated framework that not only handles technical and formal criteria in exam papers but also ensures a fair and consistent evaluation of diverse question formats. Unlike prior approaches that focus solely on accuracy, our system also emphasizes reducing evaluator stress and improving overall grading efficiency (Elbourhamy, 2024).

Research gaps and the need for an integrated approach

Despite extensive literature on exam quality and academic stress, there remains a lack of research that integrates these areas, particularly through the use of automated systems. This study addresses this gap by proposing a framework that evaluates both exam quality and test anxiety concurrently. By developing an automated system that evaluates formal and technical aspects of exam papers and correlates them with student stress levels, this research offers a comprehensive and objective method of assessment.

Significance of the proposed research

The proposed research introduces a novel approach to exam paper evaluation by combining automated analysis with psychological metrics, such as test anxiety. By linking exam quality to student well-being, this study aims to revolutionize how assessments are conducted in higher education. For educators, the system offers actionable insights into how exam design influences student stress levels, enabling improvements in exam construction. For students, the automated system ensures high-quality, fair exams that are not only cognitively challenging but also mindful of their psychological well-being.

In conclusion, this study emphasizes the critical role that exam structure and presentation play in student success and well-being. By automating the evaluation process and correlating exam quality with academic stress, this research seeks to inform and enhance both educator training and student outcomes, ultimately aiming to reduce test anxiety and improve performance in higher education.

The key research question guiding this study is: “How effective is the proposed automated system in evaluating exam papers, and what is its impact on students’ test anxiety in higher education?”

Therefore, this study proposes an innovative automated model for evaluating exam papers, utilizing specific formal and technical evaluation criteria, such as university, faculty, course, question headers, repeated questions, and repeated alternatives, etc. The goal is to foster a culture of rigorous measurement and evaluation among educators and students, supporting instructors in designing high-quality exam papers that not only meet formal standards but also reduce students’ academic stress. This, in turn, will enhance the efficiency and well-being of educational processes in higher education institutions.

Data collection

The data for this study was collected from 50 first-year English-language computer science teacher students at the Faculty of Specific Education, Kafrelsheikh University. Data collection occurred over two semesters (2021/2022 academic year), with assessments of academic stress levels taken both before and after the implementation of the automated system. Written informed consent was obtained from all participants, with the option of withdrawal without penalty.

The automated system was applied to 30 exam papers, each containing approximately 60 questions, totaling 1,800 questions. The implementation occurred in two phases: after the first-semester exams and again after the second-semester exams, focusing exclusively on English-written exams.

The evaluation criteria used in this study were developed based on an extensive review of the literature on exam quality and assessment standards (Akçay, Tunagür & Karabulut, 2020; Karatay & Dilekçi, 2019; Shepard, 2019), and in consultation with experts in measurement and evaluation. These criteria, officially adopted by the Faculty of Specific Education at Kafrelsheikh University, focus on formal and technical aspects, rather than cognitive levels.

-Formal criteria include the presence of essential information such as university and course names, exam date, time allocation, and total exam score, ensuring compliance with institutional standards (Nguyen & Habók, 2023).

-Technical criteria cover the structure and clarity of the exam content, such as clear question headers, non-redundant questions, and the avoidance of ambiguous options like “all of the above” or “none of the above” (Mate & Weidenhofer, 2022; Shultz, Whitney & Zickar, 2020).

By implementing these criteria, the automated evaluation system aims to provide a comprehensive review of exam papers, ensuring adherence to institutional standards and reducing inconsistencies that could contribute to student stress. Table 3 summarizes the formal and technical criteria used in the evaluation.

Development of the automated evaluation system

The proposed automated system was developed to assist instructors in refining exam papers and improving their quality. Figure 1 provides an overview of the system’s architecture, detailing its key components and the process flow.

Key features of the automated system:

1. Upload exam paper: Instructors upload exam papers as PDF files.

2. PDF to text conversion: The system uses the PyPDF2 library to extract text from the PDF, which is then concatenated into a single string for further processing.

(1) $$Text=\sum _{i=1}^{n}pag{e}_i.$$

• Text: The concatenated text from the entire PDF.

• n: The number of pages in the PDF.

• $pag{e}_i$: The text extracted from the i-th page of the PDF.

3. Text preprocessing: The text is preprocessed by converting it to lowercase, removing special characters, and correcting spelling errors.

(2) $$CleanedText=Lowercase\left(RemoveSpecialChars\left(Text\right)\right).$$

• $CleanedText$: The preprocessed text.

• Lowercase(x): Converts text x to lowercase.

• RemoveSpecialChars(x): Removes special characters from text x.

4. Tokenization and Stemming: The nltk library’s ‘word_tokenize’ function is used to tokenize the text into individual words, while the PorterStemmer reduces words to their root forms (Vijayarani & Janani, 2016; Mohammdi & Elbourhamy, 2021).

(3) $$\mathrm{T}\mathrm{o}\mathrm{k}\mathrm{e}\mathrm{n}\mathrm{s}=\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{d}\mathrm{\_}\mathrm{t}\mathrm{o}\mathrm{k}\mathrm{e}\mathrm{n}\mathrm{i}\mathrm{z}\mathrm{e}\left(\mathrm{C}\mathrm{l}\mathrm{e}\mathrm{a}\mathrm{n}\mathrm{e}\mathrm{d}\mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}\right).$$

• Tokens: The list of tokenized words.

• word_tokenize(x): Tokenizes the text x into individual words.

5. Feature extraction: The TF-IDF vectorizer from the scikit-learn library transforms tokenized text into numerical features suitable for machine learning algorithms.

(4) $$\mathrm{T}\mathrm{F}-\mathrm{I}\mathrm{D}\mathrm{F}\left(\mathrm{t},\mathrm{d},\mathrm{D}\right)=\mathrm{T}\mathrm{F}\left(\mathrm{t},\mathrm{d}\right)\times \mathrm{I}\mathrm{D}\mathrm{F}\left(\mathrm{t},\mathrm{D}\right).$$

• $\mathrm{T}\mathrm{F}-\mathrm{I}\mathrm{D}\mathrm{F}\left(\mathrm{t},\mathrm{d},\mathrm{D}\right)$: The TF-IDF score of term t in document d within the document set D.

• $\mathrm{T}\mathrm{F}\left(\mathrm{t},\mathrm{d}\right)$: The term frequency of term t in document d.

• $\mathrm{I}\mathrm{D}\mathrm{F}\left(\mathrm{t},\mathrm{D}\right)$: The inverse document frequency of term t in the document set D.

6. Model training: A naive Bayes classifier is trained on labeled exam questions to classify formal and technical criteria. The model was trained using labeled data and evaluated on a test set to ensure accuracy.

(5) $$P\left(c|d\right)={\displaystyle \frac{\left(d|c\right).P\left(c\right)}{P\left(d\right)}.}$$

• $P\left(c|d\right)$: The probability of class c given document d.

• $\left(d|c\right)$: The probability of document d given class c.

• $P\left(c\right)$: The prior probability of class c.

• $P\left(d\right)$: The prior probability of document d.

7. Criteria database: The SQLite-based criteria database was developed to store the formal and technical criteria. This dynamic, user-friendly database allows administrators to add or remove criteria as needed, ensuring that the system remains flexible and up-to-date. Figure 2 shows the administrative panel of the database.

8. Searching and matching: The system searches for keywords that match the criteria stored in the database. The trained naive Bayes model predicts the matching criteria for new exam papers.

(6) $$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{c}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{a}=\mathrm{M}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}.\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\left(\mathrm{T}\mathrm{F}-\mathrm{I}\mathrm{D}\mathrm{F}\mathrm{F}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}\mathrm{s}\right).$$

• $\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{c}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{a}$: The predicted criteria for the new text data.

• $\mathrm{M}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}.\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}(\mathrm{x})$: The prediction of the model for features x.

9. Automated question counting: A Python script automates the counting and classification of multiple-choice and true/false questions using regular expressions. This ensures accurate and efficient detection of question types.

Libraries used:

• PyPDF2: A library for reading PDF files.

• re (Regular Expressions): A library for matching patterns in text.

Explanation of the process:

1. extract_text_from_pdf: Extracts text from the PDF file.

2. identify_question_patterns: Defines regex patterns for identifying multiple-choice, true/false, and short answer questions.

3. classify_questions: Uses regex patterns to classify questions and returns lists of multiple-choice, true/false, and short answer questions.

4. main: Orchestrates the process of extracting text, classifying questions, and printing the results.

This automated approach ensures accurate and efficient counting of questions in exam papers, facilitating better exam preparation and analysis.

10. Evaluation of exam time correspondence: To evaluate whether the number of questions corresponds to the exam time, we set standard average times for each type of question based on recent educational research:

• Multiple-choice question: 1.5 min (Cognitive Research: Principles and Implications, 2023; Evidence Based Education, 2022).

• True/False question: 1 min (International Journal of STEM Education, 2023).

For an exam with $N_{MC}$ multiple-choice questions and $N_{TF}$ true/false questions, the expected total time needed is calculated as follows:

• 1. Expected time for multiple-choice questions:

(7) $$\mathrm{E}\mathrm{x}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{T}\mathrm{i}\mathrm{m}{\mathrm{e}}_{MC}=N_{MC}\times \mathrm{A}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{g}\mathrm{e}\mathrm{T}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{M}\mathrm{C}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}.$$

Parameters:

• $N_{MC}$: The total count of multiple-choice questions in the exam.

• $\mathrm{A}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{g}\mathrm{e}\mathrm{T}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{M}\mathrm{C}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}$: The average time allocated to answer each multiple-choice question is 1.5 min.

• 2. Expected time for true/false questions:

(8) $$\mathrm{E}\mathrm{x}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{T}\mathrm{i}\mathrm{m}{\mathrm{e}}_{TF}=N_{TF}\times \mathrm{A}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{g}\mathrm{e}\mathrm{T}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{T}\mathrm{F}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}$$

Parameters:

• $N_{TF}$: The total count of true/false questions in the exam.

• $\mathrm{A}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{g}\mathrm{e}\mathrm{T}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{T}\mathrm{F}\mathrm{Q}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}$: The average time allocated to answer each true/false question, set to 1 min.

• 3. Total expected time:

(9) $$\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}\mathrm{e}\mathrm{x}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}=\mathrm{E}\mathrm{x}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{T}\mathrm{i}\mathrm{m}{\mathrm{e}}_{MC}+\mathrm{E}\mathrm{x}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{T}\mathrm{i}\mathrm{m}{\mathrm{e}}_{TF}.$$

Parameters:

• $\mathrm{E}\mathrm{x}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{T}\mathrm{i}\mathrm{m}{\mathrm{e}}_{MC}$: The total expected time to answer all multiple-choice questions.

• $\mathrm{E}\mathrm{x}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{T}\mathrm{i}\mathrm{m}{\mathrm{e}}_{TF}$: The total expected time to answer all true/false questions.

Given that the actual allotted exam time $(T_{exam})$ is 120 min, we compare it with the total expected time $(T_{total})$. If $(T_{total})$ ≤ $(T_{exam})$, then the number of questions corresponds well with the exam time, ensuring that students have ample time to complete the exam. Otherwise, adjustments to the number of questions or the allotted time are necessary.

11. Report generation: The system compiles the results into a detailed report that includes both the formal and technical criteria of the exam paper, along with the number and types of questions and the exam’s time correspondence. This comprehensive report serves as a tool for instructors to refine and enhance the quality of their exam papers. The report can be conveniently printed or emailed directly to the instructor or the educational institution’s Measurement and Evaluation department.

Evaluation metrics and procedures

The development of the proposed automated system was guided by the goal of creating an effective and practical tool tailored specifically for educational institutions, particularly universities. This system was engineered using Python 3.11, PHP, and the Laravel framework, providing a robust and scalable foundation for evaluating exam papers. The primary aim of the system is to support teachers in evaluating exam papers efficiently and to reduce students’ academic stress by ensuring exam fairness and consistency.

System design and user interface

The user experience was a critical focus during the system’s development. The design features a user-friendly interface to simplify the evaluation process. The homepage offers an intuitive “Choose File” option, where instructors can upload exam files in PDF format (Fig. 3). This design choice ensures ease of use, allowing users to initiate the evaluation process without complexity.

The system evaluates the uploaded exam papers in six sequential stages, each focusing on specific formal and technical criteria. These stages were designed to streamline the evaluation process and ensure comprehensive checks of exam papers against key standards.

Stage 1: File upload and initial interface

-Users begin by uploading the exam file in PDF format through the homepage interface. Once the file is selected, users proceed by clicking the “Next | Upload” button, moving to the next phase.

Stage 2: Formal criteria check

-In the second stage, the system scans the exam paper for key formal criteria, such as the presence of terms like ‘university’ and ‘college’ (Fig. 4). This is initiated by clicking the “Check Required” button. The system then validates whether the exam paper meets the required institutional standards, providing immediate feedback to users.

Stage 3: Forbidden words identification

-The third stage evaluates the presence of undesired words in the exam paper, such as ‘only’ and ‘all of the above’ (Fig. 5). Users initiate this stage by selecting the “Check Forbidden” button. The system scans the document, identifying any forbidden phrases that may cause ambiguity or unfairness in the assessment.

Stage 4: Duplicated questions detection

-In the fourth stage, the system checks for duplicated questions (Fig. 6). By clicking the “Check Duplicated Questions” button, users can ensure that no question is repeated within the exam, maintaining the uniqueness and validity of the assessment.

Stage 5: Duplicated answers in multiple-choice questions

-The fifth stage involves identifying redundant alternatives in multiple-choice questions (Fig. 7). Clicking the “Check Duplicated Answers” button allows the system to verify that no answer choice is repeated across multiple questions, thus ensuring fairness and avoiding confusion.

Stage 6: Question header, number, and time analysis

-In the sixth stage, the system verifies the presence of question headers and evaluates the number of questions relative to the allocated exam time (Fig. 8). This analysis ensures that the exam length and complexity are proportionate to the available time, addressing one of the key concerns related to student stress—time management during exams.

Final Stage: Comprehensive report generation

The final stage involves generating a comprehensive report summarizing the exam’s evaluation across all formal and technical criteria (Fig. 9). This report not only indicates whether the exam adheres to institutional standards but also provides an analysis of other factors such as question type distribution and time alignment. The report can be exported as a PDF file and emailed directly to educational institution officials for further review or action.

Evaluation of the system

The effectiveness and efficiency of the proposed automated evaluation system were rigorously assessed through accuracy testing and analysis of the system’s impact on students’ academic stress. This section outlines the evaluation process, presenting both quantitative performance metrics and the system’s influence on reducing test anxiety.

Conclusion of the evaluation

The comprehensive evaluation of the system demonstrated that it is both accurate and reliable in identifying formal and technical criteria in exam papers. Furthermore, the use of the automated system positively impacted student well-being by reducing test anxiety and ensuring fair and consistent exam evaluations. The findings suggest that automated evaluation tools can play a crucial role in improving both the quality of education and the mental health of students in higher education institutions.

Ethical statement

The ethical committee at Kafrelsheikh University has reviewed the study protocol and ethically approved the study under reference No. 334-38-44972-SD.

Exploratory factor analysis

To explore the underlying factors that contribute to the relationship between exam quality and student stress, we conducted an exploratory factor analysis (EFA). This analysis helps validate the structure of the measurement scale used in the study, which was designed to assess students’ perceptions of exam quality and their associated stress levels.

-The Kaiser-Meyer-Olkin (KMO) measure of 0.82 indicated that the sample was adequate for factor analysis.

-Bartlett’s test of sphericity was significant (χ² = 512.34, p < 0.001), confirming that the dataset was suitable for EFA.

-Three factors were extracted, explaining 62.7% of the total variance as shown in Table 5:

• 1.

  Academic stress: Captures the overall levels of stress related to exam preparation and performance.

• 2.

  Exam quality perception: Reflects students’ evaluations of the clarity and fairness of the exam papers.

• 3.

  Relationship between exam quality and stress: Describes how perceptions of exam quality impact students’ stress levels.

Table 6 shows the factor loadings for each item on the three factors after Varimax rotation.

The EFA findings confirm that the scale appropriately captures the relationship between exam quality and academic stress. These factors are critical for understanding how improving exam clarity, fairness, and structure can alleviate students’ anxiety.

Comparison of the proposed system with expert evaluation

Table 7 provides a detailed comparison between the system’s predictions and expert evaluations across various formal and technical criteria. The table includes columns for:

-Expert evaluation: The manual evaluation performed by experts to determine if the criteria were present (✓) or absent (✗).

-System prediction: The system’s prediction of whether the criteria were present (✓) or absent (✗).

-Correct predictions (System): The number of correct predictions made by the system, where its output matched the expert evaluation.

-Incorrect predictions (System): The number of incorrect predictions made by the system, where its output did not match the expert evaluation.

Detailed explanation of Table 7:

In Table 7, the system’s performance is summarized by evaluating 18 different criteria, such as the presence of terms like “university” and “faculty” or technical aspects like the presence of question headers and repeated questions. The “Total” row at the bottom represents the aggregated results of all criteria evaluations.

-Expert evaluation: Indicates the expert’s manual assessment for each criterion. A value of ‘1’ denotes the presence of the criterion, while ‘✗’ means it was absent. In some cases (like "Presence of Department"), the criterion was not evaluated manually (indicated by ‘-’).

-System prediction: Reflects whether the system identified the same criteria correctly.

-Correct predictions (System): This column counts instances where the system correctly matched the expert evaluation.

-Incorrect predictions (System): This column records where the system failed to match the expert’s evaluation.

Total calculation

-The "Total" row at the bottom of the table aggregates the number of correct and incorrect predictions across all 18 criteria.

-The system’s predictions were evaluated against expert evaluations. For 18 evaluation criteria, the system correctly identified 97 instances where the criteria were present (true positives) and misclassified 1 instance (false negative). This resulted in a precision, recall, and F1-score of 98.96%, confirming the reliability of the system in automating exam paper quality assessments.

Evaluator stress assessment

Additionally, the study evaluated the stress levels of examiners during manual and automated evaluations to explore the potential of reducing evaluator stress with automated systems.

While this study primarily focuses on the impact of the automated system on student stress, it is essential to consider the burden evaluators experience when manually assessing large volumes of exam papers. In many educational settings, evaluators are responsible for grading 30 to 40 answer scripts or more, a time-consuming and repetitive process that can result in evaluator burnout, reduced efficiency, and inconsistencies in grading (Shultz, Whitney & Zickar, 2020).

By implementing the proposed automated evaluation system, institutions can alleviate some of the stress experienced by evaluators. The system ensures a more consistent, objective, and timely assessment, reducing the workload and allowing evaluators to focus on more complex and subjective evaluations, such as descriptive answers. This dual benefit—reduced evaluator stress and more reliable student assessment—highlights the broader value of automated systems within educational institutions.

System evaluation metrics

Table 8 summarizes the key evaluation metrics used to assess the system’s performance in predicting and evaluating formal and technical exam criteria. The metrics include true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). The system achieved an impressive overall accuracy of 98%, indicating that it correctly predicted the vast majority of the evaluation criteria.

In addition to accuracy, other important metrics were also calculated:

-Precision: The system achieved a precision of 98.96%, meaning that nearly all the criteria it identified as positive were correct.

-Recall: With a recall of 98.96%, the system successfully identified almost all of the actual positive criteria.

-F1-score: The F1-score, which balances both precision and recall, was also 98.96%, reflecting the system’s ability to maintain high precision and recall simultaneously.

These results demonstrate the system’s high reliability and effectiveness in ensuring accurate and consistent evaluation of formal and technical exam criteria. By minimizing both false positives and false negatives, the system ensures a high level of trustworthiness in its assessments, which can significantly improve exam paper quality and reduce evaluator bias.

Comparison with related works

This section outlines the comparative performance of the proposed system with other existing models in the field of automated exam evaluation. Table 9 presents a detailed comparison of the accuracy achieved by various models, including naive Bayes, support vector machine (SVM), rule-based, and random forest approaches.

For instance, Nguyen & Habók (2023) employed a naive Bayes model, achieving an accuracy of 90.2%, while Mate & Weidenhofer (2022) used an SVM approach and reached 88.5% accuracy. Similarly, a rule-based system developed by Akçay et al. (2020) reported a lower accuracy of 76%, and the random forest approach by Uysal et al. (2022) attained 85% accuracy.

In contrast, the proposed system, utilizing a naive Bayes classifier, significantly outperforms these models with an accuracy rate of 98%. This higher accuracy demonstrates the robustness and reliability of the proposed system in accurately evaluating exam papers based on formal and technical criteria.

These comparisons underscore the superior performance of the proposed system, not only surpassing traditional methods but also offering more precise and consistent results. By achieving the highest accuracy among the referenced models, this system proves to be a promising tool for enhancing exam paper evaluation in educational institutions, reducing human error, and promoting fairer assessments.

Analysis of exam paper quality and academic stress

A correlation analysis was conducted to explore the relationship between exam paper quality and student stress levels. The heatmap in Fig. 10 illustrates significant correlations between exam quality and academic stress indicators. Key findings include:

-I_1 and I_5 (r = 0.70): A strong positive correlation, showing that students who feel tense before exams also worry frequently about their performance.

-T_1 and T_4 (r = 0.65): Clear and understandable exam questions are associated with better student performance.

-I_1 and T_3 (r = −0.45): Lower student anxiety when students perceive the exam as a fair assessment.

-I_4 and T_1 (r = −0.40): Clear exam questions reduce students’ concentration difficulties.

This analysis highlights how well-structured, fair, and clear exams can significantly reduce student anxiety, emphasizing the practical importance of these criteria in supporting student well-being.

Insights from scatter plots and correlation analysis

Figure 11 presents scatter plots showing relationships between key variables from the correlation analysis:

1. Stress before exams vs. perceived fairness (I_1 vs. T_3): A negative correlation indicates that students experience lower stress when they perceive the exam as fair.

2. Difficulty concentrating vs. clear questions (I_4 vs. T_1): Clear questions reduce students’ concentration difficulties caused by anxiety.

These results confirm that well-designed exams, which are clear and fair, can help alleviate student stress, enhancing both student well-being and performance.

This analysis examines the perceived fairness, clarity, and alignment of exam questions (as evaluated by students) and their reported academic stress levels. Although based on subjective perceptions, the findings indicate significant correlations that underline the importance of exam paper design in reducing stress.

Comparisons between pre-and post-treatment results

The paired samples of t-test results provide a comparison between pre-and post-treatment measures of academic stress and exam quality indicators. Here is a summary of the key findings based on the data you provided:

1. Significant improvement in post-treatment scores:

-I_2, I_3, I_4, I_5, I_6, I_7, I_8: Significant improvements are observed in these measures, indicating that post-treatment scores are significantly higher than pre-treatment scores (p < 0.001).

-T_1, T_2, T_3, T_5, T_6: The post-treatment results for these measures show significant improvement compared to pre-treatment scores (p < 0.001).

-F_1: In contrast, this measure shows a significant negative difference, meaning the post-treatment score is lower than the pre-treatment score.

2. No significant changes:

-I_1: There was no statistically significant difference between pre- and post-treatment scores for this measure (p = 0.182).

3. Correlations:

-Strong positive correlations for most measures indicate consistency between pre- and post-treatment results in their relative rankings.

The significant improvement in most measures of academic stress and exam quality indicators after implementing the automated evaluation system suggests that the system positively impacts students’ experience. The improvement in items related to stress before exams (I_2, I_3, I_5, I_6, T_1, T_2) indicates that students feel more confident and less anxious, thanks to the clarity and fairness introduced by the system.

The items related to technical aspects of the exam paper (T_3, T_5, T_6) also show notable improvement, emphasizing the system’s ability to enhance the structural and technical quality of exam papers. However, the significant drop in the F_1 post-treatment score may warrant further investigation, as this could indicate an area where the system needs refinement.

So, the pre- and post-treatment comparisons indicate that the automated evaluation system has effectively reduced students’ academic stress and improved the quality of exam papers. This is evidenced by significant improvements in both stress and technical indicators. Moving forward, refinements can be made to further enhance specific aspects where discrepancies were observed.

Impact on evaluator stress

While this study primarily focuses on student stress, it is essential to consider the stress experienced by evaluators during manual assessment. The manual grading of large volumes of exam papers is time-consuming and often leads to evaluator burnout, impacting the quality and consistency of evaluations (Shultz, Whitney & Zickar, 2020). Automated systems help mitigate this burden by providing a standardized evaluation framework, ensuring that evaluators can maintain high-quality assessments without the added pressure of manually grading repetitive content (Nguyen & Habók, 2023). These benefits ultimately contribute to a more efficient and effective assessment process.

System evaluation by experts and college officials

A survey instrument was developed to evaluate the proposed system, focusing on usability, clarity, response time, and applicability. The survey was validated through expert review, digitized using Google Forms, and administered to 30 participants, including educators and administrative staff.

Before deploying the proposed evaluation system, a comprehensive survey methodology was developed to assess its effectiveness. This process involved multiple stages:

-Crafting initial survey statements to cover key aspects of the system.

-Presenting the drafts to subject matter experts for review and feedback.

-Revising the survey based on expert input to enhance its clarity and relevance.

-Finalizing and digitizing the survey using Google Forms (Form link: https://forms.gle/URaQssRpFCVG1M3QA).

The survey was then conducted among officials at the Faculty of Specific Education and experts in the field to evaluate the system across several dimensions, including usability, effectiveness, clarity, applicability, and response time. A five-point Likert scale was employed, with responses ranging from 1 (Poor) to 5 (Excellent).

The survey results (Fig. 12) revealed high satisfaction among participants, with ratings of ‘Excellent’ in usability (95%), effectiveness (92%), and clarity (90%), demonstrating the system’s readiness for institutional implementation. Specifically, the system was praised for its:

-Usability: The system’s interface was easy to use, with clear navigation.

-Effectiveness: It reliably evaluated exam papers and identified criteria accurately.

-Clarity: The processes and outputs of the system were well-structured and easy to understand.

-Applicability: The system was deemed highly applicable to real-world educational processes, with the potential to significantly improve exam evaluations.

-Response Time: The system’s speed in processing and delivering results was highly rated.

Overall, the system received positive feedback for its efficiency, clarity, and potential to enhance educational practices. The high satisfaction scores reflect the value of this automated evaluation tool in streamlining exam paper assessments and ensuring higher quality standards within educational institutions.

Effect size results

The effect sizes (Cohen’s d) for the pre- and post-test comparisons are summarized in Table 10. These results provide insight into the system’s influence on enhancing exam quality.

The results demonstrate substantial improvements, particularly for variable I_5, which shows a large effect size (d = 2.123). This indicates the system’s success in addressing fairness and alignment in exam questions. Moderate improvements in variables such as I_2 (d = 0.543) highlight meaningful enhancements in clarity and structure.

Study limitations

• The study was limited to the evaluation of exam papers from the computer science teacher program at a single institution.

• The sample size, though adequate, could be expanded for more generalized findings.

• The sample size used in this study, while adequate for preliminary findings, may not fully represent the diversity of the student population. Future research should include a larger sample size across multiple institutions.

Future directions

• Extending the study to include multiple academic years and different educational levels to validate the findings across a broader spectrum.

• Incorporate additional criteria in the evaluation system to cover a wider range of assessment aspects.

• Explore the impact of automated evaluation systems on other forms of assessments beyond objective tests.

• Conduct longitudinal studies to assess the long-term effects of improved exam paper quality on academic stress and overall student performance.

• Future work will explore the integration of descriptive-answer evaluations, and short-answer text evaluation. To address these challenges, future research will incorporate natural language processing (NLP) techniques for evaluating short-answer responses, including keyword matching and context-based algorithms to ensure accurate and fair assessments.

• Although this study primarily examined the relationship between exam paper quality and student academic stress, future research should also assess the impact of automated evaluation systems on evaluator stress. Investigating how automation alleviates the pressure faced by educators in large-scale assessments will provide a more comprehensive view of the system’s benefits for both students and educators.

• While this study focused on the impact of automated evaluation systems on exam paper quality and student anxiety, future research should explore the potential of these systems in reducing teacher stress during the evaluation process. Evaluating large volumes of exam papers can be time-consuming and stressful for educators, potentially affecting their performance and consistency. Incorporating stress metrics for teachers would provide a more comprehensive understanding of the benefits of automated systems for all stakeholders.