ASSESSING THE ASSESSORS: A COMPARATIVE ANALYSIS OF HUMAN AND AI-BASED SCORING IN EFL WRITING

Abstract

The increasing use of artificial intelligence (AI) in education has raised important questions regarding its potential role in writing assessment, particularly in English as a Foreign Language (EFL) contexts, where fairness, consistency, and efficiency are critical concerns. This study aimed to examine the comparability of human and AI-based scoring in EFL writing assessment by investigating score differences, agreement, reliability, and lecturers’ perceptions of AI-assisted assessment. A convergent mixed-methods design was employed. The quantitative data consisted of 39 student essays collected from two undergraduate EFL writing classes at Universitas Muhammadiyah Barru and assessed independently by a human rater and ChatGPT using the same analytic rubric. The qualitative data were obtained through a semi-structured interview with the human rater. Quantitative data were analyzed using paired-sample t-tests, Wilcoxon signed-rank tests, correlation analyses, Cohen’s Kappa, and Intraclass Correlation Coefficients (ICC), while qualitative data were analyzed through thematic analysis. The findings revealed no statistically significant differences between human and ChatGPT scores. Moderate positive correlations and fair categorical agreement were found between the two scoring approaches. ChatGPT also demonstrated excellent intra-rater reliability across repeated scoring sessions. The interview findings indicated that AI was particularly effective in assessing language-related aspects of writing, whereas human judgment remained essential for evaluating contextual appropriateness, originality, and authenticity. The study concludes that ChatGPT can serve as a reliable supplementary assessment tool but should complement rather than replace human evaluators in EFL writing assessment.