The context similarity problem in academic plagiarism detection: A case study on differentiating originality from collusion in a homogeneous dataset

Department of Informatics Engineering, Politeknik Negeri Bandung, 40559 Bandung, Indonesia

^* Corresponding author: jonnerh@jtk.polban.ac.id

Abstract

Intraclass collusion poses a significant challenge for automated detection systems within homogeneous datasets from e-learning platforms, where legitimate contextual overlap often leads to high false-positive rates. This “context similarity problem” questions the utility of advanced semantic models. This study confronts this issue through a quantitative comparison of a state-of-the-art Sentence-BERT model against traditional lexical methods (Levenshtein, Jaccard) on a real-world dataset of 854 student answers. Our findings reveal a compelling, counter-intuitive result: lexical methods are demonstrably more effective. Levenshtein Similarity achieved a superior F1-score (0.74) and F2-score (0.75), underpinned by a strong recall of 0.76. Conversely, the semantic model was confounded by the dataset’s homogeneity, yielding a modest F1-score of 0.57 and requiring an impractically high similarity threshold of 0.98 for optimal performance. This research provides a critical contribution by empirically demonstrating the limitations of purely semantic approaches in this specific context. We conclude that well-established lexical methods are not obsolete but remain a more reliable and practical tool for the initial screening of academic collusion, suggesting a need for hybrid strategies in modern academic integrity systems.