Preview

Informatics and education

Advanced search

Monitoring data shift in the learning success forecasting model using Shapley values

https://doi.org/10.32517/0234-0453-2025-40-3-57-68

Abstract

   Predicting academic success is one of the most critical tasks in learning analytics. Solving this problem enables educational institutions to identify students struggling with their studies in a timely manner and apply appropriate pedagogical support and interventions. For predictive models to remain effective in the long term after being integrated into the educational process, it is essential to regularly monitor data shift within these models. This helps detect changes in the distribution of educational data that could lead to decreased forecasting accuracy and affect the interpretation of predictions.
The article presents a methodology for analyzing data shift in academic performance forecasting models using Shapley values, incorporating a two-stage monitoring approach. In the first stage (during the educational process), while the forecasting model is predicting academic success, changes in predictor contributions to the final forecast are identified. In the second stage (after exams), when prediction accuracy can be evaluated, shifts in predictor contributions to the model’s loss function are analyzed. This approach enables the detection of different types of data shifts. Based on the analysis results, an educational institution can make timely decisions regarding model adjustments, retraining, or replacement. The proposed methodology was tested on models from the academic performance forecasting service Pythia, developed at Siberian Federal University. After the service was integrated into the educational process, forecasting on new data was already conducted over four academic semesters, revealing a decline in prediction accuracy. Through monitoring, several predictors responsible for covariate or concept shifts within the ensemble models were identified. As a result, recommendations for model adjustments were proposed.

About the Authors

T. A. Kustitskaya
Siberian Federal University
Russian Federation

Tatiana A. Kustitskaya, Candidate of Sciences (Physics and Mathematics), Docent, Associate Professor

School of Space and Information Technology; Department of Applied Mathematics and Data Analysis

Krasnoyarsk



R. V. Esin
Siberian Federal University
Russian Federation

Roman V. Esin, Candidate of Sciences (Education), Docent,Associate Professor

School of Space and Information Technology; Department of Applied Mathematics and Data Analysis

Krasnoyarsk



References

1. Dyulicheva Yu. Yu. Application of learning analytics in higher education: Datasets, methods and tools. Higher Education in Russia. 2024;33(5):86–111. (In Russian.) EDN: EIDBZY. DOI: 10.31992/0869-3617-2024-33-5-86-111.

2. Kustitskaya T. A., Noskov M. V. Learning analytics in Russia and abroad: Level of development, trends and prospects. RUDN Journal of Informatization in Education. 2023;20(2):150–158. EDN: KEROFN. DOI: 10.22363/2312-8631-2023-20-2-150-158.

3. Sonnleitner B., Madou T., Deceuninck M., Theodosiou F., Sagaert Y. R. Evaluation of early student performance prediction given concept drift. Computers and Education: Artificial Intelligence. 2025;8:100369. DOI: 10.1016/j.caeai.2025.100369.

4. Maia Polo F., Izbicki R., Lacerda E. G., Ibieta-Jimenez J. P., Vicente R. A unified framework for dataset shift diagnostics. Information Sciences. 2023;649:119612. DOI: 10.1016/j.ins.2023.119612.

5. Hinder F., Vaquet V., Hammer B. One or two things we know about concept drift—a survey on monitoring in evolving environments. Part A: Detecting concept drift. Frontiers in Artificial Intelligence. 2024;7:1330257. DOI: 10.3389/frai.2024.1330257.

6. Deho O. B., Liu L., Li J., Liu J., Zhan C., Joksimovic S. When the past != the future: Assessing the impact of data-set drift on the fairness of learning analytics models. IEEE Transactions on Learning Technologies. 2024;17:1007–1020. DOI: 10.1109/TLT.2024.3351352.

7. Guo L. L., Pfohl S. R., Fries J., Posada J., Fleming S. L., Aftandilian C., Shah N., Sung L. Systematic review of approaches to preserve machine learning performance in the presence of temporal dataset shift in clinical medicine. Applied Clinical Informatics. 2021;12(04):808–815. DOI: 10.1055/s-0041-1735184.

8. Sahiner B., Chen W., Samala R. K., Petrick N. Data drift in medical machine learning: implications and potential remedies. The British Journal of Radiology. 2023;96(1150):20220878. DOI: 10.1259/bjr.20220878.

9. Allogmany B., Josyula D. An approach to dealing with incremental concept drift in personalized learning systems. IEEE 4th Int. Conf. on Cognitive Machine Intelligence (CogMI). 2022:139–147. DOI: 10.1109/CogMI56440.2022.00029.

10. Tiukhova E., Vemuri P., Flores N. L., Islind A. S., Óskarsdóttir M., Poelmans S., Baesens B., Snoeck M. Explainable Learning Analytics: Assessing the stability of student success prediction models by means of explainable AI. Decision Support Systems. 2024;182:114229. DOI: 10.1016/j.dss.2024.114229.

11. Ovadia Y., Fertig E., Ren J., Nado Z., Sculley D., Nowozin S., Dillon J., Lakshminarayanan B., Snoek J. Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing systems. 2019;32.

12. Namiot D., Ilyushin E. Data shift monitoring in machine learning models. International Journal of Open Information Technologies. 2022;10(12):84–93. (In Russian.) EDN: NDOMRK.

13. Wan J. S.-W., Wang S.-D. Concept drift detection based on pre-clustering and statistical testing. Journal of Internet Technology. 2021;22(2):465–472. DOI: 10.3966/160792642021032202020.

14. Gama J., Žliobaite I., Bifet A., Pechenizkiy M., Bouchachia A. A survey on concept drift adaptation. ACM Computing Surveys (CSUR). 2014;46(4):1–37. DOI: 10.1145/2523813.

15. Gama J., Medas P., Castillo G., Rodrigues P. Learning with drift detection. Advances in Artificial Intelligence–SBIA 2004. 2004;3171:286–295. DOI: 10.1007/978-3-540-28645-5_29.

16. Baena-Garcıa M., del Campo-Ávila J., Fidalgo R., Bifet A., Gavalda R., Morales-Bueno R. Early drift detection method. Fourth Int. Workshop on Knowledge Discovery from Data Streams. 2006;6:77–86.

17. Liu A., Lu J., Liu F., Zhang G. Accumulating regional density dissimilarity for concept drift detection in data streams. Pattern Recognition. 2018;76:256–272. DOI: 10.1016/j.patcog.2017.11.009.

18. Ackerman S., Dube P., Farchi E., Raz O., Zalmanovici M. Machine learning model drift detection via weak data slices. 2021 IEEE/ACM Third Int. Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest). 2021:1–8. DOI: 10.1109/DeepTest52559.2021.00007.

19. Dong S., Wang Q., Sahri S., Palpanas T., Srivastava D. Efficiently mitigating the impact of data drift on machine learning pipelines. Proc. VLDB Endowment. 2024;17(11):3072–3081. DOI: 10.14778/3681954.3681984.

20. Ma S., Tourani R. Predictive and causal implications of using shapley value for model interpretation. Proc. 2020 KDD workshop on causal discovery. 2020:23–38.

21. Zheng S., van der Zon S. B., Pechenizkiy M., de Campos C. P., van Ipenburg W., de Harder H., Nederland R. Labelless concept drift detection and explanation. NeurIPS 2019 Workshop on Robust AI in Financial Services: Data, Fairness, Explainability, Trustworthiness, and Privacy. 2019:1–10.

22. Somova M. V. Methodological model of personalized educational process based on prediction of success in subject training. Modern High Technologies. 2023;(12-1):165–170. (In Russian.) EDN: IARKPS. DOI: 10.17513/snt.39877.

23. Kustitskaya T. A., Esin R. V., Vainshtein Y. V., Noskov M. V. Hybrid approach to predicting learning success based on digital educational history for timely identification of at-risk students. Education Sciences. 2024;14(6):657. DOI: 10.3390/educsci14060657.

24. Noskov M. V., Somova М. V., Fedotova I. M. Management of the success of student’s learning based on the Markov model. Informatics and Education. 2018;(10(299)):4–11. (In Russian.) EDN: SMQFKX. DOI: 10.32517/0234-0453-2018-33-10-4-11.

25. Noskov M. V., Vaynshteyn Yu. V., Somova M. V., Fedotova I. M. Prognostic model for assessing the success of subject learning in conditions of digitalization of education. RUDN Journal of Informatization in Education. 2023;20(1):7–19. (In Russian.) EDN: BDFDRI. DOI: 10.22363/2312-8631-2023-20-1-7-19.

26. Kustitskaya T. A., Esin R. V., Kytmanov A. A., Zykova T. V. Designing an education database in a higher education institution for the data-driven management of the educational process. Education Sciences. 2023;13(9):947. DOI: 10.3390/educsci13090947.

27. Lundberg S. M., Lee S.-I. A unified approach to interpreting model predictions. Neural Information Processing Systems. 2017;30:4768–4777. DOI: 10.48550/arXiv.1705.07874.

28. Lundberg S. M., Erion G., Chen H., DeGrave A., Prutkin J. M., Nair B., Katz R., Himmelfarb J., Bansal N., Lee S. I. From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence. 2020;2(1):56–67. DOI: 10.1038/s42256-019-0138-9.

29. Cohen J. Statistical power analysis for the behavioral sciences (2nd ed.). New Jersey, Routledge; 1988. 567 p. DOI: 10.4324/9780203771587.


Review

For citations:


Kustitskaya T.A., Esin R.V. Monitoring data shift in the learning success forecasting model using Shapley values. Informatics and education. 2025;40(3):57-68. (In Russ.) https://doi.org/10.32517/0234-0453-2025-40-3-57-68

Views: 58


Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.


ISSN 0234-0453 (Print)
ISSN 2658-7769 (Online)