infojournalИнформатика и образованиеInformatics and education0234-04532658-7769Publishing House Education and Informatics10.32517/0234-0453-2025-40-3-57-68infojournal-1280Research ArticleИНФОРМАТИЗАЦИЯ ОБРАЗОВАНИЯINFORMATIZATION OF EDUCATIONМониторинг сдвига данных в модели прогнозирования успешности обучения с использованием значений ШеплиMonitoring data shift in the learning success forecasting model using Shapley valueshttps://orcid.org/0000-0001-9854-1259КустицкаяТ. А.KustitskayaT. A.

Татьяна Алексеевна Кустицкая, канд. физ.-мат. наук, доцент, доцент кафедры

Институт космических и информационных технологий; кафедра прикладной математики и анализа данных

Красноярск

Tatiana A. Kustitskaya, Candidate of Sciences (Physics and Mathematics), Docent, Associate Professor

School of Space and Information Technology; Department of Applied Mathematics and Data Analysis

Krasnoyarsk

tkustitskaya@sfu-kras.ruhttps://orcid.org/0000-0002-9682-4690ЕсинР. В.EsinR. V.

Роман Витальевич Есин, канд. пед. наук, доцент, доцент кафедры

Институт космических и информационных технологий; кафедра прикладной математики и анализа данных

Красноярск

Roman V. Esin, Candidate of Sciences (Education), Docent,Associate Professor

School of Space and Information Technology; Department of Applied Mathematics and Data Analysis

Krasnoyarsk

resin@sfu-kras.ruСибирский федеральный университетРоссияSiberian Federal UniversityRussian Federation2025270720254035768Copyright © Кустицкая Т.А., Есин Р.В., 20252025Кустицкая Т.А., Есин Р.В.Kustitskaya T.A., Esin R.V.This work is licensed under a Creative Commons Attribution 4.0 License.https://info.infojournal.ru/jour/article/view/1280

   Прогнозирование успешности обучения — одна из важнейших задач учебной аналитики, решение которой позволяет образовательным учреждениям своевременно выявлять студентов, испытывающих затруднения в обучении, и применять к ним средства педагогической поддержки и содействия. Для эффективной работы моделей прогнозирования в долгосрочной перспективе после внедрения их в учебный процесс необходимо регулярно проводить мониторинг сдвига данных в используемых моделях. Это позволит выявлять изменения в распределении образовательных данных, которые могут стать причиной падения точности прогнозирования, а также влиять на интерпретацию полученных прогнозов. В статье предлагается методика анализа моделей прогнозирования успешности обучения на сдвиг данных с помощью значений Шепли, предполагающая двухэтапный мониторинг. На первом этапе (непосредственно в ходе образовательного процесса) с помощью исследуемой модели проводится прогнозирование успешности обучения и выявляются изменения вклада предикторов модели в итоговый прогноз. На втором этапе (после проведения промежуточной аттестации), когда уже возможна оценка точности прогнозирования, определяются изменения вклада предикторов в функцию потерь модели. Такой подход позволяет обнаруживать разные типы сдвига данных. На основании результатов анализа образовательное учреждение может принимать своевременные решения о коррекции, переобучении или замене используемой модели. Предложенная методика была апробирована на моделях из сервиса прогнозирования успеваемости «Пифия», разработанного в Сибирском федеральном университете. После внедрения сервиса в образовательный процесс прогнозирование на новых данных было проведено для четырех учебных семестров, и было отмечено падение точности прогнозирования. В результате проведенного мониторинга для каждой из моделей ансамбля были выявлены отдельные предикторы, ставшие источниками ковариационного сдвига или сдвига концепции. Были предложены варианты коррекции используемых моделей.

   Predicting academic success is one of the most critical tasks in learning analytics. Solving this problem enables educational institutions to identify students struggling with their studies in a timely manner and apply appropriate pedagogical support and interventions. For predictive models to remain effective in the long term after being integrated into the educational process, it is essential to regularly monitor data shift within these models. This helps detect changes in the distribution of educational data that could lead to decreased forecasting accuracy and affect the interpretation of predictions.The article presents a methodology for analyzing data shift in academic performance forecasting models using Shapley values, incorporating a two-stage monitoring approach. In the first stage (during the educational process), while the forecasting model is predicting academic success, changes in predictor contributions to the final forecast are identified. In the second stage (after exams), when prediction accuracy can be evaluated, shifts in predictor contributions to the model’s loss function are analyzed. This approach enables the detection of different types of data shifts. Based on the analysis results, an educational institution can make timely decisions regarding model adjustments, retraining, or replacement. The proposed methodology was tested on models from the academic performance forecasting service Pythia, developed at Siberian Federal University. After the service was integrated into the educational process, forecasting on new data was already conducted over four academic semesters, revealing a decline in prediction accuracy. Through monitoring, several predictors responsible for covariate or concept shifts within the ensemble models were identified. As a result, recommendations for model adjustments were proposed.

учебная аналитикаанализ образовательных данныхпрогнозирование успешности обучениясдвиг данныхзначение ШеплиSHAPсервис прогнозирования успеваемости «Пифия»объяснимый искусственный интеллектцифровой следlearning analyticseducation data mininglearning success predictiondata shiftShapley valueSHAPacademic performance forecasting service Pythiaexplainable artificial intelligencedigital footprintReferencesДюличева Ю. Ю. Применение учебной аналитики в высшем образовании: датасеты, методы и инструменты. Высшее образование в России. 2024;33(5):86–111. EDN: EIDBZY. DOI: 10.31992/0869-3617-2024-33-5-86-111.Dyulicheva Yu. Yu. Application of learning analytics in higher education: Datasets, methods and tools. Higher Education in Russia. 2024;33(5):86–111. (In Russian.) EDN: EIDBZY. DOI: 10.31992/0869-3617-2024-33-5-86-111.Kustitskaya T. A., Noskov M. V. Learning analytics in Russia and abroad: Level of development, trends and prospects. RUDN Journal of Informatization in Education. 2023;20(2):150–158. EDN: KEROFN. DOI: 10.22363/2312-8631-2023-20-2-150-158.Kustitskaya T. A., Noskov M. V. Learning analytics in Russia and abroad: Level of development, trends and prospects. RUDN Journal of Informatization in Education. 2023;20(2):150–158. EDN: KEROFN. DOI: 10.22363/2312-8631-2023-20-2-150-158.Sonnleitner B., Madou T., Deceuninck M., Theodosiou F., Sagaert Y. R. Evaluation of early student performance prediction given concept drift. Computers and Education: Artificial Intelligence. 2025;8:100369. DOI: 10.1016/j.caeai.2025.100369.Sonnleitner B., Madou T., Deceuninck M., Theodosiou F., Sagaert Y. R. Evaluation of early student performance prediction given concept drift. Computers and Education: Artificial Intelligence. 2025;8:100369. DOI: 10.1016/j.caeai.2025.100369.Maia Polo F., Izbicki R., Lacerda E. G., Ibieta-Jimenez J. P., Vicente R. A unified framework for dataset shift diagnostics. Information Sciences. 2023;649:119612. DOI: 10.1016/j.ins.2023.119612.Maia Polo F., Izbicki R., Lacerda E. G., Ibieta-Jimenez J. P., Vicente R. A unified framework for dataset shift diagnostics. Information Sciences. 2023;649:119612. DOI: 10.1016/j.ins.2023.119612.Hinder F., Vaquet V., Hammer B. One or two things we know about concept drift—a survey on monitoring in evolving environments. Part A: Detecting concept drift. Frontiers in Artificial Intelligence. 2024;7:1330257. DOI: 10.3389/frai.2024.1330257.Hinder F., Vaquet V., Hammer B. One or two things we know about concept drift—a survey on monitoring in evolving environments. Part A: Detecting concept drift. Frontiers in Artificial Intelligence. 2024;7:1330257. DOI: 10.3389/frai.2024.1330257.Deho O. B., Liu L., Li J., Liu J., Zhan C., Joksimovic S. When the past != the future: Assessing the impact of data-set drift on the fairness of learning analytics models. IEEE Transactions on Learning Technologies. 2024;17:1007–1020. DOI: 10.1109/TLT.2024.3351352.Deho O. B., Liu L., Li J., Liu J., Zhan C., Joksimovic S. When the past != the future: Assessing the impact of data-set drift on the fairness of learning analytics models. IEEE Transactions on Learning Technologies. 2024;17:1007–1020. DOI: 10.1109/TLT.2024.3351352.Guo L. L., Pfohl S. R., Fries J., Posada J., Fleming S. L., Aftandilian C., Shah N., Sung L. Systematic review of approaches to preserve machine learning performance in the presence of temporal dataset shift in clinical medicine. Applied Clinical Informatics. 2021;12(04):808–815. DOI: 10.1055/s-0041-1735184.Guo L. L., Pfohl S. R., Fries J., Posada J., Fleming S. L., Aftandilian C., Shah N., Sung L. Systematic review of approaches to preserve machine learning performance in the presence of temporal dataset shift in clinical medicine. Applied Clinical Informatics. 2021;12(04):808–815. DOI: 10.1055/s-0041-1735184.Sahiner B., Chen W., Samala R. K., Petrick N. Data drift in medical machine learning: implications and potential remedies. The British Journal of Radiology. 2023;96(1150):20220878. DOI: 10.1259/bjr.20220878.Sahiner B., Chen W., Samala R. K., Petrick N. Data drift in medical machine learning: implications and potential remedies. The British Journal of Radiology. 2023;96(1150):20220878. DOI: 10.1259/bjr.20220878.Allogmany B., Josyula D. An approach to dealing with incremental concept drift in personalized learning systems. IEEE 4th Int. Conf. on Cognitive Machine Intelligence (CogMI). 2022:139–147. DOI: 10.1109/CogMI56440.2022.00029.Allogmany B., Josyula D. An approach to dealing with incremental concept drift in personalized learning systems. IEEE 4th Int. Conf. on Cognitive Machine Intelligence (CogMI). 2022:139–147. DOI: 10.1109/CogMI56440.2022.00029.Tiukhova E., Vemuri P., Flores N. L., Islind A. S., Óskarsdóttir M., Poelmans S., Baesens B., Snoeck M. Explainable Learning Analytics: Assessing the stability of student success prediction models by means of explainable AI. Decision Support Systems. 2024;182:114229. DOI: 10.1016/j.dss.2024.114229.Tiukhova E., Vemuri P., Flores N. L., Islind A. S., Óskarsdóttir M., Poelmans S., Baesens B., Snoeck M. Explainable Learning Analytics: Assessing the stability of student success prediction models by means of explainable AI. Decision Support Systems. 2024;182:114229. DOI: 10.1016/j.dss.2024.114229.Ovadia Y., Fertig E., Ren J., Nado Z., Sculley D., Nowozin S., Dillon J., Lakshminarayanan B., Snoek J. Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing systems. 2019;32.Ovadia Y., Fertig E., Ren J., Nado Z., Sculley D., Nowozin S., Dillon J., Lakshminarayanan B., Snoek J. Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing systems. 2019;32.Намиот Д. Е., Ильюшин Е. А. Мониторинг сдвига данных в моделях машинного обучения. International Journal of Open Information Technologies. 2022;10(12):84–93. EDN: NDOMRK.Namiot D., Ilyushin E. Data shift monitoring in machine learning models. International Journal of Open Information Technologies. 2022;10(12):84–93. (In Russian.) EDN: NDOMRK.Wan J. S.-W., Wang S.-D. Concept drift detection based on pre-clustering and statistical testing. Journal of Internet Technology. 2021;22(2):465–472. DOI: 10.3966/160792642021032202020.Wan J. S.-W., Wang S.-D. Concept drift detection based on pre-clustering and statistical testing. Journal of Internet Technology. 2021;22(2):465–472. DOI: 10.3966/160792642021032202020.Gama J., Žliobaite I., Bifet A., Pechenizkiy M., Bouchachia A. A survey on concept drift adaptation. ACM Computing Surveys (CSUR). 2014;46(4):1–37. DOI: 10.1145/2523813.Gama J., Žliobaite I., Bifet A., Pechenizkiy M., Bouchachia A. A survey on concept drift adaptation. ACM Computing Surveys (CSUR). 2014;46(4):1–37. DOI: 10.1145/2523813.Gama J., Medas P., Castillo G., Rodrigues P. Learning with drift detection. Advances in Artificial Intelligence–SBIA 2004. 2004;3171:286–295. DOI: 10.1007/978-3-540-28645-5_29.Gama J., Medas P., Castillo G., Rodrigues P. Learning with drift detection. Advances in Artificial Intelligence–SBIA 2004. 2004;3171:286–295. DOI: 10.1007/978-3-540-28645-5_29.Baena-Garcıa M., del Campo-Ávila J., Fidalgo R., Bifet A., Gavalda R., Morales-Bueno R. Early drift detection method. Fourth Int. Workshop on Knowledge Discovery from Data Streams. 2006;6:77–86.Baena-Garcıa M., del Campo-Ávila J., Fidalgo R., Bifet A., Gavalda R., Morales-Bueno R. Early drift detection method. Fourth Int. Workshop on Knowledge Discovery from Data Streams. 2006;6:77–86.Liu A., Lu J., Liu F., Zhang G. Accumulating regional density dissimilarity for concept drift detection in data streams. Pattern Recognition. 2018;76:256–272. DOI: 10.1016/j.patcog.2017.11.009.Liu A., Lu J., Liu F., Zhang G. Accumulating regional density dissimilarity for concept drift detection in data streams. Pattern Recognition. 2018;76:256–272. DOI: 10.1016/j.patcog.2017.11.009.Ackerman S., Dube P., Farchi E., Raz O., Zalmanovici M. Machine learning model drift detection via weak data slices. 2021 IEEE/ACM Third Int. Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest). 2021:1–8. DOI: 10.1109/DeepTest52559.2021.00007.Ackerman S., Dube P., Farchi E., Raz O., Zalmanovici M. Machine learning model drift detection via weak data slices. 2021 IEEE/ACM Third Int. Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest). 2021:1–8. DOI: 10.1109/DeepTest52559.2021.00007.Dong S., Wang Q., Sahri S., Palpanas T., Srivastava D. Efficiently mitigating the impact of data drift on machine learning pipelines. Proc. VLDB Endowment. 2024;17(11):3072–3081. DOI: 10.14778/3681954.3681984.Dong S., Wang Q., Sahri S., Palpanas T., Srivastava D. Efficiently mitigating the impact of data drift on machine learning pipelines. Proc. VLDB Endowment. 2024;17(11):3072–3081. DOI: 10.14778/3681954.3681984.Ma S., Tourani R. Predictive and causal implications of using shapley value for model interpretation. Proc. 2020 KDD workshop on causal discovery. 2020:23–38.Ma S., Tourani R. Predictive and causal implications of using shapley value for model interpretation. Proc. 2020 KDD workshop on causal discovery. 2020:23–38.Zheng S., van der Zon S. B., Pechenizkiy M., de Campos C. P., van Ipenburg W., de Harder H., Nederland R. Labelless concept drift detection and explanation. NeurIPS 2019 Workshop on Robust AI in Financial Services: Data, Fairness, Explainability, Trustworthiness, and Privacy. 2019:1–10.Zheng S., van der Zon S. B., Pechenizkiy M., de Campos C. P., van Ipenburg W., de Harder H., Nederland R. Labelless concept drift detection and explanation. NeurIPS 2019 Workshop on Robust AI in Financial Services: Data, Fairness, Explainability, Trustworthiness, and Privacy. 2019:1–10.Сомова М. В. Методическая модель персонифицированного образовательного процесса на основе прогнозирования успешности предметного обучения. Современные наукоемкие технологии. 2023;(12-1):165–170. EDN: IARKPS. DOI: 10.17513/snt.39877.Somova M. V. Methodological model of personalized educational process based on prediction of success in subject training. Modern High Technologies. 2023;(12-1):165–170. (In Russian.) EDN: IARKPS. DOI: 10.17513/snt.39877.Kustitskaya T. A., Esin R. V., Vainshtein Y. V., Noskov M. V. Hybrid approach to predicting learning success based on digital educational history for timely identification of at-risk students. Education Sciences. 2024;14(6):657. DOI: 10.3390/educsci14060657.Kustitskaya T. A., Esin R. V., Vainshtein Y. V., Noskov M. V. Hybrid approach to predicting learning success based on digital educational history for timely identification of at-risk students. Education Sciences. 2024;14(6):657. DOI: 10.3390/educsci14060657.Носков М. В., Сомова М. В., Федотова И. М. Управление успешностью обучения студента на основе марковской модели. Информатика и образование. 2018;(10(299)):4–11. EDN: SMQFKX. DOI: 10.32517/0234-0453-2018-33-10-4-11.Noskov M. V., Somova М. V., Fedotova I. M. Management of the success of student’s learning based on the Markov model. Informatics and Education. 2018;(10(299)):4–11. (In Russian.) EDN: SMQFKX. DOI: 10.32517/0234-0453-2018-33-10-4-11.Носков М. В., Вайнштейн Ю. В., Сомова М. В., Федотова И. М. Прогностическая модель оценки успешности предметного обучения в условиях цифровизации образования. Вестник Российского университета дружбы народов. Серия: Информатизация образования. 2023;20(1):7–19. EDN: BDFDRI. DOI: 10.22363/2312-8631-2023-20-1-7-19.Noskov M. V., Vaynshteyn Yu. V., Somova M. V., Fedotova I. M. Prognostic model for assessing the success of subject learning in conditions of digitalization of education. RUDN Journal of Informatization in Education. 2023;20(1):7–19. (In Russian.) EDN: BDFDRI. DOI: 10.22363/2312-8631-2023-20-1-7-19.Kustitskaya T. A., Esin R. V., Kytmanov A. A., Zykova T. V. Designing an education database in a higher education institution for the data-driven management of the educational process. Education Sciences. 2023;13(9):947. DOI: 10.3390/educsci13090947.Kustitskaya T. A., Esin R. V., Kytmanov A. A., Zykova T. V. Designing an education database in a higher education institution for the data-driven management of the educational process. Education Sciences. 2023;13(9):947. DOI: 10.3390/educsci13090947.Lundberg S. M., Lee S.-I. A unified approach to interpreting model predictions. Neural Information Processing Systems. 2017;30:4768–4777. DOI: 10.48550/arXiv.1705.07874.Lundberg S. M., Lee S.-I. A unified approach to interpreting model predictions. Neural Information Processing Systems. 2017;30:4768–4777. DOI: 10.48550/arXiv.1705.07874.Lundberg S. M., Erion G., Chen H., DeGrave A., Prutkin J. M., Nair B., Katz R., Himmelfarb J., Bansal N., Lee S. I. From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence. 2020;2(1):56–67. DOI: 10.1038/s42256-019-0138-9.Lundberg S. M., Erion G., Chen H., DeGrave A., Prutkin J. M., Nair B., Katz R., Himmelfarb J., Bansal N., Lee S. I. From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence. 2020;2(1):56–67. DOI: 10.1038/s42256-019-0138-9.Cohen J. Statistical power analysis for the behavioral sciences (2nd ed.). New Jersey, Routledge; 1988. 567 p. DOI: 10.4324/9780203771587.Cohen J. Statistical power analysis for the behavioral sciences (2nd ed.). New Jersey, Routledge; 1988. 567 p. DOI: 10.4324/9780203771587.

The authors declare that there are no conflicts of interest present.