Modern approaches to the organization of the technological process of big data collection and preparation and related new professions in the IT industry: DevOps-engineer and data engineer
https://doi.org/10.32517/0234-0453-2024-39-5-21-39
Abstract
The article modern technological approaches to the organization of big data collection and preparation, as well as the distribution and use of applications with the help of the Docker platform, loading and launching a set of applications on the basis of the container approach to virtualization, are considered. The basics of big data processing task formulation and technical problems of technical platform scaling and data quality assurance are briefly outlined. On the example of deploying and using PostgreSQL database management system with management tools and Apache Superset analytics platform, a modern approach to organizing data acquisition, transformation and subsequent demonstration in the form of a dashboard is presented. Work with data is shown on the example of a real open data set processing cycle with demonstration of all the main stages — loading, cleaning, preliminary analysis and visualization. In the context of the considered technologies and approaches to workflow organization in the form of a pipeline, examples of DevOps engineer and data engineer work tasks are given, the necessity of their work as part of a team is emphasized. The technical solutions demonstrated in the article can also be used in the study of the topics “Databases”, “Infographics”, and “Data analysis” in the advanced informatics course of technological profile as an illustration of the complexity of the relevant processes and in the training of future informatics teachers.
Keywords
About the Authors
N. N. SamylkinaRussian Federation
Nadezhda N. Samylkina, Doctor of Sciences (Education),
Docent, Professor at the Department of Theory and Methodology of Informatics Education, Institute of Mathematics andInformatics
I. A. Kalinin
Russian Federation
Ilya A. Kalinin, Candidate of Sciences (Education), Docent, Head of Informatization Administration, the Department of Digital Development, Moscow State Linguistic University Associate Professor at the Department of International Information Security, Institute of Information Sciences
References
1. Samylkina N. N. Informatics, advanced level. Realiza tion of the requirements of the FSES of secondary general education: A methodological manual for teachers. Moscow, Institute for Strategy of Education Development; 2023. 226 p. (In Russian.) EDN: LGYRRI.
2. Lee S. J., Kwon K. A systematic review of AI educa tion in K-12 classrooms from 2018 to 2023: Topics, strate gies, and learning outcomes. Computers and Education: Artificial Intelligence. 2024;6:100211. DOI: 10.1016/j.caeai.2024.100211.
3. Israel-Fishelson R., Moon P. F., Tabak R., Weintrop D. Preparing students to meet their data: An evaluation of K-12 data science tools. Behaviour & Information Technology. 2023:1–20. DOI: 10.1080/0144929X.2023.2295956.
4. Hazzan O., Ragonis N., Lapidot T. Data science and computer science education. Guide to Teaching Computer Science. Springer, Cham; 2020:95–117. DOI: 10.1007/978-3-030-39360-1_6.
5. Foundations of data science for students in grades K-12: Proceedings of a workshop. Washington, DC, The National Academies Press; 2023. 152 p. DOI: 10.17226/26852.
6. Tkach T. V. Machine learning and Big Data processing in a modern school. Informatics in School. 2020;(7(160)):25–29. (In Russian.) EDN: JETIPH. DOI: 10.32517/2221-1993-2020-19-7-25-29.]
7. Samylkina N. N., Salakhova A. A. Teaching the basics of artificial intelligence and data analysis in the informatics course at the secondary general education level: Monograph. Moscow, MPSU; 2022. 242 p. (In Russian.) EDN: BACMCW. DOI: 10.31862/9785426310643.
8. Lane D. Machine learning for children. A practical introduction to artificial intelligence. Moscow, Laboratoriya Znaniy; 2023. 288 p. (In Russian.)
9. Lynch C. How do your data grow? Nature. 2008;455(7209):28–29. DOI: 10.1038/455028a.
10. Kornev M. S. History of the term “Big Data”. Dictiona ries, academic and business periodicals. Vestnik RGGU. Seriya: Istoriya. Filologiya. Kul’turologiya. Vostokovedeniye. 2018;(1(34)):81–85. (In Russian.) EDN: YXCBFP. DOI: 10.28995/2073-6355-2018-1-81-85.
11. Bessonov V. A. What opportunities do new technologies bring about for price statistics? Den’gi i Credit. 2021;80(1):120–126. (In Russian.) EDN: NCADLU. DOI: 10.31477/rjmf.202101.120.]
12. Saly V. V., Kuharenko L. V., Ischenko O. V. Digital transformation of the economy and implementation of Big Data storage in company infrastructure. Vestnik Akademii Znaniy. 2021;44(3):208–214. (In Russian.) EDN: FIQRSU. DOI: 10.24412/2304-6139-2021-11240.
13. Smirnov D. V., Grusho N. A., Zabezhailo M. I., Timoni na E. E. System for collecting and analyzing information from various sources in Big Data conditions. International Journal of Open Information Technologies. 2021;(9(4)):64–71. (In Russian.) EDN: VNVDNQ.
14. Menshchikov A. A., Perfiliev V. E., Fedosenko M. Yu., Fabziev I. R. The main problems of use of Big Data in modern information systems. Stolypin Annals. 2022;4(1):30. (In Russian.) EDN: NDZPQO.
15. Reis D., Piedade B., Correia F. F., Dias J. P., Aguiar A. Developing docker and docker-compose specifications: A developers’ survey. IEEE Access. 2022;10:2318–2329. EDN: JWRGVP. DOI: 10.1109/access.2021.3137671.
16. Egorov V. B. Some issues of software-defined datacenters. Systems and Means of Informatics. 2020;30(2):103–112. (In Russian.) EDN: FXXGJY. DOI: 10.14357/08696527200210.
17. Roy D., Dutta M. A systematic review and research perspective on recommender systems. Journal of Big Data. 2022;9:59. DOI: 10.1186/s40537-022-00592-5.
18. He Q., Li X., Cai B. Graph neural network recommenda tion algorithm based on improved dual tower model. Scientific Reports. 2024;14:3853. DOI: 10.1038/s41598-024-54376-3.
19. Broussard M. Artificial intelligence: The limits of the possible. Moscow, Alpina non-fiction; 2020. 362 p. (In Russian.)
20. Makshanov A. V., Zhuravlev A. E., Tyndykar’ L. N. Big Data: textbook for SPE. Saint Petersburg, Lan’; 2022. 188 p. (In Russian.)
21.
Review
For citations:
Samylkina N.N., Kalinin I.A. Modern approaches to the organization of the technological process of big data collection and preparation and related new professions in the IT industry: DevOps-engineer and data engineer. Informatics and education. 2024;39(5):21-39. (In Russ.) https://doi.org/10.32517/0234-0453-2024-39-5-21-39