infojournalИнформатика и образованиеInformatics and education0234-04532658-7769Publishing House Education and Informatics10.32517/0234-0453-2023-38-1-55-63infojournal-921Research ArticleКОНКУРС ИНФО-2022INFO-2022 CONTESTУчебно-методические материалы мастер-класса «Состязательные атаки на нейронные сети распознавания изображений» для студентов и школьниковEducational and methodological materials of the master class “Adversarial attacks on image recognition neural networks” for students and schoolchildrenhttps://orcid.org/0000-0003-2088-0474ПантюхинД. В.PantiukhinD. V.

Пантюхин Дмитрий Валерьевич, старший преподаватель департамента программной инженерии, факультет компьютерных наук; старший преподаватель кафедры «Интеллектуальные системы информационной безопасности», Институт кибербезопасности ицифровых технологий

Москва

Dmitry V. Pantiukhin, Senior Lecturer at School of Software Engineering, Faculty of Computer Science; Senior Lecturer at the Department of Intelligent Systems for Information Security, Institute for Cybersecurity and Digital Technologies

Moscow

dpantiukhin@hse.ruНациональный исследовательский университет «Высшая школа экономики»; МИРЭА  — Российский технологический университетРоссияNational Research University Higher School of Economics; MIREA  — Russian Technological UniversityRussian Federation2023160420233815563Copyright © Пантюхин Д.В., 20232023Пантюхин Д.В.Pantiukhin D.V.This work is licensed under a Creative Commons Attribution 4.0 License.https://info.infojournal.ru/jour/article/view/921

Проблема уязвимости нейронных сетей уже несколько лет является предметом научных исследований и экспериментов. Состязательные атаки — один из способов «обмануть» нейросеть, заставить ее принимать ошибочные классификационные решения. Сама возможность состязательной атаки кроется в особенностях машинного обучения нейросетей.

В статье показано, как свойства нейронных сетей становятся источником проблем и ограничений в их использовании. Материалы соответствующих исследований автора легли в основу мастер-класса «Состязательные атаки на нейронные сети распознавания изображений». В статье представлены учебно-методические материалы мастер-класса: теоретические основы занятия, практические материалы (в частности, описана атака на один нейрон, рассмотрен метод быстрого знака градиента для атаки на нейронную сеть), примеры экспериментов и расчетов (автор использует сверточную сеть VGG, библиотеки Torch и CleverHans), а также набор типовых ошибок обучающихся и пояснения педагога, устраняющие эти ошибки. Кроме того, в статье приведен результат эксперимента, а по указанным ссылкам доступны его полный код и примеры апробации материалов мастер-класса.

Мастер-класс предназначен как для студентов, так и для старших школьников, изучивших основы нейронных сетей и язык Python, а также может представлять практический интерес для учителей информатики, разработчиков курсов по машинному обучению и искусственному интеллекту и преподавателей вузов.

The problem of neural network vulnerability has been the subject of scientific research and experiments for several years. Adversarial attacks are one of the ways to “trick” a neural network, to force it to make incorrect classification decisions. The very possibility of adversarial attack lies in the peculiarities of machine learning of neural networks. The article shows how the properties of neural networks become a source of problems and limitations in their use. The materials of the corresponding researches of the author were used as a basis for the master class “Adversarial attacks on image recognition neural networks”.

The article presents the educational materials of the master class: the theoretical background of the class, practical materials (in particular, the attack on a single neuron is described, the fast gradient sign method for attacking a neural network is considered), examples of experiments and calculations (the author uses the convolutional network VGG, Torch and CleverHans libraries), as well as a set of typical errors of students and the teacher’s explanations of how to eliminate these errors. In addition, the result of the experiment is given in the article, and its full code and examples of approbation of the master class materials are available at the above links.

The master class is intended for both high school and university students who have learned the basics of neural networks and the Python language, and can also be of practical interest to computer science teachers, to developers of courses on machine learning and artificial intelligence as well as to university teachers.

нейронная сетьсостязательная атакаметод быстрого знака градиентагиперпараметры атакимастер-классneural networkadversarial attackfast gradient sign methodhyperparameters of attackmaster classАвтор выражает глубокую благодарность генеральному директору ООО «Лаборатория Наносемантика» Станиславу Игоревичу Ашманову, при поддержке которого были созданы упомянутый курс машинного обучения для школьников и данный мастер-класс, а также всем студентам, которые вычитывали материалы мастер-класса, коллегам, которые давали дельные советы, и своему учителю Александру Ивановичу Галушкину за то, что научил обучать нейронные сети.The author expresses his sincere gratitude to Stanislav Igorevich Ashmanov, the General Director of Nanosemantics Laboratory LLC, with whose support the mentioned machine learning course for schoolchildren and this master class were created. The author is also grateful to all students who proofread the materials of the master class, to the colleagues who gave useful advice and to his teacher Alexander Galushkin for teaching to train neural networks.ReferencesMarques L. S., Gresse von Wangenheim C., Hauck J. C. R. Teaching machine learning in school: A systematic mapping of the state of the art. Informatics in Education. 2020;19(2):283–321. DOI: 10.15388/infedu.2020.14Marques L. S., Gresse von Wangenheim C., Hauck J. C. R. Teaching machine learning in school: A systematic mapping of the state of the art. Informatics in Education. 2020;19(2):283–321. DOI: 10.15388/infedu.2020.14Briot J.-P., Hadjeres G., Pachet F.-D. Deep learning techniques for music generation. Cham, Springer; 2020. 284 p. DOI: 10.1007/978-3-319-70163-9Briot J.-P., Hadjeres G., Pachet F.-D. Deep learning techniques for music generation. Cham, Springer; 2020. 284 p. DOI: 10.1007/978-3-319-70163-9Ormazabal A., Artetxe M., Agirrrezabal M., Soroa A., Agirre E. PoeLM: A meter-and rhyme-controllable language model for unsupervised poetry generation. Findings of the Association for Computational Linguistics: EMNLP 2022. Abu Dhabi, Association for Computational Linguistics; 2022:3655–3670. DOI: 10.48550/arXiv.2205.12206Ormazabal A., Artetxe M., Agirrrezabal M., Soroa A., Agirre E. PoeLM: A meter-and rhyme-controllable language model for unsupervised poetry generation. Findings of the Association for Computational Linguistics: EMNLP 2022. Abu Dhabi, Association for Computational Linguistics; 2022:3655–3670. DOI: 10.48550/arXiv.2205.12206Пантюхин Д. В. Нейронные сети синтеза речи голосовых помощников и поющих автоматов. Речевые технологии. 2021;(3-4):3–16. EDN: PEDILK. DOI: 10.58633/2305-8129_2021_3-4_3Pantiukhin D. V. Neural networks for speech synthesis of voice assistants and singing machines. Speech Technology. 2021;(3-4):3–16. (In Russian.) EDN: PEDILK. DOI: 10.58633/2305-8129_2021_3-4_3Zhu J., Villareale J., Javvaji N., Risi S., Löwe M., Weigelt R., Harteveld C. Player-AI interaction: What neural network games reveal about AI as play. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. New York, Association for Computing Machinery; 2021:1–17. DOI: 10.1145/3411764.3445307Zhu J., Villareale J., Javvaji N., Risi S., Löwe M., Weigelt R., Harteveld C. Player-AI interaction: What neural network games reveal about AI as play. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. New York, Association for Computing Machinery; 2021:1–17. DOI: 10.1145/3411764.3445307Long T., Gao Q., Xu L., Zhou Z. A survey on adversarial attacks in computer vision: Taxonomy, visualization and future directions. Computers and Security. 2022;121:102847. DOI: 10.1016/j.cose.2022.102847Long T., Gao Q., Xu L., Zhou Z. A survey on adversarial attacks in computer vision: Taxonomy, visualization and future directions. Computers and Security. 2022;121:102847. DOI: 10.1016/j.cose.2022.102847Chakraborty A., Alam M., Dey V., Chattopadhyay A., Mukhopadhyay D. A survey on adversarial attacks and defences. CAAI Transactions on Intelligence Technology. 2021;6(1):25–45. DOI: 10.1049/cit2.12028Chakraborty A., Alam M., Dey V., Chattopadhyay A., Mukhopadhyay D. A survey on adversarial attacks and defences. CAAI Transactions on Intelligence Technology. 2021;6(1):25–45. DOI: 10.1049/cit2.12028Николенко С., Кадурин А., Архангельская Е. Глубокое обучение. Погружение в мир нейронных сетей. СПб.: Питер; 2018. 480 с.Nikolenko S., Kadurin A., Arkhangelskaya E. Deep learning. Immersion into the world of neural networks. Saint Petersburg, Piter; 2018. 480 p. (In Russian.)Гудфеллоу Я., Бенджио И., Курвилль А. Глубокое обучение. М.: ДМК-Пресс; 2018. 652 с.Goodfellow I., Bengio Y., Courville A. Deep learning. Moscow, DMK-Press; 2018. 652 p. (In Russian.)Джулли А., Пал С. Библиотека Keras — инструмент глубокого обучения. Реализация нейронных сетей с помощью библиотек Theano и TensorFlow. М.: ДМК-Пресс; 2017. 296 с.Gulli A., Pal S. Deep Learning with Keras. Implement neural networks with Keras on Theano and TensorFlow. Moscow, DMK-Press; 2017. 296 p. (In Russian.)Пойнтер Я. Программируем с PyTorch: Создание приложений глубокого обучения. СПб.: Питер; 2020. 256 с.Pointer I. Programming PyTorch for Deep Learning: Creating and Deploying Deep Learning Applications. Saint Petersburg, Piter; 2020. 256 p. (In Russian.)Ketkar N., Moolayil J. Deep learning with Python. Learn best practices of deep learning models with PyTorch. Berkeley, Apress; 2021. 306 p.Ketkar N., Moolayil J. Deep learning with Python. Learn best practices of deep learning models with PyTorch. Berkeley, Apress; 2021. 306 p.Khan A., Sohail A., Zahoora U., Qureshi A. S. A survey of the recent architectures of deep convolutional neural networks. Artificial Intelligence Review. 2020;53(8):5455–5516. DOI: 10.48550/arXiv.1901.06032Khan A., Sohail A., Zahoora U., Qureshi A. S. A survey of the recent architectures of deep convolutional neural networks. Artificial Intelligence Review. 2020;53(8):5455–5516. DOI: 10.48550/arXiv.1901.06032Hussain M., Bird J. J., Faria D. R. A study on CNN transfer learning for image classification. Advances in Computational Intelligence Systems. UKCI 2018. Advances in Intelligent Systems and Computing. Cham, Springer; 2019;840:191–202. DOI: 10.1007/978-3-319-97982-3_16Hussain M., Bird J. J., Faria D. R. A study on CNN transfer learning for image classification. Advances in Computational Intelligence Systems. UKCI 2018. Advances in Intelligent Systems and Computing. Cham, Springer; 2019;840:191–202. DOI: 10.1007/978-3-319-97982-3_16Gao B., Pavel L. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint. 2017:1–10. DOI: 10.48550/arXiv.1704.00805Gao B., Pavel L. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint. 2017:1–10. DOI: 10.48550/arXiv.1704.00805Simonyan K., Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint. 2015:1–14. DOI: 10.48550/arXiv.1409.1556Simonyan K., Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint. 2015:1–14. DOI: 10.48550/arXiv.1409.1556Kianpour M., Wen S.-F. Timing attacks on machine learning: state of the art. Advances in Intelligent Systems and Computing. 2020;1037:111–125. DOI: 10.1007/978-3-030-29516-5_10Kianpour M., Wen S.-F. Timing attacks on machine learning: state of the art. Advances in Intelligent Systems and Computing. 2020;1037:111–125. DOI: 10.1007/978-3-030-29516-5_10Goodfellow I. J., Shlens J., Szegedy Ch. Explaining and harnessing adversarial examples. arXiv preprint. 2015:1–11. DOI: 10.48550/arXiv.1412.6572Goodfellow I. J., Shlens J., Szegedy Ch. Explaining and harnessing adversarial examples. arXiv preprint. 2015:1–11. DOI: 10.48550/arXiv.1412.6572Carlini N., Wagner D. Towards evaluating the robustness of neural networks. 2017 IEEE Symposium on Security and Privacy. San Jose, 2017:39–57. DOI: 10.1109/SP.2017.49Carlini N., Wagner D. Towards evaluating the robustness of neural networks. 2017 IEEE Symposium on Security and Privacy. San Jose, 2017:39–57. DOI: 10.1109/SP.2017.49Wiyatno R., Xu A. Maximal Jacobian-based saliency map attack. arXiv preprint. 2018:1–5. DOI: 10.48550/arXiv.1808.07945Wiyatno R., Xu A. Maximal Jacobian-based saliency map attack. arXiv preprint. 2018:1–5. DOI: 10.48550/arXiv.1808.07945Eykholt K., Evtimov I., Fernandes E., Li B., Rahmati A., Xiao Ch., Prakash A., Kohno T., Song D. Robust physicalworld attacks on deep learning visual classification. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition. 2018:1625–1634. DOI: 10.1109/CVPR.2018.00175Eykholt K., Evtimov I., Fernandes E., Li B., Rahmati A., Xiao Ch., Prakash A., Kohno T., Song D. Robust physicalworld attacks on deep learning visual classification. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition. 2018:1625–1634. DOI: 10.1109/CVPR.2018.00175Kurakin A., Goodfellow I. J., Bengio S. Adversarial examples in the physical world. Artificial intelligence safety and security. New York, Chapman and Hall/CRC; 2018:99–112. DOI: DOI:10.1201/9781351251389-8 Available at: https://arxiv.org/pdf/1607.02533.pdfKurakin A., Goodfellow I. J., Bengio S. Adversarial examples in the physical world. Artificial intelligence safety and security. New York, Chapman and Hall/CRC; 2018:99–112. DOI: DOI:10.1201/9781351251389-8 Available at: https://arxiv.org/pdf/1607.02533.pdfAkhtar N., Mian A. Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access. 2018;6:14410–14430. DOI: 10.1109/ACCESS.2018.2807385Akhtar N., Mian A. Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access. 2018;6:14410–14430. DOI: 10.1109/ACCESS.2018.2807385

The authors declare that there are no conflicts of interest present.