A Comprehensive Survey on Gesture-Controlled Interfaces: Technologies, Applications, and Challenges
DOI:
https://doi.org/10.32628/IJSRST251222672Keywords:
Gesture Recognition, Human-Computer Interaction (HCI), User Experience (UX), Computer Vision, Motion TrackingAbstract
Gesture-controlled interfaces are revolutionizing the way humans interact with computers and smart devices by replacing traditional input methods with intuitive body movements. These systems leverage advanced technologies such as computer vision, depth sensing, machine learning, and wearable sensors to accurately interpret user gestures in real time. This survey paper presents a comprehensive exploration of gesture-based interaction systems, beginning with a discussion of the core technologies that enable gesture recognition and tracking. It then delves into various application domains, including gaming, healthcare, robotics, virtual and augmented reality, and smart home environments, highlighting how gesture control enhances usability and user experience in each context. The paper also addresses the major challenges faced in this field, such as gesture ambiguity, environmental sensitivity, computational complexity, and user diversity. Finally, it outlines the latest trends and future research directions, including the integration of AI, multimodal interfaces, and the development of more robust, context-aware systems. By offering an in-depth overview, this survey aims to guide researchers, developers, and industry professionals in understanding current advancements and identifying opportunities for innovation in gesture-based human-computer interaction.
Downloads
References
Devi, M., Saharia, S., Kumar Bhattacharyya, D. et al. MCM-VbF: dance hand gestures recognition with vision based features. Discov Internet Things 4, 18 (2024). https://doi.org/10.1007/s43926-024-00072-7
Sahu, A., Pal, T., Deb, S. (2023). Deep Learning-Based Real-Time Hand Gesture Recognition Using Histogram of Oriented Gradient. In: Das, A.K., Nayak, J., Naik, B., Vimal, S., Pelusi, D. (eds) Computational Intelligence in Pattern Recognition. CIPR 2022. Lecture Notes in Networks and Systems, vol725. Springer, Singapore. https://doi.org/10.1007/978-981-99-3734-9_38
Mukhtar, M.E., Sunar, N., Wahab, N.H.A., Muhammad, N.A., Rahmat, M.F. (2024). Improvement of Vision-Based Hand Gesture Recognition System with Distance Range. In: Hassan, F., Sunar, N., Mohd Basri, M.A., Mahmud, M.S.A., Ishak, M.H.I., Mohamed Ali, M.S. (eds) Methods and Applications for Modeling and Simulation of Complex Systems. AsiaSim 2023. Communications in Computer and Information Science, vol 1911. Springer, Singapore. https://doi.org/10.1007/978-981-99-7240-1_21
Bamani, E., Nissinman, E., Meir, I., Koenigsberg, L. and Sintov, A., 2024. Ultra-range gesture recognition using a web-camera in human-robot interaction. arXiv preprint arXiv:2311.15361. Available at: https://arxiv.org/abs/2311.15361
Linardakis, M., Varlamis, I. and Papadopoulos, G.Th., 2025. Survey on hand gesture recognition from visual input. arXiv preprint arXiv:2501.11992. Available at: https://arxiv.org/abs/2501.11992
Ballow, J.M. and Dey, S., 2022. Real-time hand gesture identification in thermal images. In: Image Analysis and Processing – ICIAP 2022. Springer International Publishing. pp. 491-502. ISBN 9783031064302. DOI: 10.1007/978-3-031-06430-2_41.
Mukhtar, M.E., Sunar, N., Wahab, N.H.A., Muhammad, N.A., Rahmat, M.F. (2024). Improvement of Vision-Based Hand Gesture Recognition System with Distance Range. In: Hassan, F., Sunar, N., Mohd Basri, M.A., Mahmud, M.S.A., Ishak, M.H.I., Mohamed Ali, M.S. (eds) Methods and Applications for Modeling and Simulation of Complex Systems. AsiaSim 2023. Communications in Computer and Information Science, vol 1911. Springer, Singapore. https://doi.org/10.1007/978-981-99-7240-1_21
Aithal, C.N., Ishwarya, P., Sneha, S., Yashvardhan, C.N., Kumar, D., Suresh, K.V. (2022). Hand Gesture Recognition in Complex Background. In: Guru, D.S., Y. H., S.K., K., B., Agrawal, R.K., Ichino, M. (eds) Cognition and Recognition. ICCR 2021. Communications in Computer and Information Science, vol 1697. Springer, Cham. https://doi.org/10.1007/978-3-031-22405-8_19
Uboweja, E., Tian, D., Wang, Q., Kuo, Y.-C., Zou, J., Wang, L., Sung, G. and Grundmann, M., 2023. On-device real-time custom hand gesture recognition. arXiv preprint arXiv:2309.10858. Available at: https://arxiv.org/abs/2309.10858
Kadavath, M.R.K.; Nasor, M.; Imran, A. Enhanced Hand Gesture Recognition with Surface Electromyogram and Machine Learning. Sensors 2024, 24, 5231. https://doi.org/10.3390/s24165231
Miah, Abu Saleh Musa, et al. "EMG-Based Hand Gesture Recognition through Diverse Domain Feature Enhancement and Machine Learning-Based Approach." arXiv preprint arXiv:2408.13723 (2024).
Bhattacharyya, P., Mitton, J., Page, R., Morgan, O., Powell, O., Menzies, B., Homewood, G., Jacobs, K., Baesso, P., Muhonen, T., Vigars, R. and Berridge, L., 2025. Helios 2.0: A robust, ultra-low power gesture recognition system optimised for event-sensor based wearables. arXiv preprint arXiv:2503.07825. Available at: https://arxiv.org/abs/2503.07825
Montazerin, M., Zabihi, S., Rahimian, E., Mohammadi, A. and Naderkhani, F., 2022. ViT-HGR: Vision Transformer-based hand gesture recognition from high density surface EMG signals. arXiv preprint arXiv:2201.10060. Available at: https://arxiv.org/abs/2201.10060
Ceolini, E., Taverni, G., Khacef, L., Payvand, M. and Donati, E., 2019. Sensor fusion using EMG and vision for hand gesture classification in mobile applications. arXiv preprint arXiv:1910.11126. Available at: https://arxiv.org/abs/1910.11126
Uboweja, E., Tian, D., Wang, Q., Kuo, Y.-C., Zou, J., Wang, L., Sung, G. and Grundmann, M., 2023. On-device real-time custom hand gesture recognition. arXiv preprint arXiv:2309.10858. Available at: https://arxiv.org/abs/2309.10858
Sen, A., Mishra, T.K. and Dash, R., 2023. Deep learning based hand gesture recognition system and design of a human-machine interface. arXiv preprint arXiv:2207.03112. Available at: https://arxiv.org/abs/2207.03112
Hashi, A., Hashim, S.M. and Asamah, A., 2024. A systematic review of hand gesture recognition: An update from 2018 to 2024. IEEE Access, PP, pp.1–1. DOI: 10.1109/ACCESS.2024.3421992.
Sahu, A., Pal, T., Deb, S. (2023). Deep Learning-Based Real-Time Hand Gesture Recognition Using Histogram of Oriented Gradient. In: Das, A.K., Nayak, J., Naik, B., Vimal, S., Pelusi, D. (eds) Computational Intelligence in Pattern Recognition. CIPR 2022. Lecture Notes in Networks and Systems, vol 725. Springer, Singapore. https://doi.org/10.1007/978-981-99-3734-9_38
Cunico, F., Girella, F., Avogaro, A., Emporio, M., Giachetti, A. and Cristani, M., 2023. OO-dMVMT: A deep multi-view multi-task classification framework for real-time 3D hand gesture classification and segmentation. arXiv preprint arXiv:2304.05956. Available at: https://arxiv.org/abs/2304.05956
Allemand, F., Mazzela, A., Villette, J., Aspandi, D. and Zaharia, T., 2024. Deep self-supervised learning with visualisation for automatic gesture recognition. arXiv preprint arXiv:2406.12440. Available at: https://arxiv.org/abs/2406.12440
Subudhi, B.N., Veerakumar, T., Harathas, S.R., Prabhudesai, R., Kuppili, V., Jakhetiya, V. (2023). Deep Learning in Autoencoder Framework and Shape Prior for Hand Gesture Recognition. In: Kumar, B.V., Sivakumar, P., Surendiran, B., Ding, J. (eds) Smart Computer Vision. EAI/Springer Innovations in Communication and Computing. Springer, Cham. https://doi.org/10.1007/978-3-031-20541-5_10
Hu, H., Zhao, W., Zhou, W., & Li, H., 2023. SignBERT+: Hand-Model-Aware Self-Supervised Pre-Training for Sign Language Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9), pp. 11221–11239. Available at: http://dx.doi.org/10.1109/TPAMI.2023.3269220
Bhardwaj, K., Bansal, S., Ghosh, K. (2023). A Machine Learning-Based Approach to Identify Hand Gestures. In: Chakraborty, B., Biswas, A., Chakrabarti, A. (eds) Advances in Data Science and Computing Technologies. ADSC 2022. Lecture Notes in Electrical Engineering, vol 1056. Springer, Singapore. https://doi.org/10.1007/978-981-99-3656-4_77
Garg, M., Ghosh, D. & Pradhan, P.M., 2023. Multiscaled Multi-Head Attention-Based Video Transformer Network for Hand Gesture Recognition. IEEE Signal Processing Letters, 30, pp.80–84. Available at: http://dx.doi.org/10.1109/LSP.2023.3241857
Rahim, M.A., Miah, A.S.M., Akash, H.S., Shin, J., Hossain, M.I. & Hossain, M.N., 2024. An Advanced Deep Learning Based Three-Stream Hybrid Model for Dynamic Hand Gesture Recognition. arXiv preprint arXiv:2408.08035. Available at: https://arxiv.org/abs/2408.08035
Islam, M.R., Rahman, R., Ahmed, A. and Jany, R., 2022. NFS: A hand gesture recognition based game using MediaPipe and pygame. arXiv preprint arXiv:2204.11119.
Hand Gesture Recognition for Video Player” (2024) International Research Journal on Advanced Engineering Hub (IRJAEH), 2(04), pp. 801–805.
Sen, A., Mishra, T.K. and Dash, R., 2024. Novel human machine interface via robust hand gesture recognition system using channel pruned YOLOv5s model. arXiv preprint arXiv:2407.02585.
Chen J, Zhao S, Meng H, Cheng X, Tan W. An interactive game for rehabilitation based on real-time hand gesture recognition. Front Physiol. 2022 Oct
Husna R, Brata KC, Anggraini IT, Funabiki N, Rahmadani AA, Fan C-P. An Investigation of Hand Gestures for Controlling Video Games in a Rehabilitation Exergame System. Computers. 2025; 14(1):25. https://doi.org/10.3390/computers14010025
Barona López, L.I., León Cifuentes, C.I., Muñoz Oña, J.M., Valdivieso Caraguay, A.L. and Benalcázar, M.E., 2024. Hand gesture recognition applied to the interaction with video games. In: Calvo, H., Martínez-Villaseñor, L. and Ponce, H. (eds.) Advances in Computational Intelligence. MICAI 2023. Lecture Notes in Computer Science, vol. 14391. Cham: Springer. Available at: https://doi.org/10.1007/978-3-031-47765-2_3 [Accessed 24 Apr. 2025].
Wang, J., 2024. Application of artificial intelligence in hand gesture recognition with virtual reality: survey and analysis of hand gesture hardware selection. arXiv preprint arXiv:2405.16264. Available at: https://arxiv.org/abs/2405.16264 [Accessed 24 Apr. 2025].
Park, B.J., Jang, T., Choi, J.W. and Kim, N., 2016. Gesture-controlled interface for contactless control of various computer programs with a hooking-based keyboard and mouse-mapping technique in the operating room. Computers, Mathematics and Methods in Medicine, 2016, p.5170379. Available at: https://doi.org/10.1155/2016/5170379 [Accessed 24 Apr. 2025].
Rossol, N., Cheng, I., Shen, R. and Basu, A., 2014. Touchfree medical interfaces. Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2014, pp. 6597-6600. Available at: https://doi.org/10.1109/EMBC.2014.6945140 [Accessed 24 Apr. 2025].
Jacob, M.G., Wachs, J.P. and Packer, R.A., 2013. Hand-gesture-based sterile interface for the operating room using contextual cues for the navigation of radiological images. Journal of the American Medical Informatics Association, 20(e1), pp. e183-186. Available at: https://doi.org/10.1136/amiajnl-2012-001212 [Accessed 24 Apr. 2025].
Feied, C., Gillam, M., Wachs, J., Handler, J., Stern, H. and Smith, M., 2006. A real-time gesture interface for hands-free control of electronic medical records. AMIA Annual Symposium Proceedings, 2006, p. 920. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1839426/ [Accessed 24 Apr. 2025].
Sadi MS, Alotaibi M, Islam MR, Islam MS, Alhmiedat T, Bassfar Z. Finger-Gesture Controlled Wheelchair with Enabling IoT. Sensors. 2022; 22(22):8716. https://doi.org/10.3390/s22228716
van Amsterdam, B., Clarkson, M.J. and Stoyanov, D., 2021. Gesture recognition in robotic surgery: A review. IEEE Transactions on Biomedical Engineering, 68(6), pp.2021–2035. Available at: https://doi.org/10.1109/TBME.2021.3054828 [Accessed 24 Apr. 2025].
“The Role of Gesture-Based Interaction in Improving User Satisfaction for Touchless Interfaces” (2024) International Journal of Advanced Human Computer Interaction, 2(2), pp. 20–32. Available at: https://www.ijahci.com/index.php/ijahci/article/view/17.
Ji, B., Wang, X., Liang, Z., Zhang, H., Xia, Q., Xie, L., Yan, H., Sun, F., Feng, H., Tao, K., Shen, Q. and Yin, E., 2024. Flexible strain sensor-based data glove for gesture interaction in the Metaverse: A review. International Journal of Human–Computer Interaction, 40(21), pp.6793–6812. Available at: https://doi.org/10.1080/10447318.2024.2331234 [Accessed 24 Apr. 2025].
Park, B.J., Jang, T., Choi, J.W. and Kim, N., 2016. Gesture-controlled interface for contactless control of various computer programs with a hooking-based keyboard and mouse-mapping technique in the operating room. Computers, Mathematics and Methods in Medicine, 2016, p.5170379. Available at: https://doi.org/10.1155/2016/5170379 [Accessed 24 Apr. 2025].
Alabdullah BI, Ansar H, Mudawi NA, Alazeb A, Alshahrani A, Alotaibi SS, Jalal A. Smart Home Automation-Based Hand Gesture Recognition Using Feature Fusion and Recurrent Neural Network. Sensors. 2023; 23(17):7523.
Fatima, B., Mushtaq, B., Iqbal, M. A., & Ahmed, A. (2024). IoT-based Smart Home Automation Using Gesture Control and Machine Learning for Individuals with Auditory Challenges. IECE Transactions on Internet of Things, 2(4), 74–82. https://doi.org/10.62762/TIOT.2024.723193
Suzuki, Y., Kato, K., Furui, N., Sakamoto, D. and Sugiura, Y., 2024. Exploring gestural interaction with a cushion interface for smart home control. arXiv preprint arXiv:2410.04730. Available at: https://arxiv.org/abs/2410.04730 [Accessed 24 Apr. 2025].
Li, Y., Zhang, D., Chen, J., Wan, J., Zhang, D., Hu, Y., Sun, Q. and Chen, Y., 2022. Towards domain-independent and real-time gesture recognition using mmWave signal. IEEE Transactions on Mobile Computing, pp.1–15. Available at: https://doi.org/10.1109/TMC.2022.3207570 [Accessed 24 Apr. 2025].
Karuna, M., Anandababu, Y., Thonangi, C., Sumala, C.N. and Zafreen, S., n.d. Gesture based home automation.
Małecki K, Nowosielski A, Kowalicki M. Gesture-Based User Interface for Vehicle On-Board System: A Questionnaire and Research Approach. Applied Sciences. 2020; 10(18):6620. https://doi.org/10.3390/app10186620
Khan, F., Leem, S.K. and Cho, S.H., 2017. Hand-based gesture recognition for vehicular applications using IR-UWB radar. Sensors, 17(4), p.833. Available at: https://doi.org/10.3390/s17040833 [Accessed 24 Apr. 2025].
Zheng, L., Bai, J., Zhu, X., Huang, L., Shan, C., Wu, Q. and Zhang, L., 2021. Dynamic hand gesture recognition in in-vehicle environment based on FMCW radar and transformer. Sensors, 21(19), p.6368. Available at: https://doi.org/10.3390/s21196368 [Accessed 24 Apr. 2025].
Yang, K., Kim, M., Jung, Y. and Lee, S., 2024. Hand gesture recognition using FSK radar sensors. Sensors, 24(2), p.349. Available at: https://doi.org/10.3390/s24020349 [Accessed 24 Apr. 2025].
Young, G., Milne, H., Griffiths, D., Padfield, E., Blenkinsopp, R. and Georgiou, O., 2020. Designing mid-air haptic gesture controlled user interfaces for cars. Proceedings of the ACM on Human-Computer Interaction, 4(EICS), pp.1–23. Available at: https://doi.org/10.1145/3397869 [Accessed 24 Apr. 2025].
Lee, S.H. and Yoon, S., 2020. User interface for in-vehicle systems with on-wheel finger spreading gestures and head-up displays. Journal of Computational Design and Engineering, 7(6), pp.700–721. Available at: https://doi.org/10.1093/jcde/qwaa052 [Accessed 24 Apr. 2025].
Małecki K, Nowosielski A, Kowalicki M. Gesture-Based User Interface for Vehicle On-Board System: A Questionnaire and Research Approach. Applied Sciences. 2020; 10(18):6620.
Zengeler, N., Kopinski, T. and Handmann, U., 2018. Hand gesture recognition in automotive human–machine interaction using depth cameras. Sensors, 19(1), p.59. Available at: https://doi.org/10.3390/s19010059 [Accessed 24 Apr. 2025].
Young, G., Milne, H., Griffiths, D., Padfield, E., Blenkinsopp, R. and Georgiou, O., 2020. Designing mid-air haptic gesture controlled user interfaces for cars. Proceedings of the ACM on Human-Computer Interaction, 4, pp.1–23. Available at: https://doi.org/10.1145/3397869 [Accessed 24 Apr. 2025].
Blessing, M., 2024. Applications of hand-sign recognition in assistive technologies. [online] ResearchGate. Available at: https://www.researchgate.net/publication/386177371_Applications_of_Hand-Sign_Recognition_in_Assistive_Technologies [Accessed 24 Apr. 2025].
Siddique, A., Rehna, J. and Naik, S., 2024. A survey on gesture control techniques for smart object interaction in disability support.8, pp.94–100. [Accessed 24 Apr. 2025].
Alashhab, S., Gallego, A.J. and Lozano, M.Á., 2022. Efficient gesture recognition for the assistance of visually impaired people using multi-head neural networks. Engineering Applications of Artificial Intelligence, 114, p.105188. Available at: https://doi.org/10.1016/j.engappai.2022.105188 [Accessed 24 Apr. 2025].
Rúbia E. O. Schultz Ascari, Luciano Silva, and Roberto Pereira. 2019. Personalized interactive gesture recognition assistive technology. In Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems (IHC '19). Association for Computing Machinery, New York, NY, USA, Article 38, 1–12. https://doi.org/10.1145/3357155.3358442
Erickson, Z., 2024. High-density electromyography for effective gesture-based control of physically assistive mobile manipulators. arXiv. Available at: https://arxiv.org/abs/2312.07745 [Accessed 24 Apr. 2025].
Seshan, S., 2020. Lightweight assistive technology: A wearable, optical-fiber gesture recognition system. arXiv. Available at: https://arxiv.org/abs/2009.13322 [Accessed 24 Apr. 2025].
Nelson, A. H. (2013). Gesture Based Home Automation for the Physically Disabled. Graduate Theses and Dissertations Retrieved from https://scholarworks.uark.edu/etd/720
Mohamed, N., 2025. Eye-gesture control of computer systems via artificial intelligence. F1000Research, 13, p.109. Available at: https://doi.org/10.12688/f1000research.144962.3 [Accessed 24 Apr. 2025].
Sharma, R., 2023. Hand gesture recognition systems: As assistive technology to control machines. Master's thesis. California State University, Northridge. Available at: http://localhost/files/st74cz126 [Accessed 24 Apr. 2025].
Alashhab, S., Gallego, A.J. and Lozano, M.Á., 2022. Efficient gesture recognition for the assistance of visually impaired people using multi-head neural networks. Engineering Applications of Artificial Intelligence, 114, p.105188. Available at: https://doi.org/10.1016/j.engappai.2022.105188 [Accessed 24 Apr. 2025].
Downloads
Published
Issue
Section
License
Copyright (c) 2025 International Journal of Scientific Research in Science and Technology
This work is licensed under a Creative Commons Attribution 4.0 International License.
