Artificial Intelligence in Musical Instrument Learning: A Structured Review of Audio Signal Processing, Computer Vision, and Personalized Feedback
Article Sidebar
Main Article Content
Abstract
Artificial intelligence (AI) is increasingly used to support learners who practise a musical instrument without a teacher present. This structured narrative review examines how audio signal processing, computer vision, and their combination are used to assess instrumental performance, and how the resulting assessments are turned into personalized feedback. Literature was identified through Google Scholar and publisher databases, the ISMIR and NIME proceedings archives, and backward and forward citation chasing, with a final search in October 2026; 45 sources were retained and synthesized thematically against six research questions. Audio methods for monophonic pitch and onset estimation are mature and well benchmarked, whereas polyphonic transcription and guitar-specific string–fret estimation remain difficult. Camera-based hand-landmark tracking is now accessible on consumer devices, but landmarks do not by themselves establish correct fingering; instrument-specific studies remain small and are rarely tested outside the laboratory. Audio–visual fusion is promising because each modality resolves ambiguities in the other, yet few studies compare fused and single-modality systems on identical test material. Evidence on learning outcomes is thinner still: most systems report technical accuracy, and only a handful report controlled learner studies. The review closes with an evaluation framework that separates technical, pedagogical, and usability claims, and applies it to VIBEX, a guitar-focused learning platform under development by the authors.
Article Details
Section
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
All articles published in VTIP-JMRR are made freely available under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Under this license, others may freely share, distribute, and adapt the published work, provided appropriate credit is given to the original authors.
By signing the copyright form, the author(s) warrant and represent that:
✓ The manuscript is original and has not been published previously in any journal, conference, or book (in whole or in part).
✓ The manuscript is not currently under consideration for publication elsewhere.
✓ All authors have read and approved the final manuscript and have agreed to this copyright transfer.
✓ The manuscript does not infringe upon any existing copyright, trademark, patent, or other intellectual property right of any third party.
✓ All data, figures, and tables presented are accurate, authentic, and have not been fabricated or falsified.
✓ Proper permissions have been obtained for any third-party material (images, figures, tables, text) included in the manuscript.
✓ The work complies with all applicable ethical guidelines, including those related to human and animal research.
✓ All sources of funding and conflicts of interest have been disclosed in the manuscript.
How to Cite
References
Akbari, M., & Cheng, H. (2015). Real-time piano music transcription based on computer vision. IEEE Transactions on Multimedia, 17(12), 2113–2121. https://doi.org/10.1109/TMM.2015.2473702
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the System Usability Scale. International Journal of Human–Computer Interaction, 24(6), 574–594. https://doi.org/10.1080/10447310802205776
Bello, J. P., Daudet, L., Abdallah, S., Duxbury, C., Davies, M., & Sandler, M. B. (2005). A tutorial on onset detection in music signals. IEEE Transactions on Speech and Audio Processing, 13(5), 1035–1047. https://doi.org/10.1109/TSA.2005.851998
Benetos, E., Dixon, S., Duan, Z., & Ewert, S. (2019). Automatic music transcription: An overview. IEEE Signal Processing Magazine, 36(1), 20–30. https://doi.org/10.1109/MSP.2018.2869928
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & I. L. McClelland (Eds.), Usability evaluation in industry (pp. 189–194). Taylor & Francis.
Burns, A.-M., & Wanderley, M. M. (2006). Visual methods for the retrieval of guitarist fingering. In Proceedings of the International Conference on New Interfaces for Musical Expression (pp. 196–199). https://doi.org/10.5281/zenodo.1176850
Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., & Sheikh, Y. (2021). OpenPose: Realtime multi-person 2D pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1), 172–186. https://doi.org/10.1109/TPAMI.2019.2929257
Cicconet, M., Carvalho, P., Velho, L., & Gattass, M. (2010). Visual pitch class profile: A video-based method for real-time guitar chord identification. In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP) (Vol. 2, pp. 487–490). SciTePress. https://doi.org/10.5220/0002824904870490
de Cheveigné, A., & Kawahara, H. (2002). YIN, a fundamental frequency estimator for speech and music. The Journal of the Acoustical Society of America, 111(4), 1917–1930. https://doi.org/10.1121/1.1458024
Dittmar, C., Cano, E., Abeßer, J., & Grollmisch, S. (2012). Music information retrieval meets music education. In M. Müller, M. Goto, & M. Schedl (Eds.), Multimodal music processing (Dagstuhl Follow-Ups, Vol. 3, pp. 95–120). Schloss Dagstuhl–Leibniz-Zentrum für Informatik. https://doi.org/10.4230/DFU.Vol3.11041.95
Duan, Z., Essid, S., Liem, C. C. S., Richard, G., & Sharma, G. (2019). Audiovisual analysis of music performances: Overview of an emerging field. IEEE Signal Processing Magazine, 36(1), 63–73. https://doi.org/10.1109/MSP.2018.2875511
Eremenko, V., Morsi, A., Narang, J., & Serra, X. (2020). Performance assessment technologies for the support of musical instrument learning. In Proceedings of the 12th International Conference on Computer Supported Education (CSEDU) (Vol. 1, pp. 629–640). SciTePress. https://doi.org/10.5220/0009817006290640
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363–406. https://doi.org/10.1037/0033-295X.100.3.363
Gan, Q., Wang, S., Wu, S., & Zhu, J. (2025). PianoMotion10M: Dataset and benchmark for hand motion generation in piano performance. In Proceedings of the International Conference on Learning Representations (ICLR). https://doi.org/10.48550/arXiv.2406.09326
Gfeller, B., Frank, C., Roblek, D., Sharifi, M., Tagliasacchi, M., & Velimirović, M. (2020). SPICE: Self-supervised pitch estimation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28, 1118–1128.
Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
Hawthorne, C., Elsen, E., Song, J., Roberts, A., Simon, I., Raffel, C., Engel, J., Oore, S., & Eck, D. (2018). Onsets and frames: Dual-objective piano transcription. In Proceedings of the 19th International Society for Music Information Retrieval Conference (pp. 50–57). https://doi.org/10.5281/zenodo.1492341
Kim, J. W., Salamon, J., Li, P., & Bello, J. P. (2018). CREPE: A convolutional representation for pitch estimation. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 161–165). https://doi.org/10.1109/ICASSP.2018.8461329
Kim, Y., Park, J., Bae, J., Kim, K., Kwon, T., Lerch, A., & Nam, J. (2025). PianoVAM: A multimodal piano performance dataset (arXiv:2509.08800). arXiv. https://doi.org/10.48550/arXiv.2509.08800
Koepke, A. S., Wiles, O., Moses, Y., & Zisserman, A. (2020). Sight to sound: An end-to-end approach for visual piano transcription. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE.
Lerch, A., Arthur, C., Pati, A., & Gururani, S. (2020). An interdisciplinary review of music performance analysis. Transactions of the International Society for Music Information Retrieval, 3(1), 221–245. https://doi.org/10.5334/tismir.53
Li, B., Liu, X., Dinesh, K., Duan, Z., & Sharma, G. (2019). Creating a multitrack classical music performance dataset for multimodal music analysis: Challenges, insights, and applications. IEEE Transactions on Multimedia, 21(2), 522–535. https://doi.org/10.1109/TMM.2018.2856090
Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.-L., Yong, M. G., Lee, J., Chang, W.-T., Hua, W., Georg, M., & Grundmann, M. (2019). MediaPipe: A framework for building perception pipelines (arXiv:1906.08172). arXiv. https://doi.org/10.48550/arXiv.1906.08172
Marullo, G., Amati, D., Barbarino, S., D’Onofrio, M., Sechi, T., Innocente, C., Vezzetti, E., & Ulrich, L. (2026). Three-dimensional vision-based recognition of guitar chords. Computer Music Journal. https://doi.org/10.1162/COMJ.a.690
Mauch, M., & Dixon, S. (2014). pYIN: A fundamental frequency estimator using probabilistic threshold distributions. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 659–663). https://doi.org/10.1109/ICASSP.2014.6853678
McFee, B., Raffel, C., Liang, D., Ellis, D. P. W., McVicar, M., Battenberg, E., & Nieto, O. (2015). librosa: Audio and music signal analysis in Python. In Proceedings of the 14th Python in Science Conference (pp. 18–24). https://doi.org/10.25080/Majora-7b98e3ed-003
Mohammadian, R., Pourroostaei Ardakani, S., & Zaraii Zavaraki, E. (2026). A systematic review of technology-enhanced music learning models: Impacts on music education. Computers and Education Open, 10, 100354. https://doi.org/10.1016/j.caeo.2026.100354
Ng, K., & Nesi, P. (2008). i-Maestro: Technology-enhanced learning and teaching for music. In Proceedings of the International Conference on New Interfaces for Musical Expression (pp. 225–228). https://doi.org/10.5281/zenodo.1179605
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., et al. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71
Pati, K. A., Gururani, S., & Lerch, A. (2018). Assessment of student music performances using deep neural networks. Applied Sciences, 8(4), 507. https://doi.org/10.3390/app8040507
Percival, G., Wang, Y., & Tzanetakis, G. (2007). Effective use of multimedia for computer-assisted musical instrument tutoring. In Proceedings of the International Workshop on Educational Multimedia and Multimedia Education (EMME ’07). ACM.
Provenzale, C., Di Tommaso, F., Di Stefano, N., Formica, D., & Taffoni, F. (2024). Real-time visual feedback based on MIMUs technology reduces bowing errors in beginner violin students. Sensors, 24(12), 3961. https://doi.org/10.3390/s24123961
Raffel, C., McFee, B., Humphrey, E. J., Salamon, J., Nieto, O., Liang, D., & Ellis, D. P. W. (2014). mir_eval: A transparent implementation of common MIR metrics. In Proceedings of the 15th International Society for Music Information Retrieval Conference (pp. 367–372).
Riley, X., Edwards, D., & Dixon, S. (2024). High resolution guitar transcription via domain adaptation (arXiv:2402.15258). arXiv. https://doi.org/10.48550/arXiv.2402.15258
Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/0034654307313795
Tavares, T. F., Barbedo, J. G. A., Attux, R., & Lopes, A. (2013). Survey on automatic transcription of music. Journal of the Brazilian Computer Society, 19, 589–604. https://doi.org/10.1007/s13173-013-0118-6
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221. https://doi.org/10.1080/00461520.2011.611369
Wiggins, A., & Kim, Y. E. (2019). Guitar tablature estimation with a convolutional neural network. In Proceedings of the 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands.
Wulf, G., & Shea, C. H. (2002). Principles derived from the study of simple skills do not generalize to complex skill learning. Psychonomic Bulletin & Review, 9(2), 185–211. https://doi.org/10.3758/BF03196276
Xi, Q., Bittner, R. M., Pauwels, J., Ye, X., & Bello, J. P. (2018). GuitarSet: A dataset for guitar transcription. In Proceedings of the 19th International Society for Music Information Retrieval Conference (pp. 453–460).
Xia, G., Chin, D., Zhang, Y., Zhang, T., & Zhao, J. (2020). Interactive Rainbow Score: A visual-centered multimodal flute tutoring system. In Proceedings of the International Conference on New Interfaces for Musical Expression (pp. 208–213). https://doi.org/10.5281/zenodo.4813324
Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C.-L., & Grundmann, M. (2020). MediaPipe Hands: On-device real-time hand tracking (arXiv:2006.10214). arXiv. https://doi.org/10.48550/arXiv.2006.10214
Zhang, Y., Beh, W. F., Zhang, C., & Pi, S. (2024). Transforming music education through artificial intelligence: A systematic literature review on enhancing music teaching and learning. International Journal of Interactive Mobile Technologies, 18(18), 76–93. https://doi.org/10.3991/ijim.v18i18.50545