Edge AI for Personal Desktop Assistants: A Review of Innovative Local Automation and Intelligent Interaction
Article Sidebar
Main Article Content
Abstract
Advances in Artificial Intelligence (AI) have enabled new forms of human–computer interaction through intelligent personal assistants. Conventional desktop assistants are constrained by their dependence on the cloud, data-privacy concerns, limited system-level automation and a lack of multimodal interaction. Edge AI addresses these problems by running inference on the user's own machine, which reduces response latency and keeps personal data on the device. This article presents a structured review of the techniques and tools that enable edge-based personal desktop assistants. Following a PRISMA-guided search of IEEE Xplore, the ACM Digital Library, ScienceDirect, arXiv and Google Scholar, 45 studies and software resources were included and organised into nine themes: intelligent personal assistants, edge AI and on-device inference, on-device automatic speech recognition, intent classification, small and quantised local large language models, retrieval-augmented generation over local documents, operating-system-level desktop agents, and vision-based authentication and gesture interaction. The reviewed approaches are compared qualitatively in terms of functionality, processing location, latency and resource usage, data privacy, reliability and ease of integration. Building on this synthesis, a conceptual reference architecture and formulation is presented for a multimodal desktop assistant, covering intent routing, local retrieval, a sensitivity-aware execution policy, a latency model and a multi-objective trade-off function. The discussion highlights the trade-offs between local inference and cloud-based reasoning, the cost of running several AI models on one machine, and the point at which privacy breaks down when local retrieval is paired with a cloud-hosted language model. Finally, research gaps and future directions are identified in secure command execution, private document retrieval, multimodal interaction and efficient on-device processing.
Article Details
Section
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
All articles published in VTIP-JMRR are made freely available under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Under this license, others may freely share, distribute, and adapt the published work, provided appropriate credit is given to the original authors.
By signing the copyright form, the author(s) warrant and represent that:
✓ The manuscript is original and has not been published previously in any journal, conference, or book (in whole or in part).
✓ The manuscript is not currently under consideration for publication elsewhere.
✓ All authors have read and approved the final manuscript and have agreed to this copyright transfer.
✓ The manuscript does not infringe upon any existing copyright, trademark, patent, or other intellectual property right of any third party.
✓ All data, figures, and tables presented are accurate, authentic, and have not been fabricated or falsified.
✓ Proper permissions have been obtained for any third-party material (images, figures, tables, text) included in the manuscript.
✓ The work complies with all applicable ethical guidelines, including those related to human and animal research.
✓ All sources of funding and conflicts of interest have been disclosed in the manuscript.
How to Cite
References
[1] M. B. Hoy, “Alexa, Siri, Cortana, and more: An introduction to voice assistants,” Medical Reference Services Quarterly, vol. 37, no. 1, pp. 81–88, 2018, doi: 10.1080/02763869.2018.1404391.
[2] A. de Barcelos Silva et al., “Intelligent personal assistants: A systematic literature review,” Expert Systems with Applications, vol. 147, Art. no. 113193, Jun. 2020, doi: 10.1016/j.eswa.2020.113193.
[3] P. Cheng and U. Roedig, “Personal voice assistant security and privacy—A survey,” Proceedings of the IEEE, vol. 110, no. 4, pp. 476–507, Apr. 2022, doi: 10.1109/JPROC.2022.3153167.
[4] W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, “Edge computing: Vision and challenges,” IEEE Internet of Things Journal, vol. 3, no. 5, pp. 637–646, Oct. 2016, doi: 10.1109/JIOT.2016.2579198.
[5] Z. Zhou, X. Chen, E. Li, L. Zeng, K. Luo, and J. Zhang, “Edge intelligence: Paving the last mile of artificial intelligence with edge computing,” Proceedings of the IEEE, vol. 107, no. 8, pp. 1738–1762, Aug. 2019, doi: 10.1109/JPROC.2019.2918951.
[6] S. Deng, H. Zhao, W. Fang, J. Yin, S. Dustdar, and A. Y. Zomaya, “Edge intelligence: The confluence of edge computing and artificial intelligence,” IEEE Internet of Things Journal, vol. 7, no. 8, pp. 7457–7469, Aug. 2020, doi: 10.1109/JIOT.2020.2984887.
[7] X. Wang, Z. Tang, J. Guo, T. Meng, C. Wang, T. Wang, and W. Jia, “Empowering edge intelligence: A comprehensive survey on on-device AI models,” ACM Computing Surveys, vol. 57, no. 9, Art. no. 228, 2025, doi: 10.1145/3724420.
[8] J. Xu et al., “On-device language models: A comprehensive review,” arXiv:2409.00088, 2024, doi: 10.48550/arXiv.2409.00088.
[9] P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 9459–9474, doi: 10.48550/arXiv.2005.11401.
[10] Y. Gao et al., “Retrieval-augmented generation for large language models: A survey,” arXiv:2312.10997, 2023, doi: 10.48550/arXiv.2312.10997.
[11] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection,” in Proc. 16th ACM Workshop on Artificial Intelligence and Security (AISec), 2023, pp. 79–90, doi: 10.1145/3605764.3623985.
[12] F. Schroff, D. Kalenichenko, and J. Philbin, “FaceNet: A unified embedding for face recognition and clustering,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2015, pp. 815–823, doi: 10.1109/CVPR.2015.7298682.
[13] C. Lugaresi et al., “MediaPipe: A framework for building perception pipelines,” arXiv:1906.08172, 2019, doi: 10.48550/arXiv.1906.08172.
[14] F. Zhang et al., “MediaPipe Hands: On-device real-time hand tracking,” arXiv:2006.10214, 2020, doi: 10.48550/arXiv.2006.10214.
[15] M. J. Page et al., “The PRISMA 2020 statement: An updated guideline for reporting systematic reviews,” BMJ, vol. 372, Art. no. n71, 2021, doi: 10.1136/bmj.n71.
[16] A. Dobriyal, D. Rawat, S. Vats, and V. Sharma, “AI-driven personal desktop assistant,” in Proc. 2024 IEEE 2nd Int. Conf. Innovations in High Speed Communication and Signal Processing (IHCSP), 2024, doi: 10.1109/IHCSP63227.2024.10959800.
[17] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding,” in Proc. Int. Conf. Learning Representations (ICLR), 2016, doi: 10.48550/arXiv.1510.00149.
[18] A. G. Howard et al., “MobileNets: Efficient convolutional neural networks for mobile vision applications,” arXiv:1704.04861, 2017, doi: 10.48550/arXiv.1704.04861.
[19] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 12449–12460, doi: 10.48550/arXiv.2006.11477.
[20] A. Gulati et al., “Conformer: Convolution-augmented Transformer for speech recognition,” in Proc. Interspeech, 2020, pp. 5036–5040, doi: 10.21437/Interspeech.2020-3015.
[21] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proc. 40th Int. Conf. Machine Learning (ICML), PMLR vol. 202, 2023, pp. 28492–28518, doi: 10.48550/arXiv.2212.04356.
[22] SYSTRAN, “faster-whisper: Faster Whisper transcription with CTranslate2,” GitHub repository (software; no DOI). [Online]. Available: https://github.com/SYSTRAN/faster-whisper (accessed Oct. 2026).
[23] Alpha Cephei, “Vosk speech recognition toolkit: Offline speech recognition API,” software documentation (no DOI). [Online]. Available: https://alphacephei.com/vosk/ (accessed Oct. 2026).
[24] A. Coucke et al., “Snips Voice Platform: An embedded spoken language understanding system for private-by-design voice interfaces,” arXiv:1805.10190, 2018, doi: 10.48550/arXiv.1805.10190.
[25] Q. Chen, Z. Zhuo, and W. Wang, “BERT for joint intent classification and slot filling,” arXiv:1902.10909, 2019, doi: 10.48550/arXiv.1902.10909.
[26] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” in Proc. Conf. Empirical Methods in Natural Language Processing and 9th Int. Joint Conf. Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 3982–3992, doi: 10.18653/v1/D19-1410.
[27] H. Touvron et al., “LLaMA: Open and efficient foundation language models,” arXiv:2302.13971, 2023, doi: 10.48550/arXiv.2302.13971.
[28] A. Q. Jiang et al., “Mistral 7B,” arXiv:2310.06825, 2023, doi: 10.48550/arXiv.2310.06825.
[29] M. Abdin et al., “Phi-3 technical report: A highly capable language model locally on your phone,” arXiv:2404.14219, 2024, doi: 10.48550/arXiv.2404.14219.
[30] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate post-training quantization for generative pre-trained transformers,” in Proc. Int. Conf. Learning Representations (ICLR), 2023, doi: 10.48550/arXiv.2210.17323.
[31] J. Lin et al., “AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration,” in Proc. Machine Learning and Systems (MLSys), vol. 6, 2024, doi: 10.48550/arXiv.2306.00978.
[32] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient finetuning of quantized LLMs,” in Advances in Neural Information Processing Systems, vol. 36, 2023, doi: 10.48550/arXiv.2305.14314.
[33] G. Gerganov and contributors, “llama.cpp: LLM inference in C/C++,” GitHub repository (software; no DOI). [Online]. Available: https://github.com/ggml-org/llama.cpp (accessed Oct. 2026).
[34] Ollama, “Ollama: Run large language models locally,” software documentation (no DOI). [Online]. Available: https://ollama.com (accessed Oct. 2026).
[35] Gemini Team, Google, “Gemini: A family of highly capable multimodal models,” arXiv:2312.11805, 2023, doi: 10.48550/arXiv.2312.11805.
[36] V. Karpukhin et al., “Dense passage retrieval for open-domain question answering,” in Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 6769–6781, doi: 10.18653/v1/2020.emnlp-main.550.
[37] J. Johnson, M. Douze, and H. Jégou, “Billion-scale similarity search with GPUs,” IEEE Transactions on Big Data, vol. 7, no. 3, pp. 535–547, Jul. 2021, doi: 10.1109/TBDATA.2019.2921572.
[38] S. Yao et al., “ReAct: Synergizing reasoning and acting in language models,” in Proc. Int. Conf. Learning Representations (ICLR), 2023, doi: 10.48550/arXiv.2210.03629.
[39] X. Deng et al., “Mind2Web: Towards a generalist agent for the web,” in Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), vol. 36, 2023, doi: 10.48550/arXiv.2306.06070.
[40] T. Xie et al., “OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments,” in Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), vol. 37, 2024, doi: 10.52202/079017-1650.
[41] C. Zhang et al., “UFO: A UI-focused agent for Windows OS interaction,” arXiv:2402.07939, 2024, doi: 10.48550/arXiv.2402.07939.
[42] Y. Li et al., “Personal LLM agents: Insights and survey about the capability, efficiency and security,” arXiv:2401.05459, 2024, doi: 10.48550/arXiv.2401.05459.
[43] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “ArcFace: Additive angular margin loss for deep face recognition,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4690–4699, doi: 10.1109/CVPR.2019.00482.
[44] V. Bazarevsky, Y. Kartynnik, A. Vakunov, K. Raveendran, and M. Grundmann, “BlazeFace: Sub-millisecond neural face detection on mobile GPUs,” arXiv:1907.05047, 2019, doi: 10.48550/arXiv.1907.05047.
[45] Z. Yu, Y. Qin, X. Li, C. Zhao, Z. Lei, and G. Zhao, “Deep learning for face anti-spoofing: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 5, pp. 5609–5631, May 2023, doi: 10.1109/TPAMI.2022.3215850.
[46] G. Zhang, C. Yan, X. Ji, T. Zhang, T. Zhang, and W. Xu, “DolphinAttack: Inaudible voice commands,” in Proc. ACM SIGSAC Conf. Computer and Communications Security (CCS), 2017, pp. 103–117, doi: 10.1145/3133956.3134052.