AHCI RESEARCH GROUP
Publications
Papers published in international journals,
proceedings of conferences, workshops and books.
OUR RESEARCH
Scientific Publications
How to
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
2025
Miller, C. H.
Digital Storytelling: A Creator’s Guide to Interactive Entertainment: Volume I, Fifth Edition Book
CRC Press, 2025, ISBN: 978-104034442-2 (ISBN); 978-103285888-3 (ISBN).
Abstract | Links | BibTeX | Tags: Case-studies, Chatbots, Creatives, Digital storytelling, Entertainment, Immersive environment, Interactive documentary, Interactive entertainment, Social media, Use of video, Video-games, Virtual Reality
@book{miller_digital_2025,
title = {Digital Storytelling: A Creator’s Guide to Interactive Entertainment: Volume I, Fifth Edition},
author = {C. H. Miller},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105004122515&doi=10.1201%2f9781003520092&partnerID=40&md5=894bfbb310cbd095a54409f9ac5174da},
doi = {10.1201/9781003520092},
isbn = {978-104034442-2 (ISBN); 978-103285888-3 (ISBN)},
year = {2025},
date = {2025-01-01},
volume = {1},
publisher = {CRC Press},
series = {Digital Storytelling: a Creator's Guide to Interactive Entertainment: Volume I, Fifth Edition},
abstract = {Digital Storytelling: A Creator’s Guide to Interactive Entertainment, Volume I, fifth edition delves into the fascinating and groundbreaking stories enabled by interactive digital media, examining both fictional and non-fiction narratives. This fifth edition explores monumental developments, particularly the emergence of generative AI, and highlights exciting projects utilizing this technology. Additionally, it covers social media; interactive documentaries; immersive environments; and innovative uses of video games, chatbots, and virtual reality. Carolyn Handler Miller provides insights into storytelling essentials like character development, plot, structure, dialogue, and emotion, while examining how digital media and interactivity influence these elements. This book also dives into advanced topics, such as narratives using AR, VR, and XR, alongside new forms of immersive media, including large screens, escape rooms, and theme park experiences. With numerous case studies, this edition illustrates the creative possibilities of digital storytelling and its applications beyond entertainment, such as education, training, information, and promotion. Interviews with industry leaders further enhance the understanding of this evolving universe, making it a valuable resource for both professionals and enthusiasts. Key Features: • This book includes up-to-the-minute developments in digital storytelling. • It offers case studies of noteworthy examples of digital storytelling. • It includes a glossary clearly defining new or difficult terms. • Each chapter opens with several thought-provoking questions about the chapter’s topic. • Each chapter concludes with several creative and engaging exercises to promote the reader’s understanding of the chapter’s topic. © 2025 Carolyn Handler Miller.},
keywords = {Case-studies, Chatbots, Creatives, Digital storytelling, Entertainment, Immersive environment, Interactive documentary, Interactive entertainment, Social media, Use of video, Video-games, Virtual Reality},
pubstate = {published},
tppubtype = {book}
}
Li, H.; Wang, Z.; Liang, W.; Wang, Y.
X’s Day: Personality-Driven Virtual Human Behavior Generation Journal Article
In: IEEE Transactions on Visualization and Computer Graphics, vol. 31, no. 5, pp. 3514–3524, 2025, ISSN: 10772626 (ISSN).
Abstract | Links | BibTeX | Tags: adult, Augmented Reality, Behavior Generation, Chatbots, Computer graphics, computer interface, Contextual Scene, female, human, Human behaviors, Humans, Long-term behavior, male, Novel task, Personality, Personality traits, Personality-driven Behavior, physiology, Social behavior, User-Computer Interface, Users' experiences, Virtual agent, Virtual environments, Virtual humans, Virtual Reality, Young Adult
@article{li_xs_2025,
title = {X’s Day: Personality-Driven Virtual Human Behavior Generation},
author = {H. Li and Z. Wang and W. Liang and Y. Wang},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105003864932&doi=10.1109%2fTVCG.2025.3549574&partnerID=40&md5=a865bbd2b0fa964a4f0f4190955dc787},
doi = {10.1109/TVCG.2025.3549574},
issn = {10772626 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Transactions on Visualization and Computer Graphics},
volume = {31},
number = {5},
pages = {3514–3524},
abstract = {Developing convincing and realistic virtual human behavior is essential for enhancing user experiences in virtual reality (VR) and augmented reality (AR) settings. This paper introduces a novel task focused on generating long-term behaviors for virtual agents, guided by specific personality traits and contextual elements within 3D environments. We present a comprehensive framework capable of autonomously producing daily activities autoregressively. By modeling the intricate connections between personality characteristics and observable activities, we establish a hierarchical structure of Needs, Task, and Activity levels. Integrating a Behavior Planner and a World State module allows for the dynamic sampling of behaviors using large language models (LLMs), ensuring that generated activities remain relevant and responsive to environmental changes. Extensive experiments validate the effectiveness and adaptability of our approach across diverse scenarios. This research makes a significant contribution to the field by establishing a new paradigm for personalized and context-aware interactions with virtual humans, ultimately enhancing user engagement in immersive applications. Our project website is at: https://behavior.agent-x.cn/. © 2025 IEEE. All rights reserved,},
keywords = {adult, Augmented Reality, Behavior Generation, Chatbots, Computer graphics, computer interface, Contextual Scene, female, human, Human behaviors, Humans, Long-term behavior, male, Novel task, Personality, Personality traits, Personality-driven Behavior, physiology, Social behavior, User-Computer Interface, Users' experiences, Virtual agent, Virtual environments, Virtual humans, Virtual Reality, Young Adult},
pubstate = {published},
tppubtype = {article}
}
Yadav, R.; Huzooree, G.; Yadav, M.; Gangodawilage, D. S. K.
Generative AI for personalized learning content creation Book Section
In: Transformative AI Practices for Personalized Learning Strategies, pp. 107–130, IGI Global, 2025, ISBN: 979-836938746-7 (ISBN); 979-836938744-3 (ISBN).
Abstract | Links | BibTeX | Tags: Adaptive feedback, Advanced Analytics, AI systems, Contrastive Learning, Educational contents, Educational experiences, Enhanced learning, Ethical technology, Federated learning, Immersive, Learning content creation, Personalized learning, Student engagement, Students, Supervised learning, Tools and applications, Virtual Reality
@incollection{yadav_generative_2025,
title = {Generative AI for personalized learning content creation},
author = {R. Yadav and G. Huzooree and M. Yadav and D. S. K. Gangodawilage},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005387236&doi=10.4018%2f979-8-3693-8744-3.ch005&partnerID=40&md5=904e58b9c6de83dcd431c1706dda02b3},
doi = {10.4018/979-8-3693-8744-3.ch005},
isbn = {979-836938746-7 (ISBN); 979-836938744-3 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Transformative AI Practices for Personalized Learning Strategies},
pages = {107–130},
publisher = {IGI Global},
abstract = {Generative AI has emerged as a transformative force in personalized learning, offering unprecedented opportunities to tailor educational content to individual needs. By leveraging advanced algorithms and data analysis, AI systems can dynamically generate customized materials, provide adaptive feedback, and foster student engagement. This chapter explores the intersection of generative AI and personalized learning, discussing its techniques, tools, and applications in creating immersive and adaptive educational experiences. Key benefits include enhanced learning outcomes, efficiency, and scalability. However, challenges such as data privacy, algorithmic bias, and equitable access must be addressed to ensure responsible implementation. Future trends, including the integration of immersive technologies like Virtual Reality (VR) and predictive analytics, highlight AI's potential to revolutionize education. By navigating ethical considerations and fostering transparency, generative AI can become a powerful ally in creating inclusive, engaging, and student- centered learning environments. © 2025, IGI Global Scientific Publishing. All rights reserved.},
keywords = {Adaptive feedback, Advanced Analytics, AI systems, Contrastive Learning, Educational contents, Educational experiences, Enhanced learning, Ethical technology, Federated learning, Immersive, Learning content creation, Personalized learning, Student engagement, Students, Supervised learning, Tools and applications, Virtual Reality},
pubstate = {published},
tppubtype = {incollection}
}
Shawash, J.; Thibault, M.; Hamari, J.
Who Killed Helene Pumpulivaara?: AI-Assisted Content Creation and XR Implementation for Interactive Built Heritage Storytelling Proceedings Article
In: IMX - Proc. ACM Int. Conf. Interact. Media Experiences, pp. 377–379, Association for Computing Machinery, Inc, 2025, ISBN: 979-840071391-0 (ISBN).
Abstract | Links | BibTeX | Tags: Artificial intelligence, Augmented Reality, Built heritage, Content creation, Digital heritage, Digital Interpretation, Extended reality, Human computer interaction, Human engineering, Industrial Heritage, Interactive computer graphics, Interactive computer systems, Mobile photographies, Narrative Design, Narrative designs, Production pipelines, Uncanny valley, Virtual Reality
@inproceedings{shawash_who_2025,
title = {Who Killed Helene Pumpulivaara?: AI-Assisted Content Creation and XR Implementation for Interactive Built Heritage Storytelling},
author = {J. Shawash and M. Thibault and J. Hamari},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105008003446&doi=10.1145%2f3706370.3731703&partnerID=40&md5=bc8a8d221abcf6c560446979fbd06cbc},
doi = {10.1145/3706370.3731703},
isbn = {979-840071391-0 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {IMX - Proc. ACM Int. Conf. Interact. Media Experiences},
pages = {377–379},
publisher = {Association for Computing Machinery, Inc},
abstract = {This demo presents "Who Killed Helene Pumpulivaara?", an innovative interactive heritage experience that combines crime mystery narrative with XR technology to address key challenges in digital heritage interpretation. Our work makes six significant contributions: (1) the discovery of a "Historical Uncanny Valley"effect where varying fidelity levels between AI-generated and authentic content serve as implicit markers distinguishing fact from interpretation; (2) an accessible production pipeline combining mobile photography with AI tools that democratizes XR heritage creation for resource-limited institutions; (3) a spatial storytelling approach that effectively counters decontextualization in digital heritage; (4) a multi-platform implementation strategy across web and VR environments; (5) a practical model for AI-assisted heritage content creation balancing authenticity with engagement; and (6) a pathway toward spatial augmented reality for future heritage interpretation. Using the historic Finlayson Factory in Tampere, Finland as a case study, our implementation demonstrates how emerging technologies can enrich the authenticity of heritage experiences, fostering deeper emotional connections between visitors and the histories embedded in place. © 2025 Copyright held by the owner/author(s).},
keywords = {Artificial intelligence, Augmented Reality, Built heritage, Content creation, Digital heritage, Digital Interpretation, Extended reality, Human computer interaction, Human engineering, Industrial Heritage, Interactive computer graphics, Interactive computer systems, Mobile photographies, Narrative Design, Narrative designs, Production pipelines, Uncanny valley, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
Banafa, A.
Artificial intelligence in action: Real-world applications and innovations Book
River Publishers, 2025, ISBN: 978-877004619-0 (ISBN); 978-877004620-6 (ISBN).
Abstract | Links | BibTeX | Tags: 5G, Affective Computing, AGI, AI, AI alignments, AI Ethics, AI hallucinations, AI hype, AI models, Alexa, ANI, ASI, Augmented Reality, Autoencoders, Autonomic computing, Autonomous Cars, Autoregressive models, Big Data, Big Data Analytics, Bitcoin, Blockchain, C3PO, Casual AI, Causal reasoning, ChatGPT, Cloud computing, Collective AI, Compression engines, Computer vision, Conditional Automation, Convolutional neural networks (CNNs), Cryptocurrency, Cybersecurity, Deceptive AI, Deep learning, Digital transformation, Driver Assistance, Driverless Cars, Drones, Elon Musk, Entanglement, Environment and sustainability, Ethereum, Explainable AI, Facebook, Facial Recognition, Feedforward. Neural Networks, Fog Computing, Full Automation, Future of AI, General AI, Generative Adversarial Networks (GANs), Generative AI, Google, Green AI, High Automation, Hybrid Blockchain, IEEE, Industrial Internet of Things (IIoT), Internet of things (IoT), Jarvis, Java, JavaScript, Long Short-Term Memory Networks, LTE, machine learning, Microsoft, MultiModal AI, Narrow AI, Natural disasters, Natural Language Generation (NLG), Natural Language Processing (NLP), NetFlix, Network Security, Neural Networks, Nuclear, Nuclear AI, NYTimes, Objective-driven AI, Open Source, Partial Automation, PayPal, Perfect AI, Private Blockchain, Private Cloud Computing, Programming languages, Python, Quantum Communications, Quantum Computing, Quantum Cryptography, Quantum internet, Quantum Machine Learning (QML), R2D2, Reactive machines. limited memory, Recurrent Neural Networks, Responsible AI, Robots, Sci-Fi movies, Self-Aware, Semiconductorâ??s, Sensate AI, Siri, Small Data, Smart Contracts. Hybrid Cloud Computing, Smart Devices, Sovereign AI, Super AI, Superposition, TensorFlow, Theory of Mind, Thick Data, Twitter, Variational Autoencoders (VAEs), Virtual Reality, Voice user interface (VUI), Wearable computing devices (WCD), Wearable Technology, Wi-Fi, XAI, Zero-Trust Model
@book{banafa_artificial_2025,
title = {Artificial intelligence in action: Real-world applications and innovations},
author = {A. Banafa},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105000403587&partnerID=40&md5=4b0d94be48194a942b22bef63f36d3bf},
isbn = {978-877004619-0 (ISBN); 978-877004620-6 (ISBN)},
year = {2025},
date = {2025-01-01},
publisher = {River Publishers},
series = {Artificial Intelligence in Action: Real-World Applications and Innovations},
abstract = {This comprehensive book dives deep into the current landscape of AI, exploring its fundamental principles, development challenges, potential risks, and the cutting-edge breakthroughs that are propelling it forward. Artificial intelligence (AI) is rapidly transforming industries and societies worldwide through groundbreaking innovations and real-world applications. Starting with the core concepts, the book examines the various types of AI systems, generative AI models, and the complexities of machine learning. It delves into the programming languages driving AI development, data pipelines, model creation and deployment processes, while shedding light on issues like AI hallucinations and the intricate path of machine unlearning. The book then showcases the remarkable real-world applications of AI across diverse domains. From preventing job displacement and promoting environmental sustainability, to enhancing disaster response, drone technology, and even nuclear energy innovation, it highlights how AI is tackling complex challenges and driving positive change. The book also explores the double-edged nature of AI, recognizing its tremendous potential while cautioning about the risks of misuse, unintended consequences, and the urgent need for responsible development practices. It examines the intersection of AI and fields like operating system design, warfare, and semiconductor technology, underscoring the wide-ranging implications of this transformative force. As the quest for artificial general intelligence (AGI) and superintelligent AI systems intensifies, the book delves into cutting-edge research, emerging trends, and the pursuit of multimodal, explainable, and causally aware AI systems. It explores the symbiotic relationship between AI and human creativity, the rise of user-friendly "casual AI," and the potential of AI to tackle open-ended tasks. This is an essential guide for understanding the profound impact of AI on our world today and its potential to shape our future. From the frontiers of innovation to the challenges of responsible development, this book offers a comprehensive and insightful exploration of the remarkable real-world applications and innovations driving the AI revolution. © 2025 River Publishers. All rights reserved.},
keywords = {5G, Affective Computing, AGI, AI, AI alignments, AI Ethics, AI hallucinations, AI hype, AI models, Alexa, ANI, ASI, Augmented Reality, Autoencoders, Autonomic computing, Autonomous Cars, Autoregressive models, Big Data, Big Data Analytics, Bitcoin, Blockchain, C3PO, Casual AI, Causal reasoning, ChatGPT, Cloud computing, Collective AI, Compression engines, Computer vision, Conditional Automation, Convolutional neural networks (CNNs), Cryptocurrency, Cybersecurity, Deceptive AI, Deep learning, Digital transformation, Driver Assistance, Driverless Cars, Drones, Elon Musk, Entanglement, Environment and sustainability, Ethereum, Explainable AI, Facebook, Facial Recognition, Feedforward. Neural Networks, Fog Computing, Full Automation, Future of AI, General AI, Generative Adversarial Networks (GANs), Generative AI, Google, Green AI, High Automation, Hybrid Blockchain, IEEE, Industrial Internet of Things (IIoT), Internet of things (IoT), Jarvis, Java, JavaScript, Long Short-Term Memory Networks, LTE, machine learning, Microsoft, MultiModal AI, Narrow AI, Natural disasters, Natural Language Generation (NLG), Natural Language Processing (NLP), NetFlix, Network Security, Neural Networks, Nuclear, Nuclear AI, NYTimes, Objective-driven AI, Open Source, Partial Automation, PayPal, Perfect AI, Private Blockchain, Private Cloud Computing, Programming languages, Python, Quantum Communications, Quantum Computing, Quantum Cryptography, Quantum internet, Quantum Machine Learning (QML), R2D2, Reactive machines. limited memory, Recurrent Neural Networks, Responsible AI, Robots, Sci-Fi movies, Self-Aware, Semiconductorâ??s, Sensate AI, Siri, Small Data, Smart Contracts. Hybrid Cloud Computing, Smart Devices, Sovereign AI, Super AI, Superposition, TensorFlow, Theory of Mind, Thick Data, Twitter, Variational Autoencoders (VAEs), Virtual Reality, Voice user interface (VUI), Wearable computing devices (WCD), Wearable Technology, Wi-Fi, XAI, Zero-Trust Model},
pubstate = {published},
tppubtype = {book}
}
Buldu, K. B.; Özdel, S.; Lau, K. H. Carrie; Wang, M.; Saad, D.; Schönborn, S.; Boch, A.; Kasneci, E.; Bozkir, E.
CUIfy the XR: An Open-Source Package to Embed LLM-Powered Conversational Agents in XR Proceedings Article
In: Proc. - IEEE Int. Conf. Artif. Intell. Ext. Virtual Real., AIxVR, pp. 192–197, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 9798331521578 (ISBN).
Abstract | Links | BibTeX | Tags: Augmented Reality, Computational Linguistics, Conversational user interface, conversational user interfaces, Extended reality, Head-mounted-displays, Helmet mounted displays, Language Model, Large language model, large language models, Non-player character, non-player characters, Open source software, Personnel training, Problem oriented languages, Speech models, Speech-based interaction, Text to speech, Unity, Virtual environments, Virtual Reality
@inproceedings{buldu_cuify_2025,
title = {CUIfy the XR: An Open-Source Package to Embed LLM-Powered Conversational Agents in XR},
author = {K. B. Buldu and S. Özdel and K. H. Carrie Lau and M. Wang and D. Saad and S. Schönborn and A. Boch and E. Kasneci and E. Bozkir},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105000229165&doi=10.1109%2FAIxVR63409.2025.00037&partnerID=40&md5=f11f49480d075aee04ec44cedc984844},
doi = {10.1109/AIxVR63409.2025.00037},
isbn = {9798331521578 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. - IEEE Int. Conf. Artif. Intell. Ext. Virtual Real., AIxVR},
pages = {192–197},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Recent developments in computer graphics, machine learning, and sensor technologies enable numerous opportunities for extended reality (XR) setups for everyday life, from skills training to entertainment. With large corporations offering affordable consumer-grade head-mounted displays (HMDs), XR will likely become pervasive, and HMDs will develop as personal devices like smartphones and tablets. However, having intelligent spaces and naturalistic interactions in XR is as important as tech-nological advances so that users grow their engagement in virtual and augmented spaces. To this end, large language model (LLM)-powered non-player characters (NPCs) with speech-to-text (STT) and text-to-speech (TTS) models bring significant advantages over conventional or pre-scripted NPCs for facilitating more natural conversational user interfaces (CUIs) in XR. This paper provides the community with an open-source, customizable, extendable, and privacy-aware Unity package, CUIfy, that facili-tates speech-based NPC-user interaction with widely used LLMs, STT, and TTS models. Our package also supports multiple LLM-powered NPCs per environment and minimizes latency between different computational models through streaming to achieve us-able interactions between users and NPCs. We publish our source code in the following repository: https://gitlab.lrz.de/hctl/cuify © 2025 Elsevier B.V., All rights reserved.},
keywords = {Augmented Reality, Computational Linguistics, Conversational user interface, conversational user interfaces, Extended reality, Head-mounted-displays, Helmet mounted displays, Language Model, Large language model, large language models, Non-player character, non-player characters, Open source software, Personnel training, Problem oriented languages, Speech models, Speech-based interaction, Text to speech, Unity, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
Liu, G.; Du, H.; Wang, J.; Niyato, D.; Kim, D. I.
Contract-Inspired Contest Theory for Controllable Image Generation in Mobile Edge Metaverse Journal Article
In: IEEE Transactions on Mobile Computing, vol. 24, no. 8, pp. 7389–7405, 2025, ISSN: 15361233 (ISSN), (Publisher: Institute of Electrical and Electronics Engineers Inc.).
Abstract | Links | BibTeX | Tags: Contest Theory, Deep learning, Deep reinforcement learning, Diffusion Model, Generative adversarial networks, Generative AI, High quality, Image generation, Image generations, Immersive technologies, Metaverses, Mobile edge computing, Reinforcement Learning, Reinforcement learnings, Resource allocation, Resources allocation, Semantic data, Virtual addresses, Virtual environments, Virtual Reality
@article{liu_contract-inspired_2025,
title = {Contract-Inspired Contest Theory for Controllable Image Generation in Mobile Edge Metaverse},
author = {G. Liu and H. Du and J. Wang and D. Niyato and D. I. Kim},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105000066834&doi=10.1109%2FTMC.2025.3550815&partnerID=40&md5=f95abb0df00e3112fa2c15ee77eb41bc},
doi = {10.1109/TMC.2025.3550815},
issn = {15361233 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Transactions on Mobile Computing},
volume = {24},
number = {8},
pages = {7389–7405},
abstract = {The rapid advancement of immersive technologies has propelled the development of the Metaverse, where the convergence of virtual and physical realities necessitates the generation of high-quality, photorealistic images to enhance user experience. However, generating these images, especially through Generative Diffusion Models (GDMs), in mobile edge computing environments presents significant challenges due to the limited computing resources of edge devices and the dynamic nature of wireless networks. This paper proposes a novel framework that integrates contract-inspired contest theory, Deep Reinforcement Learning (DRL), and GDMs to optimize image generation in these resource-constrained environments. The framework addresses the critical challenges of resource allocation and semantic data transmission quality by incentivizing edge devices to efficiently transmit high-quality semantic data, which is essential for creating realistic and immersive images. The use of contest and contract theory ensures that edge devices are motivated to allocate resources effectively, while DRL dynamically adjusts to network conditions, optimizing the overall image generation process. Experimental results demonstrate that the proposed approach not only improves the quality of generated images but also achieves superior convergence speed and stability compared to traditional methods. This makes the framework particularly effective for optimizing complex resource allocation tasks in mobile edge Metaverse applications, offering enhanced performance and efficiency in creating immersive virtual environments. © 2025 Elsevier B.V., All rights reserved.},
note = {Publisher: Institute of Electrical and Electronics Engineers Inc.},
keywords = {Contest Theory, Deep learning, Deep reinforcement learning, Diffusion Model, Generative adversarial networks, Generative AI, High quality, Image generation, Image generations, Immersive technologies, Metaverses, Mobile edge computing, Reinforcement Learning, Reinforcement learnings, Resource allocation, Resources allocation, Semantic data, Virtual addresses, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
Häfner, P.; Eisenlohr, F.; Karande, A.; Grethler, M.; Mukherjee, A.; Tran, N.
Leveraging Virtual Prototypes for Training Data Collection in LLM-Based Voice User Interface Development for Machines Proceedings Article
In: Proc. - IEEE Int. Conf. Artif. Intell. Ext. Virtual Real., AIxVR, pp. 281–285, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 9798331521578 (ISBN).
Abstract | Links | BibTeX | Tags: Artificial intelligence, Behavioral Research, Data collection, Language Model, Large language model, large language models, Model-based OPC, Training data, User interface development, Virtual environments, Virtual Prototype, Virtual Prototyping, Virtual Reality, Voice User Interface, Voice User Interfaces, Wizard of Oz, Wizard-of-Oz Method
@inproceedings{hafner_leveraging_2025,
title = {Leveraging Virtual Prototypes for Training Data Collection in LLM-Based Voice User Interface Development for Machines},
author = {P. Häfner and F. Eisenlohr and A. Karande and M. Grethler and A. Mukherjee and N. Tran},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105000344182&doi=10.1109%2FAIxVR63409.2025.00054&partnerID=40&md5=464de1fae1a7a9dbc4362b0a984e0cd4},
doi = {10.1109/AIxVR63409.2025.00054},
isbn = {9798331521578 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. - IEEE Int. Conf. Artif. Intell. Ext. Virtual Real., AIxVR},
pages = {281–285},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Voice User Interfaces (VUIs) are becoming increasingly valuable in industrial applications, offering hands-free control in complex environments. However, developing and validating VUIs for such applications faces challenges, including limited access to physical prototypes and high testing costs. This paper presents a methodology that utilizes virtual reality (VR) prototypes to collect training data for large language model (LLM)-based VUIs, allowing early-stage voice control development before physical prototypes are accessible. Through an immersive Wizard-of-Oz (WoZ) method, participants interact with a virtual reality representation of a machine, generating realistic, scenario-based conversational data. This combined WoZ and VR approach enables high-quality data collection and iterative model training, offering an effective solution that can be applied across various types of machine. Preliminary findings demonstrate the viability of VR in generating diverse and robust data sets that closely simulate real-world dialogs for voice interactions in industrial settings. © 2025 Elsevier B.V., All rights reserved.},
keywords = {Artificial intelligence, Behavioral Research, Data collection, Language Model, Large language model, large language models, Model-based OPC, Training data, User interface development, Virtual environments, Virtual Prototype, Virtual Prototyping, Virtual Reality, Voice User Interface, Voice User Interfaces, Wizard of Oz, Wizard-of-Oz Method},
pubstate = {published},
tppubtype = {inproceedings}
}
Tong, Y.; Qiu, Y.; Li, R.; Qiu, S.; Heng, P. -A.
MS2Mesh-XR: Multi-Modal Sketch-to-Mesh Generation in XR Environments Proceedings Article
In: Proc. - IEEE Int. Conf. Artif. Intell. Ext. Virtual Real., AIxVR, pp. 272–276, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 9798331521578 (ISBN).
Abstract | Links | BibTeX | Tags: 3D meshes, 3D object, ControlNet, Hand-drawn sketches, Hands movement, High quality, Image-based, immersive visualization, Mesh generation, Multi-modal, Pipeline codes, Realistic images, Three dimensional computer graphics, Virtual environments, Virtual Reality
@inproceedings{tong_ms2mesh-xr_2025,
title = {MS2Mesh-XR: Multi-Modal Sketch-to-Mesh Generation in XR Environments},
author = {Y. Tong and Y. Qiu and R. Li and S. Qiu and P. -A. Heng},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105000423684&doi=10.1109%2FAIxVR63409.2025.00052&partnerID=40&md5=fe9d84b91722dbe8c11d43ffe2f2041d},
doi = {10.1109/AIxVR63409.2025.00052},
isbn = {9798331521578 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. - IEEE Int. Conf. Artif. Intell. Ext. Virtual Real., AIxVR},
pages = {272–276},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {We present MS2Mesh-XR, a novel multimodal sketch-to-mesh generation pipeline that enables users to create realistic 3D objects in extended reality (XR) environments using hand-drawn sketches assisted by voice inputs. In specific, users can intuitively sketch objects using natural hand movements in mid-air within a virtual environment. By integrating voice inputs, we devise ControlNet to infer realistic images based on the drawn sketches and interpreted text prompts. Users can then review and select their preferred image, which is subsequently reconstructed into a detailed 3D mesh using the Convolutional Reconstruction Model. In particular, our proposed pipeline can generate a high-quality 3D mesh in less than 20 seconds, allowing for immersive visualization and manipulation in runtime XR scenes. We demonstrate the practicability of our pipeline through two use cases in XR settings. By leveraging natural user inputs and cutting-edge generative AI capabilities, our approach can significantly facilitate XR-based creative production and enhance user experiences. Our code and demo will be available at: https://yueqiu0911.github.io/MS2Mesh-XR/. © 2025 Elsevier B.V., All rights reserved.},
keywords = {3D meshes, 3D object, ControlNet, Hand-drawn sketches, Hands movement, High quality, Image-based, immersive visualization, Mesh generation, Multi-modal, Pipeline codes, Realistic images, Three dimensional computer graphics, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
Casas, L.; Hannah, S.; Mitchell, K.
HoloJig: Interactive Spoken Prompt Specified Generative AI Environments Journal Article
In: IEEE Computer Graphics and Applications, vol. 45, no. 2, pp. 69–77, 2025, ISSN: 02721716 (ISSN); 15581756 (ISSN), (Publisher: IEEE Computer Society).
Abstract | Links | BibTeX | Tags: 3-D rendering, Article, Collaborative workspace, customer experience, Economic and social effects, generative artificial intelligence, human, Immersive, Immersive environment, parallax, Real- time, simulation, Simulation training, speech, Time based, Virtual environments, Virtual Reality, Virtual reality experiences, Virtual spaces, VR systems
@article{casas_holojig_2025,
title = {HoloJig: Interactive Spoken Prompt Specified Generative AI Environments},
author = {L. Casas and S. Hannah and K. Mitchell},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105001182100&doi=10.1109%2FMCG.2025.3553780&partnerID=40&md5=9fafa25e4b6ddc9d2fe32d813fbabb20},
doi = {10.1109/MCG.2025.3553780},
issn = {02721716 (ISSN); 15581756 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Computer Graphics and Applications},
volume = {45},
number = {2},
pages = {69–77},
abstract = {HoloJig offers an interactive, speech-to-virtual reality (VR), VR experience that generates diverse environments in real time based on live spoken descriptions. Unlike traditional VR systems that rely on prebuilt assets, HoloJig dynamically creates personalized and immersive virtual spaces with depth-based parallax 3-D rendering, allowing users to define the characteristics of their immersive environment through verbal prompts. This generative approach opens up new possibilities for interactive experiences, including simulations, training, collaborative workspaces, and entertainment. In addition to speech-to-VR environment generation, a key innovation of HoloJig is its progressive visual transition mechanism, which smoothly dissolves between previously generated and newly requested environments, mitigating the delay caused by neural computations. This feature ensures a seamless and continuous user experience, even as new scenes are being rendered on remote servers. © 2025 Elsevier B.V., All rights reserved.},
note = {Publisher: IEEE Computer Society},
keywords = {3-D rendering, Article, Collaborative workspace, customer experience, Economic and social effects, generative artificial intelligence, human, Immersive, Immersive environment, parallax, Real- time, simulation, Simulation training, speech, Time based, Virtual environments, Virtual Reality, Virtual reality experiences, Virtual spaces, VR systems},
pubstate = {published},
tppubtype = {article}
}
Fernandez, J. A. V.; Lee, J. J.; Vacca, S. A. S.; Magana, A.; Peša, R.; Benes, B.; Popescu, V.
Hands-Free VR Proceedings Article
In: T., Bashford-Rogers; D., Meneveaux; M., Ammi; M., Ziat; S., Jänicke; H., Purchase; P., Radeva; A., Furnari; K., Bouatouch; A.A., Sousa (Ed.): Proc. Int. Jt. Conf. Comput. Vis. Imaging Comput. Graph. Theory Appl., pp. 533–542, Science and Technology Publications, Lda, 2025, ISBN: 21845921 (ISSN).
Abstract | Links | BibTeX | Tags: Deep learning, Large language model, Retrieval-Augmented Generation, Speech-to-Text, Virtual Reality
@inproceedings{fernandez_hands-free_2025,
title = {Hands-Free VR},
author = {J. A. V. Fernandez and J. J. Lee and S. A. S. Vacca and A. Magana and R. Peša and B. Benes and V. Popescu},
editor = {Bashford-Rogers T. and Meneveaux D. and Ammi M. and Ziat M. and Jänicke S. and Purchase H. and Radeva P. and Furnari A. and Bouatouch K. and Sousa A.A.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105001963646&doi=10.5220%2f0013115100003912&partnerID=40&md5=a3f2f4e16bcd5e0579b38e062c987eab},
doi = {10.5220/0013115100003912},
isbn = {21845921 (ISSN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. Int. Jt. Conf. Comput. Vis. Imaging Comput. Graph. Theory Appl.},
volume = {1},
pages = {533–542},
publisher = {Science and Technology Publications, Lda},
abstract = {We introduce Hands-Free VR, a voice-based natural-language interface for VR that allows interaction without additional hardware just using voice. The user voice command is converted into text using a fine-tuned speechto-text deep-learning model. Then, the text is mapped to an executable VR command using an LLM, which is robust to natural language diversity. Hands-Free VR was evaluated in a within-subjects study (N = 22) where participants arranged objects using either a conventional VR interface or Hands-Free VR. The results confirm that Hands-Free VR is: (1) significantly more efficient than conventional VR interfaces in task completion time and user motion metrics; (2) highly rated for ease of use, intuitiveness, ergonomics, reliability, and desirability; (3) robust to English accents (20 participants were non-native speakers) and phonetic similarity, accurately transcribing 96.7% of voice commands, and (3) robust to natural language diversity, mapping 97.83% of transcriptions to executable commands. © 2025 by SCITEPRESS–Science and Technology Publications, Lda.},
keywords = {Deep learning, Large language model, Retrieval-Augmented Generation, Speech-to-Text, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
Wang, W. -S.; Lin, C. -J.; Lee, H. -Y.; Huang, Y. -M.; Wu, T. -T.
Integrating feedback mechanisms and ChatGPT for VR-based experiential learning: impacts on reflective thinking and AIoT physical hands-on tasks Journal Article
In: Interactive Learning Environments, vol. 33, no. 2, pp. 1770–1787, 2025, ISSN: 10494820 (ISSN), (Publisher: Routledge).
Abstract | Links | BibTeX | Tags: AIoT, feedback mechanisms, generative artificial intelligence, physical hands-on tasks, reflective thinking, Virtual Reality
@article{wang_integrating_2025,
title = {Integrating feedback mechanisms and ChatGPT for VR-based experiential learning: impacts on reflective thinking and AIoT physical hands-on tasks},
author = {W. -S. Wang and C. -J. Lin and H. -Y. Lee and Y. -M. Huang and T. -T. Wu},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105001238541&doi=10.1080%2F10494820.2024.2375644&partnerID=40&md5=d75343a9e5969482f384820424b7c58d},
doi = {10.1080/10494820.2024.2375644},
issn = {10494820 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Interactive Learning Environments},
volume = {33},
number = {2},
pages = {1770–1787},
abstract = {This study investigates the application of Virtual Reality (VR) in the educational field, particularly its integration with GAI technologies such as ChatGPT to enhance the learning experience. The research indicates that while VR provides an immersive learning environment fostering student interaction and interest, the lack of a structured learning framework and personalized feedback may limit its educational effectiveness and potentially affect the transfer of VR-learned knowledge to physical hands-on tasks. Hence, it calls for the provision of more targeted and personalized feedback in VR learning environments. Through a randomized controlled trial (RCT), this study collected data from 77 university students, integrating experiential learning in VR for acquiring AIoT knowledge and practical skills, and compared the effects of traditional feedback versus GPT feedback on promoting reflective thinking, learning motivation, cognitive levels, and AIoT hands-on abilities among the students. The results show that the group receiving GPT feedback significantly outperformed the control group across these learning indicators, demonstrating the effectiveness of GAI technologies in providing personalized learning support, facilitating deep learning, and enhancing educational outcomes. This study offers new insights into the integration of GAI technology in VR learning environments, paving new pathways for the development and application of future educational technologies. © 2025 Elsevier B.V., All rights reserved.},
note = {Publisher: Routledge},
keywords = {AIoT, feedback mechanisms, generative artificial intelligence, physical hands-on tasks, reflective thinking, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
Alibrahim, Y.; Ibrahim, M.; Gurdayal, D.; Munshi, M.
AI speechbots and 3D segmentations in virtual reality improve radiology on-call training in resource-limited settings Journal Article
In: Intelligence-Based Medicine, vol. 11, 2025, ISSN: 26665212 (ISSN), (Publisher: Elsevier B.V.).
Abstract | Links | BibTeX | Tags: 3D segmentation, AI speechbots, Article, artificial intelligence chatbot, ChatGPT, computer assisted tomography, Deep learning, headache, human, Image segmentation, interventional radiology, Large language model, Likert scale, nausea, Proof of concept, prospective study, radiology, radiology on call training, resource limited setting, Teaching, Training, ultrasound, Virtual Reality, voice recognition
@article{alibrahim_ai_2025,
title = {AI speechbots and 3D segmentations in virtual reality improve radiology on-call training in resource-limited settings},
author = {Y. Alibrahim and M. Ibrahim and D. Gurdayal and M. Munshi},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105001472313&doi=10.1016%2Fj.ibmed.2025.100245&partnerID=40&md5=981139e173e781b67dba5a46be64de31},
doi = {10.1016/j.ibmed.2025.100245},
issn = {26665212 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Intelligence-Based Medicine},
volume = {11},
abstract = {Objective: Evaluate the use of large-language model (LLM) speechbot tools and deep learning-assisted generation of 3D reconstructions when integrated in a virtual reality (VR) setting to teach radiology on-call topics to radiology residents. Methods: Three first year radiology residents in Guyana were enrolled in an 8-week radiology course that focused on preparation for on-call duties. The course, delivered via VR headsets with custom software integrating LLM-powered speechbots trained on imaging reports and 3D reconstructions segmented with the help of a deep learning model. Each session focused on a specific radiology area, employing a didactic and case-based learning approach, enhanced with 3D reconstructions and an LLM-powered speechbot. Post-session, residents reassessed their knowledge and provided feedback on their VR and LLM-powered speechbot experiences. Results/discussion: Residents found that the 3D reconstructions segmented semi-automatically by deep learning algorithms and AI-driven self-learning via speechbot was highly valuable. The 3D reconstructions, especially in the interventional radiology session, were helpful and the benefit is augmented by VR where navigating the models is seamless and perception of depth is pronounced. Residents also found conversing with the AI-speechbot seamless and was valuable in their post session self-learning. The major drawback of VR was motion sickness, which was mild and improved over time. Conclusion: AI-assisted VR radiology education could be used to develop new and accessible ways of teaching a variety of radiology topics in a seamless and cost-effective way. This could be especially useful in supporting radiology education remotely in regions which lack local radiology expertise. © 2025 Elsevier B.V., All rights reserved.},
note = {Publisher: Elsevier B.V.},
keywords = {3D segmentation, AI speechbots, Article, artificial intelligence chatbot, ChatGPT, computer assisted tomography, Deep learning, headache, human, Image segmentation, interventional radiology, Large language model, Likert scale, nausea, Proof of concept, prospective study, radiology, radiology on call training, resource limited setting, Teaching, Training, ultrasound, Virtual Reality, voice recognition},
pubstate = {published},
tppubtype = {article}
}
Chen, J.; Grubert, J.; Kristensson, P. O.
Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes Proceedings Article
In: Proc. - IEEE Conf. Virtual Real. 3D User Interfaces, VR, pp. 206–216, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 9798331536459 (ISBN).
Abstract | Links | BibTeX | Tags: 3D modeling, 3D reconstruction, 3D scene editing, 3D scenes, Computer simulation languages, Editing systems, Immersive environment, Interaction pattern, Interaction strategy, Language Model, Large language model, large language models, Multimodal Interaction, Scene editing, Three dimensional computer graphics, Virtual environments, Virtual Reality
@inproceedings{chen_analyzing_2025,
title = {Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes},
author = {J. Chen and J. Grubert and P. O. Kristensson},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105002716635&doi=10.1109%2FVR59515.2025.00045&partnerID=40&md5=9db6769cd401503605578c4b711152b9},
doi = {10.1109/VR59515.2025.00045},
isbn = {9798331536459 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. - IEEE Conf. Virtual Real. 3D User Interfaces, VR},
pages = {206–216},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {As more applications of large language models (LLMs) for 3D content in immersive environments emerge, it is crucial to study user behavior to identify interaction patterns and potential barriers to guide the future design of immersive content creation and editing systems which involve LLMs. In an empirical user study with 12 participants, we combine quantitative usage data with post-experience questionnaire feedback to reveal common interaction patterns and key barriers in LLM-assisted 3D scene editing systems. We identify opportunities for improving natural language interfaces in 3D design tools and propose design recommendations. Through an empirical study, we demonstrate that LLM-assisted interactive systems can be used productively in immersive environments. © 2025 Elsevier B.V., All rights reserved.},
keywords = {3D modeling, 3D reconstruction, 3D scene editing, 3D scenes, Computer simulation languages, Editing systems, Immersive environment, Interaction pattern, Interaction strategy, Language Model, Large language model, large language models, Multimodal Interaction, Scene editing, Three dimensional computer graphics, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
Li, Y.; Pang, E. C. H.; Ng, C. S. Y.; Azim, M.; Leung, H.
Enhancing Linear Algebra Education with AI-Generated Content in the CityU Metaverse: A Comparative Study Proceedings Article
In: T., Hao; J.G., Wu; X., Luo; Y., Sun; Y., Mu; S., Ge; W., Xie (Ed.): Lect. Notes Comput. Sci., pp. 3–16, Springer Science and Business Media Deutschland GmbH, 2025, ISBN: 03029743 (ISSN); 978-981964406-3 (ISBN).
Abstract | Links | BibTeX | Tags: Comparatives studies, Digital age, Digital interactions, digital twin, Educational metaverse, Engineering education, Generative AI, Immersive, Matrix algebra, Metaverse, Metaverses, Personnel training, Students, Teaching, University campus, Virtual environments, virtual learning environment, Virtual learning environments, Virtual Reality, Virtualization
@inproceedings{li_enhancing_2025,
title = {Enhancing Linear Algebra Education with AI-Generated Content in the CityU Metaverse: A Comparative Study},
author = {Y. Li and E. C. H. Pang and C. S. Y. Ng and M. Azim and H. Leung},
editor = {Hao T. and Wu J.G. and Luo X. and Sun Y. and Mu Y. and Ge S. and Xie W.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105003632691&doi=10.1007%2f978-981-96-4407-0_1&partnerID=40&md5=c067ba5d4c15e9c0353bf315680531fc},
doi = {10.1007/978-981-96-4407-0_1},
isbn = {03029743 (ISSN); 978-981964406-3 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Lect. Notes Comput. Sci.},
volume = {15589 LNCS},
pages = {3–16},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {In today’s digital age, the metaverse is emerging as the forthcoming evolution of the internet. It provides an immersive space that marks a new frontier in the way digital interactions are facilitated and experienced. In this paper, we present the CityU Metaverse, which aims to construct a digital twin of our university campus. It is designed as an educational virtual world where learning applications can be embedded in this virtual campus, supporting not only remote and collaborative learning but also professional technical training to enhance educational experiences through immersive and interactive learning. To evaluate the effectiveness of this educational metaverse, we conducted an experiment focused on 3D linear transformation in linear algebra, with teaching content generated by generative AI, comparing our metaverse system with traditional teaching methods. Knowledge tests and surveys assessing learning interest revealed that students engaged with the CityU Metaverse, facilitated by AI-generated content, outperformed those in traditional settings and reported greater enjoyment during the learning process. The work provides valuable perspectives on the behaviors and interactions within the metaverse by analyzing user preferences and learning outcomes. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.},
keywords = {Comparatives studies, Digital age, Digital interactions, digital twin, Educational metaverse, Engineering education, Generative AI, Immersive, Matrix algebra, Metaverse, Metaverses, Personnel training, Students, Teaching, University campus, Virtual environments, virtual learning environment, Virtual learning environments, Virtual Reality, Virtualization},
pubstate = {published},
tppubtype = {inproceedings}
}
Song, T.; Pabst, F.; Eck, U.; Navab, N.
Enhancing Patient Acceptance of Robotic Ultrasound through Conversational Virtual Agent and Immersive Visualizations Journal Article
In: IEEE Transactions on Visualization and Computer Graphics, vol. 31, no. 5, pp. 2901–2911, 2025, ISSN: 10772626 (ISSN), (Publisher: IEEE Computer Society).
Abstract | Links | BibTeX | Tags: 3D reconstruction, adult, Augmented Reality, Computer graphics, computer interface, echography, female, human, Humans, Imaging, Intelligent robots, Intelligent virtual agents, Language Model, male, Medical robotics, Middle Aged, Mixed reality, Patient Acceptance of Health Care, patient attitude, Patient comfort, procedures, Real-world, Reality visualization, Robotic Ultrasound, Robotics, Three-Dimensional, three-dimensional imaging, Trust and Acceptance, Ultrasonic applications, Ultrasonic equipment, Ultrasonography, Ultrasound probes, User-Computer Interface, Virtual agent, Virtual assistants, Virtual environments, Virtual Reality, Visual languages, Visualization, Young Adult
@article{song_enhancing_2025,
title = {Enhancing Patient Acceptance of Robotic Ultrasound through Conversational Virtual Agent and Immersive Visualizations},
author = {T. Song and F. Pabst and U. Eck and N. Navab},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105003687673&doi=10.1109%2FTVCG.2025.3549181&partnerID=40&md5=0753cd3c57ac630480a19001cde28319},
doi = {10.1109/TVCG.2025.3549181},
issn = {10772626 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Transactions on Visualization and Computer Graphics},
volume = {31},
number = {5},
pages = {2901–2911},
abstract = {Robotic ultrasound systems have the potential to improve medical diagnostics, but patient acceptance remains a key challenge. To address this, we propose a novel system that combines an AI-based virtual agent, powered by a large language model (LLM), with three mixed reality visualizations aimed at enhancing patient comfort and trust. The LLM enables the virtual assistant to engage in natural, conversational dialogue with patients, answering questions in any format and offering real-time reassurance, creating a more intelligent and reliable interaction. The virtual assistant is animated as controlling the ultrasound probe, giving the impression that the robot is guided by the assistant. The first visualization employs augmented reality (AR), allowing patients to see the real world and the robot with the virtual avatar superimposed. The second visualization is an augmented virtuality (AV) environment, where the real-world body part being scanned is visible, while a 3D Gaussian Splatting reconstruction of the room, excluding the robot, forms the virtual environment. The third is a fully immersive virtual reality (VR) experience, featuring the same 3D reconstruction but entirely virtual, where the patient sees a virtual representation of their body being scanned in a robot-free environment. In this case, the virtual ultrasound probe, mirrors the movement of the probe controlled by the robot, creating a synchronized experience as it touches and moves over the patient's virtual body. We conducted a comprehensive agent-guided robotic ultrasound study with all participants, comparing these visualizations against a standard robotic ultrasound procedure. Results showed significant improvements in patient trust, acceptance, and comfort. Based on these findings, we offer insights into designing future mixed reality visualizations and virtual agents to further enhance patient comfort and acceptance in autonomous medical procedures. © 2025 Elsevier B.V., All rights reserved.},
note = {Publisher: IEEE Computer Society},
keywords = {3D reconstruction, adult, Augmented Reality, Computer graphics, computer interface, echography, female, human, Humans, Imaging, Intelligent robots, Intelligent virtual agents, Language Model, male, Medical robotics, Middle Aged, Mixed reality, Patient Acceptance of Health Care, patient attitude, Patient comfort, procedures, Real-world, Reality visualization, Robotic Ultrasound, Robotics, Three-Dimensional, three-dimensional imaging, Trust and Acceptance, Ultrasonic applications, Ultrasonic equipment, Ultrasonography, Ultrasound probes, User-Computer Interface, Virtual agent, Virtual assistants, Virtual environments, Virtual Reality, Visual languages, Visualization, Young Adult},
pubstate = {published},
tppubtype = {article}
}
Stacchio, L.; Balloni, E.; Frontoni, E.; Paolanti, M.; Zingaretti, P.; Pierdicca, R.
MineVRA: Exploring the Role of Generative AI-Driven Content Development in XR Environments through a Context-Aware Approach Journal Article
In: IEEE Transactions on Visualization and Computer Graphics, vol. 31, no. 5, pp. 3602–3612, 2025, ISSN: 10772626 (ISSN), (Publisher: IEEE Computer Society).
Abstract | Links | BibTeX | Tags: adult, Article, Artificial intelligence, Computer graphics, Computer vision, Content Development, Contents development, Context-Aware, Context-aware approaches, Extended reality, female, Generative adversarial networks, Generative AI, generative artificial intelligence, human, Human-in-the-loop, Immersive, Immersive environment, male, Multi-modal, User need, Virtual environments, Virtual Reality
@article{stacchio_minevra_2025,
title = {MineVRA: Exploring the Role of Generative AI-Driven Content Development in XR Environments through a Context-Aware Approach},
author = {L. Stacchio and E. Balloni and E. Frontoni and M. Paolanti and P. Zingaretti and R. Pierdicca},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105003746367&doi=10.1109%2FTVCG.2025.3549160&partnerID=40&md5=3356eb968b3e6a0d3c9b75716b05fac4},
doi = {10.1109/TVCG.2025.3549160},
issn = {10772626 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Transactions on Visualization and Computer Graphics},
volume = {31},
number = {5},
pages = {3602–3612},
abstract = {The convergence of Artificial Intelligence (AI), Computer Vision (CV), Computer Graphics (CG), and Extended Reality (XR) is driving innovation in immersive environments. A key challenge in these environments is the creation of personalized 3D assets, traditionally achieved through manual modeling, a time-consuming process that often fails to meet individual user needs. More recently, Generative AI (GenAI) has emerged as a promising solution for automated, context-aware content generation. In this paper, we present MineVRA (Multimodal generative artificial iNtelligence for contExt-aware Virtual Reality Assets), a novel Human-In-The-Loop (HITL) XR framework that integrates GenAI to facilitate coherent and adaptive 3D content generation in immersive scenarios. To evaluate the effectiveness of this approach, we conducted a comparative user study analyzing the performance and user satisfaction of GenAI-generated 3D objects compared to those generated by Sketchfab in different immersive contexts. The results suggest that GenAI can significantly complement traditional 3D asset libraries, with valuable design implications for the development of human-centered XR environments. © 2025 Elsevier B.V., All rights reserved.},
note = {Publisher: IEEE Computer Society},
keywords = {adult, Article, Artificial intelligence, Computer graphics, Computer vision, Content Development, Contents development, Context-Aware, Context-aware approaches, Extended reality, female, Generative adversarial networks, Generative AI, generative artificial intelligence, human, Human-in-the-loop, Immersive, Immersive environment, male, Multi-modal, User need, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
Kim, Y.; Aamir, Z.; Singh, M.; Boorboor, S.; Mueller, K.; Kaufman, A. E.
Explainable XR: Understanding User Behaviors of XR Environments Using LLM-Assisted Analytics Framework Journal Article
In: IEEE Transactions on Visualization and Computer Graphics, vol. 31, no. 5, pp. 2756–2766, 2025, ISSN: 10772626 (ISSN), (Publisher: IEEE Computer Society).
Abstract | Links | BibTeX | Tags: adult, Agnostic, Article, Assistive, Cross Reality, Data Analytics, Data collection, data interpretation, Data recording, Data visualization, Extended reality, human, Language Model, Large language model, large language models, Multi-modal, Multimodal Data Collection, normal human, Personalized assistive technique, Personalized Assistive Techniques, recorder, Spatio-temporal data, therapy, user behavior, User behaviors, Virtual addresses, Virtual environments, Virtual Reality, Visual analytics, Visual languages
@article{kim_explainable_2025,
title = {Explainable XR: Understanding User Behaviors of XR Environments Using LLM-Assisted Analytics Framework},
author = {Y. Kim and Z. Aamir and M. Singh and S. Boorboor and K. Mueller and A. E. Kaufman},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105003815583&doi=10.1109%2FTVCG.2025.3549537&partnerID=40&md5=bc5ac38eb19faa224282cf385f43799f},
doi = {10.1109/TVCG.2025.3549537},
issn = {10772626 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Transactions on Visualization and Computer Graphics},
volume = {31},
number = {5},
pages = {2756–2766},
abstract = {We present Explainable XR, an end-to-end framework for analyzing user behavior in diverse eXtended Reality (XR) environments by leveraging Large Language Models (LLMs) for data interpretation assistance. Existing XR user analytics frameworks face challenges in handling cross-virtuality - AR, VR, MR - transitions, multi-user collaborative application scenarios, and the complexity of multimodal data. Explainable XR addresses these challenges by providing a virtuality-agnostic solution for the collection, analysis, and visualization of immersive sessions. We propose three main components in our framework: (1) A novel user data recording schema, called User Action Descriptor (UAD), that can capture the users' multimodal actions, along with their intents and the contexts; (2) a platform-agnostic XR session recorder, and (3) a visual analytics interface that offers LLM-assisted insights tailored to the analysts' perspectives, facilitating the exploration and analysis of the recorded XR session data. We demonstrate the versatility of Explainable XR by demonstrating five use-case scenarios, in both individual and collaborative XR applications across virtualities. Our technical evaluation and user studies show that Explainable XR provides a highly usable analytics solution for understanding user actions and delivering multifaceted, actionable insights into user behaviors in immersive environments. © 2025 Elsevier B.V., All rights reserved.},
note = {Publisher: IEEE Computer Society},
keywords = {adult, Agnostic, Article, Assistive, Cross Reality, Data Analytics, Data collection, data interpretation, Data recording, Data visualization, Extended reality, human, Language Model, Large language model, large language models, Multi-modal, Multimodal Data Collection, normal human, Personalized assistive technique, Personalized Assistive Techniques, recorder, Spatio-temporal data, therapy, user behavior, User behaviors, Virtual addresses, Virtual environments, Virtual Reality, Visual analytics, Visual languages},
pubstate = {published},
tppubtype = {article}
}
Chen, J.; Wu, X.; Lan, T.; Li, B.
LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models Journal Article
In: IEEE Transactions on Visualization and Computer Graphics, vol. 31, no. 5, pp. 2715–2724, 2025, ISSN: 10772626 (ISSN), (Publisher: IEEE Computer Society).
Abstract | Links | BibTeX | Tags: % reductions, 3D modeling, algorithm, Algorithms, Augmented Reality, Coding errors, Computer graphics, Computer interaction, computer interface, Computer simulation languages, Extended reality, generative artificial intelligence, human, Human users, human-computer interaction, Humans, Imaging, Immersive, Language, Language Model, Large language model, large language models, Metadata, Natural Language Processing, Natural language processing systems, Natural languages, procedures, Script generation, Spatio-temporal data, Three dimensional computer graphics, Three-Dimensional, three-dimensional imaging, User-Computer Interface, Virtual Reality
@article{chen_llmer_2025,
title = {LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models},
author = {J. Chen and X. Wu and T. Lan and B. Li},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105003825793&doi=10.1109%2FTVCG.2025.3549549&partnerID=40&md5=50597473616678390f143a33082a13d3},
doi = {10.1109/TVCG.2025.3549549},
issn = {10772626 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Transactions on Visualization and Computer Graphics},
volume = {31},
number = {5},
pages = {2715–2724},
abstract = {The integration of Large Language Models (LLMs) like GPT-4 with Extended Reality (XR) technologies offers the potential to build truly immersive XR environments that interact with human users through natural language, e.g., generating and animating 3D scenes from audio inputs. However, the complexity of XR environments makes it difficult to accurately extract relevant contextual data and scene/object parameters from an overwhelming volume of XR artifacts. It leads to not only increased costs with pay-per-use models, but also elevated levels of generation errors. Moreover, existing approaches focusing on coding script generation are often prone to generation errors, resulting in flawed or invalid scripts, application crashes, and ultimately a degraded user experience. To overcome these challenges, we introduce LLMER, a novel framework that creates interactive XR worlds using JSON data generated by LLMs. Unlike prior approaches focusing on coding script generation, LLMER translates natural language inputs into JSON data, significantly reducing the likelihood of application crashes and processing latency. It employs a multi-stage strategy to supply only the essential contextual information adapted to the user's request and features multiple modules designed for various XR tasks. Our preliminary user study reveals the effectiveness of the proposed system, with over 80% reduction in consumed tokens and around 60% reduction in task completion time compared to state-of-the-art approaches. The analysis of users' feedback also illuminates a series of directions for further optimization. © 2025 Elsevier B.V., All rights reserved.},
note = {Publisher: IEEE Computer Society},
keywords = {% reductions, 3D modeling, algorithm, Algorithms, Augmented Reality, Coding errors, Computer graphics, Computer interaction, computer interface, Computer simulation languages, Extended reality, generative artificial intelligence, human, Human users, human-computer interaction, Humans, Imaging, Immersive, Language, Language Model, Large language model, large language models, Metadata, Natural Language Processing, Natural language processing systems, Natural languages, procedures, Script generation, Spatio-temporal data, Three dimensional computer graphics, Three-Dimensional, three-dimensional imaging, User-Computer Interface, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
Li, Z.; Zhang, H.; Peng, C.; Peiris, R.
Exploring Large Language Model-Driven Agents for Environment-Aware Spatial Interactions and Conversations in Virtual Reality Role-Play Scenarios Proceedings Article
In: Proc. - IEEE Conf. Virtual Real. 3D User Interfaces, VR, pp. 1–11, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 9798331536459 (ISBN).
Abstract | Links | BibTeX | Tags: Chatbots, Computer simulation languages, Context- awareness, context-awareness, Digital elevation model, Generative AI, Human-AI Interaction, Language Model, Large language model, large language models, Model agents, Role-play simulation, role-play simulations, Role-plays, Spatial interaction, Virtual environments, Virtual Reality, Virtual-reality environment
@inproceedings{li_exploring_2025,
title = {Exploring Large Language Model-Driven Agents for Environment-Aware Spatial Interactions and Conversations in Virtual Reality Role-Play Scenarios},
author = {Z. Li and H. Zhang and C. Peng and R. Peiris},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105002706893&doi=10.1109%2FVR59515.2025.00025&partnerID=40&md5=1987c128f6ec4bd24011388ef9ece179},
doi = {10.1109/VR59515.2025.00025},
isbn = {9798331536459 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. - IEEE Conf. Virtual Real. 3D User Interfaces, VR},
pages = {1–11},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Recent research has begun adopting Large Language Model (LLM) agents to enhance Virtual Reality (VR) interactions, creating immersive chatbot experiences. However, while current studies focus on generating dialogue from user speech inputs, their abilities to generate richer experiences based on the perception of LLM agents' VR environments and interaction cues remain unexplored. Hence, in this work, we propose an approach that enables LLM agents to perceive virtual environments and generate environment-aware interactions and conversations for an embodied human-AI interaction experience in VR environments. Here, we define a schema for describing VR environments and their interactions through text prompts. We evaluate the performance of our method through five role-play scenarios created using our approach in a study with 14 participants. The findings discuss the opportunities and challenges of our proposed approach for developing environment-aware LLM agents that facilitate spatial interactions and conversations within VR role-play scenarios. © 2025 Elsevier B.V., All rights reserved.},
keywords = {Chatbots, Computer simulation languages, Context- awareness, context-awareness, Digital elevation model, Generative AI, Human-AI Interaction, Language Model, Large language model, large language models, Model agents, Role-play simulation, role-play simulations, Role-plays, Spatial interaction, Virtual environments, Virtual Reality, Virtual-reality environment},
pubstate = {published},
tppubtype = {inproceedings}
}
Afzal, M. Z.; Ali, S. K. A.; Stricker, D.; Eisert, P.; Hilsmann, A.; Pérez-Marcos, D.; Bianchi, M.; Crottaz-Herbette, S.; Ioris, R.; Mangina, E.; Sanguineti, M.; Salaberria, A.; de Lacalle, O. Lopez; García-Pablos, A.; Cuadros, M.
Next Generation XR Systems - Large Language Models Meet Augmented and Virtual Reality Journal Article
In: IEEE Computer Graphics and Applications, vol. 45, no. 1, pp. 43–55, 2025, ISSN: 02721716 (ISSN); 15581756 (ISSN), (Publisher: IEEE Computer Society).
Abstract | Links | BibTeX | Tags: adult, Article, Augmented and virtual realities, Augmented Reality, Awareness, Context-Aware, human, Information Retrieval, Knowledge model, Knowledge reasoning, Knowledge retrieval, Language Model, Large language model, Mixed reality, neurorehabilitation, Position papers, privacy, Real- time, Reasoning, Situational awareness, Virtual environments, Virtual Reality
@article{afzal_next_2025,
title = {Next Generation XR Systems - Large Language Models Meet Augmented and Virtual Reality},
author = {M. Z. Afzal and S. K. A. Ali and D. Stricker and P. Eisert and A. Hilsmann and D. Pérez-Marcos and M. Bianchi and S. Crottaz-Herbette and R. Ioris and E. Mangina and M. Sanguineti and A. Salaberria and O. Lopez de Lacalle and A. García-Pablos and M. Cuadros},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105003598602&doi=10.1109%2FMCG.2025.3548554&partnerID=40&md5=94e7efe987708afc9f066b906ce232b1},
doi = {10.1109/MCG.2025.3548554},
issn = {02721716 (ISSN); 15581756 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Computer Graphics and Applications},
volume = {45},
number = {1},
pages = {43–55},
abstract = {Extended reality (XR) is evolving rapidly, offering new paradigms for human-computer interaction. This position paper argues that integrating large language models (LLMs) with XR systems represents a fundamental shift toward more intelligent, context-aware, and adaptive mixed-reality experiences. We propose a structured framework built on three key pillars: first, perception and situational awareness, second, knowledge modeling and reasoning, and third, visualization and interaction. We believe leveraging LLMs within XR environments enables enhanced situational awareness, real-time knowledge retrieval, and dynamic user interaction, surpassing traditional XR capabilities. We highlight the potential of this integration in neurorehabilitation, safety training, and architectural design while underscoring ethical considerations, such as privacy, transparency, and inclusivity. This vision aims to spark discussion and drive research toward more intelligent, human-centric XR systems. © 2025 Elsevier B.V., All rights reserved.},
note = {Publisher: IEEE Computer Society},
keywords = {adult, Article, Augmented and virtual realities, Augmented Reality, Awareness, Context-Aware, human, Information Retrieval, Knowledge model, Knowledge reasoning, Knowledge retrieval, Language Model, Large language model, Mixed reality, neurorehabilitation, Position papers, privacy, Real- time, Reasoning, Situational awareness, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
Haoyang, H.; Wang, Z.; Liang, W.; Wang, Y.
X’s Day: Personality-Driven Virtual Human Behavior Generation Journal Article
In: IEEE Transactions on Visualization and Computer Graphics, vol. 31, no. 5, pp. 3514–3524, 2025, ISSN: 10772626 (ISSN), (Publisher: IEEE Computer Society).
Abstract | Links | BibTeX | Tags: adult, Augmented Reality, Behavior Generation, Chatbots, Computer graphics, computer interface, Contextual Scene, female, human, Human behaviors, Humans, Long-term behavior, male, Novel task, Personality, Personality traits, Personality-driven Behavior, physiology, Social behavior, User-Computer Interface, Users' experiences, Virtual agent, Virtual environments, Virtual humans, Virtual Reality, Young Adult
@article{haoyang_xs_2025,
title = {X’s Day: Personality-Driven Virtual Human Behavior Generation},
author = {H. Haoyang and Z. Wang and W. Liang and Y. Wang},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105003864932&doi=10.1109%2FTVCG.2025.3549574&partnerID=40&md5=38fc6613a7c21a90f3738c048497d870},
doi = {10.1109/TVCG.2025.3549574},
issn = {10772626 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Transactions on Visualization and Computer Graphics},
volume = {31},
number = {5},
pages = {3514–3524},
abstract = {Developing convincing and realistic virtual human behavior is essential for enhancing user experiences in virtual reality (VR) and augmented reality (AR) settings. This paper introduces a novel task focused on generating long-term behaviors for virtual agents, guided by specific personality traits and contextual elements within 3D environments. We present a comprehensive framework capable of autonomously producing daily activities autoregressively. By modeling the intricate connections between personality characteristics and observable activities, we establish a hierarchical structure of Needs, Task, and Activity levels. Integrating a Behavior Planner and a World State module allows for the dynamic sampling of behaviors using large language models (LLMs), ensuring that generated activities remain relevant and responsive to environmental changes. Extensive experiments validate the effectiveness and adaptability of our approach across diverse scenarios. This research makes a significant contribution to the field by establishing a new paradigm for personalized and context-aware interactions with virtual humans, ultimately enhancing user engagement in immersive applications. Our project website is at: https://behavior.agent-x.cn/. © 2025 Elsevier B.V., All rights reserved.},
note = {Publisher: IEEE Computer Society},
keywords = {adult, Augmented Reality, Behavior Generation, Chatbots, Computer graphics, computer interface, Contextual Scene, female, human, Human behaviors, Humans, Long-term behavior, male, Novel task, Personality, Personality traits, Personality-driven Behavior, physiology, Social behavior, User-Computer Interface, Users' experiences, Virtual agent, Virtual environments, Virtual humans, Virtual Reality, Young Adult},
pubstate = {published},
tppubtype = {article}
}
Ly, D. -N.; Do, H. -N.; Tran, M. -T.; Le, K. -D.
Evaluation of AI-Based Assistant Representations on User Interaction in Virtual Explorations Proceedings Article
In: W., Buntine; M., Fjeld; T., Tran; M.-T., Tran; B., Huynh Thi Thanh; T., Miyoshi (Ed.): Commun. Comput. Info. Sci., pp. 323–337, Springer Science and Business Media Deutschland GmbH, 2025, ISBN: 18650929 (ISSN); 978-981964287-8 (ISBN).
Abstract | Links | BibTeX | Tags: 360-degree Video, AI-Based Assistant, Cultural heritages, Cultural science, Multiusers, Single users, Social interactions, Three dimensional computer graphics, User interaction, Users' experiences, Virtual environments, Virtual Exploration, Virtual Reality, Virtualization
@inproceedings{ly_evaluation_2025,
title = {Evaluation of AI-Based Assistant Representations on User Interaction in Virtual Explorations},
author = {D. -N. Ly and H. -N. Do and M. -T. Tran and K. -D. Le},
editor = {Buntine W. and Fjeld M. and Tran T. and Tran M.-T. and Huynh Thi Thanh B. and Miyoshi T.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105004253350&doi=10.1007%2f978-981-96-4288-5_26&partnerID=40&md5=5f0a8c1e356cd3bdd4dda7f96f272154},
doi = {10.1007/978-981-96-4288-5_26},
isbn = {18650929 (ISSN); 978-981964287-8 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Commun. Comput. Info. Sci.},
volume = {2352 CCIS},
pages = {323–337},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {Exploration activities, such as tourism, cultural heritage, and science, enhance knowledge and understanding. The rise of 360-degree videos allows users to explore cultural landmarks and destinations remotely. While multi-user VR environments encourage collaboration, single-user experiences often lack social interaction. Generative AI, particularly Large Language Models (LLMs), offer a way to improve single-user VR exploration through AI-driven virtual assistants, acting as tour guides or storytellers. However, it’s uncertain whether these assistants require a visual presence, and if so, what form it should take. To investigate this, we developed an AI-based assistant in three different forms: a voice-only avatar, a 3D human-sized avatar, and a mini-hologram avatar, and conducted a user study to evaluate their impact on user experience. The study, which involved 12 participants, found that the visual embodiments significantly reduce feelings of being alone, with distinct user preferences between the Human-sized avatar and the Mini hologram. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.},
keywords = {360-degree Video, AI-Based Assistant, Cultural heritages, Cultural science, Multiusers, Single users, Social interactions, Three dimensional computer graphics, User interaction, Users' experiences, Virtual environments, Virtual Exploration, Virtual Reality, Virtualization},
pubstate = {published},
tppubtype = {inproceedings}
}
Shen, Y.; Li, B.; Huang, J.; Wang, Z.
GaussianShopVR: Facilitating Immersive 3D Authoring Using Gaussian Splatting in VR Proceedings Article
In: Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW, pp. 1292–1293, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 9798331514846 (ISBN).
Abstract | Links | BibTeX | Tags: 3D authoring, 3D modeling, Digital replicas, Gaussian distribution, Gaussian Splatting editing, Gaussians, Graphical user interfaces, High quality, Immersive, Immersive environment, Interactive computer graphics, Rendering (computer graphics), Rendering pipelines, Splatting, Three dimensional computer graphics, User profile, Virtual Reality, Virtual reality user interface, Virtualization, VR user interface
@inproceedings{shen_gaussianshopvr_2025,
title = {GaussianShopVR: Facilitating Immersive 3D Authoring Using Gaussian Splatting in VR},
author = {Y. Shen and B. Li and J. Huang and Z. Wang},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005138672&doi=10.1109%2FVRW66409.2025.00292&partnerID=40&md5=2290016d250649f8d7f262212b1f59cb},
doi = {10.1109/VRW66409.2025.00292},
isbn = {9798331514846 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW},
pages = {1292–1293},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Virtual reality (VR) applications require massive high-quality 3D assets to create immersive environments. Generating mesh-based 3D assets typically involves a significant amount of manpower and effort, which makes VR applications less accessible. 3D Gaussian Splatting (3DGS) has attracted much attention for its ability to quickly create digital replicas of real-life scenes and its compatibility with traditional rendering pipelines. However, it remains a challenge to edit 3DGS in a flexible and controllable manner. We propose GaussianShopVR, a system that leverages VR user interfaces to specify target areas to achieve flexible and controllable editing of reconstructed 3DGS. In addition, selected areas can provide 3D information to generative AI models to facilitate the editing. GaussianShopVR integrates object hierarchy management while keeping the backpropagated gradient flow to allow local editing with context information. © 2025 Elsevier B.V., All rights reserved.},
keywords = {3D authoring, 3D modeling, Digital replicas, Gaussian distribution, Gaussian Splatting editing, Gaussians, Graphical user interfaces, High quality, Immersive, Immersive environment, Interactive computer graphics, Rendering (computer graphics), Rendering pipelines, Splatting, Three dimensional computer graphics, User profile, Virtual Reality, Virtual reality user interface, Virtualization, VR user interface},
pubstate = {published},
tppubtype = {inproceedings}
}
Ding, S.; Chen, Y.
RAG-VR: Leveraging Retrieval-Augmented Generation for 3D Question Answering in VR Environments Proceedings Article
In: Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW, pp. 131–136, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 9798331514846 (ISBN).
Abstract | Links | BibTeX | Tags: Ambient intelligence, Computational Linguistics, Computer interaction, Computing methodologies, Computing methodologies-Artificial intelligence-Natural language processing-Natural language generation, Computing methodology-artificial intelligence-natural language processing-natural language generation, Data handling, Formal languages, Human computer interaction, Human computer interaction (HCI), Human-centered computing, Interaction paradigm, Interaction paradigms, Language Model, Language processing, Natural language generation, Natural language processing systems, Natural languages, Virtual Reality, Word processing
@inproceedings{ding_rag-vr_2025,
title = {RAG-VR: Leveraging Retrieval-Augmented Generation for 3D Question Answering in VR Environments},
author = {S. Ding and Y. Chen},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005140593&doi=10.1109%2FVRW66409.2025.00034&partnerID=40&md5=0bd7d96a9bf05f93d17850cd3b380ff4},
doi = {10.1109/VRW66409.2025.00034},
isbn = {9798331514846 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW},
pages = {131–136},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Recent advances in large language models (LLMs) provide new opportunities for context understanding in virtual reality (VR). However, VR contexts are often highly localized and personalized, limiting the effectiveness of general-purpose LLMs. To address this challenge, we present RAG-VR, the first 3D question-answering system for VR that incorporates retrieval-augmented generation (RAG), which augments an LLM with external knowledge retrieved from a localized knowledge database to improve the answer quality. RAG-VR includes a pipeline for extracting comprehensive knowledge about virtual environments and user conditions for accurate answer generation. To ensure efficient retrieval, RAG-VR offloads the retrieval process to a nearby edge server and uses only essential information during retrieval. Moreover, we train the retriever to effectively distinguish among relevant, irrelevant, and hard-to-differentiate information in relation to questions. RAG-VR improves answer accuracy by 17.9%-41.8% and reduces end-to-end latency by 34.5%-47.3% compared with two baseline systems. © 2025 Elsevier B.V., All rights reserved.},
keywords = {Ambient intelligence, Computational Linguistics, Computer interaction, Computing methodologies, Computing methodologies-Artificial intelligence-Natural language processing-Natural language generation, Computing methodology-artificial intelligence-natural language processing-natural language generation, Data handling, Formal languages, Human computer interaction, Human computer interaction (HCI), Human-centered computing, Interaction paradigm, Interaction paradigms, Language Model, Language processing, Natural language generation, Natural language processing systems, Natural languages, Virtual Reality, Word processing},
pubstate = {published},
tppubtype = {inproceedings}
}