AHCI RESEARCH GROUP
Publications
Papers published in international journals,
proceedings of conferences, workshops and books.
OUR RESEARCH
Scientific Publications
How to
Here you can find the complete list of our publications.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
2025
Zhang, H.; Chen, P.; Xie, X.; Jiang, Z.; Wu, Y.; Li, Z.; Chen, X.; Sun, L.
FusionProtor: A Mixed-Prototype Tool for Component-level Physical-to-Virtual 3D Transition and Simulation Proceedings Article
In: Conf Hum Fact Comput Syst Proc, Association for Computing Machinery, 2025, ISBN: 979-840071394-1 (ISBN).
Abstract | Links | BibTeX | Tags: 3D modeling, 3D prototype, 3D simulations, 3d transition, Component levels, Conceptual design, Creatives, Generative AI, High-fidelity, Integrated circuit layout, Mixed reality, Product conceptual designs, Prototype tools, Prototype workflow, Three dimensional computer graphics, Usability engineering, Virtual Prototyping
@inproceedings{zhang_fusionprotor_2025,
title = {FusionProtor: A Mixed-Prototype Tool for Component-level Physical-to-Virtual 3D Transition and Simulation},
author = {H. Zhang and P. Chen and X. Xie and Z. Jiang and Y. Wu and Z. Li and X. Chen and L. Sun},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005745450&doi=10.1145%2f3706598.3713686&partnerID=40&md5=e51eac0cc99293538422d98a4070cd09},
doi = {10.1145/3706598.3713686},
isbn = {979-840071394-1 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Conf Hum Fact Comput Syst Proc},
publisher = {Association for Computing Machinery},
abstract = {Developing and simulating 3D prototypes is crucial in product conceptual design for ideation and presentation. Traditional methods often keep physical and virtual prototypes separate, leading to a disjointed prototype workflow. In addition, acquiring high-fidelity prototypes is time-consuming and resource-intensive, distracting designers from creative exploration. Recent advancements in generative artificial intelligence (GAI) and extended reality (XR) provided new solutions for rapid prototype transition and mixed simulation. We conducted a formative study to understand current challenges in the traditional prototype process and explore how to effectively utilize GAI and XR ability in prototype. Then we introduced FusionProtor, a mixed-prototype tool for component-level 3D prototype transition and simulation. We proposed a step-by-step generation pipeline in FusionProtor, effectively transiting 3D prototypes from physical to virtual and low- to high-fidelity for rapid ideation and iteration. We also innovated a component-level 3D creation method and applied it in XR environment for the mixed-prototype presentation and interaction. We conducted technical and user experiments to verify FusionProtor's usability in supporting diverse designs. Our results verified that it achieved a seamless workflow between physical and virtual domains, enhancing efficiency and promoting ideation. We also explored the effect of mixed interaction on design and critically discussed its best practices for HCI community. © 2025 Copyright held by the owner/author(s). Publication rights licensed to ACM.},
keywords = {3D modeling, 3D prototype, 3D simulations, 3d transition, Component levels, Conceptual design, Creatives, Generative AI, High-fidelity, Integrated circuit layout, Mixed reality, Product conceptual designs, Prototype tools, Prototype workflow, Three dimensional computer graphics, Usability engineering, Virtual Prototyping},
pubstate = {published},
tppubtype = {inproceedings}
}
Developing and simulating 3D prototypes is crucial in product conceptual design for ideation and presentation. Traditional methods often keep physical and virtual prototypes separate, leading to a disjointed prototype workflow. In addition, acquiring high-fidelity prototypes is time-consuming and resource-intensive, distracting designers from creative exploration. Recent advancements in generative artificial intelligence (GAI) and extended reality (XR) provided new solutions for rapid prototype transition and mixed simulation. We conducted a formative study to understand current challenges in the traditional prototype process and explore how to effectively utilize GAI and XR ability in prototype. Then we introduced FusionProtor, a mixed-prototype tool for component-level 3D prototype transition and simulation. We proposed a step-by-step generation pipeline in FusionProtor, effectively transiting 3D prototypes from physical to virtual and low- to high-fidelity for rapid ideation and iteration. We also innovated a component-level 3D creation method and applied it in XR environment for the mixed-prototype presentation and interaction. We conducted technical and user experiments to verify FusionProtor's usability in supporting diverse designs. Our results verified that it achieved a seamless workflow between physical and virtual domains, enhancing efficiency and promoting ideation. We also explored the effect of mixed interaction on design and critically discussed its best practices for HCI community. © 2025 Copyright held by the owner/author(s). Publication rights licensed to ACM.
Stroinski, M.; Kwarciak, K.; Kowalewski, M.; Hemmerling, D.; Frier, W.; Georgiou, O.
Text-to-Haptics: Enhancing Multisensory Storytelling through Emotionally Congruent Midair Haptics Journal Article
In: Advanced Intelligent Systems, vol. 7, no. 4, 2025, ISSN: 26404567 (ISSN).
Abstract | Links | BibTeX | Tags: Audiovisual, Augmented Reality, Extended reality, Haptic interfaces, Haptics, Haptics interfaces, HMI, hybrid AI, Hybrid artificial intelligences, Metaverses, Mixed reality, Multisensory, Natural Language Processing, perception, Sentiment Analysis, Sound speech, Special issue and section, Speech enhancement, Virtual environments, Visual elements
@article{stroinski_text–haptics_2025,
title = {Text-to-Haptics: Enhancing Multisensory Storytelling through Emotionally Congruent Midair Haptics},
author = {M. Stroinski and K. Kwarciak and M. Kowalewski and D. Hemmerling and W. Frier and O. Georgiou},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105002269591&doi=10.1002%2faisy.202400758&partnerID=40&md5=a4c8ce7a01c9bc90d9805a81d34df982},
doi = {10.1002/aisy.202400758},
issn = {26404567 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Advanced Intelligent Systems},
volume = {7},
number = {4},
abstract = {In multisensory storytelling, the integration of touch, sound, speech, and visual elements plays a crucial role in enhancing the narrative immersion and audience engagement. In light of this, this article presents a scalable and intelligent hybrid artificial intelligence (AI) method that uses emotional text analysis for deciding when and what midair haptics to display alongside audiovisual content generated by latent stable diffusion methods. Then, a user study involving 40 participants is described, the results of which suggest that the proposed approach enhances the audience level of engagement as they experience a short AI-generated multisensory (audio–visual–haptic) story. © 2024 The Author(s). Advanced Intelligent Systems published by Wiley-VCH GmbH.},
keywords = {Audiovisual, Augmented Reality, Extended reality, Haptic interfaces, Haptics, Haptics interfaces, HMI, hybrid AI, Hybrid artificial intelligences, Metaverses, Mixed reality, Multisensory, Natural Language Processing, perception, Sentiment Analysis, Sound speech, Special issue and section, Speech enhancement, Virtual environments, Visual elements},
pubstate = {published},
tppubtype = {article}
}
In multisensory storytelling, the integration of touch, sound, speech, and visual elements plays a crucial role in enhancing the narrative immersion and audience engagement. In light of this, this article presents a scalable and intelligent hybrid artificial intelligence (AI) method that uses emotional text analysis for deciding when and what midair haptics to display alongside audiovisual content generated by latent stable diffusion methods. Then, a user study involving 40 participants is described, the results of which suggest that the proposed approach enhances the audience level of engagement as they experience a short AI-generated multisensory (audio–visual–haptic) story. © 2024 The Author(s). Advanced Intelligent Systems published by Wiley-VCH GmbH.
Dongye, X.; Weng, D.; Jiang, H.; Tian, Z.; Bao, Y.; Chen, P.
Personalized decision-making for agents in face-to-face interaction in virtual reality Journal Article
In: Multimedia Systems, vol. 31, no. 1, 2025, ISSN: 09424962 (ISSN).
Abstract | Links | BibTeX | Tags: Decision making, Decision-making process, Decisions makings, Design frameworks, Face-to-face interaction, Feed-back based, Fine tuning, Human-agent interaction, Human–agent interaction, Integrated circuit design, Intelligent virtual agents, Language Model, Large language model, Multi agent systems, Multimodal Interaction, Virtual environments, Virtual Reality
@article{dongye_personalized_2025,
title = {Personalized decision-making for agents in face-to-face interaction in virtual reality},
author = {X. Dongye and D. Weng and H. Jiang and Z. Tian and Y. Bao and P. Chen},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85212947825&doi=10.1007%2fs00530-024-01591-7&partnerID=40&md5=d969cd926fdfd241399f2f96dbf42907},
doi = {10.1007/s00530-024-01591-7},
issn = {09424962 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Multimedia Systems},
volume = {31},
number = {1},
abstract = {Intelligent agents for face-to-face interaction in virtual reality are expected to make decisions and provide appropriate feedback based on the user’s multimodal interaction inputs. Designing the agent’s decision-making process poses a significant challenge owing to the limited availability of multimodal interaction decision-making datasets and the complexities associated with providing personalized interaction feedback to diverse users. To overcome these challenges, we propose a novel design framework that involves generating and labeling symbolic interaction data, pre-training a small-scale real-time decision-making network, collecting personalized interaction data within interactions, and fine-tuning the network using personalized data. We develop a prototype system to demonstrate our design framework, which utilizes interaction distances, head orientations, and hand postures as inputs in virtual reality. The agent is capable of delivering personalized feedback from different users. We evaluate the proposed design framework by demonstrating the utilization of large language models for data labeling, emphasizing reliability and robustness. Furthermore, we evaluate the incorporation of personalized data fine-tuning for decision-making networks within our design framework, highlighting its importance in improving the user interaction experience. The design principles of this framework can be further explored and applied to various domains involving virtual agents. © The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2024.},
keywords = {Decision making, Decision-making process, Decisions makings, Design frameworks, Face-to-face interaction, Feed-back based, Fine tuning, Human-agent interaction, Human–agent interaction, Integrated circuit design, Intelligent virtual agents, Language Model, Large language model, Multi agent systems, Multimodal Interaction, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
Intelligent agents for face-to-face interaction in virtual reality are expected to make decisions and provide appropriate feedback based on the user’s multimodal interaction inputs. Designing the agent’s decision-making process poses a significant challenge owing to the limited availability of multimodal interaction decision-making datasets and the complexities associated with providing personalized interaction feedback to diverse users. To overcome these challenges, we propose a novel design framework that involves generating and labeling symbolic interaction data, pre-training a small-scale real-time decision-making network, collecting personalized interaction data within interactions, and fine-tuning the network using personalized data. We develop a prototype system to demonstrate our design framework, which utilizes interaction distances, head orientations, and hand postures as inputs in virtual reality. The agent is capable of delivering personalized feedback from different users. We evaluate the proposed design framework by demonstrating the utilization of large language models for data labeling, emphasizing reliability and robustness. Furthermore, we evaluate the incorporation of personalized data fine-tuning for decision-making networks within our design framework, highlighting its importance in improving the user interaction experience. The design principles of this framework can be further explored and applied to various domains involving virtual agents. © The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2024.
Fernandez, J. A. V.; Lee, J. J.; Vacca, S. A. S.; Magana, A.; Peša, R.; Benes, B.; Popescu, V.
Hands-Free VR Proceedings Article
In: T., Bashford-Rogers; D., Meneveaux; M., Ammi; M., Ziat; S., Jänicke; H., Purchase; P., Radeva; A., Furnari; K., Bouatouch; A.A., Sousa (Ed.): Proc. Int. Jt. Conf. Comput. Vis. Imaging Comput. Graph. Theory Appl., pp. 533–542, Science and Technology Publications, Lda, 2025, ISBN: 21845921 (ISSN).
Abstract | Links | BibTeX | Tags: Deep learning, Large language model, Retrieval-Augmented Generation, Speech-to-Text, Virtual Reality
@inproceedings{fernandez_hands-free_2025,
title = {Hands-Free VR},
author = {J. A. V. Fernandez and J. J. Lee and S. A. S. Vacca and A. Magana and R. Peša and B. Benes and V. Popescu},
editor = {Bashford-Rogers T. and Meneveaux D. and Ammi M. and Ziat M. and Jänicke S. and Purchase H. and Radeva P. and Furnari A. and Bouatouch K. and Sousa A.A.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105001963646&doi=10.5220%2f0013115100003912&partnerID=40&md5=a3f2f4e16bcd5e0579b38e062c987eab},
doi = {10.5220/0013115100003912},
isbn = {21845921 (ISSN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. Int. Jt. Conf. Comput. Vis. Imaging Comput. Graph. Theory Appl.},
volume = {1},
pages = {533–542},
publisher = {Science and Technology Publications, Lda},
abstract = {We introduce Hands-Free VR, a voice-based natural-language interface for VR that allows interaction without additional hardware just using voice. The user voice command is converted into text using a fine-tuned speechto-text deep-learning model. Then, the text is mapped to an executable VR command using an LLM, which is robust to natural language diversity. Hands-Free VR was evaluated in a within-subjects study (N = 22) where participants arranged objects using either a conventional VR interface or Hands-Free VR. The results confirm that Hands-Free VR is: (1) significantly more efficient than conventional VR interfaces in task completion time and user motion metrics; (2) highly rated for ease of use, intuitiveness, ergonomics, reliability, and desirability; (3) robust to English accents (20 participants were non-native speakers) and phonetic similarity, accurately transcribing 96.7% of voice commands, and (3) robust to natural language diversity, mapping 97.83% of transcriptions to executable commands. © 2025 by SCITEPRESS–Science and Technology Publications, Lda.},
keywords = {Deep learning, Large language model, Retrieval-Augmented Generation, Speech-to-Text, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
We introduce Hands-Free VR, a voice-based natural-language interface for VR that allows interaction without additional hardware just using voice. The user voice command is converted into text using a fine-tuned speechto-text deep-learning model. Then, the text is mapped to an executable VR command using an LLM, which is robust to natural language diversity. Hands-Free VR was evaluated in a within-subjects study (N = 22) where participants arranged objects using either a conventional VR interface or Hands-Free VR. The results confirm that Hands-Free VR is: (1) significantly more efficient than conventional VR interfaces in task completion time and user motion metrics; (2) highly rated for ease of use, intuitiveness, ergonomics, reliability, and desirability; (3) robust to English accents (20 participants were non-native speakers) and phonetic similarity, accurately transcribing 96.7% of voice commands, and (3) robust to natural language diversity, mapping 97.83% of transcriptions to executable commands. © 2025 by SCITEPRESS–Science and Technology Publications, Lda.
Abdelmagid, A. S.; Jabli, N. M.; Al-Mohaya, A. Y.; Teleb, A. A.
In: Sustainability (Switzerland), vol. 17, no. 12, 2025, ISSN: 20711050 (ISSN).
Abstract | Links | BibTeX | Tags: Artificial intelligence, digitization, e-entrepreneurship, entrepreneur, generative artificial intelligence, green digital economy, green economy, higher education, Learning, Metaverse, Sustainable development
@article{abdelmagid_integrating_2025,
title = {Integrating Interactive Metaverse Environments and Generative Artificial Intelligence to Promote the Green Digital Economy and e-Entrepreneurship in Higher Education},
author = {A. S. Abdelmagid and N. M. Jabli and A. Y. Al-Mohaya and A. A. Teleb},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105008981835&doi=10.3390%2fsu17125594&partnerID=40&md5=0eaea40f26536c05c29c7b3f0d42d37d},
doi = {10.3390/su17125594},
issn = {20711050 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Sustainability (Switzerland)},
volume = {17},
number = {12},
abstract = {The rapid evolution of the Fourth Industrial Revolution has significantly transformed educational practices, necessitating the integration of advanced technologies into higher education to address contemporary sustainability challenges. This study explores the integration of interactive metaverse environments and generative artificial intelligence (GAI) in promoting the green digital economy and developing e-entrepreneurship skills among graduate students. Grounded in a quasi-experimental design, the research was conducted with a sample of 25 postgraduate students enrolled in the “Computers in Education” course at King Khalid University. A 3D immersive learning environment (FrameVR) was combined with GAI platforms (ChatGPT version 4.0, Elai.io version 2.5, Tome version 1.3) to create an innovative educational experience. Data were collected using validated instruments, including the Green Digital Economy Scale, the e-Entrepreneurship Scale, and a digital product evaluation rubric. The findings revealed statistically significant improvements in students’ awareness of green digital concepts, entrepreneurial competencies, and their ability to produce sustainable digital products. The study highlights the potential of immersive virtual learning environments and AI-driven content creation tools in enhancing digital literacy and sustainability-oriented innovation. It also underscores the urgent need to update educational strategies and curricula to prepare future professionals capable of navigating and shaping green digital economies. This research provides a practical and replicable model for universities seeking to embed sustainability through emerging technologies, supporting broader goals such as SDG 4 (Quality Education) and SDG 9 (Industry, Innovation, and Infrastructure). © 2025 by the authors.},
keywords = {Artificial intelligence, digitization, e-entrepreneurship, entrepreneur, generative artificial intelligence, green digital economy, green economy, higher education, Learning, Metaverse, Sustainable development},
pubstate = {published},
tppubtype = {article}
}
The rapid evolution of the Fourth Industrial Revolution has significantly transformed educational practices, necessitating the integration of advanced technologies into higher education to address contemporary sustainability challenges. This study explores the integration of interactive metaverse environments and generative artificial intelligence (GAI) in promoting the green digital economy and developing e-entrepreneurship skills among graduate students. Grounded in a quasi-experimental design, the research was conducted with a sample of 25 postgraduate students enrolled in the “Computers in Education” course at King Khalid University. A 3D immersive learning environment (FrameVR) was combined with GAI platforms (ChatGPT version 4.0, Elai.io version 2.5, Tome version 1.3) to create an innovative educational experience. Data were collected using validated instruments, including the Green Digital Economy Scale, the e-Entrepreneurship Scale, and a digital product evaluation rubric. The findings revealed statistically significant improvements in students’ awareness of green digital concepts, entrepreneurial competencies, and their ability to produce sustainable digital products. The study highlights the potential of immersive virtual learning environments and AI-driven content creation tools in enhancing digital literacy and sustainability-oriented innovation. It also underscores the urgent need to update educational strategies and curricula to prepare future professionals capable of navigating and shaping green digital economies. This research provides a practical and replicable model for universities seeking to embed sustainability through emerging technologies, supporting broader goals such as SDG 4 (Quality Education) and SDG 9 (Industry, Innovation, and Infrastructure). © 2025 by the authors.
Tang, M.; Nikolaenko, M.; Alrefai, A.; Kumar, A.
Metaverse and Digital Twins in the Age of AI and Extended Reality Journal Article
In: Architecture, vol. 5, no. 2, 2025, ISSN: 26738945 (ISSN).
Abstract | Links | BibTeX | Tags: AI, digital twin, Extended reality, Metaverse
@article{tang_metaverse_2025,
title = {Metaverse and Digital Twins in the Age of AI and Extended Reality},
author = {M. Tang and M. Nikolaenko and A. Alrefai and A. Kumar},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105008903949&doi=10.3390%2farchitecture5020036&partnerID=40&md5=3b05b81a0cf25d3c441d4701a7749d66},
doi = {10.3390/architecture5020036},
issn = {26738945 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Architecture},
volume = {5},
number = {2},
abstract = {This paper explores the evolving relationship between Digital Twins (DT) and the Metaverse, two foundational yet often conflated digital paradigms in digital architecture. While DTs function as mirrored models of real-world systems—integrating IoT, BIM, and real-time analytics to support decision-making—Metaverses are typically fictional, immersive, multi-user environments shaped by social, cultural, and speculative narratives. Through several research projects, the team investigate the divergence between DTs and Metaverses through the lens of their purpose, data structure, immersion, and interactivity, while highlighting areas of convergence driven by emerging technologies in Artificial Intelligence (AI) and Extended Reality (XR).This study aims to investigate the convergence of DTs and the Metaverse in digital architecture, examining how emerging technologies—such as AI, XR, and Large Language Models (LLMs)—are blurring their traditional boundaries. By analyzing their divergent purposes, data structures, and interactivity modes, as well as hybrid applications (e.g., data-integrated virtual environments and AI-driven collaboration), this study seeks to define the opportunities and challenges of this integration for architectural design, decision-making, and immersive user experiences. Our research spans multiple projects utilizing XR and AI to develop DT and the Metaverse. The team assess the capabilities of AI in DT environments, such as reality capture and smart building management. Concurrently, the team evaluates metaverse platforms for online collaboration and architectural education, focusing on features facilitating multi-user engagement. The paper presents evaluations of various virtual environment development pipelines, comparing traditional BIM+IoT workflows with novel approaches such as Gaussian Splatting and generative AI for content creation. The team further explores the integration of Large Language Models (LLMs) in both domains, such as virtual agents or LLM-powered Non-Player-Controlled Characters (NPC), enabling autonomous interaction and enhancing user engagement within spatial environments. Finally, the paper argues that DTs and Metaverse’s once-distinct boundaries are becoming increasingly porous. Hybrid digital spaces—such as virtual buildings with data-integrated twins and immersive, social metaverses—demonstrate this convergence. As digital environments mature, architects are uniquely positioned to shape these dual-purpose ecosystems, leveraging AI, XR, and spatial computing to fuse data-driven models with immersive and user-centered experiences. © 2025 by the authors.},
keywords = {AI, digital twin, Extended reality, Metaverse},
pubstate = {published},
tppubtype = {article}
}
This paper explores the evolving relationship between Digital Twins (DT) and the Metaverse, two foundational yet often conflated digital paradigms in digital architecture. While DTs function as mirrored models of real-world systems—integrating IoT, BIM, and real-time analytics to support decision-making—Metaverses are typically fictional, immersive, multi-user environments shaped by social, cultural, and speculative narratives. Through several research projects, the team investigate the divergence between DTs and Metaverses through the lens of their purpose, data structure, immersion, and interactivity, while highlighting areas of convergence driven by emerging technologies in Artificial Intelligence (AI) and Extended Reality (XR).This study aims to investigate the convergence of DTs and the Metaverse in digital architecture, examining how emerging technologies—such as AI, XR, and Large Language Models (LLMs)—are blurring their traditional boundaries. By analyzing their divergent purposes, data structures, and interactivity modes, as well as hybrid applications (e.g., data-integrated virtual environments and AI-driven collaboration), this study seeks to define the opportunities and challenges of this integration for architectural design, decision-making, and immersive user experiences. Our research spans multiple projects utilizing XR and AI to develop DT and the Metaverse. The team assess the capabilities of AI in DT environments, such as reality capture and smart building management. Concurrently, the team evaluates metaverse platforms for online collaboration and architectural education, focusing on features facilitating multi-user engagement. The paper presents evaluations of various virtual environment development pipelines, comparing traditional BIM+IoT workflows with novel approaches such as Gaussian Splatting and generative AI for content creation. The team further explores the integration of Large Language Models (LLMs) in both domains, such as virtual agents or LLM-powered Non-Player-Controlled Characters (NPC), enabling autonomous interaction and enhancing user engagement within spatial environments. Finally, the paper argues that DTs and Metaverse’s once-distinct boundaries are becoming increasingly porous. Hybrid digital spaces—such as virtual buildings with data-integrated twins and immersive, social metaverses—demonstrate this convergence. As digital environments mature, architects are uniquely positioned to shape these dual-purpose ecosystems, leveraging AI, XR, and spatial computing to fuse data-driven models with immersive and user-centered experiences. © 2025 by the authors.
Ademola, A.; Sinclair, D.; Koniaris, B.; Hannah, S.; Mitchell, K.
NeFT-Net: N-window extended frequency transformer for rhythmic motion prediction Journal Article
In: Computers and Graphics, vol. 129, 2025, ISSN: 00978493 (ISSN).
Abstract | Links | BibTeX | Tags: Cosine transforms, Discrete cosine transforms, Human motions, Immersive, machine learning, Machine-learning, Motion analysis, Motion prediction, Motion processing, Motion sequences, Motion tracking, Real-world, Rendering, Rendering (computer graphics), Rhythmic motion, Three dimensional computer graphics, Virtual environments, Virtual Reality
@article{ademola_neft-net_2025,
title = {NeFT-Net: N-window extended frequency transformer for rhythmic motion prediction},
author = {A. Ademola and D. Sinclair and B. Koniaris and S. Hannah and K. Mitchell},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105006724723&doi=10.1016%2fj.cag.2025.104244&partnerID=40&md5=08fd0792837332404ec9acdd16f608bf},
doi = {10.1016/j.cag.2025.104244},
issn = {00978493 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Computers and Graphics},
volume = {129},
abstract = {Advancements in prediction of human motion sequences are critical for enabling online virtual reality (VR) users to dance and move in ways that accurately mirror real-world actions, delivering a more immersive and connected experience. However, latency in networked motion tracking remains a significant challenge, disrupting engagement and necessitating predictive solutions to achieve real-time synchronization of remote motions. To address this issue, we propose a novel approach leveraging a synthetically generated dataset based on supervised foot anchor placement timings for rhythmic motions, ensuring periodicity and reducing prediction errors. Our model integrates a discrete cosine transform (DCT) to encode motion, refine high-frequency components, and smooth motion sequences, mitigating jittery artifacts. Additionally, we introduce a feed-forward attention mechanism designed to learn from N-window pairs of 3D key-point pose histories for precise future motion prediction. Quantitative and qualitative evaluations on the Human3.6M dataset highlight significant improvements in mean per joint position error (MPJPE) metrics, demonstrating the superiority of our technique over state-of-the-art approaches. We further introduce novel result pose visualizations through the use of generative AI methods. © 2025 The Authors},
keywords = {Cosine transforms, Discrete cosine transforms, Human motions, Immersive, machine learning, Machine-learning, Motion analysis, Motion prediction, Motion processing, Motion sequences, Motion tracking, Real-world, Rendering, Rendering (computer graphics), Rhythmic motion, Three dimensional computer graphics, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
Advancements in prediction of human motion sequences are critical for enabling online virtual reality (VR) users to dance and move in ways that accurately mirror real-world actions, delivering a more immersive and connected experience. However, latency in networked motion tracking remains a significant challenge, disrupting engagement and necessitating predictive solutions to achieve real-time synchronization of remote motions. To address this issue, we propose a novel approach leveraging a synthetically generated dataset based on supervised foot anchor placement timings for rhythmic motions, ensuring periodicity and reducing prediction errors. Our model integrates a discrete cosine transform (DCT) to encode motion, refine high-frequency components, and smooth motion sequences, mitigating jittery artifacts. Additionally, we introduce a feed-forward attention mechanism designed to learn from N-window pairs of 3D key-point pose histories for precise future motion prediction. Quantitative and qualitative evaluations on the Human3.6M dataset highlight significant improvements in mean per joint position error (MPJPE) metrics, demonstrating the superiority of our technique over state-of-the-art approaches. We further introduce novel result pose visualizations through the use of generative AI methods. © 2025 The Authors
Fang, A.; Chhabria, H.; Maram, A.; Zhu, H.
Social Simulation for Everyday Self-Care: Design Insights from Leveraging VR, AR, and LLMs for Practicing Stress Relief Proceedings Article
In: Conf Hum Fact Comput Syst Proc, Association for Computing Machinery, 2025, ISBN: 979-840071394-1 (ISBN).
Abstract | Links | BibTeX | Tags: design, Design insights, Language Model, Large language model, large language models, Mental health, Peer support, Professional supports, Self-care, Social simulations, Speed dating, Virtual environments, Virtual Reality, Well being
@inproceedings{fang_social_2025,
title = {Social Simulation for Everyday Self-Care: Design Insights from Leveraging VR, AR, and LLMs for Practicing Stress Relief},
author = {A. Fang and H. Chhabria and A. Maram and H. Zhu},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005770377&doi=10.1145%2f3706598.3713115&partnerID=40&md5=87d43f04dfd3231cb189fa89570824c5},
doi = {10.1145/3706598.3713115},
isbn = {979-840071394-1 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Conf Hum Fact Comput Syst Proc},
publisher = {Association for Computing Machinery},
abstract = {Stress is an inevitable part of day-to-day life yet many find themselves unable to manage it themselves, particularly when professional or peer support are not always readily available. As self-care becomes increasingly vital for mental well-being, this paper explores the potential of social simulation as a safe, virtual environment for practicing in-the-moment stress relief for everyday social situations. Leveraging the immersive capabilities of VR, AR, and LLMs to create realistic interactions and environments, we developed eight interactive prototypes for various social stress related scenarios (e.g. public speaking, interpersonal conflict) across design dimensions of modality, interactivity, and mental health guidance in order to conduct prototype-driven semi-structured interviews with 19 participants. Our qualitative findings reveal that people currently lack effective means to support themselves through everyday stress and perceive social simulation - even at low immersion and interaction levels - to fill a gap for practical, controlled training of mental health practices. We outline key design needs for developing social simulation for self-care needs, and identify important considerations including risks of trauma from hyper-realism, distrust of LLM-recommended timing for mental health recommendations, and the value of accessibility for self-care interventions. © 2025 Copyright held by the owner/author(s).},
keywords = {design, Design insights, Language Model, Large language model, large language models, Mental health, Peer support, Professional supports, Self-care, Social simulations, Speed dating, Virtual environments, Virtual Reality, Well being},
pubstate = {published},
tppubtype = {inproceedings}
}
Stress is an inevitable part of day-to-day life yet many find themselves unable to manage it themselves, particularly when professional or peer support are not always readily available. As self-care becomes increasingly vital for mental well-being, this paper explores the potential of social simulation as a safe, virtual environment for practicing in-the-moment stress relief for everyday social situations. Leveraging the immersive capabilities of VR, AR, and LLMs to create realistic interactions and environments, we developed eight interactive prototypes for various social stress related scenarios (e.g. public speaking, interpersonal conflict) across design dimensions of modality, interactivity, and mental health guidance in order to conduct prototype-driven semi-structured interviews with 19 participants. Our qualitative findings reveal that people currently lack effective means to support themselves through everyday stress and perceive social simulation - even at low immersion and interaction levels - to fill a gap for practical, controlled training of mental health practices. We outline key design needs for developing social simulation for self-care needs, and identify important considerations including risks of trauma from hyper-realism, distrust of LLM-recommended timing for mental health recommendations, and the value of accessibility for self-care interventions. © 2025 Copyright held by the owner/author(s).
Kurai, R.; Hiraki, T.; Hiroi, Y.; Hirao, Y.; Perusquia-Hernandez, M.; Uchiyama, H.; Kiyokawa, K.
MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse Platform Journal Article
In: IEEE Access, vol. 13, pp. 19132–19143, 2025, ISSN: 21693536 (ISSN).
Abstract | Links | BibTeX | Tags: Behavior design, Code programming, Computer simulation languages, Dynamic behaviors, Language Model, Large-language model, Low-code programming, Metaverse platform, Metaverses, Virtual addresses, Virtual environments, Virtual objects, Virtual Reality, Virtual-reality environment
@article{kurai_magicitem_2025,
title = {MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse Platform},
author = {R. Kurai and T. Hiraki and Y. Hiroi and Y. Hirao and M. Perusquia-Hernandez and H. Uchiyama and K. Kiyokawa},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85216011970&doi=10.1109%2fACCESS.2025.3530439&partnerID=40&md5=7a33b9618af8b4ab79b43fb3bd4317cf},
doi = {10.1109/ACCESS.2025.3530439},
issn = {21693536 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Access},
volume = {13},
pages = {19132–19143},
abstract = {To create rich experiences in virtual reality (VR) environments, it is essential to define the behavior of virtual objects through programming. However, programming in 3D spaces requires a wide range of background knowledge and programming skills. Although Large Language Models (LLMs) have provided programming support, they are still primarily aimed at programmers. In metaverse platforms, where many users inhabit VR spaces, most users are unfamiliar with programming, making it difficult for them to modify the behavior of objects in the VR environment easily. Existing LLM-based script generation methods for VR spaces require multiple lengthy iterations to implement the desired behaviors and are difficult to integrate into the operation of metaverse platforms. To address this issue, we propose a tool that generates behaviors for objects in VR spaces from natural language within Cluster, a metaverse platform with a large user base. By integrating LLMs with the Cluster Script provided by this platform, we enable users with limited programming experience to define object behaviors within the platform freely. We have also integrated our tool into a commercial metaverse platform and are conducting online experiments with 63 general users of the platform. The experiments show that even users with no programming background can successfully generate behaviors for objects in VR spaces, resulting in a highly satisfying system. Our research contributes to democratizing VR content creation by enabling non-programmers to design dynamic behaviors for virtual objects in metaverse platforms. © 2013 IEEE.},
keywords = {Behavior design, Code programming, Computer simulation languages, Dynamic behaviors, Language Model, Large-language model, Low-code programming, Metaverse platform, Metaverses, Virtual addresses, Virtual environments, Virtual objects, Virtual Reality, Virtual-reality environment},
pubstate = {published},
tppubtype = {article}
}
To create rich experiences in virtual reality (VR) environments, it is essential to define the behavior of virtual objects through programming. However, programming in 3D spaces requires a wide range of background knowledge and programming skills. Although Large Language Models (LLMs) have provided programming support, they are still primarily aimed at programmers. In metaverse platforms, where many users inhabit VR spaces, most users are unfamiliar with programming, making it difficult for them to modify the behavior of objects in the VR environment easily. Existing LLM-based script generation methods for VR spaces require multiple lengthy iterations to implement the desired behaviors and are difficult to integrate into the operation of metaverse platforms. To address this issue, we propose a tool that generates behaviors for objects in VR spaces from natural language within Cluster, a metaverse platform with a large user base. By integrating LLMs with the Cluster Script provided by this platform, we enable users with limited programming experience to define object behaviors within the platform freely. We have also integrated our tool into a commercial metaverse platform and are conducting online experiments with 63 general users of the platform. The experiments show that even users with no programming background can successfully generate behaviors for objects in VR spaces, resulting in a highly satisfying system. Our research contributes to democratizing VR content creation by enabling non-programmers to design dynamic behaviors for virtual objects in metaverse platforms. © 2013 IEEE.
Hu, H.; Wan, Y.; Tang, K. Y.; Li, Q.; Wang, X.
Affective-Computing-Driven Personalized Display of Cultural Information for Commercial Heritage Architecture Journal Article
In: Applied Sciences (Switzerland), vol. 15, no. 7, 2025, ISSN: 20763417 (ISSN).
Abstract | Links | BibTeX | Tags: Affective Computing, Cultural informations, Cultural value, Data fusion, Information display, Information fusion, Information presentation, Language Model, Large language model, Multimodal information fusion, User-generated, User-generated content, Virtual environments
@article{hu_affective-computing-driven_2025,
title = {Affective-Computing-Driven Personalized Display of Cultural Information for Commercial Heritage Architecture},
author = {H. Hu and Y. Wan and K. Y. Tang and Q. Li and X. Wang},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105002467183&doi=10.3390%2fapp15073459&partnerID=40&md5=1dc611258248d58a2bf5f44b6a0e890b},
doi = {10.3390/app15073459},
issn = {20763417 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Applied Sciences (Switzerland)},
volume = {15},
number = {7},
abstract = {The display methods for traditional cultural heritage lack personalization and emotional interaction, making it difficult to stimulate the public’s deep cultural awareness. This is especially true in commercialized historical districts, where cultural value is easily overlooked. Balancing cultural value and commercial value in information display has become one of the challenges that needs to be addressed. To solve the above problems, this article focuses on the identification of deep cultural values and the optimization of the information display in Beijing’s Qianmen Street, proposing a framework for cultural information mining and display based on affective computing and large language models. The pre-trained models QwenLM and RoBERTa were employed to analyze text and image data from user-generated content on social media, identifying users’ emotional tendencies toward various cultural value dimensions and quantifying their multilayered understanding of architectural heritage. This study further constructed a multimodal information presentation model driven by emotional feedback, mapping it into virtual reality environments to enable personalized, multilayered cultural information visualization. The framework’s effectiveness was validated through an eye-tracking experiment that assessed how different presentation styles impacted users’ emotional engagement and cognitive outcomes. The results show that the affective computing and multimodal data fusion approach to cultural heritage presentation accurately captures users’ emotions, enhancing their interest and emotional involvement. Personalized presentations of information significantly improve users’ engagement, historical understanding, and cultural experience, thereby fostering a deeper comprehension of historical contexts and architectural details. © 2025 by the authors.},
keywords = {Affective Computing, Cultural informations, Cultural value, Data fusion, Information display, Information fusion, Information presentation, Language Model, Large language model, Multimodal information fusion, User-generated, User-generated content, Virtual environments},
pubstate = {published},
tppubtype = {article}
}
The display methods for traditional cultural heritage lack personalization and emotional interaction, making it difficult to stimulate the public’s deep cultural awareness. This is especially true in commercialized historical districts, where cultural value is easily overlooked. Balancing cultural value and commercial value in information display has become one of the challenges that needs to be addressed. To solve the above problems, this article focuses on the identification of deep cultural values and the optimization of the information display in Beijing’s Qianmen Street, proposing a framework for cultural information mining and display based on affective computing and large language models. The pre-trained models QwenLM and RoBERTa were employed to analyze text and image data from user-generated content on social media, identifying users’ emotional tendencies toward various cultural value dimensions and quantifying their multilayered understanding of architectural heritage. This study further constructed a multimodal information presentation model driven by emotional feedback, mapping it into virtual reality environments to enable personalized, multilayered cultural information visualization. The framework’s effectiveness was validated through an eye-tracking experiment that assessed how different presentation styles impacted users’ emotional engagement and cognitive outcomes. The results show that the affective computing and multimodal data fusion approach to cultural heritage presentation accurately captures users’ emotions, enhancing their interest and emotional involvement. Personalized presentations of information significantly improve users’ engagement, historical understanding, and cultural experience, thereby fostering a deeper comprehension of historical contexts and architectural details. © 2025 by the authors.
Xing, Y.; Ban, J.; Hubbard, T. D.; Villano, M.; Gómez-Zará, D.
Immersed in my Ideas: Using Virtual Reality and LLMs to Visualize Users’ Ideas and Thoughts Proceedings Article
In: Int Conf Intell User Interfaces Proc IUI, pp. 60–65, Association for Computing Machinery, 2025, ISBN: 979-840071409-2 (ISBN).
Abstract | Links | BibTeX | Tags: 3-D environments, 3D modeling, Computer simulation languages, Creativity, Idea Generation, Immersive, Interactive virtual reality, Language Model, Large language model, Multimodal Interaction, Reflection, Text Visualization, Think aloud, Virtual environments, Virtual Reality, Visualization
@inproceedings{xing_immersed_2025,
title = {Immersed in my Ideas: Using Virtual Reality and LLMs to Visualize Users’ Ideas and Thoughts},
author = {Y. Xing and J. Ban and T. D. Hubbard and M. Villano and D. Gómez-Zará},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105001675169&doi=10.1145%2f3708557.3716330&partnerID=40&md5=20fb0623d2a1fff92282116b01fac4f3},
doi = {10.1145/3708557.3716330},
isbn = {979-840071409-2 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Int Conf Intell User Interfaces Proc IUI},
pages = {60–65},
publisher = {Association for Computing Machinery},
abstract = {We introduce the Voice Interactive Virtual Reality Annotation (VIVRA), an application that employs Large Language Models to facilitate brainstorming and idea exploration in an immersive 3D environment. As users think aloud to brainstorm and ideate, the application automatically detects, summarizes, suggests, and connects their ideas in real time. The experience brings participants into a room where their ideas emerge as interactive objects that embody the topics detected from their ideas. We evaluated the effectiveness of VIVRA in an exploratory study with 29 participants, followed by a user study with 10 participants comparing the application with other visualizations. Our results show that VIVRA helped participants reflect and think more about their ideas, serving as a valuable tool for personal exploration. We discuss the potential benefits and applications, highlighting the benefits of combining immersive 3D spaces and LLMs to explore, learn, and reflect on ideas. © 2025 Copyright held by the owner/author(s).},
keywords = {3-D environments, 3D modeling, Computer simulation languages, Creativity, Idea Generation, Immersive, Interactive virtual reality, Language Model, Large language model, Multimodal Interaction, Reflection, Text Visualization, Think aloud, Virtual environments, Virtual Reality, Visualization},
pubstate = {published},
tppubtype = {inproceedings}
}
We introduce the Voice Interactive Virtual Reality Annotation (VIVRA), an application that employs Large Language Models to facilitate brainstorming and idea exploration in an immersive 3D environment. As users think aloud to brainstorm and ideate, the application automatically detects, summarizes, suggests, and connects their ideas in real time. The experience brings participants into a room where their ideas emerge as interactive objects that embody the topics detected from their ideas. We evaluated the effectiveness of VIVRA in an exploratory study with 29 participants, followed by a user study with 10 participants comparing the application with other visualizations. Our results show that VIVRA helped participants reflect and think more about their ideas, serving as a valuable tool for personal exploration. We discuss the potential benefits and applications, highlighting the benefits of combining immersive 3D spaces and LLMs to explore, learn, and reflect on ideas. © 2025 Copyright held by the owner/author(s).
Intawong, K.; Worragin, P.; Khanchai, S.; Puritat, K.
Transformative Metaverse Pedagogy for Intangible Heritage: A Gamified Platform for Learning Lanna Dance in Immersive Cultural Education Journal Article
In: Education Sciences, vol. 15, no. 6, 2025, ISSN: 22277102 (ISSN).
Abstract | Links | BibTeX | Tags: Adaptive Learning, AI-assisted education, cultural preservation, Gamification, intangible cultural heritage, Lanna dance, Metaverse
@article{intawong_transformative_2025,
title = {Transformative Metaverse Pedagogy for Intangible Heritage: A Gamified Platform for Learning Lanna Dance in Immersive Cultural Education},
author = {K. Intawong and P. Worragin and S. Khanchai and K. Puritat},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105009306705&doi=10.3390%2feducsci15060736&partnerID=40&md5=9f17583b77ec54c090e50575b539f0c9},
doi = {10.3390/educsci15060736},
issn = {22277102 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Education Sciences},
volume = {15},
number = {6},
abstract = {This study explores the design of Metaverse technologies for preserving and teaching Lanna Dance, a traditional cultural heritage of Northern Thailand. It addresses the challenges of sustaining intangible cultural heritage by developing an immersive learning system that integrates motion capture, generative AI, and gamified virtual environments. Grounded in Situated Learning Theory and adaptive learning, the platform features four interactive zones, the Motion Showcase, Knowledge Exhibition, Video and AI Interaction, and Interactive Game Zone, offering learners multifaceted, context-rich experiences. Using a quasi-experimental design with 36 participants, the study evaluates learning outcomes, motivation, and user satisfaction. Results show significant improvements in knowledge acquisition and intrinsic motivation, along with high usability scores, indicating the effectiveness of immersive digital environments in enhancing cultural appreciation and skill development. The findings offer practical insights into Metaverse design for immersive cultural education, supporting educators, cultural institutions, and policymakers in developing scalable and engaging solutions for preserving intangible heritage through emerging technologies. © 2025 by the authors.},
keywords = {Adaptive Learning, AI-assisted education, cultural preservation, Gamification, intangible cultural heritage, Lanna dance, Metaverse},
pubstate = {published},
tppubtype = {article}
}
This study explores the design of Metaverse technologies for preserving and teaching Lanna Dance, a traditional cultural heritage of Northern Thailand. It addresses the challenges of sustaining intangible cultural heritage by developing an immersive learning system that integrates motion capture, generative AI, and gamified virtual environments. Grounded in Situated Learning Theory and adaptive learning, the platform features four interactive zones, the Motion Showcase, Knowledge Exhibition, Video and AI Interaction, and Interactive Game Zone, offering learners multifaceted, context-rich experiences. Using a quasi-experimental design with 36 participants, the study evaluates learning outcomes, motivation, and user satisfaction. Results show significant improvements in knowledge acquisition and intrinsic motivation, along with high usability scores, indicating the effectiveness of immersive digital environments in enhancing cultural appreciation and skill development. The findings offer practical insights into Metaverse design for immersive cultural education, supporting educators, cultural institutions, and policymakers in developing scalable and engaging solutions for preserving intangible heritage through emerging technologies. © 2025 by the authors.
Gao, H.; Xie, Y.; Kasneci, E.
PerVRML: ChatGPT-Driven Personalized VR Environments for Machine Learning Education Journal Article
In: International Journal of Human-Computer Interaction, 2025, ISSN: 10447318 (ISSN).
Abstract | Links | BibTeX | Tags: Backpropagation, ChatGPT, Curricula, Educational robots, Immersive learning, Interactive learning, Language Model, Large language model, large language models, Learning mode, Machine learning education, Machine-learning, Personalized learning, Support vector machines, Teaching, Virtual Reality, Virtual-reality environment, Virtualization
@article{gao_pervrml_2025,
title = {PerVRML: ChatGPT-Driven Personalized VR Environments for Machine Learning Education},
author = {H. Gao and Y. Xie and E. Kasneci},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005776517&doi=10.1080%2f10447318.2025.2504188&partnerID=40&md5=c2c59be3d20d02c6df7750c2330c8f6d},
doi = {10.1080/10447318.2025.2504188},
issn = {10447318 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {International Journal of Human-Computer Interaction},
abstract = {The advent of large language models (LLMs) such as ChatGPT has demonstrated significant potential for advancing educational technologies. Recently, growing interest has emerged in integrating ChatGPT with virtual reality (VR) to provide interactive and dynamic learning environments. This study explores the effectiveness of ChatGTP-driven VR in facilitating machine learning education through PerVRML. PerVRML incorporates a ChatGPT-powered avatar that provides real-time assistance and uses LLMs to personalize learning paths based on various sensor data from VR. A between-subjects design was employed to compare two learning modes: personalized and non-personalized. Quantitative data were collected from assessments, user experience surveys, and interaction metrics. The results indicate that while both learning modes supported learning effectively, ChatGPT-powered personalization significantly improved learning outcomes and had distinct impacts on user feedback. These findings underscore the potential of ChatGPT-enhanced VR to deliver adaptive and personalized educational experiences. © 2025 Taylor & Francis Group, LLC.},
keywords = {Backpropagation, ChatGPT, Curricula, Educational robots, Immersive learning, Interactive learning, Language Model, Large language model, large language models, Learning mode, Machine learning education, Machine-learning, Personalized learning, Support vector machines, Teaching, Virtual Reality, Virtual-reality environment, Virtualization},
pubstate = {published},
tppubtype = {article}
}
The advent of large language models (LLMs) such as ChatGPT has demonstrated significant potential for advancing educational technologies. Recently, growing interest has emerged in integrating ChatGPT with virtual reality (VR) to provide interactive and dynamic learning environments. This study explores the effectiveness of ChatGTP-driven VR in facilitating machine learning education through PerVRML. PerVRML incorporates a ChatGPT-powered avatar that provides real-time assistance and uses LLMs to personalize learning paths based on various sensor data from VR. A between-subjects design was employed to compare two learning modes: personalized and non-personalized. Quantitative data were collected from assessments, user experience surveys, and interaction metrics. The results indicate that while both learning modes supported learning effectively, ChatGPT-powered personalization significantly improved learning outcomes and had distinct impacts on user feedback. These findings underscore the potential of ChatGPT-enhanced VR to deliver adaptive and personalized educational experiences. © 2025 Taylor & Francis Group, LLC.
Banafa, A.
Artificial intelligence in action: Real-world applications and innovations Book
River Publishers, 2025, ISBN: 978-877004619-0 (ISBN); 978-877004620-6 (ISBN).
Abstract | Links | BibTeX | Tags: 5G, Affective Computing, AGI, AI, AI alignments, AI Ethics, AI hallucinations, AI hype, AI models, Alexa, ANI, ASI, Augmented Reality, Autoencoders, Autonomic computing, Autonomous Cars, Autoregressive models, Big Data, Big Data Analytics, Bitcoin, Blockchain, C3PO, Casual AI, Causal reasoning, ChatGPT, Cloud computing, Collective AI, Compression engines, Computer vision, Conditional Automation, Convolutional neural networks (CNNs), Cryptocurrency, Cybersecurity, Deceptive AI, Deep learning, Digital transformation, Driver Assistance, Driverless Cars, Drones, Elon Musk, Entanglement, Environment and sustainability, Ethereum, Explainable AI, Facebook, Facial Recognition, Feedforward. Neural Networks, Fog Computing, Full Automation, Future of AI, General AI, Generative Adversarial Networks (GANs), Generative AI, Google, Green AI, High Automation, Hybrid Blockchain, IEEE, Industrial Internet of Things (IIoT), Internet of things (IoT), Jarvis, Java, JavaScript, Long Short-Term Memory Networks, LTE, machine learning, Microsoft, MultiModal AI, Narrow AI, Natural disasters, Natural Language Generation (NLG), Natural Language Processing (NLP), NetFlix, Network Security, Neural Networks, Nuclear, Nuclear AI, NYTimes, Objective-driven AI, Open Source, Partial Automation, PayPal, Perfect AI, Private Blockchain, Private Cloud Computing, Programming languages, Python, Quantum Communications, Quantum Computing, Quantum Cryptography, Quantum internet, Quantum Machine Learning (QML), R2D2, Reactive machines. limited memory, Recurrent Neural Networks, Responsible AI, Robots, Sci-Fi movies, Self-Aware, Semiconductorâ??s, Sensate AI, Siri, Small Data, Smart Contracts. Hybrid Cloud Computing, Smart Devices, Sovereign AI, Super AI, Superposition, TensorFlow, Theory of Mind, Thick Data, Twitter, Variational Autoencoders (VAEs), Virtual Reality, Voice user interface (VUI), Wearable computing devices (WCD), Wearable Technology, Wi-Fi, XAI, Zero-Trust Model
@book{banafa_artificial_2025,
title = {Artificial intelligence in action: Real-world applications and innovations},
author = {A. Banafa},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105000403587&partnerID=40&md5=4b0d94be48194a942b22bef63f36d3bf},
isbn = {978-877004619-0 (ISBN); 978-877004620-6 (ISBN)},
year = {2025},
date = {2025-01-01},
publisher = {River Publishers},
series = {Artificial Intelligence in Action: Real-World Applications and Innovations},
abstract = {This comprehensive book dives deep into the current landscape of AI, exploring its fundamental principles, development challenges, potential risks, and the cutting-edge breakthroughs that are propelling it forward. Artificial intelligence (AI) is rapidly transforming industries and societies worldwide through groundbreaking innovations and real-world applications. Starting with the core concepts, the book examines the various types of AI systems, generative AI models, and the complexities of machine learning. It delves into the programming languages driving AI development, data pipelines, model creation and deployment processes, while shedding light on issues like AI hallucinations and the intricate path of machine unlearning. The book then showcases the remarkable real-world applications of AI across diverse domains. From preventing job displacement and promoting environmental sustainability, to enhancing disaster response, drone technology, and even nuclear energy innovation, it highlights how AI is tackling complex challenges and driving positive change. The book also explores the double-edged nature of AI, recognizing its tremendous potential while cautioning about the risks of misuse, unintended consequences, and the urgent need for responsible development practices. It examines the intersection of AI and fields like operating system design, warfare, and semiconductor technology, underscoring the wide-ranging implications of this transformative force. As the quest for artificial general intelligence (AGI) and superintelligent AI systems intensifies, the book delves into cutting-edge research, emerging trends, and the pursuit of multimodal, explainable, and causally aware AI systems. It explores the symbiotic relationship between AI and human creativity, the rise of user-friendly "casual AI," and the potential of AI to tackle open-ended tasks. This is an essential guide for understanding the profound impact of AI on our world today and its potential to shape our future. From the frontiers of innovation to the challenges of responsible development, this book offers a comprehensive and insightful exploration of the remarkable real-world applications and innovations driving the AI revolution. © 2025 River Publishers. All rights reserved.},
keywords = {5G, Affective Computing, AGI, AI, AI alignments, AI Ethics, AI hallucinations, AI hype, AI models, Alexa, ANI, ASI, Augmented Reality, Autoencoders, Autonomic computing, Autonomous Cars, Autoregressive models, Big Data, Big Data Analytics, Bitcoin, Blockchain, C3PO, Casual AI, Causal reasoning, ChatGPT, Cloud computing, Collective AI, Compression engines, Computer vision, Conditional Automation, Convolutional neural networks (CNNs), Cryptocurrency, Cybersecurity, Deceptive AI, Deep learning, Digital transformation, Driver Assistance, Driverless Cars, Drones, Elon Musk, Entanglement, Environment and sustainability, Ethereum, Explainable AI, Facebook, Facial Recognition, Feedforward. Neural Networks, Fog Computing, Full Automation, Future of AI, General AI, Generative Adversarial Networks (GANs), Generative AI, Google, Green AI, High Automation, Hybrid Blockchain, IEEE, Industrial Internet of Things (IIoT), Internet of things (IoT), Jarvis, Java, JavaScript, Long Short-Term Memory Networks, LTE, machine learning, Microsoft, MultiModal AI, Narrow AI, Natural disasters, Natural Language Generation (NLG), Natural Language Processing (NLP), NetFlix, Network Security, Neural Networks, Nuclear, Nuclear AI, NYTimes, Objective-driven AI, Open Source, Partial Automation, PayPal, Perfect AI, Private Blockchain, Private Cloud Computing, Programming languages, Python, Quantum Communications, Quantum Computing, Quantum Cryptography, Quantum internet, Quantum Machine Learning (QML), R2D2, Reactive machines. limited memory, Recurrent Neural Networks, Responsible AI, Robots, Sci-Fi movies, Self-Aware, Semiconductorâ??s, Sensate AI, Siri, Small Data, Smart Contracts. Hybrid Cloud Computing, Smart Devices, Sovereign AI, Super AI, Superposition, TensorFlow, Theory of Mind, Thick Data, Twitter, Variational Autoencoders (VAEs), Virtual Reality, Voice user interface (VUI), Wearable computing devices (WCD), Wearable Technology, Wi-Fi, XAI, Zero-Trust Model},
pubstate = {published},
tppubtype = {book}
}
This comprehensive book dives deep into the current landscape of AI, exploring its fundamental principles, development challenges, potential risks, and the cutting-edge breakthroughs that are propelling it forward. Artificial intelligence (AI) is rapidly transforming industries and societies worldwide through groundbreaking innovations and real-world applications. Starting with the core concepts, the book examines the various types of AI systems, generative AI models, and the complexities of machine learning. It delves into the programming languages driving AI development, data pipelines, model creation and deployment processes, while shedding light on issues like AI hallucinations and the intricate path of machine unlearning. The book then showcases the remarkable real-world applications of AI across diverse domains. From preventing job displacement and promoting environmental sustainability, to enhancing disaster response, drone technology, and even nuclear energy innovation, it highlights how AI is tackling complex challenges and driving positive change. The book also explores the double-edged nature of AI, recognizing its tremendous potential while cautioning about the risks of misuse, unintended consequences, and the urgent need for responsible development practices. It examines the intersection of AI and fields like operating system design, warfare, and semiconductor technology, underscoring the wide-ranging implications of this transformative force. As the quest for artificial general intelligence (AGI) and superintelligent AI systems intensifies, the book delves into cutting-edge research, emerging trends, and the pursuit of multimodal, explainable, and causally aware AI systems. It explores the symbiotic relationship between AI and human creativity, the rise of user-friendly "casual AI," and the potential of AI to tackle open-ended tasks. This is an essential guide for understanding the profound impact of AI on our world today and its potential to shape our future. From the frontiers of innovation to the challenges of responsible development, this book offers a comprehensive and insightful exploration of the remarkable real-world applications and innovations driving the AI revolution. © 2025 River Publishers. All rights reserved.
Ly, C.; Peng, E.; Liu, K.; Qin, A.; Howe, G.; Cheng, A. Y.; Cuadra, A.
Museum in the Classroom: Engaging Students with Augmented Reality Museum Artifacts and Generative AI Proceedings Article
In: Conf Hum Fact Comput Syst Proc, Association for Computing Machinery, 2025, ISBN: 979-840071395-8 (ISBN).
Abstract | Links | BibTeX | Tags: Artifact or System, Child/parent, Children/Parents, Digitisation, Education/Learning, Engaging students, Engineering education, Field trips, Interactive learning, Learning experiences, Rich learning experiences, Students, Teachers', Teaching
@inproceedings{ly_museum_2025,
title = {Museum in the Classroom: Engaging Students with Augmented Reality Museum Artifacts and Generative AI},
author = {C. Ly and E. Peng and K. Liu and A. Qin and G. Howe and A. Y. Cheng and A. Cuadra},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005741934&doi=10.1145%2f3706599.3719787&partnerID=40&md5=08816dd8d41bc34a0dc2d355985e2cc4},
doi = {10.1145/3706599.3719787},
isbn = {979-840071395-8 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Conf Hum Fact Comput Syst Proc},
publisher = {Association for Computing Machinery},
abstract = {Museum field trips provide a rich learning experience for children. However, they are complex and expensive for teachers to organize. Fortunately, digitization of museum artifacts makes it possible to use museum resources within the classroom. Museum in the Classroom (MITC) explores how augmented reality (AR) and generative artificial intelligence (AI) can create an interactive learning experience around museum artifacts. This iPad app allows educators to select historical topics from a curated artifact library, generating AR-based exhibits that students can explore. MITC engages students through interactive AR artifacts, AI-driven chatbots, and AI-generated quiz questions, based on a real exhibition at the Cantor Arts Center at Stanford University. A formative study with middle schoolers (N = 20) demonstrated that the app increased engagement compared to traditional learning methods. MITC also fostered a playful and comfortable environment to interact with museum artifacts. Our findings suggest that combining AR and AI has the potential to enrich classroom learning and offer a scalable alternative to traditional museum visits. © 2025 Copyright held by the owner/author(s).},
keywords = {Artifact or System, Child/parent, Children/Parents, Digitisation, Education/Learning, Engaging students, Engineering education, Field trips, Interactive learning, Learning experiences, Rich learning experiences, Students, Teachers', Teaching},
pubstate = {published},
tppubtype = {inproceedings}
}
Museum field trips provide a rich learning experience for children. However, they are complex and expensive for teachers to organize. Fortunately, digitization of museum artifacts makes it possible to use museum resources within the classroom. Museum in the Classroom (MITC) explores how augmented reality (AR) and generative artificial intelligence (AI) can create an interactive learning experience around museum artifacts. This iPad app allows educators to select historical topics from a curated artifact library, generating AR-based exhibits that students can explore. MITC engages students through interactive AR artifacts, AI-driven chatbots, and AI-generated quiz questions, based on a real exhibition at the Cantor Arts Center at Stanford University. A formative study with middle schoolers (N = 20) demonstrated that the app increased engagement compared to traditional learning methods. MITC also fostered a playful and comfortable environment to interact with museum artifacts. Our findings suggest that combining AR and AI has the potential to enrich classroom learning and offer a scalable alternative to traditional museum visits. © 2025 Copyright held by the owner/author(s).
Leininger, P.; Weber, C. J.; Rothe, S.
Understanding Creative Potential and Use Cases of AI-Generated Environments for Virtual Film Productions: Insights from Industry Professionals Proceedings Article
In: IMX - Proc. ACM Int. Conf. Interact. Media Experiences, pp. 60–78, Association for Computing Machinery, Inc, 2025, ISBN: 979-840071391-0 (ISBN).
Abstract | Links | BibTeX | Tags: 3-D environments, 3D reconstruction, 3D Scene Reconstruction, 3d scenes reconstruction, AI-generated 3d environment, AI-Generated 3D Environments, Computer interaction, Creative Collaboration, Creatives, Digital content creation, Digital Content Creation., Filmmaking workflow, Filmmaking Workflows, Gaussian distribution, Gaussian Splatting, Gaussians, Generative AI, Graphical user interface, Graphical User Interface (GUI), Graphical user interfaces, Human computer interaction, human-computer interaction, Human-Computer Interaction (HCI), Immersive, Immersive Storytelling, Interactive computer graphics, Interactive computer systems, Interactive media, Mesh generation, Previsualization, Real-Time Rendering, Splatting, Three dimensional computer graphics, Virtual production, Virtual Production (VP), Virtual Reality, Work-flows
@inproceedings{leininger_understanding_2025,
title = {Understanding Creative Potential and Use Cases of AI-Generated Environments for Virtual Film Productions: Insights from Industry Professionals},
author = {P. Leininger and C. J. Weber and S. Rothe},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105007976841&doi=10.1145%2f3706370.3727853&partnerID=40&md5=0d4cf7a2398d12d04e4f0ab182474a10},
doi = {10.1145/3706370.3727853},
isbn = {979-840071391-0 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {IMX - Proc. ACM Int. Conf. Interact. Media Experiences},
pages = {60–78},
publisher = {Association for Computing Machinery, Inc},
abstract = {Virtual production (VP) is transforming filmmaking by integrating real-time digital elements with live-action footage, offering new creative possibilities and streamlined workflows. While industry experts recognize AI's potential to revolutionize VP, its practical applications and value across different production phases and user groups remain underexplored. Building on initial research into generative and data-driven approaches, this paper presents the first systematic pilot study evaluating three types of AI-generated 3D environments - Depth Mesh, 360° Panoramic Meshes, and Gaussian Splatting - through the participation of 15 filmmaking professionals from diverse roles. Unlike commonly used 2D AI-generated visuals, our approach introduces navigable 3D environments that offer greater control and flexibility, aligning more closely with established VP workflows. Through expert interviews and literature research, we developed evaluation criteria to assess their usefulness beyond concept development, extending to previsualization, scene exploration, and interdisciplinary collaboration. Our findings indicate that different environments cater to distinct production needs, from early ideation to detailed visualization. Gaussian Splatting proved effective for high-fidelity previsualization, while 360° Panoramic Meshes excelled in rapid concept ideation. Despite their promise, challenges such as limited interactivity and customization highlight areas for improvement. Our prototype, EnVisualAIzer, built in Unreal Engine 5, provides an accessible platform for diverse filmmakers to engage with AI-generated environments, fostering a more inclusive production process. By lowering technical barriers, these environments have the potential to make advanced VP tools more widely available. This study offers valuable insights into the evolving role of AI in VP and sets the stage for future research and development. © 2025 Copyright held by the owner/author(s). Publication rights licensed to ACM.},
keywords = {3-D environments, 3D reconstruction, 3D Scene Reconstruction, 3d scenes reconstruction, AI-generated 3d environment, AI-Generated 3D Environments, Computer interaction, Creative Collaboration, Creatives, Digital content creation, Digital Content Creation., Filmmaking workflow, Filmmaking Workflows, Gaussian distribution, Gaussian Splatting, Gaussians, Generative AI, Graphical user interface, Graphical User Interface (GUI), Graphical user interfaces, Human computer interaction, human-computer interaction, Human-Computer Interaction (HCI), Immersive, Immersive Storytelling, Interactive computer graphics, Interactive computer systems, Interactive media, Mesh generation, Previsualization, Real-Time Rendering, Splatting, Three dimensional computer graphics, Virtual production, Virtual Production (VP), Virtual Reality, Work-flows},
pubstate = {published},
tppubtype = {inproceedings}
}
Virtual production (VP) is transforming filmmaking by integrating real-time digital elements with live-action footage, offering new creative possibilities and streamlined workflows. While industry experts recognize AI's potential to revolutionize VP, its practical applications and value across different production phases and user groups remain underexplored. Building on initial research into generative and data-driven approaches, this paper presents the first systematic pilot study evaluating three types of AI-generated 3D environments - Depth Mesh, 360° Panoramic Meshes, and Gaussian Splatting - through the participation of 15 filmmaking professionals from diverse roles. Unlike commonly used 2D AI-generated visuals, our approach introduces navigable 3D environments that offer greater control and flexibility, aligning more closely with established VP workflows. Through expert interviews and literature research, we developed evaluation criteria to assess their usefulness beyond concept development, extending to previsualization, scene exploration, and interdisciplinary collaboration. Our findings indicate that different environments cater to distinct production needs, from early ideation to detailed visualization. Gaussian Splatting proved effective for high-fidelity previsualization, while 360° Panoramic Meshes excelled in rapid concept ideation. Despite their promise, challenges such as limited interactivity and customization highlight areas for improvement. Our prototype, EnVisualAIzer, built in Unreal Engine 5, provides an accessible platform for diverse filmmakers to engage with AI-generated environments, fostering a more inclusive production process. By lowering technical barriers, these environments have the potential to make advanced VP tools more widely available. This study offers valuable insights into the evolving role of AI in VP and sets the stage for future research and development. © 2025 Copyright held by the owner/author(s). Publication rights licensed to ACM.
Xing, Y.; Liu, Q.; Wang, J.; Gómez-Zará, D.
sMoRe: Spatial Mapping and Object Rendering Environment Proceedings Article
In: Int Conf Intell User Interfaces Proc IUI, pp. 115–119, Association for Computing Machinery, 2025, ISBN: 979-840071409-2 (ISBN).
Abstract | Links | BibTeX | Tags: Generative adversarial networks, Generative AI, Language Model, Large language model, large language models, Mapping, Mixed reality, Mixed-reality environment, Object rendering, Rendering (computer graphics), Space Manipulation, Spatial mapping, Spatial objects, Users' experiences, Virtual environments, Virtual objects
@inproceedings{xing_smore_2025,
title = {sMoRe: Spatial Mapping and Object Rendering Environment},
author = {Y. Xing and Q. Liu and J. Wang and D. Gómez-Zará},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105001670668&doi=10.1145%2f3708557.3716337&partnerID=40&md5=8ef4c5c4ef2b3ee30d00e4b8d19d19b8},
doi = {10.1145/3708557.3716337},
isbn = {979-840071409-2 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Int Conf Intell User Interfaces Proc IUI},
pages = {115–119},
publisher = {Association for Computing Machinery},
abstract = {In mixed reality (MR) environments, understanding space and creating virtual objects is crucial to providing an intuitive user experience. This paper introduces sMoRe (Spatial Mapping and Object Rendering Environment), an MR application that combines Generative AI (GenAI) to assist users in creating, placing, and managing virtual objects within physical spaces. sMoRe allows users to use voice or typed text commands to create and place virtual objects using GenAI while specifying spatial constraints. The system employs Large Language Models (LLMs) to interpret users’ commands, analyze the current scene, and identify optimal locations. Additionally, sMoRe integrates a text-to-3D generative model to dynamically create 3D objects based on users’ descriptions. Our user study demonstrates the effectiveness of sMoRe in enhancing user comprehension, interaction, and organization of the MR environment. © 2025 Copyright held by the owner/author(s).},
keywords = {Generative adversarial networks, Generative AI, Language Model, Large language model, large language models, Mapping, Mixed reality, Mixed-reality environment, Object rendering, Rendering (computer graphics), Space Manipulation, Spatial mapping, Spatial objects, Users' experiences, Virtual environments, Virtual objects},
pubstate = {published},
tppubtype = {inproceedings}
}
In mixed reality (MR) environments, understanding space and creating virtual objects is crucial to providing an intuitive user experience. This paper introduces sMoRe (Spatial Mapping and Object Rendering Environment), an MR application that combines Generative AI (GenAI) to assist users in creating, placing, and managing virtual objects within physical spaces. sMoRe allows users to use voice or typed text commands to create and place virtual objects using GenAI while specifying spatial constraints. The system employs Large Language Models (LLMs) to interpret users’ commands, analyze the current scene, and identify optimal locations. Additionally, sMoRe integrates a text-to-3D generative model to dynamically create 3D objects based on users’ descriptions. Our user study demonstrates the effectiveness of sMoRe in enhancing user comprehension, interaction, and organization of the MR environment. © 2025 Copyright held by the owner/author(s).
Lee, S.; Jeon, J.; Choe, H.
Enhancing Pre-Service Teachers' Global Englishes Awareness with Technology: A Focus on AI Chatbots in 3D Metaverse Environments Journal Article
In: TESOL Quarterly, vol. 59, no. 1, pp. 49–74, 2025, ISSN: 00398322 (ISSN).
Abstract | Links | BibTeX | Tags:
@article{lee_enhancing_2025,
title = {Enhancing Pre-Service Teachers' Global Englishes Awareness with Technology: A Focus on AI Chatbots in 3D Metaverse Environments},
author = {S. Lee and J. Jeon and H. Choe},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85182184604&doi=10.1002%2ftesq.3300&partnerID=40&md5=51c1058910bbd828d691f03be31c45b1},
doi = {10.1002/tesq.3300},
issn = {00398322 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {TESOL Quarterly},
volume = {59},
number = {1},
pages = {49–74},
abstract = {Although Global Englishes (GE) research continues to grow in English language teaching (ELT), the role of technology in enhancing GE awareness remains underexplored. Addressing this gap, the study investigates the potential of English as a lingua franca (ELF) interactions with artificial intelligence (AI) chatbots in raising GE awareness. Using a quasi-experimental design, 97 South Korean pre-service English teachers were divided into a control group (CG},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Although Global Englishes (GE) research continues to grow in English language teaching (ELT), the role of technology in enhancing GE awareness remains underexplored. Addressing this gap, the study investigates the potential of English as a lingua franca (ELF) interactions with artificial intelligence (AI) chatbots in raising GE awareness. Using a quasi-experimental design, 97 South Korean pre-service English teachers were divided into a control group (CG
Ding, S.; Chen, Y.
RAG-VR: Leveraging Retrieval-Augmented Generation for 3D Question Answering in VR Environments Proceedings Article
In: Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW, pp. 131–136, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 979-833151484-6 (ISBN).
Abstract | Links | BibTeX | Tags: Ambient intelligence, Computational Linguistics, Computer interaction, Computing methodologies, Computing methodologies-Artificial intelligence-Natural language processing-Natural language generation, Computing methodology-artificial intelligence-natural language processing-natural language generation, Data handling, Formal languages, Human computer interaction, Human computer interaction (HCI), Human-centered computing, Interaction paradigm, Interaction paradigms, Language Model, Language processing, Natural language generation, Natural language processing systems, Natural languages, Virtual Reality, Word processing
@inproceedings{ding_rag-vr_2025,
title = {RAG-VR: Leveraging Retrieval-Augmented Generation for 3D Question Answering in VR Environments},
author = {S. Ding and Y. Chen},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005140593&doi=10.1109%2fVRW66409.2025.00034&partnerID=40&md5=36dc5fef97aeea4d6e183c83ce9fcd89},
doi = {10.1109/VRW66409.2025.00034},
isbn = {979-833151484-6 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW},
pages = {131–136},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Recent advances in large language models (LLMs) provide new opportunities for context understanding in virtual reality (VR). However, VR contexts are often highly localized and personalized, limiting the effectiveness of general-purpose LLMs. To address this challenge, we present RAG-VR, the first 3D question-answering system for VR that incorporates retrieval-augmented generation (RAG), which augments an LLM with external knowledge retrieved from a localized knowledge database to improve the answer quality. RAG-VR includes a pipeline for extracting comprehensive knowledge about virtual environments and user conditions for accurate answer generation. To ensure efficient retrieval, RAG-VR offloads the retrieval process to a nearby edge server and uses only essential information during retrieval. Moreover, we train the retriever to effectively distinguish among relevant, irrelevant, and hard-to-differentiate information in relation to questions. RAG-VR improves answer accuracy by 17.9%-41.8% and reduces end-to-end latency by 34.5%-47.3% compared with two baseline systems. © 2025 IEEE.},
keywords = {Ambient intelligence, Computational Linguistics, Computer interaction, Computing methodologies, Computing methodologies-Artificial intelligence-Natural language processing-Natural language generation, Computing methodology-artificial intelligence-natural language processing-natural language generation, Data handling, Formal languages, Human computer interaction, Human computer interaction (HCI), Human-centered computing, Interaction paradigm, Interaction paradigms, Language Model, Language processing, Natural language generation, Natural language processing systems, Natural languages, Virtual Reality, Word processing},
pubstate = {published},
tppubtype = {inproceedings}
}
Recent advances in large language models (LLMs) provide new opportunities for context understanding in virtual reality (VR). However, VR contexts are often highly localized and personalized, limiting the effectiveness of general-purpose LLMs. To address this challenge, we present RAG-VR, the first 3D question-answering system for VR that incorporates retrieval-augmented generation (RAG), which augments an LLM with external knowledge retrieved from a localized knowledge database to improve the answer quality. RAG-VR includes a pipeline for extracting comprehensive knowledge about virtual environments and user conditions for accurate answer generation. To ensure efficient retrieval, RAG-VR offloads the retrieval process to a nearby edge server and uses only essential information during retrieval. Moreover, we train the retriever to effectively distinguish among relevant, irrelevant, and hard-to-differentiate information in relation to questions. RAG-VR improves answer accuracy by 17.9%-41.8% and reduces end-to-end latency by 34.5%-47.3% compared with two baseline systems. © 2025 IEEE.
Wei, X.; Wang, L.; Lee, L. -K.; Liu, R.
Multiple Generative AI Pedagogical Agents in Augmented Reality Environments: A Study on Implementing the 5E Model in Science Education Journal Article
In: Journal of Educational Computing Research, vol. 63, no. 2, pp. 336–371, 2025, ISSN: 07356331 (ISSN).
Abstract | Links | BibTeX | Tags: 5E learning model, Augmented Reality, elementary science education, generative artificial intelligence, Pedagogical agents
@article{wei_multiple_2025,
title = {Multiple Generative AI Pedagogical Agents in Augmented Reality Environments: A Study on Implementing the 5E Model in Science Education},
author = {X. Wei and L. Wang and L. -K. Lee and R. Liu},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85211165915&doi=10.1177%2f07356331241305519&partnerID=40&md5=ab592abf16398732391a5dd3bd4ca7ed},
doi = {10.1177/07356331241305519},
issn = {07356331 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Journal of Educational Computing Research},
volume = {63},
number = {2},
pages = {336–371},
abstract = {Notwithstanding the growing advantages of incorporating Augmented Reality (AR) in science education, the pedagogical use of AR combined with Pedagogical Agents (PAs) remains underexplored. Additionally, few studies have examined the integration of Generative Artificial Intelligence (GAI) into science education to create GAI-enhanced PAs (GPAs) that enrich the learning experiences. To address these gaps, this study designed and implemented a GPA-enhanced 5E model within AR environments to scaffold students’ science learning. A mixed-methods design was conducted to investigate the effectiveness of the proposed approach on students’ academic achievement, cognitive load, and their perceptions of GPAs as learning aids through using the 5E model. Sixty sixth-grade students from two complete classes were randomly assigned to either an experimental group engaged in AR science learning with a GPA-enhanced 5E approach or a control group that followed the traditional 5E method. The findings revealed that the GPA-enhanced 5E approach in AR environments significantly improved students’ academic achievement and decreased cognitive load. Furthermore, students in the experimental group reported positive perceptions of the GPA-enhanced 5E method during the AR science lessons. The findings offer valuable insights for instructional designers and educators who leverage advanced educational technologies to support science learning aligned with constructivist principles. © The Author(s) 2024.},
keywords = {5E learning model, Augmented Reality, elementary science education, generative artificial intelligence, Pedagogical agents},
pubstate = {published},
tppubtype = {article}
}
Notwithstanding the growing advantages of incorporating Augmented Reality (AR) in science education, the pedagogical use of AR combined with Pedagogical Agents (PAs) remains underexplored. Additionally, few studies have examined the integration of Generative Artificial Intelligence (GAI) into science education to create GAI-enhanced PAs (GPAs) that enrich the learning experiences. To address these gaps, this study designed and implemented a GPA-enhanced 5E model within AR environments to scaffold students’ science learning. A mixed-methods design was conducted to investigate the effectiveness of the proposed approach on students’ academic achievement, cognitive load, and their perceptions of GPAs as learning aids through using the 5E model. Sixty sixth-grade students from two complete classes were randomly assigned to either an experimental group engaged in AR science learning with a GPA-enhanced 5E approach or a control group that followed the traditional 5E method. The findings revealed that the GPA-enhanced 5E approach in AR environments significantly improved students’ academic achievement and decreased cognitive load. Furthermore, students in the experimental group reported positive perceptions of the GPA-enhanced 5E method during the AR science lessons. The findings offer valuable insights for instructional designers and educators who leverage advanced educational technologies to support science learning aligned with constructivist principles. © The Author(s) 2024.
Sinha, Y.; Shanmugam, S.; Sahu, Y. K.; Mukhopadhyay, A.; Biswas, P.
Diffuse Your Data Blues: Augmenting Low-Resource Datasets via User-Assisted Diffusion Proceedings Article
In: Int Conf Intell User Interfaces Proc IUI, pp. 538–552, Association for Computing Machinery, 2025, ISBN: 979-840071306-4 (ISBN).
Abstract | Links | BibTeX | Tags: Data gathering, Detection models, Diffusion Model, diffusion models, Efficient Augmentation, Image Composition, Industrial context, Mixed reality, Object Detection, Objects detection, Synthetic Dataset, Synthetic datasets, Training objects
@inproceedings{sinha_diffuse_2025,
title = {Diffuse Your Data Blues: Augmenting Low-Resource Datasets via User-Assisted Diffusion},
author = {Y. Sinha and S. Shanmugam and Y. K. Sahu and A. Mukhopadhyay and P. Biswas},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105001924293&doi=10.1145%2f3708359.3712163&partnerID=40&md5=c13cb6b2ef757546239de8b3ba93fb14},
doi = {10.1145/3708359.3712163},
isbn = {979-840071306-4 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Int Conf Intell User Interfaces Proc IUI},
pages = {538–552},
publisher = {Association for Computing Machinery},
abstract = {Mixed reality applications in industrial contexts necessitate extensive and varied datasets for training object detection models, yet actual data gathering may be obstructed by logistical or cost issues. This study investigates the implementation of generative AI methods to work on this issue for mixed reality applications, with an emphasis on assembly and disassembly tasks. The novel objects found in industrial settings are difficult to describe using words, making text-based models less effective. In this study, a diffusion model is used to generate images by combining novel objects with various backgrounds. The backgrounds are selected where object detection in specific applications has been ineffective. This approach efficiently produces a diverse range of training samples. We compare three approaches: traditional augmentation methods, GAN-based augmentation, and Diffusion-based augmentation. Results show that the diffusion model significantly improved detection metrics. For instance, applying diffusion models to the dataset containing mechanical components of a pneumatic cylinder raised the F1 Score from 69.77 to 84.21 and the mAP@50 from 76.48 to 88.77, resulting in an increase in object detection performance, with a 67% less dataset size compared to the traditional augmented dataset. The proposed image composition diffusion model and user-friendly interface further simplify dataset enrichment, proving effective for augmenting data and improving the robustness of detection models. © 2025 Copyright held by the owner/author(s).},
keywords = {Data gathering, Detection models, Diffusion Model, diffusion models, Efficient Augmentation, Image Composition, Industrial context, Mixed reality, Object Detection, Objects detection, Synthetic Dataset, Synthetic datasets, Training objects},
pubstate = {published},
tppubtype = {inproceedings}
}
Mixed reality applications in industrial contexts necessitate extensive and varied datasets for training object detection models, yet actual data gathering may be obstructed by logistical or cost issues. This study investigates the implementation of generative AI methods to work on this issue for mixed reality applications, with an emphasis on assembly and disassembly tasks. The novel objects found in industrial settings are difficult to describe using words, making text-based models less effective. In this study, a diffusion model is used to generate images by combining novel objects with various backgrounds. The backgrounds are selected where object detection in specific applications has been ineffective. This approach efficiently produces a diverse range of training samples. We compare three approaches: traditional augmentation methods, GAN-based augmentation, and Diffusion-based augmentation. Results show that the diffusion model significantly improved detection metrics. For instance, applying diffusion models to the dataset containing mechanical components of a pneumatic cylinder raised the F1 Score from 69.77 to 84.21 and the mAP@50 from 76.48 to 88.77, resulting in an increase in object detection performance, with a 67% less dataset size compared to the traditional augmented dataset. The proposed image composition diffusion model and user-friendly interface further simplify dataset enrichment, proving effective for augmenting data and improving the robustness of detection models. © 2025 Copyright held by the owner/author(s).
Gaglio, G. F.; Vinanzi, S.; Cangelosi, A.; Chella, A.
Intention Reading Architecture for Virtual Agents Proceedings Article
In: O., Palinko; L., Bodenhagen; J.-J., Cabibihan; K., Fischer; S., Šabanović; K., Winkle; L., Behera; S.S., Ge; D., Chrysostomou; W., Jiang; H., He (Ed.): Lect. Notes Comput. Sci., pp. 488–497, Springer Science and Business Media Deutschland GmbH, 2025, ISBN: 03029743 (ISSN); 978-981963521-4 (ISBN).
Abstract | Links | BibTeX | Tags: Chatbots, Cognitive Architecture, Cognitive Architectures, Computer simulation languages, Intelligent virtual agents, Intention Reading, Intention readings, Language Model, Large language model, Metaverse, Metaverses, Physical robots, Video-games, Virtual agent, Virtual assistants, Virtual contexts, Virtual environments, Virtual machine
@inproceedings{gaglio_intention_2025,
title = {Intention Reading Architecture for Virtual Agents},
author = {G. F. Gaglio and S. Vinanzi and A. Cangelosi and A. Chella},
editor = {Palinko O. and Bodenhagen L. and Cabibihan J.-J. and Fischer K. and Šabanović S. and Winkle K. and Behera L. and Ge S.S. and Chrysostomou D. and Jiang W. and He H.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105002042645&doi=10.1007%2f978-981-96-3522-1_41&partnerID=40&md5=70ccc7039785bb4ca4d45752f1d3587f},
doi = {10.1007/978-981-96-3522-1_41},
isbn = {03029743 (ISSN); 978-981963521-4 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Lect. Notes Comput. Sci.},
volume = {15561 LNAI},
pages = {488–497},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {This work presents the development of a virtual agent designed specifically for use in the Metaverse, video games, and other virtual environments, capable of performing intention reading on a human-controlled avatar through a cognitive architecture that endows it with contextual awareness. The paper explores the adaptation of a cognitive architecture, originally developed for physical robots, to a fully virtual context, where it is integrated with a Large Language Model to create highly communicative virtual assistants. Although this work primarily focuses on virtual applications, integrating cognitive architectures with LLMs marks a significant step toward creating collaborative artificial agents capable of providing meaningful support by deeply understanding context and user intentions in digital environments. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.},
keywords = {Chatbots, Cognitive Architecture, Cognitive Architectures, Computer simulation languages, Intelligent virtual agents, Intention Reading, Intention readings, Language Model, Large language model, Metaverse, Metaverses, Physical robots, Video-games, Virtual agent, Virtual assistants, Virtual contexts, Virtual environments, Virtual machine},
pubstate = {published},
tppubtype = {inproceedings}
}
This work presents the development of a virtual agent designed specifically for use in the Metaverse, video games, and other virtual environments, capable of performing intention reading on a human-controlled avatar through a cognitive architecture that endows it with contextual awareness. The paper explores the adaptation of a cognitive architecture, originally developed for physical robots, to a fully virtual context, where it is integrated with a Large Language Model to create highly communicative virtual assistants. Although this work primarily focuses on virtual applications, integrating cognitive architectures with LLMs marks a significant step toward creating collaborative artificial agents capable of providing meaningful support by deeply understanding context and user intentions in digital environments. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.
Wang, C.; Sundstedt, V.; Garro, V.
Generative Artificial Intelligence for Immersive Analytics Proceedings Article
In: T., Bashford-Rogers; D., Meneveaux; M., Ammi; M., Ziat; S., Jänicke; H., Purchase; P., Radeva; A., Furnari; K., Bouatouch; A.A., Sousa (Ed.): Proc. Int. Jt. Conf. Comput. Vis. Imaging Comput. Graph. Theory Appl., pp. 938–946, Science and Technology Publications, Lda, 2025, ISBN: 21845921 (ISSN).
Abstract | Links | BibTeX | Tags: Extended reality, generative artificial intelligence, Immersive analytics, Visualization
@inproceedings{wang_generative_2025,
title = {Generative Artificial Intelligence for Immersive Analytics},
author = {C. Wang and V. Sundstedt and V. Garro},
editor = {Bashford-Rogers T. and Meneveaux D. and Ammi M. and Ziat M. and Jänicke S. and Purchase H. and Radeva P. and Furnari A. and Bouatouch K. and Sousa A.A.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105001960708&doi=10.5220%2f0013308400003912&partnerID=40&md5=cb416a11c795ea8081730f6f339a0b4b},
doi = {10.5220/0013308400003912},
isbn = {21845921 (ISSN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. Int. Jt. Conf. Comput. Vis. Imaging Comput. Graph. Theory Appl.},
volume = {1},
pages = {938–946},
publisher = {Science and Technology Publications, Lda},
abstract = {Generative artificial intelligence (GenAI) models have advanced various applications with their ability to generate diverse forms of information, including text, images, audio, video, and 3D models. In visual computing, their primary applications have focused on creating graphic content and enabling data visualization on traditional desktop interfaces, which help automate visual analytics (VA) processes. With the rise of affordable immersive technologies, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), immersive analytics (IA) has been an emerging field offering unique opportunities for deeper engagement and understanding of complex data in immersive environments (IEs). However, IA system development remains resource-intensive and requires significant expertise, while integrating GenAI capabilities into IA is still under early exploration. Therefore, based on an analysis of recent publications in these fields, this position paper investigates how GenAI can support future IA systems for more effective data exploration with immersive experiences. Specifically, we discuss potential directions and key issues concerning future GenAI-supported IA applications. © 2025 by SCITEPRESS–Science and Technology Publications, Lda.},
keywords = {Extended reality, generative artificial intelligence, Immersive analytics, Visualization},
pubstate = {published},
tppubtype = {inproceedings}
}
Generative artificial intelligence (GenAI) models have advanced various applications with their ability to generate diverse forms of information, including text, images, audio, video, and 3D models. In visual computing, their primary applications have focused on creating graphic content and enabling data visualization on traditional desktop interfaces, which help automate visual analytics (VA) processes. With the rise of affordable immersive technologies, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), immersive analytics (IA) has been an emerging field offering unique opportunities for deeper engagement and understanding of complex data in immersive environments (IEs). However, IA system development remains resource-intensive and requires significant expertise, while integrating GenAI capabilities into IA is still under early exploration. Therefore, based on an analysis of recent publications in these fields, this position paper investigates how GenAI can support future IA systems for more effective data exploration with immersive experiences. Specifically, we discuss potential directions and key issues concerning future GenAI-supported IA applications. © 2025 by SCITEPRESS–Science and Technology Publications, Lda.
Yadav, R.; Huzooree, G.; Yadav, M.; Gangodawilage, D. S. K.
Generative AI for personalized learning content creation Book Section
In: Transformative AI Practices for Personalized Learning Strategies, pp. 107–130, IGI Global, 2025, ISBN: 979-836938746-7 (ISBN); 979-836938744-3 (ISBN).
Abstract | Links | BibTeX | Tags: Adaptive feedback, Advanced Analytics, AI systems, Contrastive Learning, Educational contents, Educational experiences, Enhanced learning, Ethical technology, Federated learning, Immersive, Learning content creation, Personalized learning, Student engagement, Students, Supervised learning, Tools and applications, Virtual Reality
@incollection{yadav_generative_2025,
title = {Generative AI for personalized learning content creation},
author = {R. Yadav and G. Huzooree and M. Yadav and D. S. K. Gangodawilage},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005387236&doi=10.4018%2f979-8-3693-8744-3.ch005&partnerID=40&md5=904e58b9c6de83dcd431c1706dda02b3},
doi = {10.4018/979-8-3693-8744-3.ch005},
isbn = {979-836938746-7 (ISBN); 979-836938744-3 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Transformative AI Practices for Personalized Learning Strategies},
pages = {107–130},
publisher = {IGI Global},
abstract = {Generative AI has emerged as a transformative force in personalized learning, offering unprecedented opportunities to tailor educational content to individual needs. By leveraging advanced algorithms and data analysis, AI systems can dynamically generate customized materials, provide adaptive feedback, and foster student engagement. This chapter explores the intersection of generative AI and personalized learning, discussing its techniques, tools, and applications in creating immersive and adaptive educational experiences. Key benefits include enhanced learning outcomes, efficiency, and scalability. However, challenges such as data privacy, algorithmic bias, and equitable access must be addressed to ensure responsible implementation. Future trends, including the integration of immersive technologies like Virtual Reality (VR) and predictive analytics, highlight AI's potential to revolutionize education. By navigating ethical considerations and fostering transparency, generative AI can become a powerful ally in creating inclusive, engaging, and student- centered learning environments. © 2025, IGI Global Scientific Publishing. All rights reserved.},
keywords = {Adaptive feedback, Advanced Analytics, AI systems, Contrastive Learning, Educational contents, Educational experiences, Enhanced learning, Ethical technology, Federated learning, Immersive, Learning content creation, Personalized learning, Student engagement, Students, Supervised learning, Tools and applications, Virtual Reality},
pubstate = {published},
tppubtype = {incollection}
}
Generative AI has emerged as a transformative force in personalized learning, offering unprecedented opportunities to tailor educational content to individual needs. By leveraging advanced algorithms and data analysis, AI systems can dynamically generate customized materials, provide adaptive feedback, and foster student engagement. This chapter explores the intersection of generative AI and personalized learning, discussing its techniques, tools, and applications in creating immersive and adaptive educational experiences. Key benefits include enhanced learning outcomes, efficiency, and scalability. However, challenges such as data privacy, algorithmic bias, and equitable access must be addressed to ensure responsible implementation. Future trends, including the integration of immersive technologies like Virtual Reality (VR) and predictive analytics, highlight AI's potential to revolutionize education. By navigating ethical considerations and fostering transparency, generative AI can become a powerful ally in creating inclusive, engaging, and student- centered learning environments. © 2025, IGI Global Scientific Publishing. All rights reserved.
Hassoulas, A.; Crawford, O.; Hemrom, S.; Almeida, A.; Coffey, M. J.; Hodgson, M.; Leveridge, B.; Karwa, D.; Lethbridge, A.; Williams, H.; Voisey, A.; Reed, K.; Patel, S.; Hart, K.; Shaw, H.
A pilot study investigating the efficacy of technology enhanced case based learning (CBL) in small group teaching Journal Article
In: Scientific Reports, vol. 15, no. 1, 2025, ISSN: 20452322 (ISSN).
Abstract | Links | BibTeX | Tags: coronavirus disease 2019, Covid-19, epidemiology, female, human, Humans, Learning, male, Medical, Medical student, Pilot Projects, pilot study, problem based learning, Problem-Based Learning, procedures, SARS-CoV-2, Severe acute respiratory syndrome coronavirus 2, Students, Teaching, Virtual Reality
@article{hassoulas_pilot_2025,
title = {A pilot study investigating the efficacy of technology enhanced case based learning (CBL) in small group teaching},
author = {A. Hassoulas and O. Crawford and S. Hemrom and A. Almeida and M. J. Coffey and M. Hodgson and B. Leveridge and D. Karwa and A. Lethbridge and H. Williams and A. Voisey and K. Reed and S. Patel and K. Hart and H. Shaw},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105004223025&doi=10.1038%2fs41598-025-99764-5&partnerID=40&md5=8588cac4c3ffe437e667ba4373e010ec},
doi = {10.1038/s41598-025-99764-5},
issn = {20452322 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Scientific Reports},
volume = {15},
number = {1},
abstract = {The recent paradigm shift in teaching provision within higher education, following the COVID-19 pandemic, has led to blended models of learning prevailing in the pedagogic literature and in education practice. This shift has also resulted in an abundance of tools and technologies coming to market. Whilst the value of integrating technology into teaching and assessment has been well-established in the literature, the magnitude of choice available to educators and to students can be overwhelming. The current pilot investigated the feasibility of integrating key technologies in delivering technology-enhanced learning (TEL) case-based learning (CBL) within a sample of year two medical students. The cohort was selected at random, as was the control group receiving conventional CBL. Both groups were matched on prior academic performance. The TEL-CBL group received (1) in-person tutorials delivered within an immersive learning suite, (2) access to 3D anatomy software to explore during their self-directed learning time, (3) virtual reality (VR) guided anatomy exploration during tutorials, (4) access to a generative AI-based simulated virtual patient repository to practice key skills such as communication and history taking, and (5) an immersive medical emergency simulation. Metrics assessed included formative academic performance, student learning experience, and confidence in relation to communication and clinical skills. The results revealed that the TEL-CBL group outperformed their peers in successive formative assessments (p < 0.05), engaged thoroughly with the technologies at their disposal, and reported that these technologies enhanced their learning experience. Furthermore, students reported that access to the GenAI-simulated virtual patient platform and the immersive medical emergency simulation improved their clinical confidence and gave them a useful insight into what they can expect during the clinical phase of their medical education. The results are discussed in relation to the advantages that key emerging technologies may play in enhancing student performance, experience and confidence. © The Author(s) 2025.},
keywords = {coronavirus disease 2019, Covid-19, epidemiology, female, human, Humans, Learning, male, Medical, Medical student, Pilot Projects, pilot study, problem based learning, Problem-Based Learning, procedures, SARS-CoV-2, Severe acute respiratory syndrome coronavirus 2, Students, Teaching, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
The recent paradigm shift in teaching provision within higher education, following the COVID-19 pandemic, has led to blended models of learning prevailing in the pedagogic literature and in education practice. This shift has also resulted in an abundance of tools and technologies coming to market. Whilst the value of integrating technology into teaching and assessment has been well-established in the literature, the magnitude of choice available to educators and to students can be overwhelming. The current pilot investigated the feasibility of integrating key technologies in delivering technology-enhanced learning (TEL) case-based learning (CBL) within a sample of year two medical students. The cohort was selected at random, as was the control group receiving conventional CBL. Both groups were matched on prior academic performance. The TEL-CBL group received (1) in-person tutorials delivered within an immersive learning suite, (2) access to 3D anatomy software to explore during their self-directed learning time, (3) virtual reality (VR) guided anatomy exploration during tutorials, (4) access to a generative AI-based simulated virtual patient repository to practice key skills such as communication and history taking, and (5) an immersive medical emergency simulation. Metrics assessed included formative academic performance, student learning experience, and confidence in relation to communication and clinical skills. The results revealed that the TEL-CBL group outperformed their peers in successive formative assessments (p < 0.05), engaged thoroughly with the technologies at their disposal, and reported that these technologies enhanced their learning experience. Furthermore, students reported that access to the GenAI-simulated virtual patient platform and the immersive medical emergency simulation improved their clinical confidence and gave them a useful insight into what they can expect during the clinical phase of their medical education. The results are discussed in relation to the advantages that key emerging technologies may play in enhancing student performance, experience and confidence. © The Author(s) 2025.