AHCI RESEARCH GROUP
Publications
Papers published in international journals,
proceedings of conferences, workshops and books.
OUR RESEARCH
Scientific Publications
How to
Here you can find the complete list of our publications.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
2025
Cruz, T. A. Da; Munoz, O.; Giligny, F.; Gouranton, V.
For a Perception of Monumentality in Eastern Arabia from the Neolithic to the Bronze Age: 3D Reconstruction and Multidimensional Simulations of Monuments and Landscapes Proceedings Article
In: Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW, pp. 47–50, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 979-833151484-6 (ISBN).
Abstract | Links | BibTeX | Tags: 3D reconstruction, 4D simulations, Archaeological Site, Bronze age, Digital elevation model, Eastern Arabia, Eastern arabium, Monumentality, Multidimensional simulation, Simulation virtual realities, Spatial dimension, Temporal dimensions, Three dimensional computer graphics, Virtual Reality
@inproceedings{da_cruz_for_2025,
title = {For a Perception of Monumentality in Eastern Arabia from the Neolithic to the Bronze Age: 3D Reconstruction and Multidimensional Simulations of Monuments and Landscapes},
author = {T. A. Da Cruz and O. Munoz and F. Giligny and V. Gouranton},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005139996&doi=10.1109%2fVRW66409.2025.00018&partnerID=40&md5=14e05ff7019a4c9d712fe42aef776c8d},
doi = {10.1109/VRW66409.2025.00018},
isbn = {979-833151484-6 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW},
pages = {47–50},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {The monumentality of Neolithic and Early Bronze Age (6th to 3rd millennium BC) structures in the Arabian Peninsula has never been approached through a comprehensive approach of simulations and reconstructions. As a result, its perception remains understudied. By combining archaeological and paleoenvironmental data, 3D reconstruction, 4D simulations, virtual reality and generative AI, this PhD research project proposes to analyse the perception of monuments, exploring their spatial, visual and temporal dimensions, in order to answer to the following question: how can we reconstruct and analyse the perception of monumentality in Eastern Arabia through 4D simulations, and how can the study of this perception influence our understanding of monumentality and territories?This article presents a work in progress, after three months of research of which one month on the Dhabtiyah archaeological site (Saudi Arabia, Eastern Province). © 2025 IEEE.},
keywords = {3D reconstruction, 4D simulations, Archaeological Site, Bronze age, Digital elevation model, Eastern Arabia, Eastern arabium, Monumentality, Multidimensional simulation, Simulation virtual realities, Spatial dimension, Temporal dimensions, Three dimensional computer graphics, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
The monumentality of Neolithic and Early Bronze Age (6th to 3rd millennium BC) structures in the Arabian Peninsula has never been approached through a comprehensive approach of simulations and reconstructions. As a result, its perception remains understudied. By combining archaeological and paleoenvironmental data, 3D reconstruction, 4D simulations, virtual reality and generative AI, this PhD research project proposes to analyse the perception of monuments, exploring their spatial, visual and temporal dimensions, in order to answer to the following question: how can we reconstruct and analyse the perception of monumentality in Eastern Arabia through 4D simulations, and how can the study of this perception influence our understanding of monumentality and territories?This article presents a work in progress, after three months of research of which one month on the Dhabtiyah archaeological site (Saudi Arabia, Eastern Province). © 2025 IEEE.
Liu, G.; Du, H.; Wang, J.; Niyato, D.; Kim, D. I.
Contract-Inspired Contest Theory for Controllable Image Generation in Mobile Edge Metaverse Journal Article
In: IEEE Transactions on Mobile Computing, 2025, ISSN: 15361233 (ISSN).
Abstract | Links | BibTeX | Tags: Contest Theory, Deep learning, Deep reinforcement learning, Diffusion Model, Generative adversarial networks, Generative AI, High quality, Image generation, Image generations, Immersive technologies, Metaverses, Mobile edge computing, Reinforcement Learning, Reinforcement learnings, Resource allocation, Resources allocation, Semantic data, Virtual addresses, Virtual environments, Virtual Reality
@article{liu_contract-inspired_2025,
title = {Contract-Inspired Contest Theory for Controllable Image Generation in Mobile Edge Metaverse},
author = {G. Liu and H. Du and J. Wang and D. Niyato and D. I. Kim},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105000066834&doi=10.1109%2fTMC.2025.3550815&partnerID=40&md5=3cb5a2143b9ce4ca7f931a60f1bf239c},
doi = {10.1109/TMC.2025.3550815},
issn = {15361233 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Transactions on Mobile Computing},
abstract = {The rapid advancement of immersive technologies has propelled the development of the Metaverse, where the convergence of virtual and physical realities necessitates the generation of high-quality, photorealistic images to enhance user experience. However, generating these images, especially through Generative Diffusion Models (GDMs), in mobile edge computing environments presents significant challenges due to the limited computing resources of edge devices and the dynamic nature of wireless networks. This paper proposes a novel framework that integrates contract-inspired contest theory, Deep Reinforcement Learning (DRL), and GDMs to optimize image generation in these resource-constrained environments. The framework addresses the critical challenges of resource allocation and semantic data transmission quality by incentivizing edge devices to efficiently transmit high-quality semantic data, which is essential for creating realistic and immersive images. The use of contest and contract theory ensures that edge devices are motivated to allocate resources effectively, while DRL dynamically adjusts to network conditions, optimizing the overall image generation process. Experimental results demonstrate that the proposed approach not only improves the quality of generated images but also achieves superior convergence speed and stability compared to traditional methods. This makes the framework particularly effective for optimizing complex resource allocation tasks in mobile edge Metaverse applications, offering enhanced performance and efficiency in creating immersive virtual environments. © 2002-2012 IEEE.},
keywords = {Contest Theory, Deep learning, Deep reinforcement learning, Diffusion Model, Generative adversarial networks, Generative AI, High quality, Image generation, Image generations, Immersive technologies, Metaverses, Mobile edge computing, Reinforcement Learning, Reinforcement learnings, Resource allocation, Resources allocation, Semantic data, Virtual addresses, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
The rapid advancement of immersive technologies has propelled the development of the Metaverse, where the convergence of virtual and physical realities necessitates the generation of high-quality, photorealistic images to enhance user experience. However, generating these images, especially through Generative Diffusion Models (GDMs), in mobile edge computing environments presents significant challenges due to the limited computing resources of edge devices and the dynamic nature of wireless networks. This paper proposes a novel framework that integrates contract-inspired contest theory, Deep Reinforcement Learning (DRL), and GDMs to optimize image generation in these resource-constrained environments. The framework addresses the critical challenges of resource allocation and semantic data transmission quality by incentivizing edge devices to efficiently transmit high-quality semantic data, which is essential for creating realistic and immersive images. The use of contest and contract theory ensures that edge devices are motivated to allocate resources effectively, while DRL dynamically adjusts to network conditions, optimizing the overall image generation process. Experimental results demonstrate that the proposed approach not only improves the quality of generated images but also achieves superior convergence speed and stability compared to traditional methods. This makes the framework particularly effective for optimizing complex resource allocation tasks in mobile edge Metaverse applications, offering enhanced performance and efficiency in creating immersive virtual environments. © 2002-2012 IEEE.
Grubert, J.; Schmalstieg, D.; Dickhaut, K.
Towards Supporting Literary Studies Using Virtual Reality and Generative Artificial Intelligence Proceedings Article
In: Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW, pp. 147–149, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 979-833151484-6 (ISBN).
Abstract | Links | BibTeX | Tags: Cultural-historical, generative artificial intelligence, Immersive, literary studies, Literary study, Literary texts, Literature analysis, Textual-analysis, Virtual Reality, Visual elements
@inproceedings{grubert_towards_2025,
title = {Towards Supporting Literary Studies Using Virtual Reality and Generative Artificial Intelligence},
author = {J. Grubert and D. Schmalstieg and K. Dickhaut},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005144426&doi=10.1109%2fVRW66409.2025.00037&partnerID=40&md5=0315225b4c49f5f87bca94f82f41281c},
doi = {10.1109/VRW66409.2025.00037},
isbn = {979-833151484-6 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW},
pages = {147–149},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Literary studies critically examine fictional texts, exploring their structures, themes, stylistic features, and cultural-historical contexts. A central challenge in this field lies in bridging textual analysis with the spatial and sensory dimensions of settings described or implied in texts. Traditional methodologies often require scholars to mentally reconstruct these environments, leading to incomplete or inconsistent interpretations. Readers may be biased by their personal context or experiences, or may lack detailed knowledge of the relevant historical facts. This paper argues for the integration of virtual reality and generative artificial intelligence as supporting instruments to enhance literary research. The former enables immersive, spatially accurate reconstructions of historical environments, while the latter provides tools such as text-to-image and text-to-3D generation which let us dynamically render visual elements quoted in literary texts. Together, these technologies have the potential to significantly enhance traditional literature analysis methodologies, enabling novel approaches for contextualizing and analyzing literature in its spatial and cultural milieu. © 2025 IEEE.},
keywords = {Cultural-historical, generative artificial intelligence, Immersive, literary studies, Literary study, Literary texts, Literature analysis, Textual-analysis, Virtual Reality, Visual elements},
pubstate = {published},
tppubtype = {inproceedings}
}
Literary studies critically examine fictional texts, exploring their structures, themes, stylistic features, and cultural-historical contexts. A central challenge in this field lies in bridging textual analysis with the spatial and sensory dimensions of settings described or implied in texts. Traditional methodologies often require scholars to mentally reconstruct these environments, leading to incomplete or inconsistent interpretations. Readers may be biased by their personal context or experiences, or may lack detailed knowledge of the relevant historical facts. This paper argues for the integration of virtual reality and generative artificial intelligence as supporting instruments to enhance literary research. The former enables immersive, spatially accurate reconstructions of historical environments, while the latter provides tools such as text-to-image and text-to-3D generation which let us dynamically render visual elements quoted in literary texts. Together, these technologies have the potential to significantly enhance traditional literature analysis methodologies, enabling novel approaches for contextualizing and analyzing literature in its spatial and cultural milieu. © 2025 IEEE.
Wang, W. -S.; Lin, C. -J.; Lee, H. -Y.; Huang, Y. -M.; Wu, T. -T.
Integrating feedback mechanisms and ChatGPT for VR-based experiential learning: impacts on reflective thinking and AIoT physical hands-on tasks Journal Article
In: Interactive Learning Environments, vol. 33, no. 2, pp. 1770–1787, 2025, ISSN: 10494820 (ISSN).
Abstract | Links | BibTeX | Tags: AIoT, feedback mechanisms, generative artificial intelligence, physical hands-on tasks, reflective thinking, Virtual Reality
@article{wang_integrating_2025,
title = {Integrating feedback mechanisms and ChatGPT for VR-based experiential learning: impacts on reflective thinking and AIoT physical hands-on tasks},
author = {W. -S. Wang and C. -J. Lin and H. -Y. Lee and Y. -M. Huang and T. -T. Wu},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105001238541&doi=10.1080%2f10494820.2024.2375644&partnerID=40&md5=136ac17a3a460dfd98cf7baa6439062e},
doi = {10.1080/10494820.2024.2375644},
issn = {10494820 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Interactive Learning Environments},
volume = {33},
number = {2},
pages = {1770–1787},
abstract = {This study investigates the application of Virtual Reality (VR) in the educational field, particularly its integration with GAI technologies such as ChatGPT to enhance the learning experience. The research indicates that while VR provides an immersive learning environment fostering student interaction and interest, the lack of a structured learning framework and personalized feedback may limit its educational effectiveness and potentially affect the transfer of VR-learned knowledge to physical hands-on tasks. Hence, it calls for the provision of more targeted and personalized feedback in VR learning environments. Through a randomized controlled trial (RCT), this study collected data from 77 university students, integrating experiential learning in VR for acquiring AIoT knowledge and practical skills, and compared the effects of traditional feedback versus GPT feedback on promoting reflective thinking, learning motivation, cognitive levels, and AIoT hands-on abilities among the students. The results show that the group receiving GPT feedback significantly outperformed the control group across these learning indicators, demonstrating the effectiveness of GAI technologies in providing personalized learning support, facilitating deep learning, and enhancing educational outcomes. This study offers new insights into the integration of GAI technology in VR learning environments, paving new pathways for the development and application of future educational technologies. © 2024 Informa UK Limited, trading as Taylor & Francis Group.},
keywords = {AIoT, feedback mechanisms, generative artificial intelligence, physical hands-on tasks, reflective thinking, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
This study investigates the application of Virtual Reality (VR) in the educational field, particularly its integration with GAI technologies such as ChatGPT to enhance the learning experience. The research indicates that while VR provides an immersive learning environment fostering student interaction and interest, the lack of a structured learning framework and personalized feedback may limit its educational effectiveness and potentially affect the transfer of VR-learned knowledge to physical hands-on tasks. Hence, it calls for the provision of more targeted and personalized feedback in VR learning environments. Through a randomized controlled trial (RCT), this study collected data from 77 university students, integrating experiential learning in VR for acquiring AIoT knowledge and practical skills, and compared the effects of traditional feedback versus GPT feedback on promoting reflective thinking, learning motivation, cognitive levels, and AIoT hands-on abilities among the students. The results show that the group receiving GPT feedback significantly outperformed the control group across these learning indicators, demonstrating the effectiveness of GAI technologies in providing personalized learning support, facilitating deep learning, and enhancing educational outcomes. This study offers new insights into the integration of GAI technology in VR learning environments, paving new pathways for the development and application of future educational technologies. © 2024 Informa UK Limited, trading as Taylor & Francis Group.
Miller, C. H.
Digital Storytelling: A Creator’s Guide to Interactive Entertainment: Volume I, Fifth Edition Book
CRC Press, 2025, ISBN: 978-104034442-2 (ISBN); 978-103285888-3 (ISBN).
Abstract | Links | BibTeX | Tags: Case-studies, Chatbots, Creatives, Digital storytelling, Entertainment, Immersive environment, Interactive documentary, Interactive entertainment, Social media, Use of video, Video-games, Virtual Reality
@book{miller_digital_2025,
title = {Digital Storytelling: A Creator’s Guide to Interactive Entertainment: Volume I, Fifth Edition},
author = {C. H. Miller},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105004122515&doi=10.1201%2f9781003520092&partnerID=40&md5=894bfbb310cbd095a54409f9ac5174da},
doi = {10.1201/9781003520092},
isbn = {978-104034442-2 (ISBN); 978-103285888-3 (ISBN)},
year = {2025},
date = {2025-01-01},
volume = {1},
publisher = {CRC Press},
series = {Digital Storytelling: a Creator's Guide to Interactive Entertainment: Volume I, Fifth Edition},
abstract = {Digital Storytelling: A Creator’s Guide to Interactive Entertainment, Volume I, fifth edition delves into the fascinating and groundbreaking stories enabled by interactive digital media, examining both fictional and non-fiction narratives. This fifth edition explores monumental developments, particularly the emergence of generative AI, and highlights exciting projects utilizing this technology. Additionally, it covers social media; interactive documentaries; immersive environments; and innovative uses of video games, chatbots, and virtual reality. Carolyn Handler Miller provides insights into storytelling essentials like character development, plot, structure, dialogue, and emotion, while examining how digital media and interactivity influence these elements. This book also dives into advanced topics, such as narratives using AR, VR, and XR, alongside new forms of immersive media, including large screens, escape rooms, and theme park experiences. With numerous case studies, this edition illustrates the creative possibilities of digital storytelling and its applications beyond entertainment, such as education, training, information, and promotion. Interviews with industry leaders further enhance the understanding of this evolving universe, making it a valuable resource for both professionals and enthusiasts. Key Features: • This book includes up-to-the-minute developments in digital storytelling. • It offers case studies of noteworthy examples of digital storytelling. • It includes a glossary clearly defining new or difficult terms. • Each chapter opens with several thought-provoking questions about the chapter’s topic. • Each chapter concludes with several creative and engaging exercises to promote the reader’s understanding of the chapter’s topic. © 2025 Carolyn Handler Miller.},
keywords = {Case-studies, Chatbots, Creatives, Digital storytelling, Entertainment, Immersive environment, Interactive documentary, Interactive entertainment, Social media, Use of video, Video-games, Virtual Reality},
pubstate = {published},
tppubtype = {book}
}
Digital Storytelling: A Creator’s Guide to Interactive Entertainment, Volume I, fifth edition delves into the fascinating and groundbreaking stories enabled by interactive digital media, examining both fictional and non-fiction narratives. This fifth edition explores monumental developments, particularly the emergence of generative AI, and highlights exciting projects utilizing this technology. Additionally, it covers social media; interactive documentaries; immersive environments; and innovative uses of video games, chatbots, and virtual reality. Carolyn Handler Miller provides insights into storytelling essentials like character development, plot, structure, dialogue, and emotion, while examining how digital media and interactivity influence these elements. This book also dives into advanced topics, such as narratives using AR, VR, and XR, alongside new forms of immersive media, including large screens, escape rooms, and theme park experiences. With numerous case studies, this edition illustrates the creative possibilities of digital storytelling and its applications beyond entertainment, such as education, training, information, and promotion. Interviews with industry leaders further enhance the understanding of this evolving universe, making it a valuable resource for both professionals and enthusiasts. Key Features: • This book includes up-to-the-minute developments in digital storytelling. • It offers case studies of noteworthy examples of digital storytelling. • It includes a glossary clearly defining new or difficult terms. • Each chapter opens with several thought-provoking questions about the chapter’s topic. • Each chapter concludes with several creative and engaging exercises to promote the reader’s understanding of the chapter’s topic. © 2025 Carolyn Handler Miller.
Koizumi, M.; Ohsuga, M.; Corchado, J. M.
Development and Assessment of a System to Help Students Improve Self-compassion Proceedings Article
In: R., Chinthaginjala; P., Sitek; N., Min-Allah; K., Matsui; S., Ossowski; S., Rodríguez (Ed.): Lect. Notes Networks Syst., pp. 43–52, Springer Science and Business Media Deutschland GmbH, 2025, ISBN: 23673370 (ISSN); 978-303182072-4 (ISBN).
Abstract | Links | BibTeX | Tags: Avatar, Generative adversarial networks, Generative AI, Health issues, Mental health, Self-compassion, Students, Training program, University students, Virtual avatar, Virtual environments, Virtual Reality, Virtual Space, Virtual spaces, Visual imagery
@inproceedings{koizumi_development_2025,
title = {Development and Assessment of a System to Help Students Improve Self-compassion},
author = {M. Koizumi and M. Ohsuga and J. M. Corchado},
editor = {Chinthaginjala R. and Sitek P. and Min-Allah N. and Matsui K. and Ossowski S. and Rodríguez S.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85218979175&doi=10.1007%2f978-3-031-82073-1_5&partnerID=40&md5=b136d4a114ce5acfa89f907ccecc145f},
doi = {10.1007/978-3-031-82073-1_5},
isbn = {23673370 (ISSN); 978-303182072-4 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Lect. Notes Networks Syst.},
volume = {1259},
pages = {43–52},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {Mental health issues are becoming more prevalent among university students. The mindful self-compassion (MSC) training program, which was introduced to address this issue, has shown some efficacy. However, many people, particularly Japanese people, have difficulty recalling visual imagery or feel uncomfortable or resistant to treating themselves with compassion. This study proposes and develops a system that uses virtual space and avatars to help individuals improve their self-compassion. In the proposed system, the user first selects an avatar of a person with whom to talk (hereafter referred to as “partner”), and then talks about the problem to the avatar of his/her choice. Next, the user changes viewpoints and listens to the problem as the partner’s avatar and responds with compassion. Finally, the user returns to his/her own avatar and listens to the compassionate response spoken as the partner avatar. We first conducted surveys to understand the important system components, and then developed prototypes. In light of the results of the experiments, we improved the prototype by introducing a generative AI. The first prototype used the user’s spoken voice as it was, but the improved system uses the generative AI to organize and convert the voice and present it. In addition, we added a function to generate and add advice with compression. The proposed system is expected to contribute to the improvement of students’ self-compassion. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.},
keywords = {Avatar, Generative adversarial networks, Generative AI, Health issues, Mental health, Self-compassion, Students, Training program, University students, Virtual avatar, Virtual environments, Virtual Reality, Virtual Space, Virtual spaces, Visual imagery},
pubstate = {published},
tppubtype = {inproceedings}
}
Mental health issues are becoming more prevalent among university students. The mindful self-compassion (MSC) training program, which was introduced to address this issue, has shown some efficacy. However, many people, particularly Japanese people, have difficulty recalling visual imagery or feel uncomfortable or resistant to treating themselves with compassion. This study proposes and develops a system that uses virtual space and avatars to help individuals improve their self-compassion. In the proposed system, the user first selects an avatar of a person with whom to talk (hereafter referred to as “partner”), and then talks about the problem to the avatar of his/her choice. Next, the user changes viewpoints and listens to the problem as the partner’s avatar and responds with compassion. Finally, the user returns to his/her own avatar and listens to the compassionate response spoken as the partner avatar. We first conducted surveys to understand the important system components, and then developed prototypes. In light of the results of the experiments, we improved the prototype by introducing a generative AI. The first prototype used the user’s spoken voice as it was, but the improved system uses the generative AI to organize and convert the voice and present it. In addition, we added a function to generate and add advice with compression. The proposed system is expected to contribute to the improvement of students’ self-compassion. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
Tortora, A.; Amaro, I.; Greca, A. Della; Barra, P.
Exploring the Role of Generative Artificial Intelligence in Virtual Reality: Opportunities and Future Perspectives Proceedings Article
In: J.Y.C., Chen; G., Fragomeni (Ed.): Lect. Notes Comput. Sci., pp. 125–142, Springer Science and Business Media Deutschland GmbH, 2025, ISBN: 03029743 (ISSN); 978-303193699-9 (ISBN).
Abstract | Links | BibTeX | Tags: Ethical technology, Future perspectives, Generative AI, Image modeling, Immersive, immersive experience, Immersive Experiences, Information Management, Language Model, Personnel training, Professional training, Real- time, Sensitive data, Training design, Users' experiences, Virtual Reality
@inproceedings{tortora_exploring_2025,
title = {Exploring the Role of Generative Artificial Intelligence in Virtual Reality: Opportunities and Future Perspectives},
author = {A. Tortora and I. Amaro and A. Della Greca and P. Barra},
editor = {Chen J.Y.C. and Fragomeni G.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105007788684&doi=10.1007%2f978-3-031-93700-2_9&partnerID=40&md5=7b69183bbf8172f9595f939254fb6831},
doi = {10.1007/978-3-031-93700-2_9},
isbn = {03029743 (ISSN); 978-303193699-9 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Lect. Notes Comput. Sci.},
volume = {15788 LNCS},
pages = {125–142},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {In recent years, generative AI, such as language and image models, have started to revolutionize virtual reality (VR) by offering new opportunities for immersive and personalized interaction. This paper explores the potential of these Intelligent Augmentation technologies in the context of VR, analyzing how the generation of text and images in real time can enhance the user experience through dynamic and personalized environments and contents. The integration of generative AI in VR scenarios holds promise in multiple fields, including education, professional training, design, and healthcare. However, their implementation involves significant challenges, such as privacy management, data security, and ethical issues related to cognitive manipulation and representation of reality. Through an overview of current applications and future prospects, this paper highlights the crucial role of generative AI in enhancing VR, helping to outline a path for the ethical and sustainable development of these immersive technologies. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.},
keywords = {Ethical technology, Future perspectives, Generative AI, Image modeling, Immersive, immersive experience, Immersive Experiences, Information Management, Language Model, Personnel training, Professional training, Real- time, Sensitive data, Training design, Users' experiences, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
In recent years, generative AI, such as language and image models, have started to revolutionize virtual reality (VR) by offering new opportunities for immersive and personalized interaction. This paper explores the potential of these Intelligent Augmentation technologies in the context of VR, analyzing how the generation of text and images in real time can enhance the user experience through dynamic and personalized environments and contents. The integration of generative AI in VR scenarios holds promise in multiple fields, including education, professional training, design, and healthcare. However, their implementation involves significant challenges, such as privacy management, data security, and ethical issues related to cognitive manipulation and representation of reality. Through an overview of current applications and future prospects, this paper highlights the crucial role of generative AI in enhancing VR, helping to outline a path for the ethical and sustainable development of these immersive technologies. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
Li, Y.; Pang, E. C. H.; Ng, C. S. Y.; Azim, M.; Leung, H.
Enhancing Linear Algebra Education with AI-Generated Content in the CityU Metaverse: A Comparative Study Proceedings Article
In: T., Hao; J.G., Wu; X., Luo; Y., Sun; Y., Mu; S., Ge; W., Xie (Ed.): Lect. Notes Comput. Sci., pp. 3–16, Springer Science and Business Media Deutschland GmbH, 2025, ISBN: 03029743 (ISSN); 978-981964406-3 (ISBN).
Abstract | Links | BibTeX | Tags: Comparatives studies, Digital age, Digital interactions, digital twin, Educational metaverse, Engineering education, Generative AI, Immersive, Matrix algebra, Metaverse, Metaverses, Personnel training, Students, Teaching, University campus, Virtual environments, virtual learning environment, Virtual learning environments, Virtual Reality, Virtualization
@inproceedings{li_enhancing_2025,
title = {Enhancing Linear Algebra Education with AI-Generated Content in the CityU Metaverse: A Comparative Study},
author = {Y. Li and E. C. H. Pang and C. S. Y. Ng and M. Azim and H. Leung},
editor = {Hao T. and Wu J.G. and Luo X. and Sun Y. and Mu Y. and Ge S. and Xie W.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105003632691&doi=10.1007%2f978-981-96-4407-0_1&partnerID=40&md5=c067ba5d4c15e9c0353bf315680531fc},
doi = {10.1007/978-981-96-4407-0_1},
isbn = {03029743 (ISSN); 978-981964406-3 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Lect. Notes Comput. Sci.},
volume = {15589 LNCS},
pages = {3–16},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {In today’s digital age, the metaverse is emerging as the forthcoming evolution of the internet. It provides an immersive space that marks a new frontier in the way digital interactions are facilitated and experienced. In this paper, we present the CityU Metaverse, which aims to construct a digital twin of our university campus. It is designed as an educational virtual world where learning applications can be embedded in this virtual campus, supporting not only remote and collaborative learning but also professional technical training to enhance educational experiences through immersive and interactive learning. To evaluate the effectiveness of this educational metaverse, we conducted an experiment focused on 3D linear transformation in linear algebra, with teaching content generated by generative AI, comparing our metaverse system with traditional teaching methods. Knowledge tests and surveys assessing learning interest revealed that students engaged with the CityU Metaverse, facilitated by AI-generated content, outperformed those in traditional settings and reported greater enjoyment during the learning process. The work provides valuable perspectives on the behaviors and interactions within the metaverse by analyzing user preferences and learning outcomes. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.},
keywords = {Comparatives studies, Digital age, Digital interactions, digital twin, Educational metaverse, Engineering education, Generative AI, Immersive, Matrix algebra, Metaverse, Metaverses, Personnel training, Students, Teaching, University campus, Virtual environments, virtual learning environment, Virtual learning environments, Virtual Reality, Virtualization},
pubstate = {published},
tppubtype = {inproceedings}
}
In today’s digital age, the metaverse is emerging as the forthcoming evolution of the internet. It provides an immersive space that marks a new frontier in the way digital interactions are facilitated and experienced. In this paper, we present the CityU Metaverse, which aims to construct a digital twin of our university campus. It is designed as an educational virtual world where learning applications can be embedded in this virtual campus, supporting not only remote and collaborative learning but also professional technical training to enhance educational experiences through immersive and interactive learning. To evaluate the effectiveness of this educational metaverse, we conducted an experiment focused on 3D linear transformation in linear algebra, with teaching content generated by generative AI, comparing our metaverse system with traditional teaching methods. Knowledge tests and surveys assessing learning interest revealed that students engaged with the CityU Metaverse, facilitated by AI-generated content, outperformed those in traditional settings and reported greater enjoyment during the learning process. The work provides valuable perspectives on the behaviors and interactions within the metaverse by analyzing user preferences and learning outcomes. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.
Casas, L.; Hannah, S.; Mitchell, K.
HoloJig: Interactive Spoken Prompt Specified Generative AI Environments Journal Article
In: IEEE Computer Graphics and Applications, vol. 45, no. 2, pp. 69–77, 2025, ISSN: 02721716 (ISSN).
Abstract | Links | BibTeX | Tags: 3-D rendering, Article, Collaborative workspace, customer experience, Economic and social effects, generative artificial intelligence, human, Immersive, Immersive environment, parallax, Real- time, simulation, Simulation training, speech, Time based, Virtual environments, Virtual Reality, Virtual reality experiences, Virtual spaces, VR systems
@article{casas_holojig_2025,
title = {HoloJig: Interactive Spoken Prompt Specified Generative AI Environments},
author = {L. Casas and S. Hannah and K. Mitchell},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105001182100&doi=10.1109%2fMCG.2025.3553780&partnerID=40&md5=ec5dc44023314b6f9221169357d81dcd},
doi = {10.1109/MCG.2025.3553780},
issn = {02721716 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Computer Graphics and Applications},
volume = {45},
number = {2},
pages = {69–77},
abstract = {HoloJig offers an interactive, speech-to-virtual reality (VR), VR experience that generates diverse environments in real time based on live spoken descriptions. Unlike traditional VR systems that rely on prebuilt assets, HoloJig dynamically creates personalized and immersive virtual spaces with depth-based parallax 3-D rendering, allowing users to define the characteristics of their immersive environment through verbal prompts. This generative approach opens up new possibilities for interactive experiences, including simulations, training, collaborative workspaces, and entertainment. In addition to speech-to-VR environment generation, a key innovation of HoloJig is its progressive visual transition mechanism, which smoothly dissolves between previously generated and newly requested environments, mitigating the delay caused by neural computations. This feature ensures a seamless and continuous user experience, even as new scenes are being rendered on remote servers. © 1981-2012 IEEE.},
keywords = {3-D rendering, Article, Collaborative workspace, customer experience, Economic and social effects, generative artificial intelligence, human, Immersive, Immersive environment, parallax, Real- time, simulation, Simulation training, speech, Time based, Virtual environments, Virtual Reality, Virtual reality experiences, Virtual spaces, VR systems},
pubstate = {published},
tppubtype = {article}
}
HoloJig offers an interactive, speech-to-virtual reality (VR), VR experience that generates diverse environments in real time based on live spoken descriptions. Unlike traditional VR systems that rely on prebuilt assets, HoloJig dynamically creates personalized and immersive virtual spaces with depth-based parallax 3-D rendering, allowing users to define the characteristics of their immersive environment through verbal prompts. This generative approach opens up new possibilities for interactive experiences, including simulations, training, collaborative workspaces, and entertainment. In addition to speech-to-VR environment generation, a key innovation of HoloJig is its progressive visual transition mechanism, which smoothly dissolves between previously generated and newly requested environments, mitigating the delay caused by neural computations. This feature ensures a seamless and continuous user experience, even as new scenes are being rendered on remote servers. © 1981-2012 IEEE.
Logothetis, I.; Diakogiannis, K.; Vidakis, N.
Interactive Learning Through Conversational Avatars and Immersive VR: Enhancing Diabetes Education and Self-Management Proceedings Article
In: X., Fang (Ed.): Lect. Notes Comput. Sci., pp. 415–429, Springer Science and Business Media Deutschland GmbH, 2025, ISBN: 03029743 (ISSN); 978-303192577-1 (ISBN).
Abstract | Links | BibTeX | Tags: Artificial intelligence, Chronic disease, Computer aided instruction, Diabetes Education, Diagnosis, E-Learning, Education management, Engineering education, Gamification, Immersive virtual reality, Interactive computer graphics, Interactive learning, Large population, Learning systems, NUI, Self management, Serious game, Serious games, simulation, Virtual Reality
@inproceedings{logothetis_interactive_2025,
title = {Interactive Learning Through Conversational Avatars and Immersive VR: Enhancing Diabetes Education and Self-Management},
author = {I. Logothetis and K. Diakogiannis and N. Vidakis},
editor = {Fang X.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105008266480&doi=10.1007%2f978-3-031-92578-8_27&partnerID=40&md5=451274dfa3ef0b3f1b39c7d5a665ee3b},
doi = {10.1007/978-3-031-92578-8_27},
isbn = {03029743 (ISSN); 978-303192577-1 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Lect. Notes Comput. Sci.},
volume = {15816 LNCS},
pages = {415–429},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {Diabetes is a chronic disease affecting a large population of the world. Education and self-management of diabetes are crucial. Technologies such as Virtual Reality (VR) have presented promising results in healthcare education, while studies suggest that Artificial Intelligence (AI) can help in learning by further engaging the learner. This study aims to educate users on the entire routine of managing diabetes. The serious game utilizes VR for realistic interaction with diabetes tools and generative AI through a conversational avatar that acts as an assistant instructor. In this way, it allows users to practice diagnostic and therapeutic interventions in a controlled virtual environment, helping to build their understanding and confidence in diabetes management. To measure the effects of the proposed serious game, presence, and perceived agency were measured. Preliminary results indicate that this setup aids in the engagement and immersion of learners, while the avatar can provide helpful information during gameplay. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.},
keywords = {Artificial intelligence, Chronic disease, Computer aided instruction, Diabetes Education, Diagnosis, E-Learning, Education management, Engineering education, Gamification, Immersive virtual reality, Interactive computer graphics, Interactive learning, Large population, Learning systems, NUI, Self management, Serious game, Serious games, simulation, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
Diabetes is a chronic disease affecting a large population of the world. Education and self-management of diabetes are crucial. Technologies such as Virtual Reality (VR) have presented promising results in healthcare education, while studies suggest that Artificial Intelligence (AI) can help in learning by further engaging the learner. This study aims to educate users on the entire routine of managing diabetes. The serious game utilizes VR for realistic interaction with diabetes tools and generative AI through a conversational avatar that acts as an assistant instructor. In this way, it allows users to practice diagnostic and therapeutic interventions in a controlled virtual environment, helping to build their understanding and confidence in diabetes management. To measure the effects of the proposed serious game, presence, and perceived agency were measured. Preliminary results indicate that this setup aids in the engagement and immersion of learners, while the avatar can provide helpful information during gameplay. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
Shen, Y.; Li, B.; Huang, J.; Wang, Z.
GaussianShopVR: Facilitating Immersive 3D Authoring Using Gaussian Splatting in VR Proceedings Article
In: Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW, pp. 1292–1293, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 979-833151484-6 (ISBN).
Abstract | Links | BibTeX | Tags: 3D authoring, 3D modeling, Digital replicas, Gaussian distribution, Gaussian Splatting editing, Gaussians, Graphical user interfaces, High quality, Immersive, Immersive environment, Interactive computer graphics, Rendering (computer graphics), Rendering pipelines, Splatting, Three dimensional computer graphics, User profile, Virtual Reality, Virtual reality user interface, Virtualization, VR user interface
@inproceedings{shen_gaussianshopvr_2025,
title = {GaussianShopVR: Facilitating Immersive 3D Authoring Using Gaussian Splatting in VR},
author = {Y. Shen and B. Li and J. Huang and Z. Wang},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005138672&doi=10.1109%2fVRW66409.2025.00292&partnerID=40&md5=9b644bd19394a289d3027ab9a2dfed6a},
doi = {10.1109/VRW66409.2025.00292},
isbn = {979-833151484-6 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW},
pages = {1292–1293},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Virtual reality (VR) applications require massive high-quality 3D assets to create immersive environments. Generating mesh-based 3D assets typically involves a significant amount of manpower and effort, which makes VR applications less accessible. 3D Gaussian Splatting (3DGS) has attracted much attention for its ability to quickly create digital replicas of real-life scenes and its compatibility with traditional rendering pipelines. However, it remains a challenge to edit 3DGS in a flexible and controllable manner. We propose GaussianShopVR, a system that leverages VR user interfaces to specify target areas to achieve flexible and controllable editing of reconstructed 3DGS. In addition, selected areas can provide 3D information to generative AI models to facilitate the editing. GaussianShopVR integrates object hierarchy management while keeping the backpropagated gradient flow to allow local editing with context information. © 2025 IEEE.},
keywords = {3D authoring, 3D modeling, Digital replicas, Gaussian distribution, Gaussian Splatting editing, Gaussians, Graphical user interfaces, High quality, Immersive, Immersive environment, Interactive computer graphics, Rendering (computer graphics), Rendering pipelines, Splatting, Three dimensional computer graphics, User profile, Virtual Reality, Virtual reality user interface, Virtualization, VR user interface},
pubstate = {published},
tppubtype = {inproceedings}
}
Virtual reality (VR) applications require massive high-quality 3D assets to create immersive environments. Generating mesh-based 3D assets typically involves a significant amount of manpower and effort, which makes VR applications less accessible. 3D Gaussian Splatting (3DGS) has attracted much attention for its ability to quickly create digital replicas of real-life scenes and its compatibility with traditional rendering pipelines. However, it remains a challenge to edit 3DGS in a flexible and controllable manner. We propose GaussianShopVR, a system that leverages VR user interfaces to specify target areas to achieve flexible and controllable editing of reconstructed 3DGS. In addition, selected areas can provide 3D information to generative AI models to facilitate the editing. GaussianShopVR integrates object hierarchy management while keeping the backpropagated gradient flow to allow local editing with context information. © 2025 IEEE.
Tsai, Y. -J.; Liu, S. -T.; Hsu, S. -C.
The Development of an Interactive IoT Cross-Media Survey System and Real-Time Re-presentation of Mass Learning Proceedings Article
In: J., Wei; G., Margetis (Ed.): Lect. Notes Comput. Sci., pp. 145–157, Springer Science and Business Media Deutschland GmbH, 2025, ISBN: 03029743 (ISSN); 978-303193060-7 (ISBN).
Abstract | Links | BibTeX | Tags: Cross-media, Data Re-presentation, Internet of Things, IoT Cross-Media System, IoT cross-medium system, Learning outcome, Learning systems, Mass Learning, Media systems, Smart phones, Smartphone, Smartphones, STEM with A, Survey System, Survey systems, Surveying, Tangible User Interface, Tangible user interfaces, User interfaces, Virtual Reality
@inproceedings{tsai_development_2025,
title = {The Development of an Interactive IoT Cross-Media Survey System and Real-Time Re-presentation of Mass Learning},
author = {Y. -J. Tsai and S. -T. Liu and S. -C. Hsu},
editor = {Wei J. and Margetis G.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105008756188&doi=10.1007%2f978-3-031-93061-4_10&partnerID=40&md5=c487828eeacfdf18cf4e726e6ce28146},
doi = {10.1007/978-3-031-93061-4_10},
isbn = {03029743 (ISSN); 978-303193060-7 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Lect. Notes Comput. Sci.},
volume = {15823 LNCS},
pages = {145–157},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {In this study, we propose the Interactive IoT Cross-Media Survey System, integrating tangible interaction in a game-like manner with real-time data re-presentation. This system was implemented in the “STEM with A” Interactive Exploration Hall at National Tsing Hua University in 2020. It enabled participants to use their smartphones as tangible user interfaces to “scoop-up questions” from interactive sensing points within the exhibition areas. After completing the questions, participants could “pour-in” their responses and observe digital data re-presentation artworks generated from survey results, showcasing mass learning outcomes. Furthermore, the data re-presentation content was tailored to participants’ group characteristics, showing how their responses impact the group’s overall learning outcomes with each “pour-in response.” The study achieved several key outcomes: (1) transforming traditional surveys into a gamified survey system, enhancing participants’ engagement, (2) providing real-time, group-based data re-presentations, enabling participants to contribute to the group’s learning outcomes, and (3) implementing a grouping mechanism to foster collaboration within groups and healthy competition between them. This system provides flexible and customizable data re-presentation, making it suitable for diverse environments requiring real-time data-driven engagement. Future applications can integrate emerging technologies, such as generative AI to dynamically generate questions or virtual reality to offer immersive experiences. Additionally, data re-presentations can be designed as dynamic mass artistic creations, allowing participants to become co-creators of an evolving collective masterpiece. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.},
keywords = {Cross-media, Data Re-presentation, Internet of Things, IoT Cross-Media System, IoT cross-medium system, Learning outcome, Learning systems, Mass Learning, Media systems, Smart phones, Smartphone, Smartphones, STEM with A, Survey System, Survey systems, Surveying, Tangible User Interface, Tangible user interfaces, User interfaces, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
In this study, we propose the Interactive IoT Cross-Media Survey System, integrating tangible interaction in a game-like manner with real-time data re-presentation. This system was implemented in the “STEM with A” Interactive Exploration Hall at National Tsing Hua University in 2020. It enabled participants to use their smartphones as tangible user interfaces to “scoop-up questions” from interactive sensing points within the exhibition areas. After completing the questions, participants could “pour-in” their responses and observe digital data re-presentation artworks generated from survey results, showcasing mass learning outcomes. Furthermore, the data re-presentation content was tailored to participants’ group characteristics, showing how their responses impact the group’s overall learning outcomes with each “pour-in response.” The study achieved several key outcomes: (1) transforming traditional surveys into a gamified survey system, enhancing participants’ engagement, (2) providing real-time, group-based data re-presentations, enabling participants to contribute to the group’s learning outcomes, and (3) implementing a grouping mechanism to foster collaboration within groups and healthy competition between them. This system provides flexible and customizable data re-presentation, making it suitable for diverse environments requiring real-time data-driven engagement. Future applications can integrate emerging technologies, such as generative AI to dynamically generate questions or virtual reality to offer immersive experiences. Additionally, data re-presentations can be designed as dynamic mass artistic creations, allowing participants to become co-creators of an evolving collective masterpiece. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
Wang, W. -S.; Lin, C. -J.; Lee, H. -Y.; Huang, Y. -M.; Wu, T. -T.
Enhancing self-regulated learning and higher-order thinking skills in virtual reality: the impact of ChatGPT-integrated feedback aids Journal Article
In: Education and Information Technologies, 2025, ISSN: 13602357 (ISSN).
Abstract | Links | BibTeX | Tags: ChatGPT, Cognitive levels, feedback mechanisms, Higher-order thinking skills, Self-regulated learning, Virtual Reality
@article{wang_enhancing_2025,
title = {Enhancing self-regulated learning and higher-order thinking skills in virtual reality: the impact of ChatGPT-integrated feedback aids},
author = {W. -S. Wang and C. -J. Lin and H. -Y. Lee and Y. -M. Huang and T. -T. Wu},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105002253324&doi=10.1007%2fs10639-025-13557-x&partnerID=40&md5=d737f7160ecfb478201b4f1f960031e4},
doi = {10.1007/s10639-025-13557-x},
issn = {13602357 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Education and Information Technologies},
abstract = {This study explores the integration of ChatGPT technology into Virtual Reality (VR) learning environments to support self-regulated learning (SRL) strategies and foster higher-order thinking skills (HOTS). Utilizing a randomized controlled trial (RCT) design, 81 undergraduate students were randomly assigned to either an experimental group with GPT-feedback-supported VR learning or a control group with traditional feedback-supported VR learning. The VR learning environment focused on embedded IoT tasks, structured into six learning units aligned with Bloom’s taxonomy. Results indicate that the experimental group demonstrated significantly higher cognitive levels across memory, understanding, and application tasks compared to the control group. Notably, GPT feedback markedly improved problem-solving and critical thinking skills, though its impact on creativity was limited. This limitation may be attributed to the directive nature of GPT feedback or the structured scope of learning tasks, which may not have encouraged open-ended creative exploration. SRL abilities were also significantly improved in the experimental group, highlighting the efficacy of GPT feedback in promoting self-reflection, goal setting, and strategy adjustment. The findings underscore the potential of integrating generative AI technologies like ChatGPT into VR learning environments to provide targeted, real-time feedback, thus addressing limitations of traditional VR feedback mechanisms. This integration supports deeper engagement and more effective learning outcomes, particularly in problem-solving and critical thinking, providing valuable insights for future educational technology implementations. © The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2025.},
keywords = {ChatGPT, Cognitive levels, feedback mechanisms, Higher-order thinking skills, Self-regulated learning, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
This study explores the integration of ChatGPT technology into Virtual Reality (VR) learning environments to support self-regulated learning (SRL) strategies and foster higher-order thinking skills (HOTS). Utilizing a randomized controlled trial (RCT) design, 81 undergraduate students were randomly assigned to either an experimental group with GPT-feedback-supported VR learning or a control group with traditional feedback-supported VR learning. The VR learning environment focused on embedded IoT tasks, structured into six learning units aligned with Bloom’s taxonomy. Results indicate that the experimental group demonstrated significantly higher cognitive levels across memory, understanding, and application tasks compared to the control group. Notably, GPT feedback markedly improved problem-solving and critical thinking skills, though its impact on creativity was limited. This limitation may be attributed to the directive nature of GPT feedback or the structured scope of learning tasks, which may not have encouraged open-ended creative exploration. SRL abilities were also significantly improved in the experimental group, highlighting the efficacy of GPT feedback in promoting self-reflection, goal setting, and strategy adjustment. The findings underscore the potential of integrating generative AI technologies like ChatGPT into VR learning environments to provide targeted, real-time feedback, thus addressing limitations of traditional VR feedback mechanisms. This integration supports deeper engagement and more effective learning outcomes, particularly in problem-solving and critical thinking, providing valuable insights for future educational technology implementations. © The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2025.
Li, Z.; Zhang, H.; Peng, C.; Peiris, R.
Exploring Large Language Model-Driven Agents for Environment-Aware Spatial Interactions and Conversations in Virtual Reality Role-Play Scenarios Proceedings Article
In: Proc. - IEEE Conf. Virtual Real. 3D User Interfaces, VR, pp. 1–11, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 979-833153645-9 (ISBN).
Abstract | Links | BibTeX | Tags: Chatbots, Computer simulation languages, Context- awareness, context-awareness, Digital elevation model, Generative AI, Human-AI Interaction, Language Model, Large language model, large language models, Model agents, Role-play simulation, role-play simulations, Role-plays, Spatial interaction, Virtual environments, Virtual Reality, Virtual-reality environment
@inproceedings{li_exploring_2025,
title = {Exploring Large Language Model-Driven Agents for Environment-Aware Spatial Interactions and Conversations in Virtual Reality Role-Play Scenarios},
author = {Z. Li and H. Zhang and C. Peng and R. Peiris},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105002706893&doi=10.1109%2fVR59515.2025.00025&partnerID=40&md5=60f22109e054c9035a0c2210bb797039},
doi = {10.1109/VR59515.2025.00025},
isbn = {979-833153645-9 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. - IEEE Conf. Virtual Real. 3D User Interfaces, VR},
pages = {1–11},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Recent research has begun adopting Large Language Model (LLM) agents to enhance Virtual Reality (VR) interactions, creating immersive chatbot experiences. However, while current studies focus on generating dialogue from user speech inputs, their abilities to generate richer experiences based on the perception of LLM agents' VR environments and interaction cues remain unexplored. Hence, in this work, we propose an approach that enables LLM agents to perceive virtual environments and generate environment-aware interactions and conversations for an embodied human-AI interaction experience in VR environments. Here, we define a schema for describing VR environments and their interactions through text prompts. We evaluate the performance of our method through five role-play scenarios created using our approach in a study with 14 participants. The findings discuss the opportunities and challenges of our proposed approach for developing environment-aware LLM agents that facilitate spatial interactions and conversations within VR role-play scenarios. © 2025 IEEE.},
keywords = {Chatbots, Computer simulation languages, Context- awareness, context-awareness, Digital elevation model, Generative AI, Human-AI Interaction, Language Model, Large language model, large language models, Model agents, Role-play simulation, role-play simulations, Role-plays, Spatial interaction, Virtual environments, Virtual Reality, Virtual-reality environment},
pubstate = {published},
tppubtype = {inproceedings}
}
Recent research has begun adopting Large Language Model (LLM) agents to enhance Virtual Reality (VR) interactions, creating immersive chatbot experiences. However, while current studies focus on generating dialogue from user speech inputs, their abilities to generate richer experiences based on the perception of LLM agents' VR environments and interaction cues remain unexplored. Hence, in this work, we propose an approach that enables LLM agents to perceive virtual environments and generate environment-aware interactions and conversations for an embodied human-AI interaction experience in VR environments. Here, we define a schema for describing VR environments and their interactions through text prompts. We evaluate the performance of our method through five role-play scenarios created using our approach in a study with 14 participants. The findings discuss the opportunities and challenges of our proposed approach for developing environment-aware LLM agents that facilitate spatial interactions and conversations within VR role-play scenarios. © 2025 IEEE.
Ly, D. -N.; Do, H. -N.; Tran, M. -T.; Le, K. -D.
Evaluation of AI-Based Assistant Representations on User Interaction in Virtual Explorations Proceedings Article
In: W., Buntine; M., Fjeld; T., Tran; M.-T., Tran; B., Huynh Thi Thanh; T., Miyoshi (Ed.): Commun. Comput. Info. Sci., pp. 323–337, Springer Science and Business Media Deutschland GmbH, 2025, ISBN: 18650929 (ISSN); 978-981964287-8 (ISBN).
Abstract | Links | BibTeX | Tags: 360-degree Video, AI-Based Assistant, Cultural heritages, Cultural science, Multiusers, Single users, Social interactions, Three dimensional computer graphics, User interaction, Users' experiences, Virtual environments, Virtual Exploration, Virtual Reality, Virtualization
@inproceedings{ly_evaluation_2025,
title = {Evaluation of AI-Based Assistant Representations on User Interaction in Virtual Explorations},
author = {D. -N. Ly and H. -N. Do and M. -T. Tran and K. -D. Le},
editor = {Buntine W. and Fjeld M. and Tran T. and Tran M.-T. and Huynh Thi Thanh B. and Miyoshi T.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105004253350&doi=10.1007%2f978-981-96-4288-5_26&partnerID=40&md5=5f0a8c1e356cd3bdd4dda7f96f272154},
doi = {10.1007/978-981-96-4288-5_26},
isbn = {18650929 (ISSN); 978-981964287-8 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Commun. Comput. Info. Sci.},
volume = {2352 CCIS},
pages = {323–337},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {Exploration activities, such as tourism, cultural heritage, and science, enhance knowledge and understanding. The rise of 360-degree videos allows users to explore cultural landmarks and destinations remotely. While multi-user VR environments encourage collaboration, single-user experiences often lack social interaction. Generative AI, particularly Large Language Models (LLMs), offer a way to improve single-user VR exploration through AI-driven virtual assistants, acting as tour guides or storytellers. However, it’s uncertain whether these assistants require a visual presence, and if so, what form it should take. To investigate this, we developed an AI-based assistant in three different forms: a voice-only avatar, a 3D human-sized avatar, and a mini-hologram avatar, and conducted a user study to evaluate their impact on user experience. The study, which involved 12 participants, found that the visual embodiments significantly reduce feelings of being alone, with distinct user preferences between the Human-sized avatar and the Mini hologram. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.},
keywords = {360-degree Video, AI-Based Assistant, Cultural heritages, Cultural science, Multiusers, Single users, Social interactions, Three dimensional computer graphics, User interaction, Users' experiences, Virtual environments, Virtual Exploration, Virtual Reality, Virtualization},
pubstate = {published},
tppubtype = {inproceedings}
}
Exploration activities, such as tourism, cultural heritage, and science, enhance knowledge and understanding. The rise of 360-degree videos allows users to explore cultural landmarks and destinations remotely. While multi-user VR environments encourage collaboration, single-user experiences often lack social interaction. Generative AI, particularly Large Language Models (LLMs), offer a way to improve single-user VR exploration through AI-driven virtual assistants, acting as tour guides or storytellers. However, it’s uncertain whether these assistants require a visual presence, and if so, what form it should take. To investigate this, we developed an AI-based assistant in three different forms: a voice-only avatar, a 3D human-sized avatar, and a mini-hologram avatar, and conducted a user study to evaluate their impact on user experience. The study, which involved 12 participants, found that the visual embodiments significantly reduce feelings of being alone, with distinct user preferences between the Human-sized avatar and the Mini hologram. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.
Huang, W.; Gao, J.; Chen, X.
Predicting post-VR game experiences with wearable physiological sensors Journal Article
In: Entertainment Computing, vol. 55, 2025, ISSN: 18759521 (ISSN).
Abstract | Links | BibTeX | Tags: Brain, Correlation analysis, Depersonalization, Derealization, Electrodermal activity, Game experience, Game Experience Questionnaire, Machine learning techniques, Physiological sensors, Post-game experience, Sensory perception, Virtual Reality, Virtualization
@article{huang_predicting_2025,
title = {Predicting post-VR game experiences with wearable physiological sensors},
author = {W. Huang and J. Gao and X. Chen},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105007731332&doi=10.1016%2fj.entcom.2025.100977&partnerID=40&md5=b292fac3be095cf50627a139f0f27dc0},
doi = {10.1016/j.entcom.2025.100977},
issn = {18759521 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Entertainment Computing},
volume = {55},
abstract = {Players’ post-game experiences determine their loyalty to a virtual reality (VR) game. However, methods for identifying players’ post-game experiences in the early stages have received far less attention than those for in-game experiences. In this study, we explored the potential of using measurements from wearable physiological sensors to predict players’ post–VR game experiences. The methods employed were correlation analyses and machine learning techniques. The results showed that electrodermal activity (EDA) measurements, particularly the mean EDA and mean EDA peak, are associated with players’ post-VR game experiences after accounting for noise. By utilizing machine learning technology, physiological metrics can forecast players’ diverse reactions after playing VR games with high accuracy. The symptoms of depersonalization/derealization experienced after VR gaming are attributed to being induced by actions within the virtual environment. This research makes significant contributions to the field of user experience recognition and the progression of VR gaming by demonstrating the potential for future VR game centers to analyze player emotions remotely and cost-effectively. This achievement provides the prerequisite for these centers to create tailored new 3D game scenarios to enhance players’ post-game experiences with the support of future advanced generative artificial intelligence technologies. © 2025 Elsevier B.V.},
keywords = {Brain, Correlation analysis, Depersonalization, Derealization, Electrodermal activity, Game experience, Game Experience Questionnaire, Machine learning techniques, Physiological sensors, Post-game experience, Sensory perception, Virtual Reality, Virtualization},
pubstate = {published},
tppubtype = {article}
}
Players’ post-game experiences determine their loyalty to a virtual reality (VR) game. However, methods for identifying players’ post-game experiences in the early stages have received far less attention than those for in-game experiences. In this study, we explored the potential of using measurements from wearable physiological sensors to predict players’ post–VR game experiences. The methods employed were correlation analyses and machine learning techniques. The results showed that electrodermal activity (EDA) measurements, particularly the mean EDA and mean EDA peak, are associated with players’ post-VR game experiences after accounting for noise. By utilizing machine learning technology, physiological metrics can forecast players’ diverse reactions after playing VR games with high accuracy. The symptoms of depersonalization/derealization experienced after VR gaming are attributed to being induced by actions within the virtual environment. This research makes significant contributions to the field of user experience recognition and the progression of VR gaming by demonstrating the potential for future VR game centers to analyze player emotions remotely and cost-effectively. This achievement provides the prerequisite for these centers to create tailored new 3D game scenarios to enhance players’ post-game experiences with the support of future advanced generative artificial intelligence technologies. © 2025 Elsevier B.V.
Rasch, J.; Töws, J.; Hirzle, T.; Müller, F.; Schmitz, M.
CreepyCoCreator? Investigating AI Representation Modes for 3D Object Co-Creation in Virtual Reality Proceedings Article
In: Conf Hum Fact Comput Syst Proc, Association for Computing Machinery, 2025, ISBN: 979-840071394-1 (ISBN).
Abstract | Links | BibTeX | Tags: 3D Creation, 3D modeling, 3D object, Building process, Co-creation, Co-creative system, Co-creative systems, Creative systems, Creatives, Generative AI, Three dimensional computer graphics, User expectations, User Studies, User study, Virtual Reality, Virtualization
@inproceedings{rasch_creepycocreator_2025,
title = {CreepyCoCreator? Investigating AI Representation Modes for 3D Object Co-Creation in Virtual Reality},
author = {J. Rasch and J. Töws and T. Hirzle and F. Müller and M. Schmitz},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005742763&doi=10.1145%2f3706598.3713720&partnerID=40&md5=e6cdcb6cc7249a8836ecc39ae103cd53},
doi = {10.1145/3706598.3713720},
isbn = {979-840071394-1 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Conf Hum Fact Comput Syst Proc},
publisher = {Association for Computing Machinery},
abstract = {Generative AI in Virtual Reality offers the potential for collaborative object-building, yet challenges remain in aligning AI contributions with user expectations. In particular, users often struggle to understand and collaborate with AI when its actions are not transparently represented. This paper thus explores the co-creative object-building process through a Wizard-of-Oz study, focusing on how AI can effectively convey its intent to users during object customization in Virtual Reality. Inspired by human-to-human collaboration, we focus on three representation modes: the presence of an embodied avatar, whether the AI's contributions are visualized immediately or incrementally, and whether the areas modified are highlighted in advance. The findings provide insights into how these factors affect user perception and interaction with object-generating AI tools in Virtual Reality as well as satisfaction and ownership of the created objects. The results offer design implications for co-creative world-building systems, aiming to foster more effective and satisfying collaborations between humans and AI in Virtual Reality. © 2025 Copyright held by the owner/author(s).},
keywords = {3D Creation, 3D modeling, 3D object, Building process, Co-creation, Co-creative system, Co-creative systems, Creative systems, Creatives, Generative AI, Three dimensional computer graphics, User expectations, User Studies, User study, Virtual Reality, Virtualization},
pubstate = {published},
tppubtype = {inproceedings}
}
Generative AI in Virtual Reality offers the potential for collaborative object-building, yet challenges remain in aligning AI contributions with user expectations. In particular, users often struggle to understand and collaborate with AI when its actions are not transparently represented. This paper thus explores the co-creative object-building process through a Wizard-of-Oz study, focusing on how AI can effectively convey its intent to users during object customization in Virtual Reality. Inspired by human-to-human collaboration, we focus on three representation modes: the presence of an embodied avatar, whether the AI's contributions are visualized immediately or incrementally, and whether the areas modified are highlighted in advance. The findings provide insights into how these factors affect user perception and interaction with object-generating AI tools in Virtual Reality as well as satisfaction and ownership of the created objects. The results offer design implications for co-creative world-building systems, aiming to foster more effective and satisfying collaborations between humans and AI in Virtual Reality. © 2025 Copyright held by the owner/author(s).
Cao, X.; Ju, K. P.; Li, C.; Jain, D.
SceneGenA11y: How can Runtime Generative tools improve the Accessibility of a Virtual 3D Scene? Proceedings Article
In: Conf Hum Fact Comput Syst Proc, Association for Computing Machinery, 2025, ISBN: 979-840071395-8 (ISBN).
Abstract | Links | BibTeX | Tags: 3D application, 3D modeling, 3D scenes, Accessibility, BLV, DHH, Discrete event simulation, Generative AI, Generative tools, Interactive computer graphics, One dimensional, Runtimes, Three dimensional computer graphics, Video-games, Virtual 3d scene, virtual 3D scenes, Virtual environments, Virtual Reality
@inproceedings{cao_scenegena11y_2025,
title = {SceneGenA11y: How can Runtime Generative tools improve the Accessibility of a Virtual 3D Scene?},
author = {X. Cao and K. P. Ju and C. Li and D. Jain},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005772656&doi=10.1145%2f3706599.3720265&partnerID=40&md5=9b0bf29c3e89b70efa2d6a3e740829fb},
doi = {10.1145/3706599.3720265},
isbn = {979-840071395-8 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Conf Hum Fact Comput Syst Proc},
publisher = {Association for Computing Machinery},
abstract = {With the popularity of virtual 3D applications, from video games to educational content and virtual reality scenarios, the accessibility of 3D scene information is vital to ensure inclusive and equitable experiences for all. Previous work include information substitutions like audio description and captions, as well as personalized modifications, but they could only provide predefined accommodations. In this work, we propose SceneGenA11y, a system that responds to the user’s natural language prompts to improve accessibility of a 3D virtual scene in runtime. The system primes LLM agents with accessibility-related knowledge, allowing users to explore the scene and perform verifiable modifications to improve accessibility. We conducted a preliminary evaluation of our system with three blind and low-vision people and three deaf and hard-of-hearing people. The results show that our system is intuitive to use and can successfully improve accessibility. We discussed usage patterns of the system, potential improvements, and integration into apps. We ended with highlighting plans for future work. © 2025 Copyright held by the owner/author(s).},
keywords = {3D application, 3D modeling, 3D scenes, Accessibility, BLV, DHH, Discrete event simulation, Generative AI, Generative tools, Interactive computer graphics, One dimensional, Runtimes, Three dimensional computer graphics, Video-games, Virtual 3d scene, virtual 3D scenes, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
With the popularity of virtual 3D applications, from video games to educational content and virtual reality scenarios, the accessibility of 3D scene information is vital to ensure inclusive and equitable experiences for all. Previous work include information substitutions like audio description and captions, as well as personalized modifications, but they could only provide predefined accommodations. In this work, we propose SceneGenA11y, a system that responds to the user’s natural language prompts to improve accessibility of a 3D virtual scene in runtime. The system primes LLM agents with accessibility-related knowledge, allowing users to explore the scene and perform verifiable modifications to improve accessibility. We conducted a preliminary evaluation of our system with three blind and low-vision people and three deaf and hard-of-hearing people. The results show that our system is intuitive to use and can successfully improve accessibility. We discussed usage patterns of the system, potential improvements, and integration into apps. We ended with highlighting plans for future work. © 2025 Copyright held by the owner/author(s).
Peter, K.; Makosa, I.; Auala, S.; Ndjao, L.; Maasz, D.; Mbinge, U.; Winschiers-Theophilus, H.
Co-creating a VR Narrative Experience of Constructing a Food Storage Following OvaHimba Traditional Practices Proceedings Article
In: IMX - Proc. ACM Int. Conf. Interact. Media Experiences, pp. 418–423, Association for Computing Machinery, Inc, 2025, ISBN: 979-840071391-0 (ISBN).
Abstract | Links | BibTeX | Tags: 3D Modelling, 3D models, 3d-modeling, Co-designs, Community-based, Community-Based Co-Design, Computer aided design, Cultural heritage, Cultural heritages, Food storage, Human computer interaction, Human engineering, Indigenous Knowledge, Information Systems, Interactive computer graphics, Interactive computer systems, IVR, Namibia, OvaHimba, Ovahimbum, Photogrammetry, Sustainable development, Virtual environments, Virtual Reality
@inproceedings{peter_co-creating_2025,
title = {Co-creating a VR Narrative Experience of Constructing a Food Storage Following OvaHimba Traditional Practices},
author = {K. Peter and I. Makosa and S. Auala and L. Ndjao and D. Maasz and U. Mbinge and H. Winschiers-Theophilus},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105007984089&doi=10.1145%2f3706370.3731652&partnerID=40&md5=36f95823413852d636b39bd561c97917},
doi = {10.1145/3706370.3731652},
isbn = {979-840071391-0 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {IMX - Proc. ACM Int. Conf. Interact. Media Experiences},
pages = {418–423},
publisher = {Association for Computing Machinery, Inc},
abstract = {As part of an attempt to co-create a comprehensive virtual environment in which one can explore and learn traditional practices of the OvaHimba people, we have co-designed and implemented a VR experience to construct a traditional food storage. In collaboration with the OvaHimba community residing in Otjisa, we have explored culturally valid representations of the process. We have further investigated different techniques such as photogrammetry, generative AI and manual methods to develop 3D models. Our findings highlight the importance of context, process, and community-defined relevance in co-design, the fluidity of cultural realities and virtual representations, as well as technical challenges. © 2025 Copyright held by the owner/author(s).},
keywords = {3D Modelling, 3D models, 3d-modeling, Co-designs, Community-based, Community-Based Co-Design, Computer aided design, Cultural heritage, Cultural heritages, Food storage, Human computer interaction, Human engineering, Indigenous Knowledge, Information Systems, Interactive computer graphics, Interactive computer systems, IVR, Namibia, OvaHimba, Ovahimbum, Photogrammetry, Sustainable development, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
As part of an attempt to co-create a comprehensive virtual environment in which one can explore and learn traditional practices of the OvaHimba people, we have co-designed and implemented a VR experience to construct a traditional food storage. In collaboration with the OvaHimba community residing in Otjisa, we have explored culturally valid representations of the process. We have further investigated different techniques such as photogrammetry, generative AI and manual methods to develop 3D models. Our findings highlight the importance of context, process, and community-defined relevance in co-design, the fluidity of cultural realities and virtual representations, as well as technical challenges. © 2025 Copyright held by the owner/author(s).
Pielage, L.; Schmidle, P.; Marschall, B.; Risse, B.
Interactive High-Quality Skin Lesion Generation using Diffusion Models for VR-based Dermatological Education Proceedings Article
In: Int Conf Intell User Interfaces Proc IUI, pp. 878–897, Association for Computing Machinery, 2025, ISBN: 979-840071306-4 (ISBN).
Abstract | Links | BibTeX | Tags: Deep learning, Dermatology, Diffusion Model, diffusion models, Digital elevation model, Generative AI, Graphical user interfaces, Guidance Strategies, Guidance strategy, Image generation, Image generations, Inpainting, Interactive Generation, Medical education, Medical Imaging, Simulation training, Skin lesion, Upsampling, Virtual environments, Virtual Reality
@inproceedings{pielage_interactive_2025,
title = {Interactive High-Quality Skin Lesion Generation using Diffusion Models for VR-based Dermatological Education},
author = {L. Pielage and P. Schmidle and B. Marschall and B. Risse},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105001923208&doi=10.1145%2f3708359.3712101&partnerID=40&md5=639eec55b08a54ce813f7c1016c621e7},
doi = {10.1145/3708359.3712101},
isbn = {979-840071306-4 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Int Conf Intell User Interfaces Proc IUI},
pages = {878–897},
publisher = {Association for Computing Machinery},
abstract = {Malignant melanoma is one of the most lethal forms of cancer when not detected early. As a result, cancer screening programs have been implemented internationally, all of which require visual inspection of skin lesions. Early melanoma detection is a crucial competence in medical and dermatological education, and it is primarily trained using 2D imagery. However, given the intrinsic 3D nature of skin lesions and the importance of incorporating additional contextual information about the patient (e.g., skin type, nearby lesions, etc.), this approach falls short of providing a comprehensive and scalable learning experience. A potential solution is the use of Virtual Reality (VR) scenarios, which can offer an effective strategy to train skin cancer screenings in a realistic 3D setting, thereby enhancing medical students' awareness of early melanoma detection. In this paper, we present a comprehensive pipeline and models for generating malignant melanomas and benign nevi, which can be utilized in VR-based medical training. We use diffusion models for the generation of skin lesions, which we have enhanced with various guiding strategies to give educators maximum flexibility in designing scenarios and seamlessly placing lesions on virtual agents. Additionally, we have developed a tool which comprises a graphical user interface (GUI) enabling the generation of new lesions and adapting existing ones using an intuitive and interactive inpainting strategy. The tool also offers a novel custom upsampling strategy to achieve a sufficient resolution required for diagnostic purposes. The generated skin lesions have been validated in a user study with trained dermatologists, confirming the overall high quality of the generated lesions and the utility for educational purposes. © 2025 Copyright held by the owner/author(s).},
keywords = {Deep learning, Dermatology, Diffusion Model, diffusion models, Digital elevation model, Generative AI, Graphical user interfaces, Guidance Strategies, Guidance strategy, Image generation, Image generations, Inpainting, Interactive Generation, Medical education, Medical Imaging, Simulation training, Skin lesion, Upsampling, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
Malignant melanoma is one of the most lethal forms of cancer when not detected early. As a result, cancer screening programs have been implemented internationally, all of which require visual inspection of skin lesions. Early melanoma detection is a crucial competence in medical and dermatological education, and it is primarily trained using 2D imagery. However, given the intrinsic 3D nature of skin lesions and the importance of incorporating additional contextual information about the patient (e.g., skin type, nearby lesions, etc.), this approach falls short of providing a comprehensive and scalable learning experience. A potential solution is the use of Virtual Reality (VR) scenarios, which can offer an effective strategy to train skin cancer screenings in a realistic 3D setting, thereby enhancing medical students' awareness of early melanoma detection. In this paper, we present a comprehensive pipeline and models for generating malignant melanomas and benign nevi, which can be utilized in VR-based medical training. We use diffusion models for the generation of skin lesions, which we have enhanced with various guiding strategies to give educators maximum flexibility in designing scenarios and seamlessly placing lesions on virtual agents. Additionally, we have developed a tool which comprises a graphical user interface (GUI) enabling the generation of new lesions and adapting existing ones using an intuitive and interactive inpainting strategy. The tool also offers a novel custom upsampling strategy to achieve a sufficient resolution required for diagnostic purposes. The generated skin lesions have been validated in a user study with trained dermatologists, confirming the overall high quality of the generated lesions and the utility for educational purposes. © 2025 Copyright held by the owner/author(s).
Li, H.; Wang, Z.; Liang, W.; Wang, Y.
X’s Day: Personality-Driven Virtual Human Behavior Generation Journal Article
In: IEEE Transactions on Visualization and Computer Graphics, vol. 31, no. 5, pp. 3514–3524, 2025, ISSN: 10772626 (ISSN).
Abstract | Links | BibTeX | Tags: adult, Augmented Reality, Behavior Generation, Chatbots, Computer graphics, computer interface, Contextual Scene, female, human, Human behaviors, Humans, Long-term behavior, male, Novel task, Personality, Personality traits, Personality-driven Behavior, physiology, Social behavior, User-Computer Interface, Users' experiences, Virtual agent, Virtual environments, Virtual humans, Virtual Reality, Young Adult
@article{li_xs_2025,
title = {X’s Day: Personality-Driven Virtual Human Behavior Generation},
author = {H. Li and Z. Wang and W. Liang and Y. Wang},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105003864932&doi=10.1109%2fTVCG.2025.3549574&partnerID=40&md5=a865bbd2b0fa964a4f0f4190955dc787},
doi = {10.1109/TVCG.2025.3549574},
issn = {10772626 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Transactions on Visualization and Computer Graphics},
volume = {31},
number = {5},
pages = {3514–3524},
abstract = {Developing convincing and realistic virtual human behavior is essential for enhancing user experiences in virtual reality (VR) and augmented reality (AR) settings. This paper introduces a novel task focused on generating long-term behaviors for virtual agents, guided by specific personality traits and contextual elements within 3D environments. We present a comprehensive framework capable of autonomously producing daily activities autoregressively. By modeling the intricate connections between personality characteristics and observable activities, we establish a hierarchical structure of Needs, Task, and Activity levels. Integrating a Behavior Planner and a World State module allows for the dynamic sampling of behaviors using large language models (LLMs), ensuring that generated activities remain relevant and responsive to environmental changes. Extensive experiments validate the effectiveness and adaptability of our approach across diverse scenarios. This research makes a significant contribution to the field by establishing a new paradigm for personalized and context-aware interactions with virtual humans, ultimately enhancing user engagement in immersive applications. Our project website is at: https://behavior.agent-x.cn/. © 2025 IEEE. All rights reserved,},
keywords = {adult, Augmented Reality, Behavior Generation, Chatbots, Computer graphics, computer interface, Contextual Scene, female, human, Human behaviors, Humans, Long-term behavior, male, Novel task, Personality, Personality traits, Personality-driven Behavior, physiology, Social behavior, User-Computer Interface, Users' experiences, Virtual agent, Virtual environments, Virtual humans, Virtual Reality, Young Adult},
pubstate = {published},
tppubtype = {article}
}
Developing convincing and realistic virtual human behavior is essential for enhancing user experiences in virtual reality (VR) and augmented reality (AR) settings. This paper introduces a novel task focused on generating long-term behaviors for virtual agents, guided by specific personality traits and contextual elements within 3D environments. We present a comprehensive framework capable of autonomously producing daily activities autoregressively. By modeling the intricate connections between personality characteristics and observable activities, we establish a hierarchical structure of Needs, Task, and Activity levels. Integrating a Behavior Planner and a World State module allows for the dynamic sampling of behaviors using large language models (LLMs), ensuring that generated activities remain relevant and responsive to environmental changes. Extensive experiments validate the effectiveness and adaptability of our approach across diverse scenarios. This research makes a significant contribution to the field by establishing a new paradigm for personalized and context-aware interactions with virtual humans, ultimately enhancing user engagement in immersive applications. Our project website is at: https://behavior.agent-x.cn/. © 2025 IEEE. All rights reserved,
Song, T.; Pabst, F.; Eck, U.; Navab, N.
Enhancing Patient Acceptance of Robotic Ultrasound through Conversational Virtual Agent and Immersive Visualizations Journal Article
In: IEEE Transactions on Visualization and Computer Graphics, vol. 31, no. 5, pp. 2901–2911, 2025, ISSN: 10772626 (ISSN).
Abstract | Links | BibTeX | Tags: 3D reconstruction, adult, Augmented Reality, Computer graphics, computer interface, echography, female, human, Humans, Imaging, Intelligent robots, Intelligent virtual agents, Language Model, male, Medical robotics, Middle Aged, Mixed reality, Patient Acceptance of Health Care, patient attitude, Patient comfort, procedures, Real-world, Reality visualization, Robotic Ultrasound, Robotics, Three-Dimensional, three-dimensional imaging, Trust and Acceptance, Ultrasonic applications, Ultrasonic equipment, Ultrasonography, Ultrasound probes, User-Computer Interface, Virtual agent, Virtual assistants, Virtual environments, Virtual Reality, Visual languages, Visualization, Young Adult
@article{song_enhancing_2025,
title = {Enhancing Patient Acceptance of Robotic Ultrasound through Conversational Virtual Agent and Immersive Visualizations},
author = {T. Song and F. Pabst and U. Eck and N. Navab},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105003687673&doi=10.1109%2fTVCG.2025.3549181&partnerID=40&md5=1d46569933582ecf5e967f0794aafc07},
doi = {10.1109/TVCG.2025.3549181},
issn = {10772626 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Transactions on Visualization and Computer Graphics},
volume = {31},
number = {5},
pages = {2901–2911},
abstract = {Robotic ultrasound systems have the potential to improve medical diagnostics, but patient acceptance remains a key challenge. To address this, we propose a novel system that combines an AI-based virtual agent, powered by a large language model (LLM), with three mixed reality visualizations aimed at enhancing patient comfort and trust. The LLM enables the virtual assistant to engage in natural, conversational dialogue with patients, answering questions in any format and offering real-time reassurance, creating a more intelligent and reliable interaction. The virtual assistant is animated as controlling the ultrasound probe, giving the impression that the robot is guided by the assistant. The first visualization employs augmented reality (AR), allowing patients to see the real world and the robot with the virtual avatar superimposed. The second visualization is an augmented virtuality (AV) environment, where the real-world body part being scanned is visible, while a 3D Gaussian Splatting reconstruction of the room, excluding the robot, forms the virtual environment. The third is a fully immersive virtual reality (VR) experience, featuring the same 3D reconstruction but entirely virtual, where the patient sees a virtual representation of their body being scanned in a robot-free environment. In this case, the virtual ultrasound probe, mirrors the movement of the probe controlled by the robot, creating a synchronized experience as it touches and moves over the patient's virtual body. We conducted a comprehensive agent-guided robotic ultrasound study with all participants, comparing these visualizations against a standard robotic ultrasound procedure. Results showed significant improvements in patient trust, acceptance, and comfort. Based on these findings, we offer insights into designing future mixed reality visualizations and virtual agents to further enhance patient comfort and acceptance in autonomous medical procedures. © 1995-2012 IEEE.},
keywords = {3D reconstruction, adult, Augmented Reality, Computer graphics, computer interface, echography, female, human, Humans, Imaging, Intelligent robots, Intelligent virtual agents, Language Model, male, Medical robotics, Middle Aged, Mixed reality, Patient Acceptance of Health Care, patient attitude, Patient comfort, procedures, Real-world, Reality visualization, Robotic Ultrasound, Robotics, Three-Dimensional, three-dimensional imaging, Trust and Acceptance, Ultrasonic applications, Ultrasonic equipment, Ultrasonography, Ultrasound probes, User-Computer Interface, Virtual agent, Virtual assistants, Virtual environments, Virtual Reality, Visual languages, Visualization, Young Adult},
pubstate = {published},
tppubtype = {article}
}
Robotic ultrasound systems have the potential to improve medical diagnostics, but patient acceptance remains a key challenge. To address this, we propose a novel system that combines an AI-based virtual agent, powered by a large language model (LLM), with three mixed reality visualizations aimed at enhancing patient comfort and trust. The LLM enables the virtual assistant to engage in natural, conversational dialogue with patients, answering questions in any format and offering real-time reassurance, creating a more intelligent and reliable interaction. The virtual assistant is animated as controlling the ultrasound probe, giving the impression that the robot is guided by the assistant. The first visualization employs augmented reality (AR), allowing patients to see the real world and the robot with the virtual avatar superimposed. The second visualization is an augmented virtuality (AV) environment, where the real-world body part being scanned is visible, while a 3D Gaussian Splatting reconstruction of the room, excluding the robot, forms the virtual environment. The third is a fully immersive virtual reality (VR) experience, featuring the same 3D reconstruction but entirely virtual, where the patient sees a virtual representation of their body being scanned in a robot-free environment. In this case, the virtual ultrasound probe, mirrors the movement of the probe controlled by the robot, creating a synchronized experience as it touches and moves over the patient's virtual body. We conducted a comprehensive agent-guided robotic ultrasound study with all participants, comparing these visualizations against a standard robotic ultrasound procedure. Results showed significant improvements in patient trust, acceptance, and comfort. Based on these findings, we offer insights into designing future mixed reality visualizations and virtual agents to further enhance patient comfort and acceptance in autonomous medical procedures. © 1995-2012 IEEE.
Coronado, A.; Carvalho, S. T.; Berretta, L.
See Through My Eyes: Using Multimodal Large Language Model for Describing Rendered Environments to Blind People Proceedings Article
In: IMX - Proc. ACM Int. Conf. Interact. Media Experiences, pp. 451–457, Association for Computing Machinery, Inc, 2025, ISBN: 979-840071391-0 (ISBN).
Abstract | Links | BibTeX | Tags: Accessibility, Behavioral Research, Blind, Blind people, Helmet mounted displays, Human engineering, Human rehabilitation equipment, Interactive computer graphics, Interactive computer systems, Language Model, LLM, Multi-modal, Rendered environment, rendered environments, Spatial cognition, Virtual Reality, Vision aids, Visual impairment, Visual languages, Visually impaired people
@inproceedings{coronado_see_2025,
title = {See Through My Eyes: Using Multimodal Large Language Model for Describing Rendered Environments to Blind People},
author = {A. Coronado and S. T. Carvalho and L. Berretta},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105007991842&doi=10.1145%2f3706370.3731641&partnerID=40&md5=2f7cb1535d39d5e59b1f43f773de3272},
doi = {10.1145/3706370.3731641},
isbn = {979-840071391-0 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {IMX - Proc. ACM Int. Conf. Interact. Media Experiences},
pages = {451–457},
publisher = {Association for Computing Machinery, Inc},
abstract = {Extended Reality (XR) is quickly expanding "as the next major technology wave in personal computing". Nevertheless, this expansion and adoption could also exclude certain disabled users, particularly people with visual impairment (VIP). According to the World Health Organization (WHO) in their 2019 publication, there were at least 2.2 billion people with visual impairment, a number that is also estimated to have increased in recent years. Therefore, it is important to include disabled users, especially visually impaired people, in the design of Head-Mounted Displays and Extended Reality environments. Indeed, this objective can be pursued by incorporating Multimodal Large Language Model (MLLM) technology, which can assist visually impaired people. As a case study, this study employs different prompts that result in environment descriptions from an MLLM integrated into a virtual reality (VR) escape room. Therefore, six potential prompts were engineered to generate valuable outputs for visually impaired users inside a VR environment. These outputs were evaluated using the G-Eval, and VIEScore metrics. Even though, the results show that the prompt patterns provided a description that aligns with the user's point of view, it is highly recommended to evaluate these outputs through "expected outputs"from Orientation and Mobility Specialists, and Sighted Guides. Furthermore, the subsequent step in the process is to evaluate these outputs by visually impaired people themselves to identify the most effective prompt pattern. © 2025 Copyright held by the owner/author(s).},
keywords = {Accessibility, Behavioral Research, Blind, Blind people, Helmet mounted displays, Human engineering, Human rehabilitation equipment, Interactive computer graphics, Interactive computer systems, Language Model, LLM, Multi-modal, Rendered environment, rendered environments, Spatial cognition, Virtual Reality, Vision aids, Visual impairment, Visual languages, Visually impaired people},
pubstate = {published},
tppubtype = {inproceedings}
}
Extended Reality (XR) is quickly expanding "as the next major technology wave in personal computing". Nevertheless, this expansion and adoption could also exclude certain disabled users, particularly people with visual impairment (VIP). According to the World Health Organization (WHO) in their 2019 publication, there were at least 2.2 billion people with visual impairment, a number that is also estimated to have increased in recent years. Therefore, it is important to include disabled users, especially visually impaired people, in the design of Head-Mounted Displays and Extended Reality environments. Indeed, this objective can be pursued by incorporating Multimodal Large Language Model (MLLM) technology, which can assist visually impaired people. As a case study, this study employs different prompts that result in environment descriptions from an MLLM integrated into a virtual reality (VR) escape room. Therefore, six potential prompts were engineered to generate valuable outputs for visually impaired users inside a VR environment. These outputs were evaluated using the G-Eval, and VIEScore metrics. Even though, the results show that the prompt patterns provided a description that aligns with the user's point of view, it is highly recommended to evaluate these outputs through "expected outputs"from Orientation and Mobility Specialists, and Sighted Guides. Furthermore, the subsequent step in the process is to evaluate these outputs by visually impaired people themselves to identify the most effective prompt pattern. © 2025 Copyright held by the owner/author(s).
Tong, Y.; Qiu, Y.; Li, R.; Qiu, S.; Heng, P. -A.
MS2Mesh-XR: Multi-Modal Sketch-to-Mesh Generation in XR Environments Proceedings Article
In: Proc. - IEEE Int. Conf. Artif. Intell. Ext. Virtual Real., AIxVR, pp. 272–276, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 979-833152157-8 (ISBN).
Abstract | Links | BibTeX | Tags: 3D meshes, 3D object, ControlNet, Hand-drawn sketches, Hands movement, High quality, Image-based, immersive visualization, Mesh generation, Multi-modal, Pipeline codes, Realistic images, Three dimensional computer graphics, Virtual environments, Virtual Reality
@inproceedings{tong_ms2mesh-xr_2025,
title = {MS2Mesh-XR: Multi-Modal Sketch-to-Mesh Generation in XR Environments},
author = {Y. Tong and Y. Qiu and R. Li and S. Qiu and P. -A. Heng},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105000423684&doi=10.1109%2fAIxVR63409.2025.00052&partnerID=40&md5=caeace6850dcbdf8c1fa0441b98fa8d9},
doi = {10.1109/AIxVR63409.2025.00052},
isbn = {979-833152157-8 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. - IEEE Int. Conf. Artif. Intell. Ext. Virtual Real., AIxVR},
pages = {272–276},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {We present MS2Mesh-XR, a novel multimodal sketch-to-mesh generation pipeline that enables users to create realistic 3D objects in extended reality (XR) environments using hand-drawn sketches assisted by voice inputs. In specific, users can intuitively sketch objects using natural hand movements in mid-air within a virtual environment. By integrating voice inputs, we devise ControlNet to infer realistic images based on the drawn sketches and interpreted text prompts. Users can then review and select their preferred image, which is subsequently reconstructed into a detailed 3D mesh using the Convolutional Reconstruction Model. In particular, our proposed pipeline can generate a high-quality 3D mesh in less than 20 seconds, allowing for immersive visualization and manipulation in runtime XR scenes. We demonstrate the practicability of our pipeline through two use cases in XR settings. By leveraging natural user inputs and cutting-edge generative AI capabilities, our approach can significantly facilitate XR-based creative production and enhance user experiences. Our code and demo will be available at: https://yueqiu0911.github.io/MS2Mesh-XR/. © 2025 IEEE.},
keywords = {3D meshes, 3D object, ControlNet, Hand-drawn sketches, Hands movement, High quality, Image-based, immersive visualization, Mesh generation, Multi-modal, Pipeline codes, Realistic images, Three dimensional computer graphics, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
We present MS2Mesh-XR, a novel multimodal sketch-to-mesh generation pipeline that enables users to create realistic 3D objects in extended reality (XR) environments using hand-drawn sketches assisted by voice inputs. In specific, users can intuitively sketch objects using natural hand movements in mid-air within a virtual environment. By integrating voice inputs, we devise ControlNet to infer realistic images based on the drawn sketches and interpreted text prompts. Users can then review and select their preferred image, which is subsequently reconstructed into a detailed 3D mesh using the Convolutional Reconstruction Model. In particular, our proposed pipeline can generate a high-quality 3D mesh in less than 20 seconds, allowing for immersive visualization and manipulation in runtime XR scenes. We demonstrate the practicability of our pipeline through two use cases in XR settings. By leveraging natural user inputs and cutting-edge generative AI capabilities, our approach can significantly facilitate XR-based creative production and enhance user experiences. Our code and demo will be available at: https://yueqiu0911.github.io/MS2Mesh-XR/. © 2025 IEEE.
Timmerman, K.; Mertens, R.; Yoncalik, A.; Spriet, L.
Cities Unseen: Experiencing the Imagined Proceedings Article
In: Proc. Int. Conf. Tangible, Embed., Embodied Interact., TEI, Association for Computing Machinery, Inc, 2025, ISBN: 979-840071197-8 (ISBN).
Abstract | Links | BibTeX | Tags: Art installation, Embodiment, Immersive, Immersive Storytelling, Multiplayers, Presence, Real time interactions, Sensory Involvement, Through the lens, Urban environments, Virtual environments, Virtual Reality
@inproceedings{timmerman_cities_2025,
title = {Cities Unseen: Experiencing the Imagined},
author = {K. Timmerman and R. Mertens and A. Yoncalik and L. Spriet},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105000440301&doi=10.1145%2f3689050.3707685&partnerID=40&md5=6aa0968146eab8f91fba6eaeb30f7f9c},
doi = {10.1145/3689050.3707685},
isbn = {979-840071197-8 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. Int. Conf. Tangible, Embed., Embodied Interact., TEI},
publisher = {Association for Computing Machinery, Inc},
abstract = {"Cities Unseen: Experiencing the Imagined" is an art installation that reinterprets Italo Calvino's "Invisible Cities" through the lens of virtual reality. The project employs a physical suitcase as a portal, allowing participants to enter and explore virtual urban environments using the Oculus Quest 3. The cityscapes will be developed with generative AI, converting Calvino's descriptions into prompts, creating an immersive space for philosophical reflection on the nature of travel and the boundaries between reality and imagination. By integrating Unity's shared spatial anchors and advanced multiplayer features, "Cities Unseen" supports real-time interaction among participants, emphasizing the social and collaborative dimensions of virtual travel. © 2025 Copyright held by the owner/author(s).},
keywords = {Art installation, Embodiment, Immersive, Immersive Storytelling, Multiplayers, Presence, Real time interactions, Sensory Involvement, Through the lens, Urban environments, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
"Cities Unseen: Experiencing the Imagined" is an art installation that reinterprets Italo Calvino's "Invisible Cities" through the lens of virtual reality. The project employs a physical suitcase as a portal, allowing participants to enter and explore virtual urban environments using the Oculus Quest 3. The cityscapes will be developed with generative AI, converting Calvino's descriptions into prompts, creating an immersive space for philosophical reflection on the nature of travel and the boundaries between reality and imagination. By integrating Unity's shared spatial anchors and advanced multiplayer features, "Cities Unseen" supports real-time interaction among participants, emphasizing the social and collaborative dimensions of virtual travel. © 2025 Copyright held by the owner/author(s).