AHCI RESEARCH GROUP
Publications
Papers published in international journals,
proceedings of conferences, workshops and books.
OUR RESEARCH
Scientific Publications
How to
Here you can find the complete list of our publications.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
2025
Liu, G.; Du, H.; Wang, J.; Niyato, D.; Kim, D. I.
Contract-Inspired Contest Theory for Controllable Image Generation in Mobile Edge Metaverse Journal Article
In: IEEE Transactions on Mobile Computing, 2025, ISSN: 15361233 (ISSN).
Abstract | Links | BibTeX | Tags: Contest Theory, Deep learning, Deep reinforcement learning, Diffusion Model, Generative adversarial networks, Generative AI, High quality, Image generation, Image generations, Immersive technologies, Metaverses, Mobile edge computing, Reinforcement Learning, Reinforcement learnings, Resource allocation, Resources allocation, Semantic data, Virtual addresses, Virtual environments, Virtual Reality
@article{liu_contract-inspired_2025,
title = {Contract-Inspired Contest Theory for Controllable Image Generation in Mobile Edge Metaverse},
author = {G. Liu and H. Du and J. Wang and D. Niyato and D. I. Kim},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105000066834&doi=10.1109%2fTMC.2025.3550815&partnerID=40&md5=3cb5a2143b9ce4ca7f931a60f1bf239c},
doi = {10.1109/TMC.2025.3550815},
issn = {15361233 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Transactions on Mobile Computing},
abstract = {The rapid advancement of immersive technologies has propelled the development of the Metaverse, where the convergence of virtual and physical realities necessitates the generation of high-quality, photorealistic images to enhance user experience. However, generating these images, especially through Generative Diffusion Models (GDMs), in mobile edge computing environments presents significant challenges due to the limited computing resources of edge devices and the dynamic nature of wireless networks. This paper proposes a novel framework that integrates contract-inspired contest theory, Deep Reinforcement Learning (DRL), and GDMs to optimize image generation in these resource-constrained environments. The framework addresses the critical challenges of resource allocation and semantic data transmission quality by incentivizing edge devices to efficiently transmit high-quality semantic data, which is essential for creating realistic and immersive images. The use of contest and contract theory ensures that edge devices are motivated to allocate resources effectively, while DRL dynamically adjusts to network conditions, optimizing the overall image generation process. Experimental results demonstrate that the proposed approach not only improves the quality of generated images but also achieves superior convergence speed and stability compared to traditional methods. This makes the framework particularly effective for optimizing complex resource allocation tasks in mobile edge Metaverse applications, offering enhanced performance and efficiency in creating immersive virtual environments. © 2002-2012 IEEE.},
keywords = {Contest Theory, Deep learning, Deep reinforcement learning, Diffusion Model, Generative adversarial networks, Generative AI, High quality, Image generation, Image generations, Immersive technologies, Metaverses, Mobile edge computing, Reinforcement Learning, Reinforcement learnings, Resource allocation, Resources allocation, Semantic data, Virtual addresses, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
The rapid advancement of immersive technologies has propelled the development of the Metaverse, where the convergence of virtual and physical realities necessitates the generation of high-quality, photorealistic images to enhance user experience. However, generating these images, especially through Generative Diffusion Models (GDMs), in mobile edge computing environments presents significant challenges due to the limited computing resources of edge devices and the dynamic nature of wireless networks. This paper proposes a novel framework that integrates contract-inspired contest theory, Deep Reinforcement Learning (DRL), and GDMs to optimize image generation in these resource-constrained environments. The framework addresses the critical challenges of resource allocation and semantic data transmission quality by incentivizing edge devices to efficiently transmit high-quality semantic data, which is essential for creating realistic and immersive images. The use of contest and contract theory ensures that edge devices are motivated to allocate resources effectively, while DRL dynamically adjusts to network conditions, optimizing the overall image generation process. Experimental results demonstrate that the proposed approach not only improves the quality of generated images but also achieves superior convergence speed and stability compared to traditional methods. This makes the framework particularly effective for optimizing complex resource allocation tasks in mobile edge Metaverse applications, offering enhanced performance and efficiency in creating immersive virtual environments. © 2002-2012 IEEE.
Lv, J.; Slowik, A.; Rani, S.; Kim, B. -G.; Chen, C. -M.; Kumari, S.; Li, K.; Lyu, X.; Jiang, H.
In: Research, vol. 8, 2025, ISSN: 20965168 (ISSN).
Abstract | Links | BibTeX | Tags: Adaptive fusion, Collaborative representations, Diagnosis, Electronic health record, Generative adversarial networks, Health care application, Healthcare environments, Immersive, Learning frameworks, Metaverses, Multi-modal, Multi-modal learning, Performance
@article{lv_multimodal_2025,
title = {Multimodal Metaverse Healthcare: A Collaborative Representation and Adaptive Fusion Approach for Generative Artificial-Intelligence-Driven Diagnosis},
author = {J. Lv and A. Slowik and S. Rani and B. -G. Kim and C. -M. Chen and S. Kumari and K. Li and X. Lyu and H. Jiang},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-86000613924&doi=10.34133%2fresearch.0616&partnerID=40&md5=fdc8ae3b29db905105dada9a5657b54b},
doi = {10.34133/research.0616},
issn = {20965168 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Research},
volume = {8},
abstract = {The metaverse enables immersive virtual healthcare environments, presenting opportunities for enhanced care delivery. A key challenge lies in effectively combining multimodal healthcare data and generative artificial intelligence abilities within metaverse-based healthcare applications, which is a problem that needs to be addressed. This paper proposes a novel multimodal learning framework for metaverse healthcare, MMLMH, based on collaborative intra- and intersample representation and adaptive fusion. Our framework introduces a collaborative representation learning approach that captures shared and modality-specific features across text, audio, and visual health data. By combining modality-specific and shared encoders with carefully formulated intrasample and intersample collaboration mechanisms, MMLMH achieves superior feature representation for complex health assessments. The framework’s adaptive fusion approach, utilizing attention mechanisms and gated neural networks, demonstrates robust performance across varying noise levels and data quality conditions. Experiments on metaverse healthcare datasets demonstrate MMLMH’s superior performance over baseline methods across multiple evaluation metrics. Longitudinal studies and visualization further illustrate MMLMH’s adaptability to evolving virtual environments and balanced performance across diagnostic accuracy, patient–system interaction efficacy, and data integration complexity. The proposed framework has a unique advantage in that a similar level of performance is maintained across various patient populations and virtual avatars, which could lead to greater personalization of healthcare experiences in the metaverse. MMLMH’s successful functioning in such complicated circumstances suggests that it can combine and process information streams from several sources. They can be successfully utilized in next-generation healthcare delivery through virtual reality. © 2025 Jianhui Lv et al.},
keywords = {Adaptive fusion, Collaborative representations, Diagnosis, Electronic health record, Generative adversarial networks, Health care application, Healthcare environments, Immersive, Learning frameworks, Metaverses, Multi-modal, Multi-modal learning, Performance},
pubstate = {published},
tppubtype = {article}
}
The metaverse enables immersive virtual healthcare environments, presenting opportunities for enhanced care delivery. A key challenge lies in effectively combining multimodal healthcare data and generative artificial intelligence abilities within metaverse-based healthcare applications, which is a problem that needs to be addressed. This paper proposes a novel multimodal learning framework for metaverse healthcare, MMLMH, based on collaborative intra- and intersample representation and adaptive fusion. Our framework introduces a collaborative representation learning approach that captures shared and modality-specific features across text, audio, and visual health data. By combining modality-specific and shared encoders with carefully formulated intrasample and intersample collaboration mechanisms, MMLMH achieves superior feature representation for complex health assessments. The framework’s adaptive fusion approach, utilizing attention mechanisms and gated neural networks, demonstrates robust performance across varying noise levels and data quality conditions. Experiments on metaverse healthcare datasets demonstrate MMLMH’s superior performance over baseline methods across multiple evaluation metrics. Longitudinal studies and visualization further illustrate MMLMH’s adaptability to evolving virtual environments and balanced performance across diagnostic accuracy, patient–system interaction efficacy, and data integration complexity. The proposed framework has a unique advantage in that a similar level of performance is maintained across various patient populations and virtual avatars, which could lead to greater personalization of healthcare experiences in the metaverse. MMLMH’s successful functioning in such complicated circumstances suggests that it can combine and process information streams from several sources. They can be successfully utilized in next-generation healthcare delivery through virtual reality. © 2025 Jianhui Lv et al.
Zhang, Z.; Jiang, Y.; Wei, X.; Chen, M.; Dong, H.; Yu, S.
Generative-AI for XR Content Transmission in the Metaverse: Potential Approaches, Challenges, and a Generation-Driven Transmission Framework Journal Article
In: IEEE Network, 2025, ISSN: 08908044 (ISSN).
Abstract | Links | BibTeX | Tags: 'current, Cloud servers, Collaboration architecture, Content transmission, Decision making, Deep learning, Deep reinforcement learning, Edge server, Generative adversarial networks, Intelligence models, Large volumes, Metaverses, Network bottlenecks, Reinforcement Learning, Through current
@article{zhang_generative-ai_2025,
title = {Generative-AI for XR Content Transmission in the Metaverse: Potential Approaches, Challenges, and a Generation-Driven Transmission Framework},
author = {Z. Zhang and Y. Jiang and X. Wei and M. Chen and H. Dong and S. Yu},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-86000661262&doi=10.1109%2fMNET.2025.3547385&partnerID=40&md5=1e00d40542ec58ef1489934abb2a990c},
doi = {10.1109/MNET.2025.3547385},
issn = {08908044 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Network},
abstract = {How to efficiently transmit large volumes of Extended Reality (XR) content through current networks has been a major bottleneck in realizing the Metaverse. The recently emerging Generative Artificial Intelligence (GAI) has already revolutionized various technological fields and provides promising solutions to this challenge. In this article, we first demonstrate current networks' bottlenecks for supporting XR content transmission in the Metaverse. Then, we explore the potential approaches and challenges of utilizing GAI to overcome these bottlenecks. To address these challenges, we propose a GAI-based XR content transmission framework which leverages a cloud-edge collaboration architecture. The cloud servers are responsible for storing and rendering the original XR content, while edge servers utilize GAI models to generate essential parts of XR content (e.g., subsequent frames, selected objects, etc.) when network resources are insufficient to transmit them. A Deep Reinforcement Learning (DRL)-based decision module is proposed to solve the decision-making problems. Our case study demonstrates that the proposed GAI-based transmission framework achieves a 2.8-fold increase in normal frame ratio (percentage of frames that meet the quality and latency requirements for XR content transmission) over baseline approaches, underscoring the potential of GAI models to facilitate XR content transmission in the Metaverse. © 2025 IEEE.},
keywords = {'current, Cloud servers, Collaboration architecture, Content transmission, Decision making, Deep learning, Deep reinforcement learning, Edge server, Generative adversarial networks, Intelligence models, Large volumes, Metaverses, Network bottlenecks, Reinforcement Learning, Through current},
pubstate = {published},
tppubtype = {article}
}
How to efficiently transmit large volumes of Extended Reality (XR) content through current networks has been a major bottleneck in realizing the Metaverse. The recently emerging Generative Artificial Intelligence (GAI) has already revolutionized various technological fields and provides promising solutions to this challenge. In this article, we first demonstrate current networks' bottlenecks for supporting XR content transmission in the Metaverse. Then, we explore the potential approaches and challenges of utilizing GAI to overcome these bottlenecks. To address these challenges, we propose a GAI-based XR content transmission framework which leverages a cloud-edge collaboration architecture. The cloud servers are responsible for storing and rendering the original XR content, while edge servers utilize GAI models to generate essential parts of XR content (e.g., subsequent frames, selected objects, etc.) when network resources are insufficient to transmit them. A Deep Reinforcement Learning (DRL)-based decision module is proposed to solve the decision-making problems. Our case study demonstrates that the proposed GAI-based transmission framework achieves a 2.8-fold increase in normal frame ratio (percentage of frames that meet the quality and latency requirements for XR content transmission) over baseline approaches, underscoring the potential of GAI models to facilitate XR content transmission in the Metaverse. © 2025 IEEE.
Koizumi, M.; Ohsuga, M.; Corchado, J. M.
Development and Assessment of a System to Help Students Improve Self-compassion Proceedings Article
In: R., Chinthaginjala; P., Sitek; N., Min-Allah; K., Matsui; S., Ossowski; S., Rodríguez (Ed.): Lect. Notes Networks Syst., pp. 43–52, Springer Science and Business Media Deutschland GmbH, 2025, ISBN: 23673370 (ISSN); 978-303182072-4 (ISBN).
Abstract | Links | BibTeX | Tags: Avatar, Generative adversarial networks, Generative AI, Health issues, Mental health, Self-compassion, Students, Training program, University students, Virtual avatar, Virtual environments, Virtual Reality, Virtual Space, Virtual spaces, Visual imagery
@inproceedings{koizumi_development_2025,
title = {Development and Assessment of a System to Help Students Improve Self-compassion},
author = {M. Koizumi and M. Ohsuga and J. M. Corchado},
editor = {Chinthaginjala R. and Sitek P. and Min-Allah N. and Matsui K. and Ossowski S. and Rodríguez S.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85218979175&doi=10.1007%2f978-3-031-82073-1_5&partnerID=40&md5=b136d4a114ce5acfa89f907ccecc145f},
doi = {10.1007/978-3-031-82073-1_5},
isbn = {23673370 (ISSN); 978-303182072-4 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Lect. Notes Networks Syst.},
volume = {1259},
pages = {43–52},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {Mental health issues are becoming more prevalent among university students. The mindful self-compassion (MSC) training program, which was introduced to address this issue, has shown some efficacy. However, many people, particularly Japanese people, have difficulty recalling visual imagery or feel uncomfortable or resistant to treating themselves with compassion. This study proposes and develops a system that uses virtual space and avatars to help individuals improve their self-compassion. In the proposed system, the user first selects an avatar of a person with whom to talk (hereafter referred to as “partner”), and then talks about the problem to the avatar of his/her choice. Next, the user changes viewpoints and listens to the problem as the partner’s avatar and responds with compassion. Finally, the user returns to his/her own avatar and listens to the compassionate response spoken as the partner avatar. We first conducted surveys to understand the important system components, and then developed prototypes. In light of the results of the experiments, we improved the prototype by introducing a generative AI. The first prototype used the user’s spoken voice as it was, but the improved system uses the generative AI to organize and convert the voice and present it. In addition, we added a function to generate and add advice with compression. The proposed system is expected to contribute to the improvement of students’ self-compassion. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.},
keywords = {Avatar, Generative adversarial networks, Generative AI, Health issues, Mental health, Self-compassion, Students, Training program, University students, Virtual avatar, Virtual environments, Virtual Reality, Virtual Space, Virtual spaces, Visual imagery},
pubstate = {published},
tppubtype = {inproceedings}
}
Mental health issues are becoming more prevalent among university students. The mindful self-compassion (MSC) training program, which was introduced to address this issue, has shown some efficacy. However, many people, particularly Japanese people, have difficulty recalling visual imagery or feel uncomfortable or resistant to treating themselves with compassion. This study proposes and develops a system that uses virtual space and avatars to help individuals improve their self-compassion. In the proposed system, the user first selects an avatar of a person with whom to talk (hereafter referred to as “partner”), and then talks about the problem to the avatar of his/her choice. Next, the user changes viewpoints and listens to the problem as the partner’s avatar and responds with compassion. Finally, the user returns to his/her own avatar and listens to the compassionate response spoken as the partner avatar. We first conducted surveys to understand the important system components, and then developed prototypes. In light of the results of the experiments, we improved the prototype by introducing a generative AI. The first prototype used the user’s spoken voice as it was, but the improved system uses the generative AI to organize and convert the voice and present it. In addition, we added a function to generate and add advice with compression. The proposed system is expected to contribute to the improvement of students’ self-compassion. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
Dang, B.; Huynh, L.; Gul, F.; Rosé, C.; Järvelä, S.; Nguyen, A.
Human–AI collaborative learning in mixed reality: Examining the cognitive and socio-emotional interactions Journal Article
In: British Journal of Educational Technology, 2025, ISSN: 00071013 (ISSN).
Abstract | Links | BibTeX | Tags: Artificial intelligence agent, Collaborative learning, Educational robots, Embodied agent, Emotional intelligence, Emotional interactions, Generative adversarial networks, generative artificial intelligence, Hierarchical clustering, Human–AI collaboration, Interaction pattern, Mixed reality, ordered network analysis, Ordered network analyze, Social behavior, Social interactions, Social psychology, Students, Supervised learning, Teaching
@article{dang_humanai_2025,
title = {Human–AI collaborative learning in mixed reality: Examining the cognitive and socio-emotional interactions},
author = {B. Dang and L. Huynh and F. Gul and C. Rosé and S. Järvelä and A. Nguyen},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105007896240&doi=10.1111%2fbjet.13607&partnerID=40&md5=b58a641069461f8880d1ee0adcf42457},
doi = {10.1111/bjet.13607},
issn = {00071013 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {British Journal of Educational Technology},
abstract = {The rise of generative artificial intelligence (GAI), especially with multimodal large language models like GPT-4o, sparked transformative potential and challenges for learning and teaching. With potential as a cognitive offloading tool, GAI can enable learners to focus on higher-order thinking and creativity. Yet, this also raises questions about integration into traditional education due to the limited research on learners' interactions with GAI. Some studies with GAI focus on text-based human–AI interactions, while research on embodied GAI in immersive environments like mixed reality (MR) remains unexplored. To address this, this study investigates interaction dynamics between learners and embodied GAI agents in MR, examining cognitive and socio-emotional interactions during collaborative learning. We investigated the paired interactive patterns between a student and an embodied GAI agent in MR, based on data from 26 higher education students with 1317 recorded activities. Data were analysed using a multi-layered learning analytics approach, including quantitative content analysis, sequence analysis via hierarchical clustering and pattern analysis through ordered network analysis (ONA). Our findings identified two interaction patterns: type (1) AI-led Supported Exploratory Questioning (AISQ) and type (2) Learner-Initiated Inquiry (LII) group. Despite their distinction in characteristic, both types demonstrated comparable levels of socio-emotional engagement and exhibited meaningful cognitive engagement, surpassing the superficial content reproduction that can be observed in interactions with GPT models. This study contributes to the human–AI collaboration and learning studies, extending understanding to learning in MR environments and highlighting implications for designing AI-based educational tools. Practitioner notes What is already known about this topic Socio-emotional interactions are fundamental to cognitive processes and play a critical role in collaborative learning. Generative artificial intelligence (GAI) holds transformative potential for education but raises questions about how learners interact with such technology. Most existing research focuses on text-based interactions with GAI; there is limited empirical evidence on how embodied GAI agents within immersive environments like Mixed Reality (MR) influence the cognitive and socio-emotional interactions for learning and regulation. What this paper adds Provides first empirical insights into cognitive and socio-emotional interaction patterns between learners and embodied GAI agents in MR environments. Identifies two distinct interaction patterns: AISQ type (structured, guided, supportive) and LII type (inquiry-driven, exploratory, engaging), demonstrating how these patterns influence collaborative learning dynamics. Shows that both interaction types facilitate meaningful cognitive engagement, moving beyond superficial content reproduction commonly associated with GAI interactions. Implications for practice and/or policy Insights from the identified interaction patterns can inform the design of teaching strategies that effectively integrate embodied GAI agents to enhance both cognitive and socio-emotional engagement. Findings can guide the development of AI-based educational tools that capitalise on the capabilities of embodied GAI agents, supporting a balance between structured guidance and exploratory learning. Highlights the need for ethical considerations in adopting embodied GAI agents, particularly regarding the human-like realism of these agents and potential impacts on learner dependency and interaction norms. © 2025 The Author(s). British Journal of Educational Technology published by John Wiley & Sons Ltd on behalf of British Educational Research Association.},
keywords = {Artificial intelligence agent, Collaborative learning, Educational robots, Embodied agent, Emotional intelligence, Emotional interactions, Generative adversarial networks, generative artificial intelligence, Hierarchical clustering, Human–AI collaboration, Interaction pattern, Mixed reality, ordered network analysis, Ordered network analyze, Social behavior, Social interactions, Social psychology, Students, Supervised learning, Teaching},
pubstate = {published},
tppubtype = {article}
}
The rise of generative artificial intelligence (GAI), especially with multimodal large language models like GPT-4o, sparked transformative potential and challenges for learning and teaching. With potential as a cognitive offloading tool, GAI can enable learners to focus on higher-order thinking and creativity. Yet, this also raises questions about integration into traditional education due to the limited research on learners' interactions with GAI. Some studies with GAI focus on text-based human–AI interactions, while research on embodied GAI in immersive environments like mixed reality (MR) remains unexplored. To address this, this study investigates interaction dynamics between learners and embodied GAI agents in MR, examining cognitive and socio-emotional interactions during collaborative learning. We investigated the paired interactive patterns between a student and an embodied GAI agent in MR, based on data from 26 higher education students with 1317 recorded activities. Data were analysed using a multi-layered learning analytics approach, including quantitative content analysis, sequence analysis via hierarchical clustering and pattern analysis through ordered network analysis (ONA). Our findings identified two interaction patterns: type (1) AI-led Supported Exploratory Questioning (AISQ) and type (2) Learner-Initiated Inquiry (LII) group. Despite their distinction in characteristic, both types demonstrated comparable levels of socio-emotional engagement and exhibited meaningful cognitive engagement, surpassing the superficial content reproduction that can be observed in interactions with GPT models. This study contributes to the human–AI collaboration and learning studies, extending understanding to learning in MR environments and highlighting implications for designing AI-based educational tools. Practitioner notes What is already known about this topic Socio-emotional interactions are fundamental to cognitive processes and play a critical role in collaborative learning. Generative artificial intelligence (GAI) holds transformative potential for education but raises questions about how learners interact with such technology. Most existing research focuses on text-based interactions with GAI; there is limited empirical evidence on how embodied GAI agents within immersive environments like Mixed Reality (MR) influence the cognitive and socio-emotional interactions for learning and regulation. What this paper adds Provides first empirical insights into cognitive and socio-emotional interaction patterns between learners and embodied GAI agents in MR environments. Identifies two distinct interaction patterns: AISQ type (structured, guided, supportive) and LII type (inquiry-driven, exploratory, engaging), demonstrating how these patterns influence collaborative learning dynamics. Shows that both interaction types facilitate meaningful cognitive engagement, moving beyond superficial content reproduction commonly associated with GAI interactions. Implications for practice and/or policy Insights from the identified interaction patterns can inform the design of teaching strategies that effectively integrate embodied GAI agents to enhance both cognitive and socio-emotional engagement. Findings can guide the development of AI-based educational tools that capitalise on the capabilities of embodied GAI agents, supporting a balance between structured guidance and exploratory learning. Highlights the need for ethical considerations in adopting embodied GAI agents, particularly regarding the human-like realism of these agents and potential impacts on learner dependency and interaction norms. © 2025 The Author(s). British Journal of Educational Technology published by John Wiley & Sons Ltd on behalf of British Educational Research Association.
Linares-Pellicer, J.; Izquierdo-Domenech, J.; Ferri-Molla, I.; Aliaga-Torro, C.
Breaking the Bottleneck: Generative AI as the Solution for XR Content Creation in Education Book Section
In: Lecture Notes in Networks and Systems, vol. 1140, pp. 9–30, Springer Science and Business Media Deutschland GmbH, 2025, ISBN: 23673370 (ISSN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, Augmented Reality, Breakings, Content creation, Contrastive Learning, Development process, Educational context, Federated learning, Generative adversarial networks, Immersive learning, Intelligence models, Learning experiences, Mixed reality, Resource intensity, Technical skills, Virtual environments
@incollection{linares-pellicer_breaking_2025,
title = {Breaking the Bottleneck: Generative AI as the Solution for XR Content Creation in Education},
author = {J. Linares-Pellicer and J. Izquierdo-Domenech and I. Ferri-Molla and C. Aliaga-Torro},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85212478399&doi=10.1007%2f978-3-031-71530-3_2&partnerID=40&md5=aefee938cd5b8a74ee811a463d7409ae},
doi = {10.1007/978-3-031-71530-3_2},
isbn = {23673370 (ISSN)},
year = {2025},
date = {2025-01-01},
booktitle = {Lecture Notes in Networks and Systems},
volume = {1140},
pages = {9–30},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {The integration of Extended Reality (XR) technologies-Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR)-promises to revolutionize education by offering immersive learning experiences. However, the complexity and resource intensity of content creation hinders the adoption of XR in educational contexts. This chapter explores Generative Artificial Intelligence (GenAI) as a solution, highlighting how GenAI models can facilitate the creation of educational XR content. GenAI enables educators to produce engaging XR experiences without needing advanced technical skills by automating aspects of the development process from ideation to deployment. Practical examples demonstrate GenAI’s current capability to generate assets and program applications, significantly lowering the barrier to creating personalized and interactive learning environments. The chapter also addresses challenges related to GenAI’s application in education, including technical limitations and ethical considerations. Ultimately, GenAI’s integration into XR content creation makes immersive educational experiences more accessible and practical, driven by only natural interactions, promising a future where technology-enhanced learning is universally attainable. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.},
keywords = {Adversarial machine learning, Augmented Reality, Breakings, Content creation, Contrastive Learning, Development process, Educational context, Federated learning, Generative adversarial networks, Immersive learning, Intelligence models, Learning experiences, Mixed reality, Resource intensity, Technical skills, Virtual environments},
pubstate = {published},
tppubtype = {incollection}
}
The integration of Extended Reality (XR) technologies-Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR)-promises to revolutionize education by offering immersive learning experiences. However, the complexity and resource intensity of content creation hinders the adoption of XR in educational contexts. This chapter explores Generative Artificial Intelligence (GenAI) as a solution, highlighting how GenAI models can facilitate the creation of educational XR content. GenAI enables educators to produce engaging XR experiences without needing advanced technical skills by automating aspects of the development process from ideation to deployment. Practical examples demonstrate GenAI’s current capability to generate assets and program applications, significantly lowering the barrier to creating personalized and interactive learning environments. The chapter also addresses challenges related to GenAI’s application in education, including technical limitations and ethical considerations. Ultimately, GenAI’s integration into XR content creation makes immersive educational experiences more accessible and practical, driven by only natural interactions, promising a future where technology-enhanced learning is universally attainable. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
Shi, J.; Jain, R.; Chi, S.; Doh, H.; Chi, H. -G.; Quinn, A. J.; Ramani, K.
CARING-AI: Towards Authoring Context-aware Augmented Reality INstruction through Generative Artificial Intelligence Proceedings Article
In: Conf Hum Fact Comput Syst Proc, Association for Computing Machinery, 2025, ISBN: 979-840071394-1 (ISBN).
Abstract | Links | BibTeX | Tags: 'current, Application scenario, AR application, Augmented Reality, Context-Aware, Contextual information, Generative adversarial networks, generative artificial intelligence, Humanoid avatars, In-situ learning, Learning experiences, Power
@inproceedings{shi_caring-ai_2025,
title = {CARING-AI: Towards Authoring Context-aware Augmented Reality INstruction through Generative Artificial Intelligence},
author = {J. Shi and R. Jain and S. Chi and H. Doh and H. -G. Chi and A. J. Quinn and K. Ramani},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005725461&doi=10.1145%2f3706598.3713348&partnerID=40&md5=e88afd8426e020155599ef3b2a044774},
doi = {10.1145/3706598.3713348},
isbn = {979-840071394-1 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Conf Hum Fact Comput Syst Proc},
publisher = {Association for Computing Machinery},
abstract = {Context-aware AR instruction enables adaptive and in-situ learning experiences. However, hardware limitations and expertise requirements constrain the creation of such instructions. With recent developments in Generative Artificial Intelligence (Gen-AI), current research tries to tackle these constraints by deploying AI-generated content (AIGC) in AR applications. However, our preliminary study with six AR practitioners revealed that the current AIGC lacks contextual information to adapt to varying application scenarios and is therefore limited in authoring. To utilize the strong generative power of GenAI to ease the authoring of AR instruction while capturing the context, we developed CARING-AI, an AR system to author context-aware humanoid-avatar-based instructions with GenAI. By navigating in the environment, users naturally provide contextual information to generate humanoid-avatar animation as AR instructions that blend in the context spatially and temporally. We showcased three application scenarios of CARING-AI: Asynchronous Instructions, Remote Instructions, and Ad Hoc Instructions based on a design space of AIGC in AR Instructions. With two user studies (N=12), we assessed the system usability of CARING-AI and demonstrated the easiness and effectiveness of authoring with Gen-AI. © 2025 Copyright held by the owner/author(s).},
keywords = {'current, Application scenario, AR application, Augmented Reality, Context-Aware, Contextual information, Generative adversarial networks, generative artificial intelligence, Humanoid avatars, In-situ learning, Learning experiences, Power},
pubstate = {published},
tppubtype = {inproceedings}
}
Context-aware AR instruction enables adaptive and in-situ learning experiences. However, hardware limitations and expertise requirements constrain the creation of such instructions. With recent developments in Generative Artificial Intelligence (Gen-AI), current research tries to tackle these constraints by deploying AI-generated content (AIGC) in AR applications. However, our preliminary study with six AR practitioners revealed that the current AIGC lacks contextual information to adapt to varying application scenarios and is therefore limited in authoring. To utilize the strong generative power of GenAI to ease the authoring of AR instruction while capturing the context, we developed CARING-AI, an AR system to author context-aware humanoid-avatar-based instructions with GenAI. By navigating in the environment, users naturally provide contextual information to generate humanoid-avatar animation as AR instructions that blend in the context spatially and temporally. We showcased three application scenarios of CARING-AI: Asynchronous Instructions, Remote Instructions, and Ad Hoc Instructions based on a design space of AIGC in AR Instructions. With two user studies (N=12), we assessed the system usability of CARING-AI and demonstrated the easiness and effectiveness of authoring with Gen-AI. © 2025 Copyright held by the owner/author(s).
Behravan, M.; Matković, K.; Gračanin, D.
Generative AI for Context-Aware 3D Object Creation Using Vision-Language Models in Augmented Reality Proceedings Article
In: Proc. - IEEE Int. Conf. Artif. Intell. Ext. Virtual Real., AIxVR, pp. 73–81, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 979-833152157-8 (ISBN).
Abstract | Links | BibTeX | Tags: 3D object, 3D Object Generation, Artificial intelligence systems, Augmented Reality, Capture images, Context-Aware, Generative adversarial networks, Generative AI, generative artificial intelligence, Generative model, Language Model, Object creation, Vision language model, vision language models, Visual languages
@inproceedings{behravan_generative_2025,
title = {Generative AI for Context-Aware 3D Object Creation Using Vision-Language Models in Augmented Reality},
author = {M. Behravan and K. Matković and D. Gračanin},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105000292700&doi=10.1109%2fAIxVR63409.2025.00018&partnerID=40&md5=b40fa769a6b427918c3fcd86f7c52a75},
doi = {10.1109/AIxVR63409.2025.00018},
isbn = {979-833152157-8 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. - IEEE Int. Conf. Artif. Intell. Ext. Virtual Real., AIxVR},
pages = {73–81},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {We present a novel Artificial Intelligence (AI) system that functions as a designer assistant in augmented reality (AR) environments. Leveraging Vision Language Models (VLMs) like LLaVA and advanced text-to-3D generative models, users can capture images of their surroundings with an Augmented Reality (AR) headset. The system analyzes these images to recommend contextually relevant objects that enhance both functionality and visual appeal. The recommended objects are generated as 3D models and seamlessly integrated into the AR environment for interactive use. Our system utilizes open-source AI models running on local systems to enhance data security and reduce operational costs. Key features include context-aware object suggestions, optimal placement guidance, aesthetic matching, and an intuitive user interface for real-time interaction. Evaluations using the COCO 2017 dataset and real-world AR testing demonstrated high accuracy in object detection and contextual fit rating of 4.1 out of 5. By addressing the challenge of providing context-aware object recommendations in AR, our system expands the capabilities of AI applications in this domain. It enables users to create personalized digital spaces efficiently, leveraging AI for contextually relevant suggestions. © 2025 IEEE.},
keywords = {3D object, 3D Object Generation, Artificial intelligence systems, Augmented Reality, Capture images, Context-Aware, Generative adversarial networks, Generative AI, generative artificial intelligence, Generative model, Language Model, Object creation, Vision language model, vision language models, Visual languages},
pubstate = {published},
tppubtype = {inproceedings}
}
We present a novel Artificial Intelligence (AI) system that functions as a designer assistant in augmented reality (AR) environments. Leveraging Vision Language Models (VLMs) like LLaVA and advanced text-to-3D generative models, users can capture images of their surroundings with an Augmented Reality (AR) headset. The system analyzes these images to recommend contextually relevant objects that enhance both functionality and visual appeal. The recommended objects are generated as 3D models and seamlessly integrated into the AR environment for interactive use. Our system utilizes open-source AI models running on local systems to enhance data security and reduce operational costs. Key features include context-aware object suggestions, optimal placement guidance, aesthetic matching, and an intuitive user interface for real-time interaction. Evaluations using the COCO 2017 dataset and real-world AR testing demonstrated high accuracy in object detection and contextual fit rating of 4.1 out of 5. By addressing the challenge of providing context-aware object recommendations in AR, our system expands the capabilities of AI applications in this domain. It enables users to create personalized digital spaces efficiently, leveraging AI for contextually relevant suggestions. © 2025 IEEE.
Casas, L.; Mitchell, K.
Structured Teaching Prompt Articulation for Generative-AI Role Embodiment with Augmented Mirror Video Displays Proceedings Article
In: S.N., Spencer (Ed.): Proc.: VRCAI - ACM SIGGRAPH Int. Conf. Virtual-Reality Contin. Appl. Ind., Association for Computing Machinery, Inc, 2025, ISBN: 979-840071348-4 (ISBN).
Abstract | Links | BibTeX | Tags: Artificial intelligence, Augmented Reality, Computer interaction, Contrastive Learning, Cultural icon, Experiential learning, Generative adversarial networks, Generative AI, human-computer interaction, Immersive, Pedagogical practices, Role-based, Teachers', Teaching, Video display, Virtual environments, Virtual Reality
@inproceedings{casas_structured_2025,
title = {Structured Teaching Prompt Articulation for Generative-AI Role Embodiment with Augmented Mirror Video Displays},
author = {L. Casas and K. Mitchell},
editor = {Spencer S.N.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85217997060&doi=10.1145%2f3703619.3706049&partnerID=40&md5=7141c5dac7882232c6ee8e0bef0ba84e},
doi = {10.1145/3703619.3706049},
isbn = {979-840071348-4 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc.: VRCAI - ACM SIGGRAPH Int. Conf. Virtual-Reality Contin. Appl. Ind.},
publisher = {Association for Computing Machinery, Inc},
abstract = {We present a classroom enhanced with augmented reality video display in which students adopt snapshots of their corresponding virtual personas according to their teacher's live articulated spoken educational theme, linearly, such as historical figures, famous scientists, cultural icons, and laterally according to archetypal categories such as world dance styles. We define a structure of generative AI prompt guidance to assist teachers with focused specified visual role embodiment stylization. By leveraging role-based immersive embodiment, our proposed approach enriches pedagogical practices that prioritize experiential learning. © 2024 ACM.},
keywords = {Artificial intelligence, Augmented Reality, Computer interaction, Contrastive Learning, Cultural icon, Experiential learning, Generative adversarial networks, Generative AI, human-computer interaction, Immersive, Pedagogical practices, Role-based, Teachers', Teaching, Video display, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
We present a classroom enhanced with augmented reality video display in which students adopt snapshots of their corresponding virtual personas according to their teacher's live articulated spoken educational theme, linearly, such as historical figures, famous scientists, cultural icons, and laterally according to archetypal categories such as world dance styles. We define a structure of generative AI prompt guidance to assist teachers with focused specified visual role embodiment stylization. By leveraging role-based immersive embodiment, our proposed approach enriches pedagogical practices that prioritize experiential learning. © 2024 ACM.
Shibuya, K.
Transforming phenomenological sociology for virtual personalities and virtual worlds Journal Article
In: AI and Society, vol. 40, no. 5, pp. 3317–3331, 2025, ISSN: 09515666 (ISSN).
Abstract | Links | BibTeX | Tags: Advanced technology, Economic and social effects, Generative adversarial networks, Generative AI, Human being, Identity, Intersubjectivity, Metadata, Phenomenological Sociology, Sociology, Technological innovation, Virtual environments, Virtual Personality, Virtual Reality, Virtual worlds, Virtualization, Virtualizations
@article{shibuya_transforming_2025,
title = {Transforming phenomenological sociology for virtual personalities and virtual worlds},
author = {K. Shibuya},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85217199972&doi=10.1007%2fs00146-025-02189-x&partnerID=40&md5=aa9db1cb1f99419b605f1091469eb77c},
doi = {10.1007/s00146-025-02189-x},
issn = {09515666 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {AI and Society},
volume = {40},
number = {5},
pages = {3317–3331},
abstract = {Are there opportunities to use the plural to express the first person (“I”) of “the same person” in English? It means that the self is an entity that guarantees uniqueness and is at the core of identity. Recently, radical and rapid innovations in AI technologies have made it possible to alter our existential fundamentals. Principally, we are now interacting with “virtual personalities” generated by generative AI. Thus, there is an inevitability to explore the relationship between AI and society, and the problem domain of phenomenological sociology related to the “virtuality” of personalities and the world. Encountering and interacting with “others without subject” artificially generated by generative AI based on individual big data and attribute data is a situation that mankind has never experienced before from the perspective of sociology and phenomenological sociology related to the ego. The virtual personalities can be perceived as if it were interacting with existing humans in the form of video and audio, and it is also possible to arbitrarily change their attributes (e.g., gender, race, age, physical characteristics) and other settings, as well as to virtually create deceased persons or great figures from the past. Such technological innovation is, so to speak, a virtualization of human existential identity, and advanced technologies such as AI will transform not only the boundary between self and others but also the aspect of human existence itself (Shibuya in Digital transformation of identity in the age of artificial intelligence. Springer, Belrin, 2020). In addition, from a phenomenological viewpoint, the boundary between reality and virtuality is blurring due to technological innovation in the living world itself, and there is a concern that this will lead to an artificial state of detachment. Actually, the use of advanced technologies such as AI, VR in virtual worlds and cyberspace will not only cause people to lose their reality and actuality but will also endanger the very foundations of their existential identity. Therefore, we must ask what it means for us as existences to interact with virtual personalities in a virtually generated world, and what is the nature of the intersubjectivity formation and semantic understanding as well as the modes of existence, facts, and worlds, and what are their evidential natures. In line with what Husserl, the founder of phenomenology, once declared at the beginning of his “Cartesianische Meditationen” (Husserl in CartesianischeMeditationen, e-artnow, 2018), that “we need to begin philosophy radically anew”, as also phenomenological sociology, it can now state that “we need to begin phenomenological sociology radically anew”. Then, this paper reviews and discusses the following issues based on technological trends. Is there an intersubjectivity between the virtual personalities generated by the AI and the human being? How does the virtualization of identity, as well as the difference between self and others, transform the nature of existence? How is a mutual semantic understanding possible between a human being and the virtual personality that is generated by a generative AI and a generative AI? How can we verify discourses and propositions of fact and worldliness in our interactions with generative AIs, and how can we overcome the illusion (i.e., hallucination) that generative AIs create? What does the transformation of the world and its aspect as existence mean? How is it possible to collaborate between a human being and the virtual personality that is generated by a generative AI and a generative AI? © The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2025.},
keywords = {Advanced technology, Economic and social effects, Generative adversarial networks, Generative AI, Human being, Identity, Intersubjectivity, Metadata, Phenomenological Sociology, Sociology, Technological innovation, Virtual environments, Virtual Personality, Virtual Reality, Virtual worlds, Virtualization, Virtualizations},
pubstate = {published},
tppubtype = {article}
}
Are there opportunities to use the plural to express the first person (“I”) of “the same person” in English? It means that the self is an entity that guarantees uniqueness and is at the core of identity. Recently, radical and rapid innovations in AI technologies have made it possible to alter our existential fundamentals. Principally, we are now interacting with “virtual personalities” generated by generative AI. Thus, there is an inevitability to explore the relationship between AI and society, and the problem domain of phenomenological sociology related to the “virtuality” of personalities and the world. Encountering and interacting with “others without subject” artificially generated by generative AI based on individual big data and attribute data is a situation that mankind has never experienced before from the perspective of sociology and phenomenological sociology related to the ego. The virtual personalities can be perceived as if it were interacting with existing humans in the form of video and audio, and it is also possible to arbitrarily change their attributes (e.g., gender, race, age, physical characteristics) and other settings, as well as to virtually create deceased persons or great figures from the past. Such technological innovation is, so to speak, a virtualization of human existential identity, and advanced technologies such as AI will transform not only the boundary between self and others but also the aspect of human existence itself (Shibuya in Digital transformation of identity in the age of artificial intelligence. Springer, Belrin, 2020). In addition, from a phenomenological viewpoint, the boundary between reality and virtuality is blurring due to technological innovation in the living world itself, and there is a concern that this will lead to an artificial state of detachment. Actually, the use of advanced technologies such as AI, VR in virtual worlds and cyberspace will not only cause people to lose their reality and actuality but will also endanger the very foundations of their existential identity. Therefore, we must ask what it means for us as existences to interact with virtual personalities in a virtually generated world, and what is the nature of the intersubjectivity formation and semantic understanding as well as the modes of existence, facts, and worlds, and what are their evidential natures. In line with what Husserl, the founder of phenomenology, once declared at the beginning of his “Cartesianische Meditationen” (Husserl in CartesianischeMeditationen, e-artnow, 2018), that “we need to begin philosophy radically anew”, as also phenomenological sociology, it can now state that “we need to begin phenomenological sociology radically anew”. Then, this paper reviews and discusses the following issues based on technological trends. Is there an intersubjectivity between the virtual personalities generated by the AI and the human being? How does the virtualization of identity, as well as the difference between self and others, transform the nature of existence? How is a mutual semantic understanding possible between a human being and the virtual personality that is generated by a generative AI and a generative AI? How can we verify discourses and propositions of fact and worldliness in our interactions with generative AIs, and how can we overcome the illusion (i.e., hallucination) that generative AIs create? What does the transformation of the world and its aspect as existence mean? How is it possible to collaborate between a human being and the virtual personality that is generated by a generative AI and a generative AI? © The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2025.
Barbu, M.; Iordache, D. -D.; Petre, I.; Barbu, D. -C.; Băjenaru, L.
Framework Design for Reinforcing the Potential of XR Technologies in Transforming Inclusive Education Journal Article
In: Applied Sciences (Switzerland), vol. 15, no. 3, 2025, ISSN: 20763417 (ISSN).
Abstract | Links | BibTeX | Tags: Adaptive Learning, Adversarial machine learning, Artificial intelligence technologies, Augmented Reality, Contrastive Learning, Educational Technology, Extended reality (XR), Federated learning, Framework designs, Generative adversarial networks, Immersive, immersive experience, Immersive learning, Inclusive education, Learning platform, Special education needs
@article{barbu_framework_2025,
title = {Framework Design for Reinforcing the Potential of XR Technologies in Transforming Inclusive Education},
author = {M. Barbu and D. -D. Iordache and I. Petre and D. -C. Barbu and L. Băjenaru},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85217742383&doi=10.3390%2fapp15031484&partnerID=40&md5=3148ff2a8a8fa1bef8094199cd6d32e3},
doi = {10.3390/app15031484},
issn = {20763417 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Applied Sciences (Switzerland)},
volume = {15},
number = {3},
abstract = {This study presents a novel approach to inclusive education by integrating augmented reality (XR) and generative artificial intelligence (AI) technologies into an immersive and adaptive learning platform designed for students with special educational needs. Building upon existing solutions, the approach uniquely combines XR and generative AI to facilitate personalized, accessible, and interactive learning experiences tailored to individual requirements. The framework incorporates an intuitive Unity XR-based interface alongside a generative AI module to enable near real-time customization of content and interactions. Additionally, the study examines related generative AI initiatives that promote inclusion through enhanced communication tools, educational support, and customizable assistive technologies. The motivation for this study arises from the pressing need to address the limitations of traditional educational methods, which often fail to meet the diverse needs of learners with special educational requirements. The integration of XR and generative AI offers transformative potential by creating adaptive, immersive, and inclusive learning environments. This approach ensures real-time adaptability to individual progress and accessibility, addressing critical barriers such as static content and lack of inclusivity in existing systems. The research outlines a pathway toward more inclusive and equitable education, significantly enhancing opportunities for learners with diverse needs and contributing to broader social integration and equity in education. © 2025 by the authors.},
keywords = {Adaptive Learning, Adversarial machine learning, Artificial intelligence technologies, Augmented Reality, Contrastive Learning, Educational Technology, Extended reality (XR), Federated learning, Framework designs, Generative adversarial networks, Immersive, immersive experience, Immersive learning, Inclusive education, Learning platform, Special education needs},
pubstate = {published},
tppubtype = {article}
}
This study presents a novel approach to inclusive education by integrating augmented reality (XR) and generative artificial intelligence (AI) technologies into an immersive and adaptive learning platform designed for students with special educational needs. Building upon existing solutions, the approach uniquely combines XR and generative AI to facilitate personalized, accessible, and interactive learning experiences tailored to individual requirements. The framework incorporates an intuitive Unity XR-based interface alongside a generative AI module to enable near real-time customization of content and interactions. Additionally, the study examines related generative AI initiatives that promote inclusion through enhanced communication tools, educational support, and customizable assistive technologies. The motivation for this study arises from the pressing need to address the limitations of traditional educational methods, which often fail to meet the diverse needs of learners with special educational requirements. The integration of XR and generative AI offers transformative potential by creating adaptive, immersive, and inclusive learning environments. This approach ensures real-time adaptability to individual progress and accessibility, addressing critical barriers such as static content and lack of inclusivity in existing systems. The research outlines a pathway toward more inclusive and equitable education, significantly enhancing opportunities for learners with diverse needs and contributing to broader social integration and equity in education. © 2025 by the authors.
Stacchio, L.; Balloni, E.; Frontoni, E.; Paolanti, M.; Zingaretti, P.; Pierdicca, R.
MineVRA: Exploring the Role of Generative AI-Driven Content Development in XR Environments through a Context-Aware Approach Journal Article
In: IEEE Transactions on Visualization and Computer Graphics, vol. 31, no. 5, pp. 3602–3612, 2025, ISSN: 10772626 (ISSN).
Abstract | Links | BibTeX | Tags: adult, Article, Artificial intelligence, Computer graphics, Computer vision, Content Development, Contents development, Context-Aware, Context-aware approaches, Extended reality, female, Generative adversarial networks, Generative AI, generative artificial intelligence, human, Human-in-the-loop, Immersive, Immersive environment, male, Multi-modal, User need, Virtual environments, Virtual Reality
@article{stacchio_minevra_2025,
title = {MineVRA: Exploring the Role of Generative AI-Driven Content Development in XR Environments through a Context-Aware Approach},
author = {L. Stacchio and E. Balloni and E. Frontoni and M. Paolanti and P. Zingaretti and R. Pierdicca},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105003746367&doi=10.1109%2fTVCG.2025.3549160&partnerID=40&md5=70b162b574eebbb0cb71db871aa787e1},
doi = {10.1109/TVCG.2025.3549160},
issn = {10772626 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Transactions on Visualization and Computer Graphics},
volume = {31},
number = {5},
pages = {3602–3612},
abstract = {The convergence of Artificial Intelligence (AI), Computer Vision (CV), Computer Graphics (CG), and Extended Reality (XR) is driving innovation in immersive environments. A key challenge in these environments is the creation of personalized 3D assets, traditionally achieved through manual modeling, a time-consuming process that often fails to meet individual user needs. More recently, Generative AI (GenAI) has emerged as a promising solution for automated, context-aware content generation. In this paper, we present MineVRA (Multimodal generative artificial iNtelligence for contExt-aware Virtual Reality Assets), a novel Human-In-The-Loop (HITL) XR framework that integrates GenAI to facilitate coherent and adaptive 3D content generation in immersive scenarios. To evaluate the effectiveness of this approach, we conducted a comparative user study analyzing the performance and user satisfaction of GenAI-generated 3D objects compared to those generated by Sketchfab in different immersive contexts. The results suggest that GenAI can significantly complement traditional 3D asset libraries, with valuable design implications for the development of human-centered XR environments. © 1995-2012 IEEE.},
keywords = {adult, Article, Artificial intelligence, Computer graphics, Computer vision, Content Development, Contents development, Context-Aware, Context-aware approaches, Extended reality, female, Generative adversarial networks, Generative AI, generative artificial intelligence, human, Human-in-the-loop, Immersive, Immersive environment, male, Multi-modal, User need, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
The convergence of Artificial Intelligence (AI), Computer Vision (CV), Computer Graphics (CG), and Extended Reality (XR) is driving innovation in immersive environments. A key challenge in these environments is the creation of personalized 3D assets, traditionally achieved through manual modeling, a time-consuming process that often fails to meet individual user needs. More recently, Generative AI (GenAI) has emerged as a promising solution for automated, context-aware content generation. In this paper, we present MineVRA (Multimodal generative artificial iNtelligence for contExt-aware Virtual Reality Assets), a novel Human-In-The-Loop (HITL) XR framework that integrates GenAI to facilitate coherent and adaptive 3D content generation in immersive scenarios. To evaluate the effectiveness of this approach, we conducted a comparative user study analyzing the performance and user satisfaction of GenAI-generated 3D objects compared to those generated by Sketchfab in different immersive contexts. The results suggest that GenAI can significantly complement traditional 3D asset libraries, with valuable design implications for the development of human-centered XR environments. © 1995-2012 IEEE.
Xing, Y.; Liu, Q.; Wang, J.; Gómez-Zará, D.
sMoRe: Spatial Mapping and Object Rendering Environment Proceedings Article
In: Int Conf Intell User Interfaces Proc IUI, pp. 115–119, Association for Computing Machinery, 2025, ISBN: 979-840071409-2 (ISBN).
Abstract | Links | BibTeX | Tags: Generative adversarial networks, Generative AI, Language Model, Large language model, large language models, Mapping, Mixed reality, Mixed-reality environment, Object rendering, Rendering (computer graphics), Space Manipulation, Spatial mapping, Spatial objects, Users' experiences, Virtual environments, Virtual objects
@inproceedings{xing_smore_2025,
title = {sMoRe: Spatial Mapping and Object Rendering Environment},
author = {Y. Xing and Q. Liu and J. Wang and D. Gómez-Zará},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105001670668&doi=10.1145%2f3708557.3716337&partnerID=40&md5=8ef4c5c4ef2b3ee30d00e4b8d19d19b8},
doi = {10.1145/3708557.3716337},
isbn = {979-840071409-2 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Int Conf Intell User Interfaces Proc IUI},
pages = {115–119},
publisher = {Association for Computing Machinery},
abstract = {In mixed reality (MR) environments, understanding space and creating virtual objects is crucial to providing an intuitive user experience. This paper introduces sMoRe (Spatial Mapping and Object Rendering Environment), an MR application that combines Generative AI (GenAI) to assist users in creating, placing, and managing virtual objects within physical spaces. sMoRe allows users to use voice or typed text commands to create and place virtual objects using GenAI while specifying spatial constraints. The system employs Large Language Models (LLMs) to interpret users’ commands, analyze the current scene, and identify optimal locations. Additionally, sMoRe integrates a text-to-3D generative model to dynamically create 3D objects based on users’ descriptions. Our user study demonstrates the effectiveness of sMoRe in enhancing user comprehension, interaction, and organization of the MR environment. © 2025 Copyright held by the owner/author(s).},
keywords = {Generative adversarial networks, Generative AI, Language Model, Large language model, large language models, Mapping, Mixed reality, Mixed-reality environment, Object rendering, Rendering (computer graphics), Space Manipulation, Spatial mapping, Spatial objects, Users' experiences, Virtual environments, Virtual objects},
pubstate = {published},
tppubtype = {inproceedings}
}
In mixed reality (MR) environments, understanding space and creating virtual objects is crucial to providing an intuitive user experience. This paper introduces sMoRe (Spatial Mapping and Object Rendering Environment), an MR application that combines Generative AI (GenAI) to assist users in creating, placing, and managing virtual objects within physical spaces. sMoRe allows users to use voice or typed text commands to create and place virtual objects using GenAI while specifying spatial constraints. The system employs Large Language Models (LLMs) to interpret users’ commands, analyze the current scene, and identify optimal locations. Additionally, sMoRe integrates a text-to-3D generative model to dynamically create 3D objects based on users’ descriptions. Our user study demonstrates the effectiveness of sMoRe in enhancing user comprehension, interaction, and organization of the MR environment. © 2025 Copyright held by the owner/author(s).
2024
Lai, B.; He, J.; Kang, J.; Li, G.; Xu, M.; Zhang, T.; Xie, S.
On-demand Quantization for Green Federated Generative Diffusion in Mobile Edge Networks Proceedings Article
In: M., Valenti; D., Reed; M., Torres (Ed.): IEEE Int Conf Commun, pp. 2883–2888, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 15503607 (ISSN); 978-172819054-9 (ISBN).
Abstract | Links | BibTeX | Tags: EDGE Networks, Energy Efficient, Federated Diffusion, Federated learning, Generative adversarial networks, Generative AI, Generative Diffusion, Intelligence models, Metaverses, On demands, On-demand Quantization, Quantisation
@inproceedings{lai_-demand_2024,
title = {On-demand Quantization for Green Federated Generative Diffusion in Mobile Edge Networks},
author = {B. Lai and J. He and J. Kang and G. Li and M. Xu and T. Zhang and S. Xie},
editor = {Valenti M. and Reed D. and Torres M.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85202836582&doi=10.1109%2fICC51166.2024.10622695&partnerID=40&md5=8318ef451bb946470eb34ef50d09b5a1},
doi = {10.1109/ICC51166.2024.10622695},
isbn = {15503607 (ISSN); 978-172819054-9 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {IEEE Int Conf Commun},
pages = {2883–2888},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Generative Artificial Intelligence (GAI) shows remarkable productivity and creativity in Mobile Edge Networks, such as the metaverse and the Industrial Internet of Things. Federated learning is a promising technique for effectively training GAI models in mobile edge networks due to its data distribution. However, there is a notable issue with communication consumption when training large GAI models like generative diffusion models in mobile edge networks. Additionally, the substantial energy consumption associated with training diffusion-based models, along with the limited resources of edge devices and complexities of network environments, pose challenges for improving the training efficiency of GAI models. To address this challenge, we propose an on-demand quantized energy-efficient federated diffusion approach for mobile edge networks. Specifically, we first design a dynamic quantized federated diffusion training scheme considering various demands from the edge devices. Then, we study an energy efficiency problem based on specific quantization requirements. Numerical results show that our proposed method significantly reduces system energy consumption and transmitted model size compared to both baseline federated diffusion and fixed quantized federated diffusion methods while effectively maintaining reasonable quality and diversity of generated data. © 2024 IEEE.},
keywords = {EDGE Networks, Energy Efficient, Federated Diffusion, Federated learning, Generative adversarial networks, Generative AI, Generative Diffusion, Intelligence models, Metaverses, On demands, On-demand Quantization, Quantisation},
pubstate = {published},
tppubtype = {inproceedings}
}
Generative Artificial Intelligence (GAI) shows remarkable productivity and creativity in Mobile Edge Networks, such as the metaverse and the Industrial Internet of Things. Federated learning is a promising technique for effectively training GAI models in mobile edge networks due to its data distribution. However, there is a notable issue with communication consumption when training large GAI models like generative diffusion models in mobile edge networks. Additionally, the substantial energy consumption associated with training diffusion-based models, along with the limited resources of edge devices and complexities of network environments, pose challenges for improving the training efficiency of GAI models. To address this challenge, we propose an on-demand quantized energy-efficient federated diffusion approach for mobile edge networks. Specifically, we first design a dynamic quantized federated diffusion training scheme considering various demands from the edge devices. Then, we study an energy efficiency problem based on specific quantization requirements. Numerical results show that our proposed method significantly reduces system energy consumption and transmitted model size compared to both baseline federated diffusion and fixed quantized federated diffusion methods while effectively maintaining reasonable quality and diversity of generated data. © 2024 IEEE.
You, F.; Du, H.; Kang, J.; Ni, W.; Niyato, D.; Jamalipour, A.
Generative AI-aided Reinforcement Learning for Computation Offloading and Privacy Protection in VR-based Multi-Access Edge Computing Proceedings Article
In: Proc. - IEEE Smart World Congr., SWC - IEEE Ubiquitous Intell. Comput., Auton. Trusted Comput., Digit. Twin, Metaverse, Priv. Comput. Data Secur., Scalable Comput. Commun., pp. 2209–2214, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-833152086-1 (ISBN).
Abstract | Links | BibTeX | Tags: Computation offloading, Content services, Differential privacy, Diffusion Model, Edge computing, Generative adversarial networks, Generative diffusion model, generative diffusion models, Inverse problems, Multi-access edge computing, Multiaccess, Policy optimization, Proximal policy optimization, Reinforcement Learning, User privacy, Virtual environments, Virtual Reality
@inproceedings{you_generative_2024,
title = {Generative AI-aided Reinforcement Learning for Computation Offloading and Privacy Protection in VR-based Multi-Access Edge Computing},
author = {F. You and H. Du and J. Kang and W. Ni and D. Niyato and A. Jamalipour},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105002235341&doi=10.1109%2fSWC62898.2024.00337&partnerID=40&md5=301c91c0b737c401d54e154e13dc8d47},
doi = {10.1109/SWC62898.2024.00337},
isbn = {979-833152086-1 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - IEEE Smart World Congr., SWC - IEEE Ubiquitous Intell. Comput., Auton. Trusted Comput., Digit. Twin, Metaverse, Priv. Comput. Data Secur., Scalable Comput. Commun.},
pages = {2209–2214},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {The rapid growth of Artificial Intelligence-Generated Content (AIGC) services has led to increased mobile user participation in related computations and interactions. This development has enabled AI-generated characters to interact with Virtual Reality (VR) users in real time, making the VR experience more interactive and personalized. In this paper, we consider an MEC system where VR users engage in AIGC services, focusing on the Generative Diffusion Model (GDM)based image generation tasks. Specifically, VR users initiate requests for computing resources, while computation offloading distributes the processing load across the MEC system. To manage AIGC edge computation offloading and cloudlet-VR user connections jointly, a Data Center Operator (DCO) employs a centralized Proximal Policy Optimization (PPO) algorithm. To protect VR users' privacy while preserving PPO functionality, we employ the Generative Diffusion Model (GDM), specifically the Denoising Diffusion Implicit Model (DDIM), which first introduces noise to the PPO state, then conducts a denoising process to recover the state information. We further employ Inverse Reinforcement Learning (IRL) to infer rewards for the recovered states, using expert demonstrations trained by the PPO. The similarity between PPO-generated rewards and IRL-inferred rewards is then computed. Simulation results demonstrate that our proposed approach successfully achieves computation offloading while protecting VR users' privacy within the PPO centralized management framework. © 2024 IEEE.},
keywords = {Computation offloading, Content services, Differential privacy, Diffusion Model, Edge computing, Generative adversarial networks, Generative diffusion model, generative diffusion models, Inverse problems, Multi-access edge computing, Multiaccess, Policy optimization, Proximal policy optimization, Reinforcement Learning, User privacy, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
The rapid growth of Artificial Intelligence-Generated Content (AIGC) services has led to increased mobile user participation in related computations and interactions. This development has enabled AI-generated characters to interact with Virtual Reality (VR) users in real time, making the VR experience more interactive and personalized. In this paper, we consider an MEC system where VR users engage in AIGC services, focusing on the Generative Diffusion Model (GDM)based image generation tasks. Specifically, VR users initiate requests for computing resources, while computation offloading distributes the processing load across the MEC system. To manage AIGC edge computation offloading and cloudlet-VR user connections jointly, a Data Center Operator (DCO) employs a centralized Proximal Policy Optimization (PPO) algorithm. To protect VR users' privacy while preserving PPO functionality, we employ the Generative Diffusion Model (GDM), specifically the Denoising Diffusion Implicit Model (DDIM), which first introduces noise to the PPO state, then conducts a denoising process to recover the state information. We further employ Inverse Reinforcement Learning (IRL) to infer rewards for the recovered states, using expert demonstrations trained by the PPO. The similarity between PPO-generated rewards and IRL-inferred rewards is then computed. Simulation results demonstrate that our proposed approach successfully achieves computation offloading while protecting VR users' privacy within the PPO centralized management framework. © 2024 IEEE.
Ding, P.; Liu, J.; Sun, M.; Li, L.; Liu, H.
Enhancing Computational Processing Performance for Generative AI Large Models with Autonomous Decision-Making in Metaverse Applications Proceedings Article
In: Proc. - IEEE Int. Conf. Metaverse Comput., Netw., Appl., MetaCom, pp. 253–258, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-833151599-7 (ISBN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, AGI (Artificial General Intelligence), Artificial general intelligence, Artificial general intelligences, Autonomous decision, Autonomous Decision-Making, Data assimilation, Data integration, Decisions makings, Digital Twin Technology, Emotion Recognition, Generative adversarial networks, Generative AI large model, Generative AI Large Models, Large models, Metaverse, Metaverses, Model Acceleration, Model Compression, Multi agent systems, Multi-agent systems, Multi-modal data, Multi-Modal Data Integration, Multiagent systems (MASs), Reinforcement Learning, Reinforcement learnings, Spatio-temporal data
@inproceedings{ding_enhancing_2024,
title = {Enhancing Computational Processing Performance for Generative AI Large Models with Autonomous Decision-Making in Metaverse Applications},
author = {P. Ding and J. Liu and M. Sun and L. Li and H. Liu},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85211489063&doi=10.1109%2fMetaCom62920.2024.00048&partnerID=40&md5=ae085a7d90b12c9090f5bf7a274bc7ce},
doi = {10.1109/MetaCom62920.2024.00048},
isbn = {979-833151599-7 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - IEEE Int. Conf. Metaverse Comput., Netw., Appl., MetaCom},
pages = {253–258},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {We explore how to enhance the computational processing performance for generative AI large models with autonomous decision-making in metaverse applications. We first introduce the relationship between AI large models and the Metaverse. We elaborate on the application scenarios of generative AI large models in Metaverse, including real-time weather simulation, embodied intelligence of agents, dynamic environment interaction, and user emotion recognition. We then propose the method of Multi-Dimensional Optimization Generation Framework (MDOGF) to improve computational processing performance. The experiment results show great improvement in computational processing performance. © 2024 IEEE.},
keywords = {Adversarial machine learning, AGI (Artificial General Intelligence), Artificial general intelligence, Artificial general intelligences, Autonomous decision, Autonomous Decision-Making, Data assimilation, Data integration, Decisions makings, Digital Twin Technology, Emotion Recognition, Generative adversarial networks, Generative AI large model, Generative AI Large Models, Large models, Metaverse, Metaverses, Model Acceleration, Model Compression, Multi agent systems, Multi-agent systems, Multi-modal data, Multi-Modal Data Integration, Multiagent systems (MASs), Reinforcement Learning, Reinforcement learnings, Spatio-temporal data},
pubstate = {published},
tppubtype = {inproceedings}
}
We explore how to enhance the computational processing performance for generative AI large models with autonomous decision-making in metaverse applications. We first introduce the relationship between AI large models and the Metaverse. We elaborate on the application scenarios of generative AI large models in Metaverse, including real-time weather simulation, embodied intelligence of agents, dynamic environment interaction, and user emotion recognition. We then propose the method of Multi-Dimensional Optimization Generation Framework (MDOGF) to improve computational processing performance. The experiment results show great improvement in computational processing performance. © 2024 IEEE.
Lee, L. -K.; Chan, E. H.; Tong, K. K. -L.; Wong, N. K. -H.; Wu, B. S. -Y.; Fung, Y. -C.; Fong, E. K. S.; Hou, U. Leong; Wu, N. -I.
Utilizing Virtual Reality and Generative AI Chatbot for Job Interview Simulations Proceedings Article
In: K.T., Chui; Y.K., Hui; D., Yang; L.-K., Lee; L.-P., Wong; B.L., Reynolds (Ed.): Proc. - Int. Symp. Educ. Technol., ISET, pp. 209–212, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-835036141-4 (ISBN).
Abstract | Links | BibTeX | Tags: chatbot, Chatbots, Computer interaction, Computer simulation languages, Generative adversarial networks, Generative AI, Hong-kong, Human computer interaction, ITS applications, Job interview simulation, Job interviews, Performance, Science graduates, User friendliness, Virtual environments, Virtual Reality
@inproceedings{lee_utilizing_2024,
title = {Utilizing Virtual Reality and Generative AI Chatbot for Job Interview Simulations},
author = {L. -K. Lee and E. H. Chan and K. K. -L. Tong and N. K. -H. Wong and B. S. -Y. Wu and Y. -C. Fung and E. K. S. Fong and U. Leong Hou and N. -I. Wu},
editor = {Chui K.T. and Hui Y.K. and Yang D. and Lee L.-K. and Wong L.-P. and Reynolds B.L.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85206582338&doi=10.1109%2fISET61814.2024.00048&partnerID=40&md5=c6986c0697792254e167e143b75f14c6},
doi = {10.1109/ISET61814.2024.00048},
isbn = {979-835036141-4 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - Int. Symp. Educ. Technol., ISET},
pages = {209–212},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Stress and anxiety experienced by interviewees, particularly fresh graduates, would significantly impact their performance in job interviews. Due to the increased affordability and user-friendliness of virtual reality (VR), VR has seen a surge in its application within the educational sector. This paper presents the design and implementation of a job interview simulation system, leveraging VR and a generative AI chatbot to provide an immersive environment for computer science graduates in Hong Kong. The system aims to help graduates practice and familiarize themselves with various real-world scenarios of a job interview in English, Mandarin, and Cantonese, tailored to the unique language requirements of Hong Kong's professional environment. The system comprises three core modules: a mock question and answer reading module, an AI speech analysis module, and a virtual interview module facilitated by the generative AI chatbot, ChatGPT. We anticipate that the proposed simulator will provide valuable insights to education practitioners on utilizing VR and generative AI for job interview training, extending beyond computer science graduates. © 2024 IEEE.},
keywords = {chatbot, Chatbots, Computer interaction, Computer simulation languages, Generative adversarial networks, Generative AI, Hong-kong, Human computer interaction, ITS applications, Job interview simulation, Job interviews, Performance, Science graduates, User friendliness, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
Stress and anxiety experienced by interviewees, particularly fresh graduates, would significantly impact their performance in job interviews. Due to the increased affordability and user-friendliness of virtual reality (VR), VR has seen a surge in its application within the educational sector. This paper presents the design and implementation of a job interview simulation system, leveraging VR and a generative AI chatbot to provide an immersive environment for computer science graduates in Hong Kong. The system aims to help graduates practice and familiarize themselves with various real-world scenarios of a job interview in English, Mandarin, and Cantonese, tailored to the unique language requirements of Hong Kong's professional environment. The system comprises three core modules: a mock question and answer reading module, an AI speech analysis module, and a virtual interview module facilitated by the generative AI chatbot, ChatGPT. We anticipate that the proposed simulator will provide valuable insights to education practitioners on utilizing VR and generative AI for job interview training, extending beyond computer science graduates. © 2024 IEEE.
Taheri, M.; Tan, K.
Enhancing Presentation Skills: A Virtual Reality-Based Simulator with Integrated Generative AI for Dynamic Pitch Presentations and Interviews Proceedings Article
In: L.T., De Paolis; P., Arpaia; M., Sacco (Ed.): Lect. Notes Comput. Sci., pp. 360–366, Springer Science and Business Media Deutschland GmbH, 2024, ISBN: 03029743 (ISSN); 978-303171706-2 (ISBN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, AI feedback, Contrastive Learning, Digital elevation model, Dynamic pitch, Federated learning, feedback, Generative adversarial networks, Iterative practice, Language Model, Open source language, Open source software, Presentation skills, Simulation Design, Spoken words, Trial and error, Virtual environments, Virtual reality based simulators
@inproceedings{taheri_enhancing_2024,
title = {Enhancing Presentation Skills: A Virtual Reality-Based Simulator with Integrated Generative AI for Dynamic Pitch Presentations and Interviews},
author = {M. Taheri and K. Tan},
editor = {De Paolis L.T. and Arpaia P. and Sacco M.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85204618832&doi=10.1007%2f978-3-031-71707-9_30&partnerID=40&md5=fd649ec5c0e2ce96593fe8a129e94449},
doi = {10.1007/978-3-031-71707-9_30},
isbn = {03029743 (ISSN); 978-303171706-2 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Lect. Notes Comput. Sci.},
volume = {15027 LNCS},
pages = {360–366},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {Presenting before an audience presents challenges throughout preparation and delivery, necessitating tools to securely refine skills securely. Interviews mirror presentations, showcasing oneself to convey qualifications. Virtual environments offer safe spaces for trial and error, crucial for iterative practice without emotional distress. This research proposes a Virtual Reality-Based Dynamic Pitch Simulation with Integrated Generative AI to effectively enhance presentation skills. The simulation converts spoken words to text, then uses AI to generate relevant questions for practice. Benefits include realistic feedback and adaptability to user proficiency. Open-source language models evaluate content, coherence, and delivery, offering personalized challenges. This approach supplements learning, enhancing presentation skills effectively. Voice-to-text conversion and AI feedback create a potent pedagogical tool, fostering a prompt feedback loop vital for learning effectiveness. Challenges in simulation design must be addressed for robustness and efficacy. The study validates these concepts by proposing a real-time 3D dialogue simulator, emphasizing the importance of continual improvement in presentation skill development. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.},
keywords = {Adversarial machine learning, AI feedback, Contrastive Learning, Digital elevation model, Dynamic pitch, Federated learning, feedback, Generative adversarial networks, Iterative practice, Language Model, Open source language, Open source software, Presentation skills, Simulation Design, Spoken words, Trial and error, Virtual environments, Virtual reality based simulators},
pubstate = {published},
tppubtype = {inproceedings}
}
Presenting before an audience presents challenges throughout preparation and delivery, necessitating tools to securely refine skills securely. Interviews mirror presentations, showcasing oneself to convey qualifications. Virtual environments offer safe spaces for trial and error, crucial for iterative practice without emotional distress. This research proposes a Virtual Reality-Based Dynamic Pitch Simulation with Integrated Generative AI to effectively enhance presentation skills. The simulation converts spoken words to text, then uses AI to generate relevant questions for practice. Benefits include realistic feedback and adaptability to user proficiency. Open-source language models evaluate content, coherence, and delivery, offering personalized challenges. This approach supplements learning, enhancing presentation skills effectively. Voice-to-text conversion and AI feedback create a potent pedagogical tool, fostering a prompt feedback loop vital for learning effectiveness. Challenges in simulation design must be addressed for robustness and efficacy. The study validates these concepts by proposing a real-time 3D dialogue simulator, emphasizing the importance of continual improvement in presentation skill development. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
Gujar, P.; Paliwal, G.; Panyam, S.
Generative AI and the Future of Interactive and Immersive Advertising Proceedings Article
In: D., Rivas-Lalaleo; S.L.S., Maita (Ed.): ETCM - Ecuador Tech. Chapters Meet., Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-835039158-9 (ISBN).
Abstract | Links | BibTeX | Tags: Ad Creation, Adversarial machine learning, Advertising Technology (AdTech), Advertizing, Advertizing technology, Augmented Reality, Augmented Reality (AR), Generative adversarial networks, Generative AI, Immersive, Immersive Advertising, Immersive advertizing, Interactive Advertising, Interactive advertizing, machine learning, Machine-learning, Marketing, Mixed reality, Mixed Reality (MR), Personalization, Personalizations, User Engagement, Virtual environments, Virtual Reality, Virtual Reality (VR)
@inproceedings{gujar_generative_2024,
title = {Generative AI and the Future of Interactive and Immersive Advertising},
author = {P. Gujar and G. Paliwal and S. Panyam},
editor = {Rivas-Lalaleo D. and Maita S.L.S.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85211805262&doi=10.1109%2fETCM63562.2024.10746166&partnerID=40&md5=179c5ceeb28ed72e809748322535c7ad},
doi = {10.1109/ETCM63562.2024.10746166},
isbn = {979-835039158-9 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {ETCM - Ecuador Tech. Chapters Meet.},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Generative AI is revolutionizing interactive and immersive advertising by enabling more personalized, engaging experiences through advanced technologies like VR, AR, and MR. This transformation is reshaping how advertisers create, deliver, and optimize content, allowing for two-way communication and blurring lines between digital and physical worlds. AI enhances user engagement through predictive analytics, real-time adaptation, and natural language processing, while also optimizing ad placement and personalization. Future trends include integration with emerging technologies like 5G and IoT, fully immersive experiences, and hyper-personalization. However, challenges such as privacy concerns, transparency issues, and ethical considerations must be addressed. As AI continues to evolve, it promises to create unprecedented opportunities for brands to connect with audiences in meaningful ways, potentially blurring the line between advertising and interactive entertainment. The industry must proactively address these challenges to ensure AI-driven advertising enhances user experiences while respecting privacy and maintaining trust. © 2024 IEEE.},
keywords = {Ad Creation, Adversarial machine learning, Advertising Technology (AdTech), Advertizing, Advertizing technology, Augmented Reality, Augmented Reality (AR), Generative adversarial networks, Generative AI, Immersive, Immersive Advertising, Immersive advertizing, Interactive Advertising, Interactive advertizing, machine learning, Machine-learning, Marketing, Mixed reality, Mixed Reality (MR), Personalization, Personalizations, User Engagement, Virtual environments, Virtual Reality, Virtual Reality (VR)},
pubstate = {published},
tppubtype = {inproceedings}
}
Generative AI is revolutionizing interactive and immersive advertising by enabling more personalized, engaging experiences through advanced technologies like VR, AR, and MR. This transformation is reshaping how advertisers create, deliver, and optimize content, allowing for two-way communication and blurring lines between digital and physical worlds. AI enhances user engagement through predictive analytics, real-time adaptation, and natural language processing, while also optimizing ad placement and personalization. Future trends include integration with emerging technologies like 5G and IoT, fully immersive experiences, and hyper-personalization. However, challenges such as privacy concerns, transparency issues, and ethical considerations must be addressed. As AI continues to evolve, it promises to create unprecedented opportunities for brands to connect with audiences in meaningful ways, potentially blurring the line between advertising and interactive entertainment. The industry must proactively address these challenges to ensure AI-driven advertising enhances user experiences while respecting privacy and maintaining trust. © 2024 IEEE.
Shrestha, A.; Imamoto, K.
Generative AI based industrial metaverse creation methodology Proceedings Article
In: Proc. - Artif. Intell. Bus., AIxB, pp. 53–57, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-835039103-9 (ISBN).
Abstract | Links | BibTeX | Tags: Generative adversarial networks, Generative AI, Industrial metaverse, Industrial railroads, Investments, Maintenance and operation, Metaverses, Natural languages, Railroad transportation, Railway, Railway maintenance, Railway operations, Simple++, simulation
@inproceedings{shrestha_generative_2024,
title = {Generative AI based industrial metaverse creation methodology},
author = {A. Shrestha and K. Imamoto},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85215066217&doi=10.1109%2fAIxB62249.2024.00017&partnerID=40&md5=d6d11729f16ccaa9f69fd5452befe492},
doi = {10.1109/AIxB62249.2024.00017},
isbn = {979-835039103-9 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - Artif. Intell. Bus., AIxB},
pages = {53–57},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {The metaverse has been proposed as a suitable apparatus for the dissemination of information in a railway maintenance and operation context. However, the generation of such a metaverse environment requires significant investment with the creation of simple prototypes taking an extended duration. Although there are generative artificial intelligencebased methods to create small scenes, there is an absence of a method to do so for industrial applications. We devised a platform to create railway environments with the assistance of the language models for code creation and semantic inference without the need for reprogramming or editing of the project source meaning environments could be generated by the end users. With a natural language input and a coding paradigm output the code generation module is shown together with the example environments from real-life railway lines in Tokyo, Japan as preliminary results. By creating such environments leveraging the rapid generation with the help of generative artificial intelligence, we show generative artificial intelligence can be used to automate the task of the programmer to create new environments on demand from the user in natural language. © 2024 IEEE.},
keywords = {Generative adversarial networks, Generative AI, Industrial metaverse, Industrial railroads, Investments, Maintenance and operation, Metaverses, Natural languages, Railroad transportation, Railway, Railway maintenance, Railway operations, Simple++, simulation},
pubstate = {published},
tppubtype = {inproceedings}
}
The metaverse has been proposed as a suitable apparatus for the dissemination of information in a railway maintenance and operation context. However, the generation of such a metaverse environment requires significant investment with the creation of simple prototypes taking an extended duration. Although there are generative artificial intelligencebased methods to create small scenes, there is an absence of a method to do so for industrial applications. We devised a platform to create railway environments with the assistance of the language models for code creation and semantic inference without the need for reprogramming or editing of the project source meaning environments could be generated by the end users. With a natural language input and a coding paradigm output the code generation module is shown together with the example environments from real-life railway lines in Tokyo, Japan as preliminary results. By creating such environments leveraging the rapid generation with the help of generative artificial intelligence, we show generative artificial intelligence can be used to automate the task of the programmer to create new environments on demand from the user in natural language. © 2024 IEEE.
Jayaraman, S.; Bhavya, R.; Srihari, V.; Rajam, V. Mary Anita
TexAVi: Generating Stereoscopic VR Video Clips from Text Descriptions Proceedings Article
In: IEEE Int. Conf. Comput. Vis. Mach. Intell., CVMI, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-835037687-6 (ISBN).
Abstract | Links | BibTeX | Tags: Adversarial networks, Computer simulation languages, Deep learning, Depth Estimation, Depth perception, Diffusion Model, diffusion models, Digital elevation model, Generative adversarial networks, Generative model, Generative systems, Language Model, Motion capture, Stereo image processing, Text-to-image, Training data, Video analysis, Video-clips, Virtual environments, Virtual Reality
@inproceedings{jayaraman_texavi_2024,
title = {TexAVi: Generating Stereoscopic VR Video Clips from Text Descriptions},
author = {S. Jayaraman and R. Bhavya and V. Srihari and V. Mary Anita Rajam},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85215265234&doi=10.1109%2fCVMI61877.2024.10782691&partnerID=40&md5=8e20576af67b917ecfad83873a87ef29},
doi = {10.1109/CVMI61877.2024.10782691},
isbn = {979-835037687-6 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {IEEE Int. Conf. Comput. Vis. Mach. Intell., CVMI},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {While generative models such as text-to-image, large language models and text-to-video have seen significant progress, the extension to text-to-virtual-reality remains largely unexplored, due to a deficit in training data and the complexity of achieving realistic depth and motion in virtual environments. This paper proposes an approach to coalesce existing generative systems to form a stereoscopic virtual reality video from text. Carried out in three main stages, we start with a base text-to-image model that captures context from an input text. We then employ Stable Diffusion on the rudimentary image produced, to generate frames with enhanced realism and overall quality. These frames are processed with depth estimation algorithms to create left-eye and right-eye views, which are stitched side-by-side to create an immersive viewing experience. Such systems would be highly beneficial in virtual reality production, since filming and scene building often require extensive hours of work and post-production effort. We utilize image evaluation techniques, specifically Fréchet Inception Distance and CLIP Score, to assess the visual quality of frames produced for the video. These quantitative measures establish the proficiency of the proposed method. Our work highlights the exciting possibilities of using natural language-driven graphics in fields like virtual reality simulations. © 2024 IEEE.},
keywords = {Adversarial networks, Computer simulation languages, Deep learning, Depth Estimation, Depth perception, Diffusion Model, diffusion models, Digital elevation model, Generative adversarial networks, Generative model, Generative systems, Language Model, Motion capture, Stereo image processing, Text-to-image, Training data, Video analysis, Video-clips, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
While generative models such as text-to-image, large language models and text-to-video have seen significant progress, the extension to text-to-virtual-reality remains largely unexplored, due to a deficit in training data and the complexity of achieving realistic depth and motion in virtual environments. This paper proposes an approach to coalesce existing generative systems to form a stereoscopic virtual reality video from text. Carried out in three main stages, we start with a base text-to-image model that captures context from an input text. We then employ Stable Diffusion on the rudimentary image produced, to generate frames with enhanced realism and overall quality. These frames are processed with depth estimation algorithms to create left-eye and right-eye views, which are stitched side-by-side to create an immersive viewing experience. Such systems would be highly beneficial in virtual reality production, since filming and scene building often require extensive hours of work and post-production effort. We utilize image evaluation techniques, specifically Fréchet Inception Distance and CLIP Score, to assess the visual quality of frames produced for the video. These quantitative measures establish the proficiency of the proposed method. Our work highlights the exciting possibilities of using natural language-driven graphics in fields like virtual reality simulations. © 2024 IEEE.
Velev, D.; Steshina, L.; Petukhov, I.; Zlateva, P.
Challenges of Merging Generative AI with Metaverse for Next-Gen Education Proceedings Article
In: A.J., Tallon-Ballesteros (Ed.): Front. Artif. Intell. Appl., pp. 606–616, IOS Press BV, 2024, ISBN: 09226389 (ISSN); 978-164368569-4 (ISBN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, Augmented Reality, Contrastive Learning, Data privacy and securities, Digital literacies, Education, Federated learning, Generative adversarial networks, Generative AI, High speed internet, Instructional designs, Learning Environments, Metaverse, Metaverses, Personalized learning, Realtime processing, Teaching methods, Virtual environments, Virtual Reality
@inproceedings{velev_challenges_2024,
title = {Challenges of Merging Generative AI with Metaverse for Next-Gen Education},
author = {D. Velev and L. Steshina and I. Petukhov and P. Zlateva},
editor = {Tallon-Ballesteros A.J.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85215823646&doi=10.3233%2fFAIA241462&partnerID=40&md5=a3ed4e8486e2e32d0856a71a3a87496c},
doi = {10.3233/FAIA241462},
isbn = {09226389 (ISSN); 978-164368569-4 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Front. Artif. Intell. Appl.},
volume = {398},
pages = {606–616},
publisher = {IOS Press BV},
abstract = {The integration of Generative Artificial Intelligence (GenAI) with the Metaverse for a next-generation education is a complex but challenging task. The GenAI-enhanced Metaverse classrooms require innovative instructional designs that use virtual reality and augmented reality to enhance engagement and personalized learning. Educators must adapt to new roles over traditional teaching methods, while learners need to develop digital literacy skills that are essential for navigating and inhabiting in these environments. Such learning environments require significant advancements in real-time processing, scalability and interoperability of different platforms, while ensuring data privacy and security. The equity of access to high-speed internet and advanced devices still remains a serious barrier, which can increase the potential existing inequalities between different educational environments. Ethical considerations, including the responsible use of GenAI, the creation of unbiased educational content, and the psychological impacts of extended usage of virtual reality, are also of important consideration. The aim of the paper is to explore in detail the different challenges through a comprehensive analysis of the obstacles and potential solutions and to propose a collaborative framework involving educators, technologists, policymakers and industry stakeholders to address the effective implementation of the integration of GenAI and the Metaverse for a next generation education. © 2024 The Authors.},
keywords = {Adversarial machine learning, Augmented Reality, Contrastive Learning, Data privacy and securities, Digital literacies, Education, Federated learning, Generative adversarial networks, Generative AI, High speed internet, Instructional designs, Learning Environments, Metaverse, Metaverses, Personalized learning, Realtime processing, Teaching methods, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
The integration of Generative Artificial Intelligence (GenAI) with the Metaverse for a next-generation education is a complex but challenging task. The GenAI-enhanced Metaverse classrooms require innovative instructional designs that use virtual reality and augmented reality to enhance engagement and personalized learning. Educators must adapt to new roles over traditional teaching methods, while learners need to develop digital literacy skills that are essential for navigating and inhabiting in these environments. Such learning environments require significant advancements in real-time processing, scalability and interoperability of different platforms, while ensuring data privacy and security. The equity of access to high-speed internet and advanced devices still remains a serious barrier, which can increase the potential existing inequalities between different educational environments. Ethical considerations, including the responsible use of GenAI, the creation of unbiased educational content, and the psychological impacts of extended usage of virtual reality, are also of important consideration. The aim of the paper is to explore in detail the different challenges through a comprehensive analysis of the obstacles and potential solutions and to propose a collaborative framework involving educators, technologists, policymakers and industry stakeholders to address the effective implementation of the integration of GenAI and the Metaverse for a next generation education. © 2024 The Authors.
Rahmani, R.; Westin, T.; Nevelsteen, K.
Future Healthcare in Generative AI with Real Metaverse Proceedings Article
In: E.E., Shakshuki (Ed.): Procedia Comput. Sci., pp. 487–493, Elsevier B.V., 2024, ISBN: 18770509 (ISSN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, AI, Augmented Reality, Autism spectrum disorders, Contrastive Learning, Diseases, Edge Intelligence, Generative adversarial networks, Healthcare, Immersive learning, Independent living systems, Language Model, Large language model, LLM, Metaverses, Posttraumatic stress disorder, Real Metaverse, Social challenges, Virtual environments
@inproceedings{rahmani_future_2024,
title = {Future Healthcare in Generative AI with Real Metaverse},
author = {R. Rahmani and T. Westin and K. Nevelsteen},
editor = {Shakshuki E.E.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85214986921&doi=10.1016%2fj.procs.2024.11.137&partnerID=40&md5=3e25f2a1b023cd49f59a066a96bb2dd0},
doi = {10.1016/j.procs.2024.11.137},
isbn = {18770509 (ISSN)},
year = {2024},
date = {2024-01-01},
booktitle = {Procedia Comput. Sci.},
volume = {251},
pages = {487–493},
publisher = {Elsevier B.V.},
abstract = {The Metaverse offers a simulated environment that could transform healthcare by providing immersive learning experiences through Internet application and social form that integrates network of virtual reality environments. The Metaverse is expected to contribute to a new way of socializing, where users can enter a verse as avatars. The concept allows avatars to switch between verses seamlessly. Virtual Reality (VR) in healthcare has shown promise for social-skill training, especially for individuals with Autism Spectrum Disorder (ASD) and social challenge training of patients with Post-Traumatic Stress Disorder (PTSD) requiring adaptable support. The problem lies in the limited adaptability and functionality of existing Metaverse implementations for individuals with ASD and PTSD. While studies have explored various implementation ideas, such as VR platforms for training social skills, social challenge and context-aware Augmented Reality (AR) systems for daily activities, many lack adaptability of user input and output. A proposed solution involves a context-aware system using AI, Large Language Models (LLMs) and generative agents to support independent living for individuals with ASD and a tool to enhance emotional learning with PTSD. © 2024 The Authors.},
keywords = {Adversarial machine learning, AI, Augmented Reality, Autism spectrum disorders, Contrastive Learning, Diseases, Edge Intelligence, Generative adversarial networks, Healthcare, Immersive learning, Independent living systems, Language Model, Large language model, LLM, Metaverses, Posttraumatic stress disorder, Real Metaverse, Social challenges, Virtual environments},
pubstate = {published},
tppubtype = {inproceedings}
}
The Metaverse offers a simulated environment that could transform healthcare by providing immersive learning experiences through Internet application and social form that integrates network of virtual reality environments. The Metaverse is expected to contribute to a new way of socializing, where users can enter a verse as avatars. The concept allows avatars to switch between verses seamlessly. Virtual Reality (VR) in healthcare has shown promise for social-skill training, especially for individuals with Autism Spectrum Disorder (ASD) and social challenge training of patients with Post-Traumatic Stress Disorder (PTSD) requiring adaptable support. The problem lies in the limited adaptability and functionality of existing Metaverse implementations for individuals with ASD and PTSD. While studies have explored various implementation ideas, such as VR platforms for training social skills, social challenge and context-aware Augmented Reality (AR) systems for daily activities, many lack adaptability of user input and output. A proposed solution involves a context-aware system using AI, Large Language Models (LLMs) and generative agents to support independent living for individuals with ASD and a tool to enhance emotional learning with PTSD. © 2024 The Authors.
Paweroi, R. M.; Koppen, M.
Framework for Integration of Generative AI into Metaverse Asset Creation Proceedings Article
In: Int. Conf. Intell. Metaverse Technol. Appl., iMETA, pp. 27–33, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-835035151-4 (ISBN).
Abstract | Links | BibTeX | Tags: 3D Asset Creation, 3D Asset Diversity, 3D models, 3d-modeling, Digital assets, Digital Objects, Generative adversarial networks, Generative AI, High quality, Metaverse, Metaverses, Virtual worlds
@inproceedings{paweroi_framework_2024,
title = {Framework for Integration of Generative AI into Metaverse Asset Creation},
author = {R. M. Paweroi and M. Koppen},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85216024340&doi=10.1109%2fiMETA62882.2024.10808057&partnerID=40&md5=00373291c3d224b53759dc39ed9fd65c},
doi = {10.1109/iMETA62882.2024.10808057},
isbn = {979-835035151-4 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Int. Conf. Intell. Metaverse Technol. Appl., iMETA},
pages = {27–33},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Metaverse, a virtual world, is developing rapidly and is widely used in multi-sector. The number of users is projected to increase year over year. Due to the development of the metaverse platform, digital asset creation is demanding. Creating high-quality and diverse 3D digital objects is challenging. This study proposes the frameworks for integrating generative AI to create diverse 3D assets into the metaverse. We study different approaches for asset creation, i.e., generative 3D model-based, generative image projection-based, and generative language script-based. Creators can use this workflow to optimize the creation of 3D assets. Moreover, this study compares the results of generative AI and procedural generation on generating diverse 3D objects. The result shows that generative AI can simplify 3D creation and generate more diverse objects. © 2024 IEEE.},
keywords = {3D Asset Creation, 3D Asset Diversity, 3D models, 3d-modeling, Digital assets, Digital Objects, Generative adversarial networks, Generative AI, High quality, Metaverse, Metaverses, Virtual worlds},
pubstate = {published},
tppubtype = {inproceedings}
}
Metaverse, a virtual world, is developing rapidly and is widely used in multi-sector. The number of users is projected to increase year over year. Due to the development of the metaverse platform, digital asset creation is demanding. Creating high-quality and diverse 3D digital objects is challenging. This study proposes the frameworks for integrating generative AI to create diverse 3D assets into the metaverse. We study different approaches for asset creation, i.e., generative 3D model-based, generative image projection-based, and generative language script-based. Creators can use this workflow to optimize the creation of 3D assets. Moreover, this study compares the results of generative AI and procedural generation on generating diverse 3D objects. The result shows that generative AI can simplify 3D creation and generate more diverse objects. © 2024 IEEE.
Qin, X.; Weaver, G.
Utilizing Generative AI for VR Exploration Testing: A Case Study Proceedings Article
In: Proc. - ACM/IEEE Int. Conf. Autom. Softw. Eng. Workshops, ASEW, pp. 228–232, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-840071249-4 (ISBN).
Abstract | Links | BibTeX | Tags: Ability testing, Accuracy rate, Case Study, Case-studies, Entity selections, Field of views, Generative adversarial networks, GUI Exploration Testing, GUI testing, Localisation, Long term memory, Mixed data, Object identification, Object recognition, Virtual environments, Virtual Reality
@inproceedings{qin_utilizing_2024,
title = {Utilizing Generative AI for VR Exploration Testing: A Case Study},
author = {X. Qin and G. Weaver},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85213332710&doi=10.1145%2f3691621.3694955&partnerID=40&md5=8f3dc03520214cd2e270ed41a0fc0e19},
doi = {10.1145/3691621.3694955},
isbn = {979-840071249-4 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - ACM/IEEE Int. Conf. Autom. Softw. Eng. Workshops, ASEW},
pages = {228–232},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {As the virtual reality (VR) industry expands, the need for automated GUI testing for applications is growing rapidly. With its long-term memory and ability to process mixed data, including images and text, Generative AI (GenAI) shows the potential to understand complex user interfaces. In this paper, we conduct a case study to investigate the potential of using GenAI for field of view (FOV) analysis in VR exploration testing. Specifically, we examine how the model can assist in test entity selection and test action suggestions. Our experiments demonstrate that while GPT-4o achieves a 63% accuracy rate in object identification within an arbitrary FOV, it struggles with object organization and localization. We also identify critical contexts that can improve the accuracy of suggested actions across multiple FOVs. Finally, we discuss the limitations found during the experiment and offer insights into future research directions. © 2024 ACM.},
keywords = {Ability testing, Accuracy rate, Case Study, Case-studies, Entity selections, Field of views, Generative adversarial networks, GUI Exploration Testing, GUI testing, Localisation, Long term memory, Mixed data, Object identification, Object recognition, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
As the virtual reality (VR) industry expands, the need for automated GUI testing for applications is growing rapidly. With its long-term memory and ability to process mixed data, including images and text, Generative AI (GenAI) shows the potential to understand complex user interfaces. In this paper, we conduct a case study to investigate the potential of using GenAI for field of view (FOV) analysis in VR exploration testing. Specifically, we examine how the model can assist in test entity selection and test action suggestions. Our experiments demonstrate that while GPT-4o achieves a 63% accuracy rate in object identification within an arbitrary FOV, it struggles with object organization and localization. We also identify critical contexts that can improve the accuracy of suggested actions across multiple FOVs. Finally, we discuss the limitations found during the experiment and offer insights into future research directions. © 2024 ACM.