AHCI RESEARCH GROUP
Publications
Papers published in international journals,
proceedings of conferences, workshops and books.
OUR RESEARCH
Scientific Publications
How to
Here you can find the complete list of our publications.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
2025
Tracy, K.; Spantidi, O.
Impact of GPT-Driven Teaching Assistants in VR Learning Environments Journal Article
In: IEEE Transactions on Learning Technologies, vol. 18, pp. 192–205, 2025, ISSN: 19391382 (ISSN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, Cognitive loads, Computer interaction, Contrastive Learning, Control groups, Experimental groups, Federated learning, Generative AI, Generative artificial intelligence (GenAI), human–computer interaction, Interactive learning environment, interactive learning environments, Learning efficacy, Learning outcome, learning outcomes, Student engagement, Teaching assistants, Virtual environments, Virtual Reality (VR)
@article{tracy_impact_2025,
title = {Impact of GPT-Driven Teaching Assistants in VR Learning Environments},
author = {K. Tracy and O. Spantidi},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105001083336&doi=10.1109%2fTLT.2025.3539179&partnerID=40&md5=34fea4ea8517a061fe83b8294e1a9a87},
doi = {10.1109/TLT.2025.3539179},
issn = {19391382 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Transactions on Learning Technologies},
volume = {18},
pages = {192–205},
abstract = {Virtual reality (VR) has emerged as a transformative educational tool, enabling immersive learning environments that promote student engagement and understanding of complex concepts. However, despite the growing adoption of VR in education, there remains a significant gap in research exploring how generative artificial intelligence (AI), such as generative pretrained transformer can further enhance these experiences by reducing cognitive load and improving learning outcomes. This study examines the impact of an AI-driven instructor assistant in VR classrooms on student engagement, cognitive load, knowledge retention, and performance. A total of 52 participants were divided into two groups experiencing a VR lesson on the bubble sort algorithm, one with only a prescripted virtual instructor (control group), and the other with the addition of an AI instructor assistant (experimental group). Statistical analysis of postlesson quizzes and cognitive load assessments was conducted using independent t-tests and analysis of variance (ANOVA), with the cognitive load being measured through a postexperiment questionnaire. The study results indicate that the experimental group reported significantly higher engagement compared to the control group. While the AI assistant did not significantly improve postlesson assessment scores, it enhanced conceptual knowledge transfer. The experimental group also demonstrated lower intrinsic cognitive load, suggesting the assistant reduced the perceived complexity of the material. Higher germane and general cognitive loads indicated that students were more invested in meaningful learning without feeling overwhelmed. © 2008-2011 IEEE.},
keywords = {Adversarial machine learning, Cognitive loads, Computer interaction, Contrastive Learning, Control groups, Experimental groups, Federated learning, Generative AI, Generative artificial intelligence (GenAI), human–computer interaction, Interactive learning environment, interactive learning environments, Learning efficacy, Learning outcome, learning outcomes, Student engagement, Teaching assistants, Virtual environments, Virtual Reality (VR)},
pubstate = {published},
tppubtype = {article}
}
Virtual reality (VR) has emerged as a transformative educational tool, enabling immersive learning environments that promote student engagement and understanding of complex concepts. However, despite the growing adoption of VR in education, there remains a significant gap in research exploring how generative artificial intelligence (AI), such as generative pretrained transformer can further enhance these experiences by reducing cognitive load and improving learning outcomes. This study examines the impact of an AI-driven instructor assistant in VR classrooms on student engagement, cognitive load, knowledge retention, and performance. A total of 52 participants were divided into two groups experiencing a VR lesson on the bubble sort algorithm, one with only a prescripted virtual instructor (control group), and the other with the addition of an AI instructor assistant (experimental group). Statistical analysis of postlesson quizzes and cognitive load assessments was conducted using independent t-tests and analysis of variance (ANOVA), with the cognitive load being measured through a postexperiment questionnaire. The study results indicate that the experimental group reported significantly higher engagement compared to the control group. While the AI assistant did not significantly improve postlesson assessment scores, it enhanced conceptual knowledge transfer. The experimental group also demonstrated lower intrinsic cognitive load, suggesting the assistant reduced the perceived complexity of the material. Higher germane and general cognitive loads indicated that students were more invested in meaningful learning without feeling overwhelmed. © 2008-2011 IEEE.
Linares-Pellicer, J.; Izquierdo-Domenech, J.; Ferri-Molla, I.; Aliaga-Torro, C.
Breaking the Bottleneck: Generative AI as the Solution for XR Content Creation in Education Book Section
In: Lecture Notes in Networks and Systems, vol. 1140, pp. 9–30, Springer Science and Business Media Deutschland GmbH, 2025, ISBN: 23673370 (ISSN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, Augmented Reality, Breakings, Content creation, Contrastive Learning, Development process, Educational context, Federated learning, Generative adversarial networks, Immersive learning, Intelligence models, Learning experiences, Mixed reality, Resource intensity, Technical skills, Virtual environments
@incollection{linares-pellicer_breaking_2025,
title = {Breaking the Bottleneck: Generative AI as the Solution for XR Content Creation in Education},
author = {J. Linares-Pellicer and J. Izquierdo-Domenech and I. Ferri-Molla and C. Aliaga-Torro},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85212478399&doi=10.1007%2f978-3-031-71530-3_2&partnerID=40&md5=aefee938cd5b8a74ee811a463d7409ae},
doi = {10.1007/978-3-031-71530-3_2},
isbn = {23673370 (ISSN)},
year = {2025},
date = {2025-01-01},
booktitle = {Lecture Notes in Networks and Systems},
volume = {1140},
pages = {9–30},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {The integration of Extended Reality (XR) technologies-Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR)-promises to revolutionize education by offering immersive learning experiences. However, the complexity and resource intensity of content creation hinders the adoption of XR in educational contexts. This chapter explores Generative Artificial Intelligence (GenAI) as a solution, highlighting how GenAI models can facilitate the creation of educational XR content. GenAI enables educators to produce engaging XR experiences without needing advanced technical skills by automating aspects of the development process from ideation to deployment. Practical examples demonstrate GenAI’s current capability to generate assets and program applications, significantly lowering the barrier to creating personalized and interactive learning environments. The chapter also addresses challenges related to GenAI’s application in education, including technical limitations and ethical considerations. Ultimately, GenAI’s integration into XR content creation makes immersive educational experiences more accessible and practical, driven by only natural interactions, promising a future where technology-enhanced learning is universally attainable. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.},
keywords = {Adversarial machine learning, Augmented Reality, Breakings, Content creation, Contrastive Learning, Development process, Educational context, Federated learning, Generative adversarial networks, Immersive learning, Intelligence models, Learning experiences, Mixed reality, Resource intensity, Technical skills, Virtual environments},
pubstate = {published},
tppubtype = {incollection}
}
The integration of Extended Reality (XR) technologies-Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR)-promises to revolutionize education by offering immersive learning experiences. However, the complexity and resource intensity of content creation hinders the adoption of XR in educational contexts. This chapter explores Generative Artificial Intelligence (GenAI) as a solution, highlighting how GenAI models can facilitate the creation of educational XR content. GenAI enables educators to produce engaging XR experiences without needing advanced technical skills by automating aspects of the development process from ideation to deployment. Practical examples demonstrate GenAI’s current capability to generate assets and program applications, significantly lowering the barrier to creating personalized and interactive learning environments. The chapter also addresses challenges related to GenAI’s application in education, including technical limitations and ethical considerations. Ultimately, GenAI’s integration into XR content creation makes immersive educational experiences more accessible and practical, driven by only natural interactions, promising a future where technology-enhanced learning is universally attainable. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
Barbu, M.; Iordache, D. -D.; Petre, I.; Barbu, D. -C.; Băjenaru, L.
Framework Design for Reinforcing the Potential of XR Technologies in Transforming Inclusive Education Journal Article
In: Applied Sciences (Switzerland), vol. 15, no. 3, 2025, ISSN: 20763417 (ISSN).
Abstract | Links | BibTeX | Tags: Adaptive Learning, Adversarial machine learning, Artificial intelligence technologies, Augmented Reality, Contrastive Learning, Educational Technology, Extended reality (XR), Federated learning, Framework designs, Generative adversarial networks, Immersive, immersive experience, Immersive learning, Inclusive education, Learning platform, Special education needs
@article{barbu_framework_2025,
title = {Framework Design for Reinforcing the Potential of XR Technologies in Transforming Inclusive Education},
author = {M. Barbu and D. -D. Iordache and I. Petre and D. -C. Barbu and L. Băjenaru},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85217742383&doi=10.3390%2fapp15031484&partnerID=40&md5=3148ff2a8a8fa1bef8094199cd6d32e3},
doi = {10.3390/app15031484},
issn = {20763417 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {Applied Sciences (Switzerland)},
volume = {15},
number = {3},
abstract = {This study presents a novel approach to inclusive education by integrating augmented reality (XR) and generative artificial intelligence (AI) technologies into an immersive and adaptive learning platform designed for students with special educational needs. Building upon existing solutions, the approach uniquely combines XR and generative AI to facilitate personalized, accessible, and interactive learning experiences tailored to individual requirements. The framework incorporates an intuitive Unity XR-based interface alongside a generative AI module to enable near real-time customization of content and interactions. Additionally, the study examines related generative AI initiatives that promote inclusion through enhanced communication tools, educational support, and customizable assistive technologies. The motivation for this study arises from the pressing need to address the limitations of traditional educational methods, which often fail to meet the diverse needs of learners with special educational requirements. The integration of XR and generative AI offers transformative potential by creating adaptive, immersive, and inclusive learning environments. This approach ensures real-time adaptability to individual progress and accessibility, addressing critical barriers such as static content and lack of inclusivity in existing systems. The research outlines a pathway toward more inclusive and equitable education, significantly enhancing opportunities for learners with diverse needs and contributing to broader social integration and equity in education. © 2025 by the authors.},
keywords = {Adaptive Learning, Adversarial machine learning, Artificial intelligence technologies, Augmented Reality, Contrastive Learning, Educational Technology, Extended reality (XR), Federated learning, Framework designs, Generative adversarial networks, Immersive, immersive experience, Immersive learning, Inclusive education, Learning platform, Special education needs},
pubstate = {published},
tppubtype = {article}
}
This study presents a novel approach to inclusive education by integrating augmented reality (XR) and generative artificial intelligence (AI) technologies into an immersive and adaptive learning platform designed for students with special educational needs. Building upon existing solutions, the approach uniquely combines XR and generative AI to facilitate personalized, accessible, and interactive learning experiences tailored to individual requirements. The framework incorporates an intuitive Unity XR-based interface alongside a generative AI module to enable near real-time customization of content and interactions. Additionally, the study examines related generative AI initiatives that promote inclusion through enhanced communication tools, educational support, and customizable assistive technologies. The motivation for this study arises from the pressing need to address the limitations of traditional educational methods, which often fail to meet the diverse needs of learners with special educational requirements. The integration of XR and generative AI offers transformative potential by creating adaptive, immersive, and inclusive learning environments. This approach ensures real-time adaptability to individual progress and accessibility, addressing critical barriers such as static content and lack of inclusivity in existing systems. The research outlines a pathway toward more inclusive and equitable education, significantly enhancing opportunities for learners with diverse needs and contributing to broader social integration and equity in education. © 2025 by the authors.
2024
Fostering Personalized Learning in Data Science: Integrating Innovative Tools and Strategies for Diverse Pathways Proceedings Article
In: IEEE Int. Conf. Eng. Educ.: Dissem. Adv. Eng. Educ. using Artif. Intell., ICEED, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-835036741-6 (ISBN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, ChatGPT-4, Content recommendation, Content recommendations, Contrastive Learning, Data Science, Data science education, Federated learning, Individualized learning, Individualized learning experience framework, Learning experiences, Prerequisite skill identification, Science education, Self-directed learning, Teaching approaches, Virtual environments, Virtual Reality
@inproceedings{noauthor_fostering_2024,
title = {Fostering Personalized Learning in Data Science: Integrating Innovative Tools and Strategies for Diverse Pathways},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105001849041&doi=10.1109%2fICEED62316.2024.10923798&partnerID=40&md5=cfec507f601df5ffc3b07db0df6d80a7},
doi = {10.1109/ICEED62316.2024.10923798},
isbn = {979-835036741-6 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {IEEE Int. Conf. Eng. Educ.: Dissem. Adv. Eng. Educ. using Artif. Intell., ICEED},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {This paper introduces an innovative teaching approach in data science tailored for students in non-computer science pathways, specifically Business Information Technology (BIT) and Computing and Information Technology (CIT). Over a five-year period, a unique teaching approach has been developed incorporating a virtual reality (VR) game event and ChatGPT-4 as a generative artificial intelligence (AI) tool. To address the inherent complexities of learning data science, particularly the diverse prerequisite skills, this study introduces a framework including a diagnostic assessment centered around a specific education research question: 'How can the learning experiences of individual students be customized to address the multifaceted challenges of data science education?' Through a diagnostic assessment process, conducted via a survey completed by students, this framework identifies students' unique requirements and skill areas facilitating the delivery of personalized content recommendations within the initial week of teaching. By fostering a culture of self-directed learning, the approach aims to enable students to concentrate on essential customized learning materials. This paper also highlights the overall student satisfaction with the module averaged 4.5 out of 5 with a standard deviation of 0.9 indicating a high level of contentment with the teaching approach. The discussion encompasses the framework's implications for teaching and its alignment with educational theories. This paper contributes to the computing education field by addressing the research question and offering insights for future research and teaching practices. © 2024 IEEE.},
keywords = {Adversarial machine learning, ChatGPT-4, Content recommendation, Content recommendations, Contrastive Learning, Data Science, Data science education, Federated learning, Individualized learning, Individualized learning experience framework, Learning experiences, Prerequisite skill identification, Science education, Self-directed learning, Teaching approaches, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper introduces an innovative teaching approach in data science tailored for students in non-computer science pathways, specifically Business Information Technology (BIT) and Computing and Information Technology (CIT). Over a five-year period, a unique teaching approach has been developed incorporating a virtual reality (VR) game event and ChatGPT-4 as a generative artificial intelligence (AI) tool. To address the inherent complexities of learning data science, particularly the diverse prerequisite skills, this study introduces a framework including a diagnostic assessment centered around a specific education research question: 'How can the learning experiences of individual students be customized to address the multifaceted challenges of data science education?' Through a diagnostic assessment process, conducted via a survey completed by students, this framework identifies students' unique requirements and skill areas facilitating the delivery of personalized content recommendations within the initial week of teaching. By fostering a culture of self-directed learning, the approach aims to enable students to concentrate on essential customized learning materials. This paper also highlights the overall student satisfaction with the module averaged 4.5 out of 5 with a standard deviation of 0.9 indicating a high level of contentment with the teaching approach. The discussion encompasses the framework's implications for teaching and its alignment with educational theories. This paper contributes to the computing education field by addressing the research question and offering insights for future research and teaching practices. © 2024 IEEE.
Ding, P.; Liu, J.; Sun, M.; Li, L.; Liu, H.
Enhancing Computational Processing Performance for Generative AI Large Models with Autonomous Decision-Making in Metaverse Applications Proceedings Article
In: Proc. - IEEE Int. Conf. Metaverse Comput., Netw., Appl., MetaCom, pp. 253–258, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-833151599-7 (ISBN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, AGI (Artificial General Intelligence), Artificial general intelligence, Artificial general intelligences, Autonomous decision, Autonomous Decision-Making, Data assimilation, Data integration, Decisions makings, Digital Twin Technology, Emotion Recognition, Generative adversarial networks, Generative AI large model, Generative AI Large Models, Large models, Metaverse, Metaverses, Model Acceleration, Model Compression, Multi agent systems, Multi-agent systems, Multi-modal data, Multi-Modal Data Integration, Multiagent systems (MASs), Reinforcement Learning, Reinforcement learnings, Spatio-temporal data
@inproceedings{ding_enhancing_2024,
title = {Enhancing Computational Processing Performance for Generative AI Large Models with Autonomous Decision-Making in Metaverse Applications},
author = {P. Ding and J. Liu and M. Sun and L. Li and H. Liu},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85211489063&doi=10.1109%2fMetaCom62920.2024.00048&partnerID=40&md5=ae085a7d90b12c9090f5bf7a274bc7ce},
doi = {10.1109/MetaCom62920.2024.00048},
isbn = {979-833151599-7 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - IEEE Int. Conf. Metaverse Comput., Netw., Appl., MetaCom},
pages = {253–258},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {We explore how to enhance the computational processing performance for generative AI large models with autonomous decision-making in metaverse applications. We first introduce the relationship between AI large models and the Metaverse. We elaborate on the application scenarios of generative AI large models in Metaverse, including real-time weather simulation, embodied intelligence of agents, dynamic environment interaction, and user emotion recognition. We then propose the method of Multi-Dimensional Optimization Generation Framework (MDOGF) to improve computational processing performance. The experiment results show great improvement in computational processing performance. © 2024 IEEE.},
keywords = {Adversarial machine learning, AGI (Artificial General Intelligence), Artificial general intelligence, Artificial general intelligences, Autonomous decision, Autonomous Decision-Making, Data assimilation, Data integration, Decisions makings, Digital Twin Technology, Emotion Recognition, Generative adversarial networks, Generative AI large model, Generative AI Large Models, Large models, Metaverse, Metaverses, Model Acceleration, Model Compression, Multi agent systems, Multi-agent systems, Multi-modal data, Multi-Modal Data Integration, Multiagent systems (MASs), Reinforcement Learning, Reinforcement learnings, Spatio-temporal data},
pubstate = {published},
tppubtype = {inproceedings}
}
We explore how to enhance the computational processing performance for generative AI large models with autonomous decision-making in metaverse applications. We first introduce the relationship between AI large models and the Metaverse. We elaborate on the application scenarios of generative AI large models in Metaverse, including real-time weather simulation, embodied intelligence of agents, dynamic environment interaction, and user emotion recognition. We then propose the method of Multi-Dimensional Optimization Generation Framework (MDOGF) to improve computational processing performance. The experiment results show great improvement in computational processing performance. © 2024 IEEE.
Taheri, M.; Tan, K.
Enhancing Presentation Skills: A Virtual Reality-Based Simulator with Integrated Generative AI for Dynamic Pitch Presentations and Interviews Proceedings Article
In: L.T., De Paolis; P., Arpaia; M., Sacco (Ed.): Lect. Notes Comput. Sci., pp. 360–366, Springer Science and Business Media Deutschland GmbH, 2024, ISBN: 03029743 (ISSN); 978-303171706-2 (ISBN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, AI feedback, Contrastive Learning, Digital elevation model, Dynamic pitch, Federated learning, feedback, Generative adversarial networks, Iterative practice, Language Model, Open source language, Open source software, Presentation skills, Simulation Design, Spoken words, Trial and error, Virtual environments, Virtual reality based simulators
@inproceedings{taheri_enhancing_2024,
title = {Enhancing Presentation Skills: A Virtual Reality-Based Simulator with Integrated Generative AI for Dynamic Pitch Presentations and Interviews},
author = {M. Taheri and K. Tan},
editor = {De Paolis L.T. and Arpaia P. and Sacco M.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85204618832&doi=10.1007%2f978-3-031-71707-9_30&partnerID=40&md5=fd649ec5c0e2ce96593fe8a129e94449},
doi = {10.1007/978-3-031-71707-9_30},
isbn = {03029743 (ISSN); 978-303171706-2 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Lect. Notes Comput. Sci.},
volume = {15027 LNCS},
pages = {360–366},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {Presenting before an audience presents challenges throughout preparation and delivery, necessitating tools to securely refine skills securely. Interviews mirror presentations, showcasing oneself to convey qualifications. Virtual environments offer safe spaces for trial and error, crucial for iterative practice without emotional distress. This research proposes a Virtual Reality-Based Dynamic Pitch Simulation with Integrated Generative AI to effectively enhance presentation skills. The simulation converts spoken words to text, then uses AI to generate relevant questions for practice. Benefits include realistic feedback and adaptability to user proficiency. Open-source language models evaluate content, coherence, and delivery, offering personalized challenges. This approach supplements learning, enhancing presentation skills effectively. Voice-to-text conversion and AI feedback create a potent pedagogical tool, fostering a prompt feedback loop vital for learning effectiveness. Challenges in simulation design must be addressed for robustness and efficacy. The study validates these concepts by proposing a real-time 3D dialogue simulator, emphasizing the importance of continual improvement in presentation skill development. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.},
keywords = {Adversarial machine learning, AI feedback, Contrastive Learning, Digital elevation model, Dynamic pitch, Federated learning, feedback, Generative adversarial networks, Iterative practice, Language Model, Open source language, Open source software, Presentation skills, Simulation Design, Spoken words, Trial and error, Virtual environments, Virtual reality based simulators},
pubstate = {published},
tppubtype = {inproceedings}
}
Presenting before an audience presents challenges throughout preparation and delivery, necessitating tools to securely refine skills securely. Interviews mirror presentations, showcasing oneself to convey qualifications. Virtual environments offer safe spaces for trial and error, crucial for iterative practice without emotional distress. This research proposes a Virtual Reality-Based Dynamic Pitch Simulation with Integrated Generative AI to effectively enhance presentation skills. The simulation converts spoken words to text, then uses AI to generate relevant questions for practice. Benefits include realistic feedback and adaptability to user proficiency. Open-source language models evaluate content, coherence, and delivery, offering personalized challenges. This approach supplements learning, enhancing presentation skills effectively. Voice-to-text conversion and AI feedback create a potent pedagogical tool, fostering a prompt feedback loop vital for learning effectiveness. Challenges in simulation design must be addressed for robustness and efficacy. The study validates these concepts by proposing a real-time 3D dialogue simulator, emphasizing the importance of continual improvement in presentation skill development. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
Gujar, P.; Paliwal, G.; Panyam, S.
Generative AI and the Future of Interactive and Immersive Advertising Proceedings Article
In: D., Rivas-Lalaleo; S.L.S., Maita (Ed.): ETCM - Ecuador Tech. Chapters Meet., Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-835039158-9 (ISBN).
Abstract | Links | BibTeX | Tags: Ad Creation, Adversarial machine learning, Advertising Technology (AdTech), Advertizing, Advertizing technology, Augmented Reality, Augmented Reality (AR), Generative adversarial networks, Generative AI, Immersive, Immersive Advertising, Immersive advertizing, Interactive Advertising, Interactive advertizing, machine learning, Machine-learning, Marketing, Mixed reality, Mixed Reality (MR), Personalization, Personalizations, User Engagement, Virtual environments, Virtual Reality, Virtual Reality (VR)
@inproceedings{gujar_generative_2024,
title = {Generative AI and the Future of Interactive and Immersive Advertising},
author = {P. Gujar and G. Paliwal and S. Panyam},
editor = {Rivas-Lalaleo D. and Maita S.L.S.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85211805262&doi=10.1109%2fETCM63562.2024.10746166&partnerID=40&md5=179c5ceeb28ed72e809748322535c7ad},
doi = {10.1109/ETCM63562.2024.10746166},
isbn = {979-835039158-9 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {ETCM - Ecuador Tech. Chapters Meet.},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Generative AI is revolutionizing interactive and immersive advertising by enabling more personalized, engaging experiences through advanced technologies like VR, AR, and MR. This transformation is reshaping how advertisers create, deliver, and optimize content, allowing for two-way communication and blurring lines between digital and physical worlds. AI enhances user engagement through predictive analytics, real-time adaptation, and natural language processing, while also optimizing ad placement and personalization. Future trends include integration with emerging technologies like 5G and IoT, fully immersive experiences, and hyper-personalization. However, challenges such as privacy concerns, transparency issues, and ethical considerations must be addressed. As AI continues to evolve, it promises to create unprecedented opportunities for brands to connect with audiences in meaningful ways, potentially blurring the line between advertising and interactive entertainment. The industry must proactively address these challenges to ensure AI-driven advertising enhances user experiences while respecting privacy and maintaining trust. © 2024 IEEE.},
keywords = {Ad Creation, Adversarial machine learning, Advertising Technology (AdTech), Advertizing, Advertizing technology, Augmented Reality, Augmented Reality (AR), Generative adversarial networks, Generative AI, Immersive, Immersive Advertising, Immersive advertizing, Interactive Advertising, Interactive advertizing, machine learning, Machine-learning, Marketing, Mixed reality, Mixed Reality (MR), Personalization, Personalizations, User Engagement, Virtual environments, Virtual Reality, Virtual Reality (VR)},
pubstate = {published},
tppubtype = {inproceedings}
}
Generative AI is revolutionizing interactive and immersive advertising by enabling more personalized, engaging experiences through advanced technologies like VR, AR, and MR. This transformation is reshaping how advertisers create, deliver, and optimize content, allowing for two-way communication and blurring lines between digital and physical worlds. AI enhances user engagement through predictive analytics, real-time adaptation, and natural language processing, while also optimizing ad placement and personalization. Future trends include integration with emerging technologies like 5G and IoT, fully immersive experiences, and hyper-personalization. However, challenges such as privacy concerns, transparency issues, and ethical considerations must be addressed. As AI continues to evolve, it promises to create unprecedented opportunities for brands to connect with audiences in meaningful ways, potentially blurring the line between advertising and interactive entertainment. The industry must proactively address these challenges to ensure AI-driven advertising enhances user experiences while respecting privacy and maintaining trust. © 2024 IEEE.
Bao, Y.; Gao, N.; Weng, D.; Chen, J.; Tian, Z.
MuseGesture: A Framework for Gesture Synthesis by Virtual Agents in VR Museum Guides Proceedings Article
In: U., Eck; M., Sra; J., Stefanucci; M., Sugimoto; M., Tatzgern; I., Williams (Ed.): Proc. - IEEE Int. Symp. Mixed Augment. Real. Adjunct, ISMAR-Adjunct, pp. 337–338, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-833150691-9 (ISBN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, Embeddings, Gesture Generation, Intelligent Agents, Intelligent systems, Intelligent virtual agents, Language generation, Language Model, Large language model, large language models, Museum guide, Reinforcement Learning, Reinforcement learnings, Robust language understanding, Virtual agent, Virtual Agents, Virtual environments, Virtual reality museum guide, VR Museum Guides
@inproceedings{bao_musegesture_2024,
title = {MuseGesture: A Framework for Gesture Synthesis by Virtual Agents in VR Museum Guides},
author = {Y. Bao and N. Gao and D. Weng and J. Chen and Z. Tian},
editor = {Eck U. and Sra M. and Stefanucci J. and Sugimoto M. and Tatzgern M. and Williams I.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85214385900&doi=10.1109%2fISMAR-Adjunct64951.2024.00079&partnerID=40&md5=e71ffc28e299597557034259aab50641},
doi = {10.1109/ISMAR-Adjunct64951.2024.00079},
isbn = {979-833150691-9 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - IEEE Int. Symp. Mixed Augment. Real. Adjunct, ISMAR-Adjunct},
pages = {337–338},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {This paper presents an innovative framework named MuseGesture, designed to generate contextually adaptive gestures for virtual agents in Virtual Reality (VR) museums. The framework leverages the robust language understanding and generation capabilities of Large Language Models (LLMs) to parse tour narration texts and generate corresponding explanatory gestures. Through reinforcement learning and adversarial skill embeddings, the framework also generates guiding gestures tailored to the virtual museum environment, integrating both gesture types using conditional motion interpolation methods. Experimental results and user studies demonstrate that this approach effectively enables voice-command-controlled virtual guide gestures, offering a novel intelligent guiding system solution that enhances the interactive experience in VR museum environments. © 2024 IEEE.},
keywords = {Adversarial machine learning, Embeddings, Gesture Generation, Intelligent Agents, Intelligent systems, Intelligent virtual agents, Language generation, Language Model, Large language model, large language models, Museum guide, Reinforcement Learning, Reinforcement learnings, Robust language understanding, Virtual agent, Virtual Agents, Virtual environments, Virtual reality museum guide, VR Museum Guides},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper presents an innovative framework named MuseGesture, designed to generate contextually adaptive gestures for virtual agents in Virtual Reality (VR) museums. The framework leverages the robust language understanding and generation capabilities of Large Language Models (LLMs) to parse tour narration texts and generate corresponding explanatory gestures. Through reinforcement learning and adversarial skill embeddings, the framework also generates guiding gestures tailored to the virtual museum environment, integrating both gesture types using conditional motion interpolation methods. Experimental results and user studies demonstrate that this approach effectively enables voice-command-controlled virtual guide gestures, offering a novel intelligent guiding system solution that enhances the interactive experience in VR museum environments. © 2024 IEEE.
Arrigo, M.; Farella, M.; Fulantelli, G.; Schicchi, D.; Taibi, D.
A Task-Interaction Framework to Monitor Mobile Learning Activities Based on Artificial Intelligence and Augmented Reality Proceedings Article
In: L.T., De Paolis; P., Arpaia; M., Sacco (Ed.): Lect. Notes Comput. Sci., pp. 325–333, Springer Science and Business Media Deutschland GmbH, 2024, ISBN: 03029743 (ISSN); 978-303171706-2 (ISBN).
Abstract | Links | BibTeX | Tags: Activity-based, Adversarial machine learning, Analytic technique, Augmented Reality, Contrastive Learning, Federated learning, Generative AI, Interaction framework, Learning Activity, Learning analytic framework, Learning Analytics Framework, Learning experiences, Learning patterns, Mobile Learning, Teachers'
@inproceedings{arrigo_task-interaction_2024,
title = {A Task-Interaction Framework to Monitor Mobile Learning Activities Based on Artificial Intelligence and Augmented Reality},
author = {M. Arrigo and M. Farella and G. Fulantelli and D. Schicchi and D. Taibi},
editor = {De Paolis L.T. and Arpaia P. and Sacco M.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85204618733&doi=10.1007%2f978-3-031-71707-9_26&partnerID=40&md5=8969f18ab0f10dcddf37e54265d10518},
doi = {10.1007/978-3-031-71707-9_26},
isbn = {03029743 (ISSN); 978-303171706-2 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Lect. Notes Comput. Sci.},
volume = {15027 LNCS},
pages = {325–333},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {The complexity behind the analysis of mobile learning activities has requested the development of specifically designed frameworks. When students are involved in mobile learning experiences, they interact with the context in which the activities occur, the content they have access to, with peers and their teachers. The wider adoption of generative artificial intelligence introduces new interactions that researchers have to look at when learning analytics techniques are applied to monitor learning patterns. The task interaction framework proposed in this paper explores how AI-based tools affect student-content and student-context interactions during mobile learning activities, thus focusing on the interplay of Learning Analytics and Artificial Intelligence advances in the educational domain. A use case scenario that explores the framework’s application in a real educational context is also presented. Finally, we describe the architectural design of an environment that leverages the task interaction framework to analyze enhanced mobile learning experiences in which structured content extracted from a Knowledge Graph is elaborated by a large language model to provide students with personalized content. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.},
keywords = {Activity-based, Adversarial machine learning, Analytic technique, Augmented Reality, Contrastive Learning, Federated learning, Generative AI, Interaction framework, Learning Activity, Learning analytic framework, Learning Analytics Framework, Learning experiences, Learning patterns, Mobile Learning, Teachers'},
pubstate = {published},
tppubtype = {inproceedings}
}
The complexity behind the analysis of mobile learning activities has requested the development of specifically designed frameworks. When students are involved in mobile learning experiences, they interact with the context in which the activities occur, the content they have access to, with peers and their teachers. The wider adoption of generative artificial intelligence introduces new interactions that researchers have to look at when learning analytics techniques are applied to monitor learning patterns. The task interaction framework proposed in this paper explores how AI-based tools affect student-content and student-context interactions during mobile learning activities, thus focusing on the interplay of Learning Analytics and Artificial Intelligence advances in the educational domain. A use case scenario that explores the framework’s application in a real educational context is also presented. Finally, we describe the architectural design of an environment that leverages the task interaction framework to analyze enhanced mobile learning experiences in which structured content extracted from a Knowledge Graph is elaborated by a large language model to provide students with personalized content. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
Gao, H.; Huai, H.; Yildiz-Degirmenci, S.; Bannert, M.; Kasneci, E.
DataliVR: Transformation of Data Literacy Education through Virtual Reality with ChatGPT-Powered Enhancements Proceedings Article
In: U., Eck; M., Sra; J., Stefanucci; M., Sugimoto; M., Tatzgern; I., Williams (Ed.): Proc. - IEEE Int. Symp. Mixed Augment. Real., ISMAR, pp. 120–129, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-833151647-5 (ISBN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, Chatbots, ChatGPT, Contrastive Learning, Data driven, Data literacy, Digital transformation, Federated learning, Immersive learning, Language Model, Large language model, Learning experiences, Learning outcome, LLMs, Virtual environments, Virtual Reality
@inproceedings{gao_datalivr_2024,
title = {DataliVR: Transformation of Data Literacy Education through Virtual Reality with ChatGPT-Powered Enhancements},
author = {H. Gao and H. Huai and S. Yildiz-Degirmenci and M. Bannert and E. Kasneci},
editor = {Eck U. and Sra M. and Stefanucci J. and Sugimoto M. and Tatzgern M. and Williams I.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85213525613&doi=10.1109%2fISMAR62088.2024.00026&partnerID=40&md5=abdeba7ecfecc8b1d715d633a29bd11d},
doi = {10.1109/ISMAR62088.2024.00026},
isbn = {979-833151647-5 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - IEEE Int. Symp. Mixed Augment. Real., ISMAR},
pages = {120–129},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Data literacy is essential in today's data-driven world, emphasizing individuals' abilities to effectively manage data and extract meaningful insights. However, traditional classroom-based educational approaches often struggle to fully address the multifaceted nature of data literacy. As education undergoes digital transformation, innovative technologies such as Virtual Reality (VR) offer promising avenues for immersive and engaging learning experiences. This paper introduces DataliVR, a pioneering VR application aimed at enhancing the data literacy skills of university students within a contextual and gamified virtual learning environment. By integrating Large Language Models (LLMs) like ChatGPT as a conversational artificial intelligence (AI) chatbot embodied within a virtual avatar, DataliVR provides personalized learning assistance, enriching user learning experiences. Our study employed an experimental approach, with chatbot availability as the independent variable, analyzing learning experiences and outcomes as dependent variables with a sample of thirty participants. Our approach underscores the effectiveness and user-friendliness of ChatGPT-powered DataliVR in fostering data literacy skills. Moreover, our study examines the impact of the ChatGPT-based AI chatbot on users' learning, revealing significant effects on both learning experiences and outcomes. Our study presents a robust tool for fostering data literacy skills, contributing significantly to the digital advancement of data literacy education through cutting-edge VR and AI technologies. Moreover, our research provides valuable insights and implications for future research endeavors aiming to integrate LLMs (e.g., ChatGPT) into educational VR platforms. © 2024 IEEE.},
keywords = {Adversarial machine learning, Chatbots, ChatGPT, Contrastive Learning, Data driven, Data literacy, Digital transformation, Federated learning, Immersive learning, Language Model, Large language model, Learning experiences, Learning outcome, LLMs, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
Data literacy is essential in today's data-driven world, emphasizing individuals' abilities to effectively manage data and extract meaningful insights. However, traditional classroom-based educational approaches often struggle to fully address the multifaceted nature of data literacy. As education undergoes digital transformation, innovative technologies such as Virtual Reality (VR) offer promising avenues for immersive and engaging learning experiences. This paper introduces DataliVR, a pioneering VR application aimed at enhancing the data literacy skills of university students within a contextual and gamified virtual learning environment. By integrating Large Language Models (LLMs) like ChatGPT as a conversational artificial intelligence (AI) chatbot embodied within a virtual avatar, DataliVR provides personalized learning assistance, enriching user learning experiences. Our study employed an experimental approach, with chatbot availability as the independent variable, analyzing learning experiences and outcomes as dependent variables with a sample of thirty participants. Our approach underscores the effectiveness and user-friendliness of ChatGPT-powered DataliVR in fostering data literacy skills. Moreover, our study examines the impact of the ChatGPT-based AI chatbot on users' learning, revealing significant effects on both learning experiences and outcomes. Our study presents a robust tool for fostering data literacy skills, contributing significantly to the digital advancement of data literacy education through cutting-edge VR and AI technologies. Moreover, our research provides valuable insights and implications for future research endeavors aiming to integrate LLMs (e.g., ChatGPT) into educational VR platforms. © 2024 IEEE.
Velev, D.; Steshina, L.; Petukhov, I.; Zlateva, P.
Challenges of Merging Generative AI with Metaverse for Next-Gen Education Proceedings Article
In: A.J., Tallon-Ballesteros (Ed.): Front. Artif. Intell. Appl., pp. 606–616, IOS Press BV, 2024, ISBN: 09226389 (ISSN); 978-164368569-4 (ISBN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, Augmented Reality, Contrastive Learning, Data privacy and securities, Digital literacies, Education, Federated learning, Generative adversarial networks, Generative AI, High speed internet, Instructional designs, Learning Environments, Metaverse, Metaverses, Personalized learning, Realtime processing, Teaching methods, Virtual environments, Virtual Reality
@inproceedings{velev_challenges_2024,
title = {Challenges of Merging Generative AI with Metaverse for Next-Gen Education},
author = {D. Velev and L. Steshina and I. Petukhov and P. Zlateva},
editor = {Tallon-Ballesteros A.J.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85215823646&doi=10.3233%2fFAIA241462&partnerID=40&md5=a3ed4e8486e2e32d0856a71a3a87496c},
doi = {10.3233/FAIA241462},
isbn = {09226389 (ISSN); 978-164368569-4 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Front. Artif. Intell. Appl.},
volume = {398},
pages = {606–616},
publisher = {IOS Press BV},
abstract = {The integration of Generative Artificial Intelligence (GenAI) with the Metaverse for a next-generation education is a complex but challenging task. The GenAI-enhanced Metaverse classrooms require innovative instructional designs that use virtual reality and augmented reality to enhance engagement and personalized learning. Educators must adapt to new roles over traditional teaching methods, while learners need to develop digital literacy skills that are essential for navigating and inhabiting in these environments. Such learning environments require significant advancements in real-time processing, scalability and interoperability of different platforms, while ensuring data privacy and security. The equity of access to high-speed internet and advanced devices still remains a serious barrier, which can increase the potential existing inequalities between different educational environments. Ethical considerations, including the responsible use of GenAI, the creation of unbiased educational content, and the psychological impacts of extended usage of virtual reality, are also of important consideration. The aim of the paper is to explore in detail the different challenges through a comprehensive analysis of the obstacles and potential solutions and to propose a collaborative framework involving educators, technologists, policymakers and industry stakeholders to address the effective implementation of the integration of GenAI and the Metaverse for a next generation education. © 2024 The Authors.},
keywords = {Adversarial machine learning, Augmented Reality, Contrastive Learning, Data privacy and securities, Digital literacies, Education, Federated learning, Generative adversarial networks, Generative AI, High speed internet, Instructional designs, Learning Environments, Metaverse, Metaverses, Personalized learning, Realtime processing, Teaching methods, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
The integration of Generative Artificial Intelligence (GenAI) with the Metaverse for a next-generation education is a complex but challenging task. The GenAI-enhanced Metaverse classrooms require innovative instructional designs that use virtual reality and augmented reality to enhance engagement and personalized learning. Educators must adapt to new roles over traditional teaching methods, while learners need to develop digital literacy skills that are essential for navigating and inhabiting in these environments. Such learning environments require significant advancements in real-time processing, scalability and interoperability of different platforms, while ensuring data privacy and security. The equity of access to high-speed internet and advanced devices still remains a serious barrier, which can increase the potential existing inequalities between different educational environments. Ethical considerations, including the responsible use of GenAI, the creation of unbiased educational content, and the psychological impacts of extended usage of virtual reality, are also of important consideration. The aim of the paper is to explore in detail the different challenges through a comprehensive analysis of the obstacles and potential solutions and to propose a collaborative framework involving educators, technologists, policymakers and industry stakeholders to address the effective implementation of the integration of GenAI and the Metaverse for a next generation education. © 2024 The Authors.
Rahmani, R.; Westin, T.; Nevelsteen, K.
Future Healthcare in Generative AI with Real Metaverse Proceedings Article
In: E.E., Shakshuki (Ed.): Procedia Comput. Sci., pp. 487–493, Elsevier B.V., 2024, ISBN: 18770509 (ISSN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, AI, Augmented Reality, Autism spectrum disorders, Contrastive Learning, Diseases, Edge Intelligence, Generative adversarial networks, Healthcare, Immersive learning, Independent living systems, Language Model, Large language model, LLM, Metaverses, Posttraumatic stress disorder, Real Metaverse, Social challenges, Virtual environments
@inproceedings{rahmani_future_2024,
title = {Future Healthcare in Generative AI with Real Metaverse},
author = {R. Rahmani and T. Westin and K. Nevelsteen},
editor = {Shakshuki E.E.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85214986921&doi=10.1016%2fj.procs.2024.11.137&partnerID=40&md5=3e25f2a1b023cd49f59a066a96bb2dd0},
doi = {10.1016/j.procs.2024.11.137},
isbn = {18770509 (ISSN)},
year = {2024},
date = {2024-01-01},
booktitle = {Procedia Comput. Sci.},
volume = {251},
pages = {487–493},
publisher = {Elsevier B.V.},
abstract = {The Metaverse offers a simulated environment that could transform healthcare by providing immersive learning experiences through Internet application and social form that integrates network of virtual reality environments. The Metaverse is expected to contribute to a new way of socializing, where users can enter a verse as avatars. The concept allows avatars to switch between verses seamlessly. Virtual Reality (VR) in healthcare has shown promise for social-skill training, especially for individuals with Autism Spectrum Disorder (ASD) and social challenge training of patients with Post-Traumatic Stress Disorder (PTSD) requiring adaptable support. The problem lies in the limited adaptability and functionality of existing Metaverse implementations for individuals with ASD and PTSD. While studies have explored various implementation ideas, such as VR platforms for training social skills, social challenge and context-aware Augmented Reality (AR) systems for daily activities, many lack adaptability of user input and output. A proposed solution involves a context-aware system using AI, Large Language Models (LLMs) and generative agents to support independent living for individuals with ASD and a tool to enhance emotional learning with PTSD. © 2024 The Authors.},
keywords = {Adversarial machine learning, AI, Augmented Reality, Autism spectrum disorders, Contrastive Learning, Diseases, Edge Intelligence, Generative adversarial networks, Healthcare, Immersive learning, Independent living systems, Language Model, Large language model, LLM, Metaverses, Posttraumatic stress disorder, Real Metaverse, Social challenges, Virtual environments},
pubstate = {published},
tppubtype = {inproceedings}
}
The Metaverse offers a simulated environment that could transform healthcare by providing immersive learning experiences through Internet application and social form that integrates network of virtual reality environments. The Metaverse is expected to contribute to a new way of socializing, where users can enter a verse as avatars. The concept allows avatars to switch between verses seamlessly. Virtual Reality (VR) in healthcare has shown promise for social-skill training, especially for individuals with Autism Spectrum Disorder (ASD) and social challenge training of patients with Post-Traumatic Stress Disorder (PTSD) requiring adaptable support. The problem lies in the limited adaptability and functionality of existing Metaverse implementations for individuals with ASD and PTSD. While studies have explored various implementation ideas, such as VR platforms for training social skills, social challenge and context-aware Augmented Reality (AR) systems for daily activities, many lack adaptability of user input and output. A proposed solution involves a context-aware system using AI, Large Language Models (LLMs) and generative agents to support independent living for individuals with ASD and a tool to enhance emotional learning with PTSD. © 2024 The Authors.
Jia, Y.; Sin, Z. P. T.; Wang, X. E.; Li, C.; Ng, P. H. F.; Huang, X.; Dong, J.; Wang, Y.; Baciu, G.; Cao, J.; Li, Q.
NivTA: Towards a Naturally Interactable Edu-Metaverse Teaching Assistant for CAVE Proceedings Article
In: Proc. - IEEE Int. Conf. Metaverse Comput., Netw., Appl., MetaCom, pp. 57–64, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-833151599-7 (ISBN).
Abstract | Links | BibTeX | Tags: Active learning, Adversarial machine learning, cave automatic virtual environment, Cave automatic virtual environments, Caves, Chatbots, Contrastive Learning, Digital elevation model, Federated learning, Interactive education, Language Model, Large language model agent, Learning Activity, LLM agents, Metaverses, Model agents, Natural user interface, Students, Teaching, Teaching assistants, Virtual environments, Virtual Reality, virtual teaching assistant, Virtual teaching assistants
@inproceedings{jia_nivta_2024,
title = {NivTA: Towards a Naturally Interactable Edu-Metaverse Teaching Assistant for CAVE},
author = {Y. Jia and Z. P. T. Sin and X. E. Wang and C. Li and P. H. F. Ng and X. Huang and J. Dong and Y. Wang and G. Baciu and J. Cao and Q. Li},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85211447638&doi=10.1109%2fMetaCom62920.2024.00023&partnerID=40&md5=efefd453c426e74705518254bdc49e87},
doi = {10.1109/MetaCom62920.2024.00023},
isbn = {979-833151599-7 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - IEEE Int. Conf. Metaverse Comput., Netw., Appl., MetaCom},
pages = {57–64},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Edu-metaverse is a specialized metaverse dedicated for interactive education in an immersive environment. Its main purpose is to immerse the learners in a digital environment and conduct learning activities that could mirror reality. Not only does it enable activities that may be difficult to perform in the real world, but it also extends the interaction to personalized and CL. This is a more effective pedagogical approach as it tends to enhance the motivation and engagement of students and it increases their active participation in lessons delivered. To this extend, we propose to realize an interactive virtual teaching assistant called NivTA. To make NivTA easily accessible and engaging by multiple users simultaneously, we also propose to use a CAVE virtual environment (CAVE-VR) as a "metaverse window"into concepts, ideas, topics, and learning activities. The students simply need to step into the CAVE-VR and interact with a life-size teaching assistant that they can engage with naturally, as if they are approaching a real person. Instead of text-based interaction currently developed for large language models (LLM), NivTA is given additional cues regarding the users so it can react more naturally via a specific prompt design. For example, the user can simply point to an educational concept and ask NivTA to explain what it is. To guide NivTA onto the educational concept, the prompt is also designed to feed in an educational KG to provide NivTA with the context of the student's question. The NivTA system is an integration of several components that are discussed in this paper. We further describe how the system is designed and implemented, along with potential applications and future work on interactive collaborative edu-metaverse environments dedicated for teaching and learning. © 2024 IEEE.},
keywords = {Active learning, Adversarial machine learning, cave automatic virtual environment, Cave automatic virtual environments, Caves, Chatbots, Contrastive Learning, Digital elevation model, Federated learning, Interactive education, Language Model, Large language model agent, Learning Activity, LLM agents, Metaverses, Model agents, Natural user interface, Students, Teaching, Teaching assistants, Virtual environments, Virtual Reality, virtual teaching assistant, Virtual teaching assistants},
pubstate = {published},
tppubtype = {inproceedings}
}
Edu-metaverse is a specialized metaverse dedicated for interactive education in an immersive environment. Its main purpose is to immerse the learners in a digital environment and conduct learning activities that could mirror reality. Not only does it enable activities that may be difficult to perform in the real world, but it also extends the interaction to personalized and CL. This is a more effective pedagogical approach as it tends to enhance the motivation and engagement of students and it increases their active participation in lessons delivered. To this extend, we propose to realize an interactive virtual teaching assistant called NivTA. To make NivTA easily accessible and engaging by multiple users simultaneously, we also propose to use a CAVE virtual environment (CAVE-VR) as a "metaverse window"into concepts, ideas, topics, and learning activities. The students simply need to step into the CAVE-VR and interact with a life-size teaching assistant that they can engage with naturally, as if they are approaching a real person. Instead of text-based interaction currently developed for large language models (LLM), NivTA is given additional cues regarding the users so it can react more naturally via a specific prompt design. For example, the user can simply point to an educational concept and ask NivTA to explain what it is. To guide NivTA onto the educational concept, the prompt is also designed to feed in an educational KG to provide NivTA with the context of the student's question. The NivTA system is an integration of several components that are discussed in this paper. We further describe how the system is designed and implemented, along with potential applications and future work on interactive collaborative edu-metaverse environments dedicated for teaching and learning. © 2024 IEEE.
Bandara, E.; Foytik, P.; Shetty, S.; Hassanzadeh, A.
Generative-AI(with Custom-Trained Meta's Llama2 LLM), Blockchain, NFT, Federated Learning and PBOM Enabled Data Security Architecture for Metaverse on 5G/6G Environment Proceedings Article
In: Proc. - IEEE Int. Conf. Mob. Ad-Hoc Smart Syst., MASS, pp. 118–124, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-835036399-9 (ISBN).
Abstract | Links | BibTeX | Tags: 5G, 6G, Adversarial machine learning, Bill of materials, Block-chain, Blockchain, Curricula, Data privacy, Distance education, Federated learning, Generative adversarial networks, Generative-AI, Hardware security, Llama2, LLM, Medium access control, Metaverse, Metaverses, Network Security, Nft, Non-fungible token, Personnel training, Problem oriented languages, Reference architecture, Steganography
@inproceedings{bandara_generative-aicustom-trained_2024,
title = {Generative-AI(with Custom-Trained Meta's Llama2 LLM), Blockchain, NFT, Federated Learning and PBOM Enabled Data Security Architecture for Metaverse on 5G/6G Environment},
author = {E. Bandara and P. Foytik and S. Shetty and A. Hassanzadeh},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85210243120&doi=10.1109%2fMASS62177.2024.00026&partnerID=40&md5=70d21ac1e9c7b886da14825376919cac},
doi = {10.1109/MASS62177.2024.00026},
isbn = {979-835036399-9 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - IEEE Int. Conf. Mob. Ad-Hoc Smart Syst., MASS},
pages = {118–124},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {The Metaverse is an integrated network of 3D virtual worlds accessible through a virtual reality headset. Its impact on data privacy and security is increasingly recognized as a major concern. There is a growing interest in developing a reference architecture that describes the four core aspects of its data: acquisition, storage, sharing, and interoperability. Establishing a secure data architecture is imperative to manage users' personal data and facilitate trusted AR/VR and AI/ML solutions within the Metaverse. This paper details a reference architecture empowered by Generative-AI, Blockchain, Federated Learning, and Non-Fungible Tokens (NFTs). Within this archi-tecture, various resource providers collaborate via the blockchain network. Handling personal user data and resource provider identities is executed through a Self-Sovereign Identity-enabled privacy-preserving framework. AR/NR devices in the Metaverse are represented as NFT tokens available for user purchase. Software updates and supply-chain verification for these devices are managed using a Software Bill of Materials (SBOM) and a Pipeline Bill of Materials (PBOM) verification system. Moreover, a custom-trained Llama2 LLM from Meta has been integrated to generate PBOMs for AR/NR devices' software updates, thereby preventing malware intrusions and data breaches. This Llama2-13B LLM has been quantized and fine-tuned using Qlora to ensure optimal performance on consumer-grade hardware. The provenance of AI/ML models used in the Metaverse is encapsu-lated as Model Card objects, allowing external parties to audit and verify them, thus mitigating adversarial learning attacks within these models. To the best of our knowledge, this is the very first research effort aimed at standardizing PBOM schemas and integrating Language Model algorithms for the generation of PBOMs. Additionally, a proposed mechanism facilitates different AI/ML providers in training their machine learning models using a privacy-preserving federated learning approach. Authorization of communications among AR/VR devices in the Metaverse is conducted through a Zero-Trust security-enabled rule engine. A system testbed has been implemented within a 5G environment, utilizing Ericsson new Radio with Open5GS 5G core. © 2024 IEEE.},
keywords = {5G, 6G, Adversarial machine learning, Bill of materials, Block-chain, Blockchain, Curricula, Data privacy, Distance education, Federated learning, Generative adversarial networks, Generative-AI, Hardware security, Llama2, LLM, Medium access control, Metaverse, Metaverses, Network Security, Nft, Non-fungible token, Personnel training, Problem oriented languages, Reference architecture, Steganography},
pubstate = {published},
tppubtype = {inproceedings}
}
The Metaverse is an integrated network of 3D virtual worlds accessible through a virtual reality headset. Its impact on data privacy and security is increasingly recognized as a major concern. There is a growing interest in developing a reference architecture that describes the four core aspects of its data: acquisition, storage, sharing, and interoperability. Establishing a secure data architecture is imperative to manage users' personal data and facilitate trusted AR/VR and AI/ML solutions within the Metaverse. This paper details a reference architecture empowered by Generative-AI, Blockchain, Federated Learning, and Non-Fungible Tokens (NFTs). Within this archi-tecture, various resource providers collaborate via the blockchain network. Handling personal user data and resource provider identities is executed through a Self-Sovereign Identity-enabled privacy-preserving framework. AR/NR devices in the Metaverse are represented as NFT tokens available for user purchase. Software updates and supply-chain verification for these devices are managed using a Software Bill of Materials (SBOM) and a Pipeline Bill of Materials (PBOM) verification system. Moreover, a custom-trained Llama2 LLM from Meta has been integrated to generate PBOMs for AR/NR devices' software updates, thereby preventing malware intrusions and data breaches. This Llama2-13B LLM has been quantized and fine-tuned using Qlora to ensure optimal performance on consumer-grade hardware. The provenance of AI/ML models used in the Metaverse is encapsu-lated as Model Card objects, allowing external parties to audit and verify them, thus mitigating adversarial learning attacks within these models. To the best of our knowledge, this is the very first research effort aimed at standardizing PBOM schemas and integrating Language Model algorithms for the generation of PBOMs. Additionally, a proposed mechanism facilitates different AI/ML providers in training their machine learning models using a privacy-preserving federated learning approach. Authorization of communications among AR/VR devices in the Metaverse is conducted through a Zero-Trust security-enabled rule engine. A system testbed has been implemented within a 5G environment, utilizing Ericsson new Radio with Open5GS 5G core. © 2024 IEEE.
Federico, G.; Carrara, F.; Amato, G.; Benedetto, M. Di
Spatio-Temporal 3D Reconstruction from Frame Sequences and Feature Points Proceedings Article
In: ACM Int. Conf. Proc. Ser., pp. 52–64, Association for Computing Machinery, 2024, ISBN: 979-840071794-9 (ISBN).
Abstract | Links | BibTeX | Tags: 3D reconstruction, Adversarial machine learning, Artificial intelligence, Color motion pictures, Color photography, Contrastive Learning, De-noising, Deep learning, Denoising Diffusion Probabilistic Model, Frame features, machine learning, Machine-learning, Probabilistic models, Signed Distance Field, Signed distance fields, Spatio-temporal, Video Reconstruction, Video streaming
@inproceedings{federico_spatio-temporal_2024,
title = {Spatio-Temporal 3D Reconstruction from Frame Sequences and Feature Points},
author = {G. Federico and F. Carrara and G. Amato and M. Di Benedetto},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85203128613&doi=10.1145%2f3672406.3672415&partnerID=40&md5=2a0dc51baa15f0dcd7f9d2cca708ec15},
doi = {10.1145/3672406.3672415},
isbn = {979-840071794-9 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {ACM Int. Conf. Proc. Ser.},
pages = {52–64},
publisher = {Association for Computing Machinery},
abstract = {Reconstructing a large real environment is a fundamental task to promote eXtended Reality adoption in industrial and entertainment fields. However, the short range of depth cameras, the sparsity of LiDAR sensors, and the huge computational cost of Structure-from-Motion pipelines prevent scene replication in near real time. To overcome these limitations, we introduce a spatio-temporal diffusion neural architecture, a generative AI technique that fuses temporal information (i.e., a short temporally-ordered list of color photographs, like sparse frames of a video stream) with an approximate spatial resemblance of the explored environment. Our aim is to modify an existing 3D diffusion neural model to produce a Signed Distance Field volume from which a 3D mesh representation can be extracted. Our results show that the hallucination approach of diffusion models is an effective methodology where a fast reconstruction is a crucial target. © 2024 Owner/Author.},
keywords = {3D reconstruction, Adversarial machine learning, Artificial intelligence, Color motion pictures, Color photography, Contrastive Learning, De-noising, Deep learning, Denoising Diffusion Probabilistic Model, Frame features, machine learning, Machine-learning, Probabilistic models, Signed Distance Field, Signed distance fields, Spatio-temporal, Video Reconstruction, Video streaming},
pubstate = {published},
tppubtype = {inproceedings}
}
Reconstructing a large real environment is a fundamental task to promote eXtended Reality adoption in industrial and entertainment fields. However, the short range of depth cameras, the sparsity of LiDAR sensors, and the huge computational cost of Structure-from-Motion pipelines prevent scene replication in near real time. To overcome these limitations, we introduce a spatio-temporal diffusion neural architecture, a generative AI technique that fuses temporal information (i.e., a short temporally-ordered list of color photographs, like sparse frames of a video stream) with an approximate spatial resemblance of the explored environment. Our aim is to modify an existing 3D diffusion neural model to produce a Signed Distance Field volume from which a 3D mesh representation can be extracted. Our results show that the hallucination approach of diffusion models is an effective methodology where a fast reconstruction is a crucial target. © 2024 Owner/Author.
Sikström, P.; Valentini, C.; Sivunen, A.; Kärkkäinen, T.
Pedagogical agents communicating and scaffolding students' learning: High school teachers' and students' perspectives Journal Article
In: Computers and Education, vol. 222, 2024, ISSN: 03601315 (ISSN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, Agents communication, Augmented Reality, Contrastive Learning, Federated learning, Human communications, Human-Machine Communication, Human-to-human communication script, Human–machine communication, Human–machine communication (HMC), pedagogical agent, Pedagogical agents, Scaffolds, Scaffolds (biology), Secondary education, Student learning, Students, Teachers', Teaching, User-centered design, User-centred, Virtual environments
@article{sikstrom_pedagogical_2024,
title = {Pedagogical agents communicating and scaffolding students' learning: High school teachers' and students' perspectives},
author = {P. Sikström and C. Valentini and A. Sivunen and T. Kärkkäinen},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85202198552&doi=10.1016%2fj.compedu.2024.105140&partnerID=40&md5=dfb4a7b6c1f6352c5cc6faac213e938f},
doi = {10.1016/j.compedu.2024.105140},
issn = {03601315 (ISSN)},
year = {2024},
date = {2024-01-01},
journal = {Computers and Education},
volume = {222},
abstract = {Pedagogical agents (PAs) communicate verbally and non-verbally with students in digital and virtual reality/augmented reality learning environments. PAs have been shown to be beneficial for learning, and generative artificial intelligence, such as large language models, can improve PAs' communication abilities significantly. K-12 education is underrepresented in learning technology research and teachers' and students' insights have not been considered when developing PA communication. The current study addresses this research gap by conducting and analyzing semi-structured, in-depth interviews with eleven high school teachers and sixteen high school students about their expectations for PAs' communication capabilities. The interviewees identified relational and task-related communication capabilities that a PA should perform to communicate effectively with students and scaffold their learning. PA communication that is simultaneously affirmative and relational can induce immediacy, foster the relationship and engagement with a PA, and support students' learning management. Additionally, the teachers and students described the activities and technological aspects that should be considered when designing conversational PAs. The study showed that teachers and students applied human-to-human communication scripts when outlining their desired PA communication characteristics. The study offers novel insights and recommendations to researchers and developers on the communicational, pedagogical, and technological aspects that must be considered when designing communicative PAs that scaffold students’ learning, and discusses the contributions on human–machine communication in education. © 2024 The Authors},
keywords = {Adversarial machine learning, Agents communication, Augmented Reality, Contrastive Learning, Federated learning, Human communications, Human-Machine Communication, Human-to-human communication script, Human–machine communication, Human–machine communication (HMC), pedagogical agent, Pedagogical agents, Scaffolds, Scaffolds (biology), Secondary education, Student learning, Students, Teachers', Teaching, User-centered design, User-centred, Virtual environments},
pubstate = {published},
tppubtype = {article}
}
Pedagogical agents (PAs) communicate verbally and non-verbally with students in digital and virtual reality/augmented reality learning environments. PAs have been shown to be beneficial for learning, and generative artificial intelligence, such as large language models, can improve PAs' communication abilities significantly. K-12 education is underrepresented in learning technology research and teachers' and students' insights have not been considered when developing PA communication. The current study addresses this research gap by conducting and analyzing semi-structured, in-depth interviews with eleven high school teachers and sixteen high school students about their expectations for PAs' communication capabilities. The interviewees identified relational and task-related communication capabilities that a PA should perform to communicate effectively with students and scaffold their learning. PA communication that is simultaneously affirmative and relational can induce immediacy, foster the relationship and engagement with a PA, and support students' learning management. Additionally, the teachers and students described the activities and technological aspects that should be considered when designing conversational PAs. The study showed that teachers and students applied human-to-human communication scripts when outlining their desired PA communication characteristics. The study offers novel insights and recommendations to researchers and developers on the communicational, pedagogical, and technological aspects that must be considered when designing communicative PAs that scaffold students’ learning, and discusses the contributions on human–machine communication in education. © 2024 The Authors
2023
Feng, Y.; Zhu, H.; Peng, D.; Peng, X.; Hu, P.
RONO: Robust Discriminative Learning with Noisy Labels for 2D-3D Cross-Modal Retrieval Proceedings Article
In: Proc IEEE Comput Soc Conf Comput Vision Pattern Recognit, pp. 11610–11619, IEEE Computer Society, 2023, ISBN: 10636919 (ISSN).
Abstract | Links | BibTeX | Tags: 3D content, 3D data, 3D modeling, Adversarial machine learning, Contrastive Learning, Cross-modal, Discriminative learning, Federated learning, Heterogeneous structures, Learning mechanism, Learning performance, Metaverses, Multi-modal learning, Noisy labels, Spatio-temporal data
@inproceedings{feng_rono_2023,
title = {RONO: Robust Discriminative Learning with Noisy Labels for 2D-3D Cross-Modal Retrieval},
author = {Y. Feng and H. Zhu and D. Peng and X. Peng and P. Hu},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85170845124&doi=10.1109%2fCVPR52729.2023.01117&partnerID=40&md5=2eee285207ff3ea8e774480e29d96ec1},
doi = {10.1109/CVPR52729.2023.01117},
isbn = {10636919 (ISSN)},
year = {2023},
date = {2023-01-01},
booktitle = {Proc IEEE Comput Soc Conf Comput Vision Pattern Recognit},
volume = {2023-June},
pages = {11610–11619},
publisher = {IEEE Computer Society},
abstract = {Recently, with the advent of Metaverse and AI Generated Content, cross-modal retrieval becomes popular with a burst of 2D and 3D data. However, this problem is challenging given the heterogeneous structure and semantic discrepancies. Moreover, imperfect annotations are ubiquitous given the ambiguous 2D and 3D content, thus inevitably producing noisy labels to degrade the learning performance. To tackle the problem, this paper proposes a robust 2D-3D retrieval framework (RONO) to robustly learn from noisy multimodal data. Specifically, one novel Robust Discriminative Center Learning mechanism (RDCL) is proposed in RONO to adaptively distinguish clean and noisy samples for respectively providing them with positive and negative optimization directions, thus mitigating the negative impact of noisy labels. Besides, we present a Shared Space Consistency Learning mechanism (SSCL) to capture the intrinsic information inside the noisy data by minimizing the cross-modal and semantic discrepancy between common space and label space simultaneously. Comprehensive mathematical analyses are given to theoretically prove the noise tolerance of the proposed method. Furthermore, we conduct extensive experiments on four 3D-model multimodal datasets to verify the effectiveness of our method by comparing it with 15 state-of-the-art methods. © 2023 IEEE.},
keywords = {3D content, 3D data, 3D modeling, Adversarial machine learning, Contrastive Learning, Cross-modal, Discriminative learning, Federated learning, Heterogeneous structures, Learning mechanism, Learning performance, Metaverses, Multi-modal learning, Noisy labels, Spatio-temporal data},
pubstate = {published},
tppubtype = {inproceedings}
}
Recently, with the advent of Metaverse and AI Generated Content, cross-modal retrieval becomes popular with a burst of 2D and 3D data. However, this problem is challenging given the heterogeneous structure and semantic discrepancies. Moreover, imperfect annotations are ubiquitous given the ambiguous 2D and 3D content, thus inevitably producing noisy labels to degrade the learning performance. To tackle the problem, this paper proposes a robust 2D-3D retrieval framework (RONO) to robustly learn from noisy multimodal data. Specifically, one novel Robust Discriminative Center Learning mechanism (RDCL) is proposed in RONO to adaptively distinguish clean and noisy samples for respectively providing them with positive and negative optimization directions, thus mitigating the negative impact of noisy labels. Besides, we present a Shared Space Consistency Learning mechanism (SSCL) to capture the intrinsic information inside the noisy data by minimizing the cross-modal and semantic discrepancy between common space and label space simultaneously. Comprehensive mathematical analyses are given to theoretically prove the noise tolerance of the proposed method. Furthermore, we conduct extensive experiments on four 3D-model multimodal datasets to verify the effectiveness of our method by comparing it with 15 state-of-the-art methods. © 2023 IEEE.