AHCI RESEARCH GROUP
Publications
Papers published in international journals,
proceedings of conferences, workshops and books.
OUR RESEARCH
Scientific Publications
How to
Here you can find the complete list of our publications.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
2025
Liu, G.; Du, H.; Wang, J.; Niyato, D.; Kim, D. I.
Contract-Inspired Contest Theory for Controllable Image Generation in Mobile Edge Metaverse Journal Article
In: IEEE Transactions on Mobile Computing, 2025, ISSN: 15361233 (ISSN).
Abstract | Links | BibTeX | Tags: Contest Theory, Deep learning, Deep reinforcement learning, Diffusion Model, Generative adversarial networks, Generative AI, High quality, Image generation, Image generations, Immersive technologies, Metaverses, Mobile edge computing, Reinforcement Learning, Reinforcement learnings, Resource allocation, Resources allocation, Semantic data, Virtual addresses, Virtual environments, Virtual Reality
@article{liu_contract-inspired_2025,
title = {Contract-Inspired Contest Theory for Controllable Image Generation in Mobile Edge Metaverse},
author = {G. Liu and H. Du and J. Wang and D. Niyato and D. I. Kim},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105000066834&doi=10.1109%2fTMC.2025.3550815&partnerID=40&md5=3cb5a2143b9ce4ca7f931a60f1bf239c},
doi = {10.1109/TMC.2025.3550815},
issn = {15361233 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Transactions on Mobile Computing},
abstract = {The rapid advancement of immersive technologies has propelled the development of the Metaverse, where the convergence of virtual and physical realities necessitates the generation of high-quality, photorealistic images to enhance user experience. However, generating these images, especially through Generative Diffusion Models (GDMs), in mobile edge computing environments presents significant challenges due to the limited computing resources of edge devices and the dynamic nature of wireless networks. This paper proposes a novel framework that integrates contract-inspired contest theory, Deep Reinforcement Learning (DRL), and GDMs to optimize image generation in these resource-constrained environments. The framework addresses the critical challenges of resource allocation and semantic data transmission quality by incentivizing edge devices to efficiently transmit high-quality semantic data, which is essential for creating realistic and immersive images. The use of contest and contract theory ensures that edge devices are motivated to allocate resources effectively, while DRL dynamically adjusts to network conditions, optimizing the overall image generation process. Experimental results demonstrate that the proposed approach not only improves the quality of generated images but also achieves superior convergence speed and stability compared to traditional methods. This makes the framework particularly effective for optimizing complex resource allocation tasks in mobile edge Metaverse applications, offering enhanced performance and efficiency in creating immersive virtual environments. © 2002-2012 IEEE.},
keywords = {Contest Theory, Deep learning, Deep reinforcement learning, Diffusion Model, Generative adversarial networks, Generative AI, High quality, Image generation, Image generations, Immersive technologies, Metaverses, Mobile edge computing, Reinforcement Learning, Reinforcement learnings, Resource allocation, Resources allocation, Semantic data, Virtual addresses, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
The rapid advancement of immersive technologies has propelled the development of the Metaverse, where the convergence of virtual and physical realities necessitates the generation of high-quality, photorealistic images to enhance user experience. However, generating these images, especially through Generative Diffusion Models (GDMs), in mobile edge computing environments presents significant challenges due to the limited computing resources of edge devices and the dynamic nature of wireless networks. This paper proposes a novel framework that integrates contract-inspired contest theory, Deep Reinforcement Learning (DRL), and GDMs to optimize image generation in these resource-constrained environments. The framework addresses the critical challenges of resource allocation and semantic data transmission quality by incentivizing edge devices to efficiently transmit high-quality semantic data, which is essential for creating realistic and immersive images. The use of contest and contract theory ensures that edge devices are motivated to allocate resources effectively, while DRL dynamically adjusts to network conditions, optimizing the overall image generation process. Experimental results demonstrate that the proposed approach not only improves the quality of generated images but also achieves superior convergence speed and stability compared to traditional methods. This makes the framework particularly effective for optimizing complex resource allocation tasks in mobile edge Metaverse applications, offering enhanced performance and efficiency in creating immersive virtual environments. © 2002-2012 IEEE.
Zhang, Z.; Jiang, Y.; Wei, X.; Chen, M.; Dong, H.; Yu, S.
Generative-AI for XR Content Transmission in the Metaverse: Potential Approaches, Challenges, and a Generation-Driven Transmission Framework Journal Article
In: IEEE Network, 2025, ISSN: 08908044 (ISSN).
Abstract | Links | BibTeX | Tags: 'current, Cloud servers, Collaboration architecture, Content transmission, Decision making, Deep learning, Deep reinforcement learning, Edge server, Generative adversarial networks, Intelligence models, Large volumes, Metaverses, Network bottlenecks, Reinforcement Learning, Through current
@article{zhang_generative-ai_2025,
title = {Generative-AI for XR Content Transmission in the Metaverse: Potential Approaches, Challenges, and a Generation-Driven Transmission Framework},
author = {Z. Zhang and Y. Jiang and X. Wei and M. Chen and H. Dong and S. Yu},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-86000661262&doi=10.1109%2fMNET.2025.3547385&partnerID=40&md5=1e00d40542ec58ef1489934abb2a990c},
doi = {10.1109/MNET.2025.3547385},
issn = {08908044 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Network},
abstract = {How to efficiently transmit large volumes of Extended Reality (XR) content through current networks has been a major bottleneck in realizing the Metaverse. The recently emerging Generative Artificial Intelligence (GAI) has already revolutionized various technological fields and provides promising solutions to this challenge. In this article, we first demonstrate current networks' bottlenecks for supporting XR content transmission in the Metaverse. Then, we explore the potential approaches and challenges of utilizing GAI to overcome these bottlenecks. To address these challenges, we propose a GAI-based XR content transmission framework which leverages a cloud-edge collaboration architecture. The cloud servers are responsible for storing and rendering the original XR content, while edge servers utilize GAI models to generate essential parts of XR content (e.g., subsequent frames, selected objects, etc.) when network resources are insufficient to transmit them. A Deep Reinforcement Learning (DRL)-based decision module is proposed to solve the decision-making problems. Our case study demonstrates that the proposed GAI-based transmission framework achieves a 2.8-fold increase in normal frame ratio (percentage of frames that meet the quality and latency requirements for XR content transmission) over baseline approaches, underscoring the potential of GAI models to facilitate XR content transmission in the Metaverse. © 2025 IEEE.},
keywords = {'current, Cloud servers, Collaboration architecture, Content transmission, Decision making, Deep learning, Deep reinforcement learning, Edge server, Generative adversarial networks, Intelligence models, Large volumes, Metaverses, Network bottlenecks, Reinforcement Learning, Through current},
pubstate = {published},
tppubtype = {article}
}
How to efficiently transmit large volumes of Extended Reality (XR) content through current networks has been a major bottleneck in realizing the Metaverse. The recently emerging Generative Artificial Intelligence (GAI) has already revolutionized various technological fields and provides promising solutions to this challenge. In this article, we first demonstrate current networks' bottlenecks for supporting XR content transmission in the Metaverse. Then, we explore the potential approaches and challenges of utilizing GAI to overcome these bottlenecks. To address these challenges, we propose a GAI-based XR content transmission framework which leverages a cloud-edge collaboration architecture. The cloud servers are responsible for storing and rendering the original XR content, while edge servers utilize GAI models to generate essential parts of XR content (e.g., subsequent frames, selected objects, etc.) when network resources are insufficient to transmit them. A Deep Reinforcement Learning (DRL)-based decision module is proposed to solve the decision-making problems. Our case study demonstrates that the proposed GAI-based transmission framework achieves a 2.8-fold increase in normal frame ratio (percentage of frames that meet the quality and latency requirements for XR content transmission) over baseline approaches, underscoring the potential of GAI models to facilitate XR content transmission in the Metaverse. © 2025 IEEE.
Zhang, Z.; Wang, J.; Chen, J.; Fang, Z.; Jiang, C.; Han, Z.
A Priority-Aware AI-Generated Content Resource Allocation Method for Multi-UAV Aided Metaverse Proceedings Article
In: IEEE Wireless Commun. Networking Conf. WCNC, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 15253511 (ISSN); 979-835036836-9 (ISBN).
Abstract | Links | BibTeX | Tags: Aerial vehicle, AI-generated content, AI-generated content (AIGC), Allocation methods, Content-resources, Diffusion Model, Drones, Metaverse, Metaverses, Priority-aware, Reinforcement Learning, Reinforcement learnings, Resource allocation, Resources allocation, Target drones, Unmanned aerial vehicle, Unmanned aerial vehicle (UAV)
@inproceedings{zhang_priority-aware_2025,
title = {A Priority-Aware AI-Generated Content Resource Allocation Method for Multi-UAV Aided Metaverse},
author = {Z. Zhang and J. Wang and J. Chen and Z. Fang and C. Jiang and Z. Han},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105006408540&doi=10.1109%2fWCNC61545.2025.10978443&partnerID=40&md5=69937c6fa9be1a038b28e7884dfe586b},
doi = {10.1109/WCNC61545.2025.10978443},
isbn = {15253511 (ISSN); 979-835036836-9 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {IEEE Wireless Commun. Networking Conf. WCNC},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {With the advancement of large model technologies, AI -generated content is gradually emerging as a mainstream method for content creation. The metaverse, as a key application scenario for the next-generation communication technologies, heavily depends on advanced content generation technologies. Nevertheless, the diverse types of metaverse applications and their stringent real-time requirements constrain the full potential of AIGC technologies within this environment. In order to tackle with this problem, we construct a priority-aware multi-UAV aided metaverse system and formulate it as a Markov decision process (MDP). We propose a diffusion-based reinforcement learning algorithm to solve the resource allocation problem and demonstrate its superiority through enough comparison and ablation experiments. © 2025 IEEE.},
keywords = {Aerial vehicle, AI-generated content, AI-generated content (AIGC), Allocation methods, Content-resources, Diffusion Model, Drones, Metaverse, Metaverses, Priority-aware, Reinforcement Learning, Reinforcement learnings, Resource allocation, Resources allocation, Target drones, Unmanned aerial vehicle, Unmanned aerial vehicle (UAV)},
pubstate = {published},
tppubtype = {inproceedings}
}
With the advancement of large model technologies, AI -generated content is gradually emerging as a mainstream method for content creation. The metaverse, as a key application scenario for the next-generation communication technologies, heavily depends on advanced content generation technologies. Nevertheless, the diverse types of metaverse applications and their stringent real-time requirements constrain the full potential of AIGC technologies within this environment. In order to tackle with this problem, we construct a priority-aware multi-UAV aided metaverse system and formulate it as a Markov decision process (MDP). We propose a diffusion-based reinforcement learning algorithm to solve the resource allocation problem and demonstrate its superiority through enough comparison and ablation experiments. © 2025 IEEE.
Zhang, Z.; Wang, J.; Chen, J.; Fu, H.; Tong, Z.; Jiang, C.
Diffusion-Based Reinforcement Learning for Cooperative Offloading and Resource Allocation in Multi-UAV Assisted Edge-Enabled Metaverse Journal Article
In: IEEE Transactions on Vehicular Technology, 2025, ISSN: 00189545 (ISSN).
Abstract | Links | BibTeX | Tags: Aerial vehicle, Content creation, Content services, Contrastive Learning, Decision making, Deep learning, Deep reinforcement learning, Diffusion Model, Global industry, Helicopter services, Markov processes, Metaverse, Metaverses, Reinforcement Learning, Reinforcement learnings, Resource allocation, Resources allocation, Typical application, Unmanned aerial vehicle, Unmanned aerial vehicle (UAV), Unmanned aerial vehicles (UAV)
@article{zhang_diffusion-based_2025,
title = {Diffusion-Based Reinforcement Learning for Cooperative Offloading and Resource Allocation in Multi-UAV Assisted Edge-Enabled Metaverse},
author = {Z. Zhang and J. Wang and J. Chen and H. Fu and Z. Tong and C. Jiang},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85219108203&doi=10.1109%2fTVT.2025.3544879&partnerID=40&md5=fdbe1554f6cf7d47d4bbbb73b4b0d487},
doi = {10.1109/TVT.2025.3544879},
issn = {00189545 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IEEE Transactions on Vehicular Technology},
abstract = {As one of the typical applications of 6G, the metaverse, with its superior immersion and diversified services, has garnered widespread attention from both the global industry and academia. Simultaneously, the emergence of AI-generated content (AIGC), exemplified by ChatGPT, has revolutionized the mean of content creation in the metaverse. Providing meataverse users with diversified AIGC services anytime and anywhere to meet the demand for immersive and blended virtual-real experiences in the physical world has become a major challenge in the development of the metaverse. Considering the flexibility and mobility of unmanned aerial vehicles (UAVs), we innovatively incorporate multiple UAVs as one of the AIGC service providers and construct a multi-UAV assisted edge-enabled metaverse system in the context of AIGC-as-a-Service (AaaS) scenario. To solve the complex resource management and allocation problem in the aforementioned system, we formulate it as a Markov decision process (MDP) and propose utilizing the generative capabilities of the diffusion model in combination with the robust decision-making abilities of reinforcement learning to tackle these issues. In order to substantiate the efficacy of the proposed diffusion-based reinforcement learning framework, we propose a novel diffusion-based soft actor-critic algorithm for metaverse (Meta-DSAC). Subsequently, a series of experiments are executed and the simulation results empirically validate the proposed algorithm's comparative advantages of the ability to provide stable and substantial long-term rewards, as well as the enhanced capacity to model complex environment. © 2025 IEEE.},
keywords = {Aerial vehicle, Content creation, Content services, Contrastive Learning, Decision making, Deep learning, Deep reinforcement learning, Diffusion Model, Global industry, Helicopter services, Markov processes, Metaverse, Metaverses, Reinforcement Learning, Reinforcement learnings, Resource allocation, Resources allocation, Typical application, Unmanned aerial vehicle, Unmanned aerial vehicle (UAV), Unmanned aerial vehicles (UAV)},
pubstate = {published},
tppubtype = {article}
}
As one of the typical applications of 6G, the metaverse, with its superior immersion and diversified services, has garnered widespread attention from both the global industry and academia. Simultaneously, the emergence of AI-generated content (AIGC), exemplified by ChatGPT, has revolutionized the mean of content creation in the metaverse. Providing meataverse users with diversified AIGC services anytime and anywhere to meet the demand for immersive and blended virtual-real experiences in the physical world has become a major challenge in the development of the metaverse. Considering the flexibility and mobility of unmanned aerial vehicles (UAVs), we innovatively incorporate multiple UAVs as one of the AIGC service providers and construct a multi-UAV assisted edge-enabled metaverse system in the context of AIGC-as-a-Service (AaaS) scenario. To solve the complex resource management and allocation problem in the aforementioned system, we formulate it as a Markov decision process (MDP) and propose utilizing the generative capabilities of the diffusion model in combination with the robust decision-making abilities of reinforcement learning to tackle these issues. In order to substantiate the efficacy of the proposed diffusion-based reinforcement learning framework, we propose a novel diffusion-based soft actor-critic algorithm for metaverse (Meta-DSAC). Subsequently, a series of experiments are executed and the simulation results empirically validate the proposed algorithm's comparative advantages of the ability to provide stable and substantial long-term rewards, as well as the enhanced capacity to model complex environment. © 2025 IEEE.
2024
Liew, Z. Q.; Xu, M.; Lim, W. Y. Bryan; Niyato, D.; Kim, D. I.
AI-Generated Bidding for Immersive AIGC Services in Mobile Edge-Empowered Metaverse Proceedings Article
In: Int. Conf. Inf. Networking, pp. 305–309, IEEE Computer Society, 2024, ISBN: 19767684 (ISSN); 979-835033094-6 (ISBN).
Abstract | Links | BibTeX | Tags: Artificial intelligence generated bid, Artificial intelligence generated content, Bidding mechanism, Bidding models, Budget constraint, Budget control, Budget-constraint bidding, Constrained optimization, Content services, Immersive, Learning systems, Metaverses, Mobile edge computing, Reinforcement Learning, Semantics, Virtual tour
@inproceedings{liew_ai-generated_2024,
title = {AI-Generated Bidding for Immersive AIGC Services in Mobile Edge-Empowered Metaverse},
author = {Z. Q. Liew and M. Xu and W. Y. Bryan Lim and D. Niyato and D. I. Kim},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85198324990&doi=10.1109%2fICOIN59985.2024.10572159&partnerID=40&md5=271f5c45e8e95f01b42acaee89599bd5},
doi = {10.1109/ICOIN59985.2024.10572159},
isbn = {19767684 (ISSN); 979-835033094-6 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Int. Conf. Inf. Networking},
pages = {305–309},
publisher = {IEEE Computer Society},
abstract = {Recent advancements in Artificial Intelligence Generated Content (AIGC) provide personalized and immersive content generation services for applications such as interactive advertisements, virtual tours, and metaverse. With the use of mobile edge computing (MEC), buyers can bid for the AIGC service to enhance their user experience in real-time. However, designing strategies to optimize the quality of the services won can be challenging for budget-constrained buyers. The performance of classical bidding mechanisms is limited by the fixed rules in the strategies. To this end, we propose AI-generated bidding (AIGB) to optimize the bidding strategies for AIGC. AIGB model uses reinforcement learning model to generate bids for the services by learning from the historical data and environment states such as remaining budget, budget consumption rate, and quality of the won services. To obtain quality AIGC service, we propose a semantic aware reward function for the AIGB model. The proposed model is tested with a real-world dataset and experiments show that our model outperforms the classical bidding mechanism in terms of the number of services won and the similarity score. © 2024 IEEE.},
keywords = {Artificial intelligence generated bid, Artificial intelligence generated content, Bidding mechanism, Bidding models, Budget constraint, Budget control, Budget-constraint bidding, Constrained optimization, Content services, Immersive, Learning systems, Metaverses, Mobile edge computing, Reinforcement Learning, Semantics, Virtual tour},
pubstate = {published},
tppubtype = {inproceedings}
}
Recent advancements in Artificial Intelligence Generated Content (AIGC) provide personalized and immersive content generation services for applications such as interactive advertisements, virtual tours, and metaverse. With the use of mobile edge computing (MEC), buyers can bid for the AIGC service to enhance their user experience in real-time. However, designing strategies to optimize the quality of the services won can be challenging for budget-constrained buyers. The performance of classical bidding mechanisms is limited by the fixed rules in the strategies. To this end, we propose AI-generated bidding (AIGB) to optimize the bidding strategies for AIGC. AIGB model uses reinforcement learning model to generate bids for the services by learning from the historical data and environment states such as remaining budget, budget consumption rate, and quality of the won services. To obtain quality AIGC service, we propose a semantic aware reward function for the AIGB model. The proposed model is tested with a real-world dataset and experiments show that our model outperforms the classical bidding mechanism in terms of the number of services won and the similarity score. © 2024 IEEE.
You, F.; Du, H.; Kang, J.; Ni, W.; Niyato, D.; Jamalipour, A.
Generative AI-aided Reinforcement Learning for Computation Offloading and Privacy Protection in VR-based Multi-Access Edge Computing Proceedings Article
In: Proc. - IEEE Smart World Congr., SWC - IEEE Ubiquitous Intell. Comput., Auton. Trusted Comput., Digit. Twin, Metaverse, Priv. Comput. Data Secur., Scalable Comput. Commun., pp. 2209–2214, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-833152086-1 (ISBN).
Abstract | Links | BibTeX | Tags: Computation offloading, Content services, Differential privacy, Diffusion Model, Edge computing, Generative adversarial networks, Generative diffusion model, generative diffusion models, Inverse problems, Multi-access edge computing, Multiaccess, Policy optimization, Proximal policy optimization, Reinforcement Learning, User privacy, Virtual environments, Virtual Reality
@inproceedings{you_generative_2024,
title = {Generative AI-aided Reinforcement Learning for Computation Offloading and Privacy Protection in VR-based Multi-Access Edge Computing},
author = {F. You and H. Du and J. Kang and W. Ni and D. Niyato and A. Jamalipour},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105002235341&doi=10.1109%2fSWC62898.2024.00337&partnerID=40&md5=301c91c0b737c401d54e154e13dc8d47},
doi = {10.1109/SWC62898.2024.00337},
isbn = {979-833152086-1 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - IEEE Smart World Congr., SWC - IEEE Ubiquitous Intell. Comput., Auton. Trusted Comput., Digit. Twin, Metaverse, Priv. Comput. Data Secur., Scalable Comput. Commun.},
pages = {2209–2214},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {The rapid growth of Artificial Intelligence-Generated Content (AIGC) services has led to increased mobile user participation in related computations and interactions. This development has enabled AI-generated characters to interact with Virtual Reality (VR) users in real time, making the VR experience more interactive and personalized. In this paper, we consider an MEC system where VR users engage in AIGC services, focusing on the Generative Diffusion Model (GDM)based image generation tasks. Specifically, VR users initiate requests for computing resources, while computation offloading distributes the processing load across the MEC system. To manage AIGC edge computation offloading and cloudlet-VR user connections jointly, a Data Center Operator (DCO) employs a centralized Proximal Policy Optimization (PPO) algorithm. To protect VR users' privacy while preserving PPO functionality, we employ the Generative Diffusion Model (GDM), specifically the Denoising Diffusion Implicit Model (DDIM), which first introduces noise to the PPO state, then conducts a denoising process to recover the state information. We further employ Inverse Reinforcement Learning (IRL) to infer rewards for the recovered states, using expert demonstrations trained by the PPO. The similarity between PPO-generated rewards and IRL-inferred rewards is then computed. Simulation results demonstrate that our proposed approach successfully achieves computation offloading while protecting VR users' privacy within the PPO centralized management framework. © 2024 IEEE.},
keywords = {Computation offloading, Content services, Differential privacy, Diffusion Model, Edge computing, Generative adversarial networks, Generative diffusion model, generative diffusion models, Inverse problems, Multi-access edge computing, Multiaccess, Policy optimization, Proximal policy optimization, Reinforcement Learning, User privacy, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
The rapid growth of Artificial Intelligence-Generated Content (AIGC) services has led to increased mobile user participation in related computations and interactions. This development has enabled AI-generated characters to interact with Virtual Reality (VR) users in real time, making the VR experience more interactive and personalized. In this paper, we consider an MEC system where VR users engage in AIGC services, focusing on the Generative Diffusion Model (GDM)based image generation tasks. Specifically, VR users initiate requests for computing resources, while computation offloading distributes the processing load across the MEC system. To manage AIGC edge computation offloading and cloudlet-VR user connections jointly, a Data Center Operator (DCO) employs a centralized Proximal Policy Optimization (PPO) algorithm. To protect VR users' privacy while preserving PPO functionality, we employ the Generative Diffusion Model (GDM), specifically the Denoising Diffusion Implicit Model (DDIM), which first introduces noise to the PPO state, then conducts a denoising process to recover the state information. We further employ Inverse Reinforcement Learning (IRL) to infer rewards for the recovered states, using expert demonstrations trained by the PPO. The similarity between PPO-generated rewards and IRL-inferred rewards is then computed. Simulation results demonstrate that our proposed approach successfully achieves computation offloading while protecting VR users' privacy within the PPO centralized management framework. © 2024 IEEE.
Ding, P.; Liu, J.; Sun, M.; Li, L.; Liu, H.
Enhancing Computational Processing Performance for Generative AI Large Models with Autonomous Decision-Making in Metaverse Applications Proceedings Article
In: Proc. - IEEE Int. Conf. Metaverse Comput., Netw., Appl., MetaCom, pp. 253–258, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-833151599-7 (ISBN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, AGI (Artificial General Intelligence), Artificial general intelligence, Artificial general intelligences, Autonomous decision, Autonomous Decision-Making, Data assimilation, Data integration, Decisions makings, Digital Twin Technology, Emotion Recognition, Generative adversarial networks, Generative AI large model, Generative AI Large Models, Large models, Metaverse, Metaverses, Model Acceleration, Model Compression, Multi agent systems, Multi-agent systems, Multi-modal data, Multi-Modal Data Integration, Multiagent systems (MASs), Reinforcement Learning, Reinforcement learnings, Spatio-temporal data
@inproceedings{ding_enhancing_2024,
title = {Enhancing Computational Processing Performance for Generative AI Large Models with Autonomous Decision-Making in Metaverse Applications},
author = {P. Ding and J. Liu and M. Sun and L. Li and H. Liu},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85211489063&doi=10.1109%2fMetaCom62920.2024.00048&partnerID=40&md5=ae085a7d90b12c9090f5bf7a274bc7ce},
doi = {10.1109/MetaCom62920.2024.00048},
isbn = {979-833151599-7 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - IEEE Int. Conf. Metaverse Comput., Netw., Appl., MetaCom},
pages = {253–258},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {We explore how to enhance the computational processing performance for generative AI large models with autonomous decision-making in metaverse applications. We first introduce the relationship between AI large models and the Metaverse. We elaborate on the application scenarios of generative AI large models in Metaverse, including real-time weather simulation, embodied intelligence of agents, dynamic environment interaction, and user emotion recognition. We then propose the method of Multi-Dimensional Optimization Generation Framework (MDOGF) to improve computational processing performance. The experiment results show great improvement in computational processing performance. © 2024 IEEE.},
keywords = {Adversarial machine learning, AGI (Artificial General Intelligence), Artificial general intelligence, Artificial general intelligences, Autonomous decision, Autonomous Decision-Making, Data assimilation, Data integration, Decisions makings, Digital Twin Technology, Emotion Recognition, Generative adversarial networks, Generative AI large model, Generative AI Large Models, Large models, Metaverse, Metaverses, Model Acceleration, Model Compression, Multi agent systems, Multi-agent systems, Multi-modal data, Multi-Modal Data Integration, Multiagent systems (MASs), Reinforcement Learning, Reinforcement learnings, Spatio-temporal data},
pubstate = {published},
tppubtype = {inproceedings}
}
We explore how to enhance the computational processing performance for generative AI large models with autonomous decision-making in metaverse applications. We first introduce the relationship between AI large models and the Metaverse. We elaborate on the application scenarios of generative AI large models in Metaverse, including real-time weather simulation, embodied intelligence of agents, dynamic environment interaction, and user emotion recognition. We then propose the method of Multi-Dimensional Optimization Generation Framework (MDOGF) to improve computational processing performance. The experiment results show great improvement in computational processing performance. © 2024 IEEE.
Bao, Y.; Gao, N.; Weng, D.; Chen, J.; Tian, Z.
MuseGesture: A Framework for Gesture Synthesis by Virtual Agents in VR Museum Guides Proceedings Article
In: U., Eck; M., Sra; J., Stefanucci; M., Sugimoto; M., Tatzgern; I., Williams (Ed.): Proc. - IEEE Int. Symp. Mixed Augment. Real. Adjunct, ISMAR-Adjunct, pp. 337–338, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-833150691-9 (ISBN).
Abstract | Links | BibTeX | Tags: Adversarial machine learning, Embeddings, Gesture Generation, Intelligent Agents, Intelligent systems, Intelligent virtual agents, Language generation, Language Model, Large language model, large language models, Museum guide, Reinforcement Learning, Reinforcement learnings, Robust language understanding, Virtual agent, Virtual Agents, Virtual environments, Virtual reality museum guide, VR Museum Guides
@inproceedings{bao_musegesture_2024,
title = {MuseGesture: A Framework for Gesture Synthesis by Virtual Agents in VR Museum Guides},
author = {Y. Bao and N. Gao and D. Weng and J. Chen and Z. Tian},
editor = {Eck U. and Sra M. and Stefanucci J. and Sugimoto M. and Tatzgern M. and Williams I.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85214385900&doi=10.1109%2fISMAR-Adjunct64951.2024.00079&partnerID=40&md5=e71ffc28e299597557034259aab50641},
doi = {10.1109/ISMAR-Adjunct64951.2024.00079},
isbn = {979-833150691-9 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - IEEE Int. Symp. Mixed Augment. Real. Adjunct, ISMAR-Adjunct},
pages = {337–338},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {This paper presents an innovative framework named MuseGesture, designed to generate contextually adaptive gestures for virtual agents in Virtual Reality (VR) museums. The framework leverages the robust language understanding and generation capabilities of Large Language Models (LLMs) to parse tour narration texts and generate corresponding explanatory gestures. Through reinforcement learning and adversarial skill embeddings, the framework also generates guiding gestures tailored to the virtual museum environment, integrating both gesture types using conditional motion interpolation methods. Experimental results and user studies demonstrate that this approach effectively enables voice-command-controlled virtual guide gestures, offering a novel intelligent guiding system solution that enhances the interactive experience in VR museum environments. © 2024 IEEE.},
keywords = {Adversarial machine learning, Embeddings, Gesture Generation, Intelligent Agents, Intelligent systems, Intelligent virtual agents, Language generation, Language Model, Large language model, large language models, Museum guide, Reinforcement Learning, Reinforcement learnings, Robust language understanding, Virtual agent, Virtual Agents, Virtual environments, Virtual reality museum guide, VR Museum Guides},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper presents an innovative framework named MuseGesture, designed to generate contextually adaptive gestures for virtual agents in Virtual Reality (VR) museums. The framework leverages the robust language understanding and generation capabilities of Large Language Models (LLMs) to parse tour narration texts and generate corresponding explanatory gestures. Through reinforcement learning and adversarial skill embeddings, the framework also generates guiding gestures tailored to the virtual museum environment, integrating both gesture types using conditional motion interpolation methods. Experimental results and user studies demonstrate that this approach effectively enables voice-command-controlled virtual guide gestures, offering a novel intelligent guiding system solution that enhances the interactive experience in VR museum environments. © 2024 IEEE.
Zheng, P.; Li, C.; Fan, J.; Wang, L.
In: CIRP Annals, vol. 73, no. 1, pp. 341–344, 2024, ISSN: 00078506 (ISSN).
Abstract | Links | BibTeX | Tags: Collaboration task, Collaborative manufacturing, Deep learning, Helmet mounted displays, Human robots, Human-centric, Human-guided robot learning, Human-Robot Collaboration, Interface states, Manipulators, Manufacturing system, Manufacturing tasks, Mixed reality, Mixed reality head-mounted displays, Reinforcement Learning, Reinforcement learnings, Robot vision, Smart manufacturing
@article{zheng_vision-language-guided_2024,
title = {A vision-language-guided and deep reinforcement learning-enabled approach for unstructured human-robot collaborative manufacturing task fulfilment},
author = {P. Zheng and C. Li and J. Fan and L. Wang},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85190754943&doi=10.1016%2fj.cirp.2024.04.003&partnerID=40&md5=59c453e1931e912472e76b86b77a881b},
doi = {10.1016/j.cirp.2024.04.003},
issn = {00078506 (ISSN)},
year = {2024},
date = {2024-01-01},
journal = {CIRP Annals},
volume = {73},
number = {1},
pages = {341–344},
abstract = {Human-Robot Collaboration (HRC) has emerged as a pivot in contemporary human-centric smart manufacturing scenarios. However, the fulfilment of HRC tasks in unstructured scenes brings many challenges to be overcome. In this work, mixed reality head-mounted display is modelled as an effective data collection, communication, and state representation interface/tool for HRC task settings. By integrating vision-language cues with large language model, a vision-language-guided HRC task planning approach is firstly proposed. Then, a deep reinforcement learning-enabled mobile manipulator motion control policy is generated to fulfil HRC task primitives. Its feasibility is demonstrated in several HRC unstructured manufacturing tasks with comparative results. © 2024 The Author(s)},
keywords = {Collaboration task, Collaborative manufacturing, Deep learning, Helmet mounted displays, Human robots, Human-centric, Human-guided robot learning, Human-Robot Collaboration, Interface states, Manipulators, Manufacturing system, Manufacturing tasks, Mixed reality, Mixed reality head-mounted displays, Reinforcement Learning, Reinforcement learnings, Robot vision, Smart manufacturing},
pubstate = {published},
tppubtype = {article}
}
Human-Robot Collaboration (HRC) has emerged as a pivot in contemporary human-centric smart manufacturing scenarios. However, the fulfilment of HRC tasks in unstructured scenes brings many challenges to be overcome. In this work, mixed reality head-mounted display is modelled as an effective data collection, communication, and state representation interface/tool for HRC task settings. By integrating vision-language cues with large language model, a vision-language-guided HRC task planning approach is firstly proposed. Then, a deep reinforcement learning-enabled mobile manipulator motion control policy is generated to fulfil HRC task primitives. Its feasibility is demonstrated in several HRC unstructured manufacturing tasks with comparative results. © 2024 The Author(s)
2023
Augello, Agnese; Gaglio, Salvatore; Infantino, Ignazio; Maniscalco, Umberto; Pilato, Giovanni; Vella, Filippo
Roboception and Adaptation in a Cognitive Robot Journal Article
In: Robotics and autonomous systems (Print), pp. 104400, 2023, ISSN: 0921-8890.
Abstract | Links | BibTeX | Tags: Cognitive Architectures, Humanoid Robots, Reinforcement Learning, Roboceptions, Sensor systems, Social Robots
@article{augelloRoboceptionAdaptationCognitive2023,
title = {Roboception and Adaptation in a Cognitive Robot},
author = { Agnese Augello and Salvatore Gaglio and Ignazio Infantino and Umberto Maniscalco and Giovanni Pilato and Filippo Vella},
doi = {10.1016/j.robot.2023.104400},
issn = {0921-8890},
year = {2023},
date = {2023-01-01},
journal = {Robotics and autonomous systems (Print)},
pages = {104400},
abstract = {In robotics, perception is usually oriented at understanding what is happening in the external world, while few works pay attention to what is occurring in the robot's body. In this work, we propose an artificial somatosensory system, embedded in a cognitive architecture, that enables a robot to perceive the sensations from its embodiment while executing a task. We called these perceptions roboceptions, and they let the robot act according to its own physical needs in addition to the task demands. Physical information is processed by the robot to behave in a balanced way, determining the most appropriate trade-off between the achievement of the task and its well being. The experiments show the integration of information from the somatosensory system and the choices that lead to the accomplishment of the task.},
keywords = {Cognitive Architectures, Humanoid Robots, Reinforcement Learning, Roboceptions, Sensor systems, Social Robots},
pubstate = {published},
tppubtype = {article}
}
In robotics, perception is usually oriented at understanding what is happening in the external world, while few works pay attention to what is occurring in the robot's body. In this work, we propose an artificial somatosensory system, embedded in a cognitive architecture, that enables a robot to perceive the sensations from its embodiment while executing a task. We called these perceptions roboceptions, and they let the robot act according to its own physical needs in addition to the task demands. Physical information is processed by the robot to behave in a balanced way, determining the most appropriate trade-off between the achievement of the task and its well being. The experiments show the integration of information from the somatosensory system and the choices that lead to the accomplishment of the task.
Augello, Agnese; Gaglio, Salvatore; Infantino, Ignazio; Maniscalco, Umberto; Pilato, Giovanni; Vella, Filippo
Roboception and adaptation in a cognitive robot Journal Article
In: Robotics and autonomous systems (Print), pp. 104400, 2023, ISSN: 0921-8890.
Abstract | Links | BibTeX | Tags: Cognitive Architectures, Humanoid Robots, Reinforcement Learning, Roboceptions, Sensor systems, Social Robots
@article{augello_roboception_2023,
title = {Roboception and adaptation in a cognitive robot},
author = {Agnese Augello and Salvatore Gaglio and Ignazio Infantino and Umberto Maniscalco and Giovanni Pilato and Filippo Vella},
url = {https://www.sciencedirect.com/science/article/pii/S0921889023000398},
doi = {10.1016/j.robot.2023.104400},
issn = {0921-8890},
year = {2023},
date = {2023-01-01},
journal = {Robotics and autonomous systems (Print)},
pages = {104400},
abstract = {In robotics, perception is usually oriented at understanding what is happening in the external world, while few works pay attention to what is occurring in the robotś body. In this work, we propose an artificial somatosensory system, embedded in a cognitive architecture, that enables a robot to perceive the sensations from its embodiment while executing a task. We called these perceptions roboceptions, and they let the robot act according to its own physical needs in addition to the task demands. Physical information is processed by the robot to behave in a balanced way, determining the most appropriate trade-off between the achievement of the task and its well being. The experiments show the integration of information from the somatosensory system and the choices that lead to the accomplishment of the task.},
keywords = {Cognitive Architectures, Humanoid Robots, Reinforcement Learning, Roboceptions, Sensor systems, Social Robots},
pubstate = {published},
tppubtype = {article}
}
In robotics, perception is usually oriented at understanding what is happening in the external world, while few works pay attention to what is occurring in the robotś body. In this work, we propose an artificial somatosensory system, embedded in a cognitive architecture, that enables a robot to perceive the sensations from its embodiment while executing a task. We called these perceptions roboceptions, and they let the robot act according to its own physical needs in addition to the task demands. Physical information is processed by the robot to behave in a balanced way, determining the most appropriate trade-off between the achievement of the task and its well being. The experiments show the integration of information from the somatosensory system and the choices that lead to the accomplishment of the task.
Park, J.; Choi, J.; Kim, S. -L.; Bennis, M.
Enabling the Wireless Metaverse via Semantic Multiverse Communication Proceedings Article
In: Annu. IEEE Commun.Soc. Conf. Sens., Mesh Ad Hoc Commun. Netw. workshops, pp. 85–90, IEEE Computer Society, 2023, ISBN: 21555486 (ISSN); 979-835030052-9 (ISBN).
Abstract | Links | BibTeX | Tags: Deep learning, Extended reality (XR), Federated learning, Fertilizers, Learn+, Learning systems, Metaverse, Metaverses, Modal analysis, Multi agent systems, Multi-agent reinforcement learning, Multi-modal data, Reinforcement Learning, Semantic communication, Semantics, Signal encoding, Signaling game, Split learning, Symbolic artificial intelligence
@inproceedings{park_enabling_2023,
title = {Enabling the Wireless Metaverse via Semantic Multiverse Communication},
author = {J. Park and J. Choi and S. -L. Kim and M. Bennis},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85177465286&doi=10.1109%2fSECON58729.2023.10287438&partnerID=40&md5=b052572fb2f78ce0694c7ae5726c8daf},
doi = {10.1109/SECON58729.2023.10287438},
isbn = {21555486 (ISSN); 979-835030052-9 (ISBN)},
year = {2023},
date = {2023-01-01},
booktitle = {Annu. IEEE Commun.Soc. Conf. Sens., Mesh Ad Hoc Commun. Netw. workshops},
volume = {2023-September},
pages = {85–90},
publisher = {IEEE Computer Society},
abstract = {Metaverse over wireless networks is an emerging use case of the sixth generation (6G) wireless systems, posing unprecedented challenges in terms of its multi-modal data transmissions with stringent latency and reliability requirements. Towards enabling this wireless metaverse, in this article we propose a novel semantic communication (SC) framework by decomposing the metaverse into human/machine agent-specific semantic multiverses (SMs). An SM stored at each agent comprises a semantic encoder and a generator, leveraging recent advances in generative artificial intelligence (AI). To improve communication efficiency, the encoder learns the semantic representations (SRs) of multi-modal data, while the generator learns how to manipulate them for locally rendering scenes and interactions in the metaverse. Since these learned SMs are biased towards local environments, their success hinges on synchronizing heterogeneous SMs in the background while communicating SRs in the foreground, turning the wireless metaverse problem into the problem of semantic multiverse communication (SMC). Based on this SMC architecture, we propose several promising algorithmic and analytic tools for modeling and designing SMC, ranging from distributed learning and multi-agent reinforcement learning (MARL) to signaling games and symbolic AI. © 2023 IEEE.},
keywords = {Deep learning, Extended reality (XR), Federated learning, Fertilizers, Learn+, Learning systems, Metaverse, Metaverses, Modal analysis, Multi agent systems, Multi-agent reinforcement learning, Multi-modal data, Reinforcement Learning, Semantic communication, Semantics, Signal encoding, Signaling game, Split learning, Symbolic artificial intelligence},
pubstate = {published},
tppubtype = {inproceedings}
}
Metaverse over wireless networks is an emerging use case of the sixth generation (6G) wireless systems, posing unprecedented challenges in terms of its multi-modal data transmissions with stringent latency and reliability requirements. Towards enabling this wireless metaverse, in this article we propose a novel semantic communication (SC) framework by decomposing the metaverse into human/machine agent-specific semantic multiverses (SMs). An SM stored at each agent comprises a semantic encoder and a generator, leveraging recent advances in generative artificial intelligence (AI). To improve communication efficiency, the encoder learns the semantic representations (SRs) of multi-modal data, while the generator learns how to manipulate them for locally rendering scenes and interactions in the metaverse. Since these learned SMs are biased towards local environments, their success hinges on synchronizing heterogeneous SMs in the background while communicating SRs in the foreground, turning the wireless metaverse problem into the problem of semantic multiverse communication (SMC). Based on this SMC architecture, we propose several promising algorithmic and analytic tools for modeling and designing SMC, ranging from distributed learning and multi-agent reinforcement learning (MARL) to signaling games and symbolic AI. © 2023 IEEE.
2020
Augello, Agnese; Infantino, Ignazio; Gaglio, Salvatore; Maniscalco, Umberto; Pilato, Giovanni; Vella, Filippo
An Artificial Soft Somatosensory System for a Cognitive Robot Proceedings Article
In: Proceedings - 4th IEEE International Conference on Robotic Computing, IRC 2020, pp. 319–326, Institute of Electrical and Electronics Engineers Inc., 2020, ISBN: 978-1-72815-237-0.
Abstract | Links | BibTeX | Tags: Cognitive Architectures, Reinforcement Learning, Robotics, Social Robots, Somatosensory Systems
@inproceedings{augelloArtificialSoftSomatosensory2020,
title = {An Artificial Soft Somatosensory System for a Cognitive Robot},
author = { Agnese Augello and Ignazio Infantino and Salvatore Gaglio and Umberto Maniscalco and Giovanni Pilato and Filippo Vella},
doi = {10.1109/IRC.2020.00058},
isbn = {978-1-72815-237-0},
year = {2020},
date = {2020-01-01},
booktitle = {Proceedings - 4th IEEE International Conference on Robotic Computing, IRC 2020},
pages = {319--326},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {The paper proposes an artificial somatosensory system loosely inspired by human beings' biology and embedded in a cognitive architecture (CA). It enables a robot to receive the stimulation from its embodiment, and use these sensations, we called roboceptions, to behave according to both the external environment and the internal robot status. In such a way, the robot is aware of its body and able to interpret physical sensations can be more effective in the task while maintaining its well being. The robot's physiological urges are tightly bound to the specific physical state of the robot. Positive and negative physical information can, therefore, be processed and let the robot behave in a more realistic way adopting the right trade-off between the achievement of the task and the well-being of the robot. This goal has been achieved through a reinforcement learning approach. To test these statements we considered, as a test-bench, the execution of working performances with an SoftBank NAO robot that are modulated according its body well-being. textcopyright 2020 IEEE.},
keywords = {Cognitive Architectures, Reinforcement Learning, Robotics, Social Robots, Somatosensory Systems},
pubstate = {published},
tppubtype = {inproceedings}
}
The paper proposes an artificial somatosensory system loosely inspired by human beings' biology and embedded in a cognitive architecture (CA). It enables a robot to receive the stimulation from its embodiment, and use these sensations, we called roboceptions, to behave according to both the external environment and the internal robot status. In such a way, the robot is aware of its body and able to interpret physical sensations can be more effective in the task while maintaining its well being. The robot's physiological urges are tightly bound to the specific physical state of the robot. Positive and negative physical information can, therefore, be processed and let the robot behave in a more realistic way adopting the right trade-off between the achievement of the task and the well-being of the robot. This goal has been achieved through a reinforcement learning approach. To test these statements we considered, as a test-bench, the execution of working performances with an SoftBank NAO robot that are modulated according its body well-being. textcopyright 2020 IEEE.
Augello, Agnese; Infantino, Ignazio; Gaglio, Salvatore; Maniscalco, Umberto; Pilato, Giovanni; Vella, Filippo
An Artificial Soft Somatosensory System for a Cognitive Robot Proceedings Article
In: Proceedings - 4th IEEE International Conference on Robotic Computing, IRC 2020, pp. 319–326, Institute of Electrical and Electronics Engineers Inc., 2020, ISBN: 978-1-72815-237-0.
Abstract | Links | BibTeX | Tags: Cognitive Architectures, Reinforcement Learning, Robotics, Social Robots, Somatosensory Systems
@inproceedings{augello_artificial_2020,
title = {An Artificial Soft Somatosensory System for a Cognitive Robot},
author = {Agnese Augello and Ignazio Infantino and Salvatore Gaglio and Umberto Maniscalco and Giovanni Pilato and Filippo Vella},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85099360477&doi=10.1109%2fIRC.2020.00058&partnerID=40&md5=87b4c20a11e6bca2f17e6cf2758353f8},
doi = {10.1109/IRC.2020.00058},
isbn = {978-1-72815-237-0},
year = {2020},
date = {2020-01-01},
booktitle = {Proceedings - 4th IEEE International Conference on Robotic Computing, IRC 2020},
pages = {319–326},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {The paper proposes an artificial somatosensory system loosely inspired by human beings' biology and embedded in a cognitive architecture (CA). It enables a robot to receive the stimulation from its embodiment, and use these sensations, we called roboceptions, to behave according to both the external environment and the internal robot status. In such a way, the robot is aware of its body and able to interpret physical sensations can be more effective in the task while maintaining its well being. The robot's physiological urges are tightly bound to the specific physical state of the robot. Positive and negative physical information can, therefore, be processed and let the robot behave in a more realistic way adopting the right trade-off between the achievement of the task and the well-being of the robot. This goal has been achieved through a reinforcement learning approach. To test these statements we considered, as a test-bench, the execution of working performances with an SoftBank NAO robot that are modulated according its body well-being. © 2020 IEEE.},
keywords = {Cognitive Architectures, Reinforcement Learning, Robotics, Social Robots, Somatosensory Systems},
pubstate = {published},
tppubtype = {inproceedings}
}
The paper proposes an artificial somatosensory system loosely inspired by human beings' biology and embedded in a cognitive architecture (CA). It enables a robot to receive the stimulation from its embodiment, and use these sensations, we called roboceptions, to behave according to both the external environment and the internal robot status. In such a way, the robot is aware of its body and able to interpret physical sensations can be more effective in the task while maintaining its well being. The robot's physiological urges are tightly bound to the specific physical state of the robot. Positive and negative physical information can, therefore, be processed and let the robot behave in a more realistic way adopting the right trade-off between the achievement of the task and the well-being of the robot. This goal has been achieved through a reinforcement learning approach. To test these statements we considered, as a test-bench, the execution of working performances with an SoftBank NAO robot that are modulated according its body well-being. © 2020 IEEE.