AHCI RESEARCH GROUP
Publications
Papers published in international journals,
proceedings of conferences, workshops and books.
OUR RESEARCH
Scientific Publications
How to
Here you can find the complete list of our publications.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
2025
Xu, F.; Zhou, T.; Nguyen, T.; Bao, H.; Lin, C.; Du, J.
Integrating augmented reality and LLM for enhanced cognitive support in critical audio communications Journal Article
In: International Journal of Human Computer Studies, vol. 194, 2025, ISSN: 10715819 (ISSN).
Abstract | Links | BibTeX | Tags: Audio communications, Augmented Reality, Cognitive loads, Cognitive support, Decisions makings, Language Model, Large language model, LLM, Logic reasoning, Maintenance, Operations and maintenance, Oral communication, Situational awareness
@article{xu_integrating_2025,
title = {Integrating augmented reality and LLM for enhanced cognitive support in critical audio communications},
author = {F. Xu and T. Zhou and T. Nguyen and H. Bao and C. Lin and J. Du},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85208467299&doi=10.1016%2fj.ijhcs.2024.103402&partnerID=40&md5=153d095b837ee1666a7da0f7ed03362c},
doi = {10.1016/j.ijhcs.2024.103402},
issn = {10715819 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {International Journal of Human Computer Studies},
volume = {194},
abstract = {Operation and Maintenance (O&M) missions are often time-sensitive and accuracy-dependent, requiring rapid and precise information processing in noisy, chaotic environments where oral communication can lead to cognitive overload and impaired decision-making. Augmented Reality (AR) and Large Language Models (LLMs) offer potential for enhancing situational awareness and lowering cognitive load by integrating digital visualizations with the physical world and improving dialogue management. However, synthesizing these technologies into a real-time system that effectively aids operators remains a challenge. This study explores the integration of AR and GPT-4, an advanced LLM, in time-sensitive O&M tasks, aiming to enhance situational awareness and manage cognitive load during oral communications. A customized AR system, incorporating the Microsoft HoloLens2 for cognitive monitoring and GPT-4 for decision making assistance, was tested in a human subject experiment with 30 participants. The 2×2 factorial experiment evaluated the effects of AR and LLM assistance on task performance and cognitive load. Results demonstrated significant improvements in task accuracy and reductions in cognitive load, highlighting the effectiveness of AR and LLM integration in supporting O&M missions. These findings emphasize the need for further research to optimize operational strategies in mission critical environments. © 2024 Elsevier Ltd},
keywords = {Audio communications, Augmented Reality, Cognitive loads, Cognitive support, Decisions makings, Language Model, Large language model, LLM, Logic reasoning, Maintenance, Operations and maintenance, Oral communication, Situational awareness},
pubstate = {published},
tppubtype = {article}
}
Operation and Maintenance (O&M) missions are often time-sensitive and accuracy-dependent, requiring rapid and precise information processing in noisy, chaotic environments where oral communication can lead to cognitive overload and impaired decision-making. Augmented Reality (AR) and Large Language Models (LLMs) offer potential for enhancing situational awareness and lowering cognitive load by integrating digital visualizations with the physical world and improving dialogue management. However, synthesizing these technologies into a real-time system that effectively aids operators remains a challenge. This study explores the integration of AR and GPT-4, an advanced LLM, in time-sensitive O&M tasks, aiming to enhance situational awareness and manage cognitive load during oral communications. A customized AR system, incorporating the Microsoft HoloLens2 for cognitive monitoring and GPT-4 for decision making assistance, was tested in a human subject experiment with 30 participants. The 2×2 factorial experiment evaluated the effects of AR and LLM assistance on task performance and cognitive load. Results demonstrated significant improvements in task accuracy and reductions in cognitive load, highlighting the effectiveness of AR and LLM integration in supporting O&M missions. These findings emphasize the need for further research to optimize operational strategies in mission critical environments. © 2024 Elsevier Ltd
2024
Xu, F.; Nguyen, T.; Du, J.
Augmented Reality for Maintenance Tasks with ChatGPT for Automated Text-To-Action Journal Article
In: Journal of Construction Engineering and Management, vol. 150, no. 4, 2024, ISSN: 07339364 (ISSN).
Abstract | Links | BibTeX | Tags: Artificial intelligence systems, Augmented Reality, Augmented Reality (AR), ChatGPT, Complex sequences, Computational Linguistics, Diverse fields, Human like, Language Model, Maintenance, Maintenance tasks, Operations and maintenance, Optical character recognition, Sensor technologies, Virtual Reality
@article{xu_augmented_2024,
title = {Augmented Reality for Maintenance Tasks with ChatGPT for Automated Text-To-Action},
author = {F. Xu and T. Nguyen and J. Du},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85183669638&doi=10.1061%2fJCEMD4.COENG-14142&partnerID=40&md5=6b02d2f4f6e74a8152adf2eb30ee2d88},
doi = {10.1061/JCEMD4.COENG-14142},
issn = {07339364 (ISSN)},
year = {2024},
date = {2024-01-01},
journal = {Journal of Construction Engineering and Management},
volume = {150},
number = {4},
abstract = {Advancements in sensor technology, artificial intelligence (AI), and augmented reality (AR) have unlocked opportunities across various domains. AR and large language models like GPT have witnessed substantial progress and increasingly are being employed in diverse fields. One such promising application is in operations and maintenance (OM). OM tasks often involve complex procedures and sequences that can be challenging to memorize and execute correctly, particularly for novices or in high-stress situations. By combining the advantages of superimposing virtual objects onto the physical world and generating human-like text using GPT, we can revolutionize OM operations. This study introduces a system that combines AR, optical character recognition (OCR), and the GPT language model to optimize user performance while offering trustworthy interactions and alleviating workload in OM tasks. This system provides an interactive virtual environment controlled by the Unity game engine, facilitating a seamless interaction between virtual and physical realities. A case study (N=30) was conducted to illustrate the findings and answer the research questions. The Multidimensional Measurement of Trust (MDMT) was applied to understand the complexity of trust engagement with such a human-like system. The results indicate that users can complete similarly challenging tasks in less time using our proposed AR and AI system. Moreover, the collected data also suggest a reduction in cognitive load when executing the same operations using the AR and AI system. A divergence of trust was observed concerning capability and ethical dimensions. © 2024 American Society of Civil Engineers.},
keywords = {Artificial intelligence systems, Augmented Reality, Augmented Reality (AR), ChatGPT, Complex sequences, Computational Linguistics, Diverse fields, Human like, Language Model, Maintenance, Maintenance tasks, Operations and maintenance, Optical character recognition, Sensor technologies, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
Advancements in sensor technology, artificial intelligence (AI), and augmented reality (AR) have unlocked opportunities across various domains. AR and large language models like GPT have witnessed substantial progress and increasingly are being employed in diverse fields. One such promising application is in operations and maintenance (OM). OM tasks often involve complex procedures and sequences that can be challenging to memorize and execute correctly, particularly for novices or in high-stress situations. By combining the advantages of superimposing virtual objects onto the physical world and generating human-like text using GPT, we can revolutionize OM operations. This study introduces a system that combines AR, optical character recognition (OCR), and the GPT language model to optimize user performance while offering trustworthy interactions and alleviating workload in OM tasks. This system provides an interactive virtual environment controlled by the Unity game engine, facilitating a seamless interaction between virtual and physical realities. A case study (N=30) was conducted to illustrate the findings and answer the research questions. The Multidimensional Measurement of Trust (MDMT) was applied to understand the complexity of trust engagement with such a human-like system. The results indicate that users can complete similarly challenging tasks in less time using our proposed AR and AI system. Moreover, the collected data also suggest a reduction in cognitive load when executing the same operations using the AR and AI system. A divergence of trust was observed concerning capability and ethical dimensions. © 2024 American Society of Civil Engineers.