AHCI RESEARCH GROUP
Publications
Papers published in international journals,
proceedings of conferences, workshops and books.
OUR RESEARCH
Scientific Publications
How to
Here you can find the complete list of our publications.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
2025
Cao, X.; Ju, K. P.; Li, C.; Jain, D.
SceneGenA11y: How can Runtime Generative tools improve the Accessibility of a Virtual 3D Scene? Proceedings Article
In: Conf Hum Fact Comput Syst Proc, Association for Computing Machinery, 2025, ISBN: 979-840071395-8 (ISBN).
Abstract | Links | BibTeX | Tags: 3D application, 3D modeling, 3D scenes, Accessibility, BLV, DHH, Discrete event simulation, Generative AI, Generative tools, Interactive computer graphics, One dimensional, Runtimes, Three dimensional computer graphics, Video-games, Virtual 3d scene, virtual 3D scenes, Virtual environments, Virtual Reality
@inproceedings{cao_scenegena11y_2025,
title = {SceneGenA11y: How can Runtime Generative tools improve the Accessibility of a Virtual 3D Scene?},
author = {X. Cao and K. P. Ju and C. Li and D. Jain},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005772656&doi=10.1145%2f3706599.3720265&partnerID=40&md5=9b0bf29c3e89b70efa2d6a3e740829fb},
doi = {10.1145/3706599.3720265},
isbn = {979-840071395-8 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Conf Hum Fact Comput Syst Proc},
publisher = {Association for Computing Machinery},
abstract = {With the popularity of virtual 3D applications, from video games to educational content and virtual reality scenarios, the accessibility of 3D scene information is vital to ensure inclusive and equitable experiences for all. Previous work include information substitutions like audio description and captions, as well as personalized modifications, but they could only provide predefined accommodations. In this work, we propose SceneGenA11y, a system that responds to the user’s natural language prompts to improve accessibility of a 3D virtual scene in runtime. The system primes LLM agents with accessibility-related knowledge, allowing users to explore the scene and perform verifiable modifications to improve accessibility. We conducted a preliminary evaluation of our system with three blind and low-vision people and three deaf and hard-of-hearing people. The results show that our system is intuitive to use and can successfully improve accessibility. We discussed usage patterns of the system, potential improvements, and integration into apps. We ended with highlighting plans for future work. © 2025 Copyright held by the owner/author(s).},
keywords = {3D application, 3D modeling, 3D scenes, Accessibility, BLV, DHH, Discrete event simulation, Generative AI, Generative tools, Interactive computer graphics, One dimensional, Runtimes, Three dimensional computer graphics, Video-games, Virtual 3d scene, virtual 3D scenes, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
With the popularity of virtual 3D applications, from video games to educational content and virtual reality scenarios, the accessibility of 3D scene information is vital to ensure inclusive and equitable experiences for all. Previous work include information substitutions like audio description and captions, as well as personalized modifications, but they could only provide predefined accommodations. In this work, we propose SceneGenA11y, a system that responds to the user’s natural language prompts to improve accessibility of a 3D virtual scene in runtime. The system primes LLM agents with accessibility-related knowledge, allowing users to explore the scene and perform verifiable modifications to improve accessibility. We conducted a preliminary evaluation of our system with three blind and low-vision people and three deaf and hard-of-hearing people. The results show that our system is intuitive to use and can successfully improve accessibility. We discussed usage patterns of the system, potential improvements, and integration into apps. We ended with highlighting plans for future work. © 2025 Copyright held by the owner/author(s).
Coronado, A.; Carvalho, S. T.; Berretta, L.
See Through My Eyes: Using Multimodal Large Language Model for Describing Rendered Environments to Blind People Proceedings Article
In: IMX - Proc. ACM Int. Conf. Interact. Media Experiences, pp. 451–457, Association for Computing Machinery, Inc, 2025, ISBN: 979-840071391-0 (ISBN).
Abstract | Links | BibTeX | Tags: Accessibility, Behavioral Research, Blind, Blind people, Helmet mounted displays, Human engineering, Human rehabilitation equipment, Interactive computer graphics, Interactive computer systems, Language Model, LLM, Multi-modal, Rendered environment, rendered environments, Spatial cognition, Virtual Reality, Vision aids, Visual impairment, Visual languages, Visually impaired people
@inproceedings{coronado_see_2025,
title = {See Through My Eyes: Using Multimodal Large Language Model for Describing Rendered Environments to Blind People},
author = {A. Coronado and S. T. Carvalho and L. Berretta},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105007991842&doi=10.1145%2f3706370.3731641&partnerID=40&md5=2f7cb1535d39d5e59b1f43f773de3272},
doi = {10.1145/3706370.3731641},
isbn = {979-840071391-0 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {IMX - Proc. ACM Int. Conf. Interact. Media Experiences},
pages = {451–457},
publisher = {Association for Computing Machinery, Inc},
abstract = {Extended Reality (XR) is quickly expanding "as the next major technology wave in personal computing". Nevertheless, this expansion and adoption could also exclude certain disabled users, particularly people with visual impairment (VIP). According to the World Health Organization (WHO) in their 2019 publication, there were at least 2.2 billion people with visual impairment, a number that is also estimated to have increased in recent years. Therefore, it is important to include disabled users, especially visually impaired people, in the design of Head-Mounted Displays and Extended Reality environments. Indeed, this objective can be pursued by incorporating Multimodal Large Language Model (MLLM) technology, which can assist visually impaired people. As a case study, this study employs different prompts that result in environment descriptions from an MLLM integrated into a virtual reality (VR) escape room. Therefore, six potential prompts were engineered to generate valuable outputs for visually impaired users inside a VR environment. These outputs were evaluated using the G-Eval, and VIEScore metrics. Even though, the results show that the prompt patterns provided a description that aligns with the user's point of view, it is highly recommended to evaluate these outputs through "expected outputs"from Orientation and Mobility Specialists, and Sighted Guides. Furthermore, the subsequent step in the process is to evaluate these outputs by visually impaired people themselves to identify the most effective prompt pattern. © 2025 Copyright held by the owner/author(s).},
keywords = {Accessibility, Behavioral Research, Blind, Blind people, Helmet mounted displays, Human engineering, Human rehabilitation equipment, Interactive computer graphics, Interactive computer systems, Language Model, LLM, Multi-modal, Rendered environment, rendered environments, Spatial cognition, Virtual Reality, Vision aids, Visual impairment, Visual languages, Visually impaired people},
pubstate = {published},
tppubtype = {inproceedings}
}
Extended Reality (XR) is quickly expanding "as the next major technology wave in personal computing". Nevertheless, this expansion and adoption could also exclude certain disabled users, particularly people with visual impairment (VIP). According to the World Health Organization (WHO) in their 2019 publication, there were at least 2.2 billion people with visual impairment, a number that is also estimated to have increased in recent years. Therefore, it is important to include disabled users, especially visually impaired people, in the design of Head-Mounted Displays and Extended Reality environments. Indeed, this objective can be pursued by incorporating Multimodal Large Language Model (MLLM) technology, which can assist visually impaired people. As a case study, this study employs different prompts that result in environment descriptions from an MLLM integrated into a virtual reality (VR) escape room. Therefore, six potential prompts were engineered to generate valuable outputs for visually impaired users inside a VR environment. These outputs were evaluated using the G-Eval, and VIEScore metrics. Even though, the results show that the prompt patterns provided a description that aligns with the user's point of view, it is highly recommended to evaluate these outputs through "expected outputs"from Orientation and Mobility Specialists, and Sighted Guides. Furthermore, the subsequent step in the process is to evaluate these outputs by visually impaired people themselves to identify the most effective prompt pattern. © 2025 Copyright held by the owner/author(s).
2024
de Oliveira, E. A. Masasi; Silva, D. F. C.; Filho, A. R. G.
Improving VR Accessibility Through Automatic 360 Scene Description Using Multimodal Large Language Models Proceedings Article
In: ACM Int. Conf. Proc. Ser., pp. 289–293, Association for Computing Machinery, 2024, ISBN: 979-840070979-1 (ISBN).
Abstract | Links | BibTeX | Tags: 3D Scene, 3D scenes, Accessibility, Computer simulation languages, Descriptive information, Digital elevation model, Immersive, Language Model, Multi-modal, Multimodal large language model, Multimodal Large Language Models (MLLMs), Scene description, Virtual environments, Virtual Reality, Virtual Reality (VR), Virtual reality technology
@inproceedings{masasi_de_oliveira_improving_2024,
title = {Improving VR Accessibility Through Automatic 360 Scene Description Using Multimodal Large Language Models},
author = {E. A. Masasi de Oliveira and D. F. C. Silva and A. R. G. Filho},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85206580797&doi=10.1145%2f3691573.3691619&partnerID=40&md5=6e80800fce0e6b56679fbcbe982bcfa7},
doi = {10.1145/3691573.3691619},
isbn = {979-840070979-1 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {ACM Int. Conf. Proc. Ser.},
pages = {289–293},
publisher = {Association for Computing Machinery},
abstract = {Advancements in Virtual Reality (VR) technology hold immense promise for enriching immersive experiences. Despite the advancements in VR technology, there remains a significant gap in addressing accessibility concerns, particularly in automatically providing descriptive information for VR scenes. This paper combines the potential of leveraging Multimodal Large Language Models (MLLMs) to automatically generate text descriptions for 360 VR scenes according to Speech-to-Text (STT) prompts. As a case study, we conduct experiments on educational settings in VR museums, improving dynamic experiences across various contexts. Despite minor challenges in adapting MLLMs to VR Scenes, the experiments demonstrate that they can generate descriptions with high quality. Our findings provide insights for enhancing VR experiences and ensuring accessibility to individuals with disabilities or diverse needs. © 2024 Copyright held by the owner/author(s).},
keywords = {3D Scene, 3D scenes, Accessibility, Computer simulation languages, Descriptive information, Digital elevation model, Immersive, Language Model, Multi-modal, Multimodal large language model, Multimodal Large Language Models (MLLMs), Scene description, Virtual environments, Virtual Reality, Virtual Reality (VR), Virtual reality technology},
pubstate = {published},
tppubtype = {inproceedings}
}
Advancements in Virtual Reality (VR) technology hold immense promise for enriching immersive experiences. Despite the advancements in VR technology, there remains a significant gap in addressing accessibility concerns, particularly in automatically providing descriptive information for VR scenes. This paper combines the potential of leveraging Multimodal Large Language Models (MLLMs) to automatically generate text descriptions for 360 VR scenes according to Speech-to-Text (STT) prompts. As a case study, we conduct experiments on educational settings in VR museums, improving dynamic experiences across various contexts. Despite minor challenges in adapting MLLMs to VR Scenes, the experiments demonstrate that they can generate descriptions with high quality. Our findings provide insights for enhancing VR experiences and ensuring accessibility to individuals with disabilities or diverse needs. © 2024 Copyright held by the owner/author(s).
2019
Scianna, Andrea; Filippo, Gabriella Di
RAPID PROTOTYPING for the EXTENSION of the ACCESSIBILITY to CULTURAL HERITAGE for BLIND PEOPLE Proceedings Article
In: D., Remondino F. Toschi I. Rodriguez-Gonzalvez P. Stathopoulou E. Gonzalez-Aguilera (Ed.): International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives, pp. 1077–1082, International Society for Photogrammetry and Remote Sensing, 2019.
Abstract | Links | BibTeX | Tags: 3D Printing, Accessibility, Cultural heritage
@inproceedings{sciannaRAPIDPROTOTYPINGEXTENSION2019a,
title = {RAPID PROTOTYPING for the EXTENSION of the ACCESSIBILITY to CULTURAL HERITAGE for BLIND PEOPLE},
author = { Andrea Scianna and Gabriella Di Filippo},
editor = { Remondino F. Toschi I. Rodriguez-Gonzalvez P. Stathopoulou E. {Gonzalez-Aguilera D.}},
doi = {10.5194/isprs-archives-XLII-2-W15-1077-2019},
year = {2019},
date = {2019-01-01},
booktitle = {International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives},
volume = {42},
pages = {1077--1082},
publisher = {International Society for Photogrammetry and Remote Sensing},
abstract = {The extension of accessibility of CH to disabled people is a current topic of great importance. Today, there exist a lot of ongoing projects aimed at the extension of accessibility using the most recent and low cost technologies to improve accessibility to CH. One of the technologies, today available that can help to improve accessibility to CH, is certainly rapid prototyping. The main goal of the research here illustrated is the application of rapid prototyping for the extension of accessibility of Cultural Heritage (CH) to blind and visually impaired people. This document reports an experiment that compared the effectiveness of different 3D models for the introduction of blind and visually impaired people to the tactile use of models of monuments. In addition, this study focuses on defining standards and guidelines regarding features of 3D models, e.g. print resolution, in order to obtain greater legibility of models at different scales by the blind or visually impaired people. This paper shows a part of a wider Interreg EU research project, named I-ACCESS, aimed to study accessibility problems and solutions for CH. The experiments conducted with the sample of disabled people, allowed to obtain results on the most suitable printing parameters to be used. textcopyright 2019 International Society for Photogrammetry and Remote Sensing. All rights reserved.},
keywords = {3D Printing, Accessibility, Cultural heritage},
pubstate = {published},
tppubtype = {inproceedings}
}
The extension of accessibility of CH to disabled people is a current topic of great importance. Today, there exist a lot of ongoing projects aimed at the extension of accessibility using the most recent and low cost technologies to improve accessibility to CH. One of the technologies, today available that can help to improve accessibility to CH, is certainly rapid prototyping. The main goal of the research here illustrated is the application of rapid prototyping for the extension of accessibility of Cultural Heritage (CH) to blind and visually impaired people. This document reports an experiment that compared the effectiveness of different 3D models for the introduction of blind and visually impaired people to the tactile use of models of monuments. In addition, this study focuses on defining standards and guidelines regarding features of 3D models, e.g. print resolution, in order to obtain greater legibility of models at different scales by the blind or visually impaired people. This paper shows a part of a wider Interreg EU research project, named I-ACCESS, aimed to study accessibility problems and solutions for CH. The experiments conducted with the sample of disabled people, allowed to obtain results on the most suitable printing parameters to be used. textcopyright 2019 International Society for Photogrammetry and Remote Sensing. All rights reserved.
Scianna, Andrea; Filippo, Gabriella Di
RAPID PROTOTYPING for the EXTENSION of the ACCESSIBILITY to CULTURAL HERITAGE for BLIND PEOPLE Proceedings Article
In: D., Toschi I. Remondino F. Gonzalez-Aguilera (Ed.): International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives, pp. 1077–1082, International Society for Photogrammetry and Remote Sensing, 2019, (Issue: 2/W15).
Abstract | Links | BibTeX | Tags: 3-D printing, 3D printers, 3D Printing, Accessibility, Accessibility problems, Blind and visually impaired, Cultural heritage, Cultural heritages, Low cost technology, Rapid prototyping, Standards and guidelines, Visually impaired people
@inproceedings{scianna_rapid_2019,
title = {RAPID PROTOTYPING for the EXTENSION of the ACCESSIBILITY to CULTURAL HERITAGE for BLIND PEOPLE},
author = {Andrea Scianna and Gabriella Di Filippo},
editor = {Toschi I. Remondino F. Gonzalez-Aguilera D.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85072212590&doi=10.5194%2fisprs-archives-XLII-2-W15-1077-2019&partnerID=40&md5=b95131689b6078a224eea8a8c1cc1730},
doi = {10.5194/isprs-archives-XLII-2-W15-1077-2019},
year = {2019},
date = {2019-01-01},
booktitle = {International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives},
volume = {42},
pages = {1077–1082},
publisher = {International Society for Photogrammetry and Remote Sensing},
abstract = {The extension of accessibility of CH to disabled people is a current topic of great importance. Today, there exist a lot of ongoing projects aimed at the extension of accessibility using the most recent and low cost technologies to improve accessibility to CH. One of the technologies, today available that can help to improve accessibility to CH, is certainly rapid prototyping. The main goal of the research here illustrated is the application of rapid prototyping for the extension of accessibility of Cultural Heritage (CH) to blind and visually impaired people. This document reports an experiment that compared the effectiveness of different 3D models for the introduction of blind and visually impaired people to the tactile use of models of monuments. In addition, this study focuses on defining standards and guidelines regarding features of 3D models, e.g. print resolution, in order to obtain greater legibility of models at different scales by the blind or visually impaired people. This paper shows a part of a wider Interreg EU research project, named I-ACCESS, aimed to study accessibility problems and solutions for CH. The experiments conducted with the sample of disabled people, allowed to obtain results on the most suitable printing parameters to be used. © 2019 International Society for Photogrammetry and Remote Sensing. All rights reserved.},
note = {Issue: 2/W15},
keywords = {3-D printing, 3D printers, 3D Printing, Accessibility, Accessibility problems, Blind and visually impaired, Cultural heritage, Cultural heritages, Low cost technology, Rapid prototyping, Standards and guidelines, Visually impaired people},
pubstate = {published},
tppubtype = {inproceedings}
}
The extension of accessibility of CH to disabled people is a current topic of great importance. Today, there exist a lot of ongoing projects aimed at the extension of accessibility using the most recent and low cost technologies to improve accessibility to CH. One of the technologies, today available that can help to improve accessibility to CH, is certainly rapid prototyping. The main goal of the research here illustrated is the application of rapid prototyping for the extension of accessibility of Cultural Heritage (CH) to blind and visually impaired people. This document reports an experiment that compared the effectiveness of different 3D models for the introduction of blind and visually impaired people to the tactile use of models of monuments. In addition, this study focuses on defining standards and guidelines regarding features of 3D models, e.g. print resolution, in order to obtain greater legibility of models at different scales by the blind or visually impaired people. This paper shows a part of a wider Interreg EU research project, named I-ACCESS, aimed to study accessibility problems and solutions for CH. The experiments conducted with the sample of disabled people, allowed to obtain results on the most suitable printing parameters to be used. © 2019 International Society for Photogrammetry and Remote Sensing. All rights reserved.