AHCI RESEARCH GROUP
Publications
Papers published in international journals,
proceedings of conferences, workshops and books.
OUR RESEARCH
Scientific Publications
How to
Here you can find the complete list of our publications.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
2024
Shabanijou, M.; Sharma, V.; Ray, S.; Lu, R.; Xiong, P.
Large Language Model Empowered Spatio-Visual Queries for Extended Reality Environments Proceedings Article
In: W., Ding; C.-T., Lu; F., Wang; L., Di; K., Wu; J., Huan; R., Nambiar; J., Li; F., Ilievski; R., Baeza-Yates; X., Hu (Ed.): Proc. - IEEE Int. Conf. Big Data, BigData, pp. 5843–5846, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-835036248-0 (ISBN).
Abstract | Links | BibTeX | Tags: 3D modeling, Digital elevation model, Emerging applications, Immersive environment, Language Model, Metaverses, Modeling languages, Natural language interfaces, Query languages, spatial data, Spatial queries, Structured Query Language, Technological advances, Users perspective, Virtual environments, Visual languages, Visual query
@inproceedings{shabanijou_large_2024,
title = {Large Language Model Empowered Spatio-Visual Queries for Extended Reality Environments},
author = {M. Shabanijou and V. Sharma and S. Ray and R. Lu and P. Xiong},
editor = {Ding W. and Lu C.-T. and Wang F. and Di L. and Wu K. and Huan J. and Nambiar R. and Li J. and Ilievski F. and Baeza-Yates R. and Hu X.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85218011140&doi=10.1109%2fBigData62323.2024.10825084&partnerID=40&md5=fdd78814b8e19830d1b8ecd4b33b0102},
doi = {10.1109/BigData62323.2024.10825084},
isbn = {979-835036248-0 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - IEEE Int. Conf. Big Data, BigData},
pages = {5843–5846},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {With the technological advances in creation and capture of 3D spatial data, new emerging applications are being developed. Digital Twins, metaverse and extended reality (XR) based immersive environments can be enriched by leveraging geocoded 3D spatial data. Unlike 2D spatial queries, queries involving 3D immersive environments need to take the query user's viewpoint into account. Spatio-visual queries return objects that are visible from the user's perspective.In this paper, we propose enhancing 3D spatio-visual queries with large language models (LLM). These kinds of queries allow a user to interact with the visible objects using a natural language interface. We have implemented a proof-of-concept prototype and conducted preliminary evaluation. Our results demonstrate the potential of truly interactive immersive environments. © 2024 IEEE.},
keywords = {3D modeling, Digital elevation model, Emerging applications, Immersive environment, Language Model, Metaverses, Modeling languages, Natural language interfaces, Query languages, spatial data, Spatial queries, Structured Query Language, Technological advances, Users perspective, Virtual environments, Visual languages, Visual query},
pubstate = {published},
tppubtype = {inproceedings}
}
With the technological advances in creation and capture of 3D spatial data, new emerging applications are being developed. Digital Twins, metaverse and extended reality (XR) based immersive environments can be enriched by leveraging geocoded 3D spatial data. Unlike 2D spatial queries, queries involving 3D immersive environments need to take the query user's viewpoint into account. Spatio-visual queries return objects that are visible from the user's perspective.In this paper, we propose enhancing 3D spatio-visual queries with large language models (LLM). These kinds of queries allow a user to interact with the visible objects using a natural language interface. We have implemented a proof-of-concept prototype and conducted preliminary evaluation. Our results demonstrate the potential of truly interactive immersive environments. © 2024 IEEE.
Diaz, T. G.; Lee, X. Y.; Zhuge, H.; Vidyaratne, L.; Sin, G.; Watanabe, T.; Farahat, A.; Gupta, C.
AI+AR based Framework for Guided Visual Equipment Diagnosis Proceedings Article
In: C.S., Kulkarni; M.E., Orchard (Ed.): Proc. Annu. Conf. Progn. Health Manag. Soc., PHM, Prognostics and Health Management Society, 2024, ISBN: 23250178 (ISSN); 978-193626305-9 (ISBN).
Abstract | Links | BibTeX | Tags: Augmented Reality, Automated solutions, Customer loyalty, Customer satisfaction, Customers' satisfaction, Diagnosis, Equipment diagnosis, Failure Diagnosis, Failure repairs, High quality, Knowledge graphs, Language Model, Quality of Service, Query languages, Sales, Support services
@inproceedings{diaz_aiar_2024,
title = {AI+AR based Framework for Guided Visual Equipment Diagnosis},
author = {T. G. Diaz and X. Y. Lee and H. Zhuge and L. Vidyaratne and G. Sin and T. Watanabe and A. Farahat and C. Gupta},
editor = {Kulkarni C.S. and Orchard M.E.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85210227167&doi=10.36001%2fphmconf.2024.v16i1.3909&partnerID=40&md5=897ac8045a48e2e80aa7522870c2004f},
doi = {10.36001/phmconf.2024.v16i1.3909},
isbn = {23250178 (ISSN); 978-193626305-9 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. Annu. Conf. Progn. Health Manag. Soc., PHM},
volume = {16},
publisher = {Prognostics and Health Management Society},
abstract = {Automated solutions for effective support services, such as failure diagnosis and repair, are crucial to keep customer satisfaction and loyalty. However, providing consistent, high quality, and timely support is a difficult task. In practice, customer support usually requires technicians to perform onsite diagnosis, but service quality is often adversely affected by limited expert technicians, high turnover, and minimal automated tools. To address these challenges, we present a novel solution framework for aiding technicians in performing visual equipment diagnosis. We envision a workflow where the technician reports a failure and prompts the system to automatically generate a diagnostic plan that includes parts, areas of interest, and necessary tasks. The plan is used to guide the technician with augmented reality (AR), while a perception module analyzes and tracks the technician’s actions to recommend next steps. Our framework consists of three components: planning, tracking, and guiding. The planning component automates the creation of a diagnostic plan by querying a knowledge graph (KG). We propose to leverage Large Language Models (LLMs) for the construction of the KG to accelerate the extraction process of parts, tasks, and relations from manuals. The tracking component enhances 3D detections by using perception sensors with a 2D nested object detection model. Finally, the guiding component reduces process complexity for technicians by combining 2D models and AR interactions. To validate the framework, we performed multiple studies to:1) determine an effective prompt method for the LLM to construct the KG; 2) demonstrate benefits of our 2D nested object model combined with AR model. © 2024 Prognostics and Health Management Society. All rights reserved.},
keywords = {Augmented Reality, Automated solutions, Customer loyalty, Customer satisfaction, Customers' satisfaction, Diagnosis, Equipment diagnosis, Failure Diagnosis, Failure repairs, High quality, Knowledge graphs, Language Model, Quality of Service, Query languages, Sales, Support services},
pubstate = {published},
tppubtype = {inproceedings}
}
Automated solutions for effective support services, such as failure diagnosis and repair, are crucial to keep customer satisfaction and loyalty. However, providing consistent, high quality, and timely support is a difficult task. In practice, customer support usually requires technicians to perform onsite diagnosis, but service quality is often adversely affected by limited expert technicians, high turnover, and minimal automated tools. To address these challenges, we present a novel solution framework for aiding technicians in performing visual equipment diagnosis. We envision a workflow where the technician reports a failure and prompts the system to automatically generate a diagnostic plan that includes parts, areas of interest, and necessary tasks. The plan is used to guide the technician with augmented reality (AR), while a perception module analyzes and tracks the technician’s actions to recommend next steps. Our framework consists of three components: planning, tracking, and guiding. The planning component automates the creation of a diagnostic plan by querying a knowledge graph (KG). We propose to leverage Large Language Models (LLMs) for the construction of the KG to accelerate the extraction process of parts, tasks, and relations from manuals. The tracking component enhances 3D detections by using perception sensors with a 2D nested object detection model. Finally, the guiding component reduces process complexity for technicians by combining 2D models and AR interactions. To validate the framework, we performed multiple studies to:1) determine an effective prompt method for the LLM to construct the KG; 2) demonstrate benefits of our 2D nested object model combined with AR model. © 2024 Prognostics and Health Management Society. All rights reserved.
2010
Scianna, Andrea; Ammoscato, Alessio
3D gis data model using open source software Proceedings Article
In: A, Peled (Ed.): International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives, pp. 120–125, International Society for Photogrammetry and Remote Sensing, 2010.
Abstract | Links | BibTeX | Tags: 3-dimensional modeling, 3D Modelling, Blending, Computer software, Data visualization, Database systems, Environmental database, Environmental Technology, Free and open source softwares, Geographic information systems, Geographical Information Systems, High level languages, HTTP, Internet, Internet browsers, Internet protocols, Interoperability, Maintenance, Mapping, Maps, Open source software, Open systems, Query languages, Research management, Social networking (online), Software engineering, Spatial, Technology, Three dimensional computer graphics, Three-dimensional data, Topological information, Topology, World Wide Web
@inproceedings{scianna_3d_2010,
title = {3D gis data model using open source software},
author = {Andrea Scianna and Alessio Ammoscato},
editor = {Peled A},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-84880227655&partnerID=40&md5=502aa042af1693c18f34b5d74c4dd2bd},
year = {2010},
date = {2010-01-01},
booktitle = {International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives},
volume = {38},
pages = {120–125},
publisher = {International Society for Photogrammetry and Remote Sensing},
abstract = {Today many kinds of applications requires data containing actual three-dimensional data; fields like urban and town planning and pollution studies need 3D data, both for visualization purpose, as well as carry out many spatial analysis. This research-Management and use of distributed 3D data by open source Web-GIS software-is part of the Italian "PRIN 2007"∗ research project, aimed to build urban and suburban 3D models, and to interact with them using open source software only. Particularly free and open source software, used for the experimentation here shown, are Blender and PostGIS; the first one has been used to build and structure three-dimensional data, the second one for data allocation. These software interact using scripts, written in Python language. Buildings have been modeled upon the GIANT3D model (Geographical Interoperable Advanced Numerical Topological 3-Dimensional Model) developed in the research "PRIN 2004", regarding "Evolved structure of numerical cartography for Gis and Web-GIS". Python scripts, activated by Blender, allow to allocate data into a spatial database implemented through PostgreSQL and PostGis, that could be a remote database somewhere on the net; all geometrical and topological information, implemented in the 3D model, are so transferred in PostGIS. These information can be retrieved by Blender using other Python scripts, so Blender fully interacts with 3D data allocated in PostGIS. These data can be also accessed by many other clients, both directly using a database client, as using other protocols (like HTTP on the internet). Next step is to build an open source viewer, or a plugin for internet browsers, that allows client to visualize, explore and inquiry 3D model, retrieving data from database.},
keywords = {3-dimensional modeling, 3D Modelling, Blending, Computer software, Data visualization, Database systems, Environmental database, Environmental Technology, Free and open source softwares, Geographic information systems, Geographical Information Systems, High level languages, HTTP, Internet, Internet browsers, Internet protocols, Interoperability, Maintenance, Mapping, Maps, Open source software, Open systems, Query languages, Research management, Social networking (online), Software engineering, Spatial, Technology, Three dimensional computer graphics, Three-dimensional data, Topological information, Topology, World Wide Web},
pubstate = {published},
tppubtype = {inproceedings}
}
Today many kinds of applications requires data containing actual three-dimensional data; fields like urban and town planning and pollution studies need 3D data, both for visualization purpose, as well as carry out many spatial analysis. This research-Management and use of distributed 3D data by open source Web-GIS software-is part of the Italian "PRIN 2007"∗ research project, aimed to build urban and suburban 3D models, and to interact with them using open source software only. Particularly free and open source software, used for the experimentation here shown, are Blender and PostGIS; the first one has been used to build and structure three-dimensional data, the second one for data allocation. These software interact using scripts, written in Python language. Buildings have been modeled upon the GIANT3D model (Geographical Interoperable Advanced Numerical Topological 3-Dimensional Model) developed in the research "PRIN 2004", regarding "Evolved structure of numerical cartography for Gis and Web-GIS". Python scripts, activated by Blender, allow to allocate data into a spatial database implemented through PostgreSQL and PostGis, that could be a remote database somewhere on the net; all geometrical and topological information, implemented in the 3D model, are so transferred in PostGIS. These information can be retrieved by Blender using other Python scripts, so Blender fully interacts with 3D data allocated in PostGIS. These data can be also accessed by many other clients, both directly using a database client, as using other protocols (like HTTP on the internet). Next step is to build an open source viewer, or a plugin for internet browsers, that allows client to visualize, explore and inquiry 3D model, retrieving data from database.