AHCI RESEARCH GROUP
Publications
Papers published in international journals,
proceedings of conferences, workshops and books.
OUR RESEARCH
Scientific Publications
How to
Here you can find the complete list of our publications.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
2024
Lakhnati, Y.; Pascher, M.; Gerken, J.
Exploring a GPT-based large language model for variable autonomy in a VR-based human-robot teaming simulation Journal Article
In: Frontiers in Robotics and AI, vol. 11, 2024, ISSN: 22969144 (ISSN).
Abstract | Links | BibTeX | Tags: Assistive Robots, evaluation, GPT, Large language model, shared control, variable autonomy, Virtual Reality
@article{lakhnati_exploring_2024,
title = {Exploring a GPT-based large language model for variable autonomy in a VR-based human-robot teaming simulation},
author = {Y. Lakhnati and M. Pascher and J. Gerken},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85190520269&doi=10.3389%2ffrobt.2024.1347538&partnerID=40&md5=ba5dcbba299b475c3448d2ea6b493894},
doi = {10.3389/frobt.2024.1347538},
issn = {22969144 (ISSN)},
year = {2024},
date = {2024-01-01},
journal = {Frontiers in Robotics and AI},
volume = {11},
abstract = {In a rapidly evolving digital landscape autonomous tools and robots are becoming commonplace. Recognizing the significance of this development, this paper explores the integration of Large Language Models (LLMs) like Generative pre-trained transformer (GPT) into human-robot teaming environments to facilitate variable autonomy through the means of verbal human-robot communication. In this paper, we introduce a novel simulation framework for such a GPT-powered multi-robot testbed environment, based on a Unity Virtual Reality (VR) setting. This system allows users to interact with simulated robot agents through natural language, each powered by individual GPT cores. By means of OpenAI’s function calling, we bridge the gap between unstructured natural language input and structured robot actions. A user study with 12 participants explores the effectiveness of GPT-4 and, more importantly, user strategies when being given the opportunity to converse in natural language within a simulated multi-robot environment. Our findings suggest that users may have preconceived expectations on how to converse with robots and seldom try to explore the actual language and cognitive capabilities of their simulated robot collaborators. Still, those users who did explore were able to benefit from a much more natural flow of communication and human-like back-and-forth. We provide a set of lessons learned for future research and technical implementations of similar systems. Copyright © 2024 Lakhnati, Pascher and Gerken.},
keywords = {Assistive Robots, evaluation, GPT, Large language model, shared control, variable autonomy, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
In a rapidly evolving digital landscape autonomous tools and robots are becoming commonplace. Recognizing the significance of this development, this paper explores the integration of Large Language Models (LLMs) like Generative pre-trained transformer (GPT) into human-robot teaming environments to facilitate variable autonomy through the means of verbal human-robot communication. In this paper, we introduce a novel simulation framework for such a GPT-powered multi-robot testbed environment, based on a Unity Virtual Reality (VR) setting. This system allows users to interact with simulated robot agents through natural language, each powered by individual GPT cores. By means of OpenAI’s function calling, we bridge the gap between unstructured natural language input and structured robot actions. A user study with 12 participants explores the effectiveness of GPT-4 and, more importantly, user strategies when being given the opportunity to converse in natural language within a simulated multi-robot environment. Our findings suggest that users may have preconceived expectations on how to converse with robots and seldom try to explore the actual language and cognitive capabilities of their simulated robot collaborators. Still, those users who did explore were able to benefit from a much more natural flow of communication and human-like back-and-forth. We provide a set of lessons learned for future research and technical implementations of similar systems. Copyright © 2024 Lakhnati, Pascher and Gerken.
2022
Augello, Agnese; Infantino, Ignazio; Pilato, Giovanni; Vitale, Gianpaolo
Extending Affective Capabilities for Medical Assistive Robots Journal Article
In: Cognitive Systems Research, vol. 73, pp. 21–25, 2022, ISSN: 13890417.
Abstract | Links | BibTeX | Tags: Anthropomorphic Robots, Assistive Robots, Emotion Detection, Facial Expressions, Human computer interaction, Human Robot Interaction, Humanoid Robots, Natural Language Processing, Robotics, Wellbeing
@article{augelloExtendingAffectiveCapabilities2022,
title = {Extending Affective Capabilities for Medical Assistive Robots},
author = { Agnese Augello and Ignazio Infantino and Giovanni Pilato and Gianpaolo Vitale},
doi = {10.1016/j.cogsys.2021.12.004},
issn = {13890417},
year = {2022},
date = {2022-01-01},
journal = {Cognitive Systems Research},
volume = {73},
pages = {21--25},
abstract = {In this work, we discuss methodologies and implementation choices to enable a humanoid robot to estimate patients' mood and emotions during postoperative home rehabilitation. The approach is modular and it has been implemented into a SoftBank Pepper robotic architecture; however, the approach is general and it can be easily adapted to other robotic platforms. A sample of an interactive session for the detection of the patient's affective state is also reported. textcopyright 2022 Elsevier B.V.},
keywords = {Anthropomorphic Robots, Assistive Robots, Emotion Detection, Facial Expressions, Human computer interaction, Human Robot Interaction, Humanoid Robots, Natural Language Processing, Robotics, Wellbeing},
pubstate = {published},
tppubtype = {article}
}
In this work, we discuss methodologies and implementation choices to enable a humanoid robot to estimate patients' mood and emotions during postoperative home rehabilitation. The approach is modular and it has been implemented into a SoftBank Pepper robotic architecture; however, the approach is general and it can be easily adapted to other robotic platforms. A sample of an interactive session for the detection of the patient's affective state is also reported. textcopyright 2022 Elsevier B.V.
Augello, Agnese; Bella, Giulia Di; Infantino, Ignazio; Pilato, Giovanni; Vitale, Gianluigi
Multimodal Mood Recognition for Assistive Scenarios Proceedings Article
In: F.F., Samsonovich A. V. Ramos Corchado (Ed.): Procedia Computer Science, pp. 510–517, Elsevier B.V., 2022.
Abstract | Links | BibTeX | Tags: Assistive Robots, Emotion Analysis, Emotion Recognition, Mood
@inproceedings{augelloMultimodalMoodRecognition2022,
title = {Multimodal Mood Recognition for Assistive Scenarios},
author = { Agnese Augello and Giulia Di Bella and Ignazio Infantino and Giovanni Pilato and Gianluigi Vitale},
editor = { Samsonovich A.V. Ramos Corchado F.F.},
doi = {10.1016/j.procs.2022.11.098},
year = {2022},
date = {2022-01-01},
booktitle = {Procedia Computer Science},
volume = {213},
pages = {510--517},
publisher = {Elsevier B.V.},
abstract = {We illustrate a system performing multimodal human emotion detection from video input through the integration of audio emotional recognition, text emotional recognition, facial emotional recognition, and emotional recognition from a spectrogram. The outcomes of the four emotion recognition modalities are compared, and a final evaluation provides the most likely perceived emotion. The system has been designed to be easily implemented on cheap mini-computer based boards. It is conceived to be used as auxiliary tool in the field of telemedicine to remotely monitor the mood of patients and observe their healing process, which is closely related to their emotional condition. textcopyright 2022 The Author(s).},
keywords = {Assistive Robots, Emotion Analysis, Emotion Recognition, Mood},
pubstate = {published},
tppubtype = {inproceedings}
}
We illustrate a system performing multimodal human emotion detection from video input through the integration of audio emotional recognition, text emotional recognition, facial emotional recognition, and emotional recognition from a spectrogram. The outcomes of the four emotion recognition modalities are compared, and a final evaluation provides the most likely perceived emotion. The system has been designed to be easily implemented on cheap mini-computer based boards. It is conceived to be used as auxiliary tool in the field of telemedicine to remotely monitor the mood of patients and observe their healing process, which is closely related to their emotional condition. textcopyright 2022 The Author(s).
Augello, Agnese; Bella, Giulia Di; Infantino, Ignazio; Pilato, Giovanni; Vitale, Gianluigi
Multimodal Mood Recognition for Assistive Scenarios Proceedings Article
In: F.F., Samsonovich A. V. Ramos Corchado (Ed.): Procedia Computer Science, pp. 510–517, Elsevier B.V., 2022.
Abstract | Links | BibTeX | Tags: Assistive Robots, Emotion Analysis, Emotion Recognition, Mood
@inproceedings{augello_multimodal_2022,
title = {Multimodal Mood Recognition for Assistive Scenarios},
author = {Agnese Augello and Giulia Di Bella and Ignazio Infantino and Giovanni Pilato and Gianluigi Vitale},
editor = {Samsonovich A. V. Ramos Corchado F.F.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85146111070&doi=10.1016%2fj.procs.2022.11.098&partnerID=40&md5=5f02507ae4efe3f3fe476ce3f7dbc63d},
doi = {10.1016/j.procs.2022.11.098},
year = {2022},
date = {2022-01-01},
booktitle = {Procedia Computer Science},
volume = {213},
pages = {510–517},
publisher = {Elsevier B.V.},
abstract = {We illustrate a system performing multimodal human emotion detection from video input through the integration of audio emotional recognition, text emotional recognition, facial emotional recognition, and emotional recognition from a spectrogram. The outcomes of the four emotion recognition modalities are compared, and a final evaluation provides the most likely perceived emotion. The system has been designed to be easily implemented on cheap mini-computer based boards. It is conceived to be used as auxiliary tool in the field of telemedicine to remotely monitor the mood of patients and observe their healing process, which is closely related to their emotional condition. © 2022 The Author(s).},
keywords = {Assistive Robots, Emotion Analysis, Emotion Recognition, Mood},
pubstate = {published},
tppubtype = {inproceedings}
}
We illustrate a system performing multimodal human emotion detection from video input through the integration of audio emotional recognition, text emotional recognition, facial emotional recognition, and emotional recognition from a spectrogram. The outcomes of the four emotion recognition modalities are compared, and a final evaluation provides the most likely perceived emotion. The system has been designed to be easily implemented on cheap mini-computer based boards. It is conceived to be used as auxiliary tool in the field of telemedicine to remotely monitor the mood of patients and observe their healing process, which is closely related to their emotional condition. © 2022 The Author(s).
Augello, Agnese; Infantino, Ignazio; Pilato, Giovanni; Vitale, Gianpaolo
Extending affective capabilities for medical assistive robots Journal Article
In: Cognitive Systems Research, vol. 73, pp. 21–25, 2022, ISSN: 13890417.
Abstract | Links | BibTeX | Tags: Anthropomorphic Robots, Assistive Robots, Emotion Detection, Facial Expressions, Human computer interaction, Human Robot Interaction, Humanoid Robots, Natural Language Processing, Robotics, Wellbeing
@article{augello_extending_2022,
title = {Extending affective capabilities for medical assistive robots},
author = {Agnese Augello and Ignazio Infantino and Giovanni Pilato and Gianpaolo Vitale},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85123046436&doi=10.1016%2fj.cogsys.2021.12.004&partnerID=40&md5=6e76332f7f95333a9ae2e8f11c054622},
doi = {10.1016/j.cogsys.2021.12.004},
issn = {13890417},
year = {2022},
date = {2022-01-01},
journal = {Cognitive Systems Research},
volume = {73},
pages = {21–25},
abstract = {In this work, we discuss methodologies and implementation choices to enable a humanoid robot to estimate patients’ mood and emotions during postoperative home rehabilitation. The approach is modular and it has been implemented into a SoftBank Pepper robotic architecture; however, the approach is general and it can be easily adapted to other robotic platforms. A sample of an interactive session for the detection of the patient's affective state is also reported. © 2022 Elsevier B.V.},
keywords = {Anthropomorphic Robots, Assistive Robots, Emotion Detection, Facial Expressions, Human computer interaction, Human Robot Interaction, Humanoid Robots, Natural Language Processing, Robotics, Wellbeing},
pubstate = {published},
tppubtype = {article}
}
In this work, we discuss methodologies and implementation choices to enable a humanoid robot to estimate patients’ mood and emotions during postoperative home rehabilitation. The approach is modular and it has been implemented into a SoftBank Pepper robotic architecture; however, the approach is general and it can be easily adapted to other robotic platforms. A sample of an interactive session for the detection of the patient's affective state is also reported. © 2022 Elsevier B.V.
0000
Montalbano, Laura; Augello, Agnese; Pilato, Giovanni; Grutta, Stefania La
A Gamified Interaction with a Humanoid Robot to explain Therapeutic Procedures in Pediatric Asthma Miscellaneous
0000.
Abstract | BibTeX | Tags: Assistive Robots, Gamification, Social Robots
@misc{laura_montalbano_gamified_nodate,
title = {A Gamified Interaction with a Humanoid Robot to explain Therapeutic Procedures in Pediatric Asthma},
author = {Laura Montalbano and Agnese Augello and Giovanni Pilato and Stefania La Grutta},
publisher = {arXiv preprint arXiv:2306.04422},
abstract = {In chronic diseases, obtaining a correct diagnosis and providing the most appropriate treatments often is not enough to guarantee an improvement of the clinical condition of a patient. Poor adherence to medical prescriptions constitutes one of the main causes preventing achievement of therapeutic goals. This is generally true especially for certain diseases and specific target patients, such as children. An engaging and entertaining technology can be exploited in support of clinical practices to achieve better health outcomes. Our assumption is that a gamified session with a humanoid robot, compared to the usual methodologies for therapeutic education, can be more incisive in learning the correct inhalation procedure in children affected by asthma. In this perspective, we describe an interactive module implemented on the Pepper robotic platform and the setting of a study that was planned in 2020 to be held at the Pneumoallergology Pediatric clinic of CNR in Palermo. The study was canceled due to the COVID-19 pandemic. Our long-term goal is to assess, by means of a qualitative-quantitative survey plan, the impact of such an educational action, evaluating possible improvement in the adherence to the treatment.},
keywords = {Assistive Robots, Gamification, Social Robots},
pubstate = {published},
tppubtype = {misc}
}
In chronic diseases, obtaining a correct diagnosis and providing the most appropriate treatments often is not enough to guarantee an improvement of the clinical condition of a patient. Poor adherence to medical prescriptions constitutes one of the main causes preventing achievement of therapeutic goals. This is generally true especially for certain diseases and specific target patients, such as children. An engaging and entertaining technology can be exploited in support of clinical practices to achieve better health outcomes. Our assumption is that a gamified session with a humanoid robot, compared to the usual methodologies for therapeutic education, can be more incisive in learning the correct inhalation procedure in children affected by asthma. In this perspective, we describe an interactive module implemented on the Pepper robotic platform and the setting of a study that was planned in 2020 to be held at the Pneumoallergology Pediatric clinic of CNR in Palermo. The study was canceled due to the COVID-19 pandemic. Our long-term goal is to assess, by means of a qualitative-quantitative survey plan, the impact of such an educational action, evaluating possible improvement in the adherence to the treatment.
Licari, Amelia; Ferrante, Giuliana; Malizia, Velia; Augello, Agnese; Grutta, Stefania La
Medical Assistive Robots Book Section
In: Digital Respiratory Healthcare, pp. 16–26, 0000, ISBN: 978-1-84984-173-3.
Abstract | Links | BibTeX | Tags: Assistive Robots, Asthma, Gamification, Social Agents, Social Robots
@incollection{amelia_licari_medical_nodate,
title = {Medical Assistive Robots},
author = {Amelia Licari and Giuliana Ferrante and Velia Malizia and Agnese Augello and Stefania La Grutta},
url = {https://doi.org/10.1183/2312508X.10000523},
isbn = {978-1-84984-173-3},
booktitle = {Digital Respiratory Healthcare},
pages = {16–26},
series = {ERS Monograph)},
abstract = {Medical assistive robots (MARs) are innovative tools providing extensive support and assistance to users in different medical scenarios, enhancing patients’ health and care. Social MARs have been implemented in respiratory medicine to help manage chronic respiratory conditions, such as asthma, COPD and cystic fibrosis. To integrate MARs into routine clinical practice, more studies are needed to strengthen the evidence on the feasibility, acceptability and efficacy of MARs in chronic respiratory conditions in the long term.},
keywords = {Assistive Robots, Asthma, Gamification, Social Agents, Social Robots},
pubstate = {published},
tppubtype = {incollection}
}
Medical assistive robots (MARs) are innovative tools providing extensive support and assistance to users in different medical scenarios, enhancing patients’ health and care. Social MARs have been implemented in respiratory medicine to help manage chronic respiratory conditions, such as asthma, COPD and cystic fibrosis. To integrate MARs into routine clinical practice, more studies are needed to strengthen the evidence on the feasibility, acceptability and efficacy of MARs in chronic respiratory conditions in the long term.