AHCI RESEARCH GROUP
Publications
Papers published in international journals,
proceedings of conferences, workshops and books.
OUR RESEARCH
Scientific Publications
How to
Here you can find the complete list of our publications.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
2025
Zhang, G.; Wang, Y.; Luo, C.; Xu, S.; Ming, Y.; Peng, J.; Zhang, M.
Visual Harmony: LLM’s Power in Crafting Coherent Indoor Scenes from Images Proceedings Article
In: Z., Lin; H., Zha; M.-M., Cheng; R., He; C.-L., Liu; K., Ubul; W., Silamu; J., Zhou (Ed.): Lect. Notes Comput. Sci., pp. 3–17, Springer Science and Business Media Deutschland GmbH, 2025, ISBN: 03029743 (ISSN); 978-981978507-0 (ISBN).
Abstract | Links | BibTeX | Tags: Augmented Reality, Depth perception, Indoor scene generation, Input image, Language Model, Large language model, Metaverses, Point-clouds, Power, Scene completion, Scene Generation, Scene-graphs, Semantic Segmentation, Semantics, Virtual Reality, Visual languages
@inproceedings{zhang_visual_2025,
title = {Visual Harmony: LLM’s Power in Crafting Coherent Indoor Scenes from Images},
author = {G. Zhang and Y. Wang and C. Luo and S. Xu and Y. Ming and J. Peng and M. Zhang},
editor = {Lin Z. and Zha H. and Cheng M.-M. and He R. and Liu C.-L. and Ubul K. and Silamu W. and Zhou J.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85209374797&doi=10.1007%2f978-981-97-8508-7_1&partnerID=40&md5=5231ab0bce95fb3f09db80392acd58ff},
doi = {10.1007/978-981-97-8508-7_1},
isbn = {03029743 (ISSN); 978-981978507-0 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Lect. Notes Comput. Sci.},
volume = {15036 LNCS},
pages = {3–17},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {Indoor scene generation has recently attracted significant attention as it is crucial for metaverse, 3D animation, visual effects in movies, and virtual/augmented reality. Existing image-based indoor scene generation methods often produce scenes that are not realistic enough, with issues such as floating objects, incorrect object orientations, and incomplete scenes that only include the part of the scenes captured by the input image. To address these challenges, we propose Visual Harmony, a method that leverages the powerful spatial imagination capabilities of Large Language Model (LLM) to generate corresponding indoor scenes based on the input image. Specifically, we first extract information from the input image through depth estimation and panorama segmentation, reconstructing a semantic point cloud. Using this reconstructed semantic point cloud, we extract a scene graph that describes only the objects in the image. Then we leverage the strong spatial imagination capabilities of LLM to complete the scene graph, forming a representation of a complete room scene. Based on this fine scene graph, we can generate entire indoor scene that includes both the captured and not captured parts of the input image. Extensive experiments demonstrate that our method can generate realistic, plausible, and highly relevant complete indoor scenes related to the input image. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.},
keywords = {Augmented Reality, Depth perception, Indoor scene generation, Input image, Language Model, Large language model, Metaverses, Point-clouds, Power, Scene completion, Scene Generation, Scene-graphs, Semantic Segmentation, Semantics, Virtual Reality, Visual languages},
pubstate = {published},
tppubtype = {inproceedings}
}
Indoor scene generation has recently attracted significant attention as it is crucial for metaverse, 3D animation, visual effects in movies, and virtual/augmented reality. Existing image-based indoor scene generation methods often produce scenes that are not realistic enough, with issues such as floating objects, incorrect object orientations, and incomplete scenes that only include the part of the scenes captured by the input image. To address these challenges, we propose Visual Harmony, a method that leverages the powerful spatial imagination capabilities of Large Language Model (LLM) to generate corresponding indoor scenes based on the input image. Specifically, we first extract information from the input image through depth estimation and panorama segmentation, reconstructing a semantic point cloud. Using this reconstructed semantic point cloud, we extract a scene graph that describes only the objects in the image. Then we leverage the strong spatial imagination capabilities of LLM to complete the scene graph, forming a representation of a complete room scene. Based on this fine scene graph, we can generate entire indoor scene that includes both the captured and not captured parts of the input image. Extensive experiments demonstrate that our method can generate realistic, plausible, and highly relevant complete indoor scenes related to the input image. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.
2024
Liew, Z. Q.; Xu, M.; Lim, W. Y. Bryan; Niyato, D.; Kim, D. I.
AI-Generated Bidding for Immersive AIGC Services in Mobile Edge-Empowered Metaverse Proceedings Article
In: Int. Conf. Inf. Networking, pp. 305–309, IEEE Computer Society, 2024, ISBN: 19767684 (ISSN); 979-835033094-6 (ISBN).
Abstract | Links | BibTeX | Tags: Artificial intelligence generated bid, Artificial intelligence generated content, Bidding mechanism, Bidding models, Budget constraint, Budget control, Budget-constraint bidding, Constrained optimization, Content services, Immersive, Learning systems, Metaverses, Mobile edge computing, Reinforcement Learning, Semantics, Virtual tour
@inproceedings{liew_ai-generated_2024,
title = {AI-Generated Bidding for Immersive AIGC Services in Mobile Edge-Empowered Metaverse},
author = {Z. Q. Liew and M. Xu and W. Y. Bryan Lim and D. Niyato and D. I. Kim},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85198324990&doi=10.1109%2fICOIN59985.2024.10572159&partnerID=40&md5=271f5c45e8e95f01b42acaee89599bd5},
doi = {10.1109/ICOIN59985.2024.10572159},
isbn = {19767684 (ISSN); 979-835033094-6 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Int. Conf. Inf. Networking},
pages = {305–309},
publisher = {IEEE Computer Society},
abstract = {Recent advancements in Artificial Intelligence Generated Content (AIGC) provide personalized and immersive content generation services for applications such as interactive advertisements, virtual tours, and metaverse. With the use of mobile edge computing (MEC), buyers can bid for the AIGC service to enhance their user experience in real-time. However, designing strategies to optimize the quality of the services won can be challenging for budget-constrained buyers. The performance of classical bidding mechanisms is limited by the fixed rules in the strategies. To this end, we propose AI-generated bidding (AIGB) to optimize the bidding strategies for AIGC. AIGB model uses reinforcement learning model to generate bids for the services by learning from the historical data and environment states such as remaining budget, budget consumption rate, and quality of the won services. To obtain quality AIGC service, we propose a semantic aware reward function for the AIGB model. The proposed model is tested with a real-world dataset and experiments show that our model outperforms the classical bidding mechanism in terms of the number of services won and the similarity score. © 2024 IEEE.},
keywords = {Artificial intelligence generated bid, Artificial intelligence generated content, Bidding mechanism, Bidding models, Budget constraint, Budget control, Budget-constraint bidding, Constrained optimization, Content services, Immersive, Learning systems, Metaverses, Mobile edge computing, Reinforcement Learning, Semantics, Virtual tour},
pubstate = {published},
tppubtype = {inproceedings}
}
Recent advancements in Artificial Intelligence Generated Content (AIGC) provide personalized and immersive content generation services for applications such as interactive advertisements, virtual tours, and metaverse. With the use of mobile edge computing (MEC), buyers can bid for the AIGC service to enhance their user experience in real-time. However, designing strategies to optimize the quality of the services won can be challenging for budget-constrained buyers. The performance of classical bidding mechanisms is limited by the fixed rules in the strategies. To this end, we propose AI-generated bidding (AIGB) to optimize the bidding strategies for AIGC. AIGB model uses reinforcement learning model to generate bids for the services by learning from the historical data and environment states such as remaining budget, budget consumption rate, and quality of the won services. To obtain quality AIGC service, we propose a semantic aware reward function for the AIGB model. The proposed model is tested with a real-world dataset and experiments show that our model outperforms the classical bidding mechanism in terms of the number of services won and the similarity score. © 2024 IEEE.
Du, B.; Du, H.; Liu, H.; Niyato, D.; Xin, P.; Yu, J.; Qi, M.; Tang, Y.
YOLO-Based Semantic Communication with Generative AI-Aided Resource Allocation for Digital Twins Construction Journal Article
In: IEEE Internet of Things Journal, vol. 11, no. 5, pp. 7664–7678, 2024, ISSN: 23274662 (ISSN).
Abstract | Links | BibTeX | Tags: Cost reduction, Data transfer, Digital Twins, Edge detection, Image edge detection, Network layers, Object Detection, Object detectors, Objects detection, Physical world, Resource allocation, Resource management, Resources allocation, Semantic communication, Semantics, Semantics Information, Virtual Reality, Virtual worlds, Wireless communications
@article{du_yolo-based_2024,
title = {YOLO-Based Semantic Communication with Generative AI-Aided Resource Allocation for Digital Twins Construction},
author = {B. Du and H. Du and H. Liu and D. Niyato and P. Xin and J. Yu and M. Qi and Y. Tang},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85173060990&doi=10.1109%2fJIOT.2023.3317629&partnerID=40&md5=60507e2f6ce2b1c345248867a9c527a1},
doi = {10.1109/JIOT.2023.3317629},
issn = {23274662 (ISSN)},
year = {2024},
date = {2024-01-01},
journal = {IEEE Internet of Things Journal},
volume = {11},
number = {5},
pages = {7664–7678},
abstract = {Digital Twins play a crucial role in bridging the physical and virtual worlds. Given the dynamic and evolving characteristics of the physical world, a huge volume of data transmission and exchange is necessary to attain synchronized updates in the virtual world. In this article, we propose a semantic communication framework based on you only look once (YOLO) to construct a virtual apple orchard with the aim of mitigating the costs associated with data transmission. Specifically, we first employ the YOLOv7-X object detector to extract semantic information from captured images of edge devices, thereby reducing the volume of transmitted data and saving transmission costs. Afterwards, we quantify the importance of each semantic information by the confidence generated through the object detector. Based on this, we propose two resource allocation schemes, i.e., the confidence-based scheme and the acrlong AI-generated scheme, aimed at enhancing the transmission quality of important semantic information. The proposed diffusion model generates an optimal allocation scheme that outperforms both the average allocation scheme and the confidence-based allocation scheme. Moreover, to obtain semantic information more effectively, we enhance the detection capability of the YOLOv7-X object detector by introducing new efficient layer aggregation network-horNet (ELAN-H) and SimAM attention modules, while reducing the model parameters and computational complexity, making it easier to run on edge devices with limited performance. The numerical results indicate that our proposed semantic communication framework and resource allocation schemes significantly reduce transmission costs while enhancing the transmission quality of important information in communication services. © 2014 IEEE.},
keywords = {Cost reduction, Data transfer, Digital Twins, Edge detection, Image edge detection, Network layers, Object Detection, Object detectors, Objects detection, Physical world, Resource allocation, Resource management, Resources allocation, Semantic communication, Semantics, Semantics Information, Virtual Reality, Virtual worlds, Wireless communications},
pubstate = {published},
tppubtype = {article}
}
Digital Twins play a crucial role in bridging the physical and virtual worlds. Given the dynamic and evolving characteristics of the physical world, a huge volume of data transmission and exchange is necessary to attain synchronized updates in the virtual world. In this article, we propose a semantic communication framework based on you only look once (YOLO) to construct a virtual apple orchard with the aim of mitigating the costs associated with data transmission. Specifically, we first employ the YOLOv7-X object detector to extract semantic information from captured images of edge devices, thereby reducing the volume of transmitted data and saving transmission costs. Afterwards, we quantify the importance of each semantic information by the confidence generated through the object detector. Based on this, we propose two resource allocation schemes, i.e., the confidence-based scheme and the acrlong AI-generated scheme, aimed at enhancing the transmission quality of important semantic information. The proposed diffusion model generates an optimal allocation scheme that outperforms both the average allocation scheme and the confidence-based allocation scheme. Moreover, to obtain semantic information more effectively, we enhance the detection capability of the YOLOv7-X object detector by introducing new efficient layer aggregation network-horNet (ELAN-H) and SimAM attention modules, while reducing the model parameters and computational complexity, making it easier to run on edge devices with limited performance. The numerical results indicate that our proposed semantic communication framework and resource allocation schemes significantly reduce transmission costs while enhancing the transmission quality of important information in communication services. © 2014 IEEE.
Lin, Y.; Gao, Z.; Du, H.; Niyato, D.; Kang, J.; Xiong, Z.; Zheng, Z.
Blockchain-Based Efficient and Trustworthy AIGC Services in Metaverse Journal Article
In: IEEE Transactions on Services Computing, vol. 17, no. 5, pp. 2067–2079, 2024, ISSN: 19391374 (ISSN).
Abstract | Links | BibTeX | Tags: AI-generated content, Block-chain, Blockchain, Computational modelling, Content services, Data Mining, Digital contents, Information Management, Metaverse, Metaverses, Resource management, Semantic communication, Semantics, Virtual Reality
@article{lin_blockchain-based_2024,
title = {Blockchain-Based Efficient and Trustworthy AIGC Services in Metaverse},
author = {Y. Lin and Z. Gao and H. Du and D. Niyato and J. Kang and Z. Xiong and Z. Zheng},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85189177655&doi=10.1109%2fTSC.2024.3382958&partnerID=40&md5=5e3e80fbc88a49293b892acd762af3e9},
doi = {10.1109/TSC.2024.3382958},
issn = {19391374 (ISSN)},
year = {2024},
date = {2024-01-01},
journal = {IEEE Transactions on Services Computing},
volume = {17},
number = {5},
pages = {2067–2079},
abstract = {AI-Generated Content (AIGC) services are essential in developing the Metaverse, providing various digital content to build shared virtual environments. The services can also offer personalized content with user assistance, making the Metaverse more human-centric. However, user-assisted content creation requires significant communication resources to exchange data and construct trust among unknown Metaverse participants, which challenges the traditional centralized communication paradigm. To address the above challenge, we integrate blockchain with semantic communication to establish decentralized trust among participants, reducing communication overhead and improving trustworthiness for AIGC services in Metaverse. To solve the out-of-distribution issue in data provided by users, we utilize the invariant risk minimization method to extract invariant semantic information across multiple virtual environments. To guarantee trustworthiness of digital contents, we also design a smart contract-based verification mechanism to prevent random outcomes of AIGC services. We utilize semantic information and quality of digital contents provided by the above mechanisms as metrics to develop a Stackelberg game-based content caching mechanism, which can maximize the profits of Metaverse participants. Simulation results show that the proposed semantic extraction and caching mechanism can improve accuracy by almost 15% and utility by 30% compared to other mechanisms. © 2008-2012 IEEE.},
keywords = {AI-generated content, Block-chain, Blockchain, Computational modelling, Content services, Data Mining, Digital contents, Information Management, Metaverse, Metaverses, Resource management, Semantic communication, Semantics, Virtual Reality},
pubstate = {published},
tppubtype = {article}
}
AI-Generated Content (AIGC) services are essential in developing the Metaverse, providing various digital content to build shared virtual environments. The services can also offer personalized content with user assistance, making the Metaverse more human-centric. However, user-assisted content creation requires significant communication resources to exchange data and construct trust among unknown Metaverse participants, which challenges the traditional centralized communication paradigm. To address the above challenge, we integrate blockchain with semantic communication to establish decentralized trust among participants, reducing communication overhead and improving trustworthiness for AIGC services in Metaverse. To solve the out-of-distribution issue in data provided by users, we utilize the invariant risk minimization method to extract invariant semantic information across multiple virtual environments. To guarantee trustworthiness of digital contents, we also design a smart contract-based verification mechanism to prevent random outcomes of AIGC services. We utilize semantic information and quality of digital contents provided by the above mechanisms as metrics to develop a Stackelberg game-based content caching mechanism, which can maximize the profits of Metaverse participants. Simulation results show that the proposed semantic extraction and caching mechanism can improve accuracy by almost 15% and utility by 30% compared to other mechanisms. © 2008-2012 IEEE.
Kapadia, N.; Gokhale, S.; Nepomuceno, A.; Cheng, W.; Bothwell, S.; Mathews, M.; Shallat, J. S.; Schultz, C.; Gupta, A.
Evaluation of Large Language Model Generated Dialogues for an AI Based VR Nurse Training Simulator Proceedings Article
In: J.Y.C., Chen; G., Fragomeni (Ed.): Lect. Notes Comput. Sci., pp. 200–212, Springer Science and Business Media Deutschland GmbH, 2024, ISBN: 03029743 (ISSN); 978-303161040-0 (ISBN).
Abstract | Links | BibTeX | Tags: Bard, ChatGPT, ClaudeAI, Clinical research, Computational Linguistics, Dialogue Generation, Dialogue generations, Education computing, Extended reality, Health care education, Healthcare Education, Language Model, Language processing, Large language model, large language models, Natural Language Processing, Natural language processing systems, Natural languages, Nurse Training Simulation, Nursing, Patient avatar, Patient Avatars, Semantics, Students, Training simulation, Virtual Reality
@inproceedings{kapadia_evaluation_2024,
title = {Evaluation of Large Language Model Generated Dialogues for an AI Based VR Nurse Training Simulator},
author = {N. Kapadia and S. Gokhale and A. Nepomuceno and W. Cheng and S. Bothwell and M. Mathews and J. S. Shallat and C. Schultz and A. Gupta},
editor = {Chen J.Y.C. and Fragomeni G.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85196200653&doi=10.1007%2f978-3-031-61041-7_13&partnerID=40&md5=8890a8d0c289fdf6e7ab82e105249097},
doi = {10.1007/978-3-031-61041-7_13},
isbn = {03029743 (ISSN); 978-303161040-0 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Lect. Notes Comput. Sci.},
volume = {14706 LNCS},
pages = {200–212},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {This paper explores the efficacy of Large Language Models (LLMs) in generating dialogues for patient avatars in Virtual Reality (VR) nurse training simulators. With the integration of technology in healthcare education evolving rapidly, the potential of NLP to enhance nurse training through realistic patient interactions presents a significant opportunity. Our study introduces a novel LLM-based dialogue generation system, leveraging models such as ChatGPT, GoogleBard, and ClaudeAI. We detail the development of our script generation system, which was a collaborative endeavor involving nurses, technical artists, and developers. The system, tested on the Meta Quest 2 VR headset, integrates complex dialogues created through a synthesis of clinical expertise and advanced NLP, aimed at simulating real-world nursing scenarios. Through a comprehensive evaluation involving lexical and semantic similarity tests compared to clinical expert-generated scripts, we assess the potential of LLMs as suitable alternatives for script generation. The findings aim to contribute to the development of a more interactive and effective VR nurse training simulator, enhancing communication skills among nursing students for improved patient care outcomes. This research underscores the importance of advanced NLP applications in healthcare education, offering insights into the practicality and limitations of employing LLMs in clinical training environments. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.},
keywords = {Bard, ChatGPT, ClaudeAI, Clinical research, Computational Linguistics, Dialogue Generation, Dialogue generations, Education computing, Extended reality, Health care education, Healthcare Education, Language Model, Language processing, Large language model, large language models, Natural Language Processing, Natural language processing systems, Natural languages, Nurse Training Simulation, Nursing, Patient avatar, Patient Avatars, Semantics, Students, Training simulation, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper explores the efficacy of Large Language Models (LLMs) in generating dialogues for patient avatars in Virtual Reality (VR) nurse training simulators. With the integration of technology in healthcare education evolving rapidly, the potential of NLP to enhance nurse training through realistic patient interactions presents a significant opportunity. Our study introduces a novel LLM-based dialogue generation system, leveraging models such as ChatGPT, GoogleBard, and ClaudeAI. We detail the development of our script generation system, which was a collaborative endeavor involving nurses, technical artists, and developers. The system, tested on the Meta Quest 2 VR headset, integrates complex dialogues created through a synthesis of clinical expertise and advanced NLP, aimed at simulating real-world nursing scenarios. Through a comprehensive evaluation involving lexical and semantic similarity tests compared to clinical expert-generated scripts, we assess the potential of LLMs as suitable alternatives for script generation. The findings aim to contribute to the development of a more interactive and effective VR nurse training simulator, enhancing communication skills among nursing students for improved patient care outcomes. This research underscores the importance of advanced NLP applications in healthcare education, offering insights into the practicality and limitations of employing LLMs in clinical training environments. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
Liang, Q.; Chen, Y.; Li, W.; Lai, M.; Ni, W.; Qiu, H.
In: L., Zhang; W., Yu; Q., Wang; Y., Laili; Y., Liu (Ed.): Commun. Comput. Info. Sci., pp. 12–24, Springer Science and Business Media Deutschland GmbH, 2024, ISBN: 18650929 (ISSN); 978-981973947-9 (ISBN).
Abstract | Links | BibTeX | Tags: Augmented Reality, Glass, Identity recognition, Internet of Things, Internet of things technologies, IoT, Language learning, Learning systems, LLM, Object Detection, Objects detection, Open Vocabulary Object Detection, Recognition systems, Semantics, Telephone sets, Translation (languages), Translation systems, Visual languages, Wearable computers, Wearable device, Wearable devices
@inproceedings{liang_iknowisee_2024,
title = {iKnowiSee: AR Glasses with Language Learning Translation System and Identity Recognition System Built Based on Large Pre-trained Models of Language and Vision and Internet of Things Technology},
author = {Q. Liang and Y. Chen and W. Li and M. Lai and W. Ni and H. Qiu},
editor = {Zhang L. and Yu W. and Wang Q. and Laili Y. and Liu Y.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85200663840&doi=10.1007%2f978-981-97-3948-6_2&partnerID=40&md5=a0324ba6108674b1d39a338574269d60},
doi = {10.1007/978-981-97-3948-6_2},
isbn = {18650929 (ISSN); 978-981973947-9 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Commun. Comput. Info. Sci.},
volume = {2139 CCIS},
pages = {12–24},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {AR glasses used in daily life have made good progress and have some practical value.However, the current design concept of AR glasses is basically to simply port the content of a cell phone and act as a secondary screen for the phone. In contrast, the AR glasses we designed are based on actual situations, focus on real-world interactions, and utilize IoT technology with the aim of enabling users to fully extract and utilize the digital information in their lives. We have created two innovative features, one is a language learning translation system for users to learn foreign languages, which integrates a large language model with an open vocabulary recognition model to fully extract the visual semantic information of the scene; and the other is a social conferencing system, which utilizes the IoT cloud, pipe, edge, and end development to reduce the cost of communication and improve the efficiency of exchanges in social situations. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024.},
keywords = {Augmented Reality, Glass, Identity recognition, Internet of Things, Internet of things technologies, IoT, Language learning, Learning systems, LLM, Object Detection, Objects detection, Open Vocabulary Object Detection, Recognition systems, Semantics, Telephone sets, Translation (languages), Translation systems, Visual languages, Wearable computers, Wearable device, Wearable devices},
pubstate = {published},
tppubtype = {inproceedings}
}
AR glasses used in daily life have made good progress and have some practical value.However, the current design concept of AR glasses is basically to simply port the content of a cell phone and act as a secondary screen for the phone. In contrast, the AR glasses we designed are based on actual situations, focus on real-world interactions, and utilize IoT technology with the aim of enabling users to fully extract and utilize the digital information in their lives. We have created two innovative features, one is a language learning translation system for users to learn foreign languages, which integrates a large language model with an open vocabulary recognition model to fully extract the visual semantic information of the scene; and the other is a social conferencing system, which utilizes the IoT cloud, pipe, edge, and end development to reduce the cost of communication and improve the efficiency of exchanges in social situations. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024.
Chen, M.; Liu, M.; Wang, C.; Song, X.; Zhang, Z.; Xie, Y.; Wang, L.
Cross-Modal Graph Semantic Communication Assisted by Generative AI in the Metaverse for 6G Journal Article
In: Research, vol. 7, 2024, ISSN: 20965168 (ISSN).
Abstract | Links | BibTeX | Tags: 3-dimensional, 3Dimensional models, Cross-modal, Graph neural networks, Graph semantics, Metaverses, Multi-modal data, Point-clouds, Semantic communication, Semantic features, Semantics, Three dimensional computer graphics, Virtual scenario
@article{chen_cross-modal_2024,
title = {Cross-Modal Graph Semantic Communication Assisted by Generative AI in the Metaverse for 6G},
author = {M. Chen and M. Liu and C. Wang and X. Song and Z. Zhang and Y. Xie and L. Wang},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85192245049&doi=10.34133%2fresearch.0342&partnerID=40&md5=4a1c3e0a3ac877fcdf04937a96da32a1},
doi = {10.34133/research.0342},
issn = {20965168 (ISSN)},
year = {2024},
date = {2024-01-01},
journal = {Research},
volume = {7},
abstract = {Recently, the development of the Metaverse has become a frontier spotlight, which is an important demonstration of the integration innovation of advanced technologies in the Internet. Moreover, artificial intelligence (AI) and 6G communications will be widely used in our daily lives. However, the effective interactions with the representations of multimodal data among users via 6G communications is the main challenge in the Metaverse. In this work, we introduce an intelligent cross-modal graph semantic communication approach based on generative AI and 3-dimensional (3D) point clouds to improve the diversity of multimodal representations in the Metaverse. Using a graph neural network, multimodal data can be recorded by key semantic features related to the real scenarios. Then, we compress the semantic features using a graph transformer encoder at the transmitter, which can extract the semantic representations through the cross-modal attention mechanisms. Next, we leverage a graph semantic validation mechanism to guarantee the exactness of the overall data at the receiver. Furthermore, we adopt generative AI to regenerate multimodal data in virtual scenarios. Simultaneously, a novel 3D generative reconstruction network is constructed from the 3D point clouds, which can transfer the data from images to 3D models, and we infer the multimodal data into the 3D models to increase realism in virtual scenarios. Finally, the experiment results demonstrate that cross-modal graph semantic communication, assisted by generative AI, has substantial potential for enhancing user interactions in the 6G communications and Metaverse. Copyright © 2024 Mingkai Chen et al.},
keywords = {3-dimensional, 3Dimensional models, Cross-modal, Graph neural networks, Graph semantics, Metaverses, Multi-modal data, Point-clouds, Semantic communication, Semantic features, Semantics, Three dimensional computer graphics, Virtual scenario},
pubstate = {published},
tppubtype = {article}
}
Recently, the development of the Metaverse has become a frontier spotlight, which is an important demonstration of the integration innovation of advanced technologies in the Internet. Moreover, artificial intelligence (AI) and 6G communications will be widely used in our daily lives. However, the effective interactions with the representations of multimodal data among users via 6G communications is the main challenge in the Metaverse. In this work, we introduce an intelligent cross-modal graph semantic communication approach based on generative AI and 3-dimensional (3D) point clouds to improve the diversity of multimodal representations in the Metaverse. Using a graph neural network, multimodal data can be recorded by key semantic features related to the real scenarios. Then, we compress the semantic features using a graph transformer encoder at the transmitter, which can extract the semantic representations through the cross-modal attention mechanisms. Next, we leverage a graph semantic validation mechanism to guarantee the exactness of the overall data at the receiver. Furthermore, we adopt generative AI to regenerate multimodal data in virtual scenarios. Simultaneously, a novel 3D generative reconstruction network is constructed from the 3D point clouds, which can transfer the data from images to 3D models, and we infer the multimodal data into the 3D models to increase realism in virtual scenarios. Finally, the experiment results demonstrate that cross-modal graph semantic communication, assisted by generative AI, has substantial potential for enhancing user interactions in the 6G communications and Metaverse. Copyright © 2024 Mingkai Chen et al.
2023
Park, J.; Choi, J.; Kim, S. -L.; Bennis, M.
Enabling the Wireless Metaverse via Semantic Multiverse Communication Proceedings Article
In: Annu. IEEE Commun.Soc. Conf. Sens., Mesh Ad Hoc Commun. Netw. workshops, pp. 85–90, IEEE Computer Society, 2023, ISBN: 21555486 (ISSN); 979-835030052-9 (ISBN).
Abstract | Links | BibTeX | Tags: Deep learning, Extended reality (XR), Federated learning, Fertilizers, Learn+, Learning systems, Metaverse, Metaverses, Modal analysis, Multi agent systems, Multi-agent reinforcement learning, Multi-modal data, Reinforcement Learning, Semantic communication, Semantics, Signal encoding, Signaling game, Split learning, Symbolic artificial intelligence
@inproceedings{park_enabling_2023,
title = {Enabling the Wireless Metaverse via Semantic Multiverse Communication},
author = {J. Park and J. Choi and S. -L. Kim and M. Bennis},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85177465286&doi=10.1109%2fSECON58729.2023.10287438&partnerID=40&md5=b052572fb2f78ce0694c7ae5726c8daf},
doi = {10.1109/SECON58729.2023.10287438},
isbn = {21555486 (ISSN); 979-835030052-9 (ISBN)},
year = {2023},
date = {2023-01-01},
booktitle = {Annu. IEEE Commun.Soc. Conf. Sens., Mesh Ad Hoc Commun. Netw. workshops},
volume = {2023-September},
pages = {85–90},
publisher = {IEEE Computer Society},
abstract = {Metaverse over wireless networks is an emerging use case of the sixth generation (6G) wireless systems, posing unprecedented challenges in terms of its multi-modal data transmissions with stringent latency and reliability requirements. Towards enabling this wireless metaverse, in this article we propose a novel semantic communication (SC) framework by decomposing the metaverse into human/machine agent-specific semantic multiverses (SMs). An SM stored at each agent comprises a semantic encoder and a generator, leveraging recent advances in generative artificial intelligence (AI). To improve communication efficiency, the encoder learns the semantic representations (SRs) of multi-modal data, while the generator learns how to manipulate them for locally rendering scenes and interactions in the metaverse. Since these learned SMs are biased towards local environments, their success hinges on synchronizing heterogeneous SMs in the background while communicating SRs in the foreground, turning the wireless metaverse problem into the problem of semantic multiverse communication (SMC). Based on this SMC architecture, we propose several promising algorithmic and analytic tools for modeling and designing SMC, ranging from distributed learning and multi-agent reinforcement learning (MARL) to signaling games and symbolic AI. © 2023 IEEE.},
keywords = {Deep learning, Extended reality (XR), Federated learning, Fertilizers, Learn+, Learning systems, Metaverse, Metaverses, Modal analysis, Multi agent systems, Multi-agent reinforcement learning, Multi-modal data, Reinforcement Learning, Semantic communication, Semantics, Signal encoding, Signaling game, Split learning, Symbolic artificial intelligence},
pubstate = {published},
tppubtype = {inproceedings}
}
Metaverse over wireless networks is an emerging use case of the sixth generation (6G) wireless systems, posing unprecedented challenges in terms of its multi-modal data transmissions with stringent latency and reliability requirements. Towards enabling this wireless metaverse, in this article we propose a novel semantic communication (SC) framework by decomposing the metaverse into human/machine agent-specific semantic multiverses (SMs). An SM stored at each agent comprises a semantic encoder and a generator, leveraging recent advances in generative artificial intelligence (AI). To improve communication efficiency, the encoder learns the semantic representations (SRs) of multi-modal data, while the generator learns how to manipulate them for locally rendering scenes and interactions in the metaverse. Since these learned SMs are biased towards local environments, their success hinges on synchronizing heterogeneous SMs in the background while communicating SRs in the foreground, turning the wireless metaverse problem into the problem of semantic multiverse communication (SMC). Based on this SMC architecture, we propose several promising algorithmic and analytic tools for modeling and designing SMC, ranging from distributed learning and multi-agent reinforcement learning (MARL) to signaling games and symbolic AI. © 2023 IEEE.
Horvath, I.; Csapo, A. B.
Structured Template Language and Generative AI Driven Content Management for Personalized Workspace Reconfiguration Proceedings Article
In: IEEE Int. Conf. Cogn. Asp. Virtual Real., CVR, Institute of Electrical and Electronics Engineers Inc., 2023, ISBN: 979-835033863-8 (ISBN).
Abstract | Links | BibTeX | Tags: 3D spaces, 3D virtual reality, Cognitive infocommunications, Content management, Content management solutions, Geometric layout, Knowledge engineering, Semantic content, Semantic content management, Semantics, Simple++, Virtual Reality, Work-flows
@inproceedings{horvath_structured_2023,
title = {Structured Template Language and Generative AI Driven Content Management for Personalized Workspace Reconfiguration},
author = {I. Horvath and A. B. Csapo},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85184849596&doi=10.1109%2fCVR58941.2023.10395520&partnerID=40&md5=c3890e80798c9ec542fe453875dde253},
doi = {10.1109/CVR58941.2023.10395520},
isbn = {979-835033863-8 (ISBN)},
year = {2023},
date = {2023-01-01},
booktitle = {IEEE Int. Conf. Cogn. Asp. Virtual Real., CVR},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {This work presents a systematic approach towards personalized workspace management and reconfiguration in 3D Virtual Reality (VR) spaces, focusing on a structured template language for defining and manipulating content layout schemas, as well as a generative AI supported content management solution. Recognizing the varying requirements of different tasks and workflows, on the one hand we propose a hierarchical template language that enables, through simple steps, the adaptation of number and variety of documents within geometric layout schemas in digital 3D spaces. In the second half of the paper, we present a generative AI driven framework for integrating associative-semantic content management into such workspaces, thereby enhancing the relevance and contextuality of workspace configurations. The proposed approach aids in identifying content that is semantically linked to a given spatial, temporal and topical environment, enabling workspace designers and users to create more efficient and personalized workspace layouts. © 2023 IEEE.},
keywords = {3D spaces, 3D virtual reality, Cognitive infocommunications, Content management, Content management solutions, Geometric layout, Knowledge engineering, Semantic content, Semantic content management, Semantics, Simple++, Virtual Reality, Work-flows},
pubstate = {published},
tppubtype = {inproceedings}
}
This work presents a systematic approach towards personalized workspace management and reconfiguration in 3D Virtual Reality (VR) spaces, focusing on a structured template language for defining and manipulating content layout schemas, as well as a generative AI supported content management solution. Recognizing the varying requirements of different tasks and workflows, on the one hand we propose a hierarchical template language that enables, through simple steps, the adaptation of number and variety of documents within geometric layout schemas in digital 3D spaces. In the second half of the paper, we present a generative AI driven framework for integrating associative-semantic content management into such workspaces, thereby enhancing the relevance and contextuality of workspace configurations. The proposed approach aids in identifying content that is semantically linked to a given spatial, temporal and topical environment, enabling workspace designers and users to create more efficient and personalized workspace layouts. © 2023 IEEE.
2022
Peretokin, Vadim; Basdekis, Ioannis; Kouris, Ioannis; Maggesi, Jonatan; Sicuranza, Mario; Su, Qiqi; Acebes, Alberto; Bucur, Anca; Mukkala, Vinod; Pozdniakov, Konstantin; Kloukinas, Christos; Koutsouris, Dimitrios; Iliadou, Elefteria; Leontsinis, Ioannis; Gallo, Luigi; Pietro, Giuseppe De; Spanoudakis, George
Overview of the SMART-BEAR Technical Infrastructure Best Paper Proceedings Article
In: Proceedings of the 8th International Conference on Information and Communication Technologies for Ageing Well and E-Health, pp. 117–125, SCITEPRESS - Science and Technology Publications, Online, 2022, ISBN: 978-989-758-566-1.
Abstract | Links | BibTeX | Tags: Artificial intelligence, Balance Disorder, Cardiovascular Disease, Cloud computing, Evidence-based, GDPR, Healthcare, Hearing Loss, HL7 FHIR, Interoperability, Semantics
@inproceedings{peretokinOverviewSMARTBEARTechnical2022,
title = {Overview of the SMART-BEAR Technical Infrastructure},
author = { Vadim Peretokin and Ioannis Basdekis and Ioannis Kouris and Jonatan Maggesi and Mario Sicuranza and Qiqi Su and Alberto Acebes and Anca Bucur and Vinod Mukkala and Konstantin Pozdniakov and Christos Kloukinas and Dimitrios Koutsouris and Elefteria Iliadou and Ioannis Leontsinis and Luigi Gallo and Giuseppe De Pietro and George Spanoudakis},
doi = {10.5220/0011082700003188},
isbn = {978-989-758-566-1},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Proceedings of the 8th International Conference on Information and Communication Technologies for Ageing Well and E-Health},
pages = {117--125},
publisher = {SCITEPRESS - Science and Technology Publications},
address = {Online},
abstract = {This paper describes a cloud-based platform that offers evidence-based, personalised interventions powered by Artificial Intelligence to help support efficient remote monitoring and clinician-driven guidance to people over 65 who suffer or are at risk of hearing loss, cardiovascular diseases, cognitive impairments, balance disorders, and mental health issues. This platform has been developed within the SMART-BEAR integrated project to power its large-scale clinical pilots and comprises a standards-based data harmonisation and management layer, a security component, a Big Data Analytics system, a Clinical Decision Support tool, and a dashboard component for efficient data collection across the pilot sites.},
keywords = {Artificial intelligence, Balance Disorder, Cardiovascular Disease, Cloud computing, Evidence-based, GDPR, Healthcare, Hearing Loss, HL7 FHIR, Interoperability, Semantics},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper describes a cloud-based platform that offers evidence-based, personalised interventions powered by Artificial Intelligence to help support efficient remote monitoring and clinician-driven guidance to people over 65 who suffer or are at risk of hearing loss, cardiovascular diseases, cognitive impairments, balance disorders, and mental health issues. This platform has been developed within the SMART-BEAR integrated project to power its large-scale clinical pilots and comprises a standards-based data harmonisation and management layer, a security component, a Big Data Analytics system, a Clinical Decision Support tool, and a dashboard component for efficient data collection across the pilot sites.
Peretokin, Vadim; Basdekis, Ioannis; Kouris, Ioannis; Maggesi, Jonatan; Sicuranza, Mario; Su, Qiqi; Acebes, Alberto; Bucur, Anca; Mukkala, Vinod; Pozdniakov, Konstantin; Kloukinas, Christos; Koutsouris, Dimitrios; Iliadou, Elefteria; Leontsinis, Ioannis; Gallo, Luigi; Pietro, Giuseppe De; Spanoudakis, George
Overview of the SMART-BEAR Technical Infrastructure Proceedings Article
In: Proceedings of the 8th International Conference on Information and Communication Technologies for Ageing Well and e-Health, pp. 117–125, SCITEPRESS - Science and Technology Publications, Online, 2022, ISBN: 978-989-758-566-1.
Abstract | Links | BibTeX | Tags: Artificial intelligence, Balance Disorder, Cardiovascular Disease, Cloud computing, Evidence-based, GDPR, Healthcare, Hearing Loss, HL7 FHIR, Interoperability, Semantics
@inproceedings{peretokin_overview_2022,
title = {Overview of the SMART-BEAR Technical Infrastructure},
author = {Vadim Peretokin and Ioannis Basdekis and Ioannis Kouris and Jonatan Maggesi and Mario Sicuranza and Qiqi Su and Alberto Acebes and Anca Bucur and Vinod Mukkala and Konstantin Pozdniakov and Christos Kloukinas and Dimitrios Koutsouris and Elefteria Iliadou and Ioannis Leontsinis and Luigi Gallo and Giuseppe De Pietro and George Spanoudakis},
url = {https://www.scitepress.org/DigitalLibrary/Link.aspx?doi=10.5220/0011082700003188},
doi = {10.5220/0011082700003188},
isbn = {978-989-758-566-1},
year = {2022},
date = {2022-01-01},
urldate = {2023-03-15},
booktitle = {Proceedings of the 8th International Conference on Information and Communication Technologies for Ageing Well and e-Health},
pages = {117–125},
publisher = {SCITEPRESS - Science and Technology Publications},
address = {Online},
abstract = {This paper describes a cloud-based platform that offers evidence-based, personalised interventions powered by Artificial Intelligence to help support efficient remote monitoring and clinician-driven guidance to people over 65 who suffer or are at risk of hearing loss, cardiovascular diseases, cognitive impairments, balance disorders, and mental health issues. This platform has been developed within the SMART-BEAR integrated project to power its large-scale clinical pilots and comprises a standards-based data harmonisation and management layer, a security component, a Big Data Analytics system, a Clinical Decision Support tool, and a dashboard component for efficient data collection across the pilot sites.},
keywords = {Artificial intelligence, Balance Disorder, Cardiovascular Disease, Cloud computing, Evidence-based, GDPR, Healthcare, Hearing Loss, HL7 FHIR, Interoperability, Semantics},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper describes a cloud-based platform that offers evidence-based, personalised interventions powered by Artificial Intelligence to help support efficient remote monitoring and clinician-driven guidance to people over 65 who suffer or are at risk of hearing loss, cardiovascular diseases, cognitive impairments, balance disorders, and mental health issues. This platform has been developed within the SMART-BEAR integrated project to power its large-scale clinical pilots and comprises a standards-based data harmonisation and management layer, a security component, a Big Data Analytics system, a Clinical Decision Support tool, and a dashboard component for efficient data collection across the pilot sites.
2020
Scianna, Andrea; Guardia, Marcello La
Processing of 3D Models for Networking of CH in Geomatics Journal Article
In: Communications in Computer and Information Science, vol. 1246, pp. 267–281, 2020, (Publisher: Springer Science and Business Media Deutschland GmbH).
Abstract | Links | BibTeX | Tags: 3-D environments, 3D Environments, 3D Visualization, Cultural heritage, Cultural heritages, Geo-Spatial Informations, Geomatics, Loading configuration, Quality of information, Semantics, Surveying, Three dimensional computer graphics, Virtual fruitions, Virtual Reality, Visualization
@article{scianna_processing_2020,
title = {Processing of 3D Models for Networking of CH in Geomatics},
author = {Andrea Scianna and Marcello La Guardia},
editor = {Vettore A. Troisi S. Parente C.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85097414626&doi=10.1007%2f978-3-030-62800-0_21&partnerID=40&md5=4f0cbed228d7f218580ab8042953049d},
doi = {10.1007/978-3-030-62800-0_21},
year = {2020},
date = {2020-01-01},
journal = {Communications in Computer and Information Science},
volume = {1246},
pages = {267–281},
abstract = {In recent times the possibility of reconstruction of complex 3D Cultural Heritage (CH) environments has opened new scenarios for touristic and scientific aims. The different needs for networking or conservation purposes of CH lead to study proper structuring of 3D models. In light of this, a scientific approach has been developed in order to test the networking capabilities, comparing different loading configurations of 3D environments with multiple combinations of 3D models inside them, considering different solutions. This experimentation has been based on WebGL-HTML5 technologies and allowed to discover the true balance between performances of proposed system, the quality of visualization, and the quality of information (geometric and semantic ones) characterizing the 3D visualization of the virtual environment. The present work analyzes all of these parameters in order to find the best combination for the implementation of these models into a virtual 3D Geographic Information System (GIS) environment, based on WebGL technologies and accessible via web. This study could be considered a basic step for the development of interactive geospatial information platforms for the virtual fruition of CH. © 2020, Springer Nature Switzerland AG.},
note = {Publisher: Springer Science and Business Media Deutschland GmbH},
keywords = {3-D environments, 3D Environments, 3D Visualization, Cultural heritage, Cultural heritages, Geo-Spatial Informations, Geomatics, Loading configuration, Quality of information, Semantics, Surveying, Three dimensional computer graphics, Virtual fruitions, Virtual Reality, Visualization},
pubstate = {published},
tppubtype = {article}
}
In recent times the possibility of reconstruction of complex 3D Cultural Heritage (CH) environments has opened new scenarios for touristic and scientific aims. The different needs for networking or conservation purposes of CH lead to study proper structuring of 3D models. In light of this, a scientific approach has been developed in order to test the networking capabilities, comparing different loading configurations of 3D environments with multiple combinations of 3D models inside them, considering different solutions. This experimentation has been based on WebGL-HTML5 technologies and allowed to discover the true balance between performances of proposed system, the quality of visualization, and the quality of information (geometric and semantic ones) characterizing the 3D visualization of the virtual environment. The present work analyzes all of these parameters in order to find the best combination for the implementation of these models into a virtual 3D Geographic Information System (GIS) environment, based on WebGL technologies and accessible via web. This study could be considered a basic step for the development of interactive geospatial information platforms for the virtual fruition of CH. © 2020, Springer Nature Switzerland AG.
2018
Cascia, Marco La; Vassallo, Giorgio; Gallo, Luigi; Pilato, Giovanni; Vella, Filippo
Automatic Image Annotation Using Random Projection in a Conceptual Space Induced from Data Proceedings Article
In: 2018 14th International Conference on Signal-Image Technology Internet-Based Systems (SITIS), pp. 464–471, 2018.
Abstract | Links | BibTeX | Tags: Feature extraction, Hidden Markov models, Image annotation, Modeling, Semantics, Visualization
@inproceedings{lacasciaAutomaticImageAnnotation2018,
title = {Automatic Image Annotation Using Random Projection in a Conceptual Space Induced from Data},
author = { Marco La Cascia and Giorgio Vassallo and Luigi Gallo and Giovanni Pilato and Filippo Vella},
doi = {10.1109/SITIS.2018.00077},
year = {2018},
date = {2018-11-01},
booktitle = {2018 14th International Conference on Signal-Image Technology Internet-Based Systems (SITIS)},
pages = {464--471},
abstract = {The main drawback of a detailed representation of visual content, whatever is its origin, is that significant features are very high dimensional. To keep the problem tractable while preserving the semantic content, a dimensionality reduction of the data is needed. We propose the Random Projection techniques to reduce the dimensionality. Even though this technique is sub-optimal with respect to Singular Value Decomposition its much lower computational cost make it more suitable for this problem and in particular when computational resources are limited such as in mobile terminals. In this paper we present the use of a ``conceptual'' space, automatically induced from data, to perform automatic image annotation. Images are represented by visual features based on color and texture and arranged as histograms of visual terms and bigrams to partially preserve the spatial information [1]. Using a set of annotated images as training data, the matrix of visual features is built and dimensionality reduction is performed using the Random Projection algorithm. A new unannotated image is then projected into the dimensionally reduced space and the labels of the closest training images are assigned to the unannotated image itself. Experiments on large real collection of images showed that the approach, despite of its low computational cost, is very effective.},
keywords = {Feature extraction, Hidden Markov models, Image annotation, Modeling, Semantics, Visualization},
pubstate = {published},
tppubtype = {inproceedings}
}
The main drawback of a detailed representation of visual content, whatever is its origin, is that significant features are very high dimensional. To keep the problem tractable while preserving the semantic content, a dimensionality reduction of the data is needed. We propose the Random Projection techniques to reduce the dimensionality. Even though this technique is sub-optimal with respect to Singular Value Decomposition its much lower computational cost make it more suitable for this problem and in particular when computational resources are limited such as in mobile terminals. In this paper we present the use of a ``conceptual'' space, automatically induced from data, to perform automatic image annotation. Images are represented by visual features based on color and texture and arranged as histograms of visual terms and bigrams to partially preserve the spatial information [1]. Using a set of annotated images as training data, the matrix of visual features is built and dimensionality reduction is performed using the Random Projection algorithm. A new unannotated image is then projected into the dimensionally reduced space and the labels of the closest training images are assigned to the unannotated image itself. Experiments on large real collection of images showed that the approach, despite of its low computational cost, is very effective.
Cascia, Marco La; Vassallo, Giorgio; Gallo, Luigi; Pilato, Giovanni; Vella, Filippo
Automatic Image Annotation Using Random Projection in a Conceptual Space Induced from Data Proceedings Article
In: 2018 14th International Conference on Signal-Image Technology Internet-Based Systems (SITIS), pp. 464–471, 2018.
Abstract | Links | BibTeX | Tags: Feature extraction, Hidden Markov models, Image annotation, Modeling, Semantics, Visualization
@inproceedings{la_cascia_automatic_2018,
title = {Automatic Image Annotation Using Random Projection in a Conceptual Space Induced from Data},
author = {Marco La Cascia and Giorgio Vassallo and Luigi Gallo and Giovanni Pilato and Filippo Vella},
doi = {10.1109/SITIS.2018.00077},
year = {2018},
date = {2018-11-01},
booktitle = {2018 14th International Conference on Signal-Image Technology Internet-Based Systems (SITIS)},
pages = {464–471},
abstract = {The main drawback of a detailed representation of visual content, whatever is its origin, is that significant features are very high dimensional. To keep the problem tractable while preserving the semantic content, a dimensionality reduction of the data is needed. We propose the Random Projection techniques to reduce the dimensionality. Even though this technique is sub-optimal with respect to Singular Value Decomposition its much lower computational cost make it more suitable for this problem and in particular when computational resources are limited such as in mobile terminals. In this paper we present the use of a “conceptual” space, automatically induced from data, to perform automatic image annotation. Images are represented by visual features based on color and texture and arranged as histograms of visual terms and bigrams to partially preserve the spatial information [1]. Using a set of annotated images as training data, the matrix of visual features is built and dimensionality reduction is performed using the Random Projection algorithm. A new unannotated image is then projected into the dimensionally reduced space and the labels of the closest training images are assigned to the unannotated image itself. Experiments on large real collection of images showed that the approach, despite of its low computational cost, is very effective.},
keywords = {Feature extraction, Hidden Markov models, Image annotation, Modeling, Semantics, Visualization},
pubstate = {published},
tppubtype = {inproceedings}
}
The main drawback of a detailed representation of visual content, whatever is its origin, is that significant features are very high dimensional. To keep the problem tractable while preserving the semantic content, a dimensionality reduction of the data is needed. We propose the Random Projection techniques to reduce the dimensionality. Even though this technique is sub-optimal with respect to Singular Value Decomposition its much lower computational cost make it more suitable for this problem and in particular when computational resources are limited such as in mobile terminals. In this paper we present the use of a “conceptual” space, automatically induced from data, to perform automatic image annotation. Images are represented by visual features based on color and texture and arranged as histograms of visual terms and bigrams to partially preserve the spatial information [1]. Using a set of annotated images as training data, the matrix of visual features is built and dimensionality reduction is performed using the Random Projection algorithm. A new unannotated image is then projected into the dimensionally reduced space and the labels of the closest training images are assigned to the unannotated image itself. Experiments on large real collection of images showed that the approach, despite of its low computational cost, is very effective.
Scianna, Andrea; Gaglio, Giuseppe Fulvio; Guardia, Marcello La
BIM Modelling of Ancient Buildings Journal Article
In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11196 LNCS, pp. 344–355, 2018, (Publisher: Springer Verlag).
Abstract | Links | BibTeX | Tags: 3D Modelling, Archaeological Site, Archaeology, Architectural design, Building components, Computer aided design, Cultural heritage, Cultural heritages, Electronic data interchange, Geo-spatial, HBIM, Historic Preservation, Information Management, Information Systems, Semantics, Structural design, Surveying, Three dimensional computer graphics
@article{scianna_bim_2018,
title = {BIM Modelling of Ancient Buildings},
author = {Andrea Scianna and Giuseppe Fulvio Gaglio and Marcello La Guardia},
editor = {Wallace M. Brumana R. Fink E.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85055423142&doi=10.1007%2f978-3-030-01762-0_29&partnerID=40&md5=4803fe8c1e8d844de2a786296d5e530a},
doi = {10.1007/978-3-030-01762-0_29},
year = {2018},
date = {2018-01-01},
journal = {Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)},
volume = {11196 LNCS},
pages = {344–355},
abstract = {In the last years, new procedures on design and management of constructions, based on 3D standardised models of building elements, have been proposed. It’s the case of Building Information Modelling (BIM) applications, that, differently from CAD ones, allow to work with libraries of 3D parametrical objects (smart objects) describing geometric, structural and material properties of building elements. This methodology is based on the Industry Foundation Classes (IFC) model, that represents a global standard for the building data exchange. Initially used for the design of new architectures, BIM methodology has been even more considered also for the management and the conservation of historical buildings, thanks to the possibilities of implementation of semantic information of 3D objects, guaranteed by the connection with the external database. At the same time, the lack of regular surfaces and standardised objects are relevant problems that nowadays strongly limit the use of BIM for Cultural Heritage (CH). Anyway, in recent times, the study of parameterised objects has opened new perspectives for BIM applications on historical buildings (HBIM). The present work shows the last achievements on this topic, focusing the problems derived from the application of BIM methodology to CH. In fact, the irregular shape of ancient architectural components, the wide variety of architectural languages that characterise historical buildings, the presence, sometimes, of different stratifications, are clear examples of the difficulties of implementing HBIM methodology for CH. © 2018, Springer Nature Switzerland AG.},
note = {Publisher: Springer Verlag},
keywords = {3D Modelling, Archaeological Site, Archaeology, Architectural design, Building components, Computer aided design, Cultural heritage, Cultural heritages, Electronic data interchange, Geo-spatial, HBIM, Historic Preservation, Information Management, Information Systems, Semantics, Structural design, Surveying, Three dimensional computer graphics},
pubstate = {published},
tppubtype = {article}
}
In the last years, new procedures on design and management of constructions, based on 3D standardised models of building elements, have been proposed. It’s the case of Building Information Modelling (BIM) applications, that, differently from CAD ones, allow to work with libraries of 3D parametrical objects (smart objects) describing geometric, structural and material properties of building elements. This methodology is based on the Industry Foundation Classes (IFC) model, that represents a global standard for the building data exchange. Initially used for the design of new architectures, BIM methodology has been even more considered also for the management and the conservation of historical buildings, thanks to the possibilities of implementation of semantic information of 3D objects, guaranteed by the connection with the external database. At the same time, the lack of regular surfaces and standardised objects are relevant problems that nowadays strongly limit the use of BIM for Cultural Heritage (CH). Anyway, in recent times, the study of parameterised objects has opened new perspectives for BIM applications on historical buildings (HBIM). The present work shows the last achievements on this topic, focusing the problems derived from the application of BIM methodology to CH. In fact, the irregular shape of ancient architectural components, the wide variety of architectural languages that characterise historical buildings, the presence, sometimes, of different stratifications, are clear examples of the difficulties of implementing HBIM methodology for CH. © 2018, Springer Nature Switzerland AG.
Scianna, Andrea; Guardia, Marcello La
Globe Based 3D GIS solutions for Virtual Heritage Proceedings Article
In: G., Labetski A. Stoter J. Agugiaro (Ed.): International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives, pp. 171–177, International Society for Photogrammetry and Remote Sensing, 2018, (Issue: 4/W10).
Abstract | Links | BibTeX | Tags: Cultural heritage, Cultural heritages, Digital libraries, Geo-Spatial Informations, History, Interactive Environments, Multimedia information, Semantics, Surrounding environment, Three dimensional computer graphics, Virtual heritage, Virtual Reality, Web-GIS, WebGL
@inproceedings{scianna_globe_2018,
title = {Globe Based 3D GIS solutions for Virtual Heritage},
author = {Andrea Scianna and Marcello La Guardia},
editor = {Labetski A. Stoter J. Agugiaro G.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85056179414&doi=10.5194%2fisprs-archives-XLII-4-W10-171-2018&partnerID=40&md5=14439e37d85e8ca3c5b53ef4ba9f6491},
doi = {10.5194/isprs-archives-XLII-4-W10-171-2018},
year = {2018},
date = {2018-01-01},
booktitle = {International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives},
volume = {42},
pages = {171–177},
publisher = {International Society for Photogrammetry and Remote Sensing},
abstract = {During the last years, many solutions have been proposed for 3D Virtual Heritage representations. Recently, also new technologies for online gaming evolved, based on javascript libraries (WebGL), used to create and publish virtual interactive environments. They are based on recent Web browser's functionalities, surpassing some limitations of VRML technologies. On the side of geospatial information, technology has evolved from desktop GIS to 2D WebGIS and globe applications. The use of globe applications is, today, very diffused due to its immediate and at the same time impressive representation of the earth surface and territories. These technologies have been, also, applied to Virtual Heritage 3D reconstructions, to improve the fruition of Cultural Heritage (CH), with the achievement of interesting results. The topic of this paper is the experimentation on the fusion between globe based and gaming technologies (in our case WebGL) that allow achieving a more user-centric and powerful solution useful for publishing 3D geospatial information of CH on Web. This choice allows obtaining GIS oriented 3D models, typical of globe applications, and, at the same time, a more immersive exploration of CH and its surrounding environment. In particular, it also gives complementary text and multimedia information on the history, architectural features of each cultural good, based on querying of semantic information. The test field of the research is the construction of the 3D GIS virtual globe model of the Manfredonic Castle of Mussomeli (Sicily-Italy), developed for PON-NEPTIS EU Project, to compare open-source technologies and commercial proprietary applications. © Authors 2018. CC BY 4.0 License.},
note = {Issue: 4/W10},
keywords = {Cultural heritage, Cultural heritages, Digital libraries, Geo-Spatial Informations, History, Interactive Environments, Multimedia information, Semantics, Surrounding environment, Three dimensional computer graphics, Virtual heritage, Virtual Reality, Web-GIS, WebGL},
pubstate = {published},
tppubtype = {inproceedings}
}
During the last years, many solutions have been proposed for 3D Virtual Heritage representations. Recently, also new technologies for online gaming evolved, based on javascript libraries (WebGL), used to create and publish virtual interactive environments. They are based on recent Web browser's functionalities, surpassing some limitations of VRML technologies. On the side of geospatial information, technology has evolved from desktop GIS to 2D WebGIS and globe applications. The use of globe applications is, today, very diffused due to its immediate and at the same time impressive representation of the earth surface and territories. These technologies have been, also, applied to Virtual Heritage 3D reconstructions, to improve the fruition of Cultural Heritage (CH), with the achievement of interesting results. The topic of this paper is the experimentation on the fusion between globe based and gaming technologies (in our case WebGL) that allow achieving a more user-centric and powerful solution useful for publishing 3D geospatial information of CH on Web. This choice allows obtaining GIS oriented 3D models, typical of globe applications, and, at the same time, a more immersive exploration of CH and its surrounding environment. In particular, it also gives complementary text and multimedia information on the history, architectural features of each cultural good, based on querying of semantic information. The test field of the research is the construction of the 3D GIS virtual globe model of the Manfredonic Castle of Mussomeli (Sicily-Italy), developed for PON-NEPTIS EU Project, to compare open-source technologies and commercial proprietary applications. © Authors 2018. CC BY 4.0 License.
2014
Scianna, Andrea; Gristina, Susanna; Paliaga, Silvia
Experimental BIM applications in archaeology: A work-flow Journal Article
In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8740, pp. 490–498, 2014, (Publisher: Springer Verlag).
Abstract | Links | BibTeX | Tags: Archaeological Site, Archaeology, Architectural design, BIM, Data export, Database systems, History, Semantic Model, Semantic Web, Semantics, Social networking (online), Web documentation
@article{scianna_experimental_2014,
title = {Experimental BIM applications in archaeology: A work-flow},
author = {Andrea Scianna and Susanna Gristina and Silvia Paliaga},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-84911945081&doi=10.1007%2f978-3-319-13695-0&partnerID=40&md5=afa2444b3a602b178283e29c86786bd5},
doi = {10.1007/978-3-319-13695-0},
year = {2014},
date = {2014-01-01},
journal = {Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)},
volume = {8740},
pages = {490–498},
abstract = {In the last few decades various conceptual models, methods and techniques have been studied to allow 3D digital access to Cultural Heritage (CH). Among these is BIM (Building Information Modeling): originally built up for construction projects, it has been already experimented in the CH domain, but not enough in the archaeological field. This paper illustrates a framework to create 3D archaeological models integrated with databases using BIM. The models implemented are queryable by the connection with a Relational Database Management System and sharable on the web. Parametric solid and semantic models are integrated with 3D standardized database models that are finally manageable in the public cloud. The BIM application’s work-flow here described has been experimented on the Roman structures inside the Crypt of St. Sergius and Bacchus Church (Rome). The experiment has highlighted capabilities and limitations of BIM applications in the archaeological domain. © Springer International Publishing Switzerland 2014.},
note = {Publisher: Springer Verlag},
keywords = {Archaeological Site, Archaeology, Architectural design, BIM, Data export, Database systems, History, Semantic Model, Semantic Web, Semantics, Social networking (online), Web documentation},
pubstate = {published},
tppubtype = {article}
}
In the last few decades various conceptual models, methods and techniques have been studied to allow 3D digital access to Cultural Heritage (CH). Among these is BIM (Building Information Modeling): originally built up for construction projects, it has been already experimented in the CH domain, but not enough in the archaeological field. This paper illustrates a framework to create 3D archaeological models integrated with databases using BIM. The models implemented are queryable by the connection with a Relational Database Management System and sharable on the web. Parametric solid and semantic models are integrated with 3D standardized database models that are finally manageable in the public cloud. The BIM application’s work-flow here described has been experimented on the Roman structures inside the Crypt of St. Sergius and Bacchus Church (Rome). The experiment has highlighted capabilities and limitations of BIM applications in the archaeological domain. © Springer International Publishing Switzerland 2014.