AHCI RESEARCH GROUP
Publications
Papers published in international journals,
proceedings of conferences, workshops and books.
OUR RESEARCH
Scientific Publications
How to
Here you can find the complete list of our publications.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
You can use the tag cloud to select only the papers dealing with specific research topics.
You can expand the Abstract, Links and BibTex record for each paper.
2025
Zeng, S. -Y.; Liang, T. -Y.
PartConverter: A Part-Oriented Transformation Framework for Point Clouds Journal Article
In: IET Image Processing, vol. 19, no. 1, 2025, ISSN: 17519659 (ISSN).
Abstract | Links | BibTeX | Tags: 3D modeling, 3D models, 3d-modeling, Adversarial networks, attention mechanism, Attention mechanisms, Auto encoders, Cloud transformations, Generative Adversarial Network, Part assembler, Part-oriented, Point cloud transformation, Point-clouds
@article{zeng_partconverter_2025,
title = {PartConverter: A Part-Oriented Transformation Framework for Point Clouds},
author = {S. -Y. Zeng and T. -Y. Liang},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005775417&doi=10.1049%2fipr2.70104&partnerID=40&md5=1ee3178fd6b4a03bc7e299e1292e9694},
doi = {10.1049/ipr2.70104},
issn = {17519659 (ISSN)},
year = {2025},
date = {2025-01-01},
journal = {IET Image Processing},
volume = {19},
number = {1},
abstract = {With generative AI technologies advancing rapidly, the capabilities for 3D model generation and transformation are expanding across industries like manufacturing, healthcare, and virtual reality. However, existing methods based on generative adversarial networks (GANs), autoencoders, or transformers still have notable limitations. They primarily generate entire objects without providing flexibility for independent part transformation or precise control over model components. These constraints pose challenges for applications requiring complex object manipulation and fine-grained adjustments. To overcome these limitations, we propose PartConverter, a novel part-oriented point cloud transformation framework emphasizing flexibility and precision in 3D model transformations. PartConverter leverages attention mechanisms and autoencoders to capture crucial details within each part while modeling the relationships between components, thereby enabling highly customizable, part-wise transformations that maintain overall consistency. Additionally, our part assembler ensures that transformed parts align coherently, resulting in a consistent and realistic final 3D shape. This framework significantly enhances control over detailed part modeling, increasing the flexibility and efficiency of 3D model transformation workflows. © 2025 The Author(s). IET Image Processing published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology.},
keywords = {3D modeling, 3D models, 3d-modeling, Adversarial networks, attention mechanism, Attention mechanisms, Auto encoders, Cloud transformations, Generative Adversarial Network, Part assembler, Part-oriented, Point cloud transformation, Point-clouds},
pubstate = {published},
tppubtype = {article}
}
With generative AI technologies advancing rapidly, the capabilities for 3D model generation and transformation are expanding across industries like manufacturing, healthcare, and virtual reality. However, existing methods based on generative adversarial networks (GANs), autoencoders, or transformers still have notable limitations. They primarily generate entire objects without providing flexibility for independent part transformation or precise control over model components. These constraints pose challenges for applications requiring complex object manipulation and fine-grained adjustments. To overcome these limitations, we propose PartConverter, a novel part-oriented point cloud transformation framework emphasizing flexibility and precision in 3D model transformations. PartConverter leverages attention mechanisms and autoencoders to capture crucial details within each part while modeling the relationships between components, thereby enabling highly customizable, part-wise transformations that maintain overall consistency. Additionally, our part assembler ensures that transformed parts align coherently, resulting in a consistent and realistic final 3D shape. This framework significantly enhances control over detailed part modeling, increasing the flexibility and efficiency of 3D model transformation workflows. © 2025 The Author(s). IET Image Processing published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology.
Kurai, R.; Hiraki, T.; Hiroi, Y.; Hirao, Y.; Perusquia-Hernandez, M.; Uchiyama, H.; Kiyokawa, K.
An implementation of MagicCraft: Generating Interactive 3D Objects and Their Behaviors from Text for Commercial Metaverse Platforms Proceedings Article
In: Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW, pp. 1284–1285, Institute of Electrical and Electronics Engineers Inc., 2025, ISBN: 979-833151484-6 (ISBN).
Abstract | Links | BibTeX | Tags: 3D modeling, 3D models, 3D object, 3D Object Generation, 3d-modeling, AI-Assisted Design, Generative AI, Immersive, Metaverse, Metaverses, Model skill, Object oriented programming, Programming skills
@inproceedings{kurai_implementation_2025,
title = {An implementation of MagicCraft: Generating Interactive 3D Objects and Their Behaviors from Text for Commercial Metaverse Platforms},
author = {R. Kurai and T. Hiraki and Y. Hiroi and Y. Hirao and M. Perusquia-Hernandez and H. Uchiyama and K. Kiyokawa},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105005153642&doi=10.1109%2fVRW66409.2025.00288&partnerID=40&md5=53fa1ac92c3210f0ffa090ffa1af7e6e},
doi = {10.1109/VRW66409.2025.00288},
isbn = {979-833151484-6 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW},
pages = {1284–1285},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Metaverse platforms are rapidly evolving to provide immersive spaces. However, the generation of dynamic and interactive 3D objects remains a challenge due to the need for advanced 3D modeling and programming skills. We present MagicCraft, a system that generates functional 3D objects from natural language prompts. MagicCraft uses generative AI models to manage the entire content creation pipeline: converting user text descriptions into images, transforming images into 3D models, predicting object behavior, and assigning necessary attributes and scripts. It also provides an interactive interface for users to refine generated objects by adjusting features like orientation, scale, seating positions, and grip points. © 2025 IEEE.},
keywords = {3D modeling, 3D models, 3D object, 3D Object Generation, 3d-modeling, AI-Assisted Design, Generative AI, Immersive, Metaverse, Metaverses, Model skill, Object oriented programming, Programming skills},
pubstate = {published},
tppubtype = {inproceedings}
}
Metaverse platforms are rapidly evolving to provide immersive spaces. However, the generation of dynamic and interactive 3D objects remains a challenge due to the need for advanced 3D modeling and programming skills. We present MagicCraft, a system that generates functional 3D objects from natural language prompts. MagicCraft uses generative AI models to manage the entire content creation pipeline: converting user text descriptions into images, transforming images into 3D models, predicting object behavior, and assigning necessary attributes and scripts. It also provides an interactive interface for users to refine generated objects by adjusting features like orientation, scale, seating positions, and grip points. © 2025 IEEE.
Peter, K.; Makosa, I.; Auala, S.; Ndjao, L.; Maasz, D.; Mbinge, U.; Winschiers-Theophilus, H.
Co-creating a VR Narrative Experience of Constructing a Food Storage Following OvaHimba Traditional Practices Proceedings Article
In: IMX - Proc. ACM Int. Conf. Interact. Media Experiences, pp. 418–423, Association for Computing Machinery, Inc, 2025, ISBN: 979-840071391-0 (ISBN).
Abstract | Links | BibTeX | Tags: 3D Modelling, 3D models, 3d-modeling, Co-designs, Community-based, Community-Based Co-Design, Computer aided design, Cultural heritage, Cultural heritages, Food storage, Human computer interaction, Human engineering, Indigenous Knowledge, Information Systems, Interactive computer graphics, Interactive computer systems, IVR, Namibia, OvaHimba, Ovahimbum, Photogrammetry, Sustainable development, Virtual environments, Virtual Reality
@inproceedings{peter_co-creating_2025,
title = {Co-creating a VR Narrative Experience of Constructing a Food Storage Following OvaHimba Traditional Practices},
author = {K. Peter and I. Makosa and S. Auala and L. Ndjao and D. Maasz and U. Mbinge and H. Winschiers-Theophilus},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105007984089&doi=10.1145%2f3706370.3731652&partnerID=40&md5=36f95823413852d636b39bd561c97917},
doi = {10.1145/3706370.3731652},
isbn = {979-840071391-0 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {IMX - Proc. ACM Int. Conf. Interact. Media Experiences},
pages = {418–423},
publisher = {Association for Computing Machinery, Inc},
abstract = {As part of an attempt to co-create a comprehensive virtual environment in which one can explore and learn traditional practices of the OvaHimba people, we have co-designed and implemented a VR experience to construct a traditional food storage. In collaboration with the OvaHimba community residing in Otjisa, we have explored culturally valid representations of the process. We have further investigated different techniques such as photogrammetry, generative AI and manual methods to develop 3D models. Our findings highlight the importance of context, process, and community-defined relevance in co-design, the fluidity of cultural realities and virtual representations, as well as technical challenges. © 2025 Copyright held by the owner/author(s).},
keywords = {3D Modelling, 3D models, 3d-modeling, Co-designs, Community-based, Community-Based Co-Design, Computer aided design, Cultural heritage, Cultural heritages, Food storage, Human computer interaction, Human engineering, Indigenous Knowledge, Information Systems, Interactive computer graphics, Interactive computer systems, IVR, Namibia, OvaHimba, Ovahimbum, Photogrammetry, Sustainable development, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
As part of an attempt to co-create a comprehensive virtual environment in which one can explore and learn traditional practices of the OvaHimba people, we have co-designed and implemented a VR experience to construct a traditional food storage. In collaboration with the OvaHimba community residing in Otjisa, we have explored culturally valid representations of the process. We have further investigated different techniques such as photogrammetry, generative AI and manual methods to develop 3D models. Our findings highlight the importance of context, process, and community-defined relevance in co-design, the fluidity of cultural realities and virtual representations, as well as technical challenges. © 2025 Copyright held by the owner/author(s).
Behravan, M.; Haghani, M.; Gračanin, D.
Transcending Dimensions Using Generative AI: Real-Time 3D Model Generation in Augmented Reality Proceedings Article
In: J.Y.C., Chen; G., Fragomeni (Ed.): Lect. Notes Comput. Sci., pp. 13–32, Springer Science and Business Media Deutschland GmbH, 2025, ISBN: 03029743 (ISSN); 978-303193699-9 (ISBN).
Abstract | Links | BibTeX | Tags: 3D Model Generation, 3D modeling, 3D models, 3d-modeling, Augmented Reality, Generative AI, Image-to-3D conversion, Model generation, Object Detection, Object recognition, Objects detection, Real- time, Specialized software, Technical expertise, Three dimensional computer graphics, Usability engineering
@inproceedings{behravan_transcending_2025,
title = {Transcending Dimensions Using Generative AI: Real-Time 3D Model Generation in Augmented Reality},
author = {M. Behravan and M. Haghani and D. Gračanin},
editor = {Chen J.Y.C. and Fragomeni G.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-105007690904&doi=10.1007%2f978-3-031-93700-2_2&partnerID=40&md5=1c4d643aad88d08cbbc9dd2c02413f10},
doi = {10.1007/978-3-031-93700-2_2},
isbn = {03029743 (ISSN); 978-303193699-9 (ISBN)},
year = {2025},
date = {2025-01-01},
booktitle = {Lect. Notes Comput. Sci.},
volume = {15788 LNCS},
pages = {13–32},
publisher = {Springer Science and Business Media Deutschland GmbH},
abstract = {Traditional 3D modeling requires technical expertise, specialized software, and time-intensive processes, making it inaccessible for many users. Our research aims to lower these barriers by combining generative AI and augmented reality (AR) into a cohesive system that allows users to easily generate, manipulate, and interact with 3D models in real time, directly within AR environments. Utilizing cutting-edge AI models like Shap-E, we address the complex challenges of transforming 2D images into 3D representations in AR environments. Key challenges such as object isolation, handling intricate backgrounds, and achieving seamless user interaction are tackled through advanced object detection methods, such as Mask R-CNN. Evaluation results from 35 participants reveal an overall System Usability Scale (SUS) score of 69.64, with participants who engaged with AR/VR technologies more frequently rating the system significantly higher, at 80.71. This research is particularly relevant for applications in gaming, education, and AR-based e-commerce, offering intuitive, model creation for users without specialized skills. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.},
keywords = {3D Model Generation, 3D modeling, 3D models, 3d-modeling, Augmented Reality, Generative AI, Image-to-3D conversion, Model generation, Object Detection, Object recognition, Objects detection, Real- time, Specialized software, Technical expertise, Three dimensional computer graphics, Usability engineering},
pubstate = {published},
tppubtype = {inproceedings}
}
Traditional 3D modeling requires technical expertise, specialized software, and time-intensive processes, making it inaccessible for many users. Our research aims to lower these barriers by combining generative AI and augmented reality (AR) into a cohesive system that allows users to easily generate, manipulate, and interact with 3D models in real time, directly within AR environments. Utilizing cutting-edge AI models like Shap-E, we address the complex challenges of transforming 2D images into 3D representations in AR environments. Key challenges such as object isolation, handling intricate backgrounds, and achieving seamless user interaction are tackled through advanced object detection methods, such as Mask R-CNN. Evaluation results from 35 participants reveal an overall System Usability Scale (SUS) score of 69.64, with participants who engaged with AR/VR technologies more frequently rating the system significantly higher, at 80.71. This research is particularly relevant for applications in gaming, education, and AR-based e-commerce, offering intuitive, model creation for users without specialized skills. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
2024
Weng, S. C. -C.
Studying How Prompt-Generated 3D Models Affect the Creation Process of Mixed Reality Applications Proceedings Article
In: U., Eck; M., Sra; J., Stefanucci; M., Sugimoto; M., Tatzgern; I., Williams (Ed.): Proc. - IEEE Int. Symp. Mixed Augment. Real. Adjunct, ISMAR-Adjunct, pp. 654–655, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-833150691-9 (ISBN).
Abstract | Links | BibTeX | Tags: 3D modeling, 3D models, 3d-modeling, Creation process, Generative AI, Mixed reality, Prompt-generated 3d model, Prompt-generated 3D models, Research prototype, Study plans, User study
@inproceedings{weng_studying_2024,
title = {Studying How Prompt-Generated 3D Models Affect the Creation Process of Mixed Reality Applications},
author = {S. C. -C. Weng},
editor = {Eck U. and Sra M. and Stefanucci J. and Sugimoto M. and Tatzgern M. and Williams I.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85214403987&doi=10.1109%2fISMAR-Adjunct64951.2024.00196&partnerID=40&md5=46d553927e96356d73ffc5996fbbdc71},
doi = {10.1109/ISMAR-Adjunct64951.2024.00196},
isbn = {979-833150691-9 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - IEEE Int. Symp. Mixed Augment. Real. Adjunct, ISMAR-Adjunct},
pages = {654–655},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {In this doctoral consortium, we build upon our previous research prototype, Dream Mesh, a Mixed Reality application that generates models in MR based on user speech prompts. To evaluate the application and answer the questions derived from our pilot research, I propose a future user study plan. This plan aims to investigate how prompt-generated 3D models affect the creation process of Mixed Reality applications. © 2024 IEEE.},
keywords = {3D modeling, 3D models, 3d-modeling, Creation process, Generative AI, Mixed reality, Prompt-generated 3d model, Prompt-generated 3D models, Research prototype, Study plans, User study},
pubstate = {published},
tppubtype = {inproceedings}
}
In this doctoral consortium, we build upon our previous research prototype, Dream Mesh, a Mixed Reality application that generates models in MR based on user speech prompts. To evaluate the application and answer the questions derived from our pilot research, I propose a future user study plan. This plan aims to investigate how prompt-generated 3D models affect the creation process of Mixed Reality applications. © 2024 IEEE.
Hart, A.; Shakir, M. Z.
Realtime AI Driven Environment Development for Virtual Metaverse Proceedings Article
In: IEEE Int. Conf. Metrol. Ext. Real., Artif. Intell. Neural Eng., MetroXRAINE - Proc., pp. 313–318, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-835037800-9 (ISBN).
Abstract | Links | BibTeX | Tags: 3D modeling, 3D models, 3d-modeling, AI in Metaverse Development, Artificial intelligence in metaverse development, Digital elevation model, Digital Innovation, Digital innovations, Metaverses, Real- time, Real-Time Adaptation, Scalable virtual world, Scalable Virtual Worlds, Unity Integration, Virtual environments, Virtual worlds
@inproceedings{hart_realtime_2024,
title = {Realtime AI Driven Environment Development for Virtual Metaverse},
author = {A. Hart and M. Z. Shakir},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85216090810&doi=10.1109%2fMetroXRAINE62247.2024.10796022&partnerID=40&md5=e339d3117291e480231b7bc32f117506},
doi = {10.1109/MetroXRAINE62247.2024.10796022},
isbn = {979-835037800-9 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {IEEE Int. Conf. Metrol. Ext. Real., Artif. Intell. Neural Eng., MetroXRAINE - Proc.},
pages = {313–318},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {The integration of Artificial Intelligence (AI) into the development of Metaverse environments denotes a noteworthy shift towards crafting virtual spaces with improved interactivity, immersion, and realism. This study looks to delve into the various roles AI plays in using 3D models, enriching experiences in virtual and augmented reality, to create scalable, dynamic virtual environments. It carefully examines the challenges related to computational demands, such as processing power and data storage, scalability issues, and ethical considerations concerning privacy and the misuse of AI -generated content. By exploring AI's application in game engine platforms such as Unity through research ongoing, this paper highlights the technical achievements and ever growing possibilities unlocked by AI, such as creating lifelike virtual environments. © 2024 IEEE.},
keywords = {3D modeling, 3D models, 3d-modeling, AI in Metaverse Development, Artificial intelligence in metaverse development, Digital elevation model, Digital Innovation, Digital innovations, Metaverses, Real- time, Real-Time Adaptation, Scalable virtual world, Scalable Virtual Worlds, Unity Integration, Virtual environments, Virtual worlds},
pubstate = {published},
tppubtype = {inproceedings}
}
The integration of Artificial Intelligence (AI) into the development of Metaverse environments denotes a noteworthy shift towards crafting virtual spaces with improved interactivity, immersion, and realism. This study looks to delve into the various roles AI plays in using 3D models, enriching experiences in virtual and augmented reality, to create scalable, dynamic virtual environments. It carefully examines the challenges related to computational demands, such as processing power and data storage, scalability issues, and ethical considerations concerning privacy and the misuse of AI -generated content. By exploring AI's application in game engine platforms such as Unity through research ongoing, this paper highlights the technical achievements and ever growing possibilities unlocked by AI, such as creating lifelike virtual environments. © 2024 IEEE.
Kim, S. J.; Cao, D. D.; Spinola, F.; Lee, S. J.; Cho, K. S.
RoomRecon: High-Quality Textured Room Layout Reconstruction on Mobile Devices Proceedings Article
In: U., Eck; M., Sra; J., Stefanucci; M., Sugimoto; M., Tatzgern; I., Williams (Ed.): Proc. - IEEE Int. Symp. Mixed Augment. Real., ISMAR, pp. 544–553, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-833151647-5 (ISBN).
Abstract | Links | BibTeX | Tags: 3D modeling, 3D models, 3D reconstruction, 3d-modeling, AR-assisted image capturing, Architectural design, Augmented Reality, Augmented reality-assisted image capturing, Image capturing, Indoor 3D reconstruction, Indoor space, Mobile application, Mobile Applications, Mortar, Room layout, Texturing, Texturing quality
@inproceedings{kim_roomrecon_2024,
title = {RoomRecon: High-Quality Textured Room Layout Reconstruction on Mobile Devices},
author = {S. J. Kim and D. D. Cao and F. Spinola and S. J. Lee and K. S. Cho},
editor = {Eck U. and Sra M. and Stefanucci J. and Sugimoto M. and Tatzgern M. and Williams I.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85213494599&doi=10.1109%2fISMAR62088.2024.00069&partnerID=40&md5=0f6b9d4c44d9c55cafba7ad76651ea07},
doi = {10.1109/ISMAR62088.2024.00069},
isbn = {979-833151647-5 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - IEEE Int. Symp. Mixed Augment. Real., ISMAR},
pages = {544–553},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Widespread RGB-Depth (RGB-D) sensors and advanced 3D reconstruction technologies facilitate the capture of indoor spaces, improving the fields of augmented reality (AR), virtual reality (VR), and extended reality (XR). Nevertheless, current technologies still face limitations, such as the inability to reflect minor scene changes without a complete recapture, the lack of semantic scene understanding, and various texturing challenges that affect the 3D model's visual quality. These issues affect the realism required for VR experiences and other applications such as in interior design and real estate. To address these challenges, we introduce RoomRecon, an interactive, real-time scanning and texturing pipeline for 3D room models. We propose a two-phase texturing pipeline that integrates AR-guided image capturing for texturing and generative AI models to improve texturing quality and provide better replicas of indoor spaces. Moreover, we suggest to focus only on permanent room elements such as walls, floors, and ceilings, to allow for easily customizable 3D models. We conduct experiments in a variety of indoor spaces to assess the texturing quality and speed of our method. The quantitative results and user study demonstrate that RoomRecon surpasses state-of-the-art methods in terms of texturing quality and on-device computation time. © 2024 IEEE.},
keywords = {3D modeling, 3D models, 3D reconstruction, 3d-modeling, AR-assisted image capturing, Architectural design, Augmented Reality, Augmented reality-assisted image capturing, Image capturing, Indoor 3D reconstruction, Indoor space, Mobile application, Mobile Applications, Mortar, Room layout, Texturing, Texturing quality},
pubstate = {published},
tppubtype = {inproceedings}
}
Widespread RGB-Depth (RGB-D) sensors and advanced 3D reconstruction technologies facilitate the capture of indoor spaces, improving the fields of augmented reality (AR), virtual reality (VR), and extended reality (XR). Nevertheless, current technologies still face limitations, such as the inability to reflect minor scene changes without a complete recapture, the lack of semantic scene understanding, and various texturing challenges that affect the 3D model's visual quality. These issues affect the realism required for VR experiences and other applications such as in interior design and real estate. To address these challenges, we introduce RoomRecon, an interactive, real-time scanning and texturing pipeline for 3D room models. We propose a two-phase texturing pipeline that integrates AR-guided image capturing for texturing and generative AI models to improve texturing quality and provide better replicas of indoor spaces. Moreover, we suggest to focus only on permanent room elements such as walls, floors, and ceilings, to allow for easily customizable 3D models. We conduct experiments in a variety of indoor spaces to assess the texturing quality and speed of our method. The quantitative results and user study demonstrate that RoomRecon surpasses state-of-the-art methods in terms of texturing quality and on-device computation time. © 2024 IEEE.
Paweroi, R. M.; Koppen, M.
Framework for Integration of Generative AI into Metaverse Asset Creation Proceedings Article
In: Int. Conf. Intell. Metaverse Technol. Appl., iMETA, pp. 27–33, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-835035151-4 (ISBN).
Abstract | Links | BibTeX | Tags: 3D Asset Creation, 3D Asset Diversity, 3D models, 3d-modeling, Digital assets, Digital Objects, Generative adversarial networks, Generative AI, High quality, Metaverse, Metaverses, Virtual worlds
@inproceedings{paweroi_framework_2024,
title = {Framework for Integration of Generative AI into Metaverse Asset Creation},
author = {R. M. Paweroi and M. Koppen},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85216024340&doi=10.1109%2fiMETA62882.2024.10808057&partnerID=40&md5=00373291c3d224b53759dc39ed9fd65c},
doi = {10.1109/iMETA62882.2024.10808057},
isbn = {979-835035151-4 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Int. Conf. Intell. Metaverse Technol. Appl., iMETA},
pages = {27–33},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Metaverse, a virtual world, is developing rapidly and is widely used in multi-sector. The number of users is projected to increase year over year. Due to the development of the metaverse platform, digital asset creation is demanding. Creating high-quality and diverse 3D digital objects is challenging. This study proposes the frameworks for integrating generative AI to create diverse 3D assets into the metaverse. We study different approaches for asset creation, i.e., generative 3D model-based, generative image projection-based, and generative language script-based. Creators can use this workflow to optimize the creation of 3D assets. Moreover, this study compares the results of generative AI and procedural generation on generating diverse 3D objects. The result shows that generative AI can simplify 3D creation and generate more diverse objects. © 2024 IEEE.},
keywords = {3D Asset Creation, 3D Asset Diversity, 3D models, 3d-modeling, Digital assets, Digital Objects, Generative adversarial networks, Generative AI, High quality, Metaverse, Metaverses, Virtual worlds},
pubstate = {published},
tppubtype = {inproceedings}
}
Metaverse, a virtual world, is developing rapidly and is widely used in multi-sector. The number of users is projected to increase year over year. Due to the development of the metaverse platform, digital asset creation is demanding. Creating high-quality and diverse 3D digital objects is challenging. This study proposes the frameworks for integrating generative AI to create diverse 3D assets into the metaverse. We study different approaches for asset creation, i.e., generative 3D model-based, generative image projection-based, and generative language script-based. Creators can use this workflow to optimize the creation of 3D assets. Moreover, this study compares the results of generative AI and procedural generation on generating diverse 3D objects. The result shows that generative AI can simplify 3D creation and generate more diverse objects. © 2024 IEEE.
Weid, M.; Khezrian, N.; Mana, A. P.; Farzinnejad, F.; Grubert, J.
GenDeck: Towards a HoloDeck with Text-to-3D Model Generation Proceedings Article
In: Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW, pp. 1188–1189, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-835037449-0 (ISBN).
Abstract | Links | BibTeX | Tags: 3D content, 3D modeling, 3D models, 3d-modeling, Computational costs, Extende Reality, Human computer interaction, Immersive virtual reality, Knowledge Work, Model generation, Proof of concept, Three dimensional computer graphics, Virtual Reality, Visual fidelity
@inproceedings{weid_gendeck_2024,
title = {GenDeck: Towards a HoloDeck with Text-to-3D Model Generation},
author = {M. Weid and N. Khezrian and A. P. Mana and F. Farzinnejad and J. Grubert},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85195600251&doi=10.1109%2fVRW62533.2024.00388&partnerID=40&md5=6dab0cc05259fa2dbe0a2b3806e569af},
doi = {10.1109/VRW62533.2024.00388},
isbn = {979-835037449-0 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - IEEE Conf. Virtual Real. 3D User Interfaces Abstr. Workshops, VRW},
pages = {1188–1189},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Generative Artificial Intelligence has the potential to substantially transform the way 3D content for Extended Reality applications is produced. Specifically, the development of text-to-3D and image-to-3D generators with increasing visual fidelity and decreasing computational costs is thriving quickly. Within this work, we present GenDeck, a proof-of-concept application to experience text-to-3D model generation inside an immersive Virtual Reality environment. © 2024 IEEE.},
keywords = {3D content, 3D modeling, 3D models, 3d-modeling, Computational costs, Extende Reality, Human computer interaction, Immersive virtual reality, Knowledge Work, Model generation, Proof of concept, Three dimensional computer graphics, Virtual Reality, Visual fidelity},
pubstate = {published},
tppubtype = {inproceedings}
}
Generative Artificial Intelligence has the potential to substantially transform the way 3D content for Extended Reality applications is produced. Specifically, the development of text-to-3D and image-to-3D generators with increasing visual fidelity and decreasing computational costs is thriving quickly. Within this work, we present GenDeck, a proof-of-concept application to experience text-to-3D model generation inside an immersive Virtual Reality environment. © 2024 IEEE.
Rausa, M.; Gaglio, S.; Augello, A.; Caggianese, G.; Franchini, S.; Gallo, L.; Sabatucci, L.
Enriching Metaverse with Memories Through Generative AI: A Case Study Proceedings Article
In: IEEE Int. Conf. Metrol. Ext. Real., Artif. Intell. Neural Eng., MetroXRAINE - Proc., pp. 371–376, Institute of Electrical and Electronics Engineers Inc., St Albans, United Kingdom, 2024, ISBN: 979-835037800-9 (ISBN).
Abstract | Links | BibTeX | Tags: 3D modeling, 3D models, 3D reconstruction, 3d-modeling, Case-studies, Generative adversarial networks, Generative AI, Input modes, Metamemory, Metaverses, Synthetic Data Generation, Synthetic data generations, Textual description, Virtual environments, Virtual Reality
@inproceedings{rausa_enriching_2024,
title = {Enriching Metaverse with Memories Through Generative AI: A Case Study},
author = {M. Rausa and S. Gaglio and A. Augello and G. Caggianese and S. Franchini and L. Gallo and L. Sabatucci},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85216124702&doi=10.1109%2fMetroXRAINE62247.2024.10796338&partnerID=40&md5=580d0727ab8740a6ada62eeef5ac283f},
doi = {10.1109/MetroXRAINE62247.2024.10796338},
isbn = {979-835037800-9 (ISBN)},
year = {2024},
date = {2024-01-01},
urldate = {2025-01-07},
booktitle = {IEEE Int. Conf. Metrol. Ext. Real., Artif. Intell. Neural Eng., MetroXRAINE - Proc.},
pages = {371–376},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
address = {St Albans, United Kingdom},
abstract = {The paper introduces MetaMemory, an approach to generate 3D models from either textual descriptions or photographs of objects, offering dual input modes for enhanced representation. MetaMemory's architecture is discussed presenting the tools employed in extracting the object from the image, generating the 3D mesh from texts or images, and visualizing the object reconstruction in an immersive scenario. Afterwards, a case study in which we experienced reconstructing memories of ancient crafts is examined together with the achieved results, by highlighting current limitations and potential applications. © 2024 IEEE.},
keywords = {3D modeling, 3D models, 3D reconstruction, 3d-modeling, Case-studies, Generative adversarial networks, Generative AI, Input modes, Metamemory, Metaverses, Synthetic Data Generation, Synthetic data generations, Textual description, Virtual environments, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
The paper introduces MetaMemory, an approach to generate 3D models from either textual descriptions or photographs of objects, offering dual input modes for enhanced representation. MetaMemory's architecture is discussed presenting the tools employed in extracting the object from the image, generating the 3D mesh from texts or images, and visualizing the object reconstruction in an immersive scenario. Afterwards, a case study in which we experienced reconstructing memories of ancient crafts is examined together with the achieved results, by highlighting current limitations and potential applications. © 2024 IEEE.
Weng, S. C. -C.; Chiou, Y. -M.; Do, E. Y. -L.
Dream Mesh: A Speech-to-3D Model Generative Pipeline in Mixed Reality Proceedings Article
In: Proc. - IEEE Int. Conf. Artif. Intell. Ext. Virtual Real., AIxVR, pp. 345–349, Institute of Electrical and Electronics Engineers Inc., 2024, ISBN: 979-835037202-1 (ISBN).
Abstract | Links | BibTeX | Tags: 3D content, 3D modeling, 3D models, 3d-modeling, Augmented Reality, Digital assets, Generative AI, generative artificial intelligence, Intelligence models, Mesh generation, Mixed reality, Modeling, Speech-to-3D, Text modeling, Three dimensional computer graphics, User interfaces
@inproceedings{weng_dream_2024,
title = {Dream Mesh: A Speech-to-3D Model Generative Pipeline in Mixed Reality},
author = {S. C. -C. Weng and Y. -M. Chiou and E. Y. -L. Do},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85187218106&doi=10.1109%2fAIxVR59861.2024.00059&partnerID=40&md5=5bfe206e841f23de6458f88a0824bd4d},
doi = {10.1109/AIxVR59861.2024.00059},
isbn = {979-835037202-1 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. - IEEE Int. Conf. Artif. Intell. Ext. Virtual Real., AIxVR},
pages = {345–349},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Generative Artificial Intelligence (AI) models have risen to prominence due to their unparalleled ability to craft and generate digital assets, encompassing text, images, audio, video, and 3D models. Leveraging the capabilities of diffusion models, such as Stable Diffusion and Instruct pix2pix, users can guide AI with specific prompts, streamlining the creative journey for graphic designers. However, the primary application of these models has been to graphic content within desktop interfaces, prompting professionals in interior and architectural design to seek more tailored solutions for their daily operations. To bridge this gap, Augmented Reality (AR) and Mixed Reality (MR) technologies offer a promising solution, transforming traditional 2D artworks into engaging 3D interactive realms. In this paper, we present "Dream Mesh,"a MR application MR tool that combines a Speech-to-3D generative workflow besed on DreamFusion model without relying on pre-existing 3D content libraries. This innovative system empowers users to express 3D content needs through natural language input, promising transformative potential in real-time 3D content creation and an enhanced MR user experience. © 2024 IEEE.},
keywords = {3D content, 3D modeling, 3D models, 3d-modeling, Augmented Reality, Digital assets, Generative AI, generative artificial intelligence, Intelligence models, Mesh generation, Mixed reality, Modeling, Speech-to-3D, Text modeling, Three dimensional computer graphics, User interfaces},
pubstate = {published},
tppubtype = {inproceedings}
}
Generative Artificial Intelligence (AI) models have risen to prominence due to their unparalleled ability to craft and generate digital assets, encompassing text, images, audio, video, and 3D models. Leveraging the capabilities of diffusion models, such as Stable Diffusion and Instruct pix2pix, users can guide AI with specific prompts, streamlining the creative journey for graphic designers. However, the primary application of these models has been to graphic content within desktop interfaces, prompting professionals in interior and architectural design to seek more tailored solutions for their daily operations. To bridge this gap, Augmented Reality (AR) and Mixed Reality (MR) technologies offer a promising solution, transforming traditional 2D artworks into engaging 3D interactive realms. In this paper, we present "Dream Mesh,"a MR application MR tool that combines a Speech-to-3D generative workflow besed on DreamFusion model without relying on pre-existing 3D content libraries. This innovative system empowers users to express 3D content needs through natural language input, promising transformative potential in real-time 3D content creation and an enhanced MR user experience. © 2024 IEEE.
Scott, A. J. S.; McCuaig, F.; Lim, V.; Watkins, W.; Wang, J.; Strachan, G.
Revolutionizing Nurse Practitioner Training: Integrating Virtual Reality and Large Language Models for Enhanced Clinical Education Proceedings Article
In: G., Strudwick; N.R., Hardiker; G., Rees; R., Cook; R., Cook; Y.J., Lee (Ed.): Stud. Health Technol. Informatics, pp. 671–672, IOS Press BV, 2024, ISBN: 09269630 (ISSN); 978-164368527-4 (ISBN).
Abstract | Links | BibTeX | Tags: 3D modeling, 3D models, 3d-modeling, adult, anamnesis, clinical decision making, clinical education, Clinical Simulation, Computational Linguistics, computer interface, Computer-Assisted Instruction, conference paper, Curriculum, Decision making, E-Learning, Education, Health care education, Healthcare Education, human, Humans, Language Model, Large language model, large language models, Mesh generation, Model animations, Modeling languages, nurse practitioner, Nurse Practitioners, Nursing, nursing education, nursing student, OSCE preparation, procedures, simulation, Teaching, therapy, Training, Training program, User-Computer Interface, Virtual Reality, Virtual reality training
@inproceedings{scott_revolutionizing_2024,
title = {Revolutionizing Nurse Practitioner Training: Integrating Virtual Reality and Large Language Models for Enhanced Clinical Education},
author = {A. J. S. Scott and F. McCuaig and V. Lim and W. Watkins and J. Wang and G. Strachan},
editor = {Strudwick G. and Hardiker N.R. and Rees G. and Cook R. and Cook R. and Lee Y.J.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85199593781&doi=10.3233%2fSHTI240272&partnerID=40&md5=90c7bd43ba978f942723e6cf1983ffb3},
doi = {10.3233/SHTI240272},
isbn = {09269630 (ISSN); 978-164368527-4 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Stud. Health Technol. Informatics},
volume = {315},
pages = {671–672},
publisher = {IOS Press BV},
abstract = {This project introduces an innovative virtual reality (VR) training program for student Nurse Practitioners, incorporating advanced 3D modeling, animation, and Large Language Models (LLMs). Designed to simulate realistic patient interactions, the program aims to improve communication, history taking, and clinical decision-making skills in a controlled, authentic setting. This abstract outlines the methods, results, and potential impact of this cutting-edge educational tool on nursing education. © 2024 The Authors.},
keywords = {3D modeling, 3D models, 3d-modeling, adult, anamnesis, clinical decision making, clinical education, Clinical Simulation, Computational Linguistics, computer interface, Computer-Assisted Instruction, conference paper, Curriculum, Decision making, E-Learning, Education, Health care education, Healthcare Education, human, Humans, Language Model, Large language model, large language models, Mesh generation, Model animations, Modeling languages, nurse practitioner, Nurse Practitioners, Nursing, nursing education, nursing student, OSCE preparation, procedures, simulation, Teaching, therapy, Training, Training program, User-Computer Interface, Virtual Reality, Virtual reality training},
pubstate = {published},
tppubtype = {inproceedings}
}
This project introduces an innovative virtual reality (VR) training program for student Nurse Practitioners, incorporating advanced 3D modeling, animation, and Large Language Models (LLMs). Designed to simulate realistic patient interactions, the program aims to improve communication, history taking, and clinical decision-making skills in a controlled, authentic setting. This abstract outlines the methods, results, and potential impact of this cutting-edge educational tool on nursing education. © 2024 The Authors.
Michael, Z.; Gemeinhardt, J.; Moritz, K.
Interactive WebXR Hypertext Storytelling for Cultural Heritage Proceedings Article
In: C., Atzenbeck; J., Rubart (Ed.): Proc. Workshop Hum. Factors Hypertext, Hum. - Assoc. ACM Conf. Hypertext Soc. Media ,HT, Association for Computing Machinery, Inc, 2024, ISBN: 979-840071120-6 (ISBN).
Abstract | Links | BibTeX | Tags: 2D textures, 3D modeling, 3D models, 3d-modeling, Cultural heritage, Cultural heritages, Extended reality (XR), Generative AI, History, HTTP, Hypertext, Hypertext systems, Immersive, Machine-learning, Open source software, Open systems, Scene structure, Three dimensional computer graphics, Virtual environments, Virtual Reality, Web browsers
@inproceedings{michael_interactive_2024,
title = {Interactive WebXR Hypertext Storytelling for Cultural Heritage},
author = {Z. Michael and J. Gemeinhardt and K. Moritz},
editor = {Atzenbeck C. and Rubart J.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85211623904&doi=10.1145%2f3679058.3688635&partnerID=40&md5=60aad5a9a95e52c3fff51ebb6f670bd6},
doi = {10.1145/3679058.3688635},
isbn = {979-840071120-6 (ISBN)},
year = {2024},
date = {2024-01-01},
booktitle = {Proc. Workshop Hum. Factors Hypertext, Hum. - Assoc. ACM Conf. Hypertext Soc. Media ,HT},
publisher = {Association for Computing Machinery, Inc},
abstract = {We are presenting our approach for interactive cultural heritage storytelling in WebXR. Therefore, we are describing our scenes’ structure consisting of (stylized) photospheres of the historic locations, 3D models of 3D-scanned historic artifacts and animated 2D textures of historic characters generated with a machine learning toolset. The result is a platform-independent web-application in an immersive interactive WebXR environment running in browsers on PCs, tablets, phones and XR headsets thanks to the underlying software based on the open-source framework A-Frame. Our paper describes the process, the results and the limitations in detail. The resulting application, designed for the Fichtelgebirge region in Upper Franconia, Germany, offers users an immersive digital time travel experience in the virtual space and within a museum setting connecting real artifacts and virtual stories. © 2024 Copyright held by the owner/author(s).},
keywords = {2D textures, 3D modeling, 3D models, 3d-modeling, Cultural heritage, Cultural heritages, Extended reality (XR), Generative AI, History, HTTP, Hypertext, Hypertext systems, Immersive, Machine-learning, Open source software, Open systems, Scene structure, Three dimensional computer graphics, Virtual environments, Virtual Reality, Web browsers},
pubstate = {published},
tppubtype = {inproceedings}
}
We are presenting our approach for interactive cultural heritage storytelling in WebXR. Therefore, we are describing our scenes’ structure consisting of (stylized) photospheres of the historic locations, 3D models of 3D-scanned historic artifacts and animated 2D textures of historic characters generated with a machine learning toolset. The result is a platform-independent web-application in an immersive interactive WebXR environment running in browsers on PCs, tablets, phones and XR headsets thanks to the underlying software based on the open-source framework A-Frame. Our paper describes the process, the results and the limitations in detail. The resulting application, designed for the Fichtelgebirge region in Upper Franconia, Germany, offers users an immersive digital time travel experience in the virtual space and within a museum setting connecting real artifacts and virtual stories. © 2024 Copyright held by the owner/author(s).
2023
Wang, Z.; Joshi, A.; Zhang, G.; Ren, W.; Jia, F.; Sun, X.
Elevating Perception: Unified Recognition Framework and Vision-Language Pre-Training Using Three-Dimensional Image Reconstruction Proceedings Article
In: Proc. - Int. Conf. Artif. Intell., Human-Comput. Interact. Robot., AIHCIR, pp. 592–596, Institute of Electrical and Electronics Engineers Inc., 2023, ISBN: 979-835036036-3 (ISBN).
Abstract | Links | BibTeX | Tags: 3D Model LLM, 3D modeling, 3D models, 3D Tech, 3d-modeling, Augmented Reality, Character recognition, Component, Computer aided design, Computer vision, Continuous time systems, Data handling, Generative AI, Image enhancement, Image Reconstruction, Image to Text Generation, Medical Imaging, Pattern recognition, Pre-training, Reconstructive Training, Text generations, Three dimensional computer graphics, Virtual Reality
@inproceedings{wang_elevating_2023,
title = {Elevating Perception: Unified Recognition Framework and Vision-Language Pre-Training Using Three-Dimensional Image Reconstruction},
author = {Z. Wang and A. Joshi and G. Zhang and W. Ren and F. Jia and X. Sun},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85192837757&doi=10.1109%2fAIHCIR61661.2023.00105&partnerID=40&md5=0fe17cc622a9aa90e88b8c3e6a3bed3b},
doi = {10.1109/AIHCIR61661.2023.00105},
isbn = {979-835036036-3 (ISBN)},
year = {2023},
date = {2023-01-01},
booktitle = {Proc. - Int. Conf. Artif. Intell., Human-Comput. Interact. Robot., AIHCIR},
pages = {592–596},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {This research project explores a paradigm shift in perceptual enhancement by integrating a Unified Recognition Framework and Vision-Language Pre-Training in three-dimensional image reconstruction. Through the synergy of advanced algorithms from computer vision & language processing, the project tries to enhance the precision and depth of perception in reconstructed images. This innovative approach holds the potential to revolutionize fields such as medical imaging, virtual reality, and computer-aided design, providing a comprehensive perspective on the intersection of multimodal data processing and perceptual advancement. The anticipated research outcomes are expected to significantly contribute to the evolution of technologies that rely on accurate and contextually rich three-dimensional reconstructions. Moreover, the research aims to reduce the constant need for new datasets by improving pattern recognition through 3D image patterning on backpropagation. This continuous improvement of vectors is envisioned to enhance the efficiency and accuracy of pattern recognition, contributing to the optimization of perceptual systems over time. © 2023 IEEE.},
keywords = {3D Model LLM, 3D modeling, 3D models, 3D Tech, 3d-modeling, Augmented Reality, Character recognition, Component, Computer aided design, Computer vision, Continuous time systems, Data handling, Generative AI, Image enhancement, Image Reconstruction, Image to Text Generation, Medical Imaging, Pattern recognition, Pre-training, Reconstructive Training, Text generations, Three dimensional computer graphics, Virtual Reality},
pubstate = {published},
tppubtype = {inproceedings}
}
This research project explores a paradigm shift in perceptual enhancement by integrating a Unified Recognition Framework and Vision-Language Pre-Training in three-dimensional image reconstruction. Through the synergy of advanced algorithms from computer vision & language processing, the project tries to enhance the precision and depth of perception in reconstructed images. This innovative approach holds the potential to revolutionize fields such as medical imaging, virtual reality, and computer-aided design, providing a comprehensive perspective on the intersection of multimodal data processing and perceptual advancement. The anticipated research outcomes are expected to significantly contribute to the evolution of technologies that rely on accurate and contextually rich three-dimensional reconstructions. Moreover, the research aims to reduce the constant need for new datasets by improving pattern recognition through 3D image patterning on backpropagation. This continuous improvement of vectors is envisioned to enhance the efficiency and accuracy of pattern recognition, contributing to the optimization of perceptual systems over time. © 2023 IEEE.
Lee, S.; Lee, H.; Lee, K.
Knowledge Generation Pipeline using LLM for Building 3D Object Knowledge Base Proceedings Article
In: Int. Conf. ICT Convergence, pp. 1303–1305, IEEE Computer Society, 2023, ISBN: 21621233 (ISSN); 979-835031327-7 (ISBN).
Abstract | Links | BibTeX | Tags: 3D modeling, 3D models, 3D object, 3d-modeling, Augmented Reality, Data Mining, Knowledge Base, Knowledge based systems, Knowledge generations, Language Model, Metaverse, Metaverses, Multi-modal, MultiModal AI, Multimodal artificial intelligence, Pipelines, Virtual Reality, XR
@inproceedings{lee_knowledge_2023,
title = {Knowledge Generation Pipeline using LLM for Building 3D Object Knowledge Base},
author = {S. Lee and H. Lee and K. Lee},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85184593202&doi=10.1109%2fICTC58733.2023.10392933&partnerID=40&md5=b877638607a04e5a31a2d5723af6e11b},
doi = {10.1109/ICTC58733.2023.10392933},
isbn = {21621233 (ISSN); 979-835031327-7 (ISBN)},
year = {2023},
date = {2023-01-01},
booktitle = {Int. Conf. ICT Convergence},
pages = {1303–1305},
publisher = {IEEE Computer Society},
abstract = {With the wide spread of XR(eXtended Reality) contents such as Metaverse and VR(Virtual Reality) / AR(Augmented Reality), the utilization and importance of 3D objects are increasing. In this paper, we describe a knowledge generation pipeline of 3D object for reuse of existing 3D objects and production of new 3D object using generative AI(Artificial Intelligence). 3D object knowledge includes not only the object itself data that are generated in object editing phase but the information for human to recognize and understand objects. The target 3D model for building knowledge is the space model of office for business Metaverse service and the model of objects composing the space. LLM(Large Language Model)-based multimodal AI was used to extract knowledge from 3D model in a systematic and automated way. We plan to expand the pipeline to utilize knowledge base for managing extracted knowledge and correcting errors occurred during the LLM process for the knowledge extraction. © 2023 IEEE.},
keywords = {3D modeling, 3D models, 3D object, 3d-modeling, Augmented Reality, Data Mining, Knowledge Base, Knowledge based systems, Knowledge generations, Language Model, Metaverse, Metaverses, Multi-modal, MultiModal AI, Multimodal artificial intelligence, Pipelines, Virtual Reality, XR},
pubstate = {published},
tppubtype = {inproceedings}
}
With the wide spread of XR(eXtended Reality) contents such as Metaverse and VR(Virtual Reality) / AR(Augmented Reality), the utilization and importance of 3D objects are increasing. In this paper, we describe a knowledge generation pipeline of 3D object for reuse of existing 3D objects and production of new 3D object using generative AI(Artificial Intelligence). 3D object knowledge includes not only the object itself data that are generated in object editing phase but the information for human to recognize and understand objects. The target 3D model for building knowledge is the space model of office for business Metaverse service and the model of objects composing the space. LLM(Large Language Model)-based multimodal AI was used to extract knowledge from 3D model in a systematic and automated way. We plan to expand the pipeline to utilize knowledge base for managing extracted knowledge and correcting errors occurred during the LLM process for the knowledge extraction. © 2023 IEEE.
Yeo, J. Q.; Wang, Y.; Tanary, S.; Cheng, J.; Lau, M.; Ng, A. B.; Guan, F.
AICRID: AI-Empowered CR For Interior Design Proceedings Article
In: G., Bruder; A.H., Olivier; A., Cunningham; E.Y., Peng; J., Grubert; I., Williams (Ed.): Proc. - IEEE Int. Symp. Mixed Augment. Real. Adjunct, ISMAR-Adjunct, pp. 837–841, Institute of Electrical and Electronics Engineers Inc., 2023, ISBN: 979-835032891-2 (ISBN).
Abstract | Links | BibTeX | Tags: 3D modeling, 3D models, 3d-modeling, Architectural design, Artificial intelligence, Artificial intelligence technologies, Augmented Reality, Augmented reality technology, Interior Design, Interior designs, machine learning, Machine-learning, Model generation, Novel design, Text images, User need, Visualization
@inproceedings{yeo_aicrid_2023,
title = {AICRID: AI-Empowered CR For Interior Design},
author = {J. Q. Yeo and Y. Wang and S. Tanary and J. Cheng and M. Lau and A. B. Ng and F. Guan},
editor = {Bruder G. and Olivier A.H. and Cunningham A. and Peng E.Y. and Grubert J. and Williams I.},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85180375829&doi=10.1109%2fISMAR-Adjunct60411.2023.00184&partnerID=40&md5=b14d89dbd38a4dfe3f85b90800d42e78},
doi = {10.1109/ISMAR-Adjunct60411.2023.00184},
isbn = {979-835032891-2 (ISBN)},
year = {2023},
date = {2023-01-01},
booktitle = {Proc. - IEEE Int. Symp. Mixed Augment. Real. Adjunct, ISMAR-Adjunct},
pages = {837–841},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {Augmented Reality (AR) technologies have been utilized for interior design for years. Normally 3D furniture models need to be created manually or by scanning with specialized devices and this is usually a costly process. Additionally, users need controllers or hands for manipulating the virtual furniture which may lead to fatigue for long-time usage. Artificial Intelligence (AI) technologies have made it possible to generate 3D models from texts, images or both and show potential to automate interactions through the user's voice. We propose a novel design, AICRID in short, which aims to automate the 3D model generation and to facilitate the interactions for interior design AR by leveraging on AI technologies. Specifically, our design will allow the users to directly generate 3D furniture models with generative AI, enabling them to directly interact with the virtual objects through their voices. © 2023 IEEE.},
keywords = {3D modeling, 3D models, 3d-modeling, Architectural design, Artificial intelligence, Artificial intelligence technologies, Augmented Reality, Augmented reality technology, Interior Design, Interior designs, machine learning, Machine-learning, Model generation, Novel design, Text images, User need, Visualization},
pubstate = {published},
tppubtype = {inproceedings}
}
Augmented Reality (AR) technologies have been utilized for interior design for years. Normally 3D furniture models need to be created manually or by scanning with specialized devices and this is usually a costly process. Additionally, users need controllers or hands for manipulating the virtual furniture which may lead to fatigue for long-time usage. Artificial Intelligence (AI) technologies have made it possible to generate 3D models from texts, images or both and show potential to automate interactions through the user's voice. We propose a novel design, AICRID in short, which aims to automate the 3D model generation and to facilitate the interactions for interior design AR by leveraging on AI technologies. Specifically, our design will allow the users to directly generate 3D furniture models with generative AI, enabling them to directly interact with the virtual objects through their voices. © 2023 IEEE.
2022
Wong, S. M.; Chen, C. -W.; Pan, T. -Y.; Chu, H. -K.; Hu, M. -C.
GetWild: A VR Editing System with AI-Generated 3D Object and Terrain Proceedings Article
In: MM - Proc. ACM Int. Conf. Multimed., pp. 6988–6990, Association for Computing Machinery, Inc, 2022, ISBN: 978-145039203-7 (ISBN).
Abstract | Links | BibTeX | Tags: 3-D environments, 3-d terrains, 3D modeling, 3D models, 3D object, 3d-modeling, Editing systems, Landforms, Modeling softwares, Object generation, terrain generation, Terrain generations, Virtual Reality, Vr editing
@inproceedings{wong_getwild_2022,
title = {GetWild: A VR Editing System with AI-Generated 3D Object and Terrain},
author = {S. M. Wong and C. -W. Chen and T. -Y. Pan and H. -K. Chu and M. -C. Hu},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85151159899&doi=10.1145%2f3503161.3547733&partnerID=40&md5=668c107b586a77f7ef9bfde37d4dfb9f},
doi = {10.1145/3503161.3547733},
isbn = {978-145039203-7 (ISBN)},
year = {2022},
date = {2022-01-01},
booktitle = {MM - Proc. ACM Int. Conf. Multimed.},
pages = {6988–6990},
publisher = {Association for Computing Machinery, Inc},
abstract = {3D environment artists typically use 2D screens and 3D modeling software to achieve their creation. However, creating 3D content using 2D tools is counterintuitive. Moreover, the process would be inefficient for junior artists in the absence of a reference. We develop a system called GetWild, which employs artificial intelligence (AI) models to generate the prototype of 3D objects/terrain and allows users to further edit the generated content in the virtual space. With the aid of AI, the user can capture an image to obtain a rough 3D object model, or start with drawing simple sketches representing the river, the mountain peak and the mountain ridge to create a 3D terrain prototype. Further, the virtual reality (VR) technique is used to provide an immersive design environment and intuitive interaction (such as painting, sculpturing, coloring, and transformation) for users to edit the generated prototypes. Compared with the existing 3D modeling software and systems, the proposed VR editing system with AI-generated 3D objects/terrain provides a more efficient way for the user to create virtual artwork. © 2022 Owner/Author.},
keywords = {3-D environments, 3-d terrains, 3D modeling, 3D models, 3D object, 3d-modeling, Editing systems, Landforms, Modeling softwares, Object generation, terrain generation, Terrain generations, Virtual Reality, Vr editing},
pubstate = {published},
tppubtype = {inproceedings}
}
3D environment artists typically use 2D screens and 3D modeling software to achieve their creation. However, creating 3D content using 2D tools is counterintuitive. Moreover, the process would be inefficient for junior artists in the absence of a reference. We develop a system called GetWild, which employs artificial intelligence (AI) models to generate the prototype of 3D objects/terrain and allows users to further edit the generated content in the virtual space. With the aid of AI, the user can capture an image to obtain a rough 3D object model, or start with drawing simple sketches representing the river, the mountain peak and the mountain ridge to create a 3D terrain prototype. Further, the virtual reality (VR) technique is used to provide an immersive design environment and intuitive interaction (such as painting, sculpturing, coloring, and transformation) for users to edit the generated prototypes. Compared with the existing 3D modeling software and systems, the proposed VR editing system with AI-generated 3D objects/terrain provides a more efficient way for the user to create virtual artwork. © 2022 Owner/Author.