Recently, Assist Prof. Liu Zuozhu and his team at the Zhejiang University-University of Illinois Urbana-Champaign Institute (ZJUI), published a research paper titled "Deep learning enabled 3D multimodal fusion of cone beam CT and internal mesh scans for clinically applicable tooth bone reconstruction" in the Journal Patterns of Cell Press. The first author of this paper is Liu Jiaxiang, a doctoral student at ZJUI. The corresponding author is Assist Prof. Liu Zuozhu at ZJUI. Dr. Hao Jin from Harvard University is the co-first author, and Prof. Zhao Zhihe from the West China School/ Hospital of Stomatology Sichuan University is the co-corresponding author. Other authors include Lin Hangzheng, a 22’ undergraduate of ZJUI, and Assist Prof. Wang Gaoang at ZJUI.
This paper proposes a framework called Deep Dental Multimodal Fusion (DDMF), which can automatically fuse three-dimensional oral images such as CBCT and IOS, achieving clinically applicable three-dimensional reconstruction of tooth crowns, roots, and jawbones. This innovative achievement will provide strong and intelligent key technology support for the diagnosis and treatment in digital oral field.
Research Highlights
1. Targeted at the needs of clinical 3D dental crown-root-bone reconstruction, the multimodal fusion tasks of cone beam CT (CBCT) and intraoral scanning (IOS) have been determined.
2. Developed an end-to-end CBCT and IOS multimodal fusion framework based on deep learning.
3. Designed a new algorithm for precise single mode segmentation and multimodal fusion of CBCT and IOS
4. Established a large-scale multimodal dataset and verified its performance superiority through experiments and clinical testing.
Introduction to the paper
3D CBCT and IOS images can respectively present complex oral structures and accurate information on tooth crowns and gums, playing a crucial role in digital oral diagnosis and treatment. In clinical diagnosis and treatment, high-precision three-dimensional oral crown-root-bone models can be constructed by fusing CBCT and IOS data, thereby achieving efficient and accurate oral disease diagnosis, treatment plan design, and prognostic efficacy evaluation. However, the complexity and diversity of oral anatomical features among different patients, the modal differences between 3D CBCT images and IOS point clouds, and the lack of large-scale practical datasets and clinical experimental validation have brought many challenges to achieving clinically applicable 3D CBCT and IOS multimodal fusion and 3D corona-root bone reconstruction.
Figure 1 Framework of DDMF
In order to overcome the above challenges, this paper proposes a multimodal fusion framework for oral 3D images called DDMF, which utilizes deep learning technology to accurately segment CBCT and IOS data before performing 3D fusion, achieving high-precision dental crown-root-bone reconstruction. The DDMF framework includes a CBCT semantic segmentation module based on neural networks (TSTNet), an IOS semantic segmentation module (IOSNet), and a three-dimensional multimodal fusion module. This method significantly improves the performance of CBCT/iOS semantic segmentation and multimodal fusion by introducing data augmentation mechanism driven by prior knowledge, designing a new loss function and network architecture, and a 3D image point cloud fusion algorithm based on point of curvature and geometric spatial features.
Figure 2 Visualization Results of CBCT Semantic Segmentation
The research team constructed a large-scale multimodal dataset containing 503 CBCT and 28559 IOS data, which achieved excellent performance in numerous experiments and clinical applicability tests and reduced the multimodal oral crown-root-bone reconstruction process from traditional 5-7 hours (labor cost) to 20-25 minutes.
Figure 3 DDMF Crown-Root-Bone 3D Reconstruction Process
This study collaborated with Prof. Zhao Zhihe's research team from the West China School/ Hospital of Stomatology Sichuan University, as well as collaboration teams from Harvard University, Air Force Medical University, Angelalign, Nanyang Technological University, Shenzhen Second People's Hospital, OPT, and other corporations and institutions. The research team from Zhejiang University/West China School collaborated with Angelalign, a leading domestic oral medical device enterprise, with the aim of achieving a university-hospital-enterprise trinity of industry-university-research cooperation, provide orthodontists with a precise dental crown-root-bone structure, assist in the simulation design of diagnosis and treatment plans, and reduce various risks such as bone fenestration and dehiscence during traditional malocclusion diagnosis and treatment. In clinical practice, DDMF will be integrated into the Intelligent Root System (IRS) platform of Angelalign, to further improve diagnosis and treatment accuracy and efficiency, reduce treatment costs, provide intelligent and efficient auxiliary diagnosis and treatment methods in the digital oral field, and assist in innovative research and application development in the intersection of computer and oral medicine.
Figure 4 Clinical Risk Analysis of Orthodontic Diagnosis & Treatment based on DDMF
The work was supported by the National Natural Science Foundation of China, Zhejiang Provincial Natural Science Foundation, ZJU- Angelalign Research & Development Center for Intelligent Healthcare.
Author introduction
Liu Jiaxiang
Doctoral student
Liu Jiaxiang, a doctoral student jointly cultivated doctoral students jointly cultivated by ZJUI and the College of Computer Science and Technology, Zhejiang University in the field of artificial intelligence. His supervisor is Assist Prof. Liu Zuozhu, and his research direction is medical artificial intelligence and multimodal data processing. Currently, he has published multiple papers in journals/conferences such as Patterns, IEEE TCSVT/TMM/TETCI/ISBI, etc. and received the 2020 National Postgraduate Scholarship and the 2022/2023 Jiangsu Provincial Jicui Postgraduate Scholarship.
Dr. Hao Jin
Dr. Hao Jin graduated from Harvard University in 2022 with a doctoral degree. Currently he served as a postdoctoral researcher at the Wu Tsai Neurosciences Institute at Stanford University, and his research interests include neuroscience and digital medicine based on deep learning. Also, he served as the editorial board member of Frontiers in Neuroscience magazine and published multiple papers in flagship journals such as Patterns, Nature Communications, IEEE Trans, JDR, etc.
Dr. Feng Yang
Dr. Feng Yang obtained a doctoral degree in medical image processing from the University of Edinburgh in 2015. He is currently working at Angelalign as the head of software research and development and the discipline leader of Angelalign Research Institute. Dr. Feng Yang mainly engaged in research fields of point cloud and image processing, orthodontic analysis, and CAD automation in diagnosis and treatment. He published multiple papers in flagship journals and top-level conferences such as Patterns, IEEE TMI, ICLR, and MICCAI.
Prof. Jin Zuolin
Prof. Jin Zuolin is the Director, Professor, Chief Physician, and Doctoral Supervisor of the Orthodontics Department of the School of Stomatology at the Air Force Medical University. He is a Postdoctoral fellow in Orthodontics at Sichuan University and visiting scholar at Columbia University in the United States. He mainly engaged in periodontal tissue reconstruction and orthodontics, craniofacial malocclusion and growth development, and biomechanical research on invisible orthodontics. Also, he is responsible for 4 projects funded by the National Natural Science Foundation of China, 15 science and technology projects in Shanxi Province, published 146 papers, with 45 papers included in SCI, edited 2 works, and translated 3 works. He received the second prize for military medical achievements and the second prize for scientific and technological progress in Shanxi Province.
Prof. Zhao Zhihe
Prof. Zhao Zhihe graduated with a doctoral degree in Orthodontics from the School of Stomatology at West China University of Medical Sciences in 1992. He is currently a Professor and the Director of the Professor Committee at the West China School/ Hospital of Stomatology Sichuan University, a young and middle-aged expert with significant contributions to the National Health and Family Planning Commission, an academic and technical leader in Sichuan Province, and a Fellow at the International College of Dentists (ICD). Besides, he is the vice editor in chief of the Chinese Journal of Orthodontics, editorial board member of core journals such as the Chinese Journal of Stomatology and the West China Journal of Stomatology, and editorial board member of the International Journal of Oral Sciences indexed by SCI.
Assist Prof. Liu Zuozhu
Assist Prof. Liu Zuozhu is an Assistant Professor and PhD supervisor at ZJUI, and an Adjunct Professor at the Stomatology Hospital, Zhejiang University School of Medicine (Zhejiang Stomatology Hospital). He hosted multiple projects such as projects funded by the National Natural Science Foundation, Zhejiang Provincial Key Projects, and Zhejiang University Major Industrial Projects. And his research interests mainly include trustworthy artificial intelligence in medicine. Assist Prof. Liu published multiple papers in flagship journals and top-level conferences such as Patterns, IEEE Trans, ICLR, ACL, JDR, etc.
Article title：Deep learning-enabled 3D multimodal fusion of cone-beam CT and intraoral mesh scans for clinically applicable tooth-bone reconstruction
Article link：https://www.cell.com/patterns/fulltext/S2666-3899(23)00194-0
DOI：https://doi.org/10.1016/j.patter.2023.100825
论文网址：
https://www.cell.com/patterns/fulltext/S2666-3899(23)00194-0
DOI：
https://doi.org/10.1016/j.patter.2023.100825
