MMIS-2024@ACM MM 2024: Multi-rater Medical Image Segmentation for Radiotherapy Planning in Nasopharyngeal Carcinoma and Glioblastoma
News
[2024.06.18] We release the training set of task-1. See Section 3.4:[2024.06.17] We open the challenge website. Please check the timeline.
Overview
Image segmentation in the medical domain presents inherent subjectivity due to data-level ambiguities and diverse expert preferences. These factors lead to varying interpretations of the same target, limiting the deployment of AI models in clinical settings. This challenge invites you to address the multi-rater problem by focusing on the segmentation of diverse and personalized Gross Tumor Volumes (GTV) from MRI data, which is critical for radiotherapy planning in Nasopharyngeal Carcinoma and Glioblastoma.
Diverse segmentation offers a comprehensive view of possible interpretations, capturing a full range of expert opinions and enhancing the robustness of AI models in clinical decision-making. This approach ensures that critical variations in interpretation are not overlooked, which is particularly valuable in complex cases. Personalized segmentation, on the other hand, tailors the results to align with the specific preferences and methodologies of individual clinicians, resulting in more relevant treatment plans and reducing clinician workload. In clinical applications, these methods support improved radiotherapy planning by accommodating both the variability in expert assessments and the need for individualized treatment strategies.
Tasks
The challenge comprises two tasks, each with different training settings:
Task 1: Multi-Rater Medical Image Segmentation for Nasopharyngeal Carcinoma
(1) Each sample in this task is annotated by four different experts.(2) Participants are encouraged to utilize all annotations for model training.
Task 2: Federated Multi-Rater Medical Image Segmentation for Glioblastoma
(1) Each sample is annotated by a single expert.(2) Participants are encouraged to explore models by using only one expert annotation per sample.
Objectives
This challenge aims to enhance radiotherapy planning for Nasopharyngeal Carcinoma and Glioblastoma by developing AI models that can handle the complexity and variability of expert interpretations, ultimately leading to more personalized and effective treatment strategies.Dataset
Task 1: Three MRI sequences (T1, T2, and T1-Contrast) are given and different sequences of one sample are rigid-registered to a common subject space. For each sample, four senior radiologists from different locations (with experiences of around 5-10 years) individually annotated the GTV of Nasopharyngeal Carcinoma. We divide the 170 subjects into separate training, validation, and testing sets (i.e., 100, 20, and 50 subjects, respectively)
Task 2: We will combine several public glioblastoma datasets and further enhance them by providing more expert annotations. Data and details will be released soon.
To access the datasets, please do the registration first.
Task 2: We will combine several public glioblastoma datasets and further enhance them by providing more expert annotations. Data and details will be released soon.
To access the datasets, please do the registration first.
Agenda
| Event | Timeline | ||||
|---|---|---|---|---|---|
| Release of task-1 training/validation data | 06/18/2024 | ||||
| Release of task-2 training/validation data | 07/01/2024 | ||||
| Submission for task-1 evaluation | 07/18/2024 | ||||
| Submission for task-2 evaluation | 08/01/2024 | ||||
| Announcement of invited papers | 09/01/2024 | ||||
| Announcement of final leaderboards | 09/01/2024 | ||||
Prizes
1. Cash awards for each task ($1800 USD in total):First Prize: $500
Second Prize: $300
Third Prize: $100
2. Certificates for the top-rank teams.
3. ACM MM 2024 will open a call for challenge papers from the top 3 attending teams. Thus, for each challenge, there will be three papers (up to) invited from each team and one summary paper invited from the challenge organizers.
4. Besides, there will be a challenging paper, which will be submitted to a top-tier journal (e.g., MedIA/TMI). Top rankers will be invited as co-authors of the challenging paper.
Evaluation
For evaluation, we follow Wu_2024_CVPR, Luo_2023_deep to do the evaluation for multi-rater medical image segmentation. In general, we evaluate the segmentation results from two aspects:1. Diverse performance: we use the GED score, Dicesoft score, and two set-level metrics DiceMatch and DiceMax to compare the similarity between the results set and label set.
2. Personalized performance: we compute the Dice score for each individual expert and then use their mean value as the final metric.
For each task, we will use a combination score of these metrics for the final ranking.
Organizations
Organizer Team
•Yicheng Wu, Monash University, Australia
•Yutong Xie, Australian Institute for Machine Learning (AIML), University of Adelaide, Australia
•Xiangde Luo, Sichuan Cancer Hospital & Institute, Sichuan Cancer Center, China.
Clinician Team
•Wenjun Liao, Sichuan Cancer Hospital & Institute, Sichuan Cancer Center, China.
•Minh-Son To, Flinders Health and Medical Research Institute, Flinders University, Australia
•Peter Gorayski, Australian Bragg Centre for Proton Therapy, South Australia Health and Medical Research Institute, Australia
•Hien Le, Australian Bragg Centre for Proton Therapy, South Australia Health and Medical Research Institute, Australia
Advisor Team
•Jianfei Cai, Monash University, Australia
•Qi Wu, Australian Institute for Machine Learning (AIML), University of Adelaide, Australia
•Weidong Cai, University of Sydney, Australia
•Zhaolin Chen, Monash University, Australia
•Yong Xia, Northwestern Polytechnical University, China
Contributors
•Siqi Chen, Australian Institute for Machine Learning (AIML), University of Adelaide, Australia•Zihao Tang, University of Sydney, Australia
Organized By
Citations
1. Luo, X., Liao, W., He, Y., Tang, F., Wu, M., Shen, Y., Huang, H., Song, T., Li, K., Zhang, S., et al.: Deep learning-based accurate delineation of primary gross tumor volume of nasopharyngeal carcinoma on heterogeneous magnetic resonance imaging: A large-scale and multi-center study. Radiotherapy and Oncology 180, 109480 (2023)
2. Wu, Y., Luo, X., Xu, Z., Guo, X., Ju, L., Ge, Z., Liao, W., Cai, J.: Diversified and personalized multi-rater medical image segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11470–11479 (June 2024)
2. Wu, Y., Luo, X., Xu, Z., Guo, X., Ju, L., Ge, Z., Liao, W., Cai, J.: Diversified and personalized multi-rater medical image segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11470–11479 (June 2024)
Terms and Conditions
1. We only consider automatic segmentation methods for the final ranking.
2. No additional data is allowed. We will require all participants to provide extensive documentation on their development processes, including data and methods used. Top-ranked participants must submit their training codes and checkpoints for verification. Publicly available pre-trained models are allowed, ensuring a level playing field and transparency in the competition's evaluation process.
3. The top-ranking teams will be notified two weeks before the challenge day to prepare their presentations. Final results and awards will be announced on the challenge day.
4. Independent Publications: Participating teams are allowed to publish their results separately. However, this should be done in a manner that respects the collective efforts of the challenge.
5. Embargo Period: An embargo period allows the challenge organizers to publish a comprehensive challenge paper first. During this period, individual teams are encouraged to refrain from publishing their complete findings independently.
2. No additional data is allowed. We will require all participants to provide extensive documentation on their development processes, including data and methods used. Top-ranked participants must submit their training codes and checkpoints for verification. Publicly available pre-trained models are allowed, ensuring a level playing field and transparency in the competition's evaluation process.
3. The top-ranking teams will be notified two weeks before the challenge day to prepare their presentations. Final results and awards will be announced on the challenge day.
4. Independent Publications: Participating teams are allowed to publish their results separately. However, this should be done in a manner that respects the collective efforts of the challenge.
5. Embargo Period: An embargo period allows the challenge organizers to publish a comprehensive challenge paper first. During this period, individual teams are encouraged to refrain from publishing their complete findings independently.