An Approach to Segment Nuclei and Cytoplasm in Lung Cancer Brightfield Images Using Hybrid Swin-Unet Transformer

Purpose: Segmentation of nuclei and cytoplasm in cellular images is essential for estimating the prognosis of lung cancer disease. The detection of these organelles in the unstained brightfield microscopic images is challenging due to poor contrast and lack of separation of structures with irregular morphology. This work aims to carry out semantic segmentation of nuclei and cytoplasm in lung cancer brightfield images using the Swin-Unet Transformer. Methods: For this study, publicly available brightfield images of lung cancer cells are pre-processed and fed to the Swin-Unet for semantic segmentation. Model specific hyperparameters are identified after detailed analysis and the segmentation performance is validated using standard evaluation metrics. Results: The hyperparameter analysis provides the selection of optimum parameters as focal loss, learning rate of 0.0001, Adam optimizer, and Swin Transformer patch size of 4. The obtained results show that with these parameters, the Swin-Unet Transformer accurately segmented the nuclei and cytoplasm in the brightfield images with pixel-F1 scores of 90.71% and 79.29% respectively. Conclusion: It is observed that the model could identify nuclei and cytoplasm with varied morphologies. The detection of cytoplasm with weak and subtle edge details indicates the effectiveness of shifted window based self attention mechanism of Swin-Unet in capturing the global and long distance pixel interactions in the brightfield images. Thus, the adopted methodology in this study can be employed for the precise segmentation of nuclei and cytoplasm for assessing the malignancy of lung cancer disease.