Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Nov 2;5(1):1162.
doi: 10.1038/s42003-022-04117-x.

Self-supervised machine learning for live cell imagery segmentation

Michael C Robitaille  1 , Jeff M Byers  1 , Joseph A Christodoulides  1 , Marc P Raphael  2
Affiliations

Self-supervised machine learning for live cell imagery segmentation

Michael C Robitaille et al. Commun Biol. .

Abstract

Segmenting single cells is a necessary process for extracting quantitative data from biological microscopy imagery. The past decade has seen the advent of machine learning (ML) methods to aid in this process, the overwhelming majority of which fall under supervised learning (SL) which requires vast libraries of pre-processed, human-annotated labels to train the ML algorithms. Such SL pre-processing is labor intensive, can introduce bias, varies between end-users, and has yet to be shown capable of robust models to be effectively utilized throughout the greater cell biology community. Here, to address this pre-processing problem, we offer a self-supervised learning (SSL) approach that utilizes cellular motion between consecutive images to self-train a ML classifier, enabling cell and background segmentation without the need for adjustable parameters or curated imagery. By leveraging motion, we achieve accurate segmentation that trains itself directly on end-user data, is independent of optical modality, outperforms contemporary SL methods, and does so in a completely automated fashion-thus eliminating end-user variability and bias. To the best of our knowledge, this SSL algorithm represents a first of its kind effort and has appealing features that make it an ideal segmentation tool candidate for the broader cell biology research community.

PubMed Disclaimer

Conflict of interest statement

The authors declare no competing interests.

Figures

Fig. 1
Fig. 1. Overview of the Farneback displacement (FD) self-labeling strategy.
a The vast majority of cell segmentation techniques utilize single image frames and the static information contained within as means to distinguish ‘cell’ from ‘background’, oftentimes represented in a histogram. The self-supervised algorithm utilizes optical flow as a means to self-label pixels in an automated fashion. b Due to the prevalence of intracellular dynamics in time-lapse live cell imagery, FD can be calculated for each pair of consecutive images t−1,t. The FD can then be represented as vectors associated with each pixel (right). c The magnitude of the FD then offers a means to distinguish cells from their background, as shown in the bivariate histogram which co-plots the pixel intensity of a single image at t to the FD vector magnitudes calculated between consecutive images t−1,t. Pixels with the highest displacements can be automatically labeled ‘cell’ (left of the green dashed line) and those with the lowest can be labeled ‘background’ (right of the yellow dashed line). Pixels that do not meet either criteria remain unlabeled, while the self-labeled pixels are used to create a training data set for classification. Time increment: 600 s, scale bar = 20 µm.
Fig. 2
Fig. 2. Overview of the automated self-supervised learning algorithm.
a The contrast enhanced DIC image of several and b a single highlighted MDA-MB-231 cell illustrates the range of intensities inherent within the cells. (20X objective). c, d Unsupervised learning via FD: high threshold FD is used to select only those pixels exhibiting the highest displacement magnitudes and labels them as ‘cell’ (green pixels). Similarly, low threshold FD is used to identify pixels with a much wider range of displacement magnitudes than the high flow regime. The lowest displacement magnitude pixels are labeled ‘background’ (yellow pixels). Pixels that exhibit FD in between these regimes remain unlabeled (gray pixels). e, f Supervised learning via self-labeled training data. The self-labeled pixels (green and yellow) are then used to generate static feature vectors, which are in turn used to train the classifier model. g The blue outline is the resulting segmentation which outlines all pixels classified by the FD trained model as ‘cell’ and is also overlaid on the image in b. This process is repeated at every time step, thereby using the most recent imagery to update the training data. Scale bar: 25 µm (20X objective, time increment: 300 s).
Fig. 3
Fig. 3. Self-supervised segmentation for a range of cell types, microscope modalities, time resolutions and magnifications.
a phase contrast of Hs27 fibroblasts (10X objective, time increment: 1200 s) b transmitted light of Dictyostelium (10X objective, time increment: 60 s) c phase contrast of MDA-MB-231 (10X objective, time increment: 600 s) d IRM image of a single Hs27 cell (40X objective, time increment: 600 s). e DIC image of MDA-MB-231 cells (20X objective, time increment: 120 s) f fluorescence image of a single lifeAct (GFP-actin conjugate) transfected A549 cell (pseudo-colored) with the associated FD vector plot (100X objective, time increment: 10 s). Insets i, ii, iii highlight boxed image regions. White arrows point to examples of debris that was correctly labeled ‘background’ due either to lack of motion or automated size filtering. Images have been contrast enhanced to highlight low contrast features and background inhomogeneities. DIC image e was additionally enhanced with a sharpen filter to highlight interference induced shadowing of cell features. Scale bars: a, b, c: 50 µm; d, e: 25 µm; f: 10 µm.
Fig. 4
Fig. 4. Segmentation evaluation of Self-Supervised Machine Learning (SSL) and CellPose on the data sets used for validation in this study via F1-scores.
The top row includes the name of the data set annotated by magnification, optical modality, cell type and brief description of the imagery characteristics. #L stands for the number of annotated labels used for model training, and #O stands for the number of objects to be segmented by the model within a given data set. *CellPose has a single parameter, a size filter, that can be automatically estimated, however, for some of the data sets the best segmentation was found by manually tuning this size filter. The figures below show the ground truth (green-solid lines), SSL (cyan-large dashes), and CellPose (red-small dashes) outlines overlaid on the final image of the data set.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Sommer C, Gerlich DW. Machine learning in cell biology–teaching computers to recognize phenotypes. J. Cell Sci. 2013;126:5529–5539. - PubMed
    1. Deep learning gets scope time. Nat. Methods16, 1195–1195 (2019). - PubMed
    1. Moen E, et al. Deep learning for cellular image analysis. Nat. Methods. 2019;16:1233–1246. doi: 10.1038/s41592-019-0403-1. - DOI - PMC - PubMed
    1. Falk T, et al. U-Net: deep learning for cell counting, detection, and morphometry. Nat. Methods. 2019;16:67–70. doi: 10.1038/s41592-018-0261-2. - DOI - PubMed
    1. Lin, T.-Y. et al. European Conference on Computer Vision (Springer, 2020). - PubMed

Publication types