AI Chat Paper
Discover the SciOpen Platform and Achieve Your Research Goals with Ease.
Biological super-resolution microscopy is a new generation of imaging techniques that overcome the ~200 nm diffraction limit of conventional light microscopy in spatial resolution. By providing novel spatial or spatiotemporal information on biological processes at nanometer resolution with molecular specificity, it plays an increasingly important role in biomedical sciences. However, its technical constraints also require trade-offs to balance its spatial resolution, temporal resolution, and light exposure of samples. Recently, deep learning has achieved breakthrough performance in many image processing and computer vision tasks. It has also shown great promise in pushing the performance envelope of biological super-resolution microscopy. In this brief review, we survey recent advances in using deep learning to enhance the performance of biological super-resolution microscopy, focusing primarily on computational reconstruction of super-resolution images. Related key technical challenges are discussed. Despite the challenges, deep learning is expected to play an important role in the development of biological super-resolution microscopy. We conclude with an outlook into the future of this new research area.
Anwar S, Khan S, Barnes N (2020) A deep journey into super-resolution: a Survey. ACM Comput Surv 53(3): 1−34
Badrinarayanan V, Kendall A, Cipolla R (2017) Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE T Pattern AnalMachIntell 39(12): 2481−2495
Balzarotti F, Eilers Y, Gwosch KC, Gynnå AH, Westphal V, Stefani FD, Elf J, Hell SW (2017) Nanometer resolution imaging and tracking of fluorescent molecules with minimal photon fluxes. Science 355(6325): 606−612
Belthangady C, Royer LA (2019) Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat Methods 16(12): 1215−1225
Betzig E, Patterson GH, Sougrat R, Lindwasser OW, Olenych S, Bonifacino JS, Davidson MW, Lippincott-Schwartz J, Hess HF (2006) Imaging intracellular fluorescent proteins at nanometer resolution. Science 313(5793): 1642−1645
Boulanger J, Kervrann C, Bouthemy P, Elbau P, Sibarita J, Salamero J (2010) Patch-based nonlocal functional for denoising fluorescence microscopy image sequences. IEEE T Med Imaging 29(2): 442−454
Caicedo JC, Goodman A, Karhohs KW, Cimini BA, Ackerman J, Haghighi M, Heng C, Becker T, Doan M, McQuin C, Rohban M, Singh S, Carpenter AE (2019) Nucleus segmentation across imaging experiments: the 2018 Data Science Bowl. Nat Methods 16(12): 1247−1253
Chai X, Ba Q, Yang G (2018) Characterizing robustness and sensitivity of convolutional neural networks for quantitative analysis of mitochondrial morphology. Quant Biol 6(4): 344−358
Chen B, Gong W, Yang Z, Pan W, Verwilst P, Shin J, Yan W, Liu L, Qu J, Kim JS (2020) STORM imaging of mitochondrial dynamics using a vicinal-dithiol-proteins-targeted probe. Biomaterials 243: 119938
Ciaparrone G, Luque Sánchez F, Tabik S, Troiano L, Tagliaferri R, Herrera F (2020) Deep learning in video multi-object tracking: a survey. Neurocomputing 381: 61−88
Culley S, Albrecht D, Jacobs C, Pereira PM, Leterrier C, Mercer J, Henriques R (2018) Quantitative mapping and minimization of super-resolution optical imaging artifacts. Nat Methods 15(4): 263−266
Dertinger T, Colyer R, Iyer G, Weiss S, Enderlein J (2009) Fast, background-free, 3D super-resolution optical fluctuation imaging (SOFI). Proc Natl Acad Sci USA 106(52): 22287−22292
Dong C, Loy CC, He KM, Tang XO (2016) Image super-resolution using deep convolutional networks. IEEE T Pattern AnalMachIntell 38(2): 295−307
Fang LJ, Monroe F, Novak SW, Kirk L, Schiavon CR, Yu SB, Zhang T, Wu M, Kastner K, Latif AA, Lin ZJ, Shaw A, Kubota Y, Mendenhall J, Zhang Z, Pekkurnaz G, Harris K, Howard J, Manor U (2021) Deep learning-based point-scanning super-resolution imaging. Nat Methods 18(4): 406−416
Farsiu S, Robinson D, Elad M, Milanfar P (2004) Advances and challenges in super-resolution. Int J Imaging Syst Technol 14(2): 47−57
Fu Y, Lei Y, Wang T, Curran WJ, Liu T, Yang X (2020) Deep learning in medical image registration: a review. Phys Med Biol 65(20): 20TR01
Gaire SK, Zhang Y, Li H, Yu R, Zhang HF, Ying L (2020) Accelerating multicolor spectroscopic single-molecule localization microscopy using deep learning. Biomed Opt Express 11(5): 2705−2721
Giepmans BNG, Adams SR, Ellisman MH, Tsien RY (2006) The fluorescent toolbox for assessing protein location and function. Science 312(5771): 217−224
Gu L, Li Y, Zhang S, Xue Y, Li W, Li D, Xu T, Ji W (2019) Molecular resolution imaging by repetitive optical selective exposure. Nat Methods 16(11): 1114−1118
Gu L, Li Y, Zhang S, Zhou M, Xue Y, Li W, Xu T, Ji W (2021) Molecular-scale axial localization by repetitive optical selective exposure. Nat Methods 18(4): 369−373
Gu L, Sheng Y, Chen Y, Chang H, Zhang Y, Lv P, Ji W, Xu T (2014) High-density 3D single molecular analysis based on compressed sensing. Biophys J 106(11): 2443−2449
Guo YT, Li D, Zhang SW, Yang YR, Liu JJ, Wang XY, Liu C, Milkie DE, Moore RP, Tulu US, Kiehart DP, Hu JJ, Lippincott-Schwartz J, Betzig E, Li D (2018) Visualizing intracellular organelle and cytoskeletal interactions at nanoscale resolution on millisecond timescales. Cell 175(5): 1430−1442
Gupta A, Harrison PJ, Wieslander H, Pielawski N, Kartasalo K, Partel G, Solorzano L, Suveer A, Klemm AH, Spjuth O, Sintorn I-M, Wählby C (2019) Deep learning in image cytometry: aReview. Cytometry A 95(4): 366−380
Gustafsson MGL (2000) Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. J Microsc 198(2): 82−87
Gustafsson N, Culley S, Ashdown G, Owen DM, Pereira PM, Henriques R (2016) Fast live-cell conventional fluorophore nanoscopy with ImageJ through super-resolution radial fluctuations. Nat Commun 7(1): 12471
Haskins G, Kruger U, Yan P (2020) Deep learning in medical image registration: a survey. Mach Vis Appl 31(1): 8
Hell SW, Wichmann J (1994) Breaking the diffraction resolution limit by stimulated-emission - stimulated-emission-depletion fluorescence microscopy. Opt Lett 19(11): 780−782
Hershko E, Weiss LE, Michaeli T, Shechtman Y (2019) Multicolor localization microscopy and point-spread-function engineering by deep learning. Opt Express 27(5): 6158−6183
Hoffman DP, Slavitt I, Fitzpatrick CA (2021) The promise and peril of deep learning in microscopy. Nat Methods 18(2): 131−132
Hofmann M, Eggeling C, Jakobs S, Hell SW (2005) Breaking the diffraction barrier in fluorescence microscopy at low light intensities by using reversibly photoswitchable proteins. Proc Natl Acad Sci USA 102(49): 17565−17569
Huang F, Hartwich TMP, Rivera-Molina FE, Lin Y, Duim WC, Long JJ, Uchil PD, Myers JR, Baird MA, Mothes W, Davidson MW, Toomre D, Bewersdorf J (2013) Video-rate nanoscopy using sCMOS camera–specific single-molecule localization algorithms. Nat Methods 10(7): 653−658
Jin L, Liu B, Zhao F, Hahn S, Dong B, Song R, Elston TC, Xu Y, Hahn KM (2020) Deep learning enables structured illumination microscopy with low light levels and enhanced speed. Nat Commun 11(1): 1934
Jones SA, Shim S-H, He J, Zhuang X (2011) Fast, three-dimensional super-resolution imaging of live cells. Nat Methods 8(6): 499−505
Klar TA, Jakobs S, Dyba M, Egner A, Hell SW (2000) Fluorescence microscopy with diffraction resolution barrier broken by stimulated emission. Proc Natl Acad Sci USA 97(15): 8206−8210
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inform Process Syst 25: 1097−1105
Laissue PP, Alghamdi RA, Tomancak P, Reynaud EG, Shroff H (2017) Assessing phototoxicity in live fluorescence imaging. Nat Methods 14(7): 657−661
LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553): 436−444
Leshno M, Lin VY, Pinkus A, Schocken S (1993) Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Netw 6(6): 861−867
Li D, Shao L, Chen BC, Zhang X, Zhang MS, Moses B, Milkie DE, Beach JR, Hammer JA, Pasham M, Kirchhausen T, Baird MA, Davidson MW, Xu PY, Betzig E (2015) Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics. Science 349(6251): aab3500
Li M, Shan H, Pryshchep S, Lopez MM, Wang G (2020) Deep adversarial network for super stimulated emission depletion imaging. J Nanophoton 14(1): 016009
Li P, Wang D, Wang L, Lu H (2018) Deep visual tracking: Review and experimental comparison. Pattern Recogn 76: 323−338
Lichtman JW, Conchello J-A (2005) Fluorescence microscopy. Nat Methods 2(12): 910−919
Lin Y, Long JJ, Huang F, Duim WC, Kirschbaum S, Zhang Y, Schroeder LK, Rebane AA, Velasco MGM, Virrueta A, Moonan DW, Jiao J, Hernandez SY, Zhang Y, Bewersdorf J (2015) Quantifying and optimizing single-molecule switching nanoscopy at high speeds. PLoS One 10(5): e0128135
Liu L, Ouyang W, Wang X, Fieguth P, Chen J, Liu X, Pietikäinen M (2020) Deep learning for generic object detection: a survey. Int J Comput Vis 128(2): 261−318
Luisier F, Blu T, Unser M (2011) Image denoising in mixed Poisson–Gaussian noise. IEEE T Image Process 20(3): 696−708
McCann MT, Jin KH, Unser M (2017) Convolutional neural networks for inverse problems in imaging: a review. IEEE Signal Process Mag 34(6): 85−95
Möckl L, Roy AR, Moerner WE (2020) Deep learning in single-molecule microscopy: fundamentals, caveats, and recent developments. Biomed Opt Express 11(3): 1633−1661
Moen E, Bannon D, Kudo T, Graf W, Covert M, Van Valen D (2019) Deep learning for cellular image analysis. Nat Methods 16(12): 1233−1246
Nehme E, Freedman D, Gordon R, Ferdman B, Weiss LE, Alalouf O, Naor T, Orange R, Michaeli T, Shechtman Y (2020) DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning. Nat Methods 17(7): 734−740
Nehme E, Weiss LE, Michaeli T, Shechtman Y (2018) Deep-STORM: super-resolution single-molecule microscopy by deep learning. Optica 5(4): 458−464
Ouyang W, Aristov A, Lelek M, Hao X, Zimmer C (2018) Deep learning massively accelerates super-resolution localization microscopy. Nat Biotechnol 36(5): 460−468
Ovesný M, Křížek P, Švindrych Z, Hagen GM (2014) High density 3D localization microscopy using sparse support recovery. Opt Express 22(25): 31263−31276
Qiao C, Li D, Guo Y, Liu C, Jiang T, Dai Q, Li D (2021) Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat Methods 18(2): 194−202
Rawat W, Wang Z (2017) Deep convolutional neural networks for image classification: a comprehensive review. Neural Comput 29(9): 2352−2449
Rego EH, Shao L, Macklin JJ, Winoto L, Johansson GA, Kamps-Hughes N, Davidson MW, Gustafsson MGL (2012) Nonlinear structured-illumination microscopy with a photoswitchable protein reveals cellular structures at 50-nm resolution. Proc Natl Acad Sci USA 109(3): E135−E143
Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3): 211−252
Rust MJ, Bates M, Zhuang XW (2006) Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat Methods 3(10): 793−795
Sage D, Pham T-A, Babcock H, Lukes T, Pengo T, Chao J, Velmurugan R, Herbert A, Agrawal A, Colabrese S, Wheeler A, Archetti A, Rieger B, Ober R, Hagen GM, Sibarita J-B, Ries J, Henriques R, Unser M, Holden S (2019) Super-resolution fight club: assessment of 2D and 3D single-molecule localization microscopy software. Nat Methods 16(5): 387−395
Sahl SJ, Hell SW, Jakobs S (2017) Fluorescence nanoscopy in cell biology. Nat Rev Mol Cell Biol 18(11): 685−701
Schmarje L, Santarossa M, Schröder SM, Koch R (2021) A survey on semi-, self- and unsupervised learning for image classification. IEEE Access 9: 82146−82168
Shorten C, Khoshgoftaar TM (2019) A survey on image data augmentation for deep learning. J Big Data 6(1): 60
Sigal YM, Zhou R, Zhuang X (2018) Visualizing and discovering cellular structures with super-resolution microscopy. Science 361(6405): 880−887
Valli J, Garcia-Burgos A, Rooney LM, Vale de Melo e Oliveira B, Duncan RR, Rickman C (2021) Seeing beyond the limit: a guide to choosing the right super resolution microscopy technique. J Biol Chem 297(1): 100791
van Engelen JE, Hoos HH (2020) A survey on semi-supervised learning. Mach Learn 109(2): 373−440
van Ouwerkerk JD (2006) Image super-resolution survey. Image Vis Comput 24(10): 1039−1052
Wagner N, Beuttenmueller F, Norlin N, Gierten J, Boffi JC, Wittbrodt J, Weigert M, Hufnagel L, Prevedel R, Kreshuk A (2021) Deep learning-enhanced light-field imaging with continuous validation. Nat Methods 18(5): 557−563
Wang HD, Rivenson Y, Jin YY, Wei ZS, Gao R, Gunaydin H, Bentolila LA, Kural C, Ozcan A (2019a) Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat Methods 16(1): 103−110
Wang J, Sun K, Cheng T, Jiang B, Deng C, Zhao Y, Liu D, Mu Y, Tan M, Wang X (2020a) Deep high-resolution representation learning for visual recognition. IEEE T Pattern Anal Mach Intell
Wang Q, Ma Y, Zhao K, Tian Y (2020b) A comprehensive survey of loss functions in machine learning. Ann Data Sci
Wang Z, Chen J, Hoi SCH (2020c) Deep learning for image super-resolution: a survey. IEEE T Pattern Anal Mach Intell
Weigert M, Schmidt U, Boothe T, Müller A, Dibrov A, Jain A, Wilhelm B, Schmidt D, Broaddus C, Culley S, Rocha-Martins M, Segovia-Miranda F, Norden C, Henriques R, Zerial M, Solimena M, Rink J, Tomancak P, Royer L, Jug F, Myers EW (2018) Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat Methods 15(12): 1090−1097
Wu Y, Rivenson Y, Wang H, Luo Y, Ben-David E, Bentolila LA, Pritz C, Ozcan A (2019) Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning. Nat Methods 16(12): 1323−1331
Wu Z, Pan S, Chen F, Long G, Zhang C, Yu PS (2021) A comprehensive survey on graph neural networks. IEEE T Neural Netw Learn Syst 32: 4−24
Xing F, Xie Y, Su H, Liu F, Yang L (2018) Deep learning in microscopy image analysis: a survey. IEEE T Neural Netw Learn Syst 29: 4550−4568
Yang L, Shami A (2020) On hyperparameter optimization of machine learning algorithms: theory and practice. Neurocomputing 415: 295−316
Yang WM, Zhang XC, Tian YP, Wang W, Xue JH, Liao QM (2019) Deep learning for single image super-resolution: a brief review. IEEE T Multimedia 21(12): 3106−3121
Yu Y, Si X, Hu C, Zhang J (2019) A review of recurrent neural networks: LSTM cells and network architectures. Neural Comput 31(7): 1235−1270
Zelger P, Kaser K, Rossboth B, Velas L, Schütz GJ, Jesacher A (2018) Three-dimensional localization microscopy using deep learning. Opt Express 26(25): 33166−33179
Zhang P, Liu S, Chaurasia A, Ma D, Mlodzianoski MJ, Culurciello E, Huang F (2018) Analyzing complex single-molecule emission patterns with deep learning. Nat Methods 15(11): 913−916
Zhang Q-S, Zhu S-C (2018) Visual interpretability for deep learning: a survey. Front Inform Technol Electron Eng 19(1): 27−39
Zhao Z, Zheng P, Xu S, Wu X (2019) Object detection with deep learning: a review. IEEE T Neural Netw Learn Syst 30(11): 3212−3232
Zhou J, Cui G, Hu S, Zhang Z, Yang C, Liu Z, Wang L, Li C, Sun M (2020) Graph neural networks: a review of methods and applications. AI Open 1: 57−81
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
