Deep Learning in Microscopy

Understanding the Benefits and Limitations of Deep Learning for Microscopy

While deep learning-based algorithms and software hold much promise for supporting optical microscopy image analysis – it is important to understand that such approaches are not a “cure-all,” but rather different ways to approach a problem. Deep learning (DL) algorithms, which work by inferring correlations between data features (which may not necessarily be obvious or intuitive), will be more well-suited to some problems than classical approaches. However, often the greater benefit of artificial intelligence (AI) for microscopy is acceleration of image analysis.

Supervised DL algorithms, as used by Nikon’s NIS.ai AI modules, require extensive data for training the neural network. While Nikon pre-trains some AI modules, including Denoise.ai, Clarify.ai, and Autosignal.ai, others will need to be trained using the specific acquisition conditions for a given experiment. This process can be lengthy, but once completed it will allow for fast analysis of new data. It is important to understand that the results are not the ground truth. Ultimately, this limits the appropriate applications.

As an example of an appropriate use case, the Convert.ai module can be trained to predict DAPI nuclear staining patterns from transmitted light images. This predicted DAPI channel is useful and appropriate for cell counting, which depends mostly on the correct localization of DAPI signal. However, using predicted DAPI images for applications requiring quantification of intensity information would be inappropriate. While DL is reliable for accurately predicting DAPI localization and gross staining patterns, it is not a good choice for quantifying fine details and rare events, the latter of which would not be well represented in the training data to begin with and likely to be missed.

Streamlining Imaging and Analysis with Deep Learning

An advantage of DL methods is how quickly they can perform different and complex tasks. This allows for stacking of DL methods to perform complicated multi-part analyses in relatively short periods of time, even during the imaging process when time is limited.

For example, the Autosignal.ai module automatically determines the optimal laser power and gain within a matter of seconds when using our AX / AX R confocal microscope. A classical approach to this problem is unlikely to return a reliable solution in such a short period. Even a handful of seconds can make a significant difference when competing with factors such as photobleaching.

The Denoise.ai module is similarly fast, providing near-real time denoising. This makes for a synergistic combination with Autosignal.ai – the imaging conditions can be optimized within seconds with the push of a button. Other AI modules can be similarly stacked, such as Segment.ai for fast segmentation of image features, as shown in the video.

NIS.ai modules can be plugged into existing imaging and analysis pipelines within our NIS-Elements software. This is beneficial for optimizing the efficiency of high content-type experiments. The General Analysis 3 (GA3) module is used to craft customized measurement routines and allows users to apply NIS.ai DL-based and classical analyses together in the same pipeline. NIS.ai modules can also be embedded within the imaging routine using the JOBS experiment wizard – results obtained using an NIS.ai module within JOBS can be used to direct the course of the experiment. For example, one can automatically perform targeted acquisitions based on analysis results – a fitting choice for targeting rare events.

Deep Learning for Live Cell Imaging

One of the greatest benefits provided by many of the NIS.ai AI modules is the reduction of photon dose. Denoise.ai and Clarify.ai provide clearer images without requiring additional signal to be collected. Convert.ai allows for prediction of fluorescence imaging channels from transmitted light images (e.g., DIC), removing the need for fluorescence imaging altogether in some cases. Enhance.ai can predict higher signal images from low-signal data, minimizing the photon dose required for excitation.

Fluorescence microscopy generally requires much higher illumination intensity than transmitted light methods, has a limited photon budget, and is often the method of choice for live cell imaging, especially in research. Reduction of photon dose is useful for most imaging applications, but is essential for live cell fluorescence imaging, where photobleaching and phototoxicity are top concerns.

The Direction of AI and Deep Learning in Microscopy

The application of AI and deep learning in microscopy is still in a relatively early stage, with new research and applications consistently being published and pushing the field forward. Given the potential of DL methods, and how quickly such methods are coming into the mainstream, it is important that the AI and microscopy communities develop clear guidelines for appropriate applications and quality metrics that allow non-specialists without computational backgrounds to be confident in using these tools for peer-reviewed research.

Nikon’s pre-trained AI modules, which currently includes Clarify.ai and Denoise.ai, do not allow for manual adjustment of ambiguous settings such as analysis “strength,” avoiding an opportunity for user bias to affect the results. Everyone who uses Clarify.ai or Denoise.ai is utilizing the same pre-trained network as every other user.

For NIS.ai modules that must be trained, including Convert.ai, Enhance.ai, and Segment.ai, the module includes an intuitive graphic user interface for network training. The user can monitor the progress of training, including a real-time view of the number of iterations and training loss.