Sample-induced image-degradation remains an intricate wave-optical problem in light-sheet microscopy. Here we present biobeam, an open-source software package that enables simulation of operational light-sheet microscopes by combining data from 105–106 multiplexed and GPU-accelerated point-spread-function calculations. The wave-optical nature of these simulations leads to the faithful reproduction of spatially varying aberrations, diffraction artifacts, geometric image distortions, adaptive optics, and emergent wave-optical phenomena, and renders image-formation in light-sheet microscopy computationally tractable.
Light-sheet fluorescence microscopy is a popular tool for the volumetric imaging of developing organisms [1–5]. As light-sheet microscopes continue to be developed for progressively bigger biological samples [6,7], there is an increasing need for computers to process gigantic sets of imaging data and extract biologically relevant information [8]. With sample size, however, also light-scattering induced imaging artifacts become increasingly prevalent. During data acquisition as well as post-processing, these imaging artifacts are mostly dealt with on a purely phenomenological basis. A faithful forward model of the wave-optical imaging process, however, would i) enable rigorous benchmarks of deconvolution and segmentation strategies against ground-truth data [9], ii) serve as training platforms for machine learning approaches for image restoration and information extraction and iii) leverage the efficient use of adaptive optics [10,11] to prevent sample-induced image degradation during the acquisition process. Predicting light-tissue interactions is particularly demanding when leaving the single scattering regime [12] or strictly diffusive transport [13]. And despite significant computational advances [14], generally applicable solutions [15] remain computationally costly, effectively prohibiting the simulation of image formation in microscopy. Even when constraining simulations to the biological relevant case of predominantly forward scattering tissue [10,16], individual point spread function (PSF) calculations still require multiple seconds [17]. As aberrations deep inside tissues are unique for virtually each point in the sample [10,12] a realistically large biological specimen, i. e. an embryo with a volume ∼ 100 μm3 would require 105–106 volumetric PSF calculations in order to faithfully mimic the wave-optical imaging process, which with current methods would take several weeks. As a consequence, attempts to simulate microscopic imaging have been limited to ray optics [18] or convolution with a constant PSF [19], approaches that do not reflect the wave-optical nature of light interaction with optically heterogeneous biological samples. Here, we report on biobeam, a software package that enables the first rigorous simulations of wave-optical image formation in light-sheet microscopes by i) a novel multiplexing scheme for PSF calculations and ii) efficient GPU parallelization.
The pipeline underlying our software is based on the observation that the beam-propagation model (BPM) for fiber optics [20,21] can also be used to mimic scattering biological cells [22]. To guarantee good accuracy of BPM beyond strictly paraxial wave propagation, we use the exact propagator together with a locally adapted expansion of refractive indices (see S1 Text, Notes 1 and 4 for details and validation against analytically tractable scattering models). To massively reduce the computational cost for ∼106 wave-optical PSF calculations, we introduce a novel multiplexing scheme. In this we exploit the fact that for typical imaging scenarios, a single camera image can be constructed from sets of 100s to 1000s of mutually independent, spatially varying, non-overlapping PSFs. Such sets can be calculated within single, highly multiplexed simulations (see S3 Fig and S1 Text, Note 8), which is similar to the operational principle of highly multiplexed confocal measurements of spinning disc microscopes [23,24]. Our software is further accelerated ∼20 fold by the efficient use of GPU implementations for all low-level calculations. In this way, the propagation of an arbitrary light field such as the multiplexed set of ∼1000 PSFs through a typical volume of 10243 voxels takes less than 500ms on a single graphics card (cf. S1 Text, Note 5, S1 Table and S4 Video), corresponding to a 20. 000 fold acceleration compared to the sequential calculation of PSFs. Implementation was done as an open source Python software package. All computationally heavy parts are lifted to the GPU via OpenCL [25], thereby keeping all the advantages of Python as a dynamically typed high level language that is vastly used in the scientific community without compromising on performance. Apart from its technical focus on speed, biobeam is specifically designed to make wave-optical experiments in-silico as easy as possible, e. g. by providing a simple API that makes it easy to apply different input fields as well as PSF/aberration calculations via diffraction-limited point source propagation through tissue (see S1 Text, Note 3 for example listings).
We first demonstrate the power of our software package by presenting the first wave-optical simulation of the volumetric image-formation process in a light-sheet microscope [5,9, 18]. To start, a cylindrical sheet of light is propagated at a specific axial position through a fluorescently labelled embryo model with refractive indices of cytoplasm, nuclei and organelles as previously reported [12] (Fig 1a and S1 & S7 Videos). From this, one obtains the fluorescence excitation at every point in the volume for every axial position of the incident light field. Next, for each focal plane a full set of detection PSFs is obtained by propagating light from multiplexed, diffraction-limited point-sources orthogonally through an idealized, refocusing lens towards the camera (Fig 1c, upper left, S2 Video). This way, we obtain a quasi-continuum of spatially-dependent, volumetric PSFs with position-dependent aberrations that stem from distortions and scattering in both the illumination and the detection paths (Fig 1c left, see also S4 Fig for calculated aberration maps). Convolving the exhaustively sampled sets of spatially-varying PSFs with the fluorescent object finally yields the wave-optical image as seen by the camera (Fig 1c, right). These simulations are particularly demanding due to the large grid size (i. e. 1024 × 2048 × 1024 voxels for Fig 1) and sampling density requirements (typically 0. 5–5μm), here resulting in a total of 106 PSFs that were calculated in well under 10 minutes on a single graphics card (see also S1 and S2 Videos). In contrast, the non-multiplexed calculations would have required more than two months on a single CPU. As a result of this computational pipeline, biobeam generates faithfully calculated 3D microscopy data sets that account for both refraction and diffraction based imaging artifacts. These include image blur and contrast loss, spatially varying PSFs, speckle artifacts, image granularity, as well as sample-induced geometric distortions such as lensing, image displacements and split-screen type double images (cf. Fig 2a and 2b, S3 Video and S1 Fig for benchmarks against analytically tractable models). We additionally validated our approach by comparing simulation results for several experimental situations, such as the diffraction pattern behind an edge knife (S5 Fig), the scattering of a light-sheet at agarose embedded micro-spheres (S6 Fig) and the image distortions obtained when projecting test images through a refracting sphere (S7 and S8 Figs). Beyond the theoretical validation of the core propagation model, these successful comparisons of simulations against experiments further demonstrate that our high-level implementation works correctly in the biological relevant case of low refractive index contrasts. Being a particularly flexible software package, biobeam provides further pre-implemented illumination modalities, including Bessel lattices (Fig 1d, right, see S2 Fig, S5 Video), and allows for practical extraction of sample-induced aberrations as spatially resolved Zernike maps (see S4 Fig and S1 Text, Note 9). Next, we demonstrate biobeam’s capability to accurately simulate wave-optical effects relevant to adaptive optics (AO) imaging [10]. It is well-established and exploited in imaging that perfect imaging foci can be created behind strongly scattering screens [26] and within biological samples by appropriately shaping the wavefront before entry into the scattering medium. We recapitulated this finding by explicit simulations of light propagating through a spatially extended synthetic tissue sample represented by a Perlin-noise refractive index distribution (n = 1. 36 ± 0. 03). As expected, we find also in simulations that conjugation of the wavefront at the surface of the sample allows one to generate diffraction-limited foci inside these scattering tissues (Fig 2c, S6 Video and S9 Fig). Furthermore, we show that biobeam is capable of reproducing the shift-shift memory effect, an emergent wave-optical phenomenon responsible for the significant robustness of adaptive imaging against lateral focus displacements. According to this wave-optical phenomenon, the radius of the iso-planatic patch (distance over which an AO correction pattern works) is determined by the statistics of a tissue generated speckle field inside this medium. Based on a total of 22500 PSF calculations, we show that our simulations faithfully capture this emergent wave-optical effect. In agreement with recent analytical arguments and their empirical confirmation [10], we show by our computational microscopy experiments that the average persistence of the laterally shifted focus is precisely limited by the autocorrelation length of a speckle pattern that would result from an incident plane wave. This phenomenon is accurately reproduced even four mean-free-path lengths inside the tissue (Fig 2d). While currently using a variant of the beam propagation method, our PSF multiplexing scheme is in principle also compatible with other low-level field stepping algorithms, including those that iteratively account for multiple back-reflections [14,15], when even higher precision or penetration depths are required.
We summarize that biobeam enables faithful whole-tissue wave-optical simulations of light-sheet microscopes due to i) a novel multiplexing scheme of PSF calculations and ii) efficient GPU parallelization. Biobeam renders the biological imaging process computationally tractable, thus providing the link between wave-optically recorded image material, and the ground truth object. Given the modular nature of our software package, these simulations are easy to implement and can be flexibly adapted to custom imaging scenarios and microscopes. Beyond the reproduction and identification of commonly occurring imaging artifacts (see Supplement for comparison with experimental data), we demonstrated that biobeam is compatible with imaging scenarios in which sample-induced PSFs degradation is overcome by the use of adaptive optics. Furthermore, we showcase the optical capabilities and accuracy of our software by explicitly demonstrating that emergent wave-optical phenomena such as the shift-shift memory effect are quantitatively reproduced deep inside tissues. While we chose a variant of BPM as a low-level field stepping routine, the here presented strategy of multiplexing PSF calculation is more general in nature, and may also be used in combination with other light propagation algorithms, e. g. in scenarios where higher accuracy at large angles and/or more isotropic scattering is of relevance [14]. We conclude that biobeam is a flexible and particularly powerful platform to systematically study wave-optical image-formation by microscopes in scattering biological tissues.
Prospectively, we see biobeam helping to improve microscope design, enhancing deconvolution and segmentation strategies by providing realistic imaging data-sets along with ground truth data, and paving the way for a new generation of smart, adaptive microscopes that learn to treat the sample as a part of the optical path. Of great help in this will be the rapidly increasing knowledge of stereotypical refractive index distributions in embryos, tissues, cells, subcellular compartments and their constituents as derived from tomographic phase microscopy [12,27–29] and complemented by electron microscopy morphological data [30]. Beyond the simulation of light-sheet microscopes, our software can also be used to simulate other imaging modalities such as wide-field, laser-scanning confocal, and light-field [31] microscopes, as well as for novel micro-lens concepts [32] and the emerging field of soft photonics [33]. Especially, we emphasize the potential to improve the understanding of physiological image-formation inside the eye [34], taking into account the optics of the lens [35] as well as the retina [36–39]. Biobeam is available as open-source (BSD-3 license) python package at https: //maweigert. github. io/biobeam. Datasets can be found at https: //publications. mpi-cbg. de/6874-data/.
Modern microscopes permit to acquire high quality images of large fields of view, which is the result of a decade-long development of computer aided optical design. However, this high image quality can only be obtained at the very surface of biological specimens: when trying to penetrate deeper into biological tissues, light scattering by cells rapidly leads to severe image blur and computers have so far been unable to model the process by which light forms images in such turbid optical environments. We developed a software that allows one to simulate how microscopes record images deep inside scattering biological samples. Our software reproduces a wide range of optical effects that underlie image blur in tissues. Hence strategies to improve image quality within three-dimensional samples can now be systematically tested by computers. Specifically, our software reproduces intricate wave-optical effects that have recently been proposed as strategies to gain perfect images even in the most turbid environments. This provides the chance for a new generation of microscopes, in which computer models guide the imaging process to enable highest possible resolution even deep inside biological specimens.
fluorescence imaging
chemical characterization
diffraction
digital video imaging microscopy
medicine and health sciences
engineering and technology
light microscopy
light
electromagnetic radiation
simulation and modeling
adaptive optics
light scattering
optical analysis
microscopy
optical equipment
pharmacology
waves
research and analysis methods
tissue distribution
imaging techniques
pharmacokinetics
physics
biochemistry
scattering
equipment
biology and life sciences
physical sciences
refractive index
video microscopy
Biobeam—Multiplexed wave-optical simulations of light-sheet microscopy