Noise-Transfer2Clean: denoising cryo-EM images based on noise modeling and transfer

Abstract Motivation Cryo-electron microscopy (cryo-EM) is a widely used technology for ultrastructure determination, which constructs the 3D structures of protein and macromolecular complex from a set of 2D micrographs. However, limited by the electron beam dose, the micrographs in cryo-EM generally suffer from the extremely low signal-to-noise ratio (SNR), which hampers the efficiency and effectiveness of downstream analysis. Especially, the noise in cryo-EM is not simple additive or multiplicative noise whose statistical characteristics are quite different from the ones in natural image, extremely shackling the performance of conventional denoising methods. Results Here, we introduce the Noise-Transfer2Clean (NT2C), a denoising deep neural network (DNN) for cryo-EM to enhance image contrast and restore specimen signal, whose main idea is to improve the denoising performance by correctly learning the noise distribution of cryo-EM images and transferring the statistical nature of noise into the denoiser. Especially, to cope with the complex noise model in cryo-EM, we design a contrast-guided noise and signal re-weighted algorithm to achieve clean-noisy data synthesis and data augmentation, making our method authentically achieve signal restoration based on noise’s true properties. Our work verifies the feasibility of denoising based on mining the complex cryo-EM noise patterns directly from the noise patches. Comprehensive experimental results on simulated datasets and real datasets show that NT2C achieved a notable improvement in image denoising, especially in background noise removal, compared with the commonly used methods. Moreover, a case study on the real dataset demonstrates that NT2C can greatly alleviate the obstacles caused by the SNR to particle picking and simplify the identifying of particles. Availabilityand implementation The code is available at https://github.com/Lihongjia-ict/NoiseTransfer2Clean/. Supplementary information Supplementary data are available at Bioinformatics online.


Raw
Lowpass Gaussian BM3D TopazDe NT2C Figure S1: The whole-image denoised results on real dataset EM28 among different denoising algorithms.

Raw
TopazDe -gen NT2C Micrographs Clusters Particles Figure S3: The particles picking results on EM77 when directly combine denoising methods with meanshift algorithm (Comaniciu and Meer, 2002). Most of the particles in micrograph denoised by NT2C can be accurately identified, while the particles in raw micrograph and micrograph denoised by TopazDe are difficult to be clustered and recoginized.

S1 The imaging model in Cryo-EM
The imaging process in cryo-electron microscopy (cryo-EM) involves the conversion of the electron wave's intensity distribution into a digital signal via a detector (Vulović et al., 2013), which introduces multiple complex noises to the final micrographs. According to the principle of cryo-EM, the detected image can be written as whereĨ 0 (q) is the Fourier spectrum of the noise-free signal, DQE is the detective quantum efficiency, Φ e is the incident electron flux in [e − /area], F is Fourier transform, F −1 is inverse Fourier transform, P oiss is the Poisson described shot noise, N T F is the noise transfer function, CF is a scale factor for detectors, I rn indicates the readout noise and I dc indicates the dark current of detector. The detective quantum efficiency DQE is defined as DQE (q) = M T F 2 (q) /N T F 2 (q), where N T F 2 (q) = N P S out / CF 2 Φ e with N P S being the noise power spectrum and M T F describes transfer of the signal amplitude for different spatial frequencies.
As described in the formula, the signal propagation can be modeled as follows: (1)Ĩ 0 (q) is damped (multiplied) by the ratio between signal (M T F ) and noise (N T F ) transfer, (2) the signal is multiplied by the integrated electron flux and shot noise contributions are added, (3) the Fourier spectrum of that noisy signal is damped by the N T F , (4) the number of electrons are scaled to the image gray values in detector analog-to-digital unit [ADU ]. Therefore, the noise in cryo-EM comes from three aspects: the shot noise contributions added in the measurement process; the readout noise I rn and dark current I dc from the detector; blurring effect caused by detector point spread function PSF (x,y) whose Fourier transform is the MTF.

S2.1 Training details
The GAN noise synthesizer is trained with the weights initialized from a standard Gaussian distribution with σ = 0.02. The slope of the leak of the LeakyReLU activation for both generator and discriminator is set to 0.2. Adam optimizer is used for model training with hyper-parameters set to 0.5, weight updating β 1 = 0.999, and a learning rate of 0.0002. The model is trained on two NVIDIA 2080Ti with 12GB VRAM.
To train the CNN denoiser (both coarse denoiser and fine denoiser), the whole micrograph is cropped into 320 × 320 pixels with an overlap of 160 pixels. For each dataset, we randomly extracted 10% of images as a validation dataset. The coarse CNN denoiser used for rough micrograph enhancement is trained with simulated datasets with default initialization (Paszke et al., 2019). The fine CNN denoiser used for final denoising is trained with clean-noisy pairs generated by the GAN synthesizer and signal re-weight algorithm, initialized from the model weights of corresponding coarse CNN denoiser. These two models are all trained using the Adagrad (Lydia and Francis, 2019) with a learning rate of 0.001. Given a trained denoiser, the denoising for a full-size micrograph is performed in overlapping patches. A padding approach is adopted to avoid the artifacts occurring at the patches' edge (Bepler et al., 2020). The number of micrographs in the training set is shown in the Table S1 which the best choice we make after considering the tradeoff of training performance and training time. The model is trained on two NVIDIA 2080Ti with 12GB VRAM.

S2.2 Experiments on simulated results
We first verify the robustness of simulation-based noise pattern capture, which is a critical part of NT2C. Then we designed an experiments to demonstrate the denoising performance of NT2C on simulated dataset and compared it with four mainstream cryo-EM denoising methods.
In NT2C, the simulated datasets are generated with InsilicoTEM utilizing randomly-downloaded proteins from PDB. We generated four simulated datasets by 2wrj.pdb, 1kd1.pdb, 5lzf.pdb and 1gr5.pdb and denoted them as SIM1, SIM2, SIM3, SIM4. As shown in Figure S4, comparing with 2wrj.pdb, 5lzf.pdb has a similar 2wrj 5lzf 1kd1 1gr5 Figure S4: Illustration of the proteins used for simulated datasets generation and validation. 2wrj.pdb is specified to generate the dataset to be denoised, while 5lzf.pdb, 1kd1.pdb, and 1gr5.pdb are the proteins used to generate peer simulation used in coarse CNN denoiser. Comparing with 2wrj.pdb, 5lzf.pdb has a similar structure and size, 1kd1.pdb has a similar structure and smaller size, 1gr5.pdb has a completely different structure and smaller size.
structure and size, 1kd1.pdb has a similar structure and smaller size, 1gr5.pdb has a completely different structure and smaller size.

S2.2.1 Robustness of simulation-based noise pattern capture
The simulation based on experimental parameters is the start point for coarse CNN denoiser training, which is critical to noise patch extraction. More importantly, the simulated clean-noisy training pairs are then re-weighted with real noise to build training dataset for fine CNN denoiser, which directly affects denoising performance. However, the protein adopted in the simulation may be quite different from the unknown biological structures in real data. In order to verify the effect of this difference on the denoising performance, we conducted experiments using simulated data.
Specifying SIM1 as the dataset to be denoised (assumed to be real data without groundtruth), SIM2-4 (simulated data with noisy-clean pairs) are generated with the same simulation parameters as SIM1. We firstly trained three coarse CNN denoisers with SIM2-4 to roughly enhance the image contrast of SIM1. Then, we extracted the noise patches from SIM1 and learned the noise properties of these patches by a GAN noise synthesizer. The synthesized noise patches were then transferred to the clean signal patches of SIM2-4 for the training of fine CNN denoisers. The denoiser trained on clean-noisy pairs synthesized from xxxx.pdb is denoted as NT2C-xxxx. We tested these different denoisers on SIM1, whose results are shown in the first two rows of Figure S5. The NT2C-2wrj acts as a reference for comparison, which learns true noise patterns directly from clean-noisy pairs. It can be found that all these four denoisers have correctly resolved the specimen signal from the degenerated micrographs, resulting in a denoised output almost identical to each other.
Furthermore, with the availability of ground truth, we are able to quantitatively measure the denoising performance of different denoisers. The peak signal-to-noise ratio (PSNR) (Huynh-Thu and Ghanbari, 2008), SSIM, and Pearson correlation coefficient (Pearson CC) (Benesty et al., 2009) between the denoised micrograph and the ground-truth were calculated and summarized in Table S2. It can be found that the size and structure of protein used in simulation hardly affected the noise pattern mining and signal restoration of NT2C. Compared to NT2C-2wrj which reveals true noise distribution, the other three denoisers that trained with datasets simulated from different proteins achieved almost the same performance on all criteria.

S2.2.2 Evaluation with Simulated Noise
We compared the performance of NT2C on SIM1 with four mainstream cryo-EM denoising methods, including three conventional methods, i.e., Low-pass filter, Gaussian filter (Haddad et al., 1991), BM3D (Dabov et al., 2007), and a learning-based methods, i.e., Topaz-Denoise 1 . We also provide a coarse CNN denoiser that is used for roughly image contrast-enhancing in the noise extraction module of NT2C, denoted as NT2C-coarse (trained with clean-noisy pairs of SIM2).
2) Denoising with NT2C improves micrograph interpretability and PSNR in simulated datasets. The last two rows show comparison of different denoising methods on dataset generated with 2wrj.pdb, from left to right are Lowpass filter (2x binning), Gaussian filter, BM3D (third row), TopazDe-gen, TopazDe and NT2C-coarse (last row). Detailed views of boxed region are shown at lower right corner of the micrograph. Denoising with NT2C improves micrograph interpretability and PSNR in simulated datasets.
The last two rows of Figure S5 present the denoised results of Low-pass filter (2x binning), Gaussian filter, BM3D, TopazDe-gen, TopzaDe, and coarse-NT2C. It can be found that both TopzaDe and NT2C remove background noise while preserving the structural features significantly better than conventional methods. The very weak particle signals in the raw micrograph are clearly restored after denoised by TopzaDe and NT2C. The image contrast is greatly enhanced by TopzaDe-gen. However, the noise is not completely removed. Especially, as shown in the small representative area, the image denoised by NT2C-coarse contains artifacts around the specimen signal, which do not appear in NT2C. NT2C can correctly capture the true noise distribution in SIM1 and remove it from the micrograph. We further quantitatively assessed the performance of different methods based on the SSIM, Pearson CC and PSNR (in dB). As shown in Table S3, N2TC achieves the best performance on all three assessment criteria. Especially, our method improves SSIM by 0.05, Pearson CC by 0.06 and PSNR by >3 dB over N2TC-coarse which lacks true noise, and improves Pearson CC by 0.03 and PSNR by roughly 4 dB over TopazDe methods.

S3 Ablation study
In this work, we proposed a denoising framework for image contrast enhancement and specimen signal restoration in cryo-EM. Since this framework is flexible, though we provide an effective model or method for each component, they can be replaced by better ones in the future. The main contribution of this work is to provide an idea of denoising based on estimating the noise model of cryo-EM images over pure noise patches and transferring the statistical nature of noise into the denoiser. Therefore, we designed several ablation studies to verify the effectiveness of each component rather than a specific model: (1) To study the noise modeling ability of GAN noise synthesizer, we show the noise patches generated by GAN noise synthesizer and compared them with the Gaussian noise model which is generally used as prior knowledge in cryo-EM denoising. The denoising model trained with clean-pairs synthesized from noise patches which are generated by the Gaussian noise model is denoted as NT2C-Gaussian. (2) To verify the data augmentation effect of GAN noise synthesizer, only the extracted noise patches are used to build the dataset for fine CNN denoiser. The denoising model is denoted as NT2C-noGAN. (3) To study the effectiveness of contrast-guided noise and signal re-weighted algorithm, we generated the training dataset by directly adding signal and noise. The denoising model is denoted as NT2C-Add. (4) To show the discrepancy in the denoising ability of coarse and fine denoisers, we compared the denoising results of these two denoisers. The coarse denoiser is denoted as NT2C-coarse.

S3.1 NT2C-Gaussian
The noise model in cryo-EM is too complex to be explicitly described by an analytical expression. In NT2C, we build a GAN framework to implicitly learn the latent noise model in cryo-EM micrographs and generate A. The first row shows the noise patches extracted from real dataset EM28, generated by GAN noise synthesizer, generated from Gaussian noise model with variance=0.05, 0.1, 0.2. The second row shows the noisy patches extracted from real dataset, synthesized with noise patches generated by GAN noise synthesizer and Gaussian noise model with variance=0.05, 0.1, 0.2. B. The raw micrograph, micrograph denoised by NT2C-GAN and NT2C-Gaussian. NT2C-GAN indicates the denoiser is trained with clean-noisy pairs simulated with noise patches generated from GAN. NT2C-Gaussian indicates the denoiser is trained with clean-noisy pairs simulated with noise patches generated from Gaussian noise model. more noise samples. To intuitively demonstrate that the "noise model" is correctly captured, the first row of Figure S6A shows the noise patches extracted from real dataset EM28, generated by GAN noise synthesizer, generated from Gaussian noise model with variance=0.05, 0.1, 0.2. It can be seen that the sample generated by GAN noise synthesizer is more similar to the observed real noise sample than that by the Gaussian noise model. Correspondingly, the re-weighted noisy patch synthesized with noise patch generated by GAN is much more similar to the real noisy patch than that of the Gaussian noise model, as the second row of Figure S6A shows. In order to further verify whether the noise pattern is correctly learned, we show the corresponding denoising results in Figure S6B. In Figure S6, "NT2C-GAN" indicates the training dataset for fine denoiser is synthesized from noise patches generated by GAN, and "NT2C-Gaussian" indicates the training dataset is synthesized from noise patches generated by the Gaussian noise model. It can be found that the denoising performance of NT2C-Gaussian is significantly worse than NT2C-GAN. The reason may be that the noise in the cryo-EM image is very complicated, and NT2C-Gaussian can only remove the part of Gaussian noise. On the contrary, NT2C-GAN removes most of the noise and correctly decomplex structured features from complex noise, proving that NT2C has correctly learned the true noise distribution.

S3.2 NT2C-noGAN
The noise modeling is a critical part of the proposed framework NT2C. Generally, we can only extract a small number of pure noise patches as shown in Figure S7, which may lack diversity and not be able to summarize the statistical characteristics of noise in the entire data set. Therefore, NT2C leverages GAN to learn the noise distribution and then to generate samples with more diversity. To investigate the data augmentation effects of GAN, only the extracted noise patches are used to build the training dataset for fine CNN denoiser (NT2C-noGAN). Figure S8 shows a representative region selected from EM28 and denoised results of NT2C-noGAN, NT2C. It can be seen that NT2C achieves a stronger noise removal performance and restored clearer specimen signal than NT2C-noGAN. Furthermore, we quantitatively assessed denoising performance by measuring the SNR of micrographs denoised with NT2C-noGAN and NT2C. As shown in Table S3, the results of NT2C-noGAN are 1.8 dB, 1.85 and 0.8 dB lower than NT2C on average for datasets EM25, EM28, and EM77, respectively. It demonstrated that GAN noise synthesizer could make up for the deficiency of only using extracted blocks. Therefore, it is effective to use GAN for data augmentation to improve the denoising results.

S3.3 NT2C-Add
As described in Section S1, the noise model in cryo-EM is not simple additive noise or multiple noise. Therefore, to cope with the complex noise model in cryo-EM, we design a contrast-guided noise and signal reweighted algorithm to achieve clean-noisy data synthesis and data augmentation. To verify the effectiveness raw NT2C-noGAN NT2C Figure S8: A small region are selected from one micrograph of EM28 to illustrate the performance of NT2C-noGAN and NT2C. NT2C performs better in noise smoothing and signal restoration than NT2C-noGAN.
of this method, we generated the training dataset for fine denoiser by directly adding clean signal and noise sampled from GAN noise synthesizer. Figure S9 shows a representative region selected from dataset EM28, and the re-weighted image, directly added image generated with simulated datasets and noise samples. The re-weighted results have image contrast close to that of dataset EM28. To further study the effects on denoising, we trained two denoiser with datasets synthesized by contrast-guided noise and signal re-weighted algorithm and directly added method. The denoised results are shown in the second row of Figure S9. Due to the misestimated image contrast, the denoiser NT2C-Add harms the true signal of particles and provide inaccurate results. Moreover, as shown in Table S3, the SNRs of NT2C-Add are 4.88 dB, 7.08dB and 5.82dB lower than NT2C on average for EM25, EM28 and EM77.

RAW NT2C-Add
Noisy Denoised NT2C (Re-weighted) Figure S9: The first row shows a small region selected from EM28, the re-weighted and directly added noisy sample. The second row shows a raw micrograph from EM28 and corresponding denoised results of NT2C-Add and NT2C.

S3.4 Coarse denoiser vs Fine denoiser
Our method contains three steps, in the first step, we trained a coarse denoiser to roughly enhance the image quality and aid the extraction of noise patches. Then the GAN noise synthesizer is trained to learn the Coarse-Fine Figure S10: Comparison of the coarse and fine denoiser. A1-4 Amplitude spectrum of the raw micrographs (Raw), micrograph denoise by coarse (Coarse) and fine (Fine) denoiser. Each subfigure is divided into two parts: left and right. B1-3 The density map reconstructed from raw particles, particles denoised by coarse and fine denoiser. B4 shows the Fourier shell correlation FSC curves of three maps in B1-3.
statistical properties of noise, with pure noise patches as input and simulated noise patches as output. In the third step, a fine denoiser is trained with clean-noisy pairs synthesized from noise patches generated by GAN. The first step achieves preliminary denoising, and the third step further carries in-depth denoising. Here we evaluate the denoising performance of coarse and fine denoiser on dataset EM28 to show the discrepancy in denoising ability of these two denoisers. As shown in Figure S10, A1-4 provides the power spectrums of raw micrograph and micrographs denoised by coarse denoiser and fine denoiser, which shows patterns known as Thon rings (Thon, 1966). The Thon ring is the semi-circular intensity oscillations induced by the contrast transfer function (CTF) and reflects the image quality at different frequencies. It can be found that the Fourier amplitudes of denoised micrographs are less than in the raw images at medium and high frequencies. This is because the Fourier amplitude of the noise is at high frequency, therefore the high-frequency signal will also be harmed while denoising. However, it can be seen that compared to coarse denoisers, the fine denoisers can restore Thon rings to a higher resolution. Furthermore, we utilize the denoised particles to perform the reconstruction. We firstly performed standard single-particle cryo-EM structure determination on the raw particles ( Figure S10 B1). This reconstruction as well as the orientational parameters of each particle in the final dataset are then treated as references to quantitatively evaluate the behaviors of denoised particles in 3D reconstruction. We utilized the same particles from the denoised micrographs and calculated a 3D reconstruction using the orientation and CTF parameters determined from the raw noisy images. The reconstruction of denoised particles has the correct overall shape and some detailed structural features, but the reconstructions of particles denoised by coarse denoiser (Figure S10 B2) appears significantly blurred compared to the 3D reconstruction of particles denoised by fine denoiser (Figure S10 B3). The Figure S10 B4 shows the Fourier shell correlation (FSC) curve of these three reconstructions. The density maps reconstructed from raw particles, coarse denoised particles, and fine denoised particles achieve resolutions of 3.12Å, 4.49Å and 3.56Å, respectively. It can be known that compared with the coarse denoiser, the denoised micrographs denoised by fine denoiser retain more image details, that is, high-frequency information. It can also be proved that the noise distribution learned from extracted noise patches is correct and effective for denoising.