In this work, we propose the Sparse Deep Scattering Croisé Network (SDCSN) a novel architecture based on the Deep Scattering Network (DSN). The DSN is achieved by cascading  wavelet transform convolutions with a complex modulus and a time-invariant operator. We extend this work by first,
 crossing multiple wavelet family transforms to increase the feature diversity while avoiding any learning. Thus providing a more informative latent representation and benefit from the development of highly specialized wavelet filters over the last decades. Beside, by combining all the different wavelet representations, we reduce the amount of prior information needed regarding the signals at hand.
 Secondly, we develop an optimal thresholding strategy for over-complete filter banks that regularizes the network and controls instabilities such as inherent non-stationary noise in the signal. Our systematic and principled solution sparsifies the latent representation of the network by acting as a local mask distinguishing between activity and noise. Thus, we propose to enhance the DSN by increasing the variance of the scattering coefficients representation as well as improve its robustness with respect to non-stationary noise.
 We show that our new approach is more robust and outperforms the DSN on a bird detection task. Modern Machine Learning focuses on developing algorithms to tackle natural machine perception tasks such as speech recognition, computer vision, recommendation among others. Historically, some of the proposed models were based on well-justified mathematical tools from signal processing such as Fourier analysis. Hand-crafted features were then computed based on those tools and a classifier was trained supervised for the task of interest. However, such theory-guided approaches have become almost obsolete with the growth of computational power and the advent of high-capacity models. As such, over the past decade the standard solution evolved around deep neural networks (DNNs). While providing state-of-the-art performance on many benchmarks, at least two pernicious problems still plague DNNs: First, the absence of stability in the DNN's input-output mapping. This has famously led to adversarial attacks where small perturbations of the input lead to dramatically different outputs. In addition, this lack of control manifests in the detection thresholds (i.e: ReLU bias) of DNNs, rendering them prone to instabilities when their inputs exhibit non-stationary noise and discontinuities. Second, when inputs have low SNR, or classes are unbalanced, the stability of DNNs is cantilevered. A common approach to tackle this difficulty is to increase both the size of the training set and the number of parameters of the network resulting in a longer training time and a costly labeling process. In order to alleviate these issues we propose the use of the DSN by creating a new non-linearity based on continuous wavelet thresholding. Thus our model, inherits the mathematical guarantees intrinsic to the DSN regarding the stability, and improves the control via wavelet thresholding method. Then, in order to produce time-frequency representation that are not biased toward a single wavelet family, we propose to combine diverse wavelet families throughout the network. Increasing the variability of the scattering coefficient, we improve the linearization capability of the DSN and reduce the need of an expert knowledge regarding the choice of specific filter bank with respect to each input signal.The paper is organized as follows: 1.1 and 1.2 are devoted to the related work and contribution of the paper, the section 2 shows the theoretical results, where 2.1 is dedicated to the network architecture and its properties, and 2.2 provides the milestone of our thresholding method, then section 2.3 shows the characterization, via latent representation, of our network on different events by on the Freefield1010 1 audio scenes dataset. Finally, we evaluate our architecture and compare it to the DSN on a bird detection task are shown in 2.4. The appendix in divided into three parts, Appendix A provides both, the pre-requisite and details about building the wavelets dictionary to create our architecture; Appendix B shows additional results on the sparsity of the SDCSN latent representations; Appendix C shows mathematical details and proofs for the over-complete thresholding non-linearity. We presented an extension of the scattering network so that one can leverage multiple wavelet families simultaneously. Via a specific topology, cross family representations are performed carrying crucial information, as we demonstrated experimentally, allowing to significantly outperform standard scattering networks. We then motivated and proposed analytical derivation of an optimal overcomplete basis threhsolding being input adaptive. By providing greater sparsity in the representation but also a measure of filter-bank fitness. Again, we provided experimental validation of the use of our thresholding technique proving the robustness implied by such non-linearity. Finally, the ability to perform active denoising has been demonstrated crucial as we demonstrated that even in large scale setting, standard machine learning approach coupled with the SN fail to discard non-stationary noise. This coupled with the denoising ability of our approach should provide real world application the stability needed for consistent results and prediction control.Among the possible extensions is the one adapting the technique to convolutional neural networks such that it provides robustness with respect to adversarial attacks. Furthermore, using a joint scattering and DNN will inherit the benefits presented with our technique as our layers are the ones closer to the input. Hence, denoising will benefit the inner layers, the unconstrained standard DNN layers. Finally, it is possible to perform more consistent best basis selection a la maxout network. In fact, our thresholding technique can be linked to an optimised ReLU based thresholding. In this scheme, applying best basis selection based on the empirical risk would thus become equivalent to the pooling operator of a maxout network. A BUILDING A DEEP CROISÉ SCATTERING NETWORK A.1 CONTINOUS WAVELET TRANSFORM "By oscillating it resembles a wave, but by being localized it is a wavelet". Yves MeyerWavelets were first introduced for high resolution seismology BID21 and then developed theoretically by Meyer et al. BID14 . Formally, wavelet is a function ψ ∈ L 2 such that: DISPLAYFORM0 it is normalized such that ψ L 2 = 1. There exist two categories of wavelets, the discrete wavelets and the continuous ones. The discrete wavelets transform are constructed based on a system of linear equation. These equations represent the atom's property. These wavelet when scaled in a dyadic fashion form an orthonormal atom dictionary. Withal, the continuous wavelets have an explicit formulation and build an over-complete dictionary when successively scaled. In this work, we will focus on the continuous wavelets as they provide a more complete tool for analysis of signals. In order to perform a time-frequency transform of a signal, we first build a filter bank based on the mother wavelet. This wavelet is names the mother wavelet since it will be dilated and translated in order to create the filters that will constitute the filter bank. Notice that wavelets have a constant-Q property, thereby the ratio bandwidth to center frequency of the children wavelets are identical to the one of the mother. Then, the more the wavelet atom is high frequency the more it will be localized in time. The usual dilation parameters follows a geometric progression and belongs to the following set: DISPLAYFORM1 . Where the integers J and Q denote respectively the number of octaves, and the number of wavelets per octave. In order to develop a systematic and general principle to develop a filter bank for any wavelet family, we will consider the weighted version of the geometric progression mentioned above, that is: DISPLAYFORM2 . In fact, the implementation of wavelet filter bank can be delicate since the mother wavelet has to be define at a proper center frequency such that no artifact or redundant information will appear in the final representation. Thus, in the section A.3 we propose a principled approach that allows the computation of the filter bank of any continuous wavelet. Beside, this re-normalized scaled is crucial to the comparison between different continuous wavelet. Having selected a geometric progression ensemble, the dilated version of the mother wavelet in the time are computed as follows: DISPLAYFORM3 , and can be calculated in the Fourier domain as follows: DISPLAYFORM4 Notice that in practice the wavelets are computed in the Fourier domain as the wavelet transform will be based on a convolution operation which can be achieved with more efficiency. By construction the children wavelets have the same properties than the mother one. As a result, in the Fourier domain:ψ λ = 0, ∀λ ∈ Λ . Thus, to create a filter bank that cover all the frequency support, one needs a function that captures the low frequencies contents. The function is called the scaling function and satisfies the following criteria: DISPLAYFORM5 Finally, we denote by W x, where W ∈ C N * (J * Q)×N is a block matrix such that each block corresponds to the filters at all scales for a given time. Also, we denote by S(W x)(λ, t) the reshape operator such that, DISPLAYFORM6 where ψ is the complex conjugate of ψ λ .
We propose to enhance the Deep Scattering Network  in order to improve control and stability of any given machine learning pipeline by proposing a continuous wavelet thresholding scheme
{"src": "scitldr-AIC_train", "paper_id": "rkpqdGDeM"}