BACKGROUND

1. Field

This disclosure relates to an electronic apparatus and a controlling method thereof and, more particularly, to an electronic apparatus capable of reconstructing sound quality of audio and a controlling method thereof.

2. Description of Related Art

An artificial intelligence (AI) system is a computer system that implements a human-level intelligence and a system in which a machine learns, judges, and becomes smart, unlike an existing rule-based smart system. As the use of AI systems improves, a recognition rate and understanding or anticipation of a user's taste may be performed more accurately. As such, existing rule-based smart systems are gradually being replaced by deep learning-based AI systems.

AI technology is composed of machine learning (for example, deep learning) and elementary technologies that utilize machine learning.

Machine learning is an algorithm technology that is capable of classifying or learning characteristics of input data. Element technology is a technology that uses machine learning algorithms such as deep learning. Machine learning is composed of technical fields such as linguistic understanding, visual understanding, reasoning, prediction, knowledge representation, motion control, or the like.

Various fields in which AI technology is applied are as shown below. Linguistic understanding is a technology for recognizing, applying, and/or processing human language or characters and includes natural language processing, machine translation, dialogue system, question and answer, voice recognition or synthesis, and the like. Visual understanding is a technique for recognizing and processing objects as human vision, including object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image enhancement, and the like. Inference prediction is a technique for judging and logically inferring and predicting information, including knowledge-based and probability-based inference, optimization prediction, preference-based planning, recommendation, or the like. Knowledge representation is a technology for automating human experience information into knowledge data, including knowledge building (data generation or classification), knowledge management (data utilization), or the like. Motion control is a technique for controlling the autonomous running of the vehicle and the motion of the robot, including motion control (navigation, collision, driving), operation control (behavior control), or the like.

Recently, research has been actively conducted on machine learning, which is an algorithm capable of recognizing objects like humans and understanding information, as big data collection and storage are enabled by development of hardware technology and computer capabilities and techniques for analyzing thereof are becoming more sophisticated and accelerated. In particular, in the machine learning technical field, research on deep learning in an autonomous learning scheme using a neural network has been actively conducted.

The neural network is an algorithm for determining the final output by comparing the activation function to a particular boundary value for the sum which is acquired by multiplying a plurality of inputs by a weight, based on the intent to aggressively mimic the function of the human brain and is generally formed of a plurality of layers. A convolutional neural network (CNN), which is widely used for image recognition, a recurrent neural network (RNN), which is widely used for speech recognition, and the like are representative examples.

The disclosure provides a method for learning audio data using a neural network and reconstructing damaged audio data. When an audio signal is compressed or transmitted, an audio signal of some frequency band may be lost for efficient compression or transmission. The audio signal from which data in some frequency band is lost may have degraded sound quality or changed tone as compared to the audio signal before being lost.

An automobile is a representative location where music is consumed primarily, but due to the expanded use of the compressed/degraded sound source, a user cannot help listening to music with generally degraded sound quality.

Accordingly, if the audio signal including the lost frequency band is to be reproduced to be close to the original sound with a high sound quality, it is required to effectively reconstruct the audio signal in the lost frequency band.

SUMMARY

The disclosure provides an electronic apparatus in which an effective reconstruction is performed so that a user may enjoy a high quality sound even in a compressed or degraded sound source and a method for controlling thereof.

An electronic apparatus according to an embodiment includes a storage for storing a plurality of filters trained in a plurality of convolutional neural networks (CNNs) respectively and a processor configured to acquire a first spectrogram corresponding to a damaged audio signal, input the first spectrogram to a CNN corresponding to each frequency band to apply the plurality of filters trained in the plurality of CNNs respectively, acquire a second spectrogram by merging output values of the CNNs to which the plurality of filters are applied, and acquire an audio signal reconstructed based on the second spectrogram.

The plurality of CNNs include a first CNN into which a first spectrogram of a first frequency band is input and a second CNN into which a first spectrogram of a second frequency band is input, the plurality of filters include a first filter and a second filter trained in the first CNN and a third filter and a fourth filter trained in the second CNN, the first filter and third filter may be trained based on the first frequency band and the second filter and the fourth filter are trained based on the second frequency band, and the processor is configured to acquire a second spectrogram corresponding to the first frequency band by merging output values of the first CNN to which the first filter is applied and output values of the second CNN to which the third filter is applied, and acquire a second spectrogram corresponding to the second frequency band by merging output values of the first CNN to which the second filter is applied and output values of the second CNN to which the fourth filter is applied.

The processor is configured to identify the first spectrogram in a frame unit, group a current frame and a previous frame in a predetermined number to input the grouped frames to the CNN corresponding to each frequency band, and acquire a reconstructed current frame by merging output values of the CNN respectively.

The plurality of CNNs may be included in a first CNN layer, and the processor is configured to acquire the second spectrogram by inputting an output value of the first CNN layer to a second CNN layer comprising a plurality of other CNNs, and a size of a filter included in the second CNN layer is different from a size of a filter included in the first CNN layer.

The processor is configured to input the first spectrogram by the frequency bands to which the plurality of filters are applied to a sigmoid gate respectively, and acquire the second spectrogram by merging the first spectrogram by frequency bands output from the sigmoid gate.

The electronic apparatus may further include an inputter, and the processor is configured to transform the damaged audio signal input through the inputter to the first spectrogram based on time and frequency, and acquire the reconstructed audio signal by inverse transforming the second spectrogram to an audio signal based on time and magnitude.

The processor is configured to acquire a compensated magnitude component by acquiring a magnitude component in the first spectrogram and inputting to corresponding CNNs by frequency bands and acquire the second spectrogram by combining a phase component of the first spectrogram and the compensated magnitude component.

The processor is configured to input a frequency band which is greater than or equal to a predetermined magnitude, among frequency bands of the first spectrogram, to a corresponding CNN.

The processor is configured to normalize and input the first spectrogram to a corresponding CNN by frequency bands, denormalize the second spectrogram, and acquire the reconstructed audio signal based on the denormalized second spectrogram.

According to an embodiment, a method of controlling an electronic apparatus includes acquiring a first spectrogram corresponding to a damaged audio signal, inputting the first spectrogram to a CNN corresponding to each frequency band, applying a plurality of filters respectively trained in the CNN corresponding to each frequency band to the input first spectrogram, acquiring a second spectrogram by merging output valued of the CNNs to which the plurality of filters are applied, and acquiring an audio signal reconstructed based on the second spectrogram.

The plurality of CNNs may include a first CNN into which a first spectrogram of a first frequency band is input and a second CNN into which a first spectrogram of a second frequency band is input, the plurality of filters may include a first filter and a second filter trained in the first CNN and a third filter and a fourth filter trained in the second CNN, the first filter and third filter are trained based on the first frequency band and the second filter and the fourth filter are trained based on the second frequency band, the acquiring the second spectrogram may include acquiring a second spectrogram corresponding to the first frequency band by merging output valued of the first CNN to which the first filter is applied and output valued of the second CNN to which the third filter is applied, and acquiring a second spectrogram corresponding to the second frequency band by merging output valued of the first CNN to which the second filter is applied and output valued of the second CNN to which the fourth filter is applied.

The inputting may include identifying the first spectrogram in a frame unit, grouping a current frame and a previous frame in a predetermined number to input the grouped frames to the CNN corresponding to each frequency band, and the acquiring the second spectrogram may include acquiring a reconstructed current frame by merging output values of the CNN respectively.

The plurality of CNNs may be included in a first CNN layer, and the acquiring the second spectrogram may include acquiring the second spectrogram by inputting an output value of the first CNN layer to a second CNN layer comprising a plurality of other CNNs, and wherein a size of a filter included in the second CNN layer is different from a size of a filter included in the first CNN layer.

The acquiring the second spectrogram may include inputting first spectrogram by the frequency bands to which the plurality of filters are applied to a sigmoid gate respectively, and acquiring the second spectrogram by merging the first spectrogram by frequency bands output from the sigmoid gate.

The controlling method may include receiving a damaged audio signal, transforming the input audio signal to the first spectrogram based on time and frequency, and acquiring the reconstructed audio signal by inverse-transforming the second spectrogram to an audio signal based on time and magnitude.

The inputting may include acquiring a magnitude component in the first spectrogram and inputting to corresponding CNNs by frequency bands, and the acquiring the second spectrogram may include acquiring the second spectrogram by combining the phase component of the first spectrogram with the magnitude component compensated by the CNN.

The inputting may include inputting a frequency band which is greater than or equal to a predetermined magnitude, among frequency bands of the first spectrogram, to a corresponding CNN.

The method may further include normalizing and inputting the first spectrogram to a corresponding CNN by frequency bands, denormalizing the second spectrogram, and acquiring the reconstructed audio signal based on the denormalized second spectrogram.

A non-transitory computer readable medium having stored therein a computer instruction which is executed by a processor of an electronic apparatus to perform the method includes acquiring a first spectrogram corresponding to a damaged audio signal, inputting the first spectrogram to a convolutional neural network (CNN) corresponding to each frequency band, applying a plurality of filters respectively trained in the CNN corresponding to each frequency band to the input first spectrogram, acquiring a second spectrogram by merging output values of the CNNs to which the plurality of filters are applied, and acquiring an audio signal reconstructed based on the second spectrogram.

According to various embodiments, even a sound source degraded due to compression can enable a user to enjoy sound in a level of an original sound, and radio resource waste due to high bandwidth data transmission can be reduced.

