TECHNICAL FIELD

The present invention relates to a bandwidth extension method for extending a frequency bandwidth of an audio signal.

BACKGROUND ART

Audio bandwidth extension (BWE) technology is typically used in modern audio codecs to efficiently code wide-band audio signal at low bit rate. Its principle is to use a parametric representation of the original high frequency (HF) content to synthesize an approximation of the HF from the lower frequency (LF) data.

FIG. 1is a diagram showing such a BWE technology-based audio codec. In its encoder, a wide-band audio signal is firstly separated (101&103) into LF and HF part; its LF part is coded (104) in a waveform preserving way; meanwhile, the relationship between its LF part and HF part is analyzed (102) (typically, in frequency domain) and described by a set of HF parameters. Due to the parameter description of the HF part, the multiplexed (105) waveform data and HF parameters can be transmitted to decoder at a low bit rate.

In the decoder, the LF part is firstly decoded (107). To approximate original HF part, the decoded LF part is transformed (108) to frequency domain, the resulting LF spectrum is modified (109) to generate a HF spectrum, under the guide of some decoded HF parameters. The HF spectrum is further refined (110) by post-processing, also under the guide of some decoded HF parameters. The refined HF spectrum is converted (111) to time domain and combined with the delayed (112) LF part. As a result, the final reconstructed wide-band audio signal is outputted.

Note that in the BWE technology, one important step is to generate a HF spectrum from the LF spectrum (109). There are a few ways to realize it, such as copying the LF portion to the HF location, non-linear processing or upsampling.

A most well known audio codec that uses such a BWE technology is MPEG-4 HE-AAC, where the BWE technology is specified as SBR (spectral band replication) or SBR technology, where the HF part is generated by simply copying the LF portion within QMF representation to the HF spectral location.

Such a spectral copying operation, also called as patching, is simple and proved to be efficient for most cases. However, at very low bitrates (e.g. <20 kbits/s mono), where only small LF part bandwidths are feasible, such SBR technology can lead to undesired auditory artifact sensations such as roughness and unpleasant timbre (for example, see Non-Patent Literature (NPL) 1).

Therefore, to avoid such artifacts resulting from mirroring or copying operation presented in low bitrate coding scenario, the standard SBR technology is enhanced and extended with the following main changes (for example, see NPL 2):(1) to modify the patching algorithm from copying pattern to a phase vocoder driven patching pattern(2) to increase adaptive time resolution for post-processing parameters.

As a result of the first modification (aforementioned (1)), by spreading the LF spectrum with multiple integer factors, the harmonic continuity in the HF is ensured intrinsically. In particular, no unwanted roughness sensation due to beating effects can emerge at the border between low frequency and high frequency and between different high frequency parts (for example, see NPL 1).

And the second modification (aforementioned (2)) facilitates the refined HF spectrum to be more adaptive to the signal fluctuations in the replicated frequency bands.

As the new patching preserves harmonic relation, it is named as harmonic bandwidth extension (HBE). The advantages of the prior-art HBE over standard SBR have also been experimentally confirmed for low bit rate audio coding (for example, see NPL 1).

Note that the above two modifications only affect the HF spectrum generator (109), the remaining processes in HBE are identical to those in SBR.

FIG. 2is a diagram showing the HF spectrum generator in the prior art HBE. It should be noted that the HF spectrum generator includes a T-F transform108and a HF reconstruction109. Given a LF part of a signal, suppose its HF spectrum composes of (T−1) HF harmonic patches (each patching process produces one HF patch), from 2ndorder (the HF patch with the lowest frequency) to T-th order (the HF patch with the highest frequency). In prior art HBE, all these HF patches are generated independently in parallel derived from phase vocoders.

As shown inFIG. 2, (T−1) phase vocoders (201˜203) with different stretching factors, (from 2 to k) are employed to stretch the input LF part. The stretched outputs, with different lengths, are bandpass filtered (204˜206) and resampled (207˜209) to generate HF patches by converting time dilatation into frequency extension. By setting stretching factor as two times of resampling factor, the HF patches maintain the harmonic structure of the signal and have the double length of the LF part. Then all HF patches are delay aligned (210˜212) to compensate the potential different delay contributions from the resampling operation. In the last step, all delay-aligned HF patches are summed up and transformed (213) into QMF domain to produce the HF spectrum.

Observing the above HF spectrum generator, it has a high computation amount. The computation amount mainly comes from time stretching operation, realized by a series of Short Time Fourier Transform (STFT) and Inverse Short Time Fourier Transform (ISTFT) transforms adopted in phase vocoders, and the succeeding QMF operation, applied on time stretched HF part.

A general introduction on phase vocoder and QMF transform is described as below.

A phase vocoder is a well-known technique that uses frequency-domain transformations to implement time-stretching effect. That is, to modify a signal's temporal evolution while its local is spectral characteristics are kept unchanged. Its basic principle is described below.

FIG. 3AandFIG. 3Bare diagrams showing the basic principle of time stretching performed by the phase vocoder.

Divide audio into overlap blocks and respace these blocks where the hop size (the time-interval between successive blocks) is not the same at the input and at the output, as illustrated inFIG. 3A. Therein, the input hop size Rais smaller than the output hop size Rs, as a result, the original signal is stretched with a rate r shown in (Equation 1) below.

[Math⁢⁢1]r=RaRs(Equation⁢⁢1)

As shown inFIG. 3B, the respaced blocks are overlapped in a coherent pattern, which requires frequency domain transformation. Typically, input blocks are transformed into frequency, after a proper modification of phases, the new blocks are transformed back to output blocks.

Following the above principle, most classic phase vocoders adopt short time Fourier transform (STFT) as the frequency domain transform, and involve an explicit sequence of analysis, modification and resynthesis for time stretching.

The QMF banks transform time domain representations to joint time-frequency domain representations (and vice versa), which is typically used in parametric-based coding schemes, like the spectral band replication (SBR), parametric stereo coding (PS) and spatial audio coding (SAC), etc. A characteristic of these filter banks is that the complex-valued frequency (subband) domain signals are effectively oversampled by a factor of two. This enables post-processing operations of the subband domain signals without introducing aliasing distortion.

In more detail, given a real valued discrete time signal x(n), with the analysis QMF bank, the complex-valued subband domain signals sk(n) are obtained through (Equation 2) below.

[Math⁢⁢2]sk⁡(n)=∑l=0L-1⁢⁢x⁡(M·n-l)⁢p⁡(l)⁢ej⁢πM⁢(k+0.5)⁢(l+α)(Equation⁢⁢2)

In (Equation 2), p(n) represents a low-pass prototype filter impulse response of order L−1, α represents a phase parameter, M represents the number of bands and k the subband index with k=0, 1, . . . , M−1).

Note that like STFT, QMF transform is also a joint time-frequency transform. That means, it provides both frequency content of a signal and the change in frequency content over time, where the frequency content is represented by frequency subband and timeline is represented by time slot, respectively.

FIG. 4is a diagram showing QMF analysis and synthesis scheme.

In detail, as illustrated inFIG. 4, a given real audio input is divided into successive overlapping blocks with length of L and hopsize of M (FIG. 4 (a)), the QMF analysis process transforms each block into one time slot, composed of M complex subband signals. By this way, the L time domain input samples are transformed into L complex QMF coefficients, composed of L/M time slots and M subbands (FIG. 4 (b)). Each time slot, combined with the previous (L/M−1) time slots, is synthesized by the QMF synthesis process to reconstruct M real time domain samples (FIG. 4 (c)) with near perfect reconstruction.

CITATION LIST

Non Patent Literature

[NPL 1] Frederik Nagel and Sascha Disch, ‘A harmonic bandwidth extension method for audio codecs’, IEEE Int. Conf. on Acoustics, Speech and Signal Proc., 2009

[NPL 2] Max Neuendorf, et al, ‘A novel scheme for low bitrate unified speech and audio coding—MPEG RMO’, in 126thAES Convention, Munich, Germany, May 2009.

SUMMARY OF INVENTION

Technical Problem

A problem associated with the prior-art HBE technology is the high computation amount. The traditional phase vocoder that is adopted by HBE for stretching the signal has a higher computation amount because of applying successive FFTs and IFFTs, that is, successive FFTs (fast Fourier transforms) and IFFTs (inverse fast Fourier transforms); and the succeeding QMF transform increases the computation amount by being applied on the time stretched signal. Furthermore, in general, attempting to reduce the computation amount leads to the potential problem of quality degradation.

Thus, the present invention was conceived in view of the aforementioned problem and has as an object to provide a bandwidth extension method capable of reducing the computation amount in bandwidth extension as well as suppressing quality deterioration in the extended bandwidth.

Solution to Problem

In order to achieve the aforementioned object, the bandwidth extension method according to an aspect of the present invention is a bandwidth extension method for producing a full bandwidth signal from a low frequency bandwidth signal, the method including: transforming the low frequency bandwidth signal into a quadrature mirror filter bank (QMF) domain to generate a first low frequency QMF spectrum; generating pitch-shifted signals by applying different shifting factors on the low frequency bandwidth signal; generating a high frequency QMF spectrum by time-stretching the pitch-shifted signals in a QMF domain; modifying the high frequency QMF spectrum to satisfy high frequency energy and tonality conditions; and generating the full bandwidth signal by combining the modified high frequency QMF spectrum with the first low frequency QMF spectrum.

Accordingly, the high frequency QMF spectrum is generated by time-stretching the pitch-shifted signals in the QMF domain. Therefore, it is possible to avoid the conventional complex processing (successively repeated FFTs and IFFTs, and subsequent QMF transform), for generating the high frequency QMF spectrum, and thus the computation amount can be reduced. Note that like STFT, the QMF transform itself provides joint time-frequency resolution, thus, QMF transform replaces the series of STFT and ISTFT. In addition, in the bandwidth extension method according to an aspect of the present invention, the pitch-shifted signals are generated by applying mutually different shift coefficients instead of only one shift coefficient, and time stretching is performed on these signals, it is possible to suppress deterioration of quality of the high frequency QMF spectrum.

Furthermore, the generating of a high frequency QMF spectrum includes: transforming the pitch shifted signals into a QMF domain to generate QMF spectra; stretching the QMF spectra along a temporal dimension with different stretching factors to generate harmonic patches; time-aligning the harmonic patches; and summing up the time-aligned harmonic patches.

Furthermore, the stretching includes: calculating the amplitude and phase of a QMF spectrum among the QMF spectra; manipulating the phase to produce a new phase; and combining the amplitude with the new phase to generate a new set of QMF coefficients.

Furthermore, in the manipulating, the new phase is produced on the basis of an original phase of a whole set of QMF coefficients.

Furthermore, in the manipulating, manipulation is performed repeatedly for sets of QMF coefficients, and in the combining, new sets of QMF coefficients are generated.

Furthermore, in the manipulating, a different manipulation is performed depending on a QMF subband index.

Furthermore, in the combining, the new sets of QMF coefficients are overlap-added to generate the QMF coefficients corresponding to a temporally-extended audio signal.

Specifically, the time stretching in the bandwidth extension method according to an aspect of the present invention imitates the STFT-based stretching method by modifying phases of input QMF blocks and overlap-adding the modified QMF blocks with different hop size. From the point of view of computation amount, comparing to the successive FFTs and IFFTs in STFT-based method, such time stretching has a lower computation amount by involving only one QMF analysis transform only. Therefore, it is possible to further reduce the computation amount in bandwidth extension.

Furthermore, in order to achieve the aforementioned object, the bandwidth extension method in another aspect of the present invention is a bandwidth extension method for producing a full bandwidth signal from a low frequency bandwidth signal, the method including: transforming the low frequency bandwidth signal into a quadrature mirror filter bank (QMF) domain to generate a first low frequency QMF spectrum; generating a low order harmonic patch by time-stretching the low frequency bandwidth signal in a QMF domain; generating signals that are pitch shifted, by applying different shift coefficients to the low order harmonic patch, and generating a high frequency QMF spectrum from the signals; modifying the high frequency QMF spectrum to satisfy high frequency energy and tonality conditions; and generating the full bandwidth signal by combining the modified high frequency QMF spectrum with the first low frequency QMF spectrum.

Accordingly, the high frequency QMF spectrum is generated by time-stretching and pitch-shifting the low frequency bandwidth signal in the QMF domain. Therefore, it is possible to avoid the conventional complex processing (successively repeated FFTs and IFFTs, and subsequent QMF transform), for generating the high frequency QMF spectrum, and thus the computation amount can be reduced. In addition, since the pitch-shifted signals are generated by applying mutually different shift coefficients instead of only one shift coefficient, and the high frequency QMF spectrum is generated from these signals, it is possible to suppress deterioration of quality of the high frequency QMF spectrum. Furthermore, since the high frequency QMF spectrum is generated from the low order harmonic patch, it is possible to further suppress deterioration of quality of the high frequency QMF spectrum.

It should be noted that, in the bandwidth extension method according to another aspect of the present invention, the pitch shifting also operates in QMF domain. This is in order to decompose the LF QMF subband on the low order patch into multiple sub-subbands for higher frequency resolution, then mapping those sub-subbands into high QMF subband to generate high order patch spectrum.

Furthermore, the generating of a low order harmonic patch includes: transforming the low frequency bandwidth signal into a second low frequency QMF spectrum; bandpassing the second low frequency QMF spectrum; and stretching the bandpassed second low frequency QMF spectrum along a temporal dimension.

Furthermore, the second low frequency QMF spectrum has finer frequency resolution than the first low frequency QMF spectrum.

Furthermore, the generating of signals includes: bandpassing the low order harmonic patch to generate bandpassed patches; mapping each of the bandpassed patches into high frequency to generate high order harmonic patches; and summing up the high order harmonic patches with the low order harmonic patch.

Furthermore, the bandpassing of the low order harmonic patch includes: splitting each QMF subband in each of the bandpassed patches into multiple sub-subbands; mapping the sub-subbands to high frequency QMF subbands; and combining results of the sub-subband mapping.

Furthermore, the mapping of the sub-subbands to high frequency subbands includes: dividing the sub-subbands of each of the QMF subbands into a stop band part and a pass band part; computing transposed center frequencies of the sub-subbands on the pass band part with patch order dependent factor; mapping the sub-subbands on the pass band part into high frequency QMF subbands according to the center frequencies; and mapping the sub-subbands on the stop band part into high frequency QMF subbands according to the sub-subbands of the pass band part.

It should be noted that, in the bandwidth extension method according to the present invention, the process operations (steps) described above may be combined in any manner.

Such a bandwidth extension method as that according to the present invention is a low computation amount HBE technology which uses a computation amount-reduced HF spectrum generator, which contributes the highest computation amount to HBE. To reduce the computation amount, in the bandwidth extension method according to an aspect of the present invention, a new QMF-based phase vocoder that performs time stretching in QMF domain with a low computation amount is used. Furthermore, in the bandwidth extension method according to another aspect of the present invention, to avoid the possible quality problems associated with the solution, a new pitch shifting algorithm is used that generates high order harmonic patches from low order patch in QMF domain.

It is the object of this invention to design a QMF-based patch where time-stretching, or both time-stretching and frequency-extending can be performed in QMF domain, to make it further, to develop a low computation amount HBE technology driven by a QMF-based phase vocoder.

It should be noted that the present invention can be realized, not only as such a bandwidth extension method, but also as a bandwidth extension apparatus and an integrated circuit that extend the frequency bandwidth of an audio signal using the bandwidth extension method, as a program for causing a computer to extend a frequency bandwidth using the bandwidth extension method, and as a recording medium on which the program is recorded.

Advantageous Effects of Invention

The bandwidth extension method in the present invention designs a new harmonic bandwidth extension (HBE) technology. The core of the technology is to do time stretching or both time stretching and pitch shifting in QMF domain, rather than in traditional FFT domain and time domain, respectively. Comparing to the prior-art HBE technology, the bandwidth extension method in the present invention can provide good sound quality and significantly reduce the computation amount.

