CROSS-REFERENCE TO RELATED APPLICATIONS

The present disclosure claims priority to Chinese Patent Application 202111292710.2, entitled “Method and apparatus for editing audio, electronic device and storage medium”, filed on Nov. 3, 2021, the entire content of which is incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to a field of speech processing, in particular to a method and an apparatus for editing audio, an electronic device and a storage medium.

BACKGROUND OF THE INVENTION

The rapid development of the Internet has accelerated the dissemination of information. There are various media for learning, entertainment and communication, such as movies, videos, interactive online education and so on. The production of these media is often inseparable from audio content editing. A typical audio content editing interface displays visualization of speech, such as waveforms and/or spectrograms, and provides users with standard selecting, cutting, copying, pasting and volume adjustment, which are applied to the waveform itself, and also supports some advanced operations, such as time stretching, pitch bending and denoising. These tools provide great convenience for media producers and have a wide range of application scenarios.

Some systems allow editors to select, cut, and paste the text corresponding to the audio, and change the audio content accordingly, which is called text-based audio content editing. Traditional text-based audio content editing technology is inseparable from manual copying and pasting, which will lead to the discontinuity of modified fundamental frequency, and it is difficult to make a natural connection between synthesized speech and context, which will lead to a phenomenon that it sounds apparently unnatural. The bigger problem is that the above editing techniques do not support the ability to synthesize new words that do not appear in the transcribed text. For a person, it is easy to type a new word that does not appear in the corpus, but it is not easy to synthesize the corresponding pronunciation of the new word through the editor. Of course, new speech may be recorded, but it needs to access the original speaker, which makes the audio content editing process very difficult. With the rapid development of deep learning in speech generation tasks, machine-synthesized speech may be comparable to human beings. However, these tasks are to synthesize the corresponding speech for a given text, but may not edit specific words in the synthesis.

SUMMARY OF THE INVENTION

In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a method and an apparatus for editing audio, an electronic device and a storage medium.

In a first aspect, embodiments of the present disclosure provide a method for editing audio, including steps of:

acquiring a modified text obtained by modifying a known original text of an audio to be edited according to a known text for modification;

predicting a duration of an audio corresponding to the text for modification;

adjusting a region to be edited of the audio to be edited according to the duration of the audio corresponding to the text for modification, to obtain an adjusted audio to be edited;

based on a pre-trained audio editing model, obtaining, an edited audio according to the adjusted audio to be edited and the modified text.

Preferably, acquiring the modified text obtained by modifying the known original text of the audio to be edited according to the known text for modification includes:

obtaining the modified text by adding the known text for modification to the known original text of the audio to be edited, and/or

obtaining the modified text by replacing part of words in the original text of the audio to be edited with the known text for modification.

Preferably, predicting the duration of the audio corresponding to the text for modification is implemented by a pre-trained duration prediction model.

Preferably, the type of the text for modification comprises at least one of adding and replacing, adjusting the region to be edited of the audio to be edited according to the duration of the audio corresponding to the text for modification, to obtain the adjusted audio to be edited, includes:

adding a masked part at a designated position of the region to be edited, and ensuring that the masked part is consistent in the duration with the audio corresponding to the text for modification; and/or

masking the audio corresponding to the text to be replaced in the region to be edited to obtain the masked part, and adding or deleting frames at the masked part to make the masked part consistent in the duration with the audio corresponding to the edited text;

regarding the audio to be edited which carries the masked part as the adjusted audio to be edited.

Preferably, based on the pre-trained audio editing model, obtaining the edited audio according to the adjusted audio to be edited and the modified text includes:

extracting acoustic features from the adjusted audio to be edited;

inputting the extracted acoustic features and the modified text into a pre-trained coarse decoder to obtain coarse acoustic features corresponding to a masked part of the extracted acoustic features;

inputting the coarse acoustic features and the acoustic features after masking into a pre-trained fine decoder to obtain fine acoustic features corresponding to the masked part;

integrating and then inputting the fine acoustic features with the extracted acoustic features into a vocoder to obtain the edited audio.

Preferably, the audio editing model is trained by steps of:

extracting the original acoustic features from the known audio;

masking part of the region of the original acoustic features to obtain the acoustic features after masking;

training a known neural network model by using the acoustic features after masking, the text of the audio and the original acoustic features to obtain the audio editing model.

Preferably, the neural network model comprises a coarse decoder and a fine decoder, training the known neural network model by using the acoustic features after masking, the text of the audio and the original acoustic features to obtain the audio editing model, includes:

inputting the acoustic features after masking and the text of the known audio into the coarse decoder to obtain the coarse acoustic features corresponding to the masked part of the acoustic features after masking;

inputting the coarse acoustic features and the acoustic features after masking into the fine decoder to obtain the fine acoustic features corresponding to the masked part;

comparing the fine acoustic features with the original acoustic features;

when the difference between the fine acoustic features and the original acoustic feature is less than or equal to a preset threshold value, the current series structure of the coarse decoder and the fine decoder is regarded as the audio editing model;

when the difference between the fine acoustic features and the original acoustic feature is more than the preset threshold value, the parameters of the current coarse decoder and the fine decoder are adjusted until the difference between the fine acoustic features and the original acoustic features is less than or equal to the preset threshold value.

Preferably, inputting the extracted acoustic features and the modified text into the pre-trained coarse decoder to obtain coarse acoustic features corresponding to the masked part of the extracted acoustic features is represented by the following expression, which includes:

P⁡(ycoarse⁢❘"\[LeftBracketingBar]"x;θ)=∏t=1T′P⁡(yt⁢❘"\[LeftBracketingBar]"x,ymask;θ)

wherein, ycoarserepresents a coarse acoustic feature, ymaskrepresents the input acoustic features after masking, x represents the text information,θ represents the parameters that the model needs to learn, ytrepresents the acoustic parameters of the t-th frame.

Preferably, inputting the coarse acoustic features and the acoustic features after masking into the pre-trained fine decoder to obtain fine acoustic features corresponding to the masked part is represented by the following expression, which comprises:

P⁡(yfine⁢❘"\[LeftBracketingBar]"x;θ)=∏t=1T′P⁡(yt⁢❘"\[LeftBracketingBar]"ycoarse+ymask;θ)

wherein, yfinerepresents a fine acoustic feature, ycoarserepresents a coarse acoustic feature, ymaskrepresents the input acoustic features after masking, x represents the text information, θ represents the parameters that the model needs to learn, ytrepresents the acoustic parameters of the t-th frame.

In a second aspect, embodiments of the present disclosure provide an apparatus for editing audio, including steps of:

a modification module configured to acquire a modified text obtained by modifying a known original text of an audio to be edited according to a known text for modification;

a prediction module configured to predict the duration of an audio corresponding to the text for modification;

an adjustment module configured to adjust a region to be edited of the audio to be edited according to the duration of the audio corresponding to the text for modification, to obtain an adjusted audio to be edited;

an editing module configured to obtain, based on a pre-trained audio editing model, an edited audio according to the adjusted audio to be edited and the modified text.

In a third aspect, embodiments of the present disclosure provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, where, the processor, the communication interface and the memory communicate with each other through the communication bus;

the memory is for storing computer programs, and

the processor is for implementing steps of the method for editing audio according to the present disclosure when executing the programs stored on the memory.

In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium with a computer program stored thereon, wherein the computer program, when executed by a processor, implements steps of the method for editing audio according to the present disclosure.

Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

The method provided by the embodiments of the present disclosure, acquiring a modified text obtained by modifying a known original text of an audio to be edited according to a known text for modification; predicting the duration of an audio corresponding to the text for modification; adjusting a region to be edited of the audio to be edited according to the duration of the audio corresponding to the text for modification, to obtain an adjusted audio to be edited; obtaining, based on a pre-trained audio editing model, an edited audio according to the adjusted audio to be edited and the modified text. In the present disclosure, the edited audio obtained by the audio editing model sounds natural in the context, and supports the function of synthesizing new words that do not appear in the corpus.

