RELATED APPLICATION

This application claims priority to PCT Application No. PCT/FI2019/050256, filed on Mar. 29, 2019, which claims priority to Finnish Application No. 20185333, filed on Apr. 9, 2018, each of which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present invention relates to an apparatus, a method and a computer program for video coding and decoding.

BACKGROUND

Recently, the development of various neural network (NN) techniques, especially the ones related to deep learning, has enabled to learn algorithms for several tasks from the raw data, which algorithms may outperform algorithms which have been developed for many years using traditional (non-learning based) methods.

Some neural net approaches have been developed for implementing video/image video coding tools through deep learning. For example, a NN-based intra prediction has been introduced, where a prediction block is generated from neighboring decoded blocks used a neural net. In block-wise reduced resolution coding with neural net-based upsampling, an image block is adaptively encoded either with full spatial resolution or reduced spatial resolution, and a reconstructed reduced resolution block is upsampled to full resolution possibly using a NN-based upsampler.

However, these neural net approaches are focused on enhancing the encoding through improving the operation of the coding tools with a pre-defined neural net. On-line learning approaches for using neural net based video signal enhancement have not been utilized.

SUMMARY

Now in order to at least alleviate the above problems, an enhanced encoding method is introduced herein.

A method according to a first aspect comprises obtaining a block of a picture or a picture in an encoder; determining if the block/picture is used for on-line learning; if affirmative, encoding the block/picture; reconstructing a coarse version of the block/picture or the respective prediction error block/picture; enhancing the coarse version using a neural net; fine-tuning the neural net with a training signal based on the coarse version; determining if the block/picture is enhanced using the neural net; and if affirmative, encoding the block/picture with enhancing using the neural net.

A method according to a second aspect comprises obtaining an encoded block of a picture or an encoded picture in a decoder; determining if the block/picture is used for on-line learning; if affirmative, decoding the block/picture; reconstructing a coarse version of the block/picture or the respective prediction error block/picture; enhancing the coarse version using a neural net; fine-tuning the neural net with a training signal based on the coarse version; determining if the block/picture is enhanced using the neural net; and if affirmative, decoding the block/picture with enhancing using the neural net.

According to an embodiment, the method further comprises reconstructing the block/picture or the respective prediction error block/picture; and fine-tuning the neural net with the training signal based on the reconstructed block/picture or the respective reconstructed prediction error block/picture and on the coarse version.

According to an embodiment, the block/picture is determined to be used for on-line learning based on one or more of the following:a quantization parameter value;block/picture type(s);block/picture prediction type(s);sub-layer identifier associated with the block/picture.

According to an embodiment, the block/picture is determined to be enhanced using the neural net when the block/picture is not used for on-line learning.

According to an embodiment, the method further comprises indicating, in or along the bitstream, which blocks or pictures are used as sources for on-line learning.

According to an embodiment, said reconstructing the coarse version comprises: requantizing reconstructed transform coefficients of the prediction error block/picture; and reconstructing the respective coarse prediction error block/picture.

According to an embodiment, the method further comprises deriving the training signal from a gradient of a loss, where the loss is represented by:a difference of a reconstructed prediction error and a neural net (NN) enhanced reconstructed prediction error, where the enhancement is applied to a coarser-quantized reconstructed prediction error; ora difference of the reconstructed reference block/picture and a NN-enhanced reconstructed reference block/picture, where the enhancement is applied to a coarser-quantized reconstructed prediction error.

According to an embodiment, the method further comprises using the neural net for enhancing the reconstructed prediction error block/picture prior to reconstructing the block/picture by essentially summing up the respective prediction block/picture and the respective prediction error.

Apparatuses according to a third and a fourth aspect comprise at least one processor and at least one memory, said at least one memory stored with code thereon, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform the method.

According to a further aspect, there is provided a computer readable non-transitory storage medium stored with code thereon for use by an apparatus, which when executed by a processor, causes the apparatus to perform the method.

The apparatuses and the computer readable storage mediums stored with code thereon, as described above, are thus arranged to carry out the above methods and one or more of the embodiments related thereto.

Further aspects relate to apparatuses comprising means for performing the above methods and one or more of the embodiments related thereto.

