CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit of the filing date of Chinese Patent Application No. 201910124973.9 filed on Feb. 19, 2019, the disclosure of which is hereby incorporated in its entirety by reference.

TECHNICAL FIELD

The present application relates to the field of image processing technologies, and in particular, to an image processing method, an apparatus, an equipment and a storage medium.

BACKGROUND

Image processing is a method to manipulate images through some procedures to change their visual content, analyze or extract information. Nowadays, image processing generally refers to digital image processing that relies on digital signal processing and computational algorithms to process images.

BRIEF SUMMARY

An embodiment of the present disclosure provides an image processing method. The image processing method may include: inputting a first image and a third image to a pre-trained style transfer network model, the third image being a composited image formed by the first image and a second image; extracting content features of the third image and style features of the second image, normalizing the content features of the third image based on the style features of the second image to obtain target image features, and generating a target image based on the target image features and outputting the target image by using the pre-trained style transfer network model.

Optionally, the pre-trained style network model may include an encoder, a normalizing network and a decoder that are connected in cascade. Extracting the content features of the third image and the style features of the second image, normalizing the content features of the third image based on the style features of the second image to obtain the target image features, and generating the target image based on the target image features and outputting the target image by using the pre-trained style transfer network model, may include: extracting the content features of the third image and the style features of the second image by the encoder; normalizing the content features of the third image based on the style features of the second image to obtain the target image features by the normalizing network; and generating the target image based on the target image features and outputting the target image by the decoder.

Optionally, the encoder may be a first visual geometry group network, including at least one convolutional block, a coding pooling layer, and a coding full connection layer; which are sequentially connected in cascade, the at least one convolutional block comprising a first convolutional block comprising at least one convolutional layer connected in cascade, and other convolutional blocks each comprising a pooling layer and at least one convolutional layer connected in cascade; the normalizing network may include an adaptive instance normalization processing layer, and the coding full connection layer may be connected to the adaptive instance normalization processing layer; and the decoder may be a residual network, a dense network, or a network symmetric with the encoder.

Optionally, the decoder may be the network symmetric with the encoder. The decoder may include a decoding full connection layer, a decoding pooling layer, and at least one deconvolution block, which are sequentially connected in cascade; the at least one deconvolution block may include a last deconvolution block comprising at least one deconvolutional layer of concatenation, and a resolution of the target image output by the last deconvolution block being the same as a resolution of the second image or the third image inputted into the encoder, and other deconvolution blocks comprising at least one deconvolutional layer and a pooling layer connected in cascade.

Optionally, the style transfer network model may be pre-trained by: inputting at least one set of a first sample image and a third sample image, the third sample image being formed by compositing the first sample image and a second sample image; and iteratively training an initial style transfer network model according to at least one set of the third sample image and the second sample image until a preset convergence condition is satisfied.

Optionally, each iterative training may include: inputting the current set of the third sample image and the second sample image into the initial style transfer network model or a style transfer network model trained in the previous iteration; extracting content features of the third sample image and style features of the second sample image by using the initial style transfer network model or the style transfer network model trained in the previous iteration, normalizing the content features of the third sample image based on the style features of the second sample image to obtain sample image features, and generating a corresponding result image based on the sample image features; determining a total image loss of the result image based on the second sample image, the sample image features, and the result image; correcting a weight parameter in the initial style transfer network model or the style transfer network model trained in the previous iteration to obtain the style transfer network model trained by the iteration based on the total image loss.

Optionally, determining the total image loss of the result image based on the second sample image, the sample image features, and the result image may include: inputting the second sample image, the sample image features, and the result image into an analysis network model; extracting the style features of the second sample image and result image features of the result image by the analysis network model; and inputting the resulting image features, the sample image features, and the style features of the second sample image into a preset loss function, and determining the total image loss by the loss function.

Optionally, the analysis network model may include a second visual geometry group network, including at least one convolution block, a coding pooling layer, and a coding full connection layer; the at least one convolution block, the coding pooling layer and the coding full connection layer may be sequentially connected in cascade, the at least one convolution block comprises a first convolution block including at least one cascaded convolution layer, and other convolution blocks each including a pooling layer and at least one convolution layer connected in cascade.

Optionally, inputting the result image features, the sample image features, and the style features of the second sample image into the preset loss function, and determining the total image loss by the loss function may include: inputting the result image features and the sample image features into a first loss function in the loss function, determining a difference between the result image features and the sample image features by using the first loss function, and determining a content loss of the result image based on the difference; inputting the result image features and the style features of the second sample image into a second loss function in the loss function, and determining a mean difference and a variance difference between the result image features and the style features of the second sample image by the second loss function, and determining a style loss of the result image based on the mean difference and the variance difference; and determining the total image loss based on the content loss, a weight of the content loss, the style loss, and a weight of the style loss.

Optionally, correcting the weight parameter in the initial style transfer network model or the style transfer network model trained in the previous iteration may include modifying weight parameter of the decoder in the initial style transfer network model or the style transfer network model trained in the last iteration.

Optionally, normalizing the content features of the third image based on the style features of the second image to obtain the target image features may include: adjusting a numerical range of the content features of the third image based on the mean and variance of the content features of the third image, so that the content features of the third image match the style features of the second image; and adding the style features of the second image to the content feature of the adjusted third image to obtain the target image features.

Optionally, the first image may have a first style features, the second image may have a second style features, the first style features and the second style features may be different.

Optionally, the second image may be an element copied from a photo, the first image may be a drawing.

Another embodiment of the present disclosure provides a neural network for smart image pasting. A first image and a third image may be inputted into the neural network for smart image pasting, the third image may be a composited image formed by the first image and a second image. The neural network for smart image pasting may include: an encode configured to extract content features of the third image and style features of the second image; a normalizing network configured to normalize the content features of the third image based on the style features of the second image to obtain target image features; a decoder configured to generate the target image based on the target image features and outputting the target image.

Optionally, the encoder may be a first visual geometry group network, including at least one convolutional block, a coding pooling layer, and a coding full connection layer; which may be sequentially connected in cascade, the at least one convolutional block comprising a first convolutional block comprising at least one convolutional layer connected in cascade, and other convolutional blocks each comprising a pooling layer and at least one convolutional layer connected in cascade; the normalizing network may include an adaptive instance normalization processing layer, and the coding full connection layer may be connected to the adaptive instance normalization processing layer; and the decoder may be a residual network, a dense network, or a network symmetric with the encoder.

Optionally, the first visual geometry group network may be a VGG16 network or a VGG19 network.

Optionally, the neural network for smart image pasting may further include: a model training module. The model training module may be configured to train the neural network for smart image pasting by: inputting at least one set of a first sample image and a third sample image, the third image being a composited image formed by the first image and a second image; performing iterative training on an initial neural network for smart image pasting based on at least one set of the third sample image and the second sample image until a preset convergence condition is satisfied.

Optionally, each iteration of training may include: inputting the current set of the third sample image and the second sample image into the initial neural network for smart image pasting or a neural network for smart image pasting trained in the previous iteration; extracting content features of the third sample image and style features of the second sample image by using the initial neural network for smart image pasting or the neural network for smart image pasting trained in the previous iteration, normalizing the content features of the third sample image based on the style features of the second sample image to obtain sample image features, and generating a corresponding result image based on the sample image features; determining a total image loss of the result image based on the second sample image, the sample image features, and the result image; and correcting a weight parameter in the initial neural network for smart image pasting or the neural network for smart image pasting trained in the previous iteration to obtain the neural network for smart image pasting trained by the iteration based on the total image loss.

Optionally, the model training module may include an analysis network model. The analysis network model may be configured to input the second sample image, the sample image features and the result image, extract the style features of the second sample image and result image features of the result image, and input the resulting image features, the sample image features and the style features of the second sample image into a loss function, and determine the total image loss by the loss function.

Optionally, the analysis network model may include a second visual geometry group network, including at least one convolution block, a coding pooling layer, and a coding full connection layer; the at least one convolution block, the coding pooling layer and the coding full connection layer are sequentially connected in cascade, the at least one convolution block comprises a first convolution block including at least one cascaded convolution layer, and other convolution blocks each including a pooling layer and at least one convolution layer connected in cascade.

Another embodiment of the present disclosure provides an image processing equipment. The image processing equipment may include: a memory and a processor, the memory storing a computer program. The computer program is executed by the processor to implement the image processing method.

Another embodiment of the present disclosure provides a computer readable storage medium, which storing a computer program, the computer program being executed by a processor to implement the image processing method.

