BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a super-resolution processing apparatus, a method, and a non-transitory computer readable recording medium storing a program, and particularly to a super-resolution processing apparatus, a method, and a non-transitory computer readable recording medium storing a program that increase a resolution of image data in time and space.

2. Description of the Related Art

A super-resolution processing method of performing super-resolution processing for increasing a resolution of image data uses deep learning in some cases (Dong, C. et al., “Image Super-Resolution Using Deep Convolutional Networks”, 2014, European Conference on Computer Vision (ECCV), Jia, X. et al., “Single Image Super-Resolution Using Multi-Scale Convolutional Neural Network”, [online], 2017, arXiv, [Feb. 19, 2018 search], Internet <URL: https://arxiv.org/abs/1705.05084>, Ren, H. et al., “Image Super Resolution Based on Fusing Multiple Convolution Neural Networks”, 2017, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop, and Shi, W. et al., “Single Image Super-Resolution with Dilated Convolution based Multi-Scale Information Learning Inception Module”, 2017, IEEE International Conference on Image Processing (ICIP)). In the super-resolution processing method using deep learning, it has been proposed to improve super-resolution performance by using input images or intermediate feature amounts of a plurality of resolutions (Jia, X. et al., “Single Image Super-Resolution Using Multi-Scale Convolutional Neural Network”, [online], 2017, arXiv, [Feb. 19, 2018 search], Internet <URL: https://arxiv.org/abs/1705.05084>, Ren, H. et al., “Image Super Resolution Based on Fusing Multiple Convolution Neural Networks”, 2017, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop, and Shi, W. et al., “Single Image Super-Resolution with Dilated Convolution based Multi-Scale Information Learning Inception Module”, 2017, IEEE International Conference on Image Processing (ICIP)).

SUMMARY OF THE INVENTION

In a case where super-resolution processing is performed, a ratio of resolutions of image data before and after the super-resolution processing is often fixed. In the following description, a ratio of a resolution of image data after the super-resolution processing to a resolution of image data before the super-resolution processing is referred to as a scale factor for the super-resolution processing. In a case where a scale factor for the super-resolution processing is fixed, a resolution of image data that can be generated from image data with a certain resolution by the super-resolution processing is uniquely determined by the scale factor. Therefore, in order to generate a plurality of pieces of image data having different resolutions from image data with a certain resolution by the super-resolution processing, it is necessary to provide a plurality of super-resolution processing engines having different scale factors. In this case, since it is necessary to prepare a data set for learning and perform learning for each of the plurality of super-resolution processing engines, there is a problem that a cost required to create the super-resolution processing engine increases.

In the description of Jia, X. et al., “Single Image Super-Resolution Using Multi-Scale Convolutional Neural Network”, [online], 2017, arXiv, [Feb. 19, 2018 search], Internet <URL: https://arxiv.org/abs/1705.05084>, it is disclosed that learning of a model (general model) corresponding to a plurality of scale factors (up-scale factors) is performed. According to the technique disclosed in Jia, X. et al., “Single Image Super-Resolution Using Multi-Scale Convolutional Neural Network”, [online], 2017, arXiv, [Feb. 19, 2018 search], Internet <URL: https://arxiv.org/abs/1705.05084>, although it is possible to generate image data with a resolution corresponding to a scale factor learned in advance, image data with a resolution corresponding to any scale factor other than the scale factor learned in advance cannot be generated. In addition, in the technique described in Jia, X. et al., “Single Image Super-Resolution Using Multi-Scale Convolutional Neural Network”, [online], 2017, arXiv, [Feb. 19, 2018 search], Internet <URL: https://arxiv.org/abs/1705.05084>, sets of low-resolution (LR) and high-resolution (HR) images corresponding to the plurality of scale factors are combined into one, and it is necessary to prepare a data set for learning for each of the plurality of scale factors and perform learning.

The present invention has been made in view of such circumstances, and an object of the present invention is to provide a super-resolution processing apparatus, a method, and a non-transitory computer readable recording medium storing a program capable of generating image data with a resolution corresponding to any scale factor other than a predetermined scale factor in super-resolution processing.

In order to solve the above-mentioned problem, a super-resolution processing apparatus according to a first aspect of the present invention comprises: a conversion unit that performs super-resolution processing on input data to output data having a higher resolution than the input data, in which a resolution ratio of the data output from the conversion unit to the data input to the conversion unit is fixed; a down-sampling unit that performs down-sampling processing on the data input to the conversion unit or the data output from the conversion unit; and a processing unit that adjusts the resolution of the output data by adjusting a sampling rate in the down-sampling processing based on the resolution ratio.

According to the first aspect, by performing down-sampling before or after super-resolution processing, it becomes possible to generate image data with a resolution corresponding to any scale factor other than a scale factor learned in advance.

According to the first aspect, the super-resolution processing apparatus according to a second aspect of the present invention further comprises: a setting unit that sets a required resolution of the output data, in which the processing unit adjusts the sampling rate based on the resolution ratio so that the resolution of the output data matches the required resolution.

In the super-resolution processing apparatus according to a third aspect of the present invention, according to the first or second aspect, the conversion unit consists of a plurality of conversion units different in the resolution ratio from each other, and the processing unit generates the output data by combining pieces of data output from the conversion units.

In the super-resolution processing apparatus according to a fourth aspect of the present invention, according to the third aspect, the processing unit determines a method of combining the pieces of data output from the conversion units based on a correspondence relation between data with a first resolution learned in advance and data with a second resolution having a higher resolution than the data with a first resolution.

In the super-resolution processing apparatus according to a fifth aspect of the present invention, according to any one of the first to fourth aspects, the processing unit adjusts the sampling rate based on a correspondence relation between data with a first resolution learned in advance and data with a second resolution having a higher resolution than the data with a first resolution.

In the super-resolution processing apparatus according to a sixth aspect of the present invention, according to any one of the first to fifth aspects, the conversion unit performs the super-resolution processing based on a correspondence relation between data with a first resolution learned in advance and data with a second resolution having a higher resolution than the data with a first resolution.

In the super-resolution processing apparatus according to a seventh aspect of the present invention, according to any one of the first to sixth aspects, the conversion unit performs processing of increasing a spatial resolution of the data input to the conversion unit.

In the super-resolution processing apparatus according to an eighth aspect of the present invention, according to the seventh aspect, in a case where the data input to the conversion unit is two-dimensional or three-dimensional image data, the conversion unit performs processing of increasing a resolution of the image data in at least one direction.

In the super-resolution processing apparatus according to a ninth aspect of the present invention, according to any one of the first to eighth aspects, in a case where the data input to the conversion unit is moving image data, the conversion unit performs processing of increasing a frame rate of the moving image data.

A super-resolution processing method according to a tenth aspect of the present invention comprises: a conversion step of performing super-resolution processing on input data by a conversion unit to output data having a higher resolution than the input data, in which a resolution ratio of the data output from the conversion unit to the data input to the conversion unit is fixed; a down-sampling step of performing down-sampling processing on the data input to the conversion unit or the data output from the conversion unit; and a processing step of adjusting the resolution of the output data by adjusting a sampling rate in the down-sampling processing based on the resolution ratio.

A super-resolution processing program according to an eleventh aspect of the present invention causes a computer to realize: a conversion function of performing super-resolution processing on input data to output data having a higher resolution than the input data, in which a resolution ratio of the output data to the input data is fixed; a down-sampling function of performing down-sampling processing on the data before conversion by the conversion function or the data after conversion by the conversion function; and a processing function of adjusting the resolution of the output data by adjusting a sampling rate in the down-sampling processing based on the resolution ratio.

According to the present invention, by performing down-sampling before or after super-resolution processing, it becomes possible to generate image data with a resolution corresponding to any scale factor other than a fixed scale factor for the super-resolution processing in a conversion unit.

