Blind Super-Resolution with Iterative Kernel Correction

Jinjin Gu 1      Hannan Lu 2       Wangmeng Zuo 2      Chao Dong 3     
1 The School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen
2 School of Computer Science and Technology, Harbin Institute of Technology
3 Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences

Abstract


Deep learning based methods have dominated super-resolution (SR) field due to their remarkable performance in terms of effectiveness and efficiency. Most of these methods assume that the blur kernel during downsampling is predefined/known (e.g., bicubic). However, the blur kernels involved in real applications are complicated and unknown, resulting in severe performance drop for the advanced SR methods. In this paper, we propose an Iterative Kernel Correction (IKC) method for blur kernel estimation in blind SR problem, where the blur kernels are unknown. We draw the observation that kernel mismatch could bring regular artifacts (either over-sharpening or over-smoothing), which can be applied to correct inaccurate blur kernels. Thus we introduce an iterative correction scheme – IKC that achieves better results than direct kernel estimation. We further propose an effective SR network architecture using spatial feature transform (SFT) layers to handle multiple blur kernels, named SFTMD. Extensive experiments on synthetic and real-world images show that the proposed IKC method with SFTMD can provide visually favorable SR results and the state-of-the-art performance in blind SR problem.

Contributions

  1. We propose Iterative Kernel Correction (IKC), an intuitive and effective deep learning framework for blur kernel estimation in single image super resolution.
  2. We propose SFTMD, a new non-blind SR network using the spatial feature transform layers for multiple blur kernels. We demonstrate the superior performance of the proposed non-blind SR network.
  3. We test the blind SR performance on both carefully selected blur kernels and real images. Extensive experiments show that the combination of SFTMD and IKC achieves the state-of-the-art performance in blind SR problem.

SR Kernel Mismatch Artifacts

Our method stems from the observation that artifacts caused by kernel mismatch have regular patterns. Specifically, if the input kernel is smoother than the real one, then the output image will be blurry/over- smoothing. Conversely, if the input kernel is sharper than the correct one, then the results will be over-shapened with obvious ringing effects (see the above figure). This asymmetry of kernel mismatch effect provides us an empirical guidance on how to correct an inaccurate blur kernel.

The above figure shows the sensitivity of the SR results to kernel mismatch, where σSR denotes the kernel width used for SR. As shown in the upper-right region of the figure, where the kernel used for SR are sharper than the real one ($\sigma_{SR}<\sigma_{LR}$), the SR results are over-smoothing and the the high frequency textures are significantly blurred. In the lower-left region of the figure, where the kernel used for SR are smoother than the correct one ($\sigma_{SR}>\sigma_{SR}$), the SR results show unnatural ringing artifacts caused by over-enhancing high-frequency edges. In contrast, the results on the diagonal, which use correct blur kernels, look natural without artifacts and blurring.

See our paper for more details.


Iterative Kernel Correction (IKC)

The overall pipeline of the proposed iterative kernel correction method:

The network architecture of the predictor and corrector:

See our paper for more details.


SFTMD

We proposed a new SR model for multiple kernels using spatial feature transform (SFT) layers, namely SFTMD. In SFTMD, the kernel maps influence the output of network by applying an affine transformation to the feature maps in each middle layer by SFT layers. This affine transformation is not involved in the process of input image directly, thus providing better performance.

The architecture of the proposed SFTMD network. The design of the proposed SFT layer is shown in pink box.

See our paper for more details.

Important Declaration

There is an error in the Table 1 of the CVF Openaccess version paper, which has been corrected in the arXiv V2 version. We are very sorry for the inconvenience.

Citation

@InProceedings{gu2019blind,
            author = {Gu, Jinjin and Lu, Hannan and Zuo, Wangmeng and Dong, Chao},
            title = {Blind Super-Resolution With Iterative Kernel Correction},
            booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
            month = {June},
            year = {2019}
            }

Contact

If you have any question, please contact Jinjin Gu at jinjingu@link.cuhk.edu.cn.