Perceptual Image Processing ALgorithms (PIPAL)
a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration


Jinjin Gu1,2,3, Haoming Cai1,3, Haoyu Chen1, Xiaoxing Ye1, Jimmy S. Ren2, Chao Dong3,

1 The Chinese University of Hong Kong, Shenzhen
2 SenseTime Research
3 Shenzhen Institutes of Advanced Technology

The Image Quality Assessment (IQA) methods are developed to measure the perceptual quality of images. One of the most important applications of IQA is to measure the performance of image restoration algorithms. However, while new algorithms have been continuously improving image restoration performance, we notice an increasing inconsistency between quantitative results and perceptual quality. Especially, the invention of Generative Adversarial Networks (GAN) and GAN-based image restoration algorithms poses a great challenge for IQA, as they bring completely new characteristics to the output images. This also affected the development in the field of image restoration, as comparing them with the flawed IQA methods may not lead to better perceptual quality. In this paper, we contribute a new large-scale IQA dataset and build benchmarks for IQA methods.

Motivation

The most recent image restoration algorithms based on GANs have achieved significant improvement in visual performance, but also presented great challenges for quantitative evaluation. Notably, we observe an increasing inconsistency between perceptual quality and the evaluation results. Then we naturally raise two questions:

  1. Can existing IQA methods objectively evaluate recent IR algorithms?
  2. When focus on beating current benchmarks, are we getting better IR algorithms?

Benchmark

To understand the new challenges brought by these perceptual-driven algorithms, we construct a new IQA dataset that includes the results of these new algorithms. We term this dataset Perceptual Image Processing ALgorithms, PIPAL. PIPAL contains:

Especially, we employ the Elo rating system to assign the Mean Opinion Scores (MOS). We build a new benchmark for both IQA methods and perceptual-driven algorithms based on the proposed dataset. More information is presented in our paper.

Competition

Coming soon.

Download PIPAL

Citation

    @InProceedings{pipal,
      author = {Jinjin Gu, Haoming Cai, Haoyu Chen, Xiaoxing Ye, Jimmy Ren, Chao Dong},
      title = {PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual image Restoration},
      booktitle = {European Conference on Computer Vision (ECCV)},
      year = {2020},
      organization = {Springer}
      }
                    

Acknowledgement: This work is partially supported by SenseTime Group Limited, the National Natural Science Foundation of China (61906184), Science and Technology Service Network Initiative of Chinese Academy of Sciences (KFJ-STS-QYZX-092), Shenzhen Basic Research Program (JSGG20180507182100698, CXB201104220032A), the Joint Lab of CAS-HK, Shenzhen Institute of Artificial Intelligence and Robotics for Society.

This page has been viewed