24-10-2016, 12:57 PM
[attachment=73752]
Abstract—Existing blind image quality assessment (BIQA)
methods are mostly opinion-aware. They learn regression models
from training images with associated human subjective scores to
predict the perceptual quality of test images. Such opinion-aware
methods, however, require a large amount of training samples
with associated human subjective scores and of a variety of distortion
types. The BIQA models learned by opinion-aware methods
often have weak generalization capability, hereby limiting their
usability in practice. By comparison, opinion-unaware methods
do not need human subjective scores for training, and thus have
greater potential for good generalization capability. Unfortunately,
thus far no opinion-unaware BIQA method has shown consistently
better quality prediction accuracy than opinion-aware
methods. Here we aim to develop an opinion-unaware BIQA
method that can compete with, and perhaps outperform existing
opinion-aware methods. By integrating natural image statistics
features derived from multiple cues, we learn a multivariate
Gaussian model of image patches from a collection of pristine
natural images. Using the learned multivariate Gaussian model, a
Bhattacharyya-like distance is used to measure the quality of each
image patch, then an overall quality score is obtained by average
pooling. The proposed BIQA method does not need any distorted
sample images nor subjective quality scores for training, yet
extensive experiments demonstrate its superior quality-prediction
performance to state-of-the-art opinion-aware BIQA methods.
The Matlab source code of our algorithm is publicly available at
www.comp.polyu.edu.hk/∼cslzhang/IQA/ILNIQE/ILNIQE.htm
INTRODUCTION
I
T is a highly desirable goal to be able to faithfully evaluate
the quality of output images in many applications, such
as image acquisition, transmission, compression, restoration,
enhancement, etc. Quantitatively evaluating an image’s perceptual
quality has been among the most challenging problems of
modern image processing and computational vision research.
Perceptual image quality assessment (IQA) methods fall into
two categories: subjective assessment by humans, and objective
assessment by algorithms designed to mimic the subjective judgments. Though subjective assessment is the ultimate criterion
of an image’s quality, it is time-consuming, cumbersome,
expensive, and cannot be implemented in systems where realtime
evaluation of image or video quality is needed. Hence,
there has been an increasing interest in developing objective
IQA methods that can automatically predict image quality in
a manner that is consistent with human subjective perception.
Early no-reference IQA (NR-IQA) models commonly operated
under the assumption that the image quality is affected by
one or several particular kinds of distortions, such as blockiness
[1], [2], ringing [3], blur [4], [5], or compression [6]–[9].
Such early NR-IQA approaches therefore extract distortionspecific
features for quality prediction, based on a model of
the presumed distortion type(s). Hence, the application scope
of these methods is rather limited.
Recent studies on NR-IQA have focused on the so-called
blind image quality assessment (BIQA) problem, where prior
knowledge of the distortion types is unavailable. A majority
of existing BIQA methods are “opinion aware”, which means
that they are trained on a dataset consisting of distorted images
and associated subjective scores [10]. Representative methods
belonging to this category include [11]–[18] and they share
a similar architecture. In the training stage, feature vectors
are extracted from the distorted images, then a regression
model is learned to map the feature vectors to the associated
human subjective scores. In the test stage, a feature vector is
extracted from the test image and then fed into the learned
regression model to predict its quality score. In [11], Moorthy
and Bovik proposed a two-step framework for BIQA, called
BIQI. In BIQI, given a distorted image, scene statistics are
at first extracted and used to explicitly classify the distorted
image into one of n distortions; then, the same set of statistics
are used to evaluate the distortion-specific quality. Following
the same paradigm, Moorthy and Bovik later extended BIQI to
DIIVINE using a richer set of natural scene features [12]. Both
BIQI and DIIVINE assume that the distortion types in the test
images are represented in the training dataset, which is, however,
not the case in many practical applications. By assuming
that the statistics of DCT features can vary in a predictable
way as the image quality changes, Saad et al. [13] proposed
a BIQA model, called BLIINDS, by training a probabilistic
model based on contrast and structure features extracted in
the DCT domain. Saad et al. later extended BLIINDS to
BLIINDS-II [14] using more sophisticated NSS-based DCT
features. In [15], Mittal et al. used scene statistics of locally
normalized luminance coefficients to quantify possible losses
of naturalness in the image due to the presence of distortions,
and the resulting BIQA model is referred to BRISQUE. The model proposed in [16] extracts three sets of features based
on the statistics of natural images, distortion textures, and
blur/noise; three regression models are trained for each feature
set and finally a weighted combination of them is used to
estimate the image quality.
In [17], Ye et al. proposed an unsupervised feature learning
framework for BIQA, called CORNIA, which consists of
the following major steps: local feature extraction, codebook
construction, soft-assignment coding, max-pooling, and linear
regression. In [18], Li et al. extracted four kinds of features
from images being quality-tested: the mean value of a phase
congruency [19] map computed on an image, the entropy of
the phase congruency map, the entropy of the image, and the
gradient of the image. A generalized regression neural network
(GRNN) [20] was deployed to train the model. In [21],
Zhang and Chandler extracted image quality-related statistical
features in both the spatial and frequency domains. In the
spatial domain, locally normalized pixels and adjacent pixel
pairs were statistically modeled using log-derivative statistics;
and in the frequency domain, log-Gabor filters [22] were used
to extract the fine scales of the image. Based on the observation
that image local contrast features convey important structural
information that is related to image perceptual quality, in [23],
Xue et al. proposed a BIQA model utilizing the joint statistics
of the local image gradient magnitudes and the Laplacian of
Gaussian image responses.
The opinion-aware BIQA methods discussed above require
a large number of distorted images with human subjective
scores to learn the regression model, which causes them to
have rather weak generalization capability. In practice, image
distortion types are numerous and an image may contain
multiple interacting distortions. It is difficult to collect enough
training samples for all such manifold types and combinations
of distortions. If a BIQA model trained on a certain set
of distortion types is applied to a test image containing a
different distortion type, the predicted quality score will be
unpredictable and likely inaccurate. Second, existing trained
BIQA models have been trained on and thus are dependant to
some degree on one of the available public databases. When
applying a model learned on one database to another database,
or to real-world distorted images, the quality prediction performance
can be very poor (refer to Section IV-D for details).
Considering the shortcomings of opinion-aware BIQA
methods, it is of great interest to develop “opinion-unaware”
IQA models, which do not need training samples of distortions
nor of human subjective scores [10]. However, while
the goal of opinion-unaware BIQA is attractive, the design
methodology is more challenging due to the limited available
information. A few salient works have been reported along
this direction. In [24], Mittal et al. proposed an algorithm that
conducts probabilistic latent semantic analysis on the statistical
features of a large collection of pristine and distorted image
patches. The uncovered latent quality factors are then applied
to the image patches of the test image to infer a quality score.
The Natural Image Quality Evaluator (NIQE) model proposed
by Mittal et al. [10] extracts a set of local features from an
image, then fits the feature vectors to a multivariate Gaussian
(MVG) model. The quality of a test image is then predicted
by the distance between its MVG model and the MVG model
learned from a corpus of pristine naturalistic images. However,
since NIQE uses a single global MVG model to describe an
image, useful local image information which could be used
to better predict the image quality is lost. In [25], Xue et
al. simulated a virtual dataset wherein the quality scores of
distorted images are first estimated using the full reference
IQA algorithm FSIM [26]. A BIQA model is then learned from
the dataset by a process of patch based clustering. However,
this “quality aware clustering” (QAC) method is only able to
deal with four commonly encountered types of distortions;
hence, unlike NIQE, QAC is not a “totally blind” BIQA
method.
One distinct property of opinion-unaware BIQA methods
is that they have the potential to deliver higher generalization
capability than their opinion-aware counterparts due to the fact
that they do not depend on training samples of distorted images
and associated subjective quality scores on them. However,
thus far, no opinion-unaware method has shown better quality
prediction power than currently available opinion-aware methods.
Thus, it is of great interest and significance to investigate
whether it is possible to develop an opinion-unaware model
that outperforms state-of-the-art opinion-aware BIQA models.
We make an attempt to achieve the above goal in this paper.
It is commonly accepted that the statistics of a distorted image
will be measurably different from those of pristine images.
We use a variety of existing and new natural scene statistics
(NSS) features computed from a collection of pristine natural
image patches, and like NIQE, fit the extracted NSS features
to an MVG model. This MVG model is therefore deployed
as a pristine reference model against which to measure the
quality of a given test image. On each patch of a test image,
a best-fit MVG model is computed online, then compared
with the learned pristine MVG model. The overall quality
score of the test image is then obtained by pooling the patch
scores by averaging them. We conducted an extensive series
of experiments on large scale public benchmark databases,
and found that the proposed opinion-unaware method exhibits
superior quality prediction performance as compared to stateof-the-art
opinion-aware NR IQA models, especially on the
important cross-database tests that establish generalization
capability.
Our work is inspired by NIQE [10]; however, it performs
much better than NIQE for the following reasons. First, going
beyond the two types of NSS features used in [10], we introduce
three additional types of quality-aware features. Second,
instead of using a single global MVG model to describe the
test image, we fit the feature vector of each patch of the test
image to an MVG model, and compute a local quality score
on it. We believe that integrating multiple carefully selected
quality-aware features that are locally expressed by a local
MVG model yields a BIQA model that more comprehensively
captures local distortion artifacts. We refer to this significantly
improved “completely blind” image quality evaluator by the
monicker Integrated Local NIQE, or IL-NIQE.
The most important message of this paper is: we
demonstrate that “completely blind” opinion-unaware
IQA models can achieve more robust quality prediction performance than opinion-aware models. Such a model and
algorithm can be used in innumerable practical applications.
We hope that these results will encourage both IQA
researchers and imaging practitioners to more deeply
consider the potential of opinion-unaware “completely blind”
BIQA models. To make our results fully reproducible,
the Matlab source code of IL-NIQE and the associated
evaluation results have been made publicly available at
www.comp.polyu.edu.hk/∼cslzhang/IQA/ILNIQE/ILNIQE.htm.
The rest of this paper is organized as follows. Section II
introduces the quality-aware features used in IL-NIQE. Section
III presents the detailed design of the new BIQA index
IL-NIQE. Section IV presents the experimental results, and
Section V concludes the paper.
II. QUALITY-AWARE NSS FEATURES
It has been shown that natural scene statistics (NSS) are
excellent indicators of the degree of quality degradation of
distorted images [10]–[16]. Consequently, NSS models have
been widely used in the design of BIQA algorithms. For
example, parameters of the generalized Gaussian distribution
(GGD) which effectively model natural image wavelet coefficients
and DCT coefficients have been used as features
for quality prediction [11]–[14]. In [16], a complex pyramid
wavelet transform was used to extract similar NSS features.
All of these NSS model based BIQA methods are opinionaware
methods, and all learn a regression model to map the
extracted NSS feature vectors to subjective quality scores.
Previous studies have shown that image quality distortions
are well characterized by features of local structure [27],
contrast [23], [26], [27], multi-scale and multi-orientation
decomposition [21], [28], and color [26]. Using these considerations,
we designed a set of appropriate and effective
NSS features for accomplishing opinion-unaware BIQA. To
characterize structural distortion, we adopt two types of NSS
features (originally proposed in [10]) derived from the distribution
of locally mean subtracted and contrast normalized
(MSCN) coefficients and from the distribution of products
of pairs of adjacent MSCN coefficients. To more effectively
characterize structural distortions and to also capture contrast
distortion, we deploy quality-aware gradient features (see Sect.
II-C). In order to extract quality-related multi-scale and multiorientation
image properties, we use log-Gabor filters and
extract statistical features from the filter responses (see Sect.
II-D). Color distortions are described using statistical features
derived from the image intensity distribution in a logarithmicscale
opponent color space (see Sect. II-E). Overall, five
types of features are employed. Although all of these features
are well known in the NSS literature, we collectively adapt
them for the task of completely-blind BIQA for the first
time. Our experiments demonstrate that the new features can
significantly improve image quality prediction performance
(see Sect. IV-E for details).
A. Statistics of Normalized Luminance
Ruderman [29] pointed out that the locally normalized
luminances of a natural gray-scale photographic image I conform to a Gaussian distribution.
Abstract—Existing blind image quality assessment (BIQA)
methods are mostly opinion-aware. They learn regression models
from training images with associated human subjective scores to
predict the perceptual quality of test images. Such opinion-aware
methods, however, require a large amount of training samples
with associated human subjective scores and of a variety of distortion
types. The BIQA models learned by opinion-aware methods
often have weak generalization capability, hereby limiting their
usability in practice. By comparison, opinion-unaware methods
do not need human subjective scores for training, and thus have
greater potential for good generalization capability. Unfortunately,
thus far no opinion-unaware BIQA method has shown consistently
better quality prediction accuracy than opinion-aware
methods. Here we aim to develop an opinion-unaware BIQA
method that can compete with, and perhaps outperform existing
opinion-aware methods. By integrating natural image statistics
features derived from multiple cues, we learn a multivariate
Gaussian model of image patches from a collection of pristine
natural images. Using the learned multivariate Gaussian model, a
Bhattacharyya-like distance is used to measure the quality of each
image patch, then an overall quality score is obtained by average
pooling. The proposed BIQA method does not need any distorted
sample images nor subjective quality scores for training, yet
extensive experiments demonstrate its superior quality-prediction
performance to state-of-the-art opinion-aware BIQA methods.
The Matlab source code of our algorithm is publicly available at
www.comp.polyu.edu.hk/∼cslzhang/IQA/ILNIQE/ILNIQE.htm
INTRODUCTION
I
T is a highly desirable goal to be able to faithfully evaluate
the quality of output images in many applications, such
as image acquisition, transmission, compression, restoration,
enhancement, etc. Quantitatively evaluating an image’s perceptual
quality has been among the most challenging problems of
modern image processing and computational vision research.
Perceptual image quality assessment (IQA) methods fall into
two categories: subjective assessment by humans, and objective
assessment by algorithms designed to mimic the subjective judgments. Though subjective assessment is the ultimate criterion
of an image’s quality, it is time-consuming, cumbersome,
expensive, and cannot be implemented in systems where realtime
evaluation of image or video quality is needed. Hence,
there has been an increasing interest in developing objective
IQA methods that can automatically predict image quality in
a manner that is consistent with human subjective perception.
Early no-reference IQA (NR-IQA) models commonly operated
under the assumption that the image quality is affected by
one or several particular kinds of distortions, such as blockiness
[1], [2], ringing [3], blur [4], [5], or compression [6]–[9].
Such early NR-IQA approaches therefore extract distortionspecific
features for quality prediction, based on a model of
the presumed distortion type(s). Hence, the application scope
of these methods is rather limited.
Recent studies on NR-IQA have focused on the so-called
blind image quality assessment (BIQA) problem, where prior
knowledge of the distortion types is unavailable. A majority
of existing BIQA methods are “opinion aware”, which means
that they are trained on a dataset consisting of distorted images
and associated subjective scores [10]. Representative methods
belonging to this category include [11]–[18] and they share
a similar architecture. In the training stage, feature vectors
are extracted from the distorted images, then a regression
model is learned to map the feature vectors to the associated
human subjective scores. In the test stage, a feature vector is
extracted from the test image and then fed into the learned
regression model to predict its quality score. In [11], Moorthy
and Bovik proposed a two-step framework for BIQA, called
BIQI. In BIQI, given a distorted image, scene statistics are
at first extracted and used to explicitly classify the distorted
image into one of n distortions; then, the same set of statistics
are used to evaluate the distortion-specific quality. Following
the same paradigm, Moorthy and Bovik later extended BIQI to
DIIVINE using a richer set of natural scene features [12]. Both
BIQI and DIIVINE assume that the distortion types in the test
images are represented in the training dataset, which is, however,
not the case in many practical applications. By assuming
that the statistics of DCT features can vary in a predictable
way as the image quality changes, Saad et al. [13] proposed
a BIQA model, called BLIINDS, by training a probabilistic
model based on contrast and structure features extracted in
the DCT domain. Saad et al. later extended BLIINDS to
BLIINDS-II [14] using more sophisticated NSS-based DCT
features. In [15], Mittal et al. used scene statistics of locally
normalized luminance coefficients to quantify possible losses
of naturalness in the image due to the presence of distortions,
and the resulting BIQA model is referred to BRISQUE. The model proposed in [16] extracts three sets of features based
on the statistics of natural images, distortion textures, and
blur/noise; three regression models are trained for each feature
set and finally a weighted combination of them is used to
estimate the image quality.
In [17], Ye et al. proposed an unsupervised feature learning
framework for BIQA, called CORNIA, which consists of
the following major steps: local feature extraction, codebook
construction, soft-assignment coding, max-pooling, and linear
regression. In [18], Li et al. extracted four kinds of features
from images being quality-tested: the mean value of a phase
congruency [19] map computed on an image, the entropy of
the phase congruency map, the entropy of the image, and the
gradient of the image. A generalized regression neural network
(GRNN) [20] was deployed to train the model. In [21],
Zhang and Chandler extracted image quality-related statistical
features in both the spatial and frequency domains. In the
spatial domain, locally normalized pixels and adjacent pixel
pairs were statistically modeled using log-derivative statistics;
and in the frequency domain, log-Gabor filters [22] were used
to extract the fine scales of the image. Based on the observation
that image local contrast features convey important structural
information that is related to image perceptual quality, in [23],
Xue et al. proposed a BIQA model utilizing the joint statistics
of the local image gradient magnitudes and the Laplacian of
Gaussian image responses.
The opinion-aware BIQA methods discussed above require
a large number of distorted images with human subjective
scores to learn the regression model, which causes them to
have rather weak generalization capability. In practice, image
distortion types are numerous and an image may contain
multiple interacting distortions. It is difficult to collect enough
training samples for all such manifold types and combinations
of distortions. If a BIQA model trained on a certain set
of distortion types is applied to a test image containing a
different distortion type, the predicted quality score will be
unpredictable and likely inaccurate. Second, existing trained
BIQA models have been trained on and thus are dependant to
some degree on one of the available public databases. When
applying a model learned on one database to another database,
or to real-world distorted images, the quality prediction performance
can be very poor (refer to Section IV-D for details).
Considering the shortcomings of opinion-aware BIQA
methods, it is of great interest to develop “opinion-unaware”
IQA models, which do not need training samples of distortions
nor of human subjective scores [10]. However, while
the goal of opinion-unaware BIQA is attractive, the design
methodology is more challenging due to the limited available
information. A few salient works have been reported along
this direction. In [24], Mittal et al. proposed an algorithm that
conducts probabilistic latent semantic analysis on the statistical
features of a large collection of pristine and distorted image
patches. The uncovered latent quality factors are then applied
to the image patches of the test image to infer a quality score.
The Natural Image Quality Evaluator (NIQE) model proposed
by Mittal et al. [10] extracts a set of local features from an
image, then fits the feature vectors to a multivariate Gaussian
(MVG) model. The quality of a test image is then predicted
by the distance between its MVG model and the MVG model
learned from a corpus of pristine naturalistic images. However,
since NIQE uses a single global MVG model to describe an
image, useful local image information which could be used
to better predict the image quality is lost. In [25], Xue et
al. simulated a virtual dataset wherein the quality scores of
distorted images are first estimated using the full reference
IQA algorithm FSIM [26]. A BIQA model is then learned from
the dataset by a process of patch based clustering. However,
this “quality aware clustering” (QAC) method is only able to
deal with four commonly encountered types of distortions;
hence, unlike NIQE, QAC is not a “totally blind” BIQA
method.
One distinct property of opinion-unaware BIQA methods
is that they have the potential to deliver higher generalization
capability than their opinion-aware counterparts due to the fact
that they do not depend on training samples of distorted images
and associated subjective quality scores on them. However,
thus far, no opinion-unaware method has shown better quality
prediction power than currently available opinion-aware methods.
Thus, it is of great interest and significance to investigate
whether it is possible to develop an opinion-unaware model
that outperforms state-of-the-art opinion-aware BIQA models.
We make an attempt to achieve the above goal in this paper.
It is commonly accepted that the statistics of a distorted image
will be measurably different from those of pristine images.
We use a variety of existing and new natural scene statistics
(NSS) features computed from a collection of pristine natural
image patches, and like NIQE, fit the extracted NSS features
to an MVG model. This MVG model is therefore deployed
as a pristine reference model against which to measure the
quality of a given test image. On each patch of a test image,
a best-fit MVG model is computed online, then compared
with the learned pristine MVG model. The overall quality
score of the test image is then obtained by pooling the patch
scores by averaging them. We conducted an extensive series
of experiments on large scale public benchmark databases,
and found that the proposed opinion-unaware method exhibits
superior quality prediction performance as compared to stateof-the-art
opinion-aware NR IQA models, especially on the
important cross-database tests that establish generalization
capability.
Our work is inspired by NIQE [10]; however, it performs
much better than NIQE for the following reasons. First, going
beyond the two types of NSS features used in [10], we introduce
three additional types of quality-aware features. Second,
instead of using a single global MVG model to describe the
test image, we fit the feature vector of each patch of the test
image to an MVG model, and compute a local quality score
on it. We believe that integrating multiple carefully selected
quality-aware features that are locally expressed by a local
MVG model yields a BIQA model that more comprehensively
captures local distortion artifacts. We refer to this significantly
improved “completely blind” image quality evaluator by the
monicker Integrated Local NIQE, or IL-NIQE.
The most important message of this paper is: we
demonstrate that “completely blind” opinion-unaware
IQA models can achieve more robust quality prediction performance than opinion-aware models. Such a model and
algorithm can be used in innumerable practical applications.
We hope that these results will encourage both IQA
researchers and imaging practitioners to more deeply
consider the potential of opinion-unaware “completely blind”
BIQA models. To make our results fully reproducible,
the Matlab source code of IL-NIQE and the associated
evaluation results have been made publicly available at
www.comp.polyu.edu.hk/∼cslzhang/IQA/ILNIQE/ILNIQE.htm.
The rest of this paper is organized as follows. Section II
introduces the quality-aware features used in IL-NIQE. Section
III presents the detailed design of the new BIQA index
IL-NIQE. Section IV presents the experimental results, and
Section V concludes the paper.
II. QUALITY-AWARE NSS FEATURES
It has been shown that natural scene statistics (NSS) are
excellent indicators of the degree of quality degradation of
distorted images [10]–[16]. Consequently, NSS models have
been widely used in the design of BIQA algorithms. For
example, parameters of the generalized Gaussian distribution
(GGD) which effectively model natural image wavelet coefficients
and DCT coefficients have been used as features
for quality prediction [11]–[14]. In [16], a complex pyramid
wavelet transform was used to extract similar NSS features.
All of these NSS model based BIQA methods are opinionaware
methods, and all learn a regression model to map the
extracted NSS feature vectors to subjective quality scores.
Previous studies have shown that image quality distortions
are well characterized by features of local structure [27],
contrast [23], [26], [27], multi-scale and multi-orientation
decomposition [21], [28], and color [26]. Using these considerations,
we designed a set of appropriate and effective
NSS features for accomplishing opinion-unaware BIQA. To
characterize structural distortion, we adopt two types of NSS
features (originally proposed in [10]) derived from the distribution
of locally mean subtracted and contrast normalized
(MSCN) coefficients and from the distribution of products
of pairs of adjacent MSCN coefficients. To more effectively
characterize structural distortions and to also capture contrast
distortion, we deploy quality-aware gradient features (see Sect.
II-C). In order to extract quality-related multi-scale and multiorientation
image properties, we use log-Gabor filters and
extract statistical features from the filter responses (see Sect.
II-D). Color distortions are described using statistical features
derived from the image intensity distribution in a logarithmicscale
opponent color space (see Sect. II-E). Overall, five
types of features are employed. Although all of these features
are well known in the NSS literature, we collectively adapt
them for the task of completely-blind BIQA for the first
time. Our experiments demonstrate that the new features can
significantly improve image quality prediction performance
(see Sect. IV-E for details).
A. Statistics of Normalized Luminance
Ruderman [29] pointed out that the locally normalized
luminances of a natural gray-scale photographic image I conform to a Gaussian distribution.