07-11-2016, 10:09 AM
1465803721-06967800.pdf (Size: 3.99 MB / Downloads: 5)
Abstract— Image mosaicking applications require both geometrical
and photometrical registrations between the images that
compose the mosaic. This paper proposes a probabilistic color
correction algorithm for correcting the photometrical disparities.
First, the image to be color corrected is segmented into several
regions using mean shift. Then, connected regions are extracted
using a region fusion algorithm. Local joint image histograms of
each region are modeled as collections of truncated Gaussians
using a maximum likelihood estimation procedure. Then, local
color palette mapping functions are computed using these sets of
Gaussians. The color correction is performed by applying those
functions to all the regions of the image. An extensive comparison
with ten other state of the art color correction algorithms is
presented, using two different image pair data sets. Results show
that the proposed approach obtains the best average scores in
both data sets and evaluation metrics and is also the most robust
to failures.
INTRODUCTION
I
MAGE mosaicking and other similar variations such as
image compositing and stitching have found a vast field
of applications ranging from satellite or aerial imagery [32] to
medical imaging [8], street view maps [36], city 3D modelling
[22], super-resolution [3], texture synthesis [17] or
stereo reconstruction [19], to name a few. In general, whenever
merging two or more images of the same scene is required
for comparison or integration purposes, a mosaic is built. Two
problems are involved in the computation of an image mosaic:
the geometric and the photometric correspondence
The geometric correspondence is usually referred to as image
registration and is the process of overlaying two or more
images of the same scene taken at different times, possibly
from different viewpoints and by different sensors.
The procedure geometrically aligns the images [45]. This
problem has been extensively studied and is out of the scope
of the current paper. In this paper it is assumed that the
given images are geometrically registered. It should be noted,
however, that in most cases the alignment that is produced
by a registration method is never accurate to the pixel level.
Hence, a pixel to pixel direct mapping of color is not a feasible
solution. On the other hand, the photometrical correspondence
between images deals with the photometrical alignment of
image capturing devices. The same object, under the same
lighting conditions, should be represented by the same color
in two different images. However, even in sets of images
taken from the same camera, the colors representing an object
may differ from picture to picture. This poses a problem to
the fusion of information from several images. Hence, the
problem of how to balance the color of one picture so that
it matches the color of another must be tackled. This process
of photometrical alignment or calibration is referred to as color
correction between images.
Although there are several methods proposed to deal with
color correction, most involve strong assumptions, which
are in general, difficult to fulfill in complex environments.
Furthermore, the size of the data sets used for evaluating
their performance is relatively small. In fact, most proposed
approaches show results just for the few images depicted in the
paper and compare them to the baseline approach from [29].
The current paper proposes a new color correction algorithm
that presents several technical novelties when compared to the
state of the art: (i) the usage of truncated Gaussians to model
more accurately the color distribution; (ii) the cross modelling
of mapping probabilities followed by the fusion of the set of
Gaussians, enabling a more consistent inference of the color
mapping functions inclusively for mappings of colors that are
not observed; and (iii) a methodology to perform the expansion
of the color palette mapping functions to the non overlapping
regions of the images. To the best of our knowledge, this
paper also presents one of the most complete evaluations of
color correction algorithms for image mosaicking published in
the literature. An extensive comparison, which includes nine
other approaches, two datasets with over sixty image pairs and
two distinct evaluation metrics, is presented. It shows that the
proposed color correction algorithm achieves very good results
when compared to state of the art algorithms. The remainder
of the paper is organized as follows: Section II formulates
the problem of color correction; Section III presents the state of the art; the proposed approach is presented in Section IV;
results and conclusions are given in Sections V and VI.
II. PROBLEM FORMULATION
The general problem of compensating the photometrical
disparities between two coarsely geometrically registered
images is known as color correction. In other words, color
correction is the problem of adjusting the color palette of
an image using information from the color palette of another
image. Let two images of the same scene be referred to as
source (S, Fig. 1(a)) and target (T, Fig. 1(e)) images. Now
let a geometric registration between these images be given, so
that it is possible to build a mosaic of the scene. Note that in
the overlapping area of the mosaic in Fig. 1 (f ), the color of
a pixel is given by the average of the S and T image pixels.
Nonetheless, a clear photometrical miss-registration between
the images in the overlapping area can be appreciated. Color
correction tackles the problem of how to adjust the colors
of T, so that they resemble to the colors in S. The color
adjusted T image is called corrected image and is referred
to as Tˆ. A mosaic built using S and Tˆ should present smooth
color transitions, as in the example of Fig. 1(g). Let two new
images, Sp and Tp, be the portions of S and T that overlap,
shown in Fig. 1(b) and ©, respectively. From this overlapped
area a color correction method computes an estimation of one
or more color palette mapping functions. By applying these
functions to all the pixels of T, the color corrected image Tˆ
is obtained. The objective of any color correction approach is
to provide an estimation for this mapping function, denoted as
color palette mapping function, that maximizes the similarity
between the S and Tˆ images.
In general, the color correction operation is performed
independently for each color channel. In this sense, color
correction (correcting the three color channels) and brightness
correction (correcting a single grayscale image) are similar
Throughout this paper, the methodology for correcting a
single channel is explained. In the following sections, all the
examples shown in graphs and plots are given for the red
channel of the images. The presented results were obtained
by applying similar processes for the green and blue channels.
The term color or pixel color will be used throughout the
remainder of the paper referring to the intensity value of that
pixel for a particular color channel. Since color correction
is done independently for each channel, some authors have
proposed to use color spaces where cross channel artifacts are
less prone to occur. For example in [29] and [33], the images
are first transformed to the lαβ color space before performing
independent channel color correction. In [42], the CIECAM97
color space is employed and in [10] the chosen color space
was YCbCr. On the other hand, many other proposals use the
standard RGB color space [14], [15], [26], [39]. In Section V,
we investigate how different colorspaces affect the performance
of the color correction. In particular, we will compare
the performance of the proposed approach when using the
RGB and the lαβ colorspaces.
The estimated color palette mapping function should be as
similar as possible to an ideal mapping function. However,
since the ideal mapping function is unknown in most cases,
it is not possible to measure the quality of a color correction
method by comparing the estimated and the ideal mapping
functions. The alternative is to analyze the output of the
algorithms, i.e., to compare photometrically the corrected
image with the source image. This comparison is only possible
in the overlapping regions of the images.
III. RELATED WORK
Over the past years, several color correction approaches
have been presented. Color correction methods can be divided
into model-based parametric approaches, i.e., where the color
distribution of the images is assumed to have some statistical
distribution, and model-less, nonparametric approaches,
i.e., where no assumptions about the nature of the color distribution
are taken [40]. In 2001, Reinhard et al. [29] proposed
one of the first parametric methods for color correction. Single
Gaussians are used to model the color distributions of the
target and source images. The color distribution of the S
image is then transferred to the T image by scaling and
offsetting according to the mean and standard deviations of
the precomputed Gaussians. Although this work showed the
potential of parametric methodologies, one possible shortcoming
was that single Gaussians could be inaccurate to model
the color distribution of a complete image. With this in mind,
some subsequent works have proposed to use more complex
models, in an attempt to achieve more accurate models of
the color distributions, and as a result, more effective color
corrections. The usage of Gaussian Mixture Models (GMMs)
was proposed in [33]. The GMM was fitted to the data using
a Expectation Maximization (EM) methodology. The GMM
could not only model the color distribution more accurately,
but also assist in the probabilistic segmentation of the image
into regions. For each segmented region, a color transfer
methodology similar to the one proposed in [29] was proposed.
One other problem that may hamper the performance of
color correction methodologies is that each color channel
in the image is modelled and corrected independently. This
may lead to possible cross channel artifacts [29]. As an
attempt to solve this problem, [25] proposed the usage of
a multichannel modelling and color correction mechanism,
which employed 3D GMMs in order to model the color distribution
across the three color channels in the images. There are
some other works that try to model the three color channels
simultaneously, using multi channel image blending [4], [20].
Other model based approaches include Principal Component
Analysis [2], [42], and gain compensation methodologies [4].
There are also several examples of nonparametric methods.
In [14], the color transfer functions are obtained using a
2D tensor voting scheme. In [30] higher-dimensional Bezier
patches were used to represent color transfer functions.
Nonparametric methods also include approaches that use the
image histograms to obtain a color transfer function [10], [35].
In [26] and [27] the entire probability density function is
mapped without making assumptions on its nature. Fuzzy
logic based methodologies are proposed in [1] and [28].
Nonparametric methods also include learning frameworks.
In [23] and [41], authors propose to use neural networks to
learn the color palette mapping functions. However, learning
approaches have the limitation that they require a set of source
target image pairs for training, which could be unfeasible to
obtain in some cases. In addition, learning approaches have the
limitation of requiring a specific training for different setups.
There is also a group of techniques called nonparametric
kernel regressions, which could have a direct application to the
color correction problem, in particular when formulated using
the joint image histogram. The Nadaraya-Watson [12], [37]
estimator starts from the formulation of the conditional expectation
of variable Y, given X:
Modelling the Joint Image Histogram
As described in Section IV-B, the goal of a color correction
algorithm is to estimate a color palette mapping function ˆf i
(·)
(see eq. (4)). For a given region i, the color palette mapping
function must be inferred from the information present on the
corresponding local joint image histogram I
i
.
We assume that the observations in the joint image histogram
follow a Gaussian distribution. The reason why we
use the Gaussian distribution is that there are several works
on color correction that assume Gaussian distributions which
have achieved interesting results, see [24], [25], [29], [33].
However, one of the possible shortcomings of modelling color
with a Gaussian distribution is that it does not take into account
the fact that the range of values for the color is bounded due
to the image bit depth. Thus, it makes more sense to use a
truncated Gaussian distribution model to fit the data from the
joint image histogram, rather than a standard Gaussian.
In summary, we propose to estimate the color palette
mapping functions by fitting truncated Gaussian models to
the data from the local joint image histogram. Since these
histograms are 2D signals, the first option would be to model
it using a 2D distribution (a process similar to surface fitting).
However, that would have the disadvantage of turning our
model into a multivariate truncated Gaussian, which would
increase the overall complexity of the algorithm. For example,
it is considerably more difficult to sample values from a
truncated Gaussian in the multivariate case when compared
to the univariate case [7]. Also, the fitting of multivariate
truncated Gaussians is not straightforward: [16] proposed an
extension to the classical EM algorithm to do this, but report
that fitting a multivariate truncated Gaussian model is two
orders of magnitude slower when compared to fitting a non
truncated similar model. Finally, fitting a single multivariate
Gaussian would assume that the distribution of mappings is
not multi-modal, since that a single Gaussian is used. This is
not a feasible assumption. For example in Fig. 2(a), the observations
in the joint image histogram are clearly multi-modal.
Hence, due to the aforementioned reasons, in the current
work univariate truncated Gaussian models are used. Fitting
univariate Gaussians also assumes unimodal distributions, but
this assumption is more feasible because it affects a single
dimension.
For a given region i, the local conditional expectation Ei
that a color Y occurs, given that color X takes the value x
(denoted as X = x), is defined as follows:
Ei
(Y | X = x) =
2
n−1
y=0
y · Pi
(Y = y | X = x), (5)
where n is the bit depth of the image Sp, and
P(Y = y | X = x) is the conditional probability of Y given
X = x. Thus, to propose a color palette mapping function, we
must find an estimate for the probability P(Y = y | X = x):
Yˆ = ˆf i
(X = x) ≡
2
n−1
y=0
y · Pˆi
(Y = y | X = x), (6)
where Pˆ denotes the estimated probability. Then, using
Bayes rule, the conditional probability in eq. (6) is expanded
to:
Pˆi
(Y = y | X = x) = Pˆi
(X = x | Y = y) · Pˆi
(Y = y)
Pˆi(X = x) , (7)
where Pˆi
(X = x) and Pˆi
(Y = y) are the prior probabilities,
estimated from the normalized joint image histogram as
CONCLUSION
This paper proposes a novel color correction algorithm.
Images are color segmented using mean-shift and a region
fusion algorithm. Each segmented region is then used to
compute a local color palette mapping function by fitting
a set of univariate truncated Gaussians to the observed
color mappings. Finally, by using an extension of the color
palette mapping functions to the whole image, it is possible
to produce mosaics where no color transitions are
noticeable.
For the proper assessment of the performance of the proposed
algorithm, ten other color correction algorithms were
evaluated (#2 through #11), along with three alternatives to
the proposed approach (#12b through #12d). Each of the algorithms
was applied to two datasets, with a combined total of
63 image pairs. Three different evaluation metrics were used.
The proposed approach outperforms all other algorithms, in
most of the image pairs in the datasets, considering the PSNR
and S-CIELAB evaluation metrics. Not only it obtains some
of the best average scores but also shows to be more consistent
and robust (see Table IV). Furthermore, the difference in
performance from the proposed approach with respect to the
others is larger in the case of the real image dataset, which
seems to lead to the conclusion that the proposed approach is
well suited for handling realistic scenarios. Restuls have shown
that the proposed approach achieves very good results even
if no color segmentation preprocessing step is used. Results
have also shown that the usage of truncated Gaussians as well
as the indirect modelling of the joint image histogram also
improve the effectiveness of the color correction algorithm.
Finally, we show that both the RGB and the lαβ colorspaces
achieve similar color correction performances.