28-11-2012, 02:24 PM
Super-Resolution from Image Sequences - A Review
Super-Resolution.pdf (Size: 95.52 KB / Downloads: 25)
Abstract
Growing interest in super-resolution (SR) restoration of
video sequences and the closely related problem of construction
of SR still images from image sequences has led
to the emergence of several competing methodologies. We
review the state of the art of SR techniques using a taxonomy
of existing techniques. We critique these methods and
identify areas which promise performance improvements.
Introduction
The problem of spatial resolution enhancement of video
sequences has been an area of active research since the seminal
work by Tsai and Huang [20] which considers the problem
of resolution enhanced stills from a sequence of lowresolution
(LR) images of a translated scene. Whereas in
the traditional single image restoration problem only a single
input image is available, the task of obtaining a superresolved
image from an undersampled and degraded image
sequence can take advantage of the additional spatiotemporal
data available in the image sequence. In particular,
camera and scene motion lead to frames in the video
sequence containing similar, but not identical information.
This additional information content, as well as the inclusion
of a-priori constraints, enables reconstruction of a superresolved
image with wider bandwidth than that of any of
the individual LR frames.
Frequency Domain Methods
A major class of SR methods utilize a frequency domain
formulation of the SR problem. Frequency domainmethods
are based on three fundamental principles: i) the shifting
property of the Fourier transform (FT), ii) the aliasing relationship
between the continuous Fourier transform (CFT)
and the discrete Fourier transform (DFT), iii) the original
scene is band-limited. These properties allow the formulation
of a system of equations relating the aliased DFT
coefficients of the observed images to samples of the CFT
of the unknown scene. These equations are solved yielding
the frequency domain coefficients of the original scene,
which may then be recovered by inverse DFT. Formulation
of the system of equations requires knowledge of the
translational motion between frames to sub-pixel accuracy.
Each observation image must contribute independent equations,
which places restrictions on the inter-frame motion
that contributes useful data.
Spatial Domain Methods
In this class of SR reconstruction methods, the observation
model is formulated, and reconstruction is effected
in the spatial domain. The linear spatial domain observationmodel
can accommodate global and non-globalmotion,
optical blur, motion blur, spatially varying PSF, non-ideal
sampling, compression artifacts and more. Spatial domain
reconstruction allows natural inclusion of (possibly nonlinear)
spatial domain a-priori constraints (e.g. Markov random
fields or convex sets) which result in bandwidth extrapolation
in reconstruction.
Interpolation of NonUniformly
Spaced Samples
Registering a set of LR images using motion compensation
results in a single, dense composite image of nonuniformly
spaced samples. A SR image may be reconstructed
from this composite using techniques for reconstruction
from non-uniformly spaced samples. Restoration
techniques are sometimes applied to compensate for degradations
[17]. Iterative reconstruction techniques, based
on the Landweber iteration, have also been applied [12].
Such interpolation methods are unfortunately overly simplistic.
Since the observed data result from severely undersampled,
spatially averaged areas, the reconstruction step
(which typically assumes impulse sampling) is incapable
of reconstructing significantly more frequency content than
is present in a single LR frame. Degradation models are
limited, and no a-priori constraints are used. There is
also question as to the optimality of separate merging and
restoration steps.
Set Theoretic ReconstructionMethods
Set theoretic methods, especially the method of projection
onto convex sets (POCS), are popular as they are simple,
utilize the powerful spatial domain observation model,
and allow convenient inclusion of a priori information. In
set theoretic methods, the space of SR solution images is
intersected with a set of (typically convex) constraint sets
representing desirable SR image characteristics such as positivity,
bounded energy, fidelity to data, smoothness etc., to
yield a reduced solution space. POCS refers to an iterative
procedure which, given any point in the space SR images,
locates a point which satisfies all the convex constraint sets
Restoration Algorithms:
MAP and POCS based algorithms
are very successful and to a degree, complimentary.
Hybrid MAP/POCS restoration techniques will combine
the mathematical rigor and uniqueness of solution of
MAP estimation with the convenient a-priori constraints of
POCS. The hybridmethod isMAP based but with constraint
projections inserted into the iterative maximization of the
a-posteriori density in a generalization of the constrained
MAP optimization of [15]. Simultaneousmotion estimation
and restoration yields improved reconstructions since motion
estimation and reconstruction are interrelated. Separate
motion estimation and restoration, as is commonly done,
is sub-optimal as a result of this interdependence. Simultaneous
multi-frame SR restoration is expected to achieve
higher performance since additional spatio-temporal constraints
on the SR image ensemble may be included. This
technique has seen limited application in SR reconstruction.