23-05-2012, 02:01 PM
Robust Real-Time Super-Resolution on
FPGA and an Application to Video Enhancement
[attachment=22876]
INTRODUCTION
Every imaging system is based on an image sensor, a 2-D array of pixels that convert
incident light to an array of electrical signals (Fig. 1(a)) [Gamal and Eltoukhy 2005]. Two
types of resolution determine the quality of information collected by the sensor: the spatial
and the temporal resolution.
The spatial resolution depends on the spatial density of the photodiodes and their induced
blur. The most intuitive solution to increase the spatial resolution corresponding to
the same field of view would be reducing the pixel size, hence increasing the pixel density.
However, the smaller the photodiodes become, the smaller is the amount of incident light. As a result, a longer exposure time is required to achieve an adequate signal to noise
ratio [Gamal and Eltoukhy 2005; Farrell et al. 2006; Chen et al. 2000].
THE OBSERVATION MODEL
The proposed FPGA architecture deals with the problem of reconstructing the SR output
based on LR samples. Before looking into that problem, let us briefly describe the forward
model that forms these samples [Farsiu et al. 2004; Park et al. 2003]. This is known as the
observation model. The LR degradation channel associated with each LR frame comprises
a series of degradations, as shown in Fig. 2. The first stage involves the atmospheric blur.
A group of images that are largely affected by this type of degradation are astronomical
images. This work deals with the stages of the observation model that succeed the atmospheric
blur. These are: the motion blur, the spatial blur and the additive noise on the pixels
of the sensor
Combining an FPGA with an Adaptive Image Sensor
The adaptive image sensor should be configured in a manner that maximizes the raw information
collected from the environment. Once this information is captured, it should
be further processed to reconstruct a final output of both high temporal and high spatial
resolution. In addition, the entire system should operate in real-time, i.e. at least 25 fps, to
achieve the real-time capturing of the scene. Such throughput requirements render software
processing inadequate, due to the high computational complexity associated with the required
pixel-level processing, which scales with the number of LR samples. By exploiting
the parallelism, pipelining and data reuse possibilities offered by reconfigurable hardware,
the above objectives are feasible, as it will be explained in the sections that follow. The role
of the FPGA is twofold as illustrated in Fig. 4. It processes the raw information in realtime
and, also, configures the adaptive sensor in a way that maximizes the raw information,
according to the collected data.
An overview of the proposed video enhancement system is shown in Fig. 5. Motion
areas are located on the frame and are configured to larger pixel sizes forming LR areas,
whereas areas with slow motion or no motion are configured to HR pixels. During the HR
integration, a sequence of LR frames with reduced blur is produced at every LR area. Each
LR area is spatially enhanced using SR to estimate a high resolution frame based on the LR
inputs [Irani and Peleg 1991]. Thus, motion-deblurring is locally executed on the dynamic
regions of the scene. The focus of this paper is on the implementation of the SR block that
performs the spatial enhancement of the raw outputs.
ARCHITECTURE OF THE SR SYSTEM
Figure 8 shows an overview of the proposed system. For every new group of LR frames,
produced during a particular HR integration interval (Sect. 3.2), an SR stage occurs. At
the beginning of each SR stage an initial high-resolution approximation is produced by
applying interpolation on the reference LR frame. Once this initial phase is completed, the
iterations of the algorithm begin. When the iterations are over, the next LR group (associated
with the next HR integration interval) is processed, and so on. The rest of the section
focuses on the description of the individual blocks of the proposed architecture. The motion
vectors, describing the displacements of each LR frame with respect to the reference
frame, are treated as inputs to the implemented SR system. It should be mentioned that the
target system has 4 memory banks, each with a word-length of 4 bytes.
FPGA and an Application to Video Enhancement
[attachment=22876]
INTRODUCTION
Every imaging system is based on an image sensor, a 2-D array of pixels that convert
incident light to an array of electrical signals (Fig. 1(a)) [Gamal and Eltoukhy 2005]. Two
types of resolution determine the quality of information collected by the sensor: the spatial
and the temporal resolution.
The spatial resolution depends on the spatial density of the photodiodes and their induced
blur. The most intuitive solution to increase the spatial resolution corresponding to
the same field of view would be reducing the pixel size, hence increasing the pixel density.
However, the smaller the photodiodes become, the smaller is the amount of incident light. As a result, a longer exposure time is required to achieve an adequate signal to noise
ratio [Gamal and Eltoukhy 2005; Farrell et al. 2006; Chen et al. 2000].
THE OBSERVATION MODEL
The proposed FPGA architecture deals with the problem of reconstructing the SR output
based on LR samples. Before looking into that problem, let us briefly describe the forward
model that forms these samples [Farsiu et al. 2004; Park et al. 2003]. This is known as the
observation model. The LR degradation channel associated with each LR frame comprises
a series of degradations, as shown in Fig. 2. The first stage involves the atmospheric blur.
A group of images that are largely affected by this type of degradation are astronomical
images. This work deals with the stages of the observation model that succeed the atmospheric
blur. These are: the motion blur, the spatial blur and the additive noise on the pixels
of the sensor
Combining an FPGA with an Adaptive Image Sensor
The adaptive image sensor should be configured in a manner that maximizes the raw information
collected from the environment. Once this information is captured, it should
be further processed to reconstruct a final output of both high temporal and high spatial
resolution. In addition, the entire system should operate in real-time, i.e. at least 25 fps, to
achieve the real-time capturing of the scene. Such throughput requirements render software
processing inadequate, due to the high computational complexity associated with the required
pixel-level processing, which scales with the number of LR samples. By exploiting
the parallelism, pipelining and data reuse possibilities offered by reconfigurable hardware,
the above objectives are feasible, as it will be explained in the sections that follow. The role
of the FPGA is twofold as illustrated in Fig. 4. It processes the raw information in realtime
and, also, configures the adaptive sensor in a way that maximizes the raw information,
according to the collected data.
An overview of the proposed video enhancement system is shown in Fig. 5. Motion
areas are located on the frame and are configured to larger pixel sizes forming LR areas,
whereas areas with slow motion or no motion are configured to HR pixels. During the HR
integration, a sequence of LR frames with reduced blur is produced at every LR area. Each
LR area is spatially enhanced using SR to estimate a high resolution frame based on the LR
inputs [Irani and Peleg 1991]. Thus, motion-deblurring is locally executed on the dynamic
regions of the scene. The focus of this paper is on the implementation of the SR block that
performs the spatial enhancement of the raw outputs.
ARCHITECTURE OF THE SR SYSTEM
Figure 8 shows an overview of the proposed system. For every new group of LR frames,
produced during a particular HR integration interval (Sect. 3.2), an SR stage occurs. At
the beginning of each SR stage an initial high-resolution approximation is produced by
applying interpolation on the reference LR frame. Once this initial phase is completed, the
iterations of the algorithm begin. When the iterations are over, the next LR group (associated
with the next HR integration interval) is processed, and so on. The rest of the section
focuses on the description of the individual blocks of the proposed architecture. The motion
vectors, describing the displacements of each LR frame with respect to the reference
frame, are treated as inputs to the implemented SR system. It should be mentioned that the
target system has 4 memory banks, each with a word-length of 4 bytes.