01-02-2013, 11:05 AM
Multilayer Inpainting on Digitalized Artworks
Multilayer Inpainting.pdf (Size: 869.56 KB / Downloads: 20)
ABSTRACT
Image inpainting automatically restores damaged images and partially removed pictures.
Since most inpainting mechanisms inpaint damaged portions in a single layer, this
study proposes a multilayer inpainting mechanism by examining how Chinese paintings
are drawn in layers. The proposed multilayer inpainting mechanism employs a layer fusion
strategy to detect the optimal inpaint combination among layers to restore paintings.
Because this multilayer multi-resolution strategy considers damages in each layer from a
multi-resolution perspective, it is superior to several existing techniques for restoring
Chinese and Western paintings. In this study, the proposed algorithm is tested on more
than 1,500 still images, with evaluations showing the effectiveness of image inpainting.
The results indicate that the proposed algorithm achieves high PSNR values as well as
high user satisfactions, including inpainting in some extreme cases where more than
ninety percent of a painting are destroyed.
INTRODUCTION
Automatic digital inpainting is a technique which restores damaged image or video
by means of spatial/temporal interpolation and other mechanisms. The technique can be
used in photo restoration (e.g., scratch removal), zooming, image coding, wireless image
transmission (e.g., recovering lost blocks), and special effects (e.g., removal of objects).
Current techniques are based on extrapolation or interpolation of neighboring pixels [4],
recovery of edges, curvature-driven diffusions (according to the connectivity principle in
vision psychology) [2], and inpainting from multiple view points (i.e., image from movie,
or image from different time and view point).
THE MULTI-RESOLUTION INPAINTING ALGORITHM
The base inpainting method that is introduced in the proposed algorithm takes in to
account different levels of details of still images [8]. In general, if an image region is
seriously damaged, it is not realistic to rely on the extrapolation of neighboring pixels in
any method. Instead, global information should be used. Additionally, if the variance of
pixel colors is large in an image block, it is possible that the block contains detailed
shapes. Thus, a multi-resolution strategy should be considered.
GENERAL PRINCIPLE OF PAINTING
We study fundamental techniques of Chinese landscape painting [9]. An artist
paints trees in dark ink before distant rocks in tan colors are added. Finally, light colors
are used in the background. In general, the use of color is from dark to light. We argue
that it is important to use dark colors to inpaint missing potion of trees, to use tan colors
for rocks, and to use light colors for the background. But, color segmentation is difficult.
It is not easy to separate trees from rocks and background. However, an approximation
approach should be considered since trees and rocks could be interleaved. At least, we
can separate the painting into multiple layers, according to the use of colors. Since the
variation of colors used in traditional Chinese painting is limited, it is possible to separate
a painting efficiently according to a carefully chosen color space. For western paintings,
a similar strategy can be used. However, experience shows that western painting
has a richer usage of colors. More layers can be used. Layer separation on western paintings
is not easy but possible.
THE NEW INPAINTING STRATEGY
Most inpainting algorithms treat a damaged picture as a single layer and computes
inpaint data based on the single layer. However, single layer approach may use irrelevant
information on restoring an object, without considering the separation of the object and
its background. In our earlier approach [11], we separate a Chinese painting into several
layers. Each layer contains different objects (e.g., far mountain, near mountain, trees, and
people) and is processed separately. In the end, a fusion algorithm is used to combine the
results.
The proposed algorithm, multilayer inpainting scheme, has three main building
blocks: layer separation, inpainting, and layer fusion. These techniques are discussed in
the following subsections.
Layer Separation
Color region separation is a difficult challenge, if the objective is to precisely detect
the boundary of objects under different conditions. In digital inpainting, it is impossible
to restore an image to one hundred percent since information is lost. With this in mind,
an approximation approach is employed to separate objects into different layers. Additionally,
a naive layer separation algorithm is used to divide a painting into several layers.
It is difficult to decide the threshold of separation. However, pixels of similar colors can
be divided into groups, which represent layers.
The Inpainting Algorithm
We realize that different portion of a picture contains different levels of details.
Thus, we use a multi-resolution strategy, which looks at the details, and decide what surrounding
information to use. Level of details can be indicated by the variance of color
distribution in a portion of image. We use another percentage threshold α for color variance.
Distance of color is treated the same as that used for δ
1. Value of color variance
ranges from zero to several thousands depends on pictures. According to our experiments,
α = 80% results in a good result. Two additional percentage thresholds,
used in the inpainting algorithm for different situations of decomposition.
An input damaged picture DIBk is divided into several image blocks (i.e., IBs). If
the percentage of damaged pixels in an IB is too high, using surrounding color information
to fix a pixel is less realistic as compared to using a global average color. In some
severe cases, it is impossible to use neighborhood colors. Note that, both thresholds are
adjustable for the sake of analysis. The recursive algorithm iterates through each of the
IBs in a damaged picture. If the color variance of IB is below the percentage threshold α,
there is not much difference of pixels in the IB.
The Multilayer Fusion Strategy
After a damaged picture is decomposed into K layers, we use the above revised
multi-resolution Inpainting algorithm to inpaint damaged areas in each layer of the picture
(i.e., DIBLayer[K]). According to the decomposition, the first layer has the lightest
colors, which represent a far background. The darker the color the higher chance of a
foreground. However, to combine the separated layers after inpainting, we need a fusion
algorithm. The fusion algorithm follows a strategy. For each damaged pixel to be inpainted,
two consecutive layers are compared. A window with pixels in a distance D with
respect to an inpainted pixel P is used. The function μ [P] computes percentages of useful
pixels within distance D is applied to each inpainted pixel. Depending on the percentages,
a layer is selected. Useful pixels are non-inpainted pixels from the original image, with
respect to a layer. The far ground is firstly placed in a blank paper. The picture is restored
with a darker layer step-by-step.