17-05-2013, 04:44 PM
On-line Handwriting Recognition with Support Vector Machines— A Kernel Approach
On-line Handwriting.pdf (Size: 576.05 KB / Downloads: 19)
Abstract
In this contribution we describe a novel classification
approach for on-line handwriting recognition. The technique
combines dynamic time warping (DTW) and support
vector machines (SVMs) by establishing a new SVM
kernel. We call this kernel Gaussian DTW (GDTW) kernel.
This kernel approach has a main advantage over common
HMM techniques. It does not assume a model for the
generative class conditional densities. Instead, it directly
addresses the problem of discrimination by creating class
boundaries and thus is less sensitive to modeling assumptions.
By incorporating DTW in the kernel function, general
classification problems with variable-sized sequential
data can be handled. In this respect the proposed method
can be straightforwardly applied to all classification problems,
where DTW gives a reasonable distance measure,
e.g. speech recognition or genome processing. We show experiments
with this kernel approach on the UNIPEN handwriting
data, achieving results comparable to an HMMbased
technique.
Introduction
The utilization of support vector machine (SVM) [2, 4]
classifiers has gained immense popularity in the last years.
SVMs have achieved excellent recognition results in various
pattern recognition applications [4]. Also in off-line optical
character recognition (OCR) they have been shown to be
comparable or even superior to the standard techniques like
Bayesian classifiers or multilayer perceptrons [5]. SVMs
are discriminative classifiers based on Vapnik’s structural
risk minimization principle. They can implement flexible
decision boundaries in high dimensional feature spaces.
The implicit regularization of the classifier’s complexity
avoids overfitting and mostly this leads to good generalizations.
Some further properties are commonly seen as reasons
for the success of SVMs in real-world problems: the
optimality of the training result is guaranteed, fast training
algorithms exist and little a-priori knowledge is required,
i.e. only a labeled training set.
Gaussian dynamic time warping kernel
As indicated in the introduction, when dealing with sequential
on-line handwriting data we cannot simply employ
the basic SVM framework given by (3)–(6). Different feature
vector sequences Pi, Pj and T cannot be embedded in
the same vector space in general, as the necessary dimensions
differ.
However, an important property of (3)–(4) is that the
vectors Pi, Pj and T appear only in form of kernel evaluations.
Thus our objective, when adopting SVMs to sequential
handwriting data, can be to state a kernel definition
suitable to the particular properties of the sequential data.
Experiments
Data
The experiments are based on the 1a, 1b and 1c section
(digits, upper and lower case characters, respectively) of the
UNIPEN [7] Train-R01/V07 database. For these sections
the data set size is ≈ 16K, 28K and 61K, respectively. Examples
of UNIPEN data were shown in figure 1. Training
and test set were taken disjointly. It should be stated that
UNIPEN consists of very difficult data due to the variety of
writers and noisy or mislabeled data. We used the database
without cleaning in order to be as comparable as possible to
other classification reports.
Multi-class experiments
For a multi-class experiment we have chosen the DAGSVM
approach [13]. DAG-SVM combines a set of twoclass
SVMs into a multi-class classifier. For aK-class problem
DAG-SVM contains K · (K − 1) /2 two-class classifiers,
one for each class pair. During classification K − 1
two-class SVM evaluations are combined using a decision
directed acyclic graph (DDAG) topology.
For the multi-class case we used smaller UNIPEN subsets
due to efficiency reasons (see section 4.5). We made
experiments on two different dataset sizes in order to give
an idea of the recognizer’s dependence on this quantity. Figure
2 gives a graphical illustration of an example classification
showing a snapshot of our classification GUI.
Conclusion
We have presented a novel approach for the recognition
of on-line handwritten characters. This technique combines
dynamic time warping (DTW) and support vector machines
(SVM) by integrating DTW into a Gaussian SVM kernel.
The benefit of this approach is the absence of restrictive
assumptions about class conditional densities, as made in
conventional HMM based techniques. The only essential
assumption made is the selection of the kernel.