31-05-2012, 04:41 PM
Lane-Vehicle Detection and Tracking
[attachment=23746]
Abstract
This paper presents a monocular lane-vehicle
detection and tracking system comprising of (i) lane boundary
detection, (ii) lane region tracking, and (iii) vehicle detection
with a proposed vertical asymmetry measurement. First, a traffic
scene image is divided into possible road region. Lane boundaries
are then extracted from the region using lane markings
detection. These detected boundaries are tracked in consecutive
video frames using a linear-parabolic tracking model. Therefore,
an approximate lane region is constructed with the estimated
model parameters. By integrating the knowledge of lane region
with vehicle detection, vehicle scanning region is restricted to the
road area so as to detect the shadow underneath a vehicle
continuously with less interference from the road environment
and non-vehicle structures. A self-adjusting bounding box is used
to extract likely vehicle region for further verification. Besides
horizontal symmetry detection, a vertical asymmetry
measurement is proposed to validate the extracted region and to
obtain the center of frontal vehicle. Preliminary simulation
results revealed good performance of lane-vehicle detection and
tracking system.
Index Terms— Driver assistance systems, Lane detection,
Lane tracking, Vehicle detection.
I. INTRODUCTION
Advancement of vision-based vehicle detection has
triggered recent development of autonomous vehicular
technology in order to automatically localize moving vehicles
in complex traffic scene. Generally, the main task of
vision-based vehicle detection is to sense a leading vehicle and
thus spontaneously alert a driver of precritical conditions with
the preceding vehicle, in case he/she makes a sudden break,
slowdown driving or uniform movement [1]. Moreover,
vehicle detection can be developed to track and follow a
preceding car, and at the same time increase the efficient use
of driving space. In consequence, an effective visual scanning
of frontal road conditions is highly required for vehicle
detection system to identify the position of frontal vehicle.
Numerous vision-based techniques have been developed
over the past decade to detect vehicles in various road scenes,
as described in [1]-[11]. Tzomakas and Seelen [2] detected
vehicles based on the shadows underneath them. With the use
of entropy, a free driving space was determined and therefore,
an adaptive threshold was applied to extract vehicle edges.
Chang et al. [3] presented a preceding vehicle detection
Manuscript received October 16, 2008. Manuscript submitted to the
IAENG International Conference on Control and Automation, ICCA’09.
The authors are with the School of Electrical and Electronics Engineering,
The University of Nottingham, Malaysia Campus, Jalan Broga, Semenyih,
43500, Selangor, Malaysia. (Phone: +603-8924-8350, Fax:
+603-8924-8071; Email: {keyx7khl, Jasmine.Seng, Kenneth.Ang,
keyx8csw}[at]nottingham.edu.my).
and tracking system by finding the footprint of a vehicle on
road area and tracking the vehicle using the continuity
measurement of two consecutive frames. Kate et al. [4]
combined the knowledge-based methods such as shadow
detection, entropy analysis and horizontal symmetry
measurement for mid-range and distant vehicle detection
without prior knowledge about the road geometry.
Besides that, Khammari et al. [5] applied a horizontal
Sobel filter on the 3rd level of the Gaussian pyramid to obtain
local gradient maxima where a vehicle candidate is located. A
temporal filter was used to further remove unwanted pixels
and then a bounding box extraction was employed to retrieve a
possible vehicle region for symmetry verification. Broggi et
al. [6] presented a multi-resolution vehicle detection method
to localize vehicles with variable sizes. They computed the
symmetry property of vehicles in different sized bounding
boxes on all the columns of the regions. Liu et al. [7][8]
detected vehicle region based on the shadow underneath a
vehicle and symmetry edges. Additionally, they combined
knowledge-based and learning-based methods for vehicle
verification. In vehicle tracking, templates were dynamically
created on-line and tracking window was adaptively adjusted
with motion estimation.
On the other hand, Hoffman [9] performed a multi-sensor
fusion approach incorporating 2-D visual features such as
shadow and symmetry, with 3-D ground plane information
such as camera height for vehicle detection. These fusing
features were tracked using Interacting Multiple Model
method. Bertozzi and Broggi [10] used stereo vision-based
method to detect both generic obstacles and lane positions in a
structured environment. They utilized Inverse Perspective
Mapping technique to remove perspective effect and
reconstruct a 3-D mapping when the camera parameters and
the knowledge about road are known. Giachetti et al. [11]
developed first-order and second-order differential methods to
detect vehicle based on the motion.
However, the presence of non-vehicle structures such as
over-bridge, fly-over roadway, tunnel, buildings, sign boards
etc, in traffic scene may decrease the performance of
knowledge-based vehicle detection since these non-vehicle
structures posses the horizontal/vertical characteristics
identical to vehicle’s edges [2]-[8]. Moreover, a complex road
environment may complicate the process of vehicle detection
as there are many possibilities of human activities along the
road side [4]. Frontal vehicle with little relative motion change
or stand still has increased the difficulty of vehicle detection
based on motion flow [1][11]. Furthermore, the requirement
of 3-D transformation and the knowledge of hardware
parameters for stereo-based vehicle detection method have
highly increased the computational cost and reduced the
processing speed [9][10].
Lane-Vehicle Detection and Tracking
King Hann LIM*, Li-Minn ANG, Kah Phooi SENG and Siew Wen CHIN
Proceedings of the International MultiConference of Engineers and Computer Scientists 2009 Vol II
IMECS 2009, March 18 - 20, 2009, Hong Kong
ISBN: 978-988-17012-7-5 IMECS 2009
Fig. 1: Proposed lane-vehicle detection and tracking system.
To solve abovementioned problems, a novel monocular
lane-vehicle detection and tracking system is proposed in this
paper, as the flow diagram is depicted in Fig. 1. Lane boundary
detection is performed on road image to locate the left-right
lane edges by separating the sky-road region, and analyzing
road region to extract the prominent road features such as lane
markings. A linear-parabolic lane model without the
requirement of camera parameters is constructed to estimate
lane geometry. This is followed by lane region tracking to
restrict the vehicle searching region on the ground for every
continuous frame. As the vehicle region is bounded, vehicle
detection based on the shadow underneath a vehicle is
repeatedly performed in tracking lane region. In addition, a
bounding box extraction method is used to obtain a likely
vehicle region for further verification. Besides horizontal
symmetry detection, a proposed vertical asymmetry
measurement is applied to verify the extracted region and at
the same time, the center of vehicle is found.
Section II discusses about the lane boundary detection
while Section III explains the lane region tracking technique.
The vehicle detection with proposed verification technique is
discussed in Section IV. Some simulation results are shown in
Section V and followed by conclusion and future works.
II. LANE BOUNDARY DETECTION
Lane boundary detection is important to locate the left-right
edges of driving path on a traffic scene image. In this section, a
three-stage lane boundary detection is performed, i.e. (i)
vertical mean distribution, (ii) lane region analysis, and (iii)
lane marking detection.
A. Vertical Mean Distribution
In lane detection process, sky region is not a region of
interest (ROI). At the preliminary stage, a traffic scene image
I(x,y) is divided into sky region and road region using vertical
mean distribution [12]. Vertical mean distribution is measured
by averaging the gray values of each row on road image and
the row means are plotted in the graph depicted in Fig. 2(a).
The threshold value of horizon line is obtained through a
minimum search along the vertical mean curve, where the first
Fig. 2: (a) Vertical mean distribution, (b) Road region image (Rroi).
minimum occurs from the upper curve is the regional dividing
line. This is because sky region usually possesses higher
intensity than road pixels, and it might have a big jump of
intensity difference as sky pixels approaches ground. The
horizon line threshold is applied to generate a road image (Rroi)
as demonstrated in Fig 2(b), where all vertical coordinates
below the threshold are discarded.
B. Lane Region Analysis
Lane region analysis is performed to further classify road
region and lane markings. Usually, the bottom region in a road
image contains road pixels. By acquiring a few rows from the
bottom of image, the gray value range of road color can easily
be obtained and therefore, this color range is applied to further
remove road pixels from the Rroi map. The lane region
analysis steps are shown as follows:
Step 1: Pick 30-60 rows from bottom to avoid the
possible existence of inner part of a vehicle at the
edge of image.
Step 2: Build a vote scheme; namely VOTE for the
selected rows and the maximum vote of the row
pixels is recorded.
Step 3: Record a global maximum value for the selected
rows, as MAXTHRES.
Step 4: Define the road color range as [VOTE-25;
MAXTHRES+25]. Pixels that fall within this
range are denoted as road pixels and a binary map
(Rbin) is formed as shown in Fig. 3(a).
Step 5: Generate a difference map (Dmap) by differencing
Rroi and Rbin maps. The positive pixel values are
retained while the negative values are set to zero.
The difference map is illustrated in Fig. 3(b).
Fig. 3: (a) Lane binary map (Rbin), (b) Difference map (Dmap).
C. Lane Marking Detection
The difference map obtained in the previous stage is,
therefore used for lane marking detection. Lane markings are
the salient features on road and they are usually used to extract
the boundary of road region. Initially, an edge detection using
Sobel filter is applied to Dmap. The gradient magnitude
ÑDmap (x, y) and orientation q (x, y) are calculated in the
following equations [13]:
ÑDmap (x, y) » Dx + Dy (1)
)
( ( , ) ( , ))
2 ( , ) ( , )
tan (
2
1
( , )
( , )
2 2
1 ( , )
Î
- Î
-
=
u v R x y
u v R x y
D u v D u v
D u v D u v
q x y (2)
where u and v are the 3×3 mask coordinates. Dx is the
horizontal edge map whereas Dy is the vertical edge map.
Mean Values
Image Row
Horizon Line
Proceedings of the International MultiConference of Engineers and Computer Scientists 2009 Vol II
IMECS 2009, March 18 - 20, 2009, Hong Kong
ISBN: 978-988-17012-7-5 IMECS 2009
Fig. 4: The edge distribution function.
With reference to the constructed gradient map and its
corresponding orientation, a histogram called edge
distribution function [13] is build, with its x-axis representing
the orientation ranging from [-90º; 90º] and its y-axis
representing the gradient value of each orientation bins. The
maximum peak acquired in the graph on the negative angle
denotes the left boundary angle. Conversely, the maximum
peak at the positive region denotes the right boundary angle.
Next, these angles are used to determine the weighted-gradient
Hough Transform [13] for line construction. In the Hough
Transform measurement, the lane’s radius r() is generated
based on the following expression:
r(q ) = xo cosq + yo sinq (3)
where xo and yo are the coordinates corresponding to the
angles (xo ,yo) for both left and right lane boundaries. The
voting bins for each radius are cumulated with the gradient
edge values and the maximum vote is selected. Finally, the
measured angles and radii are used to build the left-right lane
boundary.
III. LANE REGION TRACKING
As illustrated in Fig. 5, the left right lane of interest (LOIs)
is given and drawn in black lines up to sky-road dividing line
with T1-pixel thick where T1 is a varied width for the lane
boundaries. These LOIs are lane masks used to obtain possible
lane edges in continuous video frame. At this stage, a simple
linear-parabolic tracking model proposed in [13]-[15] is used
to construct tracked lane region. Initially, an image is split into
near and far-field where ym is the border between near and
far-fields. A linear model is applied to follow the straight line
in the near-field while a parabolic model is used to mimic the
far-field lane border.
Fig. 5: Detected LOIs in road image
The left lane boundary model f(y)is defined as [14]:
+ + £
+ >
=
m
m
c dy ey if y y
a by if y y
f y
,
,
( ) 2 (4)
where
2
2a y (b d)
c m - +
= and
ym
b d
e
2
= - , which are acquired
after the imposition of continuity and differentiability
conditions on the function f(y).
Let (xni,yni) and Mni for i = 1,,m , denote the m
coordinates of the non-zero pixels of the edge image and its
corresponding magnitudes in near-field. On the other hand, let
(xfj,yfj) and Mfj for j = 1,, n represents the n coordinates
and the n edge pixels in far-field. Subsequently, the expression
in (4) can be rearranged into the n+m formula below:
+ + - = =
+ -
+ = =
=
y x j n
y
b d
dy
a y b d
a by x i m
f y
fj fj
m
fj
m
ni ni
, 1
2
( )
2
2 ( )
, 1
( ) 2 (5)
The approximated solution can then be found by minimizing
the error function as in (6).
= =
= - + -
m
i
n
j
E Mni xni f yni M fj x fj f y fj
1 1
[ ( )]2 [ ( )]2 (6)
The error is minimized when the following 3×3 linear system
is solved:
ATWAc = ATWb (7)
Where
+ - -
= + - -
2 2 2
2
1
2 2
1
( )
2
1
( )
2
1
1
( )
2
1
( )
2
1
1
1 0
1 0
fn m
m
fn m
m
f m
m
f m
m
nm
ni
y y
y
y y
y
y y
y
y y
y
y
y
A
W = diag(Mn1MnmM f 1M fn )
c = [a,b, d]T
[ ]T
b xn1 , , xnm , x f 1 , , x fn =
Hence, the left boundary parameters are estimated and used
to construct the left lane model. A similar operation is
employed to the right boundary. After the tracking system is
completed, the construction of lane region is needed to limit
the vehicle detection boundary. The linear tracking model is
applied to construct the lane region as demonstrated below:
>= + <= +
=
if others
I x y if x a b y x a b y
LROI r r l l
0 ,
( , ) , &
(8)
Orientation
Gradient Values
x
y
ym
Far-field
Near-field
Proceedings of the International MultiConference of Engineers and Computer Scientists 2009 Vol II
IMECS 2009, March 18 - 20, 2009, Hong Kong
ISBN: 978-988-17012-7-5 IMECS 2009
where (al,bl) and (ar,br) are the estimated parameters for left
and right linear lane models while I(x,y) denotes the original
image. The detected lane region of interest (LROI) is used to
restrict the vehicle searching area, which largely reduces the
computational time.
IV. VEHICLE DETECTION
After the retrieval of the lane region from the lane region
tracking in Section III, the system proceeds to vehicle
detection in order to locate the position of frontal vehicles.
Three stages are needed in identifying a vehicle: (i) shadow
detection, (ii) bounding box extraction and (iii) proposed
verification.
A. Shadow Detection
The LROI produced in previous section is used to detect the
shadow underneath a vehicle. At first, a gradient edge
transformation is applied to the LROI image with a certain
edge threshold as the binary image is shown in Fig. 6(a). The
remaining region might be a shadow or some noise produced
by lane edges. Further steps are taken to remove the noise. A
group of connected pixels which has more than T2 pixels in
horizontal is defined as the possible shadow region where T2
denotes as the threshold value for shadow detection.
Therefore, this process removes the pixels connectivity that is
less than T2 pixels remaining those possible pixels that are
larger than T2 pixels and those remaining pixels are defined as
pixels of interests (POIs), as depicted in Fig. 6(b).
Fig. 6: (a) Shadow detection, (b) POIs after noise filtering
B. Bounding Box Extraction
Vehicle verification is an important process in order to test
the detected region whether it is vehicle or non-vehicle. First,
an extracting box is used to determine the vehicle’s ROI,
where the POIs exist. Since the bottom pixels of possible
vehicle’s shadow are detected, the height from the detected
shadow pixels to the top image is calculated as Hpix while the
image height is defined as Himg. So the ratio of the image is
calculated as follows:
img
pix
ratio H
H
V = (9)
Fig. 7: Vehicle’s ROI after the adaptive bonding box extraction.
The ratio has the advantage that, if the car is at far-field, the
ratio is smaller and if the car is getting nearer, the ratio
becomes larger. This adaptive ratio is then multiplied with a
threshold (T3) to obtain the possible vehicle ROI’s height and
width for evaluation, as depicted in Fig. 7.
C. Proposed Verification
For further verification of vehicle’s ROI, symmetry
detection is performed horizontally and followed by the
vertical symmetry detection. The horizontal grayscale
symmetry axis can be found in [3] with the formula below:
+
=
+
= D =
= + D - - D
X W
j X
Y H
i Y
W
x
l
l
h
h
HS j G i j x G i j x
/ 2
1
( ) ( , ) ( , ) (10)
jsym = argmin HS( j)
where HS(j) is the horizontal symmetry measurement with the
symmetry axis located at x=j. As illustrated in Fig. 8, the
horizontal symmetry axis of the possible vehicle region occurs
at the local minimum where the point x = 31.
Fig. 8: The horizontal symmetry measurement.
The same idea is applied on the vertical asymmetry
detection with some modifications. Instead of analyzing the
grayscale symmetry, the vehicle region is turned into an edge
difference map by differencing all the columns to its first
column of the possible ROI and the difference map formula is
defined as follows:
+
=
+
=
= -
Y H
i Y
X W
j X
h
h
l
l
VS (i ) G (i , j ) G (i ,1) (11)
isym = argmaxVS(i)
where VS(i) is the vertical asymmetry measure with the
asymmetry axis located at y=i.
Fig. 9: (a) Vertical asymmetry measurement, (b) Vehicle’s difference map.
Image x-axis (j)
HS(j) Values
x=31
VS(i) Values
Image y-axis (i)
y=34
Proceedings of the International MultiConference of Engineers and Computer Scientists 2009 Vol II
IMECS 2009, March 18 - 20, 2009, Hong Kong
ISBN: 978-988-17012-7-5 IMECS 2009
As illustrated in Fig. 9(b), the color distribution at the center
row has the largest intensity compared to others due to the
great dissimilarity of vehicle’s region to surrounding
environment. The sum of each row is plotted on the graph, and
at the same time, the global maximum in the graph is selected
as the vertical asymmetry axis. As depicted in Fig. 9(a), the
peak occurs at the center of vehicle region where the vehicle
center is largely differentiated to others at the point y = 34.
Finally, the results of symmetry and asymmetry axes are
plotted on the vehicle’s ROI as shown in Fig. 10. The center of
vehicle can easily be obtained after the horizontal symmetry
and vertical asymmetry analysis and it can be extended for
vehicle tracking.
Fig. 10: Detected vehicle regions with symmetrical and asymmetrical axes.
V. SIMULATION RESULTS
All results were generated using Matlab 2006b with the Core
2 Duo processor at 1.8GHz with 1GB RAM. A video sequence
was captured under sunny day condition at around 12 p.m. in a
highway environment using Canon IXUS 65. The first image,
as depicted in Fig. 11(a), was used for the lane boundary
detection while the consecutive 12 video samples showed in
Fig. 12 and Fig. 13 were used for the evaluation of lane
tracking model and the vehicle detection. Initial values of the
following parameters, T1 = T2 = 8 and T3 = 100 were set in
these experiments.
The application of lane detection on road image is a critical
step since it determines the results and performance of the
following stages - lane tracking and vehicle detection. The
detection results demonstrated in Fig. 11(a) retrieved the
driving path successfully in frontal view. The detected lane
boundaries were therefore used to estimate the left-right LOIs
in the next step. Some other lane detection results were
demonstrated in Fig. 11(b) & © (obtained from WWW.) and
left-right lane boundaries were expected in the images.
The employment of previously detected lane outputs on the
Fig. 11: The result of lane detection (a) for the first sample of video frame,
(b) before tunnel, © in city.
first frame has activated the lane tracking process. The
purpose of lane tracking system is to predict the possible lane
model based on the detection result in order to reduce the
computational cost in a video frame. The linear-parabolic lane
model was applied to the video sequence. As a result, the lane
tracking algorithm extracted the lane region successfully, as
depicted in Fig. 12. However, the lane tracking method was
only an estimation using a lane model and it could not follow
the path exactly. As we could observe in Fig. 12(b)-(g), the
lane boundary was slightly above the lane markings, but at this
stage, road region was of interest since it was used to restrict
the vehicle searching region. With this knowledge of lane
region, the presence of vehicle could be detected in continuous
frames.
After the lane region was obtained and tracked for every
frame, vehicle searching area was limited to ground area
because vehicles were always found on the roadway. This
greatly reduced frontal vehicle scanning time and removed
noises that looked similar to be the priori knowledge of vehicle
regions such as horizontal/vertical structures of buildings and
edges. As demonstrated in Fig. 13, the vehicle region was
successfully detected in every frame and each frame indicated
the vehicle region with a self-adjusting bounding box. The
drawing of a box is based on an adjustable image ratio and it
was always well fitted to the vehicle region. At the same time,
the horizontal symmetry and vertical asymmetry measurement
described in Section IV were analyzed for vehicle verification.
The intersection of horizontal symmetry axis and vertical
asymmetry axis denoted as the vehicle’s center. Every video
frame obtained the center of vehicle correctly with the
symmetry analysis except for the Fig. 13(i).
VI. CONCLUSION
A monocular lane-vehicle detection and tracking system has
been presented in this paper with an integration of three
components, i.e. (i) lane boundary detection, (ii) lane region
tracking, and (iii) vehicle detection with a proposed vertical
asymmetry measurement. The advantages of lane-vehicle
detection and tracking system are, (i) the reduction of vehicle
searching time, and (ii) the increase performance of the
vehicle detection based on the priori knowledge regardless to
the environmental interference caused by non-vehicle
structures and sky region. At the same time, lane detection and
tracking system without any camera parameter can be applied
to other driver assistance function for the determination of
ROIs. Furthermore, horizontal symmetry analysis with an
assistance of vertical asymmetry analysis can easily obtain the
center part of vehicle and this center point can be used for
vehicle tracking in future. In the future, further investigation
will be carried out on lane-vehicle detection and tracking
system under various road conditions.
[attachment=23746]
Abstract
This paper presents a monocular lane-vehicle
detection and tracking system comprising of (i) lane boundary
detection, (ii) lane region tracking, and (iii) vehicle detection
with a proposed vertical asymmetry measurement. First, a traffic
scene image is divided into possible road region. Lane boundaries
are then extracted from the region using lane markings
detection. These detected boundaries are tracked in consecutive
video frames using a linear-parabolic tracking model. Therefore,
an approximate lane region is constructed with the estimated
model parameters. By integrating the knowledge of lane region
with vehicle detection, vehicle scanning region is restricted to the
road area so as to detect the shadow underneath a vehicle
continuously with less interference from the road environment
and non-vehicle structures. A self-adjusting bounding box is used
to extract likely vehicle region for further verification. Besides
horizontal symmetry detection, a vertical asymmetry
measurement is proposed to validate the extracted region and to
obtain the center of frontal vehicle. Preliminary simulation
results revealed good performance of lane-vehicle detection and
tracking system.
Index Terms— Driver assistance systems, Lane detection,
Lane tracking, Vehicle detection.
I. INTRODUCTION
Advancement of vision-based vehicle detection has
triggered recent development of autonomous vehicular
technology in order to automatically localize moving vehicles
in complex traffic scene. Generally, the main task of
vision-based vehicle detection is to sense a leading vehicle and
thus spontaneously alert a driver of precritical conditions with
the preceding vehicle, in case he/she makes a sudden break,
slowdown driving or uniform movement [1]. Moreover,
vehicle detection can be developed to track and follow a
preceding car, and at the same time increase the efficient use
of driving space. In consequence, an effective visual scanning
of frontal road conditions is highly required for vehicle
detection system to identify the position of frontal vehicle.
Numerous vision-based techniques have been developed
over the past decade to detect vehicles in various road scenes,
as described in [1]-[11]. Tzomakas and Seelen [2] detected
vehicles based on the shadows underneath them. With the use
of entropy, a free driving space was determined and therefore,
an adaptive threshold was applied to extract vehicle edges.
Chang et al. [3] presented a preceding vehicle detection
Manuscript received October 16, 2008. Manuscript submitted to the
IAENG International Conference on Control and Automation, ICCA’09.
The authors are with the School of Electrical and Electronics Engineering,
The University of Nottingham, Malaysia Campus, Jalan Broga, Semenyih,
43500, Selangor, Malaysia. (Phone: +603-8924-8350, Fax:
+603-8924-8071; Email: {keyx7khl, Jasmine.Seng, Kenneth.Ang,
keyx8csw}[at]nottingham.edu.my).
and tracking system by finding the footprint of a vehicle on
road area and tracking the vehicle using the continuity
measurement of two consecutive frames. Kate et al. [4]
combined the knowledge-based methods such as shadow
detection, entropy analysis and horizontal symmetry
measurement for mid-range and distant vehicle detection
without prior knowledge about the road geometry.
Besides that, Khammari et al. [5] applied a horizontal
Sobel filter on the 3rd level of the Gaussian pyramid to obtain
local gradient maxima where a vehicle candidate is located. A
temporal filter was used to further remove unwanted pixels
and then a bounding box extraction was employed to retrieve a
possible vehicle region for symmetry verification. Broggi et
al. [6] presented a multi-resolution vehicle detection method
to localize vehicles with variable sizes. They computed the
symmetry property of vehicles in different sized bounding
boxes on all the columns of the regions. Liu et al. [7][8]
detected vehicle region based on the shadow underneath a
vehicle and symmetry edges. Additionally, they combined
knowledge-based and learning-based methods for vehicle
verification. In vehicle tracking, templates were dynamically
created on-line and tracking window was adaptively adjusted
with motion estimation.
On the other hand, Hoffman [9] performed a multi-sensor
fusion approach incorporating 2-D visual features such as
shadow and symmetry, with 3-D ground plane information
such as camera height for vehicle detection. These fusing
features were tracked using Interacting Multiple Model
method. Bertozzi and Broggi [10] used stereo vision-based
method to detect both generic obstacles and lane positions in a
structured environment. They utilized Inverse Perspective
Mapping technique to remove perspective effect and
reconstruct a 3-D mapping when the camera parameters and
the knowledge about road are known. Giachetti et al. [11]
developed first-order and second-order differential methods to
detect vehicle based on the motion.
However, the presence of non-vehicle structures such as
over-bridge, fly-over roadway, tunnel, buildings, sign boards
etc, in traffic scene may decrease the performance of
knowledge-based vehicle detection since these non-vehicle
structures posses the horizontal/vertical characteristics
identical to vehicle’s edges [2]-[8]. Moreover, a complex road
environment may complicate the process of vehicle detection
as there are many possibilities of human activities along the
road side [4]. Frontal vehicle with little relative motion change
or stand still has increased the difficulty of vehicle detection
based on motion flow [1][11]. Furthermore, the requirement
of 3-D transformation and the knowledge of hardware
parameters for stereo-based vehicle detection method have
highly increased the computational cost and reduced the
processing speed [9][10].
Lane-Vehicle Detection and Tracking
King Hann LIM*, Li-Minn ANG, Kah Phooi SENG and Siew Wen CHIN
Proceedings of the International MultiConference of Engineers and Computer Scientists 2009 Vol II
IMECS 2009, March 18 - 20, 2009, Hong Kong
ISBN: 978-988-17012-7-5 IMECS 2009
Fig. 1: Proposed lane-vehicle detection and tracking system.
To solve abovementioned problems, a novel monocular
lane-vehicle detection and tracking system is proposed in this
paper, as the flow diagram is depicted in Fig. 1. Lane boundary
detection is performed on road image to locate the left-right
lane edges by separating the sky-road region, and analyzing
road region to extract the prominent road features such as lane
markings. A linear-parabolic lane model without the
requirement of camera parameters is constructed to estimate
lane geometry. This is followed by lane region tracking to
restrict the vehicle searching region on the ground for every
continuous frame. As the vehicle region is bounded, vehicle
detection based on the shadow underneath a vehicle is
repeatedly performed in tracking lane region. In addition, a
bounding box extraction method is used to obtain a likely
vehicle region for further verification. Besides horizontal
symmetry detection, a proposed vertical asymmetry
measurement is applied to verify the extracted region and at
the same time, the center of vehicle is found.
Section II discusses about the lane boundary detection
while Section III explains the lane region tracking technique.
The vehicle detection with proposed verification technique is
discussed in Section IV. Some simulation results are shown in
Section V and followed by conclusion and future works.
II. LANE BOUNDARY DETECTION
Lane boundary detection is important to locate the left-right
edges of driving path on a traffic scene image. In this section, a
three-stage lane boundary detection is performed, i.e. (i)
vertical mean distribution, (ii) lane region analysis, and (iii)
lane marking detection.
A. Vertical Mean Distribution
In lane detection process, sky region is not a region of
interest (ROI). At the preliminary stage, a traffic scene image
I(x,y) is divided into sky region and road region using vertical
mean distribution [12]. Vertical mean distribution is measured
by averaging the gray values of each row on road image and
the row means are plotted in the graph depicted in Fig. 2(a).
The threshold value of horizon line is obtained through a
minimum search along the vertical mean curve, where the first
Fig. 2: (a) Vertical mean distribution, (b) Road region image (Rroi).
minimum occurs from the upper curve is the regional dividing
line. This is because sky region usually possesses higher
intensity than road pixels, and it might have a big jump of
intensity difference as sky pixels approaches ground. The
horizon line threshold is applied to generate a road image (Rroi)
as demonstrated in Fig 2(b), where all vertical coordinates
below the threshold are discarded.
B. Lane Region Analysis
Lane region analysis is performed to further classify road
region and lane markings. Usually, the bottom region in a road
image contains road pixels. By acquiring a few rows from the
bottom of image, the gray value range of road color can easily
be obtained and therefore, this color range is applied to further
remove road pixels from the Rroi map. The lane region
analysis steps are shown as follows:
Step 1: Pick 30-60 rows from bottom to avoid the
possible existence of inner part of a vehicle at the
edge of image.
Step 2: Build a vote scheme; namely VOTE for the
selected rows and the maximum vote of the row
pixels is recorded.
Step 3: Record a global maximum value for the selected
rows, as MAXTHRES.
Step 4: Define the road color range as [VOTE-25;
MAXTHRES+25]. Pixels that fall within this
range are denoted as road pixels and a binary map
(Rbin) is formed as shown in Fig. 3(a).
Step 5: Generate a difference map (Dmap) by differencing
Rroi and Rbin maps. The positive pixel values are
retained while the negative values are set to zero.
The difference map is illustrated in Fig. 3(b).
Fig. 3: (a) Lane binary map (Rbin), (b) Difference map (Dmap).
C. Lane Marking Detection
The difference map obtained in the previous stage is,
therefore used for lane marking detection. Lane markings are
the salient features on road and they are usually used to extract
the boundary of road region. Initially, an edge detection using
Sobel filter is applied to Dmap. The gradient magnitude
ÑDmap (x, y) and orientation q (x, y) are calculated in the
following equations [13]:
ÑDmap (x, y) » Dx + Dy (1)
)
( ( , ) ( , ))
2 ( , ) ( , )
tan (
2
1
( , )
( , )
2 2
1 ( , )
Î
- Î
-
=
u v R x y
u v R x y
D u v D u v
D u v D u v
q x y (2)
where u and v are the 3×3 mask coordinates. Dx is the
horizontal edge map whereas Dy is the vertical edge map.
Mean Values
Image Row
Horizon Line
Proceedings of the International MultiConference of Engineers and Computer Scientists 2009 Vol II
IMECS 2009, March 18 - 20, 2009, Hong Kong
ISBN: 978-988-17012-7-5 IMECS 2009
Fig. 4: The edge distribution function.
With reference to the constructed gradient map and its
corresponding orientation, a histogram called edge
distribution function [13] is build, with its x-axis representing
the orientation ranging from [-90º; 90º] and its y-axis
representing the gradient value of each orientation bins. The
maximum peak acquired in the graph on the negative angle
denotes the left boundary angle. Conversely, the maximum
peak at the positive region denotes the right boundary angle.
Next, these angles are used to determine the weighted-gradient
Hough Transform [13] for line construction. In the Hough
Transform measurement, the lane’s radius r() is generated
based on the following expression:
r(q ) = xo cosq + yo sinq (3)
where xo and yo are the coordinates corresponding to the
angles (xo ,yo) for both left and right lane boundaries. The
voting bins for each radius are cumulated with the gradient
edge values and the maximum vote is selected. Finally, the
measured angles and radii are used to build the left-right lane
boundary.
III. LANE REGION TRACKING
As illustrated in Fig. 5, the left right lane of interest (LOIs)
is given and drawn in black lines up to sky-road dividing line
with T1-pixel thick where T1 is a varied width for the lane
boundaries. These LOIs are lane masks used to obtain possible
lane edges in continuous video frame. At this stage, a simple
linear-parabolic tracking model proposed in [13]-[15] is used
to construct tracked lane region. Initially, an image is split into
near and far-field where ym is the border between near and
far-fields. A linear model is applied to follow the straight line
in the near-field while a parabolic model is used to mimic the
far-field lane border.
Fig. 5: Detected LOIs in road image
The left lane boundary model f(y)is defined as [14]:
+ + £
+ >
=
m
m
c dy ey if y y
a by if y y
f y
,
,
( ) 2 (4)
where
2
2a y (b d)
c m - +
= and
ym
b d
e
2
= - , which are acquired
after the imposition of continuity and differentiability
conditions on the function f(y).
Let (xni,yni) and Mni for i = 1,,m , denote the m
coordinates of the non-zero pixels of the edge image and its
corresponding magnitudes in near-field. On the other hand, let
(xfj,yfj) and Mfj for j = 1,, n represents the n coordinates
and the n edge pixels in far-field. Subsequently, the expression
in (4) can be rearranged into the n+m formula below:
+ + - = =
+ -
+ = =
=
y x j n
y
b d
dy
a y b d
a by x i m
f y
fj fj
m
fj
m
ni ni
, 1
2
( )
2
2 ( )
, 1
( ) 2 (5)
The approximated solution can then be found by minimizing
the error function as in (6).
= =
= - + -
m
i
n
j
E Mni xni f yni M fj x fj f y fj
1 1
[ ( )]2 [ ( )]2 (6)
The error is minimized when the following 3×3 linear system
is solved:
ATWAc = ATWb (7)
Where
+ - -
= + - -
2 2 2
2
1
2 2
1
( )
2
1
( )
2
1
1
( )
2
1
( )
2
1
1
1 0
1 0
fn m
m
fn m
m
f m
m
f m
m
nm
ni
y y
y
y y
y
y y
y
y y
y
y
y
A
W = diag(Mn1MnmM f 1M fn )
c = [a,b, d]T
[ ]T
b xn1 , , xnm , x f 1 , , x fn =
Hence, the left boundary parameters are estimated and used
to construct the left lane model. A similar operation is
employed to the right boundary. After the tracking system is
completed, the construction of lane region is needed to limit
the vehicle detection boundary. The linear tracking model is
applied to construct the lane region as demonstrated below:
>= + <= +
=
if others
I x y if x a b y x a b y
LROI r r l l
0 ,
( , ) , &
(8)
Orientation
Gradient Values
x
y
ym
Far-field
Near-field
Proceedings of the International MultiConference of Engineers and Computer Scientists 2009 Vol II
IMECS 2009, March 18 - 20, 2009, Hong Kong
ISBN: 978-988-17012-7-5 IMECS 2009
where (al,bl) and (ar,br) are the estimated parameters for left
and right linear lane models while I(x,y) denotes the original
image. The detected lane region of interest (LROI) is used to
restrict the vehicle searching area, which largely reduces the
computational time.
IV. VEHICLE DETECTION
After the retrieval of the lane region from the lane region
tracking in Section III, the system proceeds to vehicle
detection in order to locate the position of frontal vehicles.
Three stages are needed in identifying a vehicle: (i) shadow
detection, (ii) bounding box extraction and (iii) proposed
verification.
A. Shadow Detection
The LROI produced in previous section is used to detect the
shadow underneath a vehicle. At first, a gradient edge
transformation is applied to the LROI image with a certain
edge threshold as the binary image is shown in Fig. 6(a). The
remaining region might be a shadow or some noise produced
by lane edges. Further steps are taken to remove the noise. A
group of connected pixels which has more than T2 pixels in
horizontal is defined as the possible shadow region where T2
denotes as the threshold value for shadow detection.
Therefore, this process removes the pixels connectivity that is
less than T2 pixels remaining those possible pixels that are
larger than T2 pixels and those remaining pixels are defined as
pixels of interests (POIs), as depicted in Fig. 6(b).
Fig. 6: (a) Shadow detection, (b) POIs after noise filtering
B. Bounding Box Extraction
Vehicle verification is an important process in order to test
the detected region whether it is vehicle or non-vehicle. First,
an extracting box is used to determine the vehicle’s ROI,
where the POIs exist. Since the bottom pixels of possible
vehicle’s shadow are detected, the height from the detected
shadow pixels to the top image is calculated as Hpix while the
image height is defined as Himg. So the ratio of the image is
calculated as follows:
img
pix
ratio H
H
V = (9)
Fig. 7: Vehicle’s ROI after the adaptive bonding box extraction.
The ratio has the advantage that, if the car is at far-field, the
ratio is smaller and if the car is getting nearer, the ratio
becomes larger. This adaptive ratio is then multiplied with a
threshold (T3) to obtain the possible vehicle ROI’s height and
width for evaluation, as depicted in Fig. 7.
C. Proposed Verification
For further verification of vehicle’s ROI, symmetry
detection is performed horizontally and followed by the
vertical symmetry detection. The horizontal grayscale
symmetry axis can be found in [3] with the formula below:
+
=
+
= D =
= + D - - D
X W
j X
Y H
i Y
W
x
l
l
h
h
HS j G i j x G i j x
/ 2
1
( ) ( , ) ( , ) (10)
jsym = argmin HS( j)
where HS(j) is the horizontal symmetry measurement with the
symmetry axis located at x=j. As illustrated in Fig. 8, the
horizontal symmetry axis of the possible vehicle region occurs
at the local minimum where the point x = 31.
Fig. 8: The horizontal symmetry measurement.
The same idea is applied on the vertical asymmetry
detection with some modifications. Instead of analyzing the
grayscale symmetry, the vehicle region is turned into an edge
difference map by differencing all the columns to its first
column of the possible ROI and the difference map formula is
defined as follows:
+
=
+
=
= -
Y H
i Y
X W
j X
h
h
l
l
VS (i ) G (i , j ) G (i ,1) (11)
isym = argmaxVS(i)
where VS(i) is the vertical asymmetry measure with the
asymmetry axis located at y=i.
Fig. 9: (a) Vertical asymmetry measurement, (b) Vehicle’s difference map.
Image x-axis (j)
HS(j) Values
x=31
VS(i) Values
Image y-axis (i)
y=34
Proceedings of the International MultiConference of Engineers and Computer Scientists 2009 Vol II
IMECS 2009, March 18 - 20, 2009, Hong Kong
ISBN: 978-988-17012-7-5 IMECS 2009
As illustrated in Fig. 9(b), the color distribution at the center
row has the largest intensity compared to others due to the
great dissimilarity of vehicle’s region to surrounding
environment. The sum of each row is plotted on the graph, and
at the same time, the global maximum in the graph is selected
as the vertical asymmetry axis. As depicted in Fig. 9(a), the
peak occurs at the center of vehicle region where the vehicle
center is largely differentiated to others at the point y = 34.
Finally, the results of symmetry and asymmetry axes are
plotted on the vehicle’s ROI as shown in Fig. 10. The center of
vehicle can easily be obtained after the horizontal symmetry
and vertical asymmetry analysis and it can be extended for
vehicle tracking.
Fig. 10: Detected vehicle regions with symmetrical and asymmetrical axes.
V. SIMULATION RESULTS
All results were generated using Matlab 2006b with the Core
2 Duo processor at 1.8GHz with 1GB RAM. A video sequence
was captured under sunny day condition at around 12 p.m. in a
highway environment using Canon IXUS 65. The first image,
as depicted in Fig. 11(a), was used for the lane boundary
detection while the consecutive 12 video samples showed in
Fig. 12 and Fig. 13 were used for the evaluation of lane
tracking model and the vehicle detection. Initial values of the
following parameters, T1 = T2 = 8 and T3 = 100 were set in
these experiments.
The application of lane detection on road image is a critical
step since it determines the results and performance of the
following stages - lane tracking and vehicle detection. The
detection results demonstrated in Fig. 11(a) retrieved the
driving path successfully in frontal view. The detected lane
boundaries were therefore used to estimate the left-right LOIs
in the next step. Some other lane detection results were
demonstrated in Fig. 11(b) & © (obtained from WWW.) and
left-right lane boundaries were expected in the images.
The employment of previously detected lane outputs on the
Fig. 11: The result of lane detection (a) for the first sample of video frame,
(b) before tunnel, © in city.
first frame has activated the lane tracking process. The
purpose of lane tracking system is to predict the possible lane
model based on the detection result in order to reduce the
computational cost in a video frame. The linear-parabolic lane
model was applied to the video sequence. As a result, the lane
tracking algorithm extracted the lane region successfully, as
depicted in Fig. 12. However, the lane tracking method was
only an estimation using a lane model and it could not follow
the path exactly. As we could observe in Fig. 12(b)-(g), the
lane boundary was slightly above the lane markings, but at this
stage, road region was of interest since it was used to restrict
the vehicle searching region. With this knowledge of lane
region, the presence of vehicle could be detected in continuous
frames.
After the lane region was obtained and tracked for every
frame, vehicle searching area was limited to ground area
because vehicles were always found on the roadway. This
greatly reduced frontal vehicle scanning time and removed
noises that looked similar to be the priori knowledge of vehicle
regions such as horizontal/vertical structures of buildings and
edges. As demonstrated in Fig. 13, the vehicle region was
successfully detected in every frame and each frame indicated
the vehicle region with a self-adjusting bounding box. The
drawing of a box is based on an adjustable image ratio and it
was always well fitted to the vehicle region. At the same time,
the horizontal symmetry and vertical asymmetry measurement
described in Section IV were analyzed for vehicle verification.
The intersection of horizontal symmetry axis and vertical
asymmetry axis denoted as the vehicle’s center. Every video
frame obtained the center of vehicle correctly with the
symmetry analysis except for the Fig. 13(i).
VI. CONCLUSION
A monocular lane-vehicle detection and tracking system has
been presented in this paper with an integration of three
components, i.e. (i) lane boundary detection, (ii) lane region
tracking, and (iii) vehicle detection with a proposed vertical
asymmetry measurement. The advantages of lane-vehicle
detection and tracking system are, (i) the reduction of vehicle
searching time, and (ii) the increase performance of the
vehicle detection based on the priori knowledge regardless to
the environmental interference caused by non-vehicle
structures and sky region. At the same time, lane detection and
tracking system without any camera parameter can be applied
to other driver assistance function for the determination of
ROIs. Furthermore, horizontal symmetry analysis with an
assistance of vertical asymmetry analysis can easily obtain the
center part of vehicle and this center point can be used for
vehicle tracking in future. In the future, further investigation
will be carried out on lane-vehicle detection and tracking
system under various road conditions.