10-08-2012, 04:58 PM
Dynamic 3D Scene Analysis from a Moving Vehicle
Abstract
In this paper, we present a system that integrates fully automatic scene geometry estimation, 2D object detection, 3D localization, trajectory estimation, and tracking for dynamic scene interpretation from a moving vehicle. Our sole inputs are two video streams from a calibrated stereo rig on
top of a car. From these streams, we estimate Structure-from-Motion (SfM) and scene geometry in real-time. In parallel, we perform multi-view/multi-category object recognition to detect cars and pedestrians in both camera images. Using the SfM self-localization, 2D object detections are
converted to 3D observations, which are accumulated in a world coordinate frame. A subsequent tracking module analyses the resulting 3D observations to find physically plausible space-time trajectories. Finally, a global optimization
criterion takes object-object interactions into account to arrive at accurate 3D localization and trajectory estimates for both cars and pedestrians. We demonstrate the performance of our integrated system on challenging real-world data showing car passages through crowded city areas.
Introduction
The task we address in this paper is dynamic scene anal-
ysis from a moving, camera-equipped vehicle. At any point
in time, we want to detect other traffic participants in the en-
vironment (cars, bicyclists, and pedestrians), localize them
in 3D, estimate their past trajectories, and predict their fu-
ture motion (as shown in Fig. 1). Such a capability has ob-
vious applications in driver assistance systems, but it also
serves as a testbed for many interesting research challenges.
Scene analysis of this sort requires multi-viewpoint,
multi-category object detection. Since we cannot control
the vehicle’s path, nor the environment it passes through,
the detectors need to be robust to a large range of light-
ing variations, noise, clutter, and partial occlusion. For 3D
localization, an accurate estimate of the scene geometry is
necessary. The ability to integrate such measurements over
time additionally requires continuous self-localization and
recalibration. In order to finally make predictions about fu-
ture states, powerful tracking is needed that can cope with a
Abstract
In this paper, we present a system that integrates fully automatic scene geometry estimation, 2D object detection, 3D localization, trajectory estimation, and tracking for dynamic scene interpretation from a moving vehicle. Our sole inputs are two video streams from a calibrated stereo rig on
top of a car. From these streams, we estimate Structure-from-Motion (SfM) and scene geometry in real-time. In parallel, we perform multi-view/multi-category object recognition to detect cars and pedestrians in both camera images. Using the SfM self-localization, 2D object detections are
converted to 3D observations, which are accumulated in a world coordinate frame. A subsequent tracking module analyses the resulting 3D observations to find physically plausible space-time trajectories. Finally, a global optimization
criterion takes object-object interactions into account to arrive at accurate 3D localization and trajectory estimates for both cars and pedestrians. We demonstrate the performance of our integrated system on challenging real-world data showing car passages through crowded city areas.
Introduction
The task we address in this paper is dynamic scene anal-
ysis from a moving, camera-equipped vehicle. At any point
in time, we want to detect other traffic participants in the en-
vironment (cars, bicyclists, and pedestrians), localize them
in 3D, estimate their past trajectories, and predict their fu-
ture motion (as shown in Fig. 1). Such a capability has ob-
vious applications in driver assistance systems, but it also
serves as a testbed for many interesting research challenges.
Scene analysis of this sort requires multi-viewpoint,
multi-category object detection. Since we cannot control
the vehicle’s path, nor the environment it passes through,
the detectors need to be robust to a large range of light-
ing variations, noise, clutter, and partial occlusion. For 3D
localization, an accurate estimate of the scene geometry is
necessary. The ability to integrate such measurements over
time additionally requires continuous self-localization and
recalibration. In order to finally make predictions about fu-
ture states, powerful tracking is needed that can cope with a