12-07-2012, 11:30 AM
Cooperative Adaptive Cruise Control: A Reinforcement Learning Approach
[attachment=27479]
INTRO DUCTION
ECENT improvements in sensing, communicating, and computing technologies have led to the development of
driver-assistance systems (DASs). Such systems aim at assist- ing drivers by either providing warning to reduce crashes or performing some of the control tasks to relieve a driver from repetitive and boring tasks. Thus, a DAS can replace a driver’s decisions and actions in some routines, without possible errors, which can lead to accidents, while achieving more regulated and smoother vehicle control [1].
ADAPTIVE CRU I S E CONTRO L : FRO M AUTONOMOUS TO COOPERATIVE SYSTEMS
New technology that actively intervenes and controls car driving may have a very great effect on comfort, safety, and traf- fic flow. ACC and CWA are representative of this technology, and currently, ACC is becoming widely available in passenger cars. The objective of ACC consists of automatically maintain- ing safe cruise driving, thus relieving a driver from manually performing a repetitive and boring task. When driving in free traffic, the ACC system holds a preset speed similar to any conventional cruise control system. When, on the other hand, a driver has to follow another vehicle, the system automatically maintains a desired time gap from the preceding vehicle [4]. An ACC can be conceived through an autonomous approach (using ranging sensors) or a cooperative approach (using V2V and/or R2V communication). Ranging sensors, e.g., radars or lasers, are generally used to measure the range and the rates of this range from the preceding vehicle. In general, ACC systems switch off when the speed is less than 30 km/h, because most of these systems are developed for highway traffic. In fact, the autonomous ACC based on ranging sensors has limited antic- ipatory capabilities, because it is impossible to react to what happens in front of the immediate predecessor.
REINFORCEMENT LEARNING
Most of the projects on CACC have relied on the classic control theory to develop autonomous controllers. However, ongoing research from the machine-learning community has yielded promising theoretical and practical results for the reso- lution of control problems in uncertain and partially observable environments, and it would be opportune to test it on CACC. One of the first research efforts to use machine learning for autonomous vehicle control was Pomerleau’s autonomous land vehicle in a neural network (ALVINN) [21], which consisted of a computer vision system, based on a neural network, that learns to correlate observations of the road to the correct action to take. This autonomous controller drove a real vehicle by itself for more than 30 mi.
CONCLUSION
This paper has proposed a novel design approach to obtain an autonomous longitudinal vehicle controller. To achieve this condition, a vehicle architecture with its ACC subsystem has been presented. With this architecture, we have also described the specific requirements for an efficient autonomous vehicle control policy through RL and the simulator in which the learning engine is embedded.
A policy-gradient algorithm estimation has been introduced and has used a backpropagation neural network for achieving the longitudinal control. Then, experimental results, through simulation, have shown that this design approach can result in efficient behavior for CACC.
Much work can still be done to improve the vehicle controller proposed in this paper. First, it is clear that some modifications to the learning process should be made to improve the resulting vehicle-following behavior. Issues related to the oscillatory behavior of our vehicle control policy can be addressed by using continuous actions. This case would require further study to efficiently implement this approach, because it brings addi- tional complexity to the learning process.
[attachment=27479]
INTRO DUCTION
ECENT improvements in sensing, communicating, and computing technologies have led to the development of
driver-assistance systems (DASs). Such systems aim at assist- ing drivers by either providing warning to reduce crashes or performing some of the control tasks to relieve a driver from repetitive and boring tasks. Thus, a DAS can replace a driver’s decisions and actions in some routines, without possible errors, which can lead to accidents, while achieving more regulated and smoother vehicle control [1].
ADAPTIVE CRU I S E CONTRO L : FRO M AUTONOMOUS TO COOPERATIVE SYSTEMS
New technology that actively intervenes and controls car driving may have a very great effect on comfort, safety, and traf- fic flow. ACC and CWA are representative of this technology, and currently, ACC is becoming widely available in passenger cars. The objective of ACC consists of automatically maintain- ing safe cruise driving, thus relieving a driver from manually performing a repetitive and boring task. When driving in free traffic, the ACC system holds a preset speed similar to any conventional cruise control system. When, on the other hand, a driver has to follow another vehicle, the system automatically maintains a desired time gap from the preceding vehicle [4]. An ACC can be conceived through an autonomous approach (using ranging sensors) or a cooperative approach (using V2V and/or R2V communication). Ranging sensors, e.g., radars or lasers, are generally used to measure the range and the rates of this range from the preceding vehicle. In general, ACC systems switch off when the speed is less than 30 km/h, because most of these systems are developed for highway traffic. In fact, the autonomous ACC based on ranging sensors has limited antic- ipatory capabilities, because it is impossible to react to what happens in front of the immediate predecessor.
REINFORCEMENT LEARNING
Most of the projects on CACC have relied on the classic control theory to develop autonomous controllers. However, ongoing research from the machine-learning community has yielded promising theoretical and practical results for the reso- lution of control problems in uncertain and partially observable environments, and it would be opportune to test it on CACC. One of the first research efforts to use machine learning for autonomous vehicle control was Pomerleau’s autonomous land vehicle in a neural network (ALVINN) [21], which consisted of a computer vision system, based on a neural network, that learns to correlate observations of the road to the correct action to take. This autonomous controller drove a real vehicle by itself for more than 30 mi.
CONCLUSION
This paper has proposed a novel design approach to obtain an autonomous longitudinal vehicle controller. To achieve this condition, a vehicle architecture with its ACC subsystem has been presented. With this architecture, we have also described the specific requirements for an efficient autonomous vehicle control policy through RL and the simulator in which the learning engine is embedded.
A policy-gradient algorithm estimation has been introduced and has used a backpropagation neural network for achieving the longitudinal control. Then, experimental results, through simulation, have shown that this design approach can result in efficient behavior for CACC.
Much work can still be done to improve the vehicle controller proposed in this paper. First, it is clear that some modifications to the learning process should be made to improve the resulting vehicle-following behavior. Issues related to the oscillatory behavior of our vehicle control policy can be addressed by using continuous actions. This case would require further study to efficiently implement this approach, because it brings addi- tional complexity to the learning process.