Advanced Planning for Autonomous Vehicles Using Reinforcement Learning and Deep Inverse Reinforcement Learning

This project addresses path planning for autonomous vehicles in traffic using reinforcement learning and deep inverse reinforcement learning to model interactions and learn expert driving behaviors

Back to Research

Autonomous vehicles promise to improve traffic safety while, at the same time, increase fuel efficiency and reduce congestion. This project concentrates on the path planning problem of autonomous vehicles in traffic, shown below. Each vehicle has a set of actions to take, such as maintaining its current speed, switching to the left/right lane, speeding up and braking. We model the interaction between the autonomous vehicle and the environment as a stochastic Markov decision process (MDP).

An aerial photograph of dense traffic on a multi-lane highway, with a red rectangle drawn around one car and the three vehicles immediately surrounding it The traffic on multi-lane road.

The state of the MDP is defined using the positions of the autonomous vehicle, and the number and positions of the environmental vehicles around the autonomous vehicle. The road geometry is taken into consideration in the MDP model in order to incorporate more diverse driving styles.

A schematic of a multi-lane road divided into cells, with the host vehicle marked HV and surrounding vehicles in various colours; three dashed boxes mark an interior nine-cell neighbourhood in red and two six-cell boundary neighbourhoods in blue, each extracted and redrawn on the right The cells and the definition of the state: (1) 9-cell internal-lane state, (2) 6-cell left-boundary state and (3) 6-cell right-boundary state.

The core problem of an MDP is to find a policy π\pi for the agent, where the policy π:S→A\pi:S\rightarrow A specifies the action to take at the current state sts_t. The goal is to find the optimal policy π∗\pi^* that maximizes the cumulative discounted reward over an infinite horizon:

π∗=arg⁡max⁡π E[∑t=0∞γtR(st,π(st))],\pi^* = \arg \max_{\pi} ~ \mathbb{E}[\sum_{t=0}^{\infty} \gamma^t R(s_t,\pi(s_t))],

The desired, expert-like driving behavior of the autonomous vehicle can be obtained using two approaches.

First, we design the reward function of the corresponding MDP and determine the optimal driving strategy for the autonomous vehicle using reinforcement learning techniques.

Second, we collect a number of demonstrations from an expert driver and learn the optimal driving strategy based on data using inverse reinforcement learning. The unknown reward function of the expert driver is approximated using a deep neural-network (DNN).

A fully connected neural network with state and action components as inputs on the left and the reward R(s_t, a_t) as the single output on the right; a red bracket labels the hidden layers as the feature, with the weights theta, the feature vector Phi and the output weight w annotated in red Deep neural-network feature function and reward.

Movies demonstrating typical driving styles are available for tailgating and overtaking.

Sponsors

This work is sponsored by Ford and NSF.

Selected Publications and Presentations

1.

You, C., Lu, J., and Tsiotras, P., “Driver Parameter Estimation Using Joint E-/UKF and Dual E-/UKF Under Nonlinear State Inequality Constraints,” IEEE Conference on Systems, Man and Cybernetics, Budapest, Hungary, Oct. 9-12, 2016.

PDF DOI
2.

Okamoto, K., and Tsiotras, P., “A New Hybrid Sensorimotor Driver Model with Model Predictive Control,” IEEE Conference on Systems, Man and Cybernetics, Budapest, Hungary, Oct. 9-12, 2016, (best student paper finalist).

PDF DOI
3.

You, C., and Tsiotras, P., “Vehicle Modeling and Parameter Estimation Using Adaptive Limited Memory Joint-State UKF,” American Control Conference, Seattle, WA, May 24-26, 2017, pp. 322-327.

PDF DOI
4.

You, C., and Tsiotras, P., “Real-Time Trail-Braking Maneuver Generation for Off-Road Vehicle Racing,” American Control Conference, Milwaukee, WI, June 27-29, 2018, pp. 4751– 4756.

PDF DOI
5.

You, C., and Tsiotras, P., “High-Speed Cornering for Autonomous Off-Road Rally Racing,” IEEE Transactions on Control Systems Technology, Vol. 29, No. 2, pp. 485-501, 2021.

PDF DOI
6.

You, C., Lu, J., Filev, D., and Tsiotras, P., “Autonomous Planning and Control for Intelligent Vehicles in Traffic,” IEEE Transactions on Intelligent Transportation Systems, Vol. 21, No. 6, pp. 2339-2349, June 2020.

PDF DOI
7.

Okamoto, K., and Tsiotras, P., “Data-Driven Human Driver Lateral Control Models for Developing Haptic-Shared Control Advanced Driver Assist Systems,” Robotics and Autonomous Systems, Vol. 114, pp. 155-171, 2020.

PDF DOI
8.

You, C., Lu, J., Filev, D., and Tsiotras, P., “Advanced Planning for Autonomous Vehicles Using Reinforcement Learning and Deep Inverse Reinforcement Learning,” Robotics and Autonomous Systems, Vol. 114, pp. 1-18, April 2019.

PDF DOI Code