KARNet

KARNet (Kalman filter Augmented Recurrent Network) learns a compact latent representation of traffic flow from front-camera images alone, and uses it to predict how the scene evolves over the next few moments. Most end-to-end driving models carry no prior knowledge of the vehicle itself; KARNet instead folds an explicit model of the ego vehicle into the architecture, with the vehicle state estimated by an error-state extended Kalman filter from IMU and GNSS measurements. Images are encoded into latent space, fused with those state estimates and propagated by a gated recurrent unit, with a multilayer perceptron acting as the controller. Supplying the network with physics it would otherwise have to infer measurably improves performance in both imitation and reinforcement learning settings, on CARLA simulations and on the real-world Udacity dataset.

Reference

H. Manjunatha, A. Pak, D. Filev and P. Tsiotras, “KARNet: Kalman Filter Augmented Recurrent Neural Network for Learning World Models in Autonomous Driving Tasks,” 2023. arXiv:2305.14644

See the Advanced Planning for Autonomous Vehicles Using Reinforcement Learning and Deep Inverse Reinforcement Learning project page for the wider context.

Visit the GitHub repository for the model architecture and training scripts.