Near End to End Driving with a CNN

Let’s look at a simple and fast to deploy architecture for autonomous driving with deep learning. It won’t give us as much control over vehicle behavior as a SLAM style system might but the needed engineering hours and computer requirements will also be lower. This architecture also has two big advantages over classical line following techniques: it can easily navigate environments where multiple lane markings are visible, and it doesn’t even need traditional lane markings to derive a concept of path!

CNN Architecture

This architecture is based off of Nvidia’s s end to end self driving car system (arXiv:1604.07316v1[cs.CV]). This network is fairly straightforward: for a single frame of road, we generated a single steering prediction¬† as a regression output:


We probably want the car to learn to turn left in a situation like this.

Our goal is to train a network to reproduce human like driving from a set of system inputs similar to those available to a human driver. This methodology is commonly refereed to as behavior cloning. At training time we feed recorded frames into the base of the neural network and use our recording of the human controller input as the ground truth for the network output to learn to reproduce.

Training CNN Architecture

To use this network autonomously we feed it the live camera data and pass a filtered output to the motor controller to manage steering. Since the current learning system is designed to be speed agnostic we don’t predict speed outputs, instead operating speed is set by the operator to a fixed value.

Autonomous CNN Architecture


In Action:

The result of all this is smooth lane following behavior after about 20 minutes of training data collection.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s