A Transportable Neural Network Controller for

Autonomous Vehicle Following*

N.C. Griswold, N.D. Kehtarnavaz and K.M. Miller
Department of Electrical Engineering, Texas A&M University
College Station, TX 77843-3 128
Phone (409) 845-7404, Fax (409) 845-6259, griswold@ee.tamu.edu
As shown in Fig. 1, the driving command
generator (DCG) subsystem translates the range and
heading angle of the lead vehicle into an appropriate
control command consisting of a steering and speed
value for the autonomous vehicle. Originally this
was accomplished through the experimental
characterization of nonlinear vehicle dynamics. For
example, the relationship between the observed
heading angle and the corresponding steering wheel
angle was determined -through experiments for
various speeds. Although the performance of the
autonomous runs was satisfactory, it was desired to
develop a control mechanism which was
independent of individual and often unobservable
nonlinear vehicle dynamics. The motivation was the
transportability of such a controller to any vehicle
regardless of its dynamics. This transportability
attribute reduces the devellopment time and hence the
cost of full-scale autonomous convoy following
scenarios planned by the Army Tank and
Automotive Command (Ti4COM).
In this paper, we discuss how such a
transportable controller for autonomous vehicle
following is achieved through the use of neural
networks. Although the neural network approach
has been used for other autonomous land navigation
applications such as road following [4], this work
constitutes the first attempt to utilize it in vehicle
following.

Abstract
This paper presents the development and testing of a
neural network controller for autonomous vehicle
following. Autonomous vehicle following is defined
as a vehicle controlling its own steering and speed
while following a lead vehicle. The strength of the
developed controller is that no characterization of
vehicle dynamics is needed. As a result it can be
transported to any vehicle regardless of its nonlinear
and often unobservable dynamics. Data for the
range and heading angle of the lead vehicle were
collected for various paths with a human driver
performing the vehicle following control function.
The data was collected for different driving
maneuvers including straight paths, lane changing
and
righaeft
turns.
Two
time-delay
backpropagation neural networks were then trained
based on the data obtained under manual control,
one network for speed control and the other for
steering control.
After training, the vehicle
following was done using the trained neural network
controller. The results obtained and presented on a
video tape indicate that it is feasible to employ a
neural network to satisfactorily perform autonomous
vehicle following.
1. Introduction

In a previous work [l-31, we developed an

autonomous vehicle, named BART, to perform
vehicle following. Autonomous vehicle following is
defined as a vehicle controlling its own steering and
speed while following a lead vehicle within a
prescribed safe headway. BART, an acronym for
Binocular Autonomous Research Team, is a Dodge
Caravan equipped with (a) a binocular vision system
to obtain range and heading angle of the lead
vehicle, (b) a PC compatible computer to process
data for generating appropriate control commands,
and (c) a microprocessor servo system to implement
steering, throttle, brakes, and transmission functions.

2. Transportable Neural1 Network Approach

Neural networks are model-free paradigms that are

capable of learning the non-linear relationship
between input/output vector pairs through examples.
In our case, this approaclh is employed to learn the
non-linear relationships tetween the observed range
and heading angle, and the controllable steering
wheel angle and speed. The incorporation of neural
network into the vehicle following system is done as
follows: A human driver manually drives the
vehicle while following a lead vehicle. Range and
heading angle data from the vision system (input)
and the corresponding commanded speed and
steering wheel angle (output) are collected for

* This work was supported by Redzone Robotics,

Inc., and the Army Tank and Automotive Command
(TACOM) under the contract 9106-PO-1 19.

195

various maneuvers.
Based on the collected
input/output vector pairs,
two time-delay
backpropagation neural networks, one for speed and
the other for steering, are trained to learn the human
driver's response. After a training period, the
control is switched to the autonomous mode or the
neural network controller to reproduce the human
driver's driving. The neural network controller can
be viewed as an autopilot for performing autonomous
vehicle following.

control and the other for steering control. Table I

summarizes a list of an extensive collection of data
for steering angle control. This data included
various steering maneuvers at 15 mph and contained
a total of 3005 samples (approximately 25 minutes).
Similarly, Table I1 summarizes a list of an extensive
collection of data for speed control. This data
included different speeds along straight paths and
contained a total of 1702 samples (approximately
14.5 minutes).

3. Joystick Interface

5. Data Conditioning

The training data collected reflect both the vehicle

dynamics and the servo dynamics. If the driver is
allowed to drive BART directly using the steering
wheel, throttle pedal, and brake pedal, the data
collected would not include compensation for the
servo dynamics. A joystick was interfaced to the PC
to allow the human driver to drive BART through
the same control as the DCG. The joystick was a
relatively simple method for separating the vehicle
dynamics from the servo dynamics.
The joystick position was quantized and
converted into a drive command that was then issued
to the microprocessor to change the steering wheel
angle and speed. The number and value of the
quantized positions were determined through
empirical testing to make the joystick manual
driving easy for the driver. The joystick used had +/.75 inches of throw. Figs. 2a and 2b show the
quantization levels used for steering wheel angle and
speed, respectively.

Once the data was collected, it was required to

condition or normalize the data for the purpose of
training the backpropagation neural networks. For
the speed control network, the range was normalized
to a range error value between -1 and +1 as follows:
range err =

range -follow distance

scaling factor

where scaling-factor is the difference of maximumvalid-range and follow-distance if range 2 followdistance, and the difference of follow-distance and
minimum-valid-range otherwise. The speed was
also normalized as follows:
Spd =

last speed commanded

maximum speed

For the steering angle network, the heading angle

was normalized to a value between -1 and +1 as
follows:

4. Data Collection and Conditioning

4.1 Data Collection

heading angle =

As illustrated in Fig. 3 , inputJoutput data is stored

during the joystick manual driving. The input data
consists of range (in ft) and heading angle (in
degrees) of the lead vehicle obtained by the stereo
vision system. The output data comprises the
steering wheel angle (in degrees) and speed (in mph)
provided by a so-called drive by wire translator. At
this point it is worth mentioning that the control
cycle in our system cannot be reduced below 0.5s for
stable operation, a constraint imposed by the servo
system. Therefore, as a result, data samples were
collected at a sampling time interval of 0.5s. All
data collection was done on a runway facility whose
course is shown in Fig. 4.
Since it was neither necessary nor easy to handle
both steering and speed control functions at the same
time, two sets of data were collected, one for speed

measured heading angle

maximum valid heading angle '

where maximum-valid-heading-angle indicates the

maximum heading angle which allows the lead
vehicle to remain within the field-of-view of the
cameras. The range was also normalized as follows:
range =

measured range
maximum valid range

For most of our runs, the follow-distance was set to

68 feet, maximum-valid-range to 120 ft. minimumvalid-range to 30 ft, maximum-speed to 20 mph, and

maximum-valid-heading-angle to 19.5 degrees
corresponding to the cameras' field-of-view of 39
degrees.

196

layer nodes was set to 12 after carrying out an

analysis of the network behavior. Table IV lists the
mean-square error and Hamming distance for several
different numbers of hidden layer nodes. Therefore,
as shown in Fig. 6, the topology of the steering
control network was considered to be a three-layer
structure having 4-12-15 nodes.

6. Network Architecture and Training

6.1 Network Architecture

It was necessary to train the neural networks based

on the current and past samples in order to be able to
detect the lead vehicle's motion trends in time and
the human driver's response to them. Hence a timedelayed-neural-network (TDNN) architecture was
selected to perform the DCG function (see Fig. 1). A
TDNN is a backpropagation neural network that uses
a time-delayed sequence as its input vector [ 5 ] . This
allows it to deal with input data that are presented
over time, such as range and heading angle samples.
Here two separate TDNNs were employed; one
TDNN was trained to perform speed control, and the
other, steering angle control.
The input vector to the speed control network
consisted of seven components or nodes: the
normalized current speed, and the current and five
previous range errors. The number of input nodes
was a function of the time dependency of the driver's
response to range. This number was determined
through experimentation, see [6] for the details. The
output vector to the speed control network was a
binary vector having four components or nodes
corresponding to the speed changes -2, -1, 0, 1 mph.
The number of hidden layer nodes was set to 21 after
canying out an analysis of the network behavior.
This analysis included the recall performance of the
network for test data which were different than the
training data. Table I11 lists the mean-square-error
and Hamming distance for several different numbers
of hidden layer nodes. Hamming distance was
included to provide a more accurate performance
measure. This is because only the value of the
output node with the largest activation is transmitted
to the microprocessor, and this output is represented
by the binary bit 1, and all other nodes by the binary
bit 0. Therefore, as shown in Fig. 5, the topology of
the speed control network was considered to be a
three-layer structure having 7-2 1-4 nodes.
The input vector to the steering control network
consisted of four components or nodes:
the
normalized current range, and the normalized
current and two previous heading angles. Again the
number of input nodes was a function of the time
dependency of the driver's response to heading angle
which was determined through experimentation, see
[6] for the details. The output vector to the steering
control network was a binary vector having 15
components or nodes corresponding to the steering
wheel angles -45, -29, -20, -15, -10, -6, -3, 0, 3, 6,
10, 15, 20, 29, 45 degrees. The number of hidden

6.2 Network Training

Backpropagation training was used to set up the

connection weights of the TDNNs. Refer to [SI for
details of backpropagation training. Figs. 7(a) and
@) illustrate typical learning errors versus the
number of training epochs for speed and steering
networks, respectively. An epoch denotes one cycle
of training data. The network training parameters
include:
(a) learning rate 70 reduced by
epoch

q = q o(05)N where 70 denotes the initial

learning rate and N a (decay rate, (b) minimum
learning rate vmin,i.e. if 77 < qmin, then 77 = 7,nin,
(c) momentum factor p, and (d) stopping criteria
consisting of maximum number of iterations and
mean-square-error threshold.
At this point it should be mentioned that the
training time was a fuinction of the number of
samples in the training data set, the stopping
criteria, and the hardware used. The training
procedure in our case was done on a PC 486
machine for the following two cases: (1) Built-in
weights - This case corresponded to extensive
training based on 3900 iteration epochs, and 25
minutes of data for steering and 14.5 minutes of data
for speed. The training time for this case took a few
hours.
(2) On-the-fly weights - This case
corresponded to a few miinutes (about 10 minutes) of
training for shorter paths;. Although in both cases,
the autonomous vehicle managed to follow the lead
vehicle successhlly, the first case provided a
smoother ride. This is because, in the first case, the
network was given enough time to learn the
optimum weight values. Furthermore, in the first
case, the network's response was based on a large
number of samples or variations whereas, in the
second case, it was basecl on only a limited number
of samples or variations.
It should be realized ithat the nonlinear mapping
learned by the neural network is applicable only to
the situations and the environment in which training
data is gathered. Situations that are completely
different than those observed during data collection
would not be identifiable by the neural network, and

197

[6]

the response of the network would be an error

optimizing generalization based on the previously
learned situations.
6.3 Experimental Results and Conclusions

K.M. Miller, A Portable Neural Network

Approach to Vehicle Tracking, MS Thesis,
Electrical Engineering Department, Texas
A&M University, May 1994.

Acknowledgments: The authors wish to thank

Texas Transportation Institute for the use of their
runway facility and for their help in producing the
video tape. They would also like to thank Mr.
Heeman Lee for writing a menu-driven software
interface for this project.

The developed neural network controller was tested

on the same runway facility used for data collection.
Figs. 8 and 9 illustrate typical steering angle and
speed commands generated by the neural network
controller for lane changing and right turn
maneuvers. Many other examples are provided on a
video tape. Fig. 10 shows a typical result obtained
when using the original algorithm developed in [l].
As can be observed, the neural network approach
emulates the human driver's response to vehicle
following, which is different than the immediate
response generated by the original algorithm.
In this work, we developed and tested a neural
network approach to control the speed and steering
of a vehicle for the purpose of performing
autonomous vehicle following. The strength of this
approach is that it does not require the
characterization of nonlinear vehicle dynamics. As a
result, the control mechanism can be transported to
any vehicle regardless of its dynamics. The results
obtained and presented on a video tape indicate that
it is feasible to employ a neural network to
satisfactorily perform autonomous vehicle following.

Videos!&

ShzqBpd-

Plxel Data RanRelHeading

DT2867LC and 486IDX2-66Mhz

Fig. 1 - Modified BART block diagram

References

[l] N.Kehtarnavaz, N.C. Griswold, and J.S. Lee,

"Visual control of an autonomous vehicle
(BART) - The vehicle following problem,"
IEEE Trans. on Vehicular Technology, vol. 40,
pp. 654-662, Aug. 1991.
[2]

[3]

N. C. Griswold, N. Kehtarnavaz, "Experiments

in real-time visual control," SPIE Proceedings
on Mobile Robots, Boston, Nov. 1990.

Fig. 2a - Final Joystick Quantization for Steering Angle

Control

N.Kehtarnavaz, N.C. Griswold, and J.K. Eem,

"Real-time visual control for an intelligent
vehicle - The convoy problem," SPIE
Proceedings on Real-Time Image Processing
and Artijcial Intelligence, Florida, Apr. 1990.

[4]

[5]

A.D. Pomerleau, "ALVINN: An autonomous

land vehicle in a neural network," in Advances
in Neural Information Processing, vol. 1,
Morgan-Kaufmann Publishing, 1989.

-2

-I

Qiianticl~tonIr\,cl

J.M. Zurada, Artificial Neural Systems, West

Publishing, 1992.

Fig. 2b - Joystick Quantization for Speed Control

198

Test Vector Fnor VI

Vidm 51pnrl

Key feature -

U Fpochs

Rnngcfltc~cltn~

Allows the human driver to control vehicle

through same interface as the driving command

Epochs

generator.

-Mean Square Error

A l g ICmmmg Dlnanw
q,=O.3, niin. q=O.Oil, N=lOOO. p=0.9, MSE threshold=O.l

Fig. 3 - Joystick Control and Data Collection Block

Diagram

Fig. 7a - Error vs. #Epochs for 12 hidden node network

with j~= 0.9.
Tcrl Vector Fnar us U Fpnchr

Fig. 4 - Training Course

Fig. 7b - Error vs. #Epochs for Speed Control TDNN

Headmg and Steenng Wheel Angle

Fig. 5 - Speed Control TDNN

Fig. 8a - Lane Changing,from A to B using CFG12

Slmng Angle

Fig. 6 - SteeringAngle Control TDNN

Fig. 8b - Lane Changing from A to B using CFG4

199

TABLE I

Data Cellmian Tnalr for Steering Angle Control TDtW

Tnd
Lead Vebclc ACtBns
Number
I
Lane chmgng beween A and B

Sampler
194

Lane changng &ween A and B

Lane changng between A and B
Lane changng bmuun Aand B
Lane changing bewm A and 8,3 nght Nml between 8 and F
3 leR mm5 bewm F and 8. lane changng betwm B and A
h e changng b m e n A and 8 . 3 nght Nms beween 8 and F
3 lefl mm5 b e e n F m d B. lane changing between B and

2
3
4

5
6

7
Lead Car

lumber of

257
210
230
S I3

537
456
508

.cFelClalcI

Table I
Data Collection Trials for SteeringAngle Control TDNN

Samples

-Range

-Commanded

Spccd

Fig. 9a - SPD2 1 TDNN Response between A and B

Range and Commanded Speed
TABLE I1
Data Collectmn Tnds for Speed Conrrol lDW4

Folloums. D i m e = 68 f a t

Vumber of
Sampler

Lead Vehicle Actions

T d
Yumber
I

0 mph. i n ~ i e a x10 IO mph. Sop. ~ncrealelo 10 mph. $lop

;ill

2
3

0 mph. increase to I 5 mph. stop. increase lo I S mph. stop

235
158

0 mph increase IO 10 mph nop, ~ n c r w

10 10 mph sop

0 mph. i n c r e i ~to I 5 mph, stop. increase to I S mph. stop

325
286

0 mph. increw to IS mph $top. increase to I8 mph stop

295

0 mph incrase 10 18 mph. stop. increm 10 I 8 mph. slop

20
O J

176

1%

216

236

256

276

2%

316

316

356

io

1-6

Sampler

-R a n 5

-Commanded

Table 11
Data Collection Trials for Speed Control TDNN

Smd

Fig. 9b - SPD21 TDNN Response between E and F

Headmg and SI-&

Wheel Angle

TABLE VI
s p e d Control lDNN error companron
usmg dlfferca number of ludden layer nodes
f

of Hidden Layer Node9 Mean Square

Enar

Avg Hmmng Dirlance

0 II62

14

0 1101

0 252

0 1097

24

0 1137

0 2283
0 252

0 2141

Table IU
Speed Control TDNN error comparison
using different number of hidden layaer nodes

Fig. 1 Oa - Training Data from Table I, Trial 7 between B

and C

TABLE IV
Steenng Angle TDNN enor compmwrn
using differen! number of hidden layer noder

#of Hidden
Layer Nodm
4

n L
I44

I64

I84

104

Tnal I Mean
Tnal I Avg
Tnal 2 M ~ M Tnd2 Abg
Square Error H m n g Distance Square Enor Hamming Dinance
0 2071
06176
02007
06176

0 2065

07206

02159

08971

12
20

0 2026

06029
06912

02005

0 7647

"a

02w7

12.

Sample3

Table IV
SteeringAngle TDNN error comparison
using different number of hidden layer nodes

Fig. 1Ob - Data Collected using Original BART Control

Algorithm between B and C

200