Thesis

THE EVALUATION OF AIR-TO-AIR
COMBAT SITUATIONS BY NAVY FIGHTER PILOTS

WITH ARTIFICIAL INTELLIGENCE APPLICATIONS
Kenneth Levin
PO T L
Monterey, California
'Y
mHMHaMH
The Evaluation of Air -to-Air

'
Combat Situations by Navy Fighter Pj .lots
with Artificial Intelligence Applications
by
Kenneth Levin
Th esis Advisor: R. S. E .ster
June 1972
Approved ôn. puhLLc. /i££e&6e.; du>&vLbuXion ayuUniltzd.

The Evaluation of Air-to-Air
Combat Situations by Navy Fighter Pilots
with Artificial Intelligence Applications
by
Kenneth Levin
Lieutenant, United States Navy
A.B., Washington University, 1966
Submitted in partial fulfillment of the

requirements for the degree of
MASTER OF SCIENCE IN COMPUTER SCIENCE
from the
NAVAL POSTGRADUATE SCHOOL

June 19 72
ABSTRACT
The evaluations mode by 36 Navy fighter pilots of 64 air-to-air
combat situations are statistically analyzed to detect any significant
differences between pilots' evaluative techniques in relation to their
rank, flight hours and combat missions. Predictor equations are computed
and used in a self-analyzing, self -modifying artificial intelligence
program modeled on an instructor-flight student interactive situation.

TABLE OF CONTENTS
I. INTRODUCTION 7
A. OBJECTIVES 7
B. DEFINITIONS 9
C. ASSUMPTIONS 9
II. EXPERIMENTAL PROCEDURE-DETERMINING DECISION 10

STRATEGY
III. RESULTS OF DATA ANALYSIS 21
A. VARIABLES ADDED AND DELETED 21
B. CORRELATIONS 23
C. VARIABLE MANIPULATION 25
D. MULTIPLE REGRESSION 3
IV. EXPERIMENTAL PROCEDURE -ARTIFICIAL INTELLIGENCE 34

APPLICATION
A. PROGRAM DESCRIPTION 38
1 . Main Program 4
2. Subroutine IN STRU 42
3. Subroutine GRADER 42
4. Subroutine KORECI 42
5. Subroutine KOREII 44
6. Subroutine KORFII 44
7. Subroutine NOVICE 45
8. Subroutine RANDOM 45
B.. RUNNING THE PROGRAM 45
V. RESULTS OF PROGRAM RUNS 48
A. AVERAGE PILOT AS STUDENT 48
B.. SYNTHETIC BEGINNER AS STUDENT 51
C. THE ARTIFICIAL INTELLIGENCE QUESTION 78
VI. CONCLUSION 79
VII. CRITICAL EVALUATION OF THE STUDY 82
VIII. CONTINUATION OF THE STUDY 84
APPENDLX A QUESTIONNAIRE 86
APPENDIX B PILOTS RESPONDING TO MAIL-OUT QUESTIONNAIRE 95
APPENDIX C PANEL OF AVIATORS 96
APPENDLX D DISCRIMINANT ANALYSIS FOR GROUPS 97

DIFFERENTIATED BY BIOGRAPHICAL DATA
APPENDIX E SELECTED PREDICTOR EQUATION COEFFICIENTS 98
APPENDIX F MEANS AND STANDARD DEVIATIONS OF DECISIONS 99

MADE
APPENDLX G VARIABLE RANKINGS FOR HOUR AND MISSION GROUPS 100
APPENDIX H MULTIPLE CORRELATION OF PREDICTOR EQUATIONS 102

WITH PREDICTED DECISION
COMPUTER PROGRAM 103
LIST OF REFERENCES 116
INITIAL DISTRIBUTION LIST 118
FORM DD 14 73 119
LIST OF TABLES
L. DESCRIPTION OF RESPONDING FIGHTER PILOTS 18
II. CORRELATION OF BIOGRAPHICAL DATA WITH PRIMARY 23

AND SECONDARY DECISIONS
ILT. CORRELATION OF SITUATIONAL VARIABLES WITH PRIMARY 24

TACTICAL DECISIONS AND SECONDARY EXPECTED ENEMY
AND FRIENDLY KILL DECISIONS
IV. CORRELATION BETWEEN PRIMARY AND SECONDARY 25
DECISIONS
V. VARIABLE MANIPULATION -AVERAGE VALUE PER SITUATION 2 7
VI. VARIABLE RANKINGS FOR ALL PILOTS AND RANK GROUPS 31

IN DECISIONS REACTED
VII. CORRELATION OF PREDICTED VALUES FOR DECISIONS 32

AND ACTUAL DECISIONS MADE BY ALL PILOTS AND
RANK GROUPINGS
VIII. RESULTS OF PROGRAM RUNS WITH TWO STUDENTS AND 47

FOUR INSTRUCTORS
LIST OF FIGURES
1. TACTICAL SITUATION PRESENTATION 15
2. QUESTIONS ASKED WITH EACH DISPLAY 16
3. MACRO -LEVEL PILOT EDUCATION. 3 6
4-6. AVERAGE STUDENT -AVERAGE INSTRUCTOR 52-54
7-9. AVERAGE STUDENT-CDR INSTRUCTOR 55-57
10-12. AVERAGE STUDENT-4000 HOUR INSTRUCTOR 58-60
13-15. AVERAGE STUDENT-300 MISSION INSTRUCTOR 61-63
16-18. BEGINNING STUDENT-AVERAGE INSTRUCTOR 65-67
19-21. BEGINNING STUDENT-CDR INSTRUCTOR 68-70
22-24. BEGINNING STUDENT-4000 HOUR INSTRUCTOR 71-73
25-27. BEGINNING STUDENT-300 MISSION INSTRUCTOR 74-76

L. INTRODUCTION
From the legendary Baron von Richtofen to the popular Snoopy-
flying his doghouse through the comic strips, the image of the fighter
pilot is one of an enigma capable of performing complex and dangerous
tasks in an environment where fractions of a second can spell the dif-
ference between success and filure, life and death. This fascinating
picture of a highly trained, assured and swaggering superbeing beckons
to be studied; to have the mystique lifted. Hence, the impetus of this
research: the fighter pilot.
Other than the satisfaction of human curiosity, the study of the
fighter pilot -in this case the Naval fighter pilot- can ultimately result
in improved training and selection of future pilots by understanding the
activities of successful pilots; improved combat techniques by adaptation
of the successful pilots' techniques; improved surface-to-air and air-to-
air communications through better understanding; and improved electronic
and mechanical pilot aids to enhance pilot performance.
A. OBJECTIVES
The objectives of this research lay in two major areas. The first
was to address the questions of how Navy fighter pilots evaluate an air-
to-air combat threat, i.e. , pilot versus pilot situations. How does the
pilot reach his decision? Can this decision be somehow quantified for
computer applications? What differences exist among pilots in their
evaluations of air-to-air threat?
The second major area lies in the field of artificial intelligence.
Classically concerned with games such as chess and checkers with well
defined rules for determining success and failure [1, 2], artificial intel-
ligence has seen little application in real life "games." A notable
exception is Clarkson's portfolio program which has reached partial
agreement with a successful bank trust administrator [3] . Heuristics,
such as the minimax technique whereby a player will always attempt to
minimize his maximum loss [4], and binary choice selection [5], although
fine for board games, find little application in games of life. As stated
by Rigney [6]:
"For decision tasks in which the outcome is of great consequence

and the time for decision is short, as in many military, business,
and medical decisions, the strategy the decision maker uses is
of special interest."
From the strategies hopefully revealed in the attainment of the first
objective, an artificial intelligence application is sought. As brought
out by Shepard [7], an obvious disparity exists between "the effortless-
ness and surety of most perceptual decisions and the painful hesitation
and doubt characteristic of these 'higher level' decisions" and strategies.
To overcome this disparity in a "game" of ill defined rules and high stakes
is the objective of the artificial intelligence application.

B.. DEFINITIONS
At this time some definitions of terms used throughout this research
should be brought forward. Artificial intelligence as used in this study
encompasses the construction of "computer programs which exhibit
behavior that we call 'intelligent behavior' when we observe it in human
beings [8]."
The area of artificial intelligence most apt for this study is
decision making in game playing. Hence the game is defined as air-
to-air combat as experienced in the Vietnam area of operations.
C. ASSUMPTIONS
Certain assumptions used throughout the study are:
(1) Data such as military rank and threat evaluation decisions

can be placed on a quantifiable continuum.
(2) Since no criteria of deciding success or failure in the game

exist short of actual combat results, expertise in the game
will decide successes or losses.
(3) Expertise is directly proportional to military rank, flight

hours and total number of combat missions.
(4) The data base used is a representative sample of Navy

fighter pilots .
.
HE.. EXPERIMENTAL PROCEDURE-DETERMINING DECISION STRATEGY
In an. attempt. to isolate and identify the important variables con-
sidered by a fighter pilot in his evaluation of an air-to-air combat
situation, a mail-out survey was designed. In the survey the subjects
were asked to assume the following hypothetical situation:
(1) The subject is an airborne Target Area Combat Air Patrol

(TARCAP) over enemy territory in Southeast Asia.
(2) The subject's aircraft is configured and armed for air-to-air

combat only; hence, a fighter.
(3) The subject's primary mission is to intercept and destroy

enemy air raids threatening friendly airborne strike forces.
(4) The TARCAP and immediate strike group is the only friendly
force in the area. Therefore, the subject is free to fire at
any threat without positive visual identification.
Eleven different items of additional information were then presented
to the subject. These were presented in pairs yielding a total of 55
combinations. The items of information were:
- (1) Relative position of enemy aircraft from the friendly strike

group
(2) Type of enemy aircraft.
(3) Number of enemy aircraft.
(4) Number of friendly aircraft.
(5) Range from friendly aircraft to enemy aircraft.
(6) Speed of enemy aircraft.
(7) Friendly aircraft capabilities.
10
(8) Altitude of enemy forces .
(9) Anti-aircraft activity in the area.
(10) Bearing of enemy from friendly forces .
(11 Closure rate of friendly and enemy forces.
The subject then had to choose which one of the pieces of infor-
mation presented in each pair would be of greater value in evaluating
the threat of, and in planning engagement tactics against, enemy air
units. An additional alternative was given whereby each member of the
pair could be evaluated as being egual in importance. Appendix A con-
tains a sample of this questionnaire. Ninety-seven Naval aviators
responded to this questionnaire of which 68 or 70.1% were pilots and
29 or 2 9.9% were Radar Intercept Officers, the second crewman in the
F-4 Phantom jet. Eleven or 11.3% were F-8 Crusader pilots, the rest
of the pilots being F-4 pilots. Sixty-three or 65.0% of the aviators had
combat experience over Korea or Vietnam. Nine or 9.4% had engaged in
actual air-to-air combat.
From a scaling of the variables from the pair-comparisons data
the variables were ranked as follows (in descending order of importance)
(1) Relative position of enemy aircraft.
(2) Bearing of enemy.
(3) Range of enemy.
(4) Altitude of enemy.
(5) Friendly aircraft capabilities .
11
(6) Number of: enemy.
(7) Rate of closure .
(8) Anti-aircraft activity.
(9) Type of enemy aircraft.
(10) Speed of enemy.
(1 1) Number of friendly .
From the results of the first survey, and with the consideration
of the type of information available to the pilot in the cockpit, a second
mail-out survey was designed. The variables considered in this survey
were limited to six in number. Fuel states and rules of engagement were
added for realism. The remaining four variables were considered as being
information a pilot would have at his access from either his cockpit
instruments or via a surface or airborne controller. The variables were
limited to two states, hence 25 or 64 possible combinations were con-
sidered by each subject. The variables and their states were:
(1) Fuel above bingo; the maximum amount of fuel that can be
utilized before fuel level drops below that necessary for a
safe return to home base.
(a) 1000 pounds.
(b) 2 500 pounds.
(2) Rules of engagement; when the subject may take the suspected
aircraft under fire.
(a) Eyeball (positive visual identification is required).
(b) Missiles free (positive visual identification is not

required) .
12
.
(3) Enemy's relative bearing from the subject.
(a) 315°
(b) 135°
(4) Enemy's range from the subject.
(a) Eyeball or within 4 nautical miles.
(b) 2 nautical miles.
(5) Enemy's true heading or direction of flight.
(a) 045°
(b) 225°
(6) Number of enemy aircraft
(a) 2
(b) 6
The subjects were asked for their rank, total number of flight
hours and total number of combat missions. The following background
scenario was used for each of the 64 situations:
"You are the flight leader of a section of F-4's armed with two
Sparrows?- and two Sidewinders .3 Assume for this exercise that the
aircraft's weapon systems are up in every respect. You are providing
A section normally consists of two F-4's
2
Beam riding guided missile.
3
Heat seeking guided missile
13
TARCAP for a division of A-7's who have just completed a strike and are
egressing from the target area. You are feet-dry over North Vietnam
(20 nm to the coast). The AA and SAM^ defenses in the immediate area
are light to moderate You have limited GCI° facilities operating for
.
you and the enemy has excellent ground radar control.
The enemy aircraft are assessed to be MIG-21's at 15,000 feet and

500 kts You are 10,000 feet and 450 kts. heading for your carrier
.
(360° relative) .
The weather in the area is clear and 15+ visibility. There are
several flights of attack aircraft still feet-dry, exact position unknown.
The MIG's have demonstrated an air-to-air missile capability."
Figure 1 shows the way the tactical situations were presented to the
subjects, analogous to a radar scope or maneuvering board's display.
Two major evaluation questions were asked of each subject in con-
junction with each of the 64 situations considered. These questions
are shown in Figure 2 .
The questionnaire was sent to a number of Navy fighter squadrons
on both the East and West Coasts. The aircraft in service at each of
the squadrons was the F-4 Phantom. Thirty-six pilots responded. These
pilots are described in Appendix B. Table 1 contains a description of
the pilots by rank, hours and combat missions. The overall average
4
Over land vice feet wet or over water,
5
Surface-to-Air Missile.
Ground Control Intercept
14
Figure 1
TACTICAL SITUATION PRESENTATION
15
.
Figure 2
QUESTIONS ASKED WITH EACH TACTICAL DISPLAY
A. In this tactical situation (check one answer):
I. I'd have no choice; there would be an engagement.
2 .. Td have a choice on whether to engage or not, and I'd

engage.
3.. I wouldn't engage.
B -.. Indicate what aircraft losses you would predict, if there were an
engagement
Enemy Losses (no. of aircraft)
Friendly Losses (no. of aircraft)
16
re spondee had a rank between Lieutenant and Lieutenant Commander
7
(3.47) , 1994.03 flight hours and 124.11 combat missions. Rank was
assumed to be an attribute providing ordinal data.
The data from the surveys were coded and placed on a magnetic
disk for statistical analysis. The rules of engagement, being qualitative
variables, were arbitrarily coded as a for eyeball and 1 for missiles-
free. A measure of central tendency (arithmetic mean) and measure of
dispersion (standard deviation) was computed for each biographical
variable (rank, flight hours, combat missions), each primary tactical
decision and each secondary enemy and friendly kill decision made for
the entire sample. Means and standard deviations of these variables
and decisions were then computed for each subset of the sample defined
by biographical data groupings, i e. , all Commanders, all Lieutenant
Commanders, all Lieutenants, all Lieutenants (junior grade), all pilots
with more than 4000 hours, all with less than 4000 and more than 3000,
all with less than 3 000 and more than 2 000, all with less than 2 000 and
more than 1000, all with less than 1000, all pilots with over 300 missions,
all with less than 300 and more than 200, all with less than 200 and
more than 100, all with less than 100 and more than one mission and
those pilots with no combat missions. Likewise, the mean and standard
deviation of each decision was computed for the subset of the sample
defined by holding each of the situational variables constant at one of
17
TABLE I
DESCRIPTION OF RESPONDING FIGHTER PILOTS
Rank: LT(jg) LT LCDR CDR

Number: 1 21 10 4
Hours: Mean 450 1394 29 2675.50 3825 00
S.D 438 54 553.69 466 61
Missions: Mean 96 52 163.40 201 75
S.D. 74 66 72.17 106 81
Hours: 0-999 1000-199! 2000-2999 3000-3999 4000+

Number: 5 17 6 6 2
Rank 7 Mean
: 2.80 3.14 3 67 4 33 5
S.D. .01 .88 47 47
Missions: Mean 127.07 111 83 242 00 106 50
S.D. 87.12 54 78 69 20 36 64
Missions: 1-99 100-199 200-299 300+

Number: 6 7 13 8 6
Rank 7 Mean
: 2.83 3.38 3.54 3.71 4.50
S.D. .37 .70 .63 .70 .50
Hours: Mean 763.33 2150.00 2065.77 2378.57 3250.00
S.D. 177.87 971.43 870.87 832.28 50.20
Ensign = 1, LT(jg) = 2, LT = 3 , LCDR = 4, CDR = 5
18
.
its values and then the other while the remaining situational variables
were free to assume either of their two values .
In attempting to find a relationship between the biographical and
situational variables and the decisions reached, correlation coefficients
were computed between these variables and decisions. These correlation
coefficients were computed for the entire sample, for each subset de-
fined by the biographical groupings, and for each respondee. These
findings were presented to a panel of Naval aviators who were asked for
their opinions and suggestions. These aviators are described in Appendix
C. From their suggestions new variables made up of combinations of
the original six variables were added to the data base and the same
analysis again undertaken. This cycle was repeated until the panel con-
sidered the results to be reasonable.
In an attempt to distinguish any discernible differences between
the decisions reached for each biographical subset within the rank, hour
and mission groupings, a discriminant analysis was performed. In this
analysis, a linear function for each subset within a group is computed
using the subset characteristic (specific rank, hours grouping, or
mission grouping) as the dependent variable and the decisions reached
as the independent variables. These functions are then evaluated for
each case and the hypothesis that the subset means are the same is
tested [9]
19
In an attempt to predict a pilot or group of pilots' decisions, a-
sequence of multiple linear equations computed using stepwise multiple
regression techniques were also produced for the entire sample and each
subset. At each step in this regression procedure, one situation variable
is added. The variable added is the one which makes the greatest re-
duction in the error sum of squares (the variable with the highest partial
correlation with the dependent variable at that step) . The production
stops when the error sum of squares decreases less than .001 or until
all variables are exhausted [10] . The correlation coefficient between
these equation predictions and the actual decision reached was then
computed for each situation examined. These equations and correlations
were then presented to the panel of aviators for their evaluation.
20
m .
RESUL1SOT_DATA ANALYSIS
A . VARIABLES ADDED AND DELETED

quantitative items of infor-
heading, although
Si nce bearing and
did not represent con-
he instant in the first survey,
ma tion shown to
manipulated, both
that could be eonveniently statistiealiy
tinuums
The reason this was done is
deleted in favor of a
new variable.
were
of 001° and 359° .
for example, bearings

apparent when evaluating,
evaluated
rose, a guantity
separated by only 2° on a oompass
Mthough
i- •! i~
negligible, iv,o ualupG are separated by
the value,
aviators as
by the panel o£
effects on a
not negligible. Also, the two variables'
357 o whieh is
to be interactive.
An enemy directly behind
pUofs evaluation appeared
\,,-,t„H riifferently
differently if the enemy was
180°) was evaluated
a friendly (bearing
away (heading 180°).
(heading 360°) or pointing
pointed at the friendly
either
to the pilot from
is not directly available
Thl s pointing information
Yet the pilot does

mentally integrate
his cochpit delays or GCI input.
information. Conse-
information into pointing
the bearing and heading
bearing and heading was created. This

quently, a function of both
equation:
angle, "i,
is describedioy the following
Hpscribed by
function, called "danger
a- m - (bearing)/ I sin
(Qin (bearing;/ 1 ) (bearing)
i_ (sin
Danger (bearing, heading)
{
+ 180) modulus 360].

x [bearing- (heading
as
and fuel-remaining
although pilots considered range
Similarly,
a
panel believed that
ortant variables independently, the aviator
imp
21
link existed between these two factors. For instance, a fighter pilot's
behavior in a situation of long range with low fuel remaining would
probably be different from that under a situation of short range and a
large amount of fuel remaining. Again, this would be a case of pilot
integration of a cockpit display (fuel remaining) and a GCI input or
radar display (range) into a range per fuel state factor. Consequently
a range-fuel ratio (MPG) was created as described by the following
equation:
MPG (range, fuel remaining) = range/fuel remaining.
As shown in the first mail-out questionnaire results, a relationship
between the enemy's speed and heading and friendly speed and heading,
or closure rate is considered by fighter pilots (See EXPERIMENTAL
PROCEDURE DETERMINING DECISION STRATEGY). Again, pilot inte-
gration of cockpit and GCI inputs appears to occur: the pilot projecting
the parallel enemy speed vector into his own. As a result, a speed
differential function was created for the tactical scenario:
Speed (heading, bearing) = 500 x cos (heading) - (cos (bearing)/
|
cos (bearing) j
) x 450.
In both the speed and danger equations, a value of +1 was assigned to
the ratio of the trigonometric function and its absolute value when that
function was equal to zero.
22
B. CORRELATIONS
The primary tactical decision made by the subjects' was assumed
to be ordinally quantifiable as indicated below:
(1) Pilot has no choice, an engagement would occur.
(2) Pilot has a choice and would elect to engage the enemy.
(3) Pilot would not engage the enemy.
The secondary decision was in two parts, number of enemy losses pre-
dicted and number of friendly losses predicted if an engagement were
to occur.
The biographical data's correlation with the decision is shown in
Table II for all pilots.
TABLE II
CORRELATION OF BIOG RAPHICAL DATA WITH PRIMARY

AND SECONDARY DECISIONS
Tactical Enemy Friendly
Decision Kills Kills
Rank -.0392 .0541 .2195

Hour -.0394 .0628 .1685
Missions -.0333 .0801 .1265
The lack of correlations significantly different from zero is striking.
Only with the relationship of rank to friendly kills predicted can a
coefficient of correlation be found of any sizeable magnitude (+.2195).
Discriminant function analysis revealed no significant differences
between the pilots when they were categorized by rank, hour or mission
23
with respect to the decisions they made (at a significance level of
< .05) (See Appendix D) . The panel of aviators considered this (post
hoc) to be a result of standardized training and operational squadron
procedures. They also expected a positive correlation between rank
and expected friendly kills as a result of more realistic evaluation
techniques acquired with time spent in operational billets. A decline
in youthful optimism as one ascends the rank and age structure was also
expected to enhance this postulated positive correlation. Appendix F
contains the means and standard deviation of the decisions reached by
the biographical groupings.
The situational variables' correlations with the decisions are
shown in Table III.
TABLE III
CORR ELATIONS OF SITUATIONAL VARIABLES WITH

PRIMARY TACTICAL DECISIONS AND SECONDARY
EXPECTED ENEMY AND FRIENDLY KILL DECISIONS
Tactical Enemy Friendly

Decision Kills Kills
MPG .4259 -.2087 -.1237

Speed .2559 -.2339 .0553
Danger .2864 -.2857 -.0201
Fuel Rei maining -.1708 .3400 -.0203
Range .4209 -.0163 -.1746
Number of Enemy .1223 .0231 .2447
Rules of Engagement -.0760 .2287 -.0512
24
The panel of aviators, however, felt a chronological relationship
existed between the decisions, that is, the pilot evaluated the situation
as a whole and then reached his primary tactical decision. After that
decision had been reached, and with that result kept in mind, it was
thought that the pilot then made his secondary decisions. Suggestions
by the author that the decisions of kill possibilities would precede and
influence the tactical decision were strongly rejected by the panel as
unrealistic from their own experiences. Table IV shows the correlation
between the primary and secondary decisions.
TABLE IV
CORRELATION BETWEEN PRIMA RY AND

SECONDARY DECISIONS
Enemy Kills Friendly Kills
Tactical Decision -.4247 -.2307
The reader should note that the biographical variable of rank is
the third most important variable when predicting secondary friendly
kills.
C. VARIABLE MANIPULATION
The effects of holding specific situation variables constant on the
decisions made is shown in Table V. MPG was not held constant since
that would imply a simultaneous manipulation of range and the fuel-
remaining variable.
25
The most striking phenomena evident in Table V is the ratio
between predicted enemy kills and predicted friendly kills of over four
to one. As explained by the senior member of the panel of aviators
after observing this phenomena:
"A pilot would be a fool to go into combat is he didn't think

the odds were stacked heavily in his favor."
The inverse relationship evident in this table between range and
friendly kills was considered by the aviators to be a function of aircraft
type.. The F-4 was considered by them to be more of a weapons platform
than a classical dog-fighter while the MIG-21, a smaller and more
maneuverable airplane, is better suited for close-in fighting. The
shorter range and, consequently, time available to the F-4 weapons
systems decreases some of this system's advantages over a MIG-21
resulting in a greater enemy opportunity to attack the friendly forces.
Interesting conjectures can be made about the results that might be
observed if the friendly plane were an F-8, a true dog-fighter. . .The
slight increase in enemy kills with decreasing range which is evident
in Table V was felt by the panel of aviators to be the result of an in-
creased friendly missile kill probability.
The pushing of the tactical decision to the "must engage" pole
with shorter ranges was felt to be an indication that a pilot naturally
considers danger more imminent as the enemy closes; the shorter ranges
decreasing the protector's options.
26
TABLE V
VARIABLE MANIPULATION -AVERAGE

VALUE PER SITUATION
Variable Value Tactical Decision Enemy Kills Friendly Kills
Range 4 1.66 1.09 .37

20 2.32 1.05 .15
Rules of Missile Free 1.93 1.31 .23

Engagement Eyeball 2.05 .84 .29
Number of 2 1.89 1.05 .11

Enemy 6 2.09 1.10 .41
Speed 1.71 1.42 .24

2.28 .73 .28
Danger 2.03 .94 .23

1.83 1.19 .29
Fuel
Remaining 1000 2.12 .73 .27
2500 1.86 1.42 .25
All Free Mean 1.99 1.07 .26

S.D. .79 1.01 .62
27
The differences displayed in the decisions as rules of engagement
are varied were also felt by the panel to reflect aircraft type. The F-4
is at its best when able to deliver the long range, unsuspected missile
under missiles free conditions. A 40% drop in predicted enemy kills can
be seen in Table V as the rule of engagement assumes the eyeball con-
dition. Waiting for ranges to shorten for eyeball identification means
losing the advantage of the long range punch. The decreased range
forced on the pilot shows in Table V, as before; in an increase in pre-
dicted friendly kills but not, surprisingly, in a tactical evaluation pushed
towards the "must engage" pole.
The greatly increased number of predicted friendly kills when the
number of enemy is high, as shown in Table V, is considered to be a
direct reflection of superiority in aircraft numbers. No matter how
effective a protector may be, once the protectors are saturated, some
of the enemy will reach the vulnerable friendly forces. Also, the more
enemy present the more enemy the friendlies may destroy. The tactical
decision's migration towards the "not engage" pole was felt by the panel
to be a sign of caution by the friendly in light of a large enemy force;
the gains of engagement being more than offset by the possible friendly
losses .
The speed differences between the friendly and enemy forces
appear to push the primary tactical decision towards the two poles of
engage or not engage. A large speed differential coupled with opening
28
. .
ranges implies, according to the panel, a friendly decision of leaving
the friendly forces exposed for a long chase or staying with the friendlies
and not engaging the enemy. The high closure rate, however, would
force the friendly to engage but, as shown in the extremely high predicted
enemy kills in Table V, would place the enemy in the F-4's most optimal
firing envelope
The enemy pointing at the friendly forces places the friendlies in
the enemy's best firing envelope and, as shown in Table V, seem to
force the friendly tactical decision to the defensive "must engage" pole.
As a result, a rise in both predicted enemy and friendly kills can be
seen. The enemy pointing away from the friendlies with its implied
opening ranges drives the tactical decision to the "not engage" side
with concurrent drops in both predicted kills
The low enemy kills and "not engage" decision evident with low
fuel state was interpreted by the panel as a sign for caution for the
friendly pilot. Friendly kills rise in this low fuel state not only from
enemy attacks but from destruction due to fuel exhaustion in combat.
Higher fuel states give the pilot a greater margin in combat for increased
maneuvering and chasing, resulting in the increased predicted enemy
kills and greater friendly protection.
29
D. MULTIPLE REGRESSION
From the multiple regression analysis a ranking of situational
variable importance was made for all pilots and for each rank, hour and
mission group. Table VI gives these rankings for all pilots and for the
rank groupings. Appendix G contains the hour and mission rankings. As
can be seen, MPG or range, one a function of the other, play an important
role in the tactical decision evaluation. Danger, except in the case of
Commander where it switches position with speed, is next most important
followed by speed.
In the importance of variables for secondary decision evaluation,
the tactical decision is the most important in evaluating enemy kills
while the number of enemy is the only variable more important in evalu-
ating friendly kills. This may reflect the pilot's integration of his
primary decision in his next evaluation, i.e. , the pilot's "mental set"
before actually making his next evaluations .
From this multiple regression analysis predictor equations were
constructed. Described in Appendix E, the equations were used in
predicting the decisions made on the decision survey. The obtained
multiple correlations are described in Table VII and Appendix H.
30
TABLE VI
VARIABLE RANKINGS FOR ALL PILOTS AN D

RANK GROUPS IN DECISIONS REACHED
A. Tactical Decision
ALL CDR LCDR LT LT(jg)

MPG 1 7 1 7
Danger 2 3 2 2 2
Speed 3 2 3 3 6
Range 4 1 1 5 1
Number of Enemy 5 4 4 4 4
Rules of Engagement 6 6 6 6 5
Fuel Remaining 7 5 5 7 3
B . Enemy Kills
ALL CDR LCDR LT LT(j

Tactical Decision 1 1 1 1 1
Rules cf Engagement 3 6 4 3 8
Danger 4 5 2 4 4
Speed 5 8 5 5 3
Range 6 2 6 6 6
MPG 8 7 8 8 7
C. Friendly Kills

Tactical Decision 2 2 1 2 4
Speed 3 4 4 3 3
Rules of Engagement 4 3 7 4 5
Danger 6 5 8 6 6
Range 7 7 3 7 2
MPG 8 6 6 8 7
31
TABLE VII
CORRELATION OF PREDICTED VALUES

FOR DECISIONS AND ACTUAL DECISIONS
MADE BY ALL PILOTS AND RANK GROUPINGS
Tactical Decision .6032 .5794 .6236 .6101 .7568
Enemy Killed .6027 .5859 .5654 .6527 .8193
Friendly Killed .4015 .4391 .5299 .3732 .5284
It appears that the equations are fairly successful for the groups studied
in predicting their tactical and enemy kill decisions.
In an attempt to secure a standard of comparison for these equations,
the panel of aviators was used to predict one of the biographical group's
decisions. The members of the panel, who were associated with the
data at some length, considered themselves to be closest to the
Lieutenant Commander group in rank, hours, and missions. All of the
Lieutenants on the panel were in the current Lieutenant Commander
selection zone. They attempted to predict the Lieutenant Commander
group's decision rather than only making their own evaluations of the
situations. The following correlations between the panel's predictions
and the actual decisions made by the Lieutenant Commander group were
found:
Tactical Decision .5863
Enemy Killed .52 00
Friendly Killed .4327
32
.
En. essence, the predictor equations fared better than the predicting
panel of aviators (See Table VII)
33
IV.. EXPERIMENTAL PROCEDURE-ARTIFICIAL INTELLIGENCE APPLICATION
One writer [11] has described and defined artificial intelligence in
the following way:
"A human being can think, learn, and create because the program
his biological endowment gives him, together with the changes in
that program produced by interaction with his environment after
birth, enables him to think, learn, and create. If a program thinks,
learns, and creates, it will be by virtue of a program that endows
it with these capacities. Clearly this will not be a program - any
more than the human's is - that calls for highly stereotyped and
repetitive behavior independent of the stimuli coming from the
environment and the task to be completed. It will be a program
that makes the system's behavior highly conditional on the task
environment - on the task goals and on the clues extracted from
the environment that indicate whether progress is being made towards
those goals. It will be a program that analyzes, by some means,
its own performance, diagnoses its failures, and makes changes
thai: enhance its future effectiveness."
With the goal of the last sentence above in mind, the panel of
aviators, all of whom have had student aviator instructor billets,
described a student pilot's behavior in interaction with his instructor.
Once past the basic ground schools, flight training appeared to entail
corrective instruction of the student's cockpit actions. The corrective
instruction can be broken down into three general areas:
(1) The student did the opposite of what he should have done.
(2) The student did too much of what he should have done.
(3) The student did not do enough of what he should have done.
For example; (1) pushing the stick to the left vice the right, (2) applying
too much throttle, and (3) not using enough flaps. Such mistakes as
34
forgetting to lower the landing gear were considered as combinations or
extremes of the three .
The student's response to the three corrections were:
(1) Reversing the sign or direction of what was incorrectly done.
(2) Lessening the magnitude of what was incorrectly done.
(3) Increasing the magnitude of what was incorrectly done.
Also, as the student pilot gains more experience his confidence level in
himself and aircraft rises. His corrections grow smaller in magnitude
resulting in a smoother development. As stated by one of the aviators
on the panel:
"Any movement which is not smooth and gentle in an aircraft is

probably an incorrect movement."
In advanced Naval pilot training where such topics as tactics and
gunnery are taught, the three instructive areas are thought to still exist.
However, at this level the pilot is developing his evaluative skills
rather than his motor skills. Since multi-level decisions are involved,
the student is not allowed to proceed to the next level until the present
level decision or action is correct. Figure 3 displays this macro-level
education concept.
This concept, if captured in a program, would enable the program
to analyze its performance and diagnose its failures (instructor correction)
and make changes to enhance its future effectiveness (student modifi-
cation) . This program could show some attributes (confidence and
35
Figure 3
MACRO-LEVEL PILOT EDUCATION
A: Present situation to student.
B: Student evaluates situation and makes primary decision
C: Is decision correct?
D: Student gains confidence and experience
E: Student makes secondary decisions.
G: Student modifies own evaluation techniques
36
experience, learning, fallibility) that, if witnessed in a living organism,
would certainly be termed intelligent. Hence, an artificial intelligence
approach to this macro-level pilot learning concept was initiated.
Certain restrictions were placed on this artificial intelligence
programming attempt. Since possible graphics adaptations were en-
visaged, memory size limitations were desired. The available graphics
terminal at Naval Postgraduate School is an AGT-10 with limited memory.
Also, as brought out by the panel of aviators, a pilot does not consciously
search through his memory looking for a situation or solution but rather
appears to act at a level of perceptual decision making [7] . Likewise,
the panel considered the classical tree searching and minimaxing pro-
cedures as unrealistic for this problem. Computer run time was to be
as short as possible for both the practical consideration of program turn
around time and because of the desire for instantaneous decisions as in
high speed air-to-air combat. Consequently, a polynomial evaluation
with adaptation of the coefficients technique was chosen along the lines
of Samuel's checker player [2] . The coefficient would, in essence, act
as a quasi-memory , reflecting change but not, necessarily, history.
FORTRAN IV was chosen for the program language over ALGOL and LISP
because of future graphics applications where no ALGOL or LISP compilers
would be available.
37
,
The "instructor" would be one of the pilot groups for which a
predictive equation existed . The instructor's memory would contain
the predictive equation and all of the decisions made concerning the
64 situations presented in the second survey. The "student" would
contain a predictive equation, either from actual data analysis, or a
dummy one. It would have no decision history memory.
A. PROGRAM DESCRIPTION
The following matrices and vectors of interest are used throughout
the program:
(1) sit (i,j) i = 1, 64; j

- 1, 6
This matrix contains the six situational variables of the
second array:
3
= 1- fuel remaining,
j
- 2 - range,
j
- 3 - bearing,
j
= 4 - heading
j
= 5 - number of enemy,
j
= 6 - rules of engagement.
They are presented in the 64 different situations, i going

from 1 to 64 .
(2) mem (i,j) = 1, 3

i = 1, 64; j
This matrix contains the three decisions made by the

instructor over the 64 situations:
j
= l- primary tactical decision,
j
= 2- secondary enemy kills decision,
j
= 3 - secondary friendly kills decision.
38
(3) prico = 1, 8
(1) i
This vector contains the instructor's primary tactical

decision predictive equation coefficients for the variables
below:
i = 1 - a constant
L = 21 - fuel remaining
1=3 - range
1 = 4 - number of enemy
i = 5 - rules of engagement
i = 6 - danger
i = 7 - speed
1=8- MPG
(4) startp = 1, 8
(i) i
Completely analogous to prico this vector contains the ,
student's primary tactical decision coefficients.
(5) secoen = 1 9
(i) i ,
This vector contains the instructor's secondary enemy kills

decision predictive equation coefficients for the same
variables as in prico. When i = 9, the coefficient is for the
primary tactical decision reached.
(6) secofr (i) i = 1, 9

This vector is the same as secoen but for the secondary
friendly kills decision predictive equation.
(7) starse (i) i = 1, 9

Completely analogous to secoen, this vector contains the
student's secondary enemy kills decision coefficients.
(8) starsf (i) i = 1, 9

Completely analogous to secofr, this vector contains the
student's secondary friendly kills decision coefficients.
(9) kor (i) i = 1 , 8

This vector is the vehicle for the instructor's correction of
the student's startp coefficients.
(10) kore (i) i = 1,9

the student's starse coefficients.
39
.
(11) korf = 1,9

(i) i

the student's starsf coefficients.
1 . Main Program
The main program contains, in essence, the previously
described macro-level pilot education concept. The 64 situations are
read in (sit (i , j)) followed by the instructor's decisions over these sit-
uations (mem(i,j)) followed by the instructor's predictive equations
(prico(i), secoen(i), secofr(i)). This constitutes the instructor's memory
and expertise, his references and standards. Next, the student is
brought into the scene by the reading of his starting evaluation equations
(startp(i), starse(i), starsf (i)) , these data representing the starting
student at the beginning of the interactive instructor-student-situational
process
Next, for each of 900 trials, one of the 64 situations is picked
at random and presented to the student. After calculating danger, speed
and MPG; a skill hopefully developed through previous training, but
probably never applied, the student makes the primary tactical decision
for situation I:
IDECIS = startp (1) + .5 + startp (2) x sit (I, 1)

+ startp (3) x sit(I,2) + startp (4) x sit(I,5)
+ startp (5) x sit (I, 6) + startp (6) x danger
+ startp (7) x speed + startp (8) x MPG.
A subroutine INSTRU is now called. This is equivalent to the
instructor grading the student by saying "You've acted in the wrong
40
direction"; "too much"; "too little"; or "correct." From this grade the
student is either sent on to make his secondary decisions (correct) or
his primary evaluation equation is sent to KORECI subroutine for modi-
fication. If the student's grade is "not correct/' upon returning from
modification he returns to the beginning to have another situation
presented without attempting to make a secondary decision. If no modi-
fication is needed, the student's experience and confidence level is
increased and the secondary decisions are made for situation I:
IIEK = starse (1) + .5 + starse (2) x sit(I,l)

+ starse (3) x sit (1 ,2) + starse (4) x sit (1 ,5)
+ starse (5) x sit (I, 6) + starse (6) x danger
+ starse (7) x speed + starse (8) x MPG
+ starse (9) x IDECIS (enemy kills),
IIFK = starsf (1) + .5 + starsf (2) x sit (1,1)

+ starsf (3) x sit (1 ,2) + starsf (4) x sit (I, 5)
+ starsf (5) x sit (1 6) + starsf (6) x danger
,
+ starsf (7) x speed + starsf (8) x MPG

+ starsf (9) x IDECIS (friendly kills) .
As in INSTRU, these decisions are graded by subroutine GRADER.
These grades are considered after which either the student's equations
are modified by KOREII or KORFII or his experience and confidence levels
are increased. In either case, the student is sent back to the beginning
for a new situation. The grading and correcting of the secondary enemy
kill and friendly kill decisions are handled independently, hence
correction may take place in one but experience and confidence may rise
in the other.
41
2 .. Subroutine INSTRU
INSTRU first determines if it agrees with the student' s
primary tactical decision by checking the instructor's memory for the
same situation (mem(I,l)) . If the instructor and the student agree this
fact is immediately returned to the main program. If there is no agree-
ment, the instructor undertakes a coefficient by coefficient comparison
of its own predictive equations (prico) with the student's evaluation
equation (startp) for the primary decision. First compared as to sign
and then as to magnitude, a grade is sent back to the main program for
each coefficient; e.g., wrong sign, too small, too large, or, agrees
with instructor's coefficient. The coefficient comparison is undertaken
only when the student's primary decision does not agree with the
instructor's .
3. Subroutine GRADER
Subroutine GRADER is essentially the same as subroutine
INSTRU but deals with the grading and coefficient comparison of the two
secondary decision equations. The two secondary decisions are handled
independently.
4. Subroutine KORECI
Subroutine KORECI handles the modification of the student's
equation for his primary tactical decision. A student's attempted cor-
rection of the same error is not necessarily stable in magnitude,
42
especially when he is first beginning to learn to fly. As a student's
experience and confidence grows, however, the modifications become
more consistent and decreasing in magnitude. Consequently, an
experience modification factor, ZZ, is set. Previous runs had shown
that multiplicative modifiers differing more than about .11 from 1 caused
an erratic and sudden convergence of the student's coefficient to that
of the instructor's. Therefore, it was decided to limit the modification
factor to the closed interval [9 , 1.1].
The experience modification factor is set by taking a uniformly
distributed random number from the open interval (0,1). A different
number is chosen each time KORECI is called. This number is then
divided by the sum 10 + . 1 x experience and confidence level. This
experience and confidence level (X) is simply the number of correct
primary tactical decisions that have been made. Hence the experience
modification factor (ZZ) is in the open interval (0 , .1). Adding or sub-
tracting ZZ from one produces a modification factor in the open interval
(.9,1.1) that approaches unity. For example; a starting random number
of .5 with no experience and confidence gives a modification factor of
.95 or 1.05, a starting random number of .5 with an experience and
confidence level of 200 gives a modification factor of .98 or 1.02, and
a starting random number of .9 with an experience and confidence level
of 1000 gives a modification factor of .99 or 1.01.
43
After the experience modification factor is set, the student's
coefficients are checked to see if the variable paired with that co-
efficient is even being considered, i.e., is the coefficient zero? If
so, subroutine NOVICE is called, which means that more intensive
instruction is given, or, a return is made to a more elementary student
state. If not, the coefficients' gradings are examined and the co-
efficients are modified according to the results of these gradings. If
a wrong sign is indicated, the student's coefficient is multiplied by -1.
If the coefficient is too great in magnitude, the student's coefficient
is multiplied by the modification factor 1-ZZ. If the coefficient is too
small, the modification factor is 1+ZZ . A coefficient determined by
INSTRU as being correct is not modified.
5. Subroutine KOREII
Subroutine KOREII is essentially the same as subroutine
KORECI but handles the secondary enemy kill decisions. The experience
and confidence level is the total number of correct secondary enemy kill
decisions made by the student.
6. Subroutine KORFII
Subroutine KORFII is identical to subroutine KOREII but
handles the secondary friendly kill decisions. The experience and con-
fidence level is the total number of correct secondary friendly kill
decisions made by the student.
44
7. Subroutine NOVICE
The object of this subroutine is to instruct the student to
consider a variable that, up to this time, was not being considered
(coefficient of zero). Since data had revealed that out of 330 coefficients
considered in the analysis of all pilots and rank, hours, and mission
groups, 329 fell into the interval (-30,30), a uniformly distributed
random number is selected from the interval (-30,3 0) each time NOVICE
is called. This is then given the sign of the instructor's coefficient
for that variable and assigned to the student's coefficient in lieu of the
previous zero value.
8. Subroutine RANDOM
This subroutine, a slightly modified version of IBM's RANDU
[12], simply generates a uniformly distributed random number in the
interval (0,1).
(All routines can be found in the Computer Printout Section
of this thesis) .
B. RUNNING THE PROGRAM
The program was run with the average pilot for the entire sample,
the statistically average pilot from each rank group, the statistically
average pilot from each hour group and the statistically average pilot
from each mission group as instructors with the same groups' predictor
equations as starting coefficients for the students. Therefore, each
45
instructor was run with all of these average pilots as students . As can
be seen in Figures 4 through 15 and Table VIII, little differences exist
between the instructor-student performances.
As a result of these runs, a synthetic beginning student was con-
structed with all starting coefficients set to zero. Using this beginner
model as a student, the program was rerun with: 1) the average pilot
for the entire sample, 2) the Commander group, 3) the 4 000-plus hour
group, and 4) the 300-plus mission group as instructors. The latter
three groups were chosen for their expertise under the assumption that
a pilot who has attained higher rank, flown 4000 non-fatal hours or
survived 300 missions must possess some expertise in flight. The
results obtained from this "beginner" student run were plotted and
compared to those of the average student (See Figures 4-15, 16-27 and
Table VIII) . These comparisons and plots were then presented to the
panel of aviators for their comments and criticisms.
46
TABLE "VIII
RESULTS OF PROGRAM RUNS WITH TWO STUDENTS

AND FOUR INSTRUCTORS
Average Pilot as Student:

Time* to Make First Correct Decision
Instructor: Average Pilot CDR 4000+ 300 +

Primary Decision 5 5 5 5
Secondary Enemy Kills 5 5 5 5
Friendly Kills 5 5 5 5
Time* to Attain 90% of Final Correct Percentage After First Correct

Decision
Secondary Enemy Kills 19 39 8 73
Final Correct Decision Proportions
Instructor: Average Pilot CDR 4 000+ 300+

Primary Decision .85 .67 .87 .83
Secondary Enemy Kills .72 .63 .60 .55
Friendly Kills .94 .71 .62 .64
Synthetic Beginner as Student:
Time* to Make First Correct Decision

Primary Decision 2 00 221 199 206
Secondary Enemy Kills 12 9 194 172 149
Time* to Attain 90% of Final Correct Percentage After First Correct

Decision
Instructor: Average Pilot CDR 4000+ 300+
Secondary Enemy Kills 13 7 40 32 104
Final Correct Decision Proportions

Primary Decision .84 .65 .86 .83
Secondary Enemy Kills . 72 .61 .60 .55
Friendly Kills .93 .71 .60 .61
*Time is in units of trials for the primary decision and in units of correct
primary decisions for the secondary decisions.
47
V.. RESULTS OF PROGRAM RUNS
A. AVERAGE PILOT AS STUDENT
For the case of the average pilot for the entire sample as the
student, a successful evaluation of the situations occurred early in the
runs, i.e. , within the first five trials for all instructors. As can be
seen in Table VIII, the average pilot instructing the average pilot had
the highest overall average of correct decisions: 85% for the primary
tactical decision, 72% for the secondary enemy kill decision and 94%
for the secondary friendly kill decision. Naturally, the student's ending
coefficients were the same as the student's starting coefficients since
they are identical to the instructor's. The extremely high percentage of
secondary friendly kill decisions is puzzling in light of the low corre-
lation coefficient of .4015 between the predictor eguation with actual
secondary kill decisions made. Varying the random number seed changed
that percentage negligibly. An explanation may lie in the fact that the
rounding of the program decision equations to the nearest integer improved
the accuracy of the prediction with respect to the correlation observed.
For example, a predicted value of 1.4 9 friendly kills would be rounded
to one friendly kill by the artificial intelligence program. Assuming the
actual friendly kill decision to be one friendly kill, the artificial intel-
ligence program would show a higher correlation than the straight
predictor equations .
48
.
On the assumption that a student, upon reaching 90% of his final
performance, is near matriculation, the time for his running percentage
of correct decisions to reach 90% of his final percentage after his first
correct decision was calculated. The running percentage of correct
decisions reached 90% of the final percentage within five trials for the
primary tactical decision, five correct primary decisions (a correct
primary decision corresponds to a trial for the secondary decision) for
the secondary friendly kill decision and 19 correct primary decisions for
the secondary enemy kill decision.
The Commander instructor, as can be seen in Table VIII, whose
predictor equations differed the most from the average pilot's, showed
the lowest final percentage of successful decisions except in the case
of the secondary friendly kill decision; 67% for the primary tactical
decision, 63% for the secondary enemy kill decision and 71% for the
secondary friendly kill decision. The high friendly kill decision per-
centage may be a reflection of a rank-friendly kill correlation witnessed
in the data analysis (See Table II) . The running percentage of correct
decisions reached 90% of the final percentage within 62 trials for the
primary decision, 39 correct primary decisions for the enemy kills and
57 correct primary decisions for the friendly kills. These longer learning
times are felt to be reflections of the instructor and student coefficient
differences
49
The 4000-plus hour instructor, as can be seen in Table VIII,
closely matched the average pilot instructor in final percentage of
primary tactical decisions with 87% and neared the Commander instructor
in final percentage of secondary enemy kill decisions with 60%. The
4 000-plus instructor displayed the lowest secondary friendly kill decisions
with 62%. The time for running percentages to reach 90% of the final
percentages for this run were moderately low at 40 trials for the primary
decision, and eight correct primary decisions for both secondary decisions.
The low number of primary decisions correct needed to reach 90% of the
secondary percentages is a reflection of the lower final percentages.
The 300-plus mission instructor, as shown in Table VIII yielded
final percentages of 83%, 55% and 64% for the primary and secondary
enemy and friendly kill decisions, respectively. Running percentages
reached 90% of final percentages within 96 trials for primary decisions,
73 correct primary decisions for the secondary enemy kill decision and
95 for the friendly kill decision. These longer learning times are felt
to be a result of the modification factor (Z3) rather than a reflection of
lower percentages or instructor student differences. Although no such
run was made, the panel of aviators felt that a changed random number
seed would show reduced learning times.
However, as can be seen on the graphs of total decisions right
versus total trials or total primary decisions correct (See Figures 4-15),
50
the results are similar and nearly linear for all cases. The panel of
aviators felt this pattern to be consistent with their own experience,
i.e. , as a result of standardization of procedures and training, different
instructors teaching similar pilots would have, for the most part, similar
results.. They also considered the student, the average pilot from the
sample, as more experienced than the actual student encountered in a
flight syllabus. However, in a refresher or replacement training situation
where the student is experienced, such rapid attainment of consistent
results as witnessed in the graphs is felt by the panel to be typical.
B. SYNTHETIC BEGINNER AS STUDENT
The synthetic beginner student, with all starting coefficients set
to zero displayed a radically different behavior than that of the average
pilot (See Table VIII) . At the end of 900 trials, all the running per-
centages for the last 100 decisions were equal to the final percentages
witnessed when the average pilot was the student. Inspection of the
runs revealed the interesting phenomenon of long error periods followed
by rapid learning to a steady percentage.
With the average pilot as instructor, the first correct primary
tactical decision was reached on the 200th trial. The first correct
secondary enemy kill decision was not reached until after 12 9 correct
primary decisions had been made, the first correct secondary friendly
kill decision was not reached until after 157 correct primary decisions.
51
Figure 4
AVERAGE STUDENT-AVERAGE INSTRUCTOR
eoo PRIMARY RIGHT VER5US TRIALS
500
400 •
<300
zoo
100
>r i t i • i
t
loo zoo eoo !SOO eoo

PRIMARY RIGHT
52
Figure 5
600 -
SECONDARY ENEMY KILL6 RIGjHT
VERSUS
PR MARY R Q HT
I I
too zoo 300 -400 ffOO ceo

SECONDARY ENEMY KIGHT
53
Figure 6
S 00 SECONDARY FRIENDLY KILLS RIQHT

VERSUS
PRIMARY Ri QHT
SCO
h
I
<r>400
*300
£00
ioo-
L
ZOO zoo 300ôo &oo eoo
SECONDARY FRIENDLY KILLS RIGHT
54
Figure 7
AVERAGE STUDENT-CDR INSTRUCTOR
soor PRIMARY RIQHT VERSUS TRIALS
£00 ZOO 3< SOO GOO

PRIMARY RIGHT
55
V "re 8
AVERAGE STUDENT-CDR INSTRUCTOR
600r SECONDARY ENEMY KILLS Rl^HT

VERSUS
PRIMARY RIQHT
ZOO 4 0Q e>oo
SECONDARY ENEMY KlUk-S RIGHT
56
Figure 9
AVERAGE S TUDENT-CDR INSTRUCTOR
600 r SECONDARY FRIENDLY KILLS RIGHT

VERSUS
VR 1 f\A A F\ Y "RIQHT
soo
<-00 -
300
cczoo
a:
<
2ioo
o.
y^
1^
„ !
100
SECONDARY y-OO
t
FRIENDLY
1 ft'-
400
KILLS
%
,500
RIGHT
•
oOO
57
Figure 10
AVERAGE STUDENT-4 000 HOUR INSTRUCTOR
Go°r PRIMAPxY KIQHT VERSUS TRIALS
PRIMARY %HT SCO 400 SQO eoo

Figure 11
AVERAGE STUDENT-4000 HOUR INSTRUCTOR
soar SECONDARY ENEMY KILLS RIQHT

VERSUS
PRIMARY RIQHT
ZOO 400 500 ©OO

iOO
SECONDARY ENEMV300 KILLS RIGHT
59
Figure 12
AVERAGE SrUDENT-4000 HOUR INSTRUCTOR
booc- SECONDARY FRIENDLY KILLS RIQHT

VERSUS
PRIMARY RiQHT
Soo -
S-QO
3oo
h
i
o
&ZOO
>•
<
5 100
4-00 500 600

SECONDARY FRIENDLY KILUS Rl GHT
60
Figure 13
AVERAGE STUDENT-300 MISSION INSTRUCTOR
GOO PFUMARY RIQHT VERSUS TRIALS
500 •
4 0G -
300
zoo
to
J
<
<*ioo
^ i
loo
, .... !
zoo 300
i
i
4-00
i
500
•
6C
PRl MA RY RIGHT
61
Figure 14
AVERAGE STUDENT-3 00 MISSION INSTRUCTOR
GOO •
SECONDARY ENEMY KILLS RIQHT

VERSUS
PRIMARY RIQHT
lOO
„
SECONDARY ENEMY ZOO KikLS400 RIGHTSO©
ZOO 600
62
Figure 15
AVERAGE STUDENT -3 00 MISSION INSTRUCTOR
600 r SECONDARY FRIENDLY KILLS RIQHT

VERSUS
PRIMARY R1QHT
lOO 2,00 300 400 TOO u<;C>

SECOMDARY FRIETNDLY R I G, HT
63
It took 40 additional trials for the correct primary decision percentage
to reach 90% of the final percentage. The secondary enemy and friendly
kill decision percentages took 13 7 and 56 additional correct primary
decisions, respectively, to reach 90% of their final percentages.
With the Commander instructor, the first correct primary decision
was attained after 221 trials, the first correct secondary enemy kill
and friendly kill decisions after 194 and 177 correct primary decisions,
respectively. After 4 7 additional trials, the correct primary decision
percentage reached 9 0% of the final. The secondary enemy and friendly
kill decision percentages reached 90% of the final percentages after 4
and 111 additional correct primary decisions, respectively.
With the 4000-plus hour instructor, the first correct primary
decision was reached after 199 trials, the first correct secondary enemy
and friendly kill decisions coming after 172 and 161 correct primary
decisions, respectively. Forty -eight additional trials were needed to
attain 90% of the final correct primary tactical decision percentage, 32
and 72 additional correct primary decisions were needed to attain 90%
of the final correct secondary enemy and friendly decision percentages.
The 300-plus mission instructor used 206 trials before the first
correct primary tactical decision was reached, 14 9 and 177 correct
primary decisions before the first correct secondary enemy and friendly
kill decisions, respectively. Eighty-three additional trials were needed
to attain 90% of the final correct primary tactical decision percentage,
64
I
TabLe 16
BEGINNING STUDENT -AVERAGE INSTRUCTOR
6©o r PRIMARY RIQHT VERSUS TRIALS
100 -4-00 i;ô 600

P I L ARY RIGHT
65
Figure 17
600 SECONDARY ENEMY KILLS RIGHT

VERSUS
PRIMARY RIQHT
500
«K;
300
h
âoo
<
0.100
40O
SECONDARY ZOO E'NETMY300 KILLS
100 ('0 I 00
RIGHT
GG
Figure 18
G-oo r SECONDARY FRIENDLY KILLS RIGHT

VETRSUS
PRIMARY RIGHT
500 600
too
SEuCGN
zoo
DARY FR\ ENDU4 00 KlULS RIGHT
67
Figure 19
BEGINNING STUDENT-CDR INSTRUCTOR
600 PRIMARY RIGHT VERSUS TRIALS
SOO -
40O
300 -
zoo
<0
J
\RY RIGHT
68
Fiqure 2
SECONDARY ENEMY KILLS RIGHT

VER5US
PRIMARY RIGHT
SOO
*oo
2300
>ZQG
ft
<
CLiOO
ZOO 400 SOO eoo

iQQ
5ECONDARY ENEMY300 KILLS RIGHT
69
Figure 21
600 r SECONDARY FRIENDLY KILLS RIGHT

VER5US
PRIMARY RIGHT
500
400
so:>
aoo
<
&.ioo
j. j. j
too zoo 300 400 Soo 6oo
S£CQN DAR.Y FRIENDLY KILLS RIGHT
70
Figure 22
BEGINNING STUDENT-4000 HOUR INSTRUCTOR
600 PRIMARY RIGHT VERSUS TRIALS
3.°a 400 600

PRIMA2.QO
RY Rt GHT
71
Figure 23
6°°r SECONDARY ENEMY KILLS RIGHT

E KS U S
\J
PRIMARY FLIGHT
£00
4O0
300
h
I
2
£00
CLlOO
^ lOO ZOO 3GO 400 500 600

SECONDARY i-TWEIVIY KILLS RIGHT
72
Figure 24
CC :> SECONDARY FRl ENDLY KILLS RIGHT

VERSUS
PRIMARY RIGHT
5QQ
< DO
^300
H
O
yZOO
<
a. loo
loo 200 300 €o o

73
Figure 25
BEGINNING STUDENT-3 00 MISSION INSTRUCTOR
COO PRIMARY RIGHT VERSUS TRIALS
500
40O -
3oo -
/
ZOO
J
<
hioo «
v*"^ i » i -»
loo zoo 3O0 400 SO© GOO

PRIMARY RIGHT
74
Figure 26
BEGINNING STUDENT -3 CIO MISSION INSTRUCTOR
eoo SECONDARY ENEMY KILLS RIGHT

VERSUS
PRIMARY RIGHT
500
400
30Q
o
5
zoo
<
E
ILiOD
tOO £OG SOO 4-0O 5TOO 6CO

SECONDARY ENEMY KIUV.S RIGHT
75
Figure 2 7
BEGINNING STUDENT-3 00 MISSION INSTRUCTOR
600
f
VERS US
PRIMARY RIGHT
SOC7
4DO •
XSOO
•
-
^2,00
<
0.
lOO r ^r
i *sr^ i l i , j
too ôo BOO 40Q 500 GOO
76
104 and 225 additional correct primary decisions to reach 90% ofthe
first correct secondary enemy and friendly kill percentages, respectively.
In comparison with the runs and graphs for the average pilot as
student, three significant points stand out (See Figures 4-15, 16-27
and Table VIII):
(1) The beginner requires a lengthy time to attain even one

correct decision, no matter what decision is being considered
(2) After a correct decision, the beginner in general, needs a

longer learning time to reach the average pilot's degree of
sophistication.
(3) Once this degree of sophistication is reached, the beginner

and average pilot are nearly identical in behavior.
In essence, the beginner is, indeed, a beginner. In fact, the
beginner could even be considered the average pilot three hundred trials
younger. The panel of aviators considered the beginner as analogous to
the beginning aviation student: the long, flat, error period being as
in his ground and basic schooling where the basic ideas and guidance
are implanted; the learning period as the time when a student's motor
skills are honed to the point that unencumbered mental integration of the
entire flight situation is developed - the transition from flight student
to a full fledged Naval aviator; and the more constant, steady period
as the reliable activity of an experienced pilot.
As can be seen in Table VI, discrepancies do appear between
instruction as in the average pilot student runs. Whether these dis-
crepancies are significant or not is a matter of conjecture. In light
77
of the data analysis showing little correlation of biographical features
with decisions reached, this question of significant instructor differences
must remain unanswered unless further research is conducted.
C THE ARTIFICIAL INTELLIGENCE QUESTION
Does this program and its resulting runs fall into the classification
of artificial intelligence? Taking the program in its entirety, one finds
that it does analyze its own performances, does diagnose its failures
and does make changes that enhance its future effectiveness and per-
formance. This behavior, if witnessed in a human being, would be
called intelligent behavior. Therefore, by previously cited criteria
[8, 11] this program does, indeed, fall into the classification of
artificial intelligence.
78
VI.. CONCLUSION
In summation, the low correlations between the biographical
factors of rank, hours and combat missions and the decisions made in
the evaluation of an air-to-air combat situation. is disconcerting. The
explanation that the standardization of training and flight procedures
may account for this lack of relationship seems of doubtful validity
to the author. The one area showing any hint of a relationship is rank
and the prediction of friendly kills. Still, the coefficient of correlation
is weak (See Table II) .
The multiple correlations between the decisions and situational
variables are moderately low to moderately high, varying from .3378 to
.8193. The lack of any high correlation coefficient values may be due
to randomness in the primary and secondary decision making among the
pilots. It may also be due to the failure of the multiple regression to
capture the decision making policy either due to missing variables or
a failure to capture a non-linear relationship with a linear function.
Still, in the case of the average pilot, the equations produced from the
multiple regression predicted 757 correct primary decisions out of 900,
548 correct enemy kill decisions out of 757, and 714 correct friendly
kill decisions out of 757. This indicates a rather adequate capturing
of the average pilot's decision policy. The relatively low correlation
statistics may, in this light, be misleading.
79
On the average, the variables most influencing a pilot in reaching
a tactical decision of the types shown in Figures I and II appear to be
MPG and danger, the least important being fuel remaining and rules of
engagement (See Table VI). The fact that fuel remaining is considered
least important while MPG, a function of fuel remaining, is considered
most important indicates the necessity of considering MPG as a separate,
though not necessarily independent, function of range and fuel remaining.
A study by Rigney and De Bow [6] has shown that 63 Combat Infor-
mation Center watch officers considered range and enemy course (heading)
as the most important air raid threat variables. Although the watch
officer's situation is radically different from that of the pilot, the fact
that range is an argument of the MPG function and course is an argument
of the danger function is interesting, especially in light of the fact that
the watch officer does not consider fuel.
The most important variables considered in making the secondary
enemy kill decisions are the tactical decision reached, fuel-remaining,
and rules of engagement (See Table VI) . The least important are MPG
and the number of enemy. An interesting reversal of importance can be
witnessed in MPG being considered most important in reaching a tactical
decision and least important in the secondary enemy kill decision. Also,
the two least important variables in the tactical decision are second and
third in importance in predicting this secondary enemy kill decision.
80
.
Number of enemy, tactical decision and speed seem most important
in predicting pilots' friendly kill expectancies . Least important in this
prediction are MPG and range. Again, in comparison with the tactical
decision, MPG is seen to switch roles. Also noticeable is the number
of enemy's rise to most important while, in the enemy kill decision, it is
the next to least important. The three decisions appear to be, in fact,
three separate decisions, the first (primary tactical) affecting the others
(enemy kills and friendly kills) but neither of the latter two affecting
each other.
Looking at the pilots individually, and within their biographical
groups, differences do appear to exist in how they reach their tactical
decisions. But these differences do not appear to be correlated with
rank, hours or missions but are, rather, insignificant random pertur-
bations. Secondary decisions show little cross-group variation (See
Table VI and Appendix G) .
The artificial intelligence program does appear to show adaptive
learning behavior when the program is taken in its entirety. The behavior
of the program can reasonably be termed intelligent. Therefore, one
could safely say that the variables and factors influencing an air-to-air
combat evaluation can, for the most part, be quantified and utilized in
an artificial intelligence application of air-to-air combat decision
making policies
VII. CRITICAL EVALUATION OF THE STUDY
Several pertinent critical comments pertaining to this study are:
(1) The group of 36 pilots from which data were gathered is too small
to allow any justifiable generalization to Navy pilots in general.
Consequently, as a result of this small size, any statistics and
findings generated may not be representative of :Navy pilots in
general.
Granted, no generalizations about all Navy fighter pilots can be

safely made from these findings. An appreciably larger sample
is required. Still, the sample did cover 2304 separate evaluations.
Any generalizations drawn do apply, at least, to the group and
situations studied.
(2) The assumption that certain factors, such as rank, primary

decisions, and rules of engagement, can be quantified is gues-
tionable, i.e., does a Commander possess five times the amount
of rank as an Ensign?
Granted, especially in the area of attempting to quantify a variable

such as rules of engagement. Still, quantifying rank and placing
the decisions on a continuum are not necessarily unrealistic.
Perhaps a better substitute for rank would be lineal number and
the subjects should be asked to give a number value on a decision
continuum; the relationships between primary decisions as presented
not necessarily being linear.
(3) The situational variables such as bearing, heading, fuel states,

range and number of enemy are not, in reality, restricted to only
two states .
Granted, however, questionnaire size limitations enforces such

a binary state. Still, a better data base could result from a
broader representation of the situational variables.
(4) In light of the low biographical-decision correlation, how can one

assume that expertise is proportional to a biographical attribute?
The question of expertise, although interesting, is essentially

a moot one. The only true criteria of expertise in the situation
would deal with the results of actual air-to-air combat. Since
the number of people with this kind of expertise is small and not
82
. .
available to the researcher, "experts" were picked by the previously

described method.. Whether, as a.rcsult of learning, the student
has developed into a better or worse fighter pilot cannot be deter-
mined is irrelevant. What is relevant is whether or not the student
approaches the standards of his declared "expert" instructor.
(5) Itappears unrealistic to ask a pilot with no combat experience to

evaluate a combat situation.
Although seemingly unrealistic on the surface, the armed forces

do. require a person with no combat experience to make an evaluation
of an actual combat situation. However, the person normally has
undergone extensive and realistic simulated combat training. All
of the 36 pilots studied have undergone such training. An interes-
ting note is the similarity in decisions between the no combat
group and Lieutenant Commander group which has an average
combat total of over 160 missions (See Table VI and Appendix G)
(6) The artificial intelligence program is simply a model of a situation-

instructor-student interaction.
Whether the artificial intelligence program is a model or does not

detract from the fact that it meets the stated criteria of artificial
intelligence
(7) Certain factors may be used for predictors several times in the
same equation, such as MPG, range and fuel, raising the multiple
correlation accordingly.
Granted, but the apparent separation of seemingly related functions

in the ranking scales (See Table VI and Appendix G) and the exper-
ience of the panel of aviators does not make this practice appear
unrealistic.
(8) The student's confidence level increases with success but does
not decrease with failure, a situation that may be unrealistic.
Granted, however, when a decreasing confidence level was imple-

mented, the modification factor in the beginning stages grew so
large as to cause erratic and rapid convergence to the instructor's
coefficients. Overcoming this problem, however, would not be
difficult.
83
VIII.. CONTINUATION OF THE STUDY
More data are desperately needed before any of the results un-
covered can be safely generalized to Navy pilots in general. Likewise,
more situations are needed for presentation to the subjects. A factor
indicating a consideration of the enemy's plight must also be sought.
For example, variables indicative of the friendly airplane pointing at
the enemy or number of friendlies could be considered.
The computer graphics field is a likely method to use in future
data collection and in developing real time interactive artificial intel-
ligence applications. For example, the cathode ray tube screen could
easily be configured to resemble a cluster of instruments and radar scope.
The situation would then be presented on the scope via simulated radar
images, digital readings on the instrument and appropriate character
generation. Via random number generation a multitude of situations
could be presented to the subject. Teletype or light pen input would
increase the subject's interaction and speed of response, allowing
several hundred responses in the space of an hour. Real time data
collection and analysis could produce desired statistics and multiple
regression equations on demand. In the same manner, by using the
artificial intelligence program described in this study, the human could
take over the role of instructor with appropriate modifications for response
collection and analysis. By not utilizing the modification subroutines /
84
the human could take the place of the student, his coefficients being
evaluated and graded by the instructor, followed by a response such
as "You are not emphasizing your fuel state enough" or "You are neglecting
your range consideration." The training, selection and research possi-
bilities in this area are boundless, limited only by the researcher's
imagination. Through this kind of work, the fighter pilot may become
less of an enigma, less of a superman; more of a human.

.
APPENDIX A
QUESTIONNAIRE
1 . a) Relative position (range and bearing) of the

enemy aircraft to the strike force.
b) Type enemy aircraft suspected or known.
c) I cannot choose between them
2. a) Composition of enemy force number of raid(s) —

and number of aircraft in each raid.
b) Number of assigned TARCAP aircraft available.
c) I cannot choose between them.
3. a) Range of enemy aircraft to the strike group.
b) Speed (absolute) of the enemy aircraft.
4. a) Relative position (range and bearing) of the

enemy aircraft to the strike group.
b) Range of enemy aircraft to the strike group.
5. a) Type enemy aircraft suspected or known.
b) Capabilities and systems operability of your

own aircraft (e.g. , fuel state, number of
missiles , etc .)
6. a) Speed (absolute) of the enemy aircraft.
b) Altitude of the enemy aircraft.
86
.
7. a) Anti-aircraft, including surface to air mis-

sile (SAM) activity in your immediate area
, .

missiles, etc .)
8. a) Altitude of the enemy aircraft
b) —
Composition of enemy force number of raid(s)
9. a) Number of assigned TARCAP aircraft available.
b) Bearing of the enemy aircraft from yourself.
10. a) Bearing of the enemy aircraft from yourself.
b) Anti-aircraft, including surface to air mis-

sile (SAM), activity in your immediate area.

b) Rate of closure of enemy aircraft to yourself. »
12. a) Rate of closure of enemy aircraft to yourself.
b) Composition of enemy force number — of raid(s)

87
13. a) Number of assigned TARCAP aircraft available
b) Range of enemy aircraft to the strike force.
b) Relative position (range and bearing) of the

15. a) Capabilities and systems operability of your

own aircraft (e.g. fuel state, number of mis-
,
siles , etc .) .


missiles , etc .) .
88
19. a) Type of enemy aircraft suspected or known.


b) Composition of enemy forces number of raid (s) —



own aircraft (e.g. , fuel state, number of mis-
siles , etc .) .

siles, etc.) .
89
.
25. a) —
Composition of enemy forces number of raid (s)
and number of aircraft in each, raid
b) Bearing of the enemy aircraft from yourself. .


29. a) Altitude of the enemy aircraft.
b) Type of enemy aircraft suspected or known.
90
31. a) Bearing of the enemy aircraft from yourself. .
b) Capabilities and system operability of your

siles, etc .) .
32. (a) —
Composition of enemy force number of raid (s)
b) Type of enemy aircraft suspected or known.
b) Rate of closure of enemy aircraft to yourself.
34. a) —
Composition of enemy force number of raid(s)

siles (SAM), activity in your immediate area.
b) Capabilities and system operability of your

siles , etc .) .
91
.
3.7, a) Speed (absolute) of the enemy aircraft . .

siles (SAM), activity in your immediate area.
38. a) Range of enemy aircraft to the strike force.
b) Altitude of the enemy aircraft..
b) Number of assigned TARCAP aircraft available


own aircraft (e.g. fuel state, number of mis-
,
siles, etc .)
92
. .
43 .. a) Anti-aircraft, including surface to air mis-

sile (SAM) , .activity in your immediate area .
b) Number of assigned TARCAP aircraft available
c) I. cannot choose between them.



siles , etc .)
b) Composition of enemy force--number of raid(s)

b) Composition of enemy force number — of raid(s)

93

b) Speed (absolute) of the enemy aircraft..

54. a) Composition of enemy force—number of raid(s)

94
APPENDIX B
PILOTS RESPONDING TO MAIL-OUT QUESTIONNAIRE
Rank Hours Missions
1. Commander 4500 70
5. Lieutenant Comma nder 3450 200
7. Lieutenant Commander 3200 140
12. Lieutenant Commander 2500 11
15. Lieutenant 2350 80
30. Lieutenant 1050
32. Lieutenant 850
33. Lieutenant 750
34. Lieutenant 750-
35. Lieutenant 730
36. Lieutenant (junior grade) 450
95
APPENDIX C
PANEL OF AVIATORS
Commander (selectee) - pilot, attack and fighter type aircraft experience-

two instructor tours in the basic training command; one
instructor tour with an attack Replacement Air Group; three
combat tours in Southeast Asia; candidate for M.S. in
Computer Science.
Lieutenant Commander - pilot; patrol type aircraft experience; one

instructor tour in the basic training command; one combat
tour in Southeast Asia; candidate for M.S. in Computer
Science.
Lieutenant - pilot; attack type aircraft experience; one instructor tour

in the Replacement Air Group; one combat tour in Southeast
Asia; candidate for M.S. in Computer Science.
Lieutenant - Naval Flight Officer; experience as airborne intercept

controller; one instructor tour in the basic training command
teaching computer fundamentals; one combat tour in South-
east Asia; candidate for M.S. in Computer Science.
Lieutenant - Naval Flight Officer; experience as airborne intercept

controller; one tour working on airborne computer display
systems; one combat tour in Southeast Asia; candidate for
M.S. in Computer Science.
96
.
APPENDIX D
DISCRIMINANT ANALYSIS FOR GROUPS DIFFERENTIATED

BY BIOGRAPHICAL DATA
Significance Level of ..05
Grouped by Rank:
Chi square = 14 .
Degrees of Freedom = 21
No significant differences between groups differentiated
by rank
Grouped by Hours:
Chi square = 24 .0
No significant difference between groups differentiated by
flight hours .
Grouped by Missions:
Chi square = 44 .0
No significant difference between groups differentiated
by combat missions .
97
APPENDIX E
SELECTED PREDTCTOR EQUATION COEFFICIENTS

Primary Tactical Decision —Corresponds to Prico Vector:
Average Pilot CDR 4000+ 300+
I.. 1.51835 1.23987 1.74982 1.12484

z... -.00014 —.00005 -.00015 -.00005
3.. .3 910 .05223 .03590 .05204
4:.. ..05065 .03086 .05952 .03043
&:.. -.122 3 9 -.07430 -,08141 -.06924
6. .003 52 .00223 .00132 .00440
7., .0002 8 .00023 .00016 .00079
8. 8.24161 5.59970 4.11829 -3.22020
Secondary Enemy Kill Decision - Corresponds to Secoen Vector:
Average Pilot CDR 4000+ 300+
1. .82003 1.20750 .23976 1.06406

2. .00034 .00018 .00038 .00024
3. .02213 .05697 .05474 .01154
4.. .03087 .12087 -.01666 .18579
5. .41223 .20282 .34197 .21952
6. -.00281 -.00306 -.00242 -.00253
7. -.00020 -.00001 .00013 -.00008
8. -7.81100 -13.56913 27.20380 11.08119
9. -.43846 -.66584 -.22533 -.57411
Secondary Friendly Kill Decision - Corresponds to Secofr Vector:
Average Pilot CDR 4000+ .

300+
1. .56915 .39760 .74056 .17621

2. -.00004 .00012 .00009 .00007
3. -.00558 -.01702 -.01714 -.01947
4. .08816 .18036 .08989 .30649
5. -.09740 -.23837 -.25172 -.26590
6. .00059 .00129 -.00023 .00278
7. .00013 .00019 .00014 .00024
8. 3.26289 28.96692 2 5. 1082 24.35280
9. -.24901 -.32516 -.33688 -.28021
98
APPENDIX F
MEAN AND STANDARD DEVIATIONS OF DECISIONS MADE
Rank Group Primary D Enemy Kills Friendly Kills
CDR X 1.95 1.40 .61

SD .82 1.09 1.01
LCDR X 1. 36 .93 .32

SD ..80 .99 .64
LT X 2.01 1.10 .18

SD .78 1.00 .48
LTJG X 2.16 .70 .08

SD .76 .75 .27
Hours Group
4000+ X 2.16 .83 .41
SD .80 .78 .55
4000 X 1.83 1.44 .51

r
SD .79 .95
3000 X 1.92 .96 .26

SD .79 .94 .61
2000 X 2.07 1.04 .16

SD .96 1.06 .45
1000 X 1.99 1.00 .26

SD .77 .84 .55
Mission Group
300 + X 1.74 1.56 .88
SD .92 1.02 1.28
300 X 2.02 1.10 .19

SD .74 1.08 .53
200 X 2.01 .96 .23

SD .80 1.01 .53
100 X 2.01 1.10 .23

SD .80 1.04 .57
X 1.97 1.07 .24

SD .75 .85 .52
X = arithmetic mean
SD = standard deviation
99
APPENDIX G
VARIABLE RANKINGS FOR HOUR AND MISSION GROUPS

HOUR GROUP:
Primary Tactical Decision
4D00+ 3000-4000 2000-3000 1000-2000 0-1000

MPG 1: 5: 7 1 7
Danger 5 2. 2 3 3
Speed 3 3 3 4 2
Range 4 i: 1 6 1
Rules of
Engagement 7 6 6 7 6
7"
Fuel Remaining 6 5 2 4
Secondary Enemy Kill Decision
4000+ 3000-4000 2000-3000 1000-2000 0-1000

IDECIS 3 1 1 2 1
Rules of
Danger 5 3 3 4 5
Speed 6 6 5 5 3
Range 2 7 6 6 -
MPG 8 8 - 8 6
Secondary Friendly Kill Decision
4000+ 3000-4000 2000-3000 1000-2000 0-1000

IDECIS 1 2 1 2 2
Speed 4 4 3 3 3
Rules of
Danger 8 3 7 6 6
Range 6 7 4 4 7
MPG 6 8 - 5 8
100
APPENDIX G. (continued)
VARIABLE RANKINGS FOR HOUR AND MISSION GROUPS

MISSION GROUP:
Primary Tactical Decision
300+ 200-300 100-200 1-100

MPG 7 i: 1 1 7
Danger 3 2. 2. 2 2
Speed i: 3 3 5 3
Range 2 4 5 4 1
Number cf Enemy 4 5
r
4 3 5
Rules of
Secondary Enemy Kill Decision
300+ 200-300 100-200 1-100

IDECIS 1 1 1 1 1
Rules of
Danger 5 3 4 4 5
Speed 7 7 6 5 3
Range 3 6 5 6 6
-
MPG 8 8 -- 8
Secondary Friendly Kill Decision
300+ 200-300 100-200 1-100

IDECIS 5 1 2 1 2
Speed 6 7 3 3 3
Rules of
Danger 4 3 7 f» 6
Range 2 8 7
MPG 7 7 8
101
APPENDIX H
MULTIPLE CORRELATIONS OF PREDICTOR EQUATIONS

WITH PREDICTED DECISION
Primary Tactical Secondary Enemy Secondary Friendly

Decision Kill Decision Kill Decision
Average Pilot .6032 6027 .4015

CDR .5794 5859 .4391
LCDR .6236 5654 .5299
LT .6101 6527 .3732
LT(jg) .7568 8193 .5284
Hours
4000+ .4587 .7930 .5840
3000-4000 .6775 .5638 .4303
2000-3000 .6249 .5988 .5217
1000-2000 .6816 .6755 .3378
0-1000 .6062 .6190 .4442
Missions
300 + .7459 .7052 .5128
200-300 .6402 .5748 .4524
100-200 .5778 .5915 .3397
1-100 .6309 .6677 .5309
.6213 .6143 .4499
102
>. 1 1 1
COMPUTER PROGRAM
_l>- ITl n
<£ n n- *-^
K-«- - « CO S
CC -U- «— o
- - o_ #
•+- or i- m
- rrtu or .-)
h- OCO ••*••
s:_i^ n o a:
z nrxj.-* or z co
m - r-KCC U_ •» (—
OH- Z
•*—!*-
• »-iUJ>-
^-Jo:
OS
LUQO.
UJ
e-i
CO K-<T 1/5 UJ 2- o
>>-u_2: *>uj co -<
sri—H-i ^-coi— f_ u_
I— U-i •»- C7>»-iQ^
w Z U_
XZV. ». ..u. UJ UJ
Z^z
Mr «a LUO»
cj>» «. •-< o
—• «> O O
o • - o. 0—2; -<
O »»UJ CJLL.UJ u. o ox
O z
~-
^
I—
«-
-
CoT
>CD
C^5:
uj
oi
U-i
00
21
O
Wars:
coo
—
~m.cc
CO—-Z
«-UJ
••
*-r *»
Oh-
u_
UJ
C_>
f-t
ot
<
o
I—
<
CC
UJ
-* <Z>- - '-O r- si" vj" >- h- Z
O SZv • C«-t— O O CC CO UJ
O h» o« OUi2 •• •• < o
w - -^» o:oz 11 it z cc cc
f— ••(mi— 10 o_v: •• *•-• »-> o <r —»—.—»~. lu
LL » HI> •• •>!— •« •« O Q —.OCOOO CO
*- C—'O — »•— «Qi •»•» <— « UJ —.— . Z *-»»™,-»^-.QN n * n •> 21
—« UJ^H- mcouj sO ro co ^C7>cr* O coooco n—ir-i,-iT-< 3
o a'ja'l s-z: •» *• co •- o •»•-•. »-. — 1 11
— •— —in-
11 11 11 z
O Uj|— O «sfOT • <—i f—l O «-r—li — i UJ i—In-ti—li-4 II » 1
)
O COLLI >-- vOOS II II Z ^H II II CO II II II

—•»—
II 1-1 • •" •> 21
*— ZLt— CC — —•'•âCL ~~) —) <S. •— 1— »— i— •>—"»-«»«-»—
—
1 • 1
II 1 •
h- D 1.. o 5: -Z
—
«- n •— < « •» O «-»•»- •« . >-< _< -! »-< z
UJ ZV
•~\- - -
•*-« uj.
SZOcvT
. •• —«
—>>—} «-^ >-
cc
•>
— —!»— <
— •*"• 2T ^-.-.owww^s* z
mwmh —ô^Cs-Uill < QCC
O
a: •
—<••. V, <• •»— O. •- CC r. -< >-< s: «~.z:a; >• o.uju-oujOa'cx'a: IX UJ
O C\| >~i_j sOcO" •- "Z. — t-t OUJU. CC l~cOooOOO<<< OCT r-IUJ z
O
—
-«
—-
oi_i »-c£f-—
<h<t3c\j
1-
m
lu s a; OOO
wuo < rvJctToioXM-ouJi-i— 1—
-<£ <r <r oi u j co c-T co co
O II CO
1—
•
' q. >—< 21 uj o. 21 U- i-— 1 r—
^NvCH- CL>— —tl— h-l— Cu CO •-' »-—' —
mDT < O+
CO OiUJUJ 1— •?"<-» \~
I—- UJ 2!
Q.^_J Di» «• ^COZ-J w CO — CO O-COCO CC
.
COCOCO'-"— '-»—.-»—
II
_J|—
ÛJt—in n <»
UJ »> h~ ••
I—<-~.(—
**UJ
O CO
*—
—n
»
Oi
*—
—»
-
Ct"
»— •»-»—
—>^-.—»
Q. t—i-»-s^»^-^^.0'-<C\im
M^-s-iCOl^HHi-IH
UJ
UJ
ÔO
roô H- 4-
•
Ocoi— « •>;- o o cn<t-vj- co

cooôiz: z
rn>3">t *•»•••»•*•«
«-< Z c\j vO CO CNJ ^-<
•—><Z ^-~ y- nh- n \— n n n m 0*~»- >OOOOvOO
«--0.<.'0 r-iO< CO ;:-
co_Jco "cDj o en o ^TLusro i-

lALnin h- r-<LnLrMr\— »—
1 —• —
la*-«— O01 jr
O — O — *-"— 2: O — —•»-UJU. UJUJUJU-' z OO
II
Z >$• OcO
» 2IO I— <— UJ O l
02 2:or<_j < o o cc 000 uj hqodhi-i-i-hi- X z

uj_j_j
2T<< 2TUJ 21<l— _l O < H < H <<< O
»-<ujuj >-i~ Ol— << h- uj co UJ co ujujuj
UJrH-J
m wmhwm h- UJ _J l— cc
cc:
3 OujiJUUJoXoiaiQrQiry: or COO<I
«< l-H
II
Oa:a:cocooi-5;- ocoqo ooooiQissrssss < •-HOO h- —

a: z. cc tz cccccc i—
OOO
«-< *
r-»l—(t-Mr-l »-l CO 1—* *-< CO h- Z

UJ
CO
o 10
< <t < < 1- UJ
UJ LU UJ UJ UJ or
C£ CC CC CC CO a.
103
4i N «
>
• ii <ii < 4i 1 » o— 4i < t1
4 - ) t <1 * -
»4
^•**-
>t<t -^co # CO
00 "-4— —«c£: — cr
in CLOO <t< <r<
or- h- + — v- *-i-
C\J CrtQ LUCO U. CO
<ILU CO + CO +
f~ J— LU ctâ cr:a
in COCL <LU <LU
HLU
—+ CO l-LU
•«*.
^-* ?;- COO. COCL

r^ c\!-n + CO + CO
-<
•* r--- ~# — «•
1—4 >»-*.- r\j— rg—.
«M« <— Q.- h- i»h-
1- k-l- t—4<-^ h-4«—
1—4 t—'C£ n_LU — LL
CO 00< Hco l-CO
s** mo: mo:
— k-
•ii-
CO CO CO< CO<
D C0 + ^ k- « l-
o «- cC — CO — CO
•* D-UJ ro + m+
o i-o ^âi N~C£
• aLZ. LULU U. LL'
— n <s<t COO coo
O— in k-O a:z: c^z:
• *~-N co*- << <;<
— +—
<t-
oco l-Q K-O
sOin ZT — vO </>;(- CO-?!-
COC* o i—1«— +— + —
•-CM 1— t-CL — vQ -—vO
o •
«>-
CO H-4|- CO H- —l«—
1 «-«: 1- •nLU ••LL
Oin <-» k-< 2: •—<co -icO
o- COv, CO M(- LU *-*c£ -*• wCfl
>- rH — in coco t— t-<co H-h-
Q *^* c* c\j *~*r-N *— i
co coo COCO
Z vj*W • (NJvjO LL >" + LU
--.—»o
?;-
'-.»—
+
LU t-*-* l*» «n** t*. LL.
t— ^H- m CL*"* .^^ LU (\| vO 1 — zr rsJsO
a: N^" »— "V 1
'
CO o «_- »»* o •w t*
u_ l-CO cc\~ 1—4 o LU»-<H- 1—4
t—1*~» -? O O z: co*--< co
LU CO 2! *• 1— CO LU o zr o oc:i--o H- C£\~
X «w*-»~4 »— —« CO-ii- a r-1 o »-H <Ik-<_I u <•—
h- Oco *-- »— +— t~4 1—4 CO 1— COLL LU 1— CO
o - 1- 4*. mm •• o CO t—t CO-5<- Q co^;-
+—
-5i-
t~ SLCC — 1—4 • «— t— h- >— o +

mmo^ > mm
t
< <LU CO «— + Cl QL O —~ LU
o «— K- -.(_. 2: CL a LU \- Q —«— -J
• *—
+ LULU Q + LL
1
O — z: CO >-H -Z hq: o -H o Q CC
—»COCO 2! — CO
ZT on< o a CO o— —•< ^-< IN c-
— 0. >-
i— Q o — •X 4
CLk- CO
— o
(—4 \~ >- z: 5: <~4Qf C^ LU —*CC
K 1 1«*-* ~l >*- 1—
<
<— oo C-CO 1—4 r-l cC —
—
UJ *-<< h-4 N-<
2: *-Z! < o CM 1—4 c£ + o 3 LU • t-< < 1 r— z: LUH-I— ctt u. In-
•—HO »— t a: IN o <— LU CL q: cy + s UO + UJ CO CO CO IL coco
o i—<>— h- o — LU I-IAD. a \- or LU|— r-< —
1
LUOI-— cC + + ai +
Cl LOCO 2 o -J W o CO *•£ a
co s + <x cÔcZ +
• >- <—o > <-«
II II LU in LU h- II 1— % >- 2 o ^-.a 3S CL o^-iclo: a: h-mo_ a: t-m
>- a: or a: ii z> >—i >- CO*— a: t— ~; 2! CL
y CL i^ 2Cl < co »-q: < co •>
s: LULU LU a LL CO cc •-I-C0 < Z! ai o z:

II ii a S^N-.->.
ii v-<-;:- Q — li
LU ou Li- LU
a
II < Ot-i— SL _l D |— X _ji~i—
t-H II ii Z! z ^«-
Z! Z1ZT lt. LU CD s LUcoa. ^—^_J — —*3 "g H- -i ala:
t
a LL'H-CO o LLH-
LU << •— a.
— ZT CL *—< o*- k- a: < LLClCl
hZZ (.O <OClCl o 1— o(N-l>-» 1— >— 1
DO c CO < a: a: •-h^^qc: CL o 1—4 1- uozz LU —i'y)LU UJ t-H CO

LU T-t Q. r-(--H O O CO r-t^H CO f-H
X Q LU LU LU LU ^-1
k- LU o LU a Q o UJ LU
LU z: < CC o
OO Q. < < Qi CO o r-i < <
—4 CO cc s: O |p-4
o s: s:
104
i 
Q ^- »— O o
*»• •— LU C\J u ro 1-4
o O
< Oh O
h-Zifl LU >-
uu o Q o -J o in
o hO O0 r-l r-H o K o
_J 00 UJ •—I >- z o
U- mD »•
O y o LU o r-i
# o UJ H LU K h- o o
—» UJ>- o Z —
C£ • ooo»-* HO
Zh
&
w o_j LU O LU o u_ -» i-H • «
c 00 O K o h- C0#- _(.-l^h- O ~
LL >Z »— >- CC
—
>- c£ #i~tt -:;-* z ooo
00 SIUJ #• c^ «-< or LU c- o; u_ ~Zt-l— h-O iÔ *
a: DUt-t i— < z \- r~l h- —! < «—
z cooaâro^ "N. rH
< za: o w- a w o • • o 00 "N.OO.LUU. • o-»-«-
h- LULL. C£ LU Or-I 2 1— .— LU C?--i z hi »-H l-« o h^ZZZUJ Oc£ •
oo LU CC LU o t— ^ + a: 4- 1 LU + o o+ ~ z zoor-i .a
ii
cz
ii ii ii
+ >->- O a: H o tuoh-H • r-H cr: • h- rH o LLOh-H h- 0—» —»—»— i i • •«

O C£C£ < o LU2 + LU c£Oa: + o U_3'+ LU oôar: + O0 Ol— t— 1— 1— 1— corjKi —
CL << CC o OLU2 oO Oir-iiua: 0J OLL s oo o.-<u_ct: M ^zzzzz^-â: .^
a: DD o ^y LUZLU LU ^ Zuu LUZ LL ^ ZLL H -DDDDDo^ 3 a a: .r.
-:«ZZ 00 O Z z *r_ 00 II z LU O z < OOOOOQ Ci.Z0.UJ
h^^y^^z Z zz
II 1 II i
II II
.-ôo _J a ~4|- X Jh-I- o ii II i-*H- II X _il — h- II 1— II
w OO
oo _J ~i ~ •—•j£. s h- _J oia: i *--S 3 1- _J <XCC oO j—— it
o^h
ii
LULU <c 00 LLLU LU <lOlulu oo U_U_ LL <OU_U_ i— h-u. ccy^cû-cc

IL. 0000 o H— >~iz 2 h- ooz; t— —«<i- z
•
H oozz \- Q.UJU-1—iCLOuCL^-tLU
i^l—
rH o O o O o
UJU-I LU LU cm LU «* LU ro LU o
DD Q ar o o o Qi o .J o
<< a: o o c* o .-I o
OCct 00 o fH 00 l—l o i-H o rH
oo >— o h— o o
105
I — 1
XXX
<•< <
Q o D :
-
.-
oc\ a:, a:
a:
U-
«* Cl
CL CL
LU X X x
Of
X
0.00
t/5 I/) (/•;
CL >- >- >-

t»
CC (_> o
CL t/) I/O. (/I
^ X X X
h- UJ UJ LU
CL
LL UJ- Ui LU
•*
1-
Ql UJ LU
•LU
»•
\~ >J _J _l
cc LU UJ UJ
Cl. CO CD CO
••
< < <
h- _J~_J 1 —
cC r.\— ».| — ».|
u. LULOUJ00UJLO
zf a < a_ < a. <
CO oo> >- —)>- i>-—i
I— •- ^ *. i— *i— »— ^
a: > — — —4. I •
co uj II II II MH 0*H-< r»>—4
o z UÛQ'UCC
o
-H
•* £0205:0
h-
aL — H— — »-| «-
O CL Q.1LUIU.I
H- z ClLULL •.o «~o cO
*—s—
-.^«Cô:h alcCa: Hiaia.1
oooonaz <<< «-— •»— 1 ».— e
• ZZDt
.-<.-|.-<^-V.Ô
. (/} »-00 o-OO «~
(/)</> 1/) I— UJI— LUI— UJ

# ooo^
-;:-
araio: •
-::-
« «
aoaocQ
Z'-'Z'-'Zm
CLQ-ClCi-I <-<--» *• 3 *»" ;?: ** 3;
Z 2: 2: uj -:<• -;:-
lh o o —<,-*.-< • •
.— — . -«-. »i— i— •• O
hOOhDvOsOÔO OOO
OLU *•»•«- OOO <t ••<[ •><!
^^^(^c^QTsO 1—1 oocoooq:xd:xc X
SQi2Q:UJU.w
LULLU.Q-ZZUJOOOOOOC XXXLUQ > > >-
0000
•
z—"-—
•
O^hhOO • • o<a<c <oooo
OOOOmUJUJLUU.'
II 11 11 11 11 11
h-l— H-l— (—CLOtOO-JCLl— «00«JOJQJ Chhl-I-

• . II II II II li
ZZZZ 1- 0:0:0: II II II II
h-OOOO I—
II II II II II II II II
KH Z
II 11
"HI-h IS 3 2 — ô3a: x >- >- a: -j 2: _j s: _; zo:3d:s

II
uj u. a.
II II
>-h
•»— 1—
SQi^LLO^t^QiCLUJLLa llod:2d:3 QoiDiOaaiaiQrj— uqx^x>- OQI2CX
1>- 1—
L
uj u_ 3 z z 2: z z: ZOLULLjLLU. OHilL(JU5S3:w2:zUJ>"-i*-i SS-hh^h-iOXOXO xzzzz
-
II II II II
•< .-CLClLLiUJ
1 1 II »-«
<
1- in CO
O o o
o o
106
Q < 1•
o
LL
«•
X
o
e
i—l
U_
X
O CO »- —
• • — CO — v^
r<-i <\jco «co *-
LL •- »C\J • CO —
• U_C\|rHC\|CO «-=.
X CO»-«LLrH •(X!CO —
I"-
»•
«-U- N U_ CM rH • CO
m Qs ~COr-llLC\JCO •
—
O •• *-» *LLCr>'— »(M *
• >»- .... CJn »-LL<\J.-H

CO 0:..0«. c~ COr-ILL
LL <>-Z>-- •• »-LLO^
•• s:s;ujq: ..a- o^ •>
x h-iujv-»<:>-^ .. ».-
co— oczqisisiilo-- •-
•-<i- O. UJ LL --< "Q
UJ -< c£
©*-< — c^zTiio-z:
• .- in — Oi C^ Ct: CL ll s: z uj
LL!
(MX COOOO *-4LU^
LL--I CO thf-htDOOofSo:
•• u_ rvj
•- 0002::
zrz: a. LULL
Xvf — »»r-4;DZ)IZ>>—"—'»— •
o —1 lc, lt\ ll a: a: a: h
h- 1— _j —i —
-h •- » »crv!— 1— H- oi c^ a: <r <
••xcrô «-oooooo<<r<i;iz::z
OHU-lLvOZZZhhHMMw
»- «. e.t—^ 1— (—*»—( OO CO 00 LL ILLL
vti-<LnLn^-
1
»..«..*»
r»»*0He»»»»-e*o»^^
LL *— • •
„„«....«..
*
„ ,.|*-f*-«,
XXLLLL .-t O
»-•--»•-»-
LUvOs0fv-r-«'
) «M* «"-* *tt=* »*—* "tat** \as* *•** «*^ Na^ ^X* w* «. • *** <W*
OOZhl-l-l--l-hl-t-l-h-h-h-hl-
II || »-.<<<<<<<<<<<<<<
h-H-f-s:5:s:5::2:s:^:s:2:s:s:s:s:?:Q.
a;3:2:Q:ct:(VQ:(2:cLa:a:c£:ct:ct:c^nr-n;oQ
ILU-OOOOOOOOaGOOOOOl-Z
Z 2 O LL LL LL LL LL LL LL LL LL LL LL LL LL LL 00 LU
OrHCMfO^LOCOo>Or-H(\JcO>tLriNO
i—4 i—4t-H^-4r-<rHi—1<—
o
107
».• i
1 — • < ! ((
. I D
CO
or
oo
a:
o
o
LUO
(/)LU
t»LU
^00
s !-!
^» •
Z—
UJO^
a*-
UlL
LUar
too
•>:*:
^-*» •* r-i c\l
CO—
W.Q-.
O O
i— rH
Ô —
OÛUJ O O
*—t-*ai h- 1—
oa:a z
UJQ.^ CC O a
o ^ ID o o
»—i^..—-» \-
roco UJ — —
t—I o.<m ai «— — <•
CC^cf. -J -j
D-sOO .—« w »—
k-.*-.^: *^ *— o o
s: r-» —,—» O o u
— uj — - •• -3«— o *—* 1~H
wÔ <
c- 1 — >-~3 .— CC a:
ID »«"- H- <_» a.-' a. a.
ar—u. o s H-a a •
1— sQoO Uj LU aro h- UJ Of
oo «-of: CC 5T <£•— O UJ
Z4-< a: • co h-a: O • *
i—«>OI— o c? »-oo a -» o *••
o UJ 1— •
>=-<00 -> «" j
0«
1
UJH oo
•• o c *-^ «-» O
Zm^
II
z 00 -J >• o o_ OCL • . O
>-<oocr> a — oo l-HOh t-tr-4 • (\jOfOUJ
H- <- o o— O »~! 0^ --<a: II r-« II II <-» II 3
DZUJH 00 UJOOZZ UJ-«, <~. < — ^ZZ
OOoo Q OOO *~)0fr-H hODl- oOohq;
OO — l-oO
1! II
aiSTor:.- O H~ M >— z: x»»» H- « r-H 1 *-H — H-— 1—

CQS<Qi UJ — a: oo oo o wa; LU —a: -J »-> a: 0:2:1-0
ZDCh-Q. Q u-CuO — U-OO O LLOOLL _J OOOOLUZ
ulO^>-i t-ii—iDX> oo wÔ a: mÔm < ii ii
ÔiiOOiUJ
UJ t-4 < s:
X O _i o 00 f-l C\IO
h- Z o o OO
o o •— .— o MO
00 or o O r-»
—I 2 h- J-
108
-* i < < ^
t ) » 5.i 4 • » »
CD
Cl.
\-
<
00
a>
U-
o
o
UJD
00 UJ
•"IJLI
-»oo
Q\\— i
*~* •>•
—Z —
iÛJO^
LLC)--
wOLL
•—LuntT
ooo
*r *^ ro IA
UJ—. o o
k-«co—» »-( r-A
t—IWCTN
•-(D"— O O
U_t_VJJ h- h-
Oh-q^: o
uuaro o O a
OOCL^
•—*»••-
•—I o o
.
z.
^—>,— o —> .— cC
UJ(V)00 H **^ •^h» Z>
o — -3 —
w
(-
lu vtai o *^ UJ
woo
m-^ a *B»^«^»»
z:
UJ
z:
LU
a:
~*.*~-
••^ » — «-*»-^ rH O a — —.— OJ
»—iLL)—. *-. ->-) O o o *»* ~)-^ o
-'src^ c^ (\j *i^- w,- I— UJ UJ O- co «—•»-' r-»
a: .^ l- «-» luz 00 00 \~ *- LL DC
UJ—«u_ o f— 00 UJ o • • o N-l OOIL o
OsOoO UJ —» CÔ H UJ o LU «^*-
o:0 t~
< £«: c£ 5: -oo UJr-JOUJ r\!om

z: LU —} >o
II
» •
*-<00C^ o • HrH • o 00 O00 O UJ o • r-«r~« •

Qi — .-ta:
II II
-.^^3 M ^Ci-H>»«rf
11 || II
f_» wr-Hr-H 1— iÔC 0""<-« O'—

3z:uj n ii oo UJ OZZ || UJ~> <~5 <
1— — Oh-
"72:
~">00
— MU.OOO
OZZ uj->
It. II
OOoOLULL H*4 wlilOOO «— *0'~»*-H

•— »*^
o:scioo o —'O t— I— If-H
—
XLUK oOUJf—oO LUh-LUI~ o M O »- 2 XLL. M-l 1—*
w -j QL CCZ. LU — u_ OOOO O —ac
1 1
CD2T<IUJLU UJ <— UJ 0000 O *—ct: UJ >— CC 1
U-COu. _j oooo a U-ooO

IJOH-OOOO
H Q u-ooo u_oo O U-O
1— M^
l/)L)^l*-' Mm QX> ii
CO M-SÔ ex: mÔm < *:oô t—lM-|QX>- 00

II
*-<bi:
-< UJ < s; LU
X O _j M-l m 00 Lao X o o
\- z: o o oo K o 2!
o o rH i— O HO O r-t
00 a: a O i— 00 K a:
*—* JS t- \~ 3
M-«
109
« l
*fr vO
o O
r-< r-*
o O
h- H
o O
CD O
or cc
u_ u_
o o
u o
UJ LU
00 00
• •
LU O
O UJ
o U-r-lOLL room
o 00 Ooo
II II O
—»r-4-^3
II UJ
r-l DC— ho:
<~3 < -5 ~52!Z:
o h-«- Ol— v—O—'t—<o£
h- 00 u_ l—oo U- H- U_ ID H
UJ —a: '*-' _! c£ c£2!H-0
o U.OOIL _1 OODOUJZ
O a:
o MÔ w < ÔÔQLUJ
< SL
>J (M 00 v0-l sfr
O o OO
o rH O HO i—4
a o i—
110
o
^ . «I
< . o 1 »<< o
t 1
i t 1 »1 i
• » <
00
0L
Cf
<
»—
00
o •»»
o
UJQ
—
~>
O0LU w
•>LU D-
— 1/} ^-
Oh
»—» •»
<*
<•
Z— H-
luo 00
O— «
•»«
uu.
luo: -3
wo ««^
-XL a
*•<• r- o t
CO—^ »—
— C£
o*-. Q.
(_>UJ —
hho: LU
<*o o
— — ro —
Q-^ lA
*. v« O > O o
rvl M
—.—. LU
M
H M t-H M
roco OO -z. + 1
*->*u^ a: i~i «k «-» o o o O

o >ta: \- _l •"^ —> h- • h- •
oo h-
o
oC _l
<o o
-* •«— r-l f-4
«— »-» LU Cl O «H^ O •_•
^•s: < »• ISl (X oo .—I h- o If o *
XLU — LL M X o •» o£ —. <-•
wÔ
—
•• LU —•I— CM < »-» "0 -» -5
w
1— 1 < .
CC Q o O \~ o »— o
O— LU- 111 LU 3 •o t— oo c CL • CL
LU vOoO »— LU— O Ol- •• •« LU t- LU h-
ai *-cf LL ooX X LU r- O _J cc _J a:
o<J-<r *— •-«# CO 00 •a o > • < <
^sDI— O *-i— »- < — r-t r- — 1- — h-
*-00 a 51 «r-l I— —> is ~> oo "0 oo
UJI— ••
2!^-^
2C o+
QO-3
II z «***.«** HO
+ O
1
o~
II
—.oa.
IIo~ II
LU O-r-H --•oa- --«

-I00O LU z-i a h- •--.i-H OOK "50I- ~3LU
—3
1- o «- <«— X> Qio-5« -HOC -HQl
DZUJ 2 q:v.o h- <LU' a. < a. < a.zz
OOoo LU MO 00 1— • cCO z. I-OH KOt- H-ôc:
QiSTC^ i— _JrvJ.-l ooÔ! o DiH-00 CCI-O0 q:i-Z)
cqzk C£ -J O 00 —~-~. ^ — < w < w- <^l— d
<MO
II
Z30H- LLI •— U-U- o 00 KOLL LU kOU- h-oujz:

00c_>00 Q. ONDii X
II
1—<»—V hO
II
(/)Ow 00 MOh OOOOiLU

t-l X h- LU < LU
LU o O r-H LU ro CM o inovf-
00 z o o a: o O ID ooo
t- LU < r~i f— o i— rM O rHOH
LU o X z LU r-i
00
111
.
i < .f - t ' <
. « . t
CO
CC
<
1-
00
*•
_
&
*M» ««•
CC *L
U- **
o —»
o —>
—
ixiO
00 UJ UJ
wLU 00
^»00 cc
CT *—
w
1
»
<
t~
Z«— 00
UJO •»
o«- —
on. ~>
tunr _»
z
^
OOO
~» »
UJ
00 —.
—o UJ
o«- 00
OU-l «-»
*-»CL UJ
QiO
D_^ 1— in — m ^-.
r* •* 0- > .
M rvj
*"***"» UJ r-4 M i-t M
coco 00 2: t— +
i-*-*** or 1—1 •• — O O
1
vkx:o K- _J <-, -5 h- i- »
>oo K CC _J vt *— r~) 1—1
«^- O --. 111 <co LU O — O w
-.21 <r tvj
M
O- 00 1—1 OO O O
Ji- «-
XUi— UL X • or ^^» ^»
wSIo •> UJ —»rH N <t — ~3 <— -J
KH •>•«—' Oi Q h- <— O •«—
f-i-Û- UJ LU 2 • t-l 00 * LU « LU
LUvOW >— UJ~^ a Oh V* tt LU O0 LU OO
a: -a: U- oox X LU i-4 O -1 CC _J a:
o*<
*:*oh- a
<~< >-<?<
t—
v*. !
a>
«-
00 O
iO
~o H i—)
t •
.—
<
J_
•
~+
<
f-
«~oo o SI «f-t H- -J .—1 r. ~> 00 -5 00
s o+ O—
1
UJt— »• II z: —*»-» -t-o II II O*- II
-z*-*~ oo-> UJ UJl-l —O —.Olu -^•OLU —

wi/)0> UJ Zr-* Q 00 • ~3<-l ->O00 ->ooo ™>LU
(- w o <w-0 z> CCQI ^-•w «--<ai —ho: -'D
Z)ZUJ z a:-N.O h- <UJUJ UJ < UJ < LU2TZ
OOoo LU MO 00 I— • QiO z OOOK 00Oh- u~)f-*cC
orxa: HH JMH O 00 oOÔl— oiH-00 CC\— 00 CL'h-Z>
C0SK cc -J **m* "*"* ^ < «-
1—1
< — <^ri— o
DOK UJ <MO —1
unc^
II
II U.UL 11
mO
00 h-OU.
WOm LU hOU. t-OUJ2!
00000 Cu X t—It-H 00 1/IOlH OOOOCUJ
r-H X h- UJ < UJ
UJ O — LU r0 fM Lno<f
00 z CC O z>
a HOH
000
\- UJ < rH rH O r-i r-t
UJ O X 2
—
LU 1—
oo a u cc
112
*i <
( o < < 1 1 «< Q
oo
<r
I—
00
•*
o
ccr ^
u. •«*
O *-»
o -5
IX.
1
Q w>
00 UJ u.
..LU OO
—.00 oi
O^t—
w <
r- H
Z~ oo
LUO *--
o— «».
uu. -)
WOT %•
ooO C£
».i(i U_
«•—% •. o
co-» o
UJ
«-c^
o—
<_>LU
O0
—
— iq: Uj
aro u —
Q.^ t— in ro *»^
•» •> o» > M M
*-^»-^ UJ O r~< M ^-i M
coco 00 z KM + 1
•»*— a: t—
—t* -~ O O \~
O O
<tal o h- _J "3
—»
h- 1 «
vOO ^- a: _J >t r-l t-l
— v^ o — UJ <xo o a. Q «— O «M
.-..s: < M D. <j>o 1—1 oo O # O ^
XLU-. u_ M X o »• a: *-» -—
—so Pt LU *-»!—»
o
C\J
o
< ~*
O
"0 .—. —>
i—< ».«-«• CC Q h- *— M-
M.-.LL- LU
—
UJ 2 •o t— O0 t U_ * u.
U_v0o0 UJ-~ a o>- »*
# UJ 00 LU 00
CtT ^CL' u_ OOX x UJ 1-1 O _l or. _J or
o<i-< t—t O
•-<?(• 00 •o o c c < c <
^>OI- a «H •• —>o l~* I— »» h- «-^ l~
.— O0 o s: «h H —> rH » "5 00 —3 00
o— O—
1
LU|— •• SL o+ II Z »M.-^ +o
ZH-, QOT
ZH
UJ ULfH -™«o
HH
II
ÔlL —
II
Oil
II
—»
MSO0CT> UJ Q 00 • -5O00 -5000 -?UJ
H- — O <— Z> occ? ^^.a** W_IC£ «—rM(V M-D
idzuj Z arv.o h- <LUU_ LL < U_ < U.ZZ
OOoO UJ MO 00 h- • aco 2 ooot— ooOt- O0»-<c^
oisia: 1— JNn< OOiÔj- o or.h-00 crr:i — 00 CtTh-Z5
COSK a: _l O O0 V,*? *.-* V M < — < » <2!l—
<NO
II
IDOh- UJ t—i a
LLU, 00 I-OIL UJ I-OIL l-OUJZ
X MM M(J)
II ii
tooô CL omo^: oOO>-« 00 ooo^- 1 OOOa: LU

r-l X H- UJ < UJ
UJ CDO i-H UJ m OvJ O ino4-
00 z O O cf O Z) 000
H- UJ < rH .H r-f rH Q MOH
uj a X 2: UU .-)
00 a o l-M ai
113
^ 1 (< ,
i ( H
CO
CL
I-
u_
o
o
llO
00 UJ
»-UJ
«-»oo —
CT*!—
^-*
o
t» rn
Z-. *•
LUC> o
o«~ m
OUL i
luo: •«^
00O _j
-^ <
*—* *• >
co— a:
wtfv U.'
O— K
2
(_>UJ
*—
•~<C£
o:o
a.*: 2
moo
i*i<f-a: LU
—
> —
<-«oo
—^_ *
O
—
i .-»
XLU.-. U- >
<-5ZCr UL «-
UJ •»-. UJ Q
o~u_ o UJ
t-Hv£)00o UJ
> »-a: O0
o-j-< o t—
ZvOh 2 w.
—00 t-H ^r ."«,
UJ(— o X
•• h-
z:»-<-^ o * cc
mc/XT" < z:>->™
1- — t~ <& v^.
ZD2TUJ 00 q:oz Z
OOoO o a: •
cCZlct < -JO — D
CQ5K _1 m 00 h- Q r-l
r>Oh- UJ < II II II LUZ
0OO0O 1— o>->
rH <
UJ
2:
UJ
o
114
t <
on
>
X *»
r-l
o o
»—
+
lij f-
UJ X <t&
V) 3 vO |
—1 muj
-. * ct: cor<-)
SL UJ CT> vJ-f-<
o CO ro r^-vO
o 2: iîO-^sO
ZL z> ITi fr-fin
< 2 •0--0<MO
aL *- t. + ^
5: QtAQ •
LU lu iu •;;-
2 O LU—.UJO
I— Z C/)Ol/}LU
H~ < t-HLUi—iLU
ZD a: ii lu 11 <s)z:
O oi/iomd:
or < LUt-HUJ II 3
co LU^-UJ_)h-D
r> UJ I/ILLOOLLUJ^
oo t— *H-• (—4 >~ C£ LU
<
on inô
UJ
2:
LU
O
115
LIST. OF/ REFERENCES
1. Newell, A.., Shaw, J.C.., and Simon, H., "Chess Playing Pro-
grams and the Problem of Complexity, " IBM Journal of Research
and Development Vol. 2, p. 320-335, October 1958.
,
2. Samuel, A.L. "Some Studies in Machine Learning Using the Game

,
of Checkers " IBM Tournal of Research and Development Vol. 3,

,
,
p. 211-229, July 1959.
3. Hunt, E., "Computer Simulation: Artificial Intelligence Studies

and Their Relevance to Psychology," Annual Review of Psychology ,
Vol. 19, Annual Reviews, 1968.
4. Ibid., p. 153
5. Feldman, J. "Simulation of Behavior in the Binary Choice

,
Experiment," Computers and Thought p. 329-346, McCraw-Hill,

,
1963.
6. Rigney, J.W., De Bow, C.H., "Multidimensional Scaling Analysis

of Decision Strategies in Threat Evaluation," Journal of Applied
Psycholo gy, Vol. 514, p. 305-310.
7. Shepard, R.N., "On Subjective Optimum Selection Among Multi-

attribute Alternates " Human Judgments and Optima lity p. 257-
,
,
281, Wiley, 1964.
8. Feigenbaum, E.A. and Feldman, J., Computers and Thought ,
p. 3, McGraw-Hill, 19 63.
9. Anderson, T.W., Introduction to Multivariate Statistical Analysis,

Section 6.7, Wiley, 1958.
10. Efroymsen, M.A., "Multiple Regression Analysis " Mathematica l

,
Methods for Digital Computers Section 5.17, Wiley, 1960.

,
11. Simon, H.A., The New Science of Management Decision, p. 25,

Harper & Row, 19 60.
12. IBM Manual C20-8011, Random Number Generation and Testing ,
1968.
116
The author believes the following references to be of interest
to the readerconcerned with additional computer applications in decision
making processes:
Rand Corporation Report R-772-PR, Decision in Battle: Breakpoint

Hypothesis and Engagement Termination Data by R. L. Helmbold,
,
June 1971.
Hunt, E..B.., andHorland, C. I. .,. "Programming a Model of Concept

Formulation," Proceedings of the Western joint Computer Con -
ference Vol. 19, p. 145-155, 1951.
,
117
INITIAL, DISTRIBUTION LIST_
No. Copies
L. Defense Documentation Center 2 .
Cameron Station
Alexandria, Virginia 22314
2. Library, Code 0212 2 .
Naval Postgraduate School

Monterey, California 93 940
3. Associate Professor R. S. Elster, Code 55Ea 1 .
Department of Operations Analysis and

Administrative Sciences
4. LT Kenneth Levin, USN 1
8647 Avers
Skokie, Illinois 60076
118
)
Security Classification
DOCUMENT CONTROL DATA - R & D

{Security classilication of title, body ot abstract and indexing annotation mo.si 6c entered when the overall report Is classitied)
I ORiGiNAriNC /tCTIVITY (Corporate author) ie. REPORT SECURITY CLASSIFICATION
Naval Postgraduate School UNCLASSIFIED

26: GROUP
J R EPOR T TITLE
The Evaluation of Air -to-Air Combat Situations by Navy Fighter Pilots with
Artificial Intelligence Applications
* DESCRIPTIVE NOTES (Type ot report and, inclusive dates)
Master's Thesis; June 1972

5 AUTHORI5I (f'ifsl name, middle initial, last name)
Kenneth Levin
REPOR T D A TE 7a. TOTAL NO. OF PAGES 76. NO. OF RE FS
June 1972 120 12

8a. CONTRACT OR GRANT NO. SO. ORIGINATOR'S REPORT NUMBER(S)
b. PROJEC T NO
96. OTHER REPOR T NO(S) (Any other numbers that may be assigned
this report)
10. DISTRIBUTION STATEMENT
Approved for public release; distribution unlimited
I I SUPPLEMENTARY NOTES 12. SPONSORING Ml LI TAR Y ACTIVITY

13. ABSTRAC T
The evaluations made by 3 6 Navy fighter pilots of 64 air-to-air combat situations

are statistically analyzed to detect any significant differences between pilots'
evaluative technigues in relation to their rank, flight hours and combat missions.
Predictor eguations are computed and used in a self-analyzing, self-modifying
artificial intelligence program modeled on an instructor-flight student interactive
situation.
DD ,
F
°r.,1473 (PAGE 1
119
S/N 0101 -807-681 1 Security Classification
A-31408
Security Classification
KEY HOBOS
Air-to -Air Combat

Artificial Intelligence
Evaluation Techniques
Decision Making
Naval Aviator
Fighter Pilot
Self Analyzing Programs
Self Modifying Programs
Flight Instruction
3 FORM
1 NOV 85 1473 (BACK)
120
010! -807-6821
Security Classification A- 3 1 40S
-
ft
2 3 9 17
Thesis 135118
L556 Levin
c.l The evaluation of air-
to-air combat situations
by Navy fighter pilots
with artificial intelli-
gence applications.
9 3 9 17
r-
ns
Thesis .135118
L55G Levin
c.l The evlauation of air-
to-air combat situations
by Navy fighter pilots
with artificial intelli-
gence applications.
thesL556
The evaluation of air-to-air combat situ
3 2768 001 03112 3

DUDLEY KNOX LIBRARY

Thesis

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Thesis

Uploaded by

Copyright:

Available Formats

THE EVALUATION OF AIR-TO-AIR

COMBAT SITUATIONS BY NAVY FIGHTER PILOTS

The Evaluation of Air -to-Air

Combat Situations by Navy Fighter Pj .lots

with Artificial Intelligence Applications

Th esis Advisor: R. S. E .ster

Approved ^on. puhLLc. /i££e&6e.; du>&vLbuXion ayuUniltzd.

Submitted in partial fulfillment of the

MASTER OF SCIENCE IN COMPUTER SCIENCE

NAVAL POSTGRADUATE SCHOOL

The evaluations mode by 36 Navy fighter pilots of 64 air-to-air

combat situations are statistically analyzed to detect any significant

differences between pilots' evaluative techniques in relation to their

and used in a self-analyzing, self -modifying artificial intelligence

program modeled on an instructor-flight student interactive situation.

II. EXPERIMENTAL PROCEDURE-DETERMINING DECISION 10

III. RESULTS OF DATA ANALYSIS 21

A. VARIABLES ADDED AND DELETED 21

IV. EXPERIMENTAL PROCEDURE -ARTIFICIAL INTELLIGENCE 34

V. RESULTS OF PROGRAM RUNS 48

A. AVERAGE PILOT AS STUDENT 48

B.. SYNTHETIC BEGINNER AS STUDENT 51

C. THE ARTIFICIAL INTELLIGENCE QUESTION 78

VII. CRITICAL EVALUATION OF THE STUDY 82

VIII. CONTINUATION OF THE STUDY 84

APPENDIX B PILOTS RESPONDING TO MAIL-OUT QUESTIONNAIRE 95

APPENDIX C PANEL OF AVIATORS 96

APPENDLX D DISCRIMINANT ANALYSIS FOR GROUPS 97

APPENDIX E SELECTED PREDICTOR EQUATION COEFFICIENTS 98

APPENDIX F MEANS AND STANDARD DEVIATIONS OF DECISIONS 99

APPENDLX G VARIABLE RANKINGS FOR HOUR AND MISSION GROUPS 100

APPENDIX H MULTIPLE CORRELATION OF PREDICTOR EQUATIONS 102

COMPUTER PROGRAM 103

LIST OF REFERENCES 116

INITIAL DISTRIBUTION LIST 118

L. DESCRIPTION OF RESPONDING FIGHTER PILOTS 18

II. CORRELATION OF BIOGRAPHICAL DATA WITH PRIMARY 23

ILT. CORRELATION OF SITUATIONAL VARIABLES WITH PRIMARY 24

V. VARIABLE MANIPULATION -AVERAGE VALUE PER SITUATION 2 7

VI. VARIABLE RANKINGS FOR ALL PILOTS AND RANK GROUPS 31

VII. CORRELATION OF PREDICTED VALUES FOR DECISIONS 32

VIII. RESULTS OF PROGRAM RUNS WITH TWO STUDENTS AND 47

1. TACTICAL SITUATION PRESENTATION 15

2. QUESTIONS ASKED WITH EACH DISPLAY 16

3. MACRO -LEVEL PILOT EDUCATION. 3 6

4-6. AVERAGE STUDENT -AVERAGE INSTRUCTOR 52-54

7-9. AVERAGE STUDENT-CDR INSTRUCTOR 55-57

10-12. AVERAGE STUDENT-4000 HOUR INSTRUCTOR 58-60

13-15. AVERAGE STUDENT-300 MISSION INSTRUCTOR 61-63

16-18. BEGINNING STUDENT-AVERAGE INSTRUCTOR 65-67

19-21. BEGINNING STUDENT-CDR INSTRUCTOR 68-70

22-24. BEGINNING STUDENT-4000 HOUR INSTRUCTOR 71-73

25-27. BEGINNING STUDENT-300 MISSION INSTRUCTOR 74-76

From the legendary Baron von Richtofen to the popular Snoopy-

pilot is one of an enigma capable of performing complex and dangerous

tasks in an environment where fractions of a second can spell the dif-

picture of a highly trained, assured and swaggering superbeing beckons

to be studied; to have the mystique lifted. Hence, the impetus of this

research: the fighter pilot.

Other than the satisfaction of human curiosity, the study of the

in improved training and selection of future pilots by understanding the

activities of successful pilots; improved combat techniques by adaptation

of the successful pilots' techniques; improved surface-to-air and air-to-

air communications through better understanding; and improved electronic