Smith 1

Adam Smith
17.31J, Fall 2004
Paper 2, Choice C (debunking Choice B as it relates to the Patriot case)

War with Words about War

“The proposal for national missile defense, as articulated now, is the wrong
answer to the right question. The right question is: what do we do about rogue
states like Saddam's Iraq?”

– Richard Butler, former chairman of the UN arms inspection effort in Iraq1

The debate surrounding the success or failure of the Patriot PAC-2 missile

in the Gulf War was explosive. Two camps, driven by different motivations,

heralded different interpretations of events. One, led by the Army and the Patriot

contractor Raytheon, claimed that the ballistic missile defense system enjoyed

near-perfect performance. These claims were motivated by efforts to protect

their budget, maintain public and ally confidence, and to make the case for more

expensive missile defense systems. The other, led by MIT professor Ted Postol,

asserted that none of the twenty-nine Patriot engagements were successful.

Postol’s camp was motivated by building a good professional reputation,

attracting attention, and gaining notoriety. Each of the two groups tried to prove

its point by establishing control over metrics and measurement, deeming

evidence as classified, and inviting third-party ad-hoc panels. This situation is a

classic example of Jasanoff’s constructivist school of thought; two scientific

adversaries attacked each other’s arguments using various tactics, allowing the

more objectively scientific side to ‘win.’

1
The Commonwealth, July 17/24 2000, p. 5
 Smith 2

This paper is divided into three sections. First, we will discuss the

motivations of each side in the debate as distinct from objective scientific truth.

With those motivations in mind, we will discuss the tactics that each side used in

the adversarial debate and how the exchange played out. Finally, we will make

observations about the results of the episode.

What is at Stake?

There were three distinct actors in the debate: the Army, the Postol team,

and the American Physical Society. Each of these entities had their own goals;

these goals ultimately drove the strategies and tactics chosen.

The Army2 primarily sought money that they could use to buy equipment

and support personnel. There are several factors which influenced its budget for

missile defense, most of which were positively effected by the original reports of

Patriot’s success. These factors and their causal relationships are illustrated in

Figure 1 below. There are two outputs of the chain illustrated: future investments

in strategic missile defense and the tactical missile defense budget. The latter

was of immediate importance because it promised immediate pay-offs. Indeed,

in 1991 the annual budget for tactical missile defense programs was $398M. By

fiscal 1992, shortly after the Gulf War, the budget was increased to $858M.3 On

the other hand, strategic missile defense was important because, although its

pay-offs were further in the future, the budget for such a program would reach

2
This includes the Army’s contractor Raytheon, as well as the defense department as a whole.
For simplicity, however, this group is referred to by their leader in this case – the Army.
3
Daniel Golden, “Missile Blower” Boston Globe Magazine, July 19, 1992.
 Smith 3

into the tens of billions of dollars.4 It would seem reasonable to look at the Patriot

case as an indicator of potential for success of such a strategic system, and thus

a tremendous set-back if the Army could not solve the much simpler problem of

building a successful Patriot.

Figure 1. Causes and effects determining the budgets for missile defense
programs. The reported success rate in the top-right was the most
accessible way for the Army to manipulate the budget in the Patriot case.

The input of the causal chain, i.e. the variable that is directly manipulable

by the Army, is the reported success rate. This was originally expressed in very

simple yet unclear terms. Raytheon claimed that the Patriot destroyed just under

90 percent of Scuds in Saudi Arabia and 50 percent in Israel. (It is not clear what

4
Congressional Budget Office, Budgetary and Technical Implications of the Administration's Plan
for National Missile Defense, April 2000, available at:
http://www.fas.org/spp/starwars/congress/2000_r/000425-cbo-nmd.htm
 Smith 4

exactly qualified as a successful destruction, a point that was later questioned.

We will discuss this in the next section.) These phenomenally high numbers had

positive effects in the causal chain of Figure 1. Public confidence was high, and

the Patriot became the symbol of American high technology success in the Gulf

War. The favorable public opinion and positive reports resulted in a higher

budget for the defense system, as mentioned before. Also, purchases from allies

were generated from the reported success; Saudi Arabia, Turkey, and Kuwait

ordered a total of 38 Patriots.

All of these things resulted in real income for the Raytheon, easing the

RTD&E costs for the Army. Thus the group was motivated to defend their claims

of success against the attacks that were later made.

Ted Postol and his collaborators found discrepancies in the Army’s case,

and after the debate began they were fighting for their professional reputations

and influential power. If Postol was proven wrong, his future whistle-blowing

power would have been significantly reduced as a result of a hit to his credibility.

Inversely, if proven correct, then he would gain notoriety and reputation.

Finally, the American Physical Society (APS) entered the debate upon

request from Postol, who asked the APS to weigh in on the legitimacy of his

evidence At issue for the APS was their credibility. If they wrote a report in favor

of either side without appropriate substantiation or evidence, they would be open

to attack by the party they deemed wrong. Thus, upon agreeing to consider the

issue, the APS panel had the responsibility to deliberate on the available

evidence and provide clear support for their position.

 Smith 5

Tactics in the Debate

If the objective truth has the largest impact on the outcome of a debate,

the tactics employed by the entities involved have the second largest impact.

This case is no different; both the Army and the Postol camp played a series of

cards against one another in an attempt to gain the upper hand in scientific and

political credibility. Three tactics were used: 1) both sides tried to strategically

define the metrics and measurement techniques of system success, 2) the Army

attempted to keep evidence hidden, and 3) to help his credibility, Postol invited a

third party group (the APS) to comment on scientific evaluations of missile

performance.

Defining Metrics and Measurements

Each group in this debate tried rigorously to control the terms of the

debate. In an oversimplified version of the situation at hand, if two adversaries

are drawing different conclusions from the same set of data, then the source of

any disparity between those conclusions must be due to the method of inference.

In this case, the method of inference has two components: the first is the

interpretation of the evidence, resulting in a set of deductions, and the second is

the tabulation of deductions into classes (e.g. ‘success’ or ‘fail’). For example, in

some Patriot engagements after the Patriot detonated near the Scud during its

flight, there was an explosion on the ground at about the time and place that the

Scud should have hit had it continued its trajectory. Postol interpreted this

explosion as evidence that the Scud was not destroyed, he thus tabulated the
 Smith 6

engagement as a ‘failure.’ The Army, however, deduced that it was possible for

the explosion to be caused by something else on the ground; the Army included

that particular case in the ‘success’ column in their tabulations.

There is no objective truth for what qualifies as ‘success’ versus ‘fail’; the

groups will always disagree on the definition of a ‘success.’ The next attempt to

arrive at an objective truth would involve classifying cases into more descriptive

buckets, instead of simply ‘success’ and ‘fail.’ Consider the following four

categories:

a. Extensive ground damage

b. Clear miss

c. Explosion on ground

d. No change in Scud trajectory or appearance after Patriot detonation

e. Scud intercepted

Roughly speaking, Postol claimed that events A through D in the list above

are ‘failures’ while the Army claimed that only event A constitutes a ‘failure.’5

Even in event B, the Army claimed, the engagement was successful because the

Scud’s target was not destroyed and the video evidence used to conclude ‘Clear

miss’ might not be reliable. Perhaps the decision-maker for whom the analysis is

being done should use their own standard for dividing the cases above into

successes and failures. In this case, the differences in the adversaries’

tabulation methods are marginalized out as a policy issue; the remaining

5
Need source
 Smith 7

scientific task is to interpret engagement evidence into deductions about which of

the five cases describes each particular engagement.

Even though we have done away with the problem of defining a metric for

success, measurement conflicts still remain. The Army gathered evidence

primarily by looking at ground explosions where a Scud landed and recovering

the Scud warheads. The warhead analysis was classified; we will discuss this

later. The ground analysis, however, is failure-positive only when the Scud hit a

target and exploded. All other cases are not ruled as failures, so are counted as

successes. For example, if the Scud was poorly maintained and came apart on

its own due to the physical forces on it during flight, then there would not be

ground damage and this means that the Army would have to tabulate the case as

a non-failure. It could be that the Scud fell apart on its own, or it could be that the

Patriot intercepted it; the extent of the evidence yielded by the Army’s

measurement techniques (ground explosion analysis) cannot distinguish these

cases.

In order to debunk the Army’s case, Postol had to rely on more

information-rich evidence sources. The group noted that reporters in the area

recorded video footage of several of the engagements, albeit with un-consistent

camera settings and filming locations. After obtaining as much footage as

possible, Postol meticulously reviewed the materials to identify which

engagement each video was of, the location of the camera, etc. Postol then

analyzed the dynamics of each engagement, and discovered that more evidence

could be found by watching the flight of the Scud with respect to the fireball
 Smith 8

created when the Patriot exploded. For example, with reference to the case in

which the Scud falls apart in mid-air because of its poor condition, Postol claimed

that the videos could distinguish that case from a clear interception case.

So, the Army used a less information-rich but more reliable measurement

method to show that Scuds often failed, and Postol used a more telling source to

posit that Patriots often did not cause the Scuds to fail.

The Army, as might be anticipated, attacked Postol’s evidence sources. It

claimed that the video was not “fast” enough to offer information about the Scud-

Patriot dynamics that occurred at velocities faster than the speed of sound. We

will discuss Postol’s response to this attack in the section below on third party

panels, and will see how the American Physical Society corroborates Postol’s

methods.

Finally, even if absolute resolution is achieved in the metric and

measurement matters, these evaluations are on the engagement-level. The

reader might ask: what percentage of engagements should be successful to

make the weapon system successful? It is fairly easy to see that this is a policy

issue. The only angle that the Army has on this is to design the system to send

several Patriots after each incoming missile; i.e. have more than a one-to-one

interceptor-to-missile ratio. This increases the probability of an engagement

success without additional engineering, but raises the cost of the system. This

did not become an issue until later in the debate because the two camps were

polarized. Postol said that the threshold did not matter because none of the

engagements was successful at all. The Army said the opposite; that the
 Smith 9

engagement success rate was so high that any sane threshold would deem the

entire system to be worthwhile.

Secrecy

The Army classified some of its evidence that it used in measuring

success. Based on publicly available sources, it seems that the classified

information concerns the Scud warheads recovered from the impact site.6 More

specifically, the Army inferred that the Patriots were successful from some

warheads which had marks that appeared to be from an exploding Patriot.

There was a legitimate need to maintain secrecy around some of the

information gained from the Scud impact site analysis. In general, the

information is principally held secret so that our enemies do not learn from it how

to defeat or weaken our weapon systems.

This is a double-edged sword, however, because it also hides information

from the public which might reveal performance under what is publicly stated.

The Army has an inherent motivation to report positive performance scores, as

outlined previously in this paper; therefore, there is a potential for abuse of

classification of information for the purposes of avoid public scrutiny.

There are two types, then, of analysis that can be done. Classified

analysis is performed based on both unclassified and classified information, the

totality of evidence available. Because the use of classified information is

necessary, the analyzers must have clearance and access. Thus, the danger

6
Ref. Postol’s paper in response to SKZ
 Smith 10

with classified analysis is that the entity doing the analysis might be biased, and

there can be far fewer checks against a possible bias.

Unclassified analysis is based strictly on unclassified information. There

are two dangers here. The first danger is that the unclassified analysis might be

flawed if the information it is based upon is missing a critical piece of the puzzle.

The second danger is that it can be trumped by classified analysis; e.g. the Army

could rebuttal against Postol’s unclassified analysis by saying that it has some

classified evidence that proves its claims. Once again, there is no natural

marketplace of ideas check on the classified analysis.

In this case, the GAO intervened. As a body that does not have the

biased motivations that the Army has (see previous section), it conducted its own

analysis of the classified evidence claims. The result of the study was essentially

that the Army was wrong. Although it could not prove foul-play, it showed that the

warhead-based inferences were incorrect since the same markings appeared on

Scuds which were not attacked by Patriot missiles.

Third-Party Reviews

Once the Army’s classified evidence was dispelled as unable to support

the Army’s claims, their primary argument against Postol was that the video

evidence is unreliable.

In response, Postol asked the American Physical Society to commission

an ad-hoc panel to evaluate the efficacy of videotape evidence. The APS

agreed, and many years later the report was released. The report backed
 Smith 11

Postol’s claims and offered detailed and technical support for its position. The

Army issued a forty-four page response paper repeating its arguments against

the video analysis and highlighting ‘errors’ in the APS report.7 In response to the

Army’s response, Postol and Lewis wrote a letter in Science & Global Security

responding point-by-point to the Army’s attack on the APS report. It does not

appear that the Army responded with a second round reply.

Was the APS report objective and correct? The APS has a good

reputation, and there does not appear to be any adverse selection in the panel.

Furthermore, the report details scientific motivations for each of its claims.

Finally, the Army did not attack the legitimacy of the panel itself, although it did

attack some of the claims in the report. Those attacks were counter-argued in

the Postol report. There is sufficient information in this series of transcripts to

build a reasonable scientific argument from the factual analysis done by both

parties.

Conclusion

The eight year exchange between the two sides of the debate seems to

have resulted in some nominal agreement. The Patriot was not nearly as

effective as originally claimed, and perhaps was not effective at all.

However, there are generalized, higher level lessons that are more

pressing. At the outset of the episode, Postol was invited to testify for the House

Armed Services Committee about the Patriot. This kind of check was a positive

thing for the issue; additional entities evaluating the case will not hurt truth in the
7
Cite SKZ report
 Smith 12

end. However, the parties were not motivated purely by truth. As shown in the

first section of this paper, each had their own motives which were un-

coincidentally different. They also happened to take different stances. The rest

plays out under the adversarial framework; each party tried to use factual

information to support its claims since factual information is the most convincing,

even though their goals might not have been pure truth. Since scientific and

factual arguments win out, in the end there should be a convergence to reality.

We also saw how other incremental minds came into the debate. The GAO and

APS offered analyses of different pieces of the arguments, and allowed a side to

gain credibility where it was factually earned, and where the party on its own

might not have been able to make the claim because of its ulterior motives.