Evaluating automated
manufacturing technologies:
Part I I - a methodology for
evaluation
SABAH U RANDHAWA and TOM M WEST
Abstract: In Part I of this paper we provided a review of the
techniques used in the evaluation of automated manufacturing
technologies. In this paper we provide a framework to integrate
the economic, technical, strategic, and qualitative factors in the
investment decision. The framework is based on combining a
simulation model with a multi-attribute decision model. The
simulation model is used to estimate the operational and
economic attributes for the investment problem. The multi-
attribute decision model is the linear additive scoring model
because of its ease of use and its ability to structure
multi-dimensional problems.
Keywords: automated manufacturing, implementation, justifi-
cation, simulation, multiattribute evaluation, scoring models
n Part I of this paper we developed a classification
I scheme for CIM evaluation techniques and briefly
reviewed some of these techniques. It was pointed
out that evaluating investments in automated technolo-
gies is a complex decision due to the large number of
attributes and their complex interactions. The task is
further complicated by the necessity of including the
non-financial and qualitative aspects of manufacturing
performance. Due to these factors, no single modelling
technique is adequate for analysing all aspects of the
manufacturing evaluation. This paper describes a
methodology based on integrating some of the tech-
niques discussed in Part I. The two techniques
primarily utilized are simulation and scoring models.
The next section provides a brief overview of the
justification methodology. This is followed by a
detailed description of each component.
Department of Industrial and ManufacturingEngineering, Oregon
State University,Corvallis, Oregon97331, USA
0951-5240/92/040276-07 (~ 1992 Butterworth-Heinemann Ltd
276
Justification methodology
An overview of the proposed methodology is shown in
Figure I. The primary input classes are: a description of
the system to be analysed; operational requirements
and environmental restrictions; and management pre-
ference structure reflected in the selection of attributes,
assessment of qualitative attributes, and in assessing
the relative importance of attributes in the decision
environment.
The basic components of the system shown in Figure
1 include:
(1) Attribute selection procedure: the decision environ-
ment is best described by a list of attributes or
characteristics. Therefore, the initial task in the
justification process is to develop a list of relevant
attributes for the situation being analysed. Some of
these attributes will be quantitative (both economic
and non-economic) in nature; others will be
qualitative.
(2) Simulation module: T h e simulation module is used
to obtain estimates for quantitative attributes, t h e
simulation module consists of three components:
(a) a resource library for describing the objects of
the physical system (equipment, piece parts,
etc.) and their interactions:
(b) a library for decision strategies that describe the
policies for equipment layout, scheduling,
inventory control, etc.; and
(c) a decision support library that contains models
for estimating modelling factors such as fore-
casts and economic parameters, and assisting in
the analysis of simulation output.
An integral part of the simulation analysis is an
integrated capability for sensitivity and risk analysis.
Computer-Integrated Manufacturing Systems

Quantitative attributes
~ Simulation module
(estimating L ,
quantitative attrioutes j
Qualitative
attributes
Strategic evaluation module
.~ Evaluate J
qualitative J
attributes J
_•
System description
Operational requirements Develop J - Ranked
Management preferences
Quantitative +
qualitative
attributes
attribute
weights
Figure 1. Methodology for evaluation
(3)Strategic evaluation module: this module uses
management's judgement and evaluation to
(a) develop estimates for qualitative attributes, and
(b) develop attribute weights, or relative import-
ance of the attributes in the decision environ-
ment.
(4) Multiattribute evaluation module: this module per-
forms two functions:
(a) computing the aggregate score for each manu-
facturing alternative consisting of total weigh-
ted attribute values for an alternative, and
(b) Performing sensitivity analyses on attribute
weights and attribute values.
The output from the system is a set of measures for
each manufacturing alternative. The measures reflect
the performance of an alternative with respect to the
defined attributes, and incorporate tradeoffs among
attribute weights and attribute values.
Attribute selection procedure
A manufacturing alternative is defined by a particular
collection of attribute values. The development of a list
of relevant attributes may be as simple or as extensive
as the analyst wishes. Often this activity is physically
limited by the time and resources available. It is always
desirable to thoroughly consider all the attributes in the
decision environment. However, the time and effort
required for such complex analyses are usually
prohibitive, and it is often possible to reduce the
dimensionality of the problem without a significant loss
of information and accuracy. A multistep iterative
procedure is usually used to identify relevant attributes.
The procedure starts by identifying an initial set of
attributes through literature search and questionnaires,
and developing operational definitions for the attri-
butes identified. Persons knowledgeable in the areas of
Vol 5 No 4 November 1992
S U RANDHAWA AND T M WEST
Qualitative attribute estimates
- economic ( NPV, IRR)
- non -economic ( utilization, lead time, throughput )
J
Qualitative Multiattribute versus simulation module
attribute levels
JCompute aggregate J
---~,cores for I
Jalternatives J
- Alternative
i evaluation
scores
Attribute
weights
r°
sensitivity
analysis
I ~-- ordered
alternatives
possible significant impact and familiar with the
operational and other technical specialties should be
asked to contribute ideas. The experience of these
experts should augment the literature review process,
and help to minimize any omissions and duplication
among attributes. Additionally, a simple model-
building approach, in which by building a model of the
manufacturing problem and identifying relevant input
and output variables, suitable attributes may be
identified, can be used in the attribute identification
process. A list of typical attributes for a CIM
application is shown in Figure 2.
There is no one best set of attributes. The attributes
chosen to serve as a basis of comparison are dependent
on the characteristics of the specific manufacturing
environment. However, it is generally desirable to keep
the attribute set as small and concise as possible.
Additionally, it is necessary to make every attribute
mutually exclusive of every other attribute. There are
primarily two reasons for keeping the attribute set to a
minimum. First, most of the available techniques for
combining the impact of attributes assume independ-
ence or near independence among attributes. This
implies that the attributes should be mutually exclusive
to avoid double counting of impacts. In practice, it is
often impossible to obtain an independent set of
attributes. However, independence is easier to justify
and approximate with a smaller attribute set than with a
larger set. Correlation among attributes may be
minimized by using statistical techniques such as factor
analysis. Second, a large number of attributes require
considerable effort in obtaining attribute values for
each alternative.
Additional attribute sets for analysing manufacturing
environments are given elsewhere 1-5. A procedure for
identifying attributes and minimiz!ng duplications
through factor analysis is given in Randhawa and
Bedworth 2.
277
Evaluating automated manufacturing technologies: Part H
Economic (quantifiable)
System acquisition and installation costs
Startup costs
Variable production costs (raw material, machine
operation, direct and indirect labour and utility costs)
Work-in-process inventory (and shortage) costs
Quality costs (inspection, manufacturing accuracy and
precision, scrap and rework, replacement)
Maintainability costs (system diagnostics, repair and
upkeep)
Operational (quantifiable)
Equipment utilization
Throughput rate
Production lead times
Product accuracy and precision
Shop floor information requirements
Space requirements and utilization
Qualitative (non-quantifiable)
Flexibility (design change accommodation, changes in
product mix or market responsiveness, routing and
scheduling flexibility, capacity growth)
Compatibility (with existing or new machines, tools,
computer control and communication)
Safety of operationsto avoid injuries and accidents
Training availability and quality
Worker morale (or acceptability and enthusiasm for use
of the system)
Technical and managementsupport
Figure 2. Attributes for CIM application
Simulation module
The objective of the simulation module is to estimate
operational and economic attributes for the investment
problem. Manufacturing systems are complex combin-
ations of computer control, communications, produc-
tion processes, and human workers. As such, the
operational (or technical) and economic components of
the system are integrated and interdependent upon
each other. Operational factors such as equipment
utilization, inventories, work in-process queuing, and
system throughput rates affect the cash flows (revenues
and costs) are used to estimate economic attributes (net
present value or internal rate of return). The oper-
ational factors, in turn, are affected by factors such as
equipment characteristics (e.g. speed and capacity),
equipment layout, and scheduling policies. For exam-
ple, component interaction in one type of layout and
scheduling priorities may result in low work in-process
inventories, but high equipment utilization; however,
changing either the layout or the scheduling priorities
may increase inventories and increase utilization.
Most evaluation approaches currently use single
parmeter economic analysis to select a manufacturing
alternative. Subsequently, the operational feasibility of
the system is established by analysing combinations of
system components. These economic analysis techni-
ques are usually insensitive to operational considera-
tions; they ignore the fact that cash flows are random
variables, the randomness resulting from the stochastic
278
nature of the operational and design variables. The
approach described below considers the economic
analysis and the operating environment as integrated
components, and evaluates the operational and the
economic viability of the system at the same time.
A number of operational factors need to be
considered in evaluating a manufacturing system.
These include different types of products with differing
production sequences; scheduling priorities; equipment
outlays; effect of machine breakdown and failures on
output; production schedules; and interaction among
system components based on schedules, breakdowns,
intermediate buffers, spatial interference, and other
such factors. These complex interactions and the
resulting economic and non-economic consequences
are very difficult to represent analytically. Simulation
provides a mechanism by which these interactions can
be represented and analysed.
The prerequisite for simulation analysis is an
accurate representation of the manufacturing process.
Simulation involves developing a model of the system
and then conducting computer-based experiments with
the model to describe, explain, and predict the
behaviour of the system. The system model generally
includes: logical components, as for example, defining
the material flow in the system; heuristic components,
as for example, defining scheduling policies on specific
machines to determine the order in which jobs are to be
processed on the machine; analytical components, as
for example, a cost estimation model to determine
revenues and operating costs; an economic model to
evaluate net present worth or internal rate of return,
growth or forecasting model for generating future
demand, and a quality evaluation model to analyse
product quality, scrap loss, etc. The simulation model
represents an integration of these three components.
Since alternative manufacturing processes may differ
significantly, simulation models designed for a specific
technology may not be readily adaptable for the
analysis of an alternative process. Developing a 'new'
model for each alternative to be evaluated is expensive,
in terms of both time and cost. This is also one of the
main reasons for relatively low acceptability of
simulation in evaluating manufacturing technologies.
The key is to develop a simulation modelling frame-
work that can be extended or modified to support the
analysis of multiple alternatives. It is not feasible to
develop a generic simulation model; instead, we
propose that the conceptual framework shown in
Figure 3 be used as a basis of developing simulation
models in analysing manufacturing alternatives.
The modelling framework contains three structures
for representing and analysing a manufacturing system.
Each of these structures is represented as a software
library in the simulation framework. The resource
library represents the physical objects (processing
stations, material handling equipment, and products in
various stages of manufacturing) in the system. The
layout for a manufacturing system is a combination of
objects from this library in a specific order.
Computer-Integrated Manufacturing Systems

The other two libraries represent conceptual con-
structs. The decision strategies library contains alterna-
tives for equipment layout, sequencing, scheduling,
and similar physical attributes. The decision support
(or model) library contains a set of analytical models to
be used in forecasting, economic evaluation, inventory
management, and analysing simulation output, and
constructs required for executing simulation such as
random number generators and event schedulers (for
generating and controlling the execution logic).
The framework of Figure 3 shows a modular
structure, independent of the implementation tech-
nique. Thus, manufacturing alternatives to be analysed
can be constructed by selecting components from the
resource library, and interfacing these components
using conceptual constructs from the decision strategies
and decision support libraries.
An advantage of using simulation is that sensitivity
and risk analyses are integral components of the
analysis, rather than representing 'post-optimality'
Resource Library
Workstations
Processing Stations
Machining
Drilling
Turning
Transfer Stations
Storage Queues (Buffers)
Material Handling
Robots
Conveyors
AGVs
Storage/Retrieval
AS/RS
Entities
Raw Material
Finished Products
Decision Strategies Library
Equipment Layout
Sequencing (or Routing) Policies
Scheduling Policies
Inventory Control Strategies
Decision Support Library
Forecasting Module
Economic Evaluator
Statistics Analyser
Random Number Generator
Event Scheduler
Figure 3. Framework for simulation modeling
Vol 5 No 4 November 1992
S U RANDHAWAAND T M WEST
analysis. In the majority of evaluation techniques which
consider parameter variability, the system is first
analysed using deterministic values. Sensitivity analyses
are then performed on the base model through equally
likely variations in attribute values to identify the more
critical attributes in the system. This is followed by risk
analysis or dermining the probability distributions
associated with outcomes of critical attributes. These
resulting distributions describe the variability associ-
ated with these attributes 6. Irrespective of the evalua-
tion technique, advanced sensitivity and risk analyses
generally require some form of simulation analysis.
In simulation analysis, the variability associated with
input parameters (costs, product demand, economic
indicators such as interest and inflation rates, and
machine performance parameters including capacities
and rates) are defined in terms of probability distri-
butions; the variance or the standard deviation of the
output measures then indicate the degree of uncer-
tainty of risk associated with predicting the final result.
Determining distributions for input parameters
depends on available information. If adequate histori-
cal data are available, then a standard distribution may
be fit to this data and verified using 'goodness-of-fit'
tests. If no standard distribution fits the observed data,
then the factor may be modelled using an empirical
distribution computed from existing or synthesized
data. If no system data is available, as is not uncommon
in evaluating new systems, a triangular distribution
with a minimum, maximum, and most likely value may
be used 7. The triangular distribution recognizes the
variability of data, and is easy to use as the parameters
may be estimated based on input from system experts.
In general, if an objective or subjective distribution
could be fit, this is preferable. Distributions allow
'extreme events' to be generated which fall outside the
range of observed data. Extreme events while rare
often may have a significant impact on the design of the
system.
In addition to sensitivity and risk analysis, certain
important characteristics such as flexibility associated
with advanced manufacturing technologies can be
evaluated using simulation. Different types of flexi-
bility may be associated with a manufacturing system8.
These include process flexibility, product flexibility,
volume flexibility, and expansion flexibility. Although
all forms of flexibility are hard to quantify, the effect on
economic and non-economic attributes can be incor-
porated in the simulation analysis by designing and
analysing different scenarios.
Strategic evaluation module
Measuring qualitative attributes
Effective representation of qualitative attributes re-
quires the approximate quantification of qualitative
attributable values. The most common quantification
method is to develop a cardinal scaling procedure
where the various applicable qualitative levels are
279
Evaluating automated manufacturing technologies: Part H
associated with a numeric scale. The choice of a 10- or Table 1. Attribute weights through palrwise compari-
lO0-point scale is usually logical, since most people son (importance scale 0 to 2; 0 = less important, 1 =
seemingly have an intuitive feel for such ranges. The same importance, 2 = more important)
most obviously identifiable values of an attribute are
the end points on the scale. For example, on a 0-10 Cost Compatibility Flexibility Sum Weight
scale, the 0 point represents the minimum attribute
value that is physically or practically attainable, while Cost 0 2 1 3 0.500
the maximum attribute value that is physically or Compatibility 0 0 1 1 0.167
Flexibility 1 1 0 2 0.333
practically attainable corresponds to the 10 point of the
scale.
To illustrate the scoring of qualitative attributes,
assume that manufacturing alternatives are to be complement of itself). This simplifies the assessment
evaluated on three attributes - quantitative attributes, procedure as only half the matrix values need to be
cost, and the qualitative attributes, compatibility and filled.
flexibility. Then, for compatibility, the 0 on the discrete This procedure may be further expanded through the
scale may correspond to 0% compatibility, and 10 may use of opinions from several 'judges' or managers.
correspond to 100% compatibility, where compatibility Concordance or logic tests may be performed on the
refers to integration of hardware and software compo- results and for large groups of input. Analysis of
nents of the system. The intermediate values on the Variance (ANOVA) may be performed to obtain
0-10 scale then represent different levels of compatibil- estimates of significant difference between attribute
ity, meaningful to the decision maker. weight estimates is.
The values on the discrete scale employ certain
mathematical relationships9. More importantly, it is Multiattribute evaluation m o d u l e
assumed that a scale value of 8.0 is twice as favourable
as 4.0. Additionally, the use of values across attributes Combining attributes
implies that the quantifiable difference between 'high' The next step in the analysis consists of computing an
and 'low' impacts for two different attributes is aggregate weighted value for each manufacturing
identical. alternative. This value is most easily obtained using the
Application of scaling techniques permit the quantifi- additive linear model that reduces the n-attribute
cation of qualitative variables but the numerical values problem to a single dimension using the following
for quantitative factors (e.g. cost) obtained using expression:
simulation analysis are usually not on a comparable
n
scale. The simplest procedure is to use a 0-10 scale with
the lower and upper values of the scale representing wj = w, j (1)
extreme but practically attainable values. Thus, for the i=1
attribute cost the 0 may represent a very high cost (an where Wj is the aggregate score for alternative j, wi is
FMS system, integrated and automated) while a 10 may the weight for attribute i, 3~jis the score of attribute i on
represent a very low cost value (manually controlled alternative j, and n is the total number of attributes in
job shop system). the model.
To use the above expression, the values for
quantifiable attributes (economic and noneconomic)
Attribute weights must be reduced to a comparable scale as that for
In any decision environment, not all attributes are qualitative attributes (see Table 2). The procedure is to
likely to be considered equally likely. The function of
weights is to express the importance of each attribute
relative to other attributes. Several techniques are Table 2. Computing aggregate scores of alternatives on
available for developing weights based upon judge- attributes based on 0 to 10 scale; 0 = unsatisfactory
mental or subjective opinions 1°'11. These include performance, 10 = superb performance)
ranking, rating, partial paired comparison 12, paired
comparison and Churchman-Ackoff-Arnoff
Alternative
method 13.
Attribute
The procedure proposed here for determining
Attribute weight A B C
attribute weights uses pairwise comparison of attri-
butes 14. Using the three attributes discussed in the
previous section, the comparison values in Table 1 Cost 0.500 6 5 9
show the importance of row attributes relative to the Compatibility 0.167 8 6 9
column attributes. Thus, cost is twice as important as Flexibility 0.333 9 7 3
compatibility. Also, the values below the principal Aggregate score on 7.33 5.83 7.00
diagonal are complement of the values above the production attributes
diagonal (0 is the complement of 2 and 1 is the

280 Computer-Integrated Manufacturing Systems


use the same scale and scaling relationships for both
types of attributes, as discussed in the section on
'measuring qualitative attributes'.
The additive weighting model can be a very powerful
tool in evaluating multi-attribute manufacturing
systems. However, the validity of the results obtained is
dependent on the development of realistically indepen-
dent attributes, the determination of comparable
attribute values, and the assignment of reasonable
importance weightings.
Sensitivity analysis
The two most important sensitivity analyses to look at
are sensitivity to changes in attribute weights and
single-attribute values. Of these two, perhaps the more
important is sensitivity to attribute weights. This is
because attribute weights are the essence of value
judgements and because weights, being purely subjec-
tive numbers about which decision makers disagree,
are more likely to be in dispute than other parameters.
A sensitivity analysis involving attribute weights
consists of investigating the sensitivity of the rankings
of the alternatives to small changes in attribute weights.
If the rankings remain unaffected as the weights are
changed by modest increments, this can be established
as evidence that small errors in the estimation of
attribute weights are not important. Weights can be
changed two at a time or three at a time, if desired. If
one or more of the weights prove to be very sensitive,
the analysis will indicate the accuracy of the estimating
weights and the stability in the ranking of alternatives.
The other sensitivity to look at is sensitivity to
single-attribute values. These values may represent
point estimates or probability distributions or may be
subjective assessments reflecting the judgement of
experts. Hence, in some cases these values may not be
known with precision, and in others may represent
values of intense controversy. A full sensitivity analysis
to attribute values requires a complete solution of the
decision problem for different values of each attribute
value. With a large number of attributes, this may
involve quite a few iterations, and the results may not
be easy to interpret. In actual practice, values
associated with attributes with high importance weights
are the most likely candidates for sensitivity analysis.
Since the weights of the associated attributes are high,
even slight changes in attribute values may result in a
change in the ranking of alternatives. Also, attribute
values which involve a high degree of uncertainty and
subjectivity in estimation should be varied in the
sensitivity analysis to investigate the effect of the
variation on the ranking of alternatives.
The other sensitivity to look at is sensitivity to
single-attribute values. These values may represent
point estimates or probability distributions or may be
subjective assessments reflecting the judgement of
experts. Hence, in some cases these values may not be
known with precision, and in others may represent
values of intense controversy. A full sensitivity analysis
Vol 5 No 4 November 1992
S U RANDHAWAANDT M WEST
to attribute values requires a complete solution of the
decision problem for different values of each attribute
value. With a large number of attributes, this may
involve quite a few iterations, and the results may not
be easy to interpret. In actual practice, values
associated with attributes with high importance weights
are the most likely candidates for sensitivity analysis.
Since the weights of the associated attributes are high,
even slight changes in attribute values may result in a
change in the ranking of alternatives. Also, attribute
values which involve a high degree of uncertainty and
subjectivity in estimation should be varied in the
sensitivity analysis to investigate the effect on the
variation on the ranking of alternatives.
As an alternative to using simulation to evaluate the
effect of variation of attribute values on alternative's
weighted score, analytic procedures can be used to
develop variability associated with the alternative
weighted value. The complete procedure is described
by West 9. To illustrate, consider the attribute cost
scaled on 0-10 scale to 3.0, 3.5, and 5.0 representing
three levels (low, most likely, and high), with relative
probabilities of 0.2, 0.7, and 0.10, respectively, for
three different scenarios (e.g. pessimistic, most likely,
and optimistic). Then the expected value of cost is 3.55
[(3.0)(0.2) + (3.5)(0.7) + (5.0)(0.10)], and the
expected variance is 0.272 [((3.0)2(0.2) + (3.5)2(0.7) +
(5.0)2(0.1)} - (3.55)2]. Once estimates of this type are
determined for each attribute, the mean aggregate
score for an alternative is determined using expression
(1) and the variance (vj) associated with this score is
determined by:
r/
V] = Z I;i W2 (2)
i=1
where vi is the variance for attribute i and wi is the
weight of attribute i.
The results give the decision maker an additional
insight as to the variability of the final values. Various
statistical methods can now be used for further analysis
of the choice being considered. Probably the most
important benefit obtained from this type of analysis is
tha ability to utilize estimates of the extreme values
which may result under actual conditions.
Conclusions
The paper presented an approach for identifying an
attribute set for a manufacturing environment, evaluat-
ing quantitative attributes, estimating qualitative fac-
tors, and developing aggregate alternative impact
scores. The methodology is based on the integration of
three modelling techniques - simulation, manage-
ment's judgement, and multi-attribute scoring model.
Our focus was to develop a methodology that integrates
economic, operational, and qualitative aspects of
manufacturing decision making, yet is easy to develop
and implement. Given the large number of attributes in
281
Evaluating automated manufacturing technologies: Part H
a manufacturing environment and the complex interac-
tions among those attributes, simulation is a cost
effective tool for analysing quantitative aspects of such
systems. A strategic component uses management's
subjective jdugements in estimating qualitative attri-
bute levels, and in estimating attribute weights. Finally,
the individual attribute values and weights are synthe-
sized using a linear additive model.
References
1 Arbel, A and Seidman, A 'Performance evaluation
of flexible manufacturing systems', IEEE Trans.
Syst. Man & Cybern., Vol 14 No 4 (1984) pp
606-617
2 Randhawa, S U and Bedworth, D D 'Methodology
identifies factors for comparing FMS with conven-
tional manufacturing systems', Ind. Eng., Vol 17
No 6 (1985) pp 40-45
3 Varney, M W, Sullivan, W G and Cochran, J F
'Justification of flexible manufacturing systems with
the analytic hierarchy process', Proc. Ann. Int. Ind.
Eng. Conf., Los Angeles, CA (May 19-23 1985) pp
181-190
4 Smith, M L, Ramesh, R, Dudek, R A and Blair, E L
'Characteristics of US flexible manufacturing
systems - a survey', in K E Stecke and R Suri (ells),
Proc. 2rid ORSA/TIMS Conf. on FMS: Operat.
Res. Models & Applic., Elsevier, Amsterdam
(1986) pp 477-486
5 Falkner, C H and Benhajla, S 'Multiattribute
decision models in the justification of CIM systems',
Eng. Economist, Vol 35 No 2 (1990) pp 91-114
282
6 Randhawa, S U and West, T M 'Uncertainty
modeling in CIM investment analysis', CIM Rev.,
Vol 6 No 1 (1989) pp 32-36
7 Law, A M 'How to choose input probability
distributions for simulation', Simon Sez, Vol 8
(Summer 1988)
8 Browne, J, Dubois, D, Rathmill, K, Sethi, S P and
Stecke, K E 'Classification of flexible manufacturing
systems', The FMS Mag. (April 1980) pp 114-117
9 West, T M 'An integrated approach to the
evaluation of multiattributed alternatives in facility
siting studies', PhD Thesis, Department of Indust-
rial Engineering, Oregon State University (1977)
10 Bartlett, C V, Heel'man, E and Rettig, S 'Compari-
son of six different scaling techniques', J. Social
Psychol. Vol 51 (1960) 343-348
11 Eckenrode R T 'Weighting multiple criteria',
Manage. Sci. Vol 12 No. 3 (1965) pp 180-192
12 Buel, W D 'A simplification of Hay's method of
recording paired comparisons', J. Appl. Psychol.,
Vol 44 (1960) pp 347-349
13 Churchman, C W, Ackoff, R L and Aenoff, E L
Introduction to Operations Research, Wiley, New
York (1957)
14 West, T M and Randhawa, S U 'Multicriteria
evaluation of manufacturing systems', in B Prasad
(ed.) CAD~CAM, Robotics and Factories of the
Future: Proc. 3rd Int. Conf. Springer-Verlag, Berlin
(1989) pp 271-275
15 Dunn-Rankin, P 'Multiple comparisons in a simpli-
fied rank method of scaling', Educ. & Psychol.
Measure., Vol 29 (1969) pp 315-329
Computer-Integrated Manufacturing Systems