Computers & Operations Research 33 (2006) 2083 2098

www.elsevier.com/locate/cor

Multicriteria decision analysis with minimum information: combining DEA with MAVT
George Mavrotas , Panagiotis Trillis
Laboratory of Industrial & Energy Economics, School of Chemical Engineering, National Technical University of Athens, Zografou Campus, Athens 15780, Greece Available online 1 January 2005

Abstract In this paper we use some basic principles from data envelopment analysis (DEA) in order to extract the necessary information for solving a multicriteria decision analysis (MCDA) problem. The proposed method (enhanced alternative cross-evaluation, ACE+) is appropriate when either the decision maker is unwilling (or hardly available) to provide information, or there are several decision makers, each one supporting his/her own option. It is similar to the AXE method of Doyle, but it goes one step further: each alternative uses its most favourable weights (as in AXE) and its most favourable value functions in order to perform a self evaluation, according to multi attribute value theory (MAVT). These self-evaluations are averaged in order to derive the overall peer-evaluation for each alternative. The minimum information required from the decision maker is to dene the weight interval for each criterion. Beside the peer evaluation and the nal rating of the alternatives, the method provides useful conclusions for the sensitivity analysis of the results. 2004 Elsevier Ltd. All rights reserved.
Keywords: Multiple criteria; DEA; Cross-efciency; Value functions

1. Introduction In multiple criteria decision analysis (MCDA) problems a number of alternatives have to be evaluated and compared using several criteria. The aim of MCDA is to provide support to the decision maker (DM) in the process of making the choice between alternatives. In other words, MCDA techniques help the DM to articulate his/her preferences in a complex decision making environment. A pre-requisite in
Corresponding author. Tel.: +30 210 7723283; fax: +30 210 7723155.

E-mail address: mavrotas@chemeng.ntua.gr (G. Mavrotas). 0305-0548/$ - see front matter doi:10.1016/j.cor.2004.11.023 2004 Elsevier Ltd. All rights reserved.

2084

G. Mavrotas, P. Trillis / Computers & Operations Research 33 (2006) 2083 2098

most MCDA methods is that the DM is able to provide the necessary information. However, in several situations the DM is hardly available or unwilling to express his/her preferences or, furthermore, there is not an unequivocal DM but a group of stakeholders that each one supports his/her own alternative. In these situations the available information in order to perform the multicriteria analysis is limited or ambiguous. In order to ll the information gap, or to reach a consensus in the latter case, information that emerges from the data set itself can be utilized. In the present work, the suggested way to extract this information is by using the basic ideas of Data Envelopment Analysis (DEA). DEA origins are found in the late 70s [1]. DEA deals with the evaluation of the performance of objects (Decision Making Units or DMU in DEAs terminology) using the transformation process of several inputs into several outputs. Relying on a technique based on Linear Programming and without having to introduce any subjective or economic prices (weights, costs, etc), DEA provides a measure of efciency of each object. Its basic aim is to separate efcient from non-efcient DMUs and to indicate for each non-efcient DMU its efcient peers. During the last 25 years thousands of papers related to DEA have been published in OR/MS journals. MCDA and DEA have been receiving considerable attention in the OR/MS literature. The problem tackled by DEA is one which may equally well be approached using MCDA. Indeed, in common with many MCDA approaches, DEA incorporates a process of assigning weights to criteria. However, despite having much in common, the two elds have developed almost entirely independently to each other [25]. It is only in the last decade the success of DEA in the area of performance evaluation together with the formal analogies existing between DEA and MCDA (which become clear replacing DMU with alternatives, outputs with maximization criteria and inputs with minimization criteria) have lead some authors to propose to use DEA as a tool for MCDA [68]. Among the rst steps towards this co-operation, was the notion of cross-efciency in DEA, originated by Sexton et al. [9] and further developed by Doyle and Green [10]. The basic aim was to increase the discriminatory power of DEA. In cross-efciency each one of the alternatives is evaluated using the most favourable set of weights (self evaluation). In the same time, the rest of the alternatives are evaluated using the above set of weights. This process is repeated for all the alternatives resulting in a square matrix of evaluations with the diagonal elements being the self evaluation scores and the off-diagonal elements the peer evaluation scores of the alternatives. The nal score for each alternative is obtained as the corresponding column average of this matrix. As Doyle and Green state . . . cross efciency, with its intuitive interpretation as peer appraisal, has less of the arbitrariness of additional constraints, and has more of the right connotations of a democratic process, as opposed to authoritarianism (externally imposed weights) or egoism (self-appraisal) [10]. These results where further elaborated by Doyle [11] in order to develop a MCDA method called alternative cross-evaluation (AXE) in its work with the intriguing title Multiattribute Choice for the Lazy Decision Maker: Let the Alternatives Decide!. The notion of cross-efciency has been emerged as an alternative DEA approach in order to increase the relative discriminating power. As a consequence, it uses DEAs mathematical representation and it is conned in linear relationships. Hence, in MCDA terminology, the value functions of the criteria are assumed linear and the only variables in the Linear Programming (LP) formulation are the relative weights. In the current work we extend the notion of cross efciency in order to allow the value functions in each criterion to be non-linear. It is accomplished with the incorporation in the model of a specially developed non-linear value functions with one parameter. The model turns inevitably to non-linear with the parameters of each criterions value function being the second set of decision variables (the rst is the set of weights). The proposed method is named enhanced alternative cross-evaluation (ACE+) and allows

G. Mavrotas, P. Trillis / Computers & Operations Research 33 (2006) 2083 2098

2085

for a fruitful insight in the problem as it can extract intra-criterion (value functions) and inter-criterion (weights) information. Moreover, it maintains the objective and democratic character of cross-efciency that makes it attractive in minimum information situations. The method can be used when the information from the DM in a MCDA problem is rather limited and we want to obtain a better insight in the problem or, even more, to solve the problem adequately under these minimum information conditions. It must be noted that in relative decision situations the usual, nave approach is to assign equal weights to the criteria, assume linear value functions and use an additive aggregation function. Using more sophisticated methods like ACE+ the underlying information in the alternatives and criteria can be better exploited leading to more robust decisions. In other words, ACE+ is not only used for the evaluation of the alternatives under minimum information conditions, but it also produces information that help the decision maker to better perceive the potential of each alternative in the specic decision context. The rest of the paper is organized as follows: In Section 2 the main topics of DEA and cross-efciency are briey reminded. In Section 3, we describe the suggested method (ACE+) in detail. In Section 4 we present the application of the method with an educational example making comparisons with other approaches. Namely, 14 countries of the European Union are compared by means of seven eco-efciency indicators. Finally, in Section 5 the most important conclusions and issues for further research are presented.

2. Data envelopment analysis and cross-efciency The efciency of a many-input, many-output decision making unit (DMU) may be dened as a weighted sum of its outputs divided by a weighted sum of its inputs (Eq. (1)). Ek = u1 y1k + u2 y2k + + un ynk , 1 x1k + 2 x2k + + m xmk (1)

where yrk is the rth output of the DMU k, xik is the ith input of the DMU k, ur is the weight of the rth output i is the weight of the ith input and Ek the efciency of kth DMU. This, so called engineering ratio is the most popular of a number of alternative measures of efciency. DEA was initiated in 1978 when Charnes, Cooper and Rhodes demonstrated how to transform this fractional linear measure of efciency into a LP model. DEA formulates choice of these weights as a linear program, which allows each DMU to appear in the best possible light (maximize its own measured efciency relative to the other DMUs). The original problem that seeks for the maximum efciency of DMU k is formulated as follows: Ek = max ST
s r=1 ur yrj m i=1 i xij s r=1 ur yrk , m i=1 i xik

1,

j = 1 . . . n, (2)

ur ,

0,

2086

G. Mavrotas, P. Trillis / Computers & Operations Research 33 (2006) 2083 2098

where s is the number of outputs, m is the number of inputs and n is the number of DMU to be evaluated. In their seminal work Charnes, Cooper and Rhodes transform the above fractional optimization problem into conventional LP by assigning the denominator of the objective function to unity. The obtained model became
n

Ek = max
r=1

ur yrk ,

ST
m i xij i=1 m i xik i=1 r=1 n

ur yrj

j = 1 . . . n,

(3)

= 1,

ur ,

0.

The necessary condition for a DMU to be characterized efcient is that the optimal value of model (3) is unity. Model (3) is solved n times, one for each DMU, in order to discover which DMU are efcient and which are not. Because the number of DMUs is usually greater than the number of inputs and outputs it is usually more convenient to solve the dual of model (3) which results in less constraints. Further information for the non-efcient DMU can be also extracted (e.g., which DMU or combination of DMUs dominates a specic non-efcient DMU). A number of alternative models have been developed, allowing for economies of scale [12], constrained weights etc. The interested reader may refer among others to Thanassoulis [13] and Cooper et al. [14] in order to read more about the theory and applications of DEA. The main focus of applications of DEA has been in performance evaluation of non-prot organizations and in estimating non-parametric production function. The initial aim of DEA was rather to identify those DMUs which are not efcient in some sense, and to assess where the inefciencies arise and not to rank the DMUs according to their efciency. In essence, DEA can be described as trying to nd a set of weights which present each DMU in the best possible light. The major advantage of this technique is that it is strongly objective, in the sense that it does not need any preference information. On the other hand, its main limitation is its small discriminating power when used for evaluation purposes. In order to overcome the latter drawback Sexton et al. in 1986 introduce the notion of cross-efciency. Since then several authors contribute in this idea (see e.g., [6,10,15]). The cross-efciency method simply calculates the efciency score of each DMU n times, using the optimal weights evaluated by the n LPs. The results of all the DEA cross-efciency scores can be summarized in a cross-efciency matrix as shown in Eq. (4): hkj =
s r=1 urk yrj m i=1 ik xij

f or

k, j = 1 . . . n.

(4)

Thus, hkj represents the score given to DMU j when optimizing for DMU k, i.e. DMU j is evaluated by the optimal weights of DMU k. Note that the diagonal elements of the cross-efciency matrix (hkk ) represent the standard DEA efciency scores (see Table 1).

G. Mavrotas, P. Trillis / Computers & Operations Research 33 (2006) 2083 2098 Table 1 Cross-efciency square matrix Evaluating DMU Evaluated DMU 1 1 2 n Column average h11 h21 hn1 e1 2 h12 h22 hn2 e2

2087

n h1n h2n hnn en

Furthermore, if the weights of the LP for DMU k are not unique, a secondary goal (either aggressive or benevolent) can be activated. In the aggressive context, DMU k chooses among the optimal weights such that it maximizes self efciency and at a secondary level minimizes the other DMUs cross-efciency levels. The secondary goal can be incorporated in the objective function as follows: max hkk = hkk
n j =1,j =k

hkj

n1

(5)

where >1. On the other hand, in the benevolent context, DMU k chooses among the optimal weights such that it maximizes self-efciency and at a secondary level maximizes the other DMUs cross efciency levels [10]. Thus, the benevolent context is implemented by reversing the sign of in Eq. (5). The cross-efciency ranking method utilizes the results of the above matrix in order to rank the DMUs. Firstwecalculateek astheaveragescoreofthencross-efciencyscoresofDMUk(seeTable1,last row).Subsequently, we rank the DMUs according to the average efciency score ek of each unit. It is argued that ek is more representative than hkk as an efciency score, since all the elements of the cross-efciency matrix are included. Furthermore, in the calculation of ek all the units are evaluated with the same set of weights. The notion of cross-efciency was further extended by Doyle in order to develop the AXE method [11]. AXE can be considered as an attempt to apply DEA in MCDA, so it was properly adjusted to MCDA terminology. In AXE the DMUs correspond to the alternatives which have to be evaluated. The inputs and outputs correspond to the criteria to be minimized and maximized respectively. The weights are the scaling factors of the additive aggregation function as in DEA. Let an MCDA problem with n alternatives (i = 1 . . . n) and m criteria (s = 1 . . . m). Assume that the DM denes value functions Vs ( ) for each criterion in order to map the performance of each alternative i (Xis ) to a value for the DM, expressed as Vs (Xi ). It is recommended to normalize Vs ( ) in [0, 1] or [0,100] in order to have a common scale. In order to perform self-evaluation of alternative i the following LP problem is solved:
m

max hii =
s=1

wis Vs (Xi ), (6)

ST hij 1, j = 1 . . . n, s = 1 . . . m.

wis 0,

2088

G. Mavrotas, P. Trillis / Computers & Operations Research 33 (2006) 2083 2098

The essence of the DEA approach is in the challenge to each alternative to choose its own weights in order to become as desirable as possible (maximize hii ) under the certain restrictions governing all the alternatives (the hij 1 condition). After solving (6), one can evaluate alternative j (j = i) under ith alternatives view (with the optimal weights for i) using the following formula:
m

hij =
s=1

wis Vs (Xj ).

opt

(7)

In this way one can obtain the cross-evaluation matrix, similar to Table 1. The column average for each alternative represents the so-called peer-evaluation score according to which the rating of alternatives is performed. 3. The suggested method: ACE+ 3.1. General description and differences with AXE ACE+ can be considered as an attempt to enhance the capabilities of AXE. It is designed to be used with even less information than AXE. In AXE the value function Vs () of each criterion is considered to be xed and given by the DM. In ACE+ the value functions are adjustable, letting each alternative use the most favourable value function during its self appraisal. In other words, the self-evaluation procedure, which is the cornerstone of cross-evaluation, is extended to comprise not only weight adjustments but also value function adjustments. The motivation is to achieve the best case scenario for each alternative during its self appraisal in the MCDA context. While AXE exploits mainly the inter-criteria information (weights of criteria) ACE+ exploits also the intra-criteria information (value function in each criterion). As a consequence, ACE+ is characterized by more degrees of freedom than AXE, and results in a wider range of performances for the alternatives. Another modication regards the objective function of the self-evaluation model. Instead of using the aggressive approach of AXE (i.e. use as primary goal the maximization of the score hii of alternative i and if there are alternative optima use as secondary goal the minimization of the average score of the rest alternatives) we suggest a somewhat different idea: rather than seeking for the combinations of weights and value functions that give to ith alternative the higher possible score, we seek those combinations of weights and value functions that give to the ith alternative the greatest possible advantage in relation to the other alternatives. In the self-appraisal model it is translated as maximization of the difference between the ith alternatives score and the average score of the rest alternatives. Belton and Stewart in an alternative approach they propose the maximization of the minimum difference in order to identify potentially optimal alternatives [19]. In the present approach the weight and the value function parameter of a criterion in the self evaluation model of alternative i, are decided taking into account the performances of the other alternatives in the specic criterion and not only the performances of alternative i in the other criteria (as it is done in AXE). As it will be shown in Section 4, this approach leads to more meaningful results. 3.2. The calculation procedure 3.2.1. Normalization The rst step in the calculation procedure is the normalization of the alternative scores in the criteria. The common scale in the criteria is necessary in order to have comparable weights in the aggregation

G. Mavrotas, P. Trillis / Computers & Operations Research 33 (2006) 2083 2098

2089

model. The performance of the alternatives in each criterion is scaled to the interval [0,1] according to the following linear formulas (see e.g., [16]): For a minimization criterion j the normalized score for alternative i is given by Nij =
U Xj Xij

U L Xj Xj

(8)

Similarly, for a maximization criterion j the normalized score for alternative i is given by Nij =
L Xij Xj

U L Xj Xj

(9)

U L The values of Xj and Xj are the lower and upper anchor values respectively of the criterion j. These values can be set exogenously but they are usually taken as the minimum and the maximum of each criterion. However, there are cases where it is more convenient to take anchor values different from the minimum and the maximum of the criterion, especially in the case of outliers. The outliers are isolated values which are away from the main body of the data, and can be identied by specic statistical tests. The use of outliers as anchor values tend to squeeze the rest of the data in the [0,1] scale reducing the discriminating power of the criterion. In order to avoid this inadequacy we propose that in the case of outliers to use as lower and upper anchor values the percentiles 5% and 95% instead of the minimum and maximum value, respectively. The normalized scores of the outliers that inevitably fall outside the interval [0,1] are adjusted to the most nearest bound. With the above procedure we alleviate the effect of outliers in the criterion scale.

3.2.2. Value functions The value functions that are used in the proposed method are the typical concave and convex functions from decision analysis, along with the linear function [1719]. They are formulated as one-parameter function with a common formula as given below: Yij = Vj (i) = 1 ecj Nij . 1 e cj (10)

The denominator in (10) is necessary in order to keep the values of the value function in [0,1]. The parameter cj denes whether the function will be concave, convex or linear as it is shown in Fig. 1. For linear functions we set cj = 0.001, instead of zero in order to avoid the disappearance of the denominator. Negative and positive values of cj characterize concave and convex functions respectively, while cj is taken to vary from 5 to 5 giving enough curvature to the value functions. The concave and convex functions indicate Decreasing and Increasing Returns to Scale respectively, in production theory terminology. In the decision theory they indicate risk averse and risk prone decision makers and they are widely used. The general notion is that in concave functions the difference between two alternatives in the beginning of the scale counts more than the same difference at the end of the scale. The opposite holds for the convex functions.

2090

G. Mavrotas, P. Trillis / Computers & Operations Research 33 (2006) 2083 2098

Fig. 1. Examples of value functions used in ACE+.

3.2.3. Model for self-evaluation and the cross-evaluation matrix In a MCDA problem with n alternatives and m criteria the self-evaluation model for each one alternative is obtained through model (11)
m

hkk = max
j =1

1 ecj Nkj wj 1 e cj

n i=1,i=k

c N m 1e j ij j =1 wj 1ecj

n1

ST
m

wj = 1,
j =1

wj wj cj cj wj

Uj , Lj , Vj , Pj , 0.

j = 1 . . . m, j = 1 . . . m, j = 1 . . . m, j = 1 . . . m, (11)

It is a non-linear programming (NLP) model with decision variables the weights (wj ) and the value function parameters (cj ). The parameters of the model are the normalized scores Nij , the bounds on the weights Uj and Lj and the bounds on the value function parameters Vj and Pj . The objective function expresses the maximization of the difference between the score of the self-evaluating alternative i and the average score of the rest alternatives. Although it looks complicated, the objective function is smooth and can be adequately tackled by a conventional NLP solver like the GRG2 code which is used in Microsoft Excels Solver module. The constraints of the model are the restriction on the sum of weights to equal to unity and the corresponding bounds on the weights and the value function parameters. The recommended

G. Mavrotas, P. Trillis / Computers & Operations Research 33 (2006) 2083 2098

2091

Fig. 2. Flowchart of the method.

upper and lower bounds of the value function parameters Vj and Pj are 5 and 5, that are considered to give enough curvature to the value functions. The upper and lower bounds on the weights depend on the number of criteria. A rule of thumb is to set these bounds to +/50% of the uniform weight value 1/m (i.e. the weight interval becomes [l/2m, 3/2m]). Solving model (11) for all n alternatives the cross-evaluation matrix is obtained. The owchart of the procedure is shown in Fig. 2. Fruitful information is carried by the cross-evaluation matrix (Table 1). Except from the self-evaluation score hkk and the peer evaluation score (column average ek ), one can obtain the range of scores for each one of the alternatives. The range of scores for each alternative is an important piece of information that can be taken into account in the nal decision. For example between two alternatives with the same average score one may select the one with the smaller range of scores (risk averse DM). In general, alternatives which are characterized by more extreme performances in the criteria result in greater range than alternatives with mediocre performances. With ACE+ this characteristic can be identied and encountered in the nal decision. In addition, one can examine the weight and the value function parameter distributions per criterion as it is obtained by the n self-evaluation problems. From these two distributions one can use the corresponding average values as representative weights and value function parameters. The weights can be considered as inter-criteria information while the value function parameters as intra-criteria information.

2092

G. Mavrotas, P. Trillis / Computers & Operations Research 33 (2006) 2083 2098

3.2.4. Optional capabilities Besides the basic framework that was described in the previous paragraphs, two optional modications can also be incorporated in the ACE+ method. First, it is the way that the representative score is obtained from the efciency score matrix. The average of the efciency scores for each alternative, although a reasonable metric for the distribution, it may lead to violations of monotonicity as it was shown by Bouyssou [20], especially when we have only few alternatives. The violation of monotonicity is the main criticism against the cross-efciency method and occurs when the position of one alternative may improve although its evaluation in one criterion deteriorates. In order to avoid this phenomenon, the midrange (instead of the average) of the distribution of the efciency scores can be used as a representative score for each alternative. As it can be shown in the example of Bouyssou in [20], the midrange metric does not violate the monotonicity principle even in the extreme case of the specic example, as it encounters only the extreme values of the cross-efciency distribution. Another modication regards the formulation of the non-linear model. In order to avoid the solution of a non-linear programming problem (which is always vulnerable to local optimal solutions) we may separate the process into two independent steps. First, we can calculate for each alternative and for each criterion the best value function regarding the rest of the other alternatives. This implies the calculation of the corresponding parameter c that maximizes the difference between the score of the optimized alternative i and the average score of the rest alternatives in the criterion j and is dened as cij . This task requires n m one-parameter non-linear optimizations of the form
cij

cj [Vj ,Pj ]

max

1 ecj Nij 1 e cj N j 1 e cj 1 e cj

avg

for i = 1 . . . n, j = 1 . . . m,
avg

(12)

where n is the number of alternatives, m the number of criteria, Nj is the average score of the criterion excluding the ith alternative and Vj and Pj are the bounds for cj . In other words, cij is the optimal value of the parameter c that is derived from the self-evaluation of the ith alternative in the j th criterion as it is compared with a virtual alternative having the average performance of the j th criterion. The optimal score of the ith alternative in the j th criterion is subsequently calculated as
sij

1 ecij Nij = 1 e cj

for i = 1 . . . n, j = 1 . . . m.

(13)

The second step is the maximization of the multicriterion score for each alternative. The already calculated optimal scores in each criterion are used and the only decision variables for the model are the weights of the criteria. For alternative i it holds
m

hii =

wj [Lj ,Uj ]

max

j =1

wj sij

for i = 1 . . . n,

(14)

where Lj and Uj are the bounds on the weights for criterion j . (i) The optimal parameters cij and weights wj for alternative i are then used for the calculation of the cross-efciencies of the other alternatives k in order to obtain the scores hik for k = 1 to n. In this way the non-linear optimization procedure is divided into two (more easy to solve) independent parts: First the calculation of the optimal parameters for each alternative and each criterion (n m one parameter

G. Mavrotas, P. Trillis / Computers & Operations Research 33 (2006) 2083 2098

2093

non-linear optimizations) and then the calculation of the optimal weights for each alternative (n linear optimizations). The results obtained from the two procedures in a few examples were almost identical regarding the ranking of the alternatives and the value of the parameters (weights and value function parameters).

4. Numerical example The example that is used in order to demonstrate the method concerns the environmental evaluation of 14 countries of the European Union (we exclude Luxemburg in most cross-national comparisons in EUcause it acts as an outlier in the data set). The cross-country comparison based on multiple indices is a common task that requires an objective evaluation procedure with little information by the decision maker. Moreover, the relative information must be extracted by the data, as usually there are no decision makers (or they are hardly available) willing to assign numbers for the required parameters. This is the decision context where ACE+ can really help. The evaluation is performed according to seven indicators that are widely used to express eco-efciency in a country. The indicators which are used as criteria in the multicriteria evaluation problem are: 1. energy intensity (ENINT) dened as the Energy to GDP ratio (toe/million ), 2. emission intensity (GHGINT) dened as the Green House Gas (GHG) emissions to GDP ratio (kg CO2 /), 3. acidifying gases intensity (ACGAS) dened as the ratio of acidifying gases (SO2 , NOx , etc) to GDP (kg/), 4. renewable energy sources (RES) share in the energy mix excluding big hydro projects (%), 5. share of population with public sewage treatment (PUBSEW) measured as %, 6. share of farming areas cultivated without chemical substances (organic farming, ORGFAR) measured as %, 7. share of paper that comes from recycle (RECPAP) measured as %. The performance of the countries in the seven criteria is shown in Table 2 and are taken from the ofcial site of the EU (http://europa.eu.int). From the above matrix we obtain the matrix of the normalized scores (Table 3) according to Eqs. (8) U L and (9) and taking as Xj and Xj the 95% and 5% percentile respectively, in order to reduce the outlier effect. The problem is rst solved with ACE+ with cj [5, 5] and wj [0.05, 0.3] and the cross-evaluation matrix is shown in Table 4. The calculation procedures of the method have been developed in Visual Basic for Applications under the platform of Microsoft Excel using the Excels Solver for the non-linear optimization module (the GRG2 algorithm incorporated in Solver found to be adequate for such type of NLP problems). Cell a(i, j ) denotes the score of alternative j if it is calculated with the weights and value function parameters that result from the self-evaluation of alternative i. Therefore the gures in the diagonal are the self-evaluation scores for each country. The last row includes the column averages as the representative value of the multicriteria score for each alternative. As it was mentioned in the previous section an important piece of information is the range of scores for each alternative. Fig. 3 illustrates the range of sores along with the average score for each country,

2094

G. Mavrotas, P. Trillis / Computers & Operations Research 33 (2006) 2083 2098

Table 2 Performances of countries in the eco-efciency criteria Code Austria Belgium Germany Denmark Spain Greece France Finland Italy Ireland Holland Portugal Sweden United Kingdom Criterion direction A B D DK E EL F FIN I IRL NL P S UK ENINT 182 30 216 167 231 345 224 266 170 191 292 319 254 255 min GHGINT 0.52 0.81 0.65 0.57 1.90 1.57 0.50 0.62 0.56 0.93 0.82 1.17 0.35 0.72 min ACGAS 0.060 0.101 0.065 0.094 0.514 0.376 0.094 0.082 0.091 0.212 0.080 0.381 0.054 0.102 min RES 10.5% 0.8% 2.8% 11.6% 4.5% 0.5% 2.3% 11.2% 4.7% 1.7% 3.7% 4.4% 5.1% 1.2% max PUBSEW 81.5 38.05 91.5 89.01 48.3 56.2 81 80 75 57.6 97.9 46.1 93 84 max ORGFAR 8.4 1.4 2.6 5.4 1.4 0.3 1.0 5.2 5.5 0.7 1.1 1.2 4.5 2.5 max RECPAP 69 38 70 50 42 29 41 57 31 12 62 40 62 40 max

Table 3 Normalized scores Code A B D DK E EL F FIN I IRL NL P S UK ENINT 0.917 0.184 0.703 1.000 0.614 0.000 0.655 0.394 0.993 0.861 0.231 0.057 0.468 0.458 GHGINT 0.944 0.712 0.842 0.903 0.000 0.093 0.957 0.866 0.908 0.615 0.699 0.415 1.000 0.785 ACGAS 0.995 0.883 0.982 0.904 0.000 0.140 0.902 0.935 0.910 0.584 0.942 0.126 1.000 0.881 RES 0.921 0.010 0.198 1.000 0.357 0.000 0.151 0.987 0.376 0.094 0.282 0.348 0.414 0.047 PUBSEW 0.743 0.000 0.937 0.889 0.098 0.251 0.733 0.714 0.617 0.278 1.000 0.055 0.967 0.792 ORGFAR 1.000 0.141 0.340 0.808 0.138 0.000 0.077 0.782 0.832 0.025 0.085 0.104 0.660 0.316 RECPAP 0.992 0.323 1.000 0.582 0.409 0.129 0.388 0.733 0.172 0.000 0.841 0.366 0.841 0.366

as they are obtained from the 14 self evaluation model runs (column ranges and column averages from Table 4). It is obvious that some alternatives display wider range than others. These alternatives are characterized by dispersed performances along the criteria and so they are more sensitive to the modication of weights. On the other hand the alternatives with more narrow range (e.g., A and EL) are characterized by more accumulated performances in the criteria as it can be veried from the relative data in Table 3.

G. Mavrotas, P. Trillis / Computers & Operations Research 33 (2006) 2083 2098 Table 4 Cross-evaluation matrix from method ACE+ Cross-evaluated alternative Code A B D 0.48 0.85 0.89 0.35 0.79 0.44 0.75 0.55 0.52 0.80 0.84 0.79 0.70 0.91
0.69

2095

DK 0.51 0.93 0.56 0.96 0.90 0.75 0.84 0.91 0.87 0.92 0.67 0.93 0.65 0.97
0.81

E 0.10 0.18 0.13 0.14 0.71 0.13 0.36 0.27 0.16 0.19 0.21 0.52 0.13 0.26
0.25

EL 0.03 0.29 0.03 0.04 0.10 0.24 0.21 0.09 0.05 0.26 0.07 0.24 0.01 0.26
0.14

F 0.17 0.78 0.39 0.25 0.70 0.40 0.83 0.35 0.39 0.73 0.47 0.69 0.55 0.76
0.53

FIN 0.49 0.86 0.52 0.63 0.84 0.67 0.76 0.90 0.57 0.73 0.62 0.87 0.54 0.94
0.71

I 0.34 0.78 0.40 0.62 0.76 0.55 0.80 0.53 0.85 0.87 0.42 0.76 0.51 0.87
0.65

IRL 0.09 0.55 0.15 0.22 0.42 0.31 0.48 0.15 0.32 0.77 0.20 0.38 0.14 0.51
0.34

NL 0.28 0.84 0.71 0.21 0.72 0.39 0.62 0.41 0.29 0.63 0.85 0.82 0.64 0.80
0.59

P 0.07 0.51 0.11 0.11 0.49 0.11 0.20 0.23 0.15 0.35 0.21 0.70 0.10 0.29
0.26

S 0.41 0.86 0.78 0.39 0.88 0.49 0.92 0.64 0.61 0.71 0.85 0.87 0.93 0.94
0.73

UK 0.13 0.77 0.40 0.26 0.56 0.38 0.66 0.44 0.37 0.67 0.47 0.59 0.42 0.86
0.50

Self evaluating alternative A 0.94 0.05 B 0.93 0.74 D 0.81 0.21 DK 0.78 0.16 E 0.95 0.39 EL 0.90 0.07 F 0.91 0.26 FIN 0.90 0.29 I 0.87 0.21 IRL 0.95 0.54 NL 0.78 0.33 P 0.94 0.50 S 0.70 0.25 UK 0.95 0.51 Avg
0.88 0.32

Fig. 3. Ranges and average values for the multicriteria scores of the alternatives.

Subsequently, in order to examine the effect of the different objective function in the self-evaluation model between the AXE and the ACE+ method we use the following procedure: We rst solve the problem with AXE and then we solve the problem with ACE+ but restricting the value functions to be linear (hereafter named as ACE+LIN). It is interesting to note that the objective function of ACE+LIN leads to more balanced weights than AXE, as it is shown in Table 5. This is explained by the following reasoning: in AXE, if a criterion has the majority of alternatives with high performances (positive skew), the corresponding self-evaluation problems will attribute this criterion with relative high

2096

G. Mavrotas, P. Trillis / Computers & Operations Research 33 (2006) 2083 2098

Table 5 Weight distribution from AXE and ACE+LIN A B D 0.05 0.05 0.30 0.05 0.20 0.05 0.30 DK 0.30 0.05 0.20 0.30 0.05 0.05 0.05 E 0.30 0.05 0.05 0.20 0.05 0.05 0.30 EL 0.05 0.05 0.30 0.05 0.30 0.05 0.20 F 0.05 0.30 0.30 0.05 0.20 0.05 0.05 FIN 0.05 0.20 0.30 0.30 0.05 0.05 0.05 I 0.30 0.20 0.30 0.05 0.05 0.05 0.05 IRL 0.30 0.30 0.20 0.05 0.05 0.05 0.05 NL 0.05 0.05 0.30 0.05 0.30 0.05 0.20 P 0.05 0.30 0.05 0.20 0.05 0.05 0.30 S 0.05 0.30 0.30 0.05 0.20 0.05 0.05 UK 0.05 0.20 0.30 0.05 0.30 0.05 0.05 Avg 0.12 0.17 0.25 0.11 0.14 0.07 0.15

Weights from AXE w1 0.05 0.05 w2 0.05 0.30 w3 0.30 0.30 w4 0.05 0.05 w5 0.05 0.05 w6 0.30 0.05 w7 0.20 0.20

Weights from ACE+LIN w1 0.05 0.05 0.05 w2 0.05 0.30 0.05 w3 0.05 0.30 0.20 w4 0.30 0.05 0.05 w5 0.05 0.05 0.30 w6 0.30 0.05 0.05 w7 0.20 0.20 0.30

0.30 0.05 0.05 0.30 0.05 0.20 0.05

0.30 0.05 0.05 0.30 0.05 0.05 0.20

0.05 0.05 0.05 0.30 0.30 0.05 0.20

0.05 0.30 0.30 0.05 0.20 0.05 0.05

0.05 0.05 0.05 0.30 0.05 0.30 0.20

0.30 0.20 0.05 0.05 0.05 0.30 0.05

0.30 0.30 0.20 0.05 0.05 0.05 0.05

0.05 0.05 0.20 0.05 0.30 0.05 0.30

0.05 0.20 0.05 0.30 0.05 0.05 0.30

0.05 0.20 0.05 0.05 0.30 0.05 0.30

0.05 0.20 0.30 0.05 0.30 0.05 0.05

0.12 0.15 0.14 0.16 0.15 0.11 0.18

weight (e.g., criterion 3 in our example). Therefore, the specic criterion will be characterized by high weights among the 14 set of weights and thus relative high average weight. In the ACE+LIN method this does not happen because the objective function does not promote alternatives with high performances independently of the other alternatives. In other words, the high performance of alternative i in criterion j (in relation to the other criteria) it is not enough to attribute high weight to the specic criterion according to the self-evaluation model (as it happens in AXE). The performance of the other alternatives in this criterion could be in average higher than the performance of the ith alternative, driving the self-evaluation model of the ith alternative not to choose a relatively high score for criterion j . Regarding the comparison between ACE+LIN and ACE+ the basic difference is that the scores of each alternative resulted from ACE+ vary in a wider range than the respective scores from ACE+LIN. This is explained by the greater number of degrees of freedom in ACE+ (adjustable and not xed value functions) which allows for more variability in the calculated scores. Conclusively, some general remarks concerning the results are outlined. It is obvious that solely the average score obtained from the cross-evaluation matrix is not enough to judge the alternatives. However, the judgment can be signicantly enriched with information concerning the corresponding score-ranges that is provided by the cross-evaluation matrix. As it was expected, the countries which present wide range of performances across the criteria result in a wide range of scores in the cross-evaluation matrix (e.g., B, F, IRL, UK). These countries are very sensitive to the weight set that is used for their evaluation. On the contrary countries with more accumulated scores in the criteria (e.g., A, EL) result in a narrower multicriteria score range and thus they are less sensitive to weights modication.

G. Mavrotas, P. Trillis / Computers & Operations Research 33 (2006) 2083 2098

2097

5. Concluding remarks The situations where a multicriteria evaluation of alternatives has to be performed based on limited information about the preferences of the DM are not rare. The self and peer evaluation of the alternatives, as it was inspired by DEA methodology, is a good remedy in such situations where the relevant information must be extracted from the data. One of the rst approaches in this direction was AXE method that uses the cross-evaluation of alternatives. In the suggested method ACE+ we are going one step further by allowing each alternative to use its own weights and value functions for the criteria. The price for the more realistic representation of the decision situation is that a more difcult problem has to be solved in the self evaluation phase of each alternative: Instead of an LP problem with the weights of criteria being the only decision variables, now we have an NLP problem with two sets of decision variables, namely the weights of criteria and the value function parameters. The use of a different objective function in the self-evaluation model of ACE+ seems to lead to more meaningful results, as it aims to optimize the relative position of the respective alternative in relation to the others, and not an absolute measure of its performance. The method ACE+ requires only a minimum of information in order to be applied and it provides a wealth of decision making relevant information. In ACE+ the DM is not limited solely to average values but he/she is encouraged to develop an information warehouse (ranges of scores, information about the weights and the value functions, pairwise comparisons) that is very useful for the nal decision. Consequently, ACE+ inherently performs a parallel sensitivity analysis of the results, leading to more robust decisions. Another attractive issue of ACE+ is that its baselines can be easily understood by non-experts. For example, ACE+ can be very useful in decision situations where each alternative is represented by a stakeholder. It is a method that easily can be agreed upon as it provides the results in an objective and democratic manner. Each stakeholder assigns its most favourable weights to the criteria in order to promote his/her alternative. The nal judgment is then emerges from the cross-evaluation (self and peer evaluation) of each one of the alternatives. Some ideas for future research regarding ACE+ include the following: Different objective functions can be tested in the self evaluation model. More types of value functions, besides the usual concave and convex value functions, may be incorporated in the model (e.g., S-curves) in order to make the model more general. Finally, it will be a good idea if the method is adjusted so that it can be used iteratively, in the case that the DM is available. The DM exploits the obtained information and expresses his/her preferences which are incorporated in the model through an interactive process, which hopefully converges to the nal decision. Acknowledgements The authors should like to thank the two anonymous referees for their stimulating comments that enrich the paper. References
[1] Charnes AW, Cooper WW, Rhodes E. Measuring efciency of decision making units. European Journal of Operational Research 1978;2:42944.

2098

G. Mavrotas, P. Trillis / Computers & Operations Research 33 (2006) 2083 2098

[2] Stewart T. Data envelopment analysis and multiple criteria decision making: a response. Omega 1994;22(2):2056. [3] Stewart T. Relationships between data envelopment analysis and multicriteria decision analysis. Journal of the Operational Research Society 1996;47:65465. [4] Doyle J, Green R. Data envelopment analysis and multiple criteria decision making. Omega 1993;21(6):7135. [5] Belton V, Vickers S. Demystifying DEAA visual ineractive approach based on multiple criteria analysis. Journal of the Operational Research Society 1993;44(9):88396. [6] Oral M, Kettani O, Lang P. A methodology for collective evaluation and selection of industrial R&D projects. Management Science 1991;37(7):87185. [7] Sarkis J. A comparative analysis of DEA as a discrete alternative multiple criteria decision tool. European Journal of Operational Research 2000;123:54357. [8] Green R, Doyle J, Cook W. Preference voting and project ranking using DEA and cross evaluation. European Journal of Operational Research 1996;90:46172. [9] Sexton T, Silkman R, Hogan A. Data envelopment analysis: critique and extensions. In: Silkman R, editor. Measuring efciency: an assessment of data envelopment analysis. San Francisco: Jossey-Bass; 1986. [10] Doyle J, Green R. Efciency and cross-efciency in DEA: derivations, meanings and uses. Operational Research Society 1994;45(5):56778. [11] Doyle J. Multiattribute choice for the lazy decision maker: let the alternatives decide!. Organizational Behavior and Human Decision Processes 1995;62:87100. [12] Banker R, Charnes A, Cooper W. Some models for estimating technical and scale inefciencies in data envelopment analysis. Management Science 1984;30:107892. [13] Thanassoulis E. Introduction to the theory and application of data envelopment analysis. Massachusetts: Kluwer Academic Publishers; 2001. [14] Cooper WW, Seiford L, Tone K. Data envelopment analysis: a comprehensive text with models applications, references and DEA-solver software. Massachusetts: Kluwer Academic Publishers; 1999. [15] Adler N, Friedman L, Sinuany-Stern Z. Review of ranking methods in the data envelopment analysis context. European Journal of Operational Research 2002;140:24965. [16] Zeleny M. Multiple criteria decision making. New York: McGraw-Hill; 1982. [17] Keeney RL, Raiffa H. Decisions with multiple objectives: preferences and value tradeoffs. New York: Wiley; 1976. [18] Von Winterfeldt D, Edwards W. Decision analysis and behavioral research. New York: Cambridge University Press; 1986. [19] Belton V, Stewart T. Multiple criteria decision analysis: an integrated approach. Massachusetts: Kluwer Academic Publishers; 2002. [20] Bouyssou D. Using DEA as a tool for MCDM: some remarks. Journal of the Operational Research Society 1999;50: 9748.