A Simulation and Evaluation of Earned Value Metrics to Forecast the Project Duration Author(s): M. Vanhouckel and S.

Vandevoorde Source: The Journal of the Operational Research Society, Vol. 58, No. 10 (Oct., 2007), pp. 13611374 Published by: Palgrave Macmillan Journals on behalf of the Operational Research Society Stable URL: http://www.jstor.org/stable/4622825 . Accessed: 12/05/2011 17:28
Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you may use content in the JSTOR archive only for your personal, non-commercial use. Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at . http://www.jstor.org/action/showPublisher?publisherCode=pal. . Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed page of such transmission. JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Palgrave Macmillan Journals and Operational Research Society are collaborating with JSTOR to digitize, preserve and extend access to The Journal of the Operational Research Society.

http://www.jstor.org

Journal of the Operational Research Society (2007) 58, 1361-1374

2007 Operational Research Society Ltd. All rights reserved. 0160-5682/07 $30.00 www.palgravejournals.com/jors

A
to

simulation
forecast the

and

evaluation

of

earned

value

metrics

project

duration

M Vanhouckel,2* and S Vandevoorde3 Ghent,Belgium;2VlerickLeuvenGhentManagement School, Ghent,Belgium;and 1GhentUniversity, 3Fabricom AirportSystems,Brussels,Belgium In this paper, we extensively review and evaluate earned value (EV)-based methods to forecast the total project duration.EV systems have been set up to deal with the complex task of controllingand adjusting the baseline project schedule duringexecution, taking into accountproject scope, timed delivery and total projectbudget.AlthoughEV systemshave been provento providereliableestimatesfor the follow-upof cost within our projectassumptions,they often fail to predictthe total durationof the project.We performance an extensive simulationstudy where we carefullycontrolthe level of uncertainty the project,the in present on influenceof the projectnetworkstructure the accuracyof the forecastsandthe time horizonwherethe EVbased measuresprovideaccurateandreliableresults.We assumea projectsettingwhereprojectactivitiesand unforeseeableevents and/or precedencerelationsare known in advanceand do not considerfundamentally unknowninteractions among variousactions that might cause entirelyunexpectedeffects in differentproject parts.This is the firststudythatinvestigatesthe potentialof a recentlydevelopedmethod,the earnedschedule method,which improvesthe connectionbetween EV metricsand the projectdurationforecasts. Journalof the Operational Research Society(2007) 58, 1361-1374. doi:10.1057/palgrave.jors.2602296 Published online 13 September 2006 simulation; Keywords:projectmanagement; forecasting

1. Introduction Earnedvalue management (EVM) is a methodologyused to measure and communicatethe real physical progress of a projectand to integratethe three criticalelements of project It management (scope, time andcost management). takesinto accountthe work complete,the time taken and the costs incurredto complete the project,and it helps to evaluateand controlprojectriskby measuring projectprogressin monetary terms.The basic principlesand the use in practicehave been describedin manysources(for an overview, comprehensively see eg Anbari (2003) or Fleming and Koppelman(2005)). AlthoughEVM has been set up to follow-up both time and of cost, the majority the researchhas been focusedon the cost the paperwrittenby Fleming and Koppelman aspect (see eg (2003) who discussEVM from a price-tagpoint-of-view).In this paper,we elaborate the recentresearchfocused on the on time aspectof EVM andcomparea newly developedmethod, called earned schedule (ES) (Lipke, 2003a), with the more traditional of approach forecastinga project'sduration. In this paper, we extensively simulate project execution based on a large set of very diverse projectnetworksand a wide set of uncertaintyscenarios.This approachallows an objective and extensive comparisonbetween various EVM
*Correspondence: M Vanhoucke,Faculty of Economics and Business Administration,Ghent University,Hoveniersberg24, Ghent, Belgium. E-mail: mario.vanhoucke@ugent.be

methodsto forecastthe projectduration. generatenumerWe ous networksbased on a networkgeneratorfrom literature andbuilda baselineplanwithrandomly generated activitydurationsand costs. We simulateprojectexecutionby meansof Monte-Carlo simulations underdifferentcontrolledscenarios andmeasureprojectperformance, basedon the actual(ie simdurations costs thatwill differfrom their and ulated)activity corresponding plannedvalues (PVs). Based on these differwe calculatethe earnedvalue (EV) metricsand report ences, our forecastingindicators.Thanksto this specific approach, we aim at drawinggeneral conclusionsthat hold for a wide thandrawscenarios,rather varietyof projectsanduncertainty ing case-specificconclusionsbased on one or a few real-life projects.The purposeof this researchis threefold.First, we evaluatethe forecastaccuracyof the threemethodsthataimto basedon EV metrics.Second, predictthetotalprojectduration we check theirrobustnessfor differentnetworktypes, based on a pre-defined of topological structure each network.Last, we study the behaviourof the differentforecastingmethods with respectto the stage of completionof the projectand the level of uncertainty each (criticalandnon-critical) for activity. The outline of the paper is as follows. In Section 2, we reviewthe current key parameters, corresponding EV the performance measuresand their use as indicatorsto predict a and project'sfinalduration cost. In Section 3, we proposethe of our simulationmodel to test the accuracyof the settings threemethodsto predicta project'sfinal duration. Section In

1362

Journal the Operational of Research Vol. Society 58, No.10

4, we discuss our results for the three methods based on different levels of uncertainty and a known structure for each project. We conclude in Section 5 with some overall conclusions and suggestions for future research. It is crucial to note that this research paper rests on a fundamental assumption that project activities and precedence relations are known in advance. Hence, we assume a project setting where estimates can be given within a certain range, even though we may not be able to predict every source of unexpected event with certainty. However, projects often do not fulfil these assumptions but, on the contrary, are commonly plagued by fundamentally unforeseeable events and/or unknown interactions among various actions and project parts (Loch et al, 2006). The straight application of the metrics and methods evaluated in this paper are certainly insufficient for projects where our assumption does not hold, but this is certainly outside the scope of this paper. We refer to the book by Loch et al (2006) for a more general framework where the authors position and classify the sources of risk by the foreseeability of the underlying influence factors and by the complexity of the project. We also cite the letter to the editor of Harvard Business Review from Cooper (2003) as a response to the article written by Fleming and Koppelman (2003). In this letter, the author argues that the use of EVM can be questioned when it is applied in highly complex projects. Due to the cycles of rework, the accuracy of the EVM metrics can be biased, leading to incorrect management decisions.

Earned Value Key Parameters

Planned Value (PV) Actual Cost (AC) Earned Schedule (ES)

Earned Value(EV)

Earned Value Performance Measures

Performance Cost Performance Schedule

Index (CPI) Cost Variance (CV) Index (SPI) Schedule Variance (SV)

Schedule Performance
Index (SPI(t)) Schedule Variance (SV(t))

Translation to time units Time Variance (TV) Earned Duration (ED)

Earned Value Forecasting Indicators

Cost
Estimate at Completion (EAC)

Duration
Estimate at Completion (EAC(t))

Duration
Estimate at Completion (EAC(t))

measuresand foreFigure 1 EVM: key parameters, performance casting indicators.

2. EV metrics revisited
In this Section, we review the different metrics of an EVM system, as used in our simulation approach. In Section 2.1, we briefly review the EVM key parameters that serve as an input for the performance measures and the forecasting indicators. In Section 2.2, we briefly review the existing performance measures and in Section 2.3, we discuss the use of these performance measures to forecast the future performance of the project. Figure 1 serves as a guideline to Sections 2.1- 2.3. 2.1. EVM key parameters Project performance should be measured throughout the life of the project and hence requires a fixed time frame (ie a baseline schedule) for the project. A project schedule defines starting times (and finishing times) for each project activity and hence, a PV for each activity, both in terms of duration and costs. The planned duration (PD) equals the total project duration as a result of the constructed critical path method (CPM) schedule and is often referred to as schedule at completion (SAC, Anbari, 2003). The actual time (AT) defines the project progress and reporting periods for performance measurement. The actual duration (AD) defines the real activity or project duration. The budget at completion (BAC) is the sum of all budgeted costs for the individual activities. EVM requires three key parameters to measure project performance, that is, the PV, the actual cost (AC) and the EV. The

PV is the time-phased budget baseline as an immediate result of the CPM schedule constructed from the project network. The PV is often called budgeted cost of work scheduled. The AC is often referred to as the actual cost of work performed and is the cumulative AC spent at a given point AT in time. The EV represents the amount budgeted for performing the work that was accomplished by a given point AT in time. It is often called the budgeted cost of work performed and equals the total activity (or project) BAC multiplied by the percentage activity (or project) completion at the particular point in time (PC x BAC). Figure 2 displays the three EVM key parameters for a fictive project under the four different possible time/cost scenarios. 2.2. EVM performance measures Project performance, both in terms of time and costs, is determined by comparing the three key parameters PV, AC and EV, resulting in four well-known performance measures: SV SPI CV CPI schedule variance (SV = EV-PV) schedule performance index (SPI = EV/PV) cost variance (CV = EV-AC) cost performance index (CPI = EV/AC)

In our specific simulation approach, we calculate these performance measures on the project level, and not on the level of each individual activity. Jacob and Kane (2004) criticize this approach and argue that the well-known performance measures are true indicators for project performance as long as they are used on the activity level, and not on the control account level or higher work breakdown structure (WBS) levels. They illustrate their statement with a simple example with two activities, leading to wrong and misleading results. As an example, a delay in a non-critical activity might give a warning signal that the project is in danger, while there is

MVanhoucke S Vandevoorde-Earnedmetrics forecast to duration1363 and value project

delay
>......

over budget

>I

sdelay
under budget

o
CLSV<0 SPI<1 CV <O CPI<1 -SV<0 SPI <1 CV> O CPI> 1

AT ...................... ..... ahead ii

Time over budget

AT

Time

&

ahead Sunder

budget budget

SSV>O

SPI>1 CV<O CPI<1+

SV>O

SPI>1

CV>O CPI>1

AT

Time

AT

Time

S PV --

AC -

EV

under 1: overbudget; Scenario fourscenarios. Scenario lateproject, PV, Figure2 TheEVMkey parameters AC andEV for a project 4: overbudget; Scenario earlyproject, and under Scenario earlyproject, 2: lateproject, under 3: budget. budget;

no problem at all since the activity only consumes part of its slack. Since we calculate the performancemeasureson the projectlevel, this will lead to a false warningsignal and hence, wrong correctiveactions can be taken.We recognize activitiescan be neuthat effects (delays) of non-performing activities (aheadof schedule)at tralizedby well-performing higherWBS levels, which might result in maskingpotential problems,but we believe that this is the only approachthat The EV metrics are set up as can be takenby practitioners. to detectin an easy andefficientway (ie earlywarningsignals thana simple at the cost accountlevel, or even higher),rather of the criticalpath-basedschedulingtools. This replacement early warningsignal, if analysedproperly,defines the need to eventually'drill-down'into lowerWBS levels. In conjunction with the projectschedule,it allows takingcorrectiveactions on those activitieswhich arein trouble(especiallythose tasks which are on the criticalpath).As a result,we calculate measures(SPI and SV) on the level of the the performance projectbased on the three key indicators(PV, AC and EV) that are calculatedper reportingperiod as the sum over all the individualactivities(which can be easily done since they are expressedin monetaryunits). The cost performance indicators theirpredictivepower and to forecastthefinal project cost (see next section) have been discussed extensively in literatureand will not be repeated

here. However, in order to trackproject time performance, the

schedule performancemeasuresneed to be translatedfrom monetary units to time units. In literature,three methods havebeen proposedto measurescheduleperformance: PV the method(Anbari,2003) and the ED method(Jacoband Kane, 2004) translatethe well-known SV and SPI indicatorsfrom monetaryunits to time units. The ES method has been recently introduced Lipke (2003a) and calculatestwo alterby native scheduleperformance measures(referredto as SV(t) and SPI(t)) that are directlyexpressedin time units. The PV methodof Anbari(2003) relies on the well-known EV metricsto forecasta project'sduration usingthe following metrics: PVR plannedvalue rate (or plannedaccomplishment rate) =BAC/PD TV time variance =SV/PVR The averagePV per time period, the PVR, is defined as the baseline BAC dividedby the PD. This measurecan be used to translate SV into time units, denotedby the TV. the Jacob and Kane (2004) introduceda new term, ED, as the productof the AD and the SPI. Jacob (2003) and Jacob

1364

Vol. Journal the Operational of Research Society 58, No.10

projectdelay

ahead of schedule

IJ SV(t)

>

SV(t)

ES

AT

Time PV -&- AC -oEz

AT

ES

Time

project. Figure3 The ES metricfor a late (left)andearly(right)

SPI(t) -------SPI -.....
0.0 -

SV

SV(t)

1.

0.0

1.0EL"0.9 . .. ...

Tim

Te-2.0-.
S-4.0

---1.0

-2.0

0.7

i . -.. ..CI) . ,.
. . . .

- 6 .0 -8.0 -

....... i i

" i

... .... ...... ..... ..... .-----,

- 3 .0 > -4.0

0.6 -.

PD

Time

PD

Time

measures. Figure4 The SPIandSV versusSPI(t) andSV(t) performance and Kane (2004) introducedthe ED method as a reliable using the SPI. methodologyfor forecastingduration ED earnedduration =AD*SPI Lipke (2003a) criticized the use of the classic SV and SPI metrics since they give false and unreliabletime forecasts nearthe end of the project.Instead,he provideda time-based measureto overcomethe quirkybehaviour the SV and SPI of indicators.This ES methodrelies on similarprinciplesof the EV method,and uses the concept of ES as follows: Find t such thatEV > PVt and EV < PVt+1
ES = t + (EV - PVt)/(PVt+ - PVt)

the totalPV plannedforthatsametimeperiod,whichis simply calculatedas a linearinterpolation between the time-spanof the time incrementt and t + 1. Figure3 illustrates translation of the EV into the ES metricto clearlyshow whethera project is behind(left) or aheadof (right)schedule. Using the ES concept, two indicatorscan be constructed of which serve as good and reliablealternatives the SV and SPI indicators,as follows: SV(t) SPI(t) schedulevariancewith earnedschedule ES-AD index with earnedschedule scheduleperformance ES/AD

with ES earnedschedule EV earnedvalue at the actualtime PVt plannedvalue at time instancet The cumulativevalue for the ES is found by using the EV to identify in which time incrementt of PV the cost value for EV occurs. ES is then equal to the cumulativetime t to the beginningof that time increment,plus a fraction(EV PVt)/(PVt+1

- PVt) of it. The fraction equals the portion of

EV extendinginto the incompletetime incrementdividedby

Figure 4 clearly displays the unreliablebehaviourof the SV and SPI metricsfor a projectthat finisheslaterthanplanned (PD = 9 weeks while the AD = 12 weeks). The last review periods of the project are unreliablesince both the SV and SPI metricsclearly show an improvingtrend.At the end of the project,both metricsgive a signalthatthe projectfinishes within time (SV = 0 and SPI = 1 at the end of the project), althoughit is 3 weeks late. The SV(t) andSPI(t) metricsgive a correctsignalalong the whole life of the project.The SV(t) equals -3 at the end of the project,which is a reflectionof the 3 weeks delay.

MVanhoucke S Vandevoorde-Earned metrics forecast and value to duration1365 project

Situation PDWR accordingto plan

Forecasting method Anbari(2003) Jacob(2003) Lipke(2003a)

PDWR will follow currentSPI trend

PDWR will follow currentSCI trend

Comments Pastperformance nota goodpredictor future is of of the Problems/opportunities thepastwill not EAC(t)Est performance. EAC(t)EDI EAC(t)pvl and workwill be done affectthefuture, theremaining to accordine plan Pastperformance a goodpredictor future is of performance of the Problems/opportunities pastwill affect EAC(t)E2 (realistic!). EAC(t)PV2 EAC(t)ED2 and workwill be corrected future performance, theremaining fortheobserved efficiencies inefficiences or are for Pastcostand schedule problems goodindicators future are (i.e. management performance costand schedule EAC(t)E3(b) EAC(t)PV3 EAC(t)ED3(a) The cost is inseparable). SCI= SPI* CPI(schedule ratio) oftencalledthecritical ratioindex Planned Earned Earned Value Rate Duration Schedule

and in on formula notappear Jacob does This (a) forecasting (2003)andhasbeenadded Vandevoorde (2003).Itis based Anbari by
Vanhoucke (2006)

and in does It on This formula notappear Lipke(2003a). is based Anbari (b) forecasting (2003)andhasbeenadded Vandevoorde by Vanhoucke (2006)

Vandevoorde Vanhoucke and on situation. PDWR (2006),andbasedon Anbari (Source: Figure5 Theestimated depending theproject (2003))

of Since the introduction the ES conceptby Lipke(2003a), other authors have investigated the potential of the new method in various ways. Henderson(2003) has shown the validity of the ES concepts on a portfolioof six projects.In anotherpaper,he extendedthis novel approach(Henderson, 2004) and used it on a small scale but time criticalinformation technology software developmentproject (Henderson, and (2006) were the firstau2005). Vandevoorde Vanhoucke thorsthatextensivelycomparedthe threemethodsandtested them to a simple one activityprojectand a real-life data set. Moreover,they summarizethe often confusing terminology the used in the ES literature, which will be used throughout currentpaper.In the currentpaper,we wish to elaborateon the ES concept and simulate the three methods on a large (generated)data set containingprojectsof moderatesize. 2.3. EVMforecasting indicators One of the primarytasks of a project manageris making decisions about the future. EVM systems are designed to of follow-upthe performance a projectandto act as a warning signal to takecorrectiveactionsin the (near)future.Forecasting total projectcost and the time to completionis crucialto take correctiveactions when problemsor opportunities arise measuresaretranslated early into (andhence,theperformance warningsignals).EVM metricsaredesignedto forecastthese measures(timeandcost) basedon two important performance the actualperformance to date and the assumptionsabout up In futureperformance. this section,we reviewsome generally and newly developedforecastingmeasuresthatwill accepted be used in our simulationstudy. The generalformulafor predictinga project'sfinal cost is cost At Completion (EAC),as follows: given by the Estimated
EAC = AC + PCWR

with EAC AC PCWR

estimatedcost at completion actualcost plannedcost of work remaining

The general and similar formula for predictinga project's is at total duration given by the estimatedduration completion as follows: (EAC(t)),
EAC(t) = AD + PDWR

with EAC(t) AD PDWR

estimateddurationat completion actualduration planneddurationof work remaining

Note that we use EAC for cost forecasting and we add a t between brackets (ie EAC(t)) for time forecasting. Cost and havebeen widely investigated by performance forecasting and numerousresearchers, is outsidethe scope of this paper. For an overview,we referto Christensen (1993) who reviews differentEACformulasand severalstudiesthatexaminetheir accuracy. In this paper,we compareand validatethe differentmethods to forecasta project'sfinal duration. Note thatEAC(t) is often referredto as the time estimateat completion(TEAC), at the estimateof duration completion(EDAC)or the Independent EDAC.In the current paper,we rely on the terminology and of Vandevoorde Vanhoucke(2006) who comparedthree methods to estimate the PDWR based on researchdone by Anbari(2003), Jacob(2003) andLipke(2003a). Eachmethod has three differentversions to predicta project'sfinal duraand of tion, dependingon the characteristics performance the in the past. Figure5 summarizes nine forecasting the project metricsused in our simulationstudy.Note thatAnbari(2003) maintainsthatthe TEACcan be calculatedbased on the SAC

1366

JournaltheOperational Society 58,No. of Research Vol. 10

or PD, actual performance to any given point AT in the up He project,andanalysisof expectedfutureperformance. specifies that 'whencurrent analysisshows thatpast scheduleperformanceis a good predictor futurescheduleperformance,' of then the TEAC 'is the sum of the cumulativeAT plus the originalscheduledtime for the remainingwork, modifiedby the cumulativeSPI'. The PV methodof Anbari(2003) does not directlygive an estimate for the PDWR, but instead, is based on the TV = SV/PVR= SV*PD/BAC. Anbari(2003) proposesthreedifferent time forecastingmeasuresreflectingthe three situations of Figure5, as follows: EAC(t)pvl = PD - TV when the durationof remaining work is as planned = of EAC(t)PV2 PD/SPI when the duration remaining work follows the currentSPI trend = of EAC(t)PV3 PD/SCIwhen the duration remaining work follows the currentSCI trend Jacob and Kane (2004) use the ED method to forecast the total project duration, as follows: EAC(t) = AD +
unearned remaining duration/ PF = AD + (PD - ED)/PF =

3. Research methodology and test design parameters In this section, we discuss our specific methodologicalapproachto measurethe forecastaccuracyof the differentforecasting measures described earlier. A summarypicture is given in Figure 6, and details are explained in the following sub-sections.In a first step, we generate 3100 diverse projectnetworksundera controlleddesign. More precisely, we varyandcontrolthe topologicalstructure. (Severalsources in literature(eg Herroelenand De Reyck (1999), Demeulemeester et al (2003) and Vanhouckeet al (2004)) refer to 'the exact shape of a network'as its 'topologicalstructure'. Other authors(eg Tavareset al (1999, 2002)) refer to it as the 'morphological of structure'.) each networkas a measure of diversity.This is the topic of Section 3.1. Each network will be transformed an earlieststartGanntchart,based on in the well-knowncritical-path-based forwardpass calculations. The resulting schedule gives the periodicallyPVs for each activity.The projectis then the subjectto uncertainty during projectexecution,which will be simulatedaccordingto the Monte-Carlo principles.More precisely,we run nine different scenariosreflectingall possible outcomes a project can have, as discussed in Section 3.2. The resultingAC and EV metricswill be comparedwith the PV, resultingin the forecastingmeasuresdescribedearlier.The resultsof the forecast in accuracyare reported Section 4. We mentionedbeforethatour uniqueapproach aims at the generationof a wide varietyof diverseprojectsand the simulationof nine uncertainty scenarios,resultingin 27 900 different possible project executions (moreover,each possible executionhas been simulatedfor 100 Monte-Carlo runs). It allows drawingobjectiveand extensive conclusions that are not based on the executionon one or a few real-lifeprojects but can be generalizedto a wide set of possible real-life scenarios,as follows: * The diverse set of networks:the generationof the 3100 projectnetworksis based on and measuredby theirtopothe logical structure(Section 3.1). Consequently, specific of structure each projectnetworkis not linked topological real-lifeproject to anypractical characteristic a particular of but is used as a diversitymeasure. * The nine uncertainty scenarios: simulaterandomvariaWe and tions in bothactivitydurations costs undera controlled distributions design (Section 3.2) by means of triangular tailed to the rightor left. We have not the intentionto link or the generatedvariabilityto practicaluncertainties systematicvariationssuch as changes in client requirements, of re-allocationsof staff, over-ambition technical people, etc. our we Moreover,throughout experiments, assumethat: * Managersdo not change their focus on control.We simulate projectexecutionunderdifferentcontrolledscenarios, actionsinto account. butdo not takeanypossiblecorrective

AD + (PD - AD*SPI)/PF. The performance factor (PF) is used to correct the estimate towardsprevious performance (see Figure5). Indeed,a PF= 1 denotesa futureperformance at the efficiency rate of the originalplan (100% efficiency). However,the future performancecan be correctedtowards the current SPI trend or the current schedule cost index SCI = CPI*SPItrend.Hence, the three forecastingmethods to predicttotal projectdurationare:
EAC(t)ED1= PD + AD*(1 - SPI) EAC(t)ED2= PD/SPI EAC(t)ED3 = AD + (PD - ED)/(CPI*SPI)

In situationswhere the projectdurationexceeds the PD, and the workis not yet completed(ie whenAD > PD andSPI < 1), the PD will be substituted the AD in the above-mentioned by formulas.In these cases, the formulasare
EAC(t)EDI = AD

+ (AD - ED)/1 = AD*(2 - SPI) = AD + (PD - ED)/CPI EAC(t)ED2 EAC(t)ED3= AD + (PD - ED)/(CPI*SPI)

The ES methodof Lipke (2003a) calculates the EAC(t) as AD+ (PD-ES)/PF. AD +unearnedremainingduration/PF= The three forecastingmetrics, based on the scenarios presentedin Figure5, are:
= EAC(t)Esl = EDAC1 AD + (PD - ES) = EDAC2 = AD + (PD - ES)/SPI(t) EAC(t)ES2 EAC(t)ES3 = EDAC3 = AD + (PD - ES)/(CPI*SPI(t))

MVanhoucke S Vandevoorde-Earnedmetrics forecast and value to duration1367 project

Network --generator

network Project
Generate networks project
With a pre-defined structure

Projectschedule
Constructan earliest startschedule (ESS) using forwardcalculations Planned Value (PV)

9 simulation scenarios

PrQiectexecution Monte-Carlo simulation for each activity's durationand cost

Actual Cost (AC) EarnedValue (EV)

Projectmonitoring
Measure the forecasting accuracy at each review period

of Figure6 The methodological approach ourresearch experiment.

Hence, once the projectstarted,it will be executedaccording to the differentexecutionscenarios. * Resources are available at all times and can be passedon to succeeding activities at no cost. This implies that activitiescan be startedearlierand/orlaterthanscheduled, dependingon the performanceof the previous activities. In doing so, we avoid that earliness in activity durations will be cancelled out later in the projectdue to resource inflexibility. 3.1. Diversity of project networks In this section, we describe our generationprocess in order to constructa set of project networksthat differ from each otherin termsof theirtopologicalstructure. Ratherthan drawingconclusions for a (limited) set of real-life projects, we aim at generatinga large set of project networks that spans the full range of complexity (Elmaghrabyand Herroelen, 1980). In doing so, we guaranteethat we have a very large and diverse set of generatednetworks that can and might occur in practice, and hence, our results can be generalized.We rely on the project network generatordeveloped by Vanhouckeet al (2004) to generateactivity-onthe-nodeprojectnetworkswhere the set of nodes represents networkactivities and the set of arcs representsthe technological precedencerelationsbetweenthe activities.These authors have proposeda networkgeneratorthat allows generating networkswith a controlledtopological structure. They have proven that their generatoris able to generate a set of very diverse networksthat differ substantially from each other from a topological structure point-of-view.Moreover, it has been shown in literaturethat the structureof a network heavily influencesthe constructedschedule (Patterson, 1976), the risk for delays (Tavareset al, 1999), the criticality of a network(Tavareset al, 2004) or the computational effort an algorithmneeds to schedulea project(Elmaghraby and Herroelen,1980). In our experiment,we carefully vary

Criticalactivities
0 1
e-

+ 7 SPl(t) > 1 AD > PD 8 SPI(t) < 1 AD > PD

4 SPl(t) > 1 AD < PD SPl(t) > 1 AD = PD 5 SPI(t) > 1 AD < PD SPI(t) = 1 AD = PD

S 2

> " o 0

3 z + SPI(t)<1
AD < PD

6 SPI(t)< 1
AD = PD

9 SPI(t)< 1
AD > PD

scenarios ourcomputational for tests. Figure7 Nine simulation

and control the design and structureof our generatednetworks, resultingin 3100 diversenetworkswith 30 activities. Formoreinformation aboutthe specifictopologicalstructures andthe generation et process,we referto Vanhoucke al (2004) or to www.projectmanagement.ugent.be/rangen.php.conThe structed dataset canbe downloaded fromwww.projectmanage ment.ugent.be/evms.php. 3.2. The nine simulationscenarios Figure 7 displays the nine simulationscenariosof our computationalexperimenton all 3100 generatednetworks,and reads as follows: Critical versus non-critical activities: A distinction has been made between critical and non-criticalactivities.Each (critical and non-critical)activity can have a actual (ie simulated) durationwhich is smaller than (-), equal to (0) or PD. largerthan (+) its corresponding Actual project performance: The actual (ie simulated) projectperformanceat completion is measuredby comparing the actualprojectdurationAD with the PD. Hence, each

1368

Journal theOperational of Research Vol. Society 58, No.10

column reflects a known scheduleconditionas follows: first column: early project completion (AD < PD), second column: on schedule (AD = PD) or last column: late project completion(AD > PD).
Measured project performance: The SPI(t) index is used to calculate the average measured performance, measured as

the average of all SPI(t) values calculatedover all review periodsduringthe entireprojectexecution. As an example,scenario3 measures-on the averageover all reportingperiods-a project delay (SPI(t) < 1), but the projectfinishes earlierthan expected (AD < PD). Scenario8 measuresan averageprojectdelay,which is a correctwarning signal since AD > PD. Our simulationsfollow the design of Figure 7 and allow the their validating threepredictive techniquesandcomparing relativeperformance. SPI(t) indicator, The whichis usedas an of averagepredictor the overallprojectperformance throughout the entireprojectlife cycle, plays a centralrole and might acts as a (good or bad)warningsignalof projectperformance. More precisely, the nine scenarios can be interpretedas follows: Scenario 1: We measure a project ahead of schedule and it's true. Scenario2: We measure a project ahead of schedule and it's true. Scenario 3: We measurea projectdelay but we are ahead of schedule. Scenario4: We measure a project ahead of schedule but thereis none. Scenario5: Everythingaccordingto schedule. Scenario 6: We measurea projectdelay but thereis none. Scenario 7: We measurea projectaheadbut the projectis behind schedule. Scenario 8: We measurea projectdelay and it's true. Scenario 9: We measurea projectdelay and it's true. We would like to stressthatall EV-basedmetricsdiscussed in this paper make no distinctionbetween critical and noncriticalactivitiesandsufferfromthefactthatall activitieshave an equal weight in the total EV calculations.Therefore,we make a distinctionbetween criticaland non-criticalactivities throughoutour simulationruns and test the potential false warning signal EVM might observe in predictingthe final (see Section4.3). Note thatfour scenarios(1, projectduration 2, 8 and 9) give a correctwarningsignal duringthe execution of the project,and four scenarios(3, 4, 6 and 7) give a false at warningsignal. One scenarioassumes no uncertainty all, that is, all PDs equal the ADs. Note that, throughoutour simulations,the randomvariations in both (critical and non-critical)activity durations distributions tailed to the and costs are based on triangular right or left. However,the choice of the specific tails depend on the scenarioto simulate.As an example, scenario 1 assumesactivitiesfinishingearlierthanplanned,whichhas been

distribution simulated meansof triangular tailedto the left by to simulatethe earliness.Scenario9 assumesthatall activities finish late, which has been simulatedby triangular distribution tailed to the right (lateness).All intermediate scenarios containsome activitieswith left-tailedtriangular distributions and some activities with right-tailedtriangular distributions, dependingon the settingsof the scenario.
3.3. The completion stage of work

The accuracyof forecastsdependson the completionstage of the project.Christensen al (1995) has shownthatno indexet based formulaalways outperforms otherswhen forecastall ing the cost of the project.In this section, we have the intentionto performa similarstudy,and measurethe accuracy of index-basedtime forecastsas a functionof the completion stage of the project.Lipke (2003a) has shown that the classic scheduleindicators(SV and SPI) areunreliableas project durationforecastingindicatorssince they show a strangebehaviour over the final third of the project. This problem is overcomeby the ES conceptwhichbehavescorrectlyoverthe complete projecthorizon (see Figure4). In orderto investiof gate the effect of the behaviour SPI andSPI(t) on the forewe have measuredand forecastthe overall castingmeasures, (duration)performancealong the completion stage of the project (expressedin their percentagecompletedEV/BAC). We divide the project horizon in three stages (early, middle and late) and simulate 3, 9 and 3 settings, respectively, as follows:
Percentage completed

1. Early stage 2. Middle stage

3. Late stage

0-20%, 0-30%, 0-40% 20-60%, 30-60%, 40-60%, 20-70%, 30-70%, 40-70%, 20-80%, 30-80%, 40-80%, 60-100%, 70-100%, 80-100%

4. Computational tests and results In this section, we displayextensiveresultsfor all our simulations.This sectionhas been dividedinto threesub-sections, proposedby Covachet al (1981) inspiredby the threecriteria of for evaluatingthe performance EAC methods,that is, accuracy,timelinessand stability. Forecastaccuracy:In Section 4.1, we evaluatethe overall forecast accuracyof the threemethods(PV, ED and ES) for the nine proposedscenariosof Section 3.2. Timeliness: Section 4.2 analysesthe behaviourof the forecasts along the completionstage of the generatedprojects, andhence, measureswhetherthe forecastingmethodsare capable of producingreliableresultsas early as possible in the life of the project. Stability:In Section 4.3, we discuss the influence of the network structureon the forecast accuracy, based on an

MVanhoucke S Vandevoorde-Earned metrics forecast and value to duration1369 project

of for Table1 Theforecast (MAPE) the threemethods the nine scenarios accuracy
PV1 Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5 0.106 0.114 0.067 0.035 0.000 PV2 0.128 0.095 0.080 0.071 0.000 PV3 0.481 0.101 0.254 0.426 0.000 ED] 0.112 0.121 0.066 0.023 0.000 ED2 0.128 0.095 0.080 0.071 0.000 ED3 0.249 0.087 0.175 0.229 0.000 ES1 0.076 0.094 0.055 0.033 0.000 ES2 0.099 0.036 0.064 0.092 0.000 ES3 0.270 0.054 0.164 0.237 0.000

6 Scenario 7 Scenario Scenario 8

Scenario 9

0.024 0.042 0.100

0.061

0.051 0.077 0.090

0.064

0.416 0.409 0.119

0.232

0.021 0.032 0.102

0.064

0.051 0.077 0.090

0.064

0.242 0.222 0.102

0.132

0.019 0.034 0.076

0.046

0.063 0.093 0.031

0.032

0.273 0.227 0.067

0.142

indicatorthat measuresthe closeness of each networkto a serial or parallelnetwork.In doing so, we measurethe robustnessor stabilityof the forecastingmeasuresfor different network structures.Note that we deviate from the original definitionof stability of Covach et al (1981) that refers to the stability of forecasting methods over the different review periods, and not over the different structuresof the network. In orderto evaluatethe forecastingmeasuresand to determine the forecastaccuracyof each technique,we calculate:
The mean percentage error (MPE) as:
T1

RD - EAC(t)rp

T
rp=l

RD

The mean absolute percentage error (MAPE):

T1

RD

- EAC(t)rP I

TE

rp=l

RD

where T is used to referto the total numberof reporting periods over the completeprojecthorizonandEAC(t)rp is used to denote the estimateddurationat completionin reporting periodrp (rp = 1, 2,..., T) (moreprecisely,at each reportdurationforecast ing period, we calculate a corresponding
EAC(t)rp). 4.1. Forecast accuracy for the nine scenarios

In order to test our nine scenarios of Section 3.2, we need to adaptthe uncertainty activitydurations in accordingto the of each scenario.As an example,the simulation specifications of scenario9 is straightforward involves randomnessin and activity durationsresultingin an increaseof the PD of each activity.However,the simulationof scenario3 (ie simulatea project ahead of schedulebut we measurethe opposite) depends heavily on the structureof the network.Indeed, the durationof the critical activities needs to be decreased,but the non-criticalactivities need to be increasedto guarantee that we measurean averageSPI(t) < 1 along the life of the project, althoughthe project finishes ahead of schedule. To obtain this scenario, the durationof the non-criticalactivities needs to be increasedas much as possible within their

activity slack (resulting in SPI(t) < 1). As an example, a projectnetworkconsistingof many serial activitieshas only a few non-criticalactivities, and hence, a carefully selected simulationis stringent. Moreprecisely,only a few criticalactivities will be decreasedto a very small extent (such that AD < PD), while the few non-critical activitiesneed to be increasedas much as possible within their slack (such that the SPI(t) value is, on the average, smaller than 1). Therefore, each scenariohas been simulatedunderstrictconditionsand betweenscenariosis of little value for the hence, comparison MPE and the MAPE. However,we use these two errormeasures to compareand evaluatethe overall forecast accuracy of the methodswithin each scenario.The cost deviationsare assumedto deviatefrom the originalbudget(BAC per activwith the duration deviation.In doing so, we ity) in correlation assumethatthe cost is expressedin per man-hour hence, and deviationsin activity durationhave an immediateeffect on the cost, due to an increaseor a decreasein the total amount of man-hoursto finish the particular activity.Althoughthis reflectsmany real-life situationsin projectmanagement, one can considerother settings where the cost deviationhas another relation(or no relation)to the durationof an activity. However,the focus of this paper is on the predictionof a project'sfinal duration,and not on cost. The SCI and SCI(t) metricsare only used to forecastthe duration,which makes sense only when the cost is correlatedwith durationperformance. Table 1 displays the MAPE for all the scenariosfor the threeproposedmethods(note that we have abbreviated each method, for example, EAC(t)pvl = PV1). The forecasting table revealsthat the ES methodoften outperforms other the but not in all scenarios.The resultscan be summamethods, rized as follows. * Projectsthat finish ahead of schedule (scenarios 1, 2 and both the PV and the ED 3): the ES method outperforms method.This is in line with the exampleproject3 of Vandevoorde and Vanhoucke(2006). In these cases, the ES methodcan be used as a reliableindicatorto detectopportunitiesin the project. * Projects that finish on schedule (scenarios 4, 5 and 6): The ED method outperformsthe other methods (note

1370

JournaltheOperational Society 58,No. of 10 Research Vol.

Table2 TheMAPEfor project of withAD < PD (ahead schedule)

Early 0-20 PV1 PV2 PV3 EDI ED2 ED3 ES1 ES2 ES3 0.89 0.34 0.43 0.94 0.34 0.41 0.91 0.24 0.47 0-30 0.83 0.32 0.42 0.89 0.32 0.37 0.86 0.22 0.44 0-40 0.78 0.31 0.41 0.84 0.31 0.35 0.80 0.20 0.41 20-60 0.64 0.28 0.39 0.72 0.28 0.28 0.67 0.16 0.33 20-70 0.60 0.27 0.39 0.69 0.27 0.25 0.62 0.14 0.31 20-80 0.56 0.27 0.39 0.65 0.27 0.23 0.57 0.13 0.28 30-60 0.60 0.26 0.39 0.69 0.26 0.25 0.63 0.14 0.31 Middle 30-70 0.56 0.26 0.38 0.66 0.26 0.23 0.58 0.13 0.28 30-80 0.52 0.25 0.38 0.62 0.25 0.21 0.53 0.12 0.26 40-60 0.56 0.25 0.38 0.66 0.25 0.23 0.58 0.13 0.28 40-70 0.52 0.24 0.39 0.62 0.24 0.21 0.53 0.12 0.26 40-80 0.47 0.24 0.39 0.58 0.24 0.19 0.48 0.11 0.23 60-100 0.31 0.23 0.39 0.44 0.23 0.14 0.24 0.06 0.12 Late 70-100 0.28 0.23 0.39 0.41 0.23 0.12 0.19 0.05 0.09 80-100 0.25 0.23 0.39 0.40 0.23 0.12 0.13 0.04 0.07

Table3 TheMPEfor project withAD < PD (ahead schedule) of

Early 0-20 0-30 0-40 20-60 20-70 20-80 30-60 Middle 30-70 30-80 40-60 40-70 40-80 60-100 -0.29 -0.14 0.38 -0.35 -0.14 0.04 -0.23 0.00 0.11 Late 70-100 -0.25 -0.14 0.39 -0.31 -0.14 0.01 -0.18 0.00 0.09 80-100 -0.21 -0.14 0.39 -0.27 -0.14 -0.01 -0.13 0.00 0.07

PV1 -0.89 -0.83 -0.78 -0.64 -0.60 -0.56 -0.60 -0.56 -0.52 -0.56 -0.52 -0.47 PV2 -0.16 -0.15 -0.16 -0.14 -0.15 -0.15 -0.14 -0.15 -0.15 -0.14 -0.14 -0.14 0.35 0.37 0.37 0.37 PV3 0.37 0.37 0.38 0.37 0.37 0.37 0.38 0.37 ED1 -0.94 -0.89 -0.84 -0.72 -0.68 -0.64 -0.68 -0.64 -0.60 -0.64 -0.60 -0.55 ED2 -0.16 -0.15 -0.16 -0.14 -0.15 -0.15 -0.14 -0.15 -0.15 -0.14 -0.14 -0.14 ED3 0.32 0.31 0.24 0.21 0.29 0.22 0.17 0.19 0.15 0.19 0.19 0.17 ES1 ES2 ES3 -0.91 -0.86 -0.80 -0.67 -0.62 -0.57 -0.63 -0.58 -0.53 -0.58 -0.53 -0.48 0.05 0.05 0.04 0.03 0.02 0.02 0.02 0.01 0.02 0.01 0.01 0.02 0.44 0.42 0.33 0.40 0.30 0.28 0.28 0.25 0.28 0.26 0.23 0.30

that ED2 and PV2 are exactly the same predictors)for scenarios 4 and 6. However, these scenarios are especially built to generate an SPI(t) indicatorthat gives a false warning signal. Hence, the forecastingmetrics will be influencedby this false indicator,resulting in wrong forecasts.The ED methoddoes not suffer from this error, since the SPI indicatortends to go to 1 at the end of the project,decreasingthe errorof the false warningsignal. * Projects that finish behind schedule (scenarios 7, 8 and the 9): the ES method outperforms other methods,which means it can be used to detect problemsin projects.HowThis, ever,in scenario7, the ED has the best performance. too, is a scenario for which the SPI(t) indicatorgives a false warningsignal. of The table also revealsthatthe performance the forecastmetricsdoes not performvery well when the SCI is used ing as a performance measure.Hence, correctingthe forecasting metricswith cost information (the CPI is used in the denomdoes not lead to reliableresultsfor the threemethods, inator) and shouldbe excluded.Note thatwe have tested the signiftest icance of all differenceswith a non-parametric in SPSS. All differencesas indicatedin the table (the best performing methodhas been indicatedin bold) were statisticallysignificant on the 5% level.

4.2. Forecast accuracy and the completion of work

In this section, we analysethe behaviourof the three schedule forecastingmethods along the completion stage of the projectas discussedin Section 3.3. We divide our computational tests into two sub-tests.In a first simulationrun (see Tables2 and 3), we analysethe MAPEand MPE of the forethatthe projectwill casting techniquesunderthe assumption end soonerthanexpected(AD < PD). In a second simulation run (see Tables4 and 5), we assume that the project is behind schedule, that is, AD > PD. To that purpose,we have influencedthe AD of each activityby randomlygeneratinga with a tail to the left numberfrom a triangular distributions, (aheadof schedule)or to the right(behindschedule). Table2 (MAPE)revealsthatthe ES methodalmostalways all of outperforms othermethods,regardless the stageof comES1 (ES3) shows a slightly worse performance pletion. Only than PV1 (PV3) in the early and middle (early) stages, but performsbetterin the late stages. Moreover,the forecastaccuracy increasesalong the stage of completion,resultingin very low absolutepercentageerrorsin the late stages. Table for 3 shows an overall overestimation all forecastingmethods with a PF = 1 (ie PV1, ED1 and ES 1). This behaviouris intuitivelyclear, since the PF equals 1 and hence, outstandin ing performance the past (SPI > 1 and SPI(t) > 1) will not

MVanhoucke S Vandevoorde-Earned metrics forecast and value to duration1371 project

Table4 The MAPEfor project withAD > PD (project delay)

Early 0-20 PV1 PV2 PV3 ED 1 ED2 ED3 ES 1 ES2 ES3 0.11 0.13 0.32 0.11 0.13 0.31 0.11 0.13 0.33 0-30 0.10 0.13 0.29 0.11 0.13 0.27 0.11 0.12 0.29 0-40 0.10 0.12 0.27 0.10 0.12 0.24 0.10 0.11 0.26 20-60 0.08 0.10 0.19 0.09 0.10 0.15 0.08 0.08 0.17 20-70 0.08 0.09 0.18 0.09 0.09 0.14 0.08 0.08 0.15 20-80 0.08 0.09 0.17 0.08 0.09 0.12 0.07 0.07 0.14 30-60 0.08 0.09 0.17 0.09 0.09 0.13 0.08 0.07 0.14 Middle 30-70 0.08 0.09 0.16 0.08 0.09 0.12 0.07 0.07 0.13 30-80 0.08 0.08 0.15 0.08 0.08 0.11 0.07 0.06 0.11 40-60 0.08 0.09 0.16 0.08 0.09 0.12 0.08 0.07 0.12 40-70 0.07 0.08 0.15 0.08 0.08 0.11 0.07 0.06 0.11 40-80 0.07 0.08 0.14 0.08 0.08 0.09 0.07 0.06 0.10 60-100 0.08 0.08 0.09 0.08 0.08 0.08 0.04 0.03 0.05 Late 70-100 0.08 0.08 0.09 0.08 0.08 0.08 0.03 0.03 0.04 80-100 0.08 0.08 0.08 0.08 0.08 0.08 0.03 0.02 0.03

Table5 TheMPEfor project withAD > PD (project delay)

Early 0-20 0-30 0-40 20-60 20-70 20-80 30-60 Middle 30-70 30-80 40-60 40-70 40-80 60-100 0.07 0.05 -0.07 0.07 0.05 0.03 0.03 0.00 -0.03 Late 70-100 0.07 0.06 -0.06 0.07 0.06 0.04 0.03 0.00 -0.02 80-100 0.08 0.07 -0.04 0.08 0.07 0.06 0.02 0.00 -0.02

0.11 0.10 0.07 0.09 0.07 0.07 0.07 0.06 0.06 PV1 0.06 0.06 0.06 PV2 -0.03 -0.03 -0.02 -0.01 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.01 PV3 -0.26 -0.24 -0.22 -0.16 -0.15 -0.14 -0.14 -0.13 -0.12 -0.14 -0.13 -0.12 ED 1 0.11 0.11 0.10 0.08 0.08 0.07 0.08 0.08 0.07 0.07 0.07 0.07 ED2 -0.03 -0.03 -0.02 -0.01 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.01 ED3 -0.24 -0.22 -0.19 -0.12 -0.10 -0.08 -0.09 -0.08 -0.06 -0.08 -0.07 -0.05 ES 1 ES2 ES3 0.11 0.10 0.07 0.10 0.08 0.07 0.07 0.07 0.06 0.07 0.07 0.06 -0.04 -0.05 -0.04 -0.03 -0.03 -0.02 -0.02 -0.02 -0.01 -0.01 -0.01 -0.01 -0.26 -0.24 -0.22 -0.14 -0.13 -0.11 -0.12 -0.10 -0.09 -0.10 -0.09 -0.08

be accountedfor the estimateof the remainingwork. However, all other methodstake a correctivefactor into account (eitherSPI, SPI(t) or these factorsmultipliedby CPI), in order to predictthe remainingworkbased on the excellentperformance of the past. However,the SPI (used for the PV2 and ED2 methods)fails to incorporate excellent perforthe mancein theirestimatesandhence,leadsto an overestimation along all stages of the project.The use of the SPI(t) metric (ES2), however, shows an excellent and improvingperformance along the completionstages (from the middle stages on, the MPE is almost always close to zero). As mentioned previously,the extracorrectionfactorCPI (PF= SCI) biases the results and leads to over-optimisticresults for all forecastingmethods.Anbari(2003) pointsout thatthe TEACadjusted for cost performance 'may providea betterindication of estimatedtime at completion,when adherenceto budget is critical to the organization'.He points out that additional time may be needed to bring the projectback on budget (by reducingresources applied to the project, taking additional time to findbetterpricesfor equipment materialand simand ilar actions).In our paper,we did not considerthe impactof cost performance the schedule. on Similar observationshave been found, althoughless extreme, when the projectfinishes later than expected. In this factors case, the SPI < 1 and SPI(t) < 1 areused as corrective

to increasethe remainingwork estimatefor the poor performance in the past. However,poor performance results in an SPI < 1 and SPI(t) < 1 in the earlyandmiddlestages,but the SPI tendsto go to 1 at the late stages even thoughthe project is late (and hence, SPI(t) < SPI in the late stages). Since the SPI andSPI(t) metricsareused in the denominator the forof mulasPV2 and ED2, the duration forecastwill be lower than the ES2 forecast (based on the reliable SPI(t) metric).This for explains the underestimation the PV2 and ED2 methods duringthe late stages, and the improvingforecast accuracy behaviourfor the ES methods.
4.3. Influence of the network structure on the forecast accuracy

Jacob and Kane (2004) argue that EV metrics and the correspondingforecastingindicatorscannot be used but on the level of an individualactivity.Indeed,a delay in a non-critical activitymight give a false warningsignal to the projectmanager,andhence, wrongcorrectiveactionscan be taken.However, we mentionedearlierthat we completely ignored this remarkin our specific simulationapproachdue to practical reasons (the projectmanageris usually interestedin the status of the overall projectand has no time to calculateevery metricon the activitylevel) andcalculatedthese performance

1372

Journal the Operational of Research Vol. Society 58, No.10

0.80.70.60.50.8 0.7 0.6 0.5-

0.8 0.70.60.50.4040.30 0.2 o0 .0.3-0 - -*

~'7---

0.3
.3--

0.1.... 1 . .................. 0.15 0.1 40

0.2

0.10-

0.10.1

090.06

o3o19-03
0.-0.050.05 -0.10.10.2 01 0.1 3 0.1 0.2 0.3 3 0.40.5 0.5 0.6 0.7 - 0.15 -0.3 -0.20.1 0.8 0.9

0.1 ...-.3 0..2.-1o -0.0-0.4

-0.3 --0.1 --0.4.25..-0.6.."--0----25--0.1

0.3
-0.5 ..2 -0.7-

-0.158 O
-0.2 -0.15 -0.3 0.35-

0.4 0.0.

-0.30.8 -0.9-

........
-0.054

-0.6

-E

E3

~-PV1

~-ED --

PV2

-?a-PV3

~-ED2

E--D3

--4~--ES1

E2--:-ES

Figure 8 The influence (MPE) of the serial or parallel networksfor the nine scenarios.

measures on the project level, and not on the level of each individual activity. The possible bias of our approach (project level) compared to the ideal approach (on the activity level) is influenced by the structure of the network, and more precisely by the number of critical activities in the networks. Therefore, we have calculated for each network from our data set how close it is to a serial or parallel network. We measure this closeness by a serial/parallel indicator, SP e [0, 1]. More precisely, when SP = 0 then all activities are in parallel, and when SP = 1 then we have a complete serial network. Between these two extreme values, networks can be closer to a serial or to parallel network. This allows us to investigate the influence of the closeness of each network to a serial or parallel network (SP) on the accuracy (measured by the MPE) of the forecasting measures. Since the SP indicator is directly linked with the number of possible critical activities in a network (the closer the SP to 1, the more critical activities in the network), this indicator might serve as an aid to detect in which cases our project level approach (compared to the activity level approach of Jacob and Kane (2004)) leads to wrong project duration forecasts. Figure 8 displays the MPE for the networks with varying values for the SP indicator, ranging from 0.1 (close to a parallel network) to 0.9 (close to a serial network), in steps of 0.1. These graphs are ranked in a similar way as the nine scenarios of Figure 7 and reads as follows: the first graph is

scenario 1, the second graph is scenario 4, the third graph is scenario 7, the fourth graph on the second line is scenario 2, etc. MPE-values larger (lower) than zero give an indication of an under-estimation (over-estimation) of the forecasting metrics for the AD. The results of Figure 8 can be summarized as follows. First, the tables reveal that the network structure clearly influences the forecast accuracy. Indeed, almost all graphs show an improving forecast performance (closer to zero) for all forecasting methods for increasing values of SP (ie more serial networks). The main reason is that the number of non-critical activities decreases for increasing SP-values, and hence, the probability to make wrong conclusions decreases (delays in non-critical activities were the cause of mis-interpretations as shown by Jacob and Kane (2004)). Only the graphs for scenarios 2 and 8 show a deteriorating trend for increasing values of SP. Again, this observation confirms the possible mis-interpretations that can be made due to effects in noncritical activities. Both scenarios have no change (neither a delay, nor a duration decrease) in the non-critical activities (see Figure 7), and hence, the aforementioned mis-interpretations are completely excluded. Secondly, most graphs reveal that the SPI or SPI(t) indicator is an important factor of the forecasting formulas and hence, might give a false warning signal. As an example, the graphs for scenarios 1, 4 and 7 all show an underestimation for the final project duration. This

MVanhoucke S Vandevoorde-Earnedmetrics forecast and value duration1373 to project

is the resultof the SPI(t) > 1 which measuresa projectahead of schedule.However,the projectof scenario4 (scenario7) finisheson time (laterthanexpected),which explainsthe unfor derestimations all the forecastingmethods.A similar,but opposite reasoningholds for scenarios 3 and 6 which both measurea projectahead (SPI(t) < 1) but have a finalproject of delay AD < PD), resultingin a clear over-estimation the duration. almostall scenariosrevealthatthe exLast, project's tracorrection factorSCI to forecasta project'sduration leads to a low forecastaccuracy. an example,the PV3, ED3 and As ES3 metricsclearly have a very low accuracyfor scenarios 1, 3, 4, 6 and 7, when SP valuesare low. Note thatscenario5 is an ideal scenario,with no deviationswhatsoever, resulting in a 0% MPE value. 5. Conclusions In this paper,we havetestedthe forecastaccuracyof threedifferentprojectdurationforecastingmethods,the PV method, the ED methodand the ES method,based on extensive simulationson a large set of generatednetworks.First, we controlled the topological structureof the generatednetworks based on existing and reliablenetworkgenerator, orderto in make the link between the projectnetworkand the forecast into accuracy. Secondly,we splitup the networkstructure critical and non-criticalactivitiesto measuretheir influenceon the forecastaccuracy. Last, we have influencedthe behaviour of the SPI(t) indicatoras a corrector false warningsignal, and hence, the type of uncertainty(risk) has been carefully controlled,resultingin nine differenttest scenarios.The reon sults reveal thatthe ES methodoutperforms, the average, all otherforecastingmethods.The closeness of a networkto a serialor parallelnetworkdirectlyinfluencesthe activityslack and has an impacton the accuracyof the forecasts. This researchis highly relevantto both academiciansand practitioners.From an academic point-of-view, it is interon esting to measurethe influence of the networkstructure the behaviourof both schedulingand monitoringtools, and to hence, this researchserves as a call to researchers focus their furtherresearchattentiontowardsspecific problem or project instances. Indeed, ratherthan developing tools and techniquesfor generalproblemsthat have an average excellent behaviour,one can develop an excellent method for a
set of specific problem instances, which belongs to a cer-

for providesa framework futureresearch purposesandhence, can be used as a guide for our futureresearch.Ourfutureresearch ideas are threefold.Firstly,we want to continue our researchin orderto improvethe forecastaccuracyof the EV metrics.More precisely,the ES methodcan be extendedand refinedby takingnetworkspecific information into account, in orderto improvethe early-warnings potentialof the metrics. Secondly,we want to use network-specific information to select a priori the best forecastingmethodfor the particular projectin execution.Based on the topological network structure simplerules-of-thumb, and softwareshouldbe able to calculate the project durationforecast and give an indication aboutthe reliabilityof the forecast.Last, we want to extendour simulationframework investigatethe best set and of correctiveactions and the requiredperformance neededto the execute these actions, given the networkstructure, forecasting method and the values for the EV metrics.In doing so, the projectmanagergains insightinto the differentpossible correctiveactions and the link with the project,the netreferencesfor this work,the selectedmethod,etc. Interesting researchtrackcan be foundin Lipke(2003b, 2004) andCioffi of (2006). Anbari(2003) highlightsthe importance such corrective and preventiveactions and points out that such forecasts help 'focus management's intereston projectsor work that need the most attention'. packages

References
AnbariFT (2003). Earnedvalue projectmanagement methodsand extensions.ProjectMngtJ 34: 12-23. A Christensen (1993).The estimateat completion DS problem: review of threestudies.ProjectMngtJ 24: 37-42. Christensen AntoliniRC andMcKinney (1995). A reviewof JW DS, EAC research. CostAnal Mngt(Spring):41-62. J CioffiDF (2006). Completing to projects according plans:An earnedvalue improvement index.J Opl Res Soc 57: 290-295. real CooperKG (2003). Yourproject's pricetag-letters to the editor. HarvardBusinessReview81: 122. CovachJ, HaydonJJ and RietherRO (1981). A Studyto Determine Indicators Methods Compute and to Estimate Completion at (EAC). ManTech International Corporation: Virginia. M Demeulemeester Vanhoucke and Herroelen (2003). RanGen: W E, A randomnetworkgenerator activity-on-the-node for networks. J 6: Scheduling 13-34. SE of Elmaghraby and HerroelenW (1980). On the measurement Eur J Opl Res 5: 223-234. complexityin activitynetworks. J real price FlemingQ and Koppelman (2003). What'syourproject's tag? HarvardBusinessReview81: 20-21. Fleming Q and Koppelman J (2005). Earned Value Project Management,3rd ed. Project ManagementInstitute:Newtowns Square,PA. Henderson (2003). Earnedschedule:A breakthrough K extensionto earnedvalue theory?A retrospective analysisof real projectdata. TheMeasurable News (Summer):13-17, 21-23. Henderson (2004). Further K in developments earnedschedule.The News (Spring):15-16,20-22. Measurable HendersonK (2005). Earned schedule in action. The Measurable News (Spring):23-28,30.

tain class or set of networkstructures. This researcharea is closely relatedto the phase transitionresearchattentionthat has been describedin many researchpapers(see eg Herroelen and De Reyck (1999) who describethe conceptof phase transitionsfrom a projectschedulingpoint-of-view).The research to evaluatethe forecast accuracyof project duration forecastingmethods is, to the best of our knowledge, completely void. Hence, this study can be used by practitioners who use these metrics and principleson a daily basis, but are unawareof the merits and the pitfalls of each individual method. Moreover,we believe that this researchpaper also

1374

Research Vol. Journal the Operational of Society 58, No.10

in HerroelenW and De Reyck B (1999). Phase transitions project

scheduling. J Opl Res Soc 50: 148-156.

JacobD (2003). Forecasting with earned projectschedulecompletion value metrics.TheMeasurable News (March):1, 7-9. schedulecompletion JacobDS andKaneM (2004). Forecasting using earned valuemetrics.., revisited.TheMeasurable News (Summer): 1, 11-17. Lipke W (2003a). Schedule is different. The MeasurableNews 31-34. (Summer): 12: LipkeW (2003b). Decidingto act. Crosstalk 22-24. Lipke W (2004). Connectingearned value to the schedule. The
Measurable News (Winter):1, 6-16.

JA Tavares Ferreira andCoelhoJS (1999). The risk of delay of a LV, of projectin termsof the morphology its network.Eur J Opl Res 119: 510-537. TavaresLV, FerreiraJA and Coelho JS (2002). A comparative sets Int analysisof benchmark of projectnetworks. J morphologic
Project Mngt 20: 475-485.

Loch CH, DeMeyerA andPich MT (2006). Managingthe Unknown:

A New Approach to Managing High Uncertainty and Risk in

Project.JohnWiley and Sons, Inc.: New Jersey. Patterson JH (1976). Project scheduling: The effects of problem structureon heuristic scheduling. Nal Res Log 23: 95-123.

indicator Tavares Ferreira and Coelho JS (2004). A surrogate JA LV, of criticalityfor stochasticnetworks.Int TransacOpl Res 11: 193-202. M of Vandevoorde and Vanhoucke (2006). A comparison different S projectduration forecastingmethodsusing earnedvalue metrics. Int J ProjectMngt,24: 289-302. LV Vanhoucke Coelho JS, Tavares and Debels D (2004). On the M, of Paper04/272, Ghent topologicalstructure a network.Working University.

ReceivedJuly 2005; acceptedJuly 2006