A Review of the Relationship Between Needs Analysis and Evaluation in Training Research, EAC 580,

A Review of the Relationship Between Needs Analysis and Evaluation in Training Research

Edward Taylor Francis EAC 580 November 17, 2009

!rancis

A Review of the Relationship Between Needs Analysis and Evaluation in Training Research, EAC 580, " There are nu#erous standard for#s of evaluation that #ost people, especially those in $estern countries and culture, are used to receiving% They #ay range fro# letter grades li&e the standard A to F grading scale used in school syste#s to nu#erical for#s of evaluation such as the 0' 00 range seen in education to #ore ne(ulous for#s of evaluation ) gold stars, s#iley faces, red Xs, and other things of this il&, *ince #ost, if not all, individuals in $estern societies are or have (een su(+ect to these assess#ents throughout their life, it would (e erroneous to say there is no understanding of the concept of evaluation a#ong the general populous% ,owever, this type of evaluation ) one that analy-es the overall #astery of an ite# or effectiveness of a course ./orrison, Ross, 0 1e#p, "002, p% "345 ) is &nown as su##ative evaluation and is only a portion of the co#plete pu--le% $hen the ter# summative evaluation was first coined it was acco#panied (y its close relative formative evaluation% 6t is this latter type of evaluation, one that atte#pts to find any strengths or deficiencies in a training progra# and then #odify the# to (etter the product ./orrison et al%, "002, p% "375, which is #ost often unrecogni-ed (y the average individual% $hile (oth types of evaluation are 8uite different in their end result, they (oth can provide valua(le infor#ation to any training course% 6n fact, so#e studies .9eathwood 0 :hillips, "000; $holey, 4475 ac&nowledge that results are #ost infor#ative when (oth for#s of evaluation, su##ative and for#ative, are used% :rior to the co#pletion of any assess#ent, however, one #ust &now what he<she is evaluating% This is where the concept of a needs analysis (eco#es i#portant% A needs analysis atte#pts to gather data to deter#ine what specific areas of a particular +o( or tas& are #ost vital, identify any gaps in perfor#ance in these areas, and conduct (oth for#ative and su##ative evaluation in order to !rancis

A Review of the Relationship Between Needs Analysis and Evaluation in Training Research, EAC 580, 3 correct these gaps ./orrison et al%, "002, p% 3"5% 6n the end, the process see#s fairly si#ple and straightforward= conduct a needs analysis and then run su##ative and for#ative evaluations to deter#ine, in the for#er case, if the training was indeed effective at #eeting its intended goals and, in the latter, how the training could (e #ade (etter% ,owever, when one loo&s at the so#ewhat li#ited research in the area, it (eco#es apparent that there is often a li#ited to non'e>istent connection (etween needs analysis and evaluation and #any evaluation practices are lac&ing in thoroughness and applica(ility%

A ?iscussion of the Analysis 9iterature

6t is perhaps prudent to (egin (y focusing on the initial portion of this process ) the needs analysis% $hile /orrison et al% ."002, p% 3"5 discuss needs analysis as a very specific and confined ter#, it is not always so narrowly defined in the literature% /c1illip ."00 5 references the ter# job analysis while /athews, @eno, 1e&ale, Rep&a, :ereira, and *ilva ."00 5 discuss a training needs assess#ent, (ut in all three instances, although the words are different, the overall idea is the si#ilar% Needs analysis is a critical co#ponent of any training progra#% Aould, 1elly, and $hite ."00B5 clai#, CTNA DTraining Needs AnalysisE plays a crucial role in ensuring that service needs are #et% 6f needs assess#ent is #isinfor#edFany su(se8uent training is unli&ely to reflect the needs of the organi-ation or its staff .p%335%G ,owever, it is interesting to note that the authors #ention in the sa#e article that when they conducted a search ranging 8 years fro# 485 to "003, they found "77 articles directly relating to TNA, only "3 of which reported the outco#es of the analysis .Aould et al%, "00B, p% 33'3B5% 9ess than 0H of articles relating to needs analysis actually provided usa(le infor#ation in ter#s of how the outco#es were !rancis

A Review of the Relationship Between Needs Analysis and Evaluation in Training Research, EAC 580, B i#pacted (y any type of needs assess#ent% *o, here one sees a topic that is not covered nearly as often nor as in'depth as the evaluation phase of the training develop#ent cycle and, further#ore, is rarely ta&en to its full conclusion when it is researched% 6nterestingly, even though Aould et al% ."00B5 point out this large discrepancy in the literature on needs analysis, the sa#e issue #anifests itself in their pu(lication% Their discussion of needs analysis centers essentially around a theoretical process% That is, they discuss how one should conduct a needs analysis, list and descri(e so#e of the &ey ideas or topics that should (e considered, (ut never delve into specific e>a#ples or provide solid evidentiary support for their #odel% Their clai# that C#uch has (een written a(out how it DTNAE should (e conducted .Aould et al%, "00B, p% 375,G (ut Cthere is a dearth of literature concerning approaches to criti8ue TNA .Aould et al%, "00B, p% 375G see#s ironic (ecause their article falls directly into that sa#e sa#ple% $hether due to the fact that Cgathering (aseline data is not always possi(le or cost'effectiveG as /orrison et al% ."002, p% 3"5 state or for other reasons, needs analysis appears to (e rarely treated as #ore than a theoretical concept% *o#ething to discuss ad nauseum in ter#s of its i#portance and proper application, (ut rarely actually practiced% Nevertheless, there is a valua(le notion in the a(ove state#ent concerning a lac& of criti8ues of needs analysis% $hen researching the literature, one finds few studies where needs analysis and evaluation are (oth directly discussed% The article either focuses on the concept of needs analysis or the idea of perfor#ing an evaluation on a topic with an assu#ed or generic set of needs% There are so#e e>ceptions, with one nota(le e>a#ple provided (y /c1illip ."00 5% Even /c1illipIs offering is not perfect as it centers on the concept of a +o( analysis vice needs analysis ) a concept si#ilar to (oth a needs analysis and a tas& analysis, (ut not !rancis

A Review of the Relationship Between Needs Analysis and Evaluation in Training Research, EAC 580, 5 a(solutely reflecting either one% ,e defines a +o( analysis as having the pri#ary purpose of identifying Cthe wor& (ehaviors and associated tas&s that are re8uired for successful +o( perfor#ance .,arvey, 440 as cited in /c1illip, "00 , p%"835 and states that once these tas&s are &nown Ctraining progra#s Dcan (eE developed ./c1illip, "00 , p%"835% 6n a "00 study, /c1illip conducted two case studies ) one concerning infor#ation technology certification and another covering li(rarian preparation ) that evaluated whether or not the re8uired training for professionals in those fields was suita(le% $hat he lac&ed in ter#s of an atte#pt to precisely conduct a needs analysis per se, he #a&es up for with a valid effort to evaluate whether or not training progra#s ade8uately cover the ite#s analy-ed as (eing i#portant for a particular +o(% 6nstead of actually conducting a thorough and co#plete needs analysis, a sa#ple group of individuals within (oth of the arenas ) Can e>pert group of /C*Es D/icrosoft Certified *yste#s EngineerE ./c1illip, "00 , p% "855G and Ca professional association of li(rarians ./c1illip, "00 , p% "875G ) were used to deter#ine which tas&s were pertinent to the career fields under study% A larger group of people within the career field were then surveyed on the specific areas chosen and rated each tas& on a five'point scale fro# . 5 Not i#portant to .55 E>tre#ely i#portant% The results of his research were plotted into a two'(y'two 6#portance':erfor#ance #atri> where Cthe J'a>is displays significance for +o( success and the K'a>is displays training e#phasis or need ./c1illip, "00 , :%"855%G The 6#portance':erfor#ance #atri> then produced four 8uadrants where, theoretically, a proper training progra# would produce a diagonal line fro# lower left to upper right when plotted on the #atri>% 6te#s of low significance should (e in the lower left indicating they receive little attention in training progra#s and those that are highly i#portant should appear in the upper right garnering a greater !rancis

A Review of the Relationship Between Needs Analysis and Evaluation in Training Research, EAC 580, 7 focus during training% !or /c1illipIs study, a scatter diagra# was used to display the results for (oth careers and the /C*E course #atched very well with its theoretical counterpart while the li(rarian course failed to #eet the desired standards% @nfortunately, the author does not delve into reasons for the discrepancy si#ply concluding, C6f the professional association were to launch a certification, these areas Dof potential needE should (e e>plored ./c1illip, "00 , p%"825%G 6t could si#ply (e that li(rarians are overstating the i#portance of certain areas in their field causing certain ite#s that are covered in training to (e considered Ccrucial co#ponentsG while those that arenIt covered auto#atically (eco#e Cpotential needs%G Regardless, despite the i#perfections of the study including a li#ited +o( analysis and surface'level evaluation, he at least atte#pts to find a relationship (etween needs and evaluation, which is rare in #ost research focusing on these topics%

A ?iscussion of the Evaluation 9iterature

A review of the literature on evaluation reveals two central pro(le#s, the first si#ilar to, (ut ulti#ately different fro#, that encountered (y needs analysis with the second one (eing uni8ue to the study of evaluation% Evaluation suffers fro# a si#ilar issue to needs analysis in that research rarely atte#pts to correlate the two concepts% ,owever, it is so#ewhat different (ecause it is possi(le, and indeed possi(ly easier, to discuss needs analysis as its own entity while an evaluation study re8uires so#ething to evaluate% Thus, there is al#ost always a Cneed,G typically identified as an o(+ective in evaluation'(ased articles, whether or not a needs analysis is perfor#ed (eforehand% The other #a+or differential is that, although there #ay (e varying definitions of a needs analysis, it is ulti#ately a single !rancis

A Review of the Relationship Between Needs Analysis and Evaluation in Training Research, EAC 580, 2 identifia(le idea whereas evaluation #ust always (e considered as (eing either for#ative or su##ative in nature% Thus, an author #ust decide whether his research will focus on one singular type of evaluation or atte#pt to enco#pass (oth in its scope% 6n the end, evaluation research is not li#ited in the sa#e #anner as needs analysis, (ut it still faces its own li#itations as the literature shows% Lne of the reasons that evaluation researchers face such a dile##a is the vast difference (etween for#ative and su##ative evaluation% ,agstro# ."0075 points out nu#erous differences (etween the two including that Csu##ative assess#ent isFrelatively straightforwardFnot open to interpretation, and can (e (roadly applied .p%"B5G while Cfor#ative assess#ent has fluctuated with changes in educational #ove#ents .p%"B5FG !urther, Csu##ative assess#entFcould have personally punitive effects .,agstro#, "007, p%"55G in contrast to for#ative assess#ent which Ccould lead to positive personal results .,agstro#, "007, p%"55%G $ininger and Nor#an ."0055 su##ari-e the differences efficiently in descri(ing su##ative assess#ent as a ther#o#eter, that is, it provides factual infor#ation, whereas for#ative assess#ent is represented (y a ther#ostat since its chief function is to help #aintain a desired level of achieve#ent% The inherent discrepancy (etween these two types of evaluation re8uires either Can a#(itious progra##e DsicE of curriculu# evaluation researchFincluding (oth for#ative and su##ative ele#ents .9eathwood 0 :hillips, "000, p%3 B5G or Creaching agree#ent on the set of perfor#ance di#ensions that are i#portant to capture and the 8uantitative or 8ualitative perfor#ance indicators that (est represent those di#ensions .$holey, 447, p% B75% /a&ing this decision is, according to $holey . 4475, the #ost difficult aspect of the process% $hat generally results are studies

!rancis

A Review of the Relationship Between Needs Analysis and Evaluation in Training Research, EAC 580, 8 that focus on either one type of evaluation or the other or provide a less'detailed analysis of (oth styles% Lnce a researcher deter#ines how he is going to approach the pro(le#, he #ust then focus on developing the o(+ectives .i%e% needs5 for his study% ,ere one finds the perfect opportunity for a needs analysis, rightM @nfortunately, that is often not the case% /athews, @eno, 1e&ale, Rep&a, 9ope- :ereira, and *ilva ."00 5 found in a cross'cultural study that, out of ten potential areas surveyed, *enior /anage#ent ?ecision and *upervisorIs Lpinions were the top two #ost influential factors in deter#ining the training needs of a corporation while Training Audits ) a needs analysis ) ran&ed either seventh or eighth depending on the country% authors clai#, CTraining auditsFis perhaps rather lower than would (e ideal ./athews et al%, "00 , p% B875FG This s&ewed influence, in turn, i#pacts training decisions within an entity% 6t affords the opportunity Cfor training needs analysis to (e overly su(+ective and with the possi(ility of coercion ./athews et al%, "00 , p%B845%G 6n this instance, the results of any evaluation are su(+ect to the whi#s of senior #anage#ent and, thus, can (e li#ited in their overall effectiveness% Even in instances where the needs are &nown and evaluation conducted, the disparity (etween for#ative and su##ative evaluation rears its head in the research% ,aw&ins, !awcett, Carroll, and Ailliland ."0075 conducted a study focusing on an intervention progra# designed to help couples transition to parenthood% $hile their study found an e>cellent response in ter#s of for#ative evaluation, it found no difference (etween the treat#ent groups and control group in ter#s of a su##ative evaluation of the progra#% 6n this instance, the progra# in and of itself was designed effectively, if not perfectly, (ut still did not achieve its desired o(+ectives% *ituations li&e this de#and further research to deter#ine why !rancis As the

A Review of the Relationship Between Needs Analysis and Evaluation in Training Research, EAC 580, 4 the progra# was ineffective in its function while (eing effective in its design% Lne potential issue encountered (y ,aw&ins et al% is one that see#s to pervade the for#ative evaluation culture% /uch for#ative evaulation Cis conducted at the Nreactions level,I usually in the post'training euphoria ./athews et al%, "00 , p% B845%G $hat this approach to evaluation affects is spurious results that donIt necessarily provide useful data with which to i#prove oneIs progra#% 6ndividualIs reactions are not always the (est source of data when atte#pting to understand deficiencies within a course and when coupled with providing the survey directly at the end of the course the reactions are even less valid (ecause the trainees havenIt had a legiti#ate chance to utili-e their new s&ills and test their application and usefulness% As an added layer of difficulty to the process, once needs are deter#ined and a type of evaluation is focused upon, the #ethods of data collecting can have an enor#ous i#pact on the validity and applica(ility of the data collected%

A !inal Analysis

6n conclusion, the se8uence appears straightforward ) perfor# a needs analysis, deter#ine which type of evaluation is pertinent to the study .ideally (oth for#ative and su##ative evaluation would (e included5, acco#plish the research, and present the findings% 6n reality, the process is #uch #ore difficult% Lutside influences, ti#e, financing, and other factors influence what is studied and how deeply the research loo&s% Needs analyses re8uire a lot of ti#e fro# (oth the researcher and, even when co#plete, si#ply for# a (asis to conduct further research costing a further invest#ent of #oney and #an'hours% As for evaluation, history indicates that C#any co#panies re#ain (lissfully unaware of how #uch they !rancis

A Review of the Relationship Between Needs Analysis and Evaluation in Training Research, EAC 580, 0 spend on training, never #ind whether it was effective or not ./athews et al%, "00 , p%B845G and that evaluation studies are often disregarded (y co#panies that have financed the# .$holey, 4475% The co#ple>ity of the scenario si#ply caused (y the natural differences (etween for#ative and su##ative evaluation create several pro(le#s in that arena alone% $hen co#(ined with the afore#entiond factors, what one finds is a li#ited connection (etween needs analysis and evaluation within the research% Needs analyses are often written a(out theoretically with little su(stantive research or evaluation provided% As it relates to evaluation studies, the needs are often either assu#ed (y the researcher or dictated (y those sta&eholders in positions of authority rarely with use of a co#plete needs analysis% Theory indicates a precise path fro# (eginning to end that provides the #ost detailed and co#plete infor#ation to (etter the training progra#s within a given institution% ,owever, practice and theory donIt always neatly coincide% 6n this instance, the relationship (etween needs analysis and evaluation so neatly espoused in theory is rarely applied in training research%

!rancis

A Review of the Relationship Between Needs Analysis and Evaluation in Training Research, EAC 580,

References
Aould, ?%, 1elly, ?%, 0 $hite, 6% ."00B5% Training Needs Analysis= An Evaluation !ra#ewor&% Nursing Standard, 18(20), 33'37% ,agstro#, !% ."0075% !or#ative 9earning and Assess#ent% Communi ation !isorders "uarterly#28(1)# "B'37% ,aw&ins, A% O%, !awcett, E% B%, Carroll, O% *%, 0 Ailliland, T% T% ."0075% The /arriage /o#ents :rogra# for Couples Transitioning to :arenthood= ?ivergent Conclusions !ro# !or#ative and Lutco#e Evaluation ?ata% $ournal of Family %sy &ology, 20('), 57 '520%

9eathwood, C%, 0 :hillips, ?% ."0005% ?eveloping Curriculu# Evaluation Research in ,igher Education= :rocess, :olitics, and :racticalities% (ig&er )du ation, '0(*), 3 3'330% /athews, B% :%, @eno, A%, 1e&ale, T%, Rep&a, /%, 9ope- :ereira, P%, 0 *ilva, A% ."00 5% Quality Training= Needs and Evaluation'findings fro# a European *urvey% +otal "uality ,anagement, 12('), B83'B40% /c1illip, O% ."00 5% Case *tudies in Oo( Analysis and Training Evaluation% -nternational $ournal of +raining and !evelo.ment# /(') , "83'"84% /orrison, A% R%, Ross, *% /%, 0 1e#p, O% E% ."0025% !esigning )ffe tive -nstru tion 0 /t& )d1 ,o(o&en, NO= Oohn $iley 0 *ons, 6nc% $holey, O% *% . 4475% !or#ative and *u##ative Evaluation= Related 6ssues in :erfor#ance /easure#ent% )valuation %ra ti e, 12(2), B5' B4% $ininger, *% R%, 0 Nor#an, A% ?% ."0055% Teacher CandidatesR E>posure to !or#ative Assess#ent in Educational :sychology Te>t(oo&s= A Content Analysis% )du ational Assessment, 10(1), 4'32%

!rancis