ABSTRACT

Evaluation research can be defined as a type of study that uses standard social research
methods for evaluative purposes, as a specific research methodology, and as an assessment
process that employs special techniques unique to the evaluation of social programs. After the
reasons for conducting evaluation research are discussed, the general principles and types are
revieed. Several evaluation methods are then presented, including input measurement,
output! performance measurement, impact!outcomes assessment, service quality assessment,
process evaluation, benchmar"ing, standards, quantitative methods, qualitative methods, cost
analysis, organi#ational effectiveness, program evaluation methods, and $%S&centered methods.
'ther aspects of evaluation research considered are the steps of planning and conducting an
evaluation study and the measurement process, including the gathering of statistics and the use
of data collection techniques. The process of data analysis and the evaluation report are also
given attention. %t is concluded that evaluation research should be a rigorous, systematic
process that involves collecting data about organi#ations, processes, programs, services,
and!or resources. Evaluation research should enhance "noledge and decision ma"ing and
lead to practical applications.
Ads by (oogle )easure consumer behavior
*ree case study on consumer behavior research
.noldus.com!consumer&behavior
+,th -lanet .)A- congress
$earn about Bead&based )ultiple.ing )ore data in less time/ Come visit0
planet.map.com
12AT %S E3A$4AT%'5 RESEARC2/
Evaluation research is not easily defined. There is not even unanimity regarding its name6 it is
referred to as evaluation research and evaluative research. Some individuals consider
evaluation research to be a specific research method6 others focus on special techniques
unique, more often than not, to program evaluation6 and yet others vie it as a research
activity that employs standard research methods for evaluative purposes. Consistent ith the
last perspective, Childers concludes, 7The differences beteen evaluative research and other
research center on the orientation of the research and not on the methods employed7 8+9:9, p.
;<+=. 1hen evaluation research is treated as a research method, it is li"ely to be seen as a
type of applied or action research, not as basic or theoretical research.
1eiss, in her standard te.tboo", defines evaluation as 7the systematic assessment of the
operation and!or the outcomes of a program or policy, compared to a set of e.plicit or implicit
standards, as a means of contributing to the improvement of the program or policy7 8+99:, p.
>6 emphasis in original=. 1hile certainly not incorrect, this definition, at least ithin a library and
information 8$%S= conte.t, is too narro or limited. 1allace and 3an *leet, for e.ample, point
out that 7evaluation has to do ith understanding library systems7 8;,,+, p. +=. As ill be noted
later in this article, evaluative methods are used for everything from evaluating library
collections to reference transactions.
12? E3A$4ATE/
But before e.amining the specific techniques and methods used in $%S evaluation research, let
us first briefly consider the question of hy evaluation is important and then identify the
desirable characteristics of evaluation, the steps involved in planning an evaluation study, and
the general approaches to evaluation. 1ith regard to the initial question, 1allace and 3an
*leet 8;,,+, pp. ..&..i= and others have noted that there are a groing number of reasons hy
it is important for librarians and other information professionals to evaluate their organi#ations@
operations, resources, and services. Among those reasons are the need for organi#ations to
+. account for ho they use their limited resources
;. e.plain hat they do
A. enhance their visibility
>. describe their impact
<. increase efficiency
B. avoid errors
C. support planning activities
:. e.press concern for their public
9. support decision ma"ing
+,. strengthen their political position.
%n addition to some of the reasons listed above, 1eiss 8+99:, pp. ;,&;:= identifies several
other purposes for evaluating programs and policies. They include the folloingD
+. Eetermining ho clients are faring
;. -roviding legitimacy for decisions
A. *ulfilling grant requirements
>. )a"ing midcourse corrections in programs
<. )a"ing decisions to continue or culminate programs
B. Testing ne ideas
C. Choosing the best alternatives
:. Recording program history
9. -roviding feedbac" to staff
+,. 2ighlighting goals
7'ver the past decade, both academics and practitioners in the field of library and information
science 8$%S= have increasingly recogni#ed the significance of assessing library services7 8Shi
F $evy, ;,,<, p. ;BB=. %n August ;,,> the 5ational Commission on $ibraries and %nformation
Science announced three strategic goals to guide its or" in the immediate future. Among
those three goals as the appraising and assessing of library and information services.
C2ARACTER%ST%CS A5E -R%5C%-$ES '* E3A$4AT%'5
Childers 8+9:9, p. ;<,=, in an article emphasi#ing the evaluation of programs, notes that
evaluation research 8+= is usually employed for decision ma"ing6 8;= deals ith research
questions about a program6 8A= ta"es place in the real orld of the program6 and 8>= usually
represents a compromise beteen pure and applied research. 1allace and 3an *leet 8;,,+=
comment that evaluation should be carefully planned, not occur by accident6 have a purpose
that is usually goal oriented6 focus on determining the quality of a product or service6 go
beyond measurement6 not be any larger than necessary6 and reflect the situation in hich it
ill occur. Similarly, evaluation should contribute to an organi#ation@s planning efforts6 be built
into e.isting programs6 provide useful, systematically collected data6 employ an outside
evaluator!consultant hen possible6 involve the staff6 not be any fancier than necessary6 and
target multiple audiences and purposes 8Some -ractical $essons on Evaluation, ;,,,=.
%n an earlier or" on the evaluation of special libraries, (riffiths and Ging 8+99+, p. A= identify
some principles for good evaluation that still bear repeatingD
+. Evaluation must have a purpose6 it must not be an end in itself
;. 1ithout the potential for some action, there is no need to evaluate
A. Evaluation must be more than descriptive6 it must ta"e into account relationships among
operational performance, users, and organi#ations
>. Evaluation should be a communication tool involving staff and users
<. Evaluation should not be sporadic
but be ongoing and provide a means for continual monitoring, diagnosis, and change
B. 'ngoing evaluation should provide a means for continual monitoring, diagnosis and change
C. 'ngoing evaluation should be dynamic in nature, reflecting ne "noledge and changes in
the environment
As has been implied, but not e.plicitly stated above, evaluation often attempts to assess the
effectiveness of a program or service. 'n a more specific level, evaluation can be used to
support accreditation revies, needs assessments, ne proHects, personnel revies, conflict
resolution, and professional compliance reports.
T?-ES '* E3A$4AT%'5 RESEARC2
Before selecting specific methods and data collection techniques to be used in an evaluation
study, the evaluator, according to 1allace and 3an *leet 8;,,+=, should decide on the general
approach to be ta"en. They categori#e
the general approaches as ad hoc!as needed!as required or evaluation conducted hen a
problem arises6 e.ternally centered, or evaluation necessitated by the need to respond to
e.ternal forces such as state library and accrediting agencies6 internally centered, or
evaluation underta"en to resolve internal problems6 and research centered, or evaluation that
is conducted so that the results can be generali#ed to similar environments. 'ther broad
categories of evaluation that can encompass a variety of methods include macroevaluation,
microevaluation, subHective evaluation, obHective evaluation, formative evaluation 8evaluation
of a program made hile it is still in progress=, and summative evaluation 8performed at the
end of a program=. The Encyclopedia of Evaluation 8)athison, ;,,>= treats forty&to different
evaluation approaches and models ranging from 7appreciative inquiry7 to 7connoisseurship7 to
7transformative evaluation.7
E3A$4AT%'5 )ET2'ES
2aving decided on the general approach to be ta"en, the evaluator must ne.t select a more
specific approach or method to be used in the evaluation study. 1hat follos are brief
overvies of several commonly used evaluation methods or groups of methods.
%nput )easurement
%nput measures are measures of the resources that are allocated to or held by an organi#ation
and represent the longest&standing, most traditional approach to assessing the quality of
organi#ations and their resources and services. E.amples of input measures for libraries
include the number of volumes held, money in the budget, and number of staff members. By
themselves they are more measurement than true evaluation and are limited in their ability to
assess quality.
'utput!-erformance )easurement
'utput or performance measures serve to indicate hat as accomplished as a result of some
programmatic activity and thus arrant being considered as a type of evaluation research.
Such measures focus on indicators of library output and effectiveness rather than merely on
input6 are closely related to the impact of the library on its community6 and, as is true for
virtually all evaluation methods, should be related to the organi#ation@s goals and obHectives.
As as Hust indicated, one critical element of performance measurement is effectiveness6
another is user satisfaction. %n addition to user satisfaction, e.amples of performance!output
measures include use of facilities and equipment, circulation of materials, document delivery
time, reference service use, subHect search success, and availability of materials. The
Association of Research $ibraries 8;,,>= identified the folloing eight output measures for
academic librariesD ease and breadth of access, user satisfaction, teaching and learning,
impact on research, cost effectiveness of library operations and services, facilities and space,
mar"et penetration, and organi#ational capacity. 'ne could argue that not all of those eight
measures represent true performance or output measures, but they are definitely measures of
effectiveness.
%mpact!'utcomes Assessment
The input or resources of a library are relatively straightforard and easy to measure. True
measurement of the performance of a library is more difficult to achieve, and it is even more
challenging to measure impact!outcomes or ho the use of library and information
resources and services actually affects users. Rossi, $ipsey, and *reeman 8;,,>= point out
that outcomes must relate to the benefits of products and services, not simply their receipt 8a
performance measure=. 2oever, given the increasing call for accountability, it is becoming
imperative for libraries to measure outcomes or impact. %ndeed, 7outcomes evaluation has
become a central focus, if not the central focus, of accountability&driven evaluation7 8-atton,
;,,;, p. +<+=.
Some authors use the terms impact and outcome synonymously6 others see them as
somehat different concepts. -atton 8;,,;, p. +B;= suggests a logical continuum that includes
inputs, activities and processes, outputs, immediate outcomes, and long&term impacts. Bettor
and )cClure, in a ;,,A article in $ibrary Trends 8pp. <99&B,,=, identified si. types of
outcomesD
+. EconomicD outcomes that relate to the financial status of library users
;. $earningD outcomes reflecting the learning s"ills and acquisition of "noledge of users
A. ResearchD outcomes that include, for e.ample, the impacts of library services and resources
on the research process of faculty and students
>. %nformation E.changeD outcomes that include the ability of users to e.change information
ith organi#ations and other individuals
<. CulturalD the impact of library resources and services on the ability of library users to benefit
from cultural activities
B. CommunityD outcomes that affect a local community and in turn affect the quality of life for
members of the community
)atthes 8;,,>, pp. +,9&++,=, in his boo" on measuring public library effectiveness, identifies
si. categories of outcomes or benefits for public libraries. Those si. categories, ith e.amples,
are as follosD
+. Cognitive resultsD refreshed memory, ne "noledge, changed ideas
;. Affective
resultsD sense of accomplishment, sense of confidence
A. )eeting e.pectationsD getting hat they needed, getting too much, see"ing substitute
sources
>. AccomplishmentsD able to ma"e better&informed decisions, achieving a higher quality
performance
<. Time aspectsD saved time, asted time, had to ait for service
B. )oney aspectsD the dollar value of results obtained, the amount of money saved, the cost of
using the service
%mpacts more relevant to academic libraries and their users include improved test scores,
better papers, publications, increased class participation, etc. 8-oell, +99<=. A boo" by
2ernon and Eugan 8;,,;= considers outcomes for both academic and public libraries. The
latter include getting ideas, ma"ing contact ith others, resting or rela.ing, and being
entertained. )ar"less and Streatfield 8;,,+= e.amine impact indicators for public, school, and
academic libraries. Among their impact targets for school libraries are 7improved quality and
type of communication beteen learners and $RC staff7 and 7enhanced user confidence7 8p.
+C<=. Seadle 8;,,A= notes that outcome&based evaluation is increasingly used for digital library
proHects.
Service Iuality
Service quality, briefly defined, is 7the difference beteen a library user@s e.pectations and
perceptions of service performance7 85itec"i, +99B, p. +:;=. As a concept, it dates bac" to at
least the +9C,s and has some roots in the total quality management 8TI)= movement. TI) is
characteri#ed by the implementation of standards of quality, the encouragement of innovation,
the measurement of results, and the ta"ing of corrective actions as needed. TI) emphasi#es
the use of a team approach to ma.imi#ing customer satisfaction. A +99B article by -ritchard
provides an e.cellent overvie of TI), as ell as other approaches to determining quality.
Iuality is an elusive concept for hich there is no commonly accepted definition, but the
assessment of service quality did get a boost from earlier research from -arasuraman, Berry,
and Jeithaml 8see 5itec"i, +99B=. They developed a conceptual frameor", the (aps )odel of
Service Iuality, and a idely used instrument, SER3&I4A$, for measuring service quality.
The (aps )odel incorporates the folloing gaps, as measured by the SER3&I4A$
questionnaireD
+. The discrepancy beteen customers@ e.pectations and managements@ perceptions of these
e.pectations
;. The discrepancy beteen managements@ perceptions of customers@ e.pectations and
service&quality specifications
A. The discrepancy beteen service&quality specifications and actual service delivery
>. The discrepancy beteen actual service delivery and hat is communicated to customers
about it
<. The discrepancy beteen customers@ e.pected services and perceived services delivered
85itec"i, +99B, p. +:;=
The most visible current iteration
of SER3&I4A$ in the library field is "non as $ibI4A$K. $ibI4A$K as developed by
faculty members of Te.as AF) 4niversity in partnership ith the Association of Research
$ibraries 8AR$= and is part of AR$@s 5e )easures %nitiative. 'ver the past fe years
$ibI4A$K studies have been conducted by hundreds of libraries, including many large
university libraries in the 4nited States. These studies are intended fo. libraries 7to solicit,
trac", understand, and act upon users@ opinions of service quality7 8$ibI4A$K, ;,,A=.
Iuestions in the $ibI4A$K questionnaire address library staff, print and electronic resources,
service hours, facilities, equipment, and document delivery and gather the data needed to
calculate the gaps described above. 2oever, according to Shi and $evy, 7the current
$ibI4A$K is not yet an adequately developed tool to measure and represent a dependable
library services assessment result7 8;,,<, p. ;C;=.
%ndividuals anting to "no more about the use of service quality methods in academic
libraries may ish to read a boo" by 2ernon and Altman 8+99B=. 'ther models of quality
assessment from a British perspective are considered by Lones, Ginnell, and 4sherood
8;,,,=.
-rocess Evaluation
The second stage in -atton@s 8;,,;= continuum described in the section on impact!outcomes
assessment as processes or activities. 7A focus on process involves loo"ing at ho
something happens rather than or in addition to e.amining outputs and outcomes7 8p. +<9=.
7-rocess data permit Hudgments about the e.tent to hich the program or organi#ation is
operating the ay it is supposed to be operating, revealing areas in hich relationships can be
improved as ell as highlighting strengths of the program that should be preserved7 8-atton,
;,,;, p. +B,=. -rocess evaluation focuses on 7hat the program actually does7 81eiss, +99:,
p. 9=. %t 7is the most frequent form of program evaluation7 8Rossi, $ipsey, F *reeman, ;,,>, p.
<C=.
-rocess indicators are somehat similar to performance measures, but they focus more on
the activities and procedures of the organi#ation than on the products of those activities. *or
e.ample, a process evaluation of an acquisitions department ould be concerned ith ho
materials are acquired and prepared for the shelf, not on ho many boo"s are ultimately used.
%n an academic library setting, process indicators might include staff training and development,
delivery styles, "noledge of the curriculum, and participation in assignments and grading
8)ar"less F Streatfield, ;,,+=. %n his boo" on public library effectiveness, )atthes 8;,,>=
places process measures in three categoriesD efficiency, staff productivity, and library
information system activity. )ore generally spea"ing, a process evaluation 7might e.amine
ho consistent the services actually delivered are ith the goals of the program, hether
services are delivered to appropriate recipients, ho ell service delivery is organi#ed, the
effectiveness of program management, the use of program resources, and other such matters7
8Rossi, $ipsey, F *reeman, ;,,>, p. <C=. And ultimately, the evaluator ould ant to "no the
e.tent to hich programs and services ere actually implemented. -atton 8;,,;= even argues
that 7implementation evaluation7 is a distinct method, and in many cases implementation
information is of greater value than outcomes information 8p. +B+=.
Benchmar"ing
'ne of the relatively recent approaches to measuring the performance of libraries and other
organi#ations is benchmar"ing. Benchmar"ing tends to fall into the 7total quality management7
category. Benchmar"ing 7represents a structured, proactive change effort designed to help
achieve high performance through comparative assessment. %t is a process that establishes an
e.ternal standard to hich internal operations can be compared7 8Luro, +99A, p. +;,=. The
;,,, Standards for College $ibraries describes benchmar"ing as the process of evaluating a
library@s points of comparison&&inputs and outputs&&against its peers and aspirational peers.
There are several types of benchmar"ing, one of hich is referred to as competitive or
performance benchmar"ing. -erformance benchmar"ing utili#es comparative data gathered
from the same field or the same type of organi#ation. The data are usually derived from
analyses of organi#ational processes and procedures. Benchmar"ing can be used to establish
best practices, identify changes to improve services, evaluate opinions and needs of users,
identify trends, e.change ideas, and develop staff. -eischl 8+99<= points out that candidates
for benchmar"ing include the services or products of an organi#ation, internal or" processes,
internal support functions, and organi#ational performance and strategy.
Standards
According to Ba"er and $ancaster, 7standards have an important role to play in the evaluation
of library services ... 1hen applied to libraries, hoever, standards refers to a set of
guidelines or recommended practices, developed by a group of e.perts, that serve as a model
for good library service7 8+99+, p. A;+=. Some general types of standards, as identified by
Ba"er and $ancaster 8+99+=, include technical standards 8for e.ample, cataloging codes=,
performance standards, output measures, input measures, qualitative standards, and
quantitative standards.
Iuantitative Evaluation
Any evaluation method that involves the measurement of quantitative! numerical
variables probably qualifies as a quantitative method, and many of the methods already
e.amined fall into this broad category. Among the strengths of quantitative methods are the
evaluator can reach conclusions ith a "non degree of confidence about the e.tent and
distribution of that the phenomenon6 they are amenable to an array of statistical techniques6
and they are generally assumed to yield relatively obHective data 81eiss, +99:, pp. :A&:>=.
E.perimental methods usually, hut not alays, deal ith quantitative data and are considered
to be the best method for certain "inds of evaluation studies. %ndeed, 7the classic design for
evaluations has been the e.periment. %t is the design of choice in many
circumstances because it guards against the threats to validity7 81eiss, +99:, p. ;+<=. The
e.periment is especially useful hen it is desirable to rule out rival e.planations for outcomes.
%n other ords, if a true e.perimental design is used properly, the evaluator should be able to
assume that any net effects of a program are due to the program and not to other e.ternal
factors.
'n the other hand, e.perimental methods are relatively ea" in producing findings that can be
generali#ed to other situations because they are usually conducted in rather controlled
settings. Also, e.periments tend to be used to test the effects of one component of a program
at a time rather than the entire program. Another limitation of the true or
randomi#ed e.periment is that it is not ell suited for evaluating programs in their early stages
of implementation. %f the program changes significantly before outcomes are measured, it ill
be difficult to determine hich version of the program produced hat effects 8Rossi, $ipsey, F
*reeman, ;,,>=.
Survey methods are often quantitative in nature but lac" the e.periment@s ability to rigorously
test the relationship beteen a program or service and its outputs or impact. Iuestionnaires
and intervies, and observation to a lesser degree, represent the most commonly used survey
data gathering techniques. 'ther quantitative methods covered by the Encyclopedia of
Evaluation 8)athison, ;,,>= include concept mapping, correlation, cross&sectional design,
matri. sampling, meta&analysis, panel studies, regression analysis, standardi#ed tests, and
time series analysis.
Iualitative Evaluation
As is true for basic research, qualitative methods are becoming increasingly popular. %n fact,
7the most stri"ing development in evaluation in recent years is the coming of age of qualitative
methods. 1here once they ere vieed as aberrant
and probably the refuge of those ho had never studied statistics, no they are recogni#ed as
valuable additions to the evaluation repertoire
7 81eiss, +99:, p. ;<;=. The Encyclopedia of Evaluation 8)athison, ;,,>= includes thirty&seven
qualitative methods. They are appropriate, of course, hen the phenomena being evaluated
do not lend themselves to quantification. A qualitative method 7tends to apply a more
holistic and natural approach to the resolution of the problem than does quantitative research.
%t also tends to give more attention to the subHective aspects of human e.perience and
behavior7 8-oell F Connaay, ;,,>, p. <9=. 7Iualitative strategies can be particularly
appropriate here the administration of standardi#ed
instruments, assigning people to comparison groups Min e.perimentsN, and!or the collection of
quantitative data ould affect program operations by being overly intrusive7 8-atton, ;,,;, p.
+9+=. %n addition, they can provide
+. greater aareness of the perspective of program participants and often a greater
responsiveness to their interests
;. capability for understanding dynamic developments in the program as it evolves
A. aareness of time and history
>. special sensitivity to the influence of conte.t
<. ability to enter the program scene ithout preconceptions or prepared instruments, and to
learn hat is happening
B. alertness to unanticipated and unplanned events
C. general fle.ibility of perspective 81eiss, +99:, p. ;<A=.
Iualitative methods do have their disadvantages as ell, of course. Among them are the
folloingD
+. $imited ability to yield obHective data
;. $imited ability to produce generali#able results
A. $imited ability to provide precise descriptions of program outcomes
>. 5ot ell suited for developing specific ansers about the relationship of particular program
strategies or events to outcomes 81eiss, +99:, pp. :<&:B=
<. 'ften relatively labor intensive to conduct
Cost Analysis
Simple cost analysis is basically a descriptive brea"don of the costs incurred in operating an
organi#ation. Cost&related techniques more concerned ith the assessment of hether monies
are being spent in an optimal fashion usually fall into one of to groups&&cost&effectiveness
studies and cost&benefit analysis. 7The term @cost&effectiveness@ implies a relationship beteen
the cost of providing some service and the level of effectiveness of that service ... Cost&
effectiveness analyses can be thought of as studies of the costs associated ith alternative
strategies for achieving a particular level of effectiveness7 8$ancaster, +99A, p. ;BC=. Some
e.amples of cost&effectiveness measures include the cost per relevant informational resource
retrieved, cost per use of a resource, cost per user, cost per capita, and cost by satisfaction
level 8$ancaster, +99A6 )atthes, ;,,>=.
Cost&effectiveness analysis can be seen as 7a truncated form of cost&benefit analysis that
stops short of putting an economic value on ... outcomes MbenefitsN of programs7 8Glarman,
+9:;, p. <:B=. 7@Cost&benefit,@ clearly, refers to a relationship beteen the cost of some activity
and the benefits derived from it. %n effect, a cost&benefit study is one that tries to Hustify the
e.istence of the activity by demonstrating that the benefits outeigh the costs7 8$ancaster,
+99A, p. ;9>=. A typical cost&benefit analysis involves determining ho benefits from and pays
for a service, identifying the costs for each group of beneficiaries, identifying the benefits for
each group, and comparing costs and benefits for each group to determine if groups have net
benefits or net costs and hether the total benefits e.ceed the total costs.
Types of cost&benefit analysis described by $ancaster 8+99A= are
+. net value approachD the ma.imum amount the user of an information service is illing to pay
minus the actual cost
;. value of reducing uncertainty in decision ma"ing
A. cost of buying service elsehere
>. librarian time replaces user time 8that is, the librarian saves the user time by performing his
or her tas"=
<. service improves organi#ation@s performance or saves it money.
'ther "inds of cost analysis discussed by 1eiss 8+99:= and )atthes 8;,,>= include the
folloingD
+. Cost&minimi#ation analysisD see"s to determine the least e.pensive ay to accomplish
some outcome
;. Cost&utility analysisD considers the value or orth of a specific outcome for an individual or
society
A. 1illingness&to&pay approachD as"s ho much individuals are illing to pay to have
something they currently do not have
>. 1illingness&to&accept approachD as"s individuals ho much they ould be illing to accept
to give up something they already have
<. Cost of time
'rgani#ational Effectiveness
The determination of the effectiveness of an organi#ation has been identified as one of the
obHectives for some of the methods described above, and, indeed, it may be more properly
thought of as an evaluation obHective than an evaluation method. Regardless, it is a crucial
element of organi#ational assessment and has received considerable attention in the
professional literature. Rubin 8cited by 1allace and 3an *leet, ;,,+, pp. +A&+>= identifies a
number of criteria for effectiveness at the organi#ational level and then describes several
models for measuring organi#ational effectiveness. Those models and their 7"ey questions7
are as follosD
+. (oalsD 2ave the established goals of the library been met/
;. Critical ConstituenciesD 2ave the needs of constituents been met/
A. ResourcesD 2ave necessary resources been acquired/
>. 2uman ResourcesD %s the library able to attract, select, and retain quality employees/
<. 'pen SystemsD %s the library able to maintain the system, adapt to threats, and survive/
B. Eecision -rocessD 2o are decisions made and evaluated/
C. Customer ServiceD 2o satisfied is the clientele ith the library/
-rogram Evaluation )ethods
%n addition to the methods already identified, there are numerous other methods primarily used
for social program evaluation. Readers interested in learning more about such methods are
referred to the or"s on evaluation already cited above, including the article by Childers
8+9:9=, and to the table by Ging in -oell and Connaay 8;,,>, pp. <C&<:=.
$%S&Centered )ethods
Another approach to categori#ing evaluation methods used in library and information science
is according to the program, service, or resource to be evaluated. The boo" by 1allace and
3an *leet 8;,,+=, for e.ample, has chapters devoted to the evaluation of reference and
information services and to library collections 8see 1hitlatch, ;,,+ for an article on the
evaluation of electronic reference services=. Baden 8+99,= presents a user&oriented approach
for the evaluation of information systems and services. An earlier issue of $ibrary Trends
8Reed, +9C>= has articles on the evaluation of administrative services, collections, processing
services, adult reference service, public services for adults, public library services for children,
and school library media services. $ancaster@s +99A te.t includes the evaluation of collections,
collection use, in&house library use, periodicals, library space, catalog use, document delivery,
reference services, and resource sharing. )ost of these methods, hoever, actually employ
techniques related to the more generic methods identified earlier in this article.
-$A55%5( T2E E3A$4AT%'5 ST4E?
As has already been indicated, evaluation should be part of an organi#ation@s overall planning
process and integral to the assessment of current services and resources, the development of
strategies for change, and the monitoring of progress toard goals and obHectives. %ndeed, in
order to be valid, an evaluation must reflect the organi#ation@s mission, goals, and obHectives.
%n planning the evaluation of a specific program, the evaluator should first gather relevant
bac"ground information. This activity might ell include revieing the professional literature,
identifying professional standards and guidelines, and netor"ing ith colleagues. 5e.t, the
evaluator should decide hat he or she actually ants to "no, that is, focus the evaluation.
This requires a determination of the purpose 8s= of the evaluation specific to the program being
e.amined. *or e.ample, the purpose may simply be to learn more about the program, or it
may be to determine if the program is meeting its obHectives.
After focusing the evaluation, decisions must be made about the overall design of the study,
the method8s= to be used, and the measurements to be made. %n other ords, the evaluator
must decide hat must be measured, choose an evaluation method, select the data collection
techniques to be employed, plan the construction and!or purchase of data collection
instruments, plan the data analysis, develop a budget for the evaluation study, and recruit
personnel. As is often the case in research studies, it is a good idea to utili#e more than one
method so as to increase the reliability and validity of the study and its findings. 2aynes 8;,,>,
p. +9=, for e.ample, argues for mi.ed&method evaluation, hich combines user&centered ith
system&centered paradigms and qualitative ith quantitative methods. %t is a good idea to rite
a thorough plan or proposal for the study at this time.
1eiss 8+99:= reminds us that the evaluator should also give careful thought to the best time to
conduct the evaluation, the types of questions to as", hether one or a series of studies ill be
necessary, and any ethical issues that might be generated by the study. Those and other
planning points are succinctly represented in the folloing 7evaluation action plan7 suggested
by 1allace and 3an *leet 8;,,+, pp. >&<=D
+. 1hat@s the problem/
;. 1hy am % doing this/
A. 1hat e.actly do % ant to "no/
>. Eoes the anser already e.ist/
<. 2o do % find out/
B. 1ho@s involved/
C. 1hat@s this going to cost/
:. 1hat ill % do ith the data/
9. 1here do % go from here/
C'5E4CT%5( T2E E3A$4AT%'5 ST4E?
After planning the evaluation, it is time, of course, to conduct the study. That is, the evaluator
is no ready to collect data or measure hat needs to be measured6 analy#e the data6 and
report the findings. 1hat follos is a brief overvie of the steps in the evaluation process.
)easurement
7)easurement, in most general terms, can be regarded as the assignment of numbers to
obHects 8or events or situations= in accord ith some rule. The property of the obHects hich
determines the assignment according to that rule is called magnitude, the measurable
attribute6 the number assigned to a particular obHect is called its measure, the amount or
degree of its magnitude7 8Gaplan, +9B>, p. +CC=. )ore generally, measurement is any process
for describing in quantitative values things, people, events, etc. )easurement by itself is not
true evaluation, but it is one of the building bloc"s for quantitative evaluation. Common types of
measures for library evaluation studies include number and types of users, number and
duration of transactions, user and staff activities, user satisfaction levels, and costs of
resources and services. They can be related to input, output, effectiveness, costs, etc.
%t is critical that the measurement process and the measures be reasonably high in reliability
and validity. Reliability refers to the degree to hich measurements can be depended upon to
secure consistent and accurate results in repeated applications. 3alidity is the degree to hich
any measure or data collection technique succeeds in doing hat it purports to do6 it refers to
the meaning of an evaluative measure or procedure. The validity and!or reliability of measures
can be affected by such factors as inconsistent data collection techniques, biases of the
observer, the data collection setting, instrumentation
, behavior of human subHects, and sampling. The use of multiple measures can help to
increase the validity and reliability of the data. They are also orth using because no single
technique is up to measuring a comple. concept, multiple measures tend to complement one
another, and separate measures can be combined to create one or more composite measures
81eiss, +99:=.
Statistics
)any measures are in the form of statistics, hich, in some cases, can be dran from already
e.isting sources of data. Types of statistics include administrative data, financial statistics,
collections and other resources or inputs, use and other output!performance measures,
outcomes, and staff and salary information. Sources of statistics include governmental
agencies, professional associations, and other organi#ations such as state library agencies.
Among the noteorthy sources of library&related statistics are the 5ational Center for
Education Statistics 85CES=, American $ibrary Association and its divisions 8such as the
-ublic $ibrary Association@s -ublic $ibrary Eata Service and the Association of College and
Research $ibraries@ Trends and Statistics series=, Association of Research $ibraries, and
federal programs such as the *ederal State Cooperative System and the %ntegrated
-ostsecondary Education Eata System.
Eata Collection Techniques
The evaluator must ne.t select or design one or more data collection techniques that are
compatible ith the method8s= to be used and that are capable of gathering the necessary
information. There are too many data collection techniques to consider here, but some of the
relatively common techniques and instruments used for evaluation studies, as ell as for other
"inds of research, include the folloingD
+. Tests 8standardi#ed and locally developed=
;. Assessments by participants
A. Assessments by e.perts
>. Iuestionnaires 8paper and electronic=
<. %ntervies, including focus groups
B. 'bservation of behavior and activities
C. Evaluation of staff performance
:. Analysis of logs or diaries of participants
9. Analysis of historical and current records
+,. Transactional log analysis
++. Content analysis
+;. Bibliometrics, especially citation analysis
+A. 4se records
+>. Anecdotal evidence
*or information about many of these techniques, readers are referred to -oell and Connaay
8;,,>= and 2ernon and )cClure 8+99,=. *or more information about techniques unique to
evaluations of library and information use, readers may ish to consult earlier te.ts by
$ancaster 8+99A= and Ba"er and $ancaster 8+99+=. 1estbroo"@s chapter in -oell and
Connaay 8;,,>=, a chapter in 1eiss 8+99:=, and the boo" by -atton 8;,,;= are among the
sources of information about qualitative data collection techniques.
Analysis of Eata
7The aim of analysis is to convert a mass of ra data into a coherent account. 1hether the
data are quantitative or qualitative, the tas" is to sort, arrange, and process them and ma"e
sense of their configuration. The intent is to produce a reading that accurately represents the
ra data and blends them into a meaningful account of events7 81eiss, +99:, p. ;C+=. The
basic tas"s of data analysis for an evaluative study are to anser the questions that must be
ansered in order to determine the success of the program or service, the quality of the
resources, etc. Those questions should, of course, be closely related to the nature of hat is
being evaluated and the goals and obHectives of the program or service. %n addition, the nature
of the data analysis ill be significantly affected by the methods and techniques used to
conduct the evaluation. According to 1eiss 8+99:=, most data analyses, hether quantitative
or qualitative in nature, ill employ some of the folloing strategiesD describing, counting,
factoring 8that is, dividing into constituent parts=, clustering, comparing, finding commonalities,
e.amining deviant
cases, finding covariation, ruling out rival e.planations, modeling, and telling the story.
Evaluators conducting quantitative data analyses ill need to be familiar ith techniques for
summari#ing and describing the data 8that is, descriptive statistics=6 and if they are engaged in
testing relationships or hypotheses and!or generali#ing findings to other situations, they ill
need to utili#e inferential statistics.
1hatever the nature of the data analysis, hoever, it cannot substitute for sound development
of the study and interpretation of the findings. Statistics can only facilitate the interpretation. %n
a quantitative study the analysis and interpretation usually follo the conduct of the study. %n a
qualitative study the data analysis is typically concurrent ith the data gathering6 7nor, in
practice, are analysis and interpretation neatly separated7 8-atton, +9:C, p. +>>=.
The Evaluation Report
As part of the planning, the evaluator should have considered ho and to hom the findings
ill be communicated and ho the results ill be applied. 1eiss 8+99:, pp. ;9B&;9C=
recommends that the typical report of a program evaluation include the folloing elementsD
+. Summary of study results
;. -roblem ith hich the program deals
A. 5ature of the programD goals and obHectives, activities, conte.t, beneficiaries, staff
>. 5ature of the evaluation
<. Comparison ith evaluations of similar programs 8optional=
B. Suggestions for further evaluation 8optional=
A good report ill be characteri#ed by clarity, effective format and graphics, timeliness, candor
about strengths and ea"nesses of the study, and generali#ability 81eiss, +99:=, as ell as
by adequacy of sources and documentation, appropriateness of data analysis and
interpretation, and basis for conclusions.
C'5C$4S%'5S
As as indicated above, evaluation research has been defined in a number of ays. %t is
vieed as a specific research methodology, as a type of study that uses standard social
research methods for evaluative purposes, and as an assessment process employing special
techniques unique to the evaluation of programs. %f treated as research, it is li"ely to be
designed as applied or action research even though it may ell use basic research methods.
But generally spea"ing, all of the approaches to evaluation tend to share the folloing
important commonalitiesD evaluation is a systematic process6 it involves collecting data about
organi#ations, processes, programs, services, and resources6 it is a process for enhancing
"noledge and decision ma"ing6 and it is e.pected to lead to practical applications 8-res"ill F
Russ&Eft, ;,,<, pp. +&;=. And finally, evaluation research should be conducted carefully and
rigorously ith consideration of many of the tenets that characteri#e good basic research.
RE*ERE5CES
Association of Research $ibraries. 8;,,>=. AR$ ne measures retreat6 Transcript. 1ashington,
ECD Association of Research $ibraries.
Ba"er, S. $., F $ancaster, *. 1. 8+99+=. The measurement and evaluation of library services
8;nd ed.=. Arlington, 3AD %nformation Resources -ress.
Baden, E. 8+99,=. 4ser&oriented evaluation of information systems and services. Broo"field,
3TD (oer.
Bertot, L. C., F )cClure, C. R. 8;,,A=. 'utcomes assessment in the netor"ed environmentD
Research questions, issues, considerations, and moving forard. $ibrary Trends, <+8>=, <9,&
B+A.
Childers, T. 8+9:9=. Evaluative research in the library and information field. $ibrary Trends,
A:8;=, ;<,&;BC.
(riffiths, L. )., F Ging, E. 1. 8+99+=. A manual on the evaluation of information centers and
services. 5e ?or"D American %nstitute of Aeronautics and Astronautics Technical %nformation
Service.
2aynes, A. 8;,,>=. Bridging the gulfD )i.ed methods and library service evaluation. Australian
$ibrary Lournal, <A8A=, ;:<&A,C.
2ernon, -., F Altman, E. 8+99B=. Service quality in academic libraries. 5orood, 5LD Able..
2ernon, -., F Eugan, R. E. 8;,,;=. An action plan for outcomes assessment in your library.
ChicagoD American $ibrary Association.
2ernon, -., F )cClure, C. R. 8+99,=. Evaluation and library decision ma"ing. 5orood, 5LD
Able..
Lones, G., Ginnell, )., F 4sherood, B. 8;,,,=. The development of self&assessment tool&"its
for the library and information sector. Lournal of Eocumentation, <B8;=, ++9&+A<.
Luro, S. R. 8+99A=. Tools for measuring and improving performance. %n S. Luro F S. B.
Barnard 8Eds.=, %ntegrating Total Iuality )anagement in a library setting 8pp. ++A&+;B=. 5e
?or"D 2aorth.
Gaplan, A. 8+9B>=. The conduct of inquiryD )ethodology for behavioral science. San *ranciscoD
Chandler.
Glarman, 2. E. 8+9:;=. The road to cost&effectiveness analysis. )ilban" )emorial *und
IuarterlyD 2ealth and Society, B,8>=, <:<&B,A.
$ancaster, *. 1. 8+99A=. %f you ant to evaluate your library ... 8;nd ed.=. Champaign, %$D
4niversity of %llinois

, (raduate School of $ibrary and %nformation Science.

$ibI4A$ K Spring ;,,A survey results. 8;,,A=. 1ashington, ECD Association of Research
$ibraries.
)ar"less, S., F Streatfield, E. 8;,,+=. Eeveloping performance and impact indicators and
targets in public and education libraries. %nternational Lournal of %nformation )anagement,
;+8;=, +BC&+C9.
)athison, S. 8Ed.=. 8;,,>=. Encyclopedia of evaluation. Thousand 'a"s, CAD Sage
-ublications. )atthes, L. R. 8;,,>=. )easuring for resultsD The dimensions of public library
effectiveness. 1estport, C5D $ibraries 4nlimited.
5itec"i, E. A. 8+99B=. Changing the concept and measure of service quality in academic
libraries. Lournal of Academic $ibrarianship, ;;8A=, +:+&+9,.
-atton, ). I. 8+9:C=. 2o to use qualitative methods in evaluation. 5ebury -ar", CAD Sage
-ublications.
-atton, ). I. 8;,,;=. Iualitative research and evaluation methods 8Ard ed.=. Thousand 'a"s,
CAD Sage -ublications.
-eischl, T. ). 8+99<=. Benchmar"ingD A process for improvement. -roceedings of the *irst
%nternational Conference on TI) and Academic $ibraries 8pp. ++9&+;;=. 1ashington, ECD
Association of Research $ibraries
-oell, R. R. 8+99<=. %mpact assessment of university libraries. %n Encyclopedia of library and
information science 83ol. <<, pp. +<+&+B>=. 5e ?or"D )arcel Ee""er.
-oell, R. R., F Connaay, $. S. 8;,,>=. Basic research methods for librarians 8>th ed.=.
1estport, C5D $ibraries 4nlimited.
-res"ill, 2., F Russ&Eft, E. 8;,,<=. Building evaluation capacityD C; activities for teaching and
training. Thousand 'a"s, CAD Sage -ublications.
-ritchard, S. ). 8+99B=. Eetermining quality in academic libraries. $ibrary Trends, >>8A=, <C;&
<9>.
Reed, S. R. 8Ed.=. 8+9C>=. Evaluation of library service MSpecial %ssuesN. $ibrary Trends, ;;8A=.
Rossi, -. 2., $ipsey, ). 1., F *reeman, 2. E. 8;,,>=. EvaluationD A systematic approach 8Cth
ed.=. Thousand 'a"s, CAD Sage -ublications.
Seadle, ). 8;,,A=. EditorialD 'utcome&based evaluation. $ibrary 2i Tech, ;+8+=, <&C.
Shi, O., F $evy, S. 8;,,<=. A theory&guided approach to library services assessment. College F
Research $ibraries, BB8A=, ;BB&;CC.
Some practical lessons on evaluation. 8;,,,=. SnapshotsD Research 2ighlights from the
5onprofit Sector Research *und, 5o. 9.
Standards for College $ibraries. 8;,,,=. ChicagoD American $ibrary Association.
1allace, E. -., F 3an *leet, C. 8;,,+=. $ibrary evaluationD A caseboo" and can&do guide.
Engleood, C'D $ibraries 4nlimited.
1eiss, C. 2. 8+99:=. EvaluationD )ethods for studying programs and policies 8;nd ed.=. 4pper
Saddle River
, 5LD -rentice 2all.
1hitlatch, L. B. 8;,,+=. Evaluating reference services in the electronic age. $ibrary Trends,
<,8;=, ;,C&;+C.
Additional References
Abels, E. (., Gantor, -. B., F Saracevic, T. 8+99B=. Studying the cost and value of library and
information services. Lournal of the American Society for %nformation Science, >C8A=, ;+C&;;C.
Bertot, L. C., )cClure, C. R., F Ryan, L. 8;,,+=. Statistics and performance measures for
public library netor"ed services. ChicagoD American $ibrary Association.
Childers, T. A., F 3an 2ouse, 5. A. 8+99A=. 1hat@s good/ Eescribing your public library@s
effectiveness. ChicagoD American $ibrary Association.
Everhart, 5. 8+99:=. Evaluating the school library media centerD Analysis techniques and
research practices. Engleood, C'D $ibraries 4nlimited.
5elson, 1. 5., F *erne"es, R. 1. 8;,,;=. Standards and assessment for academic librariesD A
or"boo". ChicagoD Association of College and Research $ibraries.
5isonget, T. E. 8;,,A=. Evaluation of library collections, access and electronic resourcesD A
literature guide and annotated bibliography. 1estport, C5D $ibraries 4nlimited.
Smith, ). $. 8+99B=. Collecting and using public library statisticsD A ho&to&do&it manual for
librarians. 5e ?or"D 5eal&Schuman.
3an 2ouse, 5. A., $ynch, ).L., )cClure, C. R., Jei#ig, E. $., F Rodger, E. L. 8+9:C=. 'utput
measures for public libraries 8;nd ed.=. ChicagoD American $ibrary Association.
3an 2ouse, 5., 1eil, B, F )cClure, C. 8+99,=. )easuring academic library performanceD A
practical approach. ChicagoD American $ibrary Association.
1hitlatch, L.B. 8;,,,=. Evaluating reference servicesD A practical guide. ChicagoD American
$ibrary Association.
Jei#ig, E. 8Ed.=. 8+99>=. The tell it0 manualD The complete program for evaluating library
performance. ChicagoD American $ibrary Association.
Ronald R. -oell is a professor in the $ibrary and %nformation Science -rogram at 1ayne
State 4niversity, Eetroit, )ichigan. -rior to becoming a library and information science
educator, he served as a university librarian and college library director. Er. -oell has taught,
conducted research, and published in the areas of research methods, collection development,
bibliographic instruction, academic libraries, the measurement and evaluation of library
resources and services, and education for librarianship. 2is publications include Basic
Research )ethods for $ibrarians 8ith $ynn Connaay=, Basic Reference Sources 8ith
)argaret Taylor=, Iualitative Research in %nformation )anagement 8ith Lac" (la#ier=, and
The 5e.t $ibrary $eadership 8ith -eter 2ernon and Arthur ?oung
=.
C'-?R%(2T ;,,B 4niversity of %llinois at 4rbana&Champaign
5o portion of this article can be reproduced ithout the e.press ritten permission from the copyright holder.
Copyright ;,,B, (ale (roup. All rights reserved. (ale (roup is a Thomson Corporation Company.