J 1744-6570 2002 tb00118 X

PERSONNEL PSYCHOLOGY
2002,ss
RECRUITMENT EVALUATION: THE CASE FOR

ASSESSING THE QUALITY OF APPLICANTS
ATTRACTED
KEVIN D. CARLSON
Department of Management
Virginia Polytechnic Institute and State University
MARY L. CONNERLEY, ROSS L. MECHAM I11
Department of Management
Virginia Polytechnic Institute and State University
We argue that assessing attraction outcomes is critical to systematic

improvement of recruitment effectiveness and offer a new assessment
framework that can be adapted by all organizations for any position in
any staffing scheme. These methods (a) permit outcomes of different
recruitment processes-attraction, status maintenance, and gaining
job acceptance-to be evaluated independently, (b) support concur-
rent evaluations of attraction outcomes, (c) enable cost-benefit analy-
ses of alternative recruitment processes, (d) allow meaningful compar-
isons of applicants from different recruitment events, and (e) support
more aggressive management of the recruitment function than is oth-
erwise possible. An illustrative example demonstrates these methods
using attraction outcome data from 5 organizations’ recruitment of as-
sociate engineers and shows that not only do attraction outcomes differ,
the value of those differences can be substantial.
It is becoming clear that effective recruitment is critical to organi-

zational success (e.g., Barber, 1998; Breaugh, 1992; Rynes, 1991). Re-
searchers at Watson Wyatt list recruiting excellence as one of five human
resource practices that affect the bottom line (Grossman, 2000). Huselid
(1995) demonstrated a positive relationship between low selection ra-
tios and organization financial performance. Lower selection ratios are
achieved by attracting larger numbers of applicants. Terpstra and Rozell
(1993) found that organizations that engaged in more of five targeted
staffing practices, including the use of follow-up studies to determine
the most effective recruitment sources, had greater annual profits and
greater annual profit growth. Yet, few organizations formally evaluate
their recruitment efforts (e.g., Davidson, 1998; Grossman, 2000; Rynes
The authors thank Jeff Arthur, T.W. Bonham, Terry Cobb, Robert Madigan and two
anonymous reviewers for their helpful comments on earlier drafts of this manuscript. We
also acknowledge the invaluable assistance of Jim Malone and Marc Hunter.
Correspondence and requests for reprints should be addressed to Kevin D. Carlson,
21 15 Pamplin Hall, Department of Management, Virginia Polytechnic Institute and State
University, Blacksburg, VA 24061; kevinc@vt.edu.
COPYRIGHT 0 2002 PERSONNEL PSYCHOLOGY, INC.
46 1
462 PERSONNEL PSYCHOLOGY
& Boudreau, 1986). The SHRM/CCH (1999) Survey of Recruiting Prac-

tices found that only 44% of the 279 organizations surveyed formally
evaluate any recruitment outcomes. If, as stated by Peter Drucker, “few
things are as important to the performance of the organization as mea-
surement” (quoted in Fitz-Enz, 1984, p. lo), general improvement in the
effectiveness of organizations’ recruitment practices is unlikely without
renewed emphasis on measurement and evaluation.
In this paper, we argue for the systematic assessment of attraction
outcomes-not just the number of applicants, but applicant quality as
well. We offer a methodological framework that integrates recruitment
theory (e.g., Barber, 1998; Breaugh, 1992), performance prediction (e.g.,
Schmidt & Hunter, 1998), and utility analysis (e.g., Boudreau, 1991;
Brogden, 1949; Cronbach & Gleser, 1965) to create a new approach to
recruitment evaluation. These methods can be adapted by any orga-
nization, tailored to any position, and used with any hiring approach.
This approach (a) supports evaluation of three different recruitment
outcomes-attraction, status maintenance, and gaining job acceptance,
(b) allows ongoing, concurrent evaluation of recruitment outcomes, (c)
permits analysis of the costs and benefits of alternative recruitment ap-
proaches, (d) enables meaningful comparisons of applicants from dif-
ferent applicant pools, and (e) supports more aggressive management
of the recruitment function than is otherwise possible.
This paper also offers several unique contributions to the recruit-
ment evaluation literature. It provides a framework for integrating the
outcomes of attraction, status maintenance, and job acceptance activities
and offers an interesting illustration of how attraction outcomes affect
the potential dollar value of performance. We address practical issues
concerning the desired properties of applicant “quality” data, methods
for developing these data, and how these data should be aggregated to
evaluate attraction outcomes. The actual attraction outcomes of five or-
ganizations competing for associate engineer applicants from the same
labor pool are used to demonstrate how differences in attraction out-
comes can be assessed. These data provide evidence that actual differ-
ences in the value of attraction outcomes can be substantial. In an ad-
ditional analysis, we simulate various levels of status maintenance and
job acceptance performance to demonstrate how attraction outcomes
determine the effect that the failure to hire the top candidates has on
recruitment outcomes.
Recruitment Evaluation
For organizations to pursue systematic, sustained improvement in re-

cruitment effectiveness, a new approach to the assessment of recruit-
KEVIN D. CARLSON ET AL. 463
ment outcomes is necessary. We believe a practical approach to recruit-

ment evaluation can be achieved (a) by isolating and evaluating attrac-
tion outcomes, (b) by directly assessing the job performance potential of
each applicant, and then (c) applying utility analysis methods to convert
differences in applicants’ job performance potential to a dollar metric.
This final step produces an estimate of the benefits of alternative attrac-
tion activities that can be compared to their costs.
Decomposing Recruitment
Recruitment is the primary mechanism used by organizations to at-

tract talent. Rynes (1991) defined recruitment as encompassing “all or-
ganizational practices and decisions that affect either the number, or
types, of individuals who are willing to apply for, or accept, a given va-
cancy” (p. 429). Although recruitment is often discussed as if it were a
single activity, it is actually a combination of activities that can be manip-
ulated independently. Barber (1998) noted there is widespread recog-
nition that recruitment contains multiple stages, though “little attention
has been paid to the identification or delineation of those stages” (p. 12).
She suggests there are three recruitment phases-(a) Attempts by the
organization to reach out to the applicant population to persuade some
portion of them to become applicants, (b) attempts to persuade appli-
cants to remain interested in the organization and to continue to pursue
the job opportunity until the organization determines whether to offer
them a position, and (c) attempts to persuade individuals to accept job
offers. Barber describes these phases as generating applicants, main-
taining applicant status, and influencing job choice.
Research on generating applicants (i.e., attraction) has examined
recruitment tactics (e.g., Garrnan & Mortensen, 1997; Lavigna, 1996;
Rynes & Barber, 1990), sources of recruits (e.g., Griffith, Hom, Fink, &
Cohen, 1997; Werbel& Landau, 1996), content of recruiting messages
(e.g., Highhouse, Stierwalt, Bachiochi, Elder, & Fisher 1999; Mason &
Belt, 1986), attributes of jobs and organizations (e.g., Barber, Wesson,
Roberson, & Taylor, 1999; Cable, Aiman-Smith, Mulvey, & Edwards,
2000), and individual differences in job seekers (e.g., Judge & Cable,
1997; Wanberg, Kanfer, & Banas, 2000). Status maintenance research
has examined applicants’ reactions to various screeninghelection tools
(e.g., Bretz & Judge, 1994; Ryan, Sacco, McFarland, & Kriska, 2000)
and the role that the recniiter plays in helping or hindering the mainte-
nance of applicant status (e.g., Connerley & Rynes, 1997; Stevens, 1997).
Job choice (i.e., acceptance) research has examined the role of congru-
ence between personal and organization values (e.g., Cable & Judge,
1996; Judge & Bretz, 1992) and applicants’ affinity for various job and
organizational factors (e.g., Saks, Wiesner, & Summers, 1996; Turban,

Campion, & Eyring, 1995). To gain acceptance of offers organizations
have used tactics like signing bonuses, “exploding” (i.e., time limited)
offers, or increased salary, benefits, and perquisites.
Separating and distinguishing among the activities intended to influ-
ence attraction, status maintenance, and job acceptance is possible be-
cause these outcomes are temporally sequenced. That is, they occur in
a specific order. Even in rapidly evolving recruitment cycles (i.e., on-
line recruitment), the boundaries between the attraction, status main-
tenance, and gaining job acceptance phases of recruitment are easily
identified (i.e., closing applications, extending offers). This temporal se-
quencing also assists in the appropriate alignment of recruitment activ-
ities with recruitment phases. For example, an activity cannot influence
an outcome that occurs prior to that activity. Further, many recruitment
activities primarily influence only one of attraction, status maintenance,
or job acceptance outcomes. Even though there can be spillover effects,
most recruiting activities, and arguably those that have the strongest ef-
fects, are designed to influence a specific outcome. For example, in pro-
fessional sports the potential to earn a signing bonus may attract more
entrants into professional sports drafts (i.e., the equivalent of applying),
but their primary purpose is to influence acceptance of a contract. Al-
ternatively, in organizations that offer a specific level of signing bonus
that is made available to all new hires, that signing bonus is primarily
designed to attract applicants, rather than to affect job acceptance.
Organizations engage in two general types of recruitment. “Batch”
recruiting involves engaging in separate recruitment activities to develop
a new applicant pool each time an organization has one or more posi-
tions to fill. Activities are easily matched to outcomes in batch recruiting
because there is only one set of attraction, status maintenance, or job ac-
ceptance outcomes. In “continuous” recruiting, organizations engage in
sustained efforts to attract applicants over longer periods of time to meet
an ongoing need to fill new positions generated by business growth, or ex-
isting positions vacated by employee promotions, transfers, or turnover.
Unlike batch recruiting, attraction and status maintenance activities in
continuous recruiting are not tied to specific hiring events. Instead, at-
traction and status maintenance activities must be aligned with the rele-
vant applicant flow. For example, a radio campaign to attract applicants
for positions would be matched to changes in the flow of applicants into
the organization during the period of time when the effects of the ad
campaign are expected to occur. Organizations that adopt continuous
recruitment systems often also use “cut-scores” to make hiring decisions.
Rather than hiring top down from within the applicant pool to fill a fixed
number of positions, cut scores depict “minimum qualifications” for the
position and any individual that achieves a score at or above the cut score
is assumed to be qualified for hire.
The importanceof attractionoutcomes. Although each of the three re-
cruitment processes is important, attraction activities occur first and are
the most crucial for determining recruitment and staffing success. At-
traction activities establish the pool of applicants from which new hires
will eventually be chosen. If top candidates do not apply, an organiza-
tion has no chance of hiring them. Thus, the maximum potential value of
a recruiting cycle (i.e., the contribution the best applicants could make
to organizational effectiveness) is fixed once the applicant pool is es-
tablished. To assess attraction outcomes, the relevant applicant pool in
continuous recruitment approaches is all applicants of record at the time
of decision making that have not already been hired, irrespective of their
current status. An alternative is to limit the definition to only those ap-
plicants that applied during some specified period of time (e.g., the pre-
vious year).
The maximum potential in an applicant pool is realized only if the
best applicants in the pool are offered, and then accept, positions. Any
other outcome (is., the failure to retain top candidates in the pool or
to gain their acceptance of offers of employment) represents a loss of
potential value. Status maintenance and gaining job acceptance activ-
ities influence the effectiveness of recruitment by reducing or avoiding
the loss of potential that can occur when the best applicants do not join
organizations. However, neither status maintenance nor efforts to in-
fluence job choice can raise the potential contribution of a recruitment
cycle beyond what is present in the applicant pool. The same is true
of selection activities. Selection practices minimize losses of potential to
the extent they ensure hiring mistakes are minimized. The use of devices
with less than perfect validity (T = l.O), though, assures that some loss
will occur. But the best that even perfectly valid selection methods can
do is to identify the best applicants in the pool-it cannot make them bet-
ter. llius, the first priority of recruitment should be attracting the best
possible applicants because attraction outcomes establish the maximum
contribution that is possible in any staffing system. Even heroic efforts
in status maintenance, selection, gaining job acceptance, or employee
retention cannot overcome poor attraction outcomes.
Evatuating Attraction Outcomes
Evaluating organizations’ effectiveness in each recruitment phase

would permit areas of strength or weakness to be isolated and would
allow specific causes of recruitment failures to be more readily identi-
fied. Further, current global assessments of recruitment effectiveness,
like assessments of the job performance of new hires, are not useful
diagnostic tools; changes in these assessments cannot be aligned with
recruitment phases. As a result, whether poor recruitment outcomes
are due to not attracting top candidates, ineffective methods for keep-
ing candidates interested until offers can be made, an inability to sign
top candidates, or some combination of these factors, can not be deter-
mined. Knowing which outcomes are poor would allow an organization
to direct its improvement efforts to those areas with the greatest poten-
tial for improving overall effectiveness.
Attraction outcomes are assessed in some organizations. Currently,
the most common method for evaluating attraction outcomes is count-
ing the number of applicants generated. Fifty percent of organiza-
tions that reported performing formal assessments of recruitment in a
SHRM/CCH (1999) survey indicated they assessed number of applicants
generated for at least some positions. Number of applicants generated
is a simple and relatively inexpensive assessment that provides a use-
ful initial indicator of attraction effectiveness. It determines whether
sufficient numbers of applicants exist to fill open positions-a primary
recruitment goal. Failing to attract enough applicants to fill open posi-
tions results in recruitment failures. Recruitment failures are expensive,
potentially requiring duplication of recruitment costs and extending the
length of position vacancies.
The number of applicants attracted also provides an indirect assess-
ment of applicant quality. If more applicants are generated than there
are positions to be filled, organizations can be selective. They can offer
positions to individuals likely to be high performers, avoiding candidates
that are less likely to perform well on the job. The larger the applicant
pool, the more selective the organization can be, and increased selec-
tivity is assumed to result in better hires and subsequent increases in
organizational performance. However, in the case of applicants, more
is not always better. As the number of applicants increases so do the
costs of administering recruitment and selection systems. Larger appli-
cant pools have more applicants to track, correspond with, and screen,
raising costs and potentially extending the time required to fill vacant
positions. A tradeoff exists. Increasing the number of applicants is valu-
able as long as the gains from being more selective are large enough to
offset increased costs.
Brogden (1949) proposed that for a given number of hires, the gains
expected from greater selectivity appear to diminish as the size of the
applicant pool increases and, therefore, an optimal applicant pool size
exists for each recruiting event. That optimal applicant pool size is a
function of the rate of selectivity and cost is based on two assumptions:
(a) job performance potential is normally distributed within the appli-
cant population (i.e., all individuals in the workforce minimally quali-

fied to hold a given position) and (b) applicant pools are random draws
from applicant populations. That is, the distribution of job performance
potential in each applicant pool is thought to mirror that of the larger ap-
plicant population and, therefore, to also have the properties of a normal
distribution.
Real applicant pools, though, are unlikely to have the idealized prop-
erties of normal distributions assumed in Brogden’s (1949) calculations.
Sampling theory tells us that each applicant pool, even if it is selected
at random, is unlikely to mirror the properties of the overall popula-
tion. The smaller the sample, the greater those deviations can be. Fur-
ther, many recruitment efforts are systematic attempts by organizations
to cause applicant pools to differ from the larger applicant population
(i.e., contain higher percentages of the most desirable candidates). Or-
ganizations strive to attract those individuals that are likely to be best
suited to their organization. Highhouse et al. (1999) and Mason and Belt
(1986) suggest that organizations can use the content and specificity of
recruitment messages to change the characteristics of applicant pools. In
addition, research on the characteristics of both jobs and organizations
suggests that individuals appear to be systematically and differentially
attracted to certain types of jobs and organizations (Saks et al., 1996;
Turban et al., 1995).
If applicant pools do not possess these expected properties, then it
cannot be assured that increasing the size of applicant pools will result
in higher quality hires, or that even larger applicant pools are better than
smaller ones. Assessments of applicant quantity are imperfectly related
to applicant quality; therefore, their usefulness for assessing and improv-
ing attraction effectiveness is limited. The only way to detect specific
changes in the quality of applicants attracted is to assess it directly.
Developing Quality Scores for All Applicants
Directly estimating applicant quality avoids relying on assumptions

about the distribution of job performance potential in applicant pools
and permits both the levels and distribution of applicant performance
potential in a pool to be evaluated. Applicant quality scores estimate
each applicant’sposition in the distribution of performance potential for
a given position. Ideally, these assessments not only indicate an indi-
vidual’s position in relation to others in the applicant pool, but against
the broader applicant population. When data referenced to the appli-
cant population are available, they can then be used to directly compare
applicants, even if they are generated at different times, from differ-
ent sources, or with different practices. This is particularly useful for
geographically and temporally dispersed recruitment efforts that gener-

ate candidates for organization-widepositions (e.g., college recruitment
for entry-level management positions) or to systematically evaluate the
effectiveness of alternative attraction strategies.
Few organizations currently develop quality scores for all applicants,
but all organizations engage in some form of personnel selection-the
fundamental practice on which these assessments are based. As a cost
saving measure, most organizations engage in some form of multiple
hurdle selection. Multiple hurdle selection systems involve the use of
several valid selection devices that are administered sequentially with
low scoring candidates dropped from consideration following the admin-
istration of each successive selection device, thus costs for administering
subsequent devices are reduced. In multiple hurdle selection systems,
the only device that is used to evaluate all candidates is the initial selec-
tion device (i.e.7often described as a screening or prescreening device).
Organizations vary in their choice of initial screening devices. For ex-
ample, many organizations use an evaluation of the data provided on a
resume or application blank for this purpose. These devices offer the
advantages of low cost and easy administration, though their validity is
often not known.
Unfortunately and irrespective of the device used, the output of ini-
tial screening is often simply the sorting of applicants into “retain for
further consideration” or “reject” categories. Multiple hurdle selection
systems use a series of these decisions to determine which applicants will
be offered positions. Retaidreject data, though, possess only ordinal
properties (i.e., better candidates are retained, those ranked lower are
rejected) and do not differentiate among candidates within categories
or permit descriptions of applicant pool characteristics beyond quantity
of applicants retained and rejected. These data are sufficient to accom-
plish selection, but do not describe how well positions are being filled.
Initial screening devices that yield ordinal data cannot be used to deter-
mine whether an opportunity to improve attraction outcomes exists or
how great that opportunity might be. This requires a quality score for
each applicant.
A New Approach
What we are suggesting is extending and formalizing organizations’

candidate evaluation activities in order to develop predictor (i.e.7 qual-
ity) scores for each applicant before less qualified applicants are re-
moved. The availability of these scores permits direct assessment of
attraction outcomes and is the foundation of this approach. A 7-step
procedure for adopting this approach is outlined in Appendix A.
Step One: Identifi Positions to Assess

Organizations should begin by assessing attraction outcomes for those
positions where assessment offers the greatest potential returns. Attrac-
tion outcomes for all positions are important and the method suggested
here can be applied to any position in any organization using any staffing
approach. However, positions that generate the greatest recruitment
and hiring activity may offer greater opportunities for return from these
efforts. Level of hiring activity, though, is only one measure of opportu-
nity. Opportunity for improvement is also based on how much improve-
ment can be made in the quality of applicants attracted. Unfortunately,
those positions that offer the greatest opportunity for improvement may
not be identifiable in the absence of assessments of applicant quality.
Step Two: Identifi and Evaluate Current Screening Mechanism

Every selection system begins with one or more initial selection de-
vices (i.e., screening or prescreening devices) that are administered to
all applicants. In most organizations, this is the first step in a multiple
hurdle selection process. For the purposes of evaluating recruitment
outcomes, ideal screening devices will (a) employ scoring procedures
that reference individual scores against the applicant population, (b)
permit adequate differentiation of applicants, (c) possess interval mea-
surement properties, and (d) have high validity for predicting job per-
formance. Referencing scores to that applicant population is consistent
with calibrating ratings so that individuals are not rated relative to oth-
ers in the applicant pool, but against any potential applicant that could
have applied. Permitting adequate differentiation means that sufficient
gradations in assessments exist to capture meaningful differences in job
performance potential. Traditional performance assessments using five
levels of performance are too coarse. They permit too few gradations
of performance to capture subtle or even moderately large differences
in job performance. Interval properties allow us to be able to quantify
differences between candidates. All existing selection devices have less
than perfect validity (i.e., T < 1.0) and result in losses of utility. Using
devices with the highest validity possible minimizes these losses.
Table 1 lists a number of selection devices that could be used as the
initial screening device in selection systems. An estimate of the validity
of each device for predicting job performance is provided. This list in-
cludes both measures of individual difference constructs (i.e., cognitive
ability, integrity) and selection methods (i.e., work sample tests, struc-
tured interviews, and biographical data). Standardized tests offer the
possibility of moderate to high validity for a wide range of positions
(Schmidt & Hunter, 1998). These might include tests of general mental
470 PERSONNELPSYCHOLOGY
TABLE 1
Validityof h m p l e s of Selection Devices that Could be used to
Initially Screen Applicants
Selection device Validity for predicting

job performance
Constructs
General mental ability tests" .51
Conscientiousness tests" .31
Integrity tests" .41
Methods
Work sample tests' .54
Job knowledge tests" .48
Structured interviews' .51
Biographical data9 .35
Grade point average" .23
Ratings of training and experience" .ll
Hunter (1980)
'w- Ones, Viswesvaran, Schmidt (1993)
'L' Hunter & Hunter (1984)
f McDaniel, Whetzel, Schmidt, & Mauer (1994)
'1 Rothstein, Schmidt, Erwin, Owens & Sparks (1990)
" Roth, BeVier, Switzer, & Schippmann (1996)

* McDaniel, Schmidt, & Hunter (1988)
ability (T = .51) or integrity (T = .41). Work on biographical data instru-

ments, though, suggests that it may be possible to achieve validities as
high as T = .35.In addition, research has shown that GPA has a validity
for predicting the job performance of new hires (T = .23; Roth, BeVier,
Switzer, Schippmann, 1996).
Basing initial screening decisions on resumes or application blanks,
as done in many organizations results in the collection of very limited
information on which to initially screen applicants. A potential concern
about constructing estimates of applicant quality on such limited infor-
mation is that these are likely to have relatively low validity (i.e., similar
to the evaluation of training and experience, T = .11). Although the use
of low validity selection devices is not recommended, using low validity
as a justification would be logically inconsistent. How could an organiza-
tion justify a decision to refuse to use scores on current screening devices
to assess attraction outcomes due to concerns about their validity yet
continue to use these same devices to make actual screening decisions?
If the validity of a current screening device is a concern, the organization
should consider an alternative device. This is discussed in Step 7.
Some organizations have developed standardized instruments or em-
ploy professionally developed tests as screening devices. In the military,
the ASVAB (Armed Services Vocational Aptitude Battery) is given to
all recruits. Although scores on this instrument are used primarily in
placement decisions, scores on the ASVAE3 could also be used to eval-

uate attraction outcomes. The postal service offers two exams for their
major entry-level positions that offer similar opportunities for evaluating
attraction outcomes. Test 460 is used to select rural carrier associates,
and Entrance Battery 470 is given to all applicants for distribution, de-
livery, and processing positions. The advantage of these instruments is
that the validity of these methods for predicting job performance is gen-
erally already known and national norms for the means and distributions
of scores already exist.
Step Three: Determine Strategy for Adapting Current Screening Device
(if necessary)
As indicated in Step 2, some commonly used screening devices do
not provide scores with the desired properties. The amount of time
and effort required to adapt current screening devices to produce scores
with the desired properties will vary. For most screening devices, some
adaptation will be necessary and this will require some amount of time
and effort. Therefore, organizations will need to decide how far they
want to go toward developing measures that yield scores with optimal
properties. It is not critical that organizations move directly to measures
that produce scores with optimal properties. Even measures that only
roughly approximate these values will be useful. A rough estimate is
better than no estimate. For example, organizations using dichotomous
retainheject decisions in their initial screening decisions could begin by
adjusting their screening practices to assign a quantitative score from
1 to 100 to each applicant. In some cases, this will involve codifymg
selection rules and mechanisms that may currently be known only to the
screeners. Organizations could then adapt measures over time to more
closely approximate scores with the desired properties.
Developing appropriate measures of applicant quality involves two
types of costs. The first is the fixed cost associated with adapting an ex-
isting screening device to produce scores with more appropriate char-
acteristics. A second is the variable cost associated with administering
the screening device and generating the applicant score. Both are rele-
vant to our decisions to implement this approach. Organizations should
monitor the costs of converting an existing screening device to generate
appropriate applicant quality scores and any changes in the variable cost
of implementing that method. Any screening device that requires adap-
tation will likely increase both types of costs, but these costs need not
be large. However, the closer scores approximate ideal properties, the
greater costs are likely to be. These costs should be evaluated against the
value of the benefits possible from improved recruitment effectiveness.
Step Four: AssessingAttraction Outcomes

Assess each applicant using the device developed in Step 3. Scores
should be developed for all applicants in the applicant pool. To evaluate
attraction outcomes in both batch and continuous recruiting, an appli-
cant is any individual that applied for a specific position, whether they
maintain their applicant status until hiring decisions occur or not. As-
sessment of applicants can occur at any point after the applicant enters
the applicant pool until screening decisions are made and low scoring
applicants are dismissed, Conducting assessments earlier in the process
permits concurrent evaluation of attraction outcomes. Data from the
analysis of applicant flows for each position should be archived in or-
der to facilitate comparisons across recruiting events and refinement of
assessment instruments.
Step Five: Matching Recruitment Activities to Recruitment Phases and
Estimating Costs
To assess attraction outcomes, it is important to capture the changes
in the quality of applicants attracted, as accomplished in Step 4,and to
also capture the costs of the activities required to generate those appli-
cants. Recruitment costs must be identified and mapped to the appro-
priate phase of recruitment (i.e., attraction, status maintenance, gain-
ing job acceptance). This requires identifying those activities directly
affecting attraction-as opposed to other recruitment outcomes-that
occurred during the period in question. For example, in college recruit-
ing, costs for campus visits to meet with placement directors and faculty,
attendance at job fairs, and the development of Web content describ-
ing opportunities within an organization are costs of attraction. Costs
of “follow-up” correspondence with applicants or establishing “self-
service” systems that allow candidates to track the progress of their can-
didacy are status maintenance costs. The cost of signing bonuses devel-
oped for specific candidates, phone calls or other correspondence de-
signed to gain offer acceptance, or the costs of any additional onsite
visits after the offer has been extended are job acceptance costs. The
cost of general recruitment activities-those not attributable to a spe-
cific position-should be evaluated against all positions influenced by
those activities and not charged to a single position.
Step Siu: Evaluating Attraction Outcomes Using UtilityAnalysis

Direct assessments of applicant “quality” provide data on the esti-
mated level and dispersion of job performance potential that exist in a
given applicant pool. These data are superior to simple quantity mea-
sures for evaluating attraction outcomes and, on their own, can sup-
port comparisons of applicants and applicant pools. However, these data

are still not capable of supporting analyses of the practices that produce
these outcomes. Recruitment practices influence “quality” outcomes,
but they also affect costs. Analyses of alternative recruitment practices,
therefore, require that the effects of attraction practices on quality and
cost be considered. Unambiguous interpretations of the effects of al-
ternative attraction practices are possible only when quality and cost are
reported in the same metric. This requires that one measure or the other
be converted.
Although it does not permit direct conversion of quality scores to dol-
lars, Utility Analysis (UA) comes close, providing a means of converting
“change” in applicant quality scores to a dollar metric. The following
equation1, provides a dollar-valued estimate of the difference in utility
resulting from two alternative attraction processes:
A U = r x y x S D y x A Z X x T x N - AC PI
In Equation 1, AU is the estimated difference in utility in dollars.
r x y is the validity of the applicant quality measure for predicting job
performance. SDY is the standard deviation of job performance in dol-
lars. AZx is the change in the average standardized applicant quality
measures resulting from different attraction processes. T is the average
number of years a new hire is expected to remain in his or her position.
N is number of individuals to be hired from the applicant pool. AC rep-
resents the difference in the total cost of the two attraction processes.
Prior research has examined the use of UA to convert differences in
recruitment outcomes to a dollar metric. Boudreau and Rynes (1985)
used a hypothetical recruitment example to demonstrate how differ-
ences in the level of applicant quality and range restriction in applicant
pools could affect overall utility and the validity of selection devices.
Murphy (1986) used UA to demonstrate how a failure to account for job
‘Boudreau’s (1991) Equation 7 is AU = ( N s ) ( S D y ) ( T : , : ~(Zx)

) - , ( C ) ( N a p p ) , where
A U represents the change in utility in dollars, N s is the number of individuals selected,
S D y is the standard deviation of performance in dollars, T . T . ~is, the validity of the selection
device for predicting job performance, Zx is the average standardized predictor score of
the selectees, C is the cost of using the selection per person, and N a p p is the number
of applicants. This equation expresses the total gain from using the predictor to select
N selectees (as compared to selecting randomly). We modified this equation in three
ways. First, we replaced Zx with A Z X . Standardized predictor scores of those hired
are hypothesized to vary across applicant pools. Second, we added tenure (T)to the
multiplicative model. Boudreau’s Equation 7 provides the utility for a single year of new
hire performance. However, the extra value contributed by these hires continues for as
long as they remain in those positions. T allows the effects of tenure other than one year
to be accounted for in the eqLation. Third, we replaced the multiplicative product (C)
( N a p p ) with AC. The focus of this equation is capturing the effects of differences in
recruitment. The term AZx captures the change in benefits. A C is conceptualized as
the change in costs associated with any differences in activities that caused A Z X
acceptance losses can lead to overestimates of utility. These examples,

though, did not employ direct estimates of applicant quality, relying in-
stead on standard assumptions that performance potential is distributed
normally in applicant populations.
Sometimes values for some variables in Equation 1 may not be known
(ie, the validity of a screening procedure) In those instances, those
values must be estimated in order to conduct UA. An estimate of a
screening device’s validity can be developed by identifying a device with
a known research-basedvalidity estimate that most closely approximates
the properties of the device used by the organization. The similarity of
devices can be used to extrapolate from the validities of known devices
to estimate the unknown validity for use in UA. This method will provide
a reasonable approximation until more definitive evidence of validity is
available.
Step Seven: Evaluating the Adequacy of Current Screening Devices

We suggest that, as a starting point, organizations use the methods
they currently employ for applicant screening to develop applicant qual-
ity scores. This minimizes any changes in current organization practice
and costs. However, organizations are encouraged to screen applicants
with the most valid device that can be cost justified. Higher validity de-
vices, or batteries of devices, reduce selection errors and, as a result,
increase utility. Determining whether it would be more advantageous to
use the current selection/screening method or an alternative device is a
decision that can also be addressed using UA methods (e.g., Boudreau,
1991; Schmidt, Hunter, McKenzie, & Muldrow, 1979). These decisions
require that both the cost and benefit consequences of the two alter-
native measures be considered. The better alternative offers the more
optimal combination of validity and cost. The decision to adopt a dif-
ferent screening device is separate from the decision to assess attraction
outcomes. Each has its own unique cost and benefit consequences.
In organizations that process large numbers of applicants, even if
the cost of developing an appropriate quality score is minimal per ap-
plicant, the total cost for the recruiting event could be substantial. One
method for reducing these costs is the expanded use of online applica-
tion and testing processes. Test automation and online testing are alter-
native delivery methods for administering screening and selection mech-
anisms. Existing research finds that online test administration can pro-
duce equivalent validity to paper-and-pencil administration (e.g., Ford,
Vitelli, & Stuckless, 1996;Lunz & Deville, 1996; Segall & Moreno, 1999;
Vispoel, Boo, & Bleiler, 2001; Vispoel & Coffman, 1994). This assumes
that appropriate controls are in place to avoid cheating. An important
cost advantage for online tests is that scores can be calculated much more
quickly. In addition, for automated evaluation of resumes or applica-

tions based on rules developed by recruiters, computers can perform this
operation far more quickly and consistently than human evaluators. This
can be critical in situations where hundreds or even thousands of appli-
cants must be screened. In addition, these systems can automatically
upload resume data to integrated human resource information systems
(HRIS) that can reduce the cost of other components of administering
hiring systems.
In summary,we recommend organizations assess attraction outcomes
using estimates of the job performance potential (i.e., quality) of each
applicant, and then use utility analysis to convert differences in attraction
outcomes to a dollar metric so they can be directly compared to costs. In
the next section, we demonstrate the assessment of recruitment benefits
using actual attraction outcomes from a college recruitment setting.
A Recruitment Evaluation Example

The applicant pools examined were generated for actual on-campus
job interviews with corporate recruiters at a southeastern university in
the fall of 1999. The data are for five full-time, permanent positions
with the job title Associate Engineer. Each position was with a different
organization; the organizations chosen represented a range of sizes and
industries. These five positions attracted a total of 380 applications.
Quality Measure
Each student applicant submitted a copy of his or her resume to be
reviewed by recruiters. Recruiters used this information to determine
which applicants would receive on-campus interviews. The resume was
the only information about the applicants available for making this ini-
tial screening decision. To develop applicant quality scores for our anal-
ysis, two different quality measures were considered-(a) a resume rat-
ing scale, similar to a weighted application blank and (b) the applicant’s
overall college grade point average (GPA). We chose to use GPA as the
applicant quality measure for this analysis. Anecdotal evidence suggests
that GPA is used as an initial screening tool by at least some organiza-
tions. In addition, unlike the resume rating scales, population norms for
GPA are known and meta-analytic estimates of the validity of GPA for
predicting job performance exist in the literature (i.e., rZy = .23; Roth
et al., 1996).
Form of the quality measure. Once individual quality score estimates
for all applicant pool members are developed, a question arises con-
cerning how individual quality score data should be aggregated to most
appropriately depict attraction outcomes. Hawk (1967) uses the mean
and standard deviations of all applicants as the appropriate descriptive

statistics, as do Sackett and Ostgaard (1994) in their analysis of Won-
derlic Personnel Test scores across applicant pools. An alternative ap-
proach is to base assessments of attraction outcomes on a subset of the
highest scoring individuals. Using data for only the highest rated appli-
cants is consistent with general recommendations for hiring top down
within applicant pools and traditional utility analyses (e.g., Boudreau,
1991), where only the standardized predictor scores (2,) of individuals
who are hired by the organization influence utility calculations. Total
number of applicants is still a useful metric, though, since individuals
that are not selected only reduce utility by increasing selection costs.
If subsets of quality scores are used, how many scores are most ap-
propriate? Depicting attraction outcomes using scores for the number
of positions to be filled may be too limiting. It does not take into account
the potentially negative effects of status maintenance and job choice out-
comes. A more appropriate measure may include a somewhat larger
subset of top applicants, for instance, the number of individuals likely to
pass initial screening or some multiple of the number to be hired. This
would limit quality estimates to the high scoring individuals from which
hires are likely to come and incorporate enough applicants to recognize
the effects of any status maintenance and job acceptance losses.
In order to compare these different levels of data aggregation, we
developed three forms of our quality measure. We created a measure
called “ALL,” consistent with those used in Hawk (1967) and Sackett
and Ostgaard (1994), that included quality score data for all applicants in
each pool. A second measure called “TOP15” included only data for the
applicants with the 15 highest quality scores. The number 15was chosen
because it matched the average number of individuals that typically pass
initial screening and receive interviews from each applicant pool; on-
campus interview schedules typically include 12-16 interviews. Third,
we constructed a measure called “TOP2” that included data for only the
two applicants with the highest quality scores. This is consistent with our
illustrative example that assumes two individuals are hired.
Utiliry Calculations
In order to use UA to convert differences in attraction outcomes to
a dollar metric, it was necessary to provide estimates of several values
required by Equation 1. These values are illustrative and the actual val-
ues used by organization decision makers would depend on the charac-
teristics of each recruitment event. In this example, we chose to hold
r x y , SDy, N , T and A C constant across all analyses. We set r x y = .23
based on the results of a meta-analysis of GPA validity data by Roth
et al. (1996). This is the estimated validity of GPA for predicting job
performance corrected for error of measurement in the criterion. We

conservatively estimated the average starting salary for associate engi-
neer positions at $38,000 per year and then set SDY equal to 40% of
starting salary (Schmidt et al. 1979). We also chose conservative values
for number of applicants hired (i.e., N = 2) and the tenure of new hires
(i.e., T = 1).
The focus of our evaluation is on the differences in applicant quality
across applicant pools (AZZ).These were calculated by determining the
difference between the appropriate standardized quality scores (mean
2,)for the comparisons of interest. Values were standardized using data
for the mean and standard deviations of GPAs for all senior students at
the university (i.e., A4 = 2.80, SD = 0.54). Individual’s ZZscores were
calculated using Equation 2 and averaged for all individuals included in
each measure (e.g., ALL, TOP15, TOP2).
ZZ= (quality score - 2.80)/0.54 PI
A note on attraction costs. This example demonstrates methods for
assessing the benefits component of attraction outcomes. We do not in-
clude specific attraction cost data because these data were not available
to us. This does not materially affect the interpretation of the utility val-
ues generated for these analyses. This is true because the relevant cost
estimate in Equation 1 is the difference in cost between alternative at-
traction practices (i.e., AC), not total cost. In our analyses, we assigned
a hypothetical “mean” total cost of attraction value to each applicant
pool. The difference in these cost values results in AC = 0 for all anal-
yses. Although it is unlikely that there are no differences in the actual
attraction costs incurred by these organizations, there is no way to esti-
mate the magnitude or the sign of any difference between the actual AC
values and our estimates.
It is interesting to note that although cost of recruitment is assessed in
many organizations, the approach discussed here requires that recruit-
ment costs be appropriately decomposed and allocated to attraction, sta-
tus maintenance, or efforts to gain job acceptance. We were unable to
identify an estimate of attraction cost in the recruiting literature. Com-
mon measures of costs, like cost per hire (SHRM/CC€I, 1999), typically
combine attraction, status maintenance, gaining job acceptance, and, in
many instances, selection costs. These processes are different and, es-
pecially in the case of selection, they contribute to staffing system effec-
tiveness in different ways; these costs should be separated.
Results of the Analysis for the Illustrative Example
Table 2 provides an overview of the five associate engineer applicant

pools used in this analysis. The columns of Table 2 list the Job ID, total
TABLE 2
All TOP15 and TOP2 Quality Scores for Five Associate Engineer Applicant Pools
Quality scores
Job Number All TOP15 TOP2 TOP15
ID of resumes Mean (SD) [Rank] Mean (SD) [Rank] Mean (SD) [Rank] quality scores
A 156 3.31 (0.471) [2] 3.97 (0.030) [ 11 4.00 [l] 4.00 4.00 4.00 4.00 4.00 3
4.00 3.98 3.97 3.97 3.95
3.95 3.95 3.93 3.93 3.92
B 105 3.35 (0.435) I11 3.92 (0.046) [2] 3.99 12) 4.00 3.98 3.97 3.95 3.95
3.94 3.93 3.90 3.90 3.90
3.90 3.90 3.87 3.86 3.84 r
C 69 3.18 (0.425) [3] 3.73 (0.129) [3] 3.94 [4] 3.96 3.92 3.90 3.86 3.84
P
2
3.74 3.70 3.70 3.69 3.67 2
3.62 3.61 3.60 3.60 3.60
D 28 3.16 (0.484) [4] 3.51 (0.338) [4] 3.97 [3] 4.00 3.95 3.90 3.84 3.80 8
r
3.18 3.60 3.38 3.30 3.20 0
3.20 3.20 3.19 3.16 3.10
E 22 2.91 (0.386) [5] 3.10 (0.283) [5] 3.65 [5] 3.78 3.53 3.42 3.20 3.14 9
3.10 3.09 3.04 3.03 3.00
2.90 2.90 2.84 2.80 2.78
number of resumes in each applicant pool, means and standard devia-

tions for the ALL, TOP15, and TOP2 quality measures, and a detailed
listing of the highest 15 quality scores (i.e., GPAs) in each applicant pool.
Comparing the three quality score measures demonstrates that these five
applicant pools vary in both the mean quality scores of applicants and
also in the distribution of those scores. Based on scores for TOP15 and
TOP2, but not ALL, Job A has the highest average score and lowest
variability of scores. In general, the larger applicant pools produced the
highest mean quality scores, but the relationship of size to quality is not
linear. Some pools with substantial differences in size had similar qual-
ity scores. In addition, the ranking of mean quality changed across forms
of the quality measures.
Table 3 reports differences in attraction outcomes among the five ap-
plicant pools. These comparisons examine differences in standardized
quality scores for all applicant pools using both TOP15 and TOP2 quality
measures. The associated difference in utility in dollars for each com-
parison is also reported. The values in bold type above the diagonal in
Table 3 are based on TOP15 quality scores and those below the diagonal
are based on TOP2 quality scores. Differences in expected utility using
TOP15 measures ranged from a high of $11,257.12 (ie., A-E) to a low of
$699.20 (i.e., A-B). Differences were somewhat smaller for TOP2 mea-
sures, ranging from a high of $4,474.88 (ie., A-E) to negative $489.44
(Le., C-D).
Next, an additional analysis was conducted to demonstrate how
attraction outcomes influence subsequent status maintenance and job
choice outcomes. As noted earlier, status maintenance and job accep-
tance activities cannot increase utility, they can only limit potential losses
by assuring top applicants become employees. In this analysis, we be-
gin with the actual attraction outcomes for these organizations and then
simulate several combinations of status maintenance and job acceptance
outcomes to examine their effects on utility estimates.
Analyses of the attraction, status maintenance, and job acceptance
outcomes for each applicant pool are presented from left to right in
Table 4. Three values are presented for each recruitment phase-the
standardized quality Score for the best two individuals available f o h w -
ing that process (Z), the difference or change in TOP2 quality Scores
attributable to that process (AZ), and the utility impact ofthe change in
quality scores ($U).
Attraction outcomes are the between-applicant pool comparisons
drawn from Table 3 representing the difference in TOP2 attraction out-
comes for each job as compared to Job A (i.e., the highest ranking aP-
plicant pool based on TOP2 quality scores). For each applicant Pool, all
possible combinations of three levels Of Status maintenance performance
TABLE 3
Comparison of Attraction OutcomesAcross Applicant Pools: Differences in Standardized Quality Scores
and Utilityfor Top15 and Top2 Quality Measures
Standardized qualitv scores Job ID

Job ID Measure Mean A B C D E
A TOP15 2.17 (AZ) .10 0.44 0.86 1.61
TOP2 2.22 (W 699.20 3,076.48 6,013.12 11,257.12
B TOP15 2.07 .02 (AZ) 34 .76 .51
TOP2 2.20 129.48 W) 2,371.28 5,313.92 10,577.92
C TOP15 1.73 .11 .09 (AZ) A2 1.17
TOP2 2.11 776.89 629.28 (W 2,936.64 8,180.64
D TOP15 1.31 .05 .02 -.07 (4 .75
TOP2 2.18 323.70 139.84 -489.44 (W 5,244.00
E TOP15 .56 0.64 .62 .53 .60 (AZ)
TOP2 1.58 4,473.59 4,335.04 3,705.76 4,195.20 W)
Notes: A-E represent different applicant pools for associate engineer positions. TOP15 and TOP2 are two different quality
measures based on the highest 15 or highest two quality measures respectively. A Z refers to the difference in standardized
quality scores between two different applicant pools. BU is the utility difference in dollars associated with the A 2 score
immediately above it. Values in bold type (i.e., those above the diagonal) are based on TOP15 quality measures. Those
values below the diagonal are based on TOP2 quality measures. In each calculation, the value of the later-lettered job is
subtracted from the value of the earlier lettered job (i.e., A-B).
TABLE 4
Attraction, Statics Maintenance and Job Acceptance Effects on Standardized Quality Measures and Utility by Applicant Pool
Attraction Status maintenance Job acceptance Overall
JobID Z AZ $lJ Perf. z AZ $U Perf. Z A2 SU impact ($)
2.22 .oo 0.00 Ideal 2.22 .OO 0.00 Ideal 2.22 .OO 0.00 0.00
Mod. 2.22 .OO 0.00 0.00
Poor 2.22 .00 0.00 0.00
Mod. 2.22 .oo 0.00 Ideal 2.22 .00 0.00 0.00
Mod. 2.22 .00 0.00 0.00 PI
Poor 2.22 .00 0.00 0.00
Poor 2.22 .00 0.00 Ideal 2.22 .oo 0.00
Mod. 2.22 .00 0.00 0.00
0.00
2?
Poor 2.18 -.05 -323.70 -323.70
2.20 -.02 -129.48 Ideal 2.20 .oo 0.00 Ideal 2.20 .OO 0.00 -129.48
Mod. 2.19 -.01 -64.74 - 194.22
Poor 2.15 -.06 -388.44 -517.93
8tl
Mod. 2.19 -.01 -64.74 Ideal 2.19 .OO 0.00 -194.22
Mod. 2.18 -.02 - 129.48 -323.70 zo
Poor 2.12 -.07 -517.93 -712.15
Poor 2.16 -.05 -323.70 Ideal 2.16 .00 0.00 -453.19
Mod. 2.15 -.01 -64.74 -517.93 ;
Poor 2.07 -.08 -582.67 -1,035.85
2.11 -.11 -776.89 Ideal 2.11 .oo 0.00 Ideal 2.11 .OO 0.00 -776.89
Mod. 2.09 -.02 -129.48 -906.37
Poor 1.98 -.13 -906.37 -1,683.26
Mod. 2.09 -.02 - 129.48 Ideal 2.09 -.00 0.00 -906.37
Mod. 2.06 -.04 -258.96 -1,165.33
Poor 1.85 -.24 -1,683.26 -2,589.63
Poor 2.02 -.as -647.41 Ideal 2.02 .00 0.00 -1,424.30
Mod. 1.91 -.11 -776.89 -2,201.19 P
Poor 1.68 -.34 -2,395.41 -3,819.70 ?2
TABLE 4 (continued)
Attraction Status maintenance Job acceptance Overall
JobID Z AZ $U Perf. Z A2 $U Perf. Z AZ $U impact ($)
D 2.18 -.05 -323.70 Ideal 2.18 .OO 0.00 Ideal 2.18 .OO 0.00 -323.70
Mod. 2.13 -.05 -323.70 -647.41
Poor 1.91 -.23 -1,618.52 -1,942.22
Mod. 2.13 -.05 -323.70 Ideal 2.13 .oo 0.00 -647.41
Mod. 2.07 -.06 -388.44 - 1,035.85
Poor 1.87 -.26 -1,812.74 -2,460.15
Poor 2.03 -.15 - 1,035.85 Ideal 2.03 .oo 0.00 -1,359.56
Mod. 1.97 -.06 -388.44 -1,748.00
Poor 1.25 -.75 -5,244.00 -6,603.56
E 1.58 -.64 --4,473.59 Ideal 1.58 .OO 0.00 Ideal 1.58 .oo 0.00 -4,473.59
Mod. 1.48 -.lo -705.67 -5,179.26
Poor .89 -.69 -4,849.08 -9,322.67
Mod. 1.48 -.lo -705.67 Ideal 1.48 .oo 0.00 -5,179.26
Mod. 1.28 -.20 -1,424.30 -6,603.56
Poor .65 -33 -5,826.67 - 11,005.93
Poor 1.05 -.54 -3,754.96 Ideal 1.05 .oo 0.00 -8,228.55
Mod. .95 -.09 -647.41 -8,875.96
Poor .46 -.58 -4,072.19 -12,300.74
Notes: A, B, C, D, and E represent different applicant pools for associate engineer positions. Z indicates the average standardized quality score
for the top two available individuals at that point. AZ refers to the difference in standardized quality scores. A 2 for attraction outcomes refers to
comparisons against the outcomes for Job A. A Z for status maintenance refers to the change in 2 due only to status maintenance effects. A 2 for job
acceptance refers to the change in Z due only to job acceptance effects. $U is the utility difference in dollars associated with the A Z score immediately
to its left. Total Impact refers to the total impact on utility from the combined effects of attraction, status maintenance, and job acceptance. Three
levels of performance (Perf.) are listed for both status maintenance and job acceptance. Ideal status maintenance is defined as having no applicants
withdraw prior to offers being extended. Moderate status maintenance performance is defined as having the Znd, 7th and 12th candidates withdraw
before offers can be made. Poor status maintenance performance is defined as having the Ist, 3rd, 5th, 7th, 9th, Ilth, and 13th candidates withdraw
before offers can be made. Ideal job acceptance is defined as no unaccepted offers. Moderate job acceptance performance is defined as having the 1st
and 3rd best candidates of those remaining accept offers. Poor job acceptance performance is defined as having both top candidates reject offers and
having the 3rd and 5th best candidates of those remaining accept offers.
(i.e., ideal, moderate, and poor) and three levels of job acceptance per-
formance (i.e., ideal, moderate, and poor) are examined. These are de-
scribed in the note to Table 4. In each case the status maintenance ef-
fect is determined first, then the job choice effect is calculated on the
remaining applicants, allowing the unique impact of each process to be
examined. The right hand column is the cumulative effect of all three re-
cruitment processes shown as a deviation from the best possible outcome
(i.e., Job A attraction with ideal status maintenance and job acceptance).
These results show that even though the same combinations of status
maintenance and job acceptance performance are applied to all appli-
cant pooIs, the consequences they produce are very different. For exam-
ple, for Job A, which has the highest overall TOP15 score and the lowest
standard deviation of scores, even poor status maintenance and job ac-
ceptance had only minor consequences for utility. The outcomes are
somewhat different for Job D. Even though the applicant pool for Job D
contains top candidates whose quality scores are only slightly lower than
those for Job A, the greater variance in scores results in substantial re-
ductions in utility when status maintenance and job acceptance perfor-
mance are poor (i.e., $6,603.56). Job E demonstrates the full potential
impact of poor attraction. Under ideal status maintenance and job ac-
ceptance, the difference in utility between Jobs A and E is $4,473.59.
But if the difference in attraction outcomes is combined with poor sta-
tus maintenance and poor job acceptance, the difference balloons to
$11,976.94. This is substantial given that it is based on (a) hiring only
two associate engineers, (b) expectations that they will remain for one
year, and (c) a selection device with validity of only T = .23. Increasing
any of these inputs would increase these utility estimates.
Discussion and Implications

Attraction outcomes are important because attraction represents
that point in the recruitment process where the maximum potential value
of recruitment processes is determined. The results of our illustrative
example demonstrate that even when applicants are drawn for similarly
titled positions from the same narrow applicant population, differences
in attraction outcomes can be substantial. Further, attraction outcomes
determine the impact that ineffective status maintenance and job ac-
ceptance activities will have for the organization. Excellent attraction
outcomes (e.g., Job A) insulated the organizations from even very poor
results in both status maintenance and job acceptance. However, poor
attraction outcomes (i.e., Job E, where few top quality applicants exist
and quality varies substantially across top applicants) can be magnified
by poor status maintenance and job choice performances.
We recommend organizations specificallyassess attraction outcomes

using quantitative assessments of applicant quality for all applicants
based on using the valid selection device the organization currently em-
ploys in its initial screening. The methods presented here can be used to
assess attraction outcomes for any type of recruitment practice used by
any organization recruiting for any type of position. Ideal measures will
(a) reference scores against the applicant population, (b) allow high lev-
els of differentiation, (c) possess interval measurement properties, and
(d) have high validity for predicting job performance potential. How-
ever, concerns about the validity of current screening practices or that
scores from those measures do not possess the desired properties should
not prevent adoption of this approach. Scores from valid measures with
less than perfect properties are better than no measures at all. Addi-
tional effort over successive iterations can improve the properties of the
scores produced. As scores approach ideal properties, the precision of
estimates will improve.
As the use of computerized screening practices increases, questions
arise concerning the validity of these practices. Simplistic database
searches using elimination by aspects criteria are likely to yield different
screening outcomes than those produced individual recruiters who likely
employ compensatory decision rules. But whether computers or humans
produce results with higher validity cannot be determined from existing
research. Concerns about the validity of the methods used by individuals
to screen resumes are just as valid. At a minimum, if more sophisticated
computer searches are developed that can mirror the compensatory na-
ture of human screeners, we can be assured that computers will apply
these decision rules consistently and impartially. More research in this
area is needed.
Appendix B provides a listing of some of the different ways that as-
sessment of attraction outcomes could be employed in organizations.
Direct assessments of attraction outcomes provide operational measures
that can be used to guide more aggressive management of recruitment
activities. These methods have direct value for both recruitment re-
search and practice. Because these methods can be used while or soon
after attraction activities are completed, they offer the possibility of al-
lowing organizations to respond to potential problems when remedies
may still exist. In addition, the separation of attraction outcomes from
other recruitment outcomes reduces the potential sources of “error”
in outcome measures, making assessments of applicant quality a more
useful diagnostic tool. When costs are also considered, these assess-
ments allow unambiguous interpretations of practices in each recruit-
ment phase. By comparison, using the number of applicants generated
as an attraction outcome measure results in a yield/cost measure that
produces unambiguous interpretations only when one alternative dom-

inates another (i.e., when the alternative that generates the most appli-
cants also has the lowest cost).
These methods for assessing recruitment outcomes proposed here
have broader application than prior attraction outcome measures. Be-
cause these assessments are made at the level of the individual applicant
and if they are referenced against the applicant population, changes in
applicant pool membership and its effects can be assessed over any time
period of interest. This permits practitioners to compare and contrast
applicant quality across different recruiting events as well as for contin-
uous recruiting. Hiring from the top 10% of applicants is not necessarily
the same as hiring from the top 10% of the applicant population. With-
out data referenced to the population, it is difficult to determine an orga-
nization’s actual attraction performance. With these data, organizations
can begin to understand how attraction is influenced by different recruit-
ment practices, important contextual differences like unemployment
rates, and the actions of competitors. However, whether the findings
for specific recruiting events generalize to others requires that decision
makers understand the dynamics of recruitment contexts. Collecting at-
traction outcome data across multiple recruiting events and comparing
them to environmental conditions and recruitment practices would pro-
vide data from which inferences concerning the generalizability of the
effectiveness of specific recruitment practices could be determined.
Conclusion
Adopting methods for evaluating attraction outcomes may be the sin-
gle most valuable step organizations can take toward improving recruit-
ment effectiveness. This method brings together recruitment theory that
suggests there are multiple recruitment phases, performance prediction,
and methods of utility analysis to form a fundamentally new and differ-
ent approach to recruitment evaluation. These methods offer a working
framework that researchers can use to better understand the factors that
influence recruitment effectiveness, and practitioners can use to more
aggressively manage the recruitment function.
REFERENCES
Barbcr AE. (1998). Recmiting employees: individual and orgnnizalionalperspectives. Thou-

sand Oaks, CA: Sage.
Barber AE, Wesson MJ, Roberson QM, Taylor MS. (1999). A tale of two job markets:
Organizational size and its effects on hiring practices and job search behavior.
PERSONNEL PSYCHOLOGY, 52,541467.
Boudreau J. (1991). Utility analysis for decisions in human resource management. In Dun-
nette MD, Hough LM (Eds.),Handbook of industrial and organizafionalp~ychology,
(Vol. 2, pp. 621-752). Palo Alto, CA: Consulting Psychologists Press.
Boudreau JW, Rynes SL. (1985). Role of recruitment in staffing utility analysis. Journal of
Applied Pvcholom, 70,354-366.
Breaugh JA. (1992). Recruitment: Science andpractice. Boston: PWS-Kent.
Bretz RD, Judge TA. (1994). The role of human resource systems in job applicant decision
processes. Journal of Management, 20,531-551.
Brogden HE. (1949). When testing pays off. PERSONNELPSYCHOLOGY, 3, 133-154.
Cable DM, Aiman-Smith L, Mulvey PW, Edwards JR. (2000). The sources and accuracy
of job applicants’ beliefs about organizational culture. Academy of Management
Journal, 43,1076-1085.
Cable DM, Judge TA. (1996). Person-organization fit, job choice decisions, and organiza-
tional entry. Organizational Behavior and Human Decision Processes, 67,294-31 1.
Connerley ML, Rynes SL. (1997). The influence of recruiter characteristics and orga-
nizational recruitment support on perceived recruiter effectiveness: Views from
applicants and recruiters. Human Relations, 50(12), 1563-1586.
Cronbach LJ, Gleser GC. (1965). Psychological tests and personnel decisions, 2nd edition.
Urbana, IL: University of Illinois Press.
Davidson L. (1998). Measuring what you bring to the bottom line. Workforce, 77(9), 34+.
FitzEnz J. (1984) How to measure human resource management New York: McGraw-Hill.
Ford BD, Vitelli R, Stuckless N. (1996). The effects of computer versus paper-and-pencil
administration on measures of anger and revenge with an inmate population. Com-
puters in Human Behavior; I 2 ( 1 ) , 159-166.
Garman AN, Mortensen S. (1997). Using targeted outreach to recruit minority students
into competitive service organizations. College Student Journal, 31(2), 174-179.
Griffeth RW, Hom PW, Fink LS, Cohen DJ. (1997), Comparative tests of multivariate
models of recruiting sources effects. Journal of Management, 23, 19-36.
Grossman RJ. (2000). Measuring up: Appropriate metrics help HR prove its worth. HR
Magazine, 45( l), 28-35.
Hawk RH. (1967). The recruitment function. New York: American Management Associa-
tion.
Highhouse S, Stierwalt SL, Bachiochi P, Elder AE, Fisher G. (1999). Effects of advertised
human resource management practices on attraction of African American appli-
cants. PERSONNEL PSYCHOLOGY, 52,425-442.
Hunter JE. (1980). Validitygeneralizationfor 12,000jobs: An application ofvnthetic vulidiq
generalization to the General Aptitude Test Battery (CATB). Washington, DC: U. S.
Department of Labor. Employment Service.
Hunter JE, Hunter RF. (1984). Validity and utility of alternative predictors of job perfor-
mance. Psychological Bulletin, 96, 72-98.
Huselid MA. (1995). The impact of human resource management practices on turnover,
productivity, and corporate financial performance. Academy of Management Jour-
nal, 38,635672.
Judge TA, Bretz RD. (1992). Effects of work values on job choice decisions. Journal of
Applied Psychology, 77,261-271.
Judge TA, Cable DM. (1997). Applicant personality, organizational culture, and organi-
zation attraction. PERSONNEL PSYCHOLOGY, 50,359-394.
Lavigna RJ. (1996). Innovation in recruiting and hiring: Attracting the best and brightest
to Wisconsin state government. Public Personnel Management, 25(4), 423-437.
Lunz ME, Deville CW. (1996). Validity of item selection: A comparison of automated
computerized adaptive and manual paper and pencil examinations. Teaching nnd
Learning in Medicine, 8(3), 152-157.
Mason NA, Belt JA. (1986). Effectiveness of specificity in recruitment advertising. Journal
oJ Management, 12,425-432.
McDaniel MA, Schmidt FL, Hunter JE. (1988). A meta-analysis of the validity of methods
for rating training and experience in personnel selection. PERSONNEL PSYCHOLOGY,
41,283-314.
McDaniel MA, Whetzel DL, Schmidt FL, Mauer SD. (1994). The validity of employment
interviews: A comprehensive review and meta-analysis. Journal ofApplied Psychol-
ogy, 79,599-616.
Murphy KR. (1986). When your top choice turns you down: Effect of rejected offers on
the utility of selection tests. Psychological Bulletin, 99, 133-138.
Ones DS, Viswesvaran C, Schmidt FL. (1993). Comprehensive meta-analysis of integrity
test validities: Findings and implications for personnel selection and theories of job
performance. Journal of Applied Psychology Monograph, 78,679-703.
Roth PL, BeVier CA, Switzer FS 111, Schippmann JS. (1996). Meta-analyzing the rela-
tionship between grades and job performance. Jounzal of Applied Pychology, 81,
548-556.
Rothstein HR, Schmidt FL, Erwin FW, Owens WA, Sparks CP. (1990). Biographical data
in employment selection: Can validities be made generalizable? Journal ofApplied
P~ychology,75, 175-184.
Ryan AM, Sacco JM, McFarland LA, Kriska SD. (2000). Applicant self-selection: Corre-
lates of withdrawal from a multiple hurdle process. Journal of Applied Psychology,
85, 163-179.
Rynes SL. (1991). Recruitment, job choice, and posthire consequences. In Dunnette MD
(Ed.), Handbook of industrial and organuationalpsychology,2nd ed., (pp. 399-444).
Palo Alto, CA: Sage.
Rynes SL, Barber AE. (1990). Applicant attraction strategies: An organizational perspec-
tive. Academy of Management Review, 15,286-310.
Rynes SL, Boudreau JW. (1986). College recruiting in large organizations: Practice, eval-
uation, and research implications. PERSONNEL PSYCHOLOGY, 39,729-757.
Sackett PR, Ostgaard DJ. (1994). Job-specific applicant pools and national norms for
cognitive ability tests: Implications for range restriction corrections in validation
research. Journal of Applied Psychology, 79,680-654.
Saks AM, Wiesner WH, Summers RJ. (1996). Effects of job previews and compensation
policy on applicant attraction and job choice. Journal of Vocational Behavior; 49,
68-85.
Schmidt FL, Hunter JE. (1998). The validity and utility of selection methods in personnel
psychology: Practical and theoretical implications of 85 years of research findings.
Pbychological Bulletin, 124, 262-274.
Schmidt FL, Hunter JE, McKenzie RC, Muldrow TW. (1979). Impact of valid selection
procedures on workforce productivity. Journal of Applied Psychology, 64,609-626.
Segall DO, Moreno KE. (1999). Development of the computerized adaptive testing
version of the armed services vocational aptitude battery. In Drasgow F, Olson-
Buchanan JB (Eds), Innovations in computerized assessmenf (pp. 35-65). Mahwah,
NJ: Erlbaum.
SHRMKCH (1999, Summer). Human resources management: Ideas and trends in person-
nel. St. Petersburg, FL: CCH Incorporated.
Stevens CK. (1997). Effects of preinterview beliefs on applicants’ reactions to campus
interviews. Academy of Mnnagement Journal, 40,947-966.
Terpstra DE, Rozell EJ. (1993) The relationship of staffing practices to organizational level
measures of performance. PERSONNEL PSYCHOLOGY, 46,27-48.
Turban DB, Campion JE, Eyring AR. (1995). Factors related to job acceptance decisions
of college recruits. Journal of Vocational Behavios 47(2), 193-213.
Vispoel WP, Boo J, Bleiler 7: (2001). Computerized and paper-and-pencil versions of
the Rosenberg Self-Esteem Scale: A comparison of psychometric features and

respondent preferences. Educational & Psychological Measurement, 61,461-474.
Vispoel WP, Coffman DD. (1994). Computerized-adaptiveand self-adapted music-listening
tests: Psychometric features and motivational benefits. AppliedMeasurement in Ed-
ucation, 7(1), 25-51.
Wanberg CR, Kanfer R, Banas JT. (2000). Predictors and outcomesof networking intensity
among unemployed job seekers. Journal of Applied Psychology, 85,491-503.
Werbel JD, Landau J. (1996). The effectiveness of different recruitment sources: A
mediating variable analysis. Journal ofApplied Psychology, 26,1337-1350.
APPENDIX A
AssessingAttraction Outcomes: A Step-by-stepOverview
Step 1. Identify positions to assess.
Description: Organizations may choose not to develop scores to eval-
uate recruitment outcomes for all positions. Those positions where as-
sessment of attraction outcomes is likely to be of greatest value are jobs
that generate several new hires and attract large numbers of applicants.
Step 2. Identify current screening mechanism and determine current
assessment properties.
Description: Organizations need to identify the current selection mech-
anism. in some organizations, screening devices are not formalized. In
these cases, efforts should be made to identify the true character of the
screening device as it currently exists.
Step 3. Determine strategy for adapting current screening device to
produce scores for each applicant and adapt changes.
Description: Depending on what screening device is currently used, the
amount of deviation between current practices and the development of
scores for each applicant with the desired properties will vary. Organi-
zations need to decide how far they want to go toward developing mea-
sures that yield scores with optimal properties. In most instances, it is
likely to be more effective for organizations to develop measures over
a series of iterations aimed at working toward desired score properties.
Doing so is likely to minimize costs. Some of the properties are more
easily developed after a particular measure has been used for a period
of time.Three general strategies exist for referencing scores to the appli-
cant population: (a) Use an instrument for which norms already exist.
(b) Use supplementary data to convert scores to a form that can be com-
pared across the applicant population (as in using SAT or ACT data for
college or degree program entrants to equate GPA data). (c) Maintain
an archive of scores on the devices you use. Decision makers should
consider the fixed costs of de;eloping these measures and any potential
changes to the variable costs of administering them. These costs should
be compared to the benefits likely to be gained from the development of
attraction outcome data.
Step 4. Assess attraction outcomes.

Description: Assess each applicant using the device developed in Step 3.
Scores for all applicants in the applicant pool in batch recruiting should
be evaluated. In continuous recruiting, all applicants, including those
that have left the pool by withdrawing themselves from consideration or
because they were hired, during the appropriate time period (defined
by the organization) should be included. Data from the analysis of ap-
plicant flows for each position should be archived in order to facilitate
comparisons across recruiting events.
Step 5. Match recruitment activities to recruitment phases and estimate
recruitment costs.
Description: In order to evaluate the cost effectiveness of various re-
cruitment practices, recruitment costs must be identified and mapped to
the appropriatephase of recruitment (ie, attraction, status maintenance,
gaining job acceptance). Estimate costs of attraction by identifylng all
activities primarily designed to influence the attraction of applicants to
this position, not general recruitment costs. The cost of general recruit-
ment activities-those not attributable to a specific position--could be
evaluated against all positions that are influenced by those activities.
Step 6. Estimate input values and complete UA estimates. (This is
required to complete evaluations that incorporate both cost and benefit
considerations.)
Description: To be able to use UA to convert differences in quality scores
to dollars, several values need to be estimated.
AU = r x y x S D y x A Z x x T x N - AC. 131
These include the validity of screening devices (rw), the standard devi-
ation of performance in dollars for the job in question (SDv), the num-
ber of individualsto be hired or the subset of applicants to be evaluated
( N ) , and the average expected tenure of applicants in their positions
once hired (T). The differencein standardized quality scores (AZZ)is
constructed by identifyingthe applicants to be compared, calculating the
average value of their predictor scores, and standardizing those values
based on the applicant population norms for screening device scores and
determining the difference between them.
Step 7. Evaluate the adequacy of current screening devices.
Description: Organizations may decide that alternative screening de-
vices may be more appropriate due to their cost, validity, or other prop-
erties of the scores they generate. Comparisons of alternative screening
methods can be performed using standard utility equations for evaluat-
ing alternative selection practices (e.g., Schmidt et 1979; Boudreau’s
(1991) Equation 7).
A u = A T ~ ~ x S D ~ X Z X -X AC TXN [41
APPENDIX B
CapabilitiesOffered by AssessingAttraction Outcomes
Capability: Evaluation of alternative attraction practices.
How it adds value: Provides a means for organizations to compare the
effectiveness of alternative recruitment activities. This approach also
allows organizations to evaluate each phase of recruitment so strengths
or weaknesses in attraction can be identified.
Capability: Comparisons of applicants across applicant pools.
How it adds value: Gives decision makers the opportunity to compare
candidates from different applicant pools. Because scores are scaled to
the applicant population rather than to specific applicant pools, scores
for candidates from different applicant pools can be compared directly.
Capability: Evaluating alternative sources of candidates.
How it adds value: Provides organizations with a means of examining
alternative sources of applicants. This can be particularly useful in col-
lege recruitment programs where organizations can objectively evaluate
the potential of the applicant pools from different schools and be able
to make determinations about which sources provide better applicants.
Capability: Evaluating the effectiveness of status maintenance and job
acceptance activities.
How it adds value: Once attraction outcomes have been assessed, the
assessment of status maintenance and job acceptance outcomes simply
involves the recognition of the value lost when top candidates withdraw
from the applicant pool prior to receiving a job offer or refuse to accept
a job offer when one is offered.
Capability: Concurrent evaluation of attraction activities.
How it adds value: Because scores are developed for each applicant,
attraction outcomes can be evaluated on an ongoing basis, even before
organizations complete recruitment processes. This gives decision mak-
ers the opportunity to identify problems and adjust activities to reduce
their negative effects while there is still time to do so.
Capability: Cost-benefit analyses of attraction activities.
How it adds value: Permits organizations to evaluate both the benefits of
alternative attraction, status maintenance, and job acceptance activities,
and their costs. Unambiguous interpretations of the relative effective-
ness are possible.
Capability: Evaluating recruitment improvement opportunities.
How it adds value: Provides organizations the opportunity to compare
how well they are performing recruitment activities against what could
be possible with the applicant populations that exist and provides a
means of estimating the value of those opportunities.

J 1744-6570 2002 tb00118 X

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

J 1744-6570 2002 tb00118 X

Uploaded by

Copyright:

Available Formats

PERSONNEL PSYCHOLOGY

RECRUITMENT EVALUATION: THE CASE FOR

We argue that assessing attraction outcomes is critical to systematic

It is becoming clear that effective recruitment is critical to organi-

& Boudreau, 1986). The SHRM/CCH (1999) Survey of Recruiting Prac-

For organizations to pursue systematic, sustained improvement in re-

ment outcomes is necessary. We believe a practical approach to recruit-

Recruitment is the primary mechanism used by organizations to at-

organizational factors (e.g., Saks, Wiesner, & Summers, 1996; Turban,

Evatuating Attraction Outcomes

Evaluating organizations’ effectiveness in each recruitment phase

cant population (i.e., all individuals in the workforce minimally quali-

Developing Quality Scores for All Applicants

Directly estimating applicant quality avoids relying on assumptions

geographically and temporally dispersed recruitment efforts that gener-

What we are suggesting is extending and formalizing organizations’

Step One: Identifi Positions to Assess

Step Two: Identifi and Evaluate Current Screening Mechanism

Selection device Validity for predicting

'1 Rothstein, Schmidt, Erwin, Owens & Sparks (1990)

" Roth, BeVier, Switzer, & Schippmann (1996)

ability (T = .51) or integrity (T = .41). Work on biographical data instru-

placement decisions, scores on the ASVAE3 could also be used to eval-

Step Four: AssessingAttraction Outcomes

Step Siu: Evaluating Attraction Outcomes Using UtilityAnalysis

port comparisons of applicants and applicant pools. However, these data

‘Boudreau’s (1991) Equation 7 is AU = ( N s ) ( S D y ) ( T : , : ~(Zx)

acceptance losses can lead to overestimates of utility. These examples,

Step Seven: Evaluating the Adequacy of Current Screening Devices

quickly. In addition, for automated evaluation of resumes or applica-

A Recruitment Evaluation Example

and standard deviations of all applicants as the appropriate descriptive

performance corrected for error of measurement in the criterion. We

Results of the Analysis for the Illustrative Example

Table 2 provides an overview of the five associate engineer applicant

number of resumes in each applicant pool, means and standard devia-

Standardized qualitv scores Job ID

Discussion and Implications

We recommend organizations specificallyassess attraction outcomes

produces unambiguous interpretations only when one alternative dom-

Barbcr AE. (1998). Recmiting employees: individual and orgnnizalionalperspectives. Thou-

the Rosenberg Self-Esteem Scale: A comparison of psychometric features and

Step 4. Assess attraction outcomes.

You might also like