The current issue and full text archive of this journal is available at www.emeraldinsight.com/0090-7324.

htm

Systematic review of research methods: the case of business instruction

Ann Manning Fiegen
California State University San Marcos, San Marcos, California, USA
Abstract
Purpose The purpose of this paper is to assess the body of business instruction literature by academic librarians against evolving models for evidence-based research. Design/methodology/approach The paper used systematic review and inter-rater reliability of the literature of business information research instruction to test two attributes of research quality: the evidence-based levels of evidence and the EBLIP critical analysis checklist. Findings Intervention questions and case studies are the most popular research methods on the EBL levels of evidence scale. The majority of articles score below 75 on the EBLIP critical appraisal checklist. Prediction questions are represented by higher levels of evidence and study quality. Intervention questions paired with the cohort design and exploratory questions paired with survey design indicate strong areas of research quality. The case study method, while most popular, showes lower scores across all question types yet revealed some high-quality benchmark examples. Research limitations/implications Error is possible when distinguishing between cohort and case study some articles may fall into one or the other study design. Rater training was conducted only once, and best practices for inter-rater reliability recommend multiple rounds to achieve higher rater agreement. Practical implications Recommendations are presented for ways to improve the evidence base of research articles and suggest areas for professional development opportunities for librarian researchers wishing to increase the quality of research publications. Originality/value The paper goes beyond the narrative review of the literature of business instruction to measure the research methods employed in those publications against two evidence-based standards. The results will show where the literature stands as a maturing discipline and provide recommendations for increasing the levels of evidence for future research. Keywords Academic libraries, Research methods Paper type Literature review

Business instruction

385
Received 12 April 2010 Revised 4 May 2010 Accepted 7 May 2010

Introduction Evidence-based practice advocates that academic librarians look to the published literature to nd reliable and valid studies as guidance. To inform and guide this researchers practice, an effort was undertaken to locate studies that could offer that guidance for information literacy instruction for business students. The result of that search is that business librarians are prolic authors of business information research
This study was funded in part by a BRASS Emerald Research Award 2009 and a California State University San Marcos, Faculty Research Grant. The author wishes to acknowledge the participation of Martha Cooney, Cheryl Delson, Nancy Dewald, Patrick Ragains, Frank Vuotto, and Diana Wu. Portions of this report were presented at the California Academic and Research Libraries Conference 2010.
Reference Services Review Vol. 38 No. 3, 2010 pp. 385-397 q Emerald Group Publishing Limited 0090-7324 DOI 10.1108/00907321011070883

RSR 38,3

386

instruction, resulting in hundreds of published studies to read and apply to practice. This wealth of information prompted this researcher to ask: What measure denes high-quality evidence-based research that can reliably be applied from the many studies published? What kind of research do business librarians undertake, and taken in the aggregate do they suggest a maturing research methodology in the discipline? And nally, against what standards can this body of literature be measured? Before the content of the articles could reliably be applied to best practice, the quality of the studies needed to be ascertained. The questions explored in this report form a portion of a larger study that analyzes the content of the literature along the dimensions of study objectives and results, institution setting sample population, and pedagogy (theorist, standard, or model) employed. The focus of this report is only on the research methods employed. Systematic review offers a model for summarizing and critiquing the literature to improve future practice and possibly encourage higher levels of research methods. A systematic literature review of 30 years should reveal evidence toward a maturing research methodology. Academic librarians have applied theory to practice and documented improvement efforts through the published literature of business instruction. The case study is a popular method for describing best practices and well suited for the action research needed by librarians. Case studies, however, are a marker for a young discipline and considered by some not to be as rigorous as compared to higher levels of research. How then can this discipline strive for higher levels of evidence? The objective of this study is to assess the body of business instruction literature in academic libraries against evolving models for evidence-based research. The results should indicate opportunities for higher levels of research methodology in keeping with a maturing discipline. Eldredge (2006) challenged librarians to apply and test his proposed evidence-based librarianship levels of evidence model he applied initially to research studies in medical librarianship. The question is whether the same can be said for the literature of instruction in business information research. Where on Eldredges proposed matrix is the state of research in this discipline, what if any are the implications for future research, and what is the case for business instruction literature? Academic librarians are introduced to research methods through graduate education and must continue to learn the process essentially independently. The expertise is gained from continuing education, professional development, and mentoring relationships and using library collections on research methods. Guidelines for evidence-based research can be used as a professional development tool to guide the new researcher and as an instrument to assess the quality of existing research. The medical eld has led this effort by developing guidelines that help the researcher assess the quality of existing research. Examples of critical appraisal tools can be accessed from the International Centre for Allied Health Evidence (2009) web site among others. Glynn (2006) studied many models to arrive at her instrument, the EBLIP critical appraisal tool for library research. This study will show how that instrument can be applied to the critical appraisal of business instruction literature by testing two attributes of research quality using the business information instruction literature as a case analysis. A systematic review of the literature will test for levels of evidence and evidence of research method quality: H1. Library articles on this subject will overwhelmingly be exploratory case studies and low on the Eldredge levels of evidence hierarchy.

H2. The majority of articles will score below 75 percent as measured by the EBLIP critical appraisal checklist. Literature review Commonly accepted social science research method denitions vary slightly depending on the discipline and the authors objectives. Widely followed for case analysis is Yin (2003) who categorizes social science research into experimental, survey, archival analysis, history, and case study. Gormon and Claytons (2005) Handbook for Information Professionals divided qualitative research methods into observational, interviewing, group discussion, and historical study. Fink (2005) emphasized assessing quality of research studies for literature reviews and categorized studies into either experimental or quasi-experimental families. Cooper (2010) cautioned about error when evaluating the quality of studies. There are a number of studies that have analyzed and critiqued the preferred research methods used by librarians. Specic examples include Watson-Boone (2000, p. 87) who examined 24 articles from Journal of Academic Librarianship ( JAL), grouped them into six research methods, and ranked them by order of frequency of research method used: survey research, action research, secondary data analysis, and case study, with evaluation research and experimental tied for last. The typical JAL article emphasized problem-solving and managerial issues, and therefore, the Watson-Boone categories cannot be generalized to this study. Most recently, Hildreth and Aytac (2007) summarized library practitioner articles from 2003 to 2005 into descriptive, exploratory, explanatory, and evaluative research. A summative approach does not support the benchmarking objective of this study. Eldredge (2002) borrows from clinical medicine with the intent of applying his model for evidence-based librarianship (EBL) levels of evidence to medical librarianship research. The matrix approach associates three types of research questions (prediction questions, intervention questions, and exploration questions) with ranked research methods. The highest ranked level of evidence is systematic review, followed by meta-analysis, summing up, prospective or retrospective cohort study, qualitative studies, descriptive study or survey, and case study. He observes that typical library research studies fall into the lower levels of descriptive survey, case study, and qualitative methods, areas where error and author bias will more typically occur (Eldredge, 2002, p. 294). Given (2006) continues a long-standing debate by arguing that the levels of evidence introduced by Eldredge favor quantitative research methods and that relegating qualitative research to the lowest level overlooks its appropriate place in social science, including library science research, continuing a long-standing debate. The Eldredge model of 2002 is adopted here as it most closely supports the objectives of this study. Edwards (1994) reported on the percent of research articles to non-research articles between 1971 and 1991 when reviewing the research of bibliographic instruction. She ranked frequency of research method used among the research articles and frequency of library instruction topic. While the Edwards study is not directly comparable for the present research, results can conrm or deny a trend. Literature reviews of library instruction are characterized by the exploratory question (what was published) and the descriptive narrative review method (summary of trends and annotation of entries for a given time period). Typical of this genre are Rader (1974, 2002)

Business instruction

387

RSR 38,3

388

and Johnson et al. (2007). Crawford and Feldt (2007) conducted a systematic analysis of library instruction literature using citations from the ERIC database as their source. It expands the narrative review by including explicit research objectives, statements of study inclusion and exclusion, and article analysis of the articles. Koufogiannakis (2006) reported on a systematic review and meta-analysis of the most effective method for teaching information literacy skills to undergraduate students and found that computer-aided instruction was as effective as traditional instruction and that traditional and self-directed instructions are more effective than no instruction. She recommended further research be conducted in comparative and validated research methods and suggested that additional replication be included in existing high-quality studies. Examples of narrative review articles that summarize the state of information literacy specic to business students include Jacobson (1993) who summarized the literature of best practices for business instruction from 1985 to 1992. Most published articles about business instruction include literature reviews. Cooney (2005) surveyed business instruction librarians at AACSB-accredited colleges to assess the extent of business instruction in libraries (indicating a trend toward more evidence-based information literacy research) and a systematic review by Zhang et al. (2007) that compared the effectiveness of face to face to computer-assisted instruction. This study will go beyond the narrative review to measure the research methods employed against two standards: the EBL levels of evidence model and the EBLIP critical appraisal checklist. Methodology Library and business education bibliographic databases were searched for English language publications between 1980 and spring 2009. The database was initially created in 2004 and repeated thereafter through spring 2009. Databases searched included EbscoHost Premier, Emerald Fulltext, ERIC, Library Literature and Information Science, LISA, ProQuest Inform Global, and the ISIs Web of Knowledge. Hand searches of cited references in the primary literature were also conducted. File drawer bias is outside the parameters of the study as this studys objective was to research only the published literature. The databases searched replicate those used by Johnson et al. (2007) in their annual review article of library instruction with the addition of Emerald and ABI Inform Global. The later were included to expand the search to internationally published reports in library science and business management education. Each index was searched for the terms: library and business and (instruct * or literac * or assess* or teach *) and (academic or higher education or college or university). Bibliographic records and abstracts were scanned resulting in an initial set of 245 articles about library instruction for business students. Further review resulted in 69 articles as the initial set of articles for this study. Criteria for inclusion were as follows: articles were authored or coauthored by a practicing academic librarian, the subject of the study was instruction in business research in academic libraries, and the article was in English and published between 1980 and 2009. Excluded were non-peer-reviewed articles, articles appearing as columns, studies authored by business faculty or by faculty teaching in library, and information science graduate programs but not coauthored by a practicing librarian and business education literature about information competencies that did not explicitly refer to library instruction with a librarian.

The bibliographic software EndNote was used to hold the data sets. Each article was read, coded, and color classied according to one of the Eldredges (2002) three research questions: predication, intervention, or exploratory. Table I shows Eldredges levels of evidence matrix where each article is categorized into its corresponding question type and research method of meta-analysis, summing up, prospective or retrospective cohort study, qualitative studies, descriptive study or survey, and case study. Eldredge (2004) distinguishes higher order levels of evidence from the lower level by their distinct hypothesis and objectives, interventions, and measurable outcomes. The denitions in Eldredges (2004) inventory of research methods guided the classication of the data set. Cohort studies used a dened population, indicated some kind of intervention even if it only described a change from status quo, and had a measurable outcome. Those studies were further identied as either using proscriptive or using retrospective data collection methods. Articles dened as case studies described an experience. According to Eldredge (2004), case studies are distinguished by their description and analysis of authors experiences, have multiple sources of evidence, and will answer how and why questions; refer to Booth and Brice (2004) and Eldredge (2004) for more complete descriptions of denitions and study design. The paired response inter-rater reliability method (Fink, 2005) was used to rate the research quality of the articles against the Glynn (2006) EBLIP Critical Appraisal Checklist. Six raters were selected to participate in the study, ve of whom were authors writing on the subject in the last ten years. One recent library and information science (LIS) graduate with prior research methods experience was also invited to participate. No rater was assigned to their own study, although one rater disclosed frequent collaboration with one of the assigned articles. It was not deemed enough of a conict to warrant exclusion. Permissions to use the Eldredge and Glynn instruments were obtained from the authors as were publisher supplied reprints of the Glynn article for all raters. Exempt status was submitted and granted by the university review board since the subjects under study, the published articles, were not human subjects. A training packet was mailed to the raters that included instructions describing the expectations and compensation for participation in the study, procedures for using the rating instrument, a copy of the Glynn article and checklist, and a sample article to rate that was not included as part of the study. Training sessions were scheduled and conducted in summer of 2009. Raters were instructed to read the sample article, use the checklist to rate the article independently by noting any comments or questions, and return their rating sheet to the principle investigator prior to the scheduled WebEx training session. At each of the three WebEx training sessions, each pair of raters and
Prediction Systematic review Meta-analysis Retrospective cohort study Prospective cohort study Survey study Case study n 8 Intervention Systematic review Meta-analysis RCT Retrospective cohort study Prospective cohort study Survey Case study n 22 Exploration Systematic review Summing up Qualitative studies Survey Case study n 17 1 8 8

Business instruction

389

2 2 1 3

2 4 3 13

Source: Used by permission of the author Eldredge (2002)

Table I. Business instruction levels of evidence

RSR 38,3

390

the principle investigator met virtually to review the rating of the training sample article with the intention of reaching agreement for the checklist elements. Raters were to be guided by their own experience as practitioner researchers, though some expressed concern about their own expertise in research methods. The discussion often centered upon reaching agreement on the denition and interpretation of the checklist questions as well as how to respond to the categories of: yes, no unclear or not applicable. Glynns (2006) annotations for using the instrument-guided discussion. For example, an Yes answer would indicate that there was an explicit statement in the study. An Yes response regarding the use of a validated instrument for data collection indicated that the instrument was pilot tested, there was evidence of revision prior to use, or it was based on a previously published study. After the training session, raters were invited to follow up with the principal investigator on any questions or concerns. Raters were sent their full set of articles with instructions to rate and return half the articles at a three-month midpoint, with the nal set due at six months. At the end of the study period, raters were debriefed and comments summarized for reporting purposes. The checklists contained no information regarding the identity of raters. Completed checklists were then paired to the corresponding articles. Paired responses were recorded, and where there was no agreement, discrepancies in scoring were broken by a vote from the principle investigator. Glynns checklist uses a scoring mechanism where each of the 29 question yes, no, unclear responses is added and averaged along four dimensions (study population, data collection, study design, and results) to arrive at an average score. Not applicable is a response option but is not used for scoring. A score of $ 75 indicates a valid research study. She cautions that numerical scoring is not the sole indicator of the quality of an article. Analysis and discussion Once each article was classied into the EBL levels of evidence matrix and then by the EBLIP critical appraisal checklist patterns emerged to show what kind of research librarians for business information instruction conduct indicated the top, mid, and lower tiers for articles for each category. A total of 69 studies were sent to raters; of those, 22 scored zero on the checklist and therefore were eliminated as not meeting the denition of evidence-based research. A total of 47 articles advanced to analysis and were plotted into the EBL levels of evidence matrix. H1 is that the majority of articles will be exploratory and does not hold but rather is led rst by intervention n (22:47) or 46 percent, second exploratory n (16:47) or 34 percent, and nally by prediction n (8:47) or 17 percent. H1 is that the majority will be case studies and low on the levels of evidence scale does hold n (24: 47) or 51 percent. Descriptive or survey studies constitute n 9:49 or 19 percent, cohort studies n 11:47 or 23 percent, and randomized controlled trials form the least used type of study n 2:47 or 04 percent, and nally one lone systematic review n 1:47 or 02 percent. When levels of evidence are viewed within each question, some patterns emerge. Prediction questions tended to favor higher level research. The cohort design (four) is slightly more likely than the case study (three). The intervention questions favored case studies (13) followed by seven cohort studies and two randomized control studies. Exploratory questions split evenly between survey design method (eight) and case

studies (eight). Table I shows how the studies are plotted against the levels of evidence matrix (Eldredge, 2002, p. 10). Table II plots the research question against the EBLIP critical appraisal checklist and indicates how groups of articles were ranked for research rigor. Overall, 15:47 or 31 percent of all studies ranked in the top third by receiving a score of 75 or above. Represented by 5:8 or 62 percent predictive studies, 4:22 or 18 percent intervention studies, and 5:17 or 29 percent were exploratory studies. As a percentage, predictive studies score higher on the EBLIP critical appraisal scale, though signicantly, high ratings are represented in each research question type. Highly rated articles described clear populations or samples and often used cohorts. Instructional design was based on validated models or industry standards. Data collection used validated or piloted survey or assessment measures. Multiple measures using both quantitative and qualitative data collection methods increased validity or supplemented self-perception surveys. If longitudinal, the time spans between pre- and post-test were a semester or longer. Statistical analysis reported more than percentages and included deviations and signicance factors. Explicit articulation of study design, methodology, and results that could facilitate replication characterized all highly ranked articles. The Appendix lists all highly ranked articles grouped by level of evidence and type of study. Articles showing some, but uneven, research quality in the middle tier scores of 50-74 were 2:8 or 25 percent predictive questions, 11:22 or 50 percent intervention studies, and 7:17 or 41 percent exploratory questions. This middle range represents the largest number of studies and suggests opportunities for additional training and professional development to increase validity and rigor for intervention and exploratory questions. Of those with low scores 1:8 or 12 percent were predictive questions, 7:22 or 32 percent were intervention questions, and 4:17 or 10 percent were exploratory questions. This validates nding that that predictive questions score higher on the EBLIP critical appraisal checklist and suggests that studies using intervention questions would benet from additional attention to research rigor Figure 1 illustrates the level of evidence against the critical appraisal ranking by type of question. Prediction questions that use higher levels of evidence represented by RCT and cohort design as a group score high on the EBLIP critical appraisal scale. Intervention questions are the most popular research method, n 22, and when viewed by EBL levels of evidence, the article set shows that the cohort design method, scoring 60 and above, outranks the case study method where all but one score below 66. This suggests that choosing a cohort design over case study tends to increase quality of research for intervention questions. An interesting phenomenon appears with exploratory studies, while represented by fewer articles, six using that methodology are in the top tier of the checklist
Predictive (n 8) n % 5 2 1 0.62 0.25 0.12 Intervention (n 22) n % 4 11 7 0.05 0.50 0.31 Exploratory (n 17) n % 6 7 4 35 14 10 Total (n 47) n % 15 20 12 0.31 0.42 0.25

Business instruction

391

EBLIP critical appraisal score 75-100 50-74 0-49

Table II. Critical appraisal score by type of research question

RSR 38,3

0 SR RCT RCT

10

20

30

40

50

60

70

80

90

100

392

COH COH COH COH COH COH COH Desc/survey Desc/survey Desc/survey Desc/survey Desc/survey Desc/survey Desc/survey Desc/survey Case Case Case Case Case Case Case Case Case Case Prediction Intervention Exploration

Figure 1. Levels of evidence and critical appraisal

Case Case Case

(one systematic, two surveys, and three cases), a stronger showing in the top tier than intervention articles. The descriptive survey design method n 8 is generally clustered high with two studies in the top tier, ve in the middle tier, and only four studies in the lower tier. This validates the hypothesis that librarians are familiar and comfortable with exploratory survey design methodology. Case studies on the other hand were

mixed with three high scores, two in the middle range, while three scored low as did most of the studies deemed ineligible. Study limitations Every effort was made to include all published business research instruction by academic librarians in the initial search but some may have been missed. Owing to the ambiguity in some of the articles, error is possible when distinguishing between cohort and case study some articles may fall into one or the other study design. Some raters, while among the higher ranked authors, expressed a need for additional training that time and resources did not allow. Rater training was conducted only once, and best practices for inter-rater reliability recommend multiple rounds to achieve higher rater agreement. The reliability of this method may increase with higher knowledge of research methodology and multiple rounds of training to increase inter-rater reliability. Kappa statistic was not conducted for this study but relied rather on simple majority. Article authors were not blind to the raters; therefore, bias is possible. This researchers own article and in another case a raters colleague were in the set; nevertheless, the articles were included. Outside the scope of this research is an important group of writers of business instruction in libraries. Articles that were solely authored by LIS professors in graduate schools or business faculty and were not coauthored by practicing librarians were not included in this data set. Although they contribute important studies, inclusion of those reports would skew results for this study population. Case studies of reports of practice are outside the scope of this study but continue to serve an important function for advancing the practice of library instruction for business students and were represented in the rst large data set. Many other studies appear as book chapters and outside the indexing and abstracting services and therefore may have been missed. Conclusion and recommendations This study sought to categorize business information literacy research articles into the two models. H1 stated that library articles about business information research instruction would overwhelmingly be represented by exploratory studies and would use the case study method that is low on the Eldredge EBL levels of evidence hierarchy. The rst part of H1 does not hold as exploratory studies comprise only 36 percent of studies, less than the 46 percent of the intervention studies, but more than the only 17 percent of prediction studies. The second part of the statement does hold since case studies represented fully 51 percent of research designs chosen. The second most popular method is the cohort study with 11 or 23 percent of articles, followed by survey design with 9 or 19 percent of articles, and nally only two randomized control trials and one systematic review. H2 stated that the majority of articles will score below 75 percent as measured by the Glynn critical appraisal checklist. H2 holds as the majority of articles scored below the Glynn denition of high-quality research. Over 245 articles over the study period of 1980-2009 met the practical screen. Only 47 met the criteria for evidence-based research studies that comprised the population under study. Of those, only 15 or 31 percent scored in the top tier of the EBLIP critical appraisal checklist, while 42 percent scored in the middle tier and 25 percent scored in the lowest tier.

Business instruction

393

RSR 38,3

394

Pursuing prediction questions yielded higher levels of evidence and study quality. Prediction questions comprised 8 percent of all studies yet overwhelmingly scored high for quality in both cohort and case study design. Intervention questions paired with the cohort design method scored above 60 and is also suggested as a method for those conducting this kind of research. For those choosing exploratory questions and using the descriptive survey method, two articles scored in the top tier with the remaining articles scoring in the middle tier. Surprisingly, the case study method (n 24) scored low across all question types with only a few exceptions 5 $ 75. Considering that it represents the most popular choice for research design, it is an important area for future training and professional development among potential authors. The ve highly ranked case studies are notable since they show that there is a place for evidence-based research quality case studies even though they are considered low on the levels of evidence scale. Future professional development for true research-based case study as opposed to the more informal description of practice would increase the evidence-based and validity of the case studies. This is especially important as the case study is and will continue to hold an important role in this research genre. One rater commented that:
There is a long tradition of case study articles in librarianship. We all like them, read them, write them, and we could denitely use a push to provide the hard evidence of success, rather than just the comfortable assurances that we succeeded.

When examining all articles by the four segments of the EBLIP critical appraisal checklist, the population segment and results segment were highly rated. The segments for data collection and to a lesser extent research methodology scored lower. Often, the lower scoring articles only lacked additional clarity, resulting in a score of unclear. More clarity in data collection reporting and study design would have increased the quality scores of those articles in the middle tier. Glynns points regarding use of validated instruments, and using questions posed to elicit precise answers, would increase rigor in data collection. Special attention to explicit documentation about the use of consent forms, ethics panel clearance and disclaimers, and divulgence about the role of authors in data collection would increase quality as measured by the checklist. Generally, these indicators suggest that research reports need only moderate improvements to become highly rated articles. All raters remarked that the checklist would serve as a useful planning guide when undertaking and writing research reports, though they also cautioned that it was not applicable for some kinds of writing. Hildreth and Aytac (2007) review the quality of LIS practitioner and academic scholar research. They also cite validity and lack of evidence as limitations in those articles. They credit journal editors and reviewers for urging higher quality studies. They suggest collaborations as a way to increase quality and notes signicantly that collaborations were more associated with other disciplines than with LIS faculty, a nding similar to this report. They suggest further research into why there are low levels of collaborative research, how they can be increased, and what the role is for the LIS graduate curriculum (Hilderth and Aytac, 2007, p. 255). Results of this research suggest education and professional development in rigorous case study method. Collaborations with business faculty as coauthors have a rich tradition and tend to increase the quality of the research reports, though the evidence shows that coauthorship with business faculty was not a deciding factor among the

highly ranked set as a group. High-quality studies authored solely by LIS faculty appeared in the initial phases of data collection yet were out of scope for this study. Yet, as Hildreth and Aytac (2007) indicated that suggestions for future collaborations between LIS graduate faculty and practicing academic librarians could increase the quality of research. In summary, business librarians conduct high-quality research at all levels of evidence. Prediction questions are represented by higher levels of evidence and study quality. Intervention questions paired with the cohort design and exploratory questions paired with survey design indicate strong areas of research quality among the set of articles studied. The case study method, while most popular showed lower scores across all question types, yet revealed some high-quality benchmark examples. Authors preferred case studies, though using cohort design tended to increase the measure of rigor. To continue to raise the levels of evidence, it is recommended that business librarians conduct more random control trials and more systematic reviews of existing research. To increase the rigor of research, the results of this study suggest closer attention to clarity of description in all segments but especially data collection and research methodology. Education and professional development in the case study method are indicated for this preferred research method. The EBLIP critical appraisal checklist or similar guidance is strongly recommended as a planning guide when undertaking a research project. This systematic review yielded 15 studies that met the criteria for evidence-based research (the Appendix). Those studies now provide a set of articles that can reliably be applied to the practice of business information literacy instruction. The many quality articles ranked in the middle tier also provide valuable reports of practice. This systematic review of the literature shows that Eldredges EBL levels of evidence and Glynns EBLIP critical appraisal checklist can be used as indicators for the maturity of business instruction research by academic librarians as a body of research and can suggest future direction for new evidence-based studies.
References Booth, A. and Brice, A. (Eds) (2004), Evidence-based Practice for Information Professionals: A Handbook, Facet, London. Cooney, M. (2005), Business information literacy instruction a survey and progress report, Journal of Business and Finance Librarianship, Vol. 11 No. 1, pp. 3-25. Cooper, H. (2010), Research Synthesis and Meta-analysis: A Step-by-step Approach, 4th ed., Sage, Thousand Oaks, CA. Crawford, G.A. and Feldt, J. (2007), An analysis of the literature on instruction in academic libraries, Reference & User Services Quarterly, Vol. 46 No. 3, pp. 77-87. Edwards, S. (1994), Bibliographic instruction research: an analysis of the journal literature from 1977 to 1991, Research Strategies, Vol. 12 No. 2, pp. 68-78. Eldredge, J.D. (2002), Evidence-based librarianship levels of evidence, Hypothesis, Vol. 16 No. 3, pp. 10-13. Eldredge, J.D. (2004), Inventory of research methods for librarianship and informatics, Journal of the Medical Librarian Association, Vol. 92 No. 1, pp. 83-90. Eldredge, J.D. (2006), Evidence-based librarianship: the EBL process, Library Hi Tech, Vol. 24 No. 3, pp. 341-54.

Business instruction

395

RSR 38,3

396

Fink, A. (2005), Conducting Research Literature Reviews: From the Internet to Paper, 2nd ed., Sage, Thousand Oaks, CA. Given, L. (2006), Qualitative research in evidence-based practice: a valuable partnership, Library Hi Tech, Vol. 24 No. 3, pp. 376-86. Glynn, L. (2006), A critical appraisal tool for library and information research, Library Hi Tech, Vol. 24 No. 3, pp. 387-99. Gormon, G.E. and Clayton, P. (2005), Qualitative Research for the Information Professional: A Practical Handbook, 2nd ed., Facet, London. Hildreth, C.R. and Aytac, S. (2007), Recent library practitioner research: a methodological analysis and critique, available at: http://myweb.cwpost.liu.edu/childret/practitionerresearch.doc (accessed 15 April 2007). International Centre for Allied Health Evidence (2009), Critical Appraisal Tools, available at: www.unisa.edu.au/cahe/CAHECATS/ (accessed 29 March 2010). Jacobson, T.E. (1993), Another look at bibliographic instruction for business students, Journal of Business and Finance Librarianship, Vol. 1 No. 14, pp. 17-24. Johnson, A.M., Jent, S. and Reynolds, L. (2007), Library instruction and information literacy 2006, Reference Services Review, Vol. 35 No. 4, pp. 584-640. Koufogiannakis, D. (2006), Effective methods for teaching information literacy skills to undergraduate students: a systematic review and meta-analysis, Evidence Based Library and Information Practice, Vol. 1 No. 3, pp. 3-43. Rader, H. (1974), Library orientation and instruction 1973: an annotated review of the literature, Reference Services Review, Vol. 2, pp. 91-3. Rader, H. (2002), Information literacy 1973-2002: a selected literature review, Library Trends, Vol. 51 No. 2, p. 242. Watson-Boone, R. (2000), Academic librarians as practitioner-researchers, Journal of Academic Libraries, Vol. 26 No. 2, pp. 85-93. Yin, R.K. (2003), Case Study Research: Design and Methods, 3rd ed., Sage, Thousand Oaks, CA. Zhang, L., Watson, E.M. and Baneld, L. (2007), The efcacy of computer-assisted instruction versus face-to-face instruction in academic libraries: a systematic review, The Journal of Academic Librarianship, Vol. 33 No. 4, pp. 478-84.

Appendix. Top-ranked business instruction articles by practioner academic business librarians based on EBL levels of evidence and EBLIP critical appraisal checklist Prediction cohort Bowers, C.V.M., Chew, B., Bowers, M.R., Ford, C.E., Smith, C. and Herrington, C. (2009), Interdisciplinary synergy: a partnership between business and library faculty and its effects on students information literacy, Journal of Business and Finance Librarianship, Vol. 14 No 2, pp. 110-27. Magi, T.J. (2003), Whats best for students? Comparing the effectiveness of a traditional print pathnder and a web-based research tool, Portal Libraries and the Academy, Vol. 3 No. 4, pp. 671-86. Orme, W.A. (2004), A study of the residual impact of the Texas information literacy tutorial on the information-seeking ability of rst year college students, College & Research Libraries, Vol. 65 No. 3, pp. 205-15.

Prediction survey Dewald, N.H. (2005), What do they tell their students? Business faculty acceptance of the web and library databases for student research, The Journal of Academic Librarianship, Vol. 31 No. 3, pp. 209-15. Intervention RCT Diamond, T. and McGee, J.E. (1995), Bibliographic instruction for business writing students: implementation of a conceptual framework, RQ: Research Quarterly, Vol. 34 No. 3, pp. 340-60. Intervention cohort Fiegen, A.M., Cherry, B. and Watson, K. (2002), Reections on collaboration: learning outcomes and information literacy assessment in the business curriculum, Reference Services Review, Vol. 30 No. 4, pp. 307-18. Lombardo, S.V. and Miree, C.E. (2003), Caught in the web: the impact of library instruction on business students perceptions and use of print and online resources, College & Research Libraries, Vol. 64, January, pp. 6-24. Intervention case Judd, V., Tims, B., Farrow, L. and Periatt, J. (2004), Evaluation and assessment of a library instruction component of an introduction to business course: a continuous process, Reference Services Review, Vol. 32 No. 3, pp. 274-83. Exploratory systematic review Song, Y.-S. (2004), International business students: a study on their use of electronic library services, Reference Services Review, Vol. 32 No. 4, pp. 366-71. Exploratory survey Cooney, M. (2005), Business information literacy instruction a survey and progress report, Journal of Business and Finance Librarianship, Vol. 11 No. 1, pp. 3-25. Dewald, N.H. (2003), Anticipating library use by business students: the uses of a syllabus study, Research Strategies, Vol. 19 No. 1, pp. 33-45. Exploratory case study Koss, A.I. (1996), Information needs of Kent State University Masters of Business Administration students, Masters thesis N.P., Kent State University, Kent, OH. Littlejohn, A.C. and Benson-Talley, L. (1990), Business students and the academic library: a second look, Journal of Business and Finance Librarianship, Vol. 1 No. 1, pp. 65-88. Thomas, J. (1994), Faculty attitudes and habits concerning library instruction: how much has changed since 1982? Research Strategies, Vol. 12 No. 4, pp. 209-23. About the author Ann Manning Fiegen is a Business Librarian at California State University, San Marcos. She received her MLS from the University of Arizona. Ann Manning Fiegen can be contacted at: aegen@csusm.edu

Business instruction

397

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com Or visit our web site for further details: www.emeraldinsight.com/reprints