Information Retrieval Behaviors among EBM Practitioners

when performing Evidence Based Medicine

Vinesha Selvarajah (1st Year PhD)
Supervised by: Anushia Inthiran, Nathorn Chaiyakunapruk, Lai Nai Ming
School of Information Technology, Monash University Malaysia,
Bandar Sunway, 47500 Subang Jaya, Selangor, Malaysia

vinesha.selvarajah@monash.edu

encouraged the practice of EBM among clinicians, physicians,

nurses, medical students and all health officers within hospital
settings [5; 6; 8]. These medical professionals integrate EBM into
their clinical practice to solve problems faced during consultation
especially when choosing best diagnostic and therapeutic
procedures for patients. It is important to practice proper
information retrieval skills when searching for information online
so that the most relevant information is retrieved. Retrieving
wrong or irrelevant information leads to making wrong medical
decisions thus putting patients at risk.

ABSTRACT
This paper reveals the preliminary analysis of how EBM
practitioners retrieve EBM related information during their
clinical rounds at a public teaching hospital in Malaysia. We
focused on studying EBM practitioners searching behaviors
during EBM process. However, in this paper we only present
analysis for query issuing pattern and result viewing behavior of
the retrieval process. We collected our preliminary data using
Moraes Key-Logging software that captures participants
information searching behavior. We recorded 30 preliminary
search sessions via convenience sampling and analyzed our data
using descriptive statistics. Our findings indicate that 90% of the
search sessions successfully returned results that the participants
searched for. Among the queries issued, 24.47% were ineffective
queries. Result viewing behaviors revealed 56 online sources
were clicked where 33.33% belonged to PubMed, Medscape and
UpToDate. Our result shows higher average query length and the
number of medical terms used in queries compared to previous
studies. We also found that result viewing behavior using multiple
tabs confuses the participants and are more likely to cause
overlooking of online resources on some of the opened tabs. The
outcome of our study leads to better initiatives on query issuing
and result viewing strategies for EBM related searches to improve
the EBM process among practitioners.

Before the existence of Internet, sources of EBM evidences were

limited to books, printed journals, and research papers. Ever since
the Internet, medical professionals turned to online search engines
such as Google.com and other medical domains such as
UpToDate and Medscape when searching for health-related
information [4]. The increasing dependency on digital devices
such as computers, mobile phones and tablets further encouraged
medical professionals to search for information online. Searching
for information requires skills and experiences that are different
from the traditional searching methods. Barriers faced when
searching for EBM related information are lack of skills and
knowledge to practice EBM, insufficient training, difficulty in
searching and accessing resources, un-suitable appraisal
techniques, and time constraints [14]. In developing countries, the
difference in quality of study and trials among the EBM resources
becomes a major challenge when choosing appropriate evidence
for medical decision making locally since most evidences are
derived from developed countries. Due to high quality threshold
standards set for publication, many lower quality studies in
developing countries are excluded [2]. Other issues includes
limited access to literature database, lack of time to attend
workshops, insufficient IT facility, poor internet resources, and
the shortage of role models practicing [3]. Despite the all the selfperceived issues reported from past researches using selfadministered questionnaires, the actual searching behavior of
EBM practitioners when searching for online EBM related
information have not been conducted in a developing country.

CCS Concepts
Information systemsInformation Retrieval

Keywords
Interactive Information Retrieval, Evidence Based Medicine,
Practitioners, Search Behaviors, NICU, Malaysia

1. INTRODUCTION
Evidence-Based Medicine (EBM) is the practice of integrating
medical evidences and clinical experiences in making medical
decisions [9]. Several studies have revealed that a decade long of
EBM related research and training in the medical field

Therefore in this paper, we attempt to answer the following

research questions: (i) what are the query issuing pattern and
result viewing behavior of EBM practitioners when performing
EBM? (ii) What are the sources used to perform EBM related
search and why were they used? We conducted an exploratory
study to analyze the searching behaviors of EBM practitioners
when retrieving EBM related information. We collected 30
preliminary search sessions to observe what the preliminary
results are showing us. The actual experiment is ongoing and will
continue until theoretical saturation is achieved. This remainder of

Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or
distributed for profit or commercial advantage and that copies bear this notice and
the full citation on the first page. Copyrights for third-party components of this
work must be honored. For all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
CHIIR '16, March 13-17, 2016, Carrboro, NC, USA
ACM 978-1-4503-3751-9/16/03
DOI: http://dx.doi.org/10.1145/2854946.2854949

369

information where 1 out of the 11 search session failed to produce

desired outcomes. An exceptional search session reported to not
have clicked any source or any links. As such, the data cannot be
generalized. That particular session however had issued 1 query
with a query length of 6 terms and found the answer on the search
result page itself. The result viewing behavior also revealed that
50% of the search sessions recorded had opened more than one
search tab to improve the result viewing process. However
13.33% of the search sessions that opened multiple tabs reported
at least one of the tabs were opened without being viewed
throughout the searching process.

this paper is divided into 3 sections. We present our adopted

methodology in Section 2, followed by our findings in Section 3
and the discussion of findings in Section 4. We then present our
remaining work in Section 5, conclude our paper with our
preliminary contributions and future works in Section 6.

2. METHODOLOGY
We conducted an exploratory study based on convenience
sampling at the Neonatal Intensive Care Unit (NICU) of a public
teaching hospital in Malaysia. We adopted user based study to
study the searching behavior of EBM practitioners, Interactive
Information Retrieval (IIR) [7] approach was applied. We used
Morae Key-Logging software version 3.3.3 to record the search
sessions. The software was installed on a HP Pavilion dm4
(model: ProBook 440 G1) laptop that is available at the NICU
ward. The laptop runs on a 64-bit Window 7 Professional
operating system, with a 2.40GHz Intel(R) Core i3-4000M
processer, and Random Access Memory (RAM) of 4.00 Gigabyte
(GB). The laptop is connected to the internet via the local network
provider. We included semi structured interview to be collected
before and after the search recordings. Data is collected when an
EBM practitioner at the NICU ward finds the need to look for
EBM related information online. The practitioners consent to
participate in this study is first obtained. The practitioners were
then interviewed (pre-search interview) before searching for
information to elicit information regarding purpose and factors
contributing to search. The practitioners then proceed on
searching for their information online via the laptop pre-installed
with the recording software. Each search session is recorded from
the beginning of search and stops when there is no more search
activity for a particular search session, or when the practitioner
decides to end the search. After the each session, the practitioners
are interviewed again (post-search interview) to collect
information on user experiences, feedback, and how the retrieved
information is used in their medical decision making. The success
of each search session was measured by the practitioners selfperception. Search sessions are claimed to be successful when
practitioners are able to find the answers they were looking for.
Note taking and memos are recorded by the researcher to record
any additional information to aid the analysis of the recordings
later. The recordings are then analyzed using Morae Manager.

4. DISCUSSION
The average number of query issued per search session showed
similar pattern to previous researches indicating around 2.34
queries were issued per search session [12; 13]. However the
average query length does not support previous study where in
our case higher average query length was issued in the search
sessions [13]. Query length differs with the purpose of search
carried out. More sophisticated search may result in higher query
length compared to simpler search. Similarly, the number of
ineffective queries issued in our study is high. The recurrent of
ineffective query in a search session lengthens the duration of the
searches and affects the practitioners time management. Search
sessions during clinical round should be kept short as it conflicts
with practitioners patient consultation time.
Our findings indicate that a higher number of medical terms were
used in the queries. This finding is supported by [13], a higher
number of medical terms used in a query increase the probability
of searching for desired information. However, our findings
differed with [11] due to the differences in the search domains
used. Our findings showed dissimilarities in average query length,
issuance of ineffective queries and the number of medical terms
used. There is a need to develop new query formulation strategy
to increase the average query length, reduce the number of
ineffective queries and increase the number of medical terms used
in the queries.
Table 1. Query issuing behavior per search

3. FINDINGS
We used descriptive statistical analysis to study the minimum
value (Min), mean, median, standard deviation (SD), and the
maximum value (Max) of our preliminary data. Overall, 27 out of
the 30 search sessions were successful searches. We provide the
result of the query issuing behavior from the searching process in
Table 1. Majority of the search sessions have an average of 2 to 3
queries issued before achieving desired result. With the majority
of successful search sessions, our analysis revealed that 24.47%
out of total number of queries issued returned zero results. These
queries are categorized as ineffective queries. We also found that
there were 120 terms used in the issued queries and 75% of terms
used were medical terms. At least 2 medical terms are found in an
issued query. Table 2 shows the result viewing behavior from the
search process. Our analysis revealed 56 different online sources
accessed and viewed with an average viewing of 2 different
sources per search session. The highest number of viewed sources
equally belonged to PubMed (11.11%), UpToDate (11.11%) and
Medscape (11.11%). Apart from the main sources, there were 11
out of 30 search referred sub links within a source to search for

Query Issuing
Behaviors
Number of Queries

Min

Mean

3.17

Ineffective Queries

0.93

Average Query
Length
Number of Medical
Terms

4.01

3.00

Median

SD

Max

2.00

2.937

14

0.00

1.437

4.00

1.587

2.00

3.877

17

Table 2. Results viewing behavior per search

Result Viewing
Behaviors
Number of sources
viewed
Number of results
clicked
Number of sub links
clicked
Number of tabs
opened

370

Min

Mean

Median

SD

Max

3.00

2.00

2.449

11

3.50

2.00

3.256

14

1.27

0.00

2.778

13

2.67

1.50

2.857

15

We found that the most visited sources are PubMed, UpToDate

and Medscape. Previous researches indicates with UptoDate being
the top referred source followed by PubMed [10]. However, our
findings are supported by a recent research stating that when
searching for medical information, the highest referral sources
were UpToDate followed by Medscape and PubMed [1]. Apart
from that, our study also revealed the drawbacks of having
multiple tab opened when viewing results. Having multiple tabs
opened confuses the user thus causing one or more tabs opened
without viewing. Proper result viewing strategy should be
established to educate practitioners on how to manage their results
viewing behavior so that they are able to cover all important
results in a shorter time. This would prevent the overlooking of
retrieved results and important information.

[4]

[5]

[6]

5. REMAINING WORK

[7]

The project is further divided into several data collection methods

including interview of the EBM practitioners (Pre-Post Search
Interviews). The analysis of the interviews is still ongoing and we
intend to use different methods of analysis on our data.
Additionally, we intend include Focus Group Discussion (FGD)
to our study to trigger discussion among EBM practitioners on
the barriers and challenges faced during the retrieval process.

[8]

6. CONCLUSIONS AND FUTURE WORK

This exploratory study analyses the query issuing and result
viewing behaviors of EBM practitioner when performing EBM
related search. We revealed issues related to ineffective queries
and the ineffective usage of tab when viewing results. The
outcome of our study provides design initiatives to query issuing
and result viewing strategy to improve the retrieval process. The
data from this study is only at its preliminary stage and analyzed
only using descriptive statistical analysis. Future work in this
study requires the completion of the remaining work described
earlier and further analysis of the data using both qualitative and
quantitative analysis method. A conceptual design framework to
assist EBM practitioners in their EBM searching strategy will be
implemented.

[9]

[10]

[11]

7. ACKNOWLEDGMENTS
The research study is funded by the Ministry of Education under
the Fundamental Research Grant Scheme (FRGS) FRGS/2/2013/ICT02/MUSM/03/1.

[12]

8. REFERENCES

[13]

[1] Boruff, J.T. and Storie, D., 2014. Mobile devices in

medicine: a survey of how medical students, residents, and
faculty use smartphones and other mobile devices to find
information. Journal of the Medical Library Association :
JMLA 102, 1 (02//received08//accepted), 22-30. DOI=
http://dx.doi.org/10.3163/15365050.102.1.006.
[2] Chinnock, P., Siegfried, N., and Clarke, M., 2005. Is
Evidence-Based Medicine Relevant to the Developing
World? PLoS Med 2, 5, e107. DOI=
http://dx.doi.org/10.1371/journal.pmed.0020107.
[3] Dans, A.L. and Dans, L.F., 2000. The need and means for
evidence-based medicine in developing countries. Evidence

[14]

371

Based Medicine 5, 4 (July 1, 2000), 100-101. DOI=

http://dx.doi.org/10.1136/ebm.5.4.100.
De Leo, G., Lerouge, C., Ceriani, C., and Niederman, F.,
2006. Websites Most Frequently Used by Physician for
Gathering Medical Information. AMIA Annual Symposium
Proceedings 2006, 902-902.
Dorsh, J.L., Aiyer, M.K., and Meyer, L.E., 2004. Impact of
an evidence-based medicine curriculum on medical students'
attitudes and skills. Journal of the Medical Library
Association 92, 4 (10//received04//accepted), 397-406.
Hersh, W.R. and Hickam, D.H., 1998. How well do
physicians use electronic information retrieval systems?: A
framework for investigation and systematic review. JAMA
280, 15, 1347-1352. DOI=
http://dx.doi.org/10.1001/jama.280.15.1347.
Kelly, D. and Sugimoto, C.R., 2013. A systematic review of
interactive information retrieval evaluation studies, 1967
2006. Journal of the American Society for Information
Science and Technology 64, 4, 745-770. DOI=
http://dx.doi.org/10.1002/asi.22799.
Majid, S., Foo, S., Luyt, B., Zhang, X., Theng, Y.-L., Chang,
Y.-K., and Mokhtar, I.A., 2011. Adopting evidence-based
practice in clinical decision making: nurses' perceptions,
knowledge, and barriers. Journal of the Medical Library
Association : JMLA 99, 3 (11//received02//accepted), 229236. DOI= http://dx.doi.org/10.3163/1536-5050.99.3.010.
Sackett, D.L., 1997. Evidence-based medicine. Seminars in
Perinatology 21, 1 (2//), 3-5. DOI=
http://dx.doi.org/10.1016/S0146-0005(97)80013-4.
Sayyah Ensan, L., Faghankhani, M., Javanbakht, A.,
Ahmadi, S.-F., and Baradaran, H.R., 2011. To compare
PubMed Clinical Queries and UpToDate in teaching
information mastery to clinical residents: a crossover
randomized controlled trial. PloS one 6, 8 (2011), e23487.
DOI= http://dx.doi.org/10.1371/journal.pone.0023487.
Yang, L., Mei, Q., Zheng, K., and Hanauer, D.A., 2011.
Query Log Analysis of an Electronic Health Record Search
Engine. AMIA Annual Symposium Proceedings 2011(10/22),
915-924.
Zhang, J., Wolfram, D., Wang, P., Hong, Y., and Gillis, R.,
2008. Visualization of health-subject analysis based on query
term co-occurrences. Journal of the American Society for
Information Science and Technology 59, 12, 1933-1947.
DOI= http://dx.doi.org/10.1002/asi.20911.
Zhang, Y., Wang, P., Heaton, A., and Winkler, H., 2012.
Health information searching behavior in MedlinePlus and
the impact of tasks. In Proceedings of the Proceedings of the
2nd ACM SIGHIT International Health Informatics
Symposium (Miami, Florida, USA2012), ACM, 2110434,
641-650. DOI= http://dx.doi.org/10.1145/2110363.2110434.
Zwolsman, S., Te Pas, E., Hooft, L., Waard, M.W.-D., and
Van Dijk, N., 2012. Barriers to GPs' use of evidence-based
medicine: a systematic review. The British Journal of
General Practice 62, 600
(06/2509/06/received10/10/revised12/09/accepted), 511e521.DOI= http://dx.doi.org/10.3399/bjgp12X652382