AN EVALUATION OF PATIENTS’

EXPERIENCED USABILITY OF A
DIABETES MHEALTH SYSTEM USING A
MULTI-METHOD APPROACH

Erika Trisliana – 13414030

Aditya Binowo - 13414057
 OUTLINE
2

▸ Introduction
▸ Materials and methods
▸ Results
▸ Discussions
▸ Conclusions
3

1.
Introduction
4
Mobile Health Technology
▸ Support systems for diabetes self-management using
Information and Communication Technology (ICT)

▸ Successful improving glucose management and aided in

regulating lifestyle changes

▸ Self-coaching by allowing users to interact with algorithm-

based systems that provide individually tailored messages
and feedback messages
5
Mobile Health Technology
▸ Need for improved usability to allow patients to get comfy in
their interaction with mHealth applications.

▸ Most studies lack methodological rigor and use a sigle

method for data collection

▸ Qualitative data analyses lack a structured approach

▸ Methodological improvements could result in more

comprehensive analysis and reproduciblity, also improved
user interaction and performances
6
Multi-Method Approach

▸ Observation, field notes

and interview first
identified by Patton

▸ Data collection used user-

based testing, Think Aloud
protocol, open-ended
interviews and a short
answer questionnaire
7
Structured Approaches for Data Analysis
▸ Framework Analysis (FA)

▸ Purpose: to assist with the analysis of descriptive, textual

source data to produce reliable and valid qualitative reports

▸ Consists of five stages (1) familiarizing oneself with the

data, (2) identifying a thematic framework to be used, (3)
indexing and applying the framework to data, (4) charting
the data, (5) mapping and interpelation
8
Structured Approaches for Data Analysis
▸ Usability Problem Taxonomy (UPT)

▸ Purpose: Providing a clear structure in usability problem

definition using classification scheme and framework for
characterizing usability programs according to their
dimensions

▸ Two perspectives: The task-artifact approach and The

Object-Action Interface Model
9
Structured Approaches for Data Analysis
10
Study Aim

▸ Using a multi-method approach for data collection

during a mHealth usability evaluation

▸ Applying structured approaches to data analysis

by using the FA method and UPT
11

2.
Materials and
Methods
12
The Evaluated System
• Interactive, SMS-based movile
intervention
• Combined mobile phone
solution and web service
• Patients interact by either
sending in or being prompted
to send in their Type 2
Diabetes, self-management
values
• Using the web service, they
can enter and/or review their
result
13
Study Sample and Setting
▸ 2317 patients met the criteria for the study

▸ 10 patients picked randomly by computer randomization and

invited to take part in usability evaluation

▸ Inclustion criteria:
(1) Patients diagnosed with Type 2 diabetes
(2) No cognitive impairment
(3) Familiarity and some knowledge and use of computer, the
Internet, and cell phone
(4) The ability to speak and understand English
14
Data Collection Methods
▸ User-related method:
(1) Think Aloud: Usability assessment method commonly
employed to determine users’ thoughts and opinions while they
peform a list of specified tasks with a system
(2) Tasks for this study were based on common patient user
interaction with the system that validated by a panel consisting
of a physician and a nurse whose specialities were diabetes
•Uploading glucose values •Interpreting a blood pressure measurement
•Interpreting a glucose measurement •Setting personal tracking goals
•Correcting recorded glucose measurement •Setting medication reminders
value •Setting a physician appointment reminder
•Exporting glucose measurement to PDF
15
Data Collection Methods
▸ User-related method:
(3) Respondents completed open-ended interview and a post-
experience questionnaire

The three questions in the interview asked patients to comment on

sections that were inadequately designed and any further
comments they might have about system usability

In post-experience questionnaire, first section consisted of short

answer question about IT, mobile phone, internet used. The second
section included open-ended questions about the system
16
Study Procedure
▸ Patients were asked to informed consent
▸ Evaluation procedure and asked if they hand any question
▸ Standardized training to simulate an actual patient educational
process in a health clinic
▸ The Think Aloud and Usability evaluation (Interaction were
digitally audio and video recorded using Morae)
▸ Patients were interviewed about their experiences an perceptions
about the system
▸ Completed post-test questionnaire on their perceptions about the
usability of the system
17
Data Analysis
18

3.
Results
19
Sample’s Characteristics
▸ Gender composition: Women (6) and Men (4)

▸ Aged between 40 – 59

▸ Used technology daily

Definition of used technology in here was used computer,
internet, and mobile phones to make and receive calls

▸ Majority agreed that using technology in health care is a

positive development and positive to their diabetes self-
management
20
Usability Problems
▸ A total 117 initial usability problems were detected using 3
methods (usability test, post interview, and questionnaire)
and consolidated into a unique list of 19

▸ The average severity for the whole site was 2.47 (serious)

▸ The views with highest number of problems were those with

several steps in a task
21
Example of Identified Usability Problems
22
Critical and Severe Usability Problems

▸ The most critical usability problems were located in Glucose

Reading View and Glucose Diary View

▸ This problems commonly caused by

▹ confusing sequence to finish the task
▹ multiple function of the button
▹ no navigation system
▹ display (location or label)
23
Results for Methodological Approaches

▸ The usability test

detected most of the
usability problems
both on its own and in
combination with
other methods
24
Problem Classification

▸ This figure indicates

distribution of problems for
each UPT classification level
and the depth of classification

▸ Critical problems were

classified into the artifact
components. Task
classifications involved
functionality concern.
25
Problem Classification (2)

▸ This figure
indicates whole
system problems
related to
presentation,
visualization,
system design and
structure (not
match with user
perspective).
26

4.
Discussion
“
The usage of multiple methods
resulted in more comprehensive
understanding of patients’
interactions with the system and
provide triangulation on severe
usability issues.
28
Usability Problem Types and Classifications

▸ The majority of problems were located in the Glucose

Readings View and two other views, the Glucose Diary View
and Blood Pressure View, had the highest severity ratings.

▸ Usability problems included issues with patients’ data (delete

and enter) and information about how to save a file and many
more as have already said in section 3.

▸ The results pointed to difficulties patients had completing

common tasks (frequent basis for diabetes self management)
so these concerns need to be resolved quickly.
29
Data Collection Methods

▸ Using a multi-method approach is beneficial to identifying

usability problems especially initial usability problems
because each method provide different point of view

▸ Usability testing (think aloud) identifies more problems,

recurring usability problems and defines the underlying
causes.

▸ In-depth interview did not contribute any unique consolidated

because may be the patients echoed only the issues they
voice during testing so they did not have any other insights
30
Data Collection Methods (2)

▸ Used open ended questions to provide an opportunity for user

to express their thoughts freely and perspective about
problems that may not surface during testing.

▸ The post questionnaire can verified significant usability

problems. But, in this study this method only generated few
issues due to study sequencing. (Questions seem similar to
users, fatigue at the end of 2 hours sessions, only mention
severe problems to them)
31
Structured Data Analyses

▸ The method used in this paper provided a beneficial way of

structuring data and coding large amounts of data.

▸ The weaknesses of UPT : vague problem descriptions and

problem could not be fully classified (due to insufficient
descriptions)

▸ The strengths of UPT : detailed, could group problems of a

similar nature easily
32
Contributions to The Literature

▹ Overall, this study adds to the body of literature because it

provides an example of a multi-method approach for
collecting data on usability problems , for standardizing data
analyses, and contributing to the reproducibility of results in
qualitative usability studies.
33
Lessons Learned and Recommendation

▸ FA analysis can generated a large amount of data even with

small number of samples but the coding need laborious
process

▸ This method is recommended for studies employing with 2

researchers

▸ UPT is easy to use for individuals familiars with HCI, usability,

and familiar with the system.
34
Limitations and Future Research

▸ Small sample size

▸ Convenience sample of patients with diabetes may not
generalize findings to the population

▸ Recommendations for future research

▹ Use purposeful sampling to select users with different characteristics to
maximize variability
▹ Involved users early and in development process
▹ Use larger samples
35
Conclusions

▸ This study provides an example of a multi-method design for

both data collection and data analyses. Multiple data
collection methods resulted in a more comprehensive set of
usability problems and helped triangulate data. The
structured data analyses allowed reproducible steps and data
validation (triangulation), a method of determining the most
severe problems for users. We recommend the use of
multiple data collection methods to uncover a variety of
problem types and severity levels. We also recommend the
use of the FA and UPT methods during data analyses.
36

THANKS!
Any questions?