Selecting a Mobile App: Evaluating the Usability of Medical Applications

mHIMSS App Usability Work Group July 2012

v1.0

Table of Contents Purpose of this Document ...................................................................................................... 1 What is an App? ................................................................................................................. 1 Mobile App Design ................................................................................................................ 2 What is Usability? ................................................................................................................. 2 Practice Guide to Evaluating App Usability ............................................................................ 7
1. 2. 3. Consider Practice Goals and Measurable Objectives................................................................7 Check Other Resources to Supplement App Market Reviews ...................................................8 Assess the Apps Usability with Typical Clinical Scenarios ........................................................9

Learn More about Usability ................................................................................................. 12 Appendix A. Mobile Design Tenets ....................................................................................... 14 Appendix B. Sample Usability Task Scenarios ....................................................................... 16
Scenario 1: Pediatric Medication Dose Calculation ........................................................................ 16 Scenario 2: Review of Patient Test Results .................................................................................... 19 Scenario 3: Diabetic Patient Education.......................................................................................... 21

Appendix C. Sample Post-Test Questionnaires ...................................................................... 25

1. System Usability Scale (SUS) ..................................................................................................... 26 2. Additional Sample Usability Post-Test Questions....................................................................... 28

Purpose of this Document

This document has been prepared as an aid to the healthcare provider or IT staff member who is engaged in the selection of mobile apps for his/her practice or hospital organization. While formal, professionally conducted, usability evaluations can provide in-depth product comparisons they may not be a practical consideration for small- to medium-sized practices. This guide outlines some basic steps, to include in a mobile app selection process, to assist practices evaluate mobile app usability, based on current usability recommendations and best practices. While the target audience of this guide is for the provider and smaller practices, the steps outlined in this document can also be used by large practices and healthcare institutions.

What is an App?
There are two general types of mobile apps, native and non-native. Native apps reside on a mobile device. Nonnative apps utilize browser interfaces or other gateways to provide the end user with information through a mobile device. A native mobile app requires download through a market place associated with the operating system (OS), such as iTunes for Apple OS devices, Androids marketplace for Android OS-powered phones, and Research in Motions App Store for Blackberry devices. Non-native apps simply require that the mobile device have the appropriate web browser or other interface enabled. Medical apps fall under many different categories, including but not limited to reference apps, like the Physicians Desk Reference or WebMD, medical calculators, and apps designed to access electronic health records (EHR) or personal health information (PHI). Medical reference apps constitute 50 percent of the apps currently present in various platform marketplaces1. However, significant growth is anticipated in mobile apps which seek to enhance or augment EHR interaction and provide additional operational functionality surrounding clinical work flow and decision support. With this expansion of mobile apps and the blurring of lines between some apps and medical devices, the Food and Drug Administration (FDA) issued draft guidance in 2011 to manufacturers, distributors and other entities outlining the types of mobile apps the FDA intends to apply its regulatory authorities to in the future2.

FirstWordSM, Dossier. (May, 2010).Trends in Mobile Medicine: Smartphone Apps for Physicians [Market Intelligence Report].
2

U.S. Food and Drug Administration. Draft Guidance for Industry and Food and Drug Administration Staff - Mobile Medical Applications. July 21, 2011 [Online]. (URL: http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm263280.htm )...

This guide focuses on the usability of native apps, presenting varied considerations and resources related to selection. Selection may ultimately be based on a number of factors/resources to include price, user reviews, usability and security. App ratings are currently available through different sources. Usability can be an additional discriminator to supplement user reviews available within the app marketplace.

Mobile App Design

A well-designed medical mobile app will result in a mobile user experience for the provider that is both useful and usable. The design should address the following key areas: data, layout, feedback, and interaction. In a nutshell: The data is the focus. The interface should be simple and intuitive, with design elements not interfering with the data on a potentially small screen. The layout incorporates typographic elements that communicate meaning and are consistent across screens. Feedback can be provided directly to the design team. Interactions with the interface engage directly with the content and task at hand, keeping cognitive burden to a minimum.

For more details, a set of mobile design tenets developed by mHIMSS is described in Appendix A. Mobile Design Tenets.

What is Usability?
Usability is the effectiveness, efficiency and satisfaction with which specific users can achieve a specific set of tasks in a particular environment3. These usability components can be evaluated and measured (either formally or informally) by means of indirect measures. Efficiency is generally the speed with which users can complete their tasks. Which tasks and clinic processes must be most efficient for success? Can you establish targets for acceptable times for completion of these tasks? Effectiveness is the accuracy and completeness with which users can complete tasks. This includes how easy it is for users to make errors using an app. User errors can lead to inaccurate or incomplete patient records can alter clinical decision-making and compromise patient safety.
3

Schoeffel R. The concept of product usability. ISO Bulletin. 2003;34:67.

Ease of learning is also an attribute of usability, although it is not easy to measure in a quick, informal assessment. User satisfaction is usually the first concept people think of in relation to usability. Satisfaction in the context of usability refers to the subjective satisfaction a user may have with a process or outcome. Satisfaction is highly subjective, but routine questionnaires can provide good insight into what and why problems or issues users have with the system. Platform Optimization is the apps effective and efficient utilization of a devices capabilities. As tablet use increases, a number of developers are optimizing apps for higher resolution, which should increase the efficiency and effectives of a tablet-optimized app. While apps developed for smartphones are often available for download and use on tablets, individuals or organizations seeking apps that are optimized for tablets should ensure the app is designed for such applications. In addition, the purchaser should consider device and platform constraints as the mobile industry regularly releases new operating systems and devices. Make sure you have the right app for the right device in the right setting. Efficiency, effectiveness and satisfaction cannot be taken in isolationall three components must be evaluated and balanced based on your practice's goals and priorities. Table 1 illustrates example questions of efficiency, effectiveness and user satisfaction. Table 1: Example of Efficiency, Effectiveness and User Satisfaction Questions
Usability Component Efficiency Example Questions Is it reasonable to take two minutes to use the app to compute drug dosing for a medication order? How many steps should it take to view vital sign measurements of patients at home through apps that remotely connect to bedside monitors? How long should it take for you to view medical images on a mobile platform? Effectiveness Is data entry too complex or confusing to perform completely and as intended? Does the small screen size on your mobile device affect your ability to accurately interpret radiological images? Is information poorly presented or subject to misinterpretation? User Satisfaction Does the app perform the way you expected?

Usability Component

Example Questions Would you use the information/data obtained from the app in your clinical decision-making process?

Usability can be compromised by the visual design of the user interface. In fact, applications that use lots of colors or visual elements probably would have a lower usability score than well thought out designs that are simpler in appearance and do not increase cognitive burden for the user. Thoughtful use of visual elements can be a supporting factor that enhances product usability. Usability and functionality are often hard to tease apart. While it is not always necessary to fully distinguish the two, it is important to understand the difference. The provider evaluation criteria for medical apps are based on both usability principles and mobile design tenets. Current evaluation methods developed to measure efficiency, effectiveness and user satisfaction of information technology are based on usability principles for all software. A subset of these principles can be applied directly to the evaluation of EMR extension apps4 or native apps. This set has been assessed in earlier literature for electronic medical record (EMR)5 and put forth as being those of key concern for medical apps. In addition, mobile design tenetsguidelines for developing health information technology on mobile devices have been translated to provider mobile app evaluation criteria. Table 2 outlines examples of how a medical app supports the usability principles and mobile design tenets.

The term EMR extension apps is used because certain functionality may differ between native apps or webenabled apps. Purchasers should fully vet the true functionality of apps that provides EMR/EHR access.
5

HIMSS Usability Task Force. Selecting an EHR for Your Practice: Evaluating Usability v1.2. [Online]. (URL: http://www.himss.org/content/files/HIMSS%20Guide%20to%20Usability_Selecting%20an%20EMR.pdf )

Table 2 Usability Principle Attribute Checklist

Usability Principle Mobile Design Tenet Let data scream Example attributes of a usable app Only information, visual elements and functionality necessary to core tasks and decisions are included. Important information stands out, and function options are easy to understand. The focus is the data. The app has a clear, clean, uncluttered screen design. Screen metaphors are familiar to everyday life, or commonly expected computer experiences for the clinician. Workflows match the clinical practice needs. The app is intuitive and easy to learn; minimal, if any, training is required. Iconography and symbols speak naturally. Graphic design and layout have the same look and feel, consistent placement of screen elements (e.g., gutters, columns, margins and captions). Terminology and data entry fields are used consistently. Understanding how one screen works helps you understand how other screens work. Its hard to lose data or destroy time-consuming effort with a wrong tap or wrong choice of buttons. If you make a mistake, the app helps you avoid it or provides a method to recover from errors gracefully (the system is forgiving). The app provides informative feedback about actions you are about to take or have taken. The app displays explanatory messages when processing information and describes how long it might take. The app uses the same words that you use (while providing mapping to standardized codes and terms used for data retrieval). List or entry-form choices are clear and unambiguous. Sentences read like your native language (e.g. English).

Simplicity

Speak my sign What interface?

Naturalness

Grid it

Consistency

Date your users

Forgiveness and Feedback

Speak my sign Date your users

Effective Use of Language

What interface? Get physical

The app minimizes the number of steps/gestures it takes to complete tasks; appropriate defaults are always provided. The app provides navigation options such as shortcuts for frequent and/or experienced users. Navigation methods minimize movements, such as

Usability Principle

Mobile Design Tenet

Example attributes of a usable app scrolling and switching between typing and tapping. Although input methods vary from OS to OS and even within devices, the design ensures that a usable method is present whether the desired input is through typing or the use of gestures. Gestures include single tap, multitap, swipe (where the method of input never leaves the screen tracing the letters to input) and auto complete. Information on screens includes sufficient white-space and large enough fonts to be read easily with high comprehension. No information is in all upper case. Colors are used to convey meaning (e.g., red to indicate medical urgency), not just for visual appeal.

Efficient Interactions

Type less + less type Color carefully

Usability Principle

Mobile Design Tenet What interface?

Example attributes of a usable app The app keeps screen changes and visual interruptions to a minimum during completion of a particular task. Visual interruptions include anything (e.g., dialog boxes) that force you to shift visual focus away from the area on the screen where you are currently reading and/or working to address something else, and then reestablish focus afterward. For example, dialog boxes should be kept to a minimum and should appear in context (adjacent to or just below the control that triggered it). Information needed for a particular task or decision is grouped together on a single screen, rather than requiring you to mentally synthesize information from multiple screens. Alerts presented are clear, concise and informative. The app performs calculations automatically.

Preservation of Context

What interface?

Minimize Cognitive Overload

Practice Guide to Evaluating App Usability

Whether you have downloaded a free medical app, are comparing similar apps or are thinking about purchasing an upgraded version of an app, the following evaluation process steps will assist you in selecting and adopting the most usable app:
1. Consider Practice Goals and Measurable Objectives

Consider what goals are most important to your practice when adopting an app, even if just for your own use, and how these goals relate to usability. For example, are you especially concerned about:

Improving the efficiency of prescription refill requests? Effectively using patient encounter data for patient education? User satisfaction with a clinical decision support tool? How much training will be required for clinicians to become adept with the app? (Ease of learning.) Think about simple baseline measures related to your goals. For example, efficiency is typically measured by how long it takes to perform sample tasks during a usability test. Satisfaction and

concerns can be assessed with questions included in a post-usability test questionnaire. Ease of learning can be assessed by having users attempt the same scenario/task with a number of test tasks and objectively measure learning based on the number of attempts and measures such as time to completion and success rate. Use your practice goals as a baseline measure for reviewing and/or comparing the apps of interest. Depending on the specific nature of effectiveness concerns, targeted questions may be appended to the usability test scenarios (answered after the scenario has been completed) or included in the overall questionnaire to be filled out after the full test has been completed. Usability testing is discussed in a subsequent section of this document, with sample scenarios and usability questions provided in the appendices.
2. Check Other Resources to Supplement App Market Reviews

Keep in mind that many of the publicly available medical app reviews are gathered through informal, non-scientific means. Most reviews and ratings are based on user satisfaction data. Organizations are only just beginning to provide information regarding perceived efficiency and effectiveness of medical apps. Use these reviews to help formulate your questions, but do not allow it to override your own assessment based on your practice goals. Hopefully in the future, a framework for assessing the usability of medical apps will be available, similar to that developed by the National Institute of Standards and Technology (NIST) for assessing the usability of EHRs6. A few resources for medical app reviews:
Resource iMedicalApps.com Description Physician curated and generated reviews with indexed search functions by app type, physician specialty, and device platform. Reviews focused on iPhone medical applications A mobile medical app store developed by healthcare professionals. Happtique is currently developing a medical mobile app certification program to verify clinical appropriateness and technical functionality based on quality and performance standards. Reviews on mobile data systems, which KLAS describes as products

iPhone Medical Apps: new, reviews, trends Happtique

KLAS
6

The EHR Usability Evaluation Protocol (EUP), Technical Evaluation, Testing and Validation of the Usability of Electronic Health Records (NIST Interagency Report 7804), is available at www.nist.gov/healthcare/usability/index.cfm. A draft version of the document was released for public comment in September 2011

Resource

mobihealthnews

Description that focus on providing physicians with access to census, results, and other patient information on a handheld device. News and reviews on the latest medical apps.

Another potential resource to consider is your peers. Check with professional organizations for your specialty to see what they may have to offer. Observe one or two colleagues using the app you are considering. Observe them working with the app. Have a checklist of questions or issues that have arisen so far in your usability evaluation process that relate to their own work. Find out what challenges these users have had learning and customizing the app with the following questions: How long did it take them to be fully up to speed? What was easy or difficult to learn? What workarounds have they had to develop and why? What tasks do they find frustrating due to inefficiencies in the app or complicated design? What kinds of errors do they find are too easy to make? How does the app increase efficiencies?
3. Assess the Apps Usability with Typical Clinical Scenarios

Perform a usability test of the app(s) under consideration. This can be done using fairly simple non-scientific methods, but is a structured means of collecting valuable usability feedback. a. Create a representative set of clinical scenarios that include the essential and frequent tasks that the app will support. Time permitting, include complex tasks. A complex example: Three different apps are used during a patient assessmenta decision support app to identify differential diagnoses, an app to review the patients recent ECG waveform captures and a medication dosing app to help guide the care plan. If you are testing the app with multiple providers in your practice suggest, including task-specific questions for the participant to answer after completing each test scenario. The questions should focus on efficiency, effectiveness, and satisfaction as well as usability concerns related to each task. See Appendix B for some sample primary care practice scenarios. A more in-depth review of usability testing is provided in the following NIST publications: Customized Common Industry Format Template for Electronic Health Record Usability Testing (NISTIR 7742) and Technical Evaluation, Testing and Validation of the Usability of Electronic Health Records (NISTIR 7804) If you are testing the app for your individual practice needs and not gathering feedback from a group of providers, suggest completing the Usability Principle Attribute Checklist outlined in Table 2 of this guide.

b. Prepare a post-usability test questionnaire that captures the overall experience with the app. Include a simple rating scale for each question. See Appendix C for examples. c. If gathering feedback from a representative set of clinicians in your practice, schedule each of them to participate in a one-on-one hands-on test of the app after a brief demo. d. Conduct the usability test. The basic elements of a simple usability test are as follows: o Load the app on one or more mobile devices that can be used for testing. Ideally, the participants should already be familiar with the mobile device type(s) (e.g., iPhone or Android phone) and its basic features. o Instruct the participants to attempt to perform the scenarios without assistance, thinking out loud as they go. o Each participant should answer scenario specific questions as each scenario is completed, and fill out the general questionnaire at the very end. e. Try to provide a test environment that closely simulates (or is) the actual usage environment and typical usage conditions. o Consider screen orientation. Does a certain usage scenario presume use in portrait or landscape mode? Test tasks in both modes if possible. o Consider how the user will hold the device. Is it likely to be used with one hand or two? o For any text entry, consider whether the users will one finger type or will be likely to double-finger or double thumb-type. Will this affect performance? o Are there any other environmental requirements? For example, if the device is to be used in surgery, test the use with a case/cover that meets surgical standards and use with surgical gloves. f. As an observer or usability test facilitator: o o o o o o o Record the time it takes for the participant to complete each scenario. Record key comments made during the test (without interrupting the user). Record whether the participant successfully completed each scenario. Track common errors made by participants. Provide each participant with the post-test questionnaire at the end of the test. Aggregate and synthesize the observations, comments, and responses. If working alone, consider using low cost video recording so you can go back and evaluate the users after the live study sessions. There are many free and low cost software apps that let you use a webcam and a laptop to capture the sessions. Below are links to a few usability testing tools/resources: Jing techsmith.com (free) http://www.usefulusability.com/24-usability-testing-tools/

f. Incorporate the Usability Findings into your Overall Assessment of the App Whatever app you choose, provide feedback to the vendor to help them improve their product. If possible, also participate in future vendor usability studies and user-centered design activities. It will give you direct input to product improvements and improve the likelihood of their success as well as your own.

Learn More about Usability

You can find more information on the web about basic usability concepts and methods from publications specifically related to app usability. Here are a few resources directed at mobile app usability and mobile app design. Fling B. (2009). Mobile Design & Development. North Sebastopol, CA: O'Reilly Media. Available at: http://mobiledesign.org/toc. HIMSS. (June 2009). Defining and Testing EMR Usability: Principles and Proposed Methods of EMR Usability Evaluation and Rating. Available at: www.himss.org/content/files/himss_definingandtestingemrusability.pdf. Accessed March 22, 2012. HIMSS. (August 2010). Selecting an EHR for Your Practice: Evaluating Usability v1.2. Available at: www.himss.org/content/files/HIMSS%20Guide%20to%20Usability_Selecting%20an%20EMR. pdf. Accessed March 22, 2012. Lowry SZ, Schumacher RM. (November 2010). Customized Common Industry Format Template for Electronic Health Record Usability Testing (Report No. NISTIR 7742). National Institute of Standards and Technology, Health Information Technology (IT), Usability Available at: www.nist.gov/customcf/get_pdf.cfm?pub_id=907312. Accessed June 12, 2012. Lowry SZ, Quinn MT, Ramaiah M, Schumacher RM, Patterson ES, Abbott P. (February 2012). Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records (Report No. NISTIR 7804). National Institute of Standards and Technology, Health Information Technology (IT), Usability. Available at: , www.nist.gov/customcf/get_pdf.cfm?pub_id=909701. mHIMSS. Usability, Accessibility, Human Factors & Design. Available at: www.mhimss.org/resource-topics/usability-accessibility-human-factors-design. Prayaga S. (n.d.). Mobile User Interface and Usability Guide [Web log post]. Available at: www.scribd.com/doc/33158304/Mobile-UI-and-Usability-Guide; http://pshyama.wordpress.com/2010/06/17/mobile-user-interface-and-usability-guide/ Roberts D. (n.d). Usability.gov. Usability and Mobile Devices. U.S. Department of Health and Human Services' Office of the Assistant Secretary for Public Affairs, HHS Web Communications and New Media Division web site. Available at: www.usability.gov/articles/newsletter/pubs/052010news.html. U.S. Food and Drug Administration, Center for Devices and Radiological Health and Center for Biologics Evaluation and Research. (July 21 2011) Draft Guidance for Industry and Food and

Drug Administration Staff - Mobile Medical Applications. Available at: www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm2632 80.htm. Accessed March 22, 2011.

Contributors
Patricia Arellano, MS, Planned Systems International, Inc. (PSI) Janet Bochinski, MSN, PNP Beth Elias, PhD, MS, University of Alabama at Birmingham Shannon Houser, PhD, MPH, RHIA, University of Alabama at Birmingham Hank Head, McKesson Staff Liaison - Thomas Martin, MBA, mHIMSS

Appendix A. Mobile Design Tenets Below is a list of mobile design tenets as of January 1, 2012 as published by the mHIMSS Design Tenet Workgroup. A well-designed mobile app should reflect the following design best practices:
Category Data Mobile Design Tenet Let data scream Description a. The focus is the data. The design of the interface does not interfere with the display of the data. b. Eighty percent of screen real estate should be dedicated to data; 20 percent to interface. c. Images, text and numbers are included where visually appropriate. a. Efficiency is critical; one piece of data, entered once only.

Layout

Only handle information once Grid it Type less + less type

a. Graphic design and layout (e.g., gutters, columns, margins,

and captions) are consistent across screens.

a. The app incorporates typographic elements that

communicate meaning and help users understand context and interactivity. Examples include: Size, weight, style, color, spacing. For readability, a single sans-serif typeface and up to six type treatments for the typeface are used. The app speaks the users language. Content is concise, requiring less reading and uses more visual guides. Layout is part of readability. Eye scan patterns matter. Scanning the interface is intuitive. Color is used sparingly. Color helps the information, the interaction, the user experience accomplish the apps intended purpose. The interface is intuitive when performing primary/common user tasks. Feedback, both good and bad, can be provided directly to the design team. Feedback from early adopters is especially encouraged. Repeat customers are cultivated as beta testers. Cognitive burden is reduced when dealing with the interface. Users engage directly with the content and task at hand without thinking about interface.

b. c. d. e.
Color carefully

a. b. a. b. c. d. a. b.

Feedback

Date your users

Interaction What interface?

Category

Mobile Design Tenet

Description

c. The interface makes the most of screen real estate. Every interface item has a function/purpose. Key tasks have priority. No clutter d. Efficiency is critical. The app directly manipulates data elements rather than
the interface. The app displays controls in a progressive manner, only the ones needed at specific points along the intended workflow. Symbolic vocabulary is globally understood. Iconography and symbols speak naturally. The app is designed for different types of users (experienced, novice, inexperienced), who will demand other services to use with the app, and purchase upgrades and future products. The design judiciously leverages keyboard shortcuts, physical engagement with screens (e.g., touch, swipe, pinch and zoom, and lasso), and power user tools. The app works within mobile device limitations such as: No hover text feature. Larger target size. Smaller display. The app leverages new capabilities. Examples include: Touch-based interactions. Location awareness (GPS). Orientation, position, motion awareness. Proximity sensitive. Integrated communications. Integrated communication tools (audio, video, photo). Push notifications (reminders). The app reflects system engineering approach to interface design and the app development lifecycle. The design addresses the entire experience, from inception to implementation; covers the lifecycle of the application.

Speak my sign Repeat customers ROCK

a. b. c.

Get physical

a. b.

c.

Lust to dust

a. b.

Appendix B. Sample Usability Task Scenarios The following examples demonstrate how you can evaluate scenarios for your usability test of an application. You should create your own based on the key practice goals you have defined, the most frequent types of tasks performed as well as the most critical tasks. Scenario specific evaluation criteria can be developed if you are gathering feedback from a representative set of clinicians in your practice. Criteria can be based on the Usability Principle Attribute Checklist identified in Table 2 of this guide. The checklist can be used in its current form to evaluate the scenarios if you are evaluating the app for your individual use. Evaluation criteria tailored to the specific scenarios are provided below.

Scenario 1: Pediatric Medication Dose Calculation

A pediatrician completes and documents an assessment of a two-year-old male patient with a diagnosis of bacterial pneumonia. The assessment data is entered in an EHR, which does not have a pediatric dose calculator. One of the treatments in the care plan is Amoxicillin Suspension. The provider uses her pediatric dose calculator via her mobile device to calculate the dose based on the patients age and weight. The dose calculator is not interfaced to the EHR. Task: Calculate pediatric patients medication dose
1. 2. 3.

Enter the patient weight and age. Select Amoxicillin Suspension from the list of drugs. Identify the correct Amoxicillin Suspension dose for the childs age and weight.

Evaluate Task: Identify the correct Amoxicillin Suspension dose for the childs age and weight
Usability Principle Example characteristics of a usable app
Patient weight and age data entry fields are the focus of the patient data

screen.
Access to drug list and drug edit functions are clearly identifiable. You are able to quickly determine the correct drug, concentration and dose. Correct dose is clearly displayed next to weight and age The app provided all the data elements required for determining the correct

Simplicity

medication dose.
You are able to complete the necessary steps to calculate the drug dose. Essential drug administration information is listed.

Usability Principle

Example characteristics of a usable app

Essential supporting information required to make a decision is available when

calculating the medication dosage.

All the important information you want to quickly reference is readily available
Naturalness

using Info Buttons on the display or quickly accessible with little disruption.
You were able to move between patient data input fields, drug lists and dosing

calculation screens quickly and easily.

Consistency

The app alerts you when a patient age and weight are not consistent with

normal ranges.

Forgiveness and Feedback

The app drug list supports trade and generic names. The patient weight is expressed in either kilograms and/or pounds. The app provided enough medication list categories or groups allowing for

timely drug identification.

Effective Use of Language

You are able to quickly find Amoxicillin in the drug list. Medications can be sorted by favorites, alphabetically, by broad categories,

Efficient Interactions

searched by name, or by category of your choice. The number of steps required to calculate the drug dosage was not excessive. You have no difficulty entering or searching for data on each screen. You do not encounter data entry errors, e.g. patient weight, due to small screen size on your mobile device. Screen size did not affect your ability to read drug names, supporting drug data or drug dosing information. Patient weight, age and other data entry fields are clearly identified and drug lists are easy to read and required drug quickly identified. Colors are used to convey meaning (e.g., red to identify drugs used in medical urgency or Broselow tape color for selection of weight ranges), not just for visual appeal.

You are able to navigate through drug dosing screens minimal steps. Pop up boxes such as do not interfere with displaying the data fields or dose

result.

Usability Principle

Example characteristics of a usable app

Screen size does not hinder viewing all data fields simultaneously.

Preservation of Context

The data on each screen was logically grouped and the number of steps

Minimize Cognitive Overload

required for calculating the Amoxicillin suspension dose allowed for rapid and accurate dose calculation. You are able to quickly confirm the dose calculation if necessary. Visual and audio alerts, such as low device memory warnings, are clear, concise and informative.

Scenario 2: Review of Patient Test Results

A provider sees patient during a follow-up visit to review abdominal ultrasound to rule out gallstones. The provider has access to the local radiology departments picture archiving and communication (PACS) systems ultrasound images via his mobile device. The providers mobile app uses cloud based radiology (DICOM) viewing and sharing web application. The provider pulls up the images and radiology report, via mobile device, to review and discuss results with patient. Task: Review patient radiology results
1. 2. 3.

Identify and select patient. Select ultrasound. Review ultrasound images.

Example characteristics of a usable app
Unique patient identifiers are clearly displayed allowing for selection of the

Usability Principle

right patient.
Patient's demographic information such as gender and age are displayed

prominently and in a location that makes it easy to reference.

Simplicity
You are able to quickly and efficiently identify the right ultrasound images for

the right patient.

Patient identifiers and image details are displayed in the same place on each

Naturalness

image and do not interfere with the ultrasound study images. Images take up the majority of the screen enhancing your ability to view and interpret. The app allows you to view multiple images at one time. Ultrasound image download time is acceptable and supports clinical workflow. Screen icons and navigation options are intuitive and are consistent with common user applications such as web-browsers

Different features of the app are used and behave in a consistent manner. The same gestures such as pinch and zoom, tapping and swiping are used

consistently across screen images.

Alerts and other informational messages follow commonly used color and

Consistency

sound schemes.

Usability Principle

Example characteristics of a usable app

The app notifies you if you select an image that is not uniquely identified to

the patient you have selected.

The app displays explanatory message when processing information such as

Forgiveness and Feedback

downloading images from the PACS, and provides feedback on the expected it will take to complete the process.
The app uses the same words that you use (while providing mapping to

Effective Use of Language

standardized codes and terms used for data retrieval). List or entry-form choices are clear, concise and unambiguous. Sentences read like natural English (or the selected language). Iconography and symbols speak naturally. Language that is not information is not included in the design.

Single tap, multi-tap, swipe gestures work consistently and produce the

desired navigation result across all ultrasound image screens.

The app minimizes the number of steps/gestures it takes to complete patient

Efficient Interactions

and ultrasound selection and navigate through ultrasound images. Images and patient information can be removed from the device easily when the visit is finished.
Patient identifiers and ultrasound image details are displayed are easy to

read.
Colors are used to convey meaning (e.g., change contrast of image), not just

for visual appeal. App allows light calibration capability to control for ambient light and indicates when image of diagnostic viewing quality and non-diagnostic viewing quality.
Pop up boxes do not interfere with the image results. Measurement overlays and contrast tools are available to enhance the MRI

Preservation of Context

image interpretation. Screen size does not affect your ability to view diagnostic quality images. Manipulating and viewing images is fluid and is not hampered by timing screen delays.
You are able to quickly synthesize and interpret image findings. Visual and audio alerts, such as low device memory warnings, are presented

are clear, concise, and informative.

 The app supports measurement overlay tool to automatically calculate

distances between specified points or measurement of an area on images so that you do not have to manually perform the calculations.
Minimize Cognitive Overload

Scenario 3: Diabetic Patient Education

A provider at a diabetes clinic is meeting with a patient to review their blood glucose measures and to help the patient meet their dietary and exercise goals. The provider has downloaded the patients blood glucose measures for the last two weeks from EHR to the mobile device and is using an app that graphs the measures. Each measure that is above or below the target range is highlighted in red. Additional data such as meal times and exercise times can be added to the graph via a tap and drag menu of icons. When the provider taps on the highlighted measure, educational material is displayed that is appropriate to the measure. Additionally, the app can email this material to the patient at the patients request. The trend graph can also be emailed. The trend graph can be resized to allow easier viewing at different levels of detail. Task: Review patient blood glucose trending with patient
1. 2. 3. 4. 5.

Identify and select patient. Select date range for display of blood glucose measures. Add meal and exercise times if desired. Display educational material. E-mail educational material and trend graph if desired.
Example characteristics of a usable app
Unique patient identifiers are clearly displayed allowing for selection of the

Usability Principle

Simplicity

right patient. Patient's demographic information such as gender and age are displayed prominently and in a location that makes it easy for you to reference. You are able to quickly and efficiently specify the date range for display of the blood glucose measures. Trend graph design is concise and uncluttered. Menu of icons for meals and exercise allows for tap and drag to the trend graph and can be hidden. Trend graph can be resized (made larger or smaller) easily to allow viewing at different levels of detail.

Usability Principle

Example characteristics of a usable app

Trend graph is on a grid with time as the X axis and measures as Y, resizing

adjusts from hours to days on the X axis.

Menu of icons for meals and exercise uses appropriate symbols for different

types of food and exercise.

Naturalness
Screen icons, navigation and email options are intuitive and are consistent

Consistency

with common user applications such as web-browsers. Different features of the app are used and behave in a consistent manner. The same gestures such as pinch and zoom, tapping and swiping are used to be consistent with common smart devices and other apps. Alerts and other informational messages follow commonly used color and sound schemes.

Usability Principle

Example characteristics of a usable app

If you select an invalid date range or did not select a date range, the app will

let you know and provides informative instructional messages.

The app displays explanatory messages when processing information (such as

while generating the trend graph) and describes how long it might take.
Forgiveness and Feedback

The app uses the same words that you use (while providing mapping to
Effective Use of Language

standardized codes and terms used for data retrieval). Trend graph markers are clear, concise and unambiguous. Educational material is written to read like plain English (or the selected language). Iconography and symbols speak naturally. Language that is not information is not included in the design. desired navigation result across all components of the screens.

Single tap, multi-tap, swipe gestures work consistently and produce the The app minimizes the number of steps/gestures it takes to complete patient

Efficient Interactions

and measures selection and navigate through educational material and email. Measures and patient information can be removed from the device easily when the visit is finished.
Patient identifiers and trend graph details are displayed are easy to read. Colors are used to convey meaning not just for visual appeal. Educational material is displayed in concise summary form, resizing increases

or decreases level of detail. Audio is used to convey meaning, not just for basic device interaction.

Preservation of Context

Pop up boxes do not interfere with the image results. Educational material is displayed with the measure in question. Screen size does not affect your ability to view graphs. Manipulating and viewing the trend graph and educational material is fluid and is not hampered by timing screen delays.

Visual design allows you to quickly synthesize and interpret the trend graph. Measures such as the average glucose level by day or by week are

automatically calculated.

Minimize Cognitive Overload

Appendix C. Sample Post-Test Questionnaires Post-usability test questionnaires are administered to a test participant immediately after completing the test scenarios and before any debriefing. The first questionnaire included in this Appendix is the Digital Equipment Corporation System Usability Scale (SUS)7. This is a general questionnaire that has been used in many usability studies; if used as intended, it has been shown to be a reliable evaluation tool. A brief overview of the SUS can be found in Wikipedia 8. For instructions on how to score responses on the SUS as well as a sample scored questionnaire see Brooke9. The second sample questionnaire illustrates how you might include questions more directly related to the usability principles discussed as key for medical app design. Include questions aligned with your practice goals. For instance, if efficiency in the review of clinical documentation is one of your key concerns, then include questions about the efficiency and ease of use of the documentation tools. This set of questions would not constitute a scientifically reliable tool, just a simple means of collecting subjective responses from your participants in a structured way. There would also be no official means of scoring responses to these questions other than face-value impressions.10

Brooke J. (1996). "SUS: a "quick and dirty" usability scale". In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & A. L. McClelland. Usability Evaluation in Industry. London: Taylor and Francis. www.usabilitynet.org/trump/documents/Suschapt.doc.
8

Wikipedia. System usability scale (2011). Retrieved from http://en.wikipedia.org/wiki/System_usability_scale.

Brooke J. (1996). "SUS: a "quick and dirty" usability scale". In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & A. L. McClelland. Usability Evaluation in Industry. London: Taylor and Francis. www.usabilitynet.org/trump/documents/Suschapt.doc.

1. System Usability Scale (SUS)

Strongly Disagree 1. I think that I would like to use this system frequently. 2. I found the system unnecessarily complex. 1 2 3 4

Strongly Agree 5

3. I thought the system was easy to use. 1

4. I think that I would need the support of a technical person to be able to use this system. 5. I found the various functions in this system were well integrated. 6. I thought there was too much inconsistency in this system. 7. I would imagine that most people would learn to use this system very quickly. 8. I found the system very cumbersome to use. 1

9. I felt very confident using the system.

10. I needed to learn a lot of things before I could get going with this system.

2. Additional Sample Usability Post-Test Questions

Strongly Disagree 1 1. The app had a clear, clean, uncluttered screen design. 2. The app kept screen changes to a minimum during completion of a task. 3. The app minimized the number of steps it took to complete tasks. 4. Information presented on screens was easy to comprehend quickly. 5. Information needed for a specific task was grouped together on a single screen. 2 3 4

Strongly Agree 5

6. Choice lists were clear and unambiguous.

7. Clinical documentation tools were efficient to use.

8. Alerts were only presented at appropriate times. 9. Data could be entered once then used in multiple places.

10. I felt confident I could make a mistake without losing my work.

General Disclaimer
The inclusion of an organization name, product or service in this document should not be construed as a HIMSS endorsement of such organization, product or service, nor is the failure to include an organization name, product or service to be construed as disapproval.