Usability Heuristic Test of Mobile Fitness Application

I just finished a master’s in healthcare informatics and through that process learned quite a few things.  I am planning a series of blog posts (three or four) that highlight several of the papers I wrote that might be of interest to readers of this blog.  The subjects will cover usability of a popular mobile fitness application, an update on what is happening in the medical device integration space, and an attempt at building a medical device integration knowledge base.

My healthcare background as a clinical engineer in the last 26 years has been mainly in operations or industry.  Any exposure I had to research was usually on the periphery.  One of the reasons I decided to pursue this masters was to see what research had to say about two areas of interest to me:  medical device integration and remote monitoring.  For most of my assignments and the final project in the degree program, I used those two subjects as themes.  Interestingly, my professors and fellow colleagues and classmates were not aware of many of the products and ideas that I had been exposed to in my clinical engineering career.  For example, in my knowledge base class, we talked about the use of intelligent agents in healthcare information technology products (electronic health records and clinical decision support).  I offered to the class in one of our discussion series an automatic external defibrillator (AED) as an example of the use of clinical decision support with an intelligent agent at the point-of-care.   Over the course of my studies, I realized I had been steeped in the techniques and ideas of informatics supporting healthcare from the beginning of my clinical engineering career; it was just at a different point in the healthcare delivery chain.  In fact, I could see that my profession had in many ways solved some of the problems that healthcare information technology was grappling with now.

Usability heuristics

With the above backdrop, I will begin this mini-blog series with the results of my usability heuristic study of a popular mobile fitness application.  First, a few definitions are in order.  A heuristic is a fancy name for a ‘rule of thumb’ and usually “employs a practical method not guaranteed to be optimal or perfect, but sufficient for the immediate goals.” Nielsen, considered one of the ‘fathers’ of usability heuristics, defines usability as a “quality attribute that assesses how easy a user interface is to use.” Nielsen developed in 1995 Nielsen’s Ten Usability Heuristics. The ten heuristics are:  visibility of system status; match between system and the real world; user control and freedom; consistency and standards, error prevention; minimize the user’s memory load; flexibility and efficiency of use; aesthetic and minimalist design; help users recognize, diagnose and recover from errors; and, help and documentation.

Mobile phones/applications and usability

Mobile phones and mobile applications pose an interesting usability challenge due to the mobile phone’s limited screen size or footprint. This leads to constraints with the ergonomics of finger and hand interaction and a limited visual presentation area.  These constraints can become even more unwieldy with older patients or patients with cognitive issues.  In the realm of chronic disease management, the use of remote monitoring and mobile phones being used as networking hubs, these constraints are more important to manage to encourage mobile application use.  In addition to those constraints, studies have shown that 74% of downloaded mobile applications are no longer used after the tenth use.  The second most prevalent reason given for no longer using the mobile application was ‘user-friendliness’ while ‘ease of navigation’ was listed as the most important feature of a mobile application.

Why did I choose a mobile fitness application to evaluate? Most chronic disease self-management profiles encourage physical activity or fitness programs as part of the protocols, and the use of an application to manage the process can provide data to be combined with physiological parameter management through remote monitoring programs to assess the overall patient progress.  Additionally, mobile fitness applications are what could be termed bridge applications between health and wellness and mobile health depending on how and who uses the application.  Lastly, this type of application could also be considered a fairly mature product in a highly dynamic market, so there would be an inherent stability in the product offerings and their capabilities and functionality.

Mobile application usability heuristic checklist

Based on the reasons above, usability should be a primary design consideration of a mobile application and especially a mobile health (mHealth) application.  With that in mind, several researchers have taken Nielsen’s Ten Usability Heuristics and modified them to apply to mobile phone based applications.  In my research for a mobile phone usability heuristic checklist, I found a very extensive one developed by Gomez, et al, which modified and expanded the ten heuristics to thirteen:  changing ‘minimizing the user’s memory load’ to ‘recognition rather than recall’, changing “help users recognize, diagnose and recover from errors’ to ‘error recovery’, and,  adding skills, pleasurable and respectful interaction, and privacy.  These thirteen heuristics are evaluated through a 228 question checklist in which a tester/user answers the question while using the mobile application.  The thirteen heuristic principles are further subdivided into application features upon which questions can be asked. Examples of questions asked for each heuristic are shown below (very small sample of the 228 questions):

Visibility of system status:

System status feedback:

If pop-up windows are used to display error messages, do they allow the user to see the field in error?  In multipage data entry screens, is each page labeled to show its relation to others?

Match between system and the real world (mental model accuracy):

Menus:

Do menu choices fit logically into categories that have readily understood meanings?

User control and freedom:

Process confirmation:

When a user's task is complete, does the system wait for a signal from the user before processing?

Consistency and standards:

Designing consistency:

Is the number of colors used constrained up to four? Are additional colors saved for occasional use only?

Error prevention:

Fat-finger syndrome:

Are touchable areas sufficiently big? (Research has shown that the best target size for widgets is 1cmx1cm for touch devices.)

Recognition rather than recall:

Memory load reduction:

Does the system provide visibility: that is, by looking, can the user tell the state of the system and the alternatives for action?

Flexibility and efficiency of use:

Search:

Are search results shown in a comprehensive manner to the user?

Aesthetic and minimalist design:

Icons:

Are all icons in a set visually and conceptually distinct?

Error recovery:

When signaling an input error in a form, is the text box that needs to be changed specifically marked?

Help and documentation:

Is the help function visible; for example, a key labeled HELP or a special menu?

Skills:

Does the system correctly anticipate and prompt for the user's probable next activity?

Pleasurable and respectful interaction:

Input Data:

Can users save history and select previously typed info?

Privacy:

Are protected or confidential areas only accessible with certain passwords?

Applying the checklist to a mobile fitness application and results

Next up was choosing a mobile fitness application to test.  I had several requirements:  free, usable on the Android OS, and still available and supported.  Fascinatingly, the ‘still available and supported’ requirement became the most constraining.  Many of the ‘best’ mobile applications listed by different organizations were no longer available after two or three years.  As an aside for product longevity, this means that an application developer needs to progress from a free to a paid product version (‘premium’ version) and/or be purchased by or collaborate with a larger corporation or source of funding to support the continuing development and support of the software.  With this scenario, the vision and/or goals of the original developer may then be altered with the business relationship and this can be reflected in the application look and feel as it evolves.  In the end I chose a free application that had progressed from an ‘independent’ and free mobile application to one supported by a large corporation.  The mobile application had a free version and a paid ‘premium’ version, advertising banners on the free version and affiliate links for in application purchases.  All three of these approaches are fairly typical marketing and funding source mechanisms for mobile applications today.

After downloading the mobile application onto my phone, I used the usability heuristic checklist by answering yes, no, yes/no (the question was general enough that the answer to the question for the way the mobile application worked was yes and no) or not applicable to each question while using the application.  From there, I summed the yes, no, and yes/no answers, normalized the results across the thirteen heuristics and calculated the overall usability score.

From look and feel as well as navigation perspective, the application relied upon a combination of Android specific platform usability interface features, icons and norms as well as some of the mobile application’s specific usability.   Common Android specific icons were the back icon/button (back arrow), menu icon/button (three parallel horizontal lines), edit icon/button (pencil), search icon (magnifying glass), notification/alarm icon/button (bell with number highlighted in red), actions available for a particular screen/menu item icon (3 parallel horizontal dots) and sync application icon/button (two curved arrows juxtaposed over each other in a clockwise direction).  The mobile application specific items dealt with menu item choices and ways to enter data into the application as well as some iconology for their main menu items (vendor specific logos).

Overall the mobile fitness application had a 64% favorable usability score as per the checklist (combination of yes answers to the checklist).  The heuristic areas of help and documentation, visibility of system status, error recovery and prevention and recognition rather than recall were the usability areas in which the application was the weakest (largest number of no and yes/no combination of answers).  Aesthetic and minimalistic design was the next area of usability weakness (highest yes/no combination of answers).  Most egregiously, the mobile fitness application lacked sufficient error prevention and handling, limited the user’s ability to get help and/or personalize the screens, required some memory recall versus recognition by the user for use and limited the user’s ability to leave the application or log out.  Additional usability issues identified were with the complex multi-level menus and some inconsistency with information and menu presentation and data entry procedures.

Where I seriously downgraded my scoring on the application was inaccessibility of certain user actions such as logging off, privacy setting and access to help.  As an example, the only way someone could logoff the application was to navigate to the main menu, to the gear (settings) icon and scroll down to the logoff menu choice.  This was also the only way to review and/or change any sharing or privacy settings.  The same series of navigation steps was needed (main menu, scroll to question mark icon) as well as retrieve any help or documentation on the mobile application.

Some concluding thoughts

The Gomez et al checklist is a comprehensive, exhaustive usability heuristic test that does a very good job of identifying usability issues that are in a mobile application.  It is also a fairly simple test for a non-expert user or evaluator to use for evaluating mobile applications. It took me between 2-3 hours to run through the checklist; with at least 50% of my time spent recreating the checklist in a spreadsheet to aid in calculating the summary statistics and capturing screen shots for my paper results.  I did not weigh any one heuristic more than another, and by normalizing my results, they were treated as equal in importance. I only tested one application on one mobile operating system platform.  Some of the heuristic questions are subjective in nature with regard to how the user perceives the application interface; I’m an engineer who is comfortable with most computing interfaces, so that can introduce bias into my score.  Nevertheless, the heuristic principles that I found the mobile application to be deficient in had fairly straightforward usability questions.

It is a bit disappointing to me that the application did not receive a higher score than 64% especially since it was one of the most popular mobile fitness applications.  If I had lumped the yes/no results into the yes results, the score would have been higher, however, with a yes/no result there is still some usability issue. In addition, the serious deficiencies with providing easy access to help and documentation, and difficulty with logging off and accessing/changing sharing and privacy showed to me a lack of respect to the user.  However, my concern may not be the same as those held by a younger cohort.

For mobile application developers and vendors, use of this fairly comprehensive usability heuristic checklist might help them self-assess any usability issues that they can address in the development and maintenance of their products.  For mobile application users, the checklist may be a bit onerous, however, if a clinician or healthcare organization is recommending an application, assessing any recommended mobile applications using this usability heuristic checklist may highlight any possible usability issues their users may encounter and/or give a way to rank the usability of their recommended applications.  The checklist statistics can be tweaked to weight certain heuristics more than others as well as compare the free and premium versions of an application. Moreover, comparison of the usability between the different operating system platforms may be interesting.  In the end, the usability of the mobile application can be a major determining factor in whether or not there is persistent and consistent use of the application by a patient or user. It is well known that compliance to the protocols to manage one’s chronic condition can determine the outcome and usability is a large contributor to that compliance. Ensuring a mobile application adheres to common usability principles can possibly increase the odds of compliance.