Calculating Usability Effects
by Kristoffer Bohmann, July 6, 2000
Time to complete a task and number of errors per task are key metrics when evaluating usability effects.
Making user tasks the level of analysis in usability evaluations is based on two assumptions. First, users are goal-oriented and focus their attention on content of relevance to the task. They want to be productive and complete tasks as opposed to using systems without a clear purpose. Second, users regard content with low relevance to the task as sources of disturbance that potentially limit performance.
This insight has lead to the emergence of a simplistic approach to Web design (Nielsen, 1999). Simplistic design argues that fewer design elements improve user performance through user interfaces optimized for fast and error-free performance.
Few Metrics in Usability
Several usability evaluation methods have emerged to identify and analyze design elements that inhibit user performance, including heuristic evaluation (Molich and Nielsen, 1990; Nielsen, 1992), cognitive walkthrough (Wharton and others, 1992; Lewis 1997), and lab studies (Fowler 1994). These methods are often used to collect observational data or interview data, while others use surveys. Very few are based on simple, quantitative metrics.
Conducting usability studies has three requirements:
- Formulating tasks that are critical to mainstream users. For instance, one to three tasks may be formulated and tested to direct design activities.
- Identifying mainstream users that represent the average user.
- Choosing a usability evaluation method.
Metrics
My purpose in this article is to develop metrics for quantitative usability evaluations. These metrics make it possible to calculate the usability effect of redesign efforts. The metrics are:
- Time to complete a task or set of tasks
- Number of errors per task.
An example is used to show how the metrics may work. In the example, a usability test has been carried out before and after a website redesign.
Overall, the redesign lead to a 62.5% improvement in overall usability as shown in table 1. Before the redesign test users spent 300 seconds to complete a critical task and made on average four tasks errors (see table 1). After the redesign test users carried out the same task in 150 seconds with one error. This equals a 100% improvement in task time and a 400% task error improvement.
Table 1: Results from Usability Evaluation
|
Task Time |
Task Errors |
Usability Index |
| Before redesign |
300 |
4 |
100 |
| After redesign |
150 |
1 |
37.5 |
| Improvement |
100% |
400% |
62.5% |
|---|
Notes: 1. Task time is measured in seconds. Task errors is average number of errors.
2. Calculation for usability index: ( (300-150)/300 + (4-1)/4 ) / 2 = 62.5%
Usability improvements in table 1 have three possible sources:
- Faster task time, i.e. three alternative ways to complete a task in an interface are reduced to one. Fewer choices focus users on doing, not thinking about alternatives.
- Fewer errors, user difficulties finding product prices are improved through refined search results.
- Combination
Metrics Defined
Task time is time to complete a task or set of tasks, measured in seconds. Task errors are things that make it difficult for users to complete their tasks. Errors are identified through user observation and measured as number of errors. Errors may be rated by their severity or user-perceived importance to prioritize redesign efforts.
Both task time and error add information to usability studies.
Errors are often the direct reason why task time is slowed down (Landauer, 1995, p. 287) and why users choose not to become regular users of a Web service.
In competitive environments error-free performance is not always enough. In particular, time is frequently mentioned as a critical factor for Web services. Speed improve convenience of use and may make more people adopt Web services.
Not Opinions Or Attitudes
What is measured is user performance, not user opinions or attitudes (e.g., satisfaction). The metrics both focus on what users do in the process (task error) and the output of their efforts (task time). Further, studies of user tasks may provide additional knowledge about what users expect from the interface.
The results are usable for designing in several ways:
- New initiatives can only be justified if speed or accuracy is improved.
- Easy to communicate to developers, designers, and managers.
- Easy to measure once key tasks have been identified and formulated.
![]() |
About the Author
Kristoffer Bohmann (biography) M.Sc. thinks and writes about high-quality user experiences. His philosophy: Users first. You can contact him at kristoffer@bohmann.dk. |
More articles...
Articles