Valuing Usability Programs
by Kristoffer Bohmann, July 14, 2000
Practical tools to cost-benefit analyze usability programs.
Assume that a website has 10.000 users per month each performing 10 tasks in the period. What will be the economic effect of a usability study that promises 40 seconds faster user task performance? One way to make the calculation is shown below.
The first step is to calculate the saved user hours per year as Number of users * Tasks per user * Estimated time savings per task. Next, the economic effect of the time savings is found using this formula: Saved user hours * User value per hour. Finally, economic impact of usability evaluation: Probability of implementing the intended design improvement is assumed to be 30 percent better when conducting usability evaluation before designing. So, economic impact of usability evaluation equals: Total Savings Per Year * Improved results due to usability evaluation.
Table 1: Valuing Usability Program Example
| Users per year (10.000 per month) |
120.000 per year |
| Tasks per user (10 per month) |
120 per year |
| Estimated time savings per task (40 seconds) |
0,0111 hours |
| Saved user hours per year |
160.000 |
|---|
| Estimated user value per hour |
$20 |
| Total Savings Per Year |
$3.200.000 |
|---|
| P(successful design|usability evaluation) |
90% |
| P(success design|no usability evaluation) |
60% |
| Probability of better results due to usability evaluation |
30% |
| Economic Impact of Usability Evaluation |
$960.000 |
|---|
Source: Calculation technique inspired by comment made by Jakob Nielsen.
Information Needed
To make the calculation information is needed about:
- Number of users per period of time: This number may be available directly from a server log (e.g., 10.000 unique visitors per month). If this number does not apply, the number of users needs to be estimated.
- Tasks per user per period of time: Estimate of how many tasks each user or each user within a user category makes per month, year, etc. (e.g., 10 tasks per user on a monthly basis).
- Estimated time savings per task: How much time can be saved through design improvements (e.g., 30 seconds)? Or, the calculation can be reversed by asking how much time needs to be saved to gain a particular economic benefit (e.g., $100.000). This requires an estimation of the user value per hour. An arbitrary number can be set (e.g., $20). Or, the number can be estimated using a more systematic procedure. One such procedure is to base the estimate on marketing costs per user. With a $200.000 Web marketing budget and 250 online orders, $800 are spent per order and $20 per user. To my knowledge, this method is the best available. The formula used to estimate user value per hour:
- User value per hour: How much is a user worth to the site per hour? Different values for different user categories (e.g., professional purchasers, engineers, and managers) can be used to refine the calculation.
- Time savings per user task and value per user hour should be estimated with great care. For instance, if the value per user hour was only $10 in the example, the total effect would also decreases by 50% to $1.600.000, and vice versa. Some time savings will not even have an economic impact mainly because they are competitively irrelevant.
Assumptions
The calculation is based on several assumptions. First, the most important assumption is that the user task is the primary level of analysis in usability studies, while page views are secondary. Effective usability efforts strive to improve overall user tasks often spanning several pages, while single Web pages only are improved to the extent they affect task performance.
Next, the user task is an aggregated level often containing several design elements and Web pages - this is especially so in interaction design. The basic unit of analysis is the design component such as a headline, URL, or graphical item. Tasks are improved through more appropriate use of design components.
Finally, the distinction between level and unit of analysis is important to carry out purposeful usability efforts that focus on critical dimensions in the user interface. So, this approach has a fundamental assumption about user behavior: Users use websites to solve tasks, not to see single pages or design components.
![]() |
About the Author
Kristoffer Bohmann (biography) M.Sc. thinks and writes about high-quality user experiences. His philosophy: Users first. You can contact him at kristoffer@bohmann.dk. |
More articles...
Articles