Evaluating Online Help Systems


Evaluation of an online help system should be done during and after the prototype stage:
  • When evaluating during the prototype stage, it is best to at least include one example of each topic type, with all relevant navigation aids, links between topic types, and other design components. Make sure the software developers play an active role in this to ensure that your ideas are cohesive with their plans for coding the links between the application and the help system. A benefit of doing this is that it allows you to see things that don't work, and allows you to fix it before further work is done to the help system. The prototype can also be shown to users or customers to obtain feedback about topic types, navigation methods, linking patterns and planned level of detail. Take into consideration all of their comments.
  • After the online help system is tested and released, it is best to once again involve users in the evaluation of the help. This can be done by providing an e-mail link in each topic that will encourage them to provide feedback.

Questions to ask about each topic when evaluating the help:
  • Does the topic contain all of the information you need at that point? (Sometimes topics contain too much information.)
  • If a topic does not contain all of the information you need, does it include links to other topics that fill in the gaps?
  • Can you tell how this topic fits in with what you want to do? Does it matter? (In some cases, it doesn't matter; in other cases, it matters very much.)
  • If a topic tells you how to do something, do you know where to find that function in the application?
  • Does the topic tell you about anything you need to do first, before you do this step?


Evaluation Strategies

Evaluate every topic for navigation and context. The following tables were developed by Whitney Quesenbery, an author of a chapter in Content and Complexity, “Dimensions of Usability,” .

Usability

The definition of usability in the ISO 9241 standard is: "The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use."

Whitney Quesenbery advocates looking the usability requirements for different aspects of the user experience. For each of the five dimensions of usability (the 5Es), we think about how it is reflected in requirements for each of the user groups. The 5Es are:
  • Effective
  • Efficient
  • Engaging
  • Error Tolerant
  • Easy to Learn

The evaluation criterion is organized into these five characteristics.
Characteristics
Definition
Effective
  1. The help label / icon is clear and prominent from all screens.
  2. Help is available for all screens for which users may need assistance or a more detailed explanation.
  3. Help is not provided for screens that are self-explanatory.
  4. Clear directions for exiting the help are available ("Close this window" button, or "Close X").
  5. Both help and the application window can be viewed simultaneously.
  6. The help window can be resized.
  7. The focus is on user tasks.
Efficient
  1. Identification of, and navigation to, the required topic is easy (context-sensitive help / clear TOC / support for keyword search)
  2. Navigation from one topic to other topics is available (mostly through the TOC, breadcrumb trails, and "see also" links).
  3. The help pages indicate where you are in the help system (through breadcrumb trails or highlighting the current topic in the TOC).
  4. The direction of navigation (to the next task / next topic / next level) is clearly indicated.
  5. A keyword index is available.
Engaging
  1. Layout is clear and aesthetically pleasing.
  2. The help system is visually appealing and motivating to use.
  3. Intuitive navigation is supported.
  4. Graphics and multimedia elements are provided (but only as required).
  5. The help system can be somewhat customized.
Error-tolerant
  1. The help system displays warnings / errors in usage (for example, possible keyword spelling corrections in keyword search).
  2. The user is prompted to go to the next logical step / level.
  3. A troubleshooting help system is available.
Easy to learn
  1. The help system includes instructions on its use.
  2. The help system layout, theme, and icon usage are consistent with the application.
  3. Help is divided into levels according to user levels.
  4. Additional or background information is provided through links.
  5. The user is motivated to learn the help system and use it often.
If you answer "Yes" for more than 20 of these points, the help system usability is high. However, if the score falls below 10, then consider further assessment of the help system.

Content

Good content should always meet the six criteria of communication:
  • Complete
  • Clear
  • Correct
  • Concise
  • Contextual
  • Consistent

The content evaluation criteria are organized divided into these six characteristics.
Characteristics
Definition
Clear
  1. The help system covers all functions and features, with illustrations and examples as required.
  2. All routine tasks / procedures are described step-by-step.
  3. Reasons are provided for a particular step, format, or restriction.
  4. The help includes a glossary of terminology.
  5. Background information / domain notes / usage guideline / best practices are provided.
  6. The help provides common workflows.
  7. The help indicates navigation to the next task (through procedure sequence or "see also" links).
  8. Instructions about how to use the help are included.
  9. The help includes basic troubleshooting information for the application.
  10. The help provides contact information for further information (such as help desk number or support site link).
Complete
  1. The help provides unambiguous in instructions and descriptions.
  2. Plain language is used.
  3. The help uses short sentences.
  4. The help avoids unnecessary wordiness.
  5. The language used is suitable to the audience.
Correct
  1. The information provided is factually correct for the objective and purpose of the application.
  2. Platform support, available features, memory usage, and the like are accurate for the referred version / release / module / part number.
  3. The help menu structure is logically aligned to the typical workflow / procedure sequence.
  4. Spelling, punctuation and grammar are correct.
  5. Language and structure are sensitive towards gender and culture.
  6. The content complies with required industry standards.
Concise
  1. Content is short and precise, with each help topic preferably limited to one non-scrolling page.
  2. Long procedures are broken down to smaller sub-procedures.
  3. Descriptive lead-ins are typically limited to one small paragraph.
  4. Where appropriate, bulleted lists, numbered lists, tables and graphics are substituted for lengthy descriptive text.
  5. Information is layered - basic information is immediately available with optional links to additional information.
Contextual
  1. Each help topic has an appropriate title describing its content.
  2. The context of a particular task / procedure is explained, with a specific reason (if applicable).
  3. Examples / cases / demonstrations are included.
  4. Help is divided into levels according to user experience levels.
Consistent
  1. The terminology and word usage (including action verbs) is consistent across the entire help system.
  2. The terminology, menu options, field labels, and action button labels are consistent with the application.
  3. The terminology, word usage (including action verbs), and usage of the help system are consistent with other applications from the same suite / group of applications.
  4. The terminology, word usage (including action verbs), and usage of the help system are consistent with internal company guidelines.
  5. The terminology, word usage, and usage of the help system are consistent with industry standards.
If you answer "Yes" for more than 30 of these points, your content exceeds expectations. However, if the Yes answers are fewer than 20, then you should perform a thorough assessment of the help and make improvements to it.

Sources
Weber, Jean Hollis. Is the Help Helpful?: How to Create Online Help That Meets Your Users' Needs. Whitefish Bay: Hentzenwerke Publishing, 2004. Print.

Quesenbery, Whitney, "Using the 5Es to Understand Users." wqusability, Whitney Interactive Design, n.d. Web. Oct 11 2012. <www.wqusability.com/articles/getting-started.html>

Dalvi, Meghashri. Evaluating Online Help. STC UUX Community Newsletter. Vol 13, No. 3. Web. Oct 11 2012.

...

Original Author: Sandra Ramirez
Contributors: Luis F. Regalado, Inez Funchess, Linda Page
Editors: Larita Clow, Wendy Anderson, Tammy Fitzpatrick