Consolidating usability problems with novice evaluators

Effie Lai Chong Law, Ebba Thora Hvannberg

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

12 Citations (Scopus)


The process of consolidating usability problems (UPs) is an integral part of usability evaluation involving multiple users/analysts. However, little is known about the mechanism of this process and its effects on evaluation outcomes, which presumably influence how developers redesign the system of interest. We conducted an exploratory research study with ten novice evaluators to examine how they performed when merging UPs in the individual and collaborative setting and how they drew consensus. Our findings indicate that collaborative merging causes the absolute number of UPs to deflate, and concomitantly the frequency of certain UP types as well as their severity ratings to inflate excessively. It can be attributed to the susceptibility of novice evaluators to persuasion in a negotiation setting, and thus they tended to aggregate UPs leniently. Such distorted UP attributes may mislead the prioritization of UPs for fixing and thus result in ineffective system redesign.

Original languageEnglish
Title of host publicationNordiCHI 2008
Subtitle of host publicationBuilding Bridges - Proceedings of the 5th Nordic Conference on Human-Computer Interaction
Number of pages4
Publication statusPublished - 2008
EventNordiCHI 2008: Building Bridges - 5th Nordic Conference on Human-Computer Interaction - Lund, Sweden
Duration: 20 Oct 200822 Oct 2008

Publication series

NameACM International Conference Proceeding Series


ConferenceNordiCHI 2008: Building Bridges - 5th Nordic Conference on Human-Computer Interaction

Other keywords

  • Confidence
  • Consensus building
  • Downstream utility
  • Evaluator effect
  • Filtering
  • Merging
  • Severity
  • Usability problems


Dive into the research topics of 'Consolidating usability problems with novice evaluators'. Together they form a unique fingerprint.

Cite this