Book Image

Remote Usability Testing

By : Inge De Bleecker, Rebecca Okoroji
Book Image

Remote Usability Testing

By: Inge De Bleecker, Rebecca Okoroji

Overview of this book

Usability testing is a subdiscipline of User Experience. Its goal is to ensure that a given product is easy to use and the user's experience with the product is intuitive and satisfying. Usability studies are conducted with study participants who are representative of the target users to gather feedback on a user interface. The feedback is then used to refine and improve the user interface. Remote studies involve fewer logistics, allow participation regardless of location and are quicker and cheaper to execute compared to in person studies, while delivering valuable insights. The users are not inhibited by being in a new environment under observation; they can act naturally in their familiar environment. Remote unmoderated studies additionally have the advantage of being independent of time zones. This book will teach you how to conduct qualitative remote usability studies, in particular remote moderated and unmoderated studies. Each chapter provides actionable tips on how to use each methodology and how to compensate for the specific nature of each methodology. The book also provides material to help with planning and executing each study type.
Table of Contents (18 chapters)
Title Page
Copyright and Credits
Packt Upsell
What to Consider When Analyzing and Presenting the Study Results

Types of usability studies

In addition to the usability testing methodologies, there are also different types of studies that can be executed using the various methodologies:

  • Formative (exploratory) versus summative (assessment)
  • Qualitative versus quantitative
  • Longitudinal studies versus single session
  • Comparative studies versus single product under testing

These types are not mutually exclusive: For example, a comparative study can also be a formative study and a longitudinal study can also be a qualitative study.

Formative and summative usability studies

The difference between formative and summative studies is determined by the development stage of the interface to be tested and the goal of the test. Formative, or exploratory, studies are run on early versions of the product when it is still being designed with the aim of understanding user behavior, needs, and wants, and how people expect to interact with the product while summative, or assessment, studies are run on half-completed to completed products, with the aim of validating the improvements made as a result of previous formative tests, validating individual features or determining a baseline usability. Formative studies are run to shape the product while summative studies are run to assess its usability. Jeff Sauro describes this as follows:

"Summative tells you how usable an interface is and formative tells you what isn't usable."

Both types are compatible with remote methodologies.

Qualitative and quantitative usability studies

Qualitative and quantitative types of study differ in the type of data that they gather. Qualitative usability studies are focused on gaining in-depth understanding based on narrative data, while quantitative studies collect numerical data in order to produce statistically relevant metrics. A qualitative study will uncover usability issues and identify why users stumble there, whereas quantitative studies will determine the task completion rate, the time-on-task, the users' satisfaction, or other relevant metrics.



A small number of participants is sufficient to provide valuable results in qualitative studies, whereas quantitative studies rely on large numbers of participants in order to provide statistically relevant metrics. How many participants are actually required for a quantitative study is determined by the study parameters, whereas qualitative studies can be run with as few as five participants, according to Nielsen (

Both study types are compatible with remote methodologies.

Longitudinal and single-session studies

In a single-session study, the participant is only required to participate once in order to complete the study, while a longitudinal study is run over a longer period of time and includes repeated contact with the participants during this time. This time period can be anything from a couple of days to weeks. The study participants are asked to use a product over that period of time. If the aim of the longitudinal study is that the participants record their behavior around and with the interface, this type of study is called a diary study.

Both longitudinal and single-session studies are compatible with remote methodologies.

Comparative and single test object studies

These studies differ in the number of interfaces that are being tested. Single test object studies will cover only a single product, while comparative studies involve two or more products in order to compare them. Comparative studies may be run in order to determine which alternative of a product’s early design better reflects the user's expectations. The aim of this type of study is to either find a preferred candidate or identify the best aspects of either alternative which can then be used to drive the further design of the product. Comparative studies can also be used to determine how competing products measure up against each other.

Another aspect of comparative studies is whether all study participants will test all test objects (within-subjects) or whether each study participant only tests one of the test objects (between-subjects). Both have their advantages and disadvantages; fewer participants are required for a within-subject study, but the study duration is necessarily longer, while a between-subjects study eliminates anchoring bias (judging subsequent test objects by comparing them to the subjective baseline made after using the first one). 

Comparative studies and studies of a single test object are compatible with remote methodologies.



Other study types

There are other, more specific, study types. One such example is the end-to-end, multichannel customer experience study. A remote unmoderated survey approach works well in gathering data on an end-to-end customer experience that includes a digital and a physical aspect; one example of an activity that includes such aspects could be ordering pizza from a fast food chain on a mobile app and then picking it up at the restaurant. The study can evaluate the ease of installing and registering on the app; ordering food, paying, and selecting the pick-up location; the quality of the messages received apropos pickup time; and the order completion progress, if available. For the actual physical pickup, the study can gather feedback on the punctuality of the order, whether it was clear where to go to pick it up in the restaurant, what information was needed to verify the identity of the orderer, and whether it was fulfilled correctly.