- Formative (exploratory) versus summative (assessment)
- Qualitative versus quantitative
- Longitudinal studies versus single session
- Comparative studies versus single product under testing
These types are not mutually exclusive: For example, a comparative study can also be a formative study and a longitudinal study can also be a qualitative study.
The difference between formative and summative studies is determined by the development stage of the interface to be tested and the goal of the test. Formative, or exploratory, studies are run on early versions of the product when it is still being designed with the aim of understanding user behavior, needs, and wants, and how people expect to interact with the product while summative, or assessment, studies are run on half-completed to completed products, with the aim of validating the improvements made as a result of previous formative tests, validating individual features or determining a baseline usability. Formative studies are run to shape the product while summative studies are run to assess its usability. Jeff Sauro describes this as follows:
"Summative tells you how usable an interface is and formative tells you what isn't usable."
– Jeff Sauro (https://measuringu.com/formative-summative/)
Both types are compatible with remote methodologies.
Qualitative and quantitative types of study differ in the type of data that they gather. Qualitative usability studies are focused on gaining in-depth understanding based on narrative data, while quantitative studies collect numerical data in order to produce statistically relevant metrics. A qualitative study will uncover usability issues and identify why users stumble there, whereas quantitative studies will determine the task completion rate, the time-on-task, the users' satisfaction, or other relevant metrics.
A small number of participants is sufficient to provide valuable results in qualitative studies, whereas quantitative studies rely on large numbers of participants in order to provide statistically relevant metrics. How many participants are actually required for a quantitative study is determined by the study parameters, whereas qualitative studies can be run with as few as five participants, according to Nielsen (https://www.nngroup.com/articles/how-many-test-users/).
Both study types are compatible with remote methodologies.
In a single-session study, the participant is only required to participate once in order to complete the study, while a longitudinal study is run over a longer period of time and includes repeated contact with the participants during this time. This time period can be anything from a couple of days to weeks. The study participants are asked to use a product over that period of time. If the aim of the longitudinal study is that the participants record their behavior around and with the interface, this type of study is called a diary study.
Both longitudinal and single-session studies are compatible with remote methodologies.
These studies differ in the number of interfaces that are being tested. Single test object studies will cover only a single product, while comparative studies involve two or more products in order to compare them. Comparative studies may be run in order to determine which alternative of a product’s early design better reflects the user's expectations. The aim of this type of study is to either find a preferred candidate or identify the best aspects of either alternative which can then be used to drive the further design of the product. Comparative studies can also be used to determine how competing products measure up against each other.
Another aspect of comparative studies is whether all study participants will test all test objects (within-subjects) or whether each study participant only tests one of the test objects (between-subjects). Both have their advantages and disadvantages; fewer participants are required for a within-subject study, but the study duration is necessarily longer, while a between-subjects study eliminates anchoring bias (judging subsequent test objects by comparing them to the subjective baseline made after using the first one).
Comparative studies and studies of a single test object are compatible with remote methodologies.
There are other, more specific, study types. One such example is the end-to-end, multichannel customer experience study. A remote unmoderated survey approach works well in gathering data on an end-to-end customer experience that includes a digital and a physical aspect; one example of an activity that includes such aspects could be ordering pizza from a fast food chain on a mobile app and then picking it up at the restaurant. The study can evaluate the ease of installing and registering on the app; ordering food, paying, and selecting the pick-up location; the quality of the messages received apropos pickup time; and the order completion progress, if available. For the actual physical pickup, the study can gather feedback on the punctuality of the order, whether it was clear where to go to pick it up in the restaurant, what information was needed to verify the identity of the orderer, and whether it was fulfilled correctly.