Book Image

Handbook of Usability Testing - Second Edition

By : Jeffrey Rubin, Dana Chisnell
Book Image

Handbook of Usability Testing - Second Edition

By: Jeffrey Rubin, Dana Chisnell

Overview of this book

Whether it's software, technical documentation, a cell phone, or a refrigerator, your customer expects your product to be easy to use. Completely updated with current industry best practices and more varied examples, it can give you that all-important marketplace advantage: products that perform the way users expect. The first part of the book introduces you to the overview of testing which covers the definition of the key terms and presents an expanded discussion of user-centered design and other usability techniques. As you progress, you’ll learn the various steps involved in performing a test. The later chapters explore the advanced testing methods and discuss how to extend one’s influence of the whole product development strategy. By the end of this book, you’ll have the exact knowledge to understand and improve the usability of your product.
Table of Contents (6 chapters)

Chapter 4

Skills for Test Moderators

The role of the test moderator or test administrator is the most critical of all the test team members, presuming that you even have the luxury of a test team. In fact, the moderator is the one team member that you absolutely must have in order to conduct the test. The moderator is ultimately responsible for all preparations including test materials, participant arrangements, and coordination of the efforts of other members of the test team.

During the test, the moderator is responsible for all aspects of administration, including greeting the participant, collecting data, assisting and probing, and debriefing the participant. After the test, he or she needs to collate the day's data collection, meet with and debrief other team members, and ensure that the testing is tracking with the test objectives. If the usability test were an athletic contest, the moderator would be the captain of the team. As such, he or she has the potential to make or break the test. An ineffective moderator can seriously negate test results and even waste much of the preliminary preparation work. This chapter discusses several alternatives for acquiring test moderators from inside and outside your organization, as well as the desired characteristics of an effective test moderator. Chapter 9.1 includes guidelines for moderating test sessions, including information about when and how to intervene, and the advantages and disadvantages of using a “think-aloud” protocol.

Who Should Moderate?

One of the basic tenets of usability testing—and of this book—is that it is almost impossible to remain objective when conducting a usability test of your own product. There is simply too strong a tendency to lead participants in a direction that you want the results to go, rather than acting as a neutral enabler of the process. This is even true for experienced test moderators who conduct the test from an external control room. In fact, asking someone to test his or her own product is like asking parents to objectively evaluate the abilities of their child. It is an impossible endeavor.

Having said that, if there is only you available to test your product, do so. In almost every case, it is still better to test than not to test, even if you must do the testing yourself. However, for the long term, you would want to be out of the self-testing business as soon as possible.

Imagine that you want to conduct a test on a product for which you have primary responsibility, and if possible you would like someone less involved with the product to conduct the test. You can help develop the test materials, make arrangements, and select participants, but you need a more objective person to handle the actual test moderating. Suppose also that your organization currently has no in-house testing staff and does not plan to introduce one shortly. To whom should you look for help?

The following sources represent a number of areas from which you can find candidates who possess the requisite skills to conduct a test, or who could head up the beginnings of an internal testing group. They may or may not already be working on your product.

Human Factors Specialist

A human factors specialist is the most likely candidate to conduct a usability test. This type of person typically has an advanced degree in psychology, industrial engineering, or similar discipline, and is familiar with experimental methodology and test rigor. Just as important, the human factors specialist is grounded in the basics of information processing, cognitive psychology, and other disciplines related to the development of usable products, systems, and support materials. This grounding is crucial in differentiating the important from the superficial usability factors in a product and ultimately in designing and conducting the test.

With the current focus on usability engineering and testing, it is highly probable that human factors specialists within your organization are already involved with testing in one form or another.

Marketing Specialist

A marketing specialist is typically customer-oriented, user-oriented, or both, has good interpersonal and communication skills, and would be very interested in improving the quality of products. This type of specialist may already be involved with your product, but usually not to the detailed level that would tend to disqualify him or her from conducting the testing.

Technical Communicator

Technical communicators, including technical writers and training specialists, often make wonderful test moderators. Many technical communicators already serve as user advocates on projects, and their profession requires them to think as a user in order to design, write, and present effective support materials.

Rotating Team Members

Let's suppose that no one from the disciplines listed previously is available to help on your project, and you are still determined not to test your own materials. Another alternative is to draw upon colleagues of similar disciplines, who are not working on the same product. An example of this approach is for technical communicators to test each other's manuals or for software engineers to test each other's program modules.

In such a scenario, the person whose product is being tested could help prepare many of the test materials and make the pretest arrangements, then turn over the actual moderating of the test to a colleague. One of the advantages of this approach is that two (or more) heads are better than one, and it is always beneficial to have someone other than yourself help prepare the test. The person acting as the test moderator would need time to become familiar with the specific product being tested and to prepare to test it in addition to the time required to actually moderator the test.

Should you decide to implement this approach, you must plan ahead in order to build the test into your mutual schedules. You cannot expect your colleague to drop everything he or she is working on to help you. Of course, you would reciprocate and serve as test moderator for your colleague's product.

External Consultant

Another option is to hire an external consultant. Many human factors, industrial design, market research, and usability engineering firms now offer usability testing as one of their services, including the use of their test laboratories. You may simply want to outsource the usability test to such a firm, or use such a firm to “kick off” a testing program in your organization.

Using an external consulting company guarantees the objectivity that testing requires. Even some organizations that employ internal human factors specialists to work on the design and development of products still outsource the testing work for the greater sense of impartiality it provides.

If you know your organization is committed to eventually forming a long-term testing program on site, then seek out a consulting company that will work with you to transfer the knowledge of testing into your organization. Even if you are unsure about the long-term prospects for testing in your company, it still might be easier to have outside help with an initial test. Just make sure that if you conduct the test off-site, its location is physically close enough to allow development team members to attend the test sessions. Do not simply farm out the test to a remote location. (Although, in a pinch, team members could observe tests from their remote locations via Morae, Camtasia, or other electronic monitoring tools.) Viewing tests in person is much more effective than watching or listening to a recording, especially for those who are skeptical about the value of testing.

Characteristics of a Good Test Moderator

Regardless of who conducts the test, either yourself or internal or external staff, and regardless of the background of that person, there are several key characteristics that the most effective test moderators share. These key characteristics are listed and described in the paragraphs that follow. If you are personally considering taking on the role of test moderator in your organization, use these key characteristics as a checklist of the skills you need to acquire. If you are considering using either an internal person or hiring an external person to perform this role, use these key characteristics to help evaluate the person's capabilities.

Grounding in the Basics of User-Centered Design

Grounding in the basics of human information processing, cognitive psychology, and user-centered design (essentially the domain of the human factors specialist) helps immensely because it enables the test moderator to sense, even before the test begins, which interactions, operations, messages, or instructions are liable to cause problems. Test moderators with this background have a knowledge of which problems can be generalized to the population at large and which are more trivial. This helps to ascertain when to probe further and what issues need to be explored thoroughly during the debriefing session. Additionally, this background can also prevent the need to test situations that are known to cause problems for users, such as the inappropriate use of color or the incorrect placing of a note in a manual. Lastly, a strong background in usability engineering helps the test moderator to focus on fixing the important issues after a test is complete.

Quick Learner

An effective test moderator need not be expert in the intricacies of the specific product being tested. For example, if the product is a database management system, the moderator need not be an expert in database management. However, he or she must be able to absorb new concepts quickly and to integrate these concepts into his or her thinking and vocabulary. The moderator also needs to absorb all the peripheral issues surrounding a product, such as its positioning in the marketplace, competitors, and historical problems. During the test itself, the moderator must be able to understand the actions and comments of the participant quickly, as well as the implications behind those actions and comments. Being a quick learner enables the moderator to probe and question effectively.

Instant Rapport with Participants

Bringing in participants to evaluate your product is an auspicious and very opportune point in the development cycle that should not be squandered. If for some reason a participant is not at ease and is not able to function as he or she normally would, it represents a lost opportunity and potentially misleading results. If you are able to test only five participants, one uneasy participant represents a potential loss of 20 percent of your test data. The test moderator's ability to quickly size up each participant's personality, make friends, and put the person at ease is essential to getting the most from the testing process. Some participants need coddling, some need stroking, and some are businesslike and require a more formal approach. Whichever the case, the test moderator must make each person feel comfortable and secure.

Excellent Memory

Some might believe that because usability test sessions are recorded, the test moderator need not rely on memory for conducting and evaluating a test session. Actually, memory is called into play well before a test session has ended. Because a test session can be rather long, the test moderator needs to remember behaviors or comments that took place earlier in the session in order to cross-check and probe those behaviors later in the session. For example, a participant may attempt to perform the same task in two or three different ways, and the test moderator may want to probe to understand why the participant performed the task differently each time.

Memory is also required to recall the results of a test session after its completion. Because there is often very little time to devote to searching the videotapes after a test, except as insurance against missing some point entirely, the test moderator often must rely heavily on memory and notes.

Good Listener

Listening skills involve the test moderator's ability to hear with “new ears” during each session and to lay aside personal biases and strong opinions about what he or she is seeing and hearing. The test moderator needs to understand both the content and the implication of a participant's comments, as there are often mixed messages of all kinds during testing. The test moderator must pick up on the subtle nuances of speech and emphasis, as a participant's comments are often indirect and less than forthcoming. It is so important to understand the rationale behind the participant's behavior, because the rationale often signals whether a change in the product is required or not.

Comfortable with Ambiguity

Usability is not a precise science consisting of formulas and black and white answers. Even if a usability test is conducted under the most rigorous conditions, which is atypical, you are still not assured that all of the results are valid and generalizable to your entire user population. Instead, usability testing can often be an imprecise, ambiguous enterprise, with varying and sometimes conflicting observations, not surprising for any venture that has human beings as its focus. A test moderator, then, must understand and be comfortable with ambiguity.

For example, prior to testing you may think that there are only two ways to perform a particular task. During testing though, you discover that the participants have found four other ways to perform the same task. Or, you discover that you are no closer to a clear-cut resolution of a product's problems after a week of testing than you were before you began. Or, when testing multiple versions of a product, no clear winner emerges. The versions are all equally bad or, if you are lucky, equally good. These situations require patience, perseverance, and very often skill at negotiation. Without tolerance for ambiguity and the patience to persevere, the test moderator tends to rationalize and to blame the participants for making unplanned choices during the test.


Another related characteristic of an effective test moderator is flexibility, which has to do with knowing when to deviate from the test plan (we discuss this further in Chapter 9.1). There are times when a particular participant does not have the expected skills or simply views the task in a completely different way than was originally intended. Jeff conducted a test when the entire high-level design of an interface became questionable after testing only two participants. He could see immediately that the premise for the design was flawed. At that time, he recommended that the company halt testing and go back to the drawing board. To continue ferreting out minor problems with the product would have been a waste of everyone's time. While this is an extreme case, the point is that one needs to be prepared for the unexpected, even if that has serious consequences.

Long Attention Span

Experienced test moderators share a secret: Usability testing can be tedious and boring. There are long stretches when seemingly nothing is happening, when participants are reading and absorbing, thinking, and sometimes just resting. The moderator cannot possess the type of personality that needs new stimulation every 5 to 10 minutes. The moderator must be able to pay attention for long periods of time because there is no predicting when a gem of a discovery will arise during a test session. In addition, because the moderator may view up to 10, 15, or 20 sessions, all of which involve observing the same or similar tasks, the ability to stay focused is extremely vital.

Empathic “People Person”

Participants will relate more readily to a test moderator who is an empathic individual. This may not be all that critical during the test session itself, especially if the session requires little probing or exploration on the part of the test moderator. However, empathy can play a major part during the debriefing session when the test moderator is trying to elicit a participant's innermost thoughts and feelings about the previous two hours of work. Participants will tend to hold back if they feel that the test moderator cannot relate to their particular situation, this being especially true if the session was unusually frustrating or difficult.

“Big Picture” Thinker

There is so much data collected during a usability test, and there is so much data that could be collected during a test that it is very easy to lose sight of the forest for the trees. The test moderator must be able to weed out the significant from the insignificant, and this ability takes two concrete forms.

  • The ability to draw together all of the various inputs, comments, and data from a single test to form a cohesive picture of a participant's performance.
  • The ability to draw together the varied inputs from different test sessions and focus on the most important and critical findings.

It is very easy to get lost in the details and focus on trivial observations. It is also easy to be influenced by the last participant and forget all that has come before.

An effective test moderator, however, avoids these difficulties by staying focused on the big picture.

Good Communicator

Good communication skills are essential in usability testing. The test moderator must communicate with individual members of the development team, participants, observers, and other individuals who may be helping to administer the test in one way or another. The test moderator must be skillful at persuading others to make changes that are in their best interest, and he or she must be able to explain the implications behind the test results. Good writing skills are also essential because the test report is often the sole means of communicating test results to those who did not attend the test sessions. The written report is also the important historical document that is relied upon months or years later to review or revisit the test results.

Good Organizer and Coordinator

A usability test is a project within a project. Even a simple test requires the management of an astonishing number of small details, events, and milestones. Ensuring that equipment is in running order, getting all participants to the site on time, and making sure that the product is ready for testing are ultimately the responsibility of the test moderator. In addition, the test moderator is the focal point for the other test team members, and must coordinate their activities as well as those of any outside consultants into a unified effort. Therefore, the test moderator should be a good organizer and coordinator.

Getting the Most out of Your Participants

Let's explore some things that the test moderator can do to enhance the process. One of the best things a test moderator can do is to develop increased sensitivity to the plight of the participants. What does it feel like to be in the testing hot seat? Figure 4.1 shows a hypothetical example of one participant's point of view compiled from many of the participants observed over the years. While it is written tongue in cheek, its point is not. Participants are often placed in awkward, stressful situations where they have little control over events. The more you put them at ease, the greater are your chances for accurate results that are applicable to real-world situations. You should become familiar with the codes of ethics of the Usability Professionals' Association and the Human Factors and Ergonomics Society. The links for each are available on the web site that accompanies this book (

Figure 4.1 A Day in the Life of the Participant


Choose the Right Format

Keeping in mind the research questions you are working with, the location of the test sessions, the time you have available, and the types of participants involved, you must consider the best way to get data from the people you have brought in to help you evaluate the product.

Sit-By Sessions versus Observing from Elsewhere

When the practice of usability testing started, most sessions were conducted in a highly clinical style, with the moderator seated in a control room to observe and collect data. Over the years, a less formal, “sit-by” style of conducting sessions has developed for circumstances under which one would want to be near participants during usability testing, including most of the types of tests we describe in this book. For example, being with the participant in the testing room is advantageous for exploratory situations in which one wants to be able to gather the first impressions of participants about the design or functionality of a product. Doing so allows the moderator to be able to ask follow-up questions easily in a more cooperative session.

Additionally, if there is very quick interaction that must be observed closely or if you are conducting paper prototyping sessions, the moderator must be with the participant to mimic the computer actions.

However, if there is a chance in an assessment or validation test that the moderator's presence may bias the data, or if the emphasis is on gathering quantitative rather than qualitative data, set the test up to let the participant work by him- or herself while you manage the sessions from another room. For a deeper discussion of the physical arrangements for testing and observation rooms, see Chapter 6.1.

“Think-Aloud” Advantages and Disadvantages

Some participants will naturally verbalize what they are going through as they work through a usability test session. These people are gold; having a running commentary (something researchers call a “verbal protocol”) from participants as they “think aloud” while they perform tasks offers many insights to why a problem exists and how someone tries to work around it. Asking participants to think aloud during their sessions also reveals important clues about how they are thinking about the product or system they are using and whether the way it works matches up with the way it was designed.

Participants always filter to some extent, however, so they may consciously or unconsciously leave things out as they talk. Likewise, it is impossible to know everything that is going through a participants' mind during the session (it is of course likely that you don't want to know everything, just what pertains to your product, anyway). Thinking aloud can also help participants think through the design problem and form ideas for recovering. While this isn't always what they would do in real life, their doing so will give you ideas about how to remedy design problems.

One important reason to avoid asking participants to think aloud is when you are measuring time on tasks. Thinking aloud slows performance significantly.

Retrospective Review

An alternative to having participants think aloud is to replay the test with the participant after all of the tasks are done. As the moderator, you will have noted where during tasks the participant had issues, questions, or concerns. After the tasks are done, you can toggle to the point in the recordings where you want the participant to discuss what issues he or she was having, what the thought processes were, and how the workarounds were arrived at. Review can be a good tool to use with participants for whom talking and working might be difficult, such as small children, very old adults, and people with cognitive disabilities.

A serious drawback to performing a review is the time it takes to do. You have completed a full slate of tasks and now must spend more time reviewing the session from the beginning. It can take as much time as or more time than the main part of the session actually took. Some practitioners also avoid retrospective review because it gives some participants opportunity to revise and rationalize their behavior rather than simply reporting on what happened and why.

Give Participants Time to Work through Hindrances

Keeping in mind that the participant may be very nervous is the first step. Skillfully working with the person's sense of frustration is the next step. There will be times when the participant becomes exceedingly frustrated during the course of a test. When you see that this is occurring, recognize this as a critical point that can be advantageous. Many test moderators, at the first sign of user frustration, will immediately tell the participant to give up and go on to the next task. They do this in order to avoid confrontation, keep things on an even keel, and maintain the participant's interest. However, because the frustration is usually related to a critical deficiency in the product, moving on too quickly misses an important opportunity. The participant's behavior at this point can be more revealing than at any other time and can help the test team to understand how the participant learns to use the product. It is important to encourage the participant to continue rather than cutting the task short too quickly. The trick is to find just that point when the participant is frustrated but is still willing to try.

Offer Appropriate Encouragement

Another reason for encouraging the participant to keep trying is to show designers and developers watching the usability test the dire consequences of certain difficulties experienced by participants. It is important to let them see just how painful and frustrating the process of using their product can be. Actually seeing the participant struggle and get very frustrated and observing the serious consequences firsthand will do more to convince a designer to revise a product than weeks of discussions and negotiations. Sometimes just that extra amount of human struggle will convince a product designer that a change is needed and that the product should not be released as is.

So how should you encourage the participant? One way is to empathize with the participant and provide an end goal or end time frame. For example, you might say, “I can sense you're getting frustrated, but this is a particularly crucial part of the software/documentation, would you please try a little bit longer?”

You might try stating that other participants in the past have also had their share of difficulty. While this runs the risk of slightly biasing the participant, you may lose the participant in any case, if he or she has already experienced great difficulty. You might say, “I see you're having a difficult time with this. This isn't the first time that I've seen someone experience some difficulty here. Would you please continue on for five more minutes?”

The most skilled test moderator can encourage a participant to work with a smile. Make it seem like you and the participant are in this together, and that what is happening in the test is not a reflection of the participant's abilities. Often, the frustration level builds due to a sense of self-consciousness and a loss of self-esteem on the part of the participant. The test moderator can help immensely by deflecting the problem back to the product. For example, the test moderator might say, “Boy this sure is turning out to be a tough section. I really appreciate your efforts in trying to get through this.” Don't be afraid to encourage the participant to verbalize what's happening. Very often, if a participant is allowed to vent while performing, frustration can be minimized. It is up to the test moderator to gauge that fine line between rescuing the participant too early and pushing him or her beyond the point at which continuing the test is no longer possible. Through practice and experience, one can find the middle road between pushing too hard and giving up too early in order to get the best results.

Troubleshooting Typical Moderating Problems

Now that you have reviewed some of the important characteristics that a test moderator should possess, as well as some pointers for moderating the smoothest possible sessions with test participants, let's review some of the behaviors that test moderators should avoid. We describe the most common “errors” that test moderators make while conducting a test, many of which we have learned from experience. Consider it a list of “what not to do.” Even experienced test moderators can benefit from taking a few moments just prior to testing to review this list. As with the previous list of characteristics, you can use this list to evaluate and improve your own performance or to evaluate the performance of someone you hire to conduct usability testing for you.

Leading Rather Than Enabling

Behavior that leads rather than enables is usually caused by the test moderator being too close to the product and unintentionally providing cues to the participant about correct performance. The test moderator's tone of voice, a nod of the head, cutting short tasks when the participant struggles, even the type of question the test moderator asks can all influence the participant and the test results. This potential problem is the main reason why assessment and validation testing is often conducted from a control room, although subtle hints are still possible even from there. To avoid the problem of leading rather than enabling, remember that you are there to collect data and to ensure that the test environment remains as neutral as possible.

Too Involved with the Act of Data Collection

While the purpose of the test is to collect as much information as possible, the act of collecting that information should not interfere with direct observation of what is occurring. The test moderator needs to stay aware of what the participant is doing at all times, even if that means that every aspect of the participant's behavior is not written down. That's one purpose of recording the test, of developing coding categories, and of having others help with the more rote collection of such things as timings and number of references to an index or online help. All of these aids help ensure that the test moderator does not become so engrossed in the collection process that he or she misses important behaviors.

Acting Too Knowledgeable

This problem occurs when the test moderator and participant are in the same room during the test. Participants will tend to defer to the test moderator and ask many questions if they feel that the test moderator knows all the answers. Being too knowledgeable can also intimidate a participant who may be somewhat nervous and self-conscious about his or her abilities.

One simple way to counteract the problem of acting too knowledgeable is for the test moderator to “play dumb.” That is, the test moderator downplays any knowledge of the product and takes on the role of a research technician who is simply collecting data. Participants then change their entire demeanor when it becomes clear that they were not going to receive any assistance from the test moderator. They begin to try harder and to behave as if they were alone in their own home or office.

Too Rigid with the Test Plan

An experienced test moderator will know when to deviate from the test plan. It is important for the moderator to sense when the test design is not achieving its objectives and is not exposing the product's deficiencies to the fullest extent. At those times, it is up to the moderator to make the appropriate changes so that a participant's time and efforts are not wasted. Sometimes a participant with a different background than what was expected will appear. Sometimes the tasks are the wrong ones for addressing the research questions. Whatever the case, it is up to the moderator to revise the plan accordingly.

Not Relating Well to Each Participant

Participants come in all shapes, sizes, and demeanors. Regardless of whether a particular participant is shy, arrogant, moody, intimidated, self-conscious, or whatever, the test moderator needs to adjust his or her style in order to allow the participant to be comfortable and at ease. The test moderator should not get involved with battling, belittling, or in any way making a participant feel like anything but a guest. As far as the test moderator is concerned, the participant is always right.

Jumping to Conclusions

Inexperienced test moderators tend to overreact to early results. This can cause other members of the test team to act on the data prematurely. It is up to the test moderator to maintain a cool, steady demeanor and remind everyone to avoid forming conclusions until all of the results are in. One of the reasons for testing multiple participants is just for that purpose: to get a rounded, comprehensive view of the product through the eyes of different types of people with different types of backgrounds. While it is important for the test moderator to pick up patterns in the behavior of participants as early as possible, this does not necessarily mean reacting to that behavior. Avoiding premature conclusions helps to keep members of the test team from making major product changes before all the data is in.

How to Improve Your Session-Moderating Skills

Conducting a usability test is an extremely challenging and worthwhile endeavor on a variety of levels. On the most ordinary level, you are working very closely with people in an attempt to design a product for maximum utility and benefit. On a deeper level, it is a very profound experience that forces you to confront your own mind and its tendency to be biased, distracted, and flighty. Monitoring a test puts you on the spot and forces you to be extremely mindful and disciplined. You spend long periods of time maintaining concentration while observing people, all the time being as unobtrusive as possible. We both have found it to be delightful, frustrating, boring, and exhausting, and sometimes these feelings result from working with just the first participant.

If you are seriously considering acting as a test moderator on a regular basis in either some official or unofficial capacity, know that it can be a very rewarding and enlightening experience. Let's look at some ways for growing into this job.

Learn the Basic Principles of Human Factors/Ergonomics

Learn the basic principles of human information processing, experimental psychology, cognitive psychology, statistics, interface design, and usability engineering. Subscribe to and read the proceedings from the major societies (listed in Chapter 14.1). Attend seminars and study basic psychology courses. Many universities and community colleges offer certificate programs in usability testing or human factors. Attend conferences hosted by professional societies. (See Chapter 14.1 for a list.)

Learn from Watching Others

Watching other test moderators is a key to success. When you have an opportunity to watch an experienced test moderator at work, you get to see what works and what doesn't firsthand. If the opportunity presents itself, ask the test moderator why he or she uses a particular technique that caught your interest. Take notes about particular techniques, behaviors, and so forth that seem particularly effective and try them out yourself. Again, do not let your concern for making mistakes prevent you from exploring new techniques.

Watch Yourself on Tape

One of the benefits of recording your test sessions is that you have an ideal medium for reviewing your own performance. Take advantage of this technology by reviewing your sessions with the intent of improving your skills. Take notes on what you do well and on behaviors that need improvement. That way you will remember to work on those aspects the next time you moderate a test.

Work with a Mentor

Work closely with an experienced test moderator. Help the test moderator work on a test and have that moderator do the same for you. If it is a test with many participants, perhaps you can conduct some of the sessions. Have your mentor watch you and critique your performance. If you hire a consultant to help conduct a test, arrange for the consultant to work closely with you in a mentor/coaching relationship, so that you can learn faster than by just observing.

Practice Moderating Sessions

Start with the right attitude. Do not be a perfectionist. You are going to make mistakes, bias participants, reveal information you should not, and invent new ways to invalidate a test session's results. This is just par for the course. Usability testing has a twofold saving grace—testing multiple participants and iterative design. Testing multiple participants means that if you invalidate something in one session, there is always another opportunity to do it right. Iterative design also makes up for any mistakes you might make, because you have several chances throughout the product development lifecycle to catch problems with the product. The important thing is not to get discouraged. Continue to practice, continue to learn, continue to improve. Even the most experienced test moderators make mistakes.

Learn to Meditate

Meditation practice, specifically the type of meditation that fosters mindfulness and awareness, can be a valuable aid in learning to see clearly and in observing subtle nuances in behavior. This type of discipline is based on the belief that to understand another's mind, you first have to master your own.

Meditation practice or mindfulness training involves setting aside a period of time to sit down on a cushion and practice a simple breathing technique, while at the same time acknowledging thoughts that arise and letting them be. Over time, the result of this practice is a very personal and heartfelt recognition of how everything we perceive is filtered and biased by our version of things. Through continual practice, one's thoughts become more transparent, which in turn frees one to perceive more clearly and directly. During a test session, this is exactly what an excellent test moderator attempts to do; observe the participant's behavior free from the tyranny of his or her own expectations and biases.

Don't take up meditation strictly to become a better test moderator; that would be missing the point. However, if you are already inclined toward a discipline to quiet the mind and gain a clearer perception, meditation practice is a natural complement to the testing discipline.

Practice “Bare Attention”

“Bare attention” practice is an adjunct to meditation practice, except that it is done within one's normal daily routine. Practicing “bare attention” can heighten your ability to concentrate during test sessions. To practice “bare attention,” set aside a period of time (15–30 minutes is more than enough to begin) when you intentionally and very deliberately heighten your awareness of whatever you happen to be doing and of your surroundings. For example, if you are working at a computer, experience very deliberately the sense of your fingers hitting the keys, of your eyes looking from the paper to the screen, of your thought process. Notice when (and how often) your mind wanders from what you are doing, and when it does, gently bring it back to the present task at hand. The intent is to stay in the present moment 100 percent of the time. Try it sometime just to see how difficult it is. This practice, as with the previously described meditation practice, helps to foster mindfulness and awareness.