How much setup information to give test participants

10 Nov 2008 - 2:36pm
6 years ago
1 reply
779 reads
Joel Tachau

I'm preparing a usability test on a major redesign of an online
marketplace for manufacturing. Our customers are going to see major
changes to the way they source RFQs using our online app. I am planning
to go into the interviews without telling the participants what has
changed. There will be emails and online training before and after the
launch. However, we all know people usually skip over this material.

Would it be fair to give the participants a brief explanation about how
the process has changed? I am inclined NOT to do this for three reasons:

1) Don't want to lead the use
2) Want to determine knowledge gaps so we can adjust launch
communications and online support
3) There will be limited help text and help icons (for popup help). We
will see in the test if the users read or click help and if they
contact customer service (email or chat). This will tell us where we
need to beef up the support.

On one hand we want the test to help determine where gaps are so we can
fix issues and develop support materials. But in a test setting, they
will read whatever instructional content we give them for each task and
we can't assume our customers will actually do this when we launch.

Thanks in advance for your help.

- Joel

Joel Tachau
Director of User Experience

Direct > 678.384.6377
Email > jtachau at
skype > mfgjoel


28 Nov 2008 - 11:03pm

I agree. Most people don't read the materials and training, so to get
value from the user testing, it is essential to see how they do
without the support material. Ideally, the new flow will be
immediately intuitive even for customers that are accustomed to the
old way. I like the smart new-user contextual messaging that's
become more common in the past year or three, prompting the user to
step through the process.

It will be interesting to see when they try to figure it on their own
and when they resort to the various levels of help.

Curious: do you expect that during user testing, users really will
contact customer service? I've not seen that structured into user
testing before. In my experience, it's just me moderating, providing
first really high-level guidance and then getting more direct if they
are still lost.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new

Syndicate content Get the feed