User Testing and Recruiting Advice

6 Feb 2009 - 10:39am
5 years ago
11 replies
536 reads
Rob Tannen
2006

Vicki - Some recommended bast practices for participant recruitment
and retention:

-Expect a 25% cancellation/no-show rate. So for every four slots you
need to fill, you need to recruit 5 people. You can schedule extra
slots or use "floaters" -- people scheduled to serve as
substitutes overlapping with specific time slots. If you're running
ahead of schedule you can include the floaters as well.

-Remind recruited participants ahead of time (week before, then day
before). Tell people to come in 15 minutes before the actual testing
slow and then call them on their cell phones if they are not there at
that time.

-Pay more money. Increase your participant incentive by 10-20% can
be a cost-effective way to improve show-up rates.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=38251

Comments

6 Feb 2009 - 2:39pm
Mary Deaton
2008

I agree with Rob, but in house recruiting is time consuming, which means it
costs the company money, but it is a hidden cost. It costs about $100 for
scheduled participant to use a recruiting service and they take on the
burden of replacing no shows and such. If you find a recruiting agency you
like, stick with them. Give them a very detailed user profile and provide
the qualifying criteria for each study and let them do the work so you can
focus on designing and administering the tests. That is what Microsoft does
because they need thousands of participants every year; they contract with
vendors to do the work to get butts into chairs.

I have done lots of remote testing, and it often works well, depending on
what you are testing or evaluating. But I have had even more no-shows with
remote testing and I think this is because they have no sense that anyone
will miss them, perhaps.

And when Rob says pay more money, that means paying people what they feel
their time is worth. You are taking 1, 2, or 3 hours of their time in order
to improve a product and make a higher profit. They know that and I think
participants sometimes feel we take advantage of them. Never expect people
to show up out of altruism.

Mary Deaton
Manager, STC Usability and User Experience

On Fri, Feb 6, 2009 at 7:39 AM, Rob Tannen <rtannen at bresslergroup.com>wrote:

> Vicki - Some recommended bast practices for participant recruitment
> and retention:
>
> -Expect a 25% cancellation/no-show rate. So for every four slots you
> need to fill, you need to recruit 5 people. You can schedule extra
> slots or use "floaters" -- people scheduled to serve as
> substitutes overlapping with specific time slots. If you're running
> ahead of schedule you can include the floaters as well.
>
> -Remind recruited participants ahead of time (week before, then day
> before). Tell people to come in 15 minutes before the actual testing
> slow and then call them on their cell phones if they are not there at
> that time.
>
> -Pay more money. Increase your participant incentive by 10-20% can
> be a cost-effective way to improve show-up rates.
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=38251
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

--
Mary Deaton
Yes we can. Yes we did. Yes we will

6 Feb 2009 - 2:48pm
Samantha LeVan
2009

My company struggles with the same issues. We try to offer decent
financial incentives but that doesn't always guarantee success.

When screening participants, gauge their level of interest and assume
that the lower the perceived interest, the lower the chance they will
arrive on time. This is a generalization but in my experience, there
is a correlation.

Consider how far your participants have to drive and send them door
to door directions the day before.

Call the participants the day before to see if they have any
questions or concerns and remind them of the scheduled time.

Let participants provide referrals of friends and family who may want
to do a study with you. Those referrals are likely to be more
successful because they will have a word-of-mouth understanding of
the usability test process.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=38251

6 Feb 2009 - 3:26pm
Jared M. Spool
2003

Hi Vicki,

I'll take a stab at answering these two, since the others are very
broad and could result in endless threads of their own on this list.

On Feb 6, 2009, at 6:44 AM, Vicki Splaine wrote:

> 1) Tips for successful recruiting using in house resources, as we can
> not use an outside recruiting agency for cost reasons?

I co-wrote an entire report on just this subject called Recruiting
Without Fear (http://is.gd/iFbB). It's 43 pages of tips on how to do
your own recruiting.

In our research, the most effective organizations do their own
recruiting, because it's a great way to extend your research
capabilities. It's not hard to do, but there is a fair amount of
administration involved, so you have to be prepared to put the
resources into it. Shirking on the resources will reduce the
effectiveness, increase your costs, and take a lot more time in the
long run.

> 2) We have considered forming a "Customer Council". A group of
> dedicated users that represent defined personas for each brand. The
> idea is to utilize these individuals to get their feedback/opinion on
> website enhancements, etc. Does anyone have an experience using a
> "Customer Council" model?

We've used customer councils and advisory boards in research with many
of our clients. When done well, it's very powerful.

A couple of caveats:

1) Not all councils / boards are the same. Before invitations are sent
out, the team really needs to know what they want to get from the
council. If you just invite folks without having clear goals, it
reduces the likelihood that the council/board will produce long-term
useful results, beyond just enhancing the power of the echo chamber.

2) Make sure you're not confusing personas (which are behavior
archetypes) with market segments (which are demographic/psychographic
groupings). For most organizations, it would be virtually impossible
to form a council around personas, since well-formed personas don't
map into specific individuals or groups.

3) Look beyond focus groups. Far beyond focus groups. Like, if someone
says, "Hey, let's just invite them together for a sort of focus
group," take that individual out and slap them around. (I really think
we don't take enough advantage of natural selection. Sometimes we just
need to thin the herd, know what I mean?) If the primary execution of
your council/board is to gather them together and have them discuss
designs, don't bother. It's not going to provide any more insight than
you currently have and, as found in many documented cases, is likely
to lead your team in the wrong direction. The council/board will be of
most use if you use them as a base for behavioral work, which probably
means going to them and watching, not discussing.

Hope that helps,

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks Twitter: jmspool
UIE Web App Summit, 4/19-4/22: http://webappsummit.com

6 Feb 2009 - 3:34pm
Jared M. Spool
2003

Rob,

With all due love and respect, I disagree with these recommendations.
At least, two of them.

On Feb 6, 2009, at 7:39 AM, Rob Tannen wrote:

> Vicki - Some recommended bast practices for participant recruitment
> and retention:
>
> -Expect a 25% cancellation/no-show rate. So for every four slots you
> need to fill, you need to recruit 5 people. You can schedule extra
> slots or use "floaters" -- people scheduled to serve as
> substitutes overlapping with specific time slots. If you're running
> ahead of schedule you can include the floaters as well.

25%!!! We have less than a 2% no-show rate across thousands of
participants (including neurosurgeons!) over the last 20 years. (We
don't get paid if participants don't show up, so we don't have any
tolerance for no-shows.)

If you have a 25% no-show rate, you've got something really wrong with
your recruiting practice. There's no excuse that 1 out of 4 folks
don't show up.

In most studies, you never have to recruit floaters. For the
occasional high-priority study, you might include a couple of extra
participants in the original schedule to be safe. We do that for one
out of every ten studies we do.

> -Remind recruited participants ahead of time (week before, then day
> before). Tell people to come in 15 minutes before the actual testing
> slow and then call them on their cell phones if they are not there at
> that time.

This is better. Though, since we almost never have no-shows, we don't
bother with the 15-minute "before" thing. No sense in wasting anyone's
time when everyone is where they are supposed to be.

> -Pay more money. Increase your participant incentive by 10-20% can
> be a cost-effective way to improve show-up rates.

Huh. We deal with many participants that can't, by law, get paid for
helping us. They still show up when they are supposed to. Money to get
people to show us is a trap that will likely lead to lower quality
sessions. You really don't want people who are there for the money.
(The money is really just to prove to them that you're serious and not
going to turn it into a timesharing sales session.)

It really sounds to me like your recruitment process could use an
overhaul if this is what you're seeing for results.

That's my $0.02.

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks Twitter: jmspool
UIE Web App Summit, 4/19-4/22: http://webappsummit.com

6 Feb 2009 - 3:56pm
Rob Tannen
2006

Jared - You're one of four people (the other three being my wife and
daughters) who I can't win an argument with, so I won't bother.
But I am curious as to whether others have recruitment experiences
that are more similar to yours or mine. Maybe I'll have to download
your report.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=38251

6 Feb 2009 - 4:00pm
Jared M. Spool
2003

In addition to our report, I should've also mentioned that Dana
Chisnell has been writing on this topic on her blog:

Yes or No: Make Your Recruiter Smarter
http://is.gd/iFrx

Why Your Screener Isn't Working
http://is.gd/iFrV

Her firm, Usability Works, are ideal recruiters and understand why UX
recruiting is not the same as market research recruiting. They also
have very low no-show rates and produce high-quality participants,
even for very unusual project requirements. They're on the top of my
list for outside recruiters. (Dana will also help you set up an
effective internal recruitment process, if that's how you want to go.)

Jared

On Feb 6, 2009, at 12:26 PM, Jared Spool wrote:

> Hi Vicki,
>
> I'll take a stab at answering these two, since the others are very
> broad and could result in endless threads of their own on this list.
>
> On Feb 6, 2009, at 6:44 AM, Vicki Splaine wrote:
>
>> 1) Tips for successful recruiting using in house resources, as we can
>> not use an outside recruiting agency for cost reasons?
>
> I co-wrote an entire report on just this subject called Recruiting
> Without Fear (http://is.gd/iFbB). It's 43 pages of tips on how to do
> your own recruiting.
>
> In our research, the most effective organizations do their own
> recruiting, because it's a great way to extend your research
> capabilities. It's not hard to do, but there is a fair amount of
> administration involved, so you have to be prepared to put the
> resources into it. Shirking on the resources will reduce the
> effectiveness, increase your costs, and take a lot more time in the
> long run.
>
>> 2) We have considered forming a "Customer Council". A group of
>> dedicated users that represent defined personas for each brand. The
>> idea is to utilize these individuals to get their feedback/opinion on
>> website enhancements, etc. Does anyone have an experience using a
>> "Customer Council" model?
>
> We've used customer councils and advisory boards in research with
> many of our clients. When done well, it's very powerful.
>
> A couple of caveats:
>
> 1) Not all councils / boards are the same. Before invitations are
> sent out, the team really needs to know what they want to get from
> the council. If you just invite folks without having clear goals, it
> reduces the likelihood that the council/board will produce long-term
> useful results, beyond just enhancing the power of the echo chamber.
>
> 2) Make sure you're not confusing personas (which are behavior
> archetypes) with market segments (which are demographic/
> psychographic groupings). For most organizations, it would be
> virtually impossible to form a council around personas, since well-
> formed personas don't map into specific individuals or groups.
>
> 3) Look beyond focus groups. Far beyond focus groups. Like, if
> someone says, "Hey, let's just invite them together for a sort of
> focus group," take that individual out and slap them around. (I
> really think we don't take enough advantage of natural selection.
> Sometimes we just need to thin the herd, know what I mean?) If the
> primary execution of your council/board is to gather them together
> and have them discuss designs, don't bother. It's not going to
> provide any more insight than you currently have and, as found in
> many documented cases, is likely to lead your team in the wrong
> direction. The council/board will be of most use if you use them as
> a base for behavioral work, which probably means going to them and
> watching, not discussing.
>
> Hope that helps,
>
> Jared
>
> Jared M. Spool
> User Interface Engineering
> 510 Turnpike St., Suite 102, North Andover, MA 01845
> e: jspool at uie.com p: +1 978 327 5561
> http://uie.com Blog: http://uie.com/brainsparks Twitter: jmspool
> UIE Web App Summit, 4/19-4/22: http://webappsummit.com
>

6 Feb 2009 - 4:15pm
Jared M. Spool
2003

On Feb 6, 2009, at 12:56 PM, Rob Tannen wrote:

> Jared - You're one of four people (the other three being my wife and
> daughters) who I can't win an argument with, so I won't bother.
> But I am curious as to whether others have recruitment experiences
> that are more similar to yours or mine. Maybe I'll have to download
> your report.

I know the feeling. *I* can't even win an article with *me* somedays.
And it really pisses me off.

Seriously. Download the report. If it doesn't help, I'll refund the
money.

And feel free to contact me to talk about how to get your no-show rate
down. It'll really make your life easier and get you better results in
the long run.

Jared

6 Feb 2009 - 10:21pm
Samantha LeVan
2009

In thinking more on this topic and after absorbing Jared's comments,
another possible problem came to mind. You have to be flexible and
accommodating to the users you really want to bring in. If you have
found truly representative users for your study, work with their
schedule. Many people with 9-5 jobs cannot be at a study at 10am.
Offer evening and weekend opportunities in addition to traditional
business hours.

Find out if they have young children. I've had studies postponed at
the last minute because of childcare difficulties. Knowing this might
happened allowed me to communicate with users so they know it's okay
to let me know if something comes up. Rather than becoming a no-show
or cancelation, it's a reschedule for another day.

Look at how you promote the opportunity. If it doesn't sound fun or
lacks an impact on products the user actually uses, it could be hard
to convince them to help without much incentive. Show the user how
their feedback could impact the design. Make them feel like they have
a say in something important. This also works really well for internal
studies on enterprise applications. Employees like having a say in the
software they use everyday.

Samantha

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=38251

6 Feb 2009 - 12:22pm
Paul H Greasby
2009

I concur with all of Rob's comments. Certainly finding the right
incentive is key. One of my clients recently said that as the world
is in economic hard times, then we should be able to pay people less.
I defended this in the context of he cost of the test, i.e. paying 5
people $20 less each is a tiny saving in the context of the whole
session (for which a majority attendance is required), and the risk
it introduces.

Other pointers include:
- collecting people's cellphone numbers
- telling people you'll remind them beforehand (ensuring of course
you do)
- telling people the incentives are cash if this is the case
- letting people know that there's only a small number of attendees
so it's really important they attend
- informing them that it's a one-to-one, not a group (i.e. they WILL
be missed)
- giving them ALL the information to ensure their arrival on time.
This can include nearby parking, rates, payment methods, and how much
additional time to allow for this
- give clear instructions on how to find the facility. Many testing
facilities are hidden inside a nondescript buildling

Basically the key is the incentivize enough and mitigate all the
little things that can make a participant late

The other thing that helps is scheduling sessions with gaps between,
rather than back to back. This allows particularly insightful
sessions to extend, but also late showing participants a chance of
still doing a full session.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=38251

7 Feb 2009 - 4:45pm
Mike Myles
2009

Regarding your question about remote testing - I've done a great deal
of remote testing and have had excellent results. I've even run
pseudo-paper prototypes remotely using PhotoShop layers to handle the
paper switching. PowerPoint can be useful for that as well

As for software to use, I've had the best results by far with
GoToMeeting. It's simple to use and the performance is very good. It
also has some handy pen & highlighter tools that are useful when
having discussions about the UI with the participant on the other
end.

Remote testing is great because people can squeeze in tests right
from their desk whenever they have a spare hour or so, and
geographical location is virtually irrelevant - though time zone &
language may be for international testing.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=38251

11 Feb 2009 - 1:08pm
Laurie Mohler
2009

My company does recruiting. We have done remote as well as on site
testing. Many of our clients use WebEx for testing. We have also
had clients use web cameras.

One of the keys to recruiting success is having the recruiters
understand who you are looking for. Most recruiting companies have a
room full of drones who aren't even trusted enough to give them
computers.

In house recruiting is certainly possible but you'll need a way to
track respondents, and if you want them to be fresh you'll need to
develop a database that tracks participation, notes, demographics and
so on.

Jared's comments that "We deal with many participants that can't,
by law, get paid for helping us." This is a different kind of
recruiting than general consumers.

General consumers should absolutely be paid well for their opinions,
just as you are. We tell participants they are being paid for doing
a job and should view the appointment in the same way they would view
showing up at work.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=38251

Syndicate content Get the feed