discount usability testing on the web, e.g. http://www.usertesting.com/

6 Dec 2007 - 8:32pm
6 years ago
11 replies
1409 reads
Dave Cortright
2005

After working mostly for big corps with deep pockets to have dedicated
usability folks on staff, I now find myself in a start-up environment where
I'm pretty much it as far as design goes. I'd like to do some user testing
and would love to outsource it.

Does anyone have experience with http://www.usertesting.com/ or other
similar services? Or more broadly, what would you recommend? I actually have
done testing myself before (in fact that was my first job in the industry as
an intern) but I'd rather spend my time on design if I can get reasonable
results otherwise.

Thanks!
·Dave

Comments

6 Dec 2007 - 8:43pm
Todd Warfel
2003

On Dec 6, 2007, at 8:32 PM, David Cortright wrote:

> Does anyone have experience with http://www.usertesting.com/ or
> other similar services?

Someone sent me this link a few days ago. I'm not sure how much faith
I would put into people using the site and recording themselves
talking about it for usability testing. You will get subjective
feedback on what they think about the site, which is good, and watch
them use parts of it, which is good, but as far as being able to
measure it, well...

How would you propose measuring something that every user could have
different experiences, chose different paths, etc.?

Don't get me wrong, I think it's a brilliant on what they're trying to
do—bringing usability to the masses. However, measuring impact and
severity is going to be an issue.

Cheers!

Todd Zaki Warfel
President, Design Researcher
Messagefirst | Designing Information. Beautifully.
----------------------------------
Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
----------------------------------
In theory, theory and practice are the same.
In practice, they are not.

6 Dec 2007 - 9:27pm
Craig Peters
2007

> On Dec 6, 2007, at 5:43 PM, Todd Zaki Warfel wrote:
>>
>> I'm not sure how much faith
>> I would put into people using the site and recording themselves
>> talking about it for usability testing. You will get subjective
>> feedback on what they think about the site, which is good, and watch
>> them use parts of it, which is good, but as far as being able to
>> measure it, well...
>>

I agree with Todd to be skeptical. I disagree, however, that it's
good to hear what people think about your site - if you're calling
this a usability study, that is. I would advise against using
something that ends up asking people to say what they like and don't
like. Or, at least, don't call it a usability study. Call it a tell-
me-what-you-think-about-our-site study.

You're going to end up with a serious validity risk.
- People behave differently than they think (there's lots of
research to support this, I believe)
- Explaining what they're doing &/or sharing likes/dislikes changes
how they behave

An experienced human moderator can set the stage and redirect
participants to *do* more and *explain* less in a usability session.

Conducting remote usability (screen sharing) with a real moderator
(phone) can save money and time compared to in-lab testing. Bolt
Peters and others do this (www.boltpeters.com). Also, I've recently
conducted some guerilla testing along the RITE approach (http://
en.wikipedia.org/wiki/RITE_Method).

craig

7 Dec 2007 - 5:32am
Alexander Baxevanis
2007

What looks really silly is what they mention on this page:

http://www.usertesting.com/WhoAreUserTesters.aspx

Just by looking at the phrase "People who are especially good at
usability testing" you can imagine that the results can be quite
skewed.

"[...] user testers are filtered out [...] because of the customer
ratings they receive. So the quality of our user testers continues to
improve."

Just great, if the testers say something about your site that you
don't like, you can always give them a bad rating...

If anyone has got the time to do an experiment, i.e. make a mock site
with a few usability issues and try to test it with this company, it
would be fun to see the results. I'm up for paying the $19 just to see
what happens...

Cheers,
Alex

On Dec 7, 2007 1:32 AM, David Cortright <davecort at gmail.com> wrote:
> After working mostly for big corps with deep pockets to have dedicated
> usability folks on staff, I now find myself in a start-up environment where
> I'm pretty much it as far as design goes. I'd like to do some user testing
> and would love to outsource it.
>
> Does anyone have experience with http://www.usertesting.com/ or other
> similar services? Or more broadly, what would you recommend? I actually have
> done testing myself before (in fact that was my first job in the industry as
> an intern) but I'd rather spend my time on design if I can get reasonable
> results otherwise.
>
> Thanks!
> ·Dave
> ________________________________________________________________
> *Come to IxDA Interaction08 | Savannah*
> February 8-10, 2008 in Savannah, GA, USA
> Register today: http://interaction08.ixda.org/
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

7 Dec 2007 - 7:48am
bminihan
2007

I agree it's pretty odd that they have determined what a good "user tester" is. The three bullets here:
They think out loud. In other words, they speak their thoughts as they browse.
They identify things that are confusing, difficult to accomplish, or other problems.
They make useful suggestions about how to improve a site.

That sounds like my mom. The reason why you ask usability study participants to think out loud, identify things that are confusing, and to make suggestions is BECAUSE you don't want to test your site with people who naturally behave this way all the time. The critical component of your usability study should be testing with likely future users, not people chosen for their testing ability.

Bryan
http://www.bryanminihan.com

-----Original Message-----
From: discuss-bounces at lists.interactiondesigners.com [mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of Alexander Baxevanis
Sent: Friday, December 07, 2007 5:32 AM
To: David Cortright
Cc: discuss at ixda.org
Subject: Re: [IxDA Discuss] discount usability testing on the web, e.g. http://www.usertesting.com/

What looks really silly is what they mention on this page:

http://www.usertesting.com/WhoAreUserTesters.aspx

Just by looking at the phrase "People who are especially good at
usability testing" you can imagine that the results can be quite
skewed.

Cheers,
Alex

7 Dec 2007 - 9:23am
Faith Peterson
2007

I've also used real-time remote user testing with satisfactory results. We
developed a script for the sessions which we ran via Webex and phone. After
some initial questions to identify the tester - what version of the product
they use, what their typical activities are, frquency of use - we handed
over control of the app and posed a series of tasks. We recorded the Webex
and voice call, but we also used two evaluators - one to conduct the session
and one to take notes.

We gathered a lot of useful information that enabled us to identify specific
factors to improve and also validated many of the benefits we expected the
feature under test to provide.

I think you do have to bring a little more empathy to the process when
working this way, at least if you're not on a video conference. There were
times we suspected from the mouse movements we observed that there was some
uncertainty to probe, but of course we didn't have any facial or body
language cues.

If there's no budget even for Webex, you might look into Yugma (
www.yugma.com).

--
Faith Peterson
f.a.peterson at gmail.com
On Dec 6, 2007 8:27 PM, Craig Peters <craig at craigp.com> wrote:

> > On Dec 6, 2007, at 5:43 PM, Todd Zaki Warfel wrote:
> >>
> >> I'm not sure how much faith
> >> I would put into people using the site and recording themselves
> >> talking about it for usability testing. >>
>
Conducting remote usability (screen sharing) with a real moderator
> (phone) can save money and time compared to in-lab testing. Bolt
> Peters and others do this (www.boltpeters.com). Also, I've recently
> conducted some guerilla testing along the RITE approach (http://
> en.wikipedia.org/wiki/RITE_Method).
>
> craig
>

7 Dec 2007 - 8:29am
Chris Shepherd
2007

I'm fairly skeptical also, as I think that the usability testing
concept has kinda gone out the window with this. Random browsing with
some opinions isn't really user testing, particularly if they also
start offering advice like this shoudln't be here.

Does anyone know if these users are given set tasks at all?

However, despite this, $19 for a person to do that for you is cheap
considering what they provide back, but i wouldn't make too many
decisions based on the feedback. It might however provoke thoughts
and other ideas on reading their feedback, as there may well be
crucial silly things that have been left on your website.

Maybe it could be classed as cheap functional testing as well, broken
links etc!

Chris

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=23315

7 Dec 2007 - 7:56am
Hussein Ahmed8
2007

Hey, guys, I think their idea is magnificent but I do agree with you
all about the effectiveness of the evaluations. There are no any
credentials stated and for sure they have nothing to be proud of.

What a constructive talk we can have is why not set up a similar
website to outsource our experience to people who do really need it.
A very powerful presence would be to represent IxDA maybe and a
significant portion of the revenue could be for the sake of raising
funds for IxDA.

I think this model have got a very high potential of success.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=23315

7 Dec 2007 - 2:13pm
Sunny Beach
2007

...The critical component of
your usability study should be testing with likely future users, not
people chosen for their testing ability.

The idea above is very important. As a graduate student at the University of Michigan in HCI, it is really nice (easy) to use peer students for user tests. The problem is that these students "know too much" and do not really represent the wider world of computer users.

The user tests are more like heuristic evaluations and tend to miss some of the more subtle (potentially confusing) elements.

Sunny

--
Sunny Beach
University of Michigan School of Information
Human-Computer Interaction Specialization
www.si.umich.edu/
sunnybeach1999 at yahoo.comcell 734.255.0261

____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

8 Dec 2007 - 3:24am
Bruno Figueiredo
2007

I've done lots of usability testing and although I find the idea
interesting, it's also flawed at its core:

- The testing protocol is missing since users roam at will through
the web site. And you can't adjust testing on the fly when you
notice that they're having trouble with a particular task.

- You can't ask them important questions like "what did you think
you could do by clicking on that" when the prototype isn't fully
functioning.

- You can't see their expressions of surprise and frustration.

- It would be very hard to compile several of this videos findings
into a consistent document and worst of all you couldn't gather
quantitative results.

Having said that, this is more like a one-off focus group session
than usability testing.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://gamma.ixda.org/discuss?post=23315

9 Dec 2007 - 11:59am
Faith Peterson
2007

Apropos of budget-priced testing, has anyone used UserVue from
TechSmith?

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=23315

9 Dec 2007 - 1:05pm
Carol J. Smith
2007

Faith Peterson <f.a.peterson at gmail.com> wrote:
>Apropos of budget-priced testing, has anyone used UserVue from TechSmith?

I've used UserVue it to conduct remote usability tests and have found it to
be easy to use and reliable. You need to of course make sure you have room
to save the recording and it must be used with Internet Explorer. Also, the
participant must be using IE in order to participate (verify during
recruitment).

It can be imported into Techsmith's Morae product giving you the ability to
do editing and make highlight videos or you can just use the .wmv file to
review the tests.

I have not used UserVue with observers, but I imagine it is just as reliable
as Morae. There can be a variety of technical issues with remote testing but
in general I have found UserVue to be a great option.

Carol J. Smith
Principal Consultant
Midwest Research - www.mw-research.com
carol at mw-research.com

Syndicate content Get the feed