Remote usability testing

6 Mar 2008 - 10:43am
6 years ago
12 replies
1381 reads
Kevin Doyle
2007

I've been recently tasked with testing users on a call center application...
here's the kicker -- the client doesn't want to foot the bill for air
travel. I will have access to video conferencing and I think that I will be
able to use some kind of web conferencing software (like WebEx), but I've
never had to do anything like this. Does anyone have experience with remote
usability testing? Any recommendations on how to carry this out?

Thanks in advance!
k.

Comments

6 Mar 2008 - 11:26am
Dante Murphy
2006

http://www.techsmith.com/uservue.asp

Dante Murphy | Director of Information Architecture | D I G I T A S H E
A L T H
229 South 18th Street | Rittenhouse Square | Philadelphia, PA 19103 |
USA
Email: dmurphy at digitashealth.com
www.digitashealth.com

-----Original Message-----

I've been recently tasked with testing users on a call center
application...
here's the kicker -- the client doesn't want to foot the bill for air
travel. I will have access to video conferencing and I think that I will
be
able to use some kind of web conferencing software (like WebEx), but
I've
never had to do anything like this. Does anyone have experience with
remote
usability testing? Any recommendations on how to carry this out?

Thanks in advance!
k.

6 Mar 2008 - 10:59am
Susan Johnson
2007

We do remote testing all the time. Check out the Morae suite of tools. if
your company will make investment you will get lots of mileage for testing
locally and remotely. Here';s more info:

http://www.techsmith.com/morae.asp?CMP=KgoogleMtmhome

Susan Johnson
netXperience Practice Director
CSC Consulting Group
266 Second Avenue
Waltham, MA  02451-1122

Direct:   781-290-1370
Mobile:  617-571-3494
Fax:       781-890-1208

CSC Consulting, Inc.
Registered Office: 29 Sawyer Road, Waltham Massachusetts 02453, USA
Registered in USA No: 042593545

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

This is a PRIVATE message. If you are not the intended recipient, please
delete without copying and kindly advise us by e-mail of the mistake in
delivery.
NOTE: Regardless of content, this e-mail shall not operate to bind CSC to
any order or other contract unless pursuant to explicit written agreement
or government initiative expressly permitting the use of e-mail for such
purpose.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

"Kevin Doyle"
<kbdoyle at gmail.co
m> To
Sent by: "IXDA list" <discuss at ixda.org>
discuss-bounces at l cc
ists.interactiond
esigners.com Subject
[IxDA Discuss] Remote usability
testing
03/06/2008 10:43
AM

I've been recently tasked with testing users on a call center
application...
here's the kicker -- the client doesn't want to foot the bill for air
travel. I will have access to video conferencing and I think that I will be
able to use some kind of web conferencing software (like WebEx), but I've
never had to do anything like this. Does anyone have experience with remote
usability testing? Any recommendations on how to carry this out?

Thanks in advance!
k.
________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... discuss at ixda.org
Unsubscribe ................ http://www.ixda.org/unsubscribe
List Guidelines ............ http://www.ixda.org/guidelines
List Help .................. http://www.ixda.org/help

6 Mar 2008 - 11:44am
Mitchell Gass
2004

At 07:43 AM 3/6/2008, Kevin Doyle wrote:
>I've been recently tasked with testing users on a call center
>application...here's the kicker -- the client doesn't want to foot
>the bill for air travel. I will have access to video conferencing
>and I think that I will be able to use some kind of web conferencing
>software (like WebEx), but I've never had to do anything like this.
>Does anyone have experience with remote usability testing? Any
>recommendations on how to carry this out?

Hi Kevin,

UserVue is especially valuable when teams want to collect their
observations during test sessions in a common log and review points
of interest in the recordings after the sessions. For the highly
iterative testing I typically do, where there isn't time to review
recordings between tests, I use Citrix GoToMeeting

http://www.gotomeeting.com/

which is less expensive and very easy for participants to use. It
includes a conference call service, whose only drawback is that it's
a toll call for participants and observers.

GoToMeeting can record sessions if you use their conference call
service. If you'd like to provide a toll-free number as a courtesy to
participants, which I typically do, you can use a two-line phone with
a Conference button to connect the toll-free call with the GoToMeeting call.

Best,

Mitchell Gass
uLab | PDA: Learning from Users | Designing with Users
Berkeley, CA 94707 USA
+1 510 525-6864 office
+1 415 637-6552 mobile
+1 510 525-4246 fax
http://www.participatorydesign.com/

Mitchell Gass
uLab | PDA: Learning from Users | Designing with Users
Berkeley, CA 94707 USA
+1 510 525-6864 office
+1 415 637-6552 mobile
+1 510 525-4246 fax
http://www.participatorydesign.com/

6 Mar 2008 - 12:04pm
Sarah Kampman
2008

Is the call center application on your machine or on the test
participants' machines? If it's on theirs, you can simply watch via
WebEx, or you can have Morae Recorder capturing the WebEx session as
you see it. If it's on yours... you have a tougher problem. Maybe
UserVue allows participants to access apps on your machine -- I
don't have it, so I can't say. But this issue has been a big one
for me, so I wanted to be sure you were considering it.

-Sarah Kampman

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=26850

6 Mar 2008 - 3:51pm
Kevin Doyle
2007

Wow! I was not expecting replies this fast, everyone! Thank you so much!

I've only recently sold my employer on the value of usability testing and,
well, he's gone gang-busters on finding usability work for me. Be careful
what you wish for, eh? ;-)

Most usability testing I've done is usually paper-based (re: Carolyn
Snyder), HTML mock-ups or off of the UAT after a majority of the development
has already been done. Your replies have been more than helpful -- thanks
again, everyone.

k.

On Thu, Mar 6, 2008 at 1:35 PM, Jason Richardson <jasonr44240 at gmail.com>
wrote:

> Hi Kevin,
> We do an extensive amount of remote usability testing at E&Y on our
> applications. From a technical point of view, we are a Notes shop so we use
> Sametime Meeting to view the screens and capture the test with Camtasia. In
> your case, if you have a video feed as well you'll be able to capture some
> physical reactions to questions or tasks which we can't do with our testing.
>
> Not sure if you were also looking for some logistical aspects to remote
> testing, but here are a few that we use.
>
>
> - We use conference calls that have a toll free option for observers
> and participants. We typically have our observers dial in 10-15 minutes
> early so we don't hear beeping throughout the test. We then stress to the
> observers to stay on mute until after the test is completed and questions
> are opened to everyone. Nothing worse then hearing a sidebar conversation
> or coughing attack that leaks into the test.
> - We send out our observation worksheets prior to the test in order
> to maintain a consistent note-taking style. Since all of our observers are
> remote, this also helps them stay on track with the questions.
> - Not sure if you're tests will be international or across time
> zones but we need to watch ours and make sure our invitations are correct.
> It sounds obvious, but coordinating 10-15 tests across the US, UK and India
> can sometimes get a little messy.
> - Translation hasn't been a problem often, but in some instances
> we've either IM'd or emailed the question to a participant if they were
> proficient at reading English but had trouble understanding the question
> over the phone.
>
>
> Jason
>
> On Thu, Mar 6, 2008 at 10:43 AM, Kevin Doyle <kbdoyle at gmail.com> wrote:
>
> > I've been recently tasked with testing users on a call center
> > application...
> > here's the kicker -- the client doesn't want to foot the bill for air
> > travel. I will have access to video conferencing and I think that I will
> > be
> > able to use some kind of web conferencing software (like WebEx), but
> > I've
> > never had to do anything like this. Does anyone have experience with
> > remote
> > usability testing? Any recommendations on how to carry this out?
> >
> > Thanks in advance!
> > k.
> > ________________________________________________________________
> > Welcome to the Interaction Design Association (IxDA)!
> > To post to this list ....... discuss at ixda.org
> > Unsubscribe ................ http://www.ixda.org/unsubscribe
> > List Guidelines ............ http://www.ixda.org/guidelines
> > List Help .................. http://www.ixda.org/help
> >
>
>

6 Mar 2008 - 1:35pm
Jason Richardson
2007

Hi Kevin,
We do an extensive amount of remote usability testing at E&Y on our
applications. From a technical point of view, we are a Notes shop so we use
Sametime Meeting to view the screens and capture the test with Camtasia. In
your case, if you have a video feed as well you'll be able to capture some
physical reactions to questions or tasks which we can't do with our testing.

Not sure if you were also looking for some logistical aspects to remote
testing, but here are a few that we use.

- We use conference calls that have a toll free option for observers
and participants. We typically have our observers dial in 10-15 minutes
early so we don't hear beeping throughout the test. We then stress to the
observers to stay on mute until after the test is completed and questions
are opened to everyone. Nothing worse then hearing a sidebar conversation
or coughing attack that leaks into the test.
- We send out our observation worksheets prior to the test in order to
maintain a consistent note-taking style. Since all of our observers are
remote, this also helps them stay on track with the questions.
- Not sure if you're tests will be international or across time zones
but we need to watch ours and make sure our invitations are correct. It
sounds obvious, but coordinating 10-15 tests across the US, UK and India can
sometimes get a little messy.
- Translation hasn't been a problem often, but in some instances we've
either IM'd or emailed the question to a participant if they were proficient
at reading English but had trouble understanding the question over the
phone.

Jason

On Thu, Mar 6, 2008 at 10:43 AM, Kevin Doyle <kbdoyle at gmail.com> wrote:

> I've been recently tasked with testing users on a call center
> application...
> here's the kicker -- the client doesn't want to foot the bill for air
> travel. I will have access to video conferencing and I think that I will
> be
> able to use some kind of web conferencing software (like WebEx), but I've
> never had to do anything like this. Does anyone have experience with
> remote
> usability testing? Any recommendations on how to carry this out?
>
> Thanks in advance!
> k.
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

6 Mar 2008 - 4:16pm
Anonymous

Hi Kevin,

We did a lot of remote usability testing while I was at Microsoft since our
customer councils were spread out across the globe. Being able to record and
annotate the recording is really critical... MS had a custom tool, but I'd
look to the tools that others have recommended for some good ideas.

Beyond that I think there are some procedural questions that you might want
to consider. Since you aren't going to be there to work with the
participants face-to-face, I recommend arranging a 10-15 minute phone call
before conducting any testing to introduce yourself and just talk about the
process. This was helpful for laying the ground work and getting the user
used to talking to you on the phone, in the same way practicing a think
aloud in person helps the user adjust to that process.

Since you're doing this remotely, you have the benefit of users using the
software to do their own tasks, as opposed to tasks that you have created
(assuming your users are already using the software). In this case, you
might consider a diary study, where users take notes on their use of the
software for 2 weeks - jotting down how they used the system, making special
notes of any high or low points. You can then use the diary for a guided
retrospective during your actual interview, and cast a wider net in terms of
issues that you find. There are certainly tradeoffs between a diary study
and a traditional one-shot usability study, but I was fairly successful in
finding a fair few critical issues with a remote diary study.

Good luck!

-Sam

6 Mar 2008 - 9:45pm
Kshitiz Anand
2008

Remote testing really becomes useful when the user-designer gap is
huge gegraphically. Like when I was in India, the clients would
mostly be in US or Europe.
This is a pretty petty thing but one thing to make sure while doing
remote user testing is that the users are taking the test in the time
zone that they are most comfortable using the application.

For example if you are testing a office mail application, you would
not want the user to take the test at night (unless ofcourse the call
center here, is working at the night).

Also, it is advisable, if you could have a person to act as the
observer at the remote location. This is especially useful to take
into consideration the user's actions, that cannot be captured by
softwares like Webex. After the meeting, the observer could send you
the noted observations.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=26850

6 Mar 2008 - 10:14pm
Melvin Jay Kumar
2007

Just to add to the list as I practically do usability testing for
various regions based on the project requirements.

1) If no budget for tools, try using Netmeeting that comes with
practically all Windows Desktops.

2) Have a good internet connection, this is very critical. ( Lag costs
a lot of problem)

3) Before the testing, call the users and get accquainted and put the
fears/issues/etc at rest. ( very important).

I think these three are important for remote testing, plus all the
normal usability testing items .

Regards,

Jay Kumar

On 3/6/08, Kevin Doyle <kbdoyle at gmail.com> wrote:
> I've been recently tasked with testing users on a call center application...
> here's the kicker -- the client doesn't want to foot the bill for air
> travel. I will have access to video conferencing and I think that I will be
> able to use some kind of web conferencing software (like WebEx), but I've
> never had to do anything like this. Does anyone have experience with remote
> usability testing? Any recommendations on how to carry this out?
>
> Thanks in advance!
> k.
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

7 Mar 2008 - 8:35am
Kevin Doyle
2007

Hi Sarah,

We are still at a requirements phase but there are some parts of the
application that are what the team considers high-risk and need to be tested
before we go into development. I'll be testing with either an HTML mock up,
Flash mock up, PowerPoint presentation or an interactive PDF document. I've
been testing the application on paper with some local users and have
received some excellent feedback, but this application is going to be used
in lots of different ways by lots of different types of users. As a result,
I've been allowed to expand my testing to their three external call centers.

We'll get into more comprehensive testing once development has started to
flesh things out, but for now, we're just testing the high-risk areas of the
application.

Heh, that was a long answer to your simple question of whether or not the
application is on my machine or theirs, eh? ;-)

k.

On Thu, 6 Mar 2008 09:04:40, SarahK <skampman at planview.com> wrote:

> Is the call center application on your machine or on the test
> participants' machines? If it's on theirs, you can simply watch via
> WebEx, or you can have Morae Recorder capturing the WebEx session as
> you see it. If it's on yours... you have a tougher problem. Maybe
> UserVue allows participants to access apps on your machine -- I
> don't have it, so I can't say. But this issue has been a big one
> for me, so I wanted to be sure you were considering it.
>
> -Sarah Kampman
>

7 Mar 2008 - 8:50am
Kevin Doyle
2007

Hmm! I never thought of the time zone difference as a factor in testing. I
believe the call centers are all here on the East Coast, but I'll double
check.

I might be able to have a video while I'm testing remotely, too -- the
client has just recently setup video teleconferencing rooms at their large
office locations. However, I think I might want to have someone observing
them, regardless.

Thanks for the tips!

On Thu, 6 Mar 2008 18:45:08, Kshitiz Anand <kshitiz_anand at yahoo.com> wrote:

> Remote testing really becomes useful when the user-designer gap is
> huge gegraphically. Like when I was in India, the clients would
> mostly be in US or Europe.
> This is a pretty petty thing but one thing to make sure while doing
> remote user testing is that the users are taking the test in the time
> zone that they are most comfortable using the application.
>
> For example if you are testing a office mail application, you would
> not want the user to take the test at night (unless ofcourse the call
> center here, is working at the night).
>
> Also, it is advisable, if you could have a person to act as the
> observer at the remote location. This is especially useful to take
> into consideration the user's actions, that cannot be captured by
> softwares like Webex. After the meeting, the observer could send you
> the noted observations.
>

7 Mar 2008 - 12:05pm
Alla Zollers
2008

Hi Kevin,

In my last company all of my usability testing was done remotely. We used GoToMeeting to share computers (normally your computer where the prototype resides), record the session, and also give the user mouse/keyboard control to let them go through the tasks.

Everyone gave really good advice. One more thing I would suggest is to have one person actually lead the usability study and take the people through the tasks, and have another person just observe and take notes. It is impossible to do both at the same time as you can miss a lot of nuanced behavior. It doesn't sound like you have a usability team since this process is fairly new, so grab a product manager to actually lead the task and you do the observing and note taking.

Good luck!

Syndicate content Get the feed