advice on usability testing for complex sites

19 Mar 2008 - 4:06pm
6 years ago
6 replies
527 reads
Meredith Noble
2010

Can anyone recommend methods for performing usability tests on large,
complex applications with lots of conceptual dependencies?

We're running into issues in our design of a test. We want to test
"Section B" of our application, but "Section B" doesn't make a lot of
sense unless you've already been exposed to "Section A". The trouble is,
Section A is pretty complicated in itself. They're definitely too big to
test together in a single 60 minute test.

What to do, what to do...! So far I've thought of (with drawbacks in
parentheses):

a) Have a facilitator walk them through Section A for 15 minutes before
they do the 60 minute Section B test (perhaps a bit overwhelming, hard
to digest)

b) Ensure the participants who test Section A come back and test Section
B (good in theory, but difficult to schedule)

c) Test the two back-to-back in a 120-minute-long test (participants
might fade)

d) Pretend the dependencies don't exist and have them test Section B
with no background knowledge (not realistic, but hey, maybe the others
are too ambitious)

Surely other people have had experience with this sort of thing - any
recommendations on what has and hasn't worked well? Am I approaching it
all wrong?

Thanks so much,

Meredith

Comments

19 Mar 2008 - 5:10pm
Katie Albers
2005

You may or may not be at liberty to do this, but my first action in a
case like this would be to examine how much of that complexity was
actually necessary and how much resulted from the development
process. I've seen more cases than I care to think about in which
that kind of examination resulted in a massively simplified
interaction. Questions like "Are we asking users to perform tasks
that could easily be performed by the system or vice versa?" "Are we
dividing up tasks/uses appropriately?" and other such points often
reveal places where you can eliminate several layers of complexity.

But as far as being stuck with what you've got...forget the 120
minute test. The user may or may not fade (probably will, but might
not), however your researcher is definitely going to go nuts during
the course of the 2nd or 3rd test. Look at how users will be
employing the app in real life: will it be something they always go
straight through? Will they be likely to leave it and return? Will
they use it daily, weekly, annually?

This is one of the few cases where I would recommend re-using a
particular set of testers. It looks like you're going to relying on
their memories in order to get any kind of actionable results.

Are there sub-tasks within the overall task? Test those as separate
entities and incorporate what you learn in the app that you test all
the way through.

[It's important to note here that I'm assuming one-on-one testing here]

If they're always going to go straight through then you really need
to test it that way. Make sure you schedule plenty of time between
for your researcher to decompress.

A possible alternative (and sort of a version of the sub-task
testing) is to test A on day 1, save the data or whatever you can or
need to have to start B, and then on day 2 test B. or you can divide
it up as morning session and afternoon session. Just make sure the
testers have been through A as recently as possible so that it can be
relatively fresh in their minds. Ask *them* to walk *you* through
what happened in the previous session. You can also build a mock-up
screen-by-screen review for the testers that they can walk you
through before they begin Section B.

On the whole, though, something where the *testing* is this complex
and difficult to design is likely to be hell to actually use. I wish
you luck.

Katie

At 6:06 PM -0400 3/19/08, Meredith Noble wrote:
>Can anyone recommend methods for performing usability tests on large,
>complex applications with lots of conceptual dependencies?
>
>We're running into issues in our design of a test. We want to test
>"Section B" of our application, but "Section B" doesn't make a lot of
>sense unless you've already been exposed to "Section A". The trouble is,
>Section A is pretty complicated in itself. They're definitely too big to
>test together in a single 60 minute test.
>
>What to do, what to do...! So far I've thought of (with drawbacks in
>parentheses):
>
>a) Have a facilitator walk them through Section A for 15 minutes before
>they do the 60 minute Section B test (perhaps a bit overwhelming, hard
>to digest)
>
>b) Ensure the participants who test Section A come back and test Section
>B (good in theory, but difficult to schedule)
>
>
>
>c) Test the two back-to-back in a 120-minute-long test (participants
>might fade)
>
>d) Pretend the dependencies don't exist and have them test Section B
>with no background knowledge (not realistic, but hey, maybe the others
>are too ambitious)
>
>Surely other people have had experience with this sort of thing - any
>recommendations on what has and hasn't worked well? Am I approaching it
>all wrong?
>
>
>
>Thanks so much,
>
>Meredith

--

----------------
Katie Albers
katie at firstthought.com

19 Mar 2008 - 4:24pm
Anthony Hempell
2007

I'd say B is really your only option ... all others involve some
compromising of the test quality and results.

B's difficulty is really just logistics. A lot of people have weekly
schedules, so scheduling two appointments, one for one day and then
again next week, might work.

In any case, you should probably plan for the inevitable no-shows for
the 2nd round so have enough scheduled that if 1/3 of the people don't
show up to the second one it won't be too much of a disaster.

On 19-Mar-08, at 3:06 PM, Meredith Noble wrote:
>
>
> a) Have a facilitator walk them through Section A for 15 minutes
> before
> they do the 60 minute Section B test (perhaps a bit overwhelming, hard
> to digest)
>
>
>
> b) Ensure the participants who test Section A come back and test
> Section
> B (good in theory, but difficult to schedule)
>
>
>
> c) Test the two back-to-back in a 120-minute-long test (participants
> might fade)
>
>
>
> d) Pretend the dependencies don't exist and have them test Section B
> with no background knowledge (not realistic, but hey, maybe the others
> are too ambitious)
>

19 Mar 2008 - 6:38pm
Marijke Rijsberman
2004

Hi Meredith,

I've done two-hour sessions, though not by choice. It's do-able if you take
a break in the middle. I would walk the participant over to a break room, so
I could be sure they had a little exercise, let them pick out something to
drink, show them where the bathroom was, etc. That way, we'd start up again
reasonably fresh.

Good luck with the testing.

marijke

19 Mar 2008 - 8:44pm
Melvin Jay Kumar
2007

Hi Meredith,

Agree on B.

But would like to add, that if you know what the dependcies are from A
to B, that would help a great deal.

So from understanding the dependencies and based on the resources and
time available ( which I rarely have enough) I would narrow down on
the most important tasks / scenarios that go from A to B.

So once that is done, then what would happen is that, you have one
session in which users get to do A and then continue to do B.

Also, if you have do test various scenarios, then you would have to
break the resources .

My 2 cents based on the simplistic understanding of your requirements.
Hope it helps.

Regards,

Jay Kumar

On 3/20/08, Meredith Noble <meredith at usabilitymatters.com> wrote:
> Can anyone recommend methods for performing usability tests on large,
> complex applications with lots of conceptual dependencies?
>
>
>
> We're running into issues in our design of a test. We want to test
> "Section B" of our application, but "Section B" doesn't make a lot of
> sense unless you've already been exposed to "Section A". The trouble is,
> Section A is pretty complicated in itself. They're definitely too big to
> test together in a single 60 minute test.
>
>
>
> What to do, what to do...! So far I've thought of (with drawbacks in
> parentheses):
>
>
>
> a) Have a facilitator walk them through Section A for 15 minutes before
> they do the 60 minute Section B test (perhaps a bit overwhelming, hard
> to digest)
>
>
>
> b) Ensure the participants who test Section A come back and test Section
> B (good in theory, but difficult to schedule)
>
>
>
> c) Test the two back-to-back in a 120-minute-long test (participants
> might fade)
>
>
>
> d) Pretend the dependencies don't exist and have them test Section B
> with no background knowledge (not realistic, but hey, maybe the others
> are too ambitious)
>
>
>
> Surely other people have had experience with this sort of thing - any
> recommendations on what has and hasn't worked well? Am I approaching it
> all wrong?
>
>
>
> Thanks so much,
>
> Meredith
>
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

20 Mar 2008 - 7:55am
Meredith Noble
2010

Wow, I just want to say thank you to everyone who weighed in on this.
This is why I love IxDA :)

I always struggle in explaining these sorts of situations without giving
away details about the actual application, but let me make an attempt.
Katie, it's not quite as dire as you might think, ha! The dependencies
aren't even *that* complex, it's just that I'd like to make the test
situation as realistic as possible. (I am wondering if this is simply
overly ambitious.)

Basically, in Section A, you define a widget. You give all sorts of
information about the widget, and are offered a bunch of different
widget options based on the information you enter.

In Section B, you're managing your widgets and various sub-widgets. (Oh
man, "sub-widgets", sorry, sigh.) What I'm struggling with is that in
almost all cases, the person who DEFINES the widget is the person
MANAGING the widget. And in the process of defining the widget, you get
all sorts of important background information about how widgets work. It
feels unrealistic to throw someone into the management of the widget
totally blind. I worry that they might struggle on certain tasks because
they don't have proper background information, background they would
almost certainly have if they defined the widget.

(Perhaps this points to other flaws in my design, ha. I will give it a
once-over and see if there are any other ways I can introduce this
background information.)

Anyway, they might very well define the widget and come back several
days later to manage it -- so option b) I described is reasonable,
although our recruiting company might hate us :), and I fully expect
some participant drop-off (the 1/3 over-recruitment strategy is a great
recommendation).

I am also strongly considering a), and letting them walk through the
widget definition flow by themselves 15 minutes before the 60 minute
test of widget management. I worry that if we prepare a prereq knowledge
document, or have a facilitator walk them through a scenario, we may
artificially emphasize important information -- information they might
actually have missed if they just read it through on their own.

Anyway, you've all given me lots of good food for thought. I will
continue pondering this. Thanks so much for your perspectives. And Paul,
whatever we do, an early pilot is DEFINITELY in order, thanks!

Meredith

> -----Original Message-----
> From: Melvin Jay Kumar [mailto:melvink2 at gmail.com]
> Sent: Wednesday, March 19, 2008 10:45 PM
> To: Meredith Noble
> Cc: discuss at ixda.org
> Subject: Re: [IxDA Discuss] advice on usability testing for complex
sites
>
> Hi Meredith,
>
> Agree on B.
>
> But would like to add, that if you know what the dependcies are from A
> to B, that would help a great deal.
>
> So from understanding the dependencies and based on the resources and
> time available ( which I rarely have enough) I would narrow down on
> the most important tasks / scenarios that go from A to B.
>
> So once that is done, then what would happen is that, you have one
> session in which users get to do A and then continue to do B.
>
> Also, if you have do test various scenarios, then you would have to
> break the resources .
>
> My 2 cents based on the simplistic understanding of your requirements.
> Hope it helps.
>
> Regards,
>
> Jay Kumar
>
>
>
> On 3/20/08, Meredith Noble <meredith at usabilitymatters.com> wrote:
> > Can anyone recommend methods for performing usability tests on
large,
> > complex applications with lots of conceptual dependencies?
> >
> >
> >
> > We're running into issues in our design of a test. We want to test
> > "Section B" of our application, but "Section B" doesn't make a lot
of
> > sense unless you've already been exposed to "Section A". The trouble
is,
> > Section A is pretty complicated in itself. They're definitely too
big to
> > test together in a single 60 minute test.
> >
> >
> >
> > What to do, what to do...! So far I've thought of (with drawbacks in
> > parentheses):
> >
> >
> >
> > a) Have a facilitator walk them through Section A for 15 minutes
before
> > they do the 60 minute Section B test (perhaps a bit overwhelming,
hard
> > to digest)
> >
> >
> >
> > b) Ensure the participants who test Section A come back and test
Section
> > B (good in theory, but difficult to schedule)
> >
> >
> >
> > c) Test the two back-to-back in a 120-minute-long test (participants
> > might fade)
> >
> >
> >
> > d) Pretend the dependencies don't exist and have them test Section B
> > with no background knowledge (not realistic, but hey, maybe the
others
> > are too ambitious)
> >
> >
> >
> > Surely other people have had experience with this sort of thing -
any
> > recommendations on what has and hasn't worked well? Am I approaching
it
> > all wrong?
> >
> >
> >
> > Thanks so much,
> >
> > Meredith
> >
> >
> >
> > ________________________________________________________________
> > Welcome to the Interaction Design Association (IxDA)!
> > To post to this list ....... discuss at ixda.org
> > Unsubscribe ................ http://www.ixda.org/unsubscribe
> > List Guidelines ............ http://www.ixda.org/guidelines
> > List Help .................. http://www.ixda.org/help
> >
>

25 Mar 2008 - 7:52pm
Dante Murphy
2006

Meredith-
The way I've approached this in the past is to use a larger sample size and create a matrix of test components based upon actual or experimental relevance.

For example, if section 1 of part A is the basis for section 4 of part B, then test that combination with 6 people, but test 4 other people with different parts of A and the same part of B.

If section 4 and section 12 of B are complex for the same reason...they both require analysis of a scatter plot, or both require the user to read German, or whatever, then ose 4 as an analogy model for 12, but test the parts of 12 that are unique contextually using time to task or success metrics isolated from the context of the whole application.

You basically have to do a lot more work in planning the test (really a series of related mini-tests), and the whole thing can look like a house of cards to the skeptics, but I've had a lot of success with it, both in getting the info I need and from various academic communities.

One nice part is that it's a great example of what Saffer calls "conservation of complexity"...the UX professional is working a lot harder so that the test subjects and project stakeholders don't have to.

If you have any other questions about this concept, feel free to contact me off-list.

Dante Murphy

________________________________

Can anyone recommend methods for performing usability tests on large,
complex applications with lots of conceptual dependencies?

Syndicate content Get the feed