User Research: Three user groups, five hours. What would you do?

2 Feb 2009 - 3:54pm
5 years ago
10 replies
762 reads
Josh Evnin
2005

To add to the great advice above:

- Make sure the people you'll be observing are prepared for you to be
there. They SHOULD NOT clear their schedules to be with you. They should
have real, regular work to get done while you're with them and you need to
set that expectation ahead of time.
- An hour or two for each observation should be a good amount of time,
but make sure that you're observing enough of the "important stuff." The
"important stuff" is the same as the Big Questions Dana talks about.
- I second Nicholas's idea of starting the day with a group meeting, but
don't let this go too long. Maybe half an hour, maybe. One thing I've done
in the past is give people some "homework" at this meeting. You can hand out
disposable cameras for people to take photos of their work spaces (if this
is alright with the organization), or ask people to think about the last
time they did that really important activity and write a quick paragraph
about it so that you can take it away afterward.
- Remember to relax, and don't make promises about things you might be
able to fix.
- Will you have anybody from your organization and/or the client's
organization to do the observations with you? My best Contextual Inquiry
research has ended with a client doing the big presentation to their own
people. In my experience, clients buy into this type of user research almost
immediately. Invite clients and coworkers to the observations if possible,
and if not, have them help you with the analysis afterward. If even *that
* is not possible, then make sure that your analysis and modeling are big
and visible, so people are interested in what you've done.
- Do your initial analysis *immediately*. Even if you take copious notes,
the things people said are going to bleed together. I would type up or
formalize your notes on the plane ride home.

That's all I've got off the top of my head. Me and a coworker presented a
paper at Agile 2008 about this topic, and I think there's a lot more in
there about my experience doing a quick Contextual Inquiry. Here's the link
to the paper:
http://ieeexplore.ieee.org/Xplore/login.jsp?url=/stamp/stamp.jsp?arnumber=4599535&isnumber=4599440

Good luck!

Josh

On Mon, Feb 2, 2009 at 2:59 PM, Nicholas Iozzo <ixda at humansize.com> wrote:

> I have found it is generally better to open the day with a group
> meeting. No matter how much explaining you do ahead of time, you will
> still likely be scheduled to meet with the wrong folks.
>
> The group meeting will allow the managers to have their say and give
> them a forum to tell you many things. You can then use that to start
> asking specific task questions. Usually during the course of these
> questions, they will say "Well, Sue does that". Great time to say,
> "I'd love to spend some time with Sue later today then to see how
> she does it."
>
> I have found that during the course of this opening meeting, names of
> individuals get brought up as the person who does X. This is the best
> way for you to then select whom you want to meet with and learn more
> about task x.
>
> Of course, as the prior posted mentioned. Preparation is the most
> important thing to do. Know what you want to observe, know how many
> events you want to observe, have research questions you want
> answered. Prepare a study guide....
>
> During the course of the day, you will be presented with more
> opportunities then you have time to follow-up on. You need to have
> spent time developing your study guide so you can make on-the-fly
> decisions on how to best use your time.
>
> Even if your research techniques are all about not interfering with
> the user and letting it naturally flow. If you have not made
> decisions about what you want to learn, then you will not learn
> anything.
>
> This is more practical then ideal. Ideally you would have lots of
> time to spend with everyone, so you will be able to learn all you
> can. Practically, you have a very limited amount of time to spend
> with a limited number of folks. So you have to plan on how to use it
> wisely.
>
> Get clearance before you even bring out any recording devices. Many
> companies do not like it.
>
> Good luck.
>
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=38073
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

--
http://josh.ev9.org/weblog

Comments

2 Feb 2009 - 4:12pm
Katie Albers
2005

I feel compelled to reiterate my note-taking plea here: Don't do it!
Note-taking splits your attention and tends to change the behavior of
the subject. It's aurally, visually and actively intrusive.
Note-taking is evil.
Use a tape recorder or a web cam or a small video camera you can mount
on the cubicle wall and aim, or similar (depending on what behaviors
you're particularly studying) but if at all possible, don't take
notes. Much of what you would take notes on can be translated into
sound simply by asking questions. Then schedule time between sessions
when you can jot down your notes and aides de memoir.

I realize that contemporary note-taking is in some cases simply
unavoidable, but make sure that you really need to do it in this case
before automatically incorporating it.

kt

Katie Albers
Founder & Principal Consultant
FirstThought
User Experience Strategy & Project Management
310 356 7550
katie at firstthought.com

On Feb 2, 2009, at 1:54 PM, Josh Evnin wrote:

> To add to the great advice above:
>
> - Make sure the people you'll be observing are prepared for you to
> be
> there. They SHOULD NOT clear their schedules to be with you. They
> should
> have real, regular work to get done while you're with them and you
> need to
> set that expectation ahead of time.
> - An hour or two for each observation should be a good amount of
> time,
> but make sure that you're observing enough of the "important
> stuff." The
> "important stuff" is the same as the Big Questions Dana talks about.
> - I second Nicholas's idea of starting the day with a group
> meeting, but
> don't let this go too long. Maybe half an hour, maybe. One thing
> I've done
> in the past is give people some "homework" at this meeting. You
> can hand out
> disposable cameras for people to take photos of their work spaces
> (if this
> is alright with the organization), or ask people to think about
> the last
> time they did that really important activity and write a quick
> paragraph
> about it so that you can take it away afterward.
> - Remember to relax, and don't make promises about things you
> might be
> able to fix.
> - Will you have anybody from your organization and/or the client's
> organization to do the observations with you? My best Contextual
> Inquiry
> research has ended with a client doing the big presentation to
> their own
> people. In my experience, clients buy into this type of user
> research almost
> immediately. Invite clients and coworkers to the observations if
> possible,
> and if not, have them help you with the analysis afterward. If
> even *that
> * is not possible, then make sure that your analysis and modeling
> are big
> and visible, so people are interested in what you've done.
> - Do your initial analysis *immediately*. Even if you take copious
> notes,
> the things people said are going to bleed together. I would type
> up or
> formalize your notes on the plane ride home.
>
> That's all I've got off the top of my head. Me and a coworker
> presented a
> paper at Agile 2008 about this topic, and I think there's a lot more
> in
> there about my experience doing a quick Contextual Inquiry. Here's
> the link
> to the paper:
> http://ieeexplore.ieee.org/Xplore/login.jsp?url=/stamp/stamp.jsp?
> arnumber=4599535&isnumber=4599440
>
> Good luck!
>
> Josh
>
>
> On Mon, Feb 2, 2009 at 2:59 PM, Nicholas Iozzo <ixda at humansize.com>
> wrote:
>
>> I have found it is generally better to open the day with a group
>> meeting. No matter how much explaining you do ahead of time, you will
>> still likely be scheduled to meet with the wrong folks.
>>
>> The group meeting will allow the managers to have their say and give
>> them a forum to tell you many things. You can then use that to start
>> asking specific task questions. Usually during the course of these
>> questions, they will say "Well, Sue does that". Great time to say,
>> "I'd love to spend some time with Sue later today then to see how
>> she does it."
>>
>> I have found that during the course of this opening meeting, names of
>> individuals get brought up as the person who does X. This is the best
>> way for you to then select whom you want to meet with and learn more
>> about task x.
>>
>> Of course, as the prior posted mentioned. Preparation is the most
>> important thing to do. Know what you want to observe, know how many
>> events you want to observe, have research questions you want
>> answered. Prepare a study guide....
>>
>> During the course of the day, you will be presented with more
>> opportunities then you have time to follow-up on. You need to have
>> spent time developing your study guide so you can make on-the-fly
>> decisions on how to best use your time.
>>
>> Even if your research techniques are all about not interfering with
>> the user and letting it naturally flow. If you have not made
>> decisions about what you want to learn, then you will not learn
>> anything.
>>
>> This is more practical then ideal. Ideally you would have lots of
>> time to spend with everyone, so you will be able to learn all you
>> can. Practically, you have a very limited amount of time to spend
>> with a limited number of folks. So you have to plan on how to use it
>> wisely.
>>
>> Get clearance before you even bring out any recording devices. Many
>> companies do not like it.
>>
>> Good luck.
>>
>>
>>
>> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
>> Posted from the new ixda.org
>> http://www.ixda.org/discuss?post=38073
>>
>>
>> ________________________________________________________________
>> Welcome to the Interaction Design Association (IxDA)!
>> To post to this list ....... discuss at ixda.org
>> Unsubscribe ................ http://www.ixda.org/unsubscribe
>> List Guidelines ............ http://www.ixda.org/guidelines
>> List Help .................. http://www.ixda.org/help
>>
>
>
>
> --
> http://josh.ev9.org/weblog
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help

2 Feb 2009 - 6:09pm
Angel Marquez
2008

I think hidden cameras are a little on the evil side...
I just read a Koan about a Zen master that was dying and he gave his pupil
his writing and the pupil tossed it into the fire.

I think incognito when nobody knows your coming is the best approach.

Your cover is already blown people are going to be playing the part.

On Mon, Feb 2, 2009 at 4:00 PM, Will Evans <will at semanticfoundry.com> wrote:

> I love you guys and your polemics - what ever happened to "it depends..."
>
> Now it's Note taking is evil and
>
> Eye tracking is voodoo.
>
> RED, ACD, GDD, UCD: It all reminds me of the religious arguments people
> used to get into between kung-fu, aikido, aikijujitsu, shotokan, judo,
> wingchun, as to which "style" was the best/most effective/most versatile.
> only neophiles got in to those arguments. the masters never did :-) New
> Practitioners used so spend as much time discussing their art/style and
> often more time discussing, than doing. We had an old saying: Shut up and
> practice. Same with [insert TLA Silver bullet methodology here] - the
> process that can be spoken is not the ultimate process. Those who speak, do
> not know, and those who know, do not speak - the master shows by doing, all
> else is void and emptiness.
> ~ will
>
> "Where you innovate, how you innovate,
> and what you innovate are design problems"
>
>
> --------------------------------------------------------------------------------
> Will Evans | User Experience Architect
> tel: +1.617.281.1281 | will at semanticfoundry.com
> http://blog.semanticfoundry.com
> aim: semanticwill
> gtalk: semanticwill
> twitter: semanticwill
>
> --------------------------------------------------------------------------------
>
>
> On Feb 2, 2009, at 3:48 PM, Jared Spool wrote:
>
>
>> On Feb 2, 2009, at 2:12 PM, Katie Albers wrote:
>>
>> I feel compelled to reiterate my note-taking plea here: Don't do it!
>>> Note-taking splits your attention and tends to change the behavior of the
>>> subject. It's aurally, visually and actively intrusive.
>>> Note-taking is evil.
>>> Use a tape recorder or a web cam or a small video camera you can mount on
>>> the cubicle wall and aim, or similar (depending on what behaviors you're
>>> particularly studying) but if at all possible, don't take notes. Much of
>>> what you would take notes on can be translated into sound simply by asking
>>> questions. Then schedule time between sessions when you can jot down your
>>> notes and aides de memoir.
>>>
>>> I realize that contemporary note-taking is in some cases simply
>>> unavoidable, but make sure that you really need to do it in this case before
>>> automatically incorporating it.
>>>
>>
>> I'd argue that note taking is very valuable and, when properly done, very
>> important to both the observer and the participant.
>>
>> (As an aside: In this case, the "subject" is not the person you're
>> observing, it's the software you're studying. The formal name in
>> phenomenalogical ethnographic studies is "informant", but many of us just
>> use "participant". Or their first name, which feels less impersonal.)
>>
>> Trying to remember everything you see, especially in an 5 to 6 hour
>> session, also splits your attention. Tape recorders, web cams, and video
>> cameras change the participants behaviors as much, if not more, than note
>> taking.
>>
>> When I'm doing field studies, I prefer to take a small audio recorder.
>> (I'm in love with the Olympus LS-10, though we often use bulkier Marantz
>> PMD-660s.) However, I still take my trusted Moleskine large-size reporter's
>> notebook, for which I take most of my notes. I would not take notes on a
>> laptop or palm-sized keyboard.
>>
>> If you've never taken notes in a live interview before, I recommend you
>> practice it. It's a learned skill and practicing definitely improves it.
>> Rehearsing your site visit by watching fellow colleagues, taking notes, then
>> writing up your daily summary -- repeating that process a couple of times --
>> is a great way to work the kinks out and get some practice.
>>
>> Jared
>>
>> Jared M. Spool
>> User Interface Engineering
>> 510 Turnpike St., Suite 102, North Andover, MA 01845
>> e: jspool at uie.com p: +1 978 327 5561
>> http://uie.com Blog: http://uie.com/brainsparks Twitter: jmspool
>> UIE Web App Summit, 4/19-4/22: http://webappsummit.com
>> ________________________________________________________________
>> Welcome to the Interaction Design Association (IxDA)!
>> To post to this list ....... discuss at ixda.org
>> Unsubscribe ................ http://www.ixda.org/unsubscribe
>> List Guidelines ............ http://www.ixda.org/guidelines
>> List Help .................. http://www.ixda.org/help
>>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

2 Feb 2009 - 9:17pm
Carol J. Smith
2007

Technology can fail in a myriad of ways. Without note taking, you risk
losing the information you spend hours planning to collect.

I *always* recommend taking notes (in any form).

I take copious notes. I create a document to help me take notes for each
participant which includes the questions I need answered and plenty of space
for the notes. Depending on the situation, I also create forms to track
common events between participants.

I then shred, recycle the paper, and plant another tree in my yard for good
measure. ;)

Carol

-------------------
Carol J. Smith
Principal Consultant, Midwest Research, LLC
http://www.mw-research.com
LinkedIn: http://www.linkedin.com/in/caroljsmith

UPA 2009 International Conference: 8-12 June, 2009
http://www.usabilityprofessionals.org/conference/2009/

3 Feb 2009 - 10:30am
jabbett
2008

I like the approach of double-teaming each interview. One person
facilitates -- maintains eye contact, asks questions, nods, actually
listens to the conversation. The other takes notes and keeps track of
which research questions have been covered. We digitally record the
interview as well, which has been invaluable.

In my limited experience, my "poor note taking" has been a barrier
during the interview itself, especially if I'm interviewing alone. I
like reviewing the tape/transcript after the fact and taking notes
from there. I find it's a good way to reengage with my project, get
the mental juices flowing.

I think the take-away is to tailor your approach for your situation.

* How many people will attend the interview? (is there room for
assigning different interview roles?)
* Will it be mostly conversation (audio), or will it be following work
on a computer (video)?
* Do you have the time/resources for transcription?
* Do you have the time spread out interviews over days/weeks, so you
can adjust your technique each time?

-Jon

On Mon, Feb 2, 2009 at 10:17 PM, Carol Smith <carol at mw-research.com> wrote:
> Technology can fail in a myriad of ways. Without note taking, you risk
> losing the information you spend hours planning to collect.
>
> I *always* recommend taking notes (in any form).
>
> I take copious notes. I create a document to help me take notes for each
> participant which includes the questions I need answered and plenty of space
> for the notes. Depending on the situation, I also create forms to track
> common events between participants.
>
> I then shred, recycle the paper, and plant another tree in my yard for good
> measure. ;)
>
> Carol
>
> -------------------
> Carol J. Smith
> Principal Consultant, Midwest Research, LLC
> http://www.mw-research.com
> LinkedIn: http://www.linkedin.com/in/caroljsmith
>
> UPA 2009 International Conference: 8-12 June, 2009
> http://www.usabilityprofessionals.org/conference/2009/
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

3 Feb 2009 - 10:32am
jabbett
2008

> What you're describing is poor note taking practice. It takes no skills to
> do a crappy job at anything you put your mind to. (Damn. I say this so often
> that I've decided to call it Spool's First Law of Competency.)

Can you recommend any resources on how to take notes well during an interview?

-Jon

4 Feb 2009 - 3:52am
Roundand
2009

2009/2/3 Jared Spool <jspool at uie.com>

>
> That's a really good question. In my experience, it's a core competency of
> any journalism or ethnography program.
>
> I've seen good stuff over the years, but am hard pressed to put my finger
> on anything current. Years ago, we had Ellen Isaacs (a UX researcher who was
> a trained journalist) do a full-day seminar on the topic at the User
> Interface Conference, but she's disappeared from the UX world of late.
>
> Over the years, I've trained our researchers and dozens of clients. It's
> not hard, but there are definitely tricks and techniques to learn. And
> practice is really what makes it work.
>
> I'll research it some more and see if I can find something current.

I am trying to improve my life-long poor note-taking skills and would be be
very interested in your recommendations. I've currently trying to adopt
Cornell Notes (http://en.wikipedia.org/wiki/Cornell_Notes) which
is definitely an improvement on what I've been doing so far.

4 Feb 2009 - 4:26am
Harry Brignull
2004

I vaguely recall reading that Microsoft OneNote can be used to record audio
and binds it with your notes as you write them. When you are viewing them
afterwards, you can double click on any line of text, and it will jump the
audio recording to that point.

Has anyone tried it, and is it any good for taking notes in interviews? Are
there any other apps that do this?

Harry

4 Feb 2009 - 9:29pm
aschechterman
2004

Oh-oh, I draw pictures.
= : ^ )

::::

Andrew Schechterman PhD

LinkedIn: http://www.linkedin.com/in/andrewschechterman

E-mail: aschechterman at gmail.com

Phone: 1-303-886-2440

:::::

On Wed, Feb 4, 2009 at 3:26 AM, Harry <harrybr at gmail.com> wrote:

> I vaguely recall reading that Microsoft OneNote can be used to record audio
> and binds it with your notes as you write them. When you are viewing them
> afterwards, you can double click on any line of text, and it will jump the
> audio recording to that point.
>
> Has anyone tried it, and is it any good for taking notes in interviews? Are
> there any other apps that do this?
>
> Harry
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

4 Feb 2009 - 11:40pm
Roundand
2009

2009/2/5 Jared Spool <jspool at uie.com>

>
> Taking notes during interviewing is a different set of skills, because (as
> Katie aptly pointed out) you need to stay present in the moment and be
> working on the dialogue with the participant *while* you're making the
> notes.
>
> The best resources are going to either be from journalism or ethnography.
> Studying note taking techniques won't work as well, nor will note taking
> techniques used for interrogation (practiced by police). They are different
> animals.
>

Now that's a different perspective - is there a story or two there? And can
you recommend any resources short of taking a journalism or ethnography
course?

5 Feb 2009 - 1:30am
Chan FoongYeen
2008

Sometime I will do mind mapping. There are tools like Mind Manager that able
to facilitate the creation of mind map quickly.

cheers,
CHAN

On Thu, Feb 5, 2009 at 11:29 AM, A S <aschechterman at gmail.com> wrote:

> Oh-oh, I draw pictures.
> = : ^ )
>
> ::::
>
>
>
> Andrew Schechterman PhD
>
> LinkedIn: http://www.linkedin.com/in/andrewschechterman
>
> E-mail: aschechterman at gmail.com
>
> Phone: 1-303-886-2440
>
>
>
> :::::
>
>
> On Wed, Feb 4, 2009 at 3:26 AM, Harry <harrybr at gmail.com> wrote:
>
> > I vaguely recall reading that Microsoft OneNote can be used to record
> audio
> > and binds it with your notes as you write them. When you are viewing them
> > afterwards, you can double click on any line of text, and it will jump
> the
> > audio recording to that point.
> >
> > Has anyone tried it, and is it any good for taking notes in interviews?
> Are
> > there any other apps that do this?
> >
> > Harry
> > ________________________________________________________________
> > Welcome to the Interaction Design Association (IxDA)!
> > To post to this list ....... discuss at ixda.org
> > Unsubscribe ................ http://www.ixda.org/unsubscribe
> > List Guidelines ............ http://www.ixda.org/guidelines
> > List Help .................. http://www.ixda.org/help
> >
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

Syndicate content Get the feed