Living the Job >> Enterprise UX Research by Doing (vs. Observation)

21 Jan 2009 - 5:55pm
5 years ago
23 replies
1124 reads
Julian Orr
2009

Hi,

Does anyone out there have the experience of actually performing a given job
(for at least a day or three, perhaps longer) as a means of really
researching context, tasks etc.? Specifically, I am thinking of an
enterprise context, where the user doesn't have choice in tools, workflow
and there are some highly developed skills (ie more than the basic web
skills of an e-commerce user). Also, I am contrasting this approach to
on-site observation, empathic modeling and user role playing.

For example, working in a call center as a first line telephone customer
care agent. Sitting down with call center agent, getting some basic
training and having that person watch your back to prevent major
catastrophes, You answer calls, use the system(s) to retrieve and enter
information etc., essentially it is you performing the job.

This was something I though of proposing ages ago when I wanted to analyze
and model the work of a particular type of system analyst. It never came to
fruition (due mainly to technical skills gap, but also legal issues with
outsider using systems) and I ended up doing standard contextual
observations. It was great for insight into high-level aspects of the
product and job that had issues (most of which we were already aware of) but
not much nuance.

It is inspired by a story I heard (circa 2001?) about a financial analyst
getting a job at an Amazon.com warehouse as a means of gauging their
likelihood of hitting/exceeding their numbers.

There are a myriad of reasons not to do this, namely resource/time
constraints, but I am curious to see if other IxDers, particularly those
with a research bent have experience with this and could provide some input
on how it compares to CI. Of course input from people with no experience
is also welcome.

What context was this performed in? (Real vs. Realistic Simulation)

Did you have some basic, prior understanding of the domain?

Did you do training?

What did you call it? (methodology)

Was it disruptive to work setting?

Does it provide a level of insight worth the time/hassle of setting it up?

Cheers,
Julian

Comments

22 Jan 2009 - 3:08am
Steve Baty
2009

Julian,

My first impression is that this is a fantastic idea! There's nothing like
actually *doing* a job to understand what it's like for the person. Just
remember that your perspective - as someone who'll get to walk away at the
end of the day(s) or week - will still be somewhat different from the person
who makes their living from it.

I've never actually undertaken this type of research; only gone so far as
contextual inquiry and observation. The only time I've done something
similar is by undertaking job interviews to learn how better to conduct them
for my own company :)

Regards
Steve

2009/1/22 Julez <hujano at gmail.com>

> Hi,
>
> Does anyone out there have the experience of actually performing a given
> job
> (for at least a day or three, perhaps longer) as a means of really
> researching context, tasks etc.? Specifically, I am thinking of an
> enterprise context, where the user doesn't have choice in tools, workflow
> and there are some highly developed skills (ie more than the basic web
> skills of an e-commerce user). Also, I am contrasting this approach to
> on-site observation, empathic modeling and user role playing.
>
> For example, working in a call center as a first line telephone customer
> care agent. Sitting down with call center agent, getting some basic
> training and having that person watch your back to prevent major
> catastrophes, You answer calls, use the system(s) to retrieve and enter
> information etc., essentially it is you performing the job.
>
> This was something I though of proposing ages ago when I wanted to analyze
> and model the work of a particular type of system analyst. It never came to
> fruition (due mainly to technical skills gap, but also legal issues with
> outsider using systems) and I ended up doing standard contextual
> observations. It was great for insight into high-level aspects of the
> product and job that had issues (most of which we were already aware of)
> but
> not much nuance.
>
> It is inspired by a story I heard (circa 2001?) about a financial analyst
> getting a job at an Amazon.com warehouse as a means of gauging their
> likelihood of hitting/exceeding their numbers.
>
> There are a myriad of reasons not to do this, namely resource/time
> constraints, but I am curious to see if other IxDers, particularly those
> with a research bent have experience with this and could provide some input
> on how it compares to CI. Of course input from people with no experience
> is also welcome.
>
> What context was this performed in? (Real vs. Realistic Simulation)
>
> Did you have some basic, prior understanding of the domain?
>
> Did you do training?
>
> What did you call it? (methodology)
>
> Was it disruptive to work setting?
>
> Does it provide a level of insight worth the time/hassle of setting it up?
>
>
> Cheers,
> Julian
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

--
Steve 'Doc' Baty | Principal | Meld Consulting | P: +61 417 061 292 | E:
stevebaty at meld.com.au | Twitter: docbaty | LinkedIn:
www.linkedin.com/in/stevebaty

Blog: http://docholdsfourth.blogspot.com
Contributor - UXMatters - www.uxmatters.com
UX Book Club: http://uxbookclub.org/ - Read, discuss, connect.

22 Jan 2009 - 9:22am
Joshua Ayers
2009

I've never heard of anyone doing this, but I have to agree that it is
an excellent idea. Having personas to help develop applications is one
thing, but to actually take on the role of a potential user is even
better!

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=37492

22 Jan 2009 - 11:00am
christine chastain
2008

Hi all,

Participant observation is well documented in anthropological literature so
I may have a look there. There are, of course, issues with this methodology,
like all others. Mainly, they are of the ethical kind - how not to influence
the observed, whether or not to let them know that you are participating in
order to observe them or not, whether you should interfere in interactions
that are uncomfortable for you or unethical in your eyes (domestic violence,
drug dealing, etc.), etc. Just a heads up because I have seen, in my own
work, that these issues do have an influence on outcomes.

Also, self-reporting techniques are useful in conjunction with participant
observation. That way, you can be a bit more sure that it isn't your
perception of participation that feeds the analysis.

Cheers,
Christine

On Thu, Jan 22, 2009 at 8:22 AM, Joshua Ayers <ayers.joshua at gmail.com>wrote:

> I've never heard of anyone doing this, but I have to agree that it is
> an excellent idea. Having personas to help develop applications is one
> thing, but to actually take on the role of a potential user is even
> better!
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=37492
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

22 Jan 2009 - 11:35am
Robert Hoekman, Jr.
2005

>
> Does anyone out there have the experience of actually performing a given
> job
> (for at least a day or three, perhaps longer) as a means of really
> researching context, tasks etc.?

I've done this a number of times. Once, for example, I learned from a waiter
the proper process for serving wine in a restaurant for an eLearning course
I was developing. I've always thought (and said out loud) that becoming
proficient in the activities you're trying to support with design is one of
the best ways to step into the mind of the user.

Did you have some basic, prior understanding of the domain?

Not at all. I don't like wine. :)

What did you call it? (methodology)
>

I think the #1 reason this approach hasn't become more prevalent is the lack
of a good name. It needs a name! It's not contextual inquiry, obviously, but
it feels like a form of contextual inquiry. Something like "contextual
participation" sounds accurate, but it also sounds ... stuffy. It should
sound as fun as it really is.

Was it disruptive to work setting?

Researchers do this all the time — Cialdini and his team, for example,
actually got jobs as salespeople and such as part of their research on
persuasion. Of course, they were completely undercover rather than entering
a supervised experience. (Perhaps even more beneficial!)

Does it provide a level of insight worth the time/hassle of setting it up?

Undoubtedly.

-r-

22 Jan 2009 - 12:58pm
leofrish
2007

Yes, I have done this on several occasions, most notably when I was building an automation application for emergency managers.

In that gig, our entire management team went through 6 months of training (off and on, not continuously!) shoulder to shoulder with our future customers/users learning the job and getting on the job experience.

It was the only way we could

1) identify the real problems these individuals faced in performing the specific tasks we were hoping to automate
2) establish the sort of empathy required to give us credibility in a very tight-knit (almost military like camaraderie) community

During some of the work sessions, we sought permission (and were usually granted) to video tape what was going on. These records were invaluable for forensic purposes and to communicate to our staff the kinds of issues our stakeholders faced.

The literature in embedded studies of this type is long and well established ( I think Christine raised some of the issues with this approach ).

In our case we were very upfront about our intentions: "we are hear to participate, pull our weight and be a part of the real work going. At the same time, we are a for-profit company ultimately interested in creating a product that we will likely resell to folks such as yourselves. Your allowing us to participate will significantly improve the chances the product will be usable and useful. And because of the time you've made for us, we will likely provide a discount not only to your agency, but to all similar public agencies."

We embedded ourselves in at least three different contexts over the course of about 1 year.

We believe this approach was a significant contributor to the usability and usefulness of the product: This product was designed and delivered in 1997. I just got a note from a user who wants to continue using it but misplaced his original disks.

Leo

22 Jan 2009 - 12:59pm
Marijke Rijsberman
2004

>>I've done this a number of times. Once, for example, I learned from a
waiter
>>the proper process for serving wine in a restaurant for an eLearning
course
>>I was developing. I've always thought (and said out loud) that becoming
>>proficient in the activities you're trying to support with design is one
of
>>the best ways to step into the mind of the user.

Point extremely well taken. However, learning from a waiter how to serve
wine is not the same as being a waiter for a number of evenings--which is
what Julian asked about.

I actually don't think it is a very practical idea, if tantalizing, to learn
what you need to know by actually doing the job. To really "get" what it is
like to do a certain job by living it, you need to have the background that
you could be hired for it, you'd have to go through the proper training, and
you'd have to do it long enough to get past the apprentice stage and
experience proficiency--and such things as the boredom that comes from
repetitiveness, the resentment that comes from being closely
supervised/criticized/harassed by customers, the anxiety that comes from the
possibility that you will be laid off, and so on.

Julian used the example of call center reps. It may be a lowly job, but for
many of these lowly call center jobs there's extensive training. If you
support health insurance, for example, you start with some 8 weeks of
training before you are allowed to take a call on your own. If you support
software, you're usually also in training for a number of weeks. Often, it
is not a matter of being able to work with "the system," but learning the
product/service, learning to be comfortable using the phones, the customer
database for logging the calls, and a variety of look-up tools (digital and
paper-based) to find answers to the less common questions, learning the
policies for handling complaints, and learning to talk and listen to the
customer while typing your notes.

While I would learn this stuff in much greater depth by doing it myself, I
don't believe that designing the right tools in the right way for the person
doing the work requires that level of depth. Simply watching a variety of
people do the work and listening to them talk about it can deliver what you
need to know, at least while exercising your imagination and practicing some
humility.

Besides, I have a nagging notion that there's an important piece in here
somewhere about acknowledging the expertise of people who do the work. Would
we ourselves think that someone would "get" our own jobs from doing it for a
day (or three)?

marijke

22 Jan 2009 - 1:32pm
Robert Hoekman, Jr.
2005

>
> Point extremely well taken. However, learning from a waiter how to serve
> wine is not the same as being a waiter for a number of evenings--which is
> what Julian asked about.

Not a great example, obviously, but I thought of it because it was one of
the more unusual cases. Other times, I have actually done the job. For
example, I actually did the work of a call center employee while working on
an intranet app that would mostly benefit the call center.

Compare the two approaches ...

1. You watch a call center employee jump from window to window in three
different applications, use the company website to look up promotional info,
IM his boss to get the answer to an approval question, take lunch orders
from the people around him because it's his turn to make the food run, and
somehow pay attention to the person on the phone and deliver great customer
service at the same time.

OR

2. You actually do all these things.

I assure you, it looks and feels very different when you're the one with the
headset on, trying to pay attention to all these things at once.

I actually don't think it is a very practical idea, if tantalizing, to learn
> what you need to know by actually doing the job. To really "get" what it is
> like to do a certain job by living it, you need to have the background that
> you could be hired for it, you'd have to go through the proper training,
> and
> you'd have to do it long enough to get past the apprentice stage and
> experience proficiency--and such things as the boredom that comes from
> repetitiveness, the resentment that comes from being closely
> supervised/criticized/harassed by customers, the anxiety that comes from
> the
> possibility that you will be laid off, and so on.

And this "doing the job" approach is somehow less practical than typical
contextual inquiry? You learn far more doing the job, even as a "tourist",
than you do by merely observing others. Of course I don't think someone
would "get" interaction design by doing my job for a few days, but here's a
short list of things they would find out very quickly:

1. Wireframe software is seriously lacking, and we end up designing some of
the same things over and over again.
2. Video conferencing is not what it should be, and using a phone to handle
client calls is ridiculous because it limits your ability to use all tools
you could be using while on the phone. Sure, there's Skype, but not everyone
uses that.
3. It takes a lot of discipline to avoid constantly being interrupted
(email, phone, IM, etc.). It can be just as hard to get into the zone while
working at home as in the typical office.
4. No matter how big my monitor is, it's not big enough.
5. Daily admin stuff is a huge time sucker.
6. There is a constant pursuit of new ways to get and stay organized so that
real work can get done.

You could observe me for a few days and learn these things, but if you
actually go through the motions, you understand and internalize the pain
points better than you ever could by simply watching them.

Sure, in some cases, the chore of getting up to speed enough to even be able
to do the job, even as a tourist, can be too time-consuming and intensive to
justify, but when you can do it, you'll learn a whole lot of things just by
going through the act of doing the job, even if you're not qualified to do
it. You'll see what situations you have to negotiate as you perform tasks,
what decisions have to be made at each step, what actually goes through your
mind.

-r-

22 Jan 2009 - 1:51pm
Bruce Esrig
2006

We did a sort of participatory observation when I was developing
documentation and training.

Depending on the situation, we would have actual field technicians, a
top-tier technical-support person, the documentation and training team, and
sometimes a video crew.

There were times when the documentation and training people picked up
wrenches and tightened nuts. That's how I found out that using two nuts
together makes a really solid way to secure a bolt.

We also had a chance to participate in a field trial at a real customer
site. The top-tier technical-support person learned how to tie down cables
with waxed cord. After that, he campaigned passionately for plastic wire
ties.

Even if you don't get exactly the right information from the experience, the
experience of having the experience makes the situation much more vivid, and
enables you to have conversations with the people who do it that you just
wouldn't have otherwise.

We found out why cabinets need to be vertical, where the slop is, and why
certain things take so incredibly long. As a result, we were able to write
better procedures. Eventually, we started creating simple reference cards
that replaced entire training courses. It was only possible because we knew
what the field technicians could do on their own and what they needed help
with.

Best wishes,

Bruce Esrig
Madison, NJ

22 Jan 2009 - 1:58pm
Dana Chisnell
2008

What context was this performed in? (Real vs. Realistic Simulation)

Poll worker on Election Day.

Did you have some basic, prior understanding of the domain?

I had voted before, but didn't know a lot about the entire election
administration process.

Did you do training?

I took the same 2 hours of training that all poll workers are
required to take in my county.

What did you call it? (methodology)

I didn't call it anything.

Was it disruptive to work setting?

No. I just acted in the role of poll worker in all the situations
required while I was there. Then I went home and wrote up notes. I
took pictures during the training and on Election Day as allowed and
appropriate.

Does it provide a level of insight worth the time/hassle of setting it
up?

WAAAYYY better than just asking someone else. It's so different
experiencing it for yourself. The boredom, the stress, the confusion,
the interactions with other people. There's no way you're going to get
that just by interviewing or even just observing each of the
particular events.

DAna

:: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: :: ::
Dana Chisnell
desk: 415.392.0776
mobile: 415.519.1148

dana AT usabilityworks DOT net

www.usabilityworks.net
http://usabilitytestinghowto.blogspot.com/

On Jan 21, 2009, at 2:55 PM, Julez wrote:

>
> What context was this performed in? (Real vs. Realistic Simulation)
>
> Did you have some basic, prior understanding of the domain?
>
> Did you do training?
>
> What did you call it? (methodology)
>
> Was it disruptive to work setting?
>
> Does it provide a level of insight worth the time/hassle of setting
> it up?

22 Jan 2009 - 1:59pm
Kate Walser
2007

Call it "walk in their shoes testing" or WITST for short (kidding on
acronym). It can work in insurance and telecom customer service rep
(CSR) scenarios with some caveats in stream of consciousness order...

*Parallel set-ups / Realistic simulation
Given the legalities and unacceptable possibility that we'd annoy or
lose real customers, we created a "parallel" environment, with a
dummy phone that listened in on the conversation but was on
perma-mute, and a view of the system and data, but didn't alter the
customer's record permanently. (if you're working with an
insurance/telecom company, they may have a training set-up like this
already)

It's not exactly the same as being on the front line, but it's
close enough to feel the anxiety and panic if the customer needs
something and you can't quickly find the options / info on the
screen / product. (*from my experience, this is the key piece that
helps it provide value over a CI - there will always be things that a
real CSR / person forgets when they're being "watched" - if you
have a chance to walk in their shoes, you can experience some of the
things they forgot to tell you)

*Be ready to sign an NDA (may want to proactively offer it, if
you're an outside consultant)
Companies protect their customers and their privacy voraciously, like
a mother bear with her cubs. How they train CSRs and handle customer
service is a bit proprietary as it can be their 'edge' over the
competition. So on all counts, expect they'll want a non-disclosure
agreement.

*Basic training
You do need some training, similar to what the CSRs would receive on
starting the job. Given high attrition in some of these industries,
that was sufficient to simulate a good portion of the workforce when
we did the studies.

*Can't simulate attitudes / motivation (or can you?)
Marijke's point is well-taken - it's hard to simulate the true
motivations and reality of someone for whom this *is* the job day in
and day out. I suppose in the CSR scenario, you could potentially
simulate a CSR's state of mind if you have colleagues give you a
warm-up by yelling at you for something over which you have no
control, much like a CSR on the front lines gets from real customers.

*Can be expensive
It can be expensive to set up the listen-in phone line, though I
suspect now it might be cheaper given Skype and phone conferencing
abilities.

*May need to forewarn customers
Your client may still want to forewarn customers that to improve
customer service, etc., during a certain week, they may have another
observer. (Some companies already have a message to this effect -
"for training purposes, your call may be monitored/recorded")

*Have someone be your notetaker / recorder
It's not realistic to record or remember your thoughts/reactions in
this situation, so either record yourself (including both parts of
the phone conversation if possible) or have someone observing you and
recording notes. Even though you won't get everything that goes
through your mind at that point, you'll have enough cues to remember
from recordings.

*Allocate time every so often (e.g., every 2 calls or every hour,
etc.) to capture your thoughts, reactions

*More than one day may be excessive
If you're working with high volume situations, several days could be
excessive - may be better to do one whole day, or do a couple half
days, again, with time every so often to capture thoughts /
reactions.

*Record the real CSR's reactions, if possible
Really worthwhile to compare the real CSR to the newbie / simulated
CSR's experience and see how they differ. You can't perfectly
attribute what you did to what a newbie would do, again given
motivations and different backgrounds, but interesting to compare
nonetheless.

*Let CSRs watch you!
If you really want CSRs to appreciate what you do (and bring humor
back into their lives), let the client invite some of the off-duty
people in to observe, and even jot down their notes on all the things
they notice that are wrong as you "experience" what they do and the
system they must use. Could be eye-opening.

Robert makes a valid point about all the "aside" tasks that a CSR /
user may have (e.g., being their turn for lunch orders, etc.) - in
some cases (i.e., small companies, tech companies, etc.) those are
legitimate situations. In big companies, that's less the case and
there may even be heavily restricted areas for CSRs where they can
focus solely on customer service and handling as many calls as
needed.

Good luck!

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=37492

22 Jan 2009 - 3:00pm
Marijke Rijsberman
2004

I'll eat a few of my words: Dana came up with an example of a job it is
practical to do whole hog--take the training and experience the work all the
way up to proficiency.

Most of the other examples involve experiencing what it is like to be a
novice.

Perhaps there are cases where it is fundamentally different to feel the
panic of the novice confronted with the full complexity of challenges in a
stressful environment, rather than imagining it. But you still have an
extremely partial view.

After a few days of the simulated novice experience, you'd still need to
observe and talk to people to know what it is like to be proficient, as well
as to understand the range of variation, the collaborations that happen
between people, to know what is truly routine and what kinds of weirdnesses
take place outside of the routine.

Of course it's good to get as close to the experience as you can get, but
it's easy to imagine you know what it's like just because you did it for a
day.

marijke

22 Jan 2009 - 5:18pm
Robert Hoekman, Jr.
2005

>
> But you still have an extremely partial view.
>
>
> Hence my use of the word "tourist". I realize, as I'm sure everyone else
> does, that this approach does not make you an expert, but it's certainly no
> worse than observation alone. I won't know what it's like to live in Dubai
> by visiting there for a few days, but I'll certainly have a much clearer
> picture of life there than I would get by interviewing residents, reading
> travel books, and watching documentaries. I may do all of these things as
> part of my research, in fact, but visiting there will give me the best
> firsthand insights by far.
>
> After a few days of the simulated novice experience, you'd still need to
>> observe and talk to people to know what it is like to be proficient
>
>
> In many cases, if not most, you'd be doing this the whole time. Working
> alongside the experts gives you a chance to bond with them in a way that
> observing ("Pretend I'm not here!") doesn't.
>
> -r-
>

22 Jan 2009 - 9:13pm
Jared M. Spool
2003

On Jan 21, 2009, at 2:55 PM, Julez wrote:

> Does anyone out there have the experience of actually performing a
> given job
> (for at least a day or three, perhaps longer) as a means of really
> researching context, tasks etc.? Specifically, I am thinking of an
> enterprise context, where the user doesn't have choice in tools,
> workflow
> and there are some highly developed skills (ie more than the basic web
> skills of an e-commerce user). Also, I am contrasting this approach
> to
> on-site observation, empathic modeling and user role playing.

It's an excellent technique to have in the toolbox.

However, there are a few things that nobody's mentioned yet:

1) It's not exclusive with observation or role playing. All of those
techniques are useful for different reasons and could be useful on the
same project. For example, you could train and work as a CSR for a few
days, then observe other CSRs to see what their workstyles and
challenges are (and how they were were different from yours), then
role play through new design solutions given what you experienced and
observed.

2) Your experience of learning and doing will probably only match that
of someone coming new into a position. However, in many jobs,
experience introduces subtleties and nuances that new folks can't see.
As I hinted above, you'll still want to observe more seasoned people
to get access to those parts of the context.

3) There are many jobs where you can't do this. For example, in the
financial services industry, you have to pass certifications before
you can answer the phones in certain contexts. Here, observation is
your only option.

4) When you're doing, it's hard to take detailed notes about your
experiences. You rely on your memory to retain much of what you
experienced. Again, observation is going to give you real advantage.

So, I'd recommend you pair this activity with observation of others.
By doing them together, I think you'll gain much greater insight than
doing them separately. I think there's a lot of synergy.

Hope that helps,

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks Twitter: jmspool
UIE Web App Summit, 4/19-4/22: http://webappsummit.com

23 Jan 2009 - 12:52pm
Michael Micheletti
2006

I've done this twice. Once to redesign a CRM system for a technical support
group and another time to design a content management application for
technical writers. Each time I basically joined the group for some time,
taking support calls or writing technical bulletins under the guidance of
the group's manager. I went to staff meetings and got good advice from my
coworkers. After I walked the walk for a while, I had a much better
understanding of what was important or not.

Both groups were very accepting of me and both projects were successful and
well-received.

In subsequent design projects I've tried to get similar experience, but the
domains generally didn't permit this. I'm also a big fan of participatory
design, where one or two real live users are on the design team - that's
worked well in projects too.

Michael Micheletti

On Wed, Jan 21, 2009 at 2:55 PM, Julez <hujano at gmail.com> wrote:

> Hi,
>
> Does anyone out there have the experience of actually performing a given
> job
> (for at least a day or three, perhaps longer) as a means of really
> researching context, tasks etc.? Specifically, I am thinking of an
> enterprise context, where the user doesn't have choice in tools, workflow
> and there are some highly developed skills (ie more than the basic web
> skills of an e-commerce user). Also, I am contrasting this approach to
> on-site observation, empathic modeling and user role playing.
>
> For example, working in a call center as a first line telephone customer
> care agent. Sitting down with call center agent, getting some basic
> training and having that person watch your back to prevent major
> catastrophes, You answer calls, use the system(s) to retrieve and enter
> information etc., essentially it is you performing the job.
>
> This was something I though of proposing ages ago when I wanted to analyze
> and model the work of a particular type of system analyst. It never came to
> fruition (due mainly to technical skills gap, but also legal issues with
> outsider using systems) and I ended up doing standard contextual
> observations. It was great for insight into high-level aspects of the
> product and job that had issues (most of which we were already aware of)
> but
> not much nuance.
>
> It is inspired by a story I heard (circa 2001?) about a financial analyst
> getting a job at an Amazon.com warehouse as a means of gauging their
> likelihood of hitting/exceeding their numbers.
>
> There are a myriad of reasons not to do this, namely resource/time
> constraints, but I am curious to see if other IxDers, particularly those
> with a research bent have experience with this and could provide some input
> on how it compares to CI. Of course input from people with no experience
> is also welcome.
>
> What context was this performed in? (Real vs. Realistic Simulation)
>
> Did you have some basic, prior understanding of the domain?
>
> Did you do training?
>
> What did you call it? (methodology)
>
> Was it disruptive to work setting?
>
> Does it provide a level of insight worth the time/hassle of setting it up?
>
>
> Cheers,
> Julian
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

23 Jan 2009 - 2:38pm
Gavin Burke
2008

With music software a lot of the people who design the software spend
their spare time using it to make music. A mix of both user/designer.
This can have its benefits as you are constantly real world testing
and refining the interface as you go through the development process.

> On Jan 21, 2009, at 2:55 PM, Julez wrote:
>
>> Does anyone out there have the experience of actually performing a
>> given job
>> (for at least a day or three, perhaps longer) as a means of really
>> researching context, tasks etc.? Specifically, I am thinking of an
>> enterprise context, where the user doesn't have choice in tools,
>> workflow
>> and there are some highly developed skills (ie more than the basic
>> web
>> skills of an e-commerce user). Also, I am contrasting this
>> approach to
>> on-site observation, empathic modeling and user role playing.
>

23 Jan 2009 - 7:04pm
Robert Hoekman, Jr.
2005

>
> I've often described how I do what I for a living as "wearing someone
> else's hat" fir a while.
>
> That could be the name. OR...
>

I crowdsourced for names on Twitter over the last couple of days, and many
people continually tried to latch onto terms that already mean other things.
David Malouf, for example, advocated hard for "ethnography", but ethnography
is like sushi — it represents a type, not a specific thing. Lots of people
know what ethnography is, but far fewer seem to know about this particular
approach to doing research. I think it needs a name of its own so that we
can reference it in conversations without having to explain it ("By
ethnography, I mean I actually did the job of a call center employee for a
few days.")

So, out of all that, I started thinking that since "contextual inquiry"
basically means going into someone's context and, well, inquiring, then
"contextual participation" could be an appropriate name, because this
technique means going into someone's context and actually, well,
participating.

Eh?

-r-

23 Jan 2009 - 7:27pm
Sharon Greenfield5
2008

I like 'ethnography', but feel it requires a specification.
eg: 'design ethnography'

On Jan 23, 2009, at 4:04 PM, Robert Hoekman Jr wrote:

>>
>> I've often described how I do what I for a living as "wearing
>> someone
>> else's hat" fir a while.
>>
>> That could be the name. OR...
>>
>
> I crowdsourced for names on Twitter over the last couple of days,
> and many
> people continually tried to latch onto terms that already mean other
> things.
> David Malouf, for example, advocated hard for "ethnography", but
> ethnography
> is like sushi — it represents a type, not a specific thing. Lots of
> people
> know what ethnography is, but far fewer seem to know about this
> particular
> approach to doing research. I think it needs a name of its own so
> that we
> can reference it in conversations without having to explain it ("By
> ethnography, I mean I actually did the job of a call center employee
> for a
> few days.")

24 Jan 2009 - 12:42am
Jared M. Spool
2003

On Jan 23, 2009, at 11:38 AM, gavin burke|FAW wrote:

> With music software a lot of the people who design the software
> spend their spare time using it to make music. A mix of both user/
> designer.
> This can have its benefits as you are constantly real world testing
> and refining the interface as you go through the development process.

I would argue that there is a difference between a designer who is
also a trained musician (creating software for themselves to make
music with) from a designer who isn't a trained musician trying to sit
in their user's shoes for a while.

The first is more Self Design (as I described here: http://is.gd/gLAQ)
versus conducting more encompassing user research. Self design has its
place, but it shouldn't be confused with research.

Jared

24 Jan 2009 - 1:06am
Tanya Rabourn
2004

On Jan 23, 2009, at 6:04 PM, Robert Hoekman Jr wrote:

>> So, out of all that, I started thinking that since "contextual
>> inquiry"
> basically means going into someone's context and, well, inquiring,
> then
> "contextual participation" could be an appropriate name, because this
> technique means going into someone's context and actually, well,
> participating.

Why does acting as a "participant-observer" not work for your
purposes? (Don't confuse it with "observing participants" or some
such.) It's a really well-known social research method. The results
don't have to be qualitative data but they usually are. It can involve
ethnography or not. The key thing is that you are a full participant
in the activity (and qualified to perform it). A lot has been written
about the difficulties with it. You have to immerse yourself in the
activity and the group yet somehow during analysis remove yourself
from the activity and "make it strange" again.

-Tanya

24 Jan 2009 - 11:07am
Chauncey Wilson
2007

A good overview of participant observation can be found in the book
"Participant Observation" by James P. Spradley. The book discusses
different levels of participation and the ethical and data collection and
interpretation issues that emerge with differing levels of
participation. The book describe a general procedure for participant
observation starting out with how to locate the observation situation, how
to choose the level of participation, how to make description observations,
how to make domain analyses and focused observations, how to take a cultural
inventory, and how to write up the overal results of the participant
observation study.

Spradley, J. P. (1980). Participant observation. Fort Worth: Harcourt Brace
College Publishers.

Another small handbook (one of the Sage series on methods) that deals with
overt and participant observation is:

Jorgensen, D. L. (1989). Participant observation: A methodology for human
studies. Sage Publications.

As part of one of my jobs, I had to listen to a day's worth of support calls
with a member of the support team and when "how to" questions came up on on
a product I was working on, the tech support person would bring me into the
call. It was very difficult and some people were outraged and you had to
calm them down first. I gained a tremendous amount of respect for tech
support as a result of a few days of being on the line with them.

Chauncey

On Thu, Jan 22, 2009 at 11:00 AM, christine chastain <
chastain.christine at gmail.com> wrote:

> Hi all,
>
> Participant observation is well documented in anthropological literature so
> I may have a look there. There are, of course, issues with this
> methodology,
> like all others. Mainly, they are of the ethical kind - how not to
> influence
> the observed, whether or not to let them know that you are participating in
> order to observe them or not, whether you should interfere in interactions
> that are uncomfortable for you or unethical in your eyes (domestic
> violence,
> drug dealing, etc.), etc. Just a heads up because I have seen, in my own
> work, that these issues do have an influence on outcomes.
>
> Also, self-reporting techniques are useful in conjunction with participant
> observation. That way, you can be a bit more sure that it isn't your
> perception of participation that feeds the analysis.
>
> Cheers,
> Christine
>
> On Thu, Jan 22, 2009 at 8:22 AM, Joshua Ayers <ayers.joshua at gmail.com
> >wrote:
>
> > I've never heard of anyone doing this, but I have to agree that it is
> > an excellent idea. Having personas to help develop applications is one
> > thing, but to actually take on the role of a potential user is even
> > better!
> >
> >
> > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> > Posted from the new ixda.org
> > http://www.ixda.org/discuss?post=37492
> >
> >
> > ________________________________________________________________
> > Welcome to the Interaction Design Association (IxDA)!
> > To post to this list ....... discuss at ixda.org
> > Unsubscribe ................ http://www.ixda.org/unsubscribe
> > List Guidelines ............ http://www.ixda.org/guidelines
> > List Help .................. http://www.ixda.org/help
> >
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

23 Jan 2009 - 4:43pm
Julian Orr
2009

Wow, thank you everyone for the great input. Stoked to see that some people
have had a chance to actually do this.

>From what I have read so far I can summarize the following;

- Is a valuable option to have in the toolkit, as a supplemental activity to
observation and as a way to mix it up.
- Works well when its possible (ie you are working internal to the company
vs. as a consultant)
- Not practical for all Design work or specific jobs, (911 dispatch is out)
- Doing the training (or portion thereof) required for the job/tool/app is a
good way to accelerate this, especially for highly skilled domains.
(Genetics)
- If you can't actually do the job you can play with the tools (or
instruments)
-Length and intensity of stint will influence your perspective, but you will
almost always be stuck with the perspective of a newbie and thus be obliged
to validate your findings with the old hands.
-There is still not a generally recognized name for this approach, but I do
like"Walk in Their Shoes" and "Contextual Participation" (clear to UX folks
if no-one else) .
-Hard to collect data while doing (does it makes sense to do this? Is there
value in channelling your experience directly into design?[with
justifications] or should it be mediated by data first?)

HOW DOES EXPERIENCE DOING THE JOB AFFECT DESIGN/PROCESS?
Being somewhat research oriented and seeing all the discussion on the
academic thread I'd be curious to see the outcome of a design
pseudo-experiment. (Ahem, calling Design Profs) Basically, have a 2-4 design
teams go head to head on a specific design brief. 2 teams perform
observation only vs. 2 teams that have a mix of training/doing with some
team members observing.

Obviously there are too many variables to eliminate here, but I am curious
to see how doing the job "might" affect the design outcome.

Will the designers who did the job have more empathy and understanding? Will
it reflect in the design? Will performing the job influence acceptance of
subsequent design by the users?

Cheers,
Julian

24 Jan 2009 - 4:28pm
Angel Marquez
2008

I just had a gig as a blu-ray programmer. It was all xml, java, ant, rake,
ruby, using linux commands. No prior experience. Their were different client
specs, player compatibility issues, design integration, localization,
different 3rd party involvement (encoders, muxing houses).
I recreated 2 existing titles using a new build system. I just took a UX
class so I employed the process that was taught in tandem.
1. I did user research found out from all parties what worked, what didn't
and why.
2. I audited the current process.
3. Conducted the ethnographic research on myself and the other programmers.
4. I made a proposal for a process that was modular and addressed the
different parties involvement. I researched other existing
solutions, sketched it, refined it, made wireframes, did a feature value
matrix, created a back end framework and submitted it with a UX brief.

>>What context was this performed in? (Real vs. Realistic Simulation)
Both. I recreated two existing titles and then created a demo. More
integration than programming.

>>Did you have some basic, prior understanding of the domain?
None. It was similar to any interactive platform. A place for everything and
everything in its place.

>>Did you do training?
No. I had to figure a lot of it out myself.

>>What did you call it? (methodology)
Never ending HACK.

>>Was it disruptive to work setting?
No. I just incorporated it along with what I was doing.

>>Does it provide a level of insight worth the time/hassle of setting it up?
It would have eliminated all known issues with a minimal learning curve,
completely scalable, and tied into all aspects of the development cycle PM,
DESIGN, PROGRAMMING, QA etc...

24 Jan 2009 - 6:17pm
whitneyq
2010

Another method I've both heard about and (briefly) used is to get a
person already doing the job to teach you how to do it.

Judy Ramey has a case study about working with healthcare
professionals reading a large set of medical images. She ran into two
problems: first, they worked so quickly that she wasn't able to follow
their actions without some explanation; second, they could not stop
and talk while they were working (both due to time demands of the job,
and because it interrupted their workflow). To address this, she:

- Received some basic training on how to "read" an image. This
primarily involved simply learning how to move through the process:
mount them for viewing, the systemmatic viewing of details, etc. This
did not, of course, make her a medical expert, but it let her have a
good feeling for the mechanics.
- Video taped some work sessions and then viewed them with the HCP in
a retrospective. In some cases, I think they actually got out the
images again for a "replay".

One concern about using "self-discovery" methods to learn about a job
is that you need to be careful that you don't assume that everyone's
reaction is the same as yours. Participation can give you a first-hand
view and can be an excellent way to learn the mechanics. But, it's
just one view (even if a well-informed and well-documented one).

Someone mentioned issues of covert vs. overt participant observation.
There's no particular reason why this exercise should be covert, in
most cases. In fact, there can be some advantages to being "one of
them" and actually trying to learn the real job.

Cognetics had a project some years ago for a utility company
(regulated, unionized workforce). We were the third company to try to
design a hand-held device for their field service folks. The first
thing the person working on the project did was a day of
"ride-alongs". There wasn't much he would be allowed to do, but just
going through an entire day provided a lot of insights. The real
pay-off however, was when the new design was reviewed with some users.
One of the ride-along participants stood up to say "These guys took
the time to really find out what we do out there."

I'm constantly hearing that user research takes too much time, and so
we should find ways to work around this problem. Perhaps the question
we ought to be asking in each situation is exactly what we need to
know, and decide on the most efficient way to learn it.

--
Whitney Quesenbery
www.wqusability.com

Storytelling for User Experience Design
www.rosenfeldmedia.com/books/storytelling

Syndicate content Get the feed