Usability Reports: A waste of time?

22 Aug 2009 - 11:44pm
5 years ago
11 replies
696 reads
Mashhoor Aldubayan
2009

Hi,

I'm wondering: how many of you think that writing a big report on
the findings/recommendations for a project is inefficient?

>From my experience, it seems that I end up discussing (and often
justifying) almost every single thing I write on the report, no
matter how logical it is.

Isn't a better way to just sit with the client's development team
and/or management, and get things done together?

I know Steve Krug is against reports, but I wanted to know what other
professionals' take on this matter.

Comments

24 Aug 2009 - 12:08am
jstanford
2003

I personally like summary reports (not huge reports but a nice manageable
size) for several reason:

- When I am designing, I go back to the findings/recommendations and makes
sure that I have addressed all of them
- In fact, sometimes some of the findings/recommendations are small
and not worth discussing in the meeting so it's nice to have them all
written down to make sure that the designer addresses them

- I refer back to reports I have written a while ago all the time to refresh
my memory on what happened on a specific project (prevents rediscovering the
wheel over and over again)

- When presenting the findings, I need a way to gather my thoughts for the
preso so I use the report for organization

The key is not having some giant unmanageable report in a crazy detailed
format but using a format that doesn't create a lot of busy work and is
instead helpful for analysis.

Julie

-----Original Message-----
From: discuss-bounces at lists.interactiondesigners.com
[mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of
Mashhoor Aldubayan
Sent: Saturday, August 22, 2009 10:44 PM
To: discuss at ixda.org
Subject: [IxDA Discuss] Usability Reports: A waste of time?

Hi,

I'm wondering: how many of you think that writing a big report on
the findings/recommendations for a project is inefficient?

>From my experience, it seems that I end up discussing (and often
justifying) almost every single thing I write on the report, no
matter how logical it is.

Isn't a better way to just sit with the client's development team
and/or management, and get things done together?

I know Steve Krug is against reports, but I wanted to know what other
professionals' take on this matter.

________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... discuss at ixda.org
Unsubscribe ................ http://www.ixda.org/unsubscribe
List Guidelines ............ http://www.ixda.org/guidelines
List Help .................. http://www.ixda.org/help

24 Aug 2009 - 5:02am
Adrian Howard
2005

On 22 Aug 2009, at 22:44, Mashhoor Aldubayan wrote:

> I'm wondering: how many of you think that writing a big report on
> the findings/recommendations for a project is inefficient?
>
>> From my experience, it seems that I end up discussing (and often
> justifying) almost every single thing I write on the report, no
> matter how logical it is.
>
> Isn't a better way to just sit with the client's development team
> and/or management, and get things done together?

Yes. I've always found this the most effective technique to actually
get changes made.

However - some clients won't let you do this without there being a big
thick wad of paper with twenty seven eight-by-ten colour glossy
photographs with circles
and arrows and a paragraph on the back of each one.

I try and get some idea of client expectations before I write the
report. I also charge for writing the report separately - so they can
what time/money they are wasting before they waste it. Beyond that -
if they still ask for it I write it.

Adrian
--
http://quietstars.com - twitter.com/adrianh - delicious.com/adrianh

24 Aug 2009 - 2:32pm
Stephen Collins
2009

On Sun, Aug 23, 2009 at 8:44 AM, Mashhoor Aldubayan<mashhoor.d at gmail.com> wrote:
> I'm wondering: how many of you think that writing a big report on
> the findings/recommendations for a project is inefficient?

Absolutely! But sometimes, it's the right way to approach the problem.
Let me explain...

> >From my experience, it seems that I end up discussing (and often
> justifying) almost every single thing I write on the report, no
> matter how logical it is.

This is often the case, and it can be incredibly frustrating. But if
the client is one of those big, bureaucracy-driven organisations,
sometimes the report is what they, and particularly senior management
who will end up okaying the recommendations you make, understand.

> Isn't a better way to just sit with the client's development team
> and/or management, and get things done together?

Definitely with the team. Especially if you can be embedded with them
and influence change on a day-to-day basis.

Again, management culture. Not every client will cope with the notion
of small, incremental change, in their development where it's not
justified or documented. I've seen this often in clients I work with,
especially those that have very bureaucratic management structures,
are very risk averse or who have strict change management models.

> I know Steve Krug is against reports, but I wanted to know what other
> professionals' take on this matter.

Given my 'druthers I'd do it Steve's way every time...

Pick your battles. Win the ones you can. Sometimes, writing a report
isn't one of them.

Steve C
--
Stephen Collins
trib at acidlabs.org
+61 410 680722
@trib

www.acidlabs.org

acidlabs
Conversation. Collaboration. Community.

This email is: [ ] bloggable [X] ask first [ ] private

25 Aug 2009 - 1:59am
Angela Arnold
2009

In my experience, usability reports go largely unread. The business
wants to know the quick wins as well as the longer term issues, but
they need to be easy to digest, engaging and prioritised: filter out
the stuff people dont need to know right now. As a UX designer
working in an agile environment, I try to save as much time as
possible, by taking screenshots to sessions for my observers to
annotate. I take developers, BAs and business owners to the sessions
to give them an overview of the main issues. I bring the screenshots
back into work and get them straight onto the wall. Whilst the
feedback is clear in my mind, I stick large index cards up next to
the screens highlighting the major issues, and include some quotes
from users. My main objective is to engage the development team
enough to read these snapshots. We always record our sessions on
video, so if we need to look back or really highlight something, a
quick video showcase can provide compelling evidence. These screens
provide a quick and easy way to communicate with the team and give a
quick round up of the major issues. They can also stay up on the wall
and act as a reference to decisions made earlier in the project.
Meanwhile I can go about implementing change rather than trawling
through video and audio material and writing long reports.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=44960

25 Aug 2009 - 10:37am
Abhay Rautela
2008

Mashoor, I'm guessing that you make a report and then mail/ hand it out.
*If* that's the case, then there's a simple solution. The issue with
preparing a report and mailing it out to stakeholders is that there is a
slim chance that it will be read by all. Taking print outs and personally
handing them over to each of them doesn't help either.

If you want to make sure all stakeholders not only go through it entirely,
but also understand well what's in it , then try shifting to presentating
the report. You can keep the deliverable as a doc if you like. I would
recommend changing to a presentation format instead since it's better suited
for presentation. As Angela mentioned, filter out the stuff they dont want
to know about. Post report creation, try and figure out with them what is it
that they want to see in there and what is it that they don't. By all means,
suggest a structure for the report that you think would work well with them,
but then modify it according to how they want it.

Once you are ready your presentation, send out a meeting request and make
sure you get all or as many stakeholders as you can to attend. As opposed to
probably only a few stakeholders reading the report and making limited sense
out of what is in it, the presentation ensures that:
1) Everybody goes through the report, from beginning to the end, since
you're presenting it to them.
2) They can make better sense of what the report contains since you're there
to discuss the details and address their queries right away.
3) There is better appreciation of your research findings and
recommendations. Equally important, you get to equally appreciate why it is
that they can not incorporate certain recommendations, what ever the reasons
may be- conflict with business goals, time/ technical constraints?

By the end of it all, everybody walks out with a better understanding of the
report and a better understanding of what should be done next. Hope this
helps.

I've mentioned some tips on creating usability test reports besides others
in this presentation:
http://www.slideshare.net/ConeTrees/tips-for-effective-usability-testing-in-india.
This is in context to India but I would think most of the tips should
come
in handy regardless of where you are located.

- Abhay
--
Cone Trees- User Research & Design
http://www.conetrees.com
http://www.twitter.com/conetrees
http://www.theuxbookmark.com
http://uxbookclub.org/doku.php?id=new_delhi
http://www.slideshare.net/group/web-accessibility

On Tue, Aug 25, 2009 at 6:29 AM, Angela Arnold <aarnold14 at googlemail.com>wrote:

> In my experience, usability reports go largely unread. The business
> wants to know the quick wins as well as the longer term issues, but
> they need to be easy to digest, engaging and prioritised: filter out
> the stuff people dont need to know right now. As a UX designer
> working in an agile environment, I try to save as much time as
> possible, by taking screenshots to sessions for my observers to
> annotate. I take developers, BAs and business owners to the sessions
> to give them an overview of the main issues. I bring the screenshots
> back into work and get them straight onto the wall. Whilst the
> feedback is clear in my mind, I stick large index cards up next to
> the screens highlighting the major issues, and include some quotes
> from users. My main objective is to engage the development team
> enough to read these snapshots. We always record our sessions on
> video, so if we need to look back or really highlight something, a
> quick video showcase can provide compelling evidence. These screens
> provide a quick and easy way to communicate with the team and give a
> quick round up of the major issues. They can also stay up on the wall
> and act as a reference to decisions made earlier in the project.
> Meanwhile I can go about implementing change rather than trawling
> through video and audio material and writing long reports.
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=44960
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

25 Aug 2009 - 2:02pm
Paul Bryan
2008

The scope and contents of a usability report should be tailored to
reflect the organizational context in which it is sponsored and
produced.

If you are internal to the organization, and the organization is
small, then I think a bullet list of recommended changes that can be
discussed in person will probably be more effective than a report. In
the event that you are working as a team member in an Agile
environment, then I think the report needs to be tailored to the
local flavor of Agile, and scheduled to conclude one cycle ahead of
development, so that it can be immediately digested and acted upon.

If you are external to the organization and it is very large, then I
think a written report of findings and recommendations can be very
useful:

1. The report is a way for you to fully explain the design direction
and changes you are advocating, so that the people who don't agree
with you will need to prepare a good case for ignoring your
recommendations.

2. Stakeholders who have a vested interest in the results but who
can't or won't sit down with you to discuss your findings can
understand the reasoning behind the changes you're recommending.

3. Third parties who get involved further down the road have a
concise, logical presentation of factors that influence the success
of the design.

4. Site owners can wave a large, weighty, well-designed document as
justification for doing what they wanted to do before you wrote the
report. Frustrating, but it happens, and I can't say that I would
turn down a project tomorrow even if knew ahead of time that was
going to happen. Why? Because I found out that this happened with a
very large client; but then a couple of years later I learned that
subsequent people had picked it up and got a lot of value from the
report.

I always try to scope in a brief user interview along with usability
testing, so that a study's findings are more generally applicable to
a customer's interaction with the company's interactive offering in
general, as well as their response to the specific design in
question. This makes the study useful long after that particular
design iteration has come and gone, because it uses specific results
to address concepts that will remain relevant. For example, in a
usability test I might find that certain types of users have
difficulty undertanding how to pair a mobile device with a carrier
plan. I will recommend a way to fix the particular design that was
tested so that users will be successful when it launches, but I will
also frame the users' challenge more generally so that the design
continually evolves to address this fundamental customer issue more
effectively.

/pb

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=44960

25 Aug 2009 - 2:48pm
Amy Jones
2009

I'll just add that nothing I've done has ever been more persuasive to a
dev team than producing a highlight reel. Watching one person struggle
with an interaction is one thing, watching 5 users have the same problem
in 5 1 minute clips is quite another.

The change in the way the work and the recommendations are perceived is
really amazing.

--Amy

-----Original Message-----
From: discuss-bounces at lists.interactiondesigners.com
[mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of
paul bryan
Sent: Tuesday, August 25, 2009 8:03 AM
To: discuss at ixda.org
Subject: Re: [IxDA Discuss] Usability Reports: A waste of time?

The scope and contents of a usability report should be tailored to
reflect the organizational context in which it is sponsored and
produced.

If you are internal to the organization, and the organization is
small, then I think a bullet list of recommended changes that can be
discussed in person will probably be more effective than a report. In
the event that you are working as a team member in an Agile
environment, then I think the report needs to be tailored to the
local flavor of Agile, and scheduled to conclude one cycle ahead of
development, so that it can be immediately digested and acted upon.

If you are external to the organization and it is very large, then I
think a written report of findings and recommendations can be very
useful:

1. The report is a way for you to fully explain the design direction
and changes you are advocating, so that the people who don't agree
with you will need to prepare a good case for ignoring your
recommendations.

2. Stakeholders who have a vested interest in the results but who
can't or won't sit down with you to discuss your findings can
understand the reasoning behind the changes you're recommending.

3. Third parties who get involved further down the road have a
concise, logical presentation of factors that influence the success
of the design.

4. Site owners can wave a large, weighty, well-designed document as
justification for doing what they wanted to do before you wrote the
report. Frustrating, but it happens, and I can't say that I would
turn down a project tomorrow even if knew ahead of time that was
going to happen. Why? Because I found out that this happened with a
very large client; but then a couple of years later I learned that
subsequent people had picked it up and got a lot of value from the
report.

I always try to scope in a brief user interview along with usability
testing, so that a study's findings are more generally applicable to
a customer's interaction with the company's interactive offering in
general, as well as their response to the specific design in
question. This makes the study useful long after that particular
design iteration has come and gone, because it uses specific results
to address concepts that will remain relevant. For example, in a
usability test I might find that certain types of users have
difficulty undertanding how to pair a mobile device with a carrier
plan. I will recommend a way to fix the particular design that was
tested so that users will be successful when it launches, but I will
also frame the users' challenge more generally so that the design
continually evolves to address this fundamental customer issue more
effectively.

/pb

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=44960

________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... discuss at ixda.org
Unsubscribe ................ http://www.ixda.org/unsubscribe
List Guidelines ............ http://www.ixda.org/guidelines
List Help .................. http://www.ixda.org/help

25 Aug 2009 - 7:54am
lkruger
2009

I agree that it largely depends on the client personality and their
expectations. I work with non-profits and the report is often
something they need to justify the costs. That being said, I realize
that all relevant stakeholders won't read a long, drawn-out report.
My preference is to do the report to document everything and then
separately, do a few slides to summarize the major highlights and
talk through them with the client.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=44960

26 Aug 2009 - 1:12am
Ali Naqvi
2008

"Isn't a better way to just sit with the client's development team
and/or management, and get things done together?"

When I worked as a Usability Logger during my studies, my task was to
ensure that the usability Reports were correct. I also asked for the
development team to be present, but at times they simply don't have
time to participate.
The reports also "remind" you of the problems being addressed. I
see the usability reports as useful.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=44960

26 Aug 2009 - 2:15am
PhillipW
2009

I don't actually like finishing reports very much - but having to
write them does force one to actually think long and hard about
what's going on.

Any many is the time about three quarters of the way through writing
the report I've suddenly thought 'Aha, so that's how its supposed
to work !'...

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=44960

26 Aug 2009 - 10:08am
Adrian Howard
2005

On 25 Aug 2009, at 21:48, Amy Jones wrote:

> I'll just add that nothing I've done has ever been more persuasive
> to a
> dev team than producing a highlight reel. Watching one person
> struggle
> with an interaction is one thing, watching 5 users have the same
> problem
> in 5 1 minute clips is quite another.
>
> The change in the way the work and the recommendations are perceived
> is
> really amazing.

I've found one thing as (maybe more) effective than that - having them
in the room during the testing (similar to what Jared Spool talks
about in these articles http://is.gd/2A2ha).

Got some really lovely insights/feedback from everybody post-test.

Would do it again.

Adrian
--
http://quietstars.com - twitter.com/adrianh - delicious.com/adrianh

Syndicate content Get the feed