The Black Art of Usability

8 Aug 2007 - 6:46pm
6 years ago
15 replies
633 reads
David Armano
2007

One of my favorite whitepapers is The Essence of Effective Rich Internet Applications by Kevin Mullet of Macromedia (2003). It provides a nice POV relevant to UX, Design and IA related disciplines. One thing that always stuck with me was this portion of it:

“Usability has always been the most difficult quality to instill in software design. It sits at the
junction of form and function to constrain and inform the design of both aspects of the
product. While still viewed as a black art by some, there is a substantial body of formal
knowledge in psychology and human factors and a long history of accepted practice in the
design disciplines that can guide the creation of usable products.“

While most of us would probably agree that making something usable isn’t easy—I’m curious how this group feels about about the “black art” part of the statement. Do you think that usability is perceived as a “black art” by some? If so—why? If not—why?

Respectfully,

David Armano
http://darmano.typepad.com

Comments

8 Aug 2007 - 7:43pm
Josh
2006

I would go further and say that software development (web or desktop) is
considered a black art by most people not directly involved in the process.

I don't know if I would say that usability as a practice is a black art,
though measuring the usability of a product probably could be. The standard
practice for setting up, running and analyzing usability tests is fairly
straightforward. Usability practitioners and user advocates have been doing
quite a bit of work to reinforce the idea that doing some usability testing
is always better than doing none, and that a usability test can be as simple
as two people, a webcam, and a paper-protoype.

The difficulty may very well stem from the fact that most of what we
consider standard usability testing is a matter of gathering qualitative not
quantitative data. Qualitative data can be problematic because it is seen as
subjective. In my experience, the user quotes garnered from frustrated users
during usability tests tend to be some of the biggest drivers of business
decisions post-analysis.

I am a huge fan of combining qualitative usability testing with quantitative
testing where the usability testing helps determine which
designs/interactions get launched and A/B or multi-variant quantitative
testing is used to optimize. In my experience, very few companies pay
adequate attention to optimizing features after they've been launched.

To me, the key question seems to be: How do companies gather the information
they need in order to make the right product decisions for their business
and for their customers?

--
Josh Viney
EastMedia Group
Company http://www.eastmedia.com
Blog http://www.kungpowthinking.com

8 Aug 2007 - 8:28pm
Steven Pautz
2006

On 8/8/07, David Armano <darmano at earthlink.net> wrote:
>
> I'm curious how this group feels about about the "black art" part of the
> statement. Do you think that usability is perceived as a "black art" by
> some? If so—why? If not—why?

I can't speak for industry, but I've met quite a few academics (both
professors and grad students, across several fields and universities but
most outspoken in Human Factors Psychology) who view usability engineering /
'classic' UCD as 100% scientific and objective beyond reproach; and any
alternative approaches or techniques as "unscientific", "wishy washy", and
"uninformed". I've never heard the phrase "black art" specifically, but I
imagine many of those academics would agree with its use (if applied to
anything outside of formal usability engineering). They would probably NOT
agree with its use as applied to their own work or beliefs.

In my personal opinion, 'classic' usability/UCD (as used by people like
that) *IS* very "black art", simply because (in the cases I've seen) all of
the observations, assumptions, inferences, and recommendations are
internalized and subjective, yet are treated as objective and absolute --
much like a magic spell known only by a special group of wizards, only the
usability authority is able to perform the secret incantation to deliver the
desired effect.

The actual body of formal knowledge in psychology and human factors is also
very "black art"-ish, I think, because it's up to the designer or evaluator
to determine which parts to use. Nearly every possible alternative design,
inference, or recommendation can be supported by (or contested by)
*something* in the literature. Presenting one's subjective judgment about
what to reference, as if it were an objective judgment, makes it very
difficult for people outside of the field/process to view the work as
anything but a black art, I believe.

To me, many of the methods and approaches from IxD/UX/etc are very
"art-like" (in that there's no single correct answer, etc), but they are
much more open, transparent, and welcoming/inviting of outsiders so they
don't have the "black art" aspects of elitism, secrecy, or deception. I can
see how an outsider might still view it as a black art, but unlike the more
authoritarian behavior surrounding (some of) the 'classic' usability
professionals, I don't think there's anything intrinsic to, for example,
personas or wireframing or strategic thinking which makes it a "black art"
rather than merely "art-like".

Just my 2 cents,
Steven Pautz
now-former HF Psych grad student

9 Aug 2007 - 12:30am
Deepak Pakhare
2005

On 8/9/07, Steven Pautz <spautz at gmail.com> wrote:

> In my personal opinion, 'classic' usability/UCD (as used by people like
> that) *IS* very "black art", simply because (in the cases I've seen) all
> of
> the observations, assumptions, inferences, and recommendations are
> internalized and subjective, yet are treated as objective and absolute --
> much like a magic spell known only by a special group of wizards, only the
> usability authority is able to perform the secret incantation to deliver
> the
> desired effect.

Usability=black art, IxD/UX=art-like is a very interesting take and I agree
to a large extent in spite of it being a bit strongly-worded. Having said
that, would you agree that IxDA/UX=art-like is a perception only within the
IxDA/UX community when outsiders view the IxDA community as a newer
"black-artish" sect going about brewing their sinister magic potion of
personas, wireframes etc. but not sinister enough since IxD/UX approaches
are so much more open-minded, inclusive. And because it doesn't seem all
that scientific, business-executive-led-design-committees with obvious clout
typically gnaw bit by bit at prototypes an IxDer has created with his bare
hands, until they get what they want coz a certain "market-segment" wants it
that way. Personas be damned, just pretty things up, okay?

Maybe it pays to be a usability guru and deliberately appear more exalted,
formidable and "scientific" when dealing with tech-management gurus. They
speak similar languages. ROI, "institutionalizing usability", so on and so
forth.

But yes, I would rather be artist-like and dirty my hands doing and re-doing
prototypes than be a classic usability professional and practice black-art.
But that's just me ;)

9 Aug 2007 - 4:53am
John Wood
2005

I have encountered quite a few people, often programmers, who view usability
as a bit of a black art. On reflection, I think this perception stems from
what appear to be arbitrary pronouncements by usability experts that seem to
contradict what the programmer heard from the same expert last week or what
they read about the subject previously. Why does this happen?

First, let's consider those people in the field who are essentially
incompetent, hiding their confusion behind jargon and an air of authority.
Their usability work is necessarily arbitrary and we won't consider this
factor further, except to say that incompetence may be responsible to some
extent for the perception of the field as a black art.

Incompetence aside, the source of this perception seems to me to be the
inherent complexity and context sensitivity of the problem domain - human
behaviour. Designing for people is not easy because people are not simple
systems and everything depends on what a person is doing, for what reason in
what context.

Now we all reduce this complexity in some way, whether using qualitative or
quantitative models. Both approaches are simplifications of a very complex
reality, and we need to use them with full awareness of their limitations.
Those who smugly privilege quantitative approaches over qualitative
approaches should bear in mind the layers of abstraction and
operationalisation of concepts that support their definitive pronouncements
on usability.

To an outsider, the process of considering this complexity and context
sensitivity may seem essentially arbitrary. This perception is exacerbated
when practitioners choose to adopt the sort of absolute certainty and
authoritative tone popularised by some usability gurus. This is tough to get
around, because if you go too far the other way and make the complexity too
apparent, you can end up sounding uncertain and undermine your client's
confidence in your work.

We need a happy medium in the way we present our work that makes it clear
that while this is not physics, it's not a roll of the dice either. This
tone is easier to strike when you are confident enough in your practice of
the field that you can leave aside dogma and see the challenges, tools and
solutions we provide for what they are. Imperfect (but often excellent)
solutions in an immensely complex problem domain.

John Wood

____
Senior Analyst, iQ Content Ltd.

________________________________________________
Message sent using UebiMiau 2.7.10

9 Aug 2007 - 6:17am
Jens Ulferts
2007

I would like to build on what John says, because I think he has a very valid point there. There is the perceived arbitrariness and the actual subjectivity which can not be avoided since uability is people oriented.

Other disciplines learn to disregard influences from people as subjective. They get fed this way of thinking during their education and work.

This is aggrevated by the fact that even the vocabulary is hard to learn for one new to the subject and that clear definitions, which is a center of programming, are hard to find. So usability specialists use terms differently. True, this also happens in the programming world but not as much.

So to warp it up I would not say usability is considered a black art. This has a negative connotation which I don't think programmers associate with it. I would say programmers consider it more as "alchemy", less scientific, and are hence very sceptical regarding the results.

Jens

-------- Original-Nachricht --------
Datum: Thu, 9 Aug 2007 10:53:06 +0100
Von: "John.wood" <john.wood at iqcontent.com>
An: discuss at ixda.org
Betreff: Re: [IxDA Discuss] The Black Art of Usability

> I have encountered quite a few people, often programmers, who view
> usability
> as a bit of a black art. On reflection, I think this perception stems from
> what appear to be arbitrary pronouncements by usability experts that seem
> to
> contradict what the programmer heard from the same expert last week or
> what
> they read about the subject previously. Why does this happen?
>
> First, let's consider those people in the field who are essentially
> incompetent, hiding their confusion behind jargon and an air of authority.
> Their usability work is necessarily arbitrary and we won't consider this
> factor further, except to say that incompetence may be responsible to some
> extent for the perception of the field as a black art.
>
> Incompetence aside, the source of this perception seems to me to be the
> inherent complexity and context sensitivity of the problem domain - human
> behaviour. Designing for people is not easy because people are not simple
> systems and everything depends on what a person is doing, for what reason
> in
> what context.
>
> Now we all reduce this complexity in some way, whether using qualitative
> or
> quantitative models. Both approaches are simplifications of a very complex
> reality, and we need to use them with full awareness of their limitations.
> Those who smugly privilege quantitative approaches over qualitative
> approaches should bear in mind the layers of abstraction and
> operationalisation of concepts that support their definitive
> pronouncements
> on usability.
>
> To an outsider, the process of considering this complexity and context
> sensitivity may seem essentially arbitrary. This perception is exacerbated
> when practitioners choose to adopt the sort of absolute certainty and
> authoritative tone popularised by some usability gurus. This is tough to
> get
> around, because if you go too far the other way and make the complexity
> too
> apparent, you can end up sounding uncertain and undermine your client's
> confidence in your work.
>
> We need a happy medium in the way we present our work that makes it clear
> that while this is not physics, it's not a roll of the dice either. This
> tone is easier to strike when you are confident enough in your practice of
> the field that you can leave aside dogma and see the challenges, tools and
> solutions we provide for what they are. Imperfect (but often excellent)
> solutions in an immensely complex problem domain.
>
>
> John Wood
>
> ____
> Senior Analyst, iQ Content Ltd.
>
> ________________________________________________
> Message sent using UebiMiau 2.7.10
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> List Guidelines ............ http://beta.ixda.org/guidelines
> List Help .................. http://beta.ixda.org/help
> Unsubscribe ................ http://beta.ixda.org/unsubscribe
> Questions .................. list at ixda.org
> Home ....................... http://beta.ixda.org

--
Psssst! Schon vom neuen GMX MultiMessenger gehört?
Der kanns mit allen: http://www.gmx.net/de/go/multimessenger

9 Aug 2007 - 8:42am
Todd Warfel
2003

On Aug 8, 2007, at 8:43 PM, Josh Viney wrote:

> [...] The difficulty may very well stem from the fact that most of
> what we consider standard usability testing is a matter of
> gathering qualitative not quantitative data. Qualitative data can
> be problematic because it is seen as subjective. In my experience,
> the user quotes garnered from frustrated users during usability
> tests tend to be some of the biggest drivers of business decisions
> post-analysis.

Let me preface this by saying that I do both qualitative and
quantitative research. And I recommend using multiple methods when
doing any kind of research to normalize data and reduce bias.

At Messagefirst, usability for us is more than just testing, it's a(n
exploratory) research method. As such, we incorporate exploratory
activities into our usability research. Sure, we run contextual (real
life) scenarios. But we don't make the customer go through the script
in the order we put it together. We let them use the product(s) as
they would natively, following their natural flow. This can be a bit
tricky to follow from a moderation standpoint, but eventually you get
used to it and find that the information you gather is much more
valuable and real.

In order to reduce the "subjective" nature of quantitative research,
we typically have more than one researcher recording information (one
w/the participant, one remotely from another room) and use a scoring
system that measures time, effort, and participant satisfaction.
While theoretically not as statistically sound as quantitative
research, there are several advantages to doing this type of
quantitative research:

1. Quantitative tells you how and what, but now why. Qualitative
tells you why and that's why we like it. Once you understand the way,
you can use that data to make more informed data driven design
decisions.

2. Qualitative is generally more contextual, more real world, less
theoretical.

In addition, quantitative can never achieve true statistical
significance anyway. Most of the quantitative focused researchers I
know complain that qualitative research can never get you (true)
statistical significance. They will argue that quantitative is the
only way to get true statistical significance and this is the
argument they use against qualitative research.

Well, the fact is that neither can quantitative. Both qualitative and
quantitative are a representative sample of a larger population -
quantitative is just a larger sample. Every sampling method includes
some bias - every one. Neither method includes the entire population
- with very few exceptions, sampling the entire population isn't
realistically feasible.

Either way, statistical significance is generally seen as having less
than a 5% chance that it isn't true. Odds are that if it has a 95%
chance that it is true, it's true. But 5% can still be significant
(Apple has less than 5% market share...).

True statistical significance relies on a truly random sampling. But
since every sampling method has bias, then truly achieving
statistical significance is realistically impossible.

> I am a huge fan of combining qualitative usability testing with
> quantitative testing where the usability testing helps determine
> which designs/interactions get launched and A/B or multi-variant
> quantitative testing is used to optimize. In my experience, very
> few companies pay adequate attention to optimizing features after
> they've been launched.

I whole-heartedly agree.

> To me, the key question seems to be: How do companies gather the
> information they need in order to make the right product decisions
> for their business and for their customers?

That should be the key question. I wish more people would get this.

Cheers!

Todd Zaki Warfel
President, Design Researcher
Messagefirst | Designing Information. Beautifully.
----------------------------------
Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
----------------------------------
In theory, theory and practice are the same.
In practice, they are not.

9 Aug 2007 - 1:17pm
David Armano
2007

Lots of good thoughts here across the board. I feel like some general
themes are that the perception may be there despite the notion that
it shouldn't. But I also think the notion of "black art" extends
beyond usability. There's a whole vernacular associated with the
world of UX%u2014a set of techniques that sometimes do conjure up
images of priests and priestesses mixing mysterious potions to create
the "perfect user experience".

It maybe symptomatic to the fact that the art or science of making
something simple is usually not easy%u2014therefore the techniques
may seem "foreign". So in the end, maybe it's a PR issue with
those outside of the discipline.

Appreciate the thoughts here.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://beta.ixda.org/discuss?post=19104

9 Aug 2007 - 2:03pm
dmitryn
2004

David,

With all respect, I'm not sure this can be written off as just a "PR
issue with those outside of the discipline".

The fact is, there is often no consensus among experienced UX
practitioners on basic questions like "what does term X mean" or "is
technique Y applicable in context Z" (a phenomenon that can be
witnessed on an almost daily basis on this list).

Given that lack of consensus, it's not surprising that UX is
occasionally perceived as a "black art". It's actually more surprising
that not everyone outside the field feels that way.

Dmitry

On 8/9/07, David Armano <darmano at earthlink.net> wrote:
> Lots of good thoughts here across the board. I feel like some general
> themes are that the perception may be there despite the notion that
> it shouldn't. But I also think the notion of "black art" extends
> beyond usability. There's a whole vernacular associated with the
> world of UX%u2014a set of techniques that sometimes do conjure up
> images of priests and priestesses mixing mysterious potions to create
> the "perfect user experience".
>
> It maybe symptomatic to the fact that the art or science of making
> something simple is usually not easy%u2014therefore the techniques
> may seem "foreign". So in the end, maybe it's a PR issue with
> those outside of the discipline.
>
> Appreciate the thoughts here.
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://beta.ixda.org/discuss?post=19104
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> List Guidelines ............ http://beta.ixda.org/guidelines
> List Help .................. http://beta.ixda.org/help
> Unsubscribe ................ http://beta.ixda.org/unsubscribe
> Questions .................. list at ixda.org
> Home ....................... http://beta.ixda.org
>

9 Aug 2007 - 2:23pm
Josh
2006

The art of creating anything can easily be considered a black art by the
uninitiated. The drawbacks associated with usability as a black art are that
the value of the service is misunderstood and the results are considered
impossible to reproduce by anyone who's not a wizard, guru, or Jakob
Nielsen.

We should work to remove the "black" from the art so that we're seen as
adding value not costing money. We should then attempt to convey that, even
though our work takes talent, we also produce real things for real people.
The term "craft" comes to mind.

I don't know if we need to get into discussions about trying to reach
consensus about our terminology or techniques. It probably wouldn't hurt if
we all agreed, but the conversations seem to wander into the world of ivory
towers and wizards too quickly. If anyone from outside looked in on one of
those email threads, it wouldn't help the cause.

--
Josh Viney
EastMedia Group
Company http://www.eastmedia.com
Blog http://www.kungpowthinking.com

9 Aug 2007 - 3:44pm
cherylkimble
2005

At 9:42 AM -0400 8/9/07, Todd Zaki Warfel wrote:
>On Aug 8, 2007, at 8:43 PM, Josh Viney wrote:
>At Messagefirst, usability for us is more than just testing, it's a(n
>exploratory) research method. As such, we incorporate exploratory
>activities into our usability research. Sure, we run contextual (real
>life) scenarios. But we don't make the customer go through the script
>in the order we put it together. We let them use the product(s) as
>they would natively, following their natural flow.

i was wondering if you could expand on this topic, todd, or if you've
written about it anywhere else. i'm working with technologists (and
clients) who think of usability as analytics and a/b testing. i would
really like them to start thinking about it as a research phase
instead.

thanks!
cheryl

9 Aug 2007 - 4:18pm
David Armano
2007

Dmitry,

My intention with the "PR" comment was not to write
off%u2014actually I think it's a serious issue. Those outside of UX
related disciplines MUST understand how we do what we do%u2014and
often times the methods though sound come off as "mysterious".

one of the learnings from my personal blogging experience has been
that speaking in vernacular which marketers, designers, strategists
etc. can comprehend%u2014lessons the "black art" halo affect.
That's a personal POV.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://beta.ixda.org/discuss?post=19104

9 Aug 2007 - 5:42pm
Michael Micheletti
2006

Josh, this is exactly what I perceive design to be: a craft. I'm an "innie"
embedded in a software dev team and the team's perception of me is similar.
To them, I have unique and useful craft skills, much as a carpenter might
consider an electrician. I'm a part of the larger team, and fill a useful
role, even though my craft may be unfamiliar to them.

Our dev team might have a very different impression though of a usability
consultant called in to perform a specific research or evaluation task. How
such a person is received would depend on their approachability as well as
on their skill and experience - their "bedside manner" as a design pro. I
can imagine scenarios in which a distinguished academic practitioner of the
black arts of usability might make a big impression on an organization -
launching a new strategic product maybe, or thumping hidebound engineers
upside the head with the oaken two-by-four of actual field research. But
this is more a parachute-in role that would be difficult to sustain in-house
over a long period of time. Somebody would eventually peek behind the
wizard's curtain and see the hot air balloonist behind it. :-)

Michael Micheletti

On 8/9/07, Josh Viney <jviney at gmail.com> wrote:
>
> The term "craft" comes to mind.
>

9 Aug 2007 - 8:21pm
Todd Warfel
2003

First, we take a baseline test of the existing product. Each
subsequent test we measure back against our baseline to determine a
measurable amount of improvement, or possibly even damage that might
have been done. That's one part of it. The other part is a little
more difficult to explain.

The research aspect comes in with exploratory methods. The first
thing we do is have a general discussion about usage with the
customer. This is in part to gain context to shape future scenarios
and part to uncover new opportunities. Once we have this context,
we'll use it for the first scenario, which is an open ended
discussion and exploration of the main screen(s). We're looking not
only for knee-jerk reactions, but also trying to evaluate a
perception of value and prioritize the items on screen. We use
include this type of open exploration with the formed scenarios as well.

For example, if we're working with a financial investment company,
we'll begin our discussion with what types of investing the customer
does and where they hold their accounts. We'll discuss how they began
investing, how they find out their information to determine what
types of investments to make, how often they invest, etc. Next, we'll
bring them to the initial landing screen and using the information
gathered in the initial discussion, discuss how the screen in front
of them either enables them or prevents them from accomplishing tasks
and goals that came up during our initial discussion.

The main thing we'll do, however, is during the entire session,
continually explore how and why they do what they're doing while
they're doing it. It's easier to understanding watching a session
than it is to try and explain it.

On Aug 9, 2007, at 4:44 PM, cheryl kimble wrote:

> At 9:42 AM -0400 8/9/07, Todd Zaki Warfel wrote:
>> On Aug 8, 2007, at 8:43 PM, Josh Viney wrote:
>> At Messagefirst, usability for us is more than just testing, it's a
>> (n exploratory) research method. As such, we incorporate
>> exploratory activities into our usability research. Sure, we run
>> contextual (real life) scenarios. But we don't make the customer
>> go through the script in the order we put it together. We let them
>> use the product(s) as
>> they would natively, following their natural flow.
>
> i was wondering if you could expand on this topic, todd, or if
> you've written about it anywhere else. i'm working with
> technologists (and clients) who think of usability as analytics and
> a/b testing. i would really like them to start thinking about it as
> a research phase instead.
>
> thanks!
> cheryl

Cheers!

Todd Zaki Warfel
President, Design Researcher
Messagefirst | Designing Information. Beautifully.
----------------------------------
Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
----------------------------------
In theory, theory and practice are the same.
In practice, they are not.

10 Aug 2007 - 3:37am
John Wood
2005

Dmitry makes a good point about the confusion of languages and concepts our
nascent profession presents to clients or to those trying to enter the
field. I guess this can't be helped. What we do is emerging out of the
confluence of many other professions in response to recent changes in
technology.

As the Internet becomes more pervasive and everything in our environment
gets driven by software and acquires more complex behaviours, our profession
has had to be cobbled together from many other disciplines. We're a bit like
Frankenstein's monster in this regard, all self-conscious and confused as we
try to get to grips with our place in the world.

I see this aspect of the Black Art perception as being purely transitory,
and my own hope is that we'll simply be absorbed into, and accepted by, the
wider community of designers, architects and other folks who shape places,
products and services. Let's not fret about this too much, time will take
care of the labels.

--------- Original Message --------
From: Dmitry Nekrasovski <mail.dmitry at gmail.com>

> Given that lack of consensus, it's not surprising that UX is
> occasionally perceived as a black art. It's actually more surprising
> that not everyone outside the field feels that way.

________________________________________________
Message sent using UebiMiau 2.7.10

10 Aug 2007 - 9:11am
Todd Warfel
2003

It's important, no critical, that the baseline is something that can
be repeated for future tests. Otherwise, you're going to have
difficulty comparing and measuring against it. We use the same
methods when we do baselines that we do for subsequent tests.

We try and use input from any resource we can get our hands on to
shape the scenarios. This typically includes interviews from stake
holders and web analytics. Sometimes we're fortunate enough to
interview some customers, have access to previous research, or
interview customer service reps to get additional guidance.

On Aug 9, 2007, at 9:47 PM, cheryl kimble wrote:

> what type of testing do you use for a baseline measurement? same
> combo of quan/qual or do you use a mix of analytics, etc...?
>

Cheers!

Todd Zaki Warfel
President, Design Researcher
Messagefirst | Designing Information. Beautifully.
----------------------------------
Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
----------------------------------
In theory, theory and practice are the same.
In practice, they are not.

Syndicate content Get the feed