Is Eye Tracking too expensive or complicated?

17 Apr 2008 - 6:11pm
6 years ago
38 replies
1774 reads
Andre Charland
2008

Hey All,

I just put together about Eye Tracking
(http://www.insideria.com/2008/04/is-eye-tracking-out-of-reach.html)
but I thought I'd put the question out to the list:

"Why isn't eye tracking used more in design and testing of rich
internet applications?"

I really have no experience with it, but am going to be working with
it this summer so looking for thoughts from some of you more
experienced with usability testing and eye tracking. I think for me
it's been cost and lack of knowledge/acccess that's kept me away from
it in the past.

Cheers!

--
André Charland
President and Co-Founder, Nitobi
e. andre.charland at gmail.com
skype. ebadre
b. http://blogs.nitobi.com/andre
w. http://www.nitobi.com

Comments

18 Apr 2008 - 11:56am
SusieComet
2006

At the Information Architecture conference in Miami last week, Jared Spool
imparted that eye tracking is just fluff to get the executives excited
about UCD.... he doesn't value it as a science beyond the obvious. We
already know that people don't look at advertisements and avoid certain
areas of the page.

While it will look good on a resume to have the experience, I tend to agree
with Jared.

18 Apr 2008 - 1:04pm
Rob Tannen
2006

It does have value as a secondary diagnostic tool. In the context of
usability testing, eye tracking does not determine the presence of a
usability problem, but helps determine what led to that problem in
conjunction with performance data, faciliator observations and user
self-reporting.

For example, different people may fail a task for different reasons
that eye tracking can reveal - overlooking a critical instruction
versus reading it but failing to understand it. In some cases users
can tell you this reliably, in others they can't.

Also, eye tracking provides a comparative metric between designs that
are equivalent on other performance measures. For example, Design A
may require greater visual scanning or workload than Design B, so all
other things being equal, Design A might be the better option.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=28208

18 Apr 2008 - 1:06pm
Rob Tannen
2006

...oops I meant Design B would be the better option (assuming less
visual workload is preferable).

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=28208

18 Apr 2008 - 2:21pm
pnuschke
2007

An interesting question, which begs for me to plug something I am
co-presenting at UPA entitled, "Practical Eyetracking."

https://www.usabilityprofessionals.org/conference/2008/program/friday.htm

If Jared Spool has been paraphrased correctly, I disagree that eyetracking
is just fluff. As Rob Tannen hinted at, eyetracking can be very helpful in
determining why things happened. Usually we have intuition do that, but
often it is not obvious.

Here's one example, we tested a design where people couldn't find the link
to login to the site, even though it was in the main navigation. Eyetracking
showed very clearly that people scanned the first few items in the
navigation and then moved on without seeing the link. Seeing exactly where
they stopped looking helped us understand why. And having a visual to show
it helped us convince the client to change their design. Without
eyetracking, we might have thought that the label was simply wrong.

The ROI is a separate discussion, but there are two things holding
eyetracking back: cost and the additional time it takes to plan studies and
to analyze the findings. Compared to alternatives, the ROI is pretty low.
But if you have one already, or have a bigger budget, it can be very useful.
And don't discount impressing the execs with great visuals.

Paul Nuschke
Electronic Ink

On Thu, Apr 17, 2008 at 7:11 PM, Andre Charland <andre.charland at gmail.com>
wrote:

> Hey All,
>
> I just put together about Eye Tracking
> (http://www.insideria.com/2008/04/is-eye-tracking-out-of-reach.html)
> but I thought I'd put the question out to the list:
>
> "Why isn't eye tracking used more in design and testing of rich
> internet applications?"
>
> I really have no experience with it, but am going to be working with
> it this summer so looking for thoughts from some of you more
> experienced with usability testing and eye tracking. I think for me
> it's been cost and lack of knowledge/acccess that's kept me away from
> it in the past.
>
> Cheers!
>
> --
> André Charland
> President and Co-Founder, Nitobi
> e. andre.charland at gmail.com
> skype. ebadre
> b. http://blogs.nitobi.com/andre
> w. http://www.nitobi.com
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

18 Apr 2008 - 1:16pm
Scott Berkun
2008

Eye-tracking has the same trap as all data collection - if you're not sure
how the data will enable you to make better decisions, having reems of it
doesn't help much. Depending on the kind of UI being designed, different
levels of data are more useful than others. Eye-tracking gives you
information about *everything* the user looks at in mili-second increments -
will that be more useful in recognizing issues and evaluating a design than
standard measures like time on task, errors, and success rates for tasks?
Often the answer is no.

There are plenty of other usability data collection methods thave have
specialized use: for example keystroke level recording and GOMS analysis
also provide highly detailed data that is great for some design problems but
overkill for most. I don't know of a single design firm or software company
that has invested in eye-tracking that uses the tool regularly. In special
cases it's poweful (Say a Heads-up display for the F-17, or Mac Finder,
complex high use items where miliseconds matter), but generally it provides
data that doesn't much help answer the design and research questions
designers have.

There are some compilations of general findings from eye-tracking, but as
interesting as they are they don't suggest changes or improvements so much
as offer validation of concepts many designers already use.
http://www.poynterextra.org/eyetrack2004/main.htm. It's great that someone
has done this, and it's cool looking data, but the information found tends
to be
general enough that few are motivated to repeat the studies to see if their
specific web-pages or software deviates from these typical saccade patterns.

Lastly, eyetracking tools are flat out expensive. This means it's harder to
find people experienced with the tools, and it takes more time and money to
interpret data that comes out of those studies.

-Scott

Scott Berkun
www.scottberkun.com

----- Original Message -----
From: "Andre Charland" <andre.charland at gmail.com>
To: <discuss at ixda.org>
Sent: Thursday, April 17, 2008 3:11 PM
Subject: [IxDA Discuss] Is Eye Tracking too expensive or complicated?

Hey All,

I just put together about Eye Tracking
(http://www.insideria.com/2008/04/is-eye-tracking-out-of-reach.html)
but I thought I'd put the question out to the list:

"Why isn't eye tracking used more in design and testing of rich
internet applications?"

18 Apr 2008 - 12:09pm
John Gibbard
2008

Thanks for sending this Andre, really interesting post.

I was watching a user session this week in one room at the Ux agency we use
and the moderator announced that, in the second room, they had setup eye
tracking so we could observe that. Everyone else rushed through to watch it
and I sat where I was as I have never gained anything useful from
eye-tracking in real-time.

That said, working out why someone has missed a header or navigation element
due to an imbalance in perceived-affordance or other visual priority is made
somewhat easier by the aggregated results of eye-tracking.

-----Original Message-----
From: discuss-bounces at lists.interactiondesigners.com
[mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of Andre
Charland
Sent: 18 April 2008 00:11
To: discuss at ixda.org
Subject: [IxDA Discuss] Is Eye Tracking too expensive or complicated?

Hey All,

I just put together about Eye Tracking
(http://www.insideria.com/2008/04/is-eye-tracking-out-of-reach.html)
but I thought I'd put the question out to the list:

"Why isn't eye tracking used more in design and testing of rich
internet applications?"

18 Apr 2008 - 12:32pm
Eva Kaniasty
2007

Eye tracking is very expensive for what it delivers, and, as with all
analytics, data interpretation can be difficult because you know nothing
about user motivation or intent. IMHO, analytics tools like session
recording and click maps can provide somewhat similar data for a MUCH lower
cost, both in terms of price and effort.

-eva

--
http://www.linkedin.com/in/kaniasty

19 Apr 2008 - 9:20am
Bruno Figueiredo
2007

Eye tracking is just like tracking mouse movement or clicks. It
doesn't really shows you what users are thinking, they're just
secondary manifestations of their thoughts. It's just like when you
twiddle your fingers on a table while thinking about what to do next.
It has nothing to do with it. Granted, there's some usefulness in the
data, since you can uncover some problems, but generally sitting with
a user and understanding it's train of thought is much more
insightful.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=28208

19 Apr 2008 - 8:46pm
Jared M. Spool
2003

I didn't say that I thought eye tracking was fluff.

I said that I thought it is a voodoo technique. Deducing information
about a design from eyetracking is equivalent to reading tea leaves
and using a ouija board.

The latter are cheaper, but just as reliable.

Every person I know who swears by eyetracking and has stories on how
its helped them can't explain how they would've gotten the same
results if some other professional had looked at the same raw data.
Until we can get to that point, the reader of the data will be more
important than the data itself, thereby making tea leaf reading a
viable alternative.

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: 1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=28208

19 Apr 2008 - 9:22pm
dszuc
2005

Have seen the phenomenon where because someone is using "Eye Tracking
Equipment" they are automatically given more status or overused
because of the "coolness factor" of just having the equipment
(independent of what the results mean and how other methods can help
compliment the results)

Also good thread about it here - http://www.ixda.org/discuss.php?post=22825&search=eye+tracking

rgds,
--
Daniel Szuc
Principal Usability Consultant
Apogee Usability Asia Ltd
www.apogeehk.com
Usability in Asia

The Usability Kit - www.theusabilitykit.com

On 19 Apr 2008, at 6:46 PM, Jared M.Spool wrote:

> I didn't say that I thought eye tracking was fluff.
>
> I said that I thought it is a voodoo technique. Deducing information
> about a design from eyetracking is equivalent to reading tea leaves
> and using a ouija board.
>
> The latter are cheaper, but just as reliable.
>
> Every person I know who swears by eyetracking and has stories on how
> its helped them can't explain how they would've gotten the same
> results if some other professional had looked at the same raw data.
> Until we can get to that point, the reader of the data will be more
> important than the data itself, thereby making tea leaf reading a
> viable alternative.
>
> Jared
>
> Jared M. Spool
> User Interface Engineering
> 510 Turnpike St., Suite 102, North Andover, MA 01845
> e: jspool at uie.com p: 1 978 327 5561
> http://uie.com Blog: http://uie.com/brainsparks
>
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=28208
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help

19 Apr 2008 - 9:27pm
SemanticWill
2007

Yes, but Jared --
There are at least 2 full sessions @ June's UPA conference dedicated to
eye-tracking, ergo it must be a valid technique! Maybe next year we will
have a "Tea-Leaf Reading Analytical Practices for Enhanced User Experience,"
which will follow "Rapid A-B testing with Mescaline & Electroshock Therapy:
Getting a Charge Out of User Testing" session.

On Sat, Apr 19, 2008 at 9:46 PM, Jared M. Spool <jspool at uie.com> wrote:

> I didn't say that I thought eye tracking was fluff.
>
> I said that I thought it is a voodoo technique. Deducing information
> about a design from eyetracking is equivalent to reading tea leaves
> and using a ouija board.
>
> The latter are cheaper, but just as reliable.
>
> Every person I know who swears by eyetracking and has stories on how
> its helped them can't explain how they would've gotten the same
> results if some other professional had looked at the same raw data.
> Until we can get to that point, the reader of the data will be more
> important than the data itself, thereby making tea leaf reading a
> viable alternative.
>
> Jared
>
> Jared M. Spool
> User Interface Engineering
> 510 Turnpike St., Suite 102, North Andover, MA 01845
> e: jspool at uie.com p: 1 978 327 5561
> http://uie.com Blog: http://uie.com/brainsparks
>
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=28208
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

--
~ will

"Where you innovate, how you innovate,
and what you innovate are design problems"

---------------------------------------------------------------------------------------------
Will Evans | User Experience Architect
tel +1.617.281.1281 || wkevans4 at gmail.com
---------------------------------------------------------------------------------------------

19 Apr 2008 - 10:54pm
Katie Albers
2005

At 10:27 PM -0400 4/19/08, Will Evans wrote:
>Yes, but Jared --
>There are at least 2 full sessions @ June's UPA conference dedicated to
>eye-tracking, ergo it must be a valid technique!

Not valid , but accepted. Surely we are all familiar with the
difference between those two....And I suspect you know it :) If
not...I know an excellent phrenologist

Katie

>Maybe next year we will
>have a "Tea-Leaf Reading Analytical Practices for Enhanced User Experience,"
>which will follow "Rapid A-B testing with Mescaline & Electroshock Therapy:
>Getting a Charge Out of User Testing" session.
>
>
>
>On Sat, Apr 19, 2008 at 9:46 PM, Jared M. Spool <jspool at uie.com> wrote:
>
>> I didn't say that I thought eye tracking was fluff.
>>
>> I said that I thought it is a voodoo technique. Deducing information
>> about a design from eyetracking is equivalent to reading tea leaves
>> and using a ouija board.
>>
>> The latter are cheaper, but just as reliable.
>>
>> Every person I know who swears by eyetracking and has stories on how
>> its helped them can't explain how they would've gotten the same
>> results if some other professional had looked at the same raw data.
>> Until we can get to that point, the reader of the data will be more
>> important than the data itself, thereby making tea leaf reading a
>> viable alternative.
>>
>> Jared
>>
>> Jared M. Spool
>> User Interface Engineering
>> 510 Turnpike St., Suite 102, North Andover, MA 01845
>> e: jspool at uie.com p: 1 978 327 5561
>> http://uie.com Blog: http://uie.com/brainsparks
>>
>>
>>
>> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
>> Posted from the new ixda.org
>> http://www.ixda.org/discuss?post=28208
>>
>>
>> ________________________________________________________________
>> Welcome to the Interaction Design Association (IxDA)!
>> To post to this list ....... discuss at ixda.org
>> Unsubscribe ................ http://www.ixda.org/unsubscribe
>> List Guidelines ............ http://www.ixda.org/guidelines
>> List Help .................. http://www.ixda.org/help
>>
>
>
>
>--
>~ will
>
>"Where you innovate, how you innovate,
>and what you innovate are design problems"
>
>---------------------------------------------------------------------------------------------
>Will Evans | User Experience Architect
>tel +1.617.281.1281 || wkevans4 at gmail.com
>---------------------------------------------------------------------------------------------
>________________________________________________________________
>Welcome to the Interaction Design Association (IxDA)!
>To post to this list ....... discuss at ixda.org
>Unsubscribe ................ http://www.ixda.org/unsubscribe
>List Guidelines ............ http://www.ixda.org/guidelines
>List Help .................. http://www.ixda.org/help

--

----------------
Katie Albers
katie at firstthought.com

20 Apr 2008 - 8:25am
SemanticWill
2007

Speaking of failed pseudo-science - I had the unfortunate opportunity to see
Ben Stein's Polemic Excretion "Expelled" on
Friday evening....sigh....

On Sat, Apr 19, 2008 at 11:54 PM, Katie Albers <katie at firstthought.com>
wrote:

> At 10:27 PM -0400 4/19/08, Will Evans wrote:
> >Yes, but Jared --
> >There are at least 2 full sessions @ June's UPA conference dedicated to
> >eye-tracking, ergo it must be a valid technique!
>
> Not valid , but accepted. Surely we are all familiar with the
> difference between those two....And I suspect you know it :) If
> not...I know an excellent phrenologist
>
> Katie
>
> >Maybe next year we will
> >have a "Tea-Leaf Reading Analytical Practices for Enhanced User
> Experience,"
> >which will follow "Rapid A-B testing with Mescaline & Electroshock
> Therapy:
> >Getting a Charge Out of User Testing" session.
> >
> >
> >
> >On Sat, Apr 19, 2008 at 9:46 PM, Jared M. Spool <jspool at uie.com> wrote:
> >
> >> I didn't say that I thought eye tracking was fluff.
> >>
> >> I said that I thought it is a voodoo technique. Deducing information
> >> about a design from eyetracking is equivalent to reading tea leaves
> >> and using a ouija board.
> >>
> >> The latter are cheaper, but just as reliable.
> >>
> >> Every person I know who swears by eyetracking and has stories on how
> >> its helped them can't explain how they would've gotten the same
> >> results if some other professional had looked at the same raw data.
> >> Until we can get to that point, the reader of the data will be more
> >> important than the data itself, thereby making tea leaf reading a
> >> viable alternative.
> >>
> >> Jared
> >>
> >> Jared M. Spool
> >> User Interface Engineering
> >> 510 Turnpike St., Suite 102, North Andover, MA 01845
> >> e: jspool at uie.com p: 1 978 327 5561
> >> http://uie.com Blog: http://uie.com/brainsparks
> >>
> >>
> >>
> >> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> >> Posted from the new ixda.org
> >> http://www.ixda.org/discuss?post=28208
> >>
> >>
> >> ________________________________________________________________
> >> Welcome to the Interaction Design Association (IxDA)!
> >> To post to this list ....... discuss at ixda.org
> >> Unsubscribe ................ http://www.ixda.org/unsubscribe
> >> List Guidelines ............ http://www.ixda.org/guidelines
> >> List Help .................. http://www.ixda.org/help
> >>
> >
> >
> >
> >--
> >~ will
> >
> >"Where you innovate, how you innovate,
> >and what you innovate are design problems"
> >
>
> >---------------------------------------------------------------------------------------------
> >Will Evans | User Experience Architect
> >tel +1.617.281.1281 || wkevans4 at gmail.com
>
> >---------------------------------------------------------------------------------------------
> >________________________________________________________________
> >Welcome to the Interaction Design Association (IxDA)!
> >To post to this list ....... discuss at ixda.org
> >Unsubscribe ................ http://www.ixda.org/unsubscribe
> >List Guidelines ............ http://www.ixda.org/guidelines
> >List Help .................. http://www.ixda.org/help
>
>
> --
>
> ----------------
> Katie Albers
> katie at firstthought.com
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

--
~ will

"Where you innovate, how you innovate,
and what you innovate are design problems"

---------------------------------------------------------------------------------------------
Will Evans | User Experience Architect
tel +1.617.281.1281 || wkevans4 at gmail.com
---------------------------------------------------------------------------------------------

20 Apr 2008 - 2:54pm
Christopher Fahey
2005

On Apr 19, 2008, at 2:46 PM, Jared M.Spool wrote:
> Every person I know who swears by eyetracking and has stories on how
> its helped them can't explain how they would've gotten the same
> results if some other professional had looked at the same raw data.

Nor can they explain why they wouldn't get *better* results and
*better* recommendations from simply showing the UI to a half-decent
user interface designer for 20 minutes.

I've never seen a eyetracking recommendation that wasn't either (a)
patently obvious to me ("Your 6-pixel-high light gray text should be
made easier to see") or (b) completely stupid ("Move the search box to
the left (where you currently have the picture of the cute little
puppy) because everyone seems to spends time looking at the left side
of the page").

-Cf

Christopher Fahey
____________________________
Behavior
http://www.behaviordesign.com
212.532.4002 x203
646.338.4002 mobile

20 Apr 2008 - 8:06am
Andy Edmonds
2004

Wow, what skepticism in this thread! I'll admit that I don't use my eye
tracker as often as split testing, but I do feel compelled to offer a
more positive view on the matter.

The #1 utility of the eye tracker, for me, is in helping me understand
the user's cognition during a test session. With real time gaze data on
a 2nd observer only screen, I don't have to work as hard on eliciting
verbal protocol. I have also used the eye gaze reports to ask the user
questions after the session -- a methodology others have developed more
fully.

Heck, I even spotted "button gravity" in my lab:
http://flickr.com/photos/andyed/450579101/

Bruno: Regarding mouse movements, it's clear that eye movements are much
higher signal, but mouse position has more data than twiddled fingers.
I've summarized research on this on my blog and in a recent publication:
http://alwaysbetesting.com/abtest/index.cfm/2007/4/29/Eye-Tracking-vs-Mouse-Tracking

There's a longstanding and largely unsuccessful effort to generate
quantitative quality metrics from eye-tracking data. That said, distance
traveled by eye has been used productively in LukeW's work on forms,
presented at Jared's Web App Summit recently. I've also been able to
show good design leads to more efficient scan paths,
http://flickr.com/photos/22582943@N03/2175663626/.

I won't dispute that many of the insights from eye-tracking are fairly
obvious (ex. no headings in a long menu?
http://flickr.com/photos/22582943@N03/2177600531/), but there's
something to be said for how well the visualizations engage consumers.

To help with these basic types of insights, we've developed a vision
simulation in a browser, "Stomper Scrutinizer" that helps reveal the
multiple fixation requirements of left aligned form labels for example.

Andy

Bruno Figueiredo wrote:
> Eye tracking is just like tracking mouse movement or clicks. It
> doesn't really shows you what users are thinking, they're just
> secondary manifestations of their thoughts. It's just like when you
> twiddle your fingers on a table while thinking about what to do next.
> It has nothing to do with it. Granted, there's some usefulness in the
> data, since you can uncover some problems, but generally sitting with
> a user and understanding it's train of thought is much more
> insightful.
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=28208
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>
>

20 Apr 2008 - 3:29pm
Mark Schraad
2006

At the cog sci lab when I was in grad school the eye tracking
equipment pretty much sat in the corner collecting dust. The general
feeling was that we had leaned all we could from that technology
years ago. We generally knowhow people see.

It does have a great 'gee wiz' effect when you show and tell. All
the execs think it is really cool.

On Apr 20, 2008, at 9:06 AM, Andy Edmonds wrote:

> Wow, what skepticism in this thread! I'll admit that I don't use
> my eye
> tracker as often as split testing, but I do feel compelled to offer a
> more positive view on the matter.

20 Apr 2008 - 3:56pm
pnuschke
2007

Jared said:

> "I said that I thought it is a voodoo technique. Deducing information
> about a design from eyetracking is equivalent to reading tea leaves
> and using a ouija board."
>

That's a pretty colorful exaggeration.

Eyetracking lets you see where people are looking in real time. Without
considering post-test analysis, this has real value in helping the
facilitator better understand what is happening without interfering. One
analogy I find useful, in terms of understanding what the participant is
doing/thinking, is that having eyetracking versus not having eyetracking is
like testing in person versus testing remotely.

I wonder, given your research background, Jared, if we are talking about
different types of eyetracking studies. For academic/generalizable research,
I have found eyetracking studies to be pretty meaningless. But for testing
real products, and only trying to interpret results for those pages, it can
be useful and not all that difficult, depending on the stimulus and tasks of
course.

I also wonder if some people have been burned by past bad experiences with
faulty eyetrackers and bad software. My lab at school had three separate
eyetrackers and none of them worked correctly. The Tobii one that I use now
is easy to use and the analysis software is very good.

Paul

On Sat, Apr 19, 2008 at 9:46 PM, Jared M. Spool <jspool at uie.com> wrote:

> I didn't say that I thought eye tracking was fluff.
>
> I said that I thought it is a voodoo technique. Deducing information
> about a design from eyetracking is equivalent to reading tea leaves
> and using a ouija board.
>
> The latter are cheaper, but just as reliable.
>
> Every person I know who swears by eyetracking and has stories on how
> its helped them can't explain how they would've gotten the same
> results if some other professional had looked at the same raw data.
> Until we can get to that point, the reader of the data will be more
> important than the data itself, thereby making tea leaf reading a
> viable alternative.
>
> Jared
>
> Jared M. Spool
> User Interface Engineering
> 510 Turnpike St., Suite 102, North Andover, MA 01845
> e: jspool at uie.com p: 1 978 327 5561
> http://uie.com Blog: http://uie.com/brainsparks
>
>
>
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Posted from the new ixda.org
> http://www.ixda.org/discuss?post=28208
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

20 Apr 2008 - 4:50pm
Jared M. Spool
2003

Paul Nuschke wrote:
> Eyetracking lets you see where people are looking in real time.

Yes. But just because you know where someone looks or doesn't look
doesn't mean you know anything about what they see, what they wanted
to see, and what they didn't see. It's not clear to me how one
interprets the "they gazed at this point on the screen for 400 ms"
information. Was that good? Was that bad?

We know that people see things through their peripheral vision, such
as the scroll bar, so that's not recorded by the eye tracker. That
means we can't even assume that when someone doesn't gaze at a spot
that it wasn't seen.

With eye trackers, we have a bunch of observations but no way to
determine the proper inferences. Instead, all of the value of an eye
tracker comes from the interpretation.

Show me a study that shows that N separate evaluators looked at the
same eye tracking data and came away with the same conclusions and
I'll change my mind.

Until then, I'll continue to group it with tarot cards and palm
reading as a fine art.

> Without considering post-test analysis, this has real value in
helping the facilitator better understand what is happening without
interfering.

Exactly my point. As the President of Best Buy, John "JT" Thompson,
once told me (while I was delivering a great presentation with a ton
of data):

"I worked for Jack Welch at GE for 17 years and if I learned
anything while I was there, it was this: If you torture data long and
hard enough, it will confess to anything you want."

>One analogy I find useful, in terms of understanding what the
participant is doing/thinking, is that having eyetracking versus not
having eyetracking is like testing in person versus testing remotely.

You lost me there.

> I wonder, given your research background, Jared, if we are talking
about different types of eyetracking studies. For
academic/generalizable research, I have found eyetracking studies to
be pretty meaningless.

Actually, that's pretty funny. I think the most exciting eye
tracking stuff is happening in research. There were a ton of good
posters and some neat presentations at CHI showing how eye tracking,
as an alternative input device, could have some really cool
applications, especially for accessibility.

I also think there are some interesting cognitive and behavioral
psych things to learn by using the devices. But I don't think
there's been anything useful in terms of using it as a tool to
enhance or inform the design process, so I'm guessing we agree
there.

> But for testing real products, and only trying to interpret results
for those pages, it can be useful and not all that difficult,
depending on the stimulus and tasks of course.

Yah, not seeing that.

What I see is that it falls nicely in the "If you can't dazzle 'em
with your brilliance, feel free to baffle 'em with your bullshit"
category of helping folks understand how to change their designs.

But then again, what do I know?

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: 1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=28208

20 Apr 2008 - 4:34pm
Pieter Jansegers
2008

I'm wondering about the need of eye tracking on any particular page.

I mean : it's nice to repeat scientific tests over and over again to
control the results.

But I don't think it could add specific value to website analysis as
such to do it on every site over and over again.

I don't see why the results would differ from a general analysis of
the website based on the principles obtained in research previously.

But, hey, if you can convince a client more easily to put his/her
money into your pocktets showing a nice reddish glow on a graph...
who I am to stop you ?

Pieter Jansegers

http://webosophy.ning.com

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=28208

20 Apr 2008 - 5:18pm
Mark Schraad
2006

I have been observing (pardon the pun) how often people look right at
something and don;t see it. I am sure there is a technical term for
this 'attention periphery' but I have not found it in the research
yet. I would love to see the results and analysis of an eye tracking
expert of subject watching the now classic dancing bear in the
basketball game.

Mark

http://www.dothetest.co.uk/

On Apr 20, 2008, at 4:56 PM, Paul Nuschke wrote:

> Eyetracking lets you see where people are looking in real time.

20 Apr 2008 - 5:40pm
Jared M. Spool
2003

On Apr 20, 2008, at 6:18 PM, mark schraad wrote:

> I am sure there is a technical term for
> this 'attention periphery' but I have not found it in the research
> yet.

Search for "situation inattentional blindness". The primary work was
done by Simons at U of Illinois at Urbana-Champagne.

Jared

20 Apr 2008 - 6:22pm
Anonymous

Mark said:
I would love to see the results and analysis of an eye tracking expert of subject watching the now classic dancing bear in the basketball game.

Here's one study: http://andrewd.ces.clemson.edu/courses/cpsc412/fall04/teams/group1_hfes.pdf

Justine

20 Apr 2008 - 6:45pm
pnuschke
2007

On Sun, Apr 20, 2008 at 5:50 PM, Jared M. Spool <jspool at uie.com> wrote:

> Paul Nuschke wrote:
> > Eyetracking lets you see where people are looking in real time.
>
> Yes. But just because you know where someone looks or doesn't look
> doesn't mean you know anything about what they see, what they wanted
> to see, and what they didn't see. It's not clear to me how one
> interprets the "they gazed at this point on the screen for 400 ms"
> information. Was that good? Was that bad?

Imagine that a user needs to click on a link to go somewhere. If she fixates
on the link and don't click it, then that's pretty good evidence that she
did not understand the link.

We know that people see things through their peripheral vision, such
> as the scroll bar, so that's not recorded by the eye tracker. That
> means we can't even assume that when someone doesn't gaze at a spot
> that it wasn't seen.

True, but that's a good thing. You can't read or see fine details in your
peripheral vision, so even if you notice something it doesn't mean that you
looked at it enough to understand what it contained (unless the important
details were very big). In the example above, even if the user noticed that
a link existed, if she did not attend to it, then she would not have been
able to read it.

Show me a study that shows that N separate evaluators looked at the
> same eye tracking data and came away with the same conclusions and
> I'll change my mind.

That some data does not make sense is not a phenomenon unique to
eyetracking. I've seen plenty of different interpretations of statistics as
well.

> >One analogy I find useful, in terms of understanding what the
> participant is doing/thinking, is that having eyetracking versus not
> having eyetracking is like testing in person versus testing remotely.
>
> You lost me there.
>

In remote testing, you loose voice quality and you don't see mannerisms,
facial expressions, etc. In "in person" testing, you have gestures and
facial expressions, and voice inflections. In eyetracking, you add the
ability to see where they are looking. You lose something too, though, in
your testing methodology, but that's another e-mail thread.

Paul

20 Apr 2008 - 8:05pm
Al Selvin
2006

This may be obvious or trivial, but it was a new insight to me. I've never
used eye tracking in software/web design or thought it would have much
utility. Nonetheless I wandered up to one of the eye-tracking vendors at the
CHI conference and got into a conversation with the rep. He said that most
of their sales and emphasis were on contexts where there was something other
than a single screen to look at -- such as automobile dashboards, complex
control panels like nuclear power plants, and the like. In those situations,
seeing where people are looking in response to stimuli like alerts, gauges,
oncoming obstacles etc., that can come from many different directions, is
very important and the eye tracking apparatus can be extremely helpful.
That made a lot of sense to me.

Al

20 Apr 2008 - 10:43pm
Jared M. Spool
2003

On Apr 20, 2008, at 7:45 PM, Paul Nuschke wrote:

> Imagine that a user needs to click on a link to go somewhere. If she
> fixates
> on the link and don't click it, then that's pretty good evidence
> that she
> did not understand the link.

All you know is that the eye tracker registered that they fixated on
the link and that they didn't click.

The notion that they didn't understand the link is one inference.

It's not the only inference. It may not be the right inference.

It is purely *your* interpretation that the user didn't understand it.

(And you could've gotten there without the eye tracking data.)

>> We know that people see things through their peripheral vision, such
>> as the scroll bar, so that's not recorded by the eye tracker. That
>> means we can't even assume that when someone doesn't gaze at a spot
>> that it wasn't seen.
>
> True, but that's a good thing. You can't read or see fine details in
> your
> peripheral vision, so even if you notice something it doesn't mean
> that you
> looked at it enough to understand what it contained (unless the
> important
> details were very big).

Again. Your inference. You don't have any evidence to actually know
that's true.

In fact, in psychographic phenomena, it's pretty amazing what people
can see and deduce from the peripheral vision. There's a lot happening
within 140 degrees of the focal point.

And it's pretty amazing what is lost within the center gaze area,
especially with people who have field issues that are frequent in
males over 40, females over 50, and anyone suffering from optic
neuritis or other immune-deficiency-based symptoms. (In MS patients,
for example, optic neuritis frequently shows up in late teens, early
20s.)

So, you are just inferring meaning to the data you're collecting.

> In the example above, even if the user noticed that
> a link existed, if she did not attend to it, then she would not have
> been
> able to read it.

Your inference. There are other likely inferences too.

>> Show me a study that shows that N separate evaluators looked at the
>> same eye tracking data and came away with the same conclusions and
>> I'll change my mind.
>
> That some data does not make sense is not a phenomenon unique to
> eyetracking. I've seen plenty of different interpretations of
> statistics as
> well.

Ok. Does that make eyetracking work?

Not buying it. Still think it's up to the interpreter of the eye
tracker.

Let me put it another way:

Would you, Paul, be comfortable letting your clients to use the eye
tracker without any help in interpreting data from you. Is the device
all they need to make the judgments necessary to provide good design
advice?

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

21 Apr 2008 - 7:33am
pnuschke
2007

On Sun, Apr 20, 2008 at 11:43 PM, Jared M. Spool <jspool at uie.com> wrote:

> All you know is that the eye tracker registered that they fixated on the
> link and that they didn't click.
> The notion that they didn't understand the link is one inference.
> It's not the only inference. It may not be the right inference.
> It is purely *your* interpretation that the user didn't understand it.
>

One way of double-checking the inferences is to ask the participant. What
I've observed in eyetracking has been confirmed by participants enough that
I know that the premises of eyetracking are true. Seems like you have had a
different experience, so I'll be curious to hear what people have to say at
UPA about it.

(And you could've gotten there without the eye tracking data.)
>

This assumes that you knew there was an issue to begin with, or that the
type of study allows you to follow up. Neither is always the case.

In fact, in psychographic phenomena, it's pretty amazing what people can see
> and deduce from the peripheral vision. There's a lot happening within 140
> degrees of the focal point.

> And it's pretty amazing what is lost within the center gaze area,
> especially with people who have field issues that are frequent in males over
> 40, females over 50, and anyone suffering from optic neuritis or other
> immune-deficiency-based symptoms. (In MS patients, for example, optic
> neuritis frequently shows up in late teens, early 20s.)

Sounds interesting, do you have a link?

Is the device all they need to make the judgments necessary to provide good
> design advice?

Of course not. In nowhere here have I said that eyetracking was the only way
to make judgments. It's just another tool. IMO, the main problems with
eyetracking are 1) the multiple participant data (heatmaps) doesn't always
make sense, 2) it is time consuming to use, and 3) the initial cost of the
equipment is ridiculously high for the benefit that you get. It is not that
the premises are wrong. On this point I think that we disagree, so let's
just leave it there.

Paul

21 Apr 2008 - 8:01am
Rob Tannen
2006

Sounds like what's missing here is a set of consistent, objective and
reliable guidelines for interpreting eye-tracking data (and
potentially usability findings in general).

For example a fixation of an a priori specified minimum duration on a
link in conjunction with a user failing to click the link AND the user
reporting that the link was seen would strongly indicate that indeed
it was seen : )

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=28208

21 Apr 2008 - 8:48am
Todd Warfel
2003

On Apr 20, 2008, at 3:54 PM, Christopher Fahey wrote:

> Nor can they explain why they wouldn't get *better* results and
> *better* recommendations from simply showing the UI to a half-decent
> user interface designer for 20 minutes

Eye-tracking should not be used on its own—if used at all, it should
be for supplementary input. Yes, it tells you what someone is looking
at, but doesn't tell you why—and the "why" is critical to finding the
appropriate design solution.

Cheers!

Todd Zaki Warfel
President, Design Researcher
Messagefirst | Designing Information. Beautifully.
----------------------------------
Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
Twitter: zakiwarfel
----------------------------------
In theory, theory and practice are the same.
In practice, they are not.

21 Apr 2008 - 12:36pm
Pieter Jansegers
2008

In the end, it's on what people click, which is really important.

And even more important is the information people are really looking for and
the findability

of that information.

It all comes down to offering the right information content in a nice way.

Without the right information, your site may be optimized the way you want,
people won't be

happy.

It's a bit back to basics, but basics are extremely important.

Pieter Jansegers
http://webosophy.ning.com

22 Apr 2008 - 2:08am
Larry Tesler
2004

Jared,

Eye tracking is of value when someone sees something in the data that
leads to a new and valuable insight.

The fact that different observers see different things in the same raw
eye tracking data is of no more concern to me than the fact that
different players count a different number of words on the same Boggle
board. Some people see words that are hidden in plain sight; some do
not. But noticed or not, the words are there. In the tea leaves, there
are no hidden words.

Adding observers to any team that is interpreting raw data makes it
more likely that someone will notice something subtle, but important,
that other observers have missed.

This is as true in design research as it is in radiology,
geophysicists or espionage.

Larry Tesler

On Apr 19, 2008, at 6:46 PM, Jared M.Spool wrote:

> ... Deducing information
> about a design from eyetracking is equivalent to reading tea leaves
> and using a ouija board.
>
> The latter are cheaper, but just as reliable.
>
> Every person I know who swears by eyetracking and has stories on how
> its helped them can't explain how they would've gotten the same
> results if some other professional had looked at the same raw data.
> Until we can get to that point, the reader of the data will be more
> important than the data itself, thereby making tea leaf reading a
> viable alternative.

22 Apr 2008 - 7:59am
Jared M. Spool
2003

On Apr 22, 2008, at 3:08 AM, Larry Tesler wrote:

> The fact that different observers see different things in the same
> raw eye tracking data is of no more concern to me than the fact that
> different players count a different number of words on the same
> Boggle board. Some people see words that are hidden in plain sight;
> some do not. But noticed or not, the words are there. In the tea
> leaves, there are no hidden words.

Larry,

I have no doubt that the observations are of interest.

My point is that the inferences drawn from those observations have
little-to-no validity, thus the tea leaf analogy.

If someone fixates on a link for a unusually large time, does that
mean they are confused by it? Or they aren't confused, but are trying
to decide if its what they want? Or they know whether they want it or
not but are considering something else?

Different inferences will lead to completely different design
solutions. Are you saying it doesn't matter which inference (and
therefore, which design solution) the observers choose?

When you back an eye-tracking supporter into a corner about this, they
all say, "Well, you should only use eye tracking in conjunction with
other data collection tools and techniques to verify your inferences."
In almost all cases, the "other data collection tools and techniques"
would yield just as much value without the eye tracking as with it, so
what's the benefit?

Second, in almost all uses of eye tracking I've seen in the last 5
years, it's in the form of twisting the meaning of the heatmap/plot
diagram/tea leaf reading into supporting whatever wacky inference the
specialist wants to support. "See that big red spot there. That means
the users are confused" v. "See that big red spot there, that means we
fixed the design."

If there really is something to this eye tracking thing, I'd think
you'd want your team members to all look at the same heat map and come
to somewhat similar design implications.

Eyetracking equipment: $30,000
Ouija Board: $5
Quality design based on solid inferences from rich, meaningful data:
Priceless

That's my take.

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

22 Apr 2008 - 8:23am
Todd Warfel
2003

On Apr 22, 2008, at 8:59 AM, Jared M. Spool wrote:

> If someone fixates on a link for a unusually large time, does that
> mean they are confused by it? Or they aren't confused, but are
> trying to decide if its what they want? Or they know whether they
> want it or not but are considering something else?

Oh, oh, oh, I know—what we need is eye tracking with mind reading.
Now, that's useful.

Cheers!

Todd Zaki Warfel
President, Design Researcher
Messagefirst | Designing Information. Beautifully.
----------------------------------
Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
Twitter: zakiwarfel
----------------------------------
In theory, theory and practice are the same.
In practice, they are not.

22 Apr 2008 - 10:34pm
Jared M. Spool
2003

Todd wrote:

> Oh, oh, oh, I know%u2014what we need is eye tracking with mind
reading. Now, that's useful.

Hah! If we had mind reading, we wouldn't need the eye tracker.

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: jspool at uie.com p: 1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=28208

23 Apr 2008 - 1:21am
Larry Tesler
2004

Jared,

If most readers think this has gone on too long, we should wind it down.

On Apr 22, 2008, at 5:59 AM, Jared M. Spool wrote:
>
> On Apr 22, 2008, at 3:08 AM, Larry Tesler wrote:
>
>> The fact that different observers see different things in the same
>> raw eye tracking data is of no more concern to me than the fact
>> that different players count a different number of words on the
>> same Boggle board. Some people see words that are hidden in plain
>> sight; some do not. But noticed or not, the words are there. In the
>> tea leaves, there are no hidden words.
>
> Larry,
>
> I have no doubt that the observations are of interest.
>
> My point is that the inferences drawn from those observations have
> little-to-no validity, thus the tea leaf analogy.

I consider them valid if they inspire us to make design changes that
lead to improvements in objective metrics.

In any study, with or without an eye tracker, I look for:
- a well-designed experiment
- clean data
- appropriate, error-free analysis
- perceptive observation (which may require several observers to point
things out to each other and reach consensus, as radiologists often do
when faced with difficult images)
- generation of hypotheses consistent with the study observations and
any other available observations
- prioritization of those hypotheses
- generation of design solutions that respond to the most likely
hypothesis
- implementation
- bucket testing
- if no improvement is seen, iterate with alternate hypotheses

> If someone fixates on a link for a unusually large time, does that
> mean they are confused by it? Or they aren't confused, but are
> trying to decide if its what they want? Or they know whether they
> want it or not but are considering something else?

If the user's mental state matters to you, ask the user what it was.
They may know. If they do not know, devise a more clever experiment.

But sometimes, the user's mental state doesn't matter. We may have run
the test because too few people were clicking on the link. We thought
perhaps they didn't even look at the area of the page that contained
the link. The tracker has refuted our hypothesis. We know that some
people look straight at the link and still do not click it. Other data
may be needed if we want to find out why. But the study was a success.
It achieved its goal.

> Different inferences will lead to completely different design
> solutions. Are you saying it doesn't matter which inference (and
> therefore, which design solution) the observers choose?

If the radiologists call your malignant tumor benign, or vice versa,
you may receive the wrong treatment, which could be a costly mistake.
But design changes that eye tracking studies inspire often entail
simple modifications to layout, color, size, typeface, etc., that help
to steer attention. They are often cheap to implement. If there are
two competing inferences, you can often try both implied solutions.

Of course, if you had both designs in mind (and particularly the one
that ultimately proved to be best) before you ran the eye tracking
study, then the study was a waste of time. If that is the situation
you are in, and you never want to run a study that may simply confirm
that you were right, then your point is valid. But it is not always
the situation. There may be no unrefuted theory about why users are
not clicking the link. There may be no considered designs that would
increase clicks. There may be too many credible designs--more than one
has the time and staff to implement and test. Or you may simply want
to confirm other data or hunches.

> When you back an eye-tracking supporter into a corner about this,
> they all say, "Well, you should only use eye tracking in conjunction
> with other data collection tools and techniques to verify your
> inferences." In almost all cases, the "other data collection tools
> and techniques" would yield just as much value without the eye
> tracking as with it, so what's the benefit?

The eye tracking test may have been the source of the first clue as to
what ails your interface. Without it, you may never have thought to
try those other tools and techniques.

Or the eye tracking test may help to rule out hypotheses that one has
generated from the use of other tools and techniques.

> Second, in almost all uses of eye tracking I've seen in the last 5
> years, it's in the form of twisting the meaning of the heatmap/plot
> diagram/tea leaf reading into supporting whatever wacky inference
> the specialist wants to support. "See that big red spot there. That
> means the users are confused" v. "See that big red spot there, that
> means we fixed the design."

I agree that features of the heat map don't tell you mental states.
But if they are inconsistent with hypotheses about mental states, they
may call those hypotheses into question. "See that big red spot over
there. Maybe they were looking at the link after all."

A big red spot may also provide evidence that the design is improved--
if the problem with the design was that nobody was looking at that
link. But there are usually more convincing methods of validating
designs than looking at heat maps, e.g., bucket tests with well-chosen
metrics.

> If there really is something to this eye tracking thing, I'd think
> you'd want your team members to all look at the same heat map and
> come to somewhat similar design implications.

I thought my previous email, with its counterexamples, refuted the
assumption that independent observers of an image must come to the
same separate conclusions for the methodology to be useful. I guess
we'll have to agree to disagree about that.

> Eyetracking equipment: $30,000

Amortized over N studies = $30,000/N. If your business is large, your
benefit may be large, too, and you could be getting a very good ROI.

> Ouija Board: $5
> Quality design based on solid inferences from rich, meaningful data:
> Priceless

If your clients' markets are too small to make eye tracking
investments pay off, and they have other sources of data that give
them useful insights, then I can see why you don't advocate eye
tracking. But for a client selling a high volume product, especially
on the web where switching costs are low, small changes can have large
effects, and eye tracking can pay off.

Larry Tesler

23 Apr 2008 - 9:06am
David Walker
2007

As others have already pointed out, the primary benefactor of eye-tracking
studies is the coordinator of the studies. After many such studies, I
gradually have absorbed what textbooks could not teach me effectively about
this science. What draws the human eye is not quite unpredictable, but the
wide variance does not lend itself to easy rule-making.

Madison Avenue has been using this technology for years for somewhat
nefarious purposes. We need to embrace it and put the knowledge to good use
across the discipline of interaction and experience design. How could we
tolerate the idea that it would be okay for marketing and advertising folks
to know more about this area than ourselves?

Here's the rub though: I have always found it challenging to use the
eye-tracking data to correct my own designs. When I tried to do this
initially, I would never get it right: I would re-submit "corrected" designs
for new eye-tracking and I'd get even worse results. Persistence was key.

Eye-tracking studies have made me a better designer. And my clients love
it.

Dave

-----Original Message-----
From: discuss-bounces at lists.interactiondesigners.com
[mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of Larry
Tesler
Sent: Tuesday, April 22, 2008 11:22 PM
To: Jared M. Spool
Cc: discuss at ixda.org
Subject: Re: [IxDA Discuss] Is Eye Tracking too expensive or complicated?

Jared,

If most readers think this has gone on too long, we should wind it down.

On Apr 22, 2008, at 5:59 AM, Jared M. Spool wrote:
>
> On Apr 22, 2008, at 3:08 AM, Larry Tesler wrote:
>
>> The fact that different observers see different things in the same
>> raw eye tracking data is of no more concern to me than the fact
>> that different players count a different number of words on the
>> same Boggle board. Some people see words that are hidden in plain
>> sight; some do not. But noticed or not, the words are there. In the
>> tea leaves, there are no hidden words.
>
> Larry,
>
> I have no doubt that the observations are of interest.
>
> My point is that the inferences drawn from those observations have
> little-to-no validity, thus the tea leaf analogy.

I consider them valid if they inspire us to make design changes that
lead to improvements in objective metrics.

In any study, with or without an eye tracker, I look for:
- a well-designed experiment
- clean data
- appropriate, error-free analysis
- perceptive observation (which may require several observers to point
things out to each other and reach consensus, as radiologists often do
when faced with difficult images)
- generation of hypotheses consistent with the study observations and
any other available observations
- prioritization of those hypotheses
- generation of design solutions that respond to the most likely
hypothesis
- implementation
- bucket testing
- if no improvement is seen, iterate with alternate hypotheses

> If someone fixates on a link for a unusually large time, does that
> mean they are confused by it? Or they aren't confused, but are
> trying to decide if its what they want? Or they know whether they
> want it or not but are considering something else?

If the user's mental state matters to you, ask the user what it was.
They may know. If they do not know, devise a more clever experiment.

But sometimes, the user's mental state doesn't matter. We may have run
the test because too few people were clicking on the link. We thought
perhaps they didn't even look at the area of the page that contained
the link. The tracker has refuted our hypothesis. We know that some
people look straight at the link and still do not click it. Other data
may be needed if we want to find out why. But the study was a success.
It achieved its goal.

> Different inferences will lead to completely different design
> solutions. Are you saying it doesn't matter which inference (and
> therefore, which design solution) the observers choose?

If the radiologists call your malignant tumor benign, or vice versa,
you may receive the wrong treatment, which could be a costly mistake.
But design changes that eye tracking studies inspire often entail
simple modifications to layout, color, size, typeface, etc., that help
to steer attention. They are often cheap to implement. If there are
two competing inferences, you can often try both implied solutions.

Of course, if you had both designs in mind (and particularly the one
that ultimately proved to be best) before you ran the eye tracking
study, then the study was a waste of time. If that is the situation
you are in, and you never want to run a study that may simply confirm
that you were right, then your point is valid. But it is not always
the situation. There may be no unrefuted theory about why users are
not clicking the link. There may be no considered designs that would
increase clicks. There may be too many credible designs--more than one
has the time and staff to implement and test. Or you may simply want
to confirm other data or hunches.

> When you back an eye-tracking supporter into a corner about this,
> they all say, "Well, you should only use eye tracking in conjunction
> with other data collection tools and techniques to verify your
> inferences." In almost all cases, the "other data collection tools
> and techniques" would yield just as much value without the eye
> tracking as with it, so what's the benefit?

The eye tracking test may have been the source of the first clue as to
what ails your interface. Without it, you may never have thought to
try those other tools and techniques.

Or the eye tracking test may help to rule out hypotheses that one has
generated from the use of other tools and techniques.

> Second, in almost all uses of eye tracking I've seen in the last 5
> years, it's in the form of twisting the meaning of the heatmap/plot
> diagram/tea leaf reading into supporting whatever wacky inference
> the specialist wants to support. "See that big red spot there. That
> means the users are confused" v. "See that big red spot there, that
> means we fixed the design."

I agree that features of the heat map don't tell you mental states.
But if they are inconsistent with hypotheses about mental states, they
may call those hypotheses into question. "See that big red spot over
there. Maybe they were looking at the link after all."

A big red spot may also provide evidence that the design is improved--
if the problem with the design was that nobody was looking at that
link. But there are usually more convincing methods of validating
designs than looking at heat maps, e.g., bucket tests with well-chosen
metrics.

> If there really is something to this eye tracking thing, I'd think
> you'd want your team members to all look at the same heat map and
> come to somewhat similar design implications.

I thought my previous email, with its counterexamples, refuted the
assumption that independent observers of an image must come to the
same separate conclusions for the methodology to be useful. I guess
we'll have to agree to disagree about that.

> Eyetracking equipment: $30,000

Amortized over N studies = $30,000/N. If your business is large, your
benefit may be large, too, and you could be getting a very good ROI.

> Ouija Board: $5
> Quality design based on solid inferences from rich, meaningful data:
> Priceless

If your clients' markets are too small to make eye tracking
investments pay off, and they have other sources of data that give
them useful insights, then I can see why you don't advocate eye
tracking. But for a client selling a high volume product, especially
on the web where switching costs are low, small changes can have large
effects, and eye tracking can pay off.

Larry Tesler

________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... discuss at ixda.org
Unsubscribe ................ http://www.ixda.org/unsubscribe
List Guidelines ............ http://www.ixda.org/guidelines
List Help .................. http://www.ixda.org/help

24 Apr 2008 - 3:23pm
Greg Edwards
2008

Andre,

In all of the discussion, the initial question got unanswered:

* Is eyetracking too expensive? *

"Expensive" is all relative. Currently, you have three options:
1) outsourced data collection and analysis: you can have someone expert in
eyetracking collect and analyze your data for you. This has the benefit that
it solves the problem many people brought up which is that experience counts
and it's harder to analyze eyetracking data than most people realize.
Eyetools, Inc. (www.eyetools.com) does this for a lot of clients and
agencies (full disclosure: my company -- I'm the founder and CEO). There are
other companies as well (actually, quite a few usability companies have
started offering eyetracking in the last year).

Cost: $2,800 - $5,000 including recruiting, incentives, and facilities for
testing a homepage/landing page/etc. (I'm not going to go into details of
pricing since this is probably not the appropriate place for that.)

2) You can buy your own equipment: there are a number of hardware
manufactures out there with pricing ranging from $20k-$44k for the hardware
and software necessary to run web usability tests. Hardware manufacturers
include LC Technologies (www.eyegaze.com), Eyetech Digital Systems
(www.eyetechds.com), SMI (www.smi.com), ASL (www.a-s-l.com, though recently
they've been doing less web usability stuff), and SmartEye (www.smarteye.se)
are some that come to the top of my mind. (full disclosure -- Eyetools is
also a Tobii reseller, though you can buy direct from Tobii also).

Cost: $44,000+ for a typical usability set up including hardware and
software necessary for doing web testing

3) You can rent a system: this raises the larger issue of "will you know how
to get valid data and analyze it correctly" which no matter what I say no
one will believe since I'm in the business of providing services. However,
experience does count, as you can well imagine. There have been some
resources appearing on the web to support people figuring it out on their
own, though it's not clear that they'll actually truly teach you all you
need to know, so again, you're on your own. This all goes to the second part
of the question which is "is it complicated" and one can look at this
discussion thread to decide for themselves.

Cost: around $7,400 for renting for two months, running 4 studies (prices
vary depending on how many studies you run -- the more you run, the more you
pay).

Best regards,
-Greg

Greg Edwards
CEO & Founder, Eyetools Inc.
greg at eyetools.com
916.792.4538

25 Apr 2008 - 3:09pm
Oleh Kovalchuke
2006

On Tue, Apr 22, 2008 at 7:23 AM, Todd Zaki Warfel <lists at toddwarfel.com>
wrote:

> Oh, oh, oh, I know—what we need is eye tracking with mind reading.
> Now, that's useful.
>

This is not too far fetched.

Jared is right: there is no 100% correlation between eye fixation and locus
of attention (or understanding what is being viewed). That can be solved
soon enough:

"The scientists used a functional magnetic resonance imaging machine -- a
real-time brain scanner -- to record the mental activity of a person looking
at thousands of random pictures: people, animals, landscapes, objects, the
stuff of everyday visual life. With those recordings the researchers built a
computational model for predicting the mental patterns elicited by looking
at any other photograph. When tested with neurological readouts generated by
a different set of pictures, the decoder passed with flying colors,
identifying the images seen with unprecedented accuracy. "
http://www.wired.com/science/discoveries/news/2008/03/mri_vision

I assume the image of shopping cart would elicit different pattern from the
image of login field or the ad banner, since they have different meaning to
the website visitor.

--
Oleh Kovalchuke
Interaction Design is design of time
http://www.tangospring.com/IxDtopicWhatIsInteractionDesign.htm

25 Apr 2008 - 3:38pm
Oleh Kovalchuke
2006

And here is one company, which can do the mind reading for you (as a side
project they can find out what your test subjects think about Saddam, W.
Bush, their reaction to pain etc.):
http://ahe6.tripod.com/cognitive.eng/id45.html

On Fri, Apr 25, 2008 at 2:09 PM, Oleh Kovalchuke <tangospring at gmail.com>
wrote:

> On Tue, Apr 22, 2008 at 7:23 AM, Todd Zaki Warfel <lists at toddwarfel.com>
> wrote:
>
>
> > Oh, oh, oh, I know—what we need is eye tracking with mind reading.
> > Now, that's useful.
> >
>
>
> This is not too far fetched.
>
> Jared is right: there is no 100% correlation between eye fixation and
> locus of attention (or understanding what is being viewed). That can be
> solved soon enough:
>
> "The scientists used a functional magnetic resonance imaging machine -- a
> real-time brain scanner -- to record the mental activity of a person looking
> at thousands of random pictures: people, animals, landscapes, objects, the
> stuff of everyday visual life. With those recordings the researchers built a
> computational model for predicting the mental patterns elicited by looking
> at any other photograph. When tested with neurological readouts generated by
> a different set of pictures, the decoder passed with flying colors,
> identifying the images seen with unprecedented accuracy. "
> http://www.wired.com/science/discoveries/news/2008/03/mri_vision
>
> I assume the image of shopping cart would elicit different pattern from
> the image of login field or the ad banner, since they have different meaning
> to the website visitor.
>
> --
> Oleh Kovalchuke
> Interaction Design is design of time
> http://www.tangospring.com/IxDtopicWhatIsInteractionDesign.htm
>

Syndicate content Get the feed