Is Eye Tracking too expensive or complicated?

Hey All,

I just put together about Eye Tracking
( http://www.insideria.com/2008 /04 /is -eye -tracking -out -of -reach.html ) but I thought I'd put the question out to the list:

"Why isn't eye tracking used more in design and testing of rich internet applications?"

I really have no experience with it, but am going to be working with it this summer so looking for thoughts from some of you more experienced with usability testing and eye tracking. I think for me it's been cost and lack of knowledge/acccess that's kept me away from it in the past.

Cheers!

-- André Charland
President and Co-Founder, Nitobi
e. andre.charland at gmail.com skype. ebadre
b. http://blogs.nitobi.com/andre
w. http://www.nitobi.com

At the Information Architecture conference in Miami last week, Jared Spool imparted that eye tracking is just fluff to get the executives excited about UCD.... he doesn't value it as a science beyond the obvious. We already know that people don't look at advertisements and avoid certain areas of the page.

While it will look good on a resume to have the experience, I tend to agree with Jared.

Thanks for sending this Andre, really interesting post.

I was watching a user session this week in one room at the Ux agency we use and the moderator announced that, in the second room, they had setup eye tracking so we could observe that. Everyone else rushed through to watch it and I sat where I was as I have never gained anything useful from eye-tracking in real-time.

That said, working out why someone has missed a header or navigation element due to an imbalance in perceived-affordance or other visual priority is made somewhat easier by the aggregated results of eye-tracking.

Original Message
From: discuss-bounces at lists.interactiondesigners.com [mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of Andre Charland
Sent: 18 April 2008 00:11
To: discuss at ixda.org
Subject: [IxDA Discuss] Is Eye Tracking too expensive or complicated?

Hey All,

I just put together about Eye Tracking
( http://www.insideria.com/2008 /04 /is -eye -tracking -out -of -reach.html ) but I thought I'd put the question out to the list:

"Why isn't eye tracking used more in design and testing of rich internet applications?"

Eye tracking is very expensive for what it delivers, and, as with all analytics, data interpretation can be difficult because you know nothing about user motivation or intent. IMHO, analytics tools like session recording and click maps can provide somewhat similar data for a MUCH lower cost, both in terms of price and effort.

-eva

-- http://www.linkedin.com/in/kaniasty

...oops I meant Design B would be the better option (assuming less visual workload is preferable).

Posted from the new ixda.org
http://www.ixda.org/discuss?post=28208

An interesting question, which begs for me to plug something I am co-presenting at UPA entitled, "Practical Eyetracking."

https://www.usabilityprofessionals.org/conference/2008/program/friday.htm

If Jared Spool has been paraphrased correctly, I disagree that eyetracking is just fluff. As Rob Tannen hinted at, eyetracking can be very helpful in determining why things happened. Usually we have intuition do that, but often it is not obvious.

Here's one example, we tested a design where people couldn't find the link to login to the site, even though it was in the main navigation. Eyetracking showed very clearly that people scanned the first few items in the navigation and then moved on without seeing the link. Seeing exactly where they stopped looking helped us understand why. And having a visual to show it helped us convince the client to change their design. Without eyetracking, we might have thought that the label was simply wrong.

The ROI is a separate discussion, but there are two things holding eyetracking back: cost and the additional time it takes to plan studies and to analyze the findings. Compared to alternatives, the ROI is pretty low. But if you have one already, or have a bigger budget, it can be very useful. And don't discount impressing the execs with great visuals.

Paul Nuschke
Electronic Ink

On Thu, Apr 17, 2008 at 7:11 PM, Andre Charland andre.charland at gmail.com wrote:

Hey All, I just put together about Eye Tracking ( http://www.insideria.com/2008 /04 /is -eye -tracking -out -of -reach.html ) but I thought I'd put the question out to the list: "Why isn't eye tracking used more in design and testing of rich internet applications?" I really have no experience with it, but am going to be working with it this summer so looking for thoughts from some of you more experienced with usability testing and eye tracking. I think for me it's been cost and lack of knowledge/acccess that's kept me away from it in the [trim]

Eye tracking is just like tracking mouse movement or clicks. It doesn't really shows you what users are thinking, they're just secondary manifestations of their thoughts. It's just like when you twiddle your fingers on a table while thinking about what to do next. It has nothing to do with it. Granted, there's some usefulness in the data, since you can uncover some problems, but generally sitting with a user and understanding it's train of thought is much more insightful.

Posted from the new ixda.org
http://www.ixda.org/discuss?post=28208

I didn't say that I thought eye tracking was fluff.

I said that I thought it is a voodoo technique. Deducing information about a design from eyetracking is equivalent to reading tea leaves and using a ouija board.

The latter are cheaper, but just as reliable.

Every person I know who swears by eyetracking and has stories on how its helped them can't explain how they would've gotten the same results if some other professional had looked at the same raw data. Until we can get to that point, the reader of the data will be more important than the data itself, thereby making tea leaf reading a viable alternative.

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845 e: jspool at uie.com p: 1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

Posted from the new ixda.org
http://www.ixda.org/discuss?post=28208

Have seen the phenomenon where because someone is using "Eye Tracking Equipment" they are automatically given more status or overused because of the "coolness factor" of just having the equipment (independent of what the results mean and how other methods can help compliment the results)

Also good thread about it here - http://www.ixda.org/discuss.php?post=22825 &search=eye +tracking

rgds, -- Daniel Szuc
Principal Usability Consultant
Apogee Usability Asia Ltd
www.apogeehk.com
Usability in Asia

The Usability Kit - www.theusabilitykit.com

On 19 Apr 2008, at 6:46 PM, Jared M.Spool wrote:

I didn't say that I thought eye tracking was fluff. I said that I thought it is a voodoo technique. Deducing information about a design from eyetracking is equivalent to reading tea leaves and using a ouija board. The latter are cheaper, but just as reliable. Every person I know who swears by eyetracking and has stories on how its helped them can't explain how they would've gotten the same results if some other professional had looked at the same raw data. Until we can get to that point, the reader of the data will be more [trim]

Yes, but Jared —
There are at least 2 full sessions @ June's UPA conference dedicated to eye-tracking, ergo it must be a valid technique! Maybe next year we will have a "Tea-Leaf Reading Analytical Practices for Enhanced User Experience, " which will follow "Rapid A-B testing with Mescaline & Electroshock Therapy: Getting a Charge Out of User Testing" session.

On Sat, Apr 19, 2008 at 9:46 PM, Jared M. Spool jspool at uie.com wrote:

I didn't say that I thought eye tracking was fluff. I said that I thought it is a voodoo technique. Deducing information about a design from eyetracking is equivalent to reading tea leaves and using a ouija board. The latter are cheaper, but just as reliable. Every person I know who swears by eyetracking and has stories on how its helped them can't explain how they would've gotten the same results if some other professional had looked at the same raw data. Until we can get to that point, the reader of the data will be more [trim]

—
~ will

"Where you innovate, how you innovate,
and what you innovate are design problems"

Will Evans | User Experience Architect
tel +1.617.281.1281 || wkevans4 at gmail.com

At 10:27 PM -0400 4/19/08, Will Evans wrote:
Yes, but Jared — There are at least 2 full sessions @ June's UPA conference dedicated to eye-tracking, ergo it must be a valid technique!

Not valid , but accepted. Surely we are all familiar with the difference between those two....And I suspect you know it : ) If not...I know an excellent phrenologist

Katie

Maybe next year we will have a "Tea-Leaf Reading Analytical Practices for Enhanced User Experience, " which will follow "Rapid A-B testing with Mescaline & Electroshock Therapy: Getting a Charge Out of User Testing" session. On Sat, Apr 19, 2008 at 9:46 PM, Jared M. Spool jspool at uie.com wrote: I didn't say that I thought eye tracking was fluff. I said that I thought it is a voodoo technique. Deducing information about a design from eyetracking is equivalent to reading tea leaves and using a ouija board. The latter are cheaper, but just as reliable. [trim]

—

Katie Albers
katie at firstthought.com

Wow, what skepticism in this thread! I'll admit that I don't use my eye tracker as often as split testing, but I do feel compelled to offer a more positive view on the matter.

The # 1 utility of the eye tracker, for me, is in helping me understand the user's cognition during a test session. With real time gaze data on a 2nd observer only screen, I don't have to work as hard on eliciting verbal protocol. I have also used the eye gaze reports to ask the user questions after the session — a methodology others have developed more fully.

Heck, I even spotted "button gravity" in my lab: http://flickr.com/photos/andyed/450579101/

Bruno: Regarding mouse movements, it's clear that eye movements are much higher signal, but mouse position has more data than twiddled fingers. I've summarized research on this on my blog and in a recent publication: http://alwaysbetesting.com/abtest /index.cfm /2007 /4 /29 /Eye -Tracking -vs -Mouse -Tracking

There's a longstanding and largely unsuccessful effort to generate quantitative quality metrics from eye-tracking data. That said, distance traveled by eye has been used productively in LukeW's work on forms, presented at Jared's Web App Summit recently. I've also been able to show good design leads to more efficient scan paths, http://flickr.com/photos/22582943@N03/2175663626/.

I won't dispute that many of the insights from eye-tracking are fairly obvious (ex. no headings in a long menu?
http://flickr.com/photos/22582943@N03/2177600531/ ), but there's something to be said for how well the visualizations engage consumers.

To help with these basic types of insights, we've developed a vision simulation in a browser, "Stomper Scrutinizer" that helps reveal the multiple fixation requirements of left aligned form labels for example.

Andy

Bruno Figueiredo wrote:
Eye tracking is just like tracking mouse movement or clicks. It doesn't really shows you what users are thinking, they're just secondary manifestations of their thoughts. It's just like when you twiddle your fingers on a table while thinking about what to do next. It has nothing to do with it. Granted, there's some usefulness in the data, since you can uncover some problems, but generally sitting with a user and understanding it's train of thought is much more insightful. Posted from the new ixda.org http://www.ixda.org/discuss?post=28208 Welcome to the Interaction Design [trim]

Speaking of failed pseudo-science - I had the unfortunate opportunity to see Ben Stein's Polemic Excretion "Expelled" on Friday evening....sigh....

On Sat, Apr 19, 2008 at 11:54 PM, Katie Albers katie at firstthought.com wrote:

At 10:27 PM -0400 4/19/08, Will Evans wrote: Yes, but Jared — There are at least 2 full sessions @ June's UPA conference dedicated to eye-tracking, ergo it must be a valid technique! Not valid , but accepted. Surely we are all familiar with the difference between those two....And I suspect you know it : ) If not...I know an excellent phrenologist Katie Maybe next year we will have a "Tea-Leaf Reading Analytical Practices for Enhanced User Experience, " which will follow "Rapid A-B testing with Mescaline & Electroshock Therapy: Getting a [trim]

—
~ will

"Where you innovate, how you innovate,
and what you innovate are design problems"

Will Evans | User Experience Architect
tel +1.617.281.1281 || wkevans4 at gmail.com

On Apr 19, 2008, at 2:46 PM, Jared M.Spool wrote:
Every person I know who swears by eyetracking and has stories on how its helped them can't explain how they would've gotten the same results if some other professional had looked at the same raw data.

Nor can they explain why they wouldn't get better results and better recommendations from simply showing the UI to a half-decent user interface designer for 20 minutes.

I've never seen a eyetracking recommendation that wasn't either (a) patently obvious to me ("Your 6-pixel-high light gray text should be made easier to see" ) or (b) completely stupid ("Move the search box to the left (where you currently have the picture of the cute little puppy) because everyone seems to spends time looking at the left side of the page" ).

-Cf

Christopher Fahey
Behavior
http://www.behaviordesign.com
212.532.4002 x203
646.338.4002 mobile

At the cog sci lab when I was in grad school the eye tracking equipment pretty much sat in the corner collecting dust. The general feeling was that we had leaned all we could from that technology years ago. We generally knowhow people see.

It does have a great 'gee wiz' effect when you show and tell. All the execs think it is really cool.

On Apr 20, 2008, at 9:06 AM, Andy Edmonds wrote:

Wow, what skepticism in this thread! I'll admit that I don't use my eye tracker as often as split testing, but I do feel compelled to offer a more positive view on the matter.

Jared said:

"I said that I thought it is a voodoo technique. Deducing information about a design from eyetracking is equivalent to reading tea leaves and using a ouija board."

That's a pretty colorful exaggeration.

Eyetracking lets you see where people are looking in real time. Without considering post-test analysis, this has real value in helping the facilitator better understand what is happening without interfering. One analogy I find useful, in terms of understanding what the participant is doing/thinking, is that having eyetracking versus not having eyetracking is like testing in person versus testing remotely.

I wonder, given your research background, Jared, if we are talking about different types of eyetracking studies. For academic/generalizable research, I have found eyetracking studies to be pretty meaningless. But for testing real products, and only trying to interpret results for those pages, it can be useful and not all that difficult, depending on the stimulus and tasks of course.

I also wonder if some people have been burned by past bad experiences with faulty eyetrackers and bad software. My lab at school had three separate eyetrackers and none of them worked correctly. The Tobii one that I use now is easy to use and the analysis software is very good.

Paul

On Sat, Apr 19, 2008 at 9:46 PM, Jared M. Spool jspool at uie.com wrote:

I didn't say that I thought eye tracking was fluff. I said that I thought it is a voodoo technique. Deducing information about a design from eyetracking is equivalent to reading tea leaves and using a ouija board. The latter are cheaper, but just as reliable. Every person I know who swears by eyetracking and has stories on how its helped them can't explain how they would've gotten the same results if some other professional had looked at the same raw data. Until we can get to that point, the reader of the data will be more [trim]

I'm wondering about the need of eye tracking on any particular page.

I mean : it's nice to repeat scientific tests over and over again to control the results.

But I don't think it could add specific value to website analysis as such to do it on every site over and over again.

I don't see why the results would differ from a general analysis of the website based on the principles obtained in research previously.

But, hey, if you can convince a client more easily to put his/her money into your pocktets showing a nice reddish glow on a graph... who I am to stop you ?

Pieter Jansegers

http://webosophy.ning.com

Posted from the new ixda.org
http://www.ixda.org/discuss?post=28208

Paul Nuschke wrote:
Eyetracking lets you see where people are looking in real time.

Yes. But just because you know where someone looks or doesn't look doesn't mean you know anything about what they see, what they wanted to see, and what they didn't see. It's not clear to me how one interprets the "they gazed at this point on the screen for 400 ms" information. Was that good? Was that bad?

We know that people see things through their peripheral vision, such as the scroll bar, so that's not recorded by the eye tracker. That means we can't even assume that when someone doesn't gaze at a spot that it wasn't seen.

With eye trackers, we have a bunch of observations but no way to determine the proper inferences. Instead, all of the value of an eye tracker comes from the interpretation.

Show me a study that shows that N separate evaluators looked at the same eye tracking data and came away with the same conclusions and I'll change my mind.

Until then, I'll continue to group it with tarot cards and palm reading as a fine art.

Without considering post-test analysis, this has real value in

helping the facilitator better understand what is happening without interfering.

Exactly my point. As the President of Best Buy, John "JT" Thompson, once told me (while I was delivering a great presentation with a ton of data):

"I worked for Jack Welch at GE for 17 years and if I learned anything while I was there, it was this: If you torture data long and hard enough, it will confess to anything you want."

One analogy I find useful, in terms of understanding what the

participant is doing/thinking, is that having eyetracking versus not having eyetracking is like testing in person versus testing remotely.

You lost me there.

I wonder, given your research background, Jared, if we are talking

about different types of eyetracking studies. For academic/generalizable research, I have found eyetracking studies to be pretty meaningless.

Actually, that's pretty funny. I think the most exciting eye tracking stuff is happening in research. There were a ton of good posters and some neat presentations at CHI showing how eye tracking, as an alternative input device, could have some really cool applications, especially for accessibility.

I also think there are some interesting cognitive and behavioral psych things to learn by using the devices. But I don't think there's been anything useful in terms of using it as a tool to enhance or inform the design process, so I'm guessing we agree there.

But for testing real products, and only trying to interpret results

for those pages, it can be useful and not all that difficult, depending on the stimulus and tasks of course.

Yah, not seeing that.

What I see is that it falls nicely in the "If you can't dazzle 'em with your brilliance, feel free to baffle 'em with your bullshit" category of helping folks understand how to change their designs.

But then again, what do I know?

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845 e: jspool at uie.com p: 1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

Posted from the new ixda.org
http://www.ixda.org/discuss?post=28208

I have been observing (pardon the pun) how often people look right at something and don;t see it. I am sure there is a technical term for this 'attention periphery' but I have not found it in the research yet. I would love to see the results and analysis of an eye tracking expert of subject watching the now classic dancing bear in the basketball game.

Mark

http://www.dothetest.co.uk/

On Apr 20, 2008, at 4:56 PM, Paul Nuschke wrote:

Eyetracking lets you see where people are looking in real time.

On Apr 20, 2008, at 6:18 PM, mark schraad wrote:

I am sure there is a technical term for this 'attention periphery' but I have not found it in the research yet.

Search for "situation inattentional blindness". The primary work was done by Simons at U of Illinois at Urbana-Champagne.

Jared

Mark said:
I would love to see the results and analysis of an eye tracking expert of subject watching the now classic dancing bear in the basketball game.

Here's one study: http://andrewd.ces.clemson.edu/courses /cpsc412 /fall04 /teams /group1 _hfes.pdf

Justine

On Sun, Apr 20, 2008 at 5:50 PM, Jared M. Spool jspool at uie.com wrote:

Paul Nuschke wrote: Eyetracking lets you see where people are looking in real time. Yes. But just because you know where someone looks or doesn't look doesn't mean you know anything about what they see, what they wanted to see, and what they didn't see. It's not clear to me how one interprets the "they gazed at this point on the screen for 400 ms" information. Was that good? Was that bad?

Imagine that a user needs to click on a link to go somewhere. If she fixates on the link and don't click it, then that's pretty good evidence that she did not understand the link.

We know that people see things through their peripheral vision, such as the scroll bar, so that's not recorded by the eye tracker. That means we can't even assume that when someone doesn't gaze at a spot that it wasn't seen.

True, but that's a good thing. You can't read or see fine details in your peripheral vision, so even if you notice something it doesn't mean that you looked at it enough to understand what it contained (unless the important details were very big). In the example above, even if the user noticed that a link existed, if she did not attend to it, then she would not have been able to read it.

Show me a study that shows that N separate evaluators looked at the same eye tracking data and came away with the same conclusions and I'll change my mind.

That some data does not make sense is not a phenomenon unique to eyetracking. I've seen plenty of different interpretations of statistics as well.

One analogy I find useful, in terms of understanding what the participant is doing/thinking, is that having eyetracking versus not having eyetracking is like testing in person versus testing remotely. You lost me there.

In remote testing, you loose voice quality and you don't see mannerisms, facial expressions, etc. In "in person" testing, you have gestures and facial expressions, and voice inflections. In eyetracking, you add the ability to see where they are looking. You lose something too, though, in your testing methodology, but that's another e-mail thread.

Paul

This may be obvious or trivial, but it was a new insight to me. I've never used eye tracking in software/web design or thought it would have much utility. Nonetheless I wandered up to one of the eye-tracking vendors at the CHI conference and got into a conversation with the rep. He said that most of their sales and emphasis were on contexts where there was something other than a single screen to look at — such as automobile dashboards, complex control panels like nuclear power plants, and the like. In those situations, seeing where people are looking in response to stimuli like alerts, gauges, oncoming obstacles etc., that can come from many different directions, is very important and the eye tracking apparatus can be extremely helpful. That made a lot of sense to me.

Al

On Apr 20, 2008, at 7:45 PM, Paul Nuschke wrote:

Imagine that a user needs to click on a link to go somewhere. If she fixates on the link and don't click it, then that's pretty good evidence that she did not understand the link.

All you know is that the eye tracker registered that they fixated on the link and that they didn't click.

The notion that they didn't understand the link is one inference.

It's not the only inference. It may not be the right inference.

It is purely your interpretation that the user didn't understand it.

(And you could've gotten there without the eye tracking data.)

We know that people see things through their peripheral vision, such as the scroll bar, so that's not recorded by the eye tracker. That means we can't even assume that when someone doesn't gaze at a spot that it wasn't seen. True, but that's a good thing. You can't read or see fine details in your peripheral vision, so even if you notice something it doesn't mean that you looked at it enough to understand what it contained (unless the important details were very big).

Again. Your inference. You don't have any evidence to actually know that's true.

In fact, in psychographic phenomena, it's pretty amazing what people can see and deduce from the peripheral vision. There's a lot happening within 140 degrees of the focal point.

And it's pretty amazing what is lost within the center gaze area, especially with people who have field issues that are frequent in males over 40, females over 50, and anyone suffering from optic neuritis or other immune-deficiency-based symptoms. (In MS patients, for example, optic neuritis frequently shows up in late teens, early 20s.)

So, you are just inferring meaning to the data you're collecting.

In the example above, even if the user noticed that a link existed, if she did not attend to it, then she would not have been able to read it.

Your inference. There are other likely inferences too.

Show me a study that shows that N separate evaluators looked at the same eye tracking data and came away with the same conclusions and I'll change my mind. That some data does not make sense is not a phenomenon unique to eyetracking. I've seen plenty of different interpretations of statistics as well.

Ok. Does that make eyetracking work?

Not buying it. Still think it's up to the interpreter of the eye tracker.

Let me put it another way:

Would you, Paul, be comfortable letting your clients to use the eye tracker without any help in interpreting data from you. Is the device all they need to make the judgments necessary to provide good design advice?

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845 e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

On Sun, Apr 20, 2008 at 11:43 PM, Jared M. Spool jspool at uie.com wrote:

All you know is that the eye tracker registered that they fixated on the link and that they didn't click. The notion that they didn't understand the link is one inference. It's not the only inference. It may not be the right inference. It is purely your interpretation that the user didn't understand it.

One way of double-checking the inferences is to ask the participant. What I've observed in eyetracking has been confirmed by participants enough that I know that the premises of eyetracking are true. Seems like you have had a different experience, so I'll be curious to hear what people have to say at UPA about it.

(And you could've gotten there without the eye tracking data.)

This assumes that you knew there was an issue to begin with, or that the type of study allows you to follow up. Neither is always the case.

In fact, in psychographic phenomena, it's pretty amazing what people can see and deduce from the peripheral vision. There's a lot happening within 140 degrees of the focal point.

And it's pretty amazing what is lost within the center gaze area, especially with people who have field issues that are frequent in males over 40, females over 50, and anyone suffering from optic neuritis or other immune-deficiency-based symptoms. (In MS patients, for example, optic neuritis frequently shows up in late teens, early 20s.)

Sounds interesting, do you have a link?

Is the device all they need to make the judgments necessary to provide good design advice?

Of course not. In nowhere here have I said that eyetracking was the only way to make judgments. It's just another tool. IMO, the main problems with eyetracking are 1) the multiple participant data (heatmaps) doesn't always make sense, 2) it is time consuming to use, and 3) the initial cost of the equipment is ridiculously high for the benefit that you get. It is not that the premises are wrong. On this point I think that we disagree, so let's just leave it there.

Paul

Sounds like what's missing here is a set of consistent, objective and reliable guidelines for interpreting eye-tracking data (and potentially usability findings in general).

For example a fixation of an a priori specified minimum duration on a link in conjunction with a user failing to click the link AND the user reporting that the link was seen would strongly indicate that indeed it was seen : )

Posted from the new ixda.org
http://www.ixda.org/discuss?post=28208

On Apr 20, 2008, at 3:54 PM, Christopher Fahey wrote:

Nor can they explain why they wouldn't get better results and - better* recommendations from simply showing the UI to a half-decent user interface designer for 20 minutes

Eye-tracking should not be used on its own—if used at all, it should be for supplementary input. Yes, it tells you what someone is looking at, but doesn't tell you why—and the "why" is critical to finding the appropriate design solution.

Cheers!

Todd Zaki Warfel
President, Design Researcher
Messagefirst | Designing Information. Beautifully. Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
Twitter: zakiwarfel
In theory, theory and practice are the same.
In practice, they are not.

In the end, it's on what people click, which is really important.

And even more important is the information people are really looking for and the findability

of that information.

It all comes down to offering the right information content in a nice way.

Without the right information, your site may be optimized the way you want, people won't be

happy.

It's a bit back to basics, but basics are extremely important.

Pieter Jansegers
http://webosophy.ning.com

Jared,

Eye tracking is of value when someone sees something in the data that leads to a new and valuable insight.

The fact that different observers see different things in the same raw eye tracking data is of no more concern to me than the fact that different players count a different number of words on the same Boggle board. Some people see words that are hidden in plain sight; some do not. But noticed or not, the words are there. In the tea leaves, there are no hidden words.

Adding observers to any team that is interpreting raw data makes it more likely that someone will notice something subtle, but important, that other observers have missed.

This is as true in design research as it is in radiology, geophysicists or espionage.

Larry Tesler

On Apr 19, 2008, at 6:46 PM, Jared M.Spool wrote:

... Deducing information about a design from eyetracking is equivalent to reading tea leaves and using a ouija board. The latter are cheaper, but just as reliable. Every person I know who swears by eyetracking and has stories on how its helped them can't explain how they would've gotten the same results if some other professional had looked at the same raw data. Until we can get to that point, the reader of the data will be more important than the data itself, thereby making tea leaf reading a viable alternative.

On Apr 22, 2008, at 3:08 AM, Larry Tesler wrote:

The fact that different observers see different things in the same raw eye tracking data is of no more concern to me than the fact that different players count a different number of words on the same Boggle board. Some people see words that are hidden in plain sight; some do not. But noticed or not, the words are there. In the tea leaves, there are no hidden words.

Larry,

I have no doubt that the observations are of interest.

My point is that the inferences drawn from those observations have little-to-no validity, thus the tea leaf analogy.

If someone fixates on a link for a unusually large time, does that mean they are confused by it? Or they aren't confused, but are trying to decide if its what they want? Or they know whether they want it or not but are considering something else?

Different inferences will lead to completely different design solutions. Are you saying it doesn't matter which inference (and therefore, which design solution) the observers choose?

When you back an eye-tracking supporter into a corner about this, they all say, "Well, you should only use eye tracking in conjunction with other data collection tools and techniques to verify your inferences." In almost all cases, the "other data collection tools and techniques" would yield just as much value without the eye tracking as with it, so what's the benefit?

Second, in almost all uses of eye tracking I've seen in the last 5 years, it's in the form of twisting the meaning of the heatmap/plot diagram/tea leaf reading into supporting whatever wacky inference the specialist wants to support. "See that big red spot there. That means the users are confused"
v. "See that big red spot there, that means we fixed the design."

If there really is something to this eye tracking thing, I'd think you'd want your team members to all look at the same heat map and come to somewhat similar design implications.

Eyetracking equipment: $30,000
Ouija Board: $5
Quality design based on solid inferences from rich, meaningful data: Priceless

That's my take.

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845 e: jspool at uie.com p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

On Apr 22, 2008, at 8:59 AM, Jared M. Spool wrote:

If someone fixates on a link for a unusually large time, does that mean they are confused by it? Or they aren't confused, but are trying to decide if its what they want? Or they know whether they want it or not but are considering something else?

Oh, oh, oh, I know—what we need is eye tracking with mind reading. Now, that's useful.

Cheers!

Todd Zaki Warfel
President, Design Researcher
Messagefirst | Designing Information. Beautifully. Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
Twitter: zakiwarfel
In theory, theory and practice are the same.
In practice, they are not.

Todd wrote:

Oh, oh, oh, I know—what we need is eye tracking with mind

reading. Now, that's useful.

Hah! If we had mind reading, we wouldn't need the eye tracker.

Jared

Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845 e: jspool at uie.com p: 1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks

Posted from the new ixda.org
http://www.ixda.org/discuss?post=28208

Jared,

If most readers think this has gone on too long, we should wind it down.

On Apr 22, 2008, at 5:59 AM, Jared M. Spool wrote: On Apr 22, 2008, at 3:08 AM, Larry Tesler wrote: The fact that different observers see different things in the same raw eye tracking data is of no more concern to me than the fact that different players count a different number of words on the same Boggle board. Some people see words that are hidden in plain sight; some do not. But noticed or not, the words are there. In the tea leaves, there are no hidden words. Larry, I have no doubt that the observations are of [trim]

I consider them valid if they inspire us to make design changes that lead to improvements in objective metrics.

In any study, with or without an eye tracker, I look for:

If someone fixates on a link for a unusually large time, does that mean they are confused by it? Or they aren't confused, but are trying to decide if its what they want? Or they know whether they want it or not but are considering something else?

If the user's mental state matters to you, ask the user what it was. They may know. If they do not know, devise a more clever experiment.

But sometimes, the user's mental state doesn't matter. We may have run the test because too few people were clicking on the link. We thought perhaps they didn't even look at the area of the page that contained the link. The tracker has refuted our hypothesis. We know that some people look straight at the link and still do not click it. Other data may be needed if we want to find out why. But the study was a success. It achieved its goal.

Different inferences will lead to completely different design solutions. Are you saying it doesn't matter which inference (and therefore, which design solution) the observers choose?

If the radiologists call your malignant tumor benign, or vice versa, you may receive the wrong treatment, which could be a costly mistake. But design changes that eye tracking studies inspire often entail simple modifications to layout, color, size, typeface, etc., that help to steer attention. They are often cheap to implement. If there are two competing inferences, you can often try both implied solutions.

Of course, if you had both designs in mind (and particularly the one that ultimately proved to be best) before you ran the eye tracking study, then the study was a waste of time. If that is the situation you are in, and you never want to run a study that may simply confirm that you were right, then your point is valid. But it is not always the situation. There may be no unrefuted theory about why users are not clicking the link. There may be no considered designs that would increase clicks. There may be too many credible designs—more than one has the time and staff to implement and test. Or you may simply want to confirm other data or hunches.

When you back an eye-tracking supporter into a corner about this, they all say, "Well, you should only use eye tracking in conjunction with other data collection tools and techniques to verify your inferences." In almost all cases, the "other data collection tools and techniques" would yield just as much value without the eye tracking as with it, so what's the benefit?

The eye tracking test may have been the source of the first clue as to what ails your interface. Without it, you may never have thought to try those other tools and techniques.

Or the eye tracking test may help to rule out hypotheses that one has generated from the use of other tools and techniques.

Second, in almost all uses of eye tracking I've seen in the last 5 years, it's in the form of twisting the meaning of the heatmap/plot diagram/tea leaf reading into supporting whatever wacky inference the specialist wants to support. "See that big red spot there. That means the users are confused"v. "See that big red spot there, that means we fixed the design."

I agree that features of the heat map don't tell you mental states. But if they are inconsistent with hypotheses about mental states, they may call those hypotheses into question. "See that big red spot over there. Maybe they were looking at the link after all."

A big red spot may also provide evidence that the design is improved— if the problem with the design was that nobody was looking at that link. But there are usually more convincing methods of validating designs than looking at heat maps, e.g., bucket tests with well-chosen metrics.

If there really is something to this eye tracking thing, I'd think you'd want your team members to all look at the same heat map and come to somewhat similar design implications.

I thought my previous email, with its counterexamples, refuted the assumption that independent observers of an image must come to the same separate conclusions for the methodology to be useful. I guess we'll have to agree to disagree about that.

Eyetracking equipment: $30,000

Amortized over N studies = $30,000/N. If your business is large, your benefit may be large, too, and you could be getting a very good ROI.

Ouija Board: $5 Quality design based on solid inferences from rich, meaningful data: Priceless

If your clients' markets are too small to make eye tracking investments pay off, and they have other sources of data that give them useful insights, then I can see why you don't advocate eye tracking. But for a client selling a high volume product, especially on the web where switching costs are low, small changes can have large effects, and eye tracking can pay off.

Larry Tesler

As others have already pointed out, the primary benefactor of eye-tracking studies is the coordinator of the studies. After many such studies, I gradually have absorbed what textbooks could not teach me effectively about this science. What draws the human eye is not quite unpredictable, but the wide variance does not lend itself to easy rule-making.

Madison Avenue has been using this technology for years for somewhat nefarious purposes. We need to embrace it and put the knowledge to good use across the discipline of interaction and experience design. How could we tolerate the idea that it would be okay for marketing and advertising folks to know more about this area than ourselves?

Here's the rub though: I have always found it challenging to use the eye-tracking data to correct my own designs. When I tried to do this initially, I would never get it right: I would re-submit "corrected" designs for new eye-tracking and I'd get even worse results. Persistence was key.

Eye-tracking studies have made me a better designer. And my clients love it.

Dave

Original Message From: discuss-bounces at lists.interactiondesigners.com [mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of Larry Tesler Sent: Tuesday, April 22, 2008 11:22 PM To: Jared M. Spool Cc: discuss at ixda.org Subject: Re: [IxDA Discuss] Is Eye Tracking too expensive or complicated? Jared, If most readers think this has gone on too long, we should wind it down. On Apr 22, 2008, at 5:59 AM, Jared M. Spool wrote: On Apr 22, 2008, at 3:08 AM, Larry Tesler wrote: The fact that different observers see different things in the same raw eye [trim]

I consider them valid if they inspire us to make design changes that lead to improvements in objective metrics.

In any study, with or without an eye tracker, I look for:

If someone fixates on a link for a unusually large time, does that mean they are confused by it? Or they aren't confused, but are trying to decide if its what they want? Or they know whether they want it or not but are considering something else?

If the user's mental state matters to you, ask the user what it was. They may know. If they do not know, devise a more clever experiment.

But sometimes, the user's mental state doesn't matter. We may have run the test because too few people were clicking on the link. We thought perhaps they didn't even look at the area of the page that contained the link. The tracker has refuted our hypothesis. We know that some people look straight at the link and still do not click it. Other data may be needed if we want to find out why. But the study was a success. It achieved its goal.

Different inferences will lead to completely different design solutions. Are you saying it doesn't matter which inference (and therefore, which design solution) the observers choose?

If the radiologists call your malignant tumor benign, or vice versa, you may receive the wrong treatment, which could be a costly mistake. But design changes that eye tracking studies inspire often entail simple modifications to layout, color, size, typeface, etc., that help to steer attention. They are often cheap to implement. If there are two competing inferences, you can often try both implied solutions.

Of course, if you had both designs in mind (and particularly the one that ultimately proved to be best) before you ran the eye tracking study, then the study was a waste of time. If that is the situation you are in, and you never want to run a study that may simply confirm that you were right, then your point is valid. But it is not always the situation. There may be no unrefuted theory about why users are not clicking the link. There may be no considered designs that would increase clicks. There may be too many credible designs—more than one has the time and staff to implement and test. Or you may simply want to confirm other data or hunches.

When you back an eye-tracking supporter into a corner about this, they all say, "Well, you should only use eye tracking in conjunction with other data collection tools and techniques to verify your inferences." In almost all cases, the "other data collection tools and techniques" would yield just as much value without the eye tracking as with it, so what's the benefit?

The eye tracking test may have been the source of the first clue as to what ails your interface. Without it, you may never have thought to try those other tools and techniques.

Or the eye tracking test may help to rule out hypotheses that one has generated from the use of other tools and techniques.

Second, in almost all uses of eye tracking I've seen in the last 5 years, it's in the form of twisting the meaning of the heatmap/plot diagram/tea leaf reading into supporting whatever wacky inference the specialist wants to support. "See that big red spot there. That means the users are confused" v. "See that big red spot there, that means we fixed the design."

I agree that features of the heat map don't tell you mental states. But if they are inconsistent with hypotheses about mental states, they may call those hypotheses into question. "See that big red spot over there. Maybe they were looking at the link after all."

A big red spot may also provide evidence that the design is improved— if the problem with the design was that nobody was looking at that link. But there are usually more convincing methods of validating designs than looking at heat maps, e.g., bucket tests with well-chosen metrics.

If there really is something to this eye tracking thing, I'd think you'd want your team members to all look at the same heat map and come to somewhat similar design implications.

I thought my previous email, with its counterexamples, refuted the assumption that independent observers of an image must come to the same separate conclusions for the methodology to be useful. I guess we'll have to agree to disagree about that.

Eyetracking equipment: $30,000

Amortized over N studies = $30,000/N. If your business is large, your benefit may be large, too, and you could be getting a very good ROI.

Ouija Board: $5 Quality design based on solid inferences from rich, meaningful data: Priceless

If your clients' markets are too small to make eye tracking investments pay off, and they have other sources of data that give them useful insights, then I can see why you don't advocate eye tracking. But for a client selling a high volume product, especially on the web where switching costs are low, small changes can have large effects, and eye tracking can pay off.

Larry Tesler

Welcome to the Interaction Design Association (IxDA)! To post to this list . discuss at ixda.org
Unsubscribe .... http://www.ixda.org/unsubscribe
List Guidelines http://www.ixda.org/guidelines
List Help http://www.ixda.org/help

Andre,

In all of the discussion, the initial question got unanswered:

- Is eyetracking too expensive? *

"Expensive" is all relative. Currently, you have three options:

1) outsourced data collection and analysis: you can have someone expert in eyetracking collect and analyze your data for you. This has the benefit that it solves the problem many people brought up which is that experience counts and it's harder to analyze eyetracking data than most people realize. Eyetools, Inc. (www.eyetools.com) does this for a lot of clients and agencies (full disclosure: my company — I'm the founder and CEO). There are other companies as well (actually, quite a few usability companies have started offering eyetracking in the last year).

Cost: $2,800 - $5,000 including recruiting, incentives, and facilities for testing a homepage/landing page/etc. (I'm not going to go into details of pricing since this is probably not the appropriate place for that.)

2) You can buy your own equipment: there are a number of hardware manufactures out there with pricing ranging from $20k-$44k for the hardware and software necessary to run web usability tests. Hardware manufacturers include LC Technologies (www.eyegaze.com), Eyetech Digital Systems (www.eyetechds.com), SMI (www.smi.com), ASL (www.a-s-l.com, though recently they've been doing less web usability stuff), and SmartEye (www.smarteye.se) are some that come to the top of my mind. (full disclosure — Eyetools is also a Tobii reseller, though you can buy direct from Tobii also).

Cost: $44,000+ for a typical usability set up including hardware and software necessary for doing web testing

3) You can rent a system: this raises the larger issue of "will you know how to get valid data and analyze it correctly" which no matter what I say no one will believe since I'm in the business of providing services. However, experience does count, as you can well imagine. There have been some resources appearing on the web to support people figuring it out on their own, though it's not clear that they'll actually truly teach you all you need to know, so again, you're on your own. This all goes to the second part of the question which is "is it complicated" and one can look at this discussion thread to decide for themselves.

Cost: around $7,400 for renting for two months, running 4 studies (prices vary depending on how many studies you run — the more you run, the more you pay).

Best regards,
-Greg

Greg Edwards
CEO & Founder, Eyetools Inc.
greg at eyetools.com
916.792.4538

On Tue, Apr 22, 2008 at 7:23 AM, Todd Zaki Warfel lists at toddwarfel.com wrote:

Oh, oh, oh, I know—what we need is eye tracking with mind reading. Now, that's useful.

This is not too far fetched.

Jared is right: there is no 100% correlation between eye fixation and locus of attention (or understanding what is being viewed). That can be solved soon enough:

"The scientists used a functional magnetic resonance imaging machine — a real-time brain scanner — to record the mental activity of a person looking at thousands of random pictures: people, animals, landscapes, objects, the stuff of everyday visual life. With those recordings the researchers built a computational model for predicting the mental patterns elicited by looking at any other photograph. When tested with neurological readouts generated by a different set of pictures, the decoder passed with flying colors, identifying the images seen with unprecedented accuracy. " http://www.wired.com/science /discoveries /news /2008 /03 /mri _vision

I assume the image of shopping cart would elicit different pattern from the image of login field or the ad banner, since they have different meaning to the website visitor.

-- Oleh Kovalchuke
Interaction Design is design of time
http://www.tangospring.com/IxDtopicWhatIsInteractionDesign.htm

And here is one company, which can do the mind reading for you (as a side project they can find out what your test subjects think about Saddam, W. Bush, their reaction to pain etc.):
http://ahe6.tripod.com/cognitive.eng/id45.html

On Fri, Apr 25, 2008 at 2:09 PM, Oleh Kovalchuke tangospring at gmail.com wrote:

On Tue, Apr 22, 2008 at 7:23 AM, Todd Zaki Warfel lists at toddwarfel.com wrote: Oh, oh, oh, I know—what we need is eye tracking with mind reading. Now, that's useful. This is not too far fetched. Jared is right: there is no 100% correlation between eye fixation and locus of attention (or understanding what is being viewed). That can be solved soon enough: "The scientists used a functional magnetic resonance imaging machine — a real-time brain scanner — to record the mental activity of a person looking at thousands of random pictures: [trim]