Testing for Learnability

22 May 2009 - 9:00am
5 years ago
6 replies
2177 reads
Kevin Silver
2010

Hi All,

After a long hiatus I'm back designing in the realm of enterprise
software and I have been shifting my thinking back to the land of the
intermediate user. Subsequently I've been thinking a lot about
designing and usability testing for learnability–how learnable is the
application? It seems to me that there is a big difference between
testing a check-out flow and a complex interface. I have some broad
assumptions in mind, but I'm sure there are many of you who have been
through this before.

So, my questions are:

How do you test for learnability?

Is there a difference between "learnability testing" and usability
testing? If so, what are they?

And maybe a pertinent question is how do you design for learnability?

Hopefully these are good friday and pre-holiday weekend questions.

Thanks,

Kevin

Comments

22 May 2009 - 10:43am
whitneyq
2010

The folks at Texas Tech have been working on this issue, and thinking
about how you adjust the flow of a session to test for both usability
and learnability.

David Edgell and Keisha McKenzie have an article coming out in UPA's
UX Magazine later this summer describing one of their projects at
Texas Tech. I'm sure they would be happy to talk to you about their
work. Their solution was to have the participant "teach back" key
tasks after using the software/site for a while.

The two qualities are intertwined. I'd say it's a question of
balancing the usability requirements: is it for speed, accuracy,
first-time learning, deeper learning, etc. I've written a bit about
this as the 5 Es (efficient, effective, engaing, error-tolerant, easy
to learn). The point is that "usability" is not a single, monolithic
quality, but must be defined for the "specific users" and "specific
contexts". This point is somewhat hidden in the ISO 9241 definition.

I've often wondered whether some of the debates about what "usability"
means really stem from conversations that cross contexts. Surely
usability for a (for example) shopping site is different than for a
(for example) equipment control panel, even though it may be composed
of similar attributes in different proportions.

There are some good techniques to help identify priorities, such as a
"game" in which the team is asked to spend $100 in play money on the
different Es. The end result is interesting, but the debate and
discussion is often very helpful in clarifying objectives.

More here: http://www.wqusability.com/articles/getting-started.html

On Fri, May 22, 2009 at 10:00 AM, Kevin Silver <kevin at kevintsilver.com> wrote:
> Hi All,
>
> After a long hiatus I'm back designing in the realm of enterprise software
> and I have been shifting my thinking back to the land of the intermediate
> user. Subsequently I've been thinking a lot about designing and usability
> testing for learnability–how learnable is the application? It seems to me
> that there is a big difference between testing a check-out flow and a
> complex interface. I have some broad assumptions in mind, but I'm sure there
> are many of you who have been through this before.
>
> So, my questions are:
>
> How do you test for learnability?
>
> Is there a difference between "learnability testing" and usability testing?
> If so, what are they?
>
> And maybe a pertinent question is how do you design for learnability?
>
> Hopefully these are good friday and pre-holiday weekend questions.
>
> Thanks,
>
> Kevin
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

--
Whitney Quesenbery
www.wqusability.com

Storytelling for User Experience Design
www.rosenfeldmedia.com/books/storytelling

23 May 2009 - 5:29am
Chauncey Wilson
2007

Hello Kevin,

The first thing you need to consider is what type of learnability are
you considering. Consider the following types:

1. New user to a product with no domain knowledge.
2. New user to a product with domain knowledge.
3. Novice to Intermediate learning (have to define "intermediate")
4. Novice, Intermediate, expert user learning new features.
5. Expert learning.
6. Transfer learning (moving from one product to another in the same area).

I see another message by Whitney that lists some attributes related
to usability. She makes a great point. One additional conceptual
issue is that when you are talking about learnability, you are
considering the shape of the learning curve. To improve learnability
is to change the shape of the learning curve. This is a constant
discussion in user assistance/tech writing groups - what can we do to
help people learn the product faster - short videos, better tutorials,
just-in-time training (for new features perhaps)....?

There is an excellent paper by some of my Autodesk colleagues in the
ACM Digital Library that relates directly to your question. They look
quite deeply at "learnability".

Grossman, T., Fitzmaurice, G., and Attar, R. 2009. A survey of
software learnability: metrics, methodologies and guidelines. In
Proceedings of the 27th international Conference on Human Factors in
Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI '09.
ACM, New York, NY, 649-658.

Here is the abstract:

"It is well-accepted that learnability is an important aspect of
usability, yet there is little agreement as to how learnability should
be defined, measured, and evaluated. In this paper, we present a
survey of the previous definitions, metrics, and evaluation
methodologies which have been used for software learnability. Our
survey of evaluation methodologies leads us to a new
question-suggestion protocol, which, in a user study, was shown to
expose a significantly higher number of learnability issues in
comparison to a more traditional think-aloud protocol. Based on the
issues identified in our study, we present a classification system of
learnability issues, and demonstrate how these categories can lead to
guidelines for addressing the associated challenges."

Chauncey

On Fri, May 22, 2009 at 10:00 AM, Kevin Silver <kevin at kevintsilver.com> wrote:
> Hi All,
>
> After a long hiatus I'm back designing in the realm of enterprise software
> and I have been shifting my thinking back to the land of the intermediate
> user. Subsequently I've been thinking a lot about designing and usability
> testing for learnability–how learnable is the application? It seems to me
> that there is a big difference between testing a check-out flow and a
> complex interface. I have some broad assumptions in mind, but I'm sure there
> are many of you who have been through this before.
>
> So, my questions are:
>
> How do you test for learnability?
>
> Is there a difference between "learnability testing" and usability testing?
> If so, what are they?
>
> And maybe a pertinent question is how do you design for learnability?
>
> Hopefully these are good friday and pre-holiday weekend questions.
>
> Thanks,
>
> Kevin
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

24 May 2009 - 4:16pm
Alan James Salmoni
2008

Essentially, learnability can be seen with two aspects:

1) How well can a user recall what they did (hence the teach back
protocol suggested by Whitney above)
2) How well can a user recognise something (slightly but not entirely
different)

Using these, memory testing is one way to test learnability. This
doesn't account for when a system is so well learned that it becomes
automatic (like using short-cuts for cut, copy and paste).

Another problem you might face is at what level are you concerned? Do
you want to look at the keystroke level (ie, very micro) or at a
higher workflow level? And another is what kind of time period are
you looking at? Are you interested in whether your users can remember
one hour after? Or perhaps several months after?

Like any research, the fundamental thing to do is to get the right
questions before you do anything else. Once you know these, you can
determine the research method, what data to collect, how to analyse,
etc. Make the questions simple (make each one address a simple thing)
and everything else becomes clear.

All the best!

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42209

26 May 2009 - 3:23pm
Kevin Silver
2010

Thanks everyone who responded to my questions. It was all very helpful
and reaffirmed most of my assumptions.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=42209

27 May 2009 - 2:51am
Geeta Bose
2009

We conduct these tests but in slightly different contexts. We use learnability testing to test the effectiveness of eLearning courses and other learning material while usability testing of products help detect the problems that users face while using the product.

Therefore, for us the primary difference between learnability testing and usability testing lies in the goal of the testing as well as in the testing process.

While usability testing is task-based, learnability testing is not. During learnability testing, we observe the cognitive flow and accuracy of the course, how well information is structured such that the learner understands and comprehends the learning material. At the same time, we also observe learners stumble, click buttons, tabs, links, skip pages or instructions, interpret instructions correctly or incorrectly as they undergo the process of trying to learn.

Therefore, in learnability testing, the focus is both on usability issues such as affordance, ease of navigation, findability etc as well as on the cognitive aspect of the course such the flow of information, language, meaning, level of difficulty or challenge as the learner tries to understand information on each screen.

I believe this is the key difference between the two. It is a huge challenge to test the learnability of a learning material as it is difficult to test the cognitive impact of the course in the long run. Asking learners to recall or restate their understanding during the test is only a short-term measure and does not assure long term retention or application.

Our research team always summarizes this by saying - "in UTs we can clearly see what is going wrong and where but in learner testing its very difficult to see whether the learner understands the concepts, and if they do, whether they will be able to retain the concepts and apply them at their workplace later."

You can have a quick look at the typical learnability testing process (http://elearning.kern-comm.com/?p=201) that we follow. Am sure most us are familiar with the usability testing process!

28 May 2009 - 1:15am
Angel Marquez
2008

>>And maybe a pertinent question is how do you design for learnability?
I just came across a couple of paragraphs that reminded me of this post. The
author mentioned a rock climbing instructor having his students attempt to
climb one of those indoor rocks & lectured after on how to properly shift
your weight etc...

The book says he created a sort of R-mode to L-mode (right side, left
side..brain) flow.

I hope this helps and you are still accepting thoughts on this. I've been
apart of a few teaching tool projects and the thought never even crossed my
mind. HA. I think hands on then research works best for me. The only
complaint with that method would be not knowing what something is and trying
to find an answer if you jump right into it without having any prior
knowledge.

Test the waters with something that sinks fast before you dive in...

Syndicate content Get the feed