Testing for Learnability

Hi All,

After a long hiatus I'm back designing in the realm of enterprise software and I have been shifting my thinking back to the land of the intermediate user. Subsequently I've been thinking a lot about designing and usability testing for learnability–how learnable is the application? It seems to me that there is a big difference between testing a check-out flow and a complex interface. I have some broad assumptions in mind, but I'm sure there are many of you who have been through this before.

So, my questions are:

How do you test for learnability?

Is there a difference between "learnability testing" and usability testing? If so, what are they?

And maybe a pertinent question is how do you design for learnability?

Hopefully these are good friday and pre-holiday weekend questions.

Thanks,

Kevin

The folks at Texas Tech have been working on this issue, and thinking about how you adjust the flow of a session to test for both usability and learnability.

David Edgell and Keisha McKenzie have an article coming out in UPA's UX Magazine later this summer describing one of their projects at Texas Tech. I'm sure they would be happy to talk to you about their work. Their solution was to have the participant "teach back" key tasks after using the software/site for a while.

The two qualities are intertwined. I'd say it's a question of balancing the usability requirements: is it for speed, accuracy, first-time learning, deeper learning, etc. I've written a bit about this as the 5 Es (efficient, effective, engaing, error-tolerant, easy to learn). The point is that "usability" is not a single, monolithic quality, but must be defined for the "specific users" and "specific contexts". This point is somewhat hidden in the ISO 9241 definition.

I've often wondered whether some of the debates about what "usability" means really stem from conversations that cross contexts. Surely usability for a (for example) shopping site is different than for a (for example) equipment control panel, even though it may be composed of similar attributes in different proportions.

There are some good techniques to help identify priorities, such as a "game" in which the team is asked to spend $100 in play money on the different Es. The end result is interesting, but the debate and discussion is often very helpful in clarifying objectives.

More here: http://www.wqusability.com/articles/getting-started.html

On Fri, May 22, 2009 at 10:00 AM, Kevin Silver kevin at kevintsilver.com wrote: Hi All, After a long hiatus I'm back designing in the realm of enterprise software and I have been shifting my thinking back to the land of the intermediate user. Subsequently I've been thinking a lot about designing and usability testing for learnability–how learnable is the application? It seems to me that there is a big difference between testing a check-out flow and a complex interface. I have some broad assumptions in mind, but I'm sure there are many of you who have been through this before. So, my questions are: How do you test for learnability? [trim]

-- Whitney Quesenbery
www.wqusability.com

Storytelling for User Experience Design
www.rosenfeldmedia.com/books/storytelling

Hello Kevin,

The first thing you need to consider is what type of learnability are you considering. Consider the following types:

1. New user to a product with no domain knowledge.

2. New user to a product with domain knowledge.

3. Novice to Intermediate learning (have to define "intermediate" )

4. Novice, Intermediate, expert user learning new features.

5. Expert learning.

6. Transfer learning (moving from one product to another in the same area).

I see another message by Whitney that lists some attributes related to usability. She makes a great point. One additional conceptual issue is that when you are talking about learnability, you are considering the shape of the learning curve. To improve learnability is to change the shape of the learning curve. This is a constant discussion in user assistance/tech writing groups - what can we do to help people learn the product faster - short videos, better tutorials, just-in-time training (for new features perhaps)....?

There is an excellent paper by some of my Autodesk colleagues in the ACM Digital Library that relates directly to your question. They look quite deeply at "learnability".

Grossman, T., Fitzmaurice, G., and Attar, R. 2009. A survey of software learnability: metrics, methodologies and guidelines. In Proceedings of the 27th international Conference on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI '09. ACM, New York, NY, 649-658.

Here is the abstract:

"It is well-accepted that learnability is an important aspect of usability, yet there is little agreement as to how learnability should be defined, measured, and evaluated. In this paper, we present a survey of the previous definitions, metrics, and evaluation methodologies which have been used for software learnability. Our survey of evaluation methodologies leads us to a new question-suggestion protocol, which, in a user study, was shown to expose a significantly higher number of learnability issues in comparison to a more traditional think-aloud protocol. Based on the issues identified in our study, we present a classification system of learnability issues, and demonstrate how these categories can lead to guidelines for addressing the associated challenges."

Chauncey

On Fri, May 22, 2009 at 10:00 AM, Kevin Silver kevin at kevintsilver.com wrote: Hi All, After a long hiatus I'm back designing in the realm of enterprise software and I have been shifting my thinking back to the land of the intermediate user. Subsequently I've been thinking a lot about designing and usability testing for learnability–how learnable is the application? It seems to me that there is a big difference between testing a check-out flow and a complex interface. I have some broad assumptions in mind, but I'm sure there are many of you who have been through this before. So, my questions are: How do you test for learnability? [trim]

Essentially, learnability can be seen with two aspects:

1) How well can a user recall what they did (hence the teach back protocol suggested by Whitney above)

2) How well can a user recognise something (slightly but not entirely different)

Using these, memory testing is one way to test learnability. This doesn't account for when a system is so well learned that it becomes automatic (like using short-cuts for cut, copy and paste).

Another problem you might face is at what level are you concerned? Do you want to look at the keystroke level (ie, very micro) or at a higher workflow level? And another is what kind of time period are you looking at? Are you interested in whether your users can remember one hour after? Or perhaps several months after?

Like any research, the fundamental thing to do is to get the right questions before you do anything else. Once you know these, you can determine the research method, what data to collect, how to analyse, etc. Make the questions simple (make each one address a simple thing) and everything else becomes clear.

All the best!

Posted from the new ixda.org
http://www.ixda.org/discuss?post=42209

Thanks everyone who responded to my questions. It was all very helpful and reaffirmed most of my assumptions.

Posted from the new ixda.org
http://www.ixda.org/discuss?post=42209

We conduct these tests but in slightly different contexts. We use learnability testing to test the effectiveness of eLearning courses and other learning material while usability testing of products help detect the problems that users face while using the product.

Therefore, for us the primary difference between learnability testing and usability testing lies in the goal of the testing as well as in the testing process.

While usability testing is task-based, learnability testing is not. During learnability testing, we observe the cognitive flow and accuracy of the course, how well information is structured such that the learner understands and comprehends the learning material. At the same time, we also observe learners stumble, click buttons, tabs, links, skip pages or instructions, interpret instructions correctly or incorrectly as they undergo the process of trying to learn.

Therefore, in learnability testing, the focus is both on usability issues such as affordance, ease of navigation, findability etc as well as on the cognitive aspect of the course such the flow of information, language, meaning, level of difficulty or challenge as the learner tries to understand information on each screen.

I believe this is the key difference between the two. It is a huge challenge to test the learnability of a learning material as it is difficult to test the cognitive impact of the course in the long run. Asking learners to recall or restate their understanding during the test is only a short-term measure and does not assure long term retention or application.

Our research team always summarizes this by saying - "in UTs we can clearly see what is going wrong and where but in learner testing its very difficult to see whether the learner understands the concepts, and if they do, whether they will be able to retain the concepts and apply them at their workplace later."

You can have a quick look at the typical learnability testing process ( http://elearning.kern-comm.com/?p=201 ) that we follow. Am sure most us are familiar with the usability testing process!

And maybe a pertinent question is how do you design for learnability?

I just came across a couple of paragraphs that reminded me of this post. The author mentioned a rock climbing instructor having his students attempt to climb one of those indoor rocks & lectured after on how to properly shift your weight etc...

The book says he created a sort of R-mode to L-mode (right side, left sidebrain) flow.

I hope this helps and you are still accepting thoughts on this. I've been apart of a few teaching tool projects and the thought never even crossed my mind. HA. I think hands on then research works best for me. The only complaint with that method would be not knowing what something is and trying to find an answer if you jump right into it without having any prior knowledge.

Test the waters with something that sinks fast before you dive in...