How many alternatives, concepts, or sketches are enough?

http://www.90percentofeverything.com/2009 /01 /14 /why -you -shouldnt -rush -into -a -solution -too -quickly /

Harry posted an interesting post on "90 percent of everything" about not rushing to design solutions too quickly. Designers should cover the design space with divergent approaches first and identify proper alternatives before converging on an idea. I think I've heard others say as well the same about iterative design, and the ability of successful designs only to evolve if the pool of ideas is rich and diverse. The idea is not exactly revolutionary but stirs a basic design question.

The question which I am wondering about then is how do we know how many alternatives are enough? How do we know we have enough sketches, alternatives or concepts before we begin choosing a satisfising solution.

Harry also pointed me to wiki entry on wicked problems http://en.wikipedia.org/wiki/Wicked_problem, where it says that it's not possible to measure the design space. "Wicked problems do not have an enumerable (or an exhaustively describable) set of potential solutions, nor is there a well-described set of permissible operations that may be incorporated into the plan." So what are we left with here?

I take the pop-corn in the microwave approach to this. take it out when the pops start to happen infrequently. But as Jonas says usually other factors create limitations before this.

BTW, sketching/exploration, is not to create "alternatives" and "iterations" but is a ideation generation process. Even though you do 100 sketches, only 10 concrete ideas may come out of the process.

-- dave

Posted from the new ixda.org
http://www.ixda.org/discuss?post=37356

I recommend that you test your alternate designs by having users use prototypes of them.

Teams I've worked with have processes for developing radically different alternatives or approaches to design problems. Then they have people who are users, or like the real users, try out one of the alternatives.

Here's how it works: Say your team decides on two design ideas (testing more than that at at time is actually really hard) that are really different. Now, find a bunch of users maybe 10 or 12 (but at least 8). Separate them into two groups. Create task scenarios that you want users to follow in trying out the prototypes. Everyone does the same tasks. Now, individually, have each person in one group do the tasks on prototype A. Then have each person in the other group do the tasks on prototype B. The whole team observes all the sessions, watching for where users have more or fewer problems.

My experience is that parts of each design work well and parts of each design work poorly, but out of this data you get from users, you can create a hybrid design that should work pretty well. (And which, by the way, you should have a few more users try out with the same tasks.)

The beauty of this approach is that it frees the designers to use their creativity and design knowledge, and then you can measure how effective those creative ideas are by having users use them. It's no longer a subjective decision among the team members.

Make sense?

Dana

:

Dana Chisnell
desk: 415.392.0776
mobile: 415.519.1148

dana AT usabilityworks DOT net

www.usabilityworks.net
http://usabilitytestinghowto.blogspot.com/

On Jan 19, 2009, at 3:13 AM, Jakub Linowski wrote:

http://www.90percentofeverything.com/2009 /01 /14 /why -you -shouldnt -rush -into -a -solution -too -quickly / Harry posted an interesting post on "90 percent of everything" about not rushing to design solutions too quickly. Designers should cover the design space with divergent approaches first and identify proper alternatives before converging on an idea. I think I've heard others say as well the same about iterative design, and the ability of successful designs only to evolve if the pool of ideas is rich and diverse. The idea is not exactly revolutionary but stirs a basic design question. The [trim]

More than the issue of "how many ideas", I always end up without adequate prioritization mechanisms/tools by which to decide alternatives to choose for inclusion in the iteration process.

On Mon, Jan 19, 2009 at 7:31 AM, Dave Malouf dave.ixd at gmail.com wrote:

I take the pop-corn in the microwave approach to this. take it out when the pops start to happen infrequently. But as Jonas says usually other factors create limitations before this. BTW, sketching/exploration, is not to create "alternatives" and "iterations" but is a ideation generation process. Even though you do 100 sketches, only 10 concrete ideas may come out of the process. — dave Posted from the new ixda.org http://www.ixda.org/discuss?post=37356 Welcome to the Interaction Design Association (IxDA)! To post to this list . discuss at ixda.org Unsubscribe .... http://www.ixda.org/unsubscribe [trim]

I would be curious to hear what tools colleagues do use for prioritization of ideas. The key issue here is what the criteria are for choosing ideas. In the early stages of ideation, the criteria might be different for choosing what to consider further (the 10 ideas out of 300) versus what to consider when you move into detailed design.

Some general methods for prioritization are:

1. The monetary method where a sample of people are given a fixed amount of "money", a list of ideas or requirements along with their relative costs and then asked to "buy" the things of most value.

2. The criterion matrix where you list the criteria (weighted or unweighted) and then calculate a score with the top scores meeting more of the criteria.

3. Q-sorting where you ask people to sort on an important criteria on a scale ranging from low to high.

4. Private voting for the best ideas

5. Public voting for the best ideas (red dots on the best ideas)

6. Consensus

7. Decision by a leader

8. Decision by another group

9. The target method (good for a first cut between good and not-good idea)

In braindrawing exercises, the design team would look at lots of sketches and mark ideas that seem worth pursuing which would be consensus or voting and would then have a product team do a second level of prioritization on specific criteria.

What other techniques do people use? This is something that doesn't seem to get discussed much.

Chauncey

On Mon, Jan 19, 2009 at 12:41 PM, christine chastain chastain.christine at gmail.com wrote:

More than the issue of "how many ideas", I always end up without adequate prioritization mechanisms/tools by which to decide alternatives to choose for inclusion in the iteration process. On Mon, Jan 19, 2009 at 7:31 AM, Dave Malouf dave.ixd at gmail.com wrote: I take the pop-corn in the microwave approach to this. take it out when the pops start to happen infrequently. But as Jonas says usually other factors create limitations before this. BTW, sketching/exploration, is not to create "alternatives" and "iterations" but is a ideation generation process. Even though you do 100 sketches, only [trim]

More than the issue of "how many ideas", I always end up without adequate prioritization mechanisms/tools by which to decide alternatives to choose for inclusion in the iteration process.

This is exactly what is hard about design.

Ideation can be learnt and performed somewhat mechanically (as many d.i.y. creativity guides and methods illustrate). It is generally not hard to teach people to create hundreds of idea seeds quickly for any given brief.

But it is obviously impossible to prototype and user-test all of the seeds to decide which one is the "best." Judgment is needed to decide which ones to pursue and elaborate further.

Experienced designers perform such judgment tasks better than novices. Part of that experience is familiarity with the design genre and the intended use situation — the judgment entails envisioning the qualities of using the product that could grow from the seed.

And as Dave points out, generative and evaluative processes are not mutually exclusive. Judging idea seeds involves adding, taking away, modifying and combining to arrive at more articulated ideas.

The conventional route to building the experience needed for early- phase judgment is, of course, to participate in design processes with more experienced peers (in professional settings) or tutors (in design school settings) — to learn criteria and values, and to study how ideas transition into actual use in the course of a design process.

Studying and practicing criticism is one additional way of strengthening judgment skills; another is to study, think about and speak systematically about the use of products/services in different genres and specifically what the qualities are that distnguish good use in those genres.

Then there is the whole issue of how the use-oriented judgments peculiar to interaction design tradeoff against other judgment criteria, such as marketability and technical feasibility, but that would take the discussion into the even fuzzier area of the organizational politics of interaction design...

Jonas Löwgren

Yes, these are methods I've used in the past and in addition K/J sorting as part of the Six Sigma process...and more recently, prioritization at various "phase gates" has run parallel to the development of business models (in the case of platforms) and business cases (in the case of concepts. But I have not been able to successfully use a holistic, predictive risk assessment tool and would love to hear of any examples.

On Mon, Jan 19, 2009 at 12:25 PM, Chauncey Wilson chauncey.wilson at gmail.com wrote:

I would be curious to hear what tools colleagues do use for prioritization of ideas. The key issue here is what the criteria are for choosing ideas. In the early stages of ideation, the criteria might be different for choosing what to consider further (the 10 ideas out of 300) versus what to consider when you move into detailed design. Some general methods for prioritization are: 1. The monetary method where a sample of people are given a fixed amount of "money", a list of ideas or requirements along with their relative costs and then asked to [trim]

All of these methods you listed strike me as limiting in they emphasize equal voting - often I don't believe everyone deserves an equal vote. Heretical perhaps, but I'd much rather let a small number of people who will be held accountable for the final design entirely drive these explorations. It's their necks on the line. They should at least win or lose on their own intuitions.

Having people vote on one sentence, or one sketch, descriptions of ideas is always a crap-shoot: people are heavily biased to the ideas they're familiar with, and they can't be equally familiar with all the ideas.

With a pile of 50 ideas and only time to explore 5, I'd sit down with the three or four people most accountable for the final result and talk it out. I would depend on intuition, debate and persuasion more than any sort of numerical/polling/ranking system.

If I did anything "methody", which I'd try to avoid, I do one of two things:

1) Have a list of criteria, or project goals, or desirable attributes up on the whiteboard during that discussion to help us frame our opinions.

2) Make the goal to pick one high risk idea, three medium risk ideas, and one low risk idea. This frames the problem of picking alternatives as a risk portfolio, where our goal is to distribute the creative risks in some way. This makes it ok to advocate a crazy idea, since that's desirable to fit the high risk slot.

But most importantly, if I didn't have the power to grant this much authority to those 3 people, my real problem is political, not the quest for the perfect number of alternatives.

-Scott

Scott Berkun
www.scottberkun.com

Original Message
From: discuss-bounces at lists.interactiondesigners.com [mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of Chauncey Wilson
Sent: Monday, January 19, 2009 10:26 AM
To: christine chastain
Cc: Dave Malouf; discuss at ixda.org
Subject: Re: [IxDA Discuss] How many alternatives, concepts,or sketches are enough?

I would be curious to hear what tools colleagues do use for prioritization of ideas. The key issue here is what the criteria are for choosing ideas. In the early stages of ideation, the criteria might be different for choosing what to consider further (the 10 ideas out of 300) versus what to consider when you move into detailed design.

Some general methods for prioritization are:

1. The monetary method where a sample of people are given a fixed amount of "money", a list of ideas or requirements along with their relative costs and then asked to "buy" the things of most value.

2. The criterion matrix where you list the criteria (weighted or unweighted) and then calculate a score with the top scores meeting more of the criteria.

3. Q-sorting where you ask people to sort on an important criteria on a scale ranging from low to high.

4. Private voting for the best ideas

5. Public voting for the best ideas (red dots on the best ideas) 6. Consensus 7. Decision by a leader 8. Decision by another group 9. The target method (good for a first cut between good and not-good idea)

You make a good point though I didn't specifically mention equal voting at all. You could have a small group who, as you say, have their necks on the line or you could have private voting of the 10 top designers in the country using polling software or you could generate criteria and have your small group use the criteria as a starting point for a deeper discussion of the type you suggest. You mention listing the criteria on the board which is a great starting point, because many groups fail to explicitly identify criteria that they are using (that method sounds like the QOC method - Questions-Options-Criteria - that is described in the "design rationale" literature.)

Some time ago, I worked with a group of people who necks were on the line and the use of a group Q-sort on the dimension of 'project risk" for particular requirements worked much as you described with the different items getting much discussion among respected team members and then getting placed into low, medium, and high risks. The discussion for each item often elaborated on what was risky for the different representatives.

Chauncey
On Mon, Jan 19, 2009 at 1:59 PM, Scott Berkun info at scottberkun.com wrote:

All of these methods you listed strike me as limiting in they emphasize equal voting - often I don't believe everyone deserves an equal vote. Heretical perhaps, but I'd much rather let a small number of people who will be held accountable for the final design entirely drive these explorations. It's their necks on the line. They should at least win or lose on their own intuitions. Having people vote on one sentence, or one sketch, descriptions of ideas is always a crap-shoot: people are heavily biased to the ideas they're familiar with, and they can't be [trim]

True - you didn't - Sorry for criticizing you for something you did not say : )

My bias is against teams pretending to quantify the unquantifiable. I like opinions. I like things that designers believe but can not prove mathematically, but can explain through argument. Any decision making process that doesn't make use of conviction, persuasion and passion is one I doubt will work out well.

Decision models/methods are great provided they're fodder - that they're used to help the discussion and debate, but not to replace it. Too often managers becomes slaves to methods, and they follow them to the letter because of the temptation to dodge their responsibility to think and be accountable: they can blame the method. Or in the case of pure democratic method, blame the team (You voted for it!). Methods can can easily encourage the tolerance for design-by-committee type decisions.

So in the case of "how many alternatives", I'm a huge advocate of delegating the design decisions to the point where a small group of people (possibly one), can easily figure this out for themselves - based on the resources they have, divided by the short ordered list of which design decisions are most important. If no such list exists, they should be motivated to make one.

If power is distributed well, you're less likely to need a "method".

-Scott

Scott Berkun
www.scottberkun.com

From: Chauncey Wilson [mailto:chauncey.wilson at gmail.com] Sent: Monday, January 19, 2009 11:44 AM
To: Scott Berkun
Cc: discuss at ixda.org
Subject: Re: [IxDA Discuss] How many alternatives, concepts, or sketches are enough?

You make a good point though I didn't specifically mention equal voting at all. You could have a small group who, as you say, have their necks on the line or you could have private voting of the 10 top designers in the country using polling software or you could generate criteria and have your small group use the criteria as a starting point for a deeper discussion of the type you suggest. You mention listing the criteria on the board which is a great starting point, because many groups fail to explicitly identify criteria that they are using (that method sounds like the QOC method - Questions-Options-Criteria - that is described in the "design rationale" literature.) \

Here's the thing, though - this is a great start but I still don't see it linked to risk assessment and ultimately the bottom line...I know, what every designer/design researcher/innovator hates to hear...

But, once again, I'm in the position of having to show, to the board of directors of a large non-profit foundation, how our budget will be used to support numerous platforms, under which reside numerous projects/concepts. Essentially, they would love to hear that one or another idea (in this case, the prioritization has already been made, based on collective criteria) will be a return on investment and I have no way, beyond presenting a business case study and linking concepts to future portfolio efforts, to provide that information. What I really need is a risk assessment/predictive model that looks at a variety of future scenarios and takes into account current and future business state/future general population need, etc. Has anyone heard of anything like that?

On Mon, Jan 19, 2009 at 1:44 PM, Chauncey Wilson
chauncey.wilson at gmail.comwrote:

You make a good point though I didn't specifically mention equal voting at all. You could have a small group who, as you say, have their necks on the line or you could have private voting of the 10 top designers in the country using polling software or you could generate criteria and have your small group use the criteria as a starting point for a deeper discussion of the type you suggest. You mention listing the criteria on the board which is a great starting point, because many groups fail to explicitly identify criteria that they are [trim]

Persuasion and passion are important and I'm seeing more references to principles of persuasion in design discussions (for example the work of Cialdini). In fact, a solid grounding in persuasion principles should be part of our professional training. I wrote an essay on the use of persuasive techniques for usability practitioners in 2007 issue of the ACM interactions. Methods (or more accurately, the output of methods which can be qualitative or quantitative) should, as you say, provide data and interpretations for discussion and debate and different methods can even provide different sides of the problem (method triangulation).

I profoundly dislike the autocratic application of "standard" methods and like to consider how different methods can examine different angles to a problem. For example, you wrote an excellent essay on how to run brainstorming sessions. Group brainstorming is a complex social environment and hard to do well. There is another technique called brainwriting (which you might have written about) which can be used to gather input when groups are shy or there are political concerns or you have a mix of old and new people. The brainwriting method can complement group brainstorming and often provides an outlet for those who may be anxious in a group setting.

So, methods and their output should be used to expose a range of issues, contribute to debate and discussion, and support the triangulation of data that will reduce risk to stakeholders and eventually customers.

I very much enjoyed your brainstorming write-up by the way and reference it in a chapter that I've written on brainstorming, brainwriting, and braindrawing.

thanks,
Chauncey

On Mon, Jan 19, 2009 at 3:01 PM, Scott Berkun info at scottberkun.com wrote:

True - you didn't - Sorry for criticizing you for something you did not say : ) My bias is against teams pretending to quantify the unquantifiable. I like opinions. I like things that designers believe but can not prove mathematically, but can explain through argument. Any decision making process that doesn't make use of conviction, persuasion and passion is one I doubt will work out well. Decision models/methods are great provided they're fodder - that they're used to help the discussion and debate, but not to replace it. Too often managers becomes slaves to methods, and they [trim]

In answer to the original question — "How many ideas qare enough?" -- I'd say it can be reassuring to have a number, as a guideline.

Here's a number, based on my experience: if you ask three to four people to separately prepare come up with substantially different ideas, then you'll typically have a saturated design space for the design of [something] at most levels of granularity.

The previous paragraph needs to be fleshed out:

Posted from the new ixda.org
http://www.ixda.org/discuss?post=37356

One less than the one that starts to hold up the development process.

MT

As the person that created the slides Harry referred to, the key point I was trying to make is that reaction to initial design sketches often sets the design direction and constrains the design space within which we explore potential solutions.

My intention with the slides was to warn practitioners of this, and encourage that they actively try to keep the design space unconstrained for as long as possible. It is difficult to do this, especially if you get positive feedback from the first sketches you share with colleagues/stakeholders/users.

As soon as we stop exploring different concepts and start iterating, we are optimising an idea rather than looking for different (innovative?) solutions.

Posted from the new ixda.org
http://www.ixda.org/discuss?post=37356

Hi Ian - thanks for speaking up. I agree with the point you made in your talk, regardless of how much I may have butchered it in this thread : )

Although I worry more about this: who are these sketches being shown to? And why are the viewers of these sketches deciding the design direction?

I'm not asking for a fantasy land where designers have mind control powers over clients and executives, but I am saying that when a sketch is shown, the person showing the sketch should take responsibility for the conversation it creates. What you describe is a common failing. But the only person to recognize / prevent / solve the failing is the person who made the sketch. They have more control over how that sketch is seen and interpreted than anyone else.

The easy trick advertisers have known forever is never show one sketch to a client. Always show three or five. Cynically they do this to make their own favorite look good, but done more constructively it establishes, right away, that there are multiple directions worthy of exploration.

-Scott

Scott Berkun
www.scottberkun.com

Original Message
From: discuss-bounces at lists.interactiondesigners.com [mailto:discuss-bounces at lists.interactiondesigners.com] On Behalf Of iain barker
Sent: Monday, January 19, 2009 3:34 PM
To: discuss at ixda.org
Subject: Re: [IxDA Discuss] How many alternatives, concepts,or sketches are enough?

As the person that created the slides Harry referred to, the key point I was trying to make is that reaction to initial design sketches often sets the design direction and constrains the design space within which we explore potential solutions.

On Mon, Jan 19, 2009 at 3:13 AM, Jakub Linowski jlinowski at gmail.com wrote:

http://www.90percentofeverything.com/2009 /01 /14 /why -you -shouldnt -rush -into -a -solution -too -quickly / The question which I am wondering about then is how do we know how many alternatives are enough? How do we know we have enough sketches, alternatives or concepts before we begin choosing a satisfising solution.

My manager is asking this too, as I'm now working on my eleventh set of concept sketches for a new software product. Probably several more ahead of me still. Sigh.

But there's a telltale indicator that lets me know when I'm getting close. It's when the design stays put for at least 24 hours with no one on the design team, including me, wanting to change anything. I call it the 24 Hour Rule. Maybe someone taught this to me - if so, thank you, whoever you were - it's been helpful. Maybe it's original; I've been using it so long I can't remember. The 24 Hour Rule means if you're still twitchy to change something, you're not done with the design yet, even if engineers are already building the product.

And I seem to recall a graphic design course where our instructor had us bring in a hundred variations of whatever it is we were drawing. But maybe we were bad and she wanted to punish us...

Michael Micheletti