Discarding Prototypes

4 Oct 2004 - 11:21am
9 years ago
34 replies
674 reads
kjnarey
2004

Hi,

Realise this is a fairly subjective topic but was wondering what the group's
take on discarding prototypes is.

At my company we tend to prototype web-applications using
HTML/JavaScript/Hard-Coded data (hi-fi I guess?) during the design phase of
a project. This mark-up is generally written using UI and front-end coding
standards and is therefore of around 75%-90% application grade quality.

I've read/heard much about discarding prototypes, but never been able to see
a good reason for throwing away this reasonably good quality code.

Can anyone assist in this regard?

Kind regards

Kevin

Comments

3 Oct 2004 - 12:00pm
Petteri Hiisilä
2004

> At my company we tend to prototype web-applications using
> HTML/JavaScript/Hard-Coded data (hi-fi I guess?) during the design phase of
> a project. This mark-up is generally written using UI and front-end coding
> standards and is therefore of around 75%-90% application grade quality.
>
> I've read/heard much about discarding prototypes, but never been able to see
> a good reason for throwing away this reasonably good quality code.

Hi Kevin,

There are many kinds of prototypes. The HTML code in the prototype you
described could and should be used in the final product, if the markup
is clean. I mean, can you do the HTML part of it any differently anyway?

The prototypes that should be discarded are technical proof-of-concepts.
Engineers use them to try if some unusual or new interaction is
technically feasible.

Those prototypes work in a whole different application layer. There's
application logic and "real code" involved. You have to open the hood to
see them :)

An example:

I've designed a spam filter that can be taught to filter or let through
certain kind of mail. Just like in Mozilla Thunderbird, except that this
is a webmail.

Mark (our persona) can mark some mail as junk. He can mark some mail as
not junk. He can "teach" the filter to adapt to mail he receives. The
webmail service as a whole utilizes the spam/ham databases created by
all users. "Collaborative teaching."

The idea is simple, but the technical implementation isn't obvious.
That's why our head engineer had to experiment. He put some old legos
together in a new way with glue and gum. Collaborative teaching now
works, but only for our team. The server runs below his desk.

There's no GUI at all. The algorithms are ugly and inelegant. The code
quality is horrible. But it works. The whole prototype was built as
quickly as possible to see, if it can be built at all. To prove the
concept. To test the idea.

If the technical concept & code aren't elegant and clean when we release
the system, it would be a disaster. Some 180 000 daily users are going
to use it every day.

He now has to discard the code and redesign the technical architecture
again, but this time he makes it production size and quality.

Best,
Petteri

--
Petteri Hiisilä
Palveluarkkitehti / Interaction Designer /
Alma Media Interactive Oy / NWS /
+358505050123 / petteri.hiisila at almamedia.fi

"The unbroken spirit
Obscured and disquiet
Finds clearness this trial demands"
- Dream Theater

3 Oct 2004 - 5:08pm
Listera
2004

kjnarey:

> I've read/heard much about discarding prototypes, but never been able to see
> a good reason for throwing away this reasonably good quality code.

Some of my milder thoughts on prototypes, in no particular order:

Prototypes should *always* be discarded.

Prototypes should not be for the sole benefit of the developers.

No reasonably complex interactive app should ever go to production without a
prototype.

Prototypes should be decoupled from development altogether.

Prototypes should never be designed to be used for production.

Prototypes should be a design artifact not a development tool.

Prototypes should be "owned" by the design team.

Prototypes should be the principal end-product of the design process,
preferably the only "deliverable."

Prototypes should never be a one-time deliverable, they should be started as
early in the project as possible and be thoroughly iterative.

Prototypes should preferably be written in a language/platform that the
developers cannot directly use for production.

Prototypes should be constructed to demonstrate interaction/functionality,
not compatibility, efficiency, scalability, speed, etc.

Prototypes should never be created by developers, unless strictly lead by
designers and when no other option is available.

Prototypes should never be considered a prelude to production code.

In my prototype-centric world, the app creation process is simply divided
into two stages: design and implementation. Prototype is what bridges the
two. I recognize that prototyping tools are not what they should be and not
all design teams are capable of designing their own prototypes, but this is
a temporary historical aberration which should disappear in time. I fully
expect designers who can't prototype to become a small minority in the
not-too-distant future. Developers may also do their own prototypes
(hopefully, for technical reasons) but that's their own internal
work-product and should not concern us.

----
Ziya

Generalizations are always inaccurate.

3 Oct 2004 - 6:16pm
Dave Malouf
2005

I think that Ziya's collolaries regarding prototypes are pretty right on.
I'd like to add a bit of context to them though. (don't mean to be putting
words in your mouth, Ziya).

Prototypes are an interaction designer's sketches and concept drawings that
are used in most of the other design areas. They need to be outside the
development cycle so that a designer can do as many as possible, throw them
away and start all over again.

Process ...

Designer does 4 completely different interaction prototypes
These prototypes are used for internal and external validation and
evaluation. Best ideas of all of them are taken and then you do two more for
the next level of validation and evaluation ... Take the best ideas of that
one and then make a final prototype, validate and evaluate again ... Edit,
and specify ... The specification is the deliverable to the development
team. Up until that point the dev team is a consultant that the interaction
designer relies upon for reality checks, as well as to make transparent the
design process through the x-functional team.

Also, at each level of iteration the fidelity gets higher and higher
(zoooming in deeper along the way).

This is akin to an industrial designer doing 50 sketches of a can opener in
his/her first round of creative thinking ... If he/she is lucky maybe 1 of
these ideas, or a new conglomerate of these ideas is taken to modeling (next
level of fidelity) and then presented to the client. Usually though you want
to give 3 options to a client. Don't ask me why 3; it just seems to be a
magic #. So the song goes!

If all this was tied to a deliverable for dev, then the freedom necessary in
this process goes out the door and then you really aren't prototyping
anymore, but are just doing development in a medium that is perceived to be
"erasable" or "flimsy" and thus changes are easier to make along the way.
Which is true, but that is not the real purpose of prototyping.

I say, to help yourselves out. Choose a platform that IS NOT the same as the
final product but is interactive. I.e. if you are making HTML products, use
Flash for the prototyping environment. You can make it look and act almost
exactly the same (actually a little better), and it can't be used, so you
won't be thinking about dev the whole time.

-- dave

-- dave

3 Oct 2004 - 9:50pm
Pradyot Rai
2004

kjnarey <asow73 at dsl.pipex.com> wrote:

> At my company we tend to prototype web-applications using
> HTML/JavaScript/Hard-Coded data (hi-fi I guess?) during the design phase of
> a project. This mark-up is generally written using UI and front-end coding
> standards and is therefore of around 75%-90% application grade quality.

Agreeing with most of the remarks about the definitions of
"prototype", I think what Kevin is doing is more than just
prototyping, or bridging the gap between design and development. He is
going to the other side and suggesting the developers to use this UI
framework (HTML prototype in this case) and prevent reinventing the
wheel. I have seen designers working on both side of the fence, so the
rule, "prototype must be throw-away" should not be considered de facto
standard. I strongly believe that it is interaction designers job too
to get involved with UI frameworks.

Prady

3 Oct 2004 - 10:50pm
Listera
2004

Prady Rai:

> He is going to the other side and suggesting the developers to use this UI
> framework (HTML prototype in this case) and prevent reinventing the wheel.

This is a subtle but exceedingly important distinction: "reinventing." In
order to re-invent you need to be in the inventing business. Or in plain
English, this would suggest that developers need to be driving IA/UI/UX
matters, as opposed to *implementing* what's conceived by the design team
and embodied by the prototype. My advocacy is to get developers out of the
IA/UI/UX invention/re-invention business all together.

This doesn't obviously mean that the design and developer teams never
consult each other, it's just that the prototype which drives IA/UI/UX is in
turn firmly driven by the design team.

> I strongly believe that it is interaction designers job too to get involved
> with UI frameworks.

I wouldn't disagree with that, but I don't think framework design (if we
agree on terminology) is the same as UI design destined to be directly
consumed by end-users.

To make a gross analogy, frameworks are often provided by OS or platform
vendors but are not directly consumed by users. Individual application
providers may use/customize those frameworks to craft their own app specific
UIs which are then directly consumed by end-users. In other words,
Macromedia hopefully employs interaction designers while creating the
bundled Flash MX components, but that says nothing about how I, as a
designer of a specific Flash or Flex app, will in fact use/customize those
very components. I'm at least a level of abstraction closer to the user than
framework/component designers.

So I consider the the task of creating (reusable) frameworks/components (for
general market or in-house use) to be a separate endeavor than designing
end-user applications.

I also realize that the notion of relegating developers to developing (as in
coding/engineering/implementing) is heretical, but, in the end, it's both
necessary and beneficial.

----
Ziya

When 2+2=4, it's development,
When 2+2>4, it's design.

4 Oct 2004 - 4:02am
Sarah Brodwall
2004

On Sun, 03 Oct 2004 18:08:59 -0400, Listera <listera at rcn.com> wrote:
> Prototypes should be the principal end-product of the design process,
> preferably the only "deliverable."

Could you elaborate on this statement? Is this sort of like the idea
that a sufficiently detailed requirements specification is
indistinguishable from the code itself? Also, would you say that
today's reality for most designers deviates significantly from your
ideal?

~Sarah

4 Oct 2004 - 4:24am
Martyn Jones BSc
2004

Ziya wrote:
> Prototypes should be decoupled from development altogether.

Just a few questions:

In the discussion of Prototypes here, are we only referring to projects that
a design team has little experience of, or are we talking about all
projects?

Also, what does a Project Lifecycle look like, where Prototyping is the main
concern of the design team (and where Prototypes should be decoupled from
development altogether.)? Is it a hybrid between the Classic Waterfall
Lifecycle and the Spiral Model?
E.g.
1. System Engineering
2. Analysis
3. Design
3a Planning
3b Risk Analysis
3c Engineering
3d User Evaluation
Loop these steps, as you move towards a completed system...
4. Code
5. Testing
6. Maintenance

Martyn?

----------------------
Martyn Jones BSc
Interaction Designer
Kode Digital Ltd.
----------------------

4 Oct 2004 - 7:23am
Gerard Torenvliet
2004

In "The Mythical Man Month", Fred Brooks coined the appropriate phrase:

"Plan to throw one away, you will anyway."

Every prototype I've seen has been thrown away (even if the team as a whole didn't know that's what was happening). The problem is that the purge sometimes doesn't happen until release 2.0 or 3.0 of a product.

-Gerard

:: Gerard Torenvliet / gerard.torenvliet at cmcelectronics.ca
:: Human Factors Engineering Design Specialist
:: CMC Electronics Inc.
::
:: Ph - 613 592 7400 x 2613
:: Fx - 613 592 7432
::
:: 415 Legget Drive, P.O. Box 13330
:: Ottawa, Ontario, CANADA, K2K 2B2
:: http://www.cmcelectronics.ca

-----Original Message-----
From: David Heller [mailto:dave at ixdg.org]
Sent: October 3, 2004 7:16 PM
To: 'Interaction Designers'
Subject: RE: [ID Discuss] Discarding Prototypes

[Please voluntarily trim replies to include only relevant quoted material.]

I think that Ziya's collolaries regarding prototypes are pretty right on.
I'd like to add a bit of context to them though. (don't mean to be putting
words in your mouth, Ziya).

Prototypes are an interaction designer's sketches and concept drawings that
are used in most of the other design areas. They need to be outside the
development cycle so that a designer can do as many as possible, throw them
away and start all over again.

Process ...

Designer does 4 completely different interaction prototypes
These prototypes are used for internal and external validation and
evaluation. Best ideas of all of them are taken and then you do two more for
the next level of validation and evaluation ... Take the best ideas of that
one and then make a final prototype, validate and evaluate again ... Edit,
and specify ... The specification is the deliverable to the development
team. Up until that point the dev team is a consultant that the interaction
designer relies upon for reality checks, as well as to make transparent the
design process through the x-functional team.

Also, at each level of iteration the fidelity gets higher and higher
(zoooming in deeper along the way).

This is akin to an industrial designer doing 50 sketches of a can opener in
his/her first round of creative thinking ... If he/she is lucky maybe 1 of
these ideas, or a new conglomerate of these ideas is taken to modeling (next
level of fidelity) and then presented to the client. Usually though you want
to give 3 options to a client. Don't ask me why 3; it just seems to be a
magic #. So the song goes!

If all this was tied to a deliverable for dev, then the freedom necessary in
this process goes out the door and then you really aren't prototyping
anymore, but are just doing development in a medium that is perceived to be
"erasable" or "flimsy" and thus changes are easier to make along the way.
Which is true, but that is not the real purpose of prototyping.

I say, to help yourselves out. Choose a platform that IS NOT the same as the
final product but is interactive. I.e. if you are making HTML products, use
Flash for the prototyping environment. You can make it look and act almost
exactly the same (actually a little better), and it can't be used, so you
won't be thinking about dev the whole time.

-- dave

-- dave

_______________________________________________
Interaction Design Discussion List
discuss at ixdg.org
--
to change your options (unsubscribe or set digest): http://discuss.ixdg.org/
--
Questions: lists at ixdg.org
--
Announcement Online List (discussion list members get announcements already)
http://subscribe-announce.ixdg.org/
--
http://ixdg.org/

4 Oct 2004 - 7:37am
Dave Malouf
2005

Why Designers shouldn't dev?

Simple, if you are building a real piece of software, there are subtleties
in code that a designer shouldn't be thinking about--i.e. performance
scaling, objectification of the code not related to the GUI, and just plain
better code.

Yes, designers should know the technology they are designing for in some
way, but that is different than taking the time to design code.

If you are creating 75% useful code two things wrong is going on.
1. You are wasting time on code and not iterating enough
2. Your dev team is not doing a satisfactory job at improving the code you
give them, especially in the area of objectification and performance. Yes,
HTML has performance issues, especially if it has JavaScript & CSS
incorporated.

-- dave

4 Oct 2004 - 8:38am
Narey, Kevin
2004

Interesting thoughts from all - thank you.

>Simple, if you are building a real piece of software,

Dave, - What is a 'real piece of software'?

>There are subtleties in code that a designer shouldn't be thinking
about--i.e. performance scaling...

I question this philosophy...............When the designer of the
aerodynamic carbon fibre mountain bike that won Lance Armstrong his 5th Tour
De France - designed the finished form of this bike was he/she not thinking
about carbon fibre in his/her dreams? Did he/she not know (because of
his/her innate interest) of all the performance criteria and compositional
data of the alloys that made up the crank shaft? Did he/she not look deeply
into the rigours of aerodynamics to find what form would suit the goals of
the persona riding the bike? Perhaps.

>and just plain better code.

Or perhaps a 'better' bike?

>Yes, designers should know the technology they are designing for in some
way, but that is different than taking the time to design code.

IMO code is never 'designed' (with the exception of the designer of that
language perhaps!). Code is a building block of which a 'form' is designed
(an object). If a designer has a thorough knowledge of the building
blocks/materials he/she is using, the knowledge of what the possibilities
(boundaries?) of that material are, starts to diminish the possibilities for
error/wasted time etc. A great deal of my knowledge of what tools are
available to me as an interaction designer, come from my knowledge of what
code can do when it performs well.

I currently see real value (saving valuable time on the project and
contributing towards re-usable components) in prototyping web front ends
with standards compliant code and still haven't seen or read anything of an
alternative concrete business case for destroying it yet.

I am ripe for turning....

Rgds

Kevin

_______________________________________________
Interaction Design Discussion List
discuss at ixdg.org
--
to change your options (unsubscribe or set digest): http://discuss.ixdg.org/
--
Questions: lists at ixdg.org
--
Announcement Online List (discussion list members get announcements already)
http://subscribe-announce.ixdg.org/
--
http://ixdg.org/

**********************************************************************
gedas united kingdom limited
Registered in England no. 1371338

This email and any files transmitted with it are confidential
and it may be privileged.

It is intended solely for the use of the individual or entity to
whom it is addressed.

If you have received this in error, please contact the sender
and delete the material immediately.
**********************************************************************

4 Oct 2004 - 8:47am
Gerard Torenvliet
2004

Kevin:

I think that something else is at work in the example of the Trek bikes designer. That designer (probably) designed the prototype bike, just thinking about how that bike should work and feel. The properties of the materials in use are a big part of this, and so were probably well-considered by the designer.

I've got to believe that the designer wasn't thinking about production economies when making that bike. He/she wasn't looking into a design that was the easiest for Trek to manufacture, store, or ship. That was someone else's concern.

Interaction designers are like the bike designer - they come up with the idea, the machine. Software developers are like those who rework the bike for manufacturing, and then do the production tests.

There is interplay. Buy from Ikea, and the design was reworked by engineering and passed back to design so that manufacturability, cost, and size requirements could be met. I'd rather buy furniture that's a one-off (the last page of Fine Woodworking always has a great example), but we pay lots for that. Lance Armstrong can afford to.

Does that make any sense?

-Gerard

:: Gerard Torenvliet / gerard.torenvliet at cmcelectronics.ca
:: Human Factors Engineering Design Specialist
:: CMC Electronics Inc.
::
:: Ph - 613 592 7400 x 2613
:: Fx - 613 592 7432
::
:: 415 Legget Drive, P.O. Box 13330
:: Ottawa, Ontario, CANADA, K2K 2B2
:: http://www.cmcelectronics.ca

-----Original Message-----
From: Narey, Kevin [mailto:Kevin.Narey at Gedas.co.uk]
Sent: October 4, 2004 9:38 AM
To: 'David Heller'
Cc: 'discuss at interactiondesigners.com'
Subject: RE: [ID Discuss] Discarding Prototypes

[Please voluntarily trim replies to include only relevant quoted material.]

Interesting thoughts from all - thank you.

>Simple, if you are building a real piece of software,

Dave, - What is a 'real piece of software'?

>There are subtleties in code that a designer shouldn't be thinking
about--i.e. performance scaling...

I question this philosophy...............When the designer of the
aerodynamic carbon fibre mountain bike that won Lance Armstrong his 5th Tour
De France - designed the finished form of this bike was he/she not thinking
about carbon fibre in his/her dreams? Did he/she not know (because of
his/her innate interest) of all the performance criteria and compositional
data of the alloys that made up the crank shaft? Did he/she not look deeply
into the rigours of aerodynamics to find what form would suit the goals of
the persona riding the bike? Perhaps.

>and just plain better code.

Or perhaps a 'better' bike?

>Yes, designers should know the technology they are designing for in some
way, but that is different than taking the time to design code.

IMO code is never 'designed' (with the exception of the designer of that
language perhaps!). Code is a building block of which a 'form' is designed
(an object). If a designer has a thorough knowledge of the building
blocks/materials he/she is using, the knowledge of what the possibilities
(boundaries?) of that material are, starts to diminish the possibilities for
error/wasted time etc. A great deal of my knowledge of what tools are
available to me as an interaction designer, come from my knowledge of what
code can do when it performs well.

I currently see real value (saving valuable time on the project and
contributing towards re-usable components) in prototyping web front ends
with standards compliant code and still haven't seen or read anything of an
alternative concrete business case for destroying it yet.

I am ripe for turning....

Rgds

Kevin

_______________________________________________
Interaction Design Discussion List
discuss at ixdg.org
--
to change your options (unsubscribe or set digest): http://discuss.ixdg.org/
--
Questions: lists at ixdg.org
--
Announcement Online List (discussion list members get announcements already)
http://subscribe-announce.ixdg.org/
--
http://ixdg.org/

**********************************************************************
gedas united kingdom limited
Registered in England no. 1371338

This email and any files transmitted with it are confidential
and it may be privileged.

It is intended solely for the use of the individual or entity to
whom it is addressed.

If you have received this in error, please contact the sender
and delete the material immediately.
**********************************************************************

_______________________________________________
Interaction Design Discussion List
discuss at ixdg.org
--
to change your options (unsubscribe or set digest): http://discuss.ixdg.org/
--
Questions: lists at ixdg.org
--
Announcement Online List (discussion list members get announcements already)
http://subscribe-announce.ixdg.org/
--
http://ixdg.org/

4 Oct 2004 - 9:19am
whitneyq
2010

At 08:37 AM 10/4/2004 -0400, David Heller wrote:
>Why Designers shouldn't dev?
>
>Simple, if you are building a real piece of software, there are subtleties
>in code that a designer shouldn't be thinking about--i.e. performance
>scaling, objectification of the code not related to the GUI, and just plain
>better code.

Whenever the question of whether a design prototype should be used as the
basis for production code comes up, I usually look the most senior
developer in the eye and ask, "Do you really want to base a product on the
code that _I_ produce?"

We can then go on to talk about what that prototype IS good for, and what
would make the most sense as a way to deliver the design. The most common
is that they would like careful attention to visual details so they can
copy them directly.

I once had a developer ask me for the Visio source for a paper prototype.
When I asked why, he told me that he wanted to be able to simply cut and
paste any text (we had worked hard on all of the messages in the program)
into his code. So, we made sure that it was carefully edited and complete
before we gave it to him.

In other words - they want the designer's prototype to pay careful
attention to the design, not to the implementation.

Whitney Quesenbery
Whitney Interactive Design, LLC
w. www.WQusability.com
e. whitneyq at wqusability.com
p. 908-638-5467

UPA - www.usabilityprofessionals.org
STC Usability SIG: www.stcsig.org/usability

4 Oct 2004 - 9:35am
Narey, Kevin
2004

Gerard,

I don't claim to know anything of major significance about the 'TREK'
production cycle so can't really defend my fairly crude analogy. However,
I'm not sure it's that much of a problem being an IxDer and having a concern
about the economies of the project. Perhaps it is seen as a diversion from
the main focus and would dilute the process/effect of design, but this is
subjective and down to the persons career objective and the extra time they
are will to invest aside from their normal (a perception) duties. I see the
malleability of the IxDer role as a positive.

Anyway, I'm still for turning on discarding these prototypes.....

BTW why would I want to re-train in prototyping with flash (or hire) for the
sake of throwing it away? What's the business case?

Rgds

Kevin

**********************************************************************
gedas united kingdom limited
Registered in England no. 1371338

This email and any files transmitted with it are confidential
and it may be privileged.

It is intended solely for the use of the individual or entity to
whom it is addressed.

If you have received this in error, please contact the sender
and delete the material immediately.
**********************************************************************

4 Oct 2004 - 9:23am
Josh Seiden
2003

Your words hold the key. You wrote:

> Code is a building block
> of which a 'form' is designed (an object). If a
designer has
> a thorough knowledge of the building blocks/materials
he/she
> is using, the knowledge of what the possibilities
> (boundaries?) of that material are, starts to
diminish the
> possibilities for error/wasted time etc.

If you examine this dynamic, you will find at its
center the fact that construction materials determine
what is possible to build. This is both indisputable
and inevitable. It DOES NOT however mean that
construction materials determine what can be
*designed.*

In user-centered design, it is incumbent upon the
designer to set aside (if only for some period of time)
that which is possible and focus instead on that which
is desirable. This setting aside becomes more difficult
when you are working in (or close to) the construction
medium, and thus being constrained by that which is
possible.

This is not to say that designers should ignore reality
or ignore the construction medium. Ultimately,
designers must engage fully with the medium. It's just
that this engagement just is not their only
responsibility.

Design "prototypes"--meaning sketches, hacks,
proofs-of-concept, visualization experiments, or
whatever--are a means to visualize the desirable.
(Engineering prototypes are another matter.) Once
you've seen what is desirable, you know where you're
going and you can try to build something similar. If
you never see it though, you never get there.

Engineering prototypes are another matter. They are
used to visualize or realize what is possible. This is
important, but it's another job.

JS

4 Oct 2004 - 9:46am
Dave Malouf
2005

A few things about Kevin's reply to my posting:

1. Industrial Design and digital design are not the same thing.
That designer did their iterative process not in form, but in virtual, or
non-tangible mediums WAY before they did any physical forms, and the first
physical forms where done in Foam way before they did anything that
resembled the expense of dealing in carbon fiber. Its just a completely
different animal.

2. Flash or other media: I wasn't suggesting that you re-train. What I was
suggesting is that as a discipline we should be training people to do
prototypes in various media so that we reduce the confusion between
production and design.

To me this all stems from the confusions that have arisen due to the various
influences that are on us as Interaction Designers. We do not all come from
strict design backgrounds--this is a good thing. But the issue is really,
how to bring design methods into IxD. Designers work on a ton of models that
are always thrown out. It is the deconstructive process before the
reconstruction of a solution begins through evaluating models. It is about
creative flow.

Like I said the Industrial Designer will do about 100 drawings if they are
any good, for any given product they are working on, before they scan the
ones they like and computer model those for further comparision.

One last thing on the bike issue. Engineering. Materials engineering is
another level of expertise that does not translate to the IxD discipline.
Will there be an IxDer who will need this expertise. Probably, but this will
be project : individual specific. Most of us so far are working in virtual
environments.

-- dave

4 Oct 2004 - 9:46am
Pradyot Rai
2004

Whitney Quesenbery <wq at sufficiently.com> wrote:

> Whenever the question of whether a design prototype should be used as the
> basis for production code comes up, I usually look the most senior
> developer in the eye and ask, "Do you really want to base a product on the
> code that _I_ produce?"
>
> We can then go on to talk about what that prototype IS good for, and what
> would make the most sense as a way to deliver the design. The most common
> is that they would like careful attention to visual details so they can
> copy them directly.

Thank you!
Just few more example where Prototype feeds into final product --

> I once had a developer ask me for the Visio source for a paper prototype.
> When I asked why, he told me that he wanted to be able to simply cut and
> paste any text (we had worked hard on all of the messages in the program)
> into his code. So, we made sure that it was carefully edited and complete
> before we gave it to him.

And the two -- "Prototype" and "final codes" are decoupled completely,
but nothing is throw-away.

Most of the time, Hi-Fi prototype incarporates approval of all the
stakeholders for visual design, interaction modal, and designs that
forms the actual widgets (just to mention a few). I have had this
argument with developers who advocated it's use to be limited for
business demos, and training purposes only. This brings the notion
that prototype is a throw away code, to which I have very strong
objection.

Thanks,

Prady

4 Oct 2004 - 10:06am
Dave Malouf
2005

If it ain't thrown away, then it isn't a prototype. It IS production.

It is fine if your organization wants designers to do this type of work.
There are positives and negatives. But once it is used for production, then
it isn't a prototype anymore.

-- dave

4 Oct 2004 - 11:58am
Narey, Kevin
2004

Dave,

Thanks for your valuable input.
You still haven't explained why taking the opportunity to write good
quality, standards compliant (i.e. it becomes almost automated after a
while) mark-up/code up once and pass it on - is not advisable.

Perhaps, as Mr Seiden alluded to, there is genuine merit in collectively
defining a family of IxD proto-'types' all of which serve certain contextual
roles/functions in project phases involving IxD input? Engineering
prototype, Design Prototype, Engineering/Design prototype, paper, hi-fi,
low-fi etc.

On the face of it, to conclude categorically that all prototypes (given it's
dictionary definition) should be discarded, appears to eliminate crucial
business related 'options' without good reason. Like Petteri said 'I mean,
can you do the HTML part of it any differently anyway'.

Rgds

Kevin

**********************************************************************
gedas united kingdom limited
Registered in England no. 1371338

This email and any files transmitted with it are confidential
and it may be privileged.

It is intended solely for the use of the individual or entity to
whom it is addressed.

If you have received this in error, please contact the sender
and delete the material immediately.
**********************************************************************

4 Oct 2004 - 12:29pm
Listera
2004

Sarah Brodwall:

>> Prototypes should be the principal end-product of the design process,
>> preferably the only "deliverable."
>
> Could you elaborate on this statement?

Surely. We conceive and design so that a digital artifact can be created by
developers. We are not (and we should not be) in the business of writing the
executable code that drives that digital artifact. However, we are the ones
to give it purpose, shape, interactivity, flow, functionality, etc. I
advocate that the prototype we create should be the principal means with
which we convey all that to the developers who will implement in code our
intentions for the product. In my opinion, nothing else comes closer than an
interactive prototype in doing that job.

We may produce lots of other stuff (Post-It notes, paper drawings,
wireframes, mind maps, card sorts, etc) in that process, but these should be
kept as internal work product. Prototype reduces the risk of ambiguity at
many levels and is far more comprehensible than all these other abstract
interim "deliverables" that often confuse non-technical stake holders. I
think any designer who exposes a UML diagram to the marketing dept ought to
be shot on the spot, for example.:-)

> Also, would you say that today's reality for most designers deviates
> significantly from your ideal?

Yes, today most designers don't/can't prototype. That's a lethal
shortcoming. It invites developers to do the prototype, at which point,
often enough, technical considerations overtake design sensibilities.

Ziya
Nullius in Verba

4 Oct 2004 - 12:39pm
Petteri Hiisilä
2004

> Design "prototypes"--meaning sketches, hacks,
> proofs-of-concept, visualization experiments, or
> whatever--are a means to visualize the desirable.
> (Engineering prototypes are another matter.) Once
> you've seen what is desirable, you know where you're
> going and you can try to build something similar. If
> you never see it though, you never get there.
>
> Engineering prototypes are another matter. They are
> used to visualize or realize what is possible. This is
> important, but it's another job.
>

Yeah. Some replies reflect that not everybody see the difference between
these two. But it's important.

What engineers call prototypes and what interaction designers call
(should call) prototypes aren't the same thing. The first one test
feasibility. The second one tests interaction design solutions. First
one is usually coded. Second one is usually drawn. Both are created to
be discarded and improved.

A detailed, but static, HTML prototype isn't really a prototype anymore.
It's more like a spec that communicates the form and behavior. And does
it well, sometimes! If the HTML markup cannot be coded any differently,
I don's see why it should be rewritten. Just embed it into the final
jsp, nsf, cfm, cgi, shtml or whatever your're going to use to implement
the dynamics.

The stuff that engineers prototype in Java or C++ or PERL, that's
usually ugly. And they know it. It was never meant to be pretty. HTML
prototypes and PERL prototypes aren't the same thing. They both start
with p. But that's practically the only thing they have in common.

We should have a different terms for them to avoid the confusion.

How about:

- technical prototype
- interaction prototype

And:

- html-based form & behavior simulation

Best,
Petteri

--
Petteri Hiisilä
Palveluarkkitehti / Interaction Designer /
Alma Media Interactive Oy / NWS /
+358505050123 / petteri.hiisila at almamedia.fi

"I was told there's a miracle for each day that I try"
- John Petrucci

4 Oct 2004 - 12:42pm
Dave Malouf
2005

> Thanks for your valuable input.
> You still haven't explained why taking the opportunity to
> write good quality, standards compliant (i.e. it becomes
> almost automated after a
> while) mark-up/code up once and pass it on - is not advisable.

If you are doing production, it is not a prototype.

You asked re: a dictionary as well ... Here goes (dictionary.com)
pro.to.type Audio pronunciation of "prototype" ( P ) Pronunciation Key
(prt-tp)
n.

1. An original type, form, or instance serving as a basis or standard for
later stages.
2. An original, full-scale, and usually working model of a new product or
new version of an existing product.
3. An early, typical example.
[Sic. I skipped #4 b/c it was bilogical in nature]

#'s 1 & 3 fit what I'm trying to say ... Basis for later stages., example.

This means it is NOT final and thus not production.

As to the "why"?
To me this is about design methodologies, whereby, models (prototypes) are
created enmasse as a creative exercise, not as a goal towards conclusion.
Now, can you have a late prototype that is close to the final iteration,
sure ... But if that becomes "too" final then I believe your design team is
becoming a x-functional team and their time and directions are not being
well managed. I believe for efficiencies it is actually better to have the
design team move on to the next thing then have them sit and write
production quality code, especially in light of the fact that the problem
all started w/ a hand-off to development, meaning there are developers who
can do this.

Also, as someone who has had to deal w/ 80% code as a developer, I can say
that I prefer starting from scratch anyway. Just too hard to make that 20%
right, put in my api stuff and what not.

But I will concede this is a philosophical debate and subjective. There is
always more than one way to skin a cat, but I believe from the point of view
of a design methodology this one is most apt.

-- dave

4 Oct 2004 - 12:52pm
Listera
2004

Martyn Jones BSc:

> Ziya wrote:
>> Prototypes should be decoupled from development altogether.

> In the discussion of Prototypes here, are we only referring to projects that
> a design team has little experience of, or are we talking about all
> projects?

What would be a project "a design team has little experience of"?

Take me, for example. I have never designed anything for cell phones. If I
were given such a project, I'd approach it like any other project. I'd
immediately familiarize myself with the technical constraints involved. The
actual product might be done in Symbian OS, but I might use HTML, VB, Flash
or Flash Lite to do my prototype.

Yes, constraints might be different, but the fundamental patterns of design,
architecture and interaction are quite the same in any/most digital
platforms/projects.

> Also, what does a Project Lifecycle look like,

Simple: I get to the prototype as soon as humanly possible, and a day
earlier. Various stakeholders may get to see it at slightly different times,
but I as a designer start with it as soon as possible. As a designer,
frankly, I don't care about stuff like:

4. Code
5. Testing
6. Maintenance

Just as I don't want developers to get involved in the IA/UI/UX design
business, I don't expect designers to be entangled in post-prototype
production/implementation issues. Those responsible for these steps
(developers) should decide and execute.

Ziya
Nullius in Verba

4 Oct 2004 - 12:53pm
Brad Becker
2004

" Like Petteri said 'I mean, can you do the HTML part of it any differently
anyway'."

Most web apps today have very little true interactivity. They're very simple
and so you're tempted to skip disposable prototyping and start cranking out
the final markup. However:
- although only HTML might be sent to the client, the pages are often built
in JSP, ASP/ASP.NET, ColdFusion, PHP, etc. So the HTML isn't really the
source file.
- You might mock up quickly using <FONT> while the final version uses CSS.
- Portions of the final HTML file might come from server-side includes where
you put it in the prototype HTML file.
- The production version might use a templating system for reuse and
consistency.
- A content management system might provide the live content that you faked
up.
- You might use static jpg's where video (becoming more prevalent) is going
to be on the final site.

Plus, if you're only thinking of 'static' HTML pages, think ahead. The web
is becoming more dynamic, more interactive. When things get really
interactive, two things happen: one, you start to really need interaction
designers, and two, these applications start getting much more complicated
and expensive to build correctly. They require a fair amount of
sophisticated coding that most designers should avoid even if they are
capable because of the time and expense. Hence the need for disposable
prototypes.

A prototype is a proposal to a user of what an interaction could be like.
You should only put into the prototype what you need to get out of it at
that particular stage in the design conversation with the user. Most of
software development is centered around how to deal with
change--particularly the changes in requirements that will happen both all
through the dev cycle and in subsequent versions of the product. Cooper
talks about "scar tissue" in code from all the changes that happen in the
code. We also know how reluctant developers are to change code once they've
put a lot of time into it. These are the main reasons that we throw away
prototypes and state from the beginning our intention to throw them away.

The final prototype can be the lion's share of a spec if done right. And
occasionally, you got lucky: you nailed the interaction, the requirements
haven't changed, and the code (if any) is so well architected that it will
scale to production.

Don't hold your breath. :)

Brad Becker
Sr. Product Designer
Macromedia, Inc.

4 Oct 2004 - 12:57pm
Brad Becker
2004

A common term for "technical prototype" you have below is "POC"--"Proof of
Concept".

-Brad

-----Original Message-----
From:
discuss-interactiondesigners.com-bounces at lists.interactiondesigners.com
[mailto:discuss-interactiondesigners.com-bounces at lists.interactiondesigners.
com] On Behalf Of Petteri Hiisilä
Sent: Monday, October 04, 2004 10:39 AM
To: discuss at interactiondesigners.com
Subject: Re: [ID Discuss] Discarding Prototypes

We should have a different terms for them to avoid the confusion.

How about:

- technical prototype
- interaction prototype

4 Oct 2004 - 1:11pm
Petteri Hiisilä
2004

>> How about:
>>
>> - technical prototype
>> - interaction prototype

> A common term for "technical prototype" you have below is
> "POC"--"Proof of Concept".

Hmm......... mmkay

Should we rewrite the subject of this conversation to

"[ID Discuss] Discarding POCs"

I mean,

If the technical prototype is actually a "POC" and interaction prototype
isn't code at all (but, say, Flash or paper), where's that "prototype"
code that we should discard?-)

- Petteri

4 Oct 2004 - 1:16pm
Brad Becker
2004

Modeling some interactions requires javascript/actionscript/lingo/whatever.
However, we're not just talking about code. You don't want to spend an hour
perfecting an image if the context, size, and even existence of that image
is still being decided.

Make sense?

-Brad

-----Original Message-----
From: Petteri Hiisilä [mailto:petteri.hiisila at almamedia.fi]
Sent: Monday, October 04, 2004 11:11 AM
To: Brad Becker
Cc: discuss at interactiondesigners.com
Subject: Re: [ID Discuss] Discarding Prototypes

>> How about:
>>
>> - technical prototype
>> - interaction prototype

> A common term for "technical prototype" you have below is
> "POC"--"Proof of Concept".

Hmm......... mmkay

Should we rewrite the subject of this conversation to

"[ID Discuss] Discarding POCs"

I mean,

If the technical prototype is actually a "POC" and interaction prototype
isn't code at all (but, say, Flash or paper), where's that "prototype"
code that we should discard?-)

- Petteri

4 Oct 2004 - 1:21pm
Petteri Hiisilä
2004

> Make sense?
>

Sure :)

- Petteri

4 Oct 2004 - 2:41pm
Pradyot Rai
2004

David Heller <dave at ixdg.org> wrote:

> You asked re: a dictionary as well ... Here goes (dictionary.com)
> pro.to.type Audio pronunciation of "prototype" ( P ) Pronunciation Key
> (prt-tp)
> n.
>
> 1. An original type, form, or instance serving as a basis or standard for
> later stages.
> 2. An original, full-scale, and usually working model of a new product or
> new version of an existing product.
> 3. An early, typical example.
> [Sic. I skipped #4 b/c it was bilogical in nature]
>
> #'s 1 & 3 fit what I'm trying to say ... Basis for later stages., example.
>
> This means it is NOT final and thus not production.

If you see # 2 again, you may see your answer change :-)
Point# 2 is the whole matter of discussion here.

> As to the "why"?
> To me this is about design methodologies, whereby, models (prototypes) are
> created enmasse as a creative exercise, not as a goal towards conclusion.
> Now, can you have a late prototype that is close to the final iteration,
> sure ... But if that becomes "too" final then I believe your design team is
> becoming a x-functional team and their time and directions are not being
> well managed.

With all due respect, I still don't see the reason of "why". I read
that you have preference of how Design team should be devoting time.
However, there are cases where this really doesn't fit well, and you
have to go climb down to the shop floor, and demonstrate 'How?". The
issue in question is High fidelity prototype, and I guess if it
requires Design team to be x-functional, so be it. It is challenge for
design team, and often is case with the design of version 1.0

> I believe for efficiencies it is actually better to have the
> design team move on to the next thing then have them sit and write
> production quality code, especially in light of the fact that the problem
> all started w/ a hand-off to development, meaning there are developers who
> can do this.

Well there are developers who can do design too. And for sake of
efficiancies, most of the managers keep the designers away from their
version 1.0. You are making lots of assumptions!

I know there are many scenarios in which Designers stays in the team
from start till the end, all for the efficiency of the team, and
better quality of the product. I also know that in some already
established products (and closer to the argument you are making), for
only subsequent releases, designers only work on visio (wireframes) to
propose designs.

Again, what I am siding with is -- "Hi-Fi Prototype" is not throw-away
code, and if you are a designer who take this challenge to do it, then
do it. It will always be better than developer asking you at the later
stage -- "what were you thinking, when you proposed this design,
Dude!"

Prady

4 Oct 2004 - 5:22pm
Listera
2004

Petteri Hiisilä:

> If the technical prototype is actually a "POC" and interaction prototype
> isn't code at all (but, say, Flash or paper), where's that "prototype"
> code that we should discard?-)

If I were the dictator of Prototypedom, I'd require the prototype file to be
obfuscated, compiled and encrypted so it can't be borrowed from. (Where's
DMCA when you need it.:-) In this year of elections, here's to the
separation of powers!

I think it'd be useful to turn the question upside down: Under what
suboptimal circumstances can/should/does the prototype go into production?

A) Company is too cheap to hire separate individuals/teams.
B) Company is too clueless to see the difference between prototype and
production code.
C) Developers are too lazy to do their own prototype.
D) Designers are too eager to play developers.
E) Project is too small for any role segmentation.
F) That's just "the way it's done here."

Any others?

Ziya
Nullius in Verba

4 Oct 2004 - 4:45pm
Joe Sokohl
2004

I've always followed Steve McConnell's approach to prototyping, specifically
his "throwaway prototyping" (Chapter 38 of "Rapid Development" and also
discussed in "Software Project Survival Guide"). Here' shis intro:
"With Throwaway Prototyping, code is developed to explore factors critical
to the system's success, and then that code is thrown away. The prototyping
implementation uses programming languages or development practices or both
that are much faster than the target language and practices. The user
interface is prototyped far more commonly than any other part of the system,
but other parts of some systems can also benefit from being prototyped.,
When used as a requirements-specification aid, the Throwaway Prototyping
practice can accelearate projects based on traditional lifecycle models,
such as DoD projects. It can be initiated at either a management or
technical level."

He also discusses "evolutionary prototyping," which is "a lifecycle model in
which the system is developed in increments so that it can readily be
modified in reponse to end-user and customer feedback." In the main, though,
it's the support of and discussion about throwaway prototyping that's
convinced me that it works better for interaction design than evolutionary
prototyping.

joe

--
GMX ProMail mit bestem Virenschutz http://www.gmx.net/de/go/mail
+++ Empfehlung der Redaktion +++ Internet Professionell 10/04 +++

5 Oct 2004 - 9:45am
Petteri Hiisilä
2004

>>If the technical prototype is actually a "POC" and interaction prototype
>>isn't code at all (but, say, Flash or paper), where's that "prototype"
>>code that we should discard?-)

That was humor. Choose irony and you might tell the best joke you're
ever going regret :)

> I think it'd be useful to turn the question upside down: Under what
> suboptimal circumstances can/should/does the prototype go into production?
>
> A) Company is too cheap to hire separate individuals/teams.
> [...]

To me this question isn't very interesting or even valid, because you
don't define what you mean by a "prototype". Like I and Joshua said,
there are at least two kind of prototypes.

What's interesting is,

1. Why any company with interaction designers would build interactive
prototypes that involve code? (other than flash/html/javascript/css etc.)

Isn't a set paper prototypes, whiteboard drawings, static bitmaps, Flash
animation, static set of HTML pages or similar enough? Why make any code
that some dumb manager would put into production?

I call these interaction prototypes and communicate them to managers by
visually showing them.

***

2. Why would an engineering department build prototypes that interact
with users?

Why don't they produce a low-level proof of concept and operate it with
test scripts or similar? Why bother with GUI for instance? Why make any
visible interface that some dumb manager would put into production?

I call these technical prototypes and communicate them to managers by
showing the results as a written report. Not a "working" GUI.

***

And I'd never talk about "just prototypes". Only when you confuse them,
you start the trouble. If you keep them separate, the dumb managers
can't force you save the code, because there's almost no code to save.

BUT!

For example, if you're doing transactional web sites, an interaction
prototype complete with working HTML/CSS/Javascript might well have some
parts that are worth using inside the "real" program logic. Then you're
not talking about a prototype anymore. It's more like a model.

That's exactly what we do. Here's how:

1. A whiteboard or paper prototype, drawn by hand (by me). We design
with this.

2a. A Photoshop or Freehand prototype, drawn with computer (by our
visual designer). We communicate with this. Sometimes we skip this phase.

2b. Engineers do their technical prototypes (proof-of-concepts) but with
no real GUI. The code is discarded. Sometimes they skip this phase.

3a. An HTML model, complete with clean Javascript and css (by our web
designer, who's also the visual designer). The pages are static, but we
use server side includes to avoid copy-pasting the code between static
pages. Some pages are linked together to simulate the key path scenarios.

3b. A form & behavior spec (by me). This is a written document that
explains how the application will work. It has lots of hyperlinks to the
static HTML prototype.

This is usually enough to communicate the whole design and behavior to
the developers, managers, customers and end users, before the
construction starts.

The product is implemented with JavaServer Pages.

Best,
Petteri

--
Petteri Hiisilä
Palveluarkkitehti / Interaction Designer /
Alma Media Interactive Oy / NWS /
+358505050123 / petteri.hiisila at almamedia.fi

"The unbroken spirit
Obscured and disquiet
Finds clearness this trial demands"
- Dream Theater

5 Oct 2004 - 11:26am
Pradyot Rai
2004

This discussion is branching away from where it started, without
converging to anything. I wish to try once, to pull our attention
again to the initial question, that was --

As per "Kevin" --
> At my company we tend to prototype web-applications using
> HTML/JavaScript/Hard-Coded data (hi-fi I guess?) during the design phase of
> a project. This mark-up is generally written using UI and front-end coding
> standards and is therefore of around 75%-90% application grade quality.

> I've read/heard much about discarding prototypes, but never been able to see
> a good reason for throwing away this reasonably good quality code.

This is a typical scenario, and designers are asked to deliver Hi-Fi
prototype for various reasons. And there has always been some
resistance from engineering to accept their quality work for the code.
Let's answer following questions --

1. What is the loss of the organization, if it allows Designers to
share "Product Code"? In other words, why not borrowing stuffs from
prototype?

2. Why Prototype (any kind of prototype) should be last stop for designers?

3. Has any of us ever witnessed that after design (with Visio,
Photoshop, freehand alone) is done and handed over to Engineering, it
still comes out as surprise? Have you ever witness yourself saying,
"No, that is not what I proposed?" How do you handle such situations?

Point # 1 & 2 covers the central theme of the discussion. If designers
are not involved in UI engineering, quality is at stake. It asks you
to wonder, what all can you do to make sure design is delivered.

Point# 3 covers one aspect of why Hi-Fi Prototyping is done for
suggesting and reinforcing requirements. I am not seen any arguments
made to convince what is the alternative.

Once again, it is about web based apps., high-fidelity prototypes, and
designers putting their hands in the code.

Thanks,

Prady

5 Oct 2004 - 12:32pm
Petteri Hiisilä
2004

Hi Prady,

Good catch. Thanks.

> 1. What is the loss of the organization, if it allows Designers to
> share "Product Code"? In other words, why not borrowing stuffs from
> prototype?

I believe it's fine, as long as it's not a managerial choice. I trust
our developers' and my own judgement on this issue.

But I don't see why interaction designers would want to create or use
"product code", except maybe HTML/CSS/Javascript...

> 2. Why Prototype (any kind of prototype) should be last stop for designers?

It shouldn't. A form and behavior spec should. :)

As every interaction designer knows, getting further than this is risky:
there's a hidden but serious conflict of interest. Easy-to-use and
easy-to-code typically aren't the same program. If you design, don't
code. It's better, quicker and more effective that way.

Cooper has written a lot about this. I especially like the
architect-engineer-programmer triad, which was mentioned in the preface
of the Inmates, 2nd ed.

It was also mentioned here:

http://www.ftponline.com/vsm/2003_09_14th/magazine/departments/softwarearchitect/

"Today, Web designers are called programmers, programmers are called
engineers, engineers are called architects, and architects never get
called."

- Architects: design for humans
- Engineers: design for CPU
- Programmers: construction for shipment

> 3. Has any of us ever witnessed that after design (with Visio,
> Photoshop, freehand alone) is done and handed over to Engineering, it
> still comes out as surprise? Have you ever witness yourself saying,
> "No, that is not what I proposed?" How do you handle such situations?

That used to happen a lot. Nowadays:

1. We have a written spec with lots of pictures, AND a static HTML
snapshot / model of the application. They're heavily interlinked.

2. We use TWiki to store the written specs. The developers put their
comments, questions and requests right into the spec.

3. We communicate the design personally to the engineers and
programmers. We make sure that they know everything and get excited
about the project. They usually have very good questions.

4. We read their IRC channel. If we notice that they're discussing
irrelevant issues, we talk to them.

5. They know that they should direct any human issues to us, and they do
tap my shoulder every now and then. They still like to call the human
issues "requirements" :)

6. We use Bugzilla with a new field: Affects. The choices are Human,
Engineer, Programmer, Admin, Manager.

7. Etc.

Best,
Petteri

--
Petteri Hiisilä
Palveluarkkitehti / Interaction Designer /
Alma Media Interactive Oy / NWS /
+358505050123 / petteri.hiisila at almamedia.fi

"I was told there's a miracle for each day that I try"
- John Petrucci

15 Oct 2004 - 1:13pm
cfmdesigns
2004

Prady Rai <pradyotrai at gmail.com> writes:

>It will always be better than developer asking you at the later
>stage -- "what were you thinking, when you proposed this design,
>Dude!"

I end up asking that all the time on the projects I work on, in both
forms of the question:

"What were you thinking when you proposed this design?" What is the
background to this decision? Did it incorporate X, Y, and Z? It
isn't clear to me that it's better than ABC. What special problem
does it solve?

"What *were* you thinking?" Did you really think this through? Why
in God's name would a user want to do this that way? You call that
discoverable? This is really just a hacked together prototype that
you didn't recode, isn't it?

The first way of asking the question is actually the one that seems
to cause the most offense. Asking developers or designers to explain
things -- not necessarily justify them, but merely explain the logic
and purpose -- tends to get big pushback.
--

----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
Jim Drew Seattle, WA jdrew at adobe.com
http://home.earthlink.net/~rubberize/Weblog/index.html (Update: 10/14)

Syndicate content Get the feed