Usability testing on the cheap- Silverback, ?

29 Jul 2008 - 1:45pm
2 years ago
11 replies
4661 reads
Mike Dunn
2008

Hey all, I'm doing some research on different tiers of usability testing,
and I love the new Silverback app that Clear Left just put out (
http://www.silverbackapp.com/) as a potential lower-end solution (we already
have partnerships in place for the high-end services). Are there any other
apps like Silverback out there? What do you guys use?

--
Michael Dunn

Comments

29 Jul 2008 - 2:57pm
Adam Korman
2004

I've used Morae in the past. It's sort of like Silverback (from what I
can see on the site), but more sophisticated. Main differences:

- it's Windows based
- in addition to recording sessions, you can stream live to observers'
computers
- it has tools for annotating sessions and creating graphs, reports,
etc.
- it's a lot pricier than Silverback

http://www.techsmith.com/morae.asp

-Adam

On Jul 29, 2008, at 12:45 PM, Michael Dunn wrote:

> Hey all, I'm doing some research on different tiers of usability
> testing,
> and I love the new Silverback app that Clear Left just put out (
> http://www.silverbackapp.com/) as a potential lower-end solution (we
> already
> have partnerships in place for the high-end services). Are there
> any other
> apps like Silverback out there? What do you guys use?

29 Jul 2008 - 5:14pm
Bojhan
2007

I am not sure what to think of Silverback, I like a really simple
solution that works really well if you do on-site testing. But Morae
(UserVue) has great remote usability testing posiblites. And nowdays for
testing websites, there are a lot of great solutions that record the
users screen with javascript, obviously this doesnt work that well if
your outside a browser envoirment. There are lots of solutions out there
that offer screen recording, audio and webcam such as GoToMeeting
(really nice), but for really simple on-sight testing with mac people
around silberback seems good.

Michael Dunn schreef:
> Hey all, I'm doing some research on different tiers of usability testing,
> and I love the new Silverback app that Clear Left just put out (
> http://www.silverbackapp.com/) as a potential lower-end solution (we already
> have partnerships in place for the high-end services). Are there any other
> apps like Silverback out there? What do you guys use?
>
>

29 Jul 2008 - 4:46pm
Mary Deaton
2008

I use Morae for all observation testing in a lab-like setting. It has
three components - Recorder, Observer, and Manager; you can buy each
individually. Recorder is installed on the participant machine and not
only records screen video and any audio, it also allows you to deliver
a survey automatically at the end of the tasks defined for the study
and, if set up to work with observer, allows the facilitator to index
events in real-time. This makes it extremely easy to get time-on-task,
error, and other event data.

Observer is set up on a computer networked to the participant
computer; you must have a separate Observer license for each computer
running observer. I have two licenses, one for the facilitator machine
and one that I typically set up in a conference room for the project
team to use to observe in real time. You could also use any screen
sharing application or conferencing application along with Observer,
but this would not allow the observers to also tag events during a
test session.

Manager is the component for doing analysis. You can view and edit the
video and audio, as well as get charts and graphs showing data for the
markers you set during observation. Without the markers, you get
virtually no analyzable data other than what you may have set to track
mouse clicks, page changes, time on pages, and this sort of data. You
can even create highlight videos based on specific markers you set
during each session or by other filtering of markers.

It is perfectly feasible to use Recorder only if the facilitator or a
note taker is in the same room as the participant and you are using it
in order to get a video and audio record. You still need to take notes
and record event data manually and do the analysis manually.

By using Recorder and Observer, the facilitator or note taker can sit
in a separate room, but you if use markers and other indexing you do
not have the integrated tools Manager gives you to do analysis of the
markers. You need all three components to make good use of Morae.

TechSmith has a remote usability product called UserVue which can be
"rented" for one month at a time and collects quite a bit of data that
can be processed in Morae Manager. Using these two together is a cost
saving approach. I have used all of these components when doing
testing in which some sessions were in a lab and some were remote.

I have worked in labs that use Usability Science equipment and in
Microsoft labs where most of the data collection tools are home grown
and not very sophisticated in my opinion. I much prefer using Morae
because it is all digital, it is portable, it is flexible, and it
allows me to use one interface to collect all data except hand-written
notes.

Morae is pricey compared to using video cameras, note taking, and a
spreadsheet, but I have found it saves me incredible amounts of time
during set-up and analysis. I can get all upgrades through an annual
maintenance agreement. It is much less pricey than systems that use
extensive video camera inputs, sound mixing boards, and pricey
consoles you cannot move out of a lab once it they are in place.

Morae is cheaper than several Web-hosted system I have looked at,
especially since I can use some of the Morae components when doing
studies for non-computer products. As a consultant, Morae lets me
supply my own tools for conducting testing, avoid high-cost lab
rentals, and carry a much smaller box of tools to a client's office or
a hotel conference room.

--
Mary Deaton
Deaton Interactive Design

29 Jul 2008 - 6:21pm
Todd Warfel
2003

We've completely removed Morea from our testing environment for a
number of reasons:
* Doesn't work on Mac (we test both Mac/Windows and need a solution
that will work on both)
* Proprietary video format that can only be edited in their video editor
* Video editor — have you tried to use this thing? It's like trying to
pull a jet ski w/a semi. Way too complicated for what you need for
editing video. Hey Morea, take some clues from iMovie.
* While the note taking and marking capabilities are nice, we never
used them. We took notes in our own research framework that allows us
to tag each observation and do much better analysis. Additionally, the
facilitator takes notes on their script.
* Reliability issues. We've had too many cases where Morea video got
corrupted. Client's weren't too happy.
* Can't record video of sites that have streaming video. It kills the
recording.
* Setting up Morea required giving ourselves a 2 hour window each time
just to make sure patches were up to date, Windows was working well,
video cameras were working, etc.

So, how do we do it now? Well, our lab consists of
* Two (2) Intel based Macs. This lets us test both Windows and Mac.
* We use OS X's built in screen sharing to view the test participant's
machine. We open an iChat session to get the picture-in-picture and
have audio.
* Recording is done via SnapZPro.

Disadvantages:
* We have to use two pieces of software (iChat, SnapZ), instead of one
integrated solution.
* Video rendering takes longer than Morea

Advantages:
* We can test both Mac/Win
* Stability — it's never crashed, never corrupted a video file
* Setup literally takes 10-15 minutes compared to 1.5-2 hours
* Videos are in a standard .mov/.mpg format that we can edit with
pretty much any video editor
* We can edit videos with something simple like iMove w/o having to
edit them
* Total cost of two Intel iMacs ($1200 ea) and SnapZ ($69) = <$2500.
For Morea, we'd need two equivalent PCs ($800-1200 ea), plus a Mac
($1200), plus Morea ($1200), plus SnapZ ($69) = $3700-4900 depending
on whether or not you buy on PC and one Mac (run Parallels), or two
PCs and one Mac.

We're going to be looking at Silverback for some future tests and give
it a try. I like the highlighting effect for clicks. I'm not sure how
the note taking capabilities will work when the participant is using
the machine. Perhaps you can take notes using a remote machine? I'll
have to see.

Cheers!

Todd Zaki Warfel
President, Design Researcher
Messagefirst | Designing Information. Beautifully.
----------------------------------
Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
Twitter: zakiwarfel
----------------------------------
In theory, theory and practice are the same.
In practice, they are not.

29 Jul 2008 - 6:28pm
SemanticWill
2007

Mary -

I had the opportunity to do some testing with Todd using his homemade system
and definitely think it's a better solution than Morea - it's streamlined,
it gets the technology out of the way so you can focus on observing the user
experience.

On Tue, Jul 29, 2008 at 8:21 PM, Todd Zaki Warfel <lists at toddwarfel.com>wrote:

> We've completely removed Morea from our testing environment for a number of
> reasons:
> * Doesn't work on Mac (we test both Mac/Windows and need a solution that
> will work on both)
> * Proprietary video format that can only be edited in their video editor
> * Video editor — have you tried to use this thing? It's like trying to pull
> a jet ski w/a semi. Way too complicated for what you need for editing video.
> Hey Morea, take some clues from iMovie.
> * While the note taking and marking capabilities are nice, we never used
> them. We took notes in our own research framework that allows us to tag each
> observation and do much better analysis. Additionally, the facilitator takes
> notes on their script.
> * Reliability issues. We've had too many cases where Morea video got
> corrupted. Client's weren't too happy.
> * Can't record video of sites that have streaming video. It kills the
> recording.
> * Setting up Morea required giving ourselves a 2 hour window each time just
> to make sure patches were up to date, Windows was working well, video
> cameras were working, etc.
>
> So, how do we do it now? Well, our lab consists of
> * Two (2) Intel based Macs. This lets us test both Windows and Mac.
> * We use OS X's built in screen sharing to view the test participant's
> machine. We open an iChat session to get the picture-in-picture and have
> audio.
> * Recording is done via SnapZPro.
>
> Disadvantages:
> * We have to use two pieces of software (iChat, SnapZ), instead of one
> integrated solution.
> * Video rendering takes longer than Morea
>
> Advantages:
> * We can test both Mac/Win
> * Stability — it's never crashed, never corrupted a video file
> * Setup literally takes 10-15 minutes compared to 1.5-2 hours
> * Videos are in a standard .mov/.mpg format that we can edit with pretty
> much any video editor
> * We can edit videos with something simple like iMove w/o having to edit
> them
> * Total cost of two Intel iMacs ($1200 ea) and SnapZ ($69) = <$2500. For
> Morea, we'd need two equivalent PCs ($800-1200 ea), plus a Mac ($1200), plus
> Morea ($1200), plus SnapZ ($69) = $3700-4900 depending on whether or not you
> buy on PC and one Mac (run Parallels), or two PCs and one Mac.
>
> We're going to be looking at Silverback for some future tests and give it a
> try. I like the highlighting effect for clicks. I'm not sure how the note
> taking capabilities will work when the participant is using the machine.
> Perhaps you can take notes using a remote machine? I'll have to see.
>
>
> Cheers!
>
> Todd Zaki Warfel
> President, Design Researcher
> Messagefirst | Designing Information. Beautifully.
> ----------------------------------
> Contact Info
> Voice: (215) 825-7423
> Email: todd at messagefirst.com
> AIM: twarfel at mac.com
> Blog: http://toddwarfel.com
> Twitter: zakiwarfel
> ----------------------------------
> In theory, theory and practice are the same.
> In practice, they are not.
>
>
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

--
~ will

"Where you innovate, how you innovate,
and what you innovate are design problems"

---------------------------------------------------------------------------------------------
Will Evans | User Experience Architect
tel +1.617.281.1281 | will at semanticfoundry.com
twitter: https://twitter.com/semanticwill
---------------------------------------------------------------------------------------------

29 Jul 2008 - 7:10pm
Todd Warfel
2003

On Jul 29, 2008, at 8:55 PM, Mary Deaton wrote:

> I have been using Morae for over two years and have never had any of
> the problems you report with patches or video corruption. It is not
> necessary to edit video in Morae, you can export it and edit in
> anything you want. However, you lose the ability to search on
> specific markers and tell it to make a highlight of only the marked
> items.

The issue with video corruption is a known problem with Morea. They
even have tech notes on it. Additionally, they have a document file
size limit (something like 2 hours), but I can't recall as it's been a
while since I spoke with their tech group. Also, they have a known
problem with not being able to record video of video. So, if you're
testing something like YouTube, Morea will not work — the video gets
corrupted. This is a known issue with Morea according to TechSmith.

> I transferred my research framework into the marker system and have
> created my own markers, finding Morae's too general. As for setup,
> it takes me only the time required to set up the participant machine
> if I go to a client's site and use their machine. That is usually
> less than 20 minutes. My biggest problem has been client networks,
> not the Morae software.

We've never ever seen it setup in less than 20 minutes. Seriously. I'm
measuring time from the point I walk in the door and touch the power
button on the computer to the point when I've run a test recording and
it works okay. The quickest we've ever seen was over an hour. Not to
mention if you're taking your own machine, then you've got cables to
plug in, boot time for Windows (that alone takes 5-10 minutes), giving
Morea machines access to each other, etc. I've never seen this done in
under 20 minutes when you have two machines — one for testing, one for
viewing.

Just two weeks ago, we were testing at a client's facility, using
their equipment w/Morea. They had done some tests the previous week.
It still took over an hour to get everything working properly. Much of
this was do to Windows, but that's still a factor in the system.

> I use a small Web cam for the participant machine if using a
> client's machine. If I use my own machine, it has a built in Web
> cam. I am not very fond of using multiple video cameras for testing
> someone sitting at a computer typing, especially since my policy is
> to never include a participants face in material given to a client
> for reasons of
> privacy.

We only use 1 video camera pointed at the participant.

> I also never do testing on Macs.

While it's only 10-20% of our participants, we prefer to provide the
most accurate data possible. So, if a participant is a Mac/FF person,
it's not going to provide the same experience putting them on Win/FF.
We prefer to filter out any bias that could be contributed to OS
differences/experience.

Additionally, we don't intentionally rule out Mac participants because
our system won't support it. It's our job to have a system that
supports the customers of our clients and the systems we test. While I
realize other companies don't take the same approach we do, as a
researcher, I feel I'm not doing my job if I use a system that doesn't
support the customer's native environment. And yes, when possible, we
go to people's homes/offices. When not possible, we do our best to
simulate the same (similar) environment at our space.

Cheers!

Todd Zaki Warfel
President, Design Researcher
Messagefirst | Designing Information. Beautifully.
----------------------------------
Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
Twitter: zakiwarfel
----------------------------------
In theory, theory and practice are the same.
In practice, they are not.

29 Jul 2008 - 8:12pm
Carol J. Smith
2007

I haven't used Silverback yet - and as a Mac user it would certainly
simplify things. I do see a big limitation for remote testing - only Mac
users (with the proper setup) can be recruited for testing. That will (in
most cases) significantly decrease your pool of participants.

I have used UserVue - participants in that case have to use Windows. I have
run remote tests with UserVue on a Mac (booting as a Windows machine) and
had no issues (beyond what was already described). It's a nice program -
though I am very curious to try Todd's solution. :-)
Mora Manager must be run on a Windows machine (for now a Mac booting as a
Windows machine does not work).

Carol

-------------------
Carol J. Smith
Principal Consultant, Midwest Research, LLC
http://www.mw-research.com

Usability Professionals' Association, Director of Global Outreach
http://www.usabilityprofessionals.org

Cell: +1 (773) 218-6568
Email: carol at mw-research.com

(Speaking only for myself in this email.)

On Tue, Jul 29, 2008 at 3:45 PM, Michael Dunn <mike at foolishstudios.com>wrote:

> Hey all, I'm doing some research on different tiers of usability testing,
> and I love the new Silverback app that Clear Left just put out (
> http://www.silverbackapp.com/) as a potential lower-end solution (we
> already
> have partnerships in place for the high-end services). Are there any other
> apps like Silverback out there? What do you guys use?
>
> --
> Michael Dunn
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... discuss at ixda.org
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>

30 Jul 2008 - 1:46pm
Darrell Benatar
2008

If you're willing to give up the ability to moderate a usability
session then for $19 per tester you can outsource the whole thing to
www.UserTesting.com (I work there). We've built an online network of
pre-screened user testers who can "think out loud" and stay on task
without a moderator. So you just post a request for users (who match
your target demographic) to do a series of tasks on your site, and in
an hour or so you're watching screencasts of them browsing your site
and speaking their thoughts.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=31694

31 Jul 2008 - 5:22am
Taras Brizitsky
2008

Screen Flow http://varasoftware.com/products/screenflow/ might look
like a better alternative for just $50 more: build-in viewer-editor,
capturing both camera and screen images with ability to arrange them
later. Plus Mic/System sound recording%u2026

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=31694

31 Jul 2008 - 11:01am
janasedivy
2008

I really live a FREE tool called OvoLogger. it's just for taking
notes (well, actually, it's a lot more than that but only the
note-taking part is free).

You can't catch mouseclicks with it but otherwise, it gives you much
of Morae functionality. You can measure task completion times, you
can preconfigure tasks, you can generate an HTML report summarizing
your users (although admittedly, it's not very pretty). You can
time stamp observations so that you can know where to look for
something in the video. You dont' have fancy graphing utilities and
no built in video editing but if you need to go cheap, it's a great
tool. Did i mention it's FREE?

Combine OvoLogger with a screen capture program (Camtasia, Captivate)
plus a web cam and microphone and you've got yourself a pretty decent
little lab for under $300.

- Jana

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=31694

26 Mar 2012 - 9:53am
violet424
2010

I've recently used silverback and like the simplicity of it but it's almost too simple.

It isn't true that you need to recruit only MAC users for remote testing. There is a workaround for that... We used Adobe COnnect web conferencing software and asked participants to "share" their screen. Then we recorded the session via Silverback using our own computer.

But I don't like that you can only select "highlights" DURING recording, not after the fact.

Syndicate content Get the feed