Usability testing on the cheap- Silverback, ?

Hey all, I'm doing some research on different tiers of usability testing, and I love the new Silverback app that Clear Left just put out ( http://www.silverbackapp.com/ ) as a potential lower-end solution (we already have partnerships in place for the high-end services). Are there any other apps like Silverback out there? What do you guys use?

-- Michael Dunn

I've used Morae in the past. It's sort of like Silverback (from what I can see on the site), but more sophisticated. Main differences:

http://www.techsmith.com/morae.asp

-Adam

On Jul 29, 2008, at 12:45 PM, Michael Dunn wrote:

Hey all, I'm doing some research on different tiers of usability testing, and I love the new Silverback app that Clear Left just put out ( http://www.silverbackapp.com/ ) as a potential lower-end solution (we already have partnerships in place for the high-end services). Are there any other apps like Silverback out there? What do you guys use?

I use Morae for all observation testing in a lab-like setting. It has three components - Recorder, Observer, and Manager; you can buy each individually. Recorder is installed on the participant machine and not only records screen video and any audio, it also allows you to deliver a survey automatically at the end of the tasks defined for the study and, if set up to work with observer, allows the facilitator to index events in real-time. This makes it extremely easy to get time-on-task, error, and other event data.

Observer is set up on a computer networked to the participant computer; you must have a separate Observer license for each computer running observer. I have two licenses, one for the facilitator machine and one that I typically set up in a conference room for the project team to use to observe in real time. You could also use any screen sharing application or conferencing application along with Observer, but this would not allow the observers to also tag events during a test session.

Manager is the component for doing analysis. You can view and edit the video and audio, as well as get charts and graphs showing data for the markers you set during observation. Without the markers, you get virtually no analyzable data other than what you may have set to track mouse clicks, page changes, time on pages, and this sort of data. You can even create highlight videos based on specific markers you set during each session or by other filtering of markers.

It is perfectly feasible to use Recorder only if the facilitator or a note taker is in the same room as the participant and you are using it in order to get a video and audio record. You still need to take notes and record event data manually and do the analysis manually.

By using Recorder and Observer, the facilitator or note taker can sit in a separate room, but you if use markers and other indexing you do not have the integrated tools Manager gives you to do analysis of the markers. You need all three components to make good use of Morae.

TechSmith has a remote usability product called UserVue which can be "rented" for one month at a time and collects quite a bit of data that can be processed in Morae Manager. Using these two together is a cost saving approach. I have used all of these components when doing testing in which some sessions were in a lab and some were remote.

I have worked in labs that use Usability Science equipment and in Microsoft labs where most of the data collection tools are home grown and not very sophisticated in my opinion. I much prefer using Morae because it is all digital, it is portable, it is flexible, and it allows me to use one interface to collect all data except hand-written notes.

Morae is pricey compared to using video cameras, note taking, and a spreadsheet, but I have found it saves me incredible amounts of time during set-up and analysis. I can get all upgrades through an annual maintenance agreement. It is much less pricey than systems that use extensive video camera inputs, sound mixing boards, and pricey consoles you cannot move out of a lab once it they are in place.

Morae is cheaper than several Web-hosted system I have looked at, especially since I can use some of the Morae components when doing studies for non-computer products. As a consultant, Morae lets me supply my own tools for conducting testing, avoid high-cost lab rentals, and carry a much smaller box of tools to a client's office or a hotel conference room.

-- Mary Deaton
Deaton Interactive Design

I am not sure what to think of Silverback, I like a really simple solution that works really well if you do on-site testing. But Morae (UserVue) has great remote usability testing posiblites. And nowdays for testing websites, there are a lot of great solutions that record the users screen with javascript, obviously this doesnt work that well if your outside a browser envoirment. There are lots of solutions out there that offer screen recording, audio and webcam such as GoToMeeting (really nice), but for really simple on-sight testing with mac people around silberback seems good.

Michael Dunn schreef:
Hey all, I'm doing some research on different tiers of usability testing, and I love the new Silverback app that Clear Left just put out ( http://www.silverbackapp.com/ ) as a potential lower-end solution (we already have partnerships in place for the high-end services). Are there any other apps like Silverback out there? What do you guys use?

Mary -

I had the opportunity to do some testing with Todd using his homemade system and definitely think it's a better solution than Morea - it's streamlined, it gets the technology out of the way so you can focus on observing the user experience.

On Tue, Jul 29, 2008 at 8:21 PM, Todd Zaki Warfel lists at toddwarfel.comwrote:

We've completely removed Morea from our testing environment for a number of reasons: Doesn't work on Mac (we test both Mac/Windows and need a solution that will work on both) Proprietary video format that can only be edited in their video editor Video editor — have you tried to use this thing? It's like trying to pull a jet ski w/a semi. Way too complicated for what you need for editing video. Hey Morea, take some clues from iMovie. While the note taking and marking capabilities are nice, we never used them. We took [trim]

—
~ will

"Where you innovate, how you innovate,
and what you innovate are design problems"

Will Evans | User Experience Architect
tel +1.617.281.1281 | will at semanticfoundry.com
twitter: https://twitter.com/semanticwill

On Jul 29, 2008, at 8:55 PM, Mary Deaton wrote:

I have been using Morae for over two years and have never had any of the problems you report with patches or video corruption. It is not necessary to edit video in Morae, you can export it and edit in anything you want. However, you lose the ability to search on specific markers and tell it to make a highlight of only the marked items.

The issue with video corruption is a known problem with Morea. They even have tech notes on it. Additionally, they have a document file size limit (something like 2 hours), but I can't recall as it's been a while since I spoke with their tech group. Also, they have a known problem with not being able to record video of video. So, if you're testing something like YouTube, Morea will not work — the video gets corrupted. This is a known issue with Morea according to TechSmith.

I transferred my research framework into the marker system and have created my own markers, finding Morae's too general. As for setup, it takes me only the time required to set up the participant machine if I go to a client's site and use their machine. That is usually less than 20 minutes. My biggest problem has been client networks, not the Morae software.

We've never ever seen it setup in less than 20 minutes. Seriously. I'm measuring time from the point I walk in the door and touch the power button on the computer to the point when I've run a test recording and it works okay. The quickest we've ever seen was over an hour. Not to mention if you're taking your own machine, then you've got cables to plug in, boot time for Windows (that alone takes 5-10 minutes), giving Morea machines access to each other, etc. I've never seen this done in under 20 minutes when you have two machines — one for testing, one for viewing.

Just two weeks ago, we were testing at a client's facility, using their equipment w/Morea. They had done some tests the previous week. It still took over an hour to get everything working properly. Much of this was do to Windows, but that's still a factor in the system.

I use a small Web cam for the participant machine if using a client's machine. If I use my own machine, it has a built in Web cam. I am not very fond of using multiple video cameras for testing someone sitting at a computer typing, especially since my policy is to never include a participants face in material given to a client for reasons of privacy.

We only use 1 video camera pointed at the participant.

I also never do testing on Macs.

While it's only 10-20% of our participants, we prefer to provide the most accurate data possible. So, if a participant is a Mac/FF person, it's not going to provide the same experience putting them on Win/FF. We prefer to filter out any bias that could be contributed to OS differences/experience.

Additionally, we don't intentionally rule out Mac participants because our system won't support it. It's our job to have a system that supports the customers of our clients and the systems we test. While I realize other companies don't take the same approach we do, as a researcher, I feel I'm not doing my job if I use a system that doesn't support the customer's native environment. And yes, when possible, we go to people's homes/offices. When not possible, we do our best to simulate the same (similar) environment at our space.

Cheers!

Todd Zaki Warfel
President, Design Researcher
Messagefirst | Designing Information. Beautifully. Contact Info
Voice: (215) 825-7423
Email: todd at messagefirst.com
AIM: twarfel at mac.com
Blog: http://toddwarfel.com
Twitter: zakiwarfel
In theory, theory and practice are the same.
In practice, they are not.

I haven't used Silverback yet - and as a Mac user it would certainly simplify things. I do see a big limitation for remote testing - only Mac users (with the proper setup) can be recruited for testing. That will (in most cases) significantly decrease your pool of participants.

I have used UserVue - participants in that case have to use Windows. I have run remote tests with UserVue on a Mac (booting as a Windows machine) and had no issues (beyond what was already described). It's a nice program - though I am very curious to try Todd's solution. :-) Mora Manager must be run on a Windows machine (for now a Mac booting as a Windows machine does not work).

Carol

Carol J. Smith
Principal Consultant, Midwest Research, LLC
http://www.mw-research.com

Usability Professionals' Association, Director of Global Outreach http://www.usabilityprofessionals.org

Cell: +1 (773) 218-6568
Email: carol at mw-research.com

(Speaking only for myself in this email.)

On Tue, Jul 29, 2008 at 3:45 PM, Michael Dunn mike at foolishstudios.comwrote:

Hey all, I'm doing some research on different tiers of usability testing, and I love the new Silverback app that Clear Left just put out ( http://www.silverbackapp.com/ ) as a potential lower-end solution (we already have partnerships in place for the high-end services). Are there any other apps like Silverback out there? What do you guys use? — Michael Dunn Welcome to the Interaction Design Association (IxDA)! To post to this list . discuss at ixda.org Unsubscribe .... http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help http://www.ixda.org/help

If you're willing to give up the ability to moderate a usability session then for $19 per tester you can outsource the whole thing to www.UserTesting.com (I work there). We've built an online network of pre-screened user testers who can "think out loud" and stay on task without a moderator. So you just post a request for users (who match your target demographic) to do a series of tasks on your site, and in an hour or so you're watching screencasts of them browsing your site and speaking their thoughts.

Posted from the new ixda.org
http://www.ixda.org/discuss?post=31694

Screen Flow http://varasoftware.com/products/screenflow/ might look like a better alternative for just $50 more: build-in viewer-editor, capturing both camera and screen images with ability to arrange them later. Plus Mic/System sound recording%u2026

Posted from the new ixda.org
http://www.ixda.org/discuss?post=31694