Traditional mechanisms for collecting feedback are well known - user studies, focus groups, usability labs, reading emails from users, visiting customers in person, etc. All of them are important and have their place. Equally important but not as well known are automated feedback gathering mechanisms. This is important since what users *do* is a much better data source than what users *say they do*.

Let's get something clear at the beginning. User privacy concerns trumps over everything else - you need to know what is ok and what isn't and more importantly, that the user knows what is going on and has opted in. If you want to know what is ok and what isn't, I suggest looking at the large companies (Microsoft, Google) and see how they do it. They typically have large user experience and legal teams so anything that comes out is typically kosher.

It is impossible to improve a product if you don't know what the top errors your users hit are. In an ideal world, you would get notified every time the user hits a problem with a debugger automatically attached. Though impossible in the real world, you can come close.

Errors in web applications/services

Errors in web applications are tricky to deal with since there are a wide variety of things that could go wrong. Here's a list of techniques I personally use, mostly due to my work on Popfly.

• Handle the window.onerror event and send the exception back to your servers
• For every custom client side error, send some data back to the server to log it.
• Log every server side error with an unique identifier and display that to the user instead of a generic error number (usually along with a helpful message). This lets you see the exact stack trace/log for the problem. This single technique has helped us tremendously at Popfly.
• Build the right tools on the backend to analyze all the data coming at you. At the very least, any member of the team should be able to get the stack trace and bug id for an error that a user sees.

Watson/Online Crash Analysis for desktop apps

The dialog on the right is a typical crash report dialog. The idea behind it is ridiculously simple - when a crash occurs, send a minidump to a server. From its humble origins in the Office world, Watson is now *the* way to find out what the top crashes are across apps on Windows. Apart from being able to load the dump in a debugger, you can also mine interesting data out of it - e.g, see what the top offending modules or what the worst extensions are. This data isn't available to Microsoft alone. If you're a software maker on Windows, you can get at crash data for your own applications.
If you don't like the Watson UI or if you're not creating Windows apps, you may want to roll your own implementation. There are third party crash reporting software you could look at as well - Google Breakpad is one that I know of. ¹

Usage information

Apart from errors and crashes, you also need to know what users are doing with your applications. What buttons are they pressing, how many files do they open, what do they click on and in what order.  Microsoft uses the 'Customer Experience Improvement Program' to collect this sort of data. This is opt-in with strict rules on what kind of data can be collected. This leads to great data-driven decisions across several products (for example, with the Office Ribbon).

The popular recent usage of such information is in search toolbars. All the major search toolbars ask users to opt-in to having their anonymous browsing history uploaded and then use that data to improve their algorithms and in some cases, show you personalized results.

Analytics

On the web, its a lot easier to find out where your users are and what they're doing with your application since all the data is coming at you over HTTP anyway. Every site should have some form of analytics turned on. I can think of almost no cases where a site owner would not want to know how much traffic the site is getting and the usage patterns that the traffic generates. Analytics are great for

• Telling you traffic patterns - user characteristics, load across time, referers, popular pages, etc
• Showing you what content works and what doesn't
• Showing you how users interact with your site. For e.g, setting up a funnel is a great way to find out where users are running into problems and dropping out.
• Telling you demographic data (with AdCenter Analytics)

I personally have both Google Analytics and AdCenter Analytics turned on in this blog.

A/B Testing

Last, but not least, is A/B Testing. The idea is simple - you run a controlled experiment where you show different sets of users different variants of a feature. You collect data and then determine² which version performed better . The amazing thing about A/B testing is that you can now resolve design decisions by letting real users show you through their actions. This is vastly different from a traditional design decision where one has to try and guess what design will work and what won't.

Amazon, Google and Microsoft all make extensive use of A/B testing (Amazon is probably the heaviest). Ronny Kohavi runs the experimentation team at Microsoft and they have some great content up at http://exp-platform.com/default.aspx. AdWords is the only popular A/B testing platform I could find for general usage - I would love pointers to others.

A/B testing is not for everything and for everyone.

• I'm still not convinced it is possible to do large scale design using A/B testing. Testing variants is one thing - designing complete user flows through A/B testing is something else
• A/B testing and QA don't go well together. Since you're testing multiple versions at the same time, someone needs to make sure that all the multiple versions work well (since they'll be seen by real users). That can put a lot of stress on whoever's testing your apps.
• It is easy to get bad data - either through external factors , by picking bad sample users or by biasing your experiment in some other way.

I strongly recommend reading Kohavi's paper on the topic to get an idea of some of the pitfalls.

Notes

1. My best experience with crash analysis came last week from an unexpected source - the e text editor. It crashed on me and popped up a custom 'do you want to send crash info' dialog. I was pleasantly surprised when I got an email from them the *very next day* that the bug that caused my crash had been fixed with a link to the new setup.

2. The 'determining' part is actually quite tricky. One would think that it is as simple as just seeing the numbers but in reality, one has to account for statistical significance, external factors, whether the experiment was run across enough users, whether the user sampling was random enough, etc.

# Thursday, May 22, 2008 3:17 AM