Speaker Details: Jen Matson

Measuring the Wrong Thing: Data-Driven Design Pitfalls

We’ve come a long way from Douglas Bowman’s infamous Google lament about having to test 41 shades of blue. Today, using data to inform and evolve designs has become the standard at large companies. And sophisticated web analytics and A/B testing tools are now available to more of us than ever before. But in our eagerness to leverage the power of quantitative data, could we possibly be measuring the wrong things? And if so, would we even know it? I’ll examine a few common pitfalls when trying to gather and use data for product design, including: Throwing Stuff Against the Wall - When you can test any idea, how do you prevent bad ones from being unleashed? The Meaning of A Click - Defining the right metrics based on the questions you seek to answer, not just what can be measured easily. Unclear Cause and Effect - Trying to determine the root cause of a change, when its connection to what you are measuring may be indirect. From working on projects where I’ve encountered each such pitfall – and sometimes more than one at a time – I’ve been able to increase my influence in the planning stage, where measurement decisions are often made, as well as improve my designs. So now when business stakeholders say “”Ship it!“” I can be confident that we are all measuring the same thing, and interpreting that data in a meaningful way.

Jen Matson

Jen Matson

Jen Matson is a Sr. UX Designer at Amazon who has been designing, variously, for web, kiosks and mobile since 1994. She has worked in a variety of industries, from entertainment to technology, with clients including Comedy Central, LEGO, Microsoft and T-Mobile.