Measuring the Wrong Thing: Data-Driven Design Pitfalls
We’ve come a long way from Douglas Bowman’s infamous Google lament about having to test 41 shades of blue. Today, using data to inform and evolve designs has become the standard at large companies. And sophisticated web analytics and A/B testing tools are now available to more of us than ever before. But in our eagerness to leverage the power of quantitative data, could we possibly be measuring the wrong things? And if so, would we even know it? I’ll examine a few common pitfalls when trying to gather and use data for product design, including: Throwing Stuff Against the Wall - When you can test any idea, how do you prevent bad ones from being unleashed? The Meaning of A Click - Defining the right metrics based on the questions you seek to answer, not just what can be measured easily. Unclear Cause and Effect - Trying to determine the root cause of a change, when its connection to what you are measuring may be indirect. From working on projects where I’ve encountered each such pitfall – and sometimes more than one at a time – I’ve been able to increase my influence in the planning stage, where measurement decisions are often made, as well as improve my designs. So now when business stakeholders say “”Ship it!“” I can be confident that we are all measuring the same thing, and interpreting that data in a meaningful way.