I have been collecting, cleaning, analyzing, and reporting data since the 1990s. That makes me old but it also means I was working with metrics before website analytics came along, with the deluge of data it produces. It was a simpler time and because of that, there was more thought to the cost of data collection and processing which, in turn, helped prioritize the data that mattered most. I learned about analyzing data from a Harvard economics professor, working in the Office of Naval Operations, and from senior consulting partners and executives. It gave me broad and deep experience across domains early in my career.
Too Much Data, Not Enough Insight
Today, if you are just starting to wade into the world of metrics, it gets overwhelming in a hurry. Between the volume of data and the complexity of measurement tools, it can feel opaque and mysterious. Ironically, that complexity is exactly what gets us in trouble when we try to present it to others.
Some of the barriers to good reporting today include:
- The copious amounts fo data that make it hard to identify what matters.
- Uneven data and reporting literacy creating a lot of poor reporting and understanding.
- Metrics defined my engineers not business analysts, which emphasizes counting objects or transactions vs. showing behavior, experience, or impact.
- Cultures of toxic positivity that discourage reporting inconvenient truths.
- Managers who use metrics as weapon to judge performance rather than as a learning tool.
Data is everywhere, over-hyped, and often misused. Business culture mentions data-driven decisions a lot but with little sophistication regarding the limits of what data can do. Data is input to decisions but it should not be the sole variable influencing decisions; to do so ignores business strategy, ethics, team preferences, and harder to measure influences. Critically, it can produce false positives; an activity that suggests value because of its quantity but is detrimental to long-term progress. The best example is disinformation or hyperbolic news headlines; in the short term it triggers action but over the long-term, it erodes trust. All too often, we ignore those things that cannot be measured and assume they are irrelevant as a result. This leads to the ridiculous need for research to tell us that spending time in nature is good for us or that hugging our children is good for them. That research confirms only that we have learned not to trust our own intuition or that others do not believe what should be obvious.
For all of these reasons, the state of metrics in business is decidedly poor. In many respects, we would often be better off doing what we think is right instead of using data poorly.
Good Data: Keep it Simple
One of the biggest issues I see in data analysis and reporting is overcomplication. The most common way this shows up is either by using and presenting too much data or by creating complex metrics that are hard to communicate and from which it is even harder to draw conclusions. My personal pet peeve is ‘index’ metrics, which often have 5 or more arguments and produce a number like 1.67 suggesting an irrelevant level of accuracy for a metric that takes significant effort to understand and interpret. If people cannot understand what actions change a metric, it’s not a great metric.
An incomplete but understood metric is far better than a more accurate but misunderstood metric.Rachel Happe
Far too often, people doubt themselves and assume that they are just not experienced enough to understand data – and because of that, say nothing and accept confusing metrics. Great metrics are clear and easy to understand. If you are confused, it is not you – it is the metric.
Good Reporting Practices
Good data reporting is not helped by most analytics platforms, largely because there are so many options, bells, and whistles. Most people, most of the time, need very little of those extras. Less is more. It costs money in the form of your time to create reports – make sure the time you spend is worth it.
The first step in reporting metrics is to create templates for regular reports – and determining what is necessary given the audience for each and the decisions to be made from them. Refine until you have only what is needed – any extra data will distract people from the purpose of the report. Next, consider what ‘data errands’ you will do and for who – can anyone walk into your office and ask for data? At a minimum, having a set of questions and requirements for them so that you can determine what they need and whether the exercise is worth it is critical. Along with that, define what you will not do – and how to tell people no; having guidelines beforehand makes that a lot easier. Without guidelines, data errands can eat up considerable time that keeps you from more strategic work and sets the expectation that you will generate data on a whim of theirs even if there is no return for that investment of time.
Below are some of the practices I find most valuable in making sense of data – both for myself and others. Some of these, like showing trends and normalizing data, are the difference between good and great analysis. Reporting raw numbers is generally best used for describing the demographic scope of data. Reporting one raw datapoint in a vacuum reveals very little. Is 2,456 good? Is it bad? Is it better or worse than before? Trends and segmentation provide critical context for figuring those questions out.
Other practices are less intuitive, largely because of poor reporting habits. Because we too often assume that data are decisions, questions are often misunderstood as criticisms or judgments. The reality is that only great reports generate questions – it demonstrates that the data revealed new. No report can provide boundless answers; good reports will surface descriptive data that isn’t that interesting on its own and provides statements of fact that can’t be argued with but also don’t reveal much.
GREAT reports provide context that allows us to see further into an issue or opportunity, propelling us down new roads of inquiry.Rachel Happe
These ten practices can save you a lot of time and grief.
Remember, you know more than you think you do, data is an opportunity for a great conversation, and every analysis has limitations. The ambiguity of analysis is a feature, not a bug.