Working Towards Cleaner Data

Picture this: A data analyst goes to the COO with a recommendation to shut down a particular logistics trade lane because it isn’t profitable. Only, they later discover that the data wasn’t reliable and they’ve actually shut down a profitable lane. This would be disastrous – not only for the business, but also the analyst who proposed the idea. 

To prevent such missteps, it’s important to make sure that the data you’re supplying your Advanced Analytics or Data Science teams with is as reliable as possible. How? By adopting a problem-solving approach to understand your data sources or how data is generated or created. When a majority of the effort goes into setting up processes to ensure that the data you capture is reliable, analysing it then becomes the easiest part. 

Common Misconceptions

If you’re an analyst, you expect your data to be as clean as possible. The goal is to be able to use this data to identify patterns that will drive business outcomes. But what happens when you find patterns of issues in data collection that render it unreliable? You’ll likely need to invest the majority of your efforts into cleaning up the data, which isn’t efficient. 

When approaching data, don’t assume that the data creation process – for instance, someone manually logging the information –  followed the guidelines or SOP’s every single time. More often than not, people and even systems don’t function as they should. Acknowledging this will help you predict some of the issues that may arise in the data creation process and solve for them preemptively. 

Secondly, don’t just look for anomalies in data. In most cases, anomalies in data occur when things are going wrong. Yet sometimes, not having anomalies in data should also be a cause for concern. If you’ve launched a marketing activity, for instance, there should be a spike in certain metrics. If you come in with the assumption that no anomalies equate to smooth sailing, you’ll have failed to spot a critical issue that needed to be addressed. This points to the issues with your data capturing processes.