Go to homepage
Get a Demo
Get a Demo


We Like Our Data Both Big and Small

Sep 11, 2019 | 13m

Gain Actionable Insights Into:

  • The goldmine of information you may already have at your fingertips
  • Why big data on its own isn’t enough to make sense of patterns
  • Real world business applications of big and small data, combined


It’s Not Rocket Science

Consumer behaviour and insights are pivotal for any brand to succeed in today’s hyper-dynamic world. All eyes are on the consumer’s attention.

As much as we might think otherwise, the consumer largely hasn’t changed: underlying motivations and behaviours at large have always remain constant. For example, in the 80s people would kill time on their train commute by reading newspapers, and now people simply use their smartphones instead. Similarly, social validation was always a thing. Earlier on, people would share printed copies of their travel photos, but now we simply hop onto Instagram.

These macro behaviours – the need to be entertained, socially validated, happy, with family – are still the same. However, the micro behaviours – consumption patterns of media, consumer goods, electronics, and so on – have drastically changed. Technology has changed the way we operate as well as the ecosystem in which we live and do business.

So now, more than ever, getting a complete grip on the finer consumer insights is the edge you need to keep your consumers engaged with your brand. After all, people have only 24 hours in a day, and there are more brands vying for their attention than ever before. A friend of mine once asked me, “do you know who Netflix’s biggest competition is?” to which I responded in a blink “Amazon Prime.”. He responded calmly, “Your sleep” (introspective silence at my end).

Big data, powered with small data is your ticket to unlocking the insights you need to make yourself visible to your consumers.

Enter Big Data

With big changes happening in the information infrastructure around us, companies providing market research and information services are changing too. Old methods of collecting information such as surveys are still relevant, but they only cover a part of the insight. Newer methods such as Big Data and Machine Learning are able to take that a step further and provide a lot more information but often lack context of the insight.

To begin, let’s get the terminology right. What is big data? Unstructured, high-volume, and primarily passive behavioural data in this case can be considered big data. Small data, on the other hand, refers to any kind of primary research (claim data) such as quantitative survey (online surveys, face-to-face surveys and so on), qualitative focus groups and even qualitative in-depth interviews.

There are big data companies out there that are good at what they do: analysing big, complex, unstructured databases or even extracting those data sets to analyse correlations or insights within them. But what these companies are unable to figure out is the insight behind the actual behaviour. There are also many research companies out there that are great at what they do, but whose analysis and research is primarily claimed by a set of respondents and not actual behavioural data.

Here’s an example of the discrepancy between a claimed response and actual behavioural data. If you ask me what I usually eat for lunch on weekends, I’d probably say salads. As I hit the gym over the weekend, I prefer eating fresh afterwards. However, if you looked at my Deliveroo receipts, they would show you orders for comfort food that I enjoyed regularly as a child.

What is your analysis of this data? You might conclude that I am one of those people who think they eat healthy but actually don’t. But that’s not the answer. On weekdays, my food orders tend to be of the healthier variety. However, if you look at my behavioural data and ask me causal questions about the comfort food orders, the answer would be that my sister visits me on most weekends, and we relive our nostalgia by ordering food from our childhood and listening to some pop beats of the time.

In essence, big data is a powerful way for you to understand the “What”, “When” and even the “Which” of a given situation, but often lacks the context of “Who”, “Why”, and “How”. This context can be established by primary research solutions.

They say the only thing that is rocket science is rocket science itself. And big data or small data is no rocket science either. Most companies, however, are only leveraging on one part of the whole process, or only using it at different intervals and probably only in a siloed manner. However, insights only come to life when all the five Ws (what, when, why, where, and who) and one H (how) are answered comprehensively.

When you’re working with data, start by asking yourself why you are doing what you’re doing. Make sure you’re clear about the larger objective. Once you’ve established that, you should look into the insights that will get you closer to achieving these objectives. Finally, the most important part is translating those insight areas into data streams that you need to tap into. Once tapped, the data streams will help you establish the when, where and what of things. You can then run the small data initiative to add context to your findings.

Low Hanging Fruits in Acquiring Big Data Information

Look closely and you’ll find that businesses are sitting on a goldmine of information: Big Data.

Things like sales records, loyalty information, financial numbers, production data, and so on, can reveal valuable insights on their own. Couple this with small data practices such as quantitative surveys or focus group discussions, and you have information that is even more telling. When you’re looking to make changes, look at what information you already have to begin with.

In terms of small data, you can always reach out to close clients or loyal customers for initial dipstick interviews. Once you have a better understanding of the results, you can then work with a research agency who will validate your conclusions with a more scientific approach.

Simply put, you can investigate the data patterns around a certain product or service by just looking into information that your company already has collected in some way. You can then use small data methods to understand the reason and context behind the performance of the product or service.

Common Mistakes

Correlations and causations aren’t the same thing, but you’ll find that many brands commonly mistake one for the other. There’s an adage in the data world that says “correlation does not imply causation”. A data pattern is a situation where two variables are working in a direct or indirect proportional manner to one another. This implies a relation, not necessarily a cause and effect association.

A classic example is actually an old fable in the data world that dates back to the 90s, probably before that as well. In it, the sale of beer rose on Friday evenings in Walmart, as did the sale of diapers. Does this mean babies are drinking beer? Probably not!

This is exactly what causation is! Just because two variables – in this case, sales of diapers and sales of beers – are correlated, doesn’t necessarily mean that there’s a cause-and-effect phenomenon at play. When you look closely into big data analysis, you’ll find that such mistakes are actually quite common.

In this particular example, the correlation between diaper and beer sales was that when new fathers were often asked to pick up diapers on their way home from work, they’d also pick up beer to enjoy over the weekend. In this case, there was a correlation but (thankfully) no causation.

When Walmart moved the diaper and beer shelves closer, they apparently saw even better results. This is how big data can help to shape some decisions which impact the bottom line of businesses. But in instances where causations are not so obvious, small data provides the context and answers behind the patterns received from big data.

On the flipside, there are common mistakes that brands do when conducting primary research (small data) too.

Someone smart once said “the plural of anecdote is not data”, but a lot of brands seem to be getting this wrong. Many times, what brands consider data is actually just anecdotal or personal thoughts about their brand. Anecdotes are called anecdotes for a reason! And data isn’t really data unless systematically retrieved and scientifically tested. Before you act upon your insights, make sure you’ve done a thorough statistical validation of your data.

I once consulted for a project around digital optimisation and insights. We were working hand-in-hand with the client on a massive campaign for which they created content across a variety of formats including video, audio, gifs, text, and so on. To my surprise, this content was tested on only 7 people from the same team before it went live!

This was not just anecdotal, as it was only tested on 7 people, it was also biased because these were the employees of the company. For an investment of that size for the campaign, we recommended that they use scientific ways to test and optimise their content before it went live. Predictably, the campaign was much more successful as compared to the previous ones. This is because the content was optimised at each stage using robust methods, without relying purely on anecdotal data. After all, content is king in the digital world.

Want to continue your read?

To view the full content, sign up for a free account and unlock 3 free podcasts, power reads or videos every month.


Siddharth Sankhe

Director | Former Global Digital Commercialization Leader

SAP Qualtrics | Nielsen



Big Data, Demystified