We Like Our Data Both Big and Small

Consumer behaviour and insights are pivotal for any brand to succeed in today’s hyper-dynamic world. All eyes are on the consumer’s attention.

As much as we might think otherwise, the consumer largely hasn’t changed: underlying motivations and behaviours at large have always remain constant. For example, in the 80s people would kill time on their train commute by reading newspapers, and now people simply use their smartphones instead. Similarly, social validation was always a thing. Earlier on, people would share printed copies of their travel photos, but now we simply hop onto Instagram.

These macro behaviours – the need to be entertained, socially validated, happy, with family – are still the same. However, the micro behaviours – consumption patterns of media, consumer goods, electronics, and so on – have drastically changed. Technology has changed the way we operate as well as the ecosystem in which we live and do business.

So now, more than ever, getting a complete grip on the finer consumer insights is the edge you need to keep your consumers engaged with your brand. After all, people have only 24 hours in a day, and there are more brands vying for their attention than ever before. A friend of mine once asked me, “do you know who Netflix’s biggest competition is?” to which I responded in a blink “Amazon Prime.”. He responded calmly, “Your sleep” (introspective silence at my end).

Big data, powered with small data is your ticket to unlocking the insights you need to make yourself visible to your consumers.

Enter Big Data

With big changes happening in the information infrastructure around us, companies providing market research and information services are changing too. Old methods of collecting information such as surveys are still relevant, but they only cover a part of the insight. Newer methods such as Big Data and Machine Learning are able to take that a step further and provide a lot more information but often lack context of the insight.

To begin, let’s get the terminology right. What is big data? Unstructured, high-volume, and primarily passive behavioural data in this case can be considered big data. Small data, on the other hand, refers to any kind of primary research (claim data) such as quantitative survey (online surveys, face-to-face surveys and so on), qualitative focus groups and even qualitative in-depth interviews.

There are big data companies out there that are good at what they do: analysing big, complex, unstructured databases or even extracting those data sets to analyse correlations or insights within them. But what these companies are unable to figure out is the insight behind the actual behaviour. There are also many research companies out there that are great at what they do, but whose analysis and research is primarily claimed by a set of respondents and not actual behavioural data.

Here’s an example of the discrepancy between a claimed response and actual behavioural data. If you ask me what I usually eat for lunch on weekends, I’d probably say salads. As I hit the gym over the weekend, I prefer eating fresh afterwards. However, if you looked at my Deliveroo receipts, they would show you orders for comfort food that I enjoyed regularly as a child.

What is your analysis of this data? You might conclude that I am one of those people who think they eat healthy but actually don’t. But that’s not the answer. On weekdays, my food orders tend to be of the healthier variety. However, if you look at my behavioural data and ask me causal questions about the comfort food orders, the answer would be that my sister visits me on most weekends, and we relive our nostalgia by ordering food from our childhood and listening to some pop beats of the time.

In essence, big data is a powerful way for you to understand the “What”, “When” and even the “Which” of a given situation, but often lacks the context of “Who”, “Why”, and “How”. This context can be established by primary research solutions.

They say the only thing that is rocket science is rocket science itself. And big data or small data is no rocket science either. Most companies, however, are only leveraging on one part of the whole process, or only using it at different intervals and probably only in a siloed manner. However, insights only come to life when all the five Ws (what, when, why, where, and who) and one H (how) are answered comprehensively.

When you’re working with data, start by asking yourself why you are doing what you’re doing. Make sure you’re clear about the larger objective. Once you’ve established that, you should look into the insights that will get you closer to achieving these objectives. Finally, the most important part is translating those insight areas into data streams that you need to tap into. Once tapped, the data streams will help you establish the when, where and what of things. You can then run the small data initiative to add context to your findings.

Low Hanging Fruits in Acquiring Big Data Information

Look closely and you’ll find that businesses are sitting on a goldmine of information: Big Data.

Things like sales records, loyalty information, financial numbers, production data, and so on, can reveal valuable insights on their own. Couple this with small data practices such as quantitative surveys or focus group discussions, and you have information that is even more telling. When you’re looking to make changes, look at what information you already have to begin with.

In terms of small data, you can always reach out to close clients or loyal customers for initial dipstick interviews. Once you have a better understanding of the results, you can then work with a research agency who will validate your conclusions with a more scientific approach.

Simply put, you can investigate the data patterns around a certain product or service by just looking into information that your company already has collected in some way. You can then use small data methods to understand the reason and context behind the performance of the product or service.

Common Mistakes

Correlations and causations aren’t the same thing, but you’ll find that many brands commonly mistake one for the other. There’s an adage in the data world that says “correlation does not imply causation”. A data pattern is a situation where two variables are working in a direct or indirect proportional manner to one another. This implies a relation, not necessarily a cause and effect association.

A classic example is actually an old fable in the data world that dates back to the 90s, probably before that as well. In it, the sale of beer rose on Friday evenings in Walmart, as did the sale of diapers. Does this mean babies are drinking beer? Probably not!

This is exactly what causation is! Just because two variables – in this case, sales of diapers and sales of beers – are correlated, doesn’t necessarily mean that there’s a cause-and-effect phenomenon at play. When you look closely into big data analysis, you’ll find that such mistakes are actually quite common.

In this particular example, the correlation between diaper and beer sales was that when new fathers were often asked to pick up diapers on their way home from work, they’d also pick up beer to enjoy over the weekend. In this case, there was a correlation but (thankfully) no causation.

When Walmart moved the diaper and beer shelves closer, they apparently saw even better results. This is how big data can help to shape some decisions which impact the bottom line of businesses. But in instances where causations are not so obvious, small data provides the context and answers behind the patterns received from big data.

On the flipside, there are common mistakes that brands do when conducting primary research (small data) too.

Someone smart once said “the plural of anecdote is not data”, but a lot of brands seem to be getting this wrong. Many times, what brands consider data is actually just anecdotal or personal thoughts about their brand. Anecdotes are called anecdotes for a reason! And data isn’t really data unless systematically retrieved and scientifically tested. Before you act upon your insights, make sure you’ve done a thorough statistical validation of your data.

I once consulted for a project around digital optimisation and insights. We were working hand-in-hand with the client on a massive campaign for which they created content across a variety of formats including video, audio, gifs, text, and so on. To my surprise, this content was tested on only 7 people from the same team before it went live!

This was not just anecdotal, as it was only tested on 7 people, it was also biased because these were the employees of the company. For an investment of that size for the campaign, we recommended that they use scientific ways to test and optimise their content before it went live. Predictably, the campaign was much more successful as compared to the previous ones. This is because the content was optimised at each stage using robust methods, without relying purely on anecdotal data. After all, content is king in the digital world.

All of this sounds fascinating in theory, but how can big and small data collaboration actually influence businesses? In this section, we’ll look at three case studies in which big data was used to make significant changes to the way businesses are run.

Target Better, Please

Let’s first look at an example of a modern retail chain that can use a combination of big data and small data to make for very impactful customer segmentation.

The big data source in this case would be store loyalty data. Using this method, stores can easily find information about each customer’s purchase history on the database. You can then run an analysis on this information to understand different basket contents and create associations, and use this to build a range of customer profiles.

However, just understanding purchase behaviours is not enough to segment customers. An added layer of quantitative survey can help you to identify insights such as common purpose of visit, family size, shopping experience, lifestyle, or occupation. This information would add a great deal of richness to the big data that you have.

Put together, the insight you derive will help you to identify solid segments within your customer base. This can help you better target your communications and messaging for each type of consumer segment. From a consumption point of view, a store may use these insights to change the design of the store to drive sales (for example, place the beer next to the diapers) or even create offers and bundles that are more relevant and possibly unique to each customer segment (consumers who buy tonic water, are more likely to pick up a bottle of gin, maybe). You could even plan communication budget efficiently, as you’d have a better understanding of the media consumption habits of your customers among other data points.

Over a decade ago, Target – the US retail giant – developed a big data algorithm that could correctly identify if a customer was pregnant based on their purchase history over a period of time. Only in this case, the pregnant girl was a teenager and wasn’t intending to be contacted. Forbes released this information in a news article titled ‘How Target Figured Out a Teen Girl was Pregnant Before Her Father Did.”. With the lack of context, Target sent out custom promotion material to her doorstep. As you can imagine, this wasn’t the best case scenario for Target or the teenager. This is why understanding nuances beyond algorithmic data is not just important, but necessary.

Kiss Those Theme Park Queues Goodbye

Now let’s look at theme parks, where big data is used to create efficiency and increase footfall to the attractions. An independent research study conducted in Asia suggested that the two biggest reasons adults avoid theme parks are queues for attractions and queues for ticketing or coupon redemption. With data, you have the tools to get rid of these obstacles.

The big data would be collected from wifi receivers or theme park smartphone apps, which collect geolocation data in real time. This data would allow you to track the movement of customers within the park. You could capture valuable information such as traffic movement, dwell time at various attractions or zones, or even the time at which a certain attraction or zone was accessed.

To complement the information received from big data, your small data research could measure visitor demographics, psychographics and their general perception and sentiment towards theme parks.

If you combined these two sets of data, you would be able to elevate the visitor experience and increase footfall to the theme park. Using small data information on demographics and then complementing it with location data can give you insight into the most frequented attractions, which would help you craft stellar targeted ads.

Imagine if your data reveals that young male visitors spend most time at the AI enabled attraction. You’d then use this data point to craft targeted social media ads that use the copy and messaging of the AI enabled attraction!

Tracking traffic movement over time and understanding likes and dislikes of visitors will also help you craft custom maps for the attractions that you can hand to each guest as they enter. Not only would you distribute the traffic, but you'd also leave the guests with a much more delightful experience (because really, nothing kills joy faster than a three-hour wait) which would result in repeat visits. Having a snapshot of traffic distribution in real-time would also help you plan more efficient evacuation strategies in case of an emergency.

Like Hot Chocolate on a Rainy Day

That is, data can help you give your customers what they want, exactly when they want it. Let’s look at how insight can be used to drive conversion for Quick Service Restaurants (QSR).

The big data in this case would be sourced from CRM or loyalty programs, location data from smartphone apps and real-time weather information.

The small data in this case could be surveys around customer profile, preferences, and consumption habits.

Let’s say a customer has installed the app of the QSR on their smartphone, through which they share and receive information. When combined with likes and preferences data, you can better target your overall ads and communication. Take this a step further. When you triangulate weather information in real-time with profile and preference data, you can effectively drive conversions in real-time!

Imagine your customer is sitting in their office on a cold, rainy day. They then receive a push notification through the QSR’s app. The notification reads, “Craving your favourite hot chocolate (through consumption data) on a rainy day (weather data)? Hop to your nearest QSR brand restaurant on 66 Orchard Road (closest based on location data) and enjoy 15% off.”.

When you are able to target your customer this effectively, you’re increasing traffic to the outlet, sales and most importantly, loyalty. You can also optimise your inventory by promoting food or beverages with shorter shelf lives. For example, if a particular outlet has a batch of cheesecakes that are due to expire, you could use location and preferences data to notify your customers of a promotion they simply won’t be able to say no to. A very successful QSR chain in Japan runs a similar version of this use case using weather data to bring about conversions through banner ads.

“Processed data is information (big data). Processed information is knowledge (small data). Processed knowledge is wisdom (synergy of big+small data)” - Ankala Subbarao

So go out there and start playing with data if you aren’t already and when you find the ‘what’, always seek to find the ‘why’ behind it. Data is the new oil indeed. Used correctly, it can power your car, but used incorrectly, it could cost you that entire car.

1. The Magic Between Big and Small Data

In an era where everyone is vying for those ten seconds of a consumer’s attention, the insights derived from a combination of both big data and small data can help you give your customers what they want, when they want it.

2. Know Your Objectives

There’s a lot of information out there, but make sure you’re only using data points that will help you solve a business challenge and achieve your objectives. Make sure you know what you want to improve and work from there.

3. Look at What You Already Have

You can start small. Look at the data you’re already collecting and conduct interviews with your most loyal customers to see what you can learn. From there, you can spring for scientific research conducted by research companies.