17 sep 2017

By Danyo Dimitrov

Solving Big Data Problems with Online Panel Solutions

We can learn a lot from people’s online behavior. Danyo Dimitrov explains how Big Data can be used effectively by bringing together the rational and the emotional.

Obtaining the best insights from respondents has always been a key objective in all research projects. There are many facets to that insights-gaining process, from the research methodology, to interaction with respondents, keeping their attention and hopefully receiving a complete point of view on the subject researched. Through the years, the pursuit of the ever-elusive insights has seen many transformations and even revolutions. With the latest development in technology and online tracking tools, this chase has been taken to a higher level yet again. Combining actively asking and passively listening may be the key to constructing a better and more comprehensive picture of consumer behavior.

The Backstory

Online market research has been around for almost 20 years now. It allows researchers to interview their respondents remotely with all the gains this brings. The online venue ensures that data quality can be monitored and actively measured via a combination of clever tools and approaches. Panels were and still are an essential building block of that construction, as they allow pre-targeting in an easy way and provide much more predictability on how big their potential consumer audience is expected to be. The level of detail increases as more and more clients and agencies move towards using online market research and online panels. Many advancements have been made over the years with gamification, adapting surveys for mobile use, perfecting survey design and question visualization to obtain the best possible level of engagement from the respondents. Making it easy for them means gaining more of those precious insights we are all after.

Greetings from the Subconscious

No matter the technology, however, the principle remains the same – asking people to actively share their opinion and provide as much detail as they possibly can, breaking down their perception of the researched subject to its smallest molecule and trying to make sense of it. After all, consumer behavior is expected to be rational, based on solid arguments of “why I buy this product”, “why I like it” and “how it is different from any other.” We all like to believe we are rational, but if we are honest, we have to admit that a portion of our consumer behavior is emotional. The rational part in us evaluates the pros and cons of certain purchases, but there is always this small somebody sitting on our shoulder, whispering in our ear: “Buy the red one, buy the red one.”
Actively gathering rational opinions gets us very far. It gives us fairly high levels of certainty on what the consumer would do. Still, the more we know about the subconscious little somebody on our shoulder, the more we look for ways to capture that emotional effect in our research findings. That “looking around” got us to think more about the data we gather but do not measure, as it makes no sense at first glance – Big Data.

The New Big Thing

In the first years of Big Data analysis used for market research purposes, there was great excitement about Big Data being the answer to all our questions. Surveying respondents started to look like a thing of the past. After all, people give us so much information with their online behavior alone. They show us what they are interested in by what they look for online, the articles they read, their preferences in the social networks, product mentions and more. There are so many sources to gather information from that actively asking became less attractive. The industry was counting on the fact that emotions were leading consumer behavior, and emotions are harder to extract in a standard research environment. They have to be deciphered from the unguided things we do when no one is watching. Clients began looking at this new solution and diverting their interest towards Big Data analysis in order to finally be able to capture and measure the subconscious drivers of their potential customers. Big Data promises speed and accuracy. One can analyze, slice and reformat the data set as to discover trends and identify connections between seemingly unrelated factors. A number of software platforms came up with diverse solutions for sorting through the endless feed of data and making valuable findings. It appeared that we had the answer to all our questions: the ultimate solution.

Questions Arising

Working more in that field soon displayed certain drawbacks. Clearly with unstructured data there are problems that pile up. The initial excitement began to give way to some important questions:
– How to weed out important points from noise in the data?
– How to identify the most useful data channels to observe and analyze the given research objective?
– Is it possible to gather information without invading personal space and sensitive data of the members of your audience?
– Where are the exact boundaries between personal space and the research-shareable information?
– Why is analysis based on Big Data only effective if done by a specialist with a certain set of technological and analytical skills, when such experts are not abundant?
– How to control costs, when collecting data from multiple sources requires significant storage space, which is bound to grow, sometimes beyond expectations?
– What is the route to putting together a 360-view of the consumer profile, bearing in mind it requires access to a wide range of data for that person, which is close to impossible to obtain and arguably crosses the limits of intrusiveness?The realization of these issues made clients and researchers alike take a step back and contemplate traditional research for solutions again. Which methodology is fast, precise and easily compatible with Big Data? The answer has always been right in front of us – online market research via panels.​

Stronger Together

We know all about online panels. They have been around for quite some time. However, they hold key points in relation to the consumer that, if used wisely, can address the problems Big Data brings. Within the online panel, members are actively asked to share significant amounts of personal information from the very beginning. They are aware of the fact that the panel owner holds their data and has a clear commitment to keeping that data safe from intrusion. Members can decide how much they want to share and whether they’d like to get involved in a given research activity, being aware of how long it lasts and what it will actually monitor. This may seem like quite a straight-forward bunch of panel advantages, but if we go back to the Big Data problems mentioned above, it is clear where the pieces start to fit.
Online Panel Solution

Pieces Fit Together Nicely

The problems and solutions listed in the table are important, but do not give any specifics on how the process actually works. Depending on the needs of the client, the mechanics of bringing together panel surveys and gathering Big Data information vary. As with any other study, it all depends on the topic and objective. Defining the audience for the panel study already brings some answers as to which channels are most appropriate to monitor and which monitoring approach to apply.
The concept is quite simple, but executing it can get complex. In essence, there is a quantitative online survey using an online panel as a sample source, which is connected to a Big Data gathering tool to ensure that questionnaire respondents are also prompted to give the tool access to other information of theirs. The key aspect of the Big Data/panel synergy is to ensure both data collection modules communicate with each other on a technological level and variables are stored properly.
With this “dialogue” secured, the scene is set for either choosing cookie tracking, Facebook or another social media tracker, even installing a type of virtual companion that would measure certain lifestyle features, based on past choices. As long as the survey participant agrees to install the given tool and is made aware of what it actually tracks and for how long, there are no constraints in working with the data afterwards and sharing the findings. Under no circumstances should the respondent’s name, address, email and other personally identifiable information (PII) be stored or shared with third parties. The respondent profile is what matters for the researcher, not the specific person. As long as the respondents are confident that no PII is being shared, they will be confident to allow us to collect the information we need for the analysis.


Real Value Added

This is where all the pieces of the puzzle fit. The active and passive listening work together to solve the mystery of consumer behavior, where Big Data bases findings on past behavior, like search items, likes and dislikes, and online panel surveys generate input by asking about intentions, views and perspectives, oriented towards the future. The synergy between traditional research methodology and Big Data analysis is what creates the real value added. A push to achieve a better understanding of what really goes into the purchase-decision process, from both the rational and emotional point of view.

Back to Blog list