Quantitative market research, most commonly executed via online surveys, is a crucial step for any business’s success. Gathering information about customers, competitors, and the industry as a whole is essential to making informed decisions about everything from product development to marketing and communications strategies. One of the most challenging aspects of any market research project lies in ensuring that your final sample reflects the natural demographic fallout of your target audience. In this blog post, we’ll explore the key strategies that we use at Flow Strategy to achieve a representative sample for each of our projects.
Can’t I Just Let the Sample Fall Out Naturally?
You can. But we wouldn’t recommend it. In our experience, even among double opt-in panels, certain demographic groups are naturally more willing or likely to click through to and complete an online survey, while others are less engaged on the panel platform. With a natural fall-out approach, you are sure to find that women aged 35 to 54 are overrepresented in your final sample, whereas 18 to 24 year old men are severely underrepresented. Your sample will also lack racial and ethnic diversity if you let those characteristics fall out naturally.
3 Approaches to Ensure Representativeness of Your Sample
At Flow Strategy, we consider a number of strategies to ensure that the final sample for all of our projects are representative of the target audience.
Weighting the Final Data
The easiest but our least preferred option is to weight the final data after capturing it using a natural fallout approach on demographic characteristics. When you weight data, you apply a multiplier to each respondent’s answers to weight their responses up if their particular demographic group is underrepresented or to weight their responses down if their particular demographic group is overrepresented.
In order to develop and apply a weighting scheme to your final data, you need to know the targets you are trying to achieve. For example, you may know that the target audience for your product or service is distributed across age and gender as follows:
Table representing target audience proportions for an online survey.
Your brand is clearly one that is trying to reach a younger audience, and one that skews more male than it does female.
However, with a natural fallout sample, we tend to see overrepresentation among women, as well as among older age groups. Thus, the sample captured through a natural fallout collection methodology might be distributed as follows:
Table representing the final sample proportions achieve via an online survey without quotas implemented.
In this scenario, the key demographic group of young males is underrepresented in your sample, with older females being highly overrepresented. As such, this sample is not reflective of your core audience.
After fielding your survey, the simplest way to bring the data in line with the core audience is to apply a weighting scheme to your final data. We leverage either cell weighting and random iterative method (RIM) weighting when weighting is required.
In cell weighting, weights are calculated on a cell-by-cell basis, so that the sample distribution is brought in line with the target distribution. The weight is computed by simply dividing the target distribution for each cell by the sample distribution for each cell:
Table representing cell weights to bring the final sample back in line with demographic targets.
These weights are then applied to the total sample in order to place greater emphasis on the demographic groups that were underrepresented in the sample and less emphasis on the demographic groups that were overrepresented in the sample. Put simply, the data for one male respondent aged 18-34 will now be weighted so that it reflects 1.77 male respondents aged 18-34 whereas the data for one female respondent aged 55+ will be weighted to reflect only 0.58 female respondents.
RIM weighting, also known as rake weighting, is a more flexible alternative to cell weighting. Instead of adjusting each demographic cell individually, RIM weighting adjusts groups of variables iteratively across multiple dimensions — such as age, gender, race, and ethnicity — until the sample aligns with the target distribution of all variables. This approach allows for greater flexibility, particularly when balancing multiple demographic variables simultaneously, and is useful when you know the target proportions for each individual demographic characteristics, but not how those demographic characteritics nest within one another.
Weighting can be a reasonable solution to bring your sample in line with targets. However, there are some caveats we like to keep in mind when applying weights to market research data:
You need to know the targets you are trying to achieve. If you are entering market research somewhat blind, you might not know who represents your target audience.
Weighting data does reduce the accuracy of your results. Sampling variance, standard deviation, and standard error all increase when weights are applied.
The risks of weighting data are compounded when your sample size is smaller. With very large sample sizes, the impact to variance, standard deviation, and standard error is quite small. However, with small sample sizes, the impact may be considerable. Use caution when weighting small sample sizes.
The more extreme the weight, the more extreme the impact. A good rule of thumb is to avoid applying a weight of less than 0.50 per respondent, as well as a weight of greater than 2.0 per respondent.
Weighting data up carries more risk than weighting data down. When you are weighting data up, sampling error is exaggerated. This is particularly dangerous if your survey includes variables that are prone to potential outliers, such as open ended numeric questions (e.g., how much did you spend on this category in the past year, how many times in the past year have you engaged in this behavior). If you apply a weight of 1.77 to a sub-group of males aged 18-34, as in the example above, each outlier in that sub-group is multiplied by 1.77. The greater the weight, the more each individual response is exaggerated.
Because of the above risks, we at Flow Strategy avoid weighting as much as we are able. The two main scenarios in which we weight sample data are:
When achieving a representative sample is not an option at the outset
In scenarios where feasibility is low and cost per interview (CPI) is high, weighting might be the best and only solution to achieving a sample that is representative of your audience, because setting quotas may not be possible, and you may just have to take what you can get when it comes to the sample.
When you have had to loosen quotas in order to come out of field
There are times when you are able to set quotas ahead of time, but because of low incidence, a need for shortened time in field, or other factors, you must loosen those quotas in order to achieve the desired total sample size within a reasonable time frame. In this scenario, weighting is perfectly reasonable in order to bring your sample in line with targets. Because you employed quotas from the outset and loosened them just to finish up fielding, chances are your quotas will be fairly minimal (i.e., not approaching the 0.5 or 2.0 limits we recommend), which means the negative risks are minimized as well.
Set Quotas
To set quotas within your survey, as with weighting, you need to know the targets you are trying to achieve for each demographic cell. Quotas can be independent of one another, or they can be nested.
Independent quotas, true to their name, treat each segmenting variable independently. For example, you might set a quota for gender (60% male / 40% female), a quota for age (39% age 18-34, 33% age 35-54, and 28% age 55+), and a quota for ethnicity (20% Hispanic; 80% non-Hispanic). Because these quotas are independent from one another, you may end up with a scenario, particularly at the end of fielding, where you are trying to capture multiple hard-to-reach quotas all at once (e.g., 18-24 year old Hispanic males).
At Flow Strategy, we prefer, when possible, to set nested quotas. When quotas are nested, they interact with one another, so you are recruiting respondents per each target cell. The above quotas nested might look something like this:
Table representing nested quotas by age, gender, and Hispanic ethnicity when nested demographic targets are known.
When possible to execute, employing nested quotas is the ideal approach to ensure your sample is representative of your core audience.
However, using quotas, especially nested quotas, does present its challenges.
You need to know the targets you are trying to achieve. As with weighting, if you are entering market research somewhat blind, you might not know who represents your target audience.
Even with set quotas and double opt-in panels, there are certain audiences that are harder to reach. Once you get close to the end of fielding, you may find yourself needing to loosen certain quotas — especially those for hard-to-reach audiences.
Our preference is to employ strict nested quotas. This allows us to capture data that is most representative of the target audience.
Balanced Starts
The two approaches we’ve discussed, weighting and set quotas, require that you know the targets you are trying to achieve for each demographic cell. However, for brands with more nascent research initiatives, that might not be possible. You might not know yet the demographic fallout of your core audience.
This is where balanced starts come in. Balanced starts, also known as click balancing, allow you to make your decisions based on the distribution of respondents who are starting the survey, not the distribution of respondents who finish the survey.
The outcome of a successfully executed survey with balanced starts should give you the appropriate breakdown of demographic cells for your target audience.
To put it simply, balanced starts on gender means that 49% of the clicks into the survey should be male and 51% should be female, based on Census data. If 60% of the respondents who qualify for and complete the survey are male and 40% are female, that tells us that the target audience for the product or service in question skews male.
When you don’t know the appropriate quotas to set for your project, balanced starts may seem like the best option. This approach will get you to a clear definition of your target audience.
That said, executing a study on balanced starts, especially through the full life cycle of the project, is incredibly difficult, and can be cost prohibitive. At Flow Strategy, we do not manage or deploy our own panel, but we are well aware of the challenges of managing a project that balances clicks into the survey. It’s not just about making sure the survey invites are balanced to Census — it’s balancing survey invites in combination with certain sub-groups’ higher or lower propensity to click in to take a survey.
Because of this, most panel partners either fall down on, charge exorbitantly for, or refuse to participate in projects involving click balancing. Furthermore, click balancing can only be executed on known variables in your panel partner’s database, such as age, gender, race, and ethnicity, among other high-level demographic variables. If your project requires quotas or balancing on other characteristics, such as behaviors or consumer segments, click balancing will not be possible because those are not known characteristics in the panel partner’s database.
While we believe click balancing is the most accurate approach to ensuring sample representativeness, especially when you don’t know very much about your core audience going into the project, it is rarely, if ever, a reasonable approach.
Approaches and Trade-Offs
We’ve discussed three approaches to sample representativeness, each of which has its benefits and drawbacks:
Table summarizing three approaches to ensuring sample representation.
Flow Strategy’s Approach
One of the most common challenges our clients face is that they do not have a clear understanding of their demographic targets. This makes setting quotas or developing a weighting scheme difficult from the outset. While balanced starts would be an ideal solution in this case, many panel partners either will not execute a balanced starts methodology, or they claim to be able to do so but struggle to deliver efficiently.
To overcome these challenges, we take a more strategic approach by leveraging one of the following methods:
Nationally Representative Omnibus Survey
We often kick off a project with a nationally representative Omnibus survey consisting of 5-10 screening questions designed to identify the right audience for our lengthier, more in-depth survey. This ensures the data is balanced from the start, allowing us to use the results to establish the appropriate quotas for the subsequent main survey.Balanced Starts Followed by Quota-Based Sampling
Alternatively, we may use balanced starts for a portion of the sample. This approach helps us understand the natural demographic breakdowns in the early stages of fielding. Once we have gathered enough data from this balanced starts portion, we can identify the necessary quotas and shift to a more traditional quota-based sampling method. This hybrid approach enables us to combine the precision of balanced starts with the practicality and feasibility of quota-based sampling.
By implementing one of these methods, we ensure that our surveys are both cost-effective and representative of your target audience, even when initial demographic targets are unknown.