We’ve said it here before, and we’ll say it here again:
There’s hundreds of ways a survey project can go totally wrong, and only one way it can go totally right.
Surveys are delicate projects. You have one chance to pulse your target audience, and any mistakes in your survey logic can make your entire dataset useless.
If you’re paying big bucks for this data, it’s even more important that you get everything 100% right the first time. There’s no going back to the same people with the same survey without paying the same fee all over again.
That said, here are some rules of thumb for what NOT to do when designing your market research survey.
1. Don’t be obvious.
We’ve talked about this before. It’s so important that only the right people get into your survey. And at the end of the day, much of that falls on you.
Imagine you’re trying to survey Dalmation (the dog breed) puppy owners. The first-instinct screener question might look something like this:
Do you have a Dalmation puppy? (Yes/No)
Straightforward enough, right?
Well, not really. This might work if every single respondent were honest. But that’s just not always true. The fact is, this screener question makes the “correct” answer obvious — dishonest respondents who just want to be compensated for completing your survey will know what to select (Yes) in order to proceed.
Because why would a screener be so specific? Who would survey people who don’t own Dalmation puppies? What could that kind of survey possibly be about?
So instead, ask a series of questions that don’t give away the right answers.
Here’s an example we used in a recent survey project:
Which of the following do you own?
1. A car
2. A Keurig coffeemaker
3. A Dell laptop
4. A dog
5. A cat
6. A vacation home
7. A parrot
8. None of these
If they select “Dog,” follow up with:
How old is your youngest dog?
1. Under 2 years
2. 2-4 years
3. 5-7 years
4. Over 7 years
Now we know who owns a puppy. It’s time to ask this:
What breed is your youngest dog?
1. Golden Retriever
2. Dalmation
3. Black Lab
4. Great Dane
5. Cocker Spaniel
6. Chihuahua
7. None of these
This approach doesn’t give away the “right” answer, and it has three points at which someone might disqualify. A much more robust approach than one simple question.
Another quality control trigger would be to disqualify respondents who select every item in the first question. While it’s possible that someone does own all of these items, it’s extremely unlikely — that’s just common sense. Better to just disqualify these respondents from your survey than risk corrupting your dataset with dishonest respondents.
Ideally, we wouldn’t have to worry about this. And at PeopleFish, we take strong measures to enhance the quality of every client’s dataset. But as consumer surveys become more ubiquitous, less-than-honest respondents find creative ways to complete surveys they’re not qualified to take. Robust screener questions (like the one above) combat this trend, and if written well, can almost entirely eliminate the risk of fraudulent responses.
2. Don’t be confusing.
This is one easy — just ask direct, simple questions.
Market research survey questions should ideally be one sentence long. And they should require almost no thought to answer honestly.
The trick here is to use branching and logic appropriately. Rather than asking questions like:
If you answered “Yes” to the previous question, please tell us more about your experience in the box below. If you answered “No,” then skip this question.
…use branching and logic so that no explainer is needed. Simply route respondents to the appropriate place in your survey, based on how they answered prior questions.
Another trick here is to separate one complicated question into multiple simple Yes or No questions. For example:
Do you own a car manufactured between 2012-2015 that has not yet had to have the tires replaced? (Yes/No)
…should become…
1. Do you own a car manufactured between 2012-2015? (Yes/No)
2. Has your car ever had its tires replaced? (Yes/No/I don’t know)
You get the same insights by splitting the question into two, with the added benefit of boosting your response rate and lowering overall survey fatigue by making the two questions almost effortless to answer — no serious thinking required!
3. Don’t be long-winded.
Market research surveys shouldn’t require more than 8 minutes to complete. 12 minutes at the absolute longest (before cost starts to increase exponentially).
The best surveys we’ve seen take 3-4 minutes, tops.
You can fit a lot of questions into 8 minutes. Consumers can answer questions about their age, gender, income, education, and region all in about 20 seconds if you ask the questions appropriately.
One tip here is to use multimedia as much as possible. Rather than present a concept to consumers with a long paragraph, make a 15-second video that accomplishes the same goal. Or create an image or wireframe that requires just a few seconds to understand and digest.
Less text, in general. Follow that rule of thumb.
4. Don’t be sneaky.
Of course, no survey designer wants to be sneaky. What’s the point in manipulating your survey results? It’s your loss, at the end of the day.
But we often come across survey questions asked in a way that force respondents to choose items they don’t actually want to choose. In the market research world, it’s called “double-barreling,” and it goes something like this:
Is this app helpful and fun? Yes/No
See the problem? What if respondents think the app is helpful, but not fun? Or fun, but not helpful?
The result here is bad data. You aren’t able to tell whether someone who thought it was helpful but not fun (or vice versa) selected Yes or No to express their honest opinion.
So avoid double-barreled question. Or in other words, don’t be sneaky!
8. Don’t be cheap.
You’ve designed your product, and a killer survey to go with it.
Don’t skimp out on collecting feedback from real consumers in your target market.
At PeopleFish, we work with clients every day who tried and failed to gather survey results from their internal network — friends, family, coworkers.
The problem is, at the end of the day, no one’s network is all that large. Even if you have 500 good friends, only a small fraction of your friends are actually going to sit down and take your market research survey — you’ll be lucky if 50 out of those 500 people take it from beginning to end.
For the rest of us who don’t have 500 good friends (more like 10-15), we’re really stuck! Sure, you can try posting your survey to relevant forums and blogs, but unless you’re offering a hefty incentive, don’t expect much movement.
That said, paying for responses is worth it. That’s why we do what we do. And it’s not expensive — $400 can get you a strong dataset of general consumers. That should be enough to pique the interest of any potential partner or investor, assuming your survey was well-designed and the data came back favorable.
So take your product idea seriously and determine your budget for a market research survey project. Your future self will thank you.
