The Problem with DIY Survey Projects

This is tough to admit.

We love surveys. We’re all about surveys. We live, eat and breathe surveys.

And we think you should love surveys, too.

But the fact is, survey projects are extremely delicate. They’re easy to mess up. One little mistake can make your entire dataset completely useless. And, unfortunately, it’s more than likely your DIY survey project is going to be a total waste of time.

So there you have it. Industry secret.

But does this mean market research surveys aren’t worth doing?

Nope. In fact, what’s funny about this is most of our clients are about 95% of the way to a good survey by the time they find us.

What does that mean? It means they know what questions they want to ask. It means they know how they want to ask them. It means they know who they want to ask and what they want their results to look like.

So why just 95%? Why not 100%?

Because surveys are about precision. The language in your questions must be absolutely perfect (not a shred of ambiguity) and your targeting must be perfect (not just close).

This is where professionals can help. That last-mile tune-up is what turns a decent survey into a great survey.

At PeopleFish, every one of our clients’ surveys is programmed by a survey research analyst. Word for word, our analysts migrate our clients’ survey instruments—typically a Word document or email text — into a field-ready survey instrument. As they do this, they review every question and answer option to make sure nothing is left to chance—no ambiguous language or unnecessarily confusing phrases. No “double-barreling” or mismatched question types.

In fact, we’ve codified this practice into a guiding rule: WE DON’T FIELD SURVEYS FOR CLIENTS THAT WE WOULDN’T FIELD FOR OURSELVES.

This means you can rest assured that your PeopleFish survey results are real-world accurate and ready for action. No wondering if you asked the questions properly or if respondents were perhaps confused by how you phrased something.

We apply this same level of analytical rigor to survey fielding strategies. Fielding surveys to consumers is, of course, our primary function. It’s what we do best. That said, we carefully review every client’s fielding strategy to ensure we’re reaching the right people with our surveys.

For example, say you’ve designed a new toy for toddlers and preschoolers. You come to PeopleFish with a request to target your survey to “parents of young children” to gather their opinions about your product concept. It’s they, after all, who will be buying this product, right?

Well, it’s not that simple. Consider that, based on secondary consumer research, mothers are five times more likely than fathers to buy toys for their children. And more than half of a child’s toys are given to him or her as a gift from other family members.

With that in mind, who you really ought to be targeting is anyone who regularly buys toys for children—not necessarily just “parents of young children.”

The bottom line is that this becomes confusing. But we can account for all of this nuance when we field your survey. It’s what we pride ourselves in doing well — adding value to our clients’ survey projects by doing everything 100%, absolutely, entirely correct.

Got a product idea? Test the waters at people.fish. We help startups and innovators survey their target market the right way, the first time. No more wasted time and money with confusing, DIY survey tools.

SurveyMonkey is Great, But Not for Startups

SurveyMonkey is an all-around great tool.

It’s tried and tested. It’s been around since 1999—one of the earliest survey platforms on the internet. More than 25 million people use SurveyMonkey, and the company recently went public, further enhancing the stability and predictability of the tool—especially for users who use SurveyMonkey as an integrated part of a larger workflow.

But all of that said, SurveyMonkey is not a good survey tool for startups.

What SurveyMonkey Does Well

As mentioned above, SurveyMonkey is not only a powerful tool, but a stable and predictable one—ideal for integrating surveys into larger project management workflows and CRM systems. This is what makes SurveyMonkey the best among its competitors, which otherwise are almost identical tools.

SurveyMonkey also has arguably the most question types and data reporting formats of any other broadly-accessible tool. In our experience, 99% of all survey research needs are covered by the options and formats SurveyMonkey offers. From more basic functions like multiple-choice questions and nested dropdowns to more advanced ones like conditional logic and SPSS variable naming, there’s little one won’t find in the suite of tools SurveyMonkey offers.

Finally, SurveyMonkey’s offers an intuitive way to add “team members” to your projects, whether they are inside or outside your organization. Perfect for collaboration on advanced surveys, or if you’re hiring an outside specialist to work on one of your survey projects, but with whom you’d rather not share your login information.

What SurveyMonkey Does Poorly

It’s simple: Pricing.

SurveyMonkey has decent rates, but not if you’re launching just one, or two, or three survey projects. The fact is, their basic license (Advantage: $384) won’t get be sufficient if your survey has any mildly-complex logic branching. You’ll need a higher license level (Standard: $444).

That’s prohibitively expensive for a startup with plans to run only a handful of surveys. With some panel providers (like PeopleFish), it’s the cost of more than 200 consumer survey responses!

Additionally, SurveyMonkey isn’t easy to use. Yes, it’s about intuitive as it can be, but to get the survey you have in mind into a usable, functional, programmed instrument takes a lot of time and experience, no matter what tool you’re using. It’s one of the most frustrating parts of any survey research project.

Here at PeopleFish, almost one-fourth of our clients have tried and failed to program and field their own survey using a self-serve tool like SurveyMonkey. The fact is, it’s just not easy to get all the survey logic and branching right, and you’ll often find that what you want to accomplish is hidden behind yet another paywall only after you’ve got your survey 90% programmed.

Talk about a frustrating waste of time and energy. And worse, whatever work you’ve put in up until you discovered that your needed logic or branching feature is going to cost another $300 isn’t transferable into another platform. Once you’ve started in SurveyMonkey, you’re stuck—you’ll have to start programming all over again if you decide to use a different tool.

What PeopleFish Does Better

With PeopleFish, you don’t need to buy a subscription. You pay only for the responses you keep.

In other words, the $444 you’d spend on a SurveyMonkey subscription can instead by used to pay for, say, 200 survey responses from your target market consumers.

On top of that, you don’t have to program your survey. We do that for you.

It’s easy to just write down what you want your survey to do. It’s harder to actually get your survey programmed exactly that way. So leave the frustration to us. We’ve programmed many thousands of surveys, and can do so quickly and efficiently. There’s no need for you to waste time learning how to use yet another survey tool. Just send us your questions in a Word doc, spreadsheet, or even in an email. We’ll take care of the tedious programming.

Finally, through strategic partnerships with a host of survey tools, PeopleFish is able to implement whatever logic or branching you need. Full stop. There’s nothing we can’t accomplish when it comes to survey programming—just tell us exactly what you want to see, and we’ll have your survey programmed in no time. We even employ JavaScript programmers to accomplish logic and branching that’s not part of any survey tool’s standard features.

The Bottom Line

At PeopleFish, we believe every entrepreneur and marketer deserves the chance to hear from consumers. To “test the waters” and validate their ideas with real-world consumer feedback.

So we make it easy. Your questions, quality responses, no minimum purchase. That’s it. No expensive consultants, frustrating survey platforms, or stacks of confusing raw data. Just ask the right questions – we’ll take care of the rest.

Survey Research is Changing. Here’s how.

When I first started PeopleFish in 2016, I called everyone I knew to sell our survey research tools.

Almost no one had work for me. I remember one visit to a friend’s company. I presented my concept, but got just one sentence in response:

“Sorry, we just don’t do surveys.”

Survey market research just wasn’t something people talked about. Sure, lots of businesses were doing market research, but it was typically left to specialists working with expensive, hard-to-understand tools.

Indeed, I was one of those specialists at a big market research firm before quitting to start my own business.

But things have changed.

What’s Changed?

Fast-forward to 2019, and it’s hard to find a business owner who isn’t curious about survey research. Who isn’t trying to survey their customers. Who doesn’t have questions for me about how to survey their target market.

I get unsolicited questions about survey every day. On LinkedIn, Instagram, Twitter. And they’re typically great questions — new challenges or hard-to-solve issues surrounding consumer survey research projects.

What changed over the past three years? I propose these three things:

  1. Companies have gotten better at harvesting customers’ contact info, such that survey research becomes POSSIBLE.
  2. Survey research tech has become more AFFORDABLE, such that even small startups can launch powerful survey research projects.
  3. Survey research tools improved but have become COMPLICATED, such that a pro is often needed to make sense of the tools and programs out there.

At first glance, that last item doesn’t seem to belong. If it’s more doable and lower-cost than ever before, why has it also become more complicated?

Before I answer, I’ll note that probably 25% of our clients come to us after one failed attempt at a survey project. They bought a SurveyMonkey license and the tool was just too complicated. They surveyed their customer database, but no one took the survey. They designed a product concept survey, but the findings were nonsensical.

That’s my evidence for survey research having become more complicated. But what’s behind this trend?

Simply put, as survey technology has improved, the list of possible ways to survey consumers becomes longer and more complex. Sure, anyone can now design a quick 10-question survey to pitch their product to consumers. But what about all those other tools or survey templates floating around? And if I can use complicated survey logic to dig deeper into consumers’ minds, shouldn’t I at least try that, too?

These things add up. Surveys get complicated.

This is true of any technology. The better it gets, the more things are possible. The more things are possible, the more complicated things become.

What Does This Mean for Business Owners?

Why does this matter?

For survey nerds like me, it’s just interesting. It helps me strategize. It helps me sell my company’s services. It helps me design better surveys.

But what about for business owners?

For one, you really shouldn’t be waiting any longer to take advantage of the survey technology that exists. Start now.

If you run a business, you have customers. If you have customers, you have their contact info. Use that, along with survey tools, to ask deep, tough questions — to uncover what your customers want, and what actions you can take that might make them happier.

Second, hire an expert. Yes, SurveyMonkey’s low monthly fee is attractive. But what if you want to run just one survey? Or what if you pay the fee, but then can’t figure it out? Or what if the tool can’t meet your business’s unique and specific needs (the most common complaint I get from new clients)?

The fact is, these tools are increasingly built for specialists and enterprise users, and their best tools are hidden behind high paywalls. While knowledge about general “how-to’s” and the value of survey research has become ubiquitous, the nuts and bolts are increasingly complicated.

So the bottom line? Reach out. I’m around. Our team is ready to field your survey to your target market. Here’s my blueprint for designing your first market research survey. And here’s my free course on surveying your target market.

If you’re starting or running a business, survey research isn’t something that can wait.

Some Best Practices for Market Research Surveys

There’s a million ways a survey project can go wrong.

We see it happen all too often. An energetic startup founder designs a survey he thinks will change the way he does business—turning his “gut instinct” into validated, data-driven insights about his customers.

But the data comes back, after weeks of work, and it’s anything but helpful. Worst case, it’s 100% useless because of some error in the way the survey was designed.

We hate seeing this happen. So we’ve put together a list of industry-secrets when it comes to designing high-quality market research surveys. Some of these may seem like common-sense, but some of them are likely to surprise you.

Remember, we at PeopleFish review every survey we field for our clients, and we won’t ever field something that will yield useless data. That said, we can’t deep-dive into the nuances of every question—ultimately, only our clients know exactly what they need to learn from their target market. That’s where these tips come in handy!


Tips for Writing Surveys

Writing a survey is harder than it seems. Here are some tips—some obvious, some not—on writing a survey that will yield useful findings for your team.

  1. Design your surveys in a word processor (Microsoft Word, Google Docs, etc.). It’s too easy to get thrown off by all the bells and whistles in a survey platform and lose track of your primary research questions and goals.
  2. Use short, simple sentences. No exceptions.
  3. No double-barreling. Don’t try to fit two questions into one.
  4. Stick to single-select, multi-select and rank-order question types. Only use other types if absolutely necessary, and don’t require respondents to rank more than seven items.
  5. Use scales wherever possible. For example, less Yes/No and more Always/Sometimes/Never. Use only five- or seven-point scales, and fully label all scale items (don’t label either end and use numbers in between).

Tips for Cleaning Data

At PeopleFish, we clean our clients’ survey data for them before sending over results. So for those of you working with us, there’s no need to worry about fully understanding these steps. But we thought it might be helpful to explain standard data-cleaning methodology—we do this each of these steps for every survey we field.

  1. Check for speeders. Use Excel to calculate the sample’s median time spent taking the survey, then flag everyone who finished in less than half the median time. Some exceptions apply, depending on the survey logic and the total length of the survey.
  2. Check for flatliners. These are respondents who selected the same scale item across multiple scales. While this doesn’t necessarily mean they should be disqualified, flag anyone that appears to have “flatlined” in this manner.
  3. Check for gibberish & logical inconsistencies. This is self-explanatory—respondents who write nonsense in open-ended boxes, or who contradict themselves across answers need to be flagged.
  4. Add up the flags for each respondents, and discard the worst offenders. Use your judgement to decide what the flag threshold should be.
  5. Track the edits you make while cleaning data. Start a new sheet after flagging for each of the three items listed above, so that someone else could conceivably review what you did to see if they agree with your flag thresholds.
  6. Your sample size shouldn’t change much through your survey. It’s OK to keep some partials, but by-and-large, the number of people who answered the first question in your survey should be close to the number who answered the last.

Tips for Reporting Data

Most of our clients report their data to someone—investors, their boss, as marketing collateral, etc. Here are some tips to make sure you make maximal use of your survey findings in the form of a report.

  1. Remember that when you segment (cut) your data, the sample sizes for each segment drop. Keep this in mind, and don’t segment if it means indefensible sample sizes within segments.
  2. Condense figures and graphs wherever possible. Don’t make clients switch between pages or slides to view obvious comparisons between questions—put like topics and similar questions together.
  3. Don’t impute meaning into respondents’ answers. Just report the facts. Draw conclusions, of course, but that’s different than imputing meaning. Conclusions come with caveats and some degree of confidence. Imputing meaning is asserting more about your respondents’ answers than is actually there.
  4. Write a Key Findings section at the beginning of your report. Most clients won’t read much beyond this anyways, so be sure it contains the most salient findings and directly answers their research questions.
  5. As when writing the survey, use short and simple sentences in your report. No exceptions.

As always, we’re open to any questions about these items. Hit us up on Twitter, or inquire here about launching a survey for your business.