I was recently talking to a colleague who teaches at a college nearby. She lamented the number of times she has seen organizations spit out useless and under-prepared surveys. I think that surveys are probably one of the most abused methods of data gathering. I get it, surveys are a tool and if that tool works well in one situation, people try to use everywhere. After all, sometimes a flat-bladed screwdriver will work on a Phillips-head screw.

I saw that a lot in big corporations too. It seemed like every time we used a new research method, all the product teams in our orbit wanted to use it. It didn’t matter if it was the best tool to answer their research questions or not. Don’t get me wrong, I love surveys. They have a bunch of important benefits:

  • Since they are usually deployed online (these days), surveys have the ability to gather data from large, geographically separated groups of people.
  • Surveys can illuminate trends in challenging constructs such as visual appeal, perceived responsiveness, usefulness etc.
  • They can also measure brand awareness, feature importance and a myriad of other challenging questions.
  • Surveys published to a large number of people make it possible to actually perform statistical analysis. This is different from qualitative methods such as in-depth interviews, or usability studies. (Qualitative data is deep and rich, but not appropriate for doing statistical analysis. That is not to say that all data from a survey is viable to use for statistical analysis.) The wording of a question and response options define the viability.

There are other considerations as well when setting out to ask questions through a survey. Below are a few things that may help ensure you can build and deploy a successful survey.

Objectives: What are you trying to learn, and why?

It is very important to keep your objectives in mind while preparing your survey. I always post my objectives at the top of my survey as a header or independent section. This way I can always refer to it while I am creating questions. I like to stay in the survey tool to keep my flow while creating questions. In reality, you can just put them on a note somewhere easy to see. Keep your objectives focused, or your survey will ramble. A rambling survey will cause your respondents to abandon before completion.

I always create more questions than I can use, to begin with. I even solicit ideas from others and look for inspiration online. It is better, but harder, to cut a survey down to the right size from a great set of questions. Pro Tip: if you do temporarily post your objectives in your survey tool for reference, remember to delete them before you publish the survey.

Length: A survey should only be long enough to answer your objective and that’s it.

Speaking of cutting questions, one of the most challenging aspects of any survey is finding the sweet spot between length and participation. As a general rule, a lengthy survey will result in more incomplete/abandoned sessions. Respondent fatigue is a real phenomenon. It is always tempting to throw everything and the kitchen sink in a survey. However, once you pass the respondents’ goodwill zone, your survey becomes a labor. If it gets too laborious, they will leave you with incomplete data. There is a reason that most surveys compensate for participation. Hardly anyone loves taking surveys for other people. To get the most out of your respondents, your survey needs to be short enough to keep it interesting, but long enough to cover the subject.

Sampling: Know who you are talking to.

It is critical that you are talking to the right people to answer your objectives. Your survey data has to come from people who have the experience to know the answers to your objectives. Thus you need to understand the profile you want to reach. Writing down a clear description of the type of person you want to hear from is the first step. After you have the description, you need to ask yourself what type of behavior would demonstrate each of the qualities in the description. Then you can build your screener by asking questions that reveal whether a person does those behaviors or not.

There are a couple of ways to screen potential respondents. You can post the screener questions as a separate survey and when the respondent matches the profile you want, you grant them access to the real survey. Alternatively, you can front-load the whole survey with your screening questions and if the respondent doesn’t fit, you jump them to the end of the survey.

Try before you buy: Consider a pilot.

You should consider running your survey with a small sample of people before launching to a large audience. We call this a pilot survey. A pilot survey verifies that the data you’re gathering comes out how you imagined it would. It also ensures that when you run the large-scale survey, everything will run smoothly. A pilot will also iron out the kinks in with any questions and the way that respondents interpret them. You may not be able to use the data, but it will make the data you do collect much better.  Below, I have a short discussion on the hazards of changing survey questions mid-flight. Mid-flight means while you are gathering data. Running a pilot greatly reduces the risk of having questions that don’t work and need alterations to get the data you want.

Compensation: How do you motivate respondents to answer the survey?

As my friend was lamenting, there are thousands of terrible surveys published every day. It is likely that whoever you want to answer your survey is getting asked to participate in other surveys too. One way of getting their attention and time is to offer a gratuity for their participation. Some people worry that gratuities influence the outcome of a survey. Unless you are providing an inordinately large compensation for taking a survey, it will not change respondents’ opinions about your product or service. Further, providing a gratuity will make it more likely to reach the people you really want to talk to.

Passionate people will always find you.

If someone loves your product they will find a way to tell you. They will email, or blog, provide a review, or find another way of telling you. The same is true of those who have had a regrettable experience, whether it is your fault or not. However, neither of those groups will give you the information that you really need. The ones who love it will continue to love it and buy more. Those that hate it probably will never come back. You need to get in touch with those people who had a good experience but did not love it. Those people will tell you real, actionable things that you can change to improve.

Here are a few gratuity methods that are very effective:

  • You can offer a small gratuity for each participant. However, if you are looking for a large number of participants this can become costly even at moderate rates.
  • Run a sweepstake and offer few but larger prizes through random selection. Make sure to consult a lawyer in the locale that you are running the survey because the laws surrounding sweepstakes differ between states and countries.
  • Offer early/special/extra access to your product or service. This type of compensation will appeal to a specialized demographic, so you should keep that in mind as it will slant the type of feedback you are going to gather.

Resources on how to write great questions.

There are resources available on the web to demonstrate great survey questions. you can models your own questions from them. If you haven’t written many surveys previously and if you’ve read this far you probably haven’t, then you can leverage the internet to find already proven examples that will greatly speed up the survey creation process. Writing a good survey is not about being original. It is about creating the right prompt so that your customers can tell you what you need to hear. The trick is doing that without compromising the integrity of the data. There are several great survey tools out there and each has their own take on how to write survey questions. I still use them as sources of inspiration. Try looking at guides for Survey Monkey, Qualtrics, or SurveyGizmo to find good examples.

Things that can make your survey fail.

Like most research methods, the key to getting great survey results is to do the prep-work before you execute. In my mind, this is even more true for surveys than other methods. That is because once you publish your survey, the questions have to stand on their own without additional explanation, clarification, or re-direction. Once loose, making changes to the survey is fraught with danger. Ideally, you don’t want to change any questions after you publish your survey.

Changing questions mid-flight

There are real reasons to not change the survey questions once it’s published. Changing the questions after publishing creates an opportunity to confound your survey results. If you change a question, even a little bit, that change can influence the respondent’s choices. Some would say that the risk is too high, and you should never make changes after publishing. However, sometimes it becomes abundantly clear that the intent of the question is not at all clear from the responses. In the real world, respondents cost money and resources, so throwing out all the data may not be acceptable.

You changed a question, how do you save your data set (and budget)?

Let’s assume some respondents have already answered a particular question (item) and you have to change it. You should treat the responses of those who saw the original and those who saw the revision as separate groups. In the analysis, you are looking for other differences between the groups’ response patterns and not just that question. If you detect other differences, it may indicate that your change influenced more of the survey. It’s true that you lose some of the power which you are probably trying to achieve with running a survey. However, it is the only way to be sure you do not inadvertently reach unsupported conclusions. The best way to avoid this problem is to carefully craft your questions, pilot them with a small group, and then iterate before you publish.

Know where you are going before you get there

Piloting a survey makes sure that you are not confusing your respondents when they see your questions. However, you should also know how to analyze a question result before you put it in your survey. With all the effort you have put in to plan and prepare a survey, it is easy to forget about the analysis until all the data is collected. That is a mistake. If you are not sure how the data from a question is going to appear during the analysis, you run the risk of collecting data that you cannot use. Additionally, you also lose a precious question. Since you are limited in the number of questions you can ask, you want each one to be a revealing experience.

Summary

Surveys are just one more tool that we use to discover the needs and wants of our customers. Running a great survey takes more planning than you would expect. But, if you plan appropriately, you can use a survey to gather reliable, bankable insights. Writing great survey questions is undeniably a skill and takes practice. With so many free tools available to create surveys, it is tempting to just dump a bunch of questions out on the Internet and take your chances. Yet, poorly constructed surveys result in more questions than answers. One thing you probably don’t want to do is create more work for yourself. I hope that you can find some usefulness in this high-level framework for your next survey. If you have an interesting experience, I’d love to hear about it. Let me know how it goes.