Avoid survey mistakes with these tips!

Have you ever been asked one of those irritating questions where you don’t know exactly how to respond? We call them survey mistakes, as the surveyor often doesn’t intend for these to happen, but they are still very annoying for the responder.

Imagine you are at a mall and go into a store. There they ask for feedback from their customers. You get prompted with a “Hi, please give us some feedback” on an iPad device. Next up you see the questions, one of which might sound something like “Did you find what you were looking for?” (Yes/No).

Now imagine how a person could potentially respond to this question. If they walked in just to browse through what you have, then the question is probably quite hard to answer. The question assumes that everyone is looking for something specific. If they found what they were looking for, but maybe not in the right size/colour, then a yes or no question also makes answering difficult. Yes, they found what they were looking for, but not exactly.

This brings us to the main focus of today’s article. We introduce some of the insights we have gathered over the years on how to ask questions. We draw upon the most common hassles that our users face and give you a way to overcome them. By the end of this article, you will know how to ask questions, and even more important, how to avoid survey mistakes. The purpose of it all is, to ask questions that are easy to answer for everyone.

The common denominator

The most common of survey mistakes is the complexity of either the survey or the questions that are asked. Complexity of questions is the length of the questions, the use of conditional clauses and vague or subjective framing.

Designing surveys is similar to writing a great sales pitch or talking to customers. You make it as easy as possible to understand. When respondents understand you it is much easier for them to connect with you. Surveys that are easy to understand will give you better and more reliable information.

“The greater the demands a question places on the responder, the less accurate the respondents’ answers.” – The Psychology of Survey Response. By Roger Tourangeau, Lance J. Rips, Kenneth Rasinski

Complexity

Now that we know that it is important to make surveys simple, other commonly seen survey mistakes include overly complex questions. Try asking yourself whether your questions are relatively easy to answer given different scenarios that the responders might be in. If not, then you have a too complex question.

A simple question aims at minimising the period in which the respondent reflects on his or her answer. If respondents need to first figure out the exact meaning of your question then as a result the survey takes longer. Hence the process could influence the response. So minimise the brainpower needed to understand the questions – make them simpler.

An example of a complex survey is when long or vague questions are used. Or when you use difficult words or sentence structures such as conditional clauses. We will go through each to help make the survey more legible.

Conditional clauses

“If you purchased something in the store today, how would you rate the experience, based on staff friendliness?”
How easy is this question to read and respond to?

Some will feel it easy. Others will have difficulty. Other survey mistakes include the use of conditional clauses in your questions. Conditional clauses are two or more statements that are linked together in one sentence in an If-then structure.

Typical conditional questions are those that ask respondents to rate something if some condition has been met. Some users string these together and form quite complex questions.

“If you could improve anything with the product, what would you change?”
“If you had a bad feeling from our customer service, how would it affect your experience?”
“Based on your experience today, would you visit us again?”

Notice how the answers to these questions depend on the first clause? By asking in this manner, people who answer your survey will need to first think and keep in mind the first clause and hereafter answer the question.

To some, this can be frustrating as they need to spend more effort to wrap their thoughts around the question. To others, this is a small trivial matter. Some might not even notice the clause. But if you were to create a survey, you probably would want to make it as easy to answer as possible, right? The good thing is that it is actually quite easy to do. We like to use the phrase “Reframe and omit” which reduces the question to the minimum.

Reframe and omit

Before
“If you could improve anything with the product, what would you change?”
After
“What would you improve on the product?”

Before
“If you had a bad feeling with our customer service, how would it affect your experience?”
After
“How would bad customer service affect your experience?”

Before
“Based on your experience today, will you visit us again?”
After
“Will you visit us again?”

Another way to go around conditional questions is to include an additional question. Conditional questions can almost always be made simpler with an additional question, as they are often comprised of an unasked question. It just requires a little more effort on your part when analyzing the data. Take the example:

“If you heard about our promotional did it influence you to visit us today?”
Not only is this a bad question as it is a conditional question. It also assumes that customers have heard of the promotional offer.

This could be circumvented with an additional question
“Have you heard about our promotional offer?”
(If yes, then) “Did this motivate you to come today?” or “Where did you hear about us?”

Finally, conditional questions should never have two questions in one sentence – this is just bad survey design and is impossible to answer without clarification.
“Have you heard of our promotional offer and did it influence you to visit us today?”

Long questions

“How was your experience of the changing rooms based on staff friendliness, cleanliness and services to help you find the perfect clothes?”

Did you read the entire question? it isn’t everybody who does. We have found that another big no-go is to make long-winded questions. These questions are often composed of many subordinated clauses or conditional clauses. Some survey designers feel that questions need to be long in order for people to respond in the most accurate way. This is not necessarily true. Let’s face it. Humans have short attention spans. If you ask something that is too long, then maybe people will probably not read the entire question.

We recommend that you break the question into parts that is relevant to your focus. Remember to keep the structure and the focus of your survey so you dont go into question overload. Also keep in mind that your feedback location should be where it is important for your research.

Examples of long questions

Another way to go about long questions is to simply make them shorter. Some elements of a question can easily be omitted without changing the meaning.

Before
“I found the content shared by mobile Sales representative (Name) at today’s event relevant to my profession and that it benefits my operational work”
After
“The information from (Name) was relevant for my profession and work”

Before
“Was the service you received from us today at our bank friendly and courteous?”
After
“Was our service friendly and courteous?”

You do not change the meaning of the question by omitting redundant information. The responders already know they are at a bank and that it’s today. So, unless they suffer from a mental condition, it’s safe to remove it.

In addition, we have found that short precise questions often give more significant data than questions that are long. The longer a question is, the more mental resources the respondents need to use to grasp what is asked.

It seems that people in a hurry do not always read the entirety of long questions and instead respond when they feel they have grasped the core of what is being asked. Consequently, this can lead to answers to unfinished questions. What if the essence of your question is at the end? How do you imagine this affects the quality of your data? Not good, right?

But if you really must have a long question in your survey then you need to be aware of a few things. First of all, long questions might affect your data. Secondly, you should structure the questions so they will need to read it entirely before they are able to answer.

Vague and highly subjective questions

Make sure that your questions are straightforward and simple to understand for everyone. Vague and subjective questions are a sure way to compromise your data. By subjective questions, we mean that the meaning of the question is up to the respondents to figure out. As a result the question can be read differently depending on who reads the question.

Subjective questions

A good example of this, is the following question (responders could answer with five smileys):
“How important was the accessibility (in terms of driving), in your choice of visiting us?”

Now imagine that you are the respondent to this question.

How would you answer if you took public transportation?
If you didn’t think a long distance was important?
If you only drove a short distance?
If short distance was extremely important to you, but you still only drove a short distance?

The list is endless. The problem is that all these different scenarios will lead to different results and therefore different data. Getting data that you can actually use to say something with is quite difficult with a question such as this.

Vague words

A similar vein to survey mistakes is highly vague word or subjective topics. Quantitative methodology cautions against using words that can have different meaning to people. The understanding of a topic such as ‘loan’ is for a student is probably different than it is for a person with a house mortgage.

“How was the experience today?” Is another example of vague wording. If your overarching goal is to measure every customers overall experience in the store, then this question might be great. But then, what can you use the data for, except as a confidence boost when things are going great.

Furthermore, if you want data that can be used then you need to know exactly what experience is being measured. A customer who comes in to your store will have various experiences from meeting sales personal, sales time, finding the best item, trying out an item etc. The list is quite long. The question becomes vague the moment you remove context from the question or it isn’t inherently apparent.

In summary

In conclusion, we believe that great questions are easy and simple to read with minimal effort to respond. Unintended survey mistakes can really inhibit how reliable and therefore how useful your data is. Consequently simple questions are crucial if you want good data. Don’t use conditional clauses to qualify the answers. Instead ask the same in two questions or simplify the question.

Secondly, you should make your questions short and to the point. Don’t include useful or nice-to-know facts. Also, try making an art out of omitting information that is irrelevant for the specific question. If this is difficult, try rephrasing the question.

Another thing is that vague questions and highly subjective words inhibit the reliability of your data. You should keep this in mind when you make your survey. Ask whether your question can mean different things to different people. Finally, try to focus your questions on a specific target or context. If you are asking about how an experience was, specify which experience – the purchase experience?

“A good survey designer puts him/herself in the shoes of the respondents and makes the survey as simple as possible. This is our ultimate truth to a good survey. “ – tabsurvey

Stay tuned for the next article where we go into complexity of the survey and how you can optimise the flow in your survey.