tabsurvey blog

Avoid making these mistakes in your survey

Get to better questions with these tips!

Have you ever been asked one of those irritating questions where you don’t know exactly how to respond?

Imagine you are at a mall and go into a store. There they ask for feedback from their customers. You get prompted with a “Hi, please give us some feedback” on an iPad device. Next up you see the questions, one of which might sound something like “Did you find what you were looking for?” (Yes/No).

Now imagine how a person could potentially respond to this question. If they walked in just to browse through what you have, then the question is quite hard to answer. The question assumes that everyone is looking for something specific. If they found what they were looking for, but not in the right size/colour, then a yes or no question also makes answering difficult. Yes, they found what they were looking for but not exactly.

This brings us to the main focus of today’s article. We introduce some of the insights we have gathered over the years on how to ask questions. We draw upon the most common hassles that our users face and give you a way to overcome them. By the end of this article, you will know how to ask questions, and even more important, what to avoid. The purpose of it all is, to ask questions that are easy to answer for everyone.

The common denominator

The most common issue in surveys is the complexity of either the survey or the questions that are asked. Complexity of questions is the length of the questions, the use of conditional clauses and vague or subjective framing.

Designing surveys is similar to writing a great sales pitch or talking to customers. You make it as easy as possible to understand. When respondents understand you it is much easier for them to connect with you. Surveys that are easy to understand will give you better and more reliable information.

“The greater the demands a question places on the responder, the less accurate the respondents’ answers.” – The Psychology of Survey Response. By Roger Tourangeau, Lance J. Rips, Kenneth Rasinski

Complexity

Now that we know that it is important to make questions simple, the next obstacle is discovering whether your survey has questions that are complex. Try asking whether your questions are relatively easy to answer given different scenarios that the responders might be in. If not, then you have a too complex question.

overcome complexity
How complex are your questions?

A simple question aims at minimising the period in which the respondent reflects on his or her answer. If respondents need to first figure out the exact meaning of your question then the survey takes longer and the process influences the response. So minimise the brainpower needed to understand the questions – make them simpler.

Typical examples of complex questions are when long or vague questions or when you use difficult words or sentence structures such as conditional clauses. We will go through each to help make the survey more legible.

Conditional clauses

“If you purchased something in the store today, how would you rate the experience, based on staff friendliness?”
How easy is this question to read and respond to?

Some will feel it easy. Others will have difficulty. We have found that a big no-no in surveys is to use conditional clauses in your questions. Conditional clauses are two or more statements that are linked together in one sentence in an If-then structure.

Typical conditional questions are those that ask respondents to rate something if some condition has been met. Some users string these together and form quite complex questions.

“If you could improve anything with the product, what would you change?”
“If you had a bad feeling from our customer service, how would it affect your experience?”
“Based on your experience today, would you visit us again?”

Notice how the answers to these questions depend on the first clause? By asking in this manner, people who answer your survey will need to first think and keep in mind the first clause and hereafter answer the question.

To some, this can be frustrating as they need to spend more effort to wrap their thoughts around the question. To others, this is a small trivial matter. Some might not even notice the clause. However, if you create a survey, wouldn’t you want to make a survey that is as easy to answer as possible? The good thing is that it is actually quite easy to do. We like to use the phrase “Reframe and omit” which reduces the question to the minimum.

Reframe and omit

Before
“If you could improve anything with the product, what would you change?”
After
“What would you improve on the product?”

Before
“If you had a bad feeling with our customer service, how would it affect your experience?”
After
“How would bad customer service affect your experience?”

Before
“Based on your experience today, will you visit us again?”
After
“Will you visit us again?”

Another way to go around conditional questions is to include an additional question. Conditional questions can almost always be made simpler with an additional question – as they are often comprised of an unasked question. It just requires a little more effort on your part when analyzing the data. Take the example:

“If you heard about our promotional did it influence you to visit us today?”
Not only is this a bad question as it is a conditional question. It also presumes that customers have heard of the promotional offer.

This could be circumvented with an additional question
“Have you heard about our promotional offer?”
(If yes, then) “Did this motivate your to come today?” or “Where did you hear about us?”

A final note on conditional questions is to never have two questions in one sentence – this is just bad survey design and is impossible to answer without clarification.
“Have you heard of our promotional offer and did it influence you to visit us today?”

question capacity
Question information overload?

Long questions

“How was your experience of the changing rooms based on staff friendliness, cleanliness and services to help you find the perfect clothes?”

Did you read the entire question? it isn’t everybody who does. We have found that another big no-go is to make long-winded questions. These questions are often composed of many subordinated clauses or conditional clauses. Some survey designers feel that questions need to be long in order for people to respond in the most accurate way. This is not necessarily true. Let’s face it. Humans have short attention spans. If you ask something that is too long, people will not read the entire question.

We recommend that you break the question into parts that is relevant to your focus. Remember to keep the structure and the focus of your survey so you dont go into question overload. Also keep in mind that your feedback location should be where it is important for your research.

Examples of long questions
Another way to go about long questions is to simply make them shorter. Some elements of a question can easily be omitted without changing the meaning.

Before
“I found the content shared by mobile Sales representative (Name) at today’s event relevant to my profession and that it benefits my operational work”
After
“The information from (Name) was relevant for my profession and work”

Before
“Was the service you received from us today at our bank friendly and courteous?”
After
“Was our service friendly and courteous?”

You do not change the meaning of the question by omitting redundant information. The responders already know they are at a bank and that it’s today. So unless they suffer from a mental condition, it’s safe to remove it.

We have found that short precise questions often give more significant data than questions that are long. The longer a question is, the more mental resources the respondents need to use to grasp what is asked.

People in a hurry do not always read the entirety of long questions and instead respond when they feel they have grasped the core of what is being asked. This can lead to answers from unfinished questions. What if the essence of your question is at the end? How do you imagine this affects the quality of your data? Not good, right?

If you really must have a long question in your survey then you need to be aware of a few things. First, long questions might affect your data. Secondly, structure the question so they will need to read it entirely before they are able to answer.

Vague and highly subjective questions

Make sure that your questions are straightforward and simple to understand for everyone. Vague and subjective questions is a sure way to compromise your data. By subjective questions, we mean that the meaning of what is asked is up to the respondents to figure out. Said in another way, that the question can be read differently depending on who reads the question.

are answers and questions aligned?
Are answers and questions aligned?

Subjective questons
A good example of precisely this is the following question (responders could answer with five smileys):
“How important was the accessibility (in terms of driving), in your choice of visiting us?”

Now imagine that you are the respondent to this question.

How would you answer if you took public transportation?
If you didn’t think a long distance was important?
If you only drove a short distance?
If short distance was extremely important but you still only drove a short distance?

The list is endless. The problem is that all these different scenarios will lead to different results and therefore different data. Getting data that you can actually use to say something with reliable certainty is quite difficult with a question such as this.

Vague words
A similar vein to the example above is highly vague word or subjective topics. Quantitative methodology cautions against using words that can have different meaning to people. The understanding of a topic such as ‘loan’ is for a student very different than it is for a person with a house mortgage.

 “How was the experience today?” Is another example of vague wording. If your overarching goal is to measure every customers overall experience in the store, then this question might be great. But then what can you use the data for except as a confidence boost when things are going great.

If you want data that can be used then you need to know exactly what experience is being measured. A customer who comes in to your store will have various experiences from meeting sales personal, sales time, finding the best item, trying out an item etc. The list is quite long. The question becomes vague the moment you remove context from the question or it isn’t inherently apparent.

In summary

We believe that great questions are easy and simple to read with minimal effort to respond. Complexity can really inhibit how reliable and therefore how useful your data is. Simple questions are crucial if you want good data. Don’t use conditional clauses to qualify the answers. Instead ask the same in two questions or simplify the question.

Secondly, make your questions short and to the point. Don’t include useful or nice-to-know facts. Make an art out of omitting information that is irrelevant for the specific question. If this is difficult, try rephrasing the question.

Vague questions and highly subjective words inhibit the reliability of your data – keep this in mind when you make your survey. Ask whether your question can mean different things to different people. Also try to focus your questions on a specific target or context. If you are asking about how an experience was, specify which experience – the purchase experience?

“A good survey designer puts him/herself in the shoes of the respondents and makes the survey as simple as possible. This is our ultimate truth to a good survey. “ – tabsurvey

Stay tuned for the next article where we go into complexity of the survey and how you can optimise the flow in your survey.

9 thoughts on “Avoid making these mistakes in your survey”

  1. you are really a just right webmaster. The website loading pace is amazing. It seems that you are doing any unique trick. Furthermore, The contents are masterwork. you have performed a excellent activity on this subject!

  2. Hey there! This is my first visit to your blog! We are a group of volunteers and starting a new project in a community in the same niche. Your blog provided us useful information to work on. You have done a marvellous job!

  3. I absolutely love your site.. Pleasant colors & theme. Did you develop this website yourself? Please reply back as I’m attempting to create my own personal website and want to find out where you got this from or just what the theme is called. Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *