- The longer the survey, the higher the drop-out rate

- Engagement is very important, and participants might be less interested in a survey that is too long or costs too much time.

- Researchers should be careful about redundant questions and only keep the necessary ones.

- If all questions are important, some of them could be randomly assigned into a subset.


Long, Long, Long Surveys

- Although open questions allow participants to explain their thoughts, they are more time-consuming and require more effort. 

- Open-ended questions should not be the focus of the survey and they should be optional.


Many Open-Ended Questions

- Finding very specific survey participants is difficult. One example of the “impossible screening criteria” from the article is “Looking for women between the ages of 25-34 who bought a washing machine on their smartphone in the last month but have never heard of Maytag.”
- Participants who only meet some of the criteria could still provide efficient responses.


Impossible Screening Criteria


- Listing too many items in a matrix could overwhelm the participants and makes the survey more time-consuming.
- Only listing the necessary items and consider re-group them into different questions.


Giant Matrix Questions

- Adding a progress bar or other types of progress indicator helps participants to keep track of their progress and encourage them to finish the survey.


No Progress Bar or Sense of Completion

- Avoid asking too personal questions, especially at the beginning of the survey. 
- While composing demographic questions, be careful about the wording of the questions and response choices.  


Really Personal Questions Right Upfront

- Although participants might be able to put “N/A” or “Not Applicable” as their responses, too many such questions will overwhelm and discourage the participant and make the survey longer than it needs to be.

Redundant and Irrelevant Questions

- Participants could be asked to answer forced rank questions, but if the question contains too many items, participants could be confused and spend too much time.
- Run a pre-test to track the time and decide if all items need to be kept.
- Consider using the Top-task approach if all items must be included.


Confusing and Challenging Questions


- Example: Please rate your level of agreement to the statements about the role of the Federal Reserve’s recent announcement on monetary policy.

- If given the chance to express their opinion, people will have one, even if it’s uninformed or if people don’t have a strong feeling about the topic. For such questions, participants may simply put N/A or Neutral.


Questions That You Have Little or No Opinion On

- Incentive is not a must-have, and sometimes the best response rates come from leveraging the interest participants have in having their opinion heard.
- However, for long surveys, incentives could attract and encourage participants to finish them.


No or Low Incentive

10 common mistakes in a survey design. Avoiding these errors will make the survey more effective.

Arizona State University

University of Michigan - Ann Arbor

Ohio State University, Columbus

User surveys

Sauro, J. (2014, February 11). MeasuringU: 10 ways to get a horrible survey response rate. MeasuringU. https://measuringu.com/horrible-responserate/


10 Ways to Get a Horrible Survey Response Rate

Types of user surveys

Rosala, M. (2020, June 7). Rating Scales in UX Research: Likert or Semantic Differential? Nielsen Norman Group. https://www.nngroup.com/articles/rating-scales/

Rating Scales in UX Research: Likert or Semantic Differential? 

Likert and Semantic Differential are two of the most common rating scale systems within survey design and UX Research. Both measure various degrees of opinion, attitudes, perceptions, user preferences. Data collected helps with improving the user experience.

Learn Before

Related

Learn After