‘Select all that apply’ questions are quick a quick and easy option for online surveys; however, studies show that the select-all-that-apply option produces less accurate results than forced-choice responses.
Before we turn to the research studies, let’s get oriented. Below is a survey question with an example of a ‘forced choice’ question and an example of a select-all-that-apply question.
Forced-choice Question
Which of the following stores have you made a purchase from in the past 3 months?
Select-all-that-apply Question
Which of the following stores have you made a purchase from in the past 3 months? Please select all that apply.
- Costco
- Walmart
- Home Depot
- Trader Joe’s
- Best Buy
- 7-11
- Gap
- Wayfair
- Tim Hortons
- Bath & Body Works
- Sephora
Academic Studies
RESEARCH: When Online Survey Respondents Only ‘Select Some That Apply’
Pew Research Centre | May 9, 2019
ARTICLE | FULL REPORT | ABOUT PEW
In 2019, the Pew Research Centre found “compelling evidence that forced-choice questions yield more accurate results than select-all-that-apply lists.” Pew randomly assigned survey respondents to see either a question with a “check all that apply” list, or the question with a required yes/no response for each list item.
Consistently, more items in the forced-response question were selected by respondents than in the select-all-that-apply question. For one of Pew’s questions, there was a 12 percentage point difference:
“The estimated share of U.S. adults saying that they or someone in their immediate family had lost a job and struggled to find another one was 63% using the forced-choice format versus 51% under select-all-that-apply.”
The Pew Research Center subsequently adopted a policy of using a forced-choice format instead of a select-all-that-apply list in its online surveys, whenever possible.
RESEARCH: Comparing Check-All And Forced-Choice Question Formats in Web Surveys
Public Opinion Quarterly. Spring 2006.
Smyth, Jolene, Don Dillman, Leah Christian and Michael Stern. Vol. 70, No. 1, pp. 66–77.
To test whether check-all-that-apply responses and forced-choice responses produced similar results in surveys, the researchers conducted 16 experimental comparisons, using two online surveys and one pencil and paper survey.
For all 16 comparisons, respondents who saw the forced-choice response format chose more options and took longer to answer the questions than respondents who saw the select-all-that-apply response format.
The researchers concluded that the forced-choice question format encourages respondents to think more deeply about the response options and thus, forced-choice survey questions are preferable to check-all-that-apply survey questions.
RESEARCH: Select-All-That-Apply versus Yes/No Forced Choice Items
MeasuringU. Jim Lewis, PhD and Jeff Sauro, PhD | December 7, 2021
Researchers Jim Lewis and Jeff Sauro conducted a literature review to identify research that compared respondent answers to a forced-choice question versus select-all-that-apply question.
They found five studies in their literature review. In all five of these studies, respondents selected more items for forced-choice questions than for the select-all-that-apply version of the same question. That said, there was substantial variation in the magnitude of difference between the forced-choice and select-all-that-apply questions. The differences varied from 1% to 24%.
The authors hypothesis that the wide range in response differences may reflect whether the forced-choice format was yes/no or a multi-point scale. They intend to do independent research to better understand their finding from the literature review.