Real World Surveys: Flaws & Fixes

When developing a survey and conducting data analysis, there is an enormous gulf between academic learning and real world applications. You may know that question order is important, but not how to avoid question order bias in your actual questionnaire. You may be a whiz at calculating statistical significance in stats class, but it can be tricky in a real survey context to figure out when to apply this concept and how to interpret it.

The Survey Science Institute reviews real world surveys to help you build your knowledge of common flaws… and how to avoid them or fix them. We offer some core information on core concepts, but our focus is on their application in actual surveys.

Hiring and recruiting for work. Find the person you are looking for. Spies and corrupt officials. Industrial and corporate espionage.

Survey Sample

Do you have a random sample survey, attempted census, census, or sample of convenience? How do you know? Why does it matter? What do you need to do to ensure that you can extrapolate from your sample to your target population as a whole. It’s tricky.

Customer feedback and review analysis by modish computer software for corporate business

Data Collection

It seems so easy. Ask people questions and get answers. How you collect data and who you collect it from are absolutely critical for data accuracy. Should you do your survey by phone, online, text, or in-person, or a mix? What are the implications for data accuracy?

Questionnaire Design

How hard can it be? The questionnaire looks good to you and your colleagues. There are many considerations when designing a questionnaire, including question order, understandability, validity and reliability, respondent fatigue, skip patterns, and much much more.

empty blackboard with drawn checked boxes

Response Scales

The question response scale is important. An appropriate scale is critical for getting the information you need. When should you use a 5-point vs 6-point Likert scale? When does allowing a “don’t know” response improve and when does it undermine data accuracy?

Data Analysis

There are a number of ubiquitous issues that arise with survey data analysis: lack of systematic analysis, inappropriate use of statistical significance and p-values, and complex, but nonsensical, statistics.

Statistical Significance

Statistical significance, margins of error, and confidence intervals estimate error related to the selection of a random sample from a target population. They are very often used inappropriately.

Reporting

The smallest word change in a survey questionnaire can change the survey findings. As such, when reporting on survey findings it is very important to avoid paraphrasing the questions and responses as much as possible.

Flaws & Fixes

Real World Surveys: Flaws & Fixes

Survey Sample

Do you have a random sample survey, attempted census, census, or sample of convenience? How do you know? Why does it matter? What do you need to do to ensure that you can extrapolate from your sample to your target population as a whole. It’s tricky.

READ MORE

Data Collection

It seems so easy. Ask people questions and get answers. How you collect data and who you collect it from are absolutely critical for data accuracy. Should you do your survey by phone, online, text, or in-person, or a mix? What are the implications for data accuracy?

READ MORE

Questionnaire Design

How hard can it be? The questionnaire looks good to you and your colleagues. There are many considerations when designing a questionnaire, including question order, understandability, validity and reliability, respondent fatigue, skip patterns, and much much more.

READ MORE

Response Scales

The question response scale is important. An appropriate scale is critical for getting the information you need. When should you use a 5-point vs 6-point Likert scale? When does allowing a “don’t know” response improve and when does it undermine data accuracy?

Data Analysis

There are a number of ubiquitous issues that arise with survey data analysis: lack of systematic analysis, inappropriate use of statistical significance and p-values, and complex, but nonsensical, statistics.

Statistical Significance

Statistical significance, margins of error, and confidence intervals estimate error related to the selection of a random sample from a target population. They are very often used inappropriately.

Reporting

The smallest word change in a survey questionnaire can change the survey findings. As such, when reporting on survey findings it is very important to avoid paraphrasing the questions and responses as much as possible.