Survey Methodology relates to the approach taken to select survey respondents (sampling) and to other key facets of a survey that affect the accuracy of the data collected. Published surveys should include a methodology section to help readers understand what was done and the accuracy of findings.
A description of a survey’s methodology typically includes a description of the following:
- Specific Target Population
- The type of survey you are intending to conduct (i.e., census, attempted census, random (probability) sample or non-probability)
- How members of the Target Population are selected/recruited
- If a random sample, what was done to ensure that key characteristics of respondents in the survey sample data are similar to known key characteristics of the Target Population (e.g., proportion of the Target Population in different age groups)
- Response rate (sample size compared to the Target Population size)
- Medium through which responses were collected (e.g., telephone, online, SMS, in-person, etc.)
- Duration of the data collection period
- Language of the survey
- Sources of systematic bias (e.g., proportion of the Target Population that does not speak the language of the survey)
A write up of the Survey Methodology may also discuss:
- Weighting of data
- Cleaning the data
- Source(s) of the survey questions and validity of survey questions
- Training of interviewers if it’s an in person survey
- Pre-testing of the methodology, if applicable
The Survey Methodology description does not usually include the actual survey questions, the survey findings, or details about how the data were analyzed.
Real World Survey Example
People for Education is a non-profit organization in Canada that conducts surveys. It is the Methodology Appendix to their 2022-23 Annual Ontario School Survey Report that prompted this blog post.
Below is an annotation of this Methodology Appendix, which is intended to help clarify what information is needed for a reader to be assured that survey data are accurate and to be advised of potential skews due to response bias. Text is purple are direct quotes the Methodology Appendix.
Findings from this report are based on 1,044 responses to People for Education’s 2022-23 Annual Ontario School Survey (AOSS) collected from principals across all 72 publicly funded school boards in the province.
This information is not helpful in helping a reader understand the accuracy of the data. It is good to know how many respondents there were, but it’s more important to know the response rate and how well the respondents reflect the Target Population.
Although not stated, the Target Population is likely all operating public and Catholic (not private) elementary, middle and secondary schools in the Province of Ontario. The Ontario school boards cover four types of schools: English public, English Catholic, French public and French Catholic. How well do the 1044 respondent in the sample reflect these divisions? Was there only one person per school permitted in the survey, or does the survey sample include more than one person from some schools?
Longitudinal data comparisons are based on the data collected from the secondary schools that participated in our 2021-22 AOSS.
It is appropriate to provide the source of the longitudinal data in the 2022-23 Report; however, without the necessary methodological information we cannot know if year-on-year differences are real or are a result of a change in the methodology.
Unless cited from other sources, the statistics and quoted material in this report originate from People for Education’s 2022-23 AOSS, the 26th annual survey of elementary schools, and the 23rd annual survey of secondary schools in Ontario.
This information is helpful, but still begs the core question of the accuracy of the current 2022-23 survey.
Surveys from the 2022-23 AOSS were completed online via SurveyMonkey in both English and French in the fall of 2022.
This statement provided critical information. It tells us that the questionnaires were completed online, the questionnaire was provided in English and French, and data collection occurred over a few months in the fall of 2022. Given the data collection in 2022, it seems odd that this survey is referred to ask the 2022-23 survey, since no data was collected in 2023.
Please note that there is a very important difference between a “survey” and a “questionnaire.” A survey includes many element of which a questionnaire is only one.
Survey responses were disaggregated to examine survey representation across provincial regions (see table below). Schools were sorted into geographical regions based on the first letter of their postal code. The GTA region includes schools with M postal codes as well as those with L postal codes located in GTA municipalities.
The above information is typically not included in the methodology as it doesn’t pertain to the accuracy of data collection or data analysis. It’s more appropriate in the body of the Report or in a footnote in the Report.
The above chart provided in the Methodology Appendix provides critical information. It tells us that the regional composition of the sample reflects the Target Population well for schools in the Eastern, Central and Southwestern regions, but schools in the Northern Region are over-represented (over-sampled), while schools in the GTA are under-represented.
To address the over-representation and under-representation, the sample data should be weighted. This is a tricky process. To correct for the regional skew alone, the data should be weighted to a representative sample of n=861 with the remaining responses considered to be over-samples.
During analysis, data collected from the survey were matched with the Median Census Family Income of Schools based on …
The discussion of the grouping of data in the analysis phase is useful, but doesn’t address the core issues related to data accuracy.
Similarly, the last two paragraphs of the Methodology Appendix provide interesting information but do not speak to how well the survey sample data reflect all schools in the Target Population, which is the primary purpose of a Survey Methodology discussion.
Elevate your game.
It is not possible to provide you here with all the detail necessary you need to design a survey methodology, analyze data, and write up findings. We do hope, however, that the quick discussion above alerts you to the necessity of having training and experience in order to design a survey that produces accurate data.