National Survey for Wales: 2018 pilot report (summary)

The main aims of the pilot were to:

• test the final questionnaire for 2018-19, in particular the new questionnaire modules
• test the up-dated survey materials (advance letter and leaflet, and showcards)
• test a new interviewing approach where two randomly selected adults should be interviewed in households with two or more adults living at a sampled address
• gather interviewer and respondent feedback
• provide more information on interview length for one versus two-adult interview scenarios and other survey process metrics

The sampling procedure for the pilot followed the same random probability sampling design as the main 2017-18 survey. There were two main differences. One was that rather than including all local authorities in Wales, nine local authorities were purposively selected, ensuring these areas covered different parts of Wales, and various levels of response rates in the previous survey year. The second difference was that where two or more adults lived at a sample address, the survey was carried out with two adults in turn (randomly selected where there were more than two living at an address).

Interviewers were provided with a Kish grid table including random numbers for each address in the quota. This random number table was used to facilitate the random selection process of adults living at sampled addresses. Previously this Kish grid table only included random numbers for the selection of one adult per household. In order to test the new two-adult interviewing process, additional random numbers for each address in a quota were added to the table.

A total of 194 individual interviews across 129 households were achieved, with one interview conducted in Welsh. This represented a household response rate of 57.8% of eligible addresses. Of those households where two adults were selected to be interviewed, 73% of the second selected adults agreed to participate in the survey, which was higher than expected. No unproductive cases were re-issued. Re-issuing unproductive cases at main stage will lead to a slightly increased response rate.

For one-adult interviews, the overall interview length was 50 minutes (median). Although this is five minutes higher than expected, the increase can be explained by the fact that some sub-samples were purposely routed to a higher proportion of people in the pilot than will be the case at main stage. This was done to achieve more accurate timings for the relevant questionnaire modules and questions. In households where two adults were interviewed, the median average was 54 minutes (median) per person. This was the opposite of the expected effect: we had anticipated that the first adult in a two-adult household would have an interview of 45 minutes on average, and the second adult an interview of 40 minutes. It was clear from the pilot that the interviewer was required to spend extra time managing the respondents when changing from the first to the second, and that individual interviews were extended as the respondent not being interviewed sometimes interjected.

It was found that the two-adult interviewing approach worked well, but required interviewers to be more flexible in planning their workload. Feedback from interviewers on the two-adult interviewing approach was mixed. Whilst the process overall seemed to be working well, interviewers drew attention to the challenges they faced planning their workload and arranging appointments considering the potentially longer interviews in households where two adults needed to be interviewed. Interviewers also highlighted the importance of being able to do both adult interviews at the same appointment to avoid a refusal from the second adult, although pilot data could not confirm this as the pilot did not record at what point the second adult refused.

The survey materials were found to be working well, but interviewers proposed some ways in which the advance letter could be improved. Interviewers also flagged up some minor amendments required to some showcards.

Interviewers reported that overall the questionnaire flowed well with no major problems occurring during the field period. Due to the interview being perceived by interviewers and respondents as quite lengthy, interviewers highlighted a number of sections that would benefit from being streamlined and being made clearer to the respondent. Interviewers also came across a number of small technical and formatting issues that ought to be addressed.

WG have given careful consideration to all elements of the experimental design (as outlined in Annex A), and in particular the potential risks and benefits resulting from it. Based on the potential savings from the change in survey design (which may be smaller than originally expected due to the longer interview length) being fairly small, WG have decided that the financial gain from that is not large enough to justify the risks to survey delivery and the potential impact on the quality of the estimates, and therefore decided not to run the experiment during the survey year 2018-19 and continue with the current sample design going forward.

Although findings with regard to the two-adult interviewing approach will not need to be considered for the mainstage survey, this pilot has highlighted a number of other suggestions on how the survey materials and questionnaire can be improved further.

The survey advance letter and leaflet were found to be working well, but suggestions on how to make them appeal more to the younger population and to tailor them more to local areas are worth considering. Whilst considerable changes to advance materials may not be introduced without thorough testing, a research project assessing the impact of the proposed minor changes on response rates could be considered.

Feedback from interviewers also highlighted potential improvements to the handling of showcards, as well as and some minor mistakes on some of the showcards.

Whilst the questionnaire was found to be working well overall, a number of issues were flagged up by interviewers suggesting where the questionnaire may not be working as expected. Interviewers also highlighted several ways in which the interview could be streamlined, to address the perception that it is lengthy. Some of these suggestions would only require small changes to the questionnaire and therefore could be applied in time for the start of the new mainstage. Other suggestions would require more detailed questionnaire development and cognitive testing to address, which may be worthwhile if WG is planning to continue relevant modules on the questionnaire in future survey years.

Martina Helme: Office for National Statistics

The full report is available on request.

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

Chris McGowan
surveys@gov.wales

Social research number: 51/2018

Image

Digital ISBN 978-1-78937-918-1