National Survey for Wales: 2019 pilot report (summary)

The main aims of the pilot were to:

• test the final questionnaire for 2019-20, in particular the new questionnaire modules
• test the up-dated survey materials (advance leaflet and showcards)
• gather interviewer feedback
• provide more information on interview length and other survey process metrics

The sampling procedure for the pilot followed the same random probability sampling design as the main 2018-19 survey, with the exception that rather than including all local authorities in Wales, six local authorities were purposively selected, ensuring these areas covered different parts of Wales, and various levels of response rates in the previous survey year.

Interviewers were provided with a Kish grid table including random numbers for each address in the quota. This random number table was used to facilitate the random selection process of adults living at sampled addresses.

A total of 198 interviews were achieved, with 1 interview conducted in a third language (other than Welsh and English).

This represented a response rate of 47.6% of eligible addresses; no unproductive cases were re-issued. The average response rate was lower than expected even for a pilot with reduced follow-up compared with mainstage. This is partly because a larger sample was drawn for this pilot due to response rates in January are typically slightly lower than at other times of the year and partly because three out of the six pilot areas (Flintshire, Swansea and Vale of Glamorgan) achieved a lower response rate than the average response rate usually achieved in these areas in the main survey. This was mainly driven by a lower than usual contact rate in these areas (compared to mainstage), as the refusal rates were similar to those normally achieved on the main survey. Interviewer feedback also did not point to any concerns around gaining response at the doorstep. At mainstage, ensuring that a minimum number of calls are made before coding out an address as a non-contact, as well as re-issuing unproductive cases, is expected to lead to an increased response rate for these areas and overall.

The median interview length was 49.3 minutes and the mean interview length was 49.7 minutes (excluding anomalies and outlier cases). Although this is four minutes longer than the target interview length for mainstage, the increase can be explained by the fact that some sub-samples were purposely routed to a higher proportion of people in the pilot than will be the case at main stage. This was done to achieve more accurate timings for the relevant questionnaire modules and questions. Also, as interviewers get used to the new modules in the questionnaire, the overall interview length is expected to decrease slightly over the course of the first couple of months of the survey year.

The survey materials were found to be working well overall, but interviewers flagged up some amendments that should be considered for the A5 postcard reminder and the questionnaire showcards.

Interviewers reported that overall the questionnaire flowed well, with no major problems occurring during the field period. However, the feedback highlighted a few sections that would benefit from being streamlined and improving the flow for the respondent.

Some minor updates may be considered for the A5 postcard, but otherwise survey materials for the pilot were deemed suitable and should be carried forward into the 2019-20 survey year.

A few changes to showcards should be considered to ensure the use of showcards flows well together with the questionnaire.

For mainstage, consideration should be given to adjusting sub-sampling for some modules to reduce the interview length, and to addressing issues highlighted with the flow and wording of some of the new questionnaire modules.

Martina Helme, Rachael Ryan and Zoe Brown: Office for National Statistics

The full report is available on request.

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

Chris McGowan
surveys@gov.wales

Social research number: 09/2020

Image

Digital ISBN 978-1-80038-065-3