National Survey for Wales: 2022 pilot report (summary)

This report presents findings from the pilot survey carried out by the Office for National Statistics (ONS) prior to the start of the main 2022-23 fieldwork for the National Survey for Wales.

We ran a small pilot in January 2022 which trialled changes planned for the 2022-23 survey year. The findings and feedback from this were used to help improve and finalise the 2022-23 National Survey questionnaire.

The main aims of the pilot were to:

• test the final questionnaire for 2022-23, in particular the new questionnaire modules
• test the updated survey materials (advance leaflet, purpose leaflet and interviewer intro card)
• test the updated online chasing spreadsheet
• gather interviewer feedback
• provide more information on interview length and other survey process metrics

There was a significant amount of change for the National Survey during the last 2 years, as a result of the coronavirus (COVID-19) pandemic.

The plan was that for 2022-23 the National Survey would move back to a more traditional set up in terms of one annual questionnaire with only necessary changes throughout the year, and one full year data delivery.

The main questionnaire would be carried out over the telephone with an interviewer, followed by a short online questionnaire, for the whole sample.

An assisted interview route was added to the questionnaire for respondents who cannot independently answer the questionnaire over the phone but can with assistance from a household member i.e. deaf respondents. Similarly, people who are unable to answer online independently can have an assisted interview with an interviewer, over the telephone.

The sampling procedure for the pilot followed the same random probability sampling design as the main 2021-22 survey. However, only seven of the 22 local authorities were selected, but we ensured these areas were representative geographically and in levels of response rate.

The next birthday method was used to randomly select one adult from the sample addresses, this selection methodology has been used on the National Survey since January 2021.  

If contact could not be made with an address, interviewers would carry out Knock to Nudge (KtN) at the address to try to obtain a phone number.

Telephone survey

A total of 193 interviews were achieved, with 11 interviews conducted in Welsh.

This represented a response rate of 33.3% of eligible addresses; no unproductive cases were re-issued. The average response rate was lower than expected, when compared with mainstage. Response rates in general tend to be slightly lower in January, however five out of the seven pilot areas – Ceredigion, Swansea, Wrexham, Gwynedd and Torfaen – were considerably lower than in the 2021-22 main survey. This was mainly driven by a lower than usual contact rate in these areas (compared to mainstage), as well as higher refusal rates in the areas Torfaen, Gwynedd and Conwy.

An increased response rate is expected at mainstage, by ensuring that a minimum number of calls are made before coding out an address as a non-contact, and by covering all areas of Wales.

The median interview length for the telephone section was 29.9 minutes and the mean interview length was 30.8 minutes (excluding anomalies and outlier cases). Although this is around five minutes longer than the target interview length for mainstage, the increase can be explained by the fact that some sub-samples were purposely routed to a higher proportion of people in the pilot than will be the case at main stage. This was done to achieve more accurate timings for the relevant questionnaire modules and questions. Also, as interviewers get used to the new modules in the questionnaire, the overall interview length is expected to decrease slightly over the course of the first couple of months of the survey year.

The survey materials were found to be working well overall, interviewers only flagged a few minor amendments that could be considered.

Interviewers reported that overall the questionnaire worked well, with no major problems occurring during the field period. However, the feedback highlighted a few sections that would benefit from being streamlined and improving the flow for the respondent. These included: the suggestion that the property question would be better at the beginning when confirming address; work and qualifications sections should be linked; and the routing could be changed so the question “Do you have a car/van” is not being asked twice.

Online survey

A total of 166 online interviews were achieved.

The median interview length was 12.4 minutes and the mean interview length was 14.2 minutes (excluding anomalies and outlier cases).

The data showed that 95 respondents received their voucher by post, whereas 71 respondents received their voucher by email.

 It is worth noting that the voucher section will be changed for the mainstage to ask the required voucher information at the end of the telephone section rather than online. This will have the benefit of routing all with internet access automatically to the e-voucher.

Some minor updates were considered for the materials, including the purpose leaflet and postcard.

For mainstage, consideration should be given to adjusting sub-sampling for some modules to reduce the interview length, and to addressing issues highlighted with the flow and wording of some of the new questionnaire modules.

Dani Evans and Elena Stylianopoulou: Office for National Statistics

The full report is available on request.

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

Chris McGowan
surveys@gov.wales

Social research number: 69/2022

Image

Digital ISBN 978-1-80391-940-9