National Survey for Wales: 2017 pilot report (summary)

The main aims of the pilot were to:

• test the final questionnaire for 2017-18, in particular the new income question modules
• test the new survey documents and field materials
• gather interviewer and respondent feedback
• provide more information on interview length, given that significant changes to the questionnaire content had been made since the 2016-17 fieldwork year

The sampling procedure for the pilot followed the same random probability sampling design as the main 2016-17 survey. The exception was that rather than including all local authorities in Wales, 13 local authorities were purposively selected, ensuring these areas covered different parts of Wales, and a mix of urban and rural areas.

The pilot sample was split into two groups, each of which received a different questionnaire. Addresses selected in the three local authorities of Flintshire, Blaenau Gwent and Monmouthshire received the full questionnaire to test the overall flow and to get an estimate on overall timings.

A short version (just including the new modules for 2017-18) was tested in the 10 other local authorities (Anglesey, Gwynned, Conwy, Denbighshire, Wrexam, Ceredigion, Pembrokeshire, Bridgend, Torfaen and Powys) that took part in the pilot. The aim of this shorter version of the questionnaire was to test the flow and understanding of new questions which had been specifically developed for 2017-18.

In order to ensure a good response rate, a conditional incentive of £20 was offered. It was sent out to respondents after they completed the interview. Interviewers were also instructed to make a maximum of four calls before coding out a case as a non-contact (which differs from the minimum of six for the main survey) to ensure resources are focused on attempting the whole issued sample and gaining productive interviews as quickly as possible.

Overall, it was found that the new questionnaire worked well. No major issues were identified in this pilot.

A total of 180 interviews were achieved from 393 issued addresses, including five interviews conducted in Welsh. This represented a response rate of 58.3% based on eligible addresses. No unproductive cases were re-issued. A lower response rate before reissue is to be expected at the main stage of fieldwork as the conditional incentive used at mainstage will only be £10. Re-issuing unproductive cases at main stage will however lead to increased response rates.

The median interview length for the sample receiving the full questionnaire was 47.0 minutes (excluding anomalies and outlier cases), two minutes longer than the expected interview length. The mean interview length was 46.8 minutes (excluding anomalies and outlier cases).

Feedback from interviewers highlighted that changing from a second advance letter (as in the 2016-17 main survey) to a postcard had mixed reviews from interviewers and respondents. On the one hand the change was well received amongst interviewers as it removed the need to write their contact details on both sides of the letter, however there were mixed views on the design of the postcard.

The change from A4 to A5 size show cards was found to be a big success with the majority of interviewers commenting that they were now ‘less daunting’ for respondents, easier to use and looked more professional.

A number of minor questionnaire improvements were suggested by interviewers, and concerns were raised about the new income questions and respondents’ reaction to them.

The updates on the survey materials for the pilot were deemed a success and should be carried forward into the 2017-18 survey year with only minor changes required.

The median interview length was two minutes higher than expected, but this is likely to reduce once the questionnaire has settled in.

Due to concerns raised by interviewers about the respondents struggling with incomerelated questions, the content of interviewer training and briefings should be reviewed before modules on income are included in the National Survey questionnaire. Response rates should be monitored closely after introducing these questions to assess whether they have a negative impact on response.

Martina Helme, Tom Pegg: Office for National Statistics

The full report is available on request.

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

Chris McGowan
surveys@gov.wales

Social research number: 59/2017

Digital ISBN 978-1-78859-612-1