National Survey for Wales: testing of recruitment materials and online survey access

The survey originally involved face-to-face interviews in people’s homes, but since May 2020 it has been carried out by telephone.

All participants complete a 35-minute survey over the phone. However, a random subsample then fill in a 15-minute online self-completion section.

The advance materials include a letter and information leaflet, which are sent together to each household. The letter asks the household to provide a phone number, either via an online portal or by calling the field office. An interviewer will then call the household and randomly select a participant using the ‘next birthday’ method. There are two versions of the letter and the leaflet: a standard version for respondents only asked to do the telephone survey, and an online version for those who are also asked to complete the online section.  

If no phone number is provided, an interviewer will send a postcard about a week later with their work mobile phone number, which gives an alternative method of contact for respondents to text or call.

This project involved testing the advance materials (letters / leaflets / postcard) for the survey, the process used to gain respondent phone numbers, and the process for accessing and completing the online section of the survey.

We sought to understand whether the advance materials and the process for providing phone numbers:

• were clear enough and understood by all participants;
• foster positive feelings about participating in the survey; and
• help to maximise response rates and minimise bias across all participant groups

For the online section, we aimed to:

• understand whether it was easy to access
• once accessed, understand whether the online section was easy to use

Also key to the testing was understanding what the motivations and barriers are for completing the survey via telephone and online.

As a result of this testing, detailed findings are provided based on the issues identified with the survey recruitment process and online section as well as recommendations on how best to address the issues.

As a first step in the process, Welsh Government provided ORS with the materials used to recruit participants to the National Survey. ORS then held an internal workshop with the project team and reviewed the materials.

This review resulted in feedback and recommendations, which we discussed with Welsh Government.

29 cognitive interviews were then held over Microsoft Teams or Zoom with participants across Wales, in three stages between 4 August and 29 October 2021. There were 10 interviews in each of the two stages of materials testing (original advance materials; revised advance materials), and nine in the Online section testing stage.

The primary selection criteria for the interviews were that participants should be: living in Wales and therefore eligible to be selected for the National Survey; from a range of geographical areas (North, Mid, South East and South West); and representative of a range of ages (18+). Three participants were from ethnic minority backgrounds, including one asylum seeker. Two participants considered themselves to be disabled.

Participants were recruited via telephone or email. They had taken part in research for ORS in the past and had agreed to be recontacted for other research purposes. We also worked with the Ethnic Minorities and Youth Support Team Wales (EYST) to recruit participants from ethnic minority backgrounds. Each participant was given a £30 gift voucher as a thank you for their time and effort in taking part.

Participants were sent a pack of materials in the post prior to their research interview, containing the documents to be reviewed.

Prior to examining the recruitment materials, interviewees were asked a series of questions about their general motivations for completing surveys. For Stage 1 and 2 participants this included the initial letter (standard and online versions), the standard leaflet, and the reminder postcard. Participants in the online section testing received the online letter and leaflet only.

All participants were asked to review the documents and give their views. Stage 2 participants were also asked to compare the original versions with the ones that had been edited after stage 1 and say which elements of the versions they preferred. Discussions focussed on their initial impressions, the tone and order of the text, the branding and whether they would feel persuaded to take part as a result of reading them, or indeed if anything about the materials would put them off.

In addition, Stage 1 participants were observed logging on to the website portal used for gathering contact information and were asked about ease of access and any improvements that could be made.

Participants in the online section testing were given time to log in and complete the online element of the National Survey. They were asked for feedback on the process of accessing and navigating through the online section, what worked well and what didn’t work so well, and if there were any issues or glitches that needed addressing.

Survey completion motivations and habits

Perhaps unsurprisingly, as this cohort of participants had agreed to take part in this research, there was a general openness to completing surveys. The most important deciding factor in whether most would complete a survey, or not, was time. For those who work, in particular, if a request to participate in a survey comes in the daytime, participants reported that the request is likely to be pushed aside to a later date, forgotten about, or ignored.

Most participants said that they would complete a survey on a topic that was of interest to them and/or is something they deem to have importance in their lives or which will impact on their local area. Surveys on local issues were seen by most participants as important and would be prioritised for completion – participants stated that anything that might impact local service delivery, including health and education, would be a particular priority for them.

Participants also placed importance on ‘official’ surveys from the Government, local authorities, or the NHS and stated they are more likely to complete these, with some saying they felt they had a ‘civic duty’ to do so. All participants had completed the Census and said they considered it important.

Stage 1: recruitment material review

Having reviewed the evidence from the Stage 1 research interviews, we recommended the following changes be made to the documents and to the portal for submitting contact details.

Recommended changes

• The opening paragraph of the standard letter should be extended to include more information about the content of the survey and why people should participate.
• The middle information box on both versions of the letter should look more like the one in the Online letter and include ‘Step 1, Step 2, Step 3’ headings.
• The wording in the line relating to conducting a ‘socially-distanced visit’ should be softened. If this is not possible, it should be located towards the bottom of the letters and not in the main instruction box.
• The leaflet should be referenced in the letter, to draw attention to it.
• The cover image in the leaflets should be changed to reflect the purpose of the survey.
• More infographics should be included in the leaflet.
• The order of the text in the leaflet should be reviewed to make the flow more logical.
• A title such as ‘Important Information’ should be added to the leaflet.
• The wording of the postcard should be reviewed to ensure the tone is more personal.
• The responsiveness of the platform should be reviewed to ensure participants are able to log in quickly and easily.
• When participants are prompted to input their access code into the three boxes on the portal, the cursor should automatically skip to the next box once four digits are inputted.
• Change the ‘save and continue’ button to ‘enter and continue’ on the portal opening page.

Stage 2: recruitment material review

Most participants agreed that they preferred the overall look of the revised standard letter, which had a box with red shading in the middle and the three steps split into further boxes within it. It was felt that this was more eye-catching, and the clearly-defined steps were welcomed. A majority also preferred the tone and structure of the edited version compared with the original. Most participants were also in favour of having more information in the opening paragraph explaining the purpose of the survey and the types of topics included.

As at Stage 1, several participants picked up on the sentence “Please provide your phone number within 7 days. If we don’t hear from you, we may make a brief, socially-distanced visit to collect your phone number”, generally commenting that it was too strong in tone. However, it was considered important to keep a focus on the seven-day time limit to provide details.

As in the case of the standard letter, most participants preferred the revised version of the online letter in terms of look, tone, and layout. In particular, the sentence “There is also a short online section at the end for anyone who uses the internet” was well received. It was considered more positive than the original “let the interviewer know if you don’t use the internet” and reassuring for those who are not digitally engaged.

All participants felt that there was enough information in the letter to indicate that this is a telephone survey with an additional online element, but many missed the detail about the online element the first time they read the letter. They were in agreement that this could be highlighted further to help participants understand the requirement.

Overall, the revised version of the standard leaflet was preferred by participants to the original version, although this finding was not as clear-cut as in the case of the letters. Although most preferred the infographics as a concept for the cover and considered this a more logical choice, many felt there was something ‘missing’ or that needed to be refined about it. Although some preferred the image of the castle aesthetically, most felt that it gave the impression of a tourism leaflet, which would not draw people to read it and may miss important information as a result.

In terms of the content of the leaflet, participants generally preferred the question-and-answer format adopted in the newly revised version. However, a few said that there could be a little more of a general introductory paragraph followed by the title ‘Frequently Asked Questions’, which they considered more logical.

A particular omission mentioned by a few participants was information on participating in the survey if you have additional needs for example, a hearing issue; or if English (or Welsh) is not your first language. Although it was recognised someone might call the helpline if they needed further information on accessibility, having the information on the leaflet would improve the feeling of inclusivity.

As in Stage 1, participants were generally supportive of the tone of the postcard in its original and edited form and did not have as much feedback to provide as on the other materials. Although the differences between the original and edited versions were small, participants generally preferred the revised version, considering it more friendly and ‘to the point’.

Participants tended to like the fact it contains direct contact details for an interviewer and generally saw the approach as more personal. A few felt it could be even more personable by including a statement such as “you can also contact me with any questions you have about the survey” if practical to do so.

In terms of practicality, all participants said they would be motivated to take part and would log in to give their details if the materials arrived through the post. It was not felt that the materials appeared suspicious at all and it was felt that the materials tie together well in the way they are branded, and the colours used were liked.

Most participants considered the length of the survey to be longer than they would like it to be, particularly when factoring in the online element. There was a general consensus that 20 minutes is about right with 30 minutes as an upper limit. However, as the materials make the length of the survey clear, it was felt people knowingly give their details while understanding the time commitment and therefore would put the time aside. There were no participants who said the length of the survey would put them off taking part completely.

Stage 3: testing of the online element

Based on the letter and leaflet, all participants would be happy to take part in the online section. There was nothing of note that would put anyone off and it was not felt by anyone that the materials appeared untrustworthy or not genuine.

Participants were unanimous in the view that the process of accessing the survey, and indeed navigating through the questions, was straightforward and felt the platform worked well. All felt it was obvious what needed doing when logging on and felt the survey was user-friendly, the questions were well laid out and well-spaced. There weren’t any glitches reported and no participants reported that any of the pages had been slow to load.

Participants that completed the survey on a computer and smartphone agreed that a clickable link would be the easiest solution for accessing the survey online immediately after the telephone section. It was felt this would improve accessibility for those less confident in using technology and the internet. Although participants did not have a significant issue with the 12-digit code, it was felt a clickable link would streamline the process. There was also some support for a shorter code which could be copied and pasted as one, but the link would make things a step easier again.

There was also support for having a progress bar on the screen, something participants reported as being useful in other surveys they’ve completed.

Based on the feedback at Stage 2 of the materials testing and the testing of the online survey element, we have made the following recommendations.

Standard letter

• The look of the revised letter should be retained, particularly the design of the central information box.
• The opening paragraph should be reviewed, it must contain as much up-front information as possible on the themes covered in the survey and who will use the data. Consideration should also be given to the look of this paragraph, in particular if there is a way to make it look ‘cleaner’.
• Consideration should be given to the sentence “please provide your phone number within 7 days. If we don’t hear from you, we may make a brief, socially-distanced visit to collect your phone number” to soften the tone for the reader. One option could be including “please provide your phone number within 7 days” in the main information box and using an asterisk or footnote to include information about the socially-distanced visit in small print.
• The Welsh translation of “the survey takes place by telephone, so please let us know your phone number” should be reviewed to ensure it matches the tone of the English version.
• Including a QR code should be considered to make accessing the portal as easy as possible.

Online letter

• The overall look and branding used in the revised version should remain.
• Another ‘step’ should be added in the central information box to indicate the online section of questions – this would improve understanding of what is entailed.

Standard and online leaflet

• Further work is required to refine the cover but the approach of including infographics is preferable. Considerations should include adding a background colour, ensuring the same style is applied to all the text, and possibly including more icons or infographics where possible.
• The title ‘Frequently Asked Questions’ should be placed below the introductory paragraph, but the Q&A style adopted thereafter should remain.
• The phrase “we value your contribution” should be included in the leaflet somewhere.
• The phrase “so we need your phone number” should be changed to “please provide your phone number”.
• Consideration should be given to including text about support provided for people with additional needs to complete the survey.
• Further consideration should be given to improving clarity around the selection of a person from a household to participate.

Postcard

• The text included in the revised version should remain.
• Consideration should be given to adding an additional sentence to enable the participant to contact the interviewer for further survey information, if appropriate.

Online survey platform

• Ease of use on a smartphone should be considered – the adoption of a unique clickable link to access the survey from a text message or email should be considered.
• If a unique access link approach is not adopted, the use of a shorter access code should be considered to allow participants to copy and paste the code into the survey login page.
• A progress bar should be added to improve survey navigation.

Report Authors: Angharad Davies, Opinion Research Services (ORS) Ltd.

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:
Chris McGowan
Email: surveys@gov.wales

Social research number: 11/2022

Image

Digital ISBN 978-1-80391-651-4