Strategic framework for Welsh Government surveys

This document provides a strategic framework within which the Welsh Government makes its decisions on the need for statistical surveys and how they are carried out.

The key aims of this strategy are:

• to ensure that the Welsh Government’s needs for information are met by the most appropriate and cost effective mechanism, whether that be by surveys or other existing data sources
• to help ensure the information collected via surveys is fit for purpose and then utilised as fully as possible
• to improve the way we run and commission surveys to ensure value for money, data quality and data security and to minimise the burden on respondents
• to promote good survey practice and raise the standard of the surveys we fund in Wales

This document summarises the survey landscape in Wales:

• the types of information statistical surveys collect
• how we use this information
• how we prioritise new survey requirements
• how we are improving the quality of surveys and ensuring value for money
• how we are working with users to help them understand and utilise the information available
• the work we are undertaking to make more use of existing data sources

Statistical surveys are used to gather evidence to inform decision-making in a number of ways. With an increasingly distinctive Welsh policy approach it is vital that we have a robust, Wales-specific evidence base. They can help to:

• better understand an issue/problem
• identify and analyse long term trends
• provide specific information or evidence
• examine and compare the merits of competing options
• evaluate the impact of policies and programmes

The types of information that can be collected through surveys.

• Factual questions: surveys often offer the only realistic way to estimate values for large groups, for example, assessments of their health status, life satisfaction and so on.
• Knowledge questions: assess what respondents know about a particular topic.
• Attitudinal questions: seek to measure respondents' opinions, beliefs, values and feelings. These typically cannot be obtained by other means for example, observation or administrative data.
• Behavioural questions: measure what people do or intend to do and how that relates to particular interventions.
• Preference questions: measure respondents’ preferences for different possible options and outcomes, including trade-offs between competing policy objectives. These can be used to elicit monetary values for different outcomes, including those not readily possessing market prices (for example, changes in air quality, health status) for use in cost-benefit analyses.

There is often no other practical, affordable, and reliable way of gathering such information but a key function of our survey advice process is to identify if other sources of information could be used (see Section 4).

The Welsh Government is committed to evidence-based policymaking. Evidence comes in many forms, each with a different role in supporting decision-making; and survey data forms a part of that overall evidence base. A key challenge for all types of evidence is ensuring that it is used effectively throughout policy development, delivery and review.

Many of the intended outcomes of actions taken by the Welsh Government can only be evidenced through the use of social surveys. For example:

• public’s perceptions of anti-social behaviour and prevalence of problematic drug misuse are monitored by the Crime Survey for England and Wales
• views on access to and satisfaction with public services such as health and education are obtained from the National Survey for Wales
• the Welsh Bus Passenger Survey provides data on satisfaction with bus services in rural areas

A number of surveys have influenced policy and delivery through providing data to plan interventions, set baselines, monitor progress, and support evaluations. One example is the Labour Force Survey (LFS), which has a large sample in Wales (around 20,000 households), and focuses on labour market activity and qualifications. Analysis of information from the LFS has led to policies designed to enhance skills and was central to the design of the Want2Work programme. With its large sample and therefore local-level accuracy, the survey has been used as part of the evidence base when considering applications for grants to support businesses in particular locations.

The surveys that are conducted in Wales can be categorised, as follows:

i) Regular surveys run by the Welsh Government
ii) Ad-hoc surveys run by the Welsh Government
iii) Surveys run by other government departments but supported financially by the Welsh Government
iv) surveys carried out in Wales, by other public bodies

An up-to-date list is published each year by the Office for National Statistics.

i) Regular surveys run by the Welsh Government and Welsh Government sponsored bodies

Regular surveys are conducted by the Welsh Government where there is a need for data to monitor progress on an ongoing basis. Examples of regular statistical surveys carried out by the Welsh Government are shown below. In 2016-17, the National Survey and the Welsh Health Survey were brought together into a single survey along with three other large-scale surveys run by Welsh Government Sponsored Bodies: the Active Adults Survey, the Welsh Outdoor Recreation Survey, and the Arts in Wales Survey. More information on this single survey is available on the National Survey web pages.

ii) Ad-hoc surveys run by the Welsh Government

These surveys are typically carried out to answer a specific research question, sometimes as part of a wider evaluation study. It is important that they are subject to the same scrutiny and quality assurance as regular surveys to ensure that they are fit-for-purpose.

iii) Surveys run by other government departments but supported financially by the Welsh Government

The Welsh Government is fortunate to have access to good comparative data with the rest of the UK through large UK or GB surveys. Financial contributions are made to existing surveys for two reasons:

• to improve the quality of the information for Wales by a boost in sample size (an example being the Labour Force Survey)
• and/or as part of a cross-administration / department consortium set up to fund a survey covering a cross cutting topic - this is often the case in the case of longitudinal studies such as Understanding Society

This can provide significant cost savings where our needs closely align to UK needs as well as providing a degree of influence over the content and conduct of these surveys.

When deciding on the means to deliver a survey the potential to piggyback on an existing survey is an important consideration. The potential cost savings are weighed up against the ability to precisely define the survey questions as there is less flexibility in the case of an existing survey.

iv) Surveys carried out in Wales, by other public bodies

A number of other surveys are carried out in Wales by UK government departments and public bodies in Wales. These are both a potential source of information as well as potential candidates for a boosted sample if that is an effective way to meet Welsh Government needs. However, many UK wide surveys in this category have a very small Welsh sample only designed to provide UK level results and as such separate results may not be available for Wales and the costs of boosting the sample could also be high.

Where it has been shown that a survey is the best way to gather information then another key function of our survey advice process (see Section 4) is to identify what survey mechanism is needed, for example:

• use of existing survey data
• boost an existing survey or add new questions
• carry out a one-off Wales specific survey
• carry out an ongoing survey specific to Wales

Before a survey is commissioned a number of steps need to be taken to ensure the right decision is made:

• checking whether the information required is already available
• checking whether the proposed approach for obtaining the information is the best way doing so
• ensuring that the appropriate professional advice is provided to assist with questionnaire and survey design, management and reporting

The survey advice and approval service within Welsh Government provides this support to officials both within Welsh Government and its sponsored bodies.

Checking whether the information required is already available

Good quality large-scale surveys are costly, especially face-to-face surveys. To ensure that we get value for money, it is important that surveys are only used to collect information when it cannot be obtained from elsewhere. A record of existing surveys is maintained to help inform this decision as well as relying on the expertise of the wider analytical community within the Welsh Government to advise on other data sources.

Checking whether the proposed approach for obtaining the information is the best way doing so

Once it has been agreed that a survey is required, the most cost-effective way of carrying it out must be identified. This could be by adding questions to or increasing the sample size of an existing Welsh Government or other survey, or commissioning a brand new survey. To ensure value for money the sample size must be sufficient but not excessive. Care needs to be taken to ensure that the information is collected from the smallest sample that allows the necessary level of detail for the required analysis.

One example of how we consider approaches to collecting survey data is in the options report that underpinned a decision to bring several surveys together into a single survey from 2016-17 onwards.

Ensuring that the appropriate professional advice is provided to assist with questionnaire and survey design, management and reporting

In many cases the same or similar questions have been asked before and are available from the National Statistics harmonised question set or from other question banks. The support provided to those developing surveys includes identifying such questions as well as guidance on how new questions could be developed. Statistical surveys fall within the definition of official statistics and this sets out our reporting standards, as does the complementary GSR publication protocols.

The Welsh Government works closely with its sponsored bodies (WGSBs) and other public sector bodies and government departments to improve the quality of its surveys and to maximise their use.

As noted above, in 2016-17 several surveys previously run separately by the Welsh Government and three Welsh Government Sponsored Bodies were brought into one survey.

We keep in regular contact with our colleagues managing surveys across government as well as participating in cross-government steering groups such as the National Statistics Harmonisation Committee.

The Survey Advice and Approval Service (see section 4) includes Welsh Government Sponsored Bodies. We also keep in touch with local authorities, fire, police and health bodies as well as the Welsh Council for Voluntary Action through our involvement with the Welsh Statistical Liaison Committee. This allows us to explore opportunities for collaboration on survey research in Wales.

We also ensure that external expertise is used to support our survey activities, for example by:

• including research academics as members of the National Survey for Wales technical advisory group
• commissioning methodology advice from the Office for National Statistics or independent survey experts

When commissioning surveys we carefully consider what information is needed and how we will analyse the results to meet these needs. Statistical outputs or research reports are made available in as much detail as is reliable as soon as is practicable on our Statistics and Research pages. Many surveys also have data published via StatsWales.

For regular surveys the data is made available on the UK Data Archive so that other researchers can access and use the data. Survey data can also be made available on request for specific purposes, subject to a data access agreement, by contacting the Welsh Government.

Making data available as statistical tabulations or for research purposes is just one of the ways to support the usability of survey data. An increasing number of surveys make provision to link survey responses to administrative data (although respondents are provided with the opportunity to opt out of doing this). Linking data can mean that surveys need to collect less data as it is already available elsewhere, and that the strengths of both survey and administrative data can be realised by combining data sets. The Office for National Statistics is currently, through its Census Transformation Project, investigating and assessing alternative options for producing population and socio-demographic data. Linking administrative data may itself reduce the need for surveys in future.

The Code of Practice for Statistics required us to report annually the estimated costs (for example, on businesses, service providers, or the public) of responding to statistical surveys and strive to develop methods that will reduce the costs to individual organisations or people.

To comply with this, the survey advice and approval service keeps a record of all regular and ad-hoc surveys that are conducted by the Welsh Government or its sponsored bodies.

This financial cost of responding to our surveys is calculated and recorded in the Online List of Government Statistical Surveys published by the Office for National Statistics.

For further information please contact:

Survey Advice and Approval
Knowledge and Analytical Services
Welsh Government
Cathays Park
Cardiff
CF10 3NQ

Email: surveysadvice@gov.wales

Social research number: 3/2018

Digital ISBN: 978-1-78903-389-2

Image