Identifying potential discontinuities in the new National Survey for Wales (summary)

A review of the way social surveys were carried out in Wales concluded that a viable approach was to bring together the five surveys into a single National Survey, and this is the approach that is being followed from 2016-17.

The new survey is longer than its predecessor surveys, and is carried out using a methodology which is different from that for some of the predecessors. This work was commissioned to investigate whether the new survey would produce different results, and so to provide information on what the change means for understanding trends over time. The project compared key results from a large-scale test of the new National Survey (carried out in summer 2015) with results for the same topics from the predecessor surveys.

The differences in the method used for the new survey compared with the previous surveys can lead to differences in the estimates. Some of the possible ways this can happen are through changes in the mode (the new survey is all interviewer-led), through other questionnaire differences such as question order effects, and through fatigue effects from the longer questionnaire.

For the purposes of this study we have focused on larger differences, of more than 5 percentage points, between the old and new surveys, and identified these as potential discontinuities. However, the sampling errors mean that is not always possible to assess whether these identified potential discontinuities are real, though they do provide helpful evidence. Smaller discontinuities may be present, and may also be important for users. The sampling variation, particularly in the estimates from the large-scale test which has a smaller sample size than the original surveys, makes it generally more difficult to identify smaller discontinuities.

For national level estimates and breakdowns into a small number of categories, direct estimation of any differences is satisfactory. For more detailed breakdowns the sample sizes are too small to produce sufficiently precise results. In these cases a small area estimation method can be used to produce more precise estimates, and in this study this approach was applied to the Welsh Health Survey as an example.

Nine variables (some with multiple possible outcomes) were investigated at national level from the National Survey for Wales, including wellbeing, Welsh language ability and internet access. Five of these were analysed at local authority level. Across all of these analyses there were very few discontinuity estimates which were significantly different from zero, and it seems likely that these have arisen by chance because of the random sampling. Therefore this project did not find any discontinuities between the variables selected from the previous design of the National Survey for Wales (from 2012 to 2015) and the new survey.

Nine variables were selected from the Welsh Health Survey and discontinuities estimated at the national level. Three of these (proportions of respondents drinking above the guidelines, drinking twice above the guidelines, and proportion of respondents eating five or more portions of fruit and vegetables a day) had lower estimates than the Welsh Health Survey. Detailed breakdowns by Health Board, gender and age show similar discontinuities for the same variables in some age groups. At the most detailed levels the variances of the estimates of discontinuities are large, so it is difficult to make an assessment of whether the observed differences arise as a result of sampling error or not. The use of small area estimates in these situations provides a much smaller sampling error. These results suggest that small area estimation is useful here. We found that there is a general issue with lower numbers of respondents in the large-scale test reporting high alcohol consumption (which may possibly be the effect of having a face-to-face survey) and eating 5 or more portions of fruit and vegetables. Some of the observed differences are in directions other than that which might be expected, and there is a general tendency for any discontinuities to arise from lower values on the new National Survey (rather than higher values).

Eighteen variables from the Active Adults Survey were analysed, including a range of participatory activities. The Hooked on Sport variable showed a discontinuity with the largescale test showing lower values. There were examples of discontinuities in both directions among the other variables.

For the Arts In Wales Survey, six variables were investigated, and four showed discontinuities at the national level. The large-scale test had lower estimates for participation variables, and a higher estimate for one barrier variable. The confidence intervals for the discontinuity estimates by local authority are particularly wide for this survey, so rather few such estimates were significantly different from zero.

Six variables were selected from the Welsh Outdoor Recreation Survey, and five of them showed discontinuities at the national level. No local authority identifier was available with this dataset, but the patterns persist nationally in gender and broad age categories.

Where discontinuities were identified, they more often suggested that for those particular variables the new National Survey will provide lower estimates than did the predecessor surveys. This project focusses on identifying and quantifying discontinuities, rather than exploring why these differences have occurred. The effects of changes in mode and in question order are possible explanations, but further investigation would be needed to identify candidate causes in each particular example.

The discontinuity estimates provide one piece of evidence about the differences between the original suite of surveys and the new National Survey. Under some strong assumptions they may provide a basis for adjustment, but are likely to be more useful as supporting information to help with the interpretation of differences. As new data accumulate from the new National Survey, it should be possible to refine the estimates using alternative models, and the full report contains information on how this might be done.

Paul A. Smith, Nikos Tzavidis, Timo Schmid, Natalia Rojas, Jan van den Brakel

The full report is available on request.

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

Chris McGowan
surveys@gov.wales

Social research number: 33/2017

Digital ISBN 978-1-4734-9488-6