Trade Survey for Wales: quality report

The Trade Survey for Wales (TSW) provides experimental statistics on the trade conducted by businesses located in Wales. It collects data relating to the annual trade businesses in Wales conduct with other parts of the UK, the EU and the rest of the world (ROW).

The results of the survey are ‘Experimental Statistics’ as the method employed is still in development and there are some issues with data quality. The volatility of estimates are assessed, with ways to further improve their quality explored. However, the data and analysis are still of value provided that users view them in the context of the data quality information provided.

Experimental statistics are published with a view to improving their quality over time through feedback from users and stakeholders. The TSW estimates will remain experimental while development work continues to improve their coverage and quality.

Limitations which have led to these statistics being badged as experimental are listed in the Strengths and limitations section.

Detailed results from each year of the survey can be found in the annual releases, along with a comparability and coherence section which compare the results to the equivalent estimates of Welsh trade produced by HM Revenue & Customs (HMRC) and the Office for National Statistics (ONS).

The Welsh Government’s Trade Policy: The Issues for Wales (2018) called for improved evidence to underpin trade policy decisions, and highlighted gaps in the trade data for Wales as well as potential methodological issues identified with existing data sources. There was no existing data on intra-UK trade from Wales, and limited data available according to business characteristics (such as size and sector). Much of the existing Wales data on international trade in goods were modelled estimates based on apportioned employment numbers from the UK, rather than direct data collections. This meant that the data for Wales at the time may not have accurately reflected the specific export markets of Welsh business units.

HMRC administer current trade statistics on the movement of goods, sourced from customs import and export entries for non-EU trade and otherwise from the EU-wide Intrastat system. The HMRC data does not include trade in services. ONS administer an Annual Survey of International Trade in Services (ITIS) that produces experimental statistics at the Wales level.

Given the limitations with the existing data TSW was piloted in 2019 with the aim of capturing robust information directly from businesses on trade flows to and from Wales.

The statistics produced from the TSW have provided Welsh Government policy makers an improved evidence with which to inform Ministers and base their decisions. The intra-UK trade figures have been particularly useful in understanding the UK internal market. The Welsh Government’s Export Action Plan also draws on the Trade Survey for Wales as part of its evidence base. The survey also gives Welsh Government the ability to undertake follow up research with respondents agreeing to re-contact, allowing further insight into key policy questions highlighted in the results.

Strengths

• The TSW is the only direct collection of trade data specific to business operations located in Wales. Businesses directly report the activity of their Welsh locations. This contrasts with existing regional statistics which collect UK level responses from which Wales level results are modelled.
• TSW is the only trade survey collecting intra UK trade data, broken down by all UK nations, by businesses in Wales.
• TSW results are the only source of intra UK trade conducted by businesses in Wales.
• Results are broken down by business size.
• Results are broken down by SIC classifications.
• Goods and services trade are collected on a consistent basis.
• The methodology allows estimates of the total trade (goods and services) businesses in Wales make with all parts of the UK, the European Union (EU) and the ROW.

Limitations                  

• As a voluntary and online survey there are low response rates which affects the ability to produce some lower breakdowns.
• Variability in the achieved sample across years affects the ability to make robust comparisons across years. The results report presents high-level comparisons between years. Breakdowns that are more granular have shown greater variability, specific comparisons are discussed in the comparability and coherence sections of the results reports.
• Incomplete coverage of businesses, sectors and total Welsh trade means the estimates do not cover all trade activity in Wales and should not be used to work out a trade balance, nor compared to total GVA/GDP. For example, only business purchases are covered, direct to consumer purchases are missing. This means Welsh consumer spending abroad (for example, tourism imports) and consumer spending in other parts of the UK has not been captured, whereas the expenditure of tourists buying from businesses in Wales is captured.
• Some respondents provided estimated values in their responses so the values provided will have varying degrees of reliability.
• Feedback suggests that some businesses found it difficult to allocate sales to customers based in other parts of the UK.
• Respondents were asked to allocate trade based on their immediate partner in the supply chain, not the original source or final destination of a purchase or sale. The movement of goods through supply chains within the UK may therefore mask an international origin or final destination in the TSW results, which may be of particular significance for the retail sector using distribution hubs within the UK.
• Within the analysis there are a number of instances where ‘unallocated’ sales or purchases data is highlighted, where the respondent was unable to allocate to a specific destination. This may have led to underestimates of the overall figures.

TSW is a voluntary sample survey of businesses. The sample is selected from the ONS Inter-Departmental Business Register (IDBR).

The sample frame includes businesses in the UK with local units in Wales. Stratified sampling takes place, with a census approach taken for larger businesses specific of the approach taken each year can be found in the technical reports which accompany results publications.

The survey is conducted using a push-to-web approach. Sampled businesses are initially sent a bilingual (Welsh and English) postal invitation letter encouraging them to go online and complete the TSW.

Welsh Government works with a contractor to deliver the survey. This includes contacting the sample, administering the online collection and verification of data and analysing responses to produce Wales level estimates.

The survey asks respondents to provide data on a calendar year basis, the first wave collected 2 years of data (2017 and 2018) with subsequent waves collecting a single calendar year of data.

Reason for method selected

Cognitive testing was conducted with 32 businesses ahead of TSW year 1. Feedback from businesses informed question development and the collection mode used.

A push-to-web approach was selected as the most appropriate to meet the aims and objectives of the TSW given the survey’s scale. The IDBR has complete records of business addresses, ensuring the TSW sample coverage consistent with other businesses surveys. However, the IDBR contains limited email addresses, so an initial postal mail-out letter contacting businesses and inviting them to respond to the online questionnaire is necessary.

Further details of the cognitive testing and development of TSW year one can be found in TSW 2018: technical report.

What it measures

The TSW collects data on the annual value of sales and purchases made by business units located in Wales. The sales and purchases are broken down by type (goods or services) and the geographical location of the trade partner. The questions cover both intra UK and international trade. Survey responses allow statistics to be produced covering the value of trade for businesses in Wales and the number of businesses conducting this trade.

Mode

Push-to-web: The survey questionnaire is completed online, with the sample invited to take part through an initial postal mail out with further reminder letters and telephone chasing.

Frequency

To date the survey has been run annually. TSW launched in November 2019, collecting annual data for 2017 and 2018. Subsequent waves collect a single calendar year of data.

Sample size

8000 businesses are selected for the sample. The survey is voluntary and in the first two years obtained overall response rates of 14% and 16% respectively.

Sample frame

The sample is selected from the ONS IDBR that lists businesses from either VAT or PAYE records. The sample frame is limited to reporting units that have local units in Wales.

Businesses with employment of fewer than 3 are excluded, along with the list of sectors below. The technical report accompanying annual results gives further details of the sample in each year.

Excluded SIC Sections

• O: public administration and defence; compulsory social security.
• Q: human health and social work activities.
• T: activities of households as employers; undifferentiated goods-and services-producing activities of households for own use.
• U: activities of extraterritorial organisations and bodies.

UK Standard Industrial Classification of Economic Activities

Further to the SIC sections list above, a number of lower SIC classifications are also excluded;

• 06 Extraction of crude petroleum and natural gas
• 84 Public administration and defence; compulsory social security
• 86 Human health activities
• 87 Residential care activities
• 88 Social work activities without accommodation
• 93 Sports activities and amusement and recreation activities
• 94 Activities of membership organisations
• 96 Other personal service activities
• 97 Activities of households as employers of domestic personnel
• 98 Undifferentiated goods and services producing activities of private households for own use
• 99 Activities of extraterritorial organizations and bodies
• 642 Activities of holding companies
• 791 ravel agency and tour operator activities
• 851 Pre-primary education
• 852 Primary education
• 8299 Other business support service activities n.e.c.
• 9101 Library and archive activities
• 03110 Marine fishing
• 47799 Retail sale of other second-hand goods in stores (not incl. antiques)
• 49320 Taxi firms
• 56290 Other food services
• 56301 Licenced clubs
• 64110 Activities of central bank
• 79901 Activities of tourist guides
• 79909 Other reservation service activities n.e.c.

Sample design

Stratified sampling takes place at the reporting unit level. The structure of the sample is reviewed each year, as development of these experimental statistics continues. The strata and number of records selected each year are provided in the technical reports.

An overarching aim of the sample is to capture as large a proportion of the total Welsh economy (business turnover) as possible. To date the sample has selected 8000 reporting units, with a focus on larger businesses. The sample each year has included all businesses with employment of 20 or more. Businesses with 3-19 employment are randomly selected based on further strata, with the sample allocated amongst SIC sections based on turnover proportions. Sectors with higher total turnover are allocated a greater proportion of the sample.

Analysis

Business size bands

Throughout the data processing and analysis of TSW, businesses are grouped into categories based on UK employee values (business size). Four size bands are used in the analysis. However, only three size bands (Small which encompasses micro businesses, medium and large) have been used for the purposes of reporting.

UK employee values are used in order to ensure base sizes are large enough for analysis, improve comparability to HMRC and ONS data, and due to anecdotal evidence from cognitive testing that most businesses make trade decisions based on their overall UK operations.

Source: Inter-Departmental Business Register (IDBR)

Sector groupings

The IDBR provides a SIC code for each business in TSW. These have been grouped to ensure base sizes are large enough to analyse. 

Standard six sector groupings and sections

Business and Other Services

SIC section:

• Information and communication
• Financial and insurance activities
• Real estate activities
• Professional, scientific and technical activities
• Administrative and support service activities
• Arts, entertainment and recreation
• Other service activities

Construction

SIC section:

• Construction

Manufacturing

SIC section:

• Manufacturing

Primary Sector and Utilities

SIC section:

• Agriculture, forestry and fishing
• Mining and quarrying
• Electricity, gas, steam and air conditioning supply
• Water supply; sewerage, waste management and remediation activities

Trade, Accommodation and Transport

SIC section:

• Wholesale and retail trade; repair of motor vehicles and motorcycles
• Transportation and storage
• Accommodation and food service activities

Non-market Services

SIC section:

• Education
• Human health and social work activities (excluded)
• Public administration and defence; compulsory social security (excluded)

Non-market services are combined with business and other services due to low base sizes. Resulting in the following five categories being used throughout the data processing and analysis of TSW.

1. Business and Other Services
2. Construction
3. Manufacturing
4. Primary Sector and Utilities
5. Trade, Accommodation and Transport

Weighting

Response data from the survey is weighted and grossed to produce Wales level estimates. Turnover and unit weights are applied to produce estimates of the value of trade and the number of businesses engaging in trade.

The weighting of results takes account of the unequal selection probabilities of businesses and the effect of non-response on the achieved sample.

As development work continues for these experimental statistics the sampling and weighting approach will be reviewed each year, details of the weighting approach each year can be found in the technical report.

Imputation

Imputation is carried out to model missing values where respondents have provided partial information. This is based on other values (responses) provided within the survey. Average proportions of sales/purchases breakdowns, amongst similar businesses within the survey are calculated. The averages are applied to the relevant total figures provided by similar respondents to populate their missing answers.

Non-response error

The largest limitation on the accuracy of results is the non-response error. Response rates for online surveys are known to be typically low compared with other, more costly, methods of data collection such as telephone or face-to-face. Further, the TSW is a voluntary survey, while similar surveys conducted by the ONS are mandatory leading to response rates of some 60% or more.

Non-response leads to reduced base sizes, therefore breakdowns of the Wales level estimates are only provided where base sizes allow.

Work is underway to incorporate other data sources to compensate for non-responses and improve the accuracy of estimates.

Sampling error

Sampling error arises because the estimates are based on a random sample of the population rather than the whole population. The results obtained for any single random sample are likely to vary by chance from the results that would be obtained if the whole population was surveyed (i.e. a census), and this variation is known as the sampling error. In general, the smaller the sample size the larger the potential sampling error.

The sample design for TSW aims to minimise this by taking a census approach to the largest businesses (by employment) and focusing the remaining sample allocation to the industrial sectors with the largest turnover.

The sampling error will be explored further as part of the as part of the development of these experimental statistics.

Missing answers

Missing answers occur for several reasons, including refusal or inability to answer a particular question. Routing within the survey prevents respondents seeing questions that are not applicable. Imputation is used to model some missing responses.

The results include an ‘unallocated’ category which is a result of respondents partially completing the questionnaire, for example; a business providing the value of total goods sales to the UK, but unable to provide a breakdown of sales to the separate UK nations. Although this limits the usefulness of some results, understanding the difficulty respondents find in answering particular questions is important while development work continues.

Incorrect answers

A potential cause of bias is respondent error when providing answers. The TSW online survey has multiple checks in place to prevent this. The online survey allows automatic checks to be conducted as the respondent is completing the questionnaire and flag questionable. Validation checks include:

• logic checks that confirm breakdowns sum to the totals provided
• providing respondents a summary of answers given, in numeric and written form, overcoming potential confusion when inputting large values

Where responses fail an automatic checks the respondent is asked to review and correct the response or confirm it is correct (with the option of providing a reason).

Incorrect answers are also prevented by routing respondents to applicable questions only, based on responses to filter questions.

Final validation

Automatic validation checks are run on each submitted survey response to flag any remaining discrepancies which would invalidate the response. Flagged responses are reviewed by the contractor, data is amended if necessary; through a combination of desk-based research and respondent call-backs where consented to.

The contractor is also required to carry out other quality assurance checks on the data once fieldwork had finished. This included looking at the comments provided within the survey to assess whether any respondents had flagged particular issues completing the survey which might impact the reliability of their responses.

Quality assurance

Welsh Government statisticians perform final quality assurance checks on the response data and results files provided by the contractor. All published statistics undergo quality assurance in line with GSS Quality statistics in government guidance.

Each wave of the TSW sample is drawn from an annual extract of the IDBR. The IDBR is updated using administrative and survey-based data sources with a variety of reference dates which means that business data could have been updated at any point up to the date of extract. This means that the business characteristic data used in sampling and weighting for the TSW is only as up-to-date as the latest version of the IDBR.

TSW fieldwork commences in Autumn, collecting data for the previous calendar year. The results are then analysed and published the following year, for example; 2018 data was collected in 2019 and results were published in 2020.

Data

All TSW data is provided in open format accessible spreadsheets.

Disclosure control

We take care to ensure that individual businesses are not identifiable from the published results. We follow the requirements for confidentiality and data access set out in the Code of Practice for Statistics. Respondents are directed towards the TSW: privacy notice for further information about the security and confidentiality of the data they provide.

Language requirements

Welsh language standards are adhered to for TSW data collections and outputs. Our invitation letters, questionnaire, telephone helpline, website, results publications, technical reports, datasets are produced in both Welsh and English. We aim to write clearly (using plain English/‘Cymraeg Clir’).

Comparing the results produced by TSW to existing data sources is a key component of our development work.

Annual results reports contain a detailed comparison of the latest TSW estimates against previous years and alternative statistics published by HMRC and ONS. Discrepancies are discussed and limitations in our estimates highlighted.

Development work on TSW continues and plans are in place to improve estimates by bringing in data from other surveys. We continue to seek feedback on our plans and on the outputs produced to date. Feedback can be sent to stats.trade@gov.wales.