Evaluability assessment of the Health and Social Care (Quality and Engagement) (Wales) Act 2020 (summary)

The Health and Social Care (Quality and Engagement) (Wales) Act 2020 (UK legislation) (hereafter referred to as ‘the Act’) received Royal Assent in June 2020 and was fully commenced in April 2023 ^{[footnote 1]}, with a new Citizen Voice Body for Health and Social Care (CVB) also becoming operational from this date.   

The Act’s provisions include the duty of quality, the duty of candour, the CVB, and the requirement for NHS trusts to have a statutory Vice Chair. Continuous improvement in the quality of services is the central concept underpinning the provisions. Quality is viewed as a system-wide way of working, to enable safe, effective, person-centred, timely, efficient, and equitable services, in the context of a learning culture.

The Welsh Government plans to evaluate the Act as part of their post-implementation review. The evaluation is intended to explore the Act’s implementation and impacts to assess the extent to which the anticipated effects, costs and benefits are realised, for whom, how and why. The evaluation is expected to begin in 2023 and will assess the first three years post-implementation.

In 2022, the Welsh Government commissioned Opinion Research Services (ORS) to undertake an Evaluability Assessment (EA) of the Act. Its aim is to make recommendations on an appropriate programme to evaluate the Act’s implementation and impact.

The EA comprised workshops and interviews with Welsh Government, NHS, and other key stakeholders, and a review of literature and documentation relating to the Act. Separate workshops were held on each of the four parts of the Act. The workshops and interviews were used to develop and refine the theory of change; explore which service users and stakeholders should be involved in the evaluation; identify evaluation questions, priorities, approaches, and data; and consider the implications of the wider policy environment, including the Well-being of Future Generations (Wales) Act 2015. Insight from the documentation review, workshops and interviews was combined with our knowledge and expertise, as well as insight from methodological literature, to inform our desk-based work to develop outcome measures; identify considerations and implications for the evaluation; and to make recommendations on an evaluation approach.

The full report includes a theory of change and a table on outcome measures and data sources for each part of the Act. At the point of data collection, discussions were still taking place to refine plans for delivering the duty of quality, duty of candour and CVB. Consequently, their theories of change, and evaluation recommendations that depend on them, will likely need to be revisited and developed further.

There are numerous existing data sources which could potentially be used in the evaluation. These include surveys with the public (e.g., the National Survey for Wales), service users (e.g., Patient-Reported Outcome Measures and Patient-Reported Experience Measures) and the workforce (e.g., NHS Staff Survey). They also include, for example, annual reports for the duty of quality, duty of candour and the CVB; policy documents; and relevant meeting papers. Where the Welsh Government does not own the data, access to the source would need to be explored. The Welsh Government should also explore opportunities to use the Secure Anonymised Information Linkage (SAIL) databank and the National Data Resource (NDR) to inform the evaluation. The Welsh Government should seek to understand what relevant research and data collection is already underway to ensure that the evaluation can dovetail with this rather than duplicate it.

Some primary data collection will also be needed. We anticipate this would largely include qualitative research with, for example, members of the public; NHS and social care staff in general; as well as service users, staff and stakeholders who have experience with specific aspects of the Act. 

The evaluation should seek to establish a baseline for each area of the Act, where relevant and feasible, to accurately measure its impact. This will be challenging due to gaps in the data.

Each part of the Act is distinct, with its own theory of change, so they should be evaluated as individual parts rather than as a whole. However, each part would ideally be combined within one overarching evaluation because of their shared themes and opportunities for data consolidation (e.g., combining qualitative research activities, wherever possible).

As the Act’s name suggests, the themes which run throughout are quality and placing the patient at the centre of service planning and delivery, particularly for the duty of quality, the duty of candour and the CVB. These themes in common are reflected in similarities in some long-term and medium-term outcomes in the theories of change. The evaluation could therefore include an overarching element that draws together the linkages in their overall aims.

Stakeholders said the Act would help to improve the health care system for the next generation, which links to the greatest extent with the well-being goal of ‘a healthier Wales’ in the Well-being of Future Generations Act. They also noted a link with ‘a more equal Wales’. The evaluation could seek to make the links between the Act and the Well-Being of Future Generations (Wales) Act 2015 more explicit, to assess the Act’s contribution to achieving the well-being goals.

Some explicit links have also been made between the Act and the five ways of working. The evaluation could explore how the ways of working are embodied in the implementation of the Act. The evaluation could also explicitly adopt the five ways of working in its approach.

Assessing the Act’s impact will be difficult due to the myriad of factors and initiatives which influence the outcomes associated with it. Several policies and strategies should be borne in mind when planning and conducting the evaluation either because they have implications for the Act’s implementation and/or could impact the evaluation’s outcomes of interest. Relevant policies and strategies include, for example, A Healthier Wales (2021), the National Clinical Framework (2021), and the updated Putting Things Right (PTR) guidance.

A mixed method, contribution analysis approach is recommended to evaluate the Act. Contribution analysis (Addressing attribution through contribution analysis: using performance measures sensibly (Jonn Mayne, 2001) (Better Evaluation); (Contribution Analysis: Coming of Age? (John Mayne, 2012) (ResearchGate)) seeks to identify to what extent observed results are caused by policies, programmes, or services through testing theories of change against new and existing evidence and identifying other influencing factors. Rather than aiming to assess the counterfactual (i.e., what would have happened without the Act), contribution analysis aims to reasonably establish causality within complex settings, acknowledging that the programme, policy, or intervention may have an influence, along with other factors.

The evaluation should combine process and outcome strands that seek to identify the effects of each part of the Act and the mechanisms by which these effects were achieved, grounded in the theories of change. This should be done by collating and combining insight from quantitative data, documentary sources, and qualitative engagement.

Assessing the Act’s value for money would be complex and options are likely to be limited. However, cost consequence analysis could be considered for some parts of the Act. The Welsh Government could consider what data on set up and ongoing delivery costs could be made available. As evaluation plans are finalised, the nature and scope of evaluation data will also become clearer. This would inform considerations of the feasibility and robustness of this type of analysis.

Given the timeframes, the evaluation should incorporate short- and medium-term outcomes, and consider how the long-term outcomes could be assessed after the initial evaluation is completed. The EA was unable to establish the anticipated magnitude of outcomes, so this should be considered as part of the evaluation. Data collection and analysis should take place at key points throughout the evaluation, including tracking progress in outcomes over time, and engaging the same service users in qualitative research at different time points where possible.

The evaluation should be conducted by suitably skilled, independent researchers to ensure objectivity. Researchers should also be trained and experienced in conducting sensitive research with vulnerable participants, a key consideration for engaging with service users who have experienced harm to evaluate the duty of candour.

[1] The only provision to be substantively commenced before this is section 24 that enables the Welsh Ministers, if they consider it appropriate, to appoint a vice chair of an NHS trust. This provision was brought into force by the end of 2021.

Authors: Phillips, L., & Lock, K

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:
Health and Social Services Research
Email: research.healthandsocialservices@gov.wales

Social research number: 93/2023
Digital ISBN: 978-1-83504-581-7

Image