Digital Communities Wales: Digital Confidence, Health and Well-being (process evaluation and theory of change summary)

The Welsh Government’s Digital Communities Wales (DCW): Digital Confidence, Health and Well-being programme is a three-year initiative delivered between July 2019 and June 2022 which aims to reduce digital exclusion and help improve basic digital skills levels across Wales. The £6 million programme is funded by the Welsh Government’s Prosperous Futures division and Health and Social Services division.

DCW aims to support individuals to gain the five basic skills set out in the UK’s essential digital skills framework with a focus upon supporting four priority groups (older people aged 50+; working age economically inactive and unemployed people; disabled people and social housing residents). DCW is expected to achieve nine key performance indicators, including to directly support 5,000 people through intensive interventions and to support 5,000 health and care staff to engage with technology in order to improve the health outcomes of users on an annual basis.

Following a competitive tendering process, the programme is delivered by the Wales Co-operative Centre, with elements of work sub-contracted to the Good Things Foundation and Swansea University.

OB3 Research, was appointed by the Welsh Government to undertake an evaluation of the DCW: Digital Confidence, Health and Well-being programme.

The aims of the evaluation are to:

• review and summarise existing evidence around the relationship between digital inclusion and health
• review the effectiveness and efficiency of the delivery of the programme
• assess the extent to which the programme aims have been achieved and targets met
• provide evidence of the outcomes of the programme for individuals and the services they access

The evaluation is being undertaken across three key stages to include this Process evaluation and Theory of Change (ToC); an interim and outcome evaluation; and a summative final evaluation.   

This process evaluation and ToC has involved:

• an inception stage, which included an inception meeting with Welsh Government officials, scoping interviews with Wales Co-operative Centre representatives and the preparation of a refined methodological approach and project plan 
• desk based research, which involved an analysis of relevant policy and strategic documents, a review of DCW programme documentation and literature relating to the relationship between digital inclusion and health and other outcomes
• preparing discussion guides for interviewing contributors to this stage of the evaluation
• interviewing Welsh Government officials, programme delivery staff from the Wales Co-operative Centre, Swansea University and the Good Things Foundation
• interviewing representatives from 20 organisations who have collaborated with DCW
• synthesising the findings of the fieldwork and desk-based research to develop a ToC logic model for the programme and preparing a peer-reviewed evaluation report

A ToC model for the programme is presented at Chapter 9 in the full report. A model setting out what DCW was expected to achieve is presented at Figure 9.1 whilst a revised model, which takes into account the findings of the process evaluation, is set out at Figure 9.2.

Programme design and implementation

• Welsh Government policy and legislation, when the current DCW programme was designed, supported continued public sector intervention to address digital exclusion and improve basic digital skills amongst key groups. The onset of the coronavirus (COVID-19) pandemic was found to have increased the underlying need for DCW and strengthened the case for intervention.
• the rationale for focusing intervention upon the health and care sector was clear and well documented. It was also appropriate for the programme to inspire interest in new technology amongst participants by focussing upon their hobbies and areas of interest. This was found to be an effective method of engaging digitally excluded groups.
• The DCW programme was found to be highly regarded, and the calibre and expertise of the delivery team was widely recognised
• Whilst the overall staffing resources allocated to the programme was appropriate it could benefit from more senior level engagement from within the Wales Co-operative Centre to support the strategic work with health organisations as well as a dedicated marketing post-holder.
• There is a need to consider whether other changes to the programme’s staffing structure are required in light of the impact COVID-19 has had upon the implementation model, such as a sustained growth in virtual training provision.
• There is much uncertainty about a new component of the programme, the Digital Inclusion Alliance (DIA), and how it can add value to the programme and the digital inclusion agenda across Wales, given the lack of progress made to date. Greater clarity and progress is required over the remaining programme period before the effectiveness of the DIA can be assessed.
• Since COVID-19, the requirements upon the programme from organisations and their staff or users has changed. Many organisations reported that their capacity to engage with the digital inclusion agenda and DCW had reduced but overall, demand for support to help organisations, staff and users to overcome the digital impact of COVID-19 increased.
• It was logical for DCW to have engaged in supporting care homes with loan devices, as it was well placed to contribute towards the COVID-19 emergency response. The programme found itself under significant pressure but despite this, responded in an agile and flexible manner to the challenge.
• COVID-19 has had a transformational impact upon the digital inclusion landscape across Wales and DCW needs to ensure that it continues to address emerging priorities associated with the digital divide.
• DCW has achieved four of its nine annual KPIs during its first year of delivery but under-achieved against the remaining five KPIs. The evaluation raises several issues relating to the nature and volume of programme KPIs and the feedback suggests that these need to be revised. The evaluation concludes that programme KPIs should be more strategic and better aligned with the intermediary outcomes expected of DCW, set out within the ToC model, as well as reflect the wider scope of the current programme in responding to the COVID-19 pandemic.

Recruiting potential recipients

• Demand has been consistently strong and grown since the advent of the COVID-19 pandemic, with an increasing reliance upon direct recruitment methods and virtual, rather than face-to-face, engagement since March 2020.
• It will be important for the programme to consider how it continues to support the new, possibly non-traditional cohort of people who have found themselves digitally excluded since COVID-19 and explore how it can focus upon specific sub-groups across its four target priority cohorts.

Engaging and support for organisations

• DCW has continued to engage with a wide range of organisations but there is scope for the programme to support, and work through, large private sector organisations to recruit potential recipients.
• Engagement with health sector organisations has been primarily at the operational level (e.g. at a ward or service level) and whilst this has been well received and is starting to make a difference, there is a lack of evidence to demonstrate that these activities lead to effective engagement with strategic, senior level representatives from across the health sector. The programme needs to consider how it can develop strategic relationships with senior representatives across the health sector, drawing upon advocacy support from key Welsh Government officials, senior Wales Co-operative Centre staff as well as other strategic digital inclusion advocates.
• There is initial evidence to suggest that DCW is making a difference to organisations’ approaches and digital inclusion practices. These outcomes related in the main to staff level changes e.g. upskilling staff, changing staff attitudes to technology; as well as allowing organisations to try new ways of working and new technology to support patient care.
• In a similar manner, DCW is having an impact upon staff and volunteers who have engaged with the programme. The evaluation found evidence that staff and volunteers who had engaged with DCW were reporting improved digital skills and confidence; were better equipped to pass on their learning to others; and were thinking differently about how digital technology could be used to support people in their community.
• The main outcomes identified by contributors for end users included increased confidence to use different types of IT equipment and technology for different purposes, improved health and well-being as well as reduced levels of loneliness and social isolation.

The evaluation offers seven recommendations for the Welsh Government to consider in the delivery of the DCW programme.

Recommendation 1

The findings of the evaluation be considered to inform the programme’s priorities over its remaining delivery period, ensuring that an appropriate balance is sought between meeting additional policy demand on the one hand and programme capacity on the other.

Recommendation 2

The extent to which there has been a significant shift in the definition of digital exclusion as a result of COVID-19 be explored, and that consideration be given to the bearing this may have upon programme implementation.

Recommendation 3

The Welsh Government explores how strategic and high-level engagement with, and commitment from, health organisations can be achieved, drawing upon robust evidence about the health and well-being outcomes that can be achieved via digital solutions as well as wider advocacy support including from Welsh Government officials, senior Wales Co-operative Centre staff, digital advocate champions and the Digital Inclusion Alliance.

Recommendation 4

The programme reviews whether there would benefit from making changes to its staffing structure in light of suggestions captured by this evaluation and to accommodate any sustained impact which COVID-19 has upon programme delivery.

Recommendation 5

The programme’s funded KPIs be reviewed with a view to setting more realistic, achievable outputs and considers adopting a small number of qualitative outcome KPIs, which adequately reflect the programme’s additional asset management work and strategic work with health organisations.

Recommendation 6

The implementation of the Digital Inclusion Alliance is accelerated, and that greater clarity is secured about its purpose and remit.

Recommendation 7

Consideration be given to the evidence DCW can gather to demonstrate that its asset management scheme for loaning devices, is leading to improved digital connectivity at care home provider settings. 

Bryer, N; and Bebb, H; (2020). Digital Communities Wales: Digital Confidence, Health and Well-being: Process evaluation and Theory of Change. Cardiff: Welsh Government, GSR report number 7/2021

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:

Siân Williams
Communities Division
Welsh Government
Merthyr Tydfil Office
Rhydycar Business Park
CF48 1UZ

Tel: 0300 025 3991
E-mail: socialjusticeresearch@gov.wales

Image

Digital ISBN 978-1-80082-844-5