Digital Communities Wales: Digital Confidence, Health and Well-being (interim process and outcome evaluation summary)

The Welsh Government’s Digital Communities Wales (DCW): Digital Confidence, Health and Well-being programme is a three-year initiative delivered between July 2019 and June 2022, which aims to reduce digital exclusion and help improve basic digital skills levels across Wales. The £6 million programme is funded by the Welsh Government’s Prosperous Futures division and Health and Social Services division.

DCW aims to support individuals to gain the five basic skills set out in the UK’s essential digital skills framework with a focus upon supporting four priority groups (older people aged 50+; working age economically inactive and unemployed people; disabled people and social housing residents). The programme is delivered by the Wales Co-operative Centre, in collaboration with the Good Things Foundation and Swansea University.

OB3 Research, was appointed by the Welsh Government to undertake an evaluation of the DCW: Digital Confidence, Health and Well-being programme.

The aims of the evaluation are to:

• review and summarise existing evidence around the relationship between digital inclusion and health
• review the effectiveness and efficiency of the delivery of the programme
• assess the extent to which the programme aims have been achieved and targets met
• provide evidence of the outcomes of the programme for individuals and the services they access. 

The evaluation is being undertaken across three key stages to include a Process evaluation and Theory of Change (ToC) published February 2021; this interim and outcome evaluation; and a summative final evaluation.

This interim and outcome evaluation has involved:

• an inception stage, which included an inception meeting with Welsh Government officials and refining the second stage work programme 
• a desk based review of recent policy and strategic documents, DCW programme documentation and monitoring data
• drafting discussion guides for interviewing contributors
• interviewing a total of nine Welsh Government officials and programme delivery staff from the Wales Co-operative Centre, Swansea University, and the Good Things Foundation
• interviewing a total of 38 individuals involved with 10 case study organisations who have collaborated with DCW, including DCW advisors and beneficiaries
• interviewing one representative from a care home
• surveying 27 Digital Inclusion Alliance Wales members
• synthesising the findings of the fieldwork and desk-review and preparing a stage two report

The evaluation found that in terms of programme rationale:

• Welsh Government digital inclusion policy has been further developed during 2020 and 2021, with the publications of Digital Inclusion Forward Look and Digital Strategy Wales. It is likely to be enhanced further by the forthcoming strategy for the health and social care sector and the work of the Chief Digital Officer for Health and Care. DCW is well positioned to help deliver these digital inclusion policies 
• a raft of other digital developments are taking place across the health and social care sector in Wales and DCW is well placed to support health and social care organisations to develop and implement their own digital inclusion strategies provided its work programme is aligned to these emerging digital priorities
• levels of digital inclusion have improved in Wales and the most robust data from the National Survey for Wales suggests that 92 per cent of households now have access to the internet and 93 per cent of people use the internet. Digital engagement levels remain lower in Wales than other UK regions and the pandemic has raised the bar in terms of the skills and confidence now required to be digitally included which justifies continued support for digitally excluded communities in Wales

The evaluation found that it terms of progress made by the programme:

• DCW now has strong leadership in place, benefits from a more appropriate senior management structure and adopts better processes for capturing data and reporting information to the Welsh Government
• a recent programme staff restructure appears to be appropriate and working well and staff resources are now better aligned to programme priorities and intended outcomes
• programme KPIs have been reviewed and new indicators better reflect programme objectives. New monitoring tools have been recently introduced and need to be applied in a consistent manner to capture comparable, useful data
• DCW is either on track or exceeding three of its new KPIs and whilst slightly behind on its fourth KPI, it looks likely that this target will be achieved over the programme period. The outcome data reported to date is encouraging, particularly in terms of individuals trained reporting positive changes to their lives
• the development of bespoke, culturally sensitive training content which meets the needs of Welsh speakers is proving very effective and should be flagged up as good practice. Similar steps to develop bespoke provision for Black, Asian and Minority Ethnic communities should be prioritised over the remaining delivery period

In terms of the progress made by Digital Inclusion Alliance Wales (DIAW), the evaluation found that it:

• has made good progress to adopt appropriate governance arrangements and publish a well-supported digital inclusion agenda which has helped to provide clarity about its purpose and remit
• would benefit from securing a broader membership base – particularly from Black, Asian and Minority Ethnic communities, Welsh language communities and the health and social care sector
• needs to consider its long-term sustainability, particularly if DCW funding were to end in June 2022

In terms of engagement with partner organisations:

• since spring 2021, the programme has been able to re-engage successfully with organisations, particularly third sector organisations, but it continues to struggle to engage the private sector
• the programme has been successful in engaging at a strategic level with health and social care sector via umbrella or member led organisations and there is scope to build upon these sector-wide approaches over the remaining duration of the programme as they offer a pan-Wales reach and a sustainable route for hosting provision in the long-term
• DCW has been able to successfully engage with some individual health boards and local authorities at a strategic level and in these cases was found to make a valuable contribution to the development of organisation-wide policies and practice
• there are numerous examples where DCW is working effectively with individual departments and services across the health and social care sector in a more evolving manner, and that these interventions have helping to introduce new digital ways of working

In terms of the experience of supported organisations, the evaluation found that:

• virtual training has been critical over the last year and DCW’s approach to virtual training is becoming more strategic and smarter. The programme would benefit from focusing on provision which can be sustained in some form in the future
• the feedback on the quality, value and benefit of training made available continues to be excellent. Case study organisations provided ample evidence that the training it helping to improve digital skills, enhance knowledge and increase confidence amongst staff
• case study organisations reported that trained staff share and apply new digital skills with colleagues and service users. Feedback also suggests that the application of new digital skills amongst service users is helping to reduce levels of loneliness and social isolation, improve mental health and help individuals feel that they belong to communities of interest
• making digital equipment or connectivity available in isolation isn’t particularly effective, despite there being several examples where such resources have been well received and utilised
• in contrast, where devices have been loaned to organisations as part of a wider support package, the resources are helping to contribute to positive digital outcomes for staff and service users. Such equipment is playing a vital role in allowing service users to maintain contact with friends and families as well as maintain or develop new hobbies and interests
• the programme has played a key role in introducing staff and service users to innovative devices, such as virtual reality equipment, and the feedback suggests that DCW is effective in helping to keep staff abreast of new and emerging technology

In terms of future priorities, the evaluation found that:

• DCW should prioritise its strategic work to embed digital inclusion into health boards’ Integrated Medium-Term Plans (IMTPs) and digital strategies, building upon the good practice established with Betsi Cadwaladr University Health Board (BCUHB)
• the Welsh Government should continue to play a key role to enable DCW to engage with, and support the implementation of, important developments across the health and social care sector
• DCW should explore how it can offer a blended training model for the remaining period of delivery and return to provide some face-to-face training provision

The evaluation offers nine recommendations for the Welsh Government to consider for the DCW programme:

1. The Welsh Government should consider extending the DCW programme post 2022, as there is strong justification for doing so in terms of need and programme performance to date.
2. The Welsh Government needs to ensure that DCW is well positioned and enabled at a strategic level to support the implementation of future digital strategies such as those set out by health boards within their IMTPs; as well as the future work programme set out by the Chief Digital Officer for Health and Care. Similarly the Welsh Government needs to ensure that DCW actively engages with the digital priorities set out within the Money and Pensions Service Strategy.
3. The DIAW would benefit from focusing upon and achieving a small number of key priorities over the remaining funding period to demonstrate its value and make a tangible difference. It would also benefit from securing representation from organisations which are currently under-represented from across health, Welsh language, and Black, Asian and Minority Ethnic Communities.
4. In the case of the DCW programme not being funded post 2022, there is an immediate need to consider the longer-term sustainability of the DIAW. Regardless of this matter, the DIAW needs to explore how it could sustain its work on an on-going basis in the future.
5. DCW needs to prioritise and reflect upon how it can better engage with the private sector, exploring other approaches where possible e.g. via trade unions to reach private sector employees.
6. DCW should build upon its successful sector-wide approach to working across the health and social care sector, by mapping out and approaching national umbrella and membership organisations it could engage with, including across other sectors where appropriate.
7. DCW should continue to develop and deliver its bespoke solutions for Welsh language audiences and prioritise its approach to engaging Black, Asian and Minority Ethnic Communities over its remaining period of delivery.
8. DCW should reflect on the lessons learnt from its approach to loaning equipment during the pandemic and in future, make available digital equipment only as part of a wider package of support to organisations and individuals rather than in isolation.
9. DCW needs to consider how it can offer a blended programme of digital training provision over its remaining period of delivery, looking where possible to produce virtual resources which can be sustained post programme funding by partner organisations.

Bryer, N; and Bebb, H; (2022). Digital Communities Wales: Digital Confidence, Health and Well-being: Interim Process and Outcome Evaluation. Cardiff: Welsh Government, GSR report number 25/2022

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:

Siân Williams
Social Justice Research
Social Research and Information Division
Knowledge and Analytical Services
Welsh Government
Cathays Park
Cardiff

Tel: 0300 025 3991
E-mail: socialjusticeresearch@gov.wales

Image

Digital ISBN 978-1-80391-868-6