Evaluation of the Cymru’n Cofio Wales Remembers Programme: 2013 to 2020 (summary)

The Internal Research Programme (Knowledge and Analytical Services, Welsh Government) was commissioned by the Culture and Sport Division, Welsh Government to undertake an evaluation of the programme delivery of Cymru’n Cofio Wales Remembers 1914-1918, the commemoration programme of Wales’ contribution to the First World War.

Cymru’n Cofio represented the Welsh Government’s contribution to the wider UK programme of marking the centenary of the First World War and in particular to mark the contribution of Welsh people, both on the front and at home. The programme’s objectives were:

• to identify and mark significant anniversaries, working with Welsh organisations and services, other UK Home Nations and international partners
• to support an educational programme that encourages schools and young people’s organisations to fully participate in commemorative activities
• to develop and support productive partnerships to deliver activities and events throughout the commemorative period to diverse audiences
• to support vibrant cultural and historical interpretation events and activities by our cultural and heritage bodies reflecting different perspectives on the period
• to work with the Heritage Lottery Fund (HLF) and other funders to support community projects telling the story of Wales and the First World War
• to ensure that information about the commemoration in Wales is easily available to everyone within Wales and beyond
• to leave a rich digital legacy of the commemoration for future generations

The aim of this evaluation was to retrospectively assess the ways in which the programme was delivered, to provide recommendations for effective delivery of future commemoration programmes and to understand how the impact of the activity funded as part of the programme could be measured. Due to the commissioning of the evaluation at the close of the project, it was not possible to measure impact effectively due to the lack of monitoring data infrastructure. Therefore this is a process evaluation; focusing on assessment of delivery of the programme based on reflections from the Programme Board, Welsh Government officials, and those who delivered community projects as part of the Programme.

The evaluation methodology included the following components.

• A Theory of Change workshop with key officials in the Culture and Sport Division and other Welsh Government (Education, Communications and Cadw) officials to retrospectively map the programme in a logic model.
• A series of qualitative interviews with members of the Cymru’n Cofio Programme Board covering aspects of coordination and delivery of the programme, the role of the communications strategy, and exploring data availability and possibilities for assessing impact in future programmes.
• A review of available data of the programme and the activity it has funded (including the Secondary Schools’ Grants Scheme and Cadw Scheme), from which recommendations were made for implementing monitoring and evaluation in future commemorative programmes.
• A short survey seeking the views of stakeholders regarding their experience of delivering their project and the nature and quality of contact with the programme board and their delivery of the programme.

Survey of community projects

The survey revealed that most of the 16 projects that responded to the survey did not seek support from the programme to deliver their project, the most common reason cited being that they weren’t aware that support was available. It is important to bear in mind here that the Cymru’n Cofio branding was in use for Welsh Government funded projects and those who were funded by partner organisations only.

Most community projects were aware of the programme branding and that it was provided by Welsh Government. Just over half of the projects surveyed reported using the branding. Awareness of the Programme Board and their role was also high, and respondents reported positive experiences of the quality and frequency of engagement, accounting for the views of projects, support to resolve issues and respecting project autonomy.

Ten of the 16 projects reported collecting monitoring data, with only one project collecting demographic information about visitors and participants. The most commonly collected data included the number of attendees to an event or initiative and feedback from visitors.

Perspectives of the Programme Board

Programme board members cited that the Board was an enabler in terms of creating opportunities for collaboration with relevant organisations to deliver on the aims of the programme. They also reported that Welsh Government provided increased capacity to move projects forward and that the involvement of the First Minister raised the profile of the commemorative programme. Whilst the ability for all Board members to express their views was considered a positive, this also meant that decision-making sometimes took longer. Other challenges for the Board included the mismatch in the pace of decision-making, which sometimes proved challenging for projects to deliver to.

The communications strategy was discussed in an overall positive light, however some Board members would have preferred the strategy to be clearer in expressing how it could offer support to projects. Internal communications were reported to be very good, although some Board members found the travel commitments to attend meetings in Cardiff were sometimes burdensome. The ICT improvements implemented as a result of the pandemic could be effectively utilised for future programme boards to eliminate this issue.

In terms of measuring the impact of the programme overall, interviews revealed that project-level evaluations had provided good feedback on how to improve future events. However, many interviewees stated that interpretations of what the programme could achieve varied between Board members and that the nature of the programme meant that success was difficult to quantify.

When engaging those with protected characteristics in the commemorative events, Board members reported that they felt that projects considered these issues in the context of their initiatives. However, as the survey shows, there are little data to support successful engagement of marginalised groups, as little demographic data on who engaged was collected. Relevant umbrella organisations were also engaged in the programme, but Board members felt that this could have been done in a more systematic way from programme inception. It was also difficult to determine whether those with protected characteristics felt included. These findings suggest that more evidence of the impact of improving inclusivity would be beneficial.

Data availability and quality

The review of programme documentation revealed that at the project level, most initiatives had submitted a business case that outlined the project objectives, how it would be delivered and resources required, as well as timescales and monitoring approaches. Projects were also obliged to complete an end of project report detailing the data collected as part of the monitoring approach and to demonstrate the impact of the project. Although these were submitted and did contain useful data, it was difficult to tie the data collected to the monitoring commitments set down in the business case. An absence of baseline data on which to compare the post-project data also made impact difficult to determine. Business cases would have benefitted in most cases from a more detailed monitoring and evaluation plan, including plans for baseline data collection. This would also have provided a clear approach for collecting and interpreting data that could feed into the overall programme objectives.

The following recommendations are therefore made for future commemorative programmes, first at programme level.

It is recommended that future programmes engage KAS colleagues at the earliest opportunity to design an evaluation approach to commemorative programmes. The following activities are recommended:

• to design an overall monitoring and evaluation plan for the programme, including an approach for assisting funded projects to monitor and evaluate their activity and determining how project data feeds into the overall assessment of impact
• hold a theory of change workshop at the inception of the programme to determine the intervention logic, and to understand which data are most appropriate and viable to collect at a project and programme level
• consider creating additional capacity within the programme team to manage data monitoring and evaluation across the duration of the commemorative programme
• secure funding for a full evaluation from the outset of the programme, to enable resources to dedicate to fully scoping data availability and collecting and analysing said data

Future commemorative programmes would benefit from earlier and co-ordinated engagement with representative bodies for protected groups to understand what is effective in terms of engagement with ethnic minority communities, for example. This would allow for a more targeted approach to engaging protected groups and understanding whether funded projects have reached the intended audience.

Greater consideration is needed in evaluation planning to understand what is meant by the term ‘legacy’ and how impact in creating legacy can be evidenced. Understanding how this can be achieved may also be useful understanding the longer term benefits of the programme. This can be built into the theory of change approach.

For funded projects, the following recommendations are made.

Commemorative programmes should provide support to projects to develop an evaluation plan that ties in with the programme level evaluation. Developed at project inception as part of the business case, this would enable baseline and mid-term collection of monitoring information that can demonstrate clear evidence of impact at project close.

Evaluation of funded projects should be co-ordinated at programme level to ensure that monitoring data collected by projects is considered within the overall programme monitoring data. This could be best achieved by using additional resource to manage monitoring and evaluation. This would enable the programme objectives and outcomes to be more clearly tied to project level objectives and outcomes.

Where possible, demographic data should be collected to understand who is engaging with the outputs of funded community projects. The majority of projects surveyed did not collect these data, leading to a gap in understanding about (i) the extent to which those with protected characteristics were engaging with commemorative activity and (ii) the extent to which marginalised groups felt represented in the commemorative activity. These data would help to determine whether the objective of widening representation and participation in commemoration had occurred.

In order to maximise the quality of monitoring data submitted, submission of baseline, mid-term and end of project data should be mandatory, with support made available to projects to enable them to submit this information. Although most projects did submit data at project close, some did not and this means that a valuable opportunity to understand impact is lost. Making the submission of data for use as part of a programme level evaluation mandatory and providing support to enable them to submit theses data would increase the reliability of data submission from projects and elevate the importance of evaluation for funded projects.

Report Authors: Coates, J, Campbell, L, and Heywood Heath, C

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:
Jo Coates
Email: rhyf.irp@gov.wales

Media: 0300 025 8099

Social research number: 9/2022
Digital ISBN: 978-1-80391-508-1

Image