Evaluation of the Co-operation and Supply Chain Development Scheme: summary

Delivered as part of the Rural Development Programme (RDP) 2014–2020, the Co-operation and Supply Chain Development Scheme (CSCDS) was funded under Measure 16 — Co-operation (Article 35 of Council Regulation (EU) 1305/2013) ^{[footnote 1]}.

The ‘vision’ for the scheme, as set out in the scheme guidance, was that businesses, organisations and communities should be working together to increase their resilience by tackling issues such as rural poverty, financial sustainability, climate change & the environment, and in-work poverty & social exclusion. Furthermore, the CSCDS was described as having been designed to ‘make things happen’, with support available for the full scope of a project, from the initial stages through delivery and into dissemination and evaluation, and having supported the development of new networks and clusters engaged in delivering specific projects.

All CSCDS applications underwent a competitive two-stage process, with applications invited periodically via ‘application windows’. An initial expression of interest (EoI) providing an outline of project activities and objectives was used for an initial eligibility assessment, with input from Welsh Government policy teams. Once that assessment was completed, if successful, applicants would then be invited to submit a full project application (the second stage). 

Final project numbers, expenditure data, and output figures were not available at the time of the evaluation, as they were subject to audit processes. Final figures will, therefore, be reported as part of the ex-post evaluation of the overall RDP in Wales. The focus of this evaluation, as set out below, was on understanding the nature of activities supported and results delivered, with an indicative analysis only of the relative importance of different types of projects within the overall project portfolio.

The aim of the evaluation was to provide an independent assessment of the implementation of the scheme and its impact to help inform decisions supporting similar types of domestic schemes in Wales after the RDP 2014–2020 comes to an end. It covered three sub-measures under Measure 16 that were delivered through the CSCDS, namely:

• support for pilot projects and for the development of new products, practices, processes and technologies
• support for horizontal and vertical co-operation among supply chain actors for the establishment and development of short supply chains and local markets and for promotion activities in a local context relating to the development of short supply chains and local markets
• co-operative Forest Planning Scheme

Research tasks

The following were the key tasks undertaken as part of the evaluation.

Documentary review

This included documents relating to the rationale behind the CSCDS in the context of the wider RDP architecture as well as all available CSCDS guidance and process documentation.

Scoping interviews

A total of 12 interviews focused on a detailed understanding of the CSCDS and its context, scheme processes, and stakeholder expectations regarding the evaluation.

A meta review of project-level evaluations

This covered 29 projects (37% of all projects) which corresponded to 18% of the total grant awarded and 28% of the total grant paid up to 24 October 2022.

Online surveys

Two online surveys were undertaken to capture insights regarding: (a) project lead experiences of CSCDS processes and their perspectives on the legacy likely to be created by CSCDS projects (27 responses were received), and (b) the participant experience of CSCDS projects (16 responses were received). Both surveys were in the field for 10 weeks (from 13 February to 30 April 2023).

Stakeholder interviews

A detailed ‘stakeholder map’ was developed that drew on suggestions from the scoping interviews as well as the evaluation team’s knowledge and experience. Experience, or at least awareness, of the CSCDS was a key sampling criterion alongside the type of organisation, the main sector of activity, and the nature and depth of CSCDS interactions. A total of 28 interviews were completed. 

Case studies

Eight case studies of specific CSCDS projects were developed to illustrate the highly diverse nature and results of the funded projects.

The research was conducted between September 2022 and April 2023, followed by the analysis and reporting tasks from May 2023 onwards.

Issues regarding data availability

The framework developed for the evaluation assumed that it would be possible to collate the quantitative and qualitative data required to cover all evaluation objectives through an analysis of existing scheme data together with additional evaluation field research. In the event, this was difficult due to a number of issues, including:

• sourcing project-level evaluation reports for the meta review (only 29 final evaluation reports could be obtained)
• sourcing contact details for primary research with Welsh Government officials, project leads, and unsuccessful applicants as a result of issues including staff turnover and out-of-date data
• the availability of comprehensive monitoring and expenditure data

It is important to underline that the timing (primarily that activity was ongoing at the time of the evaluation) and other methodological challenges encountered in undertaking this evaluation have meant that the findings and conclusions can be of an indicative nature only. However, several key insights emerge that are supported by the consistency of the evidence available, as set out below.

The range of projects supported

It is clear that the projects delivered using CSCDS funding were highly diverse, covering feasibility work, pilot and demonstration activities, as well as the development of new products, practices, processes and technologies. Thematically, activities ranged from agriculture/horticulture, food & drink marketing, and tourism activity to sustainability-focused projects in relation to energy efficiency & renewable energy, soil and water ecosystem services, and sustainable transport, as well as broader rural development activities including active travel, the Welsh language, and digital inclusion.

The nature of collaboration at the heart of CSCDS projects was equally diverse, taking in small project partnerships designed to lever complementary expertise for a particular development opportunity, co-operation between academics or rural economy intermediaries with practitioners in various activity areas, all the way to broad coalitions of key stakeholders covering key industry activities, academia, and policy partners.

Scheme strategy and project development support

A scheme such as the CSCDS holds significant strategic potential for rural economies in Wales. However, assessing the strategic value of project proposals once submitted was seen to be too reactive to maximise such value.

The strategic insights held by Welsh Government policy departments, if deployed well, could provide an important strategic steer to ensure that the cohort of projects receiving funding contributes effectively to identified strategic objectives. Providing an effective ‘strategic steer’ is, however, dependent on:

• the strategic framing of the scheme as a whole needs to clearly communicate how co-operation activities and the development, testing and piloting or demonstration of new products, processes, practices or technologies are seen to be of importance in different sectors and areas of activity
• outreach activities need to be capable of reaching and engaging those who can provide a genuine grassroots, practical perspective on strategic issues as they face organisations in the rural economies in Wales
• the active involvement of Welsh Government policy teams can reflect on and enrich strategic objectives that can then be applied to project development and selection

Stakeholder feedback suggests that while Measure 16 was intended to engage diverse rural organisations in co-operation activities, small organisations are unlikely to have the requisite financial standing and administrative capacity to develop and deliver a project under this kind of scheme. As a result, ultimate beneficiary target groups depend on larger organisations to provide such capacity and act as a ‘lead body’ for projects. The profile and motivations, project design capabilities, and the nature of interactions with beneficiaries of those organisations are, therefore, very important.

The project development stage is an important linchpin that sets projects up for success or failure. It is at this stage that a good fit between the lead organisation’s motivation and expertise and beneficiary needs can be ensured, and a sound project plan that can deliver tangible results against strategic objectives for the wider rural economy in Wales can be developed.

Programme management and reporting

Stakeholder feedback consistently highlighted the fact that the administrative processes and systems adopted for CSCDS programme management interactions with individual projects were not appropriate for the nature and scale of the kind of project that was supported through the scheme. The issues encountered included:

• delays in project approvals combined with a lack of flexibility in restructuring project plans in response to such delays
• a lack of access to appropriate advice and support
• a cumbersome claims process

Programme management aspects were, therefore, viewed as a key weakness of the scheme. In particular, the claims process received strong criticism for being overly rigid, opaque, and highly resource-intensive. Retrospective changes to compliance criteria and evidence to be provided further exacerbated this issue. The general view emerged that this mismatch is a serious challenge to the overall value for money that can be delivered by a scheme such as the CSCDS.

While there is evidence of an internal scheme management focus on understanding complementarities between projects and avoiding duplication, this appears not to have been sufficient in delivering a shared understanding of those issues with external stakeholders. Stakeholders, including project leads themselves, felt very strongly that an opportunity had been missed for project findings and results to be discussed as part of the CSCDS reporting relationship or shared more widely with a broader group of organisations. This was seen to be necessary in order to deliver more complete insights, engage a wider group of potential beneficiaries, and generate strategic momentum in different areas of activity.

Maximising project results and impacts

While there is clear evidence of many CSCDS projects effectively engaging direct project beneficiaries, the meta review and stakeholder interviews also show that the individual CSCDS projects rarely gained any traction with a wider target group of interested organisations.

The meta review and stakeholder interviews clearly suggest that, in line with the pilot nature of the funded activities, a majority of CSCDS projects have delivered intermediate outcomes rather than making a direct, quantifiable impact on direct beneficiaries or a wider cohort of organisations in rural economies in Wales.

Securing the full value for money from the CSCDS investment will, therefore, depend on ensuring that rural communities in Wales can take full advantage of project results and that learning from the projects supported is implemented. In many instances, further steps involved in bringing new products, processes, practices or technologies to fruition will require additional strategic investments. In others, meanwhile, the learning derived from projects will need to be used to steer further activity in the respective sectors and areas of activity.

Recommendations

Recommendation 1

Ensure that outreach activities are levered for Welsh Government policy teams’ strategic insights to combine with a grassroots perspective to ensure that clear strategic objectives can guide project development and selection.

Recommendation 2

Focus attention and resources on a proactive scheme management role at the project development stage, providing guidance and support and offering a ‘challenge function’ for project rationales and approaches.

Recommendation 3

Ensure that all administrative processes adopted for a scheme like the CSCDS are commensurate with the capacity of the target group and the nature and scale of anticipated projects. It is vital to avoid retrospective changes to claims and financial management processes.

Recommendation 4

Adopt a relationship-based reporting process with the capacity and adequate resources to consider project progress, identify potential synergies between projects as they develop, and ensure continuity of strategic engagement by facilitating interactions between projects funded under a scheme like the CSCDS. Welsh Government management of a scheme like the CSCDS needs to incorporate a support and facilitation function to maximise the overall value for money that can be achieved by the full cohort of projects.

Recommendation 5

Organise additional dissemination and networking activities to share the results and insights gained from CSCDS projects and make them accessible to one another and a wider cohort of potential beneficiaries in different sectors and areas of activity.

Recommendation 6

Ensure that, beyond a focus on an exit strategy for individual projects, project closure interactions include detailed consideration of project results and any requirements to ensure that they deliver anticipated outcomes and impacts.

[1] It should be noted that whilst the programme period runs until the end of 2020, spending can be incurred until the end of 2023 as a result of what is known as the N+3 rule, which applies to European-funded programmes.

Report authors: Eva Trier and Endaf Griffiths

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:
Richard Self
Email: research.evaluation@gov.wales

Social research number: 67/2024
Digital ISBN 978-1-83625-515-4

Image