Sustainable Management Scheme: process evaluation and theory of change report (summary)

The SMS is a competitive £23.3m grant funding scheme available to individuals and organisations who collaborate as groups to improve Welsh natural resources. The SMS is funded through the Welsh Government Rural Communities: Rural Development Programme (WG RC-RDP) 2014 to 2020.

The SMS aims to support collaborative approaches to land management activities which will improve natural resources and help achieve ecosystem resilience which will sustain the social and economic benefits they provide to communities.

The scheme was designed to contribute towards the implementation of the Environment (Wales) Act 2016 and was aligned to the aims of the Wellbeing of Future Generations (Wales) Act.

SMS places a significant emphasis on the principles of collaboration and engagement, and it was expected that action would be taken at the landscape (such as the catchment), rather than individual farm, scale.

The selection of projects to be funded was done over two separate stages. Applicants were expected to submit an EoI and successful ones were then invited to submit a full funding application. Five funding windows have been administered by the Welsh Government.

The SMS is administered under the RDP by Rural Payments Wales and supported by a small team from the Environment and Rural Affairs Department within the Economy, Skills and Natural Resources Group (now called Landscapes, nature and forestry) of the Welsh Government. As was the case for all RDP administrative functions, the financial management of the SMS was initially undertaken by the Scheme Management Unit before responsibility was transferred to the Rural Payments Team within the Welsh Government.

Part-way through delivery the scheme put in place a support service to guide and advise farmers, foresters and other land managers to develop new project ideas and form new partnerships or groups in order to submit an EoI to the SMS. The service also provides support to projects invited to the second stage of application to help them develop full project plans. This service is delivered as part of the Farming Connect service by Menter a Busnes.

OB3 Research, in conjunction with BRO Partnership, were appointed by the Welsh Government to undertake an evaluation of the Sustainable Management Scheme (SMS).

The aim of the evaluation is to review how the SMS is supporting collaborative action to improve natural resources and help achieve ecosystem resilience and to assess its subsequent contribution to sustaining social and economic benefits for communities.       

It is intended that the evaluation explores five main objectives:

1. the alignment of projects to the principles of Sustainable Management of Natural Resources (SMNR)
2. the management and implementation of the scheme
3. the nature and extent of collaboration and what this achieved
4. the outcomes of the scheme and action on the policy priorities
5. the contribution to the cross-cutting themes, particularly climate adaptation and mitigation

This is the first of three annual update report as part of the evaluation of the SMS. This report sets out the theory of change for the SMS and presents the key findings of a process evaluation. This report has focused on the management and implementation of the scheme and the nature of the collaboration.

The evaluation activities involved the following elements of work:

• an inception stage, which included an inception meeting with Welsh Government officials and the preparation of a refined methodological approach and project plan 
• desk-based research, which involved an analysis of relevant policy and strategic documents as well as a review of SMS scheme documentation and reports
• preparing a range of qualitative discussion guides for interviewing contributors of the evaluation  
• preparing and distributing an online survey for all successful and unsuccessful applicants to the SMS
• undertaking interviews with 18 stakeholders and 26 representatives across 18 SMS projects
• holding online focus group discussions with ten members of the facilitation support service
• interviewing ten additional key stakeholders from Welsh Government and other external organisations including Natural Resources Wales (NRW)
• synthesising the findings of the fieldwork and desk-review to develop a ToC logic model for the SMS and prepare the final report

Rationale for the scheme and alignment of projects to the principles of SMNR

• The SMS has clearly been designed and developed by Welsh Government in a way that builds on the Nature Fund, but which adopts at its heart the principles of SMNR.
• The landscape scale approach to funding, as opposed to farm scale, allows for place-based project partnerships that can generate much wider, community level benefits. It provides an opportunity for such partnerships to expand their work and strengthen their collaborative ways of working. The Welsh Government is to be commended for developing and delivering a scheme that is based on these sound principles.
• The funding criteria and application had the SMNR principles firmly embedded within the process, requiring projects to demonstrate their adoption of a holistic ecosystem approach. Place-based or catchment-based interventions have been funded that are attempting to generate greater benefits and outcomes. Making funding available to partnerships of organisations and landowners, and, in the latter stages, funding a facilitation support service to enable the establishment of such partnerships, was also very much aligned to the SMNR principles and Wellbeing of Future Generations Act’s ways of working.
• The feedback received from projects suggests that they understood the aims of the SMS and the principles of SMNR and felt that they delivered against these. However, requiring an in-depth understanding of environmental policy priorities and principles is a challenging ask of locally grown partnerships. The SMS recognised this during the early funding windows, and as such, the provision of a facilitation support service seems to have played a crucial role in explaining the principles of SMNR to landowners and farmers who would not necessarily be as familiar with these as NGO staff might be.

Management and implementation of the scheme

• The SMS has proved to be a popular funding scheme, receiving a total of 226 EoIs across the five funding windows. This healthy level of interest and awareness of the SMS across a wide range of potential applicants is a strong initial indicator of the need for and the success of the SMS.
• The SMS has also, to some degree, become a victim of its own success. The sheer demand for the scheme, the level of detail required within the EoI template and the high quality and number of EoIs submitted has meant that whilst the quality improved over each funding round, the competition increased. As a result, many projects in the later funding windows, have not been able to proceed to application stage. A number of unsuccessful (yet high quality) projects have been left feeling downtrodden with unrealised expectations.
• Whilst the decision-making process for the SMS was robust and largely appropriate, there is a strong case for simplifying and reducing the length of the forms, particularly in terms of the information requested at the EoI stage. In addition, given the length of time taken to approve applications and award funding, there is a need to streamline and speed up the process should a similar scheme be made available in the future. Based on the feedback gathered, there is also an argument for adopting a more holistic approach to the assessment process, to avoid unnecessary repetition within applications. Evaluation feedback also suggests that closer involvement with policymakers, NRW strategic staff and those with an understanding and overview of regional priorities would improve the process.
• Evidence for this first annual evaluation report is mainly gleaned from projects funded via the first three funding windows when no facilitation support service was on offer. However, early indications and initial feedback via the survey and other contributors suggests that this service has been highly valued and has enabled collaboration and achieved greater parity of access to SMS funding.
• The issues highlighted in this report seem to point to an administration capacity issue within Welsh Government which hindered their ability to undertake the assessment process within the initial timescales set. The SMS team has been praised for its support and helpful approach, but the pressure on their time and demands would suggest that their on-going support function may benefit from greater resources to help manage a heavy workload.
• The rigid and unclear requirements for financial claims reporting, and the lack of communication and timely response to queries does not seem to be in keeping with the adaptive management approach embedded in the SMNR. These issues have been numerous and widespread, hampering progress and causing severe issues for several of the partnerships funded via the SMS, particularly smaller third sector organisations or landowner-led partnerships. Immediate improvements are clearly needed to the financial claims’ procedure. There is a contradiction between the SMS encouraging local landscape-scale partnerships to apply for funding but in doing so, presenting them with a complex system which they are not (and should not be expected to be) equipped to deal with. It is difficult to see how such partnerships can continue to be encouraged to lead bids, without the significant support of NGOs or public sector systems and processes, within the current reporting system.
• The SMS places a significant emphasis on the principles of collaboration and engagement at landscape scale with an expectation that funded projects demonstrate meaningful collaboration and commitment from a wide range of organisations and individuals. Most of the funded projects that formed part of the fieldwork reported a level of existing collaboration prior to developing their applications. Whilst the SMS has strengthened these partnerships and broadened their impact, there is less evidence available to date that it has stimulated new, sustainable collaborations. 
• Whilst projects found NRW input and involvement helpful, there does not appear to be a consistent level of engagement with projects across Wales. Similarly, it appears that there are additional opportunities for greater cohesion between the SMS and NRW at a more strategic level with a real opportunity to link SMS projects more closely with the emerging priorities of the now published Area Statements.
• The evaluation identified some collaboration between SMS projects which had enabled learning about good practice and helped avoid duplication of effort. Projects were generally keen to see more opportunities to collaborate with other SMS funded projects and there is a sense of a missed opportunity here, where the programme could have done more to directly facilitate such opportunities for sharing experiences and learning. Again, this suggests a lack of capacity within the WG staff structure to fully support projects as they would have wished. 
• The key emerging lesson is that successful collaborative land management takes time - to develop trust between partners and to build up ideas. A transparent approach is required with regular communication. In this respect, and particularly for landowner-led partnerships, the evidence suggests that a significant development phase is important for the design of a funding scheme such as the SMS. A ‘hand-holding’, mentor role for the early days of the delivery phase might be useful, possibly extending the facilitator’s role.
• It is encouraging to see that some SMS projects are already contributing towards an understanding of how to develop PES approaches within environmental management. Indeed, several projects were also looking to Glastir funding or PES contracts as part of their sustainability plans. There is also a real opportunity for the lessons generated from the SMS to be taken on board by policymakers as future funding approaches for farming are developed.

Outcomes of the scheme and contribution to cross-cutting themes

• Whilst early indications suggest that the SMS is delivering against its aims and objectives and projects are generally achieving their targets and outputs, future update evaluation reports will need to consider and analyse data to assess this more robustly. The strength of the SMS’ approach is its flexibility, and although each project is required to meet set criteria, to a large extent they have been able to define and set their own targets.
• It is important that SMS projects themselves fully understand the requirements to undertake their own project level monitoring and evaluation and use this opportunity to highlight the achievement (or over-achievement) of targets and any evidence of impacts and benefits at this stage. The programme level evaluation acknowledges the difficulties in trying to aggregate and report on key programme level outputs and achievements, but subsequent reports will aim to do this as much as possible. This work will be aided by current efforts by the Welsh Government to develop common aggregated indicators for the programme. This is crucial to ensure the programme can capture, demonstrate and communicate more widely the outcomes it has achieved.
• It is also the case that the very nature of the SMS means that only direct and immediate outputs can be reported upon at this stage (e.g., number of trees planted, or hectares restored) rather than the wider, longer-term ecosystem benefits and socio-economic outcomes that can take several years to fully materialise. Future evaluation update reports will need to remain mindful of this and consider what the implications might be for project level evaluations so that as many of these wider outcomes are captured and reported upon. 
• This quandary also suggests that the design of a scheme such as the SMS may benefit from considering a much longer delivery period of at least five years to enable its desired outcomes to be fully realised and the evidence of its impact to fully emerge.
• It is promising to see such a strong contribution by SMS projects to the cross-cutting themes, the RDP cross-cutting objectives and, to the aims of the Welsh Government’s Welsh language strategy.
• Even at this early stage, the evaluation has generated evidence that should be utilised to inform the way in which future funding programmes facilitate the co-design of sustainable land management approaches and align more closely with regional priorities and needs.

The evaluation offers eight recommendations for the Welsh Government to consider for the remaining delivery period of the SMS programme and wider recommendations that might help inform future funding approaches.

Recommendation 1

Key lessons in terms of policy design for landscape scale collaborative projects should be taken on board in any future funding mechanisms. The findings and early results of this evaluation report should be shared with Welsh Government policy officials who are currently drawing up the structure for the future funding of farming in Wales.

Recommendation 2

The Welsh Government should continue to ensure that future funding mechanisms for agri-environment schemes build on the innovative and flexible collaborative approach of the SMS but consider adopting a 5+ year delivery timescale for such grant schemes in future.

Recommendation 3

Several key learning points have emerged from the SMS’ EoI and application process that need to be considered when developing an application process for any future funding scheme, including:

• a need for a shorter EoI process (no more than 15 pages long)
• a six-to-12-month development phase prior to a full three-year delivery phase
• an assessment process that assesses the whole application rather than individual sections
• greater involvement of NRW in the assessment process, to ensure alignment with regional priorities
• dates for funding windows to be outlined from the outset and adhered to
• swift turnaround from successful application to awarding of funding

Recommendation 4

The facilitation support role should be provided from the outset of a funding scheme in future and its remit extended to help projects bridge the gap and make the transition from application to delivery phase. Facilitators could undertake a quasi-project management role within collaborative projects that are not led by a public authority or NGO until the project management team are appointed.

Recommendation 5

Regional peer-to-peer networking between SMS projects to share learning and experiences needs to be facilitated and encouraged by Welsh Government. These sessions should be regular sessions (at least quarterly in each region), held on-site where possible, and include both Welsh Government and NRW input.

Recommendation 6

Opportunities for greater engagement between the SMS and NRW should also be fully explored. A policy seminar for NRW staff to disseminate the early lessons from the SMS would enable a greater understanding of the scheme. Welsh Government should also consider ways of encouraging a more consistent level of engagement of NRW with all SMS projects across Wales.

Recommendation 7

Capacity within Welsh Government should be increased so that the sufficient staff time and resources are made available to deliver the support required for such a range of complex and varied SMS funded projects for the remaining period.

Recommendation 8

The current claims procedure needs to be reviewed and adapted immediately so that it can be more responsive to the needs of the SMS. Training should be arranged to ensure all Welsh Government staff involved with claims understand the nature of landscape scale grant schemes and the implications of delays on delivery. Consideration should be given to establishing a dedicated team within RPW, to allow more streamlined assessment and approval processes, and a dedicated individual allocated to each SMS project to respond promptly to queries.

Authors: Bebb, H; and Bryer, N; (2021)

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:
Research, monitoring and evaluation team
Email: rme.mailbox@gov.wales

Media: 0300 025 8099

Social research number: 75/2021
Digital ISBN: 978-1-80391-305-6

Image