Sustainable Management Scheme: evaluation toolkit

The purpose of this document is to provide practical advice and highlight key areas specifically for SMS projects to enable them to embed good practice into their data collection, monitoring and preparing their final project evaluation.

There is detailed  guidance available for carrying out evaluations at HM Treasury Magenta Book and more specifically for RDP Measure 16 Co-operation and Supply Chain Development Scheme Projects, at CSCDS Evaluation Guidance (Gov.Wales). The Sustainable Management Scheme falls under this measure so this guidance applies to SMS and will guide you through the monitoring and evaluation requirements.

This document is supplementary guidance to the above and has been written with SMS measure 16.5 in mind allowing for a more targeted approach which will enable Welsh Government to assess the impact of SMS and to inform policy decisions for the future.

Monitoring is an important part of each SMS project, successful monitoring of activities, data collection and its management enables each project to evidence any outputs and the impact they made against their aims and objectives.

Projects need to be evaluated in order to competently make judgements on the effectiveness of the funding, interventions and decisions taken.  Evaluation is a continuous process to collate meaningful information and provide valuable insights to the audience.

As part of the requirements for the Sustainable Management Scheme - Welsh Government Rural Communities - Rural Development Programme 2014-2020, which is funded by the European Agricultural Fund for Rural Development and the Welsh Government, an independent project evaluation is required from each project. This is to allow Welsh Government to successfully evaluate schemes and programmes and collate a beneficial evidence base to inform future decisions.

The individual project evaluations will enable Welsh Government to take the learnings and outcomes from each collaborative project and incorporate this into an evidence base to inform developing future support schemes in Wales. Between now and the end of the delivery period, which is set for June 2023, we expect to see and are seeing actions contributing to very clear outcomes; Carbon Sequestration, enhanced biodiversity, ecosystem resilience, climate change and adaptation, reduced flood risk, rural skills, culture and heritage, community cohesion, health and wellbeing, improved soil, air and water quality and the development of skills, capacity and expertise. We would like to capture this effectively so individual project evaluations are paramount to this exercise.

An external independent scheme evaluation has been commissioned and OB3 Research, in conjunction with BRO Partnership, has been appointed by the Welsh Government to undertake a scheme wide evaluation of the SMS. The aim of the evaluation is to assess the effectiveness of the SMS as a mechanism for delivering the Sustainable Management of Natural Resources (SMNR) in Wales. The evaluation will focus on both the value of the collaborative approaches in delivering SMNR and the success of the scheme in achieving its outcomes. Understanding the success of the SMS as a way of delivering SMNR and the value of collaborative approaches. The evaluation therefore needs to focus on both the processes and mechanisms through which the scheme is delivered, and the outcomes achieved, providing an assessment of both the physical (environmental) and socio/economic outcomes of the SMS.

The scheme evaluation is intended to assess;

• The alignment of funded projects to the principles of SMNR
• The management and implementation of the scheme
• The nature and extent of collaboration and what this achieved
• The outcomes of the scheme and action on the policy priorities
• The contribution to cross-cutting themes, particularly climate adaptation and mitigation

The evaluation is being undertaken between January 2020 and October 2022 and involves the preparation of four reports; an initial Theory of Change (ToC) report which was completed in May 2020; two update reports and a final evaluation report in October 2022.

Projects will have the opportunity to contribute views to the WG’s independent external evaluators as lead and partner organisations, contributing to annual bilingual web surveys to tell them about the experience of applying and securing SMS funding and the experiences of delivering projects.

There will also be an opportunity to contribute to the evaluation through an interview with a researcher, on a face to face or virtual basis. The intention is to visit each project on at least one occasion over the course of the scheme evaluation to interview project leads, project partners and (where relevant) project evaluators.

The data collection, monitoring and final evaluations of your individual projects will feed into this wider evaluation.

Over the course of the RDP 2014 – 2020, SMS projects will get the opportunity to share relevant project documentation; progress reports, monitoring data, project level reports, spatial data alongside scientific publications and findings. All of this data is collated to form an evidence base with the final report evaluating the impact and usefulness of the interventions. We use these findings to inform future decisions and help us understand and demonstrate how collaborative, landscape scale working can be successful.

The purpose of this toolkit is to provide information to funded projects (and their external evaluators, where relevant) on:

• What we need from you: the information and evidence that we would find helpful from you to inform Welsh Government and the external scheme level evaluation.
• How you could approach your project level evaluation to be most effective
• Identify key research questions
• Identify a suitable methodology

We appreciate that funded projects are working to different timescales and have different approaches in place to evaluate their activities. As such, this toolkit should be considered by projects as supporting guidance and advice rather than a prescriptive approach which must be adopted.  

• The aims and objectives of your project – what it’s aiming to achieve and the key issues it’s trying to address and whether the aims and objectives were met and how?
• The value and effectiveness of any interactions, partnerships or collaborations that you may have been part of as a result of SMS funding?
• The effectiveness of your project delivery model, i.e. a bottom up approach, what has worked well and what hasn’t?
• What outputs has your project achieved against targets set; what difference it is making and what impact it has had in relation to natural resources, communities and to the economy?
• What multiple benefits have been realised when achieving your aims and objectives?
• Any lessons and important learnings which you have learned to date from implementing your project in this way?
• The extent to which any impact from new ways of working can be sustained post funding.
• Could your project have gone ahead without SMS funding and reasons behind that?

For those projects working over a longer timescale it may be appropriate to consider undertaking a mid-term and a final evaluation of project activities. In doing so, it would be useful to draw upon project documentation (e.g. funding application, progress reports, minutes of partnership meetings etc) as well as project monitoring and output data.

It would also be appropriate for the evaluation to draw upon the views of all project partners as well as representatives from the local community, including where appropriate community groups or associations and business representatives that were involved in the project to capture more intangible benefits. These views could be captured via interviews, focus groups or short surveys and for detailed guidance on methodology please refer to the links in the introduction.  

Section 4 provides a detailed template for projects to adopt when preparing project level evaluation reports.

Section 5 offers suggestions on the type of questions projects or their evaluators could ask of project partners, community members and other stakeholders to address these objectives.

This section provides some suggestions on what your project evaluation report(s) could cover. Some ideas are provided which projects (and their independent evaluators) are welcome to modify and use for preparing their evaluation reports if they so wish.

Introduction

This section could include:

• Background – a short section providing a brief introduction to the project and partnership
• Structure of report – a simple outline of the following chapters and what they will include

Overview of the SMS project

This section could succinctly set out:

• Your project aims and objectives
  • What did it plan to do and how did you plan to do it?
• The perceived need for the project and the rationale for it
  • Where did the idea come from, how was it developed, what issues was it aiming to address?
• Your partnership membership, structure and development
  • Steering groups
  • Community groups
  • Responsibility of decision making
• Your project delivery model and delivery arrangements
  • How did you plan the work, which methods did you plan on using, how was that managed?
  • What staffing or volunteer arrangements were in place?
  • What was the timescale and budget for the project?
  • Was this reasonable and proportionate?

Implementation of the SMS project

This section could review the effectiveness of how the project has been implemented, including:

• Effectiveness of partnership working and extent to which a wide range of organisations, groups and individuals have been involved
• Effectiveness of project delivery model
• Effectiveness of governance and management arrangements
• Effectiveness of project level communication and interactions
• Effectiveness of project monitoring and reporting arrangements.

Methodology

This section could outline the methodology used in the evaluative process, the research activities which have been undertaken to gather the evidence presented such as desk review, interviews or surveys with project partners and community members, ‘before and after’ photographic evidence, scientific recording and counts and any published reports.

Achievements, outcomes and impact

This section could review the evidence that the project has, to demonstrate the outcomes achieved by the project and the difference it has made, including evidence of:

• Contribution and difference made by the project to natural resources and ecosystem resilience  
• How projects have adopted and contributed to the SMNR principles in their way of working
• Contribution and difference made by project to local community and society
• Contribution and difference made by project to local economy
• Contribution and difference made by project to local peoples’ health & wellbeing
• Extent and value of any collaboration, sharing of knowledge and innovation developed with other SMS funded projects
• Extent to which project outcomes relating to ecosystem resilience will be sustained post SMS funding 
• Extent to which project partnership will be sustained post SMS funding
• Extent to which project jobs and training will be sustained post SMS funding

Conclusions and recommendations

This section could set out the evaluator’s conclusions on the implementation, outcomes and impact made by the project. Set out the lessons learnt and any key learnings and recommendations. It could highlight the multiple benefits a project may have achieved and highlight the impact.

It could also set out clear recommendations to the project management staff for the remaining project delivery period (in the case of a mid-term report) as well as transferrable lessons for the future.

There are several pilot innovative approaches being developed and existing successful sustainable management of our natural resources across the projects. The judgements and recommendations from the evaluations will be used to inform future policy and the outputs and learning from these projects will be used as part of the evidence base in developing future support schemes in Wales. So it is vital for projects to share their experiences, knowledge and success with Welsh Government and the sector to allow for future funding decisions to be well evidenced and effective. An executive summary is a very useful tool for the audience to see and would be recommended to do alongside the more detailed review.

This section offers suggestions on the type of questions which project evaluators could ask of project partners, community members and other stakeholders to inform their evaluation report and to address the project level evaluation aims and objectives set out in this toolkit.

Background

• Outline your role, organisation, and involvement with the SMS project

Aims and objectives of the SMS funded project

• What is your project aiming to achieve; highlight strategic aims and objectives that were set out in the application?
• What key issues is your project trying to address?

Monitoring and data collection and methodology

• How did you monitor your outputs and in what formats?
• What organisations helped you capture and report on findings?
• Did you record feedback to capture intangible benefits such as how a person felt about being part of something?
• Did you capture photos, videos, case studies, interviews, podcasts or blogs?
• What did you find difficult in your monitoring?
• What action have you taken to measure and capture the social value and impact of your actions?
• Have you captured qualitative and quantitative data?

Progress, managing change and identifying lessons learnt

• What environmental or economic impact have you identified as a result of your SMS funded project outputs?
• What progress has been made towards meeting your original objectives?
• What actions did you take to go about achieving your outcomes?
  • How did you identify the actions needed?
  • How did you prioritise them?
  • How did you implement them and by whom?
• If your original planned actions could not be achieved - what were the reasons for any changes you implemented how did you identify change was needed and how did you manage adaptively?
• What are the strengths and weaknesses in what has been achieved?
• What did you do to mitigate risk?
• What external factors have affected the project’s achievements?
• What nature-based solutions has the project achieved to contribute to the following outcomes:
  • Carbon Sequestration
  • Enhanced Biodiversity
  • Ecosystem Resilience
  • Climate Change and Adaptation
  • Reduce Flood Risk
  • Rural Skills, Culture & Heritage
  • Community Cohesion
  • Health and Wellbeing
  • Soil, Air and Water Quality
  • Development of Skills, Capacity and Expertise

Collaborative working

• How was your group/partnership established?
• How is your group/partnership structured?
• How is your group/partnership funded outside SMS?
• How large/small is your group/partnership?
• What barriers have you faced as a group/partnership?
• How effective is partnership working and collaboration within your project in your view?
• What is working well?
• What are the benefits to partnership, collaborative working?
• What added value is being secured from this collaboration?
• What are the challenges and how have you overcome them?
• How effective is your project’s collaborative delivery model?
• What good practice have you seen/experienced?
• What are the lessons learnt from working collaboratively?
• Have you connected with other SMS projects/collaborations in order to transfer knowledge, data or innovations?
• Who/what were these and how did you go about it?
• What have you shared and how did you go about sharing this knowledge and/or data?
• How effective are the governance and management arrangements (e.g. structures such as steering groups, reporting mechanisms) which are in place for your project? What are the main reasons for this?

• Are there any strengths and weaknesses of those mechanisms?
• How familiar were/are the partners in working together?
• What role did the facilitation support service (if any) play in supporting this?
• If you did not receive the facilitation support service, what other support if any did you receive?

Socio-economic and Health & well-being

• What benefits/outcomes and impact has your project achieved for the local community and society
  • e.g. rural skills, job creation, training, engagement with schools/community, improvements to local heritage, arts, Welsh language?
• What benefits/outcomes and impact has your project contribution made to the local economy?  
  • What benefits/outcomes has your project achieved for peoples’ health & wellbeing e.g. physical benefits, stress reduction, improved mental well-being?
• What actions have you taken to tackle social inclusion and to improve participation and enhance opportunities for disadvantaged groups?

Funding & Long term sustainability

• How likely would any project outcomes have been achieved without SMS funding?
  • Will the scale of the landscape management previously funded continue?
  • Will staff and jobs be retained?

• To what extent will the project partnership be sustained post SMS funding?
• What discussions are currently being held about sustaining the partnership post funding?
• What further support or assistance would you need to help achieve this sustainability post SMS funding?
• What is the legacy of your SMS project?
• Has the project partnership secured any other funding to support its future work? If so, what sources and levels of funding have been secured?
• What lessons have been learned to help ensure long-term sustainability for future projects and activities?