Mid-term review of the Wales Centre for Public Policy

This review was undertaken to assess progress to date of the Wales Centre for Public Policy’s (WCPP) Ministerial work stream and to inform the second phase of the grant.

The analysis is based on in-depth semi-structured interviews with key informants supplemented with responses to an online survey of policy officials.

The review finds that the WCPP is a valued institution. It is viewed as a credible, independent source of evidence and expertise for Ministers to draw on, particularly in the early stages of the policy making process. It complements existing analytical resources and helps foster closer links between academia and government.

The review concludes that there remains a requirement for Ministers to access high quality timely evidence and expertise to support the Welsh Government’s strategic priorities, either through the WCPP model or an alternative arrangement. There are also benefits from having Ministerial and wider public services work streams within a single body.

However, the review also identifies areas where both the WCPP and the Welsh Government can improve in meeting current and future evidence needs.

Recommendations are made to ensure future WCPP Ministerial work programmes better reflect Ministerial priorities; that the projects are carried out to a more consistent standard; and that new approaches are considered for disseminating findings to maximise their impact.

Background

Evidence-based policy allows governments to make well-informed decisions by placing the best available evidence from research at the heart of policy development and implementation (Sutcliffe and Court, 2005). The Welsh Government has embraced the role of evidence in the development, implementation and evaluation of policy and is committed to the use of research to inform its decision-making process across policy areas.

Much of this expertise is provided in-house through Knowledge and Analytical Services (KAS) which provides analytical support throughout the policy cycle, and is able to undertake and manage large scale research and evaluation projects. To complement this, the Public Policy Institute for Wales (PPIW) was established in 2014 with the specific aim of increasing evidence-informed policy making through activities including working with Welsh Ministers to identify and address their evidence needs

The functions of the PPIW were absorbed into the WCPP, which was launched in October 2017. The WCPP is funded via a collaborative venture between the Welsh Government and the Economic and Social Research Council (ESRC) until September 2022. It has two distinct but interlinked work streams: one providing short turn-around expert evidence and advice for Welsh Government Ministers on policy relevant issues; and another working with a wider group of public services in Wales as part of the ESRC ‘What Works’ network. 

Following a tendering exercise led by the ESRC, a contract was awarded for the Wales Centre for Public Policy (WCPP) to Cardiff University, which held the previous contract for the PPIW. The Welsh Government provides funding of £450,000 a year to the WCPP for its Ministerial work programme (The Welsh Government’s Local Government Department contributes an additional £50,000 per year to support WCPP’s wider public services work programme, in addition to the core funding from the ESRC of £500,000 a year). As part of the contract, the Welsh Government requested that an interim review take place at the grant’s mid-point (by March 2020) to inform remaining period of the award.

The literature on think tanks and policy institutes examines the different roles they can play in working with governments. Some are characterised as carrying out research on longer term policy issues at arm’s length from government in a fashion similar to traditional academics, while others may have a closer relationship or provide advice on immediate policy concerns (Weaver, 2017). This review provides an opportunity to reflect on the model employed by WCPP and whether any changes should be introduced in the second phase of its grant.

Research aims

The overall aim was to review the progress to date of the WCPP’s Ministerial work stream and inform the second phase of the grant. The specific objectives were to:

• examine the process by which Ministerial evidence priorities are identified and the subsequent work programme developed
• understand, where possible, the impact of the assignments on developing policy
• review the joint approach to Ministerial research activities and wider public services research and the extent to which it is beneficial to have both functions within a single body

The review was jointly managed and undertaken by Cabinet Office and Knowledge and Analytical Services.

Methodology

The research design combined in-depth semi-structured interviews with a selected sample of key informants, supplemented with an on online survey of policy officials who were identified as having worked with WCPP.

The key informants for the semi-structured interviews were drawn from senior Welsh Government policy officials (11 interviews); Special Advisers (2); Knowledge and Analytical Services (1); Cabinet Office (1); WCPP and their Advisory group (4).  Interviews were also conducted with the First Minister of Wales and the Permanent Secretary. In total, 21 interviews were conducted between November 2019 and February 2020 with all but one of these being conducted face to face and one interview by telephone. Two interviewers were present at each interview. One to conduct the interview, and another to take extensive notes. Interview topic guides included core questions asked of every interviewee and other questions that were tailored to reflect the background of the specific interviewee. The interview data was supplemented by an online survey that was circulated to 40 Welsh Government policy and received 13 responses.

The review benefitted from the participation of individuals with influential policy roles, thereby providing authoritative insights (Woods, 1998). Accordingly, the interview approach adopted by the researchers was informed by relevant strategies for interviewing influential policy actors (Harvey, 2011). This included limiting the number of closed questions to allow interviewees to articulate their views thoroughly, balanced with keeping the interview to the appropriate length given the busy schedules of the interviewees.

Following data collection the interview notes and survey data were categorised and coded by four researchers. A series of overarching themes were identified inductively from the data, from which key findings have been identified. 

WCPP is valued for providing credible independent evidence and expertise

There was agreement across most interviews that WCPP provides a valuable role in providing evidence and expertise to inform policy making. It is viewed as a credible, independent source of evidence and expertise for Ministers to draw on, particularly in the early stages of the policy making process.

Some interviewees noted the distinctiveness of Wales having an independent research institute that advises government across multiple policy areas. They emphasised the value of knowledge brokerage between academia and Ministers, and bringing evidence into ‘the heart of government’; a model that was viewed by some as being unique to Wales.

The speed at which WCPP are able to complete projects to align with policy decisions was also noted by some, although others commented that projects often took longer than initially envisaged. Pinch points tended to be either at the scoping stage of projects where a lot of time could be taken in agreeing research objectives, or towards the end of projects where outputs may be published at a later date than initially planned.

Interviewees noted the importance of WCPP’s independence that gives extra credibility to the outputs.  The evidence and advice provided by WCPP was mostly viewed by interviewees as being complementary to that provided by civil servants (including KAS) and Special Advisers.

WCPP was seen as having a role in challenging received wisdom or conventional thinking on certain policy issues, and providing a new source of thinking that would not be easy to provide internally. When asked what impact WCPP advice had on policy thinking, some interviewees commented that they provided ‘confirmatory impacts’ i.e. not necessarily saying something different but that they strengthened their position and provided a sound footing for moving forward with a policy, backed by credible, independent evidence.

WCPP has a mainly demand-led reactive model for identifying Ministerial evidence needs

Interviewees recognised two different ways in which evidence needs were identified – reactive and proactive approaches. Reactive approaches tended to be short term assignments in response to Ministerial announcements, committee reports, or the need for a rapid assessment of evidence to inform policy formulation. Proactive approaches were less common but were characterised as those that anticipated what longer term policy issues might be.

WCPP has developed a mainly demand-led reactive model, in line with its contract for funding from Welsh Government.  While this has resulted in impactful work across a range of policy areas, some interviewees noted that there was a missed opportunity for such an institute to do more in taking a longer term view of evidence needs for policy formulation. There were however different views on who should be responsible in identifying these longer term evidence needs.

Work programmes would benefit from a strategic alignment of projects

Some interviewees identified an absence of a systematic approach to coordinating evidence needs and ensuring these were in line with helping deliver policy priorities. Work programmes can appear to be disjointed, rather than a strategic package of work that is joined up and coherent.

This led to the perception of a work programme comprised of seemingly disconnected short term projects rather than a programme of work that is focused on delivering against longer term strategic priorities for policy in Wales. Some interviewees commented that future work programmes would benefit from having a better balance between addressing short term and longer term evidence needs.

The analysis therefore identified the need for the work programme to be on a more strategic footing and better aligned to First Minister and Ministerial priorities. This lack of configuration is exacerbated by lack of consistent process for prioritising what the strategic evidence needs are.

There is a lack of understanding about how work programmes are developed

Interviewees were often unaware of the specific processes for getting evidence needs into the work programme. The interviews revealed a lack of a consistent, robust process for how evidence needs are fed into the WCPP Ministerial work programme and how work is commissioned. This inconsistency has led to a combination of processes that include WCPP speaking directly to individual Ministers to identify evidence needs, policy officials approaching WCPP directly, officials briefing Ministers on what their evidence needs should be, and Cabinet Office identifying evidence needs.

This variation has led to work programmes that lack coordination. For some of those projects that were identified by Ministers or Special Advisers, when officials subsequently became involved, the lack of understanding around context and purpose brought risks to the projects, including lack of ownership, engagement, difficulty in scoping and decreased impact. 

The absence of a clear process and expedient engagement from Welsh Government in identifying its evidence needs meant that the work programmes invariably took a long time to be developed. This often delayed commencement of projects and sometimes meant priorities had changed by the time the programme was ready to be signed off.

Tripartite relationships are a key feature of successful projects

A consistent finding was that the best examples of successful projects were those where the Minister, Special Adviser and key policy official were all in alignment about the purpose of an assignment. Tripartite relationships were the exception rather than the rule. However, it was also recognised that such alignment is not always in place, making it challenging for WCPP to develop impactful pieces of work.

Instances of Ministers instigating a piece of research that came ‘out of the blue’ for officials sometimes led to a misunderstanding of the context or need for the research, and often had a knock on effect on the trajectory of the projects and subsequent outputs. Some interviewees highlighted instances where a lack of communication with WCPP meant that projects did not meet expectations or answer the necessary research questions, thereby affecting the potential impact that the project could have.

The need for more dialogue was strongly supported, with interviewees recognising that effective collaboration amongst Ministers, Special Advisers and policy officials on the scope of the project, the research questions and how the research could inform policy can potentially lead to projects with more successful outcomes. How this is achieved in practice is not always straightforward and will depend in part on existing relationships. However, it was also emphasised that WCPP is a resource for Ministers, and therefore alignment between Ministers, Special Adviser and officials, while beneficial, is not a necessity for a project to go ahead.

The interest of WCPP in the project was also deemed as important, with the perception among some interviewees that there were some projects WCPP clearly had more interest in than others. Consistency and quality of service for all projects was viewed as fundamental, but not always evident.

The expert roundtables events are particularly valued

The roundtable events that WCPP have held were viewed positively by interviewees, with the gatherings of experts to debate topics thought of as especially useful when policy officials were also part of the discussions.

Having access to a network of experts and facilitating discussions of the evidence focused on issues specific to the Welsh policy context was identified as a highly valuable mechanism, and one that makes the most of WCPP’s niche for brokering knowledge between experts and government. Such events allow for flexible and dynamic discussions to be held between experts and policy makers, and have the potential to foster lasting connections that can be tapped into in the future.

In contrast, there was more variability in interviewees’ assessments of projects that had been undertaken in-house, or where specific external experts had been commissioned to write a report. It was also recognised that expert roundtable events are not always necessary or the best approach for addressing the specific research questions and that it would be more helpful for WCPP to draw on a combination of approaches for meeting different evidence needs.

The experience of WCPP suggests that it is the combination of roundtables with other activities and outputs that make for successful projects. For example, face to face and written briefings with Ministers, and written reports and commentaries for a wider audience, can help ensure learning is not forgotten and reach a wider group of people not in attendance at the roundtable.

Consistent quality assurance is needed when experts are commissioned to write reports

While there were many examples of WCPP producing impactful work, the quality of the work was sometimes described as variable by interviewees. Some interviewees queried whether the right experts had been approached and suggested that in some instances the identified expert was not suitable for the work required, or wanted to take the project on a different tangent rather than answer the specific evidence question posed by Welsh Government.

To address this issue, some interviewees suggested that there should be closer collaboration with officials about which experts to work with, and that more could be done by WCPP to manage commissions once they are underway to ensure more consistent quality control and make sure projects meet the original aims of the research. However, the view that there should be closer collaboration on which experts to work with was not universally shared, with such an approach also seen as impinging on the independence of the WCPP.

There is an appetite for clear findings and actionable recommendations

Some respondents suggested that WCPP’s outputs could be an issue, with a rigid system based on producing reports as an end product. Interviewees commented that the reports could be too long, too broad or unclear, too ‘academic’ and not always tailored to use in a public policy context. 

It was recognised that effective outputs could go beyond written reports and include face to face communication (for example, making more use of meetings between Ministers, policy officials and experts), lunch and learn sessions, seminars and ensuring relevant links are made in order to broker longer term engagement.

Crucially for those developing policy, report recommendations were often viewed as being too tentative or difficult to act on. In these instances, the result was that the report was not seen as being particularly helpful in providing the insight required for determining where a public policy response would deliver impact.  Interviewees did not feel they were getting a product that was able to clearly outline options for policy based on synthesis of the evidence WCPP had provided, thus allowing Ministers and officials to decide what actions to take in light of the evidence.

It was suggested that more consideration and awareness of the policy making environment needs to be developed, with specific recognition that Ministers and officials are often required to make decisions under time constraints. Accordingly, some interviewees commented that key findings of WCPP projects should be clearer for the lay reader and, where possible, recommendations should be clear about what action and options Government could take based on the available evidence.

An important nuance here is that some WCPP projects are more amenable to providing specific recommendations than others. For example, answering ‘what works’ type questions may be more suited to making recommendations than projects where WCPP are asked to look into the nature or extent of a perceived policy issue.

The wider public services remit has the potential to positively impact on delivery

While this review focused on the Ministerial work programme, interviewees were also asked about their views of having the wider public services remit within the same body. Some interviewees did not feel able to answer this question adequately but those who did saw benefits to keeping both functions in the same body. 

Specifically, having both remits within the same body was seen as bringing opportunities for work programmes to address evidence needs that are priorities for the Welsh Government and public services more widely. This has the benefit of avoiding duplicative efforts to address similar evidence needs, while allowing insights from either work programme to feed back in to discussions around policy and evidence requirements.

Some interviewees also noted that this has the potential for expertise to be built up on policy challenges that are facing Wales as a whole, and for this to inform policy direction at a national level and delivery at a more local level. For example, where WCPP focus on questions around ‘what works in implementing this successfully?’, this may have clear relevance to public sector bodies in charge of delivery.

It was also suggested that there was an opportunity for there to be more focus on capacity building across wider public services, and for WCPP to use their role to increase knowledge mobilisation between academia, government and public services.

Broader challenges exist around raising understanding of the role and purpose of WCPP

The research identified broader challenges in the Welsh Government around the use of evidence in policy making. For example, there was variation among interviewees’ understanding of the roles performed by KAS and WCPP and some respondents were unclear who should be approached to help address evidence needs and in what circumstances.

Some interviewees were able to express what they viewed as being a complementary role for WCPP to play, with KAS being better suited for longer term research and evaluative work, and WCPP offering a resource for Ministers to meet shorter term evidence needs, and to broker relationships between government and external experts. However, this understanding was not widespread, suggesting that engagement work is required to make this clearer.

Some respondents reported that they were not aware of WCPP (or its predecessor PPIW) prior to being involved in their projects but soon came to value their work. It was also clear that not all Special Advisers were aware of WCPP or the specific purpose they serve for Ministers. For some Special Advisers, there was also lack of clarity about how to access WCPP and bring new evidence needs to their attention.

While it was clear that work needs to be done to raise the profile of the work of the WCPP within Welsh Government, this review has also highlighted a broader need for clarity of roles between WCPP and the analytical functions offered through KAS, and how this should inform policy development, implementation and evaluation in the organisation.

Recommendation 1

The Welsh Government should develop a clearer process for developing strategic work programmes.

The current way of developing the Ministerial work programme makes for a system that is too ad-hoc, difficult to manage and not always aligned to strategic priorities. The First Minister has requested a more strategic focus for the WCPP. To meet these requirements several changes are proposed for the second phase of the grant.

It is recommended that Cabinet Office should continue to manage the Ministerial work programme given their existing access to and relationships with Special Advisers and the First Minister.

The first stage should involve Cabinet Office sourcing topics of interest from the First Minister and Special Advisers. This will take place on an ongoing basis through regular Cabinet Office meetings with Special Advisers and help to identify organisational evidence needs and pull them together into a strategically focussed work programme. Any evidence needs raised at this stage should be in line with one of the three themes of the Welsh Government (green, social and economic justice).

The next stage should involve Cabinet Office and KAS working collaboratively to ensure new evidence needs fit with existing or planned work; safeguarding against duplication of work KAS may already be doing; and where relevant, feeding in evidence and data from KAS projects to inform the WCPP projects. 

It is after this stage that the First Minister would be asked to formally approve or reject the projects for the new work programme. Following First Minister approval, Cabinet Office should notify WCPP of the new evidence needs and the relevant Ministers, Special Advisers, policy and KAS officials would become involved in scoping and specifying the projects in more detail. Cabinet Office should arrange any initial scoping meetings with WCPP being responsible for taking forward any follow up meetings. At this stage Cabinet Office will also inform Directors General and other senior officials of the work programme to ensure organisational awareness of both cross-cutting and departmental research being undertaken on behalf of Minsters.

Under this model, Special Advisers will play a key role individually and collectively in translating Ministerial requirements and converting their policy ambitions into research questions for WCPP to take forward. Therefore it is important that Cabinet Office work to raise awareness among Special Advisers of both WCPP and increase understanding of the service they can provide.  This includes developing the narrative around WCPP, KAS and other available evidence sources. Such evidence needs could also be facilitated through convening evidence needs workshops as well as attending Special Advisers meetings. 

These mechanisms are intended to produce Ministerial work programmes that will address more cross-cutting evidence needs and put the work programme on a more strategic footing. The programme should retain flexibility to accommodate specific requests from the First Minister as they arise or as the public policy context changes.

Cabinet Office and WCPP will work to address the pinch points in current processes, which tend to arise at the scoping stage of projects where a lot of time is spent agreeing research objectives and WCPP scoping the research. This will involve working with Ministers and Special Advisers to ensure there are clear statements of the research issue to be explored, alongside more timely and agile responses from WCPP. 

Recommendation 2

Changes should be made to improve project and programme management and delivery.

To ensure effective inception and delivery of the work programme, Cabinet Office should have a more defined role in providing governance and oversight. Following sign off from the First Minister of the work programme, Cabinet Office will act as a conduit between Special Advisers, policy officials and lead researchers and/or statisticians from KAS, coordinating initial project specific meetings and ensuring parties are joined up and informed about the way forward.

A broad project management system will be implemented and managed by Cabinet Office for internal use, helping develop a way of accessing the status of projects in the work programme, timeframes and dissemination plans. This will help to manage dependencies and make connections across policy areas.

Regular communication between WCPP and Cabinet Office about the work programme should become the norm, including monthly catch up meetings. Regular project specific communications will be encouraged between Special Advisers, WCPP and officials.

In order to ensure the work programme is in line with overarching priorities, Cabinet Office will support meetings between WCPP and Special Advisers, senior officials and the First Minister. This will facilitate development of future work programmes and to report progress on current work plans.

To develop shared understanding of the purpose of assignments, WCPP should also consider undertaking some light touch and proportionate theory of change work at start of projects to understand assumptions that are being made by Ministers, Special Advisers and officials and, if necessary, highlight where they are different.

Recommendation 3

WCPP should consider new approaches for disseminating findings in a way that will maximise their impact.

In seeking to effectively disseminate outputs of the work programme, WCPP should give consideration to the outputs being produced, with greater emphasis and awareness of the policy environment and getting key messages from the research effectively across to policy areas that would benefit directly or indirectly.

As part of this, WCPP should ensure that recommendations made are specific, clear and potentially actionable. Inevitably WCPP’s role is in synthesising existing evidence, but in considering its application to the Welsh policy context there may be scope to show the weight of evidence for different possible actions, helping Ministers and policy makers make decisions based on the best available evidence.

While it is recognised some form of a written report will usually be an output  (for reasons of transparency and organisational memory), it is recommended that WCPP consider alternative ways for disseminating their evidence that is not solely centred on a formal written report as an end product and allows non-academic audiences to engage with their evidence. 

This could take the form of facilitating face to face advice or meetings between experts and Ministers, recognising that such meetings do not need to be one off events and could be repeated once further consideration had been given to policy advice. 

Impact of research could be maximised through better engagement between WCPP, Special Adviser and officials once projects are underway to ensure the original research aims are met. The responsibility to engage lies with officials as well as WCPP, and Cabinet Office will lead on helping facilitate this.

Impact of research could also be maximised through better engagement between WCPP, Cabinet Office and the relevant Special Advisers before a written report goes to Ministers. Face to face communication was also seen as being key in this respect. Such meetings could be used to work through the nuances and implications of the evidence and to discuss the recommendations that could be actionable.

WCPP should also build and enhance existing mechanisms of dissemination, such as the Lunch and Learn sessions, to engage officials.  Other methods should also be developed for dissemination such as presenting at existing fora or using Ministerial or policy specific meetings.

Recommendation 4

WCPP should consider medium term evidence needs as part of future work programmes.

A key finding from the interviews was the need for a Ministerial work programme that is not only structured around immediate, shorter term needs; but one that also allows for identifying and addressing questions Welsh Government will need to grapple with in the medium to long term.

Governments face perennial issues across a number of policy areas including health, social care, education and tackling poverty. Therefore, there is value in WCPP considering evidence needs around such topics over a longer time frame than their short turnaround projects. It would be more of a challenge for WCPP to give actionable recommendations for such issues but that would not necessarily be expected for such evidence needs. Rather, the aim would be to develop the evidence base over time.

As such, consideration should be given as to how the Welsh Government can commission medium to longer term assignments from the WCPP alongside the shorter term, reactive pieces of work. Building a work programme that allows for medium to longer term issues enables identification and understanding of what the potential evidence gaps are across priorities. This supports the future proofing of policy development.

Recommendation 5

The Welsh Government should give consideration as to what the remit and functions of a future institute for policy research in Wales should be.

The review concludes that there remains a requirement for Ministers to access high quality and timely evidence and expertise to support the Welsh Government’s strategic priorities, particularly in the early stages of the policy making process.

It is recommended that in the remaining two years of the WCPP contract, consideration should be given as to whether such support continues through the current WCPP model or an alternative model or arrangement.

In the first instance, the recommendations outlined in this review should be implemented for the remainder of the WCPP contract to improve ways of working, build a strategic Ministerial research work programme that delivers high quality outputs for Welsh Government and embed governance arrangements. This will be done while retaining the demand led, short term projects driven by First Minister priorities.

The opportunity should also be taken to address strategic questions of what Welsh Government needs and wants from a policy research institute. These would explore issues such as fundamental purpose, remit, focus and function.

This period should also be used to pilot different ways of working to help inform any future institute funded by Welsh Government. This could include for example, expanding the Ministerial work programme to allow space for WCPP to undertake assignments that address medium term evidence needs i.e. those issues that are not urgent priorities for Ministers now but may become so over the next two to three years. 

Helping resolve the issue of identification of medium to long term needs allows Ministers and Special Advisers to improve the way they are able to individually and collectively identify what Wales’ future policy and evidence needs might be, allowing analysis of the scale and nature of a problem. This is in line with the Welsh Government desire to become an innovative policymaking organisation.

While Welsh Government will continue to drive the priorities, there is also room for a process that allows a wider repertoire of ideas and experts to be part of the discussion, opening up the debate around complex policy issues to a broader set of ideas to draw on. It also allows issues to be addressed that are more cross cutting in nature, such as tackling poverty.

This means WCPP being proactive in working with Welsh Government in things that it might want to think about, working collaboratively with Ministers and Special Advisers to focus on key medium to long term evidence needs and raising issues that officials may not have considered. The experts who sit on the WCPP Advisory Board could also play a role in advising on longer term policy issues for consideration.

It is recognised these pieces of work are less likely to produce straightforward recommendations. However, they should help build the evidence base to inform Ministers and officials in developing policy thinking on emerging issues.

A model for commissioning medium term pieces of work could be piloted in the second phase of the current contract, with a view to embedding it in any future funded policy research institute.

Authors: Dr Angela Martin and Ian Jones (Welsh Government)

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

Ian Jones
Telephone: 0300 025 0090
Email: research.publicservices@gov.wales

Social research number: 30/2022
Digital ISBN 978-1-80364-022-8

Image