Evaluation of the ReAct III programme: 2015 to 2019 (summary)

The Welsh Government commissioned Wavehill, working in association with the Learning and Work Institute, to undertake an evaluation of their ReAct programme as it operated from 2015 to the end of 2019 (formally known as ReAct III).

It is important to note that this evaluation was completed prior to the outbreak of the COVID-19 pandemic in Wales in 2020. There is, therefore, no reference made to the impact of the pandemic on the ReAct programme within the report.

The programme

ReAct III builds on a well-established delivery model that has been in place since the original programme was established in June 2004. The programme provides support to address the needs of people who have been made redundant or are under notice of redundancy, through a series of measures designed to remove barriers to obtaining new employment. The main aim is to respond quickly and positively to all redundancy situations through a series of measures designed to alleviate the negative effects of redundancy. It aims to provide all redundant individuals with the skills necessary in order to secure new, sustainable employment in the shortest time possible.

The three strands of the programme are:

1. a vocational training grant for people who need to update their skills to return to work
2. extra support to help remove any barriers to vocational training (for example, reasonable travel, childcare, accommodation, and special equipment for participants with additional needs)
3. a contribution to wages and help with training costs for recruiting employers

ReAct is designed to complement and supplement the services offered to redundant workers by Jobcentre Plus and Careers Wales. It is an all-Wales programme funded by the Welsh Government and, in the West Wales and the Valleys region, the European Social Fund (ESF).

The evaluation and methodology

The evaluation was focused on the performance and impact of the programme, with a review of the programme design and delivery processes.

With the fieldwork undertaken from 2017 to the end of 2019, the methodology included:

• a review of literature relating to responses to redundancies
• consultation with a range of programme stakeholders, including Welsh Government officials, Careers Wales staff, and a range of external partners (including Jobcentre Plus staff) with involvement with ReAct
• consultation with supported individuals via a telephone survey of 1,155 participants[1] and in-depth interviews with 50 participants
• consultation with supported businesses via a survey (70 responses) and in-depth interviews (15 responses)
• four case studies focusing on ‘live’ redundancy situations that happened during the lifetime of the evaluation, including consultation with 180 individuals made redundant
• an impact and cost–benefit analysis

[1] Via the ESF Leavers Survey in West Wales and a survey specifically for this evaluation in other parts of Wales (both using the same research tools).

Fit with policy and the cross cutting themes

As the only programme specifically targeted at individuals being made redundant, ReAct has a clear role in the delivery of Welsh Government objectives as set out in the national strategy, i.e. Prosperity for All, and the Economic Action Plan. In terms of meeting the WEFO’s strategic objectives, ReAct fits with Priority Axis 1: Tackling Poverty through Sustainable Employment, and with Specific Objective 1: To increase the employability of those closest to the labour market who are at most risk of poverty, targeting individuals who have been impacted by redundancy.

ReAct is, as the name suggests, a reactive intervention, providing support to individuals and businesses as the need arises (specifically in redundancy situations). The programme does not target any specific part of Wales or groups of businesses or individuals beyond that criterion. Furthermore, there is an emphasis on the independent nature of the advice being provided, which is ‘person-centred’ and designed to provide support which is appropriate for the individual in question whilst also taking into account the local labour market.

This approach limits the potential of ReAct as a vehicle through which to proactively promote policy objectives, including those for the Welsh language or the programme-level Cross Cutting Themes. This is not to say that ReAct cannot contribute to achieving those policy objectives; rather, it does so in a reactive (demand-led) way. For example, a Careers Wales advisor will reference the Welsh language if it is appropriate to the individual, area or sector being discussed. The programme can therefore contribute to achieving Welsh language policy and other cross-cutting objectives. The primary drivers for the support being provided are, however, other factors such as local labour market conditions and the priorities for the individual in question.

Management and delivery

All three previous evaluations of ReAct (2008, 2011 and 2016) found that the programme has been implemented effectively, and the findings of this evaluation are the same for the ReAct III programme. However, it is acknowledged that the programme remains ‘admin-heavy’, with stakeholders highlighting the administrative complexities of delivering an ESF-funded programme.

The role of Careers Wales

Careers Wales plays a critical role in the delivery of ReAct, both as ‘gatekeepers’ to applications for support (including an important ‘quality control’ role which helps to ensure that only appropriate applications are submitted to the Welsh Government team) and in terms of the effectiveness of the information, advice and guidance (IAG) that they provide to individuals. That IAG role is crucial in terms of the ability of ReAct, via the training that it subsequently funds, to deliver positive outcomes.

The literature review found that job search assistance can have a positive impact on individuals’ chances of moving into work, especially in the short term. The integration of ReAct support with the broader range of employability support on offer from Careers Wales is, therefore, an approach that the evaluation would support.

Careers Wales advisors were strong supporters of ReAct for two reasons: firstly, for the benefit that they perceived individuals to gain from the support that it provided, and, secondly, because the prospect of receiving funding with which to undertake a training course was an important ‘carrot’ that attracted individuals to meetings with them. Again, this underlines the value of ReAct as part of a broader package of support for individuals seeking work. 

The literature review found that training can have a positive impact on movement into work, especially in the longer term. However, the evidence suggests that the impact of training programmes can vary greatly, and that good design is vital for their effectiveness. This suggests that the ReAct team need to ensure that the training that participants choose to undertake is effectively designed. This is part of the discussions between the ReAct participant and the Careers Wales advisor, although the assessment process undertaken by the Welsh Government team will also include a review of the training course that the applicant wants to undertake, so as to ensure that it is appropriate.

Targeting support

ReAct is available to anyone who has been made redundant. Although most stakeholders did not believe that ReAct should be more targeted, some acknowledged the rationale of targeting support at those with lower levels of qualification on the basis that, generally, they are more likely to need support in finding a job after having been made redundant.

The literature review supports that view. Policy documents also emphasise the importance of targeting support at those who are most in need of it. The general view of most stakeholders was, however, that ReAct should remain available to anyone who is made redundant, on the basis that even those with high levels of qualification can have gaps in their CV and skillset which need to be addressed when they are made redundant. The impact that redundancy can have on the mental health of individuals, which does not take into account experience and qualifications, also needs to be considered. 

It is interesting to note that the analysis of monitoring information finds that participants with relatively low levels of qualification (at Level 2 or below) are underrepresented in all three strands of ReAct, while those with qualifications at Level 4 or above are overrepresented in all three strands. The ‘open to all’ approach could therefore be leading to a situation in which those who (it could be argued) are most in need of support are underrepresented within the programme. Alternatively, as ReAct is demand-led, it could suggest that there is less demand for support from those groups, or this could potentially be an awareness issue amongst that group.

In truth, we cannot be certain as to why those with lower skills seem to be underrepresented within ReAct. It perhaps suggests a need to target those groups in some way to ensure that they are engaged.

The evaluation has found no evidence suggesting that the training courses that have been funded by ReAct are not well designed. However, the fact that the design of training programmes is identified by the literature as being so important suggests that it is something that should be reviewed on an ongoing basis.

Variations in the approach

The redundancy case studies illustrated that the scale of the response to redundancies in Wales can be substantial. This was the case with the closure of the Tesco call centre in Cardiff in 2017 (Tesco House), which witnessed a loss of 1,100 jobs, with a Ministerial Taskforce being set up in that case. However, the appropriateness of the scale of the response is an interesting question to consider.

Any large-scale redundancy justifies a response. However, for the case studies considered in this report, it would seem that it was the scale (and profile) of the redundancy which drove the response (rather than the need for support). For example, while the Tesco House redundancy was large-scale in terms of the numbers involved, it was described in some quarters as being ‘less challenging’ than other redundancies due to the high skill level of many of the individuals concerned and the relative buoyancy of the local job market. Moreover, substantial notice was given that the redundancies were going to happen, providing an opportunity to design an effective response.

The question is whether a less high-profile redundancy, with perhaps fewer jobs being lost but much lower prospects of local re-employment, should generate the same response. This is a difficult question to answer, although we can state that the response to each of the redundancy case studies examined for this evaluation has been comprehensive, if not perhaps as comprehensive or high-profile as in the Tesco House example.

The case studies have shown that whilst the package of support available was generally consistent, how it was delivered was adapted according to the characteristics and circumstances of each particular situation. For example, the approach for Allied Bakeries was different from that for Tesco House, taking into account the particular needs, strengths, weaknesses, etc. of the workforce in those instances. In our view, this is a positive finding and reflects the knowledge and experience that have developed within the teams delivering the support over a considerable period of time. Local knowledge and understanding were considered to be essential to the effectiveness of the support provided (e.g. understanding of the local labour market), which, again, supports the delivery of ReAct via local Careers Wales teams.

The relationship that can be built with the businesses in which the redundancies are taking place is very important. The better the relationship, the more effective the response that can be put in place. Linked to the above, the amount of time that is available to prepare a response to the redundancy is important, the longer, the better.

A main finding of the evaluation, however, is that the structure that is in place to respond to redundancies in Wales is now so well established that it can respond very quickly and very effectively, as was the case when Quinn Radiators closed its doors with no prior warning. The speed at which the response was prepared in that instance was due in large part to the fact that the process is now well established with teams and relationships in place and effectively ‘ready to go’. What is not clear is whether that response is replicable across Wales, especially in more rural locations, wherein fewer resources will be available and those resources will be more dispersed. Case studies for redundancies in a more diverse range of locations would be needed in order to explore this further.

Why some participants do not access ReAct support

It was found that the most common reason as to why individuals had not utilised ReAct support (when they were aware of it) was that they wanted/needed to get back into employment as quickly as possible and/or felt that additional training was not needed for them to find work. In-depth interviews with participants, however, highlighted the fact that individuals can find it difficult to fully consider their options in the immediate aftermath of being made redundant, and may make decisions that they later regret, such as moving back into employment too quickly.

Mental health support

Redundancy has an impact on the mental and physical health of those made redundant. Accordingly, one area in which there is potential to add to the support package available is that of the mental and physical health support provided to individuals.

While Careers Wales advice and ReAct support following redundancy are not a substitute for mental health support (other than referrals to specialist support), it is significant that many participants identified positive mental health outcomes as being a consequence of participating in ReAct.

The case studies also highlight that the impact of being made redundant for people is wider than simply losing a job, especially where they have been employed in the same place for an extended period of time, potentially losing a social and support network upon which they have become reliant in many ways.

Outcomes for individuals supported

Of advice from Careers Wales

The evaluation found that those who received advisory support from Careers Wales were generally positive about it, with respondents feeling that it gave them a better understanding of how their skills and personal qualities could be applied to the job market and their career. In most cases, however, the advice provided was not perceived to have led to a change in the action that the individual in question wanted to take. Rather, it confirmed and helped to implement a ‘direction of travel’ that was already in place in the mind of the participant. As discussed later, this is consistent with a more general finding that ReAct supports positive outcomes, rather than being the cause of them.

Of vocational training

The evaluation found a clear and positive impact in that most of the individuals made redundant who were supported by ReAct quickly moved back into employment. 80% of respondents to the survey of individuals supported by ReAct were in employment six months after the training course finished. Nonetheless, whether this can be attributed solely to ReAct is less clear.

In most cases (61%) the roles and sectors in which people worked had changed following their redundancy, sometimes in a positive way and sometimes in a negative way. On the positive side, respondents reported that they had secured employment that was higher-paid than their previous role, and that they enjoyed their new role more than their old one. The in-depth interviews with individuals highlight the work–life benefit that some have secured as a result of a change in their occupation. It seems to be clear that ReAct support in many instances has helped to facilitate those changes. In other instances, however, the change has been negative, with individuals reporting that they are now paid less than they were previously and that they preferred their old job. This is probably inevitable in a situation in which people have lost their jobs involuntarily.

In discussions, Careers Wales advisors were always keen to stress that individuals in many instances have no choice but to accept the first job that is offered to them, as they need the income. Often that job is not equivalent to the job that they previously held and/or does not fully utilise their skills, qualifications, and so on. What is more, the economic output that they generate is not maximised in such circumstances.

Furthermore, there will be circumstances in which individuals have moved into a job that is not related to the support that they received via ReAct. For example, ReAct may have funded training designed to help the individual to gain employment in Sector A but they have accepted a job in Sector B because they needed a job. In those circumstances the individual will not state that they secured their job due to the support that they received from ReAct. They still, however, have ambitions to move into a job in Sector A if and when a job becomes available, which means that the support that they have received could yet have a positive impact.

This issue is one of a number leading to a suggestion that it may be appropriate to consider whether ReAct support should continue to be made available to individuals who have been made redundant even if/when they have secured employment. There will be circumstances in which such support could have a positive impact on someone’s career prospects and, thereby, maximise the economic contribution that they can make.

Variations in outcomes

The finding that older participants are more likely to have witnessed a larger decrease in their salary is interesting to note. One interpretation of this finding is that older participants are finding it more difficult to find equivalent employment. However, several other factors are likely to have an influence, including that they may be less flexible in terms of the location of their employment than are their younger equivalents, or that they received a more generous redundancy package. We can only speculate on this matter.

The evaluation has found that outcomes of ReAct support are likely to be ‘better’ for those with higher-level qualifications. They are more likely to have found a job since being made redundant. However, the analysis suggests that ReAct support plays a less important role in driving that outcome (obtaining a job) for those with higher-level qualifications than for individuals with lower qualifications. In other words, ReAct support is more valuable to those with lower levels of qualification. Conversely, we have found that those with lower levels of qualification (who would benefit the most from the support being provided) are underrepresented within the ReAct programme.

Further research, using appropriately matched counterfactual groups, may provide further important insight in this area. However, the finding adds further weight to the argument that there should be at least some element of more targeting of support at individuals with lower levels of qualification. It is important to note that this does not necessarily mean that support should be withdrawn from those with higher levels of qualification, as targeting could involve the provision of additional support to those with lower qualifications. 

Attribution

To assess the impact of an intervention, it is important to identify which of the outcomes observed would have happened anyway. Without the identification of a robust counterfactual, such as a control group, it is not possible to quantify how many of the respondents to the survey would have found employment anyway. The survey and in-depth interview data do, however, provide some indication, as respondents were asked to provide a view on the influence of the training course undertaken on achieving the employment outcome.

A minority (14%) of survey respondents identified the training funded by ReAct as having a direct impact on whether they had been able to secure employment, with 31% reporting that it had made no difference to their employment outcome. It is important to note that this is probably not surprising, given that this is a group of people who have recently been in employment and given the relatively positive economic conditions during which the support was provided. The literature review also identifies similar findings in previous evaluations of the ReAct programme. 

Of more relevance perhaps is the fact that more than half of the respondents (55%) identified vocational training as something that had helped them to enter employment. The in-depth interviews generally told the same story, although they also highlight the key role played by ReAct support in some instances. We can conclude from this that ReAct has played a positive, if not critical, role in generating the employment outcomes that have been identified, especially where the individual in question had lower qualifications.

Outcomes of employer support

Turning our attention now to the employer elements of ReAct, previous evaluations have suggested that there was high deadweight associated with the employment subsidy and the in-work training being provided. Whilst it needs to be noted that the sample of employers surveyed for this evaluation is relatively small, the findings concur with that view: a high percentage of people would have been recruited anyway, even if the subsidy were not available, and the in-work training would largely have proceeded anyway.

Most of the businesses interviewed had utilised the employment subsidy element of ReAct (Employer Recruitment Support: ERS) but not the grant to support further training (Employer Training Support: ETS). In the majority of cases this was because it was not considered that additional training was required.

It is positive to note that a relatively large number of respondents (20, mostly microbusinesses) that had received ERS funding reported that the individual recruited had progressed to a higher-paid role within the business. A similar proportion of respondents, however, reported that employees recruited via ReAct were no longer employed by the business. There is, therefore, no clear finding from the survey in respect of the longevity of employment supported by the employer elements of ReAct.

The benefit that most businesses identified as resulting from the support that they received from ReAct was, unsurprisingly, financial: the funding that they received. However, if we look specifically at microbusinesses, recruiting workers with good work habits and a good work ethic was identified as the greatest benefit, although the small number of responses in each cohort (again) needs to be considered. Moreover, the benefits were in these circumstances, considered to be substantial.

A main finding, however, is that in the vast majority of cases the recruitment and the training were likely to have happened without ReAct support. Given that this is the case, questions must be asked about the appropriateness of the employer support elements of ReAct when going forward. The only caveat to this is that some businesses (usually microbusinesses) have benefitted substantially from the support that they received.

We conclude that if employer-related support is to be maintained, a greater degree of targeting at microbusinesses should be considered. Furthermore, there is a need to consider whether there is a strong-enough rationale for targeting a wage subsidy specifically at individuals who have been made redundant, rather than a more general wage subsidy scheme for microbusinesses. 

The literature review found that wage or recruitment subsidies can be effective in helping individuals to move into unsubsidised work, albeit only if they are well designed. Time-limited wage subsidies are better (as is the case with ReAct) and if targeted at disadvantaged workers, such as the long-term unemployed (which is not the case with ReAct). Such factors need to be considered when going forward.

Targeting in that way has been found to reduce the extent of deadweight, although substitution effects remain substantial. However, by bringing such disadvantaged workers into a job, wage subsidies enhance the supply of labour available to employers. In short, wage subsidies enable individuals with greater disadvantages or greater barriers to work to take jobs that would otherwise have been taken by workers who are more immediately employable. Thus, even with considerable substitution effects, hiring subsidies targeted at disadvantaged workers can be justified on both these positive economic grounds and the equity grounds of assisting groups who face particular difficulties in the labour market.

Impact assessment and cost–benefit analysis

Our analysis has estimated that the net additional impact of ReAct is to increase the number of people in employment over and above that which would have occurred in the absence of the programme by a total of 1,432 in Wales as a whole. Deadweight is identified as a concern in this report, as it has been in previous evaluations of ReAct, and this figure reflects such concern, with the 1,432 net jobs being generated from a total number of 8,228 participants.

A conservative assumption of 80% deadweight has been adopted for the analysis, with the employment subsidy element of ReAct assumed to be towards the top end of the range suggested by the literature review (at 90%). 

Despite this, the analysis still finds that ReAct does represent value for money, with a benefit-to-cost ratio of (at least) 1.58 (which, according to central government guidelines, represents acceptable value for money). That ratio would, however, likely improve with the implementation of the recommendations made earlier in this conclusion.

The analysis that had been planned for the impact assessment for this evaluation could not be undertaken due to weaknesses in the management information available. The original intention was to use a matching approach to assess the net impact of participation in ReAct against the counterfactual of non-participants. However, data on ReAct participants are not included in Lifelong Learning Wales Record (LLWR) data, meaning that this analysis was not possible. What is more, the alternative matching of ReAct management information on participants against data from the Labour Force Survey was not possible due to a lack of data, meaning that a more robust impact assessment was not possible. This should be addressed when going forward.

Recommendation 1

To allow potential gaps in support to be identified and addressed, there needs to be more ‘live’ monitoring of the characteristics of ReAct participants, the types of redundancies (e.g. sector and size of business), the locations, and the sectors where support is being provided via ReAct.

Recommendation 2

The potential need to take action to ensure the quality of the training being undertaken by ReAct participants should be considered. This could potentially include a regular review of a sample of the most popular courses being undertaken, for example.

Recommendation 3

In response to the finding that individuals with low levels of qualification are underrepresented within the programme, options for targeting or increasing the level of engagement of those with lower levels of qualification should be explored. The range and characteristics of participants engaged should also be monitored on an ongoing basis. 

Recommendation 4

The Welsh Government should ensure that the effective structure that the evaluation found to be in place to respond to the case study redundancies is replicable across Wales, especially in more rural locations, wherein fewer resources will be available and those resources will be more dispersed.

Recommendation 5

Redundancy can have an impact on an individual’s mental health. The provision of more comprehensive support with which to address the potential impact of redundancy on mental health should be considered as part of any future package of support.

Recommendation 6

To help each individual to secure the right job (rather than any job) and maximise their potential, the Welsh Government should consider the potential to provide further or ongoing support to ReAct participants even if/when they have secured employment.    

Recommendation 7

Those with lower skill levels benefit the most from the support that ReAct provides. Options for greater targeting of support at individuals with lower skill levels should be explored, including, for example, a higher grant rate or longer-term training options. This recommendation links to Recommendation 3.    

Recommendation 8

The appropriateness of the employer elements of ReAct should be reviewed in light of the findings of this and previous evaluations, with its withdrawal being seriously considered. If it is to be maintained, the employer elements should be more targeted potentially at microbusinesses and/or disadvantaged workers.

Recommendation 9

The management information available needs to be improved so as to allow more robust analysis of the counterfactuals, impact and cost/benefit of the ReAct programme when going forward. The process of collecting management data should be reviewed in order to ensure that it is fit for purpose. This should include a review of the potential to collect Lifelong Learning Wales Record (LLWR) data and more comprehensive outcome data.  

Authors: Endaf Griffiths, Dr Tom Marshall, Paula Gallagher, Sam Grunhut (all Wavehill), Duncan Melville, Paul Bivand, and Connor Stevens (all Learning and Work Institute)

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:
Kim Wigley
Email: kasemployabilityandskillsresearch@gov.wales

Social research number: 47/2021
Digital ISBN: 978-1-80195-600-0

Image

Image