Formative evaluation of Jobs Growth Wales+: interim report (summary)

This summary reports on the findings of an interim evaluation of Jobs Growth Wales+ (JGW+). The programme launched in April 2022 (initially to run to 2026) aiming to deliver individualised training, development, and employability support to 16 to 18-year-olds ^{[footnote 1]} who are assessed as NEET (Not in Education, Employment or Training) on joining the programme.

Designed to take forward the best elements of two previous employability programmes – Traineeships and Jobs Growth Wales 2 (JGW 2) – JGW+ is expected to have an overall budget of between £100 million – £200 million, annually equivalent to c.£25 million delivering through procured Contractors in the four regions (Lots) of Wales. 

At launch, JGW+ originally operated with a single referral system via Working Wales (delivered by Careers Wales), who operate to provide a single all-Wales entry point to employability support. Since the summer of 2022, eligible young people have also been able to directly refer themselves to JGW+ for assessment for programme entry or be referred directly by Contractors or local authority Engagement and Progression Coordinators (EPCs). This was done to improve access for young people who are the ‘hardest to reach’ and not engaging through Working Wales.

A standardised needs assessment process is used to identify the most suitable employability provision to support the individual in progressing. For referrals coming through Working Wales, the assessment is  conducted by trained careers advisors. For referrals coming through other routes, this assessment is carried out by specialist support staff. 

Based on the nature and scale of the participants’ needs/ barriers, and their distance from education or employment, the individuals are entered onto one of three support strands (engagement, advancement, or employment). Those on the engagement strand have been assessed as furthest from mainstream education or employment, and those on the employment strand have been identified as the most ready for work.

Programme adjustments

The close working relationship between Contractors and Welsh Government JGW+ staff has helped inform some adjustments to programme design since delivery started. These respond to emerging challenges experienced by young people including the COVID-19 pandemic, and ongoing cost-of-living pressures; these challenges increased the complexity of participants’ needs.

Adjustments have included: extension of Enrichment Activities and Pre-Engagement Activity ‘Get Ready’ to support engagement; increased and equalised training allowances; free meal provision; increased learner unit payments to Contractors supporting additional induction costs; expansion of qualification eligibility at Level 1 and 2; and extension of the eligible age for enrolment to 19 years.

In July 2022, Wavehill were commissioned to undertake a formative evaluation of JGW+, to:

• Track and record programme engagement, enrolment, and participation.
• Undertake a process and annual outcomes assessment of the programme.
• Establish an impact evaluation framework for the programme. 

The evaluation involves three phases of work:

• Initial scoping (August to December 2022)
• Interim (this phase) a review of JGW+ delivery in years 1 (2022/23) and 2 (2023/24); and      
• Final phase (beginning late spring 2024) impact and learning assessment of JGW+ across delivery.

Methodological approach

The evaluation has involved the following elements:

• Analysis of programme management information – including anonymised participant management information (April 2022 to October 2023), and programme-wide (published) performance data (April 2022 to September 2023).
• Stakeholder interviews (virtual) with Welsh Government staff (n=14); Contractors and Sub-Contractors staff involved in the management and delivery of the programme (n=27); and local authority representatives – typically the Engagement and Progression Coordinator role (n=23) from 17 (of the 22) local authority areas in Wales. 
• Participant Fieldwork (through telephone interviews) with three groups:
  • Participants receiving ongoing support through JGW+ (n=93). 
  • Participants who had completed JGW+ and secured a positive destination (n=69) 
  • Participants that left JGW+ in fewer than 12 weeks, prior to securing a positive destination (n=56)
• Employer Fieldwork (n=24) interviews with employers who had provided a work placement and/or subsidised employment for JGW+ participants.
• Research Advisory Group – this innovative approach led by Promo Cymru engages young people in helping to shape the evaluation and its dissemination. Six young people with protected characteristics were recruited to ensure that young people’s voices shape the evaluation delivery through commenting on: Welsh Government marketing of the JGW+ programme; participant interview schedules to advise on survey wording; young people’s experience of the labour market and job availability; and emerging participant interview findings to support analysis and interpretation.

Methodological limitations

Whilst target numbers of participant evaluation engagement were surpassed, ensuring a representative profile of JGW+ participants in terms of needs/barriers has proved challenging. This has occurred for a range of reasons including challenges in securing all participant data from Contractors for random sampling; inconsistency in survey participant contact data and changes in participant circumstances due to the time lags in data supply; and early leaver experiences being more positive than expected meaning they were ultimately a poor representation of individuals who may have had an adverse experience of the programme.

JGW+ progress and performance

Analysis of JGW+ published programme performance data and programme-wide management information has highlighted trends in relation to the monthly volume of participants enrolled by JGW+. Annually, July and October appear to present two spikes in enrolment numbers with the spike in July 2023 particularly pronounced. Whilst the caseload of JGW+ participants remained relatively static during the first 15 months of the programme; numbers have increased since July 2023.

It is estimated that over a 12-month period JGW+ is engaging with around one third of all young people NEET (aged 16 to 19) ^{[footnote 2]} in Wales. There is however wide geographical variation in rates of engagement with a 16 to 19-year-old living in Torfaen just under eight times more likely to be enrolled on JGW+ than those living in Powys. 

Participation in JGW+ is dominated by those enrolled on the engagement strand with the latest data (2022/23) showing almost two thirds (65%) enrolled onto that strand with 34% on the advancement strand and 1% on the employment strand. This reflects the fact the participants have been found to face much greater barriers to progression than had been originally anticipated, and thus have a greater distance from employment.

The programme was designed pre-COVID-19 and as such, could not have foreseen the impact on young people arising from the pandemic or the subsequent cost of living crisis and the additional challenges these events placed upon young people. The programme was adapted (as described above) to meet these additional challenges.

Outcomes are informed by the learner’s destination within four weeks of leaving the programme. For all learners, a positive outcome is either progression to higher level learning, progression into employment (full-time, part-time (16 or more hours per week) or self-employment) or progression to an Apprenticeship. For disabled learners, employment of less than 16 hours per week is also classed as a positive outcome.

Within the April 2022-March 2023 financial year, the programme fell just short of target securing a positive outcome rate of 58%. Analysis of the latest quarter of published data (July to September 2023) however shows the highest rate of positive outcomes (68%) since the programme launched exceeding the original target. 

Those enrolled at a younger age, Welsh speakers, those from Black ethnic groups, and those with higher pre-existing qualifications (Level 2 or Level 3) are more likely to secure positive outcomes than the average for the eligible cohort. There are no differences in the propensity to secure a positive outcome by gender or whether someone has any caring responsibilities.

Performance recommendations

Recommendation 1

It will be beneficial to interrogate monitoring data on the Get Ready component (if MI captures this evidence) in more detail with targeted fieldwork in the final phase of the evaluation to better understand the role it has played in driving engagement.

Recommendation 2

That evaluators gain access to, and review programme expenditure over time to assess JGW+ cost efficiency and cost effectiveness against other similarly targeted programmes. 

Recommendation 3

That further detail is captured in the next evaluation phase on participant situations prior to engaging on JGW+ and the role of the training allowance in their enrolment. 

Recommendation 4

To explore further the influences on geographical variation on the programme through further analysis of participant profiles and referral routes by geography and Contractor. 

Recommendation 5

For clearer definitions to be added to published performance data so that the public can better understand what the terminology associated with ‘programmes’ relates to.

Outcomes/Impacts of support

Learners reported wide-ranging benefits and positive outcomes from their JGW+ participation including on their job prospects. A total of 95% thought JGW+ had increased their chances of finding work and 93% felt an increased chance of gaining permanent employment. 

As a result of JGW+ participation 82% of participants reported life satisfaction improvements. 

Stakeholders concurred about the impact of support in bringing participants closer to the labour market, however their complexity of need required more sustained support to gain employment. 

More than one quarter of participants (31%) interviewed were in employment at the time of the survey however less than half of those in employment were on permanent/open-ended contracts or in full time work. 

Participants often left the programme early for positive reasons (education, employment or training). However, this was not captured in the Contractor MI; stakeholders felt that this should be rectified and participants’ reasons for leaving the programme early should be recorded in Contractor MI going forward. 

Employers reported the JGW+ subsidy enabled their programme participation and for some, had enabled them to take on a staff member. Most employers felt JGW+ met their needs. 

Multiple stakeholders raised concerns on how well the current monitoring system captures the complex, individualised outcomes/impacts achieved by learners. They highlighted the need to shift from measuring ‘hard’ outcomes towards capturing distance travelled, particularly in terms of soft skills and mental health.

Impact recommendations

Recommendation 6

The profile of all destination outcomes (positive, neutral and negative) should be taken into account in programmes’ performance assessment, and by Contractor and geography. 

Recommendation 7

That close monitoring of engagement by strand continues particularly to understand volume of participation on the employment strand to understand patterns of strand engagement and any reasons for those.

Recommendation 8

That the participant survey is repeated with greater targeting of those completing the programme to better understand all participant destinations (beyond four-weeks). 

Recommendation 9

Further analysis of detailed Management Information by protected characteristic and location with concurrent targeted fieldwork as part of the next phase of the evaluation to better understand geographical drivers of performance.

Delivery model reflections

Drawing insight from across interviews, delivery staff, wider stakeholders and participants were widely positive about the service delivery model and particularly the adjustments that were made to its design. Some raised concerns regarding the rigidity of JGW+, particularly the Assessment and Referral Report (ARR)-led method of placing learners on JGW+ strands. 

There were some mistaken assumptions amongst a few Contractor and subcontractor staff about the ability of staff to ‘override’ the system based on their professional judgement. The misconception was that Contractors could only change strand allocations by forcing a young person to ‘drop out’ and restart the assessment process, or that temporary participant absence could not be accommodated.

Stakeholders, employers and programme participants highlighted that there was a general view that marketing/promotion of JGW+ had largely been successful and awareness of the programme had increased over time. There were suggestions that promotions in schools could be strengthened, greater diversity of social media channels utilised, and a continued focus on the consistency of messaging (particularly in light of policy changes affecting the delivery of JGW+).  

It was widely felt that the introduction of direct referral to the programme led to a marked increase in the rate of enrolments with pre-engagement activity being particularly effective in increased enrolment.

The importance of pre-engagement activities was felt to relate primarily to the mental health (particularly anxiety) and an associated lack of coping mechanisms amongst prospective JGW+ participants. This pre-engagement provision was widely welcomed by stakeholders.

Participants found JGW+ easily accessible and staff understanding and supportive. 

Amongst active participants, 82% of learner respondents felt that the training allowance helped address financial pressures and played an important role in increasing young people’s interest and engagement in the programme. Stakeholders believed that placing value on young people’s time and efforts in this way helped to boost their self-esteem. 

There appeared to be little evidence of Contractors working with marginalised groups in the design of inclusive models of support. However, almost nine in ten learners felt JGW+ was satisfactory or good in terms of inclusion and accessibility.

Stakeholders reported little Welsh language provision demand from learners, whilst two Contractors struggled to recruit Welsh-speaking tutors and extra resourcing costs were considered a barrier to Welsh language provision (additional resource was provided by Welsh Government to cover costs). 

Despite some concerns around understanding of intersectionality and inclusivity and its application in Contractor design and delivery, JGW+ was viewed as more inclusive and accessible to young people than previous similar employability programmes. 

The duration of support for participants varied greatly, particularly amongst engagement strand participants however the flexibility and needs-led approach to learner support was widely welcomed. 

For those enrolled on the advancement strand, stakeholders queried whether a more sectoral targeting of skills support could better respond to future industry needs.

Amongst participants, nine in ten felt that JGW+ support had met or exceeded their expectations whilst three quarters felt there was no way support could be improved. 

Stakeholders viewed JGW+ as filling local service provision gaps but that greater integration of other service support may help ensure all support options are available to young people.

Delivery recommendations

Recommendation 10

That concerns around rigidity are explored and that Welsh Government provide messaging to support the consistent application of professional judgement by Contractors/Subcontractors across the programme. 

Recommendation 11

That understanding of issues around drop-off, restart, and use of Contractor discretion are reviewed with the aim of ensuring greater consistency in approaches enabling participants to re-engage in the service more easily, especially in relation to temporary absence. 

Recommendation 12

That consistency of marketing messaging is regularly reviewed to ensure full alignment with JGW+ policy changes, particularly to ensure that Contractors, EPCs, and young people directly referred to the programme fully understand its current delivery features. 

Recommendation 13

That the evaluation explores the nature and extent of collaborative or referral activity from JGW+ to better understand the extent of service integration with other support provision, particularly as the number of Shared Prosperity Fund (SPF) resourced activities increases. 

Recommendation 14

That indicators of wellbeing are captured more consistently as part of the enrolment process for JGW+ participants to help in capturing the social value gained through JGW+ participation.

Recommendation 15

That a consistent approach is adopted across programme for the electronic capture of participant barriers on enrolment and in the tracking and measuring of soft outcomes to help in the capture of the impact of JGW+ support.

Recommendation 16

That identifiable management information is obtained through Contractors to inform the sample frame for fieldwork.

[1] The age group for eligibility to JGW+ was extended to 19 in January 2023.

[2] The age group for eligibility to JGW+ was extended to 19 in January 2023.

Report authors: Oliver Allies, Jakob Abekhon, Paula Gallagher, Endaf Griffiths, Shanti Rao, Simon Tanner

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:
James Lundie
Senior Research Officer
Knowledge and Analytical Services
Welsh Government
Cathays Park
Cardiff
CF10 3NQ
Email: KASEmployabilityandSkillsResearch@gov.wales

Social research number: 49/2024
Digital ISBN 978-1-83625-266-5

Image