Evaluability assessment for the Wales Women’s Justice and Youth Justice Blueprints: summary

This paper summarises the findings from the Evaluability Assessment (EA) of the Wales Women’s Justice ^{[footnote 1]} and Youth Justice Blueprints ^{[footnote 2]}. The Blueprints were developed to improve partnerships and services delivered to women and children and young people in the justice system in Wales.

The EA was conducted by Opinion Research Services (ORS), with consultancy support from Dr Helen Hodges, and Dr Anthony Charles. It aimed to establish how the Blueprints can be effectively evaluated.

The four stages of the research were: a statistical and policy mapping exercise ^{[footnote 3]}; women and children and young peoples’ evaluation design groups; agency and stakeholder engagement and development of evaluation models.

The Youth Justice Blueprint

Prevention; pre-court diversion; community; custody; and resettlement and transitions are the main priorities of the Youth Justice Blueprint. Research and evaluation is a cross-cutting priority.

Prevention

Participants had different interpretations of prevention. They also highlighted the need for commissioners and services to work together; consistent staffing and funding; and to target prevention services correctly. To evaluate prevention, it needs to be measured appropriately, and have an outcomes framework, participants suggested.

Recommendations for evaluating prevention are using:

• Wales Youth Justice Indicator (WYJI) data to show engagement with education, employment, and training, and developing equivalent measures for access to other services
• Child and Adolescent Mental Health Services (CAMHS) referral to treatment time data to show service supply vs. demand;
• proxy measures such as attendance to reflect school engagement
• validated Welsh Government measures to determine improvements to health and wellbeing
• data on costs for some crime types, prosecution, and prison places to assess the cost savings of prevention
• staff training completion rates
• developing an alternative metric to FTE to capture first contact with the youth justice system (drawing on the approach used in public health Analysis Methods Notices)

Data linkage could also enhance understanding of risk and intervention triggers.

Pre-court diversion

Participants highlighted the need to define diversion clearly, and to ensure that suitable diversion services are available to all children and young people who need them. Some cultural changes were also said to be needed, such as implementing a non-punitive, child-first approach, and not criminalising children and young people who are looked after.

Participants highlighted the need to explore whether white children and young people are any more likely to be diverted than ethnic minority children and young people. The short length of pre-court diversion initiatives and differences in reoffending figures recorded by YOTs and the Ministry of Justice (MoJ) could make evaluation challenging, they felt.

Although little published data is available around pre-court diversion, recommendations for evaluating pre-court diversion are using:

• local records to determine reoffending among those who have been diverted pre-court
• linking police and YOT data with administrative datasets to track outcomes over time
• creating an alternative metric to FTE, which overcomes any inconsistencies in recording outcomes.

Community

Participants felt that agencies could further embed trauma-informed practice through appointing 'trauma champions' and delivering further training. Community services should be available early, appropriately funded, and have clear governance and accountability frameworks, they noted. Participants also suggested that evaluations of the community priority should explore the influence of trauma-informed practice on services.

Recommendations for evaluating the community priority are:

• consulting children and young people at the start and the end of their engagement with community-based support (and incorporating their grievances and complaints)
• conducting qualitative research with those who have and have not taken up community-based support
• using public health referral to treatment time measures with metrics from partner agencies to understand unmet need for services
• exploring baseline and over-time reoffending rates linked to equality and diversity measures
• collecting staff training figures and linking it to service performance management data
• linking YOT / MoJ and SAIL data to understand service user’s needs and circumstances over time

Custody

Participants highlighted the need to improve secure provision for girls, and education and training provision for all children and young people in custody. Priority should also be given to making custodial settings more family-friendly, and to implementing the proposed small unit, trauma-informed, hub-and-spoke model of custody for children and young people in Wales, they noted. Participants also felt that it would be relatively easy to engage children and young people in custody with research.

Recommendations for evaluating the custody priority are exploring:

• reoffending rate by sentence and disposal type (community vs. custodial)
• the profile of disposals for different groups, by offence type and severity, changes over time, and whether there has been an increase in the use of out of court disposals and a fall in the use of custody
• secure setting’s compliance with the Welsh Language Standard and the Social Services and Well-being Act (Wales) 2014
• the use of remand, early release, and continuity of care measures

Resettlement and transitions

Placing Welsh children and young people in custody in England was a challenge for achieving this priority, according to participants. Having suitable accommodation available upon release was essential, they felt. They also identified the need for more appropriate resettlement and transition for children and young people, which has their and their families' or role-models’ input and involves custody staff and all relevant agencies. When evaluating this priority, differences in delivering and monitoring resettlement support should be considered, as well as the general lack of data on resettlement and transitions.

Recommendations for evaluating resettlement and transitions include:

• exploring reoffending rates by sentence type, and compliance with community sentences (as with the custody priority) in terms of any differences between children and young people with different needs and/or protected characteristics
• incorporating time to justice (time from arrest to court)
• creating an intersectional profile of those leaving custody
• incorporating the use of remand
• exploring the suitability of accommodation, education, training and employment, and access to community-based treatment through continuity of care measures

Research and evaluation

Participants highlighted the following considerations for evaluating the Youth Justice Blueprint

• Incorporating input from all relevant agencies.
• Proper funding for research and evaluation.
• Building research and evaluation into services’ core business from the outset.
• Using research and evaluation to influence practice to a greater extent.
• Ensuring that research and evaluation is meaningful and not tokenistic.
• Research and evaluation should combine qualitative and quantitative data, and be independent, and longer-term.
• Data for research and evaluation should be easier for researchers to access.
• Engaging children and young people as peer researchers.
• Using age-appropriate techniques.
• Having a trusted person to introduce researchers and encourage engagement.
• Ensuring that children and young people feel validated through participating.
• Conducting research over several sessions in a location where children and young people feel safe.
• Avoiding paper-based, self-completion methods like worksheets.
• Offering suitable incentives.

In relation to research and evaluation, most children and young people said that they would welcome the opportunity to get involved in research and evaluation in future. Researcher’s gender, age, and ethnicity were unimportant to them, although they highlighted the importance of considering children and young people’s emotional state before engaging them in research. Most would prefer to participate in research in the afternoon or evening; on a one-to-one or group basis; and would rather not complete diaries for research purposes.

The Women’s Justice Blueprint

The Women’s Justice Blueprint prioritises prevention and early intervention; community-based solutions; custody and resettlement; and research and evaluation.

Prevention and early intervention

Participants highlighted the need to define and measure prevention and early intervention appropriately. They emphasised the importance of strong links between service commissioners, deliverers, and overseeing organisations. Having appropriate services for all women and encouraging them to engage with them was also seen as important, as was accessing sustainable funding for services, and overcoming competition for resources.

Recommendations for evaluating prevention and early intervention included:

• using local service data to monitor performance, e.g., referral and/or assessment, and police data, e.g., arrests and stop and search
• using data on referral to treat times for services to help determine supply vs. demand; and data on costs for some crime types, prosecutions, and prison places to help show cost savings
• considering an alternative to FTE to assess prevention and diversion drawing on the approach used in public health Analysis Methods Notices
• using data linkage to enhance evaluation (e.g., tracking women who have accessed prevention and early intervention services through administrative data)
• exploring differences in outcomes for women who were arrested and not charged, and who had different out-of-court disposals
• tracking girls who had been through Bureau/Triage to determine their outcomes

Courts and sentencing

Participants noted the need to ensure consistent understanding of alternatives to charging women, and the need to reduce the use of custody for women. Aspects for research and evaluation included the apparently lower thresholds for custodial sentences for women than for men; understanding how sentencing varies across Wales; and what works in reducing reoffending among women.

Recommendations for evaluating courts and sentencing were the need to:

• increase the evidence base on this topic
• harness the existing ways in which women can feed back on their experiences (e.g., focus groups and workshops, lived experience forums, “touchpoint” checklists, and exit interviews)
• establish whether the custodial threshold is lower for women than for men (using Data First from magistrates and crown court)
• establish any differences in the sentencing of women with protected characteristics vs those without using Data First records (data linkage would enhance this), to explore whether trends change over time, and to explore compliance issues
• explore the profile of disposals for different groups, by offence types and severity, whether this changes over time, and whether there has been an increase in the use of out-of-court disposals and a fall in the use of custody
• incorporate time to justice data
• explore decision-making and attitudes among sentencers and wider court staff through qualitative interviews
• explore the effectiveness of training for practitioners, partner agency staff, and the judiciary through conducting pre-and post-surveys and qualitative interviews

Community-based solutions

Expanding suitable accommodation, services, and information for women (especially for those who have been abused) in the community was cited as a priority for improving community-based solutions, as was establishing the proposed women’s residential centre. Women and communities should help shape community services, it was said. There was some feeling that the community and the courts and sentencing priorities within the Women’s Justice Blueprint should be linked. Defining and mapping community-based solutions, and including women with lived experience, were said to be important when evaluating community-based solutions.

Recommendations for evaluating the community-based solutions priority included:

• focusing on women’s timely access to services and support
• using public health waiting time figures to understand unmet need for services and support
• using baseline and over-time reoffending rates and linking those to equality and diversity measures
• exploring compliance with community sentences using breach and recall figures and data from electronic monitoring and drug testing, identifying reasons for non-compliance
• gathering staff training figures
• linking data from probation and from the Better Outcomes through Linked Data (BOLD) programme to find out more about service user’s needs and circumstances

Custody and resettlement

Participants felt that custody and resettlement for women in Wales was complicated by housing them in English prisons. Custody and resettlement provision could be improved through more collaboration between custodial settings and services in Wales, and ensuring consistent availability of education, training, and employment opportunities for women returning to their communities. Providing women with safe, secure accommodation was also important, as was reducing the use of short sentences.

Evaluations of custody and resettlement should explore 'revolving door' cases and different settings’ approaches to working with, monitoring, and collecting information about women.

Recommendations for evaluating custody and resettlement were incorporating:

• reoffending rates by sentence type in terms of any differences between women with and without protected characteristics
• the profile of disposals for different groups, by offence type and severity, whether this has changed over time, and whether there has been an increase in the use of out-of-court disposals and a decrease in the use of custody
• time to justice data
• data on access to support services through developing a consistent intersectional profile across all Welsh women in custody using Prison-Nomis
• the numbers of women who are applying for mother and baby units and maternity indicators
• secure establishment’s compliance with the Welsh Language Standard and the Social Services and Well-Being Act (Wales) 2014
• figures on compliance with community sentences, use of remand, release on a temporary licence (ROTL), and continuity of care measures
• exploring the suitability of accommodation on release

Research and evaluation

Participants highlighted the following should be considerations for evaluating the Women’s Justice Blueprint

• Buy-in and input from all relevant organisations.
• Being supported by a culture of open learning and a shared research strategy (across both Blueprints).
• Proper funding for research and evaluation.
• Building research and evaluation into services’ core business from the outset.
• Research and evaluation should combine qualitative and quantitative data, and be independent, and longer-term.
• Research and evaluation should seek to establish the effectiveness of multi-agency working in achieving the Blueprint’s aims.
• Data for research and evaluation should be easier for researchers to access.
• Different pieces of research and evaluation across the Blueprint should be more closely aligned.
• The difficulty of isolating the impacts of the Blueprint from those of existing initiatives should be recognised.
• Engaging women ex-offenders as peer researchers and mentors (with appropriate training and support).
• Having a trusted person to introduce researchers and to support engagement.
• Ensuring that women feel validated through taking part and know that their contribution will help to effect change.
• Using creative, visual methods.
• Selecting the 'right' women.
• Offering appropriate incentives.

Few of the women who took part in interviews as part of the project said that they had been involved with research and evaluation previously, although most recognised the importance of doing so said that they would be happy to in future. Most felt that the best way of involving women in research was simply by speaking to them, either one-to-one, or in a small group. Women expressed different opinions regarding conducting surveys and keeping written or video diaries as part of research and evaluation. Most felt that there were no good or bad times to get involved with research and evaluation, and some highlighted the importance of taking part even when they felt low. Previous experience of the criminal justice system was not seen to be essential for those conducting research. Researcher’s racial and ethnic background was not seen to be important by women, although their gender may be.

Recommended evaluation model

In line with stakeholders’ feedback, we recommend evaluating the Youth Justice and Women’s Justice Blueprints separately to capture their nuances, with an overarching summary drawing together their common themes from the evaluations to inform future policy and practice.

Ideally, the evaluation should include process and outcome strands, incorporating some existing data, such as the Integrated Offender Management (IOM) dashboard; internal analysis being undertaken by the Youth Justice Board (YJB); and publicly available datasets from MoJ, the YJB, and the Home Office.

Where gaps in this data cannot be addressed by routine data collection by partner agencies, additional data should be collected to support a more thorough evaluation. As noted, outcome measures which could be used to evaluate the Youth Justice and Women’s Justice Blueprints are listed in the full report.

If resources and time are limited, a one-off, 'snapshot' evaluation could be conducted, providing all the underpinning principles mentioned herein are addressed, but would be less insightful than a longitudinal evaluation.

The evaluation should combine quantitative data analysis with qualitative insight from interviews and/or focus groups with women/children and young people, their families, their victims, stakeholders, and practitioners. The evaluation should be longitudinal, measuring outcomes at different points in time, ideally with the same women/children and young people. This will offer the ability to benchmark and give a more long-term overview of success or otherwise. Incentives would be required to sustain participants’ engagement.

In line with a women’s and children’s rights-focused and trauma-informed approach, women/children and young people who are involved with the criminal/youth justice system at various stages should co-produce the research instruments and interview/focus group questions. This could be achieved through holding design groups.

The use of peer research and alternative methods such as videos/video diaries should also be explored, subject to appropriate training, guidance, and support.

Specific recommendations for evaluating the Youth Justice Blueprint

More individual-level data exists in relation to children and young people in the youth justice system relative to women in the criminal justice system. The YJB or MoJ could dedicate resources to undertake a systematic baselining exercise and establish the mechanism needed to enable trends in data to be monitored over time. If MoJ makes criminal justice data available within the SAIL databank, it would be possible to link it with education, social services, and health data to learn more about the needs and circumstances of those in the youth / criminal justice system. External data could also be taken into SAIL, which could enable longitudinal tracking of children and young people who have engaged in prevention and diversionary activity, which are two major gaps in the evidence base. Analytical capacity is needed to achieve this, potentially through MoJ-funded fellowships.

Specific recommendations for evaluating the Women’s Justice Blueprint

As all Welsh women who are in custody currently serve their sentences in England, they are harder to identify. This, combined with changes in probation services and related discontinuities in the data in England, make it harder to monitor trends over time.

Individual-level data on women is held by various services such as the police, courts, and local justice boards which can be aggregated to create a national picture. Disaggregating it by gender is not always possible. Low numbers of Welsh women in custody create an additional challenge. Data on protected characteristics is not collected consistently.

As with the youth justice system, notable gaps in the evidence base exist around diversion and community-based interventions. There is scope to address this through collating data from screening / referral forms, although data must be collected consistently, and questions must be harmonised between agencies. The approach used within Public Health to create Analysis Methods Notices is therefore recommended.

Expanding the women’s offending evidence base could be achieved ether by MoJ creating dedicated resource for baselining and monitoring performance, potentially creating a Welsh version of their publication Women in the Criminal Justice System. To achieve this, strategic leadership is required, and issues around data security, governance, and oversight must be resolved. MoJ data on courts, plus prison and probation data, could be anonymised and brought into SAIL and linked with other datasets to enable tracking over time, as with the youth justice data. Anonymisation would require input from Digital Health and Care Wales. Again, as with the youth justice data, funding is required for data analysts.

Underpinning principles for evaluating the Blueprints

The following principles should underpin any evaluation of the Youth Justice and Women’s Justice Blueprints.

Baselining and tracking service user’s progress through the Blueprints’ priorities is essential. To achieve this, we recommend developing a sufficiently resourced multi-agency performance and monitoring framework prior to any evaluation.

Evaluation should take an intersectional approach to understand any differences in outcomes for those from priority groups and those with protected characteristics. Agencies should ask harmonised questions in a sensitive way to identify group membership and protected characteristics.

Research and evaluation should be built into services, approaches, and interventions from commissioning stage. The suitability of existing outcome measures should be explored before creating new ones.

In advance of evaluation, agencies should define what 'success' looks like for all Blueprint-related initiatives, and data sharing agreements and/or memorandums of understanding should be in place.

Youth / criminal justice agencies should support evaluation through providing relevant data and harnessing their relationships with service users to engage them in evaluation.

A 'passport' containing key information to follow children, young people, and women through the youth / criminal justice system should be developed to standardise data collection and reduce re-traumatisation.

The Blueprints should have a coherent evaluation strategy to ensure that their delivery is evidence informed.

Research and evaluation should build on the findings from existing monitoring and evaluation. It should be led by appropriately skilled and qualified independent researchers.

Evaluation should be pragmatic and realistic about how far the effects of the Blueprints can be isolated from other activities which might be effecting change; which organisations hold the levers for change; and the extent to which change can occur in a devolved context.

[1] Supporting female offenders
[2] Supporting young offenders
[3] The findings of the statistical mapping exercise are outlined in detail in the separate Evaluability Assessment of the Women’s Justice and Youth Justice Blueprint: Statistical Mapping Report. 

Report Authors: Liz Puntan, Kelly Lock, and Helen Hodges

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:
Merisha Weeks
Email: PSLGResearchTeam@gov.wales

Social research number: 67/2022
Digital ISBN 978-1-80391-932-4

Image