Skip to main content

Research aims

The Welsh Government wants to make Wales the most LGBTQ+ friendly nation in Europe. In 2023, the Welsh Government published its LGBTQ+ Action Plan, which outlines how it intends this goal to be reached. The Action Plan falls alongside other Welsh Government action plans and strategies, such as the Anti-Racist Wales Action Plan, and they all feed into the National Equality Objectives. It is important to consider how these interact. The Welsh Government has commissioned Alma Economics to carry out an evaluability assessment to provide recommendations on how the LGBTQ+ Action Plan should be evaluated. Evaluation will help determine whether the LGBTQ+ Action Plan is successful and is achieving its intended aims of improving the lives and experiences of LGBTQ+ people in Wales.

Methodology

An assessment of the feasibility and suitability of different evaluation methodologies was carried out to determine the best approach for a future evaluation of the Action Plan. This assessment was guided by Magenta Book (HM Treasury and Evaluation Task Force) which provides guidance on evaluation, best practice and stakeholder engagement.  A review of documents relating to the LGBTQ+ Action Plan and interviews with Welsh Government officials were used to scope the project before a non-systematic literature review was performed. Stakeholders were engaged through interviews, and a Theory of Change was drafted from these conversations. The outputs from the literature review, stakeholder engagement, Theory of Change and data assessment were used to then inform the evaluability assessment.

This began with a review of key documentation relating to the LGBTQ+ Action Plan, including progress updates, recommendations of the Independent LGBTQ+ Expert Panel, and related strategies. Scoping interviews with Welsh Government officials were also conducted to deepen understanding of the LGBTQ+ Action Plan and how it relates to other Welsh Government plans and objectives. Following this, a literature review was conducted, which focused on existing evaluations of similar LGBTQ+ policies and programmes. Literature included peer-reviewed and non-peer-reviewed literature, with 52 papers identified as relevant. This was a targeted exercise with limited scope and should not be considered comprehensive. This review informed the evaluability assessment by providing an understanding of potential risks, challenges, and solutions related to particular methodologies and evaluation tools.

The Welsh Government and external stakeholders representing LGBTQ+ organisations were engaged throughout the evaluability assessment through semi-structured one-to-one and small group interviews. In total, 36 interviews were conducted with 50 interviewees. This engagement provided an understanding of what outcomes and impact could be expected from the Action Plan, how these should be measured using existing data, and how to define and measure successful implementation. Discussions also included views on potential approaches to a future evaluation. 

Findings from the interviews were also used to draft a Theory of Change which visually describes the mechanisms through which the activities and actions included in the LGBTQ+ Action Plan may generate outcomes and impact. The Theory of Change is used to support the evaluability assessment by identifying important impacts and causal mechanisms which would need to be assessed in an evaluation and should therefore be considered in the evaluability assessment. It also helps to identify key indicators to monitor the progress and impact of the LGBTQ+ Action Plan, which in turn determines what data should be collected in advance of a future evaluation for this purpose.

The outputs from the literature review, stakeholder engagement, Theory of Change and data assessment were used to inform the assessment of evaluation options to determine which approach was most feasible and suitable.

Main findings

Literature review

When evaluating complex programmes, the literature emphasises the importance of using a Theory of Change to clarify the underlying causal pathway between a policy intervention and its desired societal outcome within complex systems. The role of external influences in affecting the impact of the Action Plan should be recognised in the choice of evaluation method, as identifying causality with such outcomes is likely to be difficult. The evidence also suggests using a mixed methodologies approach to evaluation (i.e. those that use a combination of quantitative and qualitative techniques) given they will enable a flexible approach that can capture complexities and nuances of the programme, as well as any changes across time. Overall, few examples were found of evaluations of LGBTQ+ programmes and interventions, both domestically and internationally. 

Engagement with stakeholders

Interviews were conducted with 38 Welsh Government policy officials and 12 representatives of external organisations. Stakeholders consistently emphasised that quantitative measures should be used with caution when evaluating the Action Plan, because there is a risk such metrics would not capture the diversity and nuances of experiences within the LGBTQ+ community and intersectional groups. Stakeholders also emphasised the need to for an evaluation to understand the impact the Action Plan has on the perceptions and lived experiences of LGBTQ+ people and intersectional groups through qualitative methods, such as through interviews and focus groups. 

Theory of Change

Informed by interviews with stakeholders and literature review, a Theory of Change was developed. It is divided into a diagram for each of the LGBTQ+ Action Plan’s themes and explicitly highlights key assumptions and anticipated risks, which should be considered during an evaluation. The Theory of Change also includes a column to represent the Overall Vision of the LGBTQ+ Action Plan, which is informed wholly by interviews with internal and external stakeholders. It is also recognised that the Theory of Change may need to be updated nearer the evaluation as outputs or outcomes which are not yet predictable develop and require representation in the diagram. 

Assessment of data sources

There is relatively little availability of quantitative evidence that can be used to confidently measure the outcomes and impact of the LGBTQ+ Action Plan. No sources have been identified that could evidence the Action Plan’s objective ‘to make Wales the most LGBTQ+ friendly nation in Europe’. Some quantitative data does exist and is outlined in a Databank (Table 1) within the main report, and this could be used to measure important outcomes of the Action Plan. However, quantitative data should be used with caution given that, as some stakeholders pointed out, there is a risk that such metrics may oversimplify the complexities of the Action Plan and intersectional experiences. Furthermore, there does not exist any systematic data collection which captures the lived experiences and attitudes of LGBTQ+ people in Wales over time, which was consistently viewed by stakeholders as being a key feature of any future evaluation.

Assessment of evaluation options

An assessment of evaluation methodologies listed in the Magenta Book was conducted to arrive at a preferred evaluation option. This included assessing the relative merits and drawbacks of different methodologies based on the findings from the literature review, stakeholder engagement, Theory of Change development and assessment of quantitative data. 

This assessment recommended against experimental or quasi-experimental techniques being used in the evaluation due to ethical concerns and limited data availability, and therefore recommended that theory-based approaches be employed. Some theory-based approaches were ruled out as evaluation options, largely due to their high demands on stakeholder engagement and inability to capture the complexities of the LGBTQ+ Action Plan for Wales.

Furthermore, an economic evaluation is not recommended due to the absence of data to quantify and monetise key outcomes for a value-for-money appraisal and the risk of oversimplifying the complexities of the LGBTQ+ Action plan and the variety of LGBTQ+ people’s experiences.

Recommended evaluation options

Separate process and impact evaluations are recommended by this assessment. The process evaluation should be conducted as soon as possible. It should be conducted before the impact evaluation to ensure that any lessons learned from the process evaluation can be incorporated into policymaking and begin to take effect before evaluating impact. Recognising that the impact evaluation will largely be driven by peoples’ perceptions and experiences, which are expected to change slowly over time, it is recommended that the impact evaluation is conducted several years after the Action Plan has been implemented.

For the process evaluation, it is recommended that a mixed-methods approach is used to understand the extent to which the actions have been completed and how this can be improved. It should also consider any further improvements to data and monitoring processes, as well as the adequacy of structures and governance processes to engage with external stakeholders and LGBTQ+ people more generally. 

It is recommended that the impact evaluation consists of a mixed-methods approach with principles of realist evaluation. A realist evaluation is useful for understanding “what works, in which circumstances, and for whom”, making it useful for the impact and process evaluation, as well as capturing the outcomes on intersectional groups. In this case, applying a light-touch approach to the realist evaluation of the LGBTQ+ Action Plan will reduce reliance on stakeholders’ time and expertise, which can be significant barriers to conducting realist evaluations.

The impact evaluation should engage with Welsh Government officials to provide useful insight into how the Action Plan has led to changes in policymaking. It should also understand and represent the views of LGBTQ+ people and intersectional groups to provide insight into how the Action Plan may have a positive impact on their lives and lived experiences. To address existing gaps in available data collection on the lived experiences and perceptions of LGBTQ+ people and intersectional groups, it is recommended that a large-scale survey is conducted to support the evaluation. A survey can maximise an evaluation’s reach and be used to systematically record the attitudes of LGBTQ+ people, which can enhance the availability of qualitative and quantitative information to allow comparison across time and across intersectional groups. The success of the Action Plan is currently defined as “making Wales the most LGBTQ+ friendly nation in Europe”. Recognising that this objective is not measurable, it is recommended that this objective is revisited before it is used to measure success in an impact evaluation.

Recommendations to support a future evaluation

The following recommendations are made to improve the data landscape and therefore support a future evaluation of the LGBTQ+ Action Plan for Wales.

  1. The Welsh Government should oversample LGBTQ+ individuals within the National Survey for Wales to increase the quality and useability of data by sexual orientation. The Welsh Government should make data from the National Survey for Wales available to evaluators for more questions by sexual orientation. Evaluators should use this data cautiously, due to concerns over small sample sizes.
  2. The Welsh Government should add relevant questions to the National Survey for Wales, including a question to measure respondent’s awareness of anti-LGBTQ+ hate crimes and the options available to report hate crimes.
  3. Welsh public services, including the NHS, should consider more frequently collecting data on service user’s gender identities and sexual orientations in order to identify any inequalities in service delivery. Data collection should be standardised and collected consistently.
  4. The Welsh Government should contact support services and/or public services to explore the availability of data on conversion practices. If this data is unavailable, Welsh Government officials should see if data on conversion practices is available from Galop and Victim Support in Wales.
  5. Welsh Government officials should collate a list of the equality forums described in “Theme A” (Human Rights and Recognition)  of the LGBTQ+ Action Plan for Wales. This could be used to evaluate the implementation of the following action: Strengthen LGBTQ+ representation on equality forums.
  6. Welsh Government should seek data from the General Register Office on the number of people who apply for a Gender Recognition Certificate within Wales.
  7. The Welsh Government should have a greater focus on equalities issues in Overseas Network reports to inform the process evaluation. Welsh Government should include a chapter dedicated to equalities issues.
  8. The Welsh Government should request and, if necessary, collate monitoring data on hate crimes by protected characteristics provided by the Home Office on Welsh police forces to inform the evaluation. 
  9. The Welsh Government should request monitoring data from the Wales Hate Support Centre. The data request should include: monitoring data regarding the number of LGBTQ+ referrals, data regarding their engagement with LGBTQ+ people, and data measuring the impact of their services for LGBTQ+ individuals. 
  10. The Welsh Government should request data from Welsh police forces regarding LGBTQ+ representation among PCSOs and police officers.
  11. The Welsh Government should ask for regional breakdowns by sexual orientation and gender identity compared to sex recorded at birth from the Crime Survey for England and Wales. Validating the quality of this data will also be crucial, as low sample sizes are expected.
  12. Welsh Government should provide updates on the outcome of the review into VAWDASV (violence against women, domestic abuse and sexual violence) indicators and seek to ensure that indicators will be available according to sexual orientation and gender identity. 
  13. The Welsh Government should include a question of sexual orientation and gender identity in the Wales Sanctuary Service’s satisfaction survey. The data will need to be obtained and validated. This may be complex considering asylum is not a devolved policy area.
  14. The Welsh Government should seek to secure health complaints data on the number of complaints from people who identify as LGBTQ+. If this is not possible, they should seek the number of complaints which reference anti-LGBTQ+ discrimination in healthcare.
  15. It is unclear if the NHS Wales Staff Survey asks respondents to share their sexual orientation and gender identity. Welsh Government officials should investigate this, with the goal being that the survey results can be analysed according to sexual orientation, gender identity, and whether an individual’s gender aligns with the gender assigned at birth.
  16. Welsh Government officials should request monitoring data from the Welsh Gender Service – especially regarding patient-reported outcome measures (PROMs), patient-reported experience measures (PREMs), and anonymised feedback from meetings between the WGS and trans representatives – and make this data available to the evaluators.
  17. Welsh Government officials should liaise with the School Health Research Network to ensure that future Student Health and Wellbeing Surveys ask secondary school pupils questions about both sexual orientation and gender identity and make this data available to the evaluators of the LGBTQ+ Action Plan for Wales. Currently, the survey asks for the respondent’s gender and sex at birth and does not collect data on whether people identify as LGBTQ+ or any group within the community.
  18. The Welsh Government should make website analytics data available to evaluators for the “Parenting: Give it time” website pages. To improve this data’s usefulness for the evaluation, the Welsh Government should consider asking those visiting the webpage for feedback on their experience using these.
  19. The Welsh Government should strengthen the Local Government Candidates Survey by asking if people’s gender identity is the same as the gender that was described at birth. In addition, Welsh Government should seek to increase the response rate to this survey.
  20. The Welsh Government should collect data on how many National Governing Bodies of sport in Wales have trans-inclusive policies by reviewing the policies of a sample of sporting bodies. This exercise should be repeated before the evaluation begins. This will allow for a baseline to be collected against which evaluators can determine changes in trans-inclusivity in Welsh sport and the implementation of the following action: Improve the access and participation of transgender people in sport.
  21. The Welsh Government should seek monitoring data from Welsh Government-funded cultural organisations (Amgueddfa Cymru, the Arts Council of Wales, the National Library of Wales, the Royal Commission on the Ancient and Historic Monuments of Wales, and Sport Wales) to determine their role in promoting LGBTQ+ inclusion and any measurable impacts.

Contact details

Report authors: Alma Economics

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:
Victoria Saynor
Equality, Race and Disability Evidence Units
Equality, Poverty and Children’s Evidence and Support Division
Welsh Government
Cathays Park
Cardiff
CF10 3NQ

Email: EqualityEvidenceUnit@gov.wales

Social research number: 83/2024
Digital ISBN 978-1-83625-931-2

Image
GSR logo