Patterns in reading and numeracy attainment: from 2018/19 to 2022/23

Personalised assessments in Reading and Numeracy are online assessments designed to help support development of reading and numeracy skills. They are used in schools as one of a range of approaches to support progression as part of Curriculum for Wales. Annual personalised assessments are mandatory for pupils in Years 2 to 9 in maintained schools. The assessments comprise: Numeracy which is taken in two parts - Numeracy (Procedural) and Numeracy (Reasoning); Reading in Welsh and English (see note on timing of introduction of personalised assessments below, and further information on each assessment and the mandatory requirements in the main section of this report). 

These assessments provide schools with information on the reading and numeracy skills of individual pupils and an understanding of strengths and areas for improvement in these skills. Following completion of assessments, schools have access to feedback on skills, progress, and a range of reports to help plan teaching and learning. The Welsh Government is clear that the purpose of the assessments is to support progression in learning, and that the assessment outcomes are not to be used for accountability purposes at any level.

Anonymised data from personalised assessments can also provide some information on reading and numeracy skills at a national level, showing changes in attainment over time and differences between demographic groups. 

Therefore, the Welsh Government has compiled this report at the earliest opportunity to assist in understanding patterns of attainment in reading and numeracy over time and the potential impact of the coronavirus (COVID-19) pandemic. This is the first year for which national-level data is available as a time series for all assessment subjects. This report uses anonymised data from personalised assessments taken between 2018/19 (the introduction of the first assessment) and 2022/23. The publication of this report does not involve any change for schools; personalised assessments will continue to be taken in the same way and used alongside other forms of assessment designed by schools in accordance with the Curriculum for Wales framework.

This is the first of a series of releases planned on national-level personalised assessment data. The subject of this short report is attainment changes over time. A more comprehensive release on the 2018/19 to 2022/23 data, to be published in late spring 2024, will show demographic differences, for example between male and female pupils, and the gap between pupils eligible for free school meals and their peers. Thereafter, annual releases will be issued which will eventually show trends and provide important information on pupils' development of these skills over time, at a national level. 

The releases will form part of a wider range of national-level information on learner achievement, complementing, for example, the information drawn from our broader programme of national sample-based monitoring assessments, which will cover the breadth of the Curriculum for Wales. The sample-based assessments will seek to assess only a sample of schools per year, ensuring that burdens on the system are kept to a minimum while contributing to national understanding of how the Curriculum for Wales is supporting learners' development. This programme is described in further detail in the Curriculum for Wales Evaluation Plan. 

A note on timing and availability

Personalised assessments were phased in over a period of four academic years: 

• Numeracy (Procedural) from 2018/19
• Reading in Welsh and English from 2019/20
• Numeracy (Reasoning) from 2021/22

This report uses data from 2018/19 to 2022/23. However, COVID-19 disruption has resulted in a gap in data for 2019/20, the year when administration of the assessments was most impacted by the pandemic. Whilst some personalised assessments were undertaken during this period, 2019/20 is not included in this analysis due to insufficient data being available. 

This report compares the attainment of pupils in a given year group over several academic years. For example, it compares the attainment of Year 3 pupils in the Numeracy (Procedural) assessment in 2018/19, 2020/21, 2021/22, and 2022/23. This means it reports on the attainment of separate cohorts of pupils who are in a given year group at different points in time. It does not track the attainment of the same pupils over time as they progress through year groups, though future reports are planned which will investigate this. For the purposes of this report, to make comparisons at a national level, we express average higher or lower attainment in terms of months. We do this for ease of understanding and because of the nature of the assessments, which do not have a common grading scale of the type used for examinations, for example.  Please note that we round all data to the nearest month and in some cases this will mean that a displayed total may not equal the sum of the figures that make up that total.  

The impacts on learning during and after the pandemic have been the subject of several studies in the UK and internationally – see links in the evidence section below. The patterns seen in Wales are similar to the research findings in other countries.  

A note on interpreting the figures in this section can be found below. The figures show patterns in average attainment for Years 3, 6 and 9; data for other year groups is available in the accompanying spreadsheet.

The assessments are listed here in order of their introduction in schools in Wales. It should be noted this report shows patterns over a relatively short period since the introduction of personalised assessments, and that further years’ data will be required to make judgements on longer term trends. Lower take up of assessments during 2020/21, the year which saw continued disruption during and following the pandemic, may also prove to be a factor for further consideration once future years’ data becomes available on trends.

Numeracy (Procedural)

The Numeracy assessment is taken in two parts. The Numeracy (Procedural) assessment focuses on numerical facts and procedures – the numerical ‘tools’ that are needed to apply numeracy within a range of contexts. It is taken annually by all pupils in Years 2 to 9. This assessment was introduced in the 2018/19 academic year and is the only assessment for which data is available both before and after the disruption caused by COVID-19.

Figure 1: Numeracy (Procedural): difference in attainment in 2020/21, 2021/22 and 2022/23 relative to 2018/19, in months

Image

Description of Figure 1: A bar chart showing the average attainment in the Numeracy (Procedural) assessments of the cohort of pupils who were in Year 3, 6, or 9 in 2020/21, 2021/22 or 2022/23. Average attainment is presented as months higher or lower than the attainment of the same year group in 2018/19.  Note that 2019/20 is not included due to insufficient data in the year most impacted by the COVID-19 pandemic.

Main points

• In Numeracy (Procedural), pupils in 2020/21 demonstrated lower attainment on average than pupils did in 2018/19. This was equivalent to a difference of 3 months in terms of lower attainment in 2020/21 relative to pupils in 2018/19. This may be attributable to the impact of the COVID-19 pandemic, which had similar effects internationally.
• The difference in attainment in 2020/21 relative to 2018/19 was not uniform for all year groups; the difference was greater for younger pupils. In 2020/21 the attainment of pupils in Years 2 and 3 was lower by 8 months relative to Year 2 and 3 pupils in 2018/19; the attainment of pupils in Years 4 to 7 was lower by 4 months relative to pupils in 2018/19. Conversely, in 2020/21 the attainment of pupils in Years 8 and 9 was higher by 4 months relative to pupils in Years 8 and 9 in 2018/19. This suggests that younger pupils were more negatively impacted by the pandemic, whilst older pupils were better able to manage the change in learning situation.
• In 2021/22, average attainment remained similar to that seen in 2020/21, across all year groups. This is within the pattern of minor changes over time that can be expected without major external influences.
• In 2022/23 average attainment for pupils in Years 2 to 5 was very similar to that seen in 2021/22. However, for Years 6 to 9, average attainment was 1 month lower relative to 2021/22. The pattern was consistent throughout all year groups, shifting from marginally higher attainment for younger pupils to lower attainment for older ones, in a systematic manner. 

In summary, in 2022/23 the average attainment of pupils in Numeracy (Procedural) was lower than it was in 2018/19, equivalent to 4 months’ difference in attainment. Attainment for Years 2 to 3 was around 8 months lower, for Years 4 to 7 it was around 5 months lower, and for Year 8 it was down by half a month. Conversely, attainment for Year 9 pupils was 3 months higher relative to Year 9 pupils in 2018/19.

Welsh Reading 

The Welsh Reading personalised assessments focus on how well pupils understand a text in Welsh and whether they are able to make judgements about what they are reading. The Welsh Reading assessment is taken annually by pupils in Years 2 to 9 whose learning is through the medium of Welsh. The assessment was introduced in the 2019/20 academic year, however there was insufficient data available for that year due to the disruption caused by the coronavirus pandemic, and therefore 2019/20 has not been included in this analysis. 

Figure 2: Welsh Reading: difference in attainment in 2021/22 and 2022/23 relative to 2020/21, in months

Image

Description of Figure 2: A bar chart showing the average attainment in the Welsh Reading assessments of the cohort of pupils who were in Year 3, 6, or 9 in 2021/22 or 2022/23. Average attainment is presented as months higher or lower than the attainment of the same year group in 2020/21. There is insufficient data for 2019/20, the period most impacted by disruption due to the COVID-19 pandemic; the chart therefore shows data for 2 years only.

Main points

• In 2021/22, pupils’ attainment was lower on average in Welsh Reading relative to pupils in 2020/21, equivalent to 5 months’ difference on average across all year groups. 
• There was a further decrease in attainment between 2021/22 and 2022/23 equivalent to 7 months’ lower attainment on average.
• Both these patterns are consistent in direction across year groups, so we can be reasonably confident they are not solely attributable to the expected minor level of change over time.

In summary, in 2022/23 average attainment of pupils in Welsh Reading was lower than it was in 2020/21, equivalent to 11 months’ lower attainment.

English Reading 

The English Reading personalised assessments focus on how well pupils understand a text and whether they are able to make judgements about what they are reading in English. The English Reading assessment is taken annually by pupils in Years 2 to 9 whose learning is through the medium of English; for pupils whose learning is through the medium of Welsh, the English Reading personalised assessment is optional in Years 2 and 3 and mandatory in Years 4 to 9. The English Reading online assessment was introduced in the 2019/20 academic year, however there was insufficient data available for this year due to the disruption caused by COVID-19. 

Figure 3: English Reading: difference in attainment in 2021/22 and 2022/23 relative to 2020/21, in months

Image

Description of Figure 3: A bar chart showing the average attainment in the English Reading assessments of the cohort of pupils who were in year 3, 6, or 9 in 2021/22 or 2022/23.  Average attainment is presented as months higher or lower than the attainment of the same year group in 2020/21.  Note that while the Reading assessments were introduced in 2019/20, there is insufficient data for this first year which was the period most impacted by disruption due to COVID-19; the chart therefore shows data for 2 years only.

Main points

• In 2021/22, English Reading attainment was similar to 2020/21. 
• Then between 2021/22 and 2022/23 there was a change; pupils in 2022/23 demonstrated 3 months’ lower attainment on average across all year groups compared to pupils in 2021/22. 
• The change was pronounced for younger and older pupils, with Years 2 and 3 demonstrating lower attainment by 4 months on average and year 9 by 5 months on average.  
• Both these patterns are consistent in direction across year groups, so we can be reasonably confident they are not solely attributable to the expected minor level of change over time.

In summary, in 2022/23 average attainment for pupils in English Reading was lower than it was in 2020/21, equivalent to 4 months’ lower attainment.

Numeracy (Reasoning)

The Numeracy (Reasoning) personalised assessment focuses on how well pupils can use and apply what they know to solve numerical problems.
It is taken by pupils in Years 2 to 9. It was the last assessment to be introduced in the phased roll-out, and data is available for 2 years only: 2021/22 and 2022/23. 

Figure 4: Numeracy (Reasoning): difference in attainment in 2022/23 relative to 2021/22, in months

Image

Description of Figure 4: A bar chart showing the average attainment in the Numeracy (Reasoning) assessments of the cohort of pupils who were in year 3, 6, or 9 in 2022/23.  Average attainment is presented as months higher or lower than the attainment of the same year group in 2021/22.  

Main points

• In 2022/23 pupils were demonstrating higher attainment in Numeracy (Reasoning), on average and across all year groups, relative to pupils in the same year groups in 2021/22, equivalent to 6 months’ difference in attainment.
• This pattern is consistent in direction across year groups, so we can be reasonably confident it is not solely attributable to the expected minor level of change over time.
• This may be attributable to pupils becoming more familiar with the Reasoning assessments’ novel question types in their second year of use, which evidence from early trialling of these assessments suggests may be occurring.  
• Given this possibility and the limited amount of data available for Numeracy (Reasoning), we would caution against interpreting this figure as reflecting a broader trend that will bear out over a longer period.

The figures in this report show the average attainment of the pupils by year group, across a range of year groups and academic years. The horizontal axis shows year groups. The legend uses colours to differentiate each academic year. The vertical axis shows the average attainment achieved by all pupils in that year group in each academic year, contextualised in units of months.  

The attainment metric underlying the personalised assessments is ‘IRT scores’ – Item Response Theory scores which are explained in detail in the technical information section below.  In order to convert attainment into more readily understandable units of months, we used an approach based on the level of attainment pupils in each year group achieved in a ‘baseline year’.

For each year group, we considered the level of attainment achieved by pupils in that year group, and the year group above and below, in the baseline year.  We then determined how much difference there was between the attainment of pupils in these year groups in IRT score units using a statistical model.  This model is explained in more detail in the technical information section below, but we describe the approach in general terms here.  

Because we know that these pupils are 1 or 2 year groups apart, we can convert the difference in IRT scores into units that relate to how much longer the pupils have been in school; 12 or 24 months.  An example of this is shown in the graph below for Year 3 for Numeracy (Procedural).

Figure 5: Calculating attainment in months

Image

Description of Figure 5: A line chart showing an example of how to calculate average attainment in months.

In this example, we determine the average attainment of Year 2, Year 3, and Year 4 pupils in our baseline year (2018/19) via the dark blue points.  The difference between Year 2 and 4 pupils’ attainment, as shown by the black arrow, converts to 24 months (as the Year 4 pupils have been in school 24 months longer than the Year 2s).  

Having established this baseline of attainment in months, we can then use it to determine how many months lower or higher the attainment of pupils in this year group was in other academic years. In the example above, the red point shows the average attainment of Year 3 pupils in 2020/21. We work out the difference in attainment between this point and the equivalent point for the baseline year (shown by the light blue arrow), and convert this difference into months by comparing its size to our 24-month difference (i.e. comparing the size of the light blue arrow with the size of the black arrow). In this example, the light blue arrow is around 13% the size of the black arrow, showing that Year 3s in 2020/21 have average attainment around 3 months lower than Year 3s in 2018/19.

When interpreting the figures in this report, it is vital to remember that this measure is relative to the difference in attainment between three year groups, in the baseline year. The data from the personalised assessments shows that, across all subjects, there is a greater difference in attainment (in absolute IRT score terms) between lower year groups than higher ones. In other words, in the baseline year there is a greater IRT score difference between the average attainment of Years 2 and 3, than between the average attainment of Years 8 and 9.

This means that when two separate year groups show the same months’ difference in attainment relative to the baseline year, there will be a greater difference in absolute terms for the younger pupils. 

The personalised assessments are adaptive in nature. Each pupil receives a different set of questions to their peers, and each pupil’s assessment is dynamically tailored - if they get questions right, they will receive harder questions next, and if they answer questions incorrectly, they will receive easier questions next. This means that it is not valid to compare raw scores that pupils achieve on their assessments, as each pupil sees different questions of varying levels of difficulty.

Therefore, the personalised assessments make use of Item Response Theory (IRT) to work out how pupils have done in their assessments. IRT is a statistical approach that allows us to allocate a difficulty rating to each question based on how pupils across the year groups responded to it.  This means that this approach can account for the level of challenge of the different questions and produce IRT scores that are comparable no matter which questions each pupil answered. The ‘months’ attainment metric reported in the figures above is therefore based on IRT scores (otherwise known as ability estimates). 

The IRT scores used for the analysis in this report are not the same as the scores teachers, pupils and parents see on personalised assessment reports. IRT scores are the ‘internal’ or underlying scores that are used to produce age-standardised scores and to calculate progress points on pupil reports. The main reason for not using progress or age-standardised scores in this report is that they cannot be compared across year groups, whilst IRT scores can.

In addition to being adaptive, the personalised assessments follow an on-demand model, meaning schools can schedule assessments at any point during the academic year. Pupils taking assessments early in a given academic year tend to achieve at a slightly lower level than those who take them later on.  Therefore, when aiming to evaluate whether pupil attainment in one dataset differs from that in another, it is important to control for the impact of ‘learning time’.  This report therefore subsets the data to the period when the majority of assessments were taken - March to July. This mitigates (but does not entirely remove) the risk that the effects observed are due to pupils taking assessments earlier or later in one year than in another.

The entire national cohort did not take the personalised assessments in the period March to July. This means that, for the purposes of this analysis, there is a risk that the pupils who do take assessments in this period are not fully representative of the whole population. We have therefore applied weightings to the average scores reported in this paper to account for this possibility, based on demographic data available. This mitigates the risk that the effects observed are due to systematic differences in which pupils take assessments in this time period in each year.

Schools have the option to run each assessment twice within each academic year. This has also been accounted for within the weighting, as otherwise some pupils would be ‘double counted’ in the analysis. If a pupil took an assessment twice within the March-July period analysed in the same academic year, each one is allocated half the weight it would otherwise have had.

In terms of how attainment differences over time are contextualised, regression models were used to determine the trend in average attainment across each year group. Eight separate linear regression models were fitted for each subject/year of delivery, one per year group. Each year group’s regression used 2-3 year groups’ data; that from pupils in the year group in question and those in the year groups immediately above or below it.  Each regression’s formula was [IRT score ~ year group], meaning the resulting coefficient for year group could be interpreted as the amount of IRT score attainment by which we expect pupils in two different year groups to differ (on average).  

This allowed the reporting of score differences to be contextualized in units of months, as outlined in the body of the paper. The baseline for attainment for each assessment corresponds to the start of that assessment’s operation and/or the first point at which sufficient data was available. The baseline was produced on attainment data from specific academic years as follows:

• Numeracy (Procedural): 2018/19
• Welsh Reading: 2020/21
• English Reading: 2020/21
• Numeracy (Reasoning): 2021/22

This is the first release on reading and numeracy based on national-level data from personalised assessments. It is intended that releases will be made on an annual basis and in future will include information on demographic differences and trends.

The patterns over time seen in this release mirror the patterns seen elsewhere internationally as individual countries and education systems recover from the impact of the global pandemic. A few examples are listed below but this is not an exhaustive list.

National Literacy Trust article summarising multiple UK studies on the impact of Covid-19 on reading, and with reference to the attainment gap between more advantaged and less advantaged pupils.

Article by Harvard Graduate School of Education (May 2023) summarising research into Covid impact on attainment in several states in the USA, noting the negative impacts of Covid and socio-economic differences.

Study on learning loss due to school closures during the Covid-19 pandemic, based on data of primary school children from the Netherlands, published as a paper for Proceedings of the National Academy of Sciences of the United States of America, April 2021. 

Paper published in December 2022 by the World Bank Group on the educational and economic impacts of Covid-19 school closures in Poland.

An international study on learning loss following the pandemic and the educational inequalities between children from different socio-economic backgrounds published in Nature, January 2023.

Report from the National Audit Office on education recovery in schools in England following the pandemic, published February 2023.

Study by the National Foundation for Educational Research on the impact of Covid on educational attainment in England, published March 2022.

Statement of voluntary application of the Code of Practice for Statistics

These statistics are not classed as official statistics. However we have applied the principles of the Code of Practice as far as possible during development.
These statistics have been developed at pace and published the earliest possible opportunity to assist users in understanding the patterns of attainment in reading and numeracy for our younger pupils and the potential impact of the coronavirus (COVID-19) pandemic. It has not yet been possible to meet the following requirements for official statistics:

• The release of these statistics was pre-announced three weeks in advance.
• There has been no opportunity to engage with users to understand what they need from these statistics.

We will address all these points in the next publication in the series in Spring 2024.
We have therefore voluntarily applied the Code of Practice for Statistics as follows:

Trustworthiness

The data have been extracted from operational systems and tabulated by external contractors with extensive experience in analysing and presenting such information and include professional statisticians. The release has been produced with advice from statisticians who work under the supervision of the Welsh Government Chief Statistician to ensure that the statistics, data and explanatory material is presented impartially and objectively.

Although the publication of these tables has not been pre-announced in the same way as official statistics, they have been published at the earliest opportunity to assist users in understanding patterns of attainment in reading and numeracy over time and the potential impact of the coronavirus (COVID-19) pandemic. This is the first year for which national-level data is available as a time series for all assessment subjects This output will be developed further over the coming months with a view to developing a regular annual output from Spring 2024 onwards.

All personal data underlying these statistics are processed in accordance with the requirements of the Data Protection Act 2018. We have put in place a thorough governance process with the contractor to ensure that the data are securely managed and reviewed before release.  

Quality

The data in this release originate from the online adaptive personalised assessments taken annually by all pupils in years 2 to 9. This has been matched with demographic data on pupils from our annual school census. Both sources originate from schools and undergo extensive validation both on entry in individual schools and once the data has been received by us. These data sources are considered to be of sufficient quality to support this analysis. All stages in the collection, validation and production of these statistics are supported by professional statisticians working for the contractor and statisticians from the Government Statistical Group.

Our data collection tools and processes have been tailored and refined to suit the requirements of these assessments. Established and proven statistical techniques and packages have been used to ensure the robustness of the analysis.

All data are quality-assured prior to publication. The analysis has been verified by independent analysts, and all outputs have passed through multiple rounds of review.

Value

By summer 2023 pupils in Wales were assessed for the second year in numeracy (reasoning), marking the point at which there were at least two years of data for all four areas that are assessed. This provided sufficient data to be able to present a robust national picture. We have published this release at the earliest opportunity to assist users in understanding patterns of attainment in reading and numeracy over time and the potential impact of the coronavirus (COVID-19) pandemic, particularly amongst our youngest pupils.

The figures have been published in an accessible ODS format which can be shared and reused widely and which complies with the Government Analysis Function guidance on Releasing statistics in spreadsheets. Data are clearly presented in each table, with the spreadsheet also including a cover sheet listing each table. The commentary and notes in the release have been developed to try to make the information as accessible as possible to the widest range of users.

The data in this release is extracted directly from information made available to schools, pupils and teachers following the completion of the online assessments. This does not place any additional burden on pupils, teachers, schools or local authorities.

This is the first release in what will become an annual official statistics release. Following this release we will be consulting with key users on the content, timing and accessibility of this release. We will use this feedback to improve subsequent releases.

Quality

This section provides a summary of information on this output against five dimensions of quality: Relevance, Accuracy, Timeliness and Punctuality, Accessibility and Clarity, and Comparability. It also covers specific issues relating to quality of 2023 data and describes the quality management tool applied to this area of work. 

Relevance 

The purpose of personalised assessments is to support progression in reading and numeracy by providing pupils, schools, parents and carers with feedback on skills and progress. At a national level, anonymised data from personalised assessments can also provide information on patterns of attainment and demographic differences which may help support understanding of educational trends in Wales over time.   

Accuracy 

The Welsh Government works closely with the supplier appointed by the Welsh Government to deliver the online personalised assessments in order to ensure all data are validated before tables are published. Various stages of automated validation and sense-checking are built into the process to ensure a high quality of data. 

More information on the nature and administration of the personalised assessments can be found on the personalised assessments pages of the Welsh Government website. 

Timeliness and punctuality

These results are being published at the earliest opportunity to assist users in understanding patterns of attainment in reading and numeracy over time and the potential impact of the coronavirus (COVID-19) pandemic. The assessments covered in this release were undertaken up to the end of the summer term 2023.

Accessibility and clarity

This Statistical Release is pre-announced and then published on the Statistics section of the Welsh Government website. It is accompanied by an Open Document Spreadsheet.

Comparability

The personalised assessments have been designed to meet the needs of pupils and teachers in Wales and are not comparable with assessment data for other countries e.g SATS in England. 

Well-being of Future Generations Act (WFG) 

The Well-being of Future Generations Act 2015 is about improving the social, economic, environmental and cultural wellbeing of Wales. The Act puts in place seven wellbeing goals for Wales. These are for a more equal, prosperous, resilient, healthier and globally responsible Wales, with cohesive communities and a vibrant culture and thriving Welsh language. Under section (10)(1) of the Act, the Welsh Ministers must (a) publish indicators (“national indicators”) that must be applied for the purpose of measuring progress towards the achievement of the Well-being goals, and (b) lay a copy of the national indicators before Senedd Cymru. Under section 10(8) of the Well-being of Future Generations Act, where the Welsh Ministers revise the national indicators, they must as soon as reasonably practicable (a) publish the indicators as revised and (b) lay a copy of them before the Senedd. These national indicators were laid before the Senedd in 2021. The indicators laid on 14 December 2021 replace the set laid on 16 March 2016.  

Information on the indicators, along with narratives for each of the wellbeing goals and associated technical information is available in the Well-being of Wales report. 

Further information on the Well-being of Future Generations (Wales) Act 2015.

The statistics included in this release could also provide supporting narrative to the national indicators.

Statistician: Steve Hughes
Email: school.stats@gov.wales
 
Media: 0300 025 8099

SFR: 101/2023