Assessing Pandemic Learning Losses and Recoveries

07/03/2023
Featured in print Digest

This figure is a bar graph titled, Changes in Standardized Test Pass Rates, 2019 to 2022. It is subtitled, Each bar represents a bin of districts within the relevant 20 quantiles of change between 2019 and 2021.  The y axis is labeled percentage points and ranges from negative 40 to positive 20, increasing in increments of 10.  The graph is divided into two panels. The left panel is labeled English language arts and the right side is labeled Math. There are two categories of bars for each side: 2019 to 2021 change and 2021 to 2022 change. Each category has a corresponding 20 bars, making a total of 40 bars on each side of the panels.  For English language arts, the 2019 to 2021 changes are mostly negative, ranging from about negative 15 to 0. The distribution of the bars is organized in almost an exact descending order from largest decrease to smallest. The 2021 to 2022 changes are mostly positive, ranging from 0 to about positive 5.  For Math, the 2019 to 2021 changes are mostly negative and of greater magnitude than the changes seen in English language arts. These decreases range from about negative 35 to 0. The 2021 to 2022 changes are all positive and are also of greater magnitude compared to English language arts. These increases range from 0 to positive 15. For both categories, the bars are distributed in a way that appear to be descending from largest change to smallest.  The source line reads, Source: Researchers’ calculations using stat standardized assessment data from 21 states.

Test scores at the end of the 2020–21 school year revealed dramatic declines in student performance during the COVID-19 pandemic. In a data sample including schools from 21 states, only a fraction of those losses — 20 percent in English language arts and 37 percent in math — were recovered by the end of the next school year.

Those nationwide averages mask wide variations in student performance among individual states, according to Post COVID-19 Test Score Recovery: Initial Evidence from State Testing Data (NBER Working Paper 31113) by Clare HalloranClaire E. HugRebecca Jack, and Emily Oster.

Analysis of test scores in 21 states shows less than half of the losses have been regained overall, with wide variations across states.

The researchers focused on grades 3–8 and restricted their sample to states that maintained the same testing and scoring practices over the 2017–22 study period. The sample includes Arkansas, Colorado, Connecticut, Georgia, Idaho, Kansas, Louisiana, Massachusetts, Minnesota, Mississippi, Missouri, New Hampshire, Ohio, Pennsylvania, Rhode Island, South Carolina, South Dakota, Virginia, West Virginia, Wisconsin, and Wyoming.

Across states, recovery rates in English language arts varied much more than those in math. Student performance in two states, Mississippi and South Carolina, fully recovered from pandemic losses in English language arts while that in Kansas and Massachusetts, among other states, continued to decline. In math, student performance in all states experienced recovery, but none recovered completely. Mississippi and Rhode Island math scores recovered by over 70 percent, while in Arkansas and Minnesota the recovery was under 20 percent.

The researchers analyzed data for individual school districts and found that recovery rates were highly idiosyncratic. They did not significantly correlate with demographic characteristics, baseline achievement, schooling mode during the pandemic, or how states chose to allocate federal recovery funds.

The researchers were particularly interested in whether the way a state allocated its share of the $122 billion in federal relief funds that were targeted toward education affected initial losses and subsequent recovery. States varied widely in how they addressed students’ academic, social, emotional, and mental health needs. The researchers did not find any statistically significant links between intervention techniques and recovery rates, but they caution that some school districts may not yet have fully realized the benefits of programs financed by federal relief money, complicating the analysis of links between recovery rates and federal grants.

The researchers found a potential correlation between English language arts recovery and how long states have had legislation implementing reading improvement programs. The only two states that fully recovered pandemic learning losses by 2022 were also the earliest adopters of the science of reading (SOR) program: Mississippi (2013) and South Carolina (2014). Both states also took other steps to improve reading, including enhanced teacher preparation, professional development, and instruction. States that implemented the program more recently show mixed recovery rates.

Steve Maas