Measuring the Impact of the Pandemic on Grade 2-3 Reading Performance for 2020-21

For this brief report, the impact of being out of school and doing home-based learning for grade 3 students was examined in a rural school district. The sample size is approximately 300 students.

Measures:

  1. I-Ready beginning of the year (BOY) reading test overall percentile score converted to an NCE (normal curve equivalent) score.
  2. Beginning of Grade 3 (BOG) ELA test percentile rank score converted to NCE score.
  3. Difference scores between each student’s I-Ready BOY NCE and the corresponding BOG NCE score. By using intra-individual score differences, differences in overall average score from the two different cohort groups were not a confounding factor for this analysis.
  4. BOG percent proficient was computed using counts of level 3-5 and dividing it by the total number of scores. This was computed for each school.

Procedure:
For the fall 2019 and the 2020 (current year) the BOG scores were collected and matched into a single table with the previous year I-Ready BOY data score.

  1. The joined table included columns for the year, school, SID, I-Ready tier, NCE BOG, and NCE I-Ready scores.
  2. A difference score was computed for each student by subtracting the BOY I-Ready NCE score from the BOG NCE score. For example:
    2020 BOG NCE = 50 and the BOY I-Ready NCE score = 45  difference = +5
  3. For each school the average of the differences was computed. 
  4. NCE score differences were disaggregated by school and by reading tier (Tier 1, Tier 2, and At-risk for Tier 3).

Findings: 

  1. The average I-Ready to BOG NCE difference for the 2019-20 school year for the district was 6.5 with a range of 1.0 to 9.7.
  2. The average I-Ready to BOG NCE difference for the 2020-21 school year for the district was -19.8 with a range of -17.8 to -24.2. 
  3. Differences by Tier for the two different school years were as follows:
    1. 2019 Tier 1   5         Tier 2  6.7        At-Risk for Tier 3  10.1
    2. 2020 Tier 1   -26.8      Tier 2  -20.7     At-Risk for Tier 3  -9.9
  4. The 2019 BOG district proficiency was 24.1% and this is a consistent score for the last three years. The 2020 district proficiency was 11.5%.

Conclusion:
The impact of being out of school for the spring of 2020 due to the pandemic is measurable for grade 3 students using the methodology described above. Furthermore, the results suggest that there is a negative impact on reading performance for the student in this district, and the impact cuts across all reading tier levels.

Actions:
 Present this data to the schools and make a school roster of students for each school so that the school leaders and teachers can identify the students who experienced the greatest negative impact so that interventions can be provided.

Use the I-Ready diagnostic data to determine if there is a pattern of weak areas across the most negatively impacted students. For each student determine specific areas had the greatest negative impact and provide targeted individual intervention.  

Policy Impact:
While growth for grade 3 students can be determined by comparing BOG to EOG NCE scores, Percent proficient for grade 3 is quite likely to be lower than the 2018 EOY percent proficient. This is likely to be the case for ELA for grades 3-8. Accountability using the current targets will be problematic.

Test Score Analytics Using APEX

What is the value for a school district to have all of its student performance data in one database?

The answer is quick and accurate multi-year analytics. After the initial report is created, each year is uploaded and populates all of the reports.

Example

Here is an example from a school district, that is looking forward to this year’s Beginning of Grade 3 test (BOG) data and making comparisons to other test data. Using a brief SQL statement the mean scores and standard deviations can be computed and reported for each school for each year in the APEX data system.

Then the distribution of scores in the form of a histogram for each of the years can be viewed. Using a simple drop-down filter, the year and the individual school can be overlaid and viewed .

The question is then what is the relationship between the BOG scores and the i_Ready reading scores when both are converted to Normal Curve Equivalent (NCE) scores? Here is the sample SQL code for that report:

select
    BOG.SCH_CODE as SCH_CODE,
    count(BOG.BOG_NCE),
    round(CORR(BOG.BOG_NCE, I_READY_RD.IR_RD_NCE), 3)
 from I_READY_RD I_READY_RD,     BOG BOG
   where I_READY_RD.SID=BOG.SID and BOG.YEAR = 2020
      and I_READY_RD.TEST_DATE = ‘BOY’
        group by BOG.SCH_CODE
      order by BOG.SCH_CODE

The next question is: What does this data look like as a scatter plot. In the scatter plot, an entire district can be shown for a testing year or the scatter plot can be made to show one school, and then the other data color will show al of the remaining schools. In conclusion, when a district has the BOG and I_Ready (or I_Station) data for this year, it will be better able to understand the significance of the information and perhaps understand the impact on being out of school for months has had on its students.