Measuring the Impact of the Pandemic on Grade 2-3 Reading Performance for 2020-21

For this brief report, the impact of being out of school and doing home-based learning for grade 3 students was examined in a rural school district. The sample size is approximately 300 students.

Measures:

  1. I-Ready beginning of the year (BOY) reading test overall percentile score converted to an NCE (normal curve equivalent) score.
  2. Beginning of Grade 3 (BOG) ELA test percentile rank score converted to NCE score.
  3. Difference scores between each student’s I-Ready BOY NCE and the corresponding BOG NCE score. By using intra-individual score differences, differences in overall average score from the two different cohort groups were not a confounding factor for this analysis.
  4. BOG percent proficient was computed using counts of level 3-5 and dividing it by the total number of scores. This was computed for each school.

Procedure:
For the fall 2019 and the 2020 (current year) the BOG scores were collected and matched into a single table with the previous year I-Ready BOY data score.

  1. The joined table included columns for the year, school, SID, I-Ready tier, NCE BOG, and NCE I-Ready scores.
  2. A difference score was computed for each student by subtracting the BOY I-Ready NCE score from the BOG NCE score. For example:
    2020 BOG NCE = 50 and the BOY I-Ready NCE score = 45  difference = +5
  3. For each school the average of the differences was computed. 
  4. NCE score differences were disaggregated by school and by reading tier (Tier 1, Tier 2, and At-risk for Tier 3).

Findings: 

  1. The average I-Ready to BOG NCE difference for the 2019-20 school year for the district was 6.5 with a range of 1.0 to 9.7.
  2. The average I-Ready to BOG NCE difference for the 2020-21 school year for the district was -19.8 with a range of -17.8 to -24.2. 
  3. Differences by Tier for the two different school years were as follows:
    1. 2019 Tier 1   5         Tier 2  6.7        At-Risk for Tier 3  10.1
    2. 2020 Tier 1   -26.8      Tier 2  -20.7     At-Risk for Tier 3  -9.9
  4. The 2019 BOG district proficiency was 24.1% and this is a consistent score for the last three years. The 2020 district proficiency was 11.5%.

Conclusion:
The impact of being out of school for the spring of 2020 due to the pandemic is measurable for grade 3 students using the methodology described above. Furthermore, the results suggest that there is a negative impact on reading performance for the student in this district, and the impact cuts across all reading tier levels.

Actions:
 Present this data to the schools and make a school roster of students for each school so that the school leaders and teachers can identify the students who experienced the greatest negative impact so that interventions can be provided.

Use the I-Ready diagnostic data to determine if there is a pattern of weak areas across the most negatively impacted students. For each student determine specific areas had the greatest negative impact and provide targeted individual intervention.  

Policy Impact:
While growth for grade 3 students can be determined by comparing BOG to EOG NCE scores, Percent proficient for grade 3 is quite likely to be lower than the 2018 EOY percent proficient. This is likely to be the case for ELA for grades 3-8. Accountability using the current targets will be problematic.

Building a Data Culture in Schools: Data Access is a Critical Component

Building a Data Culture in Schools: Data Access is a Critical Component

There have been hundreds of articles, blog postings and presentations on using data in schools. The information presented focuses on topics such as the inquiry cycle using data, data teams,  and leadership and teacher skills for understanding data. A critical aspect of this data use process is getting the data needed to make informed decisions. Issues surrounding data availability include
1 Access in a secure user-friendly format,
2. timeliness of receiving the data, such as growth data being available which is completed months after         testing is scored and the school year has started in the fall, and
3. access to interpretation protocols.

The first critical issue is having data in a user-friendly format. Many schools provide the printouts from the scoring software with the student scores. This report requires the teacher to look up previous scores and make a comparison to determine if a student did better than the previous year. After all of the testing is completed in the district, the testing office can build spreadsheets of student scores and provide this information to the administrator. However, to be useful, the data needs to be matched to each student’s previous scores to put some sort of context to the information.

Data systems range in size and price to full data warehouse systems. These systems cost over $100,000 and require someone to manage and maintain the system. Another option is a data system for collecting and reporting the data. These systems can be secured at a cost of about $3 to 8 dollar range per ADM. However, these data systems may not be custom created and may not offer the data analysis support a school district may need.

If you would like assistance in selecting or building your own data system, contact Data Smart LLC.
www.Data-smart.net

Evaluation of Curriculum Using Data 

The Need for Data
With the EVAAS data on growth being released now, curriculum leaders can use this information for evaluation of programs and curriculum.  Unfortunately, due to the timing of the release of the growth information, making a change in direction after school has started is problematic.

Instead of waiting, your district could compute NCE Difference growth in July and use this data to gain some insight as to what is happening in the schools and answer some program evaluation questions.  Here is an example:

A school uses a math program which is intended to help students maintain skills already acquired and to build skills using a spiral approach. It is an older program and must be supplemented to meet the more rigorous math standards. Overall the EVAAS math growth has been negative for the last few years.

The Data
To add data to the analysis, the student NCE Difference scores in both reading and math were aggregated based on the PREVIOUS achievement level. For grade 7 the NCE Difference scores looked like this:

Previous Level       Count     Math NCE Diff          Count RD       Reading NCE Diff

1                      16                    -3.5                              16                    9.4

2                      26                    -3.4                              19                    5.4

3                      7                      -9.8                              10                    0.4

4                      24                    -9.54                            31                    2.4

5                      12                    -9.42                            9                      -7.2

 

It is worthwhile to note that all of the middle school grades had scores which followed this same pattern, and that there was not a big difference between teachers’ growth who taught these grades. Based on the information, math performance in terms of growth is a problem area. It is possible that all of the teachers tended to teach in “remediation mode” focusing on the students who had scored level 1 and 2. If that is the case, it is still likely that the math program is not meeting the curriculum needs of the students who are scoring level 3 and above, especially when the same pattern is found in all three grades.

Because the district has a historical data system, the district was able to go back one year and examine the pattern of growth. The district found that the same pattern of low math growth in performance levels 3 and above.

Timeliness or Information
Having this information available in early July provides the opportunity to use the summer months to investigate the situation and make changes in the curriculum and materials.  This is why having a historical data system is so important for school leaders to make curriculum and materials decisions.

If you are interested in hearing more about creating a historical data system for your district which can provide timely data analytics and reports feel free to contact me at Lew.Johnson@data-smart.net