2001-02 ABC PROGRAM INFORMATION
SETTING ANNUAL GROWTH STANDARDS
Revised January, 2002
Vol. 1, No. 3
Setting Annual Growth Standards: "The Formula"
The ABCs of Public Education is a comprehensive plan to reorganize public schools in North Carolina. This plan focuses on: (1) strong accountability, (2) emphasis on the basics and on high educational standards, and (3) maximum local control. A key component of the ABCs of Public Education is an accountability program that focuses on the performance of individual public schools (rather than school systems) in the basics of reading and mathematics. Rather than comparing different students from one year to the next, this plan—the School-based Management and Accountability Program—holds schools accountable for the educational growth of the same groups of students (cohorts) over time. The State Board of Education annually sets challenging growth expectations for each school using the formulas described in this Accountability Brief. The growth of students is determined by scores on the North Carolina End-of-Grade Tests of Reading Comprehension and mathematics. The scores from these tests are reported on developmental scales, which yield rulers to measure growth in these subject areas across time and, therefore, across grades. Just like height in inches, on average, student scores in reading and mathematics are expected to increase every year. Like height, the rate of growth is somewhat faster in the earlier grades than in the later grades. In addition, the rate of growth varies by subject area, with scores on mathematics tests in these grade levels growing faster than scores on reading tests. The Department of Public Instruction provides computer software to each local education agency to perform all of the calculations associated with determining growth expectations for each school and whether a school has met the growth standards.
What has been the actual growth rate of North Carolina students?
The North Carolina End-of-Grade Tests were first administered statewide at the end of the 1992-93 school year. Additional equivalent forms of the tests were administered at the end of the 1993-94 school year. The average statewide growth of the students (a cohort) from one grade to the next was determined by subtracting the 1992-93 scores from the 1993-94 scores. In order to determine growth for grade 3, a pretest is administered each year during the fall. For grade 3, the average statewide growth was originally determined during the 1996-97 school year and then revised in 2000-2001. These values will be constants in the growth formula of the ABCs Accountability Model until new values are approved by the State Board of Education. For example, the average score of the North Carolina grade 3 students on the reading test was 142.7 in 1992-1993 and the average score of grade 4 students was 147.9 in 1993-1994. Therefore, the average growth in reading from grade 3 to grade 4 was 5.2 scale score points.
|Pre 3 to Grade 3||
|Grade 3 to Grade 4||
|Grade 4 to Grade 5||
|Grade 5 to Grade 6||
|Grade 6 to Grade 7||
|Grade 7 to Grade 8||
Note. These values will be used until the State Board of Education approves data from different years for determining the NC Average Rate of Growth. Grades 3 through 8 based on Spring 1993 to Spring 1994. Pre 3 to Grade 3 based on revised Spring 2001 data.
What about students who are "not average"? Do they grow at different rates?
A teacher or principal might ask these questions knowing that some students in a class or school may be well below or well above average. Since several years of data are available, the student records can be matched and checked to see whether or not all students grow at the same rate.
In fact, different rates of growth are expected for two different reasons:
- Students who are more proficient might grow faster. That is how they got to be more proficient in the first place.
- Students who score high on a particular test one year may not score as high the next year, and students who score low one year may score higher the next year, partly due to "regression to the mean."
Once we have measured the magnitude of these two effects, we estimate growth for all students.
How do we estimate "true proficiency" and "regression to the mean" when we have only last year’s test scores?
Note that the two reasons for different rates of growth described above are somewhat contradictory (i.e., if last year’s scores are used as estimates of proficiency, students with high scores last year are expected to grow more (reason # 1) and less (reason # 2) and vice versa for low scores.
The end-of-grade reading and mathematics tests are correlated (the correlation coefficients range from 0.73 to 0.80). The sum of the reading and mathematics scores can be used as an index of "true proficiency." This is like using the "Total Battery Scale Score" of a norm-referenced test as an index of ability or proficiency, except end-of-grade test scores are used.
What is the "formula" for calculating growth?
To calculate the amount of growth a school is expected to make during one school year, three factors are used in an equation. The factors are:
- The North Carolina average rate of growth in the
respective grade and subject, (b0 ).
- An estimate for the "true proficiency" of the students in a
school, (b1 x Index for True Proficiency" [ ITP]
- An estimate for the movement of students’ scores due to "regression to the mean," (b2 x Index for Regression to the Mean [ IRM] ). The formula for determining expected growth is: Expected Growth = b0 + (b1 x ITP) + (b2 x IRM)
The North Carolina Average Rate of Growth (b0)
The North Carolina Average Rate of Growth (b0) is the actual growth observed during the second year of the end-of-grade testing program. The same students (in grades 3 through 8) were followed from 1992-1993 to 1993-1994 for each grade level. The values of b0 will not change in the formula (see the table on page 3 for the values for each grade and subject) unless approved by the State Board of Education.
Estimates of "true proficiency" and "regression to the mean"
The North Carolina Average Scale Scores (for grades 3 through 8) used in the indices for "true proficiency" (ITP) and "regression to the mean" (IRM) are from the 1994-1995 school year. Average Scale scores for pre to post grade 3 are from the 2000-2001 school year. These values (scores) are used to Estimate "True Proficiency" and "Regression to the Mean."
|Pre 3 to 3||
|Grade 3 to 4||
|Grade 4 to 5||
|Grade 5 to 6||
|Grade 6 to 7||
|Grade 7 to 8||
Note. These values will not change from year to year unless approved by the State Board of Education.
North Carolina Average Scale Scores: Pre 3 to 3 based on 2001 Spring data, and Grades 3 to 8 based on 1994-1995 school year.
Estimating "True Proficiency"
In order to estimate the true proficiency of the students in a school, the reading and mathematics scale scores for the end-of-grade tests are combined to give a total overall score. The index for true proficiency (ITP) is computed by subtracting the approved North Carolina averages from the local test scores (see table above). So,
ITP = (LReadSS + LMathSS) (NCReadSS + NCMathSS).
And the estimate for "true proficiency" for a school is
- "true proficiency" (Reading) = b1 x ITP, where b1 = 0.22 for all grades, and
- "true proficiency" (Math) = b1 x ITP, where b1 = 0.26 for all grades.
For example, to determine the expected growth of a group of students during the fourth grade in 2001-2002, we need to first start with their third grade scores in 2000-2001.
- Reading = 144.0
- Mathematics = 142.0
- The North Carolina averages were 143.4 on the reading test and 141.2 on the mathematics test.
For example, to estimate the average "true proficiency" of the school the following equations would be used:
- ITP = (144.0 + 142.0) (143.4 + 141.2) = +1.4
- "True proficiency" (Reading) = 0.22 x 1.4 = 0.31, and
- "True proficiency" (Math) = 0.26 x 1.4 = 0.36.
Note that this school is above average in "true proficiency" and therefore would be expected to grow at a faster rate in reading and mathematics.
Estimating "Regression to the Mean"
In order to estimate the movement of students’ scores due to "regression to the mean," the index for regression to the mean (IRM) is computed by subtracting the approved North Carolina averages from the local test scores (reading and mathematics respectively). So,
- IRM(Reading) = LReadSS NCReadSS, and
- IRM(Math) = LMathSS NCMathSS.
To estimate for "regression to the mean" for a school is
- "regression to the mean" (Reading) = b2 x IRM, where b2 = -0.60 for all grades, and
- "regression to the mean" (Math) = b2 x IRM, where b2 = -0.58 for all grades.
For example, to estimate "regression to the mean" of the school the following equations would be used:
- IRM(Reading) = 144.0 143.4 = +0.6
- "regression to the mean" (Reading) = -0.60 x 0.6 = -0.36, and
- IRM(Math) = 142.0 141.2 = +0.8
- "regression to the mean" (Math) = -0.58 x 0.8 = -0.46.
Note. Since this school is above average, the effect of "regression to the mean" will be to lower the school’s expected growth.
Calculation of Expected Growth
Now using the formula for expected growth described on page 3, the expected growth for a school would be
- Expected Growth = Expected Growth = b0 + (b1 x ITP) + (b2 x IRM).
For example, from the Table on page 2 the average growth rates for grade 3 to grade 4 are :
- Reading = 5.2 points
- Mathematics = 7.3 points
The expected growth for this fourth grade class would be
- Expected Growth (Reading) = 5.2 + 0.31 + (-0.36) = 5.15, and
- Expected Growth (Math) = 7.3 + 0.36 + (-0.46) = 7.2
2000-2001 Mathematics Tests Equating Study
In May of 1998, the State Board of Education (SBE) adopted a new K-12 mathematics curriculum. A transitional curriculum was implemented in 1999-2000 involving the teaching of the old and new mathematics curricula to accommodate both operational testing and the development of new mathematics tests. Using the mathematics field test data from spring 2000, the new (2nd Edition) mathematics tests were assembled and were administered for the first time in the spring of 2001.
In the ABCs, growth is calculated from year to year. So it was necessary to conduct a special equating study during the summer 2001 to be able to convert the 2nd edition mathematics scale scores to equivalent scores on the old mathematics scale. Once the two series of tests were equated, the existing ABCs growth formulas could continue to be used for the 2000-01 ABCs.
For the 2000-2001 ABCs, the 2nd edition mathematics scores were converted onto the old mathematics scale and the existing ABCs growth formulas were used to calculate ABCs status (with minor modification of the growth model for fall to spring growth in the third grade, because both pre and post scores in third grade were on the new mathematics scale.) The State Board of Education has approved the use of the same formulas for the 2001-2002 ABCs by converting both pre and post test mathematics scores to the old scale, using the results of the equating study. (For additional information on the equating study see 2000-2001 the appendix of A Report Card for the ABCs of Public Education, Volume I: 2000-2001 Growth and Performance of North Carolina Schools).