ABC PROGRAM INFORMATION
HISTORY OF THE ABCs PROGRAM
Table of Contents
Purpose and Uses of Accountability Data
Developmental Scale Scores
Achievement Levels (EOG)
Achievement Levels (EOC)
Student Accountability Standards
Alternate Assessments and New Accommodation
2000 - 2001 ABCs Accountability Model
Determining Growth in Other Components
End-of-Course -- Exemplary Growth
Challenges, Changes and Other Good Points
Evolution of the ABCs
The EOC Prediction Formulas
Parameters and Other Constants Used in the ABCs Growth Model for 1999-2000
2000-2001 Constants Used in the ABCs Growth Model for High Schools
NORTH CAROLINA'S ABCs OF PUBLIC EDUCATION
In 1995, the General Assembly of North Carolina passed a law (Senate Bill 16) that directed the State Board of Education (SBE) to examine the administrative organization of the Department of Public instruction (DPI) and propose a plan for reducing and/or reorganizing the department. One of the key recommendations of the plan was developing a new accountability plan that focuses on performance of public schools with a system of clear rewards and consequences. Pursuant to the 1995 legislation, the ABCs was piloted in 10 North Carolina school systems. In response to the School-Based Management and Accountability Program (SB 1139) passed by the General Assembly in June, 1996, the SBE subsequently implemented the ABCs of Public Education.
The ABCs accountability model sets growth and performance standards for each elementary, middle, and high school in the state. End-of-grade (EOG) and end-of-course (EOC) test results and selected other components are used to measure the schools' growth and performance. What makes this model different from other accountability models is that the ABCs philosophy is based on the desire that all schools have the chance to meet the accountability standards (and thus the focus on growth standards-all students can grow), as opposed to having a standard that simply looks at percent of students scoring at a selected proficiency level (which may be harder to attain for schools of certain socioeconomic/demographic conditions).
The ABCs focuses on (a) strong accountability with an emphasis on high educational standards; (b) teaching the basics; and (c) maximum local control. The first ABCs accountability model was implemented in elementary and middle schools in 1996-97; a high school model was developed the same year and was implemented for the first time in 1997-98. In 1998-99 the two accountability models were combined into one comprehensive ABCs model for elementary, middle and high schools. In 1999-2000 this model continued, augmented by a new policy for participation of alternative schools. Additionally, the 1999-2000 accountability year was the first year of reporting ABCs results for schools that are administered by the Department of Health and Human Services and the Department of Juvenile Justice and Delinquency Prevention. Their inclusion is based on action of the General Assembly, which requires their participation.
As the ABCs model has evolved, changes have been made to meet demands by the state legislators, SBE, and the federal government. In 2000-2001, the SBE adopted weighting the ABCs growth composites to eliminate concern over small groups of students having the same impact as large groups of students in the determination of growth. End-of-course (EOC) prediction formulas for 10 multiple choice EOC tests were implemented to address concerns related to comparing different cohorts over time at the high school level. A component also was added to the growth computations in high schools to reflect changes in dropout rates. Due to amendments to the Individuals with Disabilities Education Act (IDEA), adopted by the U.S. Congress in 1997, the SBE required all students with disabilities ( to the extent possible) to participate in state-and-district wide testing programs. In 2000-2001, alternate assessments were implemented to meet this requirement. The North Carolina Alternate Assessment Portfolio, which takes the place of EOG tests will be counted for the first time in the ABCs accountability model for the 2000-2001 school year.
Enhanced student performance is the centerpiece of the SBE's accountability model. A key component of the ABCs of Public Education is a new accountability program which focuses on the performance of individual public schools (rather than school systems) in the basics of reading, writing, and mathematics. Rather than comparing different students from one year to the next, this plan - the School-Based Management and Accountability Program - holds schools accountable for the educational growth of the same students over time. By focusing on growth, the accountability model stimulates improvement for all schools.
The changing educational environment, an increasingly diverse student population, higher expectations for education, and changes in curriculum will warrant constant fine tuning of the ABCs accountability model. The Evolution of the ABCs located in the appendix, provides complete documentation of the historical changes in the current ABCs model. This narrative focuses on the 2000-2001 accountability model.
The ABCs of Public Education is a comprehensive plan to improve public schools that is based on three goals of strong accountability, an emphasis on the basics and high educational standards, and on providing schools with local control ( www.ncpublicschools.org/accountability/). Each school is held accountable for the progress of its students. The SBE has set two types of annual performance goals.
- Performance standards refer to the absolute achievement or the percent of students' scores in a school at or above grade level.
- Growth standards are benchmarks set annually to measure a school's progress.
The expected growth for a school is based on (1) its previous performance, (2) statewide average growth, and (3) a statistical adjustment (regression to the mean) which is needed whenever test scores of the same students are compared from one year to the next. A formula is used to generate expected growth for each school.
Also called reasonable growth, the expected growth rate is the amount of growth that would be reasonably expected over a year's worth of time. Exemplary growth refers to growth that exceeds expected growth by an amount designated by the SBE.
Exemplary growth factors in 10% above the statewide average growth in the formula and is used in conjunction with the performance standard to identify schools that qualify for recognition and assistance. The SBE established Achievement Level III as the standard for grade level performance for all students.
The growth composite score summarizes the performance of students in a school in reading, writing, and mathematics. It is used to determine whether a school meets its expected growth standard or exemplary growth standard for incentive awards. The composite provides a representative picture of a school's overall growth. More detailed information about these composite scores is included later in this report.
The plan clearly defines the mission of the public school community: to challenge with high expectations each child to learn, to achieve, and to fulfill his or her potential. To ensure that this mission is carried out, the ABCs has four major implementation components:
The accountability portion of the ABCs uses EOC and EOG tests to determine if student achievement is improving. Each school has its own standard for expected and exemplary growth (progress). School Improvement plans (a requirement passed by the General Assembly) include strategies for improving student performance taking into account the annual performance goals for the school set by the SBE. The School Improvement plans are developed for a three-year period and amended as often as necessary to provide continuous student improvement. At the high school level, some additional factors are used to determine if schools are making adequate progress.
The ABCs recognize schools that meet exemplary or expected growth. Teachers who work in schools that reach expected or exemplary growth are entitled to financial incentives which are funded by the General Assembly.
- Certified staff in schools that make exemplary growth receive $1,500 and teacher assistants receive $500. Schools making exemplary growth also receive a certificate of recognition.
- Certified staff in schools that make expected growth receive $750 in incentive pay and teacher assistants receive $375. These schools receive a certificate of recognition.
- The 25 K-8 schools that make the most growth are recognized by a dated banner to hang in the school and a certificate. They also are honored at a banquet hosted by the SBE.
- The 10 high schools that make the most growth are recognized by a dated banner to hang in the school and a certificate. They also are honored at a banquet hosted by the SBE.
- Schools of Excellence are those that have at least 90% of their students at or above grade level and meet expected growth standards. They are recognized by a dated banner to hang in the school and a certificate. They too are special guests of the SBE at a banquet in their honor.
- Schools of Distinction have at least 80% of students at or above grade level. These schools receive a plaque and a certificate.
- Schools with No Recognition did not make their expected growth standards; but they have at least half of their students' scores at or above Achievement Level III as measured by the performance composite.
- Low-Performing Schools are those that fail to meet their expected growth standard and have significantly less than 50% of their students' scores at or above Achievement Level III. Additionally, schools that test fewer than 98% of their eligible students for two consecutive years may be designated as low-performing.
One of the major strengths of the ABCs is the assistance that is offered to schools that are designated as low-performing by the SBE and assigned an Assistance Team. Assistance Team members may be currently practicing teachers and staff, representatives of higher education, school administrators, retired educators, and others the SBE considers appropriate.
The purpose of an Assistance Team is to help low performing schools evaluate their teaching and learning environment and to provide services that will improve the education of all children in the schools. Additionally, the law requires the Assistance Team to carry out six specific functions:
- Review and investigate all facets of school operations and assist in developing recommendations for improving student performance.
- Evaluate at least semiannually the certified personnel assigned to the school and make findings and recommendations concerning their performance.
- Collaborate with school staffs, central offices, and local boards of education in the design, implementation, and monitoring of a school improvement plan that can reasonably be expected to alleviate the problems and improve student performance at that school.
- Make recommendations as the school develops and implements this plan.
- Review the school's progress.
- Report, as appropriate, to the local board of education, the community, and the SBE on the school's progress.
Intervention may involve a more directive approach from an assistance team, including recommendations to dismiss or demote personnel. Furthermore, if more than half the schools in a district are designated as low-performing, the SBE may appoint an interim superintendent for a limited period of time or take other action necessary to improve the performance of schools in that district.
If a school is designated as low-performing, the Principal and teachers of that school are evaluated under this plan by the Assistance Team. Throughout the school year, the Assistance Team collects information using the Principal Performance Appraisal Instrument (PPAI) concerning the principal's performance and abilities in the areas of instructional leadership, resource management, and communications. The Assistance Team will notify the SBE if at any time the Team determines that the principal's performance is impaired by his or her lack of general knowledge. Action taken by the SBE on this issue can result in dismissal.
Assistance team members use the Teacher Performance Appraisal Instrument (TPAI) to evaluate teachers in designated low-performing schools. The assistance team assigned to a low-performing school makes recommendations to the SBE to revoke or refuse to renew the teacher's certificate in a low-performing school.
Local superintendents recommend to their local school board whether to retain, transfer, dismiss, or demote principals in low-performing schools. The superintendent is required to submit a plan for addressing the needs of all identified low-performing schools in their school district to the SBE. If a school fails to improve student performance after assistance is provided, the assistance team may recommend that assistance continue, or that the SBE take further action. Detailed procedures are incorporated in the North Carolina 2000 Public School Law, GS 115C-105.39 (dismissal or removal of personnel; appointment of interim superintendent).
The Director of the Division of School Improvement (DSI) at DPI provides overall supervision, support, and management for the state assistance teams. Within the division, each section is responsible for one or more teams assigned to a geographic area of the state. Regional issues are addressed and support is provided by staff in the division.
Within each section of the DSI, a consultant is assigned as a liaison to one of the assistance teams. The liaison is responsible for direct communication and information to the assistance teams from the DPI, serves as first line contact for the teams and conducts regular on-site visits. The entire DPI serves as a resource for the teams. For more detailed information about assistance teams visit our web site at www.ncpublicshools.org/school_improvement/assistance_index.html.
The North Carolina EOC tests were initiated in response to legislation passed by the North Carolina General Assembly - the North Carolina Elementary and Secondary Reform Act of 1984. This act called on the implementation of the Basic Education Program through the establishment of a core curriculum for all students for each content area and the development of tests to assess the implementation of each curriculum across the state. North Carolina EOG tests were initiated in response to legislation passed in 1989 by the General Assembly. In 1992-93, the State completed the transition to a new assessment system - with state tests designed by North Carolina teachers, curriculum specialists, testing experts and DPI staff. These end-of-grade (for grades 3-8) and end-of-course (for high school and middle school) tests are revised when the curriculum is revised, a necessity since they measure the objectives outlined in the North Carolina Standard Course of Study (the curriculum).
Based on the 1984 and 1989 legislation, North Carolina end-of-course tests and end-of-grade tests were developed for two purposes:
- To provide accurate measurement of individual student skills and knowledge specified in the North Carolina Standard Course of Study, and
- To provide accurate measurement of the knowledge and skills attained by groups of students for school, school system, and state accountability.
North Carolina end-of-grade (EOG) tests in reading and mathematics at grades 3-8 were initially normed at the state level in 1993. The tests were later renormed using the state statistical distribution scores from the 1998-99 statewide administration.
Raw scores are converted to scale scores on the EOG tests. Developmental scale scores allow the different forms of the test to be equated, so that a developmental scale score of 150 (for example) in reading on Form A represents the same level of achievement as a developmental scale score of 150 in reading on Form B, even though the raw scores may be slightly different.
Reading and mathematics are reported on developmental scales, which yield rulers to measure growth in these subject areas across time. Just like height in inches, on average, student scores are expected to go up every year. Also like height, the rate of growth is somewhat faster in the earlier grades than in later grades. EOC test scores in high school are not reported on a developmental scale because the specific content covered in each course is different.
Developmental scale scores on EOG tests can be used to chart the average educational growth of students as they progress through the grades. Given the way the scales were developed, the differences in mathematics scores between grades in 1992-93 can provide a standard for comparing average growth across grade levels in the future (e.g., between where students were at the end of the third grade versus where they are at the end of the fourth grade. This anticipated growth varies from grade level to grade level and from subject to subject. Each year test results can be used to chart the actual growth of students from grade level to grade level within a subject. Schools and school systems may use the developmental scale scores to compare their own mean scale scores to the state average to evaluate the performance of their students. Scale scores can be averaged and relative comparisons made within each subject and across groups including schools and school systems.
It is not appropriate to compare developmental scale scores across subjects (i.e., mathematics vs. reading) because the typical growth across grades for each subject is different. It is appropriate to compare scale scores within subjects and types of tests (i.e., open-ended and multiple-choice) across years.
Achievement levels (cut scores) allow the comparison of student and group performance to standards based on what is expected in each subject at each grade level. Achievement levels were determined by relating the judgments of thousands of North Carolina teachers concerning the performance of each of their students to each student's performance (Contrasting Groups Method) on the EOG multiple-choice tests. Each achievement level corresponds to a range of scale scores for each subject and grade level. The four levels are:
- Level I - Students performing at this level do not have sufficient mastery of knowledge and skills in this subject area to be successful at the next grade level.
- Level II - Students performing at this level demonstrate inconsistent mastery of knowledge and skills in this subject area and are minimally prepared to be successful at the next grade level.
- Level III - Students performing at this level consistently demonstrate mastery of grade level subject matter and skills and are well prepared for the next grade level.
- Level IV - Students performing at this level consistently perform in a superior manner clearly beyond that required to be proficient at grade level work.
End-of-Course (EOC) tests are designed as curriculum-based achievement tests to measure what students know and are able to do within the context of a specific subject-area content. Raw scores are converted to scale scores on the EOC tests. Scale scores enable the different forms of the tests to be equated. Developmental scale scores are inappropriate for EOC tests because the content taught in each of the courses is different.
The scale scores for all tests were designed originally to have a mean of 50 and a standard deviation of 10. The mean and dispersion of scores have changed over time as actual student performance has changed.
Achievement levels (cut scores) allow the comparison of student and group performance to standards based on what is expected in each subject at each grade level. Like EOG, the Contrasting Groups Method was used in determining the Achievement Levels for EOC. Each EOC achievement level corresponds to a range of scale scores for each subject. The four levels are:
- Level I - Students performing at his level do not have sufficient mastery of knowledge and skills of the course to be successful at a more advanced level in the content area.
- Level II - Students performing at this level demonstrate inconsistent mastery of knowledge and skills of the course and are minimally prepared to be successful at a more advanced level in the content area.
- Level III - Students performing at this level consistently demonstrate mastery of the course subject matter and skills and are well prepared for a more advanced level in the content area.
- Level IV - Students performing at this level consistently perform in a superior manner clearly beyond that required to be proficient in the course subject matter and skills and are very well prepared for a more advanced level in the content area.
In 1999 the SBE approved Student Accountability Standards (SAS). These standards provide four Gateway Standards for student performance at grades 3, 5, 8, and 11. Students in the third, fifth, and eighth grades are required to demonstrate grade level performance in reading, writing (fifth and eighth grades only), and mathematics in order to be promoted to the next grade. To graduate, high school students need a passing score on a new exit exam of essential skills in addition to meeting existing local and state graduation requirements. The SAS became effective during the 2000-2001 school year for grade 5, 2001-2002 for grades 3 and 8, and at grade 11 (Spring 2004) for the graduating class of 2005. (www.ncpublicschools.org/student_promotion).
If a student scores below Level III on the first test administration, then the student must be offered a retest. After focused intervention, the student must be offered the opportunity to take the test again.
After the second or third test administration, a parent or teacher of a student who again scores below Level III may request that the student be promoted. If such a request is made, the principal at the school must implement the waiver review process as described in the Student Accountability Standards. (See www.ncpublicschools.org/accountability/testing/policies).
In prior years, the ABCs regulations and policies have permitted schools to exempt some students with disabilities from EOG tests if their Individualized Education Plan (IEP) team determined that they were incapable of pursuing the North Carolina Standard Course of Study or the team determined that the standard EOG tests, even with accommodations, were not an appropriate measure of the student's performance. One consequence of that decision was to remove those students from the evaluation of their school's performance under the ABCs. However, under federal law all students must now be included in the assessment of a school's performance except some students with Limited English Proficiency (LEP) students who fail to meet language proficiency requirements. The DPI, with guidance from the United States Department of Education Office of Special Education Programs (OSEP), and Office of Civil Rights (OCR), developed a system of accommodations and alternate assessments to fairly and effectively assess the performance of students with disabilities and their schools for purposes of the Student Accountability Standards and the ABCs.
When an IEP or 504 Plan specifies that the standard EOG test, with or without appropriate accommodations, is an appropriate means for assessing the performance of a student with disabilities, then the student must take the standard EOG test with any appropriate accommodations.
North Carolina developed and implemented two alternate assessments and one other accommodation for students with disabilities to address the need to include all students in the testing program.
- The North Carolina Alternate Assessment Portfolio (NCAAP) is an alternate assessment instrument that requires teachers to assess students using a year-long portfolio process in which the teacher collects evidence of student performance on tasks identified in four domains: Community, Career/Vocational, Communication, and Personal/Home. The student's score on the NCAAP will be included in the school's ABCs performance composite for purposes of classifying the school's performance. However, the scores will not be included in the ABCs growth model until sufficient data have been collected over several years to determine the best way to include these results in the growth component.
Students evaluated under the NCAAP are exempt from the Student Accountability Standards. The decision to promote or retain these students is made by the principal of the school with input from teachers and other appropriate staff. For more information please visit us at www.ncpublicschools.org/accountability/testing/alternate.
- North Carolina Alternate Assessment Academic Inventory (NCAAAI) is an alternate assessment in which teachers utilize an extensive checklist to evaluate student performance on curriculum benchmarks in the areas of reading, writing, and mathematics.
For the 2000-2001 school year, the results of NCAAAI (in terms of the number of students participating) will be part of a statewide pilot study. The participation results of the 2000-2001 administration of the NCAAAI will be reported in Volume II of the Report Card for the ABCs.
- The Computerized Adaptive Testing System (NCCATS) uses a computer application transmitted from a secure web site to assess student performance by selecting test questions from the EOG test item pool, which are appropriate for the individual student's level of functioning within the curriculum.
The NCCATS shares the test questions from the item bank used to create forms of the standard EOG tests across grades pre-3 to 8 and the High School Comprehensive Test (HSCT) in grade 10. NCCATS uses the same developmental scale scores and achievement levels as the standard EOG tests and HSCT. For 2000-2001, this new accommodation is part of a statewide pilot study. The results of the 2000-2001 administration of the NCCATS ( in terms of the number of students participating) will be reported in Volume II of the Report Card for the ABCs.
A complete overview of the North Carolina Testing Program can be viewed at www.ncpublicschools.org/accountability/testing. The document provides a description of the ABCs of Public Education, the Statewide Student Accountability Standards, and the 2000-2001 Testing Program. For updated information about testing issues visit the following web site www.ncpublicschools.org/accountability/testing/briefs.
A student with Limited English Proficiency (LEP) whose documented English language is assessed as Novice/Low to Intermediate/Low in listening and writing may be exempted from statewide testing for up to two years from the time of initial enrollment in a North Carolina School System. Although a student may be exempted because of limited proficiency, the school system is required to use other assessment methods in order to show how the student is progressing in English, as well as in other subject areas. A new policy requires that 12 months after the (LEP) student initially enters the school system the student must take the same English Language Proficiency test as was administered initially. If at this point the student is assessed as intermediate high or advanced, the student participates in all statewide assessments, though the student does not have to participate in statewide assessments on which s/he writes responses. After two years of initial enrollment in the school system a student with Limited English Proficiency must be included in all administrations of state tests, regardless of their level of English Proficiency.
In the ABCs, a school's growth and performance are summarized using composite scores. There are two types of composite scores: growth, and the performance composite. What makes this model different from other accountability models is that the ABCs philosophy is based on the desire that all schools have the chance to meet the accountability standards (and thus the focus on growth standards-all students can grow), as opposed to having a standard that simply looks at percent of students scoring at a selected proficiency level (which may be harder to attain for schools of certain socioeconomic/demographic conditions).
The total weighted growth composite for a school is the sum of the weighted growth components. Components of the model include:
- EOG Reading and Math (Grades 3-8),
- 10 EOC tests using prediction formulas (Algebra I, Algebra II, Biology, Chemistry, ELPS, English I, Geometry, Physical Science, Physics, and US History),
- English II,
- College University Prep/College Tech Prep,
- Competency Test (percent change/gain from grade 8 to grade 10),
- Comprehensive Test in Reading and Mathematics (growth from grade 8 to grade 10), and
- Change in ABCs dropout rate (1998-99 minus 1999-2000).
For computations of the performance composite, the total number of scores at or above achievement Level III in each subject included in the ABCs model is divided by the total number of valid scores. Components included are:
- EOG Reading and Math (Grades 3-8),
- 10 EOC tests using prediction formulas (Algebra I, Algebra II, Biology, Chemistry, ELPS, English I, Geometry, Physical Science, Physics, and US History),
- English II,
- Comprehensive Test in Reading and Mathematics (growth form grade 8 to grade 10),
- NCAAP (grades 3-8 and 10),
- Writing (grades 4 and 7), and
- Computer Skills test at grade 8.
The ABCs Model has specific data requirements and rules for implementation.
- Three years of data are required for English II, College University Prep/College Tech Prep, and the ABCs dropout rate change for inclusion in the ABCs growth composite. Schools that indicate that they have had major changes in demographics due to changes in attendance areas may request that these specific component(s) not be included as part of the ABCs growth.
- Schools must have a minimum of 30 student scores across all ABCs growth components.
- Students must be in membership for 91 days to be included in EOG growth results.
- Students must be in membership for 160 days to be included in EOC results.
In order to determine growth for grade 3, a pretest is administered each year during the fall. For schools with any combination of grades 3 through 8: growth is computed using pretest to posttest differences in grades 3 through 8 in reading and mathematics on a matched set of students. Students must be in attendance at the school for at least 91 days. The average statewide growth for grades 4-8 was determined in 1993-94. For grade 3, the average statewide growth was determined during the 1996-97 school year; for grades 8 to 10, it was determined during the 1997-98 school year. These values became the constants in the growth formula of the ABCs Accountability Model. The constants and parameters are located in the appendix.
The performance composite summarizes the performance of students in the school with respect to attaining Achievement Level III. It tells the percent of student test scores at or above Achievement Level III (consistent mastery of subject/course content matter) in the subjects taught in the school and included in the accountability model. There is no minimum number of days in membership at the school for students' scores to be included in the performance composite.
Both the performance composite and the expected growth composite are used when determining which schools may need special assistance (i.e., designated as low-performing). A school with an expected growth composite that is negative (less than zero) and a performance composite less than 50 (fewer than 50% of scores at Level III or above) may be identified as a low-performing school.
In the ABCs, a school's growth and performance are summarized using composite scores. There are two types of composite scores: growth, and the performance composite.
There are two growth composites-one for expected growth and one for exemplary growth. These growth composites summarize a school's growth over all grade levels and subjects included in the accountability model. The two composites indicate whether or not a school makes expected or exemplary growth, respectively. The growth composite scores allow a school to fall short of expected or exemplary growth in some areas but exceed it in others and still reach the growth standard overall (i.e., a compensatory model). If the expected growth composite equals or exceeds zero, the school makes the goal of expected growth. Similarly if the exemplary growth composite equals or exceeds zero then the school makes the goal of exemplary growth.
An outline of the process for determining these composite scores, using the accountability formula follows. Necessary constants and parameters are summarized in this document. The formula used in the calculations for End-of-Grade (EOG) tests is published in "The Accountability Brief, Setting Annual Growth Standards: The Formula," first published September 1996, revised February 2001. Vol. 1, no. 5. The formulas used in the calculations for the End-of-Course (EOC) tests are published in "The End of Course Prediction Formula" and "EOC Prediction Formulas". These documents are available on the DPI web site www.ncpublicschools.org/accountability/reporting (click on ABCs program information), and the appendix.
In order to determine the actual growth in reading and mathematics at each grade level in a particular school, data on matched groups of students (i.e., students with reading and mathematics "pretest" and "posttest" scores) are used. The mean pretest score (use pretest for grade 3, use previous year's EOG for grades 4 through 8) is subtracted from the mean posttest score (current year's EOG) at each grade level to find actual growth. For the North Carolina High School Comprehensive Test in Reading and Mathematics at grade 10, simply subtract the grade 8 EOG mean posttest score from the mean North Carolina High School Comprehensive Test score. For a more detailed description of determining the expected growth goal for reading and mathematics at each grade level computations, see "The Accountability Brief, Setting Annual Growth Standards: The Formula" in the appendix).
For each component, determine the weighted growth by multiplying the weight times the standard expected growth. Sum across all components to obtain the weighted expected growth composite. Example 1 illustrates a school with a grade 3 through 8 configuration. Example 2 illustrates a grade 10 configuration. The weight is determined by dividing the number (n) of scores for each component by the total number (N) across all components.
For the EOC tests, an equation is used to calculate an ABCs goal (or expected score) for each school on each EOC test. Each expected score is based on the achievement of the students when they were in previous grades or courses. Previous achievement (proficiency) is determined by students' performance (scores) on the North Carolina End-of-Grade (EOG) or End-of-Course (EOC) tests, which serve as predictors of the same students' performance in the course in which they are currently enrolled. The EOG or EOC test(s) serving as predictor(s) are different for each EOC test. For details, see the Accountability Brief, The EOC Prediction Formulas located in the appendix and the ABCs Model: Determining Composite Scores at www.ncpublicschools.org/accountability/reporting. (Click on ABCs program information).
Let's look at an example of a calculation to find a school's ABCs goal (expected score) For Algebra I. The equation used to compute the expected score for Algebra I is
Algebra I Expected Score = b0 + (bIMP x IMP),
- where b0 is the state average performance of schools (scale score) for the EOC = 60.4;
- bIMP is the value used to estimate the effect of the school's average math proficiency on the expected average EOC test score = 0.88;
- IMP is the index of mathematics proficiency [equals school's average EOG grade 8 Math scale score for students in Algebra I minus 176.1, (the state's average scale score in Algebra I)].
Substituting the values from the prediction formula parameters for End-of-Course performance, the equation looks like this:
Algebra I Expected Score = 60.4 + [0.88 x (Math - 176.1)]
Step 1: Identify a group of students currently enrolled in Algebra I with predictor scores. In this case, the group of students must have scores on EOG Math from grade 8.
Important considerations in selecting a matched set of students: Some students currently enrolled in Algebra I may not have a grade 8 EOG score in math for a number of reasons. There may be students who have transferred from other states or students may have been absent and failed to complete make-ups. Enrollment can change daily due to the addition of new students, transfers, and withdrawals. This makes it impossible to compute a totally accurate target score until the current enrollment is "captured" on the first day of testing for Algebra I.
Step 2: Find the average EOG Math (grade 8) score for the matched group of students; using this average in the equation, determine the expected Algebra I score. Let's say the average EOG Math score in grade 8 for this group was 178.
Algebra I Expected Score = 60.4 + [0.88 x (178 - 176.1)]
Algebra I Expected Score = 60.4 + 1.672= 62.04
This means that to reach the expected score for expected growth, the school must have an average Algebra I EOC score that equals or exceeds 62.04. Please note that full precision, though not shown here, is used in all calculations.
Over the years, other components have been included in the ABCs Model. In 2000-2001, English II, College University Prep/College Tech Prep, Competency Passing rate, and ABCs dropout rate are included in the model. A brief example of English II and ABCs dropout rate change are shared below. For details about all the components, see the Accountability Brief, The EOC Prediction Formulas located in the appendix and the "ABCs Model: Determining Composite Scores" at www.ncpublicschools.org/accountability/reporting/. (Click on ABCs program information).
Let's look at an example, calculating expected gain index for English II.
Compute the index for English II for the current year and the two previous years. Use the number for the percent of students in each Achievement Level and calculate the index using these steps:
- Multiply percent of students in Achievement Level IV times 3.
- Multiply percent of students in Achievement Level III times 2.
- Multiply percent of students in Achievement Level II times 1.
- Add the products and divide by 3; the quotient is the EOC Index.
- Add the first two years' indexes and divide by 2 to determine the baseline.
- Subtract the baseline from the current (third) year's EOC index.
- Subtract 0.1 to make zero represent gain, as it does in growth computations.
- Divide each difference by the associated standard deviation. The quotient is the standard expected gain for a given subject.
|Level||Current Year||Previous Year||Year One|
Current EOC Index = [(6X3) + (48X2) + (34X1)] / 3 = 49.3
Previous two years' EOC Indexes:
[(4X3) + (49X2) + (39X1)] / 3 = 49.7
[(6X3) + (45X2) + (32X1)] / 3 = 46.7
Baseline = (49.7 + 46.7) / 2 = 48.2
Difference = 49.3 - 48.2 = 1.10
Subtract 0.1 from the difference to ensure that zero represents growth. (1.1- 0.1) = 1.0
Divide by standard deviation = 1 / 7.6 = 0.13
Example. Completed worksheet for English II.
|Column A||Column B||Column C||Column D||Column
(B + C) / 2
(A - D)
(E - 0.1)
(F / G)
The ABCs dropout rate is based on the federal (duplicated) dropout rates, with two modifications: the ABCs dropout rate is based on dropouts in grades 9-12 rather than 7-12; and, the ABCs dropout rates have been adjusted to take into account schools' compliance with the Safe Schools Act and efforts to handle chronic behavior problems. The ABCs dropout rate is expressed as a percentage.
- ABCs dropout numerator = Total Number of dropouts in grades 9-12, minus the total number of expulsions in grades 9-12, minus the total number of long term suspensions in grades 9-12, minus the total number of students incarcerated in an Adult Facility in grades 9-12.
- ABCs dropout membership = 20th day 1999-2000 membership in grades 9-12, minus the initial enrollees in membership day 20 in grades 9-12, plus the 20th day 2000-01 membership in grades 9-12 divided by 2.
- ABCs dropout rate =
100*ABCs dropout numerator
ABCs dropout membership + ABCs dropout numerator
- Subtract ABCs dropout rate percentage in Year One from ABCs dropout rate percentage in Year Two (difference in column E)
- Divide the difference in column E by the standard deviation. This yields the standard expected growth.
- Multiply the standard expected growth by the largest number of dropouts that occurred in Year One or Year Two.
- Divide the product by the total number (N) across all components. This yields the weighted standard expected growth/gain for the ABCs dropout component.
To complete calculations for determining the expected growth composite, sum the standard expected growth for reading and math in grades 3-8 and 10; EOC, College University prep/College Tech Prep, competency passing rate, and ABCs dropout rate. If the sum equals or is greater than zero, the school has made its expected growth.
For exemplary growth, the same EOC prediction formulas are used. However, the state average performance in the formula is multiplied by 1.03. This means that the exemplary growth standard is approximately 3% greater than the expected EOC growth standard. The formula for exemplary growth/gain in Algebra I is:
From the earlier example for Algebra I, computations for determining exemplary growth would follow the two steps below.
Step 1: Multiply b0 x 1.03.
60.4 x1.03 = 62.21
Step 2: Substitute the appropriate values in the formula and complete the calculations.
62.21 + [0.88 x (178 - 176.1)]
62.21 + 1.672 = 63.88
This means that to reach the exemplary growth, the school's average performance on Algebra I EOC tests must equal or exceed 63.88 as opposed to the 62.04 to meet expected growth.
To complete the calculations for exemplary growth, add the standard exemplary growth in reading and math in grades 3-8, and 10; EOC; English II; and expected growth in writing at grades 4 and 7, and expected growth in College University Prep/ College Tech Prep, Competency Passing Rate and ABCs Dropout Rate. The sum is the exemplary growth composite. (See EOC Prediction Formula and ABCs Model: Determining Composite Scores at www.ncpublicschools.org/accountability/reporting/. (Click on ABCs program information).
Most reporting in North Carolina focuses on the school, district (school system) and state level. Student reports are provided by schools to parents from the school-level on their child's achievement level on EOG or EOC state assessments. School accountability reports provide data that include school performance on the components of the ABCs Model, recognition and status categories, SAT results, and subgroup statistics. The results appear in: (1) A Report Card for the ABCs of Public Education Volume I: Growth and Performance of Public Schools in North Carolina; and (2) A Report Card for the ABCs of Public Education Volume II: Subgroup Statistics and Supplemental Data. School system results are reported primarily in the second volume.
School-based accountability results are generated using ABC Tools. This software is customized and designed to produce growth reports for each school in North Carolina as required by the School-Based Management and Accountability Program known as the ABCs of Public Education. The software performs all calculations necessary to produce growth composite scores and performance level analyses. The program creates EOG and EOC summaries, achievement level analyses, and school membership rosters. It also creates the growth report, which shows growth in student achievement in reading and mathematics at each school from year to year. Specific data driven reports include:
- Summary Report - provides data for all students disaggregated by gender and ethnicity,
- Verification Report - used to check data in the student management system and the scan program, and
- Accountability Report- provides information necessary for determining the status of the school under the ABCs accountability model.
Another customized software program used by local school staff to produce goal summary reports for individual students is the Scan and Individual Report software. The Scan program scores EOC and EOG tests, and provides a diagnostic tool (though limited) for teachers to use in assessing their classroom students strengths and weaknesses on goals and objectives of state tests.
A Report Card for the ABCs of Public Education is published each year in accordance with the School-Based Management and Accountability Act (1996) of the North Carolina General Assembly. The growth in performance of students in each school is reported in A Report Card for the ABCs of Public Education Volume I: Growth and Performance of Public Schools in North Carolina. Subgroup statistics and supplemental data are reported in Volume II: A Report Card for the ABCs of Public Education Volume II: Subgroup Statistics and Supplemental Data. This report provides data on student performance on EOG and EOC tests, Scholastic Assessment Test (SAT), participation rates, and other supplemental data. For a copy of the report, A Report Card for the ABCs of Public Education Volume I: Growth and Performance of Public Schools in North Carolina see ( www.ncpublicschools.org/abc_results/results_00).
Adequate Yearly Progress (AYP) for Title I students is also based on the State's ABCs. Two charts in the appendix provide a brief description of how AYP is measured in Title I schools using the ABCs results.
A state report, The State of the State-Educational Performance in North Carolina is produced annually to monitor the state's public school system. The State of the State report serves three main purposes: (1) to assist policymakers in assessing the status and progress of student achievement in public schools of North Carolina; (2) to make comparisons of North Carolina student achievement with that of students throughout the nation on the National Assessment of Educational Progress (NAEP), the Iowa Tests of Basic Skills (ITBS), and the Scholastic Assessment Test (SAT); and (3) to apprise the public of the state's student achievement.
An additional report, Minority Achievement Report: Trends in Subgroup Performance provides trend data on statewide and national performance of various racial/ethnic groups by gender. In previous years this report was a section of the State of the State report. However, this year the report was expanded and will be published in the fall of 2001 as a separate report.
The DPI has examined what has made a difference in the most improved schools in respect to improving student achievement. Staff in the department have discovered that schools review existing student test data to develop instructional and curricular goals. Another positive outcome has been improved instruction at low-performing schools. State Assistance Teams in low-performing schools determined that some teachers in those schools were unfamiliar with the Standard Course of Study and so they worked to focus instruction on curriculum that students need to know to pass state tests because the tests are closely aligned with the Standard Course of Study. Assistance Teams have learned that to sustain the growth of low-performing schools requires strong leadership, focus to the school, stable professional staff, ongoing support and technical assistance, parental involvement and responsible decision-making. Table 1 provides a summary of the number and percent of public schools in North Carolina receiving awards and recognition from 1997 through 2000.
Table 1 Number and Percent of Public Schools in North Carolina Receiving Awards and Recognition 1997-2000
|Schools of Excellence||12||0.7||24||1.4||0||0.0||50||2.5||73||3.5|
|Schools of Distinction||158||9.7||289||16.8||1||0.2||408||20.6||509||24.1|
|Schools Making Exemplary Growth||531||32.5||1137||66.0||265||63.2||1156||58.2||956||45.2|
|Schools Making Expected Growth||395||24.2||308||17.9||83||19.8||456||23.0||520||24.6|
|Schools Receiving No Recognition4||583||35.7||261||15.2||50||11.9||358||18.0||595||28.1|
|Made Expected or Exemplary Growth||926||56.7||1445||83.9||348||83.1||1612||81.2||1476||69.8|
|Total ABCs Schools 5||1632||1722||419||1985||2115|
Note: The 1999-2000 results reflect State Board of Education actions through October 5, 2000.
11996-97 was the first year of
implementation of the ABCs; only K-8 schools were included in the
model for this year.
21997-98 was the first year of implementation of the ABCs in high schools. (Schools whose grades spanned K-12 were included in statistical summaries for both K-8 and high schools, so there is duplication in these counts.)
31998-99 and 1999-2000 were the first two years of the application of the comprehensive ABCs model; there is no duplication in these counts.
4This category was No Recognition in 1996-97, Adequate Performance in 1997-98, and No Recognition in 1998-99 and 1999-2000.
5Total ABCs Schools is the total number of schools participating in the ABCs for a given year; this total does not reflect the sum of the column; Schools of Excellence and Schools of Distinction may overlap with other categories.
Caution: Comparisons across years should be made with the above footnotes in mind.
Please note that a school principal may appeal their individual growth standard in writing to their local superintendent who in turn can make a formal appeal to the State Superintendent. The appeal must be received by the State Superintendent within 30 days of the school receiving its rating. The Compliance Commission (mentioned later) considers the appeal from the local superintendent and makes a recommendation to the SBE within 45 days of the date of the appeal.
Every year since the implementation of the ABCs there have been a few schools where a small number of students in a subject or grade have had an unduly positive or negative influence on the ABCs growth of a school. Weighting of the ABCs growth composites was adopted by the SBE in 2001 to deal with such disproportional impacts.
Due to the way writing is included in the ABCs and with the impact of implementation of IDEA, the SBE removed results from the writing tests for grades 4 and 7 in the ABCs growth composite. Writing test results will continue to be included in the performance composite, but will not be a part of the growth composite until three years of comparable data are available.
Several accountability models were reviewed and studied prior to selecting the growth model. On a yearly basis, the present ABCs Model is examined to make sure critical constants remain stable over time. Simulations are conducted frequently as components are added/removed from the model.
There are many challenges in the area of reporting that largely pertain to time demands for collecting and analyzing data, and generating reports for the public. Additional challenges concern the effective communication of what the assessment and accountability results mean. For example, it is difficult to communicate to the public some of the more technical procedures, including how the expected growth is calculated.
As new assessments are developed to match updated curricula, the accountability staff are faced with maintaining the growth model as the data are collected from different tests with different scales. These transitions require extra time (e.g., to complete equating studies or redevelop statistical models and the resulting outcome is the delay in the release of the ABCs results as is the case this year).
Another challenge deals with teacher perception about the ABCs accountability model. Some teachers have expressed concerns about "teaching to the test," narrowing the curriculum, administering field-tests, and spending too much time practicing for the tests. DPI staff are sensitive to these concerns and are attempting to address them (e.g., by encouraging a focus on the Standard Course of Study). The ABCs accountability model has been successfully implemented for 5 years. Regardless, the SBE has continued to refine it and the testing program. There are additional changes that go into effect in the 2001- 2002 school year. Those changes include:
- revising the ABCs categories by adding School of Progress for schools that make expected growth or better and have between 60% and 79% of students at Achievement Level III or above;
- using the term growth in place of growth/gain in all designations of meeting or exceeding growth and gain standards;
- requiring Schools of Distinction to meet expected growth or better (an accomplishment that 90% of the current Schools of Distinction already meet);
- designating schools that have between 50% and 59% of students at Achievement Level III or better as Priority Schools regardless of whether they make expected growth;
- designating schools with less than 50% of students at or above Achievement Level III that are making expected growth or better also as Priority schools; and
- eliminating three tests: ITBS at grades 5 and 8, Open-Ended in grades 4 and 8, and the High School Comprehensive Tests in Reading and Mathematics at grade 10.
As the North Carolina curriculum is modified and updated, new tests will be developed to match the curriculum; and the new tests will be equated with the old tests. This will require changes in developmental scale scores. Most recently, the math test has been updated and the new items were piloted in 1999-2000, and implemented this year. The high school exit exam is being piloted and will affect the class of 2005.
The Compliance Commission for Accountability for Public Schools of North Carolina was created in 1996 to advise the SBE on testing and other issues related to school accountability and improvement. This is a diverse group with approximately 22 members from across the state that provide critical advice to the SBE on accountability issues.
Another good point is maintaining quality control at three levels: (1) testing coordinators at school districts trained by (2) Regional Accountability Coordinators (RACs) located in six regions of the state who are trained by (3) DPI accountability staff members. Monitoring takes place often, with checkpoints at the RAC level and at DPI.
North Carolina's ABCs of Public Instruction has made a positive difference in performance of students and professional educators across the state. By using this accountability model, we have found that successful schools focus on the Standard Course of Study, use data-driven decision making, align local curriculum and instruction with testing, and in turn have increased student achievement. For additional information about North Carolina's ABCs of Public Education, please contact Lou Fabrizio, Director of the Division of Accountability Services, DPI, 301 N. Wilmington Street, Raleigh, NC 27601-2825; or email address email@example.com; or telephone at 919.807.3770.