END-OF-YEAR TESTS ARE THE RESULT OF A CAREFUL DEVELOPMENT PROCESS
When students sit down with No. 2 pencils to take the required North Carolina end-of-grade or end-of-course tests, they spend several hours completing the tests. In contrast, developing the tests takes about three years of work by testing and curriculum experts and local teachers.
"As tests become more high-stakes, parents and teachers naturally question how tests are developed and how we can be confident that North Carolina tests measure the curriculum that is taught," said Lou Fabrizio. Director of Accountability for the NC Department of Public Instruction. "In North Carolina, our testing program includes 14 development steps - 11 steps before students even take the tests - and we heavily involve classroom teachers in writing and evaluating test questions. This careful process helps us ensure that tests accurately reflect the curriculum. "
Practicing classroom teachers are involved in half of the steps that must be taken to develop a multiple choice test in North Carolina.
By their very nature and because they are used to evaluate schools and students, tests are required to be secure. That security sometimes makes tests seem mysterious. To eliminate some of the mystery, here-s the blueprint from start to finish for North Carolina end-of-year tests.
Setting Specifications - Every five years, each subject in the state's Standard Course of Study is reviewed and revised. If that subject area is one that is tested, then work begins immediately to revise the test. A team of curriculum specialists, teachers, testing experts and other educators set specifications for each grade level and content areas that are assessed. The specifications tell test developers:
- what percentage of test questions will measure specific curriculum goals;
- the types of passages and other material to include; and
- the difficulty and thinking skill level of the questions.
Test developers consider an easy question to be one that about 70 percent of students would answer correctly. Hard test questions are ones that only 20-30 percent of students would answer correctly. In the area of thinking skills, the test specifications describe the skill students must use to answer a specific question. For example, some questions may measure students' abilities to organize or to evaluate information.
Developing and Reviewing Test Items - In most instances, North Carolina's test questions are written by teachers who are recruited and given specialized training for this task. This ensures that the questions are reasonable to practicing teachers and are appropriate to the curriculum in our state. A separate group of teachers evaluate written test items. During the process, these teachers look at each written test item to make sure it matches the course objective it is designed to measure, that the language is appropriate and correct, and that the question is relevant and unbiased, among other things. A multi-step process continues until a specified number of test items are written, edited, evaluated, edited once more and completed.
Typically, during the test development process, about half of the potential test items are discarded for one reason or another. On occasion, the state has contracted with test publishing companies to assist in the development process of writing test items.
Field-testing - Test questions for each subject or grade are assembled into test forms for field-testing, usually to a random and representative sample of students from across North Carolina. Additional teacher reviews occur at this point to ensure clarity, correctness, potential bias and appropriateness of each question. Some surveys may be given as part of the field-testing to help gather information about the students taking the field tests. Field testing the test questions provides information about how well each question measures the specific content. In addition, it provides information about how well students understand questions and can reveal test items that may be misunderstood by students or have other problems. This additional information can help provide a more complete analysis of the test results.
Developing Test Forms - After field testing is completed, some test questions are dropped because of potential problems in the questions. This leaves a final pool of test questions that have survived all the reviews, field testing and other quality checks. At this point, test forms are built from this pool of questions. DPI builds between three and six forms of each new test. This helps ensure security. Test forms are built so that they have an appropriate number of test questions and a proper balance of questions to cover the curriculum. Two more rounds of test reviews are done at this point: a group of educators evaluates the assembled tests. Content specialists and editors also completely evaluate the tests.
Administration and Scoring - Tests are administered statewide, usually near the end of the school year or the end of the semester, depending on the schools' scheduling structure. All students are required to participate, and students who have disabilities or limited English proficiency may be given certain accommodations, as state policy allows. Multiple choice tests are scored locally using electronic scanning equipment, and the Department of Public Instruction provides software and other tools so that scoring and analysis can be done at the local school district level. This allows the tests to be administered near the end of the semester of the school year, but still have results back to students within a few days. Performance tests, such as the writing tests, are scored by contractors since these take several months to score.
Setting Standards - Standards for each test are set by comparing teacher judgment of student achievement within the classroom as compared to actual student performance on the tests. Analysis of these data is used to set performance standards, such as achievement level ranges (cut scores).
For more information about how North Carolina's tests are developed and how they are used, please contact Lou Fabrizio, Director of Accountability Services, or Mildred Bazemore, Section Chief for Testing, at 919.807.3769.
About the North Carolina Department of Public Instruction:
The North Carolina Department of Public Instruction provides leadership to 115 local public school districts and 126 charter schools serving over 1.5 million students in kindergarten through high school graduation. The agency is responsible for all aspects of the state's public school system and works under the direction of the North Carolina State Board of Education.
For more information:
NCDPI Communication and Information Division, 919.807.3450.