EMSC Banner for Content Pages. The banner reads, "New York State Education Department Back to EMSC Home
SED Home
Disclaimers and Notices

Large District Scoring Case Study
Grade 5  Elementary-Level
Social Studies Test

Demographic Profile

The district is responsible for scoring approximately 3,000 Grade 5 - Elementary-Level Social Studies Test papers. Accommodations will be made to score assessments that are in multiple languages, large print, or Braille, or that have had student testing modifications. To complete this task on a timely basis, 60–75 raters are needed.

The goals of our scoring-week activities are to:

  • Correct 3,000 tests as consistently and as accurately as possible.

  • Collect and return information to the schools as soon as possible.

  • Collect immediate feedback from scorers to determine what worked well, and to identify areas that need to be refined.

  • Provide professional development to teachers and specialists that will promote social studies achievement in K–6 classrooms.

 
   
 

Logistics

The raters include English Language Arts specialists; AIS specialists; and 3rd-, 4th-, and  5th- grade teachers; and special education teachers. Building principals identify and select the raters. Raters are released from their normal duties for the week of scoring. Since scoring takes place when school is in session, an off site location is needed for scoring. Six secured scoring rooms, and a logistics room, are needed to score. The grade 5 social studies test is scored during the first week of December. It takes four days to score the assessment; the fifth day is used for professional development based on the feedback from the week’s scoring session.

Tests are bundled together with a cover sheet for each student. The assessments and scoring booklets are brought to the scoring site on the Friday before scoring by each building’s test coordinator. Thirty-nine elementary buildings and programs deliver their tests bundled in boxes to the scoring site. Once all the tests are delivered, they are randomly divided into 24 boxes for distribution to the scoring rooms. After the scoring is completed, the test papers are reorganized by school and taken back to the building for secured storage.

The agenda for the week is:

Monday morning: CRQ training and scoring  
Monday afternoon: DBQ scaffolding training and scoring  
Tuesday morning: DBQ scaffolding scoring  
Tuesday afternoon: DBQ essay training and scoring  
Wednesday morning: DBQ essay scoring  
Thursday morning: DBQ essay scoring  
Thursday afternoon: DBQ essay scoring  
Friday morning: Before, during, and after the assessment
professional development and discussions
 
Friday afternoon: Planning for in-building professional development  

Preparation

Several weeks before scoring, 12 raters are selected as table leaders. These table leaders receive extensive training on the scoring booklet, the scoring process, and the role of the third reader. Each scoring day starts with a large group debriefing, discussions, and review of the scoring rubrics. This process ensures rater consistency and accuracy. All raters are given the appropriate scoring booklets when training takes place.

Breakfast and lunch are provided each day for the raters. Individual raters take breaks as needed.

All scoring rooms are equipped with chart paper, markers, highlighters, pencils, candy, and bottled water. Sheets of chart paper  with the titles "Positive Parking Lot" and "Area Needing Improvement Parking Lot" are attached to the wall. As scoring takes place, scorers write on the chart paper the patterns of improvement or the problem areas that they are encountering. Each rater is trained to scores all parts of the test.

As soon as each section is completed, raters fill in the appropriate section on the rater scanning sheets. Table leaders spot-check accuracy of the forms throughout scoring. Sheets are then complied by staff from the Research, Testing, and Evaluation Department. This department is charged with running the final scanning sheets; this process includes scorers from all parts of the assessment. Scorers will also conduct the initial data analysis from the results. Class lists are verified for absentees and for missing papers.

 
   
 

Procedure

The specialists and teachers are trained to score sections II (CRQ), IIIA (DBQ scaffolding), and III (essay). Raters are trained to score the CRQ section first. Each section follows the same training format. The NYSED scoring booklets are used in conjunction with actual test papers. Local papers are used in training and discussion sessions for all parts. Each day starts with a large group debriefing, discussions, and review of the scoring rubrics. Acceptable answers and discussion items are posted for review. Throughout the scoring process, teachers are reminded that "one bullet does not hold an entire essay hostage."

The boxes of test papers are divided evenly among scoring rooms. As a box is completed, table leaders initial the box checklist and the box is rotated into another room. No room should score multiple parts of the same box. All essays are scored once and placed in the completed box which is then moved to the next room for the second read. When the second read is completed, and it is determined that a third read is necessary, the table leader becomes the third reader. When the box is completely scored, it is moved to a secured logistics room.

At this point, the Research, Testing, and Evaluation Department checks the scanning sheets for mistakes and missing information. The scanning form is also separated from the rest of the answer sheets. After all sheets are separated, they are run through a scanning machine and tabulated.

Post-scoring Activities

Before the end of scoring week, raters are asked to reflect on what they have learned from the scoring session. They are also asked to participate in a discussion of activities that their school does before, during, and after the assessment is given. The purpose of this practice is to allow staff members from all 39 schools to hear the best practices and different approaches to improving scores. All parts of the assessment, except the scan form, are sent back to the schools for immediate use in professional development. Then they are stored in a secured location by the schools. Raters evaluate the scoring session and fill out the NYSED exam survey.

After the scores are processed, the data is reported to the superintendent, his or her instructional cabinet, and the board of education. All principals, vice-principals, instructional specialists, and teachers then receive the information. After internal notification is complete, a press release is issued.

The data is sorted according to:

  • Overall performance

  • Ethnic breakdown

  • Students-with-disabilities gap analysis

  • Comparison of results from previous years

  • Comparison to other districts in the county and state

  • Schools with common redesign models

  • Correlation to 4th-grade ELA performance

  • Data from each part of the test is analyzed and reported to building administrators and teachers. Plans are formulated to change classroom practices on the basis of the results. AIS plans are also created for students who score a "1" or a "2". These plans are placed in the student records folder and also on the electronic student information system.

     
       

    Back to New York State Social Studies Assessment