*For help with questions not addressed below, please call the Measurement Incorporated Grade 4 ELA Helpline (877) 516-2403. The line is available weekdays February 5 - 15 from 8:00 a.m. - 5:00 p.m.

INTRODUCTION - After the videotaping of the training sessions, the Scoring Leaders participated in Question and Answer sessions. Staff from Measurement, Inc. and the State Education Department provided responses to participants' questions. Many of the questions refer to particular student answer papers in the Practice Sets, and responses may refer to the Scoring Guide. The following transcript places questions of a general nature first. These are followed by questions organized by content area: Listening; Reading; and Writing and Writing Mechanics.

Q: How do I use the Videotapes?

A: Videotapes have been provided for each component of the Grade 4 English Language Arts test to assist in training scoring leaders/scorers. The trainer, in each videotape, will discuss the contents of the Scoring Guide and the Practice Set for that content area. The Scoring Guide will be presented first to demonstrate how the scoring rubric should be applied to student responses. We suggest that Training Leaders stop or pause the videotape before the videotaped trainer begins discussion of the Practice Set. This provides an opportunity for those being trained to read their Practice Sets and practice making scoring decisions.

We also suggest that scorers practice on only one or two student responses at a time, stopping and reviewing the correct score(s) before moving on to the next. The Scoring Leader may read and discuss the annotations and marginalia in their copies of the Practice Sets, or may resume the videotape at appropriate intervals. Several short practice segments followed by review maximizes the opportunity to learn by doing and assists in building scorer skill and confidence.

Q: As a Scoring Leader, how should I prepare to train Table Facilitators and Scorers?

A: Training procedures and the logistics of live scoring are covered in the Scoring Leader Handbook, which should be read thoroughly before training. You should also review your Scoring Guide and Practice Set while viewing the videotape.

Q: How should a response be scored if it’s entirely blank, or it says the student refuses to answer, or it’s written in another language?

A: A list of Condition Codes can be found near the back of the Scoring Guide, and the Scoring Leader Handbook contains the procedures for assigning such codes. Responses written in another language should always be scored as an "E" even if the Scoring Leader or another scorer understands the other language, since the test is intended to assess English language skills.

Q: What if a student writes a continuation of a response on a planning page of the test book?

A. If the planning page is in the same part of the test that is being administered, and if the student clearly labels the continuation of the response, then the continuation of a response that is written on a planning page should be scored.

Q: Sometimes a student will respond to some but not all items. What Overall score should such responses receive?

A: Near the back of the Scoring Guide is a list of Scoring Considerations. These outline the effect of missing responses on the Overall score.

Q: Suppose a student leaves the short responses blank and answers only the extended response, but in the extended response clearly demonstrates understanding of all of the questions posed in the other items? Since the Overall score is supposed to holistically reflect the understanding of the student, can such a response receive a "4"?

A: No. If only the extended response is answered, the Scoring Considerations limit the Overall score to a "2."

Q: When training is over, should scorers refer to the training materials while scoring actual student responses?

A: YES! To maintain accuracy and consistency in scoring, it is very helpful to refer occasionally to the student responses used in the training materials as examples of the various score points. These responses are often called "anchor papers" because they help to fix the acceptable range within a score point and prevent the scorer from "drifting" higher or lower in their expectations for awarding a score point.

Q: I understand that holistic scoring involves weighing and balancing various factors. What are these factors, and what weight should be given to each?

A: The scoring rubric addresses the factors that should be considered in determining the score of a response by listing characteristics that tend to occur among the score points. These characteristics reflect the degree to which focus, development, organization, and writing style are found within a response. Focus is how well the response fulfills the requirements of the task and the connections to the task found within the response. Development is how much information is presented: the details, the specifics, and the amount of elaboration on ideas. Organization is the order in which the information is presented. Does one idea logically follow another? If the response is in the form of a narrative, how tight is the sequence of events? Writing style generally concerns word choice and sentence patterns. How fluent is the response? Is it easy to read? Writing style should not be confused with writing mechanics. Style concerns what word is used, whereas mechanics concerns how the word is spelled. Style looks at how the sentence patterns create a flow of ideas, while mechanics looks at how the sentences are punctuated. Remember that writing mechanics is scored separately and should not be a factor in scoring Independent Writing. In assigning a score to an Independent Writing response, all relevant factors should be assessed. However, the most important factor by far, and the one accorded the most weight, is DEVELOPMENT. The amount of development is central to each score point. How much information are we being given? What are the details and the specifics? Are ideas or events elaborated and expanded upon? Development is not only important in and of itself; it also impacts the other factors. There must be a certain amount of information presented for a scorer to be able to assess a response’s focus, organization, and fluency. Caution: development is not synonymous with length! Obviously, the process of presenting the amount of information necessary to get to a higher score point will result in longer responses.

Q: Our scorers are experienced teachers who adhere to certain standards in their classrooms. Some scorers may find it difficult to follow the standards set by the rubric and the training materials if those standards seem higher or lower than those used by the scorer in the classroom. How should I advise a scorer who hesitates to apply the standards appropriate for this test?

A: We value the classroom experience of our scorers, and we realize that some variation of expectations may exist between districts, schools, and individuals. However, it is very important that all scorers separate their classroom expectations from the standards used in scoring this statewide test. Every scorer should use the same standards in applying the rubric to student responses. Uniform standards in scoring are crucial to obtaining the consistency and accuracy necessary for a valid assessment of student performances across the entire state. Accurate assessments ultimately benefit everyone.

Q: How can a scorer avoid "drifting" from the correct standards while scoring?

A: After scoring a number of responses, a scorer may gradually, even unconsciously, begin to accept more or less than is appropriate in awarding a particular score point. This could result in scoring inequity, where a student response could receive a different score from the same person depending on when it was scored. To maintain the consistency and accuracy of all scores, it is important to prevent any "drift" in scoring standards. This is best accomplished by frequent reference to the "anchor papers" in the training materials, and by encouraging scorers to consult their Table Facilitators or Scoring Leaders with responses that seem on the line between two score points.

Q: What if I should encounter a response where the student indicates that he or she is in a crisis situation and needs intervention? How should such sensitive responses be handled?

A: Scorers should be instructed to bring such responses to the immediate attention of the Scoring Leader. Scoring Leaders need to notify the school principal of any sensitive responses. If tests are being scored regionally, Scoring Leaders should alert the Site Coordinator who will contact the student’s principal.

Q: Some students with disabilities have extremely large handwriting and won’t be able to fit their answers to open ended questions in the empty spaces in Book 2. Are they permitted to use extra paper?

A: If the IEP/504 plan makes this test accommodation, students may use additional paper or write their answers on a word processor and attach the printed response. Notation must be made on the front of the booklet that the student had this accommodation (refer to page 3 in the School Administrator's Manual for information about Testing Accommodations).

Q: What if a student puts the correct information for a response on a different page, such as the planning page, instead of on the correct response page?

A: If the response page is blank, it must be scored to reflect that it is blank. However, if a student indicates graphically on the correct response page that a response is written or continued onto the planning page, then the scorer can follow the student’s instructions and consider the information on the indicated page.

Q: The rubric says a "4" will have "vivid language and a sense of engagement or voice." Where in each of the "4"s in the training materials can I find examples of vivid language and voice?

A: Not all "4"s will have vivid language or a sense of engagement. However, the precision of language and the manner of expression can be factors in strengthening a response if all of the other elements are present. Voice, where the personality of the student shows itself in the manner of expression, is like a cherry on the sundae. The sundae must be there first before the cherry can be seen as adding anything substantial. Keep in mind also that what is vivid language or voice for an elementary school student may be different from what you or I may consider to be vivid.

For Writing, both "3"s in the Guide have voice. In the Listening Guide, the Extended Response for 11c and 11d is thorough and fluent and contributes to the overall score of "4". However, it does not have strong voice. In the Reading Guide, the Extended Response for 12d and 12e demonstrates voice at the end when the writer explains the drawbacks of having the raven be the state bird.

Q: On borderline calls, when deciding between adjacent score points, should the scorer always give the "benefit of the doubt" to the student and award the higher score?

A: No. Such a practice can result in scoring "drift." After scoring a number of responses, a scorer may gradually, even unconsciously, begin to accept less (or demand more) than is appropriate in awarding a particular score point. Scoring "drift" can create an unfair situation where a student response could receive a different score from the same scorer depending on when the response was scored. To prevent "drift" and maintain the consistency and accuracy of all scores, it is helpful to refer occasionally to the student responses used in the training materials as examples of the various score points. These responses are often called "anchor papers" because they help to fix the acceptable range within a score point and prevent the scorer from "drifting" higher or lower in their expectations for awarding a score point. Scorers should also be encouraged to consult their Table Facilitators and Scoring Leaders with responses that seem on the line between two score points.


Q: Do all 4’s have to have Waldo’s luck changing twice (good to bad then back to good) ?

A: Yes, all 4’s we’ve seen in the Guide and Practice Set have Waldo’s luck change more than once. Sometimes this is implied, but there is specific text support to indicate the student understood this. To be " complete and thorough" as defined by the rubric a response must include both changes.

Q: How important is sequencing when evaluating the Extended Response?

A: Sequencing "correctly" is important, and is reflected in the score point definition (a "4" would have no sequencing problems). This year it is part of the question, since the student is asked to describe how Waldo’s luck changes during the course of the day.

Q: Is the extended response being given more emphasis this year than in years past?

A: No, keep in mind that every year the generic rubric remains the same. However there is always more time spent, in training, on the Extended Response because it is where most students do the majority of their writing. There is a lot to consider when we evaluate how well the student accomplished the tasks involved.

Q: Can we go over Practice Set # 3 again?

A: This is a good example of how a student’s Graphic Organizer (#29) and Short Response (#30) can help the student achieve an overall score of "2" even though the Extended Response (#31) adds nothing. There is enough accurate work done in the first two tasks to demonstrate "a partial understanding", which is the hallmark of a score point 2.

Q: Why did Practice Set # 4 not receive a score point 2?

A: Even though we have a correct Graphic Organizer (#29), the other two responses contain very minimal information. As scorers, we cannot make connections for the student. This student has not demonstrated that he/she understands how the information in the Graphic Organizer applies to the Short Response (#30) and the Extended Response (#31).


Q: In the Graphic Organizer, does the information need to be in the box or can it be somewhere else on the page?

A: As long as it is appropriate and correct, and somewhere on that page it should be considered and counted.

Q: Can we review the differences between Practice Set # 2 and #3?

A: The difference is in the Extended Response. In Practice Set #2 the student only lists, without text support. Also "smart" and "intelligent" are really the same quality. Practice Set #3 has much more development in the Extended Response. The student elaborates on "smart" and uses more specific text support. Holistically, this response achieves adequate development and demonstrates understanding.

Q: What if a student argues both sides, pro and con? For example, a student states this would be a good idea but "people wouldn’t like it".

A: This is fine, as long as the student supports with text and fulfills the requirements of the task (i.e. follows the bullets). Look at Practice Set #10, which did not follow through with text support. In the Extended Response the student does this. In the last paragraph the student changes opinion. In this case it adds nothing to the response because no new information is added. However, neither does it adversely affect the score.

Q: In the Graphic Organizer of Practice Set Paper # 7, the student uses a specific raven. Is this okay?

A: Yes, in the text, this was a specific example that demonstrated the Raven’s cleverness.

Q: For the Extended Response, do we penalize if a student does not put his/her response in the form of a letter?

A: No, it does not have to be a letter. Nothing in either rubric mentions this as an element to be considered when scoring. Also, we have never held the student to a specific format in past years.

Q: Can we review why Practice Set Paper # 10 did not get an overall score of 4?

A: The Extended Response does not add enough for the response to achieve a "4". We are not scoring down, but rather recognizing that the response falls short. It does not get there. It does not achieve the level of thoroughness required of a 4.

Q: Why is 8c in the Practice Set not a "2" in analytics?

A: It’s close, but not there. There is no connection between the text and the lesson. The scorer could perhaps infer, but that is doing the student’s work. If the student had somehow tied "ugly" to not being nice, the response would have reached the 2 score point.


Q: Is it a defect (flaw) in Guide Paper #10 that it’s not clear until late in the narrative who the friend is?

A: Holistically, the strong qualities demonstrated in this response outweigh what might be viewed as this flaw.

Q: Should we train scorers onWriting, score for Writing, and then train on Writing Mechanics etc?

A: Most experienced trainers have found it to be more efficient to train for both Writing and Writing Mechanics before going on to score. It is less time consuming and the shift from training to scoring occurs only once.


Q: Because we are scoring three separate responses as a unit, what guidelines do we have to evaluate when the three responses are of markedly different lengths?

A: Look at Practice Set #5. You can see these can be eligible for any score point.

Q: What about responses that are exempt?

A: These follow modifications as noted. If not, score what you see.