They represent a continuum in how much freedom of response is allowed, ranging from restricted-response essays on one end to extended-response essays on the other.

1. Restricted-response essay
1. limits content and response to be given
2. can limit via how narrowly question is phrased (e.g., as specific as a short-answer question)
3. can limit via scope of the problem posed (e.g., with introduction like that of an interpretive exercise)
4. therefore, can approach the objectivity of short-answer and interpretive exercises
2. Extended-response essay
1. great freedom so that allows problem formulation, organization, originality
2. therefore, shares similar scoring difficulties with performance-based tasks

Represent a continuum in complexity and breadth of learning outcomes assessed, with interpretive exercises on the left end, restricted-response essays in the middle, and extended-response essays at the right end.

1. Restricted-response essays
1. For learning outcomes not readily assessed objectively
2. Compared to extended-response questions, they target narrower learning outcomes, such as more specific mental processes (e.g., draws valid conclusions)
2. Extended-response essays
1. For learning outcomes not readily assessed objectively or with restricted response essays
2. Compared to restricted-response questions, they assess broader learning outcomes, such as integrating a set of mental processes (e.g., integrates evidence to evaluate a scientific theory)
3. Compared to interpretive exercises, both kinds of essays can assess more complex learning outcomes
3. See Table 10.1 on page 240

1. Measure complex learning outcomes not measured by other means
1. Restricted-response essays: (i) require students to supply, not just identify, the answer and (ii) can target specific mental skills
2. Extended-response essays: emphasize integration and application of high-level skills
2. Can measure writing skills in addition to (or instead of) knowledge and understanding
3. Easy to construct—but only if you don’t care what you actually measure and how reliably you do so!
4. Contribute to student learning, directly and indirectly

1. Unreliability of scoring (unless clear learning outcomes, good scoring rubrics, practice in scoring)
2. Time-consuming to score—especially if follow guidelines. Can be impossible if conscientious in scoring, give good feedback, and have many students
3. Limited sampling of content domain

• To call forth the intended student responses

1. Restrict use to learning outcomes that cannot be measured well by objective means (e.g., organization, originality)
2. Write questions that can call forth the intended mental processes
1. Easiest to do with restricted-response
2. See sample stems on pp. 243-244
3. For extended-response items, helps to state evaluation criteria in the question
4. Make sure they do not target what has not been taught
3. Phrase the question so that student’s task is clear and comparable for all
1. Easiest with restricted response
2. For extended-response, don’t define the task so tightly that its purpose is spoiled
3. Rather, give explicit instructions on type of answer desired (e.g., "Your answer should be confined to 100-150 words. It will be evaluated in terms of the appropriateness of the facts and examples presented and the skill with which it is written.")
4. Indicate approximate time limit for each question
1. Give plenty of time (should be a power test not a speed test)
2. Do not create overconcern about time
5. Avoid optional questions
1. Giving choices means students taking different tests
2. They will not study the entire domain
6. Review checklist on p. 248

• There is no single correct or best answer to an essay question, so you need guidelines—"rubrics"--for rating the quality of answers.

• Rubric = a set of guidelines for the application of performance criteria to the responses and performances of students

1. Carefully specify your scoring criteria before you finalize the exam
   1. May cause you to rethink or modify the question and its accompanying performance criteria
   2. That, in turn, enhances likelihood of calling forth the intended responses
2. Do an initial review of answers to a question to find exemplars or anchors for your scoring levels
3. Make sure you can describe the kinds of performance (e.g., "lists two of the four key points") that qualify for each scoring level ("satisfactory," 2 points, etc.)

• Enhances reliability, validity, and fairness of scoring
• Saves you a lot of time and headaches when you score the exam

• Rubric=guidelines for the application of performance criteria to the responses and performances of students
• Types
  • Analytical
  • Holistic

1. Write exemplar answer(s)
2. Decide how to give points for each part expected for a full answer
3. Decide the level of explanation necessary for full vs. partial credit

1. Specify the separate characteristics or dimensions you want to score (focus, elaboration, mechanics, etc. for an expository essay)
2. Assign a series of levels to each characteristic (1-7, poor to excellent, etc.).
3. Summarize the performance corresponding to each level ("main idea present but may not maintain consistent focus" for "adequate achievement" or 4 points for "focus"—example from table on p. 251)
4. Result is a matrix against which to judge the elements of each essay
5. Good for giving feedback to students
6. See websites for examples (e.g., www.nwrel.org/eval/toolkit/traits/index.html )

1. Provides single overall score, no separate dimensions
2. Decide how many levels.
3. Summarize the performance corresponding to each level
4. Easier to construct and apply than analytical rubrics
5. May correspond better to grading needs
6. But provides less feedback to students about strengths and weaknesses

Suggestions

1. Ease your burden by writing good questions in the first place (OH 4)
2. Prepare outlines of expected answers in advance
• Can redesign poor questions
• Provides common basis for judging all students
• Standards less likely to shift during grading
3. Use most appropriate scoring rubric, which depends on purpose
• Analytical, if focus is on multiple dimensions of performance and giving feedback
• Holistic, if focus is on overall understanding rather than writing skill
4. Decide in advance how to handle factors irrelevant to learning outcomes
• Irrelevant skills (legibility, spelling, etc.)
• Irrelevant or inaccurate factual information (risky to ignore, so consider warning in advance that will penalize)
5. Evaluate all responses to one question at a time. Reorder the papers before grading the next question.
• Maintains more uniform standards for a question
• Distributes effects of following a good or bad paper
• Minimizes halo effects
6. When possible, evaluate without seeing students’ names
7. Watch out for bluffing
• Good questions and scoring rubrics can reduce its impact
• See bluffing strategies on p. 256
8. If important decisions rest on the results, use several raters
• Reconcile any big discrepancies
• Average the scores