Activity A09—Peer Evaluation
(done in "lettered groups" over three class periods)

The goal:

Design a peer evaluation form that is "optimized" for the purpose of

We will come to a consensus on a form we can all live with, and a process for using that form, including issues of anonymity or openness, and how the form affects final course grades.

Resources at your disposal

Day 1:

Step 1: Thinking about group work
(individual, 10min, 1st day, 10:15am-10:25am)

When working in groups, there are a number of positive "traits", or "behaviors" that you might hope for—things that lead to group success.

We can consider a group successful if both the product and process are good

There are also a number of negative "traits" or "behaviors" that you would hope not to experience:

Certain traits or behaviors can be ascribed to the group, while others can be ascribed to individuals. For example:

Your job:

Think about your experience with group work—both in this course so far, as well as in previous courses, or other life experiences. Think about when groups worked well, and when they didn't work well.

By write down, I mean either

You can use positive or negative language (your choice). If you list more than the number asked for, rank the top three in each category that are most important to you.

If you don't have enough time, focus on the individual traits—those are the only ones we'll be focussing on for the rest of today's activity. The group traits, we'll come back to later. Whatever problems you've experienced in your groups this semester or in the past, make sure that your list of traits addresses those problems.

Grading Rubric: (A09.S1 out of 20 points)

Step 2: Discuss sample Peer Evaluation forms
(group, 15min, 1st day, 10:25am-10:40am)

In this step, you will review and discuss several examples of peer evaluation forms.

In your group, do the following

  1. Introduce yourselves to one another.
  2. Choose each of following (four different people)
  3. Scribe opens an email to pconrad@udel.edu and oanat@udel.edu with subject line "CISC474 A09.S2, Group X", where X is your group letter. CC the email to all group members (you can cut and paste your group members' email addresses from the group list page.) Start email by identifying name of person in each of the roles from item (2).
  4. Discussion leader chooses one of the sample peer evaluation forms to start with.
  5. Discussion leader continues through the other candidate forms following a similar procedure—except this time, for each question on the other forms, discussion leader first asks:
  6. At the end, discussion leader, go around the table and ask if there are any important traits that aren't covered on any of the candidate forms.
    1. If so, scribe record the trait
    2. Group discuss whether it is important to include, and try to come to consensus.
    3. If consensus cannot be reached, scribe should note this and group should move on.
    4. If group agrees that trait is important, design a question that would measure it—scribe records that question.
  7. When all candidate forms have been reviewed, scribe "hit send" on the email.

Grading Rubric: (A09.S2, group, out of 20 points)

Step 3: Design a consensus form
(group, 10min, 1st day 10:40am-10:50am)

Now, design a form that reflects the consensus thinking of the group about what should be in a peer evaluation form.

Don't worry about formatting: just get the content right. Decide what the questions should be, and what form the ranking should take (i.e. number on scale, "excellent/good/fair/poor", or what.).

Borrow heavily from the example forms, or do your own form, as you see fit. But get the job done within the time frame provided if at all possible.

Post your result on the web, and post a link to your groups form on the discussion board marked "A09.S3 links" on WebCT.

(If group cannot finish the form in class, it can be completed outside of class and submitted anytime before the start of class on Wednesday. It must be done by Wednesday so that we can start to narrow the choices down between Wednesday and Friday.)

Grading Rubric: (A09.S3, group, out of 20 points)

Step 4: Report back
(whole class, 10min, 1st day 10:50am-11:00am)

Instructor will call on group spokesperson to reflect on the process in the group to the whole class —what themes emerged, how things went.

 

Homework between Monday 04/03/06 and Friday 04/07/06

Step 5: Discussion of anonymity and feedback
(on WebCT between Monday and Friday)

All group members log on to WebCT and find individual discussion boards for each group.
On your group's discussion board, discuss the following questions.

For full credit, do two things:

  1. Post your thoughts about each of these questions, and
  2. follow up on the responses of other group members in a thoughtful, respectful fashion.

Questions for discussion

  1. Should evaluations be anonymous? And what feedback should the student being evaluated receive?

    Some options:
    1. student being evaluated receives only "numerical averages" for each question, not individual scores
    2. student being evaluated receives average and individual scores, but evaluators names are anonymous
    3. student receives average and individual scores plus comments, but evaluators names are anonymous
    4. student receives average and individual scores plus comments, and evaluators name are revealed

    What are the pros and cons of the various approaches?
    Are their other approaches to the issue of anonymity?
    What approach is most likely to lead to good group outcomes? Why?
  2. Should results of peer evaluations affect the course grade of students?
  3. If peer evaluations do affect a student grade, how should they?
    1. Direct computation (i.e. numeric evaluations are directly mapped to some percentage of students final grade)
    2. Triangulation—results from a single group experience will neither raise or lower a student's grade unless similar results are reported by a separate, disjoint group. That is:
      • if you get good peer ratings from two non-overlapping peer groups, your grade is raised
      • if you get bad peer raitings from two non-overlapping peer groups, your grade is lowered
      • if you get "mixed reviews", the peer review neither raises nor lowers your grade.
    3. Some other method?

Grading Rubric (A09.S5, individual, out of 30 points).

2nd Class Day, Friday 04/07/06

Step 6: In class discussion of anonymity and feedback
(whole class, 50min, 2nd day, 10:10am-11am)

Groups spokespersons will be called on to summarize the group discussions from WebCT.

More details (this section was updated on 04/07/06)

Answer three questions.

  1. Should evaluations be anonymous? What feedback should the student being evaluated receive?
  2. Should results of peer evaluations affect the course grade of students?
  3. If so, how?

We'll then see if there is a class consensus on the issue of feedback and anonymity, or not.

If not, we may modify the process outlined here to include reaching a consensus on these issues.

Results: (updated 4/12/2006)

There was a general consensus that feedback should

This was not a unanimous decision—a few students felt that feedback should include the names of students giving the feedback. We discussed some strategies to accomodate this view, but did not reach any final conclusions.

Day 3: Step 7: In class discussion of feedback forms
(Wednesday, 04/12/06—group, 30min, 3rd day, 10:15am-10:45am)

Today, the table arrangements reflect the rough consensus we noted on Wednesday:

Seating Arrangement

 

Your assignment:

When you have reached consensus

Step 8: In class discussion of final form chosen.
(whole class, 15min, 3rd day, 10:45am-11am)

If we can get here today, great. If not, we'll continue the discussion on WebCT, and try to acheive a final consensus.

Looking Ahead

Most groups designed their form as an HTML form. Such a form could be the start of a webapp that could:

How will you deal with security? There are elaborate ways (including accounts and passwords, encryption, etc.) Are there any easy ways to get something working fast without all that?

What are the security threats?

If you could do a full-blown web app with all the bells and whistles, what features would it have?

Valid XHTML 1.1 Valid CSS!