Easier Grading AND More Learning?

Ok, are you ready?   What I’d like to do now is to have you create some rubrics.  We’re going to do this very rapidly, so we’re not aiming at something polished that you can take to class next hour.  I just want you to have the experience before I talk about some key principles and give you a bunch of examples.

I’d like each of you to pick one of your courses, and create a rubric for some assignment (or all the assignments) in that course.  Can be written work, exam, oral presentation, performance, lab work, whatever.

Now, to keep this manageable, I’m going to place an artificial limit on you.  I want you to pick two or at most three things that you would be looking for in this student product, and then explain how you would differentiate between excellent, satisfactory, and unsatisfactory levels of accomplishment. 

(OV to show format.)

Let’s say I’m evaluating an essay in an introductory history course, and I decide that I’m going to judge it on just two things – inclusion of the content we have studied and whether or not it is effectively organized.  I’d put “Content” and “Organization” in these boxes on the left, and then write in my description of what excellent work, satisfactory work, and unsatisfactory work would look like in for each category.

Ok, go – you’ve got five minutes.  What do you think?  Easy?  Hard?

Let’s look at some examples of what other people have done

      Overhead of very vague rubric

What do you think?

• What does “professional” mean?   Or “Good Quality”
• Last two grammar categories clearer, more concrete
• What is the operational difference between “understood topic well,” “understood topic,” and “had basic knowledge of topic.”  
• Example of place where more than four categories creates problems.                         

In both categories, it would be easier to understand if they were more consistently specific.  For example, I think it would be a lot stronger if instead of “professional,” it said something like “no grammatical errors.”    Or, in the demonstrated understanding category, they could have done something like “consistently connected ideas with supporting facts” for excellent, and then something like “frequently made appropriate connections between supporting facts and ideas” for very good.

Ok, let me show you some better examples and we can talk about them as we go.   BY THE WAY, IF YOU HAVE SOMETHING THAT WORKS, I’D LOVE TO HAVE A COPY.  PLEASE REMEMBER TO PUT YOUR NAME ON IT SO I CAN GIVE CREDIT.  

Checklist –

Annette’s example

 If you simply want to make sure that the student did a number of fairly mechanical things correctly – like have a title page with the proper information, number the pages, use a particular citation format, and provide a bibliography, then you could use a Checklist.   As the name suggests, this form of rubric simply asks did the student include or do x, y, and z.

• Really good for reminding students of basic expectations
• Very easy tool to use if you want to include these things in the grade
• Not intended to get at more important issues and higher learning

Diane’s example

• For a group led class discussion (also has other part that deals with how much work each group member did)
• Doesn’t describe levels of achievement, but communicates the various categories of things they need to try to accomplish.  Very explicit reminder.
• Great for quick/clear grading – each category worth max of 5 points, so just enter number. 
• Then “Comments” for anything else

Holistic
Holistic rubrics are basically at the other end of the spectrum from checklists.  They provide a broad judgment of the quality of the work as a whole.  All the criteria are considered in combination and rated together on a single scale.

     GRE Essay Scoring Guide
     Spanish

What do you think?

• Clear descriptions of the differences with specifics (especially in the Spanish one)
• Would provide a fairly clear set of expectations to students
• How much help in grading
• Yes, in that creates clear criteria
• No, in that have problem of evaluating paper where student does some aspects in the highest category really well but also screws up some
• No, in that instructor can’t easily point to specific areas of strength/weakness

Analytic
This is probably the form most commonly used because it allows you make judgments about independent dimensions of a student’s work. 

• Public Speaking Competencies
• Really like the clearly separated “dimensions.”   Might occasionally be difficult to use to provide feedback – dimension #4, excellent has exceptional introduction AND clear, logical progression

       [Note errors in two boxes]

Mathematics
Would work well for assessment and grading – could easily circle specific areas. 

Letter Writing
Seems like it would be really clear and easy for grading. 
Different levels not really defined, so might not convey expectations as well.

Persuasive Essay
Very clear, specific.  Especially appropriate for self or peer evaluation, but also would work well for grading.

As you can see from things like the persuasive essay and the letter writing examples, some of rubrics are only suitable for very specific assignments, while others, of course, might fit most or all of what you have students do. 

The great value of a consistent rubric that you use all semester is that it really helps us and our students to see their improvement and remaining areas of difficulty.  If, however, we try to shoehorn a different kind of assignment into a general rubric, we’re going to have a mess.   So my message is that if you have assignments with different goals, you need to make sure that you take the time to adjust and develop appropriate rubrics.

Now, let me try to pull this together and give you some suggestions from the literature on best practice in creating rubrics. 

           HANDOUT – Creating Rubrics

Starting

• Determine what you want to measure and type of rubric needed
• Think about what you want your students to achieve
• Create list of most significant qualities or dimensions of the desired student achievement 
• Arrange list in sequence; can be importance or order in which aspects are likely to be observed
• Choose number of different performance levels.  Use only as many as can be clearly described and differentiated.  More than four is often problematic.
• Create descriptions of the qualities associated with each level, expressed in terms of observable behaviors or characteristics. 
• [Keep in mind that words mean different things to different people, and our students don’t have the same expectations as we do.  For example, terms like “thoroughly” and “adequately” are going to be translated very differently.  Need to be more specific if possible
• If desired, connect grading scale with rubric level

Review
Check your goals/expectations against sample of prior student work if possible.  Are your expectations reasonable?

• Share with colleagues
• Check for clarity
• Inclusion of all critical elements
• Exclusion of marginal elements
• Discuss with class
• Check for clarity, understanding of criteria
• (Can have class help with initial drafting)

[You can generate better communication and buy in if you ask the students to work collaboratively with you to design the rubrics.  Obviously, this takes class time so you have to weigh gains here against what you would be giving up, but people who do this argue that it really works.]

Test
With yourself
     Does it actually work in practice?  Where are the rough spots?

With students
    Can they translate the performance descriptions into specific actions?
    Is there evidence that it is improving their understanding?

With colleagues (for assessment)
   Collective grading exercises to check for common
   Understanding (ETS as example)

Revise

• Remember it is a process
• Change as you discover problems; change as you change the course objectives.  

One final big word of warning.  We can come up with the most precise, beautiful rubrics in the world, but they are going to be useless unless they actually communicate our vision to students. To overcome this problem, let me make two more suggestions. 

First, don’t do what I have done.  Don’t put your criteria on the syllabus or some other sheet of paper and just leave it at that.  If you don’t refer to the criteria frequently and use them explicitly and consistently in your grading, they are not going to be of any real help.

Second, to overcome the “language” barrier – the disconnect between our understanding of certain words and our students’ view, it is really helpful to lay out what we expect in concrete terms using the class material. For example, if you are about to have them do an exam with short answer questions, lay out what a perfect answer would be for one of the possible questions.  Or, walk them through the ideal content/organization of a lab report or paper. 

It also works to have copies of excellent work in previous classes available for them to look at, but in my experience that isn’t as powerful as doing it orally over material that is absolutely fresh to them so that they can really see what you are doing.

Primary Sources Used:

            Jim Eison, “Creating Rubrics: A Strategy for Enhancing Student Learning and Improving Grading Efficiency,” Presentation at POD Conference, October, 2001

            Kathleen Montgomery, “Authentic Tasks and Rubrics: Going Beyond Traditional Assessments in College Teaching, College Teaching, Vol 50, #1, Wnter 2002, pp. 34-39

            Barbara Walvoord and Virginia Anderson, Effective Grading

  WHY DO RUBRICS?

TEACHING    

• Forces us to think about and clarify expectations
• Helps us make sure teaching practice supports course goals
• Pushes us to match evaluation to course goals             

LEARNING

• Communicates expectations to students before evaluation
• Helps students understand our evaluation of their work
• Students can see improvement and areas needing work.  Helps students critique their own work
• Provides guidance and improves peer feedback           

GRADING

• Encourages more consistent evaluation
• Helps faculty provide feedback on specific targeted areas
• Can decrease grading time (or free faculty from spending time on basic issues)
• Students often see rubric based comments as more objective

ASSESSMENT

• Can form basis for departmental assessment
• All must agree on rubric
• Applied to sample of student work by team of readers
• Or by individual faculty grading student work
• Shared criteria means greater validity and consistency than individual grades
• Basically grading so less intrusive/threatening

Creating Rubrics

Starting

• Determine what you want to measure and type of rubric needed
• Think about what you want your students to achieve
• Create list of most significant qualities or dimensions of the desired student achievement 
• Arrange list in sequence; can be importance or order in which aspects are likely to be observed
• Choose number of different performance levels.  Use only as many as can be clearly described and differentiated.  More than four is often problematic.
• Create descriptions of the qualities associated with each level, expressed in terms of observable behaviors or characteristics
• If desired, connect grading scale with rubric level

Review

• Check your goals/expectations against sample of prior student work if possible.  Are your expectations reasonable?
• Share with colleagues
• Check for clarity
• Inclusion of all critical elements
• Exclusion of marginal elements
• Discuss with class 
• Check for clarity, understanding of criteria
• (Can have class help with initial drafting)         

Test

• With yourself
• Does it actually work in practice?  Where are the rough spots?
• With students
• Can they translate the performance descriptions into specific actions?
• Is there evidence that it is improving their understanding?
• With colleagues (for assessment)
• Collective grading exercises to check for common understanding

Revise

• Remember it is a process