How to Grade Homeworks

by Jason Eisner (2010)

This page is mainly directed at TAs and CAs for my classes.

General TA matters

There are lots of facets of being an effective TA/CA. At the start of term, we should discuss:

• Being approachable, positive, enthusiastic, etc. Teachable moments.
• Coming to class at least sometimes, both to refamiliarize yourself with the material and to keep track of what the class knows.
• Giving help on the Piazza discussion site, or via the class mailing lists.
• Weekly discussion/tutorial sections (watch the videos here for why this is important!)
  • Solving problems together (e.g., from past exams)
  • Challenging the group with a discussion question (e.g., show them something confusing and ask them to get unconfused, or ask how they'd build a system to solve a particular problem)
  • Fun activities (e.g., from the 2002/2005/2008 Teaching NLP workshops, or the NACLO practice problems, or the ML or logic community)
  • Mini-lectures at the whiteboard on things that you think deserve to be re-taught
  • Answering questions about lecture and homework
  Try using a Doodle poll to pick the times -- perhaps restrict to weekday evenings, or other ranges that the students say seem workable.
• Exam review sessions: Study the practice exams and solutions carefully. I suggest handing out the solutions only at the review session, so that students will have more of an incentive to try the questions themselves and then come to the review session.
• Drop-in office hours (especially near assignment due dates). If you decide to hold these, you may want to poll students for good times, and choose different days than the ones where I already have office hours.
• Meeting with students by appointment (you may want to announce your preferred times).
• Grading exams. I write official solutions and then we all pull a marathon grading session together (the food's on me).
• How TAs are evaluated by the students at the end of the course.
• Logging your hours and submitting timecards.
• Privacy and confidentiality (FERPA).

The rest of this page is about how to grade homeworks.

Feedback to students

Deadlines

All assignments are due back to students by 1 week after the due date, or 1 week after submission in the case of a late assignment. This is incredibly important -- I will be very unhappy if you let this slip. Schedule your other work accordingly, coordinating with other graders.

Why it's so important: Feedback is important for learning, but its value decays quickly as the students start to forget the details of the assignment. In fact, they are less likely to even read late feedback. Students also use the feedback to shape what they do on the next assignment. They would complain (legitimately) if they didn't get homeworks back promptly.

Notifications

Please email students when the assignment is graded, so that they don't have to keep checking the website. Ask them to contact you promptly if they have questions about your grading (this will simplify your life, and also gives the students extra incentive to read the feedback).

If there'll be much difference in the finishing times, please send individual emails to each student as you finish with them, so that they hear sooner rather than later.

If you have graded the first half of all assignments (for example), then it would be nice to drop a note to the class list so that people can go check those comments early, while the problem is still fresh in their mind.

When you've graded all assignments (including late assignments), please email the class some statistics about the grades, so that they can realize that they're in trouble or feel proud of doing well. It is probably enough to send mean, standard deviation, and max, but sometimes I just send a sorted list of all grades.

Sending that final email also lets me know that the assignment is graded. This would also be a good time to tell me how it went (see below).

If a student has not handed in the assignment (even late), please ask them whether they have officially dropped the class, and cc me. Unless they have officially dropped, not handing in the assignment is a sign that they are screwing up one way or another. They need to hear this from an adult before it is too late to recover.

If there is something wrong with a student's submission (e.g., some file is missing or doesn't run), email them as soon as possible to find out what is going on, and cc the staff list.

Specific feedback

Give a total grade as well as the sub-grades for individual questions, e.g., "20/25 + 3EC." You don't necessarily have to write out that you are giving "-1" for this and "-2" for that if it's too much trouble, but you should usually point out those problems.

Alongside the total grade, remind them: "You have used a total of 3 late days (including 1 on this assignment)." (Don't say "you have 7 late days left," which would sound like encouragement to go ahead and use them.)

If a student has not handed in the assignment, please send them a personal email asking what's up. If a student has handed in the assignment but has used up their last late day in doing so, please send them a personal email warning them that they have no late days left for future use.

Specific comments are expected. You'll be able to write longer, more explantory comments if you reuse them across students. Reusable comments about particular errors can be included in the grading rubric for easy cut-and-paste.

Offer praise as well as criticism.

Don't try to discuss every tiny bug. There is a limit to how much detailed feedback any student can mentally digest on one assignment. If there are many problems with the assignment, focus on the most important things. Give high-level advice, including "come see me to clear up these confusions."

It is sometimes appropriate to email the class list to debug common confusions.

Feedback to professor

After the assignment is graded, I'd like to hear how it went (in person or by email). What did students have trouble with? What were the most common comments that you put on the assignment? How should we improve or replace this assignment next year?

Please tell me about any assignments that suggest that a student might be having trouble and need extra help. We need to catch this early and get the student back on the track to success.

Please also tell me about any particularly strong or interesting assignments. Students deserve to have their professor know when they are doing neat stuff -- they'd want me to know even if they never ask me for a recommendation. And from my side, seeing good work and creativity is one of the rewards of teaching.

If grading the assignment gives you any ideas for exam problems, please send them to me.

Assignment management

Submission account

• At the start of the semester, update the submission script and change the password for the grading account.
• Keep all assignments archived so we can check for cross-year plagiarism. (Also, every several years we will need to document that year's student work for ABET re-accreditation of our B.S. degree.)
• Well before the first due date, send out submission instructions. Example:

     There is a script which allows you to submit your assignments here:
  
     http://www.ugrad.cs.jhu.edu/cgi-bin/cgiwrap/cs325sub/grades_sp_10/submitWithLog.pl
  
     Just create a password with the first option and use it to send a
     compressed file containing your assignment.  Make sure it contains a
     plain text file named "README", or "READUS" for collaborative work
     (more details on this are in the assignment handout).  If you have any
     trouble using this script, you can e-mail your assignment to me
     (XYZ@gmail.com) to make sure you get credit for completing it on
     time.
  

Recording grades

Record grades on a special spreadsheet that you can get from me. Please pay careful attention to the following:

• Edit the column headers as needed for this year's questions. Use column headers starting with "EC" for extra-credit problems (which are curved differently). Use column headers starting with "EX" for grad-only problems (which count as extra credit for undergrads).
• Remember to fill in the maximum possible score for each question on the "perfect student" line. This figures into the curve.
• When adding new rows or columns, insert them into the middle of the range, so that they are included in existing summations over that range.
• Unsubmitted work should be recorded as a blank cell (not as zero, which would pull down the curve).
• An assignment that comes in 1 minute after the deadline is a slow net connection, not a late day.

Be alert for errors. Spreadsheets are error-prone. It's easy to get a range messed up, or to accidentally hit a key and mangle just one of hundreds of copies of the same formula. (Eventually the spreadsheet will be replaced by a Dyna program that just states the rules in one place ...)

Grading Rubrics

Often I have written my own solution to the homework. So please ask me to share my own answers, code, and output with you. Please also check your grading rubric with me.

A grading rubric might look something like this:

  10 points: Question 1

     3 points: Widget design
        -1 wrong number of prongs (but four-prong design ok if they explain flipping trick)
        -0 assume earth is perfectly spherical (no points off, but give comment)

     4 points: Widget implementation
        Give at least 1 point if the program compiles.
        Give at least 2 points if it gets correct output on all test cases.

        Syntax errors (limit to -2 total):
          -1 semicolon errors
          -2 curly brace errors 
        Bugs (limit to -2 total):
          -1 inverted logic
          -1 skewed logic
          -1 rotated logic
        Poor practice (limit to -2 total):
          -1 no range checks
          -1 or -2 inadequate comments  

     3 points: Analysis and discussion
        3 = great thoughtful discussion, almost perfect
        2 = good answer but misses a few points
        1 = only shallow understanding, or careless
        0 = badly confused or missing answer
  ...

As noted above, the rubric can also include suggested written comments for particular errors. As you grade, the rubric may grow with new errors and comments. Obviously, some changes may require that you revisit already-graded assignments to ensure consistency. You will probably want to look at a sampling of student answers to design the rubric before you start assigning grades.

If a question has been used before, there may be an existing rubric for it available in the grading account. However, don't assume that this rubric is a good one. You may want to edit it or discuss it with me ...

Some fairness principles:

• Be consistent across students.
• If there are problems with the design or wording of the question, students shouldn't suffer for that.
• Don't penalize twice for the same error. E.g., give full points on question 3(b) if they clearly understood how to answer it but got the wrong answer because of an error inherited from question 3(a).
• Don't penalize for writing style or brevity, though you may want to comment on it.
• Try hard to understand what they're doing. This may be hard if the answer was brief or they didn't show their work. To make your life easier, you can warn them to be clearer or more explicit next time. It is also possible to email them for clarification.
• Extra credit should be given to people who put in extra time or thought to do something especially impressive. The amount of extra credit should generally be enough to improve their grade, but not by a crazy amount. (Extra credit corresponding to 1 standard deviation of the base grade is probably the upper limit; that has the same effect on the final grade as if their B+ on some other assignment were converted to an A.)

Feel free to ask me for advice on any of this.

Handling Grading Errors

We do want to fix clear grading errors, of course, especially systematic ones that affect multiple students or multiple problems for the same student.

When there is a grading dispute that is not clearly on one side or the other, I usually just call a truce and promise to revisit it in the unlikely event that the final course grade is on the borderline. This should be noted in the comments field at the right of the spreadsheet.

Plagiarism

It is obviously important to detect this. Discuss countermeasures with me in person.

Thank you!

Thanks in advance for your work educating the next generation of students! You play a really important role in the course.

"Learning has to occur in the students. You can do anything you like in the classroom or elsewhere -- you can stand on your head -- and it doesn't make a whit of difference unless it causes a change in behavior of your students. Learning takes place in the minds of students and nowhere else, and the effectiveness of teachers lies in what they can induce students to do."
-- Herbert A. Simon, "What We Know About Learning" (1997)