How to Serve as Program Chair of a Conference

by Jason Eisner (2007)

I was recently program chair (PC) of a moderately large CS conference, EMNLP-CoNLL 2007, which received about 400 submissions, with an acceptance rate of 27% (under 17% for talks).

To avoid duplication of effort, I saved various materials and am making them available via this page to future program chairs of ACL-style conferences.

This page concerns mainly the responsibilities of the program chair, who organizes reviewing and has final say on the program. There's obviously lots more to running a conference — local arrangements, sponsorship, publications, etc. — which is not covered here.

Many items on this page will also be useful to workshop chairs, especially if they're using the START online conference management system.


Reusable Materials For You

The most time-consuming thing for me was writing all the invitations, policies, advice, and detailed instructions. I kept an archive of all those emails — ask me for it. You can edit the emails as you see fit, of course.

Most of my advice is actually contained in these emails. Reading through them ahead of time may also give you a good sense of what's involved in the PC job. Even the list of filenames serves as a good overview:

(categorized by recipient; chronological within category)

advice-from-last-years-chairs

areachair-01-invite
areachair-02-invite-reviewers
areachair-02-invite-reviewers-reminder
areachair-03-reviewform
areachair-04-keywords
areachair-05-reviewer-policies
areachair-06-setup-bidding
areachair-07-assign-reviewers
areachair-08-assign-reviewers-resubmission
areachair-09-assign-reviewers-suggestions
areachair-10-reminders
areachair-11-invited-speakers
areachair-12-status-report
areachair-13-discussion-ranking
areachair-14-discussion-reminder
areachair-15-discussion-ranking-followup
areachair-16-revise-metareview
areachair-17-bestpaper
areachair-18-credits
areachair-19-dinner

reviewer-01-invite
reviewer-02-setup
reviewer-03-bid
reviewer-04-assign
reviewer-05-remind
reviewer-06-remind-again
reviewer-07-discuss
reviewer-08-revise
reviewer-09-thanks
reviewer-10-program

author-1-confirmation
author-2-extension
author-3-reject-without-review
author-4-accept-poster
author-4-accept-poster-shepherded
author-4-accept-talk
author-4-accept-talk-shepherded
author-4-double-submission
author-4-reject
author-5-emptyreview
author-6-accept-poster-pleaseconfirm
author-7-reminders
author-8-program

sessionchair-1-invite
sessionchair-2-instructions
sessionchair-3-reminder
And here are some useful links that appear in the above emails:
(public face of the conference)
   http://cs.jhu.edu/EMNLP-CoNLL-2007
   http://cs.jhu.edu/EMNLP-CoNLL-2007/CFP.html
   http://cs.jhu.edu/EMNLP-CoNLL-2007/CFP.txt
   http://cs.jhu.edu/EMNLP-CoNLL-2007/program.html
   http://cs.jhu.edu/EMNLP-CoNLL-2007/sessionchairs.txt

(how to configure START)
   http://cs.jhu.edu/EMNLP-CoNLL-2007/keywords 
   http://cs.jhu.edu/EMNLP-CoNLL-2007/reviewer-policies
   http://cs.jhu.edu/EMNLP-CoNLL-2007/review-form.html

(instructions to authors)
   http://cs.jhu.edu/EMNLP-CoNLL-2007/your-final-paper.html
   http://cs.jhu.edu/EMNLP-CoNLL-2007/talk.html
   http://cs.jhu.edu/EMNLP-CoNLL-2007/poster.html

General Principles (not stated in the emails above)

Things to Do Early (before deploying the emails above)

Roughly in this order:

Now you can move ahead with the emails above, starting with invitations to the area chairs.

Staying Organized

There will be a lot of details for you to keep track of. Make up your own system; I just kept several outlines in text files. Here are some examples of records you might keep.

How Not to Select the Final Program

How not to do it: Please, please, please don't just sort the papers by the 3 reviewers' average overall recommendation! There is too much variance in these scores for n=3 to be a large enough sample. Maybe reviewer #1 tends to give high scores to everyone, reviewer #2 has warped priorities, and reviewer #3 barely read the paper or barely knows the area. Whereas another paper drew a different set of 3 reviewers.

How still not to do it: Even as a first step, don't sort the papers by average recommendation. Trust me — this noisy and uncalibrated ranking isn't even a good way to triage the papers into likely accepts, likely rejects, and borderline papers that deserve a closer look. Don't risk letting it subtly influence the final decisions, or letting it doom some actual, nuanced reviews to go unread.

What I told myself: When you're working with several hundred papers, a single paper with an average score of 3.8 may seem to merit only a shrug and a coin flip. But a single false negative might harm a poor student's confidence, delay her progress to her next project, or undermine her advisor's grant proposal or promotion case. Conversely, a single false positive wastes the time of quite a lot of people in your audience.

Thus, you'll want to be pretty confident of every decision, after appropriate discussion. This is possible because you're the kind of broadly experienced person who gets to be program chair, and more importantly, because you have picked great area chairs and given them clear instructions.

How to Select the Final Program

Here's a procedure that I recommend.

Collecting rankings from area chairs: I asked each area chair to send me a careful within-area ranking, with comments. Instructions were in these emails:

   areachair-12-status-report
   areachair-13-discussion-ranking
   areachair-14-discussion-reminder
   areachair-15-discussion-ranking-followup

After all, the area chairs are the experts within each area. Not only do they understand the standards of the area and the technical content of the reviews, but also they may know their particular reviewers well enough to calibrate them. Furthermore, they are working with a lot fewer submissions than you are, so they have enough time for detail work. Where the ranking is unclear, they can ask the reviewers for further discussion, or look carefully at the submitted papers themselves.

Merging the rankings (overview): It took me a couple of days of hard work to merge these rankings and make the final 3-way decisions (talk vs. poster vs. rejection). Here is what I did, in brief. (A detailed version appears further down the page.)

  1. Create a spreadsheet with one row per submission.
  2. Assign a single numerical "strength" to each submission, by calibrating the area chairs' recommendations.
  3. Sort submissions by strength.
  4. Mark the top n submissions as talks, the next m as posters, and the rest as rejects.
  5. Identify, reconsider and tune questionable strength values where this might affect the decisions.
  6. Return to step 3 (repeat until convergence).
  7. Let the area chairs know of your provisional decisions, to give them a chance to object.

PC meeting: The above procedure doesn't require any program committee meeting. EMNLP does not traditionally have such a meeting; the final decisions are made by one or two program chairs, who exchange email with the area chairs.

But if you are going to have a physical PC meeting of all the area chairs, you might be able to adapt the above procedure:

Merging the rankings (detailed version): Anyway, here's the same procedure in more detail:

  1. Use START's Spreadsheet Maker tool to create a spreadsheet with one row per submission. (More information about this below.)

  2. Assign a single numerical absolute "strength" to each submission.

  3. Sort submissions by upper bound on strength. Break ties for now by sorting by average recommendation.

  4. Mark the top n submissions as talks, the next m as posters, and the rest as rejects.

  5. Identify, reconsider and tune questionable strength values where this might affect the decisions. (You may find yourself refining the strengths to more precise values, like 3.9 and 4.1.)
  6. Return to step 3 (repeat until convergence).

  7. Let the area chairs know of your provisional decisions, to give them a chance to object.

How to make the spreadsheet: You can get the data from START's Spreadsheet Maker tool. I pulled out the following columns, in this order. The starred columns were ones that I added myself.

   *Notes
   *Decision
   Submission ID
   *Strength (actually 2 columns for upper & lower bounds)
   *Area chair ranking
   Several columns for review subscores (averaged across all 3 reviewers):
      Recommendation 
      Appropriateness
      Clarity
      Originality
      Soundness
      References
      Depth
      Impact
      Impact of Resources
      Audience
   Title
   Authors [keep this column offscreen or omit it altogether]
   Keywords
   Submitted elsewhere
 

To get this, I believe that I generated two START spreadsheets — Submission Information and Review Information — with the default settings, and then joined them on submission ID. I then deleted irrelevant material, added the columns marked with * above, rearranged columns into the above order, and manually set column widths and colors to make the spreadsheet easier to work with. You may want to ask a careful administrative assistant to help with this clerical work.

Here is an example of how I filled in the Area Ranking field for the area chaired by Suzanne. The string format here is chosen carefully, so that sorting the spreadsheet on this column will separate the papers by area, and will order the papers within each area by the area chair's ranking, as needed for 5. above. The end of the field shows Suzanne's absolute recommendation, in her own words. Note that Suzanne's ranking contained ties (e.g., her rankings #2 through #5 are in a 4-way tie):

   Suz 01 (of 34) accept
   Suz 02-05 accept
   Suz 02-05 accept
   Suz 02-05 accept
   Suz 02-05 accept
   Suz 06 probable accept
   Suz 07 probable accept
   Suz 08 clear poster
   Suz 09 definite poster
   Suz 10 definite poster
   Suz 11 probable poster
   Suz 12-13 probable poster
   Suz 12-13 probable poster
   Suz 14-15 marginal poster
   Suz 14-15 marginal poster
   Suz reject
   Suz reject
   ... [19 recommended rejections]

Conditional Acceptance

For a few papers, you and the area chair may want to discuss the possibility of a conditional acceptance as a talk or poster. See the emails areachair-13-discussion-ranking for when this is appropriate, and author-4-accept-talk-shepherded for how it is handled with the author.

To avoid disclosing any reviewer's identity, it is usually the area chair whom you should appoint as the "shepherd" for the conditionally accepted paper. The "shepherd" must communicate with the authors, telling them what needs to be changed, and then must determine whether the camera-ready version has met these requirements. Typically, the shepherd delegates most of this work to the reviewer(s) whose concerns need to be met.

Innovations

We tried a few new things with EMNLP-CoNLL'07. Most were very well received. And here are some things that I wish we had done:

Tips on Using START

TO DO: I have some notes and memories and should write them up here. Send me additional material, too.

Handling Awkward Situations

If these happen to you, feel free to ask me how I handled them ...


This page online: http://cs.jhu.edu/~jason/advice/how-to-chair-a-conference.html
Jason Eisner - jason@cs.jhu.edu (suggestions welcome) Last Mod $Date: 2008/08/07 15:23:38 $