My previous post explained how computer scientists distribute their articles (whether refereed or unrefereed, “published” or “unpublished”). Now I will explain the strange way they referee them, because it’s probably quite unfamiliar to those in other fields of science and the humanities. I describe it not because “every discipline should do it this way” but because “it is what it is, and it is economically sustainable.”

The primary refereed venue of publication in Computer Science is the conference. Computer Science also has monographs (but extremely few these days) and journals (but they seem to be considered less and less important). The big annual all-of-computer-science meeting is not at all important, and most computer scientists have never been to it. The real conferences (of which there are hundreds) that count–for tenure, and for any kind of visibility for senior scientists–work like this:

A typical computer science conference meets once per year, attracts 100 to 500 participants, meets for three days, and has a single track of about 30 papers, each with a half-hour presentation slot. Each subdiscipline of computer science has its own conferences. Some conferences receive over 200 submissions (and accept 30 for an acceptance rate of 15%); others receive only 60 submissions (and accept 30 for an acceptance rate of 50%). Papers submitted to conferences are subject to very strict length guidelines (e.g., 12 pages, 9-point font, double-column, which is the equivalent of at least 30 pages double-space 11-point).

Each conference puts out its annual Call for Papers (CFP) listing the Program Committee (PC) and specifying the submission deadline, the accept/reject notification date, and the conference date. For example, submission deadline July 17, notification Sept 29th, conference January 17th. The PC comprises a Program Chair and 15 to 25 members. The Program Chair and the PC members are entirely unpaid, but being on PCs is helpful for one’s tenure or promotion case. Each year, the Program Chair is selected on a one-time basis by the sponsoring organization, and the Program Chair selects the PC members. It’s an honor to be asked.

Authors upload their papers through the conference web site on the day of submission (rarely earlier), and the PC members receive them the same day. Each paper is assigned to 3 or 4 PC members by the Program Chair (often using automatic software based on subject terms). Each PC member is responsible for reading and refereeing 10 to 30 papers in two months, writing referee reports, and assigning scores (typically on a scale of 1 to 5). PC members are permitted (in some conferences expected) to get referee reports from other scientists not on the program committee. PC members can (and often do) referee more papers than just those assigned to them.

Then there is a PC meeting. Typically this is a physical meeting at a location in meat-space, but sometimes it is entirely conducted by e-mail. Papers with very low scores are hardly discussed at all, other papers are hotly argued about, and finally some 30 or so papers are accepted.

Immediately after the meeting, authors are notified whether their papers have been accepted. An author of an accepted paper is expected to improve it based on the comments in the referee reports, and submit a final version one month later. In most cases, nobody checks that the final version is in any way improved or corrected. In all cases, nobody copy-edits the manuscripts for English usage. These “final version” papers are submitted camera-ready by authors as PDF files, and they appear in the final printed proceedings, which are distributed to participants at the beginning of the conference when they register. These proceedings are also distributed in print and electronically by the sponsoring organizations (ACM, IEEE, Usenix, ETAPS, etc.) as part of their periodicals offerings (and/or as part of their on-line Digital Libraries).

Papers that appear in conferences are sometimes extended, revised, strengthened, and then submitted to journals; this was once the norm but is rarer and rarer, depending on the norms of the subdiscipline.

What can we observe and conclude:

• The process is very cheap. Referees are not paid; Editor in Chief (i.e., Program Chair) is not paid; copy editors are not paid (because they don’t exist); production editors are not paid much (because all they have to do is electronically staple 30 PDF files together). The on-line web site for uploading papers and managing the uploading of referee reports is sometimes rented for money, but not much money.
• Income to pay these costs comes from conference registration fees of conference participants ($500, of which most goes to the hotel for meeting rooms and meals).
• The deadline is very important. Refereeing gets done because the PC member would be very embarrassed to come to the meeting without uploading scores and reports. Thus, authors get timely turnaround by a guaranteed date.
• The limit on length of paper is rather Procrustean, but it ensures that the PC members are not overburdened with long papers to referee.
• It’s a lot of work to be Program Chair or a PC member, but you only have to commit on a one-time basis: PC members have two months of work to do at a time predictable in advance.
• Believe it or not, it’s common for computer scientists to get to tenure without ever in their life having a paper professionally copy-edited. Consequently, computer scientists are worse in their abuse of hyphens than any other academics, because it’s the use of hyphens where professional copy editors diverge the most from common usage. (Common usage is rather haphazard.)

This entry was posted on Wednesday, October 24th, 2007 at 11:29 am and is filed under The Future of Scholarly Communication.