Referee Coordination For The Fifth Symposium On Operating Systems Principles




James L. Peterson




Department of Computer Sciences
The University of Texas at Austin
Austin, Texas 78712




January 1976




This paper has been published. It should be cited as

James L. Peterson, ``Referee Coordination for the Fifth Symposium on Operating Systems Principles'', Operating Systems Review, Volume 10, Number 1, (January 1976), pages 7-16.


Introduction

The Symposium on Operating Systems Principles has become a major conference for researchers and practitioners in operating systems. Held biannually in the Fall under the auspices of ACM and SIGOPS, the conference has been held five times since 1967. My involvement in the Fifth Symposium on Operating Systems Principles was mainly a result of being at the University of Texas at Austin, the site of the Symposium. At an early organizational meeting, I volunteered for and was given the position of refereeing coordinator, and soon found that this was an open-ended position with little in the way of guidelines or recorded history to direct my efforts or to indicate what was to be expected. It is to remedy the lack of a recorded history that this paper has been written.

This paper is a brief chronology, summary and evaluation of the refereeing and selection of papers and preparation of the proceedings of the Fifth Symposium on Operating Systems Principles. It is hoped that this will provide a history which will be considered by future conference planners, both in determining policy and procedures. It is also hoped the participants in a conference, including the planners, authors, speakers and attendees, will be better able to understand a conference through a knowledge of some of the steps involved in the preparation for the conference. The organization of this paper is chronological, beginning with the selection of referees and the receipt of papers through the refereeing and selection process to the preparation of the proceedings.

It is to be stressed, however, that this is only an abridged history of a particular role in a specific symposium. The problems, statistics and methods presented here may not be applicable to another conference for reasons of subject matter, size, personnel, finances or previous experience. A large number of people, on the Program Committee, in ACM, in SIGOPS, on the faculty and staff of the University of Texas, and elsewhere, contributed a large amount of time and effort to the Symposium. Their experiences, backgrounds and viewpoints may naturally differ from those expressed here, This paper is not to be taken as necessarily representing the views of anyone other than the author.

Selection of Referees

In January, 1975, I was invited to participate in the Symposium preparation as refereeing coordinator and I started to compile a list of referees. Initially, a list of potential referees was obtained by using the list of referees from the Third and Fourth Symposia, their program committees and the authors of presented papers. A copy of the IFIP 74 referee list was obtained and used for some names in the area of operating systems. Also recent authors of operating system papers in the Communications of the ACM were used as well as the authors of papers referenced in three or four current operating systems text books. This produced a list of 107 potential referees. These people were sent a letter (Form A) asking them to be referees and a form to indicate "yes" or "no", their address, areas of expertise and interest, and other possible referees (Form B). This last item produced another 42 potential referees who were sent forms as their names were suggested. More than 42 additional referees were suggested, but many of these names were already on our list. Thus, approximately 150 people were asked to be referees directly. An announcement was placed in the SIGOPS Operating System Review Newsletter asking for volunteer referees also.

The response to all this was

As this processing progressed, a couple of additional lists of potential referees had been acquired, but most of these people had already been asked. Two classes of people were deliberately excluded from our list of referees:

The first were excluded since it was assumed they would be working diligently on the reviewing of papers anyway; the latter were excluded due to the time involved in overseas communications.

Receipt of Papers

Papers were received from as early as April, 1975 to as late as 7 July 1975 for a 1 June deadline. The table below illustrates the number of papers received each week.

Time interval  Number of papers received
April   - May 3   5 papers
    - May 10  5
    - May 17  4
    - May 24  5
    - May 31  24
    - June 7  47
    - June 14  11
    - June 21  8
    - June 28  3
    - July 7  4

For the time around the deadline for the call for papers, we have,

May 27  12 papers
May 29  3
May 30  9
June 2  35
June 3  3
June 4  6
June 5  0
June 6  3
June 9  6

As each paper was received, a status form (Form C) was filled out, an acknowledgment letter sent (Form D) and a file created for copies of the paper and correspondence relating to that paper. Each paper was assigned an internal paper number (from 1 to, eventually, 119). Since the call for papers did not specify the number of copies, we had to contact the authors and request additional copies in most cases. We decided that 5 copies were needed, believing that we would need one for the central file, one for the program committee member assigned to the paper, and one for each of the three referees.

Selecting Referees for Papers

It was intended that papers would be sent out to referees as received, but we were unprepared for the large number of papers received and so sending them out to referees was delayed by almost three weeks with the major number of papers sent to referees the last week of June. Each referee was sent several papers, a cover letter (Form E), guidelines for refereeing by R. Stockton Gaines (Form F), and an evaluation form (Form G) for each paper. This was dated as needing to be returned within a month.

The selection of referees for individual papers was a major problem. Each referee had, of course, supplied on (Form B) his areas of interest and expertise, but these areas were very badly chosen, being much too general. This made it very difficult to determine which referees were interested in (and capable of) reviewing specific papers. In retrospect, a better list of interests would have been:

After the first sixty or so papers had been received, we started to assign each paper to a program committee member. These assignments were made on the basis of the known areas of expertise of the program committee members, and each program committee member was sent a copy of the papers assigned to him. This was to allow the program committee members to help with the selection of referees, to evaluate the papers, and to decide on acceptance/rejection. A number of program committee members did send suggested referees, but by the time these arrived, referee assignment had been completed and the papers mailed out.

The actual selection of referees was not well-organized. Originally it was planned to have 3 referees per paper but we found our list of referees to be too small for this. After trying to select, for each paper, two or three referees, it was found to be easier to select, for each referee, two or three papers. Still it was very difficult to assure at least two referees per paper, and to select appropriate referees in all cases. We added about 30 new names of persons who seemed especially appropriate for particular papers, including well-known researchers, authors of submitted papers on similar subjects, friends of the program committee and graduate students in our department.

In all we assigned two referees to 49 papers, three referees to 64 papers and four referees to 3 papers. This resulted in the following distribution of the number of papers per referee.

Number of Referees sent  1  paper  29
   2  papers  41
   3  papers  42
   4  papers  11
   5  papers  2

Plus one referee who received, reviewed, and returned 8 papers!

Of the approximately 300 papers sent out for refereeing, 21 were returned unrefereed. This was mainly the result of individuals who had recently moved, did not receive the papers until late due to forwarding problems, and did not have time to review them. This phenomena was generally limited to employees of the IBM Corporation and of the Department of Defense. Some papers were ret and some by people who had been or were going on vacation. In general it was not possible to send these papers back out for more reviews, as they were returned quite late and we were short of referees anyway. Of the remaining 288 reviews, 40 were never returned. The final number of reviews per paper received thus had a distribution of (approximately),

Number of reviews received  Number of Papers  Originally Sent
1  22  (0)
2  58  (49)
3  34  (64)
4  2  (3)

As the reviews came in, they were copied and sent to the responsible program committee member as well as being filed with the paper. A "prompt" letter was sent to each referee the last week of July, to remind them to return their reviews.

Program Selection

The first two weeks of August included two conference calls for the program committee in which the program was (roughly) determined. Of the 119 submitted papers,

Immediately following the last conference call, rejection letters (Form H) acceptance letters (Form I) and conditional acceptance letters (Form J) were mailed. Included with this letter were referee's reports and annotated manuscripts. One of the more difficult moments of my participation in the Symposium occurred a few days later when it was discovered that an acceptance letter had been mistakenly sent to the author of one of the rejected papers. Although the author seemed to understand and accepted my apologies, a little more care on my part could have prevented this incident.

Those papers which were conditionally accepted were papers which the Program Committee felt had good ideas, but the presentation of these ideas was poor, and hence major revisions were necessary. These papers would need to be re-reviewed before they could be accepted. Of the eleven, one was withdrawn and the others were eventually accepted, although a few required as many as three further revisions. Also one paper which was initially rejected was reconsidered by a member of the Program Committee and upon his evaluation and urging, the paper was moved to a conditional acceptance category and eventually accepted.

A tentative program consisting of those accepted and conditionally accepted papers was typed and mailed to all program committee members. This was also used to prepare and print a (tentative) program and registration brochure by Professor Ambler, in charge of local arrangements.

In addition to the copy of the (tentative) program, selected members of the program committee were mailed copies of all accepted papers. The members were chosen to allow sharing of copies between geographically close Program Committee members, although indications are that this sharing did not occur. This mailing was for the purpose of selecting those papers which were to appear in the special issue of the Communications of the ACM devoted to the Symposium. A preliminary screening of the papers by the faculty of the Computer Sciences Department at the University of Texas in the systems area had been made when the papers were first received. Those papers which seemed to be potential CACM papers were assigned to Dr. R. Stockton Gaines (a member of the program committee and editor of the Operating Systems Department of the CACM) as well as to another program committee member by area of interest. Twenty papers (out of the first 100 received) were thought to have CACM potential. Of these twenty, ten were accepted for the Symposium, and two were conditionally accepted. Eventually, nine papers were selected for the CACM, and of these nine, five had been pre-selected.

At the same time that the acceptance and rejection letters were mailed, "thank you" letters (Form K) were sent to all of the referees who had returned any reviews.

Proceedings

Shortly after the program was tentatively established and acceptance and rejection letters mailed, the refereeing process was complete, except for the conditionally accepted papers. These were to be re-refereed by program committee members and so we turned next to the problem of producing proceedings for the Symposium.

From the beginning, it was assumed that the proceedings would be prepared by having the authors type their papers onto special "model paper" provided by ACM. Five hundred sheets of this model paper were obtained from ACM. Shortly after the tentative program was established, we began to consider how the papers should be prepared. An author instruction sheet (Form L) was drawn up and a sample first page (Form M) typed. These, along with model paper, were sent to the contact author of each accepted or conditionally accepted paper. Conditionally accepted papers were included since if they were accepted this would decrease the time before the final paper could be returned. The number of sheets of model paper for each author was estimated by taking the number of double-spaced, one-sided pages of the submitted paper (including figures) and dividing by 2 (rounding up if need be). However, no more than 15 pages were sent for any one paper. Those authors whose papers exceeded 30 double-spaced typewritten pages were given a warning note, instructing them to shorten their paper. Graph I plots the number of pages of the returned model paper version versus the number of pages in the submitted version. Remember that the paper may have been revised between-these two versions.


GRAPH I: Showing the relationship between submitted pages and final pages.


The preparation of the Proceedings caused the greatest concern about timing during the Symposium preparation. We had planned mailing the proceedings' master copy to ACM for printing on 30 September. The kits for authors, consisting of model paper and author instructions were mailed 29 August, before the Labor day weekend. They did not start arriving at the authors until from ten days to two weeks later. Since they had been prepared with a return date of 15 September, this caused concern on the part of the authors and on our part of getting them back in time for our deadlines. The problems appears to have laid in the form of mailing. Since the model paper could not be folded, mailing tubes were used. Apparently, although the tubes were in fact first class mail, the postal service employees at each step in their handling assumed that they were not since they were not prominently marked as such and were not letter-size-objects. Duplicate kits were sent to some authors before kits began to arrive. One kit was returned to us by the Postal Service as "Found loose in the mails at Buffalo, N.Y.". Once the mailing tubes arrived, most authors prepared and returned the finished version in excellent time, using the same mailing tube to return the finished version.

Our major problem in preparing the proceedings was the finalization of the program and the scheduling of sessions. It was felt that the proceedings should present the papers by sessions. In any case, the proceedings could not be put together until the final acceptance/rejection decision was made for the conditionally accepted papers. This proved to be very difficult and final program decisions were not made until 6 October. Also, ACM publication policy prevents publishing a paper in CACM if it has appeared elsewhere. Hence, the papers chosen for presentation in the Symposium issue of the CACM could not be included in the proceedings. This decision was also made on 6 October. Sessions were decided on 7 October and the proceedings were prepared that night and mailed to ACM the next day. They were mailed air mail, special delivery, registered with return receipt requested and insured for $20,000. A Xerox copy of the proceedings was kept, which would have allowed the entire proceedings to have been retyped, if they had been lost or damaged in the mails.

The preparation of the proceedings was composed of two parts. First was the papers themselves. As this was done by the authors, this portion was not a problem. Four papers were returned in an unacceptable condition. By matching type fonts with the machines available in our Department, three were correctable with the retyping of only one or two pages and the liberal use of rubber cement and a razor blade. The fourth required complete retyping and layout. Other than this, which should not have been necessary, the only work in preparing the proceedings was preparing the foreword, the list of those who had worked on the Symposium, the table of contents, an author index and numbering the pages.

All papers had been returned to us on model paper, although those papers which were to appear in the CACM did not need this. However, that decision was not made until late. The model paper copies of the CACM papers were used to print copies of these papers for the participants of the conference.

Evaluation

Several aspects of the preparation for the Symposium created the need to make decisions concerning both policy matters and the mechanics of preparation for the Symposium. Our experience on these matters is presented here.

A computer data base was to be used for keeping track of the submitted papers and referees. However, my unfamiliarity with the system resulted in most records being kept by hand. The computer systems were not sufficiently flexible or convenient to be of any real help, although they were used to produce the address labels for the authors (four copies) and referees (nine copies). This allowed us to reduce the repetitive typing of addresses for our mailings.

Most of our communication was by mail, generally first class, but occasionally air mail. Mailing delays ranged from two days to as much as twelve days, and an unknown quantity of material was lost in the mails. This includes mailings both to and from referees and program committee members. We have information that, at least, five items did not arrive at their destination, although where they disappeared is in question. Most mail was processed both by the University of Texas, the U.S. Postal Service and an organization at the other end, allowing ample opportunity for delay, misdirection or loss.

The problem of attaining referees is major. Perhaps a solution such as that proposed by D. J. Frailey (CACM 18, 10 (October, 1975), p. 598) would work. An alternative mentioned by Dr. Browne would be for SIG/SICO to maintain referee lists for their area. Many of our referee reports resulted in very little information other than an accept/reject statement. The criteria used by the referees seemed to vary considerably and introduced a good deal of variation. It appeared that many of the program decisions were made mainly by the program committee on the basis of their own evaluation of the submitted papers, despite sometimes conflicting referees reports.

The refereeing problem could also be eased by reducing the number of papers which are refereed. The quality of submitted papers varied widely, and for perhaps half of the submitted papers, it was obvious upon receipt that they were not of sufficient quality to be accepted. These were run through the entire refereeing process out of a sense of fairness and to provide some comments to guide the authors on revisions, errors and future work. However this resulted in a heavy load on our referees and program committee, generally preventing a program committee member from reading papers other than those assigned to them. In retrospect, this may have resulted in the acceptance of some marginal papers which happened to match the dispositions of a particular program committee member. An alternative might be to have a small group scan the papers upon receipt and narrow the submitted papers to a manageable number to allow better refereeing and thorough consideration by at least two program committee members.

Another problem results from papers submitted by program committee members. Members of the program committee are specifically selected for their research abilities and prominence in the field, increasing the probability of papers authored, or coauthored, by program committee members Not allowing program committee members to submit papers would be unfair to them and their coauthors, but special precautions should be taken to prevent even the appearance of unfair consideration for their papers. In our case, special effort was taken to assure that no program committee member was responsible for his own paper and that our best referees were assigned to these papers. However, considering that 13 papers by program committee members were submitted (out of 119) and only 2 of these were accepted (out of 25), four rejected (out of 78) and the other seven were conditionally accepted (out of 11), the proportion of conditionally accepted papers for program committee members seems high. It might be wise for the organizing committee for a conference to specify early the procedures to be followed relating to papers submitted by the organizers of the conference.

Summary

We have attempted to give a history of one role, of many, in the preparation of the Fifth Symposium on Operating Systems Principles. The active work on the Symposium occurred over a period of eleven months, from January to November 1975, and at times required the full attention of a number of people. A number of problems occurred, some of which might have been preventable or at least expected had a history such as this been available from previous symposia. We hope that this record will be a help in the preparation for future conferences.