Proposal:
                       
        Reviewer of the Year Award

        Lutz Prechelt, prechelt@ira.uka.de, 1997-12-05
   
1. Frequency:
   After the end of each calendar year the editor-in-chief of the
   journal awards the 'Reviewer of the Year (RoY) Award'
   to the one individual who performed the best reviewing service
   for the journal during that year.
 
2. Methodology:
   The award is given to that reviewer who received the highest number
   of review points during that calendar year.
 
   Review points are given in three categories; each review receives
   between 1 and 5 points in each category as specified below.
   These points are summed for each review.
   Hence, any single review has at least 3 and at most 15 points.
 
   Only the three best reviews of any single reviewer are counted
   for a given year (Note 1).
   Hence, a reviewer scores at least 3 and at most 45 points for the year.
   Only reviews for initial submissions count; reviews for revised 
   versions of a paper do not (Note 2).
   Reviews are counted for the year in which they arrive at the editor, 
   regardless of when the call for review was issued.
   
   The categories for the points are:
   A. helpfulness of the review to the editor (for making an 
      editorial decision)
   B. helpfulness of the review to the authors (for improving the paper)
   C. timeliness of the review
 
3. Consequences:
   In an editorial early the next year, the editor publishes the name
   of the reviewer of the year (RoY) and describes what made his or her 
   performance best and what other reviewer may learn from it.
 
   Furthermore, a document declaring the winning reviewer to be
   "reviewer of the year" is sent to the RoY.
 
   In addition, all reviewers receive (by email) their own total 
   points score for their reviews of that year.
   They also receive a list of the anonymous points scores of all 
   reviewers so they can locate themselves in the spectrum of
   review quality.
   The list also indicates whether a score is based on 1, 2, or 3 reviews.
   
5. Criteria specification:
   These are the criteria describing how many points are scored by 
   a review in each of the three categories.
 
   A. Helpfulness of the review to the editor
     5 points: The review appears completely comprehensive, competent,
               and unbiased. The editor has full confidence in the editorial
               decision based on that review.
     4 points: The review appears competent and unbiased, but lacks
               discussion of a few of the issues relevant for the
               editorial judgement of the psper.
               Nevertheless the editor has high confidence in the editorial
               decision based on that review.
     3 points: The review is modestly incomplete or the reviewer appears
               not entirely competent to judge the paper adequately or
               the review appears modestly biased (in favor of or against
               the paper) due to the scientific or personal attitude of
               the reviewer.
               The editor has only modest confidence in the editorial
               decision based on that review.
     2 points: Several of the above-mentioned drawbacks apply 
               or one applies to a higher degree.
               The review still contains some useful information, but
               the editor regards the review as unreliable and tends to
               ignore its judgement if the other reviews are much better.
     1 point:  The review is of very little help whatsoever making an
               editorial decision.
 
   B. Helpfulness of the review to the authors
     5 points: The authors can easily identify and understand all 
               weaknesses of their paper.
               For each of these weaknesses the review suggests a concrete 
               improvement (unless that is completely obvious).
     4 points: Most weaknesses are handled in a helpful way, but
               the description of some weaknesses is incomplete, difficult
               to understand, or lacks a concrete suggestion for improvement.
     3 points: Ditto, but with a majority of all weaknesses is not 
               handled well in the review.
     2 points: The analysis of weaknesses is generally vague and provides 
               little help or guidance for actual improvements.
     1 point:  The review contains no useful indication how to
               improve the paper.
 
   C. Timeliness of the review:
     5 points: the review arrives at the editorial office within
               15 days after they were posted to the reviewer.
     4 points: ditto, within 1 month
     3 points: ditto, within 2 months
     2 points: ditto: within 4 months
     1 point:  ditto, after more than 4 months
 
6. Authority:
   The above criteria are applied by the editor, usually by the
   associate editor who handles the paper.
   The authors may be asked to provide their own judgement for
   category B.
 
7. NOTES:
   Note 1: The alternative would be to average over all reviews of that 
        reviewer. However, this might put the most frequent reviewers
        at unfair disadvantage, because a single bad review could spoil 
        their score.
 
   Note 2: It is difficult and probably not fair to compare first 
        reviews to re-reviews.
 
   Note 3: To be fair, the judgement in category A (helpfulness to the
        editor) needs to correct for the fact that it is
        much more difficult to provide help for the editorial decision if
        the paper is neither very good nor very bad.