RQC: Review Quality Collector

Lutz Prechelt, prechelt@inf.fu-berlin.de

Review Quality Collector (RQC) is a procedure to be introduced into the scientific publishing system, by which researchers can be formally recognized for their performance as peer reviewers (initially for journals and conferences, later perhaps also for funding agencies).

The first part of its software implementation is online since September 2016. Read up on the concept there:

https://reviewqualitycollector.org

This page (which was once quite long) has now been cut down to the not-yet-quite-outdated parts which may or may not be of interest, but certainly only after you visited https://reviewqualitycollector.org.


Related initiatives

This section collects (very very incompletely) links to other initiatives aiming at improving the reviewing process in various ways. If you know of good such material, please drop me a note (with a web link) to prechelt at inf.fu-berlin.de.

Providing review quality feedback to reviewers

  • Callaham et al. (2002) found in a controlled experiment that providing basic quality feedback (just a 1-to-5 point rating) confidentially from the editor to average and below-average reviewers did not improve the quality of their subsequent reviews.
  • (see also some of the entries under "Recognizing reviewers' work publicly" below)
  • Furthermore,
    • many journals send all reviewers of a paper a copy of the editor's reply to the authors (including the text of all reviews).

Recognizing reviewers' work publicly

In particular: Open peer review (OPR)

Other


Related research

  • Are non-monetary rewards effective in attracting peer reviewers? A natural experiment (Zaharie and Seeber 2018). Shows that review quality evaluation can damage intrinsic motivation if framed wrongly. (Link)
  • A review article about research on peer review (Birukou et al. 2011)
  • A fixed questionnaire, called RQI, to serve as a measurement instrument for review quality (van Rooyen et al. 1999)
  • "Many [reviewers] wanted to see feedback on their own performance, perhaps an end of year review.
    Individuals who had received an acknowledgement or some form of feedback in the past implied they were more pre-disposed to review for that journal." (Mulligan, 2004)
  • "It was felt that many of the perceived problems afflicting the refereeing process could be remedied if the review was formally recognised.
    Such a system could take the form of accreditation to a journal, society or publisher. It might identify the number of times an individual reviews in different prestigious journals; such an indicator could in part be a measure of that individual's contribution to science." (Mulligan, 2004)
    (RQC covers this idea -- and goes far beyond it.)
  • A survey (sense about science, 2009) and focus groups (Mulligan, 2004) found that
    • The dominant motivators for reviewers today are participating in the scientific process (whether for the joy of it or out of a sense of duty), helping the authors, and interest in the content of the articles.
      (So RQC can create an additional source of motivation.)
    • Editors find it becomes harder and harder to find and keep good reviewers.
    • Editors perceive the nature and quality of the reviews they receive as highly fluctuating.
    • And many other interesting attitudes.
  • see also the Callaham entry under "Providing review quality feedback" above
  • For an overview of various experimental studies on improving peer review in the biomedical field, see Jefferson et al. (2008).
  • For some fundamental thoughts on what constitutes peer review quality, see Charles Jennings' Quality and value: The true purpose of peer review?
  • The online-consultation "scientific publication system" generated some comments on reviewing: too many publications, often sloppy reviews. See Section "Results 2", point 6 in this summary.
  • A survey of senior authors and reviewers determined various attitudes towards reviewing, the indicate predominantly (64%) satisfaction, but also the feeling that improvements are possible (only 32% think they are not). (Ware 2009)
  • Publons' "Global State of Peer Review" and a summary in Nature News

References:


Other peer-review stuff


Loosely related things

Topic attachments
I Attachment Action Size Date Who Comment
RQC-receipt-dummyexample.pdfpdf RQC-receipt-dummyexample.pdf manage 21.6 K 26 Oct 2012 - 09:45 LutzPrechelt RQC receipt (conceptual example)
reviewer-of-the-year-proposal-1997.txttxt reviewer-of-the-year-proposal-1997.txt manage 5.5 K 14 Jul 2014 - 15:30 LutzPrechelt  
Topic revision: r57 - 21 Feb 2019, LutzPrechelt
 
  • Printable version of this topic (p) Printable version of this topic (p)