RQC: Review Quality Collector

Lutz Prechelt, prechelt@inf.fu-berlin.de

Review Quality Collector (RQC) is a procedure to be introduced into the scientific publishing system, by which researchers can be formally recognized for their performance as peer reviewers (initially for journals and conferences, later perhaps also for funding agencies).

The first part of its software implementation is online since September 2016. Read up on the concept there:

https://reviewqualitycollector.org

This page (which was once quite long) has now been cut down to the not-yet-quite-outdated parts which may or may not be of interest, but certainly only after you visited https://reviewqualitycollector.org.


This section collects (very very incompletely) links to other initiatives aiming at improving the reviewing process in various ways. If you know of good such material, please drop me a note (with a web link) to prechelt at inf.fu-berlin.de.

Various issues of review quality

  • JAMA 280(3), July 1998, had a number of studies related to reviewing quality

Providing review quality feedback to reviewers

  • Callaham et al. (2002) found in a controlled experiment that providing basic quality feedback (just a 1-to-5 point rating) confidentially from the editor to average and below-average reviewers did not improve the quality of their subsequent reviews.
  • (see also some of the entries under "Recognizing reviewers' work publicly" below)
  • Furthermore,
    • many journals send all reviewers of a paper a copy of the editor's reply to the authors (including the text of all reviews).

Recognizing reviewers' work publicly

In particular: Open peer review (OPR)

Other


  • Are non-monetary rewards effective in attracting peer reviewers? A natural experiment (Zaharie and Seeber 2018). Shows that review quality evaluation can damage intrinsic motivation if framed wrongly. (Link)
  • A review article about research on peer review (Birukou et al. 2011)
  • A fixed questionnaire, called RQI, to serve as a measurement instrument for review quality (van Rooyen et al. 1999)
  • "Many [reviewers] wanted to see feedback on their own performance, perhaps an end of year review.
    Individuals who had received an acknowledgement or some form of feedback in the past implied they were more pre-disposed to review for that journal." (Mulligan, 2004)
  • "It was felt that many of the perceived problems afflicting the refereeing process could be remedied if the review was formally recognised.
    Such a system could take the form of accreditation to a journal, society or publisher. It might identify the number of times an individual reviews in different prestigious journals; such an indicator could in part be a measure of that individual's contribution to science." (Mulligan, 2004)
    (RQC covers this idea -- and goes far beyond it.)
  • A survey (sense about science, 2009) and focus groups (Mulligan, 2004) found that
    • The dominant motivators for reviewers today are participating in the scientific process (whether for the joy of it or out of a sense of duty), helping the authors, and interest in the content of the articles.
      (So RQC can create an additional source of motivation.)
    • Editors find it becomes harder and harder to find and keep good reviewers.
    • Editors perceive the nature and quality of the reviews they receive as highly fluctuating.
    • And many other interesting attitudes.
  • see also the Callaham entry under "Providing review quality feedback" above
  • For an overview of various experimental studies on improving peer review in the biomedical field, see Jefferson et al. (2008).
  • For some fundamental thoughts on what constitutes peer review quality, see Charles Jennings' Quality and value: The true purpose of peer review?
  • The online-consultation "scientific publication system" generated some comments on reviewing: too many publications, often sloppy reviews. See Section "Results 2", point 6 in this summary.
  • A survey of senior authors and reviewers determined various attitudes towards reviewing, the indicate predominantly (64%) satisfaction, but also the feeling that improvements are possible (only 32% think they are not). (Ware 2009)
  • Publons' "Global State of Peer Review" and a summary in Nature News
  • 2015 Springer survey on reviewer motivation, 1400 responses: "Reviewers were generally against a points-based system or rewards based on number of reviews and instead preferred being rewarded and recognized based on the quality of the review"

References:


Other peer-review stuff


Research assessment

  • Plaudit connects an ORCID ID and an endorsement with a DOI.
  • Björn Brembs about the cost of publishing 1 article.
  • APE (Academic Publishing Europe) is a conference of mostly the publishing industry. APE 2019
  • Force11 "is a community of scholars, librarians, archivists, publishers and research funders that has arisen organically to help facilitate the change toward improved knowledge creation and sharing [...] through the effective use of information technology."