RQC: Review Quality Collector
Lutz Prechelt,
prechelt@inf.fu-berlin.de
Review Quality Collector (RQC) is a procedure
to be introduced into the scientific publishing system, by which
researchers can be formally recognized for their performance as peer reviewers
(initially for journals and conferences, later perhaps also for funding agencies).
The first part of its software implementation is online since September 2016.
Read up on the concept there:
https://reviewqualitycollector.org
This page (which was once quite long) has now been cut down to the
not-yet-quite-outdated parts which may or may not be of interest,
but certainly only
after you visited
https://reviewqualitycollector.org.
This section collects (very very incompletely)
links to other initiatives aiming at improving the reviewing process in various ways.
If you know of
good such material, please drop me a note (with a web link) to prechelt at inf.fu-berlin.de.
Various issues of review quality
- JAMA 280(3), July 1998, had a number of studies related to reviewing quality
Providing review quality feedback to reviewers
- Callaham et al. (2002) found in a controlled experiment that providing basic quality feedback (just a 1-to-5 point rating) confidentially from the editor to average and below-average reviewers did not improve the quality of their subsequent reviews.
- (see also some of the entries under "Recognizing reviewers' work publicly" below)
- Furthermore,
- many journals send all reviewers of a paper a copy of the editor's reply to the authors (including the text of all reviews).
Recognizing reviewers' work publicly
In particular: Open peer review (OPR)
- Preprint commentary/open peer review/post publication peer review venues (Google Doc)
- F1000Research is an open-access journal founded by Vitek Tracz, the inventor of BioMed Central. It performs post-publication open reviewing (and commenting and author-reviewer dialog and article revisions):
"All submissions to F1000Research will go through a rapid initial check by the in-house editorial team before being published with the status 'Awaiting Peer Review'."
Reviewing results become part of the article metadata, only positively reviewed articles are indexed.
- ScienceOpen also uses a publication style of post-publication open reviewing and commenting.
- A few journals (sometimes) publish the reviewer reports along with accepted articles, for instance
- Nature trialed "open peer review" in 2006: Authors were asked to agree to post their submitted articles publicly for receiving non-anonymous and public review comments.
Only 5% of the authors agreed to participate with their article. Only 54% of those articles received any comment at all (most of these only one). The editors judged only 4% of the comments to have the quality of normal reviews.
- Frontiers uses a format called "collaborative peer review" and has very detailed review process guidelines.
- According to the "What do we know about peer review" presentation on the BMJ reviewer resources page, three controlled trials found that signing a review did not improve its quality. (But the references are vague: Godlee et al, 1998; van Rooyen et al, 1998; van Rooyen et al, 1999)
- Blogosphere: A lot is written about open peer review in various blogs.
ScienceOpen collects such material in the ScienceOpen Blog and on the ScienceOpen facebook page.
Or see these good articles:
- Open Science:
Other
- CWTS course on responsible metrics in research evaluation (2019), refers to the Leiden Manifesto.
- PRE-Score assigns a quantitative score to a journal based on various characteristics of that journal's reviewing process, with the intention of allowing authors, readers, and libraries to identify journals where submitted work is "properly vetted".
- In Spain, in order to get promoted to a professorship, a researcher needs to bring a letter attesting his or her reviews from each journal for which the researcher claims to have reviewed.
These are collected by a central agency ANECA responsible for nationwide researcher quality assessment.
- Furthermore,
- some journals let reviewers pick manuscripts for review (rather than have an editor ask them to review a particular one), for instance
- Nature hosted an online debate on peer review in 2006 that features a number of interesting position papers.
- Casrai.org standardizes research administration information
- ASAPbio proposes to separate the evaluation function of peer review from the gatekeeping function by installing a peer feedback step before the journal review step.
- ORCID peer review early adopter program, peer review workflow
- Are non-monetary rewards effective in attracting peer reviewers? A natural experiment (Zaharie and Seeber 2018). Shows that review quality evaluation can damage intrinsic motivation if framed wrongly. (Link)
- A review article about research on peer review (Birukou et al. 2011)
- A fixed questionnaire, called RQI, to serve as a measurement instrument for review quality (van Rooyen et al. 1999)
- "Many [reviewers] wanted to see feedback on their own performance, perhaps an end of year review.
Individuals who had received an acknowledgement or some form of feedback in the past implied they were more pre-disposed to review for that journal." (Mulligan, 2004)
- "It was felt that many of the perceived problems afflicting the refereeing process could be remedied if the review was formally recognised.
Such a system could take the form of accreditation to a journal, society or publisher. It might identify the number of times an individual reviews in different prestigious journals; such an indicator could in part be a measure of that individual's contribution to science." (Mulligan, 2004)
(RQC covers this idea -- and goes far beyond it.)
- A survey (sense about science, 2009) and focus groups (Mulligan, 2004) found that
- The dominant motivators for reviewers today are participating in the scientific process (whether for the joy of it or out of a sense of duty), helping the authors, and interest in the content of the articles.
(So RQC can create an additional source of motivation.)
- Editors find it becomes harder and harder to find and keep good reviewers.
- Editors perceive the nature and quality of the reviews they receive as highly fluctuating.
- And many other interesting attitudes.
- see also the Callaham entry under "Providing review quality feedback" above
- For an overview of various experimental studies on improving peer review in the biomedical field, see Jefferson et al. (2008).
- For some fundamental thoughts on what constitutes peer review quality, see Charles Jennings' Quality and value: The true purpose of peer review?
- The online-consultation "scientific publication system" generated some comments on reviewing: too many publications, often sloppy reviews. See Section "Results 2", point 6 in this summary.
- A survey of senior authors and reviewers determined various attitudes towards reviewing, the indicate predominantly (64%) satisfaction, but also the feeling that improvements are possible (only 32% think they are not). (Ware 2009)
- Publons' "Global State of Peer Review" and a summary in Nature News
- 2015 Springer survey on reviewer motivation, 1400 responses: "Reviewers were generally against a points-based system or rewards based on number of reviews and instead preferred being rewarded and recognized based on the quality of the review"
References:
- http://asapbio.org/six-essential-reads-on-peer-review, 2018-01
- Ross-Hellauer, Deppe, Schmidt: Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers, 2017-12.
Only 44% believe open identities will help review quality; 60% ditto for open reports.
- Aliaksandr Birukou, Joseph Rushton Wakeling, Claudio Bartolini, Fabio Casati, Maurizio Marchese, Katsiaryna Mirylenka, Nardine Osman, Azzurra Ragone, Carles Sierra, Aalam Wassef, 2011: Alternatives to peer review: novel approaches for research evaluation
- Michael L. Callaham, Robert K. Knopp, E. John Gallagher, 2002: Effect of Written Feedback by Editors on Quality of Reviews: Two Randomized Trials
J. American Medial Association 287(21):2781-2783
- T. Jefferson, M. Rudin, S. Brodney Folse, F. Davidoff, 2008: Editorial Peer Review for Improving the Quality of Reports of Biomedical Studies
Cochrane Database of Systematic Reviews, John Wiley & Sons.
- Adrian Mulligan, 2004: Is Peer Review in Crisis?
Perspectives in Publishing, August.
- Sense About Science, 2009: Peer Review Survey 2009.
- Susan van Rooyen, Nick Black, Fiona Godlee, 1999: Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts, Journal of Clinical Epidemiology, 52(7):625-629.
- M. Ware, 2008: Peer review in scholarly journals: Perspective of the scholarly community-Results from an international study., Information Services & Use 28:109-112.
- Bezis, Birukou: Arbitrariness in the peer review process, Scientometrics, 2020
Other peer-review stuff
Research assessment
- Plaudit connects an ORCID ID and an endorsement with a DOI.
- Björn Brembs about the cost of publishing 1 article.
- APE (Academic Publishing Europe) is a conference of mostly the publishing industry. APE 2019
- Force11 "is a community of scholars, librarians, archivists, publishers and research funders that has arisen organically to help facilitate the change toward improved knowledge creation and sharing [...] through the effective use of information technology."
- …