Peer review

Current academic practice

Our approach to peer reviewing is informed by Whitworth and Friedman's [WaF2009a] criticism of current academic publishing as a form of competitive economics in which "scarcity reflects demand, so high journal rejection rates become quality indicators". This self-reinforcing system where journals that reject more attract more results in a situation where "avoiding faults becomes more important than new ideas. Wrongly accepting a paper with a fault gives reputation consequences, while wrongly rejecting a useful paper leaves no evidence".

Whitworth and Friedman [WaF2009b] propose an alternative evaluation system:

1. higher rating discrimination: a many-point scale, not just accept-reject
2. more submissions to be rated: rate all
3. more people to rate: community involvement
4. different ways of rating: formal review vs. informal use ratings.

Our process is also informed by Toni Prug's [Prug2010] paper on community reviewing: submission proposals will be made on our open email list. Prospective authors will then be told by our peer review community whether their proposal is appropriate for the journal and if any additional key elements are missing.

CSPP peer review process

 Prospective authors and reviewers are invited to carefully read this section.

1 PAPER PROPOSAL

1.1. Non-special issues

Authors use the contact form to send a proposal to the editor who will post it to the journal list (thereby preserving author anonymity) for community vetting. If the author is not a list-member, the editor will relay any feedback to the author.

1.2. Special issues

editors.

2. SUBMISSION AND REVIEW

Once authors have completed a full submission, they send it to the editor who  assigns it to three anonymous reviewers. The reviewers assess the paper following our suggested review categories (see Appendix A). Once the reviewers have provided any necessary recommendations for improvement, these reviews are sent to the author for possible revision and the author decides whether to follow these recommendations.

3.REVISION AND SIGNALS

The revised paper is sent back to reviewers for final evaluation following signaling categories (see Appendix B). The editor forwards the signals to the author, who then decides whether to publish the paper or not.

4. PUBLICATION

The paper is published alongside the signals.

Signals will remain anonymous to ensure frank and fearless signaling.

Background information:

- All reviews will be released alongside published papers. Reviewers  may opt whether to remain anonymous or not.

- All initial draft submissions will be released alongside published papers unless the author provides compelling reasons (such as privacy of author or subjects) why this should not be the case.

Since draft submissions will be published, authors are encouraged to use a respectful tone when dealing with other researchers' work in their drafts.

Outcomes and benefits

It is up to the author to decide whether they are happy with publishing a paper with the given signals. Letting authors decide whether to publish enables the journal to release a wider variety of submissions whilst protecting its scientific reputation.

Signaled papers will be published alongside reviewers' initial reports and recommendations. This will help to promote review quality, as well as ensure that reviewers receive public appreciation for their work.

To offer a counterpoint to expert reviews, anyone who registers an identity with the site will be able to rate articles, and comment on articles. Statistics regarding article views and ratings, and reviewer activity will be available on request for authors and reviewers, or sent directly by the journal to the required institutions.

Appendix A: Suggested categories for peer review

1. Is the subject matter relevant?
2. Is the treatment of the subject matter intellectually interesting? Are there citations or bodies of literature you think are essential to which the author has not referred?
3. Are there any noticeable problems with the author’s means of validating assumptions or making judgments?
4. Is the article well written?
5. Are there portions of the article that you recommend be shortened, excised or expanded?

Appendix B: Suggested categories for signaling

• Objective categories

  Activist

  Article proposes a critique of a policy or practice with specific action proposals or suggestions: yes / no

  Academic

  Article follows conventions of academic research article -- e.g. position in literature, cited sources, and claimed contribution: yes / no

  Prospective

  Article is based on developments that have not yet occurred: yes / no

  Formalised

  Article is based on formal logic or mathematical technique: yes / no

  Language quality

  Standard of English expression in article is excellent: yes / no

• Subjective categories

  Scope of debate

  Article addresses an issue which is widely known and debated: yes / no

  Comprehensiveness

  Most related sources are mentioned in article [this is an invitation to careful selection rather than a demonstration of prowess in citation collection -- i.e. apt and representative choices made in source citations]: yes / no

  Logical flow

  ideas are well organised in article: yes / no

  Originality

  The argument presented in article is new: yes / no

  Review impact
  

  The article has been significantly changed as a result of the review process: yes / no

  Commendation

  Reviewers indicate their appreciation of the article in the form of a 50 word statement. [Alternatively: recommendation to others 1 (only to those with a very specific interest) to 10 (essential knowledge for all)?]