When peer review goes right
by Janet Salmons, Research Community Manager
Dr. Salmons has reviewed many journal articles, book manuscripts, case studies, and countless doctoral dissertations. She has received good, bad, and ugly reviews on her own writings. Her books from SAGE Publishing can be found here. =
We are familiar with stories of peer review gone wrong. Reviewer 2 makes impossible demands! The reviewer didn’t appreciate my brilliant and ground-breaking article! There are shortcomings as Marcoci et al observe:
Peer reviewers have been shown to overlook methodological flaws and statistical errors, avoid reporting suspected instances of fraud and commonly reach a level of agreement barely exceeding what would be expected by chance. (p. 1)
Yes, there are systemic problems. But let’s celebrate what can happen when peer review goes right! How can we develop the skills needed to become great peer reviewers who help improve the process for everyone?
What is peer review and why is it important?
Peer review refers to a process that involves critical reading by one or more reviewers with expertise in the methods and subject matter verify the credibility of the research, veracity of the claims, and truthfulness of the writing. The review is usually guided by a set of criteria. In a double-blind protocol both the reviewer and writer are anonymous; in a single-blind review the reviewer knows the identity of the writer, but the reviewer is not named. Typically the editor of the book or journal evaluates the manuscript and completed reviews, and decides whether to accept it as is, request revisions, or reject it.
Peer review is what distinguishes scholarly work from other types of writing. In an era when disinformation, misinformation, and opinions offered without appropriate evidence are widely available, interested readers need to trust research-based books and articles. Disciplined scrutiny adds reliability and trustworthiness to our writings. Peer review should ensure that knowledge that is disseminated is accurate and meets the highest ethical standards (Nichols-Casebolt, 2012).
In addition to its prevalence in scholarly publishing, peer review is a part of the decision-making process for awarding contracts, funding awards and grants.
What is needed to be an excellent peer reviewer?
Excellent reviewers:
understand the review criteria,
have the necessary expertise with the content and/or the methodology,
have no conflicts of interest, personal or professional biases that can influence your review,
respect the intellectual property presented in the paper,
commit to submission of the review before the due date,
communicate concerns or problems to the editor,
read published articles to understand the style and expectations of the journal, and
conduct a thorough review by identifying strengths and limitations, and giving comments that will help the author to make improvements. a thorough review will include comments on: The paper’s contribution to the area of research or scholarship, the conceptual and theoretical soundness, adequacy of the literature review, the appropriateness of the methodology and methods, and the organization and clarity of writing. Some reviews will also consider the potential impact of the project or publication, or alignment with the mission of the publication or funder.
As a potential reviewer, be honest with the editor. If you have limited expertise with either content or methodology, say so. (If you are inexperienced with both content and methodology, say no.) If you run into a problem, such as realization that you have a conflict of interest, or inadequate time to complete the review, contact the editor. Similarly, if you encounter plagiarism or possible fraud, contact the editor. This set of guidelines from COPE (Committee on Publication Ethics) focuses on the ethical obligations of reviewers and this page outlines Principles of Transparency and Best Practice in Scholarly Publishing.
Being constructive: Not kind, not cruel
Becoming a good reviewer is a skill: It is not easy to give constructive comments, to decide what is an essential change, and to decide what suggestions are optimal and at the discretion of the author. It is not easy, knowing how much work has gone into a study, to have to recommend that it not be published. But these are essential decisions. (Morse, 2014, p. 1)
Being “constructive” does not mean being nice. You do the author no favors to let something pass that is not up to the standards of the publication or organization. At the same time, even if you think the paper is the worst thing you have ever read, a peer review is not the place to express caustic views. There is no reason to be mean - getting a request for major revisions or a rejection will be difficult enough for the author. Clase et al (2022) explained the principles that a health journal stated:
Our explicit and unique policy is to address our authors as our colleagues with the aim to improve submitted works, regardless of our decision to publish. We will be no less scientifically and constructively critical, but we will write our reviews as we would wish to be written to ourselves.
Constructive kindness requires empathy (“write as we would wish to be written to ourselves”),1 and in our training sessions, we further operationalize this by suggesting reviewer and editors write “as if to your treasured mentor or to your most junior trainee.” Kindness is also a good in itself, a prosocial virtue, related to a desire to help others effectively and to altruism.
Be respectful and clear, noting the positive aspects of the work and offering as much specificity as possible about the problems. Add suggestions for improvement, helpful examples or resources.
I find that poor submissions take much longer to review, because I read the article multiple times and give careful thought to my comments. I read my comments again to make sure I have been firm but diplomatic, especially when rejecting the work. Look at the feedback you have received - good and bad. What was helpful? What kinds of comments gave you the confidence and courage to go forward with needed revisions?
Beyond manuscript evaluation: how do peer review skills help you be a better collaborative partner?
Learning to serve as a peer reviewer is beneficial beyond the contribution you make to scholarly publishing. Mutual critique and feedback is inherent to the collaborative process. Co-researchers, co-authors, team members and partners of all kinds depend on the ability to evaluate each other’s work. A common stage in most collaborative projects involves some degree of “you do this, I’ll do that” with parts of the work divided amongst the group. At some point those elements must be melded into the final article or presentation. Partners must look carefully at each other’s efforts and decide what stays, what is cut, and what revisions must be made. In other words, they engage is a process of peer review - one that can utilizes the same kinds of skills and criteria for excellence that are described throughout this post.
Be a part of the solution!
As Clase et al (2022) observe:
Reviewing is inherently an altruistic act, motivated by a desire to improve scientific knowledge and to give back to the community.
To benefit from new knowledge and insights into contemporary problems, we need trustworthy, well-written publications that are grounded in the literature and empirical research. By supporting excellence in the peer review process, you make a contribution to your field and society. Look on the websites of your favorite journals, on the sites for conferences you attend - they need you.
Also see: 10 tips on how to be an ethical reviewer by Kanika Kamboj
References
Clase, C. M., Dicks, E., Holden, R., Sood, M. M., Levin, A., Kalantar-Zadeh, K., ... & Hartwig, S. (2022). Can peer review be kinder? Supportive peer review: a re-commitment to kindness and a call to action. Canadian Journal of Kidney Health and Disease, 9, 20543581221080327.
Fallon, L., Grapin, S., Newman, D. S., & Noltemeyer, A. (2022). Promoting equity and social justice in the peer review process: Tips for reviewers. School Psychology International, 43(1), 12–17. https://doi-org.proxy.library.cornell.edu/10.1177/01430343211070165
Marcoci, A., Vercammen, A., Bush, M., Hamilton, D. G., Hanea, A., Hemming, V., Wintle, B. C., Burgman, M., & Fidler, F. (2022). Reimagining peer review as an expert elicitation process. BMC Research Notes, 15(1), 1–7. https://doi-org.proxy.library.cornell.edu/10.1186/s13104-022-06016-0
Nichols-Casebolt, A. (2012). Research Integrity and Responsible Conduct of Research. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780195378108.001.0001
Morse, J. M. (2014). Cooperative Scholarship: The Peer Review Process. Qualitative Health Research, 24(12), 1615-1616.
Salmons, J. (2019). Learning to collaborate, collaborating to learn: Engaging students in the classroom and online. Stylus.
More Methodspace posts about research ethics
Informed consent is the term given to the agreement between researcher and participant. In this post Janet Salmons offers suggestions about the intersections of the Internet communications, ethics and participants.