Peer Review of Database Studies: Rigors in Reams of Results
Peer review of research is fundamental to journal publication, and peer review of database studies is more complicated.
Peer review has been the standard approach to vetting scientific articles for centuries. It remains a critical aspect of all types of review and consideration for publication in medical and scientific journals. It is the subject that was examined by John H. Krouse, MD, PhD, MBA, Editor in Chief of Otolaryngology–Head and Neck Surgery and OTO Open, and Jennifer J. Shin, MD, SM, the Deputy Editor.
During the on-demand session “Peer Review of Database Studies: Rigors in Reams of Results,” Dr. Krouse explored the steps in peer review for articles in the AAO-HNSF journals, while Dr. Shin dug into the intricacies of large database studies.
“Research studies need to undergo an evaluation of its rigor, of its scientific content, of its ethical nature, to look at whether or not it is relevant and something the scientific community would value,” said Dr. Krouse. “It is an incredibly important part of the acceptance process for papers in science.”
In considering a manuscript, the reviewer follows this checklist:
1. Relevance to mission: Is the information consistent with the mission of the journal?
2. Internal validity: Are the study design, conduct, and analysis described in a manner that is unbiased, appropriate, and reproducible?
3. External validity: Was the study sample chosen appropriately and described in adequate detail for results to be generalized?
4. Level of evidence: Does this manuscript significantly improve the knowledge base beyond what is already published on this topic?
5. Ethical conduct: Is the manuscript original, approved by an Institutional Review Board (if applicable) and free of undisclosed conflicts?
“Our journal has a very high rejection rate (roughly 80%) due to its high levels of expectation for the manuscripts,” he said. “Being rejected is not a reflection of the work you have done or the value, but perhaps more of the methods used.”
Papers that are not rejected then typically go through a major revision. The reviewers may have found significant but addressable concerns about study design, reporting of results, generalizability of the findings or composition of the manuscript, or significant but addressable ethical concerns. The revision may result in acceptance, rejection, or request for an additional revision.
After all the revisions have been completed and the reviewers and editorial team feel that these revisions are satisfactory, an acceptance decision will be given.
Dr. Shin explored peer review of database studies, which has infinitely more intricacies to consider as a reviewer or an author. In particular, she focused on the database evaluation process, analytics performed, and database properties. 
Jennifer J. Shin, MD, SM
Within that database evaluation process, there are three areas of focus: transparency, reproducibility, and validity, she said. “Transparency and reproducibility mean that any cleaning processing and extraction of the data are made clear within the paper. It's typically not enough to just say you used a certain database. You should describe the specifics of how you used the database,” she said.
“It is important to have a clearly defined time frame of reference as that provides a temporal anchor and reference point within the data for both exposures and outcomes,” she said. “That's important regardless of whether you're looking at a cohort from a certain starting point and following it forward or if you are looking backward.”
Using “An Original Report Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies” as a guide, she reviewed key parameters that would be used to implement a database study and a checklist of items to consider when reporting or reviewing it.
She also reviewed database analytics. She recommended that authors focus on a primary outcome.
“This often can be a challenge for authors when they are presented with a huge amount of very tempting data,” she said. “Even if you are looking at a very large data set, you ideally still want to focus in on one, or a very small number of, primary outcomes. If you don't, you can be subject to error for multiple comparisons,” she said.
She concluded by discussing database properties.
“Different databases have different strengths and weaknesses. If you understand these, you realize what the pros and cons and potential limitations of an analysis are,” she said.
This session is available on demand.
