What makes the best medical ethics journal? Some candidate answers to this question may be that it publishes the leading articles in the field or that it makes a significant difference to the practice of medical ethics in clinical or public policy settings. Part of the answer will ultimately come down to the overall quality of the contributions to the journal. Thus to answer the question of what makes the best medical ethics journal we need to examine the more fundamental consideration of how such a journal attracts high quality reviewers and authors. To this end, we investigated the factors that contribute to determining why medical ethicists choose to review and submit articles to medical ethics journals.

In our examination of this issue, we have taken a bottom-up approach to this question by consulting the people who actively review for and submit articles to medical ethics journals. We surveyed the top medical ethicists in North America to obtain their considered judgements concerning factors that increase participation (that is, reading, reviewing, and/or submitting articles) in medical ethics journals in general, and to evaluate the perceived quality of a well established medical ethics journal, in this case, the Journal of Medical Ethics (JME). We chose North American medical ethicists because they comprise the largest and most influential medical ethics community, into which non-North American journals have the greatest trouble in gaining access. On the basis of this research, we identify what should be the qualities of the best medical ethics journal.

METHODS

We surveyed medical ethicists for information on incentives and disincentives to reviewing and submitting articles to medical ethics journals. The survey instrument was constructed to examine factors that act as incentives and/or disincentives to writing and reviewing for medical ethics journals, and other related factors such as characteristics of the peer review process and a focus on North American content (see Appendix 1: available from the JME website at http://www.jmedethics.com/supplemental). The survey was pretested among a group of medical ethicists in the UK to ensure clarity and comprehensiveness. The survey was approved by the University of Oxford Central University Research Ethics Committee. Survey participants were chosen based on a review of the academic and research records of medical ethicists working in higher education institutions in North America. Survey participant selection factors included the quality of the institutions at which the ethicists were educated, the quality of the institutions where the ethicists were employed, the number, quality, and breadth of their publications, the quality of the journals in which their publications appeared, how actively they taught medical ethics and related courses at senior undergraduate and graduate levels, and their involvement in professional medical ethics organisations/associations.

An email was sent to medical ethicists identified according to their academic and research records, inviting them to complete the survey through a secure website. Before completing the survey, respondents were informed of the conditions of data gathering, storage, access, and analysis. All data collection and analysis met the standards of the UK’s Data Protection Act (1998). Consenting respondents completed the survey and the data were subsequently analysed.

RESULTS AND DISCUSSION

This study represents the first and most extensive examination of medical ethicists’ attitudes concerning the incentives and disincentives that determine why they choose to review and submit articles to medical ethics journals, their attitudes concerning the peer review process, and their impressions of a leading medical ethics journal in the field. In what follows we use this feedback to identify what should be the characteristics and qualities of the best medical ethics journal.

Of the 131 surveys sent out to North American medical ethicists (n  =  105 in the USA, n  =  26 in Canada) 82 were completed, for a respectable response rate of 63%. Responses were received during November and December 2003.

The primary research area and/or training of the respondents was philosophy (52%), medicine (21%), law (20%), religious studies (3%), history of medicine (1%), politics (1%), psychology (1%), and anthropology (1%).

Incentives and disincentives to reviewing articles (figures 1 and 2)

According to the respondents, the most important factor cited as an incentive to review journal articles was the opportunity to contribute to the field of medical ethics. Although it was good to see that this laudable reason for reviewing articles was cited as the most important, it seems that this decision is greatly informed by the reputation of the journal and/or its impact factors. While medical ethicists want to contribute to the field of medical ethics, they want to do so in venues and in a capacity that will have the greatest impact, and for which they will receive the greatest recognition.

The ability to keep up to date on current research was also largely cited as an incentive to review articles. This will be an incentive, however, only if medical ethicists acting as reviewers believe the articles they are reviewing are of sufficient quality and relevance to meet this aim. The leading disincentives to reviewing articles cited by the respondents included time constraints due to academic and/or editorial commitments, and receiving manuscripts that were either outside their scope of interest or of poor quality. As more medical ethics journals compete to attract editorial board members and reviewers, and as more medical ethicists find they have less time to devote to such duties, medical ethics journals may find it more difficult to attract high quality reviewers. One solution to this problem is the rapid rejection of poor quality articles by editorial staff prior to the review process.

Thus, in addition to reputation based factors such as the quality of the articles published in the past and the overall reputation of the journal, the best incentives to encourage ethicists to review articles would include asking reviewers to review only a very small number of manuscripts (for example, one or two per year), ensuring that these manuscripts are very close to the individual reviewer’s subject of interest/specialty, and making the review process as simple and unproblematic as possible. One way to make reviewing manuscripts more time efficient and easier would be to have a website where reviewers and authors can upload, access, communicate, and complete modifications to manuscripts entirely online. For an example of such an online interface, see the JME’s Bench>Press system (http://submit-jme.bmjjournals.com). The advantages of such a system, among others, are that it can be used at any time of the day, it provides reviewers and authors with submission records and the ability to view the progression of manuscripts through the peer review process, and reduces the likelihood that manuscripts and related correspondence are lost or misplaced.

Financial inducements were the least important incentive to review. Payment would not, on the basis of this study, be the most effective way of attracting high quality reviewers.

Incentives and disincentives to submitting articles (figures 3 and 4)

According to the respondents, the reputation of the journal is the single most important factor in whether or not to submit an article, followed closely by the quality of the scholarship previously published in the journal. Although the impact factor of a particular journal was ranked as the third most important incentive to submit an article, because a large number of medical ethics journals do not have impact factors (or impact factors that are generally known in the profession), it seems that factors that are more difficult to quantify (for example, reputation and article quality) play the greatest role in where ethicists decide to submit their manuscripts for publication. On a more practical level, the turn-around time from submission to acceptance to publication was also cited as a significant factor in this decision.

For instance, one of the most prominent disincentives to submission is a slow turn-around from submission to publication. Although a long period of time between submission and rejection/acceptance/publication unfortunately seems to be the standard amongst medical ethics journals—sometimes up to two years—any journals that can offer authors the ability to publish their articles faster present a distinct advantage within the field. Journals that, for instance, continue to accept too many articles (resulting in a backlog waiting to be published) or have an inefficient editorial infrastructure and organisation, etc., risk not receiving the best manuscripts for consideration.

No comparative data currently exist concerning editorial decision and publication timelines for medical ethics journals; however, we would suggest that journal editors could attend to a number of items concerning their performance in this area. Journals should publicise statistics on their average time from submission to rejection/acceptance/publication for non-fast-tracked articles, so that authors may make an informed decision about the importance of the speed of publication. One good example is maintained by the philosophy journal Analysis, which has a stated policy of informing authors of manuscript decisions within 8 weeks, but frequently much faster, and once manuscripts are accepted they are usually published within 9 months. Lengthy periods between acceptance and publication are frustrating for authors who have worked long and hard on their articles, and can affect employment or promotion chances. Although medical ethics journals are not unique in having long periods between submission and acceptance/publication, the best of them will strive to provide relevant information and to implement policies and procedures that reduce these times.

One factor that may help journals in this area is the prepublication of articles on the journal’s website. The JME has adopted the practice of publishing electronically on their website articles believed to be of interest or importance as preprint versions prior to final proofing. This can place the arguments and content on the web in an accessible fashion without the delay of the editorial proofing process, which can hold up final print and online publication for months. Survey respondents did mention that the JME practice of prepublication acts as an important incentive to submitting articles.

Most medical ethics journals do not have a problem attracting manuscript submissions; the vast majority reject many more manuscripts than they accept. What we have identified, however, are the incentives that top medical ethicists cite as the factors that influence where they submit their work. Incorporating as many of these incentives as possible would be a feature of the best medical ethics journal.

The peer review process

In addition to examining the incentives and disincentives to reviewing and submitting articles for medical ethics journals, we also surveyed participants on their attitudes to particular aspects of the peer review process. If it is true that this is the best method by which we ensure the overall quality of contributions to academic journals, then it seems that the nature and operation of this process is an essential component that must be at the heart of what makes the best medical ethics journal.

While attracting high quality reviewers would be essential to the best medical ethics journal, we must examine the process of peer review itself to determine what is most likely to produce the best reviews. It is often argued that peer review is an essential component used by journals to select high quality articles, not merely as a method of identification of the best articles, but also as a way to improve the quality of those already submitted.^1–3 How then do we identify reviewers who will tend to select articles and produce reviews of excellent quality? More often than not, journals attempt to find (or believe they should find) reviewers who have good research records. However, this may not guarantee that they will produce high quality reviews. Recent research published in the medical literature on this subject has shown that the expertise of reviewers is poorly correlated with their ability to produce high quality reviews, owing mostly to not having enough experience of or opportunities for experiential learning in critical assessment.^1,4,5 In an attempt to counteract this, the British Medical Journal, for instance, has recently offered free courses and developed a training programme for its reviewers in order to educate them on topics such as the state of peer review research, what constitutes a good review, and how to produce good reviews.^6,7 The overall efficacy of such programmes, however, requires further examination before such measures could be advocated for all journals.

A large part of the problem stems from the fact that it is unclear whether it is possible to provide a list of characteristics of the qualities of good reviewers. One study showed that factors such as the strength of the reviewer’s academic institution and the reviewer being known to the editor tended to produce better reviews.⁸ Another found that the specific characteristics of reviewers have little association with the quality of the reviews produced.⁹ However, one notable factor that seemed to be an independent predictor of good reviews in both these studies was the youth of reviewers.^8,9 It may be the case that younger reviewers are more familiar with current research and are concerned to a greater extent with producing more productive reviewer reports than established academics. It was also shown that review quality increased with the time spent on reviews—up to 3 hours, but not beyond.⁹ When attempting to choose reviewers who would be most likely to produce the best reviews, it seems advantageous to choose junior or mid-career academics who are not encumbered by other academic and editorial time commitments.

Be that as it may, medical ethics journals will want to insist on a basic set of qualities for referees—for example, that they have a record of publication in the topic or area of the article to be reviewed, are not known to be unsympathetic to the article’s topic or approach, or are not the target of published criticism. In addition to attempting to identify reviewers with qualities that contribute to producing better articles, receiving fair and thoughtful review comments was also cited by respondents as an incentive to submission.

Open versus closed peer review (figure 5)

Instead of examining the quality of the reviews themselves (which is a topic as yet unexamined and definitely worthy of study in medical ethics journals), we decided to investigate attitudes towards two specific aspects of the peer review process: the importance of blinded versus open peer review systems and of author nominated reviewers on the quality of manuscript review.

Respondents generally strongly agreed that both authors and reviewers should be blinded to each other’s identity (that is, double blind: closed peer review). This result is not surprising; it has often been supposed that a process of blind review ensures that the merit of an article is judged by its content and not on the reputation (or lack thereof) of the author. Blind review, it may be thought, also helps to reduce the possibility of the bias that may exist, unfortunately, for reviewers who take the ethnicity or gender of an author (based on the name on the manuscript), or the perceived inferiority of an author’s institutional affiliation or position, as subconscious factors leading to viewing a manuscript as being of lower quality.

However, it seems that recent evidence may speak against this intuition. According to a number of studies conducted on the effect of open peer review published in the medical literature, no significant effect on the quality of the review or publication recommendations was observed.^10–14 Some of these studies even showed that the quality of reviews was slightly better with open review. However, others have demonstrated that reviewers blinded to the identity of a manuscript’s author tend to produce better reviews.^8,15–18 A subsequent meta-analysis of these studies suggested that they had methodological limitations that make their outcomes ambiguous and that most of the studies on open review were inconclusive because “blinding was ineffective and confidence intervals were not sufficiently narrow to allow firm conclusions to be made” (p. 2785).²

We found that, although many respondents cited a preference for blind review, a substantial proportion were not adverse to an open review system. This result is also not surprising because such a system may result in similar or better review quality stemming from the fact that an open review system can increase accountability, fairness, and transparency. If reviewers know that their identity will be known to the author, they may be more likely to produce more constructive comments, stronger justifications for criticisms and objections, and better reasons for rejecting an article for publication, etc.

It is possible that the interdisciplinarity of the field of medical ethics affects how medical ethicists view open review. It is also possible that medical ethicists have not really reflected on their intuitions about open reviews and the associated benefits that may lead some editors to move towards such a system. As it presently stands, however, blind peer review in medical ethics journals is the standard. One possible option, currently employed by some journals such as the JME, is to give reviewers the choice of whether they want their identity revealed. This is a means of providing the opportunity to have some of the benefits highlighted above and mitigating the risk of having reviewers declining to review if they are uncomfortable with having their identity known. It is quite unclear whether a blind, open or mixed system of review will produce the highest quality reviews and articles. Further research in this area would certainly be of benefit to the field of medical ethics. Until there is conclusive evidence of the superiority of an open review system, it appears that the beliefs of medical ethicists that a closed system is superior may affect their evaluation of the quality of a journal’s review process, and hence their willingness to submit and review articles.

It was less clear whether the ethicists believed that the use of author nominated reviewers (that is, when authors submit a manuscript to a journal they are able to identify possible reviewers that would be competent to review the article) negatively affects the quality of the review. Although the respondents did not express an overwhelming negative opinion concerning author nominated reviews, there were more who favoured a non-author nominated system. However, there may be some advantages in adopting an author nominated process in reviewing manuscripts. A system of author nominated reviewers could be used to increase the likelihood of manuscripts being reviewed by experts in the field, the rationale being that authors have an increased likelihood of knowing those who would be familiar with the research. Given the disconnection between research expertise and the quality of reviewing previously noted, and the possibility of partiality under such a system, it may be thought that moving towards a system of author nominated reviewers does not present editors or journals with any substantive benefit. Again, more research in this area is needed. Such evidence is vital to changing the current beliefs of medical ethicists and ensuring their confidence in the quality of a review system.

Depending on how one feels about open versus closed peer review and author nominated reviewers, it could be a further condition that editors would want to make sure that a referee is not a colleague of the article’s author. This may be difficult because there will be areas within a discipline where colleagues are well acquainted or work closely with each other, or there may be specialised areas where a referee with the expertise to review the topic of an article will inescapably need to be a (close) colleague of the author. This being said, the open review process is not the only way to achieve this. The editor of a journal who employs a process of closed peer review can insist that all review recommendations be clear and constructive, and have sufficient justification. Reviews found to be overly critical and unhelpful could be discarded.

Whether a journal chooses to adopt an open or a closed system of peer review, it is important that authors are informed of the journal’s policy and its review procedures, and what they can expect from the process, so that they may make an informed decision.

One definite conclusion that should be taken from such findings is that developing the best system of peer review is an important issue for more than journal editors. According to an emeritus editor of the Annals of Internal Medicine, Frank Davidoff, “scholarly communities need to get serious about making high quality peer review an integral part of all aspects of professional life, including training, practice, and reward systems” (p. 657).⁵ If peer review is thought to be fundamental to critical, academic enquiry, we need to ensure that adequate support (for example, training by graduate level and professional development courses) and recognition (for example, academic considerations involving hiring, tenure, and funding decisions that take peer review contributions into account) is provided.

Medical ethicists’ impressions of JME

Results and discussion are available from the JME website at http://www.jmedethics.com/supplemental

CONCLUSION

On the basis of our research, we believe the best medical ethics journal would:

• Have a large number of institutional subscriptions to paper and electronic versions of the journal.

• Have the best reputation possible by: (1) publishing high quality articles; (2) having an editorial board comprised of well respected scholars; and (3) facilitating easy access to and/or dissemination of articles.

• Have a journal website that: (1) allows reviewers and authors to access manuscripts in an efficient and simple manner; (2) offers a website and emailed table of recent contents to readers; (3) offers advanced electronic prepublication of important and timely articles; (4) offers an archive of previously published articles; and (5) provides annual statistics on editorial decisions and practices concerning average times between submission and rejection/acceptance/publication.

• Have a manuscript review system that: (1) makes reviewing articles as efficient and simple as possible; (2) offers the ability to track the progress/status of an article through the review process; and (3) has a system of peer review that ensures fair and constructive comments and which inspires confidence in its authors and reviewers. (At present, this appears to be a closed system.)

• Have a manuscript submission system that: (1) offers a rapid turn-around time from manuscript submission to rejection/acceptance/publication; (2) offers electronic prepublication of articles; and (3) has a system of peer review that ensures fair and constructive comments.

• Have content that: (1) focuses on local and international issues; and (2) has an approach/style/length that fits the proposed readership’s interests.

• Include content that is relevant and accessible to its readership through the thematic grouping of articles, symposia, and special issues dedicated to particular topics. This should be made more accessible by emailing the contents to interested readers.

• Publish articles of different lengths, with both philosophical and medical foci.

• Be based on high quality research justifying the nature of the peer review process. Further research in this area is required.

• Ensure that reviewers are asked to review few, high quality, relevant, up to date articles. A system of rapid rejection of unsuitable articles by editorial staff prior to peer review would achieve this.

Although we have been concentrating on the most important factors influencing submission and review, we have identified many that have some effect on decisions to submit and review manuscripts. All of these are worthy of consideration.

We hope this study will initiate further research on medical ethics journal publication. Medical ethics is far behind other disciplines in examining the importance of factors that influence participation and publication.

Our primary concern was with medical ethics journals, but the conclusions of this study may be applicable to other journals with similar disciplines, such as other areas of applied philosophical inquiry and associated medical disciplines.