PEPonline
Professionalization of Exercise Physiologyonline

An international electronic
journal for exercise physiologists
ISSN 1099-5862

Vol 10 No 7 July 2007

 


Reflections of an Editor: Peer Review and Professionalism
Tommy Boone, PhD, MPH, FASEP, EPC
Professor and Chair
Director, Exercise Physiology Laboratories
Department of Exercise Physiology
The College of St. Scholastica
Duluth, MN 55811
 
Scientific peer review is the evaluation of research findings for competence, significance, and originality by qualified experts. – T. Brown [1]

This article is about two important areas of academics (a term often synonymous with "professors").  The two areas are: (1) peer review; and (2) professionalism in exercise physiology.  Peer reviewed articles are often a critical piece to a professor’s promotion or tenure.  And yet, despite its importance, many professors think of peer review more as game than a process to ensure quality, truth, and justice.  Clearly, there are problematic issues ranging from incompetence to corruption.  But, it is also clear that the work of professors must be evaluated.  How else would colleagues and administrators know whether to promote or tenure a faculty member?

Peer Reviewed Journals

Professors are educated to believe that peer-reviewed publications are better than publications in journals that are not peer reviewed.  Of course this assumption is not always correct, but no one seems interested in raising this point.  Similarly, in exercise physiology especially, few seem interested in writing about professional development issues and concerns.  But, if they don’t write the articles, debate them, and discuss them, how will exercise physiologists in general know what to think or how to think about the professionalism side of exercise physiology?  Most colleagues from other professions cannot imagine that exercise physiologists are not interested in supporting their own professional development.   

Truth about Peer Review
But, for nearly 10 years, since the founding of the American Society of Exercise Physiologists, this is clearly the case.  The academic exercise physiologists are interest in publishing articles in elite, prestigious journals with high impact factors [2].  They have been taught that peer review is the only respectable way to publish their work.  As to allegations of bias and suppression of dissenting views, that is, the ups and downs of peer review [3]; professors are not inclined to take them very seriously. In fact, most researchers simply are not interested in the well-known abuses that exist in both the peer review process and the use of impact factors to judge the quality of an article. That is why the influence of both continues to be huge, but not necessarily controversial.   

Fortunately, with the changes that have resulted from the emergence of the Internet and electronic journals, the publishing landscape is changing somewhat.  Two significant turning points with scholarly journals include open access publishing, in which the articles or the journals are freely available from the time of publication, and self-archiving, where authors post their own work freely on the web.  As to the quality of the articles in electronic journals, in particular, it is unclear whether academia will make the transition from print-based publications to Internet publishing.  It is still much more popular to publish in print-copy publications than in online publications, although everything will eventually be online!

As a college teacher and researcher for nearly 40 years, and as founder of the Journal of Exercise Physiolgyonline, it isn't difficult to understand the history of peer review and why it is important for researchers in academia. In fact, I found the 2007 article by Benos and colleagues extremely well-written and interesting.  In short, it is an excellent paper [3].  The
authors  traced the history and development of the peer review process from its inception to its present-day application.  They also address the benefits and the imperfections of peer review, bias and fraud, changes that need to be made, and the future of the peer review process.  Their paper helps to clarify why an article is submitted to a peer review journal and why publishing in a non-peer review journal is believed to be a waste of time. 

It is difficult to get a man to understand something if his salary depends upon his not understanding it. – Al Gore

Long story short, journals that rely on peer review are valued (and respected).  Non-peer review journals are not valued or respected.  In fact, in academia, there is little acceptance or even tolerance for publishing in journals that are not peer-reviewed [4].  And yet, this was the standard practice for many years.  It was not until the middle of the 20th century that refereeing became the gold standard.  But, as stated earlier, peer review has its share of problems.  If professors were to tell you what they really think, many of the faithful may even say that they no longer believe that peer review is fair or even necessary.  A few might even say that it hinders the advancement of excellent ideas that should otherwise be published.   

While there are attempts to discover the truth about peer review, the task is challenging [3].  In addition to the inertia of status quo, there are shared assumptions that favor peer review (even among the reluctant). After all, everyone who participates in peer review is part of a symbol that links them to a well-meaning process with benefits.  And yet, again, professors know that it is frequently a slow process that isn't always fair or even accurate and, occasionally, dangers, too, which raises the question:  “Do professors pay a price for not having confidence in other methods of publishing and exploring ideas (e.g., books, blogs, forums, workshops, and seminars)? 

Investigation of the peer-review system has failed to provide validation for its use.
– Jefferson, T. et. al. (2002). Measuring the Quality of Editorial Peer Review.
J Am Med Assoc. 287: 2786-90

Okay, peer review is not an exact science.  Editors understand this point and try to compensate by investing discretionary favors to reviewers who do an excellent job refereeing.  Ironically, the mentoring that editors do to encourage reviewers to evaluate manuscripts in a timely fashion without bias is not easy.  Some colleagues have defined it as a two-edged sword.  If editors work well with fair and ethical reviewers, the review generates hope and brings out the best.  But, if editors embrace reviewers who are prejudice or inept in their ability to do a fair and timely review, the process becomes weighed down with distraction and brings out the worst.   

The Role of the Editors
Quite simply, regardless of whether it is a print copy or electronic journal, the quality of the papers published depends on the integrity of the editors.  This understanding ought to be self-evident, but is it?  That said, all too often, peer review is not defined by the work of the editors.  But, many reviews would be void of accurate observations if it were not for the work of the editors.  That is why the role of the editor must be re-evaluated in light of their commitment and constant reminders to reviewers to stay focused, fair, and timely in their evaluations.   

This thinking goes back at least to 1731 when the Royal Society of Edinburgh published the first peer-reviewed collection of articles [5].  Editors developed the peer review process as a legitimate way to identify good science (i.e., worthy of publication).  Over time, editors (and the academy, in particular) developed the belief that all other articles published outside peer review are less worthy of professional recognition [6, 7].  And yet, the entire process depends on the role of the editors in locating good and fair reviewers.  Aside from being extremely expensive in many instances, Rennie [8] concludes that it is time consuming and prone to abuse.  Reviewers who are not knowledgeable or enthusiastic about research or ideas different from their own are not likely to be good reviewers [9].

Peer review is broken. It needs to be overhauled, not just tinkered with. – Kaplan, D.
(2005). How to Fix Peer Review. The Scientist. 19:1:10

Reviewers without the integrity to judge fairly their peers should not be part of peer review.  By ignoring this point, it allows for the review of papers in a non-objective manner.  Editors are aware that this makes the review unfair or biased towards status and gender.  For one thing, risky as it sounds and as popular as it is, few academics talk about peer review generating junk (as well as high-quality papers).  Why the peer review system remains the benchmark by which quality publications are determined [10] remains a mystery among some researchers? 

Lancet, pre-1976, did not implement peer review as they considered it unimportant.

Editor-Reviewed Journals
Shouldn't the process of publishing quality papers be more properly a function of editor's review? Similarly, shouldn't the quality of the papers published in non-peer reviewed but editor-reviewed electronic publications depend on the reputation of the editor?  If this thinking is correct, as many academics, researchers, and scientists seem to think, there shouldn't be any reason that authors can't build a reputation publishing outside the traditionally-defined peer review process.   

In other words, by supporting the editor-reviewed alternative, the publishing achievements of the authors in editor-reviewed journals (electronic or otherwise) will be valued as highly in promotion and tenure considerations as the conventional peer-reviewed journals.  After all, there should be room for both editor-reviewed papers and traditional peer-reviewed papers along with web-based publications.

"…Albert Einstein's revolutionary "Annus Mirabilis" papers in the 1905 issue of Annalen der Physik were not peer-reviewed. The journal's editor in chief (and father of quantum theory), Max Planck, recognized the virtue of publishing such outlandish ideas and simply had the papers published; none of the papers were sent to reviewers. The decision to publish was made exclusively by either the editor in chief, or the co-editor Wilhelm Wien…."

– http://en.wikipedia.org/wiki/Peer_review

The wikipedia quote should help clarify the lure of peer review.  Often it is more of a fantasy than a reality.  An editor-reviewed paper addresses the same concerns as the peer-reviewed paper, but despite this reality, it has rarely been the subject of discussion.  Because of this, academia is less fixed on the actual experience of the review process.  For example, they are not aware, as Arms [10] concluded, "Most of the high quality materials on the Web are not peer-reviewed and much of the peer-reviewed literature is of dubious quality."  The fact that "much of the peer-reviewed literature is of dubious quality" hits at the heart of an ever-widening problem (meaning, peer review itself is a questionable process).   

Professionalism in Exercise Physiology
In light of another truth is the question, "Why is there so little discussion about professionalism in exercise physiology?"  The answer may help explain why so few articles are submitted for publication in the ASEP electronic journal, Professionalization of Exercise Physiologyonline.  Perhaps, even worse is the fact that so few exercise physiologists seem to care about professionalism or why it is important in exercise physiology. 

Sports Medicine Influence
In other words, the belief that peer review is the only model for quality publishing is much like believing that sports medicine is the only organization for exercise physiologists.  The fact that they have, in effect, compartmentalized knowledge into discrete models is due to their willingness to be shaped by peer review and their beliefs about sports medicine and research.  However, left unchallenged, nothing changes and this isn't good.  False beliefs and out of touch realities are never good.  Thus, the larger objective is to avoid jumping to the wrong conclusions, such as:
 
  1. Exercise physiology can prosper and grow as a profession within the confines of sports medicine.
  2. It is a waste of time to publish an article on professionalism in a journal that is editor-reviewed and not peer-reviewed.
Both conclusions are wrong.  Whether exercise physiologists get this point or not depends on how they choose to define exercise physiology and quality control over publications.  Those who argue otherwise are wasting their time.  In general, the ability to think about the two erroneous conclusions would not be possible if exercise physiologists did not have the hope of something better. 

Hope is the pillar that holds up the universe. – Pliny the Elder

Message vs. Publication
It should be the exercise physiologist’s expectation that professionalism is at the center of every profession.  This is a good argument that should not lead anyone astray.  Professionalism instructs, impels, and demands its attention.  Above all, whatever the reason might be that exercise physiologists do not want to support their professional organization, they cannot, as it turns out, avoid it if they are to survive (singly or collectively). 

Regarding the second conclusion, no opportunity to publish is too small and no journal is too insignificant.  Transforming how exercise physiologists think requires being tough when necessary, and it requires the truth.  Both are absolutely central to the building of a new way to think about the purpose of publishing.  For example, while it is true that many promotion and tenure committees use published articles to evaluate professors, the ultimate purpose of the articles is to educate.  The truth of this point is unmistakable:  Articles should be published first for the message! 

But, once again, what is the problem?  Simply this: Academics do not publish in journals that are not peer-reviewed.  They say, "Why do it when the promotion and tenure committees aren't going to give me credit for the publication?  Why waste my time?"  In stark contrast, of course, the real issue is that there are other reasons for publishing.  Why not do something for the profession even if the author of the paper does not get something in return?  Failure to understand this point is often the case when different traditions interact and when professionalism is not taught. 

Profession vs. Peer Review
There is a lot of work that must be debated and discussed (e.g., code of ethics, board certification, and standards of practice to mention a few).  It isn't being done, however.  This is a marked contrast to physical therapy, nursing and other healthcare professions.  The astonishing by-product is the huge waste of time and small productivity in professional development.  Philosophically, exercise physiologists might consider that publishing their papers should not be separate from the process of managing their profession.

It is, indeed, discouraging to hear the exercise physiologist deny editor-reviewed journals and extol peer-reviewed journals (electronic or otherwise), as if to say they are separate and distinct activities.

Given that the ivory tower approach to scientific publishing has many problems [11], there is the question: "Since peer review is the publishing gatekeeper, shouldn’t it be clear to those who submit their work for publication that there are problems.  This why some researchers have established friendships with editors, that is, they think it will help them get around the problems.  Yet this is not generally discussed within tenure and promotion committees.  In short, some academics are incline to believe that peer review is a means to validating rejections without editors (or reviewers) having to state why than a process to help authors (or academics) in the development of a better paper.   

When this point is understood for what it is, particularly when reviewers act in an unfair and prejudice manner, it is as S. Rathey says [12], "Peer review is not now, and has never been, objective.  It is entirely human process and because of this, disagreement is inevitable."  Disagreements often result in negative consequences for those in academia.  This is why the integrity of the editor is so important, which agrees well with the comment by Robergs, editor-in-chief of JEPonline for the past 8 years [11]: "…when the process is unethical or unfair, there are significant human costs (e.g., failure to be promoted or even tenured)."   

There is also the cost in creativity whereby it is questionable to publish a paper that defines little or nothing of the "intuitive leaps, false starts, mistakes, loose ends, and accidents that actually cluttered up the inquiry…" [13].  Although the peer review is a well-meaning process, it has many problems (particularly, that of the reviewer taking sides).  The complaints do not stop there either.  Over the years, colleagues have been heard to say:
 
  1. The review is insulting and unhelpful.
  2. Did the reviewers even read the paper? 
  3. Aren't there any competent and open-minded reviewers?
It is past time for authors to reclaim governance of their own work.  The peer review process is not breaking down, it works but not at high level of competence.  This is evident by the publication of papers that do not meet academic standards.  To argue that good and bad does not exist in print copy peer-reviewed journals makes little sense.  Everyone in academia knows that bad research gets published all the time.  Here again, with editors who are on top of their game and have the time to invest, the process works.  But, to argue that online publications are less important than print-copy publications makes very no sense when the same editors may oversee both publication formats.  

PEPonline is Editor-Reviewed
Considering the number of electronic publications today, no one should dispute that the Internet is the future for groundbreaking ideas and research models.  It also regularly cuts the cost of publishing, as well as the bureaucratic and other structured means to publishing that delay getting new ideas and findings before colleagues and others in a timely fashion. Interestingly, with respect to the electronic journal, PEPonline, just recently concerns were raised that it is not a peer-reviewed journal.  Actually, it is but the "review" is carried out 100% by the editor, who is also the founder of the 1998 journal.  The role performed by the editor includes the following:
 
  1. Reject papers that are inconsistent with the purpose of the journal or target audience, thus filtering work that lacks quality or should be submitted to a different journal.
  2. Reject papers that are poorly written (e.g., content and readability) to ensure the quality of information [14]
  3. Accept papers that are appropriate for the journal, particularly papers that deal with ethics, professionalism, accreditation, board certification, and standards of practice.
  4. Edit papers that are accepted for publication in journal.
  5. Post papers on web page and link to the monthly issue.
  6. Archive papers for future reference.
In addition, the editor takes into consideration the following factors that should be self-evident with each submission.  For example, what is the value of the paper? How will it influence students and/or exercise physiologists in the field?  Are their important ethical implications that are timely and need publishing? Does the paper have a creative angle or new theme to an existing idea? How does the paper influence professionalism in exercise physiology?  Answers to these questions help the editor in the decision to publish or not.  Hence, the quality control of each publication is fully in the hands of the editor of PEPonline (as is the case with other journals, including many print-copy publications).  Consider the following examples:
 
  • An associate editor and 1 to 2 senior editors screen manuscripts at Ann Intern Med. About half are rejected without review because the screeners consider the material one of the following: too preliminary or of very low quality, topic matter not likely to interest the target readership, or the findings are already well-known (nothing new). -- Cindy Mulrow [15]
  • In my experience screening rather than reviewing all papers is common practice in major journals that employ several in-house editors, but much rarer among journals edited by part-time editors or those in which the editorial board does most of the reviewing. But I'd be happy to be corrected by editors. You might also be interested to note the BMJ's policy of screening papers based only on their abstracts—see Groves T and Abbasi K. (2004). Screening Research Papers by Reading Abstracts. BMJ. 329:470-471 for details. -- Liz Wager Princes Risborough, UK [15]
PEPonline and Professionalism
As editor of PEPonline, I wonder why exercise physiologists avoid publishing in the electronic journal.  Naturally, given the steps outlined above, there is a relatively low overall rejection rate.  This may not come as a "bombshell" but, yes, there are papers that have been rejected, given their unlikely chance of benefiting the profession.  This decision is often linked to the poor quality of the papers as well.  Observers would be well advised to be skeptical of the claim that anyone can publish in the journal.  This is simply not the case.  To that end, it should be clear to open-minded professionals, especially those who understand the need to think outside-of-the-box that the journal exists to support professionalism in exercise physiology.   

The electronic journal exists solely for the benefit of exercise physiology.  Because professionalism is not a hot area in exercise physiology and because some exercise physiologists have erroneously concluded that PEPonline does not review articles for publication, the number of articles submitted by exercise physiologists is small.  In contrast, having created the electronic journal, and having an intense interest in professionalism, as editor, I have written and published articles on a monthly basis since 1998. The confusion about what is credible in PEPonline in light of exercise physiologists not having an adequate background in professionalism may be exacerbated by exercise physiologists who are not interested in supporting the American Society of Exercise Physiologists [16].   

Editor, Quality Control, and Big Shoulders
Without question, as editor, I am the quality control peer who evaluates and makes judgment as to whether a paper should be published or not.  The focus is on the content of the paper.  Is it appropriate for the PEPonline audience? Are the conclusions plausible, and is the paper written clearly?  And, clearly, just as researchers can make mistakes and, in particular, exaggerate the worth of their findings, editors can find mistakes or fail to detect mistakes in the author's work.  In addition to determining whether the paper is suitable for PEPonline, as editor, I consider the author's presentation (i.e., is it understandable) and scholarship (i.e., is it current and accurate).  Are the arguments and inferences sound?  Are the references appropriate for the content and length of the paper?  All of this and more is part of any editor-reviewed journal. 

Callaham and colleagues [17] said in 1998, "the onus to keep the peer review process in good working order falls squarely on the shoulders of the editors."  In other words, even with the peer review system, editors are critical to a fair and balanced process.  This is even more important, given that the peer review system has room for improvements in fairness and consistency [18].  No doubt this is why Relman's comments [19] about peer review in scientific journals are such an eye opener: 

"…the final decision should not be made by the references.  References are consultants, not arbiters of publish ability.  Peer review is not a straw vote among references but rather a source of information and technical advice to the editors, who make the final decision.  The editors consider the opinions of several people, not just one reference, and it is the editor's responsibility to weigh the possibility of bias in the advice they receive."  [italics and bold added]

At some point in the future, when exercise physiologists wake up to the importance of professionalism, and when the number of papers becomes too much for the editor, the use of reviewers will help determine whether a paper is an original, significant piece of work.  Until then, as editor of PEPonline, the character and focus of the journal exists with me.  My job is no different from the standard peer review, that is, "…determine appropriateness and relevance and to search for error and, secondarily, to detect fraud, conflict of interest, and plagiarism…" [20].

Hence, having selected the work that is believed to be suitable for the journal and knowing that the published papers must be of high standards of presentation and publishing, exercise physiologists who are interested in publishing on professionalism will be encouraged to submit their work to the editor.  Lastly, the journal exists because exercise physiologists need it even though they haven't figured it out yet.

 

References
 
  1. Brown, T. (2006). Sense about Science. Peer Review and the Acceptance of New Scientific Ideas. [Online]. http://www.senseaboutscience.org.uk/pdf/PeerReview.pdf
  2. Boone, T. (2001). Journal Impact Factor: A Critical Review. Professionalization of Exercise Physiologyonline. 7:1 [Online]. http://www.asep.org/asep/asep/journalIMPACTfactor.html
  3. Benos, D. J., Bashari, E., Chaves, J.M., Gaggar, A., Kapoor, N., LaFrance, M., Mans, R., Mayhew, S.M., Polter, A., Qadri, Y., Sarfare, S., Schultz, K., Splittgerber, R., Stephenson, J., Tower, C., Walton, R.G., and Zotov, A. (2007). The Ups and Downs of Peer Review. Advances in Physiology Education. 31:145-152.
  4. Goldbeck-Wood, S. (1999). Evidence on Peer Review – Scientific Quality Control or Smokescreen? – Education and Debate. British Medical Journal. 318:44-45.
  5. Kronick, D.A. (1990). Peer Review in 18th-Century Scientific Journalism. Journal of the American Medical Association. 263:1321-1322. (abstract).
  6. Kassirer, J.P. and Campion, E.W. (1994). Peer Review: Crude and Understudied, but Indispensable. Journal of the American Medical Association. 272:96-97.
  7. Goldbeck-Wood, S. (1998). What Makes a Good Reviewer of Manuscripts? British Medical Journal. 316:86.
  8. Rennie, D. (1998). Peer Review Prague. Journal of the American Medical Association. 280:214-215.
  9. Silversides, A. (2000). Respect of One's Peers. University Affairs. June/July.
  10. Arms, William Y. (2002). Quality Control in Scholarly Publishing on the Web. Journal of Electronic Publishing. 8:1 [Online]. http://www.press.umich.edu/jep/08-01/arms.html
  11. Robergs, R.A. (2003). A Critical Review of Peer Review: The Need to Scrutinize the "Gatekeepers: of Research in Exercise Physiology. Journal of Exercise Physiologyonline. 6:2 [Online]. http://www.asep.org/asep/asep/EDITORI1.doc
  12. Rathey, S. (1994). Electronic Journals and Peer Review: Perils and Promises. http://www.cs.ubc.ca/spider/rathie/elecpub/paper3.html
  13. Merton, R.K. (1968). Social Theory and Social Structure. New York, NY: Fee Press. p. 4.
  14. Macrina, F.L. (2005). Authorship and Peer Review.  In Scientific Integrity: Text and Cases in Responsible Conduct of Research. (3rd Edition). Edited by F.L. Macrina. Washington, DC: ASM.
  15. World Association of Medical Editors. (2005). Rejecting Manuscripts Without Review. [Online]. http://www.wame.org/wame-listserve-discussions/rejecting-manuscripts-without-review
  16. American Society of Exercise Physiologists. (2007). Home Page. [Online]. http://www.asep.org/
  17. Callaham, M.L., Baxt, W.G., Waeckerle, J.F., and Wears, R.L. (1998). Reliability of Editors' Subjective Quality Ratings of Peer Reviews of Manuscripts. Journal of the American Medical Association. 280:3:229-231.
  18. Ernst, E. and Resch, K.L. (1994). Reviewer Bias: A Blinded Experimental Study. Journal of Laboratory Clinical Medicine. 124:2:178-182.
  19. Relman, A.S. (1990). Peer Review in Scientific Journals – What Good Is It? Western Journal of Medicine. 153:520-522.
  20. Stanley, R.J. (2005). To Peer Review or Not to Peer Review: That Seems to be the Question. American Journal of Roentgenology. I5:1101. [Online]. http://www.ajronline.org/cgi/content/full/185/5/1101

 



Copyright ©1997-2007 American Society of Exercise Physiologists   All Rights Reserved.