Professionalization of Exercise Physiologyonline    


         ISSN 1099-5862   Vol 7 No 1  January 2004 
 






 
 












 

    Editor-in-Chief
    Tommy Boone, PhD, MPH, MA, FASEP, EPC


 

Journal Impact Factor:  A Critical Review

Tommy Boone, PhD, MPH, MA, FASEP, EPC
Professor and Chair
Director, Exercise Physiology Laboratories
The College of St. Scholastica
Duluth, MN 55811   
“If you want to truly understand something, try to change it.”  -- Kurt Lewin
It is widely recognized that “impact factors” are used to rank and, therefore, identify the so-called best journal to publish one’s work.  This is true today even though most researchers really don’t understand that the Science Citation Index (SCI) is frequently misunderstood, if not used indiscriminately.  After a careful examination of the cumulative impact data, it is clear that the quantitative influence reflects more than the assumed quality of the journal [1]. From a historical point of view, Gross and Gross suggested in 1927 that scientific journals should be ranked [2].  In 1955, Garfield stated that, “reference counting could measure impact” [3].  However, it wasn’t until the early 1960s that such thinking led to the development of the SCI, which subsequently published in 1975 the Journal Citation Reports (JCR) [1].  Since then, according to Garfield [1], bibliographic “impact factor” has taken several interesting twists, for example: 
“The most used data in the JCR are impact factors – ratios obtained from dividing citations received in one year by papers published in the two previous years.  Thus, the 1995 impact factor counts the citations in 1995 journal issues to ‘items’ published in 1993 and 1994.  I say ‘items’ advisedly.  There are a dozen major categories of editorial matter.  JCR’s impact calculations are based on original research and review articles, as well as notes.  Letters of the type published in the BMJ and the Lancet are not included in the publication count.  The vast majority of research journals do not have such extensive correspondence sections.  The effects of these differences in calculating journal impact can be considerable.” [1, p. 2]
The counting of “items” per se is a serious source for misplaced use of the “impact factor."  Garfield [1] states that absolute citation counts “…preferentially give highest rank to the largest or the oldest journals.”   Journals with the highest impact factor may not be the very best journals, although the JCR is used to rank, evaluate, categorize, and compare journals.  Numerous factors go into increasing a journal’s impact, including: (a) subject matter such as dermatology vs. molecular biology; (b) papers that cite all of the relevant literature; (c) time required to review manuscripts; and (d) other subtleties such as letters, discussions, proceedings, notes, news stories, and editorials published in journals [4]. 

Another condition that favors having a high impact factor is whether the journal is a print-copy publication in contrast to an electronic journal.  For example, the Journal of Exercise Physiologyonline and the Professionalization of Exercise Physiologyonline do not have an impact factor.   Recently, a colleague from another university said:  “There isn’t any incentive for me to publish with JEPonline.  Tommy, my administrators require published articles in journals with high impact factors.  If I want tenure, that is what I must do.”  In general, the statement is consistent with the thinking of many administrators.  But, of course, there are many inconsistencies across the university system regarding what is important for promotion and tenure.   Whether exercise physiologists should be concerned when submitting a manuscript to a journal, electronic or otherwise that does not have an impact factor needs further study.  In fact, given the information in this article, it is unfortunate that exercise physiologists would submit their research to only journals with an impact factor. 

Garfield [4, p. 3] states that, “As a general rule, the journals with high impact factors are among the most prestigious today.”   This perception may be wrong.  Clearly, the notions of influence or importance are subjective if not biased.  The impact factor, as Hoeffel [5] has stated, “…is not a perfect tool to measure the quality of articles….”   In fact, the impact factor does not define the quality of scientific journals within a particular field.  A journal’s impact factor is more complicated than most researchers realize.  Finding an objective measure of the impact of different journals is not in itself a bad idea.  The “citation analysis” approach may not be the best method, however.  Just because a researcher cites a journal article does not automatically mean that the journal itself is important.  It may simply indicate that the article in the journal is important to a research’s subject matter.  Thus, the assumed influence or impact of the journal is not as much an objective measure of the journal as it is a particular topic of research. 

For example, although a journal’s impact factor is based on the numerator (which is the number of citations in the current year to any items published in a journal in the previous 2 years) and the denominator (which is the number of substantive articles published in the same 2 years) [4], there is wide variation from journal to journal in what constitutes the numerator and the denominator.  Obviously, both are important in the calculation of the impact factor.  The process tends to highlight older journals more so than journals that publish more manuscripts [1, 6].  There is room for considerable error and inflation by the inclusion of items such as letters to the editor, editorials, book reviews, and news reports as research papers or review articles [6].  These confounding variables indicate that the impact factor is not an objective measure of the influence of journals in a particular field.  Informed and careful use of the impact factor is essential to avoiding administrative mistakes.  Hence, contrary to the view by Hopkins [7] who believes in the use of the impact factor in making administrative decisions, it is clear that it should not be among the primary considerations used by exercise physiologists to improve their chances of promotion and/or tenure. 

If librarians want to use the JCR as a tool for management of library journal collections, then so be it.  However, jumping to ill-formed conclusions about the quality of published articles and/or journals is something altogether different.  The idea that publishing in a journal with a high impact factor is better than publishing in a journal with a small or no impact factor does not make sense.  It is the same unfortunate notion that “bigger is better”.  Exercise physiologists should be encouraged to publish their work when and where possible, particularly if the journal provides the timely opportunity to position the article in the scientific literature.  There really isn’t any serious advantage to career development by holding out to a so-called more influential journal.  There are many different ways to demonstrate scholarly work and/or publication that are just as objective measurements of overall quality, if not better measurements of the researchers’ work.  It is incorrect to conclude that the quality of research is somehow inferior when published in journals with small or no impact factor.  There is room for significant improvement in ranking print-copy and, potentially, electronic publications whereby the JCR data are used by advertisers to enhance the marketing of a product [8].

The first improvement might be a complete re-evaluation of the use of the bibliographic “impact factor”.  Recently, Porta [9] pointed out that Garfield [10-12] argued early on that the impact factor “…is often not the scientometric indicator of choice” [9].  The impact factor “average” is a “…highly skewed distribution (often, 85% of citations received by a journal are actually received by about 15% of the articles it published)” [9, 13, 14].  Even then, Jefferson [15] concludes that, while the cited journals are read more often than journals with lower impact factors, there is no evidence that the journals with higher impact factors are of higher quality.  Jefferson and colleagues have also questioned the measurement of quality of the editorial peer review process, stating that there is no internationally accepted definition of quality [16].  Jefferson further concludes that:  “…far from assessing the quality of scientific production, which underscores the altruistic nature of the scientific enterprise, citation rates are being used to apportion research money….” [15].  It is disconcerting to say the least that the impact factor may also be directly linked to the researchers’ pockets [17].  If so, Jefferson’s view of the impact factor as “a circulation business indicator” is likely to be used to influence healthcare research and decisions about health and wellness that may have negative consequences in the public sector.

This kind of thinking is highly unfortunate.  Researchers should be able to publish their work where possible, regardless of the so-called usefulness of a journal.  And, all journals should be given the same professional respect for publishing quality articles.  A journal’s impact factor should not be used to assess a faculty member’s publications for the purpose of accessing a job, getting tenure, or promotion.  It just doesn’t make any sense, given the vast number of factors unaccounted for in the assumed “quality analysis” of an article and/or journal.  In other words, “more is not always better” and, in this case, it is a gross violation of publication and/or scientific accountability with the notion being that only good work is published in journals with high impact factors.  Researchers who submit their work to a journal with no impact factor must, therefore, be drawing from the bottom of the barrel and, for reasons without any justification, the submission and publication must be a waste of time.  However, this is not in fact true. 

The quality of published manuscripts is defined by the integrity interwoven into the research and/or writing process.  It is not defined by impact factors per se.  Perhaps, what is needed today is increased adaptive capacity to help members of different professions respond quickly and intelligently to needed change.  If so, a whole new decision-making process must evolve to allow today’s researchers to act and to evaluate the results of their work, instead of relying on a model more than 50 years old.  Exercise physiologists, in particular, should assess the results of their work and the work of their colleagues by reading the published work and, then, critically reflecting on the merits of the published product.  This understanding is within their grasp since peer-reviewers for exercise physiology research come from within the manpower of the evolving profession of exercise physiologists.  

Although this proposal may appear radical to the exercise physiology community, before they can learn to be leaders as healthcare professionals, they must learn to think for themselves.  Academic programs in kinesiology must become exercise physiology programs, exercise physiologists as technicians must become inspired professionals and, as many have noticed with the founding of the American Society of Exercise Physiologists, exercise physiologists must learn how to recover from their lack of an exercise physiology philosophy and begin with questions such as:

  • Why is the impact factor important to publishing, promotion, and tenure?
  • Why should the widely believed notion that ranking journals is so important as to be unquestioned and, therefore, accepted as dogma?
  • What experiences are important in explaining why exercise physiologists publish in journals with the highest impact factors?
  • Is the content from journals without impact factors important to exercise physiologists as healthcare professionals?
  • What role does publishing articles play in the professional development of exercise physiology?
  • What can ASEP do to encourage new thinking about the quality of journals and articles?
  • How can exercise physiologists help others understand that journals and articles come in every size, shape, and disposition, and that each deserves the respect of the leaders, researchers, and writers in the discipline?
  • What can the ASEP leadership do to encourage risk taking, especially in developing an overriding philosophy that “to publish is to do something good”?
  • How can exercise physiologists be encouraged to speak up and set the direction in the professional development of exercise physiology?
There is growing evidence that the compulsion to use impact factors to identify quality journals and articles is seriously flawed.  With all the intellect that the world has, it is a wonder that researchers have fallen victim to nurturing such a bad idea.  Perhaps, it is true that some researchers have gotten too caught up in the apparent correlate of the “impact factor” and NIH grants, big money, elaborate laboratories, and other extraordinary and potentially troubling notions that research is just another form of competition.  Is it possible that many have forgotten why they are doing research?  It is common knowledge that poets and writers talk about the freedom to do their work, to publish, and to even make mistakes in being creative.  Why can’t researchers just enjoy doing research and publishing their work in any journal that will publish it?  It seems so pointless to hear adult men and women saying, “my journal is better than your journal” or “my article was cited more than your article.” 

Liu [18] is correct about one thing:  “…the citation number of individual papers should be adjusted according to discipline to improve on an imperfect but widely used indicator of research quality.”  However, this in itself will predictably fail to correct the problem, which is largely related to the single numerical summary measure  [19] And, as Liu [18] and Jefferson and colleagues [20] highlighted in their published works, there is no solid empirical evidence on the effectiveness of traditional peer review.  In fact, recently, Robergs [20] published an extensive article about the issues and concerns that associate with the traditional peer-review process.  It is a fearless and optimistic article about real problems in publishing that few researchers seem to have the courage to question.  In a way, to begin to understand the problem with impact factors is to understand that peer-review needs critical assessment as well. 

This article is a snapshot of the impact factor “problem” as it is presently used.  Researchers (not librarians, information scientists, publishing houses or companies, to mention a few) ought to be the primary players in deciding the actual value of the quality of their research.  The problem is that only a few researchers, exercise physiologists included, seem to understand that research, as is presently understood, is held captive in thinking that is decades old.  It will continue down the same path unless the entrepreneurs of the 21st century demand their publishing rights.  But, first, they must look deeply and critically at answering the following questions.  Why aren’t the researchers in charge of publishing?  Secondly, if researchers continue to distance themselves from the right to publish in the journal of their choosing, what are the social and professional implications?

“Leaders…can conceive and articulate goals that lift people out of their petty preoccupations, carry them above the conflicts that tear a society apart, and unite them in pursuit of objectives worthy of their best efforts.”  -- John W. Gardner, No Easy Victories 
What is presently needed in academics is strong leadership to argue the case that the Institute of Scientific Information (ISI), the database publishing company that publishes the SCI and the impact factor, has crossed the line of common sense.  In particular, since journals are moving from print-copy to electronic [22-25], librarians will have less concern about the financial costs that typically associate with hard copies of the journals in the library.  So, why not discontinue the impact factor since it is misused in determining the “…importance of individual researchers, research programs, and even the institution hosting the research” [26]?  Hecht and Hecht, and Sandberg [26] recommend abolishing the impact factor.  It should not be used to assess the relevance of research, and it should not be considered when making decisions about promotion and funding research proposals [27].  In short, the impact factory should be used with extreme care due to the many factors that influence citation rates.  Golder [28] points, for example, to the bias towards English language journals compared to journals in other languages. 

Just as many medical researchers submit their manuscripts to generalist medical journals (like the New England Journal of Medicine and the Lancet) to access the largest recognition of their published work versus publishing in specialist medical journals with a smaller audience [29], exercise physiologists often do exactly the same thing.  It is not necessarily wrong, but it doesn’t make it right either.  Publishing in a journal with a lower profile is not popular among many researchers yet, according to Sloan [29], it isn’t unusual for dentists to prefer specialist journals to generalist journals.  This kind of thinking is revolutionary.  Perhaps exercise physiologists should also question the value of using impact factors in the assessment of publication quality.  It is an old “numerology” system that is simply misused and misconstrued.  It is also an inappropriate measure of scientific quality. 

In sum, the impact factor is ill conceived, intellectually and technically flawed, and misleading effort to assess the academic worth of a paper  [30].  And, if that’s not enough, Walter and colleagues [30] point out that the impact factor (IF) has now “…spawned a range of flawed offspring, including ‘Scope-adjusted IF’, ‘Discipline-specific IF’, ‘Journal-specific influence factor’, ‘Immediacy index’ and ‘Cited half-life’”.  Exercise physiologists should reflect critically on the issues presented in this article.  This article is by no means complete in its analysis, it is nonetheless reasonably complete to help with the decision to make a clean break from the use of impact factors.   
 

References
1. Garfield, E. (1996). Fortnightly Review: How can Impact Factors be Improved?  BMJ. 313:411-413 (17 August). [Online]. http://bmj.com/cg/content/full/313/7054/411?ijkey=QHqq/zTqoo0Qk 
2. Gross, P.L.K. and Gross, E.M. (1927). College Libraries and Chemical Education. Science. 66:385-389.
3. Garfield, E. (1955). Citation Indexes for Science: A New Dimension in Documentation Through Association of Ideas. Science. 122:108-111. 
4. Garfield, E. (1999). Journal Impact Factor: A Brief Review. CMAJ. 161:1-7 (19 October). [Online]. http://www.cmaj.ca/cgi/content/full/161/8/979?ijkey=nr8.IXo1aXxvc
5. Hoeffel, C. (1998). Journal Impact Factors. [letter]. Allergy. 53:1225.
6. Joseph, K.S. and Hoey, J. (1999). CMAJ’s Impact Factor: Room for Recalculation. CMAJ. 161:1-5 (19 October). [Online]. http://www.cmaj.ca/cgi/content/full/161/8/977?ijkey=.LEQZ2kRXqDUE
7. Hopkins, W.G. (2000). Impact Factors of Journals in Sport and Exercise Science. Sportscience. 4:1-6 [Online]. http://www.sportsci.org/jour/0003/wgh.html
8. British Medical Journal. (2003). Quality of Impact Factors of General Medical Journals. [letter]. [Online]. http://bmj.com/cgi/content/full/326/7383/283
9. Porta, M. (2003). Quality Matters – and the Choice of Indicator, Too. BMJ.COM. [letter, February 6]. [Online]. http://bmj.com/cgi/eletters/326/7383/283
10. Garfield, E. (1985). Uses and Misuses of Citation Frequency. Current Contents Life Sciences. 43:3-9.
11. Garfield, E. (1986). Which Medical Journals Have the Greatest Impact? Ann Intern Med. 10:313-320.
12. Garfield, E. (1987). Prestige Versus Impact: Established Images of Journals, Like Institutions, are Resistant to Change. Current Contents Life Sciences. 38:3-4.
13. Porta, M. (1996). The Bibliographic ‘Impact Factor’ of the Institute for Scientific Information, Inc.: How Relevant is it Really for Public Health Journals? J Epidemiol Community Health. 50:606-610.
14. Seglen, P.O. (1992). How Representative is the Journal Impact Factor? Research Evaluation. 2:143-149.
15. Jefferson, T.O. (2003). Quality of Impact Factors of General Medical Journals – PRAVDA Wins Hands Down. BMJ.COM. [letter, February 8]. [Online]. http://bmj.com/cgi/eletters/326/7383/283
16. Jefferson, T.O., Wager, E., and Davidoff, F. (2002). Measuring the Quality of Editorial Peer Review. JAMA. 287:2786-2790.
17. Jimenez-Contreras, E., Delgado-Lopez-Cozar, E., Ruiz-Perez, R., and Fernandez, V.M. (2002). Impact-Factor Rewards Affect Spanish Research. Nature. 417:898.
18. Liu, J.L.Y. (2003). The Assessment of Research Quality Using a Combination of Approaches. BMJ.COM. [letter, February 19]. [Online]. http://bmj.com/cgi/eletters/326/7383/283
19. Smith, R. (1998). Unscientific Practice Flourishes in Science.  BMJ. 317:1036-1040.
20. Jefferson, T.O., Alderson, P., Davidoff, F., and Wager, E.  (2003).  Editorial Peer-Review for Improving the Quality of Reports of Biomechanical Studies (Cochrane Methodology Review). In The Cochrane Library, Issue 1.  Oxford: Update Software.
21. Robergs, R.A. (2003).  A Critical Review of Peer Review: The Need to Scrutinize the “Gatekeepers” of Research in Exercise Physiology.  Journal of Exercise Physiologyonline. 6: (2)i-xiii. [Online]. http://www.asep.org/asep/asep/EDITORI1.doc
22. E-J Miner. (2003).  Electronic Journal Miner. [Online]. http://ejournal.coalliance.org/browseTitles.cfm?filter=P
23. DOAJ. (2003). Directory of Open Access Journals. [Online]. http://www.doaj.org/alpha/
24. The University of British Columbia Library. (2003). Freely Available Electronic Journals. [Online]. http://toby.library.ubc.ca/ejournals/unrestrictedlist.cfm
25. Open Access Now. (2003). Campaigning for Freedom of Research Information. [Online]. http://www.biomedcentral.com/openaccess/archive/?page=home&issue=2
26. Hecht, F., Hecht, B.K., and Sandberg, A.A. (1998). The Journal “Impact Factor”: A Misnamed, Misleading, Misused Measure. Cancer Genet Cytogenet. 104:77-81.
27. Bloch, S. and Walter, G. (2001). The Impact Factor: Time for Change. Aust N Z J Psychiatry. 35:563-568.
28. Golder, W. (2000). Wer kontrolliert die Kontrolleure? Zehn Thesen zum sogenannten Impact Factor. Onkologie: International Journal for Cancer Research and Treatment. 23:73-75.
29. Sloan, P. and Needleman, I. (2000). Impact Factor. British Dental Journal. 189:1-2. [Online]. http://www.eastman.ucl.ac.uk/~pdarkins/iceph/Impact%20factor.pdf
30. Walter, G., Bloch, S., Hunt, G., and Fisher, K. (2003). Counting on Citations: A Flawed Way to Measure Quality. MJA. 178:280-281.
 
 
 

Return to top of page