CriticalReview

A COMMON AND PREVALENT misconception held about science in society is that it proves absolutes. However, the results of scientific research are based upon inferential statistics. Since testing an entire population is generally too costly and time consuming, inferential statistics is used in science to provide researchers with a value that represents the amount of confidence associated with a sample that is being generalized to a population. In other words, research has the ability to spark interest in a new area, strengthen an existing theory or lend further support to reject a particular theory. Without testing an entire population, absolute certainty cannot be obtained. However, one can determine the likeliness of the occurrence of a certain event that is extremely useful in real life only when the research is performed, analyzed and reported accurately. Moreover, it is the role of the researcher to prove to the audience that his or her belief has more validity than other beliefs. If that is done, it becomes the common belief. But, since the previously mentioned belief of science as conclusive prevails in society, more often than not, scientific research is left unanalyzed and unquestioned.

Therefore, the need for skepticism among professionals in reading research literature is critical. Our role of analyzing and critically evaluating research studies is crucial in order to locate weaknesses in studies that may mislead the reader to an incorrect analysis of the research. As professionals who base their practices and care on research, it is imperative for the sake of our clientele that we are able to identify deficient studies and question their conclusions.

Certain aspects of the studies yet to be mentioned may in fact be considered ethical issues in the realm of health research. One aspect is the suspicion of incomplete reporting of data. An assumption might be that certain researchers only reveal the data that they want the audience to see as opposed to reporting everything that was measured. Another facet of research that is troublesome is the continuous use and presence of ambiguous measurements. The fear is that much research is centered on measurements and/or assumptions that are not valid and therefore improper to use as a means of establishing relationships. Another aspect that is often absent in research is the practical significance of a study. Many researchers focus their discussion of the results primarily around the proposed alternative hypothesis, but fail to consider the possible benefits of other data collected in the study.

An analysis of numerous primary research articles, all testing the effects of relaxation techniques on aerobic exercise revealed a plethora of common flaws in research in all areas of science. Research on this subject completed by Benson and colleagues (1) in 1978, found in the Journal of Human Stress, laid the foundation for most of these studies. However, after analyzing later research that cites, tests and/or uses Bensons’ techniques, such as the “Benson Relaxation Response” (BRR) (1), one finds that most of this research is incomplete or improperly presented, leaving the audience to critically analyze for themselves the results and significance of the report. As a result, a gap remains in the research area of the effect of psychophysiologic self-regulation on aerobic exercise. Few researchers have shown solid support to reinforce the authors’ theory of elicitation of the relaxation response decreasing oxygen consumption during exercise, yet they fail to question his findings. Even articles being published up to the year 1999, such as the data released by Caird and colleagues (2) in Medicine and Science in Sports & Exercise refer to and compare their findings to Benson’s relaxation techniques and yet fail to question the validity of the procedures.

In 1987 in the Journal of Cardiopulmonary Rehabilitation, Hayward and colleagues (3) researched the physiologic effects of BRR during submaximal aerobic exercise in patients with coronary artery disease. It was found that there was no decrease in heart rate (HR), systolic blood pressure (SBP), or rate pressure product (RPP) during exercise. It is unknown as to whether or not other variables were measured and recorded during the study. Regardless, they were not reported or considered upon analysis. Therefore, the researchers made assumptions, based on only three variables, that especially with regard to coronary artery disease patients, a certain “exercise threshold” may exist beyond which the effects of the relaxation response cannot be obtained. This assumption appears to have been made under incomplete analysis. In fact, it seems reasonable that the research should contain 8-10 dependant variables. For example, it would have been beneficial if this study had included such physiological responses as tidal volume (Vt), expired ventilation (Ve), oxygen consujmption (VO₂), and respiratory exchange ratio (RER).

The same study as already mentioned by Benson and colleagues (1) also fails to offer the reader a complete data set. However, in this case, unlike the Caird study, the measurements were most likely taken, but not reported. This assumption is based on the suspected intent of the researcher due to the title of the article: Decreased VO₂ consumption during exercise with elicitation of the relaxation response. This title appears to be leading the reader, and possibly serves as an advertisement that sells itself and the author before the reader even reads the article. Hence, the reader enters the article with a specific mindset. Is it possible that a more complete data set was not reported in this case because the data did not yield significant results? Hence, for the sake of prominence and acclaim, the reader is left with the feeling that the non-significant results were not reported.

Much research uses methods or measurements that are not valid or fit for the study and then formulate assumptions that are inappropriate for the measurements. Referring to the study by Hayward (3) once again, the possible use of an improper method is evident here. The method of this study was based on a study done by Gervino and Veazey on young, apparently healthy, women. The sample that Hayward used consisted of men between the ages of 40 and 65. To expect replicable findings would be unlikely with two such heterogeneous samples and is essentially a useless comparison. Hayward addresses this issue, yet proceeds to suggest that the previously mentioned “exercise threshold” explanation for his results is more adequate than the dissimilarity of samples. There are many physiological, biomechanical and psychological aspects that need to be considered before making such assumptions.

A study in the Journal of Sports Sciences in 1989 by Messier and Cirillo (4) poses more than one problem with regard to ambiguity in measurements. This study was measuring running technique and went so far as to suggest an ideal running technique. Yet the argument remains as to what is the technique. Physiologically, it might be argued that the best running technique would be the one that consumes the least amount of oxygen. The authors claim that their technique consumes the least amount of oxygen. More practically, however, one might suggest that regardless of technique, the one who wins the races and comes out on top is the one with the best running technique. In addition to the ambiguity of the term “running technique”, another problem is encountered with regard to the use of rate of perceived exertion (RPE) as a means of measurement. There is evidence in literature that suggests a correlation between RPE and relative oxygen consumption. However, the true validity of this measurement is very questionable and using it to determine oxygen consumption is sketchy. Interestingly enough, the authors go so far as to mention the impossible nature of isolating mechanical variables of primary importance in running technique, which seems to be perhaps a more valid means of measurement than RPE.

A very important issue in research that is so often overlooked is the practicality of research. This idea tends to get lost as professors rush through research studies to gain tenure as well as fame and popularity. Practicality needs to be appreciated and discussed more by the researchers conducting the study. In the previously mentioned study (4), the conclusion is made that verbal and visual feedback are effective means of modifying running style in female novice runners. However, the relationship between modifications in running style and improved running economy and perceived exertion is unknown. The researchers fail to mention the practical significance, if any, of this one finding. It would be beneficial to the reader to examine what piece of information might be helpful to obtain from the research performed. Perhaps, they could have made the connection that the ideal running technique they propose might be more beneficial on the wear and tear of joints in the body. Researchers need to be able to offer an explanation as to why it is important for an athlete to modify their running style.

In the article published in the Journal of Exercise Physiologyonline in 1998, Boone and DeWeese (5) assign practical significance to the research, even though the null hypothesis was retained. The major finding in the study was that the elicitation of the relaxation response during submaximal treadmill exercise did not result in a significant decrease in VO₂. The results of the study indicated the importance of psychophysiologic self- regulation during exercise on respiratory and myocardial variables. The authors explained that there is evidence that a reduction in RPP has a positive influence on the work of the heart. Essentially, this may be more practically important than the measured VO₂ with regard to exercise tolerance.

Another critical piece of research that needs greater attention upon analysis of research is the results section of a study. Many times readers overlook this section, particularly the statistics, and in turn are not aware of poor logic, misleading statements or even mistakes in the statistically based assumptions. The following studies testing orthotic devices make conclusions that appear to be actually incorrect upon closer examination of the statistical results.

Consider the data published by Maltais (6) and colleagues in Medicine and Science in Sports & Exercise in 2000. Upon notice of the directional title, the "Use of orthoses lowers the O₂ cost of walking in children with spastic cerebral palsy," the reader should immediately become skeptical of the article and remain unbiased. The title makes it clear that the authors want the reader to believe that a significant and practical result was found. In reality, a significant decrease in oxygen consumption was found. However, the decrease is not consistent with logical thinking.

The authors’ results are based on walking efficiency (VO₂, l/min) for six conditions: (a) walking at a speed of 3 km/h without ankle foot orthoses (AFO); (b) walking at a speed of 3 km/h with AFO; (c) walking at comfortable walking speed (CWS) without AFO; (d) walking at CWS with AFO; (e) walking at 90% of fastest walking speed (FWS) without AFO; (f) walking at FWS with AFO. Statistical analysis found that the oxygen cost for walking was significantly less only for walking at an extremely low speed of 3 km/h and at a very high speed of 90% of the subject’s fastest walking speed. There was no significant difference between the use of an AFO and absence of the orthotic device at normal walking speeds! In real life, interest lies only in the results of normal walking speed. It is not logical to think that children with cerebral palsy would either essentially expend more energy to save energy or expend so much less energy that their ability to keep up with normal activity is impossible. The study is simply not relevant in the practical sense. Therefore, the title that the authors assigned to this study suggests much greater importance and practical significance than it is worthy of.

Another study published in the Archives of Physical Medicine and Rehabilitation journal in 1996 by Abdulhadi and colleagues (7), provides a statistically based conclusion that is essentially incorrect. The conclusion reported that when either a half-inch or a one-inch shoe-lift was used on the contralateral foot of an immobilized extended knee, there were statistically significant reductions in oxygen consumption. Closer examination of the statistical analysis reveals ambiguity in comparisons made by the researchers. In this study, average walking efficiency (VO₂, ml/kg/m) at CWS for four conditions was measured: (a) normal walking with no knee immobilized and no shoe-lift; (b) knee immobilized, no shoe-lift; (c) knee immobilized and one half-inch shoe-lift; (d)knee immobilized and one-inch shoe lift.

The authors claimed the oxygen consumption during walking with the knee immobilized unilaterally in extension was significantly greater, by 20%, that of normal walking. The problem arises when the authors claimed to have found a statistically significant reduction in oxygen consumption when the half-inch shoe-lift (11% above normal walking) and the one-inch shoe-lift (12% above normal walking) were applied. However, the reader is left to speculate as to why the authors are comparing the shoe-lift conditions to normal walking as opposed to the condition where the knee is immobilized without a lift, since the latter case is congruent with what the conclusion states. The question is whether a shoe-lift improves walking efficiency for those with an immobilized extended knee. Clearly, this study comes across as problematic. It is not acceptable research to be used in supporting or refuting other orthotic device theories dealing with a similar issue.

The two previously mentioned studies have serious implications in the clinical realm. Many students and professionals read articles without a deeper critical consideration of the statistics and methods and, in turn, integrate the research findings into their client care plans. The ramifications could be devastating to the client, especially if the research is poorly performed, inaccurately reported or not supported by other pieces of research of the same topic.

To conclude, it is important to use a “constructive discontent” point of view when reading research. Students, in particular, must be encouraged to critically deconstruct a piece of research and decide for themselves the importance that lies within it, if any. They must also be aware of incomplete reporting of data in research articles. Where possible and appropriate, the researchers should make an effort to measure more than one to three dependent variables when a comprehensive physiological profile is more appropriate. They should also report all the dependent variables, even those that do not reach statistical significance. Students should be especially cautious of the use of measurements that have not been proven valid when trying to show relationships. Also important is recognition of the lack of practical significance in a particular study. Finally, pay special attention to the results sections in research. Without doing so, one might overlook instances of poor logic, misleading statements and/or even mistakes in statistically based assumptions. All of these issues are of critical importance for students and professionals alike.