Proud to share my latest publication with you, “Beyond Statistical Significance: Clinical Interpretation of the Rehabilitation Literature.” I wrote most of this article while I was in the hospital for a few months. I hope you enjoy it.
Evidence-based practice requires clinicians to stay current with the scientific literature. Unfortunately, rehabilitation professionals are often faced with research literature that is difficult to interpret clinically. Clinical research data is often analyzed with traditional statistical probability (p-values), which may not give rehabilitation professionals enough information to make clinical decisions. Statistically significant differences or outcomes simply address whether to accept or reject a null or directional hypothesis, without providing information on the magnitude or direction of the difference (treatment effect). To improve the interpretation of clinical significance in the rehabilitation literature, researchers commonly include more clinically-relevant information such as confidence intervals and effect sizes. It is important for clinicians to be able to interpret confidence intervals using effect sizes, minimal clinically important differences, and magnitude-based inferences. The purpose of this commentary is to discuss the different aspects of statistical analysis and determinations of clinical relevance in the literature, including validity, significance, effect, and confidence. Understanding these aspects of research will help practitioners better utilize the evidence to improve their clinical decision-making skills.
While most research studies use statistical significance to reach their conclusions, clinical research studies should report on the “effectiveness” of a study. Statistical significance is of limited value when we want to determine if the treatment will have a clinical benefit.
For clinicians, the most fundamental question of clinical significance is usually, “Is the treatment effective, and will it change my practice?” The effect size is one of the most important indicators of clinical significance. It reflects the magnitude of the difference between treatment groups; a greater effect size indicates a larger difference between experimental and control groups. For example, if the experimental control group improves by 15 points, and the control group improves by 10 points, the change score is 5.
Cohen established effect values based on group differences (change score), divided by the combined standard deviation:
Change in Experimental Group vs. control / Combined Standard Deviation of both groups
For example, if the difference between groups (change score) is 5, and the standard deviation of both groups is 10, the Cohen score (effect size) = 0.5.
Cohen quantified effect sizes in ranges, which may be positive or negative, indicating the direction of the effect:
<0.2 = trivial effect
0.2-0.5 = small effect
0.5-0.8 = moderate effect
> 0.8 = large effect
While it’s important to support evidence in clinical practice, we have more problems translating the evidence into practice. There are several great clinical studies that are difficult to implement into practice for one reason or another. Sometimes, the article itself does a poor job of explaining the specific protocol that can be reproduced (I’m amazed journals still allow these to be published). Other times, it’s a problem with ‘external validity,’ or the relevance of the study to another population, often due to cultural differences.
As with anything related to research, scientists have attempted to quantify the translation of research into practice using the acronym, RE-AIM. Translating evidence into practice can be quantified by its Reach, Adoption, Implementation, and Maintenance:
The five steps to translate research into action are:
RE-AIM is one of the foundations of Translational Research: a field of research that helps quantify and qualify how successful we are at implementing research into practice.]]>
My friend Dr. Brad Edwards, an orthopedic surgeon in Houston Texas, is on the editorial board for the Journal of Shoulder and Elbow Surgery. He recently wrote a short article on the value of systematic reviews. There has been a recent surge in systematic reviews to journals, and I for one am frustrated to see so many of them end with, “There’s not enough evidence for a conclusion.” Rightly so, Dr. Edwards suggests if this is the conclusion to the review, then it shouldn’t be published. Couldn’t agree more. The purpose of a systematic review is to synthesize the literature and pool the data to increase the statistical power; not to tell us that we need more research, which we already know!]]>
Evidence-based practice was defined by Sackett in 1996 as “integrating the best research evidence with clinical expertise, patient values and circumstances to make clinical decisions.”
The Sports Physical Therapy Section of the American Physical Therapy Association (APTA) recently published a special issue in the International Journal of Sports Physical Therapy that reviews important topics in bringing evidence to your practice. The international authors provide information for practicing clinicians who want to perform research or learn how to integrate research into practice. The IJSPT is a PubMed-indexed journal with open access.
I was honored to contribute the paper, “Research Designs in Sports Physical Therapy.” My paper includes a discussion on the different research designs including experimental and non-experimental designs. It helps clinicians recognize the appropriate design to answer a specific question.
Other papers in interest in the October 2012 issue include:
If you don’t have time, you don’t have to read all these articles now. Keep the series as a reference for later when you are thinking about performing a study or publishing a paper. This series of articles is an outstanding and FREE resource!]]>
A report in Science Daily reviewed a study in the October 24/31 issue of JAMA. The study examined the characteristics of studies that yield large treatment effects from medical interventions. The researchers found that these studies were more likely to be smaller in size, often with limited evidence. Interestingly, when additional trials were performed, the effect sizes usually became much smaller. Read more here.]]>
I like this decision tree to help us choose which statistics we should use in research. Even if you don’t perform research, it might help you as a critical appraiser of research in determining if the study used appropriate stats.
Unfortuantely, some studies get published in journals without using the right statistical analysis….as an educated consumer, I hope this chart helps you make better evidence-based clinical decisions!
Statistics in Kinesiology, 4th edition. By William J. Vincent and Joseph P. Weir.]]>
Part 2 of More practical Evidence-Based Practice for today’s clinician
“Heterogeneity” among patient populations is key to understand. One of the problems with clinical research, and relying too heavily on it, is not appreciating the ‘sterility’ of research. The scientific method relies on homogeneity when applying results to a specific population; the scientific method also relies on controlling as many variables as possible in order to isolate an effect. Studies using patient populations with the same diagnosis often do not control for the different impairments leading to the pathology.
I’m often amazed how many clinical studies lump patients with the same diagnosis together without stratification for different impairments. Experienced clinicians treat the impairments, not the diagnosis; in addition, they know that no two patients with the same diagnosis are the same…this heterogeneity within a population forces us to adjust the application of EBP as defined previously. While it would be ideal to have clinical research generalizable to a large population, different clinical presentations of patients with the same diagnosis can make implementation of published research findings difficult.
Another limitation of clinical research is the lack of more direct, valid and reliable measures for certain clinical symptoms. One of my favorites is “proprioception”. Clinical researchers consider measures such as joint repositioning and kinesthetic awareness as valid measures of proprioception. Lephart and Fu wrote an excellent text on proprioception in 2000. Technically speaking, proprioception is likely only validly measured with somatosensory evoked potentials within the cortex! The common aforementioned clinical tests can give us a sense of the ‘processing’ of proprioception, but cannot quantify proprioception itself! Clinical research will continue to improve as technology to measure clinical entities improves.
We can’t base everything we do in the clinic on evidence alone. There is such a dearth of evidence, we would have nothing to do if we practiced true traditional EBP. I continue to be frustrated by the number of systematic reviews that conclude there’s “not enough evidence” to make a conclusion. One thing I’ve noticed over the years is that clinical research lags clinical practice by several years. We often do things in the clinic with biologic plausibility until it’s finally researched a few years later…remember that it takes several years for good quality research to be published from start to finish!
While I appreciate the push for more evidence to support our interventions, we should not base all decisions on the presence of positive or negative outcomes. “NOT PROVEN” techniques still have a place in clinical practice as long as they make sense and the “PROVEN” techniques have been attempted or ruled out. Perhaps we should better-define EBP as “evidence-led” or “evidence-informed” practice.
Several years ago, the concept of “evidence-based practice” (EBP) made its way into rehabilitation sciences…the notion that everything we do in practice needs to have evidence to support it. That raised the question then, “what is evidence”?
Practically speaking, “evidence” can give us one of three things: PROVEN, PROVEN NOT, or NOT PROVEN. In other words, research can tell us if something is proven to work, proven not to work, or there’s not enough evidence to tell us either way. In the clinic then, we should focus on using PROVEN interventions, avoiding PROVEN NOT interventions, and using our best judgment when using NOT PROVEN interventions.
Using this philosophy, the first 2 are pretty simple. “NOT PROVEN” interventions, however, should follow the lines of ‘biologic plausibility’. This concept states that something makes logical sense based on our knowledge of anatomy, biomechanics, physiology, etc….and has some ‘path’ from basic science to clinical application. This is the ‘leap of faith’ many use, for example, with animal models to interpret results to humans. Eventually, we hope that an intervention with biologic plausibility can be taken to the next level with outcomes research on actual patients. Note, however, that there are some studies (e.g. in-vitro) which have to be performed on animals that we hope translate into humans because performing them on humans would never get past IRB!
This “NOT PROVEN” area seems to be an intermediate zone compared to the “green means go” PROVEN and “red means no” PROVEN NOT, which leads us to re-think the definition of EBP and move towards a more practical definition for clinicians. I’ve always preferred this definition of EBP that I’ve adapted from several others:
Evidence-Based Practice uses the BEST AVAILABLE evidence with the EXPERIENCE of the clinician and the PATIENTS individual situation.
Bottom line: using these 3 points supports evidence-based clinical decision making! Clinical decision making is the heart of applying evidence in practice. We first rely on evidence, but can’t make all our decisions based on evidence alone. Let’s break those 3 key components in this more practical definition:
Next: Why is EBP difficult for everyday clinicians?]]>