This week we have a really unique study. Here’s the reference:
The point of this study was to check out how effective youth therapy is based on research published from 1963 to 2013. The researchers wanted to look at all the randomized controlled studies in that time period and put them all together to make some conclusions – this is called a meta-analysis.
Hang on one second though. Before we go any further, let’s just clarify a couple of things as it will help us understand the study (you can always skip this part and come back if you’re confused):
- What is a randomized-controlled study? This is the gold-standard study. There are 2 terms you need to understand: randomized and controlled. Let’s start with controlled. Imagine there are 100 people in the study needing treatment. Researchers are testing a new treatment so they are comparing this new treatment, to another treatment (or just no treatment) which is called the control. They compare to a control so they have something to compare the treatment to. Now, the researchers could just say that the first 50 people who signed up get the main treatment, and the last 50 get the control treatment – but that would be no good. For example, it could be the people who sign up first are most eager and thus if we find a difference between the treatment and the control it is actually just attributable to the eagerness of participants. To make this study even better, and stop this from happening, researchers then randomize the groups, so it is completely random who gets what treatment. This makes it really unlikely that differences between characteristics of participants could affect the study. By doing this, the researchers have created a randomized-controlled study.
- What is a meta-analysis? A meta analysis study is a study of studies. Basically, a meta analysis is a statistical technique of combining a bunch of studies that look at the same thing, to see what the general research says on a topic.
- What is an effect size? This is a statistic used to indicate the size of the difference between a treatment and a control. A large effect size just says that there’s a big difference – thus maybe the treatment really works well!
Ok, back to the study.
By looking at studies from the last 50 years they had 4 main questions.
- Is youth therapy effective?
- Does effectiveness differ for different disorders?
- Does effectiveness differ for different types of treatment?
- Does the type of control condition affect research outcomes?
This study was a MASSIVE undertaking. I have never seen anything like it before. To do this study they have to review 50 years of research…50 years!!!! They looked at over 4000 different studies to see if they were of the quality needed to be included into the meta analysis. Reviewing 4000 studies is an immense amount of work – on average, each author had to check out over 300 studies. After doing this they decided that 447 studies met their criteria (you’ll have to check out the study to see what these were). They then did some complicated statistical stuff to test out their 4 questions. Again, check out the study to see more!
Now let’s break down what they found:
Does Therapy Work?
The good news is YES! The effect size overall was .46 – which is close to medium.
What Does Therapy Work For?
Treatment for anxiety works the best. The effect size post-treatment .61 – which is medium. This makes so much sense as the research is quite established that exposure therapy, which is what a lot of the research studies conducted, works.
Interestingly, depression treatment outcomes were pretty poor, with an effect size of .29, which is low. The authors even found that based on teacher reports, which they consider to be the least biased report, depression treatment was worse than control treatment. This does suggest we need to work more on developing our depression treatments.
They also found that treatment for multiple problems was very poor, with an effect size of .15, which is below small! This makes sense because treating multiple problems does seem like it would be more difficult, but as they note, it’s a big problem as many youths come to treatment with multiple problems.
Which Therapies Work Best?
This one is most interesting to me. They found that individual CBT showed the “most robust” effects – what they mean was that youths, parents, and teachers all agreed with the improvements made in treatment. For all other treatments, this was not the case, with some disagreements where at least one rater reported very little progress. However, all the treatments basically worked as well as each other. As they say, the data are the data. This finding definitely warrants more research. One very practical suggestion as a result of this finding is that clinicians should always assess everyone to make sure there is consensus agreement.
Is the Research Influenced by Which Control Treatment was Used?
Yes. It seems that placebo treatment and “treatment as usual” control conditions are actually most effective. When treatments were compared to these groups there was less of a difference between the treatment and the control. As a heads up, this is a great reminder that when looking at research, be aware that while a new treatment could look really effective, it could just be that the researchers were comparing it to a very ineffective control. For example, while Head and Shoulders might work great, but it should be compared to other shampoos in the research, rather than no shampoo at all, to test whether it really does work!
Some Other Findings
First, their results indicate that if a youth gets a treatment, the probability that they will be better off than if they received a control treatment is 63%. As they point out, this is only just better than chance (50%), which is not good. However, this statistic is across all disorders studied – given the high effect size for anxiety treatment, it seems possible that anxiety treatments are more effective, but they don’t explore this possibility.
They also found a ton of other interesting stuff that wasn’t much discussed. I’ll just note some here:
i) Treatment outcomes weren’t different across the last 50 years – which could be interpreted as saying that treatment hasn’t become any more effective. This is definitely a possibility, but they do note that all sorts of things could be a factor. For example, research design has changed over the years, with the type of control condition used in particular becoming more stringent in the past 20 years. Researchers need to think about why there is not difference though, as it could affect future study design.
ii) Outcomes in the US were worse than international studies. Don’t know why.
iii) Treatment outcomes were the same for all ethnicities, gender, ages (they split this by under and over 12 years).
iv) This one is counter-intuitive, but they found that treatment length was negatively related to effect size. All this means is that the longer treatment goes on, the smaller the difference between treatment and control. Do not fear, this does not mean that longer treatments are a bad thing. Instead, it’s likely that the effects of treatment just taper off or plateau with time, and shorter treatments never hit this point. I wish they had discussed this more, but they didn’t so this is just my guess.
v) There is a vast problem in the research called “publication bias”. Basically, journal articles aren’t interested in publishing studies that don’t find anything. Thus, there will have been studies done where the treatments did not differ to the control. These will have been considered “disappointing”, likely just scrapped by the researchers, and never written up. As a result of the publication bias, it could be that the results of this meta-analysis are completely biased because there might be many unpublished studies that show treatment to be totally ineffective that were never written and/or published. It’s a real problem. As such, they did certain analyses to try and account for this, one of which is called the fail-safe N – this accounts for how many unpublished studies it would take to show that treatments do not differ to controls. They found that it would take 90,654 unpublished studies!!! So its fairly safe to say that the results of this meta-analysis will still stand.
This was an AMAZING study. They discuss some of the limitations. A lot of the limitations are actually a trade-off in many ways. By having large data and grouping things together, you can miss the finer analyses that would be possible if the investigation was limited only to the last 5 years. However, they noted too that in the older studies the reporting is different to today so certain information (e.g. ethnicity, gender) is not there, or diagnostic criteria changed over the years. They also noted that they only focused on English articles.
However, my biggest complaint is probably that the study wasn’t broken down further. This is an unfair complaint, not only because as I just mentioned this is a trade-off in meta-analyses, but also because journal articles limit the size of articles. However, as an anxiety clinician and researcher, I’d have really liked to see the breakdown in treatment efficacy for anxiety disorders. I’d like to know what different informers reported in treatment progress, just for the anxiety disorders, in addition to knowing how many kids with anxiety are likely to benefit from treatment beyond what control conditions achieve. Maybe in the future, other studies will examine this information with this data set.
The other thing I noticed was that the article title includes “implications for science and practice”, yet most of the discussion really focused on science. It would have been helpful, and might address my above concern, if the discussion went into the clinical practice implications. For example, just knowing that anxiety treatment seems to work, or that depression treatment in not great, doesn’t really help me clinically beyond giving basic education. Given this study reviewed so many articles, it would have been awesome to have summarized some of the literature for why some treatments were helpful, or what the active components of treatment seem to be. The most notable area this applies to is having multiple problems, given that so many people have more than just one thing going on. Since having multiple problems is associated with a poor response to treatment, they could have discussed why this might be and what has been proposed in the research to deal with this issue clinically. That would have been SUPER helpful. Instead I’m left not knowing whether to address everything at once rather than one thing at a time, or vice versa, whether to bring in the use of medications or increase treatment intensity, etc. Proposing some ideas would actually really help me take in some of the ongoing discussions in the research to inform how I go about practice. Without this information, such an incredible study doesn’t actually impact clinical practice – which unfortunately means that it doesn’t really meet its own purpose. I’m left wondering what the implications for clinical practice really are. This might seem like harsh criticism, but I think it’s a fair point. To me, it would have taken this study from an A to an A+!
Thanks for reading, please feel free to leave a comment.