The July 2012 issue of the New Zealand magazine North & South proudly bore the title for its cover story: “Do you believe in magic? The truth about alternative medicine.” The issue included an article which displayed with equal pride the title “Homeopathy – trick or treatment”. The byline to this article stated: “Homeopaths claim their formulas have “energetic powers”. Scientists say the only power in these potions is the placebo effect.”

Clive Stuart, a Tauranga homeopath, complained to the Press Council that the article and accompanying editorial were highly derogatory, inaccurate and misleading. The Press Council upheld the complaint (see www.presscouncil.org.nz/display_ruling.php?case_number=2320).

Mr Stuart’s complaint was supported by a detailed critical analysis of a key article, written by Dr David St George, a clinical epidemiologist. The key article was a meta-analysis of 110 homeopathy clinical trials and 110 conventional medical clinical trials, published in the Lancet in 2005. Sceptics often use this article to bolster up their belief that homeopathy is nothing more than a placebo. However, the Lancet article has already been challenged in the international scientific literature. Critics say that it has been subjected to selection bias through the process that the authors used to identify a subset of the clinical trials for more detailed analysis. The article’s conclusions are based on the subset analysis, but the selection bias in their methodology undermines the validity of their conclusions.

The critical analysis which supported Mr Stuart’s complaint doesn’t therefore say anything new. All of it has previously been said in the international scientific literature. What now follows is an edited version of the analysis.

Untruthful statements in “Homeopathy – Trick or Treatment?”

There are two key statements in this article, which are both untrue:

“….homeopathic remedies have failed every randomised, evidence-based scientific study seeking to verify their claims of healing powers.”

“…there is no scientific evidence of homeopathy’s efficacy.”

These statements are absolute claims about the absence of any scientific evidence and, as such, they are factually incorrect. There are indeed published “randomised, evidence-based scientific studies” which demonstrate the efficacy of homeopathy, contrary to the article’s claim.

The article’s claims appear to arise from a study published in the Lancet in 2005: “Are the clinical effects of homeopathy placebo effects? Comparative study of placebo-controlled trials of homeopathy and allopathy”, by Shang et al. This study is regularly quoted by critics of homeopathy as the definitive scientific answer to the question of whether or not homeopathy is anything more than a placebo effect.

Shang et al set out to prove that there is no scientific evidence of any therapeutic benefit from homeopathy beyond the placebo effect. The conclusion that they reached was that their findings do substantiate their a priori hypothesis of homeopathy being nothing more than a placebo effect. However, critics of Shang et al’s methodology have pointed out that there are inherent biases and limitations in their analysis. A more rigorous analysis of Shang et al’s data shows that their hypothesis cannot be substantiated.

The Shang et al study is important to understand because it sheds light the current debate about scientific evidence and homeopathy. It helps to clarify why absolute claims (such as those outlined in North & South) are being erroneously made about the absence of scientific evidence of efficacy, when in fact such evidence exists. This study therefore warrants a more detailed exploration.

The Shang et al study is a meta-analysis of 110 published placebo-controlled trials of homeopathy, compared with 110 published placebo-controlled trials of conventional medical drug treatment.

The first point to emphasise is that Shang et al’s own conclusions are not as absolute as North & South’s statements. Their interpretation section (in the summary of their published paper) says that their analysis indicated that “there was weak evidence for a specific effect of homeopathic remedies, but strong evidence for specific effects of conventional interventions.” Note that they did not say there was “no evidence” for a specific effect of homeopathic remedies, but only “weak evidence”. North & South’s absolute claim of no evidence (the second statement above), therefore, is not supported by this study.

Secondly, the majority of the 110 homeopathy trials that were included in the study are “randomised, evidence-based scientific studies” (to use North & South’s phrase). Some of them may well be considered to be lower quality studies, but, nevertheless, they still include “randomised, evidence-based scientific studies” which have clearly demonstrated evidence of a therapeutic benefit beyond the placebo effect. North & South’s first statement (above) is therefore factually incorrect and misleading.

Thirdly, the real debate is not about whether or not there are any scientific studies demonstrating homeopathy’s therapeutic benefit, because there are quite a number of such published scientific studies which do show a therapeutic benefit. The debate is somewhat different from this. It is about whether or not the published scientific studies that do clearly demonstrate homeopathy’s therapeutic benefits are of acceptable methodological quality.

What the critics of homeopathy argue is that these scientific studies are methodologically flawed and must therefore be discounted. Indeed, this is what Shang et al tried to prove.

There is a well-known statement which summarises this position: “If the facts don’t fit the theory, then the facts must be wrong.” Homeopathy is considered by many conventional biomedical thinkers to be implausible - it cannot possibly have any therapeutic benefit. Any research which appears to show a therapeutic benefit must therefore be flawed research, which should be discredited. This is an a priori stance, based on theory and belief.

This approach is hinted at in a follow-up letter about the homeopathy article that North & South published. This is from Dr Shaun Holt of Tauranga. In this letter, he said:

“..there has never been a single, reproduced study, of an acceptable standard, showing any effect whatsoever of homeopathy in people, animals or even in a test tube.”

Note that this statement is different from North & South’s, in that Dr Holt introduces the qualifier, “an acceptable standard”. There is absolutely no doubt that there are published scientific studies showing efficacy of homeopathy. However, the ones that do show efficacy are considered by Dr Holt (and others like him) to be “of an unacceptable standard”, and must therefore all be rejected. The question is whether Dr Holt and the others who take this stance are justified in doing so, from the perspective of scientific evidence (as opposed to scientific theory or belief).

If you want to locate the published scientific studies which demonstrate the efficacy of homeopathy, you only have to read Shang et al’s study in detail, because these studies are contained within it.

Below is Figure 2 from Shang et al’s publication. This compares the 110 homeopathy trials with the 110 conventional medicine trials, using what is called a “funnel plot”. Each trial is a dot on the appropriate graph.

The horizontal axis is the “odds ratio”, which (in simple terms) indicates the degree of benefit arising from the active treatment (whether homeopathy or conventional medicine) when compared with placebo treatment.

• An odds ratio of 1 (indicated by the vertical dotted line) is where the active treatment equals placebo, in terms of the degree of therapeutic benefit.
• An odds ratio to the left of the “1” (i.e., a smaller ratio than 1) indicates a therapeutic benefit from the active treatment which is greater than placebo. The further to the left the trial is positioned, the greater the benefit is from the active treatment.
• An odds ratio to the right of the “1” indicates that placebo treatment confers a greater therapeutic benefit than the active treatment.

The vertical axis (SE) is a proxy indicator of sample size. Larger trials are towards the top of the graph and smaller trials are towards the bottom.

These figures show a very similar picture for the 110 homeopathy trials and 110 conventional medicine trials. The majority of both groups show a beneficial therapeutic effect (i.e., they are to the left of the vertical dotted line). Also, in both groups, smaller trials (in the lower part of the graph) show more beneficial treatment effects than larger trials (towards the top of the graph).

With regard to North & South’s claim of “no scientific evidence of homeopathy’s efficacy” the homeopathy graph demonstrates something different. In this overall analysis, there is scientific evidence of efficacy amongst those studies which are to the left of the vertical dotted line.

However, as I have already said, the real debate is more about whether the large number of homeopathy clinical trials which do show efficacy are of sufficient methodological quality; or whether they can be rejected as being methodologically flawed.

Shang et al therefore went a stage further in their study and attempted to identify a subset of “higher quality” homeopathy and conventional medicine studies from amongst the 220 trials.

They tried to see if restricting their analysis to only “higher quality” trials would confirm their hypothesis that homeopathy is no better than placebo.

They focused on three indicators of research methodology quality: (1) the degree of randomisation and concealment of allocation; (2) the degree of masking (“blinding”) of patients, therapists and outcome assessors as to which group individual patients were in (i.e., active treatment or placebo); and (3) the degree to which the results were analysed by intention to treat (i.e., including all patients who were randomised in the analysis, not just those who were treated and followed up to the end of the study).

Using these criteria, they produced a subset of 30 trials: 21 “higher quality” homeopathy trials and 9 “higher quality” conventional medicine trials. However, they did not publish an analysis of how the two groups compared, in terms of therapeutic benefit. Instead, they restricted their analysis to only “larger” trials amongst the thirty. Using an arbitrary definition of a “larger” trial, they ended up with a subset of only 14 trials: 8 homeopathy trials and 6 allopathic trials.

Comparing the combined results of the 8 homeopathy trials with the combined results of the 6 allopathic trials, they found that the 8 selected homeopathy trials did not demonstrate any overall therapeutic benefit beyond placebo, whereas the 6 allopathic trials did.

The overall conclusion that Shang et al published (i.e., that their findings support the notion that the clinical effects of homeopathy are placebo effects) is thus based on this “8 vs 6” comparison, not on the “110 vs 110” comparison (nor on the “21 vs 9” comparison).

At this point, it is important to emphasise that the way that Shang et al undertook the subset selection and analysis did not adequately conform to the scientific method. The study itself is a methodologically flawed study.

The “scientific method” is not just about carrying out appropriately designed experiments and analyses; it is about the entire process that is driven by your intention to prove empirically and conclusively that your scientific hypothesis is correct. Because all of us (including scientists) are prone to biases and self-deception, a scientific investigator has to establish empirical investigations that adequately remove his/her own biases towards proving that the hypothesis under investigation is correct. In other words, as a scientist, you have to be “your own worst critic” in your choice of methodology. You have to demonstrate sceptically and convincingly to others that the results you have obtained are robust, objective and not distorted by your own a priori beliefs. You have to attempt to refute your own scientific hypothesis.

In practical terms, this is done by denying your own scientific hypothesis in your research methodology. Although you start the process with a scientific hypothesis, you then adopt the position opposite from this, which is often called the “null hypothesis”. You then continue your investigation under the assumption that the null hypothesis is correct (i.e., that your scientific hypothesis is wrong) and you adhere to the null hypothesis unless and until you have overwhelming evidence to the contrary. This evidence allows you to reject the null hypothesis and accept the scientific hypothesis with a high degree of confidence.

With regard to Shang et al’s study, the starting point (i.e., the scientific hypothesis) is that homeopathy is nothing more than a placebo. The purpose of their analysis was to explicitly prove this. In Shang et al’s words:

“We assumed that the (positive) effects observed in placebo-controlled trials of homeopathy could be explained by a combination of methodological deficiencies and biased reporting. Conversely, we postulated that the same biases could not explain the effects observed in comparable placebo-controlled trials of conventional medicine.”

The authors are quite open about their a priori belief in the implausibility of homeopathic treatment, and their consequent hypothesis that any apparently positive results from placebo-controlled homeopathy trials must be false positive results, arising from flawed methodology and biased reporting.

Shang et al are entitled to adopt this belief as a scientific hypothesis. However, in order to conform to scientific methodology, they should have then taken the opposite stance; i.e., they should have become their own worst critic. They should have adopted the null hypothesis that the therapeutic benefit from homeopathy is more than a placebo.

They should then have rejected this null hypothesis only if they found overwhelming and irrefutable evidence that homeopathy was indeed a placebo. They could only have achieved this through a far more robust and critical analysis.

Putting this in another way, instead of attempting to refute their own scientific hypothesis, Shang et al carried out a subset analysis in order to find the evidence to support it. They found exactly what they were looking for, in the “8 vs 6” comparison, but they made somewhat arbitrary and subjective decisions along the way, in order to get there. Their analysis is thus a long way away from what should be expected from robust scientific research. Indeed, it has already been considered to be an example of “data dredging”; i.e. looking for what you want to find, digging deeper into the data in order to find it, without critically evaluating the validity or robustness of the methodology being used.

What Shang et al should have done (as a minimum) is a standard analytical procedure known as a “sensitivity analysis”. They should have critically tested the sensitivity of their subset selection process, to see if changing the selection criteria and threshold values that they used would have resulted in a different subset of trials with different overall results.

Fortunately, such a sensitivity analysis has been carried out (by others) and published (elsewhere). It was published in 2008 in the Journal of Clinical Epidemiology as “The conclusions on the effectiveness of homeopathy highly depend on the set of analysed trials”, by Ludtke and Rutten. The authors demonstrated that some of Shang et al’s selection criteria were somewhat arbitrary and subjective. By changing the threshold of the selection criteria, different results could be obtained, in terms of the overall therapeutic benefit of homeopathy. The authors said:

“Shang et al in their article arbitrarily defined one subset of eight trials which provided an overall negative result for homeopathy. Our article shows that the choice of other meaningful subsets could lead to the opposite conclusion.”

For example, Ludtke and Rutten found that if they changed the arbitrary criterion for a “larger” trial used by Shang et al (i.e., the largest quartile of the subset, based on a sample size N=98 or larger) to the median sample size of all homeopathy trials (i.e., N=66 or larger), the subset of homeopathy trials selected by this latter criterion showed a significant therapeutic effect in favour of homeopathy. There is no objective or meaningful a priori reason for choosing Shang et al’s definition of a “larger” trial over any other definition, and therefore no objective or meaningful reason for concluding that the “8 vs 6” comparison is the one subset to be used use in order to carry out the “definitive” comparison between homeopathy and conventional medicine.

However, just as interesting is the fact that they did not publish the comparison of the subset of “higher quality” trials that they first selected, as mentioned above (i.e., the 21 homeopathy trials versus 9 conventional medicine trials). This analysis, as demonstrated by Ludtke and Rutten, showed a statistically significant therapeutic benefit for homeopathy. It is not self- evident that Shang et al had to skip over this analysis and restrict the “definitive” analysis even further, to “larger” trials only. Other things being equal, a medium-sized study with statistically significant results cannot be rejected out of hand just because of its sample size. The statistical analysis would have taken into account the sample size.

Choosing only larger studies can also introduce other biases. For example, larger homeopathy trials tend to be based on a very different homeopathic approach from smaller homeopathic trials; e.g., using a conventional medical diagnosis to define the subjects and using the same single remedy for all subjects in the larger trials. This is an approach which classical homeopaths claim dramatically reduces the effectiveness of their treatment.

Shang et al thus appear to have adopted the “data dredging” approach (i.e., pursuing analyses until they found the answer they were looking for), rather than a truly scientific approach (i.e., attempting to refute their own hypothesis). Reaching the author’s conclusion required a subjective and somewhat arbitrary choice of subset selection criteria. A different choice would have led the authors to the opposite conclusion. Shang et al’s study does not therefore provide robust scientific evidence that homeopathy is no more than placebo.

We are therefore left with the status quo with regard to scientific evidence and homeopathy; namely that there is a large group of published placebo-controlled trials of homeopathy which demonstrate therapeutic benefit for homeopathy. The quality of these trials is just as variable as the quality of conventional medicine trials.

Those who currently reject the positive homeopathy trials as being of an unacceptable methodological standard are thus doing so more because of their a priori belief that homeopathy can be nothing more than placebo, rather than from the basis of an objective, unbiased, rigorous analysis of scientific evidence.

In conclusion, this analysis supports the opinion that the article is one-sided; is based on a factually incorrect and misleading understanding of the nature of the scientific evidence concerning homeopathy’s efficacy; and is thus inaccurate, unfair and unbalanced in its treatment of homeopathy.