Australia’s NHMRC publishes flawed report despite concerns raised during public consultation

Australia’s NHMRC publishes flawed report despite concerns raised during public consultation12 March 2015

Although the HRI welcomes thorough research in homeopathy, only studies carried out using appropriate and rigorous scientific methods can produce meaningful results. We therefore lament the recent publication by Australia’s National Health and Medical Research Council (NHMRC), which fails to meet this standard.

During a public consultation on the draft version of this report, HRI and others highlighted deep flaws in how the NHMRC had analysed the evidence on homeopathy. None of these serious problems were addressed in the final publication. This raises questions as to whether the public consultation was ever meant to have any impact on the final report, leading to serious concerns about the conduct of this governmental body.

We maintain that the conclusions of the NHMRC report are inconsistent with the evidence.

The inaccuracy of the NHMRCs conclusions stem primarily from one fundamental flaw at the heart of this report – the NHMRC reviewers considered the results of all trials for one condition together as a whole, even though the individual trials were assessing very different types of homeopathic treatment.

To illustrate this flaw, the NHMRC reviewers asked, “Is homeopathy effective for condition A?’, working from the premise that a positive trial showing that one homeopathic treatment is effective is somehow negated by a negative trial which shows that a completely different homeopathic treatment for that same condition is ineffective. This is a bizarre and unprecedented way of assessing scientific evidence. In conventional research the question asked would be, “Is treatment X effective for condition A?”, not “Is conventional medicine effective for condition A?” based on combining the results of all drug trials together. Some treatments work, some don’t. The whole point of medical research is to establish which treatments are useful and which are of no value. This is no different in homeopathy.

This single methodological flaw explains why the NHMRCs has failed to find any ‘reliable’ evidence that homeopathy is effective for any of the 61 conditions under consideration.

Secondly, we are deeply perplexed as to the reasons for the exclusion of some of the best evidence for key clinical conditions. In brief:

Jacobs et al performed meta-analysis a meta-analysis of the treatment of childhood diarrhea using homeopathy in 2003, N=242 in placebo controlled trials, p-value = 0.008. This meta-analysis was excluded … why? [Link] Wiesenauer & Lüdtke conducted a meta-analysis into the treatment of hayfever in 1996, N=752 in placebo controlled RCTs, p-value <0.0001. This meta-analysis was excluded. Again we ask ourselves why? [Link] Schneider et al conducted a meta-analysis of non-inferiority trials of homeopathy compared to usual care for the treatment of vertigo, N=1388, non-inferiority was clearly demonstrated. Again excluded, again why? [Link]

A Cochrane review by Mathie et al inspected the evidence for the treatment of influenza using homeopathy concluded in favor of homeopathy (N=1259, placebo RCTs, p=0.001) yet this evidence was simply ignored in the final conclusions on the grounds of possible bias in the underlying studies [Link].

The NHMRC also need to justify their use of N=150 as a line between reliable and unreliable trials and they certainly need to explain why size is relevant at all when the findings are statistically significant.

Furthermore we do not see how there could be any justification for the absence of a homeopathy expert on the NHMRC review board. The presence of such an expert would potentially have prevented many of the issues raised here and would definitely have reassured the public about allegations of bias on the part of the NHMRC.

We note that the conclusions of the NHMRC report are at odds with the conclusions of the recent extensive meta-analysis of RCTs using homeopathy performed by Mathie and co-workers which concludes in favour of an effect of homeoapthy[Link].

The HRI does not dispute the fact that there are few high-quality, positive studies in homeopathy and that these need independent replication, but we do dispute the NHMRC’s failure to identify these positive studies in their Information Paper as promising studies which should be repeated.

Despite the considerable means spent on this report and great profession of due process and absence of bias, as demonstrated above the NHMRC has failed both in terms of the process they used and in the fairness of their assessment of the evidence.

The NHMRC documents can be found here, with the final statement here.

The HRI’s response to the initial draft can be found here, and our submission to the NHMRC can be found here. https://www.hri-research.org/2015/03/nhmrc-publishes-flawed-report-despite-concerns-raised-during-public-consultation/