Generally speaking I support the idea that academic researchers should engage with public debate. If we have evidence that could help inform policy and wider debates then it's right that we should publicise that beyond our own narrow academic communities. Blogging is one highly effective way of doing that. But every so often I read something on an academic blog that makes me pause and question whether more blogging will end up improving the quality of public debate in the long run.
Yesterday's LSE British Politics and Policy Blog on NHS competition provides a good example of the kind of post that worries me. This post criticises research about the impact of NHS competition on patient outcomes and accuses my LSE colleagues of engaging in bad science, data dredging and faulty analysis. I assume that this post will have been read by far more people than the original scientific paper. That's to be expected - after all it is what the blog is trying to achieve. But as I read through the post I became increasingly puzzled by the fact that the criticism in the the post appeared to bear little relation to the scientific papers I had read. When I went back to the original papers this morning, this confirmed my original concerns.
As I know people may not have access to the academic paper, let me give some concrete examples.
The 'major cause of reductions in AMI'?
PAPER: “using [acute myocardial infarction] AMI mortality as a quality indicator, … mortality fell more quickly (i.e. quality improved) for patients living in more competitive markets after the introduction of hospital competition (to the NHS) in January 2006"
BLOG: "The major improvements in outcome after acute myocardial infarction can be attributed to improvements in primary prevention in general practice and in hospital care".
These points are not contradictory. Couldn't the major improvement be attributable to primary prevention and hospital care while hospitals that face more competition saw mortality fall more quickly?
Heart attack victims don't choose where to be treated
BLOG: "the government’s cardiac Tzar, Sir Roger Boyle, was sufficiently angered by their claims to respond with withering criticism: AMI is a medical emergency: patients can’t choose where to have their heart attack or where to be treated!"
PAPER: "we expect that AMI mortality will decrease more quickly in more competitive markets from mid-2006 onwards after hospitals were exposed to competition created from the new NHS reimbursement system and the expansion of patient choice. While providers are not explicitly competing for AMI patients because competition in the NHS is limited to the market for elective care, we expect the market-based reforms to result in across-the-board improvements in hospital performance, which in turn will result in lower AMI death rates. To that end, Bloom et al. (2010) looked at NHS hospitals and found that better managed hospitals had significantly lower AMI mortality and that greater hospital competition was associated with better hospital management."
In short, the paper is quite clear on the mechanism. Competition on elective care care improves management which also happens to benefit AMI. Why not use elective care directly? Because hospitals can 'manipulate' statistics around those in a way that it can't with AMI precisely because patients have no choice! In other words, the authors clearly understand that patients have no choice for AMI but this helps rather than hinders them in their research.
Elective patients don't choose hospitals
BLOG: "Less than the half patients surveyed in 2008 even remember being given a choice, and only a tiny proportion made those choices based on data from the NHS choices website."
PAPER: There were three components to the health reform only one of which concerned patient choice but all three of which sharpened incentives for hospitals. Also, even if patients don't remember being given a choice: "since GPs are highly active in informing the destination of most referrals, GPs now play a substantial role dictating how money flows around the post-[reform] NHS."
There are several ways in which the reforms sharpened incentives for hospitals. Pointing to the fact that patients don't remember being given a choice doesn't seriously address whether or not these incentives worked in practice.
No biological mechanism for choice to affect outcomes
BLOG: There is no biological mechanism to explain why having a choice of providers for elective hip and knee operations surgery [...] could affect the overall outcomes from AMI where patients do not exercise choice over where they are treated."
But the paper doesn't ever claim that there is a biological mechanism. It claims there is an economic one via the incentives described above.
They ignore the existing evidence
BLOG: "They sweep aside decades of careful economic theory and evidence which shows why markets do not work in health services"
PAPER: Provides pointers to existing literature (and reviews) and specifically considers the reasons why evidence from the reforms of the mid 1990's, the internal market might not be very useful "because the internal market never created significant financial incentives for hospitals to change their behaviour"
They engage in data dredging and their work should never have been published
BLOG: "if you repeat an analysis often enough significant statistical associations will appear." The work was subsequently published in "the Economic Journal. That it got through that journal’s peer-review process is perhaps indicative of the poor understanding of healthcare and routine data from reviewers of that journal."
I don't see any basis for the first of these claims. Data 'dredging' is a serious problem - but not one that appears to apply to this paper (which shows that the results are robust to many different variations in specification - the exact opposite of the data mining problem). The Economic Journal is one of the world's leading peer-reviewed economics journals. I don't believe that peer review is everything, but simply insinuating that the referees and editors of that journal don't know what they are doing doesn't cut much weight with me.
Bad blogging versus bad science
I could go on to discuss the errors around the second paper where, e.g., the blog claims that they don't control for the mix of operations when the paper actually considers within treatment changes in outcomes (so mix is irrelevant). But I assume that the authors are perfectly capable of further defending their own research.
My point is simply that a blog that is supposed to help improve the public's understanding of the evidence is carrying a post that is pretty misleading about what the papers actually do, what they find and what claims they make about their findings. To my mind, this raises far more concerns about bad blogging than it does about bad science ...
[Disclosure: Steve Gibbons is SERC's research director and I am affiliated with the Centre for Economic Performance]
CEP Urban and Spatial Programme Blog (formerly SERC)
