Tuesday, 6 March 2012

NHS competition: bad science or bad blogging?

Generally speaking I support the idea that academic researchers should engage with public debate. If we have evidence that could help inform policy and wider debates then it's right that we should publicise that beyond our own narrow academic communities. Blogging is one highly effective way of doing that. But every so often I read something on an academic blog that makes me pause and question whether more blogging will end up improving the quality of public debate in the long run.

Yesterday's LSE British Politics and Policy Blog on NHS competition provides a good example of the kind of post that worries me. This post criticises research about the impact of NHS competition on patient outcomes and accuses my LSE colleagues of engaging in bad science, data dredging and faulty analysis. I assume that this post will have been read by far more people than the original scientific paper. That's to be expected - after all it is what the blog is trying to achieve. But as I read through the post I became increasingly puzzled by the fact that the criticism in the the post appeared to bear little relation to the scientific papers I had read. When I went back to the original papers this morning, this confirmed my original concerns.

As I know people may not have access to the academic paper, let me give some concrete examples.

The 'major cause of reductions in AMI'?

PAPER: “using [acute myocardial infarction] AMI mortality as a quality indicator, … mortality fell more quickly (i.e. quality improved) for patients living in more competitive markets after the introduction of hospital competition (to the NHS) in January 2006"

BLOG: "The major improvements in outcome after acute myocardial infarction can be attributed to improvements in primary prevention in general practice and in hospital care".

These points are not contradictory. Couldn't the major improvement be attributable to primary prevention and hospital care while hospitals that face more competition saw mortality fall more quickly?

Heart attack victims don't choose where to be treated

BLOG: "the government’s cardiac Tzar, Sir Roger Boyle, was sufficiently angered by their claims to respond with withering criticism: AMI is a medical emergency: patients can’t choose where to have their heart attack or where to be treated!"

PAPER: "we expect that AMI mortality will decrease more quickly in more competitive markets from mid-2006 onwards after hospitals were exposed to competition created from the new NHS reimbursement system and the expansion of patient choice. While providers are not explicitly competing for AMI patients because competition in the NHS is limited to the market for elective care, we expect the market-based reforms to result in across-the-board improvements in hospital performance, which in turn will result in lower AMI death rates. To that end, Bloom et al. (2010) looked at NHS hospitals and found that better managed hospitals had significantly lower AMI mortality and that greater hospital competition was associated with better hospital management."

In short, the paper is quite clear on the mechanism. Competition on elective care care improves management which also happens to benefit AMI. Why not use elective care directly? Because hospitals can 'manipulate' statistics around those in a way that it can't with AMI precisely because patients have no choice! In other words, the authors clearly understand that patients have no choice for AMI but this helps rather than hinders them in their research.

Elective patients don't choose hospitals

BLOG: "Less than the half patients surveyed in 2008 even remember being given a choice, and only a tiny proportion made those choices based on data from the NHS choices website."

PAPER: There were three components to the health reform only one of which concerned patient choice but all three of which sharpened incentives for hospitals. Also, even if patients don't remember being given a choice: "since GPs are highly active in informing the destination of most referrals, GPs now play a substantial role dictating how money flows around the post-[reform] NHS."

There are several ways in which the reforms sharpened incentives for hospitals. Pointing to the fact that patients don't remember being given a choice doesn't seriously address whether or not these incentives worked in practice.

No biological mechanism for choice to affect outcomes

BLOG: There is no biological mechanism to explain why having a choice of providers for elective hip and knee operations surgery [...] could affect the overall outcomes from AMI where patients do not exercise choice over where they are treated."

But the paper doesn't ever claim that there is a biological mechanism. It claims there is an economic one via the incentives described above.

They ignore the existing evidence

BLOG: "They sweep aside decades of careful economic theory and evidence which shows why markets do not work in health services"

PAPER: Provides pointers to existing literature (and reviews) and specifically considers the reasons why evidence from the reforms of the mid 1990's, the internal market might not be very useful "because the internal market never created significant financial incentives for hospitals to change their behaviour"

They engage in data dredging and their work should never have been published

BLOG: "if you repeat an analysis often enough significant statistical associations will appear." The work was subsequently published in "the Economic Journal. That it got through that journal’s peer-review process is perhaps indicative of the poor understanding of healthcare and routine data from reviewers of that journal."

I don't see any basis for the first of these claims. Data 'dredging' is a serious problem - but not one that appears to apply to this paper (which shows that the results are robust to many different variations in specification - the exact opposite of the data mining problem). The Economic Journal is one of the world's leading peer-reviewed economics journals. I don't believe that peer review is everything, but simply insinuating that the referees and editors of that journal don't know what they are doing doesn't cut much weight with me.

Bad blogging versus bad science

I could go on to discuss the errors around the second paper where, e.g., the blog claims that they don't control for the mix of operations when the paper actually considers within treatment changes in outcomes (so mix is irrelevant). But I assume that the authors are perfectly capable of further defending their own research.

My point is simply that a blog that is supposed to help improve the public's understanding of the evidence is carrying a post that is pretty misleading about what the papers actually do, what they find and what claims they make about their findings. To my mind, this raises far more concerns about bad blogging than it does about bad science ...


[Disclosure: Steve Gibbons is SERC's research director and I am affiliated with the Centre for Economic Performance]

8 comments:

Jonathon Tomlinson said...

Cooper and colleagues make a lot of assumptions that seem strange to a GP.

1. the idea that reduced LOS was the same as greater productivity (they didn't show that there were more elective procedures being performed)Evidence seems to suggest that reduced LOS is associated with greater costs and mortality: http://www.eurekalert.org/pub_releases/2012-02/aaoo-shs020912.php

2. "we assume that the impact of competition on pre-surgery LOS captured overall improvements in hospital efficiency" That is a big assumption. And they've shifted from productivity to efficiency without explanation.

3. They asssume GPs are making choices about where to refer their patients and that hospitals are responding competitively. As a GP, I am not aware that I have been doing this. My colleagues likewise. If you wanted to show that there was competition driven by GP choice, you would have to interview some GPs to find out about their behaviour.

David Parkin said...

Jonny, your LOS reference (related to one procedure for US Medicare patients) doesn't seem to me to say that at all. My reading of it is that there were lower costs for patients with lower LOS. And worse mortality etc is only mentioned for outpatients, so presumably not for inpatients who had a lower LOS.

The difference between efficiency and productivity is important and even economists often use the terms incorrectly, as if they are synonymous. But in this case the difference doesn't affect the Cooper et al findings, it's just loose terminology.

Jonathon Tomlinson said...

Dear David,
The LOS link was intended to demonstrate that a reduced LOS does not necessarily equate to a saving, and therefore may result in lower efficiencies. Here is another example http://findarticles.com/p/articles/mi_m3257/is_2_66/ai_n58567200/

Cooper et al do not show that reduced LOS resulted in greater productivity - they would have to demonstrate that there was a greater throughput of patients and surgery to do that. Neither do they demonstrate efficiency, e.g. lower unit costs per procedure. All they show is shorter LOS

And still they have failed to demonstrate that anyone (patients or their GPs i.e. people like me) are responding to changes in hospital behaviour that are down to competitive pressure. Reduced LOS may be an attempt to increase profits or save money and be due to greater financial pressure rather than competition from other hospitals,
Jonathon

David Parkin said...

Jonny, I do hope that you are not denying that your first link actually says the opposite of what you claimed and therefore does not demonstrate in any way either your orginal assertion (about productivity) or your new one (about 'savings' and 'efficiencies'). Your new link also does not demonstrate either of those assertions. In any case, drawing conclusions about the economics of NHS hospitals from data about US hospitals makes no real sense given that they have entirely different production, cost and market conditions. I am not arguing that shorter LOS is evidence of greater efficiency (though that seems plausible to me), just that your contrary evidence is weak.

Jonathon Tomlinson said...

David,
Cooper et al are making assumptions about LOS being the same as either efficiency or productivity (they don't specify which)
It is one thing to make an assuption and another thing to prove it.
They are not naive as to the political nature of their claims, and would do well to read Orwell, "It seems to me nonsense, that in a period like our own, to think that one can avoid writing about such [political] subjects. Everyone writes of them in one guise or another. It is simply a question of which side one takes and what approach one follows. And the moore one is conscious of one's political bias, the more chance one has of acting politically without sacrificing one's aesthetic and intellectual integrity"

Steve Gibbons said...

I thank Henry for his accurate reflections on what is actually written in our papers (I am one of the papers’ authors) in response to the blog by Pollock et al.

I response to the comments on this blog regarding our use of "LOS" as proxy for efficiency and productivity in our latest discussion paper, I agree that we use these terms loosely. The paper is, taken at face value, about the way that the relationship between hospital market structure and LOS changed over the period of the 2006-2008 choice reforms. However, this statistical evidence would be completely uninteresting and useless for informing academic debate or policy makers without some theoretical assumptions about what these numbers in the data might mean in terms of 'real world' factors. This much is true of any empirical analysis, not just ours. Our suggestion is that LOS (particularly pre-operation LOS) is one potential measure of efficiency savings (i.e. reducing any slack in the system without imposing additional costs on staff or patients). We drew this assumption from the literature, and our general understanding of the institutional environment (it is, for example, a pervasive theme in the training materials on the NHS institute for innovation and improvement website http://www.institute.nhs.uk/)

On other points raised in the comments, we have answered before the objection that these elective choices are not actually being made by GPs or patients. The evidence in the HES data (we do not report it in the current version papers) is that patterns of choice did change systematically between the pre and post-reform periods. GPS started referring to more providers and patients were less likely to attend their nearest hospital. But what is more important is that the institutional structure and information systems that facilitate choice were put in place in 2006 and it is the hospital response to these institutional changes, the threat of competition from other NHS providers, that is relevant, not whether different choices were actually being made by patients. I guess, if the GP commenting here has not made use of this choice availability, this is because he and his patients were happy with the choices he was making already and with the service being provided in his locality. The HES patient data suggests that this is not true in general of all GPs or all patients in England.

Lastly, I am baffled as to what political intent critics think we have in writing these papers. For sure, we probably all have priors that involve caution about local monopolies, and think that not being offered a choice about where you can go for e.g. a hip operation is unlikely to be a good thing, either for quality or equity. I do not think this makes us rabidly pro-market or right-wing and I am not in favour of privatising the NHS (and nor are any of my co-authors as far as I am aware). Nor do we wish to see some kind of punitive competitive regime imposed on doctors, nurses or other NHS staff. My wife is an NHS consultant, so this would personally not be in my favour. On the contrary, the research is motivated by a general interest in the role of market structure in public service provision, following in a line of academic work in economics on this issue, and for a desire to provide evidence to help inform policy.

Ironically, given the assumptions that people seem to make about our political motivations, the message from both our papers on NHS competition is in favour more local NHS hospitals, not less.

Jonathon Tomlinson said...

Thanks to Steve for commenting. I am glad that he has clarified that LOS is only a 'potential measure of efficiency savings' and not proof of increased efficiency or productivity gain.
Secondly I'm baffled at why a serious researcher should rely on HES data instead of actually asking GPs or patients how they are making choices. Once the provider market is opened up, in whatever field, it seems obvious that people will try new providers for all sorts of reasons including curiosity, convenience etc. To demonstrate the effect of competition on patient choice, you need qualitive data about how the individual choices. They don't have this.
The conclusions about the effect of choice and competition on productivity (or efficiency) carry profound political weight. I don't believe the authors are naive.

Ian Greener said...

Dear all,

I've posted a reply to Steve's comments at http://t1ber1us.wordpress.com/2012/03/08/nhs-competition-and-bad-science-a-reply/

These views are my own, not those of Allyson and Alison.