Wednesday, 6 March 2013

Problems with Portas Pilots

There has been a spate of recent coverage on the failure of (some) Portas Pilots to spend money quickly. Defenders of the scheme point to the fact that it is better to spend money wisely and that it is too early to tell whether the pilots are working.

To my mind, this misses the wider point that we will never know whether the Portas Pilots work because of the way that central government set up and run the scheme. To understand whether a pilot has a causal impact on the performance of high streets we need to be able to assess what would have happened in the absence of intervention. To do this, we need a control group that can be compared to the group of high streets that got money. Government had a perfect opportunity to create this control group because it invited bids and then only selected a few of the schemes to get funding. As I explained at the time, there were several ways that this set up could be used to allow effective evaluation of the pilot. Government didn't implement these, so we'll never be able to properly assess whether the scheme worked: if Portas Pilots do better than other high streets, it could be because government chose the places with better potential. If they do worse, it could be because government chose the hardest places to 'turn around'. Learning from pilots involves tackling these issues head-on. As this hasn't happened we will never know.

Interestingly, I've been working on the wider issues for an NAO report. One of the responses we received on that report suggested that spatial policy (like the Portas Pilots) can't be assessed using rigorous impact evaluation. As the Portas Pilots demonstrate, this isn't correct. But as long as parts of government aren't willing to embed evaluation right at the start of policy development it might as well be.


1 comment:

Anonymous said...

I too was concerned at the lack of evaluation built into the scheme – especially given the wealth of information on the limitations of area based initiatives which we have at hand. Does your report for the NAO also include the ‘’High Street Innovation Funding’’, which I believe gave 100 local authorities £100,000 to do ‘’something’’. Not only does this appear to be a mandatory amount of money to a mandatory number of councils (who then decide how to divide the money up between projects in their local authority boundary), but my understanding is that DCLG do not have any internal process for capturing the ‘’innovations’’ which would (hopefully?) emerge.

I suspect it is likely that a brief report will emerge on the DCLG website in due course which shows carefully selected case studies which have produced some tangible (short term?) benefits to some people. But is this sufficient for a £10million investment? And does the lack of capacity or willingness for policy learning from the investment highlight concerns that a ‘’what works’’ centre for local economic growth will need to actively encourage more transparency about what has not worked in a particular context and why?