It suddenly occurs to me that I am in danger of appearing contradictory. I complain that I can't learn a lot from the cross-cutting RDA evaluation but then I suggest that its an example of an area where systematising evaluation across subnational organisations may have big payoffs.
Let me clarify - I think systematic evaluation is better than no systematic evaluation. My major problem with the cross-cutting RDA evaluation is the reliance on user benefit surveys to calculate additionality. The recent practical guide from BIS notes: "To assess the net impact of an intervention, information is needed on the situation that would exist both with and without the intervention. The standard approach is to use a beneficiary survey – asking questions on the impact [...] and getting beneficiaries to estimate what would have happened otherwise. There are clearly limitations with this methodology. A more robust approach is to compare the change in activity and outputs of beneficiaries before and after the intervention against the achievements of a control group (i.e.: people/businesses that would have been eligible for support but did not receive it). However, results are dependent on identifying an appropriate control group, which is not possible in many cases. [...] In addition, a control group approach is usually more expensive than beneficiary surveys. As a result, using a survey of beneficiaries is generally the preferred approach when balancing costs and benefits of the two methods."
I agree with much of this. But the point is that when you are talking about spatial policy there are complex interactions that mean the impacts extend beyond the direct beneficiaries. Further the beneficiaries have absolutely no idea about the nature of these complex interactions. As a result beneficiary surveys are often not a great way of evaluating spatial policy. To be clear, hopefully better than nothing, but not great.
Quite simply, when you are spending large amounts of money on spatial policy (collectively the RDAs have spent around £15bn) by all means use "light touch" beneficiary evaluation for some (even most) of it. But (i) you shouldn't read too much in to the results and (ii) at least some of your evaluation work needs to try and tackle these issues through more robust analysis.
CEP Urban and Spatial Programme Blog (formerly SERC)
Thursday, 13 May 2010
Evaluation and decentralisation
After spending yesterday reading through some of the extensive material on RDA evaluations I suggested that I wasn't much clearer on the impact of RDAs. But it did get me thinking on the relationship between evaluation and decentralisation.
I confess to knowing very little about the history of the RDA evaluation exercise, but I was interested to read from the national audit office that (with the notable exception of EMDA) an assessment in 2006 found: "impact evaluation to be one of the weakest elements of their performance". BIS and the RDAs responded to this by drawing up the framework that I discussed yesterday and RDAs were then asked to use that framework to evaluate properly impacts by 2008. A follow up review in 2006 found that out of 400 projects "only in a very few cases had RDAs used a robust methodology to forecast or measure outputs and outcomes". By December 2007 external "consultants found that only about 40 per cent of [evaluations] were sufficiently compliant with the Framework". With further intervention 70% of projects had been covered by the time of the overall evaluation I discussed yesterday. Subsequently BIS has published further guidance to help make evaluations more uniform in future.
The broader question that all of this raises for the new government is what kind of frameworks for evaluation to put in place when decentralising powers to subnational organisations (LAs, RDAs - or whatever replaces them). The RDA experience would suggest that this is one area where systematising practice across organisations has big payoffs. Despite these obvious benefits, Local organisations are often hostile to this process because it appears to trample on local autonomy. Whatever changes to delivery the new government is planning it will be interesting to see how they try to square this particular circle.
I confess to knowing very little about the history of the RDA evaluation exercise, but I was interested to read from the national audit office that (with the notable exception of EMDA) an assessment in 2006 found: "impact evaluation to be one of the weakest elements of their performance". BIS and the RDAs responded to this by drawing up the framework that I discussed yesterday and RDAs were then asked to use that framework to evaluate properly impacts by 2008. A follow up review in 2006 found that out of 400 projects "only in a very few cases had RDAs used a robust methodology to forecast or measure outputs and outcomes". By December 2007 external "consultants found that only about 40 per cent of [evaluations] were sufficiently compliant with the Framework". With further intervention 70% of projects had been covered by the time of the overall evaluation I discussed yesterday. Subsequently BIS has published further guidance to help make evaluations more uniform in future.
The broader question that all of this raises for the new government is what kind of frameworks for evaluation to put in place when decentralising powers to subnational organisations (LAs, RDAs - or whatever replaces them). The RDA experience would suggest that this is one area where systematising practice across organisations has big payoffs. Despite these obvious benefits, Local organisations are often hostile to this process because it appears to trample on local autonomy. Whatever changes to delivery the new government is planning it will be interesting to see how they try to square this particular circle.
Wednesday, 12 May 2010
Bye-bye RDA's?
Pre-election neither the conservatives nor the lib-dems were particularly pro the RDA's. So I thought I would take the opportunity of a quiet day to see if I could get my head around the PwC evaluation of RDAs (which was published last year) and get a clearer idea of their effectiveness.
The crucial thing that I would like to understand is the net (i.e. additional) impact of RDA expenditure. This differs from the gross (or measured) outcomes because, loosely speaking, some of the stuff that RDAs pay for would have happened anyway. Understanding the net impact is crucial for figuring out how the benefits of RDA expenditure compare to the finanicial costs.
The PwC report provides figures for this. e.g. the RDA's have spent money creating or safeguarding 471,869 jobs (gross). The additionality percentage is 45% so more than half of these jobs would have existed anyhow in the absence of intervention, implying 212,873 jobs (net). Additionality for land remediation is higher at 71%.
The PwC evaluation arrived at these figures by reading through many individual evaluations done by consultants for the RDAs. The RDA evaluations in turn follow the framework set out by another consultancy report: "Evaluating the Impact of England's Regional Development Agencies". This report only sets out a framework for getting at additionality, referring the reader requiring more details to another consultancy report for English Partnerships: "The Additionality Guide". This provides some ready-recknors for additionality taken from a number of different sources. For example, on p.15 you can find figures for deadweight (one component needed to calculate additionality) from the City Challenge report. These figures come from a survey of beneficiaries and project managers.
To be clear, asking beneficiaries of a service (or the people who are paid by government to provide a service) whether it does any good is not a particularly robust evaluation methodology. At the very least, we might worry that they have strong incentives to say that the project did a lot of good.
Moving along, p. 16 of the Additionality Guide provides numbers for additionality from the Neighbourhood Renewals Fund Evaluation. I couldn't find the matching table in the NRF report, but I did find a very impressive looking formula for calculating additionality on p. 62:
AI = [GI x (1-L) x (1-Dp+s)/2]*[1-D]
where GI is gross impact and everything else on the right hand side measures some component of additionality (e.g. Deadweight D).
How did the NRF evalution find out the values of these things on the right hand side? Well, they asked intervention managers and coordinators (in other words the people who are being paid to deliver the service) whether they thought deadweight had been very low, low, medium, high, very high and similarly for the other components.
Notice that these are very difficult questions to answer. At least in a medical trial if someone asks whether you are "feeling" better you can provide an accurate answer. Here, these managers are being asked to answer questions about whether jobs in the area would have been created anyhow (deadweight), whether this has been at the expense of jobs at other firms (displacement) etc. The whole exercise does feel rather circular - after all I was reading the RDA evaluations hoping to find answers to these very questions because I have no idea of the additionality of the interventions. 1,000 pages to find out that I was possibly getting an idea of someone else's best guess of the effects was, should I say, slightly disappointing (and may explain the length of this blog).
But, of course, the story doesn't end there because the PwC evaluation looks at the RDA evaluations which use the framework, incorporating the additionality guide which draws on the NRF report etc. So, it is possible that the RDA evaluations themselves have better evidence drawing on, say surveys of both beneficiaries and non-beneficiaries.
Fortunately, many of the evaluations are available on the web, so I downloaded all the evaluations from Advantage West Midlands that were used by PwC. There are 9 of them comprising another 1,000 or so pages of analysis. Here is what I found:
- Regeneration Zones (£280m): additionality based on interviewing 40 project managers responsible for 50 supported projects out of a total of 300 projects
- Land and property (£261m) surveys on 39 project deliverers, interviews with 12 property developers and surveys on 88/180 occupiers who moved to the new sites
- SRB (£218m) used national SRB figures - which, btw - also appear in the additionality guide and are based on interviews in 10 case study areas
- Clusters programme (£72m) surveyed 751 beneficiaries out of 10,930 businesses
- Skills (£47m) interviewed 80 of 6029 individual beneficiaries
- Rover (£36m) used national ready recknors (I think - I was getting tired by this point)
- Technology Corridors surveyed 210/2052 business, 40/237 tenants, 65/1615 individuals (all beneficiaries)
- PARD (£32m) 38 firms out of 592 assisted
- The Market Towns Initiative, Mercia Spinner and Midlands Excellence projects (all quite small) also interviewed beneficiaries.
In short all these reports are based on surveys or interviews with beneficiaries with no comparison to non-beneficiaries. After 2,000 pages I am no clearer on the additionality provided by RDA expenditure.
This is a little depressing because, to make it very clear, I think that the careful evaluation of policy is important. Increasingly, I begin to think the problem is that government funded evaluations of spatial interventions simply try to answer too many questions (this point may apply more generally). I think we need to focus effort on getting a proper understanding of the impacts on a smaller range of outcomes for the more major policy interventions. Less important interventions or outcomes could be subjected to something much more light touch. It could be cheaper, and it certainly should be more convincing on what works. Both important things in a time of budget cuts.
Talking of cuts, I am still no clearer on RDA effectiveness ...
The crucial thing that I would like to understand is the net (i.e. additional) impact of RDA expenditure. This differs from the gross (or measured) outcomes because, loosely speaking, some of the stuff that RDAs pay for would have happened anyway. Understanding the net impact is crucial for figuring out how the benefits of RDA expenditure compare to the finanicial costs.
The PwC report provides figures for this. e.g. the RDA's have spent money creating or safeguarding 471,869 jobs (gross). The additionality percentage is 45% so more than half of these jobs would have existed anyhow in the absence of intervention, implying 212,873 jobs (net). Additionality for land remediation is higher at 71%.
The PwC evaluation arrived at these figures by reading through many individual evaluations done by consultants for the RDAs. The RDA evaluations in turn follow the framework set out by another consultancy report: "Evaluating the Impact of England's Regional Development Agencies". This report only sets out a framework for getting at additionality, referring the reader requiring more details to another consultancy report for English Partnerships: "The Additionality Guide". This provides some ready-recknors for additionality taken from a number of different sources. For example, on p.15 you can find figures for deadweight (one component needed to calculate additionality) from the City Challenge report. These figures come from a survey of beneficiaries and project managers.
To be clear, asking beneficiaries of a service (or the people who are paid by government to provide a service) whether it does any good is not a particularly robust evaluation methodology. At the very least, we might worry that they have strong incentives to say that the project did a lot of good.
Moving along, p. 16 of the Additionality Guide provides numbers for additionality from the Neighbourhood Renewals Fund Evaluation. I couldn't find the matching table in the NRF report, but I did find a very impressive looking formula for calculating additionality on p. 62:
AI = [GI x (1-L) x (1-Dp+s)/2]*[1-D]
where GI is gross impact and everything else on the right hand side measures some component of additionality (e.g. Deadweight D).
How did the NRF evalution find out the values of these things on the right hand side? Well, they asked intervention managers and coordinators (in other words the people who are being paid to deliver the service) whether they thought deadweight had been very low, low, medium, high, very high and similarly for the other components.
Notice that these are very difficult questions to answer. At least in a medical trial if someone asks whether you are "feeling" better you can provide an accurate answer. Here, these managers are being asked to answer questions about whether jobs in the area would have been created anyhow (deadweight), whether this has been at the expense of jobs at other firms (displacement) etc. The whole exercise does feel rather circular - after all I was reading the RDA evaluations hoping to find answers to these very questions because I have no idea of the additionality of the interventions. 1,000 pages to find out that I was possibly getting an idea of someone else's best guess of the effects was, should I say, slightly disappointing (and may explain the length of this blog).
But, of course, the story doesn't end there because the PwC evaluation looks at the RDA evaluations which use the framework, incorporating the additionality guide which draws on the NRF report etc. So, it is possible that the RDA evaluations themselves have better evidence drawing on, say surveys of both beneficiaries and non-beneficiaries.
Fortunately, many of the evaluations are available on the web, so I downloaded all the evaluations from Advantage West Midlands that were used by PwC. There are 9 of them comprising another 1,000 or so pages of analysis. Here is what I found:
- Regeneration Zones (£280m): additionality based on interviewing 40 project managers responsible for 50 supported projects out of a total of 300 projects
- Land and property (£261m) surveys on 39 project deliverers, interviews with 12 property developers and surveys on 88/180 occupiers who moved to the new sites
- SRB (£218m) used national SRB figures - which, btw - also appear in the additionality guide and are based on interviews in 10 case study areas
- Clusters programme (£72m) surveyed 751 beneficiaries out of 10,930 businesses
- Skills (£47m) interviewed 80 of 6029 individual beneficiaries
- Rover (£36m) used national ready recknors (I think - I was getting tired by this point)
- Technology Corridors surveyed 210/2052 business, 40/237 tenants, 65/1615 individuals (all beneficiaries)
- PARD (£32m) 38 firms out of 592 assisted
- The Market Towns Initiative, Mercia Spinner and Midlands Excellence projects (all quite small) also interviewed beneficiaries.
In short all these reports are based on surveys or interviews with beneficiaries with no comparison to non-beneficiaries. After 2,000 pages I am no clearer on the additionality provided by RDA expenditure.
This is a little depressing because, to make it very clear, I think that the careful evaluation of policy is important. Increasingly, I begin to think the problem is that government funded evaluations of spatial interventions simply try to answer too many questions (this point may apply more generally). I think we need to focus effort on getting a proper understanding of the impacts on a smaller range of outcomes for the more major policy interventions. Less important interventions or outcomes could be subjected to something much more light touch. It could be cheaper, and it certainly should be more convincing on what works. Both important things in a time of budget cuts.
Talking of cuts, I am still no clearer on RDA effectiveness ...
Tuesday, 13 April 2010
Coalfield Regeneration
Catching up on some reading - specifically, the report on Regenerating the English Coalfields by the Committee of Public Accounts. Its conclusions on the effectiveness of interventions make for depressing, although not totally surprising reading.
The interventions have spent £630m so far on bring 54 coalfield sites back in to working use. But the committee "[has] serious concerns about the value for money of the coalfield initiatives. The Department does not know what improvement the initiatives had made to the lives of people living in the coalfield areas, as it does not have a robust assessment to prove to us the true number of additional jobs created. Nor does it know the business occupancy rates for employment space on the redeveloped sites, or the number of people from former coalfield communities who have benefited. The number of jobs the initiatives had helped to create could be anywhere between 8,000 and 16,000. We are concerned that public money has been invested to create jobs that would have been created anyway."
As I said, depressing - but not that surprising - once again lots of money on new buildings (physical regeneration represents a disproportionate amount of the money) has very unclear impacts on individual deprivation.
I am not that surprised about the fact that net job creation is poorly documented - this problem bedevils the assessment of RDA spatial interventions as well.
The interventions have spent £630m so far on bring 54 coalfield sites back in to working use. But the committee "[has] serious concerns about the value for money of the coalfield initiatives. The Department does not know what improvement the initiatives had made to the lives of people living in the coalfield areas, as it does not have a robust assessment to prove to us the true number of additional jobs created. Nor does it know the business occupancy rates for employment space on the redeveloped sites, or the number of people from former coalfield communities who have benefited. The number of jobs the initiatives had helped to create could be anywhere between 8,000 and 16,000. We are concerned that public money has been invested to create jobs that would have been created anyway."
As I said, depressing - but not that surprising - once again lots of money on new buildings (physical regeneration represents a disproportionate amount of the money) has very unclear impacts on individual deprivation.
I am not that surprised about the fact that net job creation is poorly documented - this problem bedevils the assessment of RDA spatial interventions as well.
Thursday, 1 April 2010
Public Sector Relocation
It's been a very busy few weeks and I am only now getting the chance to catch up with a number of things, including the budget.
One of the big spatial issues was the decision to continue the process of reallocating jobs out of central London.
This may well deliver long term cost savings (although the case is often overstated by muddling in efficiency savings with relocation savings) but the evidence on its impact on regional economies is essentially non-existant.
The direct impact on local economies come from the "multiplier" effects of civil service expenditure on local suppliers (including that of civil sector employers). These effects are clearly positive. Offsetting this are the indirect negative effects from the fact that relatively high public sector wages mean that public sector employment crowds out private sector employment. Early reports assumed this crowding out would be complete, later reports assume it will not be. But these are simple assumptions backed up by some multiplier modelling. As far as I can see systematic ex-post evaluation of the local impact of previous relocations is not available. More on this once I have had a chance to dig a little deeper.
One of the big spatial issues was the decision to continue the process of reallocating jobs out of central London.
This may well deliver long term cost savings (although the case is often overstated by muddling in efficiency savings with relocation savings) but the evidence on its impact on regional economies is essentially non-existant.
The direct impact on local economies come from the "multiplier" effects of civil service expenditure on local suppliers (including that of civil sector employers). These effects are clearly positive. Offsetting this are the indirect negative effects from the fact that relatively high public sector wages mean that public sector employment crowds out private sector employment. Early reports assumed this crowding out would be complete, later reports assume it will not be. But these are simple assumptions backed up by some multiplier modelling. As far as I can see systematic ex-post evaluation of the local impact of previous relocations is not available. More on this once I have had a chance to dig a little deeper.
Friday, 12 March 2010
High Speed 2
In September last year, I commented on what I felt we would know after the various studies for HS2.
My starting point was that: "The direct user benefits (i.e. the benefits to people making journeys) are potentially quite large. Unfortunately, so are the costs. Both costs and benefits are highly uncertain." The government has now placed numbers on this suggesting that the project will generate £2 for every £1 spent. That might sound like a good return, but the Eddington report found "transport schemes can deliver overall benefits averaging £4 per £1 of government expenditure". So there are certainly many alternative projects which would generate better returns.
Next, I suggested that: "the wider benefits - e.g. 'regenerating the north' - are even more uncertain. I suspect the only thing we can say with any certainty is that they are likely to be overstated." Claims that Birmingham's economic output will increase by 6% suggest this process has already started.
Finally, I pointed out two certainties. First, the environmental impacts are not large and could well be negative (HS2 predict a change in average annual emissions in a range from -0.41 to +0.44 million tonnes, equivalent to just +/-0.3 per cent of current annual transport emissions) The second certainty is that any new route will not be commercially viable and will need large government subsidies (HS2 predict the cost is £30bn). The government suggest they will expect fair contributions - and point to developers and local government. Of course, most of the benefits are to passengers experiencing faster journeys so fair contributions means higher fare contributions (no pun intended). We shall see, but I can't imagine higher rail fares being that popular with voters ...
My starting point was that: "The direct user benefits (i.e. the benefits to people making journeys) are potentially quite large. Unfortunately, so are the costs. Both costs and benefits are highly uncertain." The government has now placed numbers on this suggesting that the project will generate £2 for every £1 spent. That might sound like a good return, but the Eddington report found "transport schemes can deliver overall benefits averaging £4 per £1 of government expenditure". So there are certainly many alternative projects which would generate better returns.
Next, I suggested that: "the wider benefits - e.g. 'regenerating the north' - are even more uncertain. I suspect the only thing we can say with any certainty is that they are likely to be overstated." Claims that Birmingham's economic output will increase by 6% suggest this process has already started.
Finally, I pointed out two certainties. First, the environmental impacts are not large and could well be negative (HS2 predict a change in average annual emissions in a range from -0.41 to +0.44 million tonnes, equivalent to just +/-0.3 per cent of current annual transport emissions) The second certainty is that any new route will not be commercially viable and will need large government subsidies (HS2 predict the cost is £30bn). The government suggest they will expect fair contributions - and point to developers and local government. Of course, most of the benefits are to passengers experiencing faster journeys so fair contributions means higher fare contributions (no pun intended). We shall see, but I can't imagine higher rail fares being that popular with voters ...
Thursday, 4 March 2010
The rural economy
The Rural Advocate has published his 2010 report. It focuses on how to keep young people in the countryside and includes a number of recommendations including around affordable housing, employment and skills, transport and communications.
I have written on rural housing and Digital Britain before and there is nothing new in the report that has changed my thinking on that.
My more fundamental question around this year's report is why should it be a policy objective to keep young people living in the countryside? It is hard to create jobs in rural places because they are less productive (if they were as productive as jobs in urban areas, then why would firms ever pay downtown rents?). Overcoming this productivity disadvantage requires more than just (very expensive) investment in broadband and more mobile coverage. More (heavily subsidized) public transport can only do so much to allow people to commute to jobs - rural areas close to bigger cities are usually expensive, while smaller towns struggle to generate large numbers of jobs for surrounding rural areas.
In short, we can spend a large amount of money to equalizing coverage of broadband, mobile networks and public transport but (likely) failing to generate significant numbers of rural job opportunities. Alternatively, we can recognize that, at least since the industrial revolution, rural to urban migration has played a crucial role in improving many young people's life chances. Perhaps we should be focusing on what we can do to help encourage and support it.
I have written on rural housing and Digital Britain before and there is nothing new in the report that has changed my thinking on that.
My more fundamental question around this year's report is why should it be a policy objective to keep young people living in the countryside? It is hard to create jobs in rural places because they are less productive (if they were as productive as jobs in urban areas, then why would firms ever pay downtown rents?). Overcoming this productivity disadvantage requires more than just (very expensive) investment in broadband and more mobile coverage. More (heavily subsidized) public transport can only do so much to allow people to commute to jobs - rural areas close to bigger cities are usually expensive, while smaller towns struggle to generate large numbers of jobs for surrounding rural areas.
In short, we can spend a large amount of money to equalizing coverage of broadband, mobile networks and public transport but (likely) failing to generate significant numbers of rural job opportunities. Alternatively, we can recognize that, at least since the industrial revolution, rural to urban migration has played a crucial role in improving many young people's life chances. Perhaps we should be focusing on what we can do to help encourage and support it.
Subscribe to:
Posts (Atom)