This is a brief review of impact evaluation methods. We consider the: Impact Evaluation Framework for Regional Development Agencies, the English Partnerships Additionality Guide, and the Social Return on Investment method.

We will demonstrate that:

  • it’s vital to realise that the results monitored by projects should not all count towards a measure of impact (as some of those results may have happened anyway);
  • the method of calculating impact may be as interesting as the results of those calculations and, since interpretations may vary, transparency is vital to credibility;
  • an appreciation of context is fundamental to successful impact evaluation.

From the easy-to-measure to the hard-to-prove

Each method builds upon the notion that, in order to evaluate the impact of public policy, we must consider outcomes rather than simply outputs.  That is to say, that simply counting the results of activities (or even the level of activity) is not enough - we must investigate the consequences of those results.

Each method translates gross outcomes to net outcomes. This is in response to the observation that not every observed benefit arises as a direct consequence of the intervention in question. Instead, by drawing on the notion of ‘additionally’ we may begin to question what might have happened without the support. Impact evaluation then, takes us from the easy to measure to the hard to prove. Each method recommends that a ‘theory of change’ be established so that the chain logic from inputs to impacts is explicit.

This logic chain ought to cover the whole policy process from problem to solution. An understanding of the business case for investment is therefore absolutely vital to evaluation.

Additional Impact

The methods do not include the same concepts and have markedly different recommendations on how impact ought to be calculated. Generally speaking, the following elements are considered:

  • Deadweight (aka the reference case): those benefits that might have occurred anyway (not to be confused with 'deadweight loss' that refers to the loss of consumer and producer surplus that occurs when a tax reduces the quantity of trade).
  • Displacement: the increase in benefits that is offset by a reduction elsewhere (e.g. where a business grows at the expense of competitors there is no net benefit to the economy).
  • Substitution: the possibility that people have transferred existing activity in order to benefit from the policy under evaluation (rather than through an existing route to the outcome).
  • Multipliers: the indirect, or 'second-round' benefits that arise as a consequence of the direct benefits.
  • Leakage: the possibility that some benefits accrue to people who are not part of the target group.
  • Leverage: where the intervention induces other stakeholders to contribute additional expenditure (or reduce that expenditure).
  • Unintended effects: benefits or costs that were not anticipated.
  • Attribution: those results that were not the direct result of the policy intervention under question - i.e. those results for which other groups were responsible.

These factors are usually treated as ratios that are applied to the gross value in order to calculate a net value.

The English Parnterships Additionality Guide calculates net impact as the difference between a set of intervention options and a reference case (a counter-factual hypothesis of what would have happened otherwise) where measures of leakage, displace, substation and multipliers are applied to both the invention option and the reference case.

The SROI method requires that outputs or outcomes are converted into sterling by drawing upon financial proxies that may be used to translate intangible social benefits into a standard unit of account. These are commonly the unit costs of alternate interventions or estimates from hedonic pricing models.

Theory into Practice

There is a degree of overlap between these concepts. Substitution may be also appear as displacement (was the alternative activity public or private?) or deadweight (is it fair to say that the benefits would have happened anyway if that result was attributable to another policy?). Attribution and deadweight may also be seen to overlap (again, was the result achieved by another policy?).

Furthermore, linear models that combine these factors may over-simplify the complexity of the underlying situation. For example, it will be difficult to attribute responsibility for the benefits separately to each of those involved where two partners had to work in concert to achieve the outcome (in such a case the benefit might be counted twice - once for each partner). It might also be argued that multipliers ought to be applied to substitution and displacement effects (which in turn will also have second round effects).

In reality the chain of causality will be non-linear, and may even be cyclical, with the outcomes from one stage contributing to those in another. In this situation it becomes very difficult to avoid counting outcomes more than once.

This complexity is particularly relevant when conducting Social Return on Investment analysis as this approach to impact evaluation accounts for the social value that arises from economic value. For example, where an unemployed person gains employment there is a obvious economic benefit to that individual (immediately the salary and, over the longer-term, career opportunities will improve) and also social benefits such as savings on transfer payments (as the individual will stop claiming Job Seekers Allowance and start making income tax contributions).

The rule of thumb suggested in the SROI guidance is that, to avoid double-counting, one should consider: "am I counting the same value, for the same stakeholder, twice?".

Impact is in the eye of the stakeholder

Context and perspective are vital to the interpretation of impact. Since no policy operates in a vacuum, and there will usually be related programmes of intervention, evaluators must consider how the policy interacts with other forms of public support as well as how it impacts the targeted beneficiaries.

Consideration of each stakeholder is designed into the SROI method. Indeed this method essentially advocates that the impact on each stakeholder be considered in parallel with separate impact calculations being made for each. The discussion above highlights the difficulty evaluators face in distinguishing between stakeholders. Indeed it’s not clear how collective benefits (those that may be said to be more than the sum of the individual outcomes - the inward investment that a town enjoys as a result of a better skilled workforce is more than simply the aggregate productivity increase among trainees) ought to be modelled, if at all, although a general "state" stakeholder may be called-upon as a catch-all for those benefits enjoyed by society.

The IEF method incorporates external (i.e. non-RDA) stakeholders under the assessment of "strategic added value". This models the impact that the RDA has upon partners in terms of:

  • Strategic leadership & catalyst: Articulating and communicating regional development needs, opportunities and solutions to partners and stakeholders in the region and elsewhere;
  • Strategic influence: Carrying out or stimulating activity that defines the distinctive roles of partners, gets them to commit to shared strategic objectives and to behave and allocate their funds accordingly;
  • Leverage: Providing financial and other incentives to mobilise partner and stakeholder resources – equipment, people as well as funding
  • Synergy: Using organisational capacity, knowledge and expertise to improve information exchange and knowledge transfer and coordination and/or integration of the design and delivery of interventions between partners;
  • Engagement: Setting up the mechanisms and incentives for more effective and deliberative engagement of stakeholders in the design and delivery of regional and sub-regional priorities and programmes.

By comparison with the SROI method, the notion of Strategic Added Value is less mathematical. SAV does not model the impact on each stakeholder explicitly and instead takes a ‘soft-systems’ approach where the focus is on organisational structures and processes rather than on measuring benefits accrued.

Summary

The discussion of factors of additionality demonstrates that counting gross outputs will not provide a fair perspective and that outcomes must be considered when evaluating impact.

In practice these factors may overlap and combine. An understanding of context is vital to disentangle the interplay of these factors.

These factors will vary again when viewed from the perspective of different stakeholders. A balanced impact evaluation will take this into account.