Planning is fundamental to successful evaluation. Considering these questions will help you to understand what you need to discuss with a contractor so that they can design an evaluation effectively.

What is the objective?

This ought to be defined in terms of the key questions that the evaluation ought to answer. The choice of question will determine which methods are used and what data is gathered. Typical objectives include wanting to:

  • have evidence to attract funding or investment
  • identify what has worked and what hasn’t to make improvements
  • share lessons learned with partners
  • build plan for future activity

Clearly a different approach to evaluation would be required to answer each of these questions.

In order to define the key evaluation question(s) it might be useful to consider:

  • Who is the audience is for the evaluation (your board of directors, external funders, or participants)?
  • What will you do with the evaluation once it’s complete?
  • What decisions will the evaluation help you make?

Who are your stakeholders or beneficiaries?

Different people will value your work for different reasons. Those reasons won’t always overlap and may even conflict. The evaluation might reach a different conclusion based upon the perspective you take. Answering this question will help you to determine scope and focus - indeed it’s sometimes useful to think about the maximum possible range of beneficiaries just so that you can determine those whom won’t be considered.

Thinking about stakeholders will also help with the preceding question. Often the stakeholder may also be a funder. It’s important in these cases to have the relevant people involved in the evaluation from the outset. This means discussing the objectives and proposed methods with them. Not only will this give your conclusions greater credibility with the target audience, but it will also help to ensure relevance. Further, they might be able to offer insight or resources from other work they’ve seen.

What can we do internally, where do we need support?

You can expect good evaluation to cost at least 10% of the operating costs of the service being evaluated. It not necessary to outsource all of the evaluation work. Even where you are contractually obliged to appoint an independent consultant it is still possible for you to be involved.

External support will typically be need for technical aspects such as:

  • establish a research methodology or epsitemological framework for answering your key evaluation questions
  • designing primary research (sampling, questionnaires, or topic guides)
  • devising and analysing statistical models to establish behavioural change, causal effects or to financial proxies for non-traded costs and benefits

Whereas internal work might focus on activities like collecting monitoring data.

As discussed above, it is vital that internal staff are involved in the conversations with stakeholders if the evaluation is going to be useful in future.

Do we understand the evaluation design?

In order to interpret the results of the evaluation it will be vital for you to understand how those results were reached. This doesn’t necessarily mean, for example, that you need to know how to calculate a “two-stage Heckit regression”. You do at least need to know why this method was used (i.e. in response to selection bias) so that you know that the findings do account for the fact that sample of people consulted was biased. This is important for forming your own judgements about the results and for dealing with criticisms from others.

A empirically flawless method is useless if none of your audience can make sense of the results. That’s not to say that you should avoid technical research methods; indeed this question might be better phrased as: what do we need to learn to understand the evaluation design?

What data do we already have?

Having data available may help to save on the costs of data collection. Data can come from obvious sources such as:

  • the monitoring data you’re contractually obliged to collect,
  • management information you use to direct your work, or
  • feedback forms you’ve used to ongoing service development.

Data can also come from unexpected places like:

  • administrative databases,
  • server access records or log files, and
  • email address books or messages.

You should also expect to provide evaluators with:

  • contact information for consultees - at the strategic and beneficiary level,
  • background information (particularly any work done to scope, appraise, design or commission your service), and
  • contractual and financial records.