Evaluation is about thinking about what you have done, whether it has gone well and what you might do differently in the future. Pretty much everyone would agree that this is a good idea. No one wants to keep doing the same things even if they don’t work. But in practice it isn’t always that simple. This article will introduce you to an approach for evaluating your programme and help you to see that evaluation is not an optional extra.
Taken from The Careers Leader Handbook by David Andrews and Tristram Hooley.
Reviewing existing evidence
Before you start any evaluation, you should look for existing evidence that might inform what you do and ask yourself the following questions.
- Has anyone already evaluated a similar programme anywhere else in the world?
- Does any of the existing literature give you any ideas about how best to deliver the programme or activity?
- Does any of the existing literature give you any ideas about how to approach your evaluation?
- Has any research been done that might underpin your thinking about what should work or why it should work?
- Can you use any existing research to give you baseline data?
Evidence is likely to be particularly useful when you are introducing something new into your programme. For example, if you are going to build employer mentoring into your careers programme you might find it useful to consult Tristram’s publication Effective Employer Mentoring. Once you have reviewed the evidence you can start to build, develop or change your programme to bring it into line with what the evidence suggests.
Developing a theory of change
The most important work that you do in evaluation is thinking about what you are doing, how you are doing it and what you hope will happen as a result. A theory of change describes the steps that you have to take to make something happen.
If we apply a theory of change to an aspect of your careers programme like helping students to transition to university we end up with a theory of change that looks like this.
The example above is straightforward because it is about helping a student to access a choice that they are already interested in (going on to HE). But, when you are thinking about your whole programme it gets more difficult to define the outcome.
An example of a theory of change for your whole programme might therefore look something like this.
Obviously, many of the links in this theory of change are easier to describe than they are to achieve or measure. None the less, it is important to start with a strong theory because it helps you to think about what you are trying to do and therefore about what you are evaluating against. So, in this case we have three ultimate aims.
- Students achieve their career aspirations.
- Students build successful lives.
- Students make good use of their education.
As you start to think about these things you might want to refine them and consider how they might be measured.
If we take the outcome of students achieving their aspirations, we can see that if we are going to measure this, it is going to require a school or college to record students’ aspirations while they are in education. It is also going to require some way of following up with these students to find out how far they have realised these aspirations a few years later. This might be difficult to do in practice, but just because it is challenging to measure doesn’t mean that the aim isn’t worthwhile. What is more, you might also want to reflect on whether it is OK for individuals to change their aspirations as they learn and develop. In this case you might want to adjust the aim to be something like ‘students achieve their career aspirations or are happy with the way that their career develops’.
As this example shows, building a theory of change can be challenging. It is likely to be something that you continue to develop and iterate as you learn more and think about your programme and what you are trying to achieve. But, even if it changes, it is still critical to define your theory of change if you want to evaluate your programme. A theory of change creates a road map for what you are trying to do and allows you to see if you are on track.
Deciding on your evaluation approach
Once you have developed your theory of change you need to think about what kind of evaluation you are trying to do. There are two main types of evaluation.
- A process evaluation (sometimes called a formative evaluation). This focuses on whether things are working in the way that you hoped and asks you to think about what is going wrong and how you might do things differently. A process evaluation is developmental and will help you to improve the quality of your programme.
- An outcome evaluation (sometimes called a summative evaluation). This focuses on what has happened because of the intervention that you have put in place. This type of evaluation asks you to think about whether anything different has happened because of what you have done. It also enables you to think about whether the impact that you’ve had has been worth the effort.
At this stage you should also be thinking about what you are going to evaluate. It might be tempting to decide that you want to evaluate everything to do with your programme, but really understanding what is going on with every element is likely to be difficult and time consuming. While you might try to capture the overall effectiveness of your programme, you will probably want to focus in on one or two elements, usually new things that you have just introduced or older things that you are considering getting rid of or changing. To give you the focus that you need you should define your evaluation question (or questions) before you start. An evaluation question should be specific rather than general and should be designed to be something that you can answer through your evaluation.
So don’t ask: Does our programme have an impact?
Do ask: Does providing students with labour market information result in them having broader ideas about possible careers?
As the example above shows, setting an evaluation question is a really important part of designing your evaluation. It will work alongside your theory of change and guide the analysis and reporting of your evaluation.
In a nutshell
- Evaluation is a process. It is something that you should be doing all the time rather than a one-off event.
- You should make sure that your evaluation plans are realistic and that you follow through on them.
- Developing a theory of change and some good evaluation questions is at the heart of effective evaluation.
- Effective evaluators make good use of existing data before they start trying to collect new data.
- There are lots of different ways to collect data. It can be useful to use a range of approaches to allow you to gather different kinds of data.
- Make sure that you analyse your data and write up and share your evaluation report.