Evaluation

Evaluation should itself be an opportunity for learning and this chapter will examine the role of evaluation and quality assurance in the context of a Partnership. It will offer suggestions for choosing evaluation approaches and tools appropriate to the project. It will also stress the use of a clear and unambiguous vocabulary that is shared and understood by all project participants. The process should be driven by questioning and by the desire for a high-quality project. Quality is partly defined through the impact of the project, but should also extend to relevance, efficiency, effectiveness and long-term sustainability. If evaluative evidence shows that goals have been achieved now is the time for the project to move on to further priorities and quality outcomes. Coupled with this approach is the willingness to change direction should evaluative results indicate that the anticipated outcomes are not being achieved or are not of high quality.

 Defining the role of evaluation

It is generally accepted that it is good practice to reflect on any educational project. This reflection needs to be a well organised strategy which not only fits the individual project but also conforms to the evaluative principles set out by the funder, in this instance the European Commission. A working definition for all Partnerships could be that “evaluation is the systematic application of empirical methods which aims to assess and improve the planning, implementation and impact of a project”.

In the context of a Partnership evaluation should therefore be seen as an important and productive process:

• It supports the project and acts as a check on whether the targets have been met
• It allows the results to be improved based upon judgements made about the value and quality of the project
• It simplifies decision making and can assist with fundamental changes in the project, should these be necessary
• It involves all the project partners and provides a context for open discussion, debate and agreement on project performance
• It should reveal strengths as well as weaknesses and identify obstacles to progress

Measuring the impact and quality of projects is not easy, it is much more straightforward to assess if the outputs (publications, courses, websites) have been produced, and produced on time. Many projects will then supplement this information by frantically circulating questionnaires alongside their products towards the end of the funding period. This does not fit in with the evaluative principles set out by the Lifelong Learning Programme. Evaluation is a process that must not be left to the final stages of the project. By this time outputs and results may have been achieved but without a more searching evaluation. As a consequence, the project team may have failed to put forward or even to note desirable modifications. Thus, opportunities will have been lost and the quality of the project impaired.

An explanation of terminology

A common understanding of terminology is vital in clarifying the purpose of evaluation in the context of a Partnership Project.

Formative and summative evaluation are two commonly used terms:

Formative evaluation is the fundamental tool for improving good practice and therefore the performance of any project. It takes place during the whole lifespan of the project and has the purpose of gaining feedback to ensure that the project is progressing as planned. Formative evaluation is therefore a developmental process which at its best is ‘owned’ not just by the project team but also takes the form of a dialogue with other stakeholders (often referred to as co-development). Summative evaluation happens at the end of the project and probably employs different evaluation tools and methodologies. It too has an important role in assembling what has been learned from the project’s work and making this available to audiences that can benefit directly or are in a position to take the work further.

Two additional terms are internal evaluation and external evaluation:

Internal evaluation is carried out by project partners (or by one partner) and is effectively self-evaluation. External evaluation involves the appointment by the project of an independent ‘expert’ who will follow the progress of the project and apply a range of agreed evaluation tools.

External evaluation need not follow all of the project activities but may sometimes be employed for summative purposes only. An external evaluator does not function as an inspector but should remain reasonably detached from the project in order to present an independent view. External evaluation is dealt with later in this chapter.

 Monitoring and evaluation: the potential for confusion

These two terms are frequently used together but have different meanings. Monitoring is the on-going review and assessment of performance or progress towards stated objectives. Evaluation, as the previous section states, is focussed on the quality of all the project’s processes, outputs and outcomes. This includes the quality of the monitoring process.

Monitoring can be a demanding task for the co-ordinator of a European project where the partners are scattered geographically. Some of the ICT communication tools discussed elsewhere in this document have useful applications in project monitoring. In principle review meetings can be done virtually: Collaborative tools, telephone or video conferences, chats and forums, or messenger systems like Skype can be used to compare and discuss achievements and shortcomings.

Monitoring relies on accurate and timely reporting and co-ordinators may put in place specific reporting formats for this purpose. Monitoring can also be achieved through meetings or by dedicating agenda time in meetings that have a wider purpose (face-to-face or virtual): these are review meetings.

 Stages in evaluation

The requirement for an evaluation strategy is a feature of most, if not all funding programmes inside and outside of the EU. Constructing this strategy is often not a top priority for a project team working to complete a complicated application for a rapidly approaching deadline. However, a convincing evaluation plan shows the funder that the project has the capacity to be flexible. It also helps to demonstrate that the needs and demands of the target group(s) have been taken into account.

Therefore, the evaluation processes suggested by this document aim to:

• Support the management of the project
• Simplify the consultation process within the project partnership
• Predict the development of the project and prepare the project team for pressure points in the course of the project life span
• Assist the project co-ordinator in quantifying results and relating these to the project objectives
• Ensure the quality of the processes, outputs and outcomes of the project

 Evaluation methodologies

Questionnaire

The questionnaire is a commonly used methodology. Questionnaires may follow courses or conferences, accompany newsletters or other project products or feature as ‘pop-ups’ on a project website. However, the return rates from questionnaires may be extremely low.

 Interview

Interviews may be face-to-face, telephone or virtual. If several interviewers are to be involved it is important that their questions are identical and include opportunities to ‘probe’ an interviewee with an additional question or development of an answer. Interviews may generate a considerable amount of material and require detailed analysis.

 Most Significant Change (MSC)

This is a form of participatory monitoring and evaluation that can be planned to occur throughout the project to provide data on both impact and outcomes. It involves the collection of ‘stories’ about significant changes brought about by the project. These stories will usually come from structured interviews with members of the target groups who are asked the same questions across the project partnership. When the stories have been collected, they are analysed by an independent panel whose members identify the most significant factors in the stories. MSC has become a well-used evaluative methodology in international projects.

 Peer Mentoring

There may be occasions in the life cycle of a project where Peer Mentoring has a place as an evaluative strategy to explore issues and challenges facing the project more probably in terms of the project’s processes rather than its outputs. It works well in bringing together workers who are comparatively isolated (either face-to-face or virtually) to explore issues and challenges. It is therefore part of the self-evaluation process based on Action Learning:

 Action Learning

‘…is a process underpinning the belief in individual potential: a way of learning from our actions, and from what happens to us, and around us, by taking the time to question, understand and reflect, to gain insights, and consider how to act in future.’ Krystynia Weinstein (1995): Action Learning – A Practical Guide for Managers

 Peer Mentoring works best if it follows a defined structure: 

• Project workers agree to meet collaboratively on a regular basis
• The group agrees ground rules which include complete openness and mutual trust
• Members take turns in presenting their individual situations complete with highlights, challenges and problems
• Action points are discussed and agreed
• The overall quality of the project is improved

An important part of the process is the way in which the discussion is structured so that groups (focussing on each individual member in turn) give advice and offer options for action in ways which encourage reflection and problem solving. One very important ground rule is for members to avoid simply telling their own stories as a ready means of providing a solution. In an international project it is most unlikely that working circumstances are the same or even similar, so it is important for group members to ask questions as part of the procedure.