Monitoring and Evaluation

Monitoring is:

A systematic way of recording Friendship agreement activities.
About collecting information that will help you answer questions about your Agreement
Important that this information is collected in a planned, organized, collaborative and routine way.

You can use this information to report on your activities and to help you evaluate.

How to Conduct Monitoring

First it is extremely important that you make it clear why and what you are monitoring (otherwise it can create feelings of unease, suspicion and mistrust amongst those that are being monitored, and lead to a breakdown in the relationship). The best way to monitor is to be as inclusive as possible, and to get the Timorese Friends to be their own monitors. Better still if Australian Friends work along side the Timorese Friends to implement the activities and see them through. then both Australians and Timorese can be involved in an ongoing monitoring process during the lifespan of the activity. However, if it is not possible to spend longer amounts of time in Timor, then encourage the Timorese friends to conduct ongoing monitoring. This means that instead of Australian Friends appearing at irregular intervals and asking questions about what has been done and by whom and why, the Timorese Friends can keep a very brief record of the scope of the activity, who was involved, what was achieved, who is it used by, what effect has it had, and how long it took to implement. This can then be relayed back verbally to the Australians. If it is possible to keep written records, then these can be relayed back too. If the activities requires amounts of money being spent on tools or labour or equipment or travel, it will be necessary to bear in mind that invoices and tax records etc are not readily available in Timor. Monitoring is usually best done shortly after an activity commences, then at regular intervals to ensure that the activity is progressing according to plan. This can be every two months or it might be every six months. This will depend on the scope of your activity. For example, if the activity is rehabilitating a building, then it will be necessary to keep more detailed records and monitor more closely how the work is progressing, than a short one off activity such as a bread making workshop. However, in both cases it is vital to find out how the activity was received and what impact it is having/has had. This is when monitoring becomes evaluation.

Evaluation

Evaluation is:

A systematic process which assesses whether or not Agreement objectives have been achieved.
About using monitoring and other information you collect to make judgements about your project.
About using the information to make changes and improvements.
Assessing 'value' - or the worth of something - and then taking action makes evaluation distinctive. The results of an evaluation are intended to be used.
A way of finding out the quality of your activities
A way of assessing your work so that you can improve the quality Relationship/Agreement/Activities

While monitoring is routine and ongoing, evaluation is an in-depth study, taking place at specific points in the life of the Agreement

Evaluation has two main purposes:

For learning and development
Monitoring and evaluating your services will help you assess how well you are doing in order to help you do it better. It is about asking what has happened and why - what is and what is not working. It is about using evaluation to learn more about an Agreement’s activities, and then using what has been learnt.

For accountability - to show others that you are effective
Funders and other 'stakeholders' want to know whether an Agreement has spent its money appropriately. There is pressure from funders to provide them with evidence of success. Evaluating from this perspective is about demonstrating and listing achievements.

Why is it important for Friendship groups to monitor and evaluate?

To see if the Agreement is doing what it set out to do
To learn from past experiences
To improve current activities
To assist in future planning
To assess how successful the work is, and what progress the Friendship Agreement is making towards achieving its objectives.
To find out whether or not Friendship members are satisfied with the scope and progress of the Agreement
To find out what the wider community know about the Agreement
To look at the long-term implications of the Agreement, is it sustainable?
To show the communities and organisations that provided resources or funds that the activities of the Agreement had an impact.

Who should take part in an evaluation?

It is vital that an evaluation is as inclusive as possible. Ideally all members of Australian and Timorese Agreements should be involved and their input and ideas sought. The evaluation should at a minimum include:

Present members of the Agreement
Past members of the Agreement who are no longer active
Other beneficiaries of the Agreement
A sample of community members, including members who are not happy with the Agreement
Major funders ( ie Local government officials or key donors)

How to carry out an evaluation

It is vital to create a safe environment so that people feel secure and able to express unease or dissatisfaction. Ensure that everyone knows that the aim of the exercise is to find out as much as possible about the Agreement with a view to improving it and making it work even better. Accentuate the positives but ask for ideas abut how the agreement could be improved, which activities have not worked as well and the reasons for this. It may be that the Timorese participants feel uneasy about criticising any aspects of the Agreement for fear of retaliation or that the activities and resources might be withdrawn. The Evaluation facilitators will need to ensure that it is CLEAR that the Agreement will be on going and the evaluation is about finding out how to improve upon resource exchange, relationship building etc.

Some Key Approaches to carrying out an Evaluation are:

Reviewing all the Agreement records
Holding a series of community meetings both in Timor and Australia to seek and gauge community ideas about Agreement impacts and ideas for improvement
One to one interviewing of key stakeholders (eg the District Administrators, members of the Commissions, Local government officials) using a prepared questionnaire
Small group discussions or group interviews using a prepared question sheet. This is the method that we used in April in Timor and the Timorese participants were very enthusiastic about this participatory process ( see comments about participatory evaluations Evaluation comments PDF File)

Some key questions might be:

What were the short- and long-term impacts of an Agreement?
Who benefited? Who did not benefit?
Was there genuine joint planning and consultation?
Was there genuine human growth on both sides?
How might we change the way we interact?
What are the Agreement’s strengths and weaknesses?
What are the implementation problems?
Why have things worked, or not?
What are the main challenges to be overcome? How might we overcome them?

Evaluation should not only answer questions. It should also prompt fresh thinking within your group. If you have asked the right questions, an evaluation will tell you not only what you have achieved, but also how you did it and what was most effective. It will help you find the areas where improvement or change is needed, and help you to sustain the best Agreement possible.