Evaluation

You’re working hard on the front lines, trying to ensure that more girls and young women have access to innovative violence-prevention and empowerment programs. You are testing new ideas and approaches, taking some steps forward, pausing for a moment, taking stock, and then re-adjusting in a different direction. You are creating safe, inclusive spaces where you consult young women on their ideas about how to engage in community action and advocacy. Indeed, you are “living” reflective practice – or as some in more formal circles call it – you are “evaluating!”

 

Traditionally we might think of evaluation as a requirement from a funder to provide accountability for resources used and proof that there is a cost-benefit to our programs. In a world where governments and funders are demanding increasing accountability and “evidence-based” practice, it is sometimes hard to think more creatively about how to approach evaluation.

 

Evaluations can help at all stages and aspects of a program; you do not always need to wait until the end to do an evaluation. Evaluations can be done at the beginning of a project or program, as a type of needs assessment (called a formative evaluation). Evaluations can also be done at the end to show what was accomplished (called a summative evaluation). Regardless of when an evaluation is done, it can evaluate either the process or the outcomes. However, in its simplest form, evaluation is a process of generating information that describes what you see, experience and learn in your programs, in order to better understand and communicate the value of what you do. This ultimately helps you improve how you’re doing what you do!

 

 

Re-Appropriating Evaluation to Suit Our Learning Needs…

 

More and more community organizations are realizing that they need to “appropriate” evaluation processes so that they meet the needs of not only the “external” funders, but so that they serve to answer key “internal” questions about the results and impacts their programs are having.

 

In this spirit of “appropriating evaluation,” many organizations are interested in moving beyond the measurement of hard outcomes and evidence-based results (e.g. #’s of girls in programs, # of workshops delivered etc.); and shifting the focus to more of a holistic approach of reflecting and experimenting together as an organization.

 

This type of “reflective practice” helps groups:

 

• Gain an increased understanding of problems they are tackling

• Deepen learning and understanding

• Create strengthened networks, new partnerships, and the incubation and launching of new projects

 

At Girls Action, we are experimenting with this type of developmental approach to evaluation. It is a more organic approach that focuses on asking evaluative questions, gathering information and feedback to support developmental decision-making, the testing of new approaches, continuous improvement, and adaptation of programs. It helps look at the organization from the “inside out” instead of the “outside in.”

 

Reflection helps to:

 

• Inform our practice

• Take time to learn from our actions

• Apply key learnings

• Reflect on whether we are achieving our objectives

• Reflect on whether we are aligned with our mandate

• Adapt to changes in internal and external environments

• Adjust our course

• Ensure future actions are informed by what has worked and not worked previously in certain contexts

• Recognize we are always learning and evolving based on our experiences

• Allow multiple voices to share learning and inform future direction

• Promote a constructive and positive organizational environment

• Demonstrate that your program achieved what it set out to do

• Determine how to go forward in your programming and to demonstrate to funders that their donations were used responsibly

 

Here are some reflective products and processes that will help you “look inside” at what you’re doing, how you’re doing it, and how you could be doing an even better or different job.

 

• Make a record of what has been done or accomplished over a period of time

• Encourage cohesion around a goal and get people thinking together about one certain dimension of your program

• Identify the information flow in your program. For example, what do the organizers of your girls’ program need to know from the participants? From the community? Is there enough space for discussion? Are the right people invited around the circle to contribute to the information gathering and discussion?

• Find creative ways to solicit input from diverse stakeholders in your immediate and broader community

• Encourage reflection and action in some of your team meetings

• Debrief after program workshops

• Keep journals of our thoughts about the effectiveness of our work

• Scheduled reflection meetings

• Staff retreats to debrief, learn, and plan for the future

• Document these reflections to archive and maintain the memory of your organization

• Evaluation is most effective when it is useful and participatory. If you do not know how to conduct an evaluation, try getting help from an external advisor. Factor the evaluation into your budgets and reports – it is a great way to add legitimacy to your work.

 

Our key tips around evaluation are:

 

• Evaluation should be FUN and RELEVANT

• Don’t be afraid of it

• Use it to your advantage to inform the practice of the organization and to build legitimacy

 

 

An example of an evaluation method:

 

1. Collect your data!

The following methods are simple and useful. The written materials or notes from reflection sessions become the “data” for your evaluation reports. Remember that you don’t have to use all of these suggestions, just use the methods that work for you and fit with your program.

 

• After each workshop with your girls group, write down one or a few anecdotes – stories of things girls said or did that demonstrated their learning during the workshop.

• Facilitate reflection sessions with the girls halfway through the program and near the end. Ask them what they learned in the program, how they see themselves differently now (if at all), what they liked best and least, if they act differently in school, at home or with friends since the program. Have a co-facilitator take notes and write these answers up.

• Ask the girls to fill out evaluation questionnaires. For younger girls, there are some simple ways to get at answers to the above questions. See the following sample workshop evaluation sheet.

• Do interviews with parents, school staff or community centre staff to see if they have observed any changed behaviours or attitudes in the girls (for example, more confidence, speaking up, or dealing creatively with conflict situations). Write down their answers – this can be a great way to demonstrate the impacts of your program!

 

 

2. Analyse your data

Once you have collected some “data” (often near the end of a season or year), you should analyze it. This means reading through all the questionnaires, interview notes, reflection session notes, and your facilitator’s notes. Do this with your advisory committee, or with your co-facilitators, co-workers, supervisor, or friends. (It is good to have different people’s perspectives to add to the mix.) Give everyone a copy of your program’s objectives.

 

Have a discussion about what you have read. What are the themes emerging from the girls, from parents, and from facilitators? What is most striking or interesting about what people have said? Go through your objectives and find examples in the data that show how you are moving towards your goals. Take special note of new or unexpected outcomes (things that you didn’t expect to happen, but that did – for the benefit of the girls or the program). Also take special note of the important learnings you will take into your program next time around.

 

3. Write a report or write up your notes

Write an evaluation report or if you don’t need to do this formally, just write up your notes from the analysis. In an evaluation report, explain how you collected the data and what the outcomes of your girls’ program were. A good format is to write your first outcome in a box and then underneath write quotes from girls or parents or facilitators that give examples of that outcome. Do the same for each of your outcomes (2 to 5 are enough!). You can also write a section called “observations” that explain what you have learned about the program, and some of the refinements you will make to the program next time.

 

 

For a Sample Workshop Evaluation: Head Heart Hands, and a Sample Facilitator's Evaluation: ABC Girls' Program, see Appendix 1.