• be accountable to the stakeholders who supported the development and implementation of the policy

• identify ways of improving a policy by determining what works what doesn’t work and why, improving the usefulness of the resources and materials needed to implement a policy and assessing the extent of compliance with a policy.

There are two basic types of evaluation. Process evaluation monitors the implementation of your policy. It examines the procedures and tasks involved in carrying out a policy in an effort to determine “what is happening to whom.”

Process evaluation includes

• tracking the number of people affected by a policy (e.g., number of beneficiaries, number of infractions)

• tracking the quantity and type of activities carried out to implement a policy

• assessing the quality of the activities carried out to implement your policy (e.g., participant satisfaction with a policy training workshop)

• tracking modifications to your policy over time.

Outcome evaluation assesses the results or impacts of a policy, both intended and unintended, to determine whether or not the policy made a difference. Outcome evaluations can assess both short-term results as well as  longer-term results.

In examining the Roadmap for Policy Change used as the framework throughout this Toolkit, there is another aspect of evaluation that is not part of Step 8 in the process: the vertical text to the right side “Where did the process break down?”

Where did the process break down?
Along the road, there is potential for the process to get stalled or actually fail. When this happens, it is recommended that the group working on advancing it, perhaps a committee or coalition, take the time to reflect on WHY the break down occurred. This de-briefing will assist in generating insight that can be applied the next time around – either with this group or another in your community or perhaps in another community if there is a way to share your findings. In reflecting on what happened, it may be insightful to talk with other stakeholders who have been involved to this point to get their take on the situation. When a “breakdown” occurs, it will likely be necessary to make a decision. Will you and your group:

• attempt this step again, applying what was learned to date, or
• postpone another attempt, or
• abandon the policy change process.

This type of a review is called a process evaluation and reflects attempts at mid-course correction.

Collecting Evaluation Data for Stakeholders
At the outset of the policy development process it will be important to identify what the evaluation questions are that stakeholders will be interested in answering. Methodology needs to be set in place to gather this information in a reasonable and timely manner. Once gathered, determine how the information and related findings will be shared with the stakeholders. Typically, the planning for evaluation would happen at the outset of the process, the information gathering happens throughout and the sharing of what has been learned upon completion.

TOOLS: Evaluation Plan Template

• Introduction: This section introduces the major sections of the plan as well as the primary people involved in writing the plan.
• Background: This section describes any information which is needed to provide the reader with an understanding of the background of the interactive multimedia that is being evaluated.
• Purposes: This section thoroughly describes the purposes of the evaluation. A single plan can address a variety of purposes, but all must be delineated clearly. Evaluation is always a political process and all parties must accept the purposes for the evaluation to be successful.
• Limitations: This section spells out any limitations to the interpretation and generalizability of the evaluation. It should also describe potential threats to the reliability and validity of the evaluation design and instrumentation.
• Audiences: This section specifies all the primary and secondary audiences or consumers of the evaluation. In general, it is recommended to open the evaluation up to as many people or agencies as the client will allow.
• Decisions: This section is probably the most difficult, but it should be included if the evaluation is to have meaningful impact on decision-making. Trying to anticipate the decisions which can be influenced by an evaluation takes creativity and trust. Many developers do not wish to anticipate negative outcomes for their efforts, but these too must be considered.
• Questions: A key element of a sound evaluation plan is careful specification of the questions to be addressed by the evaluation design and data collection methods. The clearer and more detailed these questions are, the more likely that you will be able to provide reliable and valid answers to them.
• Methods: This section describes the evaluation designs and procedures. There are scores of designs and hundreds of procedures which can be used. The keys to success are matching these options to the purposes and questions of your client and keeping within the budget and time line of the study.
• Sample: This section specifies exactly which students, trainers, and other personnel will participate in the evaluation. If necessary, a rationale for sample sizes should also be included.
• Instrumentation: This section describes all the evaluation instruments and tools to be used in the evaluation. Actual instruments should be included in appendices for review and approval.
• Logistics: This section spells out who will be responsible for the various implementation, analysis, and reporting aspects of the evaluation.
• Time Line: This section presents the schedule for implementation, analysis, and reporting of the evaluation.
• Budget: This section "costs out" the finances for the evaluation.

SOURCE: The University of Georgia, College of Education.
it.coe.uga.edu/~treeves/edit8350/EPT.html  

RECOMMENDED RESOURCES
Basic Guide to Program Evaluation - www.managementhelp.org/evaluatn/fnl_eval.htm
Reporting Results

TIPS
Here are some important points from THCU to consider when designing an evaluation for a policy.

• Link your evaluation measures to the goals and objectives of the policy.
  
  
• Select easy, realistic methods to collect evaluation data, such as questionnaires, surveys and focus groups. Make sure that data is collected systematically from the moment the policy is implemented. Don’t wait until a policy has been in place for a long period of time before starting to evaluate its impact.
  
• Review the evaluation results on an ongoing basis to guide appropriate changes and modifications to the policy.
  
  
• Evaluations should include a mix of both quantitative (numerical) and qualitative data. Both types of information are needed to fully understand the impact of a policy.
  Effective healthy public policies are not static; they are flexible enough to incorporate insights gained from past experience while responding to future developments and trends.

When the data has been gathered, analyzed and implications generated, write up the process and results in the appropriate format for the intended recipients. As with any strategic communication, customize the report for the audience: what do they need to know?   

THCU offers these ideas regarding what you might reflect on:

Looking back

• Is the situation better than it was before the policy was implemented?
• If the policy was not as effective as anticipated, why not? What could have been done differently?
• Are people who were involved in the policy development and implementation process happy with the results of their efforts?
• Do the people affected by the policy have a favorable view of the policy? If not, what can be done to address their concerns?

Looking ahead

• Does more need to be done to implement the policy?
• Are there any foreseeable developments that may affect the policy?

Conclusion
The Health Communication Unit (THCU) at the Centre for Health Promotion, University of Toronto offers this overall summary of the policy development process. THCU states that “ policy development does not require a new set of skills. It does, however, require looking at a problem or issue in a different way, being persistent and flexible and working with others. It involves planning and systematically approaching a process that can, at times, be unclear and unpredictable. While policy is not a panacea for all community problems, it is an effective option that has the capacity to bring about far-reaching change. In spite of the inherent challenges, the policy development process has tremendous potential to involve people and give them ownership over some of the issues that affect them. The key to success lies in helping people realize that they have the power to bring about change and providing them with the resources to do so.”