Cyclical Monitoring and Evaluation

Why is this important?

​

People working to improve health in communities often face great difficulty evaluating their work.  Many community-based organizations (CBOs), health departments (HDs) and activists have a range of skills and a wealth of experience, but they can face problems:

1. conducting thorough assessments of the overall situation and conditions in communities and systems;

2. examining the processes they use and the quality of their work; and

3. formally evaluating whether their projects are producing a wider impact. 

 

It can be especially difficult to conduct comprehensive evaluation to show that programmatic work is creating change in communities, in systems, or with change agents. It can also be especially challenging to show that the social and environmental changes achieved have improved people’s health.

 

We have found that many CBOs, HDs and activists face substantial knowledge barriers about the contextual conditions that give rise to a health issue or problem they are trying to address. They also face knowledge barriers about the nature and impact of their work.

​

Generally, CBOs and activists have difficulty collecting their own rigorous data to assess thoroughly the social and environmental conditions that cause health problems.  Instead, they tend to design projects based on their best intuition. Then, during their projects they have difficulty examining the processes and quality of their work. They tend to count activities and the number of clients/participants served. They spend a lot of time filling out forms and writing reports to show that they have met performance targets, but rarely go deep in analyzing the data they are collecting.

​

Evaluations often take place near the end of a project, well after the opportunity to adjust course has passed. Consequently, for CBOs and activists, from a project’s beginnings through its completion, the evaluation process is a “black box” that produces limited useful information for developing insightful new approaches or determining if their work has had an impact.

​

Health departments are skilled at identifying health problems through epidemiological surveillance. However, typically they have difficulty assessing thoroughly situations in communities at the “ground level” before they develop a project plan. HDs tend to develop projects by either trying an approach based on their best hypothesis or by modifying models developed in other places. When HDs launch a project in a community, they tend to focus on tracking quantifiable indicators. They have difficulty examining the quality of products and services. While most HDs are skilled at measuring the impact of a project at the individual-level, they have difficulty evaluating whether their project has changed a community as a whole or changed systems that influence health status. Many HDs have limited capacity to document and demonstrate a direct link between community engagement practices and improvements in outcomes for population health. HDs often lack staff resources and the kind of data needed to produce evaluations of long-term impacts.

​

These conditions create problems for donors and funding agencies. When people and institutions give money, they want to know that their money is producing a social return on their investment. They also want to make sure that projects are not producing unintended negative consequences. Since many CBOs, HDs and activists have difficulty conducting deep evaluations of their projects, most funders have difficulty knowing whether their investments are having a desired impact. Thus, evaluation has become a high-stakes activity.

 

Many CBOs, HDs and activists feel pressure to produce evaluations. Yet, they fear that if they conduct a typical evaluation – one of limited scope – it will show little or no social or environmental impact, and consequently they will lose funding. This is especially true when they are expected to show an impact in the short time frame. While such fears are real, they stifle the opportunity to learn and get to the deeper roots of a health issue or problem.

​

These conditions also deter CBOs, HDs and activists from taking risks to address a health issue or problem through more innovative social-environmental approaches, even when they think such approaches may be more effective. Typically, they find that they do not have the opportunities, skills and resources to evaluate a social-environmental project. For example, a CBO that has assessed hypertension in a predominantly Latino community may have good reason to believe that hypertension is primarily a by-product of the ongoing financial stress people in the community experience. The CBO might be interested in taking a social-environmental approach to address hypertension through a poverty alleviation program, but decide against a social-environmental approach because they feel that they lack the capacity to conduct a rigorous evaluation.

 

In our experience, there are several reasons why CBOs, HDs and activists have difficulties conducting comprehensive evaluations. First, social and environmental changes are difficult to measure. Second, most people have little or no training in approaches and methods for measuring social and environmental change or changes in community health. Third, most books and courses on evaluation are either highly technical or “dumbed down” so much that they are of little use.

​

Fortunately, many CBOs, HDs and activists are eager to overcome these barriers and are truly interested in becoming “learning organizations.” Over the past decade, some CBOs and HDs have experimented with community-based participatory research (CBPR). This is a positive development, but there is a scarcity of researchers who know how to do CBPR. Moreover, in most CBPR projects the researchers lead much of the evaluation work based on their interests and skills. This leaves CBOs, HDs and activists with limited power to determine the type of evaluation they want, and to learn to do their own evaluation work.

​

Overall, the main problem is that CBOs, HDs and activists lack the opportunities, skills and resources to conduct their own insightful evaluations. We have found that many CBOs, HDs and activists say they need:

​

• opportunities to think conceptually about their focus

• opportunities to “drill down” into the roots of a health issue or problem

• opportunities to think holistically about how the broader context influences health

• opportunities to think about the rationale for their work or their “theory of action”

• skills to design a rigorous evaluation

• skills to collect useful data, especially simple and reliable process measures

• skills to analyze data and make supportable inferences about cause and effect

• skills to determine what types of outside assistance they need

• resources to support training in evaluation research

• resources to help them integrate evaluation into their work

• resources to provide technical support for their evaluation activities

​

Dynamic Project Cycle Management, using Insight Change Cycles, can make it possible for organizations to overcome these obstacles.