Program Evaluation 2 – Program Rationale

As previously announced in Program Evaluation 1: Selecting the Program, I am blogging a program review in real-time. So each post will bring you up to date with where we currently are at. By following along, you can see how to do a program evaluation. This post documents the theory of change and the logic model that are the rationale for the program we are reviewing, which is the CCCC Annual Conference. These models will make fundraising easier, particularly if your mission does not produce short-term observable results. If donors understand the theory and logic of what you are doing, they will more readily fund programs that make sense to them.

Theory of Change

The theory of change is all about ensuring that you are doing the right things. It is mostly about effectiveness.

It’s important that you know why you provide a certain program and understand all of the assumptions you have made about it. The theory of change is how you document why you believe this program will result in changes that will help you fulfill your mission. It also helps you find assumptions you may not even be aware that you have made. I’ll be describing the model that I came up with for the conference. If you click on the picture of the model it will open up large enough that you can read  it.

Impact

Since you should only be working on projects that will fulfill your mission, the starting point for a theory of change is your mission statement, which is the impact that you want to have on the outside world. Since we are simultaneously doing a strategic review that will develop a new mission statement for CCCC, I have chosen to draft a statement that reflects the general idea of where I think we will end up. Our desire at CCCC is to help ministries become strong and healthy so they can fulfill their missions. That is both our mission and the impact that we assume our programs and services will make.

Assumed Problem

Why hasn’t our mission been fulfilled already? Why hasn’t the positive change we want to see already taken place? The next step is to consider what is holding your beneficiaries back from experiencing the fulfillment of your mission. Depending on your mission, you may ask, Why haven’t people come to Christ yet? Why are people still suffering? In our case, what is holding ministries back from being strong and healthy and progressively fulfilling their missions as well as they might? There could be many reasons, and if this theory of change model was for CCCC as a whole, we’d document all of them here. But this theory of change relates only to the conference, and the reason that relates to the conference is that lack of organizational knowledge results in ministries having to shift their focus, energy and resources away from their core missions into their non-core areas. Perhaps they don’t know how to receipt properly and get caught in a CRA audit that suddenly requires management’s rapt attention. Maybe their fundraising staff have damaged donor relations by taking donors for granted and they don’t know how to manage relationships.

Although there are assumptions underlying every box in this chart, I chose to highlight here one assumption that is so basic that I thought it should be featured right in the chart. It is the premise that if people had knowledge of how to do something or knew who could help them, most problems could be either avoided or quickly solved so that more focus, energy and resources could be given back to the core mission. They would then perform better and fulfill more of their mission. All other assumptions will be documented when we get to the research questions.

This post was written in real time as the evaluation was in process. For the definitive explanation of how to do a Theory of Change, please see this post.

Assumed Causes

Why does this problem exist? What is going on in the lives of your beneficiaries that they have this problem? In the case of our members, I think there are four underlying causes,  so we list them all. First, people don’t know what they don’t know, and so may be quite unaware that there even are regulations and fundraising practices they should be knowledgeable about. Second, even if they know they should be knowledgeable about a topic, many people are too busy to keep up-to-date with rapid changes. Third, some ministries can get by without direct knowledge by using expert consultants. Others, however, cannot afford consultants and so make do without their knowledge. Finally, many people do not have contact with their peers at other ministries, and so don’t have the opportunity to learn from their peers.

Corrections

When you have identified the causes of the problem, the next step will be state what the corrections for those causes are. The corrective actions are how your ministry will cause an external change that solves the problem and fulfills your mission. For each cause, I have listed a possible correction. There could be multiple corrections for one cause, or several causes may be overcome by the same correction. It’s just a matter of logically thinking through the problem and its causes to brainstorm possible corrections.

Assumed Assets

Assets are what your beneficiaries bring to the table. Think about what you assume your beneficiaries have available to them. In our case, we assume they can communicate in English because we don’t provide translation. We assume they have the time to travel and attend a conference, and we assume they have other assets. We need to check that our assumptions about their assets are true.

Other Attendee Needs

You should recognize that your beneficiaries have other things going on in their lives besides what interests you from a program standpoint. When you know their other needs you might be able to design your program to meet these as well. If their other needs intersect in any way with your program, you have the chance to modify the program so that it meets not only your mission needs, but their related needs as well. This is a huge win for both parties.

For example, many of our members are also members of professional associations and have continuing education requirements. By getting our conference recognized by their association, they can fulfill part of their CE requirements by attending our conference. A good deal for everyone. From the feedback we get, we hear that many people are motivated and refreshed by our conference, so the devotional and plenary sessions are not just nice add-ons to the workshops – they are meeting a real need and making the conference more valuable to the attendees.

Interventions

These are the actual activities you plan to carry out. They document what the program involves. Each activity should tie back to either a beneficiary’s need or a correction for a cause.

Outcomes

Next you show both the short and the long-term outcomes. Outcomes are always the changes that occur outside of the ministry. They are the changes that your beneficiaries achieve. There should be a logical connection that makes sense to people between every step in the chart. When people read your theory of change they should nod their heads, saying “Ahh yes.  That makes sense.”

The Chart

The chart has been colour-coded (although it doesn’t have to be) just to emphasize the different parts. Everything in green is either what CCCC wants or what it does. The orange columns document things from the beneficiaries’ perspective. The problem is in mauve and what the beneficiaries are responsible for is in purple.

Logic Model

The logic model is all about ensuring that you are doing things right. It is mostly about efficiency.

Once you have a theory of change, you can do a logic model. The interventions column from the theory of change comes over to the logic model. The columns are essentially the same between the two models, although I reworded one of the boxes for the logic model.

Inputs

First you document all of the inputs that make the program work. The general categories of inputs are physical assets (buildings), time and money. I’ve been a bit more detailed on this logic model because as I created the model I realized that not only were there benefits for our members, but there were benefits in the program for CCCC too. Since the goal is to ultimately compare inputs against outputs, I thought it only fair to allocate the time that benefits CCCC separately from the time that benefits our members. This simply recognizes that the content created by staff for their workshops can be used in multiple ways beyond the conference itself.

Interventions

As already mentioned, this is the list of program activities and is the same as the list in the theory of change.

Outputs

Outputs represent the completion of the program. When the outputs are done, you can say “We did our part, now the beneficiaries have to do their part.” Outputs could be the number of presentations of the Gospel, or it might be the number of people given glasses. They could be the number of Bibles handed out or the number of Sunday School classes and attendees. Whatever the outputs are, they are what you used the inputs for and did the activities for.

The outputs tell you how busy you’ve been.

Outcomes and Impact

These are the same as in the theory of change model. The outcomes and impact will tell you how effective you’ve been.

Where to from here?

Now that you have your theory of change and logic model, go over them very carefully and think hard about any assumptions you are making. You will want to check all the assumptions. Of course, at least some of your assumptions have turned up in the models themselves. For example, I made assumptions in coming up with the causes of the problem and with the attendees needs which I included in the models, but there are many, many more that I have listed and which form the basis for the research questions, which I’ll be blogging about later.

You now have two models that, put together, will help you check that you are doing the right things right. The next time I post on this program review, I’ll discuss reviewing the literature as a key part the review process.