Program evaluation 4 – Research design

A year ago I wrote about the need to check your assumptions.  That is what we are now ready to do in the program evaluation of our annual conference.  By the end of this post, you will see how we at CCCC designed the research content and process of our program review.

Deciding what research needs to be done

Having:

  1. selected the program we want to evaluate,
  2. developed the program rationale (theory of change and logic model), and
  3. completed the literature review,

we next reviewed every part of the program rationale (theory of change and logic model) looking for assumptions that we already know about and assumptions that come to light only as we think more deeply about the program.  The goal is to develop a list of questions we want to answer.  If we can answer those questions, we will know how well the program is working and how to improve it.  We also looked at the research questions that were developed at the start and thought about what questions we need to ask in order to answer the research questions.  Other ways we used to develop our questions included:

  • Reading the literature review to find decision points where we’d like to know what our members think about the options.  In our case, the literature review did not spark any questions that had not already arisen from our analysis of the program rationale, but it did provide ideas for improvement.  Decision points include such things as length, plenary/workshop mix, use of technology, etc.
  • We asked the senior team for their questions.
  • We keep a statistical analysis of each conference and have extensive feedback forms for both the conference as a whole and all of its components.  We ran reports that showed that 25-30% of the participants each year are first-timers, but conference attendance is not growing at that rate, so we’d like to understand better how the decision is made whether or not to come again.  The feedback showed that our conference is considered by most to be the best run conference they attend.  There is only one question of logistics or conference administration to ask.
  • Finally, overall responsibility for the conference rests with me, so I sat back, closed my eyes, and just thought about what I’d like to know more about.

Keep in mind that in any program evaluation you want to essentially answer two questions:

  1. Were we effective?  You need to compare the actual output to the expected output to discover if you did what you wanted to do.  That’s effectiveness.  The theory of change helps you answer the effectiveness question: Did we do the right things?
  2. Were we efficient? You need to compare the ratio of actual output to actual inputs and ask if you were good stewards of your resources.  That’s efficiency.  The logic model helps you answer the efficiency question: Did we do things right?

Specific Questions

We developed specific questions to guide the development of surveys and further research, sorted by category.

Assumed Problem and Assumed Causes questions

We wanted to know how people feel about conferences and the other means they use to stay current in their fields.  We wondered how people challenge and stimulate their own thinking.

Assumed Assets and Other Attendee Needs

These questions related to their habits, preferences and decisions about conference attendance.

Interventions

We wanted to probe what happens pre-conference, during the conference, and post-conference from the attendee’s perspective.  How does an event fit within a person’s larger context?

Short Term Outcomes

Moving away from the conference itself, we wanted to know what people do with the material afterwards.

Long Term Outcomes

Stepping back from any particular conference, respondents to our surveys were asked about their employer’s overall assessment of the usefulness of attending conferences.

Inputs

There were some questions, not for surveys but for further research, that we asked to improve the use of our time, the locations we rent, and the handout materials.

Success criteria

If there are benchmarks or standards available for the program you are reviewing, you would determine what threshold must be crossed in order to consider your program a success.  It is very difficult to set a threshold once you know what the results are because biases will enter the equation!

Ministries might ask questions such as:

  • What recidivism rate is acceptable?
  • What is an acceptable percentage of new believers who are still attending church and being discipled a year later?
  • What percentage of clients should have a job a year from now with at least three months of steady employment?
  • What percentage of our congregation is involved in some form of Christian ministry, whether with our church or with another ministry?

In our case, I cannot find any benchmarks for success, so as part of the research we will attempt to create some benchmarks by asking other associations with voluntary attendance at their conferences about their results.  We will also ask ourselves: In light of the number of ministries that benefit from the conference compared to our resources that it consumes, is it good stewardship to continue running a conference?  This criteria is a bit fuzzy for evaluation purists, but it is good enough for me given that we are primarily looking for ways to improve the conference rather than make a continue/discontinue decision.

 Methodology

We then reviewed each question and decided two things:

  1. Who should we ask this question of?
  2. How should we ask it?

We decided we would find the answers in the following ways:

  • Ask people who have attended the conference at least once since 2006 (includes members and non-members)
  • Ask members who have not attended the conference since 2005 (the conference is designed for our members so we expect them to come; non-members are not our target for the conference)
  • Ask speakers who have presented in three of the past six years
  • Review possible venues at certain locations
  • Analyze our database for attendance patterns
  • Talk with other organizations that run conferences
  • Based on the results of all the above inquiries, select a small group of people for one-on-one interviews to delve deeper to get more insight on any remaining questions.

While we chose to base the review on surveys for the most part, there are many research methodologies you could choose from:

  • Verbal data: Conduct interviews either one-on-one or in groups (focus groups)
  • Client surveys: expectations of services, use of services, satisfaction, rating of quality
  • Outcomes surveys: behaviours, beliefs and conditions that have changed as a result of your service
  • Observational data: Watch and see what happens
  • Archival data: Check data collected from running the program, your own records, plans etc.

While benchmarks may be helpful, there are always so many differences between organizations or even divisions within the same organization – circumstances, conditions, history etc. – that the best comparison is really between your current results and your past results.  Is your performance improving?

You are now ready to go ahead and do the evaluation.

Bookmark and Share

It’s finally here!!! My book is ready for sale

I feel like a new daddy again tonight as I hold my long anticipated (but too frequently edited) book that has arrived from the printer.  The desire for perfection and the conflicting desire to get it in your hands have finally converged and The Church At Work: A manual for excellent church-agency relations has been born!

You can get The Church At Work through the CCCC store (as an e-book and in hard copy with shipping included in the price).  Login as a member to get the member discount.  The newest CCCC publication is designed to help Christian ministries work better together and present a better witness to the relational life of the kingdom of God.  It examines the most strained relationship, that between churches and independent Christian agencies, and provides both a theology for how they are related to each other and a practical set of principles for how to ensure they have good relations.

The research was based on:

  • a theological forum I hosted on church structure,
  • a large attitude survey of pastors and agency leaders and what they think of each other,
  • a study of specific church-agency relationships in which both parties thought the relationship was great, and
  • a demographic survey of 100 agency leaders to find out who they are and how they got to their positions.

The research and illustrations are 100% Canadian, although we know the research is equally applicable in the United States.  My new baby has practical ‘workbooks’ for churches and agencies to work through to examine themselves and help them be more responsible for their part of the relationship.  It also includes a helpful paper that John Stackhouse wrote for the theological forum and sample covenant agreements that some ministries use.

You’ll enjoy reading the stories of people just like you.  Some of the people you may know and others may be new to you, but their stories of successful ministry relationships will inspire you to think again about the possibilities that open up when you collaborate with other ministries.

Jerry White, International President Emeritus of The Navigators, wrote the foreword of this book and says in it:

“This is the most complete discussion on the topic that exists. It is an invaluable resource for pastors and Christian workers across the range of ministries in the broader Body of Christ. Dr. Pellowe not only delves into the scriptures and into history, but also gives practical guidelines for relating among the structures. Anyone who is concerned about advancing the Kingdom of God and who wants to see unity in the Body of Christ needs this study and its practical applications.”

I have no financial interest in the success of this book, so I will shamelessly plug it and say, if you are in ministry, “You need to read it!”

The Church at Work (cover)

 

Bookmark and Share

Strategic planning that will amaze Jesus!

I sure would appreciate getting your perspective on the topic of the day!!!  How much risk are you prepared to accept for your ministry based on faith that God will provide?  Do you have big faith that God will provide and then you act?  Or do you have big faith that God will provide when you act?  I think there is a fine line between having the faith to trust God and being presumptuous in testing God.  The practical issue is how much we factor God into our strategic plans.

In a sermon I heard in Sunday School, Andy Stanley made the point that there is only one time in Scripture that it says Jesus was amazed, and it was the big faith of a Roman centurian of all people that amazed him!

When Jesus heard this, he was amazed and said to those following him, “Truly I tell you, I have not found anyone in Israel with such great faith.”
Matthew 8:10

A personal challenge

Stanley challenges us asking, “What would your life be like if you knew for sure there is a God who is all powerful, who knows you and has promised never to leave you nor forsake you?  What if you had perfect peace and no fears or anxiety about anything?”  What would we dare try for the sake of Christ?  What might we do that would demonstrate a faith that could amaze Jesus?

A ministry challenge

What would it look like if your ministry assumed no limitations that God couldn’t easily overcome?  What if we decided to trust God for the things that keep us from daring to do all that we can dream of?  If we believed God could provide the “if onlys” that we are wishing for?  Better yet, what if we prayed for them?  What if the exploits of our ministries demonstrated a faith that would amaze Jesus just like the centurion’s did?  What if…?

A planning challenge

What if our strategic plans required divine rather than human capacity?  What if our plans were so big, so bold, so far-reaching that we’d have to assign responsibility for executing parts of them to the Lord?

If we can complete the plan ourselves, I’m sure God will allow us to do it ourselves.  But maybe he wants us to plan something so stupendous and awe-inspiring that it gives him a unique role to play.  The Bible is full of stories of how God worked through individuals to accomplish great things.  What if we inserted into our plans something that only God could do?  That means, of course, that by definition our plans will be unrealistic.  Would a staff be bold enough to recommend such a plan?  Would a board be daring enough to approve such a plan?  Just wondering!!

Are we:

  • too cautious?
  • too realistic?
  • too self-reliant?

I admit that as I read these paragraphs, they scare me!  But I think they are also a healthy challenge to think about with the board and staff.

Living by faith

If we try to increase the faith component of our plans, we face the question of whether we should act first and depend on God to provide subsequently or let God act first and be ready to do what he has provided for.

In Acts 1, Jesus tells his followers not to do anything yet, but wait in Jerusalem until the Father sends his promised gift.

In Exodus 14, God told Moses to take the people of Israel to a specific place where they could be easily cornered by the Egyptians.  He basically told Moses to deliberately lead them into a trap!  Moses obeyed God and only then did the Lord part the waters and provide a way of escape.

I think there are some different approaches to the topic and each of them has a biblical basis.  I believe that which one we use will depend on how the board and staff together discern God’s will in a specific matter.

Pray and act with human wisdom

It might seem strange to start with human wisdom, but Scripture takes it seriously and so should we.  The book of Proverbs consists of the best of ‘worldly wisdom’ in that it sets out “the way things are” or “the way things work.”  God has given us great intellectual power to work out for ourselves some of the mystery behind God’s work, such as the hard and soft sciences.  Jesus commended the use of human wisdom in Luke 16:8.  French Arrington discusses the surprising position Jesus takes on human wisdom in his New Life Bible Commentary on Luke, saying:

…the sinful “people of this world” are favorably compared with the “people of the light.”  It is their resourcefulness and their dedication to the achievement of their intended goals that are held up as examples for emulation.  The implication is that the “people of the light,”  whose goal is infinitely more valuable, should be all the more resourceful and dedicated as they work toward their goal of a ‘treasure in heaven that will not be exhausted.”

Furthermore, Jesus’ advises in Luke 14:28 and 31: “Suppose one of you wants to build a tower. Won’t you first sit down and estimate the cost to see if you have enough money to complete it?…Or suppose a king is about to go to war against another king. Won’t he first sit down and consider whether he is able with ten thousand men to oppose the one coming against him with twenty thousand?”  It seems that Jesus believes that thinking ahead is just good common sense.  His point is, “Don’t become a disciple if you are not willing to go the whole way.”

Finally, Paul advises Timothy (1 Tim 5) to drink a little wine because of his stomach and many illnesses.  I’m sure Paul had already done the spiritual thing and prayed for his healing, but he also gave Timothy the best human wisdom he had to help him with his stomach problems.

My position is that, unless he has otherwise directed us, God expects us to use the best of human wisdom in leading our ministries.  Use your noggin!

Pray and wait for God to act

We can also think big, figure out what is humanly possible, pray for God to do the rest and then act when he has made it possible.  There is Scriptural support for this option.  Jesus’ instruction to wait for the gift of the Spirit is one example.  Paul raising the collection for the Jerusalem relief effort is another.  Paul committed to no specific amount, just that he would bring an offering.  The prayers of the saints for Peter in prison are yet another example.  They did not mount an attack on the prison to free him, but were waiting in prayer, and God acted and miraculously freed Peter.

This is also an approach recommended by Jesus: “Ask and it will be given to you; seek and you will find; knock and the door will be opened to you…If you, then, though you are evil, know how to give good gifts to your children, how much more will your Father in heaven give good gifts to those who ask him!” Matthew 7:7, 11.  What Jesus is telling us is: ask, receive, proceed.

Pray and then act in anticipation of God’s action

This is the basis behind faith missions.  Sometimes we have to act first and then we will see God do something wonderful.

  • Joshua had to march around Jericho seven times before he saw God bring down its walls.
  • The disciples handed out the fish and bread not knowing what was going to happen, but God provided.
  • Jesus had to go to the cross trusting that his Father would resurrect him.

Virtually all Christian ministries operate to some degree on faith because their revenue comes mostly from donations, and who knows what will come in next month?  They make long term commitments by hiring staff and acquiring property and trust that money will continue to come in.  But while some are very good at active fundraising, writing persuasive appeals and so forth, others will only do passive fundraising.  This group will pray but not tell anyone of their financial needs.  Hudson Taylor ran CIM this way, on the  assumption that “the Lord will provide.”  God has blessed this approach too.

Some advice

Any of the three strategies may be the appropriate one for you.  My comfort level is to operate the very best we can with the best knowledge we have (option 1) and to think big and wait for God to provide (option 2).

Which should you follow?  It all depends.  I think it comes down to group discernment.  I believe that God will lead the entire leadership team to one of the three options.  The board, the chief staff officer and the senior leadership team should all be in agreement about their position on faith and risk.  If one person feels the ministry should take on more risk with higher faith, then the whole group should enter into a discernment process to determine if they also feel the same.  This is what Paul did with his group.  He alone heard the call of God to go to Europe, but the whole team decided it was the right thing to do.  When Paul felt he had the answer to how to evangelise the gentiles, James and the apostles in Jerusalem discussed the issue and came to a collective decision.

If someone gave your ministry a million dollars, how would you use it?  If you don’t know, then maybe you aren’t ready to receive a million dollars!  If God said he would do anything you asked him to do to help with your ministry’s mission, what would you ask him to do?  If you don’t know, then maybe you aren’t ready to receive his miraculous help.  At the very least, have a list!!

Think BIG, pray HARD and follow the option that seems best to the team.  “It seemed good to the Holy Spirit and to us” (Acts 15:28) needs to be what you say when you announce your decisions.

Bookmark and Share

The strategy of intentional accidents

Everyone knows about penicillin’s accidental discovery, but it wasn’t that accidental, and knowing the real story behind the ‘accident’ will add a great tool to your strategic planning toolbox.  The truth is that Alexander Fleming was not an absent-minded, messy researcher but a brilliant strategist with habitual practices that made discoveries more likely.  By adopting his practices, you will likewise make it more likely that good things will happen to your ministry!

What really happened in Fleming’s lab

 

St. Mary's Hospital

Fleming's lab fills the tower bay. It is the middle level of the three vertical windows.

 

Paddington Station near St. Mary's HospitalPaddington Station, near St. Mary’s Hospital

Wikipedia and other sites describe Fleming and his lab as untidy, messy, or cluttered, but if you visit the lab, which is set up exactly as it was in 1928, you will find out that what appears messy and cluttered is just another aspect of his experimental work.  (Sorry, no pictures are allowed inside the lab.)  The museum guide told me that when Fleming was done with an experiment (the experiment he intended to do), he always left samples of his experiment out for at least several days (as an experiment to find something unintended that he obviously wasn’t looking for).

Another practice was to combine things just to see what happens.  In 1921 he had a cold, and put some of his nasal mucous in a petri dish and saw that it dissolved the bacteria in the dish.  He discovered a natural human antiseptic, lysozyme, which would have guaranteed his place in history even if he had never discovered penicillin!

Practices such as these made it possible for Fleming to be surprised by the unexpected.  He was very intentional about creating opportunities for discoveries to be made.  If this made his lab untidy or cluttered, so be it.

Furthermore, it wasn’t an accident that he just happened to notice something unusual in a sample.  He always inspected the samples very carefully, and this time he noticed a clear spot on the slide where the bacteria that he was experimenting on should have been.  The organism had been killed by something on this one part of the slide, and the killer agent turned out to be a mould that he first called mould juice and (wisely) later called penicillin!

The real story is that the discovery of penicillin was not so accidental.  Fleming deliberately left the cultures out to allow time for something unexpected to happen.  He deliberately inspected the samples to see what, if anything, had developed.  He deliberately followed up an unexpected observation to find an explanation for it.  And he deliberately invested time to think about the implications of what he had discovered.  Anyone who does what Fleming did is likely to discover something sooner or later.

In summary, Fleming’s method was:

  1. Create opportunity to be surprised
  2. Be on the lookout for surprising things
  3. Be curious about the surprises you find
  4. Look for ways to take advantage of the surprises

My discovery using Fleming’s methods

Although I did not realize I was using Fleming’s methods at the time, I discovered the idea for this post following the same principles as Fleming.  It was no accident that I stumbled upon the lab that led to the story that inspired this post.  Here’s an example of Fleming’s method at work:

Create opportunity to be surprised

  • I was on a layover in London while returning home from Zürich on the last leg of my round-the-world sabbatical trip.  I had not intended to visit the lab, but I had made the decision to make the most of my time on this once-in-a-lifetime trip by taking every opportunity to explore.  So when I had two extra hours available before I needed to leave for Heathrow airport, instead of killing time at the airport or the hotel I got out my GPS and looked to see what tourist attractions were nearby.  Fleming’s lab was the closest attraction and it was only a mile a way.  Using the GPS as my guide, I walked to St. Mary’s Hospital and found the museum.

Be on the lookout for surprising things

  • As the guide told the story of Fleming’s discovery, he made it clear that Fleming had deliberately left the cultures out after he finished his experiments with them.  He said it was not a case of being messy, but that this was his usual practice.  That I found surprising.

Be curious about the surprises you find

  • I had never heard this aspect of the story before, so I asked why he would do that.  That’s when I heard about his looking for the unexpected.  This was a surprising idea because I thought scientists always had some concept of what they were looking for, an hypothesis to prove or disprove.  Simply putting something ‘out there’ to see what happens is quite different from the scientific method I learned in high school.

Look for ways to take advantage of the surprises

  • I was not expecting to find something to blog about in that two extra hours I had, but as I heard the true story behind the discovery of penicillin I realized I had accidentally discovered a great leadership practice just as Fleming accidently discovered penicillin and I should share it on this blog.

My ‘accidents’ at CCCC

When I arrived at CCCC in September 2003 I decided to provide leadership that was open to opportunity.  In fact, we would search it out.  I decided that we would be willing to shift priorities based on comparing new opportunities with our current action plan.  To make this happen, the board and staff were told our action plans were simply default plans that we would follow as long as nothing better came up.  Then staff performance review criteria were set based on exhibiting our team values rather than reaching pre-set goals.  That way we could be flexible and nimble.  Here are a few examples of how that has worked out:

  • CCCC sponsors the Best Christian Workplace survey in Canada, and it all started because I just ‘happened’ to be attending a conference in the States where I saw the second annual presentation of awards for the American Best Christian Workplace survey.  But I was at the conference because I was looking to see what’s new and different, so I set the stage for opportunity.  I immediately connected what I saw with what I thought Canadian ministries needed because I was already on the alert for new ideas.  Within a couple of weeks, we had an agreement to do it in Canada.  I took action on an idea instead of just thinking, “That was interesting.”
  • CCCC was a partner in the 35<35 program that recognized young Christian leaders because I just ‘happened’ to run into Carson Pue in a hotel hallway.  But I attend EFC’s Presidents Day each year because it is the best place for me to meet a lot of ministry leaders at one time when I have no responsibilities of my own to attend to.  I was open to some collaborative work and wanted to talk with as many people as I could about what they are doing.  When Carson Pue and I crossed paths, we chatted a bit and when he asked what was on my mind, I told him we wanted to do something to recognize and encourage young Christian leaders at our next conference.  Wouldn’t you know it, he was wanting to find some partners to promote young leaders too and he had the idea for the 35<35 recognition program.  That turned out to be a great success for both of us (and Bridgeway Foundation and Christian Week – our other partners in that program).
  • Jerry White, International President Emeritus of The Navigators, is probably the highest ranked speaker we’ve ever had at our conference.  I ‘happened’ to have a personal connection with him that made me think of him as a speaker.  But I didn’t just ‘happen’ to meet him.  While writing my first doctoral research project in 2005 I was critiquing his 1983 book on my topic and I wondered if this author were still alive.  I tracked him down, and since then we’ve met a couple of times and had some phone calls, and he ended up not only speaking at the CCCC conference but also writing the foreword for my new book, The Church at Work.  Lots of good has come about because I didn’t just read and critique a book but took the additional step of contacting the author, not knowing at the time what was going to come of it.  I had no expectations at the beginning that anything would develop.  I just thought Jerry would be an interesting person to talk with and get to know.  But that created the environment in which other good things could develop.
  • Our horizons at CCCC are rapidly expanding as other staff members are also creating intential opportunities for ‘accidental’ discoveries!

Creating intentional accidents at your ministry

How much do you leave out to see “what might happen”?  In what ways do you create opportunity to be surprised?  To find opportunity you did not know existed?  Do you have an intentional plan to find the unexpected?  Some things you could do are:

  • Talk with people outside of your usual circles who might introduce a new idea from their sphere into your sphere.
  • Investigate what beneficiaries or clients do outside of your programs that might affect your program design.
  • Read a good book like The Eureka Effect, to stimulate your creativity.
  • Encourage your team to experiment on their own and be willing to try a small-scale pilot project.
  • Visit unrelated ministries doing a different part of the Christian mission and find out what’s new with them.
  • Just tell everyone you meet what your ministry does.  The more people who know about you, the more you are likely to hear about opportunities.
  • If you have field staff, pay close attention to what they are actually doing.  You’ll probably find a lot of good creativity on the front lines!
  • And of course, pray hard that God would bring unexpected opportunities your way!

You can never know in advance, of course, what will be discovered by following a strategy of intentional accidental discoveries, but that’s the fun and amazing part of it!  All you know at the start is that discoveries will be more likely because you have put yourself in a place that is conducive to discovery.  You won’t believe what good things will happen once you set the ball in motion by stepping outside of your zone.  Just watch what develops!

I’m sure I’ve only scratched the surface of how to make accidental discoveries more likely.  Please add your own ideas by making a comment.

Bookmark and Share

“You know, I may be wrong”

As a blogger, I spout my opinions frequently and strongly.  I think I’m right, but you know, at times I may be wrong or at least my suggestions may not be universally applicable.  In Sunday School today we watched a sermon by Andy Stanley on “Big Faith“, the first in a series on five ways that God works to build our faith.  In the introduction he says something like, ”I may be wrong of course.  Maybe there are six ways, or a completely different list of five, or perhaps God works another way entirely, so this is just an observation of five ways it appears to me that God works.”

As you and I both know, there is no one single way to lead.  There is no single formulaic five step process to successful leadership.  There are all sorts of ways one can lead, and which way you choose may be highly dependent on circumstances, your nature, and a whole bunch of other factors.

My caveat

This blog is a series of reflections on how to provide Christian leadership.  I’ve made some strong statements and I know I say ‘should’ quite a lot.  That sounds pretty directive!  Since I write based on my experience, my circumstances, my personality and so on, I think my suggestions will work well for me, but will they for you?  You must decide that for yourself.  My goal is to get you thinking and provide you with some help along the way by sharing the very best information I can find or think of about Christian leadership.  I am sure my thoughts have widespread application (or I wouldn’t bother writing), but in the end they are just observations.  They are my contribution to Christian ministry written in response to the question I assume readers are asking: ”I wonder what a Christian leader who has time to think and write about leadership issues thinks about leading in a Christian way?”

Your caveat

You as a leader are in a similar position.  You have your opinions about issues facing your ministry’s leadership and as a leader you should be sharing them.  We lead from our convictions knowing that people want to have confidence in their leaders, but that means that we tend to lead with a high degree of certainty.  Let’s not be more certain about things than we have a right to be.  In many cases, there is more than one way to “skin the cat.”  (What an awful expression, but it expresses the thought perfectly.)  This is why it is best to lead with consensus decisions, or decisions that you make but which have been shaped by group discussion.

In closing

  • Knowing that I could be wrong, or that there is more than one way to lead, is why I allow comments on this blog.  I want you to participate with me and my readers in a group discussion so we can all learn together.
  • Knowing that you could be wrong or that there is more than one way to address whatever issues face you, is why you have a team to consult with and to help you lead.  Two heads are better than one and all that!

Leaders are not required to have all the answers, but to know how to develop a good answer and to make a decision.

Plans fail for lack of counsel,
   but with many advisers they succeed.
Proverbs 15:22

Bookmark and Share
Rev. John Pellowe
   Rev. John Pellowe, MBA, DMin