{"id":5093,"date":"2011-12-03T13:47:34","date_gmt":"2011-12-03T18:47:34","guid":{"rendered":"\/news_blogs\/john\/?p=5093"},"modified":"2024-12-04T17:11:41","modified_gmt":"2024-12-04T22:11:41","slug":"program-evaluation-4-research-design","status":"publish","type":"post","link":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/","title":{"rendered":"Program Evaluation 4 &#8211; Research Design"},"content":{"rendered":"<div class=\"pps-series-post-details pps-series-post-details-variant-classic pps-series-post-details-38706\" data-series-id=\"124\"><div class=\"pps-series-meta-content\"><div class=\"pps-series-meta-text\">This entry is part 4 of 7 in the series <a href=\"https:\/\/www.cccc.org\/news_blogs\/series\/program-evaluation\/\">Program Evaluation<\/a><\/div><\/div><\/div>\n<p>A year ago I wrote about <a title=\"Mission first, then program\" href=\"\/news_blogs\/john\/2010\/12\/14\/mission-first-then-program\/\" target=\"_blank\" rel=\"noopener\">the need to check your assumptions<\/a>. That is what we are now ready to do in the <strong>program evaluation<\/strong> of our annual conference. By the end of this post, you will see how we at CCCC designed the <strong>research<\/strong> content and process of our program review.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Deciding What Research Needs to Be Done<\/h2>\n\n\n\n<p>Having:<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li><a title=\"Program Evaluation 1: Selecting the program\" href=\"\/news_blogs\/john\/2011\/10\/14\/program-evaluation-1-selecting-the-program\/\" target=\"_blank\" rel=\"noopener\">selected the program&nbsp;we want to evaluate<\/a>,<\/li><li><a title=\"Program Evaluation 2 \u2013 Program rationale\" href=\"\/news_blogs\/john\/2011\/10\/19\/program-evaluation-2-the-logic-model\/\" target=\"_blank\" rel=\"noopener\">developed the program rationale <\/a>(theory of change and logic model), and<\/li><li><a title=\"Program Evaluation 3 \u2013 Literature review\" href=\"\/news_blogs\/john\/2011\/10\/24\/program-evaluation-3-literature-review\/\" target=\"_blank\" rel=\"noopener\">completed the literature review<\/a>,<\/li><\/ol>\n\n\n\n<p>we next reviewed every part of the <strong>program rationale<\/strong> (<strong>theory of change<\/strong> and <strong>logic model<\/strong>) looking for assumptions that we already know about and <strong>assumptions<\/strong> that come to light only as we think more deeply about the program. The goal is to develop a list of questions we want to answer. If we can answer those questions, we will know how well the program is working and how to improve it. We also looked at the <a title=\"Program Evaluation 1: Selecting the program\" href=\"\/news_blogs\/john\/2011\/10\/14\/program-evaluation-1-selecting-the-program\/\" target=\"_blank\" rel=\"noopener\">research questions that were developed at the start<\/a> and thought about what questions we need to ask in order to answer the research questions. Other ways we used to develop our questions included:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Reading the literature review to find decision points where&nbsp;we&#8217;d like to know what our members think about the options. In our case, the literature review did not spark any questions that had not already arisen from our analysis of the program rationale, but it did provide ideas for improvement. Decision points include such things as length, plenary\/workshop mix, use of technology, etc.<\/li><li>We&nbsp;asked the senior team for their questions.<\/li><li>We keep a statistical analysis of each conference and have extensive feedback forms for both the conference as a whole and all of its components. We ran reports that showed that 25-30% of the participants each year are first-timers, but conference attendance is not growing at that rate, so we&#8217;d like to understand better how the decision is made whether or not to come again. The feedback showed that our conference is considered by most to be the best run conference they attend. There is only one&nbsp;question of logistics or conference administration to ask.<\/li><li>Finally, overall responsibility for the conference rests with me, so I sat back, closed my eyes, and just thought about what I&#8217;d like to know more&nbsp;about.<\/li><\/ul>\n\n\n\n<p>Keep in mind that in any program evaluation you want to essentially answer two questions:<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>Were we effective? You need to compare the actual output to the expected output to discover if you did what you wanted to do. That&#8217;s effectiveness. The theory of change helps you answer the effectiveness question: Did we do the right things?<\/li><li>Were we efficient? You need to compare the ratio of actual output to actual inputs and ask if you were good stewards of your resources. That&#8217;s efficiency. The logic model helps you answer the efficiency&nbsp;question: Did we do things right?<\/li><\/ol>\n\n\n\n<h2 class=\"wp-block-heading\">Specific Questions<\/h2>\n\n\n\n<p>We developed specific questions to guide the development of surveys and further research, sorted by category.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Assumed Problem and Assumed Causes Questions<\/h3>\n\n\n\n<p>We wanted to know how people feel about conferences and the other means they use to stay current in their fields. We wondered how people challenge and stimulate their own thinking.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Assumed Assets and Other Attendee Needs<\/h3>\n\n\n\n<p>These questions related to their habits, preferences and decisions about conference attendance.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Interventions<\/h3>\n\n\n\n<p>We wanted to probe what happens pre-conference, during the conference, and post-conference from the attendee&#8217;s perspective. &nbsp;How does an event fit within a person&#8217;s larger context?<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Short-Term Outcomes<\/h3>\n\n\n\n<p>Moving away from the conference itself, we wanted to know what people do with the material afterwards.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Long-Term Outcomes<\/h3>\n\n\n\n<p>Stepping back from any particular conference, respondents to our surveys were asked about their employer&#8217;s overall assessment of the usefulness of attending conferences.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Inputs<\/h3>\n\n\n\n<p>There were some questions, not for surveys but for further research, that we asked to improve the use of our time, the locations we rent, and the handout materials.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Success Criteria<\/h3>\n\n\n\n<p>If there are benchmarks or standards available for the program you are reviewing, you would determine what threshold must be crossed in order to consider your program a success. It is very difficult to set a threshold once you know what the results are because biases will enter the equation!<\/p>\n\n\n\n<p>Ministries might ask questions such as:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>What recidivism rate is acceptable?<\/li><li>What is an acceptable percentage of new believers who are still attending church and being discipled a year later?<\/li><li>What percentage of clients should have a job a year from now with at least three months of steady employment?<\/li><li>What percentage of our congregation is involved in some form of Christian ministry, whether with our church or with another ministry?<\/li><\/ul>\n\n\n\n<p>In our case, I cannot find any benchmarks for success, so as part of the research we will attempt to create some benchmarks by asking other associations with voluntary attendance at their conferences about their results. We will also ask ourselves: In light of the number of ministries that benefit from the conference compared to our resources that it consumes, is it good stewardship to continue running a conference? This criteria is a bit fuzzy for evaluation purists, but it is good enough for me given that we are primarily looking for ways to improve the conference rather than make a continue\/discontinue decision.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Methodology<\/h2>\n\n\n\n<p>We then reviewed each question and decided two things:<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>Who should we ask this question of?<\/li><li>How should we ask it?<\/li><\/ol>\n\n\n\n<p>We decided we would find the answers in the following ways:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Ask people who have attended the conference at least once since 2006 (includes members and non-members)<\/li><li>Ask members&nbsp;who have not attended the conference since 2005 (the conference is designed for our members so we expect them to come; non-members are not our target for the conference)<\/li><li>Ask speakers who have presented in three of the past six years<\/li><li>Review possible venues at certain locations<\/li><li>Analyze our database for attendance patterns<\/li><li>Talk with other organizations that run conferences<\/li><li>Based on the results of all the above inquiries, select a small group of people for one-on-one interviews to delve deeper to get more insight on any remaining questions.<\/li><\/ul>\n\n\n\n<p>While we chose to base the review on surveys for the most part, there are many research methodologies you could choose from:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Verbal data: Conduct interviews either one-on-one or in groups (focus groups)<\/li><li>Client surveys: expectations of&nbsp;services, use of services, satisfaction, rating of quality<\/li><li>Outcomes surveys: behaviours, beliefs and conditions that have changed as a result of your service<\/li><li>Observational data: Watch and see what happens<\/li><li>Archival data: Check data collected from running the program, your own records, plans etc.<\/li><\/ul>\n\n\n\n<p>While benchmarks may be helpful, there are always so many differences between organizations or even divisions within the same organization &#8211; circumstances, conditions, history etc. &#8211; that the best comparison is really between your current results and your past results. Is your performance improving?<\/p>\n\n\n\n<p>You are now ready to go ahead&nbsp;and do&nbsp;the evaluation.<\/p>\n\n\n\n<figure class=\"wp-block-audio\"><audio controls src=\"https:\/\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2014\/08\/Program-Evaluation-4.mp3\"><\/audio><\/figure>\n","protected":false},"excerpt":{"rendered":"<div class=\"pps-series-post-details pps-series-post-details-variant-classic pps-series-post-details-38706 pps-series-meta-excerpt\" data-series-id=\"124\"><div class=\"pps-series-meta-content\"><div class=\"pps-series-meta-text\">This entry is part 4 of 7 in the series <a href=\"https:\/\/www.cccc.org\/news_blogs\/series\/program-evaluation\/\">Program Evaluation<\/a><\/div><\/div><\/div><p>How to determine what a program&#8217;s assumptions are and why you think the program works, and then how to decide what the evaluation should research and test. <a href=\"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/\" class=\"linkbutton\">More<\/a><\/p>\n","protected":false},"author":2,"featured_media":17923,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ts_fic_featured_image_caption":"A black microscope. Used with permission.","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[553,221,554],"tags":[53,589],"series":[124],"class_list":["post-5093","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-effective","category-leadership-organizational","category-thoughtful","tag-evaluation","tag-organizational-self-awareness","series-program-evaluation"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.3 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Program Evaluation 4 - Research Design - CCCC Blogs<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Program Evaluation 4 - Research Design - CCCC Blogs\" \/>\n<meta property=\"og:description\" content=\"How to determine what a program&#039;s assumptions are and why you think the program works, and then how to decide what the evaluation should research and test. More\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/\" \/>\n<meta property=\"og:site_name\" content=\"CCCC Blogs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/CCCCCharities\" \/>\n<meta property=\"article:published_time\" content=\"2011-12-03T18:47:34+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-12-04T22:11:41+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2011\/12\/Microscope-Freeimages.jpg?fit=1488%2C2125&ssl=1\" \/>\n\t<meta property=\"og:image:width\" content=\"1488\" \/>\n\t<meta property=\"og:image:height\" content=\"2125\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"John Pellowe\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@cccccharities\" \/>\n<meta name=\"twitter:site\" content=\"@cccccharities\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"John Pellowe\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/john\\\/2011\\\/12\\\/03\\\/program-evaluation-4-research-design\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/john\\\/2011\\\/12\\\/03\\\/program-evaluation-4-research-design\\\/\"},\"author\":{\"name\":\"John Pellowe\",\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/#\\\/schema\\\/person\\\/78129b8fafee8dfe3bb7f3d7c8e33569\"},\"headline\":\"Program Evaluation 4 &#8211; Research Design\",\"datePublished\":\"2011-12-03T18:47:34+00:00\",\"dateModified\":\"2024-12-04T22:11:41+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/john\\\/2011\\\/12\\\/03\\\/program-evaluation-4-research-design\\\/\"},\"wordCount\":1182,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/john\\\/2011\\\/12\\\/03\\\/program-evaluation-4-research-design\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i0.wp.com\\\/www.cccc.org\\\/news_blogs\\\/wp-content\\\/uploads\\\/2011\\\/12\\\/Microscope-Freeimages.jpg?fit=1488%2C2125&ssl=1\",\"keywords\":[\"Evaluation\",\"Organizational Self-Awareness\"],\"articleSection\":[\"Effective\",\"Organizational Leadership\",\"Thoughtfulness\"],\"inLanguage\":\"en-CA\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/john\\\/2011\\\/12\\\/03\\\/program-evaluation-4-research-design\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/john\\\/2011\\\/12\\\/03\\\/program-evaluation-4-research-design\\\/\",\"url\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/john\\\/2011\\\/12\\\/03\\\/program-evaluation-4-research-design\\\/\",\"name\":\"Program Evaluation 4 - Research Design - CCCC Blogs\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/john\\\/2011\\\/12\\\/03\\\/program-evaluation-4-research-design\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/john\\\/2011\\\/12\\\/03\\\/program-evaluation-4-research-design\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i0.wp.com\\\/www.cccc.org\\\/news_blogs\\\/wp-content\\\/uploads\\\/2011\\\/12\\\/Microscope-Freeimages.jpg?fit=1488%2C2125&ssl=1\",\"datePublished\":\"2011-12-03T18:47:34+00:00\",\"dateModified\":\"2024-12-04T22:11:41+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/john\\\/2011\\\/12\\\/03\\\/program-evaluation-4-research-design\\\/#breadcrumb\"},\"inLanguage\":\"en-CA\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/john\\\/2011\\\/12\\\/03\\\/program-evaluation-4-research-design\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-CA\",\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/john\\\/2011\\\/12\\\/03\\\/program-evaluation-4-research-design\\\/#primaryimage\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/www.cccc.org\\\/news_blogs\\\/wp-content\\\/uploads\\\/2011\\\/12\\\/Microscope-Freeimages.jpg?fit=1488%2C2125&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/www.cccc.org\\\/news_blogs\\\/wp-content\\\/uploads\\\/2011\\\/12\\\/Microscope-Freeimages.jpg?fit=1488%2C2125&ssl=1\",\"width\":1488,\"height\":2125,\"caption\":\"Used with permission\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/john\\\/2011\\\/12\\\/03\\\/program-evaluation-4-research-design\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Program Evaluation 4 &#8211; Research Design\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/#website\",\"url\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/\",\"name\":\"CCCC News &amp; Blogs\",\"description\":\"CCCC Blogs\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-CA\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/#organization\",\"name\":\"Canadian Centre for Christian Charities\",\"url\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-CA\",\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/www.cccc.org\\\/news_blogs\\\/wp-content\\\/uploads\\\/2020\\\/10\\\/260wbRGB.png?fit=260%2C260&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/www.cccc.org\\\/news_blogs\\\/wp-content\\\/uploads\\\/2020\\\/10\\\/260wbRGB.png?fit=260%2C260&ssl=1\",\"width\":260,\"height\":260,\"caption\":\"Canadian Centre for Christian Charities\"},\"image\":{\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/CCCCCharities\",\"https:\\\/\\\/x.com\\\/cccccharities\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/canadian-council-of-christian-charities\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/#\\\/schema\\\/person\\\/78129b8fafee8dfe3bb7f3d7c8e33569\",\"name\":\"John Pellowe\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-CA\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e5439962feafa2554de0378f025969e5215f2a92edf5742f1bf3bed1765d6304?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e5439962feafa2554de0378f025969e5215f2a92edf5742f1bf3bed1765d6304?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e5439962feafa2554de0378f025969e5215f2a92edf5742f1bf3bed1765d6304?s=96&d=mm&r=g\",\"caption\":\"John Pellowe\"},\"sameAs\":[\"https:\\\/\\\/www.cccc.org\",\"bonnie.pillsworth@cccc.org\"],\"url\":\"https:\\\/\\\/www.cccc.org\\\/news_blogs\\\/author\\\/john\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Program Evaluation 4 - Research Design - CCCC Blogs","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/","og_locale":"en_US","og_type":"article","og_title":"Program Evaluation 4 - Research Design - CCCC Blogs","og_description":"How to determine what a program's assumptions are and why you think the program works, and then how to decide what the evaluation should research and test. More","og_url":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/","og_site_name":"CCCC Blogs","article_publisher":"https:\/\/www.facebook.com\/CCCCCharities","article_published_time":"2011-12-03T18:47:34+00:00","article_modified_time":"2024-12-04T22:11:41+00:00","og_image":[{"width":1488,"height":2125,"url":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2011\/12\/Microscope-Freeimages.jpg?fit=1488%2C2125&ssl=1","type":"image\/jpeg"}],"author":"John Pellowe","twitter_card":"summary_large_image","twitter_creator":"@cccccharities","twitter_site":"@cccccharities","twitter_misc":{"Written by":"John Pellowe","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/#article","isPartOf":{"@id":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/"},"author":{"name":"John Pellowe","@id":"https:\/\/www.cccc.org\/news_blogs\/#\/schema\/person\/78129b8fafee8dfe3bb7f3d7c8e33569"},"headline":"Program Evaluation 4 &#8211; Research Design","datePublished":"2011-12-03T18:47:34+00:00","dateModified":"2024-12-04T22:11:41+00:00","mainEntityOfPage":{"@id":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/"},"wordCount":1182,"commentCount":0,"publisher":{"@id":"https:\/\/www.cccc.org\/news_blogs\/#organization"},"image":{"@id":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2011\/12\/Microscope-Freeimages.jpg?fit=1488%2C2125&ssl=1","keywords":["Evaluation","Organizational Self-Awareness"],"articleSection":["Effective","Organizational Leadership","Thoughtfulness"],"inLanguage":"en-CA","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/","url":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/","name":"Program Evaluation 4 - Research Design - CCCC Blogs","isPartOf":{"@id":"https:\/\/www.cccc.org\/news_blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/#primaryimage"},"image":{"@id":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2011\/12\/Microscope-Freeimages.jpg?fit=1488%2C2125&ssl=1","datePublished":"2011-12-03T18:47:34+00:00","dateModified":"2024-12-04T22:11:41+00:00","breadcrumb":{"@id":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/#breadcrumb"},"inLanguage":"en-CA","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/"]}]},{"@type":"ImageObject","inLanguage":"en-CA","@id":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/#primaryimage","url":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2011\/12\/Microscope-Freeimages.jpg?fit=1488%2C2125&ssl=1","contentUrl":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2011\/12\/Microscope-Freeimages.jpg?fit=1488%2C2125&ssl=1","width":1488,"height":2125,"caption":"Used with permission"},{"@type":"BreadcrumbList","@id":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/12\/03\/program-evaluation-4-research-design\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.cccc.org\/news_blogs\/"},{"@type":"ListItem","position":2,"name":"Program Evaluation 4 &#8211; Research Design"}]},{"@type":"WebSite","@id":"https:\/\/www.cccc.org\/news_blogs\/#website","url":"https:\/\/www.cccc.org\/news_blogs\/","name":"CCCC News &amp; Blogs","description":"CCCC Blogs","publisher":{"@id":"https:\/\/www.cccc.org\/news_blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.cccc.org\/news_blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-CA"},{"@type":"Organization","@id":"https:\/\/www.cccc.org\/news_blogs\/#organization","name":"Canadian Centre for Christian Charities","url":"https:\/\/www.cccc.org\/news_blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-CA","@id":"https:\/\/www.cccc.org\/news_blogs\/#\/schema\/logo\/image\/","url":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2020\/10\/260wbRGB.png?fit=260%2C260&ssl=1","contentUrl":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2020\/10\/260wbRGB.png?fit=260%2C260&ssl=1","width":260,"height":260,"caption":"Canadian Centre for Christian Charities"},"image":{"@id":"https:\/\/www.cccc.org\/news_blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/CCCCCharities","https:\/\/x.com\/cccccharities","https:\/\/www.linkedin.com\/company\/canadian-council-of-christian-charities"]},{"@type":"Person","@id":"https:\/\/www.cccc.org\/news_blogs\/#\/schema\/person\/78129b8fafee8dfe3bb7f3d7c8e33569","name":"John Pellowe","image":{"@type":"ImageObject","inLanguage":"en-CA","@id":"https:\/\/secure.gravatar.com\/avatar\/e5439962feafa2554de0378f025969e5215f2a92edf5742f1bf3bed1765d6304?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/e5439962feafa2554de0378f025969e5215f2a92edf5742f1bf3bed1765d6304?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e5439962feafa2554de0378f025969e5215f2a92edf5742f1bf3bed1765d6304?s=96&d=mm&r=g","caption":"John Pellowe"},"sameAs":["https:\/\/www.cccc.org","bonnie.pillsworth@cccc.org"],"url":"https:\/\/www.cccc.org\/news_blogs\/author\/john\/"}]}},"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2011\/12\/Microscope-Freeimages.jpg?fit=1488%2C2125&ssl=1","jetpack-related-posts":[{"id":5049,"url":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/10\/14\/program-evaluation-1-selecting-the-program\/","url_meta":{"origin":5093,"position":0},"title":"Program Evaluation 1 &#8211; Selecting the Program","author":"John Pellowe","date":"October 14, 2011","format":false,"excerpt":"Here are some factors to help you decide which program you should review next.","rel":"","context":"In &quot;Effective&quot;","block_context":{"text":"Effective","link":"https:\/\/www.cccc.org\/news_blogs\/category\/effective\/"},"img":{"alt_text":"Multiple Choice test","src":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2011\/10\/Multiple-Choice-Freeimages.jpg?fit=767%2C1200&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2011\/10\/Multiple-Choice-Freeimages.jpg?fit=767%2C1200&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2011\/10\/Multiple-Choice-Freeimages.jpg?fit=767%2C1200&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2011\/10\/Multiple-Choice-Freeimages.jpg?fit=767%2C1200&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":5091,"url":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/10\/19\/program-evaluation-2-program-rationale\/","url_meta":{"origin":5093,"position":1},"title":"Program Evaluation 2 &#8211; Program Rationale","author":"John Pellowe","date":"October 19, 2011","format":false,"excerpt":"Every program evaluation needs to be based on the rationale for why the program exists in the first place. Individual programs only exist because they support the ministry's theory of change that describes how its vision will be accomplished through its programs. Only then can you develop a rationale for\u2026","rel":"","context":"In &quot;Exemplary&quot;","block_context":{"text":"Exemplary","link":"https:\/\/www.cccc.org\/news_blogs\/category\/exemplary\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2021\/11\/shutterstock_154767518-scaled.jpg?fit=1200%2C847&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2021\/11\/shutterstock_154767518-scaled.jpg?fit=1200%2C847&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2021\/11\/shutterstock_154767518-scaled.jpg?fit=1200%2C847&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2021\/11\/shutterstock_154767518-scaled.jpg?fit=1200%2C847&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2021\/11\/shutterstock_154767518-scaled.jpg?fit=1200%2C847&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":5101,"url":"https:\/\/www.cccc.org\/news_blogs\/john\/2012\/02\/02\/program-evaluation-5-wrapping-it-up\/","url_meta":{"origin":5093,"position":2},"title":"Program Evaluation 5 &#8211; Wrapping It Up","author":"John Pellowe","date":"February 2, 2012","format":false,"excerpt":"Well, the program review is now complete and it is time to figure out what to do with the results.\u00a0 The process started with selecting which program to evaluate and went on from there with development of the theory of change and logic model, the literature review, and then the\u2026","rel":"","context":"In &quot;Effective&quot;","block_context":{"text":"Effective","link":"https:\/\/www.cccc.org\/news_blogs\/category\/effective\/"},"img":{"alt_text":"Wrapped gift with a bow","src":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2012\/02\/Wrapped-gift.jpg?fit=639%2C770&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2012\/02\/Wrapped-gift.jpg?fit=639%2C770&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2012\/02\/Wrapped-gift.jpg?fit=639%2C770&ssl=1&resize=525%2C300 1.5x"},"classes":[]},{"id":5095,"url":"https:\/\/www.cccc.org\/news_blogs\/john\/2011\/10\/24\/program-evaluation-3-literature-review\/","url_meta":{"origin":5093,"position":3},"title":"Program Evaluation 3 &#8211; Literature Review","author":"John Pellowe","date":"October 24, 2011","format":false,"excerpt":"A well done literature review will identify issues to research and generate new knowledge and insights that should lead to a cutting-edge program. Here's how to do one well.","rel":"","context":"In &quot;Creativity&quot;","block_context":{"text":"Creativity","link":"https:\/\/www.cccc.org\/news_blogs\/category\/exemplary\/intellectual-creativity\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2021\/01\/freestocks-RgKmrxpIraY-unsplash-scaled.jpg?fit=1200%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2021\/01\/freestocks-RgKmrxpIraY-unsplash-scaled.jpg?fit=1200%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2021\/01\/freestocks-RgKmrxpIraY-unsplash-scaled.jpg?fit=1200%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2021\/01\/freestocks-RgKmrxpIraY-unsplash-scaled.jpg?fit=1200%2C800&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2021\/01\/freestocks-RgKmrxpIraY-unsplash-scaled.jpg?fit=1200%2C800&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":4849,"url":"https:\/\/www.cccc.org\/news_blogs\/john\/2010\/12\/14\/mission-first-then-program\/","url_meta":{"origin":5093,"position":4},"title":"Mission First, Then Program","author":"John Pellowe","date":"December 14, 2010","format":false,"excerpt":"Be committed to your mission, not your programs. Suggestions for meaningful program evaluations","rel":"","context":"In &quot;Creativity&quot;","block_context":{"text":"Creativity","link":"https:\/\/www.cccc.org\/news_blogs\/category\/exemplary\/intellectual-creativity\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2021\/01\/glen-carrie-YtLAiN5YJ-8-unsplash-scaled.jpg?fit=1200%2C900&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2021\/01\/glen-carrie-YtLAiN5YJ-8-unsplash-scaled.jpg?fit=1200%2C900&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2021\/01\/glen-carrie-YtLAiN5YJ-8-unsplash-scaled.jpg?fit=1200%2C900&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2021\/01\/glen-carrie-YtLAiN5YJ-8-unsplash-scaled.jpg?fit=1200%2C900&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2021\/01\/glen-carrie-YtLAiN5YJ-8-unsplash-scaled.jpg?fit=1200%2C900&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":27064,"url":"https:\/\/www.cccc.org\/news_blogs\/john\/2018\/03\/19\/evaluation-avoiding-the-blame-game\/","url_meta":{"origin":5093,"position":5},"title":"Evaluation: Avoiding the Blame Game","author":"John Pellowe","date":"March 19, 2018","format":false,"excerpt":"For your ministry to \"step up its game,\" it must have an objective understanding how it is performing now. But is your organization\u00a0ready to evaluate itself?","rel":"","context":"In &quot;Effective&quot;","block_context":{"text":"Effective","link":"https:\/\/www.cccc.org\/news_blogs\/category\/effective\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2018\/02\/blame-iStock-608538642.jpg?fit=1200%2C1199&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2018\/02\/blame-iStock-608538642.jpg?fit=1200%2C1199&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2018\/02\/blame-iStock-608538642.jpg?fit=1200%2C1199&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2018\/02\/blame-iStock-608538642.jpg?fit=1200%2C1199&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.cccc.org\/news_blogs\/wp-content\/uploads\/2018\/02\/blame-iStock-608538642.jpg?fit=1200%2C1199&ssl=1&resize=1050%2C600 3x"},"classes":[]}],"jetpack_shortlink":"https:\/\/wp.me\/p30X8p-1k9","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.cccc.org\/news_blogs\/wp-json\/wp\/v2\/posts\/5093","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.cccc.org\/news_blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cccc.org\/news_blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cccc.org\/news_blogs\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cccc.org\/news_blogs\/wp-json\/wp\/v2\/comments?post=5093"}],"version-history":[{"count":0,"href":"https:\/\/www.cccc.org\/news_blogs\/wp-json\/wp\/v2\/posts\/5093\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.cccc.org\/news_blogs\/wp-json\/wp\/v2\/media\/17923"}],"wp:attachment":[{"href":"https:\/\/www.cccc.org\/news_blogs\/wp-json\/wp\/v2\/media?parent=5093"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cccc.org\/news_blogs\/wp-json\/wp\/v2\/categories?post=5093"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cccc.org\/news_blogs\/wp-json\/wp\/v2\/tags?post=5093"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/www.cccc.org\/news_blogs\/wp-json\/wp\/v2\/series?post=5093"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}