Discussion: Program Evaluation: Benefits and Concerns of Stakeholders
Dudley (2020) points out that social work practice is [frequently] embedded in programs. In Week 3, you developed a program for a single client in single system research design. This week, you will shift your focus to the methods used in program evaluation.
Program evaluators use a variety of models to answer different questions about the operation and outcomes of social work programs, including product (also known as outcome) evaluations, process evaluations, \\\”needs assessments, program monitoring, client satisfaction studies, …[and] cost benefit [analyses. Each type of evaluation involves collecting and analyzing] many different types of data to answer different questions.
This week (Week 6), our focus will be on program evaluation. Program evaluations serve many purposes. They provide information for determining the need for a program, the quality of the program, its outcomes, and effectiveness. Funding sources such as government agencies and private foundations require periodic program evaluations to comply with the law and ensure accountability to funding organizations and the public. Information from program evaluations contributes to improvement of services. Adapted from SOCW 6311, Week 6 Discussion, Walden University
Our task as program evaluators involves selecting the program evaluation model that is best suited for addressing different questions about the program, collecting and analyzing data to answer those questions, and reporting results to stakeholders.
The goal of this week’s information is to demonstrate methods for evaluating social service programs. If you work in an agency, you may be involved in evaluating programs at some point in your career. The format we will use in this week’s work is widely used in social service agencies, hospitals, schools, and other human service organizations.
This week, we will explore two types of program evaluations: process evaluations and product (also called outcome) evaluations. Understanding the difference between process and product evaluations is essential for success with the discussion and assignment this week.
We will also explore the important roles played by stakeholders in developing, implementing, and evaluating programs.
WHAT TO STUDY THIS WEEK
- Read: Pages 221-227, Program Evaluation Studies (Logan and Royse, 2010) from Chapter 13 of B. Thyer (Ed.), The Handbook of Social Work Research Methods (2nd ed.) for an overview of the methods used in program evaluation. In this chapter, Logan and Royse (2010) describe process and outcome or impact evaluations. Note that outcome evaluations are also called product evaluations. We will use those terms interchangeably. Impact evaluations are more technical and involve comparing a program to a control situation through experimental or quasi-experimental methods. We will not discuss the details of impact evaluations in this course.
- Read: Pages 26-27 of A Step-by-Step Guide to Program Evaluation, from TSNE, an organization that assists non-profits evaluate its programs for definitions of process and product evaluations. (https://www.tsne.org/blog/process-evaluation-vs-outcome-evaluation).
- Read: Glesmann (2016) (https://www.nccdglobal.org/newsroom/nccd-blog/comparing-process-and-outcome-evaluations) for a detailed description of process and product evaluations and the questions that might be answered by them.
- Read: Chapter 3 of the W.K. Kellogg Foundation Evaluation Step-by-Step Guide to Evaluation (pp. 25-34). The link to the W. K. Kellogg Foundation Evaluation Handbook (https://www.wkkf.org/resource-directory/resources/2017/11/the-step-by-step-guide-to-evaluation–how-to-become-savvy-evaluation-consumers) is on the Dashboard for this week. When the page opens, there are several booklets shown; select the Evaluation Handbook (the 3rd entry on the main page near the right margin) and download it. When it opens, it will be entitled Step-by-step process of evaluating programs. Download and keep this booklet because we will use it in future weeks.
- Skim the webpage from the University of Kansas Center for Community Health and Development on program evaluation and notice the questions that are asked in process and product evaluations. You’ll find the information here: Center for Community Health and Development. (2017). Chapter 36, Section 1: A Framework for Program Evaluation: A Gateway to Tools. https://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/framework-for-evaluation/main
- Read about stakeholders, their importance, role, and concerns regarding evaluations here:
The Center for Community Health and Development. (2017). Chapter 36, Section 1: A Framework for Program Evaluation: A Gateway to Tools. Look on the left margin and open Section 3: Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests? (https://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/interests-of-leaders-evaluators-funders/main)
- Skim this handout, entitled Common Questions For Program Evaluations adapted from Krupar (2019), the CDC (2012), and Community Toolbox (n.d.) for a list of questions often included in evaluations to address stakeholders\\\’ concerns. Many more questions for an evaluation are possible, based on the phase of the program (part-way through or at the end), local community issues, and the features of the target population.
WHAT TO LEARN THIS WEEK
- The two major types of evaluations (that is, product (also called outcome) and process evaluations) and the essential differences between them;
- The purposes for conducting evaluations;
- The role and importance of including stakeholders in the evaluation process.
- The need to communicate clearly to all who will be involved in the evaluation and apply methods to ensure that evaluations are ethical, fair, and accurate.
WHAT TO DO THIS WEEK
By Day 3
By Day 3, submit your original post, answering the questions listed below. Please include \\\”Original Post\\\” in your header and use the format listed below these instructions. If you choose to revise information in your post, you must resubmit the entire corrected post with references, and title it “Original post – revised”.
For this discussion, you will describe aspects of evaluating a program with which you have had some experience as a planner, provider, or recipient, if possible. The program should address a need or solve a problem in your community, but the program does not have to be a social work program. School, crime reduction, nutrition, and recreation programs are acceptable. Your program must be currently in operation or one that has been completed at least once. Do not include programs that are in the planning stages.
If you do not know of any programs, you may have to read about some programs on the Internet and use your imagination to answer some of these questions. The Brookings Institute has a web page, Social programs that work, (https://www.brookings.edu/blog/up-front/2014/12/01/social-programs-that-work/) with programs from which you may choose if you wish.
Include the following elements in your post:
1. Name a program with which you may be familiar, either through work in an agency, as an interested community member, or as a planner, provider, or participant. Make up a name for the program and organization that delivers it (if any) to keep information about the program private. If you do not have experience with a program, choose one from the Internet. (Use a few words only to name the program). (1 sentence)
2. In one sentence, categorize the purpose of the program. A list of purposes can be found in the Optional Information and Ideas section below, entitled Examples of Social Service Programs: https://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/framework-for-evaluation/main
3. In 1-3 sentences identify the intended recipients of the program (children, adolescents, persons with XYZ illness or disability, persons struggling with…, and so on).
4. Identify the general goals of the program. Look for inspiring words (empower, enable, heal, build, improve health, and so on). These are often stated in the mission statement for the program or somewhere on the program website
5. Name the specific task of the program or intended outcomes of the program for the target recipients. These might include helping parents navigate online learning at Walden Middle School, finding shelter space for families to stay together, offering a support group for families who have lost a member due to COVID, or a summer camp for children who have had cancer, or a class to teach children of wounded veterans how to handle their parent\\\’s service dog appropriately, for example.
6. Select and name TWO different types of program stakeholders by their roles from the table below or others that were involved in an actual evaluation. Do not include target recipients for this step, although they are a type of stakeholder, too. Each stakeholder must have a different role or relationship with the program. Since they have different relationships with the program, they may have different perspectives, which will be provided in Question 7 and Question 8.
If you don’t know the stakeholders of your program or your program is fictional, use your judgment to identify two types of people who may be affected by the success or failure of the program. Do not use actual names of individuals. You may use made-up names or identify the stakeholders by their role in the agency or the community (administrator, concerned parent, parole officer, etc.). A maximum of two students may choose the same example of a stakeholder role.
7. List two relevant questions about the program that each type of stakeholder would want answered by your evaluation (you may need to use your imagination for this, and you may copy and paste from the list in the Optional Information and Ideas section). Typical concerns of stakeholders are located in the Optional Information and Ideas section as well as in the learning resources. (2 sentences, with two relevant concerns for each of the different stakeholders)
8. Describe the type of evaluation (process evaluation or product evaluation) that would answer your stakeholders\\\’ questions and provide your rationale (3-4 sentences, 1-2 for each stakeholder).
9. Identify two concerns about the methods used for the evaluation that each of the stakeholders might have (2 sentences, with two relevant concerns about the evaluation from each of the two different stakeholders).
10. Explain how you would address one of one stakeholder\\\’s program evaluation concerns and the one of the other stakeholder\\\’s concerns about the methods for evaluating the program. (2-4 sentences)