15.1 Evaluation research

As you may recall from the definition provided in Chapter 1, evaluation research is research conducted to assess the effects of specific programs or policies. Evaluation research is often used when some form of policy intervention is planned, such as welfare reform or school curriculum change. The focus on interventions and social problems makes it natural fit for social work researchers. It might be used to assess the extent to which intervention is necessary by attempting to define and diagnose social problems in a social worker’s service area, and it might also be used to understand whether their agency’s interventions have had their intended consequences.

I often remind my students they will eventually have bright ideas about what programs or interventions their agency should try. Moreover, they will eventually be so good at their job they will take on additional administrative and supervisory responsibilities. As a result, you will need to prove to your agency, and in particular, the people who fund your agency that your interventions are successful. Government and private grants almost universally come with a requirement that outcomes be measured and reported in order to maintain funding.

An outcomes assessment is an evaluation designed to discover if a program achieved its intended outcomes. Much like all of research, it comes with its own peculiar terminology that resembles an assembly line at a factory (Engel & Schutt, 2016). ^[1] Inputs are the resources needed for the program to operate. These include physical location, any equipment needed, staff (and experience/knowledge of those staff), monetary funding, and most importantly, the clients. Program administrators pull together the necessary resources to run an intervention or program. The program is the intervention your clients receive—perhaps giving them access to housing vouchers or enrolling them in a smoking cessation class.

The outputs of programs are tangible results of the program process—i.e., the boring things that come out of your program. Outputs in a program might include the number of clients served, staff members trained to implement the intervention, mobility assistance devices distributed, nicotine patches distributed, etc. By contrast, outcomes speak to the purpose of the program itself.

Outcomes are the observed changes, whether intended or unintended, that occurred due to the program or intervention. By looking at each of these domains, evaluation researchers can obtain a comprehensive view of the program.

Let’s run through an example from my wife’s social work practice. She runs an after-school bicycling club called Pedal Up for children with mental health issues. She has a lot of inputs in her program. First, there are the children who enroll, the volunteer and paid staff members who supervise the kids (and their knowledge about bicycles and children’s mental health), the bicycles and equipment that all clients and staff use, the community center room they use as a home base, the paths of our city where they ride their bikes, and the public and private grants they use to fund the program. Next, the program itself is a twice weekly after-school program in which children learn about bicycle maintenance and bicycle safety for about 30 minutes each day and then spend at least an hour riding around the city on bicycle trails.

In measuring the outputs of this program, she has many options. She would probably include the number of children and staff participating in the program or the number of bike rides or lessons given. Other outputs might include the number of miles logged by the children over the school year, the number of bicycle helmets or spare tires distributed, etc. Finally, the outcomes of the programs might include providing surveys to family members or teachers to see if each child’s mental health symptoms have improved, counting any behavioral issues at school, or conducting a child-friendly survey with the children themselves.

Outcomes assessments are performed at the end of a program or at specific points during the grant reporting process. What if a social worker wants to assess earlier on in the process if the program is on target to achieve its outcomes? In that case a process assessment is recommended, which evaluates a program in its earlier stages. Faulkner and Faulkner (2016) ^[2] describe three main goals for conducting a process evaluation.

The first is program description, in which the researcher simply tries to understand how the program looks like in everyday life for clients and staff members. In our Pedal Up example, assessing program description might involve measuring in the first few weeks the hours children spent riding their bikes, the number of children and staff in attendance, etc. This data will provide those in charge of the program an idea of how their ideas have translated from the grant proposal to the real world. If, for example, not enough children are showing up or if children are only able to ride their bikes for ten minutes each day, it may indicate that something is wrong.

Another important goal of process assessment is program monitoring. If you have some social work practice experience already, it’s likely you’ve encountered program monitoring. Agency administrators may look at sign-in sheets for groups, hours billed by clinicians, or other metrics to track how services are utilized over time. They may also assess whether clinicians are following the program correctly or if they are deviating from how the program was designed. This can be an issue in program evaluations of specific treatment models, as any differences between what the administrators conceptualized and what the clinicians implemented jeopardize the internal validity of the evaluation. If, in our Pedal Up example, we have a staff member who does not review bike safety each week or does not enforce helmet laws for some students, we could catch that through program monitoring.

The final goal of process assessments is quality assurance. At its most simple level, quality assurance may involve sending out satisfaction questionnaires to clients and staff members. If there are serious issues, it’s better to know them early on in a program so the program can be adapted to meet the needs of clients and staff. It is important to solicit staff feedback in addition to consumer feedback, as they have insight into how the program is working in practice and areas in which they may be falling short of what the program should be. In our example, we could spend some time talking with parents when they pick their children up from the program or hold a staff meeting to provide opportunities for those most involved in the program to provide feedback.

Evaluation research is a part of all social workers’ toolkits. It ensures that social work interventions achieve their intended effects. This protects our clients and ensures that money and other resources are not spent on programs that do not work. Evaluation research uses the skills of quantitative and qualitative research to ensure clients receive interventions that have been shown to be successful.