Staff Perspective: But, Does It Work? Resources for Designing, Implementing, and Interpreting Program Evaluation for Behavioral Health Programs
Are you a director charged with managing and optimizing a clinic, program, or department? Have you developed a program to address a specific need in the field of behavioral health and need evidence of its effectiveness to share with stakeholders? Are you a provider interested in tracking and improving the outcomes of your clinic or caseload? A yes to any or all of the questions suggests that you are in a position to benefit from standardized and methodologically sound program evaluation. That may sound like a daunting prospect, particularly since you are likely busy with the day-to-day tasks of running your clinic, program, practice, etc. But with the correct information and resources, an effective and successful evaluation of any size program is achievable.
In a previous blog, I described the ongoing program evaluation efforts here at the Center for Deployment Psychology (CDP). CDP uses a wide variety of evaluation and assessment techniques that have been gleaned from a diversity of resources. Today I’d like to share some of those resources with our readers in hopes that you will find a resource (or several) that can help initiate or improve program evaluation in your clinic, program, or practice.
- For Training Programs - The Kirkpatrick Model
In my previous post, I referenced CDP’s use of the Kirkpatrick Model as a foundational perspective in our program evaluation efforts. This model was initially developed in the 1950s and updated more recently as the New World Kirkpatrick Model to include additional assessment and indicators of performance for more contemporary goals and practices. The Kirkpatrick Model focuses on evaluation of training activities and may be particularly helpful for clinics or programs that have an educational or training component. With a four-level format, the Kirkpatrick Model guides programs through the assessment of (1) reaction, (2) learning, (3) behavior, and (4) results to determine overall program effectiveness. The linked website provides helpful resources and training options for those interested in exploring the use of this model further.
- For Psychological Health (PH) and Traumatic Brain Injury (TBI) Programs – Defense Centers of Excellence (DCoE) Program Evaluation Resources
DCoE has a well-established program evaluation and improvement arm that provides numerous trainings and resources for individuals interested in assessing the effectiveness of their program, clinic, or organization. Although many of these resources were designed with military applications in mind, the basic information that they impart would be beneficial to a variety of programs that focus on PH and TBI. Their program evaluation website includes links to the DCoE Program Evaluation Guide, 2nd Edition (Modules 1, 2, 3, 4, 5, & 6), a 6-step guide to designing and conducting evaluation of PH and TBI programs. They also offer a library of two dozen archived webinars (2014-15 webinars, 2016 webinars) with transcripts and slides that explain important evaluation concepts, provide information for designing and implementing evaluation, and suggest methods for interpreting and applying evaluation outcomes.
- For Public Health Programs - Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide (CDC)
The Program Performance and Evaluation Office (PPEO) of the Centers for Disease Control (CDC) released the Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide in October 2011. Available in both an online HTML and downloadable PDF format, the PPEO describes this manual as “a ‘how to’ guide… intended to assist managers and staff of public, private, and community public health programs to plan, design, implement and use comprehensive evaluations in a practical way”. The guide walks program staff through a 6-step self-study process that culminates in the productive use/application of the evaluation findings and the sharing of lessons learned with program staff and stakeholders. Of particular interest is a section in the Introduction that provides criteria to distinguish Program Evaluation activities from Research activities, a distinction that may be difficult depending on the design and nature of the program being evaluated.
If none of the above resources fit your specific program evaluation needs or if you’re interested in an even broader library to choose from, you may want to visit the Substance Abuse and Mental Health Services Administration (SAMHSA) Evaluation Tools and Resources page. SAMHSA has collected more than a dozen links from government and academic institutions to a wide array of tools, aides, tip sheets, and other resources for individuals seeking guidance in the design and implementation of evaluations.
Are you a director charged with managing and optimizing a clinic, program, or department? Have you developed a program to address a specific need in the field of behavioral health and need evidence of its effectiveness to share with stakeholders? Are you a provider interested in tracking and improving the outcomes of your clinic or caseload? A yes to any or all of the questions suggests that you are in a position to benefit from standardized and methodologically sound program evaluation. That may sound like a daunting prospect, particularly since you are likely busy with the day-to-day tasks of running your clinic, program, practice, etc. But with the correct information and resources, an effective and successful evaluation of any size program is achievable.
In a previous blog, I described the ongoing program evaluation efforts here at the Center for Deployment Psychology (CDP). CDP uses a wide variety of evaluation and assessment techniques that have been gleaned from a diversity of resources. Today I’d like to share some of those resources with our readers in hopes that you will find a resource (or several) that can help initiate or improve program evaluation in your clinic, program, or practice.
- For Training Programs - The Kirkpatrick Model
In my previous post, I referenced CDP’s use of the Kirkpatrick Model as a foundational perspective in our program evaluation efforts. This model was initially developed in the 1950s and updated more recently as the New World Kirkpatrick Model to include additional assessment and indicators of performance for more contemporary goals and practices. The Kirkpatrick Model focuses on evaluation of training activities and may be particularly helpful for clinics or programs that have an educational or training component. With a four-level format, the Kirkpatrick Model guides programs through the assessment of (1) reaction, (2) learning, (3) behavior, and (4) results to determine overall program effectiveness. The linked website provides helpful resources and training options for those interested in exploring the use of this model further.
- For Psychological Health (PH) and Traumatic Brain Injury (TBI) Programs – Defense Centers of Excellence (DCoE) Program Evaluation Resources
DCoE has a well-established program evaluation and improvement arm that provides numerous trainings and resources for individuals interested in assessing the effectiveness of their program, clinic, or organization. Although many of these resources were designed with military applications in mind, the basic information that they impart would be beneficial to a variety of programs that focus on PH and TBI. Their program evaluation website includes links to the DCoE Program Evaluation Guide, 2nd Edition (Modules 1, 2, 3, 4, 5, & 6), a 6-step guide to designing and conducting evaluation of PH and TBI programs. They also offer a library of two dozen archived webinars (2014-15 webinars, 2016 webinars) with transcripts and slides that explain important evaluation concepts, provide information for designing and implementing evaluation, and suggest methods for interpreting and applying evaluation outcomes.
- For Public Health Programs - Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide (CDC)
The Program Performance and Evaluation Office (PPEO) of the Centers for Disease Control (CDC) released the Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide in October 2011. Available in both an online HTML and downloadable PDF format, the PPEO describes this manual as “a ‘how to’ guide… intended to assist managers and staff of public, private, and community public health programs to plan, design, implement and use comprehensive evaluations in a practical way”. The guide walks program staff through a 6-step self-study process that culminates in the productive use/application of the evaluation findings and the sharing of lessons learned with program staff and stakeholders. Of particular interest is a section in the Introduction that provides criteria to distinguish Program Evaluation activities from Research activities, a distinction that may be difficult depending on the design and nature of the program being evaluated.
If none of the above resources fit your specific program evaluation needs or if you’re interested in an even broader library to choose from, you may want to visit the Substance Abuse and Mental Health Services Administration (SAMHSA) Evaluation Tools and Resources page. SAMHSA has collected more than a dozen links from government and academic institutions to a wide array of tools, aides, tip sheets, and other resources for individuals seeking guidance in the design and implementation of evaluations.