All of the discussions and written assignments in this course will focus on a program evaluation of one specific non-profit agency. Each week's discussion thread will focus on the issues that will be relevant to that week's written assignment. Use the discussion as a means to clarify your understanding of the issues related to the topics each week. This week, you will begin by selecting an agency, interviewing key personnel at the agency regarding the programs they provide, and developing a logic model that provides a graphic overview of the program's purpose, services, inputs, activities, outputs, and outcomes that will be evaluated.
Please note that, due to the very short duration of this course, it will likely not be possible to actually collect and analyze data from that agency. You will have the option to collect agency data (with the permission of the agency, of course) or to create data files using hypothetical data in future weeks.
Review the PowerPoint presentation of a program evaluation conducted at Program Evaluation for Broward Partnership for the Homeless.pptx
In this discussion thread, please identify the following. Items 1, 2, and 3 are due by Wednesday; item 4 is due by Sunday.
- The name of the agency that you will be evaluating in this course NOTE: If the agency prefers, you may use a pseudonym for confidentiality purposes. The agency should provide psychological and/or social services and be a not-for-profit agency.
- Discuss the ethical issues involved with being a consultant for a social services agency. How will the confidentiality of the clients be maintained throughout your program evaluation project? Are there other ethical or legal issues that need to be considered?
- By the end of the week, you will need to interview at least one key personnel at the agency. When deciding who to interview, consider the following:
- Who can provide the best information about the services delivered by the agency?
- Who can provide the best information about how the agency currently measures outcomes related to effectiveness of their programs?
- Review the components that are required for your logic model. The person(s) interviewed should be able to explain the inputs, activities, outputs, and desired short-term, intermediate-term, and long-term outcomes.
Broward Partnership for the Homeless, Inc. (BPHI)
Program Evaluation Study
Single-site program evaluation
Homeless shelter outcomes
1 ½ year duration
2001 – 2003
BPHI
Homeless shelter in Broward County, FL
Approximately 1700 residents per year
Up to one-year stay
Average six months
200 beds
10 family units
BPHI – Services
Case Management
Psychiatric Treatment
Substance abuse, mental health counseling
Job training
Resume Writing
Education
GED
Referrals for educational services
Employment Services
BPHI – Services
Three meals a day
Medical evaluation and treatment
Dental office
Child care
Library
Clothing
Hair salon
Message service
BPHI – Need for Evaluation
Funding agencies require evidence of treatment effectiveness
Donors often require evidence
No data to support effectiveness of services
BPHI – Program Evaluation
Grant from Health Foundation of South Florida
Goals
Increase behavioral health services
Increase engagement in center residents
Improve behavioral stability in center residents
Reduce symptom severity
Increase functionality
Provide evidence of results achieved
Research Design Considerations
Pre-Post design with control group
What is the main purpose of assessment?
Who will do the assessments?
What type of assessment?
Is diagnostic interview necessary?
Assessment versus screening
BPHI – Measures
Public Domain = Free!
CAGE for drug/alcohol screening
Mental Health Screening Form (MHSF)
SOFAS
GAF
BPHI – CAGE
1. Have you felt you ought to cut down on your drinking or drug use?
2. Have people annoyed you by criticizing your drinking or drug use?
3. Have you felt bad or guilty about your drinking or drug use?
4. Have you ever had a drink or used drugs first thing in the morning to steady your nerves or to get rid of a hangover (eye-opener)?
BPHI – MHSF
Mental Health Screening Form (MHSF; Carroll & McGinley, 2000).
Screen for mental health symptoms
Designed for use with co-occurring disorders
17 items
Instrument available here
BPHI – SOFAS
Social and Occupational Functioning Scale (SOFAS; American Psychiatric Association, 2000)
Focus on social, occupational functioning
Impairment not exclusively due symptom severity
Includes general medical conditions
Case Manager rated at intake and discharge
BPHI – GAF
Global Assessment of Functioning (GAF; American Psychiatric Association, 2000)
Impairment due to symptom severity
Impairment not due to general medical conditions
Rated by therapist/psychiatrist each session
BPHI – Issues
Training
Oversight
Supervision
Data Collection
Data Entry
Results
BPHI |
Compared those treated to not treated |
↓ Substance abuse symptoms |
↓ Psychiatric symptoms |
↑ SOFAS scores |
↑ GAF scores |
↑ attained housing |
References
American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text revision). Washington, DC: Author.
Carroll, J.F.X., & McGinley, J.J. (2000). Mental Health Screening Form III. Tampa, FL: Project Return Foundation. Retrieved from http://smchealth.org/sites/default/files/docs/1309587941MentalHealthScreeningFormIII.pdf