Administration for Children and Families Children's Bureau regional partnership grants : semi-annual report.

              Administration for Children and Families Children’s Bureau
Regional Partnership Grants
(Semi-Annual Report)
Grantee Name and Address:
Oklahoma Department of Mental Health and Substance Abuse Services
1200 NE 13th Street
Oklahoma City, Oklahoma 73152
Grant Number: 90CU0038
Period covered by report: 9/30/2009 through 3/31/2010
Principal Investigator/Project Director:
Jessica Hawkins, Principal Investigator
Elicia Chandler, Project Director
Author’s Name and Telephone Number:
Elicia Chandler (405) 522-0077
Name of Federal Project Officer:
Melissa Lim Brodowski
Name of Grants Management Specialist:
Bernard Morgan
I. RPG Project and Evaluation Implementation and Activities
This section of the semi-annual report should provide a descriptive report of your projects
planning and implementation activities and plans for the next 6 month period.
A. Major activities and accomplishments
•
Describe activities the project engaged in over the past six months, focusing on the key program and evaluation goals, objectives and activities.
Revised 03/2009 1
•
Using the attached table format, provide a listing of program and evaluation goals or objectives, dates completed (if applicable), a brief report of work accomplished, and anticipated activities in the next six months.
A. Major activities and accomplishments:
The Oklahoma Partnership Initiative (OPI) has fully implemented all four (4) project objectives. During this reporting period recruitment and retention effort improved, and there has been an increased request for community education trainings concerning substance exposed newborns.
•
Recruitment of substance abuse providers for statewide implementation of Strengthening Families Program Curriculum.
•
Tulsa County substance abuse providers trained for Strengthening Families Program Curriculum.
•
Continuous early intervention and preventive services for children of substance abusing parents through evidence-based programs. New Directions Program will begin its fifth intervention group cycle in May, 2010, focused on helping children in foster care overcome the effects of parental substance abuse. Specialized Outpatient Services implemented the Strengthening Families Program Curriculum, and they have completed over forty-two weeks of family group sessions.
•
New Directions group treatment curriculum was enhanced and revised to incorporate feedback from the treatment team, participating families, and expert consultants, Jerry Moe, National Director of Children’s Programs at the Betty Ford Center, and Robin Gurwich, Program Coordinator at National Center for School Crisis and Bereavement.
•
There has been an increased retention rate with New Directions treatment groups.
•
Project staff participated in OKDHS quarterly training providing brochures and information to case workers and staff.
•
New Directions Program provided all group participants with Developmental screenings, IQ/Cognitive screenings, mental health assessments, education. Linkage, and advocacy provided to foster parents, parent-child reunification support sessions (as needed), and individual therapy services.
•
Statewide pre and postnatal substance exposure training provided to over 170 medical and health professionals as a result of projects goal to expand accessibility of services to newborns with prenatal exposure through enhanced identification and intervention. Substantial request from statewide substance abuse conference organizers for interventions trainings.
•
The UNCOPE Screening tool is embedded within OKDHS new Family Functioning Assessment in pilot counties, statewide implementation expected, June 2010.
•
OPI Steering Committee, Data Sharing Group, and Screening Tool and Sustainability Sub-Committees continue to convene monthly meetings.
Revised 03/2009 2
•
Attendance at February, 2010, RPG Special Topics Conference with New Directions and Specialized Outpatient Services staff, and Project Evaluators.
•
Participation in all Child Focused Cluster Calls.
•
Participation in all OPI Project related update calls with FPO, PML, and Project Evaluators.
•
Ongoing participation in community awareness events for program recruitment and retention efforts.
Six (6) Month Plan:
•
Contract with Strengthening Families Program developers to expand the ages of children that Specialized Outpatient Services (SOS) can provide evidence based training to include ages 3-5. (OKDHS data indicates that there are a significant number of potential participants in 3-5 year age bracket.)
•
Increased collaborative efforts with foster and therapeutic foster serving agencies.
•
Ongoing and Continuous training of OKDHS child welfare staff statewide on UNCOPE screening tool.
•
The UNCOPE Universal Screening Tool implemented statewide. (Currently being piloted in several Oklahoma Counties).
•
Project evaluators collaborate with OKDHS to develop a mechanism to obtain the UNCOPE data during the pendency of state legislative action.
•
Ongoing recruitment and retention efforts for New Directions and Strengthening Families Programs.
•
New Directions (The University of Oklahoma Health Sciences Center) and OKDHS execute legal agreement regarding the procedure for providing consent for children in state custody.
•
Attendance at October 2010, Grantee Meeting in Washington, D.C., by OPI Project Director, New Directions staff, Specialized Outpatient Services Staff, and Project Evaluators.
•
Continued efforts to develop OPI Project sustainability plan with Performance Management Liaison and Children’s and Families Future’s technical assistance staff.
•
New Directions Program project staff will develop a preliminary sustainability plan
for their intervention.
Revised 03/2009 3
B. Program and Evaluation Challenges/Barriers
•
Describe any challenges or barriers encountered during the reporting period and their effect on project implementation. Include programmatic barriers (e.g., lower referrals than expected, inability to provide services as planned) or evaluation challenges (e.g., difficulty accessing needed data on target population, challenges in engaging program staff to participate in evaluation activities).
•
Describe efforts and outcomes in overcoming the identified challenges/barriers.
•
Include a description of any key lessons learned regarding program and evaluation implementation.
For purposes of this semi-annual report, activities are divided by OPI interventions, and grouped where applicable. Project evaluation services have focused work on providing support and guidance in increasing recruitment and referrals, assistance with data uploads and reporting requirements, resolve data issues surrounding the UNCOPE Universal Drug and Alcohol Screening Tool, and the resolution of outstanding data sharing agreement issues. Project evaluation activities pertaining to plans, progress and problems are outlined below:
Increasing Recruitment: New Directions and Strengthening Families Program enrollment have been lower than initially projected. In the past six months numbers have almost doubled, and one of the two programs has a waiting list. The New Directions program is a new intervention; therefore they had a very slow start up period, and had to do extensive community promotion. As a result of New Directions community promotion efforts they now have a waiting list, and are considering starting concurrent group sessions. We originally proposed a wait list control group, and we received approval to use an alternative option and create a comparison group through the use of propensity score matching in the DHS-AFCARS database. The rationale for variation from the original proposal was due to the fact that New Directions staff worked diligently to get families referred, and the child welfare workers were against their families being placed on a wait list as a control group; consequently the procedure was modified to ensure family participation when sufficient referrals were obtained.
Strengthening Families Program (SFP) has also increased their numbers served, and project evaluators continue to actively work with them to increase their program visibility. Troubleshooting concerning low numbers revealed that the major referral source for SFP (as the intervention was designed to be implemented) was not yielding the program referrals. There was a miscommunication between the referring source and the provider, and this has been corrected and numbers have
Revised 03/2009 4
improved. Additionally, there is a need for SFP providers at Specialized Outpatient Services to expand the ages of children that they can provide SFP to include ages
3-5. Project Director plans to contract with SFP developers to expand ages served.
The issues related to recruitment are important to the overall success of the project. As a result of the numbers being lower than originally proposed, evaluators have not had a sufficient sample size for meaningful analyses. The SFP cases have been tested for program effects, and those findings will be included in the semi-annual report.
Data Uploads and RPG related activities: Evaluators for this project also have focused attention on refining the process of getting data for the required uploads, and ensuring the process is running smoothly. New Directions is submitting information on the instrument specific upload (PSI), and this has required a considerable amount of work, as the data was not being kept in a form that could be directly uploaded. A new protocol and procedure has been established, and future uploads should operate more efficiently. New Directions staff has been unable to consent foster children to participate in the research portion of the program since early fall of 2009. The consent issue is the result of a newly developed OKDHS policy regarding procedures for procuring research consent for children in OKDHS custody.
Data Issues Surrounding the UNCOPE implementation: The OPI Steering Committee selected the UNCOPE Universal Drug and Alcohol Screening Tool for implementation. During the selection process, OKDHS Child Welfare was undergoing changes in their assessment process, and developed a new assessment tool, the Family Functioning Assessment (FFA). Training is occurring statewide, and OKDHS offices have been utilizing the new FFA in piloted counties. The UNCOPE was embedded into this new screening tool, and will be utilized throughout the state as part of the “roll-out” of the State of Oklahoma’s Performance Improvement Plan. Unfortunately, the family functional assessment tool is not being entered into the state child welfare database, as there are delays in getting the necessary changes made at the state level. The Oklahoma legislature enacted legislation last year that mandated prioritization of children’s health records for data system changes, and any other changes to the state’s database will have to occur after finalization of health record activity. These changes could take up to two years to occur, and this has resulted in an inability of the evaluators to track how the UNCOPE is being used, and the outcomes of the assessments. Evaluators are currently in discussions with child welfare leadership to devise an alternative option for accessing data.
Data Sharing Agreement Issues: The fourth objective for OPI is improvements in the cross-system information sharing mechanisms between ODMHSAS and OKDHS to ensure consistent data collection across the substance abuse and child welfare systems. Evaluators have maintained existing data sharing agreements with OKDHS in order to work on other child welfare related projects. However, as it relates to the OPI project it was decided that the preexisting
Revised 03/2009 5
agreements were insufficient and required revisions. A resolution to this issue remains outstanding, and an untimely agreement could potentially delay future data uploads. ODMHSAS, OKDHS, and project evaluators are currently in discussions to resolve pending issues.
C. Contextual Events or Community Changes:
•
Describe any significant contextual conditions, events or community changes that took place over the reporting period which have already had or will likely have an influence on your project or impact the outcomes you are measuring for your target population. Include things such as the implementation of other child welfare initiatives, policies or programs; community events such as a child death or high profile case that might impact caseloads; changes in judicial officers who hear dependency cases (if relevant); changes in agency or community leadership; implementation of other new legislation, policies or procedures that affect your program or target population; changes to State/county budgets or funding that affect services to your target population; changes in child welfare or substance use trends; other related community developments, etc.
•
State of Oklahoma’s Performance Improvement Plan has been approved.
•
The state of Oklahoma is experiencing a major budget crisis with significant reductions occurring in treatment and mental health services. Much of the social service infrastructure, both public and private, is in danger of being lost due to the anticipated budget cuts.
•
Substance abuse agencies have experienced a 20% increase in waiting lists over the past two months.
•
OKDHS data reflects an approximate 11% decrease in the number of children in out of home placement.
D. Departures, Additions and Deletions from Original Application or
Implementation Plan:
•
There were no departures, additions, or deletions from original application during this reporting period.
E. Technical Assistance Activities:
•
Describe programmatic and evaluation technical assistance needs. To request technical assistance, contact your PML or Larisa Owen at rpgsc@cffutures.org.
•
Ongoing and continuous support and collaboration with ACF Regional Coordinator, Dana Huckabee. Ms. Huckabee has provided linkage and support within the OKDHS system, and she also continuously provides OPI Steering
Revised 03/2009 6
Committee members with referrals and resources concerning our project objectives.
•
Continuous technical assistance on behalf of Children and Family Futures staff concerning the development of Oklahoma’s Sustainability Plan.
F. How Knowledge and Information from Grantee Meetings is
Being Used:
OPI Project evaluators attended the pre-conference evaluation session, and indicated that it was a much needed addition to the grantee meeting format, and appreciated the opportunity to dialogue with evaluators across the entire grantee continuum, instead of the usual cluster focused formats. Evaluators advised that the session allowed them the opportunity to discuss on a large scale their concerns, hurdles, and highlights.
Project staff participation in the sustainability session was for ongoing support as our project endeavors to formulate a sustainability plan. Some of the ideas and support provided definitely enhanced my ability to stimulate our sustainability committee members thought processes during our subcommittee meetings. We were encouraged to follow-up for technical assistance, and they provided guidance on additional forms and tools to use during the process, and we are certainly optimistic that our participation will help advance our efforts.
The TF-CBT cultural adaptation for American Indians was attended by evidence based intervention staff whose program also utilizes TF-CBT and they advised that the presentation stimulated their desires to improve services and client engagement. The story telling techniques session was attended as our project has experienced some difficulties with recruitment and referrals and this was selected to improve our techniques as we attempt to educate potential referral sources, and exceptional ideas were given concerning marketing our projects.
II. RPG Program Evaluation
In this section, focus on program outcome and process evaluation data (both quantitative and qualitative) obtained during the reporting period.
A. Overview of Major Findings:
•
Describe any major findings from evaluation activities and your interpretation of these findings. Findings may be preliminary and based on initial observations or data collection activities (e.g., descriptions of program participants at baseline; short-term outcomes specified in the logic model). Attach any local evaluation reports completed during this reporting period, in addition to or as an alternative to this section.
Revised 03/2009 7
The overall goal of this project is to intervene effectively and early to prevent and reduce the risks for children associated with parental methamphetamine and/or other substance abuse. The following are the projects four (4) objectives:
Objective 1: Implementation of Universal Alcohol and Drug Screening for parents
in child welfare systems. “IMPLEMENTED in piloted counties, Statewide implementation expected summer, 2010.”
OPI has selected and embedded the UNCOPE screening tool within the
Oklahoma Department of Human Services (OKDHS) “newly” revised Family Functioning Assessment. The new OKDHS assessment
is currently being piloted in several Oklahoma Counties, and is scheduled
for use across the entire state by mid-year 2010. Tracking the usage and
outcomes associated with the UNCOPE is in progress as OKDHS is
currently in process of updating its OKKIDS data system. Project
evaluators are currently in communication with OKDHS in an effort to seek
a resolution to this issue.
Objective 2: Expansion of accessibility of services to newborns with prenatal
substance exposure through enhanced identification and
intervention with this population. “FULLY IMPLEMENTED”
Oklahoma University Health Sciences Center (OUHSC), has fully
developed and implemented a community education training for
health care professionals on substance-exposed newborns and related
issues. During this reporting period OUHSC and Dr. Robin Gurwitch
facilitated six (6) community education trainings, with over 170 participants from medical, health, child welfare, juvenile justice, corrections, courts, substance abuse, and ODMHSAS in attendance.
Objective 3: Early intervention and preventive services for children and
adolescents of substance abusing parents through evidence-based
programs. “FULLY IMPLEMENTED ongoing recruitment efforts”
There have been two evidence-based programs implemented under
the OPI project. New Directions targets families in foster care and the
foster parent. Specialized Outpatient Services is providing Strengthening
Families Program curriculum to children in out of home placement due to their parents substance use. The interventions participants have been behind initial projections, however, project staff have identified barriers to recruitment and developed plans to address priority barriers.
Process evaluation (fidelity check) was done during this reporting period, along with an evaluation report from developers of Strengthening Families Program Curriculum.
Revised 03/2009 8
Objective 4: Improvements in cross-system information sharing mechanisms to
Ensure consistent data collection across the substance abuse and
child welfare systems. “FULLY IMPLEMENTED”
As a result of the implementation of the Oklahoma Partnership Initiative (OPI) an over-arching data sharing agreement was developed, finalized and executed by the Oklahoma Department of Mental Health
and Substance Abuse Services, and the Oklahoma Department of Human
Services. The agreement is in place, and data is cooperatively being
exchanged across systems.
Client Vignettes:
There was a five-year-old boy who rarely showed emotions, especially mad or sad. Tw weeks after the children’s groups participated in the feelings curriculum, the boy’s foster parents reported a “breakthrough.” The foster child suddenly burst into tears at home, repeatedly crying aloud, “mommy.” He had not ever cried in front of his foster family despite living with them for over ten months. The foster parents quickly remembered to use skills they had learned in the group, by providing emotional support to their foster child and trying to calm him. Once he was calm, he told his foster parent that he learned in New Directions group that it was “okay” to be sad and cry. The foster parents reported that this experience helped them understand their foster child’s behavior and how to help him cope.
A nine-year-old boy participating in the New Directions Program advised staff that after attending the first group he realized that he was not “alone” because he met other children in the group who have moms and dads that use alcohol and drugs, and who are also in foster care. This child’s foster mom stated that during group sessions the kids have “light bulb” moments, and the child suddenly “gets” why they are in foster case. Foster mom also reported that the child appeared calmer and less disruptive for several days following his revelation.
The foster mom of a five-year-old girl was given the responsibility of telling the girl that her parents’ rights were being terminated. The foster mother did not know how she could deliver such awful news as she knew the girl would be very upset. She talked about the situation in the New Directions caregivers’ group and received significant support from the other foster parents. There was guidance provided by the therapists, she used children’s books about termination of parental rights and adoption to tell her foster child the news concerning her parents. The foster mother reported that the books and suggestions from the other foster parents made it easier for her to talk to her foster child, answer the child’s questions, and to provide emotional support.
A child welfare case worker contacted SFP staff to discuss a parent/child interaction that she had witnessed between a client who had graduated the Strengthening Families Program. The case worker advised that she was very impressed with the positive
Revised 03/2009 9
integrations that she had witnessed with the family, and was amazed at the parents commitment to maintain the skills she were taught while participating in the group. She stated that she overheard the mother holding her child responsible for his poor behavior at school by the use of the term “person power.” Utilizing this terminology the child was able to understand that he was being held accountable for his choices and behavior.
Revised 03/2009 10
`
STRENGTHENING FAMILIES PROGRAM SUBSTANCE ABUSE PREVENTION INITIATIVE
Revised 03/2009 11
State of Oklahoma
Strengthening Families Program
YEAR TWO EVALUATION REPORT
Reporting Period:
October 1, 2008 – September 30 2009
This report was prepared by:
Karol L. Kumpfer, Ph.D.,
Jeanie Ahearn Greene, Ph.D., MSW
Keeley Cofrin, Ph.D.
Henry Whiteside, Ph.D.
Qing Qing Hu, M.S.
LutraGroup, Inc.
5215 Pioneer Fork Road
Salt Lake City, Utah 84108
(801) 582-1652
State of Oklahoma
Strengthening Families Program Initiative
YEAR TWO EVALUATION REPORT
(October 1, 2008 – Sept. 30, 2009)
I. INTRODUCTION AND OVERVIEW
The State of Oklahoma in conjunction with the University of Kansas has implemented an evidence-based model parenting program as part of a state-wide strategy for the prevention of child maltreatment in children and their families affected by methamphetamine or other substance abuse. The children involved in this parenting program are either 1) at risk of removal from their homes or 2) have been removed with the goal of reintegration. They have participated in an educational family skills training program (Strengthening Families Program) to positively impact the following domains: parenting, family attachment, parental substance use, understanding risk and protective factors to avoid substance use, and child behavior. It is anticipated that through participation in this program, children will be maintained in their own homes or reintegration for those already out of home will occur more quickly.
State of Oklahoma in conjunction with the University of Kansas has implemented SFP with children of substance abuse with funding from a grant from the Administration of Children and Families (ACF) in Washington, D.C. Based on assessed community needs and risk factors, the State chose to implement the Strengthening Families Program (SFP). SFP is designed to reduce environmental risk factors and to increase personal resilience and improve protective factors to drug use in high-risk youth. Research confirms that SFP is also effective in reducing risk precursors for mental disorders and juvenile delinquency. This program has been adapted, translated and implemented in a variety of cultures and communities including the type of families served by this project.
This report is the Year 02 annual evaluation report for this project, which spans from October 1, 2008 to September 30, 2009. At the end of Year 01, the State had successfully implemented SFP at one agency in the state, including training staff in the evidence-based model, pilot testing SFP groups for feasibility in child abuse prevention family services under the coordination of the State of Oklahoma project coordinator, Elicia Chandler. However, there was not enough data to conduct the outcome evaluation until Year Two. This year we have data from two SFP groups to conduct the outcomes data analysis. In Year Three, LutraGroup was also contracted to conduct a fidelity evaluation with site visits to the agencies implementing SFP as well as the outcome evaluation of the pilot cycles. The following sites participated in this initiative in Year 01 and in Year 02:
Revised 03/2009 12
Year Two State of Oklahoma Strengthening Families Program
Child Abuse Prevention Initiative Cycles
2008-2009
Cycle
Agency
Cycle Start Date
SFP Site Location
Curriculum
Spring 2009
Specialized Outpatient Services. (SOS)
2/26/2009
OKC, OK
SFP (6-11)
7 pre /6 post
Fall 2009
Specialized Outpatient Services. (SOS)
8/19/09
OKC, OK
SFP (6-11)
6 pre-tests
Fall 2009
Specialized Outpatient Services. (SOS)
11/12/2009
OKC, OK
SFP (6-11)
2 pre & post
Spring 2010
Specialized Outpatient Services. (SOS)
1/07/2010
OKC, OK
SFP (6-11)
5 pre
An independent evaluation is being conducted by the University of Kansas, with a subcontract to LutraGroup, Inc., to measure the implementation fidelity and program effectiveness of SFP. The evaluation includes a process evaluation (to begin in Year 3) to assess fidelity and an outcome evaluation (all years) to measure effectiveness with comparisons to the SFP National Norms. The report is organized with an introduction, methodology of the evaluation, outcome evaluation findings and conclusion with summary and recommendations for future directions for the initiative.
SFP Program Description. The Strengthening Families Program 3-16 Years (Kumpfer & DeMarsh, 1989; Kumpfer, DeMarsh, & Child, 1989) is an evidence-based 14-week family skills training program. There are four age versions of SFP—3-5 Years, 6-11 Years, 12-16 years that were designed for children of drug abusing parents. SFP is one of only four parenting programs developed and tested specifically for children of substance abusers. Additionally, SFP is the only one with independent replications in randomized control trials that also demonstrates significant improvements in the outcomes for the children rather than just improved parenting skills and reductions in child maltreatment (Kumpfer & Johnson, 2008; in press).
SFP is unique and most effective because it involves the whole family in three classes run on the same night once a week. The parents and or foster or kinships caretakers of drug endangered children in child protective services attend the SFP Parent Training Program in the first hour. At the same time their children attend the SFP Children’s Skills Training Program. In the second hour, the families participate together in a SFP Family Skills Training Program. The program utilizes four Group Leaders and a Site Coordinator to run the program. Multiple replications of SFP in randomized control trials with different ethnic groups by independent evaluators have found SFP to be an effective program in reducing multiple risk factors for later drug abuse, mental health problems, and delinquency by increasing family strengths, children’s social competencies, and improving parent’s parenting skills (Kumpfer, Alvarado, Smith, & Bellamy, 2002).
Revised 03/2009 13
II. SCOPE AND METHOD OF THE EVALUATION
The major goal of this evaluation is to determine if the program is still working and effective when implemented with less research controls and determine the effectiveness of the program at provided by one State of Oklahoma drug treatment contracted agency to families in their catchment area in relation to the reported levels of fidelity based on SFP identified best practices (Appendix 1).
This includes only an outcome evaluation conducted by an outside contractor to assure the fidelity and effectiveness of SFP. The process evaluation has begun in Year 3 to include a fidelity survey of funded cycles and a site visit to assess program fidelity. The outcome evaluation involves a repeated measures retrospective pre- and post-test design with standardized instruments being administered to parents attending the program. The outcome evaluation assesses program effectiveness for identified and targeted parent, child and family risk and protective factors for substance abuse and delinquency prevention.
Evaluation Contractors: LutraGroup, Inc. in collaboration with Ahearn Greene Associates. The contracted evaluator to measure the fidelity and effectiveness of SFP with the participating families is LutraGroup, Inc. The evaluation contractor is comprised of a team of health and human service professionals with combined expertise in evaluation, research, substance abuse treatment and prevention, mental health and multi-system intervention. The professionals in this consulting company are very experienced in conducting research and evaluations of the Strengthening Families Program over the last 20 years. The SFP program developer, Dr. Karol Kumpfer, is the Evaluation Director for LutraGroup. LutraGroup is also the sole authorized contractor for SFP training, technical assistance, program evaluation and program development in the United States, Canada, and Europe. They have provided the SFP training of group leaders, evaluation and technical assistance for evidence-based SFP implementations throughout the United States since 1988. They are familiar with the community context and issues effecting at-risk families, with special attention to families involved with child protective services and histories of substance abuse.
Ahearn Greene Associates, through a subcontract with LutraGroup is responsible for the process evaluation of the Oklahoma SFP. The Oklahoma SFP process evaluator, Jeanie Ahearn Greene, Ph.D., MSW, has combined expertise in research and clinical practice and is responsible for technical assistance, building of evidence- base, training and program development of multiple established evidence-based substance abuse and health promotion programs since 1993. They are familiar with the community context and issues effecting at-risk families, particularly in implementation of evidence-based practices with special populations, including child protective involvement, criminal justice involvement and substance abuse. Dr. Ahearn Greene is responsible for the fidelity evaluation design, site visits, quality assurance, and reporting of fidelity levels for the Oklahoma SFP site.
III. Year Two Outcome Evaluation Report
Revised 03/2009 14
Year One and Two Client Change Outcomes:
As can be seen from the tables below, the SPSS data analysis found 19 of the 21 outcomes (91%) of the outcomes had significant positive results with small to medium effect sizes for most of the parent and family outcomes as found in Strengthening Families sites nationwide. This is better than last year when 18 of 21 (86%) of the outcomes had significant positive results. Although it is unusual to have so many significant outcomes (91%), as will be discussed in more detail below, the effect sizes in Year 1 were about half the size typically found for SFP outcomes. However, the effect sizes are larger this year probably because the group leaders are more experienced. Another reason for the smaller effect sizes is likely because of the requirement to have a regular pre-test rather than also doing a retrospective pre-test at posttest as is done for most of the families in the SFP National Database. Such high-risk families as are in this Kansas sample that have open cases with child protective services have been found to have a self-reporting bias at baseline to indicate that there are no problems in their parenting style or their children. Indeed this is the case for this sample in Kansas. For example, they report all outcomes as better at baseline for the Kansas families than for the National Sample except in family conflict and depression. Having such high scores at the pre-test tends to make the effect sizes smaller because they are already almost “perfect” and can’t report much more improvement by the post-test. This is called a “ceiling effect” in analysis.
Outcome Evaluation Methods. The Experimental Evaluation Design consists of a quasi-experimental, repeated measures, pre- and post-test design with post-hoc subgroup comparisons as recommended by Campbell & Stanley (1967) to control for most threats to internal and external sources of validity. An “SFP Retrospective Parent Pre/posttest”, using standardized CSAP and NIDA core measures, was developed and used because of the need for a short, non-research quality, practitioner-friendly evaluation instrument. Instruments are to be delivered by the site staff. These instruments are designed to assess child and parent mental health, substance abuse risk and resiliencies, family management and cohesiveness, and parent and child social skills and attitudes. The data is recorded by the parents on printed questionnaires. These data on the pre and post-tests will be hand-entered and analyzed using SPSS by Dr. Keely Cofrin of LutraGroup using standardized scales for 20 outcome variables.
Evaluation Measurement Instruments. A multi-measure, multi-informant (child, parent, and group leader) data collection strategy will be used to improve triangulation of the data to approximate real changes being measured. Hence, immediately before at an orientation session and after completion of SFP at graduation, participating families will complete a number of outcome instruments selected to measure the hypothesized change variables or outcomes for the family changes, child changes and the parent changes. The risk and protective factor precursors of substance abuse include negative or positive child behaviors, parenting stress and depression or substance use and lack of effective discipline methods and family dysfunction. The children’s change outcomes will be measured by the Parent Observation of Children’s Activities (POCA) by Dr. Shep Kellam at Johns Hopkins University. This standardized test is a modification of the Child Behavior Checklist (Achenbach & Edelbrock, 1988) to be more sensitive to change (has a 5-point scale vs. only at 3-point scale and less offensive or clinical wording to be more acceptable to parents. We measure 8 child outcome scales as shown below in the table. The children’s social and life skills will be measured by selected items from the Gresham and Elliot Social Skills Scale Revised 03/2009 15
(1990). The parent’s parenting efficacy and skills will be measured by the 10-item Kumpfer Parenting Skills scales that are taken from the Alabama Parenting test. The family conflict, organization, communication and cohesion will be measured by these four scales from the Family Environment Scales, (Moos, 1974). Most of these outcome instruments are standardized and were used by the original program developer. These instruments are discussed in greater detail below.
Parent Change Measures. The parent alcohol, and illicit drug use including age of first use and 30-day substance use rates for tobacco, alcohol, marijuana, binge drinking, and other illicit drugs will be measured using the CSAP/GPRA drug use measures from the Monitoring the Future (Johnston, O’Malley, and Bachman, 1998) and the National Household Survey (SAMHSA/OAS, 2000). Parental depression is measured by the 20-item scale on the CES-D (Radloff, 1977) included in the Strengthening Families Program Adult Parenting Questionnaire.
Child Risk Behavior Change Measures. The risk and protective factor precursors of substance abuse include negative child behaviors and lack of effective discipline methods. The children’s change outcomes will be measured by the Kellam Parent Observation of Children’s Activities (POCA), which is a modification of the Achenbach and Edelbrock (1988) Child Behavior Checklist (CBCL) to improve parent acceptability and understanding of the constructs measured. The POCA has a five-point scale and is more change sensitive than the CBCL and the wording is simpler for low education families. Six child negative outcomes are measured: child overt and covert aggression, depression, hyperactivity, concentration problems, and criminal behavior.
Child Protective Factor Behavior Changes. The parent and child version of the Social Skills Rating System (SSRS) (Gresham & Elliott, 1990) will be used for measuring social/life skills. The SSRS measures the following dimensions: Cooperation, Assertion, Responsibility, and Self-Control. In addition, it measures problem behaviors, which are classified as internalizing behaviors, externalizing behaviors, and hyperactivity. The parents completed both parent versions of the SSRS and CBCL, and the children completed the student version of the SSRS. For the main SSRS subscales, higher scores indicate more positive outcomes (e.g. more cooperation, assertion, responsibility and self-control). For the problem behavior subscales, lower scores indicate more positive outcomes (e.g. fewer internalizing, externalizing, hyperactivity problems).
Family Environment or Functioning Measures. The family change outcomes were measured by the Moos Family Environment Scale ( FES) (Moos & Moos, 1994) and the Children’s Version of the Family Environment Scale (Pino, Simons, & Slawinoski, 1983) that include scales for the level of family conflict, communication, organization and family cohesion. A separate 12 item Family Strengths and Resilience Scale is also measured (Kumpfer & Dunst, 1995). See description of each scale and source below.
Psychometric Properties. These measurement instruments and scales have been found to have high reliability and validity in prior SFP studies with similar participants. To reduce testing burden, in some cases only sub-scales of selected instruments were used for evaluation. They match the hypothesized dependent variables and were used in the construction of the testing batteries. Each of the
Revised 03/2009 16
program goals and objectives as listed above are matched to the standardized testing scale or measure in the Table below.
Table 11. SFP Hypothesized Outcomes Matched to Measures
SFP Outcome Variables Measures
Parent Immediate Change Objectives
1.
increase positive parenting 1. SFP parenting skills
2.
increase in parenting skills 2. SFP parenting skills
3.
increase parental supervision 3. SFP parenting skills
4. increase parental efficacy 4. Alabama Parenting Scale
5. increase in parental involvement 5. Alabama Parenting Scale
6. decrease in parental substance use or misuse 6. CSAP30-day use rates
7. parental depression 7. Radloff CES-D
Child Change Objectives
1.
increase social skills (cooperation, assertion, 1. Social Skills Rating Scale
responsibility, and self-control) (parent and child)
2. reduced overt aggression 3. POCA Child Rating Scale
3. reduced covert aggression 4. POCA covert aggression scale
4. reduced concentration problems (ADD) 5. POCA ADD scale
6. reduced criminal behavior 7. POCA criminal behavior scale
7. reduced hyperactivity 8. POCA hyperactivity scale
8. reduced depression 9. POCA depression scale
Family Change Objectives
1.
increase positive parent/child relationship
or family cohesion 1. Moos FES cohesion
2. reduce family conflict 2. Moos FES family conflict
3. increase family organization and order 3. Moos FES family organization
4. increase family communication skills 4. Moos FES communication
5. increased overall family strengths and resilience 5. Kumpfer & Dunst Family Strengths and Resilience scale
Data Analysis. All outcome data will be collected on the SFP parent, youth, and group leader questionnaires by the local evaluation team or the site coordinator/group leaders. After data cleaning (removing any names, assuring readable marks, checking for missing data and random markings) by the SFP outcome researchers at the University of Utah, the data will be entered into a computer for analysis on a network PC using SPSS for Windows by Dr. Keely Cofrin at the Utah State Health Department. She manages all outcome analyses for an external evaluation and has the SFP norms database of over 3,000 families for a quasi-experimental comparison group to international norms.
For this study, only the de-identified (coded) parent pre- and post-test quantitative data will be used using SPSS program.
A total change score is calculated as well as summed scores for the parent, child and family outcomes. The effect sizes of the outcomes are calculated using both Cohen’s d and the
Revised 03/2009 17
eta squared or d’ statistics for the three outcome cluster variables (parent, family and child outcomes) and 20 individual outcome variables related to parent, family, and child risk factor improvements and improved protective factors for substance abuse. Analyses of Variance (ANOVAs) and the Effect Sizes for the pre- to post-test changes are conducted and reported in outcome tables categorically by parent, family and child variables.
The State of Oklahoma SFP 6 to 11 Years Pre- to Posttest Outcomes
In Year 2 there were 7 families who completed both a pre- and a posttest for the Fall 2009 SFP groups. As can be seen from the table below, there were significant positive results for SFP 6 -11 Years for 19 of the 21 outcomes (95%) measured by parent, child and family outcome variables. The best news is that the effect sizes or amount of positive changes in the families is consistently larger than the SFP norms from other similar agencies. The only two outcomes not to be statistically significant was decreases in Hyperactivity and Criminal Behavior in the children. However, the effect sizes were larger than the norms, so with a larger sample size these outcomes could be statistically significant. The sample size as only seven parents from two groups that had both pre-and posttest data even though we had pretest data on 15 parents or twice that number.
This year, five of the five or 100% of the parenting outcomes were statistically significant and 5 of the 5 (100%) of the family outcomes. Six of eight (75%) of the children’s outcomes were significantly improved. The six significantly improved children’s outcomes are namely overt aggression (p. = .04), covert aggression (p. = .04), depression (p. = .03), social behavior (p. = .05), concentration problems or reduced ADD (p. = .02) and Child Cluster Scale (p. = .02). The two child outcomes that were not reduced significantly were hyperactivity (p. = .47) and criminal behavior (p. = 1.00) that had no reduction. However, the base rates reported by the parents was too low in such young children.
The results for the child, parent and family improvements are excellent and the effect sizes are slightly larger than normally found for SFP even in Year 2.. The parents are self-reporting higher parenting skills and a more positive family environment on all of the five the family outcomes all five of the parenting outcomes than the SFP national norms. These parents who have had open child abuse cases are not likely to be higher in parenting and family relationship skills than the other parents nationally participating in SFP. The only time we see this positive bias in the pretest results is when a regular pretest is used with parents at risk of loosing their children or those who have lost their children and the children are not living with them. In this case practice sessions to improve skills are very difficult and can result in lower changes.
In any case these 19 statistically significant outcomes are reflective of very positive improvements and the effect sizes are larger than those reported in the SFP National Database. These families are very high risk and are ending SFP with higher scores on the parenting and family scales than generally found. These statistically significant outcomes are not solely due to a large sample size because there were only 7 families in this FY ’08 –’09 analysis of the SFP 6 -11 groups. The major reason is the significant mean changes and effect sizes.
Revised 03/2009 18
The effect sizes (d) for the parent outcomes ranged from a high of d. = .85 for improvements in Parenting Efficacy, d. = .78 for Positive Parenting. All of the parenting outcomes were in the range of d = .67 for Parental Supervision to d. = .85 for Parenting Efficacy. Most of the family outcomes were larger. Family Strengths/Resilience had largest improvement with an effect size of d. = .90, followed closely by Family Communication with an effect size of d. = .81. Family Organization also improved with an effect size of d. = .76 and Family Cohesion was d. = .63.
As is generally found, the children’s outcomes were smaller as these take longer to change. Also these families often did not live with their children. Hence, it is remarkable that six of eight children’s outcomes were statistically significant outcomes. The Cohen’s d effect sizes were from a medium to small improvements by the immediate posttest (within 14 weeks). They range from rather large d = .61 for improvements in Concentration to d= .51 for Social Behavior.
Reported in the tables below are the significant level or p. value for pre to posttest changes as well as a more important statistical outcome called “effect size”. Similar to percent change, effect size is a more scientific way that researchers today report how much participants in an intervention have changed. The effect sizes reported are calculated in SPSS software by eta squared or Cohen’s d as well as d’. It can be seen that they are very large and replicate the large effect sizes found to SFP in randomized control trials (Kumpfer & DeMarsh, 1986; Spoth, et al., 1999; 2002; 2003; Trudeau & Spoth, 2005), Gottfredson, Kumpfer, et al., 2006), except they are even larger. The overall effect size in reducing alcohol and drug use of all youth-only substance abuse prevention programs is d = .10. The effect size of the DARE program was d =.08 and the best social skills training prevention programs only have an effect size of about d = .30 (Tobler & Stratton, 1997; Tobler & Kumpfer, 2000). Parenting and family interventions have larger effect sizes. See table below.
SFP 6 -11 Years Effect Sizes or Amount of Individual Change
The families reported Effect Sizes (d) at least .11 Effect Size or greater in 20 of the 21 outcome variables as shown below in the following table. Nighteen of the effect sizes are greater than d = .41 and sixteen of the effect sizes are over d = .51 or larger effect sizes (Parental Involvement, Parental Supervision, Parenting Efficacy, Positive Parenting, Parenting Skills, Parent Cluster Scale, Alcohol and Drug Use, Family Cohesion, Family Communication, Family Organization, Family Strengths/Resilience, Family and Child Cluster Scale, reduced Overt Aggression, and improved Concentration and Social Behavior in children). The effect sizes are much larger we normally find for families participating in SFP as shown in the tables below for the SFP database of other high risk parents with drug problems, some also with child protection reports. While the State of Oklahoma and the five agencies are clearly doing a very good job at recruiting the right families that are high risk and also implementing the program very well, using a regular pre-test and not doing another retrospective pre-test reduces the size of the results because an inflated base rate.
Revised 03/2009 19
Family Outcomes
As can be seen in the table below, as is typically found for SFP the second largest changes being reported are in the area of family dynamics. 100% or all five family measures were found to be statistically significant positive changes. Additionally, all of the family outcomes for these SFP groups were larger in effect size or amount of change than the SFP National Norms. In family conflict, the families were higher in reported family conflict as seen in the pre-test means of (m = 2.79 vs 2.36 for the norms).
All family environment outcomes for SFP 6 -11 groups changed from d = .45 to .90 or large effect sizes. The largest effect size was for Family Strengths and Resilience Scale (d = .90), followed by Family Communication (d. = .81). The Family Strengths and Resilience Scale measures overall emotional, behavioral, parenting, educational, and spiritual strengths of the family. Hence, this large change indicated that these Oklahoma agencies are making major improvements in these families. Family Organization was also reported to have improved (Effect Size = .76) and Family Cohesion’s effect size is .63. Improvements in Family Cohesion was statistically significant at the p= .02. These changes within 4 months are higher of the average effect sizes for the total Family Cluster Scale (mean of all five family outcomes) which as d = .67 compared to d = .84 in Oklahoma families.
The outcomes are much higher for that found in meta-analyses for evidence-based parent training programs (d. = .33) where the families are not brought together. Also, the results are higher than that found for the best long-term family therapies (d = .45), which are much more costly than SFP. Additionally, these family outcomes are smaller than those of the SFP National Norms as is shown in the table below.
These local results are bigger effects than found in other federally funded research studies conducted for National Institute of Drug Abuse (NIDA) research SFP studies (Gottfredson, Kumpfer, et al., 2005; Spoth, et al., 2003) and the Center for Substance Abuse Prevention (CSAP) (Kumpfer, Alvarado, Smith & Bellamy, 2002; Kumpfer, Alvarado, Tait & Turner, 2002). Overall Family Strengths and Resilience Effect Size d was .34, which was smaller than the national norms for SFP that was d = .61. These effect sizes are smaller for all family variables (except Family Conflict) than in the SFP National Database of all national sites submitting data on SFP groups to LutraGroup. For some reason the families are not reporting as much family conflict at intake as in prior years so they have less room for improvement.
Table 12. OK SFP 6 –11 Years Family Outcomes for Pre- to Posttest Changes
Protective Factor Sig. Level (p=) 08-09 (d) vs. Nat’l Norms (d)
1. Family Organization .01 .76 vs. .64 (large)
2. Family Cohesion .02 .63 vs. .48 (large)
3. Family Communication .001 .81 vs. .64 (large)
Revised 03/2009 20
4. Family Conflict .03 .45 vs. .18 (medium)
5. Family Resilience .001 .90 vs. .62 (large)
The following table reports the actual pretest to posttest means for the group as well as the mean changes along with the p values and two different types of effect size, d and d’. These are compared to the descriptive statistics for the SFP National Norms on about 4512 families from agencies all over the country. It can be seen that the families are reporting themselves to be higher risk at base line or pretest for all the family outcomes than families that generally participate in SFP groups.
The ANOVA comparison of the experimental to the control (norms) groups suggests that there were statistically significant differences from the control (norms) for all family outcomes.
Revised 03/2009 21
Table 13. OK SFP Mean Changes in Family Risk and Protective Factors Compared to SFP National Norms
Scale Name
# fam
Pre-Test
SD
Post-Test
SD
Change
F
sig
Effect Size d
ES d'
Family Cohesion
2.73
0.10
0.00
0.06
National Norms
4020
3.60
0.97
4.37
0.66
0.78
3659.82
0.00
0.48
1.92
Oklahoma Sites
7
3.57
1.13
4.86
0.24
1.29
10.02
0.02
0.63
2.58
Family Communication
10.36
0.00
0.00
National Normsl 4049
3.13
0.79
4.07
0.65
0.94
6940.69
0.00
0.64
Oklahoma Sites
7
2.86
1.17
4.67
0.51
1.81
25.94
0.00
0.81
4.16
Family Conflict
10.32
0.00
0.00
0.11
National Norms
3960
2.36
1.09
1.99
0.87
(0.37)
839.28
0.00
0.18
0.93
Oklahoma Sites
7
2.79
1.16
1.44
0.60
(1.35)
4.86
0.03
0.45
1.80
Family Organization
7.62
0.01
0.00
0.09
National Norms
3966
2.62
0.91
3.81
0.78
1.19
7200.21
0.00
0.64
2.70
Oklahoma Sites
40
3.36
1.14
4.16
0.81
0.80
18.48
0.01
0.76
3.51
Family Strengths/Resilience
2.53
0.11
0.00
0.06
National Norms
4026
2.62
0.92
3.81
0.78
1.18
6343.52
0.00
0.62
2.54
Oklahoma Sites
7
2.96
1.14
4.68
0.43
1.71
53.97
0.00
0.90
6.00
Family Cluster Scale
7.69
0.01
0.00
0.09
National Norms
4008
3.34
0.85
4.24
0.61
0.90
8180.43
0.00
0.67
2.84
Oklahoma Sites
7
3.05
0.65
4.70
0.31
1.65
31.29
0.00
0.84
4.57
Parenting Skills and Behaviors
The largest changes were in the area of parenting skills and behaviors. Five of the five (100%) of the parent outcomes had the significantly significance in p. values, and had large effect sizes ranging from d = .67 (Parental Supervision) to d= .85 (Parenting Efficacy).
Revised 03/2009 22
Table 14: OK SFP 6 –11 Years Parenting Outcomes for Pre- to Posttest Changes
Protective Factor Sig. Level (p=) 08-09 (d) vs. Nat’l Norms
1. Positive Parenting .001 .78 vs. .52 (large)
2. Parental Involvement .01 .68 vs. .47 (large)
3. Parenting Skills .01 .71 vs. .42 (large)
4. Parental Supervision .01 .67 vs. .39 (large)
5. Parenting Efficacy .001 .85 vs. .52 (large)
The area of Parenting Efficacy and Positive Parenting had the largest amount of positive change SFP (Effect Size d = .85 and .78). Next largest change was reported in Parenting Skills (Effect Size d = .71). Parental Involvement (Effect Size d = .68), or increases in the amount of time spent with the child(ren), and Parental Supervision (Effect Size d=.67) have the lower effective sizes compared with other parenting outcomes.
The smallest change in the parenting area was for Parental Supervision (Effect Size d = .67). This area improved, and higher than the Effect Size d = .39 in the SFP National Data Base.
Overall, these are increases in parent child management skills with Cohen d effect sizes ranging from .67 for parental supervision to .85 for parenting efficacy. Parental supervision (d= .67) is a critical risk factor for children’s later drug and alcohol use, so improvements in this area as well as communication bode well for the long-term effectiveness of this program in preventing later behavioral problems and substance use in the children.
Table 15. OK SFP Mean Changes in Parenting Risk and Protective Factors Compared to the SFP National Norms
Scale Name
# fam
Pre-Test
SD
Post-Test
SD
Change
F
Sig
Effect Size d
ES d'
Parental Involvement
1.51
0.22
0.00
0.00
National Norms
4008
3.49
0.96
4.24
0.72
0.74
3540.07
0.00
0.47
1.89
Oklahoma Sites
7
3.32
1.19
4.43
0.62
1.11
12.93
0.01
0.68
2.94
Parental Supervision
2.84
0.09
0.00
0.06
National Norms
4464
3.25
0.78
3.83
0.66
0.58
2799.31
0.00
0.39
1.60
Oklahoma Sites
7
3.34
0.93
4.40
0.37
1.06
12.30
0.01
0.67
2.86
Revised 03/2009 23
Parenting Efficacy
12.60
0.00
0.00
0.11
National Norms
4037
3.26
0.88
4.07
0.69
0.81
4239.99
0.00
0.52
2.07
Oklahoma Sites
7
2.81
0.84
4.67
0.38
1.86
32.83
0.00
0.85
4.67
Positive Parenting
5.62
0.02
0.00
0.06
National Norms
4031
3.78
0.91
4.55
0.57
0.76
4274.02
0.00
0.52
2.08
Oklahoma Sites
7
3.48
0.94
4.90
0.16
1.43
20.61
0.00
0.78
3.71
SFP Parenting Skills
2.29
0.13
0.00
0.06
National Norms
4031
3.36
0.72
3.83
0.67
0.47
2857.55
0.00
0.42
1.70
Oklahoma Sites
7
3.93
0.51
4.71
0.25
0.79
14.34
0.01
0.71
3.09
Parent Cluster Scale
5.37
0.02
0.00
0.06
National Norms
4485
3.45
0.66
4.00
0.59
0.55
2927.55
0.00
0.40
1.64
Oklahoma Sites
7
3.43
0.73
4.60
0.28
1.17
22.10
0.00
0.79
3.83
Parent Substance Abuse
Reported alcohol and drug use by the parents is reasonably low at the intake at mean score of 1.81 for parents (just below 2.00 of “some use”) at pre-test and decreased to no use or 1.00 by the posttest. The reduction in use is statistically significant at p. = .02 for the parents. So this is a good result for a program that does not directly target drug and alcohol use in the parents. Possibly to other recovery services provided by these agencies are contributing to the self-reported significant decrease in substance use in the parents by the posttest 14 to 16 weeks later. An effect size of d =. 64 is a large effect size, and much larger than the d =. 18 for the norms. With 7 families power was large enough to detect a significant decrease in substance use.
These improvements are much better than the SFP National Norms (d=. 03) showing significant improvement in the parent’s substance use.
Table 16. OK SFP Changes in Parent Alcohol and Drug Use
Scale Name
# fam
Pre-Test
SD
Post-Test
SD
Change
F
sig
Effect Size d
ES d'
Alcohol & Drug Use
26.89
0.00
0.01
0.17
National Norms
3928
1.41
0.60
1.34
0.57
(0.07)
118.97
0.00
0.03
0.35
Oklahoma Sites
7
1.81
0.66
1.00
0.00
(0.81)
10.67
0.02
0.64
2.67 Revised 03/2009 24
Children’s Behavioral and Emotional Improvements
As can be seen above, the first outcomes to improve are the family environment measured outcomes with the larger effect sizes, next are the parenting outcomes with large effect sizes. The most difficult to change within a 4 month time period are the children’s behaviors, which are generally small to medium in size. Larger improvements are found at the 6-month and 12-month follow-ups assessments.
This year, if we conducted a one-tail test for reductions in child Depression and Covert Aggression that should be done when a hypothesis predicts a direction of change, the p values were statistically significant with a two-tailed test. This year six of eight (75%) of the children’s outcomes were significantly improved. The six significantly improved children’s outcomes are namely overt aggression (p. = .04), covert aggression (p. = .04), depression (p. = .03), social behavior (p. = .05), concentration problems or reduced ADD (p. = .02) and Child Cluster Scale (p. = .02). The two child outcomes that were not reduced significantly were hyperactivity (p. = .47) and criminal behavior (p. = 1.00) that had no reduction. However, the base rates reported by the parents was too low in such young children.
These changes generally occur later with the 6 and 12-month follow-up tests. Most studies of SFP find increased positive results with time in the children rather than diminished results (Kumpfer, et al, 2002). Spoth and his associates have recently reported 2 to 3 times reductions in lifetime diagnoses of any type of mental health problem (depression, anxiety disorder, social phobias, and even personality disorder) in 22 year old youth who had participating in SFP 10-14 ten years earlier (Trudeau & Spoth, 2005; Spoth & Trudeau, 2005). This possibly makes SFP the most effective mental health initiative that any state or county could implement. These results also suggests that SFP results are not specific to just major reductions in tobacco, alcohol and drug abuse, but also in mental health and juvenile delinquency services costs.
In this preliminary analysis of the data, we only have the first 4 months of data. Regardless of these caveats, the data suggest significant positive changes in five of the youth change variables. The sites are collecting 6- and 12-month follow-up that will be analyzed separately when there are enough tests.
Table 17. OK SFP 6 –11 Years Child Outcomes for Pre- to Posttest Changes
Protective Factor Sig. Level (p=) 08-09 (d) vs. Nat’l Norms
1. Overt Aggression. .04 .52 vs. .28 (large)
2. Covert Aggression .045 .41 vs. .15 (medium)
3. Concentration Problems .02 .61 vs. .46 (large)
Revised 03/2009 25
4. Criminal Behavior 1.00 .00 vs. .01 (small)
5. Hyperactivity .47 .11 vs. .00* (small)
6. Social Behavior .05 .51 vs. .28 (large)
7. Depression .03 .48 vs. .22 (medium)
The table below shows all of the statistical outcomes for the children’s changes for SFP 6-11 compared to the National Norms for SFP in over 4512 families from all over the country. The effect sizes for the statistically significant outcomes ranged from medium d = .41 for Covert Aggression if we did the one-tail test to large d = .61 for improvements in Concentration in the children.
Table 18. OK SFP Means, SDs, Changes, F and P values, d and d’ in Children’s Risk and Protective Factors
Scale Name
# fam
Pre-Test
SD
Post-Test
SD
Change
F
sig
Effect Size d
ES d'
Concentration
11.34
0.00
0.00
0.11
National Norms
4112
3.25
0.71
4.12
0.52
0.86
3381.45
0.00
0.46
1.84
Oklahoma Sites
7
3.04
0.87
4.68
0.27
1.64
9.53
0.02
0.61
2.52
Covert Aggression
0.28
0.60
0.00
0.00
National Norms
3975
2.04
0.64
1.82
0.55
(0.22)
664.07
0.00
0.15
0.82
Oklahoma Sites
7
1.88
0.61
1.55
0.64
(0.33)
4.16
0.04
0.41
1.66
Criminal Behavior
0.04
0.84
0.00
0.00
National Norms
3881
1.11
0.40
1.09
0.37
(0.03)
26.53
0.00
0.01
0.17
Oklahoma Sites
6
1.42
1.02
1.42
1.02
0.00
.
.
.
######
Depression
2.58
0.11
0.00
0.06
National Norms
4072
2.13
0.74
1.84
0.62
(0.30)
1101.36
0.00
0.22
1.05
Oklahoma Sites
7
2.32
0.53
1.68
0.70
(0.64)
5.59
0.03
0.48
1.93
Hyperactivity
0.16
0.69
0.00
0.00
National Norms
3930
2.73
0.90
2.75
0.89
0.02
3.81
0.05
0.00
0.06 Revised 03/2009 26
Oklahoma Sites
6
2.72
0.25
2.83
0.46
0.11
0.63
0.47
0.11
0.71
Overt Aggression
0.61
0.43
0.00
0.00
National Norms
4094
2.16
0.74
1.81
0.58
(0.35)
1572.39
0.00
0.28
1.25
Oklahoma Sites
7
1.77
0.66
1.25
0.33
(0.51)
6.60
0.04
0.52
2.10
Social Behavior
2.32
0.13
0.00
0.06
National Norms
4418
3.82
0.70
4.11
0.66
0.30
1683.56
0.00
0.28
1.25
Oklahoma Sites
7
3.67
0.76
4.24
0.70
0.58
6.20
0.05
0.51
2.03
Child Cluster Scale
0.63
0.43
0.00
0.00
National Norms
4512
3.68
0.52
4.03
0.51
0.35
2874.53
0.00
0.39
1.62
Oklahoma Sites
7
3.92
0.53
4.40
0.38
0.48
8.99
0.02
0.60
2.45
Overt Aggression. The hardest child outcome variables to change by the post-test are Criminal Behavior and Hyperactivity. However, the Overt Aggression variable is also generally found to difficult to change and sometimes does not improve significantly by the posttest. However, in State of Oklahoma children, overt aggression improvements were significantly reduced (p. = .04) with a large effect size of .52. The effect size was twice as large as SFP National Norms (d = .52 vs. .28). This are excellent outcomes even if the children are not living with the parents as it will likely lead to a better relationship between the child and the caretakers whether parents, relatives or foster parents that could result in less family conflict and abusive interactions in the future. Additionally, overt aggression is a cause of failed placements in the child welfare system that can lead to multiple placements. Multiple placements are a predictor of later involvement in the juvenile justice system and substance abuse (Chamberlain, et al., 2002).
In the Washington D.C. randomized control trial study (Gottfredson, Kumpfer, et al., 2005) overt aggression did not have a statistically significant improvement. The effect size is also moderate in the SFP National Database (d =. 29). With a large effect size, this amount of positive change represents an impressive 4-month posttest outcome for just a 14-session parenting and family program.
Covert Aggression. Positive outcomes for Covert Aggression were statistically significant at the p= .09 level. However, if we conducted the one-tail test, the outcomes for Covert Aggression were statistically significant at the p= .045 level. Generally girls are more likely to engage in covert aggression (stealing, lying, gossiping, whispering, eye rolling, character assignation) than boys. The effectiveness of the SFP for covert aggression was effect size of d = .41 compared to nationally norm of .15. The effect size was .41 in this year. When we get enough data we will conduct a gender analysis to see if covert aggression is higher in girls and whether SFP is as successful in reducing covert aggression as overt aggression in girls and boys separately.
Revised 03/2009 27
Improved Concentration or Reduced Attention Deficit. The effect size for reductions in attention deficit or problems in concentration in the children is the highest of all seven of the child behavioral measures compared with other child outcomes. The effect size this year (2008 to 2009) for SFP is d = .61. This compares higher than the d = .46 found for the national norms. For some reason the parents are reporting their children to have fewer attention or concentration problems at the baseline intake which makes it harder to get large effect sizes (m=3.04 OK sites vs. m=3.25 SFP national norms). Larger standard deviations reduce effect sizes.
A major complaint of parents is that children today do not focus and pay attention. This change in the children’s ability to concentrate, at least in the view of the parents, is positive. Inability to concentrate causes children to have school academic problems, which is a major risk factor for later association with antisocial peers and drug use (Kumpfer, Alvarado, & Whiteside, 2003).
Criminal Behavior. Antisocial criminal behavior was reported by parents to be very low at a mean of only 1.42 or basically none for the children at the pretest resulting in the same to 1.42 by the posttest. Because of a floor effect there was very little room for improvement. This is not surprising because the children in SFP 6-11 Years are very young and not likely to already be arrested. The effect size of d = .00 was small and the results were not statistically significant (p. = 1.00). If the rate of criminal behavior is so low, it is hard to make it much lower. The non-significant increase is so small it could be a reporting error.
Child Hyperactivity. Child Hyperactivity was reported to be similar at baseline or intake for the children (mean 2.72) than the national average (mean 2.73). Moreover, hyperactivity did not increase significantly (p. = .47). In the SFP national database, hyperactivity also is not decreasing much (-.02). The SFP National Database does not generally find significant improvements in Hyperactivity in the children (Effect Size =.05). We have conducted a study within this national database and found that group leaders who are warmer and well liked tend to promote better changes in the clients, except for increasing the children’s hyperactivity and the parent’s depression (Park & Kumpfer, 2005).
Social Behavior. Social Behavior improved with large changes in the effect sizes of the youth’s Social Skills and Competencies (d = .51) compared to d = .28 for the SFP norms. This is larger than the effect sizes for the best social skills training programs at d = .25 for all life or social skills training programs included in the Tobler meta-analysis study. Also, these results are much better than the national SFP norms of d = .28. SFP includes a 14 session children social skills curriculum based on the best evidence based social skills models, such as Shure and Spivack’s I can Problem Solve Program. It includes sessions on problem solving, decision making, communication skills, coping with anger and depression, and even dating relationships in the older adolescent version of SFP 12 – 16 Years (Kumpfer & Whiteside, 2006). Children’s Depression. There was a statistically significant decrease in depression if we conducted the one-tail test (p. = .03). The effect sizes for Year 2 were medium size (d = .48) compared to a d of .22 in National Norms. The medium size for Oklahoma families is more than twice as for SFP National Norms. SFP includes a 14- session children social skills curriculum based on the best evidence based social skills models, such as Shure and Spivack’s I can
Revised 03/2009 28
Problem Solve Program. It includes sessions on communication skills and coping with anger and depression. In addition, the improvements in the way the parents are treating their children with less corporal punishment and more attention for wanted behaviors can contribute to reduced depression. Children whose parents begin the recovery process also have a reduction in depression because they become hopeful of a better family life and relationship with their parent. Randomized control trials suggest there is a long term impact on reducing children’s lifetime rates of diagnosed depression by 280% by age 22 years of SFP improving family relationships (Spoth, et al., 2007). Depression and anxiety have been found in research to be highly correlated to substance abuse including genetic studies of identical and fraternal twins, suggesting a genetic basis for both depression and substance abuse. Hence, reducing the children’s depression might later reduce their use of drugs and alcohol that perpetuate the inner generational transmission of family violence and neglect.
Overall Strengthening Families Program Results for Year Two (FY ’08 to ’09) Compared to SFP Norms
The following Table 21 reports on the total data tables for the SFP program for 2008 to 2009 participants (n = 7 families). Table 19 also includes comparison of this agencies data to that of the national database of all participant families that has send data to LutraGroup (n = 4512 families). This analysis included the effect sizes calculated by both the d’ prime and Cohen’s d as calculated by eta squared. The statistical significance values are to pre-to posttest ANOVA within-S analyses.
These are the raw results reported on below in Table 19. They suggest very good outcomes that are better for reducing parent Alcohol and Drug use, increasing Parental Supervision, Parental Involvement, Parenting Efficacy, and reducing Family Conflict and also children’s Covert and Overt Aggression than the average mean changes found for almost 4400 families in the Strengthening Families Program National Database.
Table 19. State of Oklahoma SFP Compared to SFP National Norms for All 21 Outcome Variables (Pre- to Posttest Means, SDs, Change Scores, Fs, p-values, and Effect Sizes for All Outcome Variables
Strengthening Family Program Evaluation Project
Oklahoma Sites
Friday, April 09, 2010
Scale Name
Sample
Pre-Test
SD
Post-Test
SD
Change
F
sig
Effect Size d
ES d'
Parental Involvement
1.51
0.22
0.00
0.00
National Norms
4008
3.49
0.96
4.24
0.72
0.74
3540.07
0.00
0.47
1.89
Oklahoma Sites
7
3.32
1.19
4.43
0.62
1.11
12.93
0.01
0.68
2.94
Parental Supervision
2.84
0.09
0.00
0.06
National Norms
4464
3.25
0.78
3.83
0.66
0.58
2799.31
0.00
0.39
1.60 Revised 03/2009 29
Oklahoma Sites
7
3.34
0.93
4.40
0.37
1.06
12.30
0.01
0.67
2.86
Parenting Efficacy
12.60
0.00
0.00
0.11
National Norms
4037
3.26
0.88
4.07
0.69
0.81
4239.99
0.00
0.52
2.07
Oklahoma Sites
7
2.81
0.84
4.67
0.38
1.86
32.83
0.00
0.85
4.67
Positive Parenting
5.62
0.02
0.00
0.06
National Norms
4031
3.78
0.91
4.55
0.57
0.76
4274.02
0.00
0.52
2.08
Oklahoma Sites
7
3.48
0.94
4.90
0.16
1.43
20.61
0.00
0.78
3.71
SFP Parenting Skills
2.29
0.13
0.00
0.06
National Norms
4031
3.36
0.72
3.83
0.67
0.47
2857.55
0.00
0.42
1.70
Oklahoma Sites
7
3.93
0.51
4.71
0.25
0.79
14.34
0.01
0.71
3.09
Parent Cluster Scale
5.37
0.02
0.00
0.06
National Norms
4485
3.45
0.66
4.00
0.59
0.55
2927.55
0.00
0.40
1.64
Oklahoma Sites
7
3.43
0.73
4.60
0.28
1.17
22.10
0.00
0.79
3.83
Family Cohesion
2.73
0.10
0.00
0.06
National Norms
4020
3.60
0.97
4.37
0.66
0.78
3659.82
0.00
0.48
1.92
Oklahoma Sites
7
3.57
1.13
4.86
0.24
1.29
10.02
0.02
0.63
2.58
Family Communication
10.36
0.00
0.00
0.11
National Norms
4049
3.13
0.79
4.07
0.65
0.94
6940.69
0.00
0.64
2.64
Oklahoma Sites
7
2.86
1.17
4.67
0.51
1.81
25.94
0.00
0.81
4.16
Family Conflict
10.32
0.00
0.00
0.11
National Norms
3960
2.36
1.09
1.99
0.87
(0.37)
839.28
0.00
0.18
0.93
Oklahoma Sites
7
2.79
1.16
1.44
0.60
(1.35)
4.86
0.03
0.45
1.80
Family Organization
7.62
0.01
0.00
0.09
National Norms
3966
2.62
0.91
3.81
0.78
1.19
7200.21
0.00
0.64
2.70
Oklahoma Sites
40
3.36
1.14
4.16
0.81
0.80
18.48
0.01
0.76
3.51
Family Strengths/Resilience
2.53
0.11
0.00
0.06
National Norms
4026
2.62
0.92
3.81
0.78
1.18
6343.52
0.00
0.62
2.54
Oklahoma Sites
7
2.96
1.14
4.68
0.43
1.71
53.97
0.00
0.90
6.00
Family Cluster Scale
7.69
0.01
0.00
0.09
National Norms
4008
3.34
0.85
4.24
0.61
0.90
8180.43
0.00
0.67
2.84 Revised 03/2009 30
Oklahoma Sites
7
3.05
0.65
4.70
0.31
1.65
31.29
0.00
0.84
4.57
Concentration
11.34
0.00
0.00
0.11
National Norms
4112
3.25
0.71
4.12
0.52
0.86
3381.45
0.00
0.46
1.84
Oklahoma Sites
7
3.04
0.87
4.68
0.27
1.64
9.53
0.02
0.61
2.52
Covert Aggression
0.28
0.60
0.00
0.00
National Norms
3975
2.04
0.64
1.82
0.55
(0.22)
664.07
0.00
0.15
0.82
Oklahoma Sites
7
1.88
0.61
1.55
0.64
(0.33)
4.16
0.04
0.41
1.66
Criminal Behavior
0.04
0.84
0.00
0.00
National Norms
3881
1.11
0.40
1.09
0.37
(0.03)
26.53
0.00
0.01
0.17
Oklahoma Sites
6
1.42
1.02
1.42
1.02
0.00
.00
1.00.
0.00
0.00
Depression
2.58
0.11
0.00
0.06
National Norms
4072
2.13
0.74
1.84
0.62
(0.30)
1101.36
0.00
0.22
1.05
Oklahoma Sites
7
2.32
0.53
1.68
0.70
(0.64)
5.59
0.03
0.48
1.93
Hyperactivity
0.16
0.69
0.00
0.00
National Norms
3930
2.73
0.90
2.75
0.89
0.02
3.81
0.05
0.00
0.06
Oklahoma Sites
6
2.72
0.25
2.83
0.46
0.11
0.63
0.47
0.11
0.71
Overt Aggression
0.61
0.43
0.00
0.00
National Norms
4094
2.16
0.74
1.81
0.58
(0.35)
1572.39
0.00
0.28
1.25
Oklahoma Sites
7
1.77
0.66
1.25
0.33
(0.51)
6.60
0.04
0.52
2.10
Social Behavior
2.32
0.13
0.00
0.06
National Norms
4418
3.82
0.70
4.11
0.66
0.30
1683.56
0.00
0.28
1.25
Oklahoma Sites
7
3.67
0.76
4.24
0.70
0.58
6.20
0.05
0.51
2.03
Child Cluster Scale
0.63
0.43
0.00
0.00
National Norms
4512
3.68
0.52
4.03
0.51
0.35
2874.53
0.00
0.39
1.62
Oklahoma Sites
7
3.92
0.53
4.40
0.38
0.48
8.99
0.02
0.60
2.45
Alcohol & Drug Use
26.89
0.00
0.01
0.17
National Norms
3928
1.41
0.60
1.34
0.57
(0.07)
118.97
0.00
0.03
0.35
Oklahoma Sites
7
1.81
0.66
1.00
0.00
(0.81)
10.67
0.02
0.64
2.67
Revised 03/2009 31
V. SUMMARY, CONCLUSION AND RECOMMENDATIONS
The State of Oklahoma has implemented a an evidence-based parenting program in one agency in Tulsa, Oklahoma to improve parenting, improve family functioning and prevent substance abuse and juvenile delinquency in families involved with child protective services and with a parental history of substance abuse. This has been affected through one agency, SOS, utilizing the Strengthening Families Program (6-11) and Strengthening Families Program (3-5). The State of Oklahoma has delivered an evidence-based model program that accommodates the communities’ assessed cultural and risk-based needs in an effort to reduce child abuse or maltreatment, improve reintegration rates, or prevent later removal of the child. This has included delivery of the SFP curriculums to youth and their parents at one substance abuse treatment agency conducting a total of 3 cycles in the two-year reporting period of cycles conducted from January 2008 through June 2009, at two-year reporting period. Further, this has been done with excellent outcome results with large effect sizes that exceeded the national norms.
This evaluation has reported on findings from Year 01 and Year 02. We are just waiting for the post tests for the first half of Year 3 of the current five-year funding grant from the Administration for Children and Families in Washington, D.C. to the State of Oklahoma. In Year 02, the SFP initiative delivered three cycles of SFP. This family-based strategy targets families with children age 3-12 with children at risk for child abuse. Risk factors include parental child protective involvement and substance abuse history. This evaluation has utilized a combined methods design to measure program fidelity and program effect. An independent evaluation is being conducted by the University of Kansas with a subcontract to LutraGroup, the implementation and outcome evaluation center for SFP. The evaluation did not include a process evaluation to assess fidelity in the first two years but that has been added for Year 3 along with an outcome evaluation to measure effectiveness. The major goal of this evaluation is to determine if the program is still working and effective when implemented with less research controls and determine the effectiveness of the program at provided by one of State of Oklahoma child welfare contracted agencies to families in their catchment area in relation to the reported levels of fidelity based on SFP identified best practices (Appendix 1). This report includes the outcome evaluation conducted by an outside contractor to measure the effectiveness of SFP. This Year 3 we have begun an extensive process evaluation that includes a fidelity survey of funded cycles and a site visit to assess program fidelity. The outcome evaluation involves a repeated measures retrospective pre- and post-test design with standardized instruments being administered to parents attending the program.
Program Effectiveness and Outcomes Summary
The experimental outcome evaluation design consisted of a quasi-experimental repeated measures, pre- and post-test design with post-hoc subgroup comparisons as recommended by Campbell & Stanley (1967) to control for most threats to internal and external sources of validity. The “SFP Retrospective Parent Pre/posttest” that uses standardized Center for Substance Abuse (CSAP) and National Institute on Drug Abuse (NIDA) core measures was developed and used because of the need for a short, non-research quality, practitioner friendly evaluation instrument (Appendix 3). This instrument was selected for the ACF grant with the
Revised 03/2009 32
State of Oklahoma evaluation for child abuse prevention. Instruments administered by the agency cycles site staff. These instruments are designed to assess child and parent mental health, substance abuse risk and resiliencies, family management and cohesiveness, and parent and child social skills and attitudes. The data were recorded by the parents on printed questionnaires. These data on the pre and post-tests were hand-entered and analyzed using SPSS by Dr. Keely Cofrin using standardized scales for 21 outcome variables. The sample size for the two-year period is a total of 15 families with pretests but only 7 with posttests. These findings were compared with the SFP national database with over 4400 families.
This comparison group was the norms for the SFP National Database of 4400 families. There are two major reasons for the positive outcomes for this agency: 1) the implementation of the program in terms of quality of delivery must be better than other SFP sites nationally and 2) the families were higher risk but reported fewer problems at baseline entry into the program. The reason for this is because the evaluation design required using a regular pre-test rather than a retrospective pre-test done at posttest also.
In Year 2 with a sample size of 7 families, as can be seen from the table above there were significant positive results for SFP 6 -11 Years for 19 of the 21 outcomes (91%) measured by parent, child and family outcome variables.
This year, all five of the five or 100% of the parenting outcomes were statistically significant and 5 of the 5 (100%) of the family outcomes which is the same as last year. Six of eight (75%) of the children’s outcomes were significantly improved. The six significantly improved outcomes are namely overt and covert aggression (p. < .04), depression (p. < .03), social behavior (p. < .05, concentration problems or reduced ADD (p. < .02) and Child Cluster Scale (p. < .02). The two child outcomes that were not reduced significantly were hyperactivity (p. = .47), and criminal behavior (p. < 1.00) that had no reduction, but the base rates reported by the parents was too low in such young children.
The results for the parent and family improvements are excellent given the children have been removed and are not living with the parents. The results are twice as good as Year 1 more equivalent to the SFP Norms. These results are for the retrospective pre and posttests outcomes, which are typically larger in effect sizes than regular pre- and posttests. Many of the sites also collected regular pre-and posttests, but these data have not been analyzed yet because of client ID matching issues in the database. Retrospective tests do not suffer from this matching issue.
The Oklahoma outcomes are larger than the SFP norms in effect sizes. There are many reasons for this to occur with this population of drug-abusing parents or those who have open child protection cases. The parents are self-reporting lower parenting skills and a less positive family environment on all of the five the family outcomes all five of the parenting outcomes than the SFP national norms. Their Family Conflict scale pretest scores are considerably higher than the norms. Hence, these parents who have had open child abuse cases appear to be answering more honestly than we typically see for similar types of parents participating in SFP. Often we get a “positive response bias” in the pretest results when a regular pretest is used. This is because parents fear any identifiers on the test to match their pretest to their posttest so they do not believe their results will be kept confidential. They then are concerned about disclosing any
Revised 03/2009 33
problems in their parenting or children for fear of loosing their children. Those parents who have lost their children also tend to have a positive response bias on the pretest for similar reason, but also the amount of positive posttest change can be diminished because the children are not living with them. In this case practice sessions to improve skills are very difficult and can result in lower changes.
In any case these 19 statistically significant outcomes are reflective of very positive improvements and the effect sizes are larger than those reported since the baseline than the national SFP norms. These families are very high risk and are ending SFP with higher scores on the parenting and family scales than generally found. These statistically significant outcomes are not solely due to a large sample size because there were only 7 families in this FY ’08 –’09 analysis of the SFP groups. The major reason is the significant mean changes and effect sizes.
100% or five of five family measures were found to be statistically significant positive changes. The next largest changes were in the area of parenting skills and behaviors. Five of five (100%) of the parent outcomes had the significantly significance in p value, with large effect sizes ranging from the largest d of .67 for Parenting Efficacy to the smallest d of .67 for Parenting Supervision.. Reported alcohol and drug use by the parents is reasonably low at the intake at mean score of 1.81 for parents (just below 2.00 of “some use”) at pre-test and decreased to 1.00 or no use by the posttest. The reduction in use is statistically significant at p. < .02 for the parents. The effect size is large at d = .67 compared to a small improvement of d = .03 in the parents in the SFP norms, so this SOS drug treatment agency is doing an excellent job in reducing the parents alcohol and drug abuse. So this is a good result for a program that does not directly target drug and alcohol use in the parents.
Five of the seven child outcomes (all except Child Criminality and Hyperactivity) were statistically significant and averaged the same mean pre-to posttest changes as the SFP norms. The areas or outcomes with significant improvements were Overt and Covert Aggression, Concentration Problems, Social Behavior, and Depression.
These are the results reported on above and suggest very good outcomes that are better for reducing parent Alcohol and Drug use, improving Parent Involvement, Parent Efficacy, and reducing Family Conflict and also children’s Covert Aggression than the average results found for almost 4400 families in the Strengthening Families Program National Database.
Recommendations
It is recommended that this program continue as it is being delivered presently with the following recommendations for maintaining and improving program fidelity, effectiveness and evaluation:
Revised 03/2009 34
•
Increase Numbers of Families or Sites. Because of the low numbers of families participating in this evaluation, it is important to increase these numbers per year to increase the power of the analysis.
•
Increase the Collection of Posttests. Less than half of the families with pretests completed posttests. The agency has to be more diligent in assuring that they have collected a posttest for all the families who participated in a pretest even if they do not complete the program. This way we can also examine the benefit of participation in even fewer sessions than the required 14 sessions.
•
Evaluation Design. It is recommended that the evaluation design be continued in Year 03. The combined pretest and retrospective posttest should be administered. Site visits that started in Year 3 should be continued to all agencies and the Site Information should continue to be submitted.
•
Longitudinal Outcome Study. Efforts should be intensified to collect the follow-up posttest at six-month and one-year after completion of the program. Results will be included in this annual report when there is sufficient sample size available to provide adequate statistical power for the analysis.
•
Retention and Recruitment. There were substantial increases in capacity and completion rates from Year 01 and Year 02. It is anticipated that Year 03 will experience additional capacity and higher completion rates. The agency needs to improve their low retention rate and should be provided with technical assistance early in Year 03 to strategize and implement actions to increase their numbers. More families should be recruited with a target enrolling 8-12 families per cycle and retaining a minimum of 6 families per cycle in Year 03 cycles. Recruitment efforts should continue and be aggressive. Strategies and innovations should be documented and disseminated for the benefit of others facing similar challenges. This agency could benefit from reaching out to other sites in the similar Kansas ACF grant to find out what strategies are successful at other sites and adapt them for their community.
•
Sustainability. Efforts are already underway to sustain the program at the end of the 5-year grant funding. This topic should be an agenda item on monthly calls. Staff and managers should be provided technical assistance and workshops to help them “think outside the box” beyond the current research protocols and restrictions to an community implementation that will leverage/unify resources, build partnerships and identify alternative community agencies and funding sources that might continue and/or replicate SFP for the community as a whole, including the high-risk families currently being served by this project.
•
Dissemination of Results. Dissemination of program findings should be done both at the professional level (conferences and publications), community level and to the stakeholders that are referring to the program and experiencing the benefits of the families’ success (e.g., courts, child protective workers, elected officials, schools, foster parents). For the evaluators and project directors, publications and presentations at professional conferences should be sought to present the aggregate findings and additional project findings that can contribute and extend best practices in child abuse,
Revised 03/2009 35
child neglect and substance abuse prevention. For example, dissemination of strategies for building capacity, parent-child relations in out-of-home placement families, benefits of using telephonic supervision across sites are among the topics to be considered.
FINAL COMMENT
Despite the low numbers of parenting participants in the SFP the State of Oklahoma has implemented an exceptional Strengthening Families Program for drug-involved families as evidenced by their exceptional outcomes from the small sample of families over the two-year project period. During this time 3 cycles have been conducted by seven sites. Notable is that the mean family size approaches 3 persons per family. Thus with 6 families, this would equate to 24 participants which is more reflective of the programs impact. The challenges faced in the first year of implementation have been expected and the collaboration between the agencies, State of Oklahoma social services and the evaluation team have successfully overcome by the end of Year 2 by having effective aggressive strategies to rise to the challenge of working with this particularly high-risk population of families involved with child protective services and at-risk for termination of parental rights. The rest of Year 03 will provide for a testing of the model, study of the effect on fidelity and outcomes and documentation and reporting of the results. As they continue to implement and adapt for high-risk families, it will be important that they work with the evaluators to measure and document the strategies and the effectiveness of the adaptations in the coming years of this project. The Oklahoma SFP is providing a valuable contribution to the field to establish best practices extending evidence-based practice from substance abuse prevention to the prevention of child abuse and neglect.
Revised 03/2009 36
REFERENCES AND BIBLIOGRAPHY
Aktan, G. (1995). Organizational framework for a substance use prevention program. International Journal of Addictions 30: 185-201.
Aktan, G., Kumpfer, K. L., & Turner, C. (1996). The Safe Haven program: Effectiveness of a family skills training program for substance abuse prevention with inner city African American families. International Journal of the Addictions. 31, 158-175.
Alvarado, R. & Kumpfer, K.L. (2000). Strengthening America=s families. Juvenile Justice, 7 (2), 8-18.
Biglan, T., Mrazek, P.J., Carnine, D., & Flay, B.R. (in press). The integration of research and practice in the prevention of youth problems. American Psychologist.
Bry, B. H., Catalano, R. F., Kumpfer, K. L., Lochman, J. E. & Szapocznik, J. (1998). Scientific findings from family prevention intervention research. In Ashery, Robertson, & Kumpfer (Eds.) Family focused prevention of drug abuse: Research and interventions. NIDA Research Monograph, Washington, DC: Superintendent of Documents, US Government printing office, 103-129.
DeMarsh, J. K., & Kumpfer, K. L. (1985). Family environmental and genetic influences on children's future chemical dependency. Journal of Children in Contemporary Society: Advances in Theory and Applied Research, 18 (1/2), 117-152.
Harrison, S., Proskauer, S., & Kumpfer, K. L. (1995). Final evaluation report on Utah CSAP/CYAP project. Submitted to the Utah State Division of Substance Abuse. Social Research Institute, University of Utah.
Harrison, S., Boyle, S.W., & Farley, O.W. (1999). Evaluating the outcomes of a family-based intervention for troubled children: A pretest-posttest study. Research on Social Work Practice, 9 (6), 640-655.
Kumpfer, K.L. (1991). How to get hard-to-reach parents involved in parenting programs. In Pines, D., Crute, D., & Rogers, E. (Eds.), Parenting as prevention: Preventing alcohol and other drug abuse problems in the family (pp.87-95). Rockville, MD: Office of Substance Abuse Prevention Monograph.
Kumpfer, K.L. (2000). Strengthening family involvement in school substance abuse programs. In W.B. Hansen, S.M.Giles, & M.D. Fearnow-Kenney (Eds.). Improving Prevention Effectiveness, (pp. 127-140), Tanglewood Research, Inc., Greensboro, North Carolina.
Kumpfer, K.L. (1999). Factors and processes contributing to resilience: The resilience framework. In M.D. Glantz and J.L. Johnson (Eds.) Resilience and Development: Positive Life Adaptions, 179-224. New York: Kluwer Academic/Plenum Publishers.
Revised 03/2009 37
Kumpfer, K.L. (1998). The Strengthening Families Program. In R.S. Ashery, E. Robertson, & K.L. Kumpfer (Eds.) (1998). Drug Abuse Prevention Through Family Interventions, NIDA Research Monograph #177, DHHS, National Institute on Drug Abuse, Rockville, MD, NIH Publication No. 97-4135.
Kumpfer, K.L, & Alvarado, R. (in press). Family interventions for the prevention of drug abuse. American Psychologist, (special issue on prevention). Editors: Weissberg, R., and Kumpfer, K.L.
Kumpfer, K.L., & Alvarado, R (1998). Effective Family Strengthening Interventions. Juvenile Justice Bulletin, Family Strengthening Series. Washington, DC: U.S. Department of Justice, Office of Justice Programs, Office of Juvenile Justice and Delinquency Prevention (OJJDP). November, 1998.
Kumpfer, K.L. & DeMarsh, J.P. (1983). Strengthening families program: Parent training curriculum manual. (Prevention Services to Children of Substance-abusing Parents). Social Research Institute, Graduate School of Social Work, University of Utah
Kumpfer, K. L., & DeMarsh, J. P. (1985). Prevention of chemical dependency in children of alcohol and drug abusers. NIDA Notes, 5, 2-3.
Kumpfer, K. L., & Kaftarian, S. J. (2000). Bridging the gap between family-focused research and substance abuse prevention practice: Preface. Journal of Primary Prevention, 21(2), 169-183.
Kumpfer, K. L.; DeMarsh, J. P.; & Child, W. (1989). Strengthening families program: Children's skills training curriculum manual, parent training manual, children's skill training manual, and family skills training manual (Prevention Services to Children of Substance-abusing Parents). Social Research Institute, Graduate School of Social Work, University of Utah.
Kumpfer, K.L., & Turner, C.W. (1990-1991). The social ecology model of adolescent substance abuse: Implications for prevention. The International Journal of the Addictions, 25(4A), 435-463.
Kumpfer, K.L., Molgaard, V., & Spoth, R. (1996). The Strengthening Families Program for prevention of delinquency and drug use in special populations. In R. DeV Peters, & R. J. McMahon, (Eds.) Childhood Disorders, Substance Abuse, and Delinquency: Prevention and Early Intervention Approaches. Newbury Park, CA: Sage Publications.
Kumpfer, K. L., Alvarado, R., Smith, P., & Bellamy, N. (2002). Cultural sensitivity in family-based prevention interventions. In K. Kavanaugh, R. Spoth, & T. Dishion (Special Edition Eds.), Prevention Science, New York, Klewer Academic/Plenum Publishers,
Kumpfer, K.L., Alvarado, R., Tait, C., & Turner, C. (2002). Effectiveness of school-based family and children’s skills training for substance abuse prevention among 6-8 year old rural children, Psychology of Addictive Behavior (Special Issue), Editors:, R. Tarter, P.Tolan, & S. Sambrano.
Revised 03/2009 38
Molgaard, V., Kumpfer, K. L. & Spoth, R. (1994). The Iowa strengthening families program for pre and early teens. Ames, IA: Iowa State University.
Spoth, R. & Molgaard, V. (1999). Project Family: A partnership integrating research with the practice of promoting family and youth competencies. In T.R. Chibucos & R. Lerner (Eds.). Serving children and families through community-university partnerships: Success stories (pp.127-137). Boston: Kluwer Academic.
Spoth, R., Redmond, C., Hockaday, C., & Shin, C. (1996). Barriers to participation in family skills preventive interventions and their evaluations: A replication and extension. Family Relations 45, 247-254.
Spoth, R., Redmond, C., & Lepper, H. (1999). Alcohol initiation outcomes of universal family-focused preventive interventions: one- and two-year follow-ups of a controlled study. Journal of Studies on Alcohol 13, 103-111.
Spoth, R., Guyll, M., & Day, S. (2002) Universal family-focused interventions in alcohol-use disorder prevention: Cost-effectiveness and cost-benefit analysis of two interventions. Journal of Studies on Alcohol, volume # 219-228.
Revised 03/2009 39
APPENDIX 1-
FIDELITY BENCHMARKS: STRENGTHENING FAMILIES PROGRAM RECOMMENDED BEST PRACTICES AND PROGRAM STANDARDS
SFP is designed to reduce family environmental risk factors and improve protective factors with the ultimate objective of increasing personal resilience to drug use in high-risk youth. Research has demonstrated that the program is equally effective in reducing risk precursors for mental disorders and juvenile delinquency. SFP has been recommended as a best practice program by all federal agencies conducting expert reviews of individual programs, such as NIDA, CSAP, CMHS, DoEd Safe and Drug-free Schools, NIAAA, and OJJDP. For the purposes of assessing fidelity to the model three domains have been identified for this for this evaluation: 1) Community/Environmental Fidelity; 2) Population Fidelity; 3) Program Fidelity; 4) Staffing Fidelity; and, 5) Curriculum/Content Fidelity. The primary criteria for each of these domains are described as benchmarks in relation to established program standards.
COMMUNITY/ENVIRONMENTAL FIDELITY
Funding
The Strengthening Families Program budget provides for an itemized and general budget allowances. The recommended budget is based on a capacity of 12 families, with a recommendation for over-recruiting to maintain a functionally sized group of about 8 families. Expenses for conducting the program include site coordination, group leaders for delivering the program to families, food for a family meal, supplies, photocopying of manuals, in-session incentives, graduation incentives, graduation celebration, transportation, childcare and booster sessions. In-kind contributions are encouraged.
The budget is based on delivering the complete 14 sessions with fidelity of SFP (3-5), SFP (6-11) or SFP (13-17). Completion is usually at least one parent and child in a family attending 70% of the sessions (e.g., 10/14 sessions).
Community Context- The delivery of the program and fidelity measures are considered within the context of the host community, including population density and diversity. Variation is to be expected based on the interagency partnerships and mission of the agencies delivering the program. Variation is also expected based on the demographic make-up of the community (e.g., rural v. suburban v. seasonal v. urban) and community characteristics such as language differences and population age differences.
Safe and Welcoming Environment - Sites are selected based on accessibility and appropriateness for families to come together for a positive skills building program. The site must avoid stigmatizing or labeling attending families based on the local community’s perception of the activities and persons that generally frequent the site. For example, in some communities the substance abuse treatment center is only frequented by persons who are diagnosed with substance abuse treatment disorders, which deters families from “being seen there.” Some correctional facilities do not permit or are not considered appropriate for children. The site must be accessible by public transportation in those communities where the families
Revised 03/2009 40
utilize such transportation and/or have parking available in convenient well-lit lots. The site must not only be safe, but must be perceived as safe, particularly for young and vulnerable children.
Site Facilities - The program recommends that the site have adequate facilities for separate rooms for the children and parents to meet for one hour and for the families to meet together for a meal and one hour of program curriculum. Additionally, there must adequate space for childcare while parents are attending sessions. If the meal is to be prepared or stored on-site, there must be adequate facilities for food safety.
Transportation - Transportation to and from the program needs to be assured and coordinated within the resources of the community and program. This is particularly true since the families this program targets often do not have access to private transportation and/or cannot afford the gas to attend a program of this duration. Additionally, many of these families do not want and should not have to disclose that transportation is the barrier, particularly in the recruitment and early sessions of the program. Taking “hand outs” can be stigmatizing and shaming for some families. It is understood that in some communities public transportation does not exist; and in other communities public transportation may not be considered safe or accessible for families with young or multiple children. Transportation should not be a barrier to attending the program; but the “provision” of transportation should be measured within the community context of availability and community culture/expectations.
Community Calendar Congruity. The sessions are to be congruent with the local community calendar and events regularly attended by the target families (i.e., holidays, sporting activities, work schedules). Particularly, the sessions need to be coordinated with the host site (e.g., school vacations for school-sites; religious observations for faith-based sites; average length of stay for institutional settings). Changes in the calendar over the 14-session time span need to be anticipated and planned for.
TARGET POPULATION
Intervention Level - SFP was originally designed for selective or at-risk and high risk groups of children. This risk assessment has been extended to consider the family risk level, not just the children. The program has been delivered and tested and found successful with universal, selected, and indicated populations and has been tested with all three types of primary prevention approaches.
Age of Children - Three SFP curriculums are funded. The programs target three age ranges of children:
•
SFP (3-5)- targets children age 3-5;
•
SFP (6-11)- targets children age 6-11; and,
•
SFP (12-16)- targets children age 12-16.
Family Composition - The program requires that at least one child in a family and one parent participate in the three curriculum sessions. Parent is defined as the child’s primary caregiver(s) and is interpreted in a broad context (e.g., foster parents, boyfriends, step parents, adoptive
Revised 03/2009 41
parents, kinship care, etc.). Multiple caregivers are encouraged to attend the program. Multiple children in the age range are encouraged to attend.
Level of Risk- SFP was originally designed for high-risk and at-risk families, and has been successfully implemented with universal populations. It is important to determine and consider the specific risk and protective factors of the eligible and attending families a level of risk is assessed to determine if the population is of high risk, at-risk or moderate risk. Level of risk has been found to impact the retention and outcomes and overall program fidelity.
Special Populations- Some implementations have additional eligibility requirements or target intervention in specific communities or with specific family characteristics. These are noted and it is considered if these contribute to the risk level of the families.
Cultural Competence- based on the specific social, economic, racial, faith-based, language, ethnicity, disability or familial background, it is noted to be assured that the context, environment, program, staffing and curriculum accommodate the needs of the attending families. The curriculum and evaluation materials are available in both Spanish and English and has been shown to be effective across a wide range of cultural populations (e.g., White, African- American, Hispanic, Native American, Pacific Islander, rural, urban and suburban).
Recruitment- The NJ SFP initiative targets recruitment of at-risk and high-risk families for participation in the program. The recruitment is the responsibility of the site and largely reliant on relationships, partnerships and outreach on the part of the site coordinator and host agency. This initiative promotes recruitment of families involved in substance abuse treatment, the criminal justice system, child welfare (DYFS) and Temporary Aid to Needy Families (WorkFirst NJ). Over-recruitment is encouraged in order to maintain required retention levels since drop-outs prior to and during the first session are common. The relational ratio of referred to eligible to enrolled families provides determination of targeted capacity achievement rates.
PROGRAM FIDELITY
Program Length/Dosage - The Strengthening Families Program is designed to be conducted in consecutive sessions, with each session lasting approximately 1 ½ - 2 hours (2 - 2 ½ hours when combined with the meal). The prescribed number of sessions is 14.
The sessions are usually delivered once a week but may be delivered twice a week. Recent analysis of the data in the NIDA research study suggests that the results for reductions in antisocial behavior is not as good if the program doesn’t run for 14 weeks.
Meal - Generally a light meal is served to families as they arrive, making the activities 2 ½ hours in duration at each session. This meal is a time for group leaders to observe the families; for families to practice and check in regarding skill development; to reinforce family rituals; and, to remove a barrier to retention.
Following the general welcome, the first hour is spent with the parents and children meeting in their own respective groups. At the end of these groups, families are reunited and have a short
Revised 03/2009 42
break together. The second hour is spent in the Family Skills Training portion of the program. Depending on the number of participants, this group may be divided into smaller groups or may remain together.
Retention - SFP is a 14 session curriculum that allows for adequate time and dosage for families to learn, implement, practice and evaluate their progress in skill building, particularly in areas of family communication, positive discipline and family organization. Retention of families in a 14-session program today is very challenging (Kumpfer, 1991). SFP recommends meals, childcare, transportation, and culturally matched group leaders to increase retention. Originally, SFP required that families complete 12 of 14 sessions to graduate. However, in the last decade, we are happy to have families complete 8 of the 14 sessions. The average completion rate is now about 8.5 sessions across the SFP implementations. We have found that having the families sign a contract to complete at least 8 sessions does increase retention. The New Jersey sites were sent this contract to help improve retention.
Attrition has been higher in the initial implementation and retention should increase in subsequent cycles. Across a number of prior evaluations, it was found that only about 50% of families graduate in the first group an agency runs as they are in training or this first group can be considered their pilot group. With more experience, over 4 to 5 groups, the group leaders should become more effective at retaining families and end with about 85% of the families graduating (Aktan, Kumpfer, & Turner, 1995). Of course, this does vary with the ease of attendance and closeness of the clients to the staff and agency. Some sites with very engaging and respectful group leaders with great incentives and a needy group of parents have been known to graduate 100% of their clients in the first group.
Incentives - Program incentives for participation increase retention and reinforce the program. Incentives that are tied to, build on and reinforce the curriculum are recommended. These include a family meal provided at each session, transportation, childcare, graduation certificates and completion rewards, and intermittent grab bags and supplies necessary for the family to complete the homework assignments and weekly curriculum activities. Many programs offer additional incentives, including weekly vouchers for attendance with cash value.
Child Care - Childcare is recommended to be provided at the site during the sessions. Since the program is promoting parental responsibility and family organization, the program needs to facilitate and assure age appropriate care for other children in the family, both younger and older than the participating children. Childcare provision or babysitting is to be in keeping with providing safety and fun for children not including in the skills training. Some sites provide the older children with the junior high school version of SFP (Kumpfer, Molgaard, & Spoth, 1997), which is a very effective program with one of the largest cost-benefit ratios of any drug prevention program—a $9.60 savings for every dollar spend on it (Spoth, Guyll, & Day, 2002). Some sites hire some of the older youth to help in the babysitting with a partial salary. It is important that it meet local and host or partner agency regulations and guidelines, keeping in mind that childcare is only provided for 2 hours and the parents are onsite. Hence, it is possible to call this “babysitting” which often avoids the extra requirements of licensed childcare programs.
Revised 03/2009 43
Follow-up Booster Sessions - Following the completion of the fourteen sessions, programs need to address follow-up and on-going support for families. This includes linkage when necessary to community services. This also includes any plan for a 6-month Follow-up or Booster Session. At these sessions the families come together again. It is an opportunity for the families to reflect on the programs impact on their lives, receive assistance in content areas unclear or problematic, to receive new educational or family skill building, participate in program evaluation and, moreover, reinforce the positive bonds they built with each other in the program. The format for these sessions is flexible and determined by the needs of the families, programs, evaluators and funding prerequisites.
STAFFING FIDELITY
Group Leaders - A total of four group leaders are recommended to deliver the program, with a group leader and co-group leader for the Parent Training group and another group leader and co-group leader for the Children's Skill Training group. During the Family Skills Training sessions, all four group leaders facilitate the session. It is strongly recommended that the two group leaders be gender balanced (both a man and a woman) and ethnically matched to the participants.
Site Coordinator - A Site Coordinator is responsible for oversight, logistics, staff supervision and coordinating the program implementation and delivery. This includes being accessible to families between sessions, towards assuring retention. The site coordinator is a fifth staff person that is present at the sessions. They are NOT a group leader.
Training - The staff implementing SFP is to have completed the SFP two-day training. Trainings are conducted for each of the three curriculums. This training provides group leaders and site coordinators with training in program implementation, curriculum content and theory and working with at-risk and high-risk families. It is not necessary for staff to be credentialed in mental health, substance abuse treatment or prevention, although it may be helpful with some higher-risk populations.
Additional Staff - Additional staff includes childcare providers, food preparation, staff and van drives, as needed for program implementation. Childcare providers are recommended to provide on-site childcare and supervision of families’ youth not participating in the curriculum due to age inappropriateness. In some communities staff includes food preparation, staff and van drivers.
CURRICULUM FIDELITY
Three Skills Training Curriculums- Curriculum fidelity is dependent on group leaders’ delivering all prescribed sessions, assigning and reviewing homework and including the content areas specified for each session in sequence. Skills training methods for the parents', children's and family groups include lecture, demonstration, discussion, role playing, audio-visuals, charts, homework assignments, practicum exercises, peer support, puppet shows, games, Child's Game, Parents' Game, supervised practice and video-taping practicum exercises. Actual delivery of the direct/verbatim curriculum will vary depending on the individual characteristics of the group leaders and the group members. The curriculum is spelled out in manuals complete with
Revised 03/2009 44
instructions for delivery, key lecture content, details of activities, lists of materials needed, homework assignments and handouts for copying and distribution. An overview of the Parent Training, Child Training and Family Training curriculum is indicated in the Table of Contents of each module.
Developmental Appropriateness- Sites are encouraged to use the SFP version that was originally designed for the age range of the families attending in order to assure developmental appropriateness of the curriculum.
Modeling the Program Skills. Group leaders are expected to model the tenants of the program when interacting with the families, including at the family meal. Activities and skills are designed for and appropriate for both parents and children according to the developmental age of the children targeted in the age specifications of the curriculum.
Cultural and Local Adaptations. It is recommended that each local site tailor the program to accommodate cultural and community diversity. The program is designed to provide a framework and an outline of activities that will meet each program lessons objectives. The skills and activities are prescriptive and designed to be sequentially lead to the families (both children and parents) developing skills proven to result in improved family, child and parent behavioral and affective outcomes and reduced risk behaviors. (These outcomes are assessed in the outcome evaluation instruments). However, the group leaders are encouraged to make the program more culturally and locally appropriate by changing the names of people in the stories or puppet plays, using more appropriate ethnic stories for storytelling, adding food, cultural and dances or games that the participants find reflect their traditional family values.
Creative and Interactive Delivery. The program may be adapted but not modified. Group leaders are encouraged not to read from the training manuals during the sessions, but rather to present the material in a well-thought out professional manner. They are encouraged to use personally developed flip charts or poster boards for visual outlines of their major points. This helps visual learners to learn better, personalizes the program (vs. power point presentations or overheads), and helps the Group Leaders not to read from their books. They look better prepared and respectful to the families with prepared material in advance of the group.
The program does not have to be implemented in a word-for-word manner. Examples of what to say are included in the manual as examples. Group leaders should personalize the delivery to fit their style, local language and examples.
The Group Leaders are also encouraged to “get creative” with their program materials by having concrete hand-on materials (e.g., art projects, puppets, hats, and props such as a flower pot with sun flower seeds for demonstrating following directions or a “tool kit” to demonstrate that the parents are learning new tools for their parenting tool box, etc.). In addition, they are asked to make the program more interactive with role plays, games, and exercises that will get the major points across, but in a manner that is culturally or locally sensitive and appropriate, while making it more fun for the families.
Revised 03/2009 45
LUTRAGROUP SP
5215 Pioneer Fork Road, Salt Lake City, Utah 84108-1678 .
Phone: 801.583.4601 ♦ hwhiteside@lutragroup,com ♦ Fax: 801.583.7979
State of Oklahoma
Strengthening Families Program
Site Visit Reporting Memo
Evaluation Consultant: Jeanie Ahearn Greene Phone: 240-460-3931
Contracted Agency: Specialized Outpatient Services, OKC, OK Phone: 405-810-1776
Project Officer: Elicia Chandler
Date of Visit: 11/5/2009
SESSION #: 11
Site Location: SOS, Oklahoma City, OK
Time Started: 5:30 P.M. Number of Group Leaders: 4
Number Families Enrolled: 6 Number of Families Attending: 2
Session Fidelity Summary: (SFP 6-11)
The session was conducted at a substance abuse treatment agency. The site was safe and welcoming with adequate parking, good space for the meal, parent, child and family groups. Children were transported by foster parents. Attending parents arranged their own transportation. The meal was appetizing and the families sat in family groups with group leaders in attendance. The meal was followed by one hour parent/child sessions and 45 minute family session. In all sessions the curriculum was followed, homework was reviewed and family members were engaged and actively participating in the program. Each session had at least one group leader that had completed the SFP Group Leader Training. All key fidelity benchmarks were met with no indications of modifications to the curriculum that would compromise program effect.
Notable Program Strengths:
The staffs were familiar with and comfortable with the attending families/target population. The parents were supportive to each other and talked among themselves at the meal, during breaks and the session. The child session was very active and the children participated in the role plays and were eager to practice and demonstrate peer refusal skills. The group leaders allowed for their “activity” while maintaining control of the group, which at time posed a challenge.
Of note was that all families were from very different situations with one single mother with five children and one two-parent family with. As the focus was on parenting skills the parents were respectful of each other and shared strategies between themselves.
The group leaders developed a family activity of making a family crest providing an example of how British Crests work and each family was given materials to make one of their own for
Revised 03/2009 46
display at graduation and in their homes. This was an excellent activity as it accommodated all of the family members, particularly when working with families that ranged from one to five children. The families were particularly engaged and when the time for the session was over, asked to take it home to complete it which provided a great segue to home practice.
Fidelity Concerns:
The group leaders were eager to deliver the curriculum with fidelity but could use some additional support in assuring that the focus is on the skills and reward based. As is usual in initial cycles, the low number of families limits the support in the parents group and focus on the skills, instead of individual family interventions. Continued technical assistance and support should be provided to increase the number of families and attendance. Again, this often resolves itself in subsequent cycles as the program becomes better known by the referral sources and barriers to attendance are addressed prior to them becoming an issue (e.g., transportation, schedule conflicts and other drop out issues, program appeal).
Recommendations:
This staff would benefit from advanced training with the other sites, as child group management and adaptation are cross-site issues that sites would benefit from shared strategies that are successful with this high-risk population. As the parents were learning about discipline, in particular the group leaders could benefit from additional training and technical assistance in limit setting, discipline and “punishment”.
As some of the staff had not completed the group leader training, this advanced training could be addressed in a general group leader training with veteran staff attending and sharing their expertise for the new recruits. It is also recommended that SOS train additional staff so that they have sub substitutes when needed. Strong efforts should be made to recruit male and diverse group leaders, particularly for the children’s group.
Final Comments:
This program is operating well organized staff delivering the program with fidelity to the targeted families. The two attending families were engaged and demonstrating learning and using the intended skills. A future concern is the potential for high numbers of children and parents in the families, particularly if the program achieves the intended capacity of 10 families. If larger group sizes are affected, then appropriate staffing levels must be determined. Additionally, this program should be tracked and provided technical support for enrolling, engaging and retaining families.
Revised 03/2009 47
B. Partnership and Client Data
This section provides a snapshot of the RPG operation —both as an independent initiative and as a part of the larger community. It includes information regarding clients served, new partner agencies, and trainings conducted.
Project Implementation: Report on Partnership and Client Data
For all client data, please include the number for this reporting period, in addition to the total number since the inception of your grant. Only include data reflecting your treatment or intervention population; do not include data from your comparison groups. These should be unduplicated counts of adults, children, and families served.
Client Data
Number
Notes
This Reporting Period
Since Start of Grant
Number of target population families served by the RPG Program who received services
Adults Served
33
80
Children Served
27
92
Families Served
24
55
Number of target population families served by the RPG Program who received substance abuse treatment services (Note: data for children/youth should reflect those receiving treatment for their own substance use disorder, as opposed to support or other services they received as a result of their parent/caregiver’s substance use disorder)
Adults Served
05
31
Children/Youth Served
00
00
Number of new admissions to the RPG Program
Adults Admitted
33
80
Children Admitted
27
92
Families Admitted
24
55
Number of discharges from the RPG Program
Adults Discharged
06
23
Children Discharged
13
40
Families Discharged
08
20
Of children served by the RPG Program:
Number of open child welfare cases (defined by grantee/state)
27
92
Specify how open cases are defined:
Substantiated reports of abuse;
Of children served by the RPG Program, number of children in:
In-home care
00
00
Out-of-home care
27
92
Revised 03/2009 48
Performance Indicator F4. Coordinated Case Management
Grantees who are reporting treatment and/or control or comparison group data on indicator F4. Coordinated Case Management should also complete the following table. If you selected this measure and indicated that you would report only certain aspects of this indicator, or you do not have control/comparison group data, please put N/A for the elements you are not reporting.
If this is indicator is not part of your final set of indicators, please put N/A here: _N/A .
Client Data
Number: Treatment Group
Number: Control/ Comparison Group
Notes
a. Of the number of families served this reporting period (as indicated above), number who report active involvement in various aspects of case planning
Denominator information for parts b. and c.:
Number of families served this reporting period who have open cases in both the child welfare and substance abuse treatment systems
b. Of the number of families served this reporting period who have open cases in both the child welfare and substance abuse treatment systems, the number who received joint case management services coordinated between the substance abuse and child welfare systems
c. Of the number of families served this reporting period who have open cases in both the child welfare and substance abuse treatment systems and received joint case management services, the number who received a cross-agency assessment conference every 90 days or less
Revised 03/2009 49
Changes in Regional Partnership Membership
New Partner Agencies this Reporting Period
List Name of Each New Agency or Service Delivery System Partner for this reporting period
(List agency/system name, not individual person; if no new members, put “None” or “NA”)
Was Written Partnership Agreement/Memorandum of Understanding (MOU) Established with This New Partner Agency?
The Center for Therapeutic Interventions
In Process
Written Partnership Agreements or MOUs with Existing Partner Agencies
this Reporting Period
List Any Existing Partner Agencies for which Written Agreements/MOUs were Established this Reporting Period
(If none, put “None” or “NA”)
Indicate if New or Renewed Agreement
Oklahoma Health Sciences Center (OUHSC)-New Directions
Renewed
Specialized Outpatient Services (SOS)
Renewed
Number and Types of Staff and Substitute Caregiver Trainings
Indicate number of trainings and staff who participated for each training topic area.
Please Note: For grantees who are reporting on indicator F5. Substance abuse education and training for foster care parents and other substitute caregivers, please report this information in the designated part of this table that follows staff trainings.
TOTAL Number of Staff Trainings and Staff Participating
Please indicate the total number of training events staff participated in (whether conducted by the RPG or another organization) and the number of staff who participated.
TOTAL Number Staff Training Events
TOTAL Number All Staff Participating
Total Number of Staff Participating From:
Child Welfare
Substance Abuse
Courts
Other
OUHSC/Child Abuse and Neglect Staff
Other ODMHSAS
Staff
38
428
126
153
60
74
15
Revised 03/2009 50
Number of Staff Trainings and Staff Participating – BY TOPIC
Please provide a breakdown of the number of training events and staff participating by topic. Add additional rows as needed for other topics not specified. Note: The sum of the number of trainings by top