Cross Site Evaluation - Annual Evaluation Report (Year 3), October, 2009 |
Previous | 1 of 1 | Next |
|
small (250x250 max)
medium (500x500 max)
Large
Extra Large
large ( > 500x500)
Full Resolution
|
This page
All
|
Evaluation of the Mental Health Transformation State Incentive Grant (MHT SIG) Program Annual Evaluation Report October 22, 2009 Prepared for The Department of Health and Human Services Substance Abuse and Mental Health Services Administration Center for Mental Health Services Prepared by MANILA Consulting Group, Inc., and Human Services Research Institute (HSRI) MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 Contents INTRODUCTION .......................................................................................................................................... 1 ACCOMPLISHMENTS ................................................................................................................................ 1 Transformation Tracker Guidance and Support ....................................................................................... 1 POC Guidance and Support ..................................................................................................................... 1 State Site Visits ........................................................................................................................................ 1 Federal Agency Site Visit .......................................................................................................................... 2 Consumer and Family Member Focus Groups ......................................................................................... 2 Evaluation Work Group Meetings ............................................................................................................. 2 2008 Annual Grantee Meeting .................................................................................................................. 2 Active Involvement of Consumer and Family Member Consultants ......................................................... 3 MHT SIG Web Site .................................................................................................................................. 3 Leadership Survey ................................................................................................................................... 3 Provider Survey ....................................................................................................................................... 3 Other Accomplishments........................................................................................................................... 4 FINDINGS .................................................................................................................................................... 4 Transformation Tracker Data: GPRA Activities and the President’s New Freedom Commission Goals . 4 Proof of Concept Studies ........................................................................................................................ 18 State Site Visits ...................................................................................................................................... 25 Federal Agency Site Visit ........................................................................................................................ 26 Consumer, Youth, and Family Member Focus Groups .......................................................................... 28 National Outcome Measures Data ......................................................................................................... 34 Analysis of State Employment Initiatives ................................................................................................ 36 Leadership Survey ................................................................................................................................. 36 Provider Survey ..................................................................................................................................... 38 State Specific Evaluation Activities ......................................................................................................... 38 SUMMARY ................................................................................................................................................. 39 APPENDIX A: GPRA TARGETS ............................................................................................................... 46 APPENDIX B: GPRA RESULTS ................................................................................................................ 48 APPENDIX C: PROOF OF CONCEPT STUDIES INFORMATION ........................................................... 50 APPENDIX D: NATIONAL OUTCOME MEASURES FROM THE CMHS UNIFORM REPORTING SYSTEM, 2004–2008 ................................................................................................................................ 54 MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 Tables TABLE 1. 2007-2009 NUMBER OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ........................................ 6 TABLE 2. 2007-2009 PERCENTAGE OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ................................. 7 TABLE 3. 2007 NUMBER OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ................................................. 8 TABLE 4. 2007 PERCENTAGE OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS .......................................... 9 TABLE 5. 2008 NUMBER OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ............................................... 10 TABLE 6. 2008 PERCENTAGE OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ........................................ 11 TABLE 7. 2009 NUMBER OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ............................................... 12 TABLE 8. 2009 PERCENTAGE OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ........................................ 13 TABLE 9. SUMMARY OF 2007 TRANSFORMATION ACTIVITIES BY NEW FREEDOM COMMISSION GOALS ............... 15 TABLE 10. SUMMARY OF 2008 TRANSFORMATION ACTIVITIES BY NEW FREEDOM COMMISSION GOAL ............... 16 TABLE 11. SUMMARY OF 2009 TRANSFORMATION ACTIVITIES BY NEW FREEDOM COMMISSION GOAL ............... 17 TABLE 12. STATUS OF RECOVERY STUDIES .................................................................................................... 18 TABLE 13. STATUS OF RESILIENCY STUDIES ................................................................................................... 19 TABLE 14. STATUS OF STATE PROOF OF CONCEPT STUDIES ........................................................................... 21 TABLE 15. RANKING OF NFC RECOMMENDATIONS BY MHT SIG PROGRAM MANAGERS AND CONNECTORS ...... 27 TABLE 16. AVERAGE CYF RATING OF FREQUENCY AND TYPE OF CONSUMER, YOUTH, AND FAMILY MEMBER INVOLVEMENT IN GRANT ACTIVITIES ...................................................................................................... 29 TABLE 17. AVERAGE CFY RATINGS REGARDING VARIOUS ASPECTS OF CFY PARTICIPATION IN MHT SIG GRANT ACTIVITIES .......................................................................................................................................... 31 TABLE 18. NUMBER OF MHT SIG GRANTEE STATES REPORTING INCREASES, DECREASES, AND NO CHANGES IN THEIR NOMS DATA FROM 2004 TO 2008 .............................................................................................. 35 TABLE 19. RESULTS OF PILOT TEST OF PROVIDER SURVEY ............................................................................. 38 TABLE 20. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING MENTAL HEALTH PROGRAM/SERVICES/SYSTEMS/CLIENTS .................................................. 40 TABLE 21. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING STIGMA ............................................................................................................................. 41 TABLE 22. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING FUNDING & RESOURCES .................................................................................................... 41 TABLE 23. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING WORKFORCE AND TRAINING ............................................................................................... 42 TABLE 24. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING CYF, CYF ORGANIZATIONS, & COMMUNITY STAKEHOLDERS ............................................... 43 TABLE 25. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING DISPARITIES, PREVALENCE, & STATISTICS .......................................................................... 44 TABLE 26. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING CRIMINAL/JUVENILE JUSTICE .............................................................................................. 44 TABLE 27. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING TRANSFORMATION WORKING GROUPS AND COMMITTEES .................................................... 45 TABLE 28. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING OTHER TOPICS .................................................................................................................. 45 TABLE 29. CUMULATIVE GPRA TARGET NUMBERS ......................................................................................... 46 TABLE 30. CUMULATIVE GPRA RESULTS/ACCOMPLISHMENTS ........................................................................ 48 TABLE 31. STATE INTERVENTIONS EXAMINED IN RECOVERY (ADULT) STUDY .................................................... 50 TABLE 32. STATE INTERVENTIONS EXAMINED IN RESILIENCE (CHILD) STUDY .................................................... 51 TABLE 33. GPRA MEASURES ADDRESSED RECOVERY (ADULT) STUDIES ........................................................ 51 TABLE 34. NFC GOALS ADDRESSED BY RECOVERY (ADULT) STUDIES ............................................................. 51 TABLE 35. GPRA MEASURES ADDRESSED BY RESILIENCE (CHILD) STUDIES ................................................... 52 TABLE 36. NFC GOALS ADDRESSED BY RESILIENCE (CHILD) STUDIES ............................................................ 52 TABLE 37. INDIVIDUAL-LEVEL RECOVERY INSTRUMENTS BEING USED BY GRANTEE STATES .............................. 52 TABLE 38. SYSTEM-LEVEL RECOVERY ORIENTATION INSTRUMENTS BEING USED BY GRANTEE STATES ............ 52 TABLE 39. INDIVIDUAL-LEVEL RESILIENCE INSTRUMENTS BEING USED BY GRANTEE STATES ............................. 52 TABLE 40. SYSTEM-LEVEL RESILIENCE ORIENTATION INSTRUMENTS BEING USED BY GRANTEE STATES (OPTIONAL) ......................................................................................................................................... 53 MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 TABLE 41. NOM 2: INCREASED OR RETAINED EMPLOYMENT AND SCHOOL ENROLLMENT: PERCENTAGE OF ADULTS WITH KNOWN STATUS WHO ARE EMPLOYED ............................................................................. 54 TABLE 42. NOM 4: INCREASED STABILITY IN FAMILY AND LIVING CONDITIONS: PERCENTAGE OF ADULTS WITH KNOWN STATUS WHO ARE LIVING IN PRIVATE RESIDENCE ..................................................................... 54 TABLE 43. NOM 5: INCREASED ACCESS TO SERVICES: PERCENTAGE OF PERSONS SERVED BY AGE: PERCENTAGE OF CLIENTS WHO ARE UNDER AGE 18 OR ARE AGE 65 OR OVER ...................................... 55 TABLE 44. NOM 5: INCREASED ACCESS TO SERVICES: PERCENTAGE OF PERSONS SERVED BY RACE/ETHNICITY (PERCENT OF CLIENTS WHO ARE WHITE) .............................................................................................. 55 TABLE 45. NOM 6: DECREASED UTILIZATION OF PSYCHIATRIC INPATIENT BEDS: MENTAL HEALTH HOSPITAL UTILIZATION RATE PER 1,000 ................................................................................................................ 56 TABLE 46. NOM 7: INCREASED SOCIAL SUPPORT/SOCIAL CONNECTEDNESS: PERCENTAGES FOR ADULTS AND CHILDREN/FAMILIES .............................................................................................................................. 56 TABLE 47. NOM 8: INCREASED POSITIVE REPORTING BY CLIENTS ABOUT OUTCOMES: PERCENTAGE OF ADULT CONSUMERS WITH POSITIVE REPORTING OF OUTCOMES ....................................................................... 56 TABLE 48. NOM 10: INCREASED USE OF EVIDENCE-BASED PRACTICES: PERCENTAGE OF STATES IMPLEMENTING ADULT SUPPORTED HOUSING ................................................................................................................ 57 TABLE 49. NOM 10: INCREASED USE OF EVIDENCE-BASED PRACTICES: PERCENTAGE OF STATES IMPLEMENTING ADULT SUPPORTED EMPLOYMENT ......................................................................................................... 57 TABLE 50. NOM 10: INCREASED USE OF EVIDENCE-BASED PRACTICES: PERCENTAGE OF STATES IMPLEMENTING ADULT ASSERTIVE COMMUNITY TREATMENT .......................................................................................... 57 TABLE 51. NOM 10: INCREASED USE OF EVIDENCE-BASED PRACTICES: PERCENTAGE OF STATES IMPLEMENTING THERAPEUTIC FOSTER CARE FOR CHILDREN AND ADOLESCENTS ............................................................ 58 MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 Figures FIGURE 1. AVERAGE RATING OF GRANT IMPACT BY STATE .............................................................................. 25 FIGURE 2. AVERAGE RATING OF GRANT IMPACT BY TYPE OF STATE AGENCY ................................................... 26 FIGURE 3. AVERAGE RATINGS ON STATEMENTS PROVIDED BY CONSUMER AND FAMILY MEMBER FOCUS GROUP PARTICIPANTS ACROSS ALL STATES ...................................................................................................... 34 MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 1 INTRODUCTION This annual report identifies accomplishments and findings related to the third year of the Mental Health Transformation State Incentive Grant (MHT SIG) cross-site evaluation project. The third year of the project began September 15, 2008, and ended September 14, 2009. The evaluation of the MHT SIG program is designed to measure the success of transformation efforts in grantee States to meet New Freedom Commission (NFC) goals. The project supports nine States as they implement their MHT SIG grants. The first cohort of States, awarded Center for Mental Health Services (CMHS) grants in 2005, includes seven States (Connecticut, Maryland, New Mexico, Ohio, Oklahoma, Texas and Washington). The second cohort, awarded grants in 2006, includes two States (Hawaii and Missouri). ACCOMPLISHMENTS During this year of the project, the project team—MANILA Consulting Group, Inc. (MANILA), and Human Services Research Institute (HSRI)—focused efforts on completing the consumer, youth, and family focus groups in all nine grantee States and developing associated reports, conducting site visits to Hawaii and Missouri, launching Transformation Tracker Explorer, coordinating with States regarding Proof of Concept (POC) study requirements, completing leadership and provider surveys, and examining methods for conducting the various cross-site analyses for the final report. Transformation Tracker Guidance and Support The cross-site evaluation collects information in Transformation Tracker on planned MHT SIG activities, including the nature of the activities, their scheduling, and whether or not they are completed. These activities are coded by States as to New Freedom Commission Goals addressed and SAMHSA Government Performance and Results Act (GPRA) categories. The MHT SIG program has seven Government Performance and Results Act (GPRA) categories of infrastructure changes for which States must provide annual progress updates: policy changes, workforce trainings, financial policy changes, organizational changes, data sharing and analysis, membership in consumer/family member (CFM) networks, and implementation of practices consistent with the State’s Comprehensive Mental Health Plan (CMHP). The project team provided detailed guidance and support to Hawaii and Missouri as they completed initial entry of their activities into Transformation Tracker. In addition, guidance was provided to other States regarding various issues with their Transformation Tracker entries and GPRA 6 (membership in CFM networks) supporting information. Additionally, the cross-site team began discussions of how to clean, code, and analyze the Transformation Tracker data from the perspectives of New Freedom Commission Goals and Recommendations and SAMHSA GPRA categories. POC Guidance and Support The evaluation design requires States to identify services for adults with serious mental illness and children/youth with serious emotional disturbance that have been or are anticipated to be impacted by the MHT SIG project (the impacted group) as well as comparable services that have not been impacted by the project (the nonimpacted group). The goal is to determine if positive changes in recovery (for adults) and resilience (for youth) result from “best case” transformative initiatives consisting of both infrastructure and service changes. During the past year, for these recovery and resiliency studies, the project team focused on different activities by cohort. For the first cohort of States, the evaluation team clarified details of their POC proposals and approved completed proposals, completed successful test data submissions for all instruments to be used, supported and monitored data collection efforts and early stages of data submission, and developed preliminary data analysis plans based on anticipated data collection rates. For the second cohort of States, the primary focus was on supporting the development of their POC proposals. State Site Visits MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 2 State site visits are conducted to determine the effect the MHT SIG projects have had on their mental health service systems, to understand why the States have chosen the approaches they have, and to learn about any barriers or needs associated with the grant program. The initial site visit with each grantee is conducted in grant year 3, with a second one occurring in grant year 5. The cross-site evaluators conducted site visits in Hawaii on December 17 and 18, 2008, and in Missouri on January 28 and 29, 2009. Draft site visit reports were prepared and provided to both States. Federal Agency Site Visit From December 2008 to early March 2009, the project team interviewed key CMHS managers, all six Connectors (the MHT SIG program’s Federal Project Officers), and two other CMHS staff to understand the development and implementation of the MHT SIG program to date. All interviewees also were asked to rank the New Freedom Commission recommendations in order of importance. Consumer and Family Member Focus Groups Consumer and family member focus groups with up to 15 individuals per State were designed to identify the nature of consumer/family member involvement in transformation grant activities at both the State and local levels. Within each State, the focus groups were to consist of persons who were aware of the grant—some who were involved in grant activities and some who were not. The process of identifying CFM focus group participants and scheduling meetings was found to be more difficult than anticipated. In particular, it was challenging to find individuals who were aware of the mental health transformation grants, regardless of whether they were involved in grant activities. Other difficulties with recruitment included reluctance to participate due to limited familiarity with the Federal grant program and lack of responsiveness from CFM organizations that were expected to be potential sources of participants. In addition, some persons who initially indicated interest in participation did not respond to subsequent followup attempts. Focus groups were conducted during the State site visits with all the CFMs who were available. All other focus group interviews were completed by phone afterwards. By the end of the third year, all focus group interviews had been completed in all States. Reports for focus groups with the first cohort of States were provided on July 17, and public versions of these reports were developed and provided on September 3. Reports for Hawaii and Missouri are under development. Evaluation Work Group Meetings Evaluation Work Group meetings provide a forum in which State evaluators can update the evaluation team on the status of their efforts, identify issues and needs associated with their project, and obtain peer-to-peer support. During the third project year, 11 Evaluation Work Group meetings were held by telephone. Calls were used to provide guidance on POC studies and discuss implementation issues as well as the status of State-specific evaluation studies, the status of leadership and provider surveys, the potential development and subsequent results of launching the publicly available Transformation Tracker Explorer Web site, State-specific evaluation activities, and topics for the November 2009 Annual Grantee Meeting. Participation by evaluators from all States was consistently strong for all of the meetings. 2008 Annual Grantee Meeting The 2008 Annual Grantee Meeting was conducted on November 12 through 14 in Bethesda, Maryland. This meeting provided an opportunity for program staff and grantees to share accomplishments and identify objectives for the coming year. Three sessions were devoted to evaluation topics: Cross-Site Evaluation Progress Report: The Project Officer and Principal Investigator provided information about accomplishments in the cross-site evaluation and preliminary findings from the State planned MHT SIG activity data. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 3 Evaluation Informing Grant Implementation: Challenges and Opportunities: Evaluators from Connecticut, Hawaii, Oklahoma, and Washington made presentations about strengths-based approaches that can facilitate the process of evaluation and grant project activities informing one another. Cross-Site Evaluators Meeting: With no preset agenda for this session, Hawaii, Maryland, New Mexico, Ohio, Oklahoma, and Washington used the time to describe particular evaluation activities/initiatives in their States. The session also included general discussion about disparities in access to mental health services, POC issues, State-specific evaluation activities, and methods for determining when transformation has occurred. Active Involvement of Consumer and Family Member Consultants The evaluation project has six CFM Consultants who participate in all aspect of the project to ensure that CFM perspectives are incorporated into the design and implementation of the evaluation (e.g., site visits, focus groups, Evaluation Work Group calls, annual grantee meetings). During the third project year, the CFM Consultants were involved in the two State site visits and associated focus group interviews. They also reviewed the site visit and focus group reports for the States they visited. Three of the Consultants were able to participate in the cross-site team meeting in Boston in August 2009. One Consultant has been active with the MHT SIG Consumer, Family, and Youth Work Group and has participated in many of its monthly conference calls. The Consultants also reviewed and commented on the Transformation Tracker Explorer test site prior to its launch in May 2009. MHT SIG Web Site This Web site was developed early in the evaluation project to provide grantees with resource and project information and a system in which States could enter their MHTSIG activities. The major change to the Web site this year was the launch of Transformation Tracker Explorer (http://mhtsigdata.samhsa.gov) on May 19, 2009. This is the public part of the MHT SIG evaluation project Web site that allows the public to see, rate, and comment on GPRA activities being undertaken in all nine grantee States. This Web site is thought to be one of the first of its kind, providing current information about grantee activities and accomplishments during the course of an evaluation project. On August 18, 2009, the project team submitted a 90-day report detailing Web site traffic to date and identifying the MHTSIG activity ratings and comments that had been submitted. By the end of the third year, there were approximately 1000 visits by approximately 500 unique individuals from 42 States and 16 other countries. Leadership Survey Conducted in grant years 3 and 5, the leadership survey is designed to capture leadership characteristics of persons heading up the MHT SIG transformation efforts within the States. “Leaders” are the Project Directors and Transformation Working Group (TWG) chairs, while “raters” are TWG members or TWG subcommittee members. For the first cohort of States, the online leadership survey was launched in October 2008; 10 of 11 leaders and 51 of 76 raters completed the survey by March 2009. This survey will be conducted in Hawaii and Missouri in November 2009 to maintain the same timing of the survey as for the first cohort of States. Provider Survey The provider survey is designed to determine whether providers are aware of the MHT SIG grant in their State, and if so, how their State has been affected by mental health transformation grant initiatives. In the original evaluation design, each grantee was to survey 26 providers in each of grant years 3, 4, and 5. In conjunction with the Project Officer, it was decided that a pilot survey of providers with a smaller group of participants in year 3 would be a useful approach in understanding how providers might respond to the survey questions and whether any additional probes might be useful. The pilot survey was completed in all States except Hawaii; three of four provider surveys in Hawaii still need to be completed. The MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 4 remaining surveys will be completed during grant years 4 and 5, beginning in late September 2009. Providers will be encouraged to complete the survey online to make efficient use of staff time (i.e., staff will not need to schedule and conduct phone surveys or perform data entry). Other Accomplishments Other project accomplishments include the following: An analysis of employment initiatives by grantee States was done in January 2009 to determine the types of initiatives being undertaken, including participation in competitive labor force efforts and any U.S. Department of Labor initiatives. Presentation at the National Association of State Mental Health Program Directors (NASMHPD) Research Institute: On April 15, 2009, the Project Officer and a project staff member presented information about the MHT SIG evaluation project and preliminary findings on integration of mental health and physical health care. Meeting with MHT SIG Connectors: On April 21, 2009, project staff met with the Connectors to discuss the ways GPRA information could be used for program management purposes and the plans to launch Transformation Tracker Explorer. Support for the MHT SIG Consumer, Family, and Youth Work Group: At the request of the MHT SIG Consumer, Family, and Youth Advisor/Consultant, the project staff provided a demonstration to this group about how to use Transformation Tracker Explorer during a conference call on May 20, 2009. Cross-Site Team Meeting: On August 25 and 26, 2009, the entire cross-site team met to discuss and make decisions about the cross-site analyses to be completed by the end of the project. The Project Officer attended this meeting along with three of the six CFM Consultants. Participation on Planning Committee for the 2009 Annual Grantee Meeting: In July and August 2009, the project participated in Planning Committee meetings to help determine agenda topics and presenters for the upcoming November 2009 meeting. Analysis of Other State Data: Public health epidemiological datasets to measure statewide outcomes were obtained from the following other state data sources: SAMHSA’s National Survey on Drug Use and Health (NSDUH) for 2002 to 2007 and Behavioral Risk Factor Surveillance Survey (BRFSS) from the Centers for Disease Control and Prevention (CDC) for 2002 to 2008. These datasets and other public data sources are being examined to identify relevant mental health outcomes and relate them to CMHS National Outcomes Measures (NOMs). We also are reviewing other longitudinal datasets (e.g., the data on which the NAMI Grading the States Reports were based) that might have information on outcomes or infrastructure changes in States. In addition to analytic approaches that compare MHTSIG States with Non-MHTSIG control States over time, we are also exploring “dose-effect” analytic methods for analyzing the associations between transformative practices and infrastructure and outcome changes within MHTSIG States. FINDINGS The findings from the third year of the cross-site evaluation project are discussed in the following sections on Transformation Tracker data, POC studies, State site visits, Federal agency site visit, consumer and family member focus groups, statewide National Outcome Measures (NOMs) data, analysis of State employment initiatives, leadership survey, provider survey, and State-specific evaluation activities. Transformation Tracker Data: GPRA Activities and the President’s New Freedom Commission Goals MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 5 The cross-site evaluation project has tracked MHT SIG State activities in Transformation Tracker, where entries are coded by GPRA measure and by the President’s New Freedom Commission goal. Given the central role of the NFC report in the origination of the MHT SIG program, Tables 1 through 8 below present a cross-tabulation of MHTSIG activities by NFC goal for entries scheduled to be completed in the 2007 to 2009 fiscal years and for each of the years separately. Each activity item can be assigned to only one GPRA category but may be assigned to multiple NFC goals. Data in these tables as well as Tables 9 through 11 were extracted from Transformation Tracker on October 5, 2009. The individual percentages in these tables may not total 100 percent due to rounding. Appendix A provides target activity information by State and year, while Appendix B presents activity results by State and year. All tables that describe GPRA and NFC goal activity in this section and in Appendices A and B are provisional and are intended to illustrate the analyses that will eventually be implemented for the summative cross-site evaluation. A primary reason for caution about the data in these tables is that, as of yet, the data have not been “cleaned” (i.e., they have not been reviewed to ensure the accuracy of coding or the presence of duplicate activities). In addition, the GPRA and NFC tables within this report do not reflect State activities for the 2006 baseline year because the grantees were not required to report GPRA data until 2007. At least one grantee State, however, has entered its baseline MHTSIG activities into Transformation Tracker. Tables 1 through 8 show numbers and percentages of GPRA activity items by NFC goal. If these data are accurately coded by the States, the most frequently addressed NFC goals in 2007, 2008, and 2009 were NFC 2 (care is consumer and family driven) and NFC 5 (excellent care is provided), while the least frequently addressed ones were NFC 3 (disparities are eliminated) and NFC 6 (technology is used). They also show that most State activities were classified under GPRA 1 (policy changes) and GPRA 2 (training activities). Currently a total of 1,076 unique MHTSIG activity items have been entered in Transformation Tracker as ending in fiscal years 2007 to 2009 across all States). As mentioned previously, this count does not reflect baseline year activities. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 6 Table 1. 2007-2009 Number of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 82 72 23 60 79 37 GPRA 2: Training 101 147 57 78 118 15 GPRA 3: Financial Policy Changes 43 41 20 32 53 19 GPRA 4: Organizational Changes 47 55 19 47 44 17 GPRA 5: Data Sharing 8 17 3 11 23 13 GPRA 6: Consumer, Youth, & Family Networks 3 29 0 0 2 0 GPRA 7: CMHP Practices 63 34 18 44 54 20 Non-GPRA 22 53 12 14 32 16 Total Number 369 448 152 286 405 137 Percent of Total Activities 21% 25% 8% 16% 23% 8% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 7 Table 2. 2007-2009 Percentage of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 22% 16% 15% 21% 20% 27% GPRA 2: Training 27% 33% 38% 27% 29% 11% GPRA 3: Financial Policy Changes 12% 9% 13% 11% 13% 14% GPRA 4: Organizational Changes 13% 12% 13% 16% 11% 12% GPRA 5: Data Sharing 2% 4% 2% 4% 6% 10% GPRA 6: Consumer, Youth, & Family Networks 1% 6% 0% 0% 1% 0% GPRA 7: CMHP Practices 17% 8% 12% 15% 13% 15% Non-GPRA 6% 12% 8% 5% 8% 12% Total Number 100% 100% 100% 100% 100% 100% Percent of Total Activities 21% 25% 8% 16% 23% 8% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 8 Table 3. 2007 Number of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 13 25 4 13 17 8 GPRA 2: Training 4 7 4 4 9 0 GPRA 3: Financial Policy Changes 9 9 3 8 11 5 GPRA 4: Organizational Changes 5 7 2 5 6 3 GPRA 5: Data Sharing 1 1 0 1 0 2 GPRA 6: Consumer, Youth, & Family Networks 0 3 0 0 1 0 GPRA 7: CMHP Practices 6 2 1 4 3 1 Non-GPRA 0 2 1 1 1 5 Total Number 38 56 15 36 48 24 Percent of Total Activities 18% 26% 7% 17% 22% 11% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 9 Table 4. 2007 Percentage of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 34% 45% 27% 36% 35% 33% GPRA 2: Training 11% 13% 27% 11% 19% 0% GPRA 3: Financial Policy Changes 24% 16% 20% 22% 23% 21% GPRA 4: Organizational Changes 13% 13% 13% 14% 13% 13% GPRA 5: Data Sharing 3% 2% 0% 3% 0% 8% GPRA 6: Consumer, Youth, & Family Networks 0% 5% 0% 0% 2% 0% GPRA 7: CMHP Practices 16% 4% 7% 11% 6% 4% Non-GPRA 0% 4% 7% 3% 2% 21% Total Number 100% 100% 100% 100% 100% 100% Percent of Total Activities 18% 26% 7% 17% 22% 11% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 10 Table 5. 2008 Number of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 26 23 12 22 29 18 GPRA 2: Training 42 76 29 39 57 10 GPRA 3: Financial Policy Changes 11 17 10 14 29 11 GPRA 4: Organizational Changes 19 23 8 18 17 4 GPRA 5: Data Sharing 1 8 2 4 8 5 GPRA 6: Consumer, Youth, & Family Networks 2 16 0 0 0 0 GPRA 7: CMHP Practices 18 10 7 14 17 7 Non-GPRA 12 30 7 6 16 9 Total Number 131 203 75 117 173 64 Percent of Total Activities 17% 27% 10% 15% 23% 8% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 11 Table 6. 2008 Percentage of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 20% 11% 16% 19% 17% 28% GPRA 2: Training 32% 37% 39% 33% 33% 16% GPRA 3: Financial Policy Changes 8% 8% 13% 12% 17% 17% GPRA 4: Organizational Changes 15% 11% 11% 15% 10% 6% GPRA 5: Data Sharing 1% 4% 3% 3% 5% 8% GPRA 6: Consumer, Youth, & Family Networks 2% 8% 0% 0% 0% 0% GPRA 7: CMHP Practices 14% 5% 9% 12% 10% 11% Non-GPRA 9% 15% 9% 5% 9% 14% Total Number 100% 100% 100% 100% 100% 100% Percent of Total Activities 17% 27% 10% 15% 23% 8% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 12 Table 7. 2009 Number of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 43 24 7 25 33 11 GPRA 2: Training 55 64 24 35 52 5 GPRA 3: Financial Policy Changes 23 15 7 10 13 3 GPRA 4: Organizational Changes 23 25 9 24 21 10 GPRA 5: Data Sharing 6 8 1 6 15 6 GPRA 6: Consumer, Youth, & Family Networks 1 10 0 0 1 0 GPRA 7: CMHP Practices 39 22 10 26 34 12 Non-GPRA 10 21 4 7 15 2 Total Number 200 189 62 133 184 49 Percent of Total Activities 25% 23% 8% 16% 23% 6% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 13 Table 8. 2009 Percentage of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 22% 13% 11% 19% 18% 23% GPRA 2: Training 28% 34% 39% 26% 28% 10% GPRA 3: Financial Policy Changes 12% 8% 11% 8% 7% 6% GPRA 4: Organizational Changes 12% 13% 15% 18% 11% 20% GPRA 5: Data Sharing 3% 4% 2% 5% 8% 12% GPRA 6: Consumer, Youth, & Family Networks 1% 5% 0% 0% 1% 0% GPRA 7: CMHP Practices 20% 12% 16% 20% 19% 25% Non-GPRA 5% 11% 7% 5% 8% 4% Total Number 100% 100% 100% 100% 100% 100% Percent of Total Activities 25% 23% 8% 16% 23% 6% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 14 As noted earlier, the Transformation Tracker activity items are coded by NFC goal(s), and additional information is provided about the history and current status of each activity. Tables 9, 10, and 11 below summarize the most important historical information associated with the NFC activity items in 2007, 2008, and 2009, respectively; each table presents, by NFC goal, the number of activities, the percentage of States with those activities, the percentage of activities representing that goal across the States (presented as a range), the percentage begun before the MHT SIG grant, the percentage raised in the Needs Assessment and Resource Inventory (NARI), the percentage noted in the CMHP, the percentage recommended by the TWG, and the percentage with results noted in Transformation Tracker. If these data are accurately coded by the States, the characteristics of activity items associated with the NFC goals have changed over time. For example, in 2007, activity items associated with NFC 2 (care is consumer and family driven) were less likely to be recommended by TWGs than activity items associated with other NFC goals. In 2008, a greater proportion of NFC 2 activity items had been TWG-approved, though these items were still approved by the TWG at a lower rate than activities related to other NFC goals. Interestingly, Goal 2 activities in 2009 had a much higher approval rate than in the previous 2 years, giving these items a TWG approval rate consistent with those for other NFC goal activities. One possible explanation is that earlier activity items had not been presented to TWGs at the time the activity items were entered into Transformation Tracker, and that this information has not yet been updated. The cross-site evaluation team will investigate the reason for this pattern when conducting the data cleaning for the summative evaluation. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 15 Table 9. Summary of 2007 Transformation Activities by New Freedom Commission Goals NFC Goals Number of Activities Percent of States Percent Range of Activity Items Percent Begun Before MHT SIG [Yes] Percent Begun Before MHT SIG [No] Percent Raised in NARI [Yes] Percent Raised in NARI [No] Percent Noted in CMHP [Yes] Percent Noted in CMHP [No] Percent Recomm by TWG [Yes] Percent Recomm by TWG [No] Percent of Activities With Activity Results NFC 2: Care Is Consumer/ Family Driven 54 89% 0-31% 19% 81% 61% 39% 80% 20% 50% 50% 96% NFC 5: Excellent Care Is Provided 48 89% 0-31% 19% 81% 88% 13% 94% 6% 73% 27% 98% NFC 1: Mental Health Is Essential 38 67% 0-50% 18% 82% 82% 18% 89% 11% 82% 18% 95% NFC 4: Early Screening Is Provided 36 67% 0-39% 25% 75% 83% 17% 97% 3% 64% 36% 94% NFC 6: Technology Is Used 23 67% 0-35% 9% 91% 96% 4% 96% 4% 83% 17% 83% NFC 3: Disparities Are Eliminated 15 56% 0-60% 0% 100% 87% 13% 93% 7% 87% 13% 93% Total 214 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A Average per Goal 35.7 N/A N/A 15% 85% 83% 17% 92% 8% 73% 27% 93% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 16 Table 10. Summary of 2008 Transformation Activities by New Freedom Commission Goal NFC Goals Number of Activities Percent of States Percent Range of Activity Items Percent Begun Before MHT SIG [Yes] Percent Begun Before MHT SIG [No] Percent Raised in NARI [Yes] Percent Raised in NARI [No] Percent Noted in CMHP [Yes] Percent Noted in CMHP [No] Percent Recomm by TWG [Yes] Percent Recomm by TWG [No] Percent of Activities With Activity Results NFC 2: Care Is Consumer/ Family Driven 194 100% 1-25% 19% 81% 74% 26% 87% 13% 57% 43% 93% NFC 5: Excellent Care Is Provided 177 89% 0-22% 25% 75% 78% 22% 93% 7% 77% 23% 90% NFC 1: Mental Health Is Essential 133 89% 0-31% 27% 73% 75% 25% 92% 8% 78% 22% 88% NFC 4: Early Screening Is Provided 118 89% 0-32% 30% 70% 86% 14% 92% 8% 86% 14% 86% NFC 3: Disparities Are Eliminated 76 89% 0-29% 29% 71% 74% 26% 100% 0% 70% 30% 79% NFC 6: Technology Is Used 66 89% 0-24% 12% 88% 86% 14% 98% 2% 89% 11% 86% Total 764 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A Average per Goal 127.3 N/A N/A 24% 76% 79% 21% 94% 6% 76% 24% 87% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 17 Table 11. Summary of 2009 Transformation Activities by New Freedom Commission Goal NFC Goals Number of Activities Percent of States Percent Range of Activity Items Percent Begun Before MHT SIG [Yes] Percent Begun Before MHT SIG [No] Percent Raised in NARI [Yes] Percent Raised in NARI [No] Percent Noted in CMHP [Yes] Percent Noted in CMHP [No] Percent Recomm by TWG [Yes] Percent Recomm by TWG [No] Percent of Activities With Activity Results NFC 1: Mental Health Is Essential 199 100% 2-34% 12% 88% 75% 25% 91% 9% 85% 15% 82% NFC 2: Care Is Consumer/ Family Driven 185 89% 0-23% 23% 77% 73% 27% 91% 9% 71% 29% 65% NFC 5: Excellent Care Is Provided 184 100% 2-21% 23% 77% 81% 19% 89% 11% 81% 19% 72% NFC 4: Early Screening Is Provided 132 89% 0-36% 20% 80% 77% 23% 89% 11% 75% 25% 64% NFC 3: Disparities Are Eliminated 62 100% 3-21% 19% 81% 79% 21% 95% 5% 87% 13% 65% NFC 6: Technology Is Used 49 89% 0-27% 12% 88% 86% 14% 92% 8% 92% 8% 76% Total 811 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A Average per Goal 135.2 N/A N/A 18% 82% 78% 22% 91% 8% 82% 18% 71% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 18 Proof of Concept Studies The status of the recovery and resiliency studies (as of September 14, 2009) is shown in Tables 12 and 13. For both types of studies, each State is required to enroll 75 persons in the impacted group and 75 in the nonimpacted group. Overall, across impacted and nonimpacted groups, the first cohort of States had completed baseline data collection with 68 percent of the enrollment target for their recovery studies and 51 percent for their resiliency studies by the end of the third year. Table 12. Status of Recovery Studies State Start Date Consents Past Week Declines Past Week Interviews Past Week Consents to Date Declines to Date Interviews to Date Connecticut Impacted 7/09 Not available Not available Not available Not available Not available 75 Connecticut Nonimpacted 7/09 Not available Not available Not available Not available Not available 75 Maryland Impacted 5/09 N/A 0 2 N/A 66 51 Maryland Nonimpacted 5/09 N/A 0 0 N/A 15 27 New Mexico Impacted 5/09 N/A N/A N/A 10 N/A 10 New Mexico Nonimpacted 6/09 N/A N/A N/A 17 N/A 17 Ohio Impacted Complete N/A N/A N/A N/A N/A 751 Ohio Nonimpacted Complete N/A N/A N/A N/A N/A 752 Oklahoma Impacted 2/09 9 0 9 70 0 70 Oklahoma Nonimpacted 2/09 11 0 11 69 0 69 Texas Impacted 4/09 4 0 2 74 2 35 Texas Nonimpacted 4/09 N/A N/A 2 Not available Not available 25 Washington Impacted 8/09 8 4 8 14 9 14 Washington Nonimpacted 5/08 0 0 0 98 Not available 983 Total (Percent of Target Number) N/A N/A N/A N/A N/A N/A 716 (68%) 1Final number may be more than 75. 2Final number may be more than 75. 3Sequential design: This is followup data collection. Baseline data collection was completed 7/08. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 19 Table 13. Status of Resiliency Studies4 Site Start Date Consents Past Week Declines Past Week Interviews Past Week Consents to Date Declines to Date Interviews to Date Connecticut Impacted 6/09 0 0 0 0 0 0 Connecticut Nonimpacted 6/09 0 0 0 0 0 0 Maryland Impacted 1/09 0 0 0 79 22 50 Maryland Nonimpacted 1/09 0 0 1 35 17 28 New Mexico Impacted 6/08 N/A N/A N/A 48 N/A 48 New Mexico Nonimpacted 6/08 N/A N/A N/A N/A N/A 0 Ohio Impacted Complete N/A N/A N/A N/A N/A 751 Ohio Nonimpacted Complete N/A N/A N/A N/A N/A 751 Oklahoma Impacted Complete 9 N/A 9 78 N/A 781 Oklahoma Nonimpacted Complete 8 N/A 8 78 N/A 781 Texas Impacted 4/09 Not available Not available Not available 83 10 71 Texas Nonimpacted 4/09 Not available Not available Not available 24 5 16 Washington Impacted 3/09 0 0 0 39 13 37 Washington Nonimpacted 3/09 0 0 0 23 7 23 Total (Percent of Target Number) N/A N/A N/A N/A N/A N/A 579 (51%) Beginning in April 2009, State evaluators were asked to provide updates on the status of their POC studies on each monthly Evaluation Work Group call. By June 2009, it became apparent that at least several sites would likely not reach the recruitment target of 150 participants for their POC studies by the mid-August deadline (the last possible date that would allow a 1-year followup by the project’s end). Accordingly, the evaluation team responded in two ways: 1. By providing additional monitoring and support for the sites. In late July and early August, a series of calls with sites at risk of not reaching the full complement of 150 participants (i.e., Connecticut, Maryland, Oklahoma, New Mexico, Texas, Washington) were held to determine the current status of enrollment, the nature of the difficulties encountered, and possible strategies for overcoming them. In general, delays had resulted from two factors: (1) The interventions were not implemented as planned by the State mental health agency, and (2) there had been a lack of responsiveness on the part of State treatment programs, especially those representing nonimpacted sites, to refer clients to the POC studies. 4 Final number may be more than 75. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 20 At-risk States were called to ask whether they intended to request a no-cost extension to allow sufficient time to enroll the full number of participants and preserve the 1-year followup period. None of the States indicated that they planned to make such a request. The project team provided more focused monitoring and support for the sites that appeared to be having the most difficulty (e.g., Maryland, New Mexico, Texas) to identify the nature of the obstacles they were encountering and explore means of overcoming them. 2. By exploring analytic strategies to address the possible shortfall in the number of study participants and the potential to have a shorter followup period. As it became apparent that some POC studies were unlikely to reach the target of 150 persons by the deadline of mid-August, the cross-site evaluation team began to explore analytic strategies to address the possibility of having fewer than 150 participants for each study. The project’s statistical consultant conducted further power analyses and found that 100 participants per site would provide adequate power. It appears that most, if not all, of the sites will achieve this level of enrollment. However, this approach will require further consideration if the number of participants in each study is not fairly equally balanced between impacted and nonimpacted groups, which appears likely to be the case in some States. The second approach, having a followup period of less than 1 year (e.g., 9 months) was examined to determine whether the various interventions being studied and measures being used would permit a dose-effect analysis whereby it would be possible to identify a trend line representing outcomes as a function of time of exposure to the intervention. A preliminary review of the interventions and the measures being used in the POC studies indicated that this was in fact the case—the effects of the interventions may be expected to be incremental and the outcomes are measured as continuous variables. The Project Officer set a final deadline for baseline data collection of November 1, 2009, based on guidance from the evaluators that this date would provide the optimum balance between number of participants and length of followup period. Subsequently, several States expressed various concerns about this deadline, such as additional burden related to data collection and Institutional Review Board (IRB) modifications as well as methodological issues. These concerns were addressed through email and telephone discussion with the evaluation team. Both of these approaches will be explored more thoroughly in the coming year as baseline data become available. Table 14 provides an overview of the status of POC studies by State (as of September 14, 2009), including each State’s plan to request a no-cost extension. Appendix C provides more detailed information about each of the State POC studies, including the impacted and nonimpacted groups, the GPRA indicators and NFC goals being addressed, and the survey instruments being used. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 21 Table 14. Status of State Proof of Concept Studies Site (Intervention) Status of Data Collection Date Expect To Complete Baseline Data Collection Problems Encountered/ Actions To Expedite Potential Problems Method Issues/Help Requested Plans to Request No-Cost Extension Connecticut Recovery (Staff trained in individual recovery planning) IRB approval obtained, data collection completed Impacted and nonimpacted interviews completed IRB delays— now complete None anticipated (based on previous projects with same clinicians and consumers) None at this time Not expected Connecticut Resilience (Wraparound in juvenile justice) Final IRB obtained in September 2009, requiring only sign-off; data collection to begin immediately End of October 2009 IRB questions— now resolved None anticipated, if recruitment proceeds as rapidly as was the case with adult study 1. “More vs. less impacted” (i.e., subject exposure varies at baseline— admin data, and site fidelity measure may allow adjustment 2. Possibly shortened timeframe (11 months) 3. Type of sites vary (school, probation, welfare) No, if shorter followup period is acceptable Hawaii Recovery (Network of Care/WRAP Self-management) Proposal submitted to cross-site evaluation team, working with the sites on logistics Between December 2009 and February 2010 Delays in State Mental Health Agency (SMHA) implementation of programs— meetings with SMHA and Project Directors None additional at this time Questions about follow up period—advised to conform to protocol of 1-year follow up Don’t know Hawaii Resilience (Under development) Partners and sites selected, study details being worked out, expect to submit proposal in October 2009 Unknown at this time Difficulty obtaining cooperation of SMHA—delayed implementation None additional at this time None at this time Don’t know MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 22 Site (Intervention) Status of Data Collection Date Expect To Complete Baseline Data Collection Problems Encountered/ Actions To Expedite Potential Problems Method Issues/Help Requested Plans to Request No-Cost Extension Maryland Recovery (Cultural competence training) Underway; have IRB approval to extend to end of October 2009 End of October 2009 High refusal and bad contact information— adjustments made None additional at this time Initial concern about obtaining diagnosis, but will be available from administrative data Hope not Maryland Resilience (Wraparound) Consent to be contacted forms completed, continuing to follow up November 1, 2009; however, probably will have fewer than 150 participants Fewer than expected receiving services—staff have been very active contacting sites Need to complete before other grants start up Possibility of having less than 150 participants More time will not help, as subject pool is limited and need to complete before new grants begin Missouri Recovery (Peer specialists) POC proposal submitted to cross-site evaluators, sites to be determined, expect to start data collection in January 2010 Uncertain N/A None at this time None at this time Don’t know Missouri Resilience (Family support specialist proposal is under development) Still in planning stage, submission to cross-site evaluators has not been made; expect to start data collection in November/ December 2009 Uncertain N/A None at this time None at this time Don’t know MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 23 Site (Intervention) Status of Data Collection Date Expect To Complete Baseline Data Collection Problems Encountered/ Actions To Expedite Potential Problems Method Issues/Help Requested Plans to Request No-Cost Extension New Mexico Recovery (Integrated Dual Diagnosis Treatment) Underway but proceeding very slowly November 1, 2009; however, probably will have fewer than 150 participants In July, State Managed Care Organization (MCO) change; MCO was not contacting sites, turmoil at impact site—Met with Deputy Secretary, site incentives increased, more education, recruitment now bypassing MCO and going directly to sites No additional anticipated, expect to proceed more smoothly None at this time Don’t know New Mexico Resilience (Clinical home) Underway for a year but proceeding very slowly, already doing some followup interviews though baseline incomplete November 1, 2009; however, probably will have fewer than 150 participants Same as above plus turmoil at impact site— Deputy Secretary authorized five new sites Same as above Though still collecting baseline data, started in June 2008 and some are due for followup—will be maintaining 1-year interval for now Don’t know Texas Recovery (Self-directed care— control funds) Proceeding slowly November 1, 2009; however, probably will have fewer than 150 participants Difficulty with provider referrals, especially nonimpacted— doing more outreach and modified enrollment criteria None additional at this time None at this time Don’t know MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 24 Site (Intervention) Status of Data Collection Date Expect To Complete Baseline Data Collection Problems Encountered/ Actions To Expedite Potential Problems Method Issues/Help Requested Plans to Request No-Cost Extension Texas Resilience (Trauma cognitive behavioral therapy) Proceeding slowly November 1, 2009; however, probably will have fewer than 150 participants Obtaining referrals from nonimpacted sites Imbalance in number of participants with much smaller number of nonimpacted None at this time Don’t know Washington Recovery (Wellness) Sequential design: nonimpacted started June 2008, completed September 2009 Impacted baseline will start in September 2009 None at this time None at this time None at this time Hope not Washington Resilience (School-based services) Only partial numbers obtained by end of school year, will continue beginning 2009 school year End of October 2009 IRB modification was approved to allow default parent contact (opt-out letter) None at this time 1. 6-month interval between two baseline groups 2. Possible shortened timeframe Hope not MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 25 State Site Visits During the site visits, State agency directors were asked to rate the grant’s impact on their agency using a 10-point scale from 1 (lowest) to 10 (highest). Charts 1 and 2 illustrate ratings by State and type of State agency, respectively. (The States were randomly coded to maintain their anonymity.) Three States (States 4, 5, and 9) had the highest ratings of 7 or more, while two States (States 3 and 7) had the lowest ratings of 5 or less. Agency directors were asked to explain the rationale for their rating; their responses indicate that they used different approaches in assigning their rating. For example, some State agency staff based their rating on progress completed to date, while others rated based on the potential impact of the grant. Regarding impact by agency type, mental health and juvenile justice agencies reported the most impact (with ratings 7 or above), while other agencies reported slightly lower levels of impact (ratings of around 6). About 10 percent of the agency directors did not provide any rating, often because they viewed an assessment of the grant’s impact as premature or had limited involvement with the grant. Figure 1. Average Rating of Grant Impact by State5 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 Individual States Average Impact Rating 5 States have been randomly coded. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 26 Figure 2. Average Rating of Grant Impact by Type of State Agency 1 2 3 4 5 6 7 8 9 10 Mental Health Criminal Justice Juvenile Justice Child Welfare Housing Vocational Rehab Medicaid Type of State Agency Average Impact Rating Federal Agency Site Visit As part of the Federal agency site visit, Federal staff (program managers and Connectors) were asked to rank NFC recommendations by order of importance. The results of this ranking exercise are shown in Table 15. Review of these rankings reveals that managers and Connectors share four of their six highest ranked recommendations and five of their seven lowest ranked recommendations, showing a relatively high level of agreement between the groups. The greatest levels of agreement between manager and Connector rankings are described below: Recommendation 2.2 (involve CFMs in orienting mental health systems toward recovery) is the highest ranked recommendation of both groups. Recommendations 1.1 (advance antistigma and antisuicide campaigns), 1.2 (address mental health with same urgency as physical health), and 4.1 (promote the mental health of young children) are in the six highest ranked recommendations of both groups. The following recommendations are in the seven lowest ranked recommendations of both groups: 3.2 (improve access in rural and geographically remote areas). 5.1 (accelerate research). 5.4 (develop the knowledge base in four understudied areas). 6.1 (use technology to improve access and coordination of care). 6.2 (develop electronic health records). The following recommendations are in the six mid-ranked recommendations of both groups (neither highest nor lowest ranked): 4.3 (screen for co-occurring disorders and link with integrated treatment). MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 27 5.2 (advance evidence-based practices and use public/private partnerships to advance). The greatest discrepancies between manager and Connector rankings are found for: Recommendation 2.4 (develop CMHP)—ranked low by managers (16) and high by Connectors (2). Recommendation 4.2 (improve and expand school mental health programs)—ranked low by managers (17) and in the middle by Connectors (7). Recommendation 4.4 (screen for mental health disorders in primary health care, across the lifespan, and connect to treatment)—ranked in the middle by managers (7) and low by Connectors (16). Recommendation 2.3 (align Federal programs to improve access and accountability)—ranked high by managers (4) and in the middle by Connectors (12). Table 15. Ranking of NFC Recommendations by MHT SIG Program Managers and Connectors6 6 Code for shading/font: Turquoise square/bold font indicates recommendation is ranked in top 6; yellow shading/italic font indicates recommendation is ranked in bottom 7. NFC Recommendations Average Rank for Managers Overall Rank for Managers Average Rank for Connectors Overall Rank for Connectors 1.1 Advance and implement a national campaign to reduce the stigma of seeking care and a national strategy for suicide prevention 7.7 5 8.3 5 1.2 Address mental health with the same urgency as physical health 4.3 2 6.2 4 2.1 Develop an individualized plan of care for every adult with a serious mental illness and child with a serious emotional disturbance 11.5 12 11.8 14 2.2 Involve consumers and families fully in orienting the mental health system toward recovery 1.3 1 4.2 1 2.3 Align relevant Federal programs to improve access and accountability for mental health services 7.3 4 10.7 12 2.4 Create a Comprehensive State Mental Health Plan 13.3 16 4.5 2 2.5 Protect and enhance the rights of people with mental illnesses 8.7 8 8.3 5 3.1 Improve access to quality care that is culturally competent 6.3 3 9.5 8 3.2 Improve access to quality care in rural and geographically remote areas 12.3 14 11.5 13 4.1 Promote the mental health of young children 7.7 5 4.7 3 4.2 Improve and expand school mental health programs 15.3 17 8.8 7 4.3 Screen for co-occurring mental and substance use disorders and link with integrated treatment strategies 11.0 11 10.0 9 MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 28 Consumer, Youth, and Family Member Focus Groups Focus groups were conducted with convenience samples of CFMs; these samples should not be considered representative of all CFM perspectives across all grantee States. Of the 130 persons who participated in the focus groups: 101 participants were consumers (17 were youths, 77 were adults, and 7 were older adults). 39 participants were family members of youth consumers, and 62 were family members of adult consumers. A total of 40 persons identified as being both a consumer and family member. Involvement was greater at the State level (101 participants, or 78 percent) than at the local level (75 participants, or 58 percent). 107 participants (82 percent) reported being members of CFM stakeholder organizations. Tables 16 and 17 provide information about the frequency and type of CFM involvement in State and local activities and the involvement of CFMs in specific types of grant activities. In both these tables, the individual States have been randomly coded. Table 16 shows CFM ratings for the frequency and type of CFM involvement at the State and local level for individual States and for all States combined. Overall, the frequency of consumer and family member involvement was viewed as “occasionally” and their type of involvement as “minor” at both the State and local levels. 4.4 Screen for mental disorders in primary health care, across the lifespan, and connect to treatment and supports 8.0 7 12.8 16 5.1 Accelerate research to promote recovery and resilience, and ultimately to cure and prevent mental illnesses 15.3 17 15.7 19 5.2 Advance evidence-based practices using dissemination and demonstration projects and create a public-private partnership to guide their implementation 10.0 10 10.3 10 5.3 Improve and expand the workforce providing evidence-based mental health services and supports 9.5 9 10.3 10 5.4 Develop the knowledge base in four understudied areas: mental health disparities, long-term effects of medications, trauma, and acute care 12.7 15 14.5 17 6.1 Use health technology and telehealth to improve access and coordination of mental health care, especially for Americans in remote areas or in underserved populations 11.7 13 12.3 15 6.2 Develop and implement integrated electronic health record and personal health information systems 15.7 19 15.5 18 MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 29 Table 16. Average CYF Rating of Frequency and Type of Consumer, Youth, and Family Member Involvement in Grant Activities7 Consumer/Youth/Family Involvement State 1 State 2 State 3 State 4 State 5 State 6 State 7 State 8 State 9 Frequency of Involvement8 in Local Activities 2.3 1.7 1.5 1.7 1.4 1.9 1.7 1.4 1.4 1.8 Occasionally Type of Involvement9 in Local Activities 1.2 1.1 0.6 1.1 0.7 1.3 1.3 0.7 0.9 0.9 Minor Role Frequency of Involvement10 in State Activities 1.9 2.1 0.7 2.1 1.0 1.4 1.9 1.1 1.2 1.6 Occasionally Type of Involvement11 in State Activities 1.1 1.1 0.5 1.1 0.5 1.1 1.2 0.5 1.0 0.8 Minor Role 7States have been randomly coded. 8Never = 0, Rarely = 1, Occasionally = 2, Frequently = 3, Almost always = 4. 9No role = 0, Minor role = 1, Major role = 2. 10Never = 0, Rarely = 1, Occasionally = 2, Frequently = 3, Almost always = 4. 11No role = 0, Minor role = 1, Major role = 2. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 30 Focus group participants were also asked to rate their level of agreement with 35 statements about their involvement in grant activities. Table 17 shows the average ratings by focus group participants for these statements by individual States and for all States combined. For the purpose of describing ratings for all States combined, the ratings have been grouped into the following categories: 1 to 2.7 (disagree), 2.8 to 3.2 (generally neither agree nor disagree), and 3.3 to 5 (agree). Across all States, consumers and family members generally agreed with the following statements: During project meetings, the opinions of CFMs are discussed. Involvement of CFMs has made a difference. The grant promotes collaboration among CFMs. Disagreements are handled respectfully within this project. Efforts are being made to evaluate CFM involvement. Stigma/discrimination is not accepted at any level of this grant. Grant staff are able to work collaboratively with CFMs. Mental health service users and their family members are positively affected by this grant. Persons of all cultural and ethnic origins are respected within this grant. Across all States, consumers and family members generally disagreed with the following statements: There are enough CFMs involved in the mental health grant. CFMs have meaningful involvement in decisions on funding of State programs. CFMs have meaningful involvement in decisions on funding of local programs. CFMs have meaningful involvement in making State policy. CFMs receive the training and support they need to participate effectively in the mental health transformation grant. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 31 Table 17. Average CFY Ratings regarding Various Aspects of CFY Participation in MHT SIG Grant Activities1213 Statements State 1 State 2 State 3 State 4 State 5 State 6 State 7 State 8 State 9 Overall Average Rating14 1. CFMs have meaningful involvement in: Setting local goals 4.1 2.3 2.1 3.7 2.8 3.6 3.9 2.1 2.4 3.0 Neither Making local policy 3.7 2.3 1.8 3.5 2.5 3.3 3.8 2.0 2.1 2.8 Neither Designing local programs 3.8 2.8 2.0 3.3 2.5 3.3 3.0 2.2 2.6 2.9 Neither Implementing local programs 3.8 2.8 2.2 3.3 2.3 3.6 2.9 2.5 2.3 2.9 Neither Evaluating local programs 3.8 2.5 2.0 3.4 2.2 3.6 2.9 2.5 2.4 2.8 Neither Decisions on funding of local programs 3.3 2.3 1.6 3.4 2.4 2.9 2.8 1.9 2.4 2.5 Disagree Setting State goals 3.6 3.2 2.2 3.2 2.3 3.2 3.6 2.2 2.9 2.9 Neither Making State policy 3.2 2.6 1.8 2.8 1.9 2.8 3.4 2.0 2.8 2.6 Disagree Designing State programs 3.5 3.0 2.0 3.1 2.1 3.0 3.2 1.9 2.9 2.8 Neither Implementing State programs 3.5 3.4 1.8 3.7 2.1 2.8 3.7 1.7 2.9 2.8 Neither Evaluating State programs 3.8 2.8 2.0 3.3 2.1 2.9 3.8 1.8 3.1 2.8 Neither Decisions on funding of State programs 3.1 2.4 1.5 2.8 1.7 2.1 3.4 1.8 2.5 2.4 Disagree 2. There are enough CFMs involved in the mental health grant 3.2 2.8 1.4 3.3 1.1 1.8 3.2 1.2 2.0 2.2 Disagree 12 States have been randomly coded. 13 Strongly disagree = 1, Disagree = 2, Neither = 3, Agree = 4, Strongly Agree = 5. 14The ratings were grouped into the following categories: 1 to 2.7 (disagree), 2.8 to 3.2 (generally neither agree nor disagree), and 3.3 to 5 (agree). MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 32 Statements State 1 State 2 State 3 State 4 State 5 State 6 State 7 State 8 State 9 Overall Average Rating14 3. Mental health service users and their family members are positively affected by this grant 4.2 3.9 2.9 2.7 2.8 3.8 3.8 2.0 3.2 3.3 Agree 4. CFMs are adequately compensated for their roles 3.9 3.4 1.8 3.4 1.7 3.8 3.3 1.2 3.1 2.9 Neither 5. Involvement of CFMs has made a difference 4.2 4.2 3.9 2.7 2.6 4.1 3.8 2.3 3.3 3.5 Agree 6. Grant staff are able to work collaboratively with CFMs 4.5 4.2 3.3 2.9 2.0 4.0 4.0 1.7 3.5 3.4 Agree 7. The grant promotes collaboration among CFMs 4.4 4.1 3.0 2.9 2.6 3.6 3.9 2.4 3.5 3.5 Agree 8. State leaders are sensitive to cultural and linguistic issues 4.1 3.6 2.6 2.8 1.6 3.5 3.4 2.5 3.0 3.1 Neither 9. CFMs receive the training and support they need to participate effectively in the mental health transformation grant 3.6 3.3 2.0 2.2 1.4 3.1 3.3 1.9 2.8 2.7 Disagree 10. The grant promotes CFM understanding of the process behind developing policy 4.0 3.1 2.0 2.8 1.6 3.4 3.6 1.6 3.0 2.8 Neither 11. The grant promotes CFM understanding of current policy issues 3.8 3.1 2.3 2.6 1.8 3.6 3.7 2.0 3.4 3.0 Neither 12. The leaders of this grant make involvement by CFMs a priority 4.1 3.9 1.8 2.6 2.3 4.1 3.9 2.1 3.6 3.3 Neither 13. During project meetings, the opinions of CFMs are discussed 4.4 4.2 3.3 3.1 3.0 4.0 4.0 2.0 3.5 3.6 Agree 14. Action is taken as a result of CFM opinions 4.2 4.0 2.6 2.4 1.9 3.9 3.7 1.9 3.0 3.1 Neither 15. A partnership exists between CFMs and persons who are not CFMs 4.2 3.4 2.9 2.7 2.2 3.8 3.4 1.7 3.5 3.1 Neither 16. As a result of the grant, CFMs have the knowledge to educate the community on important issues. 3.7 3.4 2.2 2.6 2.3 3.5 4.0 1.8 3.0 3.0 Neither MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 33 Statements State 1 State 2 State 3 State 4 State 5 State 6 State 7 State 8 State 9 Overall Average Rating14 17. The grant promotes CFMs to take the lead in this transformation project. 3.9 3.4 2.0 2.7 1.8 3.7 3.9 1.9 2.9 2.9 Neither 18. Disagreements are handled respectfully within this project. 4.1 4.2 2.6 3.2 2.5 3.8 4.3 2.4 3.0 3.5 Agree 19. Information about the mental health transformation grant is readily available to CFMs. 3.9 3.4 2.4 3.9 2.4 3.5 3.9 2.0 3.2 3.2 Neither 20. Professionals use language that is easily understood by nonprofessional participants. 3.7 3.6 2.4 3.6 2.7 3.5 4.0 2.9 2.4 3.2 Neither 21. Efforts are made to evaluate CFM involvement. 4.4 4.1 3.0 3.0 2.6 4.0 4.1 2.3 3.5 3.5 Agree 22. CFMs are excited about the progress of the grant. 4.0 3.7 2.2 2.9 2.0 3.8 3.6 1.8 3.2 3.2 Neither 23. Persons of all cultural and ethnic origins are respected within this grant. 4.2 3.6 2.9 3.0 2.4 3.5 3.6 2.6 3.3 3.3 Agree 24. Stigma/discrimination is not accepted at any level of this grant. 4.3 4.3 2.4 2.7 2.5 4.0 4.4 2.7 3.6 3.5 Agree MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 34 Across all States, CFMs neither agreed nor disagreed with 21 (60 percent) of the statements (see Chart 3). CFMs agreed with nine of the statements (26 percent) and disagreed with five others (14 percent). The lack of agreement or disagreement with most statements raises a question of the true level and nature of CFM involvement in grant activities. Figure 3. Average Ratings on Statements Provided by Consumer and Family Member Focus Group Participants Across All States National Outcome Measures Data State NOMs data for 2004 to 2008 were analyzed to determine the completeness of data by individual measures and summarize the findings. Appendix D provides the actual NOMs data by State. Data for three measures are in development and are not currently available: NOM 1 (decreased mental illness symptomatology), NOM 3 (decreased criminal justice system involvement), and NOM 9 (increased cost-effectiveness). It should be noted that States sometimes collect NOMs data using different methods and samples. A full understanding of how these data can be used in the Other State Data Analyses will depend on a careful analysis of how these data are collected by MHT SIG states. The missing State data by NOMs measure are: NOM 2: Percentage of adults who are employed: Maryland in 2004 and Missouri in 2006. NOM 4: Percentage of individuals living in a private residence: Maryland in 2004, Missouri in 2005 and 2006, and Ohio in 2004. NOM 5: Percentage of persons served by age (under 18 and age 65+) and race/ethnicity: New Mexico in 2004 (age data only). NOM 6: Mental health hospital utilization per 1,000: Ohio in 2004 to 2008. NOM 7: Percentage of adults and children/families indicating increased social support/connectedness: In 2004 and 2005, this was a developmental measure. For adults, 2006 0% 10% 20% 30% 40% 50% 60% 70% Neither Agree nor Disagree Disagree Agree Response Percent of Respondents MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 35 data are missing for Hawaii, Missouri, New Mexico, and Washington. For children/families, 2006 data are missing for Hawaii, Missouri, Ohio, and Washington. Other missing data on children/families include 2007 and 2008 data for Ohio and 2007 data for Oklahoma. NOM 8: Percentage of adults with positive reporting for outcomes: Missouri in 2004 and 2006. NOM 10: Four different measures are used (percentages of States implementing adult supported housing, adult supported employment, adult assertive community treatment, and therapeutic foster care for children and adolescents), and there are missing data for many of these measures. The adult supported employment data are most complete, followed by adult supported housing. State trends for NOMs 2, 4, 6, 7, and 8 are shown in Table 18. Six of nine grantee States reported positive changes over the time period for all NOMs measures except NOM 8 (positive reporting on outcomes by adults). Table 18. Number of MHT SIG Grantee States Reporting Increases, Decreases, and No Changes in Their NOMs Data from 2004 to 2008 National Outcome Measure Increase Decrease No Change Not Applicable NOM 2: Employment 6 1 2 0 NOM 4: Living in Private Residence 6 3 0 0 NOM 6: Inpatient Utilization Rate151 6 2 0 1 NOM 7: Social Support/Social Connectedness—Adults 6 1 2 0 NOM 7: Social Support/Social Connectedness—Child/Family 6 1 1 1 NOM 8: Positive Reporting About Outcomes—Adults 3 4 2 0 The analysis of NOM 5 data was done for age groups (under 18 and 65+) and race/ethnicity using Tables 43 and 44 from Appendix D. The results showed that: Five States had higher percentages of youth in treatment in 2008 than were represented in the State’s general population, while four States had lower percentages. It is possible that higher percentages are associated with greater access to mental health services for youth. During the period from 2004 to 2008, four States had increases in their percentages of youth in treatment, while the remaining five States had decreases. All nine States had lower percentages of persons age 65+ in treatment in 2008 than were represented in the State’s general population. For six States, 3 percent or fewer of persons in treatment were age 65+, while individuals aged 65+ represented 10 to15 percent of grantee State populations. For the period 2004 to 2008, all States but Ohio had decreases in their percentages of persons aged 65+ in their client population; Ohio’s percentage remained unchanged. All but one grantee State had a lower percentage of Whites in treatment in 2008 than was represented in the State’s general population. As noted in the technical evaluation plan, issues do exist regarding statewide comparison of NOMs data (e.g., comparable data periods and measurement across States, lack of year-to-year consistency within individual States). Until these issues are examined in more detail at the level of the individual NOMs for the full data collection period, the usefulness of the NOMs data in the cross-site evaluation will be unclear. 15A reduced inpatient utilization rate is a change in a positive direction. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 36 Analysis of State Employment Initiatives In January 2009, an analysis of State employment initiatives was completed. The findings are summarized as follows: All nine MHT SIG States have employment activities focused on consumer representation in the mental health care workforce. Employment opportunities for consumers include: Peer specialists Positions in consumer-operated services Research, evaluation, or quality improvement team members Consumer positions in non-mental health service agencies Consumer positions on Transformation Working Groups Consumer positions as liaisons with other consumers (e.g., in offices of consumer affairs, on consumer engagement teams) General positions open to anyone (e.g., clinicians, State administrators) In addition, some States are addressing competitive labor force participation in the following subject areas: Supported employment (Maryland and Ohio) Ticket to Work (Maryland and Ohio) Medicaid (Ohio, Texas, and Washington) Vocational rehabilitation system (Ohio, Texas, and Washington) Criminal justice system (Washington) Temporary Assistance for Needy Families (New Mexico) Working with employers (Texas) Individual Development Accounts (Texas) Employment assessment as part of treatment planning (New Mexico and Ohio) Consumer-run moving business (New Mexico) Transportation barriers (Texas) None of the States reported activities that involve working with U.S. Department of Labor initiatives such as One-Stop Career Centers. Leadership Survey During the third project year, the leadership survey was administered for the first time to leaders and raters in the first cohort of States. Leadership changes had already occurred in most of the States when the survey was administered. For some States, these changes could be categorized as significant. For instance, two States experienced gubernatorial changes involving political party transitions. For these two States, leadership was impacted at three levels: State (gubernatorial), project (TWG Chairperson), and project management (Project Director). These two States had Project Directors who were involved in leadership of the TWG in one way or another; one Project Director operated in dual roles (as Project Director and TWG Chairperson) and the other was involved in coordination related to the TWG. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 37 Across both cohorts, four States have individuals serving in the dual roles of Project Director and TWG Chairperson. Changes in Project Directors occurred in seven States (including the two States that experienced gubernatorial changes). Implementation of the survey was challenging in some respects due to these changes in leadership. A common issue was getting previous leaders to complete the survey. Another challenge was that leaders and raters had to rely on their long-term memory. Raters in particular were asked to complete the survey based on their observations of the previous leader’s leadership style; for some, this aspect of their State’s MHT SIG project was difficult to recall. Preliminary findings from the leadership survey include the following: Leaders indicated they frequently seek differing perspectives when solving problems, consider the moral and ethical consequences of their decisions, and express satisfaction when others meet expectations. They reported that once in a while, they are absent when needed, delay responding to urgent questions, and avoid making decisions. Raters indicated that leaders fairly often consider the moral and ethical consequences of their decisions, express satisfaction when the rater meets expectations, and treat the rater as an individual rather than a member of a group. Raters reported that once in a while, leaders avoid making decisions, delay in responding to urgent questions, and are absent when needed. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 38 Provider Survey The pilot survey was completed in all States except Hawaii. A total of 37 providers completed the survey, with 59 percent reporting they were aware of the grant. Table 19 shows the results of the pilot survey as of September 14, 2009. Knowledge about the grant most often came from State mental health agencies, provider meetings, and news releases/emails. Table 19. Results of Pilot Test of Provider Survey State Number Surveyed Percent Aware of Grant How Providers Heard About Grant Connecticut 2 100% State Department of Mental Health and Addiction Services; participation on TWG subcommittee Hawaii 0 N/A N/A Maryland 6 67% Governor’s press announcement; NAMI meetings; co-occurring disorder meetings Missouri 4 50% Public mental health authority; CEO of their organization who works on the grant project New Mexico 7 43% Networking; CEO of their organization; local collaborative; provider meetings Ohio 4 50% Federal Department of Health and Human Services and SAMHSA; State Department of Mental Health email Oklahoma 6 50% Oklahoma Department of Mental Health and Substance Abuse Services (ODMHSAS) Director meetings; ODMHSAS publications; email to community mental health centers Texas 5 80% Provider meetings and mental health/mental retardation collaborative; news clippings; Texas Department of State Health Services grant notification Washington 3 67% Grant implementation committee; email notification Total 37 59% N/A State Specific Evaluation Activities Based on a request from State evaluators, a listing of actual and planned State-specific evaluation activities (i.e., activities that were not part of the cross-site evaluation requirements) was developed in May 2009 through review of 2009 State continuation applications and reviews by State MHT SIG staff and evaluators. Tables 20 through 28 list the 128 different activities identified by the grantee State evaluators. The average number of activities in each State is 14, with range from 10 in Texas to 26 in Connecticut.16 The most common types of evaluation activities in the States are as follows: Antistigma efforts to determine impact or effectiveness (5 States) Mental Health First Aid (4 States) State surveys of evidence-based practices (4 States) 16 Note that Hawaii and Missouri received less MHT SIG funding than the States that were awarded grants in 2005. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 39 SUMMARY The third year of the MHT SIG cross-site evaluation project was productive. Site visits were completed in Hawaii and Missouri, with site visit reports to be completed early in year 4. Close coordination with the States has continued regarding their POC studies. The project team completed focus groups in all States (with associated reports completed for the first cohort of States), the leadership survey for the first cohort of States, and the pilot provider survey in all but one State. Other support provided to the MHT SIG grantees included launching and maintaining Transformation Tracker Explorer and participating in the annual grantee meeting. The cross-site evaluation team values the important work in the States to realize transformation of their mental health systems and meet New Freedom Commission goals. The team is enthusiastic about collaborating with the States and continuing the evaluation in the upcoming year. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 40 Table 20. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Mental Health Program/Services/Systems/Clients Topic CT HI MD MO NM OH OK TX WA Mental Health First Aid N N Y Y Y N Y N N State survey of evidence-based practices N N N Y Y N Y N Y Consumer surveys on recovery orientation N N N N Y N Y N Y Wraparound Fidelity Assessment System (WFAS) Y N Y N N N N N N Systems of care N N Y N N N Y N N Expanded school mental health program N N Y N Y N N N N Early Childhood Mental Health Consultation Evaluation N N Y N Y N N N N Evidenced-based practices fidelity and outcomes evaluation for a number of specific programs N N Y N Y N N N N Collection of standardized information on all MHT SIG projects N N N N Y N N Y N Evaluation of Program of Assertive Community Treatment (PACT) teams statewide N N N N N N Y N Y Indepth interviews with consumers, youth, and family members regarding the current mental health system and needed improvements N N N N N N Y N Y Consumer-, youth-, and family-driven Quality Improvement Collaborative to identify and implement recovery- and resilience-oriented performance measurements for the evaluation of the mental health service delivery models and programs Y N N N N N N N N Use of seclusion and restraint Y N N N N N N N N Behavioral Workforce Collaborative Y N N N N N N N N Interagency determination of questions about mental health curricula to add to CDC biannual School Health Profile Y N N N N N N N N Followup study of three Co-Occurring State Incentive Grant (CO-SIG) sites (in process) N Y N N N N N N N Community minigrants N Y N N N N N N N Department of Human Resources/Transformation Level of Intensity Evaluation— Development of tool to evaluate youth in group homes, treatment foster care, and independent living N N Y N N N N N N Project Connections, a community mental health outreach program for persons living in communities with high rates of poverty, violence, and instability N N Y N N N N N N RESPECT program N N N Y N N N N N Regional Behavioral Health Initiative activities regarding cultural competence/ RESPECT program N N N Y N N N N N Communities of Hope program (planned) N N N Y N N N N N State survey of provider needs N N N Y N N N N N Drop-in program analysis N N N Y N N N N N Behavioral health recovery studies N N N N Y N N N N Youth suicide prevention programming N N N N Y N N N N Hospital to community trauma-informed care study (planned) N N N N N Y N N N Statewide consumer focus group interviews for insight into perceptions of treatment planning and instrumentation development N N N N N Y N N N Access to mental health and substance abuse services N N N N N N Y N N MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 41 Topic CT HI MD MO NM OH OK TX WA Development and implementation of centralized electronic health record and reporting tool by local mental health authorities and substance abuse treatment providers N N N N N N N Y N Determination of the effectiveness of mental health infrastructure innovations and related factors using the Texas MHT Conceptual Model N N N N N N N Y N Development and assessment of use of Web-based comprehensive listing of evidence-based practices N N N N N N N Y N Service integration via use of liaison position to navigate consumers through local mental health authorities and primary care at Federally Qualified Health Centers N N N N N N N Y N General Assistance—Unemployable Clients: Challenges and Opportunities N N N N N N N N Y Washington’s Family Integrated Transition program N N N N N N N N Y Evaluation of King County Rethinking Care Pilot Project (to improve access to mental health services for Federally Qualified Health Center clients) N N N N N N N N Y Evaluation of statewide wraparound pilot for youth with serious emotional and behavioral disorders and their families N N N N n N N N Y Table 21. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Stigma Topic CT HI MD MO NM OH OK TX WA Evaluation of impact or effectiveness of statewide social marketing campaign Y Y N Y N N Y N Y Antistigma Y N Y N N N Y N N Stigma surveys N N N Y Y N Y N N Survey of provider organizations regarding stigma N N N N N N N N Y Table 22. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Funding & Resources Topic CT HI MD MO NM OH OK TX WA Medicaid 1915c home and community-based waiver Y N Y N N N N N N Expedited Medicaid review for prisoners study N N N N N N Y N Y NARI: Fiscal Resource Inventory Addendum, August 2008 N Y N N N N N N N Provider documentation project report N N N N N Y N N N Implementation of E2SSB 5763 (county option to impose 1/10 of 1% sales tax for mental health and other services) N N N N N N N N Y MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 42 Table 23. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Workforce and Training Topic CT HI MD MO NM OH OK TX WA Peer specialists (satisfaction survey) N N N Y Y N N N N Connecticut Recovery Employment Consultation Service (focuses on recruiting, preparing, and placing consumer providers within human service agencies) Y N N N N N N N N Providers and supervisors in outpatient and inpatient settings in person-centered, rehabilitative services, and trauma-informed care Y N N N N N N N N Community stakeholders (parents, teachers, behavioral health staff, and judicial staff) in wraparound practices and principles Y N N N N N N N N University professors and graduate students in intensive in-home family treatment Y N N N N N N N N Supervision competencies Y Nn N N N N N N N Department of Labor health and human services staff in recovery and mental health Y N N N N N N N N Adult consumers to become recovery supporters Y N N N N N N N N Community minigrants N Y N N N N N N N Family Engagement Training Program N N Y N N N N N N Cultural competence trainings N N Y N N N N N N Family support workers (satisfaction survey) N N N Y N N N N N Peer specialists (whether interaction between giver and recipient is beneficial) N N N Y N N N N N Workforce N N N N N N Y N N Training—Texas Collaborative for Emotional Development in Schools/Education Service Centers Train-the-Trainer N N N N N N N Y N Survey of persons who completed consumer peer-to-peer counselor training N N N N N N N N Y MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 43 Table 24. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding CYF, CYF Organizations, & Community Stakeholders Topic CT HI MD MO NM OH OK TX WA Community minigrants Y Y N N N N N N Y Surveys on consumer and family member perspectives about transformation efforts N N N Y Y Y N N N Network of Care Web site—Monitoring of perceptions/use of information (about mental health and substance abuse treatment and services as well as resources for persons across the lifespan) or effectiveness Y N N Y N N N N N Leadership training to parents Y N N Y N N N N N Assessment of level of consumer, youth, and family involvement in grant activities and of ways it can be enhanced throughout the service delivery system Y N N N N N N N N Evaluation of Consumer/Youth/Family Research and Evaluation Network (e.g., transformation involvement survey, minigrant projects, focus group information from mental health agency disparities initiative) Y N N N N N N N N Consumer-run organizations N N Y N N N N N N Community Evaluation and Research Training Academy for consumers and community stakeholders N N N N Y N N N N Consumer participation in Local Collaborative process N N N N Y N N N N State Annual Local Collaborative Self-Assessment Survey N N N N Y N N N N State Pre K-6 Parent Drug Prevention Programs N N N N Y N N N N Family advocacy focus group evaluation N N N N N Y N N N Consumer/family member involvement standards N N N N N N Y N N Consumer Network and Training—Training and Technical Assistance Center N N N N N N N Y N Consumer, youth, and family conference (satisfaction survey) N Y N Y Y N N N N MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 44 Table 25. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Disparities, Prevalence, & Statistics Topic CT HI MD MO NM OH OK TX WA Geographic disparities in mental health workforce N N N Y N N Y N Y Disparities in mental health services between urban and rural communities N N N N N N Y N Y Data interoperability to identify and eliminate behavioral health disparities Y N N N N N N N N State alcohol consumption and consequences annual survey N N N N Y N N N N State epidemiological drug use surveys N N N N Y N N N N Analysis of statewide database on family health and mental health N N N N N Y N N N Business case for trauma-informed intervention programming N N N N N Y N N N Analysis of utilization data for transition-age youth N N N N N Y N N N Analysis of the Youth Risk Behavior Survey N N N N N Y N N N Table 26. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Criminal/Juvenile Justice Topic CT HI MD MO NM OH OK TX WA Prison reentry study N N N N N N Y N Y Mental Health and Juvenile Justice Consultant Technical Report N Y N N N N N N N Community minigrants N Y N N N N N N N Juvenile justice supportive housing project pilot survey—adolescents N N N N Y N N N N Criminal justice (to determine whether released persons are getting necessary and quality services and have appropriate outcomes) N N N N N N Y N N Data sharing to promote jail diversion N N N N N N N Y N Service integration—court competency restoration program N N N N N N N Y N Arrests Among Working-Age Disabled (General Assistance—Unemployable) Clients evaluation N N N N N N N N Y Implementation of SSB 5533 (jail diversion) N N N N N N N N Y MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 45 Table 27. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Transformation Working Groups and Committees Topic CT HI MD MO NM OH OK TX WA Formative feedback to MHT SIG partners about progress to date and ways to better meet CMHP objectives Y N N N Y Y N N N Transformation Working Group—satisfaction with meetings, transformation process, engagement, and suggestions for continued transformation N N Y Y N Y N N N Monthly reporting template as part of formative evaluation of grant activities Y N N N Y N N N N Summary Evaluation of Sub-Working Group Process N Y N Y N N N N N Interviews and surveys with Content Working Groups, other agencies, and community constituents to evaluate overall transformation effectiveness and process of transformation efforts N N N Y N Y N N N Detailed evaluation reports on major initiatives of the Resource Investment and Strategy Subcommittee Y N N N N N N N N Summary Evaluation of Task Groups Process (draft developed) N Y N N N N N N N Local Collaborative Process Assessment Survey of Local Collaborative leaders N N N N Y N N N N MHT SIG Summative and Sustainability Report N N N N Y N N N N Table 28. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Other Topics Topic CT HI MD MO NM OH OK TX WA Psychiatric advance directives Y N N N N N N N N Policy analysis N N N N N N Y N N Technology Use Among Community Collaboratives (completed) N N N N N N N Y N Tracking of numbers of consumers, youth, and family members hired as a result of MHT SIG initiatives Y N N N N N N N N Pilot Housing Resilience Survey N N N N Y N N N N MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 46 Appendix A: GPRA Targets Table 29. Cumulative GPRA Target Numbers17 GPRA Measure FY CT HI MD MO NM OH OK TX WA Program 118 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline 1 2007 15 Baseline 0 Baseline 2 2 34 14 13 80 1 2008 33 0 5 20 21 63 46 64 27 279 1 2009 39 1 7 53 22 171 65 77 35 470 1 2010 41 1 8 71 27 223 78 81 48 578 1 2011 N/A 1 N/A 80 N/A N/A N/A N/A N/A 587 219 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline 2 2007 0 Baseline 2 Baseline 150 580 1,186 352 930 3,200 2 2008 1,780 179 159 1,143 152 6,908 8,243 13,884 6,261 38,709 2 2009 2,385 1,040 1,277 42,488 1,287 8,700 9,158 17,597 7,561 91,493 2 2010 2,505 1,229 1,886 44,741 3,516 10,420 11,004 26,196 17,596 119,093 2 2011 N/A 1,768 N/A 47,346 N/A N/A N/A N/A N/A 122,237 320 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline 3 2007 0 Baseline 1 Baseline 17 0 11 15 3 47 3 2008 0 0 5 31 19 6 15 33 7 116 3 2009 2 0 6 42 19 7 21 56 9 162 3 2010 2 0 7 50 25 10 27 65 30 216 3 2011 N/A 0 N/A 53 N/A N/A N/A N/A N/A 219 421 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline 4 2007 1 Baseline 0 Baseline 2 0 47 10 1 61 4 2008 30 1 16 19 11 1 68 22 15 183 4 2009 63 6 47 36 47 5 78 34 15 331 4 2010 67 9 50 55 114 5 183 43 44 570 17 Please note that data are displayed as reported by the States and have not been cleaned by the cross-site evaluation team. Numbers may change upon cleaning of the data. “Missing” means that target numbers should have been entered but were not. “N/A” means there are/were no GPRA reporting requirements for a State for that year. Each target reflects infrastructure changes that are projected to be completed by September 30 of the target grant year; therefore, changes projected to be completed after September 30 count toward the next year, and changes projected to be completed after the end of a State’s grant do not appear in the table. Data were extracted from Transformation Tracker on October 1, 2009. For all GPRA indicators except GPRA 6, targets were derived by adding the target numbers listed by year within the “Target Population” column of the “Key Features of Infrastructure Activities” report of Transformation Tracker. 18 GPRA 1: Number of policy changes completed as a consequence of the CMHP 19 GPRA 2: Number of persons in the mental health care and related workforce who have been trained in service improvements recommended by the CMHP 20GPRA 3: Number of financing policy changes completed as a consequence of the CMHP 21GPRA 4: Number of organizational changes completed as a consequence of the CMHP MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 47 GPRA Measure FY CT HI MD MO NM OH OK TX WA Program 4 2011 N/A 9 N/A 56 N/A N/A N/A N/A N/A 571 522 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline 5 2007 0 Baseline 0 Baseline 16 0 2 23 0 41 5 2008 0 0 0 96 33 498 38 27 0 692 5 2009 11 1 3 6,709 66 514 192 27 10 7,533 5 2010 324 14 42 6,863 246 532 407 27 21 8,476 5 2011 N/A 14 N/A 6,873 N/A N/A N/A N/A N/A 8,486 623 2007 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline 6 2008 14,158 Baseline 4,826 Baseline 1,562 17,155 3,351 15,500 10,136 66,688 6 2009 14,474 Missing 4,906 Missing Missing 30,055 3,656 16,500 12,522 82,113 6 2010 14,802 Missing 4,986 Missing Missing 31,355 3,961 17,500 13,019 85,623 6 2011 N/A Missing N/A Missing N/A N/A N/A N/A N/A 85,623 724 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline 7 2007 0 Baseline 0 Baseline 0 515 86 3 1 605 7 2008 0 0 19 199 60 940 101 99 1 1,419 7 2009 4 11 79 661 98 1,239 281 290 12 2,675 7 2010 4 17 217 831 196 1,601 589 580 326 4,361 7 2011 N/A 21 N/A 1,046 N/A N/A N/A N/A N/A 4,580 22 GPRA 5: Number of organizations that regularly obtain and analyze data relevant to the goals of the CMHP 23 GPRA 6: Number of consumers and family members that are members of Statewide consumer- and family-run networks. For GPRA 6, targets were derived from the “GPRA 6 Membership Numbers” table within the “Overview of Infrastructure Activities” section of Transformation Tracker. 24 GPRA 7: Number of programs implementing practices consistent with the CMHP MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 48 Appendix B: GPRA Results Table 30. Cumulative GPRA Results/Accomplishments25 GPRA Measure Year CT HI MD MO NM OH OK TX WA Program 126 2006 0 N/A 0 N/A 0 0 14 1 0 15 1 2007 12 0 0 5 3 0 43 13 15 91 1 2008 33 0 5 21 6 18 57 57 32 229 1 2009 33 0 5 54 6 20 61 70 32 281 1 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 1 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11 227 2006 0 N/A 0 N/A 0 0 0 0 0 0 2 2007 0 0 22 0 0 180 406 395 2,602 3,605 2 2008 2,915 179 1,455 1,769 2,236 4,516 13,928 18,729 10,740 56,467 2 2009 2,920 1,219 2,319 45,534 2,236 7,016 15,177 22,764 17,475 116,660 2 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 2 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11 328 2006 0 N/A 0 N/A 0 0 8 0 0 8 3 2007 0 0 2 6 19 0 12 9 3 51 3 2008 0 0 5 28 32 2 20 43 17 147 3 2009 1 0 5 39 32 2 26 58 19 182 3 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 3 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11 429 2006 0 N/A 0 N/A 0 0 6 0 0 6 4 2007 1 0 4 4 0 0 26 15 2 52 4 2008 29 0 44 18 20 1 44 32 8 196 4 2009 29 0 52 35 20 1 52 44 34 267 4 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 25 Please note that data are displayed as reported by the States and have not been cleaned by the cross-site evaluation team. Numbers may change upon cleaning of the data. “Missing” means there were no results reported as of September 30 of the target year. “N/A” means there are/were no GPRA reporting requirements for a State for that year. Each result reflects infrastructure changes with impacts reported up to September 30 of the target year; therefore, changes completed after September 30 of a calendar year count toward the next year, and changes completed outside of a State’s grant period do not appear in the table. Data were extracted from Transformation Tracker on October 1, 2009. For all GPRA indicators except GPRA 6, results were derived by summing the reported “impact” fields in the downloaded “Activity Results” entries for each State. Results were summed by fiscal year based on reported completion date. 26GPRA 1: Number of policy changes completed as a consequence of the CMHP 27GPRA 2: Number of persons in the mental health care and related workforce who have been trained in service improvements recommended by the CMHP 28GPRA 3: Number of financing policy changes completed as a consequence of the CMHP 29GPRA 4: Number of organizational changes completed as a consequence of the CMHP MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 49 GPRA Measure Year CT HI MD MO NM OH OK TX WA Program 4 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11 530 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11 5 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 5 2009 0 0 17 6,772 65 0 39 9 18 6,920 5 2008 0 0 7 106 65 0 31 5 18 232 5 2007 0 0 0 0 30 0 2 5 0 37 5 2006 0 N/A 0 N/A 0 0 1 0 0 1 631 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11 6 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 6 2009 Missing Missing Missing Missing Missing Missing 4,871 Missing Missing 4,871 6 2008 14,386 1,520 6,431 909 1,562 17,476 3,754 13,256 7,679 66,973 6 2007 13,842 N/A 4,739 N/A Missing 19,395 3,046 14,700 6,689 62,411 732 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11 7 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 7 2009 0 0 159 702 67 838 167 316 255 2,504 7 2008 0 0 114 170 67 753 135 99 247 1,585 7 2007 0 0 0 0 0 117 58 4 1 180 7 2006 0 N/A 0 N/A 0 0 28 0 0 28 30GPRA 5: Number of organizations that regularly obtain and analyze data relevant to the goals of the CMHP 31GPRA 6: Number of consumers and family members that are members of Statewide consumer- and family-run networks. For GPRA 6, results were derived from the “GPRA 6 Membership Numbers” table within the “Overview of Infrastructure Activities” section of Transformation Tracker. 32GPRA 7: Number of programs implementing practices consistent with the CMHP MHT SIG Evaluation Project, Annual Report, October 22, 2009 50 Appendix C: Proof of Concept Studies Information Table 31. State Interventions Examined in Recovery (Adult) Study State Impacted Group Nonimpacted Group Connecticut Individualized Recovery Planning tools with advanced training, infrastructure support, technical assistance (TA)— professional and peer staff331 Individualized Recovery Planning tools with 2-day training (not receiving advanced training, infrastructure support, or TA) Maryland Training and consultation providers and consumers in psychiatric rehabilitation programs regarding Cultural Competence and elimination of racial disparities in service provision1 Psychiatric rehabilitation programs not receiving training New Mexico Integrated Dual Diagnosis Treatment (IDDT) for adults diagnosed with co-occurring disorders in nine Certified Comprehensive Community Support Services (CCSS) Contractor Sites Provider sites not yet trained or certified in IDDT or the CCSS Model Ohio Using Outcomes in Person-Centered Treatment Planning (training for staff and consumers) Wait-list agencies Oklahoma Illness Management and Recovery— structured course with guided curriculum for Psychiatric Rehabilitation Service consumers Wait-list Texas Self-directed/consumer-directed care— control over funds to purchase services Services as usual Washington Journey to Life Wellness/Home Team— peer support and education on recovery and use of wellness tools Sequential design (pre and post) will do concurrent (randomized) next year if funds permit 33Intervention aims to enhance existing intervention/programs. MHT SIG Evaluation Project, Annual Report, October 22, 2009 51 Table 32. State Interventions Examined in Resilience (Child) Study State Impacted Group Nonimpacted Group Connecticut Wraparound with infrastructure support1 Wraparound without infrastructure support Maryland Wraparound Service Model Local Coordinating Councils (jurisdictions not yet implementing WSM) New Mexico Clinical Home—Comprehensive Service Agency model for referrals from Juvenile Justice division, eventually Protective Services Division Youth assessment center with services as usual Ohio Trauma Affect Regulation Guide for Education and Therapy (TARGET)— manualized treatment and prevention for Department of Youth Services Juvenile Correctional Facilities Mental health services as usual Oklahoma Cross-agency care coordination oversight for children, youth, and their families with high-level service use, funding redirected from residential to community-based services Comparable population, randomized to services as usual Texas Enhancing Fidelity to a Resiliency and Disease Management System: staff training and consultation1 Resiliency and Disease Management programs without training and consultation Washington Mental health services in middle-school, school-based health centers Schools without school-based health centers Tables 33 through 36 provide information about the GPRA measures and NFC goals being addressed by the State POC studies. Shaded boxes indicate the specific measures that are addressed by each State’s recovery (Tables 33 and 34) and resilience (Tables 35 and 36) POC studies. Table 33. GPRA Measures Addressed Recovery (Adult) Studies State 1 2 3 4 5 6 7 Connecticut n Y(2) n n Y n n Maryland n Y n n n n Y New Mexico n n Y n n Y Ohio n Y n n n n n Oklahoma Y Y n Y Y n Y Texas Y n Y n Y n n Washington n Y Y n Y n Y Table 34. NFC Goals Addressed by Recovery (Adult) Studies NFC 1 2 3 4 5 6 Connecticut n Y n n n n Maryland Y Y Y n Y n New Mexico n Y n n Y n Ohio n Y n n n n Oklahoma Y Y n n Y n Texas n Y n n Y n Washington Y Y n n n n MHT SIG Evaluation Project, Annual Report, October 22, 2009 52 Table 35. GPRA Measures Addressed by Resilience (Child) Studies State 1 2 3 4 5 6 7 Connecticut Y Y n Y Y(2) n n Maryland n Y Y n Y n Y New Mexico n n n Y n n Y Ohio n Y n n n n n Oklahoma Y Y n Y Y n Y Texas n n n n n n n Washington1 Y Y Y Y Y Y Y Table 36. NFC Goals Addressed by Resilience (Child) Studies State 1 2 3 4 5 6 Connecticut n Y n n Y n Maryland Y Y Y Y Y n New Mexico n n n n Y n Ohio n n n Y Y Y Oklahoma n Y Y Y Y n Texas n Y n n n n Washington1 Y Y Y Y Y Y Tables 37 through 40 below present the instruments for measuring system- and individual-level recovery in adults and individual resilience in children selected by each site from the approved list. It is apparent that there is some variability, with no more than three sites using any one measure. Table 37. Individual-level Recovery Instruments Being Used by Grantee States Instrument States Using Recovery Markers Questionnaire (RMQ) CT, NM, WA Mental Health Recovery Measure (MHRM) MD, OK Ohio Mental Health Consumer Outcomes system OH Recovery Assessment Scale (RAS) TX Table 38. System-level Recovery Orientation Instruments Being Used by Grantee States Instrument States Using Recovery Self-Assessment Revised (RSA-R) CT, MD, OH Recovery Oriented System Indicators Measure (ROSI) WA Recovery Enhancing Environment Measures (REE) NM Elements of a Recovery Facilitating System (ERFS–Adult) OK, TX Optional/other: Discrimination Experience Subscale/ISMI WA Table 39. Individual-level Resilience Instruments Being Used by Grantee States Instrument States Using Developmental Assets Profile (DAP) CT, WA Behavioral and Emotional Resilience Scale (BERS-2) MD, TX Ohio Youth Problems, Functioning, and Satisfaction Scales OH, OK Resilience Scale NM Optional/other: Strengths and Difficulties Questionnaire OK MHT SIG Evaluation Project, Annual Report, October 22, 2009 53 Table 40. System-level Resilience Orientation Instruments Being Used by Grantee States (Optional) Instrument States Using Elements of a Recovery Facilitating System (ERFS–Child Adapted) CT Community Supports for Wrap-Around Inventory WA Recovery Self-Assessment Revised (RSA-R) OH Optional/Other: Consumer Perception of Program Cultural Competence MD MHT SIG Evaluation Project, Annual Report, October 22, 2009 54 Appendix D: National Outcome Measures from the CMHS Uniform Reporting System, 2004–2008 Table 41. NOM 2: Increased or Retained Employment and School Enrollment: Percentage of Adults With Known Status Who Are Employed State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut 24 24 24 25 -- + Hawaii 19 24 23 23 -- + Maryland -- 14 14 16 -- + Missouri 19 17 -- 15 -- - New Mexico 21 20 25 21 -- None Ohio 16 22 23 23 -- + Oklahoma 18 18 19 20 -- + Texas 15 16 17 17 -- + Washington 12 11 11 12 -- None United States 21 21 22 21 -- None Table 42. NOM 4: Increased Stability in Family and Living Conditions: Percentage of Adults With Known Status Who Are Living in Private Residence State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut 72.6 76.2 73.7 78.7 80.8 + Hawaii 75.0 65.3 65.9 67.2 70.0 - Maryland -- 77.1 79.2 80.1 80.8 + Missouri 33.0 -- -- 80.1 78.5 + New Mexico 90.1 85.8 89.6 89.7 90.9 + Ohio -- 82.3 83.2 83.4 82.6 + Oklahoma 85.3 85.8 86.5 87.9 89.7 + Texas 91.7 90.4 90.6 91.2 91.4 - Washington 78.6 79.5 78.0 77.9 78.1 - United States 74.9 79.6 79.5 80.1 80.8 + MHT SIG Evaluation Project, Annual Report, October 22, 2009 55 Table 43. NOM 5: Increased Access to Services: Percentage of Persons Served by Age: Percentage of Clients Who Are Under Age 18 or Are Age 65 or Over State Categories 2004 2005 2006 2007 2008 Percent of State/U.S. 2008 Population Connecticut Under 18 42.1 35.0 35.6 35.1 35.0 23.2 Age 65+ 4.7 5.6 4.3 4.2 4.4 13.7 Hawaii Under 18 21.0 16.6 22.1 16.7 15.6 22.1 Age 65+ 8.0 7.0 5.4 4.2 6.8 14.8 Maryland Under 18 48.1 47.0 50.7 45.0 43.8 23.8 Age 65+ 1.4 1.3 1.2 1.2 1.2 12.1 Missouri Under 18 20.4 21.2 22.5 22.4 21.9 24.0 Age 65+ 3.1 2.8 2.8 2.9 2.8 13.6 New Mexico Under 18 -- 43.1 40.1 40.9 42.3 25.3 Age 65+ 3.8 1.9 1.8 1.7 1.7 13.1 Ohio Under 18 33.8 34.2 33.8 34.1 34.0 23.8 Age 65+ 2.9 2.9 2.8 2.8 2.9 13.7 Oklahoma Under 18 11.4 12.9 13.3 12.9 13.4 24.9 Age 65+ 2.6 2.3 2.2 1.7 1.7 13.5 Texas Under 18 18.2 18.7 19.1 19.4 19.8 27.6 Age 65+ 3.2 3.1 2.9 2.8 2.7 10.2 Washington Under 18 27.4 27.5 27.4 26.7 26.1 23.5 Age 65+ 7.6 7.2 6.5 6.5 6.6 12.0 United States Under 18 25.0 24.8 24.6 27.0 27.0 24.3 Age 65+ 4.8 4.6 4.5 4.8 4.8 12.8 Table 44. NOM 5: Increased Access to Services: Percentage of Persons Served by Race/Ethnicity (Percent of Clients Who Are White) State 2004 2005 2006 2007 2008 Percent of State/U.S. 2008 Population Connecticut 59.2 59.3 55.2 63.0 65.1 83.9 Hawaii 27.2 26.6 26.5 24.6 23.9 27.3 Maryland 42.6 44.1 44.1 45.9 46.6 61.8 Missouri 76.1 75.8 74.7 75.1 73.6 84.9 New Mexico 62.4 45.2 54.8 51.4 50.2 75.8 Ohio 74.1 73.5 73.5 73.3 73.1 84.7 Oklahoma 78.1 78.2 78.4 78.8 78.9 77.3 Texas 74.9 75.6 74.6 74.6 74.5 74.7 Washington 71.5 71.1 69.8 69.2 69.2 83.7 United States 74.3 74.1 73.9 63.3 63.3 77.6 MHT SIG Evaluation Project, Annual Report, October 22, 2009 56 Table 45. NOM 6: Decreased Utilization of Psychiatric Inpatient Beds: Mental Health Hospital Utilization Rate per 1,000 State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut .53 .56 .46 .43 .43 negative Hawaii .27 .32 .33 .32 .30 + Maryland .67 .62 .60 .51 .47 negative Missouri 1.5 1.39 1.33 1.27 1.15 negative New Mexico .48 .56 .60 1.24 .54 + Ohio -- -- -- -- -- N/A Oklahoma .83 .95 .85 .72 .74 negative Texas .69 .68 .67 .65 .61 negative Washington .57 .53 .52 .53 .52 negative United States .61 .63 .59 .60 .59 negative Table 46. NOM 7: Increased Social Support/Social Connectedness: Percentages for Adults and Children/Families34 State Adult 2006 Adult 2007 Adult 2008 Adult 2006- 2008 Change Child/Family 2006 Child/Family 2007 Child/Family 2008 Child/Family 2006-2008 Change Connecticut 73 77 75 + 86 83 90 + Hawaii -- 67 69 + -- 75 83 + Maryland 68 73 69 + 83 84 86 + Missouri -- 69 69 None -- 81 83 + New Mexico -- 97 97 None 84 93 89 + Ohio 36 37 37 + -- -- -- N/A Oklahoma 93 93 95 + 91 -- 98 + Texas 62 63 61 - 79 77 72 negative Washington -- 62 63 + -- 86 86 None United States 73 72 73 None 79 84 86 + Table 47. NOM 8: Increased Positive Reporting by Clients About Outcomes: Percentage of Adult Consumers With Positive Reporting of Outcomes State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut 78 81 78 80 77 negative Hawaii 68 69 71 70 76 + Maryland 63 61 66 70 63 None Missouri -- 81 -- 71 70 negative New Mexico 74 70 80 91 83 + Ohio 68 69 67 85 68 None Oklahoma 96 96 97 90 92 negative Texas 60 60 53 57 55 negative Washington 58 58 66 65 60 + United States 71 71 71 71 72 + 34 Measures of Social Support/Social Connectedness had been in developmental status prior to 2006. MHT SIG Evaluation Project, Annual Report, October 22, 2009 57 Table 48. NOM 10: Increased Use of Evidence-Based Practices: Percentage of States Implementing Adult Supported Housing State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut -- 9.8 9.8 10.7 9.9 + Hawaii -- 6.6 7.6 6.1 4.6 negative Maryland -- 18.6 14.4 14.8 15.2 negative Missouri -- -- -- -- -- N/A New Mexico 3.2 2.4 -- -- 0.3 negative Ohio -- -- -- -- -- N/A Oklahoma 1.0 0.9 0.8 0.6 0.5 negative Texas 4.4 2.7 2.3 2.8 2.7 negative Washington 0.3 -- -- 4.1 5.4 + United States 4.7 5.0 5.5 5 3 negative Table 49. NOM 10: Increased Use of Evidence-Based Practices: Percentage of States Implementing Adult Supported Employment State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut -- -- -- -- 3.8 N/A Hawaii -- 6.1 6.5 4.3 3.2 negative Maryland 57.91 4.3 4.5 5.3 5.7 + Missouri -- -- -- 2.7 9.1 + New Mexico 9.6 6.6 2.8 2.2 1.7 negative Ohio -- 100351 1.7 1.7 1.5 negative Oklahoma 0.2 0.1 0.1 0.1 0.1 negative Texas 3.5 2.7 1.7 1.6 1.2 negative Washington 1.8 -- -- 2.9 3.1 + United States 3.1 2.5 2.7 2 2 negative Table 50. NOM 10: Increased Use of Evidence-Based Practices: Percentage of States Implementing Adult Assertive Community Treatment State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut -- -- -- -- -- N/A Hawaii -- 7.6 6.7 4.5 3.5 negative Maryland -- 3.2 3.8 4.3 4.6 + Missouri -- -- -- -- 0.2 N/A New Mexico -- -- 0.2 0.2 0.3 + Ohio -- -- -- 1.7 1.8 + Oklahoma -- 1.9 2.4 2.9 2.4 + Texas -- 2.6 2.1 2.1 1.8 negative Washington -- -- -- 1.2 1.3 + United States 2.2 2.2 2.8 3 2 negative 35These data appear to be in error. MHT SIG Evaluation Project, Annual Report, October 22, 2009 58 Table 51. NOM 10: Increased Use of Evidence-Based Practices: Percentage of States Implementing Therapeutic Foster Care for Children and Adolescents State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut -- 5.0 5.0361 -- -- None Hawaii -- 12.6 9.1 10.3 12.6 None Maryland -- -- -- -- 5.2 N/A Missouri -- -- -- 1.7 -- N/A New Mexico -- 5.8 5.0 4.2 5.4 negative Ohio -- -- -- -- -- N/A Oklahoma -- -- -- -- -- N/A Texas -- -- -- -- -- N/A Washington -- -- -- 1.8 1.9 + United States -- 2.1 2.0 3 2 negative 362006 figure for Connecticut is based on communication with State staff.
Object Description
Description
Title | Cross Site Evaluation - Annual Evaluation Report (Year 3), October, 2009 |
Publication Date | 2009-10-22 |
Purpose | "This annual report identifies accomplishments and findings related to the third year of the...project." |
OkDocs Class# | M1400.3 E92m 2009 |
Digital Format | PDF, Adobe Reader required |
ODL electronic copy | Downloaded from agency website: www.ok.gov/.../Cross%20Site%20Evaluation%20-%20Annual%20Evaluation%20Report%20(Year%203),%20October,... |
Rights and Permissions | This Oklahoma state government publication is provided for educational purposes under U.S. copyright law. Other usage requires permission of copyright holders. |
Language | English |
Full text | Evaluation of the Mental Health Transformation State Incentive Grant (MHT SIG) Program Annual Evaluation Report October 22, 2009 Prepared for The Department of Health and Human Services Substance Abuse and Mental Health Services Administration Center for Mental Health Services Prepared by MANILA Consulting Group, Inc., and Human Services Research Institute (HSRI) MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 Contents INTRODUCTION .......................................................................................................................................... 1 ACCOMPLISHMENTS ................................................................................................................................ 1 Transformation Tracker Guidance and Support ....................................................................................... 1 POC Guidance and Support ..................................................................................................................... 1 State Site Visits ........................................................................................................................................ 1 Federal Agency Site Visit .......................................................................................................................... 2 Consumer and Family Member Focus Groups ......................................................................................... 2 Evaluation Work Group Meetings ............................................................................................................. 2 2008 Annual Grantee Meeting .................................................................................................................. 2 Active Involvement of Consumer and Family Member Consultants ......................................................... 3 MHT SIG Web Site .................................................................................................................................. 3 Leadership Survey ................................................................................................................................... 3 Provider Survey ....................................................................................................................................... 3 Other Accomplishments........................................................................................................................... 4 FINDINGS .................................................................................................................................................... 4 Transformation Tracker Data: GPRA Activities and the President’s New Freedom Commission Goals . 4 Proof of Concept Studies ........................................................................................................................ 18 State Site Visits ...................................................................................................................................... 25 Federal Agency Site Visit ........................................................................................................................ 26 Consumer, Youth, and Family Member Focus Groups .......................................................................... 28 National Outcome Measures Data ......................................................................................................... 34 Analysis of State Employment Initiatives ................................................................................................ 36 Leadership Survey ................................................................................................................................. 36 Provider Survey ..................................................................................................................................... 38 State Specific Evaluation Activities ......................................................................................................... 38 SUMMARY ................................................................................................................................................. 39 APPENDIX A: GPRA TARGETS ............................................................................................................... 46 APPENDIX B: GPRA RESULTS ................................................................................................................ 48 APPENDIX C: PROOF OF CONCEPT STUDIES INFORMATION ........................................................... 50 APPENDIX D: NATIONAL OUTCOME MEASURES FROM THE CMHS UNIFORM REPORTING SYSTEM, 2004–2008 ................................................................................................................................ 54 MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 Tables TABLE 1. 2007-2009 NUMBER OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ........................................ 6 TABLE 2. 2007-2009 PERCENTAGE OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ................................. 7 TABLE 3. 2007 NUMBER OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ................................................. 8 TABLE 4. 2007 PERCENTAGE OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS .......................................... 9 TABLE 5. 2008 NUMBER OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ............................................... 10 TABLE 6. 2008 PERCENTAGE OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ........................................ 11 TABLE 7. 2009 NUMBER OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ............................................... 12 TABLE 8. 2009 PERCENTAGE OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ........................................ 13 TABLE 9. SUMMARY OF 2007 TRANSFORMATION ACTIVITIES BY NEW FREEDOM COMMISSION GOALS ............... 15 TABLE 10. SUMMARY OF 2008 TRANSFORMATION ACTIVITIES BY NEW FREEDOM COMMISSION GOAL ............... 16 TABLE 11. SUMMARY OF 2009 TRANSFORMATION ACTIVITIES BY NEW FREEDOM COMMISSION GOAL ............... 17 TABLE 12. STATUS OF RECOVERY STUDIES .................................................................................................... 18 TABLE 13. STATUS OF RESILIENCY STUDIES ................................................................................................... 19 TABLE 14. STATUS OF STATE PROOF OF CONCEPT STUDIES ........................................................................... 21 TABLE 15. RANKING OF NFC RECOMMENDATIONS BY MHT SIG PROGRAM MANAGERS AND CONNECTORS ...... 27 TABLE 16. AVERAGE CYF RATING OF FREQUENCY AND TYPE OF CONSUMER, YOUTH, AND FAMILY MEMBER INVOLVEMENT IN GRANT ACTIVITIES ...................................................................................................... 29 TABLE 17. AVERAGE CFY RATINGS REGARDING VARIOUS ASPECTS OF CFY PARTICIPATION IN MHT SIG GRANT ACTIVITIES .......................................................................................................................................... 31 TABLE 18. NUMBER OF MHT SIG GRANTEE STATES REPORTING INCREASES, DECREASES, AND NO CHANGES IN THEIR NOMS DATA FROM 2004 TO 2008 .............................................................................................. 35 TABLE 19. RESULTS OF PILOT TEST OF PROVIDER SURVEY ............................................................................. 38 TABLE 20. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING MENTAL HEALTH PROGRAM/SERVICES/SYSTEMS/CLIENTS .................................................. 40 TABLE 21. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING STIGMA ............................................................................................................................. 41 TABLE 22. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING FUNDING & RESOURCES .................................................................................................... 41 TABLE 23. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING WORKFORCE AND TRAINING ............................................................................................... 42 TABLE 24. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING CYF, CYF ORGANIZATIONS, & COMMUNITY STAKEHOLDERS ............................................... 43 TABLE 25. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING DISPARITIES, PREVALENCE, & STATISTICS .......................................................................... 44 TABLE 26. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING CRIMINAL/JUVENILE JUSTICE .............................................................................................. 44 TABLE 27. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING TRANSFORMATION WORKING GROUPS AND COMMITTEES .................................................... 45 TABLE 28. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES REGARDING OTHER TOPICS .................................................................................................................. 45 TABLE 29. CUMULATIVE GPRA TARGET NUMBERS ......................................................................................... 46 TABLE 30. CUMULATIVE GPRA RESULTS/ACCOMPLISHMENTS ........................................................................ 48 TABLE 31. STATE INTERVENTIONS EXAMINED IN RECOVERY (ADULT) STUDY .................................................... 50 TABLE 32. STATE INTERVENTIONS EXAMINED IN RESILIENCE (CHILD) STUDY .................................................... 51 TABLE 33. GPRA MEASURES ADDRESSED RECOVERY (ADULT) STUDIES ........................................................ 51 TABLE 34. NFC GOALS ADDRESSED BY RECOVERY (ADULT) STUDIES ............................................................. 51 TABLE 35. GPRA MEASURES ADDRESSED BY RESILIENCE (CHILD) STUDIES ................................................... 52 TABLE 36. NFC GOALS ADDRESSED BY RESILIENCE (CHILD) STUDIES ............................................................ 52 TABLE 37. INDIVIDUAL-LEVEL RECOVERY INSTRUMENTS BEING USED BY GRANTEE STATES .............................. 52 TABLE 38. SYSTEM-LEVEL RECOVERY ORIENTATION INSTRUMENTS BEING USED BY GRANTEE STATES ............ 52 TABLE 39. INDIVIDUAL-LEVEL RESILIENCE INSTRUMENTS BEING USED BY GRANTEE STATES ............................. 52 TABLE 40. SYSTEM-LEVEL RESILIENCE ORIENTATION INSTRUMENTS BEING USED BY GRANTEE STATES (OPTIONAL) ......................................................................................................................................... 53 MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 TABLE 41. NOM 2: INCREASED OR RETAINED EMPLOYMENT AND SCHOOL ENROLLMENT: PERCENTAGE OF ADULTS WITH KNOWN STATUS WHO ARE EMPLOYED ............................................................................. 54 TABLE 42. NOM 4: INCREASED STABILITY IN FAMILY AND LIVING CONDITIONS: PERCENTAGE OF ADULTS WITH KNOWN STATUS WHO ARE LIVING IN PRIVATE RESIDENCE ..................................................................... 54 TABLE 43. NOM 5: INCREASED ACCESS TO SERVICES: PERCENTAGE OF PERSONS SERVED BY AGE: PERCENTAGE OF CLIENTS WHO ARE UNDER AGE 18 OR ARE AGE 65 OR OVER ...................................... 55 TABLE 44. NOM 5: INCREASED ACCESS TO SERVICES: PERCENTAGE OF PERSONS SERVED BY RACE/ETHNICITY (PERCENT OF CLIENTS WHO ARE WHITE) .............................................................................................. 55 TABLE 45. NOM 6: DECREASED UTILIZATION OF PSYCHIATRIC INPATIENT BEDS: MENTAL HEALTH HOSPITAL UTILIZATION RATE PER 1,000 ................................................................................................................ 56 TABLE 46. NOM 7: INCREASED SOCIAL SUPPORT/SOCIAL CONNECTEDNESS: PERCENTAGES FOR ADULTS AND CHILDREN/FAMILIES .............................................................................................................................. 56 TABLE 47. NOM 8: INCREASED POSITIVE REPORTING BY CLIENTS ABOUT OUTCOMES: PERCENTAGE OF ADULT CONSUMERS WITH POSITIVE REPORTING OF OUTCOMES ....................................................................... 56 TABLE 48. NOM 10: INCREASED USE OF EVIDENCE-BASED PRACTICES: PERCENTAGE OF STATES IMPLEMENTING ADULT SUPPORTED HOUSING ................................................................................................................ 57 TABLE 49. NOM 10: INCREASED USE OF EVIDENCE-BASED PRACTICES: PERCENTAGE OF STATES IMPLEMENTING ADULT SUPPORTED EMPLOYMENT ......................................................................................................... 57 TABLE 50. NOM 10: INCREASED USE OF EVIDENCE-BASED PRACTICES: PERCENTAGE OF STATES IMPLEMENTING ADULT ASSERTIVE COMMUNITY TREATMENT .......................................................................................... 57 TABLE 51. NOM 10: INCREASED USE OF EVIDENCE-BASED PRACTICES: PERCENTAGE OF STATES IMPLEMENTING THERAPEUTIC FOSTER CARE FOR CHILDREN AND ADOLESCENTS ............................................................ 58 MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 Figures FIGURE 1. AVERAGE RATING OF GRANT IMPACT BY STATE .............................................................................. 25 FIGURE 2. AVERAGE RATING OF GRANT IMPACT BY TYPE OF STATE AGENCY ................................................... 26 FIGURE 3. AVERAGE RATINGS ON STATEMENTS PROVIDED BY CONSUMER AND FAMILY MEMBER FOCUS GROUP PARTICIPANTS ACROSS ALL STATES ...................................................................................................... 34 MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 1 INTRODUCTION This annual report identifies accomplishments and findings related to the third year of the Mental Health Transformation State Incentive Grant (MHT SIG) cross-site evaluation project. The third year of the project began September 15, 2008, and ended September 14, 2009. The evaluation of the MHT SIG program is designed to measure the success of transformation efforts in grantee States to meet New Freedom Commission (NFC) goals. The project supports nine States as they implement their MHT SIG grants. The first cohort of States, awarded Center for Mental Health Services (CMHS) grants in 2005, includes seven States (Connecticut, Maryland, New Mexico, Ohio, Oklahoma, Texas and Washington). The second cohort, awarded grants in 2006, includes two States (Hawaii and Missouri). ACCOMPLISHMENTS During this year of the project, the project team—MANILA Consulting Group, Inc. (MANILA), and Human Services Research Institute (HSRI)—focused efforts on completing the consumer, youth, and family focus groups in all nine grantee States and developing associated reports, conducting site visits to Hawaii and Missouri, launching Transformation Tracker Explorer, coordinating with States regarding Proof of Concept (POC) study requirements, completing leadership and provider surveys, and examining methods for conducting the various cross-site analyses for the final report. Transformation Tracker Guidance and Support The cross-site evaluation collects information in Transformation Tracker on planned MHT SIG activities, including the nature of the activities, their scheduling, and whether or not they are completed. These activities are coded by States as to New Freedom Commission Goals addressed and SAMHSA Government Performance and Results Act (GPRA) categories. The MHT SIG program has seven Government Performance and Results Act (GPRA) categories of infrastructure changes for which States must provide annual progress updates: policy changes, workforce trainings, financial policy changes, organizational changes, data sharing and analysis, membership in consumer/family member (CFM) networks, and implementation of practices consistent with the State’s Comprehensive Mental Health Plan (CMHP). The project team provided detailed guidance and support to Hawaii and Missouri as they completed initial entry of their activities into Transformation Tracker. In addition, guidance was provided to other States regarding various issues with their Transformation Tracker entries and GPRA 6 (membership in CFM networks) supporting information. Additionally, the cross-site team began discussions of how to clean, code, and analyze the Transformation Tracker data from the perspectives of New Freedom Commission Goals and Recommendations and SAMHSA GPRA categories. POC Guidance and Support The evaluation design requires States to identify services for adults with serious mental illness and children/youth with serious emotional disturbance that have been or are anticipated to be impacted by the MHT SIG project (the impacted group) as well as comparable services that have not been impacted by the project (the nonimpacted group). The goal is to determine if positive changes in recovery (for adults) and resilience (for youth) result from “best case” transformative initiatives consisting of both infrastructure and service changes. During the past year, for these recovery and resiliency studies, the project team focused on different activities by cohort. For the first cohort of States, the evaluation team clarified details of their POC proposals and approved completed proposals, completed successful test data submissions for all instruments to be used, supported and monitored data collection efforts and early stages of data submission, and developed preliminary data analysis plans based on anticipated data collection rates. For the second cohort of States, the primary focus was on supporting the development of their POC proposals. State Site Visits MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 2 State site visits are conducted to determine the effect the MHT SIG projects have had on their mental health service systems, to understand why the States have chosen the approaches they have, and to learn about any barriers or needs associated with the grant program. The initial site visit with each grantee is conducted in grant year 3, with a second one occurring in grant year 5. The cross-site evaluators conducted site visits in Hawaii on December 17 and 18, 2008, and in Missouri on January 28 and 29, 2009. Draft site visit reports were prepared and provided to both States. Federal Agency Site Visit From December 2008 to early March 2009, the project team interviewed key CMHS managers, all six Connectors (the MHT SIG program’s Federal Project Officers), and two other CMHS staff to understand the development and implementation of the MHT SIG program to date. All interviewees also were asked to rank the New Freedom Commission recommendations in order of importance. Consumer and Family Member Focus Groups Consumer and family member focus groups with up to 15 individuals per State were designed to identify the nature of consumer/family member involvement in transformation grant activities at both the State and local levels. Within each State, the focus groups were to consist of persons who were aware of the grant—some who were involved in grant activities and some who were not. The process of identifying CFM focus group participants and scheduling meetings was found to be more difficult than anticipated. In particular, it was challenging to find individuals who were aware of the mental health transformation grants, regardless of whether they were involved in grant activities. Other difficulties with recruitment included reluctance to participate due to limited familiarity with the Federal grant program and lack of responsiveness from CFM organizations that were expected to be potential sources of participants. In addition, some persons who initially indicated interest in participation did not respond to subsequent followup attempts. Focus groups were conducted during the State site visits with all the CFMs who were available. All other focus group interviews were completed by phone afterwards. By the end of the third year, all focus group interviews had been completed in all States. Reports for focus groups with the first cohort of States were provided on July 17, and public versions of these reports were developed and provided on September 3. Reports for Hawaii and Missouri are under development. Evaluation Work Group Meetings Evaluation Work Group meetings provide a forum in which State evaluators can update the evaluation team on the status of their efforts, identify issues and needs associated with their project, and obtain peer-to-peer support. During the third project year, 11 Evaluation Work Group meetings were held by telephone. Calls were used to provide guidance on POC studies and discuss implementation issues as well as the status of State-specific evaluation studies, the status of leadership and provider surveys, the potential development and subsequent results of launching the publicly available Transformation Tracker Explorer Web site, State-specific evaluation activities, and topics for the November 2009 Annual Grantee Meeting. Participation by evaluators from all States was consistently strong for all of the meetings. 2008 Annual Grantee Meeting The 2008 Annual Grantee Meeting was conducted on November 12 through 14 in Bethesda, Maryland. This meeting provided an opportunity for program staff and grantees to share accomplishments and identify objectives for the coming year. Three sessions were devoted to evaluation topics: Cross-Site Evaluation Progress Report: The Project Officer and Principal Investigator provided information about accomplishments in the cross-site evaluation and preliminary findings from the State planned MHT SIG activity data. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 3 Evaluation Informing Grant Implementation: Challenges and Opportunities: Evaluators from Connecticut, Hawaii, Oklahoma, and Washington made presentations about strengths-based approaches that can facilitate the process of evaluation and grant project activities informing one another. Cross-Site Evaluators Meeting: With no preset agenda for this session, Hawaii, Maryland, New Mexico, Ohio, Oklahoma, and Washington used the time to describe particular evaluation activities/initiatives in their States. The session also included general discussion about disparities in access to mental health services, POC issues, State-specific evaluation activities, and methods for determining when transformation has occurred. Active Involvement of Consumer and Family Member Consultants The evaluation project has six CFM Consultants who participate in all aspect of the project to ensure that CFM perspectives are incorporated into the design and implementation of the evaluation (e.g., site visits, focus groups, Evaluation Work Group calls, annual grantee meetings). During the third project year, the CFM Consultants were involved in the two State site visits and associated focus group interviews. They also reviewed the site visit and focus group reports for the States they visited. Three of the Consultants were able to participate in the cross-site team meeting in Boston in August 2009. One Consultant has been active with the MHT SIG Consumer, Family, and Youth Work Group and has participated in many of its monthly conference calls. The Consultants also reviewed and commented on the Transformation Tracker Explorer test site prior to its launch in May 2009. MHT SIG Web Site This Web site was developed early in the evaluation project to provide grantees with resource and project information and a system in which States could enter their MHTSIG activities. The major change to the Web site this year was the launch of Transformation Tracker Explorer (http://mhtsigdata.samhsa.gov) on May 19, 2009. This is the public part of the MHT SIG evaluation project Web site that allows the public to see, rate, and comment on GPRA activities being undertaken in all nine grantee States. This Web site is thought to be one of the first of its kind, providing current information about grantee activities and accomplishments during the course of an evaluation project. On August 18, 2009, the project team submitted a 90-day report detailing Web site traffic to date and identifying the MHTSIG activity ratings and comments that had been submitted. By the end of the third year, there were approximately 1000 visits by approximately 500 unique individuals from 42 States and 16 other countries. Leadership Survey Conducted in grant years 3 and 5, the leadership survey is designed to capture leadership characteristics of persons heading up the MHT SIG transformation efforts within the States. “Leaders” are the Project Directors and Transformation Working Group (TWG) chairs, while “raters” are TWG members or TWG subcommittee members. For the first cohort of States, the online leadership survey was launched in October 2008; 10 of 11 leaders and 51 of 76 raters completed the survey by March 2009. This survey will be conducted in Hawaii and Missouri in November 2009 to maintain the same timing of the survey as for the first cohort of States. Provider Survey The provider survey is designed to determine whether providers are aware of the MHT SIG grant in their State, and if so, how their State has been affected by mental health transformation grant initiatives. In the original evaluation design, each grantee was to survey 26 providers in each of grant years 3, 4, and 5. In conjunction with the Project Officer, it was decided that a pilot survey of providers with a smaller group of participants in year 3 would be a useful approach in understanding how providers might respond to the survey questions and whether any additional probes might be useful. The pilot survey was completed in all States except Hawaii; three of four provider surveys in Hawaii still need to be completed. The MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 4 remaining surveys will be completed during grant years 4 and 5, beginning in late September 2009. Providers will be encouraged to complete the survey online to make efficient use of staff time (i.e., staff will not need to schedule and conduct phone surveys or perform data entry). Other Accomplishments Other project accomplishments include the following: An analysis of employment initiatives by grantee States was done in January 2009 to determine the types of initiatives being undertaken, including participation in competitive labor force efforts and any U.S. Department of Labor initiatives. Presentation at the National Association of State Mental Health Program Directors (NASMHPD) Research Institute: On April 15, 2009, the Project Officer and a project staff member presented information about the MHT SIG evaluation project and preliminary findings on integration of mental health and physical health care. Meeting with MHT SIG Connectors: On April 21, 2009, project staff met with the Connectors to discuss the ways GPRA information could be used for program management purposes and the plans to launch Transformation Tracker Explorer. Support for the MHT SIG Consumer, Family, and Youth Work Group: At the request of the MHT SIG Consumer, Family, and Youth Advisor/Consultant, the project staff provided a demonstration to this group about how to use Transformation Tracker Explorer during a conference call on May 20, 2009. Cross-Site Team Meeting: On August 25 and 26, 2009, the entire cross-site team met to discuss and make decisions about the cross-site analyses to be completed by the end of the project. The Project Officer attended this meeting along with three of the six CFM Consultants. Participation on Planning Committee for the 2009 Annual Grantee Meeting: In July and August 2009, the project participated in Planning Committee meetings to help determine agenda topics and presenters for the upcoming November 2009 meeting. Analysis of Other State Data: Public health epidemiological datasets to measure statewide outcomes were obtained from the following other state data sources: SAMHSA’s National Survey on Drug Use and Health (NSDUH) for 2002 to 2007 and Behavioral Risk Factor Surveillance Survey (BRFSS) from the Centers for Disease Control and Prevention (CDC) for 2002 to 2008. These datasets and other public data sources are being examined to identify relevant mental health outcomes and relate them to CMHS National Outcomes Measures (NOMs). We also are reviewing other longitudinal datasets (e.g., the data on which the NAMI Grading the States Reports were based) that might have information on outcomes or infrastructure changes in States. In addition to analytic approaches that compare MHTSIG States with Non-MHTSIG control States over time, we are also exploring “dose-effect” analytic methods for analyzing the associations between transformative practices and infrastructure and outcome changes within MHTSIG States. FINDINGS The findings from the third year of the cross-site evaluation project are discussed in the following sections on Transformation Tracker data, POC studies, State site visits, Federal agency site visit, consumer and family member focus groups, statewide National Outcome Measures (NOMs) data, analysis of State employment initiatives, leadership survey, provider survey, and State-specific evaluation activities. Transformation Tracker Data: GPRA Activities and the President’s New Freedom Commission Goals MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 5 The cross-site evaluation project has tracked MHT SIG State activities in Transformation Tracker, where entries are coded by GPRA measure and by the President’s New Freedom Commission goal. Given the central role of the NFC report in the origination of the MHT SIG program, Tables 1 through 8 below present a cross-tabulation of MHTSIG activities by NFC goal for entries scheduled to be completed in the 2007 to 2009 fiscal years and for each of the years separately. Each activity item can be assigned to only one GPRA category but may be assigned to multiple NFC goals. Data in these tables as well as Tables 9 through 11 were extracted from Transformation Tracker on October 5, 2009. The individual percentages in these tables may not total 100 percent due to rounding. Appendix A provides target activity information by State and year, while Appendix B presents activity results by State and year. All tables that describe GPRA and NFC goal activity in this section and in Appendices A and B are provisional and are intended to illustrate the analyses that will eventually be implemented for the summative cross-site evaluation. A primary reason for caution about the data in these tables is that, as of yet, the data have not been “cleaned” (i.e., they have not been reviewed to ensure the accuracy of coding or the presence of duplicate activities). In addition, the GPRA and NFC tables within this report do not reflect State activities for the 2006 baseline year because the grantees were not required to report GPRA data until 2007. At least one grantee State, however, has entered its baseline MHTSIG activities into Transformation Tracker. Tables 1 through 8 show numbers and percentages of GPRA activity items by NFC goal. If these data are accurately coded by the States, the most frequently addressed NFC goals in 2007, 2008, and 2009 were NFC 2 (care is consumer and family driven) and NFC 5 (excellent care is provided), while the least frequently addressed ones were NFC 3 (disparities are eliminated) and NFC 6 (technology is used). They also show that most State activities were classified under GPRA 1 (policy changes) and GPRA 2 (training activities). Currently a total of 1,076 unique MHTSIG activity items have been entered in Transformation Tracker as ending in fiscal years 2007 to 2009 across all States). As mentioned previously, this count does not reflect baseline year activities. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 6 Table 1. 2007-2009 Number of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 82 72 23 60 79 37 GPRA 2: Training 101 147 57 78 118 15 GPRA 3: Financial Policy Changes 43 41 20 32 53 19 GPRA 4: Organizational Changes 47 55 19 47 44 17 GPRA 5: Data Sharing 8 17 3 11 23 13 GPRA 6: Consumer, Youth, & Family Networks 3 29 0 0 2 0 GPRA 7: CMHP Practices 63 34 18 44 54 20 Non-GPRA 22 53 12 14 32 16 Total Number 369 448 152 286 405 137 Percent of Total Activities 21% 25% 8% 16% 23% 8% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 7 Table 2. 2007-2009 Percentage of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 22% 16% 15% 21% 20% 27% GPRA 2: Training 27% 33% 38% 27% 29% 11% GPRA 3: Financial Policy Changes 12% 9% 13% 11% 13% 14% GPRA 4: Organizational Changes 13% 12% 13% 16% 11% 12% GPRA 5: Data Sharing 2% 4% 2% 4% 6% 10% GPRA 6: Consumer, Youth, & Family Networks 1% 6% 0% 0% 1% 0% GPRA 7: CMHP Practices 17% 8% 12% 15% 13% 15% Non-GPRA 6% 12% 8% 5% 8% 12% Total Number 100% 100% 100% 100% 100% 100% Percent of Total Activities 21% 25% 8% 16% 23% 8% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 8 Table 3. 2007 Number of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 13 25 4 13 17 8 GPRA 2: Training 4 7 4 4 9 0 GPRA 3: Financial Policy Changes 9 9 3 8 11 5 GPRA 4: Organizational Changes 5 7 2 5 6 3 GPRA 5: Data Sharing 1 1 0 1 0 2 GPRA 6: Consumer, Youth, & Family Networks 0 3 0 0 1 0 GPRA 7: CMHP Practices 6 2 1 4 3 1 Non-GPRA 0 2 1 1 1 5 Total Number 38 56 15 36 48 24 Percent of Total Activities 18% 26% 7% 17% 22% 11% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 9 Table 4. 2007 Percentage of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 34% 45% 27% 36% 35% 33% GPRA 2: Training 11% 13% 27% 11% 19% 0% GPRA 3: Financial Policy Changes 24% 16% 20% 22% 23% 21% GPRA 4: Organizational Changes 13% 13% 13% 14% 13% 13% GPRA 5: Data Sharing 3% 2% 0% 3% 0% 8% GPRA 6: Consumer, Youth, & Family Networks 0% 5% 0% 0% 2% 0% GPRA 7: CMHP Practices 16% 4% 7% 11% 6% 4% Non-GPRA 0% 4% 7% 3% 2% 21% Total Number 100% 100% 100% 100% 100% 100% Percent of Total Activities 18% 26% 7% 17% 22% 11% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 10 Table 5. 2008 Number of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 26 23 12 22 29 18 GPRA 2: Training 42 76 29 39 57 10 GPRA 3: Financial Policy Changes 11 17 10 14 29 11 GPRA 4: Organizational Changes 19 23 8 18 17 4 GPRA 5: Data Sharing 1 8 2 4 8 5 GPRA 6: Consumer, Youth, & Family Networks 2 16 0 0 0 0 GPRA 7: CMHP Practices 18 10 7 14 17 7 Non-GPRA 12 30 7 6 16 9 Total Number 131 203 75 117 173 64 Percent of Total Activities 17% 27% 10% 15% 23% 8% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 11 Table 6. 2008 Percentage of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 20% 11% 16% 19% 17% 28% GPRA 2: Training 32% 37% 39% 33% 33% 16% GPRA 3: Financial Policy Changes 8% 8% 13% 12% 17% 17% GPRA 4: Organizational Changes 15% 11% 11% 15% 10% 6% GPRA 5: Data Sharing 1% 4% 3% 3% 5% 8% GPRA 6: Consumer, Youth, & Family Networks 2% 8% 0% 0% 0% 0% GPRA 7: CMHP Practices 14% 5% 9% 12% 10% 11% Non-GPRA 9% 15% 9% 5% 9% 14% Total Number 100% 100% 100% 100% 100% 100% Percent of Total Activities 17% 27% 10% 15% 23% 8% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 12 Table 7. 2009 Number of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 43 24 7 25 33 11 GPRA 2: Training 55 64 24 35 52 5 GPRA 3: Financial Policy Changes 23 15 7 10 13 3 GPRA 4: Organizational Changes 23 25 9 24 21 10 GPRA 5: Data Sharing 6 8 1 6 15 6 GPRA 6: Consumer, Youth, & Family Networks 1 10 0 0 1 0 GPRA 7: CMHP Practices 39 22 10 26 34 12 Non-GPRA 10 21 4 7 15 2 Total Number 200 189 62 133 184 49 Percent of Total Activities 25% 23% 8% 16% 23% 6% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 13 Table 8. 2009 Percentage of GPRA Activities Associated with NFC Goals GPRA Measure NFC Goal 1: Mental Health Is Essential NFC Goal 2: Care Is Consumer/ Family Driven NFC Goal 3: Disparities Are Eliminated NFC Goal 4: Early Screening Is Provided NFC Goal 5: Excellent Care Is Provided NFC Goal 6: Technology Is Used GPRA 1: Policy Changes 22% 13% 11% 19% 18% 23% GPRA 2: Training 28% 34% 39% 26% 28% 10% GPRA 3: Financial Policy Changes 12% 8% 11% 8% 7% 6% GPRA 4: Organizational Changes 12% 13% 15% 18% 11% 20% GPRA 5: Data Sharing 3% 4% 2% 5% 8% 12% GPRA 6: Consumer, Youth, & Family Networks 1% 5% 0% 0% 1% 0% GPRA 7: CMHP Practices 20% 12% 16% 20% 19% 25% Non-GPRA 5% 11% 7% 5% 8% 4% Total Number 100% 100% 100% 100% 100% 100% Percent of Total Activities 25% 23% 8% 16% 23% 6% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 14 As noted earlier, the Transformation Tracker activity items are coded by NFC goal(s), and additional information is provided about the history and current status of each activity. Tables 9, 10, and 11 below summarize the most important historical information associated with the NFC activity items in 2007, 2008, and 2009, respectively; each table presents, by NFC goal, the number of activities, the percentage of States with those activities, the percentage of activities representing that goal across the States (presented as a range), the percentage begun before the MHT SIG grant, the percentage raised in the Needs Assessment and Resource Inventory (NARI), the percentage noted in the CMHP, the percentage recommended by the TWG, and the percentage with results noted in Transformation Tracker. If these data are accurately coded by the States, the characteristics of activity items associated with the NFC goals have changed over time. For example, in 2007, activity items associated with NFC 2 (care is consumer and family driven) were less likely to be recommended by TWGs than activity items associated with other NFC goals. In 2008, a greater proportion of NFC 2 activity items had been TWG-approved, though these items were still approved by the TWG at a lower rate than activities related to other NFC goals. Interestingly, Goal 2 activities in 2009 had a much higher approval rate than in the previous 2 years, giving these items a TWG approval rate consistent with those for other NFC goal activities. One possible explanation is that earlier activity items had not been presented to TWGs at the time the activity items were entered into Transformation Tracker, and that this information has not yet been updated. The cross-site evaluation team will investigate the reason for this pattern when conducting the data cleaning for the summative evaluation. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 15 Table 9. Summary of 2007 Transformation Activities by New Freedom Commission Goals NFC Goals Number of Activities Percent of States Percent Range of Activity Items Percent Begun Before MHT SIG [Yes] Percent Begun Before MHT SIG [No] Percent Raised in NARI [Yes] Percent Raised in NARI [No] Percent Noted in CMHP [Yes] Percent Noted in CMHP [No] Percent Recomm by TWG [Yes] Percent Recomm by TWG [No] Percent of Activities With Activity Results NFC 2: Care Is Consumer/ Family Driven 54 89% 0-31% 19% 81% 61% 39% 80% 20% 50% 50% 96% NFC 5: Excellent Care Is Provided 48 89% 0-31% 19% 81% 88% 13% 94% 6% 73% 27% 98% NFC 1: Mental Health Is Essential 38 67% 0-50% 18% 82% 82% 18% 89% 11% 82% 18% 95% NFC 4: Early Screening Is Provided 36 67% 0-39% 25% 75% 83% 17% 97% 3% 64% 36% 94% NFC 6: Technology Is Used 23 67% 0-35% 9% 91% 96% 4% 96% 4% 83% 17% 83% NFC 3: Disparities Are Eliminated 15 56% 0-60% 0% 100% 87% 13% 93% 7% 87% 13% 93% Total 214 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A Average per Goal 35.7 N/A N/A 15% 85% 83% 17% 92% 8% 73% 27% 93% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 16 Table 10. Summary of 2008 Transformation Activities by New Freedom Commission Goal NFC Goals Number of Activities Percent of States Percent Range of Activity Items Percent Begun Before MHT SIG [Yes] Percent Begun Before MHT SIG [No] Percent Raised in NARI [Yes] Percent Raised in NARI [No] Percent Noted in CMHP [Yes] Percent Noted in CMHP [No] Percent Recomm by TWG [Yes] Percent Recomm by TWG [No] Percent of Activities With Activity Results NFC 2: Care Is Consumer/ Family Driven 194 100% 1-25% 19% 81% 74% 26% 87% 13% 57% 43% 93% NFC 5: Excellent Care Is Provided 177 89% 0-22% 25% 75% 78% 22% 93% 7% 77% 23% 90% NFC 1: Mental Health Is Essential 133 89% 0-31% 27% 73% 75% 25% 92% 8% 78% 22% 88% NFC 4: Early Screening Is Provided 118 89% 0-32% 30% 70% 86% 14% 92% 8% 86% 14% 86% NFC 3: Disparities Are Eliminated 76 89% 0-29% 29% 71% 74% 26% 100% 0% 70% 30% 79% NFC 6: Technology Is Used 66 89% 0-24% 12% 88% 86% 14% 98% 2% 89% 11% 86% Total 764 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A Average per Goal 127.3 N/A N/A 24% 76% 79% 21% 94% 6% 76% 24% 87% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 17 Table 11. Summary of 2009 Transformation Activities by New Freedom Commission Goal NFC Goals Number of Activities Percent of States Percent Range of Activity Items Percent Begun Before MHT SIG [Yes] Percent Begun Before MHT SIG [No] Percent Raised in NARI [Yes] Percent Raised in NARI [No] Percent Noted in CMHP [Yes] Percent Noted in CMHP [No] Percent Recomm by TWG [Yes] Percent Recomm by TWG [No] Percent of Activities With Activity Results NFC 1: Mental Health Is Essential 199 100% 2-34% 12% 88% 75% 25% 91% 9% 85% 15% 82% NFC 2: Care Is Consumer/ Family Driven 185 89% 0-23% 23% 77% 73% 27% 91% 9% 71% 29% 65% NFC 5: Excellent Care Is Provided 184 100% 2-21% 23% 77% 81% 19% 89% 11% 81% 19% 72% NFC 4: Early Screening Is Provided 132 89% 0-36% 20% 80% 77% 23% 89% 11% 75% 25% 64% NFC 3: Disparities Are Eliminated 62 100% 3-21% 19% 81% 79% 21% 95% 5% 87% 13% 65% NFC 6: Technology Is Used 49 89% 0-27% 12% 88% 86% 14% 92% 8% 92% 8% 76% Total 811 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A Average per Goal 135.2 N/A N/A 18% 82% 78% 22% 91% 8% 82% 18% 71% MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 18 Proof of Concept Studies The status of the recovery and resiliency studies (as of September 14, 2009) is shown in Tables 12 and 13. For both types of studies, each State is required to enroll 75 persons in the impacted group and 75 in the nonimpacted group. Overall, across impacted and nonimpacted groups, the first cohort of States had completed baseline data collection with 68 percent of the enrollment target for their recovery studies and 51 percent for their resiliency studies by the end of the third year. Table 12. Status of Recovery Studies State Start Date Consents Past Week Declines Past Week Interviews Past Week Consents to Date Declines to Date Interviews to Date Connecticut Impacted 7/09 Not available Not available Not available Not available Not available 75 Connecticut Nonimpacted 7/09 Not available Not available Not available Not available Not available 75 Maryland Impacted 5/09 N/A 0 2 N/A 66 51 Maryland Nonimpacted 5/09 N/A 0 0 N/A 15 27 New Mexico Impacted 5/09 N/A N/A N/A 10 N/A 10 New Mexico Nonimpacted 6/09 N/A N/A N/A 17 N/A 17 Ohio Impacted Complete N/A N/A N/A N/A N/A 751 Ohio Nonimpacted Complete N/A N/A N/A N/A N/A 752 Oklahoma Impacted 2/09 9 0 9 70 0 70 Oklahoma Nonimpacted 2/09 11 0 11 69 0 69 Texas Impacted 4/09 4 0 2 74 2 35 Texas Nonimpacted 4/09 N/A N/A 2 Not available Not available 25 Washington Impacted 8/09 8 4 8 14 9 14 Washington Nonimpacted 5/08 0 0 0 98 Not available 983 Total (Percent of Target Number) N/A N/A N/A N/A N/A N/A 716 (68%) 1Final number may be more than 75. 2Final number may be more than 75. 3Sequential design: This is followup data collection. Baseline data collection was completed 7/08. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 19 Table 13. Status of Resiliency Studies4 Site Start Date Consents Past Week Declines Past Week Interviews Past Week Consents to Date Declines to Date Interviews to Date Connecticut Impacted 6/09 0 0 0 0 0 0 Connecticut Nonimpacted 6/09 0 0 0 0 0 0 Maryland Impacted 1/09 0 0 0 79 22 50 Maryland Nonimpacted 1/09 0 0 1 35 17 28 New Mexico Impacted 6/08 N/A N/A N/A 48 N/A 48 New Mexico Nonimpacted 6/08 N/A N/A N/A N/A N/A 0 Ohio Impacted Complete N/A N/A N/A N/A N/A 751 Ohio Nonimpacted Complete N/A N/A N/A N/A N/A 751 Oklahoma Impacted Complete 9 N/A 9 78 N/A 781 Oklahoma Nonimpacted Complete 8 N/A 8 78 N/A 781 Texas Impacted 4/09 Not available Not available Not available 83 10 71 Texas Nonimpacted 4/09 Not available Not available Not available 24 5 16 Washington Impacted 3/09 0 0 0 39 13 37 Washington Nonimpacted 3/09 0 0 0 23 7 23 Total (Percent of Target Number) N/A N/A N/A N/A N/A N/A 579 (51%) Beginning in April 2009, State evaluators were asked to provide updates on the status of their POC studies on each monthly Evaluation Work Group call. By June 2009, it became apparent that at least several sites would likely not reach the recruitment target of 150 participants for their POC studies by the mid-August deadline (the last possible date that would allow a 1-year followup by the project’s end). Accordingly, the evaluation team responded in two ways: 1. By providing additional monitoring and support for the sites. In late July and early August, a series of calls with sites at risk of not reaching the full complement of 150 participants (i.e., Connecticut, Maryland, Oklahoma, New Mexico, Texas, Washington) were held to determine the current status of enrollment, the nature of the difficulties encountered, and possible strategies for overcoming them. In general, delays had resulted from two factors: (1) The interventions were not implemented as planned by the State mental health agency, and (2) there had been a lack of responsiveness on the part of State treatment programs, especially those representing nonimpacted sites, to refer clients to the POC studies. 4 Final number may be more than 75. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 20 At-risk States were called to ask whether they intended to request a no-cost extension to allow sufficient time to enroll the full number of participants and preserve the 1-year followup period. None of the States indicated that they planned to make such a request. The project team provided more focused monitoring and support for the sites that appeared to be having the most difficulty (e.g., Maryland, New Mexico, Texas) to identify the nature of the obstacles they were encountering and explore means of overcoming them. 2. By exploring analytic strategies to address the possible shortfall in the number of study participants and the potential to have a shorter followup period. As it became apparent that some POC studies were unlikely to reach the target of 150 persons by the deadline of mid-August, the cross-site evaluation team began to explore analytic strategies to address the possibility of having fewer than 150 participants for each study. The project’s statistical consultant conducted further power analyses and found that 100 participants per site would provide adequate power. It appears that most, if not all, of the sites will achieve this level of enrollment. However, this approach will require further consideration if the number of participants in each study is not fairly equally balanced between impacted and nonimpacted groups, which appears likely to be the case in some States. The second approach, having a followup period of less than 1 year (e.g., 9 months) was examined to determine whether the various interventions being studied and measures being used would permit a dose-effect analysis whereby it would be possible to identify a trend line representing outcomes as a function of time of exposure to the intervention. A preliminary review of the interventions and the measures being used in the POC studies indicated that this was in fact the case—the effects of the interventions may be expected to be incremental and the outcomes are measured as continuous variables. The Project Officer set a final deadline for baseline data collection of November 1, 2009, based on guidance from the evaluators that this date would provide the optimum balance between number of participants and length of followup period. Subsequently, several States expressed various concerns about this deadline, such as additional burden related to data collection and Institutional Review Board (IRB) modifications as well as methodological issues. These concerns were addressed through email and telephone discussion with the evaluation team. Both of these approaches will be explored more thoroughly in the coming year as baseline data become available. Table 14 provides an overview of the status of POC studies by State (as of September 14, 2009), including each State’s plan to request a no-cost extension. Appendix C provides more detailed information about each of the State POC studies, including the impacted and nonimpacted groups, the GPRA indicators and NFC goals being addressed, and the survey instruments being used. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 21 Table 14. Status of State Proof of Concept Studies Site (Intervention) Status of Data Collection Date Expect To Complete Baseline Data Collection Problems Encountered/ Actions To Expedite Potential Problems Method Issues/Help Requested Plans to Request No-Cost Extension Connecticut Recovery (Staff trained in individual recovery planning) IRB approval obtained, data collection completed Impacted and nonimpacted interviews completed IRB delays— now complete None anticipated (based on previous projects with same clinicians and consumers) None at this time Not expected Connecticut Resilience (Wraparound in juvenile justice) Final IRB obtained in September 2009, requiring only sign-off; data collection to begin immediately End of October 2009 IRB questions— now resolved None anticipated, if recruitment proceeds as rapidly as was the case with adult study 1. “More vs. less impacted” (i.e., subject exposure varies at baseline— admin data, and site fidelity measure may allow adjustment 2. Possibly shortened timeframe (11 months) 3. Type of sites vary (school, probation, welfare) No, if shorter followup period is acceptable Hawaii Recovery (Network of Care/WRAP Self-management) Proposal submitted to cross-site evaluation team, working with the sites on logistics Between December 2009 and February 2010 Delays in State Mental Health Agency (SMHA) implementation of programs— meetings with SMHA and Project Directors None additional at this time Questions about follow up period—advised to conform to protocol of 1-year follow up Don’t know Hawaii Resilience (Under development) Partners and sites selected, study details being worked out, expect to submit proposal in October 2009 Unknown at this time Difficulty obtaining cooperation of SMHA—delayed implementation None additional at this time None at this time Don’t know MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 22 Site (Intervention) Status of Data Collection Date Expect To Complete Baseline Data Collection Problems Encountered/ Actions To Expedite Potential Problems Method Issues/Help Requested Plans to Request No-Cost Extension Maryland Recovery (Cultural competence training) Underway; have IRB approval to extend to end of October 2009 End of October 2009 High refusal and bad contact information— adjustments made None additional at this time Initial concern about obtaining diagnosis, but will be available from administrative data Hope not Maryland Resilience (Wraparound) Consent to be contacted forms completed, continuing to follow up November 1, 2009; however, probably will have fewer than 150 participants Fewer than expected receiving services—staff have been very active contacting sites Need to complete before other grants start up Possibility of having less than 150 participants More time will not help, as subject pool is limited and need to complete before new grants begin Missouri Recovery (Peer specialists) POC proposal submitted to cross-site evaluators, sites to be determined, expect to start data collection in January 2010 Uncertain N/A None at this time None at this time Don’t know Missouri Resilience (Family support specialist proposal is under development) Still in planning stage, submission to cross-site evaluators has not been made; expect to start data collection in November/ December 2009 Uncertain N/A None at this time None at this time Don’t know MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 23 Site (Intervention) Status of Data Collection Date Expect To Complete Baseline Data Collection Problems Encountered/ Actions To Expedite Potential Problems Method Issues/Help Requested Plans to Request No-Cost Extension New Mexico Recovery (Integrated Dual Diagnosis Treatment) Underway but proceeding very slowly November 1, 2009; however, probably will have fewer than 150 participants In July, State Managed Care Organization (MCO) change; MCO was not contacting sites, turmoil at impact site—Met with Deputy Secretary, site incentives increased, more education, recruitment now bypassing MCO and going directly to sites No additional anticipated, expect to proceed more smoothly None at this time Don’t know New Mexico Resilience (Clinical home) Underway for a year but proceeding very slowly, already doing some followup interviews though baseline incomplete November 1, 2009; however, probably will have fewer than 150 participants Same as above plus turmoil at impact site— Deputy Secretary authorized five new sites Same as above Though still collecting baseline data, started in June 2008 and some are due for followup—will be maintaining 1-year interval for now Don’t know Texas Recovery (Self-directed care— control funds) Proceeding slowly November 1, 2009; however, probably will have fewer than 150 participants Difficulty with provider referrals, especially nonimpacted— doing more outreach and modified enrollment criteria None additional at this time None at this time Don’t know MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 24 Site (Intervention) Status of Data Collection Date Expect To Complete Baseline Data Collection Problems Encountered/ Actions To Expedite Potential Problems Method Issues/Help Requested Plans to Request No-Cost Extension Texas Resilience (Trauma cognitive behavioral therapy) Proceeding slowly November 1, 2009; however, probably will have fewer than 150 participants Obtaining referrals from nonimpacted sites Imbalance in number of participants with much smaller number of nonimpacted None at this time Don’t know Washington Recovery (Wellness) Sequential design: nonimpacted started June 2008, completed September 2009 Impacted baseline will start in September 2009 None at this time None at this time None at this time Hope not Washington Resilience (School-based services) Only partial numbers obtained by end of school year, will continue beginning 2009 school year End of October 2009 IRB modification was approved to allow default parent contact (opt-out letter) None at this time 1. 6-month interval between two baseline groups 2. Possible shortened timeframe Hope not MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 25 State Site Visits During the site visits, State agency directors were asked to rate the grant’s impact on their agency using a 10-point scale from 1 (lowest) to 10 (highest). Charts 1 and 2 illustrate ratings by State and type of State agency, respectively. (The States were randomly coded to maintain their anonymity.) Three States (States 4, 5, and 9) had the highest ratings of 7 or more, while two States (States 3 and 7) had the lowest ratings of 5 or less. Agency directors were asked to explain the rationale for their rating; their responses indicate that they used different approaches in assigning their rating. For example, some State agency staff based their rating on progress completed to date, while others rated based on the potential impact of the grant. Regarding impact by agency type, mental health and juvenile justice agencies reported the most impact (with ratings 7 or above), while other agencies reported slightly lower levels of impact (ratings of around 6). About 10 percent of the agency directors did not provide any rating, often because they viewed an assessment of the grant’s impact as premature or had limited involvement with the grant. Figure 1. Average Rating of Grant Impact by State5 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 Individual States Average Impact Rating 5 States have been randomly coded. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 26 Figure 2. Average Rating of Grant Impact by Type of State Agency 1 2 3 4 5 6 7 8 9 10 Mental Health Criminal Justice Juvenile Justice Child Welfare Housing Vocational Rehab Medicaid Type of State Agency Average Impact Rating Federal Agency Site Visit As part of the Federal agency site visit, Federal staff (program managers and Connectors) were asked to rank NFC recommendations by order of importance. The results of this ranking exercise are shown in Table 15. Review of these rankings reveals that managers and Connectors share four of their six highest ranked recommendations and five of their seven lowest ranked recommendations, showing a relatively high level of agreement between the groups. The greatest levels of agreement between manager and Connector rankings are described below: Recommendation 2.2 (involve CFMs in orienting mental health systems toward recovery) is the highest ranked recommendation of both groups. Recommendations 1.1 (advance antistigma and antisuicide campaigns), 1.2 (address mental health with same urgency as physical health), and 4.1 (promote the mental health of young children) are in the six highest ranked recommendations of both groups. The following recommendations are in the seven lowest ranked recommendations of both groups: 3.2 (improve access in rural and geographically remote areas). 5.1 (accelerate research). 5.4 (develop the knowledge base in four understudied areas). 6.1 (use technology to improve access and coordination of care). 6.2 (develop electronic health records). The following recommendations are in the six mid-ranked recommendations of both groups (neither highest nor lowest ranked): 4.3 (screen for co-occurring disorders and link with integrated treatment). MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 27 5.2 (advance evidence-based practices and use public/private partnerships to advance). The greatest discrepancies between manager and Connector rankings are found for: Recommendation 2.4 (develop CMHP)—ranked low by managers (16) and high by Connectors (2). Recommendation 4.2 (improve and expand school mental health programs)—ranked low by managers (17) and in the middle by Connectors (7). Recommendation 4.4 (screen for mental health disorders in primary health care, across the lifespan, and connect to treatment)—ranked in the middle by managers (7) and low by Connectors (16). Recommendation 2.3 (align Federal programs to improve access and accountability)—ranked high by managers (4) and in the middle by Connectors (12). Table 15. Ranking of NFC Recommendations by MHT SIG Program Managers and Connectors6 6 Code for shading/font: Turquoise square/bold font indicates recommendation is ranked in top 6; yellow shading/italic font indicates recommendation is ranked in bottom 7. NFC Recommendations Average Rank for Managers Overall Rank for Managers Average Rank for Connectors Overall Rank for Connectors 1.1 Advance and implement a national campaign to reduce the stigma of seeking care and a national strategy for suicide prevention 7.7 5 8.3 5 1.2 Address mental health with the same urgency as physical health 4.3 2 6.2 4 2.1 Develop an individualized plan of care for every adult with a serious mental illness and child with a serious emotional disturbance 11.5 12 11.8 14 2.2 Involve consumers and families fully in orienting the mental health system toward recovery 1.3 1 4.2 1 2.3 Align relevant Federal programs to improve access and accountability for mental health services 7.3 4 10.7 12 2.4 Create a Comprehensive State Mental Health Plan 13.3 16 4.5 2 2.5 Protect and enhance the rights of people with mental illnesses 8.7 8 8.3 5 3.1 Improve access to quality care that is culturally competent 6.3 3 9.5 8 3.2 Improve access to quality care in rural and geographically remote areas 12.3 14 11.5 13 4.1 Promote the mental health of young children 7.7 5 4.7 3 4.2 Improve and expand school mental health programs 15.3 17 8.8 7 4.3 Screen for co-occurring mental and substance use disorders and link with integrated treatment strategies 11.0 11 10.0 9 MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 28 Consumer, Youth, and Family Member Focus Groups Focus groups were conducted with convenience samples of CFMs; these samples should not be considered representative of all CFM perspectives across all grantee States. Of the 130 persons who participated in the focus groups: 101 participants were consumers (17 were youths, 77 were adults, and 7 were older adults). 39 participants were family members of youth consumers, and 62 were family members of adult consumers. A total of 40 persons identified as being both a consumer and family member. Involvement was greater at the State level (101 participants, or 78 percent) than at the local level (75 participants, or 58 percent). 107 participants (82 percent) reported being members of CFM stakeholder organizations. Tables 16 and 17 provide information about the frequency and type of CFM involvement in State and local activities and the involvement of CFMs in specific types of grant activities. In both these tables, the individual States have been randomly coded. Table 16 shows CFM ratings for the frequency and type of CFM involvement at the State and local level for individual States and for all States combined. Overall, the frequency of consumer and family member involvement was viewed as “occasionally” and their type of involvement as “minor” at both the State and local levels. 4.4 Screen for mental disorders in primary health care, across the lifespan, and connect to treatment and supports 8.0 7 12.8 16 5.1 Accelerate research to promote recovery and resilience, and ultimately to cure and prevent mental illnesses 15.3 17 15.7 19 5.2 Advance evidence-based practices using dissemination and demonstration projects and create a public-private partnership to guide their implementation 10.0 10 10.3 10 5.3 Improve and expand the workforce providing evidence-based mental health services and supports 9.5 9 10.3 10 5.4 Develop the knowledge base in four understudied areas: mental health disparities, long-term effects of medications, trauma, and acute care 12.7 15 14.5 17 6.1 Use health technology and telehealth to improve access and coordination of mental health care, especially for Americans in remote areas or in underserved populations 11.7 13 12.3 15 6.2 Develop and implement integrated electronic health record and personal health information systems 15.7 19 15.5 18 MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 29 Table 16. Average CYF Rating of Frequency and Type of Consumer, Youth, and Family Member Involvement in Grant Activities7 Consumer/Youth/Family Involvement State 1 State 2 State 3 State 4 State 5 State 6 State 7 State 8 State 9 Frequency of Involvement8 in Local Activities 2.3 1.7 1.5 1.7 1.4 1.9 1.7 1.4 1.4 1.8 Occasionally Type of Involvement9 in Local Activities 1.2 1.1 0.6 1.1 0.7 1.3 1.3 0.7 0.9 0.9 Minor Role Frequency of Involvement10 in State Activities 1.9 2.1 0.7 2.1 1.0 1.4 1.9 1.1 1.2 1.6 Occasionally Type of Involvement11 in State Activities 1.1 1.1 0.5 1.1 0.5 1.1 1.2 0.5 1.0 0.8 Minor Role 7States have been randomly coded. 8Never = 0, Rarely = 1, Occasionally = 2, Frequently = 3, Almost always = 4. 9No role = 0, Minor role = 1, Major role = 2. 10Never = 0, Rarely = 1, Occasionally = 2, Frequently = 3, Almost always = 4. 11No role = 0, Minor role = 1, Major role = 2. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 30 Focus group participants were also asked to rate their level of agreement with 35 statements about their involvement in grant activities. Table 17 shows the average ratings by focus group participants for these statements by individual States and for all States combined. For the purpose of describing ratings for all States combined, the ratings have been grouped into the following categories: 1 to 2.7 (disagree), 2.8 to 3.2 (generally neither agree nor disagree), and 3.3 to 5 (agree). Across all States, consumers and family members generally agreed with the following statements: During project meetings, the opinions of CFMs are discussed. Involvement of CFMs has made a difference. The grant promotes collaboration among CFMs. Disagreements are handled respectfully within this project. Efforts are being made to evaluate CFM involvement. Stigma/discrimination is not accepted at any level of this grant. Grant staff are able to work collaboratively with CFMs. Mental health service users and their family members are positively affected by this grant. Persons of all cultural and ethnic origins are respected within this grant. Across all States, consumers and family members generally disagreed with the following statements: There are enough CFMs involved in the mental health grant. CFMs have meaningful involvement in decisions on funding of State programs. CFMs have meaningful involvement in decisions on funding of local programs. CFMs have meaningful involvement in making State policy. CFMs receive the training and support they need to participate effectively in the mental health transformation grant. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 31 Table 17. Average CFY Ratings regarding Various Aspects of CFY Participation in MHT SIG Grant Activities1213 Statements State 1 State 2 State 3 State 4 State 5 State 6 State 7 State 8 State 9 Overall Average Rating14 1. CFMs have meaningful involvement in: Setting local goals 4.1 2.3 2.1 3.7 2.8 3.6 3.9 2.1 2.4 3.0 Neither Making local policy 3.7 2.3 1.8 3.5 2.5 3.3 3.8 2.0 2.1 2.8 Neither Designing local programs 3.8 2.8 2.0 3.3 2.5 3.3 3.0 2.2 2.6 2.9 Neither Implementing local programs 3.8 2.8 2.2 3.3 2.3 3.6 2.9 2.5 2.3 2.9 Neither Evaluating local programs 3.8 2.5 2.0 3.4 2.2 3.6 2.9 2.5 2.4 2.8 Neither Decisions on funding of local programs 3.3 2.3 1.6 3.4 2.4 2.9 2.8 1.9 2.4 2.5 Disagree Setting State goals 3.6 3.2 2.2 3.2 2.3 3.2 3.6 2.2 2.9 2.9 Neither Making State policy 3.2 2.6 1.8 2.8 1.9 2.8 3.4 2.0 2.8 2.6 Disagree Designing State programs 3.5 3.0 2.0 3.1 2.1 3.0 3.2 1.9 2.9 2.8 Neither Implementing State programs 3.5 3.4 1.8 3.7 2.1 2.8 3.7 1.7 2.9 2.8 Neither Evaluating State programs 3.8 2.8 2.0 3.3 2.1 2.9 3.8 1.8 3.1 2.8 Neither Decisions on funding of State programs 3.1 2.4 1.5 2.8 1.7 2.1 3.4 1.8 2.5 2.4 Disagree 2. There are enough CFMs involved in the mental health grant 3.2 2.8 1.4 3.3 1.1 1.8 3.2 1.2 2.0 2.2 Disagree 12 States have been randomly coded. 13 Strongly disagree = 1, Disagree = 2, Neither = 3, Agree = 4, Strongly Agree = 5. 14The ratings were grouped into the following categories: 1 to 2.7 (disagree), 2.8 to 3.2 (generally neither agree nor disagree), and 3.3 to 5 (agree). MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 32 Statements State 1 State 2 State 3 State 4 State 5 State 6 State 7 State 8 State 9 Overall Average Rating14 3. Mental health service users and their family members are positively affected by this grant 4.2 3.9 2.9 2.7 2.8 3.8 3.8 2.0 3.2 3.3 Agree 4. CFMs are adequately compensated for their roles 3.9 3.4 1.8 3.4 1.7 3.8 3.3 1.2 3.1 2.9 Neither 5. Involvement of CFMs has made a difference 4.2 4.2 3.9 2.7 2.6 4.1 3.8 2.3 3.3 3.5 Agree 6. Grant staff are able to work collaboratively with CFMs 4.5 4.2 3.3 2.9 2.0 4.0 4.0 1.7 3.5 3.4 Agree 7. The grant promotes collaboration among CFMs 4.4 4.1 3.0 2.9 2.6 3.6 3.9 2.4 3.5 3.5 Agree 8. State leaders are sensitive to cultural and linguistic issues 4.1 3.6 2.6 2.8 1.6 3.5 3.4 2.5 3.0 3.1 Neither 9. CFMs receive the training and support they need to participate effectively in the mental health transformation grant 3.6 3.3 2.0 2.2 1.4 3.1 3.3 1.9 2.8 2.7 Disagree 10. The grant promotes CFM understanding of the process behind developing policy 4.0 3.1 2.0 2.8 1.6 3.4 3.6 1.6 3.0 2.8 Neither 11. The grant promotes CFM understanding of current policy issues 3.8 3.1 2.3 2.6 1.8 3.6 3.7 2.0 3.4 3.0 Neither 12. The leaders of this grant make involvement by CFMs a priority 4.1 3.9 1.8 2.6 2.3 4.1 3.9 2.1 3.6 3.3 Neither 13. During project meetings, the opinions of CFMs are discussed 4.4 4.2 3.3 3.1 3.0 4.0 4.0 2.0 3.5 3.6 Agree 14. Action is taken as a result of CFM opinions 4.2 4.0 2.6 2.4 1.9 3.9 3.7 1.9 3.0 3.1 Neither 15. A partnership exists between CFMs and persons who are not CFMs 4.2 3.4 2.9 2.7 2.2 3.8 3.4 1.7 3.5 3.1 Neither 16. As a result of the grant, CFMs have the knowledge to educate the community on important issues. 3.7 3.4 2.2 2.6 2.3 3.5 4.0 1.8 3.0 3.0 Neither MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 33 Statements State 1 State 2 State 3 State 4 State 5 State 6 State 7 State 8 State 9 Overall Average Rating14 17. The grant promotes CFMs to take the lead in this transformation project. 3.9 3.4 2.0 2.7 1.8 3.7 3.9 1.9 2.9 2.9 Neither 18. Disagreements are handled respectfully within this project. 4.1 4.2 2.6 3.2 2.5 3.8 4.3 2.4 3.0 3.5 Agree 19. Information about the mental health transformation grant is readily available to CFMs. 3.9 3.4 2.4 3.9 2.4 3.5 3.9 2.0 3.2 3.2 Neither 20. Professionals use language that is easily understood by nonprofessional participants. 3.7 3.6 2.4 3.6 2.7 3.5 4.0 2.9 2.4 3.2 Neither 21. Efforts are made to evaluate CFM involvement. 4.4 4.1 3.0 3.0 2.6 4.0 4.1 2.3 3.5 3.5 Agree 22. CFMs are excited about the progress of the grant. 4.0 3.7 2.2 2.9 2.0 3.8 3.6 1.8 3.2 3.2 Neither 23. Persons of all cultural and ethnic origins are respected within this grant. 4.2 3.6 2.9 3.0 2.4 3.5 3.6 2.6 3.3 3.3 Agree 24. Stigma/discrimination is not accepted at any level of this grant. 4.3 4.3 2.4 2.7 2.5 4.0 4.4 2.7 3.6 3.5 Agree MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 34 Across all States, CFMs neither agreed nor disagreed with 21 (60 percent) of the statements (see Chart 3). CFMs agreed with nine of the statements (26 percent) and disagreed with five others (14 percent). The lack of agreement or disagreement with most statements raises a question of the true level and nature of CFM involvement in grant activities. Figure 3. Average Ratings on Statements Provided by Consumer and Family Member Focus Group Participants Across All States National Outcome Measures Data State NOMs data for 2004 to 2008 were analyzed to determine the completeness of data by individual measures and summarize the findings. Appendix D provides the actual NOMs data by State. Data for three measures are in development and are not currently available: NOM 1 (decreased mental illness symptomatology), NOM 3 (decreased criminal justice system involvement), and NOM 9 (increased cost-effectiveness). It should be noted that States sometimes collect NOMs data using different methods and samples. A full understanding of how these data can be used in the Other State Data Analyses will depend on a careful analysis of how these data are collected by MHT SIG states. The missing State data by NOMs measure are: NOM 2: Percentage of adults who are employed: Maryland in 2004 and Missouri in 2006. NOM 4: Percentage of individuals living in a private residence: Maryland in 2004, Missouri in 2005 and 2006, and Ohio in 2004. NOM 5: Percentage of persons served by age (under 18 and age 65+) and race/ethnicity: New Mexico in 2004 (age data only). NOM 6: Mental health hospital utilization per 1,000: Ohio in 2004 to 2008. NOM 7: Percentage of adults and children/families indicating increased social support/connectedness: In 2004 and 2005, this was a developmental measure. For adults, 2006 0% 10% 20% 30% 40% 50% 60% 70% Neither Agree nor Disagree Disagree Agree Response Percent of Respondents MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 35 data are missing for Hawaii, Missouri, New Mexico, and Washington. For children/families, 2006 data are missing for Hawaii, Missouri, Ohio, and Washington. Other missing data on children/families include 2007 and 2008 data for Ohio and 2007 data for Oklahoma. NOM 8: Percentage of adults with positive reporting for outcomes: Missouri in 2004 and 2006. NOM 10: Four different measures are used (percentages of States implementing adult supported housing, adult supported employment, adult assertive community treatment, and therapeutic foster care for children and adolescents), and there are missing data for many of these measures. The adult supported employment data are most complete, followed by adult supported housing. State trends for NOMs 2, 4, 6, 7, and 8 are shown in Table 18. Six of nine grantee States reported positive changes over the time period for all NOMs measures except NOM 8 (positive reporting on outcomes by adults). Table 18. Number of MHT SIG Grantee States Reporting Increases, Decreases, and No Changes in Their NOMs Data from 2004 to 2008 National Outcome Measure Increase Decrease No Change Not Applicable NOM 2: Employment 6 1 2 0 NOM 4: Living in Private Residence 6 3 0 0 NOM 6: Inpatient Utilization Rate151 6 2 0 1 NOM 7: Social Support/Social Connectedness—Adults 6 1 2 0 NOM 7: Social Support/Social Connectedness—Child/Family 6 1 1 1 NOM 8: Positive Reporting About Outcomes—Adults 3 4 2 0 The analysis of NOM 5 data was done for age groups (under 18 and 65+) and race/ethnicity using Tables 43 and 44 from Appendix D. The results showed that: Five States had higher percentages of youth in treatment in 2008 than were represented in the State’s general population, while four States had lower percentages. It is possible that higher percentages are associated with greater access to mental health services for youth. During the period from 2004 to 2008, four States had increases in their percentages of youth in treatment, while the remaining five States had decreases. All nine States had lower percentages of persons age 65+ in treatment in 2008 than were represented in the State’s general population. For six States, 3 percent or fewer of persons in treatment were age 65+, while individuals aged 65+ represented 10 to15 percent of grantee State populations. For the period 2004 to 2008, all States but Ohio had decreases in their percentages of persons aged 65+ in their client population; Ohio’s percentage remained unchanged. All but one grantee State had a lower percentage of Whites in treatment in 2008 than was represented in the State’s general population. As noted in the technical evaluation plan, issues do exist regarding statewide comparison of NOMs data (e.g., comparable data periods and measurement across States, lack of year-to-year consistency within individual States). Until these issues are examined in more detail at the level of the individual NOMs for the full data collection period, the usefulness of the NOMs data in the cross-site evaluation will be unclear. 15A reduced inpatient utilization rate is a change in a positive direction. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 36 Analysis of State Employment Initiatives In January 2009, an analysis of State employment initiatives was completed. The findings are summarized as follows: All nine MHT SIG States have employment activities focused on consumer representation in the mental health care workforce. Employment opportunities for consumers include: Peer specialists Positions in consumer-operated services Research, evaluation, or quality improvement team members Consumer positions in non-mental health service agencies Consumer positions on Transformation Working Groups Consumer positions as liaisons with other consumers (e.g., in offices of consumer affairs, on consumer engagement teams) General positions open to anyone (e.g., clinicians, State administrators) In addition, some States are addressing competitive labor force participation in the following subject areas: Supported employment (Maryland and Ohio) Ticket to Work (Maryland and Ohio) Medicaid (Ohio, Texas, and Washington) Vocational rehabilitation system (Ohio, Texas, and Washington) Criminal justice system (Washington) Temporary Assistance for Needy Families (New Mexico) Working with employers (Texas) Individual Development Accounts (Texas) Employment assessment as part of treatment planning (New Mexico and Ohio) Consumer-run moving business (New Mexico) Transportation barriers (Texas) None of the States reported activities that involve working with U.S. Department of Labor initiatives such as One-Stop Career Centers. Leadership Survey During the third project year, the leadership survey was administered for the first time to leaders and raters in the first cohort of States. Leadership changes had already occurred in most of the States when the survey was administered. For some States, these changes could be categorized as significant. For instance, two States experienced gubernatorial changes involving political party transitions. For these two States, leadership was impacted at three levels: State (gubernatorial), project (TWG Chairperson), and project management (Project Director). These two States had Project Directors who were involved in leadership of the TWG in one way or another; one Project Director operated in dual roles (as Project Director and TWG Chairperson) and the other was involved in coordination related to the TWG. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 37 Across both cohorts, four States have individuals serving in the dual roles of Project Director and TWG Chairperson. Changes in Project Directors occurred in seven States (including the two States that experienced gubernatorial changes). Implementation of the survey was challenging in some respects due to these changes in leadership. A common issue was getting previous leaders to complete the survey. Another challenge was that leaders and raters had to rely on their long-term memory. Raters in particular were asked to complete the survey based on their observations of the previous leader’s leadership style; for some, this aspect of their State’s MHT SIG project was difficult to recall. Preliminary findings from the leadership survey include the following: Leaders indicated they frequently seek differing perspectives when solving problems, consider the moral and ethical consequences of their decisions, and express satisfaction when others meet expectations. They reported that once in a while, they are absent when needed, delay responding to urgent questions, and avoid making decisions. Raters indicated that leaders fairly often consider the moral and ethical consequences of their decisions, express satisfaction when the rater meets expectations, and treat the rater as an individual rather than a member of a group. Raters reported that once in a while, leaders avoid making decisions, delay in responding to urgent questions, and are absent when needed. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 38 Provider Survey The pilot survey was completed in all States except Hawaii. A total of 37 providers completed the survey, with 59 percent reporting they were aware of the grant. Table 19 shows the results of the pilot survey as of September 14, 2009. Knowledge about the grant most often came from State mental health agencies, provider meetings, and news releases/emails. Table 19. Results of Pilot Test of Provider Survey State Number Surveyed Percent Aware of Grant How Providers Heard About Grant Connecticut 2 100% State Department of Mental Health and Addiction Services; participation on TWG subcommittee Hawaii 0 N/A N/A Maryland 6 67% Governor’s press announcement; NAMI meetings; co-occurring disorder meetings Missouri 4 50% Public mental health authority; CEO of their organization who works on the grant project New Mexico 7 43% Networking; CEO of their organization; local collaborative; provider meetings Ohio 4 50% Federal Department of Health and Human Services and SAMHSA; State Department of Mental Health email Oklahoma 6 50% Oklahoma Department of Mental Health and Substance Abuse Services (ODMHSAS) Director meetings; ODMHSAS publications; email to community mental health centers Texas 5 80% Provider meetings and mental health/mental retardation collaborative; news clippings; Texas Department of State Health Services grant notification Washington 3 67% Grant implementation committee; email notification Total 37 59% N/A State Specific Evaluation Activities Based on a request from State evaluators, a listing of actual and planned State-specific evaluation activities (i.e., activities that were not part of the cross-site evaluation requirements) was developed in May 2009 through review of 2009 State continuation applications and reviews by State MHT SIG staff and evaluators. Tables 20 through 28 list the 128 different activities identified by the grantee State evaluators. The average number of activities in each State is 14, with range from 10 in Texas to 26 in Connecticut.16 The most common types of evaluation activities in the States are as follows: Antistigma efforts to determine impact or effectiveness (5 States) Mental Health First Aid (4 States) State surveys of evidence-based practices (4 States) 16 Note that Hawaii and Missouri received less MHT SIG funding than the States that were awarded grants in 2005. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 39 SUMMARY The third year of the MHT SIG cross-site evaluation project was productive. Site visits were completed in Hawaii and Missouri, with site visit reports to be completed early in year 4. Close coordination with the States has continued regarding their POC studies. The project team completed focus groups in all States (with associated reports completed for the first cohort of States), the leadership survey for the first cohort of States, and the pilot provider survey in all but one State. Other support provided to the MHT SIG grantees included launching and maintaining Transformation Tracker Explorer and participating in the annual grantee meeting. The cross-site evaluation team values the important work in the States to realize transformation of their mental health systems and meet New Freedom Commission goals. The team is enthusiastic about collaborating with the States and continuing the evaluation in the upcoming year. MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 40 Table 20. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Mental Health Program/Services/Systems/Clients Topic CT HI MD MO NM OH OK TX WA Mental Health First Aid N N Y Y Y N Y N N State survey of evidence-based practices N N N Y Y N Y N Y Consumer surveys on recovery orientation N N N N Y N Y N Y Wraparound Fidelity Assessment System (WFAS) Y N Y N N N N N N Systems of care N N Y N N N Y N N Expanded school mental health program N N Y N Y N N N N Early Childhood Mental Health Consultation Evaluation N N Y N Y N N N N Evidenced-based practices fidelity and outcomes evaluation for a number of specific programs N N Y N Y N N N N Collection of standardized information on all MHT SIG projects N N N N Y N N Y N Evaluation of Program of Assertive Community Treatment (PACT) teams statewide N N N N N N Y N Y Indepth interviews with consumers, youth, and family members regarding the current mental health system and needed improvements N N N N N N Y N Y Consumer-, youth-, and family-driven Quality Improvement Collaborative to identify and implement recovery- and resilience-oriented performance measurements for the evaluation of the mental health service delivery models and programs Y N N N N N N N N Use of seclusion and restraint Y N N N N N N N N Behavioral Workforce Collaborative Y N N N N N N N N Interagency determination of questions about mental health curricula to add to CDC biannual School Health Profile Y N N N N N N N N Followup study of three Co-Occurring State Incentive Grant (CO-SIG) sites (in process) N Y N N N N N N N Community minigrants N Y N N N N N N N Department of Human Resources/Transformation Level of Intensity Evaluation— Development of tool to evaluate youth in group homes, treatment foster care, and independent living N N Y N N N N N N Project Connections, a community mental health outreach program for persons living in communities with high rates of poverty, violence, and instability N N Y N N N N N N RESPECT program N N N Y N N N N N Regional Behavioral Health Initiative activities regarding cultural competence/ RESPECT program N N N Y N N N N N Communities of Hope program (planned) N N N Y N N N N N State survey of provider needs N N N Y N N N N N Drop-in program analysis N N N Y N N N N N Behavioral health recovery studies N N N N Y N N N N Youth suicide prevention programming N N N N Y N N N N Hospital to community trauma-informed care study (planned) N N N N N Y N N N Statewide consumer focus group interviews for insight into perceptions of treatment planning and instrumentation development N N N N N Y N N N Access to mental health and substance abuse services N N N N N N Y N N MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 41 Topic CT HI MD MO NM OH OK TX WA Development and implementation of centralized electronic health record and reporting tool by local mental health authorities and substance abuse treatment providers N N N N N N N Y N Determination of the effectiveness of mental health infrastructure innovations and related factors using the Texas MHT Conceptual Model N N N N N N N Y N Development and assessment of use of Web-based comprehensive listing of evidence-based practices N N N N N N N Y N Service integration via use of liaison position to navigate consumers through local mental health authorities and primary care at Federally Qualified Health Centers N N N N N N N Y N General Assistance—Unemployable Clients: Challenges and Opportunities N N N N N N N N Y Washington’s Family Integrated Transition program N N N N N N N N Y Evaluation of King County Rethinking Care Pilot Project (to improve access to mental health services for Federally Qualified Health Center clients) N N N N N N N N Y Evaluation of statewide wraparound pilot for youth with serious emotional and behavioral disorders and their families N N N N n N N N Y Table 21. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Stigma Topic CT HI MD MO NM OH OK TX WA Evaluation of impact or effectiveness of statewide social marketing campaign Y Y N Y N N Y N Y Antistigma Y N Y N N N Y N N Stigma surveys N N N Y Y N Y N N Survey of provider organizations regarding stigma N N N N N N N N Y Table 22. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Funding & Resources Topic CT HI MD MO NM OH OK TX WA Medicaid 1915c home and community-based waiver Y N Y N N N N N N Expedited Medicaid review for prisoners study N N N N N N Y N Y NARI: Fiscal Resource Inventory Addendum, August 2008 N Y N N N N N N N Provider documentation project report N N N N N Y N N N Implementation of E2SSB 5763 (county option to impose 1/10 of 1% sales tax for mental health and other services) N N N N N N N N Y MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 42 Table 23. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Workforce and Training Topic CT HI MD MO NM OH OK TX WA Peer specialists (satisfaction survey) N N N Y Y N N N N Connecticut Recovery Employment Consultation Service (focuses on recruiting, preparing, and placing consumer providers within human service agencies) Y N N N N N N N N Providers and supervisors in outpatient and inpatient settings in person-centered, rehabilitative services, and trauma-informed care Y N N N N N N N N Community stakeholders (parents, teachers, behavioral health staff, and judicial staff) in wraparound practices and principles Y N N N N N N N N University professors and graduate students in intensive in-home family treatment Y N N N N N N N N Supervision competencies Y Nn N N N N N N N Department of Labor health and human services staff in recovery and mental health Y N N N N N N N N Adult consumers to become recovery supporters Y N N N N N N N N Community minigrants N Y N N N N N N N Family Engagement Training Program N N Y N N N N N N Cultural competence trainings N N Y N N N N N N Family support workers (satisfaction survey) N N N Y N N N N N Peer specialists (whether interaction between giver and recipient is beneficial) N N N Y N N N N N Workforce N N N N N N Y N N Training—Texas Collaborative for Emotional Development in Schools/Education Service Centers Train-the-Trainer N N N N N N N Y N Survey of persons who completed consumer peer-to-peer counselor training N N N N N N N N Y MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 43 Table 24. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding CYF, CYF Organizations, & Community Stakeholders Topic CT HI MD MO NM OH OK TX WA Community minigrants Y Y N N N N N N Y Surveys on consumer and family member perspectives about transformation efforts N N N Y Y Y N N N Network of Care Web site—Monitoring of perceptions/use of information (about mental health and substance abuse treatment and services as well as resources for persons across the lifespan) or effectiveness Y N N Y N N N N N Leadership training to parents Y N N Y N N N N N Assessment of level of consumer, youth, and family involvement in grant activities and of ways it can be enhanced throughout the service delivery system Y N N N N N N N N Evaluation of Consumer/Youth/Family Research and Evaluation Network (e.g., transformation involvement survey, minigrant projects, focus group information from mental health agency disparities initiative) Y N N N N N N N N Consumer-run organizations N N Y N N N N N N Community Evaluation and Research Training Academy for consumers and community stakeholders N N N N Y N N N N Consumer participation in Local Collaborative process N N N N Y N N N N State Annual Local Collaborative Self-Assessment Survey N N N N Y N N N N State Pre K-6 Parent Drug Prevention Programs N N N N Y N N N N Family advocacy focus group evaluation N N N N N Y N N N Consumer/family member involvement standards N N N N N N Y N N Consumer Network and Training—Training and Technical Assistance Center N N N N N N N Y N Consumer, youth, and family conference (satisfaction survey) N Y N Y Y N N N N MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 44 Table 25. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Disparities, Prevalence, & Statistics Topic CT HI MD MO NM OH OK TX WA Geographic disparities in mental health workforce N N N Y N N Y N Y Disparities in mental health services between urban and rural communities N N N N N N Y N Y Data interoperability to identify and eliminate behavioral health disparities Y N N N N N N N N State alcohol consumption and consequences annual survey N N N N Y N N N N State epidemiological drug use surveys N N N N Y N N N N Analysis of statewide database on family health and mental health N N N N N Y N N N Business case for trauma-informed intervention programming N N N N N Y N N N Analysis of utilization data for transition-age youth N N N N N Y N N N Analysis of the Youth Risk Behavior Survey N N N N N Y N N N Table 26. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Criminal/Juvenile Justice Topic CT HI MD MO NM OH OK TX WA Prison reentry study N N N N N N Y N Y Mental Health and Juvenile Justice Consultant Technical Report N Y N N N N N N N Community minigrants N Y N N N N N N N Juvenile justice supportive housing project pilot survey—adolescents N N N N Y N N N N Criminal justice (to determine whether released persons are getting necessary and quality services and have appropriate outcomes) N N N N N N Y N N Data sharing to promote jail diversion N N N N N N N Y N Service integration—court competency restoration program N N N N N N N Y N Arrests Among Working-Age Disabled (General Assistance—Unemployable) Clients evaluation N N N N N N N N Y Implementation of SSB 5533 (jail diversion) N N N N N N N N Y MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 45 Table 27. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Transformation Working Groups and Committees Topic CT HI MD MO NM OH OK TX WA Formative feedback to MHT SIG partners about progress to date and ways to better meet CMHP objectives Y N N N Y Y N N N Transformation Working Group—satisfaction with meetings, transformation process, engagement, and suggestions for continued transformation N N Y Y N Y N N N Monthly reporting template as part of formative evaluation of grant activities Y N N N Y N N N N Summary Evaluation of Sub-Working Group Process N Y N Y N N N N N Interviews and surveys with Content Working Groups, other agencies, and community constituents to evaluate overall transformation effectiveness and process of transformation efforts N N N Y N Y N N N Detailed evaluation reports on major initiatives of the Resource Investment and Strategy Subcommittee Y N N N N N N N N Summary Evaluation of Task Groups Process (draft developed) N Y N N N N N N N Local Collaborative Process Assessment Survey of Local Collaborative leaders N N N N Y N N N N MHT SIG Summative and Sustainability Report N N N N Y N N N N Table 28. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Other Topics Topic CT HI MD MO NM OH OK TX WA Psychiatric advance directives Y N N N N N N N N Policy analysis N N N N N N Y N N Technology Use Among Community Collaboratives (completed) N N N N N N N Y N Tracking of numbers of consumers, youth, and family members hired as a result of MHT SIG initiatives Y N N N N N N N N Pilot Housing Resilience Survey N N N N Y N N N N MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 46 Appendix A: GPRA Targets Table 29. Cumulative GPRA Target Numbers17 GPRA Measure FY CT HI MD MO NM OH OK TX WA Program 118 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline 1 2007 15 Baseline 0 Baseline 2 2 34 14 13 80 1 2008 33 0 5 20 21 63 46 64 27 279 1 2009 39 1 7 53 22 171 65 77 35 470 1 2010 41 1 8 71 27 223 78 81 48 578 1 2011 N/A 1 N/A 80 N/A N/A N/A N/A N/A 587 219 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline 2 2007 0 Baseline 2 Baseline 150 580 1,186 352 930 3,200 2 2008 1,780 179 159 1,143 152 6,908 8,243 13,884 6,261 38,709 2 2009 2,385 1,040 1,277 42,488 1,287 8,700 9,158 17,597 7,561 91,493 2 2010 2,505 1,229 1,886 44,741 3,516 10,420 11,004 26,196 17,596 119,093 2 2011 N/A 1,768 N/A 47,346 N/A N/A N/A N/A N/A 122,237 320 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline 3 2007 0 Baseline 1 Baseline 17 0 11 15 3 47 3 2008 0 0 5 31 19 6 15 33 7 116 3 2009 2 0 6 42 19 7 21 56 9 162 3 2010 2 0 7 50 25 10 27 65 30 216 3 2011 N/A 0 N/A 53 N/A N/A N/A N/A N/A 219 421 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline 4 2007 1 Baseline 0 Baseline 2 0 47 10 1 61 4 2008 30 1 16 19 11 1 68 22 15 183 4 2009 63 6 47 36 47 5 78 34 15 331 4 2010 67 9 50 55 114 5 183 43 44 570 17 Please note that data are displayed as reported by the States and have not been cleaned by the cross-site evaluation team. Numbers may change upon cleaning of the data. “Missing” means that target numbers should have been entered but were not. “N/A” means there are/were no GPRA reporting requirements for a State for that year. Each target reflects infrastructure changes that are projected to be completed by September 30 of the target grant year; therefore, changes projected to be completed after September 30 count toward the next year, and changes projected to be completed after the end of a State’s grant do not appear in the table. Data were extracted from Transformation Tracker on October 1, 2009. For all GPRA indicators except GPRA 6, targets were derived by adding the target numbers listed by year within the “Target Population” column of the “Key Features of Infrastructure Activities” report of Transformation Tracker. 18 GPRA 1: Number of policy changes completed as a consequence of the CMHP 19 GPRA 2: Number of persons in the mental health care and related workforce who have been trained in service improvements recommended by the CMHP 20GPRA 3: Number of financing policy changes completed as a consequence of the CMHP 21GPRA 4: Number of organizational changes completed as a consequence of the CMHP MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 47 GPRA Measure FY CT HI MD MO NM OH OK TX WA Program 4 2011 N/A 9 N/A 56 N/A N/A N/A N/A N/A 571 522 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline 5 2007 0 Baseline 0 Baseline 16 0 2 23 0 41 5 2008 0 0 0 96 33 498 38 27 0 692 5 2009 11 1 3 6,709 66 514 192 27 10 7,533 5 2010 324 14 42 6,863 246 532 407 27 21 8,476 5 2011 N/A 14 N/A 6,873 N/A N/A N/A N/A N/A 8,486 623 2007 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline 6 2008 14,158 Baseline 4,826 Baseline 1,562 17,155 3,351 15,500 10,136 66,688 6 2009 14,474 Missing 4,906 Missing Missing 30,055 3,656 16,500 12,522 82,113 6 2010 14,802 Missing 4,986 Missing Missing 31,355 3,961 17,500 13,019 85,623 6 2011 N/A Missing N/A Missing N/A N/A N/A N/A N/A 85,623 724 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline 7 2007 0 Baseline 0 Baseline 0 515 86 3 1 605 7 2008 0 0 19 199 60 940 101 99 1 1,419 7 2009 4 11 79 661 98 1,239 281 290 12 2,675 7 2010 4 17 217 831 196 1,601 589 580 326 4,361 7 2011 N/A 21 N/A 1,046 N/A N/A N/A N/A N/A 4,580 22 GPRA 5: Number of organizations that regularly obtain and analyze data relevant to the goals of the CMHP 23 GPRA 6: Number of consumers and family members that are members of Statewide consumer- and family-run networks. For GPRA 6, targets were derived from the “GPRA 6 Membership Numbers” table within the “Overview of Infrastructure Activities” section of Transformation Tracker. 24 GPRA 7: Number of programs implementing practices consistent with the CMHP MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 48 Appendix B: GPRA Results Table 30. Cumulative GPRA Results/Accomplishments25 GPRA Measure Year CT HI MD MO NM OH OK TX WA Program 126 2006 0 N/A 0 N/A 0 0 14 1 0 15 1 2007 12 0 0 5 3 0 43 13 15 91 1 2008 33 0 5 21 6 18 57 57 32 229 1 2009 33 0 5 54 6 20 61 70 32 281 1 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 1 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11 227 2006 0 N/A 0 N/A 0 0 0 0 0 0 2 2007 0 0 22 0 0 180 406 395 2,602 3,605 2 2008 2,915 179 1,455 1,769 2,236 4,516 13,928 18,729 10,740 56,467 2 2009 2,920 1,219 2,319 45,534 2,236 7,016 15,177 22,764 17,475 116,660 2 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 2 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11 328 2006 0 N/A 0 N/A 0 0 8 0 0 8 3 2007 0 0 2 6 19 0 12 9 3 51 3 2008 0 0 5 28 32 2 20 43 17 147 3 2009 1 0 5 39 32 2 26 58 19 182 3 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 3 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11 429 2006 0 N/A 0 N/A 0 0 6 0 0 6 4 2007 1 0 4 4 0 0 26 15 2 52 4 2008 29 0 44 18 20 1 44 32 8 196 4 2009 29 0 52 35 20 1 52 44 34 267 4 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 25 Please note that data are displayed as reported by the States and have not been cleaned by the cross-site evaluation team. Numbers may change upon cleaning of the data. “Missing” means there were no results reported as of September 30 of the target year. “N/A” means there are/were no GPRA reporting requirements for a State for that year. Each result reflects infrastructure changes with impacts reported up to September 30 of the target year; therefore, changes completed after September 30 of a calendar year count toward the next year, and changes completed outside of a State’s grant period do not appear in the table. Data were extracted from Transformation Tracker on October 1, 2009. For all GPRA indicators except GPRA 6, results were derived by summing the reported “impact” fields in the downloaded “Activity Results” entries for each State. Results were summed by fiscal year based on reported completion date. 26GPRA 1: Number of policy changes completed as a consequence of the CMHP 27GPRA 2: Number of persons in the mental health care and related workforce who have been trained in service improvements recommended by the CMHP 28GPRA 3: Number of financing policy changes completed as a consequence of the CMHP 29GPRA 4: Number of organizational changes completed as a consequence of the CMHP MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 49 GPRA Measure Year CT HI MD MO NM OH OK TX WA Program 4 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11 530 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11 5 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 5 2009 0 0 17 6,772 65 0 39 9 18 6,920 5 2008 0 0 7 106 65 0 31 5 18 232 5 2007 0 0 0 0 30 0 2 5 0 37 5 2006 0 N/A 0 N/A 0 0 1 0 0 1 631 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11 6 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 6 2009 Missing Missing Missing Missing Missing Missing 4,871 Missing Missing 4,871 6 2008 14,386 1,520 6,431 909 1,562 17,476 3,754 13,256 7,679 66,973 6 2007 13,842 N/A 4,739 N/A Missing 19,395 3,046 14,700 6,689 62,411 732 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11 7 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 7 2009 0 0 159 702 67 838 167 316 255 2,504 7 2008 0 0 114 170 67 753 135 99 247 1,585 7 2007 0 0 0 0 0 117 58 4 1 180 7 2006 0 N/A 0 N/A 0 0 28 0 0 28 30GPRA 5: Number of organizations that regularly obtain and analyze data relevant to the goals of the CMHP 31GPRA 6: Number of consumers and family members that are members of Statewide consumer- and family-run networks. For GPRA 6, results were derived from the “GPRA 6 Membership Numbers” table within the “Overview of Infrastructure Activities” section of Transformation Tracker. 32GPRA 7: Number of programs implementing practices consistent with the CMHP MHT SIG Evaluation Project, Annual Report, October 22, 2009 50 Appendix C: Proof of Concept Studies Information Table 31. State Interventions Examined in Recovery (Adult) Study State Impacted Group Nonimpacted Group Connecticut Individualized Recovery Planning tools with advanced training, infrastructure support, technical assistance (TA)— professional and peer staff331 Individualized Recovery Planning tools with 2-day training (not receiving advanced training, infrastructure support, or TA) Maryland Training and consultation providers and consumers in psychiatric rehabilitation programs regarding Cultural Competence and elimination of racial disparities in service provision1 Psychiatric rehabilitation programs not receiving training New Mexico Integrated Dual Diagnosis Treatment (IDDT) for adults diagnosed with co-occurring disorders in nine Certified Comprehensive Community Support Services (CCSS) Contractor Sites Provider sites not yet trained or certified in IDDT or the CCSS Model Ohio Using Outcomes in Person-Centered Treatment Planning (training for staff and consumers) Wait-list agencies Oklahoma Illness Management and Recovery— structured course with guided curriculum for Psychiatric Rehabilitation Service consumers Wait-list Texas Self-directed/consumer-directed care— control over funds to purchase services Services as usual Washington Journey to Life Wellness/Home Team— peer support and education on recovery and use of wellness tools Sequential design (pre and post) will do concurrent (randomized) next year if funds permit 33Intervention aims to enhance existing intervention/programs. MHT SIG Evaluation Project, Annual Report, October 22, 2009 51 Table 32. State Interventions Examined in Resilience (Child) Study State Impacted Group Nonimpacted Group Connecticut Wraparound with infrastructure support1 Wraparound without infrastructure support Maryland Wraparound Service Model Local Coordinating Councils (jurisdictions not yet implementing WSM) New Mexico Clinical Home—Comprehensive Service Agency model for referrals from Juvenile Justice division, eventually Protective Services Division Youth assessment center with services as usual Ohio Trauma Affect Regulation Guide for Education and Therapy (TARGET)— manualized treatment and prevention for Department of Youth Services Juvenile Correctional Facilities Mental health services as usual Oklahoma Cross-agency care coordination oversight for children, youth, and their families with high-level service use, funding redirected from residential to community-based services Comparable population, randomized to services as usual Texas Enhancing Fidelity to a Resiliency and Disease Management System: staff training and consultation1 Resiliency and Disease Management programs without training and consultation Washington Mental health services in middle-school, school-based health centers Schools without school-based health centers Tables 33 through 36 provide information about the GPRA measures and NFC goals being addressed by the State POC studies. Shaded boxes indicate the specific measures that are addressed by each State’s recovery (Tables 33 and 34) and resilience (Tables 35 and 36) POC studies. Table 33. GPRA Measures Addressed Recovery (Adult) Studies State 1 2 3 4 5 6 7 Connecticut n Y(2) n n Y n n Maryland n Y n n n n Y New Mexico n n Y n n Y Ohio n Y n n n n n Oklahoma Y Y n Y Y n Y Texas Y n Y n Y n n Washington n Y Y n Y n Y Table 34. NFC Goals Addressed by Recovery (Adult) Studies NFC 1 2 3 4 5 6 Connecticut n Y n n n n Maryland Y Y Y n Y n New Mexico n Y n n Y n Ohio n Y n n n n Oklahoma Y Y n n Y n Texas n Y n n Y n Washington Y Y n n n n MHT SIG Evaluation Project, Annual Report, October 22, 2009 52 Table 35. GPRA Measures Addressed by Resilience (Child) Studies State 1 2 3 4 5 6 7 Connecticut Y Y n Y Y(2) n n Maryland n Y Y n Y n Y New Mexico n n n Y n n Y Ohio n Y n n n n n Oklahoma Y Y n Y Y n Y Texas n n n n n n n Washington1 Y Y Y Y Y Y Y Table 36. NFC Goals Addressed by Resilience (Child) Studies State 1 2 3 4 5 6 Connecticut n Y n n Y n Maryland Y Y Y Y Y n New Mexico n n n n Y n Ohio n n n Y Y Y Oklahoma n Y Y Y Y n Texas n Y n n n n Washington1 Y Y Y Y Y Y Tables 37 through 40 below present the instruments for measuring system- and individual-level recovery in adults and individual resilience in children selected by each site from the approved list. It is apparent that there is some variability, with no more than three sites using any one measure. Table 37. Individual-level Recovery Instruments Being Used by Grantee States Instrument States Using Recovery Markers Questionnaire (RMQ) CT, NM, WA Mental Health Recovery Measure (MHRM) MD, OK Ohio Mental Health Consumer Outcomes system OH Recovery Assessment Scale (RAS) TX Table 38. System-level Recovery Orientation Instruments Being Used by Grantee States Instrument States Using Recovery Self-Assessment Revised (RSA-R) CT, MD, OH Recovery Oriented System Indicators Measure (ROSI) WA Recovery Enhancing Environment Measures (REE) NM Elements of a Recovery Facilitating System (ERFS–Adult) OK, TX Optional/other: Discrimination Experience Subscale/ISMI WA Table 39. Individual-level Resilience Instruments Being Used by Grantee States Instrument States Using Developmental Assets Profile (DAP) CT, WA Behavioral and Emotional Resilience Scale (BERS-2) MD, TX Ohio Youth Problems, Functioning, and Satisfaction Scales OH, OK Resilience Scale NM Optional/other: Strengths and Difficulties Questionnaire OK MHT SIG Evaluation Project, Annual Report, October 22, 2009 53 Table 40. System-level Resilience Orientation Instruments Being Used by Grantee States (Optional) Instrument States Using Elements of a Recovery Facilitating System (ERFS–Child Adapted) CT Community Supports for Wrap-Around Inventory WA Recovery Self-Assessment Revised (RSA-R) OH Optional/Other: Consumer Perception of Program Cultural Competence MD MHT SIG Evaluation Project, Annual Report, October 22, 2009 54 Appendix D: National Outcome Measures from the CMHS Uniform Reporting System, 2004–2008 Table 41. NOM 2: Increased or Retained Employment and School Enrollment: Percentage of Adults With Known Status Who Are Employed State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut 24 24 24 25 -- + Hawaii 19 24 23 23 -- + Maryland -- 14 14 16 -- + Missouri 19 17 -- 15 -- - New Mexico 21 20 25 21 -- None Ohio 16 22 23 23 -- + Oklahoma 18 18 19 20 -- + Texas 15 16 17 17 -- + Washington 12 11 11 12 -- None United States 21 21 22 21 -- None Table 42. NOM 4: Increased Stability in Family and Living Conditions: Percentage of Adults With Known Status Who Are Living in Private Residence State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut 72.6 76.2 73.7 78.7 80.8 + Hawaii 75.0 65.3 65.9 67.2 70.0 - Maryland -- 77.1 79.2 80.1 80.8 + Missouri 33.0 -- -- 80.1 78.5 + New Mexico 90.1 85.8 89.6 89.7 90.9 + Ohio -- 82.3 83.2 83.4 82.6 + Oklahoma 85.3 85.8 86.5 87.9 89.7 + Texas 91.7 90.4 90.6 91.2 91.4 - Washington 78.6 79.5 78.0 77.9 78.1 - United States 74.9 79.6 79.5 80.1 80.8 + MHT SIG Evaluation Project, Annual Report, October 22, 2009 55 Table 43. NOM 5: Increased Access to Services: Percentage of Persons Served by Age: Percentage of Clients Who Are Under Age 18 or Are Age 65 or Over State Categories 2004 2005 2006 2007 2008 Percent of State/U.S. 2008 Population Connecticut Under 18 42.1 35.0 35.6 35.1 35.0 23.2 Age 65+ 4.7 5.6 4.3 4.2 4.4 13.7 Hawaii Under 18 21.0 16.6 22.1 16.7 15.6 22.1 Age 65+ 8.0 7.0 5.4 4.2 6.8 14.8 Maryland Under 18 48.1 47.0 50.7 45.0 43.8 23.8 Age 65+ 1.4 1.3 1.2 1.2 1.2 12.1 Missouri Under 18 20.4 21.2 22.5 22.4 21.9 24.0 Age 65+ 3.1 2.8 2.8 2.9 2.8 13.6 New Mexico Under 18 -- 43.1 40.1 40.9 42.3 25.3 Age 65+ 3.8 1.9 1.8 1.7 1.7 13.1 Ohio Under 18 33.8 34.2 33.8 34.1 34.0 23.8 Age 65+ 2.9 2.9 2.8 2.8 2.9 13.7 Oklahoma Under 18 11.4 12.9 13.3 12.9 13.4 24.9 Age 65+ 2.6 2.3 2.2 1.7 1.7 13.5 Texas Under 18 18.2 18.7 19.1 19.4 19.8 27.6 Age 65+ 3.2 3.1 2.9 2.8 2.7 10.2 Washington Under 18 27.4 27.5 27.4 26.7 26.1 23.5 Age 65+ 7.6 7.2 6.5 6.5 6.6 12.0 United States Under 18 25.0 24.8 24.6 27.0 27.0 24.3 Age 65+ 4.8 4.6 4.5 4.8 4.8 12.8 Table 44. NOM 5: Increased Access to Services: Percentage of Persons Served by Race/Ethnicity (Percent of Clients Who Are White) State 2004 2005 2006 2007 2008 Percent of State/U.S. 2008 Population Connecticut 59.2 59.3 55.2 63.0 65.1 83.9 Hawaii 27.2 26.6 26.5 24.6 23.9 27.3 Maryland 42.6 44.1 44.1 45.9 46.6 61.8 Missouri 76.1 75.8 74.7 75.1 73.6 84.9 New Mexico 62.4 45.2 54.8 51.4 50.2 75.8 Ohio 74.1 73.5 73.5 73.3 73.1 84.7 Oklahoma 78.1 78.2 78.4 78.8 78.9 77.3 Texas 74.9 75.6 74.6 74.6 74.5 74.7 Washington 71.5 71.1 69.8 69.2 69.2 83.7 United States 74.3 74.1 73.9 63.3 63.3 77.6 MHT SIG Evaluation Project, Annual Report, October 22, 2009 56 Table 45. NOM 6: Decreased Utilization of Psychiatric Inpatient Beds: Mental Health Hospital Utilization Rate per 1,000 State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut .53 .56 .46 .43 .43 negative Hawaii .27 .32 .33 .32 .30 + Maryland .67 .62 .60 .51 .47 negative Missouri 1.5 1.39 1.33 1.27 1.15 negative New Mexico .48 .56 .60 1.24 .54 + Ohio -- -- -- -- -- N/A Oklahoma .83 .95 .85 .72 .74 negative Texas .69 .68 .67 .65 .61 negative Washington .57 .53 .52 .53 .52 negative United States .61 .63 .59 .60 .59 negative Table 46. NOM 7: Increased Social Support/Social Connectedness: Percentages for Adults and Children/Families34 State Adult 2006 Adult 2007 Adult 2008 Adult 2006- 2008 Change Child/Family 2006 Child/Family 2007 Child/Family 2008 Child/Family 2006-2008 Change Connecticut 73 77 75 + 86 83 90 + Hawaii -- 67 69 + -- 75 83 + Maryland 68 73 69 + 83 84 86 + Missouri -- 69 69 None -- 81 83 + New Mexico -- 97 97 None 84 93 89 + Ohio 36 37 37 + -- -- -- N/A Oklahoma 93 93 95 + 91 -- 98 + Texas 62 63 61 - 79 77 72 negative Washington -- 62 63 + -- 86 86 None United States 73 72 73 None 79 84 86 + Table 47. NOM 8: Increased Positive Reporting by Clients About Outcomes: Percentage of Adult Consumers With Positive Reporting of Outcomes State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut 78 81 78 80 77 negative Hawaii 68 69 71 70 76 + Maryland 63 61 66 70 63 None Missouri -- 81 -- 71 70 negative New Mexico 74 70 80 91 83 + Ohio 68 69 67 85 68 None Oklahoma 96 96 97 90 92 negative Texas 60 60 53 57 55 negative Washington 58 58 66 65 60 + United States 71 71 71 71 72 + 34 Measures of Social Support/Social Connectedness had been in developmental status prior to 2006. MHT SIG Evaluation Project, Annual Report, October 22, 2009 57 Table 48. NOM 10: Increased Use of Evidence-Based Practices: Percentage of States Implementing Adult Supported Housing State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut -- 9.8 9.8 10.7 9.9 + Hawaii -- 6.6 7.6 6.1 4.6 negative Maryland -- 18.6 14.4 14.8 15.2 negative Missouri -- -- -- -- -- N/A New Mexico 3.2 2.4 -- -- 0.3 negative Ohio -- -- -- -- -- N/A Oklahoma 1.0 0.9 0.8 0.6 0.5 negative Texas 4.4 2.7 2.3 2.8 2.7 negative Washington 0.3 -- -- 4.1 5.4 + United States 4.7 5.0 5.5 5 3 negative Table 49. NOM 10: Increased Use of Evidence-Based Practices: Percentage of States Implementing Adult Supported Employment State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut -- -- -- -- 3.8 N/A Hawaii -- 6.1 6.5 4.3 3.2 negative Maryland 57.91 4.3 4.5 5.3 5.7 + Missouri -- -- -- 2.7 9.1 + New Mexico 9.6 6.6 2.8 2.2 1.7 negative Ohio -- 100351 1.7 1.7 1.5 negative Oklahoma 0.2 0.1 0.1 0.1 0.1 negative Texas 3.5 2.7 1.7 1.6 1.2 negative Washington 1.8 -- -- 2.9 3.1 + United States 3.1 2.5 2.7 2 2 negative Table 50. NOM 10: Increased Use of Evidence-Based Practices: Percentage of States Implementing Adult Assertive Community Treatment State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut -- -- -- -- -- N/A Hawaii -- 7.6 6.7 4.5 3.5 negative Maryland -- 3.2 3.8 4.3 4.6 + Missouri -- -- -- -- 0.2 N/A New Mexico -- -- 0.2 0.2 0.3 + Ohio -- -- -- 1.7 1.8 + Oklahoma -- 1.9 2.4 2.9 2.4 + Texas -- 2.6 2.1 2.1 1.8 negative Washington -- -- -- 1.2 1.3 + United States 2.2 2.2 2.8 3 2 negative 35These data appear to be in error. MHT SIG Evaluation Project, Annual Report, October 22, 2009 58 Table 51. NOM 10: Increased Use of Evidence-Based Practices: Percentage of States Implementing Therapeutic Foster Care for Children and Adolescents State 2004 2005 2006 2007 2008 2004–2008 Change Connecticut -- 5.0 5.0361 -- -- None Hawaii -- 12.6 9.1 10.3 12.6 None Maryland -- -- -- -- 5.2 N/A Missouri -- -- -- 1.7 -- N/A New Mexico -- 5.8 5.0 4.2 5.4 negative Ohio -- -- -- -- -- N/A Oklahoma -- -- -- -- -- N/A Texas -- -- -- -- -- N/A Washington -- -- -- 1.8 1.9 + United States -- 2.1 2.0 3 2 negative 362006 figure for Connecticut is based on communication with State staff. |
Date created | 2011-07-27 |
Date modified | 2011-07-27 |
Tags
Add tags for Cross Site Evaluation - Annual Evaluation Report (Year 3), October, 2009