Evaluation of the Mental Health Transformation State Incentive Grant (MHT SIG) Program : annual evaluation report.

              Evaluation of the Mental Health Transformation State Incentive Grant
(MHT SIG) Program
Annual Evaluation Report
October 22, 2009
Prepared for
The Department of Health and Human Services
Substance Abuse and Mental Health Services Administration
Center for Mental Health Services
Prepared by
MANILA Consulting Group, Inc., and
Human Services Research Institute (HSRI)
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009
Contents
INTRODUCTION .......................................................................................................................................... 1
ACCOMPLISHMENTS ................................................................................................................................ 1
Transformation Tracker Guidance and Support ....................................................................................... 1
POC Guidance and Support ..................................................................................................................... 1
State Site Visits ........................................................................................................................................ 1
Federal Agency Site Visit .......................................................................................................................... 2
Consumer and Family Member Focus Groups ......................................................................................... 2
Evaluation Work Group Meetings ............................................................................................................. 2
2008 Annual Grantee Meeting .................................................................................................................. 2
Active Involvement of Consumer and Family Member Consultants ......................................................... 3
MHT SIG Web Site .................................................................................................................................. 3
Leadership Survey ................................................................................................................................... 3
Provider Survey ....................................................................................................................................... 3
Other Accomplishments........................................................................................................................... 4
FINDINGS .................................................................................................................................................... 4
Transformation Tracker Data: GPRA Activities and the President’s New Freedom Commission Goals . 4
Proof of Concept Studies ........................................................................................................................ 18
State Site Visits ...................................................................................................................................... 25
Federal Agency Site Visit ........................................................................................................................ 26
Consumer, Youth, and Family Member Focus Groups .......................................................................... 28
National Outcome Measures Data ......................................................................................................... 34
Analysis of State Employment Initiatives ................................................................................................ 36
Leadership Survey ................................................................................................................................. 36
Provider Survey ..................................................................................................................................... 38
State Specific Evaluation Activities ......................................................................................................... 38
SUMMARY ................................................................................................................................................. 39
APPENDIX A: GPRA TARGETS ............................................................................................................... 46
APPENDIX B: GPRA RESULTS ................................................................................................................ 48
APPENDIX C: PROOF OF CONCEPT STUDIES INFORMATION ........................................................... 50
APPENDIX D: NATIONAL OUTCOME MEASURES FROM THE CMHS UNIFORM REPORTING
SYSTEM, 2004–2008 ................................................................................................................................ 54
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009
Tables
TABLE 1. 2007-2009 NUMBER OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ........................................ 6
TABLE 2. 2007-2009 PERCENTAGE OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ................................. 7
TABLE 3. 2007 NUMBER OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ................................................. 8
TABLE 4. 2007 PERCENTAGE OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS .......................................... 9
TABLE 5. 2008 NUMBER OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ............................................... 10
TABLE 6. 2008 PERCENTAGE OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ........................................ 11
TABLE 7. 2009 NUMBER OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ............................................... 12
TABLE 8. 2009 PERCENTAGE OF GPRA ACTIVITIES ASSOCIATED WITH NFC GOALS ........................................ 13
TABLE 9. SUMMARY OF 2007 TRANSFORMATION ACTIVITIES BY NEW FREEDOM COMMISSION GOALS ............... 15
TABLE 10. SUMMARY OF 2008 TRANSFORMATION ACTIVITIES BY NEW FREEDOM COMMISSION GOAL ............... 16
TABLE 11. SUMMARY OF 2009 TRANSFORMATION ACTIVITIES BY NEW FREEDOM COMMISSION GOAL ............... 17
TABLE 12. STATUS OF RECOVERY STUDIES .................................................................................................... 18
TABLE 13. STATUS OF RESILIENCY STUDIES ................................................................................................... 19
TABLE 14. STATUS OF STATE PROOF OF CONCEPT STUDIES ........................................................................... 21
TABLE 15. RANKING OF NFC RECOMMENDATIONS BY MHT SIG PROGRAM MANAGERS AND CONNECTORS ...... 27
TABLE 16. AVERAGE CYF RATING OF FREQUENCY AND TYPE OF CONSUMER, YOUTH, AND FAMILY MEMBER
INVOLVEMENT IN GRANT ACTIVITIES ...................................................................................................... 29
TABLE 17. AVERAGE CFY RATINGS REGARDING VARIOUS ASPECTS OF CFY PARTICIPATION IN MHT SIG GRANT
ACTIVITIES .......................................................................................................................................... 31
TABLE 18. NUMBER OF MHT SIG GRANTEE STATES REPORTING INCREASES, DECREASES, AND NO CHANGES IN
THEIR NOMS DATA FROM 2004 TO 2008 .............................................................................................. 35
TABLE 19. RESULTS OF PILOT TEST OF PROVIDER SURVEY ............................................................................. 38
TABLE 20. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES
REGARDING MENTAL HEALTH PROGRAM/SERVICES/SYSTEMS/CLIENTS .................................................. 40
TABLE 21. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES
REGARDING STIGMA ............................................................................................................................. 41
TABLE 22. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES
REGARDING FUNDING & RESOURCES .................................................................................................... 41
TABLE 23. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES
REGARDING WORKFORCE AND TRAINING ............................................................................................... 42
TABLE 24. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES
REGARDING CYF, CYF ORGANIZATIONS, & COMMUNITY STAKEHOLDERS ............................................... 43
TABLE 25. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES
REGARDING DISPARITIES, PREVALENCE, & STATISTICS .......................................................................... 44
TABLE 26. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES
REGARDING CRIMINAL/JUVENILE JUSTICE .............................................................................................. 44
TABLE 27. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES
REGARDING TRANSFORMATION WORKING GROUPS AND COMMITTEES .................................................... 45
TABLE 28. STATES IMPLEMENTING OR PLANNING TO IMPLEMENT STATE-SPECIFIC EVALUATION ACTIVITIES
REGARDING OTHER TOPICS .................................................................................................................. 45
TABLE 29. CUMULATIVE GPRA TARGET NUMBERS ......................................................................................... 46
TABLE 30. CUMULATIVE GPRA RESULTS/ACCOMPLISHMENTS ........................................................................ 48
TABLE 31. STATE INTERVENTIONS EXAMINED IN RECOVERY (ADULT) STUDY .................................................... 50
TABLE 32. STATE INTERVENTIONS EXAMINED IN RESILIENCE (CHILD) STUDY .................................................... 51
TABLE 33. GPRA MEASURES ADDRESSED RECOVERY (ADULT) STUDIES ........................................................ 51
TABLE 34. NFC GOALS ADDRESSED BY RECOVERY (ADULT) STUDIES ............................................................. 51
TABLE 35. GPRA MEASURES ADDRESSED BY RESILIENCE (CHILD) STUDIES ................................................... 52
TABLE 36. NFC GOALS ADDRESSED BY RESILIENCE (CHILD) STUDIES ............................................................ 52
TABLE 37. INDIVIDUAL-LEVEL RECOVERY INSTRUMENTS BEING USED BY GRANTEE STATES .............................. 52
TABLE 38. SYSTEM-LEVEL RECOVERY ORIENTATION INSTRUMENTS BEING USED BY GRANTEE STATES ............ 52
TABLE 39. INDIVIDUAL-LEVEL RESILIENCE INSTRUMENTS BEING USED BY GRANTEE STATES ............................. 52
TABLE 40. SYSTEM-LEVEL RESILIENCE ORIENTATION INSTRUMENTS BEING USED BY GRANTEE STATES
(OPTIONAL) ......................................................................................................................................... 53
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009
TABLE 41. NOM 2: INCREASED OR RETAINED EMPLOYMENT AND SCHOOL ENROLLMENT: PERCENTAGE OF
ADULTS WITH KNOWN STATUS WHO ARE EMPLOYED ............................................................................. 54
TABLE 42. NOM 4: INCREASED STABILITY IN FAMILY AND LIVING CONDITIONS: PERCENTAGE OF ADULTS WITH
KNOWN STATUS WHO ARE LIVING IN PRIVATE RESIDENCE ..................................................................... 54
TABLE 43. NOM 5: INCREASED ACCESS TO SERVICES: PERCENTAGE OF PERSONS SERVED BY AGE:
PERCENTAGE OF CLIENTS WHO ARE UNDER AGE 18 OR ARE AGE 65 OR OVER ...................................... 55
TABLE 44. NOM 5: INCREASED ACCESS TO SERVICES: PERCENTAGE OF PERSONS SERVED BY RACE/ETHNICITY
(PERCENT OF CLIENTS WHO ARE WHITE) .............................................................................................. 55
TABLE 45. NOM 6: DECREASED UTILIZATION OF PSYCHIATRIC INPATIENT BEDS: MENTAL HEALTH HOSPITAL
UTILIZATION RATE PER 1,000 ................................................................................................................ 56
TABLE 46. NOM 7: INCREASED SOCIAL SUPPORT/SOCIAL CONNECTEDNESS: PERCENTAGES FOR ADULTS AND
CHILDREN/FAMILIES .............................................................................................................................. 56
TABLE 47. NOM 8: INCREASED POSITIVE REPORTING BY CLIENTS ABOUT OUTCOMES: PERCENTAGE OF ADULT
CONSUMERS WITH POSITIVE REPORTING OF OUTCOMES ....................................................................... 56
TABLE 48. NOM 10: INCREASED USE OF EVIDENCE-BASED PRACTICES: PERCENTAGE OF STATES IMPLEMENTING
ADULT SUPPORTED HOUSING ................................................................................................................ 57
TABLE 49. NOM 10: INCREASED USE OF EVIDENCE-BASED PRACTICES: PERCENTAGE OF STATES IMPLEMENTING
ADULT SUPPORTED EMPLOYMENT ......................................................................................................... 57
TABLE 50. NOM 10: INCREASED USE OF EVIDENCE-BASED PRACTICES: PERCENTAGE OF STATES IMPLEMENTING
ADULT ASSERTIVE COMMUNITY TREATMENT .......................................................................................... 57
TABLE 51. NOM 10: INCREASED USE OF EVIDENCE-BASED PRACTICES: PERCENTAGE OF STATES IMPLEMENTING
THERAPEUTIC FOSTER CARE FOR CHILDREN AND ADOLESCENTS ............................................................ 58
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009
Figures
FIGURE 1. AVERAGE RATING OF GRANT IMPACT BY STATE .............................................................................. 25
FIGURE 2. AVERAGE RATING OF GRANT IMPACT BY TYPE OF STATE AGENCY ................................................... 26
FIGURE 3. AVERAGE RATINGS ON STATEMENTS PROVIDED BY CONSUMER AND FAMILY MEMBER FOCUS GROUP
PARTICIPANTS ACROSS ALL STATES ...................................................................................................... 34
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 1
INTRODUCTION
This annual report identifies accomplishments and findings related to the third year of the Mental Health
Transformation State Incentive Grant (MHT SIG) cross-site evaluation project. The third year of the
project began September 15, 2008, and ended September 14, 2009. The evaluation of the MHT SIG
program is designed to measure the success of transformation efforts in grantee States to meet New
Freedom Commission (NFC) goals. The project supports nine States as they implement their MHT SIG
grants. The first cohort of States, awarded Center for Mental Health Services (CMHS) grants in 2005,
includes seven States (Connecticut, Maryland, New Mexico, Ohio, Oklahoma, Texas and Washington).
The second cohort, awarded grants in 2006, includes two States (Hawaii and Missouri).
ACCOMPLISHMENTS
During this year of the project, the project team—MANILA Consulting Group, Inc. (MANILA), and Human
Services Research Institute (HSRI)—focused efforts on completing the consumer, youth, and family focus
groups in all nine grantee States and developing associated reports, conducting site visits to Hawaii and
Missouri, launching Transformation Tracker Explorer, coordinating with States regarding Proof of Concept
(POC) study requirements, completing leadership and provider surveys, and examining methods for
conducting the various cross-site analyses for the final report.
Transformation Tracker Guidance and Support
The cross-site evaluation collects information in Transformation Tracker on planned MHT SIG activities,
including the nature of the activities, their scheduling, and whether or not they are completed. These
activities are coded by States as to New Freedom Commission Goals addressed and SAMHSA
Government Performance and Results Act (GPRA) categories. The MHT SIG program has seven
Government Performance and Results Act (GPRA) categories of infrastructure changes for which States
must provide annual progress updates: policy changes, workforce trainings, financial policy changes,
organizational changes, data sharing and analysis, membership in consumer/family member (CFM)
networks, and implementation of practices consistent with the State’s Comprehensive Mental Health Plan
(CMHP). The project team provided detailed guidance and support to Hawaii and Missouri as they
completed initial entry of their activities into Transformation Tracker. In addition, guidance was provided to
other States regarding various issues with their Transformation Tracker entries and GPRA 6 (membership
in CFM networks) supporting information. Additionally, the cross-site team began discussions of how to
clean, code, and analyze the Transformation Tracker data from the perspectives of New Freedom
Commission Goals and Recommendations and SAMHSA GPRA categories.
POC Guidance and Support
The evaluation design requires States to identify services for adults with serious mental illness and
children/youth with serious emotional disturbance that have been or are anticipated to be impacted by the
MHT SIG project (the impacted group) as well as comparable services that have not been impacted by
the project (the nonimpacted group). The goal is to determine if positive changes in recovery (for adults)
and resilience (for youth) result from “best case” transformative initiatives consisting of both infrastructure
and service changes. During the past year, for these recovery and resiliency studies, the project team
focused on different activities by cohort. For the first cohort of States, the evaluation team clarified details
of their POC proposals and approved completed proposals, completed successful test data submissions
for all instruments to be used, supported and monitored data collection efforts and early stages of data
submission, and developed preliminary data analysis plans based on anticipated data collection rates.
For the second cohort of States, the primary focus was on supporting the development of their POC
proposals.
State Site Visits
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 2
State site visits are conducted to determine the effect the MHT SIG projects have had on their mental
health service systems, to understand why the States have chosen the approaches they have, and to
learn about any barriers or needs associated with the grant program. The initial site visit with each
grantee is conducted in grant year 3, with a second one occurring in grant year 5. The cross-site
evaluators conducted site visits in Hawaii on December 17 and 18, 2008, and in Missouri on January 28
and 29, 2009. Draft site visit reports were prepared and provided to both States.
Federal Agency Site Visit
From December 2008 to early March 2009, the project team interviewed key CMHS managers, all six
Connectors (the MHT SIG program’s Federal Project Officers), and two other CMHS staff to understand
the development and implementation of the MHT SIG program to date. All interviewees also were asked
to rank the New Freedom Commission recommendations in order of importance.
Consumer and Family Member Focus Groups
Consumer and family member focus groups with up to 15 individuals per State were designed to identify
the nature of consumer/family member involvement in transformation grant activities at both the State and
local levels. Within each State, the focus groups were to consist of persons who were aware of the
grant—some who were involved in grant activities and some who were not. The process of identifying
CFM focus group participants and scheduling meetings was found to be more difficult than anticipated. In
particular, it was challenging to find individuals who were aware of the mental health transformation
grants, regardless of whether they were involved in grant activities. Other difficulties with recruitment
included reluctance to participate due to limited familiarity with the Federal grant program and lack of
responsiveness from CFM organizations that were expected to be potential sources of participants. In
addition, some persons who initially indicated interest in participation did not respond to subsequent
followup attempts.
Focus groups were conducted during the State site visits with all the CFMs who were available. All other
focus group interviews were completed by phone afterwards. By the end of the third year, all focus group
interviews had been completed in all States. Reports for focus groups with the first cohort of States were
provided on July 17, and public versions of these reports were developed and provided on September 3.
Reports for Hawaii and Missouri are under development.
Evaluation Work Group Meetings
Evaluation Work Group meetings provide a forum in which State evaluators can update the evaluation
team on the status of their efforts, identify issues and needs associated with their project, and obtain
peer-to-peer support. During the third project year, 11 Evaluation Work Group meetings were held by
telephone. Calls were used to provide guidance on POC studies and discuss implementation issues as
well as the status of State-specific evaluation studies, the status of leadership and provider surveys, the
potential development and subsequent results of launching the publicly available Transformation Tracker
Explorer Web site, State-specific evaluation activities, and topics for the November 2009 Annual Grantee
Meeting. Participation by evaluators from all States was consistently strong for all of the meetings.
2008 Annual Grantee Meeting
The 2008 Annual Grantee Meeting was conducted on November 12 through 14 in Bethesda, Maryland.
This meeting provided an opportunity for program staff and grantees to share accomplishments and
identify objectives for the coming year. Three sessions were devoted to evaluation topics:
 Cross-Site Evaluation Progress Report: The Project Officer and Principal Investigator provided
information about accomplishments in the cross-site evaluation and preliminary findings from the
State planned MHT SIG activity data.
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 3
 Evaluation Informing Grant Implementation: Challenges and Opportunities: Evaluators from
Connecticut, Hawaii, Oklahoma, and Washington made presentations about strengths-based
approaches that can facilitate the process of evaluation and grant project activities informing one
another.
 Cross-Site Evaluators Meeting: With no preset agenda for this session, Hawaii, Maryland, New
Mexico, Ohio, Oklahoma, and Washington used the time to describe particular evaluation
activities/initiatives in their States. The session also included general discussion about disparities
in access to mental health services, POC issues, State-specific evaluation activities, and methods
for determining when transformation has occurred.
Active Involvement of Consumer and Family Member Consultants
The evaluation project has six CFM Consultants who participate in all aspect of the project to ensure that
CFM perspectives are incorporated into the design and implementation of the evaluation (e.g., site visits,
focus groups, Evaluation Work Group calls, annual grantee meetings). During the third project year, the
CFM Consultants were involved in the two State site visits and associated focus group interviews. They
also reviewed the site visit and focus group reports for the States they visited. Three of the Consultants
were able to participate in the cross-site team meeting in Boston in August 2009. One Consultant has
been active with the MHT SIG Consumer, Family, and Youth Work Group and has participated in many of
its monthly conference calls. The Consultants also reviewed and commented on the Transformation
Tracker Explorer test site prior to its launch in May 2009.
MHT SIG Web Site
This Web site was developed early in the evaluation project to provide grantees with resource and project
information and a system in which States could enter their MHTSIG activities. The major change to the
Web site this year was the launch of Transformation Tracker Explorer (http://mhtsigdata.samhsa.gov) on
May 19, 2009. This is the public part of the MHT SIG evaluation project Web site that allows the public to
see, rate, and comment on GPRA activities being undertaken in all nine grantee States. This Web site is
thought to be one of the first of its kind, providing current information about grantee activities and
accomplishments during the course of an evaluation project. On August 18, 2009, the project team
submitted a 90-day report detailing Web site traffic to date and identifying the MHTSIG activity ratings and
comments that had been submitted. By the end of the third year, there were approximately 1000 visits by
approximately 500 unique individuals from 42 States and 16 other countries.
Leadership Survey
Conducted in grant years 3 and 5, the leadership survey is designed to capture leadership characteristics
of persons heading up the MHT SIG transformation efforts within the States. “Leaders” are the Project
Directors and Transformation Working Group (TWG) chairs, while “raters” are TWG members or TWG
subcommittee members. For the first cohort of States, the online leadership survey was launched in
October 2008; 10 of 11 leaders and 51 of 76 raters completed the survey by March 2009. This survey will
be conducted in Hawaii and Missouri in November 2009 to maintain the same timing of the survey as for
the first cohort of States.
Provider Survey
The provider survey is designed to determine whether providers are aware of the MHT SIG grant in their
State, and if so, how their State has been affected by mental health transformation grant initiatives. In the
original evaluation design, each grantee was to survey 26 providers in each of grant years 3, 4, and 5. In
conjunction with the Project Officer, it was decided that a pilot survey of providers with a smaller group of
participants in year 3 would be a useful approach in understanding how providers might respond to the
survey questions and whether any additional probes might be useful. The pilot survey was completed in
all States except Hawaii; three of four provider surveys in Hawaii still need to be completed. The
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 4
remaining surveys will be completed during grant years 4 and 5, beginning in late September 2009.
Providers will be encouraged to complete the survey online to make efficient use of staff time (i.e., staff
will not need to schedule and conduct phone surveys or perform data entry).
Other Accomplishments
Other project accomplishments include the following:
 An analysis of employment initiatives by grantee States was done in January 2009 to determine
the types of initiatives being undertaken, including participation in competitive labor force efforts
and any U.S. Department of Labor initiatives.
 Presentation at the National Association of State Mental Health Program Directors (NASMHPD)
Research Institute: On April 15, 2009, the Project Officer and a project staff member presented
information about the MHT SIG evaluation project and preliminary findings on integration of
mental health and physical health care.
 Meeting with MHT SIG Connectors: On April 21, 2009, project staff met with the Connectors to
discuss the ways GPRA information could be used for program management purposes and the
plans to launch Transformation Tracker Explorer.
 Support for the MHT SIG Consumer, Family, and Youth Work Group: At the request of the MHT
SIG Consumer, Family, and Youth Advisor/Consultant, the project staff provided a demonstration
to this group about how to use Transformation Tracker Explorer during a conference call on May
20, 2009.
 Cross-Site Team Meeting: On August 25 and 26, 2009, the entire cross-site team met to discuss
and make decisions about the cross-site analyses to be completed by the end of the project. The
Project Officer attended this meeting along with three of the six CFM Consultants.
 Participation on Planning Committee for the 2009 Annual Grantee Meeting: In July and August
2009, the project participated in Planning Committee meetings to help determine agenda topics
and presenters for the upcoming November 2009 meeting.
 Analysis of Other State Data: Public health epidemiological datasets to measure statewide
outcomes were obtained from the following other state data sources: SAMHSA’s National Survey
on Drug Use and Health (NSDUH) for 2002 to 2007 and Behavioral Risk Factor Surveillance
Survey (BRFSS) from the Centers for Disease Control and Prevention (CDC) for 2002 to 2008.
These datasets and other public data sources are being examined to identify relevant mental
health outcomes and relate them to CMHS National Outcomes Measures (NOMs). We also are
reviewing other longitudinal datasets (e.g., the data on which the NAMI Grading the States
Reports were based) that might have information on outcomes or infrastructure changes in
States. In addition to analytic approaches that compare MHTSIG States with Non-MHTSIG
control States over time, we are also exploring “dose-effect” analytic methods for analyzing the
associations between transformative practices and infrastructure and outcome changes within
MHTSIG States.
FINDINGS
The findings from the third year of the cross-site evaluation project are discussed in the following sections
on Transformation Tracker data, POC studies, State site visits, Federal agency site visit, consumer and
family member focus groups, statewide National Outcome Measures (NOMs) data, analysis of State
employment initiatives, leadership survey, provider survey, and State-specific evaluation activities.
Transformation Tracker Data: GPRA Activities and the President’s New Freedom
Commission Goals
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 5
The cross-site evaluation project has tracked MHT SIG State activities in Transformation Tracker, where
entries are coded by GPRA measure and by the President’s New Freedom Commission goal. Given the
central role of the NFC report in the origination of the MHT SIG program, Tables 1 through 8 below
present a cross-tabulation of MHTSIG activities by NFC goal for entries scheduled to be completed in the
2007 to 2009 fiscal years and for each of the years separately. Each activity item can be assigned to only
one GPRA category but may be assigned to multiple NFC goals. Data in these tables as well as Tables 9
through 11 were extracted from Transformation Tracker on October 5, 2009. The individual percentages
in these tables may not total 100 percent due to rounding. Appendix A provides target activity information
by State and year, while Appendix B presents activity results by State and year.
All tables that describe GPRA and NFC goal activity in this section and in Appendices A and B are
provisional and are intended to illustrate the analyses that will eventually be implemented for the
summative cross-site evaluation. A primary reason for caution about the data in these tables is that, as of
yet, the data have not been “cleaned” (i.e., they have not been reviewed to ensure the accuracy of coding
or the presence of duplicate activities). In addition, the GPRA and NFC tables within this report do not
reflect State activities for the 2006 baseline year because the grantees were not required to report GPRA
data until 2007. At least one grantee State, however, has entered its baseline MHTSIG activities into
Transformation Tracker.
Tables 1 through 8 show numbers and percentages of GPRA activity items by NFC goal. If these data are
accurately coded by the States, the most frequently addressed NFC goals in 2007, 2008, and 2009 were
NFC 2 (care is consumer and family driven) and NFC 5 (excellent care is provided), while the least
frequently addressed ones were NFC 3 (disparities are eliminated) and NFC 6 (technology is used). They
also show that most State activities were classified under GPRA 1 (policy changes) and GPRA 2 (training
activities). Currently a total of 1,076 unique MHTSIG activity items have been entered in Transformation
Tracker as ending in fiscal years 2007 to 2009 across all States). As mentioned previously, this count
does not reflect baseline year activities.
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 6
Table 1. 2007-2009 Number of GPRA Activities Associated with NFC Goals
GPRA Measure
NFC Goal 1:
Mental Health
Is Essential
NFC Goal 2:
Care Is
Consumer/
Family Driven
NFC Goal 3:
Disparities
Are
Eliminated
NFC Goal 4:
Early
Screening Is
Provided
NFC Goal 5:
Excellent
Care Is
Provided
NFC Goal 6:
Technology Is
Used
GPRA 1: Policy Changes 82 72 23 60 79 37
GPRA 2: Training 101 147 57 78 118 15
GPRA 3: Financial Policy Changes 43 41 20 32 53 19
GPRA 4: Organizational Changes 47 55 19 47 44 17
GPRA 5: Data Sharing 8 17 3 11 23 13
GPRA 6: Consumer, Youth, & Family
Networks 3 29 0 0 2 0
GPRA 7: CMHP Practices 63 34 18 44 54 20
Non-GPRA 22 53 12 14 32 16
Total Number 369 448 152 286 405 137
Percent of Total Activities 21% 25% 8% 16% 23% 8%
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 7
Table 2. 2007-2009 Percentage of GPRA Activities Associated with NFC Goals
GPRA Measure
NFC Goal 1:
Mental Health
Is Essential
NFC Goal 2:
Care Is
Consumer/
Family Driven
NFC Goal 3:
Disparities
Are
Eliminated
NFC Goal 4:
Early
Screening Is
Provided
NFC Goal 5:
Excellent
Care Is
Provided
NFC Goal 6:
Technology Is
Used
GPRA 1: Policy Changes 22% 16% 15% 21% 20% 27%
GPRA 2: Training 27% 33% 38% 27% 29% 11%
GPRA 3: Financial Policy Changes 12% 9% 13% 11% 13% 14%
GPRA 4: Organizational Changes 13% 12% 13% 16% 11% 12%
GPRA 5: Data Sharing 2% 4% 2% 4% 6% 10%
GPRA 6: Consumer, Youth, & Family
Networks 1% 6% 0% 0% 1% 0%
GPRA 7: CMHP Practices 17% 8% 12% 15% 13% 15%
Non-GPRA 6% 12% 8% 5% 8% 12%
Total Number 100% 100% 100% 100% 100% 100%
Percent of Total Activities 21% 25% 8% 16% 23% 8%
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 8
Table 3. 2007 Number of GPRA Activities Associated with NFC Goals
GPRA Measure
NFC Goal 1:
Mental Health
Is Essential
NFC Goal 2:
Care Is
Consumer/
Family Driven
NFC Goal 3:
Disparities
Are
Eliminated
NFC Goal 4:
Early
Screening Is
Provided
NFC Goal 5:
Excellent
Care Is
Provided
NFC Goal 6:
Technology Is
Used
GPRA 1: Policy Changes 13 25 4 13 17 8
GPRA 2: Training 4 7 4 4 9 0
GPRA 3: Financial Policy Changes 9 9 3 8 11 5
GPRA 4: Organizational Changes 5 7 2 5 6 3
GPRA 5: Data Sharing 1 1 0 1 0 2
GPRA 6: Consumer, Youth, & Family
Networks 0 3 0 0 1 0
GPRA 7: CMHP Practices 6 2 1 4 3 1
Non-GPRA 0 2 1 1 1 5
Total Number 38 56 15 36 48 24
Percent of Total Activities 18% 26% 7% 17% 22% 11%
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 9
Table 4. 2007 Percentage of GPRA Activities Associated with NFC Goals
GPRA Measure
NFC Goal 1:
Mental Health
Is Essential
NFC Goal 2:
Care Is
Consumer/
Family Driven
NFC Goal 3:
Disparities
Are
Eliminated
NFC Goal 4:
Early
Screening Is
Provided
NFC Goal 5:
Excellent
Care Is
Provided
NFC Goal 6:
Technology Is
Used
GPRA 1: Policy Changes 34% 45% 27% 36% 35% 33%
GPRA 2: Training 11% 13% 27% 11% 19% 0%
GPRA 3: Financial Policy Changes 24% 16% 20% 22% 23% 21%
GPRA 4: Organizational Changes 13% 13% 13% 14% 13% 13%
GPRA 5: Data Sharing 3% 2% 0% 3% 0% 8%
GPRA 6: Consumer, Youth, & Family
Networks 0% 5% 0% 0% 2% 0%
GPRA 7: CMHP Practices 16% 4% 7% 11% 6% 4%
Non-GPRA 0% 4% 7% 3% 2% 21%
Total Number 100% 100% 100% 100% 100% 100%
Percent of Total Activities 18% 26% 7% 17% 22% 11%
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 10
Table 5. 2008 Number of GPRA Activities Associated with NFC Goals
GPRA Measure
NFC Goal 1:
Mental Health
Is Essential
NFC Goal 2:
Care Is
Consumer/
Family Driven
NFC Goal 3:
Disparities
Are
Eliminated
NFC Goal 4:
Early
Screening Is
Provided
NFC Goal 5:
Excellent
Care Is
Provided
NFC Goal 6:
Technology Is
Used
GPRA 1: Policy Changes 26 23 12 22 29 18
GPRA 2: Training 42 76 29 39 57 10
GPRA 3: Financial Policy Changes 11 17 10 14 29 11
GPRA 4: Organizational Changes 19 23 8 18 17 4
GPRA 5: Data Sharing 1 8 2 4 8 5
GPRA 6: Consumer, Youth, & Family
Networks 2 16 0 0 0 0
GPRA 7: CMHP Practices 18 10 7 14 17 7
Non-GPRA 12 30 7 6 16 9
Total Number 131 203 75 117 173 64
Percent of Total Activities 17% 27% 10% 15% 23% 8%
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 11
Table 6. 2008 Percentage of GPRA Activities Associated with NFC Goals
GPRA Measure
NFC Goal 1:
Mental Health
Is Essential
NFC Goal 2:
Care Is
Consumer/
Family Driven
NFC Goal 3:
Disparities
Are
Eliminated
NFC Goal 4:
Early
Screening Is
Provided
NFC Goal 5:
Excellent
Care Is
Provided
NFC Goal 6:
Technology Is
Used
GPRA 1: Policy Changes 20% 11% 16% 19% 17% 28%
GPRA 2: Training 32% 37% 39% 33% 33% 16%
GPRA 3: Financial Policy Changes 8% 8% 13% 12% 17% 17%
GPRA 4: Organizational Changes 15% 11% 11% 15% 10% 6%
GPRA 5: Data Sharing 1% 4% 3% 3% 5% 8%
GPRA 6: Consumer, Youth, & Family
Networks 2% 8% 0% 0% 0% 0%
GPRA 7: CMHP Practices 14% 5% 9% 12% 10% 11%
Non-GPRA 9% 15% 9% 5% 9% 14%
Total Number 100% 100% 100% 100% 100% 100%
Percent of Total Activities 17% 27% 10% 15% 23% 8%
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 12
Table 7. 2009 Number of GPRA Activities Associated with NFC Goals
GPRA Measure
NFC Goal 1:
Mental Health
Is Essential
NFC Goal 2:
Care Is
Consumer/
Family Driven
NFC Goal 3:
Disparities
Are
Eliminated
NFC Goal 4:
Early
Screening Is
Provided
NFC Goal 5:
Excellent
Care Is
Provided
NFC Goal 6:
Technology Is
Used
GPRA 1: Policy Changes 43 24 7 25 33 11
GPRA 2: Training 55 64 24 35 52 5
GPRA 3: Financial Policy Changes 23 15 7 10 13 3
GPRA 4: Organizational Changes 23 25 9 24 21 10
GPRA 5: Data Sharing 6 8 1 6 15 6
GPRA 6: Consumer, Youth, & Family
Networks 1 10 0 0 1 0
GPRA 7: CMHP Practices 39 22 10 26 34 12
Non-GPRA 10 21 4 7 15 2
Total Number 200 189 62 133 184 49
Percent of Total Activities 25% 23% 8% 16% 23% 6%
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 13
Table 8. 2009 Percentage of GPRA Activities Associated with NFC Goals
GPRA Measure
NFC Goal 1:
Mental Health
Is Essential
NFC Goal 2:
Care Is
Consumer/
Family Driven
NFC Goal 3:
Disparities
Are
Eliminated
NFC Goal 4:
Early
Screening Is
Provided
NFC Goal 5:
Excellent
Care Is
Provided
NFC Goal 6:
Technology Is
Used
GPRA 1: Policy Changes 22% 13% 11% 19% 18% 23%
GPRA 2: Training 28% 34% 39% 26% 28% 10%
GPRA 3: Financial Policy Changes 12% 8% 11% 8% 7% 6%
GPRA 4: Organizational Changes 12% 13% 15% 18% 11% 20%
GPRA 5: Data Sharing 3% 4% 2% 5% 8% 12%
GPRA 6: Consumer, Youth, & Family
Networks 1% 5% 0% 0% 1% 0%
GPRA 7: CMHP Practices 20% 12% 16% 20% 19% 25%
Non-GPRA 5% 11% 7% 5% 8% 4%
Total Number 100% 100% 100% 100% 100% 100%
Percent of Total Activities 25% 23% 8% 16% 23% 6%
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 14
As noted earlier, the Transformation Tracker activity items are coded by NFC goal(s), and additional
information is provided about the history and current status of each activity. Tables 9, 10, and 11 below
summarize the most important historical information associated with the NFC activity items in 2007, 2008,
and 2009, respectively; each table presents, by NFC goal, the number of activities, the percentage of
States with those activities, the percentage of activities representing that goal across the States
(presented as a range), the percentage begun before the MHT SIG grant, the percentage raised in the
Needs Assessment and Resource Inventory (NARI), the percentage noted in the CMHP, the percentage
recommended by the TWG, and the percentage with results noted in Transformation Tracker. If these
data are accurately coded by the States, the characteristics of activity items associated with the NFC
goals have changed over time. For example, in 2007, activity items associated with NFC 2 (care is
consumer and family driven) were less likely to be recommended by TWGs than activity items associated
with other NFC goals. In 2008, a greater proportion of NFC 2 activity items had been TWG-approved,
though these items were still approved by the TWG at a lower rate than activities related to other NFC
goals. Interestingly, Goal 2 activities in 2009 had a much higher approval rate than in the previous 2
years, giving these items a TWG approval rate consistent with those for other NFC goal activities. One
possible explanation is that earlier activity items had not been presented to TWGs at the time the activity
items were entered into Transformation Tracker, and that this information has not yet been updated. The
cross-site evaluation team will investigate the reason for this pattern when conducting the data cleaning
for the summative evaluation.
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 15
Table 9. Summary of 2007 Transformation Activities by New Freedom Commission Goals
NFC Goals
Number
of
Activities
Percent
of
States
Percent
Range
of
Activity
Items
Percent
Begun
Before
MHT
SIG
[Yes]
Percent
Begun
Before
MHT
SIG [No]
Percent
Raised
in NARI
[Yes]
Percent
Raised
in NARI
[No]
Percent
Noted in
CMHP
[Yes]
Percent
Noted in
CMHP
[No]
Percent
Recomm
by TWG
[Yes]
Percent
Recomm
by TWG
[No]
Percent
of
Activities
With
Activity
Results
NFC 2:
Care Is
Consumer/
Family
Driven
54 89% 0-31% 19% 81% 61% 39% 80% 20% 50% 50% 96%
NFC 5:
Excellent
Care Is
Provided
48 89% 0-31% 19% 81% 88% 13% 94% 6% 73% 27% 98%
NFC 1:
Mental
Health Is
Essential
38 67% 0-50% 18% 82% 82% 18% 89% 11% 82% 18% 95%
NFC 4:
Early
Screening Is
Provided
36 67% 0-39% 25% 75% 83% 17% 97% 3% 64% 36% 94%
NFC 6:
Technology
Is Used
23 67% 0-35% 9% 91% 96% 4% 96% 4% 83% 17% 83%
NFC 3:
Disparities
Are
Eliminated
15 56% 0-60% 0% 100% 87% 13% 93% 7% 87% 13% 93%
Total 214 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
Average
per Goal 35.7 N/A N/A 15% 85% 83% 17% 92% 8% 73% 27% 93%
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 16
Table 10. Summary of 2008 Transformation Activities by New Freedom Commission Goal
NFC Goals
Number
of
Activities
Percent
of
States
Percent
Range
of
Activity
Items
Percent
Begun
Before
MHT
SIG
[Yes]
Percent
Begun
Before
MHT
SIG [No]
Percent
Raised
in NARI
[Yes]
Percent
Raised
in NARI
[No]
Percent
Noted in
CMHP
[Yes]
Percent
Noted in
CMHP
[No]
Percent
Recomm
by TWG
[Yes]
Percent
Recomm
by TWG
[No]
Percent of
Activities
With
Activity
Results
NFC 2:
Care Is
Consumer/
Family
Driven
194 100% 1-25% 19% 81% 74% 26% 87% 13% 57% 43% 93%
NFC 5:
Excellent
Care Is
Provided
177 89% 0-22% 25% 75% 78% 22% 93% 7% 77% 23% 90%
NFC 1:
Mental
Health Is
Essential
133 89% 0-31% 27% 73% 75% 25% 92% 8% 78% 22% 88%
NFC 4:
Early
Screening Is
Provided
118 89% 0-32% 30% 70% 86% 14% 92% 8% 86% 14% 86%
NFC 3:
Disparities
Are
Eliminated
76 89% 0-29% 29% 71% 74% 26% 100% 0% 70% 30% 79%
NFC 6:
Technology
Is Used
66 89% 0-24% 12% 88% 86% 14% 98% 2% 89% 11% 86%
Total 764 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
Average
per Goal 127.3 N/A N/A 24% 76% 79% 21% 94% 6% 76% 24% 87%
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 17
Table 11. Summary of 2009 Transformation Activities by New Freedom Commission Goal
NFC Goals
Number
of
Activities
Percent
of
States
Percent
Range
of
Activity
Items
Percent
Begun
Before
MHT
SIG
[Yes]
Percent
Begun
Before
MHT
SIG [No]
Percent
Raised
in NARI
[Yes]
Percent
Raised
in NARI
[No]
Percent
Noted in
CMHP
[Yes]
Percent
Noted in
CMHP
[No]
Percent
Recomm
by TWG
[Yes]
Percent
Recomm
by TWG
[No]
Percent of
Activities
With
Activity
Results
NFC 1:
Mental
Health Is
Essential
199 100% 2-34% 12% 88% 75% 25% 91% 9% 85% 15% 82%
NFC 2:
Care Is
Consumer/
Family
Driven
185 89% 0-23% 23% 77% 73% 27% 91% 9% 71% 29% 65%
NFC 5:
Excellent
Care Is
Provided
184 100% 2-21% 23% 77% 81% 19% 89% 11% 81% 19% 72%
NFC 4:
Early
Screening Is
Provided
132 89% 0-36% 20% 80% 77% 23% 89% 11% 75% 25% 64%
NFC 3:
Disparities
Are
Eliminated
62 100% 3-21% 19% 81% 79% 21% 95% 5% 87% 13% 65%
NFC 6:
Technology
Is Used
49 89% 0-27% 12% 88% 86% 14% 92% 8% 92% 8% 76%
Total 811 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
Average
per Goal 135.2 N/A N/A 18% 82% 78% 22% 91% 8% 82% 18% 71%
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 18
Proof of Concept Studies
The status of the recovery and resiliency studies (as of September 14, 2009) is shown in Tables 12 and
13. For both types of studies, each State is required to enroll 75 persons in the impacted group and 75 in
the nonimpacted group. Overall, across impacted and nonimpacted groups, the first cohort of States had
completed baseline data collection with 68 percent of the enrollment target for their recovery studies and
51 percent for their resiliency studies by the end of the third year.
Table 12. Status of Recovery Studies
State Start Date
Consents
Past
Week
Declines
Past
Week
Interviews
Past
Week
Consents
to Date
Declines
to Date
Interviews
to Date
Connecticut
Impacted 7/09 Not
available
Not
available
Not
available
Not
available
Not
available 75
Connecticut
Nonimpacted 7/09 Not
available
Not
available
Not
available
Not
available
Not
available 75
Maryland
Impacted 5/09 N/A 0 2 N/A 66 51
Maryland
Nonimpacted 5/09 N/A 0 0 N/A 15 27
New Mexico
Impacted 5/09 N/A N/A N/A 10 N/A 10
New Mexico
Nonimpacted 6/09 N/A N/A N/A 17 N/A 17
Ohio
Impacted Complete N/A N/A N/A N/A N/A 751
Ohio
Nonimpacted Complete N/A N/A N/A N/A N/A 752
Oklahoma
Impacted 2/09 9 0 9 70 0 70
Oklahoma
Nonimpacted 2/09 11 0 11 69 0 69
Texas
Impacted 4/09 4 0 2 74 2 35
Texas
Nonimpacted 4/09 N/A N/A 2 Not
available
Not
available 25
Washington
Impacted 8/09 8 4 8 14 9 14
Washington
Nonimpacted 5/08 0 0 0 98 Not
available 983
Total
(Percent of
Target
Number)
N/A N/A N/A N/A N/A N/A 716 (68%)
1Final number may be more than 75.
2Final number may be more than 75.
3Sequential design: This is followup data collection. Baseline data collection was completed 7/08.
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 19
Table 13. Status of Resiliency Studies4
Site Start Date
Consents
Past
Week
Declines
Past
Week
Interviews
Past Week
Consents
to Date
Declines
to Date
Interviews
to Date
Connecticut
Impacted 6/09 0 0 0 0 0 0
Connecticut
Nonimpacted 6/09 0 0 0 0 0 0
Maryland
Impacted 1/09 0 0 0 79 22 50
Maryland
Nonimpacted 1/09 0 0 1 35 17 28
New Mexico
Impacted 6/08 N/A N/A N/A 48 N/A 48
New Mexico
Nonimpacted 6/08 N/A N/A N/A N/A N/A 0
Ohio
Impacted Complete N/A N/A N/A N/A N/A 751
Ohio
Nonimpacted Complete N/A N/A N/A N/A N/A 751
Oklahoma
Impacted Complete 9 N/A 9 78 N/A 781
Oklahoma
Nonimpacted Complete 8 N/A 8 78 N/A 781
Texas
Impacted 4/09 Not
available
Not
available
Not
available 83 10 71
Texas
Nonimpacted 4/09 Not
available
Not
available
Not
available 24 5 16
Washington
Impacted 3/09 0 0 0 39 13 37
Washington
Nonimpacted 3/09 0 0 0 23 7 23
Total
(Percent of
Target
Number)
N/A N/A N/A N/A N/A N/A 579 (51%)
Beginning in April 2009, State evaluators were asked to provide updates on the status of their POC
studies on each monthly Evaluation Work Group call. By June 2009, it became apparent that at least
several sites would likely not reach the recruitment target of 150 participants for their POC studies by the
mid-August deadline (the last possible date that would allow a 1-year followup by the project’s end).
Accordingly, the evaluation team responded in two ways:
1. By providing additional monitoring and support for the sites.
 In late July and early August, a series of calls with sites at risk of not reaching the full
complement of 150 participants (i.e., Connecticut, Maryland, Oklahoma, New Mexico,
Texas, Washington) were held to determine the current status of enrollment, the nature of
the difficulties encountered, and possible strategies for overcoming them. In general,
delays had resulted from two factors: (1) The interventions were not implemented as
planned by the State mental health agency, and (2) there had been a lack of
responsiveness on the part of State treatment programs, especially those representing
nonimpacted sites, to refer clients to the POC studies.
4 Final number may be more than 75.
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 20
 At-risk States were called to ask whether they intended to request a no-cost extension to
allow sufficient time to enroll the full number of participants and preserve the 1-year
followup period. None of the States indicated that they planned to make such a request.
 The project team provided more focused monitoring and support for the sites that
appeared to be having the most difficulty (e.g., Maryland, New Mexico, Texas) to identify
the nature of the obstacles they were encountering and explore means of overcoming
them.
2. By exploring analytic strategies to address the possible shortfall in the number of study
participants and the potential to have a shorter followup period.
 As it became apparent that some POC studies were unlikely to reach the target of 150
persons by the deadline of mid-August, the cross-site evaluation team began to explore
analytic strategies to address the possibility of having fewer than 150 participants for
each study. The project’s statistical consultant conducted further power analyses and
found that 100 participants per site would provide adequate power. It appears that most,
if not all, of the sites will achieve this level of enrollment. However, this approach will
require further consideration if the number of participants in each study is not fairly
equally balanced between impacted and nonimpacted groups, which appears likely to be
the case in some States.
 The second approach, having a followup period of less than 1 year (e.g., 9 months) was
examined to determine whether the various interventions being studied and measures
being used would permit a dose-effect analysis whereby it would be possible to identify a
trend line representing outcomes as a function of time of exposure to the intervention. A
preliminary review of the interventions and the measures being used in the POC studies
indicated that this was in fact the case—the effects of the interventions may be expected
to be incremental and the outcomes are measured as continuous variables.
 The Project Officer set a final deadline for baseline data collection of November 1, 2009,
based on guidance from the evaluators that this date would provide the optimum balance
between number of participants and length of followup period. Subsequently, several
States expressed various concerns about this deadline, such as additional burden related
to data collection and Institutional Review Board (IRB) modifications as well as
methodological issues. These concerns were addressed through email and telephone
discussion with the evaluation team.
 Both of these approaches will be explored more thoroughly in the coming year as
baseline data become available.
Table 14 provides an overview of the status of POC studies by State (as of September 14, 2009),
including each State’s plan to request a no-cost extension. Appendix C provides more detailed
information about each of the State POC studies, including the impacted and nonimpacted groups, the
GPRA indicators and NFC goals being addressed, and the survey instruments being used.
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 21
Table 14. Status of State Proof of Concept Studies
Site (Intervention)
Status of Data
Collection
Date Expect
To Complete
Baseline Data
Collection
Problems
Encountered/
Actions To
Expedite
Potential
Problems
Method Issues/Help
Requested
Plans to
Request
No-Cost
Extension
Connecticut
Recovery (Staff
trained in individual
recovery planning)
IRB approval
obtained, data
collection
completed
Impacted and
nonimpacted
interviews
completed
IRB delays—
now complete
None anticipated
(based on previous
projects with same
clinicians and
consumers)
None at this time Not
expected
Connecticut
Resilience
(Wraparound in
juvenile justice)
Final IRB obtained
in September
2009, requiring
only sign-off; data
collection to begin
immediately
End of October
2009
IRB questions—
now resolved
None anticipated, if
recruitment
proceeds as rapidly
as was the case
with adult study
1. “More vs. less
impacted” (i.e.,
subject exposure
varies at baseline—
admin data, and
site fidelity measure
may allow
adjustment
2. Possibly shortened
timeframe (11
months)
3. Type of sites vary
(school, probation,
welfare)
No, if
shorter
followup
period is
acceptable
Hawaii Recovery
(Network of
Care/WRAP Self-management)
Proposal
submitted to
cross-site
evaluation team,
working with the
sites on logistics
Between
December
2009 and
February 2010
Delays in State
Mental Health
Agency (SMHA)
implementation
of programs—
meetings with
SMHA and
Project Directors
None additional at
this time
Questions about follow
up period—advised to
conform to protocol of
1-year follow up
Don’t know
Hawaii Resilience
(Under
development)
Partners and sites
selected, study
details being
worked out,
expect to submit
proposal in
October 2009
Unknown at
this time
Difficulty
obtaining
cooperation of
SMHA—delayed
implementation
None additional at
this time
None at this time Don’t know
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 22
Site (Intervention)
Status of Data
Collection
Date Expect
To Complete
Baseline Data
Collection
Problems
Encountered/
Actions To
Expedite
Potential
Problems
Method Issues/Help
Requested
Plans to
Request
No-Cost
Extension
Maryland Recovery
(Cultural
competence
training)
Underway; have
IRB approval to
extend to end of
October 2009
End of October
2009
High refusal and
bad contact
information—
adjustments
made
None additional at
this time
Initial concern about
obtaining diagnosis,
but will be available
from administrative
data
Hope not
Maryland Resilience
(Wraparound)
Consent to be
contacted forms
completed,
continuing to
follow up
November 1,
2009; however,
probably will
have fewer
than 150
participants
Fewer than
expected
receiving
services—staff
have been very
active
contacting sites
Need to complete
before other grants
start up
Possibility of having
less than 150
participants
More time
will not help,
as subject
pool is
limited and
need to
complete
before new
grants begin
Missouri Recovery
(Peer specialists)
POC proposal
submitted to
cross-site
evaluators, sites to
be determined,
expect to start
data collection in
January 2010
Uncertain N/A None at this time None at this time Don’t know
Missouri Resilience
(Family support
specialist proposal is
under development)
Still in planning
stage, submission
to cross-site
evaluators has not
been made;
expect to start
data collection in
November/
December 2009
Uncertain N/A None at this time None at this time Don’t know
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 23
Site (Intervention)
Status of Data
Collection
Date Expect
To Complete
Baseline Data
Collection
Problems
Encountered/
Actions To
Expedite
Potential
Problems
Method Issues/Help
Requested
Plans to
Request
No-Cost
Extension
New Mexico
Recovery
(Integrated Dual
Diagnosis
Treatment)
Underway but
proceeding very
slowly
November 1,
2009; however,
probably will
have fewer
than 150
participants
In July, State
Managed Care
Organization
(MCO) change;
MCO was not
contacting sites,
turmoil at impact
site—Met with
Deputy
Secretary, site
incentives
increased, more
education,
recruitment now
bypassing MCO
and going
directly to sites
No additional
anticipated, expect
to proceed more
smoothly
None at this time Don’t know
New Mexico
Resilience (Clinical
home)
Underway for a
year but
proceeding very
slowly, already
doing some
followup
interviews though
baseline
incomplete
November 1,
2009; however,
probably will
have fewer
than 150
participants
Same as above
plus turmoil at
impact site—
Deputy
Secretary
authorized five
new sites
Same as above Though still collecting
baseline data, started
in June 2008 and
some are due for
followup—will be
maintaining 1-year
interval for now
Don’t know
Texas Recovery
(Self-directed care—
control funds)
Proceeding slowly November 1,
2009; however,
probably will
have fewer
than 150
participants
Difficulty with
provider
referrals,
especially
nonimpacted—
doing more
outreach and
modified
enrollment
criteria
None additional at
this time
None at this time Don’t know
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 24
Site (Intervention)
Status of Data
Collection
Date Expect
To Complete
Baseline Data
Collection
Problems
Encountered/
Actions To
Expedite
Potential
Problems
Method Issues/Help
Requested
Plans to
Request
No-Cost
Extension
Texas Resilience
(Trauma cognitive
behavioral therapy)
Proceeding slowly November 1,
2009; however,
probably will
have fewer
than 150
participants
Obtaining
referrals from
nonimpacted
sites
Imbalance in
number of
participants with
much smaller
number of
nonimpacted
None at this time Don’t know
Washington
Recovery (Wellness)
Sequential design:
nonimpacted
started June 2008,
completed
September 2009
Impacted
baseline will
start in
September
2009
None at this
time
None at this time None at this time Hope not
Washington
Resilience (School-based
services)
Only partial
numbers obtained
by end of school
year, will continue
beginning 2009
school year
End of October
2009
IRB modification
was approved to
allow default
parent contact
(opt-out letter)
None at this time 1. 6-month interval
between two
baseline groups
2. Possible shortened
timeframe
Hope not
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 25
State Site Visits
During the site visits, State agency directors were asked to rate the grant’s impact on their agency using a
10-point scale from 1 (lowest) to 10 (highest). Charts 1 and 2 illustrate ratings by State and type of State
agency, respectively. (The States were randomly coded to maintain their anonymity.) Three States
(States 4, 5, and 9) had the highest ratings of 7 or more, while two States (States 3 and 7) had the lowest
ratings of 5 or less. Agency directors were asked to explain the rationale for their rating; their responses
indicate that they used different approaches in assigning their rating. For example, some State agency
staff based their rating on progress completed to date, while others rated based on the potential impact of
the grant. Regarding impact by agency type, mental health and juvenile justice agencies reported the
most impact (with ratings 7 or above), while other agencies reported slightly lower levels of impact
(ratings of around 6). About 10 percent of the agency directors did not provide any rating, often because
they viewed an assessment of the grant’s impact as premature or had limited involvement with the grant.
Figure 1. Average Rating of Grant Impact by State5
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 8 9
Individual States
Average Impact Rating
5 States have been randomly coded.
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 26
Figure 2. Average Rating of Grant Impact by Type of State Agency
1
2
3
4
5
6
7
8
9
10
Mental
Health
Criminal
Justice
Juvenile
Justice
Child
Welfare
Housing Vocational
Rehab
Medicaid
Type of State Agency
Average Impact Rating
Federal Agency Site Visit
As part of the Federal agency site visit, Federal staff (program managers and Connectors) were asked to
rank NFC recommendations by order of importance. The results of this ranking exercise are shown in
Table 15. Review of these rankings reveals that managers and Connectors share four of their six highest
ranked recommendations and five of their seven lowest ranked recommendations, showing a relatively
high level of agreement between the groups. The greatest levels of agreement between manager and
Connector rankings are described below:
 Recommendation 2.2 (involve CFMs in orienting mental health systems toward recovery) is the
highest ranked recommendation of both groups.
 Recommendations 1.1 (advance antistigma and antisuicide campaigns), 1.2 (address mental
health with same urgency as physical health), and 4.1 (promote the mental health of young
children) are in the six highest ranked recommendations of both groups.
 The following recommendations are in the seven lowest ranked recommendations of both groups:
 3.2 (improve access in rural and geographically remote areas).
 5.1 (accelerate research).
 5.4 (develop the knowledge base in four understudied areas).
 6.1 (use technology to improve access and coordination of care).
 6.2 (develop electronic health records).
 The following recommendations are in the six mid-ranked recommendations of both groups
(neither highest nor lowest ranked):
 4.3 (screen for co-occurring disorders and link with integrated treatment).
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 27
 5.2 (advance evidence-based practices and use public/private partnerships to advance).
The greatest discrepancies between manager and Connector rankings are found for:
 Recommendation 2.4 (develop CMHP)—ranked low by managers (16) and high by Connectors
(2).
 Recommendation 4.2 (improve and expand school mental health programs)—ranked low by
managers (17) and in the middle by Connectors (7).
 Recommendation 4.4 (screen for mental health disorders in primary health care, across the
lifespan, and connect to treatment)—ranked in the middle by managers (7) and low by
Connectors (16).
 Recommendation 2.3 (align Federal programs to improve access and accountability)—ranked
high by managers (4) and in the middle by Connectors (12).
Table 15. Ranking of NFC Recommendations by MHT SIG Program Managers and Connectors6
6 Code for shading/font: Turquoise square/bold font indicates recommendation is ranked in top 6; yellow shading/italic
font indicates recommendation is ranked in bottom 7.
NFC Recommendations
Average
Rank for
Managers
Overall Rank
for Managers
Average
Rank for
Connectors
Overall Rank
for
Connectors
1.1 Advance and implement a national
campaign to reduce the stigma of
seeking care and a national strategy
for suicide prevention
7.7 5 8.3 5
1.2 Address mental health with the
same urgency as physical health 4.3 2 6.2 4
2.1 Develop an individualized plan of
care for every adult with a serious
mental illness and child with a
serious emotional disturbance
11.5 12 11.8 14
2.2 Involve consumers and families fully
in orienting the mental health system
toward recovery
1.3 1 4.2 1
2.3 Align relevant Federal programs to
improve access and accountability
for mental health services
7.3 4 10.7 12
2.4 Create a Comprehensive State
Mental Health Plan 13.3 16 4.5 2
2.5 Protect and enhance the rights of
people with mental illnesses 8.7 8 8.3 5
3.1 Improve access to quality care that
is culturally competent 6.3 3 9.5 8
3.2 Improve access to quality care in
rural and geographically remote
areas
12.3 14 11.5 13
4.1 Promote the mental health of young
children 7.7 5 4.7 3
4.2 Improve and expand school mental
health programs 15.3 17 8.8 7
4.3 Screen for co-occurring mental and
substance use disorders and link
with integrated treatment strategies
11.0 11 10.0 9
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 28
Consumer, Youth, and Family Member Focus Groups
Focus groups were conducted with convenience samples of CFMs; these samples should not be
considered representative of all CFM perspectives across all grantee States. Of the 130 persons who
participated in the focus groups:
101 participants were consumers (17 were youths, 77 were adults, and 7 were older adults).
39 participants were family members of youth consumers, and 62 were family members of adult
consumers.
A total of 40 persons identified as being both a consumer and family member.
Involvement was greater at the State level (101 participants, or 78 percent) than at the local level (75
participants, or 58 percent).
107 participants (82 percent) reported being members of CFM stakeholder organizations.
Tables 16 and 17 provide information about the frequency and type of CFM involvement in State and
local activities and the involvement of CFMs in specific types of grant activities. In both these tables, the
individual States have been randomly coded. Table 16 shows CFM ratings for the frequency and type of
CFM involvement at the State and local level for individual States and for all States combined. Overall,
the frequency of consumer and family member involvement was viewed as “occasionally” and their type
of involvement as “minor” at both the State and local levels.
4.4 Screen for mental disorders in
primary health care, across the
lifespan, and connect to treatment
and supports
8.0 7 12.8 16
5.1 Accelerate research to promote
recovery and resilience, and
ultimately to cure and prevent
mental illnesses
15.3 17 15.7 19
5.2 Advance evidence-based practices
using dissemination and
demonstration projects and create a
public-private partnership to guide
their implementation
10.0 10 10.3 10
5.3 Improve and expand the workforce
providing evidence-based mental
health services and supports
9.5 9 10.3 10
5.4 Develop the knowledge base in four
understudied areas: mental health
disparities, long-term effects of
medications, trauma, and acute care
12.7 15 14.5 17
6.1 Use health technology and
telehealth to improve access and
coordination of mental health care,
especially for Americans in remote
areas or in underserved populations
11.7 13 12.3 15
6.2 Develop and implement integrated
electronic health record and
personal health information systems
15.7 19 15.5 18
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 29
Table 16. Average CYF Rating of Frequency and Type of Consumer, Youth, and Family Member Involvement in Grant Activities7
Consumer/Youth/Family
Involvement
State
1
State
2
State
3
State
4
State
5
State
6
State
7
State
8
State
9
Frequency of Involvement8 in
Local Activities 2.3 1.7 1.5 1.7 1.4 1.9 1.7 1.4 1.4 1.8
Occasionally
Type of Involvement9 in Local
Activities 1.2 1.1 0.6 1.1 0.7 1.3 1.3 0.7 0.9 0.9
Minor Role
Frequency of Involvement10 in
State Activities 1.9 2.1 0.7 2.1 1.0 1.4 1.9 1.1 1.2 1.6
Occasionally
Type of Involvement11 in State
Activities 1.1 1.1 0.5 1.1 0.5 1.1 1.2 0.5 1.0 0.8
Minor Role
7States have been randomly coded.
8Never = 0, Rarely = 1, Occasionally = 2, Frequently = 3, Almost always = 4.
9No role = 0, Minor role = 1, Major role = 2.
10Never = 0, Rarely = 1, Occasionally = 2, Frequently = 3, Almost always = 4.
11No role = 0, Minor role = 1, Major role = 2.
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 30
Focus group participants were also asked to rate their level of agreement with 35 statements about their
involvement in grant activities. Table 17 shows the average ratings by focus group participants for these
statements by individual States and for all States combined. For the purpose of describing ratings for all
States combined, the ratings have been grouped into the following categories: 1 to 2.7 (disagree), 2.8 to
3.2 (generally neither agree nor disagree), and 3.3 to 5 (agree).
Across all States, consumers and family members generally agreed with the following statements:
 During project meetings, the opinions of CFMs are discussed.
 Involvement of CFMs has made a difference.
 The grant promotes collaboration among CFMs.
 Disagreements are handled respectfully within this project.
 Efforts are being made to evaluate CFM involvement.
 Stigma/discrimination is not accepted at any level of this grant.
 Grant staff are able to work collaboratively with CFMs.
 Mental health service users and their family members are positively affected by this grant.
 Persons of all cultural and ethnic origins are respected within this grant.
Across all States, consumers and family members generally disagreed with the following statements:
 There are enough CFMs involved in the mental health grant.
 CFMs have meaningful involvement in decisions on funding of State programs.
 CFMs have meaningful involvement in decisions on funding of local programs.
 CFMs have meaningful involvement in making State policy.
 CFMs receive the training and support they need to participate effectively in the mental health
transformation grant.
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 31
Table 17. Average CFY Ratings regarding Various Aspects of CFY Participation in MHT SIG Grant Activities1213
Statements
State
1
State
2
State
3
State
4
State
5
State
6
State
7
State
8
State
9
Overall Average
Rating14
1. CFMs have meaningful
involvement in:
Setting local goals 4.1 2.3 2.1 3.7 2.8 3.6 3.9 2.1 2.4 3.0
Neither
Making local policy 3.7 2.3 1.8 3.5 2.5 3.3 3.8 2.0 2.1 2.8
Neither
Designing local programs 3.8 2.8 2.0 3.3 2.5 3.3 3.0 2.2 2.6 2.9
Neither
Implementing local programs 3.8 2.8 2.2 3.3 2.3 3.6 2.9 2.5 2.3 2.9
Neither
Evaluating local programs 3.8 2.5 2.0 3.4 2.2 3.6 2.9 2.5 2.4 2.8
Neither
Decisions on funding of local
programs 3.3 2.3 1.6 3.4 2.4 2.9 2.8 1.9 2.4 2.5
Disagree
Setting State goals 3.6 3.2 2.2 3.2 2.3 3.2 3.6 2.2 2.9 2.9
Neither
Making State policy 3.2 2.6 1.8 2.8 1.9 2.8 3.4 2.0 2.8 2.6
Disagree
Designing State programs 3.5 3.0 2.0 3.1 2.1 3.0 3.2 1.9 2.9 2.8
Neither
Implementing State programs 3.5 3.4 1.8 3.7 2.1 2.8 3.7 1.7 2.9 2.8
Neither
Evaluating State programs 3.8 2.8 2.0 3.3 2.1 2.9 3.8 1.8 3.1 2.8
Neither
Decisions on funding of State
programs 3.1 2.4 1.5 2.8 1.7 2.1 3.4 1.8 2.5 2.4
Disagree
2. There are enough CFMs
involved in the mental health
grant
3.2 2.8 1.4 3.3 1.1 1.8 3.2 1.2 2.0 2.2
Disagree
12 States have been randomly coded.
13 Strongly disagree = 1, Disagree = 2, Neither = 3, Agree = 4, Strongly Agree = 5.
14The ratings were grouped into the following categories: 1 to 2.7 (disagree), 2.8 to 3.2 (generally neither agree nor disagree), and 3.3 to 5 (agree).
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 32
Statements
State
1
State
2
State
3
State
4
State
5
State
6
State
7
State
8
State
9
Overall Average
Rating14
3. Mental health service users and
their family members are
positively affected by this grant
4.2 3.9 2.9 2.7 2.8 3.8 3.8 2.0 3.2 3.3
Agree
4. CFMs are adequately
compensated for their roles 3.9 3.4 1.8 3.4 1.7 3.8 3.3 1.2 3.1 2.9
Neither
5. Involvement of CFMs has made
a difference 4.2 4.2 3.9 2.7 2.6 4.1 3.8 2.3 3.3 3.5
Agree
6. Grant staff are able to work
collaboratively with CFMs 4.5 4.2 3.3 2.9 2.0 4.0 4.0 1.7 3.5 3.4
Agree
7. The grant promotes
collaboration among CFMs 4.4 4.1 3.0 2.9 2.6 3.6 3.9 2.4 3.5 3.5
Agree
8. State leaders are sensitive to
cultural and linguistic issues 4.1 3.6 2.6 2.8 1.6 3.5 3.4 2.5 3.0 3.1
Neither
9. CFMs receive the training and
support they need to participate
effectively in the mental health
transformation grant
3.6 3.3 2.0 2.2 1.4 3.1 3.3 1.9 2.8 2.7
Disagree
10. The grant promotes CFM
understanding of the process
behind developing policy
4.0 3.1 2.0 2.8 1.6 3.4 3.6 1.6 3.0 2.8
Neither
11. The grant promotes CFM
understanding of current policy
issues
3.8 3.1 2.3 2.6 1.8 3.6 3.7 2.0 3.4 3.0
Neither
12. The leaders of this grant make
involvement by CFMs a priority 4.1 3.9 1.8 2.6 2.3 4.1 3.9 2.1 3.6 3.3
Neither
13. During project meetings, the
opinions of CFMs are discussed 4.4 4.2 3.3 3.1 3.0 4.0 4.0 2.0 3.5 3.6
Agree
14. Action is taken as a result of
CFM opinions 4.2 4.0 2.6 2.4 1.9 3.9 3.7 1.9 3.0 3.1
Neither
15. A partnership exists between
CFMs and persons who are not
CFMs
4.2 3.4 2.9 2.7 2.2 3.8 3.4 1.7 3.5 3.1
Neither
16. As a result of the grant, CFMs
have the knowledge to educate
the community on important
issues.
3.7 3.4 2.2 2.6 2.3 3.5 4.0 1.8 3.0 3.0
Neither
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 33
Statements
State
1
State
2
State
3
State
4
State
5
State
6
State
7
State
8
State
9
Overall Average
Rating14
17. The grant promotes CFMs to
take the lead in this
transformation project.
3.9 3.4 2.0 2.7 1.8 3.7 3.9 1.9 2.9 2.9
Neither
18. Disagreements are handled
respectfully within this project. 4.1 4.2 2.6 3.2 2.5 3.8 4.3 2.4 3.0 3.5
Agree
19. Information about the mental
health transformation grant is
readily available to CFMs.
3.9 3.4 2.4 3.9 2.4 3.5 3.9 2.0 3.2 3.2
Neither
20. Professionals use language that
is easily understood by
nonprofessional participants.
3.7 3.6 2.4 3.6 2.7 3.5 4.0 2.9 2.4 3.2
Neither
21. Efforts are made to evaluate
CFM involvement. 4.4 4.1 3.0 3.0 2.6 4.0 4.1 2.3 3.5 3.5
Agree
22. CFMs are excited about the
progress of the grant. 4.0 3.7 2.2 2.9 2.0 3.8 3.6 1.8 3.2 3.2
Neither
23. Persons of all cultural and ethnic
origins are respected within this
grant.
4.2 3.6 2.9 3.0 2.4 3.5 3.6 2.6 3.3 3.3
Agree
24. Stigma/discrimination is not
accepted at any level of this
grant.
4.3 4.3 2.4 2.7 2.5 4.0 4.4 2.7 3.6 3.5
Agree
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 34
Across all States, CFMs neither agreed nor disagreed with 21 (60 percent) of the statements (see Chart
3). CFMs agreed with nine of the statements (26 percent) and disagreed with five others (14 percent).
The lack of agreement or disagreement with most statements raises a question of the true level and
nature of CFM involvement in grant activities.
Figure 3. Average Ratings on Statements Provided by Consumer and Family Member Focus
Group Participants Across All States
National Outcome Measures Data
State NOMs data for 2004 to 2008 were analyzed to determine the completeness of data by individual
measures and summarize the findings. Appendix D provides the actual NOMs data by State. Data for
three measures are in development and are not currently available: NOM 1 (decreased mental illness
symptomatology), NOM 3 (decreased criminal justice system involvement), and NOM 9 (increased cost-effectiveness).
It should be noted that States sometimes collect NOMs data using different methods and
samples. A full understanding of how these data can be used in the Other State Data Analyses will
depend on a careful analysis of how these data are collected by MHT SIG states.
The missing State data by NOMs measure are:
 NOM 2: Percentage of adults who are employed: Maryland in 2004 and Missouri in 2006.
 NOM 4: Percentage of individuals living in a private residence: Maryland in 2004, Missouri in
2005 and 2006, and Ohio in 2004.
 NOM 5: Percentage of persons served by age (under 18 and age 65+) and race/ethnicity: New
Mexico in 2004 (age data only).
 NOM 6: Mental health hospital utilization per 1,000: Ohio in 2004 to 2008.
 NOM 7: Percentage of adults and children/families indicating increased social
support/connectedness: In 2004 and 2005, this was a developmental measure. For adults, 2006
0%
10%
20%
30%
40%
50%
60%
70%
Neither Agree
nor Disagree
Disagree Agree
Response
Percent of Respondents
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 35
data are missing for Hawaii, Missouri, New Mexico, and Washington. For children/families, 2006
data are missing for Hawaii, Missouri, Ohio, and Washington. Other missing data on
children/families include 2007 and 2008 data for Ohio and 2007 data for Oklahoma.
 NOM 8: Percentage of adults with positive reporting for outcomes: Missouri in 2004 and 2006.
 NOM 10: Four different measures are used (percentages of States implementing adult supported
housing, adult supported employment, adult assertive community treatment, and therapeutic
foster care for children and adolescents), and there are missing data for many of these measures.
The adult supported employment data are most complete, followed by adult supported housing.
State trends for NOMs 2, 4, 6, 7, and 8 are shown in Table 18. Six of nine grantee States reported
positive changes over the time period for all NOMs measures except NOM 8 (positive reporting on
outcomes by adults).
Table 18. Number of MHT SIG Grantee States Reporting Increases, Decreases, and No Changes in
Their NOMs Data from 2004 to 2008
National Outcome Measure Increase Decrease No Change
Not
Applicable
NOM 2: Employment 6 1 2 0
NOM 4: Living in Private
Residence 6 3 0 0
NOM 6: Inpatient Utilization
Rate151 6 2 0 1
NOM 7: Social Support/Social
Connectedness—Adults 6 1 2 0
NOM 7: Social Support/Social
Connectedness—Child/Family 6 1 1 1
NOM 8: Positive Reporting
About Outcomes—Adults 3 4 2 0
The analysis of NOM 5 data was done for age groups (under 18 and 65+) and race/ethnicity using Tables
43 and 44 from Appendix D. The results showed that:
 Five States had higher percentages of youth in treatment in 2008 than were represented in the
State’s general population, while four States had lower percentages. It is possible that higher
percentages are associated with greater access to mental health services for youth. During the
period from 2004 to 2008, four States had increases in their percentages of youth in treatment,
while the remaining five States had decreases.
 All nine States had lower percentages of persons age 65+ in treatment in 2008 than were
represented in the State’s general population. For six States, 3 percent or fewer of persons in
treatment were age 65+, while individuals aged 65+ represented 10 to15 percent of grantee State
populations. For the period 2004 to 2008, all States but Ohio had decreases in their percentages
of persons aged 65+ in their client population; Ohio’s percentage remained unchanged.
 All but one grantee State had a lower percentage of Whites in treatment in 2008 than was
represented in the State’s general population.
As noted in the technical evaluation plan, issues do exist regarding statewide comparison of NOMs data
(e.g., comparable data periods and measurement across States, lack of year-to-year consistency within
individual States). Until these issues are examined in more detail at the level of the individual NOMs for
the full data collection period, the usefulness of the NOMs data in the cross-site evaluation will be
unclear.
15A reduced inpatient utilization rate is a change in a positive direction.
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 36
Analysis of State Employment Initiatives
In January 2009, an analysis of State employment initiatives was completed. The findings are
summarized as follows:
 All nine MHT SIG States have employment activities focused on consumer representation in the
mental health care workforce. Employment opportunities for consumers include:
 Peer specialists
 Positions in consumer-operated services
 Research, evaluation, or quality improvement team members
 Consumer positions in non-mental health service agencies
 Consumer positions on Transformation Working Groups
 Consumer positions as liaisons with other consumers (e.g., in offices of consumer affairs,
on consumer engagement teams)
 General positions open to anyone (e.g., clinicians, State administrators)
 In addition, some States are addressing competitive labor force participation in the following
subject areas:
 Supported employment (Maryland and Ohio)
 Ticket to Work (Maryland and Ohio)
 Medicaid (Ohio, Texas, and Washington)
 Vocational rehabilitation system (Ohio, Texas, and Washington)
 Criminal justice system (Washington)
 Temporary Assistance for Needy Families (New Mexico)
 Working with employers (Texas)
 Individual Development Accounts (Texas)
 Employment assessment as part of treatment planning (New Mexico and Ohio)
 Consumer-run moving business (New Mexico)
 Transportation barriers (Texas)
 None of the States reported activities that involve working with U.S. Department of Labor
initiatives such as One-Stop Career Centers.
Leadership Survey
During the third project year, the leadership survey was administered for the first time to leaders and
raters in the first cohort of States. Leadership changes had already occurred in most of the States when
the survey was administered. For some States, these changes could be categorized as significant. For
instance, two States experienced gubernatorial changes involving political party transitions. For these two
States, leadership was impacted at three levels: State (gubernatorial), project (TWG Chairperson), and
project management (Project Director). These two States had Project Directors who were involved in
leadership of the TWG in one way or another; one Project Director operated in dual roles (as Project
Director and TWG Chairperson) and the other was involved in coordination related to the TWG.
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 37
Across both cohorts, four States have individuals serving in the dual roles of Project Director and TWG
Chairperson. Changes in Project Directors occurred in seven States (including the two States that
experienced gubernatorial changes).
Implementation of the survey was challenging in some respects due to these changes in leadership. A
common issue was getting previous leaders to complete the survey. Another challenge was that leaders
and raters had to rely on their long-term memory. Raters in particular were asked to complete the survey
based on their observations of the previous leader’s leadership style; for some, this aspect of their State’s
MHT SIG project was difficult to recall.
Preliminary findings from the leadership survey include the following:
 Leaders indicated they frequently seek differing perspectives when solving problems, consider
the moral and ethical consequences of their decisions, and express satisfaction when others
meet expectations. They reported that once in a while, they are absent when needed, delay
responding to urgent questions, and avoid making decisions.
 Raters indicated that leaders fairly often consider the moral and ethical consequences of their
decisions, express satisfaction when the rater meets expectations, and treat the rater as an
individual rather than a member of a group. Raters reported that once in a while, leaders avoid
making decisions, delay in responding to urgent questions, and are absent when needed.
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 38
Provider Survey
The pilot survey was completed in all States except Hawaii. A total of 37 providers completed the survey,
with 59 percent reporting they were aware of the grant. Table 19 shows the results of the pilot survey as
of September 14, 2009. Knowledge about the grant most often came from State mental health agencies,
provider meetings, and news releases/emails.
Table 19. Results of Pilot Test of Provider Survey
State
Number
Surveyed
Percent
Aware of
Grant How Providers Heard About Grant
Connecticut 2 100% State Department of Mental Health and Addiction
Services; participation on TWG subcommittee
Hawaii 0 N/A N/A
Maryland 6 67% Governor’s press announcement; NAMI meetings;
co-occurring disorder meetings
Missouri 4 50% Public mental health authority; CEO of their
organization who works on the grant project
New Mexico 7 43% Networking; CEO of their organization; local
collaborative; provider meetings
Ohio 4 50%
Federal Department of Health and Human Services
and SAMHSA; State Department of Mental Health
email
Oklahoma 6 50%
Oklahoma Department of Mental Health and
Substance Abuse Services (ODMHSAS) Director
meetings; ODMHSAS publications; email to
community mental health centers
Texas 5 80%
Provider meetings and mental health/mental
retardation collaborative; news clippings; Texas
Department of State Health Services grant
notification
Washington 3 67% Grant implementation committee; email notification
Total 37 59% N/A
State Specific Evaluation Activities
Based on a request from State evaluators, a listing of actual and planned State-specific evaluation
activities (i.e., activities that were not part of the cross-site evaluation requirements) was developed in
May 2009 through review of 2009 State continuation applications and reviews by State MHT SIG staff
and evaluators. Tables 20 through 28 list the 128 different activities identified by the grantee State
evaluators. The average number of activities in each State is 14, with range from 10 in Texas to 26 in
Connecticut.16 The most common types of evaluation activities in the States are as follows:
 Antistigma efforts to determine impact or effectiveness (5 States)
 Mental Health First Aid (4 States)
 State surveys of evidence-based practices (4 States)
16 Note that Hawaii and Missouri received less MHT SIG funding than the States that were awarded
grants in 2005.
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 39
SUMMARY
The third year of the MHT SIG cross-site evaluation project was productive. Site visits were completed in
Hawaii and Missouri, with site visit reports to be completed early in year 4. Close coordination with the
States has continued regarding their POC studies. The project team completed focus groups in all States
(with associated reports completed for the first cohort of States), the leadership survey for the first cohort
of States, and the pilot provider survey in all but one State. Other support provided to the MHT SIG
grantees included launching and maintaining Transformation Tracker Explorer and participating in the
annual grantee meeting. The cross-site evaluation team values the important work in the States to realize
transformation of their mental health systems and meet New Freedom Commission goals. The team is
enthusiastic about collaborating with the States and continuing the evaluation in the upcoming year.
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 40
Table 20. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Mental Health
Program/Services/Systems/Clients
Topic CT HI MD MO NM OH OK TX WA
Mental Health First Aid N N Y Y Y N Y N N
State survey of evidence-based practices N N N Y Y N Y N Y
Consumer surveys on recovery orientation N N N N Y N Y N Y
Wraparound Fidelity Assessment System (WFAS) Y N Y N N N N N N
Systems of care N N Y N N N Y N N
Expanded school mental health program N N Y N Y N N N N
Early Childhood Mental Health Consultation Evaluation N N Y N Y N N N N
Evidenced-based practices fidelity and outcomes evaluation for a number of specific
programs N N Y N Y N N N N
Collection of standardized information on all MHT SIG projects N N N N Y N N Y N
Evaluation of Program of Assertive Community Treatment (PACT) teams statewide N N N N N N Y N Y
Indepth interviews with consumers, youth, and family members regarding the current
mental health system and needed improvements N N N N N N Y N Y
Consumer-, youth-, and family-driven Quality Improvement Collaborative to identify
and implement recovery- and resilience-oriented performance measurements for the
evaluation of the mental health service delivery models and programs
Y
N N N N N N N N
Use of seclusion and restraint Y N N N N N N N N
Behavioral Workforce Collaborative Y N N N N N N N N
Interagency determination of questions about mental health curricula to add to CDC
biannual School Health Profile Y N N N N N N N N
Followup study of three Co-Occurring State Incentive Grant (CO-SIG) sites (in
process) N Y N N N N N N N
Community minigrants N Y N N N N N N N
Department of Human Resources/Transformation Level of Intensity Evaluation—
Development of tool to evaluate youth in group homes, treatment foster care, and
independent living
N
N
Y
N N N N N N
Project Connections, a community mental health outreach program for persons living
in communities with high rates of poverty, violence, and instability N N Y N N N N N N
RESPECT program N N N Y N N N N N
Regional Behavioral Health Initiative activities regarding cultural competence/
RESPECT program N N N Y N N N N N
Communities of Hope program (planned) N N N Y N N N N N
State survey of provider needs N N N Y N N N N N
Drop-in program analysis N N N Y N N N N N
Behavioral health recovery studies N N N N Y N N N N
Youth suicide prevention programming N N N N Y N N N N
Hospital to community trauma-informed care study (planned) N N N N N Y N N N
Statewide consumer focus group interviews for insight into perceptions of treatment
planning and instrumentation development N N N N N Y N N N
Access to mental health and substance abuse services N N N N N N Y N N
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 41
Topic CT HI MD MO NM OH OK TX WA
Development and implementation of centralized electronic health record and reporting
tool by local mental health authorities and substance abuse treatment providers N N N N N N N Y N
Determination of the effectiveness of mental health infrastructure innovations and
related factors using the Texas MHT Conceptual Model N N N N N N N Y N
Development and assessment of use of Web-based comprehensive listing of
evidence-based practices N N N N N N N Y N
Service integration via use of liaison position to navigate consumers through local
mental health authorities and primary care at Federally Qualified Health Centers N N N N N N N Y N
General Assistance—Unemployable Clients: Challenges and Opportunities N N N N N N N N Y
Washington’s Family Integrated Transition program N N N N N N N N Y
Evaluation of King County Rethinking Care Pilot Project (to improve access to mental
health services for Federally Qualified Health Center clients) N N N N N N N N Y
Evaluation of statewide wraparound pilot for youth with serious emotional and
behavioral disorders and their families N N N N n N N N Y
Table 21. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Stigma
Topic CT HI MD MO NM OH OK TX WA
Evaluation of impact or effectiveness of statewide social marketing campaign Y Y N Y N N Y N Y
Antistigma Y N Y N N N Y N N
Stigma surveys N N N Y Y N Y N N
Survey of provider organizations regarding stigma N N N N N N N N Y
Table 22. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Funding & Resources
Topic CT HI MD MO NM OH OK TX WA
Medicaid 1915c home and community-based waiver Y N Y N N N N N N
Expedited Medicaid review for prisoners study N N N N N N Y N Y
NARI: Fiscal Resource Inventory Addendum, August 2008 N Y N N N N N N N
Provider documentation project report N N N N N Y N N N
Implementation of E2SSB 5763 (county option to impose 1/10 of 1% sales tax for
mental health and other services) N N N N N N N N Y
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 42
Table 23. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Workforce and Training
Topic CT HI MD MO NM OH OK TX WA
Peer specialists (satisfaction survey) N N N Y Y N N N N
Connecticut Recovery Employment Consultation Service (focuses on recruiting,
preparing, and placing consumer providers within human service agencies) Y N N N N N N N N
Providers and supervisors in outpatient and inpatient settings in person-centered,
rehabilitative services, and trauma-informed care Y N N N N N N N N
Community stakeholders (parents, teachers, behavioral health staff, and judicial staff)
in wraparound practices and principles Y N N N N N N N N
University professors and graduate students in intensive in-home family treatment Y N N N N N N N N
Supervision competencies Y Nn N N N N N N N
Department of Labor health and human services staff in recovery and mental health Y N N N N N N N N
Adult consumers to become recovery supporters Y N N N N N N N N
Community minigrants N Y N N N N N N N
Family Engagement Training Program N N Y N N N N N N
Cultural competence trainings N N Y N N N N N N
Family support workers (satisfaction survey) N N N Y N N N N N
Peer specialists (whether interaction between giver and recipient is beneficial) N N N Y N N N N N
Workforce N N N N N N Y N N
Training—Texas Collaborative for Emotional Development in Schools/Education
Service Centers Train-the-Trainer N N N N N N N Y N
Survey of persons who completed consumer peer-to-peer counselor training N N N N N N N N Y
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 43
Table 24. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding CYF, CYF Organizations, &
Community Stakeholders
Topic CT HI MD MO NM OH OK TX WA
Community minigrants Y Y N N N N N N Y
Surveys on consumer and family member perspectives about transformation efforts N N N Y Y Y N N N
Network of Care Web site—Monitoring of perceptions/use of information (about
mental health and substance abuse treatment and services as well as resources for
persons across the lifespan) or effectiveness
Y N N Y N N N N N
Leadership training to parents Y N N Y N N N N N
Assessment of level of consumer, youth, and family involvement in grant activities and
of ways it can be enhanced throughout the service delivery system Y N N N N N N N N
Evaluation of Consumer/Youth/Family Research and Evaluation Network (e.g.,
transformation involvement survey, minigrant projects, focus group information from
mental health agency disparities initiative)
Y N N N N N N N N
Consumer-run organizations N N Y N N N N N N
Community Evaluation and Research Training Academy for consumers and
community stakeholders N N N N Y N N N N
Consumer participation in Local Collaborative process N N N N Y N N N N
State Annual Local Collaborative Self-Assessment Survey N N N N Y N N N N
State Pre K-6 Parent Drug Prevention Programs N N N N Y N N N N
Family advocacy focus group evaluation N N N N N Y N N N
Consumer/family member involvement standards N N N N N N Y N N
Consumer Network and Training—Training and Technical Assistance Center N N N N N N N Y N
Consumer, youth, and family conference (satisfaction survey) N Y N Y Y N N N N
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 44
Table 25. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Disparities, Prevalence, &
Statistics
Topic CT HI MD MO NM OH OK TX WA
Geographic disparities in mental health workforce N N N Y N N Y N Y
Disparities in mental health services between urban and rural communities N N N N N N Y N Y
Data interoperability to identify and eliminate behavioral health disparities Y N N N N N N N N
State alcohol consumption and consequences annual survey N N N N Y N N N N
State epidemiological drug use surveys N N N N Y N N N N
Analysis of statewide database on family health and mental health N N N N N Y N N N
Business case for trauma-informed intervention programming N N N N N Y N N N
Analysis of utilization data for transition-age youth N N N N N Y N N N
Analysis of the Youth Risk Behavior Survey N N N N N Y N N N
Table 26. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Criminal/Juvenile Justice
Topic CT HI MD MO NM OH OK TX WA
Prison reentry study N N N N N N Y N Y
Mental Health and Juvenile Justice Consultant Technical Report N Y N N N N N N N
Community minigrants N Y N N N N N N N
Juvenile justice supportive housing project pilot survey—adolescents N N N N Y N N N N
Criminal justice (to determine whether released persons are getting necessary and
quality services and have appropriate outcomes) N N N N N N Y N N
Data sharing to promote jail diversion N N N N N N N Y N
Service integration—court competency restoration program N N N N N N N Y N
Arrests Among Working-Age Disabled (General Assistance—Unemployable) Clients
evaluation N N N N N N N N Y
Implementation of SSB 5533 (jail diversion) N N N N N N N N Y
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 45
Table 27. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Transformation Working Groups
and Committees
Topic CT HI MD MO NM OH OK TX WA
Formative feedback to MHT SIG partners about progress to date and ways to better
meet CMHP objectives Y N N N Y Y N N N
Transformation Working Group—satisfaction with meetings, transformation process,
engagement, and suggestions for continued transformation N N Y Y N Y N N N
Monthly reporting template as part of formative evaluation of grant activities Y N N N Y N N N N
Summary Evaluation of Sub-Working Group Process N Y N Y N N N N N
Interviews and surveys with Content Working Groups, other agencies, and community
constituents to evaluate overall transformation effectiveness and process of
transformation efforts
N N N Y N Y N N N
Detailed evaluation reports on major initiatives of the Resource Investment and
Strategy Subcommittee Y N N N N N N N N
Summary Evaluation of Task Groups Process (draft developed) N Y N N N N N N N
Local Collaborative Process Assessment Survey of Local Collaborative leaders N N N N Y N N N N
MHT SIG Summative and Sustainability Report N N N N Y N N N N
Table 28. States Implementing or Planning to Implement State-Specific Evaluation Activities Regarding Other Topics
Topic CT HI MD MO NM OH OK TX WA
Psychiatric advance directives Y N N N N N N N N
Policy analysis N N N N N N Y N N
Technology Use Among Community Collaboratives (completed) N N N N N N N Y N
Tracking of numbers of consumers, youth, and family members hired as a result of
MHT SIG initiatives Y N N N N N N N N
Pilot Housing Resilience Survey N N N N Y N N N N
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 46
Appendix A: GPRA Targets
Table 29. Cumulative GPRA Target Numbers17
GPRA Measure FY CT HI MD MO NM OH OK TX WA Program
118 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline
1 2007 15 Baseline 0 Baseline 2 2 34 14 13 80
1 2008 33 0 5 20 21 63 46 64 27 279
1 2009 39 1 7 53 22 171 65 77 35 470
1 2010 41 1 8 71 27 223 78 81 48 578
1 2011 N/A 1 N/A 80 N/A N/A N/A N/A N/A 587
219 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline
2 2007 0 Baseline 2 Baseline 150 580 1,186 352 930 3,200
2 2008 1,780 179 159 1,143 152 6,908 8,243 13,884 6,261 38,709
2 2009 2,385 1,040 1,277 42,488 1,287 8,700 9,158 17,597 7,561 91,493
2 2010 2,505 1,229 1,886 44,741 3,516 10,420 11,004 26,196 17,596 119,093
2 2011 N/A 1,768 N/A 47,346 N/A N/A N/A N/A N/A 122,237
320 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline
3 2007 0 Baseline 1 Baseline 17 0 11 15 3 47
3 2008 0 0 5 31 19 6 15 33 7 116
3 2009 2 0 6 42 19 7 21 56 9 162
3 2010 2 0 7 50 25 10 27 65 30 216
3 2011 N/A 0 N/A 53 N/A N/A N/A N/A N/A 219
421 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline
4 2007 1 Baseline 0 Baseline 2 0 47 10 1 61
4 2008 30 1 16 19 11 1 68 22 15 183
4 2009 63 6 47 36 47 5 78 34 15 331
4 2010 67 9 50 55 114 5 183 43 44 570
17 Please note that data are displayed as reported by the States and have not been cleaned by the cross-site evaluation team. Numbers
may change upon cleaning of the data.
 “Missing” means that target numbers should have been entered but were not.
 “N/A” means there are/were no GPRA reporting requirements for a State for that year.
 Each target reflects infrastructure changes that are projected to be completed by September 30 of the target grant year; therefore, changes
projected to be completed after September 30 count toward the next year, and changes projected to be completed after the end of a State’s
grant do not appear in the table. Data were extracted from Transformation Tracker on October 1, 2009.
 For all GPRA indicators except GPRA 6, targets were derived by adding the target numbers listed by year within the “Target Population”
column of the “Key Features of Infrastructure Activities” report of Transformation Tracker.
18 GPRA 1: Number of policy changes completed as a consequence of the CMHP
19 GPRA 2: Number of persons in the mental health care and related workforce who have been trained in service improvements recommended by
the CMHP
20GPRA 3: Number of financing policy changes completed as a consequence of the CMHP
21GPRA 4: Number of organizational changes completed as a consequence of the CMHP
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 47
GPRA Measure FY CT HI MD MO NM OH OK TX WA Program
4 2011 N/A 9 N/A 56 N/A N/A N/A N/A N/A 571
522 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline
5 2007 0 Baseline 0 Baseline 16 0 2 23 0 41
5 2008 0 0 0 96 33 498 38 27 0 692
5 2009 11 1 3 6,709 66 514 192 27 10 7,533
5 2010 324 14 42 6,863 246 532 407 27 21 8,476
5 2011 N/A 14 N/A 6,873 N/A N/A N/A N/A N/A 8,486
623 2007 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline
6 2008 14,158 Baseline 4,826 Baseline 1,562 17,155 3,351 15,500 10,136 66,688
6 2009 14,474 Missing 4,906 Missing Missing 30,055 3,656 16,500 12,522 82,113
6 2010 14,802 Missing 4,986 Missing Missing 31,355 3,961 17,500 13,019 85,623
6 2011 N/A Missing N/A Missing N/A N/A N/A N/A N/A 85,623
724 2006 Baseline N/A Baseline N/A Baseline Baseline Baseline Baseline Baseline Baseline
7 2007 0 Baseline 0 Baseline 0 515 86 3 1 605
7 2008 0 0 19 199 60 940 101 99 1 1,419
7 2009 4 11 79 661 98 1,239 281 290 12 2,675
7 2010 4 17 217 831 196 1,601 589 580 326 4,361
7 2011 N/A 21 N/A 1,046 N/A N/A N/A N/A N/A 4,580
22 GPRA 5: Number of organizations that regularly obtain and analyze data relevant to the goals of the CMHP
23 GPRA 6: Number of consumers and family members that are members of Statewide consumer- and family-run networks. For GPRA 6, targets
were derived from the “GPRA 6 Membership Numbers” table within the “Overview of Infrastructure Activities” section of Transformation Tracker.
24 GPRA 7: Number of programs implementing practices consistent with the CMHP
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 48
Appendix B: GPRA Results
Table 30. Cumulative GPRA Results/Accomplishments25
GPRA Measure Year CT HI MD MO NM OH OK TX WA Program
126 2006 0 N/A 0 N/A 0 0 14 1 0 15
1 2007 12 0 0 5 3 0 43 13 15 91
1 2008 33 0 5 21 6 18 57 57 32 229
1 2009 33 0 5 54 6 20 61 70 32 281
1 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10
1 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11
227 2006 0 N/A 0 N/A 0 0 0 0 0 0
2 2007 0 0 22 0 0 180 406 395 2,602 3,605
2 2008 2,915 179 1,455 1,769 2,236 4,516 13,928 18,729 10,740 56,467
2 2009 2,920 1,219 2,319 45,534 2,236 7,016 15,177 22,764 17,475 116,660
2 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10
2 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11
328 2006 0 N/A 0 N/A 0 0 8 0 0 8
3 2007 0 0 2 6 19 0 12 9 3 51
3 2008 0 0 5 28 32 2 20 43 17 147
3 2009 1 0 5 39 32 2 26 58 19 182
3 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10
3 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11
429 2006 0 N/A 0 N/A 0 0 6 0 0 6
4 2007 1 0 4 4 0 0 26 15 2 52
4 2008 29 0 44 18 20 1 44 32 8 196
4 2009 29 0 52 35 20 1 52 44 34 267
4 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10
25 Please note that data are displayed as reported by the States and have not been cleaned by the cross-site evaluation team. Numbers
may change upon cleaning of the data.
 “Missing” means there were no results reported as of September 30 of the target year.
 “N/A” means there are/were no GPRA reporting requirements for a State for that year.
 Each result reflects infrastructure changes with impacts reported up to September 30 of the target year; therefore, changes completed after
September 30 of a calendar year count toward the next year, and changes completed outside of a State’s grant period do not appear in the
table. Data were extracted from Transformation Tracker on October 1, 2009.
 For all GPRA indicators except GPRA 6, results were derived by summing the reported “impact” fields in the downloaded “Activity Results”
entries for each State. Results were summed by fiscal year based on reported completion date.
26GPRA 1: Number of policy changes completed as a consequence of the CMHP
27GPRA 2: Number of persons in the mental health care and related workforce who have been trained in service improvements recommended by
the CMHP
28GPRA 3: Number of financing policy changes completed as a consequence of the CMHP
29GPRA 4: Number of organizational changes completed as a consequence of the CMHP
MHT SIG Evaluation Project, Annual Evaluation Report, October 22, 2009 49
GPRA Measure Year CT HI MD MO NM OH OK TX WA Program
4 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11
530 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11
5 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10
5 2009 0 0 17 6,772 65 0 39 9 18 6,920
5 2008 0 0 7 106 65 0 31 5 18 232
5 2007 0 0 0 0 30 0 2 5 0 37
5 2006 0 N/A 0 N/A 0 0 1 0 0 1
631 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11
6 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10
6 2009 Missing Missing Missing Missing Missing Missing 4,871 Missing Missing 4,871
6 2008 14,386 1,520 6,431 909 1,562 17,476 3,754 13,256 7,679 66,973
6 2007 13,842 N/A 4,739 N/A Missing 19,395 3,046 14,700 6,689 62,411
732 2011 N/A TBD-11 N/A TBD-11 N/A N/A N/A N/A N/A TBD-11
7 2010 TBD-10 Oct-10 TBD-10 Oct-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10 TBD-10
7 2009 0 0 159 702 67 838 167 316 255 2,504
7 2008 0 0 114 170 67 753 135 99 247 1,585
7 2007 0 0 0 0 0 117 58 4 1 180
7 2006 0 N/A 0 N/A 0 0 28 0 0 28
30GPRA 5: Number of organizations that regularly obtain and analyze data relevant to the goals of the CMHP
31GPRA 6: Number of consumers and family members that are members of Statewide consumer- and family-run networks. For GPRA 6, results
were derived from the “GPRA 6 Membership Numbers” table within the “Overview of Infrastructure Activities” section of Transformation Tracker.
32GPRA 7: Number of programs implementing practices consistent with the CMHP
MHT SIG Evaluation Project, Annual Report, October 22, 2009 50
Appendix C: Proof of Concept Studies Information
Table 31. State Interventions Examined in Recovery (Adult) Study
State Impacted Group Nonimpacted Group
Connecticut Individualized Recovery Planning tools
with advanced training, infrastructure
support, technical assistance (TA)—
professional and peer staff331
Individualized Recovery Planning tools with
2-day training (not receiving advanced
training, infrastructure support, or TA)
Maryland Training and consultation providers and
consumers in psychiatric rehabilitation
programs regarding Cultural
Competence and elimination of racial
disparities in service provision1
Psychiatric rehabilitation programs not
receiving training
New Mexico Integrated Dual Diagnosis Treatment
(IDDT) for adults diagnosed with co-occurring
disorders in nine Certified
Comprehensive Community Support
Services (CCSS) Contractor Sites
Provider sites not yet trained or certified in
IDDT or the CCSS Model
Ohio Using Outcomes in Person-Centered
Treatment Planning (training for staff
and consumers)
Wait-list agencies
Oklahoma Illness Management and Recovery—
structured course with guided
curriculum for Psychiatric Rehabilitation
Service consumers
Wait-list
Texas Self-directed/consumer-directed care—
control over funds to purchase services
Services as usual
Washington Journey to Life Wellness/Home Team—
peer support and education on recovery
and use of wellness tools
Sequential design (pre and post) will do
concurrent (randomized) next year if funds
permit
33Intervention aims to enhance existing intervention/programs.
MHT SIG Evaluation Project, Annual Report, October 22, 2009 51
Table 32. State Interventions Examined in Resilience (Child) Study
State Impacted Group Nonimpacted Group
Connecticut Wraparound with infrastructure support1 Wraparound without infrastructure support
Maryland Wraparound Service Model Local Coordinating Councils (jurisdictions
not yet implementing WSM)
New Mexico Clinical Home—Comprehensive
Service Agency model for referrals from
Juvenile Justice division, eventually
Protective Services Division
Youth assessment center with services as
usual
Ohio Trauma Affect Regulation Guide for
Education and Therapy (TARGET)—
manualized treatment and prevention
for Department of Youth Services
Juvenile Correctional Facilities
Mental health services as usual
Oklahoma Cross-agency care coordination
oversight for children, youth, and their
families with high-level service use,
funding redirected from residential to
community-based services
Comparable population, randomized to
services as usual
Texas Enhancing Fidelity to a Resiliency and
Disease Management System: staff
training and consultation1
Resiliency and Disease Management
programs without training and consultation
Washington Mental health services in middle-school,
school-based health centers
Schools without school-based health
centers
Tables 33 through 36 provide information about the GPRA measures and NFC goals being addressed by
the State POC studies. Shaded boxes indicate the specific measures that are addressed by each State’s
recovery (Tables 33 and 34) and resilience (Tables 35 and 36) POC studies.
Table 33. GPRA Measures Addressed Recovery (Adult) Studies
State 1 2 3 4 5 6 7
Connecticut n Y(2) n n Y n n
Maryland n Y n n n n Y
New Mexico n n Y n n Y
Ohio n Y n n n n n
Oklahoma Y Y n Y Y n Y
Texas Y n Y n Y n n
Washington n Y Y n Y n Y
Table 34. NFC Goals Addressed by Recovery (Adult) Studies
NFC 1 2 3 4 5 6
Connecticut n Y n n n n
Maryland Y Y Y n Y n
New Mexico n Y n n Y n
Ohio n Y n n n n
Oklahoma Y Y n n Y n
Texas n Y n n Y n
Washington Y Y n n n n
MHT SIG Evaluation Project, Annual Report, October 22, 2009 52
Table 35. GPRA Measures Addressed by Resilience (Child) Studies
State 1 2 3 4 5 6 7
Connecticut Y Y n Y Y(2) n n
Maryland n Y Y n Y n Y
New Mexico n n n Y n n Y
Ohio n Y n n n n n
Oklahoma Y Y n Y Y n Y
Texas n n n n n n n
Washington1 Y Y Y Y Y Y Y
Table 36. NFC Goals Addressed by Resilience (Child) Studies
State 1 2 3 4 5 6
Connecticut n Y n n Y n
Maryland Y Y Y Y Y n
New Mexico n n n n Y n
Ohio n n n Y Y Y
Oklahoma n Y Y Y Y n
Texas n Y n n n n
Washington1 Y Y Y Y Y Y
Tables 37 through 40 below present the instruments for measuring system- and individual-level recovery
in adults and individual resilience in children selected by each site from the approved list. It is apparent
that there is some variability, with no more than three sites using any one measure.
Table 37. Individual-level Recovery Instruments Being Used by Grantee States
Instrument States Using
Recovery Markers Questionnaire (RMQ) CT, NM, WA
Mental Health Recovery Measure (MHRM) MD, OK
Ohio Mental Health Consumer Outcomes system OH
Recovery Assessment Scale (RAS) TX
Table 38. System-level Recovery Orientation Instruments Being Used by Grantee States
Instrument States Using
Recovery Self-Assessment Revised (RSA-R) CT, MD, OH
Recovery Oriented System Indicators Measure (ROSI) WA
Recovery Enhancing Environment Measures (REE) NM
Elements of a Recovery Facilitating System (ERFS–Adult) OK, TX
Optional/other: Discrimination Experience Subscale/ISMI WA
Table 39. Individual-level Resilience Instruments Being Used by Grantee States
Instrument States Using
Developmental Assets Profile (DAP) CT, WA
Behavioral and Emotional Resilience Scale (BERS-2) MD, TX
Ohio Youth Problems, Functioning, and Satisfaction Scales OH, OK
Resilience Scale NM
Optional/other: Strengths and Difficulties Questionnaire OK
MHT SIG Evaluation Project, Annual Report, October 22, 2009 53
Table 40. System-level Resilience Orientation Instruments Being Used by Grantee States
(Optional)
Instrument States Using
Elements of a Recovery Facilitating System (ERFS–Child Adapted) CT
Community Supports for Wrap-Around Inventory WA
Recovery Self-Assessment Revised (RSA-R) OH
Optional/Other: Consumer Perception of Program Cultural Competence MD
MHT SIG Evaluation Project, Annual Report, October 22, 2009 54
Appendix D: National Outcome Measures from the CMHS Uniform Reporting System, 2004–2008
Table 41. NOM 2: Increased or Retained Employment and School Enrollment: Percentage of
Adults With Known Status Who Are Employed
State 2004 2005 2006 2007 2008
2004–2008
Change
Connecticut 24 24 24 25 -- +
Hawaii 19 24 23 23 -- +
Maryland -- 14 14 16 -- +
Missouri 19 17 -- 15 -- -
New Mexico 21 20 25 21 -- None
Ohio 16 22 23 23 -- +
Oklahoma 18 18 19 20 -- +
Texas 15 16 17 17 -- +
Washington 12 11 11 12 -- None
United States 21 21 22 21 -- None
Table 42. NOM 4: Increased Stability in Family and Living Conditions: Percentage of Adults With
Known Status Who Are Living in Private Residence
State 2004 2005 2006 2007 2008
2004–2008
Change
Connecticut 72.6 76.2 73.7 78.7 80.8 +
Hawaii 75.0 65.3 65.9 67.2 70.0 -
Maryland -- 77.1 79.2 80.1 80.8 +
Missouri 33.0 -- -- 80.1 78.5 +
New Mexico 90.1 85.8 89.6 89.7 90.9 +
Ohio -- 82.3 83.2 83.4 82.6 +
Oklahoma 85.3 85.8 86.5 87.9 89.7 +
Texas 91.7 90.4 90.6 91.2 91.4 -
Washington 78.6 79.5 78.0 77.9 78.1 -
United States 74.9 79.6 79.5 80.1 80.8 +
MHT SIG Evaluation Project, Annual Report, October 22, 2009 55
Table 43. NOM 5: Increased Access to Services: Percentage of Persons Served by Age:
Percentage of Clients Who Are Under Age 18 or Are Age 65 or Over
State Categories 2004 2005 2006 2007 2008
Percent of
State/U.S.
2008
Population
Connecticut Under 18 42.1 35.0 35.6 35.1 35.0 23.2
Age 65+ 4.7 5.6 4.3 4.2 4.4 13.7
Hawaii Under 18 21.0 16.6 22.1 16.7 15.6 22.1
Age 65+ 8.0 7.0 5.4 4.2 6.8 14.8
Maryland Under 18 48.1 47.0 50.7 45.0 43.8 23.8
Age 65+ 1.4 1.3 1.2 1.2 1.2 12.1
Missouri Under 18 20.4 21.2 22.5 22.4 21.9 24.0
Age 65+ 3.1 2.8 2.8 2.9 2.8 13.6
New Mexico Under 18 -- 43.1 40.1 40.9 42.3 25.3
Age 65+ 3.8 1.9 1.8 1.7 1.7 13.1
Ohio Under 18 33.8 34.2 33.8 34.1 34.0 23.8
Age 65+ 2.9 2.9 2.8 2.8 2.9 13.7
Oklahoma Under 18 11.4 12.9 13.3 12.9 13.4 24.9
Age 65+ 2.6 2.3 2.2 1.7 1.7 13.5
Texas Under 18 18.2 18.7 19.1 19.4 19.8 27.6
Age 65+ 3.2 3.1 2.9 2.8 2.7 10.2
Washington Under 18 27.4 27.5 27.4 26.7 26.1 23.5
Age 65+ 7.6 7.2 6.5 6.5 6.6 12.0
United States Under 18 25.0 24.8 24.6 27.0 27.0 24.3
Age 65+ 4.8 4.6 4.5 4.8 4.8 12.8
Table 44. NOM 5: Increased Access to Services: Percentage of Persons Served by Race/Ethnicity
(Percent of Clients Who Are White)
State 2004 2005 2006 2007 2008
Percent of
State/U.S. 2008
Population
Connecticut 59.2 59.3 55.2 63.0 65.1 83.9
Hawaii 27.2 26.6 26.5 24.6 23.9 27.3
Maryland 42.6 44.1 44.1 45.9 46.6 61.8
Missouri 76.1 75.8 74.7 75.1 73.6 84.9
New Mexico 62.4 45.2 54.8 51.4 50.2 75.8
Ohio 74.1 73.5 73.5 73.3 73.1 84.7
Oklahoma 78.1 78.2 78.4 78.8 78.9 77.3
Texas 74.9 75.6 74.6 74.6 74.5 74.7
Washington 71.5 71.1 69.8 69.2 69.2 83.7
United States 74.3 74.1 73.9 63.3 63.3 77.6
MHT SIG Evaluation Project, Annual Report, October 22, 2009 56
Table 45. NOM 6: Decreased Utilization of Psychiatric Inpatient Beds: Mental Health Hospital
Utilization Rate per 1,000
State 2004 2005 2006 2007 2008
2004–2008
Change
Connecticut .53 .56 .46 .43 .43 negative
Hawaii .27 .32 .33 .32 .30 +
Maryland .67 .62 .60 .51 .47 negative
Missouri 1.5 1.39 1.33 1.27 1.15 negative
New Mexico .48 .56 .60 1.24 .54 +
Ohio -- -- -- -- -- N/A
Oklahoma .83 .95 .85 .72 .74 negative
Texas .69 .68 .67 .65 .61 negative
Washington .57 .53 .52 .53 .52 negative
United States .61 .63 .59 .60 .59 negative
Table 46. NOM 7: Increased Social Support/Social Connectedness: Percentages for Adults and
Children/Families34
State
Adult
2006
Adult
2007
Adult
2008
Adult
2006-
2008
Change
Child/Family
2006
Child/Family
2007
Child/Family
2008
Child/Family
2006-2008
Change
Connecticut 73 77 75 + 86 83 90 +
Hawaii -- 67 69 + -- 75 83 +
Maryland 68 73 69 + 83 84 86 +
Missouri -- 69 69 None -- 81 83 +
New
Mexico
-- 97 97 None 84 93 89 +
Ohio 36 37 37 + -- -- -- N/A
Oklahoma 93 93 95 + 91 -- 98 +
Texas 62 63 61 - 79 77 72 negative
Washington -- 62 63 + -- 86 86 None
United
States
73 72 73 None 79 84 86 +
Table 47. NOM 8: Increased Positive Reporting by Clients About Outcomes: Percentage of Adult
Consumers With Positive Reporting of Outcomes
State 2004 2005 2006 2007 2008
2004–2008
Change
Connecticut 78 81 78 80 77 negative
Hawaii 68 69 71 70 76 +
Maryland 63 61 66 70 63 None
Missouri -- 81 -- 71 70 negative
New Mexico 74 70 80 91 83 +
Ohio 68 69 67 85 68 None
Oklahoma 96 96 97 90 92 negative
Texas 60 60 53 57 55 negative
Washington 58 58 66 65 60 +
United States 71 71 71 71 72 +
34 Measures of Social Support/Social Connectedness had been in developmental status prior to 2006.
MHT SIG Evaluation Project, Annual Report, October 22, 2009 57
Table 48. NOM 10: Increased Use of Evidence-Based Practices: Percentage of States
Implementing Adult Supported Housing
State 2004 2005 2006 2007 2008
2004–2008
Change
Connecticut -- 9.8 9.8 10.7 9.9 +
Hawaii -- 6.6 7.6 6.1 4.6 negative
Maryland -- 18.6 14.4 14.8 15.2 negative
Missouri -- -- -- -- -- N/A
New Mexico 3.2 2.4 -- -- 0.3 negative
Ohio -- -- -- -- -- N/A
Oklahoma 1.0 0.9 0.8 0.6 0.5 negative
Texas 4.4 2.7 2.3 2.8 2.7 negative
Washington 0.3 -- -- 4.1 5.4 +
United States 4.7 5.0 5.5 5 3 negative
Table 49. NOM 10: Increased Use of Evidence-Based Practices: Percentage of States
Implementing Adult Supported Employment
State 2004 2005 2006 2007 2008
2004–2008
Change
Connecticut -- -- -- -- 3.8 N/A
Hawaii -- 6.1 6.5 4.3 3.2 negative
Maryland 57.91 4.3 4.5 5.3 5.7 +
Missouri -- -- -- 2.7 9.1 +
New Mexico 9.6 6.6 2.8 2.2 1.7 negative
Ohio -- 100351 1.7 1.7 1.5 negative
Oklahoma 0.2 0.1 0.1 0.1 0.1 negative
Texas 3.5 2.7 1.7 1.6 1.2 negative
Washington 1.8 -- -- 2.9 3.1 +
United States 3.1 2.5 2.7 2 2 negative
Table 50. NOM 10: Increased Use of Evidence-Based Practices: Percentage of States
Implementing Adult Assertive Community Treatment
State 2004 2005 2006 2007 2008
2004–2008
Change
Connecticut -- -- -- -- -- N/A
Hawaii -- 7.6 6.7 4.5 3.5 negative
Maryland -- 3.2 3.8 4.3 4.6 +
Missouri -- -- -- -- 0.2 N/A
New Mexico -- -- 0.2 0.2 0.3 +
Ohio -- -- -- 1.7 1.8 +
Oklahoma -- 1.9 2.4 2.9 2.4 +
Texas -- 2.6 2.1 2.1 1.8 negative
Washington -- -- -- 1.2 1.3 +
United States 2.2 2.2 2.8 3 2 negative
35These data appear to be in error.
MHT SIG Evaluation Project, Annual Report, October 22, 2009 58
Table 51. NOM 10: Increased Use of Evidence-Based Practices: Percentage of States
Implementing Therapeutic Foster Care for Children and Adolescents
State 2004 2005 2006 2007 2008
2004–2008
Change
Connecticut -- 5.0 5.0361 -- -- None
Hawaii -- 12.6 9.1 10.3 12.6 None
Maryland -- -- -- -- 5.2 N/A
Missouri -- -- -- 1.7 -- N/A
New Mexico -- 5.8 5.0 4.2 5.4 negative
Ohio -- -- -- -- -- N/A
Oklahoma -- -- -- -- -- N/A
Texas -- -- -- -- -- N/A
Washington -- -- -- 1.8 1.9 +
United States -- 2.1 2.0 3 2 negative
362006 figure for Connecticut is based on communication with State staff.