Tools In Development:
Measuring Recovery at the Individual, Program, and System Levels
For many SMHA administrators, offering reimbursable programs and services
is the only way to maintain a viable mental health organization. This
scenario is not always conducive to implementing new methods and services,
many of which are not yet considered evidence-based practices. Recovery
should be measurable, though, because for SMHAs it’s an accountability
issue, and there are quantitative and qualitative aspects to it at the
individual, program, and system levels. Today’s SMHAs need concrete,
replicable, measurable services, and therefore many state mental health
officials are looking for tools to help quantify recovery-based care.
Despite the reality that research on recovery measurement is in a relatively
fledgling state, there are a variety of recovery-themed measurement tools
in various stages of development that are being created to help administrators,
clinicians, peer providers, and consumers institute recovery-based care
into mental health settings. This special report features a sampling of
three separate recovery measurement tools—all in development—including
descriptions and progress updates written by representatives of the three
efforts.
Ruth Ralph, PhD, offers a report on the effort to create a personal measure
of recovery within the Recovery Measurement Tool and the Recovery Model.
Priscilla Ridgway, MSW, provides an update on the Recovery Enhancing Environment
Measure, which is intended to assess the recovery orientation of mental
health programs, and Steve Onken, PhD, offers a look at a system-wide
tool, the Recovery Oriented System Indicators (ROSI) Measure.
Editor’s note: This is not an exclusive list of measures, as
there are other measures for recovery in use and/or in development across
the country. NASMHPD/NTAC does not necessarily endorse the aforementioned
three measures over any others.
At the Individual Level:
A Personal Measure of Recovery
By Ruth O. Ralph, Ph.D.
Dr. Ralph, a retired senior research associate with the Edmund S.
Muskie School of Public Service at the University of Southern Maine, is
a consumer researcher who has conducted mental health research and evaluation
for over 25 years.
The Recovery Measurement Tool (RMT) is based upon The Recovery Advisory
Group Recovery Model (Ralph & The Recovery Advisory Group, 1999).
It is important to review this model in order to understand the background
of the RMT.
When the Recovery Advisory Group was formed in 1999, monthly teleconferences
were held to discuss recovery issues. The discussions were based on the
considerable experience of the group’s members and exchange and
discourse of readings about recovery, both published and unpublished.
It was the hope of the group’s CMHS funders that these discussions
would lead to measurement of recovery.
However, the group focused on definition(s) of recovery, and realized
that there needed to be a visual way of portraying recovery. Thus, The
Recovery Model was developed, discussed, and revised until members of
the group felt it truly portrayed a viable description of recovery.
There are several parts to the model. The first shows a series of ovals
to portray the stages one goes through on the way to recovery (MS
PowerPoint, pdf).
These are shown as: anguish, awakening, insight, action plan, determination
to be well, and well-being/recovery. The progress through these stages
is not linear, but involves considerable moving back and forth, from one
stage to another, and possibly backward when things aren’t going
well. The ovals are arranged in a beginning spiral, to show that one can
move through the spiral, and possibly start again, but maybe on a higher
plane.
The second part of the model is a grid that shows these stages across
the top, with internal and external domains down the left side (MS
Word, pdf).
In the model, each “box” has a statement that describes recovery
at that point in the journey. This grid became an important component
in the development of the Recovery Measurement Tool. The third part of
the model shows both negative and positive external influences on a person
who is working on recovery (MS
PowerPoint, pdf).
The fourth part summarizes the model by placing the person who is going
through the stages in the center (represented by a circle) surrounded
by both negative and positive external influences (MS
PowerPoint, pdf).
This part is called “The Big Picture.”
The development of The Recovery Measurement Tool (RMT) was funded by
the Center for the Study of Public Issues in Mental Health located in
New York State. A group of consumers from Maine met and reviewed the Recovery
Model, with special attention given to the grid with the descriptive statements
about the stages of recovery. Using the grid, we developed one or more
items for each of the intersecting boxes in the grid. For example, the
item “There is hope for me even when I do not feel well,”
is in the Insight/Emotional box. The item “I visit a number of places
to see where I can make friends,” is in the Determined Commitment/Activities
box.
The result, over a number of meetings, was the development of 100 items.
While we worked on this project, we read and discussed information about
how to write items—how they should have only one idea, and how they
should be clearly written so that they could be understood easily.
We also learned about research issues, and how to “test a test.”
We came together as strangers, but through this learning experience we
bonded as co-workers and friends. We found that while we came from different
backgrounds and had different experiences, “our feelings were the
same, and we learned about ways to express them in order that others might
also be able to express theirs” (Kidder, 2001).
In selecting a response set for these items, we reviewed a number of
different types. However, because we wanted this to be a personal measure
of recovery, we decided to make the responses range from “not at
all like me,” through “not very much like me,” and “somewhat
like me” to “quite a bit like me” and “very much
like me.” We also included a column of “not applicable.”
A thorough review of the items revealed nine items that were duplicates
or near duplicates of other items, so they were deleted. The result is
a total of 91 items.
Items were randomly arranged with the response format placed at the right
of each item. Some information about demographics, e.g., age, sex, education,
race/ethnicity, and location were added. This instrument has not been
tested with any group at this time. It is hoped that this can be done
soon, and a shorter, valid instrument can be constructed from the results.
References
Kidder, K. (2001) The role of advocacy in reducing stigma and increasing
hope and self-esteem in people recovering from mental health disorders.
Presentation for Mental Health Awareness Week sponsored by NAMI Choices.
Portland, ME.
Ralph, R. O., & The Recovery Advisory Group. (1999). The recovery
advisory group recovery model. National Conference on Mental Health
Statistics.
Back to Top
At the Program Level:
The Recovery Enhancing Environment Measure
By Priscilla Ridgway, M.S.W.
Ms. Ridgway consults with mental health agencies and is a doctoral
student at the University of Kansas (KU), School of Social Welfare.
Increasing importance is being placed on moving mental health systems
toward a recovery orientation. The need to shift the driving focus of
service delivery is evident in strong recommendations made by the Surgeon
General (US DHHS, 1999) and the President’s New Freedom Commission
on Mental Health (2003). Given the call for such profound change, mental
health administrators across the country are asking questions such as:
What services and supports are important in a recovery-oriented mental
health system? What recovery-facilitating practices are currently underdeveloped
in our system? Where are the people we serve on their journey of recovery?
How well are we facilitating people’s potential for resilience and
recovery? The Recovery Enhancing Environment measure (REE) was designed
to provide empirical answers to such pressing concerns.
Developing and Testing the REE Measure
REE was created in 1999. The content of the measure was developed based
on: 1) an examination of first person accounts of the process of mental
health recovery and the services and supports people say enhance their
recovery; 2) a review of emerging promising practices that promote recovery
drawn from informal literature, workshop descriptions, and progressive
programs; and, 3) a literature review of factors that facilitate resilience,
or that help people rebound from adversity, in general.
The REE measure was reviewed and pre-tested by Kansans involved in a
Consumer-as-Provider training program, and later by persons served by
a day treatment program. Items were revised, dropped, and added based
on consumer input. The measure underwent technical edits and the format
was refined based on the input of colleagues Allan Press and Patricia
E. Deegan.
Two formal pilot tests were conducted on the REE. In 2002, the Kansas
Department of Social and Rehabilitation Services funded a mail survey
of those served in the seven largest community support programs in the
state. It was conducted by the Kansas University Office of Mental Health
Research and Training in collaboration with participating community mental
health centers (Ridgway, Press, Ratzlaff, Davidson, & Rapp, 2003).
More recently, Pat Deegan & Associates trained a cadre of mental health
consumers to gather REE data in face-to-face interviews with nearly half
of those served by a large Massachusetts mental health agency (Ridgway,
Press, Anderson, & Deegan, in preparation). More than 500 people completed
the REE in the two pilots. Preliminary statistical analyses indicate that
the instrument is psychometrically sound.
Content of the Measure
The REE instrument examines personal recovery by looking at the respondent’s
self-perceived stage of recovery and the markers of recovery (intermediate
outcomes) they currently experience. Respondents rate the degree to which
24 elements are important to their personal recovery to increase our understanding
of the process, using a 5 point scale ranging from “strongly agree”
to “strongly disagree.” These elements include hope, being
able to manage symptoms, overall health and wellness, having one’s
rights respected, being involved in and a part of the larger community,
having meaningful activities, taking on normal social roles, having positive
relationships, identifying and building on personal strengths, having
one’s basic needs met, self-help and peer support, and others.
In order to assess the recovery orientation of the agency, people rate
the current status of three specific staff behaviors or services that
support each recovery element. Consumer perceptions are also gathered
on the importance and existence of qualities of the service environment
that have been found to enhance the potential for resilience. These include
the presence of caring and compassionate helpers; opportunities for meaningful
participation and contribution; being connected to others in positive
ways; and feeling valued, respected, and powerful, among others. The degree
to which the agency meets the perceived needs of people on dual journeys
of recovery (those who experience dual diagnoses, trauma survivors), those
from minority cultural backgrounds and sexual orientations, and recipients
who are parents is also examined.
Findings
The findings of the pilots show that mental health recovery is a multi-dimensional
process. People are able to place themselves within a particular stage
of recovery. Most people are in an active phase of recovery, but many
are not yet in recovery, a few experience setbacks, and some view themselves
as fully recovered, but having to maintain their gains. Many respondents
have one or more special needs. Almost all people can identify at least
a few indicators of recovery in their lives, and many are able to claim
several markers of personal recovery. These markers include being involved
in productive activities, having trusted people to turn to for help, having
goals one is working to achieve, controlling important decisions, feeling
hopeful about the future, having one’s symptoms under control, working,
having a sense of belonging, and several others. These markers have performed
well as a measure of change over time in other research. People gain ground
concurrent with, or in part due to, exposure to recovery-enhancing programming.
REE begins to define a complex set of activities that enhance the potential
for recovery from the perspective of service recipients. Consumers were
able to differentiate higher performing programs from lower performing
programs on the basis of statistically significant differences in the
mean rating of program performance, as well as significant differences
within and among programs along specific dimensions of practice.
Listening to the Voice of Lived Experience
The REE includes open-ended questions that ask people about the lessons
they have learned in their own recovery and that give advice to staff
and to others just starting out on a journey of recovery. These data were
analyzed by themes. A few quotes give the flavor of the wisdom people
generously shared.
“Never give up on you. Keep your head up. Everyone else might give
up,
but don’t give up on you, because if you do, there is nothing else
to live for.
Do the best you can do every day. Never quit. Don’t say I can’t.
Can’t never did anything.”
“Don’t go it alone. We are survivors, and although the trip
is hard, it is well worth the wealth of knowledge, understanding and the
happiness you will be
able to regain.”
“You will need a philosophy. I have one for you. It is four words,
all action
verbs: work, play, rest, love (what you can).”
Discussion
The REE allows programs to gather data from the recipients of services
to assess gaps in important dimensions of recovery-enhancing programming,
and to understand the needs of those who have not yet begun to recover
or to experience recovery outcomes. In the spirit of appreciative inquiry,
REE data can also be used to identify and celebrate positive elements
of the program that are already in place that support people in their
recovery, and the progress and wisdom of participants who are already
moving forward in recovery.
Data from the REE can be used to plan recovery interventions, retrain
staff, and target program innovations. The measure is intended to serve
as a useful tool in processes of strategic planning that involve consumers,
staff and administrators in point-in-time or on-going agency self-assessment
or program evaluation efforts.
REE requires a significant investment of time and resources, but these
are well spent if the data are used to identify priorities for program
development and to set specific targets for transforming mental health
agencies, thereby enhancing the potential for recovery among people using
mental health services. REE is one of several tools that can help the
field move toward fulfilling the vision that all mental health systems
assume a recovery orientation, and that recovery is possible for everyone.
The REE measure is complete and in a scantrons (computer read) format.
Permission to use the measure and ordering information are available from
PRidgway@ku.edu. A brief user’s
guide, an MS Word version of the measure, a booklet summarizing the wisdom
of people in recovery, and reports on the pilots should be available as
of September 2004, from the Office of Mental Health Research and Training,
University of Kansas, School of Social Welfare, Lawrence KS, 66045.
References
The President’s New Freedom Commission on Mental Health. (2003).
Achieving the promise: Transforming mental health care in America.
Final report. DHHS Pub. No. SMA-03-3832. Rockville, MD: U.S. Department
of Health and Human Services.
Ridgway, P., Press, A., Anderson, D., & Deegan, P.E. (in preparation).
Pilot testing the Recovery Enhancing Environment Measure: The Massachusetts
experience. Byfield, MA: Pat Deegan & Associates.
Ridgway, P., Press, A., Ratzlaff, S., Davidson, L., & Rapp, C.A.
(2003). Report on field testing the Recovery Enhancing Environment
Measure. Lawrence, KS: School of Social Welfare, Office of Mental
Health Research and Training.
U.S. Department of Health and Human Services (US DHHS). (1999). Mental
health: A report of the Surgeon General. Rockville, MD: U.S. Department
of Health and Human Services, Substance Abuse and Mental Health Services
Administration, Center for Mental Health Services, National Institutes
of Health, National Institute of Mental Health.
Back to Top
At the System Level:
Consumer Self-Report Survey and Administrative-Data Profile
By Steven J. Onken, PhD, representing his fellow researcher team
members Jeanne M. Dumont, PhD, Priscilla Ridgway, MSW, Douglas H. Dornan,
MS, and Ruth O. Ralph, PhD.
Dr. Onken is an assistant professor at the Columbia University School
of Social Work and a co-principal investigator of the Mental Health Recovery:
What Helps and What Hinders? National Research Project.
The Recovery Oriented System Indicators (ROSI) Measure is the product
of the Mental Health Recovery: What Helps and What Hinders? A National
Research Project for the Development of Recovery Facilitating System Performance
Indicators effort. This research project evolved from collaborative efforts
among a team of consumer and non-consumer researchers, state mental health
authorities (SMHAs), and a consortium of sponsors working to operationalize
a set of mental health system performance indicators for mental health
recovery.
Conceptualized and directed by a five-member research team (the majority
of whom are primary consumers) as a three-phase process (i.e. grounded
theory inquiry concerning the phenomenon of recovery; creation of prototype
systems-level performance indicators; and large-scale pilot testing),
Phase One and Two have been completed. This briefing summarizes the research
process and resulting ROSI measure.
Phase One involved a grounded theory, multi-site qualitative design to
identify the person-in-environment factors that help or hinder recovery
for people experiencing severe and persistent mental illness. Nine SMHAs
used purposive sampling to recruit 115 consumers that participated in
10 structured focus groups. Researchers used rigorous, constant and comparative
analytic methods involving qualitative coding, codebook development, cross
coding and recoding of the focus group transcripts to develop a single
set of findings. All nine SMHAs conducted member checks with focus group
participants regarding the coding report for their respective focus group.
The research achieved a “confirmability index” (agreement
that the coding captured the original content) of 99.47% among the 59
who responded (51% of the original sample).
A conceptual paradigm for organizing and interpreting the phenomenon
of mental health recovery emerged from the findings. While recovery is
a deeply personal journey, there are many commonalities in people’s
experiences. Recovery is facilitated or impeded through the dynamic interplay
of many forces that are complex, synergistic, and linked. Recovery is
a product of dynamic interaction among characteristics of the individual
(self-agency, holism, hope, a sense of meaning and purpose), characteristics
of the environment (basic material resources, social relationships, meaningful
activities, peer support, formal services and staff), and the characteristics
of the exchange (hope, choice, empowerment, referent power, independence,
interdependence). Each of these emergent domains/themes contain a rich
and complex network of helping and hindering elements.
In Phase Two, the research team used these findings to develop recovery
oriented performance indicators. Two sets emerged, 73 consumer self-report
data items and 30 administrative data items. In partnership with the participating
states, the team refined the self-report set based on consumer review
(a Think Aloud process) and a readability check and then conducted a prototype
indicator test involving a diverse cross-section of 219 consumer/survivors
in seven states. The research team then used the prototype self-report
data results to evaluate each item as to: (a) importance rating, (b) factor
loading values within a varimax rotated component matrix, (c) response
scale distribution and direction, (d) Phase One originating theme, (e)
items assessing similar content, (e) clarity of wording, and (f) Phase
One member check priorities. Selected demographic variables (e.g., housing
status; parent status, etc.) were also cross-tabbed with selected item
importance mean ratings to determine whether significant differences exist
and therefore if an item should be retained or specified for a particular
subpopulation.
The research team generated specific measure definitions (i.e. numerators
and denominators) for the administrative data items. The 10 participating
SMHAs and the National Association of Consumer/Survivor Mental Health
Administrators (NAC/SMHA) were then surveyed on the administrative data
items as to (a) the feasibility of implementing each, (b) the importance
of each for improving system recovery orientation, (c) whether or not
the data articulated in the definition was currently being collected,
and (d) specific comments on each. Nine SMHAs and three NAC/SMHA members
responded. Through a series of teleconferences the research team evaluated
each measure as to importance rating, feasibility rating and comments.
These analyses led to further refinement with a concentrated effort towards
parsimony, resulting in 42 self-report items being crafted into an adult
consumer self-report survey, and 23 administrative-data items into an
authority/provider profile for the Recovery Oriented System Indicators
(ROSI) measure. A factor analysis of the 42 self-report items resulted
in domains of Person-Center Decision-Making & Choice, Invalidated
Personhood, Self-Care & Wellness, Basic Life Resources, Meaningful
Activities & Roles, Peer Advocacy, Staff Treatment Knowledge, and
Access. The 23 administrative-data items also include the domains of Peer
Support, Staffing Ratios, Consumer Inclusion in Governance, and Coercion.
The ROSI bridges the gap between the principles of recovery and self-help—choice,
hope, purpose, relationships, self-determination, empowerment, citizenship,
resources, opportunities—and the real-world application of these
principles in the everyday work of staff and service systems. The ROSI
is targeted for large-scale pilot testing. A subset of these items are
being incorporated into the Mental Health Statistics Improvement Program
Quality Report Version 2.0, the Decision Support 2000+ and other national
data collection requirements in order to generate comparable data across
state and local mental health systems.
The Phase One Technical Report, a.k.a. Mental Health Recovery: What Helps
and What Hinders? A National Research Project for the Development of Recovery
Facilitating System Performance Indicators, is available online at www.nasmhpd.org/publications.cfm#techreports
The Phase Two Technical Report is being written and NTAC hopes to release
this by the end of the calendar year. If you are interested in serving
as a site for the pilot test, please contact Steven Onken at so280@columbia.edu
Back to Top
Download FREE
Adobe Acrobat® Reader™ to view PDF files located on this
site. An accessible,
text-only Adobe Reader download page is available if you are using
a screen reader or need an earlier version of Adobe Reader. |
e-Report Table of Contents
Printer-Friendly Version of This Article (pdf,
Word)
Printer-Friendly Version of All Articles in One Document (pdf,
Word)
|
