Evaluation of systems-change efforts is critical to sustaining the energy and
attention required to have a long-term impact on services for persons with
co-occurring mental illness and
substance abuse/chemical dependency. This issue of The Change Agent’s TOOL BOX focuses on how to create efficient mechanisms for defining outcomes and evaluating progress toward shared goals.
In the realm of systems integration, or any systems change process for that matter, it is important to be able to
measure whether change is occurring. Additionally, it is important to determine whether changes being made in the
system structures and service
configuration benefit consumers.
Ideally, systems integration should result in increased access to services for
persons with multiple problems and
improved treatment outcomes at the
individual client level. At the same time, systems integration activities should not disrupt or diminish existing services for persons who are not diagnosed with
co-occurring disorders.
Without evaluating the core outcomes
related to increased access and
improved treatment results, as well as other outcomes of particular interest to local stakeholders, there is no legitimate method for justifying the time,
energy and resources that are dedicated to the systems-integration process. People with multiple problems need
to get better as a result of integration
efforts. It matters little if an
organizational chart appears stream-lined or bureaucratic barriers are
removed if clients do not benefit from these changes.
Furthermore, reductions in funding that are marketed as integration efforts must be questioned if the efficiencies gained at an administrative level result in
negative consequences for line staff or clients, whether these consequences are increased caseload size, less individualized care or more frequent episodes of hospitalization or incarceration.
Effective outcome evaluations use simple language to describe how well a system is (or is not) doing in meeting its integration goals and objectives.
Evaluation provides a mechanism for reporting to consumers, providers, funders and other stakeholders about specific successes and failures.
Outcome-based program evaluation can offer useful information about whether initial efforts have met the desired
goals or could benefit from specific
adjustments. Outcome reports can help to establish a foundation for
recommending that pilot projects be
supported on an ongoing basis.
Unless data are available to
substantiate the claim that integrated
services are more effective services, few policymakers or funders will
support institutionalizing changes that may have been initiated on a temporary or experimental basis.
Evaluation Can Make
People Nervous
Planners, administrators, line staff and consumers may react cautiously to evaluation activities. There may be a variety of reasons for this reaction. Some may anticipate that evaluators will not
understand the complexity and
challenges of providing human services under difficult circumstances or in
less-than-ideal environments. Others may express the view that monies
earmarked for evaluation activities should instead be used for direct services or integration activities.
Past experiences with outcome data collection may have included evaluations that were used to justify funding cuts, program consolidation or service
reductions.
To win the cooperation of essential
participants in the program evaluation process, the evaluation should focus on learning about what is working, what is not working as well as it might, and the nature and extent of the impact that specific activities are having on individual clients and the system as a whole. Evaluation should be viewed as a tool to help improve the system and channel energies into productive avenues rather than as an opportunity to identify individual or
programmatic deficiencies that must be remedied through “corrective action plans” that undermine rather than
support creative energy and productive problem solving.
Evaluate the Right Things
at the Right Time
One of the great dangers inherent in evaluating system-integration initiatives is the expectation that change will occur quickly and that the impact of integration efforts at the direct-services level will be measurable overnight. Government
officials, policy-makers and planners are often anxious to demonstrate concrete results to voters, funders and other key
constituents. This concern can result
in pressure on change agents and
providers to produce results with
unreasonable speed. Many valuable
integration activities have been terminated prematurely because too much was expected too soon in the change
process.
The remedy for this pitfall in systems-change efforts is not to delay evaluation. Instead, change agents and other stakeholders need to develop evaluation
measures that evolve over time and
measure the progressive changes at many levels of a system that are required for integration concepts to take hold and
produce long-term benefits for clients.
This continuum of evaluation efforts should begin with “process outcomes” that measure what the various systems involved in the change effort are
actually doing to promote integration. Examples of process outcome measures include the:
- extent to which the necessary
range of stakeholders has been
assembled,
- frequency with which stake-
holders meet to discuss and plan
integration activities,
- completion of strategic planning
efforts focused on service- and
system-level changes to existing programs and systems,
- development of consensus on best practice models for integrated
programming at the direct-service level,
- identification of resources among participating systems that can be targeted or leveraged for integration activities, and
- creation of integrated information systems which have the capacity to track and manage the flow of
clients and services among multiple systems.
None of the activities reflected in these process-related measures are easy to accomplish; yet successful systems-integration efforts depend on mobilizing and completing these tasks. It is, therefore, highly reasonable to collect data about how effectively these process-related tasks are completed and to give credit to the stakeholders involved in the endeavors. A reasonable evaluation timetable for the first several years of a systems-integration initiative would
focus primarily on these process-type measures.
If policy-makers, funders, consumers and other stakeholders understand the utility of an evaluation continuum as they enter into systems-change initiatives, they will be much more likely to accept that specific client outcome measures may not be possible until after key
process outcomes have been achieved. Only when the essential process tasks have been completed and measured can an integrated system begin to move
toward measurement of changes in the services clients receive and the extent to which individual clients stabilize and recover from both substance abuse and mental illness. Once again, identifying the right things to measure in a system is the key to developing outcomes that support the systems-integration process rather than enhance resistance to change.
Capture Objective and
Subjective Information
The range of outcome measures should reflect the full continuum of systems-
integration activities that have been
undertaken. This means that outcome measures that capture both the
objective changes in the system and the subjective changes that are perceived by key stakeholders are necessary and useful.
Objective measures include specific changes at the system, service and client levels such as:
- changes in the structural system management that promote
integration;
- creation of truly integrated
programming at the service level
for persons with co-occurring
disorders;
- blending of resources across
multiple systems to fund integrated
programming; and
- improved treatment outcomes for persons with co-occurring disorders, including fewer episodes of
incarceration and hospitalization and reduced use of more expensive and
restrictive settings.
Subjective measures include the perceptions of key stakeholders about integration at the service and systems levels resulting from the change process.
These may include:
- funders reporting that providers are interested in and responsive to
requests for proposals to mobilize
integrated services;
- providers perceiving that integrated care is easier to fund and mobilize;
- family members and advocates
reporting that treatment for persons with co-occurring disorders is more readily accessible; and
- individuals with co-occurring
disorders reporting satisfaction with the care they are receiving, including attainment of treatment goals that move them closer to recovery.
Develop a Systems-Integration Report Card
As outcomes are identified and appropriate measures are developed, it is
useful to aggregate and disseminate evaluation information to interested parties, including elected officials, policy-makers, administrators, service providers, consumers, family members, advocates and members of the public.
At the time the County Council of King County, Washington, approved the
integration of the mental health and
substance abuse treatment systems into a single authority, the Council called for the development and dissemination of an annual “report card” on progress toward systems-integration goals. By doing so, local elected officials were seeking to ensure accountability in the systems-
integration process and create a
mechanism for helping to determine whether all of the energy being poured into the integration initiative proved worthwhile in the long run.
The report card was designed by the county’s Systems Integration Advisory Council (SIAC), which is composed of stakeholders in all of the systems affected by the integration effort. The SIAC followed the procedures recommended above, calling for the collection of data on objective and subjective measures reflecting changes at the system, service and client levels. The SIAC made it clear that outcomes should be tracked as a chronological progression and that measurable change in different areas should be evaluated at different points in time. The specific categories and time frames for measurement of progress toward integration included the following:
Objective Measures:
System Change Outcomes
(12-18 months)
- Development of planning
structures
- Mobilization of cross-training
- Articulation of policies and
procedures
- Integrated systems management and administration
- Development of leadership
- Implementation of financial
incentives
- Advocacy for larger systems change (state and federal levels)
Service Change Outcomes
(18-24 months)
- Co-location of services
- Development of leadership
- Development of integrated
programs
- Delivery of integrated services
- Provision of culturally and
linguistically competent services
- Implementation of financial
incentives
Client Change Outcomes
(24-36 months)
- Identification of key clients
- Identification and definition of
key client outcome measures
Subjective Measures:
(18-36 months)
System Assessment
- Implementation of systems-integration assessment tool(s) to measure levels of integration achieved
Client Satisfaction
- Implementation of linguistically
and culturally appropriate client
satisfaction surveys targeting clients with co-occurring disorders
By suggesting specific time frames for these evaluation activities, the SIAC
emphasized that progress would be gradual and cumulative and that the
success of the integration initiative needed to be measured in years rather than months. Changes would probably be noted first at the systems level (e.g., administrative reorganization of the county treatment authority), then at the service level (e.g., mobilization of integrated programming supported by blended funding) and finally at the client level (e.g., improved outcomes for clients with co-occurring disorders).
The SIAC agreed to work with
the integrated mental health and
substance abuse treatment authority
to refine the measures over time,
based in part upon the availability of
data sets relevant to the various
outcomes identified. Rather than
creating an adversarial relationship
between the SIAC and county
managers, the report card became
an opportunity to explore and discuss which measures represented realistic
expectations and how the realization
of these outcomes could be paced evenly over time without placing an excessive burden for evaluation on a system that was already stretched extremely thin.
Ensure Adequate Funding
Effective evaluation activities cannot
be accomplished without adequate
resources. The collection and analysis of data requires time, infrastructure and specific staff expertise. Data elements must be defined; the data-collection methodology must be designed; and the hardware, software and technically
competent staff necessary for data analysis must be available. Program evaluation is not an activity that can be tacked on to existing job descriptions of program managers or planners. Evaluation staff should either be included in the core structure of the systems involved in the integration initiative or provided through contracts with independent entities that specialize in evaluation. Sometimes a combination of these two strategies offers the most useful approach. An in-house evaluation unit oversees the
activities of independent contractors whose objectivity is increased by being one step removed from the system
being studied.
Evaluation costs (which often are
calculated at up to 10 percent of the cost of the program being evaluated)
should be built into the budget for
the integrated services initiative.
Although some may express concern
that evaluation activities are
taking scarce resources away from
direct services, the savings that
evaluations generate over time by
helping to redirect programs away from less effective treatment interventions make them well worth the up-front
investment. Suggestions that evaluation activities be cut or drastically reduced at the first sign of budget problems should be resisted.
It may be helpful to identify and secure specialized funding for evaluation
activities. Some federal agencies and
private foundations and organizations that do not fund direct services can often be tapped as sources of funding for evaluation activities. Again, researching these funding sources and building them into the overall systems-change design is well worth the effort.
This material was prepared by David M. Wertheimer, M.S.W., M. Div., Principal Consultant, Kelly Point Partners, a Washington state-based human services consulting firm. Previously, Mr. Wertheimer served as Systems Integration Administrator for the King County (WA) Department of Community and Human Services. For additional information on this topic, contact him at
davidmwertheimer@aol.com or at
(206) 914-4475.
