Development of a Program for Evaluating
Treatment Outcome
In A Community Mental Health Center
Jeffrey Braunstein, Ph.D.
ResearchConsultation.com
Purpose
1. Measurement
of treatment outcome has become increasingly important in the mental health care
industry. With this increased focus, a system for accurately measuring outcome
needs to be designed and implemented.
2. Clinical
staff and consumers have expressed concern about administration difficulties and
consumer friendliness of the existing outcome measure, the Basis-32. Problems
with readability and lengthy administration time are among some of the concerns
reported. In addition, clinicians reported needing to interpret questions to
consumers who have difficulty understanding ambiguous test items.
3. To
determine if the Basis-32 adequately assesses psychopathology. Convergent /
construct validity analysis comparing the Basis-32’s overall score of
psychopathology with clinician ratings (GAF and LOI) will be conducted to help
assess whether the Basis-32 accurately measures psychopathology. Assessing
convergent / construct validity will help us determine if the Basis-32 can be
used for assessing outcome.
4. To
compare the Basis-32 and clinician ratings with a new measure of
psychopathology, the Personality Assessment Screener (PAS), to further
investigate consumer friendliness and our ability to adequately measure
psychopathology. Our ability to accurately measure psychopathology directly
impacts our ability to effectively measure treatment outcome.
5. Once
we determine which instrument and ratings most accurately measure pathology, we
can improve the accuracy of assessing the severity of illness during the intake
evaluation. Improving the accuracy of assessing severity of illness at intake
will also improve our ability to effectively measure outcome.
6. Assessing
mean differences (improvement) on psychological instruments and clinician
ratings will help to measure treatment outcome. In addition, we can measure the
treatment effectiveness of individual clinicians and groups of clinicians by
comparing treatment improvement with the amount of treatment consumers receive.
Overview of Program
Study 1: Convergent/Construct
validity between clinician ratings (GAF & LOI) and the Basis-32 total score at
intake. Post-hoc pre-post outcome measure design: Basis-32 at intake and
current.
Study 2: Convergent validity
between clinician ratings (GAF & LOI) and psychological measures (Basis-32 &
PAS) at intake. Post-hoc pre-post outcome measure design: Basis-32 and PAS at
intake and current.
Study 3: The goal of this study is to improve the
accuracy of assessing the severity of illness during the intake evaluation.
Comparing the variate of clinician ratings (VX-1) with the variate of
self-report instruments (VX-1) for both individual intake clinicians
and intake clinicians as an entire group can help to improve patient treatment
placement. Analyzing and interpreting the Redundancy Coefficients (RC2)
will help determine the accuracy of clinician ratings when compared to patients’
scores on self-report instruments. In addition, the results of clinician
ratings and self-report instruments at intake provide the baseline for measuring
treatment outcome.
Recommended Statistic: Canonical Correlation
Study 4: Assessing Treatment
Outcome. We can measure treatment outcome for individual patients, specific
clinicians and groups of clinicians. Recommended Statistic: Repeated Measure
Design ANOVA.
Example: Clinician X
|
Mean |
Std. Deviation |
N |
PAS Intake |
89.74 |
13.02 |
95 |
PAS Current |
52.87 |
10.65 |
95 |
(I) Outcome 1 (J)
Outcome 2 |
Mean Difference (I-J) |
Standard Error |
Sig. |
1 2 |
36.87** |
6.83 |
.001** |
2 1 |
-36.87** |
6.83 |
.001** |
Study 5: Estimating Treatment
Effectiveness. We can construct an “Effectiveness Score” for individual
clinicians and groups of clinicians. We first add the PAS Mean Difference (I-J)
to the number of treatment sessions* (constant). We then divide this score by
the number of treatment sessions to derive an Effectiveness Score (ES).
Clinicians should be compared with other clinicians who provide treatment to
similar patient populations to control for severity of psychopathology (e.g. SMI
with other SMI).
Example: Clinician X
PAS Mean Difference
(I-J) |
Avg. # Sessions |
PAS (I-J) + # of
Sessions |
Divided by Avg. # of
Sessions |
Effectiveness Score
(ES) |
36.87 |
12 |
48.87 |
12 |
4.07 |
Example: Clinician Y
PAS Mean Difference
(I-J) |
Avg. # Sessions |
PAS (I-J) + # of
Sessions |
Divided by Avg. # of
Sessions |
Effectiveness Score
(ES) |
36.87 |
6 |
42.87 |
6 |
7.15 |
*Weighted Values for Number of
Treatment Sessions (Indiv/Group/Family)
30 minutes = 1.00
session
< 30 minutes = 0.50 session
Summary of Results: Study 1
·
Review of demographic data (N = 118).
·
Mean LOI (1.38) is close to the 1B level (1.50). Mean
of GAF = 55.53. Mean of the Basis-32 at intake is 1.57 (between “A Little
Difficulty” and “Moderate Difficulty”) on a 0-4 likert scale.
·
Clinician ratings (GAF and LOI) have a statistically
significant (p < .01) strong negative correlation (-.615) with each other
accounting for 38% of the variance.
·
GAF rating has a statistically significant (p
< .01) moderate negative correlation (-.458) with the Basis-32 accounting for
21% of the variance.
·
LOI rating has a statistically significant (p
< .01) low positive correlation (.297) with the Basis-32 accounting for 9% of
the variance.
·
Post-hoc analysis (N = 41): Statistically significant
mean difference
(p <
.01) between pre (intake) and post (current) Basis-32 time intervals (1.64 to
1.29) suggest symptom reduction during the course of treatment.
Discussion: Study 1
·
Is the Basis-32 measuring psychopathology adequately?
·
Are our clinicians measuring psychopathology
adequately?
·
Are our clinician ratings accurate determinants of
psychopathology?
·
Maybe our consumers are under or over-reporting
psychopathology due to the nature of illness, lack of insight or secondary gain
issues.
·
We need to add another psychological measure to our
design to further assess our accuracy for measuring psychopathology, consumer
friendliness of the instrument and treatment outcome.
COMPARISON OF THE BASIS-32 AND PERSONALITY
ASSESSMENT SCREENER (PAS)
Dimension |
Basis 32 |
PAS |
Number of Items |
32 |
22 |
Administration Time |
10-20 minutes |
5 minutes or less |
Scoring System |
Scantron |
< than 5 min. by hand |
Total score |
Yes |
Yes |
Total score prediction
with a parent measure |
No |
Yes (P Score:
probability estimate of problematic full PAI) |
Number of subscales |
5 |
10 |
Reading Level (Grade) |
6.90 |
4.40 |
Number of Words |
421 |
178 |
Age of administration |
14 and up |
18 and up |
General Normative
Information |
Originally designed
for inpatient and hospital populations. |
Census-matched norming
with community, student & clinical populations. |
Outpatient Normative
Information |
No outpatient norms
existed until 1999 study with 407 patients (Eisen et. al 1999). |
Census-matched norms
outpatient, inpatient, substance abuse & forensic populations. |
Informal Feedback by
Clinicians and Consumers |
Clinicians report
needing to interpret questions to consumers who have difficulty
understanding items. |
Consumers report PAS
is easier to understand. Less clinician assistance needed. |
Summary of Results: Study 2
·
Review of demographic data (N = 58).
·
Mean LOI (1.52) is at the 1B level. Mean of GAF =
58.88.
·
Mean of the Basis-32 at intake is 1.71 (between “A
Little Difficulty” and “Moderate Difficulty”) on a 0-4 likert scale.
·
Mean of the PAS Total Score at intake is 74.03
(75.00P to 99.81P is Marked Impairment range).
·
Basis-32 and PAS have a statistically significant (p
< .01) strong positive correlation with each other (.559) accounting for 31% of
the variance.
·
Clinician ratings (GAF and LOI) have a statistically
significant (p < .01) moderate negative correlation with each other
(-.447) accounting for 20% of the variance.
·
GAF rating has a statistically significant (p
< .01) but low negative correlation (-.310) with the PAS accounting for 10% of
the variance and low statistically significant (p < .05) negative
correlation (-.260) with the Basis-32 accounting for 7% of the variance.
·
LOI rating had a statistically significant (p
< .05) but low positive correlation (.252) with the Basis-32 accounting for 6%
of the variance and low statistically significant (p < .05) positive
correlation (.236) with the PAS accounting for 6% of the variance.
Discussion: Study 2
·
The PAS is reporting a moderate to marked level of
psychopathology. In contrast, the GAF, LOI and Basis-32 report low to moderate
levels of disturbance. A follow-up study is needed comparing the PAS and
Basis-32 at pre and post time periods to assess disparity in measuring outcome.
Reports of greater disturbance at intake (pre) would allow for the possibility
of measuring greater outcome at follow-up (post).
·
There is a significant difference between clinician
ratings and psychological self-report ratings. Why is this so? Conducting an
intake accuracy study (Study 3 in proposal) may provide information to assess
the reason for these differences. In addition, Study 2 is limited due to sample
size. A greater sample size may yield more definitive results.
·
The PAS appears to be easier to administer and score
(less clinician involvement, results immediately). Extensive outpatient
normative and validation studies have been conducted. In addition, the PAS
seems to be more consumer friendly (e.g. reading level, length of test).
·
Once the aforementioned issues are investigated, we
can begin to accurately assess treatment outcome and eventually measure
treatment effectiveness.
Recommendations for Improving Research
Procedures
In A Community Mental Health Center
1.
Employing a research assistant or volunteer would have facilitated data
collection and entry. A doctoral student could be recruited to assist in this
area with an arrangement to use a portion of the data for the student’s
dissertation or doctoral project.
2.
Designing a research protocol for collecting data efficiently needs to be
implemented system-wide to facilitate data collection. If a specific research
protocol is integrated into the existing clinical protocol, research and
continued assessment will be a natural process of treatment.
3.
Providing in-service training to clinical and support staff may increase
compliance with a research protocol. The in-services should emphasize the
importance of continued assessment for improving treatment and the importance of
research for acquiring contracts and grants.
|