Development of a Program for Evaluating Treatment Outcome
In A Community Mental Health Center

Jeffrey Braunstein, Ph.D.
ResearchConsultation.com

Purpose

1.  Measurement of treatment outcome has become increasingly important in the mental health care industry.  With this increased focus, a system for accurately measuring outcome needs to be designed and implemented.

2.  Clinical staff and consumers have expressed concern about administration difficulties and consumer friendliness of the existing outcome measure, the Basis-32.  Problems with readability and lengthy administration time are among some of the concerns reported.  In addition, clinicians reported needing to interpret questions to consumers who have difficulty understanding ambiguous test items.

3.  To determine if the Basis-32 adequately assesses psychopathology. Convergent / construct validity analysis comparing the Basis-32’s overall score of psychopathology with clinician ratings (GAF and LOI) will be conducted to help assess whether the Basis-32 accurately measures psychopathology. Assessing convergent / construct validity will help us determine if the Basis-32 can be used for assessing outcome.

4.  To compare the Basis-32 and clinician ratings with a new measure of psychopathology, the Personality Assessment Screener (PAS), to further investigate consumer friendliness and our ability to adequately measure psychopathology.  Our ability to accurately measure psychopathology directly impacts our ability to effectively measure treatment outcome.

5.  Once we determine which instrument and ratings most accurately measure pathology, we can improve the accuracy of assessing the severity of illness during the intake evaluation.  Improving the accuracy of assessing severity of illness at intake will also improve our ability to effectively measure outcome.

6.  Assessing mean differences (improvement) on psychological instruments and clinician ratings will help to measure treatment outcome.  In addition, we can measure the treatment effectiveness of individual clinicians and groups of clinicians by comparing treatment improvement with the amount of treatment consumers receive.

Overview of Program

Study 1: Convergent/Construct validity between clinician ratings (GAF & LOI) and the Basis-32 total score at intake.  Post-hoc pre-post outcome measure design: Basis-32 at intake and current.

Study 2: Convergent validity between clinician ratings (GAF & LOI) and psychological measures (Basis-32 & PAS) at intake.  Post-hoc pre-post outcome measure design: Basis-32 and PAS at intake and current.

Study 3: The goal of this study is to improve the accuracy of assessing the severity of illness during the intake evaluation.  Comparing the variate of clinician ratings (V_X-1) with the variate of self-report instruments (V_X-1) for both individual intake clinicians and intake clinicians as an entire group can help to improve patient treatment placement.  Analyzing and interpreting the Redundancy Coefficients (RC²) will help determine the accuracy of clinician ratings when compared to patients’ scores on self-report instruments.  In addition, the results of clinician ratings and self-report instruments at intake provide the baseline for measuring treatment outcome. 

Recommended Statistic: Canonical Correlation

Study 4: Assessing Treatment Outcome.  We can measure treatment outcome for individual patients, specific clinicians and groups of clinicians. Recommended Statistic: Repeated Measure Design ANOVA.

Example: Clinician X

Study 5: Estimating Treatment Effectiveness.  We can construct an “Effectiveness Score” for individual clinicians and groups of clinicians.  We first add the PAS Mean Difference (I-J) to the number of treatment sessions* (constant).  We then divide this score by the number of treatment sessions to derive an Effectiveness Score (ES). Clinicians should be compared with other clinicians who provide treatment to similar patient populations to control for severity of psychopathology (e.g. SMI with other SMI).  

Example: Clinician X

Example: Clinician Y

*Weighted Values for Number of Treatment Sessions (Indiv/Group/Family)

             30 minutes = 1.00 session

          < 30 minutes = 0.50 session

Summary of Results: Study 1

·        Review of demographic data (N = 118).

·        Mean LOI (1.38) is close to the 1B level (1.50). Mean of GAF = 55.53.  Mean of the Basis-32 at intake is 1.57 (between “A Little Difficulty” and “Moderate Difficulty”) on a 0-4 likert scale.

·        Clinician ratings (GAF and LOI) have a statistically significant (p < .01) strong negative correlation (-.615) with each other accounting for 38% of the variance.

·        GAF rating has a statistically significant (p < .01) moderate negative correlation (-.458) with the Basis-32 accounting for 21% of the variance.

·        LOI rating has a statistically significant (p < .01) low positive correlation (.297) with the Basis-32 accounting for 9% of the variance.

·        Post-hoc analysis (N = 41): Statistically significant mean difference

(p < .01) between pre (intake) and post (current) Basis-32 time intervals (1.64 to 1.29) suggest symptom reduction during the course of treatment.

Discussion: Study 1

·        Is the Basis-32 measuring psychopathology adequately?

·        Are our clinicians measuring psychopathology adequately?

·        Are our clinician ratings accurate determinants of psychopathology?

·        Maybe our consumers are under or over-reporting psychopathology due to the nature of illness, lack of insight or secondary gain issues.

·        We need to add another psychological measure to our design to further assess our accuracy for measuring psychopathology, consumer friendliness of the instrument and treatment outcome.

COMPARISON OF THE BASIS-32 AND PERSONALITY ASSESSMENT SCREENER (PAS)

Summary of Results: Study 2

·        Review of demographic data (N = 58).

·        Mean LOI (1.52) is at the 1B level. Mean of GAF = 58.88. 

·        Mean of the Basis-32 at intake is 1.71 (between “A Little Difficulty” and “Moderate Difficulty”) on a 0-4 likert scale.

·        Mean of the PAS Total Score at intake is 74.03 (75.00P to 99.81P is Marked Impairment range).

·        Basis-32 and PAS have a statistically significant (p < .01) strong positive correlation with each other (.559) accounting for 31% of the variance.

·        Clinician ratings (GAF and LOI) have a statistically significant (p < .01) moderate negative correlation with each other (-.447) accounting for 20% of the variance.

·        GAF rating has a statistically significant (p < .01) but low negative correlation (-.310) with the PAS accounting for 10% of the variance and low statistically significant (p < .05) negative correlation (-.260) with the Basis-32 accounting for 7% of the variance.

·        LOI rating had a statistically significant (p < .05) but low positive correlation (.252) with the Basis-32 accounting for 6% of the variance and low statistically significant (p < .05) positive correlation (.236) with the PAS accounting for 6% of the variance.

Discussion: Study 2

·        The PAS is reporting a moderate to marked level of psychopathology.  In contrast, the GAF, LOI and Basis-32 report low to moderate levels of disturbance.  A follow-up study is needed comparing the PAS and Basis-32 at pre and post time periods to assess disparity in measuring outcome.  Reports of greater disturbance at intake (pre) would allow for the possibility of measuring greater outcome at follow-up (post).

·        There is a significant difference between clinician ratings and psychological self-report ratings.  Why is this so?  Conducting an intake accuracy study (Study 3 in proposal) may provide information to assess the reason for these differences.  In addition, Study 2 is limited due to sample size.  A greater sample size may yield more definitive results.

·        The PAS appears to be easier to administer and score (less clinician involvement, results immediately).  Extensive outpatient normative and validation studies have been conducted.  In addition, the PAS seems to be more consumer friendly (e.g. reading level, length of test).

·        Once the aforementioned issues are investigated, we can begin to accurately assess treatment outcome and eventually measure treatment effectiveness.

Recommendations for Improving Research Procedures

In A Community Mental Health Center

1.     Employing a research assistant or volunteer would have facilitated data collection and entry.  A doctoral student could be recruited to assist in this area with an arrangement to use a portion of the data for the student’s dissertation or doctoral project.

2.     Designing a research protocol for collecting data efficiently needs to be implemented system-wide to facilitate data collection.  If a specific research protocol is integrated into the existing clinical protocol, research and continued assessment will be a natural process of treatment.   

3.     Providing in-service training to clinical and support staff may increase compliance with a research protocol.  The in-services should emphasize the importance of continued assessment for improving treatment and the importance of research for acquiring contracts and grants.