R Yates Coley, Jennifer M Boggs, Arne Beck, Gregory E Simon
Highlights:
Abstract: Predictive analytics with electronic health record (EHR) data holds promise for improving outcomes of psychiatric care. This study evaluated models for predicting outcomes of psychotherapy for depression in a clinical practice setting.
EHR data from two large integrated health systems (Kaiser Permanente Colorado and Washington) included 5,554 new psychotherapy episodes with a baseline Patient Health Questionnaire (PHQ-9) score ≥ 10 and a follow-up PHQ-9 14–180 days after treatment initiation. Baseline predictors included demographics and diagnostic, medication, and encounter history. Prediction models for two outcomes — follow-up PHQ-9 score and treatment response (≥ 50% PHQ-9 reduction) — were trained in a random sample of 70% of episodes and validated in the remaining 30%. Two methods were used for modeling: generalized linear regression models with variable selection and random forests. Sensitivity analyses considered alternate predictor, outcome, and model specifications.
Predictions of follow-up PHQ-9 scores poorly estimated observed outcomes (mean squared error = 31 for linear regression, 40 for random forest). Predictions of treatment response had low discrimination (AUC = 0.57 for logistic regression, 0.61 for random forest), low classification accuracy, and poor calibration. Sensitivity analyses showed similar results. We note that prediction model performance may vary for settings with different care or EHR documentation practices.
In conclusion, prediction models did not accurately predict depression treatment outcomes despite using rich EHR data and advanced analytic techniques. Health systems should proceed cautiously when considering prediction models for psychiatric outcomes using baseline intake information. Transparent research should be conducted to evaluate performance of any model intended for clinical use.
Toward Gender Equity in Academic Promotions. In press.
Nationwide Use of Telehealth Among Commercially Insured Individuals 2007-2017
Considerations of Bias and Reliability in Publicly Reported Physician Ratings.
Clinical risk prediction models and informative cluster size: assessing the performance of a suicide risk prediction algorithm. In press.
How should we reconcile statistical and clinical predictions of suicide risk? In press.
Predicting outcomes of psychotherapy for depression with electronic health record data
Positive predictive values and potential success of suicide prediction models
Defining success in measurement-based care for depression: a comparison of common metrics.
Measuring outcome of depression: It is complicated
Assessing and minimizing re-identification risk in research data derived from healthcare records
Safer use of antipsychotics in youth (SUAY) pragmatic trial protocol
Contact: Susan Brandzel
CATALyST Program Administrator
Susan.X.Brandzel@kp.org
206-287-2380
Land Acknowledgment
Our Seattle offices sit on the occupied land of the Duwamish and by the shared waters of the Coast Salish people, who have been here thousands of years and remain. Learn about practicing land acknowledgment.