Our Preliminary Results and Analysis

Overview. Our preliminary results and analyses include the results of our prospective, multicenter, CDR derivation and validation studies, and extensive secondary analyses designed to [1] further validate the CDR’s screening performance, [2] assess site-by-site disparities in current AHT screening and evaluation practices, [3] demonstrate the CDR’s application as an assistive prediction tool, [4] estimate the magnitude of the CDR’s potential clinical impacts and cost benefit, [5] identify factors that could impact the CDR’s application in PICU settings, and [6] estimate baseline CDR “acceptability” among potential physician users.

Our CDR derivation and validation studies. These two prospective, cross-sectional, strictly observational studies were conducted in accordance with established standards for CDR development at 18 sites in the U.S. and Canada.  Ten sites participated in both studies.  In both studies, doctors were kept blinded to the CDR’s final form and content.  The two studies used the same inclusion and exclusion criteria, data forms, methods, and a priori definitional criteria for AHT.  To minimize circular reasoning, these criteria contained no references to any of the clinical variables considered for inclusion in the CDR.  Eligible patients were children less than three years old who were hospitalized for intensive care of symptomatic, acute, closed, traumatic, cranial or intracranial injuries confirmed on initial head CT or MRI.  Patients were excluded if their head injuries resulted from collisions involving motor vehicles or if initial neuroimaging revealed clear evidence of pre-existing brain malformation, disease, infection, or hypoxia-ischemia.

 

The CDR derivation study was conducted between February 2010 and August 2011 at 14 PICU sites. The goal was to identify a cluster of reliable predictor variables that would perform effectively as an AHT screening tool.  To this end, physician-investigators [1] captured historical, clinical, and radiological data regarding 209 eligible patients; [2] identified 13 highly discriminating (p ≤ 0.001) and highly reliable (κ = 0.68-0.95) clinical variables readily available at or near the time of PICU admission; [3] calculated each variable’s isolated predictive qualities (sensitivity, specificity, predictive values, likelihood ratios, post-test probabilities); [4] applied a recursive partitioning algorithm designed to penalize missed cases of AHT; and [5] identified four clusters of these 13 variables that, alone or in combination, detected AHT with sensitivity ≥ 0.92.  These four clusters of predictor variables became our four candidate CDRs.

 

The CDR validation study was conducted between March 2012 and July 2013 at 14 PICU sites. The goal was to verify the candidate CDRs’ AHT screening performance in a new, equivalent, patient population.  Physician-investigators [1] captured equivalent historical, clinical, and radiological data for 291 additional patients; [2] confirmed the AHT screening performance of all four candidate CDRs in a new, equivalent, patient population; and [3] determined that a four-variable CDR demonstrated the optimal combination of simplicity and AHT screening performance.   All measures of this four-variable CDR’s screening performance matched or exceeded equivalent results from the completed derivation study. Most importantly, the CDR demonstrated sensitivity of 0.96 in both studies.

 

Secondary analyses. We also conducted multiple secondary analyses of the captured data regarding the 500 acutely head-injured PICU patients in our derivation and validation study datasets.

 

Our CDR has been broadly validated. Analysis of the CDR’s screening performance in novel patient cohorts revealed that it would have performed with high sensitivity (0.95 to 0.98) in different geographical regions, in large and small PICUs (0.94 to 0.99), and in the same PICUs over time (0.97 to 0.99).

 

Current AHT screening and evaluation practices demonstrate site-by-site disparities. The percentage of eligible patients who underwent both skeletal survey and retinal examination varied from 33% to 100% across the 18 participating sites.  The percentage of eligible patients who underwent at least one of these abuse evaluations varied from 43% to 100%, and the percentage who underwent neither abuse evaluation varied from 0% to 57%.

 

Non-clinical factors appear to impact doctors’ decisions to launch or forgo child abuse evaluations in their young, acutely head-injured patients. Exploratory analyses identified four non-clinical factors associated with significant differences (p < .05) in the percentages of high or low risk patients evaluated for child abuse.  These factors included [1] caregiver denial of any head trauma, [2] patients of minority race/ethnicity, [3] patients not yet cruising or walking prior to PICU admission, and [4] patient age <6 months at the time of PICU admission.

 

Applied as a directive decision tool, the CDR is highly sensitive—no matter how we define AHT. Gold standard definitional criteria for AHT do not exist.  As defined by our a priori criteria, the prevalence of AHT in our combined patient population was 0.44.  The CDR would have correctly identified (categorized as high risk) 96% of patients who met these criteria. Applying several modified (more inclusive and less inclusive) AHT criteria to iteratively re-sort the patient population, our resulting calculations of AHT prevalence ranged from 0.28 to 0.52.  Subsequent analyses revealed that the CDR would have correctly identified (categorized as high risk) at least 96% of AHT patients—no matter how we defined AHT.  Applied at the time of PICU admission, the CDR would also have correctly identified 98% of patients ultimately diagnosed with probable or definitive AHT, and 99% of patients whose completed skeletal surveys and/or retinal exams revealed corroborating findings of abuse.

 

Applied as an assistive prediction tool, the CDR’s patient-specific estimates of abuse probability are good predictors of the results of completed abuse evaluations. To calculate an evidence-based, patient-specific, estimate of abuse probability for every patient, we [1] divided the patient population (N = 500) into its 16 subpopulations defined by each patients’ specific combination of the CDR’s four predictor variables, [2] calculated the percentage of patients within each subpopulation who met our a priori criteria for AHT, and [3] assigned that same value—expressed as an estimate of abuse probability—to every patient within that same subpopulation. The resulting patient-specific estimates of abuse probability correlated positively and very strongly (R = 0.708) with the overall diagnostic yields of the patients’ completed abuse evaluations (the percentage of patients within each subpopulation whose completed skeletal survey and/or retinal exam revealed corroborating findings of abuse).  This correlation increased significantly (R = 0.946) when we eliminated the single patient subpopulation that contained only one patient.


Applied accurately and consistently as a directive decision tool, the CDR can improve AHT screening accuracy. To estimate the CDR’s potential impact on AHT screening accuracy in PICU settings, we applied mean (patient-specific) estimates of abuse probability to predict additional positive abuse evaluations (cases of missed AHT) among high risk and low risk patients lacking skeletal survey and/or retinal exam, and then used these predictions to extrapolate and compare “current PICU screening for AHT” vs. “AHT screening guided by the CDR.”  Our results suggest that, applied accurately and consistently as a directive AHT screening tool, the CDR could [1] increase AHT detection (from 87% to 96%), [2] increase the overall diagnostic yield of patients’ completed abuse evaluations (from 49% to 56%), [3] reduce (mean) evidence-based estimates of abuse probability (estimates of missed AHT) among patients not evaluated thoroughly for abuse (from 0.19 to 0.07), and [4] reduce unnecessary abuse evaluations of patients with non-AHT (from 67% to 60%).

 

Our CDR could decrease AHT-associated acute health care costs in PICU settings. Applying average skeletal survey, retinal exam, and visit costs obtained from the MarketScan Commercial Claims and Encounters database, we estimate that accurate and consistent CDR application as a directive AHT screening tool could reduce the average cost per correctly identified child with AHT from $458 to $389—a reduction of 15.1%.  Applying published data on re-injury rates in children with missed AHT and the costs of emergency department/clinic visits and hospital re-admissions resulting from re-injury, we estimate a 72.4% reduction in total AHT-associated acute health care costs stemming from the anticipated reduction in missed AHT cases.

 

Our preliminary assessment of CDR “acceptability” among physicians yielded favorable results. The Ottawa Acceptability of Decision Rules Instrument (OADRI) is a validated 12-item survey that evaluates decision rule acceptability among clinicians. The OADRI’s final score is calculated as the mean of all 12 items, resulting in a score that ranges from 0 to 6 with 6 being greatest acceptability.  To estimate the CDR’s baseline acceptability among potential users, we distributed a brief, voluntary, anonymous survey to pediatric intensivists and child abuse pediatricians who participate in the dedicated listservs of the PALISI research network and the Ray E. Helfer Society.  Survey participants were provided a one-page document that summarized the CDR’s derivation, validation, and potential clinical impact.  Participants were asked to answer the OADRI’s 12 questions “as if they were considering using the rule.”  A total of 30 pediatric intensivists, 51 child abuse pediatricians, and 2 other physicians completed the survey. The mean OADRI score for the total sample (N= 83) was 4.86 (SD =1.32).  A clear majority of participants endorsed items indicating that the CDR was “easy to use”, “easy to understand”, “useful”, “clear”, and “will benefit patients”.  Though not generalizable, the results suggest that many physicians will actively consider the CDR’s application as an AHT screening tool.