The Laura P. and Leland K. Whittier Virtual PICU

What is a Common Information Space?

Fifteen years ago, the common information space seemed a strange concept. The space, as imagined then, represented a combination of communication technologies that would bring pediatric critical care together as one Virtual ICU. Since then, this virtual space has been used to care for children geographically distant from academic ICUs, educate caregivers, conduct research and support quality improvement.

Today, the common information space is readily understood as a space where knowledge is shared and is bounded only by the internet and defined by myriad communication technologies developed by the VPICU.

Nowhere is rapid decision making more urgent, or more complex, than in managing critically ill children. Important to that decision making is the ability to identify similar patients. Patient similarity has long been a challenging thorn to many researchers because of its subjective nature.

The Laura P. and Leland K. Whittier Virtual Pediatric Intensive Care Unit Data Science team works closely with physicians at Children’s Hospital Los Angeles to develop a framework to address this challenge.

The clinical status of an individual patient can be defined by multiple contexts. To understand a patient’s clinical status, the VPICU Patient Similarity Framework combines these contexts as needed by corresponding modules. A module generates a representation for each individual patient that enables the computation of a mathematical distance between any two patients within a given clinical context (e.g. mortality, diagnosis, volatility, etc). The summation of the distances from various contexts of interest enables the retrieval of similar patients.

Dynamically Evolving Severity of Illness Score

Severity of illness (SOI) scores have been developed since the early 1980s to aid clinicians assess their critically ill patients. Examples of clinically-used systems are the Pediatric Index of Mortality (PIM) and Pediatric Risk of Mortality (PRISM). The VPICU developed a Recurrent Neural Network (RNN) that continuously generates individual SOI scores by predicting an individual child’s risk of ICU mortality. By incorporating many more variables than traditional models and integrating measurements of those variables in a dynamic manner, the RNN model demonstrated significantly higher discrimination, measured by the Area Under the Receiver Operating Characteristic (AUROC), than other models. The VPICU’s work demonstrates the RNN’s potential to provide accurate, continuous, and real-time assessment of a child’s condition in the ICU.

Area Under the Receiver Operating Characteristic (AUROC) curve of RNN mortality model predictions at different times after ICU admission. Shown for reference are the AUROCs of PRISM III (12th hour version), PIM2, and PELOD Day 1 scores in the same cohort.

Predicting Cardiac Death Within 1 Hour After Terminal Extubation

In US pediatric ICUs, limitation or withdrawal of life sustaining treatment is the most common mode of death. Intensivists are often called upon to counsel families about the dying process and to estimate time to death after treatment withdrawal. Some patients that have planned treatment withdrawal may be eligible for organ donations after cardiac death (DCD). DCD presents significant challenges; these include identifying patients who are anticipated to die within a set timeframe after treatment withdrawal, managing family expectations in case the donor does not die within the timeframe for donation, and utilizing operating rooms and organ procurement teams.

Using historical data from CHLA’s PICU, the VPICU team developed a LSTM-based RNN and a Cox Proportional Hazard (CPH) model to predict death within an hour of terminal extubation. The team also implemented the Dallas Predictor Tool (DPT) developed by Children’s Medical Center Dallas for comparison. The RNN and CPH models showed higher discrimination (as measured by AUROC) than the DPT model, both in the overall cohort and in the subcohort of DCD candidates. At any fixed sensitivity, the VPICU’s RNN and CPH models also achieved a lower number of needed to alert than the DPT model, showing the potential for the VPICU’s models to help identify potential candidates for DCD with minimal institutional waste in terms of preparing the operating room. The models developed by VPICU were able to identify 93% of potential organ donors with a number needed to alert of 1.08, meaning that 13 of 14 prepared operating rooms would have yielded a viable organ. The VPICU is now collaborating with several institutions to improve and validate the CPH model.

Interpreting a Recurrent Neural Network Model for ICU Mortality

Deep learning algorithms have demonstrated success in a wide range of applications; however the lack of transparency into how they generate predictions has impeded their acceptance in healthcare, where decisions may be lifesaving. The ability to determine how input features contribute to a model’s predictions is important for several reasons. First, it may provide useful information for clinical intervention or new avenues for investigation. Second, it facilitates an environment in which users can interact with the model and learn its strengths and weaknesses. Third, the information can be used to improve the model itself.

The VPICU developed Learned Binary Masks (LBM) to identify which input features were used by a Recurrent Neural Network (RNN) model to predict an individual child’s risk of mortality (ROM). In essence, the LBM finds which observations must remain unchanged between consecutive measurements to drive the ROM predictions to zero. The team also modified KernelSHAP to provide additional insights into the model predictions. In either method, feature contributions evolve over time within one individual; they also vary across different individuals. The VPICU aggregated and analyzed feature contributions in different ways to facilitate different levels of analysis of the RNN model and its predictions: (1) over a volatile time period within an individual patient’s predictions; (2) over populations of ICU patients sharing specific diagnoses; (3) across the general population of critically ill children.

The figure on the right illustrates the evolving ROM predictions of the RNN model for an individual diagnosed with pneumonia and acute respiratory distress syndrome (top panel), feature contributions to the evolving predictions (middle 2 panels, where the horizontal axis represents time), and the top contributing features to the RNN’s predictions during the patient’s volatile period (bottom panel). Despite the RNN model not using diagnoses as inputs, features related to the respiratory system (EtCO2, Pulse Oximetry, Respiratory Rate), or associated with infections (Temperature, Heart Rate, Systolic/Diastolic/Mean Arterial Blood Pressure, and blood gas measurements (ABG pH, ABG PO2, ABG PCO2, ABG HCO3, ABG TCO2) were identified.

Analyzing the Impact of Extraneous Features on Recurrent Neural Networks

Electronic Medical Records (EMR) are a rich source of patient information that can be used to generate predictions on a wide array of clinical tasks. Identifying which features are relevant in predicting different clinical outcomes can be challenging. The VPICU conducted a series of experiments, wherein an RNN’s predictive performance on various clinical modeling tasks was measured with and without the presence of extraneous features in the inputs. The extraneous features were randomly drawn from theoretical and empirical distributions.

The figure below displays that inclusion of extraneous features as inputs to the RNN had negligible effect on its ability to predict both mortality and ICU-free days. The results support the notion that RNNs can be trained without meticulous efforts spent on feature selection.

Background

The adoption of electronic medical record (EMR) systems has enabled researchers to apply a wide range of emergent machine and deep learning methodologies to the collected data with the goal of discovering and applying knowledge for better diagnosis, prognosis and patient treatment. At the recent AIMED conference in Laguna Nigel CA it was abundantly clear, as it has been to many investigators in health care big data research for years, that one of the major impediments to achieving the ‘big data’ promise of improved care for children, is the lack of sufficiently large datasets. This is particularly true in pediatric critical care. Although we capture and record large amounts of clinical data it is not widely available for researchers. This problem stems from: the lack of suitably large amounts of high quality data; and from the siloed nature of datasets used by various groups and studies, which make it difficult to make “apples-to-apples” comparisons. Furthermore the nature of pediatric critical care, wherein one unit may see a very limited number of certain diagnoses, makes this collaboration even more essential.

• Members contribute and share critical care EMR data
• Data conform to a common schema to facilitate algorithm development, benchmarks and valid standard datasets for development
• Members gather to share, discuss, and prioritize data aggregation, algorithm development, and clinical deployment

Overview

The PICU Data Collaborative will be comprised of institutions who contribute anonymized pediatric critical care EMR data to a shared data platform which resides in a private cloud-computing environment.

Collaborative Technical Overview

Each Collaborative member will be responsible for collecting and aggregating data from its various sources based upon an agreed data schema and element list. The anonymization algorithm will be shared across members to ensure compliance and usability. Standard EMR data (including demographics, diagnoses, bedside and laboratory measurements, medications and procedures for each patient) would be aggregated during the initial phase with the subsequent potential to add additional data including notes, waveform data, imaging data and even genomic data at a later stage as the Collaborative matures. To facilitate new research and development, the data platform will enable automated or semi-automated methods converting the disparate data sets into a format that is easily digestible by data scientists. The platform will also provide artificial intelligence workflows and access to data science workspaces, empowering members to develop the science instead of wrangling with the technicalities of the data.

Common Schema and Benchmarks

Benchmark data sets historically have led to rapid gains in their associated fields. The most noteworthy example of this is the ImageNet database which is freely available and easily accessible to all researchers. Since 2010, annual challenges using this standardized database have led to rapid advances in computer vision capabilities. Well-curated medical datasets that are openly available for benchmarks are scarce, and this dearth makes the tracking of real progress in EMR algorithm development nearly impossible. The MIMIC-III database, which includes vital measurements, laboratory results, notes, fluid balance, procedure codes, diagnoses, imaging reports, among other data, is the most comprehensive and only freely accessible dataset of its kind. It is primarily an adult (16 years or above) critical care database with more than 38,000 patients, but it also contains almost 8,000 neonates. This database has enabled significant progress, but its single-institution nature places uncertainty on the transferability of algorithm developments. Discoveries made from the MIMIC database are rarely validated using databases from other institutions.

Our proposed rich collection of curated and standardized data can be used for benchmarking algorithm development geared towards improving pediatric critical care. The Collaborative could define tasks or problems akin to the ImageNet challenges. Regardless of the task or challenge, a common test set would be set aside to compare algorithms and track progress. Access to the data set will be controlled by the collaborative.

Symposia for Collaborative Members

An important activity of the Collaborative would be an annual symposium, where researchers and clinicians gather to: