Understanding the Turnaround Time Effects of an AI-based Prioritization System for Intracranial Hemorrhage

A Simulation Study on the Benefits and Risks of Utilizing AI for the Prioritization of Intracranial Hemorrhage

Derek Zhi, Yue Pan, & Feng Gao

Intracranial hemorrhage (ICH) is a type of bleeding that happens inside the skull. Every year, 67,000 patients in the US suffer from ICH. Although ICH affects a small percentage of the population, it is a very serious condition—the 1-month mortality rate is 35% to 52%, with nearly half of the resulting mortality occurring in the first 24 hours of onset [1]. ICH condition requires immediate medical attention after diagnosis to mitigate the ensuing damage. Currently, a medical diagnosis of ICH is based on non-contrast head computed tomography (CT). A non-contrast head CT study only takes about seven minutes to interpret, but it may sit in the radiologist’s worklist for an extended period before being read due to the backlog of existing studies in the worklist. In the standard of care, studies appear on the worklist and are arranged based on “STAT” or “Routine” status in a first-in-first-out order, with “STAT” scans of higher priority than “Routine” scans. In an AI integrated workflow, non-contrast head CT studies will be analyzed by the software to identify ICH findings, which could be used in the worklist for notification or prioritization purposes. Radiologists can also configure the worklist application’s sorting rules to allow the worklist to automatically elevate the suspected ICH-positive patients in the queue. The illustration below depicts the standard of care worklist in comparison with an AI-assisted worklist prioritization.

We built a Monte Carlo simulation of a one-week workflow of a radiologist under two workload settings: “low” workload and “high” workload. The simulation was repeated for 2,000 realizations. The standard of care was used as the baseline for comparison. Simulation parameters were extracted from previous literature, which include utilizations of medical-image modalities, prevalence of ICH case among STAT and Routine respectively, prevalence of a critical finding in a medical image, distribution of radiologist interpretation time for a specific imaging modality, and distribution of medical-image arrival time interval [1,2,3,4]. We investigated the positional change and the change of turnaround time (TAT) in three groups: ICH true positive group, ICH false negative group, and other critical finding group. ICH true positive is defined as an ICH study successfully detected by the AI algorithms; ICH false negative is defined as an ICH study incorrectly predicted as ICH negative by the AI algorithms; the other critical finding is defined as a study that contains a critical finding other than ICH. Two endpoints were evaluated: the median positional change and median change in TAT. A positional change is defined as the change in queuing position of a study in the AI-integrated worklist compared to the standard of care. A TAT change is defined as either the reduction or addition of TAT in the AI-integrated worklist compared to the standard of care.

Observed simulation results show that the Routine ICH true positive studies were elevated to a noticeably earlier queue position, with a positional advance of approximately 200 in the low workload setting, and approximately 1,000 in the high workload setting. The TAT reduction relative to the standard of care worklist under different parameters settings ranged from 1,394 minutes (23.2 hours) to 3,144 minutes (52.4 hours) in the low workload setting and 4,790 minutes (79.8 hours) to 11,532 minutes (192.2 hours) in the high workload setting. Thus, the simulation demonstrates that the AI-integrated worklist is effective in the prioritization of Routine ICH true positive studies. Furthermore, when the percentage of STAT cases is high (60%), STAT ICH true positive studies showed a positional advance of 350 and TAT reduction of 1,700 minutes (28.3 hours).

While the benefits are demonstrated by the analysis above, the corresponding risks associated with the ICH false negative group and the other critical finding group must also be considered. Both groups experienced a slight delay in both low and high workload settings. However, the observed delays are minor in the context of the clear benefit illustrated above and represent a relatively low impact overall. As an example, the longest TAT delay (133 minutes) occurs in the Routine false negative studies. However, on average, only two incidents of this scenario occur during an entire 1-week simulation period.

In summary, a Monte Carlo simulation was performed to investigate the clinical benefits and risks of the AI integrated work list prioritization.  The results demonstrate that AI can help prioritize ICH true positive studies with only minimal delays in the rarely occurring ICH false negative studies, and the other studies containing critical findings.