systematic review
citation
A methodological systematic review of validation and performance of sepsis
real-time prediction models. npj Digital Medicine, 2025.
urlhttps://www.nature.com/articles/s41746-025-01587-1
studies reviewed91
note58.2% derive from the PhysioNet/CinC Challenge — a shared, narrow base.
auroc
internal partial window
6h pre onset0.886
12h pre onset0.861
external partial window
6h pre onset0.860
12h pre onset0.860
noteOnly 18 performances reported — thin.
full window
internal0.811
IQR 0.760–0.842
external0.783
IQR 0.755–0.865
utility score
---- THE CENTRAL FINDING ------------------------------------------------- Utility Score accounts for the cost of false alarms and missed cases. It goes NEGATIVE on external validation: the model does net harm on patients it was not trained on. Meanwhile AUROC falls only 0.811 -> 0.783 and looks fine. This is the single most important number in this file. A build that reports AUROC alone would show a model that appears to work and does not.
internal0.381
IQR 0.313–0.409
external-0.164
IQR -0.216 to -0.090
significancep < 0.001
auroc utility correlation0.483
implication
AUROC and clinical usefulness are only loosely related. Report both,
and treat a negative utility as disqualifying regardless of AUROC.
methodological failures
"Only 54.9% applied full-window validation"
"87 of 91 studies used US datasets — nothing from Indian populations"
"78 of 91 confined to ICU populations"
"Only 2 studies performed prospective external validation"
"High AUROC frequently masks poor sensitivity, so cases are missed"
what helped
>
Hand-crafted features embedding clinical expertise ("Medical Index"
constructs) outperformed algorithmic feature learning. Clinical
knowledge beat model sophistication.
lead times
hrv wavelet
signalHeart rate variability, wavelet analysis
lead time hours35
measure25% drop from baseline, mean across patients
citation
Continuous Multi-Parameter Heart Rate Variability Analysis Heralds
Onset of Sepsis in Adults. PLOS One 2009.
urlhttps://pmc.ncbi.nlm.nih.gov/articles/PMC2721415/
sample size17
cases14
caution
VERY small sample — 17 patients, 14 cases. Individual lead times ranged
0 to 227 hours; the authors treated anything beyond 120h as likely false
alarm. Directionally interesting, not a basis for a threshold.
onset definition
SIRS plus clinically suspected infection requiring treatment. Antibiotic
administration used as the diagnosis marker — a TREATMENT PROXY, with
physician diagnosis preceding it by roughly 30–90 minutes.
hrv composite
signalComposite HRV measure
lead time hours60
measuremedian, before sepsis diagnosis
decline begins hours140
acceleration hours42
citationMonitoring and Identification of Sepsis Development through a Composite Measure of HRV. PLOS One 2012.
urlhttps://journals.plos.org/plosone/article?id=10.1371/journal.pone.0045666
hrv alone vs combined
hrv only auc0.61
hrv plus clinical auc0.83
implication
Beat-to-beat data alone is weak. Its value is as an ADDITION to clinical
variables, not a replacement. Any claim that waveform data alone
predicts sepsis is not supported.
deterioration trajectory
citationMachine learning predicts sepsis deterioration trajectories. npj Digital Medicine 2026.
urlhttps://www.nature.com/articles/s41746-026-02565-x
auroc development0.92
auroc internal0.89
auroc mimic iii0.84
auroc eicu0.77
median warning time hours17.6
sensitivity0.83
specificity0.88
note
Clean illustration of transfer decay0.92 -> 0.89 -> 0.84 -> 0.77 as
the data moves further from where the model was built. Expect the same
or worse moving to Indian data.
engine
nameSepsis Onset Engine
version0.2.0
method
---- The core method -----------------------------------------------------
Anchor on a confirmed case, then walk backwards. Sepsis onset is
established at the highest fidelity available and held fixed. The engine
then examines the window BEFORE onset and asks, at each lookback point,
what was already separable from patients who did not become septic.
algorithm
- step1
nameAnchor
detail
Establish onset per patient. Clinical criteria preferred; treatment
proxy accepted but flagged, because it encodes clinician behaviour.
- step2
nameBlank
detail
Discard all data at and after onset, plus a blanking window, so the
model cannot learn the treatment response instead of the warning signs.
- step3
nameSweep
detail
At each lookback point, measure separability per signal and combined.
Report where each signal first carries information.
- step4
nameCompare
detail
Two models per point — chartable variables only, versus everything a
monitor records. The gap is the machine's actual contribution. If the
gap is near zero, continuous monitoring adds nothing and must not be
claimed to.
- step5
nameReport both metrics
detail
AUROC and a utility measure. The systematic review shows utility can
go negative while AUROC looks acceptable. Negative utility is
disqualifying.
lookback points minutes[1, 5, 10, 15, 30, 60, 120, 180, 360, 720, 1440, 2160]
signal tiers
chartable
Appears on any observation chart. A nurse can see this. HR, BP, RR,
temperature, SpO2.
ratio
Arithmetic on chartable values — shock index, pulse pressure fraction.
Trivial per instance; nobody computes them continuously on every patient.
This is where deterministic calculation earns its place.
beat to beat
Requires continuous monitoring. HRV and its derivatives. Not humanly
observable at all. Level III only.
perfusion
Measurable but intermittently recorded — capillary refill, urine output.
ISCCM names both; neither needs equipment.
assumptions
- idno_real_patient_data
severitycritical
statement
This build has NOT been run on real patient data. The synthetic
generator produces a deterioration curve that was written by hand.
Any performance figure it produces measures the generator, not sepsis.
Numbers from a synthetic run are a test that the pipeline works, and
nothing more.
- idliterature_is_us_derived
severityhigh
statement
87 of 91 reviewed studies used US data. No published performance figure
here describes an Indian population. Transfer decay is documented
(0.92 -> 0.77 across datasets) and should be assumed worse for India.
- idonset_has_no_objective_timestamp
severityhigh
statement
Sepsis has no bell. Every published lead time depends on how onset was
defined, and several anchor on antibiotic administration — a record of
clinician behaviour. Lead times measured against a treatment proxy are
lead times relative to when staff acted.
- idhrv_evidence_is_thin
severityhigh
statement
The 35-hour HRV lead time rests on 17 patients with 14 cases, with
individual values from 0 to 227 hours. Directionally interesting.
Not a threshold.
- idutility_not_auroc
severitycritical
statement
External Utility Score is negative (-0.164) across the reviewed
literature while AUROC remains ~0.78. A model that looks acceptable on
AUROC may do net harm. Both must be reported.
- idclinical_features_beat_algorithms
severitymedium
statement
The review found hand-crafted clinically informed features outperformed
algorithmic feature learning. This argues for clinician input into
feature design over model sophistication — which is the case for clinician
involvement, not merely a courtesy.