Up to 0.864 in U.S. and Austrian ICUs, 0.698 in China
ELDER-ICU, an ICU mortality prediction model for patients 65+, reached up to 0.864 AUROC in U.S. and Austrian cohorts but 0.698 at a site in China. The results highlight that performance can vary across settings, so hospitals buying AI built elsewhere should validate it on local patients and ensure contracts allow recalibration or retraining when needed.
This story was produced through MarketScale. See how Healthcare teams put it to work with Executive Thought Leadership.
Key takeaways
A vendor's validation claim is only as good as the patients it was tested on: one ICU mortality model scored 0.804 to 0.864 AUROC in U.S. and Austrian hospitals and 0.698 in China.
Match the fix to the failure. Recalibration improved probability accuracy at every site, while only retraining on local data recovered the bigger ranking losses at the South Korean and Chinese sites (+0.066 and +0.076 AUROC).
Checking AI on local patients takes staff and contract terms: whether the license permits local recalibration or retraining, and who on the team can run it.
Free workspace
Turn your Healthcare expertise into content.
Record interviews, organize footage, and write with AI on a free trial of the MarketScale platform for qualifying companies. No demo required, no credit card.
A risk prediction model for older ICU patients scored as high as 0.864 AUROC in U.S. and Austrian hospitals. At a site in China, the same model scored 0.698. For anyone signing a contract for a prediction tool trained on another health system's patients, that gap is a purchasing problem as much as a research finding.
The study behind those numbers, which tested a model called ELDER-ICU, was cited in Telehealth.org's coverage of the Telehealth Connect Summit 2026, held in Tunis from Sept. 29 to Oct. 1. Health ministries, the World Health Organization, the International Telecommunication Union, the Global Digital Health Partnership (GDHP) and private technology companies met there to get digital health and AI out of pilot mode. The site's Nov. 9 report by Julia Ivanova drew on attendee Maya Sherne, Telehealth.org's general manager.
The idea most useful to a hospital technology buyer came from the Multilateral Perspectives on Digital Health panel. Speakers laid out AI adoption in three moves: adopt an existing model, adapt it to the national context, then advance it with local expertise and data. Their real point was capacity. Countries need people who can evaluate AI systems themselves instead of relying entirely on technology built somewhere else.
That sounds like a ministry-level aspiration, but it applies just as well to a single hospital. A vendor's validation claim is only as good as the patients it was tested on. For a chief medical information officer, the practical questions are how far a model's accuracy travels and what it takes to bring it back when it doesn't. The ELDER-ICU study puts numbers on both.
An ICU model tested in four countries
The study, published in npj Digital Medicine, tested ELDER-ICU, a machine learning tool that predicts in-hospital death for ICU patients aged 65 and older using clinical variables recorded in the first 24 hours after admission. Its developers originally built and validated it on data from the United States and Amsterdam. This round, researchers ran it at 12 ICU sites in the U.S., Austria, South Korea and China, testing both geographic differences and shifts over time, including the COVID-19 period.
In the U.S. and Austrian cohorts, the model held up, with AUROC between 0.804 and 0.864. At the South Korean site it slipped to 0.753. At the Chinese site it fell to 0.698.
The patients behind those scores were very different. Mortality ranged from about 6% at the South Korean site to about 22% at the Chinese site, with every other site falling between 10% and 21%. The authors concluded that rigorous validation and proactive adaptation are essential before clinical use wherever populations and clinical practice vary.
The study compared countries. The same reasoning could apply wherever a hospital's patient mix or care practices differ from the data a model learned on.
AUROC measures how well a model ranks patients by risk: 0.5 is a coin flip, and 1.0 means it perfectly separates patients who died from those who survived. Calibration is a separate test of whether the predicted probabilities match real outcomes, which matters when clinicians act on a stated risk percentage.
Three fixes, each with a different price
The study gets most useful after the drop. Researchers tried three ways of updating the model, and each one asks more of the hospital than the last.
Recalibration is the lightest. It adjusts the probabilities the model outputs to match local outcomes without rebuilding anything. Two standard methods, isotonic regression and Platt scaling, improved calibration across sites. Incremental training goes further: it fine-tunes the model on newly collected local data while keeping its core architecture, and it improved performance at most centers. The authors note this method has seen little use in critical care so far.
Retraining is the heaviest option, and it produced the biggest gains where they were needed most. AUROC rose by 0.066 at the South Korean site and 0.076 at the Chinese site, the two places where the original model had slipped furthest.
Taken together, the results suggest a practical order of work. Measure performance on local patients first. Recalibrate when the model ranks patients well but its stated probabilities are off. Save retraining for sites where the ranking itself breaks down. Each step needs something different: outcome data the hospital can label, people (in-house or vendor) to run the update, and contract terms that allow the model to be changed at all.
The same study also shows where this matters less. Performance stayed strong in the U.S. cohorts, the country the model was first built on, and in Austria. A hospital whose patients resemble a vendor's development population may need only a local check and a calibration pass, a far smaller project than the retraining the Asian sites needed.
Why validation so often goes undone
The same panel offered a reason that work gets skipped: agreeing on technical standards doesn't mean anyone puts them into practice. Speakers described fragmented projects piling up when health ministries, telecom agencies and private companies build systems on their own. Their answer was national interoperability standards, digital public infrastructure and government oversight to tie the efforts together.
Standards still need people to apply them, and in the U.S. that capacity gap shows up in rural planning. A May 2026 National Consortium of Telehealth Resource Centers webinar on state Rural Health Transformation Fund plans noted that rural providers increasingly rely on technical and operational support that may not be fully reflected in current workforce strategies. That matters for AI. If no one on staff can run a local validation, a hospital could be left relying on whatever the vendor tested.
Questions for the next AI contract
Before signing, ask for performance results from sites whose patients look like yours, whether the license allows local recalibration or retraining, and who runs and pays for that work after go-live.
The summit's final day turned to execution, with workshops on building interoperable health systems and designing telehealth and AI projects that move from planning to implementation. Panelists also argued that tight resources can push health systems toward new ways of delivering care, and that countries can learn from each other instead of copying models built in wealthier systems. A teleradiology session with participants from Egypt, Tunisia and the WHO focused on that kind of exchange.
For hospitals whose patients differ most from a model's training data, the ELDER-ICU results make the stakes concrete: at the Chinese site, retraining improved AUROC by 0.076. That work is possible only if the license allows the model to be changed and someone, in-house or at the vendor, is assigned to run it after go-live.
Sources
Your experts belong here
Every story in MarketScale Healthcare starts with a company putting its clinicians, service-line leaders, and field engineers on the record. Buyers are already reading this topic. The only question is whose experts they find.
Service-line buyers vet vendors quietly, and your clinicians become the proof they find while doing it.
About the author
The MarketScale Newsroom reports on the companies, technologies, and trends shaping 16 B2B industries. It turns primary sources and expert commentary into clear, useful coverage for the people doing the work.