A Churn Prediction Model You Never Train: Kumo Relational Wins the RelBench Average and Loses All Three Churn Tasks

A churn prediction model is usually the first thing a data team trains and the last thing it stops retraining. NVIDIA's pitch for Kumo Relational, published on 21 October, is that you stop: hand it your tables and a few thousand labelled rows as context, and it predicts churn, demand or fraud in one forward pass with no training run. The announcement calls it the first few-shot foundation model to outperform supervised machine learning on common relational benchmarks.

The news is the blog post, not the model. The KumoRFM-2 paper behind it reached arXiv on 14 April, and the Hugging Face repository was created on 20 July with a release commit dated 28 September. What changed this month is that the PyPI package the model card tells you to install, an empty placeholder when we checked Kumo Tabular on 30 September, became a real 0.2.1 release on 14 October.

The claim, stated fairly

The headline spans 41 tasks on 15 databases. On RelBench V1's classification tasks, the paper reports KumoRFM-2 beating gradient-boosted trees on expert-engineered features by 3.98 AUROC points on average, and the strongest supervised relational model by 1.54 points, with as little as 0.2 percent of the training rows as context. It also reports the model as the top in-context entry on Stanford's RelBench leaderboard.

That last part checks out. RelBench's board is a JSON file of self-reported submissions hosted by the Stanford STAR project: 27 entries, 20 of them on the binary classification board. The Kumo Relational entry, submitted on 9 October, averages 0.7926 AUROC across 12 classification tasks. Only Kapso, a Leeroo agent system entered as a trained model, sits above it at 0.8119. The next in-context entry, RDB-PFN, is at 0.7506.

Where the churn prediction model claim flips

Three of those 12 tasks are churn tasks. RelBench defines rel-amazon user-churn as whether a customer writes no review in the next three months, item-churn as whether a product gets none, and rel-hm user-churn the same way over H&M purchases. On all three, trained models sit above it.

On rel-hm user-churn the in-context model is level, to four decimal places, with the plain graph neural network baseline RelBench itself submitted. The one task Kumo Relational wins outright is rel-f1 driver-dnf, at 0.8434.

The paper names the mechanism. In its own comparison set, KumoRFM-2 wins by wide margins on the f1, avito, event and trial databases and concedes a slight advantage to supervised relational models on amazon, stack and hm, which it attributes to their larger training sets. RelBench's training table for rel-amazon user-churn has 4,732,555 rows and rel-hm user-churn has 3,871,410, so a 20,000-row context sees well under one percent of the labels. The rel-f1 tasks have 11,411 and 1,353. Size is not the whole rule: the paper's ratebeer database has up to 2.56 million training rows and 221 columns, and KumoRFM-2 wins both churn tasks there, though against only LightGBM, GraphSAGE and its own predecessor.

What the measurement leaves out

No training is not no tuning. NVIDIA's RelBench example lists per-task settings found through the RelArena-α protocol: a 20,000-row context for the churn tasks against a default of 10,000, eight estimators against a default of one, and neighbour fan-outs of 8, 32 and 1 per hop against a default of 16. Run the defaults and you are not running the submitted numbers.

The baselines were copied, and the protocol differs. The paper says it relies on publicly reported baseline numbers where possible. Prior Labs, which built RelArena-α to re-run every method under one data state, notes that KumoRFM-2's own evaluation does not freeze the database at the test cutoff as RelBench specifies, and re-ran it frozen. Its churn numbers are 0.6771, 0.8018 and 0.6781, between 1.4 and 2.0 points below the paper's 0.6910, 0.8217 and 0.6927, and on item-churn and H&M below the GNN baseline RelBench submitted.

The Pareto claim is self-run. The announcement places Kumo Relational on RelArena-α's accuracy-efficiency frontier, about 430 times faster than RT-PluRel. The live RelArena-α boards have no Kumo row. NVIDIA's own adapter README reports an Elo of 1832.2 and third of 12 on the models board as of 29 September, with a mean task runtime of one minute against 351 for RT-PluRel and 47 for GraphSAGE, all measured by NVIDIA.

The enterprise benchmark is gated. The SAP SALT results, including 7 MRR points over XGBoost, come from a dataset that requires approved access and is licensed CC-BY-NC-SA-4.0.

What survives contact with production

The weights are two checkpoint files of about 120 MB each under OpenMDW-1.1, fine-tuned from TabICLv2 under BSD-3-Clause, with Apache-2.0 code. Commercial licensing was the real news about Kumo Tabular and it holds here. The paper also shows the other half of the decision: fine-tuning on SALT lifts mean MRR from 0.83 to 0.89 in under two minutes per task. Whether to buy the in-context behaviour or spend those minutes is the choice we laid out in buy, build or fine-tune.

The central claim of this post: on churn targets with millions of labelled rows, the kind RelBench measures, a trained relational model or tuned pipeline still beats Kumo Relational in context, and the foundation model earns its place where labels are scarce. It is wrong if an in-context Kumo entry tops rel-amazon user-churn or rel-hm user-churn on the RelBench board. Check:

curl -sL https://raw.githubusercontent.com/stanford-star/relbench/main/leaderboard/leaderboard.json | jq -r --arg t rel-hm/user-churn '[.[] | select(.boards.binary_classification) | {name, in_context, auroc: .boards.binary_classification.results[$t]}] | sort_by(-.auroc) | .[] | [.auroc, .in_context, .name] | @tsv' | head -8

On 24 October the first in-context row is seventh. Until that changes:

• Count the labelled rows behind your churn target. Below about 20,000, run the in-context model first. Above a few million, keep your trained model as the bar it has to clear.
• Install from PyPI and run NVIDIA's RelBench example twice, with the listed per-task settings and with defaults, and time the one you would deploy.
• Freeze your database at the prediction cutoff before building context, as RelArena-α does, and compare on a time-ordered holdout.
• If the gap is small, try the fine-tuning path before scaling context.

Deciding which churn and scoring models to replace, and which to keep training, is the evaluation our machine learning consulting work exists for.