PatchTST-FM-r2 Beats r1 on Average and Loses on 24 of 97 GIFT-Eval Configurations

IBM released Granite PatchTST-FM-r2 on 9 September, and the Hugging Face announcement leads with where it sits among zero-shot time series forecasting models: second on the GIFT-Eval leaderboard, and first among those under a commercial-friendly license. I recomputed that ranking from the result files the benchmark publishes, and it holds. What the headline does not show is the quarter of the benchmark where the new model is worse than the one it replaces.

What shipped

The checkpoint is ibm-granite/granite-timeseries-patchtst-fm-r2. Its safetensors metadata counts 384,600,208 float32 parameters, against 257,895,552 for r1, and the weights file grows from about 1.03 GB to 1.54 GB. The config shows where the parameters went: 30 blocks instead of 20, each a conformer block that adds a convolution after the attention layer, with kernel sizes cycling through 5, 5, 3, 3. Patches stay 16 steps wide but now overlap at a stride of 8, so a full 8,192-step context becomes 1,023 patches where r1 had 512. Context length, the 64-step native output and the 99-quantile head are unchanged.

Running it needs granite-tsfm 0.3.9 or later. That release, tagged 28 August, is where the new config fields such as the block type and the kernel list first appear, and it widens the PyTorch pin to 2.10 up to but not including 2.12.

How time series forecasting models rank on GIFT-Eval, filter by filter

GIFT-Eval divides each model's error on each of 97 dataset configurations by a seasonal naive forecast's error, then takes a geometric mean. Applying the Space's own normalisation to the result files of the 127 models with complete runs matches what IBM reported: a CRPS of 0.467 and a MASE of 0.6846.

The rank depends on which boxes are ticked:

• Zero-shot only, no test leakage, replication code published: 31 models, and r2 is second on both metrics, behind Google Research's TimesFM-3 at a CRPS of 0.4557.
• Add pretrained models, which may train on the benchmark's own train splits: r2 falls to third on CRPS and fourth on MASE.
• Drop the replication-code requirement: fourth, behind EXAONE-Forecast and DeOSAlphaTimeGPTPredictor-2025.
• Everything on the board, agentic and fine-tuned entries included: 28th on CRPS.

The license qualifier in the announcement is doing real work. TimesFM-3's Hub card names its license as non-commercial; the third-placed TiRex-2 zero-shot checkpoint is tagged Apache 2.0 and trails r2 by about 2% on CRPS. The claim is accurate, and it is as much about licensing as accuracy: the only model ahead of r2 under that filter ships under a license its own card calls non-commercial.

The model card has not caught up on one point. It still describes the GIFT-Eval results as sitting in a pending pull request. The benchmark repository merged that pull request, number 204, on 31 August.

What changes for a team already on r1

Across the benchmark, r2 lowers geometric-mean CRPS by 4.2% and MASE by 4.5% against r1. The gains are uneven: CRPS drops 13.0% on the five healthcare configurations and MASE 18.3% on the four sales ones, while CRPS on energy and transport improves by only 2.6%.

The average also hides regressions. On 24 of the 97 configurations, r2's CRPS is worse than r1's. The worst are solar/W/short, 29.8% worse; bizitobs_application/10S/long, 13.3% worse; and two hourly Bitbrains short horizons, both nearly 12% worse. On three short-horizon configurations, r2's point forecast scores worse than repeating the last season: its MASE sits above the seasonal naive baseline on daily ETT1, weekly ETT2 and M4 daily.

That is the central claim of this post: moving from r1 to r2 is a per-series decision, not a version bump, and on the benchmark's own evidence roughly one series in four gets worse. If your backtest shows r2 winning on nearly every series you forecast, that regression rate did not transfer to your data, and a fleet-wide swap is fine.

Item one on the PoC-to-production checklist is evaluation you can defend, with regressions caught before they ship. A leaderboard rank is a proof of concept for a model, and its geometric mean averages away exactly the regressions that checklist asks you to catch. The backtest on your own series is the part that ships, and it is the first deliverable a machine learning consulting engagement on forecasting should produce.

What it breaks

License gates. The LICENSE file offers a choice of OpenMDW 1.0 or Apache 2.0. The Hub metadata carries only one of them:

curl -s https://huggingface.co/api/models/ibm-granite/granite-timeseries-patchtst-fm-r2 | jq -r .cardData.license

That prints openmdw-1.0; the same query for r1 prints apache-2.0. A dependency allowlist that reads Hub tags and approved r1 as Apache will now meet a license it has never reviewed. OpenMDW grants broad rights over weights, data and code together, asks you to keep the license and notices, and ends the grant for anyone who sues claiming the materials infringe a patent. A scanner knows none of that. Record which of the two licenses you elect.

Signature verification, partly. r2 ships a model.sig, which r1 did not: a Sigstore bundle signed with an IBM code-signing certificate issued by DigiCert and valid until November 2027. Its manifest lists nine files. Eight of their digests match what the Hub serves today, the 1.54 GB weights included. The ninth, digicert-root.pem, returns a 404.

Support expectations. Both model cards carry IBM's disclosure that it has no obligation to provide updates or support and will not maintain the code. The library shipped three releases between 31 July and 28 August, but the disclaimer, not the cadence, is what you can hold IBM to.

Verdict

If you run r1 today, make r2 the default candidate; the gains are real and the ranking reproduces. Do not flip it everywhere:

1. Install with pip install 'granite-tsfm>=0.3.9' and pin the Hub commit, b125275 at the time of writing.
2. Backtest r1 and r2 on the same rolling origins at your production horizon, scoring CRPS per series.
3. Move a series to r2 only where it wins. If more than a third of your series regress, stay on r1 and treat r2 as a per-domain tool.
4. Keep seasonal naive in that backtest. r2's MASE was worse than seasonal naive's on three GIFT-Eval configurations, and it costs nothing to run.
5. File the license election in writing before a scanner files it for you.