# Preliminary CAMS benchmark, January-May 2026

Run date: 6 July 2026.

Status: **internal retrospective benchmark; not yet an independent or blind
validation result**.

## Result

On the 107 forecast initializations in the checkpoint's existing 2026 test split,
TCOscope has lower TROPOMI-grid RMSE than operational CAMS at every lead.

| Lead | TCOscope RMSE, DU | CAMS RMSE, DU | Persistence RMSE, DU | TCOscope reduction vs CAMS |
| --- | ---: | ---: | ---: | ---: |
| T+1 | 9.78 | 16.75 | 19.68 | 41.6% |
| T+2 | 11.08 | 17.10 | 25.75 | 35.2% |
| T+3 | 12.10 | 17.55 | 28.33 | 31.0% |
| T+4 | 13.32 | 18.30 | 29.70 | 27.2% |
| T+5 | 15.30 | 19.45 | 31.03 | 21.3% |
| **All** | **12.46** | **17.86** | **27.19** | **30.2%** |

Additional checks:

- TCOscope mean bias: -1.87 DU; CAMS mean bias: -7.46 DU.
- Centered RMSE after removing each system's single aggregate bias: 12.32 DU for
  TCOscope and 16.23 DU for CAMS.
- Area-weighted RMSE: 10.81 DU for TCOscope and 16.28 DU for CAMS, a 33.6%
  reduction.
- Skill against persistence: 0.542 for TCOscope and 0.343 for CAMS.
- TCOscope has lower daily RMSE on 107/107 init dates at T+1 through T+4 and on
  106/107 dates at T+5.
- A paired seven-day circular block bootstrap gives a 95% interval of 27.4% to
  32.7% for the overall RMSE reduction. The paired TCOscope-minus-CAMS RMSE
  difference interval is -5.92 to -4.83 DU.

Per-lead bootstrap intervals for RMSE reduction:

| Lead | 95% interval |
| --- | ---: |
| T+1 | 39.5%...43.4% |
| T+2 | 32.6%...37.5% |
| T+3 | 28.1%...33.6% |
| T+4 | 23.7%...30.2% |
| T+5 | 17.9%...24.4% |

## Latitude-band comparison

The same shared observation mask was split into five fixed latitude belts. The
primary values below are unweighted within each belt, matching the existing model
test aggregation.

| Band | TCOscope RMSE, DU | CAMS RMSE, DU | Reduction | 95% block-bootstrap interval |
| --- | ---: | ---: | ---: | ---: |
| Antarctic, 90S...60S | 10.46 | 13.80 | 24.2% | 12.5%...33.8% |
| SH midlatitudes, 60S...30S | 9.41 | 15.85 | 40.6% | 39.0%...42.2% |
| Tropics, 30S...30N | 4.95 | 8.24 | 39.9% | 37.9%...41.8% |
| NH midlatitudes, 30N...60N | 17.79 | 26.79 | 33.6% | 31.1%...35.8% |
| Arctic, 60N...90N | 20.10 | 25.57 | 21.4% | 16.8%...26.3% |

RMSE reduction relative to CAMS by lead:

| Band | T+1 | T+2 | T+3 | T+4 | T+5 |
| --- | ---: | ---: | ---: | ---: | ---: |
| Antarctic | 39.7% | 30.0% | 23.9% | 19.0% | 14.2% |
| SH midlatitudes | 52.1% | 47.7% | 43.1% | 37.2% | 28.9% |
| Tropics | 50.4% | 42.9% | 39.2% | 36.6% | 33.5% |
| NH midlatitudes | 42.7% | 38.4% | 35.5% | 31.7% | 24.5% |
| Arctic | 34.1% | 25.7% | 20.9% | 18.0% | 14.1% |

Daily TCOscope wins out of 107 init dates:

| Band | T+1 | T+2 | T+3 | T+4 | T+5 |
| --- | ---: | ---: | ---: | ---: | ---: |
| Antarctic | 104 | 99 | 89 | 81 | 76 |
| SH midlatitudes | 107 | 107 | 107 | 107 | 107 |
| Tropics | 106 | 106 | 106 | 106 | 106 |
| NH midlatitudes | 107 | 107 | 107 | 107 | 106 |
| Arctic | 107 | 104 | 98 | 94 | 88 |

The strongest and most stable advantage is in the southern midlatitudes and the
tropics. Absolute errors are highest in the northern midlatitudes and Arctic,
where January-May ozone variability is also largest. The relative advantage
decreases with lead in both polar belts, but remains positive in every belt and
lead. In the Antarctic T+5 case the result is less uniform day to day (76/107 wins),
which reinforces the need for a separate ozone-hole-season evaluation.

Spherical area weighting does not remove the result. The all-lead TCOscope/CAMS
RMSE pairs are 10.29/14.91 DU in the Antarctic, 9.12/15.27 in SH midlatitudes,
4.89/8.19 in the tropics, 17.38/26.16 in NH midlatitudes, and 20.66/27.40 in the
Arctic.

## Protocol

- Cases: 107 consecutive T0 dates, 2026-01-21 through 2026-05-07.
- Valid dates: 2026-01-22 through 2026-05-12, depending on lead.
- Truth: the exact TROPOMI daily target and observed-cell mask used by the model
  evaluation pipeline.
- TCOscope: production `earthformer-ema-v1` checkpoint, SHA256
  `73162ed9a6491b6d78423f9ee2f570dab1eb4eee4b42d2ef9a4ef7aec612d2eb`.
- CAMS: ADS `total_column_ozone`, GEMS chemistry field `gtco3`, GRIB parameter
  210206, 00 UTC cycles, T+24/48/72/96/120.
- CAMS units: converted from kg m-2 using `1 DU = 2.1415e-5 kg m-2`.
- CAMS remapping: conservative spherical-area remapping from the regular 0.4-degree
  grid to the actual TCOscope 1.2-degree cell bounds.
- Aggregation: squared error accumulated over the shared TROPOMI observation mask;
  the primary table is unweighted to reproduce the existing test metric exactly.
- Uncertainty: paired circular block bootstrap over init dates, seven-day blocks,
  10,000 iterations, seed 42.

The reproducible implementation is
[`forecast/src/evaluation/compare_cams.py`](../forecast/src/evaluation/compare_cams.py).
Generated machine-readable outputs are written to
`forecast/outputs/cams_benchmark_2026/{summary.json,cases.csv,latitude_cases.csv}`
and are intentionally ignored as local research output.

## What this result does and does not establish

This is strong preliminary evidence that TCOscope predicts the future TROPOMI-like
TCO field more accurately than CAMS over this period. The advantage is present on
nearly every day, remains after removal of aggregate bias, and becomes slightly
larger under spherical area weighting. It is therefore not explained only by one
bad CAMS date, polar oversampling, or a constant CAMS offset.

It is **not yet sufficient** to claim general superiority over CAMS because:

1. TCOscope was trained to reproduce the TROPOMI product used as truth. The result
   may include sensor/retrieval specialization.
2. The current validation/test split has overlapping raw dates at the 2025/2026
   boundary, and 2026 metrics were inspected during model development.
3. CAMS instantaneous 00 UTC lead fields are compared with daily satellite
   retrieval grids; a later benchmark must match the observation time explicitly.
4. The period covers Northern Hemisphere winter and spring but not the Antarctic
   ozone-hole season.
5. Subsequent OMPS and ground-based WOUDC comparisons are now available. They
   support the result but remain retrospective; see
   [`omps-benchmark-2026-preliminary.md`](omps-benchmark-2026-preliminary.md) and
   [`woudc-benchmark-2026-preliminary.md`](woudc-benchmark-2026-preliminary.md).

Allowed wording at this stage:

> In a retrospective TROPOMI-consistency benchmark over 107 initializations from
> January to May 2026, TCOscope reduced RMSE by 30.2% relative to operational CAMS
> after both systems were evaluated on the same 1.2-degree grid and observation
> mask.

Not yet allowed:

> TCOscope is more accurate than CAMS in general.
