Clarification on Global Mean RMSE Oscillation for Near-Surface Temperatures

[yashdiggikar]

Hi
I’m observing an interesting behavior while evaluating GraphCast forecasts against ERA5 data.

When plotting the global mean of near-surface temperature variables, specifically t1000 and t2m, the forecast mean oscillates in sync with ERA5, which looks physically correct and expected.

However, the global mean RMSE computed for a single 40-step (6-hour interval) run also shows a noticeable oscillatory pattern instead of a smooth or gradually increasing trend.

Is this RMSE oscillation expected, or does it indicate an underlying issue with how RMSE is being computed or with the model’s short-term stability?

Details:
Model: GraphCast-Small (1.0° / 13 levels)
Forecast length: 40 steps (6-hour intervals)
RMSE computed as global cosine-latitude-weighted mean over all grid points and times
Both forecast and ERA5 data are aligned on valid time coordinates before evaluation

RMSE typically increases smoothly with lead time, so I wanted to confirm whether oscillations in RMSE are normal behavior for GraphCast for near-surface temperature variables.

[yashdiggikar]

Hi, just checking in to see if anyone has insights on this.
I’m still observing a clear oscillation in the global RMSE for t2m/t1000 across the 40-step forecast, even though the forecast means track ERA5 correctly. I’ve verified time alignment and cosine-latitude weighting, but the oscillatory RMSE pattern persists.

Would appreciate any guidance on whether this behavior is expected for GraphCast or if it suggests something I should double-check in my evaluation setup. Thanks!

[andrewlkd]

Hey,

Could you elaborate on what you're plotting/seeing?

Running the GraphCast colab with GraphCast Small, rolling out 40 steps and computing the globally averaged RMSE

from graphcast import losses
mse = (predictions['2m_temperature'] - eval_targets['2m_temperature'])**2
weighted_mse = mse * losses.normalized_latitude_weights(eval_targets)
global_avg_weighted_rmse = weighted_mse.mean(dim=['lat', 'lon']).squeeze('batch') ** 0.5
global_avg_weighted_rmse.plot()

we see

Best,

Andrew