ECMWF Newsletter #175

Forecast performance 2022

Thomas Haiden
Matthieu Chevallier
David Richardson

 

ECMWF maintains a comprehensive range of verification statistics to evaluate the accuracy of its forecasts. Each year, a summary of verification results is presented to ECMWF’s Technical Advisory Committee (TAC). Their views about the performance of the operational forecasting system in 2022 are given in the box.

Cycle 47r3 of ECMWF’s Integrated Forecasting System (IFS), implemented on 12 October 2021, included a major upgrade to the moist physics of the model. Together with a number of other changes, this cycle improved upper-air forecasts in the early medium range such that in 2022 new high points in skill relative to forecasts based on the ERA5 reanalysis system were reached (see the figure). This cycle has been ported to the Atos high-performance computing facility in Bologna (Italy), where it has been running operationally since 18 October 2022.

Upper-air skill of the ENS relative to forecasts based on the ERA5 reanalysis system.
Upper-air skill of the ENS relative to forecasts based on the ERA5 reanalysis system. Skill of the ENS for upper-air parameters at day 5 in the northern extratropics, relative to a Gaussian-dressed ERA5 forecast. Values are running 12-month averages, and verification is performed against own analysis.

Compared to forecasts from other global centres, ECMWF has been able to maintain the overall lead for upper-air parameters in the medium range. For surface parameters such as 2 m temperature and precipitation, some of the other centres have drawn closer to ECMWF in the medium range and partly taken the lead in the short range. However, the number of large 2 m temperature errors in ensemble forecasts (ENS) has been further reduced according to one of ECMWF’s headline scores, defined as the continuous ranked probability score (CRPS) being larger than 5 K for forecast day 5.

The skill of the Extreme Forecast Index (EFI) has increased for 10‑metre wind speed and 2‑metre temperature, and slightly decreased for precipitation. In the extended range, the skill of predicting terciles of 2‑metre temperature anomalies in the extratropics in week 2 now consistently exceeds that of persistence by about 10% and that of climatology by 40%. Corresponding values for skill over weeks 3 and 4 combined are about 5% (with respect to persistence) and 20% (with respect to climatology). Summer 2022 positive temperature anomalies over Europe and northern Russia were captured at weeks 1 and 2 in ECMWF medium-range and early extended-range forecasts, whilst negative temperature and positive precipitation anomalies with signals of severe flooding over Pakistan were indicated up to three weeks ahead in extended-range forecasts.

Position errors for forecasts of tropical cyclones were similar to those in the year before. There is also still a notable slow bias in terms of propagation speed. With regard to forecast skill for ocean wave parameters, there has been a substantial improvement in wave peak period due to IFS Cycle 47r3. ECMWF is now leading compared to forecasts from other global centres both in peak period and significant wave height.

Early in 2022, ECMWF’s seasonal forecast predicted a gradual return from La Niña towards more neutral conditions within a few months, but in the observations La Niña conditions persisted throughout the year. Later in 2022, the forecast got closer to observations and shifted the return to neutral into early 2023. In the extratropics, a strong signal for a warm anomaly in the summer of 2022 in Europe was present in the forecast but its magnitude towards Scandinavia and Siberia was underestimated. The negative temperature anomaly over Pakistan, linked to a strongly positive precipitation anomaly leading to widespread severe flooding in the area, was indicated by the seasonal forecast for the summer of 2022.

The complete set of annual verification results is available in ECMWF Technical Memorandum No. 902 on ‘Evaluation of ECMWF forecasts, including the 2021 upgrade’, downloadable from http://www.ecmwf.int/en/publications/technical-memoranda.

The following are other sources of information about verification and forecasting system changes.

Assessment of ECMWF’s Technical Advisory Committee, 6–7 October 2022

With regard to its overall view of the performance of ECMWF’s operational forecasting system, the Committee:

  1. congratulated ECMWF on progress with the new high-performance computing facility in Bologna, noting that a test system for the CAMS global analyses and forecasts is already running, and is looking forward to the successful switch-over of the operational forecast system on 18 October 2022;
  2. recognised that ECMWF maintains a lead compared to other centres for a range of upper-air and stratospheric verification scores, medium-range ensemble precipitation, and the medium-range spread-skill relationship;
  3. noted, also, that ECMWF now performs less well than some other centres for some surface parameters in the shorter range, for example ensemble 2 m temperature, whilst recognising that proposed improvements in 48r1 and 49r1 will address some of these issues; also, in the southern hemisphere ECMWF's lead in some scores for upper-air parameters has narrowed relative to other centres;
  4. congratulated ECMWF on the successful implementation of IFS Cycle 47r3 in October 2021, which included an improved moist physics package and some new products introduced in response to user requests;
  5. noted that the introduction of 47r3 had a positive impact on many scores, including precipitation and upper parameters and improvements visible in the early medium-range ensemble scores. However, in verification against SYNOP 47r3 showed limited impact on 2 m temperature and 10 m wind scores and led to a reduction in skill in total cloud cover, which will be addressed in future model cycles;
  6. noted that ECMWF has maintained its lead compared to other centres for significant wave height and that the introduction of 47r3 has had a positive impact on peak period skill scores with ECMWF now leading other centres;
  7. recognised that EFI ROC skill scores over Europe aimed at high-impact weather showed improved skill for 2 m temperature and 10 m wind speed with new high points reached, whilst scores for 24 h precipitation fell away slightly but still remained high;
  8. noted that ERA5 is a useful benchmark for accounting for year-to-year variability and welcomed that the causes of the improvement seen in both HRES and ERA5 precipitation headline scores are to be investigated;
  9. noted improvement in tropical cyclone HRES and ENS position errors and speed, whilst central pressure errors remained similar to last year;
  10. noted that in the extended range the week 2 forecast continues to show improvement relative to persistence, but no statistically significant trends were apparent in weeks 3 and 4 compared to persistence;
  11. recognised that in the extended range the JJA 2022 positive temperature anomalies over southern Europe and northern Russia and negative temperature anomaly/positive precipitation anomaly over Pakistan were captured at weeks 1 and 2 as well as at longer lead times; signals for the Pakistan flooding appeared with 3 weeks’ lead time, whilst the positive temperature anomalies over northern Europe were, however, not identified at lead times of 3 weeks or longer;
  12. noted that on longer-range seasonal time scales ECMWF, like some other centres, returned ENSO to a neutral state too quickly and so did not capture the ongoing La Niña at lead times of 3 months and longer;
  13. noted that seasonal model outputs for DJF 2021–22 and JJA 2022 represented signals related to La Niña as well as capturing some element of the JJA 2022 warm anomaly over Europe and cold anomaly over Pakistan. However, during DJF 2021–22 in more northerly latitudes signals were more poorly captured, for example the warm anomaly over northern Russia and the cold anomaly over eastern North America;
  14. acknowledged the continuing high quality of atmospheric composition products and increase in performance of some outputs in CAMS; noted upper-air scores were degraded in 47r3 and recognised that this will be addressed soon;
  15. welcomed the introduction of additional datasets in verification, for example TOA net shortwave radiation and downward solar radiation at the surface, and the greater focus on near-surface variables in verification; appreciated the insightful presentation of verification data, for example scale-dependent aspects of precipitation at different lead times; suggested, based on forecaster feedback, that a verification metric reflecting duration of blocking situations, which can be underestimated, be developed;
  16. appreciated the continued development of new diagnostics and very good support ECMWF provided to Member and Co‑operating States over the last year, with engagement via many mechanisms including online support, the annual UEF, online seminars, site visits and meteorological representatives at Member States; the increase in in‑person contact as the COVID‑19 pandemic evolves is also noted.