Long range cyclone forecast skill
Long range tropical cyclone forecasts using the Bureau of Meteorology's new ensemble-based models have been developed. The OWZ tracking scheme developed by Kevin Tory at the Bureau of Meteorology is applied to both models to identify regions which are conducive to sustained tropical cyclone genesis. Having detected such regions, the tracking scheme then follows storms which meet user-defined thresholds.

These forecasts use the ACCESS-GE2 Numerical Weather Prediction model to provide guidance for days 4-10, and the ACCESS-S1 seasonal forecast model for guidance from days 4-28. Forecasts inside three days are not possible with the OWZ tracking scheme.

ACCESS-GE2 and S1 both use the same atmospheric model running at the same spatial resolution. They use different methods to initialise their ensemble spreads. ACCESS-GE2 contains 24 ensemble members while ACCESS-S1 contains 33*. ACCESS-S1 has a dynamic ocean model (which is coupled to the atmosphere) which allows it to make long-range forecasts. In contrast, ACCESS-GE2 uses fixed ocean Sea Surface Temperatures. However ACCESS-GE2 uses a more sophisticated atmospheric data assimilation scheme to initialise the ensemble members.

*Of the 33, 11 run for six months while the remaining 22 are valid only for 42 days to give better skill in the multi-week forecast space.
ACCESS-GE2 skill
Skill for ACCESS-GE2 was assessed during the 2017-18 cyclone season, with 180 forecasts initialised from November 1st 2017 to 29th April 2018 inclusive. Verification was performed across the entire southern hemisphere (0 - 30 degrees south latitude, 30 - 240 degrees west longitude). This domain was first divided into 66 overlapping regions of 15 x 20 degree resolution (denoted 'SH'). Additional verification on the same domain was performed using 150 smaller regions with 10 x 15 degree resolution ('SH-SMALL').

Brier Skill Scores (BSS) and Reliability diagrams for each regions for 4-10 day forecasts are shown below.

BSS for 4-10 day forecasts SH SH-SMALL
Raw 0.227 0.196
w/ 14 m/s threshold 0.379 0.331
SH
map
SH
map
SH-SMALL
map
SH-SMALL
map

A forecast has skill if BSS is greater than zero. A perfectly reliable forecast would feature all points of the solid dotted line falling onto the diagonal (i.e. an event with a forecast probability of 40% will be observed 40% of the time).

Earlier work assessing ACCESS-S1 skill showed that using an 850 hPa wind-speed threshold of 14 m/s gave the best agreement with observations (i.e. higher BSS). Many of the false alarms created by the OWZ tracking scheme used on these ACCESS models are removed using this threshold. For ACCESS-GE2, the wind-speed threshold clearly increases skill and reliability.

Example products for ACCESS-GE2 are shown below. Raw tracks whose 850 hPa wind-speed doesn't reach 14 m/s are shown in dotted lines. These forecast storm tracks are less likely to reach cyclone strength.

Strike probabilities are computed by finding the proportion of forecast ensemble tracks which pass within a specified distance of every grid point (120 km for the case of ACCESS-GE2).
ACCESS-S1 skill
ACCESS-S1 skill has been assessed against the model hindcast and forecasts made during the 2017-18 cyclone season.
ACCESS-S1 hindcast skill
To assess model bias for seasonal forecasting ACCESS-S1 was run in hindcast mode (i.e. forecasts ran in the past) from 1990-2012 with four forecasts per month (issued on the 1st, 9th, 17th and 26th of each month). Each forecast had 11 ensemble members. Forecasts from November-April were included in the verification statistics The same verification regions outlined above for ACCESS-GE2 were used to verify ACCESS-S1 during this period. The hindcast is used to account for model biases using a Leave-One-Out-Cross-Validation calibration method.

Brier Skill Scores (BSS) and Reliability diagrams for each regions for multi-week forecasts are shown below.

SH Days 4-7 Days 8-14 Days 15-21 Days 22-28
Raw 0.216 -0.042 -0.143 -0.185
w/ 14 m/s threshold 0.216 0.128 0.044 0.023
Calibrated 0.268 0.155 0.096 0.074

SH-SMALL Days 4-7 Days 8-14 Days 15-21 Days 22-28
Raw 0.180 -0.041 -0.133 -0.167
w/ 14 m/s threshold 0.192 0.111 0.030 0.013
Calibrated 0.220 0.133 0.076 0.059
SH Raw
map
SH Raw
map
SH Raw
map
SH Raw
map
SH Raw w/threshold
map
SH Raw w/threshold
map
SH Raw w/threshold
map
SH Raw w/threshold
map
SH w/CAL
map
SH w/CAL
map
SH w/CAL
map
SH w/CAL
map
SH-SMALL Raw
map
SH-SMALL Raw
map
SH-SMALL Raw
map
SH-SMALL Raw
map
SH-SMALL Raw w/threshold
map
SH-SMALL Raw w/threshold
map
SH-SMALL Raw w/threshold
map
SH-SMALL Raw w/threshold
map
SH-SMALL w/CAL
map
SH-SMALL w/CAL
map
SH-SMALL w/CAL
map
SH-SMALL w/CAL
map
Using the 14 m/s wind speed threshold gives skill in the SH verification region out to four weeks, although the forecast is still overconfident. Skill assessed at higher spatial resolution (i.e. SH-SMALL) is lower and reliability for the 3rd week forecast (days 15-21) remains poor.

Using the calibrated probabilities gives the best improvement in skill and reliability at the expense of sharpness (i.e. we lose the ability to forecast very high probability events).

In addition to producing maps of raw tracks and strike probabilities, the ACCESS-S1 forecast can also produce calibrated forecasts.
ACCESS-S1 forecast skill
ACCESS-S1 skill was also assessed during the 2017-18 cyclone season. 157 forecast from the November 11th to April 28th (some forecast dates were missing due to machine maintenance).

The forecast skill for the 2017-18 season was better than the hindcast skill. This could be attributed to
  • Increased number of ensembles (33 in the operational forecasts vs. 11 in the hindcast)
  • Use of ACCESS-G to provide the atmospheric control member in the operational forecast vs. ERA analysis in the hindcast
  • Strong MJO events may have increased predictability of storm formation in 2017-18 (i.e. it might have been an easy season to forecast)
BSS and reliability plots for the 2017-18 season are shown below. During this season the raw tracks (with wind-speed threshold applied) gave better skill out to week two, with the calibrated forecasts giving better skill for weeks three and four.
SH Days 4-7 Days 8-14 Days 15-21 Days 22-28
Raw 0.221 0.089 -0.006 -0.060
w/ 14 m/s threshold 0.304 0.209 0.125 0.108
Calibrated 0.279 0.182 0.140 0.121

SH-SMALL Days 4-7 Days 8-14 Days 15-21 Days 22-28
Raw 0.168 0.051 -0.024 -0.054
w/ 14 m/s threshold 0.269 0.180 0.096 0.071
Calibrated 0.243 0.177 0.114 0.092

SH Raw
map
SH Raw
map
SH Raw
map
SH Raw
map
SH Raw w/threshold
map
SH Raw w/threshold
map
SH Raw w/threshold
map
SH Raw w/threshold
map
SH w/CAL
map
SH w/CAL
map
SH w/CAL
map
SH w/CAL
map
SH-SMALL Raw
map
SH-SMALL Raw
map
SH-SMALL Raw
map
SH-SMALL Raw
map
SH-SMALL Raw w/threshold
map
SH-SMALL Raw w/threshold
map
SH-SMALL Raw w/threshold
map
SH-SMALL Raw w/threshold
map
SH-SMALL w/CAL
map
SH-SMALL w/CAL
map
SH-SMALL w/CAL
map
SH-SMALL w/CAL
map