In various benchmarks, ClimaX outperforms state-of-the-art models and can be fine-tuned for a variety of prediction applications.
ClimaX is a deep learning foundation model for weather and climate modelling that was open sourced by researchers from Microsoft’s Autonomous Systems and Robotics Research team. With two significant changes, ClimaX is based on the Vision Transformer (ViT) concept, which was first created for processing picture data. The first is variable tokenization, which enables the model to accept information from datasets with a range of input variable counts. The second method, known as variable aggregation, aggregates all input variables for a certain spatial position. Five datasets from the CMIP6 collection are used for the model’s pre-training.
The Microsoft team chose to use the same strategy for weather and climate prediction jobs because of the effectiveness of pre-trained foundation language models, which can be adjusted for cutting-edge performance on a range of downstream NLP tasks. The fundamental structure of ClimaX is an image-to-image vision transformer; the input is a two-dimensional grid, but each grid element stores a variety of heterogeneous meteorological variables, such as temperature and air pressure, rather as RGB pixel values. The model’s job is to provide an image that depicts the weather at some point in the future.
The team tested ClimaX by fine-tuning it for both tasks that used variables the model had never seen before and tasks whose input variables were identical to those used during pre-training. The first category of tasks includes sub-seasonal to seasonal prediction, global weather forecasting, and regional weather forecasting. The second type, which was not employed during pre-training, used the ClimateBench benchmark for predicting the state of the climate. In this example, the input variables are the amounts of gases like carbon dioxide. ClimaX outperformed baselines in terms of temperature prediction, while it underperformed in terms of precipitation prediction.
