GOME2_NO2_2024_banner.png

Title

This notebook is an AC SAF notebook prepared by BIRA-IASB (Gaia Pinardi), on illustrating the use of:

A sample zip file is provided with this notebook and instructions on how to register and download more data from the AC SAF archive are also provided below.

AC SAF recomend the usage of HARP for reading, filtering, and gridding, as shown in a companion notebook (also available here: https://acsaf.org/notebooks.php).

Table of Contents

This notebook contains several sections:

Data description

Documentation

The GOME-2/MetOp total column products are generated at DLR in the framework of EUMETSAT’s AC SAF.

For information and access to all AC SAF products, please refer to the AC SAF web page and help desk: AC SAF Web page: http://acsaf.org AC SAF Helpdesk: helpdesk@acsaf.org

Additional information can be found in EUMETSAT's product Navigator accessible through the following link: http://navigator.eumetsat.int

Current and historical quick look images as well as value added products generated by DLR are available at: http://atmos.eoc.dlr.de/gome2

The ATBD (Algorithm Theoretical Basis Documents) and PUM (PRODUCT USER MANUAL) are available here: https://acsaf.org/docs/atbd/Algorithm_Theoretical_Basis_Document_NTO_OTO_Nov_2019.pdf and https://acsaf.org/docs/pum/Product_User_Manual_NTO_OTO_Nov_2019.pdf

Data download is described here: https://acsaf.org/offline_access.php and rely on accessing an ftp server.

Downloading test data

For this example, the data are included in the zip folder available with this notebook: GOME2A_201812.zip

save then the zip folder into the data folder, at the same level than this notebook

Data files

The GOME-2 GDP 4.8/4.9 L2 reprocessed datafiles are structures as one file per orbit, ie 15 files per day:

image.png

HDF5 Product Format Description

The product content comprises the following groups:

image.png

Several gases are provided into the same orbital file, as mentionned in the file name:

i.e., GOME_O3-NO2-NO2Tropo-BrO-SO2-H2O-HCHO_L2_20181201134102_051_METOPA_62878_DLR_06.HDF5 will include O3, NO2, NO2Tropo, BrO, SO2, H2O and HCHO data.

In the TOTAL_COLUMNS, the final VCD values are given for all the gases, in a order specified in the file name and in the variable /META_DATA/MainSpecies:

image.png

Imports

Standard library / numerics / IO imports for what follows: h5py/netCDF4/xarray to read the HDF5 file with three different approaches, numpy for arrays, cf_units for unit handling, datetime/warnings as usual helpers (glob for multi-file patterns, matplotlib/cartopy for the maps built later).

**AC SAF suggestion: use HARP** HARP (https://atmospherictoolbox.org/harp/) is **one of the best tools to read GOME-2 ACSAF Level-2 HDF5 products**, because it already contains built-in ingestion definitions for many satellite products (including GOME-2, OMI, TROPOMI). **HARP converts the native product structure into a standardized format so that different instruments have the same variable names and dimensions.** In this notebook we will use the commonly-used xarray package, but AC SAF suggest the use of HARP toolkit, see the companion notebook here https://acsaf.org/notebooks.php.

Reading the data

Here we points at a sample day of GOME-2 data on the downloaded and unzipped folder, builds the exact file name for that day/orbit and then lists the matching files on disk as a quick existence check before opening anything.

There are several possibilities in python to read an HDF5 file. We show here a few classical ones (with h5py and netCDF4) that preserve the original file structure and then illustate using xarray.

First option: inspecting the data with h5py

Lists the sub-groups (variable names) inside DETAILED_RESULTS and GEOLOCATION, the two groups used throughout the rest of this notebook, as a quick way to see what's available without opening a separate HDF5 viewer.

Second option: inspecting the data with netCDF4

Reused from an EUMETSAT example notebook (credited in the comment): pretty-prints every group's variable names, 3 per line, so the whole file structure can be scanned at a glance via the netCDF4 interface.

reading the data with xarray

depending on the xarray version number, the ingestion of structured files (ie containing groups and not only variables) is possible easelly, or more complicated. We show here how to read part of the file, focusing here on the 'DETAILED_RESULTS/NO2' group.

Please note that more recent xarray versions allow reading the full file with:

ds = xr.open_dataset(G2file, engine="h5netcdf", decode_cf=False)

As we are interested in the Tropospheric vertical column density of Nitrogen Dioxide (NO2), VCDTropo, we can retrieve the parameter using square brackets [] and the name of the parameter of interest. We will get an xarray DataArrary, which has a shape of a one-dimensional vector.

The xarray DataArray also store the variable attributes, like the unit, title, ... of the variable.

note on dimensions:

Lists the dataset's dimension names and sizes:

The 'phony_dim_7' and 'phony_dim_8' are actually dimensions that xarray invents bacause the HDF5 files lack proper NetCDF dimension scales. We could rename them, ie:

ds = ds.rename({"phony_dim_7": "ground_pixel"})

Unlike NetCDF4 files, many satellite HDF5 files do not store dimension scales, so xarray cannot determine which variables share dimensions. The NetCDF/HDF backend therefore invents phony_dim_0, phony_dim_1, ...

see attributes

We want to see the variable's metadata (title, unit, fill value, ...), that we can use later to build plot labels and units automatically.

We now have a DataArrary, but we would need geolocations to plot the data on a map. We can thus get them from the netCDF Dataset (previously called file: file = Dataset(G2file, diskless=True, persist=False))

create a xarray dataArray with geolocations

check the loaded NO2 values:

Here, the filled value is -1 and it is clear that the 'tropospheric_NO2_column_number_density' data are not given for all pixels, as it is not always valid.

As discussed in the PUM:

"The tropospheric NO2 column is only available for measurements with latitude < 70º and SZA < 80º. The error in the retrieved tropospheric NO2 columns can become (very) large for measurements with cloud fractions > 20% (cloud radiance fraction > 50%), and these tropospheric NO2 measurements should normally not be used. Therefore, the NO2Tropo entry in the TOTAL_COLUMNS group is not available for these cloudy conditions. "

This is what we see above, with values put to nan for these cases. These case be filtered out using the NO2_Flag

( For expert users and to facilitate detailed data analyses, the tropospheric NO2 columns for cloudy conditions remain available via the VCDTropo entry in DETAILED_RESULTS/NO2. As discussed in the PUM, several variables are reported in several places in the files: The NO2Tropo entry in the TOTAL_COLUMNS group contains the same value as the VCDTropo entry in DETAILED_RESULTS/NO2. These two entries contain the tropospheric NO2 column density. )

NO2_Flag:

NO2 Flag is degined in the PUM as:

image.png

Filtering the data

**The data need to be filtered** The data need to be filtered for clouds and for quality of the tropopsheric NO2 variable. We use the `NO2_Flag`: The tropospheric NO2 column and pollution corrected total NO2 column should only be used for measurements with **NO2Tropo_Flag set to 1 or 2 (as discussed in the [PUM](https://acsaf.org/docs/pum/Product_User_Manual_NTO_OTO_Nov_2019.pdf))**. We can **add the flag to the created dataset**

filter on "validity":

from PUM: keep NO2_Flag=1 or 2. This is translated in HARP into the variable "NO2_column_number_density_validity", and the filtering can be done at the ingestion level:

add the flag into a newly created dataset:

Builds a fresh xarray.Dataset from the no2_da DataArray (naming the variable NO2Tropo), attaches the NO2Tropo_Flag quality-flag variable alongside it, adds a few descriptive attributes (author, creation date, per-variable long name/unit), then re-displays the dataset to check the result.

We could built the same dataset from scratch (constructing data_vars/coords/attrs by hand) rather than starting from to_dataset() and assigning variables afterwards. We show it in a commented cell.

to avoid the problem of the different axis ("phony_dim_7" with the same dimension), force the flag (and others) to have the same dimension:

Re-displays the dataset now that NO2Tropo_Flag has an explicit ground_pixel dimension matching the other variables (fixing the phony_dim mismatch noted earlier).

now I have both variables and I can filter

as mentionned above, we want to keep only data with the NO2Tropo_Flag set to 1 or 2:

Inspects the dataset now that it also holds NO2Tropo_filtered, the NO2 column masked to keep only flag values ≤ 2.

add other variables:

sanity-check for both cloud_fraction and solar_zenith_angle, that are then attached to the dataset as new ground_pixel-indexed variables, and the dataset is redisplayed to confirm.

other possibility is to create mask with more complex situations:

and we can also filter all the variables in once, and create a new, filtered dataset:

Re-displays NO2Tropo_filtered, then checks its NaN-aware mean, and finally compares that mean against the boolean-mask version (NO2Tropo_filtered2) and the dataset-level .where(mask) version (no2_ds_sel) — a sanity check that the three filtering approaches agree.

Compare several of the loaded data (different options):

We can now compare the three products built so far side by side: the basic loading (the unfiltered product) and the distinct validity-flag values with different extra options. We show the pixel counts that each filter level keeps and confirm e.g. that product_sel/product_sel2 only contain flags ≤ 2.

Plotting of the data

To plot the tropospheric NO2 data, we create a function that works on arrays. We provide as input some arguments such as the type of projection, the colorbar type, the colorbar min and max, ...

create a fct for the plotting

plot from the xarray product:

We call plot_map_GOME2; the next two cells inspect the plotted array and its summary statistics.