fre.cmor.cmor_mixer module

FRE / CMOR Metadata Mixing and Rewriting (CMORization)

This module provides routines which rewrite post-processed FRE/FMS model output in a community-driven, standardized way. This module relies heavily on PCMDI’s CMOR module and it’s python API. It is the core implementation for fre cmor run operations- mixing and matching GFDL’s and FRE’s conventions to CMOR’s expectations, so that participation in model-intercomparison projects may be eased. For more usage details, see the project README.md, the FRE documentation, and PCMDI’s CMOR module documentation available at https://cmor.llnl.gov/.

This module currently follows a composite pattern, analogous to nested/russian doll, where a large piece contains a smaller piece which contains another. fre cmor run leads directly to cmor_run_subtool, which calls cmorize_all_variables_in_dir once, which calls cmorize_all_variables_in_dir once, which calls cmorize_target_var_files once per variable in a variable list, and calls rewrite_netcdf_file_var once per found datetime for a given variable. Functions within cmor_helpers assist with the CMORization process.

Functions

  • rewrite_netcdf_file_var(...)

  • cmorize_target_var_files(...)

  • cmorize_all_variables_in_dir(...)

  • cmor_run_subtool(...)

Note

The name “mixer” comes from a conversation between Chris Blanton, the original code author (Sergey Nikonov), and the next author/maintainer, Ian Laflotte, in 2022. Chris wanted to change the name, and Sergey kind of enjoyed the original CMORCommander.py, and so did not have any suggestions. Ian, whom was very new and knew nothing, suggested “cmor mixer”, not truly understanding why. Chris and Sergey decided to go with it.

fre.cmor.cmor_mixer.cmor_run_subtool(indir: str = None, json_var_list: str = None, json_table_config: str = None, json_exp_config: str = None, outdir: str = None, run_one_mode: bool | None = False, opt_var_name: str | None = None, grid: str | None = None, grid_label: str | None = None, nom_res: str | None = None, start: str | None = None, stop: str | None = None, calendar_type: str | None = None) → int

Main entry point for CMORization workflow, steering all routines in this file.

Parameters:

  • indir (str) – Directory containing NetCDF files to process.

  • json_var_list (str) – Path to JSON file with variable mapping (local to target names).

  • json_table_config (str) – Path to MIP table JSON file (per-variable metadata).

  • json_exp_config (str) – Path to experiment configuration JSON file (for header metadata).

  • outdir (str) – Output directory root for CMORized files.

  • run_one_mode (bool, optional) – If True, process only one file per variable.

  • opt_var_name (str, optional) – If provided, only process this variable.

  • grid (str, optional) – Grid description (if gridding is specified).

  • grid_label (str, optional) – Grid label (must match controlled vocabulary if provided).

  • nom_res (str, optional) – Nominal resolution for grid (must match controlled vocabulary if provided).

  • start (str, optional) – Start year (YYYY) for files to process.

  • stop (str, optional) – Stop year (YYYY) for files to process.

  • calendar_type (str, optional) – CF-compliant calendar type.

Raises:

  • ValueError – If required parameters are missing or inconsistent.

  • FileNotFoundError – If required files do not exist.

Returns:

0 if successful.

Return type:

int

Note

Updates grid, label, and calendar fields in experiment config if needed.

Note

Loads variable mapping and MIP table, filters variables, and orchestrates file processing.

fre.cmor.cmor_mixer.cmorize_all_variables_in_dir(vars_to_run: Dict[str, Any], indir: str, iso_datetime_range_arr: List[str], name_of_set: str, json_exp_config: str, outdir: str, mip_var_cfgs: Dict[str, Any], json_table_config: str, run_one_mode: bool) → int

CMORize all variables in a directory according to a variable mapping.

Parameters:

  • vars_to_run (dict) – Mapping of local variable names (in filenames) to target variable names (in NetCDF).

  • indir (str) – Directory containing NetCDF files to process.

  • iso_datetime_range_arr (list of str) – List of ISO datetime strings to identify files.

  • name_of_set (str) – Post-processing component or set label.

  • json_exp_config (str) – Path to experiment configuration JSON file.

  • outdir (str) – Output directory root for CMORized files.

  • mip_var_cfgs (dict) – Variable table from the MIP table JSON config.

  • json_table_config (str) – Path to MIP table JSON file.

  • run_one_mode (bool) – If True, process only one file per variable.

Returns:

0 if last file processed was successful, 1 if last file processed failed, -1 if no files were processed.

Return type:

int

Note

Errors for individual variables are logged and processing continues (except for run_one_mode).

fre.cmor.cmor_mixer.cmorize_target_var_files(indir: str = None, target_var: str = None, local_var: str = None, iso_datetime_range_arr: List[str] = None, name_of_set: str = None, json_exp_config: str = None, outdir: str = None, mip_var_cfgs: Dict[str, Any] = None, json_table_config: str = None, run_one_mode: bool = False)

CMORize a target variable across all NetCDF files in a directory.

Parameters:

  • indir (str) – Path to the directory containing NetCDF files to process.

  • target_var (str) – Name of the variable to process in each file.

  • local_var (str) – Local/filename variable name (often identical to target_var).

  • iso_datetime_range_arr (list of str) – List of ISO datetime strings, each identifying a specific file.

  • name_of_set (str) – Post-processing component or label for the targeted files.

  • json_exp_config (str) – Path to experiment configuration JSON file.

  • outdir (str) – Output directory root for CMORized files.

  • mip_var_cfgs (dict) – Variable table from the MIP table JSON config.

  • json_table_config (str) – Path to MIP table JSON file.

  • run_one_mode (bool, optional) – If True, processes only one file and exits.

Raises:

  • ValueError – See function body for details.

  • OSError – See function body for details.

  • Exception – See function body for details.

Returns:

None

Return type:

None

Note

Copies files to a temporary directory, runs CMORization, moves results to output, cleans up temp files.

fre.cmor.cmor_mixer.rewrite_netcdf_file_var(mip_var_cfgs: dict = None, local_var: str = None, netcdf_file: str = None, target_var: str = None, json_exp_config: str = None, json_table_config: str = None, prev_path: str | None = None) → str

Rewrite the input NetCDF file for a target variable in a CMIP-compliant manner and write output using CMOR.

Parameters:

  • mip_var_cfgs (dict) – Variable table, as loaded from the MIP table JSON config.

  • local_var (str) – Variable name used for finding files locally.

  • netcdf_file (str) – Path to the input NetCDF file to be CMORized.

  • target_var (str) – Name of the variable to be processed.

  • json_exp_config (str) – Path to experiment configuration JSON file (for dataset metadata).

  • json_table_config (str) – Path to MIP table JSON file.

  • prev_path (str, optional) – Path to previous file (used for finding statics file for tripolar grids).

Raises:

  • ValueError – If unsupported vertical dimensions or inconsistent grid dimensions are found.

  • FileNotFoundError – If required statics file for tripolar ocean grid is missing.

  • Exception – For other errors in the metadata, file IO, or CMOR calls.

Returns:

Absolute path to the output file written by cmor.close.

Return type:

str

Note

This function performs extensive setup of axes and metadata, and conditionally handles tripolar ocean grids.