Tools

`fre`

The click based entry point to the rest of the package at the command line for fre-cli. fre has a command structure like, fre TOOL COMMAND, akin to the CLI’s provided by popular packages like git and cylc. This enables discovery of the tooling capability, useful for complex tools with multiple options and detailed configurations.

arguments

(optional) --help, help flag
- print information on TOOL options
(optional) -v[v], verbosity flag, up to two
- increments fre’s verbosity from the default logging.WARNING to logging.INFO with one -v, and again to logging.DEBUG with -vv
(optional) -q, quiet flag, up to one
- sets fre’s verbosity from the default logging.WARNING to logging.ERROR.
(optional) -l, --log_file PATH, argument specifying a log file in which to store logging messages
- the given (or default) verbosity will be used as-specified
- the text will still be seen in the terminal, and the format within the log_file is the same as what is printed to screen.

Tools that require arguments to run will alert user about missing arguments, and will also list the rest of the optional parameters if --help is executed. e.g.:

fre TOOL COMMAND --help

Argument flags are not positional, can be specified in any order. Some arguments expect sub-arguments.

`fre app`

fre app tools are intended to be a collection of single-purpose tools.

`fre catalog`

For more complete information on the catalogbuilder tool, please see its official documentation.

`build`

Generate a catalog.

Builds json and csv format catalogs from user input directory path
Minimal Syntax: fre catalog build -i [input path] -o [output path]
Module(s) needed: n/a
Example: fre catalog build -i /archive/am5/am5/am5f3b1r0/c96L65_am5f3b1r0_pdclim1850F/gfdl.ncrc5-deploy-prod-openmp/pp -o ~/output --overwrite

`validate`

Validate a catalog

Runs the comprehensive validator tool (validates vocabulary and ensures catalogs were generated properly)
Minimal Syntax: fre catalog validate [json_path] --vocab OR
fre catalog validate [json_path] --proper_generation
Module(s) needed: n/a
Example: fre catalog validate ~/example_catalog.json --vocab

`fre cmor`

See also, fre cmor’s README
See also, fre cmor’s project board

This set of tools leverages the external cmor python package within the fre ecosystem. cmor is an acronym for “climate model output rewriter”. The process of rewriting model-specific output files for model intercomparisons (MIPs) using the cmor module is, quite cleverly, referred to as “CMORizing”.

CMOR Subcommands Overview

fre cmor rewrites climate model output files with CMIP-compliant metadata. Both CMIP6 and CMIP7 workflows are supported. Available subcommands:

fre cmor run - Rewrite individual directories of netCDF files
fre cmor yaml - Process multiple directories/tables using YAML configuration
fre cmor find - Search MIP tables for variable definitions
fre cmor varlist - Generate variable lists from netCDF files
fre cmor config - Generate a CMOR YAML configuration from a post-processing directory tree

`run`

Rewrites netCDF files in a directory to be CMIP-compliant
Requires MIP tables and controlled vocabulary configuration
Minimal Syntax: fre cmor run -d [indir] -l [varlist] -r [table_config] -p [exp_config] -o [outdir] [options]
Required Options:
- -d, --indir TEXT - Input directory with netCDF files
- -l, --varlist TEXT - Variable list dictionary mapping local to MIP variable names
- -r, --table_config TEXT - MIP table JSON configuration
- -p, --exp_config TEXT - Experiment/model metadata JSON
- -o, --outdir TEXT - Output directory prefix
Optional:
- -v, --opt_var_name TEXT - Target specific variable
- --run_one - Process one file for testing
- -g, --grid_label TEXT - Grid type (e.g. “gn”, “gr”)
- --grid_desc TEXT - Grid description
- --nom_res TEXT - Nominal resolution
- --start TEXT - Minimum year (YYYY)
- --stop TEXT - Maximum year (YYYY)
- --calendar TEXT - Calendar type
Example: fre cmor run --run_one -g gr --nom_res "10000 km" -d input/ -l varlist.json -r CMIP6_Omon.json -p exp_config.json -o output/

`yaml`

Processes YAML configuration to CMORize multiple directories/tables
Requires FRE-flavored YAML files with experiment configuration
Minimal Syntax: fre cmor yaml -y [yamlfile] -e [experiment] -p [platform] -t [target] [options]
Required Options:
- -y, --yamlfile TEXT - YAML file to parse
- -e, --experiment TEXT - Experiment name
- -p, --platform TEXT - Platform name
- -t, --target TEXT - Target name
Optional:
- -o, --output TEXT - Output file
- --run_one - Process one file for testing
- --dry_run - Print planned calls without executing
- --print_cli_call/--no-print_cli_call - In dry-run mode, print the equivalent CLI invocation (default) or the Python cmor_run_subtool() call
- --start TEXT - Minimum year (YYYY)
- --stop TEXT - Maximum year (YYYY)
Example: fre cmor yaml -y am5.yaml -e c96L65_am5f7b12r1_amip -p ncrc5.intel -t prod-openmp --dry_run

`find`

Searches MIP tables for variable definitions
Minimal Syntax: fre cmor find -r [table_config_dir] [options]
Required Options:
- -r, --table_config_dir TEXT - Directory with MIP tables
Optional:
- -l, --varlist TEXT - Variable list file
- -v, --opt_var_name TEXT - Specific variable to search
Example: fre cmor find -r cmip6-cmor-tables/Tables/ -v sos

`varlist`

Generates variable list from netCDF files in a directory
Minimal Syntax: fre cmor varlist -d [dir_targ] -o [output_file]
Required Options:
- -d, --dir_targ TEXT - Target directory
- -o, --output_variable_list TEXT - Output file path
Optional:
- -t, --mip_table TEXT - MIP table JSON file to filter variables against
Example: fre cmor varlist -d ocean_data/ -o varlist.json

`config`

Generates a CMOR YAML configuration file by scanning a post-processing directory tree and cross-referencing against MIP tables
Creates per-component variable list JSON files and the structured YAML that fre cmor yaml consumes
Minimal Syntax: fre cmor config -p [pp_dir] -t [mip_tables_dir] -m [mip_era] -e [exp_config] -o [output_yaml] -d [output_dir] -l [varlist_dir]
Required Options:
- -p, --pp_dir TEXT - Root post-processing directory
- -t, --mip_tables_dir TEXT - Directory containing MIP table JSON files
- -m, --mip_era TEXT - MIP era identifier (e.g. cmip6, cmip7)
- -e, --exp_config TEXT - Path to experiment configuration JSON
- -o, --output_yaml TEXT - Path for the output CMOR YAML file
- -d, --output_dir TEXT - Root output directory for CMORized data
- -l, --varlist_dir TEXT - Directory for per-component variable list files
Optional:
- --freq TEXT - Temporal frequency (default: monthly)
- --chunk TEXT - Time chunk string (default: 5yr)
- --grid TEXT - Grid label anchor name (default: g99)
- --overwrite - Overwrite existing variable list files
- --calendar TEXT - Calendar type (default: noleap)
Example: fre cmor config -p /path/to/pp -t /path/to/tables -m cmip7 -e exp_config.json -o cmor.yaml -d /path/to/output -l /path/to/varlists

For comprehensive documentation, see CMORize Postprocessed Output.

`fre make`

`checkout`

fre make checkout-script [options]

Purpose: Creates the checkout script and can check out source code (with execute option)
Options:
- -y, --yamlfile [experiment yaml] (required)
- -p, --platform [platform] (required)
- -t, --target [target] (required)
- -j, --njobs [number of jobs to run simultaneously]
- -npc, --no-parallel-checkout (for container build)
- -e, --execute

`makefile`

fre make makefile [options]

Purpose: Creates the makefile
Options:
- -y, --yamlfile [experiment yaml] (required)
- -p, --platform [platform] (required)
- -t, --target [target] (required)

`compile`

fre make compile-script [options]

Purpose: Creates the compile script and compiles the model (with execute option)
Options:
- -y, --yamlfile [experiment yaml] (required)
- -p, --platform [platform] (required)
- -t, --target [target] (required)
- -j, --njobs [number of jobs to run simultaneously]
- -n, --parallel [number of concurrent module compiles]
- -e, --execute

`dockerfile`

fre make dockerfile [options]

Purpose: Creates the dockerfile and creates the container (with execute option)
With the creation of the dockerfile, the Makefile, checkout script, and any other necessary script is copied into the container from a temporary location
Options:
- -y, --yamlfile [experiment yaml] (required)
- -p, --platform [platform] (required)
- -t, --target [target] (required)

`all`

fre make all [options]

Purpose: Create the checkout script, Makefile, compile script, and dockerfile (platform dependent) for the compilation of the model
Options:
- -y, --yamlfile [experiment yaml] (required)
- -p, --platform [platform] (required)
- -t, --target [target] (required)
- -npc, --no-parallel-checkout (for container build)
- -j, --njobs [number of jobs to run simultaneously]
- -n, --parallel [number of concurrent module compiles]

`fre pp`

The identifiers for the Cylc workflows are built from 3 pieces, separated with two underscores:

$(experiment)__$(platform)__$(target)

For the tools called by `fre pp $tool`, these usually correspond to the -e, -p. -t arguments.

The experiment is a short description of the model configuration that was run. It corresponds to one of the experiment names that are displayed by `fre list exps -y $yamlfile`, and the workflow tools with throw an error if -e does not match one of the experiment names in the yaml.

The platform is the combination of the platform upon which the model was run and the compiler with which it was built. Bronx added ‘gfdl’ to the platform for all postprocessing experiments, since those are run on the postprocessing nodes at the Geophysical Fluid Dynamics Lab.

The target is the option that was used for the model compiler. There are currently 5-ish options we expect to see:

prod - Used for production experiments. Runs most slowly, but with highest precision.
repro - Used when reproducing past results for regression tests. Slower than debug, faster than production.
debug - Used when debugging. Fastest runtime.
openmp - Used when compiled and linked with OpenMP. Can be put onto the end of another target (e.g. prod-openmp)
hdf5 - Added directives to generate HDF5-based netCDF files. Can be put onto the end of another target (e.g. repro-hdf5)

For more information see: https://sites.google.com/noaa.gov/oar-gfdl-msd-docs/fre-documentation/fre-documentation/compiling-an-experiment#h.52hq1w2v4klg

`all`

Executes the fre postprocessing steps in order (fre pp configure-yaml, fre pp checkout, fre pp validate, fre pp install, fre pp run, trigger and status)
Minimal syntax: fre pp all -e experiment_name -p platform_name -T target_name -c config_file [ -b [branch] -t [time] ]
Module(s) needed: n/a
Example: fre pp all -e c96L65_am5f4b4r0_amip -p ncrc5.intel23 -T prod -c ./am5.yaml -b main

`configure-yaml`

Postprocessing yaml configuration
Minimal Syntax: fre pp configure-yaml -y [user-edit yaml file] -e experiment_name -p platform_name -t target_name
Module(s) needed: n/a
Example: fre pp configure-yaml -y /home/$user/pp/ue2/user-edits/edits.yaml -e c96L65_am5f4b4r0_amip -p gfdl.ncrc5-deploy -t prod-openmp

`checkout`

Checkout template file and clone https://github.com/NOAA-GFDL/fre-workflows.git git repository to ~/cylc_src
Minimal Syntax: fre pp checkout -e experiment_name -p platform_name -t target_name [ -b [branch] ]
Module(s) needed: n/a
Example: fre pp checkout -e c96L65_am5f4b4r0_amip -p gfdl.ncrc5-deploy -t prod-openmp

`install`

Installs an experiment configuration into ~/cylc_run/$(experiment)_$(platform)_$(target)
Minimal Syntax: fre pp install -e experiment_name -p platform_name -t target_name
Module(s) needed: n/a
Example: fre pp install -e c96L65_am5f4b4r0_amip -p gfdl.ncrc5-deploy -t prod-openmp

`run`

Submits postprocessing job
Minimal Syntax: fre pp run -e experiment_name -p platform_name -t target_name [ --pause --no-wait ]
Module(s) needed: n/a
Example: fre pp run -e c96L65_am5f4b4r0_amip -p gfdl.ncrc5-deploy -t prod-openmp

`trigger`

Starts postprocessing history files belonging to an experiment that represent a specific chunk of time (e.g. 1979-1981)
Minimal Syntax: fre pp trigger -e experiment_name -p platform_name -T target_name -t time
Module(s) needed: n/a
Example: fre pp trigger -e c96L65_am5f4b4r0_amip -p gfdl.ncrc5-deploy -T prod-openmp -t 00010101

`split-netcdf`

Splits single netcdf file into separate netcdf files with one data variable per file
Minimal Syntax: fre pp split-netcdf -f netcdf_file -o output_directory -v var_1,var_2...var_n
Module(s) needed: n/a
Example: fre pp split-netcdf -f 19790101.atmos_tracer.tile6.nc -o output/ -v tasmax,tasmin

`split-netcdf-wrapper`

Given a directory structure with netcdf files, calls split-netcdf on individual netcdf files
If split-netcdf-wrapper is called with the the argument –split-all-vars, then -c and -y are not required.
Minimal Syntax: fre pp split-netcdf-wrapper -i input/ -o output/ -s history_source --split-all-vars [--use-subdirs]
Otherwise, -c and -y requirements are necessary.
Minimal Syntax: fre pp split-netcdf-wrapper -i input/ -o output/ -s history_source -c yaml_component -y yamlfile.yml [--use-subdirs]
Module(s) needed: n/a
Example 1: fre pp split-netcdf-wrapper -i input/ -o output/ --split-all-vars --use-subdirs
Example 2: fre pp split-netcdf-wrapper -i input/ -o output/ -c ocean_cobalt_tracers_year_z -y ./ESM4.5v06_b08_cobv3_mekep_piC.yaml --use-subdirs

`nccheck`

Verify netCDF file contains expected number of time records
Minimal Syntax: fre pp nccheck -f [netCDF file path] -n [number of expected time records]
Module(s) needed: n/a
Example: fre pp nccheck -f /home/$user/some_netcdf_file -n 10

`histval`

Run nccheck over all files found in diag manifest
Minimal Syntax: fre pp histval --history [path to directory containing history files] --date_string [date_string] [ --warn ]
Module(s) needed: n/a
Example: fre pp histval --history /some_path/dir_with_history_files/ --date_string 00010101

`ppval`

Run nccheck over postprocessed time-series files
Minimal Syntax: fre pp ppval --path [path to file]
Module(s) needed: n/a
Example: fre pp ppval --path /some_path/example.200001-200412.nc

`fre yamltools`

`combine-yamls`

fre yamltools combine-yamls [options]

Purpose: Creates a combined yaml file for either compilation or post-processing.
If --use compile, the model yaml is combined with the compile and platforms yaml. If --use pp, the model yaml is combined with post-processing and analysis yamls. If --use cmor, the model yaml is combined with cmor yamls.
Options:
- -y, --yamlfile [experiment yaml] (required)
- -e, --experiment [experiment name]
- -p, --platform [platform] (required)
- -t, --target [target] (required)
- --use [compile|pp|cmor] (required)

`fre list`

`exps`

fre list exps [options]

Purpose: Lists available post-processing experiments included in the yaml configurations
Options:
- -y, --yamlfile [experiment yaml]

`platforms`

fre list platforms [options]

Purpose: Lists available platforms included in the yaml configurations
Options:
- -y, --yamlfile [experiment yaml]

`pp-components`

fre list pp-components [options]

Purpose: Lists components that have the postprocess_on key set to True in the postprocessing yaml configurations
Options:
- -y, --yamlfile [experiment yaml]
- -e, --experiment [experiment to be post-processed]

Tools

fre

arguments

fre app

fre catalog

build

validate

fre cmor

CMOR Subcommands Overview

run

yaml

find

varlist

config

fre make

checkout

makefile

compile

dockerfile

all

fre pp

all

configure-yaml

checkout

install

run

trigger

split-netcdf

split-netcdf-wrapper

nccheck

histval

ppval

fre yamltools

combine-yamls

fre list

exps

platforms

pp-components

`fre`

`fre app`

`fre catalog`

`build`

`validate`

`fre cmor`

`run`

`yaml`

`find`

`varlist`

`config`

`fre make`

`checkout`

`makefile`

`compile`

`dockerfile`

`all`

`fre pp`

`all`

`configure-yaml`

`checkout`

`install`

`run`

`trigger`

`split-netcdf`

`split-netcdf-wrapper`

`nccheck`

`histval`

`ppval`

`fre yamltools`

`combine-yamls`

`fre list`

`exps`

`platforms`

`pp-components`