GCAM tool (gt
)¶
The gt script unifies GCAM workflow managment functionality into a single script with sub-commands. Generic sub-commands are implemented directly by the pygcam library. Project-specific features can be added via plugins.
The sub-commands support all the major workflow setups, including
- Modify XML files and configuration.xml to set up a modeling experiment (See the setup sub-command and GCAM XML-Setup for more information.)
- Run GCAM in an automatically-created workspace, allowing multiple instances of GCAM to run simultaneously, e.g., on parallel computing systems (See the gcam sub-command.)
- Execute batch queries against the XML database to extract GCAM results, with on-the-fly regionalization based on a simple region-mapping file. (See the query sub-command.)
- Compute differences between policy and baseline scenarios, including linear annualization of values between time-steps, and (See the diff sub-command.)
- Plot results, with flexible control of figure features including title, axis labels, scale, and so on. (See the chart sub-command.)
- Manage (create, delete, rename, run commands in) automatically-created workspaces. (See the sandbox sub-command.)
In addition, the run sub-command allows workflow steps to be
defined in an XML file so that individual or groups of steps can be executed for one
or more scenarios. The run
sub-command supports direct invocation of other
workflow steps as well as running arbitrary programs of the user’s choosing.
Finally, gt allows all project steps to be run on a compute node in a High-Performance Computing environment by specifying ‘-b’ or ‘–batch’’ on the command-line. (Note that this is not available on Mac OS X or Windows.)
For example, the command:
gt -b -P MyProject run -S MyScenario
runs all steps for scenario MyScenario
in the project MyProject
by
queuing a batch job on the default queue. Arguments to gt
allow
the user to set various resource requirements and to select the queue to use.
The command to run to queue the batch job is taken from the configuration
file parameter GCAM.BatchCommand
. Example batch commands for the SLURM
and PBS job management systems are provided in variables GCAM.QueueSLURM
and GCAM.QueuePBS
, respectively.
Command-line usage is described below. Note that some command-line (e.g., batch-related) options must precede the sub-command, whereas sub-command specific options must follow it.
Usage¶
usage: gt [-h] [-b] [-B] [-e ENVIROVARS] [-j JOBNAME] [-l level] [-L LOGFILE]
[-m MINUTES] [--mcs {trial,gensim}] [-P name] [-q QUEUENAME]
[-r RESOURCES] [--set name=value] [-v] [--version]
{chart,config,diff,gcam,new,protect,query,run,sandbox,setup} ...
- Options:
-b=False, --batch=False Run the commands by submitting a batch job using the command given by config variable GCAM.BatchCommand. (Linux only) -B=False, --showBatch=False Show the batch command to be run, but don’t run it. (Linux only) -e, --enviroVars Comma-delimited list of environment variable assignments to pass to queued batch job, e.g., -E “FOO=1,BAR=2”. (Linux only) -j=gt, --jobName=gt Specify a name for the queued batch job. Default is “gt”. (Linux only) -l, --logLevel Sets the log level of the program.
Possible choices: notset, debug, info, warning, error, critical
-L, --logFile Sets the name of a log file for batch runs. Default is “gt-$j.out” where “$j” is replaced by “%%j”, which (in SLURM) is the jobid. If the argument is not an absolute pathname, it is treated as relative to the value of GCAM.LogDir. -m, --minutes Set the number of minutes to allocate for the queued batch job. Overrides config parameter GCAM.Minutes. (Linux only) --mcs Used only when running gcamtool from gcammcs.
Possible choices: trial, gensim
-P, --projectName The project name (the config file section to read from), which defaults to the value of config variable GCAM.DefaultProject -q, --queueName Specify the name of the queue to which to submit the batch job. Default is given by config variable GCAM.DefaultQueue. (Linux only) -r=, --resources= Specify resources for the queued batch command. Can be a comma-delimited list of assignments of the form NAME=value, e.g., -r ‘pvmem=6GB’. (Linux only) --set=[] Assign a value to override a configuration file parameter. For example, to set batch commands to start after a prior job of the same name completes, use –set “GCAM.OtherBatchArgs=-d singleton”. Enclose the argument in quotes if it contains spaces or other characters that would confuse the shell. Use multiple –set flags and arguments to set multiple variables. -v=False, --verbose=False Show diagnostic output --version show program’s version number and exit - Sub-commands:
- chart
The
chart
sub-command generates plots from GCAM-style ”.csv” files. Two types of plots are currently supported: (i) stacked bar plots based on summing values over all years (with optional interpolation of annual values), by the given ‘indexCol’ (default is ‘region’), and (ii) stacked bar plots by year for some data column, where the data are grouped by and summed across elements with the indicated ‘indexCol’. The first option is indicated by using the ‘-S’ (‘–sumYears’) option. Numerous options allow the appearance to be customized.usage: gt chart [-h] [-b] [-B] [-c COLUMNS] [-C CONSTRAINT] [-d OUTPUTDIR] [-D WORKINGDIR] [-e] [-f FROMFILE] [-F DIVISORFILE] [--format FORMAT] [-g] [-i] [-I INDEXCOL] [-k] [-l] [-L LABELCOLOR] [-m MULTIPLIER] [-M MULTIPLIERFILE] [-n NCOL] [-N SCENARIO] [--negate] [-o OUTFILE] [-O] [-p PALETTE] [-r ROTATION] [-R REFERENCE] [--region REGION] [-s SKIPROWS] [-S] [-t YEARSTEP] [-T TITLE] [--timeseries] [--transparent] [-u UNSTACKED] [-U UNSTACKEDREGION] [-v VALUECOL] [-V DIVISOR] [--version] [-x SUFFIX] [-X XLABEL] [-Y YLABEL] [-y YEARS] [--ymax YMAX] [--ymin YMIN] [-z] [--legendY LEGENDY] [--barWidth BARWIDTH] [csvFile]
- Positional arguments:
csvFile The file containing the data to plot. - Options:
-b=False, --box=False Draw a box around the plot. Default is no box. -B=False, --byRegion=False Generate one plot per region. Region names are read from the CSV file, so they reflect any regional aggregation produced by the query. -c=output, --columns=output Specify the column whose values identify the segments in the stacked bar chart. (These appear in the legend.) -C, --constraint Apply a constraint to limit the rows of data to plot. The constraint can be any constraint string that is valid for the DataFrame.query() method, e.g., -C ‘input == “biomass”’ -d=., --outputDir=. The directory into which to write image files. Default is ”.” -D=., --workingDir=. The directory to change to before performing any operations -e=False, --enumerate=False Prefix image filenames with sequential number for easy reference. Used only with –fromFile -f, --fromFile A file from which to read argument strings, one per line. These are read as if chartGCAM.py were called on each line individually, but avoiding the ~2 sec startup time for the bigger python packages. -F, --divisorFile A file containing a floating point value to divide data by before plotting. See also -V. --format Specify a format for the Y-axis. Possible values are ‘.’ for float, ‘,’ for int with commas, or any format recognized by print, e.g., “%%.2f” to Y values as floats with 2 decimal places. -g=False, --ygrid=False Show light grey horizontal lines at the major Y-axis ticks. Default is no grid. -i=False, --interpolate=False Interpolate (linearly) annual values between timesteps. -I=region, --indexCol=region A column to use as the index column, or blank for None. Default is “region”. -k=False, --yticks=False Show tick marks on Y-axis. Default is no tick marks. -l=False, --label=False Add text along the right side of the figure showing the filename. -L, --labelColor Color for the text label, which defaults to lightgrey. Some users may prefer “black”, for example. (Implies -l) -m, --multiplier A value to multiply data by before generating the plot. The argument can be a floating point number or the name of any variable in pygcam.unitConversion.py. For example, “-m 3.667” and “-m C_to_CO2” are equivalent, and effectively convert values from Tg C to Tg CO2. Be sure to adjust the Y axis label. See also -M. -M, --multiplierFile A file containing a floating point value to multiply data by before plotting. See also -m. -n=5, --ncol=5 The number of columns with which to display the legend. Default is 5. -N=, --scenario= When using the ‘–fromFile’ option, this argument is used to specify one or more scenario names (delimited by commas if more than one). These are substituted into each line read from the file as the value for “{scenario}” wherever it appears on each line read from the ‘fromFile’. --negate=False Multiply data by -1 before plotting, which can make interpretation of some figures more intuitive. The string “-negated” is added to the file label, displayed if the “-l” or “-L” flag is specified. -o=, --outFile= The name of the image file to create. Format is determined from filename extension. All common formats are allowed, e.g., png, pdf, tif, and gif. Try it; it probably works. Default is the name of the data file substituting ”.png” for ”.csv” -O=False, --open=False Open the plot file after generating it. -p, --palette The name of a color palette to use. Some good options include hls, husl, and Paired. See http://stanford.edu/~mwaskom/software/seaborn/tutorial/color_palettes.html -r=90, --rotation=90 Set the rotation angle for X-axis labels. Defaults to 90 degrees (vertical). Use 0 for horizontal labels. -R=reference, --reference=reference When using the ‘–fromFile’ option, this argument is used to specify the name of the reference scenario. The “other” scenario is given using the “-N” option. These are substituted into each line read from the file as the value for “{scenario}” and “{reference}” (without the quotes) wherever they appear on each line read from the ‘fromFile’. Defaults to “reference” --region Plot values only for the given region. -s=1, --skiprows=1 The number of rows of the CSV file to skip before reading the data (starting with a header row with column names.) Default is 1, which works for GCAM batch query output. -S=False, --sumYears=False Sum across the time horizon, typically by region. This results in a stacked bar plot. When not summed over years (the default) a stacked area plot is generated showing values grouped and summed by indexCol (-I) and presented by year. -t=5, --yearStep=5 The spacing of year labels on X-axis for time-series plots. Defaults to 5. -T=, --title= Adds a title to the plot. Default is no title. The string can have LaTeX math language in it, e.g., ‘CO$_2$’ causes the 2 to be subscripted, and ‘MJ$^{-1}$’ results in “MJ” with a superscripted exponent of -1. The string ‘$\Delta$’ results in a capital Greek delta. See LaTeX documentation for more options. Be sure to enclose the title in single quotes so the “$” is not (mis)interpreted by the shell. --timeseries=False Plot the data as a time series. --transparent=False Save the plot with a transparent background. (Default is white.) -u, --unstacked Draw an unstacked bar plot for the column given as an argument to this option, showing three groups of bars: the region, all other regions, and the total. -U, --unstackedRegion The region to plot separately from Rest of World in an unstacked plot. Ignored if –byRegion is specified, in which case a plot is created for all regions. -v, --valueCol Identify a single column (e.g., a year) to plot for bar plots. -V=0.0, --divisor=0.0 A value to divide year column values by before plotting. The argument can be a floating point number or the name of any variable in pygcam.unitConversion.py. See also -F. --version show program’s version number and exit -x, --suffix A suffix to append to the basename of the input csv file to create the name for the output file. For example, if processing my_data.csv, indicating -x ‘-by-region.pdf’ results in an output file named my_data-by-region.pdf. -X=, --xlabel= Defines a label for the X-axis; defaults to blank. LaTeX math language is supported. (See the -T flag for more info.) -Y=EJ, --ylabel=EJ Label for the Y-axis; defaults to “EJ”. LaTeX math language is supported. (See the -T flag for more info.) -y=, --years= Takes a parameter of the form XXXX-YYYY, indicating start and end years of interest. Data for all other years are dropped. --ymax Set the scale of a figure by indicating the value to show as the maximum Y value. (By default, scale is set according to the data.) --ymin Set the scale of a figure by indicating the value (given as abs(value), but used as -value) to show as the minimum Y value -z=False, --zeroLine=False Whether to show a line at Y=0 --legendY The Y position of the legend. Useful for fixing poorly formatted figures. Note that to pass a negative value, use the syntax –legendY=”-xxx.xxx”, otherwise the hyphen is interpreted as indicating a command-line argument. --barWidth=0.5 The relative width of bars. Helpful when plotting only 1 or 2 bar, so they aren’t obnoxiously wide. Default is 0.5
- config
The config command list the values of configuration variables from ~/.pygcam.cfg. With no arguments, it displays the values of all variables for the default project. Use the
-d
flag to show only values from the[DEFAULT]
section.If an argument
name
is provided, it is treated as a substring pattern, unless the-x
flag is given (see below). All configuration variables containing the give name are displayed with their values. The match is case-insensitive.If the
-x
or--exact
flag is specified, the argument is treated as an exact variable name (case-sensitive) and only the value is printed. This is useful mainly for scripting. For general use the substring matching is more convenient.Examples:
$ gt config project [MyProject] GCAM.DefaultProject = MyProject GCAM.ProjectRoot = /Users/rjp/bitbucket/myProject GCAM.ProjectXmlFile = /Users/rjp/bitbucket/myProject/etc/project.xml $ gt config -x GCAM.DefaultProject MyProject $ gt config sand [MyProject] GCAM.SandboxRoot = /Users/rjp/ws/myProject $ gt config sand -d [DEFAULT] GCAM.SandboxRoot = /Users/rjp/ws
usage: gt config [-h] [-d] [-e] [-x] [-t] [--version] [name]
- Positional arguments:
name Show the names and values of all parameters whose name contains the given value. The match is case-insensitive. If not specified, all variable values are shown. - Options:
-d=False, --useDefault=False Indicates to operate on the DEFAULT section rather than the project section. -e=False, --edit=False Edit the configuration file. The command given by the value of config variable GCAM.TextEditor is run with the .pygcam.cfg file as an argument. -x=False, --exact=False Treat the text not as a substring to match, but as the name of a specific variable. Match is case-sensitive. Prints only the value. -t=False, --test=False Test the settings in the configuration file to ensure that the basic setup is ok, i.e., required parameters have values that make sense. If specified, no variables are displayed. --version show program’s version number and exit
- diff
The
diff
sub-command script computes the differences between results from two or more CSV files generated from batch queries run on a GCAM database, saving the results in either a CSV or XLSX file, according to the extension given to the output file. If not provided, the output filename defaults to differences.csv.If multiple otherFiles are given (i.e., the referenceFile plus 2 or more other files named on the command-line), the resulting CSV file will contain one difference matrix for each otherFile, with a label indicating which pair of files were used to produce each result.
When the output file is in XLSX format, each result is written to a separate worksheet. If the -c flag is specified, no differences are computed; rather, the .csv file contents are combined into a single .xlsx file.
usage: gt diff [-h] [-D WORKINGDIR] [-g GROUPSUM] [-i] [-o OUTFILE] [-c] [-q QUERYFILE] [-r REWRITESETSFILE] [-S] [-s SKIPROWS] [--version] [-y YEARS] [-Y STARTYEAR] csvFiles [csvFiles ...]
- Positional arguments:
csvFiles The files to process. For difference operations, the first file is treated as the reference file whose time-series data is subtracted from that of each other file. If missing, ”.csv” suffixes are added to all arguments (the ”.csv” is optional). - Options:
-D=., --workingDir=. The directory to change to before performing any operations -g=, --groupSum= Group data for each timestep (or interpolated annual values) by the given column, and sum all members of each group to produce a timeseries for each group. Takes precedence over the simpler “-S” (“–sum”) option. -i=False, --interpolate=False Interpolate (linearly) annual values between timesteps. -o=differences.csv, --outFile=differences.csv The name of the ”.csv” or ”.xlsx” file containing the differences between each scenario and the reference. Default is “differences.csv”. -c=False, --convertOnly=False Convert the given CSV files into an Excel workbook, one sheet per CSV file. -q=, --queryFile= If the extension is ”.xml” (case insensitive), the argument must be an XML file holding a list of queries to run, with optional mappings specified to rewrite output. This file has the same structure as the <queries> element in project.xml. If the file doesn’t end in ”.xml”, it must be a text file listing the names of queries to process, one per line. NOTE: When –queryFile is specified, the two positional arguments are required: the names of the baseline and policy scenarios, in that order. -r, --rewriteSetsFile An XML file defining query maps by name (default taken from config parameter “GCAM.RewriteSetsFile”) -S=False, --sum=False Sum all timestep (or interpolated annual values) to produce a single time-series. -s=1, --skiprows=1 The number of rows to skip. Default is 1, which works for GCAM batch query output. Use -s0 for outFile.csv --version show program’s version number and exit -y=, --years= Takes a parameter of the form XXXX-YYYY, indicating start and end years of interest. Other years are dropped (except for annual outputs.) -Y=0, --startYear=0 The year at which to begin interpolation
- gcam
The
gcam
sub-command runs the GCAM executable on the designated configuration file, scenario, or workspace. Typical use (e.g., from aproject.xml
file) would be to run GCAM by referencing a directory named the same as a scenario, holding a file calledconfig.xml
, as is generated by thesetup
sub-command. (See GCAM XML-Setup.)If a workspace is specified on the command-line, it is used. Otherwise, if a scenario is specified, the workspace defined by {GCAM.SandboxDir}/{scenario} is used. If neither workspace nor scenario are defined, the value of config variable
GCAM.RefWorkspace
is used, i.e., GCAM is run in the reference workspace.If the workspace doesn’t exist, it is created based on the reference GCAM workspace, defined by the configuration variable
GCAM.RefWorkspace
. By default, read-only directories (e.g., input and libs) are symbolically linked from the new workspace to the reference one. (See the new sub-command for more information on the creation of workspaces.)Directories into which GCAM writes results (e.g., output and exe) are created in the new workspace, but read-only files within exe (e.g., the GCAM executable) are symbolically linked (with the same caveat for Windows users.)
Usage example:
gt gcam -S ~/MyProject/scenarios -s MyScenario -w ~/sandboxes/MyProject/MyScenario
would run the scenario
MyScenario
in the newly created sandbox (workspace)~/sandboxes/MyProject/MyScenario
using the configuration file~/MyProject/scenarios/MyScenario/config.xml
.usage: gt gcam [-h] [-C CONFIGFILE] [-f] [-g GROUPDIR] [-n] [-r REFWORKSPACE] [-s SCENARIO] [-S SCENARIOSDIR] [--version] [-w WORKSPACE] [-W]
- Options:
-C, --configFile Specify the one or more GCAM configuration filenames, separated by commas. If multiple configuration files are given, the are run in succession in the same “job” on the cluster. -f=False, --forceCreate=False Re-create the workspace, even if it already exists. -g=, --groupDir= The scenario group directory name, if any. -n=False, --noRun=False Don’t run GCAM; just print the command that would be run. -r, --refWorkspace The reference workspace to use to create the new sandbox. This is used only if the indicated or implied workspace doesn’t exist. Defaults to the value of GCAM.RefWorkspace. -s=, --scenario= The scenario to run. -S=, --scenariosDir= Specify the directory holding scenario files. Default is the value of config variable GCAM.ScenariosDir, if set, otherwise it’s the current directory. --version show program’s version number and exit -w, --workspace Specify the path to the GCAM workspace to use. If it doesn’t exist, the named workspace will be created. If not specified on the command-line, the path constructed as {GCAM.SandboxDir}/{scenario} is used. -W=False, --noWrapper=False Do not run gcam within a wrapper that detects errors as early as possible and terminates the model run. By default, the wrapper is used.
- new
Create the directory structure and basic files required for a new pygcam project. If a directory is specified with the
-r
flag, the project is created with the given name in that directory; otherwise the project is created in the directory identified by the config variableGCAM.ProjectRoot
.This sub-command creates examples of
xmlsrc/scenarios.py
,etc/protection.xml
,etc/project.xml
,etc/queries.xml
,etc/rewriteSets.xml
, andetc/scenarioSetup.xml
that can be edited to fit the needs of your project. See the fileInstructions.txt
that is created in the new “etc” directory.If the
-c
flag is given, a basic entry for the new project is added to the users configuration file,$HOME/.pygcam.cfg
. Before modifying the config file, a backup is created in$HOME/.pygcam.cfg~
. For example, the commandgt new -c foo
generates and entry like this:
[foo] # Added by "new" sub-command Mon Aug 29 15:00:19 2016 GCAM.ProjectDir = %(GCAM.ProjectRoot)s/foo #GCAM.LandProtectionXmlFile = %(GCAM.ProjectDir)s/etc/protection.xml #GCAM.RewriteSetsFile = %(GCAM.ProjectDir)s/etc/rewriteSets.xml #GCAM.ScenarioSetupFile = %(GCAM.ProjectDir)s/etc/scenarios.xml
Variables controlling linking vs copying
Several configuration variables control which files are copied versus linked (on systems on which the user has the ability to create symlinks; if not all files are copied):
GCAM.WorkspaceFilesToCopy
- A list of paths relative to GCAM.RefWorkspace that should be copied to the same relative location under {GCAM.SandboxDir}/Workspace.
GCAM.WorkspaceFilesToLink
- A list of paths relative to GCAM.RefWorkspace that should be symlinked to same relative location under {GCAM.SandboxDir}/Workspace.
GCAM.SandboxFilesToCopy
- A list of paths relative to {GCAM.SandboxDir}/Workspace that should be copied to the same relative location in the current scenario directory.
GCAM.SandboxFilesToLink
- A list of paths relative to {GCAM.SandboxDir}/Workspace that should be symlinked to the same relative location in the current scenario directory.
GCAM.OtherFilesToCopy
- This variable is provided to allow the user to specify additional files to copy. Its value is appended to both GCAM.WorkspaceFilesToCopy and GCAM.SandboxFilesToCopy.
GCAM.OtherFilesToLink
- This variable is provided to allow the user to specify additional files to symlink. Its value is appended to both GCAM.WorkspaceFilesToLink and GCAM.SandboxFilesToLink.
Note that when running on Windows, if pygcam detects that the user does not have permission to create symlinks, it sets the configuration variable
GCAM.CopyAllFiles
toTrue
, resulting in copies of the reference files rather than symbolic links.usage: gt new [-h] [-c] [--overwrite] [-r PATH] [--version] name
- Positional arguments:
name Create the structure for the named project, and copy example XML files into the “etc” directory. - Options:
-c=False, --addToConfig=False Add a section for the new project to $HOME/.pygcam.cfg after making a backup of the file in $HOME/.pygcam.cfg~ --overwrite=False If files that are to be copied to the project directory exist, overwrite them. By default, existing files are not overwritten. -r, --projectRoot The directory in which to create a subdirectory for the named project. Default is the value of config variable GCAM.ProjectRoot --version show program’s version number and exit
- protect
Generate versions of GCAM’s land_input XML files that protect a given fraction of land of the given land types in the given regions by subtracting the required land area from the “managed” land classes, thereby removing them from consideration in land allocations.
Simple protection scenarios can be specified on the command-line. More complex scenarios can be specified in an XML file, landProtection.xml.
Examples:
# Create and modify copies of the reference land files, renaming them with # "prot\_" prefix. Protect 80% of the "UnmanagedForest" and "UnmanagedPasture" # land classes in the specified regions only. CLASSES=UnmanagedForest,UnmanagedPasture REGIONS='Australia_NZ,Canada,EU-12,EU-15,Japan,Middle East,Taiwan,USA' OUTDIR="$HOME/tmp/xml" gt protect -f 0.8 "$INFILES" -l "$CLASSES" -r "$REGIONS" -o "$OUTDIR" -t 'prot_{filename}'
# Run the land protection scenario "s1", described in the file ``$HOME/protect.xml``, # placing the results in the directory ``$HOME/ws/workspace1`` gt protect -s s1 -S "$HOME/protect.xml" -w "$HOME/ws/workspace1"
usage: gt protect [-h] [-b] [-f FRACTION] [--inPlace] [-l LANDCLASSES] [-m] [-o OUTDIR] [-O] [-t TEMPLATE] [-r REGIONS] [-s SCENARIO] [-S SCENARIOFILE] [-V] [-w WORKSPACE]
- Options:
-b=False, --backup=False Make a copy of the output file, if it exists (with an added ~ after filename) before writing new output. This option is ignored if a scenario file is specified. -f, --fraction The fraction of land in the given land classes to protect. (Required, unless a scenario file is specified, in which case this option is ignored.) --inPlace=False Edit the file in place. This must be given explicitly, to avoid overwriting files by mistake. -l, --landClasses The land class or classes to protect in the given regions. Multiple, comma-delimited land types can be given in a single argument, or the -l flag can be repeated to indicate additional land classes. By default, all unmanaged land classes are protected. Allowed land classes are [‘UnmanagedPasture’, ‘UnmanagedForest’, ‘Shrubland’, ‘Grassland’]. This option is ignored if a scenario file is specified. -m=False, --mkdir=False Make the output dir if necessary. -o=., --outDir=. The directory into which to write the modified files. Default is current directory. -O=False, --otherArable=False Include OtherArableLand in the list of default land classes to protect. This flag is ignored if the -l (–landClasses) argument is used. -t=prot_{fraction}_{filename}, --template=prot_{fraction}_{filename} Specify a template to use for output filenames. The keywords {fraction}, {filename}, {regions}, and {classes} (with surrounding curly braces) are replaced by the following values and used to form the name of the output files, written to the given output directory. fraction: 100 times the given fraction (i.e., int(fraction * 100)); filename: the name of the input file being processed (e.g., land_input_2.xml or land_input_3.xml); basename: the portion of the input filename prior to the extension (i.e., before ‘.xml’); regions: the given regions, separated by ‘-‘, or the word ‘global’ if no regions are specified; classes: the given land classes, separated by ‘-‘, or the word ‘unmanaged’ if no land classes are specified. The default pattern is “prot_{fraction}_{filename}”. This option is ignored if a scenario file is specified. -r, --regions The region or regions for which to protect land. Multiple, comma-delimited regions can be given in a single argument, or the -r flag can be repeated to indicate additional regions. By default, all regions are protected. This option is ignored if a scenario file is specified. -s, --scenario The name of a land-protection scenario defined in the file given by the –scenarioFile argument or it’s default value. -S, --scenarioFile An XML file defining land-protection scenarios. Default is the value of configuration file parameter GCAM.LandProtectionXmlFile. -V, --version show program’s version number and exit -w, --workspace Specify the path to the GCAM workspace to use. The files in {workspace}/input/gcam-data-system/xml/aglu-xml/land_input_{2,3}.xml are used as inputs. Default is value of configuration parameter GCAM.RefWorkspace.
- query
Run one or more GCAM database queries by generating and running the named XML queries. The results are placed in a file in the specified output directory with a name composed of the basename of the XML query file plus the scenario name. For example,
gt query -o. -s MyReference,MyPolicyCase liquids-by-region
would run the
liquids-by-region
query on two scenarios, MyReference and MyPolicyCase. Query results will be stored in the files./liquids-by-region-MyReference.csv
and./liquids-by-region-MyPolicyCase.csv
.The named queries are located using the value of config variable
GCAM.QueryPath
, which can be overridden with the-Q
argument. The QueryPath consists of one or more colon-delimited (on Unix) or semicolon-delimited (on Windows) elements that can identify directories or XML files. The elements of QueryPath are searched in order until the named query is found. If a path element is a directory, the filename composed of the query + ‘.xml’ is sought in that directory. If the path element is an XML file, a query with a title matching the query name (first literally, then by replacing'_'
and'-'
characters with spaces) is sought. Note that query names are case-sensitive.usage: gt query [-h] [-d XMLDB] [-D] [-g GROUPDIR] [-n] [-o OUTPUTDIR] [-p] [-q QUERYXMLFILE] [-Q QUERYPATH] [-r REGIONS] [-R REGIONMAP] [-s SCENARIO] [-S REWRITESETSFILE] [--version] [-w WORKSPACE] [queryName [queryName ...]]
- Positional arguments:
queryName A file or files, each holding an XML query to run. (The ”.xml” suffix will be added if needed.) If an argument is preceded by the “@” sign, it is read and its contents substituted as the values for this argument. That means you can store queries to run in a file (one per line) and just reference the file by preceding the filename argument with “@”. - Options:
-d, --xmldb The XML database to query (default is computed as {GCAM.SandboxDir}/output/{GCAM.DbFile}. Overrides the -w flag. -D=False, --noDelete=False Don’t delete any temporary file created by extracting a query from a query file. Used mainly for debugging. -g=, --groupDir= The scenario group directory name, if any. Used with to compute default for –workspace argument. -n=False, --noRun=False Show the command to be run, but don’t run it -o, --outputDir Where to output the result (default taken from config parameter “GCAM.OutputDir”) -p=False, --prequery=False Generate the XMLDBDriver.properties file and associated batch file to be run by GCAM when GCAM.BatchMultipleQueries or GCAM.InMemoryDatabase are True. -q, --queryXmlFile An XML file holding a list of queries to run, with optional mappings specified to rewrite output. This file has the same structure as the <queries> element in project.xml. -Q, --queryPath A semicolon-delimited list of directories or filenames to look in to find query files. Defaults to value of config parameter GCAM.QueryPath -r, --regions A comma-separated list of regions on which to run queries found in query files structured like Main_Queries.xml. If not specified, defaults to querying all 32 regions. -R, --regionMap A file containing tab-separated pairs of names, the first being a GCAM region and the second being the name to map this region to. Lines starting with “#” are treated as comments. Lines without a tab character are also ignored. This arg overrides the value of config variable GCAM.RegionMapFile. -s=Reference, --scenario=Reference The scenario to run the query/queries for (default is “Reference”) Note that this must refers to a scenarios in the XML database. -S, --rewriteSetsFile An XML file defining query maps by name (default taken from config parameter “GCAM.RewriteSetsFile”) --version show program’s version number and exit -w=, --workspace= The workspace directory in which to find the XML database. Defaults computed as {GCAM.SandboxDir}/{groupDir}/{scenario}. Overridden by the -d flag.
- run
This sub-command reads instructions from the file project.xml, the location of which is taken from the user’s ~/.pygcam.cfg file. The workflow steps indicated in the XML file and command-line determine which commands to run.
Examples:
Run all steps for the default scenario group for project ‘Foo’:
gt -P Foo run
Run all steps for scenario group ‘test’ for project ‘Foo’, but only for scenarios ‘baseline’ and ‘policy-1’:
gt -P Foo run -g test -S baseline,policy1
or, equivalently:
gt -P Foo run --group test --scenario baseline --step policy1
Run only the ‘setup’ and ‘gcam’ steps for scenario ‘baseline’ in the default scenario group:
gt -P Foo run -s setup,gcam -S baseline,policy-1
Same as above, but queue a batch job to run these commands on the queue ‘short’:
gt -b -q short -P Foo run -s setup,gcam -S baseline,policy-1
Show the commands that would be executed for the above command, but don’t run them:
gt -P Foo run -s setup,gcam -S baseline,policy-1 -n
usage: gt run [-h] [-f PROJECTFILE] [-g GROUP] [-G] [-k SKIPSTEPS] [-K SKIPSCENARIOS] [-l] [-L] [-n] [-p PROJECT] [-q] [-s STEPS] [-S SCENARIOS] [--version] [--vars] [-x SANDBOXDIR]
- Options:
-f, --projectFile The XML file describing the project. If set, command-line argument takes precedence. Otherwise, value is taken from config file variable GCAM.ProjectXmlFile, if defined, otherwise the default is ‘./project.xml’. -g, --group The name of the scenario group to process. If not specified, the group with attribute default=”1” is processed. -G=False, --listGroups=False List the scenario groups defined in the project file and exit. -k, --skipStep Steps to skip. These must be names of steps defined in the project.xml file. Multiple steps can be given in a single (comma-delimited) argument, or the -k flag can be repeated to indicate additional steps. By default, all steps are run. -K, --skipScenario Scenarios to skip. Multiple scenarios can be given in a single (comma-delimited) argument, or the -K flag can be repeated to indicate additional scenarios. By default, all scenarios are run. -l=False, --listSteps=False List the steps defined for the given project and exit. Dynamic variables (created at run-time) are not displayed. -L=False, --listScenarios=False List the scenarios defined for the given project and exit. Dynamic variables (created at run-time) are not displayed. -n=False, --noRun=False Display the commands that would be run, but don’t run them. -p, --project The name of the project to run. -q=False, --noQuit=False Don’t quit if an error occurs when processing a scenario, just move on to processing the next scenario, if any. -s, --step The steps to run. These must be names of steps defined in the project.xml file. Multiple steps can be given in a single (comma-delimited) argument, or the -s flag can be repeated to indicate additional steps. By default, all steps are run. -S, --scenario Which of the scenarios defined for the given project should be run. Multiple scenarios can be given in a single (comma-delimited) argument, or the -S flag can be repeated to indicate additional scenarios. By default, all active scenarios are run. --version show program’s version number and exit --vars=False List variables and their values -x, --sandboxDir The directory in which to create the run-time sandbox workspace. Defaults to value of {GCAM.SandboxProjectDir}/{scenarioGroup}.
- sandbox
The
sandbox
sub-command allows you to create, delete, show the path of, or run a shell command in a workspace. If the--scenario
argument is given, the operation is performed on a scenario-specific workspace within a project directory. If--scenario
is not specified, the operation is performed on the project directory that contains individual scenario workspaces. Note that the gcam sub-command automatically creates workspaces as needed.N.B. You can run
sandbox
with the--path
option before performing any operations to be sure of the directory that will be operated on, or use the--noExecute
option to show the command that would be executed by--run
.usage: gt sandbox [-h] [--create] [--delete] [--recreate] [-g GROUPDIR] [-n] [-p] [-r RUN] [-s SCENARIO] [--version]
- Options:
--create=False Create the identified sandbox. If used with –delete, the deletion occurs first. --delete=False Delete the identified sandbox’ If used with –create, the deletion occurs first. --recreate=False Recreate the identified sandbox. Equivalent to using the –delete and –create options together. -g=, --groupDir= The name of the scenario group subdir -n=False, --noExecute=False Print the command that would be executed by –run, but don’t execute it. -p=False, --path=False Print the absolute path to the identified sandbox. -r, --run Run the given command in the identified sandbox. -s=, --scenario= The scenario for the computed sandbox root. --version show program’s version number and exit
- setup
The
setup
sub-command automates modification to copies of GCAM’s input XML files and construction of a corresponding configuration XML file. See GCAM XML-Setup for a detailed description.usage: gt setup [-h] [-b BASELINE] [-d] [-f] [-g GROUP] [-G] [-m MODULEPATH] [-M MODULESPEC] [-p period-or-year] [-r REFWORKSPACE] [-R RESULTSDIR] [-s SCENARIO] [-S SUBDIR] [--setupXml SETUPXML] [-u] [-x XMLSOURCEDIR] [-X XMLOUTPUTROOT] [-w WORKSPACE] [-y YEARS]
- Options:
-b, --baseline Identify the baseline the selected scenario is based on. Note: at least one of –baseline (-b) / –scenario (-s) must be used. -d=False, --dynamicOnly=False Generate only dynamic XML for dyn-xml: don’t create static XML. -f=False, --forceCreate=False Re-create the workspace, even if it already exists. -g, --group The scenario group to process. Defaults to the group labeled default=”1”. -G=False, --staticOnly=False Generate only static XML for local-xml: don’t create dynamic XML. -m, --modulePath The path to a scenario definition module. See -M flag for more info. -M, --moduleSpec The “dot spec” for the Python module holding the setup classes and a function called ‘scenarioMapper’ or a dictionary called ‘ClassMap’ which map scenario names to classes. If the function ‘scenarioMapper’ exists, it is used. If not, the ‘ClassMap’ is used. Default is “{xmlsrc}/subdir/scenarios.py” (if subdir is defined) or “{xmlsrc}/scenarios.py” (if subdir is undefined) under the current ProjectRoot. -p, --stop The number of the GCAM period or the year to stop after -r=, --refWorkspace= A reference workspace to use instead of the value of GCAM.RefWorkspace -R, --resultsDir The parent directory holding the GCAM output workspaces -s, --scenario Identify the scenario to run. Note: at least one of –baseline (-b) / –scenario (-s) must be used. -S=, --subdir= A sub-directory to use instead of scenario name --setupXml An XML scenario definition file. Overrides configuration variable GCAM.ScenarioSetupFile. -u=False, --useGroupDir=False Use the group name as a subdir below xmlsrc, local-xml, and dyn-xml -x, --xmlSourceDir The location of the xmlsrc directory. -X, --xmlOutputRoot The root directory into which to generate XML files. -w, --workspace The pathname of the workspace to operate on. -y=2015-2100, --years=2015-2100 Years to generate constraints for. Must be of the form XXXX-YYYY. Default is “2015-2100”
Extending gt using plug-ins¶
The gt script will load any python files whose name ends in
_plugin.py
, found in any of the directories indicated in the config
file variable GCAM.PluginPath
. The value of GCAM.PluginPath
must
be a sequence of directory names separated by colons (:
) on Unix-like
systems or by semi-colons (;
) on Windows.
See pygcam.subcommand for documentation of the plug-in API.