GCAM tool (gt)

The gt script unifies GCAM workflow managment functionality into a single script with sub-commands. Generic sub-commands are implemented directly by the pygcam library. Project-specific features can be added via plugins.

Note

Quick links to sub-commands: chart, config, diff, gcam, new, protect, query, run, setup, sandbox

The sub-commands support all the major workflow setups, including

  • Modify XML files and configuration.xml to set up a modeling experiment (See the setup sub-command and GCAM XML-Setup for more information.)
  • Run GCAM in an automatically-created workspace, allowing multiple instances of GCAM to run simultaneously, e.g., on parallel computing systems (See the gcam sub-command.)
  • Execute batch queries against the XML database to extract GCAM results, with on-the-fly regionalization based on a simple region-mapping file. (See the query sub-command.)
  • Compute differences between policy and baseline scenarios, including linear annualization of values between time-steps, and (See the diff sub-command.)
  • Plot results, with flexible control of figure features including title, axis labels, scale, and so on. (See the chart sub-command.)
  • Manage (create, delete, rename, run commands in) automatically-created workspaces. (See the sandbox sub-command.)

In addition, the run sub-command allows workflow steps to be defined in an XML file so that individual or groups of steps can be executed for one or more scenarios. The run sub-command supports direct invocation of other workflow steps as well as running arbitrary programs of the user’s choosing.

Finally, gt allows all project steps to be run on a compute node in a High-Performance Computing environment by specifying ‘-b’ or ‘–batch’’ on the command-line. (Note that this is not available on Mac OS X or Windows.)

For example, the command:

gt -b -P MyProject run -S MyScenario

runs all steps for scenario MyScenario in the project MyProject by queuing a batch job on the default queue. Arguments to gt allow the user to set various resource requirements and to select the queue to use.

The command to run to queue the batch job is taken from the configuration file parameter GCAM.BatchCommand. Example batch commands for the SLURM and PBS job management systems are provided in variables GCAM.QueueSLURM and GCAM.QueuePBS, respectively.

Command-line usage is described below. Note that some command-line (e.g., batch-related) options must precede the sub-command, whereas sub-command specific options must follow it.

Usage

usage: gt [-h] [-b] [-B] [-e ENVIROVARS] [-j JOBNAME] [-l level] [-L LOGFILE]
          [-m MINUTES] [--mcs {trial,gensim}] [-P name] [-q QUEUENAME]
          [-r RESOURCES] [--set name=value] [-v] [--version]
          {chart,config,diff,gcam,new,protect,query,run,sandbox,setup} ...
Options:
-b=False, --batch=False
 Run the commands by submitting a batch job using the command given by config variable GCAM.BatchCommand. (Linux only)
-B=False, --showBatch=False
 Show the batch command to be run, but don’t run it. (Linux only)
-e, --enviroVars
 Comma-delimited list of environment variable assignments to pass to queued batch job, e.g., -E “FOO=1,BAR=2”. (Linux only)
-j=gt, --jobName=gt
 Specify a name for the queued batch job. Default is “gt”. (Linux only)
-l, --logLevel

Sets the log level of the program.

Possible choices: notset, debug, info, warning, error, critical

-L, --logFile Sets the name of a log file for batch runs. Default is “gt-$j.out” where “$j” is replaced by “%%j”, which (in SLURM) is the jobid. If the argument is not an absolute pathname, it is treated as relative to the value of GCAM.LogDir.
-m, --minutes Set the number of minutes to allocate for the queued batch job. Overrides config parameter GCAM.Minutes. (Linux only)
--mcs

Used only when running gcamtool from gcammcs.

Possible choices: trial, gensim

-P, --projectName
 The project name (the config file section to read from), which defaults to the value of config variable GCAM.DefaultProject
-q, --queueName
 Specify the name of the queue to which to submit the batch job. Default is given by config variable GCAM.DefaultQueue. (Linux only)
-r=, --resources=
 Specify resources for the queued batch command. Can be a comma-delimited list of assignments of the form NAME=value, e.g., -r ‘pvmem=6GB’. (Linux only)
--set=[] Assign a value to override a configuration file parameter. For example, to set batch commands to start after a prior job of the same name completes, use –set “GCAM.OtherBatchArgs=-d singleton”. Enclose the argument in quotes if it contains spaces or other characters that would confuse the shell. Use multiple –set flags and arguments to set multiple variables.
-v=False, --verbose=False
 Show diagnostic output
--version show program’s version number and exit
Sub-commands:
chart

The chart sub-command generates plots from GCAM-style ”.csv” files. Two types of plots are currently supported: (i) stacked bar plots based on summing values over all years (with optional interpolation of annual values), by the given ‘indexCol’ (default is ‘region’), and (ii) stacked bar plots by year for some data column, where the data are grouped by and summed across elements with the indicated ‘indexCol’. The first option is indicated by using the ‘-S’ (‘–sumYears’) option. Numerous options allow the appearance to be customized.

usage: gt chart [-h] [-b] [-B] [-c COLUMNS] [-C CONSTRAINT] [-d OUTPUTDIR]
                [-D WORKINGDIR] [-e] [-f FROMFILE] [-F DIVISORFILE]
                [--format FORMAT] [-g] [-i] [-I INDEXCOL] [-k] [-l]
                [-L LABELCOLOR] [-m MULTIPLIER] [-M MULTIPLIERFILE] [-n NCOL]
                [-N SCENARIO] [--negate] [-o OUTFILE] [-O] [-p PALETTE]
                [-r ROTATION] [-R REFERENCE] [--region REGION] [-s SKIPROWS]
                [-S] [-t YEARSTEP] [-T TITLE] [--timeseries] [--transparent]
                [-u UNSTACKED] [-U UNSTACKEDREGION] [-v VALUECOL] [-V DIVISOR]
                [--version] [-x SUFFIX] [-X XLABEL] [-Y YLABEL] [-y YEARS]
                [--ymax YMAX] [--ymin YMIN] [-z] [--legendY LEGENDY]
                [--barWidth BARWIDTH]
                [csvFile]
Positional arguments:
csvFile The file containing the data to plot.
Options:
-b=False, --box=False
 Draw a box around the plot. Default is no box.
-B=False, --byRegion=False
 Generate one plot per region. Region names are read from the CSV file, so they reflect any regional aggregation produced by the query.
-c=output, --columns=output
 Specify the column whose values identify the segments in the stacked bar chart. (These appear in the legend.)
-C, --constraint
 Apply a constraint to limit the rows of data to plot. The constraint can be any constraint string that is valid for the DataFrame.query() method, e.g., -C ‘input == “biomass”’
-d=., --outputDir=.
 The directory into which to write image files. Default is ”.”
-D=., --workingDir=.
 The directory to change to before performing any operations
-e=False, --enumerate=False
 Prefix image filenames with sequential number for easy reference. Used only with –fromFile
-f, --fromFile A file from which to read argument strings, one per line. These are read as if chartGCAM.py were called on each line individually, but avoiding the ~2 sec startup time for the bigger python packages.
-F, --divisorFile
 A file containing a floating point value to divide data by before plotting. See also -V.
--format Specify a format for the Y-axis. Possible values are ‘.’ for float, ‘,’ for int with commas, or any format recognized by print, e.g., “%%.2f” to Y values as floats with 2 decimal places.
-g=False, --ygrid=False
 Show light grey horizontal lines at the major Y-axis ticks. Default is no grid.
-i=False, --interpolate=False
 Interpolate (linearly) annual values between timesteps.
-I=region, --indexCol=region
 A column to use as the index column, or blank for None. Default is “region”.
-k=False, --yticks=False
 Show tick marks on Y-axis. Default is no tick marks.
-l=False, --label=False
 Add text along the right side of the figure showing the filename.
-L, --labelColor
 Color for the text label, which defaults to lightgrey. Some users may prefer “black”, for example. (Implies -l)
-m, --multiplier
 A value to multiply data by before generating the plot. The argument can be a floating point number or the name of any variable in pygcam.unitConversion.py. For example, “-m 3.667” and “-m C_to_CO2” are equivalent, and effectively convert values from Tg C to Tg CO2. Be sure to adjust the Y axis label. See also -M.
-M, --multiplierFile
 A file containing a floating point value to multiply data by before plotting. See also -m.
-n=5, --ncol=5 The number of columns with which to display the legend. Default is 5.
-N=, --scenario=
 When using the ‘–fromFile’ option, this argument is used to specify one or more scenario names (delimited by commas if more than one). These are substituted into each line read from the file as the value for “{scenario}” wherever it appears on each line read from the ‘fromFile’.
--negate=False Multiply data by -1 before plotting, which can make interpretation of some figures more intuitive. The string “-negated” is added to the file label, displayed if the “-l” or “-L” flag is specified.
-o=, --outFile=
 The name of the image file to create. Format is determined from filename extension. All common formats are allowed, e.g., png, pdf, tif, and gif. Try it; it probably works. Default is the name of the data file substituting ”.png” for ”.csv”
-O=False, --open=False
 Open the plot file after generating it.
-p, --palette The name of a color palette to use. Some good options include hls, husl, and Paired. See http://stanford.edu/~mwaskom/software/seaborn/tutorial/color_palettes.html
-r=90, --rotation=90
 Set the rotation angle for X-axis labels. Defaults to 90 degrees (vertical). Use 0 for horizontal labels.
-R=reference, --reference=reference
 When using the ‘–fromFile’ option, this argument is used to specify the name of the reference scenario. The “other” scenario is given using the “-N” option. These are substituted into each line read from the file as the value for “{scenario}” and “{reference}” (without the quotes) wherever they appear on each line read from the ‘fromFile’. Defaults to “reference”
--region Plot values only for the given region.
-s=1, --skiprows=1
 The number of rows of the CSV file to skip before reading the data (starting with a header row with column names.) Default is 1, which works for GCAM batch query output.
-S=False, --sumYears=False
 Sum across the time horizon, typically by region. This results in a stacked bar plot. When not summed over years (the default) a stacked area plot is generated showing values grouped and summed by indexCol (-I) and presented by year.
-t=5, --yearStep=5
 The spacing of year labels on X-axis for time-series plots. Defaults to 5.
-T=, --title= Adds a title to the plot. Default is no title. The string can have LaTeX math language in it, e.g., ‘CO$_2$’ causes the 2 to be subscripted, and ‘MJ$^{-1}$’ results in “MJ” with a superscripted exponent of -1. The string ‘$\Delta$’ results in a capital Greek delta. See LaTeX documentation for more options. Be sure to enclose the title in single quotes so the “$” is not (mis)interpreted by the shell.
--timeseries=False
 Plot the data as a time series.
--transparent=False
 Save the plot with a transparent background. (Default is white.)
-u, --unstacked
 Draw an unstacked bar plot for the column given as an argument to this option, showing three groups of bars: the region, all other regions, and the total.
-U, --unstackedRegion
 The region to plot separately from Rest of World in an unstacked plot. Ignored if –byRegion is specified, in which case a plot is created for all regions.
-v, --valueCol Identify a single column (e.g., a year) to plot for bar plots.
-V=0.0, --divisor=0.0
 A value to divide year column values by before plotting. The argument can be a floating point number or the name of any variable in pygcam.unitConversion.py. See also -F.
--version show program’s version number and exit
-x, --suffix A suffix to append to the basename of the input csv file to create the name for the output file. For example, if processing my_data.csv, indicating -x ‘-by-region.pdf’ results in an output file named my_data-by-region.pdf.
-X=, --xlabel= Defines a label for the X-axis; defaults to blank. LaTeX math language is supported. (See the -T flag for more info.)
-Y=EJ, --ylabel=EJ
 Label for the Y-axis; defaults to “EJ”. LaTeX math language is supported. (See the -T flag for more info.)
-y=, --years= Takes a parameter of the form XXXX-YYYY, indicating start and end years of interest. Data for all other years are dropped.
--ymax Set the scale of a figure by indicating the value to show as the maximum Y value. (By default, scale is set according to the data.)
--ymin Set the scale of a figure by indicating the value (given as abs(value), but used as -value) to show as the minimum Y value
-z=False, --zeroLine=False
 Whether to show a line at Y=0
--legendY The Y position of the legend. Useful for fixing poorly formatted figures. Note that to pass a negative value, use the syntax –legendY=”-xxx.xxx”, otherwise the hyphen is interpreted as indicating a command-line argument.
--barWidth=0.5 The relative width of bars. Helpful when plotting only 1 or 2 bar, so they aren’t obnoxiously wide. Default is 0.5
config

The config command list the values of configuration variables from ~/.pygcam.cfg. With no arguments, it displays the values of all variables for the default project. Use the -d flag to show only values from the [DEFAULT] section.

If an argument name is provided, it is treated as a substring pattern, unless the -x flag is given (see below). All configuration variables containing the give name are displayed with their values. The match is case-insensitive.

If the -x or --exact flag is specified, the argument is treated as an exact variable name (case-sensitive) and only the value is printed. This is useful mainly for scripting. For general use the substring matching is more convenient.

Examples:

 $ gt config project
 [MyProject]
 GCAM.DefaultProject = MyProject
 GCAM.ProjectRoot = /Users/rjp/bitbucket/myProject
 GCAM.ProjectXmlFile = /Users/rjp/bitbucket/myProject/etc/project.xml

 $ gt config -x GCAM.DefaultProject
 MyProject

 $ gt config sand
[MyProject]
      GCAM.SandboxRoot = /Users/rjp/ws/myProject

 $ gt config sand -d
 [DEFAULT]
      GCAM.SandboxRoot = /Users/rjp/ws
usage: gt config [-h] [-d] [-e] [-x] [-t] [--version] [name]
Positional arguments:
name Show the names and values of all parameters whose name contains the given value. The match is case-insensitive. If not specified, all variable values are shown.
Options:
-d=False, --useDefault=False
 Indicates to operate on the DEFAULT section rather than the project section.
-e=False, --edit=False
 Edit the configuration file. The command given by the value of config variable GCAM.TextEditor is run with the .pygcam.cfg file as an argument.
-x=False, --exact=False
 Treat the text not as a substring to match, but as the name of a specific variable. Match is case-sensitive. Prints only the value.
-t=False, --test=False
 Test the settings in the configuration file to ensure that the basic setup is ok, i.e., required parameters have values that make sense. If specified, no variables are displayed.
--version show program’s version number and exit
diff

The diff sub-command script computes the differences between results from two or more CSV files generated from batch queries run on a GCAM database, saving the results in either a CSV or XLSX file, according to the extension given to the output file. If not provided, the output filename defaults to differences.csv.

If multiple otherFiles are given (i.e., the referenceFile plus 2 or more other files named on the command-line), the resulting CSV file will contain one difference matrix for each otherFile, with a label indicating which pair of files were used to produce each result.

When the output file is in XLSX format, each result is written to a separate worksheet. If the -c flag is specified, no differences are computed; rather, the .csv file contents are combined into a single .xlsx file.

usage: gt diff [-h] [-D WORKINGDIR] [-g GROUPSUM] [-i] [-o OUTFILE] [-c]
               [-q QUERYFILE] [-r REWRITESETSFILE] [-S] [-s SKIPROWS]
               [--version] [-y YEARS] [-Y STARTYEAR]
               csvFiles [csvFiles ...]
Positional arguments:
csvFiles The files to process. For difference operations, the first file is treated as the reference file whose time-series data is subtracted from that of each other file. If missing, ”.csv” suffixes are added to all arguments (the ”.csv” is optional).
Options:
-D=., --workingDir=.
 The directory to change to before performing any operations
-g=, --groupSum=
 Group data for each timestep (or interpolated annual values) by the given column, and sum all members of each group to produce a timeseries for each group. Takes precedence over the simpler “-S” (“–sum”) option.
-i=False, --interpolate=False
 Interpolate (linearly) annual values between timesteps.
-o=differences.csv, --outFile=differences.csv
 The name of the ”.csv” or ”.xlsx” file containing the differences between each scenario and the reference. Default is “differences.csv”.
-c=False, --convertOnly=False
 Convert the given CSV files into an Excel workbook, one sheet per CSV file.
-q=, --queryFile=
 If the extension is ”.xml” (case insensitive), the argument must be an XML file holding a list of queries to run, with optional mappings specified to rewrite output. This file has the same structure as the <queries> element in project.xml. If the file doesn’t end in ”.xml”, it must be a text file listing the names of queries to process, one per line. NOTE: When –queryFile is specified, the two positional arguments are required: the names of the baseline and policy scenarios, in that order.
-r, --rewriteSetsFile
 An XML file defining query maps by name (default taken from config parameter “GCAM.RewriteSetsFile”)
-S=False, --sum=False
 Sum all timestep (or interpolated annual values) to produce a single time-series.
-s=1, --skiprows=1
 The number of rows to skip. Default is 1, which works for GCAM batch query output. Use -s0 for outFile.csv
--version show program’s version number and exit
-y=, --years= Takes a parameter of the form XXXX-YYYY, indicating start and end years of interest. Other years are dropped (except for annual outputs.)
-Y=0, --startYear=0
 The year at which to begin interpolation
gcam

The gcam sub-command runs the GCAM executable on the designated configuration file, scenario, or workspace. Typical use (e.g., from a project.xml file) would be to run GCAM by referencing a directory named the same as a scenario, holding a file called config.xml, as is generated by the setup sub-command. (See GCAM XML-Setup.)

If a workspace is specified on the command-line, it is used. Otherwise, if a scenario is specified, the workspace defined by {GCAM.SandboxDir}/{scenario} is used. If neither workspace nor scenario are defined, the value of config variable GCAM.RefWorkspace is used, i.e., GCAM is run in the reference workspace.

If the workspace doesn’t exist, it is created based on the reference GCAM workspace, defined by the configuration variable GCAM.RefWorkspace. By default, read-only directories (e.g., input and libs) are symbolically linked from the new workspace to the reference one. (See the new sub-command for more information on the creation of workspaces.)

Directories into which GCAM writes results (e.g., output and exe) are created in the new workspace, but read-only files within exe (e.g., the GCAM executable) are symbolically linked (with the same caveat for Windows users.)

Usage example:

gt gcam -S ~/MyProject/scenarios -s MyScenario -w ~/sandboxes/MyProject/MyScenario

would run the scenario MyScenario in the newly created sandbox (workspace) ~/sandboxes/MyProject/MyScenario using the configuration file ~/MyProject/scenarios/MyScenario/config.xml.

usage: gt gcam [-h] [-C CONFIGFILE] [-f] [-g GROUPDIR] [-n] [-r REFWORKSPACE]
               [-s SCENARIO] [-S SCENARIOSDIR] [--version] [-w WORKSPACE] [-W]
Options:
-C, --configFile
 Specify the one or more GCAM configuration filenames, separated by commas. If multiple configuration files are given, the are run in succession in the same “job” on the cluster.
-f=False, --forceCreate=False
 Re-create the workspace, even if it already exists.
-g=, --groupDir=
 The scenario group directory name, if any.
-n=False, --noRun=False
 Don’t run GCAM; just print the command that would be run.
-r, --refWorkspace
 The reference workspace to use to create the new sandbox. This is used only if the indicated or implied workspace doesn’t exist. Defaults to the value of GCAM.RefWorkspace.
-s=, --scenario=
 The scenario to run.
-S=, --scenariosDir=
 Specify the directory holding scenario files. Default is the value of config variable GCAM.ScenariosDir, if set, otherwise it’s the current directory.
--version show program’s version number and exit
-w, --workspace
 Specify the path to the GCAM workspace to use. If it doesn’t exist, the named workspace will be created. If not specified on the command-line, the path constructed as {GCAM.SandboxDir}/{scenario} is used.
-W=False, --noWrapper=False
 Do not run gcam within a wrapper that detects errors as early as possible and terminates the model run. By default, the wrapper is used.
new

Create the directory structure and basic files required for a new pygcam project. If a directory is specified with the -r flag, the project is created with the given name in that directory; otherwise the project is created in the directory identified by the config variable GCAM.ProjectRoot.

This sub-command creates examples of xmlsrc/scenarios.py, etc/protection.xml, etc/project.xml, etc/queries.xml, etc/rewriteSets.xml, and etc/scenarioSetup.xml that can be edited to fit the needs of your project. See the file Instructions.txt that is created in the new “etc” directory.

If the -c flag is given, a basic entry for the new project is added to the users configuration file, $HOME/.pygcam.cfg. Before modifying the config file, a backup is created in $HOME/.pygcam.cfg~. For example, the command

gt new -c foo

generates and entry like this:

[foo]
# Added by "new" sub-command Mon Aug 29 15:00:19 2016
GCAM.ProjectDir = %(GCAM.ProjectRoot)s/foo
#GCAM.LandProtectionXmlFile = %(GCAM.ProjectDir)s/etc/protection.xml
#GCAM.RewriteSetsFile       = %(GCAM.ProjectDir)s/etc/rewriteSets.xml
#GCAM.ScenarioSetupFile     = %(GCAM.ProjectDir)s/etc/scenarios.xml

Variables controlling linking vs copying

Several configuration variables control which files are copied versus linked (on systems on which the user has the ability to create symlinks; if not all files are copied):

GCAM.WorkspaceFilesToCopy
A list of paths relative to GCAM.RefWorkspace that should be copied to the same relative location under {GCAM.SandboxDir}/Workspace.
GCAM.WorkspaceFilesToLink
A list of paths relative to GCAM.RefWorkspace that should be symlinked to same relative location under {GCAM.SandboxDir}/Workspace.
GCAM.SandboxFilesToCopy
A list of paths relative to {GCAM.SandboxDir}/Workspace that should be copied to the same relative location in the current scenario directory.
GCAM.SandboxFilesToLink
A list of paths relative to {GCAM.SandboxDir}/Workspace that should be symlinked to the same relative location in the current scenario directory.
GCAM.OtherFilesToCopy
This variable is provided to allow the user to specify additional files to copy. Its value is appended to both GCAM.WorkspaceFilesToCopy and GCAM.SandboxFilesToCopy.
GCAM.OtherFilesToLink
This variable is provided to allow the user to specify additional files to symlink. Its value is appended to both GCAM.WorkspaceFilesToLink and GCAM.SandboxFilesToLink.

Note that when running on Windows, if pygcam detects that the user does not have permission to create symlinks, it sets the configuration variable GCAM.CopyAllFiles to True, resulting in copies of the reference files rather than symbolic links.

usage: gt new [-h] [-c] [--overwrite] [-r PATH] [--version] name
Positional arguments:
name Create the structure for the named project, and copy example XML files into the “etc” directory.
Options:
-c=False, --addToConfig=False
 Add a section for the new project to $HOME/.pygcam.cfg after making a backup of the file in $HOME/.pygcam.cfg~
--overwrite=False
 If files that are to be copied to the project directory exist, overwrite them. By default, existing files are not overwritten.
-r, --projectRoot
 The directory in which to create a subdirectory for the named project. Default is the value of config variable GCAM.ProjectRoot
--version show program’s version number and exit
protect

Generate versions of GCAM’s land_input XML files that protect a given fraction of land of the given land types in the given regions by subtracting the required land area from the “managed” land classes, thereby removing them from consideration in land allocations.

Simple protection scenarios can be specified on the command-line. More complex scenarios can be specified in an XML file, landProtection.xml.

Examples:

# Create and modify copies of the reference land files, renaming them with
# "prot\_" prefix. Protect 80% of the "UnmanagedForest" and "UnmanagedPasture"
# land classes in the specified regions only.

CLASSES=UnmanagedForest,UnmanagedPasture
REGIONS='Australia_NZ,Canada,EU-12,EU-15,Japan,Middle East,Taiwan,USA'
OUTDIR="$HOME/tmp/xml"

gt protect -f 0.8 "$INFILES" -l "$CLASSES" -r "$REGIONS" -o "$OUTDIR" -t 'prot_{filename}'
# Run the land protection scenario "s1", described in the file ``$HOME/protect.xml``,
# placing the results in the directory ``$HOME/ws/workspace1``

gt protect -s s1 -S "$HOME/protect.xml" -w "$HOME/ws/workspace1"
usage: gt protect [-h] [-b] [-f FRACTION] [--inPlace] [-l LANDCLASSES] [-m]
                  [-o OUTDIR] [-O] [-t TEMPLATE] [-r REGIONS] [-s SCENARIO]
                  [-S SCENARIOFILE] [-V] [-w WORKSPACE]
Options:
-b=False, --backup=False
 Make a copy of the output file, if it exists (with an added ~ after filename) before writing new output. This option is ignored if a scenario file is specified.
-f, --fraction The fraction of land in the given land classes to protect. (Required, unless a scenario file is specified, in which case this option is ignored.)
--inPlace=False
 Edit the file in place. This must be given explicitly, to avoid overwriting files by mistake.
-l, --landClasses
 The land class or classes to protect in the given regions. Multiple, comma-delimited land types can be given in a single argument, or the -l flag can be repeated to indicate additional land classes. By default, all unmanaged land classes are protected. Allowed land classes are [‘UnmanagedPasture’, ‘UnmanagedForest’, ‘Shrubland’, ‘Grassland’]. This option is ignored if a scenario file is specified.
-m=False, --mkdir=False
 Make the output dir if necessary.
-o=., --outDir=.
 The directory into which to write the modified files. Default is current directory.
-O=False, --otherArable=False
 Include OtherArableLand in the list of default land classes to protect. This flag is ignored if the -l (–landClasses) argument is used.
-t=prot_{fraction}_{filename}, --template=prot_{fraction}_{filename}
 Specify a template to use for output filenames. The keywords {fraction}, {filename}, {regions}, and {classes} (with surrounding curly braces) are replaced by the following values and used to form the name of the output files, written to the given output directory. fraction: 100 times the given fraction (i.e., int(fraction * 100)); filename: the name of the input file being processed (e.g., land_input_2.xml or land_input_3.xml); basename: the portion of the input filename prior to the extension (i.e., before ‘.xml’); regions: the given regions, separated by ‘-‘, or the word ‘global’ if no regions are specified; classes: the given land classes, separated by ‘-‘, or the word ‘unmanaged’ if no land classes are specified. The default pattern is “prot_{fraction}_{filename}”. This option is ignored if a scenario file is specified.
-r, --regions The region or regions for which to protect land. Multiple, comma-delimited regions can be given in a single argument, or the -r flag can be repeated to indicate additional regions. By default, all regions are protected. This option is ignored if a scenario file is specified.
-s, --scenario The name of a land-protection scenario defined in the file given by the –scenarioFile argument or it’s default value.
-S, --scenarioFile
 An XML file defining land-protection scenarios. Default is the value of configuration file parameter GCAM.LandProtectionXmlFile.
-V, --version show program’s version number and exit
-w, --workspace
 Specify the path to the GCAM workspace to use. The files in {workspace}/input/gcam-data-system/xml/aglu-xml/land_input_{2,3}.xml are used as inputs. Default is value of configuration parameter GCAM.RefWorkspace.
query

Run one or more GCAM database queries by generating and running the named XML queries. The results are placed in a file in the specified output directory with a name composed of the basename of the XML query file plus the scenario name. For example,

gt query -o. -s MyReference,MyPolicyCase liquids-by-region

would run the liquids-by-region query on two scenarios, MyReference and MyPolicyCase. Query results will be stored in the files ./liquids-by-region-MyReference.csv and ./liquids-by-region-MyPolicyCase.csv.

The named queries are located using the value of config variable GCAM.QueryPath, which can be overridden with the -Q argument. The QueryPath consists of one or more colon-delimited (on Unix) or semicolon-delimited (on Windows) elements that can identify directories or XML files. The elements of QueryPath are searched in order until the named query is found. If a path element is a directory, the filename composed of the query + ‘.xml’ is sought in that directory. If the path element is an XML file, a query with a title matching the query name (first literally, then by replacing '_' and '-' characters with spaces) is sought. Note that query names are case-sensitive.

usage: gt query [-h] [-d XMLDB] [-D] [-g GROUPDIR] [-n] [-o OUTPUTDIR] [-p]
                [-q QUERYXMLFILE] [-Q QUERYPATH] [-r REGIONS] [-R REGIONMAP]
                [-s SCENARIO] [-S REWRITESETSFILE] [--version] [-w WORKSPACE]
                [queryName [queryName ...]]
Positional arguments:
queryName A file or files, each holding an XML query to run. (The ”.xml” suffix will be added if needed.) If an argument is preceded by the “@” sign, it is read and its contents substituted as the values for this argument. That means you can store queries to run in a file (one per line) and just reference the file by preceding the filename argument with “@”.
Options:
-d, --xmldb The XML database to query (default is computed as {GCAM.SandboxDir}/output/{GCAM.DbFile}. Overrides the -w flag.
-D=False, --noDelete=False
 Don’t delete any temporary file created by extracting a query from a query file. Used mainly for debugging.
-g=, --groupDir=
 The scenario group directory name, if any. Used with to compute default for –workspace argument.
-n=False, --noRun=False
 Show the command to be run, but don’t run it
-o, --outputDir
 Where to output the result (default taken from config parameter “GCAM.OutputDir”)
-p=False, --prequery=False
 Generate the XMLDBDriver.properties file and associated batch file to be run by GCAM when GCAM.BatchMultipleQueries or GCAM.InMemoryDatabase are True.
-q, --queryXmlFile
 An XML file holding a list of queries to run, with optional mappings specified to rewrite output. This file has the same structure as the <queries> element in project.xml.
-Q, --queryPath
 A semicolon-delimited list of directories or filenames to look in to find query files. Defaults to value of config parameter GCAM.QueryPath
-r, --regions A comma-separated list of regions on which to run queries found in query files structured like Main_Queries.xml. If not specified, defaults to querying all 32 regions.
-R, --regionMap
 A file containing tab-separated pairs of names, the first being a GCAM region and the second being the name to map this region to. Lines starting with “#” are treated as comments. Lines without a tab character are also ignored. This arg overrides the value of config variable GCAM.RegionMapFile.
-s=Reference, --scenario=Reference
 The scenario to run the query/queries for (default is “Reference”) Note that this must refers to a scenarios in the XML database.
-S, --rewriteSetsFile
 An XML file defining query maps by name (default taken from config parameter “GCAM.RewriteSetsFile”)
--version show program’s version number and exit
-w=, --workspace=
 The workspace directory in which to find the XML database. Defaults computed as {GCAM.SandboxDir}/{groupDir}/{scenario}. Overridden by the -d flag.
run

This sub-command reads instructions from the file project.xml, the location of which is taken from the user’s ~/.pygcam.cfg file. The workflow steps indicated in the XML file and command-line determine which commands to run.

Examples:

Run all steps for the default scenario group for project ‘Foo’:

gt -P Foo run

Run all steps for scenario group ‘test’ for project ‘Foo’, but only for scenarios ‘baseline’ and ‘policy-1’:

gt -P Foo run -g test -S baseline,policy1

or, equivalently:

gt -P Foo run --group test --scenario baseline --step policy1

Run only the ‘setup’ and ‘gcam’ steps for scenario ‘baseline’ in the default scenario group:

gt -P Foo run -s setup,gcam -S baseline,policy-1

Same as above, but queue a batch job to run these commands on the queue ‘short’:

gt -b -q short -P Foo run -s setup,gcam -S baseline,policy-1

Show the commands that would be executed for the above command, but don’t run them:

gt -P Foo run -s setup,gcam -S baseline,policy-1 -n
usage: gt run [-h] [-f PROJECTFILE] [-g GROUP] [-G] [-k SKIPSTEPS]
              [-K SKIPSCENARIOS] [-l] [-L] [-n] [-p PROJECT] [-q] [-s STEPS]
              [-S SCENARIOS] [--version] [--vars] [-x SANDBOXDIR]
Options:
-f, --projectFile
 The XML file describing the project. If set, command-line argument takes precedence. Otherwise, value is taken from config file variable GCAM.ProjectXmlFile, if defined, otherwise the default is ‘./project.xml’.
-g, --group The name of the scenario group to process. If not specified, the group with attribute default=”1” is processed.
-G=False, --listGroups=False
 List the scenario groups defined in the project file and exit.
-k, --skipStep Steps to skip. These must be names of steps defined in the project.xml file. Multiple steps can be given in a single (comma-delimited) argument, or the -k flag can be repeated to indicate additional steps. By default, all steps are run.
-K, --skipScenario
 Scenarios to skip. Multiple scenarios can be given in a single (comma-delimited) argument, or the -K flag can be repeated to indicate additional scenarios. By default, all scenarios are run.
-l=False, --listSteps=False
 List the steps defined for the given project and exit. Dynamic variables (created at run-time) are not displayed.
-L=False, --listScenarios=False
 List the scenarios defined for the given project and exit. Dynamic variables (created at run-time) are not displayed.
-n=False, --noRun=False
 Display the commands that would be run, but don’t run them.
-p, --project The name of the project to run.
-q=False, --noQuit=False
 Don’t quit if an error occurs when processing a scenario, just move on to processing the next scenario, if any.
-s, --step The steps to run. These must be names of steps defined in the project.xml file. Multiple steps can be given in a single (comma-delimited) argument, or the -s flag can be repeated to indicate additional steps. By default, all steps are run.
-S, --scenario Which of the scenarios defined for the given project should be run. Multiple scenarios can be given in a single (comma-delimited) argument, or the -S flag can be repeated to indicate additional scenarios. By default, all active scenarios are run.
--version show program’s version number and exit
--vars=False List variables and their values
-x, --sandboxDir
 The directory in which to create the run-time sandbox workspace. Defaults to value of {GCAM.SandboxProjectDir}/{scenarioGroup}.
sandbox

The sandbox sub-command allows you to create, delete, show the path of, or run a shell command in a workspace. If the --scenario argument is given, the operation is performed on a scenario-specific workspace within a project directory. If --scenario is not specified, the operation is performed on the project directory that contains individual scenario workspaces. Note that the gcam sub-command automatically creates workspaces as needed.

N.B. You can run sandbox with the --path option before performing any operations to be sure of the directory that will be operated on, or use the --noExecute option to show the command that would be executed by --run.

usage: gt sandbox [-h] [--create] [--delete] [--recreate] [-g GROUPDIR] [-n]
                  [-p] [-r RUN] [-s SCENARIO] [--version]
Options:
--create=False Create the identified sandbox. If used with –delete, the deletion occurs first.
--delete=False Delete the identified sandbox’ If used with –create, the deletion occurs first.
--recreate=False
 Recreate the identified sandbox. Equivalent to using the –delete and –create options together.
-g=, --groupDir=
 The name of the scenario group subdir
-n=False, --noExecute=False
 Print the command that would be executed by –run, but don’t execute it.
-p=False, --path=False
 Print the absolute path to the identified sandbox.
-r, --run Run the given command in the identified sandbox.
-s=, --scenario=
 The scenario for the computed sandbox root.
--version show program’s version number and exit
setup

The setup sub-command automates modification to copies of GCAM’s input XML files and construction of a corresponding configuration XML file. See GCAM XML-Setup for a detailed description.

usage: gt setup [-h] [-b BASELINE] [-d] [-f] [-g GROUP] [-G] [-m MODULEPATH]
                [-M MODULESPEC] [-p period-or-year] [-r REFWORKSPACE]
                [-R RESULTSDIR] [-s SCENARIO] [-S SUBDIR]
                [--setupXml SETUPXML] [-u] [-x XMLSOURCEDIR]
                [-X XMLOUTPUTROOT] [-w WORKSPACE] [-y YEARS]
Options:
-b, --baseline Identify the baseline the selected scenario is based on. Note: at least one of –baseline (-b) / –scenario (-s) must be used.
-d=False, --dynamicOnly=False
 Generate only dynamic XML for dyn-xml: don’t create static XML.
-f=False, --forceCreate=False
 Re-create the workspace, even if it already exists.
-g, --group The scenario group to process. Defaults to the group labeled default=”1”.
-G=False, --staticOnly=False
 Generate only static XML for local-xml: don’t create dynamic XML.
-m, --modulePath
 The path to a scenario definition module. See -M flag for more info.
-M, --moduleSpec
 The “dot spec” for the Python module holding the setup classes and a function called ‘scenarioMapper’ or a dictionary called ‘ClassMap’ which map scenario names to classes. If the function ‘scenarioMapper’ exists, it is used. If not, the ‘ClassMap’ is used. Default is “{xmlsrc}/subdir/scenarios.py” (if subdir is defined) or “{xmlsrc}/scenarios.py” (if subdir is undefined) under the current ProjectRoot.
-p, --stop The number of the GCAM period or the year to stop after
-r=, --refWorkspace=
 A reference workspace to use instead of the value of GCAM.RefWorkspace
-R, --resultsDir
 The parent directory holding the GCAM output workspaces
-s, --scenario Identify the scenario to run. Note: at least one of –baseline (-b) / –scenario (-s) must be used.
-S=, --subdir= A sub-directory to use instead of scenario name
--setupXml An XML scenario definition file. Overrides configuration variable GCAM.ScenarioSetupFile.
-u=False, --useGroupDir=False
 Use the group name as a subdir below xmlsrc, local-xml, and dyn-xml
-x, --xmlSourceDir
 The location of the xmlsrc directory.
-X, --xmlOutputRoot
 The root directory into which to generate XML files.
-w, --workspace
 The pathname of the workspace to operate on.
-y=2015-2100, --years=2015-2100
 Years to generate constraints for. Must be of the form XXXX-YYYY. Default is “2015-2100”

Extending gt using plug-ins

The gt script will load any python files whose name ends in _plugin.py, found in any of the directories indicated in the config file variable GCAM.PluginPath. The value of GCAM.PluginPath must be a sequence of directory names separated by colons (:) on Unix-like systems or by semi-colons (;) on Windows.

See pygcam.subcommand for documentation of the plug-in API.