pygcam.xmlEditor

This module defines common variables, functions, and classes for manipulating XML files to setup GCAM modeling experiments. An introduction to the XML-Setup system is available on the GCAM XML-Setup page; command-line features are documented with the setup sub-command.

The basic approach is to create a directory for each defined scenario, in which modified files and a corresponding configuration XML file are stored.

To allow functions to be called in any order or combination, each checks for a local copy of the file to be edited, and if not present, copies the original file to the local directory. Local copies are modified in place. Each function updates the local config file so that it loads the modified file.

XML Starlet

The xmlEditor module relies on the XML Starlet program, a command-line tool that can search and edit XML files (among other tricks.) It is available for all three GCAM platforms. Download XML Starlet. It should be included on Linux systems. It is available in binary (executable) form for Windows, but must be compiled on Mac OS X.

API

class pygcam.xmlEditor.XMLEditor(baseline, scenario, xmlOutputRoot, xmlSourceDir, refWorkspace, groupDir, subdir, parent=None)

Base class for scenario setup. Actual scenarios must subclass this. Represents the information required to setup a scenario, i.e., to generate and/or copy the required XML files into the XML output dir.

addMarketConstraint(target, policy, dynamic=False)

Adds references to a pair of files comprising a policy, i.e., a policy definition file and a constraint file. References to the two files–assumed to be named XXX-{subsidy,tax}.xml and XXX-{subsidy,tax}-constraint.xml for policy target XXX–are added to the configuration file.

Parameters:
  • target – (str) the subject of the policy, e.g., corn-etoh, cell-etoh, ft-biofuel, biodiesel
  • policy – (str) one of subsidy or tax
  • dynamic – (str) True if the XML file was dynamically generated, and thus found in dyn-xml rather than local-xml
Returns:

none

addScenarioComponent(name, xmlfile)

Add a new <ScenarioComponent> to the configuration file, at the end of the list of components.

Parameters:
  • name – (str) the name to assign to the new scenario component
  • xmlfile – (str) the location of the XML file, relative to the exe directory
Returns:

none

cfgPath()

Compute the name of the GCAM config file for the current scenario.

Returns:(str) the pathname to the XML configuration file.
delMarketConstraint(target, policy)

Delete the two elements defining a market constraint from the configuration file. The filenames are constructed as indicated in

Parameters:
  • target – (str) the subject of the policy, e.g., corn-etoh, cell-etoh, ft-biofuel, biodiesel
  • policy – (str) one of subsidy or tax
Returns:

none

deleteScenarioComponent(name)

Delete a <ScenarioComponent> identified by the <Value> element name.

Parameters:name – (str) the name of the ScenarioComponent to delete
Returns:none
getLocalCopy(pathname)

Get the filename for the most local version (in terms of scenario hierarchy) of the XML file pathname, and copy the file to our scenario dir if not already there.

Parameters:pathname – (str) the pathname of an XML file
Returns:(str, str) a tuple of the relative and absolute path of the local (i.e., within the current scenario) copy of the file.
insertScenarioComponent(name, xmlfile, after)

Insert a <ScenarioComponent> to the configuration file, following the entry named by after.

Parameters:
  • name – (str) the name to assign to the new scenario component
  • xmlfile – (str) the location of the XML file, relative to the exe directory
  • after – (str) the name of the element after which to insert the new component
Returns:

none

makeScenarioComponentsUnique()

Give all reference ScenarioComponents a unique “name” tag to facilitate manipulation via xmlstarlet.

Returns:none
parseRelPath(relPath)

Parse a relative pathname and return a tuple with the scenario prefix, the tail part (after the prefix) and the absolute path to this file. If a scenario doesn’t recognize the prefix as its own, it recursively asks its parent, unless the parent is None, in which case the standard GCAM prefix is checked, and if not present, and error is raised.

Parameters:relPath – (str) a relative pathname
Returns:(str) the pathname of the closest copy of the file
protectLand(fraction, landClasses=None, otherArable=False, regions=None, unprotectFirst=False)

Modify land_input files to protect a constant fraction of unmanaged land of the given classes, in the given regions. Callable from XML setup files.

Parameters:
  • fraction – (float) the fraction of land in the given land classes to protect
  • landClasses – a string or a list of strings, or None. If None, all “standard” unmanaged land classes are modified.
  • otherArable – (bool) if True, land class ‘OtherArableLand’ is included in default land classes.
  • regions – a string or a list of strings, or None. If None, all regions are modified.
renameScenarioComponent(name, xmlfile)

Modify the name of a ScenarioComponent, located by the XML file path it holds. This is used in to create a local reference XML that has unique names for all scenario components, which allows all further modifications to refer only to the (now unique) names.

Parameters:
  • name – (str) the new name for the scenario component
  • xmlfile – (str) the XML file path used to locate the scenario component
Returns:

none

setClimateOutputInterval(years)

Sets the the frequency at which climate-related outputs are saved to the XML database to the given number of years, e.g., <Value name="climateOutputInterval">1</Value>. Callable from XML setup files.

Parameters:years – (coercible to int) the number of years to set as the climate (GHG) output interval
Returns:none
setEnergyTechnologyCoefficients(subsector, technology, energyInput, values)

Set the coefficients in the global technology database for the given energy input of the given technology in the given subsector. Callable from XML setup files.

Parameters:
  • subsector – (str) the name of the subsector
  • technology – (str) The name of the technology, e.g., ‘cellulosic ethanol’, ‘FT biofuel’, etc.
  • energyInput – (str) the name of the minicam-energy-input
  • values – A sequence of tuples or object with iteritems method returning (year, coefficient). For example, to set the coefficients for cellulosic ethanol for years 2020 and 2025 to 1.234, the pairs would be ((2020, 1.234), (2025, 1.234)).
Returns:

none

setGlobalTechNonEnergyCost(sector, subsector, technology, values)

Set the non-energy cost of for technology in the global-technology-database, given a list of values of (year, price). The price is applied to all years indicated by the range. Callable from XML setup files.

Parameters:
  • sector – (str) the name of a GCAM sector
  • subsector – (str) the name of a GCAM subsector within sector
  • technology – (str) the name of a GCAM technology in subsector
  • values – (dict-like or iterable of tuples of (year, price)) year can be a single year (as string or int), or a string specifying a range of years, of the form “xxxx-yyyy”, which implies 5 year timestep, or “xxxx-yyyy:s”, which provides an alternative timestep. If values is dict-like (e.g. a pandas Series) a list of tuples is created by calling values.iteritems() after which the rest of the explanation above applies. The price can be anything coercible to float.
setGlobalTechShareWeight(sector, subsector, technology, values, xmlBasename='en_transformation.xml', configFileTag='energy_transformation')

Create a modified version of en_transformation.xml with the given share-weights for technology in sector based on the data in values. Callable from XML setup files.

Parameters:
  • sector – (str) the name of a GCAM sector
  • technology – (str) the name of a GCAM technology in sector
  • values – (dict-like or iterable of tuples of (year, shareWeight)) year can be a single year (as string or int), or a string specifying a range of years, of the form “xxxx-yyyy”, which implies 5 year timestep, or “xxxx-yyyy:s”, which provides an alternative timestep. If values is dict-like (e.g. a pandas Series) a list of tuples is created by calling values.iteritems() after which the rest of the explanation above applies. The shareWeight can be anything coercible to float.
  • xmlBasename – (str) the name of an xml file in the energy-xml folder to edit.
  • configFileTag – (str) the ‘name’ of a <File> element in the <ScenarioComponents> section of a config file. This must match xmlBasename.
Returns:

none

setGlobalTechShutdownRate(sector, subsector, technology, values)

Create a modified version of en_transformation.xml with the given shutdown rates for technology in sector based on the data in values. Callable from XML setup files.

Parameters:
  • sector – (str) the name of a GCAM sector
  • subsector – (str) the name of a GCAM subsector within sector
  • technology – (str) the name of a GCAM technology in subsector
  • values – (dict-like or iterable of tuples of (year, shutdownRate)) year can be a single year (as string or int), or a string specifying a range of years, of the form “xxxx-yyyy”, which implies 5 year timestep, or “xxxx-yyyy:s”, which provides an alternative timestep. If values is dict-like (e.g. a pandas Series) a list of tuples is created by calling values.iteritems() after which the rest of the explanation above applies. The shutdownRate can be anything coercible to float.
  • xmlBasename – (str) the name of an xml file in the energy-xml folder to edit.
  • configFileTag – (str) the ‘name’ of a <File> element in the <ScenarioComponents> section of a config file. This must match xmlBasename.
Returns:

none

setInterpolationFunction(region, supplysector, subsector, fromYear, toYear, funcName, applyTo='share-weight', stubTechnology=None)

Set the interpolation function for the share-weight of the subsector of supplysector to funcName between years fromYear to toYear in region. Callable from XML setup files.

Parameters:
  • region – the GCAM region to operate on
  • supplysector – the name of a supply sector
  • subsector – the name of a sub-sector
  • fromYear – the year to start interpolating
  • toYear – the year to stop interpolating
  • funcName – the name of an interpolation function
  • applyTo – what the interpolation function is applied to
Returns:

none

setRegionalShareWeights(region, sector, subsector, values, stubTechnology=None, xmlBasename='en_transformation.xml', configFileTag='energy_transformation')

Create a modified version of en_transformation.xml with the given share-weights for technology in sector based on the data in values. Callable from XML setup files.

Parameters:
  • region – if not None, changes are made in a specific region, otherwise they’re made in the global-technology-database.
  • sector – (str) the name of a GCAM sector
  • technology – (str) the name of a GCAM technology in sector
  • values – (dict-like or iterable of tuples of (year, shareWeight)) year can be a single year (as string or int), or a string specifying a range of years, of the form “xxxx-yyyy”, which implies 5 year timestep, or “xxxx-yyyy:s”, which provides an alternative timestep. If values is dict-like (e.g. a pandas Series) a list of tuples is created by calling values.iteritems() after which the rest of the explanation above applies. The shareWeight can be anything coercible to float.
  • xmlBasename – (str) the name of an xml file in the energy-xml folder to edit.
  • configFileTag – (str) the ‘name’ of a <File> element in the <ScenarioComponents> section of a config file. This must match xmlBasename.
Returns:

none

setStopPeriod(yearOrPeriod)

Sets the model stop period. If stopPeriod is <= 22, the stop period is set to the given value. If the value > 2000, the value is treated as a year and converted to the correct stop period for the configuration file.

Parameters:yearOrPeriod – (coercible to int) this argument is treated as a literal stop period if the value is < 1000. (N.B. 2015 = step 4, 2020 = step 5, and so on.) If yearOrPeriod >= 1000, it is treated as a year and converted to a stopPeriod for use in the GCAM configuration file.
Returns:none
Raises:SetupException
setup(args)

Calls setupStatic and/or setupDynamic, depending on flags set in args.

Parameters:args – (argparse.Namespace) arguments passed from the top-level call to setup
Returns:none
setupDynamic(args)

Create dynamic XML files in dyn-xml. These files are generated for policy scenarios when XML file contents must be computed from baseline results.

Parameters:args – (argparse.Namespace) arguments passed from the top-level call to setup sub-command
Returns:none
setupSolver(solutionTolerance=None, broydenTolerance=None, maxModelCalcs=None, maxIterations=None)

Set the model solution tolerance to the given values for the solver “driver” (solutionTolerance) and, optionally for the Broyden component (broydenTolerance).

Parameters:
  • solutionTolerance – (coercible to float, > 0.0) the value to set for the driver tolerance
  • broydenTolerance – (coercible to float, > 0.0) the value to set for the Broyden component tolerance. (If both are provided, the function requires that componentTolerance <= driverTolerance.)
  • maxModelCalcs – (coercible to int, > 0) maximum number of calculations to run in the driver
  • maxIterations – (coercible to int, > 0) maximum number of iterations to allow in the Broyden component
Returns:

none

setupStatic(args)

Create static XML files in local-xml. By “static”, we mean files whose contents are independent of baseline results. In comparison, policy scenarios may generate dynamic XML files whose contents are computed from baseline results.

Parameters:args – (argparse.Namespace) arguments passed from the top-level call to setup sub-command.
Returns:none
updateConfigComponent(group, name, value=None, writeOutput=None, appendScenarioName=None)

Update the value of an arbitrary element in GCAM’s configuration.xml file, i.e., <{group}><Value name="{name}>{value}</Value></{group}>

Optional args are used only for <Files> group, which has entries like <Value write-output="1" append-scenario-name="0" name="outFileName">outFile.csv</Value> Values for the optional args can be passed as any of [0, 1, "0", "1", True, False].

Parameters:
  • group – (str) the name of a group of config elements in GCAM’s configuration.xml
  • name – (str) the name of the element to be updated
  • value – (str) the value to set between the <Value></Value> elements
  • writeOutput – (coercible to int) for <Files> group, this sets the optional write-output attribute
  • appendScenarioName – (coercible to int) for <Files> group, this sets the optional append-scenario-name attribute.
Returns:

none

updateScenarioComponent(name, xmlfile)

Set a new filename for a ScenarioComponent identified by the <Value> element name.

Parameters:
  • name – (str) the name of the scenario component to update
  • xmlfile – (str) the location of the XML file, relative to the exe directory, that should replace the existing value
Returns:

none

pygcam.xmlEditor.copyIfMissing(src, dst, makedirs=False)

Copy file src to dst, but only if dst doesn’t already exist.

Parameters:
  • src – (str) pathname of the file to copy
  • dst – (str) pathname of the copy to create
  • makedirs – if True, make any missing directories
Returns:

none

pygcam.xmlEditor.expandYearRanges(seq)

Expand a sequence of (year, value) tuples, or a dict keyed by year, where the year argument may be a string containing identifying range of values with an optional “step” value indicated after a ”:”. The default step is 5 years. For example, “2015-2030” expands to (2015, 2020, 2025, 2030), and “2015-2020:1” expands to (2015, 2016, 2017, 2018, 2019, 2020). When a range is given, the tuple is replaced with a sequence of tuples naming each year explicitly. Typical usage is for year, price in expandYearRanges(values): ....

Parameters:seq_or_dict – The sequence of (year, value) tuples, or any object with an iteritems() method that returns (year, value) pairs.
Returns:A list of tuples with the expanded sequence.
pygcam.xmlEditor.extractStubTechnology(region, srcFile, dstFile, sector, subsector, technology, sectorElement='supplysector', fromRegion=False)

Extract a definition from the global-technology-database based on sector, subsector, and technology, defined in srcFile and create a new file, dstFile with the extracted bit as a stub-technology definition for the given region. If fromRegion is True, extract the stub-technology from the regional definition, rather than from the global-technology-database.

Parameters:
  • region – (str) the name of the GCAM region for which to copy the technology
  • srcFile – (str) the pathname of a source XML file with a global-technology-database
  • dstFile – (str) the pathname of the file to create
  • sector – (str) the name of a GCAM sector
  • subsector – (str) the name of a GCAM subsector within sector
  • technology – (str) the name of a GCAM technology within sector and subsector
  • sectorElement – (str) the name of the XML element to create (or search for, if fromRegion is True) between the <region> and <subsector> XML elements. Defaults to ‘supplysector’.
  • fromRegion – (bool) if True, the definition is extracted from a regional definition rather than from the global-technology-database.
Returns:

True on success, else False

pygcam.xmlEditor.makeDirPath(elements, require=False, create=False, mode=509)

Join the tuple of elements to create a path to a directory, optionally checking that it exists or creating intermediate directories as needed.

Parameters:
  • elements – a tuple of pathname elements to join
  • require – if True, raise an error if the path doesn’t exist
  • create – if True, create the path if it doesn’t exist
  • mode – file mode used when making directories
Returns:

the joined path

Raises:

pygcam.error.SetupException

pygcam.xmlEditor.xmlEdit(filename, *rest)

Edit the XML file filename in place, using the xmlstarlet arguments passed in rest.

Parameters:
  • filename – the file to edit
  • rest – (iterable) values to pass as arguments to xmlstarlet
Returns:

True on success, else False

pygcam.xmlEditor.xmlSel(filename, *rest)

Return True if the XML component identified by the xmlstarlet arguments in rest exists in filename. Useful for deciding whether to edit or insert an XML element.

Parameters:
  • filename – (str) the file to edit
  • rest – (iterable) values to pass as arguments to xmlstarlet
Returns:

pygcam.xmlEditor.xmlStarlet(*args)

Run the XML Starlet executable in a subprocess, passing the given args and return True if success, else False.

Parameters:args – (iterable) these values are passed as arguments to xmlstarlet. See xmlstarlet documentation for details.
Returns:True if exit status was 0, else False

Sector-specific config editors

These classes subclass XMLEditor to create sector-specific setup functionality.

class pygcam.sectorEditors.BioenergyEditor(baseline, scenario, xmlOutputRoot, xmlSourceDir, workspaceDir, groupDir, subdir, parent=None)

BioenergyEditor adds knowledge of biomass and biofuels.

adjustForestResidueSupply(region, loPrice, loFract, hiPrice, hiFract)

Change supply curves for forest residues in the USA only. loPrice and hiPrice are the new prices to set; loFract and hiFract are the new fractions to assign to these prices.

adjustResidueSupply(loTarget, loPrice, loFract, hiTarget, hiPrice, hiFract, target)

Change supply curves for residues, as per arguments. loTarget and hiTarget identify the price to match in the XML file; loPrice and hiPrice are the new prices to set; loFract and hiFract are the new fractions to assign to these prices. Target must be one of {all, us-crops, us-corn, global-corn} Note that the standard reference supply curve for residue is: 0% at 1975$0; 25% at $1.2; 65% at $1.5; 100% at $10.

localizeCellEthanolTechnologyUSA()

Same as corn ethanol above, but for cellulosic ethanol

localizeCornEthanolTechnologyUSA()

Copy the stub-technology for corn ethanol to CornEthanolUSA.xml and CornEthanolUSA2.xml so they can be manipulated in the US only.

localizeFtBiofuelsTechnologyUSA()

Same as cellulosic ethanol above

purposeGrownOffInRegion(region)

Turn off the “isNewTechnology” flag for all land-leaf nodes to turn off purpose-grown biomass. The line(s) we need to edit in land_input_3.xml is: <isNewTechnology fillout=”1” year=”2020”>1</isNewTechnology>

setBiofuelBiomassCoefficients(fuelName, pairs)

Set new coefficients for biomass conversion for the given fuel

Parameters:
  • fuelName – The name of the liquid fuel, e.g., ‘cellulosic ethanol’, ‘FT biofuel’, etc.
  • pairs – A sequence of tuples of the form (year, coefficient). For example, to set the coefficients for cellulosic ethanol for years 2020 and 2025 to 1.234, the pairs would be ((2020, 1.234), (2025, 1.234)).
Returns:

nothing

setCellEthanolShareWeightUSA(year, shareweight)

Create modified version of cellEthanolUSA.xml with the given share-weight for the given fuel in the given year.

setCornEthanolCoefficients(cornCoef, gasCoef=None, elecCoef=None)

Set corn ethanol performance coefficients, i.e., the (regional) corn, gas, and electricity required per GJ of ethanol. These appear in two files, so we edit them both.

setCornEthanolCoefficientsUSA(cornCoef, gasCoef=None, elecCoef=None)

Set corn ethanol performance coefficients: (regional) corn, gas, and electricity required per GJ of ethanol. Modified from superclass version to operate on biofuelTechUSA.xml.