pygcam.utils
¶
This module contains constants, functions, and classes that are used by various other modules of the pygcam toolkit.
API¶
-
class
pygcam.utils.
TempFile
(path=None, suffix='', tmpDir=None, delete=True, openFile=False, text=True, createDir=False)¶ Class to create and track temporary files in one place so they can be deleted before an application exits.
-
deleteFile
()¶ Remove the file for a TempFile instance if
self.delete
is True. In either case, delete the instance from the class instance dict.Returns: none Raises: PygcamException if the path is not related to a TempFile instance.
-
classmethod
remove
(filename, raiseError=True)¶ Remove a temporary file and delete the TempFile instance from the dict.
Parameters: - filename – (str) the name of a temp file created by this class
- raiseError – (bool) if True, raise an exception if the filename is not a known TempFile.
Returns: none
Raises: PygcamException – if the path is not related to a TempFile instance.
-
-
class
pygcam.utils.
XMLFile
(xmlFile, schemaFile=None, raiseError=True, rootClass=None, removeComments=True)¶ Represents an XML file, which is parsed by lxml.etree and stored internally.
Parameters: - xmlFile – pathname of the XML file
- schemaFile – optional XMLSchema file (or stream) to use for validation
- raiseError – if True, raise an error if validation fails
- rootClass – optional root class, which is instantiated for the parsed tree and stored internally
Raises: FileFormatError
-
pygcam.utils.
chunkify
(lst, chunks)¶ Iterator to turn a list into the number of lists given by chunks. In the case that len(lst) % chunksize != 0, all chunks are made as close to the same size as possible.
Parameters: - lst – (list) a list of values
- chunks – (int) the number of chunks to create
Returns: iterator that returns one chunk at a time
-
pygcam.utils.
coercible
(value, type, raiseError=True)¶ Attempt to coerce a value to type and raise an error on failure.
Parameters: - value – any value coercible to type
- type – any Python type
Returns: (type) the coerced value, if it’s coercible, otherwise None if raiseError is False
Raises: PygcamException – if not coercible and raiseError is True
-
pygcam.utils.
copyFileOrTree
(src, dst)¶ Copy src to dst, where the two can both be files or directories. If src and dst are directories, dst must not exist yet.
Parameters: - src – (str) path to a source file or directory
- dst – (str) path to a destination file or directory.
Returns: none
-
pygcam.utils.
copyResource
(relpath, dest, overwrite=True)¶ Copy a resource from the ‘pygcam’ package to the given destination.
Parameters: - relpath – (str) a path relative to the pygcam package
- dest – (str) the pathname of the file to create by copying the resource.
- overwrite – (bool) if False, raise an error if the destination file already exists.
Returns: none
-
pygcam.utils.
createTrialString
(lst)¶ Assemble a list of numbers into a compact list using hyphens to identify ranges. This function is the inverse of
parseTrialString()
.
-
pygcam.utils.
deleteFile
(filename)¶ Delete the given filename, but ignore errors, like “rm -f”
Parameters: filename – (str) the file to remove Returns: none
-
pygcam.utils.
digitColumns
(df, asInt=False)¶ Get a list of columns with integer names (as strings, e.g., “2007”) in df. If asInt is True return as a list of integers, otherwise as strings.
-
pygcam.utils.
ensureCSV
(file)¶ Ensure that the file has a ‘.csv’ extension by replacing or adding the extension, as required.
Parameters: file – (str) a filename Returns: (str) the filename with a ‘.csv’ extension.
-
pygcam.utils.
ensureExtension
(filename, ext)¶ Force a filename to have the given extension, ext, adding it to any other extension, if present. That is, if filename is
foo.bar
, and ext isbaz
, the result will befoo.bar.baz
. If ext doesn’t start with a ”.”, one is added.Parameters: - filename – filename
- ext – the desired filename extension
Returns: filename with extension ext
-
pygcam.utils.
flatten
(listOfLists)¶ Flatten one level of nesting given a list of lists. That is, convert [[1, 2, 3], [4, 5, 6]] to [1, 2, 3, 4, 5, 6].
Parameters: listOfLists – a list of lists, obviously Returns: the flattened list
-
pygcam.utils.
getBatchDir
(scenario, resultsDir)¶ Get the name of the directory holding batch query results. This differs when running in pygcam’s “gt” or when running in GCAM-MCS.
Parameters: - scenario – (str) the name of a scenario
- resultsDir – (str) the directory in which the batch results directory should be created
Returns: (str) the pathname to the batch results directory
-
pygcam.utils.
getBooleanXML
(value)¶ Get a value from an XML file and convert it into a boolean True or False.
Parameters: value – any value (it’s first converted to a string) Returns: True if the value is in [‘true’, ‘1’], False if the value is in [‘false’, ‘0’]. An exception is raised if any other value is passed. Raises: PygcamException
-
pygcam.utils.
getResource
(relpath)¶ Extract a resource (e.g., file) from the given relative path in the pygcam package.
Parameters: relpath – (str) a path relative to the pygcam package Returns: the file contents
-
pygcam.utils.
getTempDir
(suffix='', tmpDir=None, delete=True)¶ Convenience function for common use pattern, which is to get the name of a temporary directory that needs to be deleted on app exit.
Parameters: - suffix – (str) an extension to give the temporary file
- tmpDir – (str) the directory in which to create the new temporary directory (Defaults to the value of configuration file variable ‘GCAM.TempDir’, or ‘/tmp’ if the variable is not found.
- delete – (bool) if False, don’t delete the file on cleanup. (This is useful for debugging.)
Returns: (str) pathname of a new temporary directory
-
pygcam.utils.
getTempFile
(suffix='', tmpDir=None, text=True, delete=True)¶ Convenience function for common use pattern, which is to get the name of a temp file that needs to be deleted on app exit.
Parameters: - suffix – (str) an extension to give the temporary file
- tmpDir – (str) the directory in which to create the file. (Defaults to the value of configuration file variable ‘GCAM.TempDir’, or ‘/tmp’ if the variable is not found.
- text – True if this will be a text file
- delete – (bool) if False, don’t delete the file on cleanup. (This is useful for debugging.)
Returns: (str) pathname of a new temporary file
-
pygcam.utils.
getYearCols
(years, timestep=5)¶ Generate a list of names of year columns in GCAM result files from a string indicating a year range.
Parameters: - years – (str) a string of the form “2020-2050”
- timestep – (int) the number of years between timesteps
Returns: (list of strings) the names of the corresponding columns
-
pygcam.utils.
importFrom
(modname, objname, asTuple=False)¶ Import modname and return reference to objname within the module.
Parameters: - modname – (str) the name of a Python module
- objname – (str) the name of an object in module modname
- asTuple – (bool) if True a tuple is returned, otherwise just the object
Returns: (object or (module, object)) depending on asTuple
-
pygcam.utils.
importFromDotSpec
(spec)¶ Import an object from an arbitrary dotted sequence of packages, e.g., “a.b.c.x” by splitting this into “a.b.c” and “x” and calling importFrom().
Parameters: spec – (str) a specification of the form package.module.object Returns: none Raises: PygcamException – if the import fails
-
pygcam.utils.
loadModuleFromPath
(modulePath, raiseOnError=True)¶ Load a module from a ‘.py’ or ‘.pyc’ file from a path that ends in the module name, i.e., from “foo/bar/Baz.py”, the module name is ‘Baz’.
Parameters: - modulePath – (str) the pathname of a python module (.py or .pyc)
- raiseOnError – (bool) if True, raise an error if the module cannot be loaded
Returns: (module) a reference to the loaded module, if loaded, else None.
Raises: PygcamException
-
pygcam.utils.
loadObjectFromPath
(objName, modulePath, required=True)¶ Load a module and return a reference to a named object in that module. If ‘required’ and the object is not found, an error is raised, otherwise, None is returned if the object is not found.
Parameters: - objName – (str) the name of an object to find in the modulePath
- modulePath – (str) the path to a python module to load
- required – (bool) if True and the object cannot be loaded, raise an error.
Returns: a reference to the loaded object, if loaded. If not loaded and required is False, return None.
Raises: PygcamException
-
pygcam.utils.
mkdirs
(newdir, mode=504)¶ Try to create the full path newdir and ignore the error if it already exists.
Parameters: newdir – the directory to create (along with any needed parent directories) Returns: nothing
-
pygcam.utils.
parseTrialString
(string)¶ Converts a comma-separated list of ranges into a list of numbers. Ex. 1,3,4-6,2 becomes [1,3,4,5,6,2]. Duplicates are deleted. This function is the inverse of
createTrialString()
.Parameters: string – (str) comma-separate list of ints or int ranges indicated by two ints separated by a hyphen. Returns: (list) a list of ints
-
pygcam.utils.
printSeries
(series, label, header='', loglevel='DEBUG')¶ Print a series of values, with a give label.
Parameters: - series – (convertible to pandas Series) the values
- label – (str) a label to print for the data
Returns: none
-
pygcam.utils.
pushd
(*args, **kwds)¶ Context manager that changes to the given directory and then returns to the original directory. Usage is
with pushd('/foo/bar'): ...
:param directory: (str) a directory to chdir to temporarily :return: none
-
pygcam.utils.
readScenarioName
(configFile)¶ Read the file configFile and extract the scenario name.
Parameters: configFile – (str) the path to a GCAM configuration file Returns: (str) the name of the scenario defined in configFile
-
pygcam.utils.
removeFileOrTree
(path, raiseError=True)¶ Remove a file or an entire directory tree. Handles removal of symlinks on Windows, as these are treated differently in that system.
Parameters: - path – (str) the pathname of a file or directory.
- raiseError – (bool) if True, re-raise any error that occurs during the file operations, else errors are ignored.
Returns: none
-
pygcam.utils.
resourceStream
(relpath)¶ Return a stream on the resource found on the given path relative to the pygcam package.
Parameters: relpath – (str) a path relative to the pygcam package Returns: (file-like stream) a file-like buffer opened on the desired resource.
-
pygcam.utils.
saveToFile
(txt, dirname='', filename='')¶ Save the given text to a file in the given directory.
Parameters: - txt – (str) the text to save
- dirname – (str) path to a directory
- filename – (str) the name of the file to create
Returns: none
-
pygcam.utils.
shellCommand
(command, shell=True, raiseError=True)¶ Run a shell command and optionally raise PygcamException error.
Parameters: - command – the command to run, with arguments. This can be expressed either as a string or as a list of strings.
- shell – if True, run command in a shell, otherwise run it directly.
- raiseError – if True, raise ToolError on command failure.
Returns: exit status of executed command
Raises: ToolError
-
pygcam.utils.
simpleFormat
(s, varDict)¶ Simple version of str.format that does not treat ‘.’ as an attribute reference.
Parameters: - s – (str) string with args in curly braces
- varDict – (dict) dictionary of var names and values
Returns: (str) formatted string
-
pygcam.utils.
systemOpenFile
(path)¶ Ask the operating system to open a file at the given pathname.
Parameters: path – (str) the pathname of a file to open Returns: none
-
pygcam.utils.
unixPath
(path, rmFinalSlash=False)¶ Convert a path to use Unix-style slashes, optionally removing the final slash, if present.
Parameters: - path – (str) a pathname
- rmFinalSlash – (bool) True if a final slash should be removed, if present.
Returns: (str) the modified pathname
-
pygcam.utils.
writeXmldbDriverProperties
(outputDir='.', inMemory=True, filterFile='', batchFile='', batchLog='')¶ Write a XMLDBDriver.properties file using the values passed in the arguments.
Parameters: - outputDir – (str) where to write the file
- inMemory – (bool) if True, the
in-memory
attribute is set to True - filterFile – (str) a file that filters GCAM query output to limit what’s written to the database
- batchFile – (str) the path to an XML batch query file
- batchLog – (str) the path to a log file into which to direct batch query messages (queries can be pretty verbose...)
Returns: none