Pyam (compat.pyam)

Package documentation

The “IAMC data structure” is a particular data structure with either 6 or 7 dimensions: model, scenario, region, variable, unit, either year or time, and optionally subannual. Data with this structure are usually stored in a tablular “IAMC format,” wherein each dimension is stored as one column, and the remaining column, labeled value, contains observation values.

Using add_as_pyam() (Computer.add(..., "as_pyam", ...)):

  • model and scenario are populated from the attributes of the object returned by the Computer key scenario;

  • variable contains the name(s) of each of the quantities, or others constructed by collapse (see below);

  • unit contains the units associated with each of the quantities; and

  • year, time, and optionally subannual can be created using rename or collapse operations.

A callback function (collapse) can be supplied that modifies the data before it is converted to an IamDataFrame; for instance, to concatenate extra dimensions into variable labels. Other dimensions can simply be dropped (with drop). Dimensions that are not collapsed or dropped will appear as additional columns in the resulting IamDataFrame; this is valid, but non-standard data per the IAMC format.

For example, here the labels for the MESSAGEix t (technology) and m (mode) dimensions are appended to a fixed string to construct variable labels:

c = Computer

def m_t(df):
   """Collapse `t` and `m` dimensions to an IAMC 'Variable' string."""
   df["variable"] = "Activity|" + df["t"] + "|" + df["m"]
   return df

ACT = c.full_key('ACT')
keys = c.add(ACT, "as_pyam", "ya", collapse=m_t, drop=["t", "m"])
genno.compat.pyam.HAS_PYAM = True

bool indicating whether pyam: analysis and visualization  of integrated-assessment & macro-energy scenarios is available.

as_pyam(scenario, quantity, *[, rename, ...])

Return a pyam.IamDataFrame containing the data from quantity.

add_as_pyam(func, c, quantities[, tag])

Computer.add() helper for as_pyam().

This module also registers implementations of concat() and write_report() that handle pyam.IamDataFrame objects.

genno.compat.pyam.operator.add_as_pyam(func, c: Computer, quantities: Key | str | Iterable[Key | str], tag='iamc', /, **kwargs)[source]

Computer.add() helper for as_pyam().

Add conversion of one or more quantities to the IAMC data structure.

Parameters:

  • quantities (str or Key or list of str or Key) – Keys for quantities to transform.

  • tag (str, optional) – Tag to append to new Keys.

  • kwargs – Any keyword arguments accepted by as_pyam().

Returns:

Each task converts a Quantity into a pyam.IamDataFrame.

Return type:

list of Key

genno.compat.pyam.operator.as_pyam(scenario, quantity: AnyQuantity, *, rename: Mapping[str, str] | None = None, collapse: Callable | None = None, replace={}, drop: Collection[str] | str = 'auto', unit=None, prepend_name: bool = True, model_name: str | None = None, scenario_name: str | None = None)[source]

Return a pyam.IamDataFrame containing the data from quantity.

Warnings are logged if the arguments result in additional, unhandled columns in the resulting data frame that are not part of the IAMC spec.

The conversion has the following steps:

  1. quantity is converted to a temporary pandas.DataFrame.

  2. Labels for the following IAMC dimensions are filled:

    • model, scenario: from attributes of the scenario, model_name, and/or scenario_name argument(s).

    • variable: from the name of quantity, if any.

    • unit: from the units of quantity, if any.

  3. The actions specified by the optional arguments rename, collapse, replace, drop, and unit, if any, are applied in that order.

  4. The resulting data frame is converted to pyam.IamDataFrame.

Parameters:

  • scenario – Any object with model and scenario attributes of type str, for instance an ixmp.Scenario or ScenarioInfo; or a str, which is equivalent to scenario_name.

  • quantity (Quantity) – Quantity to convert to IAMC data structure.

  • rename (dict, optional) – Mapping from dimension names in quantity (str) to column names (str); either IAMC dimension names, or others that are consumed by collapse.

  • collapse (typing.Callable, optional) – Function that takes a pandas.DataFrame and returns the same type. This function may collapse 2 or more dimensions, for example to construct labels for the IAMC variable dimension, or any other.

  • replace (optional) – Values to be replaced and their replaced. Passed directly to pandas.DataFrame.replace().

  • drop (str or collections.abc.Collection of str, optional) – Columns to drop. Passed to util.drop(), so if not given, all non-IAMC columns are dropped.

  • unit (str, optional) – Label for the IAMC unit dimension. Passed to clean_units().

  • prepend_name (bool, optional) – If True, the Quantity.name of quantity is prepended to the IAMC variable dimension.

  • model_name (str, optional) – Value for the IAMC model dimension.

  • scenario_name (str, optional) – Value for the IAMC scenario dimension.

Raises:

  • ValueError – If the resulting data frame has duplicate keys in the IAMC dimensions. pyam.IamDataFrame cannot handle such data.

  • TypeError – If both scenario and scenario_name are non-empty str.

genno.compat.pyam.operator.quantity_from_iamc(qty: AnyQuantity | pyam.IamDataFrame | pandas.DataFrame, variable: str, *, fail: int | str = 'warning') → AnyQuantity[source]

Extract data for a single measure from qty with IAMC-like structure.

Parameters:

  • qty – Must have at least 2 dimensions named ‘v’ (or ‘variable’, any case) and ‘u’ (or ‘unit’, any case).

  • variable (str) – Regular expression to match full labels on the v dimension of qty. If the expression contains match groups, they are used to rewrite v labels: only the contents of the first match group are kept. This may be used to discard a portion of the label.

Returns:

The ‘variable’ dimension contains reduced labels. The Quantity.units attribute contains the unique units for the subset of data.

Return type:

Quantity

See also

unique_units_from_dim

Configuration

compat.pyam adds a handler for a iamc: configuration file section.

pyam.iamc(info)

Handle one entry from the iamc: config section.

Computer-specific configuration.

Invokes add_as_pyam() and adds additional computations to convert data from Quantity to a pyam.IamDataFrame. Each entry contains:

variable: (str)

Variable name. This is used two ways: it is placed in the variable label of the resulting IamDataFrame; and the Computer key for executing the conversion is <variable>:iamc.

base: (str)

Key for the quantity to convert.

select: (dict, optional)

Keyword arguments to operator.select(). This selection is performed while data is in Quantity format, before it is passed to as_pyam().

rename: (dict, optional)

Passed to as_pyam().

replace: (dict, optional)

Passed to as_pyam().

drop: (list of str, optional)

Passed to as_pyam().

unit: (str, optional)

Passed to as_pyam().

Any further additional entries are passed as keyword arguments to collapse(), which is then given as the collapse callback for as_pyam().

collapse() formats the variable labels of the IamDataFrame. The variable name replacements from the iamc variable names: section of the config file are applied to all variables.

Utilities

genno.compat.pyam.util.IAMC_DIMS = frozenset({'model', 'region', 'scenario', 'time', 'unit', 'variable', 'year'})

Dimensions of the IAMC data structure used by pyam: analysis and visualization  of integrated-assessment & macro-energy scenarios.

genno.compat.pyam.util.clean_units(df: DataFrame, unit=None) → DataFrame[source]

Convert magnitudes and units of df to unit in str.

Raises:

ValueError – if there is more than one unit.

genno.compat.pyam.util.collapse(df: DataFrame, columns: Mapping[str, Sequence[str]] = {}, sep='|') → DataFrame[source]

Collapse columns into the IAMC columns of df.

genno.compat.pyam.util.drop(df: DataFrame, columns: Collection[str] | str) → DataFrame[source]

Drop columns if given, or all non-IAMC columns.