What’s new#
v1.26.0 (2024-03-27)#
Expose
sphinx
utilities as extensions ingenno.compat.sphinx
(PR #137).
v1.25.0 (2024-03-26)#
Migration notes#
The
Quantity
class implementation is simplified in this release. The changes should not be noticeable and should require no action for users who do not:change the default Quantity implementation (
AttrSeries
)use both Quantity implementations within the same code including test suites, for instance via the
parametrize_quantity_class()
test fixture, oruse
Quantity
for type annotation.
Users who do either (a) or (b): see
set_class()
for hints on how toimport Quantity
safely so that the correct class is instantiated.Users who do (c) should generally change
Quantity
toAnyQuantity
in type hints:from typing import TYPE_CHECKING if TYPE_CHECKING: from genno.types import AnyQuantity def my_operator(qty: "AnyQuantity", arg: int) -> "AnyQuantity": ...
The global variable
genno.config.STORE
is deprecated. Instead of:import genno.config genno.config.STORE.add("my_config_key")
…register the built-in
store()
handler for the target configuration key/section:from genno.config import handles, store handles("my_config_key", False, False)(store)
All changes#
Improvements to
Quantity
(PR #135).The constructor can use
xarray.DataArray
-style arguments (data=...
andcoords=...
) directly, without a need to first instantiate a DataArray.The binary operations
+ - * / **
are supported on Quantity objects directly, with Quantity as the left and/or right operand. Units, name, and other attributes are preserved.
New top-level functions
set_class()
to choose the Quantity implementation (AttrSeries
orSparseDataArray
);get_class()
to query the current settings, andassert_quantity()
(PR #135)New operators:
as_quantity()
,clip()
,unique_units_from_dim()
,where()
(PR #135). New corresponding methodsQuantity.clip()
andQuantity.where()
.New operator in Pyam (compat.pyam):
quantity_from_iamc()
(PR #135).as_pyam()
can accept “model” and “scenario” names from separate keyword arguments.caching
now uses the Apache Parquet file format instead of Python’spickle
forQuantity
andpandas.DataFrame
(#128, PR #135). This reduces file size and increases input/output speed.If no other location is configured, cached files are stored and read in the
platformdirs.user_cache_path()
, (PR #135).SparseDataArray
can be instantiated withint
data (PR #135). Becausesparse
does not support nullable integer dtypes, values are automatically cast tofloat
and a warning is logged.Configuration handling is simplified using a
ConfigHandler
class (PR #135).
v1.24.1 (2024-03-14)#
Handle pandas Copy-on-Write (CoW) set to
True
(PR #133). This setting will be the default in pandas v3.0 (expected April or May 2024). It is already pre-emptively enabled in some packages (such as dask-expr) sometimes used withgenno
. This can cause errors in other packages that expect the setting to beFalse
.Add
disable_copy_on_write()
compatibility function,parametrize_copy_on_write()
test fixture.Use
disable_copy_on_write()
inPlot.save()
.
v1.24.0 (2024-02-09)#
v1.23.1 (2024-02-01)#
Bugfix: genno 1.23.0 would return
AttrSeries
withpandas.Index
(instead of 1-Dpandas.MultiIndex
) fromsqueeze()
(#120, PR #122).
v1.23.0 (2024-01-26)#
New operators in SDMX (compat.sdmx):
dataset_to_quantity()
,quantity_to_dataset()
,quantity_to_message()
(#21, PR #115).
v1.22.0 (2023-12-13)#
New attribute
Plot.path
, allowing control of the full path used to write plots (PR #110).Bugfix:
AttrSeries.sel()
with a scalar indexer (for instance,qty.sel(x="foo")
) formerly did not drop the selected dimension; this was in contrast toxarray.DataArray.sel()
. The behaviour is now consistent (PR #110):qty.sel(x=["foo"])
, a length-1 sequence of indexers: the dimension is retained.qty.sel(x="foo")
, a single scalar indexer: the dimension is dropped.
Small fixes in
SparseDataArray
(PR #110):The
Quantity.name
is preserved when anxarray.DataArray
is passed to the constructor.to_series()
works with 0-D (scalar) quantities.
Provide typed signature for
Quantity.squeeze()
for the benefit of downstream applications (PR #110).
v1.21.0 (2023-11-28)#
write_report()
andconcat()
are single-dispatch functions for simpler extension in user code (PR #108).New argument to
write_report()
:kwargs
, including “header_comment” to write a header comment at the start of a.csv
file (PR #108).Fix many cross-references in the documentation (PR #108).
v1.20.0 (2023-10-28)#
As indicated in the v1.18.0 (2023-08-31) release notes, genno.computations
is renamed genno.operator
(PR #105).
The former module can still be imported and used to access particular operators, but will trigger a FutureWarning
.
Migration notes#
Adjust imports of/from genno.computations
to genno.operator
.
All changes#
genno supports and is tested on Python 3.12 (PR #105).
operator.add()
,div()
, andsub()
gain the same “helper” functionality asmul()
: when used withComputer.add()
, the returned key will automatically include the expected dimensions of the result (PR #105). See the documentation ofadd_binop()
.The standard Python operators
+
,-
,*
, and/
can be used withQuantity
objects directly; the resulting values, units, and dimensions are the same as usingadd()
etc. respectively (PR #105).Improved
Key
arithmetic (PR #105). See the documentation.Computer.apply()
will return anyKey
or keys that are returned or yielded by the function passed as its first argument (PR #105).
v1.19.0 (2023-09-11)#
Allow use of regular expressions in
aggregate()
(#35, PR #103).Improve documentation of
add_queue()
(#18, PR #103).New compatibility module for SDMX (compat.sdmx) (PR #103).
Ensure consistent
drop=True
behaviour ofselect()
andAttrSeries.sel()
withSparseDataArray.sel()
(PR #103). See the docstring for details.Creating a 1-D
AttrSeries
with an “anonymous” (un-named) dimension or index results in a dimension name"dim_0"
, consistent withxarray.DataArray
(PR #103).Don’t allow exceptions to raise from
Graph.__contains__()
; returnFalse
(PR #103).Reduce verbosity of logging from
collect_units()
(PR #103).Provide typed signatures for
Quantity.shape()
andsize
for the benefit of downstream applications (PR #103).
v1.18.1 (2023-08-31)#
Show the caller’s context in
DeprecationWarning
fromComputer
methods that wereBugfix: restore behaviour of genno ≤ 1.17.2 in
config.aggregate()
(PR #101). Specifically: when there are multiple_quantities:
to be aggregated, a failure to match any one key results in the whole item failing and being re-appended to the queue to be retried after other configuration items. Giving_fail: warning
(or anything less than “error”) causes the behaviour to be permissive: missing keys are logged but tolerated. This functionality was broken in 1.18.0.Allow for zero positional/only keyword arguments when formatting a
DeprecationWarning
fromComputer.convert_pyam()
(PR #101).
v1.18.0 (2023-08-31)#
This release adjusts the documentation by using language more carefully and precisely in line with Dask
(#34):
A computation is any entry in the
Computer.graph
: even a simple alias, or a list of other keys with no specific operation to be performed.A task is a specific kind of computation: a tuple that consists of a callable first item (usually a function) and other items—including keys and literal values—that are arguments to that callable.
An operator is a function or callable that is used as the first item in a task. The new
Operator
class is named to align with this meaning.
To complete this shift, in future releases of genno
:
The module
genno.computations
will be renamed togenno.operator
. At this point, imports fromgenno.computations
will continue to function, but will trigger aFutureWarning
.genno.computations
will be removed entirely. This will happen no sooner than 6 months after (1), and with at least 1 minor version in between.
Migration notes#
Code that uses the deprecated Computer
convenience methods can be adjusted to use the corresponding add_tasks()
helpers—which give equivalent behaviour—via Computer.add()
.
See the documentation of the deprecated methods and/or warnings at runtime for examples and hints.
Computer.add_file()
→ useload_file()
and its helper.Computer.add_product()
→ usemul()
and its helper.Computer.aggregate()
→ useaggregate()
orsum()
and its helper.Computer.convert_pyam()
→ useas_pyam()
and its helper.Computer.disaggregate()
anddisaggregate_shares()
: usemul()
and its helper.
For Plot.make_task()
similarly change, for instance, c.add("plot", DemoPlot.make_task("x:t", "y:t"))
to c.add("plot", DemoPlot, "x:t", "y:t")
.
All changes#
New class
Operator
(PR #98). This class allows to combine a function/callable for use in computations with an optionalhelper()
convenience method for adding tasks to aComputer
.Computer.add()
calls these helpers automatically, if they exist.New method
Computer.eval()
for using Python code-like expressions to define tasks and keys (PR #97).-
New method
Key.rename()
.Key supports the Python operations
+
(=add_tag()
),*
(=append()
a dimension),/
(=drop()
a dimension).
Add
.computations.sub
(PR #97`).Provide typed signatures for
Quantity.astype()
,data
, andpipe()
, and__neg__()
for the benefit of downstream applications (PR #97).~.genno.computations.concat
handles N-dimensional quantities with dimensions in any order (#38, PR #97).~.computations.pow
will derive units if the exponent is a Quantity with all identical integer values (PR #97).Adjust for pandas 2.1.0 to prevent
RecursionError
that could occur usingAttrSeries.sel()
(PR #99).Deprecations:
Computer.add_file()
,add_product()
,aggregate()
,convert_pyam()
, anddisaggregate()
(PR #98).Plot.make_task()
; the Plot class now has aadd_tasks()
method, analogous toadd_tasks()
, and so a Plot subclass can be provided directly toComputer.add()
(PR #98).
v1.17.2 (2023-07-11)#
Improve
Computer.visualize()
; see the example usage and output (PR #92, PR #95).aggregate()
tolerates missing labels along any of the dimensions to be aggregated (PR #95). Previously, this would raiseKeyError
.Performance improvement: avoid creating a large, empty
DataArray
in order to provideAttrSeries.coords
(PR #95). In some cases this would triggerMemoryError
, a segmentation fault, or similar, even though the actual data was smaller than available memory.
v1.17.1 (2023-05-30)#
Adjust for changed exception types in Pint 0.22 (PR #90).
v1.17.0 (2023-05-15)#
Bug fix: genno v1.16.1 (PR #85) introduced
ValueError
for some usages of.computations.sum
(PR #88).Provide typed signatures for
Quantity.bfill()
,cumprod()
,ffill()
for the benefit of downstream applications (PR #88).Ensure and test that
Quantity.name
andunits
pass through all~genno.computations
, in particularaggregate()
,convert_units()
, andsum()
(PR #88).Simplify arithmetic operations (
div()
,mul()
,pow()
) so they are agnostic as to theQuantity
class in use (PR #88).Ensure
AttrSeries.index
is alwayspandas.MultiIndex
(PR #88).
v1.16.1 (2023-05-13)#
Bug fix:
select()
raisedKeyError
if the indexers contained values not appearing in the coords of theQuantity
(PR #85). This occurred with pandas 2.x, but not with earlier versions. The documentation now states explicitly that extraneous values are silently ignored.All
~genno.computations
are type hinted for the benefit of downstream code (PR #85).Implement
AttrSeries.shape
(PR #85).Bug fix:
Computer.add()
now correctly handles positional-only keyword arguments to computations that specify these (PR #85).
v1.16.0 (2023-04-29)#
genno supports and is tested on Python 3.11 (PR #83).
Update dependencies (PR #83):
General:
importlib_resources
(the independent backport ofimportlib.resources
) is added for Python 3.9 and earlier.genno[sparse]
: new set of optional dependencies, includingsparse
. Install this set in order to useSparseDataArray
forQuantity
.Note that sparse depends on
numba
, and thusllvmlite
, and both of these package can lag new Python versions by several months. For example, as of this release, they do not yet support Python 3.11, and thussparse
andSparseDataArray
can only be used with Python 3.10 and earlier.genno[tests]
:ixmp
is removed;jupyter
andnbclient
are added. Testing utilities ingenno.testing.jupyter
are duplicated fromixmp.testing.jupyter
.
Adjust
AttrSeries.interp()
for compatibility with pandas 2.0.0 (released 2023-04-03) (PR #81).
v1.15.0 (2023-04-02)#
dask.core.quote()
is now re-exported bygenno
for convenience (PR #79):from genno import quote
Add new computations
drop_vars()
,round()
(PR #79).Improve existing computation
div()
to accept a scalar numerator. (PR #79):Improve
AttrSeries
(PR #79):Handle
dim=None
incumprod()
.Handle 1-D quantities in
shift()
.Preserve
AttrSeries.name
in more operations.
Computer.configure()
can accept adict
instead of keyword arguments (PR #79).Adjust for compatibility with pyam 1.7 (released 2022-12-19) (PR #78).
v1.14.1 (2022-10-28)#
Fix
cumprod()
for 1-dimensionalAttrSeries
(PR #74).Adjust for compatibility with pint 0.20 (released 2022-10-25) (PR #74).
v1.14.0 (2022-09-27)#
Add new computations
assign_units()
andconvert_units()
. These have simpler behaviour thanapply_units()
and should be preferred in most situations (PR #72).Following pandas (v1.4.0, released 2022-01-22) and xarray (v0.21.0, released 2022-01-27), support for Python 3.7 is dropped (PR #72). genno supports and is tested on Python 3.8 and newer.
v1.13.0 (2022-08-17)#
1-dimensional quantities are handled in
relabel()
and as weights insum()
(PR #68).load_file()
will read a header comment like# Units: kg / s
and apply the indicated units to the resulting quantity (PR #68).div()
andmul()
become the canonical names, matchingoperator
and other parts of the Python standard library (PR #68).ratio()
andproduct()
are retained as aliases, for compatibility.Ensure data passed to
Plot.generate()
has a “value” column; use short units format by default (PR #68).
v1.12.0 (2022-07-18)#
New computation
index_to()
(PR #65).general: configuration items are more flexible (PR #65).
comp: null
or omitted allows to specify a simple collection of other computations.A bare string
key:
is left as-is; only keys with (a) dimension(s) and/or tag are parsed toKey
.
Bug fix:
Computer.convert_pyam()
handles its tag argument correctly, generating keys likefoo:x-y-z:iamc
orbar::iamc
when applied to existing keys likefoo:x-y-z
orbar
(PR #65). Previously the generated keys would be e.g.bar:iamc
, which incorrectly treats “iamc” as a (sole) dimension rather than a tag.Computer.require_compat()
can handle arbitrary module names as strings, as well as module objects (PR #63).
v1.11.0 (2022-04-20)#
Migration notes#
The index keyword argument to Computer.add_single()
/ add()
is deprecated (PR #60) and will be removed in or after v3.0.0.
Indexing behaviour changes slightly: Computer.full_key()
always returns the Key
which matches its arguments and has the greatest number of dimensions.
For instance:
c.add_product("foo", "bar:a-b-c", "baz:x-y-z", sums=True)
…will generate a key <foo:a-b-c-x-y-z>
and all partial sums over subsets of its dimensions; c.full_key("foo")
will return this key.
Care should be taken to avoid adding 2+ keys with the same name, tag, and number of dimensions:
c.add("foo:a-b-c", ...)
c.add("foo:l-m-n", ...)
c.add("foo:x-y-z", ...)
This situation is ambiguous and the behaviour of Computer.full_key()
is undefined.
Instead, add a tag
to disambiguate.
All changes#
Key.product()
acceptsstr
arguments (PR #60).New class method
Key.bare_name()
(PR #60).Infer dimensions for e.g.
X:*:tag
in general: configuration items (PR #60).Handle the fail argument to
Computer.aggregate()
through aggregate: configuration items (PR #60).New computations
relabel()
andrename_dims()
(PR #60).Improve perfomance for adding large number of computations to
Computer
(PR #60).
v1.10.0 (2022-03-31)#
Add
testing.assert_units()
(PR #58).
v1.9.2 (2022-03-03)#
Silence
collect_units()
when units are explicitly “”, rather thanNone
(PR #56).Add explicit implementations of
__radd__()
,__rmul__()
,__rsub__()
and__rtruediv__()
for e.g.4.2 * Quantity(...)
(PR #55)Improve typing of
Quantity.shift()
(PR #55)
v1.9.1 (2022-01-27)#
Note that installing genno[pyam]
(including via genno[compat]
) currently forces the installation of an old version of pint
; version 0.17 or earlier.
Users wishing to use genno.compat.pyam
should first install genno[pyam]
, then pip install --upgrade pint
to restore a recent version of pint (0.18 or newer) that is usable with genno.
.computations.concat
works withAttrSeries
with misaligned dimensions (PR #53).Improve typing of
Quantity
andComputer
to help with using mypy on code that usesgenno
(PR #53).
v1.9.0 (2021-11-23)#
Fix error messages raised by
AttrSeries.sel()
on incorrect usage (PR #52).genno
no longer supports Python 3.6 or earlier, followingxarray
(PR #52).
v1.8.1 (2021-07-27)#
Bug fixes#
pathlib.Path
not serialized correctly incaching
(PR #51).
v1.8.0 (2021-07-27)#
Improve caching (PR #50):
Handle a lambda functions in
cache()
-decorated code.Add
Encoder.register()
andEncoder.ignore()
for downstream code to extend hashing of function arguments into cache keys.Expand docs.
v1.7.0 (2021-07-22)#
Add
.computations.interpolate
and supportingAttrSeries.interp()
(PR #48). This code works around issues pandas#25460 and pandas#31949.Computer.cache()
now also invalidates cache if the compiled bytecode of the decorated function changes (PR #48).Separate and expand docs of Caching to show how to check modification time and/or contents of files (#49, PR #48).
Add
Quantity.units
attribute for access to units (PR #48).AttrSeries.dims
andAttrSeries.coords
behave likeDataArray
for 1-D quantities (PR #48)
v1.6.0 (2021-07-07)#
Add
Key.permute_dims()
(PR #47).Improve performance of
Computer.check_keys()
(PR #47).
v1.5.2 (2021-07-06)#
v1.5.1 (2021-07-01)#
Bug fix:
infer_keys()
raisesAttributeError
under some circumstances (PR #45).
v1.5.0 (2021-06-27)#
Adjust
test_assign_coords()
for xarray 0.18.2 (PR #43).Make
Key.dims
order-insensitive so thatKey("foo", "ab") == Key("foo", "ba")
(PR #42); make corresponding changes toComputer
(PR #44).Fix “
AttributeError
: ‘COO’ object has no attribute ‘item’” onSparseDataArray.item()
(PR #41).
v1.4.0 (2021-04-26)#
plotnine.Plot.save()
automatically converts inputs (specified withPlot.inputs
) fromQuantity
toDataFrame
, but others (e.g. basic Python types) are passed through unchanged (PR #40).plotnine.Plot.save()
generates no output file ifgenerate()
returnsNone
/emptylist
.Quote
dict
argument toComputer.aggregate()
(for grouped aggregation) to avoid collisions between its contents and other graph keys.
v1.3.0 (2021-03-22)#
Bump minimum version of
sparse
from 0.10 to 0.12 and adjust to changes in this version (PR #39)Remove
SparseDataArray.equals()
, obviated by improvements insparse
.
Improve
AttrSeries
(PR #39)Implement
drop_vars()
andexpand_dims()
.assign_coords()
can relabel an entire dimension.sel()
can acceptDataArray
indexers and rename/combine dimensions.
v1.2.1 (2021-03-08)#
Bug fix: Provide abstract
Quantity.to_series()
method for type checking in packages that depend ongenno
.
v1.2.0 (2021-03-08)#
Quantity
becomes an actual class, rather than a factory function;AttrSeries
andSparseDataArray
are subclasses (PR #37).AttrSeries
gains methodsbfill()
,cumprod()
,ffill()
, andshift()
(PR #37).computations.load_file
uses the skipinitialspace parameter topandas.read_csv()
; extra dimensions not mentioned in the dims parameter are preserved (PR #37).AttrSeries.sel()
acceptsxarray.DataArray
for xarray-style indexing (PR #37).
v1.1.1 (2021-02-22)#
Bug fix:
Computer.add_single()
incorrectly callscheck_keys()
on iterables (e.g.pandas.DataFrame
) that are not computations (PR #36).
v1.1.0 (2021-02-16)#
.computations.add
transforms compatible units, and raises an exception for incompatible units (PR #31).Improve handling of scalar quantities (PR #31).
Plot
is fault-tolerant: if any of the input quantities are missing, it becomes a no-op (PR #31).Computer.configure()
accepts a fail argument, allowing partial handling of configuration data/files, with errors logged but not raised (PR #31).New
.computations.pow
(PR #31).
v1.0.0 (2021-02-13)#
Adjust for usage by
ixmp.reporting
andmessage_ix.reporting
(PR #28):Reduce minimum Python version to 3.6. This is lower than the minimum version for xarray (3.7), but matches ixmp, etc.
Remove submodule
compat.ixmp
; this code has been moved toixmp.reporting
, replacing what was there. Likewise, remove submodulecompat.message_ix
.Simplify the form & parsing of
iamc:
section entries in configuration files:Remove unused feature to add
group_sum()
to the chain of tasks.Keys now conform more closely to the arguments of
Computer.convert_pyam()
.
Move argument-checking from
as_pyam()
toconvert_pyam()
.- Simplify semantics of
genno.config.handles()
decorator. Remove
CALLBACKS
feature, for now.
- Simplify semantics of
Computer.get_comp()
andrequire_compat()
are now public methods.Expand tests.
Protect
Computer
configuration fromdask.optimization.cull()
; this prevents infinite recursion if the configuration contains strings matching keys in the graph. Addunquote()
(#25, PR #26).Simplify
collect_units()
and improve unit handling inratio()
(#25, PR #26).Add file-based caching via
Computer.cache()
andgenno.caching
(#20, PR #24).
v0.4.0 and earlier#
v0.4.0 (2021-02-07)#
Add file-based configuration in
genno.config
and associated documentation (#8, PR #16).
v0.3.0 (2021-02-05)#
Add Plotnine (compat.plotnine) compatibility (PR #15).
Add a Concepts and usage overview to the documentation (PR #13).
v0.2.0 (2021-01-18)#
Increase test coverage to 100% (PR #12).
Port code from
message_ix.reporting
(PR #11).Add
compat.pyam
.Add a name parameter to
load_file()
.
v0.1.0 (2021-01-10)#
Initial code port from
ixmp.reporting
.