Programming style

Python lets you write concise, readable code that you can maintain and further develop with reasonable effort. However, code quality also depends on the programmer’s experience and available time. In hydrology, PhD students with little programming experience often write much model code under pressure not only to get their model running but also to tackle their scientific questions and publish their results. The resulting source code is understandably often a mess. Even the relatively good software results often prove inadequate when transferring the software into practical applications or sharing it with other researchers.

During the long development of HydPy, which also started as a quick side project during a PhD thesis, we made many misleading design decisions ourselves. However, through extensive refactoring and consolidation, we arrived at a software architecture that, in our opinion, should be easily extensible and applicable in many contexts.

This section defines the steadily growing “HydPy Style Guide”, an attempt to explain the principles in the development of HydPy and make sure the contributions of different developers are as consistent as possible. Please understand the “HydPy Style Guide” as a refinement of PEP 8 — the “official” Style Guide for Python Code. PEP 8 gives coding conventions that help to write clear code. If everyone follows these conventions, diving into existing source code becomes much more straightforward, as you spend less effort unravelling the mysteries of overly creative programming solutions.

In some regards, the HydPy Style Guide deviates from PEP 8, primarily due to the following two aims:

• We design the HydPy framework as a Python library applicable to hydrologists with little or no programming experience. Ideally, users of such a framework should not even notice they are writing valid Python code while preparing configuration files or working interactively in the Python shell.

• We try to close the gap between the model code, model documentation and model tests as well as possible. By reading (and testing) the documentation of a specific model, one should understand exactly how this model works within the corresponding version of the HydPy framework.

When contributing to the codebase, be aware that even slight changes can significantly affect the applicability of HydPy, and future developers must cope with your work. So, always check for possible side effects of your code changes. Structure your code with a clear (mainly object-oriented) design, and use Black to format it automatically. Refactor thoroughly enough to avoid code duplication. Last but not least, create well-thought-out APIs for your objects so everyone can use them smoothly in both doctests and the Python shell.

We use Black and Pylint in our Travis CI continuous integration workflow. Black checks that all committed files follow its standards. Pylint is an additional style checker that recognises missing documentation sections, repeated or inconsistent method definitions, and much more. The pylintrc file configures the general behaviour and strictness of Pylint. You may turn off some checks locally if you provide a good explanation. At best, add a link to a related issue explaining why Pylint is wrong in your particular code section, using the following pattern:

>>> # pylint: disable=abstract-method
>>> # due to pylint issue https://github.com/PyCQA/pylint/issues/179

This section describes some specific conventions for the development of HydPy but is not guidance on how to write good source code in general. If you have little programming experience, start by learning the basics of Python through tutorials. Afterwards, improve your knowledge of code quality by reading more advanced literature, such as this book on object-oriented design.

Project structure

For HydPy, we prefer a flat folder structure with two subpackage levels. The individual modules can be of arbitrary length to cover particular topics completely. For example, module parametertools defines all base classes for creating model-specific parameter classes and related collection classes.

Subpackage core provides the essential features of HydPy, used for implementing hydrological models and workflows. One example is module parametertools. Modules defined in subpackage core should never import features provided by modules of other subpackages, except those of subpackage cythons.

Subpackage auxs provides auxiliary features, only necessary for selected HydPy models and applications. One example is the module anntools defining artificial neural network classes usable as complex model parameters, currently relevant for the dam model only. Modules defined in subpackage auxs are allowed to import features from subpackages core and cythons.

Subpackage models contains the implemented hydrological models. Base models as dam are additional subpackages, providing, for example, different kinds of sequence classes in separate submodules. Application models as dam_v001, selecting valid combinations of base model features, are defined within single modules. Please follow the naming patterns of the modules of the already available models carefully when implementing new ones.

Subpackage exe provides features that ease the execution of HydPy. Module commandtools (in combination with script hyd), for example, allows controlling HydPy from the command line.

Subpackage cythons is related to all Cython features of HydPy:

• It implements functionalities for “cythonizing” the Python models defined in subpackage models.

• It contains Cython extension files, which mostly correspond to Python modules of other subpackages. For example, the extension file annutils provides time-critical implementation details to module anntools.

• It contains the additional subpackage autogen, including all automatically generated extension files and Dynamic Link Library files (pyd files on Windows and so files on Linux).

Extension files should not import any features from other subpackages. Python files that control the automatic generation of extension files can import from subpackage core.

Note that the names of the modules of subpackages core, auxs, and exe end in almost all cases with “tools” and those of the modules and extension files of subpackage cythons with “utils”, which helps to identify the different module types immediately and to circumvent name conflicts between and within modules.

Subpackage conf contains configuration files (currently XML schema files and coefficients for numerical integration algorithms), which might be generated automatically during HydPy’s build process.

Subpackage data provides example data usable within doctests, currently only the HydPy-H-Lahn example project.

Subpackage docs contains different subpackages. sphinx controls the automatic generation of the HTML documentation. rst contains all reStructuredText files written manually. figs contains all manually generated figures in the PNG format. After the build process, html contains the Plotly plots automatically generated during testing. Note that the actual HTML generation takes place in the auto folder, which the build process creates and fills with information.

Subpackage tests deals with testing. Its subpackage iotesting is designated to store data temporarily during testing.

Imports

As recommended in PEP 8, clarify your import sources. Always use the following pattern at the top of a new module and list the imports of a section in alphabetical order:

>>> import os
>>> import sys
...
>>> import numpy
...
>>> from hydpy.core import sequencetools
>>> from hydpy.cythons import pointerutils

Note that each import command stands on a separate line. Always import complete modules from HydPy without changing their names. — No wildcard imports!

We lift the wildcard ban for writing configuration files. For example, when using parameter control files, it’s not always convenient to write something like this:

>>> from hydpy.models import hland
>>> model = hland.Model()
>>> from hydpy.core import parametertools
>>> model.parameters = parametertools.Parameters({"model": model})
>>> model.parameters.control = hland.ControlParameters(model.parameters.control)
>>> model.parameters.control.nmbzones = 2
>>> model.parameters.control.nmbzones
nmbzones(2)

Here, a wildcard import (and the “magic” of function parameterstep()) allows for a much cleaner syntax:

>>> del model
>>> from hydpy.models.hland import *
>>> parameterstep("1d")
>>> nmbzones(2)
>>> nmbzones
nmbzones(2)

The wildcard import is acceptable here because there is only one import statement. There is no danger of name conflicts.

Besides the wildcard exception explained above, there is another one related to modelimports.

Defensive programming

HydPy is intended for researchers and practitioners who are not Python experts and may have little programming experience in general. Hence, it is desirable to anticipate errors due to misleading input as thoroughly as possible and report them as soon as possible. So, in contradiction to PEP 8, it is often preferable not to expose the names of simple public attributes. Whenever sensible, use protected attributes (defined by property or the more specific property features provided by module propertytools) to ensure that the internal states of objects remain consistent. One example is that it is not allowed to assign an unknown string to the outputfiletype of an instance of class SequenceManager:

>>> from hydpy.core.filetools import SequenceManager
>>> sequencemanager = SequenceManager()
>>> sequencemanager.filetype = "test"
Traceback (most recent call last):
  ...
ValueError: The given sequence file type `test` is not implemented.  Please choose one of the following file types: `npy`, `asc`, and `nc`.

Of course, extensive use of protected attributes increases source code length and slows computation time. However, regarding the first point, writing a graphical user interface would require much more source code (and still decrease flexibility). Regarding the second point, the computation times of the general framework functionalities discussed here should be negligible compared with the computation times of hydrological simulations in most cases.

Error messages

Unmodified Python error messages are often not sufficiently informative for HydPy applications for two reasons. First, someone unfamiliar with Python’s exception-handling system is likely to read them. Second, they do not indicate which hydrological context the problem occurs in. Knowing only that a parameter object of a particular type has been misspecified is little help without knowing which sub-catchment. Hence, try to add as much helpful information to error messages as possible. Additionally, follow the rules below so that all messages look alike. The same rules apply to warnings.

Write complete sentences that start with a capital letter and end with a period, and separate sentences by two spaces. Explain what is required and what happened instead, preferably in the form “… requires …, but …”.

Tell where the problem occurred using the functions elementphrase(), nodephrase(), devicephrase(), and submodelphrase(), which determine the relevant device name and already include the required backticks (see below):

>>> from hydpy.models.hland import *
>>> parameterstep("1d")
>>> from hydpy import Element
>>> e1 = Element("e1", outlets="n1")
>>> e1.model = model
>>> from hydpy.core import objecttools
>>> objecttools.elementphrase(k)
'`k` of element `e1`'
>>> k(hq=10.0, khq=2.0)
Traceback (most recent call last):
...
RuntimeError: For the alternative calculation of parameter `k` of element `e1`, either
the keyword argument `alpha` must be given or the value of parameter `alpha` must be
defined beforehand.

Another recommended approach is exception chaining via function augment_excmessage(), which adds context information to an error raised by code you call. Call it within except clauses only (preferably except BaseException). Let its prefix start with “While trying to” and end without punctuation. Function augment_excmessage() appends “, the following error occurred:” and the original message, so nested calls result in a readable chain of explanations:

>>> def set_threshold(value):
...     try:
...         return float(value)
...     except BaseException:
...         objecttools.augment_excmessage(
...             f"While trying to set the threshold value `{value}`"
...         )
>>> set_threshold("x")
Traceback (most recent call last):
...
ValueError: While trying to set the threshold value `x`, the following error occurred:
could not convert string to float: 'x'

If the original message would rather confuse users than help them, raise a new exception with from None instead.

Never write “(s)” or append plural suffixes manually if the relevant number is known when constructing the message. Instead, use the class methods of Inflect, which wrap the required functionalities of the inflect library (do not import inflect directly). plural() returns the singular or plural form of nouns and verbs, a() adds the indefinite article or returns the plural form, and count() combines a number with a noun and, optionally, a verb:

>>> from hydpy.core.objecttools import Inflect
>>> Inflect.plural("node", 2), Inflect.plural("is", 2)
('nodes', 'are')
>>> Inflect.a("output node", 1), Inflect.a("output node", 2)
('an output node', 'output nodes')
>>> def check_names(*names):
...     if (n := len(names)) != 2:
...         raise ValueError(
...             f"Exactly two node names are required, but "
...             f"{Inflect.count(n, 'name', 'is')} given."
...         )
>>> check_names("n1", "n2", "n3")
Traceback (most recent call last):
...
ValueError: Exactly two node names are required, but three names are given.
>>> check_names()
Traceback (most recent call last):
...
ValueError: Exactly two node names are required, but no names are given.

listing() combines a noun and, optionally, a verb with an enumeration of the given items:

>>> def check_nodes(*names):
...     if unknown := sorted(n for n in names if n not in ("n1", "n2")):
...         raise RuntimeError(
...             f"The {Inflect.listing('node', unknown, 'is')} unknown."
...         )
>>> check_nodes("n1", "n3")
Traceback (most recent call last):
...
RuntimeError: The node `n3` is unknown.
>>> check_nodes("n4", "n3")
Traceback (most recent call last):
...
RuntimeError: The nodes `n3` and `n4` are unknown.

Keep “(s)” only if the number is unknown at the time of constructing the message.

Make sure the message also reads well if an enumeration of varying length consists of a single item. For example, do not write “the following” in front of such enumerations (“the following node n3” sounds odd). “The following” is fine for introducing fixed lists after a colon (“Please choose one of the following modes: none and mean.”). Phrases like “any of” or “at least one of” require a separate wording for a single item (“is not an instance of class int” instead of “is not an instance of any of the classes int”).

Spell out counts up to ten and write larger counts in digits. This rule applies to cardinal numbers (“three values”) and ordinal numbers (“the third value”) and is the default behaviour of cardinal(), ordinal(), and count():

>>> Inflect.cardinal(10), Inflect.cardinal(11)
('ten', '11')
>>> Inflect.ordinal(3), Inflect.ordinal(22)
('third', '22nd')

For zero, count() writes “no” by default (“no names are given”). When a number stands alone, write “none” instead (“but none are given”):

>>> Inflect.cardinal(0, zero="none")
'none'

Write all other numbers, like positions, indices, and shapes, in digits.

Use backticks for names of HydPy and Python objects (devices, models, parameters, sequences, methods, classes, modules, keywords, options, files, and so on) and for values as users would type them or as repr() shows them. Do not use backticks for numbers that are part of the sentence, like counts and positions. When enumerating names, put each name into separate backticks but not the conjunction. listing() does this by default, and function enumeration() does it if you set its argument backticks to True. Values of a single vector form a single value, so put them together into one pair of backticks without a conjunction (see functions repr_values() and repr_list()). Some examples:

The value `1.5` of parameter `area` of element `e1` is invalid.
Parameter `k` requires exactly one argument, but three are given.
The nodes `n1`, `n2`, and `n3` are unavailable.
The given values `1.0, 2.0, 2.0` do not increase strictly monotonically.
The submodel at position 3 is missing.

Naming conventions

The naming conventions of PEP 8 apply. Additionally, we encourage naming classes and their instances as similar as possible whenever reasonable, often simply switching from CamelCase to lowercase, as shown in the following examples:

Class Name	Instance Name	Note
Sequences	sequences	each Model instance handles exactly one Sequence instance: model.sequences
InputSequences	inputs	“inputsequences” would be redundant for attribute access: model.sequences.inputs

If reasonable, each instance should define its preferred name via the name attribute:

>>> from hydpy.models.hland import *
>>> InputSequences(None).name
'inputs'

Classes like Element or Node, where names (and not namespaces) are used to differentiate between instances, should implement instance name attributes when reasonable:

>>> from hydpy import Node
>>> Node("gauge1").name
'gauge1'

Group instances of the same type in collection objects with the same name, except an attached letter “s”. For example, we store different Element objects in an instance of class Elements and different Node objects in an instance of the class Nodes.

Collection classes

The subsection above deals with the naming (of the instances) of collection classes. Additionally, consider the following recommendations when implementing new collection classes.

Each collection object must be iterable:

>>> from hydpy import Nodes
>>> nodes = Nodes("gauge1", "gauge2")
>>> for node in nodes:
...     print(repr(node))
Node("gauge1", variable="Q")
Node("gauge2", variable="Q")

To help users working interactively in the Python shell, collection objects should expose their handled objects as attributes and let the function “dir” return attribute names identical to the handled objects’ name attributes:

>>> nodes.gauge1
Node("gauge1", variable="Q")
>>> nodes.gauge2
Node("gauge2", variable="Q")
>>> "gauge1" in dir(nodes)
True

Also provide item access as a more type-safe, and eventually more efficient, alternative for writing complex scripts:

>>> nodes["gauge1"]
Node("gauge1", variable="Q")

Whenever useful, define convenience functions to simplify the handling of collection objects:

>>> nodes += Node("gauge1")
>>> nodes.gauge1 is Node("gauge1")
True
>>> len(nodes)
2
>>> "gauge1" in nodes
True
>>> nodes.gauge1 in nodes
True
>>> newnodes = nodes.copy()
>>> nodes is newnodes
False
>>> nodes.gauge1 is newnodes.gauge1
True
>>> nodes -= "gauge1"
>>> 'gauge1' in nodes
False

String representations

Be aware of the difference between str and repr(). Often, str is supposed to return strings describing objects in a condensed form for end-users when executing a program. In contrast, repr() should return strings containing all details of an object for developers debugging a program. Some argue that, given its limited use, giving repr() much attention is a waste of time in many cases. For HydPy, we think differently. Defining comprehensive repr() return values simplifies reading the doctests in the online documentation and working interactively in the Python shell, making it highly relevant for end-users, too. On the other hand, str is less relevant because it mainly serves as an alternative for generating exception messages. Hence, focus primarily on repr(), and use str when repr()’s return value is too complicated for exception messages.

A good return value of repr() is one that a non-Python programmer does not identify as a string. The first ideal case is that copy-pasting the string representation and evaluating it within the Python shell returns a reference to the same object.

A Python example:

>>> repr(None)
'None'
>>> eval("None") is None
True

A HydPy example:

>>> from hydpy import Node
>>> Node("gauge1")
Node("gauge1", variable="Q")
>>> eval('Node("gauge1", variable="Q")') is Node("gauge1")
True

In the second ideal case, evaluating the string representation results in an equal object.

A Python example:

>>> x = 1.5
>>> x
1.5
>>> eval("1.5") is x
False
>>> eval("1.5") == x
True

A HydPy example:

>>> from hydpy import Period
>>> Period("1d")
Period("1d")
>>> eval('Period("1d")') is Period("1d")
False
>>> eval('Period("1d")') == Period("1d")
True

For nested objects, the above goals may be hard to accomplish, but sometimes it’s worth it.

A Python example:

>>> [1., "a"]
[1.0, 'a']
>>> eval("[1.0, 'a']") == [1.0, "a"]
True

A HydPy example:

>>> from hydpy import Timegrid
>>> Timegrid("01.11.1996", "1.11.2006", "1d")
Timegrid("01.11.1996 00:00:00",
         "01.11.2006 00:00:00",
         "1d")
>>> eval('Timegrid("01.11.1996 00:00:00", "01.11.2006 00:00:00", "1d")') == Timegrid("01.11.1996", "1.11.2006", "1d")
True

For deeply nested objects, this strategy becomes infeasible, of course. Then try to find a way to “flatten” the string representation without losing too much information:

>>> from hydpy import Element, Elements
>>> Elements(Element("e_1", outlets="n_1"), Element("e_2", outlets="n_2"))
Elements("e_1", "e_2")

Finally, always consider using functions provided by module objecttools to simplify the definition of repr() and str return values and keep the string representations of different HydPy objects, at least to a certain degree, consistent. For example, use function repr_ to let the user control the maximum number of decimal places of scalar floating-point values:

>>> from hydpy import pub, repr_
>>> class Number(float):
...     def __repr__(self):
...         return repr_(self)
>>> pub.options.reprdigits = 3
>>> Number(1./3.)
0.333

Introspection

One nice feature of Python is its “introspection” capability, allowing analysis (and, when necessary, modification) of objects at runtime with little effort.

HydPy makes extensive use of these introspection features whenever they relieve non-programmers of writing code that does not deal with hydrological modelling directly. Section Imports discusses wildcard imports in parameter control files, where the real comfort comes from the “magic” implemented in function parameterstep(). Invoking this function not only sets the time interval length for the following parameter values. It also initialises a new model instance (if such an instance does not already exist) and directly exposes its control parameter objects in the local namespace. For the user’s comfort, each parameter control file purports to be a simple configuration file that somehow checks its own validity. On the downside, changing the operating principle of HydPy’s parameter control files requires more thought than a simpler, direct approach.

We encourage adding introspection features as long as they improve usability for non-programmers and don’t harm HydPy’s reliability. However, please be particularly cautious and document why and how thoroughly. To ensure traceability, one should usually add such code to modules like modelutils, importtools, and autodoctools. Module modelutils deals with all introspection needed to “cythonize” Python models automatically. Module importtools contains the function parameterstep() and related features. Module autodoctools improves automatic online documentation generation.

Typing

Python is a strongly but dynamically typed programming language, allowing you to write very condensed, readable, and flexible (scripting) code. However, missing type information has also its drawbacks. With the HydPy sources reaching a certain size, we began to introduce static typing annotations based on module typing. In our experience, the additional information helps a lot, allowing code inspection and refactoring tools to analyse and modify the code more efficiently. We plan to increase our efforts in this direction, but we don’t yet have a “HydPy Typing Style Guide”. Please add any typing annotations you find useful. The minimum requirement for Python core modules is to declare the return type (or, when necessary, to declare the union of possible return types) of each new function or method:

>>> def test(nmb) -> list[int]:
...     return list(range(nmb))

For Cython extension files, adding type information understandable to Python tools is of even greater importance. So, accompany each Cython extension file with a stub file that annotates all public (sub)members.

Implementing models

Please inspect the source files of the already available hydrological models in detail to understand how to implement new ones correctly. HydPy provides many standard features, allowing you to write straightforward model source code in many cases. However, you can implement any missing functionality (see, for example, the complex “connect” method defined by the exch_branch_hbv96 model). If those functionalities might matter to other models as well, consider generalising them and adding them to the appropriate subpackage.

The main effort in creating new models is not writing the source code, but thoroughly documenting it and proving it works correctly. Each docstring of a calculation method must contain at least a short description, lists of the required, calculated, and updated variables (linked via substitutions), the basic equation in LaTeX style, and doctests covering all anticipated usages of the method, even the unlikely ones. The docstrings of all Parameter or Sequence_ subclasses containing “special” source code (for example, modifications of trim()) must contain doctests addressing these code sections. Finally, write integration tests for each application model based on class IntegrationTest, explaining all model functionalities in detail both with text and Plotly plots, and preventing future regression by sufficiently complete tabulated calculation results.