Programming style¶
Python allows for writing concise and easily readable software code that can be maintained and further developed with reasonable effort. However, code quality also depends on the experience and available time of the programmer writing it. In hydrology, much model code is written by PhD students with little programming experience that are under pressure not only to get their model running but also to tackle their scientific questions and publish their results. The source code resulting from such a rush is understandably often a mess. Even the relatively goodsoftware results often prove inadequate when transferring the software into practical applications or sharing it with other researchers.
In the long development process of HydPy, which also started as a quick side-project during a PhD thesis, we made many misleading design decisions ourselves. However, through much effort spent in periods of refactoring and consolidation, we came to a software architecture that, in our opinion, should be easily extensible and applicable in many contexts.
This section defines the steadily growing “HydPy Style Guide”, an attempt to explain the principles in the development of HydPy and make sure the contributions of different developers are as consistent as possible. Please understand the “HydPy Style Guide” as a refinement of PEP 8 — the “official” Style Guide for Python Code. PEP 8 gives coding conventions that help to write clear code. If everyone follows these conventions, diving into existing source code becomes much more straightforward, as one has less effort to unravel the mysteries of overly creative programming solutions.
In some regards, the HydPy Style Guide deviates from PEP 8, primarily due to the following two aims:
* We design the *HydPy* framework as a Python library applicable to
hydrologists with little or no programming experience. Ideally, such
framework users should not even notice that they write valid Python code
while preparing their configuration files or working interactively in the
Python shell.
* We try to close the gap between the model code, model documentation and
model tests as well as possible. By reading (and testing) the documentation
of a specific model, one should exactly understand how this model works
within the corresponding version of the *HydPy* framework.
When contributing to the code basis, be aware that even slight changes can significantly affect the applicability of HydPy, and future developers must cope with your work. So, always make sure to check for possible side-effects of your code changes. Structure your code in a clear (mainly object-oriented) design and use Black for automatically formatting your code. Refactor thoroughly enough to avoid code duplicates. Last but not least, create smartly thought-through APIs for your objects, allowing everyone to use them smoothly both within doctests and within the Python shell.
Be aware of the usage of Black and Pylint in our Travis CI continuous integration workflow. Black checks that all committed files follow its standards. Pylint is an additional style checker that recognises missing documentation sections, repeated or inconsistent method definitions, and much more. The pylintrc file configures the general behaviour and strictness of Pylint. You are allowed to disable some checks locally in case you provide a good explanation. At best, simply at a link to a related issue explaining why Pylint is wrong in your particular code section, using the following pattern:
>>> # pylint: disable=abstract-method
>>> # due to pylint issue https://github.com/PyCQA/pylint/issues/179
This section describes some specific conventions for the development of HydPy but is no guidance on how to write good source code in general. If you have little experience in programming, first make sure to learn the basics of Python through some Python tutorials. Afterwards, improve your knowledge of code quality through reading more advanced literature like this book on object-oriented design.
Project structure¶
For HydPy, we prefer a flat folder structure with two subpackage levels. The
individual modules can be of arbitrary length to cover particular topics
completely. For example, module parametertools
defines all base classes for
creating model-specific parameter classes and related collection classes.
Subpackage core provides the essential features of HydPy, used for
implementing hydrological models and workflows. One example is the mentioned
parametertools
module. Modules defined in subpackage core should never
import features provided by modules of other subpackages, excepts those of
subpackage cythons.
Subpackage auxs provides auxiliary features, only necessary for selected
HydPy models and applications. One example is the module anntools
defining
artificial neural network classes usable as complex model parameters, currently
relevant for the dam
model only. Modules defined in subpackage auxs are
allowed to import features from subpackages core and cythons.
Subpackage models contains the implemented hydrological models. Base models
as dam
are additional subpackages, providing, for example, different kinds of
sequence classes in separate submodules. Application models as dam_v001
,
selecting valid combinations of base model features, are defined within single
modules. Please follow the naming patterns of the modules of the already
available models carefully when implementing new ones.
Subpackage exe provides features easing the execution of HydPy.
Module commandtools
(in combination with script hyd
), for example,
allows controlling HydPy from the command line.
Subpackage cythons is related to all Cython features of HydPy:
* It implements functionalities for "cythonizing" the Python models defined in
subpackage `models`_.
* It contains `Cython`_ extension files, which mostly correspond to Python
modules of other subpackages. For example, the extension file |annutils|
provides time-critical implementation details to module |anntools|.
* it contains the additional subpackage `autogen`_, including all
automatically generated extension files and Dynamic Link Library files
(*pyd* files on Windows and *so* files on Linux).
Extension files should not import any features from other subpackages. Python files controlling the automatic generation of extension files can import from the subpackage core.
Note that the names of the modules of subpackages core, auxs, and exe end in almost all cases with “tools” and those of the modules and extension files of subpackage cythons with “utils”, which helps to identify the different module types immediately and to circumvent name conflicts between and within modules.
Subpackage conf contains configuration files (currently XML schema files and coefficients for numerical integration algorithms), which might be generated automatically during HydPy’s build process.
Subpackage data provides example data usable within doctests, currently only the LahnH example project.
Subpackage docs contains different subpackages. sphinx controls the automatic generation of the HTML documentation. rst contains all reStructuredText files written manually. figs contains all manually generated figures in the png format. After the build process, html contains the plotly plots automatically generated during testing. Note that the actual HTML generation takes place in a folder auto, automatically created and filled with information during the process.
Subpackage tests deals with testing. As explained in section Tests & documentation, the contained unit test modules are deprecated. Its subpackage iotesting is the place designated to store data during testing temporarily.
Imports¶
As recommended in PEP 8, clarify the sources of your imports. Always use the following pattern at the top of a new module and list the imports of a section in alphabetical order:
>>> # import...
>>> # ...from standard library
>>> import os
>>> import sys
>>> # ...from site-packages
>>> import numpy
>>> # ...from HydPy
>>> from hydpy.core import sequencetools
>>> from hydpy.cythons import pointerutils
Note that each import command stands in a separate line. Always import complete modules from HydPy without changing their names. — No wildcard imports!
We lift the wildcard ban for writing configuration files. Using the example of parameter control files, it would not be convenient always to write something like:
>>> from hydpy.models import hland
>>> model = hland.Model()
>>> from hydpy.core import parametertools
>>> model.parameters = parametertools.Parameters({"model": model})
>>> model.parameters.control = hland.ControlParameters(model.parameters.control)
>>> model.parameters.control.nmbzones = 2
>>> model.parameters.control.nmbzones
nmbzones(2)
Here a wildcard import (and the “magic” of function parameterstep()
),
allows for a much cleaner syntax:
>>> del model
>>> from hydpy.models.hland import *
>>> parameterstep("1d")
>>> nmbzones(2)
>>> nmbzones
nmbzones(2)
Note that the wildcard import is acceptable here, as there is only one import statement. There is no danger of name conflicts.
Besides the wildcard exeption explained above, there is another one
related to modelimports
.
Defensive programming¶
HydPy is intended to be applicable by researchers and practitioners who are
no Python experts and may have little experience in programming in general.
Hence, it is desirable to anticipate errors due to misleading input as thorough
as possible and report them as soon as possible. So, in contradiction
to PEP 8, it is often preferable to not just expose the names of
simple public attributes. Whenever sensible, use protected attributes
(defined by property
or the more specific property features provided by
module propertytools
) to assure that the internal states of objects
remain consistent. One example is that it is not allowed to assign an unknown
string to the outputfiletype of an instance ofclass SequenceManager
:
>>> from hydpy.core.filetools import SequenceManager
>>> sequencemanager = SequenceManager()
>>> sequencemanager.filetype = "test"
Traceback (most recent call last):
...
ValueError: The given sequence file type `test` is not implemented. Please choose one of the following file types: npy, asc, and nc.
Of course, the extensive usage of protected attributes increases the length of the source code and slows computation time. However, regarding the first point, writing a graphical user interface would require much more source code (and still decrease flexibility). Regarding the second point, one should take into account that the computation times of the general framework functionalities discussed here should be negligible in comparison with the computation times of hydrological simulations in the majority of cases.
Exceptions¶
Unmodified Python error messages are often not sufficiently informative
for HydPy applications due to two reasons. First, they are probably
read by someone who has no experience in understanding Python’s exception
handling system. Second, they do not tell in which hydrological context
a problem occurs. It would be of little help to only know that the value
of a parameter object of a particular type has been misspecified but not
to know in which sub-catchment. Hence, try to add as much helpful
information to error messages as possible. One useful helper function
for doing so is elementphrase()
, trying to determine the name of the
relevant Element
object and add it to the error message:
>>> from hydpy.models.hland import *
>>> parameterstep("1d")
>>> from hydpy import Element
>>> e1 = Element("e1", outlets="n1")
>>> e1.model = model
>>> k(hq=10.0)
Traceback (most recent call last):
...
ValueError: For the alternative calculation of parameter `k` of element `e1`, at least the keywords arguments `khq` and `hq` must be given.
Another recommended approach is exception chaining, for which we
recommend using the function augment_excmessage()
:
>>> e1.keywords = "correct", "w r o n g"
Traceback (most recent call last):
...
ValueError: While trying to add the keyword `w r o n g` to device e1, the following error occurred: The given name string `w r o n g` does not define a valid variable identifier. Valid identifiers do not contain characters like `-` or empty spaces, do not start with numbers, cannot be mistaken with Python built-ins like `for`...)
Naming conventions¶
The naming conventions of PEP 8 apply. Additionally, we encouraged to name classes and their instances as similar as possible whenever reasonable, often simply switching from CamelCase to lowercase, as shown in the following examples:
Class Name |
Instance Name |
Note |
---|---|---|
Sequences |
sequences |
each Model instance handles exactly one Sequence instance: model.sequences |
InputSequences |
inputs |
“inputsequences” would be redundant for attribute access: model.sequences.inputs |
If reasonable, each instance should define its preferred name via name attribute:
>>> from hydpy.models.hland import *
>>> InputSequences(None).name
'inputs'
Classes like Element
or Node
, where names (and not namespaces) are
used to differentiate between instances, should implement instance name
attributes when reasonable:
>>> from hydpy import Node
>>> Node("gauge1").name
'gauge1'
Group instances of the same type in collection objects with the same name,
except an attached letter “s”. For example, we store different Element
objects in an instance of class Elements
and different Node
objects
in an instance of the class Nodes
.
Collection classes¶
The subsection above deals with the naming (of the instances) of collection classes. Additionally, consider the following recommendations when implementing new collection classes.
Each collection object must be iterable:
>>> from hydpy import Nodes
>>> nodes = Nodes("gauge1", "gauge2")
>>> for node in nodes:
... print(repr(node))
Node("gauge1", variable="Q")
Node("gauge2", variable="Q")
For assisting the user when working interactively in the Python shell, collection objects should expose their handled objects as attributes and let function “dir” return the attribute names, being identical with the name attributes of the handled objects:
>>> nodes.gauge1
Node("gauge1", variable="Q")
>>> nodes.gauge2
Node("gauge2", variable="Q")
>>> "gauge1" in dir(nodes)
True
Additionally, provide item access as a more type-safe and eventually more efficient alternative for writing complex scripts:
>>> nodes["gauge1"]
Node("gauge1", variable="Q")
Whenever useful, define convenience functions to simplify the handling of collection objects:
>>> nodes += Node("gauge1")
>>> nodes.gauge1 is Node("gauge1")
True
>>> len(nodes)
2
>>> "gauge1" in nodes
True
>>> nodes.gauge1 in nodes
True
>>> newnodes = nodes.copy()
>>> nodes is newnodes
False
>>> nodes.gauge1 is newnodes.gauge1
True
>>> nodes -= "gauge1"
>>> 'gauge1' in nodes
False
String representations¶
Be aware of the difference between str
and repr()
. Often, str
is
supposed to return strings describing objects in a condensed form for
end-users when executing a program, while repr()
is supposed to return
strings containing all details of an object for developers when debugging
a program. Some argue, due to its limited usage, giving repr()
much
attention is a waste of time in many cases. For HydPy, we think different.
Defining comprehensive repr()
return values simplifies reading the
doctests of the online documentation and working interactively within
the Python shell, thus being of high relevance for end-users, too. On
the other hand, str
is a little less relevant due to mainly being an
alternative for the generation of exception messages. Hence, focus
primarily on repr()
and concentrate on str
when the return value of
repr()
is too complicated for exception messages.
A good return value of repr()
is one that a non-Python-programmer does
not identify as a string. The first ideal case is that copy-pasting
the string representation and evaluating it within the Python shell
returns a reference to the same object.
A Python example:
>>> repr(None)
'None'
>>> eval("None") is None
True
A HydPy example:
>>> from hydpy import Node
>>> Node("gauge1")
Node("gauge1", variable="Q")
>>> eval('Node("gauge1", variable="Q")') is Node("gauge1")
True
In the second ideal case, evaluating the string representation results in an equal object.
A Python example:
>>> x = 1.5
>>> x
1.5
>>> eval("1.5") is x
False
>>> eval("1.5") == x
True
A HydPy example:
>>> from hydpy import Period
>>> Period("1d")
Period("1d")
>>> eval('Period("1d")') is Period("1d")
False
>>> eval('Period("1d")') == Period("1d")
True
For nested objects, the above goals may be hard to accomplish, but sometimes it’s worth it.
A Python example:
>>> [1., "a"]
[1.0, 'a']
>>> eval("[1.0, 'a']") == [1.0, "a"]
True
A HydPy example:
>>> from hydpy import Timegrid
>>> Timegrid("01.11.1996", "1.11.2006", "1d")
Timegrid("01.11.1996 00:00:00",
"01.11.2006 00:00:00",
"1d")
>>> eval('Timegrid("01.11.1996 00:00:00", "01.11.2006 00:00:00", "1d")') == Timegrid("01.11.1996", "1.11.2006", "1d")
True
For deeply nested objects, this strategy becomes infeasible, of course. Then try to find a way to “flatten” the string representation without losing too much information:
>>> from hydpy import Element, Elements
>>> Elements(Element("e_1", outlets="n_1"), Element("e_2", outlets="n_2"))
Elements("e_1", "e_2")
Finally, always consider using functions provided by module objecttools
for simplifying the definition of repr()
and str
return values
to keep the string representations of different HydPy objects, at least
to a certain degree, consistent. For example, use function repr_
to
let the user control the maximum number of decimal places of scalar
floating-point values:
>>> from hydpy import pub, repr_
>>> class Number(float):
... def __repr__(self):
... return repr_(self)
>>> pub.options.reprdigits = 3
>>> Number(1./3.)
0.333
Introspection¶
One nice feature of Python is its “introspection” capability, allowing to analyse (and, when necessary, modify) objects at runtime with little effort.
HydPy makes extensive use of these introspection features whenever it
serves the purpose of relieving non-programmers from writing code lines
that do not deal with hydrological modelling directly. Section Imports
discusses the usage of wildcard imports in parameter control files,
where the real comfort comes from the “magic” implemented in function
parameterstep()
. Invoking this function does not only define the time
interval length for the following parameter values. It also initialises
a new model instance (if such an instance does not already exist) and
directly exposes its control parameter objects in the local namespace.
For the sake of the user’s comfort, each parameter control file purports
to be a simple configuration file that somehow checks its own validity.
On the downside, modifying the operating principle of HydPy’s parameter
control files requires more thought than a more simple direct approach would.
We encourage to implement additional introspection features as long as
they improve the intuitive usability for non-programmers and do not harm
HydPy’s reliability. However, please be particularly cautious when doing
so and document why and how thoroughly. To ensure traceability, one
should usually add such code to modules like modelutils
, importtools
,
and autodoctools
. Module modelutils
deals with all introspection
needed to “cythonize” Python models automatically. Module importtools
contains the function parameterstep()
and related features. Module
autodoctools
serves the purpose to improve the automatic generation of
the online documentation.
Typing¶
Python is a strongly but dynamically typed programming language, allowing
to write very condensed, readable, and flexible (scripting) code. However,
missing type information has also its drawbacks. With the HydPy sources
reaching a certain size, we began to introduce static typing annotations
based on module typing
. In our experience, the additional information
helps a lot, allowing code inspection and refactoring tools to analyse
and modify the code more efficiently. We are going to increase our efforts
in this direction, but do not have a “HydPy Typing Style Guide” at hand,
so far. So please add the typing annotations you find useful. The minimum
requirement for Python core modules is to declare the return type (or, when
necessary, to declare the Union
of possible return types) of
each new function or method:
>>> from typing import List
>>> def test(nmb) -> List[int]:
... return list(range(nmb))
For Cython extension files, adding type information understandable to Python tools is of even greater importance. Hence, accompany each Cython extension file with a stub file, annotating all public (sub)members.
Implementing models¶
Please inspect the source files of the already available hydrological
models in detail to understand how to implement new ones correctly.
HydPy provides many standard features, allowing you to write straightforward
model source code in many cases. However, you are free to implement any
functionalities you find missing (see, for example, the complex “connect”
method defined by the hbranch
model). If those functionalities might
be of importance to other models as well, consider generalising them and
adding them to the suitable subpackage.
The main effort of creating new models is not to write the source code
but to document it thoroughly and to prove it is working correctly.
Each docstring of a calculation method must contain at least a short
description, lists of the required, calculated, and updated variables
(linked via substitutions), the basic equation in LaTeX style,
and doctests covering all anticipated usages of the method, even the
unlikely ones. The docstrings of all Parameter
or Sequence_
subclasses containing “special” source code (for example, modifications
of trim()
) must contain doctests addressing these code sections.
Finally, write integration tests for each application model based on
class IntegrationTest
, explaining all model functionalities in detail
both with text and plotly plots, and preventing future regression by
sufficiently complete tabulated calculation results.