Tests & documentation

The scientific community broadly discusses the capabilities and shortcomings of hydrological models from a theoretical perspective. For example, many sensitivity studies address the negative impacts of low data quality. By contrast, we are unaware of any studies that estimate the adverse effects of bugs and misleading documentation in hydrological computer models. Because evaluation processes (e.g. peer review) pay little attention to these issues, impaired model results may be published, possibly compromising the conclusions drawn. See, for example, Hutton et al.’s commentary, which addresses this topic from a scientific perspective.

This section describes strategies to limit the risk of serious errors and outdated documentation. We try to keep the code and the documentation in sync by connecting them as strongly as possible, using Python’s “docstring” and “doctest” features.

The first “connection” is writing each piece of the documentation close to the related source code. Therefore, we prefer to write all explanations addressing specific HydPy features as docstrings. Docstrings are documentation strings attached to the Python objects they explain. When extending HydPy, each newly implemented public member (including sub-members, e.g. the user-available methods of a class) must have its own docstring. Our Travis CI-based continuous integration workflow recognises any missing docstrings using Pylint and reports them as errors.

The second “connection” is to use and extend Sphinx’s functionality, which collects the source code, docstrings, and standard documentation files to generate the online documentation. Sphinx relies on the reStructuredText format; therefore, use this format for docstrings and regular documentation files. However, instead of using its regular referencing style, use “substitutions” as defined by class Substituter of module autodoctools. For example, write the class name “Substituter” within vertical bars to reference the corresponding class properly. This short syntax makes it easy to use substitutions frequently. A helpful side effect is that, during HTML generation, incorrect substitutions trigger warnings that our Travis CI-based continuous integration workflow interprets as errors (see section Continuous Integration). This mechanism increases the chances that, when documentation adjustments do not accompany future code changes, the Travis CI-based workflow breaks, forcing the responsible programmer to update the documentation.

The third “connection” is to define a sufficient number of doctests. Doctests are documentation sections containing valid Python code followed by the expected results. For developers, doctest is not always as convenient as other unit testing frameworks (e.g. unittest), but it offers the great advantage of defining tests that are understandable for non-programmers. At best, each (sub)member should specify its own doctests, in conjunction with some “normal” explanations about its purpose and usage. Non-programmers should be enabled to learn using HydPy by repeating the doctests. Beyond their intuitiveness, doctests (like substitutions) keep source code and documentation in sync. For example, as long as the following three-line doctest remains in the documentation, one can be sure that the current core package contains a module named “objecttools”:

>>> from hydpy.core import objecttools
>>> objecttools.classname(objecttools)
'module'

The Python script run_doctests.py collects the doctests of all modules and executes them. Before executing the doctests of a module or documentation file, it calls function prepare_testing(), which, for example, sets all options to the values the doctests expect. To execute doctests with other tools, like the doctest runner of PyCharm, under the same conditions, set the environment variable HYDPY_PREPARE_TESTING to python or cython (see function prepare_testing_via_environment()). In PyCharm, you can do so for all doctest run configurations by adding HYDPY_PREPARE_TESTING=python to the environment variables of the “Doctests” configuration template.

We measure the “code coverage” (the number of executed code lines divided by the total number of code lines), determined by the coverage library. The current code coverage of HydPy is 100 %. Travis CI reports future changes that introduce uncovered lines as failures, ensuring future changes do not undo this favourable situation.