servertools

This module facilitates using HydPy as an HTTP server application.

HydPy is designed to be used interactively or by executing individual Python scripts. Consider the typical steps of calibrating model parameters. Usually, one first prepares an instance of class HydPy, then changes some parameter values and performs a simulation, and finally inspects whether the new simulation results are better than the ones of the original parameterisation or not. One can perform these steps manually (in a Python console) or apply optimisation tools like those provided by scipy (usually in a Python script).

Performing or implementing such procedures is relatively simple, as long as all tools are written in Python or come with a Python interface, which is not the case for some relevant optimisation tools. One example is OpenDA, being written in Java, which was the original reason for adding module servertools to the HydPy framework.

Module servertools solves such integration problems by running HydPy within an HTTP server. After starting such a server, one can use any HTTP client (e.g. curl) to perform the above steps.

The server’s API is relatively simple, allowing performing a “normal” calibration using only a few server methods. However, it is also more restrictive than controlling HydPy within a Python process. Within a Python process, you are free to do anything. Using the HydPy server, you are much more restricted to what was anticipated by the framework developers.

Commonly but not mandatory, one configures the initial state of a HydPy server with an XML file. As an example, we prepare the LahnH project by calling function prepare_full_example_1(), which contains the XML configuration file multiple_runs_alpha.xml:

>>> from hydpy.examples import prepare_full_example_1
>>> prepare_full_example_1()

To start the server in a new process, open a command-line tool and insert the following command (see module hyd for general information on how to use HydPy via the command line):

>>> command = "hyd.py start_server 8080 LahnH multiple_runs_alpha.xml"
>>> from hydpy import run_subprocess, TestIO
>>> with TestIO():
...     process = run_subprocess(command, blocking=False, verbose=False)
...     result = run_subprocess("hyd.py await_server 8080 10", verbose=False)

The HydPy server should now be running on port 8080. You can use any HTTP client to check it is working. For example, you can type the following URLs in your web browser to get information on the types and initial values of the exchange items defined in multiple_runs_alpha.xml (in a format required by the HydPy-OpenDA-Black-Box-Model-Wrapper):

>>> from urllib import request
>>> url = "http://127.0.0.1:8080/query_itemtypes"
>>> print(str(request.urlopen(url).read(), encoding="utf-8"))
alpha = Double0D
dill_nodes_sim_series = TimeSeries0D

>>> url = "http://127.0.0.1:8080/query_initialitemvalues"
>>> print(str(request.urlopen(url).read(), encoding="utf-8"))
alpha = 2.0
dill_nodes_sim_series = [nan, nan, nan, nan, nan]

It is generally possible to control the HydPy server via invoking each method with a separate HTTP request. However, alternatively, one can use methods GET_execute() and POST_execute() to execute many methods with only one HTTP request. We now define three such metafunctions. The first one changes the value of the parameter Alpha The second one runs a simulation. The third one prints the newly calculated discharge at the outlet of the headwater catchment Dill. All of this is very similar to what the HydPy-OpenDA-Black-Box-Model-Wrapper does.

Function set_itemvalues wraps the POST methods POST_register_simulationdates(), POST_register_parameteritemvalues(), and POST_register_conditionitemvalues(). The HydPy server will execute these methods in the given order. The arguments firstdate_sim, lastdate_sim, and alpha allow changing the start and end date of the simulation period and the value of parameter Alpha later:

>>> def set_itemvalues(id_, firstdate, lastdate, alpha):
...     content = (f"firstdate_sim = {firstdate}\n"
...                f"lastdate_sim = {lastdate}\n"
...                f"alpha = {alpha}").encode("utf-8")
...     methods = ",".join(("POST_register_simulationdates",
...                         "POST_register_parameteritemvalues",
...                         "POST_register_conditionitemvalues"))
...     url = f"http://127.0.0.1:8080/execute?id={id_}&methods={methods}"
...     request.urlopen(url, data=content)

Function simulate wraps only GET methods and triggers the next simulation run. As for all GET and POST methods, one should pass the query parameter id, used by the HydPy server for internal bookmarking:

>>> def simulate(id_):
...     methods = ",".join(("GET_activate_simulationdates",
...                         "GET_activate_parameteritemvalues",
...                         "GET_load_internalconditions",
...                         "GET_activate_conditionitemvalues",
...                         "GET_simulate",
...                         "GET_save_internalconditions",
...                         "GET_update_conditionitemvalues",
...                         "GET_update_getitemvalues"))
...     url = f"http://127.0.0.1:8080/execute?id={id_}&methods={methods}"
...     request.urlopen(url)

Function print_itemvalues also wraps only GET methods and prints the current value of parameter Alpha as well as the lastly simulated discharge values corresponding to the given id value:

>>> from hydpy import print_values
>>> def print_itemvalues(id_):
...     methods = ",".join(("GET_query_simulationdates",
...                         "GET_query_parameteritemvalues",
...                         "GET_query_conditionitemvalues",
...                         "GET_query_getitemvalues"))
...     url = f"http://127.0.0.1:8080/execute?id={id_}&methods={methods}"
...     data = str(request.urlopen(url).read(), encoding="utf-8")
...     for line in data.split("\n"):
...         if line.startswith("alpha"):
...             alpha = line.split("=")[1].strip()
...         if line.startswith("dill"):
...             discharge = eval(line.split("=")[1])
...     print(f"{alpha}: ", end="")
...     print_values(discharge)

For the sake of brevity, we also define do_everything for calling the other functions at once:

>>> def do_everything(id_, firstdate, lastdate, alpha):
...     set_itemvalues(id_, firstdate, lastdate, alpha)
...     simulate(id_)
...     print_itemvalues(id_)

In the simplest example, we perform a simulation throughout five days for an Alpha value of 2:

>>> do_everything("1a", "1996-01-01", "1996-01-06", 2.0)
2.0: 35.537828, 7.741064, 5.018981, 4.501784, 4.238874

The following example shows interlocked simulation runs. The first call only triggers a simulation run for the first initialised day:

>>> do_everything("1b", "1996-01-01", "1996-01-02", 2.0)
2.0: 35.537828

The second call repeats the first one with a different id value:

>>> do_everything("2", "1996-01-01", "1996-01-02", 2.0)
2.0: 35.537828

The third call covers the first three initialisation days:

>>> do_everything("3", "1996-01-01", "1996-01-04", 2.0)
2.0: 35.537828, 7.741064, 5.018981

The fourth call continues the simulation of the first call, covering the last four initialised days:

>>> do_everything("1b", "1996-01-02", "1996-01-06", 2.0)
2.0: 7.741064, 5.018981, 4.501784, 4.238874

The results of the very first call of function do_everything (with`id=1`) are identical with the pulled-together discharge values of the calls with id=1b, made possible by the internal bookmarking feature of the HydPy server. Here we use numbers, but any other strings are valid id values.

This example extends the last one by applying different parameter values:

>>> do_everything("4", "1996-01-01", "1996-01-04", 2.0)
2.0: 35.537828, 7.741064, 5.018981
>>> do_everything("5", "1996-01-01", "1996-01-04", 1.0)
1.0: 11.78038, 8.901179, 7.131072
>>> do_everything("4", "1996-01-04", "1996-01-06", 2.0)
2.0: 4.501784, 4.238874
>>> do_everything("5", "1996-01-04", "1996-01-06", 1.0)
1.0: 6.017787, 5.313211
>>> do_everything("5", "1996-01-01", "1996-01-06", 1.0)
1.0: 11.78038, 8.901179, 7.131072, 6.017787, 5.313211

The order in which function do_everything calls its subfunctions seems quite natural, but some tools might require do deviate from it. For example, OpenDA offers ensemble-based algorithms triggering the simulation of all memberse before starting to query any simulation results. The final example shows that the underlying atomic methods support such an execution sequence:

>>> set_itemvalues("6", "1996-01-01", "1996-01-03", 2.0)
>>> simulate("6")
>>> set_itemvalues("7", "1996-01-01", "1996-01-03", 1.0)
>>> simulate("7")
>>> print_itemvalues("6")
2.0: 35.537828, 7.741064
>>> print_itemvalues("7")
1.0: 11.78038, 8.901179

When working in parallel mode, OpenDA might not always call the functions set_itemvalues and simulate for the same id directly one after another, which also causes no problem:

>>> set_itemvalues("6", "1996-01-03", "1996-01-06", 2.0)
>>> set_itemvalues("7", "1996-01-03", "1996-01-06", 1.0)
>>> simulate("6")
>>> simulate("7")
>>> print_itemvalues("6")
2.0: 5.018981, 4.501784, 4.238874
>>> print_itemvalues("7")
1.0: 7.131072, 6.017787, 5.313211

Finally, we close the server and kill its process (just closing your command-line tool works likewise):

>>> _ = request.urlopen("http://127.0.0.1:8080/close_server")
>>> process.kill()
>>> _ = process.communicate()

The above description focussed on coupling HydPy to OpenDA. However, the applied atomic submethods of class HydPyServer also allow coupling HydPy with other software products. See the documentation on class HydPyServer for further information.

Module servertools implements the following members:

• ID Type for strings that identify “artificial” HydPy instances (from a client’s point of view).

• ServerState Singleton class handling states like the current HydPy instance exchange items.

• HydPyServer The API of the HydPy server.

• start_server() Start the HydPy server using the given socket.

• await_server() Block the current process until either the HydPy server is responding on the given port or the given number of seconds elapsed.

---

class hydpy.exe.servertools.ServerState(projectname: str, xmlfile: str, load_conditions: bool = True, load_series: bool = True)

Bases: object

Singleton class handling states like the current HydPy instance exchange items.

The instance of class ServerState is available as the member state of class HydPyServer after calling the function start_server(). You could create other instances (like we do in the following examples), but you most likely shouldn’t. The primary purpose of this instance is to store information between successive initialisations of class HydPyServer.

We use the LahnH project and its (complicated) XML configuration file multiple_runs.xml as an example (module xmltools provides information on interpreting this file):

>>> from hydpy.examples import prepare_full_example_1
>>> prepare_full_example_1()
>>> from hydpy import print_values, TestIO
>>> from hydpy.exe.servertools import ServerState
>>> with TestIO():  
...     state = ServerState("LahnH", "multiple_runs.xml")
Start HydPy project `LahnH` (...).
Read configuration file `multiple_runs.xml` (...).
Interpret the defined options (...).
Interpret the defined period (...).
Read all network files (...).
Activate the selected network (...).
Read the required control files (...).
Read the required condition files (...).
Read the required time series files (...).

After initialisation, all defined exchange items are available:

>>> for item in state.parameteritems:
...     print(item)
SetItem("alpha", "hland_v1", "control.alpha", None, "global")
SetItem("beta", "hland_v1", "control.beta", None, "global")
SetItem("lag", "musk_classic", "control.nmbsegments", "lag", "global")
SetItem("damp", "musk_classic", "control.coefficients", "damp", "global")
AddItem("sfcf_1", "hland_v1", "control.sfcf", "control.rfcf", "global")
AddItem("sfcf_2", "hland_v1", "control.sfcf", "control.rfcf", "global")
AddItem("sfcf_3", "hland_v1", "control.sfcf", "control.rfcf", "subunit")
MultiplyItem("k4", "hland_v1", "control.k4", "control.k", "global")
>>> for item in state.conditionitems:
...     print(item)
SetItem("ic_lahn_2", "hland_v1", "states.ic", None, "device")
SetItem("ic_lahn_1", "hland_v1", "states.ic", None, "subunit")
SetItem("sm_lahn_2", "hland_v1", "states.sm", None, "device")
SetItem("sm_lahn_1", "hland_v1", "states.sm", None, "subunit")
SetItem("quh", "hland_v1", "logs.quh", None, "device")
>>> for item in state.getitems:
...     print(item)
GetItem("?", "hland_v1", "factors.tmean")
GetItem("current_discharge", "hland_v1", "fluxes.qt")
GetItem("entire_discharge_series", "hland_v1", "fluxes.qt.series")
GetItem("?", "hland_v1", "states.sm")
GetItem("?", "hland_v1", "states.sm.series")
GetItem("?", "nodes", "nodes.sim.series")

The initialisation also memorises the initial conditions of all elements:

>>> for element in state.init_conditions:
...     print(element)
land_dill
land_lahn_1
land_lahn_2
land_lahn_3
stream_dill_lahn_2
stream_lahn_1_lahn_2
stream_lahn_2_lahn_3

The initialisation also prepares all selected series arrays and reads the required input data:

>>> print_values(state.hp.elements.land_dill.model.sequences.inputs.t.series)
-0.298846, -0.811539, -2.493848, -5.968849, -6.999618
>>> state.hp.nodes.dill.sequences.sim.series
InfoArray([nan, nan, nan, nan, nan])

interface: XMLInterface

hp: HydPy

parameteritems: List[ChangeItem]

inputitems: List[SetItem]

conditionitems: List[SetItem]

outputitems: List[SetItem]

getitems: List[GetItem]

initialparameteritemvalues: Dict[Name, Any]

initialinputitemvalues: Dict[Name, Any]

initialconditionitemvalues: Dict[Name, Any]

initialgetitemvalues: Dict[Name, Any]

conditions: Dict[ID, Dict[int, Dict[str, Dict[str, Dict[str, float | ndarray[Any, dtype[float64]]]]]]]

parameteritemvalues: Dict[ID, Dict[Name, Any]]

inputitemvalues: Dict[ID, Dict[Name, Any]]

conditionitemvalues: Dict[ID, Dict[Name, Any]]

outputitemvalues: Dict[ID, Dict[Name, Any]]

getitemvalues: Dict[ID, Dict[Name, str]]

init_conditions: Dict[str, Dict[str, Dict[str, float | ndarray[Any, dtype[float64]]]]]

timegrids: Dict[ID, Timegrid]

serieswriterdirs: Dict[ID, str]

seriesreaderdirs: Dict[ID, str]

inputconditiondirs: Dict[ID, str]

outputconditiondirs: Dict[ID, str]

outputcontroldirs: Dict[ID, str]

idx1: int

idx2: int

class hydpy.exe.servertools.HydPyServer(request, client_address, server)

Bases: BaseHTTPRequestHandler

The API of the HydPy server.

Technically and strictly speaking, HydPyServer is, only the HTTP request handler for the real HTTP server class (from the standard library).

After initialising the HydPy server, each communication via a GET or POST request is handled by a new instance of HydPyServer. This handling occurs in a unified way using either method do_GET() or [HydPyServer.do_POST|, which select and apply the actual GET or POST method. All methods provided by class HydPyServer starting with “GET” or “POST” are accessible via HTTP.

In the main documentation on module servertools, we use the multiple_runs_alpha.xml file of the LahnH project as an example. However, now we select the more complex XML configuration file multiple_runs.xml, covering a higher number of cases:

>>> from hydpy.examples import prepare_full_example_1
>>> prepare_full_example_1()
>>> from hydpy import run_subprocess, TestIO
>>> with TestIO():
...     process = run_subprocess(
...         "hyd.py start_server 8080 LahnH multiple_runs.xml debugging=enable",
...         blocking=False, verbose=False)
...     result = run_subprocess("hyd.py await_server 8080 10", verbose=False)

We define a test function that simplifies sending the following requests and offers two optional arguments. When passing a value to id_, test adds this value as the query parameter id to the URL. When passing a string to data, test sends a POST request containing the given data; otherwise, a GET request without additional data:

>>> from urllib import request
>>> def test(name, id_=None, data=None, return_result=False):
...     url = f"http://127.0.0.1:8080/{name}"
...     if id_:
...         url = f"{url}?id={id_}"
...     if data:
...         data = bytes(data, encoding="utf-8")
...     response = request.urlopen(url, data=data)
...     result = str(response.read(), encoding="utf-8")
...     print(result)
...     return result if return_result else None

Asking for its status tells us that the server is ready (which may take a while, depending on the project’s size):

>>> test("status")
status = ready

You can query the current version number of the HydPy installation used to start the server:

>>> result = test("version", return_result=True)  
version = ...
>>> hydpy.__version__ in result
True

HydPyServer returns the error code 400 if it realises the URL to be wrong:

>>> test("missing")
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 400: RuntimeError: No method `GET_missing` available.

The error code is 500 in all other cases of error:

>>> test("register_parameteritemvalues", id_="0", data="alpha = []")
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: RuntimeError: While trying to execute method `POST_register_parameteritemvalues`, the following error occurred: A value for parameter item `beta` is missing.

Some methods require identity information, passed as query parameter id, used for internal bookmarking:

>>> test("query_parameteritemvalues")
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: RuntimeError: While trying to execute method `GET_query_parameteritemvalues`, the following error occurred: For the GET method `query_parameteritemvalues` no query parameter `id` is given.

POST methods always expect an arbitrary number of lines, each one assigning some values to some variable (in most cases, numbers to exchange items):

>>> test("parameteritemvalues",
...      id_="a",
...      data=("x = y\n"
...            "   \n"
...            "x == y\n"
...            "x = y"))
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 400: RuntimeError: The POST method `parameteritemvalues` received a wrongly formated data body.  The following line has been extracted but cannot be further processed: `x == y`.

Before explaining the more offical methods, we introduce the method POST_evaluate(), which evaluates arbitrary valid Python code within the server process. Its most likely use-case is to access the (sub)attributes of the single instance of class ServerState, available as a member of class HydPyServer. This method can help when being puzzled about the state of the HydPy server. Use it, for example, to find out which Node objects are available and to see which one is the outlet node of the Element object land_dill:

>>> test("evaluate",
...      data=("nodes = HydPyServer.state.hp.nodes\n"
...            "elements = HydPyServer.state.hp.elements.land_dill"))
nodes = Nodes("dill", "lahn_1", "lahn_2", "lahn_3")
elements = Element("land_dill", outlets="dill", keywords="catchment")

Method GET_query_itemtypes(), already described in the main documentation of module servertools, returns all available exchange item types at once. However, it is also possible to query those that are related to setting parameter values (GET_query_parameteritemtypes()), setting condition values (GET_query_conditionitemtypes()), setting input time series (GET_query_inputitemtypes()), getting values or series of factors or fluxes in the “setitem style” (GET_query_outputitemtypes()) and getting different kinds of values or series in the “getitem style” (GET_query_getitemtypes()) separately:

>>> test("query_parameteritemtypes")
alpha = Double0D
beta = Double0D
lag = Double0D
damp = Double0D
sfcf_1 = Double0D
sfcf_2 = Double0D
sfcf_3 = Double1D
k4 = Double0D
>>> test("query_conditionitemtypes")
ic_lahn_2 = Double1D
ic_lahn_1 = Double1D
sm_lahn_2 = Double1D
sm_lahn_1 = Double1D
quh = Double1D
>>> test("query_inputitemtypes")
t_headwaters = TimeSeries1D
>>> test("query_outputitemtypes")
swe_headwaters = TimeSeries1D
>>> test("query_getitemtypes")
land_dill_factors_tmean = Double0D
land_dill_fluxes_qt = Double0D
land_dill_fluxes_qt_series = TimeSeries0D
land_dill_states_sm = Double1D
land_lahn_1_states_sm = Double1D
land_lahn_2_states_sm = Double1D
land_lahn_3_states_sm = Double1D
land_lahn_3_states_sm_series = TimeSeries1D
dill_nodes_sim_series = TimeSeries0D

The same holds for the initial values of the exchange items. Method GET_query_initialitemvalues() returns them all at once, while the methods GET_query_initialparameteritemvalues(), GET_query_initialconditionitemvalues(), GET_query_initialinputitemvalues(), GET_query_initialoutputitemvalues(), and (GET_query_initialgetitemvalues() return the relevant subgroup only. Note that for the exchange items related to state sequence SM (sm_lahn_1 and sm_lahn_2), the initial values stem from the XML file. For the items related to state sequence Ic and input sequence T, the XML file does not provide such information. Thus, the initial values of ic_lahn_1, ic_lahn_2, and t_headwaters stem from the corresponding sequences themselves (and thus, indirectly, from the respective condition and time series files):

>>> test("query_initialparameteritemvalues")
alpha = 2.0
beta = 1.0
lag = 5.0
damp = 0.5
sfcf_1 = 0.3
sfcf_2 = 0.2
sfcf_3 = [0.1, 0.2, 0.1, 0.2, 0.1, 0.2, 0.1, 0.2, 0.1, 0.2, 0.1, 0.2, 0.2, 0.2]
k4 = 10.0
>>> test("query_initialconditionitemvalues")
ic_lahn_2 = [1.184948]
ic_lahn_1 = [0.96404, 1.36332, 0.96458, 1.46458, 0.96512, 1.46512, 0.96565, 1.46569, 0.96617, 1.46617, 0.96668, 1.46668, 1.46719]
sm_lahn_2 = [123.0]
sm_lahn_1 = [110.0, 120.0, 130.0, 140.0, 150.0, 160.0, 170.0, 180.0, 190.0, 200.0, 210.0, 220.0, 230.0]
quh = [10.0]
>>> test("query_initialinputitemvalues")
t_headwaters = [[-0.298846, -0.811539, -2.493848, -5.968849, -6.999618], [-0.705395, -1.505553, -4.221268, -7.446349, -8.119366]]
>>> test("query_initialoutputitemvalues")
swe_headwaters = [[nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan]]
>>> test("query_initialgetitemvalues")  
land_dill_factors_tmean = nan
land_dill_fluxes_qt = nan
land_dill_fluxes_qt_series = [nan, nan, nan, nan, nan]
land_dill_states_sm = [185.13164...]
land_lahn_1_states_sm = [99.27505...]
land_lahn_2_states_sm = [138.31396...]
land_lahn_3_states_sm = [101.31248...]
land_lahn_3_states_sm_series = [[nan, ...], [nan, ...], ..., [nan, ...]]
dill_nodes_sim_series = [nan, nan, nan, nan, nan]

Some external tools require ways to identify specific sub-values of different exchange items. For example, they need to map those sub-values to location data available in a separate database. Method GET_query_itemsubnames() provides artificial sub names suitable for such a mapping. See property subnames of class ChangeItem and method yield_name2subnames() of class GetItem for the specification of the sub names. Here, note the special handling for change items addressing the global level, for which we cannot define a meaningful sub name. Method GET_query_itemsubnames() returns the string *global* in such cases:

>>> test("query_itemsubnames") 
alpha = *global*
beta = *global*
lag = *global*
damp = *global*
sfcf_1 = *global*
sfcf_2 = *global*
sfcf_3 = [land_lahn_3_0, ..., land_lahn_3_13]
k4 = *global*
t_headwaters = [land_dill, land_lahn_1]
ic_lahn_2 = [land_lahn_2]
ic_lahn_1 = [land_lahn_1_0, ..., land_lahn_1_12]
sm_lahn_2 = [land_lahn_2]
sm_lahn_1 = [land_lahn_1_0, ..., land_lahn_1_12]
quh = [land_lahn_2]
swe_headwaters = [land_dill, land_lahn_1]
land_dill_factors_tmean = land_dill
land_dill_fluxes_qt = land_dill
land_dill_fluxes_qt_series = land_dill
land_dill_states_sm = ('land_dill_0', ..., 'land_dill_11')
land_lahn_1_states_sm = ('land_lahn_1_0', ..., 'land_lahn_1_12')
land_lahn_2_states_sm = ('land_lahn_2_0', ..., 'land_lahn_2_9')
land_lahn_3_states_sm = ('land_lahn_3_0', ..., 'land_lahn_3_13')
land_lahn_3_states_sm_series = ('land_lahn_3_0', ..., 'land_lahn_3_13')
dill_nodes_sim_series = dill

The init time grid is immutable once the server is ready. Method GET_query_initialisationtimegrid() returns the fixed first date, last date, and stepsize of the whole initialised period:

>>> test("query_initialisationtimegrid")
firstdate_init = 1996-01-01T00:00:00+01:00
lastdate_init = 1996-01-06T00:00:00+01:00
stepsize = 1d

The dates of the sim time grid, on the other hand, are mutable and can vary for different id query parameters. This flexibility makes things a little more complicated, as the Timegrids object of the global pub module handles only one sim object at once. Hence, we differentiate between registered simulation dates of the respective id values and the current simulation dates of the Timegrids object.

Method GET_query_simulationdates() asks for registered simulation dates and thus fails at first:

>>> test("query_simulationdates", id_="0")
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: RuntimeError: While trying to execute method `GET_query_simulationdates`, the following error occurred: Nothing registered under the id `0`.  There is nothing registered, so far.

After logging new simulation dates via the POST method POST_register_simulationdates(), method GET_query_simulationdates() returns them correctly:

>>> test("register_simulationdates", id_="0",
...      data=("firstdate_sim = 1996-01-01\n"
...            "lastdate_sim = 1996-01-02"))

>>> test("query_simulationdates", id_="0")
firstdate_sim = 1996-01-01T00:00:00+01:00
lastdate_sim = 1996-01-02T00:00:00+01:00

Our initial call to the POST method POST_register_simulationdates() did not affect the currently active simulation dates. We need to do this manually by calling method GET_activate_simulationdates():

>>> test("evaluate", data="lastdate = hydpy.pub.timegrids.sim.lastdate")
lastdate = Date("1996-01-06T00:00:00")
>>> test("activate_simulationdates", id_="0")

>>> test("evaluate", data="lastdate = hydpy.pub.timegrids.sim.lastdate")
lastdate = Date("1996-01-02 00:00:00")

Generally, passing a missing id while others are available results in error messages like the following:

>>> test("activate_simulationdates", id_="1")
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: RuntimeError: While trying to execute method `GET_activate_simulationdates`, the following error occurred: Nothing registered under the id `1`.  The available ids are: 0.

The logic of the parameter-related GET and POST methods is very similar to the one of the simulation date-related methods discussed above. Method POST_register_parameteritemvalues() registers new values of the exchange items, and method GET_activate_parameteritemvalues() activates them (assigns them to the relevant parameters):

>>> test("register_parameteritemvalues", id_="0",
...      data=("alpha = 3.0\n"
...            "beta = 2.0\n"
...            "lag = 1.0\n"
...            "damp = 0.5\n"
...            "sfcf_1 = 0.3\n"
...            "sfcf_2 = 0.2\n"
...            "sfcf_3 = 0.1\n"
...            "k4 = 10.0\n"))

>>> control = "HydPyServer.state.hp.elements.land_dill.model.parameters.control"
>>> test("evaluate",
...      data=(f"alpha = {control}.alpha\n"
...            f"sfcf = {control}.sfcf"))
alpha = alpha(1.0)
sfcf = sfcf(1.1)

>>> test("activate_parameteritemvalues", id_="0")

>>> test("evaluate",
...      data=(f"alpha = {control}.alpha\n"
...            f"sfcf = {control}.sfcf"))
alpha = alpha(3.0)
sfcf = sfcf(1.34283)

The list of exchange items must be complete:

>>> test("register_parameteritemvalues", id_="0",
...      data=("alpha = 3.0\n"
...            "beta = 2.0"))
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: RuntimeError: While trying to execute method `POST_register_parameteritemvalues`, the following error occurred: A value for parameter item `lag` is missing.

Note that the related query method (GET_query_parameteritemvalues()) returns the logged values of the ChangeItem objects instead of the (eventually modified) values of the Parameter objects:

>>> test("query_parameteritemvalues", id_="0")
alpha = 3.0
beta = 2.0
lag = 1.0
damp = 0.5
sfcf_1 = 0.3
sfcf_2 = 0.2
sfcf_3 = 0.1
k4 = 10.0

The condition-related methods POST_register_conditionitemvalues(), GET_activate_conditionitemvalues(), and GET_query_conditionitemvalues() work like the parameter-related methods described above:

>>> test("register_conditionitemvalues", id_="0",
...      data=("sm_lahn_2 = 246.0\n"
...            "sm_lahn_1 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]\n"
...            "ic_lahn_2 = 642.0\n"
...            "ic_lahn_1 = [13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1]\n"
...            "quh = 1.0\n"))

>>> test("query_conditionitemvalues", id_="0")
ic_lahn_2 = 642.0
ic_lahn_1 = [13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1]
sm_lahn_2 = 246.0
sm_lahn_1 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
quh = 1.0

Note the trimming of the too-high value for the state sequence SM to its highest possible value defined by control parameter FC):

>>> for element in ("land_lahn_1", "land_lahn_2"):
...     sequences = f"HydPyServer.state.hp.elements.{element}.model.sequences"
...     test("evaluate",
...          data=(f"sm = {sequences}.states.sm \n"
...                f"quh = {sequences}.logs.quh"))  
sm = sm(99.27505, ..., 142.84148)
quh = quh(0.0)
sm = sm(138.31396, ..., 164.63255)
quh = quh(0.7, 0.0)
>>> test("activate_conditionitemvalues", id_="0")

>>> for element in ("land_lahn_1", "land_lahn_2"):
...     sequences = f"HydPyServer.state.hp.elements.{element}.model.sequences"
...     test("evaluate",
...          data=(f"sm = {sequences}.states.sm \n"
...                f"quh = {sequences}.logs.quh"))  
sm = sm(1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0)
quh = quh(0.0)
sm = sm(197.0, 197.0, 197.0, 197.0, 197.0, 197.0, 197.0, 197.0, 197.0, 197.0)
quh = quh(1.0, 0.0)

The methods POST_register_inputitemvalues(), GET_activate_inputitemvalues(), and GET_query_inputitemvalues() always focus on the currently relevant simulation time grid:

>>> test("update_inputitemvalues", id_="0")

>>> test("query_inputitemvalues", id_="0")
t_headwaters = [[-0.298846], [-0.705395]]
>>> t = "HydPyServer.state.hp.elements.land_lahn_1.model.sequences.inputs.t"
>>> test("evaluate", data=(f"t_series = {t}.series\n"
...                        f"t_simseries = {t}.simseries\n"))
t_series = InfoArray([-0.70539496, -1.50555283, -4.22126769, -7.44634946, -8.11936591])
t_simseries = InfoArray([-0.70539496])

>>> test("register_inputitemvalues", id_="0",
...      data="t_headwaters = [[1.0], [2.0]]\n")

>>> test("activate_inputitemvalues", id_="0")

>>> test("evaluate", data=(f"t_series = {t}.series\n"
...                        f"t_simseries = {t}.simseries\n"))
t_series = InfoArray([ 2.        , -1.50555283, -4.22126769, -7.44634946, -8.11936591])
t_simseries = InfoArray([2.])

The “official” way to gain information on modified parameters or conditions is to use the method GET_query_getitemvalues():

>>> test("query_getitemvalues", id_="0")
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: RuntimeError: While trying to execute method `GET_query_getitemvalues`, the following error occurred: Nothing registered under the id `0`.  There is nothing registered, so far.

As the error message explains, we first need to fill the registry for the given id parameter. Unlike the examples above, we do not do this by sending external data via a POST request but by retrieving the server’s currently active data. We accomplish this task by calling the GET method GET_update_getitemvalues():

>>> test("update_getitemvalues", id_="0")

>>> test("query_getitemvalues", id_="0")  
land_dill_factors_tmean = nan
land_dill_fluxes_qt = nan
land_dill_fluxes_qt_series = [nan]
land_dill_states_sm = [185.13164, ...]
land_lahn_1_states_sm = [1.0, 2.0, ..., 12.0, 13.0]
land_lahn_2_states_sm = [197.0, ..., 197.0]
land_lahn_3_states_sm = [101.31248, ...]
land_lahn_3_states_sm_series = [[nan, ..., nan]]
dill_nodes_sim_series = [nan]

Besides the “official” way for retrieving information (which we sometimes call the “getitem style), some sequences types (namely those derived from FactorSequence and FluxSequence) also allow retrieving information in the so-called “setitem style” via the methods GET_update_outputitemvalues() and GET_query_outputitemvalues():

>>> test("query_outputitemvalues", id_="0")
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: RuntimeError: While trying to execute method `GET_query_outputitemvalues`, the following error occurred: Nothing registered under the id `0`.  There is nothing registered, so far.
>>> test("update_outputitemvalues", id_="0")

>>> test("query_outputitemvalues", id_="0")
swe_headwaters = [[nan], [nan]]

We now modify the parameter, condition, and input time series values again, but this time in one step through calling POST_register_changeitemvalues() and GET_activate_changeitemvalues():

>>> test("register_changeitemvalues", id_="0",
...      data=("alpha = 1.0\n"
...            "beta = 1.0\n"
...            "lag = 0.0\n"
...            "damp = 0.0\n"
...            "sfcf_1 = 0.0\n"
...            "sfcf_2 = 0.0\n"
...            "sfcf_3 = 0.0\n"
...            "k4 = 5.0\n"
...            "ic_lahn_1 = 1.0\n"
...            "ic_lahn_2 = 2.0\n"
...            "sm_lahn_1 = 50.0\n"
...            "sm_lahn_2 = 100.0\n"
...            "quh = 0.0\n"
...            "t_headwaters = [[-0.29884643], [-0.70539496]]\n"))

>>> test("activate_changeitemvalues", id_="0")

>>> test("query_changeitemvalues", id_="0")  
alpha = 1.0
beta = 1.0
lag = 0.0
damp = 0.0
sfcf_1 = 0.0
sfcf_2 = 0.0
sfcf_3 = 0.0
k4 = 5.0
t_headwaters = [[-0.29884...], [-0.70539...]]
ic_lahn_2 = 2.0
ic_lahn_1 = 1.0
sm_lahn_2 = 100.0
sm_lahn_1 = 50.0
quh = 0.0

Next, we trigger a simulation run by calling the GET method GET_simulate():

>>> test("simulate", id_="0")

Calling methods GET_update_getitemvalues() and GET_query_getitemvalues() as well as methods GET_update_outputitemvalues() and GET_query_outputitemvalues() reveals how the simulation results:

>>> test("update_getitemvalues", id_="0")

>>> test("query_getitemvalues", id_="0")  
land_dill_factors_tmean = -0.572053
land_dill_fluxes_qt = 5.515321
...
land_lahn_2_states_sm = [99.848023, ..., 99.848023]
...
dill_nodes_sim_series = [5.515321]
>>> test("update_outputitemvalues", id_="0")

>>> test("query_outputitemvalues", id_="0")
swe_headwaters = [[0.0], [0.0]]

So far, we have explained how the HydPy server memorises different exchange item values for different values of query parameter id. Complicating matters, memorising condition values must also consider the relevant time point. You load conditions for the simulation period’s current start date with method GET_load_internalconditions() and save them for the current end date with method GET_save_internalconditions(). For example, we first save the states calculated for the end time of the last simulation run (January 2):

>>> test("query_simulationdates", id_="0")
firstdate_sim = 1996-01-01T00:00:00+01:00
lastdate_sim = 1996-01-02T00:00:00+01:00
>>> test("evaluate",
...      data=f"sm_lahn2 = {sequences}.states.sm")  
sm_lahn2 = sm(99.848023, ..., 99.848023)
>>> test("save_internalconditions", id_="0")

Calling method GET_load_internalconditions() without changing the simulation dates reloads the initial conditions for January 1, originally read from disk:

>>> test("load_internalconditions", id_="0")

>>> test("evaluate",
...      data=f"sm_lahn2 = {sequences}.states.sm")  
sm_lahn2 = sm(138.31396, ..., 164.63255)

If we set the first date of the simulation period to January 2, method GET_load_internalconditions() loads the conditions we saved for January 2 previously:

>>> test("register_simulationdates", id_="0",
...      data=("firstdate_sim = 1996-01-02\n"
...            "lastdate_sim = 1996-01-03"))

>>> test("activate_simulationdates", id_="0")

>>> test("load_internalconditions", id_="0")

>>> test("evaluate",
...      data=f"sm_lahn2 = {sequences}.states.sm")  
sm_lahn2 = sm(99.848023, ..., 99.848023)

Loading condition values for a specific time point requires saving them before:

>>> test("register_simulationdates", id_="0",
...      data=("firstdate_sim = 1996-01-03\n"
...            "lastdate_sim = 1996-01-05"))

>>> test("activate_simulationdates", id_="0")

>>> test("load_internalconditions", id_="0")
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: RuntimeError: While trying to execute method `GET_load_internalconditions`, the following error occurred: Conditions for ID `0` and time point `1996-01-03 00:00:00` are required, but have not been calculated so far.

For example, when restarting data assimilation subsequent forecasting periods, you might need to get and set all internal conditions from the client side. Then, you have two options. The more efficient way relies on methods GET_query_internalconditions() and POST_register_internalconditions(). Method GET_query_internalconditions() returns the information registered for the end of the current simulation period. All data is within a single nested dict object (created by the conditions property of class HydPy):

>>> test("register_simulationdates", id_="0",
...      data=("firstdate_sim = 1996-01-01\n"
...            "lastdate_sim = 1996-01-02"))

>>> test("activate_simulationdates", id_="0")

>>> conditions = test("query_internalconditions", id_="0",
...                   return_result=True)[13:]  
conditions = {'land_dill': {'states': {'ic': array([0.69171697, 1.19171697...

Due to the steps above, the returned dictionary agrees with the current state of the HydPy instance:

>>> sequences = f"HydPyServer.state.hp.elements.land_dill.model.sequences"
>>> test("evaluate",
...      data=f"ic_dill = {sequences}.states.ic")  
ic_dill = ic(0.691717, 1.191717, 0.692897,...

To show that registering new internal conditions also works, we first convert the string representation of the data to actual Python objects by using Python’s eval() function. Therefore, we need to clarify that “array” means the array creation function array() of numpy:

>>> import numpy
>>> conditions = eval(conditions, {"array": numpy.array})

Next, we modify an arbitrary state and convert the dictionary back to a single-line string:

>>> conditions["land_dill"]["states"]["ic"][:2] = 0.5, 2.0
>>> conditions = str(conditions).replace("\n", " ")

Now we can send the modified data back to the server by using the POST_register_internalconditions() method, which stores it for the start of the simulation period:

>>> test("register_internalconditions", id_="0", data=f"conditions = {conditions}")

>>> ic_dill = "self.state.conditions['0'][0]['land_dill']['states']['ic']"
>>> test("evaluate",
...      data=f"ic_dill = {ic_dill}")  
ic_dill = array([0.5       , 2.        , 0.69289697,...

After calling method GET_load_internalconditions(), the freshly registered states are ready to be used by the next simulation run:

>>> test("load_internalconditions", id_="0")

>>> test("evaluate",
...      data=f"ic_dill = {sequences}.states.ic")  
ic_dill = ic(0.5, 1.5, 0.692897,...

Keeping the internal conditions for multiple time points can use plenty of RAM. Use the GET method GET_deregister_internalconditions() to remove all conditions data available under the given id to avoid that:

>>> test("query_internalconditions", id_="0")  
conditions = {'land_dill': {'states': {'ic': array([0.6917...
>>> test("deregister_internalconditions", id_="0")

>>> test("query_internalconditions", id_="0")
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: RuntimeError: While trying to execute method `GET_query_internalconditions`, the following error occurred: No internal conditions registered under the id `0` for `1996-01-02 00:00:00`.

Some algorithms provide new information about initial conditions and require information on how they evolve during a simulation. For such purposes, you can use method GET_update_conditionitemvalues() to store the current conditions under an arbitrary id and use method GET_query_conditionitemvalues() to query them later. Note that this approach so far only works when using SetItem objects that modify their target sequence on the device or subunit level (please tell us if you encounter other relevant use-cases):

>>> test("update_conditionitemvalues", id_="0")

>>> test("query_conditionitemvalues", id_="0")  
ic_lahn_2 = [0.953246...]
ic_lahn_1 = [0.738365, ...]
sm_lahn_2 = [99.84802...]
sm_lahn_1 = [49.92944...]
quh = [0.00038...]

The second option for handling multiple “simultaneous” initial conditions is telling the HydPy server to read them from and write them to disk, which is easier but often less efficient due to higher IO activity. Use methods GET_load_conditions() and GET_save_conditions() for this purpose. Reading from or writing to different directories than those defined in multiple_runs.xml requires registering them beforehand. If we, for example, create a new empty directory with method POST_register_inputconditiondir(), loading conditions from it must fail:

>>> test("register_inputconditiondir", id_="0", data="inputconditiondir = new")

>>> test("load_conditions", id_="0")  
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: FileNotFoundError: While trying to execute method `GET_load_conditions`, the following error occurred: While trying to load the initial conditions of element `land_dill`, the following error occurred: [Errno 2] No such file or directory: ...land_dill.py'

>>> test("register_outputconditiondir", id_="0", data="outputconditiondir = new")

>>> test("save_conditions", id_="0")

Hence, we better first write suitable conditions into the new directory:

>>> lz_dill = "self.state.hp.elements.land_dill.model.sequences.states.lz"
>>> test("evaluate", data=f"lz_dill = {lz_dill}")  
lz_dill = lz(9.4921...)

To prove reading and writing conditions works, we first set the current value of sequence LZ of catchment “Dill” to zero:

>>> test("evaluate", data=f"nothing = {lz_dill}(0.0)")
nothing = None
>>> test("evaluate", data=f"lz_dill = {lz_dill}")
lz_dill = lz(0.0)

As expected, applying GET_load_conditions() on the previously written data resets the value of LZ:

>>> test("load_conditions", id_="0")

>>> test("evaluate", data=f"lz_dill = {lz_dill}")  
lz_dill = lz(9.4921...)

Use the GET methods GET_query_inputconditiondir() and GET_deregister_inputconditiondir() to query or remove the currently registered input condition directory:

>>> test("query_inputconditiondir", id_="0")
inputconditiondir = new
>>> test("deregister_inputconditiondir", id_="0")

>>> test("query_inputconditiondir", id_="0")
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: RuntimeError: While trying to execute method `GET_query_inputconditiondir`, the following error occurred: Nothing registered under the id `0`.  There is nothing registered, so far.

Use the GET methods GET_query_outputconditiondir() and GET_deregister_outputconditiondir() to query or remove the currently registered output condition directory:

>>> test("query_outputconditiondir", id_="0")
outputconditiondir = new
>>> test("deregister_outputconditiondir", id_="0")

>>> test("query_outputconditiondir", id_="0")
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: RuntimeError: While trying to execute method `GET_query_outputconditiondir`, the following error occurred: Nothing registered under the id `0`.  There is nothing registered, so far.

Above, we explained the recommended way to query the initial values of all or a subgroup of the available exchange items. Alternatively, you can first register the initial values and query them later, which is a workaround for retrieving initial and intermediate values with the same HTTP request (an OpenDA requirement):

>>> test("register_initialitemvalues", id_="1")

>>> test("query_itemvalues", id_="1")  
alpha = 2.0
beta = 1.0
lag = 5.0
damp = 0.5
sfcf_1 = 0.3
sfcf_2 = 0.2
sfcf_3 = [0.1, 0.2, 0.1, 0.2, 0.1, 0.2, 0.1, 0.2, 0.1, 0.2, 0.1, 0.2, 0.2, 0.2]
k4 = 10.0
t_headwaters = [[-0.29884...], [-0.70539...]]
ic_lahn_2 = [1.18494...]
ic_lahn_1 = [0.96404...]
sm_lahn_2 = [123.0]
sm_lahn_1 = [110.0, 120.0, 130.0, 140.0, 150.0, 160.0, 170.0, 180.0, 190.0, 200.0, 210.0, 220.0, 230.0]
quh = [10.0]
swe_headwaters = [[nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan]]
land_dill_factors_tmean = nan
land_dill_fluxes_qt = nan
land_dill_fluxes_qt_series = [nan, nan, nan, nan, nan]
land_dill_states_sm = [185.13164...]
land_lahn_1_states_sm = [99.27505...]
land_lahn_2_states_sm = [138.31396...]
land_lahn_3_states_sm = [101.31248...]
land_lahn_3_states_sm_series = [[nan, ...], [nan, ...], ..., [nan, ...]]
dill_nodes_sim_series = [nan, nan, nan, nan, nan]

In contrast to running a single simulation via method run_simulation(), the HydPy server does (usually) not write calculated time series automatically. Instead, one must manually call method GET_save_allseries():

>>> test("save_allseries", id_="0")

According to the fixed configuration of multiple_runs.xml, GET_save_allseries() wrote averaged soil moisture values into the directory mean_sm:

>>> import netCDF4
>>> from hydpy import print_values
>>> filepath = "LahnH/series/mean_sm/hland_v1_state_sm_mean.nc"
>>> with TestIO(), netCDF4.Dataset(filepath) as ncfile:
...     print_values(ncfile["state_sm"][:, 0])
211.238178, 0.0, 0.0, 0.0, 0.0

To save the results of subsequent simulations without overwriting the previous ones, change the current series writer directory by the GET method POST_register_serieswriterdir():

>>> test("register_serieswriterdir", id_="0", data="serieswriterdir = sm_averaged")

>>> test("save_allseries", id_="0")

>>> filepath = "LahnH/series/sm_averaged/hland_v1_state_sm_mean.nc"
>>> with TestIO(), netCDF4.Dataset(filepath) as ncfile:
...     print_values(ncfile["state_sm"][:, 0])
211.238178, 0.0, 0.0, 0.0, 0.0

GET_deregister_serieswriterdir() removes the currently set directory from the registry so that the HydPy server falls back to the configuration of multiple_runs.xml:

>>> test("query_serieswriterdir", id_="0")
serieswriterdir = sm_averaged
>>> test("deregister_serieswriterdir", id_="0")

>>> test("query_serieswriterdir", id_="0")
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: RuntimeError: While trying to execute method `GET_query_serieswriterdir`, the following error occurred: Nothing registered under the id `0`.  There is nothing registered, so far.

The same holds for time series to be written “just in time” during simulation runs. The temperature writer in multiple_runs.xml select the jit mode. This setting triggered that the HydPy server wrote the time series of sequences T and TN to the directory temperature during the last simulation:

>>> filepath = "LahnH/series/temperature/hland_v1_input_t.nc"
>>> with TestIO(), netCDF4.Dataset(filepath) as ncfile:
...     print_values(ncfile["input_t"][:, 0])
-0.298846, 0.0, 0.0, 0.0, 0.0

The input sequences P and EPN are instead reading their time series “just in time” (reading and writing data for the same IOSequence object is not supported). We query the last read value of EPN for the Dill catchment:

>>> epn = f"HydPyServer.state.hp.elements.land_dill.model.sequences.inputs.epn"
>>> test("evaluate", data=f"epn_dill = {epn}")  
epn_dill = epn(0.2854...)

We can change the series writer directory before starting another simulation run to write the time series of T and TN to another directory:

>>> test("register_serieswriterdir", id_="0", data="serieswriterdir = temp")

>>> test("simulate", id_="0")

>>> filepath = "LahnH/series/temp/hland_v1_input_t.nc"
>>> with TestIO(), netCDF4.Dataset(filepath) as ncfile:
...     print_values(ncfile["input_t"][:, 0])
-0.298846, 0.0, 0.0, 0.0, 0.0

The “just in time” reading of the series of “P” and “EPN” still worked, showing the registered series directory “temp” only applied for writing data:

>>> test("evaluate", data=f"epn_dill = {epn}")  
epn_dill = epn(0.2854...)

Changing the series reader directory works as explained for the series writer directory. After setting it to an empty folder, GET_load_allseries() and GET_simulate() cannot find suitable files and report this problem:

>>> test("register_seriesreaderdir", id_="0", data="seriesreaderdir = no_data")

>>> test("query_seriesreaderdir", id_="0")
seriesreaderdir = no_data

>>> test("load_allseries", id_="0")  
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: FileNotFoundError: While trying to execute method `GET_load_allseries`, the following error occurred: While trying to load the time-series data of sequence `t` of element `land_dill`, the following error occurred: [Errno 2] No such file or directory: ...land_dill_input_t.asc'

>>> test("simulate", id_="0")  
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: FileNotFoundError: While trying to execute method `GET_simulate`, the following error occurred: While trying to prepare NetCDF files for reading or writing data "just in time" during the current simulation run, the following error occurred: No file `...hland_v1_input_p.nc` available for reading.

After deregistering the “no_data” directory, both methods work again:

>>> test("deregister_seriesreaderdir", id_="0")

>>> test("query_seriesreaderdir", id_="0")
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: RuntimeError: While trying to execute method `GET_query_seriesreaderdir`, the following error occurred: Nothing registered under the id `0`.  There is nothing registered, so far.

>>> test("load_allseries", id_="0")

>>> test("simulate", id_="0")

As described for time series, one must explicitly pass (comparable) requests to the HydPy Server to let it write parameter control files. The control files reflect the current parameter values of all model instances:

>>> test("register_outputcontroldir", id_="0", data="outputcontroldir = calibrated")

>>> test("query_outputcontroldir", id_="0")
outputcontroldir = calibrated

>>> test("save_controls", id_="0")

>>> with TestIO(), open("LahnH/control/calibrated/land_dill.py") as file_:
...     print(file_.read())  
# -*- coding: utf-8 -*-

from hydpy.models.hland_v1 import *

simulationstep("1d")
parameterstep("1d")
...
beta(1.0)
...

>>> parameterstep = "hydpy.pub.options.parameterstep"
>>> simulationstep = "hydpy.pub.options.simulationstep"
>>> beta = "HydPyServer.state.hp.elements.land_dill.model.parameters.control.beta"
>>> test("evaluate", data=(f"simulationstep = {simulationstep}\n"
...                        f"parameterstep = {parameterstep}\n"
...                        f"beta = {beta}"))
simulationstep = Period("1d")
parameterstep = Period("1d")
beta = beta(1.0)

>>> test("deregister_outputcontroldir", id_="0")

>>> test("query_outputcontroldir", id_="0")
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: RuntimeError: While trying to execute method `GET_query_outputcontroldir`, the following error occurred: Nothing registered under the id `0`.  There is nothing registered, so far.

To close the HydPy server, call GET_close_server():

>>> test("close_server")

>>> process.kill()
>>> _ = process.communicate()

server: _HTTPServerBase

state: ClassVar[ServerState]

extensions_map: ClassVar[Dict[str, str]]

do_GET() → None

Select and apply the currently requested GET method.

do_POST() → None

Select and apply the currently requested POST method.

GET_execute() → None

Execute an arbitrary number of GET methods.

The method names must be passed as query parameters, as explained in the main documentation on module servertools.

POST_execute() → None

Execute an arbitrary number of POST and GET methods.

The method names must be passed as query parameters, as explained in the main documentation on module servertools.

POST_evaluate() → None

Evaluate any valid Python expression with the HydPy server process and get its result.

Method POST_evaluate() serves to test and debug, primarily. The main documentation on class HydPyServer explains its usage.

For safety purposes, method POST_evaluate() only works if you start the HydPy Server in debug mode by writing “debugging=enable”, as we do in the examples of the main documentation on class HydPyServer. When not working in debug mode, invoking this method results in the following error message:

>>> from hydpy.examples import prepare_full_example_1
>>> prepare_full_example_1()
>>> from hydpy import run_subprocess, TestIO
>>> with TestIO():
...     process = run_subprocess(
...         "hyd.py start_server 8080 LahnH multiple_runs_alpha.xml",
...         blocking=False, verbose=False)
...     _ = run_subprocess("hyd.py await_server 8080 10", verbose=False)
>>> from urllib import request
>>> request.urlopen("http://127.0.0.1:8080/evaluate", data=b"")
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 500: RuntimeError: While trying to execute method `POST_evaluate`, the following error occurred: You can only use the POST method `evaluate` if you have started the `HydPy Server` in debugging mode.

>>> _ = request.urlopen("http://127.0.0.1:8080/close_server")
>>> process.kill()
>>> _ = process.communicate()

GET_status() → None

Return “status = ready” as soon as possible.

GET_version() → None

Return Hydpy’s version number.

GET_close_server() → None

Stop and close the HydPy server.

GET_query_itemtypes() → None

Get the types of all current exchange items.

GET_query_changeitemtypes() → None

Get the types of all current exchange items supposed to change the values of Parameter, StateSequence, or LogSequence objects.

GET_query_parameteritemtypes() → None

Get the types of all current exchange items supposed to change the values of Parameter objects.

GET_query_inputitemtypes() → None

Get the types of all current exchange items supposed to change the series of InputSequence objects.

GET_query_conditionitemtypes() → None

Get the types of all current exchange items supposed to change the values of StateSequence or LogSequence objects.

GET_query_outputitemtypes() → None

Get the types of all current exchange items supposed to return the values or series of FactorSequence or FluxSequence objects in the “setitem style”.

GET_query_getitemtypes() → None

Get the types of all current exchange items supposed to return the values of Parameter or Sequence_ objects or the time series of IOSequence objects in the “getitem style”.

GET_query_itemsubnames() → None

Get names (suitable as IDs) describing the individual values of all current exchange objects.

GET_query_changeitemsubnames() → None

Get names (suitable as IDs) describing the individual values of all current exchange objects supposed to change the values of Parameter, StateSequence, or LogSequence objects.

GET_query_parameteritemnames() → None

Get names (suitable as IDs) describing the individual values of all current exchange objects supposed to change the values of Parameter objects.

GET_query_inputitemnames() → None

Get names (suitable as IDs) describing the individual values of all current exchange objects supposed to change the values of InputSequence objects.

GET_query_conditionitemnames() → None

Get names (suitable as IDs) describing the individual values of all current exchange objects supposed to change the values of StateSequence or LogSequence objects.

GET_query_outputitemnames() → None

Get names (suitable as IDs) describing the individual values of all current exchange objects supposed to return the values of or series of FactorSequence or FluxSequence objects in the “setitem style”.

GET_query_getitemsubnames() → None

Get names (suitable as IDs) describing the individual values of all current exchange objects supposed to return the values of Parameter or Sequence_ objects or the time series of IOSequence objects in the “getitem style”.

GET_query_initialitemvalues() → None

Get the initial values of all current exchange items.

GET_register_initialitemvalues() → None

Register the initial values of all current exchange items under the given id.

Implemented as a workaround to support OpenDA. Better use method GET_query_initialitemvalues().

GET_query_initialchangeitemvalues() → None

Get the initial values of all current exchange items supposed to change the values of Parameter, InputSequence, StateSequence, or LogSequence objects.

GET_register_initialchangeitemvalues() → None

Register the initial values of all current exchange items supposed to change the values of Parameter, InputSequence, StateSequence, or LogSequence objects under the given id.

Implemented as a workaround to support OpenDA. Better use method GET_query_initialchangeitemvalues().

GET_query_initialparameteritemvalues() → None

Get the initial values of all current exchange items supposed to change the values of Parameter objects.

GET_register_initialparameteritemvalues() → None

Register the initial values of all current exchange items supposed to change the values of Parameter objects under the given id.

Implemented as a workaround to support OpenDA. Better use method GET_query_initialparameteritemvalues().

GET_query_initialinputitemvalues() → None

Get the initial values of all current exchange items supposed to change the series of InputSequence objects.

GET_register_initialinputitemvalues() → None

Register the initial series of all current exchange items supposed to change the values of InputSequence objects under the given id.

Implemented as a workaround to support OpenDA. Better use method GET_query_initialinputitemvalues().

GET_query_initialconditionitemvalues() → None

Get the initial values of all current exchange items supposed to change the values of StateSequence or LogSequence objects.

GET_register_initialconditionitemvalues() → None

Register the initial values of all current exchange items supposed to change the values of StateSequence or LogSequence objects under the given id.

Implemented as a workaround to support OpenDA. Better use method GET_query_initialconditionitemvalues().

GET_query_initialoutputitemvalues() → None

Get the initial values of all current exchange items supposed to return the values or sequences of FactorSequence or FluxSequence objects in the “setitem style”.

GET_register_initialoutputitemvalues() → None

Register the initial values of all current exchange items supposed to return the values or sequences of FactorSequence or FluxSequence objects in the “setitem style” under the given id.

Implemented as a workaround to support OpenDA. Better use method GET_query_initialoutputitemvalues().

GET_query_initialgetitemvalues() → None

Get the initial values of all current exchange items supposed to return the values of Parameter or Sequence_ objects or the time series of IOSequence objects in the “getitems style”.

GET_register_initialgetitemvalues() → None

Register the initial values of all current exchange items supposed to return the values of Parameter or Sequence_ objects or the time series of IOSequence objects in the “getitem style” under the given id.

Implemented as a workaround to support OpenDA. Better use method GET_query_initialgetitemvalues().

GET_query_initialisationtimegrid() → None

Return the general init time grid.

POST_register_simulationdates() → None

Register the send simulation dates under the given id.

GET_activate_simulationdates() → None

Activate the simulation dates registered under the given id.

GET_query_simulationdates() → None

Return the simulation dates registered under the given id.

GET_query_itemvalues() → None

Get the values of all ExchangeItem objects registered under the given id.

POST_register_changeitemvalues() → None

Register the send values of all ChangeItem objects under the given id.

GET_activate_changeitemvalues() → None

Activate the values of the ChangeItem objects registered under the given id.

GET_query_changeitemvalues() → None

Get the values of all ChangeItem objects registered under the given id.

POST_register_parameteritemvalues() → None

Register the send parameter values under the given id.

GET_activate_parameteritemvalues() → None

Activate the parameter values registered under the given id.

GET_query_parameteritemvalues() → None

Return the parameter values registered under the given id.

POST_register_inputitemvalues() → None

Register the send input item values under the given id.

GET_activate_inputitemvalues() → None

Apply the input item values registered under the given id to modify the current InputSequence values.

GET_update_inputitemvalues() → None

Convert the current InputSequence values to input item values (when necessary) and register them under the given id.

GET_query_inputitemvalues() → None

Return the input item values registered under the given id.

POST_register_conditionitemvalues() → None

Register the send condition item values under the given id.

GET_activate_conditionitemvalues() → None

Apply the condition item values registered under the given id to modify the current StateSequence and LogSequence values.

GET_update_conditionitemvalues() → None

Convert the current StateSequence and LogSequence values to condition item values (when necessary) and register them under the given id.

GET_query_conditionitemvalues() → None

Return the condition item values registered under the given id.

GET_update_outputitemvalues() → None

Convert the current FactorSequence and FluxSequence values or series to output item values (when necessary) and register them under the given id.

GET_query_outputitemvalues() → None

Return the output item values registered under the given id.

GET_save_internalconditions() → None

Register the StateSequence and LogSequence values of the HydPy instance for the current simulation endpoint under the given id.

GET_load_internalconditions() → None

Activate the StateSequence or LogSequence values registered for the current simulation start point under the given id.

When the simulation start point is identical with the initialisation time point, and you do not register alternative conditions manually, method GET_load_internalconditions() uses the “original” initial conditions of the current process (usually those of the conditions files of the respective HydPy project).

POST_register_internalconditions() → None

Register the send internal conditions under the given id.

GET_deregister_internalconditions() → None

Remove all internal condition directories registered under the given id.

GET_query_internalconditions() → None

Get the internal conditions registered under the given id.

POST_register_inputconditiondir() → None

Register the send input condition directory under the given id.

GET_deregister_inputconditiondir() → None

Remove the input condition directory registered under the id.

GET_query_inputconditiondir() → None

Return the input condition directory registered under the id.

POST_register_outputconditiondir() → None

Register the send output condition directory under the given id.

GET_deregister_outputconditiondir() → None

Remove the output condition directory registered under the id.

GET_query_outputconditiondir() → None

Return the output condition directory registered under the id.

GET_load_conditions() → None

Load the (initial) conditions.

GET_save_conditions() → None

Save the (resulting) conditions.

GET_update_getitemvalues() → None

Register the current GetItem values under the given id.

For GetItem objects observing time series, method GET_update_getitemvalues() registers only the values within the current simulation period.

GET_query_getitemvalues() → None

Get the GetItem values registered under the given id.

GET_simulate() → None

Perform a simulation run.

POST_register_seriesreaderdir() → None

Register the send series reader directory under the given id.

GET_deregister_seriesreaderdir() → None

Remove the series reader directory registered under the id.

GET_query_seriesreaderdir() → None

Return the series reader directory registered under the id.

POST_register_serieswriterdir() → None

Register the send series writer directory under the given id.

GET_deregister_serieswriterdir() → None

Remove the series writer directory registered under the id.

GET_query_serieswriterdir() → None

Return the series writer directory registered under the id.

GET_load_allseries() → None

Load the time series of all sequences selected for (non-jit) reading.

GET_save_allseries() → None

Save the time series of all sequences selected for (non-jit) writing.

POST_register_outputcontroldir() → None

Register the send output control directory under the given id.

GET_deregister_outputcontroldir() → None

Remove the output control directory registered under the id.

GET_query_outputcontroldir() → None

Return the output control directory registered under the id.

GET_save_controls() → None

Save the control files of all model instances.

hydpy.exe.servertools.start_server(socket: int | str, projectname: str, xmlfilename: str, load_conditions: bool | str = True, load_series: bool | str = True, maxrequests: int | str = 5, debugging: Literal['enable', 'disable'] = 'disable') → None

Start the HydPy server using the given socket.

The folder with the given projectname must be available within the current working directory. The XML configuration file must be placed within the project folder unless xmlfilename is an absolute file path. The XML configuration file must be valid concerning the schema file HydPyConfigMultipleRuns.xsd (see class ServerState for further information).

The HydPyServer allows for five still unhandled requests before refusing new connections by default. Use the optional maxrequests argument to increase this number (which might be necessary when parallelising optimisation or data assimilation):

>>> from hydpy.examples import prepare_full_example_1
>>> prepare_full_example_1()
>>> command = (
...     "hyd.py start_server 8080 LahnH multiple_runs_alpha.xml "
...     "debugging=enable maxrequests=100")
>>> from hydpy import run_subprocess, TestIO
>>> with TestIO():
...     process = run_subprocess(command, blocking=False, verbose=False)
...     result = run_subprocess("hyd.py await_server 8080 10", verbose=False)

>>> from urllib import request
>>> command = "maxrequests = self.server.request_queue_size"
>>> response = request.urlopen("http://127.0.0.1:8080/evaluate",
...                            data=bytes(command, encoding="utf-8"))
>>> print(str(response.read(), encoding="utf-8"))
maxrequests = 100

>>> _ = request.urlopen("http://127.0.0.1:8080/close_server")
>>> process.kill()
>>> _ = process.communicate()

Please see the documentation on method POST_evaluate() that explains the “debugging” argument.

Note that function start_server() tries to read the “mime types” from a dictionary stored in the file mimetypes.txt available in subpackage conf and passes it as attribute extension_map to class HydPyServer. The reason is to avoid the long computation time of function init() of module mimetypes, usually called when defining class BaseHTTPRequestHandler of module http.server. If file mimetypes.txt does not exist or does not work for , start_server() calls init() as usual, (over)writes mimetypes.txt and tries to proceed as expected.

hydpy.exe.servertools.await_server(port: int | str, seconds: float | str) → None

Block the current process until either the HydPy server is responding on the given port or the given number of seconds elapsed.

>>> from hydpy import run_subprocess, TestIO
>>> with TestIO():  
...     result = run_subprocess("hyd.py await_server 8080 0.1")
Invoking hyd.py with arguments `await_server, 8080, 0.1` resulted in the following error:
<urlopen error Waited for 0.1 seconds without response on port 8080.>
...

>>> from hydpy.examples import prepare_full_example_1
>>> prepare_full_example_1()
>>> with TestIO():
...     process = run_subprocess(
...         "hyd.py start_server 8080 LahnH multiple_runs.xml",
...         blocking=False, verbose=False)
...     result = run_subprocess("hyd.py await_server 8080 10", verbose=False)

>>> from urllib import request
>>> _ = request.urlopen("http://127.0.0.1:8080/close_server")
>>> process.kill()
>>> _ = process.communicate()