Introduction
Reconciling models with observations is often ill-conditioned,
especially when single performance measures, such as mean square
errors, are used to infer models . This ill-condition is
often attributed to our attempt to extract higher dimensional
information (about the model) from a single dimension of information
given by the measure. It is therefore often recommended to select
hydrological models either using multiple signatures of hydrological
response or multiple objectives, the idea being to constrain the model
selection exercise . Different signatures
or multiple objectives are different measures of
closeness, which when orthogonal, provide complementary pieces of
information to select a better constrained model
. The constraints imposed on the model
selection exercise in effect may condition the problem well.
But is the issue of ill-conditionness limited to the discourse of the
number of measures used? Can we say something about the nature of
conditionness first before addressing the question of how it can be
ameliorated by, for example, the use of multiple signatures or
objectives? A definition of ill-conditioness and the consequences of
an ill-conditioned hydrological model problem are therefore
needed. are the first to formally introduce the notion
of ill-posedness in hydrological modeling and emphasized the
importance of prior specification in correcting or properly
conditioning ill-posed model selection problems. Their approach appears
to have been motivated by the issue of non-identifiability, that not
all parameters of interest are often decipherable from limited
rainfall–runoff information . We ask an equivalent
question and attempt to formalize what ill-conditionness means: what
happens when an ill conditioned model is selected to represent the
underlying hydrological system? Since it fails to exploit interesting
information in the data, there is uncertainty in system representation
. Should not this uncertainty in assessing
structure deficiency depend on the class of model structures which are
used to assess deficiencies? The characteristics of uncertainty in
system representation can then identify the consequences of
ill-conditioned model selection problem and hence define
ill-conditioned model selection.
We characterize uncertainty in hydrological system representation as
composed of non-uniqueness and instability in system
representation. Non-uniqueness in system representation (of the
underlying processes) is synonymous to equifinality
. Meanwhile, instability refers to inconsistency in
process representation as more information on the underlying processes
is available. That is, a set of models that appears to be more
representative on smaller pieces of information is not the best
representation as more pieces of information are brought to bear
. The instability of model representation is not
specific to the case when multiple measures define different pieces of
information as elucidated by . Instability may exist
even when using a single measure of performance but when the
information content increases as the amount of available data
increases. This is equivalent to suggesting that information content
of smaller datasets of similar lengths is dissimilar. Instability can
then be rephrased as a changing set of good system representations
(models) of underlying processes when different data sets of similar
lengths are used since different data sets of similar lengths may have
dissimilar information content. This may especially be the case when
data size is small and noisy, assuming that the observations are
samples from a probability distribution defined by the underlying
processes. The small data sets suffer from sampling uncertainty.
In other words, different systems representations may appear to be
suitable on different realizations. This may also partly explain how
equifinal models may distinguish themselves when additional pieces of
information are provided. Equifinal models on one set of data,
assuming use of a single measure of performance, may no longer be
equifinal on another set of data (or on another piece of information)
if the two data sets contain different information. This paper
demonstrates that instability of a given model over different
realizations of data can be understood and controlled by what we term
as model output space. Ill-conditionness of model identification can
then be corrected by constraining the extent of model output space. We
call the extent of model output space as the measure of complexity
since its regularization would lead to a stabler representation of
underlying processes .
If an unstable system representation (model) is used for model
prediction on yet unseen data (or on another realization of underlying
processes), its instability directly translates into uncertainty in
its prediction. Instability in model representation can also be seen
as poor representative of underlying processes since that selected
model will not be a good representation of the underlying processes on
another realization. Here by prediction we mean model simulation of
a variable of interest conditioned by certain future values of input
(forcing) variables. The regularization of the model selection problem
by complexity, which corrects for the instability in system
representation, then ameliorates prediction uncertainty. Complexity
controlled model selection selects a model that predicts future values
of a variable of interest with least uncertainty amongst the set of
competing models .
The Bayesian treatment of prediction uncertainty and model complexity
is through its specification of a marginal likelihood function of
a hydrological model structure. The marginal likelihoods of
hydrological model structures are often approximated by measures such
as AIC (Akaike Information Criterion), BIC (Bayesian Information
Criterion) and KIC (Kashyap Information Criterion)
. These measures therefore embody Bayesian
interpretation of model prediction uncertainty.
Less complex hydrological models are often preferred for stable system
representaiton . Low computational complexity of
simulations of models is also often desired . We
here however only explore the concept of model complexity in context
of stable system representation. Often models with low parameter
dimensionality (i.e. less number of parameters) are considered less
complex and hence are associated with low prediction
uncertainty. Whether this is always the case remains to be explored.
We follow an alternate to Bayesian, i.e. frequentist, approach
to model complexity to explore whether parameter
dimensionality is the only indicator of model complexity, instability
in system represention and hence prediction uncertainty. One strength
of a frequentist approach is the ease with which unstable model
representation can be geometrically interpreted
. It also makes less
restrictive assumptions. After illustrating the context in which model
complexity has been defined, i.e. in context of unstable model
representation of underlying processes and prediction uncertainty, we
explore the question of whether a hydrological model with more
parameters is more complex or less complex in context of its influence
on stability of system representation and hence prediction
uncertainty. Within this context, we test the hypothesis that model
complexity also depends on the magnitude of parameters that define
constitutive relationships and model architecture.
The paper is organized as follows. Section 2 on methodology provides
the theory, the models structures, datasets and the algorithms
used. The theory first explores the connection between unstable
process representation and model complexity and then provides
justification for complexity regularized model selection to ameliorate
instability in system representation. It then follows up with how
hydrological model complexity may be calculated. Algorithms for
estimating complexity of an arbitrary hydrological model is then
presented and the data sets to be used are introduced. Finally the two
model structures, SAC-SMA (Sacramento Soil Moisture Accounting model)
and SIXPAR (Six Parameter model), are introduced. Section 3 presents
and discusses the results. Here complexities of the two model
structures are estimated to demonstrate the applicability of the
algorithms. Then parameter ranges of SIXPAR are varied in a controlled
manner to demonstrate the effect of the magnitude of parameters on
model complexity, in particular in comparison with complexity computed
for SAC-SMA model structure. Finally Sect. 4 concludes.
Methodology
Unstable system representation and model complexity
We now illustrate that instability of a given model over different
realizations of data can be measured by what we term as model output
space. Thereupon we demonstrate that ill-conditionedness of model
structure identification is corrected by constraining (in a certain
fashion, to be deliberated upon further) the extent of model structure
output space (that is a union of output spaces of models that comprise
a model structure).
In order to do so, we first define what we mean by model output
space. If N is the sample size, the model output space is defined
in a N-dimensional space. It is a collection of all model outputs
that are obtained for all possible N-dimensional input forcings that
underlying input processes may generate. Let distance in this space be
measured by a metric such as mean of absolute deviations or by any
other measure of similarity. It is however required that the measure
of similarity obeys the conditions of being a metric (see Appendix
A for further details). Figure illustrates the concept of
model output space.
We define instability of a given model by the variability in the
differences between its outputs over two different realizations of
data. A model then is more unstable if it tends to have larger
differences between model simulations for any given pair of data
realizations. Such a definition is sufficient to encapsulate the
notion of inconsistency in process representation by a model. This is
because it is quite likely that a highly unstable model that appears
to be a suitable representation of the underlying system on one piece
of information may not be a suitable representation on another or more
pieces of information. Figure illustrates this concept
further.
Let T represent a set of observed output values for different
realizations of input forcing. For illustration purposes we have
assumed N=2 in Fig. , hence we have a 2 dimensional space
in which the output space is defined. Let o1=(o11,o12) represent one observed output value for a given input
forcing. Let p1 be the simulation of a model parameterized by
θ1 corresponding to the same input forcing. A collection of
such simulations over all possible input forcings define the output
space M(θ1) of the model. Let o2 and
p2 be another pair of points in sets T and
M(θ1) corresponding to another realization of input
forcing. Let A, B, C and
D be the vectors connecting the 4
points (see Fig. ). Let ‖.‖ measure the magnitude of
vectors and define the metric used. For example ‖A‖=d(p2,o2), where d(p2,o2) is
a metric that measures the nearness between two points in the model
output space, for example mean absolute error or any other measure
that satisfies the conditions of being a metric (see Appendix A). Thus
‖A‖ and ‖C‖ measure the similarity of model
representations of the output to the observed values for two different
input forcings. Meanwhile ‖B‖ measures the closeness of two
model representations themselves and ‖D‖ measures the
closeness of the two observed time series.
Using the triangle inequality, see Appendix B, it can then be shown that
|‖A‖-‖C‖|⩽‖B‖+‖D‖. Thus the deviation in performance of a model
over two different information sources is bounded by ‖B‖
that measures how large is the model output space. If we now consider
another model parameterized by θ2 that belongs to the same
model structure as the model parameterized by θ1, we can
define a model structure (here a model structure is defined as
composed of of models corresponding to parameter sets θ1 and
θ2) output space that is a union of model output spaces
M(θ1) and M(θ2) (Fig. ). One
can thus obtain model structure output spaces for arbitrary model
structures.
We now consider a case of nested model structures ∧1 and
∧2 such that all process representations possessed by
∧2 are also possessed by ∧1 but not vice versa
(Fig. ). This is to elucidate the role of the size of model
structure output space in controlling the uncertainty in representing
underlying processes. For an observed data point let o1 be
an observation of underlying processes and let p11 and
p12 be the best model representations provided by
two model structures ∧1 and ∧2. The two simulations
p11 and p12 correspond to models parameterised by
θ1∗ and θ2∗, obtained from model structures
∧1 and ∧2 respectively, which are most similar to
o1 in simulations. Since ∧2 is nested within
∧1, if p11 is not the same as p12 then
p11 is closer to the observation o1. However, if one
observes another realization o2 of the underlying processes,
the performance of model parameterized by θ1∗ has more
possibility to vary than the performance of model parameterized by
θ2∗, since the output space of ∧2 lies nested
inside the output space of ∧1. If p22 is the
response provided by θ2∗ to the input forcing
corresponding to o2, the response provided by
θ1∗ to the same input forcings may vary widely, such as
p21 or p2′1, in terms of its distance
from o2. This possibility of more variable response to the
same input forcing emerges from the larger output space of ∧1
in comparison to ∧2.
We illustrate this further through a synthetic case study. Appendix C
describes the set up in detail. 100 pairs of synthetic data sets,
corrupted by input and output noises, are generated from a simple
single linear reservoir model. Two nested model structures are then
considered. These are model structures defined by linear reservoir
models (∧2) and by two reservoir models (∧1). In the
case of the latter, each of the two reservoirs are linear reservoirs. The
top reservoir feeds the lower reservoir via percolation as well as
produces runoff. Meanwhile the lower reservoir produces only
runoff. It is evident that (∧1) is more flexible than
(∧2) and therefore intuitively has more propensity to produce
unstable system representations. The differences |‖A‖-‖C‖| are calculated for each of the 100 pairs and
kernel density estimates of Pr(|‖A‖-‖C‖|⩾ϵ) are
produced. Similarly Pr(‖B‖⩾ϵ) is estimated. Both these probability of
exceedences are plotted in Fig. .
Let E be some event and let Pr(E)
define the probability of occurance of that event. We first note that
Pr(‖B‖⩾ϵ) is larger for
two reservoir model structure ∧1 than for single reservoir
model structure ∧2 for nearly all ϵ⩾0. Let
E[‖B‖] be the expected value of
‖B‖ over multiple realizations of data. If the extent of
a model structure output space is measured by
E[‖B‖], i.e. what is the distance
between two arbitrary model simulations in expected sense, we note
that the extent of ∧1 is larger than ∧2. This is
because E[‖B‖]=∫0∞Pr(‖B‖⩾ϵ)dϵ. Thus the distance between any
two simulations is expected to be larger for ∧1 than for
∧2 since Fig. b demonstrates that
Pr(‖B‖⩾ϵ) is larger for
∧1 for nearly all ϵ⩾0. The extent of model
output space as measured by E[‖B‖] may
be able to distinguish between model structures in terms of stability
in system representation. We later provide further motivation for why
it can be used as a measure to control for instability in system
representation and, by doing so, we provide the context for defining
it a measure of model complexity.
Imagine uncertainty in process representation as the possibility of
more variable responses to the same input forcing. In the case of
nested model structures, it is due to larger size of structure output
space. Hence it is a measure of structural complexity since larger
complexity leads to higher possibility of more variable responses. The
definition of complexity is intuitive since structure output space of
∧1 is larger than ∧2 because it has model concepts not
in ∧2. Hence it is more complex. Thus, uncertainty in system
representation can be controlled by controlling for model structure
complexity, at least when nested model structures are considered.
Figure a also demonstrates that deviation in performance of
system representations from model structure ∧2 is often larger
than ∧1, to the extent that Pr(|‖A‖-‖C‖|⩾ϵ) is larger for nearly
all ϵ⩾0. Thus, process representations from model
structure ∧2 is expected to be more unstable than
∧1. The similarity in the ordering of complexity and
instability thus suggest that constraining the complexity of model
structures can control instability in representation of underlying
processes. Further, model structure complexity is a measure of
instability in process representation in the sense that larger model
structure complexity implies larger possibility of unstable process
representation (or higher uncertainty in process representation).
One can now flip the notion of uncertainty in process representation
by considering the variability in system representations when
a modeler has the liberty to select a new representation as new
information in the form of another realization of observations comes
to fore. Since ∧1 is more complex than ∧2, the
variation, in best representations, over different
realizations of observations, obtained from ∧1 is larger than
when they are obtained from ∧2. This is because ∧2 is
nested within ∧1 and this leads to the possibility of larger
variation in distances between best model representations and
observations for the latter. This is also illustrated in
Fig. . One realization of observations o1 results
in a selection of models corresponding to θ1∗ and
θ2∗ from model structures ∧1 and ∧2
respectively. However for another realization o2, the model
structure ∧2 retains the same model representation while the
model structure ∧1, owing to its more flexible structure,
allows the selection of another model representation
θ̃1. Since the model structure ∧2 is nested
within ∧1, model representations chosen from ∧1 would
at least be as unstable as those chosen from ∧1, if not more,
but never less. The figure therefore illustrates that a more complex
model structure results in a more unstable representation of the
underlying processes. Thus it is necessary to control the complexity
of a model selection problem in a certain fashion if a “stable”
process representation is desired.
Following the synthetic case study presented in Appendix C,
Fig. demonstrates the variability in best system
representations from the two model
structures. Figure a plots the kernel density estimate of
variability in process representations from ∧1 over 100 data
pair realizations while Fig. b plots the pairwise kernel
density estimate of the same for ∧1. It is evident from
Fig. that ∧1 offers more flexibility to accomodate
sample variability since it has higher complexity, especially by the
tradeoff between k3 and k1. One can observed this behavior
by noting that bivariate densities of θ1∗ often have
higher values of k1 and lower values of k3 when compared
with the bivariate densities of θ̃1. The parametric
variation offered by ∧2 is rather limited as witnessed by the
cumulative density functions of θ2∗ and
θ̃2.
Complexity regularized model selection
Abstract parameterization
Both Figs. and suggest that controlling for
the complexity in a model selection exercise may stabilize
the representation of underlying processes. This is akin to
“correcting” the ill-posedness of model selection
problem by constraining the complexity of the model structures
used. This is equivalent to regularized model selection problem.
Let a vector y0={y0(1),y0(2),…,y0(N)} define
the set of observations of a variable of prediction interest such as
streamflow. It represents a realization of observations
o. Similarly, let forcing be represented by
x={x1,x2,…,xN} where x1 may not be univariate,
though assumed here to be univariate for simplicity without any loss
of generality. Further let a model from a model structure ∧ be
represented by a parameter set θ that for given forcing
x simulates y(x;θ)={y(t,x;θ)}t=1,…,N. The prediction variable thus represents
p. Let ξN(y0,x;θ) be defined as
empirical risk that measures the performance of the model in terms of
deviations of its predictions from the observed, for example by mean
absolute error,
ξN(y0,x;θ)=∑t=1N|y(t,x;θ)-y0(t)|N.
This represents ‖p-o‖, where we have assumed mean of
absolute deviations as the metric.
Let us now reformulate the definition of a model structure wherein its
internal architecture of how various subsystem representations are
connected as well as its parameters can both be defined by an abstract
parameter set α. That is, both a model structure, for
e.g. ∧, and a model from the structure, for e.g. θ, are
parameterized by α. Consider the linear and the two linear
reservoirs model (as discussed in the previous section). The linear
reservoir model has only 1 parameter, i.e. the recession parameter
k∈[0,1] (dimension: [1/T]). Meanwhile, the two reservoir model has
3 recession parameters k={k1,k2,k3}∈[0,1]3
(dimensions: [1/T]3). If we now define the abstract parameter set
α={α1,α2,α3}∈[0,1]3 then we can
describe both the model structures. Model structure of a single
reservoir model can be described by the set ∧1={0}×{0}×[0,1], in which case α1 and α2
is restricted to 0 while α3 is allowed to vary between
[0,1]. The two reservoir model structure can be described by the same
parameter α, which is less constrained and belong to
[0,1]×[0,1]×[0,1]. Thus such a representation not only
distinguishes between two model structures in terms of its different
subsystem architecture (one vs. two reservoir model structure) but also
distinguishes in terms of its parameter magnitudes. For example, in
this representation a two reservoir model structure defined by
{α:α∈[0,1]×[0.5,1]×[0,1]} that only
permits fast flow from the second reservoir is different from a model
structure {α:α∈[0,1]×[0,1]×[0,1]} that
does not restrict the nature of flow from the second
reservoir. Equation () can be reformulated in terms of
α as,
ξN(y0,x;α)=∑t=1N|y(t,x;α)-y0(t)|N.
This represents ‖p-o‖.
By doing this we no longer distinguish between a model and a model
structure and allow models to seamlessly change their model structure
by changing α. Then a model and its corresponding model
structure is represented by y(t,x;α). We suppress t,x and represent a model by y(α). Further, since
a distinction between a model and a model structure has been dissolved
by using α, any compact set of αs can now be called
a model structure.
A continuum of model structures defined by complexity
Let Φ(y(α)) be the complexity (here the extent of model
output space) of the model y(α). We now note that the model
output of any hydrological model is continuous in its
parameters. Further, the extent of model output space is continuous in
model outputs (the extent of one model output space is smaller than
another if two simulations of the former are closer than the latter
for any given pair of input forcing). Therefore, Φ(y(α)) is
continuous in α. In other words, a set ∧={α:Φ(y(α))⩽c} is compact and defines a model
structure. By extension, we can obtain a sequence of model structures
∧m using the inequality Φ(∧)⩽cm for
a sequence of cm, where m=1,2,…,j... What this says is
that if the difference between any two upper bounds on
Φ(y(α)) is small, the corresponding model structures are
similar. Based on our construction, we note that a model structure
here is a result not just of the architecture of how various model
components are interconnected but also how they are
parameterised. Thus model structures with different architecture and
parameterization may be deemed similar.
A model structure is then nested within another model structure if the
complexity of the former is smaller than the latter. Formally,
∧1={α:Φ(y(α))≤c1} is nested within
∧2={α:Φ(y(α))≤c2} if c1≤c2. A continuum of model structures may therefore be obtained by
a sequence of c. The nomenclature “continuum of model structures”
has also been invoked elsewhere .
This is interesting because a definitive statement on structure
complexity based on parameter ranges or parameter dimensionality,
i.e. without knowing their complexity in advance, can only be made if
the corresponding structures are nested. For a given model structure,
such a statement can only be made if parameter ranges of one are
a subset of another. But the effect of parameter dimensionality on
model complexity, jointly with parameter magnitudes is not always
clear. This is because the abstract parameters corresponding to
parameter dimensionality and their interaction with other “real”
parameters is not evident. The effect of parameter magnitudes on model
complexity is also not clear. Hence, complexities of model structures
and their effect on prediction uncertainties may be
counterintuitive. For example, a model structure that has higher
number of parameters than another may be less complex than the other
for certain parameter ranges. This is where the number of parameters
and parameter magnitudes jointly effect model complexity, uncertainty
in process representation and consequent prediction uncertainty.
Consider the example of the linear reservoir model structure of Appendix
C. In this case one can state that a model structure with
k∈[0,0.5] is less complex than a model structure with
k∈[0,1]. However, no statement can yet be made on how the
complexity of the model structure k∈[0,0.5] fares with complexity
of model structure with k∈[0.5,1]. Now if we consider the 3
parameter model structure in Appendix C alongside the single reservoir
model structure, one can still state that a single reservoir model
structure with k∈[0,1], i.e. with
α∈[0,1]×{0}×{0}, is less complex than the 3
reservoir model structure with k∈[0,1]3, i.e. with
α∈[0,1]3. This is because the former structure is a subset
of the latter. However one cannot state anything about structure
complexities of the two model structures with
α∈[0,1]×{0}×{0} and
α∈[0,0.5]×[0,0.3]×[0,0.5] respectively unless their
complexities are computed. This is because we cannot say that one
model structure is nested within the other.
Stable system representation and top-down modelling approach
Let us now revisit the definition of a stable system representation:
a problem of system representation is stable if for two realizations
of observations that are δ-close, corresponding selected system
representations are ϵ-close such that as δ becomes
small so does ϵ. Here by ϵ-close one means that
distance between two system representations is not larger than
ϵ. Intuitively, it means that the problem of model selection
is bounded such that the selected representations do not differ
dramatically for two different realizations of data. Thus a model
selection process is stable if the models (or model structures)
selected on similar realizations of observations are similar. Now note
that the demands of stable model selection are two two-fold: the need
for a good representation of the underlying processes and the need to
have a bounded representation, i.e. no two representations are
drastically different when confronted with similar observations. Since
the complexity measure expressed in the form of
Φ(y(α))⩽cm ensures that model structures
corresponding to Φ(y(α))⩽cm are similar if two
values of cm are similar, complexity measure acts as a natural
constraint to ensure stable model selection. Thus two objectives need
to be considered, (i) maximize finite sample performance by minimizing
ξN, which ensures that a good model on a given sample is selected
and (ii) obey a constraint on model complexity for some value of
cm, say c∗, which ensures that model complexity is controlled
for. Such a model selection problem can be posed as,
minαξN(y0,x;α)s.t.Φ(y(α))⩽c∗.
The above can alternatively be written as
Ξ(αN∗)=minαξN(y0,x;α)+λ∗Φ(y(α)).
Here λ∗≥0 still has to be estimated. This is often done
on a set of observations that is independent of the observations used
to estimate models. Thus the choice of λ∗ depends on the
model structures used and the underlying hydrological system since its
estimation is based on observations. It gauges how ill-posed is the
problem of system representation and how tightly should the model
selection problem be controlled by complexity so that it can be
stabilized.
Since the measure of complexity, in the context defined here,
“stabilizes” system representation, a complexity regularized model
selection problem yields least uncertain system representation over
future unseen data. If the representation is used to predict system
behavior, such a representation also has least predictive
uncertainty. It is in this sense that complexity controls predictive
uncertainty if the problem of identifying system representation is
regularized by complexity, i.e. it controls predictive uncertainty by
“stabilizing” the problem of system representation. In other words,
complexity is a measure of predictive uncertainty since higher
complexity of system representation of underlying processes leads to
more unstable system representation, which in turn implies higher
predictive uncertainty.
An approach wherein additional process representations are added in
a stepwise manner, or a top down approach, increases complexity in
a stepwise manner . Additional complexity
with more detailed or additional process representations trades off
with the accuracy with which the processes are represented. Thus more
complexity may be acceptable when it sufficiently improves the
representation of the underlying processes. Equation ()
describes this tradeoff. The multiplier λ∗≥0 is the
minimum amount of improvement in system representation that is desired
in order for a unit increment in model complexity.
Thus λ∗ measures the tradeoff between improvement in model
performance and corresponding increase in model complexity. This
allows a formal framework to assess how much additional model
complexity is warranted, especially in a top down modelling
approach. This is because it also suggests that a more complex model
is not selected if it provides “really bad” system representation
. Thus model complexity may be increased in
a step-wise manner till model performance begins to decline.
Continuum of models and model complexity: parameter magnitude vs. dimensionality
The continuum of model structures is an important construct since it
dissolves the distinction between model architecture and model
parameters. Model structures can be defined based on constraints on
model parameters or model outcomes or both, ofcourse not excluding the
case when structures are induced by different architectures. Then
complexity of such structures, now defined as a set of abstract
parameters, can be defined as the combined (union of) extent of output
spaces of models corresponding to the parameters.
Since model complexity does not distinguish model architecture from
model parameter magnitudes (by using abstract parameters), one can
assess the relative effect of model architecture over parameter
magnitude on model complexity. Again, we can do so because the concept
of model complexity presented here depends on how a model transform
input forcings to model simulations. This depends both on the
architecture and strengths of constitutive relationships.
The effects of number of parameters (as a result of model
architecture) and magnitude of parameters (as a result of the strength
of constitutive relatioships) on model complexity can be
decomposed. This can be done by estimating model complexity of two
model structures when their parameter ranges are “equivalent” and
then fixing the model architecture and varying parameter
magnitudes. Equivalent parameter ranges ensure that two model
architectures have, for example, similar water storage capacities and
water residence times but are different in model
architectures. Meanwhile variation in parameter mangnitudes for the
same model architecture provides model structures that differ in
storage capacities and residence times. Overall model complexity can
then be thought of as the combined effect of architecture and
parameter magnitudes.
Estimation of model complexity
Section 2.1 suggests that E[‖B‖] is able to
distinguish between model (structures) in terms of stability in
process representation and can serve as a measure of model complexity
(the extent of model output space) in the context of stabilizing
system representation. We however note that it is one statistic of the
distribution Pr(‖B‖⩾ϵ). A distributed
measure of complexity may well be desired but we leave an investigaton
of this for future research. Here we demonstrate how
E[‖B‖] can be estimated in a step by step manner
see also. By doing so we also explain the
algorithms presented in Sect. 2.4.
First we note that E[‖B‖] is the expected
difference in a model's simulations for two realizations of
observations. We now translate what it means for an arbitrary
hydrological represented by y(α).
Definition 1: Let E[‖B‖]=E[‖y(x1;α)-y(x2;α)‖], where ‖y(x;α)-E[y(x;α)]‖=∑t=1,N|y(t,x;α)-E[y(x;α)]|N. Thus we assume that the
mean of absolute deviations is the metric used.
The statistic provided in Definition 1 is similar to
E[‖y(x;α)-E[y(x;α)]‖], which also measures
variation in simulations of a model parameterized by α. We will
use the latter to represent E[‖B‖].
Also, note that the expectation is obtained by taking the average of
‖y(x;α)-E[y(x;α)]‖ over a large number, say M,
of realizations of observations, i.e.
E[‖y(x;α)-E[y(x;α)]‖]=limM→∞∑k=1M‖y(xk;α)-E[y(xk;α)]‖M.
This is because a very large set of observations of size N′
can be divided into very large M subsets of observations of size N
such that N′=MN. The above thus allows us to estimate
complexity E[‖B‖] by estimating
‖y(xk;α)-E[y(xk;α)]‖ on k=1,…,M sets of
observations of size N. Also, M→∞ is indicative of
a very large M. Often, M may not be required to be large if
variation in ‖B‖ asymptotes after some finite value of M,
whereupon E[‖B‖] can be estimated with high
confidence. The estimation of model complexity as presented here thus
rests on estimating E[‖y(x;α)-E[y(x;α)]‖].
Definition 2: Let us denote
E[‖y(x;α)-E[y(x;α)]‖], that measures the
complexity of a model parameterised by α, by γ̃.
Then, by definition, the probability that
E[‖y(x;α)-E[y(x;α)]‖]⩾γ is 1 when
γ⩽γ̃ and 0 otherwise for all γ⩾0. This is because
Pr(E[‖y(x;α)-E[y(x;α)]‖]⩾γ)=Pr(γ̃⩾γ), which is equal to 1 when
γ⩽γ̃. It is equal to 0 otherwise. Thus,
Pr(E[‖y(x;α)-E[y(x;α)]‖]⩾γ)=1if γ⩽γ̃0otherwise.
We now show that E[‖y(x;α)-E[y(x;α)]‖] can be expressed as
limN′→∞∑N′|y(t,x;α)-E[y(t,x;α)]|N′, where N′=MN.
E[‖y(x;α)-E[y(x;α)]‖]=limM→∞∑k=1,.,M‖y(xk;α)-E[y(xk;α)]‖M=limM→∞∑k=1,…,M1M∑t=1,…,N|y(t,xk;α)-E[y(t,xk;α)]|N=limN′→∞∑N′|y(t,x;α)-E[y(t,x;α)]|N′.
From equation system () we note that
Pr(E[‖y(x;α)-E[y(x;α)]‖]⩾γ)=PrlimN′→∞∑N′|y(t,x;α)-E[y(t,x;α)]|N′⩾γ.
The argument of the Right Hand Side (RHS) of Eq. ()
therefore also measures complexity, i.e. E[‖B‖],
since the argument of Left Hand Side (LHS) measure it as per
definition 1. We now note as a consequence of Proposition 1.1.1 of
that
PrlimN′→∞∑N′|y(t,x;α)-E[y(t,x;α)]|N′⩾γ=limN′→∞Pr∑N′|y(t,x;α)-E[y(t,x;α)]|N′⩾γ.
Equation () states that the limit of the probability is the same as the probability of the limit. Readers are referred to the Supplement of for additional details.
Definition 3: Let PN,γ be defined as
Pr∑N|y(t,x;α)-E[y(t,x;α)]|N⩾γ.
We now estimate PN,γ, since its argument contains the
measure of complexity as per definition 2 and Eqs. ()
and (). How the measure of complexity is extracted from the
argument is now demonstrated.
For this we first invoke Markov's Lemma, which states that for any
X⩾0 and t>0 the following inequality holds,
Pr(X⩾0)⩽E[X2]t2.
By substituting X by ∑N|y(t,x;α)-E[y(t,x;α)]|N in inequality (), we
obtain the following inequality,
PN,γ=Pr∑N|y(t,x;α)-E[y(t,x;α)]|N⩾γ⩽E∑N|y(t,x;α)-E[y(t,x;α)]|2N2γ2.
The inequality () can now be rearranged to obtained an
expression for PN,γN2γ2, the motivation behind
invoking Markov's inequality. We then obtain
PN,γN2γ2⩽E∑N|y(t,x;α)-E[y(t,x;α)]|2.
Several points are in order based on inequality (). The Right Hand Side
is independent of γ. It is a sum of N(N+1)2
non-negative numbers, thus it can be bounded by some function of
N2. Since the inequality is not strict, a maximum of the Left Hand
Side, i.e. PN,γN2γ2, with respect to γ can be
equated to the Right Hand Side. Thus
maxγPN,γN2γ2 is a function only of N, while
PN,γ is both a function of γ and N. Since the Right
Hand Side is O(g(N2)) (a function f(x)=O(g(N2)) means that |f(x)|≤c|g(N2)|, where c>0),
we assume it to be a quadratic functon of form
f(h,N)=β2N2+β1N+β0 with
h={β2,β1,β0}. We therefore have
maxγPN,γN2γ2=f(h,N).
Let γN∗ represent the γ that maximizes
PN,γN2γ2, then PN,γN∗=f(h,N)N2γN∗2=1γN∗2β2+β1N+β0N2. Finally,
if we represent γ∗ as the γ that maximizes limN→∞PN,γN2γ2, then
PN,γN∗→β2γ∗2 as
N→∞.
We now show that γ∗=E[‖y(x;α)-E[y(x;α)]‖]=γ̃ maximizes
PN,γN2γ2 as N→∞. This is because of
two reasons. First, as N→∞, PN,γ→Pr(E[‖y(x;α)-E[y(x;α)]‖]⩾γ) from
Eqs. () and (). But then
Pr(E[‖y(x;α)-E[y(x;α)]‖]⩾γ) is either
1 (maximum value) or 0 (minimum value). The maximum value is achieved
when γ⩽E[‖y(x;α)-E[y(x;α)]‖] and the minimum value is
achieved when γ>E[‖y(x;α)-E[y(x;α)]‖] respectively (see Eq. ). Meanwhile γ2 is increasing in γ. Thus
γ∗ that maximizes PN,γN2γ2 as
N→∞ is the maximum possible value of γ for
which lim→∞PN,γ=1. This is
γ∗=E[‖y(x;α)-E[y(x;α)]‖]=γ̃, which is
a measure of complexity.
Thus as N becomes large we note the following based on the arguments
above: (i) PN,γN∗2 becomes 1, and (ii)
PN,γN∗2 becomes
β2γN∗2 and (iii) γN∗2
becomes the measure of complexity
E[‖y(x;α)-E[y(x;α)]‖]2. These 3 points therefore
suggest that hydrological model complexity can be estimated if we
estimate β2 since E[‖y(x;α)-E[y(x;α)]‖] becomes β2 as
N becomes large.
All we now have to do is estimate β2 to estimate complexity,
which in turn can be estimated based on Eq. (). We study
the behavior of PN,γ=Pr(∑N|y(t,x;α)-E[y(t,x;α)]|N⩾γ) for a given
model on synthetically generated data in order to estimate
β2. In particular we study the maximum of PNN2γ2 for
various values of N and when it asymptotes we obtain the measure of
model complexity, β2.
We summarize the above arguments to estimate complexity based on
expressions (), (), () and () in the
following steps.
Let γN∗ be the one that maximizes
PN,γN2γ2. Then from equality ()
PN,γ∗=f(h,N)N2γN∗2.
Let limN→∞γN∗=γ∗. From inequality (),
limN→∞PN,γN∗=PrlimN→∞∑N|y(t,x;α)-E[y(t,x;α)]|N⩾γ∗.
From expression (),
PrE[‖y(x;α)-E[y(x;α)]‖]⩾γ∗=PrlimN→∞∑N|y(t,x;α)-E[y(t,x;α)]|N⩾γ∗.
From expression () we have
Pr(E[‖y(x;α)-E[y(x;α)]‖]⩾γ) is
either 0 or 1 for different values of γ. Since, from steps
(1)–(3), γ∗ maximizes
limN→∞PN,γN2γ2, we require
PrE[‖y(x;α)-E[y(x;α)]‖]⩾γ∗=1 and γ∗=γ̃=E[‖y(x;α)-E[y(x;α)]‖].
From steps (4) to (2), we obtain
limN→∞PN,γ̃=1.
From steps (5), (4), (2) and
(1) we obtain
limN→∞β2N2+β1N+β0N2γ̃2=β2γ̃2=1.
From steps (6) and (4), we obtain
E[‖y(x;α)-E[y(x;α)]‖]=γ̃=β2.
Thus complexity can be estimated by β2. The parameter set h,
that includes β2, are estimated by regressing a quadratic
function to maxγPN,γN2γ2 that is numerically
estimated for various values of N. Algorithms 1 and 2 peform this
task. Using Eq. (4) we further define a measure of
complexity, F(h,N)=f(h,N)N2, that is dependent on N
such that limN→∞F(h,N)=β2. We call β2
asymptotic complexity in this context.
Algorithms and data
The computation of model complexity requires a synthetically generated
input forcing data set because PN,γ needs to be estimated in
order to estimate maxγPN,γN2γ2 for each
N. This inturn requires the estimation of E[y(t,x;α)], which can be estimated based on
synthetically generated input forcing data.
We here note that a vector E[y(x;α)]={E[y(t,x;α)]}t=1,…,N is desired that
preserves the autocorrelation that a model simulation may bring. It
also represents the expectation of a N-vector in the N-dimensional
model output space. Here x is a N-dimensional input
forcing, i.e. x=(x1,x2,…,xt,…,xN). For
notational simplicity we have assumed xt is a one-dimensional
varaible. Also, note that the intention is to use it to estimate
Pr(‖y(x;α)-E[y(x;α)]‖≥γ). Thus if have M
realizations of input forcings,
i.e. {x1,x2,…,xk,…,xM},
we estimate the expectation of N-dimensional model simulations as
E[y(x;α)]={∑k=1,…,My(1,xk;α)M,∑k=1,…,My(2,xk;α)M,…,∑k=1,…,My(t,xk;α)M,…,∑k=1,…,My(N,xk;α)M}.
We now present an algorithm that computes the expectation operator on
the synthetically generated data set . The input
forcing basin datasets for this algorithm are obtained from the MOPEX
data sets . 5 basins from different hydroclimatic
regions are used. By doing so we test whether the ordering in terms of
its complexity of various model structure set-ups changes with
different data sets. Insensitivity of the ordering of structure
complexities to the data sets used for input forcings is crucial for
any robust statement about the role of parameter magnitudes in
determining model complexity. Table provides
characteristics such as area, mean annual precipitation and
evaporation and hydrologic ratios such as runoff ratio and dryness
index, for the basins used in this study. Figure displays
them.
The algorithm is a resampler that block bootstraps time series from
a given sample of data .
discuss that the weather resampler bootstraps blocks of wet/dry spell
pairs where each block contains one wet/dry spell pair. The algorithm
can be improved by increasing the number of contiguous wet/dry samples
within each block. We use basin input forcing data set (of
precipitation and potential evapotranspiration) and generate multiple
realizations for the complexity, one for each sampled parameter. We
also partially account for the sensitivity of complexity computation
by permuting data at monthly scale in such a way that intra-annual
autocorrelation in forcing time series is randomized. Sensitivity of
complexity computation is also tested against multiple basins and
different wet-dry spell identification by choosing basins from
different regions of the US (Fig. ).
Algorithm 1:
Extract daily precipitation and potential evapotranspiration
data for a basin.
Identify a block of contiguous wet (a set of contiguous days
with positive precipitation) and dry (a set of contiguous days with
zero precipitation) spell pairs for each month: determine the amount
and length of spell pairs and attach an identifier to each spell.
Construct a one month sample for each month: conditioned on
a selected month, randomly sample (with replacement) blocks of spell
pairs, along with potential evaporation values for the same days,
across different years for the same month, appending these blocks
till the total length of the sequence exceeds 30 days.
Go to step 3 for other months until all 12 months of a year have
been sampled.
Permute the months (if correlation between months is to be
removed), while maintaining the order of sequences within each
month, to create one year sample.
Repeat steps 4 and 5 to create a realization of input forcings
at daily time steps with N‾ datapoints.
Go to step 6 until M realizations of N‾ datapoints are
created.
The algorithm resamples forcing data from an observed dataset of
a basin such that auto (and cross) correlation of the variables are
preserved at certain scale. For each month, for example January,
wet-dry spell pairs are identified and a resample for the month is
generated by bootstrapping such pairs with replacement (i.e. the pairs
are put back in the month and can be resampled again). A resample for
a month is created once the total length of days resampled in such
a manner is at least 30. Then if the auto-correlation is to be
preserved at certain scale, for example at 3 month scale (called
“Medium 4”), then the ordering of 3 month blocks of monthly
(re-)samples is permuted. The “4” in “Medium 4” therefore
represents the number of blocks in a year that need to be
permuted. That is, the ordering of the set of 3-tuples
{JFM,AMJ,JAS,OND} is permuted, where each letter stands for
the beginning letter of a resampled month (“JFM” for
January-February-March, “AMJ” for April-May-June, and so on). Thus
a resample of forcing data for a year that preserves correlation at
3 month scale can be {AMJ, JFM, OND, JAS}. Repeating the
process for multiple years thus re-samples (or stochastically
generates) forcing data for multiple years and correlation is
preserved at certain scale. The preservation of the entire seasonal
cycle (“Complete”), of the monthly correlation at 6 month scale
(“Medium2”), of the monthly auto-correlation at 3 month scale
(“Medium 4”) and of no month to month autocorrelation (“None”) is
currently allowed.
Using the weather resampler, M=2000 sequences of N‾=5000
datapoints for daily precipitation and potential evaporation are
obtained. For each realization, input forcings of smaller sample sizes
N=200:50:N‾ are obtained by sampling its first N data
points. Since SIXPAR model structure does not explicitly incorporate
evaporation (see Supplement), the precipitation data used
for SIXPAR is assumed to be equal to a maximum of the precipitation
minus potential evaporation and zero.
Once multiple realizations of input forcing data have been generated
(resampled), Algorithm 2 computes the complexity of models for
a sampled parameter set as outlined in the previous
section. It uses the M realizations of input forcings to first
estimate expected value of model simulations of size N‾,
i.e. E[y(x;α)] and then estimate
probabilities of exceedences for γ=0:γ‾, where
PN,γ=Pr(∑N|y(t,x;α)-E[y(t,x;α)]|N⩾γ). These are
the steps involved in step 1 of Algorithm 2.
Algorithm 2:
For each parameter set of a model structure set up,
estimate PN,γ, for a given value of N and γ using
M samples of data set of size N, obtained from Algorithm 1.
Estimate the maximum f̃(N) of
PN,γN2γ2 with respect to γ for each N. Let
the maximizing γ be γmaxN.
Repeat steps 1 and 2 for N=200:50:N‾.
Determine the set of coefficients h={β2,β1,β0} of f(h,N)=β2N2+β1N+β0
that fits data points {f̃(N),N=200:50:N‾}. The set
of coefficients h defines the model complexity.
Repeat step 1–4 to estimate complexity
for different parameter sets of a model structure.
In total 500 parameter sets are sampled from each range presented in
Tables and .
Model structures and parameter ranges
SAC-SMA and SIXPAR model structures
The two model structures that are used are SAC-SMA (Sacramention Soil
Moisture Accounting model) and SIXPAR (Six Parameter model). SAC-SMA
is a complex hydrological model structure with a two layer reservoir
architecture and a nonlinear percolation conceptualization. The two
upper zone reservoirs represent a free water zone and a tension water
zone, wherein the former controls the percolation to the lower zones
while the tension water zone mainly controls the evaporation and feeds
the free water zone. The percolation is a nonlinear complex function
of demand from the lower reservoirs and available supply of water from
the upper zone reservoirs. Both the upper and lower zones also control
the outflows. The SIXPAR model structure, which is a conceptual
simplification of the SAC-SMA model with one upper and lower zone,
evaporation and the concept of tension water zones but retains the
complex conceptualization of percolation. These models are run at
daily time steps using input forcing from selected basins (in
Table ). Additional details on the models can be found
elsewhere .The code used and further
explanation for SIXPAR is provided in the Supplement.
Parameter ranges as model structures
Table provides the “reference” parameter ranges for
SAC-SMA. Table provides various parameter ranges of SIXPAR,
including so called “reference” ranges and “equivalent”
ranges. The model structures and various parameter ranges that govern
parameter magnitudes of models that are sampled from these ranges
allow us to study the (decomposed) effect of structure architecture
and parameter magnitudes on computed complexity. We note that these
two effects are mixed when arbitrary (here called “reference”)
parameter ranges of SAC-SMA and SIXPAR are considered. However the
effect of structure architecture (and the role of the number of
parameters) on complexity emerges when we control the ranges of the
parameters. This is when we have “equivalent” parameter ranges for
the two model structures.
The parameter range of SIXPAR model structure is made “equivalent”
to the “reference” parameter range of SAC-SMA model structure by
ensuring that (i) the upper bounds on the reservoir capacities of the
two layers of SIXPAR is equal to the sum of upper bounds on the
reservoir capacities of the corresponding layers of SAC-SMA model
structure and (ii) the corresponding lower and upper bounds of the
recession parameter ranges of SIXPAR model structure are the geometric
means of the corresponding lower and upper bounds of the SAC-SMA
recession parameters. In terms of abstract parameters, this would then
mean that the set of αs (abstract parameters) corresponding to
the SIXPAR model structure are a subset of αs corresponding to
the SAC-SMA model structure. Hence SIXPAR model structure would be
nested within SAC-SMA structure in “abstract” sense.
In order to study the effect of parameter magnitudes on model
complexity, we restrict our attention to the “reference” ranges of
SIXPAR and contrain the paramater ranges in three ways. The three
parameter ranges are called (i) “High recession”, (ii) “Low
recession”, and (iii) “High storage/Low recession”. These
correspond to the “reference” parameter ranges for SIXPAR except
that (i) corresponds to the case where the lower bounds of the
recession ranges for the two layers are higher than the means of the
corresponding “reference” ranges, (ii) corresponds to the case where
the upper bounds of the recession ranges are lower than the means of
the corresponding “reference” ranges and (iii) corresponds to the
case where the means of storage capacities are larger than the means
of the corresponding “reference” ranges and where the recession
ranges are the same as in (ii). The three parameters ranges define
three different model structures. The “High recession” and “Low
recession” model structures are nested within the “reference” model
structure of SIXPAR, while the “Low recession” SIXPAR model
structure is nested within “High storage/Low recession” model
structure. Finally, both the “Low recession” and “High storage/Low
recession” model structures are nested wthin the “equivalent”
SIXPAR model structure.
The complexities of SIXPAR model structures for “reference”,
“equivalent”, and (i)–(iii) ranges are computed on the selected
hydrological data sets of MOPEX basins (see Table ) and
compared with the SAC-SMA model structure complexities computed on the
same basins for its “reference” parameter range. The complexities of
the model structures corresponding to the specified ranges are
computed using Algorithm 2. It uses resampled basin scale potential
evaporation and precipitation data using Algorithm 1.
Results
The Algorithm 2 provides complexity computations for each of the two
structures for the parameter sets sampled from ranges defined in
Tables and . The algorithm uses input forcing
realizations resampled by Algorithm 1 from input forcings of the
selected MOPEX basins. The parameters are sampled using Latin
Hypercube Sampling. As a result, Algorithm 2 provides a collection of
{β1,β2,β3} corresponding to parameter sets that are
sampled from a specified range for each model structure. Note that the
algorithm computes one set of {β1,β2,β3}
corresponding to one sampled parameter set. A corresponding
distribution of F(h,N)=f(h,N)N2=β2+β1N+β0N2 as a function of N
can therefore be obtained. Figure demonstrates the
variation of 50th percentile values of F(h,N) (over the 500
parameters sampled from “equivalent” parameter ranges) with N for
the SIXPAR model structure using data from basin “NC”.
The different curves correspond to different month permutations (step
5 of Algorithm 1) of the resampled input forcing data set. We note
that the estimation of the curve is insensitive to the type of
permutation in step 5 of Algorithm 1. We further note that F(h,N)
declines with increasing N and reaches an asymptote for large
N. Since F(h,N) is a function of complexity, represented by
“h”, and N, the value of F(h,N) at large N (when F(h,N)
asymptotes and becomes insensitive to N) reveals the measure of
complexity (β2). The asymptotic value of F(h,N) is used to
compare the complexity of different model structure set-ups. We here
note that F(h,N) already asymptotes around 2500 data points. Since
Algorithm 1 resamples daily datasets, this means that N=2500 is as
large as N→∞ with regards to computing asymptotic
complexity. It also means that when N=2500 datapoints are enough to
obtain a representation of the underlying processes that is not
influenced by sampling uncertainty. In other words, this sample size
is large enough to accurately reveal how unstable is the
representation of underlying processes by SIXPAR model structure that
is “equivalent” to SAC-SMA.
Figure demonstrates that the asymptotic complexity for
parameter ranges of SAC-SMA sampled from its “reference” ranges
(Table ) appears to be less complex than the asymptotic
complexity of SIXPAR when sampled from its “reference” ranges
(Table ). This may appear counterintuitive since SIXPAR
model structure is a conceptual simplification of SAC-SMA. However,
similar conclusions have been drawn elsewhere for regression problems,
where it has been shown that model complexity is both a function of
magnitude and dimensionality of model parameters. For example,
and find that the complexity of ANNs
(Artificial Neural Networks) and SVMs (Support Vector Machines) are
not only dependent on the dimensionality of the regressors but also crucially
depend on the magnitude of the parameters. Ridge regression
also regularizes the linear regression problem by penalizing the
magnitude of the parameters .
Based on our construct of a continuum of model structures, which does
not distinguish between model structures and parameter magnitudes, it
may be possible that effect of parameter magnitudes on model
complexity may compensate for the effect of structure architecture. We
know that higher parameter dimensionality as a result of more
complicated structure architecture leads to higher complexity. Thus
magnitudes of parameters sampled from “reference” parameter range
for SIXPAR compared to “reference” parameter range for SAC-SMA must
have some compensating effect to reduce model complexity to such an
extent, inspite of higher parameter dimensionality. We now look for
those possible effect of parameter magnitudes on model complexity.
Figure further studies the effect of sampling SIXPAR
parameters from various ranges in Table on its
complexity. It suggests that complexity is less sensitive to recession
parameters at lower magnitudes than it is at higher magnitudes since
the median complexity for “low recession” range is closer to median
complexity for “reference” recession range than the median
complexity for “high recession” range. Further, the model complexity
increases when the magnitudes of the recession parameters are
increased. Finally, an increase in reservoir storage capacities leads
to a reduction in model complexity. This can be seen from the median
complexities of box plots corresponding to “Low recession” and
“HS/LR” (i.e. high storage with low recession). Finally
“Equivalent” SIXPAR model structure has the lowest complexity. We
note that parameters sampled for “Equivalent” SIXPAR tends to have
high storage and low recession when compared to the parameters sampled
for the “Reference” SIXPAR. It is this effect of sampling high
storage capacities and low recession parameters that brings down the
complexity of SIXPAR in its “Equivalent” version.
The figure therefore suggests that high values of recession
coefficients, i.e. small residence times, and low storage capacities
lead to high complexity. This is intuitive, models with smaller
residence time and lower storage capacities are more sensitive to
perturbation in input forcing and hence have higher possibility of
leading us to an unstable systen representation.
Over all, this demonstrates that the magnitude of parameters appear to
have an effect on model complexity. Figure shows
a comparative variation of computed complexity with sample size N
for SAC-SMA and SIXPAR. Figure a shows the comparison
between the two models when parameters are sampled from “reference”
parameter ranges and Fig. b compares the two model
structures when the parameters are sampled from “equivalent”
parameter ranges. The y axis, PN=PN,γN∗, where
γN∗ is the one that maximizes
PN,γN2γ2 in Eq. (). Then from equality (), PN,γ∗=f(h,N)N2γ∗2 is
an increasing function of model complexity as defined by the 3-tuple
{β2,β1,β0}.
Both the figures demonstrate that the differences in complexities of
the two model structures are more evident for small sample sizes. In
effect, this figure demonstrates the decomposed effect of parameter
dimensionality and parameter magnitudes on model
complexity. Figure a suggests that SIXPAR model structure is
more complex, due to sampled parameters having higher recession values
and lower reservoir storage capacities for all sample sizes
N. Meanwhile Fig. b shows SAC-SMA is more complex for all
sample sizes N when the sampled parameter sets of SIXPAR are from
ranges that are “equivalent” to SAC-SMA parameter ranges. The change
in complexity and ordering from Fig. b to Fig. a is due to the
effect of parameter magnitudes. The comparison suggests that parameter
magnitudes also play a role in model complexity and that parameter
dimensionality is an incomplete measure of complexity and hence
prediction uncertainty. Figure presents the case again for
the asymptotic complexities β2 of “reference” SAC-SMA,
“reference” SIXPAR and “equivalent” SIXPAR.
The complexities are computed using input forcings from historical
dataset of “NC” basin using Algorithm 1. Is the conclusion that
parameter magnitudes may have an effect on model complexity sensitive
to the basin that is selected for resampling of input forcings?
Figure plots the asymptotic complexities for the same
ranges of SIXPAR model structure on input forcing resampled from CA,
IA, GA and ME MOPEX basins that are from different hydro-climatic
regions of continental US (Table ). We observe
a similar pattern in asymptotic complexities with parameter ranges and
hence with parameter magnitudes.
Discussion
The evidence from Figs. and suggest that (i)
model complexity is increasing in parameter dimensionality when
parameter magnitudes of two model structures are “equivalent” and
(ii) model complexity depends on the magnitudes of model parameters
irrespective (to a certain extent) of model parameter
dimensionality. Since model complexity is linked to instability in
process representation and hence predictive uncertainty, it then
follows that predictive uncertainty of a model structure need not be
lower if it has lower number of parameters. A SIXPAR model in one
application with lower number of parameters but with “high
recession” parameter values may have higher predictive uncertainty
than an application of SAC-SMA model that is parameterized from
“reference” parameter ranges (given in Table ).
An important implication for complexity controlled model selection is
then that parameter range specification should be application
dependent. The modelling of a fast catchment with shallow unsaturated
or saturated zones requires high recession and low reservoir capacity
ranges. Our results (though for SIXPAR but may be extended to other
models as well) demonstrate that complexity and hence predictive
uncertainty is more sensitive to these parameters ranges since model
complexity is high for such recession and reservoir capacity
values. Model selection should consider parameter magnitudes in
addition to parametric dimensionality when modelling such
catchments. On the other hand, model parameter dimensionality may be
a sufficient criterion to select a model with low prediction uncertainty
in modelling slower basins.
Figure demonstrated that for a given specification of
parameter range, the magnitudes of asymptotic complexities are
different for different basins. This indicates the influence of basin
specific wetness conditions since higher magnitude of input forcings
leads to a larger model output space and hence larger magnitudes of
estimated complexities. The hydrologic ratios of these basins
presented in Table may be indicative of this. The CA basin
is extremely dry with low runoff ratio while IA basin is moderately
dry with a moderate ratio of annual evaporation to annual potential
evaporation EEP. The asymptotic complexities of
CA are lower than those of IA for corresponding SIXPAR variants
(Fig. ). Yet their asymptotic complexities are a lot lower
than those of the remaining 3 basins. Incidently these 3 basins have
dryness indices EPP<1. They also have higher
EEP ratios. Thus the last 3 basins are
comparatively wetter. Had we normalized the input forcings (by
substracting the mean and dividing by SD), the correlation structure
in input forcings on model structure complexities would have been
revealed. A detailed analysis of such effect on computing model
complexity and of its own interpretation of complexity is left for
future research.
Farmer et al. (2003) noted that more complex model structures are
needed for dry catchments. This is because the runoff response of
these catchments is more sensitive to small perturbations in input
forcing than wetter catchments. Dry catchments experience more
disruption of connectivity than wet catchments. The notion of
complexity as proposed in this paper is also defined as a measure of
sensitivity of modelled responses to perturbations in input
forcings. The paper formally builds the notion of complexity and
measures it. The context of model complexity is how stable or unstable
the model is to input pertubations. This then leads to instability of
system representation and prediction uncertainty. We find that
sensitivity of model outcomes to input perturbations does not just
depend on how complicated the architecture of model structure is (for
example the structure variants of Farmer et al. (2003), Bai et al. (2009)
and others) but also on the magnitude of the parameters that
define a model structure. Higher recession coefficients and smaller
storage capacities result in response variability at finer/shorter
time scales (assuming the input forcing remains the same).
This is not to say that more complex model structures are always
unsuitable. A top-down modelling approach proposes to increase model
structure complexity in a nested fashion, starting with model
structures of lower complexity and increasing the complexity till
system representation degrades. However the formalism presented here
takes one step further. It allows the possibility of not always
rejecting a more complex model, even if it has higher complexity, if
the ratio of reduced performance with increased complexity is less
than a certain threshold (λ∗ in Eq. ). This
allows the possibility that a more complex model with poorer
performance on one realization of observations may perform better on
another realization. This “acceptability” threshold is derived from
the information embedded within the observations of the underlying
processes.
We note that the notion of complexity and stable process
representation is not limited to the use of one performance
metric. Any performance measure, such as based on flow duration curve
or Nash–Shutcliffe efficiency, can be used as long as it is a valid
metric. Thus the results obtained are general and testable on a wide
variety of empirical evidence on appropriate model complexity and
process representation that has been documented so far. Further, the
method to estimate complexity is independent of the type of
hydrological model used. Hence it is applicable for conceptual models,
physically based models, empirical models as well as data driven
models. For example estimated the complexities of
flexible conceptual rainfall–runoff models and demonstrated the
applicability of the theory for a class of linear regression models.
Finally, we here highlight one limitation of the approach. The notion
of complexity control on prediction uncertainty is based on a triangle
inequality, wherein prediction uncertainty is bounded by the measure
of complexity. It thus rests on the idea that controlling the measure
of complexity only avoids the possibility that variation in model
performance over two different realizations of data is not large. In
context of top-down modeling approach, if we gradually ease the
control on complexity, i.e. make models more complex, variation in
model performance gradually increases as well. However, if this
increase in model complexity is guided by better system
representation, the possible increase in variation of model
performance may be compensated by better average model performance to
a certain extent.