A methodology for quantifying the accuracy of snow depth measurement are demonstrated in this study by using the equation of error propagation for the same type sensors and by compariong autimatic measurement with manual observation. Snow depth was measured at the Centre for Atmospheric Research Experiments (CARE) site of the Environment Canada (EC) during the 2013–2014 winter experiment. The snow depth measurement system at the CARE site was comprised of three bases. Three ultrasonic and one laser snow depth sensors and twelve snow stakes were placed on each base. Data from snow depth sensors are quality-controlled by range check and step test to eliminate erroneous data such as outliers and discontinuities.
In comparison with manual observations, bias errors were calculated to show the spatial distribution of snow depth by considering snow depth measured from four snow stakes located on the easternmost side of the site as reference. The bias error of snow stakes on the west side of the site was largest. The uncertainty of all pairs of stakes and the average uncertainty for each base were 1.81 and 1.52 cm, respectively. The bias error and normalized bias removed root mean square error (NBRRMSE) for each snow depth sensor were calculated to quantify the systematic error and random error in comparison of snow depth sensors with manual observations that share the same snow depth target. The snow depth sensors on base 12A (11A) measured snow depth larger (less) than manual observation up to 10.8 cm (5.21 cm), and the NBRRMSEs ranged from 5.10 to 16.5 %. Finally, the instrumental uncertainties of each snow depth sensor were calculated by comparing three sensors of the same type installed at the different bases. The instrumental uncertainties ranged from 0.62 to 3.08 cm.
Solid precipitation has a significant effect on human life, as it can lead to issues such as flight delays and slippery roads, harm to crops, and building collapses (Rasmussen et al., 2003). The impact of these issues can be mitigated with more accurate weather forecasts and representative engineering standards, the development of which require accurate solid precipitation and snow on the ground measurements. In addition, precipitation and snow on the ground measurements are important for various meteorological and hydrological applications, such as climate change, remote sensing calibration, and water supply forecasts (e.g. Michelson, 2004; Rasmussen et al., 2012; Theriault et al., 2012). However, accurate measurement of solid precipitation is difficult due to wind-induced loss and the spatial and temporal variability of snow shape, size, and density (Roebber et al., 2003; Nitu, 2013). The measurement of snow on the ground is also prone to numerous errors due to snow redistribution, blowing snow, and compaction (Ryal et al., 2008). Thus, there is a requirement for the accuracy of solid precipitation and snow on the ground measurements to be evaluated systematically. This study focuses on the measurement of snow depth, which is the total vertical height of snow on the ground within the observation period.
Graduated rulers or snow stakes are used to measure snow depth
by trained human observers; these manual measurements are
considered to be the reference for snow depth
measurements. However, manual snow depth measurements have
significant limitations such as consistency, continuity,
spatial and temporal resolution, and time and manpower
consumption (Ryan and Doesken, 2007). Meanwhile, snow depth
sensors based on various operating principles have been
developed as a result of the automation of meteorological
observation systems (Nitu et al., 2012). Automatic snow depth
sensors can help to overcome the limitations of manual snow
depth measurements, but they also have limitations (Ryal
et al., 2008; Fischer, 2008; Haij, 2011). Ultrasonic snow
depth sensors are currently the most frequently used, due to
their ease of use and low power consumption. Two aspects of
the operation of these sensors need to be properly managed:
first, the temperature dependency of ultrasonic pulses; and
second, the risk of interference within the field of view,
since ultrasonic pulses have the shape of a cone, for example
a cone of 22
The required resolution and uncertainty for snow on the ground
measurements are suggested by the World Meteorological
Organization (WMO). The snow depth should be reported with
0.1
In 2010, the WMO initiated the Solid Precipitation InterComparison Experiment (SPICE). The main objective of SPICE is to define the (automatic) operational reference system and to evaluate the performance of various automatic snow depth and snowfall measurement instruments (WMO/CIMO, 2011, 2012a, 2012b, 2013). The Centre for Atmospheric Research Experiments (CARE) in Egbert, Ontario, Canada, is one of the lead SPICE sites and hosts one of the SPICE experiments for snow on ground. On this site, several models of snow depth sensors (ultrasonic, laser) are tested against manual reference measurements, as part of an Environment Canada (EC) study project, as well as a contribution to SPICE.
The main purpose of this study is to document methodology for analyzing the uncertainty of snow depth measurements from automatic snow depth sensors. This procedure is demonstrated using data collected at the CARE site during the 2013–2014 winter season. The quality control (QC) procedures applied to the data sets used are similar to those established for the first level of QC in SPICE, and constitute the baseline for any advanced QC that may be required. SPICE will investigate and report on more advanced QC techniques, tailored to specific sensors. In Sect. 2, the methodology for the quantification of uncertainty in snow depth measurements is proposed. The procedures for snow depth measurements are described in Sect. 3. The manual and automatic snow depth data are shown in Sect. 4, along with suggested QC procedures for automatic snow depth measurements. The uncertainties in manual and automatic snow depth measurements are detailed in Sect. 5. Section 6 summarizes the results and provides conclusions.
The uncertainty analysis is performed using two approaches: statistical measures and the propagation of error. The standard quantities for measuring the accuracy are defined under statistical measures. In the propagation of error, the uncertainty of individual instruments is calculated from the difference of two measurements of the same type. These approaches are explained in further detail below.
Standard statistical measures are used to quantify the
uncertainty of snow depth measurements. The Bias Error (BE),
Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and
Bias Removed Root Mean Square Error (BRRMSE) are defined as
follows:
In comparisons between manual observations, the BE is
calculated to investigate the spatial distribution of snow
depth relative to the average snow depth measured by snow
stakes at each target. The average snow depth of four stakes
on the same snow depth target is considered as
The error propagation equation is used to quantify the
uncertainty of manual snow depth measurements and automatic
snow depth sensors. When
The terms
The CARE site (44
The layout of the snow depth measurement system at the CARE
site during the 2013–2014 winter season is shown in
Fig. 1. This system is configured around three bases, each
with four snow depth sensors (three ultrasonic and one
laser-based) pointing at three snow depth targets developed by
EC. Four snow stakes, used for the manual measurements, are
posted at the corners of each snow depth target, in the shape
of a square. A total of 36 manual observations are
performed. Snow depth targets are composed of grey plastic
decking (Fig. 2a). The size of each target is
The manual observations taken using wooden snow stakes
(Fig. 2b) are considered to be the reference observations for
snow depth. Gradations with 0.5
Three automatic snow depth sensors are installed on each base
as in Fig. 2c. The pole on each base has the three arms. One
ultrasonic snow depth sensor is installed on each arm. The
laser snow depth sensors, which are installed at the top of
the pole, point to the middle snow depth target. Care has been
taken during the installation of the laser snow depth sensor
to avoid pointing at the holes of the snow depth target. The
output of all snow depth sensors under test is collected every
30
In this experiment the following snow depth sensors are being
tested, the FELIX SL300 (hereafter, FEL), SOMMER USH-8
(hereafter, SOM), CAMPBELL SR-50A (hereafter, SR50A), and
JENOPTIK SHM 30 (hereafter, JEN). One each of these sensors is
installed on each base. The FEL, SOM, and SR50A are ultrasonic
snow depth sensors, and the JEN is a laser snow depth
sensor. The general characteristics of the snow depth sensors
are outlined in Table 1 (Sommer Mess-Systemtechnik, 2008;
Jenoptik, 2009; Campbell Scientific Corp., 2011; Felix
Technology Inc., 2014). The measurement range of FEL, SOM,
SR50A, and JEN are 0.43–6.10, 0.00–8.00, 0.50–10.0, and
0.00–15.0
Snow depth data from the manual observations and automatic snow depth sensors (FEL, SOM, SR50A, and JEN) are used to demonstrate the proposed methodology for the analysis of uncertainty in snow depth measurements. The data were collected at the CARE site from 20 December 2013 to 26 March 2014.
Figure 3 shows the time series of snow depth at each snow
stake from the manual observations at each base (Fig. 3a–c),
and the average snow depths from the four snow stakes at each
target (Fig. 3d). The maximum snow depths recorded at bases
12A, 20, and 11A during the observation period were 40.0,
42.5, and 44.0
The data collection for this experiment has been configured
such that the JEN, FEL, and SR50A data are reported with one
millimeter resolution, while the SOM data is reported with one
centimeter resolution. The time series of raw (unfiltered)
snow depth data from each sensor show apparent erroneous data,
such as outliers and discontinuities (Fig. 5). Some of the
data from FEL, SOM, and JEN fall outside the reasonable range
for a given site and observation period (Fig. 5a, b,
and d). Snow depth sensor data over 1000
The following QC procedures are applied to snow depth sensor
data in this study: (1) range check, (2) step test; and (3)
conversion of data from 30 s temporal resolution into
1
In the first QC stage, a range check is applied to remove
outliers. In this study, values of 0 and 60
The snow depth data after applying the QC procedure are shown
in Fig. 6. It is evident that the outliers and discontinuities
observed in Fig. 5 have been removed. Several spikes still
remain in Fig. 6a, underscoring the challenge of selecting
general QC thresholds for data from different sensors. The
data collected from FEL (base 20) after 21 February 2014 are
removed by the QC procedure, since they exceed the range check
threshold. From Fig. 6, it is evident that the maximum snow
depth measured by automatic sensors during the winter season
was 58.9
It is acknowledged that the experimental configuration is such that any two identical sensors do not measure the snow deposition on the same target. Owing to the variability in snow distribution, there are differences in the depth of snow accumulated on different targets, which leads to differences in the data reported by sensors. These errors are likely well represented by the variation of manual measurements at each target. In addition, Fig. 7 shows that the snow depth sensor data show almost all zero snow depth prior to the snow accumulation (ground clear), except for FEL (11A). Thus, the uncertainty determined in this study is most likely related to the sensor response to snow signal in various conditions, and not due to sensor malfunctions.
The BEs and uncertainties of manual snow depth measurements
are calculated to analyze the spatial distribution of snow
depth and uncertainty of manual snow depth measurements
(Fig. 8 and Table 2). For calculation of BEs, the average snow
depth of snow stakes 1 to 4 on base 12A is considered to be
the reference for the purpose of this analysis. Figure 8a and
Table 2 show that the BEs of base 12A (0.00, 0.86, and
3.20
The snow depth measured by automatic snow depth sensors is
compared with the average of the manual observations on the
same snow depth target in Fig. 9. The automatic snow depth
observations at the halfway point of the manual observations
(that is, 10
Figure 10 and Table 3 show the BEs and NBRRMSEs of each snow
depth sensor. The automatic snow depth sensors on base 12A
(11A) tend to measure snow depth as much as
5.02–10.8
In general, the NBE (BE) ranges from
The snow depth measured by two snow depth sensors of the same type on different bases is compared to quantify the instrumental uncertainty of individual snow depth sensors (Fig. 11). The data quality during a snow event could be poor for ultrasonic sensors, since it is a known limitation of these sensors that the sound waves are returned by the falling snow before reaching the target. This may have an impact on the calculated uncertainty. A significant bias is shown in the comparison, and should be eliminated to quantify instrumental uncertainty. In addition, bimodal distributions are observed for a few sensor pairs (SOM on base 20 vs. 12A and 11A vs. 12A; SR50A on base 20 vs. 12A, JEN on base 20 vs. 12A). The physical reasons are not known for this peculiar characteristic.
The BEs and instrumental uncertainties of each snow depth sensor are shown in Fig. 12. The snow depth sensors on base 12A are considered to be the reference for the calculation of BE (diamonds in Fig. 12a), similar to the approach used for the assessment of manual observations. The circles in Fig. 12a represent the spatial distribution of snow depth measured by the automatic sensors. To calculate these values, the BEs in Figs. 8a and 10a are added and the snow depths from sensors on base 12A are used as the reference. The BEs of snow depth sensors on base 20 and 11A are negative. This could result from the spatial distribution of snow depth, and/or the systematic bias of snow depth sensors. The snow depths measured at bases 12A and 20 are lower than that measured at base 11A, based on the result from comparison of manual observations (Fig. 8a). Meanwhile, the snow depth sensors on bases 12A and 20 (11A) overestimate (underestimate) snow depth relative to the manual observations (Fig. 10a). Thus, the BEs of bases 20 and 11A are negative, as snow depths measured by the snow depth sensors on base 12A are larger than those measured by snow depth sensors on bases 20 and 11A.
When comparing each base, the instrumental uncertainties of
each snow depth sensor on base 12A (2.08–3.08
This paper introduced a methodology for assessing the global uncertainty of instruments measuring snow depth using statistical measures and error propagation analysis. The standard statistics such as BE (NBE), MAE (NMAE), RMSE (NRMSE), and BRRMSE (NBRRMSE) are calculated in statistical measures. The BEs of manual snow depth measurements indicate the spatial distribution of snow depth on the CARE site. In addition, those computed in the comparison between manual and automatic snow depth measurements provide information about the systematic bias of each snow depth sensor. For error propagation analysis, the matrix is created from measurement pairs among three sensors of same type. The uncertainties of manual and automatic snow depth measurements are calculated from the constructed matrix. This methodology is demonstrated with its application on the snow depth data collected at the CARE site from 20 December 2013 to 26 March 2014, to both manual and automatic sensor measurements.
The manual snow depth measurements are performed once a day,
while the automatic measurements are output every 30 s. Over
the course of the study period, the reported snow depth varied
by location and method of measurement, likely being influenced
by the configuration of the experimental site. The values of
maximum snow depth recorded by manual observations are lower
than those reported by automatic snow depth sensors. The snow
depth is larger on base 11A (west) than 12A (east) at the CARE
site when we select the average snow depth of stakes 1–4 on
base 12A as the reference. The uncertainties of manual
observations for all pairs, on each base, and for each snow
target were 1.81, 1.53, and 1.33
As illustrated by the results of this study, the uncertainty of measurements can vary among similar instruments collocated on the same site. The variability of results obtained through this study may indicate that other additional factors could influence the uncertainty of measurement of any sensor. The identification and treatment of the influence of other factors could further improve the uncertainty of measurement, and should be further investigated as part of SPICE. Two categories of factors are recognized to influence the uncertainty of measurements that would require further investigation. The first is related to the site and sensor configuration, while the second is specific to a sensors ability to detect and measure snow on the ground.
On the topic of site and sensor configuration, at the CARE site, although the similar instruments are located in close proximity, they do not measure snowfall and snow on the ground on the same sample area. The differences in the uncertainty of measurements for similar sensors would include the differences in the accumulation due to topography, wind influence, etc. Additionally, the accuracy of measurement of the initial distance between the sensing element and the ground is critical, as is the ability to maintain this distance throughout operations. This is best illustrated by the uncertainty of measurement calculated for periods of time with no snow on the ground, before and after the snow season. The stability of the configuration would influence the ability to derive accurate snow depth measurements.
The second category of factors is related to how the sensor data is sampled and treated. All snow depth sensors tested at the CARE site have their own internal data processing algorithms and report data based on their own internal processing of multiple raw samples. These sensors also output signal quality indicators, reflective of the interpretation of the returned raw signals, as processed by the internal sensor algorithms. Some manufacturers provide recommendations on the approaches for data filtering.
The QC methodology reported in this paper has not taken into account the specific approaches recommended by manufacturers, focusing on testing generic approaches. Additional analysis of uncertainty of measurement and error propagation using more advanced data quality controlled will be included in the SPICE work.
The proposed methodology for assessing the uncertainty of measurement of automatic sensors is an effective tool to quantify and compare the ability to measure of various sensors, and SPICE will further investigate its use at various time scales and in different conditions, to develop recommendations on how to characterize the quality of measurements of automatic measurements, in general.
The authors acknowledge the roles of Sorin Pinzariu and Jeffery Hoover of Environment Canada in the configuration of the snow depth measurement experiment at the Centre for Atmospheric Research Experiments (CARE) site and the provision of details of the measurement system, and the contributions of the site team and observers at CARE in collecting snow depth data and maintaining instruments. This work was funded by the Korea Meteorological Administration Research and Development Program under Grant CATER 2013–2040.
General characteristics of snow depth
sensors. The L,
BEs and uncertainties (
BEs and NBRRMSEs of each snow depth sensor in comparison between manual observation and snow depth sensors.
Layout of snow depth measurement system at the CARE site during the 2013–2014 winter season. The large circles (squares) represent bases (snow depth targets). The orange rectangles and small circles indicate snow stakes and snow depth sensors, respectively (courtesy: Environment Canada).
Snow depth measurement system.
Time series of snow depth from snow stakes on base
Photograph of the uneven snow deposition on the surface of snow depth targets.
Time series of snow depth from
Same as in Fig. 5 except for data after applying the QC procedure described in Sect. 4.3.
Same as in Fig. 6 except for the events prior to snow accumulation. Snow was mostly not seen on the ground on 08 November 2013 and from 12 November 2013 to 15 November 2013.
Scatter plots of snow depth measured by automatic snow depth
sensors (
Scatter plots of snow depth measured by two snow depth sensors of the same type on bases 20 vs. 12A (left), 11A vs. 12A (middle), and 11A vs. 20 (right): FELIX (first column), SOMMER (second column), SR50A (third column), JENOPTIK (fourth column). The values in brackets indicate the NBE, NMAE, NRMSE, and NBRRMSE.