[rrd-users] Measuring rain from weatherstation

[Kurt Sejr Hansen]

Dear allI’m using rrd with a weather station (Sirius) from TFA, get data
every 5 minutes, and an Raspberry PI for producing graph and upload them to
a web server – interested see here <http://sejr-hansen.dk/vejr/vejr.htm>
(Encrypted in Danish) Raincounter from the TFA weather station is a
“counter” in rrd language, and my problem is that I can’t figure out how to
produce graphs and can’t see how I get data the tells about e.g. rain the
last week, day or one hour. I found this
<https://lists.oetiker.ch/pipermail/rrd-users/2000-November/001697.html> :
and has tried to copy this to this shell
script---------#!/bin/bashDIR="/home/pi/rrd/"#Rain the last hourrrdtool
graph dummy --start -1h
\DEF:rain=$DIR/hjemmedatabase.rrd:raincounter:AVERAGE
\CDEF:totalrain=rain,1,3600,*,* \PRINT:totalrain:AVERAGE:"%4.2lf mm"#Reain
last 24 hourrrdtool graph dummy --start -1d
\DEF:rain=$DIR/hjemmedatabase.rrd:raincounter:AVERAGE
\CDEF:totalrain=rain,1,86400,*,* \PRINT:totalrain:AVERAGE:"%4.2lf mm"#rain
the last weekrrdtool graph dummy --start -1w
\DEF:rain=$DIR/hjemmedatabase.rrd:raincounter:AVERAGE
\CDEF:totalrain=rain,1,86400,*,* \PRINT:totalrain:AVERAGE:"%4.2lf mm"#rain
the last monthrrdtool graph dummy --start -1m
\DEF:rain=$DIR/hjemmedatabase.rrd:raincounter:AVERAGE
\CDEF:totalrain=rain,1,86400,*,* \PRINT:totalrain:AVERAGE:"%6.2lf mm"#rain
the last 3 monthrrdtool graph dummy --start -3m
\DEF:rain=$DIR/hjemmedatabase.rrd:raincounter:AVERAGE
\CDEF:totalrain=rain,1,86400,*,* \PRINT:totalrain:AVERAGE:"%6.2lf
mm"-------I can’t see why this is not working. It seems the last hour and 24
hour is correct – but rest?Hope that I can get some help from/Kurt Sejr
Hansen



--
View this message in context: http://rrd-mailinglists.937164.n2.nabble.com/Measuring-rain-from-weatherstation-tp7582709.html
Sent from the RRDtool Users Mailinglist mailing list archive at Nabble.com.

[Alex van den Bogaerdt]

----- Original Message -----
From: "Kurt Sejr Hansen" <***@sejr-hansen.dk>
To: <rrd-***@lists.oetiker.ch>
Sent: Saturday, December 06, 2014 11:24 AM
Subject: [rrd-users] Measuring rain from weatherstation

Post by Kurt Sejr Hansen
Dear allI’m using rrd with a weather station (Sirius) from TFA, get data
every 5 minutes, and an Raspberry PI for producing graph and upload them
to
a web server – interested see here <http://sejr-hansen.dk/vejr/vejr.htm>

Expect some more tips later, i'm a bit in a hurry now

Post by Kurt Sejr Hansen
(Encrypted in Danish) Raincounter from the TFA weather station is a
“counter” in rrd language, and my problem is that I can’t figure out how
to
produce graphs and can’t see how I get data the tells about e.g. rain the
last week, day or one hour. I found this

Are you sure it is COUNTER ? I'm not saying it's wrong, I am only asking
it!
I would expect it to be ABSOLUTE instead of counter.

Post by Kurt Sejr Hansen
<https://lists.oetiker.ch/pipermail/rrd-users/2000-November/001697.html>
and has tried to copy this to this shell
script---------#!/bin/bashDIR="/home/pi/rrd/"#Rain the last hourrrdtool
graph dummy --start -1h
\DEF:rain=$DIR/hjemmedatabase.rrd:raincounter:AVERAGE
\CDEF:totalrain=rain,1,3600,*,* \PRINT:totalrain:AVERAGE:"%4.2lf mm"#Reain
last 24 hourrrdtool graph dummy --start -1d

<snip>

I see "start -1h" which means it should show roughly one hour.

Post by Kurt Sejr Hansen
\DEF:rain=$DIR/hjemmedatabase.rrd:raincounter:AVERAGE
\CDEF:totalrain=rain,1,86400,*,* \PRINT:totalrain:AVERAGE:"%4.2lf mm"#rain
the last weekrrdtool graph dummy --start -1w

and here you try to define "rain" another time, so I expect to see an error.

Don't copy stuff from the internet without understanding what it is supposed
to do. You may end up with a pretty picture, not representing your data at
all.

While debugging this problem, make sure you keep track of the numbers you
get from your device. It would, for instance, be useful to know what the
values are during some rain. Do not just keep the values, also keep the
corresponding timestamps.

Understand the different options to RRDtool. My tutorial and website may
help, or find another source of information. It is especially necessary to
have the right input data type. If that's wrong, it's hard or impossible to
fix in rrdtool graph.

First step is to make sure data going into RRDtool is processed correctly,
so that you end up with <something-per-second>. Then later worry about
translating this during "rrdtool graph" into useful output. You cannot
produce useful output if your database contains gibberish.

As your graph shows rain in megameters, I don't trust your current database.
But don't delete it yet, it could be something weird in your graph
statements. First things first, show the output of "rrdtool info" and gather
some data to post here.

cheers,
alex

[Alex van den Bogaerdt]

Post by Alex van den Bogaerdt

Post by Kurt Sejr Hansen
Dear allI’m using rrd with a weather station (Sirius) from TFA, get data
every 5 minutes, and an Raspberry PI for producing graph and upload them
to
a web server – interested see here <http://sejr-hansen.dk/vejr/vejr.htm>

Expect some more tips later, i'm a bit in a hurry now

First a suggestion for your wind direction graph. Try "--lower-limit
0 --upper limit 360 --rigid --y-grid 30:3".
Play a bit with the settings for y-grid if you don't like what you see.

Then something about large values. I've seen unlikely rates, such as 14
million mm, but also 140m/s wind speed. I can explain why such numbers occur
when using COUNTER data types, but not when using GAUGE as I expect to see
for your wind speed measurements. Please check how this happened. 140m/s is
504 km/h which would be a new record.

For the COUNTER type: if you increment from 1 at midnight to 0 at 00:05,
meaning a reset of the counter has happened, then without protection RRDtool
will assume the counter moved forward by one less than 2^32 in 300 seconds,
thus 14316557,65 per second. There is your 14 million. I will give you a way
to overcome this furtheron in this email.


I found some documentation for a Sirius 300 which seems to indicate the
amount of rain is shown as specific values:
0, <0,5 , <1 , <2 , <5 , <10 , <20 , <50
liters per sqare meter, accumulated for the current day.
This seems to be increasing during the day, and then be reset at midnight.

Is this also what you are getting? If so: there are basically two choices:

#1 poll the value multiple times between resets. In that case it is a
COUNTER, but you will want to do something around midnight to signal a
reset. DERIVE may be a better choice, in combination with a minimum of zero
and optionally with some extra processing. DERIVE in combination with a
minimum of zero will automatically notice a counter reset has happened. And
with a bit of extra work just after midnight, you can even prevent your
database from having unknowns for every interval after midnight.

Use the DERIVE counter type. Set the minimum allowed value to zero or else
the rate of rainfal could become negative. Optionally but recommended, you
could, at midnight, do the following:
1: get the value valid for midnight.
2: process this as usual (use fixed time when updating, e.g. the timestamp
1417910400 instead of 'now' !
3: insert a value of zero just after it (thus 1417910401). RRDtool will
notice the computed rate is invalid and set this one second interval to
unknown. It also resets the internal value it remembers to zero so a future
update compares against zero and will work as desired.

#2 poll the value only once, after which it is reset. In that case it is an
ABSOLUTE data type.

In both cases: A little bit of rain could make the value increase from <1 to
<2 whereas the same amount of rain could also go undetected e.g. from 6.5 to
8.5 is both <10. I would not expect your graph to really work for shorter
intervals of time but also not for longer intervals.

I don't think you can, but just in case you or someone else reading this
can:
If you can reset the rain counter when you want it, then the better option
is to do that. Find a good balance between frequent updates and allowing the
counter to change from e.g. <1 to <2. Read the amount of rain, reset the
counter, continue. If you implement this, then it is ABSOLUTE. It would be
a mistake to do this every 5 minutes. You won't record much rain, unless it
is raining very very hard.

Whichever method you used, you now have liters per square meter per second
in your database. That's also known as mm/s. Multiply by the amount of time
on the graph to get a total for that amount of time.

This won't change the way your graph looks. It is still showing mm/s. You
could multiply that by e.g. 3600 to see 'mm per hour', even if the intervals
on the graph are only 5 minutes. That is not a problem. The speedometer in a
car is showing km/h, same non-problem.

You will not see a running total, you will see an amount of rain per second.
That is what RRDtool does: per second. If you really want to have running
totals during the day (as your device gives you) but also during a week,
month, year, then you will have to do much more programming. Don't think
about it now, get your results correct first.

It is important that you know the amount of seconds in your graph, make sure
to use specific start and end times. E.g. do not use "" (nothing, meaning
the default) for the end time, but rather choose to use "--end 1417910400"
and similar. Combined with "--start end-86400" gives you a full day.

Make sure you understand how RRDtool stores data in RRAs. Each entry in an
RRA represents an amount of time, with specific start and end times. As
long as a whole number of such entries fits perfectly in your graph, RRDtool
does not have to tweak your settings, and because you have programmed the
graph you know how many seconds are shown and by how many seconds you need
to multiply to get an amount.

If the total amount of time shown on a graph is exactly 1 day, then this
will show a line representing rain in mm per hour, and total mm for that
day:
DEF:rain=$DIR/hjemmedatabase.rrd:raincounter:AVERAGE
CDEF:rainperhour=rain,3600,*
LINE1:rainperhour#0000FF:"rain in mm/h"
CDEF:raintotal=rain,86400,*
GPRINT:raintotal:AVERAGE:"%6.2lf mm"

Do remember that you are measuring "less than ... mm/s".
After getting above mentioned stuff right, you could try to increase the
accuracy of your rainfall meter. It involves remembering values read from
it. An example works best:

At time 09:55 you get "<1".
At time 10:00 you get "<2".
Upto and including time 13:55 you get "<2".
At time 14:00 you get "<5".
Upto and including time 17:55 you get "<5".
At time 18:00 you get "<10".
So far you have been logging this as "1", "2", "5" and "10", but with some
programming you could achieve more accurate results.

If you remember the read values and timestamps, then you can do something
like this:
* do not update the database for rainfall, unless the value changes
* do update the database every midnight, even if there was no change
* as soon as the read value changes, you record the previous value in the
database.

The result: at 10:00 you record "1". Then you wait. At time 14:00 you record
"2". RRDtool will automatically fill the time intervals between 10:00 and
14:00 with the new computed rate of 1mm in 4 hours = 1/14400 mm/s.
At time 18:00 you feed RRDtool the number 5. The computed rate will be 4mm
in 4 hours = 1/3600 mm/s.

If the value does not change anymore, then at midnight you have a choice.
Either log 5 again, or log 10, or log 7.5 being the average of those two.
You really do not know how much rain has fallen, you only know it is less
than 10 but not less than 5.

[Kurt Sejr Hansen]

Hi Alex

Great with all these suggestions – I appreciate all this.
My idea was that if I have average values over time, and multiply by time I
should get the total amount of rain. But I will look at your proposals –
including the wind of cause.

/Kurt




--
View this message in context: http://rrd-mailinglists.937164.n2.nabble.com/Measuring-rain-from-weatherstation-tp7582709p7582713.html
Sent from the RRDtool Users Mailinglist mailing list archive at Nabble.com.

[Alex van den Bogaerdt]

----- Original Message -----
From: "Kurt Sejr Hansen" <***@sejr-hansen.dk>
To: <rrd-***@lists.oetiker.ch>
Sent: Sunday, December 07, 2014 2:18 PM
Subject: Re: [rrd-users] Measuring rain from weatherstation

Post by Kurt Sejr Hansen
Hi Alex
Great with all these suggestions – I appreciate all this.
My idea was that if I have average values over time, and multiply by time
I

Sure. But then you need to have a proper average, and a known amount of
time.

My point is that I am afraid you may have neither, so I elaborated on what's
needed and how to get there.

[Simon Hobson]

Post by Alex van den Bogaerdt
At time 09:55 you get "<1".
At time 10:00 you get "<2".
Upto and including time 13:55 you get "<2".
At time 14:00 you get "<5".
Upto and including time 17:55 you get "<5".
At time 18:00 you get "<10".
So far you have been logging this as "1", "2", "5" and "10", but with some programming you could achieve more accurate results.
* do not update the database for rainfall, unless the value changes
* do update the database every midnight, even if there was no change
* as soon as the read value changes, you record the previous value in the database.
The result: at 10:00 you record "1". Then you wait. At time 14:00 you record "2". RRDtool will automatically fill the time intervals between 10:00 and 14:00 with the new computed rate of 1mm in 4 hours = 1/14400 mm/s.
At time 18:00 you feed RRDtool the number 5. The computed rate will be 4mm in 4 hours = 1/3600 mm/s.
If the value does not change anymore, then at midnight you have a choice. Either log 5 again, or log 10, or log 7.5 being the average of those two. You really do not know how much rain has fallen, you only know it is less than 10 but not less than 5.

If you have the option of resetting the counter after an arbitrary reading, then I could see a way of getting it more accurate.

Read the value "quite often". While it is returning "<0.5" then assume 0. When it switched to "<1" then use 0.5 and reset the counter. You will potentially lose a tiny amount of rainfall, but you'd have to be resetting the counter just at the point where the bucket tips (assuming it's a tipping bucket system) - but unless your reading points are a long way apart, that's unlikely.

You now have a value that's normally 0 but "pulses" to 0.5 each time 0.5 units of rain are measured. My mind isn't working well enough right now to work out the best way to put that into the RRD database.
One thought is to accumulate "units" of rain into a file which is read by your script that updates the rest of the weather parameters.
Another is just to insert 0 or 0.5 at frequent intervals - you'll get a spike each time the rainfall reaches 0.5mm, which can be averaged out by RRD. Probably need to use derive and insert 0.5 followed by 0 one second later.

You'd need to allow for larger values - in case your script gets delayed for some reason.

[Alex van den Bogaerdt]

Post by Simon Hobson
If you have the option of resetting the counter after an arbitrary
reading, then I could see a way of getting it more accurate.
Read the value "quite often". While it is returning "<0.5" then assume 0.
When it switched to "<1" then use 0.5 and reset the counter. You will
potentially lose a tiny amount of rainfall, but you'd have to be resetting
the counter just at the point where the bucket tips (assuming it's a
tipping bucket system) - but unless your reading points are a long way
apart, that's unlikely.
You now have a value that's normally 0 but "pulses" to 0.5 each time 0.5
units of rain are measured. My mind isn't working well enough right now to
work out the best way to put that into the RRD database.
One thought is to accumulate "units" of rain into a file which is read by
your script that updates the rest of the weather parameters.
Another is just to insert 0 or 0.5 at frequent intervals - you'll get a
spike each time the rainfall reaches 0.5mm, which can be averaged out by
RRD. Probably need to use derive and insert 0.5 followed by 0 one second
later.

As before: don't write to RRDtool unless the value tipped over to the next
number. There won't be spikes. The amount of rain will be divided by the
amount of seconds since the last update.

[Simon Hobson]

As before: don't write to RRDtool unless the value tipped over to the next number. There won't be spikes. The amount of rain will be divided by the amount of seconds since the last update.

So use derive data type, write 0.5 (or any larger value that's accumulated) followed by 0 whenever the counter tips over ?
Guess we need to write 0 periodically to avoid getting long periods of unknown.

So we end up with :

forever
read value
if value > min
reset counter
reset timer
write T:value to rrd
writeT+1:0 to rrd
else
if timer > some_period
write T:0 to rrd
reset timer
sleep for a bit
go back and do it all again

[Alex van den Bogaerdt]

Post by Simon Hobson
forever
read value
if value > min
reset counter
reset timer
write T:value to rrd
writeT+1:0 to rrd
else
if timer > some_period
write T:0 to rrd
reset timer
sleep for a bit
go back and do it all again

Not quite. I think you are mixing two different solutions.

Option 1: don't reset the counter every time it is read. The counter will
reset itself just after midnight. In this case use DERIVE and optionally do
write that zero. Writing the zero is optional, it will help a bit in
preventing an unknown at the beginning of each day.

Option 2: (if possible) reset the counter every time it changed. This means
using ABSOLUTE as counter type. Write the appropriate amount to RRD. Do not
write an extra zero to the rrd, this is already taken care of by using
ABSOLUTE.

But yes, something does need to be done before the hearbeat timer expires,
in both cases.