[rrd-users] Strange issue with large step value

[_ JvG _]

Hi,

I'm trying to import some old data (collected once every week at 19.00 hour
on Sunday)

This is what I did:

rrdtool create pymeter-weekdata.rrd \
--start 1419184800 \
--step 604800 \
DS:qtotal:COUNTER:907200:0:100000 \
RRA:AVERAGE:0.5:1:53

rrdtool update pymeter-weekdata.rrd 1419789600:50628
rrdtool update pymeter-weekdata.rrd 1420398000:50803
rrdtool update pymeter-weekdata.rrd 1421002800:50959
rrdtool update pymeter-weekdata.rrd 1421607600:51120

rrdtool xport \
--start 1419184800 \
--end 1425236400 \
--step 604800 \
DEF:Qtotal=pymeter-weekdata.rrd:qtotal:AVERAGE \
CDEF:QTot=Qtotal,604800,*,100,/ \
XPORT:QTot:"Usage\n"

and I get this:

<xport>
<meta>
<start>1419465600</start>
<step>604800</step>
<end>1419465600</end>
<rows>11</rows>
<columns>1</columns>
<legend>
<entry>Usage\n</entry>
</legend>
</meta>
<data>
<row><t>1419465600</t><v>NaN</v></row>
<row><t>1420070400</t><v>NaN</v></row>
<row><t>1420675200</t><v>1.6573076923e+00</v></row>
<row><t>1421280000</t><v>1.5829166667e+00</v></row>
<row><t>1421884800</t><v>NaN</v></row>
<row><t>1422489600</t><v>NaN</v></row>
<row><t>1423094400</t><v>NaN</v></row>
<row><t>1423699200</t><v>NaN</v></row>
<row><t>1424304000</t><v>NaN</v></row>
<row><t>1424908800</t><v>NaN</v></row>
<row><t>1425513600</t><v>NaN</v></row>
</data>
</xport>

while I would expect these results:

508.03 - 506.28 = 1,75
509.59 - 508.03 = 1.56
511.20 - 509.59 = 1.61

What did I do wrong or what do I need to correct to get the expected
results?

JJ

--

[Alex van den Bogaerdt]

I guess the problem may have to do with the fact that your time stamps are
not a whole multiple of your step size.

Not as a solution, but as a means to investigate the problem: try to
populate an RRD with a step size of an hour, and plenty of room in a couple
of RRAs. See if your data makes it into the database and is as expected.

----- Original Message -----
From: "_ JvG _" <***@knowzone.org>
To: <rrd-***@lists.oetiker.ch>
Sent: Monday, May 04, 2015 4:24 PM
Subject: [rrd-users] Strange issue with large step value

Post by _ JvG _
Hi,
I'm trying to import some old data (collected once every week at 19.00
hour
on Sunday)
rrdtool create pymeter-weekdata.rrd \
--start 1419184800 \
--step 604800 \
DS:qtotal:COUNTER:907200:0:100000 \
RRA:AVERAGE:0.5:1:53
rrdtool update pymeter-weekdata.rrd 1419789600:50628
rrdtool update pymeter-weekdata.rrd 1420398000:50803
rrdtool update pymeter-weekdata.rrd 1421002800:50959
rrdtool update pymeter-weekdata.rrd 1421607600:51120
rrdtool xport \
--start 1419184800 \
--end 1425236400 \
--step 604800 \
DEF:Qtotal=pymeter-weekdata.rrd:qtotal:AVERAGE \
CDEF:QTot=Qtotal,604800,*,100,/ \
XPORT:QTot:"Usage\n"
<xport>
<meta>
<start>1419465600</start>
<step>604800</step>
<end>1419465600</end>
<rows>11</rows>
<columns>1</columns>
<legend>
<entry>Usage\n</entry>
</legend>
</meta>
<data>
<row><t>1419465600</t><v>NaN</v></row>
<row><t>1420070400</t><v>NaN</v></row>
<row><t>1420675200</t><v>1.6573076923e+00</v></row>
<row><t>1421280000</t><v>1.5829166667e+00</v></row>
<row><t>1421884800</t><v>NaN</v></row>
<row><t>1422489600</t><v>NaN</v></row>
<row><t>1423094400</t><v>NaN</v></row>
<row><t>1423699200</t><v>NaN</v></row>
<row><t>1424304000</t><v>NaN</v></row>
<row><t>1424908800</t><v>NaN</v></row>
<row><t>1425513600</t><v>NaN</v></row>
</data>
</xport>
508.03 - 506.28 = 1,75
509.59 - 508.03 = 1.56
511.20 - 509.59 = 1.61
What did I do wrong or what do I need to correct to get the expected
results?
JJ
--

--------------------------------------------------------------------------------

Post by _ JvG _
_______________________________________________
rrd-users mailing list
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

[_ JvG _]

:)
my bad (make not for self -- do not calculate epoch dates by hand)

rrdtool create pymeter-weekdata.rrd \
--start 1419184800 \
--step 604800 \
DS:qtotal:COUNTER:907200:0:100000 \
RRA:AVERAGE:0.5:1:53

rrdtool update pymeter-weekdata.rrd 1419789600:50628
rrdtool update pymeter-weekdata.rrd 1420394400:50803
rrdtool update pymeter-weekdata.rrd 1420999200:50959
rrdtool update pymeter-weekdata.rrd 1421604000:51120

rrdtool xport \
--start 1419184800 \
--end 1425236400 \
--step 604800 \
DEF:Qtotal=pymeter-weekdata.rrd:qtotal:AVERAGE \
CDEF:QTot=Qtotal,604800,*,100,/ \
XPORT:QTot:"Usage\n"

<?xml version="1.0" encoding="ISO-8859-1"?>

<xport>
<meta>
<start>1419465600</start>
<step>604800</step>
<end>1419465600</end>
<rows>11</rows>
<columns>1</columns>
<legend>
<entry>Usage\n</entry>
</legend>
</meta>
<data>
<row><t>1419465600</t><v>NaN</v></row>
<row><t>1420070400</t><v>NaN</v></row>
<row><t>1420675200</t><v>1.6617857143e+00</v></row>
<row><t>1421280000</t><v>1.5832142857e+00</v></row>
<row><t>1421884800</t><v>NaN</v></row>
<row><t>1422489600</t><v>NaN</v></row>
<row><t>1423094400</t><v>NaN</v></row>
<row><t>1423699200</t><v>NaN</v></row>
<row><t>1424304000</t><v>NaN</v></row>
<row><t>1424908800</t><v>NaN</v></row>
<row><t>1425513600</t><v>NaN</v></row>
</data>
</xport>

still not the expected numbers...

Please explain how I convert the above example to an hourly database
without recalculating the values

JJ

On Mon, May 4, 2015 at 10:50 PM, Alex van den Bogaerdt <

Post by Alex van den Bogaerdt
I guess the problem may have to do with the fact that your time stamps are
not a whole multiple of your step size.
Not as a solution, but as a means to investigate the problem: try to
populate an RRD with a step size of an hour, and plenty of room in a couple
of RRAs. See if your data makes it into the database and is as expected.
Sent: Monday, May 04, 2015 4:24 PM
Subject: [rrd-users] Strange issue with large step value
Hi,

Post by _ JvG _
I'm trying to import some old data (collected once every week at 19.00
hour
on Sunday)
rrdtool create pymeter-weekdata.rrd \
--start 1419184800 \
--step 604800 \
DS:qtotal:COUNTER:907200:0:100000 \
RRA:AVERAGE:0.5:1:53
rrdtool update pymeter-weekdata.rrd 1419789600:50628
rrdtool update pymeter-weekdata.rrd 1420398000:50803
rrdtool update pymeter-weekdata.rrd 1421002800:50959
rrdtool update pymeter-weekdata.rrd 1421607600:51120
rrdtool xport \
--start 1419184800 \
--end 1425236400 \
--step 604800 \
DEF:Qtotal=pymeter-weekdata.rrd:qtotal:AVERAGE \
CDEF:QTot=Qtotal,604800,*,100,/ \
XPORT:QTot:"Usage\n"
<xport>
<meta>
<start>1419465600</start>
<step>604800</step>
<end>1419465600</end>
<rows>11</rows>
<columns>1</columns>
<legend>
<entry>Usage\n</entry>
</legend>
</meta>
<data>
<row><t>1419465600</t><v>NaN</v></row>
<row><t>1420070400</t><v>NaN</v></row>
<row><t>1420675200</t><v>1.6573076923e+00</v></row>
<row><t>1421280000</t><v>1.5829166667e+00</v></row>
<row><t>1421884800</t><v>NaN</v></row>
<row><t>1422489600</t><v>NaN</v></row>
<row><t>1423094400</t><v>NaN</v></row>
<row><t>1423699200</t><v>NaN</v></row>
<row><t>1424304000</t><v>NaN</v></row>
<row><t>1424908800</t><v>NaN</v></row>
<row><t>1425513600</t><v>NaN</v></row>
</data>
</xport>
508.03 - 506.28 = 1,75
509.59 - 508.03 = 1.56
511.20 - 509.59 = 1.61
What did I do wrong or what do I need to correct to get the expected
results?
JJ
--

--------------------------------------------------------------------------------
_______________________________________________

Post by _ JvG _
rrd-users mailing list
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

_______________________________________________
rrd-users mailing list
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

[Alex van den Bogaerdt]

Post by _ JvG _
Please explain how I convert the above example to an hourly database
without recalculating the values

Create a new database to test, investigate and learn.

The step size should be divided by 168 (and then becomes 3600), the amount
of rows should be multiplied by 168 (to store the same amount of time), and
then feed the same data (rrdtool will update more than one row at a time, no
problem).

--step 3600
RRA:AVERAGE:0.5:1:8904

Now use "rrdtool dump" and open the resulting file with a text editor to get
an idea what is inside this database.

Repeat the excercise, now with two RRAs:

--step 3600
RRA:AVERAGE:0.5:1:8904
RRA:AVERAGE:0.5:168:53

This second RRA should be very much the same as what you get now: one entry
for every week, 53 times.

Depending on the results you can test further. Still no data? Perhaps
there's something wrong with your heartbeat settings, or your min/max
settings. Data, but not what you expect? Then the input is not what you
expect.

After looking at this, do your "rrdtool xport" again and compare against the
"rrdtool dump" results. Does it do what you think it should do.

[_ JvG _]

I did what you asked:

rrdtool create pymeter-weekdata.rrd \
--start 1419184800 \
--step 3600 \
DS:qtotal:COUNTER:604800:0:100000 \
RRA:AVERAGE:0.5:1:8904 \
RRA:AVERAGE:0.5:168:53

rrdtool update pymeter-weekdata.rrd 1419789600:50628
rrdtool update pymeter-weekdata.rrd 1420394400:50803
rrdtool update pymeter-weekdata.rrd 1420999200:50959
rrdtool update pymeter-weekdata.rrd 1421604000:51120

rrdtool xport \
--start 1419184800 \
--end 1425236400 \
--step 604800 \
DEF:Qtotal=pymeter-weekdata.rrd:qtotal:AVERAGE \
CDEF:QTot=Qtotal,604800,*,100,/ \
XPORT:QTot:"Usage\n"

<?xml version="1.0" encoding="ISO-8859-1"?>

<xport>
<meta>
<start>1419465600</start>
<step>604800</step>
<end>1419465600</end>
<rows>11</rows>
<columns>1</columns>
<legend>
<entry>Usage\n</entry>
</legend>
</meta>
<data>
<row><t>1419465600</t><v>NaN</v></row>
<row><t>1420070400</t><v>NaN</v></row>
<row><t>1420675200</t><v>1.6617857143e+00</v></row>
<row><t>1421280000</t><v>1.5832142857e+00</v></row>
<row><t>1421884800</t><v>NaN</v></row>
<row><t>1422489600</t><v>NaN</v></row>
<row><t>1423094400</t><v>NaN</v></row>
<row><t>1423699200</t><v>NaN</v></row>
<row><t>1424304000</t><v>NaN</v></row>
<row><t>1424908800</t><v>NaN</v></row>
<row><t>1425513600</t><v>NaN</v></row>
</data>
</xport>


the data in the new table is even more off...


<row><t>1419804000</t><v>1.0416666667e-02</v></row>
<row><t>1420380000</t><v>1.0416666667e-02</v></row>
.
.
.
<row><t>1420398000</t><v>1.0190476190e-02</v></row>
<row><t>1420416000</t><v>9.2857142857e-03</v></row>
.
.
.
<row><t>1420992000</t><v>9.2857142857e-03</v></row>
<row><t>1421010000</t><v>9.4642857143e-03</v></row>
<row><t>1421028000</t><v>9.5833333333e-03</v></row>
.
.
.
<row><t>1421604000</t><v>9.5833333333e-03</v></row>


Ans the problem is still there.
I've did some more tests with different heartbeats with the same results.
I do not get your remark that my input is wrong...

Please help



On Tue, May 5, 2015 at 7:51 AM, Alex van den Bogaerdt <

Post by _ JvG _
Please explain how I convert the above example to an hourly database

Post by _ JvG _
without recalculating the values

Create a new database to test, investigate and learn.
The step size should be divided by 168 (and then becomes 3600), the amount
of rows should be multiplied by 168 (to store the same amount of time), and
then feed the same data (rrdtool will update more than one row at a time,
no problem).
--step 3600
RRA:AVERAGE:0.5:1:8904
Now use "rrdtool dump" and open the resulting file with a text editor to
get an idea what is inside this database.
--step 3600
RRA:AVERAGE:0.5:1:8904
RRA:AVERAGE:0.5:168:53
This second RRA should be very much the same as what you get now: one
entry for every week, 53 times.
Depending on the results you can test further. Still no data? Perhaps
there's something wrong with your heartbeat settings, or your min/max
settings. Data, but not what you expect? Then the input is not what you
expect.
After looking at this, do your "rrdtool xport" again and compare against
the "rrdtool dump" results. Does it do what you think it should do.
_______________________________________________
rrd-users mailing list
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

[Alex van den Bogaerdt]

[...]

Post by _ JvG _
rrdtool xport \

Post by Alex van den Bogaerdt
Now use "rrdtool dump" and open the resulting file with a text editor to
get an idea what is inside this database.

[_ JvG _]

<rra>
<cf>AVERAGE</cf>
<pdp_per_row>168</pdp_per_row> <!-- 604800 seconds -->

<params>
<xff>5.0000000000e-01</xff>
</params>
<cdp_prep>
<ds>
<primary_value>2.6177484883e-04</primary_value>
<secondary_value>2.6620370370e-04</secondary_value>
<value>2.3958333333e-02</value>
<unknown_datapoints>0</unknown_datapoints>
</ds>
</cdp_prep>
<database>
<!-- 2014-01-16 01:00:00 CET / 1389830400 -->
<row><v>NaN</v></row>
<!-- 2014-01-23 01:00:00 CET / 1390435200 -->
<row><v>NaN</v></row>
<!-- 2014-01-30 01:00:00 CET / 1391040000 -->
<row><v>NaN</v></row>
<!-- 2014-02-06 01:00:00 CET / 1391644800 -->
<row><v>NaN</v></row>
<!-- 2014-02-13 01:00:00 CET / 1392249600 -->
<row><v>NaN</v></row>
<!-- 2014-02-20 01:00:00 CET / 1392854400 -->
<row><v>NaN</v></row>
<!-- 2014-02-27 01:00:00 CET / 1393459200 -->
<row><v>NaN</v></row>
<!-- 2014-03-06 01:00:00 CET / 1394064000 -->
<row><v>NaN</v></row>
<!-- 2014-03-13 01:00:00 CET / 1394668800 -->
<row><v>NaN</v></row>
<!-- 2014-03-20 01:00:00 CET / 1395273600 -->
<row><v>NaN</v></row>
<!-- 2014-03-27 01:00:00 CET / 1395878400 -->
<row><v>NaN</v></row>
<!-- 2014-04-03 02:00:00 CEST / 1396483200 -->
<row><v>NaN</v></row>
<!-- 2014-04-10 02:00:00 CEST / 1397088000 -->
<row><v>NaN</v></row>
<!-- 2014-04-17 02:00:00 CEST / 1397692800 -->
<row><v>NaN</v></row>
<!-- 2014-04-24 02:00:00 CEST / 1398297600 -->
<row><v>NaN</v></row>
<!-- 2014-05-01 02:00:00 CEST / 1398902400 -->
<row><v>NaN</v></row>
<!-- 2014-05-08 02:00:00 CEST / 1399507200 -->
<row><v>NaN</v></row>
<!-- 2014-05-15 02:00:00 CEST / 1400112000 -->
<row><v>NaN</v></row>
<!-- 2014-05-22 02:00:00 CEST / 1400716800 -->
<row><v>NaN</v></row>
<!-- 2014-05-29 02:00:00 CEST / 1401321600 -->
<row><v>NaN</v></row>
<!-- 2014-06-05 02:00:00 CEST / 1401926400 -->
<row><v>NaN</v></row>
<!-- 2014-06-12 02:00:00 CEST / 1402531200 -->
<row><v>NaN</v></row>
<!-- 2014-06-19 02:00:00 CEST / 1403136000 -->
<row><v>NaN</v></row>
<!-- 2014-06-26 02:00:00 CEST / 1403740800 -->
<row><v>NaN</v></row>
<!-- 2014-07-03 02:00:00 CEST / 1404345600 -->
<row><v>NaN</v></row>
<!-- 2014-07-10 02:00:00 CEST / 1404950400 -->
<row><v>NaN</v></row>
<!-- 2014-07-17 02:00:00 CEST / 1405555200 -->
<row><v>NaN</v></row>
<!-- 2014-07-24 02:00:00 CEST / 1406160000 -->
<row><v>NaN</v></row>
<!-- 2014-07-31 02:00:00 CEST / 1406764800 -->
<row><v>NaN</v></row>
<!-- 2014-08-07 02:00:00 CEST / 1407369600 -->
<row><v>NaN</v></row>
<!-- 2014-08-14 02:00:00 CEST / 1407974400 -->
<row><v>NaN</v></row>
<!-- 2014-08-21 02:00:00 CEST / 1408579200 -->
<row><v>NaN</v></row>
<!-- 2014-08-28 02:00:00 CEST / 1409184000 -->
<row><v>NaN</v></row>
<!-- 2014-09-04 02:00:00 CEST / 1409788800 -->
<row><v>NaN</v></row>
<!-- 2014-09-11 02:00:00 CEST / 1410393600 -->
<row><v>NaN</v></row>
<!-- 2014-09-18 02:00:00 CEST / 1410998400 -->
<row><v>NaN</v></row>
<!-- 2014-09-25 02:00:00 CEST / 1411603200 -->
<row><v>NaN</v></row>
<!-- 2014-10-02 02:00:00 CEST / 1412208000 -->
<row><v>NaN</v></row>
<!-- 2014-10-09 02:00:00 CEST / 1412812800 -->
<row><v>NaN</v></row>
<!-- 2014-10-16 02:00:00 CEST / 1413417600 -->
<row><v>NaN</v></row>
<!-- 2014-10-23 02:00:00 CEST / 1414022400 -->
<row><v>NaN</v></row>
<!-- 2014-10-30 01:00:00 CET / 1414627200 -->
<row><v>NaN</v></row>
<!-- 2014-11-06 01:00:00 CET / 1415232000 -->
<row><v>NaN</v></row>
<!-- 2014-11-13 01:00:00 CET / 1415836800 -->
<row><v>NaN</v></row>
<!-- 2014-11-20 01:00:00 CET / 1416441600 -->
<row><v>NaN</v></row>
<!-- 2014-11-27 01:00:00 CET / 1417046400 -->
<row><v>NaN</v></row>
<!-- 2014-12-04 01:00:00 CET / 1417651200 -->
<row><v>NaN</v></row>
<!-- 2014-12-11 01:00:00 CET / 1418256000 -->
<row><v>NaN</v></row>
<!-- 2014-12-18 01:00:00 CET / 1418860800 -->
<row><v>NaN</v></row>
<!-- 2014-12-25 01:00:00 CET / 1419465600 -->
<row><v>NaN</v></row>
<!-- 2015-01-01 01:00:00 CET / 1420070400 -->
<row><v>NaN</v></row>
<!-- 2015-01-08 01:00:00 CET / 1420675200 -->
<row><v>2.7476615646e-04</v></row>
<!-- 2015-01-15 01:00:00 CET / 1421280000 -->
<row><v>2.6177484883e-04</v></row>
</database>
</rra>
</rrd>

On Tue, May 5, 2015 at 9:56 AM, Alex van den Bogaerdt <

Post by Alex van den Bogaerdt

Post by _ JvG _
[...]
rrdtool xport \

Now use "rrdtool dump" and open the resulting file with a text editor to

Post by _ JvG _

Post by _ JvG _
get an idea what is inside this database.