[rrd-users] XML dump/restore: How to handle "cdp_prep"?

[Christian Schrötter]

Hello list!

I'm writing a small PHP script to dump a (Munin) RRD file, change some
things with SimpleXML and restore it to a RRD file:

* GROW each RRA like rrdresize by a new format definition.
* Import old (deleted) data from another (dumped) RRD file.

This works great, but I'm not sure what to do with the "cdp_prep" XML
node, especially with the *value elements. Am I supposed to leave it
unchanged (the rrdresize way), delete the whole node or change it to
another value?

<cdp_prep>
<ds>
<primary_value>7.0714381033e+01</primary_value>
<secondary_value>6.5151843957e+01</secondary_value>
<value>7.0289869143e+01</value>
<unknown_datapoints>0</unknown_datapoints>
</ds>
</cdp_prep>

Because I don't understand the usage of this, I'm a little bit
confused.
Or is there a right way to "update/sync" it? Thanks for any hints! :-)

[Alex van den Bogaerdt]

----- Original Message -----
From: "Christian Schrötter" <***@lists.kb19.at>
To: "rrd-users Liste" <rrd-***@lists.oetiker.ch>
Sent: Saturday, August 16, 2014 6:07 PM
Subject: [rrd-users] XML dump/restore: How to handle "cdp_prep"?

Post by Christian Schrötter
Hello list!
I'm writing a small PHP script to dump a (Munin) RRD file, change some
* GROW each RRA like rrdresize by a new format definition.

Explain "by a new format definition".

Post by Christian Schrötter
* Import old (deleted) data from another (dumped) RRD file.
This works great, but I'm not sure what to do with the "cdp_prep" XML
node, especially with the *value elements. Am I supposed to leave it
unchanged (the rrdresize way), delete the whole node or change it to
another value?

Rrdtool resize adds rows to the past of the data. It does not do anything
with the present time where the current CDP is being built. If your version
is doing something to the data (e.g. change amount of time per CDP) then you
should change cdp_prep accordingly. If not, not.

Post by Christian Schrötter
Because I don't understand the usage of this, I'm a little bit confused.
Or is there a right way to "update/sync" it? Thanks for any hints! :-)

If you don't know how CDPs are built, may I suggest you visit my site (
http://rrdtool.vandenbogaerdt.nl ) and also look in the source code of
rrdtool.


HTH
Alex

[Christian Schrötter]

Sorry Alex, forgot the list as recipient... ;-)

Post by Alex van den Bogaerdt

Post by Christian Schrötter
* GROW each RRA like rrdresize by a new format definition.

Explain "by a new format definition".

1st step: Only more rows in the past (like rrdresize):
* RRA 0/1/2: 576 rows -> 1584 rows (+1008 rows at the beginning)
* RRA 3/4/5: xxx rows -> xxxx rows (+xxx rows at the beginning)
* ...

Post by Alex van den Bogaerdt

Post by Christian Schrötter
* Import old (deleted) data from another (dumped) RRD file.
This works great, but I'm not sure what to do with the "cdp_prep"
XML node, especially with the *value elements. Am I supposed to leave
it unchanged (the rrdresize way), delete the whole node or change it
to another value?

Rrdtool resize adds rows to the past of the data. It does not do
anything with the present time where the current CDP is being built.
If your version is doing something to the data (e.g. change amount of
time per CDP) then you should change cdp_prep accordingly. If not,
not.

2nd step: Import some data into the added rows. The current rows (=
existing ones) are not touched. Is this problematic too?

Post by Alex van den Bogaerdt
If you don't know how CDPs are built, may I suggest you visit my site
( http://rrdtool.vandenbogaerdt.nl ) and also look in the source code
of rrdtool.

Thanks, I'll take a look at this too.

[Alex van den Bogaerdt]

Post by Christian Schrötter
* RRA 0/1/2: 576 rows -> 1584 rows (+1008 rows at the beginning)
* RRA 3/4/5: xxx rows -> xxxx rows (+xxx rows at the beginning)
* ...
2nd step: Import some data into the added rows. The current rows (=
existing ones) are not touched. Is this problematic too?

If nothing changes, then I think you should keep the cdp_prep data and do
not touch it. RRDtool can continue to work as before, build a new CDP (it
may be halfway doing so), as if nothing happened. The only change is that
you have more rows available for the past. You are filling in those rows
using historic data, which is okay and does not affect the present time.

I hope others will review what I claim here.

HTH
Alex

[Christian Schrötter]

Post by Alex van den Bogaerdt
If nothing changes, then I think you should keep the cdp_prep data
and do not touch it. RRDtool can continue to work as before, build a
new CDP (it may be halfway doing so), as if nothing happened. The
only
change is that you have more rows available for the past. You are
filling in those rows using historic data, which is okay and does not
affect the present time.

I've uploaded some samples for interested parties:
http://www.happytec.at/upload_files_list/rrd/2014-08-17/

I'll test it tomorrow with some real RRD files from my
munin-master-server at home. I'll let you know the results :-)

[Christian Schrötter]

Post by Christian Schrötter

Post by Alex van den Bogaerdt
If nothing changes, then I think you should keep the cdp_prep data
and do not touch it. RRDtool can continue to work as before, build a
new CDP (it may be halfway doing so), as if nothing happened. The
only
change is that you have more rows available for the past. You are
filling in those rows using historic data, which is okay and does
not
affect the present time.

I'll test it tomorrow with some real RRD files from my
munin-master-server at home. I'll let you know the results :-)

Done! I've successfully resized/imported (1 year -> 3 years) ~1700 RRD
files in ~12 hours on a weak ARM system. Thanks for your input :-)

[Steve Shipway]

I have a utility I've written to merge multiple historic RRD files into a
single RRD with longer RRAs (it was originally written to merge the old-style
RRD archives from MRTG/RRD/Routers2 into new-style single extended RRD files).
It can also perform more complex merges from different RRD files with
non-matching RRAs or DSs. It pulls older data where possible to make up the
gaps.

If you'd like a copy of this, it is included with the latest Routers2 package,
(2.23beta5) or drop me a line and I can email it to you. It is written in
Perl and works by building a new XML for import.

Steve

Steve Shipway
***@auckland.ac.nz

[Christian Schrötter]

Post by Steve Shipway
I have a utility I've written to merge multiple historic RRD files
[…]
If you'd like a copy of this […] drop me a line and I can email it to
you.

My personal RRD migration task is finished, but for the future: Yes,
please! Thx :-)