[dev.icinga.com #1173] int(11) to small for some of ido tables #538

icinga-migration · 2011-01-31T10:05:12Z

This issue has been migrated from Redmine: https://dev.icinga.com/issues/1173

Created by berk on 2011-01-31 10:05:12 +00:00

Assignee: Tommi
Status: Closed (closed on 2011-05-19 11:07:26 +00:00)
Target Version: (none)
Last Update: 2014-12-08 15:18:43 +00:00 (in Redmine)

Icinga Version: 1.10.0
OS Version: any

Hi Michael,

you discussed that stuff with Tobias. We think that int(11) is to small for some cases. In the past there where not a lot of users using the db results for stuff like report and web interface, but in the future using icinga-web and addons like lconf the support for larger id.

I think using bigint would be a good idea.

Regards

Bernd

Attachments

mysql_alter_icinga13_numbers.sql Tommi - 2011-05-05 15:08:41 +00:00 - Script for mysql
pgsql_alter_icinga13_types.sql Tommi - 2011-05-05 15:08:41 +00:00 - Script for Postgresql
oracle_alter_icinga13_numbers.sql Tommi - 2011-05-05 15:08:41 +00:00 - Script for Oracle

Changesets

2011-03-27 20:16:09 +00:00 by Tommi 3ee455b

oracle update scripts v1.3.0->1.4.0
fixes #1173
implements #1354, #1355 as update scripts
Committer: Thomas Dreßler

2011-05-03 20:36:10 +00:00 by Tommi c46e5e1

Fix int(11) to small #1173
Fix integer not big enough #1401
refs #1173,#1401

2011-05-03 20:59:45 +00:00 by Tommi 874e2ad

includes all patchesi and features in upgrade/*.sql
refs #1354,#1355,#1358,#1363,#1401,#1173

2011-05-03 21:08:39 +00:00 by Tommi ed19196

includes all fixes from upgrade
refs #1401,#1173,#1355

2011-05-03 21:27:58 +00:00 by Tommi c74b02e

fixes for mysql int(11) to small #1173,
integer not big enough #1401
remove not null constraints #1355
refs #1355,#1401,#1173

2011-05-03 21:48:11 +00:00 by Tommi 2f773b2

includes all fixes from upgrade_1.4
remove unwanted characters
refs #1173,#1401,#1355

2011-05-04 16:52:26 +00:00 by Tommi 2b806b2

increase serveroutput buffer size to prevent buffer overflow #1173
refs #1173,#1401

2011-05-05 16:05:21 +00:00 by mfriedrich a3a031c

idoutils: s/'%lu'/%lu/g s/'%d'/%d/g

seems like initial instance_id get_uint failed, so
changing foobar all over.

refs #1173

2011-05-09 12:47:46 +00:00 by mfriedrich a2dc45f

idoutils: fix s/get_uint/get_ulonglong/ for 8 byte unsigned integer using libdbi #1173

fixed some debug output issues too.

refs #1173

Relations:

relates #1173
relates #1401
duplicates #1173

The text was updated successfully, but these errors were encountered:

icinga-migration · 2011-01-31T10:05:58Z

Updated by berk on 2011-01-31 10:05:58 +00:00

Subject changed from int(11) too small for some of ido tables to int(11) to small for some of ido tables

icinga-migration · 2011-01-31T14:24:22Z

Updated by mfriedrich on 2011-01-31 14:24:22 +00:00

Status changed from New to Feedback

mh discussed is the wrong phrase for that - he told me that he found the id running out of int(11) on the customvariablestatus - which is a rather not commonly used table. there must have been quite a few restarts on that table to get the pk that high.

i'd like to see a proposal which tables should be affected because a larger number might lack a bit of performance then.

icinga-migration · 2011-01-31T15:02:21Z

Updated by berk on 2011-01-31 15:02:21 +00:00

Hi Michi,

is the lack of performance large enough to split model definition into int and bigint?

Regards

Bernd

icinga-migration · 2011-01-31T15:04:48Z

Updated by mfriedrich on 2011-01-31 15:04:48 +00:00

no, but i don't like big upgrade scripts, and even more 'check if all rdbms can do that the same way/behavior'. and furthermore, i'd prefer full git patches ;p

icinga-migration · 2011-02-01T17:51:14Z

Updated by mfriedrich on 2011-02-01 17:51:14 +00:00

Assigned to deleted ~~mfriedrich~~
Target Version changed from 1.3 to 1.4

the more reason i'm also asking - those ids are used for table relations. so it's not only the general ids but mainly the relation ids which need to be adapted.
i think this could be a quickfix for now, but needs to be tested in all setups before applying that to the users (although i don't think it will make problems).

thoughts?

icinga-migration · 2011-03-14T21:09:11Z

Updated by mfriedrich on 2011-03-14 21:09:11 +00:00

Status changed from Feedback to Assigned
Assigned to set to mfriedrich

since this was already reported in other stages, and in conclusion with other dbas ("wurscht"), this makes sense to be replaced in 1.4 for the complete schema and upgrade scripts.

icinga-migration · 2011-03-16T17:34:26Z

Updated by mfriedrich on 2011-03-16 17:34:26 +00:00

pgsql uses SERIAL for pks, INTEGER for fks. that should be fine, and not targetted.

oracle uses number(.) while mysql int(.) and smallint(.*) (smallint would be number(6)).

icinga-migration · 2011-03-24T22:06:51Z

Updated by mfriedrich on 2011-03-24 22:06:51 +00:00

Assigned to changed from mfriedrich to Tommi

yours :)

icinga-migration · 2011-03-27T20:52:00Z

Updated by Tommi on 2011-03-27 20:52:00 +00:00

Status changed from Assigned to Feedback
Done % changed from 0 to 100

commit 3ee455b
->script alter_icinga13_numbers.sql as update script
commit df45798
->script create_icinga_objects_oracle.sql for new creation

icinga-migration · 2011-03-27T21:38:44Z

Updated by Tommi on 2011-03-27 21:38:44 +00:00

Status changed from Feedback to Resolved

Applied in changeset 3ee455b.

icinga-migration · 2011-04-13T20:56:50Z

Updated by mfriedrich on 2011-04-13 20:56:50 +00:00

Status changed from Resolved to Assigned

that's still missing for mysql and pgsql relatives.

icinga-migration · 2011-04-13T22:28:23Z

Updated by brianatb on 2011-04-13 22:28:23 +00:00

Regarding #1405 (bigints for mysql), what's the best thing to do as a work around until it's released in v1.4?

icinga-migration · 2011-04-13T22:33:42Z

Updated by mfriedrich on 2011-04-13 22:33:42 +00:00

Done % changed from 100 to 50

alter the columns for each table and pk from int(11) to bigint. if you happen to do that, please put the sql over here - we can use that as upgrade script then, and it will save much duplicated work :)

icinga-migration · 2011-04-15T19:41:12Z

Updated by Tommi on 2011-04-15 19:41:12 +00:00

File added mysql_alter.sql
File added mysql_new.sql
Status changed from Assigned to Feedback
Done % changed from 50 to 70

attached mysql scripts should change the datatype for xxx_id columns to "bigint unsigned not null unique auto_increment"(="serial" in mysql 5.1+). If somebody can confirm its working in this way i will commit this to git.
pgsql should be already fine because its still using this serial datatype.

Changing this for other fields may require more code changes when switching from int to long long.

icinga-migration · 2011-04-27T17:23:29Z

Updated by mfriedrich on 2011-04-27 17:23:29 +00:00

the sqls themselves work. but the main problem are the relations - everywhere an instance_id or *_object_id is being used, this needs to changed too - e.g.

mysql> show create table icinga_objects;
+----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table          | Create Table                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
+----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| icinga_objects | CREATE TABLE `icinga_objects` (
  `object_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `instance_id` smallint(6) NOT NULL DEFAULT '0',
  `objecttype_id` smallint(6) NOT NULL DEFAULT '0',
  `name1` varchar(128) CHARACTER SET latin1 COLLATE latin1_general_cs NOT NULL DEFAULT '',
  `name2` varchar(128) CHARACTER SET latin1 COLLATE latin1_general_cs DEFAULT NULL,
  `is_active` smallint(6) NOT NULL DEFAULT '0',
  PRIMARY KEY (`object_id`),
  UNIQUE KEY `object_id` (`object_id`),
  UNIQUE KEY `object_id_2` (`object_id`),
  KEY `objecttype_id` (`objecttype_id`,`name1`,`name2`),
  KEY `objects_objtype_id_idx` (`objecttype_id`),
  KEY `objects_name1_idx` (`name1`),
  KEY `objects_name2_idx` (`name2`),
  KEY `objects_inst_id_idx` (`instance_id`)
) ENGINE=InnoDB AUTO_INCREMENT=48 DEFAULT CHARSET=latin1 COMMENT='Current and historical objects of all kinds' |
+----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

where it says that the pk is being changed, but the related instance_id is not...

  `object_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `instance_id` smallint(6) NOT NULL DEFAULT '0',

so this needs a little more rework.

icinga-migration · 2011-04-27T19:39:07Z

Updated by Tommi on 2011-04-27 19:39:07 +00:00

Target Version changed from 1.4 to 1.5
Done % changed from 70 to 20

You are complete right. All _id colums will need the larger fields. There are also other fields which needs to inspect. For this reason i would like to suggest a review of the data model regarding size of datypes and expected values. Afterwards we will have a new datamodel which we have to bring it into the c code. This will cause very large changes and needs to be tested very heavy before releasing it to the people

icinga-migration · 2011-04-27T19:45:06Z

Updated by mfriedrich on 2011-04-27 19:45:06 +00:00

i would split that - one for the current fixes (finding all table relations, or simply replacing the too small ints all over in mysql) and then a complete review.

i can live with a quickfix on the size, people will do the same - but larger changes need time and testing stages, right. so if you could create another bunch of replacing/alter statements, those still using the old int definitions, it would be great.

icinga-migration · 2011-04-27T20:08:11Z

Updated by Tommi on 2011-04-27 20:08:11 +00:00

sql itself is not the problem. But if you are using a 64bit id in dbqueries.c as current with asprintf("%lu",(unsigned long) data[0],.. we will introduce more problems. finding and changing all these code lines will cause the most effort.

icinga-migration · 2011-05-05T15:08:41Z

Updated by Tommi on 2011-05-05 15:08:41 +00:00

File added mysql_alter_icinga13_numbers.sql
File added pgsql_alter_icinga13_types.sql
File added oracle_alter_icinga13_numbers.sql
Category changed from 57 to 24
Done % changed from 20 to 30

sql prepared for V1.4. Attached Scripts should be run as part of Version update. Previous posted scripts mysql_alter.sql and mysl_new.sql are obsolete (cant be removed from here)

icinga-migration · 2011-05-05T15:08:47Z

Updated by Tommi on 2011-05-05 15:08:47 +00:00

File deleted mysql_alter.sql

icinga-migration · 2011-05-05T15:08:52Z

Updated by Tommi on 2011-05-05 15:08:52 +00:00

File deleted mysql_new.sql

icinga-migration · 2011-05-10T06:19:25Z

Updated by mfriedrich on 2011-05-10 06:19:25 +00:00

http://libdbi.sourceforge.net/docs/programmers-guide/reference-field.html

icinga-migration · 2011-05-11T18:51:47Z

Updated by tgelf on 2011-05-11 18:51:47 +00:00

I'm not sure whether it was a good idea to change instance_id from SMALLINT to BIGINT...

icinga-migration · 2011-05-11T19:08:56Z

Updated by Tommi on 2011-05-11 19:08:56 +00:00

Did you found problems with this setting?
From now there is a domain "ID", defined as uint64, which will applied to all _ID fields in all rdbms in the same matter.
I know in the most installations we spend 3 bytes zeros for instance_id, but other rdbms are not aware such small number types (and today servers will not have problems even with millions of rows:-))).

icinga-migration · 2011-05-12T12:34:35Z

Updated by tgelf on 2011-05-12 12:34:35 +00:00

Well... no issue right now. At least nothing but a user having had big trouble with Icinga 1.4 minus two days he run into BIGINTs fetched with get_uint :p This has been fixed by dnsmichi prior to the final 1.4 release.

Call me conservative, but I'm usually trying to keep indexed columns as small as possible. And I see absolutely no need for more than 2^16 Icinga instances - never seen an installation even nearly as large as that. And if so: wouldn't 2^32 be large enough? MySQL is a very special case, other RDBMs have just a single numeric data type and that's it. MySQL does far less automagic work to optimize indexes and data structures. I blindly guess that MySQL is also unable to use many indexes applied to the IDO db right now. It can rarely combine multiple indexes, and things are even more tricky once JOINs, GROUP_BYs and ORDERs are involved.

Unfortunately (and obviously, what else should they do) Icinga-Web is making intensive use of all this features and doesn't really care about query performance. NDO's database structure has IMO not been designed with performance in mind, and using 4 times the space formerly reserved for instance_id will definitively not help to improve performance. And as most tables are referencing instance_id (IMO not a good design choice either, but that could be discussed elsewhere) there is far more than just a single index growing as of this "small change".

My concerns are not storage- but memory-related: larger indexes also mean more wasted buffer space, caches filling up earlier and so on. These are sideeffects that could lead to earlier temp table operations - and therefore cause more I/O. Once again: no concrete issue at all. But why shall we waste ressources with no need for doing so at all? Saying "Hey, we changed one INT to BIGINT, let's change all the others too" is an excuse I personally would never accept.

I have been running MySQL clusters with hundreds of millions of rows - and with pretty large keys. That's not an issue at all. At least not unless your queries allow your database to operate on indexed columns only. As soon they don't: sit back and relax... it could take a few minutes to serve your request ;-)

Regards,
Thomas Gelf

icinga-migration · 2011-05-13T12:28:36Z

Updated by mfriedrich on 2011-05-13 12:28:36 +00:00

tgelf wrote:

Well... no issue right now. At least nothing but a user having had big trouble with Icinga 1.4 minus two days he run into BIGINTs fetched with get_uint :p This has been fixed by dnsmichi prior to the final 1.4 release.

as said on irc, people not telling using git, but blaming the release are hard to support.
even using a git version in production during the feature freeze of 2 weeks might the need of the question - "why?"

so this is more or less a user issue.

Call me conservative, but I'm usually trying to keep indexed columns as small as possible. And I see absolutely no need for more than 2^16 Icinga instances - never seen an installation even nearly as large as that. And if so: wouldn't 2^32 be large enough? MySQL is a very special case, other RDBMs have just a single numeric data type and that's it. MySQL does far less automagic work to optimize indexes and data structures. I blindly guess that MySQL is also unable to use many indexes applied to the IDO db right now. It can rarely combine multiple indexes, and things are even more tricky once JOINs, GROUP_BYs and ORDERs are involved.

the thing i don't get is, why one demands a change, the other one reflects that change after being released. i thought that you guys had a talk about that before actually demanding the change as important.

Unfortunately (and obviously, what else should they do) Icinga-Web is making intensive use of all this features and doesn't really care about query performance. NDO's database structure has IMO not been designed with performance in mind, and using 4 times the space formerly reserved for instance_id will definitively not help to improve performance. And as most tables are referencing instance_id (IMO not a good design choice either, but that could be discussed elsewhere) there is far more than just a single index growing as of this "small change".

well icinga-web should actually care about query performance. joining this, where'ing that, grouping that isn't always the best thing to do, even with a unified api on the queries (doctrine).

the database design is ready for fast inserts, but yet keeping active/inactive config in place. most of the actions done affects one single table, only referenced configs will require more action.

My concerns are not storage- but memory-related: larger indexes also mean more wasted buffer space, caches filling up earlier and so on. These are sideeffects that could lead to earlier temp table operations - and therefore cause more I/O. Once again: no concrete issue at all. But why shall we waste ressources with no need for doing so at all? Saying "Hey, we changed one INT to BIGINT, let's change all the others too" is an excuse I personally would never accept.

and you're not committing a patch either, so this decision is left to those actually doing it. in that special case, thomas did it and i trust his expertise to decide

a. is it worth making an exception for just one id
b. does the performance lack of that change for just this id

if there would have been such a patch in the first place, telling exactly which ids might be affected and are important, things will be a lot easier.

I have been running MySQL clusters with hundreds of millions of rows - and with pretty large keys. That's not an issue at all. At least not unless your queries allow your database to operate on indexed columns only. As soon they don't: sit back and relax... it could take a few minutes to serve your request ;-)

as usual - patches welcome. theoratically it could be slower. show us performance diagrams between 1.3.1 and 1.4.0 and mark those significants disadvantages you're talking about.

icinga-migration · 2011-05-13T13:21:16Z

Updated by tgelf on 2011-05-13 13:21:16 +00:00

dnsmichi wrote:

the thing i don't get is, why one demands a change, the other one reflects that change after being released. i thought that you guys had a talk about that before actually demanding the change as important.

I'm not "you guys". I accidentally stumbled over this change as of the previously mentioned pre-1.4 git issue. And yes, I was so keen to put this change into question. I have not been part of the discussion mentioned in this ticket - but from the information I'm able to gather from the comments available right here, no one ever requested to change instance_id.

It hasn't been changed unless a few days before 1.4, as far as I could find out at first only object_id's have been touched. But then (for whatever reason) all INTs have been changed - and that's what I'm unhappy with.

the database design is ready for fast inserts, but yet keeping active/inactive config in place. most of the actions done affects one single table, only referenced configs will require more action.

That's true. But don't expect me to be a big fan of a database designed to fit a specific program's internal data structures just to speed up INSERTs. Throwing more INDEXes in will not help frontend developers to achieve better performance. But let's discuss that somewhere else - it's off topic.

and you're not committing a patch either, so this decision is left to those actually doing it. in that special case, thomas did it and i trust his expertise to decide

A patch? Rolling back only those changes involving instance_id would suffice :p

a. is it worth making an exception for just one id

IMO: yes. Every single change to a db schema should be questioned and evaluated individually.

b. does the performance lack of that change for just this id

It would definitively not get better, but chances are good that it could become worse. This strongly depends on how much RAM for MySQL buffers you are going to throw in.

if there would have been such a patch in the first place, telling exactly which ids might be affected and are important, things will be a lot easier.

That's probably addressed to those who started the discussion / opened the ticket. IMO not wiping/refilling the DB on every Icinga reload would also have solved their problems. If those raising this issue did not specify which ID they where referring to they could also have been asked for such details. But if so: why has it been done correctly first? It's the deciscion to "change all ints" that I'm unhappy with - and this happened after the first attempt to fix what has been requested.

as usual - patches welcome. theoratically it could be slower. show us performance diagrams between 1.3.1 and 1.4.0 and mark those significants disadvantages you're talking about.

As soon as Icinga-Web is offering some profiling instruments I'd be happy to do so. Today I shipped a patch reducing Icinga-Web's web traffic to a fraction of what it has been before. That's far more than just my two Eurocent for today ;-)

Cheers,
Thomas

icinga-migration · 2011-05-19T09:15:58Z

Updated by mfriedrich on 2011-05-19 09:15:58 +00:00

Done % changed from 30 to 90

as said, feel free to provide a suggestion/patch what exactly needs to be changed. and make sure checking the c sources too, as the instance_name=>instance_id select happens in the first place, and by putting uint instead of ulonglong e.g. will result in null values all over there place.

so if the instance_id being bigint is really a problem, someone provide a patch. i don't see a magnificant severity to change that myself.

icinga-migration · 2011-05-19T10:39:25Z

Updated by tgelf on 2011-05-19 10:39:25 +00:00

Well... Asking someone for a patch to roll back a modification recently done by yourself is brashly impertinent, isn't it? I'm not interested in such pointless discussions, so please feel free to close this issue. I have had really great times at Kindergarten, but that's been a very long time ago.

In my opinion this is still a risky modification, it hasn't been intensively tested and right now probably no one really knows if third party tools (Icinga-Web, Nagvis) working with the IDO database are able to handle all those new 64bit Integers correctly. Let me sum it all up, just for the sake of completeness:

Intensive tests are not required prior to such changes, especially not if they occur within time periods thought for feature freezes
Performance impacts of such changes aren't worth to be evaluated bevore releasing them unless someone proves that things got worse after a new stable release
Third party tools have to find out the hard way that things have been changed. There is no need to ask at least the most known ones for feedback first. PHP tools are believed to behave fine with values greater than 2^31.
Inofficial strategy is "change everything to BIGINT and let's hope that no one ever reaches such large INTs"
Special upgrade warnings in the documentation as been there for 1.0 are not required. Everything will be fine if people change db structure while the former ido2db version is still running. There will be no issue with doubled entries (instance 0 VS instance 1) at all, in no potential scenario.

What else shall I say? Go for it, live with it. Don't care about hopeless pessimists like me. I really didn't want to waste your time and feel sorry if I did so.

Regards,
Thomas

icinga-migration · 2011-05-19T11:07:26Z

Updated by mfriedrich on 2011-05-19 11:07:26 +00:00

Status changed from Feedback to Closed
Target Version deleted ~~1.5~~

for gods sake i did only approve those diffs, but not implement them. prior testing would have been much appreciated in the first place. did it happen? no. everyone awaits the release, then applies the upgrade path, and then the issues get updated.

i for myself tested those changes on the rdbms, also tested them with icinga-web from git during the feature freeze test phasis. i'm regularly doing this to keep quality assurance safe before a release.

if you don't feel comfortable to support us in that way of testing, but reporting such things afterwards, then please don't use icinga with idoutils. go use ancient ndoutils instead, it might be compatible and still supports the old fashioned mysql schema like it was 2 years ago.

this is not what i do expect from viral development. and like you said, it's a waste of time discussing about such a behavior. i will close this issue for now, if there are any complaints about

performance
index problems
etc

then open a new issue providing

test cases, queries
performance tests
detailed problem descriptions

over and out.

icinga-migration · 2014-12-08T14:59:32Z

Updated by postmaster@digimarc.com on 2014-12-08 14:59:32 +00:00

Delivery has failed to these recipients or groups:

brian+icinga@attributor.com<mailto:brian%2Bicinga@attributor.com>
The e-mail address you entered couldn't be found. Please check the recipient's e-mail address and try to resend the message. If the problem continues, please contact your helpdesk.

Diagnostic information for administrators:

Generating server: digimarc.com

[dev.icinga.com #1173] int(11) to small for some of ido tables #538

[dev.icinga.com #1173] int(11) to small for some of ido tables #538

Comments

icinga-migration commented Jan 31, 2011

icinga-migration commented Jan 31, 2011

icinga-migration commented Jan 31, 2011

icinga-migration commented Jan 31, 2011

icinga-migration commented Jan 31, 2011

icinga-migration commented Feb 1, 2011

icinga-migration commented Mar 14, 2011

icinga-migration commented Mar 16, 2011

icinga-migration commented Mar 24, 2011

icinga-migration commented Mar 27, 2011

icinga-migration commented Mar 27, 2011

icinga-migration commented Apr 13, 2011

icinga-migration commented Apr 13, 2011

icinga-migration commented Apr 13, 2011

icinga-migration commented Apr 15, 2011

icinga-migration commented Apr 27, 2011

icinga-migration commented Apr 27, 2011

icinga-migration commented Apr 27, 2011

icinga-migration commented Apr 27, 2011

icinga-migration commented May 5, 2011

icinga-migration commented May 5, 2011

icinga-migration commented May 5, 2011

icinga-migration commented May 10, 2011

icinga-migration commented May 11, 2011

icinga-migration commented May 11, 2011

icinga-migration commented May 12, 2011

icinga-migration commented May 13, 2011

icinga-migration commented May 13, 2011

icinga-migration commented May 19, 2011

icinga-migration commented May 19, 2011

icinga-migration commented May 19, 2011

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014

icinga-migration commented Dec 8, 2014