[dev.icinga.com #9208] Monitoring Module does not take multiple instance names into account

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/9208

Created by mfriedrich on 2015-04-30 15:50:12 +00:00

Assignee: (none)
Status: Closed (closed on 2016-10-05 07:52:43 +00:00)
Target Version: (none)
Last Update: 2016-10-05 07:52:43 +00:00 (in Redmine)

---

General problem

It seems that Icinga Web 2 is generally not aware of instances (instance_name, instance_id) and only shows one active instance.

I do have two instances configured (1.x and 2.x) but only the 2.x instance is taking into account for some strange reason.

Please explain how this is supposed to be working, and fix it accordingly to support multiple instances inside one (database/livestatus) backend.

Monitoring health summary

Only shows one instance.

MariaDB [icinga]> select i.instance_name, ps.program_version, ps.status_update_time from icinga_programstatus ps join icinga_instances i on ps.instance_id=i.instance_id;
+---------------+--------------------+---------------------+
| instance_name | program_version    | status_update_time  |
+---------------+--------------------+---------------------+
| icinga-dev    | 1.11.8             | 2014-10-17 18:13:35 |
| default       | v2.3.3-63-gbb59065 | 2015-04-30 17:17:31 |
+---------------+--------------------+---------------------+
2 rows in set (0,00 sec)

The instance_id cannot be filtered either.

UI merges all instances

The web interface does show all hosts and services from both instances, even if 'icinga-dev' (my 1.x instance) is not running. The monitoring health view does not take that into account either.

MariaDB [icinga]> select i.instance_name, count(*) from icinga_instances i join icinga_servicestatus ss on i.instance_id=ss.instance_id join icinga_objects oss on ss.service_object_id=oss.object_id where oss.is_active=1 group by i.instance_id;
+---------------+----------+
| instance_name | count(*) |
+---------------+----------+
| default       |      112 |
| icinga-dev    |      104 |
+---------------+----------+
2 rows in set (0,09 sec)

Commands

When sending a command through the detail views, it should take the instance_id and instance_name into account and then select the command pipe by instance name. This can be configured through the gui/config but currently does not match. So it seems this is ignored.

Code

Seems it is not there.

modules/monitoring/library/Monitoring/Backend/Ido/Query/ContactgroupQuery.php:            | service_contactgroup_id | instance_id | service_id | contactgroup_object_id |
modules/monitoring/library/Monitoring/Backend/Ido/Query/CommandQuery.php:            'command_instance_id'   => 'c.instance_id',
modules/monitoring/library/Monitoring/DataView/Command.php:            'command_instance_id',

Attachments

• Auswahl_164.png mfriedrich - 2015-04-30 15:40:43 +00:00
• Auswahl_165.png mfriedrich - 2015-04-30 15:41:15 +00:00
• Auswahl_166.png mfriedrich - 2015-04-30 15:45:27 +00:00
• Auswahl_167.png mfriedrich - 2015-04-30 15:52:37 +00:00
• Auswahl_168.png mfriedrich - 2015-04-30 15:55:01 +00:00

---

Subtasks:

• Feature 9943 - Offer instance_name as query column
• Feature 9945 - Show instance_name in a host's and service's detail view
• Feature 9651 - Automatically use the correct instance configuration based on a host's or service's instance
• Feature 9213 - Make Monitoring Health View multiple backend aware

Relations:

• relates #9208
• relates #9208

[icinga-migration]

Updated by mfriedrich on 2015-04-30 15:53:08 +00:00

• File added Auswahl_167.png

In Icinga Web 1.x this is visible through the program status cronk and a dedicated instances view.

PS: Duplicated comments, objects, etc might be related to that as well.

[icinga-migration]

Updated by mfriedrich on 2015-04-30 15:55:11 +00:00

• File added Auswahl_168.png

And a filtered view with the instance name column in Web 1.x

[icinga-migration]

Updated by mfriedrich on 2015-04-30 15:56:32 +00:00

• Relates set to 9207

[icinga-migration]

Updated by tgelf on 2015-05-01 10:04:05 +00:00

• Status changed from New to Feedback
• Assigned to set to mfriedrich
• Priority changed from High to Normal

Hi Michael,

I'd like to let this open / use this ticket for farther discussion if you agree. Our plans are to silently fade out / deprecate support for multiple instances. This is an ancient feature thought to solve the "I have multiple monitoring nodes and want to see all objects in one place". For environments with multiple instances we are trying to hide the fact that there are such. As you noted, all objects are merged, we try to do as if there was only one instance in a least-effort way.

In practice "instances" had a couple of issues:

• They lead to a very inefficient DB schema, with instance_id involved in multiple places all over the schema. We had (and have) lots of inefficient indexes. As object_id is unique this was completely pointless. However it is nothing we can easily change right now.
• First symptom of misunderstood "instances" are multiple hosts with the same name, with "localhost" being the most prominent one.
• Multiple instances where thought for "one ndo2db daemon, forking multiple childs for multiple nagios instances". Your example showing Icinga 1.x and Icinga 2 in the very same DB is technically working fine, but a pretty courageous interpretation of that design. IMO you should use multiple database backends for this, at least as soon as Icinga Web 2 supports them (see below). Your experiment is only working thanks to the fact that you are "forcefully" syncing the IDO schema for 1.x and 2 in the respective projects. No way to run let's say Icinga 1.9 and 2.3 together or to upgrade them independently - this will not happen "in the wild".
• Seen from a DBA point of view we have a frightening history of relaxing schema constraints instead of fixing root causes of duplicate items - we definitively must get better in that discipline. Our instance_ids are not a great help to approach that goal, one more reason to fade them out. Especially as we have far better options for distributed environments with Icinga 2.

I completely agree with you in two of your major points: IF there are multiple instances AND you configured multiple "command instances", they MUST be respected when sending commands. And "monitoring health" MUST be aware of multiple instances. If there are no issues for those two problems yet we should create them. My love and support for multiple instances ends here.

From a technical point of view in Icinga Web 2 it would be easier as ever before to provide a correct and complete "multiple instance implementation". However I'm currently not willing to do so. If you have a good real-world usecase for them please let me know, I'm always open for new input enlarging my horizon. But let me first explain our farther plans:

• "instance" (or instance_name) should be available as a column in our most prominent views, that's something we are more or less aware of, but no one is working of this I guess. This would allow one to filter for instances and use the instance name with &addColumns=instance. We could also show the instance name in our detail views, but only if there are multiple ones in your DB. This is a candidate for one or two new issues on this tracker I guess.
• Icinga Web 2 allows one to configure multiple backends, but they are currently unused, as Web 2 will support so-called "environments". An environment consists of a backend (db, livestatus, combo), a command instance (local or remote, "api" in the near future), and other components provided by modules, such as graphers, issue trackers and more
• We will not show objects from multiple environments in single list views, but you will be allowed to combine them on your dashboards. All links will behave as expected. Combining multiple sources in lists might be technically possible, but I'll refuse to implement such as there is no way to do so in an efficient way. We MIGHT add exceptions for special use cases, like "show all problems from all backends in one list". Even this relaxed versionis something I'd like to avoid.

To sum it all up: we well support multiple backends, we do not like "instances" as they exist right now. We must support existing ones to get them working as expected (commands) and to highlight issues in such environments (health). More related ideas and suggestions for improvement are highly appreciated.

Cheers,
Thomas

[icinga-migration]

Updated by mfriedrich on 2015-05-01 11:01:27 +00:00

• Assigned to changed from mfriedrich to tgelf

Hi Tom,

I'm totally fine with this explanation. I was just a little irritated when discussing that with Alex (basically the monitoring health overview showing the instance health) not finding this implemented afterall. Having two or more instances inside the database makes the data select in the monitoring health confusing. Although this should be hidden from the users - the only thing I'd like to see is the version of the backend (helps with problem support).

From a technical point of view I fully agree that instance_id generates quite a lot of bloated implementation and problematic sql queries. Especially when IDO connects, selecting the right instance_id based on its configured name. 99% of the users never changed the instance_name from 'default' to something different, only in a multiple instances scenario. Or being a developer testing all capabilities wiht Core 1.x and Web 1.x.

My primary concern is still that we've been building distributed environments based on that instance_id and instance_name columns in the past with Icinga Web 1.x. So I guess there exist certain installations with this kind of data integrity where instance_id is the only qualifier next to object_id to identify unique objects. Maybe not at customer projects (I've found different distributed assets there in the past 2 years) but generally speaking this was in the documentation mentioned all over. My guess is that users migrating from Web 1.x to Web 2.x will encounter this kind of problem somehow.

So if that's the general strategy to go with multiple databases, please explain that in the documentation once this feature is fully implemented (no idea combo backends currently work). And it would certainly be nice to filter by "backend" then. Further add a note somewhere that instance_id is generally replaced by multiple backends and explain that over there too. A guide similar to the "Icinga 1.x vs Icinga 2.x" migration documentation which already exists will certainly help users (and colleagues) to help prepare a clean migration.

I don't expect that to happen now though, but it would be nice to have such docs for the final 2.0.0 release somewhere :)

PS: Multiple backends support was a long demanded feature for Web 1.x which was never fully implemented, and also confused with instance_id support (ask Jannis about his nightmares in #892).

Kind regards,
Michael

[icinga-migration]

Updated by tgelf on 2015-05-01 13:24:26 +00:00

• Assigned to changed from tgelf to mfriedrich

Hi Michael,

great to hear that. As stated above, "health" MUST be instance-aware, so if it is not and there is no such issue it would be great if you could create one. I hope that our current implementation (given that we get multiple command instances working correctly together with a single idb connection) will satisfy most if not all existing web 1.x users.

There will be no "filtering by backend" I guess as there will be no single view showing data from multiple backends. List views will of course have to provide some knob allowing you to switch environment. What we could also implement would be hard-coded filters per backend (implemented exactly like permissions, so not much coding involved) allowing you to pre-filter provided data.

That way you could create multiple environments for multiple "legacy instances", each of them pre-filtered by instance. Not sure why one might want to do so, but it could be a workaround for those who believe to really need this feature.

How to go on with this issue? I guess we should make sure to create dedicated issues for the remaining tasks, eventually link them to this one for farther information - and close this ticket as soon as all the others have been created.

Cheers,
Thomas

[icinga-migration]

Updated by mfriedrich on 2015-05-01 13:37:48 +00:00

• Relates set to 9212

[icinga-migration]

Updated by mfriedrich on 2015-05-01 13:42:27 +00:00

• Relates set to 9213

[icinga-migration]

Updated by mfriedrich on 2015-05-01 13:45:44 +00:00

• Relates set to 9214

[icinga-migration]

Updated by mfriedrich on 2015-05-01 13:47:13 +00:00

• Assigned to changed from mfriedrich to tgelf

Hi Tom,

tgelf wrote:

great to hear that. As stated above, "health" MUST be instance-aware, so if it is not and there is no such issue it would be great if you could create one. I hope that our current implementation (given that we get multiple command instances working correctly together with a single idb connection) will satisfy most if not all existing web 1.x users.

If the command backends are linked to data backend resources, this should be fine. Afaik this check is implemented, and there also are some references to instance_id but it looks incomplete.

For monitoring health itself see #9213 ... I did not update #9207 as this is only the program version which probably works fine.

One thing which would make sense - check if there are multiple rows in the icinga_instances table, and add a warning during the setup wizard/monitoring health. See #9214

There will be no "filtering by backend" I guess as there will be no single view showing data from multiple backends. List views will of course have to provide some knob allowing you to switch environment. What we could also implement would be hard-coded filters per backend (implemented exactly like permissions, so not much coding involved) allowing you to pre-filter provided data.

Sounds good to me :)

That way you could create multiple environments for multiple "legacy instances", each of them pre-filtered by instance. Not sure why one might want to do so, but it could be a workaround for those who believe to really need this feature.

Well, it will help sort out issues if there's dead data in one of the interfaces. This is more of a maintenance thought, as we all know that monitoring environments see different admins, and at some point it might get confusing. It's not necessarily something we should have with 2.0.0 but would add a little more convenience on troubleshooting. But I would wait with that until users made their experience with that feature before actually implementing this visible to the user.

How to go on with this issue? I guess we should make sure to create dedicated issues for the remaining tasks, eventually link them to this one for farther information - and close this ticket as soon as all the others have been created.

Works for me :) Others having the same "problem" might just find this issue and the entitled discussion/solution,

Kind regards,
Michael

[icinga-migration]

Updated by jmeyer on 2015-05-05 11:27:55 +00:00

• Relates deleted 9207

[icinga-migration]

Updated by elippmann on 2015-07-15 12:17:37 +00:00

• Status changed from Feedback to New
• Assigned to deleted tgelf
• Target Version set to 2.0.0

[icinga-migration]

Updated by jmeyer on 2015-07-16 13:46:17 +00:00

• Relates set to 9651

[icinga-migration]

Updated by jmeyer on 2015-08-18 11:45:19 +00:00

• Relates deleted 9651

[icinga-migration]

Updated by jmeyer on 2015-08-18 11:46:20 +00:00

• Relates deleted 9213

[icinga-migration]

Updated by elippmann on 2015-10-01 21:44:18 +00:00

• Target Version changed from 2.0.0 to 273

[icinga-migration]

Updated by elippmann on 2015-11-20 13:10:57 +00:00

• Target Version changed from 273 to Backlog

[icinga-migration]

Updated by elippmann on 2016-10-05 07:52:43 +00:00

• Status changed from New to Closed
• Target Version deleted Backlog

Won't fix.