[dev.icinga.com #13143] Automation sync jobs order

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/13143

Created by dzalewski on 2016-11-10 14:37:35 +00:00

Assignee: dzalewski
Status: Feedback
Target Version: (none)
Last Update: 2016-11-10 16:03:29 +00:00 (in Redmine)

---

Hi,

I have 4 sync jobs in Director pulling hosts, host groups, user and user groups from CMDB (iTop) via SQL query.

Hosts sync job won't be able to assign a host to a host group if its not there already. I have to first run host group sync jobs and then hosts sync job.

It would be nice if we were able to set sync jobs order or perhaps dependencies on other sync jobs.

I'm not sure if this the most elegant solution. Is there is alreadya better way of solving this issue?

Thanks

Dominik

---

Relations:

• relates #13145

[icinga-migration]

Updated by dzalewski on 2016-11-10 14:55:10 +00:00

Perhaps a job could have an option to sync both hosts and hostgroups object at once?

[icinga-migration]

Updated by tgelf on 2016-11-10 15:03:14 +00:00

First, order doesn't really matter. There could always be a race condition where a related group is created in between two import runs. One of the related syncs will fail, will succeed the next time, so what. I could hardcode a specific order to solve that, like datalist -> hostgroup -> host -> service and similar and this would work for most scenarios just out of the box. Eventually we could allow to manually tweak this order, but I think it isn't even worth the effort.

Syncing at once would make a couple of things easier, but has one big disadvantage: the transaction runs longer and there is much more memory I cannot free in the meantime. Could become deadly in large environments. Hostgroups wouldn't make a difference, but with the same argumentation I'd be required to put services into the mix, and then... well.

Would a hardcoded object-type based execution order solve your issue?

[icinga-migration]

Updated by tgelf on 2016-11-10 15:04:04 +00:00

• Status changed from New to Feedback
• Assigned to set to dzalewski

[icinga-migration]

Updated by dzalewski on 2016-11-10 15:27:05 +00:00

"One of the related syncs will fail, will succeed the next time"

I agree its not end of the world. I don't think it you can see failed sync in the job history correct?

"Would a hardcoded object-type based execution order solve your issue?"

Could you please elaborate more?

Thanks

Dominik

[icinga-migration]

Updated by tgelf on 2016-11-10 15:35:59 +00:00

dzalewski wrote:

> "One of the related syncs will fail, will succeed the next time"
I agree its not end of the world. I don't think it you can see failed sync in the job history correct?

Correct. However, we could easily add this. Just: a job scheduled to run once a minute could then generate 200k entries on a single weekend. So, not sure on this.

> "Would a hardcoded object-type based execution order solve your issue?"
Could you please elaborate more?

Instead of providing an option allowing you to sort your rules and forcing every user to figure out the right order for himself, I'd prefer to tell Director to always run Datalist imports first, then Zones, Endpoints, XyGroups, Commands, Hosts, Services - all in their "natural" order. Hosts have no dependency on Services but on Commands and Groups - and so on. There is no loop condition here, the core acts similar when building it's running config at startup time.

[icinga-migration]

Updated by dzalewski on 2016-11-10 15:56:35 +00:00

Syncing all in their "natural" order sounds good!

Absolutely 200k entries in a history won't be good. It is probably outside of this topic but how would you then know if a sync has failed let say because database resource isn't available? Will it be logged by icingaweb2 or director in db or a logfile? Just trying to think how would one monitor their director syncs are working. Thanks.

[icinga-migration]

Updated by tgelf on 2016-11-10 16:03:29 +00:00

dzalewski wrote:

Syncing all in their "natural" order sounds good!

Absolutely 200k entries in a history won't be good. It is probably outside of this topic but how would you then know if a sync has failed let say because database resource isn't available? Will it be logged by icingaweb2 or director in db or a logfile? Just trying to think how would one monitor their director syncs are working. Thanks.

It should be logged to the Icinga Web 2 log. But while job startup is logged, seems that failure isn't - just overall failures related to "all the jobs" are, but those should be rare. Opened #13145 for this. When a Job continues to fail this is perfectly visible in the GUI, but as soon as it recovers this information vanishes.

[icinga-migration]

Updated by tgelf on 2016-11-10 16:03:38 +00:00

• Relates set to 13145

[Thomas-Gelf]

There where some bugs related to persisting job errors, this should have been improved with v1.7.0. No plans for hard-coded "ordering" right now.