[dev.icinga.com #11419] Config parser should not log names of included files by default

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/11419

Created by tgelf on 2016-03-19 22:28:23 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2016-07-05 13:45:10 +00:00)
Target Version: 2.5.0
Last Update: 2016-08-22 11:48:32 +00:00 (in Redmine)

Backport?: Not yet backported
Include in Changelog: 1

---

System description:

• mid-sized, 40k monitored objects
• IDO knows 860 scheduled downtimes
• more than twice as many (?!) files in _api (could be expired ones, I didn't check)
• fqdn names are not that short, could be 50+ chars long (but never 64+)

Problem: "information/ConfigCompiler: Compiling config file: ..." lines make up more than half a megabyte of data at each restart. And I didn't even add those nasty "warning/ConfigObject: Ignoring config object ... due to errors" lines to that number.

We should either find a more efficient way to store that kind of data (read: one single file for all of them) or get rid of those log lines.

Cheers,
Thomas

Changesets

2016-07-05 13:43:48 +00:00 by mfriedrich 915ebe1

ConfigCompiler should not log included files by default

fixes #11419

[icinga-migration]

Updated by mfriedrich on 2016-03-21 08:04:21 +00:00

• Duplicates set to 10717

[icinga-migration]

Updated by mfriedrich on 2016-03-21 08:10:56 +00:00

• Status changed from New to Rejected

Hi Tom,

we already have an issue for that :)

Kind regards,
Michael

[icinga-migration]

Updated by tgelf on 2016-03-21 08:18:51 +00:00

• Status changed from Rejected to Feedback
• Assigned to set to mfriedrich

Hi Michael,

I'm perfectly aware of #10717, but please re-evaluate this issue. It targets another problem, I'm talking about systems running with hundreds or even thousands valid and non-expired comments or downtimes. Erroneous ones like those in #10717 make it worse, but this problem wouldn't change once that is fixed.

Thanks,
Thomas

[icinga-migration]

Updated by mfriedrich on 2016-03-21 19:44:12 +00:00

• Duplicates deleted 10717

[icinga-migration]

Updated by mfriedrich on 2016-03-21 19:46:28 +00:00

• Category set to Configuration
• Status changed from Feedback to New
• Assigned to deleted mfriedrich
• Parent Id set to 11312

Discussed it offline. The problem is with a lot of comment/downtime objects put into separate files, causing a long list of 'compiling file' entries. Though that's the default output of the config compiler - it might be a discussion point to filter that away somehow for the API packages startup.log file.

[icinga-migration]

Updated by tgelf on 2016-03-21 20:00:22 +00:00

Hi Michi,

if it was only for the API I wouldn't care. I could perfectly filter away that noise before storing to the Director DB. But such massive log outputs might influence a lot of other components:

• tools using systemd output
• loggers
• Config validation driven by configuration management

We probably do not know where all this data is going to finish. To me that's a good reason to generate as less noise as possible. Several thousand lines for a reload seems a little bit too much to me. This way the related (startup) log data written to disk is larger than the real comment/downtime payload.

Cheers,
Thomas

[icinga-migration]

Updated by mfriedrich on 2016-07-05 13:38:35 +00:00

• Tracker changed from Bug to Feature
• Subject changed from _Insane startup log size as of downtimes in api package to ConfigCompiler should not log included files by default
• Status changed from New to Assigned
• Assigned to set to mfriedrich
• Target Version set to 2.5.0

One thing which feels much more comfortable - hide loading libraries and including each file into the "notice" log level. Adding an additional "loading config files" log entry to let the user know when it starts. With the change in #12098 you'll only have certain warnings for unreadable files, but won't have to visually grep between "Information" and "Warning".

mbmif /usr/local/icinga2/etc/icinga2/conf.d (master) # icinga2 daemon -C
information/cli: Icinga application loader (version: v2.4.10-513-g63f83e8; debug)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: mbmif.int.netways.de
warning/ApplyRule: Apply rule 'satellite-host' (in /usr/local/icinga2/etc/icinga2/conf.d/satellite.conf: 29:1-29:41) for type 'Dependency' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freef:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freeg:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freeh:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freei:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freej:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freek:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freel:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freem:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freen:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freeo:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freep:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freeq:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freer:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-frees:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freet:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freeu:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freev:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freew:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freex:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freey:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'wmi-disk-freez:' (in /usr/local/icinga2/etc/icinga2/tests/for-apply.conf: 4:9-4:60) for type 'Service' does not match anywhere!
information/ConfigItem: Instantiated 1 FileLogger.
information/ConfigItem: Instantiated 1 Endpoint.
information/ConfigItem: Instantiated 3 Zones.
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 3 Comments.
information/ConfigItem: Instantiated 1 ScheduledDowntime.
information/ConfigItem: Instantiated 2 Dependencies.
information/ConfigItem: Instantiated 170 Services.
information/ConfigItem: Instantiated 4 UserGroups.
information/ConfigItem: Instantiated 5 ServiceGroups.
information/ConfigItem: Instantiated 1 Downtime.
information/ConfigItem: Instantiated 9 TimePeriods.
information/ConfigItem: Instantiated 19 Users.
information/ConfigItem: Instantiated 68 Hosts.
information/ConfigItem: Instantiated 10 HostGroups.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 175 CheckCommands.
information/ConfigItem: Instantiated 5 NotificationCommands.
information/ConfigItem: Instantiated 29 Notifications.
information/ConfigItem: Instantiated 1 StatusDataWriter.
information/ConfigItem: Instantiated 1 ExternalCommandListener.
information/ConfigItem: Instantiated 3 ApiUsers.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ConfigItem: Instantiated 1 LivestatusListener.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ScriptGlobal: Dumping variables to file '/usr/local/icinga2/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).

[icinga-migration]

Updated by mfriedrich on 2016-07-05 13:45:10 +00:00

• Status changed from Assigned to Resolved
• Done % changed from 0 to 100

Applied in changeset 915ebe1.

[icinga-migration]

Updated by tgelf on 2016-07-05 14:03:16 +00:00

@dnsmichi: I know that you are used to modify subjects to obtain a good-looking changelog, allowing you to benefit from the "Include in Changelog" flag.

However, please consider opening a dedicated ticket for that purpose when modifying the subject of a user-generated issue to something completely different than initially requested. When someone doesn't get what he wants that's something he has to live with. But when history overview suggests that he has been in some way responsible for a change he didn't ask for, he could remain ... well ... not amused.

[icinga-migration]

Updated by mfriedrich on 2016-08-08 11:21:00 +00:00

• Parent Id deleted 11312

[icinga-migration]

Updated by gbeutner on 2016-08-22 11:48:32 +00:00

• Subject changed from ConfigCompiler should not log included files by default to Config parser should not log names of included files by default