[dev.icinga.com #9983] Replay logs consume all threads

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/9983

Created by aledermueller on 2015-08-24 07:57:42 +00:00

Assignee: (none)
Status: Closed (closed on 2015-10-14 12:47:37 +00:00)
Target Version: (none)
Last Update: 2015-10-14 12:47:37 +00:00 (in Redmine)

Icinga Version: v2.3.8
Backport?: Not yet backported
Include in Changelog: 1

---

Hi,

replaying the logs causes icinga2 to consume 100% cpu and the process hangs forever. A stacktrace of the threads is attached.

Clients (Zones): 400
log_duration client: 15m

Master (zone with 2 endpoints)
log_duration masters: 6h (approx. 10GB of logfiles)

Thanks, Achim

Attachments

• debug aledermueller - 2015-08-24 07:48:42 +00:00 - Stacktrace threads

---

Relations:

• relates #9976
• relates #9983
• relates #9730
• relates #10002

[icinga-migration]

Updated by mfrosch on 2015-08-24 08:03:56 +00:00

• Relates set to 9976

[icinga-migration]

Updated by mfrosch on 2015-08-24 08:49:01 +00:00

• Relates set to 9986

[icinga-migration]

Updated by mfrosch on 2015-08-24 08:50:12 +00:00

We try to fix this with #9986

[icinga-migration]

Updated by mfrosch on 2015-08-25 12:16:44 +00:00

• Status changed from New to Feedback

Please try to set log_rotation to "0" on all Endpoints that are only a agent.

This should disable any massive log read on the master, and will only allow Agent -> Master messages being spooled in a log (agent side)

Better solution will be something like #9730

[icinga-migration]

Updated by mfrosch on 2015-08-25 12:16:54 +00:00

• Relates set to 9730

[icinga-migration]

Updated by aledermueller on 2015-08-27 07:19:05 +00:00

We set log_duration on the master for all agents to 0 and on all agents for the masters to 0. Between the masters, we set it to 2m. With this settings our masters die with #10002. I'm not sure if i should increase the value for the masters to 6h or 1d, or if i should wait for #10002 to be solved.

Thanks, Achim

[icinga-migration]

Updated by mfriedrich on 2015-08-27 14:52:45 +00:00

• Relates set to 10002

[icinga-migration]

Updated by mfriedrich on 2015-08-27 14:53:01 +00:00

• Category set to Cluster
• Status changed from Feedback to New

[icinga-migration]

Updated by mfriedrich on 2015-08-31 13:45:57 +00:00

• Target Version set to Backlog

[icinga-migration]

Updated by mfriedrich on 2015-09-05 15:21:42 +00:00

• Priority changed from Normal to High
• Target Version deleted Backlog

[icinga-migration]

Updated by gbeutner on 2015-10-14 12:47:37 +00:00

• Status changed from New to Closed

This should be fixed in the master branch.