[dev.icinga.com #12085] deadlock in ido reconnect

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/12085

Created by tobiasvdk on 2016-07-01 09:48:57 +00:00

Assignee: gbeutner
Status: Resolved (closed on 2016-07-12 10:00:08 +00:00)
Target Version: 2.5.0
Last Update: 2016-07-12 10:00:08 +00:00 (in Redmine)

Icinga Version: r2.4.10-509-g208fd53
Backport?: Not yet backported
Include in Changelog: 1

---

My icinga 2 master hangs - __lll_lock_wait in multiple threads. This happened after an icinga reload using mariadb (10.1.14+maria-1~jessie) on debian jessie.

icinga2 - The Icinga 2 network monitoring daemon (version: r2.4.10-509-g208fd53)

Copyright (c) 2012-2016 Icinga Development Team (https://www.icinga.org/)
License GPLv2+: GNU GPL version 2 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Debian GNU/Linux
  Platform version: 8 (jessie)
  Kernel: Linux
  Kernel version: 4.6.0-0.bpo.1-amd64
  Architecture: x86_64

last log messages:

[2016-07-01 08:48:41 +0000] information/IdoMysqlConnection: Query queue items: 2000253, query rate: 9045.48/s (542729/min 923285/5min 923310/15min); empty in infinite time, your database isn't able to keep up
[2016-07-01 08:48:56 +0000] information/IdoMysqlConnection: Query queue items: 2000253, query rate: 9047.78/s (542867/min 1078544/5min 1078569/15min); empty in infinite time, your database isn't able to keep up
[2016-07-01 08:49:11 +0000] information/IdoMysqlConnection: Query queue items: 2000253, query rate: 9038.02/s (542281/min 1233021/5min 1233046/15min); empty in infinite time, your database isn't able to keep up
[2016-07-01 08:49:26 +0000] information/IdoMysqlConnection: Query queue items: 2000253, query rate: 7699.67/s (461980/min 1307774/5min 1307799/15min); empty in infinite time, your database isn't able to keep up

(gdb) info threads
  Id   Target Id         Frame 
  109  Thread 0x2b51a0355700 (LWP 25360) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  108  Thread 0x2b51a0e00700 (LWP 25361) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  107  Thread 0x2b51a1001700 (LWP 25362) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  106  Thread 0x2b51a1202700 (LWP 25363) "icinga2" 0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  105  Thread 0x2b51a1403700 (LWP 25364) "icinga2" 0x00002b519acbc438 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  104  Thread 0x2b51a1604700 (LWP 25365) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  103  Thread 0x2b51a8e00700 (LWP 25414) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  102  Thread 0x2b51a9001700 (LWP 25415) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  101  Thread 0x2b51a9202700 (LWP 25416) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  100  Thread 0x2b51a9403700 (LWP 25417) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  99   Thread 0x2b51a9604700 (LWP 25418) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  98   Thread 0x2b51a9805700 (LWP 25419) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  97   Thread 0x2b51a9a06700 (LWP 25420) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  96   Thread 0x2b51a9c07700 (LWP 25421) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  95   Thread 0x2b51a9e08700 (LWP 25422) "icinga2" 0x00002b519acbc438 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  94   Thread 0x2b51cc600700 (LWP 25423) "icinga2" 0x00002b519d84ad3d in poll () from /lib/x86_64-linux-gnu/libc.so.6
* 93   Thread 0x2b51cd600700 (LWP 25424) "icinga2" 0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  92   Thread 0x2b51cde00700 (LWP 25425) "icinga2" 0x00002b519acbeccd in accept () from /lib/x86_64-linux-gnu/libpthread.so.0
  91   Thread 0x2b51ce001700 (LWP 25426) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  90   Thread 0x2b51d0a00700 (LWP 25428) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  89   Thread 0x2b51d0c01700 (LWP 25429) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  88   Thread 0x2b51d0e02700 (LWP 25430) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  87   Thread 0x2b51d1003700 (LWP 25431) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  86   Thread 0x2b51d1204700 (LWP 25432) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  85   Thread 0x2b51d2200700 (LWP 25433) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  84   Thread 0x2b51d2e00700 (LWP 25434) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  83   Thread 0x2b51d3600700 (LWP 25435) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  82   Thread 0x2b51d9200700 (LWP 25436) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  81   Thread 0x2b51d9401700 (LWP 25437) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  80   Thread 0x2b51d9e01700 (LWP 25438) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  79   Thread 0x2b51d9602700 (LWP 25439) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  78   Thread 0x2b51da002700 (LWP 25440) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  77   Thread 0x2b51da203700 (LWP 25442) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  76   Thread 0x2b51f5600700 (LWP 25444) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  75   Thread 0x2b520ca00700 (LWP 25446) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  74   Thread 0x2b522e200700 (LWP 25448) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  73   Thread 0x2b526c200700 (LWP 25455) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  72   Thread 0x2b526c602700 (LWP 25457) "icinga2" 0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  71   Thread 0x2b526c401700 (LWP 25459) "icinga2" 0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
  70   Thread 0x2b51ce202700 (LWP 25460) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  69   Thread 0x2b526ce00700 (LWP 25461) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  68   Thread 0x2b526d801700 (LWP 25463) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  67   Thread 0x2b526d600700 (LWP 25464) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  66   Thread 0x2b526da02700 (LWP 25465) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  65   Thread 0x2b526e401700 (LWP 25467) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  64   Thread 0x2b526e200700 (LWP 25474) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  63   Thread 0x2b526e602700 (LWP 25475) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  62   Thread 0x2b526e803700 (LWP 25476) "icinga2" 0x00002b519d84ad3d in poll () from /lib/x86_64-linux-gnu/libc.so.6
  61   Thread 0x2b526ea04700 (LWP 25477) "icinga2" 0x00002b519d84ad3d in poll () from /lib/x86_64-linux-gnu/libc.so.6
  60   Thread 0x2b526ec05700 (LWP 25478) "icinga2" 0x00002b519d84ad3d in poll () from /lib/x86_64-linux-gnu/libc.so.6
  59   Thread 0x2b526ee06700 (LWP 25479) "icinga2" 0x00002b519d84ad3d in poll () from /lib/x86_64-linux-gnu/libc.so.6
  58   Thread 0x2b526f208700 (LWP 25482) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  57   Thread 0x2b526f007700 (LWP 25485) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  56   Thread 0x2b526f60a700 (LWP 25488) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  55   Thread 0x2b5270801700 (LWP 25490) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  54   Thread 0x2b5270600700 (LWP 25496) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  53   Thread 0x2b5270a02700 (LWP 25505) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  52   Thread 0x2b5271200700 (LWP 25508) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  51   Thread 0x2b526f409700 (LWP 25509) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  50   Thread 0x2b5271e00700 (LWP 25520) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  49   Thread 0x2b5273001700 (LWP 25528) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  48   Thread 0x2b5272e00700 (LWP 25531) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  47   Thread 0x2b5287a00700 (LWP 25535) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  46   Thread 0x2b5273202700 (LWP 25547) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  45   Thread 0x2b5305200700 (LWP 25548) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  44   Thread 0x2b5315600700 (LWP 25552) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  43   Thread 0x2b5315801700 (LWP 25553) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  42   Thread 0x2b5315a02700 (LWP 25569) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  41   Thread 0x2b5316e00700 (LWP 25573) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  40   Thread 0x2b5317001700 (LWP 25578) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  39   Thread 0x2b5317202700 (LWP 25579) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  38   Thread 0x2b5317604700 (LWP 25610) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  37   Thread 0x2b5317403700 (LWP 25614) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  36   Thread 0x2b5317805700 (LWP 25615) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  35   Thread 0x2b5317a06700 (LWP 25616) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  34   Thread 0x2b5317c07700 (LWP 25620) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  33   Thread 0x2b5318009700 (LWP 25622) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  32   Thread 0x2b5317e08700 (LWP 25627) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  31   Thread 0x2b531820a700 (LWP 25628) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  30   Thread 0x2b531840b700 (LWP 25632) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  29   Thread 0x2b531860c700 (LWP 25633) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  28   Thread 0x2b531880d700 (LWP 25634) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  27   Thread 0x2b5318a0e700 (LWP 25635) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  26   Thread 0x2b5318c0f700 (LWP 25646) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  25   Thread 0x2b5318e10700 (LWP 25649) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  24   Thread 0x2b531a600700 (LWP 25655) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  23   Thread 0x2b531ae00700 (LWP 25659) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  22   Thread 0x2b531c600700 (LWP 25678) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  21   Thread 0x2b531d600700 (LWP 25692) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  20   Thread 0x2b531d801700 (LWP 25699) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  19   Thread 0x2b531da02700 (LWP 25703) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  18   Thread 0x2b531dc03700 (LWP 25704) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  17   Thread 0x2b531de04700 (LWP 25721) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  16   Thread 0x2b531e005700 (LWP 25731) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  15   Thread 0x2b531e206700 (LWP 25736) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  14   Thread 0x2b531e407700 (LWP 25740) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  13   Thread 0x2b531e608700 (LWP 25773) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  12   Thread 0x2b531e809700 (LWP 25775) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  11   Thread 0x2b531ea0a700 (LWP 25778) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  10   Thread 0x2b531ec0b700 (LWP 25781) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  9    Thread 0x2b531ee0c700 (LWP 25784) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  8    Thread 0x2b531f00d700 (LWP 25787) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  7    Thread 0x2b531f20e700 (LWP 25867) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  6    Thread 0x2b531f40f700 (LWP 25895) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  5    Thread 0x2b531f610700 (LWP 25907) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  4    Thread 0x2b531f811700 (LWP 26006) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  3    Thread 0x2b531fa12700 (LWP 26013) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  2    Thread 0x2b531fc13700 (LWP 26324) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
  1    Thread 0x2b5199edef00 (LWP 25359) "icinga2" 0x00002b519acbc08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0

(gdb) thread 93
[Switching to thread 93 (Thread 0x2b51cd600700 (LWP 25424))]

#9  0x00002b51a4ed83a5 in icinga::DbConnection::UpdateObject (this=this@entry=0x2b51aae60c00, object=(boost::intrusive_ptr) 0x2b51bb17d600)

at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/db_ido/dbconnection.cpp:399
399 dbobj->SendConfigUpdate();
(gdb) bt

#0  0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00002b519acba4d4 in _L_lock_952 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00002b519acba336 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x00002b51a4ef8935 in lock (this=) at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
#4  icinga::ObjectLock::LockMutex (object=object@entry=0x2b51b71b72e0) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:74
#5  0x00002b51a4ed6249 in Lock (this=0x2b51cd5fdd50) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:100
#6  ObjectLock (object=..., this=0x2b51cd5fdd50) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:49
#7  icinga::DbObject::SendVarsConfigUpdate (this=this@entry=0x2b5202c642e0) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/db_ido/dbobject.cpp:164
#8  0x00002b51a4ed7bec in icinga::DbObject::SendConfigUpdate (this=0x2b5202c642e0) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/db_ido/dbobject.cpp:86
#9  0x00002b51a4ed83a5 in icinga::DbConnection::UpdateObject (this=this@entry=0x2b51aae60c00, object=(boost::intrusive_ptr) 0x2b51bb17d600)

at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/db_ido/dbconnection.cpp:399

#10 0x00002b51a4ed87d5 in icinga::DbConnection::UpdateAllObjects (this=this@entry=0x2b51aae60c00) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/db_ido/dbconnection.cpp:415
#11 0x00002b51a464ed68 in icinga::IdoMysqlConnection::Reconnect (this=0x2b51aae60c00) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/db_ido_mysql/idomysqlconnection.cpp:413
#12 0x00002b519afcfd63 in operator() (this=0x2b51cd5ffc50) at /usr/include/boost/function/function_template.hpp:767
#13 icinga::WorkQueue::WorkerThreadProc (this=0x2b51aae60ea8) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/workqueue.cpp:228
#14 0x00002b519a312aea in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:164
#15 0x00002b519acb80a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00002b519d85387d in clone () from /lib/x86_64-linux-gnu/libc.so.6

(gdb) thread 106 [Switching to thread 106 (Thread 0x2b51a1202700 (LWP 25363))]

#0  0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0

(gdb) bt

#0  0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00002b519acba4d4 in _L_lock_952 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00002b519acba336 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x00002b51a3ad7552 in lock (this=) at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
#4  icinga::ObjectLock::LockMutex (object=object@entry=0x2b51bc059600) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:74
#5  0x00002b51a3a46957 in Lock (this=0x2b51a1200780) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:100
#6  ObjectLock (object=..., this=0x2b51a1200780) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:49
#7  icinga::CIB::CalculateServiceCheckStats () at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/cib.cpp:137
#8  0x00002b51a3a9a391 in icinga::CIB::StatsFunc (status=(boost::intrusive_ptr) 0x2b51c9889380, perfdata=...) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/cib.cpp:292
#9  0x00002b519afaa6aa in operator() (a1=(boost::intrusive_ptr) 0x1, a0=(boost::intrusive_ptr) 0x2b51c9889380, this=0x2b51a08b6ad8)

at /usr/include/boost/function/function_template.hpp:767

#10 icinga::StatsFunction::Invoke (this=this@entry=0x2b51a08b6ac0, status=(boost::intrusive_ptr) 0x2b51c9889380, perfdata=(boost::intrusive_ptr) 0x2b51c988a280)

at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/statsfunction.cpp:32

#11 0x00002b51a3a46ebb in icinga::CIB::GetFeatureStats () at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/cib.cpp:260
#12 0x00002b51a3e67f78 in icinga::ClusterCheckTask::ScriptFunc (checkable=(boost::intrusive_ptr) 0x2b51b5967d80, cr=(boost::intrusive_ptr) 0x2b51c9845080, 

resolvedMacros=(boost::intrusive_ptricinga::Dictionary) 0x0, useResolvedMacros=useResolvedMacros@entry=false) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/methods/clusterchecktask.cpp:59

#13 0x00002b51a3e70b00 in icinga::FunctionWrapperV const&, boost::intrusive_ptr const&, boost::intrusive_ptr const&, bool>

(function=
0x2b51a3e67ed0 <icinga::ClusterCheckTask::ScriptFunc(boost::intrusive_ptricinga::Checkable const&, boost::intrusive_ptricinga::CheckResult const&, boost::intrusive_ptricinga::Dictionary const&, bool)>, arguments=std::vector of length 4, capacity 4 = {...}) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/functionwrapper.hpp:171

#14 0x00002b51a3e6c42d in operator()&, const boost::intrusive_ptr&, const boost::intrusive_ptr&, bool), const std::vector&), boost::_bi::list1&> > (f=, a=, this=)

at /usr/include/boost/bind/bind.hpp:303

#15 operator() > (a1=..., this=) at /usr/include/boost/bind/bind_template.hpp:47
#16 boost::detail::function::function_obj_invoker1 const&, boost::intrusive_ptr const&, boost::intrusive_ptr const&, bool), std::vector > const&), boost::_bi::list2 const&, boost::intrusive_ptr const&, boost::intrusive_ptr const&, bool)>, boost::arg<1> > >, icinga::Value, std::vector > const&>::invoke (function_obj_ptr=..., a0=...) at /usr/include/boost/function/function_template.hpp:132
#17 0x00002b519afaa61d in operator() (a0=std::vector of length 4, capacity 4 = {...}, this=) at /usr/include/boost/function/function_template.hpp:767
#18 icinga::Function::Invoke (this=, arguments=std::vector of length 4, capacity 4 = {...}) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/function.cpp:34
#19 0x00002b51a3a2b44b in icinga::CheckCommand::Execute (this=0x2b51ab419640, checkable=..., cr=(boost::intrusive_ptr) 0x2b51c9845080, 

resolvedMacros=(boost::intrusive_ptricinga::Dictionary) 0x0, useResolvedMacros=) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/checkcommand.cpp:36

#20 0x00002b51a3ab1dff in icinga::Checkable::ExecuteCheck (this=0x2b51b5967d80) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/checkable-check.cpp:448
#21 0x00002b51a40bc0bb in icinga::CheckerComponent::ExecuteCheckHelper (this=0x2b51ab4a8180, checkable=(boost::intrusive_ptr) 0x2b51b5967d80)

at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/checker/checkercomponent.cpp:204

#22 0x00002b519afe0187 in operator() (this=0x2b51a12019d0) at /usr/include/boost/function/function_template.hpp:767
#23 icinga::ThreadPool::WorkerThread::ThreadProc (this=0x2b519b2dc2b0 , queue=...) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/threadpool.cpp:138
#24 0x00002b519a312aea in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:164
#25 0x00002b519acb80a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#26 0x00002b519d85387d in clone () from /lib/x86_64-linux-gnu/libc.so.6

(gdb) thread 72 [Switching to thread 72 (Thread 0x2b526c602700 (LWP 25457))]

#0  0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0

(gdb) bt

#0  0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00002b519acba4d4 in _L_lock_952 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00002b519acba336 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x00002b519b01dfd5 in lock (this=) at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
#4  icinga::ObjectLock::LockMutex (object=object@entry=0x2b51bc059600) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:74
#5  0x00002b519afa717e in Lock (this=0x2b526c6014b0) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:100
#6  ObjectLock (object=0x2b51bc059600, this=0x2b526c6014b0) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:56
#7  icinga::ConfigObject::SetAuthority (this=0x2b51bc059600, authority=) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/configobject.cpp:468
#8  0x00002b519b9b29c6 in icinga::ApiListener::UpdateObjectAuthority () at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/remote/authority.cpp:78
#9  0x00002b519b05999c in operator() (a0=..., this=) at /usr/include/boost/function/function_template.hpp:767
#10 m_invoke&)>, 0u, const boost::intrusive_ptr&> (args=..., func=..., this=)

at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:92

#11 operator()&)>, const boost::intrusive_ptr&, 1ul> (args=..., func=..., this=)

at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:81

#12 m_invoke >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (connectionBody=..., this=)

at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:117

#13 operator() >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (connectionBody=..., this=)

at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:110

#14 dereference (this=0x2b526c601720) at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:82
#15 dereference&>, std::_List_iterator >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (f=...)

at /usr/include/boost/iterator/iterator_facade.hpp:514

#16 operator* (this=0x2b526c601720) at /usr/include/boost/iterator/iterator_facade.hpp:639
#17 operator()&>, std::_List_iterator >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (last=..., 

first=..., this=) at /usr/include/boost/signals2/optional_last_value.hpp:55

#18 operator(), boost::signals2::detail::slot_call_iterator_t&>, std::_List_iterator >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body >, boost::signals2::slot&), boost::function&)> >, boost::signals2::mutex> > > (last=..., first=..., combiner=..., this=) at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:64
#19 boost::signals2::detail::signal_impl const&), boost::signals2::optional_last_value, int, std::less, boost::function const&)>, boost::function const&)>, boost::signals2::mutex>::operator()(boost::intrusive_ptr const&) (this=, args#0=(boost::intrusive_ptr) 0x2b51a085ebf0) at /usr/include/boost/signals2/detail/signal_template.hpp:246
#20 0x00002b519b000bd9 in operator() (args#0=(boost::intrusive_ptr) 0x2b51a085ebf0, this=0x2b51a085ec08) at /usr/include/boost/signals2/detail/signal_template.hpp:695
#21 icinga::Timer::Call (this=0x2b51a085ebf0) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/timer.cpp:94
#22 0x00002b519afe0187 in operator() (this=0x2b526c6019d0) at /usr/include/boost/function/function_template.hpp:767
#23 icinga::ThreadPool::WorkerThread::ThreadProc (this=0x2b519b2dbfb0 , queue=...) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/threadpool.cpp:138
#24 0x00002b519a312aea in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:164
#25 0x00002b519acb80a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#26 0x00002b519d85387d in clone () from /lib/x86_64-linux-gnu/libc.so.6

(gdb) thread 71 [Switching to thread 71 (Thread 0x2b526c401700 (LWP 25459))]

#0  0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0

(gdb) bt

#0  0x00002b519acbe7fc in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00002b519acba4d4 in _L_lock_952 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00002b519acba336 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x00002b51a3ad7552 in lock (this=) at /usr/include/boost/thread/pthread/recursive_mutex.hpp:110
#4  icinga::ObjectLock::LockMutex (object=object@entry=0x2b51b595ef00) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:74
#5  0x00002b51a3a46957 in Lock (this=0x2b526c3ff810) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:100
#6  ObjectLock (object=..., this=0x2b526c3ff810) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/objectlock.hpp:49
#7  icinga::CIB::CalculateServiceCheckStats () at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/cib.cpp:137
#8  0x00002b51a3e62da6 in icinga::IcingaCheckTask::ScriptFunc (service=(boost::intrusive_ptr) 0x2b51b5d62600, cr=(boost::intrusive_ptr) 0x2b526a678000, 

resolvedMacros=(boost::intrusive_ptricinga::Dictionary) 0x0, useResolvedMacros=useResolvedMacros@entry=false) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/methods/icingachecktask.cpp:66

#9  0x00002b51a3e70b00 in icinga::FunctionWrapperV const&, boost::intrusive_ptr const&, boost::intrusive_ptr const&, bool>

(function=
0x2b51a3e61bf0 <icinga::IcingaCheckTask::ScriptFunc(boost::intrusive_ptricinga::Checkable const&, boost::intrusive_ptricinga::CheckResult const&, boost::intrusive_ptricinga::Dictionary const&, bool)>, arguments=std::vector of length 4, capacity 4 = {...}) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/functionwrapper.hpp:171

#10 0x00002b51a3e6c42d in operator()&, const boost::intrusive_ptr&, const boost::intrusive_ptr&, bool), const std::vector&), boost::_bi::list1&> > (f=, a=, this=)

at /usr/include/boost/bind/bind.hpp:303

#11 operator() > (a1=..., this=) at /usr/include/boost/bind/bind_template.hpp:47
#12 boost::detail::function::function_obj_invoker1 const&, boost::intrusive_ptr const&, boost::intrusive_ptr const&, bool), std::vector > const&), boost::_bi::list2 const&, boost::intrusive_ptr const&, boost::intrusive_ptr const&, bool)>, boost::arg<1> > >, icinga::Value, std::vector > const&>::invoke (function_obj_ptr=..., a0=...) at /usr/include/boost/function/function_template.hpp:132
#13 0x00002b519afaa61d in operator() (a0=std::vector of length 4, capacity 4 = {...}, this=) at /usr/include/boost/function/function_template.hpp:767
#14 icinga::Function::Invoke (this=, arguments=std::vector of length 4, capacity 4 = {...}) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/function.cpp:34
#15 0x00002b51a3a2b44b in icinga::CheckCommand::Execute (this=0x2b51aa863280, checkable=..., cr=(boost::intrusive_ptr) 0x2b526a678000, 

resolvedMacros=(boost::intrusive_ptricinga::Dictionary) 0x0, useResolvedMacros=) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/checkcommand.cpp:36

#16 0x00002b51a3ab1dff in icinga::Checkable::ExecuteCheck (this=0x2b51b5d62600) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/icinga/checkable-check.cpp:448
#17 0x00002b51a40bc0bb in icinga::CheckerComponent::ExecuteCheckHelper (this=0x2b51ab4a8180, checkable=(boost::intrusive_ptr) 0x2b51b5d62600)

at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/checker/checkercomponent.cpp:204

#18 0x00002b519afe0187 in operator() (this=0x2b526c4009d0) at /usr/include/boost/function/function_template.hpp:767
#19 icinga::ThreadPool::WorkerThread::ThreadProc (this=0x2b519b2dbfd0 , queue=...) at /build/icinga2-7VnrWK/icinga2-2.5.0/lib/base/threadpool.cpp:138
#20 0x00002b519a312aea in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:164
#21 0x00002b519acb80a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#22 0x00002b519d85387d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Attachments

• gdb_hangs.txt tobiasvdk - 2016-07-01 09:48:50 +00:00

Changesets

2016-07-12 09:44:36 +00:00 by gbeutner ea57f00

Fix deadlock in DbObject::SendVars{Config,Status}Update

fixes #12127
fixes #12085

---

Relations:

• relates #12127

[icinga-migration]

Updated by gbeutner on 2016-07-12 09:45:57 +00:00

• Category set to DB IDO
• Status changed from New to Assigned
• Assigned to set to gbeutner
• Target Version set to 2.5.0

[icinga-migration]

Updated by gbeutner on 2016-07-12 09:46:10 +00:00

• Relates set to 12127

[icinga-migration]

Updated by gbeutner on 2016-07-12 10:00:08 +00:00

• Status changed from Assigned to Resolved
• Done % changed from 0 to 100

Applied in changeset ea57f00.