Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dev.icinga.com #12460] DB IDO started before daemonizing (no systemd) #4501

Closed
icinga-migration opened this issue Aug 17, 2016 · 2 comments
Closed

[dev.icinga.com #12460] DB IDO started before daemonizing (no systemd) #4501

icinga-migration opened this issue Aug 17, 2016 · 2 comments
Labels
area/distributed Distributed monitoring (master, satellites, clients) bug Something isn't working
Milestone
2.5.0

Comments

@icinga-migration
Copy link

This issue has been migrated from Redmine: https://dev.icinga.com/issues/12460

Created by mfriedrich on 2016-08-17 07:18:53 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2016-08-17 07:20:05 +00:00)
Target Version: 2.5.0
Last Update: 2016-08-17 07:20:05 +00:00 (in Redmine)

Icinga Version: 2.4.10
Backport?: Not yet backported
Include in Changelog: 1

Steps to reproduce:

Requires a configuration which takes longer than 30s to validate.

Macbook Pro, I5.

const countHosts = 3000;
const countServices = 30;


for (i in range(countHosts)) {
  object Host "many-test-" + i use (i) {
    check_command = "random"
    address = "127.0.0.1"

    if (i == 2000) {
      vars.test = "abc"
    }
  }
}

for (j in range(countServices)) {
  apply Service "many-test-" + j {
    check_command = "http"

    vars.http_uri = "/"

    for (k in range(5)) {
      vars["key" + k] = k
    }
    //log(host.name)
    //var f = get_host(host.name)
    assign where match("many*", host.name)
  }
}

mbmif /usr/local/icinga2/etc/icinga2 (master) # time icinga2 daemon -C
information/cli: Icinga application loader (version: v2.4.10-684-g6688f11; debug)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: mbmif.int.netways.de
information/DbConnection: Resuming IDO connection: ido-mysql
warning/ApplyRule: Apply rule 'satellite-host' (in /usr/local/icinga2/etc/icinga2/conf.d/satellite.conf: 29:1-29:41) for type 'Dependency' does not match anywhere!
information/ConfigItem: Instantiated 2 FileLoggers.
information/ConfigItem: Instantiated 1 Endpoint.
information/ConfigItem: Instantiated 2 Zones.
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 3 ApiUsers.
information/ConfigItem: Instantiated 1 ScheduledDowntime.
information/ConfigItem: Instantiated 189 CheckCommands.
information/ConfigItem: Instantiated 1 Dependency.
information/ConfigItem: Instantiated 1 Comment.
information/ConfigItem: Instantiated 93065 Services.
information/ConfigItem: Instantiated 10 ServiceGroups.
information/ConfigItem: Instantiated 4 TimePeriods.
information/ConfigItem: Instantiated 31 Users.
information/ConfigItem: Instantiated 9 UserGroups.
information/ConfigItem: Instantiated 2 Downtimes.
information/ConfigItem: Instantiated 3064 Hosts.
information/ConfigItem: Instantiated 15 HostGroups.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 5 NotificationCommands.
information/ConfigItem: Instantiated 32 Notifications.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ConfigItem: Instantiated 1 ExternalCommandListener.
information/ConfigItem: Instantiated 1 StatusDataWriter.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.
information/ConfigItem: Instantiated 1 LivestatusListener.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ScriptGlobal: Dumping variables to file '/usr/local/icinga2/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).

mbmif /usr/local/icinga2/etc/icinga2 (master) # lldb -- /usr/local/icinga2/lib/icinga2/sbin/icinga2 daemon -d
(lldb) target create "/usr/local/icinga2/lib/icinga2/sbin/icinga2"
Current executable set to '/usr/local/icinga2/lib/icinga2/sbin/icinga2' (x86_64).
(lldb) settings set -- target.run-args  "daemon" "-d"
(lldb) b fork
Breakpoint 1: where = libsystem_c.dylib`fork, address = 0x000000000000ff70
(lldb) r
Process 48075 launched: '/usr/local/icinga2/lib/icinga2/sbin/icinga2' (x86_64)
[2016-08-17 09:01:50 +0200] information/cli: Icinga application loader (version: v2.4.10-684-g6688f11; debug)
[2016-08-17 09:01:50 +0200] information/cli: Loading configuration file(s).
[2016-08-17 09:01:51 +0200] information/ConfigItem: Committing config item(s).
[2016-08-17 09:01:51 +0200] information/ApiListener: My API identity: mbmif.int.netways.de
[2016-08-17 09:02:19 +0200] information/DbConnection: Resuming IDO connection: ido-mysql
[2016-08-17 09:02:35 +0200] warning/ApplyRule: Apply rule 'satellite-host' (in /usr/local/icinga2/etc/icinga2/conf.d/satellite.conf: 29:1-29:41) for type 'Dependency' does not match anywhere!
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 2 Zones.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 3 ApiUsers.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 1 Comment.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 1 Dependency.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 93065 Services.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 10 ServiceGroups.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 4 TimePeriods.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 31 Users.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 9 UserGroups.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 2 Downtimes.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 3064 Hosts.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 15 HostGroups.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 5 NotificationCommands.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 32 Notifications.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 189 CheckCommands.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 1 StatusDataWriter.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 1 LivestatusListener.
[2016-08-17 09:02:35 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2016-08-17 09:02:35 +0200] information/ScriptGlobal: Dumping variables to file '/usr/local/icinga2/var/cache/icinga2/icinga2.vars'
Process 48075 stopped
* thread #1: tid = 0x3b3b3a, 0x00007fff91503f70 libsystem_c.dylib`fork, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x00007fff91503f70 libsystem_c.dylib`fork
libsystem_c.dylib`fork:
->  0x7fff91503f70 <+0>: pushq  %rbp
    0x7fff91503f71 <+1>: movq   %rsp, %rbp
    0x7fff91503f74 <+4>: pushq  %rbx
    0x7fff91503f75 <+5>: pushq  %rax
(lldb) bt all
* thread #1: tid = 0x3b3b3a, 0x00007fff91503f70 libsystem_c.dylib`fork, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x00007fff91503f70 libsystem_c.dylib`fork
    frame #1: 0x0000000100330055 libcli.dylib`Daemonize() + 21 at daemoncommand.cpp:61
    frame #2: 0x000000010032ef94 libcli.dylib`icinga::DaemonCommand::Run(this=0x0000000101a0ffe0, vm=0x00007fff5fbfe608, ap=0x00007fff5fbfc960) const + 6516 at daemoncommand.cpp:266
    frame #3: 0x000000010000cc5f icinga2`Main() + 28879 at icinga.cpp:488
    frame #4: 0x000000010000d074 icinga2`main(argc=3, argv=0x00007fff5fbff748) + 260 at icinga.cpp:754
    frame #5: 0x00007fff9244f5ad libdyld.dylib`start + 1
    frame #6: 0x00007fff9244f5ad libdyld.dylib`start + 1

  thread #8: tid = 0x3b3e63, 0x00007fff8904edb6 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff8904edb6 libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff8db1c728 libsystem_pthread.dylib`_pthread_cond_wait + 767
    frame #2: 0x00000001010f9e78 libbase.dylib`boost::condition_variable::wait(this=0x0000000103801018, m=0x000070000059e7f8) + 152 at condition_variable.hpp:77
    frame #3: 0x0000000101174686 libbase.dylib`icinga::WorkQueue::WorkerThreadProc(this=0x0000000103800fb0) + 1654 at workqueue.cpp:218
    frame #4: 0x0000000101177ad9 libbase.dylib`boost::_mfi::mf0::operator(this=0x0000000112162760, p=0x0000000103800fb0)(icinga::WorkQueue*) const + 105 at mem_fn_template.hpp:49
    frame #5: 0x0000000101178420 libbase.dylib`void boost::_bi::list1 >::operator(this=0x0000000112162770, (null)=type @ 0x000070000059ee48, f=0x0000000112162760, a=0x000070000059ee70, (null)=0), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) + 80 at bind.hpp:259
    frame #6: 0x00000001011783ca libbase.dylib`boost::_bi::bind_t, boost::_bi::list1 > >::operator(this=0x0000000112162760)() + 58 at bind.hpp:1222
    frame #7: 0x000000010117835c libbase.dylib`boost::detail::thread_data, boost::_bi::list1 > > >::run(this=0x0000000112162560) + 28 at thread.hpp:116
    frame #8: 0x0000000100080804 libboost_thread-mt.dylib`boost::(anonymous namespace)::thread_proxy(void*) + 164
    frame #9: 0x00007fff8db1b99d libsystem_pthread.dylib`_pthread_body + 131
    frame #10: 0x00007fff8db1b91a libsystem_pthread.dylib`_pthread_start + 168
    frame #11: 0x00007fff8db19351 libsystem_pthread.dylib`thread_start + 13
(lldb) b icinga::IdoMysqlConnection::Resume
Breakpoint 2: where = libdb_ido_mysql.dylib`icinga::IdoMysqlConnection::Resume() + 18 at idomysqlconnection.cpp:79, address = 0x00000001058bcd02
(lldb) r
There is a running process, kill it and restart?: [Y/n] y
Process 48075 exited with status = 9 (0x00000009)
Process 48177 launched: '/usr/local/icinga2/lib/icinga2/sbin/icinga2' (x86_64)
[2016-08-17 09:03:08 +0200] information/cli: Icinga application loader (version: v2.4.10-684-g6688f11; debug)
[2016-08-17 09:03:08 +0200] information/cli: Loading configuration file(s).
[2016-08-17 09:03:09 +0200] information/ConfigItem: Committing config item(s).
[2016-08-17 09:03:09 +0200] information/ApiListener: My API identity: mbmif.int.netways.de
Process 48177 stopped
* thread #5: tid = 0x3b449a, 0x00000001057bcd02 libdb_ido_mysql.dylib`icinga::IdoMysqlConnection::Resume(this=0x0000000102129800) + 18 at idomysqlconnection.cpp:79, stop reason = breakpoint 2.1
    frame #0: 0x00000001057bcd02 libdb_ido_mysql.dylib`icinga::IdoMysqlConnection::Resume(this=0x0000000102129800) + 18 at idomysqlconnection.cpp:79
   76
   77   void IdoMysqlConnection::Resume(void)
   78   {
-> 79       DbConnection::Resume();
   80
   81       SetConnected(false);
   82
(lldb) bt
* thread #5: tid = 0x3b449a, 0x00000001057bcd02 libdb_ido_mysql.dylib`icinga::IdoMysqlConnection::Resume(this=0x0000000102129800) + 18 at idomysqlconnection.cpp:79, stop reason = breakpoint 2.1
  * frame #0: 0x00000001057bcd02 libdb_ido_mysql.dylib`icinga::IdoMysqlConnection::Resume(this=0x0000000102129800) + 18 at idomysqlconnection.cpp:79
    frame #1: 0x0000000100fc5738 libbase.dylib`icinga::ConfigObject::SetAuthority(this=0x0000000102129800, authority=true) + 184 at configobject.cpp:472
    frame #2: 0x00000001006e5f36 libremote.dylib`icinga::ApiListener::UpdateObjectAuthority() + 3750 at authority.cpp:78

Changesets

2016-08-17 07:19:58 +00:00 by mfriedrich 87bc291

Fix that HA enabled objects are started before config validation has finished

fixes #12460

Relations:
@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-08-17 07:20:05 +00:00

  • Status changed from Assigned to Resolved
  • Done % changed from 0 to 100

Applied in changeset 87bc291.

@icinga-migration
Copy link
Author

Updated by gbeutner on 2016-08-17 07:37:20 +00:00

  • Duplicated set to 12147

@icinga-migration icinga-migration added bug Something isn't working area/distributed Distributed monitoring (master, satellites, clients) labels Jan 17, 2017
@icinga-migration icinga-migration added this to the 2.5.0 milestone Jan 17, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/distributed Distributed monitoring (master, satellites, clients) bug Something isn't working
Projects
None yet
Milestone
2.5.0
Development

No branches or pull requests
1 participant
@icinga-migration