[dev.icinga.com #12667] Crash in HttpRequest::Parse while processing HTTP request #4644

icinga-migration · 2016-09-07T12:19:50Z

This issue has been migrated from Redmine: https://dev.icinga.com/issues/12667

Created by de1m on 2016-09-07 12:19:50 +00:00

Assignee: gbeutner
Status: Resolved (closed on 2016-11-17 08:10:03 +00:00)
Target Version: 2.6.0
Last Update: 2016-11-21 10:48:57 +00:00 (in Redmine)

Icinga Version: r2.5.4-1
Backport?: Not yet backported
Include in Changelog: 1

Hi,
I've installed yesterday the new version of icinga2 - r2.5.4-1
So, my problem now, that this service crashed after ca. 2 hours.
I found this in crash log

Application information:
  Application version: r2.5.4-1
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Ubuntu
  Platform version: 16.04.1 LTS (Xenial Xerus)
  Kernel: Linux
  Kernel version: 3.13.0-95-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 5.4.0
  Build host: lgw01-29
Stacktrace:

        (0) libpthread.so.0:  (+0x113d0) [0x7f82374fc3d0]
        (1) libc.so.6: gsignal (+0x38) [0x7f82361c1418]
        (2) libc.so.6: abort (+0x16a) [0x7f82361c301a]
        (3) libc.so.6:  (+0x2dbd7) [0x7f82361b9bd7]
        (4) libc.so.6:  (+0x2dc82) [0x7f82361b9c82]
        (5) libremote.so: icinga::HttpRequest::Parse(icinga::StreamReadContext&, bool) (+0x7f8) [0x7f8235972d78]
        (6) libremote.so: icinga::HttpServerConnection::ProcessMessage() (+0x3c) [0x7f823597310c]
        (7) libremote.so: icinga::HttpServerConnection::DataAvailableHandler() (+0x88) [0x7f8235973a08]
        (8) libremote.so: icinga::HttpServerConnection::Start() (+0x18d) [0x7f8235973ced]
        (9) libremote.so: icinga::ApiListener::NewClientHandlerInternal(boost::intrusive_ptr const&, icinga::String const&, icinga::ConnectionRole) (+0x46a) [0x7f82359abefa]
        (10) libremote.so: icinga::ApiListener::NewClientHandler(boost::intrusive_ptr const&, icinga::String const&, icinga::ConnectionRole) (+0x25) [0x7f82359ad325]
        (11) libboost_thread.so.1.58.0:  (+0x115d5) [0x7f8237b9b5d5]
        (12) libpthread.so.0:  (+0x76fa) [0x7f82374f26fa]
        (13) libc.so.6: clone (+0x6d) [0x7f8236292b5d]

This version running in a docker container. Can you found here https://hub.docker.com/r/adito/icinga2/

Changesets

2016-11-17 08:08:22 +00:00 by gbeutner f58d3a1

Fix crash in HttpRequest::Parse

fixes #12667

Relations:

relates #12667
relates #12757
relates #12667
relates #12667
relates #12667
relates #12667
duplicates #12667
duplicates #12667
duplicates #12667

The text was updated successfully, but these errors were encountered:

icinga-migration · 2016-09-13T20:16:26Z

Updated by gbeutner on 2016-09-13 20:16:26 +00:00

Relates set to 12709

icinga-migration · 2016-09-13T20:17:14Z

Updated by gbeutner on 2016-09-13 20:17:14 +00:00

Status changed from New to Assigned
Assigned to set to mfriedrich

dnsmichi: Please have a look at this.

icinga-migration · 2016-09-21T08:15:38Z

Updated by mfriedrich on 2016-09-21 08:15:38 +00:00

Relates set to 12757

icinga-migration · 2016-09-21T08:16:08Z

Updated by mfriedrich on 2016-09-21 08:16:08 +00:00

Category changed from libicinga to API
Priority changed from Normal to High

icinga-migration · 2016-09-28T13:51:07Z

Updated by mfriedrich on 2016-09-28 13:51:07 +00:00

Target Version set to 2.6.0

icinga-migration · 2016-09-29T14:00:49Z

Updated by gbeutner on 2016-09-29 14:00:49 +00:00

Duplicated set to 12824

icinga-migration · 2016-09-29T20:51:09Z

Updated by magnus on 2016-09-29 20:51:09 +00:00

Target version set to 2.6.0

Does this mean it will not get fixed until 2.6.0?

Regards
Magnus

icinga-migration · 2016-09-30T07:10:31Z

Updated by mfriedrich on 2016-09-30 07:10:31 +00:00

It does mean that is planned being investigated and fixed in the next development sprint. We do not assign tickets to bug fix versions, rather fix them in master and then backport fixes if necessary.

Kind regards,
Michael

icinga-migration · 2016-09-30T08:39:58Z

Updated by magnus on 2016-09-30 08:39:58 +00:00

Ok thanks for the information and for your time in developing icinga and
especially the api which I think clearly puts icinga ahead of nagios.

Regards
Magnus

On 2016-09-30 09:10, development@icinga.org wrote:

Issue #12667 has been updated by dnsmichi.

It does mean that is planned being investigated and fixed in the next development sprint. We do not assign tickets to bug fix versions, rather fix them in master and then backport fixes if necessary.

Kind regards,
Michael

Bug #12667: Icinga2 Service stopped after 2 hours
https://dev.icinga.org/issues/12667#change-55169

* Author: de1m
* Status: Assigned
* Priority: High
* Assignee: dnsmichi
* Category: API
* Target version: 2.6.0
* Icinga Version: r2.5.4-1
* Package Repository:
* Backport?: Not yet backported
* Include in Changelog: Yes

Hi,
I've installed yesterday the new version of icinga2 - r2.5.4-1
So, my problem now, that this service crashed after ca. 2 hours.
I found this in crash log

> Application information:
>    Application version: r2.5.4-1
>    Installation root: /usr
>    Sysconf directory: /etc
>    Run directory: /run
>    Local state directory: /var
>    Package data directory: /usr/share/icinga2
>    State path: /var/lib/icinga2/icinga2.state
>    Modified attributes path: /var/lib/icinga2/modified-attributes.conf
>    Objects path: /var/cache/icinga2/icinga2.debug
>    Vars path: /var/cache/icinga2/icinga2.vars
>    PID path: /run/icinga2/icinga2.pid
>
> System information:
>    Platform: Ubuntu
>    Platform version: 16.04.1 LTS (Xenial Xerus)
>    Kernel: Linux
>    Kernel version: 3.13.0-95-generic
>    Architecture: x86_64
>
> Build information:
>    Compiler: GNU 5.4.0
>    Build host: lgw01-29
> Stacktrace:
>
>          (0) libpthread.so.0:  (+0x113d0) [0x7f82374fc3d0]
>          (1) libc.so.6: gsignal (+0x38) [0x7f82361c1418]
>          (2) libc.so.6: abort (+0x16a) [0x7f82361c301a]
>          (3) libc.so.6:  (+0x2dbd7) [0x7f82361b9bd7]
>          (4) libc.so.6:  (+0x2dc82) [0x7f82361b9c82]
>          (5) libremote.so: icinga::HttpRequest::Parse(icinga::StreamReadContext&, bool) (+0x7f8) [0x7f8235972d78]
>          (6) libremote.so: icinga::HttpServerConnection::ProcessMessage() (+0x3c) [0x7f823597310c]
>          (7) libremote.so: icinga::HttpServerConnection::DataAvailableHandler() (+0x88) [0x7f8235973a08]
>          (8) libremote.so: icinga::HttpServerConnection::Start() (+0x18d) [0x7f8235973ced]
>          (9) libremote.so: icinga::ApiListener::NewClientHandlerInternal(boost::intrusive_ptr const&, icinga::String const&, icinga::ConnectionRole) (+0x46a) [0x7f82359abefa]
>          (10) libremote.so: icinga::ApiListener::NewClientHandler(boost::intrusive_ptr const&, icinga::String const&, icinga::ConnectionRole) (+0x25) [0x7f82359ad325]
>          (11) libboost_thread.so.1.58.0:  (+0x115d5) [0x7f8237b9b5d5]
>          (12) libpthread.so.0:  (+0x76fa) [0x7f82374f26fa]
>          (13) libc.so.6: clone (+0x6d) [0x7f8236292b5d]
>

This version running in a docker container. Can you found here https://hub.docker.com/r/adito/icinga2/

icinga-migration · 2016-11-02T06:53:14Z

Updated by gbeutner on 2016-11-02 06:53:14 +00:00

Duplicated set to 13004

icinga-migration · 2016-11-02T06:58:12Z

Updated by gbeutner on 2016-11-02 06:58:12 +00:00

Duplicated set to 12987

icinga-migration · 2016-11-02T07:00:15Z

Updated by gbeutner on 2016-11-02 07:00:15 +00:00

Relates set to 12996

icinga-migration · 2016-11-11T07:52:11Z

Updated by oldunixman on 2016-11-11 07:52:11 +00:00

I have a similar Problem: every day crash the daemon. In HA Setup only the configuration master(ubuntu16.04) crash. Same icinga version on secondary server with (ubuntu 12.04) running fine.

root@monp1:/var/log/icinga2# ps -ef|grep [i]cinga2
root@monp1:/var/log/icinga2#

last debug.log entry:
[2016-11-11 02:00:56 +0100] critical/Application: Icinga 2 has terminated unexpectedly. Additional information can be found in '/var/log/icinga2/crash/report.1478826056.595711'

[2016-11-11 02:00:56 +0100] notice/Process: PID 17455 ('/etc/icinga2/scripts/mail-service-notification.sh') terminated with exit code 0
[2016-11-11 02:00:56 +0100] notice/Process: PID 17506 ('/etc/icinga2/scripts/mail-service-notification.sh') terminated with exit code 0

last icinga2.log entry:

[2016-11-11 02:00:56 +0100] warning/ApiListener: Discarding 'config update object' message from 'pXXX.de'.
[2016-11-11 02:00:56 +0100] warning/ApiListener: Discarding 'config update object' message from 'pXXX.de'.
[2016-11-11 02:00:56 +0100] warning/ApiListener: Discarding 'config update object' message from 'pXXX.de'.
[2016-11-11 02:00:56 +0100] warning/ApiListener: Discarding 'config update object' message from 'pXXX.de'.
[2016-11-11 02:00:56 +0100] warning/ApiListener: Discarding 'config update object' message from 'pXXX.de'.
[2016-11-11 02:00:56 +0100] warning/ApiListener: Discarding 'config update object' message from 'pXXX.de'.
[2016-11-11 02:00:56 +0100] warning/ApiListener: Discarding 'config update object' message from 'pXXX.de'.
[2016-11-11 02:00:56 +0100] warning/ApiListener: Discarding 'config update object' message from 'pXXX.de'.
[2016-11-11 02:00:56 +0100] warning
~

Crash:

Application information:
Application version: r2.5.4-1
Installation root: /usr
Sysconf directory: /etc
Run directory: /run
Local state directory: /var
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

System information:
Platform: Ubuntu
Platform version: 16.04.1 LTS (Xenial Xerus)
Kernel: Linux
Kernel version: 4.4.0-47-generic
Architecture: x86_64

Build information:
Compiler: GNU 5.4.0
Build host: lgw01-29
Stacktrace:

(0) libpthread.so.0: (+0x113e0) [0x7fb8ca90a3e0]
(1) libc.so.6: gsignal (+0x38) [0x7fb8c95cf428]
(2) libc.so.6: abort (+0x16a) [0x7fb8c95d102a]
(3) libc.so.6: (+0x2dbd7) [0x7fb8c95c7bd7]
(4) libc.so.6: (+0x2dc82) [0x7fb8c95c7c82]
(5) libicinga.so: icinga::ClusterEvents::SendNotificationsAPIHandler(boost::intrusive_ptricinga::MessageOrigin const&, boost::intrusive_ptricinga::Dictionary const&) (+0xa35) [0x7fb8c43c8d55]
(6) libremote.so: boost::detail::function::function_invoker2<icinga::Value (*)(boost::intrusive_ptricinga::MessageOrigin const&, boost::intrusive_ptricinga::Dictionary const&), icinga::Value, boost::intrusive_ptricinga::MessageOrigin const&, boost::intrusive_ptricinga::Dictionary const&>::invoke(boost::detail::function::function_buffer&, boost::intrusive_ptricinga::MessageOrigin const&, boost::intrusive_ptricinga::Dictionary const&) (+0x23) [0x7fb8c8dc53f3]
(7) libremote.so: icinga::ApiFunction::Invoke(boost::intrusive_ptricinga::MessageOrigin const&, boost::intrusive_ptricinga::Dictionary const&) (+0x36) [0x7fb8c8d64216]
(8) libremote.so: icinga::JsonRpcConnection::MessageHandler(icinga::String const&) (+0x4cc) [0x7fb8c8db792c]
(9) libremote.so: icinga::JsonRpcConnection::MessageHandlerWrapper(icinga::String const&) (+0x73) [0x7fb8c8db94a3]
(10) libbase.so: icinga::WorkQueue::WorkerThreadProc() (+0x529) [0x7fb8c9ff55f9]
(11) libboost_thread.so.1.58.0: (+0x115d5) [0x7fb8cafa95d5]
(12) libpthread.so.0: (+0x770a) [0x7fb8ca90070a]
(13) libc.so.6: clone (+0x6d) [0x7fb8c96a082d]

This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
please submit a bug report at https://dev.icinga.org/ and include this stack trace as well as any other
information that might be useful in order to reproduce this problem.
*****

icinga-migration · 2016-11-11T08:41:31Z

Updated by mfriedrich on 2016-11-11 08:41:31 +00:00

That's a different issue handled with #13151 and #12718.

icinga-migration · 2016-11-16T15:40:54Z

Updated by mfriedrich on 2016-11-16 15:40:54 +00:00

Relates set to 12898

icinga-migration · 2016-11-17T08:09:03Z

Updated by gbeutner on 2016-11-17 08:09:03 +00:00

Assigned to changed from mfriedrich to gbeutner

icinga-migration · 2016-11-17T08:09:51Z

Updated by gbeutner on 2016-11-17 08:09:51 +00:00

Subject changed from Icinga2 Service stopped after 2 hours to Crash in HttpRequest::Parse while processing HTTP request

icinga-migration · 2016-11-17T08:10:03Z

Updated by gbeutner on 2016-11-17 08:10:03 +00:00

Status changed from Assigned to Resolved
Done % changed from 0 to 100

Applied in changeset f58d3a1.

icinga-migration · 2016-11-18T09:23:57Z

Updated by mfriedrich on 2016-11-18 09:23:57 +00:00

Relates set to 13237

icinga-migration · 2016-11-21T10:17:02Z

Updated by gbeutner on 2016-11-21 10:17:02 +00:00

I believe this should be fixed in the master branch. Unfortunately I am unable to reproduce this problem here so I'm going to have to rely on feedback to see if this issue reoccurs. Please test the snapshot packages. :)

icinga-migration · 2016-11-21T10:24:31Z

Updated by de1m on 2016-11-21 10:24:31 +00:00

Thanks for info, where can I get the spanshot package?

icinga-migration · 2016-11-21T10:48:57Z

Updated by Nagodar on 2016-11-21 10:48:57 +00:00

I have updated to the snapshot2016.11.20 snapshot. I'll come back once tested :)

EDIT:
tried also de 20161121 snapshot, but the server stops constantly with:
[2016-11-21 12:17:05 +0100] information/Application: Received request to shut down.
[2016-11-21 12:17:06 +0100] information/Application: Shutting down...
[2016-11-21 12:17:06 +0100] information/CheckerComponent: Checker stopped.

it didn't happen with 2.5.4 :/... or at least after that the icinga2 daemon started again (could be something related to update via api?)

regards,

icinga-migration · 2016-12-06T16:10:50Z

Updated by mfriedrich on 2016-12-06 16:10:50 +00:00

Relates set to 12765

icinga-migration closed this as completed Nov 17, 2016

icinga-migration added blocker Blocks a release or needs immediate attention bug Something isn't working area/api REST API labels Jan 17, 2017

icinga-migration added this to the 2.6.0 milestone Jan 17, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dev.icinga.com #12667] Crash in HttpRequest::Parse while processing HTTP request #4644

[dev.icinga.com #12667] Crash in HttpRequest::Parse while processing HTTP request #4644

icinga-migration commented Sep 7, 2016

icinga-migration commented Sep 13, 2016

icinga-migration commented Sep 13, 2016

icinga-migration commented Sep 21, 2016

icinga-migration commented Sep 21, 2016

icinga-migration commented Sep 28, 2016

icinga-migration commented Sep 29, 2016

icinga-migration commented Sep 29, 2016

icinga-migration commented Sep 30, 2016

icinga-migration commented Sep 30, 2016

* Author: de1m
* Status: Assigned
* Priority: High
* Assignee: dnsmichi
* Category: API
* Target version: 2.6.0
* Icinga Version: r2.5.4-1
* Package Repository:
* Backport?: Not yet backported
* Include in Changelog: Yes

icinga-migration commented Nov 2, 2016

icinga-migration commented Nov 2, 2016

icinga-migration commented Nov 2, 2016

icinga-migration commented Nov 11, 2016

icinga-migration commented Nov 11, 2016

icinga-migration commented Nov 16, 2016

icinga-migration commented Nov 17, 2016

icinga-migration commented Nov 17, 2016

icinga-migration commented Nov 17, 2016

icinga-migration commented Nov 18, 2016

icinga-migration commented Nov 21, 2016

icinga-migration commented Nov 21, 2016

icinga-migration commented Nov 21, 2016

icinga-migration commented Dec 6, 2016

[dev.icinga.com #12667] Crash in HttpRequest::Parse while processing HTTP request #4644

[dev.icinga.com #12667] Crash in HttpRequest::Parse while processing HTTP request #4644

Comments

icinga-migration commented Sep 7, 2016

icinga-migration commented Sep 13, 2016

icinga-migration commented Sep 13, 2016

icinga-migration commented Sep 21, 2016

icinga-migration commented Sep 21, 2016

icinga-migration commented Sep 28, 2016

icinga-migration commented Sep 29, 2016

icinga-migration commented Sep 29, 2016

icinga-migration commented Sep 30, 2016

icinga-migration commented Sep 30, 2016

* Author: de1m * Status: Assigned * Priority: High * Assignee: dnsmichi * Category: API * Target version: 2.6.0 * Icinga Version: r2.5.4-1 * Package Repository: * Backport?: Not yet backported * Include in Changelog: Yes

icinga-migration commented Nov 2, 2016

icinga-migration commented Nov 2, 2016

icinga-migration commented Nov 2, 2016

icinga-migration commented Nov 11, 2016

icinga-migration commented Nov 11, 2016

icinga-migration commented Nov 16, 2016

icinga-migration commented Nov 17, 2016

icinga-migration commented Nov 17, 2016

icinga-migration commented Nov 17, 2016

icinga-migration commented Nov 18, 2016

icinga-migration commented Nov 21, 2016

icinga-migration commented Nov 21, 2016

icinga-migration commented Nov 21, 2016

icinga-migration commented Dec 6, 2016

* Author: de1m
* Status: Assigned
* Priority: High
* Assignee: dnsmichi
* Category: API
* Target version: 2.6.0
* Icinga Version: r2.5.4-1
* Package Repository:
* Backport?: Not yet backported
* Include in Changelog: Yes