[dev.icinga.com #11436] Add LogstashWriter for Logstash in addition to GelfWriter

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/11436

Created by twidhalm on 2016-03-22 15:49:55 +00:00

Assignee: mfriedrich
Status: Assigned
Target Version: 2.7.0
Last Update: 2016-11-09 15:11:53 +00:00 (in Redmine)

Backport?: Not yet backported
Include in Changelog: 1

---

The GELF protocol can use UDP or TCP according to the documentation.

Since some GELF receivers (like Logstash) can only receive UDP and the Icinga 2 GELF writer can only send TCP we have the situation that 2 tools talking the same protocol still can't talk to each other.

Could you add a protocol switcher to the GELF writer object?

Changesets

2016-06-15 13:21:28 +00:00 by kgoller 8ad95bd

Delete unused files

refs #11436

2016-06-15 13:27:26 +00:00 by kgoller ce7a23c

Delete more unused files

refs #11436

2016-07-06 11:51:59 +00:00 by kgoller cd785ae

add UDP support

- icinga can now use UDP to communicate with logstash;
	switch protocol in logstash.conf

refs #11436

2016-07-06 12:07:15 +00:00 by kgoller 50e4c35

delete comment

refs #11436

2016-07-11 19:03:41 +00:00 by kgoller ea5aadf

fix several small issues

-removed some unneeded changes
-fixed several typo errors
-moved TcpSocket::Connect() into class Socket

refs #11436

2016-07-14 07:21:20 +00:00 by kgoller 1db4852

Remove pointer

refs #11436

---

Relations:

• blocks #11436

[icinga-migration]

Updated by mfriedrich on 2016-03-22 20:31:33 +00:00

• Category set to Perfdata
• Target Version set to Backlog

[icinga-migration]

Updated by mfriedrich on 2016-03-22 20:31:33 +00:00

Part of Kai's internship.

[icinga-migration]

Updated by mfriedrich on 2016-03-24 10:46:13 +00:00

• Status changed from New to Assigned
• Assigned to set to kgoller

Hi Kai,

this will be part of your project in the next months. We will discuss it further then.

Kind regards,
Michael

[icinga-migration]

Updated by kowalskimn on 2016-03-24 16:23:30 +00:00

widhalmt wrote:

The GELF protocol can use UDP or TCP according to the documentation.

Since some GELF receivers (like Logstash) can only receive UDP and the Icinga 2 GELF writer can only send TCP we have the situation that 2 tools talking the same protocol still can't talk to each other.

Could you add a protocol switcher to the GELF writer object?

I can already tell you it won't be enough to get logstash working.

There is a plugin for logstash that grabs gelf over tcp, and it's having issues with format of various fields. Not sure if that's for logstash or icinga2 to fix.

[icinga-migration]

Updated by mfriedrich on 2016-03-29 15:08:33 +00:00

Thanks, that will also be part of Kai's task to evaluate whether the existing GELF specification works with Logstash or if a different format specification is required.

[icinga-migration]

Updated by mfriedrich on 2016-03-29 15:09:09 +00:00

• Subject changed from Add UDP communication to GELF writer to _Evaluate whether the GelfWriter works with Logstash _

[icinga-migration]

Updated by mfriedrich on 2016-04-18 11:36:19 +00:00

UDP

https://www.cs.rutgers.edu/~pxk/417/notes/sockets/udp.html

Logstash Input

UDP, JSON?

https://www.elastic.co/guide/en/logstash/current/input-plugins.html
https://github.com/logstash-plugins/logstash-input-udp/blob/master/lib/logstash/inputs/udp.rb
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-udp.html

https://www.elastic.co/guide/en/logstash/current/config-examples.html
http://untergeek.com/2012/10/11/getting-apache-to-output-json-for-logstash/
https://docs.saltstack.com/en/latest/ref/configuration/logging/handlers/salt.log.handlers.logstash\_mod.html
https://github.com/logstash/logstash-logback-encoder
http://untergeek.com/2012/10/11/using-rsyslog-to-send-pre-formatted-json-to-logstash/

https://github.com/sazze/winston-logstash-udp/blob/development/lib/winston-logstash-udp.js

[icinga-migration]

Updated by mfriedrich on 2016-04-18 11:46:34 +00:00

Logstash GELF Input

https://github.com/logstash-plugins/logstash-input-gelf/blob/master/lib/logstash/inputs/gelf.rb

# The main use case for this input is to leverage existing GELF
# logging libraries such as the GELF log4j appender. A library used
# by this plugin has a bug which prevents it parsing uncompressed data.

[icinga-migration]

Updated by mfriedrich on 2016-09-28 13:49:04 +00:00

• Target Version changed from Backlog to 2.7.0

[icinga-migration]

Updated by mfriedrich on 2016-11-09 15:11:53 +00:00

• Assigned to changed from kgoller to mfriedrich

[icinga-migration]

Updated by mfriedrich on 2016-11-09 15:13:30 +00:00

• Blocked set to 13111

[dnsmichi]

Code was not in a good shape, and at least the performance data part is still incomplete.

Tests

Use the icinga2x-elastic box. Logstash 5.x requires the udp plugin to be manually installed.

/usr/share/logstash/bin/logstash-plugin install logstash-input-udp logstash-codec-json
systemctl restart logstash

Configure the input and output.

vim /etc/logstash/conf.d/icinga2.conf

input {
  udp {
    port => 5555
    codec => json
  }
}
output {
  elasticsearch {
    hosts => [ "localhost:9200" ]
  }
}

Then point Icinga 2 to it inside features-enabled/logstash.conf.

object LogstashWriter "logstash" {
  host = "192.168.33.7"
  port = 5555
  //port = 9201
  //socket_type = "udp"
}

Note: The logstash input port is freely assignable, there is no sane default.

Discussion

I'm not entirely convinced if the udp input makes sense since both logstash and elasticsearch provide a HTTP API. Elastic beats are providing output methods for both interfaces, so they probably do not differ that much.

The JSON blob is entirely written to Logstash, so you'll need to use the json codec on your own if your prefer just that.

[sgovaerts]

@dnsmichi Any update about this?

[dnsmichi]

Still on my TODO list. Mainly tests and docs are missing.

[dnsmichi]

I'm postponing this feature until there is a general decision about an ElasticWriter feature. There's a possible sponsor for that.

[sgovaerts]

@dnsmichi Just wondering if anything's changed..

[dnsmichi]

There are plans to introduce a generic ElasticWriter, sponsored by a customer. Nothing new in this issue though.

[sgovaerts]

Ok, thank you for the quick response. Was just wondering since I couldn't find anything under ElasticWriter.

[dnsmichi]

We'll create an issue once the spec and assigned devs are clarified.

[dnsmichi]

Closing this in favor of #5538.