Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dev.icinga.com #10480] Add InfluxDbWriter feature #3562

Closed
icinga-migration opened this issue Oct 28, 2015 · 16 comments
Closed

[dev.icinga.com #10480] Add InfluxDbWriter feature #3562

icinga-migration opened this issue Oct 28, 2015 · 16 comments
Labels
area/metrics General metrics handling enhancement New feature or request
Milestone

Comments

@icinga-migration
Copy link

This issue has been migrated from Redmine: https://dev.icinga.com/issues/10480

Created by mfriedrich on 2015-10-28 18:55:17 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2016-05-03 12:15:03 +00:00)
Target Version: 2.5.0
Last Update: 2016-05-09 09:43:57 +00:00 (in Redmine)

Backport?: Not yet backported
Include in Changelog: 1

InfluxDB 0.9.x could still use the GraphiteWriter feature although they changed the tree notation from Graphite into a tag based system similar to OpenTSDB. By default the Graphite tree is stored "flat" and does not benefit from the InfluxDB tags and indexes. So there's not really a point in using the Graphite input with InfluxDB at this point.

influxdata/influxdb#2996
influxdata/influxdb#3125

Changesets

2016-05-03 12:12:51 +00:00 by spjmurray 79c1e88

Add InfluxDB Writer

Adds an Icinga2 object to directly interface with InfluxDB's native HTTP API.
This supports optional basic authorization, and TLS transport.  InfluxDB didn't
appear to like having the TLS stream kept open, so instead this object buffers
data points which are then flushed to InfluxDB as a batch write, either driven
by a configurable timeout or threshold.

As InfluxDB is a schema-less database the host and service templates are user
configurable allowing both the measurement field and tags to be set by the
end user via macro expansion.  This allows access to tag fields from arbitrary
data associated with host.vars or service.vars.  If a particular value is
unable to be resolved, the tag will be dropped and not transmitted to InfluxDB.

Also alters URL handling to omit array brackets when only a single value is
attached to a key, otherwise InfluxDB has a strop with non-standard syntax.

fixes #10480

Signed-off-by: Michael Friedrich <michael.friedrich@netways.de>

Relations:

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2015-10-28 18:55:48 +00:00

  • Status changed from New to Assigned
  • Assigned to set to mfriedrich

I'd like to play with it in my spare time :)

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2015-10-30 23:35:20 +00:00

http://stackoverflow.com/questions/33350314/usecases-influxdb-vs-prometheus/

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2015-11-25 16:04:52 +00:00

  • Status changed from Assigned to New
  • Assigned to deleted mfriedrich

I'm postponing this until InfluxDB reaches 1.0.0 ... talked to many at OSMC and it seems 0.9.5 again broke the backend.

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2015-12-03 17:01:54 +00:00

  • Blocks set to 9072

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-03-31 06:11:24 +00:00

https://influxdata.com/blog/update-on-influxdb-clustering-high-availability-and-monetization/

@icinga-migration
Copy link
Author

Updated by spjmurray on 2016-04-12 14:44:56 +00:00

As discussed on IRC I'm happy to pick this up and maintain it. Here's what I imagine the configuration to look like:

library "perfdata"

object InfluxdbWriter "influxdb" {
  host = "influxdb.example.com"
  port = 8086
  ssl_enable = true
  ssl_ca_cert = "/var/lib/puppet/ssl/certs/ca.pem"
  ssl_cert = "/var/lib/puppet/ssl/certs/icinga2.example.com.pem"
  ssl_key = "/var/lib/puppet/ssl/private_keys/icinga2.example.com.pem"
  host_template = "$host.check_command$,hostname=$host.name$domain$"
  service_template = "$service.check_command$,hostname=$host.name$,service=$service.name$"
  enable_send_thresholds = true
  enable_send_metadata = false
}

The important bits:

  • Optional server side SSL support with ssl_enable and ssl_ca_cert
  • Optional mutually verified SSL support with ssl_enable, ssl_ca_cert, ssl_cert and ssl_key
  • Flexible configuration allows the measurement and tags to be specified to your preference
  • Tags can be utterly arbitrary e.g. domain=$host.vars.domain$
  • Icinga2 implicitly adds metric and type tags which maps to a nagios perfdata label and a perfdata element (e.g. value, warn, crit. min, max) respectively

InfluxDB also advises not to have more that one piece of data in a tag. Say if we had an MTU check applied to an interface we'd potentially generate mtueth0 as the service name from the check command mtu and the key we applied the check to eth0. In this case the mtu part is redundant as it's already available as $service.check_command$ and all we really care about is this data point is tagged to the interface eth0. This could be possibly surfaced by attaching the key to the service object.

Another idea is to make tags programmable:

service_template = {
  measurement = "$service.check_command$"
  tags        = {
    hostname = "$host.name$"
    instance = "$service.instance$"
  }
}

In this example rather than encoding the template as a fixed string, it can be encoded as a hash. The measurement field is required, tags is optional. Any unresolved macros are omitted from the final set of tags to be sent to InfluxDB.

All I can think of right about now. Feel free to highlight any technical snafus I may be making.

@icinga-migration
Copy link
Author

Updated by spjmurray on 2016-04-14 13:31:39 +00:00

Working branch: https://github.com/spjmurray/icinga2/tree/feature/influxdb-writer-10480

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-04-19 15:24:52 +00:00

  • Status changed from New to Assigned
  • Assigned to set to mfriedrich
  • Target Version changed from Backlog to 2.5.0

You're way too fast for us catching up with a review. Much appreciated though :)

I've seen your blog post, leaving as a reference here: http://www.datacentred.co.uk/blog/integrating-icinga2-with-influxdb-and-grafana/

I'll take it for a review though it might take a while. Since it is a new feature it should target 2.5.0 then.

Kind regards,
Michael

@icinga-migration
Copy link
Author

Updated by spjmurray on 2016-04-19 16:24:53 +00:00

shroud has already kicked my ass into shape so most of the major points should have been fixed! Bloody hell, the marketing droids don't hang about... well at least you get a bigger picture of what this is about.

@icinga-migration
Copy link
Author

Updated by spjmurray on 2016-05-03 12:15:03 +00:00

  • Status changed from Assigned to Resolved
  • Done % changed from 0 to 100

Applied in changeset 79c1e88.

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-05-03 12:35:40 +00:00

You are actually using the http bits from libremote and fixed a bug over there, nice :)

The rest looks like you know your development foo, also helped by shroud :)

I've partially updated the documentation for addons/features and amended those changes into your patch.

I have no objections against the patch though I'm lacking a test setup for now. Once the snapshot packages are built, please invite others to test them before they are hitting the release 2.5.0 then.

@icinga-migration
Copy link
Author

Updated by spjmurray on 2016-05-05 05:26:05 +00:00

Thanks Michael. I take it I can start testing the snapshots in staging once 2.4.8 is released?

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-05-05 07:03:16 +00:00

Snapshot packages are built on each git commit for el7 (vagrant/docker trsts) and via cron for all other distributions once a day. You don't have to wait for the bugfix release 2.4.8, that release will just backport certsin fixes from the master development branch which currently targets 2.5.0 :)

TL;DR - start now.

@icinga-migration
Copy link
Author

Updated by spjmurray on 2016-05-09 09:43:57 +00:00

Soak test started with 2.4.7+snapshot2016.05.09+1~trusty as of 10:41 09/05/16...

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-06-07 19:00:36 +00:00

  • Relates set to 11912

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2016-06-07 20:01:58 +00:00

  • Relates set to 11904

@icinga-migration icinga-migration added enhancement New feature or request area/metrics General metrics handling labels Jan 17, 2017
@icinga-migration icinga-migration added this to the 2.5.0 milestone Jan 17, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/metrics General metrics handling enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant