This repository has been archived by the owner on Jan 15, 2019. It is now read-only.
[dev.icinga.com #2342] Long output data causes wrong data in database #873
Comments
|
Updated by mfriedrich on 2012-02-22 11:59:27 +00:00 and the database table data looks how? |
|
Updated by jmosshammer on 2012-02-22 14:03:46 +00:00 Like I said, it's written like to the database as icinga-web displays it: current_state is always zero, sometimes the check/notification parameters are modified. if it helps, long_output is truncated and completely unrelated data to the output appended at the end (in this case the DN from LConf, which should not appear in the plugin output): ,97262=DN:0:cn=xyz-check-checksum,ou=template-linux-xyz,ou=Templates,ou=LConf,dc=xyz,dc=org --------------------- --------------------- --------------------- |
|
Updated by Tommi on 2012-02-22 20:37:38 +00:00 long_output is a text column with max 2^16 =65536 bytes (we are using UTF8, means less characters than bytes) Next bigger type is longtext which allows 2^32 bytes. |
|
Updated by jmosshammer on 2012-02-23 13:52:56 +00:00 FYI, today the plugin produced less output (2200 Chars) and worked fine. I agree to Tommis solution with limiting output data (and truncating it) - it's better to have 32K of truncated output and a correct check result/no sideeffects. |
|
Updated by Tommi on 2012-03-03 12:02:04 +00:00
i will add the truncate to mysql code. Again, What about the size? In Ido2db.h is defined IDO2DB_MAX_BUFLEN=16384. In Idomod.h we have IDOMOD_MAX_BUFLEN=49152. As discussed above i would set it to the half of mysql_net_buffer=65535 => 32768, because the whole sql must fit into this buffer, not only the column we mentioned. |
|
Updated by Tommi on 2012-03-03 19:51:23 +00:00
applied in changeset 42917a36b337b62ca1e23df76348de105579e24b |
|
Updated by Tommi on 2012-03-04 13:52:24 +00:00 commit message in changeset had wrong ticket number. Did amend and set correct ticket id. After push got another changeset id. |
|
Updated by jmosshammer on 2012-03-05 07:46:28 +00:00 Thanks, we'll give it try! |
|
Updated by jmosshammer on 2012-03-15 10:01:33 +00:00 Just got feedback from our monitoring guys, now there are no more strange events like notifications disabled, etc. but the check returns status Pending and no output according to them. |
|
Updated by Tommi on 2012-03-15 19:04:06 +00:00 My patch only restricts size of long_output data, nothing more. With the information provided i cannot guess whats going wrong. Maybe you suggest a full debug trace and scan for the sql and the lines around and send me the related output together with database content and your expection how should it look like. |
|
Updated by Tommi on 2012-03-30 16:00:39 +00:00 if there are any news on this topic? Otherwise i will go to set this ticket to resolved. |
|
Updated by mfriedrich on 2012-03-31 10:55:27 +00:00 i need to test your changes. given the last feedback i would expect buggy behaviour an rather revert the commit to prevent unwanted bugs? |
|
Updated by mfriedrich on 2012-04-02 14:26:30 +00:00
mostly what i expected - now it says less data provided for writing than indicated. rhel 5.8 x64, db connection via (server version 11.2.11, client version [1110], ocilib version 3.9.2) remote oracle box for the record - i ran oracle-upgrade-1.70.sql cleanly before testing your branch. syslog ido2db.debug |
|
Updated by mfriedrich on 2012-04-02 14:27:48 +00:00 btw - i can't alter session for tracing, that's prohibited by dba. |
|
Updated by Tommi on 2012-04-02 16:03:21 +00:00 regarding 24811 message: according http://sourceforge.net/projects/orclib/forums/forum/470801/topic/5000325 this is not an error, only a warning. Content should be correct. Its related to the UTF8 vs. Ansi discussion. I tought i cleaned already the misleading message. I will check this. But anyway, it is not related to the ticket topic, its more a general lob handling issue. For enabling the trace of own sessions "alter session" priviledge is required(and documented). Its not a standard right, but is safe also from the dba perspective, especially for dev environments. This because i trapped the missing right into the log entries you mentioned to give the hint again. You can ask your DBA why it's not possible to grant this right. |
|
Updated by mfriedrich on 2012-04-02 16:10:38 +00:00 well it's treated as oci error and therefore written to syslog (or any other facility) that's why it made me nervous. it happens only on hoststatus with some check_multi html output though from a testcase in #674 |
|
Updated by Tommi on 2012-04-02 18:47:37 +00:00 OK, its not the case i assumed just before, this looks really whorse. Error is the same we had in #2303 It clams to have 1184 bytes to store. First chunk is 1024 bytes and was successfully. Next chunk should be the remaining 160bytes. Correct. Now oracle says not enough data available. But the string buffer we handover is exactly 160 Bytes long and doesnt contains any multibyte characters. Now added an explicit reset for char counter within the loop and an utf8 ready char count function. Hope this solves the problem. Finally, i reduced chunk size to 16Byte to check handover of chunks. For now its looks OK in my environment. Please check again. Finally, the chunk size in db.h should be increase before release to 1024 or more. |
|
Updated by mfriedrich on 2012-04-03 11:14:23 +00:00 thanks looks better with smaller junks and the explicit counter. i'll go retest with 1024k junks, will post an update then. |
|
Updated by mfriedrich on 2012-04-03 11:26:19 +00:00
looks good with 1024. feel free to change accordingly to your tests. |
|
Updated by Tommi on 2012-04-03 19:07:08 +00:00
fine. chunksize i set now to 2048 This is a c&p error(wrong variable assigned) and is fixed now in the same changeset |
|
Updated by mfriedrich on 2012-04-19 13:02:19 +00:00
seen that, runs fine over here as well. thanks for the fix. |
|
Updated by mfriedrich on 2013-01-02 19:20:23 +00:00
the original issue is not fixed, as it's not the long output on mysql causing data irritation, but inner idoutils buffer size limits. below is an ido2db debug log which shows an irritation between the forced service check and the status update triggered, which directly leads into buffer problems from within idomod, as data is being worked on in a sequential manner, it cannot be mixed by ido2db, only working on the received data itsself. it finally receives only garbage after the long_output has been dumped, which may be a reason for the gargabe actually dumped into the database. |
|
Updated by mfriedrich on 2013-01-02 19:53:28 +00:00 basically, idomod does not care about any string limits of its own, and creates funky strings, just truncating them in the end for the socket. a while back, i've increased that buffer size from 16k to 50k from an Opsview patch, but ever since we easily hit 50k with the given plugin long output, this just causes data irritation rather than a safety lock on the socket buffer. meaning to say - limiting the socket buffer is just plain stupid, it only makes sense if you do not give a fuck about the data/protocol being sent over. |
|
Updated by mfriedrich on 2013-01-02 20:50:13 +00:00
requires some more testing, but fixed in next and master. |
|
Updated by mfriedrich on 2013-01-16 19:55:43 +00:00
fixed in 1.8.4 already. |
|
Updated by mfriedrich on 2014-12-08 14:37:34 +00:00
|
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
This issue has been migrated from Redmine: https://dev.icinga.com/issues/2342
Created by jmosshammer on 2012-02-22 11:18:12 +00:00
Assignee: mfriedrich
Status: Resolved (closed on 2013-01-16 19:55:43 +00:00)
Target Version: 1.9
Last Update: 2014-12-08 14:37:34 +00:00 (in Redmine)
We encountered the following issue:
We have a plugin that produces verly large output (~800 Lines, not perfdata).
In Icinga-Web, we noticed that the service state is always ok, even if the classic interface showed critical/warning.
Also, the service was sometimes displayed with 'disable_notifications', sometimes the check is written as disabled to the db. (in classic everything is ok, and those flags are not set) - Perhaps there's an overflow in the idoutils?
This is written to the database in this way, so it's not an icinga-web issue.
Modifying a existing plugin like check_echo so that it produces a large output caused the same effect.
Pluginoutput looks like this:
Differences on mysql5g
TABLE CHUNK CNT_DIFF CRC_DIFF CHUNK_INDEX LOWER_BOUNDARY UPPER_BOUNDARY
blog.site_xyp 38 0 1 id 8033859 8158659
blog.site_xyp 39 0 1 id 8158661 8283521
...
inxyz.datapoint 12109 0 1 PRIMARY 776,776,4,776,4,1328652000 776,776,6,776,6,1313550000
inxyz.datapoint 12138 0 1 PRIMARY 778,778,4,778,4,1326385800 778,778,6,778,6,1312416000
inxyz_live.datapoint 28619 0 1 PRIMARY 1736,1736,4,1736,4,1327199400 1736,1736,6,1736,6,1313193600
...
(about 800 lines)
Core: 1.6
DB: MySQL 5.0
Web: 1.6.2dev
Changesets
2012-03-04 08:35:09 +00:00 by Tommi f04c704
2012-04-02 18:34:50 +00:00 by Tommi 703426f
2012-04-03 18:48:27 +00:00 by Tommi 0906122
2013-01-02 19:55:51 +00:00 by (unknown) 00aebd0
2013-01-02 20:23:00 +00:00 by (unknown) cfa3d76
Relations:
The text was updated successfully, but these errors were encountered: