[dev.icinga.com #11855] logrotate uses incorrect group on CentOS 7

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/11855

Created by rbu on 2016-05-29 09:31:40 +00:00

Assignee: mfriedrich
Status: Assigned
Target Version: (none)
Last Update: 2016-06-13 06:55:06 +00:00 (in Redmine)

Icinga Version: 2.4.10
Backport?: Not yet backported
Include in Changelog: 1

---

logrotate errors on the first day:

/etc/cron.daily/logrotate:

error: error setting owner of /var/log/icinga2/icinga2.log-20160527.gz to uid 994 and gid 991: Operation not permitted

Then, on subsequent days, it errors with:

error: error creating output file /var/log/icinga2/icinga2.log-20160527.gz: File exists

The directory after the logrotate run looks like this:

drwxr-x---  4 icinga icingacmd  4096 29. Mai 03:39 .
drwxr-xr-x 10 root   root       4096 29. Mai 03:39 ..
drwxr-x---  3 icinga icingacmd  4096 26. Mai 22:02 compat
drwxr-x---  2 icinga icinga     4096 19. Mai 14:27 crash
-rw-------  1 icinga icingacmd     0 26. Mai 22:03 error.log
-rw-r--r--  1 icinga icinga    44694 29. Mai 10:56 icinga2.log
-rw-r--r--  1 icinga icingacmd 56791 28. Mai 03:06 icinga2.log-20160528
-rw-------  1 icinga icinga        0 29. Mai 03:39 icinga2.log-20160528.gz

The user ids are as follows:

# id icinga
uid=994(icinga) gid=992(icinga) groups=992(icinga),991(icingacmd)

The logorate script contains:

    su icinga icinga
    compress
    delaycompress
...
    create 644 icinga icinga

The root of the problem seems to be the following: Icinga, when started for the first time and at every service restart (with systemctl), will ensure that its log file is owned by icinga:icingacmd (994:991). logrotate changes to icinga:icinga (994:992) and performs the following actions:

• rename the existing log file to append the date (keeping permissions with gid 991)
• create a new file for icinga to write to with gid 992
• create a new file to save compressed content to, attempting to keep the previous log file's group (gid 991).

That is where it fails.

My first guess for a resolution would be that if icinga2 wants the group to be icingacmd, then the logrotate recipe must use that group too.

Versions:

• I have installed icinga2 from the "release" packages for EPEL at http://packages.icinga.org/
• icinga2.x86_64 0:2.4.10-1.el7.centos
• CentOS 7
• logrotate 3.8.6

This issue is a reoccurrence of #8868, but due to another cause it seems to me.

---

Relations:

• relates #9249
• relates #8868

[icinga-migration]

Updated by dgoetz on 2016-06-01 14:17:51 +00:00

So the create directive is wrong when Icinga 2 expects ownership icinga:icingacmd, but this is only for creation of the new file after rotation and before the postscript.
The problem is perhaps the su directive added in the latest versions, which forces logrotate to do the rotation not as the default user (root) so instead using icinga:icinga. But from the permissions on the directory this should not be a problem.

Can you try "su icinga icingacmd" in the logrotation and if it works fine?

Also I am quite not sure if the su directive is required at all. Typically it is used to avoid insecure privileges Icinga 2 does not assign to the log directory.

[icinga-migration]

Updated by wols on 2016-06-06 10:58:45 +00:00

Is patched on Gentoo icinga-2.4.10-r1 and works.

A pull request is here https://github.com/wols/icinga2/pull/1

[icinga-migration]

Updated by mfriedrich on 2016-06-06 12:46:57 +00:00

• Relates set to 9249

[icinga-migration]

Updated by mfriedrich on 2016-06-06 12:47:07 +00:00

• Relates set to 8868

[icinga-migration]

Updated by mfriedrich on 2016-06-06 15:58:02 +00:00

#9249 already adds the "su" usage. The Gentoo patch just adds another hardcoded entry which is not required on CentOS7.

That's the config of a fresh centos Docker container with icinga2 v2.4.10 installed.

[root@60a806c67ed3 /]# cat /etc/logrotate.d/icinga2
/var/log/icinga2/icinga2.log /var/log/icinga2/debug.log {
    daily
    rotate 7
    su icinga icinga
    compress
    delaycompress
    missingok
    notifempty
    create 644 icinga icinga
    postrotate
        /bin/kill -USR1 $(cat /run/icinga2/icinga2.pid 2> /dev/null) 2> /dev/null || true
    endscript
}

/var/log/icinga2/error.log {
    daily
    rotate 90
    su icinga icinga
    compress
    delaycompress
    missingok
    notifempty
    create 644 icinga icinga
    # TODO: figure out how to get Icinga to re-open this log file
}

I would rather suspect the package setting the wrong directory ownership (icinga:icingacmd) for /var/log/icinga2 which somehow influences the file creation/rotation here in combination with SELinux.

[icinga-migration]

Updated by rbu on 2016-06-07 20:01:32 +00:00

dnsmichi wrote:

That's the config of a fresh centos Docker container with icinga2 v2.4.10 installed.

I can confirm that is the same configuration I had a problem with. Can you reproduce the issue in your container?

I can also confirm now that setting this config fixes the bug:

/var/log/icinga2/icinga2.log /var/log/icinga2/debug.log {
        daily
        rotate 7
        su icinga icingacmd
        compress
        delaycompress
        missingok
        notifempty
        create 644 icinga icingacmd
        postrotate
                /bin/kill -USR1 $(cat /run/icinga2/icinga2.pid 2> /dev/null) 2> /dev/null || true
        endscript
}

I would rather suspect the package setting the wrong directory ownership (icinga:icingacmd) for /var/log/icinga2 which somehow influences the file creation/rotation here in combination with SELinux.

The system in question has SELinux disabled. The problem is that icinga changes the file ownership (group) when it is restarted.
So with the original config, the logrotate run creates a new file "icinga2.log" owned by group icinga. Then icinga2 is restarted, it changes the file ownership to icingacmd. Then the next time logrotate runs, it renames the log file to icinga2.log-20160528, but fails to change the ownership as it has a different group.

I understand that running as user "icinga", logrotate should have the permission to write to the directory in question. So no idea why this happens.

[icinga-migration]

Updated by gbeutner on 2016-06-13 06:55:06 +00:00

• Status changed from New to Assigned
• Assigned to set to mfriedrich

[xenuser]

I can confirm this on SLES 12 SP1. The configuration should use icinga:icingacmd instead of icinga:icinga for rotating the logs.

[xenuser]

Sorry for that double post.

This logrotate config works for me on SLES 12 SP1:

/var/log/icinga2/icinga2.log /var/log/icinga2/debug.log {
daily
rotate 7
su icinga icingacmd
compress
delaycompress
missingok
create 644 icinga icingacmd
postrotate
/bin/kill -USR1 $(cat /run/icinga2/icinga2.pid 2> /dev/null) 2> /dev/null || true
endscript
}

/var/log/icinga2/error.log {
daily
rotate 90
su icinga icingacmd
compress
delaycompress
missingok
create 644 icinga icingacmd
}

It's a slightly modified version of the config which ships with the official package.

[maccadia]

Hi,

I get a similar issue on Ubuntu 16.04 :

/etc/cron.daily/logrotate:
error: destination /var/log/icinga2/icinga2.log.1.gz already exists, renaming to /var/log/icinga2/icinga2.log.1.gz-2018040306.backup
error: error setting owner of /var/log/icinga2/icinga2.log.1.gz to uid 114 and gid 33: Operation not permitted
run-parts: /etc/cron.daily/logrotate exited with return code 1

$ls -la /var/log/icinga2/
total 14260
drwxr-s--x  4 nagios adm          4096 avril  3 06:25 ./
drwxr-xr-x 15 root   syslog       4096 avril  3 06:25 ../
drwxr-s--x  3 nagios adm          4096 mars   8 16:04 compat/
drwxr-s--x  2 nagios adm          4096 janv. 17 12:40 crash/
-rw-r--r--  1 nagios www-data 10531040 avril  3 10:13 icinga2.log
-rw-rw----  1 nagios www-data  4042222 mars  23 06:25 icinga2.log.1
-rw-------  1 nagios adm             0 avril  3 06:25 icinga2.log.1.gz
-rw-------  1 nagios adm             0 mars  24 06:25 icinga2.log.1.gz-2018032506.backup
-rw-------  1 nagios adm             0 mars  25 06:25 icinga2.log.1.gz-2018032606.backup
-rw-------  1 nagios adm             0 mars  26 06:25 icinga2.log.1.gz-2018032706.backup
-rw-------  1 nagios adm             0 mars  27 06:25 icinga2.log.1.gz-2018032806.backup
-rw-------  1 nagios adm             0 mars  28 06:25 icinga2.log.1.gz-2018032906.backup
-rw-------  1 nagios adm             0 mars  29 06:25 icinga2.log.1.gz-2018033006.backup
-rw-------  1 nagios adm             0 mars  30 06:25 icinga2.log.1.gz-2018033106.backup
-rw-------  1 nagios adm             0 mars  31 06:25 icinga2.log.1.gz-2018040106.backup
-rw-------  1 nagios adm             0 avril  1 06:25 icinga2.log.1.gz-2018040206.backup
-rw-------  1 nagios adm             0 avril  2 06:25 icinga2.log.1.gz-2018040306.backup

$ sudo cat /etc/logrotate.d/icinga2 
/var/log/icinga2/icinga2.log /var/log/icinga2/debug.log {
	daily
	rotate 7
	su nagios nagios
	compress
	delaycompress
	missingok
	notifempty
	create 644 nagios nagios
	postrotate
		/bin/kill -USR1 $(cat /run/icinga2/icinga2.pid 2> /dev/null) 2> /dev/null || true
	endscript
}

/var/log/icinga2/error.log {
	daily
	rotate 90
	su nagios nagios
	compress
	delaycompress
	missingok
	notifempty
	create 644 nagios nagios
	# TODO: figure out how to get Icinga to re-open this log file
}

Replacing nagios nagios with nagios www-data in /etc/logrotate.d/icinga2 fix this.

[dnsmichi]

I guess there isn't a specific generic solution for this, packages might need to provide different configuration adjustments here. Modify at your own will in the meantime, we won't fix this here.

[maccadia]

I agree, but could this bug be sent / assigned to packaging team ?