remote_syslog 1.6.14 keeps deleted logfiles open

[rolandkool]

We noticed that remote_syslog is keeping logfiles open that have been deleted.
An example lsof:
ruby 4894 root 10r REG 253,2 151396 65567 /path/to/catalina.out (deleted)
ruby 4894 root 11r DIR 0,10 0 649 inotify
ruby 4894 root 12r REG 253,2 151396 65567 /path/to/catalina.out (deleted)
ruby 4894 root 13r REG 253,2 13646 65556 /path/to/catalina.out (deleted)
ruby 4894 root 14r REG 253,7 29238 507906 /path/to/catalina.out

Is there a way, besides restarting the service, to prevent this or get rid of them?

[joesolly]

This happens when the files are symbolic links too, which means we can't stop logging by just removing a symbolic link but instead have to restart the service.

[ChrisLundquist]

Still happens in remote_syslog (1.6.15)

[ChrisLundquist]

This seems to be an interaction with logrotate.

$ sudo lsof | grep deleted
# many other remote_syslog files that are deleted
# ...
remote_sy 26216 26218       root   25r      REG              202,1 391179605     176530 /var/log/nginx/core.access.log.1 (deleted)

The default logrotate conf that ubuntu 14.04 ships

/var/log/nginx/*.log {
    weekly
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    prerotate
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
            run-parts /etc/logrotate.d/httpd-prerotate; \
        fi \
    endscript
    postrotate
        [ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
    endscript
}

and my log_files.yml looks like:

files:
   - # stuff
   - /var/log/nginx/core.access.log
destination:
   host: logs.papertrailapp.com
   port: #####

[troy]

I think @ChrisLundquist's problem is unrelated to @rolandkool's. @ChrisLundquist: check out copytruncate; that may prevent the leftover open file descriptors.

[ChrisLundquist]

@troy Thanks for the speedy reply!

Yeah, my goal in the above was to give to simplest way to replicate an issue. You're right that the default nginx logrotate configuration doesn't use copytruncate. I inherited this setup, so it is possible there was documentation that got missed in the implementation. It seems to me that using papertrail with nginx is a pretty common use case. In my ideal world, remote_syslog would work with a vanilla sudo apt-get install nginx.

My rails app logs use copytruncate and see a similar issue.

$ sudo lsof | grep deleted
remote_sy 26216             root   23r      REG              202,1   2295124      86059 /path/to/rails/app/e2e590b3d1b4942191c7fd20fbc74997d8b9d63c/build/log/production.log (deleted)

$ cat /etc/logrotate.d/core_service
/path/to/rails/app/current/log/*.log {
  daily
  missingok
  rotate 7
  compress
  delaycompress
  notifempty
  copytruncate
}

This could be related to the symlink issue, but I'm not entirely sure. current is a symlink, but it looks like File.expand_path is called to resolve *.log to an absolute path.

[troy]

@ChrisLundquist no problem for the quick reply. I'm going to transition your last email into a thread in our support system (so we can explore your environment more specifically), so please expect an email from support@ late Wednesday or Thursday US time.