File Handles Left Open for Logs #720
Description
It appears TimedRotatingFileHandler is not handling multiple writer threads well. http://thinlight.org/2011/08/10/python-logging-from-multiple-processes/ is an article highlighting the issue.
I ran a test with the vanilla agent installed on Ubuntu 16.04 on Aug 31st and found that logs are getting rotated, but the original rotated log file is kept open.
root@ip-x-x-x-x:/var/log/diamond# ls -lh
total 25M
-rw-r--r-- 1 root root 1.2M Sep 4 13:04 archive.log
-rw-r--r-- 1 root root 533K Aug 31 23:59 archive.log.2018-08-31
-rw-r--r-- 1 root root 2.1M Sep 1 23:59 archive.log.2018-09-01
-rw-r--r-- 1 root root 2.1M Sep 2 23:59 archive.log.2018-09-02
-rw-r--r-- 1 root root 2.1M Sep 3 23:59 archive.log.2018-09-03
-rw-r--r-- 1 root root 2.5M Sep 4 13:04 diamond.log
-rw-r--r-- 1 root root 1.2M Aug 31 23:59 diamond.log.2018-08-31
-rw-r--r-- 1 root root 4.5M Sep 1 23:59 diamond.log.2018-09-01
-rw-r--r-- 1 root root 4.5M Sep 2 23:59 diamond.log.2018-09-02
-rw-r--r-- 1 root root 4.5M Sep 3 23:59 diamond.log.2018-09-03
root@ip-x-x-x-x:/var/log/diamond# lsof | grep diamond
bash 16131 theconnman cwd DIR 202,1 4096 305452 /var/log/diamond
sudo 16189 root cwd DIR 202,1 4096 305452 /var/log/diamond
su 16190 root cwd DIR 202,1 4096 305452 /var/log/diamond
bash 16191 root cwd DIR 202,1 4096 305452 /var/log/diamond
lsof 16635 root cwd DIR 202,1 4096 305452 /var/log/diamond
grep 16636 root cwd DIR 202,1 4096 305452 /var/log/diamond
lsof 16637 root cwd DIR 202,1 4096 305452 /var/log/diamond
python 30116 root 2w REG 202,1 545051 305510 /var/log/diamond/archive.log.2018-08-31
python 30116 root 3w REG 202,1 1194245 305507 /var/log/diamond/diamond.log.2018-08-31
python 30117 root 3w REG 202,1 1194245 305507 /var/log/diamond/diamond.log.2018-08-31
python 30117 30127 root 3w REG 202,1 1194245 305507 /var/log/diamond/diamond.log.2018-08-31
python 30117 30378 root 3w REG 202,1 1194245 305507 /var/log/diamond/diamond.log.2018-08-31
python 30117 30379 root 3w REG 202,1 1194245 305507 /var/log/diamond/diamond.log.2018-08-31
python 30117 30406 root 3w REG 202,1 1194245 305507 /var/log/diamond/diamond.log.2018-08-31
python 30117 30420 root 3w REG 202,1 1194245 305507 /var/log/diamond/diamond.log.2018-08-31
python 30117 30447 root 3w REG 202,1 1194245 305507 /var/log/diamond/diamond.log.2018-08-31
python 30117 30473 root 3w REG 202,1 1194245 305507 /var/log/diamond/diamond.log.2018-08-31
python 30125 root 2w REG 202,1 1173857 305523 /var/log/diamond/archive.log
python 30125 root 3w REG 202,1 2571314 305522 /var/log/diamond/diamond.log
python 30128 root 2w REG 202,1 545051 305510 /var/log/diamond/archive.log.2018-08-31
python 30128 root 3w REG 202,1 1194245 305507 /var/log/diamond/diamond.log.2018-08-31
python 30130 root 2w REG 202,1 545051 305510 /var/log/diamond/archive.log.2018-08-31
python 30130 root 3w REG 202,1 1194245 305507 /var/log/diamond/diamond.log.2018-08-31
python 30132 root 2w REG 202,1 545051 305510 /var/log/diamond/archive.log.2018-08-31
python 30132 root 3w REG 202,1 1194245 305507 /var/log/diamond/diamond.log.2018-08-31
python 30135 root 2w REG 202,1 545051 305510 /var/log/diamond/archive.log.2018-08-31
python 30135 root 3w REG 202,1 1194245 305507 /var/log/diamond/diamond.log.2018-08-31
python 30139 root 2w REG 202,1 545051 305510 /var/log/diamond/archive.log.2018-08-31
python 30139 root 3w REG 202,1 1194245 305507 /var/log/diamond/diamond.log.2018-08-31
python 30143 root 2w REG 202,1 545051 305510 /var/log/diamond/archive.log.2018-08-31
python 30143 root 3w REG 202,1 1194245 305507 /var/log/diamond/diamond.log.2018-08-31