Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 12 May 2015 03:08:18 +0000
From:      bugzilla-noreply@freebsd.org
To:        freebsd-bugs@FreeBSD.org
Subject:   [Bug 200139] Auditdistd suddenly stops working and leaves untransmitted files.
Message-ID:  <bug-200139-8@https.bugs.freebsd.org/bugzilla/>

next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200139

            Bug ID: 200139
           Summary: Auditdistd suddenly stops working and leaves
                    untransmitted files.
           Product: Base System
           Version: 11.0-CURRENT
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: bin
          Assignee: freebsd-bugs@FreeBSD.org
          Reporter: peter@FreeBSD.org
                CC: pjd@FreeBSD.org

I'm not sure why, but we are routinely getting an auditidistd failure in the
freebsd.org cluster.

A couple of log samples and data points..

root@ref11-amd64:~ # ls -l /var/audit/dist/
total 262049
-r--r-----  2 root  audit  10535025 May 11 23:00 20150511225958.20150511230009
-r--r-----  2 root  audit  11280061 May 11 23:23 20150511232340.20150511232357
-r--r-----  2 root  audit  10521949 May 11 23:24 20150511232417.20150511232437
-r--r-----  2 root  audit  15688526 May 11 23:25 20150511232533.20150511232553
-r--r-----  2 root  audit  15477527 May 11 23:27 20150511232646.20150511232704
-r--r-----  2 root  audit  13751547 May 11 23:27 20150511232741.20150511232753
-r--r-----  2 root  audit   9764920 May 11 23:28 20150511232840.20150511232849
-r--r-----  2 root  audit  10504847 May 11 23:29 20150511232904.20150511232912
-r--r-----  2 root  audit   9942299 May 11 23:30 20150511233004.20150511233010
-r--r-----  2 root  audit  23953683 May 11 23:30 20150511233010.20150511233025
-r--r-----  2 root  audit  10894790 May 11 23:30 20150511233031.20150511233038
-r--r-----  2 root  audit   7714981 May 11 23:31 20150511233056.20150511233101
-r--r-----  2 root  audit   7938553 May 11 23:31 20150511233138.20150511233145
-r--r-----  2 root  audit  12498960 May 11 23:31 20150511233145.20150511233156
-r--r-----  2 root  audit  10511961 May 11 23:33 20150511233258.20150511233308
-r--r-----  2 root  audit  11620541 May 11 23:33 20150511233308.20150511233319
-r--r-----  2 root  audit  10507759 May 11 23:33 20150511233343.20150511233352
-r--r-----  2 root  audit  12708772 May 11 23:35 20150511233446.20150511233500
-r--r-----  2 root  audit  10542898 May 11 23:35 20150511233524.20150511233538
-r--r-----  2 root  audit  10564417 May 11 23:36 20150511233641.20150511233655
-r--r-----  2 root  audit  10499802 May 11 23:37 20150511233728.20150511233739
-r--r-----  2 root  audit  10571378 May 11 23:38 20150511233809.20150511233818
-r--r-----  2 root  audit   8769659 May 12 02:59 20150512011254.not_terminated

root@ref11-amd64:~ # grep 20150511233809.20150511233818 /var/log/messages
root@ref11-amd64:~ # bzgrep 20150511233809.20150511233818 /var/log/messages*
/var/log/messages.0.bz2:May 11 23:38:24 ref11-amd64 auditd[797]: renamed
/var/audit/20150511233809.not_terminated to
/var/audit/20150511233809.20150511233818
/var/log/messages.0.bz2:May 11 23:38:24 ref11-amd64 root: audit warning:
closefile /var/audit/20150511233809.20150511233818


root@ref11-amd64:~ # bzgrep 20150511233809.20150511233818 /var/log/messages*
/var/log/messages.0.bz2:May 11 23:38:24 ref11-amd64 auditd[797]: renamed
/var/audit/20150511233809.not_terminated to
/var/audit/20150511233809.20150511233818
/var/log/messages.0.bz2:May 11 23:38:24 ref11-amd64 root: audit warning:
closefile /var/audit/20150511233809.20150511233818

root@ref11-amd64:~ # bzgrep auditdist /var/log/messages*
/var/log/messages.0.bz2:May 11 23:00:16 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511225958.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:24:15 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511232340.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:24:51 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511232417.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:26:10 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511232533.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:27:18 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511232646.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:28:07 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511232741.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:29:02 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511232840.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:29:35 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511232904.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:30:22 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511233004.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:30:28 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511233010.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:30:52 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511233031.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:31:08 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511233056.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:31:54 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511233138.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:32:00 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511233145.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:33:18 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511233258.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:33:24 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511233308.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:33:58 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511233343.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:35:08 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511233446.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:35:49 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511233524.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:37:02 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511233641.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:37:44 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511233728.not_terminated", skipping: No such file or
directory.
/var/log/messages.0.bz2:May 11 23:38:24 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511233809.not_terminated", skipping: No such file or
directory.

This is the most telling:

/var/log/messages.0.bz2:May 11 23:00:05 ref11-amd64 auditd[797]: renamed
/var/audit/20150511225947.not_terminated to
/var/audit/20150511225947.20150511225958
/var/log/messages.0.bz2:May 11 23:00:05 ref11-amd64 auditd[797]: New audit file
is /var/audit/20150511225958.not_terminated
/var/log/messages.0.bz2:May 11 23:00:05 ref11-amd64 root: audit warning:
closefile /var/audit/20150511225947.20150511225958
/var/log/messages.0.bz2:May 11 23:00:15 ref11-amd64 auditd[797]: renamed
/var/audit/20150511225958.not_terminated to
/var/audit/20150511225958.20150511230009
/var/log/messages.0.bz2:May 11 23:00:16 ref11-amd64 root: audit warning:
closefile /var/audit/20150511225958.20150511230009
/var/log/messages.0.bz2:May 11 23:00:16 ref11-amd64 auditdistd[84495]:
[audit.ysv.FreeBSD.org] (sender) Unable to open file
"/var/audit/dist/20150511225958.not_terminated", skipping: No such file or
directory.

It looks like when it loses a rename race (under high loads), it ignores that
audit chunk forever.

-- 
You are receiving this mail because:
You are the assignee for the bug.



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-200139-8>