Date: Wed, 5 Oct 2011 01:00:25 +0100 From: "Steven Hartland" <killing@multiplay.co.uk> To: <freebsd-stable@freebsd.org> Cc: liv3d@multiplay.co.uk Subject: serious packet routing issue causing ntpd high load? Message-ID: <2740DA0E7C0F495BA00E6D4BC0FE8F37@multiplay.co.uk>
next in thread | raw e-mail | index | archive | help
We just updated a machine to 8-STABLE and I've noticed
that ntpd is using notible amounts of CPU 5-7% which
is very high for such a trivial daemon.
8.2-STABLE FreeBSD 8.2-STABLE #16: Tue Oct 4 09:53:17 UTC 2011
truss indicates its constantly checking and reading
from a socket
0.047297485 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1 (0x1)
0.047513160 clock_gettime(0,{1317770389.969538247 }) = 0 (0x0)
0.047604515 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 }) = 1 (0x1)
0.047668212 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) = 184 (0xb8)
0.049395293 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1 (0x1)
0.049503689 clock_gettime(0,{1317770389.971526820 }) = 0 (0x0)
0.049606219 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 }) = 1 (0x1)
0.049669916 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) = 184 (0xb8)
0.049809882 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1 (0x1)
...
running with debug enabled it sits looping outputting:-
routing message op = 7: ignored
routing message op = 7: ignored
routing message op = 7: ignored
routing message op = 7: ignored
routing message op = 7: ignored
routing message op = 7: ignored
routing message op = 7: ignored
...
It seems socket 28 is a duplicate of an internal routing socket
as seen here in the trace:-
0.044544269 socket(PF_ROUTE,SOCK_RAW,0) = 4 (0x4)
0.044595394 fcntl(4,F_DUPFD,0x14) = 28 (0x1c)
0.044645960 close(4) = 0 (0x0)
0.044695968 fcntl(28,F_SETFL,O_NONBLOCK) = 0 (0x0)
Now this looks like its RTM_MISS as defined:-
sys/net/route.h:#define RTM_MISS 0x7 /* Lookup failed on this address */
So the question was why is PF_ROUTE socket constantly
spamming RTM_MISS?
route -n monitor on this machines shows:-
got message of size 184 on Tue Oct 4 23:46:36 2011
RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno 0, flags:<DONE>
locks: inits:
sockaddrs: <DST>
::A.B.C.D
got message of size 184 on Tue Oct 4 23:46:36 2011
RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno 0, flags:<DONE>
locks: inits:
sockaddrs: <DST>
::A.B.C.D
This seems very much like the following pr which was fixed:-
"Remove a bogusly introduced rtalloc_ign() in rev. 1.335/SVN 178029,
generating an RTM_MISS for every IP packet forwarded making user space
routing daemons unhappy":-
http://www.freebsd.org/cgi/query-pr.cgi?pr=124540
The box is doing no routing, its fairly basic install with
1 main IP on em0 + 1 alias + gw addres and 1 private ip on em1.
Its running mysql and thats about it.
Any ideas?
Regards
Steve
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2740DA0E7C0F495BA00E6D4BC0FE8F37>