From owner-freebsd-stable@FreeBSD.ORG Wed Oct 5 23:43:44 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4FC10106564A for ; Wed, 5 Oct 2011 23:43:44 +0000 (UTC) (envelope-from qing.li@bluecoat.com) Received: from whisker.bluecoat.com (whisker.bluecoat.com [216.52.23.28]) by mx1.freebsd.org (Postfix) with ESMTP id 096D38FC0A for ; Wed, 5 Oct 2011 23:43:43 +0000 (UTC) Received: from PWSVL-EXCHTS-01.internal.cacheflow.com ([10.2.2.122]) by whisker.bluecoat.com (8.14.2/8.14.2) with ESMTP id p95NWuNR016745 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Wed, 5 Oct 2011 16:32:56 -0700 (PDT) Received: from PWSVL-EXCMBX-01.internal.cacheflow.com ([fe80::15bc:12e2:4676:340f]) by PWSVL-EXCHTS-01.internal.cacheflow.com ([fe80::5c50:e2ba:8115:4223%20]) with mapi id 14.01.0289.001; Wed, 5 Oct 2011 16:32:50 -0700 From: "Li, Qing" To: Steven Hartland , "freebsd-stable@freebsd.org" Thread-Topic: serious packet routing issue causing ntpd high load? Thread-Index: AQHMgvN1PVqhp1q7OEaxelBmE0OkKZVuZ0Aw Date: Wed, 5 Oct 2011 23:32:50 +0000 Message-ID: References: <2740DA0E7C0F495BA00E6D4BC0FE8F37@multiplay.co.uk> In-Reply-To: <2740DA0E7C0F495BA00E6D4BC0FE8F37@multiplay.co.uk> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.2.2.106] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "liv3d@multiplay.co.uk" Subject: RE: serious packet routing issue causing ntpd high load? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Oct 2011 23:43:44 -0000 Hi, > > RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno > 0, flags: > locks: inits: > sockaddrs: > ::A.B.C.D > Would it be possible for you to email me what exactly does "::A.B.C.D"=20 map into WRT your system or infrastructure ? And are you able to share your "ifconfig -a" and "netstat -rn" output with me privately ? --Qing > -----Original Message----- > From: owner-freebsd-stable@freebsd.org [mailto:owner-freebsd- > stable@freebsd.org] On Behalf Of Steven Hartland > Sent: Tuesday, October 04, 2011 5:00 PM > To: freebsd-stable@freebsd.org > Cc: liv3d@multiplay.co.uk > Subject: serious packet routing issue causing ntpd high load? >=20 > We just updated a machine to 8-STABLE and I've noticed > that ntpd is using notible amounts of CPU 5-7% which > is very high for such a trivial daemon. >=20 > 8.2-STABLE FreeBSD 8.2-STABLE #16: Tue Oct 4 09:53:17 UTC 2011 >=20 > truss indicates its constantly checking and reading > from a socket >=20 > 0.047297485 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) =3D 1 > (0x1) > 0.047513160 clock_gettime(0,{1317770389.969538247 }) =3D 0 (0x0) > 0.047604515 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 }) > =3D 1 (0x1) > 0.047668212 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) =3D 184 > (0xb8) > 0.049395293 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) =3D 1 > (0x1) > 0.049503689 clock_gettime(0,{1317770389.971526820 }) =3D 0 (0x0) > 0.049606219 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 }) > =3D 1 (0x1) > 0.049669916 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) =3D 184 > (0xb8) > 0.049809882 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) =3D 1 > (0x1) > ... >=20 > running with debug enabled it sits looping outputting:- > routing message op =3D 7: ignored > routing message op =3D 7: ignored > routing message op =3D 7: ignored > routing message op =3D 7: ignored > routing message op =3D 7: ignored > routing message op =3D 7: ignored > routing message op =3D 7: ignored > ... >=20 > It seems socket 28 is a duplicate of an internal routing socket > as seen here in the trace:- > 0.044544269 socket(PF_ROUTE,SOCK_RAW,0) =3D 4 (0x4) > 0.044595394 fcntl(4,F_DUPFD,0x14) =3D 28 (0x1c) > 0.044645960 close(4) =3D 0 (0x0) > 0.044695968 fcntl(28,F_SETFL,O_NONBLOCK) =3D 0 (0x0) >=20 > Now this looks like its RTM_MISS as defined:- > sys/net/route.h:#define RTM_MISS 0x7 /* Lookup failed on > this address */ >=20 > So the question was why is PF_ROUTE socket constantly > spamming RTM_MISS? >=20 > route -n monitor on this machines shows:- > got message of size 184 on Tue Oct 4 23:46:36 2011 > RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno > 0, flags: > locks: inits: > sockaddrs: > ::A.B.C.D >=20 > got message of size 184 on Tue Oct 4 23:46:36 2011 > RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno > 0, flags: > locks: inits: > sockaddrs: > ::A.B.C.D >=20 > This seems very much like the following pr which was fixed:- > "Remove a bogusly introduced rtalloc_ign() in rev. 1.335/SVN 178029, > generating an RTM_MISS for every IP packet forwarded making user space > routing daemons unhappy":- > http://www.freebsd.org/cgi/query-pr.cgi?pr=3D124540 >=20 > The box is doing no routing, its fairly basic install with > 1 main IP on em0 + 1 alias + gw addres and 1 private ip on em1. >=20 > Its running mysql and thats about it. >=20 > Any ideas? >=20 > Regards > Steve >=20 >=20 >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > This e.mail is private and confidential between Multiplay (UK) Ltd. and > the person or entity to whom it is addressed. In the event of > misdirection, the recipient is prohibited from using, copying, printing > or otherwise disseminating it or any information contained in it. >=20 > In the event of misdirection, illegible or incomplete transmission > please telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable- > unsubscribe@freebsd.org"