From owner-freebsd-stable@FreeBSD.ORG Wed Oct 5 00:11:18 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 187FC106566B for ; Wed, 5 Oct 2011 00:11:18 +0000 (UTC) (envelope-from prvs=1259955070=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 82A028FC08 for ; Wed, 5 Oct 2011 00:11:17 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Wed, 05 Oct 2011 01:00:07 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 05 Oct 2011 01:00:07 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50015351279.msg for ; Wed, 05 Oct 2011 01:00:07 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1259955070=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-stable@freebsd.org Message-ID: <2740DA0E7C0F495BA00E6D4BC0FE8F37@multiplay.co.uk> From: "Steven Hartland" To: Date: Wed, 5 Oct 2011 01:00:25 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: liv3d@multiplay.co.uk Subject: serious packet routing issue causing ntpd high load? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Oct 2011 00:11:18 -0000 We just updated a machine to 8-STABLE and I've noticed that ntpd is using notible amounts of CPU 5-7% which is very high for such a trivial daemon. 8.2-STABLE FreeBSD 8.2-STABLE #16: Tue Oct 4 09:53:17 UTC 2011 truss indicates its constantly checking and reading from a socket 0.047297485 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1 (0x1) 0.047513160 clock_gettime(0,{1317770389.969538247 }) = 0 (0x0) 0.047604515 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 }) = 1 (0x1) 0.047668212 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) = 184 (0xb8) 0.049395293 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1 (0x1) 0.049503689 clock_gettime(0,{1317770389.971526820 }) = 0 (0x0) 0.049606219 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 }) = 1 (0x1) 0.049669916 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) = 184 (0xb8) 0.049809882 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1 (0x1) ... running with debug enabled it sits looping outputting:- routing message op = 7: ignored routing message op = 7: ignored routing message op = 7: ignored routing message op = 7: ignored routing message op = 7: ignored routing message op = 7: ignored routing message op = 7: ignored ... It seems socket 28 is a duplicate of an internal routing socket as seen here in the trace:- 0.044544269 socket(PF_ROUTE,SOCK_RAW,0) = 4 (0x4) 0.044595394 fcntl(4,F_DUPFD,0x14) = 28 (0x1c) 0.044645960 close(4) = 0 (0x0) 0.044695968 fcntl(28,F_SETFL,O_NONBLOCK) = 0 (0x0) Now this looks like its RTM_MISS as defined:- sys/net/route.h:#define RTM_MISS 0x7 /* Lookup failed on this address */ So the question was why is PF_ROUTE socket constantly spamming RTM_MISS? route -n monitor on this machines shows:- got message of size 184 on Tue Oct 4 23:46:36 2011 RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: ::A.B.C.D got message of size 184 on Tue Oct 4 23:46:36 2011 RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: ::A.B.C.D This seems very much like the following pr which was fixed:- "Remove a bogusly introduced rtalloc_ign() in rev. 1.335/SVN 178029, generating an RTM_MISS for every IP packet forwarded making user space routing daemons unhappy":- http://www.freebsd.org/cgi/query-pr.cgi?pr=124540 The box is doing no routing, its fairly basic install with 1 main IP on em0 + 1 alias + gw addres and 1 private ip on em1. Its running mysql and thats about it. Any ideas? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.