From owner-freebsd-bugs@FreeBSD.ORG Sat Oct 22 13:20:09 2011 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 753721065673 for ; Sat, 22 Oct 2011 13:20:09 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 519798FC17 for ; Sat, 22 Oct 2011 13:20:09 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9MDK9rH040516 for ; Sat, 22 Oct 2011 13:20:09 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9MDK987040515; Sat, 22 Oct 2011 13:20:09 GMT (envelope-from gnats) Resent-Date: Sat, 22 Oct 2011 13:20:09 GMT Resent-Message-Id: <201110221320.p9MDK987040515@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Steven Hartland Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C552106564A for ; Sat, 22 Oct 2011 13:15:29 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 4B6E58FC08 for ; Sat, 22 Oct 2011 13:15:29 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p9MDFSOq066936 for ; Sat, 22 Oct 2011 13:15:28 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id p9MDFSTN066935; Sat, 22 Oct 2011 13:15:28 GMT (envelope-from nobody) Message-Id: <201110221315.p9MDFSTN066935@red.freebsd.org> Date: Sat, 22 Oct 2011 13:15:28 GMT From: Steven Hartland To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/161899: Repeating RTM_MISS packets causing high CPU load for ntpd X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Oct 2011 13:20:09 -0000 >Number: 161899 >Category: kern >Synopsis: Repeating RTM_MISS packets causing high CPU load for ntpd >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Oct 22 13:20:08 UTC 2011 >Closed-Date: >Last-Modified: >Originator: Steven Hartland >Release: 8.2-STABLE >Organization: Multiplay >Environment: FreeBSD db4.multiplay.co.uk 8.2-STABLE FreeBSD 8.2-STABLE #16: Tue Oct 4 09:53:17 UTC 2011 root@db3-old.multiplay.co.uk:/usr/obj/usr/src/sys/MULTIPLAY amd64 >Description: We just updated a machine to 8-STABLE and I've noticed that ntpd is using notible amounts of CPU 5-7% which is very high for such a trivial daemon. 8.2-STABLE FreeBSD 8.2-STABLE #16: Tue Oct 4 09:53:17 UTC 2011 truss indicates its constantly checking and reading from a socket 0.047297485 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1 (0x1) 0.047513160 clock_gettime(0,{1317770389.969538247 }) = 0 (0x0) 0.047604515 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 }) = 1 (0x1) 0.047668212 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) = 184 (0xb8) 0.049395293 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1 (0x1) 0.049503689 clock_gettime(0,{1317770389.971526820 }) = 0 (0x0) 0.049606219 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 }) = 1 (0x1) 0.049669916 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) = 184 (0xb8) 0.049809882 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1 (0x1) .. running with debug enabled it sits looping outputting:- routing message op = 7: ignored routing message op = 7: ignored routing message op = 7: ignored routing message op = 7: ignored routing message op = 7: ignored routing message op = 7: ignored routing message op = 7: ignored .. It seems socket 28 is a duplicate of an internal routing socket as seen here in the trace:- 0.044544269 socket(PF_ROUTE,SOCK_RAW,0) = 4 (0x4) 0.044595394 fcntl(4,F_DUPFD,0x14) = 28 (0x1c) 0.044645960 close(4) = 0 (0x0) 0.044695968 fcntl(28,F_SETFL,O_NONBLOCK) = 0 (0x0) Now this looks like its RTM_MISS as defined:- sys/net/route.h:#define RTM_MISS 0x7 /* Lookup failed on this address */ So the question was why is PF_ROUTE socket constantly spamming RTM_MISS? route -n monitor on this machines shows:- got message of size 184 on Tue Oct 4 23:46:36 2011 RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: ::A.B.C.D got message of size 184 on Tue Oct 4 23:46:36 2011 RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: ::A.B.C.D This seems very much like the following pr which was fixed:- "Remove a bogusly introduced rtalloc_ign() in rev. 1.335/SVN 178029, generating an RTM_MISS for every IP packet forwarded making user space routing daemons unhappy":- http://www.freebsd.org/cgi/query-pr.cgi?pr=124540 The box is doing no routing, its fairly basic install with 1 main IP on em0 + 1 alias + gw addres and 1 private ip on em1. Its running mysql and thats about it. Additional discussion can be found in the the thread topic freebsd-stable list "serious packet routing issue causing ntpd high load?" >How-To-Repeat: According to Vlad Galu, which Li Qing confirms on the original discussion thread: "serious packet routing issue causing ntpd high load?" 1. Start listening for routing messages on a non-default FIB, e.g. setfib 2 route monitor 2. Add any static route within that FIB. 3. The machine I run the test on is a heavy DNS client, it generates a few dozen requests per second. The monitor process starts getting RTM_MISS messages for each outgoing DNS request (the dst sockaddr is the same as my first resolv.conf entry, seq is always 0). >Fix: >Release-Note: >Audit-Trail: >Unformatted: