From owner-freebsd-bugs@FreeBSD.ORG Sat Mar 20 23:10:03 2010 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3F6F0106566B for ; Sat, 20 Mar 2010 23:10:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 02FA28FC14 for ; Sat, 20 Mar 2010 23:10:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2KNA2hd080859 for ; Sat, 20 Mar 2010 23:10:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2KNA2M6080858; Sat, 20 Mar 2010 23:10:02 GMT (envelope-from gnats) Resent-Date: Sat, 20 Mar 2010 23:10:02 GMT Resent-Message-Id: <201003202310.o2KNA2M6080858@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Doychin Dokov Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CAD311065672 for ; Sat, 20 Mar 2010 23:06:36 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id A01028FC14 for ; Sat, 20 Mar 2010 23:06:36 +0000 (UTC) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o2KN6a5Q038745 for ; Sat, 20 Mar 2010 23:06:36 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id o2KN6Z55038744; Sat, 20 Mar 2010 23:06:35 GMT (envelope-from nobody) Message-Id: <201003202306.o2KN6Z55038744@www.freebsd.org> Date: Sat, 20 Mar 2010 23:06:35 GMT From: Doychin Dokov To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/144917: Flowtable crashes system X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2010 23:10:03 -0000 >Number: 144917 >Category: kern >Synopsis: Flowtable crashes system >Confidential: no >Severity: serious >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Mar 20 23:10:02 UTC 2010 >Closed-Date: >Last-Modified: >Originator: Doychin Dokov >Release: 8.0-RELEASE-p2 >Organization: Data Syst Ltd. >Environment: FreeBSD wilma.net1.cc 8.0-RELEASE-p2 FreeBSD 8.0-RELEASE-p2 #0: Tue Jan 5 21:11:58 UTC 2010 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 >Description: It seems like flowtable has been merged and enabled by default in 8.0.... which is a really really bad idea. On a system which handles two full BGP tables it makes one of the CPU cores run at 100% right after most of the prefixes get installed in the routing table. Performance is seriously harmed, as well is ZFS speed - it's dramatically slow (sometimes 3x-4x). The system crashes in random time - between 1 and 10 minutes right after it has booted and started installing the routes in the kernel routing table. Observations with top -PS show that a process called 'flowcontroller' is overwhelming the CPU. Adding net.inet.flowtable.enable=0 to /etc/sysctl.conf AND rebooting the system fixes the problem. Changing the sysctl after the problem has appeared does not seem to fix it - the system still crashes in a while. This is very very embarassing when upgrading from 7.x, and is also very undocumented: wilma# man flowcontroller No manual entry for flowcontroller wilma# man flowtable No manual entry for flowtable I think this feature should be disabled by default (according to what i've found about that, it helps with heavy traffic like 10G links - it's far more frequent for one to use lots of prefixes than 10G), or - at least - better documented and maybe automatically disabled by packages which deal with lots of prefixes (e.g. quagga, openbgpd, etc.) >How-To-Repeat: 1. Install 8.0-RELEASE (does not matter if you update to the latest patchlevel with freebsd-update) 2. Install quagga or openbgpd or some other routing daemon 3. Feed it with lots of routes (e.g. full internet routing table) 4. Grab a coffee 5. It'll crash until your cup is still smoking. P.S. I've only tested with FreeBSD installed on ZFS root >Fix: Add this to /etc/sysctl.conf: net.inet.flowtable.enable=0 >Release-Note: >Audit-Trail: >Unformatted: