From owner-freebsd-amd64@FreeBSD.ORG Mon May 28 02:28:07 2012 Return-Path: Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D4A5A1065670; Mon, 28 May 2012 02:28:07 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id A8C458FC15; Mon, 28 May 2012 02:28:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q4S2S7hf056816; Mon, 28 May 2012 02:28:07 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q4S2S7Vt056810; Mon, 28 May 2012 02:28:07 GMT (envelope-from linimon) Date: Mon, 28 May 2012 02:28:07 GMT Message-Id: <201205280228.q4S2S7Vt056810@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-amd64@FreeBSD.org, freebsd-bugs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/168342: [mbuf] mbuf exhaustion hangs all daemons in keglimit state X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 02:28:07 -0000 Old Synopsis: mbuf exhaustion hangs all daemons in keglimit state New Synopsis: [mbuf] mbuf exhaustion hangs all daemons in keglimit state Responsible-Changed-From-To: freebsd-amd64->freebsd-bugs Responsible-Changed-By: linimon Responsible-Changed-When: Mon May 28 02:27:38 UTC 2012 Responsible-Changed-Why: A customer of mine is also seeing this. However, I do not believe it is amd64-specific. http://www.freebsd.org/cgi/query-pr.cgi?pr=168342 From owner-freebsd-amd64@FreeBSD.ORG Mon May 28 02:29:51 2012 Return-Path: Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 63147106566C; Mon, 28 May 2012 02:29:51 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 369CA8FC08; Mon, 28 May 2012 02:29:51 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q4S2TpXg056915; Mon, 28 May 2012 02:29:51 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q4S2TpN3056911; Mon, 28 May 2012 02:29:51 GMT (envelope-from linimon) Date: Mon, 28 May 2012 02:29:51 GMT Message-Id: <201205280229.q4S2TpN3056911@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-amd64@FreeBSD.org, freebsd-bugs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/168320: [hptiop] [patch] make the hptiop driver support the RR4310 controller card X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 02:29:51 -0000 Old Synopsis: hptiop driver should support the RR4310 controller card. New Synopsis: [hptiop] [patch] make the hptiop driver support the RR4310 controller card Responsible-Changed-From-To: freebsd-amd64->freebsd-bugs Responsible-Changed-By: linimon Responsible-Changed-When: Mon May 28 02:28:16 UTC 2012 Responsible-Changed-Why: reclassify. http://www.freebsd.org/cgi/query-pr.cgi?pr=168320 From owner-freebsd-amd64@FreeBSD.ORG Mon May 28 11:07:22 2012 Return-Path: Delivered-To: freebsd-amd64@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CEBDF106564A for ; Mon, 28 May 2012 11:07:22 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B90B28FC1D for ; Mon, 28 May 2012 11:07:22 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q4SB7Mx3063278 for ; Mon, 28 May 2012 11:07:22 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q4SB7MBx063276 for freebsd-amd64@FreeBSD.org; Mon, 28 May 2012 11:07:22 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 28 May 2012 11:07:22 GMT Message-Id: <201205281107.q4SB7MBx063276@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-amd64@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-amd64@FreeBSD.org X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 11:07:23 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o amd64/167582 amd64 Compile of MySQL NDB Cluster Fails 8.2 AMD64 o amd64/167543 amd64 [kernel] Install FreeBSD can show error message with c o amd64/167393 amd64 [boot] MacBook4,1 hangs on SMP boot o amd64/166639 amd64 [boot] Syscons issue Intel D2700 o amd64/166229 amd64 [boot] Unable to install FreeBSD 9 on Acer Extensa 522 o amd64/165850 amd64 [build] 8.3-RC1 (amd64): world doesn't build with CPUT o amd64/165845 amd64 [build] Unable to build kernel on 8.2-STABLE o amd64/165351 amd64 [boot] Error while installing or booting the freeBSD O o amd64/164773 amd64 [boot] 9.0 amd64 fails to boot on HP DL145 G3 [regress o amd64/164707 amd64 FreeBSD 9 installer does not work with IBM uefi o amd64/164643 amd64 Kernel Panic at 9.0-RELEASE o amd64/164619 amd64 when logged in as root the user and group applications o amd64/164457 amd64 [install] Can't install FreeBSD 9.0 (amd64) on HP Blad o amd64/164301 amd64 [install] 9.0 - Can't install, no DHCP lease o amd64/164136 amd64 after fresh install 8.1 release or 8.2 release the har o amd64/164116 amd64 [boot] FreeBSD 9.0-RELEASE installations mediums fails o amd64/164089 amd64 FreeBSD-9.0-RELEASE-amd64-memstick.img does not boot o amd64/164073 amd64 /etc/rc warning after booting o amd64/164036 amd64 [keyboard] Moused fails on 9_0_RELENG o amd64/163736 amd64 Freebsd 8.2 with MPD5 and about 100 PPPoE clients pani o amd64/163710 amd64 setjump in userboot.so causes stack corruption o amd64/163625 amd64 Install problems of RC3 amd64 on ASRock N68 GE3 UCC o amd64/163568 amd64 hard drive naming o amd64/163285 amd64 when installing gnome2-lite not all dependent packages o amd64/163284 amd64 print manager failed to install correctly o amd64/163114 amd64 no boot on Via Nanao netbook Samsung NC20 o amd64/163092 amd64 FreeBSD 9.0-RC2 fails to boot from raid-z2 if AHCI is o amd64/163048 amd64 normal user cant mount ntfs-3g o amd64/162936 amd64 fails boot and destabilizes other OSes on FreeBSD 9 RC o amd64/162489 amd64 After some time X blanks the screen and does not respo o amd64/162314 amd64 not able to install FreeBSD-8.2-RELEASE-amd64-dvd1 as o amd64/162219 amd64 [REGRESSION] In KDE 4.7.2 cant enable OpenGL,in 4.6.5 o amd64/162170 amd64 Unable to install due to freeze at "run_interrupt_driv o amd64/161974 amd64 FreeBSD 9 new installer installs succesful, renders ma o kern/160833 amd64 Keyboard USB doesn't work o amd64/157386 amd64 [powerd] Enabling powerd(8) with default settings on I o amd64/156106 amd64 [boot] boot0 fails to start o amd64/155135 amd64 [boot] Does Not Boot On a Very Standard Hardware o amd64/154957 amd64 [boot] Install boot CD won't boot up - keeps rebooting o amd64/154629 amd64 [panic] Fatal trap 9: general protection fault while i o amd64/153935 amd64 [hang] system hangs while trying to do 'shutdown -h no o amd64/153831 amd64 [boot] CD bootloader won't on Tyan s2912G2nr o amd64/153496 amd64 [hyper-v] [install] Install on Hyper-V leaves corrupt o amd64/153372 amd64 [panic] kernel panic o amd64/153175 amd64 [amd64] Kernel Panic on only FreeBSD 8 amd64 o amd64/152874 amd64 [install] 8.1 install fails where 7.3 works due to lac o amd64/152430 amd64 [boot] HP ProLiant Microserver n36l cannot boot into i o amd64/145991 amd64 [NOTES] [patch] Add a requires line to /sys/amd64/conf o amd64/144405 amd64 [build] [patch] include /usr/obj/lib32 in cleanworld t s amd64/143173 amd64 [ata] Promise FastTrack TX4 + SATA DVD, installer can' p amd64/141413 amd64 [hang] Tyan 2881 m3289 SMDC freeze o amd64/137942 amd64 [pci] 8.0-BETA2 having problems with Asus M2N-SLI-delu o amd64/127640 amd64 [amd64] gcc(1) will not build shared libraries with -f o amd64/115194 amd64 LCD screen remains blank after Dell XPS M1210 lid is c 54 problems total. From owner-freebsd-amd64@FreeBSD.ORG Tue May 29 12:48:26 2012 Return-Path: Delivered-To: freebsd-amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F17E9106566C; Tue, 29 May 2012 12:48:26 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id C65F18FC08; Tue, 29 May 2012 12:48:26 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 3C56AB91A; Tue, 29 May 2012 08:48:26 -0400 (EDT) From: John Baldwin To: freebsd-amd64@freebsd.org, Ziyan Maraikar Date: Tue, 29 May 2012 08:12:40 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p13; KDE/4.5.5; amd64; ; ) References: <201205252034.q4PKYKcB038870@nanuoya.pdn.ac.lk> In-Reply-To: <201205252034.q4PKYKcB038870@nanuoya.pdn.ac.lk> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201205290812.40093.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 29 May 2012 08:48:26 -0400 (EDT) Cc: FreeBSD-gnats-submit@freebsd.org, Darshana Jayasinghe Subject: Re: amd64/168342: mbuf exhaustion hangs all daemons in keglimit state X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 12:48:27 -0000 On Friday, May 25, 2012 4:34:20 pm Ziyan Maraikar wrote: > > >Number: 168342 > >Category: amd64 > >Synopsis: mbuf exhaustion hangs all daemons in keglimit state > >Confidential: no > >Severity: serious > >Priority: medium > >Responsible: freebsd-amd64 > >State: open > >Quarter: > >Keywords: > >Date-Required: > >Class: sw-bug > >Submitter-Id: current-users > >Arrival-Date: Fri May 25 20:40:01 UTC 2012 > >Closed-Date: > >Last-Modified: > >Originator: Ziyan Maraikar > >Release: FreeBSD 9.0-RELEASE amd64 > >Organization: > Department of computer engineering, University of Peradeniya > >Environment: > System: FreeBSD nanuoya.pdn.ac.lk 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 > HP Proliant DL165 4-core, 8G RAM > 4x igb NICs -- 1 interface assigned 6 IPv4 aliases. > 3x 1TB SATA zfs RAID-Z pool (zfs boot) > > >Description: > This machine has been running DHCP, BIND, NFS and, openldap serving a lab of about 40 machines. The machine recently began to experience very frequentlockups in all network services including, ssh. The services all hang in state keglimit, even under very light load. I have tried disbling TSO and hardware checksum on igb as suggested in related mailing list posts, but it has no effect. > > >How-To-Repeat: > Several ssh attempts after boot is enough to make all daemons hang in keglimit. > # netstat -m > 25034/1602/26636 mbufs in use (current/cache/total) > 24892/708/25600/25600 mbuf clusters in use (current/cache/total/max) > 24642/708 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/9/9/12800 4k (page size) jumbo clusters in use (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 56053K/1852K/57905K bytes allocated to network (current/cache/total) > 0/1697/1209 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines Have you tried increasing kern.ipc.nmbclusters? Alternatively, have you tried restricting igb to only using 1 queue? It sounds like all your igb interfaces are allocating all of your mbuf clusters for their receive rings. -- John Baldwin From owner-freebsd-amd64@FreeBSD.ORG Tue May 29 16:44:09 2012 Return-Path: Delivered-To: freebsd-amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF2831065670; Tue, 29 May 2012 16:44:09 +0000 (UTC) (envelope-from ziyanm@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 91CC68FC14; Tue, 29 May 2012 16:44:09 +0000 (UTC) Received: by pbbro2 with SMTP id ro2so6460551pbb.13 for ; Tue, 29 May 2012 09:44:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=KYEt+gulYN3+SHIDERkcNljcoqi08BKEuapEiDbPo7E=; b=NhqUmxZfLjNgQfd9qhkxkbzx1dRvywDKatyLN6tb/yFYaf31G+heWeCs/3HfILR8vx PDDMro5zsdMMklAJCrjd9pZyyiDNU1bWw10zwyX5soY6TBDHpTkVQuAytnkJFwTKvkRD FtAA5TC0IYrdH9vceBMoBuAM8fy6kk8p5yo5G8FiBcdh934WK5rNOwwJ2oXXOE/ousvt ZoPnyZTinnc279l4BSP8ZAV2PZNB/Z6ngVKgxOFRgvipmYiUN3J23cF3yxHZx+STRNTc 5TrLv+UrS8lsDooUw++d1YxwdRCGzn9DvkpoTD0UIMGHfXgLTUv0FNx+6tgUvv3PiZ6q gQ2Q== Received: by 10.68.226.73 with SMTP id rq9mr38717571pbc.145.1338309849114; Tue, 29 May 2012 09:44:09 -0700 (PDT) Received: from [192.168.1.102] ([112.134.101.120]) by mx.google.com with ESMTPS id ol1sm4218674pbb.25.2012.05.29.09.44.05 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 29 May 2012 09:44:08 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: text/plain; charset=us-ascii From: Ziyan Maraikar In-Reply-To: <201205290812.40093.jhb@freebsd.org> Date: Tue, 29 May 2012 22:14:02 +0530 Content-Transfer-Encoding: quoted-printable Message-Id: <1966F26E-3E73-4E78-8F54-DBDC11195954@gmail.com> References: <201205252034.q4PKYKcB038870@nanuoya.pdn.ac.lk> <201205290812.40093.jhb@freebsd.org> To: John Baldwin X-Mailer: Apple Mail (2.1257) X-Mailman-Approved-At: Tue, 29 May 2012 16:57:04 +0000 Cc: FreeBSD-gnats-submit@freebsd.org, freebsd-amd64@freebsd.org, Darshana Jayasinghe Subject: Re: amd64/168342: mbuf exhaustion hangs all daemons in keglimit state X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 16:44:09 -0000 Hello John, Thanks for the response. >=20 > Have you tried increasing kern.ipc.nmbclusters? Alternatively, have = you tried=20 > restricting igb to only using 1 queue? It sounds like all your igb = interfaces=20 > are allocating all of your mbuf clusters for their receive rings. >=20 I found this very suggestion on several mailing list discussions [1] and = set these values on Saturday. kern.ipc.nmbclusters=3D"131072" hw.igb.num_queues=3D"2" So far everything seems to back to normal, and netstat -m shows plenty = of headroom now.=20 The problem cropped up after running several months on 9.0-RELEASE when = I brought up another interface. Disabling the new interface didn't = restore normal operation, however. I also tried 8.3-RELEASE but the = problem was worse on it. [1] http://osdir.com/ml/freebsd-stable/2012-02/msg00563.html __ Regards Ziyan.=