From owner-freebsd-amd64@FreeBSD.ORG Fri May 25 20:40:02 2012 Return-Path: Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8A34A106566C for ; Fri, 25 May 2012 20:40:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6295B8FC18 for ; Fri, 25 May 2012 20:40:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q4PKe2sb040732 for ; Fri, 25 May 2012 20:40:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q4PKe24v040731; Fri, 25 May 2012 20:40:02 GMT (envelope-from gnats) Resent-Date: Fri, 25 May 2012 20:40:02 GMT Resent-Message-Id: <201205252040.q4PKe24v040731@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-amd64@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Ziyan Maraikar Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ED18B1065672 for ; Fri, 25 May 2012 20:34:23 +0000 (UTC) (envelope-from admin@nanuoya.pdn.ac.lk) Received: from nanuoya.pdn.ac.lk (unknown [IPv6:2401:dd00:30::1f00]) by mx1.freebsd.org (Postfix) with ESMTP id 4DD2C8FC17 for ; Fri, 25 May 2012 20:34:22 +0000 (UTC) Received: from nanuoya.pdn.ac.lk (localhost [127.0.0.1]) by nanuoya.pdn.ac.lk (8.14.5/8.14.5) with ESMTP id q4PKYKAj038871; Sat, 26 May 2012 02:04:20 +0530 (IST) (envelope-from admin@nanuoya.pdn.ac.lk) Received: (from admin@localhost) by nanuoya.pdn.ac.lk (8.14.5/8.14.5/Submit) id q4PKYKcB038870; Sat, 26 May 2012 02:04:20 +0530 (IST) (envelope-from admin) Message-Id: <201205252034.q4PKYKcB038870@nanuoya.pdn.ac.lk> Date: Sat, 26 May 2012 02:04:20 +0530 (IST) From: Ziyan Maraikar To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 X-Mailman-Approved-At: Fri, 25 May 2012 21:55:31 +0000 Cc: Darshana Jayasinghe Subject: amd64/168342: mbuf exhaustion hangs all daemons in keglimit state X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Ziyan Maraikar List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 May 2012 20:40:02 -0000 >Number: 168342 >Category: amd64 >Synopsis: mbuf exhaustion hangs all daemons in keglimit state >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-amd64 >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri May 25 20:40:01 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Ziyan Maraikar >Release: FreeBSD 9.0-RELEASE amd64 >Organization: Department of computer engineering, University of Peradeniya >Environment: System: FreeBSD nanuoya.pdn.ac.lk 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 HP Proliant DL165 4-core, 8G RAM 4x igb NICs -- 1 interface assigned 6 IPv4 aliases. 3x 1TB SATA zfs RAID-Z pool (zfs boot) >Description: This machine has been running DHCP, BIND, NFS and, openldap serving a lab of about 40 machines. The machine recently began to experience very frequentlockups in all network services including, ssh. The services all hang in state keglimit, even under very light load. I have tried disbling TSO and hardware checksum on igb as suggested in related mailing list posts, but it has no effect. >How-To-Repeat: Several ssh attempts after boot is enough to make all daemons hang in keglimit. # netstat -m 25034/1602/26636 mbufs in use (current/cache/total) 24892/708/25600/25600 mbuf clusters in use (current/cache/total/max) 24642/708 mbuf+clusters out of packet secondary zone in use (current/cache) 0/9/9/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 56053K/1852K/57905K bytes allocated to network (current/cache/total) 0/1697/1209 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines >Fix: >Release-Note: >Audit-Trail: >Unformatted: