From owner-freebsd-net@FreeBSD.ORG Wed Aug 11 16:43:23 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E57CA106567B for ; Wed, 11 Aug 2010 16:43:22 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-pv0-f182.google.com (mail-pv0-f182.google.com [74.125.83.182]) by mx1.freebsd.org (Postfix) with ESMTP id 7C87B8FC2B for ; Wed, 11 Aug 2010 16:43:21 +0000 (UTC) Received: by pvg4 with SMTP id 4so136056pvg.13 for ; Wed, 11 Aug 2010 09:43:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=V/YD4Izf0pINMemfiJZBGozgXI1spH7Dn6V8/UNG/hU=; b=NA2Ng+pSUgxPLdstpAnVXgvxQkDhGbXlx0d0dxA6AMHfFWJcY3cKn7c1dBT7DmhNxO Svco9Fi1vpspZ/lnCCx3d5meP90nIg99t1KwbdZDDp1m7b6IfdnHI8YeEbSct6woIjUL FQfaSlMd9pwL30BivqOSt0veE9pKz6yb3paoU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=Fu2S674tJX4w8xBYklD9O6b22tGcEsMdas+oCLO5XnCg4edePpS7039gobOuG0C6iZ SnyH/Sp1zoKG6Svt2p+LzbbFQoeiwt809sG2kgRVw8fVYRD2Vjmv958ZBZtkfk5QmV5K uv5YNoYONmh/TupQDxApPPX/vxEPWzPcY09uY= Received: by 10.114.93.19 with SMTP id q19mr22259699wab.206.1281545001063; Wed, 11 Aug 2010 09:43:21 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id x9sm497308waj.3.2010.08.11.09.43.18 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 11 Aug 2010 09:43:19 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Wed, 11 Aug 2010 09:43:18 -0700 From: Pyun YongHyeon Date: Wed, 11 Aug 2010 09:43:18 -0700 To: Victor Ophof Message-ID: <20100811164318.GB15858@michelle.cdnetworks.com> References: <20100810213754.GH6960@michelle.cdnetworks.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: "RX ring hdr initialization error" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Aug 2010 16:43:23 -0000 On Wed, Aug 11, 2010 at 10:19:11AM +0200, Victor Ophof wrote: > > > > From: pyunyh@gmail.com > Date: Tue, 10 Aug 2010 14:37:54 -0700 > To: mr4hughz@hotmail.com > CC: jfv@freebsd.org; freebsd-net@freebsd.org > Subject: Re: "RX ring hdr initialization error" > > On Tue, Aug 10, 2010 at 12:52:56PM +0200, Victor Ophof wrote: > > > > > > > > > > Hi > > > > I've bought a asus M4a78-EM Motherboard. to build a NAS on, > > thinking the onboard Realtek would be sufficant speed > > unfortunatly the onboard fives 16/31 mbs at best > > > > ps later It improved with enabeling "polling" in the kernel (duh) > > > > so I had a PCI intel GT nic around, what gave intermittent tcp/ip connections in a other machine (ESXi) > > unfortunatly this was the same with Freebsd (card issue?) The card is still in the machine > > even with the Intel supplied BSD driver > > > > now I bought a PCIe intel CT nic, put it in and the kernel panic with > > "RX ring hdr initialization error" > > so replaced the intel with the freebsd one by doing > > intel overwrites the freebsd one /boot/kernel/if_em.ko > > # cd /usr/src/sys/modules/em/ && make obj depend all install > > (was already in the kernel) > > > > > > still panic > > anybody got some idea's howto fix ? > > > --- reaction pyunyh --- > I have been using the attached patch for em(4)/igb(4) controllers. > These drivers explicitly calls panic(9) when memory allocation > failure happens. I don't think it's good idea to panic the box > under resource shortage condition as it's common to see this > situation on heavily loaded servers. > > The patch does not solve the one issue yet. The panic caused by > RX buffer allocation failure condition which in turn means you're > allocating a lot of buffers. Reduce number of descriptors if you > increased that too high and see whether the issue could be gone. > ---/reaction pyunyh ---What buffers /descriptors I need to reduce? I have 2gb ram and set the following in /boot/loader.conf vm.kmem_size_max="1024m" The loader tunables are hw.em.txd and hw.em.rxd. I thought you increased TX/RX descriptor size to large value(e.g. 4096). > vm.kmem_size="1024m" > #vfs.zfs.prefetch_disable=1 > vm.kmem_size="2048M" > vfs.zfs.arc_min="1024M" > vfs.zfs.arc_max="1536M" > vfs.zfs.vdev.min_pending=2 > vfs.zfs.vdev.max_pending=8 > vfs.zfs.txg.timeout=5 > aio_load="YES" > ahci_load="YES" > I guess zfs consumed a lot of memory such that em(4) was not able to allocate RX buffers. It seems there is nothing can be done in this case unless some memory is reclaimed from zfs. I'm not familiar with zfs internals but others can comment on this. However the patch should fix the panic under these resource shortage situation.