From owner-freebsd-stable@FreeBSD.ORG Thu May 27 16:18:39 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 635E21065674 for ; Thu, 27 May 2010 16:18:39 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 019F68FC0A for ; Thu, 27 May 2010 16:18:38 +0000 (UTC) Received: by vws12 with SMTP id 12so147559vws.13 for ; Thu, 27 May 2010 09:18:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=1SQVyyCxeKw8F25GEAC90ccTLR4ElXic7hnza8ZutVc=; b=n+3YslzgmcJXifcF9lBWc/0qEovFcxY7uqUlBTBBtIXz3otORYkFWDOH70YCLLpfWf BtVdkI7rNQ9vWg8uWwKqD/bOnqdroco6SEz4Hyl9km+qoyMxpUYXtBbsUMzrhv4gzZAY ARTrpV7WFPENxFPzm8Zsbtlc5DBQL8aSgDpjI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=VuJ0EjymkUcN9WhiAOhprrp78EdOaZhcjixkJjtGCkhaqGG4oOxLJ+2eEU3Ns0YHKT FgVLXLLc/2NXr6gTt8CuUZliof2BI7iJYBAS6y0v3YZwJoozgOAJSvo5erI06WLXL7hR Wxx9tVdYFg1N3OqCR1umMldps720OeCy96K/Q= MIME-Version: 1.0 Received: by 10.229.218.19 with SMTP id ho19mr2331121qcb.143.1274977117441; Thu, 27 May 2010 09:18:37 -0700 (PDT) Received: by 10.229.184.76 with HTTP; Thu, 27 May 2010 09:18:37 -0700 (PDT) In-Reply-To: <619287.68177.qm@web120513.mail.ne1.yahoo.com> References: <619287.68177.qm@web120513.mail.ne1.yahoo.com> Date: Thu, 27 May 2010 09:18:37 -0700 Message-ID: From: Jack Vogel To: Kirill Yelizarov Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable@freebsd.org Subject: Re: Strange igb befavior X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 May 2010 16:18:39 -0000 Well, it might make sense to increase the mbuf pool, do: sysctl kern.ipc.nmbclusters this is a parameter that can be set in /etc/sysctl.conf. Our testers set this on all their machines to 262144. As for the aim stuff, and that includes those latency tuneables, they have all been replaced in my latest code with a simplified, more automatic approach. Jack On Thu, May 27, 2010 at 3:30 AM, Kirill Yelizarov wrote: > Thank You Jack > > i'll keep it at 2048 now. I have plans to add two more igb interfaces. > Should i decrease values to 1024 in case i will have four interfaces in > server? > > I found there are additional tweaks for igb card available: > hw.igb.enable_aim=1 this one enabled by default and i understand its > meaning > Where can i read about the rest? > hw.igb.low_latency=1000 > hw.igb.ave_latency=2000 > hw.igb.bulk_latency=4000 > hw.igb.rx_process_limit=400 > hw.igb.fc_setting=0 > hw.igb.lro=0 > > Some years ago i downloaded an article about em card from intel site. > Perhaps there is one for igb as well? > > Kirill > > --- On Thu, 5/27/10, Jack Vogel wrote: > > > From: Jack Vogel > > Subject: Re: Strange igb befavior > > To: "Kirill Yelizarov" > > Cc: freebsd-stable@freebsd.org > > Date: Thursday, May 27, 2010, 1:06 PM > > Panic is due to a failure to get > > enough mbufs, when you make your ring that > > big you > > hit the problem, I have been experimenting with a change to > > fix it but am > > not yet > > completely confident, for the moment don't make your ring > > so big :) > > > > Jack > > > > > > On Thu, May 27, 2010 at 1:08 AM, Kirill Yelizarov > > wrote: > > > > > Hi > > > > > > I'm having reproducible panics with 8-Stable of May13 > > 2010. Panic occurs in > > > igb code. Panic start to happen when i set > > hw.igb.rxd="4096" and > > > hw.igb.txd="4096" in /boot/loader.conf. Panic happens > > immediately after boot > > > in igb1 code in my case. igb1 is connected to 100Mbit > > 3COM switch and switch > > > is not connected to anything else. > > > > > > Here is dmesg for igb > > > # dmesg | grep igb > > > igb0: > - 1.9.5> port > > > 0x2020-0x203f mem > > 0xb1a20000-0xb1a3ffff,0xb1a44000-0xb1a47fff irq 40 at > > > device 0.0 on pci1 > > > igb0: Using MSIX interrupts with 5 vectors > > > igb0: [ITHREAD] > > > igb0: [ITHREAD] > > > igb0: [ITHREAD] > > > igb0: [ITHREAD] > > > igb0: [ITHREAD] > > > igb0: Ethernet address: 00:15:17:ba:2e:00 > > > igb1: > - 1.9.5> port > > > 0x2000-0x201f mem > > 0xb1a00000-0xb1a1ffff,0xb1a40000-0xb1a43fff irq 28 at > > > device 0.1 on pci1 > > > igb1: Using MSIX interrupts with 5 vectors > > > igb1: [ITHREAD] > > > igb1: [ITHREAD] > > > igb1: [ITHREAD] > > > igb1: [ITHREAD] > > > igb1: [ITHREAD] > > > igb1: Ethernet address: 00:15:17:ba:2e:01 > > > igb1: link state changed to UP > > > igb0: link state changed to UP > > > > > > border2# ifconfig > > > igb0: > > flags=8843 > > metric 0 mtu 1500 > > > > > options=13b > > > ether 00:15:17:ba:2e:00 > > > inet 192.168.10.2 netmask > > 0xffffff00 broadcast 192.168.10.255 > > > inet 192.168.10.201 netmask > > 0xffffffff broadcast 192.168.10.201 > > > inet 192.168.10.202 netmask > > 0xffffffff broadcast 192.168.10.202 > > > inet 192.168.10.203 netmask > > 0xffffffff broadcast 192.168.10.203 > > > inet 192.168.10.204 netmask > > 0xffffffff broadcast 192.168.10.204 > > > media: Ethernet autoselect > > (1000baseT ) > > > status: active > > > igb1: > > flags=8843 > > metric 0 mtu 1500 > > > > > options=13b > > > ether 00:15:17:ba:2e:01 > > > inet XXX.74.229.230 netmask > > 0xfffffff0 broadcast XXX.74.229.239 > > > inet XXX.74.229.226 netmask > > 0xffffffff broadcast XXX.74.229.226 > > > inet XXX.74.229.227 netmask > > 0xffffffff broadcast XXX.74.229.227 > > > media: Ethernet autoselect > > (100baseTX ) > > > status: active > > > lo0: flags=8049 > > metric 0 mtu 16384 > > > > > options=3 > > > inet 127.0.0.1 netmask > > 0xff000000 > > > pfsync0: flags=0<> metric 0 mtu 1460 > > > syncpeer: 224.0.0.240 > > maxupd: 128 > > > pflog0: flags=141 metric 0 > > mtu 33152 > > > > > > #pciconf -lv > > > igb0@pci0:1:0:0: > > class=0x020000 card=0x34de8086 chip=0x10a78086 > > > rev=0x02 hdr=0x00 > > > vendor = 'Intel > > Corporation' > > > device = '82575EB > > Gigabit Network Connection' > > > class = network > > > subclass = ethernet > > > igb1@pci0:1:0:1: > > class=0x020000 card=0x34de8086 chip=0x10a78086 > > > rev=0x02 hdr=0x00 > > > vendor = 'Intel > > Corporation' > > > device = '82575EB > > Gigabit Network Connection' > > > class = network > > > subclass = ethernet > > > > > > # netstat -i > > > Name Mtu Network > > Address > > Ipkts Ierrs Idrop Opkts > > > Oerrs Coll > > > igb0 1500 > > 00:15:17:ba:2e:00 2315 > > 0 0 > > 1415 > > > 0 0 > > > igb0 1500 192.168.10.0 > > border2 > > 1664 - > > - 1412 > > > - - > > > igb0 1500 192.168.10.20 > > mysql-border2 > > 0 - - > > 0 > > > - - > > > igb0 1500 192.168.10.20 > > apache-border2 > > 0 - > > - 0 > > > - - > > > igb0 1500 192.168.10.20 > > squid-border2 > > 0 - - > > 0 > > > - - > > > igb0 1500 192.168.10.20 > > postgresql-border 1 > > - - > > 0 > > > - - > > > igb1 1500 > > 00:15:17:ba:2e:01 129 > > 0 0 > > 0 > > > 0 0 > > > igb1 1500 XXX.74.229.22 border2 > > > > 0 - - > > 0 > > > - - > > > igb1 1500 XXX.74.229.22 > > apache-border2 > > 0 - > > - 0 > > > - - > > > igb1 1500 XXX.74.229.22 > > squid-border2 > > 0 - - > > 0 > > > - - > > > lo0 16384 > > > > 1 > > 0 0 > > 1 > > > 0 0 > > > lo0 16384 your-net > > localhost > > 0 - > > - 1 > > > - - > > > pfsyn 1460 > > > > 0 > > 0 0 > > 0 > > > 0 0 > > > pflog 33152 > > > > 0 > > 0 0 > > 0 > > > 0 0 > > > > > > There are several jails on this server. > > > > > > When i set hw.igb.rxd="2048" and hw.igb.txd="2048" i > > don't have panic > > > anymore. > > > > > > The reason i tried to add more buffers than default is > > because i have NFS > > > export on this server. But it is on igb0. igb1 is > > currently doing nothing > > > but it will soon once the server will be ready for > > production. > > > > > > I didn't get core dump because my system is rather > > old. But i can do it if > > > needed. > > > > > > Regards, > > > Kirill > > > > > > > > > > > > _______________________________________________ > > > freebsd-stable@freebsd.org > > mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > > To unsubscribe, send any mail to " > freebsd-stable-unsubscribe@freebsd.org" > > > > > _______________________________________________ > > freebsd-stable@freebsd.org > > mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org > " > > > > > >