From owner-freebsd-net@FreeBSD.ORG Tue Jun 14 17:17:26 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9BB10106564A for ; Tue, 14 Jun 2011 17:17:26 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4B71F8FC14 for ; Tue, 14 Jun 2011 17:17:25 +0000 (UTC) Received: by vws18 with SMTP id 18so6676661vws.13 for ; Tue, 14 Jun 2011 10:17:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=MB1ZPJklTV4M/uFzgjNMMV+7BD8h5uifrJSViGXmzbQ=; b=sz0l8Xw4AvyAKeU2srtUAw01SqhlF54xs4rKlQ+ZsZUu5A5wJ8V7VKYym1a0qF3vmS 5Du4ugsTmMbQ8zWwig31hWzlhbV8Qv5fmp/WAnsaBJ6J7Pq49CzWXJnwrr83ZQjZ5Afh 7XucWth0W9Ha/9sYUyEnt2mFb/swPOz7QZ34g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=nf3YtrXJH3Mlb29pTMsaqsavIu7hOyNKnE7Mxws9sCLpgbkl/pRII8GaqDdjRgYUqd 06mz4fF2gwjwNFEybEHeKcWhjbaMRIciTLQfMUUwHieBKI0Zu/i+vIb23ouWxjFzG3E3 A1e4hnzC7OGuv5CVJamJ8HtStkjXHfJ2aPZjA= MIME-Version: 1.0 Received: by 10.52.96.134 with SMTP id ds6mr7951544vdb.27.1308071845224; Tue, 14 Jun 2011 10:17:25 -0700 (PDT) Received: by 10.52.115.138 with HTTP; Tue, 14 Jun 2011 10:17:25 -0700 (PDT) In-Reply-To: References: Date: Tue, 14 Jun 2011 10:17:25 -0700 Message-ID: From: Jack Vogel To: Mario Spinthiras Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: strange igb interface performance problems X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jun 2011 17:17:26 -0000 PS. My driver does not smell :), however if you want the 'freshest' smell possible maybe you should update to the version in STABLE/8 :) Jack On Tue, Jun 14, 2011 at 10:15 AM, Jack Vogel wrote: > Why do you have TSO off. If you run the same connection without the vlan > what do you see? > > Jack > > > > On Tue, Jun 14, 2011 at 7:38 AM, Mario Spinthiras < > spinthiras.mario@gmail.com> wrote: > >> Hello everyone, >> >> I'm back with more updates on the issue. Forgive my lack of elaboration >> on >> the topic in my previous post; over the frustration of this problem I >> couldn't see this topic through so I let it go for a few days to revisit >> with a clear head perhaps. >> >> As I mentioned in a previous post, the problem is between 2 nodes running >> PfSense 2.0 RC1 over a point to point link. The uname from the machines: >> >> Endpoint A: >> >> [2.0-RC1][root@pfSense.localdomain]/root(3): uname -a >> FreeBSD pfSense.localdomain 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 #0: Sat >> Feb 26 18:05:58 EST 2011 >> root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org: >> /usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8 >> amd64 >> [2.0-RC1][root@pfSense.localdomain]/root(4): >> >> Enpoint B: >> >> [2.0-RC1][root@pfsense.localdomain]/root(1): uname -a >> FreeBSD pfsense.localdomain 8.1-RELEASE-p3 FreeBSD 8.1-RELEASE-p3 #1: Tue >> Apr 26 20:56:16 EDT 2011 >> sullrich@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org: >> /usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8 >> i386 >> [2.0-RC1][root@pfsense.localdomain]/root(2): >> >> >> The link has a RTT of 130ms. The link's traffic is for the most part TCP >> with UDP not being used much. The performance of the link in one direction >> is fine (expected 160 odd Mbps). In the other direction it seems the link >> struggles to climb even with a 3 MB window size and all sorts of tuning on >> the stack. An output of the iperf test running for 20 seconds produces the >> following results: >> >> Node A: >> >> [2.0-RC1][root@pfSense.localdomain]/root(24): iperf -c b.b.b.b -w 3M -t >> 20 >> -i 5 >> ------------------------------------------------------------ >> Client connecting to b.b.b.b, TCP port 5001 >> TCP window size: 3.00 MByte (WARNING: requested 3.00 MByte) >> ------------------------------------------------------------ >> [ 3] local a.a.a.a port 22995 connected with b.b.b.b port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0- 5.0 sec 75.3 MBytes 126 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 5.0-10.0 sec 107 MBytes 180 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 10.0-15.0 sec 107 MBytes 180 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 15.0-20.0 sec 108 MBytes 181 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0-20.2 sec 404 MBytes 167 Mbits/sec >> [2.0-RC1][root@pfSense.localdomain]/root(25): >> >> >> Node B: >> >> [2.0-RC1][root@pfsense.localdomain]/root(2): iperf -c a.a.a.a -t 20 -i 10 >> -w >> 3M >> ------------------------------------------------------------ >> Client connecting to a.a.a.a, TCP port 5001 >> TCP window size: 3.00 MByte (WARNING: requested 3.00 MByte) >> ------------------------------------------------------------ >> [ 3] local b.b.b.b port 44160 connected with a.a.a.a port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0-10.0 sec 7.73 MBytes 6.49 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 10.0-20.0 sec 9.95 MBytes 8.34 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0-20.3 sec 18.1 MBytes 7.47 Mbits/sec >> [2.0-RC1][root@pfsense.localdomain]/root(3): >> >> The performance is terrible in Node B. >> >> Their corresponding interfaces look like this: >> >> Node A: >> >> [2.0-RC1][root@pfSense.localdomain]/root(26): ifconfig igb1 >> igb1: flags=28943 >> metric 0 mtu 1500 >> >> >> options=100bb >> ether 00:1b:21:9b:6e:ab >> inet6 fe80::21b:21ff:fe9b:6eab%igb1 prefixlen 64 scopeid 0x4 >> nd6 options=3 >> media: Ethernet autoselect (1000baseT ) >> status: active >> [2.0-RC1][root@pfSense.localdomain]/root(27): >> >> [2.0-RC1][root@pfSense.localdomain]/root(28): ifconfig igb1_593 >> igb1_593: flags=8843 metric 0 mtu >> 1500 >> options=3 >> ether 00:1b:21:9b:6e:ab >> inet6 fe80::225:90ff:fe20:5510%igb1_593 prefixlen 64 scopeid 0xa >> inet a.a.a.a netmask 0xfffffff8 broadcast x.x.x.x >> nd6 options=3 >> media: Ethernet autoselect (1000baseT ) >> status: active >> vlan: 593 parent interface: igb1 >> [2.0-RC1][root@pfSense.localdomain]/root(29): >> >> Node B: >> >> [2.0-RC1][root@pfsense.localdomain]/root(48): ifconfig igb3 >> igb3: flags=8843 metric 0 mtu 1500 >> >> >> options=101bb >> ether 00:1b:21:8e:2b:5f >> inet6 fe80::21b:21ff:fe8e:2b5f%igb3 prefixlen 64 scopeid 0x4 >> nd6 options=3 >> media: Ethernet autoselect (1000baseT ) >> status: active >> [2.0-RC1][root@pfsense.localdomain]/root(49): ifconfig igb3_vlan593 >> igb3_vlan593: flags=8843 metric 0 >> mtu 1500 >> options=3 >> ether 00:1b:21:8e:2b:5f >> inet6 fe80::21b:21ff:fe8e:2b5c%igb3_vlan593 prefixlen 64 scopeid >> 0xb >> inet b.b.b.b netmask 0xfffffff8 broadcast x.x.x.x >> nd6 options=3 >> media: Ethernet autoselect (1000baseT ) >> status: active >> vlan: 593 parent interface: igb3 >> [2.0-RC1][root@pfsense.localdomain]/root(50): >> >> >> As you can see from the ifconfig output, the link runs dot1q with a VID of >> 593. >> >> The netstat -m counters: >> >> Node A: >> >> [2.0-RC1][root@pfSense.localdomain]/root(29): netstat -m >> 10261/10606/20867 mbufs in use (current/cache/total) >> 10241/5957/16198/25600 mbuf clusters in use (current/cache/total/max) >> 10240/5120 mbuf+clusters out of packet secondary zone in use >> (current/cache) >> 0/3989/3989/12800 4k (page size) jumbo clusters in use >> (current/cache/total/max) >> 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) >> 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) >> 25612K/33173K/58785K bytes allocated to network (current/cache/total) >> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) >> 0/0/0 requests for jumbo clusters denied (4k/9k/16k) >> 0/0/0 sfbufs in use (current/peak/max) >> 0 requests for sfbufs denied >> 0 requests for sfbufs delayed >> 0 requests for I/O initiated by sendfile >> 0 calls to protocol drain routines >> [2.0-RC1][root@pfSense.localdomain]/root(30): >> >> Node B: >> >> [2.0-RC1][root@pfsense.localdomain]/root(50): netstat -m >> 9691/4274/13965 mbufs in use (current/cache/total) >> 6355/1891/8246/25600 mbuf clusters in use (current/cache/total/max) >> 6354/1070 mbuf+clusters out of packet secondary zone in use >> (current/cache) >> 1216/1271/2487/12800 4k (page size) jumbo clusters in use >> (current/cache/total/max) >> 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) >> 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) >> 20091K/9934K/30026K bytes allocated to network (current/cache/total) >> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) >> 0/0/0 requests for jumbo clusters denied (4k/9k/16k) >> 0/8/6656 sfbufs in use (current/peak/max) >> 0 requests for sfbufs denied >> 0 requests for sfbufs delayed >> 0 requests for I/O initiated by sendfile >> 0 calls to protocol drain routines >> [2.0-RC1][root@pfsense.localdomain]/root(51): >> >> >> dmesg output on both nodes. >> >> Node A: >> >> igb1: port >> 0xe880-0xe89f mem >> 0xf97e0000-0xf97fffff,0xf9c00000-0xf9ffffff,0xf97dc000-0xf97dffff irq 17 >> at >> device 0.1 on pci3 >> igb1: Using MSIX interrupts with 5 vectors >> igb1: [ITHREAD] >> igb1: [ITHREAD] >> igb1: [ITHREAD] >> igb1: [ITHREAD] >> igb1: [ITHREAD] >> >> >> Node B: >> >> igb3: port >> 0xc880-0xc89f mem >> 0xf9fe0000-0xf9ffffff,0xfa000000-0xfa3fffff,0xf9fdc000-0xf9fdffff irq 37 >> at >> device 0.1 on pci6 >> igb3: Using MSIX interrupts with 5 vectors >> igb3: [ITHREAD] >> igb3: [ITHREAD] >> igb3: [ITHREAD] >> igb3: [ITHREAD] >> igb3: [ITHREAD] >> igb3: link state changed to UP >> igb3_vlan593: link state changed to UP >> >> >> pciconf output: >> >> Node A: >> >> igb1@pci0:3:0:1: class=0x020000 card=0xa03c8086 chip=0x10c98086 >> rev=0x01hdr=0x00 >> class = network >> subclass = ethernet >> bar [10] = type Memory, range 32, base 0xf97e0000, size 131072, >> enabled >> bar [14] = type Memory, range 32, base 0xf9c00000, size 4194304, >> enabled >> bar [18] = type I/O Port, range 32, base 0xe880, size 32, enabled >> bar [1c] = type Memory, range 32, base 0xf97dc000, size 16384, >> enabled >> >> >> Node B: >> >> igb3@pci0:6:0:1: class=0x020000 card=0xa02b8086 chip=0x10e88086 >> rev=0x01 hdr=0x00 >> class = network >> subclass = ethernet >> bar [10] = type Memory, range 32, base 0xf9fe0000, size 131072, >> enabled >> bar [14] = type Memory, range 32, base 0xfa000000, size 4194304, >> enabled >> bar [18] = type I/O Port, range 32, base 0xc880, size 32, enabled >> bar [1c] = type Memory, range 32, base 0xf9fdc000, size 16384, >> enabled >> >> >> I've been looking at issues regarding performance with the igb issue from >> a >> few years back at this thread : >> http://lists.freebsd.org/pipermail/freebsd-doc/2009-June/015983.html. >> These >> are very similar problems to what I'm having however disabling LRO and TSO >> does not work for me. >> >> I'm quite frustrated with this because I'm not getting an error or not >> looking in the right place. Can someone out there point me in the right >> direction? Any help will be much appreciated. >> >> Warm Regards, >> Mario Spinthiras >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > >