From owner-freebsd-net@FreeBSD.ORG Tue Jun 14 17:15:10 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 348F7106567A for ; Tue, 14 Jun 2011 17:15:10 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id A3A108FC1C for ; Tue, 14 Jun 2011 17:15:08 +0000 (UTC) Received: by vxc34 with SMTP id 34so6689110vxc.13 for ; Tue, 14 Jun 2011 10:15:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=Muz4IERSe2ZiNw/ojbqrWjsYr+XLoZY9V5Dam1eU+cA=; b=VJ3E3AVoQaM7dl/acnDxWo8F34gcFd75eszFNNs/WOeTRSMuLmEtLGK7I6Z6Ho9jVl gJYpV5XvKmFAR+pbaJX4MwZL15qtOlxsdWPET45haFY3aiZiaBSzyxUZ2aYFFPVlt+c6 vlzyFSY0N+aY4YRAVgpzhoJ1dXukoP45scS+c= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=RZTnptd7lc0++TU6MoRti56gsTzxBQeV0PbXpCZxXlG6gpbwjsuAKuTjoT5tG+IQsq /Oa8GMTFpPbdaZaXpU5WL030YzLOO2whHGh6W+lbWcuzHwIsg9VPeliMHMMj5MAElLrB tHF6ryrNKE3HL/E1wh3RwGP+N5Tb7Niadat6I= MIME-Version: 1.0 Received: by 10.52.115.163 with SMTP id jp3mr5147089vdb.187.1308071707214; Tue, 14 Jun 2011 10:15:07 -0700 (PDT) Received: by 10.52.115.138 with HTTP; Tue, 14 Jun 2011 10:15:07 -0700 (PDT) In-Reply-To: References: Date: Tue, 14 Jun 2011 10:15:07 -0700 Message-ID: From: Jack Vogel To: Mario Spinthiras Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: strange igb interface performance problems X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jun 2011 17:15:10 -0000 Why do you have TSO off. If you run the same connection without the vlan what do you see? Jack On Tue, Jun 14, 2011 at 7:38 AM, Mario Spinthiras < spinthiras.mario@gmail.com> wrote: > Hello everyone, > > I'm back with more updates on the issue. Forgive my lack of elaboration on > the topic in my previous post; over the frustration of this problem I > couldn't see this topic through so I let it go for a few days to revisit > with a clear head perhaps. > > As I mentioned in a previous post, the problem is between 2 nodes running > PfSense 2.0 RC1 over a point to point link. The uname from the machines: > > Endpoint A: > > [2.0-RC1][root@pfSense.localdomain]/root(3): uname -a > FreeBSD pfSense.localdomain 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 #0: Sat > Feb 26 18:05:58 EST 2011 > root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org: > /usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8 > amd64 > [2.0-RC1][root@pfSense.localdomain]/root(4): > > Enpoint B: > > [2.0-RC1][root@pfsense.localdomain]/root(1): uname -a > FreeBSD pfsense.localdomain 8.1-RELEASE-p3 FreeBSD 8.1-RELEASE-p3 #1: Tue > Apr 26 20:56:16 EDT 2011 > sullrich@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org: > /usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8 > i386 > [2.0-RC1][root@pfsense.localdomain]/root(2): > > > The link has a RTT of 130ms. The link's traffic is for the most part TCP > with UDP not being used much. The performance of the link in one direction > is fine (expected 160 odd Mbps). In the other direction it seems the link > struggles to climb even with a 3 MB window size and all sorts of tuning on > the stack. An output of the iperf test running for 20 seconds produces the > following results: > > Node A: > > [2.0-RC1][root@pfSense.localdomain]/root(24): iperf -c b.b.b.b -w 3M -t 20 > -i 5 > ------------------------------------------------------------ > Client connecting to b.b.b.b, TCP port 5001 > TCP window size: 3.00 MByte (WARNING: requested 3.00 MByte) > ------------------------------------------------------------ > [ 3] local a.a.a.a port 22995 connected with b.b.b.b port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0- 5.0 sec 75.3 MBytes 126 Mbits/sec > [ ID] Interval Transfer Bandwidth > [ 3] 5.0-10.0 sec 107 MBytes 180 Mbits/sec > [ ID] Interval Transfer Bandwidth > [ 3] 10.0-15.0 sec 107 MBytes 180 Mbits/sec > [ ID] Interval Transfer Bandwidth > [ 3] 15.0-20.0 sec 108 MBytes 181 Mbits/sec > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-20.2 sec 404 MBytes 167 Mbits/sec > [2.0-RC1][root@pfSense.localdomain]/root(25): > > > Node B: > > [2.0-RC1][root@pfsense.localdomain]/root(2): iperf -c a.a.a.a -t 20 -i 10 > -w > 3M > ------------------------------------------------------------ > Client connecting to a.a.a.a, TCP port 5001 > TCP window size: 3.00 MByte (WARNING: requested 3.00 MByte) > ------------------------------------------------------------ > [ 3] local b.b.b.b port 44160 connected with a.a.a.a port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 7.73 MBytes 6.49 Mbits/sec > [ ID] Interval Transfer Bandwidth > [ 3] 10.0-20.0 sec 9.95 MBytes 8.34 Mbits/sec > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-20.3 sec 18.1 MBytes 7.47 Mbits/sec > [2.0-RC1][root@pfsense.localdomain]/root(3): > > The performance is terrible in Node B. > > Their corresponding interfaces look like this: > > Node A: > > [2.0-RC1][root@pfSense.localdomain]/root(26): ifconfig igb1 > igb1: flags=28943 > metric 0 mtu 1500 > > > options=100bb > ether 00:1b:21:9b:6e:ab > inet6 fe80::21b:21ff:fe9b:6eab%igb1 prefixlen 64 scopeid 0x4 > nd6 options=3 > media: Ethernet autoselect (1000baseT ) > status: active > [2.0-RC1][root@pfSense.localdomain]/root(27): > > [2.0-RC1][root@pfSense.localdomain]/root(28): ifconfig igb1_593 > igb1_593: flags=8843 metric 0 mtu > 1500 > options=3 > ether 00:1b:21:9b:6e:ab > inet6 fe80::225:90ff:fe20:5510%igb1_593 prefixlen 64 scopeid 0xa > inet a.a.a.a netmask 0xfffffff8 broadcast x.x.x.x > nd6 options=3 > media: Ethernet autoselect (1000baseT ) > status: active > vlan: 593 parent interface: igb1 > [2.0-RC1][root@pfSense.localdomain]/root(29): > > Node B: > > [2.0-RC1][root@pfsense.localdomain]/root(48): ifconfig igb3 > igb3: flags=8843 metric 0 mtu 1500 > > > options=101bb > ether 00:1b:21:8e:2b:5f > inet6 fe80::21b:21ff:fe8e:2b5f%igb3 prefixlen 64 scopeid 0x4 > nd6 options=3 > media: Ethernet autoselect (1000baseT ) > status: active > [2.0-RC1][root@pfsense.localdomain]/root(49): ifconfig igb3_vlan593 > igb3_vlan593: flags=8843 metric 0 > mtu 1500 > options=3 > ether 00:1b:21:8e:2b:5f > inet6 fe80::21b:21ff:fe8e:2b5c%igb3_vlan593 prefixlen 64 scopeid 0xb > inet b.b.b.b netmask 0xfffffff8 broadcast x.x.x.x > nd6 options=3 > media: Ethernet autoselect (1000baseT ) > status: active > vlan: 593 parent interface: igb3 > [2.0-RC1][root@pfsense.localdomain]/root(50): > > > As you can see from the ifconfig output, the link runs dot1q with a VID of > 593. > > The netstat -m counters: > > Node A: > > [2.0-RC1][root@pfSense.localdomain]/root(29): netstat -m > 10261/10606/20867 mbufs in use (current/cache/total) > 10241/5957/16198/25600 mbuf clusters in use (current/cache/total/max) > 10240/5120 mbuf+clusters out of packet secondary zone in use > (current/cache) > 0/3989/3989/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 25612K/33173K/58785K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > [2.0-RC1][root@pfSense.localdomain]/root(30): > > Node B: > > [2.0-RC1][root@pfsense.localdomain]/root(50): netstat -m > 9691/4274/13965 mbufs in use (current/cache/total) > 6355/1891/8246/25600 mbuf clusters in use (current/cache/total/max) > 6354/1070 mbuf+clusters out of packet secondary zone in use (current/cache) > 1216/1271/2487/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 20091K/9934K/30026K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/8/6656 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > [2.0-RC1][root@pfsense.localdomain]/root(51): > > > dmesg output on both nodes. > > Node A: > > igb1: port > 0xe880-0xe89f mem > 0xf97e0000-0xf97fffff,0xf9c00000-0xf9ffffff,0xf97dc000-0xf97dffff irq 17 at > device 0.1 on pci3 > igb1: Using MSIX interrupts with 5 vectors > igb1: [ITHREAD] > igb1: [ITHREAD] > igb1: [ITHREAD] > igb1: [ITHREAD] > igb1: [ITHREAD] > > > Node B: > > igb3: port > 0xc880-0xc89f mem > 0xf9fe0000-0xf9ffffff,0xfa000000-0xfa3fffff,0xf9fdc000-0xf9fdffff irq 37 at > device 0.1 on pci6 > igb3: Using MSIX interrupts with 5 vectors > igb3: [ITHREAD] > igb3: [ITHREAD] > igb3: [ITHREAD] > igb3: [ITHREAD] > igb3: [ITHREAD] > igb3: link state changed to UP > igb3_vlan593: link state changed to UP > > > pciconf output: > > Node A: > > igb1@pci0:3:0:1: class=0x020000 card=0xa03c8086 chip=0x10c98086 > rev=0x01hdr=0x00 > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xf97e0000, size 131072, > enabled > bar [14] = type Memory, range 32, base 0xf9c00000, size 4194304, > enabled > bar [18] = type I/O Port, range 32, base 0xe880, size 32, enabled > bar [1c] = type Memory, range 32, base 0xf97dc000, size 16384, enabled > > > Node B: > > igb3@pci0:6:0:1: class=0x020000 card=0xa02b8086 chip=0x10e88086 > rev=0x01 hdr=0x00 > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xf9fe0000, size 131072, > enabled > bar [14] = type Memory, range 32, base 0xfa000000, size 4194304, > enabled > bar [18] = type I/O Port, range 32, base 0xc880, size 32, enabled > bar [1c] = type Memory, range 32, base 0xf9fdc000, size 16384, enabled > > > I've been looking at issues regarding performance with the igb issue from a > few years back at this thread : > http://lists.freebsd.org/pipermail/freebsd-doc/2009-June/015983.html. > These > are very similar problems to what I'm having however disabling LRO and TSO > does not work for me. > > I'm quite frustrated with this because I'm not getting an error or not > looking in the right place. Can someone out there point me in the right > direction? Any help will be much appreciated. > > Warm Regards, > Mario Spinthiras > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >