From owner-freebsd-net@FreeBSD.ORG Tue Jun 14 14:39:33 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E26A1065672 for ; Tue, 14 Jun 2011 14:39:33 +0000 (UTC) (envelope-from spinthiras.mario@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id F37978FC0A for ; Tue, 14 Jun 2011 14:39:32 +0000 (UTC) Received: by vxc34 with SMTP id 34so6494822vxc.13 for ; Tue, 14 Jun 2011 07:39:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=ABS+6DY+mcCUijjg9tKZhRIZ1YoXcPPdLSPWNwwkhbU=; b=aAtbFpHFT5X4y0ubVM01/hcemFIf8Hq92ch332oXmnmNwkqKKoT7u0pAG8WVn0N8PM bprovl7SZPuE7RN/2GcZvGILd/7teQ2Go7R8w9+XqCLV6t9tYy2i7Vkf1zexJXRkHfjJ NyIwUWsYuJL4YQgjwY8qoMlZZBdJQixkfcbFQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=HSZy+05PvFgvhs9ygKi0msiJ7yAAbNPmLlZAnx49rDxQG6FNaLmECoEU3yNbCqICzc J3HnsaP/SKuLpgkkC+6B3VsC7pmkzXMVyb5qfPN2iWNZ0+ZGsXUJsjHTPaCNRwVEdOqW PERAzQ5tz0s39AIOMy0NeRBovAQp7Ls/Z4590= Received: by 10.52.177.104 with SMTP id cp8mr5743009vdc.132.1308062372184; Tue, 14 Jun 2011 07:39:32 -0700 (PDT) MIME-Version: 1.0 Received: by 10.52.186.4 with HTTP; Tue, 14 Jun 2011 07:38:52 -0700 (PDT) In-Reply-To: References: From: Mario Spinthiras Date: Tue, 14 Jun 2011 15:38:52 +0100 Message-ID: To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: strange igb interface performance problems X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jun 2011 14:39:33 -0000 Hello everyone, I'm back with more updates on the issue. Forgive my lack of elaboration on the topic in my previous post; over the frustration of this problem I couldn't see this topic through so I let it go for a few days to revisit with a clear head perhaps. As I mentioned in a previous post, the problem is between 2 nodes running PfSense 2.0 RC1 over a point to point link. The uname from the machines: Endpoint A: [2.0-RC1][root@pfSense.localdomain]/root(3): uname -a FreeBSD pfSense.localdomain 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 #0: Sat Feb 26 18:05:58 EST 2011 root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64 [2.0-RC1][root@pfSense.localdomain]/root(4): Enpoint B: [2.0-RC1][root@pfsense.localdomain]/root(1): uname -a FreeBSD pfsense.localdomain 8.1-RELEASE-p3 FreeBSD 8.1-RELEASE-p3 #1: Tue Apr 26 20:56:16 EDT 2011 sullrich@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8 i386 [2.0-RC1][root@pfsense.localdomain]/root(2): The link has a RTT of 130ms. The link's traffic is for the most part TCP with UDP not being used much. The performance of the link in one direction is fine (expected 160 odd Mbps). In the other direction it seems the link struggles to climb even with a 3 MB window size and all sorts of tuning on the stack. An output of the iperf test running for 20 seconds produces the following results: Node A: [2.0-RC1][root@pfSense.localdomain]/root(24): iperf -c b.b.b.b -w 3M -t 20 -i 5 ------------------------------------------------------------ Client connecting to b.b.b.b, TCP port 5001 TCP window size: 3.00 MByte (WARNING: requested 3.00 MByte) ------------------------------------------------------------ [ 3] local a.a.a.a port 22995 connected with b.b.b.b port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 5.0 sec 75.3 MBytes 126 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 5.0-10.0 sec 107 MBytes 180 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 10.0-15.0 sec 107 MBytes 180 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 15.0-20.0 sec 108 MBytes 181 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 0.0-20.2 sec 404 MBytes 167 Mbits/sec [2.0-RC1][root@pfSense.localdomain]/root(25): Node B: [2.0-RC1][root@pfsense.localdomain]/root(2): iperf -c a.a.a.a -t 20 -i 10 -w 3M ------------------------------------------------------------ Client connecting to a.a.a.a, TCP port 5001 TCP window size: 3.00 MByte (WARNING: requested 3.00 MByte) ------------------------------------------------------------ [ 3] local b.b.b.b port 44160 connected with a.a.a.a port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 7.73 MBytes 6.49 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 10.0-20.0 sec 9.95 MBytes 8.34 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 0.0-20.3 sec 18.1 MBytes 7.47 Mbits/sec [2.0-RC1][root@pfsense.localdomain]/root(3): The performance is terrible in Node B. Their corresponding interfaces look like this: Node A: [2.0-RC1][root@pfSense.localdomain]/root(26): ifconfig igb1 igb1: flags=28943 metric 0 mtu 1500 options=100bb ether 00:1b:21:9b:6e:ab inet6 fe80::21b:21ff:fe9b:6eab%igb1 prefixlen 64 scopeid 0x4 nd6 options=3 media: Ethernet autoselect (1000baseT ) status: active [2.0-RC1][root@pfSense.localdomain]/root(27): [2.0-RC1][root@pfSense.localdomain]/root(28): ifconfig igb1_593 igb1_593: flags=8843 metric 0 mtu 1500 options=3 ether 00:1b:21:9b:6e:ab inet6 fe80::225:90ff:fe20:5510%igb1_593 prefixlen 64 scopeid 0xa inet a.a.a.a netmask 0xfffffff8 broadcast x.x.x.x nd6 options=3 media: Ethernet autoselect (1000baseT ) status: active vlan: 593 parent interface: igb1 [2.0-RC1][root@pfSense.localdomain]/root(29): Node B: [2.0-RC1][root@pfsense.localdomain]/root(48): ifconfig igb3 igb3: flags=8843 metric 0 mtu 1500 options=101bb ether 00:1b:21:8e:2b:5f inet6 fe80::21b:21ff:fe8e:2b5f%igb3 prefixlen 64 scopeid 0x4 nd6 options=3 media: Ethernet autoselect (1000baseT ) status: active [2.0-RC1][root@pfsense.localdomain]/root(49): ifconfig igb3_vlan593 igb3_vlan593: flags=8843 metric 0 mtu 1500 options=3 ether 00:1b:21:8e:2b:5f inet6 fe80::21b:21ff:fe8e:2b5c%igb3_vlan593 prefixlen 64 scopeid 0xb inet b.b.b.b netmask 0xfffffff8 broadcast x.x.x.x nd6 options=3 media: Ethernet autoselect (1000baseT ) status: active vlan: 593 parent interface: igb3 [2.0-RC1][root@pfsense.localdomain]/root(50): As you can see from the ifconfig output, the link runs dot1q with a VID of 593. The netstat -m counters: Node A: [2.0-RC1][root@pfSense.localdomain]/root(29): netstat -m 10261/10606/20867 mbufs in use (current/cache/total) 10241/5957/16198/25600 mbuf clusters in use (current/cache/total/max) 10240/5120 mbuf+clusters out of packet secondary zone in use (current/cache) 0/3989/3989/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 25612K/33173K/58785K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines [2.0-RC1][root@pfSense.localdomain]/root(30): Node B: [2.0-RC1][root@pfsense.localdomain]/root(50): netstat -m 9691/4274/13965 mbufs in use (current/cache/total) 6355/1891/8246/25600 mbuf clusters in use (current/cache/total/max) 6354/1070 mbuf+clusters out of packet secondary zone in use (current/cache) 1216/1271/2487/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 20091K/9934K/30026K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/8/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines [2.0-RC1][root@pfsense.localdomain]/root(51): dmesg output on both nodes. Node A: igb1: port 0xe880-0xe89f mem 0xf97e0000-0xf97fffff,0xf9c00000-0xf9ffffff,0xf97dc000-0xf97dffff irq 17 at device 0.1 on pci3 igb1: Using MSIX interrupts with 5 vectors igb1: [ITHREAD] igb1: [ITHREAD] igb1: [ITHREAD] igb1: [ITHREAD] igb1: [ITHREAD] Node B: igb3: port 0xc880-0xc89f mem 0xf9fe0000-0xf9ffffff,0xfa000000-0xfa3fffff,0xf9fdc000-0xf9fdffff irq 37 at device 0.1 on pci6 igb3: Using MSIX interrupts with 5 vectors igb3: [ITHREAD] igb3: [ITHREAD] igb3: [ITHREAD] igb3: [ITHREAD] igb3: [ITHREAD] igb3: link state changed to UP igb3_vlan593: link state changed to UP pciconf output: Node A: igb1@pci0:3:0:1: class=0x020000 card=0xa03c8086 chip=0x10c98086 rev=0x01hdr=0x00 class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xf97e0000, size 131072, enabled bar [14] = type Memory, range 32, base 0xf9c00000, size 4194304, enabled bar [18] = type I/O Port, range 32, base 0xe880, size 32, enabled bar [1c] = type Memory, range 32, base 0xf97dc000, size 16384, enabled Node B: igb3@pci0:6:0:1: class=0x020000 card=0xa02b8086 chip=0x10e88086 rev=0x01 hdr=0x00 class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xf9fe0000, size 131072, enabled bar [14] = type Memory, range 32, base 0xfa000000, size 4194304, enabled bar [18] = type I/O Port, range 32, base 0xc880, size 32, enabled bar [1c] = type Memory, range 32, base 0xf9fdc000, size 16384, enabled I've been looking at issues regarding performance with the igb issue from a few years back at this thread : http://lists.freebsd.org/pipermail/freebsd-doc/2009-June/015983.html. These are very similar problems to what I'm having however disabling LRO and TSO does not work for me. I'm quite frustrated with this because I'm not getting an error or not looking in the right place. Can someone out there point me in the right direction? Any help will be much appreciated. Warm Regards, Mario Spinthiras