Date: Wed, 6 Jun 2007 09:41:59 +0200 From: David =?utf-8?q?Sieb=C3=B6rger?= <drs@rucus.ru.ac.za> To: freebsd-net@freebsd.org Subject: bge interfaces: poor transmit performance? Message-ID: <200706060941.59943.drs@rucus.ru.ac.za>
next in thread | raw e-mail | index | archive | help
I'm experiencing a problem with BCM5721 bge interfaces, which seem to be able to receive at almost 1Gbps but can only transmit at =< 560Mbps. I first noticed the problem on a firewall which routes between a number of subnets, all connected as VLAN interfaces on bge0. In its case, transmitted traffic and received traffic should be roughly equal (minus those packets which the firewall drops) but I've seen that throughput hits a limit at 560Mbps (measured from the interface stats using slurm). To investigate further, I've connected two of the same machines with a flylead and run some iperf tests: Test: iperf -c172.30.3.x -w256k -t30 -P8 Client Server Result -------------------------------------------------------- 1. 6.2-STABLE 7.0-CURRENT 499 Mbits/sec 2. 7.0-CURRENT 6.2-STABLE 526 Mbits/sec 3. 6.2-STABLE Linux 500 Mbits/sec 4. Linux 6.2-STABLE 941 Mbits/sec 5. Linux Linux 941 Mbits/sec Test: iperf -c172.30.3.x -w256k -t30 -P4 -d Client Server Result -------------------------------------------------------- 6. 6.2-STABLE 7.0-CURRENT 381 & 388 Mbits/sec 7. 7.0-CURRENT 6.2-STABLE 369 & 405 Mbits/sec 8. 6.2-STABLE Linux 423 & 537 Mbits/sec 9. Linux 6.2-STABLE 421 & 554 Mbits/sec 10. Linux Linux 833 & 830 Mbits/sec (Hardware: Dell PE860, onboard BCM5721 NICs, 2.4 GHz Xeon 3060 CPU. FreeBSD tuning: net.inet.ip.fw.enable=0, kern.ipc.maxsockbuf=8192000, net.inet.tcp.sendspace=262144, net.inet.tcp.recvspace=262144, WITNESS and INVARIANTS disabled on -CURRENT. Linux: Knoppix 5.1.1 with kernel 2.6.19, using tg3 driver.) The most interesting result I see there is the difference between tests 3 and 4: just changing the direction of traffic flow makes a major difference to performance. Possible causes that (I think) have been eliminated: * IRQ sharing. 'vmstat -i | grep bge' on all machines looks similar to this: irq16: bge0 374558 5 irq17: bge1 324860 4 * Hardware architecture. If Linux can make it perform well, there's no fundamental PCI bus bandwidth limitation or anything like that. * Network errors. 'netstat -i' has always shown Ierrs and Oerrs = 0. * SMP. I've tried building SMP and non-SMP kernels and got almost idenitical results. * HZ. I've tried kernels with the default HZ=1000 and with HZ=2500. Does anyone have any ideas as to what could be causing the problem, or any other tests I could try that might shed light on the problem? -- David Siebörger drs@rucus.ru.ac.za
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200706060941.59943.drs>