Date: Thu, 26 Aug 2010 12:53:57 -0700 From: Pyun YongHyeon <pyunyh@gmail.com> To: MAI JIN <Jin.Mai@alcatel-sbell.com.cn> Cc: freebsd-net@freebsd.org Subject: Re: HELP. FreeBSD 8.1 polling issue Message-ID: <20100826195357.GC16395@michelle.cdnetworks.com> In-Reply-To: <1DB91DF937A4544C81E636468B91C21C0728EA30@CNSHGSMBS03.ad4.ad.alcatel.com> References: <1DB91DF937A4544C81E636468B91C21C0728EA30@CNSHGSMBS03.ad4.ad.alcatel.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Aug 26, 2010 at 01:34:45PM +0800, MAI JIN wrote: > Hi, > > I got a freeBSD 8.1 polling issue on my PC. It is a dual-core Intel > Pentium x86 PC (2.8GHz each core). The Ethernet interface is Broadcom > NetXtreme 57xx Gigabit Ethernet interface. > I set the following options (enable polling and zero-buffer copy) and > rebuilt the kernel: > > Code: > # To make an SMP kernel, the next two lines are needed > options SMP # Symmetric MultiProcessor > Kernel > device apic # I/O APIC > > options DEVICE_POLLING # Open Polling > options HZ=1000 > options ZERO_COPY_SOCKETS > The following were appended to the /etc/sysctl.conf > > Code: > kern.polling.enable=1 > # increase BPF buffer to 10M > net.bpf.bufsize=10485760 > net.bpf.maxbufsize=10485760 > kern.polling.idle_poll=1 > kern.polling.burst_max=1000 > After installed and rebooted the system, kern.polling.enable was not > found in MIB so I had to ignore this error. Looks like > kern.polling.enable is removed from FreeBSD v8.1? > Everything looked good so build my application to received data from > another HP server. I wrote the application using libpcap-1.1.1 with BFP > zero-copy turned on (I found the #define HAVE_ZEROCOPY_BPF 1 in > config.h). Attached please find the source code of my application. > > Before running the application, I set the following parameters: > > Code: > ifconfig bge0 polling # This will turn on the polling of the > Broadcom driver. > Code: > sysctl -w net.bpf.bufsize=10485760 > sysctl -w net.bpf.maxbufsize=10485760 > sysctl -w kern.polling.idle_poll=1 > sysctl -w kern.polling.burst_max=1000 > sysctl -w kern.polling.each_burst=128 > sysctl -w net.inet.ip.intr_queue_maxlen=256 > Then I ran the application to receive data from the HP server. I ran > multiple iperf on the HP server to send around 133Mbits/s UDP load to > the PC under test. The UDP payload size was 47 bytes. The entire IP > packet size is 76 bytes. > > First of all, the receiving application worked well and received around > 205K packets/second without packet losing (I checked the receiving > status using pcap_stats). However, after 2 minutes, the application can > not received data any more. The packets/second is 0. I ran the ping from > the PC under test and found that the ping reporting timeout and > destination unreachable (the ping from HP to the PC also failed). Looked > like the link between the HP server and PC was broken so the application > could receive data. No packet was dropped. Then I restart the bge0 > interface using: ifconfig bge0 down && ifconfig bge0 up > > And then I re-ran the application and it continued receiving data. But > after 1 or 2 minutes, the link broke again. I think it was my > application that caused the bge0 interface down. I started the tcpdump > and it worked well without breaking the link. > > I tried to increase the kern.polling.each_burst from 128 to 500 but the > application would cause the bge0 down within 1 minute. No packet was > dropped before the link was down. > > I checked the CPU usage of the PC. The sys used is around 90% (might be > caused by kern.polling.idle_poll=1), user land is 13%. > I don't understand why the application would break the bge0. > > I tried changing the parameters: > options HZ=2000 > > sysctl -w net.bpf.bufsize=20485760 > sysctl -w net.bpf.maxbufsize=20485760 > sysctl -w kern.polling.idle_poll=1 > sysctl -w kern.polling.burst_max=10000 > sysctl -w kern.polling.each_burst=5000 > > The performance was better: I got 307K packet/second (the HP server > sended around 250Mbits/s, my PC got 200Mbits/s). But after 2 minutes, > the bge0 was down again. > I'm not a fan of polling(4) especially for intelligent controllers like bge(4) but it seems bge(4) was dead under high network load. Would you show me the output of both verbose dmesg and "pciconf -lcbv"? > Could anybody have a look at this issue? How can <<cap.cpp>> I optimize > the performance of the polling? > > Thanks, > Jin >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100826195357.GC16395>