Date: Tue, 23 Feb 2010 11:58:34 +0400 From: rihad <rihad@mail.ru> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: freebsd-net@freebsd.org Subject: Re: Slow speeds experienced with Dummynet Message-ID: <4B838AAA.7080104@mail.ru> In-Reply-To: <20100220095941.GA82976@onelab2.iet.unipi.it> References: <4B7EDD00.5080308@mail.ru> <20100220095941.GA82976@onelab2.iet.unipi.it>
next in thread | previous in thread | raw e-mail | index | archive | help
Luigi Rizzo wrote: > On Fri, Feb 19, 2010 at 10:48:32PM +0400, rihad wrote: >> Hi, all, >> >> Recalling my old posting "dummynet dropping too many packets" dated >> October 4, 2009, the problem isn't over just yet. This time, there are >> no interface i/o drops (just a reminder: we have 2 bce(4) GigE cards >> connected to a Cisco router, one for input, and one for output. The box >> itself does some traffic accounting and enforces speed limits w/ >> ipfw/dummynet. There are normally around 5-6k users online). > > If i remember well, the previous discussion ended when you > raised the intr_queue_maxlen (and perhaps increased HZ) to > avoid that the bursts produced by periodic invocation of > dummynet_io() could overflow that queue. > >>From the rest of your post it is not completely clear if you have > not found any working configuration, or there is some setting (e.g. > with "queue 1000" or larger) which do produce a smooth experience > for your customers. > Nope, I haven't been able to find it :( From as low as 50 to 2000 slots. I've also tried 1 and 2s worth queue sizes (in Kbytes). After about 8 p.m., when most users are online, speed problems are the most apparent. It all boils down to big differences between some subsequent "systat -if" reads, both for input and output, when dummynet is in use. It isn't normal for two reads to differ in as much as 100-150 mbps (20-25%). I can only hope that Intel's expensive 10 GigE cards have much larger tolerance to traffic load spikes, and are better suited for ISP usage patterns. > Another thing i'd like to understand is whether all of your pipes > have a /32 mask, or there are some which cover multiple hosts. > Typical TCP connections have around 50 packets in flight when the > connection is fully open (which in turn is hard to happen on a 512k > pipe) so a queue of 100-200 is unlikely to overflow. > > In fact, long queues are very detrimental for customers because > they increase the delay of the congestion control loop -- as a rule > of thumb, you should try to limit the queue size to at most 1-2s > of data. > > cheers > luigi > >> Traffic shaping is accomplished by this ipfw rule: >> pipe tablearg ip from any to table(0) out >> where table(0) contains those 5-6k IP addresses. The pipes themselves >> are GRED (or taildrop, it doesn't matter): >> ipfw pipe 512 config bw 512kbit/s mask dst-ip 0xffffffff gred >> 0.002/900/1000/0.1 queue 1000 >> Taking this template the speeds range from 512 to tens of mbps. With the >> setup as above very many users complain about very slow downloads, slow >> browsing. systat -ifstat, refreshed every 5 seconds, does reveal large >> differences between subsequent displays: if around 800-850 mbps is >> what's to be expected, it doesn't stay within those limits, also jumping >> to as low as 620+, and to somewhere in the 700's, Now imagine this: once >> I turn dummynet off (by short-circuiting a "allow ip from any to any" >> before the "pipe tablearg" rule) all user complaints stop, with traffic >> load staying stably at around 930-950 mbps. >> >> Does this have anything to do with "dummynet bursts"? How can I beat >> that? If I keep the pipe queue size at 2000 slots, the >> net.inet.ip.dummynet.io_pkt_drops sysctl stops increasing, once I start >> tweaking the value to as low as 100 slots, the value starts raising >> constantly at about 300-500 pps. I had hoped that smaller queue sizes >> would battle the negative effects of dummynet burstiness, it did so, I >> guess, but not in a very decisive manner. >> >> >> FreeBSD 7.1-RELEASE-p10 >> kern.hz=4000 >> kern.ipc.nmbclusters=111111 >> net.inet.ip.fastforwarding=1 >> net.inet.ip.dummynet.io_fast=1 >> net.isr.direct=0 >> net.inet.ip.intr_queue_maxlen=5000 >> net.inet.ip.dummynet.hash_size=512 >> net.inet.ip.dummynet.max_chain_len=16 >> >> net.inet.ip.intr_queue_drops: 0 >> systat -ip shows zero output drops at times of trouble. netstat -s's >> "output packets dropped due to no bufs, etc." is also fine. netstat -m >> shows nothing suspicious. >> >> p.s: Two "bloody expensive" Intel 10 GigE's are on their way to us to >> replace the Broadcom cards, meanwhile what should I try doing? Thanks >> for reading. >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4B838AAA.7080104>