Date: Tue, 06 Oct 2009 20:28:35 +0500 From: rihad <rihad@mail.ru> To: Eugene Grosbein <eugen@kuzbass.ru> Cc: freebsd-net@freebsd.org, Luigi Rizzo <rizzo@iet.unipi.it>, Julian Elischer <julian@elischer.org> Subject: Re: dummynet dropping too many packets Message-ID: <4ACB6223.1000709@mail.ru> In-Reply-To: <20091006142152.GA42350@svzserv.kemerovo.su> References: <4AC9D87E.7000005@mail.ru> <20091005120418.GA63131@onelab2.iet.unipi.it> <4AC9E29B.6080908@mail.ru> <20091005123230.GA64167@onelab2.iet.unipi.it> <4AC9EFDF.4080302@mail.ru> <4ACA2CC6.70201@elischer.org> <4ACAFF2A.1000206@mail.ru> <4ACB0C22.4000008@mail.ru> <20091006100726.GA26426@svzserv.kemerovo.su> <4ACB42D2.2070909@mail.ru> <20091006142152.GA42350@svzserv.kemerovo.su>
next in thread | previous in thread | raw e-mail | index | archive | help
Eugene Grosbein wrote: > On Tue, Oct 06, 2009 at 06:14:58PM +0500, rihad wrote: > >>> No, generally handles much more. Please show your ipfw rule(s) >>> containing 'tablearg'. >> 01031 x x allow ip from any to any >> 01040 x x skipto 1100 ip from table(127) to any out >> recv bce0 xmit bce1 >> 01060 x x pipe tablearg ip from any to table(0) out >> recv bce0 xmit bce1 >> 01070 x x allow ip from any to table(0) out recv bce0 >> xmit bce1 >> 01100 x x pipe tablearg ip from any to table(2) out >> 65535 x x allow ip from any to any >> >> table(127) contains country-wide ISPs' netblocks (under 100 entries). >> table(0) and table(2) contain same user IP addresses, but different pipe >> IDs - normally around 3-4k entries each. >> >> Now please pay special attention to rule 1031. I've added it to bypass >> dummynet and stop packets from being dropped for now. Normally the rule >> isn't there. >> >> As I found out today after rebooting, drops only start occurring when >> the number of entries in table(0) exceeds 2000 or so (please see my >> previous email). Maybe it's a coincidence - I don't know. Global traffic >> load doesn't matter - it was approximately the same before and after the >> drops (around 450 mbit/s). > > It's possible that pipe lookup by its number is inefficient > and firewall keeps its lock for too much time while searching the pipe, > just a guess. And packets start to drop, eh? > > Try setting net.isr.direct to 0 and make large net.inet.ip.intr_queue_maxlen. > This way, one of your cores may run bce's thread, enqueue incoming > packets and return to work immediately. The rest of processing may be > performed by another kernel thread, hopefully using another core. > Just to see if this changes anything. top -S should help here too. > Since this is a remote production box, I'm really scared of toggling such on/off flags I've never used before, particularly under heavy traffic loads, they're way too eager to lock up the whole system. I might try this tomorrow morning, though, first on another less critical box. p.s.: I've just tested toggling the flag on my virtual machine, it went fine. I don't think net.inet.ip.intr_queue_maxlen is relevant to this problem, as net.inet.ip.intr_queue_drops is normally zero or very close to it at all times.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4ACB6223.1000709>