Date: Tue, 20 Nov 2012 20:16:49 -0600 From: khatfield@socllc.net To: Adrian Chadd <adrian@freebsd.org> Cc: Barney Cordoba <barney_cordoba@yahoo.com>, Jim Thompson <jim@netgate.com>, Alfred Perlstein <bright@mu.org>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: FreeBSD boxes as a 'router'... Message-ID: <250266404.35502.1353464214924@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> In-Reply-To: <CAJ-Vmok8Ybdi%2BY8ZguMTKC7%2BF5=OxVDog27i4UgY-s3MCZkGcQ@mail.gmail.com> References: <1353448328.76219.YahooMailClassic@web121602.mail.ne1.yahoo.com> <E1F4816E-676C-4630-9FA1-817F737D007D@netgate.com> <50AC08EC.8070107@mu.org> <832757660.33924.1353460119408@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> <CAJ-Vmok8Ybdi%2BY8ZguMTKC7%2BF5=OxVDog27i4UgY-s3MCZkGcQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I may be misstating. Specifically under high burst floods either routed or being dropped by pf w= e would see the system go unresponsive to user-level applications / SSH for= example. The system would still function but it was inaccessible. To clarify as well= this was any number of floods or attacks to any ports, the behavior remain= ed. (These were not SSH ports being hit) Now we did a lot of sysctl resource tuning to correct this with some floods= but high rate would still cause the behavior. Other times the system would= simply drop all traffic (like a buffer filled or max connections) but it w= as not either case.=20 The attacks were also well within bandwidth capabilities for the pipe and n= etwork gear. All of these issues stopped upon adding polling or the overall threshold wa= s increased tremendously with polling. Yet, polling has some downsides not necessarily due to FreeBSD but applicat= ion issues. Haproxy is one example where we had handshake/premature connect= ions terminated with polling. Those issues were not present with polling di= sabled.=20 So that is my reasoning for saying that it was perfect for some things and = not for others. In the end, we spent years tinkering and it was always satisfactory but nev= er perfect. Finally we grew to the point of replacing the edge with MX80's = and left BSD to load balancing and the like. This finally resolved all issu= es for us. Albeit, we were a DDoS mitigation company running high PPS and lots of burs= ting. BSD was beautiful until we ended up needing 10Gps+ on the edge and it= was time to go Juniper. I still say BSD took us from nothing to a $30M company. So despite somethin= g's requiring tinkering with I think it is still worth the effort to put in= the testing to find what is best for your gear and environment. I got off-track but we did find one other thing. We found ipfw did seem to = reduce load on the interrupts (likely because we couldn't do near the scrub= bing with it vs pf) at any rate less filtering may also fix the issue with = the op.=20 Your forwarding - we found doing forwarding via a simple pf rule and a GRE = tunnel to an app server or by using a tool like haproxy on the router itsel= f seemed to reduce a large majority of our original stability issues (verse= s pure fw-based packet forwarding) *I also agree because as I mentioned in a previous email... (To me) our ove= rall PPS seemed to decrease from FBSD 7 to 9. No idea why but we seemed to = begin having less effect with polling as we seemed to get with polling on 7= .4. Not to say that this wasn't due to error on our part or some issue with th= e Juniper switches but we seemed to just run into more issues with newer re= leases when it came to performance with Intel 1Gbps NIC's. this later cause= d us to move more app servers to Linux because we never could get to the bo= ttom of some of those things. We do intend to revisit BSD with our new CDN = company to see if we can restandardize it for high volume traffic servers. Best, Kevin=20 On Nov 20, 2012, at 7:19 PM, "Adrian Chadd" <adrian@freebsd.org> wrote: > Ok, so since people are talking about it, and i've been knee deep in > at least the older intel gige interrupt moderation - at maximum pps, > how exactly is the interrupt moderation giving you a livelock > scenario? >=20 > The biggest benefit I found when doing some forwarding work a few > years ago was to write a little daemon that actually sat there and > watched the interrupt rates and packet drop rates per-interface - and > then tuned the interrupt moderation parameters to suit. So at the > highest pps rates I wasn't swamped with interrupts. >=20 > I think polling here is hiding some poor choices in driver design and > network stack design.. >=20 >=20 >=20 > adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?250266404.35502.1353464214924>