Date: Mon, 22 Sep 2014 11:11:04 -0700 From: Rumen Telbizov <telbizov@gmail.com> To: Adrian Chadd <adrian@freebsd.org> Cc: Tom Elite <qingli@freebsd.org>, "freebsd-stable@freebsd.org" <freebsd-stable@freebsd.org>, "K. Macy" <kmacy@freebsd.org> Subject: Re: FreeBSD 10 network performance problems Message-ID: <CAENR%2B_UW-POPbSgsqZA-pUV0Be4AABUOEfTZqbTbJi3OZR5sTQ@mail.gmail.com> In-Reply-To: <CAJ-VmomJckhbJP4Ks_eiomNeSVaMH9g62%2BBimBZcWpMooSA%2BDQ@mail.gmail.com> References: <CAENR%2B_VDVvnY0zWKVXHOjz2vWw27s%2ByVrz9ZFokZ=p6P6oFNvw@mail.gmail.com> <1411259605.674669006.339g4pd4@frv35.fwdcdn.com> <CAENR%2B_UwjMGoOqKkhCeL4zmLnSgTiABoeR-7x71MBvOnCF8z%2BA@mail.gmail.com> <CAHM0Q_MXmn2P=vfFgaj5pZSqcTSm3h%2BKPAotc4K_8qpQUFh1dQ@mail.gmail.com> <CAJ-VmomJckhbJP4Ks_eiomNeSVaMH9g62%2BBimBZcWpMooSA%2BDQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Thank you all for the answers and directions. I tried all of the suggested changes to sysctl.conf and loader.conf settings above - they made no difference whatsoever. I believe they might help in certain situations but will improve things marginally. What I am dealing with is a pretty hard and sudden drop of performance due to lock contention. Adrian: What you're saying makes sense. If we can avoid the locking completely, this problem might go away. I'll see if I can get some help and try to tackle this challenge. Cheers, Rumen Telbizov On Sun, Sep 21, 2014 at 11:29 PM, Adrian Chadd <adrian@freebsd.org> wrote: > Hi! > > On 21 September 2014 15:08, K. Macy <kmacy@freebsd.org> wrote: > > What you're dealing with is hardly an edge case. Most people don't need > to > > push more than a couple of Gbps in production. > > > > Flowtable is hardly "untested." However, it has been a source of friction > > at times because it can be somewhat brittle, having limits on the number > of > > cache entries that it can store that are frequently too low for people > with > > very large numbers of active flows. Without raising this limit > > substantially these systems will fail in a rather spectacular fashion. > > Additionally, flowtable was not written with the intent of being a > routing > > cache. It was developed to support stateful multipath routing for load > > balancing. In its current incarnation, stripped of much of the code for > its > > initial purpose, it's really just a band-aid around locking problems in > > routing. That said, the handful of commercial users of FreeBSD that do > have > > large amounts of traffic (10s of Gbps) per system that I personally know > of > > all have flowtable enabled. > > > > Unfortunately, at least in terms of what is in HEAD, little has been done > > to fix the contention that flowtable works around. For your purposes the > > response that Adrian gave you is the closest to "optimal." > > Gleb and I spent a bunch of time late last year and early this year > finding and fixing a lot of the corner cases with Flowtable handling. > > I'm pretty sure that it'll behave predictably and reliably for people > now. If it doesn't then please file PRs. It's no longer some corner of > the codebase that isn't well understood. At least two people besides > the author (Gleb and I) understand what it is, how it works and how it > ties into things. > > What's missing is someone sitting down and adding flowtable support to > the rest of the forwarding path(s). It shouldn't be too hard - as long > as you have an mbuf that has the IPv4/IPv6 header setup as that's what > flowtable currently expects when doing lookups - but it has to be > done. > > So I thoroughly recommend someone who has the test setup and the > desire to do so and post results. I have enough equipment now to test > this out and develop it but I'm really busy doing work, wireless RSS > stuff. So I just don't have the spare cycles to do it. > > I do think it'll be a pretty simple task. > > In theory - once you have the flowtable code working correctly in the > forwarding path you shouldn't see any rtentry lock contention except > during route changes (which will invalidate flowtable entries and > cause normal routing table lookups until the flowtable has all the > route entries in question.) > > > > -a > -- Rumen Telbizov Unix Systems Administrator <http://telbizov.com>;
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAENR%2B_UW-POPbSgsqZA-pUV0Be4AABUOEfTZqbTbJi3OZR5sTQ>