Date: Wed, 8 Mar 2017 05:25:57 -0700 From: Kevin Bowling <kevin.bowling@kev009.com> To: "Eugene M. Zheganin" <emz@norma.perm.ru> Cc: freebsd-net <freebsd-net@freebsd.org> Subject: Re: about that DFBSD performance test Message-ID: <CAK7dMtDiT-PKyy5LkT1WEg5g-nwqv501F=Ap4dNCdwzwr_1dqA@mail.gmail.com> In-Reply-To: <b91a6e40-9956-1ad9-ac59-41a281846147@norma.perm.ru> References: <b91a6e40-9956-1ad9-ac59-41a281846147@norma.perm.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
Right off the bat, FreeBSD doesn't really understand NUMA in any sufficient capacity. Unfortunately at companies like the one I work at, we take that to mean "OK buy a high bin CPU and only populate one socket" which serves us well and may ultimately be the best value but does nothing to address the reality that multi-socket NUMA systems are common place and FreeBSD should run in the same league on them as other operating systems to be taken seriously. I'd be interested in seeing how the contenders look by removing one of the CPUs as it might at least put a spotlight on that issue. With respect to SO_REUSEPORT, we are investigating implementing this round robin behavior right now. We think librss is a better way to go for content serving, but software will get written with Linux APIs in mind so I think it'll be a good idea to have a distributing SO_REUSEPORT anyway for software that does not understand librss natively. I have no commercial need for kernel IP forwarding but Matt Macy is doing a lot of driver work for me and we are at least trying to keep it in line with the legacy drivers. Since sephe's test was ixgbe, it'd be interesting to test https://github.com/mattmacy/networking/commits/HEAD_MERGE/iflib-ixgbe-current which makes the FreeBSD ixgbe driver much more like cxgbe (same ring code, improved coalescing, various prefetching insights for 10g+). In talking with Matt there's probably some low hanging fruit at the driver layer for this workload around batching and distribution. I do think the use of polling(4) in the benchmark is a bit hacky, it should be possible to get similar performance to Linux by doing batching and distribution as we are fine tuning in the iflib project. I'd be looking at netmap before polling(4) in any new product development that does heavy forwarding. But there's probably more to do up stack in routing and I don't know of anyone working on it with significant commercial backing. Ultimately I have to say congrats to DFBSD. They pulled that performance off with a small team and very little to no commercial backing. FreeBSD has a lot of full time commercial contributors, and while this is generally a good thing for longevity, I think workplace culture (even/especially at fabled places, that just means you're super busy) defeats some of the lucid insight and tenacity that ardent personal contributors are able to pull off of their own accord. Regards, On Tue, Mar 7, 2017 at 9:00 PM, Eugene M. Zheganin <emz@norma.perm.ru> wrote: > Hi. > > Some have probably seen this already - http://lists.dragonflybsd.org/ > pipermail/users/2017-March/313254.html > > So, could anyone explain why FreeBSD was owned that much. Test is split > into two parts, one is nginx part, and the other is the IPv4 forwarding > part. I understand that nginx ownage was due to SO_REUSEPORT feature, which > we do formally have, but in DFBSD and Linux it does provide a kernel socket > multiplexor, which eliminates locking, and ours does not. I have only found > traces of discussion that DFBSD implementation is too hackish. Well, > hackish or not, but it's 4 times faster, as it turns out. The IPv4 > forwarding loss is pure defeat though. > > Please not that although they use HEAD it these tests, they also mention > that this is the GENERIC-NODEBUG kernel which means this isn't related to > the WITNESS stuff. > > Please also don't consider this trolling, I'm a big FreeBSD fan through > the years, so I'm asking because I'm kind of concerned. > > Eugene. > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAK7dMtDiT-PKyy5LkT1WEg5g-nwqv501F=Ap4dNCdwzwr_1dqA>