Date: Sat, 8 Sep 2007 22:27:10 +0200 From: Fabien THOMAS <thomas.fabien@gmail.com> To: freebsd-net@freebsd.org Cc: daffy@xview.net Subject: Re: new version of polling for FreeBSD 6.x Message-ID: <6DED4888-AE5E-4BF6-85CF-6968ED7BEBF4@gmail.com>
next in thread | raw e-mail | index | archive | help
> > Hello Fabien, Hello :) > > 1- I have noticed you are not using GENERIC config file, can you > provide us more information on how your KERNCONF differs from > GENERIC ? > I am pretty sure you have removed all the debug OPTIONs from the > kernel, isn't it ? It's a GENERIC kernel conf with polling and SMP activated. With FreeBSD 7.x i've removed witness and invariant. > > 2- Did you get a chance to try Jumbo Frames and evaluate jumbo > mbufs (I never success to make them work for me, did someone had > more chance ?). > In any cases, PPS values are important for such tests. Andre is > right, with Fast Forward you get the best perfs for such test. No i havent done any jumbo frame test, maybe on next try. Yes fastforward is better but my goal was to stress the IP stack so i've not integrated the fastforward result in the pdf. (but you can find some results in my reply to Andre). > > 3- Did you monitor the DUT to see where CPU cycles were spend in > each test ? Not during the real test. Profiling using hwpmc and LOCK_PROFILING have been done under the same condition but ignoring results. hwpmc use the callgraph patch published by Joseph Koshy. > > 4- Have you considered measuring the time it takes for an interrupt > to be handled and processed by the kernel Bottom/Top Half and ISR ? > [1] No > > 5- When I have performed some test using a Spirent SmartBits > (SmartFlow) last summer I got the following results [2]. (For > comparison purposes) It's really difficult to compare. For all my test i'm always using a reference hardware (not too powerful to be in the range of test tools). > > 6- In the test with Spirent Avalanche, you are using Lighttpd as > webserver, did you enable kqueue ? how many workers ? > You are using HTTP 1.0 wo Keep-Alive, what was your > net.inet.tcp.msl MIB's value ? The goal of the application test was simple: i've pollng that works better than interrupt in all forwarding case but is my socket application will works better ? For that i've just installed the port with the default config (log disabled, default is one worker). The result on this test show that polling is a great benefit to network application vs interrupt (near than two times more connections per seconds). > > 7- Polling is known to introduce higher latency, I would expect its > benefits to be less in 7-CURRENT compared to 6.x since (Scott ?) > a FAST INTR Handler has been introduced around a year ago. Yes it cost more in term of packet latency (where FreeBSD 4.11 was better than 6.x / 7.x in all mode) but under high pps with interrupt the DUT is unresponsive (ithread, filter, em fastintr). > Nonetheless, what you report sounds like a perf regression...Have > you filled a PR ? Luigi might have a good explaination here. :-D For us polling have always worked better than interrupt under FreeBSD 4.x, under FreeBSD 6.x it is not the case and under one of my application benchmark you can see that it really have a problem to sustain the load. Behind the new model there is more than a regression fix: I think it will scale better to SMP and provide a good acceleration for packet inspection. > > 8- Lock profiling information were obtanied through KTR ? no: LOCK_PROFILING > > 9- I was wondering if you have explored Intr CPU affinity [3] and > IRQ Swizzling [4] ? No, but one idea is the hybrid mode used under Solaris (intr when load is low and polling). > > Thanks for your efforts and your valuable contribution, best regards, > /Olivier Kind regards, Fabien > > [1] http://xview.net/papers/interrupt_latency_scheduler/ (WARNING: > Never find the time to finish that doc and publish it) && > http://xview.net/research/RTBench/ > > [2] http://xview.net/papers/jumbo_mbufs_freebsd/ > > [3] http://www.intel.com/cd/ids/developer/asmo-na/eng/188935.htm? > prn=3DY > > [4] http://download.intel.com/design/chipsets/applnots/31433702.pdf > -- > Olivier Warin > > > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6DED4888-AE5E-4BF6-85CF-6968ED7BEBF4>