Date: Thu, 16 Mar 2017 15:50:42 -0400 From: John Jasen <jjasen@gmail.com> To: Navdeep Parhar <nparhar@gmail.com> Cc: Slawa Olhovchenkov <slw@zxy.spb.ru>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, "Caraballo-vega, Jordan A." <jordancaraballo87@gmail.com> Subject: Re: bad throughput performance on multiple systems: Re: Fwd: Re: Disappointing packets-per-second performance results on a Dell,PE R530 Message-ID: <CAACLuR29xQhDWATRheBaOU2vtiYp61JgDKHaXum%2BU32MBDLBzw@mail.gmail.com> In-Reply-To: <CAPFoGT-BAMpj34wtB06dxMKk%2B87OEOs5-qu%2BRLVz=aPrhX6hDA@mail.gmail.com> References: <ebb04a3e-bcde-6d50-af63-348e8d06fcba@gmail.com> <40a413f3-2c44-ee9d-9961-67114d8dffca@gmail.com> <20170205175531.GA20287@dwarf> <7d349edd-0c81-2e3f-d3b9-27af232de76d@gmail.com> <20170209153409.GG41673@dwarf> <6ad029e0-86c6-af3d-8fc3-694d4bcdc683@gmail.com> <b460cc49-364a-34fb-e212-582730546dee@gmail.com> <20170312231826.GV15630@zxy.spb.ru> <74654520-b8b6-6118-2e46-902a8ea107ac@gmail.com> <CAPFoGT9k4HfDCQ7wJPDFMTrJTtDyc9uK_ma9ubneDhVSsS-jcA@mail.gmail.com> <173fffac-7ae2-786a-66c0-e9cd7ab78f44@gmail.com> <CAPFoGT-BAMpj34wtB06dxMKk%2B87OEOs5-qu%2BRLVz=aPrhX6hDA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
As a few points of note, partial resolution, and curiosity: Following down leads that 11-STABLE had tryforward improvements over 11-RELENG, I upgraded. The same tests (24 client streams over UDP with small packets), the system went from passing 1.7m pps to about 2.5m. Following indications from Navdeep Parhar that UDP queue hashing is not as efficient as it could be, we started running the tests with various powers of 2 streams (2,4,8,16,32) -- and were able to push the system up to 5m pps. We are currently seeing in the tests approximately 10-11m pps on the outside interface, around 5-6m dropped, and 5 million passed. On Mon, Mar 13, 2017 at 1:31 PM, Navdeep Parhar <nparhar@gmail.com> wrote: > On Mon, Mar 13, 2017 at 10:13 AM, John Jasen <jjasen@gmail.com> wrote: > > On 03/13/2017 01:03 PM, Navdeep Parhar wrote: > > > >> On Sun, Mar 12, 2017 at 5:35 PM, John Jasen <jjasen@gmail.com> wrote: > >>> UDP traffic. dmesg reports 16 txq, 8 rxq -- which is the default for > >>> Chelsio. > >>> > >> I don't recall offhand, but UDP might be using 2-tuple hashing by > >> default and that might affect the distribution of flows across queues. > >> Are there senders generating IP fragments by any chance (that'll > >> depend on the "send size" that your UDP application is using)? > > > > No, they're not fragmenting. > > > >> Have you tried limiting the adapter's rx ithreads to the CPU that the > >> PCIe slot with the adapter is wired to? > > > > Above and beyond the use of cpuset, you mean? > > I meant cpuset. > > If possible, try your experiments on a single socket system. > > Regards, > Navdeep >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAACLuR29xQhDWATRheBaOU2vtiYp61JgDKHaXum%2BU32MBDLBzw>