Date: Tue, 11 Aug 2015 15:16:26 -0700 From: hiren panchasara <hiren@strugglingcoder.info> To: Adrian Chadd <adrian.chadd@gmail.com> Cc: Maxim Sobolev <sobomax@freebsd.org>, FreeBSD Net <freebsd-net@freebsd.org>, freebsd@intel.com, Jev Bj?rsell <jev@sippysoft.com> Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 Message-ID: <20150811221626.GE96509@strugglingcoder.info> In-Reply-To: <CAJ-Vmo=7XzE0SYfG__Y7qee9jZ1qKOOuNPY2TFPJfD2-06Mk5g@mail.gmail.com> References: <CAH7qZftMB34PM1CqNhdg7AWhsq6YknUDgc60ASfT2Z0L1z8XCQ@mail.gmail.com> <CAJ-Vmo=7XzE0SYfG__Y7qee9jZ1qKOOuNPY2TFPJfD2-06Mk5g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--k3qmt+ucFURmlhDS Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 08/11/15 at 03:01P, Adrian Chadd wrote: > hi, >=20 > Are you able to graph per-queue interrupt rates? >=20 > It looks like the traffic is distributed differently (the first two > queues are taking interrupts). Yeah, also check out "# sysctl dev.ix | grep packets" >=20 > Does 10.1 have the flow director code disabled? I remember there was > some .. interesting behaviour with ixgbe where it'd look at traffic > and set up flow director rules to try and "balance" things. It was > buggy and programmed the hardware badly, so we disabled it in at least > -HEAD. Looks like we don't build with IXGBE_FDIR by default on 10 so I assume it's off. There were some lagg/hashing related changes recently so let us know if that is hurting you. Cheers, Hiren >=20 >=20 >=20 > -adrian >=20 >=20 > On 11 August 2015 at 14:18, Maxim Sobolev <sobomax@freebsd.org> wrote: > > Hi folks, > > > > We've trying to migrate some of our high-PPS systems to a new hardware = that > > has four X540-AT2 10G NICs and observed that interrupt time goes through > > roof after we cross around 200K PPS in and 200K out (two ports in LACP). > > The previous hardware was stable up to about 350K PPS in and 350K out. I > > believe the old one was equipped with the I350 and had the identical LA= CP > > configuration. The new box also has better CPU with more cores (i.e. 24 > > cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. > > > > After hitting this limit with the default settings, I've tried to tweak= the > > following settings: > > > > hw.ix.rx_process_limit=3D"-1" > > hw.ix.tx_process_limit=3D"-1" > > hw.ix.enable_aim=3D"0" > > hw.ix.max_interrupt_rate=3D"-1" > > hw.ix.rxd=3D"4096" > > hw.ix.txd=3D"4096" > > > > dev.ix.0.fc=3D0 > > dev.ix.1.fc=3D0 > > dev.ix.2.fc=3D0 > > dev.ix.3.fc=3D0 > > > > hw.intr_storm_threshold=3D0 > > > > But there is little or no effect on the performance. The workload is ju= st > > lot of small UDP packets being relayed between bunch of hosts. The symp= toms > > are always the same - the box runs nice and cool until it his the said = PPS > > threshold, with kernel spending just few percent in the interrupts and = then > > it jumps straight to 100% interrupt time, thereby scaring some traffic = away > > due to packet loss and such, so that the load drops and the system goes > > into the "cool" state again. It looks very much like some contention in= the > > driver or in the hardware. Linked are some monitoring screenshots > > displaying the issue unfolding as well as systat -vm screenshots from t= he > > "cool" state. > > > > http://sobomax.sippysoft.com/ScreenShot387.png <- CPU utilization right > > before the "bang event" > > http://sobomax.sippysoft.com/ScreenShot382.png <- issue itself > > http://sobomax.sippysoft.com/ScreenShot385.png <- systat -vm few minutes > > after traffic declined somewhat > > > > We are now trying to get customer install 1Gig NIC so that we can run it > > and compare performance with the rest of the hardware and software being > > essentially the same. > > > > Any ideas on how to improve/resolve this problem are welcome. Thanks! > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" --k3qmt+ucFURmlhDS Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQF8BAEBCgBmBQJVynQ6XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/l3/gIAIJscBPgQ3wE0WFGWJ1kC4OW rj1drAXAW72tSX8quAfMcnlw99RVtVU7AUFgPHsPokrZpPEFXPHXAQobM5CJfm5O UWkjpY02N1q+90N1kxfQ1Sg8J9rP7xM8rx0T63tecnlpM51KTpOw9oGyHfCIJzUu F38Ir9PGseZT5Ouc+szpqZiqtpZ9vP1hduI8hdA/Aa9iC5heYmu5W/gI5+7xaLWb XroSa8emwcaOQGViJ0GHzncwXnnEM/RNNrW/BUZQKurYscmQhWt/Oxj9JUNREjE3 cbVs1nQ5KifDSFilUjxIlebXhYMUKEOyQj+SpRIS44+K3HDl3d119xJkLB34zkU= =iddr -----END PGP SIGNATURE----- --k3qmt+ucFURmlhDS--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20150811221626.GE96509>