Date: Fri, 23 Oct 2015 21:36:10 +0000 From: Eric Joyner <erj@freebsd.org> To: Bruce Evans <brde@optusnet.com.au> Cc: "Eggert, Lars" <lars@netapp.com>, Ian Smith <smithi@nimnet.asn.au>, Kevin Oberman <rkoberman@gmail.com>, "net@freebsd.org" <net@freebsd.org>, "jfv@FreeBSD.org" <jfv@freebsd.org>, Luigi Rizzo <rizzo@iet.unipi.it>, Giuseppe Lettieri <g.lettieri@iet.unipi.it> Subject: Re: ixl 40G bad performance? Message-ID: <CA%2Bb0zg92iMfLfRo%2BZBiMBnpBS_WBG==h3a3NCSXQGyeD7gio8A@mail.gmail.com> In-Reply-To: <20151023162337.L1149@besplex.bde.org> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <CA%2BhQ2%2Bj5PkmRWe-ydahExfvOj%2BzestFEn%2B5DZ9_acG2Z_v=8_w@mail.gmail.com> <C81B629B-67D5-4E5B-A215-452E837BE673@netapp.com> <CA%2BhQ2%2BiGk5Bx5mZ8v7K9cPdiNcJgBz99kmP8=twFnETw-bi2vw@mail.gmail.com> <CAN6yY1uZJ5MhMdXLGy7REk8i6Z%2BjgJwMXFSu899%2BQwVaz%2BvP4w@mail.gmail.com> <20151020190541.B15983@sola.nimnet.asn.au> <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> <20151020232218.G1833@besplex.bde.org> <20151023162337.L1149@besplex.bde.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Bruce mostly has it right -- ITR is the minimum latency between interrupts. But, it does actually guarantee a minimum period between interrupts. Though, Fortville actually is unique a little bit in that there is another ITR setting that can ensure a certain average number of interrupts per second (called Interrupt Rate Limiting), though, but I don't think this is used in the current version of the driver. I see that the sysctl does clobber the global value, but have you tried lowering the interval / raising the rate? You could try something like 10usecs, and see if that helps. We'll do some more investigation here -- 3Gb/s on a 40Gb/s using default settings is terrible, and we shouldn't let that be happening. - Eric On Thu, Oct 22, 2015 at 10:36 PM Bruce Evans <brde@optusnet.com.au> wrote: > On Wed, 21 Oct 2015, Bruce Evans wrote: > > > Fix for em: > > > > X diff -u2 if_em.c~ if_em.c > > X --- if_em.c~ 2015-09-28 06:29:35.000000000 +0000 > > X +++ if_em.c 2015-10-18 18:49:36.876699000 +0000 > > X @@ -609,8 +609,8 @@ > > X em_tx_abs_int_delay_dflt); > > X em_add_int_delay_sysctl(adapter, "itr", > > X - "interrupt delay limit in usecs/4", > > X + "interrupt delay limit in usecs", > > X &adapter->tx_itr, > > X E1000_REGISTER(hw, E1000_ITR), > > X - DEFAULT_ITR); > > X + 1000000 / MAX_INTS_PER_SEC); > > X X /* Sysctl for limiting the amount of work done in the taskqueue */ > > > > "delay limit" is fairly good wording. Other parameters tend to give long > > delays, but itr limits the longest delay due to interrupt moderation to > > whatever the itr respresents. > > Everything in the last paragraph is backwards (inverted). Other > parameters tend to give short delays. They should be set to small > values to minimise latency. Then under load, itr limits the interrupt > _rate_ from above. The interrupt delay is the inverse of the interrupt > rate, so it is limited from below. So "delay limit" is fairly bad > wording. Normally, limits are from above, but the inversion makes > the itr limit from below. > > This is most easily understood by converting itr to a rate: itr = 125 > means a rate limit of 8000 Hz. It doesn't quite mean that the latency > is at least 125 usec. No one wants to ensure large latencies, and the > itr setting only ensures a minimal average latency them under load. > > Bruce >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2Bb0zg92iMfLfRo%2BZBiMBnpBS_WBG==h3a3NCSXQGyeD7gio8A>