Date: Tue, 20 Oct 2015 11:03:14 +0000 From: "Eggert, Lars" <lars@netapp.com> To: Ian Smith <smithi@nimnet.asn.au> Cc: Kevin Oberman <rkoberman@gmail.com>, Luigi Rizzo <rizzo@iet.unipi.it>, "jfv@FreeBSD.org" <jfv@freebsd.org>, "ricera10@gmail.com" <ricera10@gmail.com>, Giuseppe Lettieri <g.lettieri@iet.unipi.it>, "net@freebsd.org" <net@freebsd.org> Subject: Re: ixl 40G bad performance? Message-ID: <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> In-Reply-To: <20151020190541.B15983@sola.nimnet.asn.au> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <CA%2BhQ2%2Bj5PkmRWe-ydahExfvOj%2BzestFEn%2B5DZ9_acG2Z_v=8_w@mail.gmail.com> <C81B629B-67D5-4E5B-A215-452E837BE673@netapp.com> <CA%2BhQ2%2BiGk5Bx5mZ8v7K9cPdiNcJgBz99kmP8=twFnETw-bi2vw@mail.gmail.com> <CAN6yY1uZJ5MhMdXLGy7REk8i6Z%2BjgJwMXFSu899%2BQwVaz%2BvP4w@mail.gmail.com> <20151020190541.B15983@sola.nimnet.asn.au>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_D6D2EEA7-5DE1-488B-A132-48E5E1E99B3B
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii
Hi,
On 2015-10-20, at 10:24, Ian Smith <smithi@nimnet.asn.au> wrote:
> Actually, you want to set hw.acpi.cpu.cx_lowest=3DC1 instead.
Done.
On 2015-10-19, at 17:55, Luigi Rizzo <rizzo@iet.unipi.it> wrote:
> On Mon, Oct 19, 2015 at 8:34 AM, Eggert, Lars <lars@netapp.com> wrote:
>> The only other sysctls in ixl(4) that look relevant are:
>>=20
>> hw.ixl.rx_itr
>> The RX interrupt rate value, set to 8K by default.
>>=20
>> hw.ixl.tx_itr
>> The TX interrupt rate value, set to 4K by default.
>>=20
>=20
> yes those. raise to 20-50k and see what you get in
> terms of ping latency.
While ixl(4) talks about 8K and 4K, the defaults actually seem to be:
hw.ixl.tx_itr: 122
hw.ixl.rx_itr: 62
Doubling those values *increases* flood ping latency to ~200 usec (from =
~116 usec).
Halving them to 62/31 decreases flood ping latency to ~50 usec, but =
still doesn't increase iperf throughput (still 2.8 Gb/s). Going to 31/16 =
further drops latency to 24 usec, with no change in throughput.
(Looking at the "interrupt Moderation parameters" #defines in =
sys/dev/ixl/ixl.h it seems that ixl likes to have its irq rates =
specified with some weird divider scheme.)
With 5/5 (which corresponds to IXL_ITR_100K), I get down to 16 usec. =
Unfortunately, throughput is then also down to about 2 Gb/s.
One thing I noticed in top is that one queue irq is using quite a bit of =
CPU when I run iperf:
11 0 -92 - 0K 1152K CPU2 2 0:19 50.98% =
intr{irq293: ixl1:q2}
11 0 -92 - 0K 1152K WAIT 3 0:02 5.18% =
intr{irq294: ixl1:q3}
0 0 -92 0 0K 8944K - 25 0:01 1.07% =
kernel{ixl1 que}
11 0 -92 - 0K 1152K WAIT 1 0:01 0.00% =
intr{irq292: ixl1:q1}
11 0 -92 - 0K 1152K WAIT 0 0:00 0.00% =
intr{irq291: ixl1:q0}
0 0 -92 0 0K 8944K - 22 0:00 0.00% =
kernel{ixl1 adminq}
0 0 -92 0 0K 8944K - 31 0:00 0.00% =
kernel{ixl1 que}
0 0 -92 0 0K 8944K - 31 0:00 0.00% =
kernel{ixl1 que}
0 0 -92 0 0K 8944K - 31 0:00 0.00% =
kernel{ixl1 que}
11 0 -92 - 0K 1152K WAIT -1 0:00 0.00% =
intr{irq290: ixl1:aq}
With 10G ix interfaces and a throughput of ~9Gb/s, the CPU load is much =
lower:
11 0 -92 - 0K 1152K WAIT 0 0:05 7.67% =
intr{irq274: ix0:que }
0 0 -92 0 0K 8944K - 27 0:00 0.29% =
kernel{ix0 que}
0 0 -92 0 0K 8944K - 10 0:00 0.00% =
kernel{ix0 linkq}
11 0 -92 - 0K 1152K WAIT 1 0:00 0.00% =
intr{irq275: ix0:que }
11 0 -92 - 0K 1152K WAIT 3 0:00 0.00% =
intr{irq277: ix0:que }
11 0 -92 - 0K 1152K WAIT 2 0:00 0.00% =
intr{irq276: ix0:que }
11 0 -92 - 0K 1152K WAIT 18 0:00 0.00% =
intr{irq278: ix0:link}
0 0 -92 0 0K 8944K - 0 0:00 0.00% =
kernel{ix0 que}
0 0 -92 0 0K 8944K - 0 0:00 0.00% =
kernel{ix0 que}
0 0 -92 0 0K 8944K - 0 0:00 0.00% =
kernel{ix0 que}
Lars
--Apple-Mail=_D6D2EEA7-5DE1-488B-A132-48E5E1E99B3B
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="signature.asc"
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Message signed with OpenPGP using GPGMail
-----BEGIN PGP SIGNATURE-----
iQCVAwUBViYfcdZcnpRveo1xAQiS1gP/bGeKFsEN+BVIzUbtCDdGkoDsDL7eZIuZ
Qfdw8J63Azn4DfnPvgKlH330ESm45UaSYvojSQev16rO6BfU53Sa0K+7nzaSyqzf
kkyiOseAXHTTsAOnL/IA4djPcdYLSD2qGRyx1LpeNc3uMaVshcxlzeCgFmSLyJa4
17FAveCAMk4=
=1HCx
-----END PGP SIGNATURE-----
--Apple-Mail=_D6D2EEA7-5DE1-488B-A132-48E5E1E99B3B--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6CD6754D-FC0E-4B24-AAEC-7C9D68284141>