Date: Tue, 20 Oct 2015 11:03:14 +0000 From: "Eggert, Lars" <lars@netapp.com> To: Ian Smith <smithi@nimnet.asn.au> Cc: Kevin Oberman <rkoberman@gmail.com>, Luigi Rizzo <rizzo@iet.unipi.it>, "jfv@FreeBSD.org" <jfv@freebsd.org>, "ricera10@gmail.com" <ricera10@gmail.com>, Giuseppe Lettieri <g.lettieri@iet.unipi.it>, "net@freebsd.org" <net@freebsd.org> Subject: Re: ixl 40G bad performance? Message-ID: <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> In-Reply-To: <20151020190541.B15983@sola.nimnet.asn.au> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <CA%2BhQ2%2Bj5PkmRWe-ydahExfvOj%2BzestFEn%2B5DZ9_acG2Z_v=8_w@mail.gmail.com> <C81B629B-67D5-4E5B-A215-452E837BE673@netapp.com> <CA%2BhQ2%2BiGk5Bx5mZ8v7K9cPdiNcJgBz99kmP8=twFnETw-bi2vw@mail.gmail.com> <CAN6yY1uZJ5MhMdXLGy7REk8i6Z%2BjgJwMXFSu899%2BQwVaz%2BvP4w@mail.gmail.com> <20151020190541.B15983@sola.nimnet.asn.au>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
Hi,
On 2015-10-20, at 10:24, Ian Smith <smithi@nimnet.asn.au> wrote:
> Actually, you want to set hw.acpi.cpu.cx_lowest=C1 instead.
Done.
On 2015-10-19, at 17:55, Luigi Rizzo <rizzo@iet.unipi.it> wrote:
> On Mon, Oct 19, 2015 at 8:34 AM, Eggert, Lars <lars@netapp.com> wrote:
>> The only other sysctls in ixl(4) that look relevant are:
>>
>> hw.ixl.rx_itr
>> The RX interrupt rate value, set to 8K by default.
>>
>> hw.ixl.tx_itr
>> The TX interrupt rate value, set to 4K by default.
>>
>
> yes those. raise to 20-50k and see what you get in
> terms of ping latency.
While ixl(4) talks about 8K and 4K, the defaults actually seem to be:
hw.ixl.tx_itr: 122
hw.ixl.rx_itr: 62
Doubling those values *increases* flood ping latency to ~200 usec (from ~116 usec).
Halving them to 62/31 decreases flood ping latency to ~50 usec, but still doesn't increase iperf throughput (still 2.8 Gb/s). Going to 31/16 further drops latency to 24 usec, with no change in throughput.
(Looking at the "interrupt Moderation parameters" #defines in sys/dev/ixl/ixl.h it seems that ixl likes to have its irq rates specified with some weird divider scheme.)
With 5/5 (which corresponds to IXL_ITR_100K), I get down to 16 usec. Unfortunately, throughput is then also down to about 2 Gb/s.
One thing I noticed in top is that one queue irq is using quite a bit of CPU when I run iperf:
11 0 -92 - 0K 1152K CPU2 2 0:19 50.98% intr{irq293: ixl1:q2}
11 0 -92 - 0K 1152K WAIT 3 0:02 5.18% intr{irq294: ixl1:q3}
0 0 -92 0 0K 8944K - 25 0:01 1.07% kernel{ixl1 que}
11 0 -92 - 0K 1152K WAIT 1 0:01 0.00% intr{irq292: ixl1:q1}
11 0 -92 - 0K 1152K WAIT 0 0:00 0.00% intr{irq291: ixl1:q0}
0 0 -92 0 0K 8944K - 22 0:00 0.00% kernel{ixl1 adminq}
0 0 -92 0 0K 8944K - 31 0:00 0.00% kernel{ixl1 que}
0 0 -92 0 0K 8944K - 31 0:00 0.00% kernel{ixl1 que}
0 0 -92 0 0K 8944K - 31 0:00 0.00% kernel{ixl1 que}
11 0 -92 - 0K 1152K WAIT -1 0:00 0.00% intr{irq290: ixl1:aq}
With 10G ix interfaces and a throughput of ~9Gb/s, the CPU load is much lower:
11 0 -92 - 0K 1152K WAIT 0 0:05 7.67% intr{irq274: ix0:que }
0 0 -92 0 0K 8944K - 27 0:00 0.29% kernel{ix0 que}
0 0 -92 0 0K 8944K - 10 0:00 0.00% kernel{ix0 linkq}
11 0 -92 - 0K 1152K WAIT 1 0:00 0.00% intr{irq275: ix0:que }
11 0 -92 - 0K 1152K WAIT 3 0:00 0.00% intr{irq277: ix0:que }
11 0 -92 - 0K 1152K WAIT 2 0:00 0.00% intr{irq276: ix0:que }
11 0 -92 - 0K 1152K WAIT 18 0:00 0.00% intr{irq278: ix0:link}
0 0 -92 0 0K 8944K - 0 0:00 0.00% kernel{ix0 que}
0 0 -92 0 0K 8944K - 0 0:00 0.00% kernel{ix0 que}
0 0 -92 0 0K 8944K - 0 0:00 0.00% kernel{ix0 que}
Lars
[-- Attachment #2 --]
-----BEGIN PGP SIGNATURE-----
iQCVAwUBViYfcdZcnpRveo1xAQiS1gP/bGeKFsEN+BVIzUbtCDdGkoDsDL7eZIuZ
Qfdw8J63Azn4DfnPvgKlH330ESm45UaSYvojSQev16rO6BfU53Sa0K+7nzaSyqzf
kkyiOseAXHTTsAOnL/IA4djPcdYLSD2qGRyx1LpeNc3uMaVshcxlzeCgFmSLyJa4
17FAveCAMk4=
=1HCx
-----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6CD6754D-FC0E-4B24-AAEC-7C9D68284141>