Date: Thu, 02 Apr 2015 00:04:18 +0200 From: Tobias Oberstein <tobias.oberstein@gmail.com> To: Jim Harris <jim.harris@gmail.com>, Konstantin Belousov <kostikbel@gmail.com> Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, Michael Fuckner <michael@fuckner.net>, Alan Somers <asomers@freebsd.org> Subject: Re: NVMe performance 4x slower than expected Message-ID: <551C6B62.7080205@gmail.com> In-Reply-To: <CAJP=Hc87FMYCrQYGfAtefQ8PLT3WtnvPfPSppp3zRF-0noQR9Q@mail.gmail.com> References: <551BC57D.5070101@gmail.com> <CAOtMX2jVwMHSnQfphAF%2Ba2%2Bo7eLp62nHmUo4t%2BEahrXLWReaFQ@mail.gmail.com> <CAJP=Hc-RNVuhPePg7bnpmT4ByzyXs_CNvAs7Oy7ntXjqhZYhCQ@mail.gmail.com> <551C5A82.2090306@gmail.com> <20150401212303.GB2379@kib.kiev.ua> <CAJP=Hc87FMYCrQYGfAtefQ8PLT3WtnvPfPSppp3zRF-0noQR9Q@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> Is this vmstat after the test ? No, it wasn't (I ran vmstat hours after the test). Here is right after test (shortened test duration, otherwise exactly the same FIO config): https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_vmstat.md#nvd7 > Somewhat funny is that nvme does not use MSI(X). > > > Yes - this is exactly the problem. > > nvme does use MSI-X if it can allocate the vectors (one per core). With > 48 cores, > I suspect we are quickly running out of vectors, so NVMe is reverting to > INTx. > > Could you actually send vmstat -ia (I left off the 'a' previously) - > just so we can > see all allocated interrupt vectors. > > As an experiment, can you try disabling hyperthreading - this will > reduce the The CPUs in this box root@s4l-zfs:~/src/sys/amd64/conf # sysctl hw.model hw.model: Intel(R) Xeon(R) CPU E7-8857 v2 @ 3.00GHz don't have hyperthreading (we deliberately selected CPU model for max. clock rather than HT) http://ark.intel.com/products/75254/Intel-Xeon-Processor-E7-8857-v2-30M-Cache-3_00-GHz > number of cores and should let you get MSI-X vectors allocated for at least > the first couple of NVMe controllers. Then please re-run your performance > test on one of those controllers. > You mean I should run against nvdN where N is a controller that still got MSI-X while other controllers did not? How would I find out which controller N? I don't know which nvdN is mounted in a PCIe slot directly assigned to which CPU socket, and I don't know which one's still got MSI-X and which not. I could arrange for disabling all but 1 CPU and retest. Would that help? === Right after running against nvd7 irq56: nvme0 6440 0 ... irq106: nvme7 145056 3 Then, immediately thereafter, running against nvd0 https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_vmstat.md#nvd0 irq56: nvme0 9233 0 ... irq106: nvme7 145056 3 === Earlier this day, I ran multiple longer tests .. all against nvd7. So if these are cumulative numbers since last boot, that would make sense.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?551C6B62.7080205>