Date: Fri, 3 Apr 2015 01:10:29 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Tobias Oberstein <tobias.oberstein@gmail.com> Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, Michael Fuckner <michael@fuckner.net>, Jim Harris <jim.harris@gmail.com>, Alan Somers <asomers@freebsd.org> Subject: Re: NVMe performance 4x slower than expected Message-ID: <20150402221029.GE2379@kib.kiev.ua> In-Reply-To: <551D4E5F.9090400@gmail.com> References: <551BC57D.5070101@gmail.com> <CAOtMX2jVwMHSnQfphAF%2Ba2%2Bo7eLp62nHmUo4t%2BEahrXLWReaFQ@mail.gmail.com> <CAJP=Hc-RNVuhPePg7bnpmT4ByzyXs_CNvAs7Oy7ntXjqhZYhCQ@mail.gmail.com> <551C5A82.2090306@gmail.com> <20150401212303.GB2379@kib.kiev.ua> <CAJP=Hc87FMYCrQYGfAtefQ8PLT3WtnvPfPSppp3zRF-0noQR9Q@mail.gmail.com> <551C6B62.7080205@gmail.com> <CAJP=Hc_6BFpoWqkSRyZaxsN1Zn=-D14CXOQjMb4zjnZRKhMb-g@mail.gmail.com> <551D4E5F.9090400@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Apr 02, 2015 at 04:12:47PM +0200, Tobias Oberstein wrote: > > You can also try a debug tunable that is in the nvme driver. > > > > hw.nvme.per_cpu_io_queues=0 > > I have rerun tests with kernel that has INVARIANTS off, and above sysctl > in loader.conf. > > Results are the same. > > vmstat now: > > root@s4l-zfs:~/oberstet # vmstat -ia | grep nvme > irq371: nvme0 8 0 > irq372: nvme0 7478 0 > irq373: nvme1 8 0 > irq374: nvme1 7612 0 > irq375: nvme2 8 0 > irq376: nvme2 7695 0 > irq377: nvme3 7 0 > irq378: nvme3 7716 0 > irq379: nvme4 8 0 > irq380: nvme4 7622 0 > irq381: nvme5 7 0 > irq382: nvme5 7561 0 > irq383: nvme6 8 0 > irq384: nvme6 7609 0 > irq385: nvme7 7 0 > irq386: nvme7 15373417 1174 > > === > > I was advised (off list) to run tests against a pure ramdisk. > > Here are results from a single socket E3: > > https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#xeon-e3-machine > > and here are results for the 48 core box > > https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#48-core-big-machine > > Performance with this box is 1/10 on this test compared to single socket E3! > > Something is severely wrong. It seems, there might be multiple issues > (not only NVMe). And this is after already running with 3 patches to > make it even boot. The speed of dd already looks wrong. Check the CPU frequency settings in BIOS, and check what sysctl dev.cpu reports. Ensure that cpufreq.ko is loaded from loader.conf. > > Well. I'm running out of ideas to try, and also patience with the users > waiting for this box =( > > /Tobias
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20150402221029.GE2379>