Date: Thu, 02 Apr 2015 16:12:47 +0200 From: Tobias Oberstein <tobias.oberstein@gmail.com> To: Jim Harris <jim.harris@gmail.com> Cc: Konstantin Belousov <kostikbel@gmail.com>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, Michael Fuckner <michael@fuckner.net>, Alan Somers <asomers@freebsd.org> Subject: Re: NVMe performance 4x slower than expected Message-ID: <551D4E5F.9090400@gmail.com> In-Reply-To: <CAJP=Hc_6BFpoWqkSRyZaxsN1Zn=-D14CXOQjMb4zjnZRKhMb-g@mail.gmail.com> References: <551BC57D.5070101@gmail.com> <CAOtMX2jVwMHSnQfphAF%2Ba2%2Bo7eLp62nHmUo4t%2BEahrXLWReaFQ@mail.gmail.com> <CAJP=Hc-RNVuhPePg7bnpmT4ByzyXs_CNvAs7Oy7ntXjqhZYhCQ@mail.gmail.com> <551C5A82.2090306@gmail.com> <20150401212303.GB2379@kib.kiev.ua> <CAJP=Hc87FMYCrQYGfAtefQ8PLT3WtnvPfPSppp3zRF-0noQR9Q@mail.gmail.com> <551C6B62.7080205@gmail.com> <CAJP=Hc_6BFpoWqkSRyZaxsN1Zn=-D14CXOQjMb4zjnZRKhMb-g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> You can also try a debug tunable that is in the nvme driver. > > hw.nvme.per_cpu_io_queues=0 I have rerun tests with kernel that has INVARIANTS off, and above sysctl in loader.conf. Results are the same. vmstat now: root@s4l-zfs:~/oberstet # vmstat -ia | grep nvme irq371: nvme0 8 0 irq372: nvme0 7478 0 irq373: nvme1 8 0 irq374: nvme1 7612 0 irq375: nvme2 8 0 irq376: nvme2 7695 0 irq377: nvme3 7 0 irq378: nvme3 7716 0 irq379: nvme4 8 0 irq380: nvme4 7622 0 irq381: nvme5 7 0 irq382: nvme5 7561 0 irq383: nvme6 8 0 irq384: nvme6 7609 0 irq385: nvme7 7 0 irq386: nvme7 15373417 1174 === I was advised (off list) to run tests against a pure ramdisk. Here are results from a single socket E3: https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#xeon-e3-machine and here are results for the 48 core box https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#48-core-big-machine Performance with this box is 1/10 on this test compared to single socket E3! Something is severely wrong. It seems, there might be multiple issues (not only NVMe). And this is after already running with 3 patches to make it even boot. Well. I'm running out of ideas to try, and also patience with the users waiting for this box =( /Tobias
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?551D4E5F.9090400>