Date: Tue, 15 Sep 2015 17:29:22 +0200 From: Tobias Oberstein <tobias.oberstein@gmail.com> To: CadSoftware <structuralcadsoftware@gmail.com>, freebsd-hackers@freebsd.org Subject: Re: NVMe performance 4x slower than expected Message-ID: <55F83952.3070407@gmail.com> In-Reply-To: <1442330273152-6039235.post@n5.nabble.com> References: <CAOtMX2jVwMHSnQfphAF%2Ba2%2Bo7eLp62nHmUo4t%2BEahrXLWReaFQ@mail.gmail.com> <CAJP=Hc-RNVuhPePg7bnpmT4ByzyXs_CNvAs7Oy7ntXjqhZYhCQ@mail.gmail.com> <551C5A82.2090306@gmail.com> <20150401212303.GB2379@kib.kiev.ua> <CAJP=Hc87FMYCrQYGfAtefQ8PLT3WtnvPfPSppp3zRF-0noQR9Q@mail.gmail.com> <CAJP=Hc-WLKe3%2BDQ=2o21CY=aaQAjADrzEfnD7NVO1Cotu4vcGg@mail.gmail.com> <5526EA33.6090004@gmail.com> <CAJ-VmonecBDemkfS=3nV2jiuJfOFJg7bZOacxOKXvTWktxBd9A@mail.gmail.com> <5527F554.2030806@gmail.com> <CAJP=Hc8RTmFK5DL%2BToiRbcSMQbBtTKVkGMtp2FJWbz=AY%2BQQBw@mail.gmail.com> <1442330273152-6039235.post@n5.nabble.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> I believe FreeBSD 10.2-RELEASE includes an updated nvme (4) driver. Do these > changes fix the performance issues described in this thread? I don't know. The problems with the machine we've deployed however run deeper than "just NVMe": RAM: 3TB .. the FBSD kernel patches that fixed this might be in 10.2 .. dunno amount of PCIx resources: the box has 8x NVMe, 2x SAS controller, 2x dual-port 10GbE .. FBSD freaked out due to this .. there were kernel patches also .. again no clue if those are in mainline now And finally, the bummer: NUMA performance. The box has 4 sockets, 48 cores, and we are moving to 64 cores. My unscientific impression was, that FBSD isn't yet able to cope with that. Anyway, in the meantime, we've moved from SLES 12 to Ubuntu 15.04 ( Linux kernel 3.19.3) .. due to recent Linux developments: blkmq and such. 8kB random write performance now is now at >1 million IOPS (over a test duration of 10h and a 1TB dataset at block device level - RAID-0 over 8x 2TB NVMe). That's nearly 40% more than on SLES 12, and the CPU load is lower also. So we can saturate the hardware IOPS wise. Cheers, /Tobias
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55F83952.3070407>