Date: Thu, 02 Apr 2015 10:44:14 -0400 From: Kurt Lidl <lidl@pix.net> To: freebsd-hackers@freebsd.org Subject: Re: NVMe performance 4x slower than expected Message-ID: <551D55BE.3020904@pix.net> In-Reply-To: <551D4E5F.9090400@gmail.com> References: <551BC57D.5070101@gmail.com> <CAOtMX2jVwMHSnQfphAF%2Ba2%2Bo7eLp62nHmUo4t%2BEahrXLWReaFQ@mail.gmail.com> <CAJP=Hc-RNVuhPePg7bnpmT4ByzyXs_CNvAs7Oy7ntXjqhZYhCQ@mail.gmail.com> <551C5A82.2090306@gmail.com> <20150401212303.GB2379@kib.kiev.ua> <CAJP=Hc87FMYCrQYGfAtefQ8PLT3WtnvPfPSppp3zRF-0noQR9Q@mail.gmail.com> <551C6B62.7080205@gmail.com> <CAJP=Hc_6BFpoWqkSRyZaxsN1Zn=-D14CXOQjMb4zjnZRKhMb-g@mail.gmail.com> <551D4E5F.9090400@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 4/2/15 10:12 AM, Tobias Oberstein wrote: > I was advised (off list) to run tests against a pure ramdisk. > > Here are results from a single socket E3: > > https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#xeon-e3-machine > > > and here are results for the 48 core box > > https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#48-core-big-machine > > > Performance with this box is 1/10 on this test compared to single socket > E3! > > Something is severely wrong. It seems, there might be multiple issues > (not only NVMe). And this is after already running with 3 patches to > make it even boot. Offhand, I'd guess the performance difference between the single-socket machine and the quad-socket machine has to do with the NUMA effects of the memory in the multi-socket system. FreeBSD does not have per-socket memory allocation/affliation at this time. So, some of the memory allocated to your ramdisk might be accessible to your process only over the QPI interconnect between the different CPU sockets. You could install the latest and greatest intel-pcm tools from /usr/ports and see what that says about the memory while you are running your randomio/fio tests. -Kurt
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?551D55BE.3020904>