From owner-freebsd-hackers@FreeBSD.ORG  Thu Apr  2 14:44:16 2015
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id BEB6DCE
 for <freebsd-hackers@freebsd.org>; Thu,  2 Apr 2015 14:44:16 +0000 (UTC)
Received: from hydra.pix.net (hydra.pix.net [IPv6:2001:470:e254::4])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client CN "mail.pix.net", Issuer "Pix.Com Technologies,
 LLC CA" (not verified))
 by mx1.freebsd.org (Postfix) with ESMTPS id 65164E8B
 for <freebsd-hackers@freebsd.org>; Thu,  2 Apr 2015 14:44:16 +0000 (UTC)
Received: from torb.pix.net (verizon.pix.net [71.178.232.3])
 (authenticated bits=0)
 by hydra.pix.net (8.15.1/8.15.1) with ESMTPA id t32EiEQ4087995;
 Thu, 2 Apr 2015 10:44:14 -0400 (EDT) (envelope-from lidl@pix.net)
X-Authentication-Warning: hydra.pix.net: Host verizon.pix.net [71.178.232.3]
 claimed to be torb.pix.net
Message-ID: <551D55BE.3020904@pix.net>
Date: Thu, 02 Apr 2015 10:44:14 -0400
From: Kurt Lidl <lidl@pix.net>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10;
 rv:31.0) Gecko/20100101 Thunderbird/31.5.0
MIME-Version: 1.0
To: freebsd-hackers@freebsd.org
Subject: Re: NVMe performance 4x slower than expected
References: <551BC57D.5070101@gmail.com>	<CAOtMX2jVwMHSnQfphAF+a2+o7eLp62nHmUo4t+EahrXLWReaFQ@mail.gmail.com>	<CAJP=Hc-RNVuhPePg7bnpmT4ByzyXs_CNvAs7Oy7ntXjqhZYhCQ@mail.gmail.com>	<551C5A82.2090306@gmail.com>	<20150401212303.GB2379@kib.kiev.ua>	<CAJP=Hc87FMYCrQYGfAtefQ8PLT3WtnvPfPSppp3zRF-0noQR9Q@mail.gmail.com>	<551C6B62.7080205@gmail.com>
 <CAJP=Hc_6BFpoWqkSRyZaxsN1Zn=-D14CXOQjMb4zjnZRKhMb-g@mail.gmail.com>
 <551D4E5F.9090400@gmail.com>
In-Reply-To: <551D4E5F.9090400@gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
 <freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-hackers>, 
 <mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers/>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
 <mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 02 Apr 2015 14:44:16 -0000

On 4/2/15 10:12 AM, Tobias Oberstein wrote:
> I was advised (off list) to run tests against a pure ramdisk.
>
> Here are results from a single socket E3:
>
> https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#xeon-e3-machine
>
>
> and here are results for the 48 core box
>
> https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#48-core-big-machine
>
>
> Performance with this box is 1/10 on this test compared to single socket
> E3!
>
> Something is severely wrong. It seems, there might be multiple issues
> (not only NVMe). And this is after already running with 3 patches to
> make it even boot.

Offhand, I'd guess the performance difference between the single-socket
machine and the quad-socket machine has to do with the NUMA effects of
the memory in the multi-socket system.

FreeBSD does not have per-socket memory allocation/affliation at this
time.  So, some of the memory allocated to your ramdisk might be
accessible to your process only over the QPI interconnect between the
different CPU sockets.

You could install the latest and greatest intel-pcm tools from
/usr/ports and see what that says about the memory while you are
running your randomio/fio tests.

-Kurt