From owner-freebsd-hackers@FreeBSD.ORG  Sat Apr  4 23:48:46 2015
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 11B40A70;
 Sat,  4 Apr 2015 23:48:46 +0000 (UTC)
Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1])
 (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 717D5DA2;
 Sat,  4 Apr 2015 23:48:45 +0000 (UTC)
Received: from tom.home (kostik@localhost [127.0.0.1])
 by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t34NmdYk083083
 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO);
 Sun, 5 Apr 2015 02:48:39 +0300 (EEST)
 (envelope-from kostikbel@gmail.com)
DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t34NmdYk083083
Received: (from kostik@localhost)
 by tom.home (8.14.9/8.14.9/Submit) id t34NmdkV083082;
 Sun, 5 Apr 2015 02:48:39 +0300 (EEST)
 (envelope-from kostikbel@gmail.com)
X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com
 using -f
Date: Sun, 5 Apr 2015 02:48:39 +0300
From: Konstantin Belousov <kostikbel@gmail.com>
To: Tobias Oberstein <tobias.oberstein@gmail.com>
Subject: Re: NVMe performance 4x slower than expected
Message-ID: <20150404234839.GM2379@kib.kiev.ua>
References: <CAOtMX2jVwMHSnQfphAF+a2+o7eLp62nHmUo4t+EahrXLWReaFQ@mail.gmail.com>
 <CAJP=Hc-RNVuhPePg7bnpmT4ByzyXs_CNvAs7Oy7ntXjqhZYhCQ@mail.gmail.com>
 <551C5A82.2090306@gmail.com> <20150401212303.GB2379@kib.kiev.ua>
 <CAJP=Hc87FMYCrQYGfAtefQ8PLT3WtnvPfPSppp3zRF-0noQR9Q@mail.gmail.com>
 <551C6B62.7080205@gmail.com>
 <CAJP=Hc_6BFpoWqkSRyZaxsN1Zn=-D14CXOQjMb4zjnZRKhMb-g@mail.gmail.com>
 <551D4E5F.9090400@gmail.com> <20150402221029.GE2379@kib.kiev.ua>
 <551FC7B3.80203@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <551FC7B3.80203@gmail.com>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00,
 DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no
 autolearn_force=no version=3.4.0
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home
Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>,
 Michael Fuckner <michael@fuckner.net>, Jim Harris <jim.harris@gmail.com>,
 Alan Somers <asomers@freebsd.org>
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
 <freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-hackers>, 
 <mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers/>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
 <mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 04 Apr 2015 23:48:46 -0000

On Sat, Apr 04, 2015 at 01:14:59PM +0200, Tobias Oberstein wrote:
> >> I was advised (off list) to run tests against a pure ramdisk.
> >>
> >> Here are results from a single socket E3:
> >>
> >> https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#xeon-e3-machine
> >>
> >> and here are results for the 48 core box
> >>
> >> https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#48-core-big-machine
> >>
> >> Performance with this box is 1/10 on this test compared to single socket E3!
> >>
> >> Something is severely wrong. It seems, there might be multiple issues
> >> (not only NVMe). And this is after already running with 3 patches to
> >> make it even boot.
> > The speed of dd already looks wrong.
> 
> Yes.
Try some simple checks for the performance anomalies.  E.g., verify the
raw memory bandwidth, check that it is in the expected range.  Quick search
popped up the following tool, e.g.:
https://zsmith.co/bandwidth.html

> 
> >
> > Check the CPU frequency settings in BIOS, and check what sysctl dev.cpu
> > reports.  Ensure that cpufreq.ko is loaded from loader.conf.
> 
> It's loaded now, and CPU clock is at maximum: dev.cpu.0.freq: 3000
> 
> Unfortuanetly, performance (randomio/fio) did not change ..