From owner-freebsd-hackers@FreeBSD.ORG Thu Apr 2 22:10:40 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 56928A2C; Thu, 2 Apr 2015 22:10:40 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ED0D2DB5; Thu, 2 Apr 2015 22:10:39 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t32MATC3059242 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Fri, 3 Apr 2015 01:10:29 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t32MATC3059242 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t32MAT9l059241; Fri, 3 Apr 2015 01:10:29 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 3 Apr 2015 01:10:29 +0300 From: Konstantin Belousov To: Tobias Oberstein Subject: Re: NVMe performance 4x slower than expected Message-ID: <20150402221029.GE2379@kib.kiev.ua> References: <551BC57D.5070101@gmail.com> <551C5A82.2090306@gmail.com> <20150401212303.GB2379@kib.kiev.ua> <551C6B62.7080205@gmail.com> <551D4E5F.9090400@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <551D4E5F.9090400@gmail.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: "freebsd-hackers@freebsd.org" , Michael Fuckner , Jim Harris , Alan Somers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Apr 2015 22:10:40 -0000 On Thu, Apr 02, 2015 at 04:12:47PM +0200, Tobias Oberstein wrote: > > You can also try a debug tunable that is in the nvme driver. > > > > hw.nvme.per_cpu_io_queues=0 > > I have rerun tests with kernel that has INVARIANTS off, and above sysctl > in loader.conf. > > Results are the same. > > vmstat now: > > root@s4l-zfs:~/oberstet # vmstat -ia | grep nvme > irq371: nvme0 8 0 > irq372: nvme0 7478 0 > irq373: nvme1 8 0 > irq374: nvme1 7612 0 > irq375: nvme2 8 0 > irq376: nvme2 7695 0 > irq377: nvme3 7 0 > irq378: nvme3 7716 0 > irq379: nvme4 8 0 > irq380: nvme4 7622 0 > irq381: nvme5 7 0 > irq382: nvme5 7561 0 > irq383: nvme6 8 0 > irq384: nvme6 7609 0 > irq385: nvme7 7 0 > irq386: nvme7 15373417 1174 > > === > > I was advised (off list) to run tests against a pure ramdisk. > > Here are results from a single socket E3: > > https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#xeon-e3-machine > > and here are results for the 48 core box > > https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_ramdisk.md#48-core-big-machine > > Performance with this box is 1/10 on this test compared to single socket E3! > > Something is severely wrong. It seems, there might be multiple issues > (not only NVMe). And this is after already running with 3 patches to > make it even boot. The speed of dd already looks wrong. Check the CPU frequency settings in BIOS, and check what sysctl dev.cpu reports. Ensure that cpufreq.ko is loaded from loader.conf. > > Well. I'm running out of ideas to try, and also patience with the users > waiting for this box =( > > /Tobias