From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 1 22:04:21 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3E000E66; Wed, 1 Apr 2015 22:04:21 +0000 (UTC) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BE424104; Wed, 1 Apr 2015 22:04:20 +0000 (UTC) Received: by wibgn9 with SMTP id gn9so83385704wib.1; Wed, 01 Apr 2015 15:04:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=e6+8Cx4RDKgAtPb6N7f5AX5PNR7KrvoP5TaAhEhni18=; b=UKLHoHcVsI2HlnTwzjivr7Te+nf2j9UMye38i7zRznOpVgFSp5+bfeQHVBiYSztUmb slvBzlKF4O+JnaKTQXvCadLbRiCw2ji41O2L9xl7WcXj4DtbG//Hc3dZuJFXjeb1BHNy SUTsKxfm9K43XOEpl9vz7k9yh+Yz8pLqpA+IFR/lzZ/4ZvGYBpSlRkBhPzV1G9MoiZBq wKxWQougJ8f5pTmNH3s/AIM/Hst5nyz35o743/2o4JcGdHGGEM69uqkx9aY2TTU6gYpQ KQodYj3qm+a0QSR8oebOs12v3SJBo0D49ViIhcHxf+tJiwAMoOh42ovUEgY+ofgJdQ3r wj6w== X-Received: by 10.180.208.46 with SMTP id mb14mr18914615wic.31.1427925859360; Wed, 01 Apr 2015 15:04:19 -0700 (PDT) Received: from [172.22.3.210] (p578b7aa0.dip0.t-ipconnect.de. [87.139.122.160]) by mx.google.com with ESMTPSA id r14sm4957351wiv.13.2015.04.01.15.04.18 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 01 Apr 2015 15:04:18 -0700 (PDT) Message-ID: <551C6B62.7080205@gmail.com> Date: Thu, 02 Apr 2015 00:04:18 +0200 From: Tobias Oberstein User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: Jim Harris , Konstantin Belousov Subject: Re: NVMe performance 4x slower than expected References: <551BC57D.5070101@gmail.com> <551C5A82.2090306@gmail.com> <20150401212303.GB2379@kib.kiev.ua> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" , Michael Fuckner , Alan Somers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Apr 2015 22:04:21 -0000 > Is this vmstat after the test ? No, it wasn't (I ran vmstat hours after the test). Here is right after test (shortened test duration, otherwise exactly the same FIO config): https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_vmstat.md#nvd7 > Somewhat funny is that nvme does not use MSI(X). > > > Yes - this is exactly the problem. > > nvme does use MSI-X if it can allocate the vectors (one per core). With > 48 cores, > I suspect we are quickly running out of vectors, so NVMe is reverting to > INTx. > > Could you actually send vmstat -ia (I left off the 'a' previously) - > just so we can > see all allocated interrupt vectors. > > As an experiment, can you try disabling hyperthreading - this will > reduce the The CPUs in this box root@s4l-zfs:~/src/sys/amd64/conf # sysctl hw.model hw.model: Intel(R) Xeon(R) CPU E7-8857 v2 @ 3.00GHz don't have hyperthreading (we deliberately selected CPU model for max. clock rather than HT) http://ark.intel.com/products/75254/Intel-Xeon-Processor-E7-8857-v2-30M-Cache-3_00-GHz > number of cores and should let you get MSI-X vectors allocated for at least > the first couple of NVMe controllers. Then please re-run your performance > test on one of those controllers. > You mean I should run against nvdN where N is a controller that still got MSI-X while other controllers did not? How would I find out which controller N? I don't know which nvdN is mounted in a PCIe slot directly assigned to which CPU socket, and I don't know which one's still got MSI-X and which not. I could arrange for disabling all but 1 CPU and retest. Would that help? === Right after running against nvd7 irq56: nvme0 6440 0 ... irq106: nvme7 145056 3 Then, immediately thereafter, running against nvd0 https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_vmstat.md#nvd0 irq56: nvme0 9233 0 ... irq106: nvme7 145056 3 === Earlier this day, I ran multiple longer tests .. all against nvd7. So if these are cumulative numbers since last boot, that would make sense.