From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 1 20:05:03 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 90B04F9A; Wed, 1 Apr 2015 20:05:03 +0000 (UTC) Received: from mail-qg0-x235.google.com (mail-qg0-x235.google.com [IPv6:2607:f8b0:400d:c04::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 44BFB1D8; Wed, 1 Apr 2015 20:05:03 +0000 (UTC) Received: by qgh3 with SMTP id 3so52765716qgh.2; Wed, 01 Apr 2015 13:05:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=CYHxygrnAyQNNTLjSMXb5oNsh6QgtB4DGpbHP7GLodA=; b=BAFraz9p9dHt7lKihWMgan1cJhfQbq3y7AcLrmIvjzYA89ho0LD05hyJRd5o9hFgd5 zvpsbTj6TeTXTvWHUW03NtLBddRclcOwDEmuLLzGQ3D8V+1yMjzxtMcJShzx5oiXB4i7 lZ/smqkndtC5ihEaWx4OZEJFQrQ+A7FapVJe11pLN0We+Ji6VlBjdjiVophoKhJJC1n9 k7NH6YpzfVS5/mzoSK+NQoR2+fDefPDourufa7NKTqCM9U255oA2LSUIyVhPqTUeJW0Y +0Yt5QqpGt2Y00n/IhIucQ04WcrRrCGcG6Xw7cgUuz8kNyOixTad8niAgKkV1X6+PF5a +ryg== MIME-Version: 1.0 X-Received: by 10.140.83.19 with SMTP id i19mr32958061qgd.97.1427918702349; Wed, 01 Apr 2015 13:05:02 -0700 (PDT) Received: by 10.140.38.73 with HTTP; Wed, 1 Apr 2015 13:05:02 -0700 (PDT) In-Reply-To: References: <551BC57D.5070101@gmail.com> Date: Wed, 1 Apr 2015 13:05:02 -0700 Message-ID: Subject: Re: NVMe performance 4x slower than expected From: Jim Harris To: Alan Somers Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-hackers@freebsd.org" , Tobias Oberstein , Michael Fuckner , Konstantin Belousov X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Apr 2015 20:05:03 -0000 On Wed, Apr 1, 2015 at 7:42 AM, Alan Somers wrote: > On Wed, Apr 1, 2015 at 4:16 AM, Tobias Oberstein > wrote: > > Hi, > > > > I am testing performance of a NVMe device (Intel P3700) using FIO at the > > block device level and get 4x slower performance than expected: > > > > 4kB Random Read > > > > Intel Datasheet FIO Measurement Match > > P3700 450,000 107,092 24% > > DC S3700 75,000 67,186 90% > > > > The 2nd line are results for an Intel DC S3700 for comparison (with this > > device, I do see the performance expected, but not for the P3700). > > > > Hardware: > > > > - 4 sockets, 48 core x86-64, 3TB RAM > > - 8 x Intel P3700 2TB > > - 12 x Intel DC S3700 800GB (via LSI HBAs) > > > > Software: > > > > FreeBSD 11 Current with patches (DMAR and ZFS patches, otherwise the box > > doesn't boot at all .. because of 3TB RAM and the amount of periphery). > > > Do you still have WITNESS and INVARIANTS turned on in your kernel > config? They're turned on by default for Current, but they do have > some performance impact. To turn them off, just build a > GENERIC-NODEBUG kernel . > > Could you also post full dmesg output as well as vmstat -i? > > > > > Complete info and test logs are here: > > > > > https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/perftests.md > > > > Right now I am running Linux on the box (openSUSE 13.2). Using the exact > > same FIO control file, the values for the DC S3700 are very close to > > FreeBSD, but the values for the P3700 are much higher: > > > > > https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/perftests.md#more-numbers-linux > > > > I am looking for tuning hints or general advice for FreeBSD and NVMe. > > > > I would like to go with FreeBSD (a major aspect is ZFS), but the > performance > > issues with NVMe might be a deal breaker. > > > > Cheers, > > /Tobias > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to " > freebsd-hackers-unsubscribe@freebsd.org" >