From owner-freebsd-stable@FreeBSD.ORG  Fri Jan 16 09:13:23 2015
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id A7AB6430
 for <freebsd-stable@freebsd.org>; Fri, 16 Jan 2015 09:13:23 +0000 (UTC)
Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 13D961AF
 for <freebsd-stable@freebsd.org>; Fri, 16 Jan 2015 09:13:23 +0000 (UTC)
Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD))
 (envelope-from <slw@zxy.spb.ru>)
 id 1YC2xx-000NAb-3C; Fri, 16 Jan 2015 12:13:17 +0300
Date: Fri, 16 Jan 2015 12:13:17 +0300
From: Slawa Olhovchenkov <slw@zxy.spb.ru>
To: Mihai Vintila <unixro@gmail.com>
Subject: Re: Poor performance on Intel P3600 NVME driver
Message-ID: <20150116091317.GA83755@zxy.spb.ru>
References: <54B7F769.40605@gmail.com> <20150115175927.GA19071@zxy.spb.ru>
 <54B8C7E9.3030602@gmail.com> <54B8CEF4.9040206@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <54B8CEF4.9040206@gmail.com>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: slw@zxy.spb.ru
X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false
Cc: freebsd-stable@freebsd.org
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 16 Jan 2015 09:13:23 -0000

On Fri, Jan 16, 2015 at 10:42:28AM +0200, Mihai Vintila wrote:

> Actually seems that i've was wrong vfs.zfs.top_maxinflight does not have 
> any influence and this is confirmed by the nvmecontrol perftests.
>   nvmecontrol perftest -n 32 -o read -s 512 -t30 nvme0ns1
> Threads: 32 Size:    512  READ Time:  30 IO/s:  270212 MB/s:  131
> nvmecontrol perftest -n 32 -o write -s 512 -t30 nvme0ns1
> Threads: 32 Size:    512 WRITE Time:  30 IO/s:   13658 MB/s:    6

I see datasheet specified performance metrics for 4K (not 512bytes) random r/w.
Can you try this? This is may be impact write IO/s at least.

> I was able to recover from the errors from previous message. They were 
> cause by the fact that i've commented hw.nvme.per_cpu_io_queues=0 in 
> loader.conf . After setting it back things got back to "slow normal"
> Performance is half of what it should be and  issue seems to be the nvme 
> driver. I'll try it on another OS to confirm it's not a hardware setting 
> issue.