From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 08:42:32 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7C2A09D4 for ; Fri, 16 Jan 2015 08:42:32 +0000 (UTC) Received: from mail-wi0-x22e.google.com (mail-wi0-x22e.google.com [IPv6:2a00:1450:400c:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0DBC3E00 for ; Fri, 16 Jan 2015 08:42:32 +0000 (UTC) Received: by mail-wi0-f174.google.com with SMTP id n3so655744wiv.1 for ; Fri, 16 Jan 2015 00:42:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=mA9L0vesN/gdkLqQ1EUVzvDnfu1u/cEbCysDUXsDWNc=; b=gzkCu+LlJeA2IMHeW4vuj0y8QVv+oYGoBP7K/fZWbcwP6Jxz0uRtAQ7eXqTwv4OEdy sMlRGMfM1b0g7V9oUu7NrZEByFjmM3uyAET9z9hqnliIE8X8wBaSBmYbPIY8cNXrgtrE yOkhty2ETjgwiAiVPpfjRlx27wxBN7VZZ3Blo3GWp03OgTwZvVvIvT2tYgzQReAxNiFs rdqapS21U3jJ1gCHnPgCR1dy3BNoPJlee/8mvyoIc2fnrd7wK53vShiU0FLcEvi8LfZk qinn9ThZfEX98GqEseF6aT9i8qQEEQLI/oO16W4y0HvGAa6/joD/zNucQEo1et+Hr4TB O0yA== X-Received: by 10.194.190.162 with SMTP id gr2mr17755770wjc.13.1421397750329; Fri, 16 Jan 2015 00:42:30 -0800 (PST) Received: from [192.168.3.105] ([193.148.0.35]) by mx.google.com with ESMTPSA id x16sm2039679wia.15.2015.01.16.00.42.29 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 16 Jan 2015 00:42:29 -0800 (PST) Message-ID: <54B8CEF4.9040206@gmail.com> Date: Fri, 16 Jan 2015 10:42:28 +0200 From: Mihai Vintila User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: Poor performance on Intel P3600 NVME driver References: <54B7F769.40605@gmail.com> <20150115175927.GA19071@zxy.spb.ru> <54B8C7E9.3030602@gmail.com> In-Reply-To: <54B8C7E9.3030602@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 08:42:32 -0000 Actually seems that i've was wrong vfs.zfs.top_maxinflight does not have any influence and this is confirmed by the nvmecontrol perftests. nvmecontrol perftest -n 32 -o read -s 512 -t30 nvme0ns1 Threads: 32 Size: 512 READ Time: 30 IO/s: 270212 MB/s: 131 nvmecontrol perftest -n 32 -o write -s 512 -t30 nvme0ns1 Threads: 32 Size: 512 WRITE Time: 30 IO/s: 13658 MB/s: 6 I was able to recover from the errors from previous message. They were cause by the fact that i've commented hw.nvme.per_cpu_io_queues=0 in loader.conf . After setting it back things got back to "slow normal" Performance is half of what it should be and issue seems to be the nvme driver. I'll try it on another OS to confirm it's not a hardware setting issue. Best regards, Vintila Mihai Alexandru On 1/16/2015 10:12 AM, Mihai Vintila wrote: > I've tried to increase it and obtained around 90k iops for write, but > now i only get : > > nvme0: WRITE sqid:6 cid:125 nsid:1 lba:3907028496 len:16 > nvme0: WRITE FAULTS (02/80) sqid:6 cid:125 cdw0:0 > nvme0: async event occurred (log page id=0x1) > nvme0: WRITE sqid:6 cid:126 nsid:1 lba:528 len:16 > nvme0: WRITE FAULTS (02/80) sqid:6 cid:126 cdw0:0 > nvme0: async event occurred (log page id=0x1) > nvme0: WRITE sqid:6 cid:127 nsid:1 lba:3907027984 len:16 > nvme0: WRITE FAULTS (02/80) sqid:6 cid:127 cdw0:0 > nvme0: async event occurred (log page id=0x1) > > Best regards, > Vintila Mihai Alexandru > > On 1/15/2015 7:59 PM, Slawa Olhovchenkov wrote: >> On Thu, Jan 15, 2015 at 07:22:49PM +0200, Mihai Vintila wrote: >> >>> /etc/sysctl.conf >>> >>> net.inet.tcp.fast_finwait2_recycle=1 >>> net.inet.ip.portrange.randomized=0 >>> net.inet.ip.portrange.first=1024 >>> net.inet.ip.portrange.last=65535 >>> net.inet.tcp.recvbuf_max=16777216 >>> net.inet.tcp.sendbuf_max=16777216 >>> net.inet.tcp.recvbuf_inc=65536 >>> vfs.zfs.vdev.trim_on_init=0 >>> #close time_wait connections at 2*7500ms >>> net.inet.tcp.msl=7500 >>> kern.ipc.somaxconn=4096 >>> net.inet.icmp.icmplim=2000 >>> >>> #zfs >>> vfs.zfs.txg.timeout=5 >>> vfs.zfs.prefetch_disable=1 >> >>> Any help to bring this device to proper speed will be welcomed. >> Do you try to increase vfs.zfs.top_maxinflight? >