From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 17:09:00 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6466D787 for ; Fri, 16 Jan 2015 17:09:00 +0000 (UTC) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E8AF6DEB for ; Fri, 16 Jan 2015 17:08:59 +0000 (UTC) Received: by mail-wi0-f177.google.com with SMTP id l15so5202642wiw.4 for ; Fri, 16 Jan 2015 09:08:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=uOMIfacjHTMzgTz8ecVE0sqjr4zGByEtKZyEIMyfX1U=; b=L3wk6ZMd56o044IuL5udJrXoaelbnVGm5cEchivASUrvZMQBuQLX5w+IpKk/ZLeCzC NR0MGcPp0UXy2rNZYHcKN/hTdB7xt+MCHhwJfancQayF0R6flf1SlVJ28QLptqnRx+wJ lregsBcW1gwo9689UC9i8o49NE/zwHkxxm3tLba/1HGj5yX6liGjushO1bN/yn48t+MY 75ymOS2qNG/6cg5CBGAsC4LXn9ZZ2UDJcsI5Q47m4MBVAyNWOGAPRGx3iShcOF40b75y dhxO7ETJNu8exmtAE5o+QtPdR4jrTx74ymS1uzUR6ucITOqeUUuVojFnDuOqT43AzJ9R grBg== X-Received: by 10.180.77.114 with SMTP id r18mr8183146wiw.8.1421428138004; Fri, 16 Jan 2015 09:08:58 -0800 (PST) Received: from [192.168.3.105] ([193.148.0.35]) by mx.google.com with ESMTPSA id u3sm3725945wiw.24.2015.01.16.09.08.56 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 16 Jan 2015 09:08:57 -0800 (PST) Message-ID: <54B945A8.10408@gmail.com> Date: Fri, 16 Jan 2015 19:08:56 +0200 From: Mihai Vintila User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: Poor performance on Intel P3600 NVME driver References: <54B7F769.40605@gmail.com> <20150115175927.GA19071@zxy.spb.ru> In-Reply-To: <20150115175927.GA19071@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 17:09:00 -0000 There are clearly issues with the controller driver on FreeBSD. Not only zfs related: I've did some testing under Centos 7 with ext4 Command line used: iozone -Rb /root/output_iops.wks -O -i 0 -i 1 -i 2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -r 512K -s 8G Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 8388608 4 156982 0 859809 0 656572 208156 8388608 8 82545 0 503549 0 435005 111282 8388608 32 21620 0 129740 0 135738 31361 8388608 64 11070 0 66027 0 71660 16175 8388608 128 5558 0 31800 0 35361 8066 8388608 512 1383 0 7520 0 8472 2068 Same test on BSD with ufs Command line used: iozone -Rb /root/output.wks -O -i 0 -i 1 -i 2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -s 1G Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 1048576 4 120642 0 368188 0 252302 41650 1048576 8 61871 0 253074 0 189851 35015 1048576 32 15925 0 89190 0 80004 19357 1048576 64 7958 0 47190 0 59120 10070 1048576 128 4024 0 23619 0 22435 5542 ZFS test on BSD: Command line used: iozone -Rb /root/output.wks -O -i 0 -i 1 -i 2 -e -+n -r4K -s 1G Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 1048576 4 69600 0 99750 0 92007 49556 Perftest while write is close to what it should be, read takes a 4x penalty nvmecontrol perftest -n 32 -o write -s 4096 -t30 nvme0ns1 Threads: 32 Size: 4096 WRITE Time: 30 IO/s: 211294 MB/s: 825 nvmecontrol perftest -n 32 -o read -s 4096 -t30 nvme0ns1 Threads: 32 Size: 4096 READ Time: 30 IO/s: 221365 MB/s: 864 What i've tried for zfs: vfs.zfs.vdev.sync_read_min_active: 32 vfs.zfs.vdev.sync_read_max_active: 32 vfs.zfs.vdev.sync_write_min_active: 32 vfs.zfs.vdev.sync_write_max_active: 32 vfs.zfs.vdev.async_read_min_active: 32 vfs.zfs.vdev.async_read_max_active: 32 vfs.zfs.vdev.async_write_min_active: 32 vfs.zfs.vdev.async_write_max_active: 32 No effect what so ever in tests. increasing vfs.zfs.top_maxinflight - > no effect While i've managed to make it a little bit more stable so that controller doesn't just disappear by disabling ASPM in bios i still get : nvme0: resetting controller and after this i have to reboot. Some other errors i had today which where fixed with ssd firmware update and by forcing PCIeX to 16x : 1511.530545] Buffer I/O error on device nvme0n1, logical block 0 [ 1511.530594] lost page write due to I/O error on nvme0n1 [ 1517.694014] Buffer I/O error on device nvme1n1, logical block 0 [ 1517.694064] lost page write due to I/O error on nvme1n1 Best regards, Vintila Mihai Alexandru On 1/15/2015 7:59 PM, Slawa Olhovchenkov wrote: > On Thu, Jan 15, 2015 at 07:22:49PM +0200, Mihai Vintila wrote: > >> /etc/sysctl.conf >> >> net.inet.tcp.fast_finwait2_recycle=1 >> net.inet.ip.portrange.randomized=0 >> net.inet.ip.portrange.first=1024 >> net.inet.ip.portrange.last=65535 >> net.inet.tcp.recvbuf_max=16777216 >> net.inet.tcp.sendbuf_max=16777216 >> net.inet.tcp.recvbuf_inc=65536 >> vfs.zfs.vdev.trim_on_init=0 >> #close time_wait connections at 2*7500ms >> net.inet.tcp.msl=7500 >> kern.ipc.somaxconn=4096 >> net.inet.icmp.icmplim=2000 >> >> #zfs >> vfs.zfs.txg.timeout=5 >> vfs.zfs.prefetch_disable=1 > >> Any help to bring this device to proper speed will be welcomed. > Do you try to increase vfs.zfs.top_maxinflight?