From owner-freebsd-fs@FreeBSD.ORG Thu Mar 19 20:21:22 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 876C38FF; Thu, 19 Mar 2015 20:21:22 +0000 (UTC) Received: from gate.pik.ru (gate.pik.ru [IPv6:2a03:5a00:31:40::25]) by mx1.freebsd.org (Postfix) with ESMTP id 03C0A171; Thu, 19 Mar 2015 20:21:21 +0000 (UTC) Received: from delta.hotplug.ru (localhost [127.0.0.1]) by gate.pik.ru (Postfix) with ESMTP id 2CD8311B99; Thu, 19 Mar 2015 23:21:19 +0300 (MSK) Received: from delta.hotplug.ru (unknown [141.101.202.35]) by gate.pik.ru (Postfix) with ESMTP id F0D4F11B98; Thu, 19 Mar 2015 23:21:18 +0300 (MSK) Received: from ghost-pc.home.lan (unknown [IPv6:2a02:290:2:1f9:912b:bd09:2a07:2f23]) by delta.hotplug.ru (Postfix) with ESMTPSA id B9AD74197; Thu, 19 Mar 2015 23:21:16 +0300 (MSK) Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: freebsd-fs@freebsd.org, "Alexander Motin" Subject: Re: CAM Target over FC and UNMAP problem References: <54F88DEA.2070301@hotplug.ru> <54F8AB96.6080600@FreeBSD.org> <54F9782A.90501@hotplug.ru> <54F98135.5000908@FreeBSD.org> <550979C8.2070603@hotplug.ru> <550AD6ED.50201@FreeBSD.org> Date: Thu, 19 Mar 2015 23:21:16 +0300 MIME-Version: 1.0 Content-Transfer-Encoding: Quoted-Printable From: "Emil Muratov" Message-ID: In-Reply-To: <550AD6ED.50201@FreeBSD.org> User-Agent: Opera Mail/12.17 (Win32) X-KLMS-Rule-ID: 1 X-KLMS-Message-Action: clean X-KLMS-AntiSpam-Lua-Profiles: 74804 [Mar 19 2015] X-KLMS-AntiSpam-Version: 5.5.4 X-KLMS-AntiSpam-Envelope-From: gpm@hotplug.ru X-KLMS-AntiSpam-Rate: 40 X-KLMS-AntiSpam-Status: not_detected X-KLMS-AntiSpam-Method: none X-KLMS-AntiSpam-Moebius-Timestamps: 3437332, 3437395, 0 X-KLMS-AntiSpam-Info: LuaCore: 175 2015-03-18_14-10-30 59b0fb5d1fe0bc13ab72a23d6aa445f4185e0a58, {relay has no DNS name}, Auth:dmarc=none header.from=hotplug.ru; spf=none smtp.mailfrom=hotplug.ru; dkim=none, {rdns complete}, {DNS response errors} X-KLMS-AntiSpam-Interceptor-Info: scan successful X-KLMS-AntiVirus: Kaspersky Security 8.0 for Linux Mail Server 8.0.0.455, not checked X-KLMS-AntiVirus-Status: NotChecked: not checked, skipped X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Mar 2015 20:21:22 -0000 Alexander Motin =D0=BF=D0=B8=D1=81=D0=B0=D0=BB(=D0=B0)= =D0=B2 =D1=81=D0=B2=D0=BE=D1=91=D0=BC =D0=BF=D0=B8=D1=81=D1=8C=D0=BC=D0= =B5 Thu, 19 Mar 2015 = 17:02:21 +0300: >> I looked through ctl code and changed hardcoded values for 'unmap LBA= >> count' and 'unmap block descr count' to 8Mb and 128. >> With this values UNMAP works like a charm! No more IO blocks, IO >> timeouts, log error, high disk loads or anything, only a medium >> performance drop-down during even very large unmaps. But this >> performance drop is nothing compared with those all-blocking issues. = No >> problems over FiberChannel transport too. > In my present understanding of SBC-4 specification, implemented also i= n > FreeBSD initiator, MAXIMUM UNMAP LBA COUNT is measured not per segment= , > but per command. Hmm.. my understanding of SBC specs is close to 0 :) Just checked it, = looks like you were right - sounds like it must be the total block count= = per command. My first assumption was based on SG_UNMAP(8) notes from = SG3_UTILS, it defines NUM as a value constrained by MAXIMUM UNMAP LBA = COUNT, but there can be more than one LBA,NUM pairs. Not sure how it was= = implemented in the sg_unmap code itself. Anyway, based on the wrong = assumption I was lucky to hit the the jackpot :) > From such perspective limiting it to 8MB per UNMAP > command is IMHO an overkill. Could you try to increase it to 2097152, > which is 1GB, while decrease MAXIMUM UNMAP BLOCK DESCRIPTOR COUNT from= > 128 to 64? Will it give acceptable results? Just did it, it was as bad as with the default values, same io blocking,= = errors and timeouts. I'll try to test some more values between 1G and 8M= :) Have no idea what is the basis for choosing this values without = undestanding ZFS internals. We have a t10 compliant Hitachi HUS-VM FC-storage with a set of options = = for different initiators. A standart t10-compliant setup gives this valu= es = in bl VPD: Block limits VPD page (SBC): Write same no zero (WSNZ): 0 Maximum compare and write length: 1 blocks Optimal transfer length granularity: 128 blocks Maximum transfer length: 0 blocks Optimal transfer length: 86016 blocks Maximum prefetch length: 0 blocks Maximum unmap LBA count: 4294967295 Maximum unmap block descriptor count: 1 Optimal unmap granularity: 86016 Unmap granularity alignment valid: 0 Unmap granularity alignment: 0 Maximum write same length: 0x80000 blocks Very odd values (86016 blocks), no idea how this works inside HUSVM but = = large unmaps is not a problem there. BTW, msdn mentions that ws2012 implements only SBC3 unmap, but not unmap= = through WRITE_SAME. I will try to test if unmap with sg_write_same behav= es = as bad on ZFS vol with a default large write_same length.