From owner-freebsd-fs@freebsd.org Tue Dec 1 17:34:14 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AF186A3E5AC for ; Tue, 1 Dec 2015 17:34:14 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-pa0-x235.google.com (mail-pa0-x235.google.com [IPv6:2607:f8b0:400e:c03::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7F7D315FF for ; Tue, 1 Dec 2015 17:34:14 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: by pacej9 with SMTP id ej9so11448386pac.2 for ; Tue, 01 Dec 2015 09:34:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=OxBxRCN2ZF7QV7KJr5gLzqRvvmFVx+4KtIeDs1zZjg0=; b=ltOWYTajnSbQbuyQ1AGdzD6ATn99TYiJ9otadgWLEWpHDl1v6vRLm8W+rsxpi/LZPX 3obq7Ssd9gcsq8wG4k1caiMLM90AEjgHjrJjoNjf3BV2kvE9ouYhgE34wWATzskAcxoN g5Hgy0ZSPT5oOXc9Kap2lefwr/kSwvTfS/JFO+O6Z46u5BuDv2OpwwK7qTb9aTjHM7g5 WL6oCOANP1o38o5mMo1oMAtUpd9g/h+wJYIX8wg45/Ec+U2eosnR6d5I5KXZn0fbZ5ty FpYT4ivWZwzNqCMNyadkY6TFdLQpl6wNQY81dXLo2jwe4r9moE8Jv6CBUqh7YCrvp+9r NAbA== X-Received: by 10.98.43.67 with SMTP id r64mr81886505pfr.3.1448991254043; Tue, 01 Dec 2015 09:34:14 -0800 (PST) Received: from mavbook.mavhome.dp.ua ([12.229.62.2]) by smtp.googlemail.com with ESMTPSA id u76sm59131101pfa.88.2015.12.01.09.34.13 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 01 Dec 2015 09:34:13 -0800 (PST) Sender: Alexander Motin Message-ID: <565DDA14.4010006@FreeBSD.org> Date: Tue, 01 Dec 2015 19:34:12 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Christopher Forgeron , Emil Muratov CC: FreeBSD Filesystems Subject: Re: CAM Target over FC and UNMAP problem References: <54F88DEA.2070301@hotplug.ru> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Dec 2015 17:34:14 -0000 Not really. But just as an idea you may try to set tloader unable vfs.zfs.trim.enabled=0 . Aside of obvious disabling TRIM, that is no-op for non-SSD ZFS pool, it also changes the day deletes are handled, possibly making them less blocking. I see no issue here from CTL side -- it does well. It indeed does not limit size of single UNMAP operation, that is not good, but that is because it has no idea about performance of backing store. On 01.12.2015 16:39, Christopher Forgeron wrote: > Did this ever progress further? I'm load testing 10.2 zvol / UNMAP, and > having similar lockup issues. > > On Thu, Mar 5, 2015 at 1:10 PM, Emil Muratov > wrote: > > > I've got an issue with CTL UNMAP and zvol backends. > Seems that UNMAP from the initiator passed to the underlying disks > (without trim support) causes IO blocking to the whole pool. Not sure > where to address this problem. > > My setup: > - plain SATA 7.2 krpm drives attached to Adaptec aacraid SAS controller > - zfs raidz pool over plain drives, no partitioning > - zvol created with volmode=dev > - Qlogic ISP 2532 FC HBA in target mode > - FreeBSD 10.1-STABLE #1 r279593 > > Create a new LUN with a zvol backend > > ctladm realsync off > ctladm port -o on -p 5 > ctladm create -b block -o file=/dev/zvol/wd/tst1 -o unmap=on -l 0 -d > wd.tst1 -S tst1 > > Both target an initiator hosts connected to the FC fabric. Initiator is > Win2012 server, actually it is a VM with RDM LUN to the guest OS. > Formating, reading and writing large amounts of data (file copy/IOmeter) > - so far so good. > But as soon as I've tried to delete large files all IO to the LUN > blocks, initiator system just iowaits. gstat on target shows that > underlying disk load bumped to 100%, queue up to 10, but no iowrites > actually in progress, only decent amount of ioreads. After a minute or > so IO unblocks for a second or two than blocks again and so on again > until all UNMAPs are done, it could take up to 5 minutes to delete 10Gb > file. I can see that 'logicalused' property of a zvol shows that the > deleted space was actually released. System log is filled with CTL msgs: > > > kernel: (ctl2:isp1:0:0:3): ctlfestart: aborted command 0x12aaf4 > discarded > kernel: (2:5:3/3): WRITE(10). CDB: 2a 00 2f d4 74 b8 00 00 08 00 > kernel: (2:5:3/3): Tag: 0x12ab24, type 1 > kernel: (2:5:3/3): ctl_process_done: 96 seconds > kernel: (ctl2:isp1:0:0:3): ctlfestart: aborted command 0x12afa4 > discarded > kernel: (ctl2:isp1:0:0:3): ctlfestart: aborted command 0x12afd4 > discarded > kernel: ctlfedone: got XPT_IMMEDIATE_NOTIFY status 0x36 tag 0xffffffff > seq 0x121104 > kernel: (ctl2:isp1:0:0:3): ctlfe_done: returning task I/O tag 0xffffffff > seq 0x1210d4 > > > I've tried to tackle some sysctls, but no success so far. > > vfs.zfs.vdev.bio_flush_disable: 1 > vfs.zfs.vdev.bio_delete_disable: 1 > vfs.zfs.trim.enabled=0 > > > Disabling UNMAP in CTL (-o unmap=off) resolves the issue completely but > than there is no space reclamation for zvol. > > Any hints would be appreciated. > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org > " > > -- Alexander Motin