From owner-freebsd-fs@FreeBSD.ORG Thu Mar 5 19:16:44 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A90DEF9B for ; Thu, 5 Mar 2015 19:16:44 +0000 (UTC) Received: from mail-wg0-x234.google.com (mail-wg0-x234.google.com [IPv6:2a00:1450:400c:c00::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 37424A77 for ; Thu, 5 Mar 2015 19:16:44 +0000 (UTC) Received: by wghl18 with SMTP id l18so1286258wgh.11 for ; Thu, 05 Mar 2015 11:16:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=QnDj822+Kwnhf9YtGl0uMuJF+SjK//wOVsmnbXH92+k=; b=YmesJiYtDgwxZwPT2m0rKN3RViOhMCAe/wOTgT/jhLGOQZuOyUaUB+gBhj4Z7kuUt2 aLwYNdpPr4w3+0LV2npG+eWYApAVcO4ZwTwoX4Rt36d31Girhp0l6KkYI7Wz5m1O2F9v vaW6D1vw2gTs//+0Wi2OdKsuiigHPUvZAlS9CMhbfoWdesGl+KzzOImQO3dRXSczm7si 4eRONu0EMVwD6qJg7SUsIQL+UFFuSHq1Bk7cxlfRpHITOfeMFTTcw8SMB14KvmAKIt1A rIbBXdkzExDJ/GmtjRUn+NJF/SPlrOUJfkmu/kLNyRXmuN6B9fOGlmF1B9k832KE9p0O mekg== X-Received: by 10.180.12.233 with SMTP id b9mr24989471wic.49.1425583001737; Thu, 05 Mar 2015 11:16:41 -0800 (PST) Received: from mavbook.mavhome.dp.ua ([91.198.175.1]) by mx.google.com with ESMTPSA id kj8sm11763737wjc.29.2015.03.05.11.16.40 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Mar 2015 11:16:40 -0800 (PST) Sender: Alexander Motin Message-ID: <54F8AB96.6080600@FreeBSD.org> Date: Thu, 05 Mar 2015 21:16:38 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: Emil Muratov , freebsd-fs@freebsd.org Subject: Re: CAM Target over FC and UNMAP problem References: <54F88DEA.2070301@hotplug.ru> In-Reply-To: <54F88DEA.2070301@hotplug.ru> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Mar 2015 19:16:44 -0000 Hi. On 05.03.2015 19:10, Emil Muratov wrote: > I've got an issue with CTL UNMAP and zvol backends. > Seems that UNMAP from the initiator passed to the underlying disks > (without trim support) causes IO blocking to the whole pool. Not sure > where to address this problem. There is no direct relations between UNMAP sent to ZVOl and UNMAP/TRIM to underlying disks. ZVOL UNMAP only frees some pool space, that may later be trimmed if disks support it. > My setup: > - plain SATA 7.2 krpm drives attached to Adaptec aacraid SAS controller > - zfs raidz pool over plain drives, no partitioning > - zvol created with volmode=dev > - Qlogic ISP 2532 FC HBA in target mode > - FreeBSD 10.1-STABLE #1 r279593 > Create a new LUN with a zvol backend > > ctladm realsync off Are you sure you need this? Your data are so uncritical to ignore even explicit cache flushes? > ctladm port -o on -p 5 > ctladm create -b block -o file=/dev/zvol/wd/tst1 -o unmap=on -l 0 -d > wd.tst1 -S tst1 Just for note, this configuration can now be alternatively done via ctld and /etc/ctl.conf. > Both target an initiator hosts connected to the FC fabric. Initiator is > Win2012 server, actually it is a VM with RDM LUN to the guest OS. > Formating, reading and writing large amounts of data (file copy/IOmeter) > - so far so good. > But as soon as I've tried to delete large files all IO to the LUN > blocks, initiator system just iowaits. gstat on target shows that > underlying disk load bumped to 100%, queue up to 10, but no iowrites > actually in progress, only decent amount of ioreads. After a minute or > so IO unblocks for a second or two than blocks again and so on again > until all UNMAPs are done, it could take up to 5 minutes to delete 10Gb > file. I can see that 'logicalused' property of a zvol shows that the > deleted space was actually released. System log is filled with CTL msgs: > > > kernel: (ctl2:isp1:0:0:3): ctlfestart: aborted command 0x12aaf4 discarded > kernel: (2:5:3/3): WRITE(10). CDB: 2a 00 2f d4 74 b8 00 00 08 00 > kernel: (2:5:3/3): Tag: 0x12ab24, type 1 > kernel: (2:5:3/3): ctl_process_done: 96 seconds > kernel: (ctl2:isp1:0:0:3): ctlfestart: aborted command 0x12afa4 discarded > kernel: (ctl2:isp1:0:0:3): ctlfestart: aborted command 0x12afd4 discarded > kernel: ctlfedone: got XPT_IMMEDIATE_NOTIFY status 0x36 tag 0xffffffff > seq 0x121104 > kernel: (ctl2:isp1:0:0:3): ctlfe_done: returning task I/O tag 0xffffffff > seq 0x1210d4 > > > I've tried to tackle some sysctls, but no success so far. > > vfs.zfs.vdev.bio_flush_disable: 1 > vfs.zfs.vdev.bio_delete_disable: 1 > vfs.zfs.trim.enabled=0 > > > Disabling UNMAP in CTL (-o unmap=off) resolves the issue completely but > than there is no space reclamation for zvol. > > Any hints would be appreciated. There were number of complains on UNMAP performance in Illumos lists too. Six month ago there were some fixes committed and merged to stable/10 that substantially improved the situation. Since that time I haven't observed problems with that on my tests. What's about the large amount of reads during UNMAP, I have two guesses: 1) it may be read of metadata absent in ARC. Though I doubt that there are so much metadata to read them during several minutes. 2) if UNMAP ranges were not aligned to ZVOL block, I guess ZFS could try to read blocks that need partial "unmap". I've made experiment with unmapping 512 bytes of 8K ZVOL block, and it indeed zeroed specified 512 bytes, from SCSI perspective while it would be fine to just ignore the request. -- Alexander Motin