From owner-freebsd-current@freebsd.org Sat Nov 28 01:44:08 2015 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 288EAA3B28C for ; Sat, 28 Nov 2015 01:44:08 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [38.97.5.131]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 888A61774; Sat, 28 Nov 2015 01:44:07 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx1.shrew.net (8.14.7/8.14.7) with ESMTP id tAS1fiTT042855; Fri, 27 Nov 2015 19:41:44 -0600 (CST) (envelope-from mgrooms@shrew.net) Received: from [10.22.200.30] (cpe-72-179-24-154.austin.res.rr.com [72.179.24.154]) by mail.shrew.net (Postfix) with ESMTPSA id 6C98518C1F4; Fri, 27 Nov 2015 19:41:39 -0600 (CST) Subject: Re: Resizing a zpool as a VMware ESXi guest ... To: freebsd-current@freebsd.org, trasz@FreeBSD.org References: <543841B8.4070007@shrew.net> <20141016081016.GA4670@brick.home> <5657F135.6080902@shrew.net> <56581F5A.4010009@digiware.nl> <56589C1A.1020702@shrew.net> <5658A764.5030508@shrew.net> From: Matthew Grooms Message-ID: <565906E8.9060005@shrew.net> Date: Fri, 27 Nov 2015 19:44:08 -0600 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <5658A764.5030508@shrew.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mx1.shrew.net [10.24.10.10]); Fri, 27 Nov 2015 19:41:45 -0600 (CST) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Nov 2015 01:44:08 -0000 On 11/27/2015 12:56 PM, Matthew Grooms wrote: > I thought it would be useful to get more output from the geom layer, > similar to the camcontrol debug output ... > > [root@iscsi-i /home/mgrooms]# sysctl kern.geom.debugflags=0x81 > > When I resize the iSCSI LUN and run the 'camcontrol readcap da2 -h', I > see this in the log output ... > > [root@iscsi-i /home/mgrooms]# tail -f /var/log/messages > ... > Nov 27 12:20:07 iscsi-i kernel: (pass3:iscsi1:0:0:0): Capacity data > has changed > Nov 27 12:20:07 iscsi-i kernel: g_post_event_x(0xffffffff80973850, > 0xfffff80003f4e000, 1, 0) > Nov 27 12:20:07 iscsi-i kernel: g_post_event_x(0xffffffff8097a6b0, > 0xfffff80003f42b60, 2, 0) > Nov 27 12:20:07 iscsi-i kernel: > g_resize_provider_event(0xfffff800038c6700) > Nov 27 12:20:07 iscsi-i kernel: g_part_resize(da2) > Nov 27 12:20:07 iscsi-i kernel: GEOM_PART: da2 was automatically resized. > Nov 27 12:20:07 iscsi-i kernel: Use `gpart commit da2` to save changes > or `gpart undo da2` to revert them. > Nov 27 12:20:07 iscsi-i kernel: g_raid_taste(RAID, da2) > Nov 27 12:20:07 iscsi-i kernel: g_attach(0xfffff80003e64780, > 0xfffff800038c6700) > Nov 27 12:20:07 iscsi-i kernel: g_detach(0xfffff80003e64780) > Nov 27 12:20:07 iscsi-i kernel: g_destroy_consumer(0xfffff80003e64780) > Nov 27 12:20:07 iscsi-i kernel: > g_destroy_geom(0xfffff800038c9c00(raid:taste)) > Nov 27 12:20:07 iscsi-i kernel: g_label_taste(LABEL, da2) > > However, when I resize the VMDK disk and run the 'camcontrol readcap > da1 -h' command, I see nothing in the log output. So it would appear > that even though cam is reporting the new capacity in the command line > output, the this info is not being forwarded to geom in this case. > Maybe the VMware virtual SAS device ( mpt0 ) isn't reporting some > special capability bit to cam which causes it to squelch the info? > > I'm not sure if this is useful but here is what the device info looks > like at boot time ... > > mpt0: port 0x4000-0x40ff mem > 0xfd4ec000-0xfd4effff,0xfd4f0000-0xfd4fffff irq 18 at device 0.0 on pci3 > mpt0: MPI Version=1.5.0.0 > ... > da0 at mpt0 bus 0 scbus2 target 0 lun 0 > da0: Fixed Direct Access SCSI-2 device > da0: 300.000MB/s transfers > da0: Command Queueing enabled > da0: 20480MB (41943040 512 byte sectors) > da0: quirks=0x40 > da1 at mpt0 bus 0 scbus2 target 1 lun 0 > da1: Fixed Direct Access SCSI-2 device > da1: 300.000MB/s transfers > da1: Command Queueing enabled > da1: 20480MB (41943040 512 byte sectors) > da1: quirks=0x40 > ... > da2 at iscsi1 bus 0 scbus3 target 0 lun 0 > da2: Fixed Direct Access SPC-4 SCSI device > da2: Serial Number MYSERIAL 0 > da2: 150.000MB/s transfers > da2: Command Queueing enabled > da2: 16384MB (33554432 512 byte sectors) > I spent the day looking over the FreeBSD cam and scsi_da source code. After sprinkling a bunch of printf's around to see what code paths were being called, It's obvious that Edward was correct in assuming that ESXi doesn't return any 'Unit Attention' sense information in response to a 'Read Capacity' request. This kinda makes sense as ESXi emulates SCSI-2 disk devices and, as far as I can tell, the 0x2A/0x09 ASC/ASCQ sense code that denotes 'Capacity Data Has Changed' wasn't defined until the SCSCI-3 spec. It's frustrating that the only way to get the scsci_da code to call reprobe() is by receiving a command from the device. Would something like this work? ... 1) Register a callback using xpt_register_async( daasync, AC_REPROBE_DEVICE, path ) that calls reprobe() 2) Implement a new IOCTL in cam_xpt that camcontrol can call with the bus:target:lun as the argument 3) have cam_xpt capture the IOCTL request and call xpt_async( AC_REPROBE_DEVICE, path ) as a result This way users would have the option of manually asking cam to communicate the new size to geom. The only option now is one or more reboots to gain access to the increased disk capacity. If this sounds like a reasonable approach, I'll take a stab at implementing it. Thanks, -Matthew