From owner-freebsd-current@freebsd.org Sun Nov 29 00:10:10 2015 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 250A4A3B552 for ; Sun, 29 Nov 2015 00:10:10 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [38.97.5.132]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 70F1B1AA2; Sun, 29 Nov 2015 00:10:09 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx2.shrew.net (8.14.7/8.14.7) with ESMTP id tAT07k1U069043; Sat, 28 Nov 2015 18:07:46 -0600 (CST) (envelope-from mgrooms@shrew.net) Received: from [10.22.200.30] (cpe-72-179-24-154.austin.res.rr.com [72.179.24.154]) by mail.shrew.net (Postfix) with ESMTPSA id B0DEA18BAFE; Sat, 28 Nov 2015 18:07:35 -0600 (CST) Subject: Re: Resizing a zpool as a VMware ESXi guest ... To: freebsd-current@freebsd.org References: <543841B8.4070007@shrew.net> <20141016081016.GA4670@brick.home> <5657F135.6080902@shrew.net> <56581F5A.4010009@digiware.nl> <56589C1A.1020702@shrew.net> <5658A764.5030508@shrew.net> <565906E8.9060005@shrew.net> From: Matthew Grooms Cc: mav@FreeBSD.org, trasz@FreeBSD.org Message-ID: <565A425C.7020101@shrew.net> Date: Sat, 28 Nov 2015 18:10:04 -0600 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <565906E8.9060005@shrew.net> Content-Type: multipart/mixed; boundary="------------030409010104040509000407" X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mx2.shrew.net [10.24.10.11]); Sat, 28 Nov 2015 18:07:46 -0600 (CST) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Nov 2015 00:10:10 -0000 This is a multi-part message in MIME format. --------------030409010104040509000407 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit On 11/27/2015 7:44 PM, Matthew Grooms wrote: > I spent the day looking over the FreeBSD cam and scsi_da source code. > After sprinkling a bunch of printf's around to see what code paths > were being called, It's obvious that Edward was correct in assuming > that ESXi doesn't return any 'Unit Attention' sense information in > response to a 'Read Capacity' request. This kinda makes sense as ESXi > emulates SCSI-2 disk devices and, as far as I can tell, the 0x2A/0x09 > ASC/ASCQ sense code that denotes 'Capacity Data Has Changed' wasn't > defined until the SCSCI-3 spec. It's frustrating that the only way to > get the scsci_da code to call reprobe() is by receiving a command from > the device. Would something like this work? ... > > 1) Register a callback using xpt_register_async( daasync, > AC_REPROBE_DEVICE, path ) that calls reprobe() > 2) Implement a new IOCTL in cam_xpt that camcontrol can call with the > bus:target:lun as the argument > 3) have cam_xpt capture the IOCTL request and call xpt_async( > AC_REPROBE_DEVICE, path ) as a result > > This way users would have the option of manually asking cam to > communicate the new size to geom. The only option now is one or more > reboots to gain access to the increased disk capacity. If this sounds > like a reasonable approach, I'll take a stab at implementing it. > Here is a proof of concept patch. I'm a complete noob when it comes to cam, scsi or freebsd kernel development for that matter, so I'm sure it could have been done a better way. In any case, I added a new command to camcontrol that allows you to specify a bus, target and lun as an argument. For example ... # camcontrol readcap da1 -h Device Size: 32 G, Block Length: 512 bytes # gpart show da1 => 40 58720176 da1 GPT (28G) 40 58720176 1 freebsd-ufs (28G) Note, I resized the VMDK disk in ESXi. The camcontrol output shows the size as 32G but geom thinks its 28G. # camcontrol devlist at scbus1 target 0 lun 0 (cd0,pass0) at scbus2 target 0 lun 0 (pass1,da0) at scbus2 target 1 lun 0 (pass2,da1) at scbus3 target 0 lun 0 (da2,pass3) # camcontrol reprobe 2:1:0 This generates an event that is captured by the scsci da device to forces a reprobe. The kernel output looks almost identical to when the 'Unit Attention' sense data is received ... Nov 28 17:46:13 iscsi-i kernel: (da1:mpt0:0:1:0): Re-probe requested Nov 28 17:46:13 iscsi-i kernel: GEOM_PART: da1 was automatically resized. Nov 28 17:46:13 iscsi-i kernel: Use `gpart commit da1` to save changes or `gpart undo da1` to revert them. Now that geom knows about the increased disk capacity, I can increase the partition size and grow the fs ... [root@iscsi-i /home/mgrooms]# gpart show da1 => 40 67108784 da1 GPT (32G) 40 58720176 1 freebsd-ufs (28G) 58720216 8388608 - free - (4.0G) # gpart resize -i 1 da1 da1p1 resized # growfs da1p1 Device is mounted read-write; resizing will result in temporary write suspension for /var/data1. It's strongly recommended to make a backup before growing the file system. OK to grow filesystem on /dev/da1p1, mounted on /var/data1, from 28GB to 32GB? [Yes/No] Yes super-block backups (for fsck_ffs -b #) at: 58983232, 60265472, 61547712, 62829952, 64112192, 65394432, 66676672 # df -h Filesystem Size Used Avail Capacity Mounted on /dev/da0p3 18G 5.3G 12G 31% / devfs 1.0K 1.0K 0B 100% /dev /dev/da1p1 31G 32M 28G 0% /var/data1 /dev/da2p1 15G 32M 14G 0% /var/data2 Sure would be nice to have something like this in the tree. It's really a drag to have to reboot production VMs to increase disk capacity when it could be easily avoided. I'm not sure what the correct IOCTL should look like. Maybe CAMIOCOMMAND is a better way to go? If someone with some experience with the cam/scsi subsystems was willing to give me some direction I'd be willing to try and rewrite the patch in a way that would be commit worthy. I just need some direction. Thanks, -Matthew --------------030409010104040509000407 Content-Type: text/plain; charset=UTF-8; name="cam-reprobe.diff" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="cam-reprobe.diff" Index: lib/libcam/camlib.c =================================================================== --- lib/libcam/camlib.c (revision 291390) +++ lib/libcam/camlib.c (working copy) @@ -752,3 +752,41 @@ bcopy(src, dst, sizeof(struct cam_device)); } + +/* + * Send a reprobe unit request for a given bus, target and lun + */ +int +cam_reprobe_btl(path_id_t path_id, target_id_t target_id, lun_id_t target_lun) +{ + int fd; + char *func_name = "cam_reprobe_btl"; + union ccb ccb; + + if ((fd = open(XPT_DEVICE, O_RDWR)) < 0) { + snprintf(cam_errbuf, CAM_ERRBUF_SIZE, + "%s: couldn't open %s\n%s: %s", func_name, XPT_DEVICE, + func_name, strerror(errno)); + return(-1); + } + + /* Setup our request ccb */ + bzero(&ccb.ccb_h, sizeof(struct ccb_hdr)); + ccb.ccb_h.path_id = path_id; + ccb.ccb_h.target_id = target_id; + ccb.ccb_h.target_lun = target_lun; + + /* + * Attempt to issue a unit reprobe request. This ioctl will fail if the + * kernel can't find a unit for the given path, target and lun. + */ + if (ioctl(fd, CAMUNITREPROBE, &ccb) != 0) { + snprintf(cam_errbuf, CAM_ERRBUF_SIZE, + "%s: failed to issue unit reprobe request", func_name ); + close(fd); + return(-1); + } + + close(fd); + return 0; +} Index: lib/libcam/camlib.h =================================================================== --- lib/libcam/camlib.h (revision 291390) +++ lib/libcam/camlib.h (working copy) @@ -146,6 +146,8 @@ struct cam_device *dst); int cam_get_device(const char *path, char *dev_name, int devnamelen, int *unit); +int cam_reprobe_btl(path_id_t path_id, target_id_t target_id, + lun_id_t target_lun); /* * Buffer encoding/decoding routines, from the old SCSI library. Index: sbin/camcontrol/camcontrol.c =================================================================== --- sbin/camcontrol/camcontrol.c (revision 291390) +++ sbin/camcontrol/camcontrol.c (working copy) @@ -100,7 +100,8 @@ CAM_CMD_APM = 0x00000021, CAM_CMD_AAM = 0x00000022, CAM_CMD_ATTRIB = 0x00000023, - CAM_CMD_OPCODES = 0x00000024 + CAM_CMD_OPCODES = 0x00000024, + CAM_CMD_REPROBE = 0x00000025 } cam_cmdmask; typedef enum { @@ -193,6 +194,7 @@ #endif /* MINIMALISTIC */ {"rescan", CAM_CMD_RESCAN, CAM_ARG_NONE, NULL}, {"reset", CAM_CMD_RESET, CAM_ARG_NONE, NULL}, + {"reprobe", CAM_CMD_REPROBE, CAM_ARG_NONE, NULL}, #ifndef MINIMALISTIC {"cmd", CAM_CMD_SCSI_CMD, CAM_ARG_NONE, scsicmd_opts}, {"command", CAM_CMD_SCSI_CMD, CAM_ARG_NONE, scsicmd_opts}, @@ -3127,6 +3129,34 @@ } static int +doreprobe(int argc, char **argv) +{ + static const char must[] = + "you must specify a bus:target:lun to reprobe"; + int rv; + path_id_t bus = CAM_BUS_WILDCARD; + target_id_t target = CAM_TARGET_WILDCARD; + lun_id_t lun = CAM_LUN_WILDCARD; + char *tstr; + + if (argc < 3) { + warnx(must); + return(1); + } + + tstr = argv[optind]; + while (isspace(*tstr) && (*tstr != '\0')) + tstr++; + rv = parse_btl(argv[optind], &bus, &target, &lun, &arglist); + if (rv != 3) { + warnx(must); + return(1); + } + + return(cam_reprobe_btl(bus, target, lun)); +} + +static int rescan_or_reset_bus(path_id_t bus, int rescan) { union ccb ccb, matchccb; @@ -8685,6 +8715,7 @@ #endif /* MINIMALISTIC */ " camcontrol rescan \n" " camcontrol reset \n" +" camcontrol reprobe bus:target:lun\n" #ifndef MINIMALISTIC " camcontrol defects [dev_id][generic args] <-f format> [-P][-G]\n" " [-q][-s][-S offset][-X]\n" @@ -9053,6 +9084,7 @@ */ if ((cmdlist == CAM_CMD_RESCAN) || (cmdlist == CAM_CMD_RESET) + || (cmdlist == CAM_CMD_REPROBE) || (cmdlist == CAM_CMD_DEVTREE) || (cmdlist == CAM_CMD_USAGE) || (cmdlist == CAM_CMD_DEBUG)) @@ -9204,6 +9236,9 @@ case CAM_CMD_RESET: error = dorescan_or_reset(argc, argv, 0); break; + case CAM_CMD_REPROBE: + error = doreprobe(argc, argv); + break; #ifndef MINIMALISTIC case CAM_CMD_READ_DEFECTS: error = readdefects(cam_dev, argc, argv, combinedopt, Index: sys/amd64/conf/GENERIC =================================================================== --- sys/amd64/conf/GENERIC (revision 291390) +++ sys/amd64/conf/GENERIC (working copy) @@ -366,3 +366,6 @@ # The crypto framework is required by IPSEC device crypto # Required by IPSEC + +# Cam Debug Support +options CAMDEBUG Index: sys/cam/cam_ccb.h =================================================================== --- sys/cam/cam_ccb.h (revision 291390) +++ sys/cam/cam_ccb.h (working copy) @@ -767,6 +767,7 @@ * Definitions for the asynchronous callback CCB fields. */ typedef enum { + AC_UNIT_REPROBE = 0x8000,/* Device reprobe user request */ AC_UNIT_ATTENTION = 0x4000,/* Device reported UNIT ATTENTION */ AC_ADVINFO_CHANGED = 0x2000,/* Advance info might have changes */ AC_CONTRACT = 0x1000,/* A contractual callback */ Index: sys/cam/cam_xpt.c =================================================================== --- sys/cam/cam_xpt.c (revision 291390) +++ sys/cam/cam_xpt.c (working copy) @@ -717,7 +717,41 @@ } xpt_unlock_buses(); break; + } + case CAMUNITREPROBE: { + union ccb *inccb; + struct cam_path path; + + inccb = (union ccb *)addr; + + /* + * Make sure a specific bus, target and lun were specified. + */ + if (inccb->ccb_h.path_id == CAM_BUS_WILDCARD || + inccb->ccb_h.target_id == CAM_TARGET_WILDCARD || + inccb->ccb_h.target_lun == CAM_LUN_WILDCARD) { + error = EINVAL; + break; } + + /* + * Compile a path using the bus, target, and lun the user passed in. + */ + if (xpt_compile_path(&path, NULL, + inccb->ccb_h.path_id, + inccb->ccb_h.target_id, + inccb->ccb_h.target_lun) != + CAM_REQ_CMP){ + error = EINVAL; + break; + } + + /* + * Notify interested parties + */ + xpt_async(AC_UNIT_REPROBE, &path, NULL); + break; + } default: error = ENOTTY; break; Index: sys/cam/scsi/scsi_da.c =================================================================== --- sys/cam/scsi/scsi_da.c (revision 291390) +++ sys/cam/scsi/scsi_da.c (working copy) @@ -1702,6 +1702,17 @@ } break; } + case AC_UNIT_REPROBE: + { + softc = (struct da_softc *)periph->softc; + + xpt_print(periph->path, + "Re-probe requested\n"); + softc->flags &= ~DA_FLAG_PROBED; + dareprobe(periph); + + break; + } case AC_UNIT_ATTENTION: { union ccb *ccb; @@ -2240,7 +2251,7 @@ * would be to not attach the device on failure. */ xpt_register_async(AC_SENT_BDR | AC_BUS_RESET | AC_LOST_DEVICE | - AC_ADVINFO_CHANGED | AC_SCSI_AEN | AC_UNIT_ATTENTION, + AC_ADVINFO_CHANGED | AC_SCSI_AEN | AC_UNIT_ATTENTION | AC_UNIT_REPROBE, daasync, periph, periph->path); /* @@ -3244,12 +3255,11 @@ if (have_sense) scsi_sense_print( &done_ccb->csio); - else { + else xpt_print(periph->path, "got CAM status %#x\n", done_ccb->ccb_h.status); - } - + xpt_print(periph->path, "fatal error, " "failed to attach to device\n"); Index: sys/cam/scsi/scsi_pass.h =================================================================== --- sys/cam/scsi/scsi_pass.h (revision 291390) +++ sys/cam/scsi/scsi_pass.h (working copy) @@ -38,5 +38,6 @@ */ #define CAMIOCOMMAND _IOWR(CAM_VERSION, 2, union ccb) #define CAMGETPASSTHRU _IOWR(CAM_VERSION, 3, union ccb) +#define CAMUNITREPROBE _IOWR(CAM_VERSION, 4, union ccb) #endif --------------030409010104040509000407--