Date: Sat, 28 Nov 2015 22:03:18 -0600 From: Matthew Grooms <mgrooms@shrew.net> To: freebsd-current@freebsd.org, mav@FreeBSD.org, trasz@FreeBSD.org Subject: Re: Resizing a zpool as a VMware ESXi guest ... Message-ID: <565A7906.60106@shrew.net> In-Reply-To: <565A425C.7020101@shrew.net> References: <543841B8.4070007@shrew.net> <20141016081016.GA4670@brick.home> <5657F135.6080902@shrew.net> <56581F5A.4010009@digiware.nl> <56589C1A.1020702@shrew.net> <5658A764.5030508@shrew.net> <565906E8.9060005@shrew.net> <565A425C.7020101@shrew.net>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format.
--------------090907040403050103020002
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
On 11/28/2015 6:10 PM, Matthew Grooms wrote:
> On 11/27/2015 7:44 PM, Matthew Grooms wrote:
>> I spent the day looking over the FreeBSD cam and scsi_da source code.
>> After sprinkling a bunch of printf's around to see what code paths
>> were being called, It's obvious that Edward was correct in assuming
>> that ESXi doesn't return any 'Unit Attention' sense information in
>> response to a 'Read Capacity' request. This kinda makes sense as ESXi
>> emulates SCSI-2 disk devices and, as far as I can tell, the 0x2A/0x09
>> ASC/ASCQ sense code that denotes 'Capacity Data Has Changed' wasn't
>> defined until the SCSCI-3 spec. It's frustrating that the only way to
>> get the scsci_da code to call reprobe() is by receiving a command
>> from the device. Would something like this work? ...
>>
>> 1) Register a callback using xpt_register_async( daasync,
>> AC_REPROBE_DEVICE, path ) that calls reprobe()
>> 2) Implement a new IOCTL in cam_xpt that camcontrol can call with the
>> bus:target:lun as the argument
>> 3) have cam_xpt capture the IOCTL request and call xpt_async(
>> AC_REPROBE_DEVICE, path ) as a result
>>
>> This way users would have the option of manually asking cam to
>> communicate the new size to geom. The only option now is one or more
>> reboots to gain access to the increased disk capacity. If this sounds
>> like a reasonable approach, I'll take a stab at implementing it.
>>
>
> Here is a proof of concept patch. I'm a complete noob when it comes to
> cam, scsi or freebsd kernel development for that matter, so I'm sure
> it could have been done a better way. In any case, I added a new
> command to camcontrol that allows you to specify a bus, target and lun
> as an argument. For example ...
>
> # camcontrol readcap da1 -h
> Device Size: 32 G, Block Length: 512 bytes
>
> # gpart show da1
> => 40 58720176 da1 GPT (28G)
> 40 58720176 1 freebsd-ufs (28G)
>
> Note, I resized the VMDK disk in ESXi. The camcontrol output shows the
> size as 32G but geom thinks its 28G.
>
> # camcontrol devlist
> <NECVMWar VMware IDE CDR10 1.00> at scbus1 target 0 lun 0 (cd0,pass0)
> <VMware Virtual disk 1.0> at scbus2 target 0 lun 0 (pass1,da0)
> <VMware Virtual disk 1.0> at scbus2 target 1 lun 0 (pass2,da1)
> <FREEBSD CTLDISK 0001> at scbus3 target 0 lun 0 (da2,pass3)
>
> # camcontrol reprobe 2:1:0
>
> This generates an event that is captured by the scsci da device to
> forces a reprobe. The kernel output looks almost identical to when the
> 'Unit Attention' sense data is received ...
>
> Nov 28 17:46:13 iscsi-i kernel: (da1:mpt0:0:1:0): Re-probe requested
> Nov 28 17:46:13 iscsi-i kernel: GEOM_PART: da1 was automatically resized.
> Nov 28 17:46:13 iscsi-i kernel: Use `gpart commit da1` to save changes
> or `gpart undo da1` to revert them.
>
> Now that geom knows about the increased disk capacity, I can increase
> the partition size and grow the fs ...
>
> [root@iscsi-i /home/mgrooms]# gpart show da1
> => 40 67108784 da1 GPT (32G)
> 40 58720176 1 freebsd-ufs (28G)
> 58720216 8388608 - free - (4.0G)
>
> # gpart resize -i 1 da1
> da1p1 resized
>
> # growfs da1p1
> Device is mounted read-write; resizing will result in temporary write
> suspension for /var/data1.
> It's strongly recommended to make a backup before growing the file
> system.
> OK to grow filesystem on /dev/da1p1, mounted on /var/data1, from 28GB
> to 32GB? [Yes/No] Yes
> super-block backups (for fsck_ffs -b #) at:
> 58983232, 60265472, 61547712, 62829952, 64112192, 65394432, 66676672
>
> # df -h
> Filesystem Size Used Avail Capacity Mounted on
> /dev/da0p3 18G 5.3G 12G 31% /
> devfs 1.0K 1.0K 0B 100% /dev
> /dev/da1p1 31G 32M 28G 0% /var/data1
> /dev/da2p1 15G 32M 14G 0% /var/data2
>
> Sure would be nice to have something like this in the tree. It's
> really a drag to have to reboot production VMs to increase disk
> capacity when it could be easily avoided. I'm not sure what the
> correct IOCTL should look like. Maybe CAMIOCOMMAND is a better way to
> go? If someone with some experience with the cam/scsi subsystems was
> willing to give me some direction I'd be willing to try and rewrite
> the patch in a way that would be commit worthy. I just need some
> direction.
>
Ok, last post until I get some feedback. Here's a new version of the
patch complete with man page updates. It communicates via CAMIOCOMMAND
instead of introducing a new ioctl value. I tried to model it after the
device reset option, hopefully with some degree of success. Functionally
it should be the same as the first patch.
Thanks,
-Matthew
--------------090907040403050103020002
Content-Type: text/plain; charset=UTF-8;
name="cam-reprobe-v2.diff"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="cam-reprobe-v2.diff"
Index: sbin/camcontrol/camcontrol.8
===================================================================
--- sbin/camcontrol/camcontrol.8 (revision 291390)
+++ sbin/camcontrol/camcontrol.8 (working copy)
@@ -104,6 +104,9 @@
.Ic reset
.Aq all | bus Ns Op :target:lun
.Nm
+.Ic reprobe
+.Aq bus:target:lun
+.Nm
.Ic defects
.Op device id
.Op generic args
@@ -548,6 +551,9 @@
connecting to that device.
Note that this can have a destructive impact
on the system.
+.It Ic reprobe
+Tell the kernel to re-probe the given bus:target:lun
+(XPT_REPROBE_DEV) by issuing an async event to peripheral device.
.It Ic defects
Send the
.Tn SCSI
Index: sbin/camcontrol/camcontrol.c
===================================================================
--- sbin/camcontrol/camcontrol.c (revision 291390)
+++ sbin/camcontrol/camcontrol.c (working copy)
@@ -100,7 +100,8 @@
CAM_CMD_APM = 0x00000021,
CAM_CMD_AAM = 0x00000022,
CAM_CMD_ATTRIB = 0x00000023,
- CAM_CMD_OPCODES = 0x00000024
+ CAM_CMD_OPCODES = 0x00000024,
+ CAM_CMD_REPROBE = 0x00000025
} cam_cmdmask;
typedef enum {
@@ -193,6 +194,7 @@
#endif /* MINIMALISTIC */
{"rescan", CAM_CMD_RESCAN, CAM_ARG_NONE, NULL},
{"reset", CAM_CMD_RESET, CAM_ARG_NONE, NULL},
+ {"reprobe", CAM_CMD_REPROBE, CAM_ARG_NONE, NULL},
#ifndef MINIMALISTIC
{"cmd", CAM_CMD_SCSI_CMD, CAM_ARG_NONE, scsicmd_opts},
{"command", CAM_CMD_SCSI_CMD, CAM_ARG_NONE, scsicmd_opts},
@@ -273,6 +275,7 @@
static int rescan_or_reset_bus(path_id_t bus, int rescan);
static int scanlun_or_reset_dev(path_id_t bus, target_id_t target,
lun_id_t lun, int scan);
+static int reprobe_dev(path_id_t bus, target_id_t target, lun_id_t lun);
#ifndef MINIMALISTIC
static int readdefects(struct cam_device *device, int argc, char **argv,
char *combinedopt, int retry_count, int timeout);
@@ -3127,6 +3130,34 @@
}
static int
+doreprobe(int argc, char **argv)
+{
+ static const char must[] =
+ "you must specify a bus:target:lun to reprobe";
+ int rv;
+ path_id_t bus = CAM_BUS_WILDCARD;
+ target_id_t target = CAM_TARGET_WILDCARD;
+ lun_id_t lun = CAM_LUN_WILDCARD;
+ char *tstr;
+
+ if (argc < 3) {
+ warnx(must);
+ return(1);
+ }
+
+ tstr = argv[optind];
+ while (isspace(*tstr) && (*tstr != '\0'))
+ tstr++;
+ rv = parse_btl(argv[optind], &bus, &target, &lun, &arglist);
+ if (rv != 3) {
+ warnx(must);
+ return(1);
+ }
+
+ return(reprobe_dev(bus, target, lun));
+}
+
+static int
rescan_or_reset_bus(path_id_t bus, int rescan)
{
union ccb ccb, matchccb;
@@ -3382,6 +3413,59 @@
}
}
+static int
+reprobe_dev(path_id_t bus, target_id_t target, lun_id_t lun)
+{
+ union ccb ccb;
+ struct cam_device *device;
+ int fd;
+
+ device = NULL;
+
+ if (bus == CAM_BUS_WILDCARD) {
+ warnx("invalid bus number %d", bus);
+ return(1);
+ }
+
+ if (target == CAM_TARGET_WILDCARD) {
+ warnx("invalid target number %d", target);
+ return(1);
+ }
+
+ if (lun == CAM_LUN_WILDCARD) {
+ warnx("invalid lun number %jx", (uintmax_t)lun);
+ return(1);
+ }
+
+ fd = -1;
+
+ bzero(&ccb, sizeof(union ccb));
+
+ if ((fd = open(XPT_DEVICE, O_RDWR)) < 0) {
+ warnx("error opening transport layer device %s\n",
+ XPT_DEVICE);
+ warn("%s", XPT_DEVICE);
+ return(1);
+ }
+
+ ccb.ccb_h.func_code = XPT_REPROBE_DEV;
+ ccb.ccb_h.path_id = bus;
+ ccb.ccb_h.target_id = target;
+ ccb.ccb_h.target_lun = lun;
+
+ if (ioctl(fd, CAMIOCOMMAND, &ccb) < 0) {
+ warn("CAMIOCOMMAND ioctl failed");
+ close(fd);
+ return(1);
+ }
+ close(fd);
+
+ fprintf(stdout, "Re-probe of %d:%d:%jx was successful\n",
+ bus, target, (uintmax_t)lun);
+
+ return(0);
+}
+
#ifndef MINIMALISTIC
static struct scsi_nv defect_list_type_map[] = {
@@ -8685,6 +8769,7 @@
#endif /* MINIMALISTIC */
" camcontrol rescan <all | bus[:target:lun]>\n"
" camcontrol reset <all | bus[:target:lun]>\n"
+" camcontrol reprobe bus:target:lun\n"
#ifndef MINIMALISTIC
" camcontrol defects [dev_id][generic args] <-f format> [-P][-G]\n"
" [-q][-s][-S offset][-X]\n"
@@ -9053,6 +9138,7 @@
*/
if ((cmdlist == CAM_CMD_RESCAN)
|| (cmdlist == CAM_CMD_RESET)
+ || (cmdlist == CAM_CMD_REPROBE)
|| (cmdlist == CAM_CMD_DEVTREE)
|| (cmdlist == CAM_CMD_USAGE)
|| (cmdlist == CAM_CMD_DEBUG))
@@ -9204,6 +9290,9 @@
case CAM_CMD_RESET:
error = dorescan_or_reset(argc, argv, 0);
break;
+ case CAM_CMD_REPROBE:
+ error = doreprobe(argc, argv);
+ break;
#ifndef MINIMALISTIC
case CAM_CMD_READ_DEFECTS:
error = readdefects(cam_dev, argc, argv, combinedopt,
Index: sys/amd64/conf/GENERIC
===================================================================
--- sys/amd64/conf/GENERIC (revision 291390)
+++ sys/amd64/conf/GENERIC (working copy)
@@ -366,3 +366,6 @@
# The crypto framework is required by IPSEC
device crypto # Required by IPSEC
+
+# Cam Debug Support
+options CAMDEBUG
Index: sys/cam/cam_ccb.h
===================================================================
--- sys/cam/cam_ccb.h (revision 291390)
+++ sys/cam/cam_ccb.h (working copy)
@@ -193,6 +193,11 @@
* Set SIM specific knob values.
*/
+ XPT_REPROBE_DEV = 0x1a,
+ /*
+ * Re-probe the sepcified SCSI device.
+ */
+
XPT_SMP_IO = 0x1b | XPT_FC_DEV_QUEUED,
/* Serial Management Protocol */
@@ -767,6 +772,7 @@
* Definitions for the asynchronous callback CCB fields.
*/
typedef enum {
+ AC_UNIT_REPROBE = 0x8000,/* Device reprobe user request */
AC_UNIT_ATTENTION = 0x4000,/* Device reported UNIT ATTENTION */
AC_ADVINFO_CHANGED = 0x2000,/* Advance info might have changes */
AC_CONTRACT = 0x1000,/* A contractual callback */
Index: sys/cam/cam_xpt.c
===================================================================
--- sys/cam/cam_xpt.c (revision 291390)
+++ sys/cam/cam_xpt.c (working copy)
@@ -433,6 +433,13 @@
return (EINVAL);
}
break;
+ case XPT_REPROBE_DEV:
+ if (inccb->ccb_h.target_id == CAM_TARGET_WILDCARD ||
+ inccb->ccb_h.target_lun == CAM_LUN_WILDCARD) {
+ xpt_release_bus(bus);
+ return (EINVAL);
+ }
+ break;
default:
break;
}
@@ -472,6 +479,29 @@
xpt_free_ccb(ccb);
break;
+ case XPT_REPROBE_DEV: {
+ struct cam_path path;
+
+ /*
+ * Compile a path using the bus, target, and lun the
+ * user passed in.
+ */
+ if (xpt_compile_path(&path, NULL,
+ inccb->ccb_h.path_id,
+ inccb->ccb_h.target_id,
+ inccb->ccb_h.target_lun) !=
+ CAM_REQ_CMP){
+ error = EINVAL;
+ break;
+ }
+
+ /*
+ * Notify interested parties
+ */
+ xpt_async(AC_UNIT_REPROBE, &path, NULL);
+ break;
+
+ }
case XPT_DEBUG: {
union ccb ccb;
@@ -717,7 +747,7 @@
}
xpt_unlock_buses();
break;
- }
+ }
default:
error = ENOTTY;
break;
Index: sys/cam/scsi/scsi_da.c
===================================================================
--- sys/cam/scsi/scsi_da.c (revision 291390)
+++ sys/cam/scsi/scsi_da.c (working copy)
@@ -1702,6 +1702,17 @@
}
break;
}
+ case AC_UNIT_REPROBE:
+ {
+ softc = (struct da_softc *)periph->softc;
+
+ xpt_print(periph->path,
+ "Re-probe requested\n");
+ softc->flags &= ~DA_FLAG_PROBED;
+ dareprobe(periph);
+
+ break;
+ }
case AC_UNIT_ATTENTION:
{
union ccb *ccb;
@@ -2240,7 +2251,7 @@
* would be to not attach the device on failure.
*/
xpt_register_async(AC_SENT_BDR | AC_BUS_RESET | AC_LOST_DEVICE |
- AC_ADVINFO_CHANGED | AC_SCSI_AEN | AC_UNIT_ATTENTION,
+ AC_ADVINFO_CHANGED | AC_SCSI_AEN | AC_UNIT_ATTENTION | AC_UNIT_REPROBE,
daasync, periph, periph->path);
/*
@@ -3244,12 +3255,11 @@
if (have_sense)
scsi_sense_print(
&done_ccb->csio);
- else {
+ else
xpt_print(periph->path,
"got CAM status %#x\n",
done_ccb->ccb_h.status);
- }
-
+
xpt_print(periph->path, "fatal error, "
"failed to attach to device\n");
--------------090907040403050103020002--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?565A7906.60106>