Date: Sat, 28 Nov 2015 18:10:04 -0600 From: Matthew Grooms <mgrooms@shrew.net> To: freebsd-current@freebsd.org Cc: mav@FreeBSD.org, trasz@FreeBSD.org Subject: Re: Resizing a zpool as a VMware ESXi guest ... Message-ID: <565A425C.7020101@shrew.net> In-Reply-To: <565906E8.9060005@shrew.net> References: <543841B8.4070007@shrew.net> <20141016081016.GA4670@brick.home> <5657F135.6080902@shrew.net> <56581F5A.4010009@digiware.nl> <56589C1A.1020702@shrew.net> <5658A764.5030508@shrew.net> <565906E8.9060005@shrew.net>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format.
--------------030409010104040509000407
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
On 11/27/2015 7:44 PM, Matthew Grooms wrote:
> I spent the day looking over the FreeBSD cam and scsi_da source code.
> After sprinkling a bunch of printf's around to see what code paths
> were being called, It's obvious that Edward was correct in assuming
> that ESXi doesn't return any 'Unit Attention' sense information in
> response to a 'Read Capacity' request. This kinda makes sense as ESXi
> emulates SCSI-2 disk devices and, as far as I can tell, the 0x2A/0x09
> ASC/ASCQ sense code that denotes 'Capacity Data Has Changed' wasn't
> defined until the SCSCI-3 spec. It's frustrating that the only way to
> get the scsci_da code to call reprobe() is by receiving a command from
> the device. Would something like this work? ...
>
> 1) Register a callback using xpt_register_async( daasync,
> AC_REPROBE_DEVICE, path ) that calls reprobe()
> 2) Implement a new IOCTL in cam_xpt that camcontrol can call with the
> bus:target:lun as the argument
> 3) have cam_xpt capture the IOCTL request and call xpt_async(
> AC_REPROBE_DEVICE, path ) as a result
>
> This way users would have the option of manually asking cam to
> communicate the new size to geom. The only option now is one or more
> reboots to gain access to the increased disk capacity. If this sounds
> like a reasonable approach, I'll take a stab at implementing it.
>
Here is a proof of concept patch. I'm a complete noob when it comes to
cam, scsi or freebsd kernel development for that matter, so I'm sure it
could have been done a better way. In any case, I added a new command to
camcontrol that allows you to specify a bus, target and lun as an
argument. For example ...
# camcontrol readcap da1 -h
Device Size: 32 G, Block Length: 512 bytes
# gpart show da1
=> 40 58720176 da1 GPT (28G)
40 58720176 1 freebsd-ufs (28G)
Note, I resized the VMDK disk in ESXi. The camcontrol output shows the
size as 32G but geom thinks its 28G.
# camcontrol devlist
<NECVMWar VMware IDE CDR10 1.00> at scbus1 target 0 lun 0 (cd0,pass0)
<VMware Virtual disk 1.0> at scbus2 target 0 lun 0 (pass1,da0)
<VMware Virtual disk 1.0> at scbus2 target 1 lun 0 (pass2,da1)
<FREEBSD CTLDISK 0001> at scbus3 target 0 lun 0 (da2,pass3)
# camcontrol reprobe 2:1:0
This generates an event that is captured by the scsci da device to
forces a reprobe. The kernel output looks almost identical to when the
'Unit Attention' sense data is received ...
Nov 28 17:46:13 iscsi-i kernel: (da1:mpt0:0:1:0): Re-probe requested
Nov 28 17:46:13 iscsi-i kernel: GEOM_PART: da1 was automatically resized.
Nov 28 17:46:13 iscsi-i kernel: Use `gpart commit da1` to save changes
or `gpart undo da1` to revert them.
Now that geom knows about the increased disk capacity, I can increase
the partition size and grow the fs ...
[root@iscsi-i /home/mgrooms]# gpart show da1
=> 40 67108784 da1 GPT (32G)
40 58720176 1 freebsd-ufs (28G)
58720216 8388608 - free - (4.0G)
# gpart resize -i 1 da1
da1p1 resized
# growfs da1p1
Device is mounted read-write; resizing will result in temporary write
suspension for /var/data1.
It's strongly recommended to make a backup before growing the file system.
OK to grow filesystem on /dev/da1p1, mounted on /var/data1, from 28GB to
32GB? [Yes/No] Yes
super-block backups (for fsck_ffs -b #) at:
58983232, 60265472, 61547712, 62829952, 64112192, 65394432, 66676672
# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/da0p3 18G 5.3G 12G 31% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/da1p1 31G 32M 28G 0% /var/data1
/dev/da2p1 15G 32M 14G 0% /var/data2
Sure would be nice to have something like this in the tree. It's really
a drag to have to reboot production VMs to increase disk capacity when
it could be easily avoided. I'm not sure what the correct IOCTL should
look like. Maybe CAMIOCOMMAND is a better way to go? If someone with
some experience with the cam/scsi subsystems was willing to give me some
direction I'd be willing to try and rewrite the patch in a way that
would be commit worthy. I just need some direction.
Thanks,
-Matthew
--------------030409010104040509000407
Content-Type: text/plain; charset=UTF-8;
name="cam-reprobe.diff"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="cam-reprobe.diff"
Index: lib/libcam/camlib.c
===================================================================
--- lib/libcam/camlib.c (revision 291390)
+++ lib/libcam/camlib.c (working copy)
@@ -752,3 +752,41 @@
bcopy(src, dst, sizeof(struct cam_device));
}
+
+/*
+ * Send a reprobe unit request for a given bus, target and lun
+ */
+int
+cam_reprobe_btl(path_id_t path_id, target_id_t target_id, lun_id_t target_lun)
+{
+ int fd;
+ char *func_name = "cam_reprobe_btl";
+ union ccb ccb;
+
+ if ((fd = open(XPT_DEVICE, O_RDWR)) < 0) {
+ snprintf(cam_errbuf, CAM_ERRBUF_SIZE,
+ "%s: couldn't open %s\n%s: %s", func_name, XPT_DEVICE,
+ func_name, strerror(errno));
+ return(-1);
+ }
+
+ /* Setup our request ccb */
+ bzero(&ccb.ccb_h, sizeof(struct ccb_hdr));
+ ccb.ccb_h.path_id = path_id;
+ ccb.ccb_h.target_id = target_id;
+ ccb.ccb_h.target_lun = target_lun;
+
+ /*
+ * Attempt to issue a unit reprobe request. This ioctl will fail if the
+ * kernel can't find a unit for the given path, target and lun.
+ */
+ if (ioctl(fd, CAMUNITREPROBE, &ccb) != 0) {
+ snprintf(cam_errbuf, CAM_ERRBUF_SIZE,
+ "%s: failed to issue unit reprobe request", func_name );
+ close(fd);
+ return(-1);
+ }
+
+ close(fd);
+ return 0;
+}
Index: lib/libcam/camlib.h
===================================================================
--- lib/libcam/camlib.h (revision 291390)
+++ lib/libcam/camlib.h (working copy)
@@ -146,6 +146,8 @@
struct cam_device *dst);
int cam_get_device(const char *path, char *dev_name,
int devnamelen, int *unit);
+int cam_reprobe_btl(path_id_t path_id, target_id_t target_id,
+ lun_id_t target_lun);
/*
* Buffer encoding/decoding routines, from the old SCSI library.
Index: sbin/camcontrol/camcontrol.c
===================================================================
--- sbin/camcontrol/camcontrol.c (revision 291390)
+++ sbin/camcontrol/camcontrol.c (working copy)
@@ -100,7 +100,8 @@
CAM_CMD_APM = 0x00000021,
CAM_CMD_AAM = 0x00000022,
CAM_CMD_ATTRIB = 0x00000023,
- CAM_CMD_OPCODES = 0x00000024
+ CAM_CMD_OPCODES = 0x00000024,
+ CAM_CMD_REPROBE = 0x00000025
} cam_cmdmask;
typedef enum {
@@ -193,6 +194,7 @@
#endif /* MINIMALISTIC */
{"rescan", CAM_CMD_RESCAN, CAM_ARG_NONE, NULL},
{"reset", CAM_CMD_RESET, CAM_ARG_NONE, NULL},
+ {"reprobe", CAM_CMD_REPROBE, CAM_ARG_NONE, NULL},
#ifndef MINIMALISTIC
{"cmd", CAM_CMD_SCSI_CMD, CAM_ARG_NONE, scsicmd_opts},
{"command", CAM_CMD_SCSI_CMD, CAM_ARG_NONE, scsicmd_opts},
@@ -3127,6 +3129,34 @@
}
static int
+doreprobe(int argc, char **argv)
+{
+ static const char must[] =
+ "you must specify a bus:target:lun to reprobe";
+ int rv;
+ path_id_t bus = CAM_BUS_WILDCARD;
+ target_id_t target = CAM_TARGET_WILDCARD;
+ lun_id_t lun = CAM_LUN_WILDCARD;
+ char *tstr;
+
+ if (argc < 3) {
+ warnx(must);
+ return(1);
+ }
+
+ tstr = argv[optind];
+ while (isspace(*tstr) && (*tstr != '\0'))
+ tstr++;
+ rv = parse_btl(argv[optind], &bus, &target, &lun, &arglist);
+ if (rv != 3) {
+ warnx(must);
+ return(1);
+ }
+
+ return(cam_reprobe_btl(bus, target, lun));
+}
+
+static int
rescan_or_reset_bus(path_id_t bus, int rescan)
{
union ccb ccb, matchccb;
@@ -8685,6 +8715,7 @@
#endif /* MINIMALISTIC */
" camcontrol rescan <all | bus[:target:lun]>\n"
" camcontrol reset <all | bus[:target:lun]>\n"
+" camcontrol reprobe bus:target:lun\n"
#ifndef MINIMALISTIC
" camcontrol defects [dev_id][generic args] <-f format> [-P][-G]\n"
" [-q][-s][-S offset][-X]\n"
@@ -9053,6 +9084,7 @@
*/
if ((cmdlist == CAM_CMD_RESCAN)
|| (cmdlist == CAM_CMD_RESET)
+ || (cmdlist == CAM_CMD_REPROBE)
|| (cmdlist == CAM_CMD_DEVTREE)
|| (cmdlist == CAM_CMD_USAGE)
|| (cmdlist == CAM_CMD_DEBUG))
@@ -9204,6 +9236,9 @@
case CAM_CMD_RESET:
error = dorescan_or_reset(argc, argv, 0);
break;
+ case CAM_CMD_REPROBE:
+ error = doreprobe(argc, argv);
+ break;
#ifndef MINIMALISTIC
case CAM_CMD_READ_DEFECTS:
error = readdefects(cam_dev, argc, argv, combinedopt,
Index: sys/amd64/conf/GENERIC
===================================================================
--- sys/amd64/conf/GENERIC (revision 291390)
+++ sys/amd64/conf/GENERIC (working copy)
@@ -366,3 +366,6 @@
# The crypto framework is required by IPSEC
device crypto # Required by IPSEC
+
+# Cam Debug Support
+options CAMDEBUG
Index: sys/cam/cam_ccb.h
===================================================================
--- sys/cam/cam_ccb.h (revision 291390)
+++ sys/cam/cam_ccb.h (working copy)
@@ -767,6 +767,7 @@
* Definitions for the asynchronous callback CCB fields.
*/
typedef enum {
+ AC_UNIT_REPROBE = 0x8000,/* Device reprobe user request */
AC_UNIT_ATTENTION = 0x4000,/* Device reported UNIT ATTENTION */
AC_ADVINFO_CHANGED = 0x2000,/* Advance info might have changes */
AC_CONTRACT = 0x1000,/* A contractual callback */
Index: sys/cam/cam_xpt.c
===================================================================
--- sys/cam/cam_xpt.c (revision 291390)
+++ sys/cam/cam_xpt.c (working copy)
@@ -717,7 +717,41 @@
}
xpt_unlock_buses();
break;
+ }
+ case CAMUNITREPROBE: {
+ union ccb *inccb;
+ struct cam_path path;
+
+ inccb = (union ccb *)addr;
+
+ /*
+ * Make sure a specific bus, target and lun were specified.
+ */
+ if (inccb->ccb_h.path_id == CAM_BUS_WILDCARD ||
+ inccb->ccb_h.target_id == CAM_TARGET_WILDCARD ||
+ inccb->ccb_h.target_lun == CAM_LUN_WILDCARD) {
+ error = EINVAL;
+ break;
}
+
+ /*
+ * Compile a path using the bus, target, and lun the user passed in.
+ */
+ if (xpt_compile_path(&path, NULL,
+ inccb->ccb_h.path_id,
+ inccb->ccb_h.target_id,
+ inccb->ccb_h.target_lun) !=
+ CAM_REQ_CMP){
+ error = EINVAL;
+ break;
+ }
+
+ /*
+ * Notify interested parties
+ */
+ xpt_async(AC_UNIT_REPROBE, &path, NULL);
+ break;
+ }
default:
error = ENOTTY;
break;
Index: sys/cam/scsi/scsi_da.c
===================================================================
--- sys/cam/scsi/scsi_da.c (revision 291390)
+++ sys/cam/scsi/scsi_da.c (working copy)
@@ -1702,6 +1702,17 @@
}
break;
}
+ case AC_UNIT_REPROBE:
+ {
+ softc = (struct da_softc *)periph->softc;
+
+ xpt_print(periph->path,
+ "Re-probe requested\n");
+ softc->flags &= ~DA_FLAG_PROBED;
+ dareprobe(periph);
+
+ break;
+ }
case AC_UNIT_ATTENTION:
{
union ccb *ccb;
@@ -2240,7 +2251,7 @@
* would be to not attach the device on failure.
*/
xpt_register_async(AC_SENT_BDR | AC_BUS_RESET | AC_LOST_DEVICE |
- AC_ADVINFO_CHANGED | AC_SCSI_AEN | AC_UNIT_ATTENTION,
+ AC_ADVINFO_CHANGED | AC_SCSI_AEN | AC_UNIT_ATTENTION | AC_UNIT_REPROBE,
daasync, periph, periph->path);
/*
@@ -3244,12 +3255,11 @@
if (have_sense)
scsi_sense_print(
&done_ccb->csio);
- else {
+ else
xpt_print(periph->path,
"got CAM status %#x\n",
done_ccb->ccb_h.status);
- }
-
+
xpt_print(periph->path, "fatal error, "
"failed to attach to device\n");
Index: sys/cam/scsi/scsi_pass.h
===================================================================
--- sys/cam/scsi/scsi_pass.h (revision 291390)
+++ sys/cam/scsi/scsi_pass.h (working copy)
@@ -38,5 +38,6 @@
*/
#define CAMIOCOMMAND _IOWR(CAM_VERSION, 2, union ccb)
#define CAMGETPASSTHRU _IOWR(CAM_VERSION, 3, union ccb)
+#define CAMUNITREPROBE _IOWR(CAM_VERSION, 4, union ccb)
#endif
--------------030409010104040509000407--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?565A425C.7020101>