Date: Wed, 20 Jun 2012 22:29:25 -0600 From: "Kenneth D. Merry" <ken@freebsd.org> To: current@freebsd.org Subject: minor GEOM disk API change coming Message-ID: <20120621042925.GA44903@nargothrond.kdm.org>
next in thread | raw e-mail | index | archive | help
--TB36FDmn/VVEgNH/
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Hi folks,
I have attached some patches that fix an object lifetime issue between CAM
and GEOM.
Fixing the bug required adding a callback to the GEOM disk code, and adding
a callback that a GEOM class can register to get notified when a provider
is destroyed.
The probable commit message is below.
If I don't hear any objections, I will commit it on Friday, June 22nd.
============
Fix a bug which causes a panic in daopen(). The panic is caused by
a da(4) instance going away while GEOM is still probing it.
In this case, the GEOM disk class instance has been created by
disk_create(), and the taste of the disk is queued in the GEOM
event queue.
While that event is queued, the da(4) instance goes away. When the
open call comes into the da(4) driver, it dereferences the freed
(but non-NULL) peripheral pointer provided by GEOM, which results
in a panic.
The solution is to add a callback to the GEOM disk code that is
called when all of its resources are cleaned up. This is
implemented inside GEOM by adding an optional callback that is
called when all consumers have detached from a provider, and the
provider is about to be deleted.
scsi_cd.c,
scsi_da.c: In the register routine for the cd(4) and da(4)
routines, acquire a reference to the CAM peripheral
instance just before we call disk_create().
Use the new GEOM disk d_gone() callback to register
a callback (dadiskgonecb()/cddiskgonecb()) that
decrements the peripheral reference count once GEOM
has finished cleaning up its resources.
In the cd(4) driver, clean up open and close
behavior slightly. GEOM makes sure we only get one
open() and one close call, so there is no need to
set an open flag and decrement the reference count
if we are not the first open.
In the cd(4) driver, use cam_periph_release_locked()
in a couple of error scenarios to avoid extra mutex
calls.
geom.h: Add a new, optional, providergone callback that
is called when a provider is about to be deleted.
geom_disk.h: Add a new d_gone() callback to the GEOM disk
interface.
Bump the DISK_VERSION to version 2. This probably
should have been done after a couple of previous
changes, especially the addition of the d_getattr()
callback.
geom_disk.c: Add a providergone callback for the disk class,
g_disk_providergone(), that calls the user's
d_gone() callback if it exists.
Bump the DISK_VERSION to 2.
geom_subr.c: In g_destroy_provider(), call the providergone
callback if it has been provided.
In g_new_geomf(), propagate the class's
providergone callback to the new geom instance.
disk.9: Update the disk(9) man page to include information
on the new d_gone() callback, as well as the
previously added d_getattr() callback, d_descr
field, and HBA PCI ID fields.
============
Ken
--
Kenneth Merry
ken@FreeBSD.ORG
--TB36FDmn/VVEgNH/
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="cam_geom_callback.20120620.1.txt"
==== //depot/users/kenm/FreeBSD-test/share/man/man9/disk.9#1 - /usr/home/kenm/perforce4/kenm/FreeBSD-test/share/man/man9/disk.9 ====
*** /tmp/tmp.81866.21 Wed Jun 20 22:19:20 2012
--- /usr/home/kenm/perforce4/kenm/FreeBSD-test/share/man/man9/disk.9 Wed Jun 20 21:30:45 2012
***************
*** 145,150 ****
--- 145,160 ----
.Xr dumpon 8 ,
this function is invoked from a very restricted system state after a
kernel panic to record a copy of the system RAM to the disk.
+ .It Vt "disk_getattr_t *" Va d_getattr
+ Optional: if this method is provided, it gives the disk driver the
+ opportunity to override the default GEOM response to BIO_GETATTR requests.
+ This function should return -1 if the attribute is not handled, 0 if the
+ attribute is handled, or an errno to be passed to g_io_deliver().
+ .It Vt "disk_gone_t *" Va d_gone
+ Optional: if this method is provided, it will be called after disk_gone()
+ is called, once GEOM has finished its cleanup process.
+ Once this callback is called, it is safe for the disk driver to free all of
+ its resources, as it will not be receiving further calls from GEOM.
.El
.Ss Mandatory Media Properties
The following fields identify the size and granularity of the disk device.
***************
*** 180,186 ****
.Pa src/sys/geom/notes
for details.
.It Vt char Va d_ident[DISK_IDENT_SIZE]
! This field can and should be used to store disk's serial number.
.El
.Ss Driver Private Data
This field may be used by the device driver to store a pointer to
--- 190,212 ----
.Pa src/sys/geom/notes
for details.
.It Vt char Va d_ident[DISK_IDENT_SIZE]
! This field can and should be used to store disk's serial number if the
! d_getattr method described above isn't implemented, or if it does not
! support the GEOM::ident attribute.
! .It Vt char Va d_descr[DISK_IDENT_SIZE]
! This field can be used to store the disk vendor and product description.
! .It Vt uint16_t Va d_hba_vendor
! This field can be used to store the PCI vendor ID for the HBA connected to
! the disk.
! .It Vt uint16_t Va d_hba_device
! This field can be used to store the PCI device ID for the HBA connected to
! the disk.
! .It Vt uint16_t Va d_hba_subvendor
! This field can be used to store the PCI subvendor ID for the HBA connected to
! the disk.
! .It Vt uint16_t Va d_hba_subdevice
! This field can be used to store the PCI subdevice ID for the HBA connected to
! the disk.
.El
.Ss Driver Private Data
This field may be used by the device driver to store a pointer to
==== //depot/users/kenm/FreeBSD-test/sys/cam/scsi/scsi_cd.c#7 - /usr/home/kenm/perforce4/kenm/FreeBSD-test/sys/cam/scsi/scsi_cd.c ====
*** /tmp/tmp.81866.119 Wed Jun 20 22:19:20 2012
--- /usr/home/kenm/perforce4/kenm/FreeBSD-test/sys/cam/scsi/scsi_cd.c Wed Jun 20 21:30:48 2012
***************
*** 103,110 ****
CD_FLAG_RETRY_UA = 0x0200,
CD_FLAG_VALID_MEDIA = 0x0400,
CD_FLAG_VALID_TOC = 0x0800,
! CD_FLAG_SCTX_INIT = 0x1000,
! CD_FLAG_OPEN = 0x2000
} cd_flags;
typedef enum {
--- 103,109 ----
CD_FLAG_RETRY_UA = 0x0200,
CD_FLAG_VALID_MEDIA = 0x0400,
CD_FLAG_VALID_TOC = 0x0800,
! CD_FLAG_SCTX_INIT = 0x1000
} cd_flags;
typedef enum {
***************
*** 358,363 ****
--- 357,376 ----
}
}
+ /*
+ * Callback from GEOM, called when it has finished cleaning up its
+ * resources.
+ */
+ static void
+ cddiskgonecb(struct disk *dp)
+ {
+ struct cam_periph *periph;
+
+ periph = (struct cam_periph *)dp->d_drv1;
+
+ cam_periph_release(periph);
+ }
+
static void
cdoninvalidate(struct cam_periph *periph)
{
***************
*** 389,395 ****
camq_remove(&softc->changer->devq, softc->pinfo.index);
disk_gone(softc->disk);
! xpt_print(periph->path, "lost device\n");
}
static void
--- 402,408 ----
camq_remove(&softc->changer->devq, softc->pinfo.index);
disk_gone(softc->disk);
! xpt_print(periph->path, "lost device, %d refs\n", periph->refcount);
}
static void
***************
*** 726,731 ****
--- 739,745 ----
softc->disk->d_open = cdopen;
softc->disk->d_close = cdclose;
softc->disk->d_strategy = cdstrategy;
+ softc->disk->d_gone = cddiskgonecb;
softc->disk->d_ioctl = cdioctl;
softc->disk->d_name = "cd";
cam_strvis(softc->disk->d_descr, cgd->inq_data.vendor,
***************
*** 747,752 ****
--- 761,779 ----
softc->disk->d_hba_device = cpi.hba_device;
softc->disk->d_hba_subvendor = cpi.hba_subvendor;
softc->disk->d_hba_subdevice = cpi.hba_subdevice;
+
+ /*
+ * Acquire a reference to the periph before we register with GEOM.
+ * We'll release this reference once GEOM calls us back (via
+ * dadiskgonecb()) telling us that our provider has been freed.
+ */
+ if (cam_periph_acquire(periph) != CAM_REQ_CMP) {
+ xpt_print(periph->path, "%s: lost periph during "
+ "registration!\n", __func__);
+ cam_periph_lock(periph);
+ return (CAM_REQ_CMP_ERR);
+ }
+
disk_create(softc->disk, DISK_VERSION);
cam_periph_lock(periph);
***************
*** 1000,1013 ****
cam_periph_lock(periph);
if (softc->flags & CD_FLAG_INVALID) {
cam_periph_unlock(periph);
- cam_periph_release(periph);
return(ENXIO);
}
if ((error = cam_periph_hold(periph, PRIBIO | PCATCH)) != 0) {
cam_periph_unlock(periph);
- cam_periph_release(periph);
return (error);
}
--- 1027,1040 ----
cam_periph_lock(periph);
if (softc->flags & CD_FLAG_INVALID) {
+ cam_periph_release_locked(periph);
cam_periph_unlock(periph);
return(ENXIO);
}
if ((error = cam_periph_hold(periph, PRIBIO | PCATCH)) != 0) {
+ cam_periph_release_locked(periph);
cam_periph_unlock(periph);
return (error);
}
***************
*** 1024,1037 ****
CAM_DEBUG(periph->path, CAM_DEBUG_TRACE, ("leaving cdopen\n"));
cam_periph_unhold(periph);
! /* Closes aren't symmetrical with opens, so fix up the refcounting. */
! if ((softc->flags & CD_FLAG_OPEN) == 0) {
! softc->flags |= CD_FLAG_OPEN;
! cam_periph_unlock(periph);
! } else {
! cam_periph_unlock(periph);
! cam_periph_release(periph);
! }
return (0);
}
--- 1051,1057 ----
CAM_DEBUG(periph->path, CAM_DEBUG_TRACE, ("leaving cdopen\n"));
cam_periph_unhold(periph);
! cam_periph_unlock(periph);
return (0);
}
***************
*** 1070,1080 ****
/*
* We'll check the media and toc again at the next open().
*/
! softc->flags &= ~(CD_FLAG_VALID_MEDIA|CD_FLAG_VALID_TOC|CD_FLAG_OPEN);
cam_periph_unhold(periph);
cam_periph_unlock(periph);
- cam_periph_release(periph);
return (0);
}
--- 1090,1100 ----
/*
* We'll check the media and toc again at the next open().
*/
! softc->flags &= ~(CD_FLAG_VALID_MEDIA|CD_FLAG_VALID_TOC);
cam_periph_unhold(periph);
+ cam_periph_release_locked(periph);
cam_periph_unlock(periph);
return (0);
}
==== //depot/users/kenm/FreeBSD-test/sys/cam/scsi/scsi_da.c#12 - /usr/home/kenm/perforce4/kenm/FreeBSD-test/sys/cam/scsi/scsi_da.c ====
*** /tmp/tmp.81866.135 Wed Jun 20 22:19:20 2012
--- /usr/home/kenm/perforce4/kenm/FreeBSD-test/sys/cam/scsi/scsi_da.c Wed Jun 20 21:32:11 2012
***************
*** 1233,1238 ****
--- 1233,1252 ----
}
}
+ /*
+ * Callback from GEOM, called when it has finished cleaning up its
+ * resources.
+ */
+ static void
+ dadiskgonecb(struct disk *dp)
+ {
+ struct cam_periph *periph;
+
+ periph = (struct cam_periph *)dp->d_drv1;
+
+ cam_periph_release(periph);
+ }
+
static void
daoninvalidate(struct cam_periph *periph)
{
***************
*** 1255,1261 ****
--- 1269,1280 ----
bioq_flush(&softc->bio_queue, NULL, ENXIO);
bioq_flush(&softc->delete_queue, NULL, ENXIO);
+ /*
+ * Tell GEOM that we've gone away, we'll get a callback when it is
+ * done cleaning up its resources.
+ */
disk_gone(softc->disk);
+
xpt_print(periph->path, "lost device - %d outstanding, %d refs\n",
softc->outstanding_cmds, periph->refcount);
}
***************
*** 1633,1638 ****
--- 1652,1658 ----
softc->disk->d_strategy = dastrategy;
softc->disk->d_dump = dadump;
softc->disk->d_getattr = dagetattr;
+ softc->disk->d_gone = dadiskgonecb;
softc->disk->d_name = "da";
softc->disk->d_drv1 = periph;
if (cpi.maxio == 0)
***************
*** 1655,1660 ****
--- 1675,1693 ----
softc->disk->d_hba_device = cpi.hba_device;
softc->disk->d_hba_subvendor = cpi.hba_subvendor;
softc->disk->d_hba_subdevice = cpi.hba_subdevice;
+
+ /*
+ * Acquire a reference to the periph before we register with GEOM.
+ * We'll release this reference once GEOM calls us back (via
+ * dadiskgonecb()) telling us that our provider has been freed.
+ */
+ if (cam_periph_acquire(periph) != CAM_REQ_CMP) {
+ xpt_print(periph->path, "%s: lost periph during "
+ "registration!\n", __func__);
+ mtx_lock(periph->sim->mtx);
+ return (CAM_REQ_CMP_ERR);
+ }
+
disk_create(softc->disk, DISK_VERSION);
mtx_lock(periph->sim->mtx);
==== //depot/users/kenm/FreeBSD-test/sys/dev/xen/blkfront/blkfront.c#8 - /usr/home/kenm/perforce4/kenm/FreeBSD-test/sys/dev/xen/blkfront/blkfront.c ====
*** /tmp/tmp.81866.147 Wed Jun 20 22:19:20 2012
--- /usr/home/kenm/perforce4/kenm/FreeBSD-test/sys/dev/xen/blkfront/blkfront.c Wed Jun 20 21:30:01 2012
***************
*** 230,236 ****
sc->xb_disk->d_mediasize = sectors * sector_size;
sc->xb_disk->d_maxsize = sc->max_request_size;
sc->xb_disk->d_flags = 0;
! disk_create(sc->xb_disk, DISK_VERSION_00);
return error;
}
--- 230,236 ----
sc->xb_disk->d_mediasize = sectors * sector_size;
sc->xb_disk->d_maxsize = sc->max_request_size;
sc->xb_disk->d_flags = 0;
! disk_create(sc->xb_disk, DISK_VERSION);
return error;
}
==== //depot/users/kenm/FreeBSD-test/sys/geom/geom.h#2 - /usr/home/kenm/perforce4/kenm/FreeBSD-test/sys/geom/geom.h ====
*** /tmp/tmp.81866.158 Wed Jun 20 22:19:20 2012
--- /usr/home/kenm/perforce4/kenm/FreeBSD-test/sys/geom/geom.h Wed Jun 20 21:30:45 2012
***************
*** 76,81 ****
--- 76,82 ----
typedef void g_start_t (struct bio *);
typedef void g_spoiled_t (struct g_consumer *);
typedef void g_attrchanged_t (struct g_consumer *, const char *attr);
+ typedef void g_provgone_t (struct g_provider *);
typedef void g_dumpconf_t (struct sbuf *, const char *indent, struct g_geom *,
struct g_consumer *, struct g_provider *);
***************
*** 102,107 ****
--- 103,109 ----
g_start_t *start;
g_spoiled_t *spoiled;
g_attrchanged_t *attrchanged;
+ g_provgone_t *providergone;
g_dumpconf_t *dumpconf;
g_access_t *access;
g_orphan_t *orphan;
***************
*** 133,138 ****
--- 135,141 ----
g_start_t *start;
g_spoiled_t *spoiled;
g_attrchanged_t *attrchanged;
+ g_provgone_t *providergone;
g_dumpconf_t *dumpconf;
g_access_t *access;
g_orphan_t *orphan;
==== //depot/users/kenm/FreeBSD-test/sys/geom/geom_disk.c#5 - /usr/home/kenm/perforce4/kenm/FreeBSD-test/sys/geom/geom_disk.c ====
*** /tmp/tmp.81866.209 Wed Jun 20 22:19:20 2012
--- /usr/home/kenm/perforce4/kenm/FreeBSD-test/sys/geom/geom_disk.c Wed Jun 20 21:30:46 2012
***************
*** 75,80 ****
--- 75,81 ----
static g_start_t g_disk_start;
static g_ioctl_t g_disk_ioctl;
static g_dumpconf_t g_disk_dumpconf;
+ static g_provgone_t g_disk_providergone;
static struct g_class g_disk_class = {
.name = "DISK",
***************
*** 84,89 ****
--- 85,91 ----
.start = g_disk_start,
.access = g_disk_access,
.ioctl = g_disk_ioctl,
+ .providergone = g_disk_providergone,
.dumpconf = g_disk_dumpconf,
};
***************
*** 487,492 ****
--- 489,513 ----
g_error_provider(pp, 0);
}
+ /*
+ * We get this callback after all of the consumers have gone away, and just
+ * before the provider is freed. If the disk driver provided a d_gone
+ * callback, let them know that it is okay to free resources -- they won't
+ * be getting any more accesses from GEOM.
+ */
+ static void
+ g_disk_providergone(struct g_provider *pp)
+ {
+ struct disk *dp;
+ struct g_disk_softc *sc;
+
+ sc = (struct g_disk_softc *)pp->geom->softc;
+ dp = sc->dp;
+
+ if (dp->d_gone != NULL)
+ dp->d_gone(dp);
+ }
+
static void
g_disk_destroy(void *ptr, int flag)
{
***************
*** 550,556 ****
disk_create(struct disk *dp, int version)
{
! if (version != DISK_VERSION_00 && version != DISK_VERSION_01) {
printf("WARNING: Attempt to add disk %s%d %s",
dp->d_name, dp->d_unit,
" using incompatible ABI version of disk(9)\n");
--- 571,577 ----
disk_create(struct disk *dp, int version)
{
! if (version != DISK_VERSION_02) {
printf("WARNING: Attempt to add disk %s%d %s",
dp->d_name, dp->d_unit,
" using incompatible ABI version of disk(9)\n");
==== //depot/users/kenm/FreeBSD-test/sys/geom/geom_disk.h#2 - /usr/home/kenm/perforce4/kenm/FreeBSD-test/sys/geom/geom_disk.h ====
*** /tmp/tmp.81866.251 Wed Jun 20 22:19:20 2012
--- /usr/home/kenm/perforce4/kenm/FreeBSD-test/sys/geom/geom_disk.h Wed Jun 20 21:30:46 2012
***************
*** 50,55 ****
--- 50,56 ----
typedef int disk_close_t(struct disk *);
typedef void disk_strategy_t(struct bio *bp);
typedef int disk_getattr_t(struct bio *bp);
+ typedef void disk_gone_t(struct disk *);
typedef int disk_ioctl_t(struct disk *, u_long cmd, void *data,
int fflag, struct thread *td);
/* NB: disk_ioctl_t SHALL be cast'able to d_ioctl_t */
***************
*** 77,82 ****
--- 78,84 ----
disk_ioctl_t *d_ioctl;
dumper_t *d_dump;
disk_getattr_t *d_getattr;
+ disk_gone_t *d_gone;
/* Info fields from driver to geom_disk.c. Valid when open */
u_int d_sectorsize;
***************
*** 110,116 ****
#define DISK_VERSION_00 0x58561059
#define DISK_VERSION_01 0x5856105a
! #define DISK_VERSION DISK_VERSION_01
#endif /* _KERNEL */
#endif /* _GEOM_GEOM_DISK_H_ */
--- 112,119 ----
#define DISK_VERSION_00 0x58561059
#define DISK_VERSION_01 0x5856105a
! #define DISK_VERSION_02 0x5856105b
! #define DISK_VERSION DISK_VERSION_02
#endif /* _KERNEL */
#endif /* _GEOM_GEOM_DISK_H_ */
==== //depot/users/kenm/FreeBSD-test/sys/geom/geom_subr.c#2 - /usr/home/kenm/perforce4/kenm/FreeBSD-test/sys/geom/geom_subr.c ====
*** /tmp/tmp.81866.345 Wed Jun 20 22:19:20 2012
--- /usr/home/kenm/perforce4/kenm/FreeBSD-test/sys/geom/geom_subr.c Wed Jun 20 21:32:22 2012
***************
*** 351,356 ****
--- 351,357 ----
gp->start = mp->start;
gp->spoiled = mp->spoiled;
gp->attrchanged = mp->attrchanged;
+ gp->providergone = mp->providergone;
gp->dumpconf = mp->dumpconf;
gp->access = mp->access;
gp->orphan = mp->orphan;
***************
*** 634,639 ****
--- 635,647 ----
LIST_REMOVE(pp, provider);
gp = pp->geom;
devstat_remove_entry(pp->stat);
+ /*
+ * If a callback was provided, send notification that the provider
+ * is now gone.
+ */
+ if (gp->providergone != NULL)
+ gp->providergone(pp);
+
g_free(pp);
if ((gp->flags & G_GEOM_WITHER))
g_do_wither();
--TB36FDmn/VVEgNH/--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120621042925.GA44903>