Date: Mon, 17 Sep 2012 19:23:01 +0000 (UTC) From: Jim Harris <jimharris@FreeBSD.org> To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r240616 - in head/sys/dev: nvd nvme Message-ID: <201209171923.q8HJN1IZ073080@svn.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: jimharris Date: Mon Sep 17 19:23:01 2012 New Revision: 240616 URL: http://svn.freebsd.org/changeset/base/240616 Log: This is the first of several commits which will add NVM Express (NVMe) support to FreeBSD. A full description of the overall functionality being added is below. nvmexpress.org defines NVM Express as "an optimized register interface, command set and feature set fo PCI Express (PCIe)-based Solid-State Drives (SSDs)." This commit adds nvme(4) and nvd(4) driver source code and Makefiles to the tree. Full NVMe functionality description: Add nvme(4) and nvd(4) drivers and nvmecontrol(8) for NVM Express (NVMe) device support. There will continue to be ongoing work on NVM Express support, but there is more than enough to allow for evaluation of pre-production NVM Express devices as well as soliciting feedback. Questions and feedback are welcome. nvme(4) implements NVMe hardware abstraction and is a provider of NVMe namespaces. The closest equivalent of an NVMe namespace is a SCSI LUN. nvd(4) is an NVMe consumer, surfacing NVMe namespaces as GEOM disks. nvmecontrol(8) is used for NVMe configuration and management. The following are currently supported: nvme(4) - full mandatory NVM command set support - per-CPU IO queues (enabled by default but configurable) - per-queue sysctls for statistics and full command/completion queue dumps for debugging - registration API for NVMe namespace consumers - I/O error handling (except for timeoutsee below) - compilation switches for support back to stable-7 nvd(4) - BIO_DELETE and BIO_FLUSH (if supported by controller) - proper BIO_ORDERED handling nvmecontrol(8) - devlist: list NVMe controllers and their namespaces - identify: display controller or namespace identify data in human-readable or hex format - perftest: quick and dirty performance test to measure raw performance of NVMe device without userspace/physio/GEOM overhead The following are still work in progress and will be completed over the next 3-6 months in rough priority order: - complete man pages - firmware download and activation - asynchronous error requests - command timeout error handling - controller resets - nvmecontrol(8) log page retrieval This has been primarily tested on amd64, with light testing on i386. I would be happy to provide assistance to anyone interested in porting this to other architectures, but am not currently planning to do this work myself. Big-endian and dmamap sync for command/completion queues are the main areas that would need to be addressed. The nvme(4) driver currently has references to Chatham, which is an Intel-developed prototype board which is not fully spec compliant. These references will all be removed over time. Sponsored by: Intel Contributions from: Joe Golio/EMC <joseph dot golio at emc dot com> Added: head/sys/dev/nvd/ head/sys/dev/nvd/nvd.c (contents, props changed) head/sys/dev/nvme/ head/sys/dev/nvme/nvme.c (contents, props changed) head/sys/dev/nvme/nvme.h (contents, props changed) head/sys/dev/nvme/nvme_ctrlr.c (contents, props changed) head/sys/dev/nvme/nvme_ctrlr_cmd.c (contents, props changed) head/sys/dev/nvme/nvme_ns.c (contents, props changed) head/sys/dev/nvme/nvme_ns_cmd.c (contents, props changed) head/sys/dev/nvme/nvme_private.h (contents, props changed) head/sys/dev/nvme/nvme_qpair.c (contents, props changed) head/sys/dev/nvme/nvme_sysctl.c (contents, props changed) head/sys/dev/nvme/nvme_test.c (contents, props changed) head/sys/dev/nvme/nvme_uio.c (contents, props changed) Added: head/sys/dev/nvd/nvd.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sys/dev/nvd/nvd.c Mon Sep 17 19:23:01 2012 (r240616) @@ -0,0 +1,318 @@ +/*- + * Copyright (C) 2012 Intel Corporation + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <sys/cdefs.h> +__FBSDID("$FreeBSD$"); + +#include <sys/param.h> +#include <sys/bio.h> +#include <sys/kernel.h> +#include <sys/malloc.h> +#include <sys/module.h> +#include <sys/systm.h> +#include <sys/taskqueue.h> + +#include <geom/geom.h> +#include <geom/geom_disk.h> + +#include <dev/nvme/nvme.h> + +struct nvd_disk; + +static disk_ioctl_t nvd_ioctl; +static disk_strategy_t nvd_strategy; + +static void create_geom_disk(void *, struct nvme_namespace *ns); +static void destroy_geom_disk(struct nvd_disk *ndisk); + +static int nvd_load(void); +static void nvd_unload(void); + +MALLOC_DEFINE(M_NVD, "nvd", "nvd(4) allocations"); + +struct nvme_consumer *consumer_handle; + +struct nvd_disk { + + struct bio_queue_head bioq; + struct task bioqtask; + struct mtx bioqlock; + + struct disk *disk; + struct taskqueue *tq; + struct nvme_namespace *ns; + + uint32_t cur_depth; + + TAILQ_ENTRY(nvd_disk) tailq; +}; + +TAILQ_HEAD(, nvd_disk) nvd_head; + +static int nvd_modevent(module_t mod, int type, void *arg) +{ + int error = 0; + + switch (type) { + case MOD_LOAD: + error = nvd_load(); + break; + case MOD_UNLOAD: + nvd_unload(); + break; + default: + break; + } + + return (error); +} + +moduledata_t nvd_mod = { + "nvd", + (modeventhand_t)nvd_modevent, + 0 +}; + +DECLARE_MODULE(nvd, nvd_mod, SI_SUB_DRIVERS, SI_ORDER_ANY); +MODULE_VERSION(nvd, 1); +MODULE_DEPEND(nvd, nvme, 1, 1, 1); + +static int +nvd_load() +{ + + TAILQ_INIT(&nvd_head); + consumer_handle = nvme_register_consumer(create_geom_disk, NULL); + + return (consumer_handle != NULL ? 0 : -1); +} + +static void +nvd_unload() +{ + struct nvd_disk *nvd; + + while (!TAILQ_EMPTY(&nvd_head)) { + nvd = TAILQ_FIRST(&nvd_head); + TAILQ_REMOVE(&nvd_head, nvd, tailq); + destroy_geom_disk(nvd); + free(nvd, M_NVD); + } + + nvme_unregister_consumer(consumer_handle); +} + +static void +nvd_strategy(struct bio *bp) +{ + struct nvd_disk *ndisk; + + ndisk = (struct nvd_disk *)bp->bio_disk->d_drv1; + + mtx_lock(&ndisk->bioqlock); + bioq_insert_tail(&ndisk->bioq, bp); + mtx_unlock(&ndisk->bioqlock); + taskqueue_enqueue(ndisk->tq, &ndisk->bioqtask); +} + +static int +nvd_ioctl(struct disk *ndisk, u_long cmd, void *data, int fflag, + struct thread *td) +{ + int ret = 0; + + switch (cmd) { + default: + ret = EIO; + } + + return (ret); +} + +static void +nvd_done(void *arg, const struct nvme_completion *status) +{ + struct bio *bp; + struct nvd_disk *ndisk; + + bp = (struct bio *)arg; + + ndisk = bp->bio_disk->d_drv1; + + if (atomic_fetchadd_int(&ndisk->cur_depth, -1) == NVME_QD) + taskqueue_enqueue(ndisk->tq, &ndisk->bioqtask); + + /* + * TODO: add more extensive translation of NVMe status codes + * to different bio error codes (i.e. EIO, EINVAL, etc.) + */ + if (status->sf_sc || status->sf_sct) { + bp->bio_error = EIO; + bp->bio_flags |= BIO_ERROR; + bp->bio_resid = bp->bio_bcount; + } else + bp->bio_resid = 0; + + biodone(bp); +} + +static void +nvd_bioq_process(void *arg, int pending) +{ + struct nvd_disk *ndisk = arg; + struct bio *bp; + int err; + + for (;;) { + if (atomic_load_acq_int(&ndisk->cur_depth) >= NVME_QD) + break; + + mtx_lock(&ndisk->bioqlock); + bp = bioq_takefirst(&ndisk->bioq); + mtx_unlock(&ndisk->bioqlock); + if (bp == NULL) + break; + +#ifdef BIO_ORDERED + /* + * BIO_ORDERED flag dictates that all outstanding bios + * must be completed before processing the bio with + * BIO_ORDERED flag set. + */ + if (bp->bio_flags & BIO_ORDERED) { + while (ndisk->cur_depth > 0) { + pause("nvd flush", 1); + } + } +#endif + + bp->bio_driver1 = NULL; + atomic_add_acq_int(&ndisk->cur_depth, 1); + + err = nvme_ns_bio_process(ndisk->ns, bp, nvd_done); + + if (err) { + atomic_add_acq_int(&ndisk->cur_depth, -1); + bp->bio_error = EIO; + bp->bio_flags |= BIO_ERROR; + bp->bio_resid = bp->bio_bcount; + biodone(bp); + } + +#ifdef BIO_ORDERED + /* + * BIO_ORDERED flag dictates that the bio with BIO_ORDERED + * flag set must be completed before proceeding with + * additional bios. + */ + if (bp->bio_flags & BIO_ORDERED) { + while (ndisk->cur_depth > 0) { + pause("nvd flush", 1); + } + } +#endif + } +} + +static void +create_geom_disk(void *arg, struct nvme_namespace *ns) +{ + struct nvd_disk *ndisk; + struct disk *disk; + + ndisk = malloc(sizeof(struct nvd_disk), M_NVD, M_ZERO | M_NOWAIT); + + disk = disk_alloc(); + disk->d_strategy = nvd_strategy; + disk->d_ioctl = nvd_ioctl; + disk->d_name = "nvd"; + disk->d_drv1 = ndisk; + + disk->d_maxsize = nvme_ns_get_max_io_xfer_size(ns); + disk->d_sectorsize = nvme_ns_get_sector_size(ns); + disk->d_mediasize = (off_t)nvme_ns_get_size(ns); + + if (TAILQ_EMPTY(&nvd_head)) + disk->d_unit = 0; + else + disk->d_unit = TAILQ_FIRST(&nvd_head)->disk->d_unit + 1; + + disk->d_flags = 0; + + if (nvme_ns_get_flags(ns) & NVME_NS_DEALLOCATE_SUPPORTED) + disk->d_flags |= DISKFLAG_CANDELETE; + + if (nvme_ns_get_flags(ns) & NVME_NS_FLUSH_SUPPORTED) + disk->d_flags |= DISKFLAG_CANFLUSHCACHE; + + strlcpy(disk->d_ident, nvme_ns_get_serial_number(ns), + sizeof(disk->d_ident)); + +#if __FreeBSD_version >= 900034 + strlcpy(disk->d_descr, nvme_ns_get_model_number(ns), + sizeof(disk->d_descr)); +#endif + + disk_create(disk, DISK_VERSION); + + ndisk->ns = ns; + ndisk->disk = disk; + ndisk->cur_depth = 0; + + mtx_init(&ndisk->bioqlock, "NVD bioq lock", NULL, MTX_DEF); + bioq_init(&ndisk->bioq); + + TASK_INIT(&ndisk->bioqtask, 0, nvd_bioq_process, ndisk); + ndisk->tq = taskqueue_create("nvd_taskq", M_WAITOK, + taskqueue_thread_enqueue, &ndisk->tq); + taskqueue_start_threads(&ndisk->tq, 1, PI_DISK, "nvd taskq"); + + TAILQ_INSERT_HEAD(&nvd_head, ndisk, tailq); +} + +static void +destroy_geom_disk(struct nvd_disk *ndisk) +{ + struct bio *bp; + + taskqueue_free(ndisk->tq); + disk_destroy(ndisk->disk); + + mtx_lock(&ndisk->bioqlock); + for (;;) { + bp = bioq_takefirst(&ndisk->bioq); + if (bp == NULL) + break; + bp->bio_error = EIO; + bp->bio_flags |= BIO_ERROR; + bp->bio_resid = bp->bio_bcount; + + biodone(bp); + } + mtx_unlock(&ndisk->bioqlock); + + mtx_destroy(&ndisk->bioqlock); +} Added: head/sys/dev/nvme/nvme.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sys/dev/nvme/nvme.c Mon Sep 17 19:23:01 2012 (r240616) @@ -0,0 +1,408 @@ +/*- + * Copyright (C) 2012 Intel Corporation + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <sys/cdefs.h> +__FBSDID("$FreeBSD$"); + +#include <sys/param.h> +#include <sys/bus.h> +#include <sys/conf.h> +#include <sys/module.h> + +#include <dev/pci/pcivar.h> + +#include "nvme_private.h" + +struct nvme_consumer { + nvme_consumer_cb_fn_t cb_fn; + void *cb_arg; +}; + +struct nvme_consumer nvme_consumer[NVME_MAX_CONSUMERS]; + +MALLOC_DEFINE(M_NVME, "nvme", "nvme(4) memory allocations"); + +static int nvme_probe(device_t); +static int nvme_attach(device_t); +static int nvme_detach(device_t); + +static devclass_t nvme_devclass; + +static device_method_t nvme_pci_methods[] = { + /* Device interface */ + DEVMETHOD(device_probe, nvme_probe), + DEVMETHOD(device_attach, nvme_attach), + DEVMETHOD(device_detach, nvme_detach), + { 0, 0 } +}; + +static driver_t nvme_pci_driver = { + "nvme", + nvme_pci_methods, + sizeof(struct nvme_controller), +}; + +DRIVER_MODULE(nvme, pci, nvme_pci_driver, nvme_devclass, 0, 0); +MODULE_VERSION(nvme, 1); + +static struct _pcsid +{ + u_int32_t type; + const char *desc; +} pci_ids[] = { + { 0x01118086, "NVMe Controller" }, + { CHATHAM_PCI_ID, "Chatham Prototype NVMe Controller" }, + { IDT_PCI_ID, "IDT NVMe Controller" }, + { 0x00000000, NULL } +}; + +static int +nvme_probe (device_t device) +{ + u_int32_t type = pci_get_devid(device); + struct _pcsid *ep = pci_ids; + + while (ep->type && ep->type != type) + ++ep; + + if (ep->desc) { + device_set_desc(device, ep->desc); + return (BUS_PROBE_DEFAULT); + } else + return (ENXIO); +} + +static void +nvme_load(void) +{ +} + +static void +nvme_unload(void) +{ +} + +static void +nvme_shutdown(void) +{ + device_t *devlist; + struct nvme_controller *ctrlr; + union cc_register cc; + union csts_register csts; + int dev, devcount; + + if (devclass_get_devices(nvme_devclass, &devlist, &devcount)) + return; + + for (dev = 0; dev < devcount; dev++) { + /* + * Only notify controller of shutdown when a real shutdown is + * in process, not when a module unload occurs. It seems at + * least some controllers (Chatham at least) don't let you + * re-enable the controller after shutdown notification has + * been received. + */ + ctrlr = DEVICE2SOFTC(devlist[dev]); + cc.raw = nvme_mmio_read_4(ctrlr, cc); + cc.bits.shn = NVME_SHN_NORMAL; + nvme_mmio_write_4(ctrlr, cc, cc.raw); + csts.raw = nvme_mmio_read_4(ctrlr, csts); + while (csts.bits.shst != NVME_SHST_COMPLETE) { + DELAY(5); + csts.raw = nvme_mmio_read_4(ctrlr, csts); + } + } + + free(devlist, M_TEMP); +} + +static int +nvme_modevent(module_t mod, int type, void *arg) +{ + + switch (type) { + case MOD_LOAD: + nvme_load(); + break; + case MOD_UNLOAD: + nvme_unload(); + break; + case MOD_SHUTDOWN: + nvme_shutdown(); + break; + default: + break; + } + + return (0); +} + +moduledata_t nvme_mod = { + "nvme", + (modeventhand_t)nvme_modevent, + 0 +}; + +DECLARE_MODULE(nvme, nvme_mod, SI_SUB_DRIVERS, SI_ORDER_FIRST); + +void +nvme_dump_command(struct nvme_command *cmd) +{ + printf("opc:%x f:%x r1:%x cid:%x nsid:%x r2:%x r3:%x " + "mptr:%qx prp1:%qx prp2:%qx cdw:%x %x %x %x %x %x\n", + cmd->opc, cmd->fuse, cmd->rsvd1, cmd->cid, cmd->nsid, + cmd->rsvd2, cmd->rsvd3, + (long long unsigned int)cmd->mptr, + (long long unsigned int)cmd->prp1, + (long long unsigned int)cmd->prp2, + cmd->cdw10, cmd->cdw11, cmd->cdw12, cmd->cdw13, cmd->cdw14, + cmd->cdw15); +} + +void +nvme_dump_completion(struct nvme_completion *cpl) +{ + printf("cdw0:%08x sqhd:%04x sqid:%04x " + "cid:%04x p:%x sc:%02x sct:%x m:%x dnr:%x\n", + cpl->cdw0, cpl->sqhd, cpl->sqid, + cpl->cid, cpl->p, cpl->sf_sc, cpl->sf_sct, cpl->sf_m, + cpl->sf_dnr); +} + +void +nvme_payload_map(void *arg, bus_dma_segment_t *seg, int nseg, int error) +{ + struct nvme_tracker *tr; + struct nvme_qpair *qpair; + struct nvme_prp_list *prp_list; + uint32_t cur_nseg; + + KASSERT(error == 0, ("nvme_payload_map error != 0\n")); + + tr = (struct nvme_tracker *)arg; + qpair = tr->qpair; + + /* + * Note that we specified PAGE_SIZE for alignment and max + * segment size when creating the bus dma tags. So here + * we can safely just transfer each segment to its + * associated PRP entry. + */ + tr->cmd.prp1 = seg[0].ds_addr; + + if (nseg == 2) { + tr->cmd.prp2 = seg[1].ds_addr; + } else if (nseg > 2) { + KASSERT(tr->prp_list, + ("prp_list needed but not attached to tracker\n")); + cur_nseg = 1; + prp_list = tr->prp_list; + tr->cmd.prp2 = (uint64_t)prp_list->bus_addr; + while (cur_nseg < nseg) { + prp_list->prp[cur_nseg-1] = + (uint64_t)seg[cur_nseg].ds_addr; + cur_nseg++; + } + } + + nvme_qpair_submit_cmd(qpair, tr); +} + +struct nvme_tracker * +nvme_allocate_tracker(struct nvme_controller *ctrlr, boolean_t is_admin, + nvme_cb_fn_t cb_fn, void *cb_arg, uint32_t payload_size, void *payload) +{ + struct nvme_tracker *tr; + struct nvme_qpair *qpair; + uint32_t modulo, offset, num_prps; + boolean_t alloc_prp_list = FALSE; + + if (is_admin) { + qpair = &ctrlr->adminq; + } else { + if (ctrlr->per_cpu_io_queues) + qpair = &ctrlr->ioq[curcpu]; + else + qpair = &ctrlr->ioq[0]; + } + + num_prps = payload_size / PAGE_SIZE; + modulo = payload_size % PAGE_SIZE; + offset = (uint32_t)((uintptr_t)payload % PAGE_SIZE); + + if (modulo || offset) + num_prps += 1 + (modulo + offset - 1) / PAGE_SIZE; + + if (num_prps > 2) + alloc_prp_list = TRUE; + + tr = nvme_qpair_allocate_tracker(qpair, alloc_prp_list); + + memset(&tr->cmd, 0, sizeof(tr->cmd)); + + tr->qpair = qpair; + tr->cb_fn = cb_fn; + tr->cb_arg = cb_arg; + tr->payload_size = payload_size; + + return (tr); +} + +static int +nvme_attach(device_t dev) +{ + struct nvme_controller *ctrlr = DEVICE2SOFTC(dev); + int status; + + status = nvme_ctrlr_construct(ctrlr, dev); + + if (status != 0) + return (status); + + /* + * Reset controller twice to ensure we do a transition from cc.en==1 + * to cc.en==0. This is because we don't really know what status + * the controller was left in when boot handed off to OS. + */ + status = nvme_ctrlr_reset(ctrlr); + if (status != 0) + return (status); + + status = nvme_ctrlr_reset(ctrlr); + if (status != 0) + return (status); + + ctrlr->config_hook.ich_func = nvme_ctrlr_start; + ctrlr->config_hook.ich_arg = ctrlr; + + config_intrhook_establish(&ctrlr->config_hook); + + return (0); +} + +static int +nvme_detach (device_t dev) +{ + struct nvme_controller *ctrlr = DEVICE2SOFTC(dev); + struct nvme_namespace *ns; + int i; + + if (ctrlr->taskqueue) { + taskqueue_drain(ctrlr->taskqueue, &ctrlr->task); + taskqueue_free(ctrlr->taskqueue); + } + + for (i = 0; i < NVME_MAX_NAMESPACES; i++) { + ns = &ctrlr->ns[i]; + if (ns->cdev) + destroy_dev(ns->cdev); + } + + if (ctrlr->cdev) + destroy_dev(ctrlr->cdev); + + for (i = 0; i < ctrlr->num_io_queues; i++) { + nvme_io_qpair_destroy(&ctrlr->ioq[i]); + } + + free(ctrlr->ioq, M_NVME); + + nvme_admin_qpair_destroy(&ctrlr->adminq); + + if (ctrlr->resource != NULL) { + bus_release_resource(dev, SYS_RES_MEMORY, + ctrlr->resource_id, ctrlr->resource); + } + +#ifdef CHATHAM2 + if (ctrlr->chatham_resource != NULL) { + bus_release_resource(dev, SYS_RES_MEMORY, + ctrlr->chatham_resource_id, ctrlr->chatham_resource); + } +#endif + + if (ctrlr->tag) + bus_teardown_intr(ctrlr->dev, ctrlr->res, ctrlr->tag); + + if (ctrlr->res) + bus_release_resource(ctrlr->dev, SYS_RES_IRQ, + rman_get_rid(ctrlr->res), ctrlr->res); + + if (ctrlr->msix_enabled) + pci_release_msi(dev); + + return (0); +} + +static void +nvme_notify_consumer(struct nvme_consumer *consumer) +{ + device_t *devlist; + struct nvme_controller *ctrlr; + int dev, ns, devcount; + + if (devclass_get_devices(nvme_devclass, &devlist, &devcount)) + return; + + for (dev = 0; dev < devcount; dev++) { + ctrlr = DEVICE2SOFTC(devlist[dev]); + for (ns = 0; ns < ctrlr->cdata.nn; ns++) + (*consumer->cb_fn)(consumer->cb_arg, &ctrlr->ns[ns]); + } + + free(devlist, M_TEMP); +} + +struct nvme_consumer * +nvme_register_consumer(nvme_consumer_cb_fn_t cb_fn, void *cb_arg) +{ + int i; + + /* + * TODO: add locking around consumer registration. Not an issue + * right now since we only have one nvme consumer - nvd(4). + */ + for (i = 0; i < NVME_MAX_CONSUMERS; i++) + if (nvme_consumer[i].cb_fn == NULL) { + nvme_consumer[i].cb_fn = cb_fn; + nvme_consumer[i].cb_arg = cb_arg; + + nvme_notify_consumer(&nvme_consumer[i]); + return (&nvme_consumer[i]); + } + + printf("nvme(4): consumer not registered - no slots available\n"); + return (NULL); +} + +void +nvme_unregister_consumer(struct nvme_consumer *consumer) +{ + + consumer->cb_fn = NULL; + consumer->cb_arg = NULL; +} + Added: head/sys/dev/nvme/nvme.h ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sys/dev/nvme/nvme.h Mon Sep 17 19:23:01 2012 (r240616) @@ -0,0 +1,738 @@ +/*- + * Copyright (C) 2012 Intel Corporation + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef __NVME_H__ +#define __NVME_H__ + +#ifdef _KERNEL +#include <sys/types.h> +#endif + +#define NVME_IDENTIFY_CONTROLLER _IOR('n', 0, struct nvme_controller_data) +#define NVME_IDENTIFY_NAMESPACE _IOR('n', 1, struct nvme_namespace_data) +#define NVME_IO_TEST _IOWR('n', 2, struct nvme_io_test) +#define NVME_BIO_TEST _IOWR('n', 4, struct nvme_io_test) + +/* + * Use to mark a command to apply to all namespaces, or to retrieve global + * log pages. + */ +#define NVME_GLOBAL_NAMESPACE_TAG ((uint32_t)0xFFFFFFFF) + +union cap_lo_register { + uint32_t raw; + struct { + /** maximum queue entries supported */ + uint32_t mqes : 16; + + /** contiguous queues required */ + uint32_t cqr : 1; + + /** arbitration mechanism supported */ + uint32_t ams : 2; + + uint32_t reserved1 : 5; + + /** timeout */ + uint32_t to : 8; + } bits __packed; +} __packed; + +union cap_hi_register { + uint32_t raw; + struct { + /** doorbell stride */ + uint32_t dstrd : 4; + + uint32_t reserved3 : 1; + + /** command sets supported */ + uint32_t css_nvm : 1; + + uint32_t css_reserved : 3; + uint32_t reserved2 : 7; + + /** memory page size minimum */ + uint32_t mpsmin : 4; + + /** memory page size maximum */ + uint32_t mpsmax : 4; + + uint32_t reserved1 : 8; + } bits __packed; +} __packed; + +union cc_register { + uint32_t raw; + struct { + /** enable */ + uint32_t en : 1; + + uint32_t reserved1 : 3; + + /** i/o command set selected */ + uint32_t css : 3; + + /** memory page size */ + uint32_t mps : 4; + + /** arbitration mechanism selected */ + uint32_t ams : 3; + + /** shutdown notification */ + uint32_t shn : 2; + + /** i/o submission queue entry size */ + uint32_t iosqes : 4; + + /** i/o completion queue entry size */ + uint32_t iocqes : 4; + + uint32_t reserved2 : 8; + } bits __packed; +} __packed; + +enum shn_value { + NVME_SHN_NORMAL = 0x1, + NVME_SHN_ABRUPT = 0x2, +}; + +union csts_register { + uint32_t raw; + struct { + /** ready */ + uint32_t rdy : 1; + + /** controller fatal status */ + uint32_t cfs : 1; + + /** shutdown status */ + uint32_t shst : 2; + + uint32_t reserved1 : 28; + } bits __packed; +} __packed; + +enum shst_value { + NVME_SHST_NORMAL = 0x0, + NVME_SHST_OCCURRING = 0x1, + NVME_SHST_COMPLETE = 0x2, +}; + +union aqa_register { + uint32_t raw; + struct { + /** admin submission queue size */ + uint32_t asqs : 12; + + uint32_t reserved1 : 4; + + /** admin completion queue size */ + uint32_t acqs : 12; + + uint32_t reserved2 : 4; + } bits __packed; +} __packed; + +struct nvme_registers +{ + /** controller capabilities */ + union cap_lo_register cap_lo; + union cap_hi_register cap_hi; + + uint32_t vs; /* version */ + uint32_t intms; /* interrupt mask set */ + uint32_t intmc; /* interrupt mask clear */ + + /** controller configuration */ + union cc_register cc; + + uint32_t reserved1; + uint32_t csts; /* controller status */ + uint32_t reserved2; + + /** admin queue attributes */ + union aqa_register aqa; + + uint64_t asq; /* admin submission queue base addr */ + uint64_t acq; /* admin completion queue base addr */ + uint32_t reserved3[0x3f2]; + + struct { + uint32_t sq_tdbl; /* submission queue tail doorbell */ + uint32_t cq_hdbl; /* completion queue head doorbell */ + } doorbell[1] __packed; +} __packed; + +struct nvme_command +{ + /* dword 0 */ + uint16_t opc : 8; /* opcode */ + uint16_t fuse : 2; /* fused operation */ + uint16_t rsvd1 : 6; + uint16_t cid; /* command identifier */ + + /* dword 1 */ + uint32_t nsid; /* namespace identifier */ + + /* dword 2-3 */ + uint32_t rsvd2; + uint32_t rsvd3; + + /* dword 4-5 */ + uint64_t mptr; /* metadata pointer */ + + /* dword 6-7 */ + uint64_t prp1; /* prp entry 1 */ + + /* dword 8-9 */ + uint64_t prp2; /* prp entry 2 */ + + /* dword 10-15 */ + uint32_t cdw10; /* command-specific */ + uint32_t cdw11; /* command-specific */ + uint32_t cdw12; /* command-specific */ + uint32_t cdw13; /* command-specific */ + uint32_t cdw14; /* command-specific */ + uint32_t cdw15; /* command-specific */ +} __packed; + +struct nvme_completion { + + /* dword 0 */ + uint32_t cdw0; /* command-specific */ + + /* dword 1 */ + uint32_t rsvd1; + + /* dword 2 */ + uint16_t sqhd; /* submission queue head pointer */ + uint16_t sqid; /* submission queue identifier */ + + /* dword 3 */ + uint16_t cid; /* command identifier */ + uint16_t p : 1; /* phase tag */ + uint16_t sf_sc : 8; /* status field - status code */ + uint16_t sf_sct : 3; /* status field - status code type */ + uint16_t rsvd2 : 2; + uint16_t sf_m : 1; /* status field - more */ + uint16_t sf_dnr : 1; /* status field - do not retry */ +} __packed; + +struct nvme_dsm_range { + + uint32_t attributes; + uint32_t length; + uint64_t starting_lba; +} __packed; + +/* status code types */ +enum nvme_status_code_type { + NVME_SCT_GENERIC = 0x0, + NVME_SCT_COMMAND_SPECIFIC = 0x1, + NVME_SCT_MEDIA_ERROR = 0x2, + /* 0x3-0x6 - reserved */ + NVME_SCT_VENDOR_SPECIFIC = 0x7, +}; + +/* generic command status codes */ +enum nvme_generic_command_status_code { + NVME_SC_SUCCESS = 0x00, *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201209171923.q8HJN1IZ073080>