From owner-svn-src-projects@freebsd.org Sun May 22 04:28:56 2016 Return-Path: Delivered-To: svn-src-projects@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 87CA9B41A1D for ; Sun, 22 May 2016 04:28:56 +0000 (UTC) (envelope-from ngie@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3AF371F12; Sun, 22 May 2016 04:28:56 +0000 (UTC) (envelope-from ngie@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id u4M4Stsa023977; Sun, 22 May 2016 04:28:55 GMT (envelope-from ngie@FreeBSD.org) Received: (from ngie@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id u4M4StJu023975; Sun, 22 May 2016 04:28:55 GMT (envelope-from ngie@FreeBSD.org) Message-Id: <201605220428.u4M4StJu023975@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: ngie set sender to ngie@FreeBSD.org using -f From: Garrett Cooper Date: Sun, 22 May 2016 04:28:55 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r300396 - projects/vmware_pvscsi/sys/dev/vmware/vmw_pvscsi X-SVN-Group: projects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 May 2016 04:28:56 -0000 Author: ngie Date: Sun May 22 04:28:55 2016 New Revision: 300396 URL: https://svnweb.freebsd.org/changeset/base/300396 Log: Commit the 1.0.1.0-k driver obtained from VMware under a BSD 2-clause license Obtained from: Isilon OneFS Sponsored by: EMC / Isilon Storage Division Added: projects/vmware_pvscsi/sys/dev/vmware/vmw_pvscsi/ projects/vmware_pvscsi/sys/dev/vmware/vmw_pvscsi/vmw_pvscsi.c (contents, props changed) projects/vmware_pvscsi/sys/dev/vmware/vmw_pvscsi/vmw_pvscsi.h (contents, props changed) Added: projects/vmware_pvscsi/sys/dev/vmware/vmw_pvscsi/vmw_pvscsi.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ projects/vmware_pvscsi/sys/dev/vmware/vmw_pvscsi/vmw_pvscsi.c Sun May 22 04:28:55 2016 (r300396) @@ -0,0 +1,2745 @@ +/* + * Copyright (c) 2014 EMC Corporation, Inc. All rights reserved. + */ + +/* ************************************************************************ + * Copyright 2008 VMware, Inc. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + * DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY + * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF + * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * ************************************************************************/ + +/* + * pvscsi.c -- + * + * This is a driver for the VMware PVSCSI HBA adapter. + */ + +#ifndef __FreeBSD__ +#include "driver-config.h" + +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include "compat_module.h" +#include "compat_scsi.h" +#include "compat_pci.h" +#include "compat_interrupt.h" +#include "compat_workqueue.h" + +#include "pvscsi_defs.h" +#include "pvscsi_version.h" +#include "scsi_defs.h" +#include "vm_device_version.h" +#include "vm_assert.h" + +#define PVSCSI_LINUX_DRIVER_DESC "VMware PVSCSI driver" + +MODULE_DESCRIPTION(PVSCSI_LINUX_DRIVER_DESC); +MODULE_AUTHOR("VMware, Inc."); +MODULE_LICENSE("GPL v2"); +MODULE_VERSION(PVSCSI_DRIVER_VERSION_STRING); + +/* + * Starting with SLE10sp2, Novell requires that IHVs sign a support agreement + * with them and mark their kernel modules as externally supported via a + * change to the module header. If this isn't done, the module will not load + * by default (i.e., neither mkinitrd nor modprobe will accept it). + */ +MODULE_INFO(supported, "external"); +#else +#include "vmw_pvscsi.h" +#endif /* __FreeBSD__ */ + +#define PVSCSI_DEFAULT_NUM_PAGES_PER_RING 8 +#define PVSCSI_DEFAULT_NUM_PAGES_MSG_RING 1 +#define PVSCSI_DEFAULT_QUEUE_DEPTH 64 + +#ifndef __FreeBSD__ +/* MSI-X has horrible performance in < 2.6.19 due to needless mask frobbing */ +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 19) +#define PVSCSI_DISABLE_MSIX 0 +#else +#define PVSCSI_DISABLE_MSIX 1 +#endif +#endif /* __FreeBSD__ */ + + +#ifndef __FreeBSD__ +#define HOST_ADAPTER(host) ((struct pvscsi_adapter *)(host)->hostdata) + +#define LOG(level, fmt, args...) \ +do { \ + if (pvscsi_debug_level > level) \ + printk(KERN_DEBUG "pvscsi: " fmt, args); \ +} while (0) +#endif /* __FreeBSD__ */ + + +#ifdef __FreeBSD__ +typedef struct pvscsi_adapter pvscsinst_t; + +#define MODNM pvscsi +enum { + PVSCSI_MSIX_VEC0 = 0, /* Only one MSI-X interrupt required */ + PVSCSI_NUM_MSIX +}; + +MALLOC_DEFINE(M_PVSCSI, "pvscsi", "VMware's para-virtualized scsi driver"); +MALLOC_DEFINE(M_PVSCSI_PCI, "pvscsi_pci", "pvscsi's ring queues"); +MALLOC_DEFINE(M_PVSCSI_SGL, "pvscsi_sgl", "pvscsi's scatter-gather list"); +#endif /* __FreeBSD__ */ + + + +/* Command line parameters */ +#ifndef __FreeBSD__ +static int pvscsi_debug_level; +#endif /* __FreeBSD__ */ +static int pvscsi_ring_pages = PVSCSI_DEFAULT_NUM_PAGES_PER_RING; +static int pvscsi_msg_ring_pages = PVSCSI_DEFAULT_NUM_PAGES_MSG_RING; +#ifndef __FreeBSD__ +static int pvscsi_cmd_per_lun = PVSCSI_DEFAULT_QUEUE_DEPTH; +static compat_mod_param_bool pvscsi_disable_msi; +static compat_mod_param_bool pvscsi_disable_msix = PVSCSI_DISABLE_MSIX; +static compat_mod_param_bool pvscsi_use_msg = TRUE; +#else +static bool pvscsi_use_msg = true; +#endif /* __FreeBSD__ */ + +#ifndef __FreeBSD__ +#define PVSCSI_RW (S_IRUSR | S_IWUSR) + +module_param_named(debug_level, pvscsi_debug_level, int, PVSCSI_RW); +MODULE_PARM_DESC(debug_level, "Debug logging level - (default=0)"); + +module_param_named(ring_pages, pvscsi_ring_pages, int, PVSCSI_RW); +MODULE_PARM_DESC(ring_pages, "Number of pages per req/cmp ring - (default=" + __stringify(PVSCSI_DEFAULT_NUM_PAGES_PER_RING) ")"); + +module_param_named(msg_ring_pages, pvscsi_msg_ring_pages, int, PVSCSI_RW); +MODULE_PARM_DESC(msg_ring_pages, "Number of pages for the msg ring - (default=" + __stringify(PVSCSI_DEFAULT_NUM_PAGES_MSG_RING) ")"); + +module_param_named(cmd_per_lun, pvscsi_cmd_per_lun, int, PVSCSI_RW); +MODULE_PARM_DESC(cmd_per_lun, "Maximum commands per lun - (default=" + __stringify(PVSCSI_MAX_REQ_QUEUE_DEPTH) ")"); + +module_param_named(disable_msi, pvscsi_disable_msi, bool, PVSCSI_RW); +MODULE_PARM_DESC(disable_msi, "Disable MSI use in driver - (default=0)"); + +module_param_named(disable_msix, pvscsi_disable_msix, bool, PVSCSI_RW); +MODULE_PARM_DESC(disable_msix, "Disable MSI-X use in driver - (default=" + __stringify(PVSCSI_DISABLE_MSIX) ")"); + +module_param_named(use_msg, pvscsi_use_msg, bool, PVSCSI_RW); +MODULE_PARM_DESC(use_msg, "Use msg ring when available - (default=1)"); + +static const struct pci_device_id pvscsi_pci_tbl[] = { + { PCI_DEVICE(PCI_VENDOR_ID_VMWARE, PCI_DEVICE_ID_VMWARE_PVSCSI) }, + { 0 } +}; + +MODULE_DEVICE_TABLE(pci, pvscsi_pci_tbl); +#else +#define pvscsi_dev(adapter) ((adapter)->pvs_dev) +#define pvscsi_simdev(sim) (pvscsi_dev((pvscsinst_t *)cam_sim_softc(sim))) + +/* + * To share code between the Linux and Isilon ports we key off the variable + * "irq" to figure out if we need to actually lock or not + */ +#define spin_lock_irqsave(l,f) if (irq) PVSCSILCK +#define spin_unlock_irqrestore(l,f) if (irq) PVSCSIULCK + +#define SCSI_SENSE_BUFFERSIZE sizeof(csio->sense_data) +#define SIMPLE_QUEUE_TAG MSG_SIMPLE_Q_TAG +#define HEAD_OF_QUEUE_TAG MSG_HEAD_OF_Q_TAG +#define ORDERED_QUEUE_TAG MSG_ORDERED_Q_TAG + +enum dma_data_direction { + DMA_BIDIRECTIONAL = 0, + DMA_TO_DEVICE = 1, + DMA_FROM_DEVICE = 2, + DMA_NONE = 3, +}; + +struct scsi_cmnd { + struct ccb_scsiio *qsc_csio; + void *sense_buffer; + unsigned short cmd_len; + unsigned char *cmn; + struct scsi_device *device; + enum dma_data_direction sc_data_direction; + unsigned char tag; + unsigned char *cmnd; + pvscsinst_t *adapter; +}; + +struct scsi_device { + unsigned int id, lun, channel; + bool tagged_supported; +}; +#endif /* __FreeBSD__ */ + +static struct pvscsi_ctx * +#ifndef __FreeBSD__ +pvscsi_find_context(const struct pvscsi_adapter *adapter, struct scsi_cmnd *cmd) +#else /* __FreeBSD__ */ +pvscsi_find_context(struct pvscsi_adapter *adapter, struct ccb_scsiio *cmd) +#endif /* __FreeBSD__ */ +{ + struct pvscsi_ctx *ctx, *end; + + end = &adapter->cmd_map[adapter->req_depth]; + for (ctx = adapter->cmd_map; ctx < end; ctx++) + if (ctx->cmd == cmd) + return ctx; + + return NULL; +} + +static struct pvscsi_ctx * +pvscsi_acquire_context(struct pvscsi_adapter *adapter, struct scsi_cmnd *cmd) +{ + struct pvscsi_ctx *ctx; + + if (list_empty(&adapter->cmd_pool)) + return NULL; + + ctx = list_entry(adapter->cmd_pool.next, struct pvscsi_ctx, list); +#ifndef __FreeBSD__ + ctx->cmd = cmd; +#else + ctx->cmd = cmd->qsc_csio; + ctx->toed = false; + ctx->debugerr_checked = false; +#endif /* __FreeBSD__ */ + list_del(&ctx->list); + + return ctx; +} + +static void pvscsi_release_context(struct pvscsi_adapter *adapter, + struct pvscsi_ctx *ctx) +{ + ctx->cmd = NULL; + list_add(&ctx->list, &adapter->cmd_pool); +} + +/* + * Map a pvscsi_ctx struct to a context ID field value; we map to a simple + * non-zero integer. + */ +static u64 pvscsi_map_context(const struct pvscsi_adapter *adapter, + const struct pvscsi_ctx *ctx) +{ + return ctx - adapter->cmd_map + 1; +} + +static struct pvscsi_ctx * +pvscsi_get_context(const struct pvscsi_adapter *adapter, u64 context) +{ + return &adapter->cmd_map[context - 1]; +} + +/************************************************************** + * + * VMWARE PVSCSI Hypervisor Communication Implementation + * + * This code is largely independent of any Linux internals. + * + **************************************************************/ + +static void pvscsi_reg_write(const struct pvscsi_adapter *adapter, + u32 offset, u32 val) +{ +#ifndef __FreeBSD__ + writel(val, adapter->mmioBase + offset); +#else /* __FreeBSD__ */ + bus_space_write_4(adapter->pvs_mmtag, adapter->pvs_mmhndl, offset, val); +#endif /* __FreeBSD__ */ +} + +static u32 pvscsi_reg_read(const struct pvscsi_adapter *adapter, u32 offset) +{ +#ifndef __FreeBSD__ + return readl(adapter->mmioBase + offset); +#else /* __FreeBSD__ */ + return bus_space_read_4(adapter->pvs_mmtag, adapter->pvs_mmhndl, offset); +#endif /* __FreeBSD__ */ +} + +static u32 pvscsi_read_intr_status(const struct pvscsi_adapter *adapter) +{ + return pvscsi_reg_read(adapter, PVSCSI_REG_OFFSET_INTR_STATUS); +} + +static void pvscsi_write_intr_status(const struct pvscsi_adapter *adapter, + u32 val) +{ + pvscsi_reg_write(adapter, PVSCSI_REG_OFFSET_INTR_STATUS, val); +} + +static void pvscsi_unmask_intr(const struct pvscsi_adapter *adapter) +{ + u32 intr_bits; + + intr_bits = PVSCSI_INTR_CMPL_MASK; + if (adapter->use_msg) { + intr_bits |= PVSCSI_INTR_MSG_MASK; + } + + pvscsi_reg_write(adapter, PVSCSI_REG_OFFSET_INTR_MASK, intr_bits); +} + +static void pvscsi_mask_intr(const struct pvscsi_adapter *adapter) +{ + pvscsi_reg_write(adapter, PVSCSI_REG_OFFSET_INTR_MASK, 0); +} + +static void pvscsi_write_cmd_desc(const struct pvscsi_adapter *adapter, + u32 cmd, const void *desc, size_t len) +{ +#ifndef __FreeBSD__ + u32 *ptr = (u32 *)desc; +#else /* __FreeBSD__ */ + const u32 *ptr = (const u32 *)desc; +#endif /* __FreeBSD__ */ + unsigned i; + + len /= sizeof(u32); + pvscsi_reg_write(adapter, PVSCSI_REG_OFFSET_COMMAND, cmd); + for (i = 0; i < len; i++) + pvscsi_reg_write(adapter, + PVSCSI_REG_OFFSET_COMMAND_DATA, ptr[i]); +} + +#ifndef __FreeBSD__ +static void pvscsi_abort_cmd(const struct pvscsi_adapter *adapter, + const struct pvscsi_ctx *ctx) +{ + struct PVSCSICmdDescAbortCmd cmd = { 0 }; + + cmd.target = ctx->cmd->device->id; +#else /* __FreeBSD__ */ +static void pvscsi_abort_cmd(struct pvscsi_adapter *adapter, + struct pvscsi_ctx *ctx, + target_id_t trg) +{ + struct PVSCSICmdDescAbortCmd cmd = { 0 }; + cmd.target = trg; +#endif /* __FreeBSD__ */ + cmd.context = pvscsi_map_context(adapter, ctx); + + pvscsi_write_cmd_desc(adapter, PVSCSI_CMD_ABORT_CMD, &cmd, sizeof cmd); +} + +static void pvscsi_kick_rw_io(const struct pvscsi_adapter *adapter) +{ + pvscsi_reg_write(adapter, PVSCSI_REG_OFFSET_KICK_RW_IO, 0); +} + +static void pvscsi_process_request_ring(const struct pvscsi_adapter *adapter) +{ + pvscsi_reg_write(adapter, PVSCSI_REG_OFFSET_KICK_NON_RW_IO, 0); +} + +static int scsi_is_rw(unsigned char op) +{ + return op == READ_6 || op == WRITE_6 || + op == READ_10 || op == WRITE_10 || + op == READ_12 || op == WRITE_12 || + op == READ_16 || op == WRITE_16; +} + +static void pvscsi_kick_io(const struct pvscsi_adapter *adapter, + unsigned char op) +{ + if (scsi_is_rw(op)) + pvscsi_kick_rw_io(adapter); + else + pvscsi_process_request_ring(adapter); +} + +static void ll_adapter_reset(const struct pvscsi_adapter *adapter) +{ + LOG(0, "Adapter Reset on %p\n", adapter); + + pvscsi_write_cmd_desc(adapter, PVSCSI_CMD_ADAPTER_RESET, NULL, 0); +} + +static void ll_bus_reset(const struct pvscsi_adapter *adapter) +{ + LOG(0, "Reseting bus on %p\n", adapter); + + pvscsi_write_cmd_desc(adapter, PVSCSI_CMD_RESET_BUS, NULL, 0); +} + +static void ll_device_reset(const struct pvscsi_adapter *adapter, u32 target) +{ + struct PVSCSICmdDescResetDevice cmd = { 0 }; + + LOG(0, "Reseting device: target=%u\n", target); + +#ifdef __FreeBSD__ + device_t device = pvscsi_dev(adapter); + device_printf(device, "Resetting target %u\n", target); +#endif /* _FreeBSD__ */ + cmd.target = target; + + pvscsi_write_cmd_desc(adapter, PVSCSI_CMD_RESET_DEVICE, + &cmd, sizeof cmd); +#ifdef __FreeBSD__ + device_printf(device, "Done resetting target %u\n", target); +#endif /* _FreeBSD__ */ +} + + +#ifndef __FreeBSD__ +/************************************************************** + * + * VMWARE Hypervisor ring / SCSI mid-layer interactions + * + * Functions which have to deal with both ring semantics + * and Linux SCSI internals are placed here. + * + **************************************************************/ + +static void pvscsi_create_sg(struct pvscsi_ctx *ctx, + struct scatterlist *sg, unsigned count) +{ + unsigned i; + struct PVSCSISGElement *sge; + + BUG_ON(count > PVSCSI_MAX_NUM_SG_ENTRIES_PER_SEGMENT); + + sge = &ctx->sgl->sge[0]; + for (i = 0; i < count; i++, sg++) { + sge[i].addr = sg_dma_address(sg); + sge[i].length = sg_dma_len(sg); + sge[i].flags = 0; + } +} +#endif /* __FreeBSD__ */ + +#ifdef __FreeBSD__ +/* + * Takes a list of physical segments and translates them into the VMware + * device emulation's scatter/gather format. It does not initiate the I/O. + * It reports any errors in the translation through the ctx structure. + * + * The bus_dma_segment_t pointed to by dm_segs is allocated on the stack. + */ +static void +pvscsi_queue_io(void *arg, bus_dma_segment_t *dm_segs, int nseg, int error) +{ + struct pvscsi_ctx *ctx = (struct pvscsi_ctx *)arg; + + if (error || ctx->dmamapping_errno) { + ctx->dmamapping_errno = error; + return; + } + + if (nseg > PVSCSI_MAX_NUM_SG_ENTRIES_PER_SEGMENT) { + ctx->dmamapping_errno = EFBIG; + return; + } + + unsigned i; + struct PVSCSISGElement *sge = &ctx->sgl->sge[0]; + + struct PVSCSIRingReqDesc *e = ctx->e; + e->flags |= PVSCSI_FLAG_CMD_WITH_SG_LIST; + + for (i = 0; i < nseg; i++) { + sge[i].addr = dm_segs[i].ds_addr; + sge[i].length = dm_segs[i].ds_len; + sge[i].flags = 0; + } + + ctx->dmamapping_errno = 0; + e->flags |= PVSCSI_FLAG_CMD_WITH_SG_LIST; + ctx->sglPA = pci_map_single(adapter->dev, ctx->sgl, + PAGE_SIZE, PCI_DMA_TODEVICE); + e->dataAddr = ctx->sglPA; +} +#define scsi_bufflen(cmd) (cmd)->qsc_csio->dxfer_len +#define pvscsi_create_sg(a,b,c) +#define scsi_sg_count(a) 2 +#define scsi_dma_map(a) 2 + +static inline dma_addr_t +sg_dma_address_fn(void) +{ + panic("This code-path shouldn't have been taken"); + return 0; +} +#define sg_dma_address(sg) sg_dma_address_fn() +#define IRQ_RETVAL(a) 0 + +#define SAM_STAT_GOOD SCSI_STATUS_OK +#define SAM_STAT_CHECK_CONDITION SCSI_STATUS_CHECK_COND +#define SAM_STAT_COMMAND_TERMINATED SCSI_STATUS_CMD_TERMINATED +#endif /* __FreeBSD__ */ +/* + * Map all data buffers for a command into PCI space and + * setup the scatter/gather list if needed. + */ +static void pvscsi_map_buffers(struct pvscsi_adapter *adapter, + struct pvscsi_ctx *ctx, struct scsi_cmnd *cmd, + struct PVSCSIRingReqDesc *e) +{ + unsigned count; + unsigned bufflen = scsi_bufflen(cmd); + + e->dataLen = bufflen; + e->dataAddr = 0; + if (bufflen == 0) + return; + +#ifdef __FreeBSD__ + struct ccb_scsiio *csio = cmd->qsc_csio; + device_t device = pvscsi_dev(adapter); + + ctx->e = e; + ctx->dmamapping_errno = 0; + + switch(csio->ccb_h.flags & (CAM_DATA_PHYS|CAM_SCATTER_VALID)) { + case CAM_DATA_PHYS|CAM_SCATTER_VALID: { + pvscsi_queue_io(ctx, + (struct bus_dma_segment *)csio->data_ptr, + csio->sglist_cnt, 0); + LOG(0, "CAM_DATA_PHYS|CAM_SCATTER_VALID\n"); + break; + } + + case CAM_SCATTER_VALID: { + device_printf(device, "No support yet for list of VAs\n"); + csio->ccb_h.status = CAM_REQ_CMP_ERR; + LOG(0, "CAM_SCATTER_VALID\n"); + break; + } + + case CAM_DATA_PHYS: { + struct bus_dma_segment seg; + seg.ds_addr = (bus_addr_t)(vm_offset_t)csio->data_ptr; + seg.ds_len = csio->dxfer_len; + pvscsi_queue_io(ctx, &seg, 1, 0); + LOG(0, "CAM_DATA_PHYS\n"); + break; + } + + case 0: { + int error; + + if (csio->bio && !csio->data_ptr) + error = bus_dmamap_load_bio(adapter->pvs_dmat, + ctx->dmap, csio->bio, + pvscsi_queue_io, ctx, + BUS_DMA_NOWAIT); + else + error = bus_dmamap_load(adapter->pvs_dmat, + ctx->dmap, csio->data_ptr, + csio->dxfer_len, + pvscsi_queue_io, ctx, + BUS_DMA_NOWAIT); + if (error) + ctx->dmamapping_errno = error; + + LOG(0, "single VA %p %p %lx\n", csio->bio, + csio->data_ptr, virt_to_phys(csio->data_ptr)); + break; + } + + default: { + panic("Unknown case %d", + (csio->ccb_h.flags & + (CAM_DATA_PHYS|CAM_SCATTER_VALID))); + } + } + + if (ctx->dmamapping_errno) { + if (ctx->dmamapping_errno == EFBIG) + csio->ccb_h.flags = CAM_REQ_TOO_BIG; + else + csio->ccb_h.flags = CAM_REQ_CMP_ERR; + } + + /* + * Setup 'count' and 'segs' so that we choose the path that sets + * PVSCSI_FLAG_CMD_WITH_SG_LIST and uses a scatter/gather list + */ +#endif /* __FreeBSD__ */ + count = scsi_sg_count(cmd); + if (count != 0) { +#ifndef __FreeBSD__ + struct scatterlist *sg = scsi_sglist(cmd); + int segs = pci_map_sg(adapter->dev, sg, count, + cmd->sc_data_direction); +#else + int segs = 2; /* Force the more generic path below */ +#endif /* __FreeBSD__ */ + if (segs > 1) { + pvscsi_create_sg(ctx, sg, segs); + + e->flags |= PVSCSI_FLAG_CMD_WITH_SG_LIST; + ctx->sglPA = pci_map_single(adapter->dev, ctx->sgl, + PAGE_SIZE, PCI_DMA_TODEVICE); + e->dataAddr = ctx->sglPA; + } else + e->dataAddr = sg_dma_address(sg); + } else { +#ifndef __FreeBSD__ + ctx->dataPA = pci_map_single(adapter->dev, + scsi_request_buffer(cmd), bufflen, + cmd->sc_data_direction); +#endif /* __FreeBSD__ */ + e->dataAddr = ctx->dataPA; + } +} + +static void pvscsi_unmap_buffers(const struct pvscsi_adapter *adapter, + struct pvscsi_ctx *ctx) +{ +#ifndef __FreeBSD__ + struct scsi_cmnd *cmd; + unsigned bufflen; + + cmd = ctx->cmd; + bufflen = scsi_bufflen(cmd); + + if (bufflen != 0) { + unsigned count = scsi_sg_count(cmd); + + if (count != 0) { + pci_unmap_sg(adapter->dev, scsi_sglist(cmd), count, + cmd->sc_data_direction); + if (ctx->sglPA) { + pci_unmap_single(adapter->dev, ctx->sglPA, + PAGE_SIZE, PCI_DMA_TODEVICE); + ctx->sglPA = 0; + } + } else + pci_unmap_single(adapter->dev, ctx->dataPA, bufflen, + cmd->sc_data_direction); + } + if (cmd->sense_buffer) + pci_unmap_single(adapter->dev, ctx->sensePA, + SCSI_SENSE_BUFFERSIZE, PCI_DMA_FROMDEVICE); +#else /* __FreeBSD__ */ + struct ccb_scsiio *csio = ctx->cmd; + + if (csio->dxfer_len && + !(csio->ccb_h.flags & (CAM_DATA_PHYS|CAM_SCATTER_VALID))) { + bus_dmamap_unload(adapter->pvs_dmat, ctx->dmap); + } +#endif /* __FreeBSD__ */ +} + +static int pvscsi_allocate_rings(struct pvscsi_adapter *adapter) +{ + adapter->rings_state = pci_alloc_consistent(adapter->dev, PAGE_SIZE, + &adapter->ringStatePA); + if (!adapter->rings_state) + return -ENOMEM; + + adapter->req_pages = min(PVSCSI_MAX_NUM_PAGES_REQ_RING, + pvscsi_ring_pages); + adapter->req_depth = adapter->req_pages + * PVSCSI_MAX_NUM_REQ_ENTRIES_PER_PAGE; + adapter->req_ring = pci_alloc_consistent(adapter->dev, + adapter->req_pages * PAGE_SIZE, + &adapter->reqRingPA); + if (!adapter->req_ring) + return -ENOMEM; + + adapter->cmp_pages = min(PVSCSI_MAX_NUM_PAGES_CMP_RING, + pvscsi_ring_pages); + adapter->cmp_ring = pci_alloc_consistent(adapter->dev, + adapter->cmp_pages * PAGE_SIZE, + &adapter->cmpRingPA); + if (!adapter->cmp_ring) + return -ENOMEM; + +#ifndef __FreeBSD__ + BUG_ON(adapter->ringStatePA & ~PAGE_MASK); + BUG_ON(adapter->reqRingPA & ~PAGE_MASK); + BUG_ON(adapter->cmpRingPA & ~PAGE_MASK); +#else + BUG_ON(adapter->ringStatePA & PAGE_MASK); + BUG_ON(adapter->reqRingPA & PAGE_MASK); + BUG_ON(adapter->cmpRingPA & PAGE_MASK); +#endif /* __FreeBSD__ */ + + if (!adapter->use_msg) + return 0; + + adapter->msg_pages = min(PVSCSI_MAX_NUM_PAGES_MSG_RING, + pvscsi_msg_ring_pages); + adapter->msg_ring = pci_alloc_consistent(adapter->dev, + adapter->msg_pages * PAGE_SIZE, + &adapter->msgRingPA); + if (!adapter->msg_ring) + return -ENOMEM; +#ifndef __FreeBSD__ + BUG_ON(adapter->msgRingPA & ~PAGE_MASK); +#else + BUG_ON(adapter->msgRingPA & PAGE_MASK); +#endif /* __FreeBSD__ */ + + return 0; +} + +static void pvscsi_setup_all_rings(const struct pvscsi_adapter *adapter) +{ + struct PVSCSICmdDescSetupRings cmd = { 0 }; + dma_addr_t base; + unsigned i; + + cmd.ringsStatePPN = adapter->ringStatePA >> PAGE_SHIFT; + cmd.reqRingNumPages = adapter->req_pages; + cmd.cmpRingNumPages = adapter->cmp_pages; + + base = adapter->reqRingPA; + for (i = 0; i < adapter->req_pages; i++) { + cmd.reqRingPPNs[i] = base >> PAGE_SHIFT; + base += PAGE_SIZE; + } + + base = adapter->cmpRingPA; + for (i = 0; i < adapter->cmp_pages; i++) { + cmd.cmpRingPPNs[i] = base >> PAGE_SHIFT; + base += PAGE_SIZE; + } + + memset(adapter->rings_state, 0, PAGE_SIZE); + memset(adapter->req_ring, 0, adapter->req_pages * PAGE_SIZE); + memset(adapter->cmp_ring, 0, adapter->cmp_pages * PAGE_SIZE); + + pvscsi_write_cmd_desc(adapter, PVSCSI_CMD_SETUP_RINGS, + &cmd, sizeof cmd); + + if (adapter->use_msg) { + struct PVSCSICmdDescSetupMsgRing cmd_msg = { 0 }; + + cmd_msg.numPages = adapter->msg_pages; + + base = adapter->msgRingPA; + for (i = 0; i < adapter->msg_pages; i++) { + cmd_msg.ringPPNs[i] = base >> PAGE_SHIFT; + base += PAGE_SIZE; + } + memset(adapter->msg_ring, 0, adapter->msg_pages * PAGE_SIZE); + + pvscsi_write_cmd_desc(adapter, PVSCSI_CMD_SETUP_MSG_RING, + &cmd_msg, sizeof cmd_msg); + } +} + +/* + * Pull a completion descriptor off and pass the completion back + * to the SCSI mid layer. + */ +void pvscsi_complete_request(struct pvscsi_adapter *adapter, + const struct PVSCSIRingCmpDesc *e) +{ + struct pvscsi_ctx *ctx; +#ifndef __FreeBSD__ + struct scsi_cmnd *cmd; +#else /* __FreeBSD__ */ + bool toed = false; + struct ccb_scsiio *cmd; + device_t device = pvscsi_dev(adapter); + int debugerr = 0; +#endif /* __FreeBSD__ */ + u32 btstat = e->hostStatus; + u32 sdstat = e->scsiStatus; + + ctx = pvscsi_get_context(adapter, e->context); + cmd = ctx->cmd; +#ifdef __FreeBSD__ + + /* + * check debugerr failpoints now so that we can do nothing if we're + * delaying the completion with a timer. Only check them once per + * command. + */ + if (!ctx->debugerr_checked) { + ctx->debugerr_checked = true; /* For when this very routine is + * invoked from the FP callout */ + debugerr = pvscsi_debugerr_check(adapter, ctx); + if (debugerr == PVSCSI_DEBUGERR_QUEUED) + return; + } + + callout_stop(&ctx->calloutx); /* disables ABORT or SCSI IO callout */ + toed = ctx->toed; + if (toed) { + device_printf(device, "ccb:%p marked for timeout returned with," + "ctx:%p, h:%u s:%u\n", cmd, ctx, btstat, sdstat); + } +#endif /* __FreeBSD__ */ + pvscsi_unmap_buffers(adapter, ctx); + pvscsi_release_context(adapter, ctx); + +#ifndef __FreeBSD__ + cmd->result = 0; +#endif /* __FreeBSD__ */ + + if (sdstat != SAM_STAT_GOOD && + (btstat == BTSTAT_SUCCESS || + btstat == BTSTAT_LINKED_COMMAND_COMPLETED || + btstat == BTSTAT_LINKED_COMMAND_COMPLETED_WITH_FLAG)) { +#ifndef __FreeBSD__ + if (sdstat == SAM_STAT_COMMAND_TERMINATED) + cmd->result = (DID_RESET << 16); + else { + cmd->result = (DID_OK << 16) | sdstat; + if (sdstat == SAM_STAT_CHECK_CONDITION && + cmd->sense_buffer) + cmd->result |= (DRIVER_SENSE << 24); + } +#else /* __FreeBSD__ */ + cmd->scsi_status = sdstat; + if (sdstat == SAM_STAT_COMMAND_TERMINATED) + cmd->ccb_h.status = CAM_SCSI_BUS_RESET; + else if (sdstat == SAM_STAT_CHECK_CONDITION) + cmd->ccb_h.status = CAM_SCSI_STATUS_ERROR | + CAM_AUTOSNS_VALID; + else + cmd->ccb_h.status = CAM_SCSI_STATUS_ERROR; +#endif /* __FreeBSD__ */ + } else + switch (btstat) { + case BTSTAT_SUCCESS: + case BTSTAT_LINKED_COMMAND_COMPLETED: + case BTSTAT_LINKED_COMMAND_COMPLETED_WITH_FLAG: + /* If everything went fine, let's move on.. */ +#ifndef __FreeBSD__ + cmd->result = (DID_OK << 16); +#else /* __FreeBSD__ */ + cmd->scsi_status = sdstat; + cmd->ccb_h.status = CAM_REQ_CMP; +#endif /* __FreeBSD__ */ + break; + + case BTSTAT_DATARUN: + case BTSTAT_DATA_UNDERRUN: +#ifndef __FreeBSD__ + /* Report residual data in underruns */ + scsi_set_resid(cmd, scsi_bufflen(cmd) - e->dataLen); + cmd->result = (DID_ERROR << 16); +#else /* __FreeBSD__ */ + cmd->scsi_status = sdstat; + cmd->ccb_h.status = CAM_DATA_RUN_ERR; + cmd->resid = cmd->dxfer_len - e->dataLen; +#endif /* __FreeBSD__ */ + break; + + case BTSTAT_SELTIMEO: + /* Our emulation returns this for non-connected devs */ +#ifndef __FreeBSD__ + cmd->result = (DID_BAD_TARGET << 16); +#else /* __FreeBSD__ */ + cmd->scsi_status = sdstat; + cmd->ccb_h.status = CAM_SEL_TIMEOUT; +#endif /* __FreeBSD__ */ + break; + + case BTSTAT_LUNMISMATCH: + case BTSTAT_TAGREJECT: + case BTSTAT_BADMSG: +#ifndef __FreeBSD__ + cmd->result = (DRIVER_INVALID << 24); +#else /* __FreeBSD__ */ + cmd->scsi_status = sdstat; + cmd->ccb_h.status = CAM_LUN_INVALID; + break; +#endif /* __FreeBSD__ */ + /* fall through */ + + case BTSTAT_HAHARDWARE: + case BTSTAT_INVPHASE: + case BTSTAT_HATIMEOUT: + case BTSTAT_NORESPONSE: + case BTSTAT_DISCONNECT: + case BTSTAT_HASOFTWARE: + case BTSTAT_BUSFREE: + case BTSTAT_SENSFAILED: +#ifndef __FreeBSD__ + cmd->result |= (DID_ERROR << 16); +#else /* __FreeBSD__ */ + cmd->scsi_status = sdstat; + cmd->ccb_h.status = CAM_REQ_CMP_ERR; +#endif /* __FreeBSD__ */ + break; + + case BTSTAT_SENTRST: + case BTSTAT_RECVRST: + case BTSTAT_BUSRESET: +#ifndef __FreeBSD__ + cmd->result = (DID_RESET << 16); +#else /* __FreeBSD__ */ + cmd->scsi_status = sdstat; + cmd->ccb_h.status = CAM_SCSI_BUS_RESET; +#endif /* __FreeBSD__ */ + break; + + case BTSTAT_ABORTQUEUE: +#ifndef __FreeBSD__ + /* + * Linux seems to do better with DID_BUS_BUSY instead of + * DID_ABORT. + */ + cmd->result = (DID_BUS_BUSY << 16); +#else /* __FreeBSD__ */ + cmd->scsi_status = sdstat; + device_printf(device, "Command %s\n", toed ? + "timedout" : "aborted"); + if(toed) { + cmd->ccb_h.status = CAM_CMD_TIMEOUT; + } else { + cmd->ccb_h.status = CAM_REQ_ABORTED; + } +#endif /* __FreeBSD__ */ + break; + + case BTSTAT_SCSIPARITY: +#ifndef __FreeBSD__ + cmd->result = (DID_PARITY << 16); +#else /* __FreeBSD__ */ + cmd->scsi_status = sdstat; + cmd->ccb_h.status = CAM_UNCOR_PARITY; +#endif /* __FreeBSD__ */ + break; + + default: +#ifndef __FreeBSD__ + cmd->result = (DID_ERROR << 16); + LOG(0, "Unknown completion status: 0x%x\n", btstat); +#else /* __FreeBSD__ */ + cmd->scsi_status = sdstat; + cmd->ccb_h.status = CAM_REQ_CMP_ERR; + device_printf(device, "Unknown completion status: " + "0x%x\n", btstat); +#endif /* __FreeBSD__ */ + } + +#ifndef __FreeBSD__ + LOG(3, "cmd=%p %x ctx=%p result=0x%x status=0x%x,%x\n", + cmd, cmd->cmnd[0], ctx, cmd->result, btstat, sdstat); + + cmd->scsi_done(cmd); +#else /* __FreeBSD__ */ + if (debugerr != 0) { + /* inject an error */ + union ccb *ccb = (union ccb *)cmd; + ccb->ccb_h.status = CAM_UNCOR_PARITY; + ccb->csio.scsi_status = SCSI_STATUS_CHECK_COND; + } + xpt_done((union ccb *)cmd); +#endif /* __FreeBSD__ */ +} + +/* + * barrier usage : Since the PVSCSI device is emulated, there could be cases + * where we may want to serialize some accesses between the driver and the + * emulation layer. We use compiler barriers instead of the more expensive + * memory barriers because PVSCSI is only supported on X86 which has strong + * memory access ordering. + */ +static void pvscsi_process_completion_ring(struct pvscsi_adapter *adapter) +{ + struct PVSCSIRingsState *s = adapter->rings_state; + struct PVSCSIRingCmpDesc *ring = adapter->cmp_ring; + u32 cmp_entries = s->cmpNumEntriesLog2; + + while (s->cmpConsIdx != s->cmpProdIdx) { + struct PVSCSIRingCmpDesc *e = ring + (s->cmpConsIdx & + MASK(cmp_entries)); + /* + * This barrier() ensures that *e is not dereferenced while + * the device emulation still writes data into the slot. + * Since the device emulation advances s->cmpProdIdx only after + * updating the slot we want to check it first. + */ + barrier(); + pvscsi_complete_request(adapter, e); + /* + * This barrier() ensures that compiler doesn't reorder write + * to s->cmpConsIdx before the read of (*e) inside + * pvscsi_complete_request. Otherwise, device emulation may + * overwrite *e before we had a chance to read it. *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***