From owner-svn-src-head@freebsd.org Mon Jul 6 18:27:43 2015 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5D061A275; Mon, 6 Jul 2015 18:27:43 +0000 (UTC) (envelope-from zbb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BC2B11E0; Mon, 6 Jul 2015 18:27:43 +0000 (UTC) (envelope-from zbb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.14.9/8.14.9) with ESMTP id t66IRhO8016283; Mon, 6 Jul 2015 18:27:43 GMT (envelope-from zbb@FreeBSD.org) Received: (from zbb@localhost) by repo.freebsd.org (8.14.9/8.14.9/Submit) id t66IRgWw016275; Mon, 6 Jul 2015 18:27:42 GMT (envelope-from zbb@FreeBSD.org) Message-Id: <201507061827.t66IRgWw016275@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: zbb set sender to zbb@FreeBSD.org using -f From: Zbigniew Bodek Date: Mon, 6 Jul 2015 18:27:42 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r285213 - in head/sys: arm64/arm64 arm64/include conf X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jul 2015 18:27:43 -0000 Author: zbb Date: Mon Jul 6 18:27:41 2015 New Revision: 285213 URL: https://svnweb.freebsd.org/changeset/base/285213 Log: Introduce ITS support for ARM64 Add ARM ITS (Interrupt Translation Services) support required to bring-up message signalled interrupts on some ARM64 platforms. Obtained from: Semihalf Sponsored by: The FreeBSD Foundation Added: head/sys/arm64/arm64/gic_v3_its.c (contents, props changed) Modified: head/sys/arm64/arm64/gic_v3.c head/sys/arm64/arm64/gic_v3_fdt.c head/sys/arm64/arm64/gic_v3_reg.h head/sys/arm64/arm64/gic_v3_var.h head/sys/arm64/include/param.h head/sys/conf/files.arm64 Modified: head/sys/arm64/arm64/gic_v3.c ============================================================================== --- head/sys/arm64/arm64/gic_v3.c Mon Jul 6 18:27:18 2015 (r285212) +++ head/sys/arm64/arm64/gic_v3.c Mon Jul 6 18:27:41 2015 (r285213) @@ -236,19 +236,18 @@ gic_v3_dispatch(device_t dev, struct tra break; if (__predict_true((active_irq >= GIC_FIRST_PPI && - active_irq <= GIC_LAST_SPI))) { + active_irq <= GIC_LAST_SPI) || active_irq >= GIC_FIRST_LPI)) { arm_dispatch_intr(active_irq, frame); continue; } - if (active_irq <= GIC_LAST_SGI || active_irq >= GIC_FIRST_LPI) { + if (active_irq <= GIC_LAST_SGI) { /* - * TODO: Implement proper SGI/LPI handling. + * TODO: Implement proper SGI handling. * Mask it if such is received for some reason. */ device_printf(dev, - "Received unsupported interrupt type: %s\n", - active_irq >= GIC_FIRST_LPI ? "LPI" : "SGI"); + "Received unsupported interrupt type: SGI\n"); PIC_MASK(dev, active_irq); } } @@ -275,6 +274,8 @@ gic_v3_mask_irq(device_t dev, u_int irq) } else if (irq >= GIC_FIRST_SPI && irq <= GIC_LAST_SPI) { /* SPIs in distributor */ gic_r_write(sc, 4, GICD_ICENABLER(irq), GICD_I_MASK(irq)); gic_v3_wait_for_rwp(sc, DIST); + } else if (irq >= GIC_FIRST_LPI) { /* LPIs */ + lpi_mask_irq(dev, irq); } else panic("%s: Unsupported IRQ number %u", __func__, irq); } @@ -293,6 +294,8 @@ gic_v3_unmask_irq(device_t dev, u_int ir } else if (irq >= GIC_FIRST_SPI && irq <= GIC_LAST_SPI) { /* SPIs in distributor */ gic_d_write(sc, 4, GICD_ISENABLER(irq), GICD_I_MASK(irq)); gic_v3_wait_for_rwp(sc, DIST); + } else if (irq >= GIC_FIRST_LPI) { /* LPIs */ + lpi_unmask_irq(dev, irq); } else panic("%s: Unsupported IRQ number %u", __func__, irq); } Modified: head/sys/arm64/arm64/gic_v3_fdt.c ============================================================================== --- head/sys/arm64/arm64/gic_v3_fdt.c Mon Jul 6 18:27:18 2015 (r285212) +++ head/sys/arm64/arm64/gic_v3_fdt.c Mon Jul 6 18:27:41 2015 (r285213) @@ -35,6 +35,8 @@ __FBSDID("$FreeBSD$"); #include #include +#include + #include #include #include @@ -51,11 +53,27 @@ __FBSDID("$FreeBSD$"); static int gic_v3_fdt_probe(device_t); static int gic_v3_fdt_attach(device_t); +static struct resource *gic_v3_ofw_bus_alloc_res(device_t, device_t, int, int *, + u_long, u_long, u_long, u_int); +static const struct ofw_bus_devinfo *gic_v3_ofw_get_devinfo(device_t, device_t); + static device_method_t gic_v3_fdt_methods[] = { /* Device interface */ DEVMETHOD(device_probe, gic_v3_fdt_probe), DEVMETHOD(device_attach, gic_v3_fdt_attach), + /* Bus interface */ + DEVMETHOD(bus_alloc_resource, gic_v3_ofw_bus_alloc_res), + DEVMETHOD(bus_activate_resource, bus_generic_activate_resource), + + /* ofw_bus interface */ + DEVMETHOD(ofw_bus_get_devinfo, gic_v3_ofw_get_devinfo), + DEVMETHOD(ofw_bus_get_compat, ofw_bus_gen_get_compat), + DEVMETHOD(ofw_bus_get_model, ofw_bus_gen_get_model), + DEVMETHOD(ofw_bus_get_name, ofw_bus_gen_get_name), + DEVMETHOD(ofw_bus_get_node, ofw_bus_gen_get_node), + DEVMETHOD(ofw_bus_get_type, ofw_bus_gen_get_type), + /* End */ DEVMETHOD_END }; @@ -71,6 +89,11 @@ EARLY_DRIVER_MODULE(gic_v3, ofwbus, gic_ 0, 0, BUS_PASS_INTERRUPT + BUS_PASS_ORDER_MIDDLE); /* + * Helper functions declarations. + */ +static int gic_v3_ofw_bus_attach(device_t); + +/* * Device interface. */ static int @@ -109,6 +132,17 @@ gic_v3_fdt_attach(device_t dev) err = gic_v3_attach(dev); if (err) goto error; + /* + * Try to register ITS to this GIC. + * GIC will act as a bus in that case. + * Failure here will not affect main GIC functionality. + */ + if (gic_v3_ofw_bus_attach(dev) != 0) { + if (bootverbose) { + device_printf(dev, + "Failed to attach ITS to this GIC\n"); + } + } return (err); @@ -122,3 +156,155 @@ error: return (err); } + +/* OFW bus interface */ +struct gic_v3_ofw_devinfo { + struct ofw_bus_devinfo di_dinfo; + struct resource_list di_rl; +}; + +static const struct ofw_bus_devinfo * +gic_v3_ofw_get_devinfo(device_t bus __unused, device_t child) +{ + struct gic_v3_ofw_devinfo *di; + + di = device_get_ivars(child); + return (&di->di_dinfo); +} + +static struct resource * +gic_v3_ofw_bus_alloc_res(device_t bus, device_t child, int type, int *rid, + u_long start, u_long end, u_long count, u_int flags) +{ + struct gic_v3_ofw_devinfo *di; + struct resource_list_entry *rle; + int ranges_len; + + if ((start == 0UL) && (end == ~0UL)) { + if ((di = device_get_ivars(child)) == NULL) + return (NULL); + if (type != SYS_RES_MEMORY) + return (NULL); + + /* Find defaults for this rid */ + rle = resource_list_find(&di->di_rl, type, *rid); + if (rle == NULL) + return (NULL); + + start = rle->start; + end = rle->end; + count = rle->count; + } + /* + * XXX: No ranges remap! + * Absolute address is expected. + */ + if (ofw_bus_has_prop(bus, "ranges")) { + ranges_len = OF_getproplen(ofw_bus_get_node(bus), "ranges"); + if (ranges_len != 0) { + if (bootverbose) { + device_printf(child, + "Ranges remap not supported\n"); + } + return (NULL); + } + } + return (bus_generic_alloc_resource(bus, child, type, rid, start, end, + count, flags)); +} + +/* Helper functions */ + +/* + * Bus capability support for GICv3. + * Collects and configures device informations and finally + * adds ITS device as a child of GICv3 in Newbus hierarchy. + */ +static int +gic_v3_ofw_bus_attach(device_t dev) +{ + struct gic_v3_ofw_devinfo *di; + device_t child; + phandle_t parent, node; + pcell_t addr_cells, size_cells; + + parent = ofw_bus_get_node(dev); + if (parent > 0) { + addr_cells = 2; + OF_getencprop(parent, "#address-cells", &addr_cells, + sizeof(addr_cells)); + size_cells = 2; + OF_getencprop(parent, "#size-cells", &size_cells, + sizeof(size_cells)); + /* Iterate through all GIC subordinates */ + for (node = OF_child(parent); node > 0; node = OF_peer(node)) { + /* Allocate and populate devinfo. */ + di = malloc(sizeof(*di), M_GIC_V3, M_WAITOK | M_ZERO); + if (ofw_bus_gen_setup_devinfo(&di->di_dinfo, node)) { + if (bootverbose) { + device_printf(dev, + "Could not set up devinfo for ITS\n"); + } + free(di, M_GIC_V3); + continue; + } + + /* Initialize and populate resource list. */ + resource_list_init(&di->di_rl); + ofw_bus_reg_to_rl(dev, node, addr_cells, size_cells, + &di->di_rl); + + /* Should not have any interrupts, so don't add any */ + + /* Add newbus device for this FDT node */ + child = device_add_child(dev, NULL, -1); + if (!child) { + if (bootverbose) { + device_printf(dev, + "Could not add child: %s\n", + di->di_dinfo.obd_name); + } + resource_list_free(&di->di_rl); + ofw_bus_gen_destroy_devinfo(&di->di_dinfo); + free(di, M_GIC_V3); + continue; + } + + device_set_ivars(child, di); + } + } + + return (bus_generic_attach(dev)); +} + +static int gic_v3_its_fdt_probe(device_t dev); + +static device_method_t gic_v3_its_fdt_methods[] = { + /* Device interface */ + DEVMETHOD(device_probe, gic_v3_its_fdt_probe), + + /* End */ + DEVMETHOD_END +}; + +DEFINE_CLASS_1(gic_v3_its, gic_v3_its_fdt_driver, gic_v3_its_fdt_methods, + sizeof(struct gic_v3_its_softc), gic_v3_its_driver); + +static devclass_t gic_v3_its_fdt_devclass; + +EARLY_DRIVER_MODULE(gic_v3_its, gic_v3, gic_v3_its_fdt_driver, + gic_v3_its_fdt_devclass, 0, 0, BUS_PASS_INTERRUPT + BUS_PASS_ORDER_MIDDLE); + +static int +gic_v3_its_fdt_probe(device_t dev) +{ + + if (!ofw_bus_status_okay(dev)) + return (ENXIO); + + if (!ofw_bus_is_compatible(dev, GIC_V3_ITS_COMPSTR)) + return (ENXIO); + + device_set_desc(dev, GIC_V3_ITS_DEVSTR); + return (BUS_PROBE_DEFAULT); +} Added: head/sys/arm64/arm64/gic_v3_its.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sys/arm64/arm64/gic_v3_its.c Mon Jul 6 18:27:41 2015 (r285213) @@ -0,0 +1,1448 @@ +/*- + * Copyright (c) 2015 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Semihalf under + * the sponsorship of the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include +#include + +#include +#include +#include +#include + +#include "gic_v3_reg.h" +#include "gic_v3_var.h" + +#include "pic_if.h" + +/* Device and PIC methods */ +static int gic_v3_its_attach(device_t); + +static device_method_t gic_v3_its_methods[] = { + /* Device interface */ + DEVMETHOD(device_attach, gic_v3_its_attach), + /* + * PIC interface + */ + /* MSI-X */ + DEVMETHOD(pic_alloc_msix, gic_v3_its_alloc_msix), + DEVMETHOD(pic_map_msix, gic_v3_its_map_msix), + /* MSI */ + DEVMETHOD(pic_alloc_msi, gic_v3_its_alloc_msi), + DEVMETHOD(pic_map_msi, gic_v3_its_map_msix), + + /* End */ + DEVMETHOD_END +}; + +DEFINE_CLASS_0(gic_v3_its, gic_v3_its_driver, gic_v3_its_methods, + sizeof(struct gic_v3_its_softc)); + +MALLOC_DEFINE(M_GIC_V3_ITS, "GICv3 ITS", GIC_V3_ITS_DEVSTR); + +static int its_alloc_tables(struct gic_v3_its_softc *); +static void its_free_tables(struct gic_v3_its_softc *); +static void its_init_commandq(struct gic_v3_its_softc *); +static int its_init_cpu(struct gic_v3_its_softc *); +static void its_init_cpu_collection(struct gic_v3_its_softc *); + +static int its_cmd_send(struct gic_v3_its_softc *, struct its_cmd_desc *); + +static void its_cmd_mapc(struct gic_v3_its_softc *, struct its_col *, uint8_t); +static void its_cmd_mapvi(struct gic_v3_its_softc *, struct its_dev *, uint32_t, + uint32_t); +static void its_cmd_mapi(struct gic_v3_its_softc *, struct its_dev *, uint32_t); +static void its_cmd_inv(struct gic_v3_its_softc *, struct its_dev *, uint32_t); +static void its_cmd_invall(struct gic_v3_its_softc *, struct its_col *); + +static void lpi_init_conftable(struct gic_v3_its_softc *); +static void lpi_bitmap_init(struct gic_v3_its_softc *); +static void lpi_init_cpu(struct gic_v3_its_softc *); +static int lpi_config_cpu(struct gic_v3_its_softc *); + +const char *its_ptab_cache[] = { + [GITS_BASER_CACHE_NCNB] = "(NC,NB)", + [GITS_BASER_CACHE_NC] = "(NC)", + [GITS_BASER_CACHE_RAWT] = "(RA,WT)", + [GITS_BASER_CACHE_RAWB] = "(RA,WB)", + [GITS_BASER_CACHE_WAWT] = "(WA,WT)", + [GITS_BASER_CACHE_WAWB] = "(WA,WB)", + [GITS_BASER_CACHE_RAWAWT] = "(RAWA,WT)", + [GITS_BASER_CACHE_RAWAWB] = "(RAWA,WB)", +}; + +const char *its_ptab_share[] = { + [GITS_BASER_SHARE_NS] = "none", + [GITS_BASER_SHARE_IS] = "inner", + [GITS_BASER_SHARE_OS] = "outer", + [GITS_BASER_SHARE_RES] = "none", +}; + +const char *its_ptab_type[] = { + [GITS_BASER_TYPE_UNIMPL] = "Unimplemented", + [GITS_BASER_TYPE_DEV] = "Devices", + [GITS_BASER_TYPE_VP] = "Virtual Processors", + [GITS_BASER_TYPE_PP] = "Physical Processors", + [GITS_BASER_TYPE_IC] = "Interrupt Collections", + [GITS_BASER_TYPE_RES5] = "Reserved (5)", + [GITS_BASER_TYPE_RES6] = "Reserved (6)", + [GITS_BASER_TYPE_RES7] = "Reserved (7)", +}; + +static struct gic_v3_its_softc *its_sc; + +#define gic_its_read(sc, len, reg) \ + bus_read_##len(&sc->its_res[0], reg) + +#define gic_its_write(sc, len, reg, val) \ + bus_write_##len(&sc->its_res[0], reg, val) + +static int +gic_v3_its_attach(device_t dev) +{ + struct gic_v3_its_softc *sc; + uint64_t gits_tmp; + uint32_t gits_pidr2; + int rid; + int ret; + + sc = device_get_softc(dev); + + /* + * Initialize sleep & spin mutex for ITS + */ + /* Protects ITS device list and assigned LPIs bitmaps. */ + mtx_init(&sc->its_mtx, "ITS sleep lock", NULL, MTX_DEF); + /* Protects access to ITS command circular buffer. */ + mtx_init(&sc->its_spin_mtx, "ITS spin lock", NULL, MTX_SPIN); + + rid = 0; + sc->its_res = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &rid, + RF_ACTIVE); + if (sc->its_res == NULL) { + device_printf(dev, "Could not allocate memory\n"); + return (ENXIO); + } + + sc->dev = dev; + + gits_pidr2 = gic_its_read(sc, 4, GITS_PIDR2); + switch (gits_pidr2 & GITS_PIDR2_ARCH_MASK) { + case GITS_PIDR2_ARCH_GICv3: /* fall through */ + case GITS_PIDR2_ARCH_GICv4: + if (bootverbose) { + device_printf(dev, "ITS found. Architecture rev. %u\n", + (u_int)(gits_pidr2 & GITS_PIDR2_ARCH_MASK) >> 4); + } + break; + default: + device_printf(dev, "No ITS found in the system\n"); + gic_v3_its_detach(dev); + return (ENODEV); + } + + /* 1. Initialize commands queue */ + its_init_commandq(sc); + + /* 2. Provide memory for any private ITS tables */ + ret = its_alloc_tables(sc); + if (ret != 0) { + gic_v3_its_detach(dev); + return (ret); + } + + /* 3. Allocate collections. One per-CPU */ + sc->its_cols = malloc(sizeof(*sc->its_cols) * MAXCPU, + M_GIC_V3_ITS, (M_WAITOK | M_ZERO)); + + /* 4. Enable ITS in GITS_CTLR */ + gits_tmp = gic_its_read(sc, 4, GITS_CTLR); + gic_its_write(sc, 4, GITS_CTLR, gits_tmp | GITS_CTLR_EN); + + /* 5. Initialize LPIs configuration table */ + lpi_init_conftable(sc); + + /* 6. LPIs bitmap init */ + lpi_bitmap_init(sc); + + /* 7. CPU init */ + (void)its_init_cpu(sc); + + /* 8. Init ITS devices list */ + TAILQ_INIT(&sc->its_dev_list); + + arm_register_msi_pic(dev); + + /* + * XXX ARM64TODO: We need to have ITS software context + * when being called by the interrupt code (mask/unmask). + * This may be used only when one ITS is present in + * the system and eventually should be removed. + */ + KASSERT(its_sc == NULL, + ("Trying to assign its_sc that is already set")); + its_sc = sc; + + return (0); +} + +/* Will not detach but use it for convenience */ +int +gic_v3_its_detach(device_t dev) +{ + device_t parent; + struct gic_v3_softc *gic_sc; + struct gic_v3_its_softc *sc; + u_int cpuid; + int rid = 0; + + sc = device_get_softc(dev); + cpuid = PCPU_GET(cpuid); + + /* Release what's possible */ + + /* Command queue */ + if ((void *)sc->its_cmdq_base != NULL) { + contigfree((void *)sc->its_cmdq_base, + ITS_CMDQ_SIZE, M_GIC_V3_ITS); + } + /* ITTs */ + its_free_tables(sc); + /* Collections */ + free(sc->its_cols, M_GIC_V3_ITS); + /* LPI config table */ + parent = device_get_parent(sc->dev); + gic_sc = device_get_softc(parent); + if ((void *)gic_sc->gic_redists.lpis.conf_base != NULL) { + contigfree((void *)gic_sc->gic_redists.lpis.conf_base, + LPI_CONFTAB_SIZE, M_GIC_V3_ITS); + } + if ((void *)gic_sc->gic_redists.lpis.pend_base[cpuid] != NULL) { + contigfree((void *)gic_sc->gic_redists.lpis.pend_base[cpuid], + roundup2(LPI_PENDTAB_SIZE, PAGE_SIZE_64K), M_GIC_V3_ITS); + } + + /* Resource... */ + bus_release_resource(dev, SYS_RES_MEMORY, rid, sc->its_res); + + /* XXX ARM64TODO: Reset global pointer to ITS software context */ + its_sc = NULL; + + return (0); +} + +static int +its_alloc_tables(struct gic_v3_its_softc *sc) +{ + uint64_t gits_baser, gits_tmp; + uint64_t type, esize, cache, share, psz; + uint64_t gits_typer; + size_t page_size, npages, nitspages, nidents, tn; + size_t its_tbl_size; + vm_offset_t ptab_vaddr; + vm_paddr_t ptab_paddr; + boolean_t first = TRUE; + + page_size = PAGE_SIZE_64K; + + /* Read features first */ + gits_typer = gic_its_read(sc, 8, GITS_TYPER); + + for (tn = 0; tn < GITS_BASER_NUM; tn++) { + gits_baser = gic_its_read(sc, 8, GITS_BASER(tn)); + type = GITS_BASER_TYPE(gits_baser); + /* Get the Table Entry size */ + esize = GITS_BASER_ESIZE(gits_baser); + + switch (type) { + case GITS_BASER_TYPE_UNIMPL: /* fall through */ + case GITS_BASER_TYPE_RES5: + case GITS_BASER_TYPE_RES6: + case GITS_BASER_TYPE_RES7: + continue; + case GITS_BASER_TYPE_DEV: + nidents = (1 << GITS_TYPER_DEVB(gits_typer)); + its_tbl_size = esize * nidents; + its_tbl_size = roundup2(its_tbl_size, page_size); + npages = howmany(its_tbl_size, PAGE_SIZE); + break; + default: + npages = howmany(page_size, PAGE_SIZE); + break; + } + + /* Allocate required space */ + ptab_vaddr = (vm_offset_t)contigmalloc(npages * PAGE_SIZE, + M_GIC_V3_ITS, (M_WAITOK | M_ZERO), 0, ~0UL, PAGE_SIZE, 0); + + sc->its_ptabs[tn].ptab_vaddr = ptab_vaddr; + sc->its_ptabs[tn].ptab_pgsz = PAGE_SIZE; + sc->its_ptabs[tn].ptab_npages = npages; + + ptab_paddr = vtophys(ptab_vaddr); + KASSERT((ptab_paddr & GITS_BASER_PA_MASK) == ptab_paddr, + ("%s: Unaligned PA for Interrupt Translation Table", + device_get_name(sc->dev))); + + /* Set defaults: WAWB, IS */ + cache = GITS_BASER_CACHE_WAWB; + share = GITS_BASER_SHARE_IS; + + for (;;) { + nitspages = howmany(its_tbl_size, page_size); + + switch (page_size) { + case PAGE_SIZE: /* 4KB */ + psz = GITS_BASER_PSZ_4K; + break; + case PAGE_SIZE_16K: /* 16KB */ + psz = GITS_BASER_PSZ_4K; + break; + case PAGE_SIZE_64K: /* 64KB */ + psz = GITS_BASER_PSZ_64K; + break; + default: + device_printf(sc->dev, + "Unsupported page size: %zuKB\n", + (page_size / 1024)); + its_free_tables(sc); + return (ENXIO); + } + + /* Clear fields under modification first */ + gits_baser &= ~(GITS_BASER_VALID | + GITS_BASER_CACHE_MASK | GITS_BASER_TYPE_MASK | + GITS_BASER_ESIZE_MASK | GITS_BASER_PA_MASK | + GITS_BASER_SHARE_MASK | GITS_BASER_PSZ_MASK | + GITS_BASER_SIZE_MASK); + /* Construct register value */ + gits_baser |= + (type << GITS_BASER_TYPE_SHIFT) | + ((esize - 1) << GITS_BASER_ESIZE_SHIFT) | + (cache << GITS_BASER_CACHE_SHIFT) | + (share << GITS_BASER_SHARE_SHIFT) | + (psz << GITS_BASER_PSZ_SHIFT) | + ptab_paddr | (nitspages - 1) | + GITS_BASER_VALID; + + gic_its_write(sc, 8, GITS_BASER(tn), gits_baser); + /* + * Verify. + * Depending on implementation we may encounter + * shareability and page size mismatch. + */ + gits_tmp = gic_its_read(sc, 8, GITS_BASER(tn)); + if (((gits_tmp ^ gits_baser) & GITS_BASER_SHARE_MASK) != 0) { + share = gits_tmp & GITS_BASER_SHARE_MASK; + share >>= GITS_BASER_SHARE_SHIFT; + continue; + } + + if (((gits_tmp ^ gits_baser) & GITS_BASER_PSZ_MASK) != 0) { + switch (page_size) { + case PAGE_SIZE_16K: + /* Drop to 4KB page */ + page_size = PAGE_SIZE; + continue; + case PAGE_SIZE_64K: + /* Drop to 16KB page */ + page_size = PAGE_SIZE_16K; + continue; + } + } + /* + * All possible adjustments should + * be applied by now so just break the loop. + */ + break; + } + /* + * Do not compare Cacheability field since + * it is implementation defined. + */ + gits_tmp &= ~GITS_BASER_CACHE_MASK; + gits_baser &= ~GITS_BASER_CACHE_MASK; + + if (gits_tmp != gits_baser) { + device_printf(sc->dev, + "Could not allocate ITS tables\n"); + its_free_tables(sc); + return (ENXIO); + } + + if (bootverbose) { + if (first) { + device_printf(sc->dev, + "Allocated ITS private tables:\n"); + first = FALSE; + } + device_printf(sc->dev, + "\tPTAB%zu for %s: PA 0x%lx," + " %lu entries," + " cache policy %s, %s shareable," + " page size %zuKB\n", + tn, its_ptab_type[type], ptab_paddr, + (page_size * nitspages) / esize, + its_ptab_cache[cache], its_ptab_share[share], + page_size / 1024); + } + } + + return (0); +} + +static void +its_free_tables(struct gic_v3_its_softc *sc) +{ + vm_offset_t ptab_vaddr; + size_t size; + size_t tn; + + for (tn = 0; tn < GITS_BASER_NUM; tn++) { + ptab_vaddr = sc->its_ptabs[tn].ptab_vaddr; + if (ptab_vaddr == 0) + continue; + size = sc->its_ptabs[tn].ptab_pgsz; + size *= sc->its_ptabs[tn].ptab_npages; + + if ((void *)ptab_vaddr != NULL) + contigfree((void *)ptab_vaddr, size, M_GIC_V3_ITS); + + /* Clear the table description */ + memset(&sc->its_ptabs[tn], 0, sizeof(sc->its_ptabs[tn])); + } +} + +static void +its_init_commandq(struct gic_v3_its_softc *sc) +{ + uint64_t gits_cbaser, gits_tmp; + uint64_t cache, share; + vm_paddr_t cmdq_paddr; + device_t dev; + + dev = sc->dev; + /* Allocate memory for command queue */ + sc->its_cmdq_base = contigmalloc(ITS_CMDQ_SIZE, M_GIC_V3_ITS, + (M_WAITOK | M_ZERO), 0, ~0UL, ITS_CMDQ_SIZE, 0); + /* Set command queue write pointer (command queue empty) */ + sc->its_cmdq_write = sc->its_cmdq_base; + + /* Save command queue pointer and attributes */ + cmdq_paddr = vtophys(sc->its_cmdq_base); + + /* Set defaults: Normal Inner WAWB, IS */ + cache = GITS_CBASER_CACHE_NIWAWB; + share = GITS_CBASER_SHARE_IS; + + gits_cbaser = (cmdq_paddr | + (cache << GITS_CBASER_CACHE_SHIFT) | + (share << GITS_CBASER_SHARE_SHIFT) | + /* Number of 4KB pages - 1 */ + ((ITS_CMDQ_SIZE / PAGE_SIZE) - 1) | + /* Valid bit */ + GITS_CBASER_VALID); + + gic_its_write(sc, 8, GITS_CBASER, gits_cbaser); + gits_tmp = gic_its_read(sc, 8, GITS_CBASER); + + if (((gits_tmp ^ gits_cbaser) & GITS_CBASER_SHARE_MASK) != 0) { + if (bootverbose) { + device_printf(dev, + "Will use cache flushing for commands queue\n"); + } + /* Command queue needs cache flushing */ + sc->its_flags |= ITS_FLAGS_CMDQ_FLUSH; + } + + gic_its_write(sc, 8, GITS_CWRITER, 0x0); +} + +static int +its_init_cpu(struct gic_v3_its_softc *sc) +{ + device_t parent; + struct gic_v3_softc *gic_sc; + + /* + * Check for LPIs support on this Re-Distributor. + */ + parent = device_get_parent(sc->dev); + gic_sc = device_get_softc(parent); + if ((gic_r_read(gic_sc, 4, GICR_TYPER) & GICR_TYPER_PLPIS) == 0) { + if (bootverbose) { + device_printf(sc->dev, + "LPIs not supported on CPU%u\n", PCPU_GET(cpuid)); + } + return (ENXIO); + } + + /* Initialize LPIs for this CPU */ + lpi_init_cpu(sc); + + /* Initialize collections */ + its_init_cpu_collection(sc); + + return (0); +} + +static void +its_init_cpu_collection(struct gic_v3_its_softc *sc) +{ + device_t parent; + struct gic_v3_softc *gic_sc; + uint64_t typer; + uint64_t target; + vm_offset_t redist_base; + u_int cpuid; + + cpuid = PCPU_GET(cpuid); + parent = device_get_parent(sc->dev); + gic_sc = device_get_softc(parent); + + typer = gic_its_read(sc, 8, GITS_TYPER); + if ((typer & GITS_TYPER_PTA) != 0) { + redist_base = + rman_get_bushandle(gic_sc->gic_redists.pcpu[cpuid]); + /* + * Target Address correspond to the base physical + * address of Re-Distributors. + */ + target = vtophys(redist_base); + } else { + /* Target Address correspond to unique processor numbers */ + typer = gic_r_read(gic_sc, 8, GICR_TYPER); + target = GICR_TYPER_CPUNUM(typer); + } + + sc->its_cols[cpuid].col_target = target; + sc->its_cols[cpuid].col_id = cpuid; + + its_cmd_mapc(sc, &sc->its_cols[cpuid], 1); + its_cmd_invall(sc, &sc->its_cols[cpuid]); +} + +static void +lpi_init_conftable(struct gic_v3_its_softc *sc) +{ + device_t parent; + struct gic_v3_softc *gic_sc; + vm_offset_t conf_base; + uint8_t prio_default; + + parent = device_get_parent(sc->dev); + gic_sc = device_get_softc(parent); + /* + * LPI Configuration Table settings. + * Notice that Configuration Table is shared among all + * Re-Distributors, so this is going to be created just once. + */ + conf_base = (vm_offset_t)contigmalloc(LPI_CONFTAB_SIZE, + M_GIC_V3_ITS, (M_WAITOK | M_ZERO), 0, ~0UL, PAGE_SIZE_64K, 0); + + if (bootverbose) { + device_printf(sc->dev, + "LPI Configuration Table at PA: 0x%lx\n", + vtophys(conf_base)); + } + + /* + * Let the default priority be aligned with all other + * interrupts assuming that each interrupt is assigned + * MAX priority at startup. MAX priority on the other + * hand cannot be higher than 0xFC for LPIs. + */ + prio_default = GIC_PRIORITY_MAX; + + /* Write each settings byte to LPI configuration table */ + memset((void *)conf_base, + (prio_default & LPI_CONF_PRIO_MASK) | LPI_CONF_GROUP1, + LPI_CONFTAB_SIZE); + + cpu_dcache_wb_range((vm_offset_t)conf_base, roundup2(LPI_CONFTAB_SIZE, + PAGE_SIZE_64K)); + + gic_sc->gic_redists.lpis.conf_base = conf_base; +} + +static void +lpi_init_cpu(struct gic_v3_its_softc *sc) +{ + device_t parent; + struct gic_v3_softc *gic_sc; + vm_offset_t pend_base; + u_int cpuid; + + parent = device_get_parent(sc->dev); + gic_sc = device_get_softc(parent); + + /* + * LPI Pending Table settings. + * This has to be done for each Re-Distributor, hence for each CPU. + */ + cpuid = PCPU_GET(cpuid); + + pend_base = (vm_offset_t)contigmalloc( + roundup2(LPI_PENDTAB_SIZE, PAGE_SIZE_64K), M_GIC_V3_ITS, + (M_WAITOK | M_ZERO), 0, ~0UL, PAGE_SIZE_64K, 0); + + /* Clean D-cache so that ITS can see zeroed pages */ + cpu_dcache_wb_range((vm_offset_t)pend_base, + roundup2(LPI_PENDTAB_SIZE, PAGE_SIZE_64K)); + + if (bootverbose) { + device_printf(sc->dev, + "LPI Pending Table for CPU%u at PA: 0x%lx\n", + cpuid, vtophys(pend_base)); + } + + gic_sc->gic_redists.lpis.pend_base[cpuid] = pend_base; + + lpi_config_cpu(sc); +} + +static int +lpi_config_cpu(struct gic_v3_its_softc *sc) +{ + device_t parent; + struct gic_v3_softc *gic_sc; + vm_offset_t conf_base, pend_base; + uint64_t gicr_xbaser, gicr_temp; + uint64_t cache, share, idbits; + uint32_t gicr_ctlr; + u_int cpuid; + + parent = device_get_parent(sc->dev); + gic_sc = device_get_softc(parent); + cpuid = PCPU_GET(cpuid); + + conf_base = gic_sc->gic_redists.lpis.conf_base; + pend_base = gic_sc->gic_redists.lpis.pend_base[cpuid]; + + /* Disable LPIs */ + gicr_ctlr = gic_r_read(gic_sc, 4, GICR_CTLR); + gicr_ctlr &= ~GICR_CTLR_LPI_ENABLE; + gic_r_write(gic_sc, 4, GICR_CTLR, gicr_ctlr); + /* Perform full system barrier */ + dsb(sy); + + /* + * Set GICR_PROPBASER + */ + + /* + * Find out how many bits do we need for LPI identifiers. + * Remark 1.: Even though we have (LPI_CONFTAB_SIZE / 8) LPIs + * the notified LPI ID still starts from 8192 + * (GIC_FIRST_LPI). + * Remark 2.: This could be done on compilation time but there + * seems to be no sufficient macro. + */ + idbits = flsl(LPI_CONFTAB_SIZE + GIC_FIRST_LPI) - 1; + + /* Set defaults: Normal Inner WAWB, IS */ + cache = GICR_PROPBASER_CACHE_NIWAWB; + share = GICR_PROPBASER_SHARE_IS; + + gicr_xbaser = vtophys(conf_base) | + ((idbits - 1) & GICR_PROPBASER_IDBITS_MASK) | + (cache << GICR_PROPBASER_CACHE_SHIFT) | + (share << GICR_PROPBASER_SHARE_SHIFT); + + gic_r_write(gic_sc, 8, GICR_PROPBASER, gicr_xbaser); + gicr_temp = gic_r_read(gic_sc, 8, GICR_PROPBASER); + + if (((gicr_xbaser ^ gicr_temp) & GICR_PROPBASER_SHARE_MASK) != 0) { + if (bootverbose) { + device_printf(sc->dev, + "Will use cache flushing for LPI " + "Configuration Table\n"); + } + gic_sc->gic_redists.lpis.flags |= LPI_FLAGS_CONF_FLUSH; + } + + /* + * Set GICR_PENDBASER + */ + + /* Set defaults: Normal Inner WAWB, IS */ + cache = GICR_PENDBASER_CACHE_NIWAWB; + share = GICR_PENDBASER_SHARE_IS; + + gicr_xbaser = vtophys(pend_base) | + (cache << GICR_PENDBASER_CACHE_SHIFT) | + (share << GICR_PENDBASER_SHARE_SHIFT); + + gic_r_write(gic_sc, 8, GICR_PENDBASER, gicr_xbaser); + + /* Enable LPIs */ + gicr_ctlr = gic_r_read(gic_sc, 4, GICR_CTLR); + gicr_ctlr |= GICR_CTLR_LPI_ENABLE; + gic_r_write(gic_sc, 4, GICR_CTLR, gicr_ctlr); + + dsb(sy); + *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***