From owner-svn-src-stable@freebsd.org Wed Nov 30 06:26:39 2016 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E41A4C5D5C2; Wed, 30 Nov 2016 06:26:39 +0000 (UTC) (envelope-from dexuan@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AC9991451; Wed, 30 Nov 2016 06:26:39 +0000 (UTC) (envelope-from dexuan@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id uAU6QcBx077450; Wed, 30 Nov 2016 06:26:38 GMT (envelope-from dexuan@FreeBSD.org) Received: (from dexuan@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id uAU6QcBq077442; Wed, 30 Nov 2016 06:26:38 GMT (envelope-from dexuan@FreeBSD.org) Message-Id: <201611300626.uAU6QcBq077442@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dexuan set sender to dexuan@FreeBSD.org using -f From: Dexuan Cui Date: Wed, 30 Nov 2016 06:26:38 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r309313 - in stable/10/sys: conf dev/hyperv/pcib dev/hyperv/vmbus modules/hyperv/pcib modules/hyperv/vmbus X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Nov 2016 06:26:40 -0000 Author: dexuan Date: Wed Nov 30 06:26:37 2016 New Revision: 309313 URL: https://svnweb.freebsd.org/changeset/base/309313 Log: MFC: 308723-308725,308793-308795,309127 Approved by: sephe (mentor) r308723 hyperv/vmbus: add a new method to get vcpu_id vcpu_id is host's representation of guest CPU. We get the mapping between vcpu_id and FreeBSD kernel's cpu id when VMBus driver is loaded. Later, when a driver, like the coming pcib driver, talks to the host and needs to refer to a guest CPU, the driver must use the vcpu_id. Reviewed by: jhb, sephe Approved by: sephe (mentor) Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8410 r308724 hyperv/vmbus: add new vmbus methods to support PCIe pass-through The new methods will be used by the coming pcib driver. Reviewed by: sephe Approved by: sephe (mentor) Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8409 r308725 hyperv/pcib: enable PCIe pass-through (a.k.a. Discrete Device Assignment) The feature enables us to pass through physical PCIe devices to FreeBSD VM running on Hyper-V (Windows Server 2016) to get near-native performance with low CPU utilization. The patch implements a PCI bridge driver to support the feature: 1) The pcib driver talks to the host to discover device(s) and presents the device(s) to FreeBSD's pci driver via PCI configuration space (note: to access the configuration space, we don't use the standard I/O port 0xCF8/CFC method; instead, we use an MMIO-based method supplied by Hyper-V, which is very similar to the 0xCF8/CFC method). 2) The pcib driver allocates resources for the device(s) and initialize the related BARs, when the device driver's attach method is invoked; 3) The pcib driver talks to the host to create MSI/MSI-X interrupt remapping between the guest and the host; 4) The pcib driver supports device hot add/remove. Reviewed by: sephe Approved by: sephe (mentor) Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8332 r308793 hyperv/pcib: Fix the build for some kernel configs Add the dependency on pci explicitly for the pcib and vmbus drivers. The related Makefiles are updated accordingly too. Reviewed by: sephe Approved by: sephe (mentor) Sponsored by: Microsoft r308794 hyperv/vmbus,pcib: Add MODULE_DEPEND on pci We'd better add this dependency explicitly, though usually the pci driver is built into the kernel by default. Reviewed by: sephe Approved by: sephe (mentor) Sponsored by: Microsoft r308795 hyperv/pcib: change the file name: pcib.c -> vmbus_pcib.c This makes the file name and the variable naming in the file consistent. Reviewed by: sephe Approved by: sephe (mentor) Sponsored by: Microsoft r309127 hyperv/vmbus,pcib: unbreak build in case NEW_PCIB is undefined vmbus_pcib requires NEW_PCIB, but in case that's not defined, we at least shouldn't break build. Reviewed by: sephe Approved by: sephe (mentor) Sponsored by: Microsoft Added: stable/10/sys/dev/hyperv/pcib/ - copied from r308725, head/sys/dev/hyperv/pcib/ stable/10/sys/dev/hyperv/pcib/vmbus_pcib.c - copied, changed from r308795, head/sys/dev/hyperv/pcib/vmbus_pcib.c stable/10/sys/modules/hyperv/pcib/ - copied from r308725, head/sys/modules/hyperv/pcib/ Deleted: stable/10/sys/dev/hyperv/pcib/pcib.c Modified: stable/10/sys/conf/files.amd64 stable/10/sys/conf/files.i386 stable/10/sys/dev/hyperv/vmbus/vmbus.c stable/10/sys/dev/hyperv/vmbus/vmbus_if.m stable/10/sys/dev/hyperv/vmbus/vmbus_var.h stable/10/sys/modules/hyperv/pcib/Makefile stable/10/sys/modules/hyperv/vmbus/Makefile Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/conf/files.amd64 ============================================================================== --- stable/10/sys/conf/files.amd64 Wed Nov 30 06:20:43 2016 (r309312) +++ stable/10/sys/conf/files.amd64 Wed Nov 30 06:26:37 2016 (r309313) @@ -261,6 +261,7 @@ dev/hwpmc/hwpmc_uncore.c optional hwpmc dev/hwpmc/hwpmc_piv.c optional hwpmc dev/hwpmc/hwpmc_tsc.c optional hwpmc dev/hwpmc/hwpmc_x86.c optional hwpmc +dev/hyperv/pcib/vmbus_pcib.c optional hyperv pci dev/hyperv/netvsc/hn_nvs.c optional hyperv dev/hyperv/netvsc/hn_rndis.c optional hyperv dev/hyperv/netvsc/if_hn.c optional hyperv @@ -272,7 +273,7 @@ dev/hyperv/utilities/hv_timesync.c opt dev/hyperv/utilities/hv_util.c optional hyperv dev/hyperv/vmbus/hyperv.c optional hyperv dev/hyperv/vmbus/hyperv_busdma.c optional hyperv -dev/hyperv/vmbus/vmbus.c optional hyperv +dev/hyperv/vmbus/vmbus.c optional hyperv pci dev/hyperv/vmbus/vmbus_br.c optional hyperv dev/hyperv/vmbus/vmbus_chan.c optional hyperv dev/hyperv/vmbus/vmbus_et.c optional hyperv Modified: stable/10/sys/conf/files.i386 ============================================================================== --- stable/10/sys/conf/files.i386 Wed Nov 30 06:20:43 2016 (r309312) +++ stable/10/sys/conf/files.i386 Wed Nov 30 06:26:37 2016 (r309313) @@ -238,6 +238,7 @@ dev/hwpmc/hwpmc_piv.c optional hwpmc dev/hwpmc/hwpmc_ppro.c optional hwpmc dev/hwpmc/hwpmc_tsc.c optional hwpmc dev/hwpmc/hwpmc_x86.c optional hwpmc +dev/hyperv/pcib/vmbus_pcib.c optional hyperv pci dev/hyperv/netvsc/hn_nvs.c optional hyperv dev/hyperv/netvsc/hn_rndis.c optional hyperv dev/hyperv/netvsc/if_hn.c optional hyperv @@ -249,7 +250,7 @@ dev/hyperv/utilities/hv_timesync.c opt dev/hyperv/utilities/hv_util.c optional hyperv dev/hyperv/vmbus/hyperv.c optional hyperv dev/hyperv/vmbus/hyperv_busdma.c optional hyperv -dev/hyperv/vmbus/vmbus.c optional hyperv +dev/hyperv/vmbus/vmbus.c optional hyperv pci dev/hyperv/vmbus/vmbus_br.c optional hyperv dev/hyperv/vmbus/vmbus_chan.c optional hyperv dev/hyperv/vmbus/vmbus_et.c optional hyperv Copied and modified: stable/10/sys/dev/hyperv/pcib/vmbus_pcib.c (from r308795, head/sys/dev/hyperv/pcib/vmbus_pcib.c) ============================================================================== --- head/sys/dev/hyperv/pcib/vmbus_pcib.c Fri Nov 18 06:44:18 2016 (r308795, copy source) +++ stable/10/sys/dev/hyperv/pcib/vmbus_pcib.c Wed Nov 30 06:26:37 2016 (r309313) @@ -27,6 +27,8 @@ #include __FBSDID("$FreeBSD$"); +#ifdef NEW_PCIB + #include #include #include @@ -1789,3 +1791,5 @@ DEFINE_CLASS_0(pcib, vmbus_pcib_driver, DRIVER_MODULE(vmbus_pcib, vmbus, vmbus_pcib_driver, pcib_devclass, 0, 0); MODULE_DEPEND(vmbus_pcib, vmbus, 1, 1, 1); MODULE_DEPEND(vmbus_pcib, pci, 1, 1, 1); + +#endif /* NEW_PCIB */ Modified: stable/10/sys/dev/hyperv/vmbus/vmbus.c ============================================================================== --- stable/10/sys/dev/hyperv/vmbus/vmbus.c Wed Nov 30 06:20:43 2016 (r309312) +++ stable/10/sys/dev/hyperv/vmbus/vmbus.c Wed Nov 30 06:26:37 2016 (r309313) @@ -45,11 +45,14 @@ __FBSDID("$FreeBSD$"); #include #include +#include #include +#include #include #include #include +#include #include #include @@ -60,6 +63,7 @@ __FBSDID("$FreeBSD$"); #include #include "acpi_if.h" +#include "pcib_if.h" #include "vmbus_if.h" #define VMBUS_GPADL_START 0xe1e10 @@ -76,9 +80,25 @@ static int vmbus_read_ivar(device_t, d uintptr_t *); static int vmbus_child_pnpinfo_str(device_t, device_t, char *, size_t); +static struct resource *vmbus_alloc_resource(device_t dev, + device_t child, int type, int *rid, + rman_res_t start, rman_res_t end, + rman_res_t count, u_int flags); +static int vmbus_alloc_msi(device_t bus, device_t dev, + int count, int maxcount, int *irqs); +static int vmbus_release_msi(device_t bus, device_t dev, + int count, int *irqs); +static int vmbus_alloc_msix(device_t bus, device_t dev, + int *irq); +static int vmbus_release_msix(device_t bus, device_t dev, + int irq); +static int vmbus_map_msi(device_t bus, device_t dev, + int irq, uint64_t *addr, uint32_t *data); static uint32_t vmbus_get_version_method(device_t, device_t); static int vmbus_probe_guid_method(device_t, device_t, const struct hyperv_guid *); +static uint32_t vmbus_get_vcpu_id_method(device_t bus, + device_t dev, int cpu); static int vmbus_init(struct vmbus_softc *); static int vmbus_connect(struct vmbus_softc *, uint32_t); @@ -133,10 +153,27 @@ static device_method_t vmbus_methods[] = DEVMETHOD(bus_print_child, bus_generic_print_child), DEVMETHOD(bus_read_ivar, vmbus_read_ivar), DEVMETHOD(bus_child_pnpinfo_str, vmbus_child_pnpinfo_str), + DEVMETHOD(bus_alloc_resource, vmbus_alloc_resource), + DEVMETHOD(bus_release_resource, bus_generic_release_resource), + DEVMETHOD(bus_activate_resource, bus_generic_activate_resource), + DEVMETHOD(bus_deactivate_resource, bus_generic_deactivate_resource), + DEVMETHOD(bus_setup_intr, bus_generic_setup_intr), + DEVMETHOD(bus_teardown_intr, bus_generic_teardown_intr), +#if __FreeBSD_version >= 1100000 + DEVMETHOD(bus_get_cpus, bus_generic_get_cpus), +#endif + + /* pcib interface */ + DEVMETHOD(pcib_alloc_msi, vmbus_alloc_msi), + DEVMETHOD(pcib_release_msi, vmbus_release_msi), + DEVMETHOD(pcib_alloc_msix, vmbus_alloc_msix), + DEVMETHOD(pcib_release_msix, vmbus_release_msix), + DEVMETHOD(pcib_map_msi, vmbus_map_msi), /* Vmbus interface */ DEVMETHOD(vmbus_get_version, vmbus_get_version_method), DEVMETHOD(vmbus_probe_guid, vmbus_probe_guid_method), + DEVMETHOD(vmbus_get_vcpu_id, vmbus_get_vcpu_id_method), DEVMETHOD_END }; @@ -151,6 +188,7 @@ static devclass_t vmbus_devclass; DRIVER_MODULE(vmbus, acpi, vmbus_driver, vmbus_devclass, NULL, NULL); MODULE_DEPEND(vmbus, acpi, 1, 1, 1); +MODULE_DEPEND(vmbus, pci, 1, 1, 1); MODULE_VERSION(vmbus, 1); static __inline struct vmbus_softc * @@ -1054,6 +1092,74 @@ vmbus_sysctl_version(SYSCTL_HANDLER_ARGS return sysctl_handle_string(oidp, verstr, sizeof(verstr), req); } +/* + * We need the function to make sure the MMIO resource is allocated from the + * ranges found in _CRS. + * + * For the release function, we can use bus_generic_release_resource(). + */ +static struct resource * +vmbus_alloc_resource(device_t dev, device_t child, int type, int *rid, + rman_res_t start, rman_res_t end, rman_res_t count, u_int flags) +{ + device_t parent = device_get_parent(dev); + struct resource *res; + +#ifdef NEW_PCIB + if (type == SYS_RES_MEMORY) { + struct vmbus_softc *sc = device_get_softc(dev); + + res = pcib_host_res_alloc(&sc->vmbus_mmio_res, child, type, + rid, start, end, count, flags); + } else +#endif + { + res = BUS_ALLOC_RESOURCE(parent, child, type, rid, start, + end, count, flags); + } + + return (res); +} + +static device_t +get_nexus(device_t vmbus) +{ + device_t acpi = device_get_parent(vmbus); + device_t nexus = device_get_parent(acpi); + return (nexus); +} + +static int +vmbus_alloc_msi(device_t bus, device_t dev, int count, int maxcount, int *irqs) +{ + return (PCIB_ALLOC_MSI(get_nexus(bus), dev, count, maxcount, irqs)); +} + +static int +vmbus_release_msi(device_t bus, device_t dev, int count, int *irqs) +{ + return (PCIB_RELEASE_MSI(get_nexus(bus), dev, count, irqs)); +} + +static int +vmbus_alloc_msix(device_t bus, device_t dev, int *irq) +{ + return (PCIB_ALLOC_MSIX(get_nexus(bus), dev, irq)); +} + +static int +vmbus_release_msix(device_t bus, device_t dev, int irq) +{ + return (PCIB_RELEASE_MSIX(get_nexus(bus), dev, irq)); +} + +static int +vmbus_map_msi(device_t bus, device_t dev, int irq, uint64_t *addr, + uint32_t *data) +{ + return (PCIB_MAP_MSI(get_nexus(bus), dev, irq, addr, data)); +} + static uint32_t vmbus_get_version_method(device_t bus, device_t dev) { @@ -1073,6 +1179,148 @@ vmbus_probe_guid_method(device_t bus, de return ENXIO; } +static uint32_t +vmbus_get_vcpu_id_method(device_t bus, device_t dev, int cpu) +{ + const struct vmbus_softc *sc = device_get_softc(bus); + + return (VMBUS_PCPU_GET(sc, vcpuid, cpu)); +} + +#ifdef NEW_PCIB +#define VTPM_BASE_ADDR 0xfed40000 +#define FOUR_GB (1ULL << 32) + +enum parse_pass { parse_64, parse_32 }; + +struct parse_context { + device_t vmbus_dev; + enum parse_pass pass; +}; + +static ACPI_STATUS +parse_crs(ACPI_RESOURCE *res, void *ctx) +{ + const struct parse_context *pc = ctx; + device_t vmbus_dev = pc->vmbus_dev; + + struct vmbus_softc *sc = device_get_softc(vmbus_dev); + UINT64 start, end; + + switch (res->Type) { + case ACPI_RESOURCE_TYPE_ADDRESS32: + start = res->Data.Address32.Address.Minimum; + end = res->Data.Address32.Address.Maximum; + break; + + case ACPI_RESOURCE_TYPE_ADDRESS64: + start = res->Data.Address64.Address.Minimum; + end = res->Data.Address64.Address.Maximum; + break; + + default: + /* Unused types. */ + return (AE_OK); + } + + /* + * We don't use <1MB addresses. + */ + if (end < 0x100000) + return (AE_OK); + + /* Don't conflict with vTPM. */ + if (end >= VTPM_BASE_ADDR && start < VTPM_BASE_ADDR) + end = VTPM_BASE_ADDR - 1; + + if ((pc->pass == parse_32 && start < FOUR_GB) || + (pc->pass == parse_64 && start >= FOUR_GB)) + pcib_host_res_decodes(&sc->vmbus_mmio_res, SYS_RES_MEMORY, + start, end, 0); + + return (AE_OK); +} + +static void +vmbus_get_crs(device_t dev, device_t vmbus_dev, enum parse_pass pass) +{ + struct parse_context pc; + ACPI_STATUS status; + + if (bootverbose) + device_printf(dev, "walking _CRS, pass=%d\n", pass); + + pc.vmbus_dev = vmbus_dev; + pc.pass = pass; + status = AcpiWalkResources(acpi_get_handle(dev), "_CRS", + parse_crs, &pc); + + if (bootverbose && ACPI_FAILURE(status)) + device_printf(dev, "_CRS: not found, pass=%d\n", pass); +} + +static void +vmbus_get_mmio_res_pass(device_t dev, enum parse_pass pass) +{ + device_t acpi0, pcib0 = NULL; + device_t *children; + int i, count; + + /* Try to find _CRS on VMBus device */ + vmbus_get_crs(dev, dev, pass); + + /* Try to find _CRS on VMBus device's parent */ + acpi0 = device_get_parent(dev); + vmbus_get_crs(acpi0, dev, pass); + + /* Try to locate pcib0 and find _CRS on it */ + if (device_get_children(acpi0, &children, &count) != 0) + return; + + for (i = 0; i < count; i++) { + if (!device_is_attached(children[i])) + continue; + + if (strcmp("pcib0", device_get_nameunit(children[i]))) + continue; + + pcib0 = children[i]; + break; + } + + if (pcib0) + vmbus_get_crs(pcib0, dev, pass); + + free(children, M_TEMP); +} + +static void +vmbus_get_mmio_res(device_t dev) +{ + struct vmbus_softc *sc = device_get_softc(dev); + /* + * We walk the resources twice to make sure that: in the resource + * list, the 32-bit resources appear behind the 64-bit resources. + * NB: resource_list_add() uses INSERT_TAIL. This way, when we + * iterate through the list to find a range for a 64-bit BAR in + * vmbus_alloc_resource(), we can make sure we try to use >4GB + * ranges first. + */ + pcib_host_res_init(dev, &sc->vmbus_mmio_res); + + vmbus_get_mmio_res_pass(dev, parse_64); + vmbus_get_mmio_res_pass(dev, parse_32); +} + +static void +vmbus_free_mmio_res(device_t dev) +{ + struct vmbus_softc *sc = device_get_softc(dev); + + pcib_host_res_free(dev, &sc->vmbus_mmio_res); +} +#endif /* NEW_PCIB */ + static int vmbus_probe(device_t dev) { @@ -1109,6 +1357,11 @@ vmbus_doattach(struct vmbus_softc *sc) if (sc->vmbus_flags & VMBUS_FLAG_ATTACHED) return (0); + +#ifdef NEW_PCIB + vmbus_get_mmio_res(sc->vmbus_dev); +#endif + sc->vmbus_flags |= VMBUS_FLAG_ATTACHED; sc->vmbus_gpadl = VMBUS_GPADL_START; @@ -1252,6 +1505,10 @@ vmbus_detach(device_t dev) mtx_destroy(&sc->vmbus_prichan_lock); mtx_destroy(&sc->vmbus_chan_lock); +#ifdef NEW_PCIB + vmbus_free_mmio_res(dev); +#endif + return (0); } Modified: stable/10/sys/dev/hyperv/vmbus/vmbus_if.m ============================================================================== --- stable/10/sys/dev/hyperv/vmbus/vmbus_if.m Wed Nov 30 06:20:43 2016 (r309312) +++ stable/10/sys/dev/hyperv/vmbus/vmbus_if.m Wed Nov 30 06:26:37 2016 (r309313) @@ -45,3 +45,9 @@ METHOD int probe_guid { device_t dev; const struct hyperv_guid *guid; }; + +METHOD uint32_t get_vcpu_id { + device_t bus; + device_t dev; + int cpu; +}; Modified: stable/10/sys/dev/hyperv/vmbus/vmbus_var.h ============================================================================== --- stable/10/sys/dev/hyperv/vmbus/vmbus_var.h Wed Nov 30 06:20:43 2016 (r309312) +++ stable/10/sys/dev/hyperv/vmbus/vmbus_var.h Wed Nov 30 06:26:37 2016 (r309313) @@ -31,8 +31,11 @@ #include #include +#include #include +#include +#include /* * NOTE: DO NOT CHANGE THIS. @@ -77,6 +80,10 @@ struct vmbus_pcpu_data { struct task message_task; /* message task */ } __aligned(CACHE_LINE_SIZE); +#if __FreeBSD_version < 1100000 +typedef u_long rman_res_t; +#endif + struct vmbus_softc { void (*vmbus_event_proc)(struct vmbus_softc *, int); u_long *vmbus_tx_evtflags; @@ -120,6 +127,11 @@ struct vmbus_softc { /* Complete channel list */ struct mtx vmbus_chan_lock; TAILQ_HEAD(, vmbus_channel) vmbus_chans; + +#ifdef NEW_PCIB + /* The list of usable MMIO ranges for PCIe pass-through */ + struct pcib_host_resources vmbus_mmio_res; +#endif }; #define VMBUS_FLAG_ATTACHED 0x0001 /* vmbus was attached */ Modified: stable/10/sys/modules/hyperv/pcib/Makefile ============================================================================== --- head/sys/modules/hyperv/pcib/Makefile Wed Nov 16 09:25:00 2016 (r308725) +++ stable/10/sys/modules/hyperv/pcib/Makefile Wed Nov 30 06:26:37 2016 (r309313) @@ -4,8 +4,8 @@ ${.CURDIR}/../../../dev/hyperv/vmbus KMOD= vmbus_pcib -SRCS= pcib.c -SRCS+= bus_if.h device_if.h vmbus_if.h +SRCS= vmbus_pcib.c +SRCS+= bus_if.h device_if.h pci_if.h pcib_if.h vmbus_if.h CFLAGS+= -I${.CURDIR}/../../../dev/hyperv/pcib Modified: stable/10/sys/modules/hyperv/vmbus/Makefile ============================================================================== --- stable/10/sys/modules/hyperv/vmbus/Makefile Wed Nov 30 06:20:43 2016 (r309312) +++ stable/10/sys/modules/hyperv/vmbus/Makefile Wed Nov 30 06:26:37 2016 (r309313) @@ -13,7 +13,7 @@ SRCS= hyperv.c \ vmbus_et.c \ vmbus_if.c \ vmbus_xact.c -SRCS+= acpi_if.h bus_if.h device_if.h opt_acpi.h vmbus_if.h +SRCS+= acpi_if.h bus_if.h device_if.h opt_acpi.h pci_if.h pcib_if.h vmbus_if.h # XXX: for assym.s SRCS+= opt_kstack_pages.h opt_nfs.h opt_apic.h opt_hwpmc_hooks.h opt_compat.h