From owner-svn-src-all@freebsd.org Fri Aug 26 20:15:24 2016 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 46381B7401B; Fri, 26 Aug 2016 20:15:24 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0A6D82CF1; Fri, 26 Aug 2016 20:15:23 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id u7QKFNQn077412; Fri, 26 Aug 2016 20:15:23 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id u7QKFNTS077409; Fri, 26 Aug 2016 20:15:23 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201608262015.u7QKFNTS077409@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Fri, 26 Aug 2016 20:15:23 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r304858 - in head/sys/amd64/vmm: . io X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Aug 2016 20:15:24 -0000 Author: jhb Date: Fri Aug 26 20:15:22 2016 New Revision: 304858 URL: https://svnweb.freebsd.org/changeset/base/304858 Log: Enable I/O MMU when PCI pass through is first used. Rather than enabling the I/O MMU when the vmm module is loaded, defer initialization until the first attempt to pass a PCI device through to a guest. If the I/O MMU fails to initialize or is not present, than fail the attempt to pass a PCI device through to a guest. The hw.vmm.force_iommu tunable has been removed since the I/O MMU is no longer enabled during boot. However, the I/O MMU support can be disabled by setting the hw.vmm.iommu.enable tunable to 0 to prevent use of the I/O MMU on any systems where it is buggy. Reviewed by: grehan MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D7448 Modified: head/sys/amd64/vmm/io/iommu.c head/sys/amd64/vmm/io/iommu.h head/sys/amd64/vmm/vmm.c Modified: head/sys/amd64/vmm/io/iommu.c ============================================================================== --- head/sys/amd64/vmm/io/iommu.c Fri Aug 26 19:23:17 2016 (r304857) +++ head/sys/amd64/vmm/io/iommu.c Fri Aug 26 20:15:22 2016 (r304858) @@ -38,6 +38,7 @@ __FBSDID("$FreeBSD$"); #include #include +#include #include #include "vmm_util.h" @@ -51,6 +52,10 @@ static int iommu_avail; SYSCTL_INT(_hw_vmm_iommu, OID_AUTO, initialized, CTLFLAG_RD, &iommu_avail, 0, "bhyve iommu initialized?"); +static int iommu_enable = 1; +SYSCTL_INT(_hw_vmm_iommu, OID_AUTO, enable, CTLFLAG_RDTUN, &iommu_enable, 0, + "Enable use of I/O MMU (required for PCI passthrough)."); + static struct iommu_ops *ops; static void *host_domain; @@ -148,7 +153,7 @@ IOMMU_DISABLE(void) (*ops->disable)(); } -void +static void iommu_init(void) { int error, bus, slot, func; @@ -156,6 +161,9 @@ iommu_init(void) const char *name; device_t dev; + if (!iommu_enable) + return; + if (vmm_is_intel()) ops = &iommu_ops_intel; else if (vmm_is_amd()) @@ -174,8 +182,13 @@ iommu_init(void) */ maxaddr = vmm_mem_maxaddr(); host_domain = IOMMU_CREATE_DOMAIN(maxaddr); - if (host_domain == NULL) - panic("iommu_init: unable to create a host domain"); + if (host_domain == NULL) { + printf("iommu_init: unable to create a host domain"); + IOMMU_CLEANUP(); + ops = NULL; + iommu_avail = 0; + return; + } /* * Create 1:1 mappings from '0' to 'maxaddr' for devices assigned to @@ -216,7 +229,16 @@ iommu_cleanup(void) void * iommu_create_domain(vm_paddr_t maxaddr) { + static volatile int iommu_initted; + if (iommu_initted < 2) { + if (atomic_cmpset_int(&iommu_initted, 0, 1)) { + iommu_init(); + atomic_store_rel_int(&iommu_initted, 2); + } else + while (iommu_initted == 1) + cpu_spinwait(); + } return (IOMMU_CREATE_DOMAIN(maxaddr)); } Modified: head/sys/amd64/vmm/io/iommu.h ============================================================================== --- head/sys/amd64/vmm/io/iommu.h Fri Aug 26 19:23:17 2016 (r304857) +++ head/sys/amd64/vmm/io/iommu.h Fri Aug 26 20:15:22 2016 (r304858) @@ -61,7 +61,6 @@ struct iommu_ops { extern struct iommu_ops iommu_ops_intel; extern struct iommu_ops iommu_ops_amd; -void iommu_init(void); void iommu_cleanup(void); void *iommu_host_domain(void); void *iommu_create_domain(vm_paddr_t maxaddr); Modified: head/sys/amd64/vmm/vmm.c ============================================================================== --- head/sys/amd64/vmm/vmm.c Fri Aug 26 19:23:17 2016 (r304857) +++ head/sys/amd64/vmm/vmm.c Fri Aug 26 20:15:22 2016 (r304858) @@ -224,11 +224,6 @@ SYSCTL_INT(_hw_vmm, OID_AUTO, trace_gues &trace_guest_exceptions, 0, "Trap into hypervisor on all guest exceptions and reflect them back"); -static int vmm_force_iommu = 0; -TUNABLE_INT("hw.vmm.force_iommu", &vmm_force_iommu); -SYSCTL_INT(_hw_vmm, OID_AUTO, force_iommu, CTLFLAG_RDTUN, &vmm_force_iommu, 0, - "Force use of I/O MMU even if no passthrough devices were found."); - static void vm_free_memmap(struct vm *vm, int ident); static bool sysmem_mapping(struct vm *vm, struct mem_map *mm); static void vcpu_notify_event_locked(struct vcpu *vcpu, bool lapic_intr); @@ -358,8 +353,6 @@ vmm_handler(module_t mod, int what, void switch (what) { case MOD_LOAD: vmmdev_init(); - if (vmm_force_iommu || ppt_avail_devices() > 0) - iommu_init(); error = vmm_init(); if (error == 0) vmm_initialized = 1; @@ -396,9 +389,6 @@ static moduledata_t vmm_kmod = { /* * vmm initialization has the following dependencies: * - * - iommu initialization must happen after the pci passthru driver has had - * a chance to attach to any passthru devices (after SI_SUB_CONFIGURE). - * * - VT-x initialization requires smp_rendezvous() and therefore must happen * after SMP is fully functional (after SI_SUB_SMP). */ @@ -893,6 +883,8 @@ vm_assign_pptdev(struct vm *vm, int bus, ("vm_assign_pptdev: iommu must be NULL")); maxaddr = sysmem_maxaddr(vm); vm->iommu = iommu_create_domain(maxaddr); + if (vm->iommu == NULL) + return (ENXIO); vm_iommu_map(vm); }