From owner-svn-src-all@freebsd.org Wed Dec 4 15:14:14 2019 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DC3FB1C65C9; Wed, 4 Dec 2019 15:14:14 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Sj7B5WbPz3xHT; Wed, 4 Dec 2019 15:14:14 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A0CFE1F8A; Wed, 4 Dec 2019 15:14:14 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xB4FEEkv090272; Wed, 4 Dec 2019 15:14:14 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xB4FEEmu090271; Wed, 4 Dec 2019 15:14:14 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201912041514.xB4FEEmu090271@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Wed, 4 Dec 2019 15:14:14 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-12@freebsd.org Subject: svn commit: r355391 - stable/12/sys/dev/ntb/ntb_hw X-SVN-Group: stable-12 X-SVN-Commit-Author: mav X-SVN-Commit-Paths: stable/12/sys/dev/ntb/ntb_hw X-SVN-Commit-Revision: 355391 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Dec 2019 15:14:14 -0000 Author: mav Date: Wed Dec 4 15:14:14 2019 New Revision: 355391 URL: https://svnweb.freebsd.org/changeset/base/355391 Log: MFC r355165: Make DMAR allow Intel NTB device to access its own BAR0. I have no good explanation why it happens, but I found that in B2B mode at least Xeon v4 NTB leaks accesses to its configuration memory at BAR0 originated from the link side to its host side. DMAR predictably blocks those, making access to remote scratchpad registers in B2B mode impossible. This change creates identity mapping in DMAR covering the BAR0 addresses, making the NTB work fine with DMAR enabled. It seems like allowing single 4KB range at 32KB offset may be enough, but I don't see a reason to be so specific. Modified: stable/12/sys/dev/ntb/ntb_hw/ntb_hw_intel.c Directory Properties: stable/12/ (props changed) Modified: stable/12/sys/dev/ntb/ntb_hw/ntb_hw_intel.c ============================================================================== --- stable/12/sys/dev/ntb/ntb_hw/ntb_hw_intel.c Wed Dec 4 15:12:09 2019 (r355390) +++ stable/12/sys/dev/ntb/ntb_hw/ntb_hw_intel.c Wed Dec 4 15:14:14 2019 (r355391) @@ -218,6 +218,9 @@ struct ntb_softc { struct ntb_pci_bar_info *peer_lapic_bar; struct callout peer_msix_work; + bus_dma_tag_t bar0_dma_tag; + bus_dmamap_t bar0_dma_map; + struct callout heartbeat_timer; struct callout lr_timer; @@ -790,6 +793,29 @@ intel_ntb_map_pci_bars(struct ntb_softc *ntb) if (rc != 0) goto out; + /* + * At least on Xeon v4 NTB device leaks to host some remote side + * BAR0 writes supposed to update scratchpad registers. I am not + * sure why it happens, but it may be related to the fact that + * on a link side BAR0 is 32KB, while on a host side it is 64KB. + * Without this hack DMAR blocks those accesses as not allowed. + */ + if (bus_dma_tag_create(bus_get_dma_tag(ntb->device), 1, 0, + BUS_SPACE_MAXADDR, BUS_SPACE_MAXADDR, NULL, NULL, + bar->size, 1, bar->size, 0, NULL, NULL, &ntb->bar0_dma_tag)) { + device_printf(ntb->device, "Unable to create BAR0 tag\n"); + return (ENOMEM); + } + if (bus_dmamap_create(ntb->bar0_dma_tag, 0, &ntb->bar0_dma_map)) { + device_printf(ntb->device, "Unable to create BAR0 map\n"); + return (ENOMEM); + } + if (bus_dma_dmar_load_ident(ntb->bar0_dma_tag, ntb->bar0_dma_map, + bar->pbase, bar->size, 0)) { + device_printf(ntb->device, "Unable to load BAR0 map\n"); + return (ENOMEM); + } + bar = &ntb->bar_info[NTB_B2B_BAR_1]; bar->pci_resource_id = PCIR_BAR(2); rc = map_memory_window_bar(ntb, bar); @@ -942,6 +968,12 @@ intel_ntb_unmap_pci_bar(struct ntb_softc *ntb) struct ntb_pci_bar_info *bar; int i; + if (ntb->bar0_dma_map != NULL) { + bus_dmamap_unload(ntb->bar0_dma_tag, ntb->bar0_dma_map); + bus_dmamap_destroy(ntb->bar0_dma_tag, ntb->bar0_dma_map); + } + if (ntb->bar0_dma_tag != NULL) + bus_dma_tag_destroy(ntb->bar0_dma_tag); for (i = 0; i < NTB_MAX_BARS; i++) { bar = &ntb->bar_info[i]; if (bar->pci_resource != NULL)