Date: Thu, 02 Apr 2015 00:24:45 +0200 From: Tobias Oberstein <tobias.oberstein@gmail.com> To: Konstantin Belousov <kostikbel@gmail.com> Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, Michael Fuckner <michael@fuckner.net>, Jim Harris <jim.harris@gmail.com>, Alan Somers <asomers@freebsd.org> Subject: Re: NVMe performance 4x slower than expected Message-ID: <551C702D.2070009@gmail.com> In-Reply-To: <20150401212303.GB2379@kib.kiev.ua> References: <551BC57D.5070101@gmail.com> <CAOtMX2jVwMHSnQfphAF%2Ba2%2Bo7eLp62nHmUo4t%2BEahrXLWReaFQ@mail.gmail.com> <CAJP=Hc-RNVuhPePg7bnpmT4ByzyXs_CNvAs7Oy7ntXjqhZYhCQ@mail.gmail.com> <551C5A82.2090306@gmail.com> <20150401212303.GB2379@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
Am 01.04.2015 um 23:23 schrieb Konstantin Belousov: > On Wed, Apr 01, 2015 at 10:52:18PM +0200, Tobias Oberstein wrote: >>> > FreeBSD 11 Current with patches (DMAR and ZFS patches, otherwise the box >>> > doesn't boot at all .. because of 3TB RAM and the amount of periphery). >>> >>> Do you still have WITNESS and INVARIANTS turned on in your kernel >>> config? They're turned on by default for Current, but they do have >>> some performance impact. To turn them off, just build a >>> GENERIC-NODEBUG kernel . >> >> WITNESS is off, INVARIANTS is still on. > INVARIANTS are costly. ah, ok. will rebuild without this option. > I have the following patch for a long time, it allowed to increase pps > in iperf and similar tests when DMAR is enabled. In your case it could > reduce the rate of the DMAR interrupts. You mean these lines from vmstat? irq257: dmar0:qi 22312 0 irq259: dmar1:qi 22652 0 irq261: dmar2:qi 261874194 6911 irq263: dmar3:qi 124939 3 So these dmar2 interrupts come from DMAR region 2 which is used by nvd7? From dmesg: dmar0: <DMA remap> iomem 0xc7ffc000-0xc7ffcfff on acpi0 dmar1: <DMA remap> iomem 0xe3ffc000-0xe3ffcfff on acpi0 dmar2: <DMA remap> iomem 0xfbffc000-0xfbffcfff on acpi0 dmar3: <DMA remap> iomem 0xabffc000-0xabffcfff on acpi0 mpr0: dmar3 pci0:4:0:0 rid 400 domain 4 mgaw 48 agaw 48 re-mapped mpr1: dmar2 pci0:195:0:0 rid c300 domain 2 mgaw 48 agaw 48 re-mapped nvme0: dmar0 pci0:65:0:0 rid 4100 domain 0 mgaw 48 agaw 48 re-mapped nvme1: dmar0 pci0:67:0:0 rid 4300 domain 1 mgaw 48 agaw 48 re-mapped nvme2: dmar0 pci0:69:0:0 rid 4500 domain 2 mgaw 48 agaw 48 re-mapped nvme3: dmar1 pci0:129:0:0 rid 8100 domain 0 mgaw 48 agaw 48 re-mapped nvme4: dmar1 pci0:131:0:0 rid 8300 domain 1 mgaw 48 agaw 48 re-mapped nvme5: dmar1 pci0:132:0:0 rid 8400 domain 2 mgaw 48 agaw 48 re-mapped nvme6: dmar2 pci0:193:0:0 rid c100 domain 0 mgaw 48 agaw 48 re-mapped nvme7: dmar2 pci0:194:0:0 rid c200 domain 1 mgaw 48 agaw 48 re-mapped unknown: dmar3 pci0:0:29:0 rid e8 domain 0 mgaw 48 agaw 48 re-mapped unknown: dmar3 pci0:0:26:0 rid d0 domain 1 mgaw 48 agaw 48 re-mapped ix0: dmar3 pci0:1:0:0 rid 100 domain 2 mgaw 48 agaw 48 re-mapped ix1: dmar3 pci0:1:0:1 rid 101 domain 3 mgaw 48 agaw 48 re-mapped ix0: Using MSIX interrupts with 49 vectors ix1: Using MSIX interrupts with 49 vectors -- So the LSI HBAs, Intel NICs and NVMe are all using DMAR, but only the NICs use MSI-X? But 2 * 49 = 98, and that is smaller than the 191 which Jim mentions. And what are those "unknown" devices on dmar3? Actually, I don't really know what I am talking about here .. just puzzled. /Tobias
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?551C702D.2070009>