From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 1 22:24:50 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 918C7908; Wed, 1 Apr 2015 22:24:50 +0000 (UTC) Received: from mail-wg0-x236.google.com (mail-wg0-x236.google.com [IPv6:2a00:1450:400c:c00::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1E32A36C; Wed, 1 Apr 2015 22:24:50 +0000 (UTC) Received: by wgoe14 with SMTP id e14so67750731wgo.0; Wed, 01 Apr 2015 15:24:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=sbwvBEDLPZc4xuoFlDvRpXBuO2lCPfN09/YZMISFyso=; b=CvkbQ3pOrYo93mSaVb1dJS1I8jMExWq+6/U3+rblJeunowsWVIAcBje8UoI4/DRXkb FxBvAtQWFrW4XeYaCapPolXVjtkH9MZKf84idL5CbZpGzayah3KFFOSoF8HxdpNUJn0A b9BOdFMiJvUfNNsEiovMI0+sIj8NWpqN9OfGcDahE14OvmDHY3MABUqwwE0VpIOBFGhm oqrqc8rRED9OTAD60UiRhO0LhaZGuRsN8KK3eXYbci7WWGq4p2WYjy5D0b8op8zd71Oe G5Rk1xHGu8Th5SKONegGzvFLeBG1JzlTmeH0uZy08yr69cdSxMY6yt5/DaDR2cXNsAe0 ZrJA== X-Received: by 10.194.177.132 with SMTP id cq4mr84237742wjc.99.1427927088497; Wed, 01 Apr 2015 15:24:48 -0700 (PDT) Received: from [172.22.3.210] (p578b7aa0.dip0.t-ipconnect.de. [87.139.122.160]) by mx.google.com with ESMTPSA id 17sm4521152wjt.45.2015.04.01.15.24.47 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 01 Apr 2015 15:24:47 -0700 (PDT) Message-ID: <551C702D.2070009@gmail.com> Date: Thu, 02 Apr 2015 00:24:45 +0200 From: Tobias Oberstein User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: NVMe performance 4x slower than expected References: <551BC57D.5070101@gmail.com> <551C5A82.2090306@gmail.com> <20150401212303.GB2379@kib.kiev.ua> In-Reply-To: <20150401212303.GB2379@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" , Michael Fuckner , Jim Harris , Alan Somers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Apr 2015 22:24:50 -0000 Am 01.04.2015 um 23:23 schrieb Konstantin Belousov: > On Wed, Apr 01, 2015 at 10:52:18PM +0200, Tobias Oberstein wrote: >>> > FreeBSD 11 Current with patches (DMAR and ZFS patches, otherwise the box >>> > doesn't boot at all .. because of 3TB RAM and the amount of periphery). >>> >>> Do you still have WITNESS and INVARIANTS turned on in your kernel >>> config? They're turned on by default for Current, but they do have >>> some performance impact. To turn them off, just build a >>> GENERIC-NODEBUG kernel . >> >> WITNESS is off, INVARIANTS is still on. > INVARIANTS are costly. ah, ok. will rebuild without this option. > I have the following patch for a long time, it allowed to increase pps > in iperf and similar tests when DMAR is enabled. In your case it could > reduce the rate of the DMAR interrupts. You mean these lines from vmstat? irq257: dmar0:qi 22312 0 irq259: dmar1:qi 22652 0 irq261: dmar2:qi 261874194 6911 irq263: dmar3:qi 124939 3 So these dmar2 interrupts come from DMAR region 2 which is used by nvd7? From dmesg: dmar0: iomem 0xc7ffc000-0xc7ffcfff on acpi0 dmar1: iomem 0xe3ffc000-0xe3ffcfff on acpi0 dmar2: iomem 0xfbffc000-0xfbffcfff on acpi0 dmar3: iomem 0xabffc000-0xabffcfff on acpi0 mpr0: dmar3 pci0:4:0:0 rid 400 domain 4 mgaw 48 agaw 48 re-mapped mpr1: dmar2 pci0:195:0:0 rid c300 domain 2 mgaw 48 agaw 48 re-mapped nvme0: dmar0 pci0:65:0:0 rid 4100 domain 0 mgaw 48 agaw 48 re-mapped nvme1: dmar0 pci0:67:0:0 rid 4300 domain 1 mgaw 48 agaw 48 re-mapped nvme2: dmar0 pci0:69:0:0 rid 4500 domain 2 mgaw 48 agaw 48 re-mapped nvme3: dmar1 pci0:129:0:0 rid 8100 domain 0 mgaw 48 agaw 48 re-mapped nvme4: dmar1 pci0:131:0:0 rid 8300 domain 1 mgaw 48 agaw 48 re-mapped nvme5: dmar1 pci0:132:0:0 rid 8400 domain 2 mgaw 48 agaw 48 re-mapped nvme6: dmar2 pci0:193:0:0 rid c100 domain 0 mgaw 48 agaw 48 re-mapped nvme7: dmar2 pci0:194:0:0 rid c200 domain 1 mgaw 48 agaw 48 re-mapped unknown: dmar3 pci0:0:29:0 rid e8 domain 0 mgaw 48 agaw 48 re-mapped unknown: dmar3 pci0:0:26:0 rid d0 domain 1 mgaw 48 agaw 48 re-mapped ix0: dmar3 pci0:1:0:0 rid 100 domain 2 mgaw 48 agaw 48 re-mapped ix1: dmar3 pci0:1:0:1 rid 101 domain 3 mgaw 48 agaw 48 re-mapped ix0: Using MSIX interrupts with 49 vectors ix1: Using MSIX interrupts with 49 vectors -- So the LSI HBAs, Intel NICs and NVMe are all using DMAR, but only the NICs use MSI-X? But 2 * 49 = 98, and that is smaller than the 191 which Jim mentions. And what are those "unknown" devices on dmar3? Actually, I don't really know what I am talking about here .. just puzzled. /Tobias