From owner-freebsd-stable@FreeBSD.ORG Sun Jan 11 08:07:59 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 942AEA43 for ; Sun, 11 Jan 2015 08:07:59 +0000 (UTC) Received: from mail-pd0-x22d.google.com (mail-pd0-x22d.google.com [IPv6:2607:f8b0:400e:c02::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6365B28B for ; Sun, 11 Jan 2015 08:07:59 +0000 (UTC) Received: by mail-pd0-f173.google.com with SMTP id ft15so25240070pdb.4 for ; Sun, 11 Jan 2015 00:07:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:mime-version:to:from:subject:date :content-transfer-encoding:content-type; bh=FQqqsthfVygSAblLv2VSVzMOMbd32W8qaWqysn+tZiU=; b=i9EO8nIf0y94Br0FBckxyhVosXEraZ1cQXmZ/OoK+DSyj5KH6Odf58slG2/HjdmJHM c7YugT6ejyPyYKIs+EQiUWVEqGhB2Eualc1x8Xf6q20KE4Ck8zE+b/zZha1FzKpXRr7f 6HCLVvJ+IkKeB4rP6WBCc2gGjY8FUqdL2NJqqqbfUPczXapTBIX/Xm+zA6kKW9ktqVSE CkInk7TCMqT3EapDZssofMit9Oh/zAX47j4eUtcVvSRG5x93X3+ockSBsRrkDaRzwPW5 KolYwMGcbKjcKT4o78lr99h6CYFHKCqhXmRC1qST+5X8CwJaxqijGQI0Mp7uOtWxRRv7 HGZQ== X-Received: by 10.68.224.234 with SMTP id rf10mr1175285pbc.124.1420963678944; Sun, 11 Jan 2015 00:07:58 -0800 (PST) Received: from [192.168.0.2] (cpe-104-33-36-107.socal.res.rr.com. [104.33.36.107]) by mx.google.com with ESMTPSA id rg6sm11237314pbc.43.2015.01.11.00.07.58 for (version=SSLv3 cipher=RC4-SHA bits=128/128); Sun, 11 Jan 2015 00:07:58 -0800 (PST) Message-ID: <54b22f5e.06e1440a.6cb0.fffff431@mx.google.com> To: From: Blanca Suarez Subject: Adult FriendFinder email address change Date: Sun, 11 Jan 2015 00:07:53 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="Windows-1252" X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jan 2015 08:07:59 -0000 From owner-freebsd-stable@FreeBSD.ORG Sun Jan 11 15:36:33 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C61118CE for ; Sun, 11 Jan 2015 15:36:33 +0000 (UTC) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 37BB6E92 for ; Sun, 11 Jan 2015 15:36:32 +0000 (UTC) Received: from mh0.gentlemail.de (mh0.gentlemail.de [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id t0BFaSph079526; Sun, 11 Jan 2015 16:36:28 +0100 (CET) (envelope-from h.schmalzbauer@omnilan.de) Received: from titan.inop.mo1.omnilan.net (titan.inop.mo1.omnilan.net [IPv6:2001:a60:f0bb:1::3:1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 49A8D64A; Sun, 11 Jan 2015 16:36:27 +0100 (CET) Message-ID: <54B29870.5020706@omnilan.de> Date: Sun, 11 Jan 2015 16:36:16 +0100 From: Harald Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Jack Vogel Subject: Re: igb(4) watchdog timeout, lagg(4) fails References: <54ACC6A2.1050400@omnilan.de> <54AE565D.50208@omnilan.de> <54AE5A6B.7040601@omnilan.de> <54AFA784.6020102@omnilan.de> <54B10432.8050909@omnilan.de> In-Reply-To: X-Enigmail-Version: 1.1.2 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigC234BA70BC6B9ED425A25F1C" X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Sun, 11 Jan 2015 16:36:28 +0100 (CET) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) Cc: FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jan 2015 15:36:33 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigC234BA70BC6B9ED425A25F1C Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Bez=FCglich Jack Vogel's Nachricht vom 10.01.2015 19:32 (localtime): > Did you say this system is a VM under ESX? Yes, but igb(4) isn't used as VF (SR-IOV), but via VT-d. Like mentioned, the setup never showed any unexpected errors and has been reliably withstanding single-component-failures as designed for more than one year. But now the watchdog timeout leaves one igb(4) nic in unoperational state, but lagg(4) doesn't notice that there's a problem with that nic :-= ( I think a watchdog reset should bring if_igb down and only up again if it's confirmed that the reset succeeded and the nic is operational again.= Or lagg(0) would need to do some line testing. It's done with LACP, but in that environment I don't have LACP stackable available. So I'm left with l2hash=85 Regarding the timeout, I think I found the solution =96 but it's not evident yet. Besides the FreeBSD update from 9.1 to 10.1 (-stable) I noticed that the VT-d routing changed. So I spread the PCI-slot numbers differently to make igb0 share it's virtual APIC line with some other cards. No more watchdog timeout until now=85 Thanks, -Harry --------------enigC234BA70BC6B9ED425A25F1C Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAlSymHsACgkQLDqVQ9VXb8iTOACfWeMX8xtnytGICVtiZ1Xvaqei s7AAn3sGO7FrxcfoWQPvFcgjvOmoMwga =yOl3 -----END PGP SIGNATURE----- --------------enigC234BA70BC6B9ED425A25F1C-- From owner-freebsd-stable@FreeBSD.ORG Mon Jan 12 06:11:05 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4BF92900 for ; Mon, 12 Jan 2015 06:11:05 +0000 (UTC) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B2919A7F for ; Mon, 12 Jan 2015 06:11:04 +0000 (UTC) Received: from mh0.gentlemail.de (mh0.gentlemail.de [78.138.80.135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id t0C6AxWZ088414; Mon, 12 Jan 2015 07:10:59 +0100 (CET) (envelope-from h.schmalzbauer@omnilan.de) Received: from titan.inop.mo1.omnilan.net (titan.inop.mo1.omnilan.net [IPv6:2001:a60:f0bb:1::3:1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id CDCB2777; Mon, 12 Jan 2015 07:10:58 +0100 (CET) Message-ID: <54B36572.3040108@omnilan.de> Date: Mon, 12 Jan 2015 07:10:58 +0100 From: Harald Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Jack Vogel Subject: Re: igb(4) watchdog timeout, lagg(4) fails References: <54ACC6A2.1050400@omnilan.de> <54AE565D.50208@omnilan.de> <54AE5A6B.7040601@omnilan.de> <54AFA784.6020102@omnilan.de> <54B10432.8050909@omnilan.de> <54B29870.5020706@omnilan.de> In-Reply-To: <54B29870.5020706@omnilan.de> X-Enigmail-Version: 1.1.2 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig137F57397351B1270FF0E03C" X-Greylist: ACL 119 matched, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [78.138.80.130]); Mon, 12 Jan 2015 07:10:59 +0100 (CET) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: 78.138.80.135; Sender-helo: mh0.gentlemail.de; ) Cc: FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jan 2015 06:11:05 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig137F57397351B1270FF0E03C Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Bez=FCglich Harald Schmalzbauer's Nachricht vom 11.01.2015 16:36 (localt= ime): > cards. No more watchdog timeout until now=85 until 5am this morning: Jan 12 05:10:04 vega kernel: igb0: Watchdog timeout -- resetting Jan 12 05:10:04 vega kernel: igb0: Queue(127840256) tdh =3D 33019, hw tdt= =3D 139 Jan 12 05:10:04 vega kernel: igb0: TX(127840256) desc avail =3D 0,Next TX= to Clean =3D 0 Jan 12 05:10:04 vega kernel: igb0: link state changed to DOWN Jan 12 05:10:07 vega kernel: igb0: link state changed to UP Jan 12 05:10:07 vega devd: Executing '/etc/rc.d/dhclient quietstart igb0'= Jan 12 05:19:43 vega kernel: igb0: Watchdog timeout -- resetting Jan 12 05:19:43 vega kernel: igb0: Queue(127840256) tdh =3D 33019, hw tdt= =3D 139 Jan 12 05:19:43 vega kernel: igb0: TX(127840256) desc avail =3D 0,Next TX= to Clean =3D 0 Jan 12 05:19:43 vega kernel: igb0: link state changed to DOWN Jan 12 05:19:46 vega kernel: igb0: link state changed to UP =85 But this time the interface went down. Unfortunately also up again, so lagg(4) failed again, dropping most iSCSI sessions :-( I'll have to wait until I can reboot the machine and try disabling aim. Were there changes between 9.1 and 10.1 regarding aim? Thanks, -Harry --------------enig137F57397351B1270FF0E03C Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAlSzZXMACgkQLDqVQ9VXb8hXQACgtU9ZCvvxPFEVjv9/K9C+J4sH zN8AoKehxT0pDqvfRnAS0RB1XMnYE4+A =G308 -----END PGP SIGNATURE----- --------------enig137F57397351B1270FF0E03C-- From owner-freebsd-stable@FreeBSD.ORG Mon Jan 12 16:36:54 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D9A5EB14 for ; Mon, 12 Jan 2015 16:36:54 +0000 (UTC) Received: from mail-wg0-f47.google.com (mail-wg0-f47.google.com [74.125.82.47]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 677833E7 for ; Mon, 12 Jan 2015 16:36:53 +0000 (UTC) Received: by mail-wg0-f47.google.com with SMTP id n12so20348331wgh.6 for ; Mon, 12 Jan 2015 08:36:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=Mkiul+e8JIWgQiADL2I1YVo5In0RW7bT5NahLbPw6cg=; b=b+l4KAmHwH+UQGmFqjcKEH8yqcHq6AdZR7bm5BhEcSTr6QqQWRvreZjUXZxWo6ugj5 lgkaXia7wcDnS87COu2rH9UtlMsjy8ey2G+b+ML5p9n8MKaGexpuuEWmINcHFg7t06bh +ngXKFTUGyqsqqErdn92Q8DPt6MCj3QoTeCK8qTYR9LkAOGOEF3pkPeW8jM5m/9DOHJG YlHDOMeRgtxbgFZAX7WqugHByTKcTcrXCTD1NI0HkKannMozIo7nJJDM/KXZJLUoSwPj 5+zh7vWh+ys8a3iYsnIsRpfLjSHTbN4E70/JnScbPuPEyQxjThSfOAfDDkw8abvwg3aI 7MCg== X-Gm-Message-State: ALoCoQkeQ4aki3dOH05TT9xSKPZ7noDev6niZGMg4R/vmr0pQLqnh/swHx0HR0zf0xhm329+K5jH X-Received: by 10.180.91.37 with SMTP id cb5mr32042748wib.1.1421080605919; Mon, 12 Jan 2015 08:36:45 -0800 (PST) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id gb10sm22250600wjb.21.2015.01.12.08.36.44 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Jan 2015 08:36:45 -0800 (PST) Message-ID: <54B3F81A.8080400@multiplay.co.uk> Date: Mon, 12 Jan 2015 16:36:42 +0000 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: Creating a bootable ZFS disk? References: <54a048f2.45c1c20a.6ffd.ffffe6d7SMTPIN_ADDED_BROKEN@mx.google.com> <54A062FE.6020500@multiplay.co.uk> <54A067D0.4050606@multiplay.co.uk> <349F0A87-5F85-4367-9A5C-E77DBFA16588@karthauser.co.uk> <7E1DA790-822F-4253-A3F6-1E5F5EFFEE04@karthauser.co.uk> <54A1129F.3040004@multiplay.co.uk> <54A18986.4000002@multiplay.co.uk> <5AFFE5CE-ABC7-4D9C-B8E7-0AC9C3327D6B@tao.org.uk> <54A2D3FA.6000404@multiplay.co.uk> <18C4C5D1-04D8-46B3-9E6F-FDAEB49BED1A@tao.org.uk> In-Reply-To: <18C4C5D1-04D8-46B3-9E6F-FDAEB49BED1A@tao.org.uk> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jan 2015 16:36:54 -0000 On 01/01/2015 17:18, Dr Josef Karthauser wrote: > On 30 Dec 2014, at 16:34, Steven Hartland wrote: >> This is strengthened by the fact that ATI's previous generation HW (SB600) had MSI disabled by r245875 due to a very similar issue. >> >> So given all the evidence so far ahci.0.msi=1 may well be the fix. >> > Is there any benefit to also trying with mdi > 1 < 8? > > Any joy with this Josef? From owner-freebsd-stable@FreeBSD.ORG Tue Jan 13 00:20:20 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BDD17F46 for ; Tue, 13 Jan 2015 00:20:20 +0000 (UTC) Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by mx1.freebsd.org (Postfix) with ESMTP id 51A563D5 for ; Tue, 13 Jan 2015 00:20:19 +0000 (UTC) Received: from ppp14-2-33-47.lns21.adl2.internode.on.net (HELO midget.dons.net.au) ([14.2.33.47]) by ipmail06.adl6.internode.on.net with ESMTP; 13 Jan 2015 10:50:11 +1030 Received: from [10.0.2.26] ([10.0.2.26]) (authenticated bits=0) by midget.dons.net.au (8.15.1/8.14.9) with ESMTPSA id t0D0K1FV001473 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 13 Jan 2015 10:50:10 +1030 (CST) (envelope-from darius@dons.net.au) From: "O'Connor, Daniel" Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Tue, 13 Jan 2015 10:50:00 +1030 Subject: DTrace and function names To: FreeBSD Stable Mailing List Message-Id: Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) X-Mailer: Apple Mail (2.1993) X-Spam-Score: -2.9 () ALL_TRUSTED,BAYES_00 X-Scanned-By: MIMEDefang 2.75 on 10.0.2.1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2015 00:20:20 -0000 Hi, I am trying to debug why astro/gpsd uses a lot of CPU so I thought I'd = try the 'hotuser' Dtrace toolkit script but the output looks like so.. [midget 9:24] ~ >sudo /usr/local/share/DTraceToolkit/hotuser -p 37690 Password: Sampling... Hit Ctrl-C to end. ^C FUNCTION COUNT PCNT libc.so.7`0x801631777 1 0.0% libc.so.7`strtol 1 0.0% libc.so.7`0x801631658 1 0.0% libc.so.7`0x801631625 1 0.0% libc.so.7`0x801631636 1 0.0% libc.so.7`0x801631566 1 0.0% libc.so.7`0x801631623 1 0.0% gpsd`0x405ba6 1 0.0% gpsd`0x404fd6 1 0.0% libc.so.7`0x801631797 10 0.0% libc.so.7`0x8016315f9 50 0.2% libc.so.7`0x80163165c 63 0.2% libc.so.7`0x80163179b 115 0.4% libc.so.7`0x8016315b9 119 0.4% libc.so.7`0x801631794 135 0.5% libc.so.7`0x801631791 170 0.6% libc.so.7`0x801631601 212 0.8% libc.so.7`0x80163165d 226 0.8% libc.so.7`0x801631757 233 0.8% libc.so.7`0x8016315bd 235 0.8% gpsd`0x4028bc 237 0.8% libc.so.7`0x801631736 238 0.9% libc.so.7`0x80163156a 239 0.9% libc.so.7`0x801631617 243 0.9% gpsd`0x4084ea 246 0.9% gpsd`0x408508 251 0.9% libc.so.7`0x8016317a0 251 0.9% libc.so.7`0x801631571 252 0.9% libc.so.7`0x801631749 252 0.9% libc.so.7`0x801631583 253 0.9% libc.so.7`0x8016315c7 256 0.9% libc.so.7`0x801631727 259 0.9% gpsd`0x4084fa 261 0.9% libc.so.7`0x8016315ed 264 0.9% libc.so.7`0x8016315a6 264 0.9% libc.so.7`0x801631775 264 0.9% libc.so.7`0x801631789 266 1.0% libc.so.7`0x801631564 267 1.0% libc.so.7`0x8016316e4 268 1.0% libc.so.7`0x8016316eb 271 1.0% libc.so.7`0x8016315d9 274 1.0% libc.so.7`0x801631771 276 1.0% libc.so.7`0x801631656 278 1.0% libc.so.7`0x8016315dd 287 1.0% libc.so.7`0x8016316e0 288 1.0% libc.so.7`0x801631615 292 1.0% libc.so.7`0x801631782 295 1.1% libc.so.7`0x801631611 306 1.1% libc.so.7`0x80163175c 336 1.2% libc.so.7`0x801631790 349 1.3% libc.so.7`0x8016315b3 485 1.7% libc.so.7`0x80163177e 529 1.9% libc.so.7`0x801631767 550 2.0% libc.so.7`clock_gettime 769 2.8% libc.so.7`0x801631764 796 2.9% libc.so.7`0x8016315d5 800 2.9% libc.so.7`0x80163164f 1029 3.7% libc.so.7`0x801631607 1106 4.0% libc.so.7`0x801631593 1335 4.8% libc.so.7`0x8016315a0 1634 5.9% libc.so.7`0x80163159a 1849 6.6% libc.so.7`0x80163160c 2132 7.6% gpsd`gpsd_report 2162 7.8% libc.so.7`0x80163179d 3052 10.9% So it shows _some_ function names from libc but mostly not.. Is there a = way to improve it? -- Daniel O'Connor "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C From owner-freebsd-stable@FreeBSD.ORG Tue Jan 13 13:42:59 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1F316FC2 for ; Tue, 13 Jan 2015 13:42:59 +0000 (UTC) Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by mx1.freebsd.org (Postfix) with ESMTP id AF910F58 for ; Tue, 13 Jan 2015 13:42:58 +0000 (UTC) Received: from ppp14-2-13-162.lns21.adl2.internode.on.net (HELO leader.local) ([14.2.13.162]) by ipmail07.adl2.internode.on.net with ESMTP; 14 Jan 2015 00:07:48 +1030 Message-ID: <54B51FA0.6040307@ShaneWare.Biz> Date: Wed, 14 Jan 2015 00:07:36 +1030 From: Shane Ambler User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: FreeBSD stable Subject: Question about sysctl's Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2015 13:42:59 -0000 I am curious about two particular sysctl's, they are described as vm.stats.vm.v_wire_count: Wired pages vm.max_wired: System-wide limit to wired page count Now to me that sounds like v_wire_count should never get higher than max_wired, yet today I have a v_wire_count that is ~40% above the max_wired. vm.stats.vm.v_wire_count: 938417 vm.max_wired: 662820 So is this an error or do these two work differently than they appear? FreeBSD leader.local 10.1-STABLE FreeBSD 10.1-STABLE #0 r276527: Fri Jan 2 21:55:56 ACDT 2015 root@leader.local:/usr/obj/usr/src/sys/GENERIC amd64 -- FreeBSD - the place to B...Software Developing Shane Ambler From owner-freebsd-stable@FreeBSD.ORG Tue Jan 13 18:26:58 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9745D330 for ; Tue, 13 Jan 2015 18:26:58 +0000 (UTC) Received: from tensor.andric.com (unknown [IPv6:2001:7b8:3a7:1:2d0:b7ff:fea0:8c26]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "tensor.andric.com", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 50CBA7B4 for ; Tue, 13 Jan 2015 18:26:58 +0000 (UTC) Received: from [IPv6:2001:7b8:3a7::c148:42bb:70c4:c38e] (unknown [IPv6:2001:7b8:3a7:0:c148:42bb:70c4:c38e]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 7C583B80A; Tue, 13 Jan 2015 19:26:52 +0100 (CET) Subject: Re: Upgrading from stable/8 to stable/9 blocked by file 5.21 (r276416) Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) Content-Type: multipart/signed; boundary="Apple-Mail=_D4BA8B24-22ED-44CD-993E-FEB44200E602"; protocol="application/pgp-signature"; micalg=pgp-sha1 X-Pgp-Agent: GPGMail 2.5b4 From: Dimitry Andric In-Reply-To: Date: Tue, 13 Jan 2015 19:26:42 +0100 Message-Id: <6E1192E7-DC34-452E-891C-A53DC92C3B0A@FreeBSD.org> References: To: =?utf-8?Q?Trond_Endrest=C3=B8l?= X-Mailer: Apple Mail (2.1993) Cc: FreeBSD stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2015 18:26:58 -0000 --Apple-Mail=_D4BA8B24-22ED-44CD-993E-FEB44200E602 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 On 04 Jan 2015, at 18:41, Trond Endrest=F8l = wrote: >=20 > I'm investigating how to convert my stable/8 systems to stable/9, and > subsequently to stable/10. ... > =3D=3D=3D> lib/libmagic (obj,build-tools) > cc -O2 -pipe -DMAGIC=3D'"/usr/share/misc/magic"' -DHAVE_CONFIG_H = -I/usr/src/lib/libmagic -I/usr/src/lib/libmagic/../../contrib/file/src = -std=3Dgnu99 -I/usr/obj/usr/src/tmp/legacy/usr/include -DCOMPILE_ONLY = -L/usr/obj/usr/src/tmp/legacy/usr/lib -o mkmagic = /usr/src/lib/libmagic/../../contrib/file/src/apprentice.c = /usr/src/lib/libmagic/../../contrib/file/src/cdf_time.c = /usr/src/lib/libmagic/../../contrib/file/src/encoding.c = /usr/src/lib/libmagic/../../contrib/file/src/funcs.c = /usr/src/lib/libmagic/../../contrib/file/src/magic.c = /usr/src/lib/libmagic/../../contrib/file/src/print.c -lz -legacy > In file included from = /usr/src/lib/libmagic/../../contrib/file/src/apprentice.c:32: > /usr/src/lib/libmagic/../../contrib/file/src/file.h:488:21: error: = xlocale.h: No such file or directory FYI, I have submitted a fix for this issue here, to be reviewed: https://reviews.freebsd.org/D1518 -Dimitry --Apple-Mail=_D4BA8B24-22ED-44CD-993E-FEB44200E602 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.26 iEYEARECAAYFAlS1Y2oACgkQsF6jCi4glqNj3QCfcVP5vZUB/JXbe0jtj+kKxHJz 1J4AoNxtqlM4JebPfvKODCEZ//EDkrkK =HOv/ -----END PGP SIGNATURE----- --Apple-Mail=_D4BA8B24-22ED-44CD-993E-FEB44200E602-- From owner-freebsd-stable@FreeBSD.ORG Tue Jan 13 18:41:51 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 81E0177A for ; Tue, 13 Jan 2015 18:41:51 +0000 (UTC) Received: from mail.ultra-secure.de (mail.ultra-secure.de [88.198.178.88]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C1781A01 for ; Tue, 13 Jan 2015 18:41:49 +0000 (UTC) Received: (qmail 43073 invoked by uid 89); 13 Jan 2015 18:41:57 -0000 Received: from unknown (HELO ?192.168.1.200?) (rainer@ultra-secure.de@217.71.83.52) by mail.ultra-secure.de with ESMTPA; 13 Jan 2015 18:41:57 -0000 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) Subject: Re: FreeBSD 10.1-amd64 -> booting on a HP DL380 Gen9 results in panic From: Rainer Duffner In-Reply-To: Date: Tue, 13 Jan 2015 19:41:39 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: =?utf-8?Q?Hans_J=C3=B8rgen_Jakobsen?= X-Mailer: Apple Mail (2.1993) Cc: freebsd-stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2015 18:41:51 -0000 > Am 19.12.2014 um 09:01 schrieb Hans J=C3=B8rgen Jakobsen = : >=20 > Got to work: > System Options -> Processor Options -> Processor x2APIC Support = [Disabled] > /hjj >=20 Hi,=20 just for reference: the gentleman above mailed me privately with this = tip. Due to holidays etc. it got sort of lost. But the above BIOS-settings allowed my co-worker to install FreeBSD 10.1 = with the UEFI installer. Most helpful - thanks a lot! Best Regards, Rainer From owner-freebsd-stable@FreeBSD.ORG Tue Jan 13 22:30:05 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 90974DFC for ; Tue, 13 Jan 2015 22:30:05 +0000 (UTC) Received: from dnvrco-oedge-vip.email.rr.com (dnvrco-outbound-snat.email.rr.com [107.14.73.232]) by mx1.freebsd.org (Postfix) with ESMTP id 6158876D for ; Tue, 13 Jan 2015 22:30:04 +0000 (UTC) Received: from [96.28.178.143] ([96.28.178.143:59017] helo=localhost) by dnvrco-oedge01 (envelope-from ) (ecelerity 3.5.0.35861 r(Momo-dev:tip)) with ESMTP id CB/68-13566-56C95B45; Tue, 13 Jan 2015 22:29:58 +0000 Date: Tue, 13 Jan 2015 22:29:57 +0000 Message-ID: From: "Thomas Mueller" To: freebsd-stable@freebsd.org Subject: Can't make buildworld: very quick error X-RR-Connecting-IP: 107.14.64.118:25 X-Authority-Analysis: v=2.1 cv=QOzmR27L c=1 sm=1 tr=0 a=RKm8ZHSrUWUxlfG+7GhaOw==:117 a=RKm8ZHSrUWUxlfG+7GhaOw==:17 a=ayC55rCoAAAA:8 a=pedpZTtsAAAA:8 a=Twlkf-z8AAAA:8 a=MFL0tfOqBP9cj3q6SrIA:9 X-Cloudmark-Score: 0 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2015 22:30:05 -0000 I got a very quick error on "make buildworld" for i386 after success from the same source tree on amd64. It looked like something with clang, so I checked UPDATING file, no clue for 10-STABLE in contrast to HEAD. So I look stuck, though I could try to cross-build from just-finished update of FreeBSD 10-STABLE amd64. Is it some silly little oversight on my part, or is something really awry? Log file follows: -------------------------------------------------------------- >>> World build started on Tue Jan 13 21:39:45 UTC 2015 -------------------------------------------------------------- -------------------------------------------------------------- >>> Rebuilding the temporary build tree -------------------------------------------------------------- rm -rf /STABLE1/usr/obj.i386/STABLE1/usr/src/tmp mkdir -p /STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/lib mkdir -p /STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/usr mkdir -p /STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/legacy/bin mkdir -p /STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/legacy/usr mtree -deU -f /STABLE1/usr/src/etc/mtree/BSD.usr.dist -p /STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/legacy/usr >/dev/null mtree -deU -f /STABLE1/usr/src/etc/mtree/BSD.groff.dist -p /STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/legacy/usr >/dev/null mtree -deU -f /STABLE1/usr/src/etc/mtree/BSD.usr.dist -p /STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/usr >/dev/null mtree -deU -f /STABLE1/usr/src/etc/mtree/BSD.include.dist -p /STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/usr/include >/dev/null ln -sf /STABLE1/usr/src/sys /STABLE1/usr/obj.i386/STABLE1/usr/src/tmp -------------------------------------------------------------- >>> stage 1.1: legacy release compatibility shims -------------------------------------------------------------- cd /STABLE1/usr/src; MAKEOBJDIRPREFIX=/STABLE1/usr/obj.i386/STABLE1/usr/src/tmp INSTALL="sh /STABLE1/usr/src/tools/install.sh" PATH=/STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/legacy/usr/sbin:/STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/legacy/usr/bin:/STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/legacy/usr/games:/STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/legacy/bin:/sbin:/bin:/usr/sbin:/usr/bin WORLDTMP=/STABLE1/usr/obj.i386/STABLE1/usr/src/tmp VERSION="FreeBSD 10.1-STABLE i386 1001506" MAKEFLAGS="-m /STABLE1/usr/src/tools/build/mk -m /STABLE1/usr/src/share/mk" COMPILER_TYPE=clang make -f Makefile.inc1 DESTDIR= BOOTSTRAPPING=1000710 SSP_CFLAGS= -DWITHOUT_HTML -DWITHOUT_INFO -DNO_LINT -DWITHOUT_MAN -DNO_PIC -DNO_PROFILE -DNO_SHARED _BOOTSTRAP_MAKEINFO=yes -DNO_CPU_CFLAGS -DNO_WARNS -DNO_CTF -DEARLY_BUILD -DNO_TESTS legacy ===> tools/build (obj,includes,depend,all,install) /STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/STABLE1/usr/src/tools/build created for /STABLE1/usr/src/tools/build set -e; cd /STABLE1/usr/src/tools/build; make buildincludes; make installincludes rm -f .depend mkdep -f .depend -a -I/STABLE1/usr/src/tools/build/../../contrib/libc-pwcache -I/STABLE1/usr/src/tools/build/../../lib/libc/include -I/STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/legacy/usr/include -std=gnu99 /STABLE1/usr/src/tools/build/../../contrib/libc-pwcache/pwcache.c Stack dump: 0. Program arguments: /usr/bin/cc -cc1 -triple i386-unknown-freebsd10.0 -Eonly -disable-free -disable-llvm-verifier -main-file-name pwcache.c -mrelocation-model static -mdisable-fp-elim -masm-verbose -mconstructor-aliases -target-cpu i486 -resource-dir /usr/bin/../lib/clang/3.4.1 -dependency-file - -MT pwcache.o -sys-header-deps -I /STABLE1/usr/src/tools/build/../../contrib/libc-pwcache -I /STABLE1/usr/src/tools/build/../../lib/libc/include -I /STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/legacy/usr/include -std=gnu99 -fdebug-compilation-dir /STABLE1/usr/obj.i386/STABLE1/usr/src/tmp/STABLE1/usr/src/tools/build -ferror-limit 19 -fmessage-length 0 -mstackrealign -fobjc-runtime=gnustep -fdiagnostics-show-option -vectorize-slp -x c /STABLE1/usr/src/tools/build/../../contrib/libc-pwcache/pwcache.c cc: error: unable to execute command: Segmentation fault (core dumped) cc: error: clang frontend command failed due to signal (use -v to see invocation) FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 Target: i386-unknown-freebsd10.0 Thread model: posix cc: note: diagnostic msg: PLEASE submit a bug report to http://llvm.org/bugs/ and include the crash backtrace, preprocessed source, and associated run script. cc: error: unable to execute command: Segmentation fault (core dumped) cc: note: diagnostic msg: Error generating preprocessed source(s). mkdep: compile failed *** Error code 1 Stop. make[3]: stopped in /STABLE1/usr/src/tools/build *** Error code 1 Stop. make[2]: stopped in /STABLE1/usr/src *** Error code 1 Stop. make[1]: stopped in /STABLE1/usr/src *** Error code 1 Stop. make: stopped in /STABLE1/usr/src Tom From owner-freebsd-stable@FreeBSD.ORG Tue Jan 13 22:30:57 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 860B8FE8; Tue, 13 Jan 2015 22:30:57 +0000 (UTC) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2994D816; Tue, 13 Jan 2015 22:30:56 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.9/8.14.9) with ESMTP id t0DMUsQx061186; Tue, 13 Jan 2015 17:30:54 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.9/8.14.4/Submit) id t0DMUs2c061183; Tue, 13 Jan 2015 17:30:54 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <21685.40094.453028.585630@hergotha.csail.mit.edu> Date: Tue, 13 Jan 2015 17:30:54 -0500 From: Garrett Wollman To: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Some filesystem performance numbers X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (hergotha.csail.mit.edu [127.0.0.1]); Tue, 13 Jan 2015 17:30:55 -0500 (EST) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS autolearn=disabled version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on hergotha.csail.mit.edu X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2015 22:30:57 -0000 I recently bought a copy of the SPECsfs2014 benchmark, and I've been using it to test out our NFS server platform. One scenario of interest to me is identifying where the limits are in terms of the local CAM/storage/filesystem implementation versus bottlenecks unique to the NFS server, and to that end I've been running the benchmark suite directly on the server's local disk. (This is of course also the way you'd benchmark for shared-nothing container-based virtualization.) I have found a few interesting results on my test platform: 1) I can quantify the cost of using SHA256 vs. fletcher4 as the ZFS checksum algorithm. On the VDA workload (essentially a simulated video streaming/recording application), my server can do about half as many "streams" with SHA256 as it can with fletcher4. 2) Both L2ARC and separate ZIL have small but measurable performance impacts. I haven't examined the differences closely. 3) LZ4 compression also makes a small performance impact, but as advertised, less than LZJB for mostly-incompressible data. I hope to be able to present the actual benchmark results at some point, as well as some results for the other three workloads. -GAWollman From owner-freebsd-stable@FreeBSD.ORG Wed Jan 14 03:19:01 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8D03F605 for ; Wed, 14 Jan 2015 03:19:01 +0000 (UTC) Received: from slim.berklix.org (slim.berklix.org [94.185.90.68]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 024278AB for ; Wed, 14 Jan 2015 03:19:00 +0000 (UTC) Received: from mart.js.berklix.net (p57BCF35A.dip0.t-ipconnect.de [87.188.243.90]) (authenticated bits=128) by slim.berklix.org (8.14.5/8.14.5) with ESMTP id t0E2DS9V081052; Wed, 14 Jan 2015 03:13:28 +0100 (CET) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by mart.js.berklix.net (8.14.3/8.14.3) with ESMTP id t0E2BBV4061614; Wed, 14 Jan 2015 03:11:12 +0100 (CET) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.14.7/8.14.7) with ESMTP id t0E2AeZY003052; Wed, 14 Jan 2015 03:11:11 +0100 (CET) (envelope-from jhs@berklix.com) Message-Id: <201501140211.t0E2AeZY003052@fire.js.berklix.net> To: freebsd-stable@freebsd.org Subject: 9 stable is in worse shape than current ! Some fixes. From: "Julian H. Stacey" Organization: http://berklix.com BSD Linux Unix Consultants, Munich Germany User-agent: EXMH on FreeBSD http://www.berklix.com/free/ X-URL: http://www.berklix.com/~jhs/cv/ Date: Wed, 14 Jan 2015 03:10:40 +0100 Cc: "Julian H. Stacey" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2015 03:19:01 -0000 Hi freebsd-stable@freebsd.org, 9 stable is a lot worse than current to build ! Suprising as in the old days it used to be the other way, but on 2 current boxes here I have very little trouble building, (usually just new includes needed), whereas 9 stable is lots of trouble: My env.: 9-stable ( .ctm_status src-9 1374, .svn_revision 277102 ) (within a prison with 9.2 FreeBSD 9.2-STABLE FreeBSD 9.2-STABLE #3 r264390: Sun Apr 13 12:16:37 CEST 2014 :/usr/obj/usr/src/sys/GENERIC amd64 ) The jail has all ist own binaries, not shared with prison... & with nothing in /etc/make.conf except NO_FSCHG=YES To ease debugging of include paths after interrupted dependent makes etc, I did not use a /usr/obj/ (though I do normally). Problem 1 - Solved: 9-stable default : cc -v # gcc version 4.2.1 11-Current default : cc -v # clang version 3.5.0 In both cases my boxes use Unchanged default cc. It seems developers only tested make world & bsd.sys.mk with clang ! These errors: ===> lib/libfetch (all) SSL cc1: warnings being treated as errors common.c: In function 'fetch_ssl': common.c:808: warning: unused parameter 'URL' ===> lib/libmagic (all) cc1: warnings being treated as errors /usr/src/lib/libmagic/../../contrib/file/src/apprentice.c:942: warning: 'apprentice_list' defined but not used Can be avoided by applying this emergency patch-out: --------- *** 9-stable/src//share/mk/bsd.sys.mk Wed Jan 14 02:02:26 2015 --- new/src/share/mk/bsd.sys.mk Wed Jan 14 02:03:23 2015 *************** *** 32,38 **** CWARNFLAGS+= -Wsystem-headers .if !defined(NO_WERROR) && (${COMPILER_TYPE} != "clang" \ || !defined(NO_WERROR.clang)) ! CWARNFLAGS+= -Werror .endif # !NO_WERROR && (!CLANG || !NO_WERROR.clang) .endif # WARNS >= 1 .if ${WARNS} >= 2 --- 32,38 ---- CWARNFLAGS+= -Wsystem-headers .if !defined(NO_WERROR) && (${COMPILER_TYPE} != "clang" \ || !defined(NO_WERROR.clang)) ! ### CWARNFLAGS+= -Werror .endif # !NO_WERROR && (!CLANG || !NO_WERROR.clang) .endif # WARNS >= 1 .if ${WARNS} >= 2 *************** *** 97,103 **** .endif # CLANG .if !defined(NO_WERROR) && (${COMPILER_TYPE} != "clang" \ || !defined(NO_WERROR.clang)) ! CWARNFLAGS+= -Werror .endif # !NO_WERROR && (!CLANG || !NO_WERROR.clang) .endif # WFORMAT > 0 .endif # WFORMAT --- 97,103 ---- .endif # CLANG .if !defined(NO_WERROR) && (${COMPILER_TYPE} != "clang" \ || !defined(NO_WERROR.clang)) ! ### CWARNFLAGS+= -Werror .endif # !NO_WERROR && (!CLANG || !NO_WERROR.clang) .endif # WFORMAT > 0 .endif # WFORMAT --------- Problem 2 - Not Solved # ===> lib/libarchive (all) # /usr/src/lib/libarchive/../../contrib/libarchive/libarchive/archive_hash.h:129:20: error: sha1.h: No such file or directory Problem 3 - Not Solved ===> libexec/telnetd ... undefined reference ... Problem 4 - Not Solved - in /etc/src.conf I had to add: WITHOUT_ATM="YES" # sbin/atm/atmconfig WITHOUT_OPENSSL="YES" WITHOUT_RESCUE="YES" # WITHOUT_BSNMP="YES" # lib/libbsnmp/libbsnmp # No longer need to avoid that, maybe fixed by bsd.sys.mk.diff Anyone else see these problems ? Suggestions ? These observations are on a production server I've temporarily patched out from active service, but I want to return it soon, so unless there's some quick fixes, I'll have to down grade it from 9-stable to 9.3-RELEASE, cos I dont care about things like atm, but I do need ssl & ssh. Cheers, Julian -- Julian Stacey, BSD Linux Unix C Sys Eng Consultant Munich http://berklix.com Indent previous with "> ". Interleave reply paragraphs like a play script. Send plain text, not quoted-printable, HTML, base64, or multipart/alternative. - - - - - - - Practice French & support democracy ? Buy on 14 Jan http://www.charliehebdo.fr BBC said: A special print run of 3 million in 16 languages not just French. In Munich on 15th: Purchasable at Haupt Bahn Hof International Presse. From owner-freebsd-stable@FreeBSD.ORG Wed Jan 14 07:41:27 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DFF0E122 for ; Wed, 14 Jan 2015 07:41:27 +0000 (UTC) Received: from nm19-vm7.access.bullet.mail.bf1.yahoo.com (nm19-vm7.access.bullet.mail.bf1.yahoo.com [216.109.115.102]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8E4E4759 for ; Wed, 14 Jan 2015 07:41:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bellsouth.net; s=s2048; t=1421221130; bh=+l09pVBnjK8kR5B2C7Is+jBmQTsoY1I3DoV+/gXH4do=; h=Date:From:To:CC:References:Subject:From:Subject; b=ks91Ada/bxNuOHuX6ZPJaGyvvpMSNK4+BhTelvnbJ9qP7SoXA4d801GqfjkW2/Qzlv6sSWEI/zN9ivEDWXgvr4OR2WTEmAzEuHpbmX5zxtQAVXYxYHO+RSA/1Ii+XnJoNAdDhHAI6vx5SDlwNGgJ6cdDOHBj/JfbjY26LiId6wLTZa/HrDLSsyAgzgVZzF6ryy/cvEDHdvswvPAJ5JCKV9w2P2KPw9mcHFHiT3XaDnUa8Xt2DNdOtmkIZ/A91dSOq2px87qyAfBac6e0IUH4I/nFC64Y6dNy73x3Xg60cYbj27W45G/hQER/YzI7vz4g3RyuEfbPPAJ5ypPPLtdB1A== Received: from [66.196.81.165] by nm19.access.bullet.mail.bf1.yahoo.com with NNFMP; 14 Jan 2015 07:38:50 -0000 Received: from [98.138.226.241] by tm11.access.bullet.mail.bf1.yahoo.com with NNFMP; 14 Jan 2015 07:38:50 -0000 Received: from [127.0.0.1] by smtp112.sbc.mail.ne1.yahoo.com with NNFMP; 14 Jan 2015 07:38:50 -0000 X-Yahoo-Newman-Id: 25905.40218.bm@smtp112.sbc.mail.ne1.yahoo.com Message-ID: <25905.40218.bm@smtp112.sbc.mail.ne1.yahoo.com> Date: Tue, 13 Jan 2015 23:38:50 -0800 (PST) X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: 1b5oVu4VM1lPqwFaOmbITmJBfwDmyjQar0eTsPKs3uERHhz _DRqaPa2bBREuMOldXdnjwM8a9moNMjjjfMLnOS0fQDYmOAZj.gtzuLWj8H_ Zhh76eCbm94BC3lgpsK.Q0GrahOmoRZfiNKemx7X4YgtRnjpv4jhsW62tlMa _Shb9XYdEeMqk1OARcIVRtuMtTpDDtcjBcAcP9saFIX5rXwNgMH5HtG7POzv 2hJCZcvwsbUQUGCLnH9zeKWIXARIBYiamkMXn8dk1g3WrDbNRRDO43.iaOwc YGcp650WHoRlIocZfsXJGyQJoRYnmx66nkISY92vqBiUyEwhCx198D79hJDM EAS7IWfeH4SZnZ6C4HAaWVUajA.bLVAeJLMO1mNT497hOV6LLIRgR0aHCXhD 9_oOlnwbJxzBiitufN98vY72DE5T.KHKHIZkBPDsyyqh.88ia2dLxCuqCKMw MXX.rNboX7obDCF3n.LG91mJdC44uAs1dAe9.AjKTYaupmQv2cxsozxxPgrs 2a4xAf4zob1nFCIk7Wt4oTtxuHxbB5rP2_9ZsWt5v6EoZmUXK5qb8uFivqg- - X-Yahoo-SMTP: Kz_aW1.swBBYof3zAD7.RWzXz9ZAQVDMml1VADsbgPT4Kq79LC0- From: "Thomas Mueller" To: freebsd-stable@freebsd.org References: <20150113223353.GF1151@albert.catwhisker.org> Subject: Re: Can't make buildworld: very quick error X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2015 07:41:28 -0000 My original post didn't appear on the list as downloaded, but one response indicates it made the list. Anyway, I found the reason for the buildworld failure as I was later unable to update ports, even pkg, with portmaster. Filesystem was corrupted, as evidenced by booting back to 10.1-STABLE amd64 and running fsck_ffs -y on the partition in question, not mounted. I kept getting errors on unreadable sectors, and it might have gone on forever had I not stopped it with Ctrl-C. Corrupted installation, 10-STABLE i386, was on a 32 GB Kingston Data Traveler USB 3.0. So all I can do is give up on this USB stick, it looks kaputt. End of story on this issue. Tom From owner-freebsd-stable@FreeBSD.ORG Wed Jan 14 14:17:07 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5E0D7FD5 for ; Wed, 14 Jan 2015 14:17:07 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1B5757EE for ; Wed, 14 Jan 2015 14:17:07 +0000 (UTC) Received: from new-host.home (pool-173-70-85-31.nwrknj.fios.verizon.net [173.70.85.31]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id C6899B91E; Wed, 14 Jan 2015 09:17:04 -0500 (EST) Message-ID: <54B67A60.2000708@FreeBSD.org> Date: Wed, 14 Jan 2015 09:17:04 -0500 From: John Baldwin User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: "O'Connor, Daniel" , FreeBSD Stable Mailing List Subject: Re: DTrace and function names References: In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 14 Jan 2015 09:17:04 -0500 (EST) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2015 14:17:07 -0000 On 1/12/15 7:20 PM, O'Connor, Daniel wrote: > Hi, > I am trying to debug why astro/gpsd uses a lot of CPU so I thought I'd try the 'hotuser' Dtrace toolkit script but the output looks like so.. > [midget 9:24] ~ >sudo /usr/local/share/DTraceToolkit/hotuser -p 37690 > Password: > Sampling... Hit Ctrl-C to end. > ^C > FUNCTION COUNT PCNT > libc.so.7`0x801631777 1 0.0% > libc.so.7`strtol 1 0.0% > libc.so.7`0x801631658 1 0.0% > libc.so.7`0x801631625 1 0.0% > libc.so.7`0x801631636 1 0.0% > libc.so.7`0x801631566 1 0.0% > libc.so.7`0x801631623 1 0.0% > gpsd`0x405ba6 1 0.0% > gpsd`0x404fd6 1 0.0% > libc.so.7`0x801631797 10 0.0% > libc.so.7`0x8016315f9 50 0.2% > libc.so.7`0x80163165c 63 0.2% > libc.so.7`0x80163179b 115 0.4% > libc.so.7`0x8016315b9 119 0.4% > libc.so.7`0x801631794 135 0.5% > libc.so.7`0x801631791 170 0.6% > libc.so.7`0x801631601 212 0.8% > libc.so.7`0x80163165d 226 0.8% > libc.so.7`0x801631757 233 0.8% > libc.so.7`0x8016315bd 235 0.8% > gpsd`0x4028bc 237 0.8% > libc.so.7`0x801631736 238 0.9% > libc.so.7`0x80163156a 239 0.9% > libc.so.7`0x801631617 243 0.9% > gpsd`0x4084ea 246 0.9% > gpsd`0x408508 251 0.9% > libc.so.7`0x8016317a0 251 0.9% > libc.so.7`0x801631571 252 0.9% > libc.so.7`0x801631749 252 0.9% > libc.so.7`0x801631583 253 0.9% > libc.so.7`0x8016315c7 256 0.9% > libc.so.7`0x801631727 259 0.9% > gpsd`0x4084fa 261 0.9% > libc.so.7`0x8016315ed 264 0.9% > libc.so.7`0x8016315a6 264 0.9% > libc.so.7`0x801631775 264 0.9% > libc.so.7`0x801631789 266 1.0% > libc.so.7`0x801631564 267 1.0% > libc.so.7`0x8016316e4 268 1.0% > libc.so.7`0x8016316eb 271 1.0% > libc.so.7`0x8016315d9 274 1.0% > libc.so.7`0x801631771 276 1.0% > libc.so.7`0x801631656 278 1.0% > libc.so.7`0x8016315dd 287 1.0% > libc.so.7`0x8016316e0 288 1.0% > libc.so.7`0x801631615 292 1.0% > libc.so.7`0x801631782 295 1.1% > libc.so.7`0x801631611 306 1.1% > libc.so.7`0x80163175c 336 1.2% > libc.so.7`0x801631790 349 1.3% > libc.so.7`0x8016315b3 485 1.7% > libc.so.7`0x80163177e 529 1.9% > libc.so.7`0x801631767 550 2.0% > libc.so.7`clock_gettime 769 2.8% > libc.so.7`0x801631764 796 2.9% > libc.so.7`0x8016315d5 800 2.9% > libc.so.7`0x80163164f 1029 3.7% > libc.so.7`0x801631607 1106 4.0% > libc.so.7`0x801631593 1335 4.8% > libc.so.7`0x8016315a0 1634 5.9% > libc.so.7`0x80163159a 1849 6.6% > libc.so.7`0x80163160c 2132 7.6% > gpsd`gpsd_report 2162 7.8% > libc.so.7`0x80163179d 3052 10.9% > > So it shows _some_ function names from libc but mostly not.. Is there a way to improve it? Build with debug symbols? For libc you can do that via: % cd /usr/src/lib/libc % make cleandir % make obj # May want to use "-O -g" to reduce inlining % make DEBUG_FLAGS="-g" depend all install For gpsd you'll have to figure out a way to get it built with extra symbols (if the port has a DEBUG option, enable that, otherwise you might have to hack the port). -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Wed Jan 14 14:18:58 2015 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A408A14E for ; Wed, 14 Jan 2015 14:18:58 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7DF2080E for ; Wed, 14 Jan 2015 14:18:58 +0000 (UTC) Received: from new-host.home (pool-173-70-85-31.nwrknj.fios.verizon.net [173.70.85.31]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 8E257B91E; Wed, 14 Jan 2015 09:18:57 -0500 (EST) Message-ID: <54B67AD1.5050609@FreeBSD.org> Date: Wed, 14 Jan 2015 09:18:57 -0500 From: John Baldwin User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: Shane Ambler , FreeBSD stable Subject: Re: Question about sysctl's References: <54B51FA0.6040307@ShaneWare.Biz> In-Reply-To: <54B51FA0.6040307@ShaneWare.Biz> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 14 Jan 2015 09:18:57 -0500 (EST) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2015 14:18:58 -0000 On 1/13/15 8:37 AM, Shane Ambler wrote: > > I am curious about two particular sysctl's, they are described as > > vm.stats.vm.v_wire_count: Wired pages > vm.max_wired: System-wide limit to wired page count > > Now to me that sounds like v_wire_count should never get higher than > max_wired, yet today I have a v_wire_count that is ~40% above the > max_wired. > > vm.stats.vm.v_wire_count: 938417 > vm.max_wired: 662820 > > So is this an error or do these two work differently than they appear? Not all wired allocations are subject to the limit (which isn't obvious from the description). Anything that calls 'malloc()' in the kernel will use wired memory for example, and those generally aren't subject to the limit. However, user-initiated operations (like a sysctl buffer or user pages wired via mlock()) are. -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Wed Jan 14 21:12:57 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DC10EC4 for ; Wed, 14 Jan 2015 21:12:57 +0000 (UTC) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AC76BC07 for ; Wed, 14 Jan 2015 21:12:57 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id BEF4C21D15 for ; Wed, 14 Jan 2015 16:12:56 -0500 (EST) Received: from web3 ([10.202.2.213]) by compute1.internal (MEProxy); Wed, 14 Jan 2015 16:12:56 -0500 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=message-id:x-sasl-enc:from:to :mime-version:content-transfer-encoding:content-type:in-reply-to :references:subject:date; s=smtpout; bh=jm8WCNTtPPGy/7bqGLgfvZEk ER0=; b=ZasXO4JCdCUmZz51Usl+6AaJYUycsUrBQlEJKywZU2uStYPJ2NeCEKaJ h0FToj4KkpgQxfM8YgBKpUwR6ropVaIsqRyW0ig4Jt24/WKK8ylzPoBPqhFECEh5 HBoLopoYJawbXTOrasRtaTy7FmkOCTuqCZc4Sp7lz79/F8RC5zk= Received: by web3.nyi.internal (Postfix, from userid 99) id 9AC9411054C; Wed, 14 Jan 2015 16:12:56 -0500 (EST) Message-Id: <1421269976.1116901.213997149.582CB93B@webmail.messagingengine.com> X-Sasl-Enc: 6ODmneHGYHkSzS9kMb00XjkF129T4uT5BVIwU72gIPvh 1421269976 From: Mark Felder To: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: MessagingEngine.com Webmail Interface - ajax-46f3f2c7 In-Reply-To: <54AA5613.4050303@omnilan.de> References: <54A17F33.2020708@ish.com.au> <54A1ED2F.2070305@heuristicsystems.com.au> <54AA5613.4050303@omnilan.de> Subject: Re: PMTU (must fragment) with ipsec [Was: Re: ipsec routing issue] Date: Wed, 14 Jan 2015 15:12:56 -0600 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2015 21:12:57 -0000 On Mon, Jan 5, 2015, at 03:14, Harry Schmalzbauer wrote: > Bez=FCglich Dewayne Geraghty's Nachricht vom 30.12.2014 01:09 (localtime= ): > > Ari, > > > > Bjoern offers good advise (as usual). This practical example might >=20 > Hello, >=20 > I'm quiet familar with ipsec(4), enc(1) and companions, but I haven't > found a way to make routers return ICMP "must fragment" with gif-less > tunnels. > My last attempt was adding disc(4), assign it a MTU of 1420 and add a > static route which points to disc. > That works for 'route get remotelan' on the router itself, it's > reporting correctly the mtu of 1420, but nevertheless, the router never > returns "must fragment" (which I'd need because FreeBSD has PMTU on and > we use jumbo frames). > Apperently fragementation is handled before packets arrive at the > outgoing interface. Of course, kernel policy "steals" the packet before > ot reaches "outgoing" state. > Do I miss any trick? > You can apply an MTU to a route instead of an interface, so perhaps that would work better? Just add -mtu 1420 at the end of your route statement and it will work its magic. :-) From owner-freebsd-stable@FreeBSD.ORG Wed Jan 14 22:01:58 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3D4C5AD5; Wed, 14 Jan 2015 22:01:58 +0000 (UTC) Received: from slim.berklix.org (slim.berklix.org [94.185.90.68]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 75C3C3B1; Wed, 14 Jan 2015 22:01:56 +0000 (UTC) Received: from mart.js.berklix.net (p5DCBD518.dip0.t-ipconnect.de [93.203.213.24]) (authenticated bits=128) by slim.berklix.org (8.14.5/8.14.5) with ESMTP id t0EM46V7087809; Wed, 14 Jan 2015 23:04:07 +0100 (CET) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by mart.js.berklix.net (8.14.3/8.14.3) with ESMTP id t0EM1m06067568; Wed, 14 Jan 2015 23:01:49 +0100 (CET) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.14.7/8.14.7) with ESMTP id t0EM1HMb054891; Wed, 14 Jan 2015 23:01:48 +0100 (CET) (envelope-from jhs@berklix.com) Message-Id: <201501142201.t0EM1HMb054891@fire.js.berklix.net> To: freebsd-stable@freebsd.org Subject: Re: 9 stable is in worse shape than current ! Some fixes. From: "Julian H. Stacey" Organization: http://berklix.com BSD Unix Linux Consultants, Munich Germany User-agent: EXMH on FreeBSD http://berklix.com/free/ X-URL: http://www.berklix.com In-reply-to: Your message "Wed, 14 Jan 2015 03:10:40 +0100." <201501140211.t0E2AeZY003052@fire.js.berklix.net> Date: Wed, 14 Jan 2015 23:01:17 +0100 Cc: current@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2015 22:01:58 -0000 Added cc current@ (source of broken commits to stable, most likely) + text added below. "Julian H. Stacey" wrote Wed, 14 Jan 2015 03:10:40 +0100: > Hi freebsd-stable@freebsd.org, > > 9 stable is a lot worse than current to build ! > Suprising as in the old days it used to be the other way, but on > 2 current boxes here I have very little trouble building, (usually > just new includes needed), whereas 9 stable is lots of trouble: > > My env.: > 9-stable ( .ctm_status src-9 1374, .svn_revision 277102 ) > (within a prison with 9.2 > FreeBSD 9.2-STABLE FreeBSD 9.2-STABLE #3 r264390: > Sun Apr 13 12:16:37 CEST 2014 > :/usr/obj/usr/src/sys/GENERIC amd64 ) > The jail has all ist own binaries, not shared with prison... > & with nothing in /etc/make.conf except NO_FSCHG=YES > To ease debugging of include paths after interrupted dependent > makes etc, I did not use a /usr/obj/ (though I do normally). > > Problem 1 - Solved: > 9-stable default : cc -v # gcc version 4.2.1 > 11-Current default : cc -v # clang version 3.5.0 > In both cases my boxes use Unchanged default cc. > It seems developers only tested make world & bsd.sys.mk with clang ! > > These errors: > ===> lib/libfetch (all) SSL > cc1: warnings being treated as errors > common.c: In function 'fetch_ssl': > common.c:808: warning: unused parameter 'URL' > > ===> lib/libmagic (all) > cc1: warnings being treated as errors > /usr/src/lib/libmagic/../../contrib/file/src/apprentice.c:942: warning: 'apprentice_list' defined but not used > > Can be avoided by applying this emergency patch-out: > --------- > *** 9-stable/src//share/mk/bsd.sys.mk Wed Jan 14 02:02:26 2015 > --- new/src/share/mk/bsd.sys.mk Wed Jan 14 02:03:23 2015 > *************** > *** 32,38 **** > CWARNFLAGS+= -Wsystem-headers > .if !defined(NO_WERROR) && (${COMPILER_TYPE} != "clang" \ > || !defined(NO_WERROR.clang)) > ! CWARNFLAGS+= -Werror > .endif # !NO_WERROR && (!CLANG || !NO_WERROR.clang) > .endif # WARNS >= 1 > .if ${WARNS} >= 2 > --- 32,38 ---- > CWARNFLAGS+= -Wsystem-headers > .if !defined(NO_WERROR) && (${COMPILER_TYPE} != "clang" \ > || !defined(NO_WERROR.clang)) > ! ### CWARNFLAGS+= -Werror > .endif # !NO_WERROR && (!CLANG || !NO_WERROR.clang) > .endif # WARNS >= 1 > .if ${WARNS} >= 2 > *************** > *** 97,103 **** > .endif # CLANG > .if !defined(NO_WERROR) && (${COMPILER_TYPE} != "clang" \ > || !defined(NO_WERROR.clang)) > ! CWARNFLAGS+= -Werror > .endif # !NO_WERROR && (!CLANG || !NO_WERROR.clang) > .endif # WFORMAT > 0 > .endif # WFORMAT > --- 97,103 ---- > .endif # CLANG > .if !defined(NO_WERROR) && (${COMPILER_TYPE} != "clang" \ > || !defined(NO_WERROR.clang)) > ! ### CWARNFLAGS+= -Werror > .endif # !NO_WERROR && (!CLANG || !NO_WERROR.clang) > .endif # WFORMAT > 0 > .endif # WFORMAT > --------- > > > Problem 2 - Not Solved > # ===> lib/libarchive (all) > # /usr/src/lib/libarchive/../../contrib/libarchive/libarchive/archive_hash.h:129:20: error: sha1.h: No such file or directory > > > Problem 3 - Not Solved > ===> libexec/telnetd > ... undefined reference ... > > > Problem 4 - Not Solved - in /etc/src.conf I had to add: > > WITHOUT_ATM="YES" # sbin/atm/atmconfig > WITHOUT_OPENSSL="YES" > WITHOUT_RESCUE="YES" > > # WITHOUT_BSNMP="YES" # lib/libbsnmp/libbsnmp > # No longer need to avoid that, maybe fixed by bsd.sys.mk.diff > > Anyone else see these problems ? Suggestions ? > > These observations are on a production server I've temporarily > patched out from active service, but I want to return it soon, > so unless there's some quick fixes, I'll have to down grade it > from 9-stable to 9.3-RELEASE, cos I dont care about things like > atm, but I do need ssl & ssh. Downgrading from broken 9-stable src/ to 9.3-RELEASE solved everything! Broken code in bsd.sys.mk relate to 11-current's use of clang V. gcc in 9. There's other broken in 9-stable too It is easy to check without rebooting, just mount -t devfs dev /9stable/dev ; chroot /9stable ; cd /usr/src ; make I've tested this on my current box, re-making a 9.3-RELEASE I hope commiters try it, & back out broken 9stable commits. Thanks Cheers, Julian -- Julian Stacey, BSD Linux Unix C Sys Eng Consultant Munich http://berklix.com Indent previous with "> ". Interleave reply paragraphs like a play script. Send plain text, not quoted-printable, HTML, base64, or multipart/alternative. - - - - - - - Practice French & support democracy ? Buy on 14 Jan http://www.charliehebdo.fr A special print run of 5 million in 16 languages, not just French. In Munich on 15th at Haupt Bahn Hof International Presse. From owner-freebsd-stable@FreeBSD.ORG Thu Jan 15 09:07:19 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E0A95242; Thu, 15 Jan 2015 09:07:18 +0000 (UTC) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6C17AE2; Thu, 15 Jan 2015 09:07:18 +0000 (UTC) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [78.138.80.135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id t0F97DEb024486; Thu, 15 Jan 2015 10:07:13 +0100 (CET) (envelope-from h.schmalzbauer@omnilan.de) Received: from titan.inop.mo1.omnilan.net (titan.inop.mo1.omnilan.net [IPv6:2001:a60:f0bb:1::3:1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id F3B7972A; Thu, 15 Jan 2015 10:07:12 +0100 (CET) Message-ID: <54B78340.8090806@omnilan.de> Date: Thu, 15 Jan 2015 10:07:12 +0100 From: Harald Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Mark Felder Subject: Re: PMTU (must fragment) with ipsec [Was: Re: ipsec routing issue] References: <54A17F33.2020708@ish.com.au> <54A1ED2F.2070305@heuristicsystems.com.au> <54AA5613.4050303@omnilan.de> <1421269976.1116901.213997149.582CB93B@webmail.messagingengine.com> In-Reply-To: <1421269976.1116901.213997149.582CB93B@webmail.messagingengine.com> X-Enigmail-Version: 1.1.2 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig4892B0CC6ADC31C6ED5AB460" X-Greylist: ACL 119 matched, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [78.138.80.130]); Thu, 15 Jan 2015 10:07:13 +0100 (CET) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: 78.138.80.135; Sender-helo: mh0.gentlemail.de; ) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2015 09:07:19 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig4892B0CC6ADC31C6ED5AB460 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Bez=FCglich Mark Felder's Nachricht vom 14.01.2015 22:12 (localtime): =85 >> My last attempt was adding disc(4), assign it a MTU of 1420 and add a >> static route which points to disc. >> That works for 'route get remotelan' on the router itself, it's >> reporting correctly the mtu of 1420, but nevertheless, the router neve= r >> returns "must fragment" (which I'd need because FreeBSD has PMTU on an= d >> we use jumbo frames). >> Apperently fragementation is handled before packets arrive at the >> outgoing interface. Of course, kernel policy "steals" the packet befor= e >> ot reaches "outgoing" state. >> Do I miss any trick? >> > You can apply an MTU to a route instead of an interface, so perhaps tha= t > would work better? Just add -mtu 1420 at the end of your route statemen= t > and it will work its magic. :-) Thanks for the hint! But essentially the same happens for both types of MTU propagation. The local routing table forces packet length for outgoing packets on the router. In the gif(4)-less IPSec-tunnel scenario, there is no "outgoing" packet on the router. So hosts which forward packets to the router will never receive a "must fragement" icmp answer to packets larger than the MTU set on the router. I had to set the MTU on every single client in the lan=85 Not what I'm looking for, I'd like to get my router informing clients! I still have no idea how to accomplish :-( Thanks for further hints in advance, -Harry --------------enig4892B0CC6ADC31C6ED5AB460 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAlS3g0AACgkQLDqVQ9VXb8hp7gCfSg7NYOcJYMWl3ZvdOux6qX5Q DsAAnRBQ2UMSo+fauB8CnEC+UKHK4chr =3vhG -----END PGP SIGNATURE----- --------------enig4892B0CC6ADC31C6ED5AB460-- From owner-freebsd-stable@FreeBSD.ORG Thu Jan 15 17:22:52 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B647AF60 for ; Thu, 15 Jan 2015 17:22:52 +0000 (UTC) Received: from mail-we0-x233.google.com (mail-we0-x233.google.com [IPv6:2a00:1450:400c:c03::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B0D2239 for ; Thu, 15 Jan 2015 17:22:52 +0000 (UTC) Received: by mail-we0-f179.google.com with SMTP id q59so15962004wes.10 for ; Thu, 15 Jan 2015 09:22:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=pzHKFLMphev5AI1tANp4xbmaIbTWPJxsfP/95SRgqOU=; b=RLN5npwaDRuEGF4XFPpUbvVppu24bV38nKgOkcNYJHVZI99LA1xh+KqANDDp2NaUdr 1p+7FnJX2OGjAKrEnkNYbsLJNLjbOm9KrUC1fYkMYV1DSUKvNQsCAH/XhKGbZIu0Qxge 94jaoU3BnntQ4s47g2hsq+dB0ZyG63PdYd/qHGCirUThwrgCnvjYTneR3Z3EaKDVqUTK Blat19zrdwn+pJfVsNpwF0MkjcjvR8lvaGS4ixF9sxSQTL9IbDy8iFPQa6noVC8xCWf3 pr8qM9MNY3/7KRFTn7s10b+G3t83x6u69jMfGAe37vNEpziDb9cdCbtNcKwHsrD/cc1G m37A== X-Received: by 10.194.203.104 with SMTP id kp8mr15888541wjc.103.1421342570667; Thu, 15 Jan 2015 09:22:50 -0800 (PST) Received: from [192.168.3.105] ([193.148.0.35]) by mx.google.com with ESMTPSA id u18sm2843427wjq.42.2015.01.15.09.22.49 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Jan 2015 09:22:49 -0800 (PST) Message-ID: <54B7F769.40605@gmail.com> Date: Thu, 15 Jan 2015 19:22:49 +0200 From: Mihai Vintila User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Poor performance on Intel P3600 NVME driver Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2015 17:22:52 -0000 Hi all, I've got a: 10.1-RELEASE FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 21:02:49 UTC 2014 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 hw.machine: amd64 hw.model: Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz hw.ncpu: 12 hw.machine_arch: amd64 With 2 Intel P3600 nvmecontrol devlist nvme0: INTEL SSDPE2ME020T4 nvme0ns1 (1907729MB) nvme1: INTEL SSDPE2ME020T4 nvme1ns1 (1907729MB) That i've put in a mirror or single drive zfs pool. Issue that i'm having is that performance is very poor on the P3600 drives, they are really close to a S3500 which is really poor: Here is a iozone benchmark on a single drive pool with recordsize set to 4k and compression to lz4 Command line used: iozone -Rb /root/output.wks -O -i 0 -i 1 -i 2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -s 1G Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 1048576 4 74609 0 104268 0 95699 49975 1048576 8 36554 0 62927 0 59419 25778 1048576 32 9869 0 19148 0 18757 7134 1048576 64 5014 0 9612 0 9528 3813 1048576 128 2586 0 4908 0 4883 1962 While on S3500 i have: Command line used: iozone -Rb /root/output_nexenta.wks -O -i 0 -i 1 -i 2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -s 1G Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 1048576 4 66215 0 204121 0 162069 35408 1048576 8 54523 0 168679 0 137393 29711 1048576 32 10293 0 80652 0 75063 10462 1048576 64 23065 0 49044 0 46684 20179 1048576 128 16755 0 25715 0 25240 16125 Settings that i have apart default are: cat /boot/loader.conf zfs_load="YES" kern.geom.label.gptid.enable="0" nvme_load="YES" nvd_load="YES" vfs.zfs.vdev.trim_on_init=0 vfs.zfs.trim.enabled=0 kern.ipc.nmbjumbo16=262144 kern.ipc.nmbjumbo9=262144 kern.ipc.nmbclusters=262144 kern.ipc.nmbjumbop=262144 net.inet.tcp.maxtcptw=163840 hw.intr_storm_threshold="9000" vfs.zfs.cache_flush_disable=1 #avoid sending flushes to prevent useless delays in buggy low end ssds vfs.zfs.vdev.cache.bshift=13 net.inet.tcp.tcbhashsize=32768 vfs.zfs.arc_max=34359738368 /etc/sysctl.conf net.inet.tcp.fast_finwait2_recycle=1 net.inet.ip.portrange.randomized=0 net.inet.ip.portrange.first=1024 net.inet.ip.portrange.last=65535 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_inc=65536 vfs.zfs.vdev.trim_on_init=0 #close time_wait connections at 2*7500ms net.inet.tcp.msl=7500 kern.ipc.somaxconn=4096 net.inet.icmp.icmplim=2000 #zfs vfs.zfs.txg.timeout=5 vfs.zfs.prefetch_disable=1 What i've tried is setting hw.nvme.per_cpu_io_queues to 0, but doesn't seem to have any effect. Setting hw.nvme.force_intx=1 leads to system not booting at all. From what it seems issues is on nvme driver as perftest outputs: nvmecontrol perftest -n 32 -o read -s 512 -t30 nvme0ns1 Threads: 32 Size: 512 READ Time: 30 IO/s: 270212 MB/s: 131 nvmecontrol perftest -n 32 -o write -s 512 -t30 nvme0ns1 Threads: 32 Size: 512 WRITE Time: 30 IO/s: 13658 MB/s: 6 Any help to bring this device to proper speed will be welcomed. -- Best regards, Vintila Mihai Alexandru From owner-freebsd-stable@FreeBSD.ORG Thu Jan 15 17:54:24 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3AC2A9CA; Thu, 15 Jan 2015 17:54:24 +0000 (UTC) Received: from mail.egr.msu.edu (hill.egr.msu.edu [35.9.37.162]) by mx1.freebsd.org (Postfix) with ESMTP id 0CBBC7E7; Thu, 15 Jan 2015 17:54:20 +0000 (UTC) Received: from hill (localhost [127.0.0.1]) by mail.egr.msu.edu (Postfix) with ESMTP id 7EF253067E; Thu, 15 Jan 2015 12:54:19 -0500 (EST) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mail.egr.msu.edu ([127.0.0.1]) by hill (hill.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 84qXl0yu8Qvt; Thu, 15 Jan 2015 12:54:19 -0500 (EST) Received: from EGR authenticated sender Message-ID: <54B7FECB.2010002@egr.msu.edu> Date: Thu, 15 Jan 2015 12:54:19 -0500 From: Adam McDougall User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Patch to fix SSH and Kerberos entries in OptionalObsoleteFiles.inc Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Cc: op@freebsd.org, amdmi3@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2015 17:54:24 -0000 My local patch count for FreeBSD 10 builds are down to one; could someone help me bring it to zero by committing my small patch and merging it to 10 at least? https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193980 It adds some missing entries for OpenSSH and Heimdal Kerberos. This patch makes it easier for people to switch to OpenSSH or Kerberos from ports without leaving a broken and/or messy base system behind. I am willing to regenerate the patch if needed. Thank you. From owner-freebsd-stable@FreeBSD.ORG Thu Jan 15 17:59:31 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0A4B2DAC for ; Thu, 15 Jan 2015 17:59:31 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B9BEC843 for ; Thu, 15 Jan 2015 17:59:30 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YBohb-00056m-GY; Thu, 15 Jan 2015 20:59:27 +0300 Date: Thu, 15 Jan 2015 20:59:27 +0300 From: Slawa Olhovchenkov To: Mihai Vintila Subject: Re: Poor performance on Intel P3600 NVME driver Message-ID: <20150115175927.GA19071@zxy.spb.ru> References: <54B7F769.40605@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <54B7F769.40605@gmail.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2015 17:59:31 -0000 On Thu, Jan 15, 2015 at 07:22:49PM +0200, Mihai Vintila wrote: > /etc/sysctl.conf > > net.inet.tcp.fast_finwait2_recycle=1 > net.inet.ip.portrange.randomized=0 > net.inet.ip.portrange.first=1024 > net.inet.ip.portrange.last=65535 > net.inet.tcp.recvbuf_max=16777216 > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.recvbuf_inc=65536 > vfs.zfs.vdev.trim_on_init=0 > #close time_wait connections at 2*7500ms > net.inet.tcp.msl=7500 > kern.ipc.somaxconn=4096 > net.inet.icmp.icmplim=2000 > > #zfs > vfs.zfs.txg.timeout=5 > vfs.zfs.prefetch_disable=1 > Any help to bring this device to proper speed will be welcomed. Do you try to increase vfs.zfs.top_maxinflight? From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 02:55:30 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D7AF541C; Fri, 16 Jan 2015 02:55:30 +0000 (UTC) Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by mx1.freebsd.org (Postfix) with ESMTP id 42DE0875; Fri, 16 Jan 2015 02:55:30 +0000 (UTC) Received: from ppp14-2-33-47.lns21.adl2.internode.on.net (HELO midget.dons.net.au) ([14.2.33.47]) by ipmail04.adl6.internode.on.net with ESMTP; 16 Jan 2015 13:25:20 +1030 Received: from [10.0.2.26] ([10.0.2.26]) (authenticated bits=0) by midget.dons.net.au (8.15.1/8.14.9) with ESMTPSA id t0G2tDh0041155 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 16 Jan 2015 13:25:19 +1030 (CST) (envelope-from darius@dons.net.au) Subject: Re: DTrace and function names Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) Content-Type: text/plain; charset=windows-1252 From: "O'Connor, Daniel" In-Reply-To: <54B67A60.2000708@FreeBSD.org> Date: Fri, 16 Jan 2015 13:25:18 +1030 Content-Transfer-Encoding: quoted-printable Message-Id: References: <54B67A60.2000708@FreeBSD.org> To: John Baldwin X-Mailer: Apple Mail (2.1993) X-Spam-Score: -2.9 () ALL_TRUSTED,BAYES_00 X-Scanned-By: MIMEDefang 2.75 on 10.0.2.1 Cc: FreeBSD Stable Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 02:55:30 -0000 > On 15 Jan 2015, at 00:47, John Baldwin wrote: >> So it shows _some_ function names from libc but mostly not.. Is there = a way to improve it? >=20 > Build with debug symbols? For libc you can do that via: >=20 > % cd /usr/src/lib/libc > % make cleandir > % make obj > # May want to use "-O -g" to reduce inlining > % make DEBUG_FLAGS=3D"-g" depend all install OK, I guess I was thinking it wasn't necessary since some of the symbols = showed up :( > For gpsd you'll have to figure out a way to get it built with extra > symbols (if the port has a DEBUG option, enable that, otherwise you > might have to hack the port). Done that - it was the easy part ;) -- Daniel O'Connor "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 08:12:29 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 30EE83AD for ; Fri, 16 Jan 2015 08:12:29 +0000 (UTC) Received: from mail-we0-x233.google.com (mail-we0-x233.google.com [IPv6:2a00:1450:400c:c03::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B4E88B36 for ; Fri, 16 Jan 2015 08:12:28 +0000 (UTC) Received: by mail-we0-f179.google.com with SMTP id q59so18863701wes.10 for ; Fri, 16 Jan 2015 00:12:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=iPv3GyzDYdpsLva1j34S3SnakCZkPTbijboqWGuIt2g=; b=Jho+yjofEZrNHDbMmR008SRU2vm7n8vzYVVWjDC5oUW0dxY/97CcpGyzrualKxTy4m XbmJo08lQ399Y7bg3xKS1LdlDQxbXv27bvx39hqeB4UOjcX62GfAZgBfXPqtP6e+dhz9 2lqhsCXFOq11guLqJAXR9jfRcwfJvkJ7RNszvqGNMy2h45SsuVJSouU9Ll5hysTqZQgH zATQCckjaD6bm+bOZXr14DnedswTUreHUK/EcUKe+vSXrGgZWsJCkiPVucaBm+YkNVuC aZbzSAyMtT59oIjxGKCIvco8iCQ6Y/tm4Ge8la+Sva6uRi9549cd52JXzvF0FEIiXEUe werQ== X-Received: by 10.180.211.169 with SMTP id nd9mr3724440wic.4.1421395947080; Fri, 16 Jan 2015 00:12:27 -0800 (PST) Received: from [192.168.3.105] ([193.148.0.35]) by mx.google.com with ESMTPSA id e18sm5020840wjz.27.2015.01.16.00.12.25 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 16 Jan 2015 00:12:26 -0800 (PST) Message-ID: <54B8C7E9.3030602@gmail.com> Date: Fri, 16 Jan 2015 10:12:25 +0200 From: Mihai Vintila User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: Poor performance on Intel P3600 NVME driver References: <54B7F769.40605@gmail.com> <20150115175927.GA19071@zxy.spb.ru> In-Reply-To: <20150115175927.GA19071@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 08:12:29 -0000 I've tried to increase it and obtained around 90k iops for write, but now i only get : nvme0: WRITE sqid:6 cid:125 nsid:1 lba:3907028496 len:16 nvme0: WRITE FAULTS (02/80) sqid:6 cid:125 cdw0:0 nvme0: async event occurred (log page id=0x1) nvme0: WRITE sqid:6 cid:126 nsid:1 lba:528 len:16 nvme0: WRITE FAULTS (02/80) sqid:6 cid:126 cdw0:0 nvme0: async event occurred (log page id=0x1) nvme0: WRITE sqid:6 cid:127 nsid:1 lba:3907027984 len:16 nvme0: WRITE FAULTS (02/80) sqid:6 cid:127 cdw0:0 nvme0: async event occurred (log page id=0x1) Best regards, Vintila Mihai Alexandru On 1/15/2015 7:59 PM, Slawa Olhovchenkov wrote: > On Thu, Jan 15, 2015 at 07:22:49PM +0200, Mihai Vintila wrote: > >> /etc/sysctl.conf >> >> net.inet.tcp.fast_finwait2_recycle=1 >> net.inet.ip.portrange.randomized=0 >> net.inet.ip.portrange.first=1024 >> net.inet.ip.portrange.last=65535 >> net.inet.tcp.recvbuf_max=16777216 >> net.inet.tcp.sendbuf_max=16777216 >> net.inet.tcp.recvbuf_inc=65536 >> vfs.zfs.vdev.trim_on_init=0 >> #close time_wait connections at 2*7500ms >> net.inet.tcp.msl=7500 >> kern.ipc.somaxconn=4096 >> net.inet.icmp.icmplim=2000 >> >> #zfs >> vfs.zfs.txg.timeout=5 >> vfs.zfs.prefetch_disable=1 > >> Any help to bring this device to proper speed will be welcomed. > Do you try to increase vfs.zfs.top_maxinflight? From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 08:42:32 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7C2A09D4 for ; Fri, 16 Jan 2015 08:42:32 +0000 (UTC) Received: from mail-wi0-x22e.google.com (mail-wi0-x22e.google.com [IPv6:2a00:1450:400c:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0DBC3E00 for ; Fri, 16 Jan 2015 08:42:32 +0000 (UTC) Received: by mail-wi0-f174.google.com with SMTP id n3so655744wiv.1 for ; Fri, 16 Jan 2015 00:42:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=mA9L0vesN/gdkLqQ1EUVzvDnfu1u/cEbCysDUXsDWNc=; b=gzkCu+LlJeA2IMHeW4vuj0y8QVv+oYGoBP7K/fZWbcwP6Jxz0uRtAQ7eXqTwv4OEdy sMlRGMfM1b0g7V9oUu7NrZEByFjmM3uyAET9z9hqnliIE8X8wBaSBmYbPIY8cNXrgtrE yOkhty2ETjgwiAiVPpfjRlx27wxBN7VZZ3Blo3GWp03OgTwZvVvIvT2tYgzQReAxNiFs rdqapS21U3jJ1gCHnPgCR1dy3BNoPJlee/8mvyoIc2fnrd7wK53vShiU0FLcEvi8LfZk qinn9ThZfEX98GqEseF6aT9i8qQEEQLI/oO16W4y0HvGAa6/joD/zNucQEo1et+Hr4TB O0yA== X-Received: by 10.194.190.162 with SMTP id gr2mr17755770wjc.13.1421397750329; Fri, 16 Jan 2015 00:42:30 -0800 (PST) Received: from [192.168.3.105] ([193.148.0.35]) by mx.google.com with ESMTPSA id x16sm2039679wia.15.2015.01.16.00.42.29 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 16 Jan 2015 00:42:29 -0800 (PST) Message-ID: <54B8CEF4.9040206@gmail.com> Date: Fri, 16 Jan 2015 10:42:28 +0200 From: Mihai Vintila User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: Poor performance on Intel P3600 NVME driver References: <54B7F769.40605@gmail.com> <20150115175927.GA19071@zxy.spb.ru> <54B8C7E9.3030602@gmail.com> In-Reply-To: <54B8C7E9.3030602@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 08:42:32 -0000 Actually seems that i've was wrong vfs.zfs.top_maxinflight does not have any influence and this is confirmed by the nvmecontrol perftests. nvmecontrol perftest -n 32 -o read -s 512 -t30 nvme0ns1 Threads: 32 Size: 512 READ Time: 30 IO/s: 270212 MB/s: 131 nvmecontrol perftest -n 32 -o write -s 512 -t30 nvme0ns1 Threads: 32 Size: 512 WRITE Time: 30 IO/s: 13658 MB/s: 6 I was able to recover from the errors from previous message. They were cause by the fact that i've commented hw.nvme.per_cpu_io_queues=0 in loader.conf . After setting it back things got back to "slow normal" Performance is half of what it should be and issue seems to be the nvme driver. I'll try it on another OS to confirm it's not a hardware setting issue. Best regards, Vintila Mihai Alexandru On 1/16/2015 10:12 AM, Mihai Vintila wrote: > I've tried to increase it and obtained around 90k iops for write, but > now i only get : > > nvme0: WRITE sqid:6 cid:125 nsid:1 lba:3907028496 len:16 > nvme0: WRITE FAULTS (02/80) sqid:6 cid:125 cdw0:0 > nvme0: async event occurred (log page id=0x1) > nvme0: WRITE sqid:6 cid:126 nsid:1 lba:528 len:16 > nvme0: WRITE FAULTS (02/80) sqid:6 cid:126 cdw0:0 > nvme0: async event occurred (log page id=0x1) > nvme0: WRITE sqid:6 cid:127 nsid:1 lba:3907027984 len:16 > nvme0: WRITE FAULTS (02/80) sqid:6 cid:127 cdw0:0 > nvme0: async event occurred (log page id=0x1) > > Best regards, > Vintila Mihai Alexandru > > On 1/15/2015 7:59 PM, Slawa Olhovchenkov wrote: >> On Thu, Jan 15, 2015 at 07:22:49PM +0200, Mihai Vintila wrote: >> >>> /etc/sysctl.conf >>> >>> net.inet.tcp.fast_finwait2_recycle=1 >>> net.inet.ip.portrange.randomized=0 >>> net.inet.ip.portrange.first=1024 >>> net.inet.ip.portrange.last=65535 >>> net.inet.tcp.recvbuf_max=16777216 >>> net.inet.tcp.sendbuf_max=16777216 >>> net.inet.tcp.recvbuf_inc=65536 >>> vfs.zfs.vdev.trim_on_init=0 >>> #close time_wait connections at 2*7500ms >>> net.inet.tcp.msl=7500 >>> kern.ipc.somaxconn=4096 >>> net.inet.icmp.icmplim=2000 >>> >>> #zfs >>> vfs.zfs.txg.timeout=5 >>> vfs.zfs.prefetch_disable=1 >> >>> Any help to bring this device to proper speed will be welcomed. >> Do you try to increase vfs.zfs.top_maxinflight? > From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 09:13:23 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A7AB6430 for ; Fri, 16 Jan 2015 09:13:23 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 13D961AF for ; Fri, 16 Jan 2015 09:13:23 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YC2xx-000NAb-3C; Fri, 16 Jan 2015 12:13:17 +0300 Date: Fri, 16 Jan 2015 12:13:17 +0300 From: Slawa Olhovchenkov To: Mihai Vintila Subject: Re: Poor performance on Intel P3600 NVME driver Message-ID: <20150116091317.GA83755@zxy.spb.ru> References: <54B7F769.40605@gmail.com> <20150115175927.GA19071@zxy.spb.ru> <54B8C7E9.3030602@gmail.com> <54B8CEF4.9040206@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <54B8CEF4.9040206@gmail.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 09:13:23 -0000 On Fri, Jan 16, 2015 at 10:42:28AM +0200, Mihai Vintila wrote: > Actually seems that i've was wrong vfs.zfs.top_maxinflight does not have > any influence and this is confirmed by the nvmecontrol perftests. > nvmecontrol perftest -n 32 -o read -s 512 -t30 nvme0ns1 > Threads: 32 Size: 512 READ Time: 30 IO/s: 270212 MB/s: 131 > nvmecontrol perftest -n 32 -o write -s 512 -t30 nvme0ns1 > Threads: 32 Size: 512 WRITE Time: 30 IO/s: 13658 MB/s: 6 I see datasheet specified performance metrics for 4K (not 512bytes) random r/w. Can you try this? This is may be impact write IO/s at least. > I was able to recover from the errors from previous message. They were > cause by the fact that i've commented hw.nvme.per_cpu_io_queues=0 in > loader.conf . After setting it back things got back to "slow normal" > Performance is half of what it should be and issue seems to be the nvme > driver. I'll try it on another OS to confirm it's not a hardware setting > issue. From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 09:33:08 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C565D9F8 for ; Fri, 16 Jan 2015 09:33:08 +0000 (UTC) Received: from mail-wi0-x235.google.com (mail-wi0-x235.google.com [IPv6:2a00:1450:400c:c05::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 559355EE for ; Fri, 16 Jan 2015 09:33:08 +0000 (UTC) Received: by mail-wi0-f181.google.com with SMTP id hi2so2580165wib.2 for ; Fri, 16 Jan 2015 01:33:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=j9ee4bCW6G+6gMojTywdJnjHHsEme2muTzLKkly/1g0=; b=SIZxtILY2JDpz0/aLUdSklBXWwXaIijs/MVaW6iiDBQQs+dGLSz814aCEEDy2NHMvK flzO+3TbsK27QrJs0gHobYZnDWLNZyCfLzJTaXN3KkHYH9PJNBPCOsDon89VUkw3AHak piwr8lN537r3QrTgh2/T1DCNAoakT1x+XeoqtMLFteVGpIFNoIFTZ0bTGayPl6QTS6Xh qBjQ6twQGPFDedEysEVIPGgNEPmANsLhZkF6zcOQU71MLCNG1hOdg74o8mrzlVw5rNYK Tm417iSycFFpwdP0tr4PxkCWFI+w+gYom48g56mWGzi2VISQg6SRR6WFBCn4zoTT5tTo R33Q== X-Received: by 10.180.93.1 with SMTP id cq1mr4368150wib.2.1421400786025; Fri, 16 Jan 2015 01:33:06 -0800 (PST) Received: from [192.168.3.105] ([193.148.0.35]) by mx.google.com with ESMTPSA id e4sm5255741wjw.48.2015.01.16.01.33.05 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 16 Jan 2015 01:33:05 -0800 (PST) Message-ID: <54B8DAD0.5040606@gmail.com> Date: Fri, 16 Jan 2015 11:33:04 +0200 From: Mihai Vintila User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: Poor performance on Intel P3600 NVME driver References: <54B7F769.40605@gmail.com> <20150115175927.GA19071@zxy.spb.ru> <54B8C7E9.3030602@gmail.com> <54B8CEF4.9040206@gmail.com> <20150116091317.GA83755@zxy.spb.ru> In-Reply-To: <20150116091317.GA83755@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 09:33:08 -0000 I've tried it and are similar. Performance on this should be in the 450/70 read/write on 4k and i'm getting 120/40. Best regards, Vintila Mihai Alexandru On 1/16/2015 11:13 AM, Slawa Olhovchenkov wrote: > On Fri, Jan 16, 2015 at 10:42:28AM +0200, Mihai Vintila wrote: > >> Actually seems that i've was wrong vfs.zfs.top_maxinflight does not have >> any influence and this is confirmed by the nvmecontrol perftests. >> nvmecontrol perftest -n 32 -o read -s 512 -t30 nvme0ns1 >> Threads: 32 Size: 512 READ Time: 30 IO/s: 270212 MB/s: 131 >> nvmecontrol perftest -n 32 -o write -s 512 -t30 nvme0ns1 >> Threads: 32 Size: 512 WRITE Time: 30 IO/s: 13658 MB/s: 6 > I see datasheet specified performance metrics for 4K (not 512bytes) random r/w. > Can you try this? This is may be impact write IO/s at least. > >> I was able to recover from the errors from previous message. They were >> cause by the fact that i've commented hw.nvme.per_cpu_io_queues=0 in >> loader.conf . After setting it back things got back to "slow normal" >> Performance is half of what it should be and issue seems to be the nvme >> driver. I'll try it on another OS to confirm it's not a hardware setting >> issue. From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 15:37:15 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 88BC0BC6 for ; Fri, 16 Jan 2015 15:37:15 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 60DBF119 for ; Fri, 16 Jan 2015 15:37:15 +0000 (UTC) Received: from new-host-2.home (pool-173-70-85-31.nwrknj.fios.verizon.net [173.70.85.31]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id E9A59B97B; Fri, 16 Jan 2015 10:37:13 -0500 (EST) Message-ID: <54B9302A.7070002@FreeBSD.org> Date: Fri, 16 Jan 2015 10:37:14 -0500 From: John Baldwin User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: "O'Connor, Daniel" Subject: Re: DTrace and function names References: <54B67A60.2000708@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 16 Jan 2015 10:37:14 -0500 (EST) Cc: FreeBSD Stable Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 15:37:15 -0000 On 1/15/15 9:55 PM, O'Connor, Daniel wrote: > >> On 15 Jan 2015, at 00:47, John Baldwin wrote: >>> So it shows _some_ function names from libc but mostly not.. Is there a way to improve it? >> >> Build with debug symbols? For libc you can do that via: >> >> % cd /usr/src/lib/libc >> % make cleandir >> % make obj >> # May want to use "-O -g" to reduce inlining >> % make DEBUG_FLAGS="-g" depend all install > > OK, I guess I was thinking it wasn't necessary since some of the symbols showed up :( Yeah, libc will always include symbols for the public functions it exports, but without -g you won't have symbols for any internal functions (and that's generally true of any shared library AFAIK). -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 17:09:00 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6466D787 for ; Fri, 16 Jan 2015 17:09:00 +0000 (UTC) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E8AF6DEB for ; Fri, 16 Jan 2015 17:08:59 +0000 (UTC) Received: by mail-wi0-f177.google.com with SMTP id l15so5202642wiw.4 for ; Fri, 16 Jan 2015 09:08:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=uOMIfacjHTMzgTz8ecVE0sqjr4zGByEtKZyEIMyfX1U=; b=L3wk6ZMd56o044IuL5udJrXoaelbnVGm5cEchivASUrvZMQBuQLX5w+IpKk/ZLeCzC NR0MGcPp0UXy2rNZYHcKN/hTdB7xt+MCHhwJfancQayF0R6flf1SlVJ28QLptqnRx+wJ lregsBcW1gwo9689UC9i8o49NE/zwHkxxm3tLba/1HGj5yX6liGjushO1bN/yn48t+MY 75ymOS2qNG/6cg5CBGAsC4LXn9ZZ2UDJcsI5Q47m4MBVAyNWOGAPRGx3iShcOF40b75y dhxO7ETJNu8exmtAE5o+QtPdR4jrTx74ymS1uzUR6ucITOqeUUuVojFnDuOqT43AzJ9R grBg== X-Received: by 10.180.77.114 with SMTP id r18mr8183146wiw.8.1421428138004; Fri, 16 Jan 2015 09:08:58 -0800 (PST) Received: from [192.168.3.105] ([193.148.0.35]) by mx.google.com with ESMTPSA id u3sm3725945wiw.24.2015.01.16.09.08.56 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 16 Jan 2015 09:08:57 -0800 (PST) Message-ID: <54B945A8.10408@gmail.com> Date: Fri, 16 Jan 2015 19:08:56 +0200 From: Mihai Vintila User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: Poor performance on Intel P3600 NVME driver References: <54B7F769.40605@gmail.com> <20150115175927.GA19071@zxy.spb.ru> In-Reply-To: <20150115175927.GA19071@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 17:09:00 -0000 There are clearly issues with the controller driver on FreeBSD. Not only zfs related: I've did some testing under Centos 7 with ext4 Command line used: iozone -Rb /root/output_iops.wks -O -i 0 -i 1 -i 2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -r 512K -s 8G Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 8388608 4 156982 0 859809 0 656572 208156 8388608 8 82545 0 503549 0 435005 111282 8388608 32 21620 0 129740 0 135738 31361 8388608 64 11070 0 66027 0 71660 16175 8388608 128 5558 0 31800 0 35361 8066 8388608 512 1383 0 7520 0 8472 2068 Same test on BSD with ufs Command line used: iozone -Rb /root/output.wks -O -i 0 -i 1 -i 2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -s 1G Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 1048576 4 120642 0 368188 0 252302 41650 1048576 8 61871 0 253074 0 189851 35015 1048576 32 15925 0 89190 0 80004 19357 1048576 64 7958 0 47190 0 59120 10070 1048576 128 4024 0 23619 0 22435 5542 ZFS test on BSD: Command line used: iozone -Rb /root/output.wks -O -i 0 -i 1 -i 2 -e -+n -r4K -s 1G Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 1048576 4 69600 0 99750 0 92007 49556 Perftest while write is close to what it should be, read takes a 4x penalty nvmecontrol perftest -n 32 -o write -s 4096 -t30 nvme0ns1 Threads: 32 Size: 4096 WRITE Time: 30 IO/s: 211294 MB/s: 825 nvmecontrol perftest -n 32 -o read -s 4096 -t30 nvme0ns1 Threads: 32 Size: 4096 READ Time: 30 IO/s: 221365 MB/s: 864 What i've tried for zfs: vfs.zfs.vdev.sync_read_min_active: 32 vfs.zfs.vdev.sync_read_max_active: 32 vfs.zfs.vdev.sync_write_min_active: 32 vfs.zfs.vdev.sync_write_max_active: 32 vfs.zfs.vdev.async_read_min_active: 32 vfs.zfs.vdev.async_read_max_active: 32 vfs.zfs.vdev.async_write_min_active: 32 vfs.zfs.vdev.async_write_max_active: 32 No effect what so ever in tests. increasing vfs.zfs.top_maxinflight - > no effect While i've managed to make it a little bit more stable so that controller doesn't just disappear by disabling ASPM in bios i still get : nvme0: resetting controller and after this i have to reboot. Some other errors i had today which where fixed with ssd firmware update and by forcing PCIeX to 16x : 1511.530545] Buffer I/O error on device nvme0n1, logical block 0 [ 1511.530594] lost page write due to I/O error on nvme0n1 [ 1517.694014] Buffer I/O error on device nvme1n1, logical block 0 [ 1517.694064] lost page write due to I/O error on nvme1n1 Best regards, Vintila Mihai Alexandru On 1/15/2015 7:59 PM, Slawa Olhovchenkov wrote: > On Thu, Jan 15, 2015 at 07:22:49PM +0200, Mihai Vintila wrote: > >> /etc/sysctl.conf >> >> net.inet.tcp.fast_finwait2_recycle=1 >> net.inet.ip.portrange.randomized=0 >> net.inet.ip.portrange.first=1024 >> net.inet.ip.portrange.last=65535 >> net.inet.tcp.recvbuf_max=16777216 >> net.inet.tcp.sendbuf_max=16777216 >> net.inet.tcp.recvbuf_inc=65536 >> vfs.zfs.vdev.trim_on_init=0 >> #close time_wait connections at 2*7500ms >> net.inet.tcp.msl=7500 >> kern.ipc.somaxconn=4096 >> net.inet.icmp.icmplim=2000 >> >> #zfs >> vfs.zfs.txg.timeout=5 >> vfs.zfs.prefetch_disable=1 > >> Any help to bring this device to proper speed will be welcomed. > Do you try to increase vfs.zfs.top_maxinflight? From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 17:41:02 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 737C0442 for ; Fri, 16 Jan 2015 17:41:02 +0000 (UTC) Received: from pit.databus.com (Databus-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:80b::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2E3041FD for ; Fri, 16 Jan 2015 17:41:02 +0000 (UTC) Received: by pit.databus.com (Postfix, from userid 202) id A5FBE54BD; Fri, 16 Jan 2015 12:41:00 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=databus.com; s=20140217; t=1421430060; bh=JG6g0Wb9/jgXxjgXzIAgq+kkhshQLjjQM+kkp4aCnLk=; l=2549; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=RHvbzy1Q9WPNWsg7f+RtlZlY9APSUh+38BUsdUh4BiHNMFKv0fydF6slbHAwncUJ+ OOUSME78w2exNDxq//jSvZPdcPZ8fB1+z+oMXZrp4Gpkk2UmHcn4ZYmuswg7Y/XH/Y Wy6sv6wndZjZhUZxgjnzmrh6FMDvBmHEFKdSkrBbks3ZvZw91I0B4aUry5p6EE/Ofh Cylh0C1r/2qJY+WuECKOAhFTtVxyWr9XoYiTYYYRCIE0x7nkYwsMOsmjvJH/KyNxOt 5gm+mPSUtyadoS/SEgJ3MHB9oZ/vjj8bWiTPj+M/tLdpjJ51Xs9bymPo1nmprEMZSP tRcP75ZDERcLg== Date: Fri, 16 Jan 2015 12:41:00 -0500 From: Barney Wolff To: Mihai Vintila Subject: Re: Poor performance on Intel P3600 NVME driver Message-ID: <20150116174100.GA9487@pit.databus.com> References: <54B7F769.40605@gmail.com> <20150115175927.GA19071@zxy.spb.ru> <54B945A8.10408@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <54B945A8.10408@gmail.com> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 17:41:02 -0000 Apology if I've missed a mention, but are you sure there aren't zfs snapshots in existence, and things like access time updates differing between Linux and FreeBSD? On Fri, Jan 16, 2015 at 07:08:56PM +0200, Mihai Vintila wrote: > There are clearly issues with the controller driver on FreeBSD. Not only > zfs related: > I've did some testing under Centos 7 with ext4 > Command line used: iozone -Rb /root/output_iops.wks -O -i 0 -i > 1 -i 2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -r 512K -s 8G > Time Resolution = 0.000001 seconds. > Processor cache size set to 1024 kBytes. > Processor cache line size set to 32 bytes. > File stride size set to 17 * record size. > random random bkwd record stride > kB reclen write rewrite read reread read > write read rewrite read fwrite frewrite fread freread > 8388608 4 156982 0 859809 0 656572 208156 > 8388608 8 82545 0 503549 0 435005 111282 > 8388608 32 21620 0 129740 0 135738 31361 > 8388608 64 11070 0 66027 0 71660 16175 > 8388608 128 5558 0 31800 0 35361 8066 > 8388608 512 1383 0 7520 0 8472 2068 > > > Same test on BSD with ufs > Command line used: iozone -Rb /root/output.wks -O -i 0 -i 1 -i > 2 -e -+n -r4K -r 8K -r 32K -r 64K -r 128K -s 1G > Time Resolution = 0.000001 seconds. > Processor cache size set to 1024 Kbytes. > Processor cache line size set to 32 bytes. > File stride size set to 17 * record size. > random random bkwd record stride > KB reclen write rewrite read reread read > write read rewrite read fwrite frewrite fread freread > 1048576 4 120642 0 368188 0 252302 41650 > 1048576 8 61871 0 253074 0 189851 35015 > 1048576 32 15925 0 89190 0 80004 19357 > 1048576 64 7958 0 47190 0 59120 10070 > 1048576 128 4024 0 23619 0 22435 5542 > > > ZFS test on BSD: > Command line used: iozone -Rb /root/output.wks -O -i 0 -i 1 -i > 2 -e -+n -r4K -s 1G > Time Resolution = 0.000001 seconds. > Processor cache size set to 1024 Kbytes. > Processor cache line size set to 32 bytes. > File stride size set to 17 * record size. > random random bkwd record stride > KB reclen write rewrite read reread read > write read rewrite read fwrite frewrite fread freread > 1048576 4 69600 0 99750 0 92007 49556 > > > > Perftest while write is close to what it should be, read takes a 4x penalty > nvmecontrol perftest -n 32 -o write -s 4096 -t30 nvme0ns1 > Threads: 32 Size: 4096 WRITE Time: 30 IO/s: 211294 MB/s: 825 > nvmecontrol perftest -n 32 -o read -s 4096 -t30 nvme0ns1 > Threads: 32 Size: 4096 READ Time: 30 IO/s: 221365 MB/s: 864 From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 18:19:42 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 56ACD6EA for ; Fri, 16 Jan 2015 18:19:42 +0000 (UTC) Received: from mail-we0-x22a.google.com (mail-we0-x22a.google.com [IPv6:2a00:1450:400c:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DFE9598E for ; Fri, 16 Jan 2015 18:19:41 +0000 (UTC) Received: by mail-we0-f170.google.com with SMTP id w61so21706463wes.1 for ; Fri, 16 Jan 2015 10:19:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=AHUQ3/rhi1/VHomrfAon4Sl20Fg2KupwKoZlGAgjw3U=; b=z6oHPWE1b534CQFREKGRVziNHHxfFo/YbIjx01ujJmQtwlLg4yyRJTbwxJlvW21Vp6 RWiugubgfyXQP/p6kM9DmOUhepFg3Na+RR4aA4Ef94ghSD3tnJ6F/sl49TQh8gOzRug2 rBW3I0hN4cKGCew1+H31UaBl7ncK/Ws1YDsTdSSu5CmvZLTGckMSqwB3aprWCcNuDxjX TfeqsQiXAPGLUPU9uOtqpHO+CynUeuQ2y78ynVy/kLoU3Ag3bVuSU35T5rQIVLZhHslx Z3DJ2y8UYeoVZuBdimDm9A5kZiGjixt2a9GtJnogWC0ZqrQbtRVtIYWV+LgF7BnYclBx ru8w== MIME-Version: 1.0 X-Received: by 10.180.210.228 with SMTP id mx4mr8768744wic.57.1421432380261; Fri, 16 Jan 2015 10:19:40 -0800 (PST) Received: by 10.27.6.143 with HTTP; Fri, 16 Jan 2015 10:19:40 -0800 (PST) In-Reply-To: <54B8C7E9.3030602@gmail.com> References: <54B7F769.40605@gmail.com> <20150115175927.GA19071@zxy.spb.ru> <54B8C7E9.3030602@gmail.com> Date: Fri, 16 Jan 2015 10:19:40 -0800 Message-ID: Subject: Re: Poor performance on Intel P3600 NVME driver From: Chuck Tuffli To: Mihai Vintila Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 18:19:42 -0000 On Fri, Jan 16, 2015 at 12:12 AM, Mihai Vintila wrote: > I've tried to increase it and obtained around 90k iops for write, but now i > only get : > > nvme0: WRITE sqid:6 cid:125 nsid:1 lba:3907028496 len:16 > nvme0: WRITE FAULTS (02/80) sqid:6 cid:125 cdw0:0 > nvme0: async event occurred (log page id=0x1) > nvme0: WRITE sqid:6 cid:126 nsid:1 lba:528 len:16 > nvme0: WRITE FAULTS (02/80) sqid:6 cid:126 cdw0:0 > nvme0: async event occurred (log page id=0x1) > nvme0: WRITE sqid:6 cid:127 nsid:1 lba:3907027984 len:16 > nvme0: WRITE FAULTS (02/80) sqid:6 cid:127 cdw0:0 > nvme0: async event occurred (log page id=0x1) If it helps, 02/80 means media error / LBA out of range, and log page 0x1 contains command error information which will probably show that the starting LBA field for the write command was the issue. --chuck From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 19:12:45 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DD7AF49B for ; Fri, 16 Jan 2015 19:12:45 +0000 (UTC) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 507581AB for ; Fri, 16 Jan 2015 19:12:45 +0000 (UTC) Received: from mh0.gentlemail.de (mh0.gentlemail.de [78.138.80.135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id t0GJCgbO043691 for ; Fri, 16 Jan 2015 20:12:42 +0100 (CET) (envelope-from h.schmalzbauer@omnilan.de) Received: from titan.inop.mo1.omnilan.net (titan.inop.mo1.omnilan.net [IPv6:2001:a60:f0bb:1::3:1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 0AE67A93; Fri, 16 Jan 2015 20:12:41 +0100 (CET) Message-ID: <54B962A3.9010308@omnilan.de> Date: Fri, 16 Jan 2015 20:12:35 +0100 From: Harald Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: FreeBSD Stable Subject: lagg(8) causes ghost queue with igb(4) X-Enigmail-Version: 1.1.2 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigF3C3217F2F3F5BD778CE7AB2" X-Greylist: ACL 119 matched, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [78.138.80.130]); Fri, 16 Jan 2015 20:12:42 +0100 (CET) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: 78.138.80.135; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 19:12:46 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigF3C3217F2F3F5BD778CE7AB2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi all, while investigating a watchdog timeout problem (on FreeBSD-10.1-stable r276295) I noticed "ghost" queues consuming interrupts with igb(4)[82576] if igb(4) is member of lagg(4) (laggproto loadbalance lagghash l2). Mysteriously only sometimes (1 of 2 runs). Sending machine has only one ssh session open, where I do the following: 'dd if=3D/dev/zero | nc vegashare 3333' (of course on vegashare is a listener [nc -l vegashare 3333 > /dev/null]) I'm transfering 123.5*10^6 Bytes/sec and on the sender, systat reports: 2682 igb0:que 0 (1 of 4 queues causes moderate irq load hw.igb.aim=3D1, otherhise it were 16k irqs/s, nothing on the other 3 queues) But roughly every second run, I see a another queue consuming much more irqs/s while transferring exactly the same at exactly the same speed: 7740 igb0:que 0 2640 igb0:que 3 Here's again one queue with 2.6k irqs/s, but another one with 7.7k irqs/s which I can't understand what this is doing. It's useless for sure, because with only one queue I get the same payload transported on the same hardware with the same speed and 3 queues idle=E2=80=A6 I removed lagg(4) from the party and the 2nd queue hasn't showed up for at least 20 test runs. So I guess it's caused by lagg(4). Does anybody have an explanation for this? Thanks, -Harry --------------enigF3C3217F2F3F5BD778CE7AB2 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAlS5YqkACgkQLDqVQ9VXb8gP5gCbBo9rGkv1uutSUAP+DApsBSsU xNMAnRgpWnq8bax31ARa372k7B18CcjX =qUmT -----END PGP SIGNATURE----- --------------enigF3C3217F2F3F5BD778CE7AB2-- From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 20:21:10 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 760F99B3 for ; Fri, 16 Jan 2015 20:21:10 +0000 (UTC) Received: from mail-lb0-x231.google.com (mail-lb0-x231.google.com [IPv6:2a00:1450:4010:c04::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EB5A9B21 for ; Fri, 16 Jan 2015 20:21:09 +0000 (UTC) Received: by mail-lb0-f177.google.com with SMTP id b6so20234389lbj.8 for ; Fri, 16 Jan 2015 12:21:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=uctsunqbunNhA6PdJCOxV1MHpT6DBPsknJKqMVpeA+k=; b=X7tV1EQq06wNOoEidb1T06sjBhMQWMvVfkb1rKJ1lTs/LCAws5wIbT+4yMaOx7n+N5 aVTEvhnxQAsvB3mYBvQ7OFBHm+M5sbA6INXaXvH+4TMKC4uHNHs+QGPqV9QZhTUXj9yM Pb+w9B3tFwA2Yg9ilO8ktdTZykGQxOoIwYmoAKD6DOiy60n3pzMBhjMx1uGifehS/UzL yweqUy+qQxUMj/ga/DNSaw6rV0MAUq0wkIsqyuE4MfGx3HAmVUuh5pM1Ur0dgJHlaaFu NuWXyqMfOhyIWgcB7anzZucKE13f3Dl4UHVk7Jmv0Yzec+gPgNn7yABLB57aEOFNsHF9 j7Vw== MIME-Version: 1.0 X-Received: by 10.152.2.8 with SMTP id 8mr17062527laq.97.1421439667813; Fri, 16 Jan 2015 12:21:07 -0800 (PST) Received: by 10.112.155.229 with HTTP; Fri, 16 Jan 2015 12:21:07 -0800 (PST) In-Reply-To: References: <54B7F769.40605@gmail.com> <20150115175927.GA19071@zxy.spb.ru> <54B8C7E9.3030602@gmail.com> Date: Fri, 16 Jan 2015 22:21:07 +0200 Message-ID: Subject: Re: Poor performance on Intel P3600 NVME driver From: Mihai-Alexandru Vintila To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 20:21:10 -0000 @ Chuck Tuffli - error was caused by older firmware + the fact that the nvme controller didn't negotiate on x16 so i had to force the pcie to x16 from bios. With the update is fixed, i've just added it to be indexed since i couldn't find anything on google. @Barney Wolff it's a new pool with only changes recordsize=4k and compression=lz4 . On linux test is on ext4 with default values. Penalty is pretty high. Also there is a read penalty for read as well between ufs and zfs. Even on nvmecontrol perftest you can see the read penalty it's not normal to have same result for both write and read On Fri, Jan 16, 2015 at 8:19 PM, Chuck Tuffli wrote: > On Fri, Jan 16, 2015 at 12:12 AM, Mihai Vintila wrote: > > I've tried to increase it and obtained around 90k iops for write, but > now i > > only get : > > > > nvme0: WRITE sqid:6 cid:125 nsid:1 lba:3907028496 len:16 > > nvme0: WRITE FAULTS (02/80) sqid:6 cid:125 cdw0:0 > > nvme0: async event occurred (log page id=0x1) > > nvme0: WRITE sqid:6 cid:126 nsid:1 lba:528 len:16 > > nvme0: WRITE FAULTS (02/80) sqid:6 cid:126 cdw0:0 > > nvme0: async event occurred (log page id=0x1) > > nvme0: WRITE sqid:6 cid:127 nsid:1 lba:3907027984 len:16 > > nvme0: WRITE FAULTS (02/80) sqid:6 cid:127 cdw0:0 > > nvme0: async event occurred (log page id=0x1) > > If it helps, 02/80 means media error / LBA out of range, and log page > 0x1 contains command error information which will probably show that > the starting LBA field for the write command was the issue. > > --chuck > From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 22:51:19 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0F86E955; Fri, 16 Jan 2015 22:51:19 +0000 (UTC) Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by mx1.freebsd.org (Postfix) with ESMTP id 6D579BDB; Fri, 16 Jan 2015 22:51:18 +0000 (UTC) Received: from ppp14-2-33-47.lns21.adl2.internode.on.net (HELO midget.dons.net.au) ([14.2.33.47]) by ipmail04.adl6.internode.on.net with ESMTP; 17 Jan 2015 09:21:15 +1030 Received: from [10.0.2.26] ([10.0.2.26]) (authenticated bits=0) by midget.dons.net.au (8.15.1/8.14.9) with ESMTPSA id t0GMp9wC020738 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 17 Jan 2015 09:21:14 +1030 (CST) (envelope-from darius@dons.net.au) Subject: Re: DTrace and function names Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) Content-Type: text/plain; charset=windows-1252 From: "O'Connor, Daniel" In-Reply-To: <54B9302A.7070002@FreeBSD.org> Date: Sat, 17 Jan 2015 09:21:09 +1030 Content-Transfer-Encoding: quoted-printable Message-Id: References: <54B67A60.2000708@FreeBSD.org> <54B9302A.7070002@FreeBSD.org> To: John Baldwin X-Mailer: Apple Mail (2.1993) X-Spam-Score: -2.9 () ALL_TRUSTED,BAYES_00 X-Scanned-By: MIMEDefang 2.75 on 10.0.2.1 Cc: FreeBSD Stable Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 22:51:19 -0000 > On 17 Jan 2015, at 02:07, John Baldwin wrote: > On 1/15/15 9:55 PM, O'Connor, Daniel wrote: >>=20 >>> On 15 Jan 2015, at 00:47, John Baldwin wrote: >>>> So it shows _some_ function names from libc but mostly not.. Is = there a way to improve it? >>>=20 >>> Build with debug symbols? For libc you can do that via: >>>=20 >>> % cd /usr/src/lib/libc >>> % make cleandir >>> % make obj >>> # May want to use "-O -g" to reduce inlining >>> % make DEBUG_FLAGS=3D"-g" depend all install >>=20 >> OK, I guess I was thinking it wasn't necessary since some of the = symbols showed up :( >=20 > Yeah, libc will always include symbols for the public functions it > exports, but without -g you won't have symbols for any internal > functions (and that's generally true of any shared library AFAIK). Right, I guess it seems surprising so many internal functions show up.. I wonder if dtrace has an option to collapse the internal functions into = their public parents (too much to hope for I imagine :) -- Daniel O'Connor "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 23:07:55 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 59E95C9D for ; Fri, 16 Jan 2015 23:07:55 +0000 (UTC) Received: from mail-wg0-x236.google.com (mail-wg0-x236.google.com [IPv6:2a00:1450:400c:c00::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 37792CFB for ; Fri, 16 Jan 2015 23:07:54 +0000 (UTC) Received: by mail-wg0-f54.google.com with SMTP id z12so22971444wgg.13 for ; Fri, 16 Jan 2015 15:07:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type; bh=cTAbAXCpAhR6CcpwFjo/KhuV6Vtn29RoCbPvX114gCs=; b=WDfuSqnxGe5+GT2/5kUXSc8weAFvpCDt1ikIbDV1mCbocJwBVtLUHw0RHQzl51ZkL4 5q17btZaJ5FWUec8P82/KP6x/haFIWU5i2jq9xVjgNfdYTk72SMTRQFQFoipZcTmu81c bCNd/QfS9mYLApeB4K980XnYWsOBWIb5SCkMMW6Zji8C/GLhgbrQpTA830Il89jYkW8d ya2wZCQheOVGb8TMWGi4OHRYuMbxPYcTjPNGJ32uU1cRAFQlrSQu8Dj1ko71fVRazBJg +mj2Fnc4x46N5PCNzNrEM2VZnVG2Jn5S/bod/+Rqkn+/M4lfmi0o6ld/0lsvvdsiNYsp SAzg== X-Received: by 10.194.235.226 with SMTP id up2mr34195337wjc.9.1421449672659; Fri, 16 Jan 2015 15:07:52 -0800 (PST) Received: from [192.168.3.105] ([193.148.0.35]) by mx.google.com with ESMTPSA id x16sm4701335wia.15.2015.01.16.15.07.51 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 16 Jan 2015 15:07:51 -0800 (PST) Message-ID: <54B999C6.2090909@gmail.com> Date: Sat, 17 Jan 2015 01:07:50 +0200 From: Mihai Vintila User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: Poor performance on Intel P3600 NVME driver References: <54B7F769.40605@gmail.com> <20150115175927.GA19071@zxy.spb.ru> <54B8C7E9.3030602@gmail.com> <20150116221344.GA72201@pit.databus.com> In-Reply-To: <20150116221344.GA72201@pit.databus.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 23:07:55 -0000 I've remade the test with atime=off. Drive has 512b physical, but I've created it with 4k gnop anyway. Results are similar with atime Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bk wd record stride KB reclen write rewrite read reread read write re ad rewrite read fwrite frewrite fread freread 1048576 4 74427 0 101744 0 93529 47925 1048576 8 39072 0 64693 0 61104 25452 I've also tried to increase vfs.zfs.vdev.aggregation_limit and ended up with a crash (screenshot attached) I'm attaching zfs tunables: sysctl -a|grep vfs.zfs vfs.zfs.arc_max: 34359738368 vfs.zfs.arc_min: 4294967296 vfs.zfs.arc_average_blocksize: 8192 vfs.zfs.arc_meta_used: 5732232 vfs.zfs.arc_meta_limit: 8589934592 vfs.zfs.l2arc_write_max: 8388608 vfs.zfs.l2arc_write_boost: 8388608 vfs.zfs.l2arc_headroom: 2 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_feed_min_ms: 200 vfs.zfs.l2arc_noprefetch: 1 vfs.zfs.l2arc_feed_again: 1 vfs.zfs.l2arc_norw: 1 vfs.zfs.anon_size: 32768 vfs.zfs.anon_metadata_lsize: 0 vfs.zfs.anon_data_lsize: 0 vfs.zfs.mru_size: 17841664 vfs.zfs.mru_metadata_lsize: 858624 vfs.zfs.mru_data_lsize: 13968384 vfs.zfs.mru_ghost_size: 0 vfs.zfs.mru_ghost_metadata_lsize: 0 vfs.zfs.mru_ghost_data_lsize: 0 vfs.zfs.mfu_size: 4574208 vfs.zfs.mfu_metadata_lsize: 465408 vfs.zfs.mfu_data_lsize: 4051456 vfs.zfs.mfu_ghost_size: 0 vfs.zfs.mfu_ghost_metadata_lsize: 0 vfs.zfs.mfu_ghost_data_lsize: 0 vfs.zfs.l2c_only_size: 0 vfs.zfs.dedup.prefetch: 1 vfs.zfs.nopwrite_enabled: 1 vfs.zfs.mdcomp_disable: 0 vfs.zfs.dirty_data_max: 4294967296 vfs.zfs.dirty_data_max_max: 4294967296 vfs.zfs.dirty_data_max_percent: 10 vfs.zfs.dirty_data_sync: 67108864 vfs.zfs.delay_min_dirty_percent: 60 vfs.zfs.delay_scale: 500000 vfs.zfs.prefetch_disable: 1 vfs.zfs.zfetch.max_streams: 8 vfs.zfs.zfetch.min_sec_reap: 2 vfs.zfs.zfetch.block_cap: 256 vfs.zfs.zfetch.array_rd_sz: 1048576 vfs.zfs.top_maxinflight: 32 vfs.zfs.resilver_delay: 2 vfs.zfs.scrub_delay: 4 vfs.zfs.scan_idle: 50 vfs.zfs.scan_min_time_ms: 1000 vfs.zfs.free_min_time_ms: 1000 vfs.zfs.resilver_min_time_ms: 3000 vfs.zfs.no_scrub_io: 0 vfs.zfs.no_scrub_prefetch: 0 vfs.zfs.metaslab.gang_bang: 131073 vfs.zfs.metaslab.fragmentation_threshold: 70 vfs.zfs.metaslab.debug_load: 0 vfs.zfs.metaslab.debug_unload: 0 vfs.zfs.metaslab.df_alloc_threshold: 131072 vfs.zfs.metaslab.df_free_pct: 4 vfs.zfs.metaslab.min_alloc_size: 10485760 vfs.zfs.metaslab.load_pct: 50 vfs.zfs.metaslab.unload_delay: 8 vfs.zfs.metaslab.preload_limit: 3 vfs.zfs.metaslab.preload_enabled: 1 vfs.zfs.metaslab.fragmentation_factor_enabled: 1 vfs.zfs.metaslab.lba_weighting_enabled: 1 vfs.zfs.metaslab.bias_enabled: 1 vfs.zfs.condense_pct: 200 vfs.zfs.mg_noalloc_threshold: 0 vfs.zfs.mg_fragmentation_threshold: 85 vfs.zfs.check_hostid: 1 vfs.zfs.spa_load_verify_maxinflight: 10000 vfs.zfs.spa_load_verify_metadata: 1 vfs.zfs.spa_load_verify_data: 1 vfs.zfs.recover: 0 vfs.zfs.deadman_synctime_ms: 1000000 vfs.zfs.deadman_checktime_ms: 5000 vfs.zfs.deadman_enabled: 1 vfs.zfs.spa_asize_inflation: 24 vfs.zfs.txg.timeout: 5 vfs.zfs.vdev.cache.max: 16384 vfs.zfs.vdev.cache.size: 0 vfs.zfs.vdev.cache.bshift: 16 vfs.zfs.vdev.trim_on_init: 0 vfs.zfs.vdev.mirror.rotating_inc: 0 vfs.zfs.vdev.mirror.rotating_seek_inc: 5 vfs.zfs.vdev.mirror.rotating_seek_offset: 1048576 vfs.zfs.vdev.mirror.non_rotating_inc: 0 vfs.zfs.vdev.mirror.non_rotating_seek_inc: 1 vfs.zfs.vdev.max_active: 1000 vfs.zfs.vdev.sync_read_min_active: 32 vfs.zfs.vdev.sync_read_max_active: 32 vfs.zfs.vdev.sync_write_min_active: 32 vfs.zfs.vdev.sync_write_max_active: 32 vfs.zfs.vdev.async_read_min_active: 32 vfs.zfs.vdev.async_read_max_active: 32 vfs.zfs.vdev.async_write_min_active: 32 vfs.zfs.vdev.async_write_max_active: 32 vfs.zfs.vdev.scrub_min_active: 1 vfs.zfs.vdev.scrub_max_active: 2 vfs.zfs.vdev.trim_min_active: 1 vfs.zfs.vdev.trim_max_active: 64 vfs.zfs.vdev.aggregation_limit: 131072 vfs.zfs.vdev.read_gap_limit: 32768 vfs.zfs.vdev.write_gap_limit: 4096 vfs.zfs.vdev.bio_flush_disable: 0 vfs.zfs.vdev.bio_delete_disable: 0 vfs.zfs.vdev.trim_max_bytes: 2147483648 vfs.zfs.vdev.trim_max_pending: 64 vfs.zfs.max_auto_ashift: 13 vfs.zfs.min_auto_ashift: 9 vfs.zfs.zil_replay_disable: 0 vfs.zfs.cache_flush_disable: 0 vfs.zfs.zio.use_uma: 1 vfs.zfs.zio.exclude_metadata: 0 vfs.zfs.sync_pass_deferred_free: 2 vfs.zfs.sync_pass_dont_compress: 5 vfs.zfs.sync_pass_rewrite: 2 vfs.zfs.snapshot_list_prefetch: 0 vfs.zfs.super_owner: 0 vfs.zfs.debug: 0 vfs.zfs.version.ioctl: 4 vfs.zfs.version.acl: 1 vfs.zfs.version.spa: 5000 vfs.zfs.version.zpl: 5 vfs.zfs.vol.mode: 1 vfs.zfs.trim.enabled: 0 vfs.zfs.trim.txg_delay: 32 vfs.zfs.trim.timeout: 30 vfs.zfs.trim.max_interval: 1 And nvm: ev.nvme.%parent: dev.nvme.0.%desc: Generic NVMe Device dev.nvme.0.%driver: nvme dev.nvme.0.%location: slot=0 function=0 handle=\_SB_.PCI0.BR3A.D08A dev.nvme.0.%pnpinfo: vendor=0x8086 device=0x0953 subvendor=0x8086 subdevice=0x370a class=0x010802 dev.nvme.0.%parent: pci4 dev.nvme.0.int_coal_time: 0 dev.nvme.0.int_coal_threshold: 0 dev.nvme.0.timeout_period: 30 dev.nvme.0.num_cmds: 811857 dev.nvme.0.num_intr_handler_calls: 485242 dev.nvme.0.reset_stats: 0 dev.nvme.0.adminq.num_entries: 128 dev.nvme.0.adminq.num_trackers: 16 dev.nvme.0.adminq.sq_head: 12 dev.nvme.0.adminq.sq_tail: 12 dev.nvme.0.adminq.cq_head: 8 dev.nvme.0.adminq.num_cmds: 12 dev.nvme.0.adminq.num_intr_handler_calls: 7 dev.nvme.0.adminq.dump_debug: 0 dev.nvme.0.ioq0.num_entries: 256 dev.nvme.0.ioq0.num_trackers: 128 dev.nvme.0.ioq0.sq_head: 69 dev.nvme.0.ioq0.sq_tail: 69 dev.nvme.0.ioq0.cq_head: 69 dev.nvme.0.ioq0.num_cmds: 811845 dev.nvme.0.ioq0.num_intr_handler_calls: 485235 dev.nvme.0.ioq0.dump_debug: 0 dev.nvme.1.%desc: Generic NVMe Device dev.nvme.1.%driver: nvme dev.nvme.1.%location: slot=0 function=0 handle=\_SB_.PCI0.BR3B.H000 dev.nvme.1.%pnpinfo: vendor=0x8086 device=0x0953 subvendor=0x8086 subdevice=0x370a class=0x010802 dev.nvme.1.%parent: pci5 dev.nvme.1.int_coal_time: 0 dev.nvme.1.int_coal_threshold: 0 dev.nvme.1.timeout_period: 30 dev.nvme.1.num_cmds: 167 dev.nvme.1.num_intr_handler_calls: 163 dev.nvme.1.reset_stats: 0 dev.nvme.1.adminq.num_entries: 128 dev.nvme.1.adminq.num_trackers: 16 dev.nvme.1.adminq.sq_head: 12 dev.nvme.1.adminq.sq_tail: 12 dev.nvme.1.adminq.cq_head: 8 dev.nvme.1.adminq.num_cmds: 12 dev.nvme.1.adminq.num_intr_handler_calls: 8 dev.nvme.1.adminq.dump_debug: 0 dev.nvme.1.ioq0.num_entries: 256 dev.nvme.1.ioq0.num_trackers: 128 dev.nvme.1.ioq0.sq_head: 155 dev.nvme.1.ioq0.sq_tail: 155 dev.nvme.1.ioq0.cq_head: 155 dev.nvme.1.ioq0.num_cmds: 155 dev.nvme.1.ioq0.num_intr_handler_calls: 155 dev.nvme.1.ioq0.dump_debug: 0 Best regards, Vintila Mihai Alexandru On 1/17/2015 12:13 AM, Barney Wolff wrote: > I suspect Linux defaults to noatime - at least it does on my rpi. I > believe the FreeBSD default is the other way. That may explain some > of the difference. > > Also, did you use gnop to force the zpool to start on a 4k boundary? > If not, and the zpool happens to be offset, that's another big hit. > Same for ufs, especially if the disk has logical sectors of 512 but > physical of 4096. One can complain that FreeBSD should prevent, or > at least warn about, this sort of foot-shooting. > > On Fri, Jan 16, 2015 at 10:21:07PM +0200, Mihai-Alexandru Vintila wrote: >> @Barney Wolff it's a new pool with only changes recordsize=4k and >> compression=lz4 . On linux test is on ext4 with default values. Penalty is >> pretty high. Also there is a read penalty for read as well between ufs and >> zfs. Even on nvmecontrol perftest you can see the read penalty it's not >> normal to have same result for both write and read From owner-freebsd-stable@FreeBSD.ORG Fri Jan 16 23:24:32 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BCDA2F67 for ; Fri, 16 Jan 2015 23:24:32 +0000 (UTC) Received: from mail-wg0-f44.google.com (mail-wg0-f44.google.com [74.125.82.44]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CED0EA6 for ; Fri, 16 Jan 2015 23:24:31 +0000 (UTC) Received: by mail-wg0-f44.google.com with SMTP id y19so23028816wgg.3 for ; Fri, 16 Jan 2015 15:24:23 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=p3ePwt1JcBD+PJuDFFbqjJ7Z3N05McsexsArhvsm14k=; b=Mkmdw///4z4QUU0cN4zoJvf+6GZUv59CYesPMdM91/YsgC/bwpd8kZa4StQzGN1gFK EQf+hajV6YgfE0qGh6l3VnE9P6rBhOdtprJIfIHjL8rvykkLKiFGR6JFXiGYcIFh9R39 02Lba8jbVnoV8L/ynjl5hczKrvsV16o3nHubATNaklyArok1yB9POvMiuihhlDQwMC9N 8C5QXCUxWHxhOrIc0o0gYd7a6+6amOqIhy9KHhnO1VWen92x14yYuwoNHKCw2r5bbbde M199VjshULaqVH8xX/3lzBmdU23X6NqcDEIFkPTkCB6u5Rphzj62PW5T4ohpaeU8narV 8MdA== X-Gm-Message-State: ALoCoQlK9uUwl5Ob/bj9nBMA4QGRZ3DuDb0w2TOEdT/gDfjAxu1TExYwgz9XBUOfU2p9XdNgwcUc X-Received: by 10.181.8.193 with SMTP id dm1mr10574868wid.55.1421450663839; Fri, 16 Jan 2015 15:24:23 -0800 (PST) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id hg1sm7801264wjb.43.2015.01.16.15.24.22 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 16 Jan 2015 15:24:23 -0800 (PST) Message-ID: <54B99DA1.2000001@multiplay.co.uk> Date: Fri, 16 Jan 2015 23:24:17 +0000 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: Poor performance on Intel P3600 NVME driver References: <54B7F769.40605@gmail.com> <20150115175927.GA19071@zxy.spb.ru> <54B8C7E9.3030602@gmail.com> <20150116221344.GA72201@pit.databus.com> <54B999C6.2090909@gmail.com> In-Reply-To: <54B999C6.2090909@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 23:24:32 -0000 Any difference if you disable trim? On 16/01/2015 23:07, Mihai Vintila wrote: > I've remade the test with atime=off. Drive has 512b physical, but I've > created it with 4k gnop anyway. Results are similar with atime > Processor cache line size set to 32 bytes. > File stride size set to 17 * record size. > random random bk wd record stride > KB reclen write rewrite read reread read write > re ad rewrite read fwrite frewrite fread freread > 1048576 4 74427 0 101744 0 93529 47925 > 1048576 8 39072 0 64693 0 61104 25452 > > I've also tried to increase vfs.zfs.vdev.aggregation_limit and ended > up with a crash (screenshot attached) > > I'm attaching zfs tunables: > sysctl -a|grep vfs.zfs > vfs.zfs.arc_max: 34359738368 > vfs.zfs.arc_min: 4294967296 > vfs.zfs.arc_average_blocksize: 8192 > vfs.zfs.arc_meta_used: 5732232 > vfs.zfs.arc_meta_limit: 8589934592 > vfs.zfs.l2arc_write_max: 8388608 > vfs.zfs.l2arc_write_boost: 8388608 > vfs.zfs.l2arc_headroom: 2 > vfs.zfs.l2arc_feed_secs: 1 > vfs.zfs.l2arc_feed_min_ms: 200 > vfs.zfs.l2arc_noprefetch: 1 > vfs.zfs.l2arc_feed_again: 1 > vfs.zfs.l2arc_norw: 1 > vfs.zfs.anon_size: 32768 > vfs.zfs.anon_metadata_lsize: 0 > vfs.zfs.anon_data_lsize: 0 > vfs.zfs.mru_size: 17841664 > vfs.zfs.mru_metadata_lsize: 858624 > vfs.zfs.mru_data_lsize: 13968384 > vfs.zfs.mru_ghost_size: 0 > vfs.zfs.mru_ghost_metadata_lsize: 0 > vfs.zfs.mru_ghost_data_lsize: 0 > vfs.zfs.mfu_size: 4574208 > vfs.zfs.mfu_metadata_lsize: 465408 > vfs.zfs.mfu_data_lsize: 4051456 > vfs.zfs.mfu_ghost_size: 0 > vfs.zfs.mfu_ghost_metadata_lsize: 0 > vfs.zfs.mfu_ghost_data_lsize: 0 > vfs.zfs.l2c_only_size: 0 > vfs.zfs.dedup.prefetch: 1 > vfs.zfs.nopwrite_enabled: 1 > vfs.zfs.mdcomp_disable: 0 > vfs.zfs.dirty_data_max: 4294967296 > vfs.zfs.dirty_data_max_max: 4294967296 > vfs.zfs.dirty_data_max_percent: 10 > vfs.zfs.dirty_data_sync: 67108864 > vfs.zfs.delay_min_dirty_percent: 60 > vfs.zfs.delay_scale: 500000 > vfs.zfs.prefetch_disable: 1 > vfs.zfs.zfetch.max_streams: 8 > vfs.zfs.zfetch.min_sec_reap: 2 > vfs.zfs.zfetch.block_cap: 256 > vfs.zfs.zfetch.array_rd_sz: 1048576 > vfs.zfs.top_maxinflight: 32 > vfs.zfs.resilver_delay: 2 > vfs.zfs.scrub_delay: 4 > vfs.zfs.scan_idle: 50 > vfs.zfs.scan_min_time_ms: 1000 > vfs.zfs.free_min_time_ms: 1000 > vfs.zfs.resilver_min_time_ms: 3000 > vfs.zfs.no_scrub_io: 0 > vfs.zfs.no_scrub_prefetch: 0 > vfs.zfs.metaslab.gang_bang: 131073 > vfs.zfs.metaslab.fragmentation_threshold: 70 > vfs.zfs.metaslab.debug_load: 0 > vfs.zfs.metaslab.debug_unload: 0 > vfs.zfs.metaslab.df_alloc_threshold: 131072 > vfs.zfs.metaslab.df_free_pct: 4 > vfs.zfs.metaslab.min_alloc_size: 10485760 > vfs.zfs.metaslab.load_pct: 50 > vfs.zfs.metaslab.unload_delay: 8 > vfs.zfs.metaslab.preload_limit: 3 > vfs.zfs.metaslab.preload_enabled: 1 > vfs.zfs.metaslab.fragmentation_factor_enabled: 1 > vfs.zfs.metaslab.lba_weighting_enabled: 1 > vfs.zfs.metaslab.bias_enabled: 1 > vfs.zfs.condense_pct: 200 > vfs.zfs.mg_noalloc_threshold: 0 > vfs.zfs.mg_fragmentation_threshold: 85 > vfs.zfs.check_hostid: 1 > vfs.zfs.spa_load_verify_maxinflight: 10000 > vfs.zfs.spa_load_verify_metadata: 1 > vfs.zfs.spa_load_verify_data: 1 > vfs.zfs.recover: 0 > vfs.zfs.deadman_synctime_ms: 1000000 > vfs.zfs.deadman_checktime_ms: 5000 > vfs.zfs.deadman_enabled: 1 > vfs.zfs.spa_asize_inflation: 24 > vfs.zfs.txg.timeout: 5 > vfs.zfs.vdev.cache.max: 16384 > vfs.zfs.vdev.cache.size: 0 > vfs.zfs.vdev.cache.bshift: 16 > vfs.zfs.vdev.trim_on_init: 0 > vfs.zfs.vdev.mirror.rotating_inc: 0 > vfs.zfs.vdev.mirror.rotating_seek_inc: 5 > vfs.zfs.vdev.mirror.rotating_seek_offset: 1048576 > vfs.zfs.vdev.mirror.non_rotating_inc: 0 > vfs.zfs.vdev.mirror.non_rotating_seek_inc: 1 > vfs.zfs.vdev.max_active: 1000 > vfs.zfs.vdev.sync_read_min_active: 32 > vfs.zfs.vdev.sync_read_max_active: 32 > vfs.zfs.vdev.sync_write_min_active: 32 > vfs.zfs.vdev.sync_write_max_active: 32 > vfs.zfs.vdev.async_read_min_active: 32 > vfs.zfs.vdev.async_read_max_active: 32 > vfs.zfs.vdev.async_write_min_active: 32 > vfs.zfs.vdev.async_write_max_active: 32 > vfs.zfs.vdev.scrub_min_active: 1 > vfs.zfs.vdev.scrub_max_active: 2 > vfs.zfs.vdev.trim_min_active: 1 > vfs.zfs.vdev.trim_max_active: 64 > vfs.zfs.vdev.aggregation_limit: 131072 > vfs.zfs.vdev.read_gap_limit: 32768 > vfs.zfs.vdev.write_gap_limit: 4096 > vfs.zfs.vdev.bio_flush_disable: 0 > vfs.zfs.vdev.bio_delete_disable: 0 > vfs.zfs.vdev.trim_max_bytes: 2147483648 > vfs.zfs.vdev.trim_max_pending: 64 > vfs.zfs.max_auto_ashift: 13 > vfs.zfs.min_auto_ashift: 9 > vfs.zfs.zil_replay_disable: 0 > vfs.zfs.cache_flush_disable: 0 > vfs.zfs.zio.use_uma: 1 > vfs.zfs.zio.exclude_metadata: 0 > vfs.zfs.sync_pass_deferred_free: 2 > vfs.zfs.sync_pass_dont_compress: 5 > vfs.zfs.sync_pass_rewrite: 2 > vfs.zfs.snapshot_list_prefetch: 0 > vfs.zfs.super_owner: 0 > vfs.zfs.debug: 0 > vfs.zfs.version.ioctl: 4 > vfs.zfs.version.acl: 1 > vfs.zfs.version.spa: 5000 > vfs.zfs.version.zpl: 5 > vfs.zfs.vol.mode: 1 > vfs.zfs.trim.enabled: 0 > vfs.zfs.trim.txg_delay: 32 > vfs.zfs.trim.timeout: 30 > vfs.zfs.trim.max_interval: 1 > > And nvm: > ev.nvme.%parent: > dev.nvme.0.%desc: Generic NVMe Device > dev.nvme.0.%driver: nvme > dev.nvme.0.%location: slot=0 function=0 handle=\_SB_.PCI0.BR3A.D08A > dev.nvme.0.%pnpinfo: vendor=0x8086 device=0x0953 subvendor=0x8086 > subdevice=0x370a class=0x010802 > dev.nvme.0.%parent: pci4 > dev.nvme.0.int_coal_time: 0 > dev.nvme.0.int_coal_threshold: 0 > dev.nvme.0.timeout_period: 30 > dev.nvme.0.num_cmds: 811857 > dev.nvme.0.num_intr_handler_calls: 485242 > dev.nvme.0.reset_stats: 0 > dev.nvme.0.adminq.num_entries: 128 > dev.nvme.0.adminq.num_trackers: 16 > dev.nvme.0.adminq.sq_head: 12 > dev.nvme.0.adminq.sq_tail: 12 > dev.nvme.0.adminq.cq_head: 8 > dev.nvme.0.adminq.num_cmds: 12 > dev.nvme.0.adminq.num_intr_handler_calls: 7 > dev.nvme.0.adminq.dump_debug: 0 > dev.nvme.0.ioq0.num_entries: 256 > dev.nvme.0.ioq0.num_trackers: 128 > dev.nvme.0.ioq0.sq_head: 69 > dev.nvme.0.ioq0.sq_tail: 69 > dev.nvme.0.ioq0.cq_head: 69 > dev.nvme.0.ioq0.num_cmds: 811845 > dev.nvme.0.ioq0.num_intr_handler_calls: 485235 > dev.nvme.0.ioq0.dump_debug: 0 > dev.nvme.1.%desc: Generic NVMe Device > dev.nvme.1.%driver: nvme > dev.nvme.1.%location: slot=0 function=0 handle=\_SB_.PCI0.BR3B.H000 > dev.nvme.1.%pnpinfo: vendor=0x8086 device=0x0953 subvendor=0x8086 > subdevice=0x370a class=0x010802 > dev.nvme.1.%parent: pci5 > dev.nvme.1.int_coal_time: 0 > dev.nvme.1.int_coal_threshold: 0 > dev.nvme.1.timeout_period: 30 > dev.nvme.1.num_cmds: 167 > dev.nvme.1.num_intr_handler_calls: 163 > dev.nvme.1.reset_stats: 0 > dev.nvme.1.adminq.num_entries: 128 > dev.nvme.1.adminq.num_trackers: 16 > dev.nvme.1.adminq.sq_head: 12 > dev.nvme.1.adminq.sq_tail: 12 > dev.nvme.1.adminq.cq_head: 8 > dev.nvme.1.adminq.num_cmds: 12 > dev.nvme.1.adminq.num_intr_handler_calls: 8 > dev.nvme.1.adminq.dump_debug: 0 > dev.nvme.1.ioq0.num_entries: 256 > dev.nvme.1.ioq0.num_trackers: 128 > dev.nvme.1.ioq0.sq_head: 155 > dev.nvme.1.ioq0.sq_tail: 155 > dev.nvme.1.ioq0.cq_head: 155 > dev.nvme.1.ioq0.num_cmds: 155 > dev.nvme.1.ioq0.num_intr_handler_calls: 155 > dev.nvme.1.ioq0.dump_debug: 0 > > Best regards, > Vintila Mihai Alexandru > > On 1/17/2015 12:13 AM, Barney Wolff wrote: >> I suspect Linux defaults to noatime - at least it does on my rpi. I >> believe the FreeBSD default is the other way. That may explain some >> of the difference. >> >> Also, did you use gnop to force the zpool to start on a 4k boundary? >> If not, and the zpool happens to be offset, that's another big hit. >> Same for ufs, especially if the disk has logical sectors of 512 but >> physical of 4096. One can complain that FreeBSD should prevent, or >> at least warn about, this sort of foot-shooting. >> >> On Fri, Jan 16, 2015 at 10:21:07PM +0200, Mihai-Alexandru Vintila wrote: >>> @Barney Wolff it's a new pool with only changes recordsize=4k and >>> compression=lz4 . On linux test is on ext4 with default values. >>> Penalty is >>> pretty high. Also there is a read penalty for read as well between >>> ufs and >>> zfs. Even on nvmecontrol perftest you can see the read penalty it's not >>> normal to have same result for both write and read > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Sat Jan 17 00:43:15 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BA80BE9C for ; Sat, 17 Jan 2015 00:43:15 +0000 (UTC) Received: from orthanc.ca (orthanc.ca [IPv6:2607:f2f8:abf8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "ftp.orthanc.ca", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 86E4691C for ; Sat, 17 Jan 2015 00:43:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by orthanc.ca (8.14.7/8.14.7) with ESMTP id t0H0AsPw056846 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 16 Jan 2015 16:11:40 -0800 (PST) (envelope-from lyndon@orthanc.ca) Date: Fri, 16 Jan 2015 16:10:54 -0800 (PST) From: Lyndon Nerenberg To: freebsd-stable@freebsd.org Subject: SIIG 8-port USB RS-232 Message-ID: User-Agent: Alpine 2.11 (BSF 23 2013-08-11) Organization: The Frobozz Magic Homing Pigeon Company MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Jan 2015 00:43:15 -0000 Are any of you using this this beast (specifically, the JU-SC0211-S1) with FreeBSD 10.1, and in particular, running in a B+ Raspberry Pi? We are thinking about rolling this pairing out as a console server in a couple of remote datacentres, but I'm a bit nervous about the reliability of the USB port on the Pi. --lyndon From owner-freebsd-stable@FreeBSD.ORG Sat Jan 17 01:16:45 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BE9B16E9 for ; Sat, 17 Jan 2015 01:16:45 +0000 (UTC) Received: from mail.bitblocks.com (ns1.bitblocks.com [173.228.5.8]) by mx1.freebsd.org (Postfix) with ESMTP id A5B75C02 for ; Sat, 17 Jan 2015 01:16:45 +0000 (UTC) Received: from bitblocks.com (localhost [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id 8EEEEB827; Fri, 16 Jan 2015 17:07:15 -0800 (PST) To: Lyndon Nerenberg Subject: Re: SIIG 8-port USB RS-232 In-reply-to: Your message of "Fri, 16 Jan 2015 16:10:54 PST." References: Comments: In-reply-to Lyndon Nerenberg message dated "Fri, 16 Jan 2015 16:10:54 -0800." Date: Fri, 16 Jan 2015 17:07:15 -0800 From: Bakul Shah Message-Id: <20150117010715.8EEEEB827@mail.bitblocks.com> Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Jan 2015 01:16:45 -0000 On Fri, 16 Jan 2015 16:10:54 PST Lyndon Nerenberg wrote: > Are any of you using this this beast (specifically, the JU-SC0211-S1) with > FreeBSD 10.1, and in particular, running in a B+ Raspberry Pi? > > We are thinking about rolling this pairing out as a console server in a > couple of remote datacentres, but I'm a bit nervous about the reliability > of the USB port on the Pi. IIRC there were lots of issues with USB split-transactions and the RPi Foundations guys worked very hard at solving them. I don't know the end of that saga. You should check if this problem is solved well enough (and may be run Linux on the pi). You may be better off using BBB if you want freebsd. There is also the $37 odroid-c1, with separate 1G ethernet & USB + 1GB mem + quad core A5 but you'll have to run linux. From owner-freebsd-stable@FreeBSD.ORG Sat Jan 17 09:26:17 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 527BFE48 for ; Sat, 17 Jan 2015 09:26:17 +0000 (UTC) Received: from mail-we0-x229.google.com (mail-we0-x229.google.com [IPv6:2a00:1450:400c:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CD360EB1 for ; Sat, 17 Jan 2015 09:26:16 +0000 (UTC) Received: by mail-we0-f169.google.com with SMTP id p10so362834wes.0 for ; Sat, 17 Jan 2015 01:26:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:references:from:content-type:in-reply-to:message-id:date:to :content-transfer-encoding:mime-version; bh=SwQ2u2efYDm/1cgS+/lO+Fx6oFYQkk8eqyK7gcPkY+g=; b=G/7CDSn8rGVFweo9ACepzh6KJci5EiVnwGNk28CR7WQwH0qXgTt6LTWykKCAn76HfI N41vsjAkFA9wP+rZoT3tA9NEGTJ4Ihjgpia2gvf8iXAWOTsPAg3vfde0hmkzwNLCo51a JAqTyLBBDxNzBSVf0cf4N/ucwDkHyFtizqsT5MKsBtFAH/KNmmiRLK67rrZ2Y83Kst7F /sTiK6uwRlKWsUdXi8jBrxDR5jvcF4TOiB/30sOoWj+EKFHr49u4mALbxjN5QHtzlp9H pdjcBSWGAx2JvFRR4Tzw5NjIDfN0LIeVCwgA7JaIurru5YXPSckYJhcyQ0swUYUyEmrg 1x/g== X-Received: by 10.194.79.226 with SMTP id m2mr37197893wjx.60.1421486775236; Sat, 17 Jan 2015 01:26:15 -0800 (PST) Received: from [192.168.0.133] ([178.156.136.152]) by mx.google.com with ESMTPSA id i3sm9126416wjw.2.2015.01.17.01.26.13 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 17 Jan 2015 01:26:14 -0800 (PST) Subject: Re: Poor performance on Intel P3600 NVME driver References: <54B7F769.40605@gmail.com> <20150115175927.GA19071@zxy.spb.ru> <54B8C7E9.3030602@gmail.com> <20150116221344.GA72201@pit.databus.com> <54B999C6.2090909@gmail.com> <54B99DA1.2000001@multiplay.co.uk> From: Mihai-Alexandru Vintila Content-Type: text/plain; charset=us-ascii X-Mailer: iPhone Mail (12B440) In-Reply-To: <54B99DA1.2000001@multiplay.co.uk> Message-Id: Date: Sat, 17 Jan 2015 11:26:14 +0200 To: "freebsd-stable@freebsd.org" Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Jan 2015 09:26:17 -0000 Trim is already disabled as you can see in previous mail Best regards, Mihai Vintila > On 17 ian. 2015, at 01:24, Steven Hartland wrote= : >=20 > Any difference if you disable trim? >=20 >> On 16/01/2015 23:07, Mihai Vintila wrote: >> I've remade the test with atime=3Doff. Drive has 512b physical, but I've c= reated it with 4k gnop anyway. Results are similar with atime >> Processor cache line size set to 32 bytes. >> File stride size set to 17 * record size. >> random random bk wd record stride >> KB reclen write rewrite read reread read write re a= d rewrite read fwrite frewrite fread freread >> 1048576 4 74427 0 101744 0 93529 47925 >> 1048576 8 39072 0 64693 0 61104 25452 >>=20 >> I've also tried to increase vfs.zfs.vdev.aggregation_limit and ended up w= ith a crash (screenshot attached) >>=20 >> I'm attaching zfs tunables: >> sysctl -a|grep vfs.zfs >> vfs.zfs.arc_max: 34359738368 >> vfs.zfs.arc_min: 4294967296 >> vfs.zfs.arc_average_blocksize: 8192 >> vfs.zfs.arc_meta_used: 5732232 >> vfs.zfs.arc_meta_limit: 8589934592 >> vfs.zfs.l2arc_write_max: 8388608 >> vfs.zfs.l2arc_write_boost: 8388608 >> vfs.zfs.l2arc_headroom: 2 >> vfs.zfs.l2arc_feed_secs: 1 >> vfs.zfs.l2arc_feed_min_ms: 200 >> vfs.zfs.l2arc_noprefetch: 1 >> vfs.zfs.l2arc_feed_again: 1 >> vfs.zfs.l2arc_norw: 1 >> vfs.zfs.anon_size: 32768 >> vfs.zfs.anon_metadata_lsize: 0 >> vfs.zfs.anon_data_lsize: 0 >> vfs.zfs.mru_size: 17841664 >> vfs.zfs.mru_metadata_lsize: 858624 >> vfs.zfs.mru_data_lsize: 13968384 >> vfs.zfs.mru_ghost_size: 0 >> vfs.zfs.mru_ghost_metadata_lsize: 0 >> vfs.zfs.mru_ghost_data_lsize: 0 >> vfs.zfs.mfu_size: 4574208 >> vfs.zfs.mfu_metadata_lsize: 465408 >> vfs.zfs.mfu_data_lsize: 4051456 >> vfs.zfs.mfu_ghost_size: 0 >> vfs.zfs.mfu_ghost_metadata_lsize: 0 >> vfs.zfs.mfu_ghost_data_lsize: 0 >> vfs.zfs.l2c_only_size: 0 >> vfs.zfs.dedup.prefetch: 1 >> vfs.zfs.nopwrite_enabled: 1 >> vfs.zfs.mdcomp_disable: 0 >> vfs.zfs.dirty_data_max: 4294967296 >> vfs.zfs.dirty_data_max_max: 4294967296 >> vfs.zfs.dirty_data_max_percent: 10 >> vfs.zfs.dirty_data_sync: 67108864 >> vfs.zfs.delay_min_dirty_percent: 60 >> vfs.zfs.delay_scale: 500000 >> vfs.zfs.prefetch_disable: 1 >> vfs.zfs.zfetch.max_streams: 8 >> vfs.zfs.zfetch.min_sec_reap: 2 >> vfs.zfs.zfetch.block_cap: 256 >> vfs.zfs.zfetch.array_rd_sz: 1048576 >> vfs.zfs.top_maxinflight: 32 >> vfs.zfs.resilver_delay: 2 >> vfs.zfs.scrub_delay: 4 >> vfs.zfs.scan_idle: 50 >> vfs.zfs.scan_min_time_ms: 1000 >> vfs.zfs.free_min_time_ms: 1000 >> vfs.zfs.resilver_min_time_ms: 3000 >> vfs.zfs.no_scrub_io: 0 >> vfs.zfs.no_scrub_prefetch: 0 >> vfs.zfs.metaslab.gang_bang: 131073 >> vfs.zfs.metaslab.fragmentation_threshold: 70 >> vfs.zfs.metaslab.debug_load: 0 >> vfs.zfs.metaslab.debug_unload: 0 >> vfs.zfs.metaslab.df_alloc_threshold: 131072 >> vfs.zfs.metaslab.df_free_pct: 4 >> vfs.zfs.metaslab.min_alloc_size: 10485760 >> vfs.zfs.metaslab.load_pct: 50 >> vfs.zfs.metaslab.unload_delay: 8 >> vfs.zfs.metaslab.preload_limit: 3 >> vfs.zfs.metaslab.preload_enabled: 1 >> vfs.zfs.metaslab.fragmentation_factor_enabled: 1 >> vfs.zfs.metaslab.lba_weighting_enabled: 1 >> vfs.zfs.metaslab.bias_enabled: 1 >> vfs.zfs.condense_pct: 200 >> vfs.zfs.mg_noalloc_threshold: 0 >> vfs.zfs.mg_fragmentation_threshold: 85 >> vfs.zfs.check_hostid: 1 >> vfs.zfs.spa_load_verify_maxinflight: 10000 >> vfs.zfs.spa_load_verify_metadata: 1 >> vfs.zfs.spa_load_verify_data: 1 >> vfs.zfs.recover: 0 >> vfs.zfs.deadman_synctime_ms: 1000000 >> vfs.zfs.deadman_checktime_ms: 5000 >> vfs.zfs.deadman_enabled: 1 >> vfs.zfs.spa_asize_inflation: 24 >> vfs.zfs.txg.timeout: 5 >> vfs.zfs.vdev.cache.max: 16384 >> vfs.zfs.vdev.cache.size: 0 >> vfs.zfs.vdev.cache.bshift: 16 >> vfs.zfs.vdev.trim_on_init: 0 >> vfs.zfs.vdev.mirror.rotating_inc: 0 >> vfs.zfs.vdev.mirror.rotating_seek_inc: 5 >> vfs.zfs.vdev.mirror.rotating_seek_offset: 1048576 >> vfs.zfs.vdev.mirror.non_rotating_inc: 0 >> vfs.zfs.vdev.mirror.non_rotating_seek_inc: 1 >> vfs.zfs.vdev.max_active: 1000 >> vfs.zfs.vdev.sync_read_min_active: 32 >> vfs.zfs.vdev.sync_read_max_active: 32 >> vfs.zfs.vdev.sync_write_min_active: 32 >> vfs.zfs.vdev.sync_write_max_active: 32 >> vfs.zfs.vdev.async_read_min_active: 32 >> vfs.zfs.vdev.async_read_max_active: 32 >> vfs.zfs.vdev.async_write_min_active: 32 >> vfs.zfs.vdev.async_write_max_active: 32 >> vfs.zfs.vdev.scrub_min_active: 1 >> vfs.zfs.vdev.scrub_max_active: 2 >> vfs.zfs.vdev.trim_min_active: 1 >> vfs.zfs.vdev.trim_max_active: 64 >> vfs.zfs.vdev.aggregation_limit: 131072 >> vfs.zfs.vdev.read_gap_limit: 32768 >> vfs.zfs.vdev.write_gap_limit: 4096 >> vfs.zfs.vdev.bio_flush_disable: 0 >> vfs.zfs.vdev.bio_delete_disable: 0 >> vfs.zfs.vdev.trim_max_bytes: 2147483648 >> vfs.zfs.vdev.trim_max_pending: 64 >> vfs.zfs.max_auto_ashift: 13 >> vfs.zfs.min_auto_ashift: 9 >> vfs.zfs.zil_replay_disable: 0 >> vfs.zfs.cache_flush_disable: 0 >> vfs.zfs.zio.use_uma: 1 >> vfs.zfs.zio.exclude_metadata: 0 >> vfs.zfs.sync_pass_deferred_free: 2 >> vfs.zfs.sync_pass_dont_compress: 5 >> vfs.zfs.sync_pass_rewrite: 2 >> vfs.zfs.snapshot_list_prefetch: 0 >> vfs.zfs.super_owner: 0 >> vfs.zfs.debug: 0 >> vfs.zfs.version.ioctl: 4 >> vfs.zfs.version.acl: 1 >> vfs.zfs.version.spa: 5000 >> vfs.zfs.version.zpl: 5 >> vfs.zfs.vol.mode: 1 >> vfs.zfs.trim.enabled: 0 >> vfs.zfs.trim.txg_delay: 32 >> vfs.zfs.trim.timeout: 30 >> vfs.zfs.trim.max_interval: 1 >>=20 >> And nvm: >> ev.nvme.%parent: >> dev.nvme.0.%desc: Generic NVMe Device >> dev.nvme.0.%driver: nvme >> dev.nvme.0.%location: slot=3D0 function=3D0 handle=3D\_SB_.PCI0.BR3A.D08A= >> dev.nvme.0.%pnpinfo: vendor=3D0x8086 device=3D0x0953 subvendor=3D0x8086 s= ubdevice=3D0x370a class=3D0x010802 >> dev.nvme.0.%parent: pci4 >> dev.nvme.0.int_coal_time: 0 >> dev.nvme.0.int_coal_threshold: 0 >> dev.nvme.0.timeout_period: 30 >> dev.nvme.0.num_cmds: 811857 >> dev.nvme.0.num_intr_handler_calls: 485242 >> dev.nvme.0.reset_stats: 0 >> dev.nvme.0.adminq.num_entries: 128 >> dev.nvme.0.adminq.num_trackers: 16 >> dev.nvme.0.adminq.sq_head: 12 >> dev.nvme.0.adminq.sq_tail: 12 >> dev.nvme.0.adminq.cq_head: 8 >> dev.nvme.0.adminq.num_cmds: 12 >> dev.nvme.0.adminq.num_intr_handler_calls: 7 >> dev.nvme.0.adminq.dump_debug: 0 >> dev.nvme.0.ioq0.num_entries: 256 >> dev.nvme.0.ioq0.num_trackers: 128 >> dev.nvme.0.ioq0.sq_head: 69 >> dev.nvme.0.ioq0.sq_tail: 69 >> dev.nvme.0.ioq0.cq_head: 69 >> dev.nvme.0.ioq0.num_cmds: 811845 >> dev.nvme.0.ioq0.num_intr_handler_calls: 485235 >> dev.nvme.0.ioq0.dump_debug: 0 >> dev.nvme.1.%desc: Generic NVMe Device >> dev.nvme.1.%driver: nvme >> dev.nvme.1.%location: slot=3D0 function=3D0 handle=3D\_SB_.PCI0.BR3B.H000= >> dev.nvme.1.%pnpinfo: vendor=3D0x8086 device=3D0x0953 subvendor=3D0x8086 s= ubdevice=3D0x370a class=3D0x010802 >> dev.nvme.1.%parent: pci5 >> dev.nvme.1.int_coal_time: 0 >> dev.nvme.1.int_coal_threshold: 0 >> dev.nvme.1.timeout_period: 30 >> dev.nvme.1.num_cmds: 167 >> dev.nvme.1.num_intr_handler_calls: 163 >> dev.nvme.1.reset_stats: 0 >> dev.nvme.1.adminq.num_entries: 128 >> dev.nvme.1.adminq.num_trackers: 16 >> dev.nvme.1.adminq.sq_head: 12 >> dev.nvme.1.adminq.sq_tail: 12 >> dev.nvme.1.adminq.cq_head: 8 >> dev.nvme.1.adminq.num_cmds: 12 >> dev.nvme.1.adminq.num_intr_handler_calls: 8 >> dev.nvme.1.adminq.dump_debug: 0 >> dev.nvme.1.ioq0.num_entries: 256 >> dev.nvme.1.ioq0.num_trackers: 128 >> dev.nvme.1.ioq0.sq_head: 155 >> dev.nvme.1.ioq0.sq_tail: 155 >> dev.nvme.1.ioq0.cq_head: 155 >> dev.nvme.1.ioq0.num_cmds: 155 >> dev.nvme.1.ioq0.num_intr_handler_calls: 155 >> dev.nvme.1.ioq0.dump_debug: 0 >>=20 >> Best regards, >> Vintila Mihai Alexandru >>=20 >>> On 1/17/2015 12:13 AM, Barney Wolff wrote: >>> I suspect Linux defaults to noatime - at least it does on my rpi. I >>> believe the FreeBSD default is the other way. That may explain some >>> of the difference. >>>=20 >>> Also, did you use gnop to force the zpool to start on a 4k boundary? >>> If not, and the zpool happens to be offset, that's another big hit. >>> Same for ufs, especially if the disk has logical sectors of 512 but >>> physical of 4096. One can complain that FreeBSD should prevent, or >>> at least warn about, this sort of foot-shooting. >>>=20 >>>> On Fri, Jan 16, 2015 at 10:21:07PM +0200, Mihai-Alexandru Vintila wrote= : >>>> @Barney Wolff it's a new pool with only changes recordsize=3D4k and >>>> compression=3Dlz4 . On linux test is on ext4 with default values. Penal= ty is >>>> pretty high. Also there is a read penalty for read as well between ufs a= nd >>>> zfs. Even on nvmecontrol perftest you can see the read penalty it's not= >>>> normal to have same result for both write and read >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"= >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Sat Jan 17 11:42:45 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 558CA164 for ; Sat, 17 Jan 2015 11:42:45 +0000 (UTC) Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "tensor.andric.com", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0E9F8D06 for ; Sat, 17 Jan 2015 11:42:44 +0000 (UTC) Received: from [IPv6:2001:7b8:3a7::ac38:cb7f:ad0f:4f2a] (unknown [IPv6:2001:7b8:3a7:0:ac38:cb7f:ad0f:4f2a]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 96E55B80A; Sat, 17 Jan 2015 12:42:35 +0100 (CET) Subject: Re: Upgrading from stable/8 to stable/9 blocked by file 5.21 (r276416) Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) Content-Type: multipart/signed; boundary="Apple-Mail=_67793A0B-F0C5-4137-94D6-794534420958"; protocol="application/pgp-signature"; micalg=pgp-sha1 X-Pgp-Agent: GPGMail 2.5b4 From: Dimitry Andric In-Reply-To: <6E1192E7-DC34-452E-891C-A53DC92C3B0A@FreeBSD.org> Date: Sat, 17 Jan 2015 12:42:29 +0100 Message-Id: <86C905BB-6313-4DC3-A673-34E8EEEC077D@FreeBSD.org> References: <6E1192E7-DC34-452E-891C-A53DC92C3B0A@FreeBSD.org> To: =?utf-8?Q?Trond_Endrest=C3=B8l?= X-Mailer: Apple Mail (2.1993) Cc: FreeBSD stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Jan 2015 11:42:45 -0000 --Apple-Mail=_67793A0B-F0C5-4137-94D6-794534420958 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 On 13 Jan 2015, at 19:26, Dimitry Andric wrote: >=20 > On 04 Jan 2015, at 18:41, Trond Endrest=F8l = wrote: >>=20 >> I'm investigating how to convert my stable/8 systems to stable/9, and >> subsequently to stable/10. > ... >> =3D=3D=3D> lib/libmagic (obj,build-tools) >> cc -O2 -pipe -DMAGIC=3D'"/usr/share/misc/magic"' -DHAVE_CONFIG_H = -I/usr/src/lib/libmagic -I/usr/src/lib/libmagic/../../contrib/file/src = -std=3Dgnu99 -I/usr/obj/usr/src/tmp/legacy/usr/include -DCOMPILE_ONLY = -L/usr/obj/usr/src/tmp/legacy/usr/lib -o mkmagic = /usr/src/lib/libmagic/../../contrib/file/src/apprentice.c = /usr/src/lib/libmagic/../../contrib/file/src/cdf_time.c = /usr/src/lib/libmagic/../../contrib/file/src/encoding.c = /usr/src/lib/libmagic/../../contrib/file/src/funcs.c = /usr/src/lib/libmagic/../../contrib/file/src/magic.c = /usr/src/lib/libmagic/../../contrib/file/src/print.c -lz -legacy >> In file included from = /usr/src/lib/libmagic/../../contrib/file/src/apprentice.c:32: >> /usr/src/lib/libmagic/../../contrib/file/src/file.h:488:21: error: = xlocale.h: No such file or directory >=20 > FYI, I have submitted a fix for this issue here, to be reviewed: >=20 > https://reviews.freebsd.org/D1518 The fix has been merged to stable/10 and stable/9 in r277296. -Dimitry --Apple-Mail=_67793A0B-F0C5-4137-94D6-794534420958 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.26 iEYEARECAAYFAlS6SqoACgkQsF6jCi4glqP1vACeNbqHQBSYv/+WhWRPbGu2DK8s 12IAoLywMSZZeSeaOJYEZqWNwQDOO3yA =YZX2 -----END PGP SIGNATURE----- --Apple-Mail=_67793A0B-F0C5-4137-94D6-794534420958-- From owner-freebsd-stable@FreeBSD.ORG Sat Jan 17 13:37:47 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BD5C39A9 for ; Sat, 17 Jan 2015 13:37:47 +0000 (UTC) Received: from mail-yk0-f169.google.com (mail-yk0-f169.google.com [209.85.160.169]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7C7AA9D2 for ; Sat, 17 Jan 2015 13:37:47 +0000 (UTC) Received: by mail-yk0-f169.google.com with SMTP id 79so11560433ykr.0 for ; Sat, 17 Jan 2015 05:37:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=3V78gmUVvywHMpEj7sNOIse3v1lIlUTqqrAsCfJgfX8=; b=IMl6ZL5qtbsN3OGeYdOtZSZq0uDPgnqxI7di1T6P0yZ9UwpM9u2H4FB7k8sPP6Pkdk pSTGV9UpqEGULsrk4Ep1SUzwsWGMzbhEzQbMjLvqztQLaVhyG0ub9YEMNjZcpVEUdBBx ciaYf+eS2SGJ6TMLdEVtAvPDNE+2HkAYunXzU1c2ydb65g7IsS6rLsbCQaqTJawJuwoP U/FgftAIOIV962rmAL4N+8C3f4wSc2JDOWCfq3qyEAdDD3RlC2ksJnocWiDj4iPL5kfI Bwq1CrJXcfvgU5UoR6hO3KiFPQJTz3eLeEDxbbP2zkhl0UYnIX/ELV6dyW8NTwa3mII9 HRfQ== X-Gm-Message-State: ALoCoQnCFBRAGivcLLS+IGhDmfvOVfiv2FcUuY7rQ9XHIIq63uA3PQEfufn2reH84kBktVbp3IAK MIME-Version: 1.0 X-Received: by 10.236.0.195 with SMTP id 43mr2079733yhb.146.1421501396497; Sat, 17 Jan 2015 05:29:56 -0800 (PST) Received: by 10.170.46.213 with HTTP; Sat, 17 Jan 2015 05:29:56 -0800 (PST) In-Reply-To: References: <54B7F769.40605@gmail.com> <20150115175927.GA19071@zxy.spb.ru> <54B8C7E9.3030602@gmail.com> <20150116221344.GA72201@pit.databus.com> <54B999C6.2090909@gmail.com> <54B99DA1.2000001@multiplay.co.uk> Date: Sat, 17 Jan 2015 14:29:56 +0100 Message-ID: Subject: Re: Poor performance on Intel P3600 NVME driver From: Oliver Pinter To: Mihai-Alexandru Vintila , jimharris Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Jan 2015 13:37:47 -0000 Added Jim to thread, as he is the nvme driver's author. On Sat, Jan 17, 2015 at 10:26 AM, Mihai-Alexandru Vintila wrote: > Trim is already disabled as you can see in previous mail > > Best regards, > Mihai Vintila > >> On 17 ian. 2015, at 01:24, Steven Hartland wrote: >> >> Any difference if you disable trim? >> >>> On 16/01/2015 23:07, Mihai Vintila wrote: >>> I've remade the test with atime=off. Drive has 512b physical, but I've created it with 4k gnop anyway. Results are similar with atime >>> Processor cache line size set to 32 bytes. >>> File stride size set to 17 * record size. >>> random random bk wd record stride >>> KB reclen write rewrite read reread read write re ad rewrite read fwrite frewrite fread freread >>> 1048576 4 74427 0 101744 0 93529 47925 >>> 1048576 8 39072 0 64693 0 61104 25452 >>> >>> I've also tried to increase vfs.zfs.vdev.aggregation_limit and ended up with a crash (screenshot attached) >>> >>> I'm attaching zfs tunables: >>> sysctl -a|grep vfs.zfs >>> vfs.zfs.arc_max: 34359738368 >>> vfs.zfs.arc_min: 4294967296 >>> vfs.zfs.arc_average_blocksize: 8192 >>> vfs.zfs.arc_meta_used: 5732232 >>> vfs.zfs.arc_meta_limit: 8589934592 >>> vfs.zfs.l2arc_write_max: 8388608 >>> vfs.zfs.l2arc_write_boost: 8388608 >>> vfs.zfs.l2arc_headroom: 2 >>> vfs.zfs.l2arc_feed_secs: 1 >>> vfs.zfs.l2arc_feed_min_ms: 200 >>> vfs.zfs.l2arc_noprefetch: 1 >>> vfs.zfs.l2arc_feed_again: 1 >>> vfs.zfs.l2arc_norw: 1 >>> vfs.zfs.anon_size: 32768 >>> vfs.zfs.anon_metadata_lsize: 0 >>> vfs.zfs.anon_data_lsize: 0 >>> vfs.zfs.mru_size: 17841664 >>> vfs.zfs.mru_metadata_lsize: 858624 >>> vfs.zfs.mru_data_lsize: 13968384 >>> vfs.zfs.mru_ghost_size: 0 >>> vfs.zfs.mru_ghost_metadata_lsize: 0 >>> vfs.zfs.mru_ghost_data_lsize: 0 >>> vfs.zfs.mfu_size: 4574208 >>> vfs.zfs.mfu_metadata_lsize: 465408 >>> vfs.zfs.mfu_data_lsize: 4051456 >>> vfs.zfs.mfu_ghost_size: 0 >>> vfs.zfs.mfu_ghost_metadata_lsize: 0 >>> vfs.zfs.mfu_ghost_data_lsize: 0 >>> vfs.zfs.l2c_only_size: 0 >>> vfs.zfs.dedup.prefetch: 1 >>> vfs.zfs.nopwrite_enabled: 1 >>> vfs.zfs.mdcomp_disable: 0 >>> vfs.zfs.dirty_data_max: 4294967296 >>> vfs.zfs.dirty_data_max_max: 4294967296 >>> vfs.zfs.dirty_data_max_percent: 10 >>> vfs.zfs.dirty_data_sync: 67108864 >>> vfs.zfs.delay_min_dirty_percent: 60 >>> vfs.zfs.delay_scale: 500000 >>> vfs.zfs.prefetch_disable: 1 >>> vfs.zfs.zfetch.max_streams: 8 >>> vfs.zfs.zfetch.min_sec_reap: 2 >>> vfs.zfs.zfetch.block_cap: 256 >>> vfs.zfs.zfetch.array_rd_sz: 1048576 >>> vfs.zfs.top_maxinflight: 32 >>> vfs.zfs.resilver_delay: 2 >>> vfs.zfs.scrub_delay: 4 >>> vfs.zfs.scan_idle: 50 >>> vfs.zfs.scan_min_time_ms: 1000 >>> vfs.zfs.free_min_time_ms: 1000 >>> vfs.zfs.resilver_min_time_ms: 3000 >>> vfs.zfs.no_scrub_io: 0 >>> vfs.zfs.no_scrub_prefetch: 0 >>> vfs.zfs.metaslab.gang_bang: 131073 >>> vfs.zfs.metaslab.fragmentation_threshold: 70 >>> vfs.zfs.metaslab.debug_load: 0 >>> vfs.zfs.metaslab.debug_unload: 0 >>> vfs.zfs.metaslab.df_alloc_threshold: 131072 >>> vfs.zfs.metaslab.df_free_pct: 4 >>> vfs.zfs.metaslab.min_alloc_size: 10485760 >>> vfs.zfs.metaslab.load_pct: 50 >>> vfs.zfs.metaslab.unload_delay: 8 >>> vfs.zfs.metaslab.preload_limit: 3 >>> vfs.zfs.metaslab.preload_enabled: 1 >>> vfs.zfs.metaslab.fragmentation_factor_enabled: 1 >>> vfs.zfs.metaslab.lba_weighting_enabled: 1 >>> vfs.zfs.metaslab.bias_enabled: 1 >>> vfs.zfs.condense_pct: 200 >>> vfs.zfs.mg_noalloc_threshold: 0 >>> vfs.zfs.mg_fragmentation_threshold: 85 >>> vfs.zfs.check_hostid: 1 >>> vfs.zfs.spa_load_verify_maxinflight: 10000 >>> vfs.zfs.spa_load_verify_metadata: 1 >>> vfs.zfs.spa_load_verify_data: 1 >>> vfs.zfs.recover: 0 >>> vfs.zfs.deadman_synctime_ms: 1000000 >>> vfs.zfs.deadman_checktime_ms: 5000 >>> vfs.zfs.deadman_enabled: 1 >>> vfs.zfs.spa_asize_inflation: 24 >>> vfs.zfs.txg.timeout: 5 >>> vfs.zfs.vdev.cache.max: 16384 >>> vfs.zfs.vdev.cache.size: 0 >>> vfs.zfs.vdev.cache.bshift: 16 >>> vfs.zfs.vdev.trim_on_init: 0 >>> vfs.zfs.vdev.mirror.rotating_inc: 0 >>> vfs.zfs.vdev.mirror.rotating_seek_inc: 5 >>> vfs.zfs.vdev.mirror.rotating_seek_offset: 1048576 >>> vfs.zfs.vdev.mirror.non_rotating_inc: 0 >>> vfs.zfs.vdev.mirror.non_rotating_seek_inc: 1 >>> vfs.zfs.vdev.max_active: 1000 >>> vfs.zfs.vdev.sync_read_min_active: 32 >>> vfs.zfs.vdev.sync_read_max_active: 32 >>> vfs.zfs.vdev.sync_write_min_active: 32 >>> vfs.zfs.vdev.sync_write_max_active: 32 >>> vfs.zfs.vdev.async_read_min_active: 32 >>> vfs.zfs.vdev.async_read_max_active: 32 >>> vfs.zfs.vdev.async_write_min_active: 32 >>> vfs.zfs.vdev.async_write_max_active: 32 >>> vfs.zfs.vdev.scrub_min_active: 1 >>> vfs.zfs.vdev.scrub_max_active: 2 >>> vfs.zfs.vdev.trim_min_active: 1 >>> vfs.zfs.vdev.trim_max_active: 64 >>> vfs.zfs.vdev.aggregation_limit: 131072 >>> vfs.zfs.vdev.read_gap_limit: 32768 >>> vfs.zfs.vdev.write_gap_limit: 4096 >>> vfs.zfs.vdev.bio_flush_disable: 0 >>> vfs.zfs.vdev.bio_delete_disable: 0 >>> vfs.zfs.vdev.trim_max_bytes: 2147483648 >>> vfs.zfs.vdev.trim_max_pending: 64 >>> vfs.zfs.max_auto_ashift: 13 >>> vfs.zfs.min_auto_ashift: 9 >>> vfs.zfs.zil_replay_disable: 0 >>> vfs.zfs.cache_flush_disable: 0 >>> vfs.zfs.zio.use_uma: 1 >>> vfs.zfs.zio.exclude_metadata: 0 >>> vfs.zfs.sync_pass_deferred_free: 2 >>> vfs.zfs.sync_pass_dont_compress: 5 >>> vfs.zfs.sync_pass_rewrite: 2 >>> vfs.zfs.snapshot_list_prefetch: 0 >>> vfs.zfs.super_owner: 0 >>> vfs.zfs.debug: 0 >>> vfs.zfs.version.ioctl: 4 >>> vfs.zfs.version.acl: 1 >>> vfs.zfs.version.spa: 5000 >>> vfs.zfs.version.zpl: 5 >>> vfs.zfs.vol.mode: 1 >>> vfs.zfs.trim.enabled: 0 >>> vfs.zfs.trim.txg_delay: 32 >>> vfs.zfs.trim.timeout: 30 >>> vfs.zfs.trim.max_interval: 1 >>> >>> And nvm: >>> ev.nvme.%parent: >>> dev.nvme.0.%desc: Generic NVMe Device >>> dev.nvme.0.%driver: nvme >>> dev.nvme.0.%location: slot=0 function=0 handle=\_SB_.PCI0.BR3A.D08A >>> dev.nvme.0.%pnpinfo: vendor=0x8086 device=0x0953 subvendor=0x8086 subdevice=0x370a class=0x010802 >>> dev.nvme.0.%parent: pci4 >>> dev.nvme.0.int_coal_time: 0 >>> dev.nvme.0.int_coal_threshold: 0 >>> dev.nvme.0.timeout_period: 30 >>> dev.nvme.0.num_cmds: 811857 >>> dev.nvme.0.num_intr_handler_calls: 485242 >>> dev.nvme.0.reset_stats: 0 >>> dev.nvme.0.adminq.num_entries: 128 >>> dev.nvme.0.adminq.num_trackers: 16 >>> dev.nvme.0.adminq.sq_head: 12 >>> dev.nvme.0.adminq.sq_tail: 12 >>> dev.nvme.0.adminq.cq_head: 8 >>> dev.nvme.0.adminq.num_cmds: 12 >>> dev.nvme.0.adminq.num_intr_handler_calls: 7 >>> dev.nvme.0.adminq.dump_debug: 0 >>> dev.nvme.0.ioq0.num_entries: 256 >>> dev.nvme.0.ioq0.num_trackers: 128 >>> dev.nvme.0.ioq0.sq_head: 69 >>> dev.nvme.0.ioq0.sq_tail: 69 >>> dev.nvme.0.ioq0.cq_head: 69 >>> dev.nvme.0.ioq0.num_cmds: 811845 >>> dev.nvme.0.ioq0.num_intr_handler_calls: 485235 >>> dev.nvme.0.ioq0.dump_debug: 0 >>> dev.nvme.1.%desc: Generic NVMe Device >>> dev.nvme.1.%driver: nvme >>> dev.nvme.1.%location: slot=0 function=0 handle=\_SB_.PCI0.BR3B.H000 >>> dev.nvme.1.%pnpinfo: vendor=0x8086 device=0x0953 subvendor=0x8086 subdevice=0x370a class=0x010802 >>> dev.nvme.1.%parent: pci5 >>> dev.nvme.1.int_coal_time: 0 >>> dev.nvme.1.int_coal_threshold: 0 >>> dev.nvme.1.timeout_period: 30 >>> dev.nvme.1.num_cmds: 167 >>> dev.nvme.1.num_intr_handler_calls: 163 >>> dev.nvme.1.reset_stats: 0 >>> dev.nvme.1.adminq.num_entries: 128 >>> dev.nvme.1.adminq.num_trackers: 16 >>> dev.nvme.1.adminq.sq_head: 12 >>> dev.nvme.1.adminq.sq_tail: 12 >>> dev.nvme.1.adminq.cq_head: 8 >>> dev.nvme.1.adminq.num_cmds: 12 >>> dev.nvme.1.adminq.num_intr_handler_calls: 8 >>> dev.nvme.1.adminq.dump_debug: 0 >>> dev.nvme.1.ioq0.num_entries: 256 >>> dev.nvme.1.ioq0.num_trackers: 128 >>> dev.nvme.1.ioq0.sq_head: 155 >>> dev.nvme.1.ioq0.sq_tail: 155 >>> dev.nvme.1.ioq0.cq_head: 155 >>> dev.nvme.1.ioq0.num_cmds: 155 >>> dev.nvme.1.ioq0.num_intr_handler_calls: 155 >>> dev.nvme.1.ioq0.dump_debug: 0 >>> >>> Best regards, >>> Vintila Mihai Alexandru >>> >>>> On 1/17/2015 12:13 AM, Barney Wolff wrote: >>>> I suspect Linux defaults to noatime - at least it does on my rpi. I >>>> believe the FreeBSD default is the other way. That may explain some >>>> of the difference. >>>> >>>> Also, did you use gnop to force the zpool to start on a 4k boundary? >>>> If not, and the zpool happens to be offset, that's another big hit. >>>> Same for ufs, especially if the disk has logical sectors of 512 but >>>> physical of 4096. One can complain that FreeBSD should prevent, or >>>> at least warn about, this sort of foot-shooting. >>>> >>>>> On Fri, Jan 16, 2015 at 10:21:07PM +0200, Mihai-Alexandru Vintila wrote: >>>>> @Barney Wolff it's a new pool with only changes recordsize=4k and >>>>> compression=lz4 . On linux test is on ext4 with default values. Penalty is >>>>> pretty high. Also there is a read penalty for read as well between ufs and >>>>> zfs. Even on nvmecontrol perftest you can see the read penalty it's not >>>>> normal to have same result for both write and read >>> >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >> >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"