From owner-freebsd-fs@freebsd.org Sun Jun 18 15:15:49 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE991D8835D for ; Sun, 18 Jun 2017 15:15:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DC7A172ABA for ; Sun, 18 Jun 2017 15:15:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5IFFmsf064464 for ; Sun, 18 Jun 2017 15:15:49 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 212608] nfsd(8) listens on UDP port 2049, but it is reported incorrectly in sockstat(1) and lsof(8) Date: Sun, 18 Jun 2017 15:15:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: m.bueker@berlin.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Jun 2017 15:15:50 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212608 --- Comment #5 from m.bueker@berlin.de --- I can confirm this bug on 11.0, but only for the UDP port: # sockstat -4l | grep 2049 root nfsd 646 5 tcp4 *:2049 *:* ? ? ? ? udp4 *:2049 *:* --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Jun 18 17:06:24 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7FBF9D89F62 for ; Sun, 18 Jun 2017 17:06:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6DEDF7533B for ; Sun, 18 Jun 2017 17:06:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5IH6N5p052588 for ; Sun, 18 Jun 2017 17:06:24 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220105] boot1.efi does not detect ZFS pools on whole disks Date: Sun, 18 Jun 2017 17:06:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Jun 2017 17:06:24 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220105 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Jun 18 20:40:57 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7227D8D9D1 for ; Sun, 18 Jun 2017 20:40:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 955337B434 for ; Sun, 18 Jun 2017 20:40:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5IKevGJ057533 for ; Sun, 18 Jun 2017 20:40:57 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220105] boot1.efi does not detect ZFS pools on whole disks Date: Sun, 18 Jun 2017 20:40:57 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kallix@kallix.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Jun 2017 20:40:57 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220105 --- Comment #2 from Kevin Allix --- Commenting out the two lines in boot1.c allowed boot1.efi to detect my zfs pool, and to load loader.efi . However, loader.efi could not find any zfs pool neither (nothing in the ZFS section of the output of lsdev) --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Jun 18 21:01:17 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DEE14D8E277 for ; Sun, 18 Jun 2017 21:01:17 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D087C7C38D for ; Sun, 18 Jun 2017 21:01:17 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5IL01L6025316 for ; Sun, 18 Jun 2017 21:01:17 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201706182101.v5IL01L6025316@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention Date: Sun, 18 Jun 2017 21:01:17 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Jun 2017 21:01:18 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic New | 217062 | for file systems mounted with -o noexec, exec=off Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS 7 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Sun Jun 18 21:59:28 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D800BD8F48E for ; Sun, 18 Jun 2017 21:59:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BC8AC7E26A for ; Sun, 18 Jun 2017 21:59:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5ILxSZt027159 for ; Sun, 18 Jun 2017 21:59:28 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219972] Unable to zpool export following some zfs recv Date: Sun, 18 Jun 2017 21:59:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: pfribeiro@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Jun 2017 21:59:28 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219972 --- Comment #2 from pfribeiro@gmail.com --- I would just like to add that this behaviour is reproducible using a device= as a file, rather than just usb devices, eg: mdconfig -a -t vnode blob00, and = then sending a snapshot from md0 to md1 using the 'zfs send' and 'zfs recv' flag= s as in comment #1. An accompanying thread that I opened earlier about this behaviour is availa= ble at: https://forums.freebsd.org/threads/61291/. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Jun 19 22:14:17 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48913DA5D8C for ; Mon, 19 Jun 2017 22:14:17 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on0062.outbound.protection.outlook.com [104.47.36.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CAC971BFB for ; Mon, 19 Jun 2017 22:14:15 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weatherford.onmicrosoft.com; s=selector1-ca-weatherford-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=DIQdfXrb4dfm+rNrwkJ9SPhKtHm7Ky2p/SxlrsDujVA=; b=BIiS1JJC6aNRaj/D2Rd6XLfwvD0gpir6KMZ2yhAkLMwpOjJ83zNj9f/jyp1c/q5TyIg3stWWaGcZ7jpyMxdwazNs2kmxjQCPz8Bjq91jPgrvdYQQ/GDhftNIeCXx6oic4ewUHlmsnoTMzY7ejSACxqqx55OcU6Ek2q4pKQB79M4= Received: from BN3PR03CA0079.namprd03.prod.outlook.com (10.167.1.167) by CY4PR03MB3270.namprd03.prod.outlook.com (10.171.246.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Mon, 19 Jun 2017 22:14:14 +0000 Received: from BY2FFO11FD011.protection.gbl (2a01:111:f400:7c0c::147) by BN3PR03CA0079.outlook.office365.com (2a01:111:e400:7a4d::39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14 via Frontend Transport; Mon, 19 Jun 2017 22:14:13 +0000 Authentication-Results: spf=pass (sender IP is 23.103.226.20) smtp.mailfrom=ca.weatherford.com; freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=bestguesspass action=none header.from=ca.weatherford.com; Received-SPF: Pass (protection.outlook.com: domain of ca.weatherford.com designates 23.103.226.20 as permitted sender) receiver=protection.outlook.com; client-ip=23.103.226.20; helo=032-smtp-out.weatherford.com; Received: from 032-smtp-out.weatherford.com (23.103.226.20) by BY2FFO11FD011.mail.protection.outlook.com (10.1.14.129) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1157.12 via Frontend Transport; Mon, 19 Jun 2017 22:14:13 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net (141.251.110.81) by DM2PR58MB016.032d.mgd.msft.net (141.251.110.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Mon, 19 Jun 2017 22:14:11 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) by DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) with mapi id 15.01.1178.018; Mon, 19 Jun 2017 22:14:11 +0000 From: "Caza, Aaron" To: "freebsd-fs@freebsd.org" Subject: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Topic: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Index: AdLpSTjOAvUi3ktqRja7BT87AbAIaw== Date: Mon, 19 Jun 2017 22:14:11 +0000 Message-ID: Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [141.251.136.132] x-ms-publictraffictype: Email X-MS-TrafficTypeDiagnostic: DM2PR58MB016:|CY4PR03MB3270: X-MS-Office365-Filtering-Correlation-Id: fb18870c-ce11-4169-5226-08d4b760846d MIME-Version: 1.0 X-OrganizationHeadersPreserved: DM2PR58MB016.032d.mgd.msft.net X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:23.103.226.20; IPV:NLI; CTRY:; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(39400400002)(39410400002)(39840400002)(39450400003)(39850400002)(39860400002)(2980300002)(438002)(199003)(189002)(9170700003)(2351001)(9686003)(189998001)(33646002)(86362001)(106466001)(575784001)(478600001)(72206003)(8936002)(81166006)(24736003)(22756006)(7736002)(356003)(8676002)(2906002)(54356999)(110136004)(5890100001)(50986999)(38730400002)(2501003)(66066001)(6306002)(54896002)(6916009)(7696004)(55016002)(53936002)(5660300001)(42882006)(5640700003)(3846002)(108616004)(86146001)(84326002)(512954002)(2900100001)(102836003)(260700001)(790700001); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR03MB3270; H:032-smtp-out.weatherford.com; FPR:; SPF:Pass; MLV:sfv; A:0; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD011; 1:AQm8X5jRL7Unla0Or2GnM+mNmgIMeRP9SK0IFJ+Vw7liIwb7LlNpYTwJvbKFBil6cWu+1m35Tf8YWdFbOqFAj5bM3Eq5LiS/3e1eCxSt7p9giEbO6M4XyMwHpBuXVInNtrECmWnMGZL5F5qUEdrpdRPaiTvk7eCkJjIXL6kKSl+oJlUsnd1TGTDdt6u5ed7asEuP6d1RTqxGW5fYMU6xFw994u/UpNNtlqr+10b0aqtw1IUHi4Ma24QWfq6V3IZHXVAsta94xkJzykNHqVMDXRrTTwdT9SSnp67bOAco9UG36luUVfj4IbOH6nD1bIHKqret0k5IiadFLK9b4gE41/lH5RfcHidrrc33pCKSZJzilYJhF7EKxaArrOoqX6rNhGWH0AVAEOsFSm9HfMufgyjhbvH1S8+JsdQKg9VMlc+N8qRPnWWbfsSlwqSVbGNj2GVYXwS+N6D6GyRMOkBa/Q9fH+CFVDJBVP+sJuz4Z1jkL7RoBaZTjxqFWY5nCh+l+XhL5DupwFKDkqgVqBlNHQ== X-CrossPremisesHeadersPromoted: BY2FFO11FD011.protection.gbl X-CrossPremisesHeadersFiltered: BY2FFO11FD011.protection.gbl X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(8251501002)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:CY4PR03MB3270; X-Microsoft-Exchange-Diagnostics: 1; CY4PR03MB3270; 3:SzyVPTQmTLnNuWriDcf1iz5/303lxmwh4AVXvbW+BsNAeSDJvFQqhteZT9WBzywGg7SapGcyGO/6CHEaY+8Qj8uR6G9sdyAKtUJpvnSpfni0iRvOTOtZMHnCgns8Bjc5Knn/9a42YTxsimIuR9kev/yRvDPFTsY/JYRVBd16r+7a5nmQFp30YgR3YD2JDUyZdD7Kc3usZNDslrOLVAtipVjONkFOZCEAEjyJk6GRa2O/ZOIuoFUrisSwmn+DmvErMBBXea2w8ahnL6EBA+i1vUz0NqK2u1nIGQHyegXz10A9a4EqV/+KQDf8gxRXzVTIGXA4Vh6MexCyio9scqjzxBMc47RA8l33+Ei074cewlwNINrTCVBj5Rl/zgYBoTPSyLctdoLhbhxNQhFPrHxvDJOF0fFK8HDpZPI8Wic27xWnLloY7HN9+7TowIWnl2QrmHS/eRlY48QOWYzmlUDiec8G+hyrS3VlZoip0RxsVOUeGY66tdZr+Q3dew2UYVLqOeadNKvmGyQJPb9nyY/QPA== X-Microsoft-Exchange-Diagnostics: 1; CY4PR03MB3270; 25:yLech/z2nLvK7PtiXTnkLkrttYKUydJH7Wt7Qp2r74/rkeSUznfoyAVAc1Y+ETd/Rg4tvhrqLwmZwHzPJNf4/RxVFpQurIVGOOeTQiLbRd7xKYZNO5c0OK3qQZ3MpHbwKcCNUgsbFgFkiAItzIot4OFi58/3DuI813Ts3zI932wMuG4Z+SUuTv/3xGuE0IThP6lLd8sWh9KzCAIf2zl+ZViJjfPfpY7adR30HzTmhlpRPxue5GYgeMDnGHeSy0ZPsTea0uiOt9pt+RdimiiJSj3zNadWyYI9daBqLQx7LaPg8YGkTwMwNg7PLo5EtwXL9EBp7SFDK5U14uBbS61XOhzavfVei+EhS1moV8PCgn5JaKaPNZ1EnlBdBjsUXoKUrsSNMFOkII+Zut7sscl1q9uWNpcu11ntNvoAHxwPvpRJDnrqTRe4JiC0W+MLXr0D2vYiaazis44zOkbr9o/CCasYrGZvbRpinMc8KTa2Plc=; 31:ZA4zc69mu6IBlkow7ETPBVhpDF6IEaquawKjPZ9BQB1NPPDTm5JqgttyfvFLQMEytOcb4x1xkI3vo1/RDLNzziqFU94m8dxyabLQx/ub1C3tXfCO79kxHeELyWy01JVL9wpW1iVQzvCM1pQLVJlHhwapc9kim3rMweQrze9PjL3/yCRSkwsQneZf/2W4fUxKiIK+XCn4HgubcVwRqqbYlmXrJMSoU8GxyG+d1NKM2+ZWFlbdAJpRKJUEYhqEkplQwfEb81wjnJmmM9tE4QF0HA== X-Microsoft-Exchange-Diagnostics: 1; CY4PR03MB3270; 20:vn5G8kiJaxzJ86akjNcldU/5j8f1pDXAKuWlk/IawflcDtV7l2bDod9GYHGYNBLlwHuClQwia69/E6ObjXEieu0acRktMGbsuYF7s+57ssHW1TBrW3K5wH7V6boXmenp73Nu+7RmySyEbGWfUBe1xaSOkOb8I+Cr0XxdE30Li4RMc6l2FXIL7yAx5TrwvgnHCa+vL1k5DJj4Y7BUYRE7ZnwFTbCMXCoacH1ANBYYkQ4uCOmeGAkg66XCqJzgtCW386e0sBj5XPzmAiqlfBP86CstKMFtp/0cGeRzsvbG+iJCBFCVp2SYR98sMCc320iWWv96hdVT5Q49UsH/8y77wdeoO5t3YnJdPJgjKY9iLHgugsioxOyNUY6dx7oe7i1kPpah60D3Sli9btoqKdOj5gJMH8Rnd4ApW67W4tNMEcXsuNCbbKfbM9doMWRg8P7Utz88H2BzkgPgXdn6rTcpfM1F5rtelELbIvL2qAV/jAgXvTkcf2768ao8xCRj/sup X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(209352067349851)(192374486261705)(21748063052155); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(11241501159)(6040450)(601004)(2401047)(8121501046)(13016025)(5005006)(13018025)(93006095)(93004095)(3002001)(100000703101)(100105400095)(10201501046)(6055026)(6041248)(20161123558100)(20161123562025)(20161123555025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:CY4PR03MB3270; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:CY4PR03MB3270; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR03MB3270; 4:lBYO8LSYDx2tnvFfv/O9975zvqumhypm3kcjFZF4nl?= =?us-ascii?Q?WpfsPodxrMXlCrFLhfr8S2vei3R2JPloTSiCqlg0nu1H51G0o09txxy7FUQ9?= =?us-ascii?Q?8JQ5xVbAWpNvwGryPFj+kYpLltCgxGjwoN1CH3GWIfA45Rjwp1bhC5ergYin?= =?us-ascii?Q?HsixjdTefY4aGerodOtL68krnliUzD2JdOjSmj6t6Wmrz1oHAPS4gy4B/e+H?= =?us-ascii?Q?Pyt4BlzX+4ZGwzcPtP3Sf1KfzFNZ+MtWsfpyKOiGTmVPQKGUNNbvyqz6KT8+?= =?us-ascii?Q?uxHi0dlOPXKvR9EAuIjHFsPsmGMr7flb+Zn8vFfmZAObznvbHlp3YH25wunY?= =?us-ascii?Q?XUCO3WVXQUmpft3R04gZ19OS2EKuoUTemT6cNj26Yj9Ci308636K7lgDRJYp?= =?us-ascii?Q?aWEQwK3gDeKuHiKCkPZ4zM2w1zfeH7GiIvPZoBfStMaRu1d+moJNkrebGWsP?= =?us-ascii?Q?UnG2MfH6fjB0RQn+gk9B0FiylO7hjXmBX28IoA0ktG3MwjwJc8yKZSU/JnF7?= =?us-ascii?Q?/Upip/suqPCDCVA3JP+52hIwOGbYavUplc1HUNezI0C92ttZcHqWm9gkhcJY?= =?us-ascii?Q?aDAcLP+efF8YcvSjrxxg/a0GQf54OoohtWQPNt1cMPNjqUQtE4wdoh6e0xEH?= =?us-ascii?Q?QZT6LfAGbLIdCh8Cor++3xxzsbhNeAmUobKEU4F2ke8R41bJQ10x9C/vbv87?= =?us-ascii?Q?3ylgsE66PpUfgElhxvMX0Bfbq8ZuWvBzdpvXU6UZ9e3DdkkodIrmNL80TswX?= =?us-ascii?Q?U4KD1df6qUtrTULXziBFYmsetpWI6qDdshmsyqmN4ya3kZLnaFLbSBhqt6Xd?= =?us-ascii?Q?iDcfwQKX5D/AP4oJO6pMejAu6opzLa+oJKnEFBczxjoYIBkn54+uQWrpiVQ+?= =?us-ascii?Q?X430zTmKQrNVm2S2FzG8zV6RiWSAliCZnTsCVF0ZutdGRumUppxdZD49Oa4K?= =?us-ascii?Q?wY4wu6qCKbeizTq3o+LxpTa4fZ9YkJPC8Uf/uhjyQWGy/B8VV848ydRbL6ub?= =?us-ascii?Q?X1BC5HWgfIGH1qgkVRsm7rVQMqKdNkYt0tWGq72KjdPKnl7Bq9yg7GVZWroN?= =?us-ascii?Q?qA8Vb7FquHpZkR9Kb0ol/q5LkppF4fWydL3v93vzJ7mzkrUcLX7FoxyjXmhY?= =?us-ascii?Q?Ru5ynmUGUK4j5cSPNjRDTcapzaRfMZlitbUMC2bet+BlQW66RTA5l1BaxafY?= =?us-ascii?Q?C1jNZitMaXMf1EA/vtHXKhm0e0EZHp+PsvhMb1YAYwzQ6eHrbZOSxBY8xPt3?= =?us-ascii?Q?B5jQyzsMmr5NQ8eBdg4dzBwUc5viGpNV8Gugi7ovaq8w0fuKiTWuI03O97kH?= =?us-ascii?Q?5ZCpech6Kwy7kXZcjBF0TwUfrolryjkEZN2FYDYn+E?= X-Forefront-PRVS: 0343AC1D30 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR03MB3270; 23:nNhAiGcnT6ChKIRRbKQ5EHHqHZrVIdmvdYgUe8bTU?= =?us-ascii?Q?KrHTyaMMd9SoeW79YTwCCChzhitEqGXluZ2L3ITjfl4SnyZmEvKUS2t+TmTH?= =?us-ascii?Q?eoMNHEzlX9xnZwzmIHD1yXusK5OnNTnPfEJPkCjVQM6stfKQeMXjf2I30+PT?= =?us-ascii?Q?BgL63N/A5VKTEgA2zGp+8dsnFSiAoAiic0FIbncL0c/H4ftH5F19Veo4WmS8?= =?us-ascii?Q?arduTP4/fe4VDAgA8qwz0U4XSXtT0Vl8WO7OCDabblX3qRsgPXRbN17wSx0u?= =?us-ascii?Q?xtmhTA8kBHtChIPT5NMWwFcOvKDtDWR/dQVlPMrRb4t7DB7Wd0KvMxx9g3yy?= =?us-ascii?Q?PJwAht4HsN0QPiZukVFVA5yUEsxVhG9KGSO/pcuB9v+mcjrrQKxwaGe5L9sK?= =?us-ascii?Q?0NvS2mjxz0KUPOqzt5ng8Yjexzrw8XaOdlgnxbX72PIrFeAKThb1oMC8Fw6Y?= =?us-ascii?Q?L7GtgXeHOMdxH53mjTYZXxoi3AuKlHEYsM55Q+F3xJAuPX1VOLZq9GaKBjU+?= =?us-ascii?Q?ZsxLcFCuSoUbNMMVKtQOmkMJL12TYaO+xT0MsFq0GgI15zWkrm7WHNsbAV7D?= =?us-ascii?Q?4atmgp1G6ayDjBB1gS/2yyc44Ka6ry4lbfMCgKam6NLzUJesUO2793TjoJbL?= =?us-ascii?Q?pf+VwhqeOdxsyO0SrALA3ln9k9RWttn0aVZ2oOX5/wnnxavcUWDBYpH5frOl?= =?us-ascii?Q?WW3zdfKXv1+NiVa7mwNfjc4a7m5Z6YxCuvPkEqEoOiMsG6qQMYzXqvQI68WX?= =?us-ascii?Q?+pTLywvikgvl4YNcgwF3YqsJ1gWZvpeIq3Mjq+nuVLUPGNe6buS62Ia9iIiN?= =?us-ascii?Q?I9Fj4vcoUAV3l+69sP6GpDrXUYGf7B0QINMH5rvEBIPDyVVFm/Lw/VAxqLaZ?= =?us-ascii?Q?k+MIowKwWpxq2/BvX+vop/tqE88/boOdaf1iwH5eDwZ50vyCept5kqJjKuMa?= =?us-ascii?Q?iR1jdcdoR74ChGXcotoXV/Ri+YxdRzTQdxYK1jBwEHE+/xqsHKK49NVIjC60?= =?us-ascii?Q?zXNj/j+EhvvsWTAZqiwoJqimMus6xByjSeKYXBdL9D/UNvzAZ3xWkHCHp3UC?= =?us-ascii?Q?dGrBOmxIlOZyW2fM96/5uBMdYmJPu8N0hoJa9wO519qPUIJiBiIR9MAHUelE?= =?us-ascii?Q?u+5QM4xuAsKn8fcJAYRaEOZ/q7LummPFEOFp/H7P7zQ4klxRNpmLK8ID/EjN?= =?us-ascii?Q?iZkhGhMOnJSNELkNlv0v8QU7V81SMNlqJ6/CXPDY2qaYXct8bqoacMwTM9FZ?= =?us-ascii?Q?eDZisoXNSGhzgdEu4waACJRSTfFTVeCxwQrIYIK7EX37nAd7xklqVTwbZhU2?= =?us-ascii?Q?gpU7S8iVz6xNBSAHgm74XQ=3D?= X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR03MB3270; 6:vy8xM4Bgxc/xXlSf/E0Y4UWUWQNnbIyggS/llpeNE5?= =?us-ascii?Q?+E257ojyaodHSWu0BjusJ2UjU5F2GDPGObZ4l37oYhVZ7JSLJVjWWTAIT68k?= =?us-ascii?Q?rdABlDsNsRvvfO9FB0HsZs0q4kSQzkzVfGzbUGTDjqibPlHQLlGFxE/dE0AZ?= =?us-ascii?Q?yjuXtQB4c7ePm6+RR3QF7eCdGzTVSnKUYGmQNE0dFru7nfAyDy6e+sAQRH9+?= =?us-ascii?Q?Yoq/ozXKdZzEgKgzLGzaHkoUAqHnYr3mBpL1XqBqZeoYTxLUe0VTk7+ZzUoQ?= =?us-ascii?Q?+o6LoRblj5X5ioEY8lDycOEBr3NkcByuhPxi/qQ1ObdvaJppa1ezqfBYDe3r?= =?us-ascii?Q?kQ7pm+1jouKWQNTlYQ094vvoiB8Z6q1wIwEdUatCnQWEkQsO8KwkLZ7XMEDq?= =?us-ascii?Q?dyjyf8Kb+K01N9sO2MMU0MhvhxUlgAW5RQlCLm9Ma41Y76nlbKO925cTIarR?= =?us-ascii?Q?TZLi1qFOoBAzZRX+hd0LeHzR5JOL1tNUQrVTg8igNGHSrq6/NlZOa2BZDXzK?= =?us-ascii?Q?o98MMkZA3eKKAaIEycEjJsCyY9R+tZZ0xMJio8GJta4pUszNSckyxONPz+hs?= =?us-ascii?Q?9eODwDPEyZUvkHOfzGYQYOAEx/pZKzbud/fqMqAgLNUryE4Deua6CCRw4eh2?= =?us-ascii?Q?E6p/1ynqr1tKeJz44WvlZhKhQpxj2AV2mVyj3tr0CUPbaeGxAgppDEljvOYs?= =?us-ascii?Q?KZSyWmUBLSKCUMcmFXSh6kGS15ETB36dU+IcK3+kfS4GOIDAZXlNXwl+z4s1?= =?us-ascii?Q?IhSR+u11rIMgabS6wUQMAf2fm7BRZ8eHS+1Q8jYMSSJGSD4O6r8Zbu5Z2ezl?= =?us-ascii?Q?ppdQPtTkVIm+B9L3YRY6zwiZVEJO7+EX/Phv7KermIIryGJtbPUk0JuhIF5K?= =?us-ascii?Q?DHCIhUVPPpushyZ1TR9OU8lbsWoH9MHpM0R57BclGcPLBaqrRb/6KSEwgN9W?= =?us-ascii?Q?7BZkzuFoqUqZiFksBD32K5sHnNq2jtO7gVWa62yZ6RPe5mufhpvt8SXu2lV7?= =?us-ascii?Q?jf8Rai5ZTa8HXhmii1JiVe?= X-Microsoft-Exchange-Diagnostics: 1; CY4PR03MB3270; 5:ySsDoZeewOgfvnkSRIqmy+qsDFiUMf+xFTF08/ltV0nCni4QQje5yTp7rAA1Dujy2ZijWTDxK4IxnNtiTk2N7aTeSWzHpDPJpojyWS72Ark/j1dmOCBeiaHXcDpngXgxOtO1h61uVus4plwAqr4BCXrZDcAnHSU4czWaIbaWRAfhi/bDUNOodW6FmJwuLPEW+3k0JhNL+2d8JXeiS3WJfksIfEpqTFrCYLfjq/xHG+iqBZXp5jkv30E79+fOYcuzcMJ0NY+R2kmsKjTiEaZ++TTyfXOEv2IUrAzL61MAJr2F9OcrAzM+aWMORuqV0IYnuwvXXDi9yeqz6fq5AJ0TTB/9iKk8pJSiEeiiDNQIx4rkn/oJwl0H5hvGkpSKU2YLvHKLId6MehUcTubP8/mbK9+KTq3pxpJ4TDSVrfSyn5bPEZcwKSsQQ+LtQnJ+MoQqVEFYDYi7+plF4f6qzXwSCODROnTmjGW13AFKeTh/KPP7WGO6fjdFVqZkv7laFOGJ; 24:fMwF2oJtdK1RWdcSgt7IzYLwso8J0QavUbPwHeqTojh42oeFH5ACsSQfEKtTF6NiHPxIC55PK2WRY/2PcX7vAjyBAcagHoX7hsNV6M6Y46I= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; CY4PR03MB3270; 7:HnTn3IdHqm2RviPAFb6Z1ClHCQ8xntny/G6fiVZUmpZoPFXVVPbDapMS53rDpRXmcty7kBBVrIPf1rwHAyjCIlKGSP7T2PQrnquQQN1OgD6sU4J0SmHavt5/baspks0U6jTx4mNtIPIQST/dh9dwrsKxQBAxySo9ZOsyWS4blUo97aCBXLj5t2ICxlhCR5xu4KrCx4eIFHmYfLaL/Mx+4Tj3WSoTmAQaOcq3Fdv0NJ2TO098IRw5/7WogGXE2ZRhvaAcezE1+YAgRnneiypAUlhpT7wTTQrpK92DV+sg58B4WuONVDum6tLX9ouvY5CuiUOxerFBeW/MRgx8nsB/a782ya8ZC4oqAqK12asD3RHJL9lPFsVD87QOnoTBUrLnjUPA073yrwgIdnoGONDwUIjW9omVvpkkNYPprxyc/8rFvdoIKyTK5PXCwjbO48PxH8vJ3RzAW39ppMutSV3E+Xa/4b4QG+eLy2zmqOnViqQGJtV3qQOAVLykGlhZzlpwwXBxdK9N5aqjiERnfho/BpvstVzbghe3eLyRxgaWtd2Aoe0uAsvbsDsg0zXFVd0bfBYOxoB1P7ysq5uDErVjwWcqwGfb8K6LArWWlEhu1MlsyBMmZPoJ0SkEuPqcBaxREBiot3Pf1aT7oU9UudnZ1QdRvNCvuY8kqeZS39jWwvcTMy/l8CXeeWveoh8+V2eo5e4MQiSw+ONqzNY5E7Td5YTYg5cpPIE7TNyYAGnLo35yh5oltvZpArAv20PlIbYpyWBzjdETwvpVoP3rxkBJEaZMNQBllTgOcCrl2XNJSJc= X-OriginatorOrg: ca.weatherford.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jun 2017 22:14:13.1802 (UTC) X-MS-Exchange-CrossTenant-Id: dd63fb60-07f6-4d96-8d40-ebeca61a524e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=dd63fb60-07f6-4d96-8d40-ebeca61a524e; Ip=[23.103.226.20]; Helo=[032-smtp-out.weatherford.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR03MB3270 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jun 2017 22:14:17 -0000 I've been having a problem with FreeBSD ZFS SSD performance inexplicably d= egrading after < 24 hours uptime as described in a separate e-mail thread.= In an effort to get down to basics, I've now performed a ZFS-on-Root inst= all of FreeBSD 11.1 Beta 2 amd64 using the default Auto(ZFS) install using = the default 4k sector emulation (vfs.zfs.min_auto_ashift=3D12) setting (no = swap, not encrypted). Firstly, the vfs.zfs.min_auto_ashift=3D12 is set correctly in the /boot/loa= der.conf file, but doesn't appear to work because when I log in and do "sys= tctl -a | grep min_auto_ashift" it's set to 9 and not 12 as expected. I tr= ied setting it to vfs.zfs.min_auto_ashift=3D"12" in /boot/loader.conf but t= hat didn't make any difference so I finally just added it to /etc/sysctl.co= nf where it seems to work. So, something needs to be changed to make this = functionaly work correctly. Next, after reboot I was expecting somewhere in the neighborhood of 950MB/s= from the ZFS mirrored zpool of 2 Samsung 850 Pro 256GB SSDs that I'm using= as I was previously seeing this before with my previous FreeBSD 11-Stable = setup which, admittedly, is a different from the way the bsdinstall script = does it. However, I'm seeing half that on bootup. Performance result: Starting 'dd' test of large file...please wait 16000+0 records in 16000+0 records out 16777216000 bytes transferred in 37.407043 secs (448504207 bytes/sec) Zpool Status: pool: zroot state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0p2 ONLINE 0 0 0 ada1p2 ONLINE 0 0 0 /boot/loader.conf: kern.geom.label.disk_ident.enable=3D"0" kern.geom.label.gptid.enable=3D"0" vfs.zfs.min_auto_ashift=3D12 vfs.zfs.arc_min=3D"50M" vfs.zfs.arc_max=3D"51M" zfs_load=3D"YES" /etc/sysctl.conf: vfs.zfs.min_auto_ashift=3D12 Is this the expected behavior now in FreeBSD 11.1? -- Aaron This message may contain confidential and privileged information. If it has= been sent to you in error, please reply to advise the sender of the error = and then immediately delete it. If you are not the intended recipient, do n= ot read, copy, disclose or otherwise use this message. The sender disclaims= any liability for such unauthorized use. PLEASE NOTE that all incoming e-m= ails sent to Weatherford e-mail accounts will be archived and may be scanne= d by us and/or by external service providers to detect and prevent threats = to our systems, investigate illegal or inappropriate behavior, and/or elimi= nate unsolicited promotional e-mails (spam). This process could result in d= eletion of a legitimate e-mail before it is read by its intended recipient = at our organization. Moreover, based on the scanning results, the full text= of e-mails and attachments may be made available to Weatherford security a= nd other personnel for review and appropriate action. If you have any conce= rns about this process, please contact us at dataprivacy@weatherford.com. From owner-freebsd-fs@freebsd.org Mon Jun 19 22:22:52 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CF7C5DA604A for ; Mon, 19 Jun 2017 22:22:52 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: from mail-lf0-x22c.google.com (mail-lf0-x22c.google.com [IPv6:2a00:1450:4010:c07::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 43865207A for ; Mon, 19 Jun 2017 22:22:52 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: by mail-lf0-x22c.google.com with SMTP id l13so3984007lfl.1 for ; Mon, 19 Jun 2017 15:22:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphix.com; s=google; h=mime-version:from:date:message-id:subject:to; bh=BI5+ws2Y4wHV/R9mHpiiUmRNpLjoc5T+GoUUGPGapOM=; b=MP5b/WJyDc/A0rOdJ8oeakXXTCnu+TEeSvfT4K0xt7PNx5Q+0FsGz0afUatnEpPfgJ MfV9TmGQfoTnGZ9d1NMjsBmvOZBPca9d8BkpvsY1zavByNiKzeF4CKGFC5eQAxvO+S9d 0agaz58Ku0acNX48P9x+AzV+Je4umcrHOb+LE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=BI5+ws2Y4wHV/R9mHpiiUmRNpLjoc5T+GoUUGPGapOM=; b=TzlytWHcls66a0C1caWHeXWAd8ZRTp5KDNO0GhYJcDnEXgqQX/lKPipmD8KshhezE5 9HnbsUIraN/397k861r+xkAej8bhIvYStVlK609IYXSzFnLIuie2IbfUtiskBdmBI4eb bB+NKvcthWm6jGrdCua2/gATgGLlq5miMhbQZmHq9NBX/AZ09OeBKiiD78tIATefNzM1 hXgGcu5jCPcelsoOJLcf+4EJTgp2VQDdDk7DRJ+lLpf6Nb63JMQa4A+k5duQkYH+izt3 FuLqWSUym8QlaeiVswFVjfHSs2i9lV1BtQ0VxzuWFcn3g+vLektB56iQ7BBvTy4p/7FV NY9A== X-Gm-Message-State: AKS2vOznYpwB9EQmqPtXRU08z+puHCZsScfzZqp+Jy28MxE5BJH96/j9 JW1mFFcYiy4v9jiKuMXfjLX0HAV+dQPR X-Received: by 10.25.18.154 with SMTP id 26mr8654805lfs.176.1497910970283; Mon, 19 Jun 2017 15:22:50 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.221.150 with HTTP; Mon, 19 Jun 2017 15:22:49 -0700 (PDT) From: Matthew Ahrens Date: Mon, 19 Jun 2017 15:22:49 -0700 Message-ID: Subject: OpenZFS Developer Summit 2017 - announcement & CFP To: "admin@open-zfs.org" , developer , illumos-zfs , zfs-devel@list.zfsonlinux.org, zfs-devel@freebsd.org, freebsd-fs , zfs-discuss Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jun 2017 22:22:52 -0000 The fifth annual OpenZFS Developer Summit will be held in San Francisco, *October 24-25, 2017* (Tuesday-Wednesday). All OpenZFS developers are invited to participate, and registration is now open. See http://www.open-zfs.org/wiki/OpenZFS_Developer_Summit for all of the details, including slides and videos from previous years' conferences. The goal of the event is to foster cross-community discussions of OpenZFS work and to make progress on some of the projects we have proposed. Like last year, the first day will be set aside for presentations. *If you would like to give a talk, submit a proposal via email to admin@open-zfs.org , including a 1-2 paragraph abstract.* I will review all of the submissions and select talks that will be most interesting for the audience. The registration fee will be waived for speakers. The deadlines are as follows: September 4, 2016 All abstracts/proposals submitted to admin@open-zfs.org September 15, 2016 Proposal submitters notified September 26-27, 2016 OpenZFS Developer Summit The second day of the event will be a hackathon, where you will have the opportunity to work with OpenZFS developers that you might not normally sit next to, with the goal of having something, no matter how insignificant, to demo at the end of the day. Please add your hackathon ideas to the summit wiki page. This event is only possible because of the efforts and contributions of the community of developers and companies that sponsor the event. Special thanks to our early Platinum sponsors: Delphix , Datto , and OpenDrives Additional sponsorship opportunities are available. Please see the website for details and send email to admin@open-zfs.org if you have any questions. --matt From owner-freebsd-fs@freebsd.org Mon Jun 19 22:24:53 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7F00FDA60CC for ; Mon, 19 Jun 2017 22:24:53 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x229.google.com (mail-wr0-x229.google.com [IPv6:2a00:1450:400c:c0c::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2EC692123 for ; Mon, 19 Jun 2017 22:24:53 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x229.google.com with SMTP id y25so42708415wrd.2 for ; Mon, 19 Jun 2017 15:24:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=RgIygCDVcxa5esqhSZ4h/lsvaaIFg3umh/y/HLCrmOQ=; b=0pVlmMV6Fms9trZvBJ5gPoWdSJwplsTetyE7qrsp6JpvIaHUS0joY9NW7A4PRMVKwH Clyid9bzzeQcJ++/b1mov2T3i5vNZ2OiAUl6hQtqj+ziq6exwHF4VqZBsOQHhkXBB1Lb Oq7z+DwVvWM+vgy9eYWjfvxLeQAcbLdo2tg+nIDAsmsgkWDr43sov7jbkyFcjPWrr8WE AWMq1pPOS0vtCwSM+JS3xnygYHWITMy9sAzuJVqJk8t3cnA/vKJOcJP+x5Vb0FhLLMnf 6rFXSvOjYcOcgh6gZhruZfMH31IQ4tbXDOdVwx3le3pnDomYWNJ6t4Ee7UjAQZuqNY+z bndA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=RgIygCDVcxa5esqhSZ4h/lsvaaIFg3umh/y/HLCrmOQ=; b=Sno22Wa4jp5hWJ81oSGAXgu4uv9Gbcm1EPSYZOm6+hhfwuKmOyZwTfymk6natgRZwg G1cSZKpeQ7dGgcxg6L56SRbhGxBIjZU4aTbo1AhVb2+uFCIK9+Z0oBrwASpPnsVY50Bg r6YCEc9Azz0JGClsNoNQRpmXjsRIOFAOys8r2CwUHgGvagaQfo1TMdZCde1h5tmjueDm 7qU3wuGrw1MXk6C3RrfiosjrAH6+gVi4fIHHudWWmD2KhX+J3hHBKUR2UPS+n/IBKojc bjfoAeHOitmTPdOsqMDnRv/3aE1pSADeXz0eVbz5SMEzWGQThQmZLT2k43ce+E1u8aou 4ZuQ== X-Gm-Message-State: AKS2vOxNwcUNzCpbjHsOiXUutIGt1KhDE3aHKZTRzM2zMD3zz/smZzO1 GTk7nLV87LnLL5E+N/Ao1A== X-Received: by 10.223.134.157 with SMTP id 29mr842225wrx.157.1497911091028; Mon, 19 Jun 2017 15:24:51 -0700 (PDT) Received: from [10.10.1.111] ([185.97.61.15]) by smtp.gmail.com with ESMTPSA id l17sm5860822wre.25.2017.06.19.15.24.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Jun 2017 15:24:50 -0700 (PDT) Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs To: freebsd-fs@freebsd.org References: From: Steven Hartland Message-ID: <27c6a211-e2d6-2f40-6bb0-469ce7a42bb6@multiplay.co.uk> Date: Mon, 19 Jun 2017 23:24:49 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jun 2017 22:24:53 -0000 vfs.zfs.min_auto_ashift is a sysctl only its not a tuneable, so setting i= t in /boot/loader.conf won't have any effect. There's no need for it to be a tuneable as it only effects vdevs when the= y are created, which an only be done once the system is running. You don't explain why you believe there is degrading performance? What is the exact dd command your running as that can have a huge impact = on performance. On 19/06/2017 23:14, Caza, Aaron wrote: > I've been having a problem with FreeBSD ZFS SSD performance inexplicab= ly degrading after < 24 hours uptime as described in a separate e-mail t= hread. In an effort to get down to basics, I've now performed a ZFS-on-R= oot install of FreeBSD 11.1 Beta 2 amd64 using the default Auto(ZFS) inst= all using the default 4k sector emulation (vfs.zfs.min_auto_ashift=3D12) = setting (no swap, not encrypted). > > Firstly, the vfs.zfs.min_auto_ashift=3D12 is set correctly in the /boot= /loader.conf file, but doesn't appear to work because when I log in and d= o "systctl -a | grep min_auto_ashift" it's set to 9 and not 12 as expecte= d. I tried setting it to vfs.zfs.min_auto_ashift=3D"12" in /boot/loader.= conf but that didn't make any difference so I finally just added it to /e= tc/sysctl.conf where it seems to work. So, something needs to be changed= to make this functionaly work correctly. > > Next, after reboot I was expecting somewhere in the neighborhood of 950= MB/s from the ZFS mirrored zpool of 2 Samsung 850 Pro 256GB SSDs that I'm= using as I was previously seeing this before with my previous FreeBSD 11= -Stable setup which, admittedly, is a different from the way the bsdinsta= ll script does it. However, I'm seeing half that on bootup. > > Performance result: > Starting 'dd' test of large file...please wait > 16000+0 records in > 16000+0 records out > 16777216000 bytes transferred in 37.407043 secs (448504207 bytes/sec) > > Zpool Status: > pool: zroot > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > zroot ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > ada0p2 ONLINE 0 0 0 > ada1p2 ONLINE 0 0 0 > > /boot/loader.conf: > kern.geom.label.disk_ident.enable=3D"0" > kern.geom.label.gptid.enable=3D"0" > vfs.zfs.min_auto_ashift=3D12 > vfs.zfs.arc_min=3D"50M" > vfs.zfs.arc_max=3D"51M" > zfs_load=3D"YES" > > /etc/sysctl.conf: > vfs.zfs.min_auto_ashift=3D12 > > > Is this the expected behavior now in FreeBSD 11.1? > > -- > Aaron > This message may contain confidential and privileged information. If it= has been sent to you in error, please reply to advise the sender of the = error and then immediately delete it. If you are not the intended recipie= nt, do not read, copy, disclose or otherwise use this message. The sender= disclaims any liability for such unauthorized use. PLEASE NOTE that all = incoming e-mails sent to Weatherford e-mail accounts will be archived and= may be scanned by us and/or by external service providers to detect and = prevent threats to our systems, investigate illegal or inappropriate beha= vior, and/or eliminate unsolicited promotional e-mails (spam). This proce= ss could result in deletion of a legitimate e-mail before it is read by i= ts intended recipient at our organization. Moreover, based on the scannin= g results, the full text of e-mails and attachments may be made available= to Weatherford security and other personnel for review and appropriate a= ction. If you have any concerns about this process, > please contact us at dataprivacy@weatherford.com. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Tue Jun 20 00:57:14 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 401AADA7C5E for ; Tue, 20 Jun 2017 00:57:14 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) Received: from NAM01-BN3-obe.outbound.protection.outlook.com (mail-bn3nam01on0059.outbound.protection.outlook.com [104.47.33.59]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A4E3765A0F for ; Tue, 20 Jun 2017 00:57:12 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weatherford.onmicrosoft.com; s=selector1-ca-weatherford-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=fpvYwEwcAa2oAlosDCKnA98KwHNWsniPdHpQswIr7JU=; b=RyQJwYKb4xtIKdnMYcxVaDliCeEN7P0qmm5TsVTl4uJ2ocQ47GEqutSBvYr5UkjtnYPZx7kJutjM+oBMhXwNgoNngmNJAt7VgXED38rlChcR353PgHBMKxqRp363qW6NdoDPD6qppbi8KAkxZlR0NN6/jPJsNadO3lApfBjU8z8= Received: from BN6PR03CA0090.namprd03.prod.outlook.com (10.164.122.156) by CY4PR03MB3272.namprd03.prod.outlook.com (10.171.246.141) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 00:57:10 +0000 Received: from BY2FFO11FD011.protection.gbl (2a01:111:f400:7c0c::152) by BN6PR03CA0090.outlook.office365.com (2603:10b6:405:6f::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14 via Frontend Transport; Tue, 20 Jun 2017 00:57:10 +0000 Authentication-Results: spf=pass (sender IP is 23.103.226.20) smtp.mailfrom=ca.weatherford.com; freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=bestguesspass action=none header.from=ca.weatherford.com; Received-SPF: Pass (protection.outlook.com: domain of ca.weatherford.com designates 23.103.226.20 as permitted sender) receiver=protection.outlook.com; client-ip=23.103.226.20; helo=032-smtp-out.weatherford.com; Received: from 032-smtp-out.weatherford.com (23.103.226.20) by BY2FFO11FD011.mail.protection.outlook.com (10.1.14.129) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1157.12 via Frontend Transport; Tue, 20 Jun 2017 00:57:10 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net (141.251.110.81) by DM2PR58MB016.032d.mgd.msft.net (141.251.110.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 00:57:08 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) by DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) with mapi id 15.01.1178.018; Tue, 20 Jun 2017 00:57:08 +0000 From: "Caza, Aaron" To: "freebsd-fs@freebsd.org" Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Topic: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Index: AdLpVvY5oAKJNs3ZTx6kAWjOND0OCw== Date: Tue, 20 Jun 2017 00:57:08 +0000 Message-ID: <431d958c658a408d8bfd4c574a565439@DM2PR58MB013.032d.mgd.msft.net> Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [141.251.110.69] x-ms-publictraffictype: Email X-MS-TrafficTypeDiagnostic: DM2PR58MB016:|CY4PR03MB3272: X-MS-Office365-Filtering-Correlation-Id: 30af2779-b18a-49a8-0e46-08d4b77747e9 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OrganizationHeadersPreserved: DM2PR58MB016.032d.mgd.msft.net X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:23.103.226.20; IPV:NLI; CTRY:; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(39400400002)(39850400002)(39840400002)(39860400002)(39450400003)(39410400002)(2980300002)(438002)(189002)(40224003)(13734003)(24454002)(199003)(9170700003)(356003)(46406003)(2351001)(478600001)(6116002)(966005)(106466001)(3846002)(50466002)(33646002)(23726003)(110136004)(8676002)(8936002)(53936002)(50986999)(54356999)(72206003)(305945005)(189998001)(81166006)(6916009)(42882006)(53546009)(86146001)(38730400002)(8746002)(86362001)(229853002)(97756001)(2501003)(575784001)(5640700003)(55016002)(47776003)(5890100001)(2900100001)(6306002)(22756006)(2906002)(5660300001)(24736003)(7696004)(66066001)(9686003)(299355004); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR03MB3272; H:032-smtp-out.weatherford.com; FPR:; SPF:Pass; MLV:sfv; MX:1; A:0; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD011; 1:Rlx9+kV9VXe31cLaLxoM1KoKBhGYBbbIr1SJGXp7toM/BUK+f+d7TG5Wr2Klg0BxHuhsVD5QkjYT+kd0rj+BitrZc2Hfss09IHok005UbTXfAmxdd5+tjepgvNFvML7tEPgl98g7N2UnSwFakTSmqh5JPfwGT5JLb2svojVwj/AL9j6LLQ4QKH67m1w0S8c9mb5XnDnUMZEWEaNUlqURXNnzSrTQQbDyxOEHUDf/lZOuQ1fAJbBWlcFq+EYyWEUzh/0DgYJMqiKEyVz7ljl9uqNHM2nSDWzIUouz6C0hznAcIE+4F6tLEqFedBTXx49kL+6FldcI89ftrD6KH5zg5biprGWImEUJ2cuXbvEUCjhw38M0LsAHl7g/aZOoJDpDDoWwD4fgRtsEcOStVMlVoZpJ6K4H/+zuExtfaiRvPRIBPd9vaAiqjalfZbfev5PktJreTbjf4yxasMcz/m9C/476Bw4OeV8ZjkpSMaJnd+zqU4s4ObKEv/gmO4+fXzJVe5atMQSOMUFpwZ2ubrRMoQ== X-CrossPremisesHeadersPromoted: BY2FFO11FD011.protection.gbl X-CrossPremisesHeadersFiltered: BY2FFO11FD011.protection.gbl X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(8251501002)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:CY4PR03MB3272; X-Microsoft-Exchange-Diagnostics: 1; CY4PR03MB3272; 3:XARSO3q55PxVDwqlN4zI0U+IS1jvc+HRQ8eSONfgrs5dCBdZMHuqAV3fmH6jF6hBQ/g/WOTFfrI1HSeFEOBa/9D/RSbs/S+ugLYSq6VTQIyrYwDbHUggPteNht5ShxpmteZjrhoHnEU06mQ8baP4gGUCq8+BBTAZHlCqPmZ5O+MA8wXyuyxIzRm02fNg3e5p4ijVYS/H7al1mpBIIC1Fwc2Dav42NLsbqqE5C8aTP8dHyJu7R1kUzhkeG/c7x24l6ZtE70swVRgwxNPZ5HGJ3eN9HqZshAi3MkGZZ7w9VoM5+jJERhsuTfDTzPLhcU4Q2Y9CC5ZenA19cpvbTgDfn5nFsHw+UGaw0fbGBHHvu4ZV4PreG6uYcDA/RAmD36d6Q6TQpgWSIbnOc/yT17D6opoRJXtihIP+bIrkfZkzztAvKIun7uEhRNcLW5p1HOibvgRq++//BtH3ZxRDDX52gHX1+RyVjkC/ic0sRcF9DaIDLO9uU//S1zadDMKqBoMdPOmtXlR62b6ePtUb2OqH3Q== X-Microsoft-Exchange-Diagnostics: 1; CY4PR03MB3272; 25:wKWWx6wMavz9CnvDXRc3cO4oDiKQvmwmrNqzG8dm9J3+2220ZZXnoz5ilKxwynAsGmI1CiBPDkgL4s/Ioj5zvA2yfUeNKfsEU/hTUOQomjpVT0nybW+hzSBcjX0d/6Y/68VbaCzVj9xQViRjzDAMP3ar3VfZLuM6ElOrHW5j80AHz6wozEii1QA+V6wwDRPJIBoQUucxFZXg3lOAzMosl0Xzmwp6kDQ7JL0PP2kPC8FtLzEXdf0NO9FnF84yBVrJBhngUdg54V0jkDbihBzPW29N66fLUUbbyv1jpna1r+Sx9QQ6fCJwdYdQGGgT/KBCg0xcwqDLK6HIqmA66UVju+u30LiIPudWO0WRvw8KiS9T15m9J+QNb1ncRisl1pn1/pdYXz8XSY0Dp4LZAGjsaQHcOlCrbD/zkBJgopSAJNsUN0qk2zBakqqjd8nJdCjEbFZouZ9vWqNj0Xs2JnFPSIkqAjj1PjX2WKQ83wZKLVE=; 31:GBSJF2ojp1lGa+bjuzN9lz0V4BrTU4FAxH7hZq2C1SX1NYxk4NOXu7nk+ovMHo9M/rq3fsxzHcs66x7qwRP+ZLDcuWCKxDQZVSWP1UncQA5cVjZf7UPda1DqOyERHyX7qdNifss0+5P+NzAhsgKbzsxkm9Ads3fNRp66ySbcnVlOKSGoz85yUJzJKlwTcmWE/t41SDlQG3hKzAqOXB79sb9EH5maeji+84uTF7rYU4DbgNSZO8RTp/Y0b/R1sJRXbUjdSS0VPKuuOXG7CAeS0A== X-Microsoft-Exchange-Diagnostics: 1; CY4PR03MB3272; 20:7+tfuhmVWc92nowtr7o35GPnvWVRKhbIvA6pfef8nbXL4ybIx/t3UoTvTqeBr0P8Uf45DFg9vq4FIyAmoIfBGW5iAqb/f/SmoOHPYmm7BTyjO7fzRDqVljXsIThVqOPtVU/I/wt6HQmWSGPlJbZTG4bVaKKHd3cfW+ZV4SFMryr9TThkTetcSEHKPuytSPyQ0XXQR4luhHH1MQr/9KxwUONKDuloSO2lQHUtx02UYXDDI4x1jY/07oWxAWJGHTiMpkcz3K/4ifl5L2Y7iaVgST1iJuF5EYhZBa7rByrZNmja/hCOnd7uqBm7oe4tbZLWUNrY82ZzyGloUGxfismc7JiDg1SmtXTCXwW+SnNxj4mOeaE2bQkB92RgifNwFLu6V6hLdv+f2FgaullSYWjDc2K5vpo2xxN6nSTlVRYCLsVxvxoUm9khPdq7L7moZaBWrWdRqueCHmvAPD7KubXcAyUW8q9e/Tw9v74ceCoMvTtwYDtcESG+J6kGQF90IjGy X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(158342451672863)(209352067349851)(192374486261705); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(11241501159)(6040450)(601004)(2401047)(13018025)(8121501046)(13016025)(5005006)(10201501046)(100000703101)(100105400095)(93006095)(93004095)(3002001)(6055026)(6041248)(20161123562025)(20161123564025)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123560025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:CY4PR03MB3272; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:CY4PR03MB3272; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR03MB3272; 4:ebdfKt0AyhBawDMV77UJZOOJkSVBXL5en7CW9gFspQ?= =?us-ascii?Q?aqg/L1L34qcMA2iYV+zEo6Mu1dTG9vja/tLWBf2cBs0FrqUobxAeMOtZc6WJ?= =?us-ascii?Q?cdhkgz9d2meA+Ljhsmfgu7IEwFR9iO6NWE2/7lGbm+P+FGUj+w7sraW+dCUc?= =?us-ascii?Q?h6Rbw6j/kZuI3z/+HKJEpg3cF2fotTBV124ElqANOvFq89DKdjRWlb6SChqd?= =?us-ascii?Q?B6NySzuHJr7/gDrHtDd5OXajQD6Q/u40SpaGO8BgBZL5wNgdBRbM+c6Fhksw?= =?us-ascii?Q?tOxlOh74SCmrGcOoagmWfc41J7V5ctgOYRR0PZv6i0hFoD0x5OMQT9T1UWpg?= =?us-ascii?Q?Uc9XLG1M9YujNAOWThmbsO0ExqbtAm5byboFrpA4xFuPnAcGPKEhUIkI0A6o?= =?us-ascii?Q?d2s+5Xjb3sWG+nfkEZ8VNCKE4eWpx6IkJ1/bebQKN4hVgdusVZxxweV0ZbDX?= =?us-ascii?Q?39tXCYnKnAXgjWZ0OioM8xvASi4QtBsPWui/sCIGbS4jeeL06hubVpOrAyll?= =?us-ascii?Q?1I6JWwiQ3wBGWJi6+YFlI0exrfSnuO7BsYYJUpGdMoZWr18hTgL7N1GPtI4F?= =?us-ascii?Q?n+12rB9kosPRJc+gpoYsmk/waJoBqVWstcsO6PfGWcPivBQxn862TqHG2nuO?= =?us-ascii?Q?Hk8wf0f/MiqRNnDiW+Y2jrNpJgl8YyWCjzAE+h29tW8jBIzWfhZz+TnD7n/P?= =?us-ascii?Q?rJuwXWJJf6QDXIAn0J9OSLKxondU1T0UNpE1dlam2v6wbjLKYT8gkUafcXKg?= =?us-ascii?Q?XVzR0q+cw+KeD0NVWiNgapCOTTWlBxEakp8ZB1mXVLtFOehS+264/wvHMI85?= =?us-ascii?Q?fwEGXZs7y/qnxxyhhnUpWDDX4Ef8ai302fbDG2rM0Hn31q77eHVlQitDoPYr?= =?us-ascii?Q?Q74MSduhwxPme/7JNHbg9VO9rwgddqmHl8pyVdEZqi0wFxFBv/+T51QJorCX?= =?us-ascii?Q?I4UtSzB6xBqPtS5x2n1PmmwMfr2Y5sxsI9IX13xkrkb/zfViH0W0g9DRHrdY?= =?us-ascii?Q?RmriOFnaV82Dns0mlbaIRRfuKRDkO6uyulRKrX1/tt5o1S2egaUmzoiIKe10?= =?us-ascii?Q?Ya9ATjvh28WzJ3NHlgbi5uhGZYvFrdnSejB+hrDCn/FQOsFvryS09py3X58n?= =?us-ascii?Q?kkSk96wiHDbdQtdBfloBFjV484jI1rczLE6GJJaN6pzShBj1nUgiKWokpQ0G?= =?us-ascii?Q?2Xdtu3Ezf5GccSIJEhCTASPf54qWFXacdhEkRbAE0j902emE0nJ4GUR1Se64?= =?us-ascii?Q?IzOtp+ZOV1txKNVI72R/cPuUZA6EEaJ0041K2rsM/twuICRkGIvGfjWsL/x9?= =?us-ascii?Q?vXh2wzxHA2qxHP1AfIcBmDWMB/KQJjW0eRH/Pi6A/WyfRxiYpt1oZr4qIUSF?= =?us-ascii?Q?UgDw=3D=3D?= X-Forefront-PRVS: 03449D5DD1 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR03MB3272; 23:/7Axl30SDG4N6Ibzp8mMHrFQuVv5BhR8Z8fyATPlI?= =?us-ascii?Q?LK7FlFNRsS0x44uvhqoUFx+JTc53SNNQLX3xRZO7bUHuwybjiG/ti8HqsHpE?= =?us-ascii?Q?VXFLnyMSh0nMkYiTQs88BDms00rszwdB2p4GNVuJcsk0Wub4GCnrpeOWRdqx?= =?us-ascii?Q?r0UnEAV7DSd434ZC5RsEp+we0Ro66RZWTmH3zYVcPFa7Nmdy/DbwIQL5m4DT?= =?us-ascii?Q?dWRpkh5Vdb0VpNwHelX05XhtQ2D9tKKp6bi7njVvZ+v8Mdxhcmk5HOKMXCVe?= =?us-ascii?Q?J6jDAdwtNnbxvGAcknORWEORJAhR6fAkmFGAfc58S71rIf9cEMEUn2B0UPYc?= =?us-ascii?Q?fFZUrxZPpWnGgfWH4o0bvA3WC0T4REgPPo9hzjSCNwOlBRokYEJ+J1Y2l5Xw?= =?us-ascii?Q?dUgi3CH+TlZUzSX+CFH3wFjWmwBVybnHssx44TSNjDO2fRfSmUesi0mVGxMg?= =?us-ascii?Q?aair5VkLGvAghvUJ674Xw1YugSOsrFvuR4It5okrXEq2iS/aKQPqyz+E7rMl?= =?us-ascii?Q?VCsCUwaCL6YShZ2LQppy1tdHzpTAxm3wXCpz3IJC+65elc7Ox4DUqhApk0bU?= =?us-ascii?Q?UnRkfm+vmomM/fzR6yXypli98WNaxRJLg5kMTG9jWTzdkHxjgDVvF7cau25X?= =?us-ascii?Q?lV8iWSymhFzt1dUYBzmOK2mO1t+KO18SjbVOrPvpSEdxeyn613fIOk8ewL/z?= =?us-ascii?Q?b8G+jPVXO8y3MYckQXh5yYypJHqxWn+b/+rgHusV9efjVzWLTdkEk1ClyF4t?= =?us-ascii?Q?x6zVZ7Xn82j/+i2VafNWLG103ABX47o3/hvamyPa0OaLvXcMC/yuU0qI1ZTZ?= =?us-ascii?Q?IbHcbhUL3vJtpsqpi/tqUi/yKsRft+t+Ay8ojQV86cR/G+hOOBQyo+wFj0Jd?= =?us-ascii?Q?o5LUNtfH1t4fZ7/CPgloLHIx+emr9epztm0aTwEfaPefWPx4M/RpQYzxoDNk?= =?us-ascii?Q?efVMgxryOatZLVr/3bQ/GQUzNDjpb/4D5jkLwD0qQB1BIe4wOZbD8JO1DZC5?= =?us-ascii?Q?gW79Q+/H+X2UKMBsqybJNQgL0jJJSlcMDvDJwbkfqLmN0ixiN4X9ytcMsUYv?= =?us-ascii?Q?teFvemcdUq+TrUL2Qgpfise/41Z6BWagdnMiI9DJv8Syd70sVtVfO1RIQAwY?= =?us-ascii?Q?9S+L3pXoTtt7NerLC9ardShcgpfps5wei8N66ppcD2C014vfn0BAPrSl5EKL?= =?us-ascii?Q?11TZMblASjof/a5mllqdpz4GDkdHtegelB+PSw6EXPdcVm8UShYcv1EvPkZ1?= =?us-ascii?Q?IuReoy8b8zn3PlkbTvGWHbXvkSpyfSJK+1iyOUE9m07IdIiDtCVtcZUdva1v?= =?us-ascii?Q?NYkykUvPmdfgoBvaLreEoADzPbjVvk0PpViAJ1IhFS5KiyVzDj4NLJLDCrzM?= =?us-ascii?Q?7JzXdtLioMBhUNWtb8WaEZQ0Zx5X0dvA3+GlTKy6yxyhozxw1k33xT+On7w4?= =?us-ascii?Q?NqJSdBqsZc9dsAEQneqsYSTP4UpzAlRW54j8fOFaFy/SiK30GK+?= X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR03MB3272; 6:nF8pfRA20ybPkiP3JRjhylHm7n4hV7I9srgXybMk5v?= =?us-ascii?Q?HeBxW5WwS3a3m3QrkFZOp/XbG+NW/uXD2uEKVxAZz7llpes5OPICKm70pJiO?= =?us-ascii?Q?Tjpix0Wl5lvqzODKTbKzm3zSA+qve3KRGkpI4j33Y2ZYSGZ0RXZbRlTUshxv?= =?us-ascii?Q?QuwDWo3Z0olp3ovc/geAk5TQe8vXLhfCsSXeJeS8SRRwNLMt0MSZwJIxnC+V?= =?us-ascii?Q?iCdI/d2sSB+kvQv/BMHRX8koCeEUl/Yjlf4ybnR5tHhUtBoPp5bNxmbLYj7F?= =?us-ascii?Q?+yEdjMbxzZHBixY8TjTFoL5bPX2EW2Z2X9e/ESQAQnWS4xRX3S4IOEWjGb16?= =?us-ascii?Q?NCiXsl3J5Gno+Ve6myu32xIImNFoIHqXVZ1jS5BIL0uHkRHXWACEl4W79K7N?= =?us-ascii?Q?LPNw9phQ1O3e31QkLDVkn6vyGZOrmbMHNvGhukUUV1PK2R0ErFb3FY2a6Lhq?= =?us-ascii?Q?FlOEqb94l4t6EvP+PHeecQn3M0opfGqSGdBt9FGYnr2m14Rglz59ZOBG3F6F?= =?us-ascii?Q?SCMgSnF+vEH+PVroKYrxGoRkXSLh26Bbel5kAO/LTbojg0g2XpLXknYDP+N+?= =?us-ascii?Q?njMMRK2Aj5kJau5mmPdg4E0HbSCim9Ree1ui56pTcZZ7RBHrZtzrv05sS6Ed?= =?us-ascii?Q?fXtTXSJvSieY/zgyrj3jJ2A0kjKjTSe+eJcJ2hPSo3XBlPaNxjzrkxbtlbjk?= =?us-ascii?Q?ytXHZuVtuEU2C9W6nqG4obtxPzYLFFIXGvteFuajjuGZOjxMdei+tIR/oEsW?= =?us-ascii?Q?w5teOppiUX3V1nX1Bd/LCPEWbQ5RTY4ALkU/LQ+oJn+ksJQG4gMfPqfgz9UE?= =?us-ascii?Q?5fM1AeEKZdXioPk2ZhA7OA1+qoQAT/0DdsnvMqYS9326JOIloatsCG/ccSr1?= =?us-ascii?Q?xxocxctFJwTuenk/glprhvaMtxJUd8XhofwDzGzRyMCF+Tx2qlRUWzSzH4tt?= =?us-ascii?Q?Cu/viiwQWT+aofyexKIuaStpGcZvOeartrzeEkirEVNk8kO1Dq9wBBpOxYoE?= =?us-ascii?Q?9V1k5jO7NFkTmO4AuXEcfA?= X-Microsoft-Exchange-Diagnostics: 1; CY4PR03MB3272; 5:mfCYTROCn90V2Izvnm1yAcQVG3tc68LtfwzCuhBRXawGsp1CALYsA8/LMIsNYd3K1BX9ldITZNZkmU2cNkbkTa6cH9CPtflNr7EfZ3q7zTw3R+DCzjTSLo5/+E0o9ZoLTkVi2vNfCtpVZSsIoWG2jzRV8ZqdeyGY2UcSoYvq+yYCxy+Zzara5ARYxPyq2f0j3S2s8qYggitqykBKNE60fLylFwqP4/ERuvpVPgjnTWIWmP7TRqJ5K/sSzQbmyO/ukgobfN0v0Dl56gk/tFSyWXLKkw/TLxSUxzkaPACjywhVB6heDzjy+zBwtdxUUY7osFHltOreU7K6xs/W9F55lclULqgZqoHIU0lTq9zoz63abHdMkr8tfpG4T/q9Iw6ITqRwrlAx0A6HXDKUKBVg426sZYUuW+iEL2JgkkYYjoIrfT/qmQwkHm0EFSvO9M+cLGohzb2kQ8THb+mM265BEt+1V49NrVVdMNSx7rEi/pUr2bTONNUwN3qNSt3/UVN1; 24:KuHCDjceQW1Zerr0aDV07u0fdma4oUGhmTtCaQFIxX2wtv3X2fcS6MXnZ32AC7JMuL9QQUa2tTwpAVXonF9MB2j1bnacM3xSLxeEGjcdZf4= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; CY4PR03MB3272; 7:jPnyzrAKN3YCqupnXB3EX1ebFfl/mwRnqYyuH7J13TaK3snaWKpjrkoGyliqp3qncedHiBeHGHM6PvfgHRGuh0746xSuoFtuj6C4EIpmCjoRUODF2X3wI+sFxrq4TYxvJvj8x/YWSpqa9Sl4FhC1LeY/SnWZ7IgR2vZue0TAmoaJyEJP0ksCGlXNuXqCRvQnaZyk3uiA3mqvQx8tvxyKMyr1cHg962IcIEyRYhF5XAaz8ghWFLxUgvASZxZfi39fS/QCk4arA43l0i6TgISbOjmo8NJO/YNHOJ6JPsLD0aKaqi6GI/llbU/lpAwjuX1JadwlkkeVdB81Yds4h5KEzdxgUccR8lNPyLmzlUoKk1bm5hFTYOz8BffzGGagykGHQaOjufogd85y2MZ/bR9PxhFtK8ltt6RM7yGXS04X5Xavh8U/StPL74PBVWauPVyG57XsZTf2KrvyFYzH8gKtL1rRE/cgbZP4aK77/2htd7P9UUzJoPXTT3aAp+vm6v8HR0KcLAvILU644T3/q9NqUp/osJVgkWX8ykM7gF4gYFki44LqhGcgFWT6d3u/ksmghsHt9m6rVGjik3lsJa7jaYQqidbwYb6kDQtUbwZkJ5XKHh/V+eg1K762GVxePys+4FdZucrFdA5p+jSeuqmActYgPwF5dFK0xyhJihBhPFKLy0kjtNpA7n6Ym0QjswMdZbNHiHfIkKrVpygFgl81wqTkcqm0l6dvtb39N+bQb+ikZ4x0adptmMGQa6V3xzbe+EW80LxDnEdZpQbK77bANAAi5zZglNmSJPjUYBZCCxY= X-OriginatorOrg: ca.weatherford.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jun 2017 00:57:10.0593 (UTC) X-MS-Exchange-CrossTenant-Id: dd63fb60-07f6-4d96-8d40-ebeca61a524e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=dd63fb60-07f6-4d96-8d40-ebeca61a524e; Ip=[23.103.226.20]; Helo=[032-smtp-out.weatherford.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR03MB3272 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 00:57:14 -0000 > vfs.zfs.min_auto_ashift is a sysctl only its not a tuneable, so setting i= t in /boot/loader.conf won't have any effect. > > There's no need for it to be a tuneable as it only effects vdevs when the= y are created, which an only be done once the system is running. > The bsdinstall install script itself set vfs.zfs.min_auto_shift=3D12 in /bo= ot/loader.conf yet, as you say, this doesn't do anything. As a user, this = is a bit confusing to see it in /boot/loader.conf but do a 'sysctl -a | gre= p min_auto_ashift' and see 'vfs.zfs.min_auto_ashift: 9' so felt it was wort= h mentioning. > You don't explain why you believe there is degrading performance? As I related in my post, my previous FreeBSD 11-Stable setup using this sam= e hardware, I was seeing 950MB/s after bootup. I've been posting to the fr= eebsd-hackers list, but have moved to freebsd-fs list as this seemingly has= something to do with FreeBSD+ZFS behavior and user Jov had previously cros= s-posted to this list for me: https://docs.freebsd.org/cgi/getmsg.cgi?fetch=3D2905+0+archive/2017/freebsd= -fs/20170618.freebsd-fs I've been using FreeBSD+ZFS ever since FreeBSD 9.0, admittedly, with a diff= erent zpool layout which is essentially as follows: adaXp1 - gptboot loader adaXp2 - 1GB UFS partition adaXp3 - UFS with UUID labeled partition hosting a GEOM ELI layer using= NULL encryption to emulate 4k sectors (done before ashift was an option) So, adaXp3 would show up as something like the following: /dev/gpt/b62feb20-554b-11e7-989b-000bab332ee8 /dev/gpt/b62feb20-554b-11e7-989b-000bab332ee8.eli Then, the zpool mirrored pair would be something like the following: pool: wwbase state: ONLINE scan: none requested config: NAME STATE READ WR= ITE CKSUM wwbase ONLINE 0 = 0 0 mirror-0 ONLINE 0 = 0 0 gpt/b62feb20-554b-11e7-989b-000bab332ee8.eli ONLINE 0 = 0 0 gpt/4c596d40-554c-11e7-beb1-002590766b41.eli ONLINE 0 = 0 0 Using the above zpool configuration on this same hardware on FreeBSD 11-Sta= ble, I was seeing read speeds of 950MB/s using dd (dd if=3D/testdb/test of= =3D/dev/null bs=3D1m). However, after anywhere from 5 to 24 hours, perform= ance would degrade down to less than 100MB/s for unknown reasons - server w= as essentially idle so it's a mystery to me why this occurs. I'm seeing th= is behavior on FreeBSD 10.3R amd64 up through FreeBSD11.0 Stable. As I was= n't making any headway in resolving this, I opted today to use the FreeBSD1= 1.1 Beta 2 memstick image to create a basic FreeBSD 11.1 Beta 2 amd64 Auto(= ZFS) installation to see if this would resolve the original issue I was hav= ing as I would be using ZFS-on-root and vfs.zfs.min_auto_ashift=3D12 instea= d of my own emulation as described above. However, instead of seeing the 9= 50MB/s that I expected - which it what I see it with my alternative emulati= on - I'm seeing 450MB/s. I've yet to determine if this zpool setup as done= by the bsdinstall script will suffer from the original performance degrada= tion I observed. > What is the exact dd command your running as that can have a huge impact = on performance. dd if=3D/testdb/test of=3D/dev/null bs=3D1m Note that file /testdb/test is 16GB, twice the size of ram available in thi= s system. The /testdb directory is a ZFS file system with recordsize=3D8k,= chosen as ultimately it's intended to host a PostgreSQL database which use= s an 8k page size. My understanding is that a ZFS mirrored pool with two drives can read from = both drives at the same time hence double the speed. This is what I've act= ually observed ever since I first started using this in FreeBSD 9.0 with th= e GEOM ELI 4k sector emulation. This is actually my first time using FreeB= SD's native installer's Auto(ZFS) setup with 4k sectors emulated using vfs.= zfs.min_auto_ashift=3D12. As it's a ZFS mirrored pool, I still expected it= to be able to read at double-speed as it does with the GEOM ELI 4k sector = emulation; however, it does not. /var/run/dmesg.boot: Copyright (c) 1992-2017 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.1-BETA2 #0 r319993: Fri Jun 16 02:32:38 UTC 2017 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM = 4.0.0) VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E31240 @ 3.30GHz (3292.60-MHz K8-class CPU) Origin=3D"GenuineIntel" Id=3D0x206a7 Family=3D0x6 Model=3D0x2a Steppi= ng=3D7 Features=3D0xbfebfbff Features2=3D0x1dbae3ff AMD Features=3D0x28100800 AMD Features2=3D0x1 XSAVE Features=3D0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory =3D 8589934592 (8192 MB) avail memory =3D 8232431616 (7851 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads random: unblocking device. ioapic0 irqs 0-23 on motherboard SMP: AP CPU #1 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #5 Launched! Timecounter "TSC-low" frequency 1646297938 Hz quality 1000 random: entropy device external interface kbd1 at kbdmux0 netmap: loaded module module_register_init: MOD_LOAD (vesa, 0xffffffff80f5a190, 0) error 19 nexus0 vtvga0: on motherboard cryptosoft0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 550 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pcib0: _OSC returned error 0x10 pci0: on pcib0 em0: port 0xf020-0xf03f mem = 0xfba00000-0xfba1ffff,0xfba24000-0xfba24fff irq 20 at device 25.0 on pci0 em0: Using an MSI interrupt em0: Ethernet address: 00:25:90:76:6b:41 em0: netmap queues/slots: TX 1/1024, RX 1/1024 ehci0: mem 0xfba23000-0xfba233ff ir= q 16 at device 26.0 on pci0 usbus0: EHCI version 1.0 usbus0 on ehci0 usbus0: 480Mbps High Speed USB v2.0 pcib1: irq 17 at device 28.0 on pci0 pci1: on pcib1 pcib2: irq 17 at device 28.4 on pci0 pci2: on pcib2 em1: port 0xe000-0xe01f mem = 0xfb900000-0xfb91ffff,0xfb920000-0xfb923fff irq 16 at device 0.0 on pci2 em1: Using MSIX interrupts with 3 vectors em1: Ethernet address: 00:25:90:76:6b:40 em1: netmap queues/slots: TX 1/1024, RX 1/1024 ehci1: mem 0xfba22000-0xfba223ff ir= q 23 at device 29.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci1 usbus1: 480Mbps High Speed USB v2.0 pcib3: at device 30.0 on pci0 pci3: on pcib3 vgapci0: mem 0xfe000000-0xfe7fffff,0xfb800000-0xfb= 803fff,0xfb000000-0xfb7fffff irq 23 at device 3.0 on pci3 vgapci0: Boot video device isab0: at device 31.0 on pci0 isa0: on isab0 ahci0: port 0xf070-0xf077,0xf060-= 0xf063,0xf050-0xf057,0xf040-0xf043,0xf000-0xf01f mem 0xfba21000-0xfba217ff = irq 19 at device 31.2 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahciem0: on ahci0 acpi_button0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse Explorer, device ID 4 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0 ppc0: cannot reserve I/O port range est0: on cpu0 est1: on cpu1 est2: on cpu2 est3: on cpu3 est4: on cpu4 est5: on cpu5 est6: on cpu6 est7: on cpu7 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec nvme cam probe device init ugen0.1: at usbus0 ugen1.1: at usbus1 uhub0: on usbus0 uhub1: on usbus1 ses0 at ahciem0 bus 0 scbus2 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ACS-2 ATA SATA 3.x device ada0: Serial Number S39KNB0HB00482Y ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) ada0: Command Queueing enabled ada0: 244198MB (500118192 512 byte sectors) ada0: quirks=3D0x3<4K,NCQ_TRIM_BROKEN> ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada1: ACS-2 ATA SATA 3.x device ada1: Serial Number S39KNB0HB00473Z ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) ada1: Command Queueing enabled ada1: 244198MB (500118192 512 byte sectors) ada1: quirks=3D0x3<4K,NCQ_TRIM_BROKEN> Trying to mount root from zfs:zroot/ROOT/default []... Root mount waiting for: usbus1 usbus0 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered Root mount waiting for: usbus1 usbus0 ugen0.2: at usbus0 uhub2 on uhub0 uhub2: on = usbus0 ugen1.2: at usbus1 uhub3 on uhub1 uhub3: on = usbus1 Root mount waiting for: usbus1 usbus0 uhub2: 6 ports with 6 removable, self powered uhub3: 6 ports with 6 removable, self powered ugen1.3: at usbus1 umodem0 on uhub3 umodem0: on usbus1 umodem0: data interface 1, has CM over data, has break > On 19/06/2017 23:14, Caza, Aaron wrote: > > I've been having a problem with FreeBSD ZFS SSD performance inexplicab= ly degrading after < 24 hours uptime as described in a separate e-mail thr= ead. In an effort to get down to basics, I've now performed a ZFS-on-Root = install of FreeBSD 11.1 Beta 2 amd64 using the default Auto(ZFS) install us= ing the default 4k sector emulation (vfs.zfs.min_auto_ashift=3D3D12) settin= g (no swap, not encrypted). > > > > Firstly, the vfs.zfs.min_auto_ashift=3D3D12 is set correctly in the /bo= ot=3D/loader.conf file, but doesn't appear to work because when I log in an= d do "systctl -a | grep min_auto_ashift" it's set to 9 and not 12 as expect= ed. I tried setting it to vfs.zfs.min_auto_ashift=3D3D"12" in /boot/loader= .conf but that didn't make any difference so I finally just added it to /et= c/sysctl.conf where it seems to work. So, something needs to be changed to= make this functionaly work correctly. > > > > Next, after reboot I was expecting somewhere in the neighborhood of 950= MB/s from the ZFS mirrored zpool of 2 Samsung 850 Pro 256GB SSDs that I'm u= sing as I was previously seeing this before with my previous FreeBSD 11-Sta= ble setup which, admittedly, is a different from the way the bsdinstall scr= ipt does it. However, I'm seeing half that on bootup. > > > > Performance result: > > Starting 'dd' test of large file...please wait > > 16000+0 records in > > 16000+0 records out > > 16777216000 bytes transferred in 37.407043 secs (448504207 bytes/sec) > > > > Zpool Status: > > pool: zroot > > state: ONLINE > > scan: none requested > > config: > > > > NAME STATE READ WRITE CKSUM > > zroot ONLINE 0 0 0 > > mirror-0 ONLINE 0 0 0 > > ada0p2 ONLINE 0 0 0 > > ada1p2 ONLINE 0 0 0 > > > > /boot/loader.conf: > > kern.geom.label.disk_ident.enable=3D3D"0" > > kern.geom.label.gptid.enable=3D3D"0" > > vfs.zfs.min_auto_ashift=3D3D12 > > vfs.zfs.arc_min=3D3D"50M" > > vfs.zfs.arc_max=3D3D"51M" > > zfs_load=3D3D"YES" > > > > /etc/sysctl.conf: > > vfs.zfs.min_auto_ashift=3D3D12 > > > > > > Is this the expected behavior now in FreeBSD 11.1? > > > > -- > > Aaron -- Aaron This message may contain confidential and privileged information. If it has= been sent to you in error, please reply to advise the sender of the error = and then immediately delete it. If you are not the intended recipient, do n= ot read, copy, disclose or otherwise use this message. The sender disclaims= any liability for such unauthorized use. PLEASE NOTE that all incoming e-m= ails sent to Weatherford e-mail accounts will be archived and may be scanne= d by us and/or by external service providers to detect and prevent threats = to our systems, investigate illegal or inappropriate behavior, and/or elimi= nate unsolicited promotional e-mails (spam). This process could result in d= eletion of a legitimate e-mail before it is read by its intended recipient = at our organization. Moreover, based on the scanning results, the full text= of e-mails and attachments may be made available to Weatherford security a= nd other personnel for review and appropriate action. If you have any conce= rns about this process, please contact us at dataprivacy@weatherford.com. From owner-freebsd-fs@freebsd.org Tue Jun 20 01:28:07 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DCBDFD87A09 for ; Tue, 20 Jun 2017 01:28:07 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [67.205.158.196]) by mx1.freebsd.org (Postfix) with ESMTP id 9964F674AE for ; Tue, 20 Jun 2017 01:28:07 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id E7986274D0 for ; Mon, 19 Jun 2017 21:28:02 -0400 (EDT) Received: from [192.168.10.20] (D10.Denninger.Net [192.168.10.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id D702660360 for ; Mon, 19 Jun 2017 20:27:54 -0500 (CDT) Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs To: freebsd-fs@freebsd.org References: <431d958c658a408d8bfd4c574a565439@DM2PR58MB013.032d.mgd.msft.net> From: Karl Denninger Message-ID: <5033585e-591e-99b5-c14e-0c0fb70f63fb@denninger.net> Date: Mon, 19 Jun 2017 20:27:50 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: <431d958c658a408d8bfd4c574a565439@DM2PR58MB013.032d.mgd.msft.net> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms060306060902090607050007" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 01:28:08 -0000 This is a cryptographically signed message in MIME format. --------------ms060306060902090607050007 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Just one note below... On 6/19/2017 19:57, Caza, Aaron wrote: > > Note that file /testdb/test is 16GB, twice the size of ram available in= this system. The /testdb directory is a ZFS file system with recordsize= =3D8k, chosen as ultimately it's intended to host a PostgreSQL database w= hich uses an 8k page size. Do not make this assumption blindly. Yes, I know the docs say to set recordsize=3D8k but this is something you need to benchmark against your actual working data set. MANY Postgres workloads are MUCH faster (2x or more!) if you use a default page size and lz4 compression -- including one I have in production and have extensively benchmarked. The difference is NOT small= =2E =2E... zroot/ticker compressratio 1.53x - zroot/ticker mounted yes - zroot/ticker quota none default= zroot/ticker reservation none default= zroot/ticker recordsize 128K default= zroot/ticker mountpoint /usr/local/pgsql/data-ticker local zroot/ticker sharenfs off default= zroot/ticker checksum fletcher4 =20 inherited from zroot zroot/ticker compression lz4 =20 inherited from zroot zroot/ticker atime off =20 inherited from zroot You may also want to consider setting logbias=3Dthroughput. In some case= s the improvement there can be quite material as well -- depending on the insert/update traffic to the database in question. --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms060306060902090607050007 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC BlwwggZYMIIEQKADAgECAgE9MA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE2MTIxODE5NDUzNVoXDTIxMTIxNzE5NDUzNVowVzEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxGzAZBgNVBAMUEmthcmxAZGVubmluZ2VyLm5ldDCCAiIwDQYJKoZIhvcNAQEBBQADggIP ADCCAgoCggIBAM2N5maxs7NkoY9g5NMxFWll0TYiO7gXrGZTo3q25ZJgNdPMwrntLz/5ewE9 07TEbwJ3ah/Ep9BfZm7JF9vTtE1HkgKtXNKi0pawNGm1Yn26Dz5AbUr1byby6dFtDJr14E07 trzDCtRRvTkOVSBj6PQPal0fAnDtkIYQBVcuMkXkuMCtyfE95pjm8g4K9l7lAcKii3T1/3rE hCc1o2nBnb7EN1/XwBeCDGB+I2SN/ftZDbKQqGAF5q9dUn+iXU7Z/CVSfUWmhVh6cVZA4Ftv TglUqj410OuPx+cUQch3h1kFgsuhQR63HiJc3HbRJllHsV0rihvL1CjeARQkhnA6uY9NLFST p5I/PfzBzW2MSmtN/tGZvmfKKnmtbfUNgkzbIR1K3lsum+yEL71kB93Xtz/4f1demEx5c8TJ RBIniDHjDeLGK1aoBu8nfnvXAvgthFNTWBOEoR49AHEPjC3kZj0l8JQml1Y8bTQD5gtC5txl klO60WV0EufU7Hy9CmynMuFtjiA2v71pm097rXeCdrAKgisdYeEESB+SFrlY65rLiLv4n8o1 PX7DqRfqKkOYIakZ0ug/yHVKcq2EM3RiJxwzls5gT70CoOBlKbrC98O8TA6teON0Jq30M06t NTI2HhvNbJDLbBH+Awf4h1UKB+0ufENwjVvF5Jfz8Ww/FaSDAgMBAAGjgfQwgfEwNwYIKwYB BQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgwCQYD VR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIBDQQf Fh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUpfAI3y+751pp9A0w 6vJHx8RoR/MwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYwFIES a2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBiB6MlugxYJdccD8boZ/u8 d8VxmLkJCtbfyYHRjYdyoABLW5hE3k3xSpYCM9L7vzWyV/UWwDYKi4ZzxHo4g+jG/GQZfKhx v38BQjL2G9xD0Hn2d+cygOq3UPjVYlbbfQoew6JbyCFXrrZ7/0jvRMLAN2+bRC7ynaFUixPH Whnj9JSH7ieYdzak8KN+G2coIC2t2iyfXVKehzi5gdNQ0vJ7+ypbGsRm4gE8Mdo9N/WgFPvZ HPFqR9Dwas7Z+aHwOabpk5r/336SyjOaZsn3MqKJQZL6GqDKusVOCWt+9uFAD8kadg7FetZe atIoD9I+zbp59oVoMnkMDMx7Hi85faU03csusqMGsjSsAzWSI1N8PJytZlchLiykokLKc3OL G87QKlErotlou7cfPX2BbEAH5wmkj9oiqZhxIL/wwAUA+PkiTbEmksKBNompSjUq/6UsR8EA s74gnu17lmijv8mrg2qMlwRirE7qG8pnE8egLtCDxcjd0Of9WMi2NJskn0/ovC7P+J60Napl m3ZIgPJst1piYSE0Zc1FIat4fFphMfK5v4iLblo1tFSlkdx1UNDGdg/U+LaXkNVXlMp8fyPm R80V6cIrCAlEWnBJNxG1UyfbbsvNMCCZBM4faGGsR/hhQOiydlruxhjL6P8J2WV8p11DdeGx KymWoil2s1J5WTGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxv cmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRww GgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5c3Rl bXMgTExDIENBAgE9MA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcN AQcBMBwGCSqGSIb3DQEJBTEPFw0xNzA2MjAwMTI3NTBaME8GCSqGSIb3DQEJBDFCBED/uqLW CwFhNffhD7NPkHjiRA9DEmbPYx+7TAjiy2XzpHix2FBytl7TPFigtjom7/6ykebG3j6Wzn1n eWgSuNEyMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggq hkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZI hvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNVBAgT B0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExM QzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3VkYSBT eXN0ZW1zIExMQyBDQQIBPTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYTAlVT MRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEg U3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0B CQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECAT0wDQYJKoZIhvcNAQEBBQAEggIANzTMGcdpmDdi 7dIDf50Y+YVhl1jhdcR++xxXQreAN6rnfQ2kaIgjwfj9VIXufgvS4ji2sEXR3BloNwojE+ab WZRAgMvJQWxDVAecOuCPFSutiNqskQZ3qCsf1sGZsFisFj/PdLcK0cmGwgTgCocTcNzPPuFJ sggcdHbvPD5EdlhTh19vi8XUmn0uSAB7LsxhqysOj4LAQ9DlrFyiGqJfIDWlmr0dmZRxlIdC 2Pxg+GQHB7WKzZwMengQ0sBgcWXWxFi91bS60WXjhbot1EPKfsz56bDpj/DxWXaAmv+Xqe/n 3ISXPTbL59st+BrsA7R2E9KgzDnvnFKKjl6iNKIoXIZ0jhQWrbg4CiZb1JFhvbbre58WpyxX uYcLChvc6Df8HcNhR+ZBP9WFUuYwH9cvr5VLVFk6Q5sC0RBdOPkdFJ+uqJe/LdcArkyeDOOV 80nZmUB+nR6KfMlADpzzaA7XM9q7yQP5S7VZrbsOQP8c/Ek/oAbBghTbgFAGn/TMm7BWvsXO jYzqNo0f3lb6zin331fopVv3Vla6g0gIBI13hQz8V9s1Ej3g4DP6tAjNGd+RwqSFzw6Sl14U 69wQRZRePJC+3d+Gw8mjT0WMxSwn/GkFkxYJYWnvgFtkL9oAvO2OyHJ2KtG5L2MogyjYLehN oj+UtRs7HwdLf65cqMauUR4AAAAAAAA= --------------ms060306060902090607050007-- From owner-freebsd-fs@freebsd.org Tue Jun 20 01:32:03 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 398FCD87E8B for ; Tue, 20 Jun 2017 01:32:03 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x22b.google.com (mail-wm0-x22b.google.com [IPv6:2a00:1450:400c:c09::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D5E16786E for ; Tue, 20 Jun 2017 01:32:02 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x22b.google.com with SMTP id m125so7881282wmm.1 for ; Mon, 19 Jun 2017 18:32:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=xmu5yYCWW/jBC40hpO3TmsG+TuWNM+hdp7d6DdVFMj4=; b=NN3FKzhO9+XCj6dJC4g34ex6uODe30xFHOmh/VcMLM4lIQ/H0vp/DTszxKB9aEKGpk 5+sPzPPdLm1PsADfxqeQFYh4+ryX8Nj4GGNjoRvGnWycjprecJ1tkDfyJdHCyluWOtQt zQGhYe+jngHgV2bRtvadQZtFp2nBsQlft0pBgmURIhEIBeHtMivxFJgeulBBXjN9PMs3 lI+Kby097KQvvi38pUMHyC+oThT8rK9pC2LO9dU1eo6Gc3kFng9NSeHwvpqVOm2km7FS imhLAzMAA2HCwCFMnOxTFsTaxy37py9hs9/zDmbEvWh97a3nvTT/Tq7xnpcUa13lR4Ak hCKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=xmu5yYCWW/jBC40hpO3TmsG+TuWNM+hdp7d6DdVFMj4=; b=qJB5rxnVmG16NR1Zx1oIOVW0RJA2Z4y/arudIRG5cP9g2YlC5sHfmUQX32NXIhR6cH zzWwwkfaTTQvNVONn6Z04U1HJxA2ZMQq03cCxjUg3aJnnu/lXxCyvHztQtvp7nAJeeiV SiyLDomSQ82j5RFCxMTLNI8njO8CGTUfzmyrk8yHr6979BD7OOCLbARdFRqVf8L4egPR 8D2JSl1Z3vbz0fP4hE3OxfoXcuikuhLJCZd02uVlpsc120AzIf7rZ47eCcPV1lflzyJt hNW8H1s+iSZNWJHqULiUvc3QUGYQEafCGLrOsghnB1yNYhveTzhK4vBhUlpBqtuVI8vV +8MA== X-Gm-Message-State: AKS2vOxg693g7fbyQgJjKb6JDEwAg7IuYYBCzc7N88rEJNdSoYIMNqJi MilIajHgDLofMCPwCbZ17Q== X-Received: by 10.28.32.131 with SMTP id g125mr843843wmg.49.1497922320328; Mon, 19 Jun 2017 18:32:00 -0700 (PDT) Received: from [10.10.1.111] ([185.97.61.15]) by smtp.gmail.com with ESMTPSA id 2sm15380272wrx.26.2017.06.19.18.31.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Jun 2017 18:31:59 -0700 (PDT) Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs To: freebsd-fs@freebsd.org References: <431d958c658a408d8bfd4c574a565439@DM2PR58MB013.032d.mgd.msft.net> From: Steven Hartland Message-ID: <990886ae-b7c1-8630-1cef-f6678f0b5b63@multiplay.co.uk> Date: Tue, 20 Jun 2017 02:31:59 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: <431d958c658a408d8bfd4c574a565439@DM2PR58MB013.032d.mgd.msft.net> Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 01:32:03 -0000 On 20/06/2017 01:57, Caza, Aaron wrote: >> vfs.zfs.min_auto_ashift is a sysctl only its not a tuneable, so settin= g it in /boot/loader.conf won't have any effect. >> >> There's no need for it to be a tuneable as it only effects vdevs when = they are created, which an only be done once the system is running. >> > The bsdinstall install script itself set vfs.zfs.min_auto_shift=3D12 in= /boot/loader.conf yet, as you say, this doesn't do anything. As a user,= this is a bit confusing to see it in /boot/loader.conf but do a 'sysctl = -a | grep min_auto_ashift' and see 'vfs.zfs.min_auto_ashift: 9' so felt i= t was worth mentioning. Absolutely, patch is in review here: https://reviews.freebsd.org/D11278 > >> You don't explain why you believe there is degrading performance? > As I related in my post, my previous FreeBSD 11-Stable setup using this= same hardware, I was seeing 950MB/s after bootup. I've been posting to = the freebsd-hackers list, but have moved to freebsd-fs list as this seemi= ngly has something to do with FreeBSD+ZFS behavior and user Jov had previ= ously cross-posted to this list for me: > https://docs.freebsd.org/cgi/getmsg.cgi?fetch=3D2905+0+archive/2017/fre= ebsd-fs/20170618.freebsd-fs > > I've been using FreeBSD+ZFS ever since FreeBSD 9.0, admittedly, with a = different zpool layout which is essentially as follows: > adaXp1 - gptboot loader > adaXp2 - 1GB UFS partition > adaXp3 - UFS with UUID labeled partition hosting a GEOM ELI layer = using NULL encryption to emulate 4k sectors (done before ashift was an op= tion) > > So, adaXp3 would show up as something like the following: > > /dev/gpt/b62feb20-554b-11e7-989b-000bab332ee8 > /dev/gpt/b62feb20-554b-11e7-989b-000bab332ee8.eli > > Then, the zpool mirrored pair would be something like the following: > > pool: wwbase > state: ONLINE > scan: none requested > config: > > NAME STATE RE= AD WRITE CKSUM > wwbase ONLINE = 0 0 0 > mirror-0 ONLINE = 0 0 0 > gpt/b62feb20-554b-11e7-989b-000bab332ee8.eli ONLINE = 0 0 0 > gpt/4c596d40-554c-11e7-beb1-002590766b41.eli ONLINE = 0 0 0 > > Using the above zpool configuration on this same hardware on FreeBSD 11= -Stable, I was seeing read speeds of 950MB/s using dd (dd if=3D/testdb/te= st of=3D/dev/null bs=3D1m). However, after anywhere from 5 to 24 hours, = performance would degrade down to less than 100MB/s for unknown reasons -= server was essentially idle so it's a mystery to me why this occurs. I'= m seeing this behavior on FreeBSD 10.3R amd64 up through FreeBSD11.0 Stab= le. As I wasn't making any headway in resolving this, I opted today to u= se the FreeBSD11.1 Beta 2 memstick image to create a basic FreeBSD 11.1 B= eta 2 amd64 Auto(ZFS) installation to see if this would resolve the origi= nal issue I was having as I would be using ZFS-on-root and vfs.zfs.min_au= to_ashift=3D12 instead of my own emulation as described above. However, = instead of seeing the 950MB/s that I expected - which it what I see it wi= th my alternative emulation - I'm seeing 450MB/s. I've yet to determine = if this zpool setup as done by the bsdinstall script will s > uffer from the original performance degradation I observed. > >> What is the exact dd command your running as that can have a huge impa= ct on performance. > dd if=3D/testdb/test of=3D/dev/null bs=3D1m > > Note that file /testdb/test is 16GB, twice the size of ram available in= this system. The /testdb directory is a ZFS file system with recordsize= =3D8k, chosen as ultimately it's intended to host a PostgreSQL database w= hich uses an 8k page size. > > My understanding is that a ZFS mirrored pool with two drives can read f= rom both drives at the same time hence double the speed. This is what I'= ve actually observed ever since I first started using this in FreeBSD 9.0= with the GEOM ELI 4k sector emulation. This is actually my first time u= sing FreeBSD's native installer's Auto(ZFS) setup with 4k sectors emulate= d using vfs.zfs.min_auto_ashift=3D12. As it's a ZFS mirrored pool, I sti= ll expected it to be able to read at double-speed as it does with the GEO= M ELI 4k sector emulation; however, it does not. > > /var/run/dmesg.boot: > Copyright (c) 1992-2017 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 199= 4 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 11.1-BETA2 #0 r319993: Fri Jun 16 02:32:38 UTC 2017 > root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on L= LVM 4.0.0) > VT(vga): resolution 640x480 > CPU: Intel(R) Xeon(R) CPU E31240 @ 3.30GHz (3292.60-MHz K8-class CPU) > Origin=3D"GenuineIntel" Id=3D0x206a7 Family=3D0x6 Model=3D0x2a S= tepping=3D7 > Features=3D0xbfebfbff > Features2=3D0x1dbae3ff > AMD Features=3D0x28100800 > AMD Features2=3D0x1 > XSAVE Features=3D0x1 > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID > TSC: P-state invariant, performance statistics > real memory =3D 8589934592 (8192 MB) > avail memory =3D 8232431616 (7851 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs > FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads > random: unblocking device. > ioapic0 irqs 0-23 on motherboard > SMP: AP CPU #1 Launched! > SMP: AP CPU #7 Launched! > SMP: AP CPU #2 Launched! > SMP: AP CPU #3 Launched! > SMP: AP CPU #4 Launched! > SMP: AP CPU #6 Launched! > SMP: AP CPU #5 Launched! > Timecounter "TSC-low" frequency 1646297938 Hz quality 1000 > random: entropy device external interface > kbd1 at kbdmux0 > netmap: loaded module > module_register_init: MOD_LOAD (vesa, 0xffffffff80f5a190, 0) error 19 > nexus0 > vtvga0: on motherboard > cryptosoft0: on motherboard > acpi0: on motherboard > acpi0: Power Button (fixed) > cpu0: on acpi0 > cpu1: on acpi0 > cpu2: on acpi0 > cpu3: on acpi0 > cpu4: on acpi0 > cpu5: on acpi0 > cpu6: on acpi0 > cpu7: on acpi0 > attimer0: port 0x40-0x43 irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > atrtc0: port 0x70-0x71 irq 8 on acpi0 > Event timer "RTC" frequency 32768 Hz quality 0 > hpet0: iomem 0xfed00000-0xfed003ff on acpi= 0 > Timecounter "HPET" frequency 14318180 Hz quality 950 > Event timer "HPET" frequency 14318180 Hz quality 550 > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pcib0: _OSC returned error 0x10 > pci0: on pcib0 > em0: port 0xf020-0xf03f = mem 0xfba00000-0xfba1ffff,0xfba24000-0xfba24fff irq 20 at device 25.0 on = pci0 > em0: Using an MSI interrupt > em0: Ethernet address: 00:25:90:76:6b:41 > em0: netmap queues/slots: TX 1/1024, RX 1/1024 > ehci0: mem 0xfba23000-0xfba233f= f irq 16 at device 26.0 on pci0 > usbus0: EHCI version 1.0 > usbus0 on ehci0 > usbus0: 480Mbps High Speed USB v2.0 > pcib1: irq 17 at device 28.0 on pci0 > pci1: on pcib1 > pcib2: irq 17 at device 28.4 on pci0 > pci2: on pcib2 > em1: port 0xe000-0xe01f = mem 0xfb900000-0xfb91ffff,0xfb920000-0xfb923fff irq 16 at device 0.0 on p= ci2 > em1: Using MSIX interrupts with 3 vectors > em1: Ethernet address: 00:25:90:76:6b:40 > em1: netmap queues/slots: TX 1/1024, RX 1/1024 > ehci1: mem 0xfba22000-0xfba223f= f irq 23 at device 29.0 on pci0 > usbus1: EHCI version 1.0 > usbus1 on ehci1 > usbus1: 480Mbps High Speed USB v2.0 > pcib3: at device 30.0 on pci0 > pci3: on pcib3 > vgapci0: mem 0xfe000000-0xfe7fffff,0xfb800000-= 0xfb803fff,0xfb000000-0xfb7fffff irq 23 at device 3.0 on pci3 > vgapci0: Boot video device > isab0: at device 31.0 on pci0 > isa0: on isab0 > ahci0: port 0xf070-0xf077,0xf= 060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf000-0xf01f mem 0xfba21000-0xfba= 217ff irq 19 at device 31.2 on pci0 > ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported > ahcich0: at channel 0 on ahci0 > ahcich1: at channel 1 on ahci0 > ahciem0: on ahci0 > acpi_button0: on acpi0 > atkbdc0: port 0x60,0x64 irq 1 on acpi0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > psm0: irq 12 on atkbdc0 > psm0: [GIANT-LOCKED] > psm0: model IntelliMouse Explorer, device ID 4 > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0= > orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa= 0 > ppc0: cannot reserve I/O port range > est0: on cpu0 > est1: on cpu1 > est2: on cpu2 > est3: on cpu3 > est4: on cpu4 > est5: on cpu5 > est6: on cpu6 > est7: on cpu7 > ZFS filesystem version: 5 > ZFS storage pool version: features support (5000) > Timecounters tick every 1.000 msec > nvme cam probe device init > ugen0.1: at usbus0 > ugen1.1: at usbus1 > uhub0: on usbus= 0 > uhub1: on usbus= 1 > ses0 at ahciem0 bus 0 scbus2 target 0 lun 0 > ses0: SEMB S-E-S 2.00 device > ses0: SEMB SES Device > ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 > ada0: ACS-2 ATA SATA 3.x device > ada0: Serial Number S39KNB0HB00482Y > ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) > ada0: Command Queueing enabled > ada0: 244198MB (500118192 512 byte sectors) > ada0: quirks=3D0x3<4K,NCQ_TRIM_BROKEN> > ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 > ada1: ACS-2 ATA SATA 3.x device > ada1: Serial Number S39KNB0HB00473Z > ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) > ada1: Command Queueing enabled > ada1: 244198MB (500118192 512 byte sectors) > ada1: quirks=3D0x3<4K,NCQ_TRIM_BROKEN> > Trying to mount root from zfs:zroot/ROOT/default []... > Root mount waiting for: usbus1 usbus0 > uhub0: 2 ports with 2 removable, self powered > uhub1: 2 ports with 2 removable, self powered > Root mount waiting for: usbus1 usbus0 > ugen0.2: at usbus0 > uhub2 on uhub0 > uhub2: = on usbus0 > ugen1.2: at usbus1 > uhub3 on uhub1 > uhub3: = on usbus1 > Root mount waiting for: usbus1 usbus0 > uhub2: 6 ports with 6 removable, self powered > uhub3: 6 ports with 6 removable, self powered > ugen1.3: at usbus1 > umodem0 on uhub3 > umodem0: on usbus1 > umodem0: data interface 1, has CM over data, has break > >> On 19/06/2017 23:14, Caza, Aaron wrote: >>> I've been having a problem with FreeBSD ZFS SSD performance inexplic= ably degrading after < 24 hours uptime as described in a separate e-mail= thread. In an effort to get down to basics, I've now performed a ZFS-on= -Root install of FreeBSD 11.1 Beta 2 amd64 using the default Auto(ZFS) in= stall using the default 4k sector emulation (vfs.zfs.min_auto_ashift=3D3D= 12) setting (no swap, not encrypted). >>> >>> Firstly, the vfs.zfs.min_auto_ashift=3D3D12 is set correctly in the /= boot=3D/loader.conf file, but doesn't appear to work because when I log i= n and do "systctl -a | grep min_auto_ashift" it's set to 9 and not 12 as = expected. I tried setting it to vfs.zfs.min_auto_ashift=3D3D"12" in /boo= t/loader.conf but that didn't make any difference so I finally just added= it to /etc/sysctl.conf where it seems to work. So, something needs to b= e changed to make this functionaly work correctly. >>> >>> Next, after reboot I was expecting somewhere in the neighborhood of 9= 50MB/s from the ZFS mirrored zpool of 2 Samsung 850 Pro 256GB SSDs that I= 'm using as I was previously seeing this before with my previous FreeBSD = 11-Stable setup which, admittedly, is a different from the way the bsdins= tall script does it. However, I'm seeing half that on bootup. >>> >>> Performance result: >>> Starting 'dd' test of large file...please wait >>> 16000+0 records in >>> 16000+0 records out >>> 16777216000 bytes transferred in 37.407043 secs (448504207 bytes/sec)= Can you show the output from gstat -pd during this DD please. >>> >>> Zpool Status: >>> pool: zroot >>> state: ONLINE >>> scan: none requested >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> zroot ONLINE 0 0 0 >>> mirror-0 ONLINE 0 0 0 >>> ada0p2 ONLINE 0 0 0 >>> ada1p2 ONLINE 0 0 0 >>> >>> /boot/loader.conf: >>> kern.geom.label.disk_ident.enable=3D3D"0" >>> kern.geom.label.gptid.enable=3D3D"0" >>> vfs.zfs.min_auto_ashift=3D3D12 >>> vfs.zfs.arc_min=3D3D"50M" >>> vfs.zfs.arc_max=3D3D"51M" >>> zfs_load=3D3D"YES" >>> >>> /etc/sysctl.conf: >>> vfs.zfs.min_auto_ashift=3D3D12 >>> >>> >>> Is this the expected behavior now in FreeBSD 11.1? >>> >>> -- >>> Aaron > -- > Aaron > > This message may contain confidential and privileged information. If it= has been sent to you in error, please reply to advise the sender of the = error and then immediately delete it. If you are not the intended recipie= nt, do not read, copy, disclose or otherwise use this message. The sender= disclaims any liability for such unauthorized use. PLEASE NOTE that all = incoming e-mails sent to Weatherford e-mail accounts will be archived and= may be scanned by us and/or by external service providers to detect and = prevent threats to our systems, investigate illegal or inappropriate beha= vior, and/or eliminate unsolicited promotional e-mails (spam). This proce= ss could result in deletion of a legitimate e-mail before it is read by i= ts intended recipient at our organization. Moreover, based on the scannin= g results, the full text of e-mails and attachments may be made available= to Weatherford security and other personnel for review and appropriate a= ction. If you have any concerns about this process, > please contact us at dataprivacy@weatherford.com. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Tue Jun 20 14:38:51 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EFBD1D9A1AE for ; Tue, 20 Jun 2017 14:38:51 +0000 (UTC) (envelope-from lkateley@kateley.com) Received: from mail-it0-x235.google.com (mail-it0-x235.google.com [IPv6:2607:f8b0:4001:c0b::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B9459947 for ; Tue, 20 Jun 2017 14:38:51 +0000 (UTC) (envelope-from lkateley@kateley.com) Received: by mail-it0-x235.google.com with SMTP id m47so15396799iti.1 for ; Tue, 20 Jun 2017 07:38:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kateley-com.20150623.gappssmtp.com; s=20150623; h=reply-to:subject:to:references:from:organization:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=PGpI1zxA2P8qs+VKw1naU4Rw21aZnm1FrWeb9V2z8QI=; b=vYxO5niRX3Idb0eL7j+cOlloaSGsAviUa0B41Oa5Yxn2/IwlpTkb2ml9oltsapqv10 wMpRuLjkvoWc2yygFLKYDZyCvcJPKbfV7it/WKDO59JnBfB4HvJ1NToHaUkrEKjjGzsL JBTP3lwiTKWTCm+jcZAa6Dn0jW17eb2NdC4pmC+zeguSx798hYDgvAk5+6i2LRUWR3V0 QDntRcIoF0EnRDVCwVDXnX/bLMBASz9gUyoZlkS9HRDKLjJJq9sTOPVUbhmjQVPKJIaE ZwOZVMjZXieoWxFyclsM6rYkugCqJaT639tmXe1lfjLnPkdqVw9igVNG8/uDbO/3P7Pe m6Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:subject:to:references:from:organization :message-id:date:user-agent:mime-version:in-reply-to :content-transfer-encoding:content-language; bh=PGpI1zxA2P8qs+VKw1naU4Rw21aZnm1FrWeb9V2z8QI=; b=uLyWoPn6CGdo7Avq7pk4Ij0TCpNtilTajX5ZuP/f6RzAx4MjZ2OkCkKKt8j569kwwC xM6Yu0C4NZymSrAxdB0g01vUG7qqlxWZz35t3g/j45r32qiAbk9zhz6zloWSPxG+GQ23 Ib+2/Od/E9hK07G0P3DikgRlHiFb6m2UQy+tEoQMl7k3kOXyxI24SZQkYShkUSlpchrc Wj7v7gLAvYFPKees6Vb2prMgOt5+AZAZllZ3wEAviz/JU1roFjmfwQUeJhkUEzZrpIXt U7EnXepSnpnsFdDvhtYOUaJ4fH+1a3t98aKXPylPrmSR0xqvYiuAgd+s68f1q8Fvnhdf TOAw== X-Gm-Message-State: AKS2vOwKqUj+38RPLMr+8sTAZhkPR8g9/iZi1gem4otiZTErExch+LVb bjHpBq+O+zWXWx9AY40= X-Received: by 10.36.116.23 with SMTP id o23mr3957685itc.107.1497969530381; Tue, 20 Jun 2017 07:38:50 -0700 (PDT) Received: from Kateleyco-iMac.local (c-75-72-30-128.hsd1.mn.comcast.net. [75.72.30.128]) by smtp.googlemail.com with ESMTPSA id 131sm8566156itl.11.2017.06.20.07.38.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 20 Jun 2017 07:38:49 -0700 (PDT) Reply-To: linda@kateley.com Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs To: freebsd-fs@freebsd.org References: <431d958c658a408d8bfd4c574a565439@DM2PR58MB013.032d.mgd.msft.net> From: Linda Kateley Organization: Kateley Company Message-ID: <17bddb07-9a4f-9287-a77e-0e2f6d81cac9@kateley.com> Date: Tue, 20 Jun 2017 09:38:49 -0500 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: <431d958c658a408d8bfd4c574a565439@DM2PR58MB013.032d.mgd.msft.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Language: en-US X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 14:38:52 -0000 This does seem odd. One theory could be the arc isn't releasing=20 correctly. Can you give us a look at cache structures at the point of=20 performance degradation? arc_summary.py On 6/19/17 7:57 PM, Caza, Aaron wrote: >> vfs.zfs.min_auto_ashift is a sysctl only its not a tuneable, so settin= g it in /boot/loader.conf won't have any effect. >> >> There's no need for it to be a tuneable as it only effects vdevs when = they are created, which an only be done once the system is running. >> > The bsdinstall install script itself set vfs.zfs.min_auto_shift=3D12 in= /boot/loader.conf yet, as you say, this doesn't do anything. As a user,= this is a bit confusing to see it in /boot/loader.conf but do a 'sysctl = -a | grep min_auto_ashift' and see 'vfs.zfs.min_auto_ashift: 9' so felt i= t was worth mentioning. > >> You don't explain why you believe there is degrading performance? > As I related in my post, my previous FreeBSD 11-Stable setup using this= same hardware, I was seeing 950MB/s after bootup. I've been posting to = the freebsd-hackers list, but have moved to freebsd-fs list as this seemi= ngly has something to do with FreeBSD+ZFS behavior and user Jov had previ= ously cross-posted to this list for me: > https://docs.freebsd.org/cgi/getmsg.cgi?fetch=3D2905+0+archive/2017/fre= ebsd-fs/20170618.freebsd-fs > > I've been using FreeBSD+ZFS ever since FreeBSD 9.0, admittedly, with a = different zpool layout which is essentially as follows: > adaXp1 - gptboot loader > adaXp2 - 1GB UFS partition > adaXp3 - UFS with UUID labeled partition hosting a GEOM ELI layer = using NULL encryption to emulate 4k sectors (done before ashift was an op= tion) > > So, adaXp3 would show up as something like the following: > > /dev/gpt/b62feb20-554b-11e7-989b-000bab332ee8 > /dev/gpt/b62feb20-554b-11e7-989b-000bab332ee8.eli > > Then, the zpool mirrored pair would be something like the following: > > pool: wwbase > state: ONLINE > scan: none requested > config: > > NAME STATE RE= AD WRITE CKSUM > wwbase ONLINE = 0 0 0 > mirror-0 ONLINE = 0 0 0 > gpt/b62feb20-554b-11e7-989b-000bab332ee8.eli ONLINE = 0 0 0 > gpt/4c596d40-554c-11e7-beb1-002590766b41.eli ONLINE = 0 0 0 > > Using the above zpool configuration on this same hardware on FreeBSD 11= -Stable, I was seeing read speeds of 950MB/s using dd (dd if=3D/testdb/te= st of=3D/dev/null bs=3D1m). However, after anywhere from 5 to 24 hours, = performance would degrade down to less than 100MB/s for unknown reasons -= server was essentially idle so it's a mystery to me why this occurs. I'= m seeing this behavior on FreeBSD 10.3R amd64 up through FreeBSD11.0 Stab= le. As I wasn't making any headway in resolving this, I opted today to u= se the FreeBSD11.1 Beta 2 memstick image to create a basic FreeBSD 11.1 B= eta 2 amd64 Auto(ZFS) installation to see if this would resolve the origi= nal issue I was having as I would be using ZFS-on-root and vfs.zfs.min_au= to_ashift=3D12 instead of my own emulation as described above. However, = instead of seeing the 950MB/s that I expected - which it what I see it wi= th my alternative emulation - I'm seeing 450MB/s. I've yet to determine = if this zpool setup as done by the bsdinstall script will s > uffer from the original performance degradation I observed. > >> What is the exact dd command your running as that can have a huge impa= ct on performance. > dd if=3D/testdb/test of=3D/dev/null bs=3D1m > > Note that file /testdb/test is 16GB, twice the size of ram available in= this system. The /testdb directory is a ZFS file system with recordsize= =3D8k, chosen as ultimately it's intended to host a PostgreSQL database w= hich uses an 8k page size. > > My understanding is that a ZFS mirrored pool with two drives can read f= rom both drives at the same time hence double the speed. This is what I'= ve actually observed ever since I first started using this in FreeBSD 9.0= with the GEOM ELI 4k sector emulation. This is actually my first time u= sing FreeBSD's native installer's Auto(ZFS) setup with 4k sectors emulate= d using vfs.zfs.min_auto_ashift=3D12. As it's a ZFS mirrored pool, I sti= ll expected it to be able to read at double-speed as it does with the GEO= M ELI 4k sector emulation; however, it does not. > > /var/run/dmesg.boot: > Copyright (c) 1992-2017 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 199= 4 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 11.1-BETA2 #0 r319993: Fri Jun 16 02:32:38 UTC 2017 > root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on L= LVM 4.0.0) > VT(vga): resolution 640x480 > CPU: Intel(R) Xeon(R) CPU E31240 @ 3.30GHz (3292.60-MHz K8-class CPU) > Origin=3D"GenuineIntel" Id=3D0x206a7 Family=3D0x6 Model=3D0x2a S= tepping=3D7 > Features=3D0xbfebfbff > Features2=3D0x1dbae3ff > AMD Features=3D0x28100800 > AMD Features2=3D0x1 > XSAVE Features=3D0x1 > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID > TSC: P-state invariant, performance statistics > real memory =3D 8589934592 (8192 MB) > avail memory =3D 8232431616 (7851 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs > FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads > random: unblocking device. > ioapic0 irqs 0-23 on motherboard > SMP: AP CPU #1 Launched! > SMP: AP CPU #7 Launched! > SMP: AP CPU #2 Launched! > SMP: AP CPU #3 Launched! > SMP: AP CPU #4 Launched! > SMP: AP CPU #6 Launched! > SMP: AP CPU #5 Launched! > Timecounter "TSC-low" frequency 1646297938 Hz quality 1000 > random: entropy device external interface > kbd1 at kbdmux0 > netmap: loaded module > module_register_init: MOD_LOAD (vesa, 0xffffffff80f5a190, 0) error 19 > nexus0 > vtvga0: on motherboard > cryptosoft0: on motherboard > acpi0: on motherboard > acpi0: Power Button (fixed) > cpu0: on acpi0 > cpu1: on acpi0 > cpu2: on acpi0 > cpu3: on acpi0 > cpu4: on acpi0 > cpu5: on acpi0 > cpu6: on acpi0 > cpu7: on acpi0 > attimer0: port 0x40-0x43 irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > atrtc0: port 0x70-0x71 irq 8 on acpi0 > Event timer "RTC" frequency 32768 Hz quality 0 > hpet0: iomem 0xfed00000-0xfed003ff on acpi= 0 > Timecounter "HPET" frequency 14318180 Hz quality 950 > Event timer "HPET" frequency 14318180 Hz quality 550 > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pcib0: _OSC returned error 0x10 > pci0: on pcib0 > em0: port 0xf020-0xf03f = mem 0xfba00000-0xfba1ffff,0xfba24000-0xfba24fff irq 20 at device 25.0 on = pci0 > em0: Using an MSI interrupt > em0: Ethernet address: 00:25:90:76:6b:41 > em0: netmap queues/slots: TX 1/1024, RX 1/1024 > ehci0: mem 0xfba23000-0xfba233f= f irq 16 at device 26.0 on pci0 > usbus0: EHCI version 1.0 > usbus0 on ehci0 > usbus0: 480Mbps High Speed USB v2.0 > pcib1: irq 17 at device 28.0 on pci0 > pci1: on pcib1 > pcib2: irq 17 at device 28.4 on pci0 > pci2: on pcib2 > em1: port 0xe000-0xe01f = mem 0xfb900000-0xfb91ffff,0xfb920000-0xfb923fff irq 16 at device 0.0 on p= ci2 > em1: Using MSIX interrupts with 3 vectors > em1: Ethernet address: 00:25:90:76:6b:40 > em1: netmap queues/slots: TX 1/1024, RX 1/1024 > ehci1: mem 0xfba22000-0xfba223f= f irq 23 at device 29.0 on pci0 > usbus1: EHCI version 1.0 > usbus1 on ehci1 > usbus1: 480Mbps High Speed USB v2.0 > pcib3: at device 30.0 on pci0 > pci3: on pcib3 > vgapci0: mem 0xfe000000-0xfe7fffff,0xfb800000-= 0xfb803fff,0xfb000000-0xfb7fffff irq 23 at device 3.0 on pci3 > vgapci0: Boot video device > isab0: at device 31.0 on pci0 > isa0: on isab0 > ahci0: port 0xf070-0xf077,0xf= 060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf000-0xf01f mem 0xfba21000-0xfba= 217ff irq 19 at device 31.2 on pci0 > ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported > ahcich0: at channel 0 on ahci0 > ahcich1: at channel 1 on ahci0 > ahciem0: on ahci0 > acpi_button0: on acpi0 > atkbdc0: port 0x60,0x64 irq 1 on acpi0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > psm0: irq 12 on atkbdc0 > psm0: [GIANT-LOCKED] > psm0: model IntelliMouse Explorer, device ID 4 > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0= > orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa= 0 > ppc0: cannot reserve I/O port range > est0: on cpu0 > est1: on cpu1 > est2: on cpu2 > est3: on cpu3 > est4: on cpu4 > est5: on cpu5 > est6: on cpu6 > est7: on cpu7 > ZFS filesystem version: 5 > ZFS storage pool version: features support (5000) > Timecounters tick every 1.000 msec > nvme cam probe device init > ugen0.1: at usbus0 > ugen1.1: at usbus1 > uhub0: on usbus= 0 > uhub1: on usbus= 1 > ses0 at ahciem0 bus 0 scbus2 target 0 lun 0 > ses0: SEMB S-E-S 2.00 device > ses0: SEMB SES Device > ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 > ada0: ACS-2 ATA SATA 3.x device > ada0: Serial Number S39KNB0HB00482Y > ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) > ada0: Command Queueing enabled > ada0: 244198MB (500118192 512 byte sectors) > ada0: quirks=3D0x3<4K,NCQ_TRIM_BROKEN> > ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 > ada1: ACS-2 ATA SATA 3.x device > ada1: Serial Number S39KNB0HB00473Z > ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) > ada1: Command Queueing enabled > ada1: 244198MB (500118192 512 byte sectors) > ada1: quirks=3D0x3<4K,NCQ_TRIM_BROKEN> > Trying to mount root from zfs:zroot/ROOT/default []... > Root mount waiting for: usbus1 usbus0 > uhub0: 2 ports with 2 removable, self powered > uhub1: 2 ports with 2 removable, self powered > Root mount waiting for: usbus1 usbus0 > ugen0.2: at usbus0 > uhub2 on uhub0 > uhub2: = on usbus0 > ugen1.2: at usbus1 > uhub3 on uhub1 > uhub3: = on usbus1 > Root mount waiting for: usbus1 usbus0 > uhub2: 6 ports with 6 removable, self powered > uhub3: 6 ports with 6 removable, self powered > ugen1.3: at usbus1 > umodem0 on uhub3 > umodem0: on usbus1 > umodem0: data interface 1, has CM over data, has break > >> On 19/06/2017 23:14, Caza, Aaron wrote: >>> I've been having a problem with FreeBSD ZFS SSD performance inexplic= ably degrading after < 24 hours uptime as described in a separate e-mail= thread. In an effort to get down to basics, I've now performed a ZFS-on= -Root install of FreeBSD 11.1 Beta 2 amd64 using the default Auto(ZFS) in= stall using the default 4k sector emulation (vfs.zfs.min_auto_ashift=3D3D= 12) setting (no swap, not encrypted). >>> >>> Firstly, the vfs.zfs.min_auto_ashift=3D3D12 is set correctly in the /= boot=3D/loader.conf file, but doesn't appear to work because when I log i= n and do "systctl -a | grep min_auto_ashift" it's set to 9 and not 12 as = expected. I tried setting it to vfs.zfs.min_auto_ashift=3D3D"12" in /boo= t/loader.conf but that didn't make any difference so I finally just added= it to /etc/sysctl.conf where it seems to work. So, something needs to b= e changed to make this functionaly work correctly. >>> >>> Next, after reboot I was expecting somewhere in the neighborhood of 9= 50MB/s from the ZFS mirrored zpool of 2 Samsung 850 Pro 256GB SSDs that I= 'm using as I was previously seeing this before with my previous FreeBSD = 11-Stable setup which, admittedly, is a different from the way the bsdins= tall script does it. However, I'm seeing half that on bootup. >>> >>> Performance result: >>> Starting 'dd' test of large file...please wait >>> 16000+0 records in >>> 16000+0 records out >>> 16777216000 bytes transferred in 37.407043 secs (448504207 bytes/sec)= >>> >>> Zpool Status: >>> pool: zroot >>> state: ONLINE >>> scan: none requested >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> zroot ONLINE 0 0 0 >>> mirror-0 ONLINE 0 0 0 >>> ada0p2 ONLINE 0 0 0 >>> ada1p2 ONLINE 0 0 0 >>> >>> /boot/loader.conf: >>> kern.geom.label.disk_ident.enable=3D3D"0" >>> kern.geom.label.gptid.enable=3D3D"0" >>> vfs.zfs.min_auto_ashift=3D3D12 >>> vfs.zfs.arc_min=3D3D"50M" >>> vfs.zfs.arc_max=3D3D"51M" >>> zfs_load=3D3D"YES" >>> >>> /etc/sysctl.conf: >>> vfs.zfs.min_auto_ashift=3D3D12 >>> >>> >>> Is this the expected behavior now in FreeBSD 11.1? >>> >>> -- >>> Aaron > -- > Aaron > > This message may contain confidential and privileged information. If it= has been sent to you in error, please reply to advise the sender of the = error and then immediately delete it. If you are not the intended recipie= nt, do not read, copy, disclose or otherwise use this message. The sender= disclaims any liability for such unauthorized use. PLEASE NOTE that all = incoming e-mails sent to Weatherford e-mail accounts will be archived and= may be scanned by us and/or by external service providers to detect and = prevent threats to our systems, investigate illegal or inappropriate beha= vior, and/or eliminate unsolicited promotional e-mails (spam). This proce= ss could result in deletion of a legitimate e-mail before it is read by i= ts intended recipient at our organization. Moreover, based on the scannin= g results, the full text of e-mails and attachments may be made available= to Weatherford security and other personnel for review and appropriate a= ction. If you have any concerns about this process, > please contact us at dataprivacy@weatherford.com. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Tue Jun 20 16:58:41 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 067BFD9CFEF for ; Tue, 20 Jun 2017 16:58:41 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-bl2nam02on0076.outbound.protection.outlook.com [104.47.38.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5497566801 for ; Tue, 20 Jun 2017 16:58:39 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weatherford.onmicrosoft.com; s=selector1-ca-weatherford-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=vWcU7KOYIwymPdS76cqejL7diqQc9NJ0twEzZfufZ44=; b=YhlbvJintgHRenXr8yYaNyFMLNhjPdsfkgjvz2b8Rih7jSlHPT+U4gYmnJdzvNvAxFXr776INzExF0TJL7IaMzKY5MCVg8nwrcoBU6lYMHeoh2O+vmR8gjXdI9J93FZTJS4nqgBBWxIg8AlczkHxZApiziq+litdRPsB2LzxWEQ= Received: from CY1PR03CA0031.namprd03.prod.outlook.com (10.174.128.41) by MWHPR03MB3278.namprd03.prod.outlook.com (10.174.249.34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 16:58:37 +0000 Received: from BL2FFO11FD049.protection.gbl (2a01:111:f400:7c09::198) by CY1PR03CA0031.outlook.office365.com (2603:10b6:600::41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14 via Frontend Transport; Tue, 20 Jun 2017 16:58:37 +0000 Authentication-Results: spf=pass (sender IP is 23.103.226.20) smtp.mailfrom=ca.weatherford.com; multiplay.co.uk; dkim=none (message not signed) header.d=none;multiplay.co.uk; dmarc=bestguesspass action=none header.from=ca.weatherford.com; Received-SPF: Pass (protection.outlook.com: domain of ca.weatherford.com designates 23.103.226.20 as permitted sender) receiver=protection.outlook.com; client-ip=23.103.226.20; helo=032-smtp-out.weatherford.com; Received: from 032-smtp-out.weatherford.com (23.103.226.20) by BL2FFO11FD049.mail.protection.outlook.com (10.173.161.211) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1157.12 via Frontend Transport; Tue, 20 Jun 2017 16:58:35 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net (141.251.110.81) by DM2PR58MB015.032d.mgd.msft.net (141.251.110.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 16:58:33 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) by DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) with mapi id 15.01.1178.018; Tue, 20 Jun 2017 16:58:33 +0000 From: "Caza, Aaron" To: Steven Hartland , "freebsd-fs@freebsd.org" Subject: RE: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Topic: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Index: AQHS6eZz0MQHJ5XH+0mGKOVoyWjUKw== Date: Tue, 20 Jun 2017 16:58:33 +0000 Message-ID: References: <431d958c658a408d8bfd4c574a565439@DM2PR58MB013.032d.mgd.msft.net> <990886ae-b7c1-8630-1cef-f6678f0b5b63@multiplay.co.uk> In-Reply-To: <990886ae-b7c1-8630-1cef-f6678f0b5b63@multiplay.co.uk> Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [141.251.205.196] x-ms-publictraffictype: Email X-MS-TrafficTypeDiagnostic: DM2PR58MB015:|MWHPR03MB3278: X-MS-Office365-Filtering-Correlation-Id: 35a4ed5d-e8a2-4bd5-2c8a-08d4b7fd978a Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-OrganizationHeadersPreserved: DM2PR58MB015.032d.mgd.msft.net X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:23.103.226.20; IPV:NLI; CTRY:; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(39840400002)(39450400003)(39860400002)(39410400002)(39400400002)(39850400002)(2980300002)(438002)(377454003)(24454002)(13464003)(189002)(199003)(9170700003)(6246003)(6116002)(76176999)(22756006)(7696004)(2906002)(3846002)(102836003)(50986999)(6306002)(9686003)(38730400002)(2900100001)(53936002)(5890100001)(2501003)(86362001)(106466001)(53546009)(54356999)(5660300001)(305945005)(55016002)(33646002)(86146001)(478600001)(8676002)(8936002)(356003)(81166006)(72206003)(2950100002)(229853002)(42882006)(189998001)(24736003)(7736002)(66066001)(23676002)(50466002)(966005)(108616004)(47776003)(299355004); DIR:OUT; SFP:1101; SCL:1; SRVR:MWHPR03MB3278; H:032-smtp-out.weatherford.com; FPR:; SPF:Pass; MLV:sfv; MX:1; A:0; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BL2FFO11FD049; 1:sArArtyXRPno7InGOXkKxNempB4hKYj/E/BGgK1YRW8tg+yQmHNYEpGWyGIjhH1JvC5ohjYiXdawp0R9KtxUKl1zGPNCawmG2GLEyT3IBD68ungId3EFCgWxzDZvhcB/d/pQc0635NcE2H9oZSt4+6pl3Tv19LyiGgyrNdSX2ohNIBAxmUl7TJcSjEwD5fW+CFhY7FXY8ZtLD0lNFB0N0mkNWNg5FxPNAwYDGCnZgBrIXIZZUf3//c7DPL8Hp7gCKum+Eml70J0a6a0LZvBmLtOIijPMFObrOtVBIgVhgvlzrnNE1phDhUi0yD/ysdjBY49GHr1mF9UNCbZV3EA1oAzCIE80+QpA51ECf4+7NrGWhmacNgLq285ujdigbIDl01F7i77K8s5agSQK/wQzuA8Fez9e6tctf0rKTC62Ui5Z2/SCjIsDnUPgCFw82k96ZGOi53FLychdO4scLMhraJ/iQzuNBSvNiADzNiyOGIcsAeSqujgUb8hqE1F8XqCzliCbV0zrLvWxqa3FGUY43jyZcq171MKlNO+OAQfbhpbMs3jMMieFiXPsoyvF3LlV3RrwlVCoVoM8nyKHCkMBfftAiQYQpxW1KOY9Yn8PcjvpsP/JBzHtjP6DDJYxGmlkvy5wSPuWSN6yT8eBZThVnqOmWTuwJWOOaGzibKiDV+M97jvlwsmmVc3zPAdYq7+MEzuuEb97Q8MurA98Hfmwj7aeClrbIuMNHuH0ZHs3BenyPZw/weU/bsqrL7GyvK8kN9pN3Si8zNTO3NieLaIP2a2gdhRjxZyOCa0YSlBW7uOh2AZDP9rp9SLNLfuG6PIQq9eXlkl/BPh/vJyFApHyXMbz04gmGU/KCvrqgzNOLywx4C/SyzcvJX+mmvd1mFM2 X-CrossPremisesHeadersPromoted: BL2FFO11FD049.protection.gbl X-CrossPremisesHeadersFiltered: BL2FFO11FD049.protection.gbl X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(8251501002)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:MWHPR03MB3278; X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3278; 3:ZtVg2LI9cc1ds8kES1wxXZ0Fej8Gmlj5AiU/l+edWBikrFLH9Sr13yqcgMcUq3aeqzUbXMv6dRGGH+kM1nJSR99+SLnUJCpW4KPTB4ZJ4NjaQYNn1+dUe3kf3P2IQuGkDv4KFO11/b1Ui4Vet7WCuE/jYF7/nRleDImZ3cp73A5GyeICLcU6FFyrM7xDP29TrFlgPUWtLkox0/fZ+WbPQEaQ/BeA90Hmjxt9IXikt8eZIyRg5WvnvLc1APdf1L3WaBkIyQiCA3Fh1GR8r3p+sKXN3d8esZzAbQN5+Nb9XLhbFOLg+p9dBzmnLC7eo5UtIO+a362SymAa4NK5l3dErmPW9wqLLi/o1VV41l57lCX3LxSPFrLv2c99mn1ER/UVXurChoratCuCK3AYhGqrgAj1x5USSdCVP2FbGHu9zQu1WVdK93ZZ7umEznnZdvu+5nrLQPprTy7RMXp++1IbOUQZIXum/Dd7afcc5x4GE0Tm2jsT+kRR5dcxkjNIIDzPlL456RaJYCUjfqUKTa2sgw== X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3278; 25:mJHfbnSHMwBjCoAX+AFaild/i1XYrR/jRanFoIo5yJdwF2w0exM8k5oo1EsI4iQnWAFqm1MMnQND/U5UdtnnzL3yt9H3WpZM82jkHw0Vr3sk56BG8yyTzb40nLGDJk6cVxZNwglzjyIsCVCXiA0l093uhMWG18TRoqu3mk2DnxuRaWnbuzd1sbIJqx5uvsmkaPeyG1hKR2P9Jg7oPTHHu3FyWAAaledVA3qJUvIGtWrgnrUh9sWPumTnGbRipeXW2CWCpOIfOPc9KDn/roc8Tm/Nq7QYcbbD7y0iV1+RYdrhJEeZuMdqJSQMMxhCOs/s6kB7/OYQBfJ3VsmEavliDUcwguC1n67/XJdSWzm/3ly2F2lWTax3YjVFXq26w+5AApNmmTRvKm2Z3EeqC3bHzUTjZZMYgKdUNQtkaCfKpjJUi9hjfKtMZelE/v6xAQiD+8jCvAuvrxxvq9OpXnLnC7lQkQEK6UT6iMrisR5IAFc=; 31:kYt/4xFwiAWh6/DuOrrhif65l/43YfZ7Re6/yc4UD7V5hH0z212vrTReD+j48rqWUe5oB9tAX0wRZPm3FU5IKsa7aFQqLlqg9aXifB8WENL+eJ6Rl80/q67g7+6Inx8HRfuZM1aaRG5m76x1G6oCsZ2y+1/k3u5IPOe1en0UjvsHfdMYPVB55oJGsQs/hqOZlqBmOCOGxjI2qZd2JqHgRpi0vJktrfKcSrNlycK7g3czbUmzPzGpVzys0id5BqRWOxFoUdLPKVrdGnd0lnfGjA== X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3278; 20:WaHpQMusuC8vRWRZ82GXsQbY12BS0/CniXs/sneZiu10QGemiryy1aj4MXK3WiiLTCzoRl1TONqlf2ddqcEnez87Z9NV4lZME81Q8ZQKCM8AgvkRwU6oJOlm+Izvp2IDnChXpdeansiq1CUFcrMNJOri85MdbXmdX5DExHe83CDlILHacy+i4tcyA6Qhb9Go0qmMCkkfEY2gakgY2/Ey/QAXbxJ6SWFrVIxVZF2tFS9S3QsgCzYGhmLwhPjPcA83RAeLGfYU+aHCFrt2mU+Nk7zfXQ5Rmf5BI9Ck7nBi3JmUshFnLubLHCbXYwMSG9i0yodSTeeYgwyL+kQ7TFHkJ2I19pceEH0WGZxKWDZ3yCQkjE2/K9B5u5B2NyhZhRSoOTDXOWQCBHsG2OYETgAgOviTzH6mqImHu/HQIaODnWReZc7EbIiT0ecAsduJ4HXOu8RVwWEFT/AwJ/YOGl+csRj7MJVWbmBci8cHlVrgCDwBisi0Vok4whN509Ang07d X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(158342451672863)(209352067349851)(192374486261705); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(11241501159)(6040450)(601004)(2401047)(13018025)(8121501046)(13016025)(5005006)(10201501046)(100000703101)(100105400095)(3002001)(93006095)(93004095)(6055026)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(20161123562025)(20161123555025)(20161123564025)(20161123558100)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:MWHPR03MB3278; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:MWHPR03MB3278; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtNV0hQUjAzTUIzMjc4OzQ6RDl6WHg4cVcrMW1UdkVQSTFWcmdCeU1GNlp3?= =?utf-8?B?UWMzb1hxSW1aYXNMazBFQ0JEVVpwRVVTVXN3emRtL1E4VmJyYnoreUd1Q1ht?= =?utf-8?B?YXZDWmVHYjUyTWxta3hIRFVOcFRuWGVyY2Uzby9wRTZiai9uc0c0MjZCNFp3?= =?utf-8?B?TTdramljUUQ5QzhGS2gvcXJsZW1nV1gxS3VpZTdUdUFEb1ZRSC8zdDVhNTNn?= =?utf-8?B?OVQvY0FmazdWRVYvUWZMUmhBYkNGcHFaQTBPbU5oZVNWTjJJSHh4SGxyOFcr?= =?utf-8?B?eXkwMGZ6OWR6ZzJmcmNIaVVGRjBsOWsvS1JhVjBXMkpOQUxzRE9BK0NhdnVZ?= =?utf-8?B?MHI0eHhYU2NLVmNkK0NzTUNzbmtEYXhGTzN4TmswdUYzRHBvVy80RThlWU1Q?= =?utf-8?B?UEdFOURnTU1YSEZSaGgzMGxtTjVObXI4ODVHdDFML0Yya3U1ZUx3U2Q5Tnk0?= =?utf-8?B?RkIvTk0wanNneDZ4c2lhWjFnRkF4Y0RFYkRvcGFacDRQWGErVUJ3Vnp3ZjFp?= =?utf-8?B?bWNIbHl3bFNkYUt2MEJEaE5FcDhuSlZKMkVzN2docVBBRXNqVWU4cmtIU3lR?= =?utf-8?B?eHFJNVlCcXlSSEtGSXRCT1piKzdMUGxOc0lWMW1jQi9qYmRIU0NZNC94d2Z2?= =?utf-8?B?VDVlZUJyelNTejJ4aGhFUVB3OGR5M09Oak5lT0w5bWZQc1hRWnJHNDlESFdi?= =?utf-8?B?cDhsUUpUSjRhd21wbUtEeVV6QnZib25IVnlJWU9vYkRWM0EwWW1qOUlUOW5Q?= =?utf-8?B?djV3aDZpb01LcFJWczBRdTk5S1FLU2JIWlVxZHlOb1BqTGJvMC9KSFR1QU9i?= =?utf-8?B?TmhxeGpYTVBRQlJHck02YkYyWjlwb0huWkVFRzhpSDlySFNXNVJWRThHWkZC?= =?utf-8?B?Q29wTmdLM0d5ZzNiZjVDbEk0bjFrUlBwd1RsZGRRVm4vVmV1bFh1bFF3ajBE?= =?utf-8?B?K01WRmowdTMvKzR6TkxDcGdXbm1HT1g5VW4vVXl2MjVtNURHY3JlSWRBRGdX?= =?utf-8?B?S3cwMm1yV3B2TERNaGE0U2l6QkszbCtxSTQ4bm9Pd0hIYlVjMWF4b001UnB0?= =?utf-8?B?VHlvcXJMTndrcmdrN0VMQ2V0QytCNmRUM1p1cURLTWtlTmlXalZ1MmE5N2Fx?= =?utf-8?B?alRtQTBLZmZPR2F5emVldktsak9nYUUzVG9SYXhlTWh4MC95RmdZYlhIVzZk?= =?utf-8?B?VmhjTTRKMHY4WnYrbjlRY1pCZEkvdEt2T3ZGWDVOTjlaZWZZN2IzTlZMSk53?= =?utf-8?B?d1lWaG5LWkVQOFNXNTVaczU5UVZxdXNpKzR3d3poMStKWTVwSjNHdFdzNVRi?= =?utf-8?B?Z2ZxR1VJNGFiS3ZLNUV4RGoyWU9POWhSYmp2UFFOc2NlOWhGQU5YNnF2YVQw?= =?utf-8?B?c05id0lINERkb3RBblc3TnpIMHQxWXloY29ESXNGRHo5aktjaTBxNXlhZnBK?= =?utf-8?B?MzQ4YklWT243eGphUE9peDdkWm8vN0xLUk1FeUZvZE0zdnR1RENDUUkyRFBw?= =?utf-8?B?WjVzNDArQk80STVnNWMvcmlmRDBOY2pNMnBZYkdiUThyanJyL3Z6ZUhuN3Bo?= =?utf-8?B?OWU2Rkc3QWcrazdVam5pQjFPRnN0MUNsS0dQQUlDUXZCQnJQbzRVUTQ2Ti9T?= =?utf-8?B?MzVKUkQ0YVBPbW9Jd1RtaUpxeWJzWEErcHNXVUpMWXdJK0hrNjZWTE9hZ2FV?= =?utf-8?B?aDJldy91VnhXK2thNEUwSTNyS1g4T1FKZ1o1ajkraUlQUWpUL05WY3VBVysw?= =?utf-8?B?dXk5T2VPeXJ4M0NBQ1FvZ0QrMGxYbXdna0R5U0pFaFpKWitsdHViMGdoNlVu?= =?utf-8?Q?zNcWRDypePB/?= X-Forefront-PRVS: 03449D5DD1 X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtNV0hQUjAzTUIzMjc4OzIzOmhOR3BpRmw4S1RXd0ttakZ4RlZPYUFqYktu?= =?utf-8?B?L25uQnBTRkFOd2ZCaldaNjN5SzhJc0JNOVhnV3BGSWU0NGVpVW9XOEk2UUlF?= =?utf-8?B?M0M1Z2YzWVZCMG1vMVhqWGlISWw5UThXbmVqN1ZrMDJpdlkzS1EwN1BlWktE?= =?utf-8?B?V0wxU0ozSXBzWnB1eUxZWFJCeGxQOUJsR3E3NFk0OUtMYnJNTEIvMHJSTGNi?= =?utf-8?B?SzRlUU1oS2I0Ukc0MHFEWXdmaGRKWG1ma1BKZGw3MGxIQnlFZUxrZkswYWoy?= =?utf-8?B?Q015V0NDSFdIaEt1U0JjMVFHZHVmMUU1ZWZLVVNHN1ZIZkVQMFpHZ2ZUdGtB?= =?utf-8?B?MksyNHRWSnZBa3N6UGMwYUhMWkMrbVFVNnBGZkIveVRqanJlTzV6VGs3M2Vi?= =?utf-8?B?ZUVEMEU5WDNlaG1GYTQrNHhhUUVjd1RpT0pkWnRreVJZY1ZkSitZcGxybTZ4?= =?utf-8?B?c2ZwR2hSZko1UGRMMjBnN1BRSHZDZHZWY0VvSTR4YkVyM2JYYzlkSmVINnFP?= =?utf-8?B?aFFTRUVJQ29wYTZEUE1hb29IOUVrN2lxTmg0QjlyMHZ2VGdHVWdYUmdSMTFN?= =?utf-8?B?ZWhpbXp2dW9wVDBzeXlPSCs2YUdQWTVLa242cnFOMWJ3d0NtVVFtTzNka2ZU?= =?utf-8?B?cXF2RzZwY0xKKzlMMzI3N1dWUEZFazZ0aDlESW9keitrMlhYK3pNdU1ZcWxN?= =?utf-8?B?RThDZzJGdGppbUJIc1NGdTQxQ3RkRmpqdkZPMWZZZmtUV0xWMUpTSXRQVFFl?= =?utf-8?B?cGFHcGRYRWVtbzFXL2hkZnVhcHgrVEFLekJ4dTVpWHN1cmxkaE0yVGZtNGFu?= =?utf-8?B?cjJIb2t0WGxPRFNZeWdkditTMjVzeE1KdGVpbUxmbDNkYUV3Wk52WnIvRXd1?= =?utf-8?B?MkxMb3dESjJyU1RRcXFHNzJ0VHVBZWlmNDVtRE1TUGZZVG1ULzZiT3dXa1FQ?= =?utf-8?B?c2tLN3dEU0NFZ3VDL3JNejRoNkxrUkQzYkZPRUN2azdGeGkvYTZUUGJOaGlM?= =?utf-8?B?eGlIbUQ4Z2tIV21NR1N0RkNma2IvbitnSG50SkNCYWpXN3lWS2JXZ3YzR2dI?= =?utf-8?B?N3JCN1laZUJRVEVqMkZSWnBUczFseGNGVVViUmlqTWoxM205a0ZFYTUreFNy?= =?utf-8?B?NmVGNzIrQ0xmMmhJTUZxYmtlKzV3OFljTmREVVB3K2lpa1YzV2NpSkZDenNm?= =?utf-8?B?QWtFYTlkY3drQWxDOFVxMnRicWFTamlSMnlxZzM3UUs0cllJUjMxdVVxT3l2?= =?utf-8?B?NHNYTy9IS2h2R0VDQjRBb3RpS25nQy9pNnJFRXZUQTlrU3ZldGI3U0pPY0xY?= =?utf-8?B?WU1vVXdPcGVoNll3SVo1K0VLaVJQdzhreTFUbVE2eU5iTkgvYjUwNzQ0UG00?= =?utf-8?B?eGNIWlBjckZaQ1JEcXdzSXFPZ0dyLzEyYmFDTnZGU0preGFXRDlrRHl6SE5x?= =?utf-8?B?WVZBVEpYdjRVL0pLVnNmL25GTDE5eFhsSUFlZktrZTc2cDVMR2RSdFUyeXlm?= =?utf-8?B?TExweStGUkEyYmxwblZBZjVGSHYxVVpyQTFiK1FkSnIyaVZidFZhdzY2d0Fj?= =?utf-8?B?MHl4OVpRYU93eS9xSkJ5Mk5pL3dBcHZUMFEzUHphYVU2YXczUk9aa2VXaVFi?= =?utf-8?B?V3FSeTFTK1Y1cUthZ1dscWRCSlNDYXNyUzFLZTU5Yjdlb2pJdFNydG1vMU1X?= =?utf-8?B?b0lFM3FDODBGdGdXeWh4cmhMdkVzdnp3T2M5Z3htV2Ixbmg4Y1ptODRZVDlG?= =?utf-8?B?TWorcmxxNW9JTU1FSGF6cmF5TzV5VDJxVXFGOGMzU3dHaU1FMWtvbVNoeTlD?= =?utf-8?B?TXZaVWs4QVNpSmEyN0FZRzcxdnNzZ3RHcWUyYnRDdXVLY0t6UHYrQ0hlSFRh?= =?utf-8?B?SmJGcU15cTFTbzgxVEhIUG9mV0c2VWx6M2VjTk1scE5IN0xLeS9UZ0d3T1Vp?= =?utf-8?B?U0NtditOcE1RPT0=?= X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtNV0hQUjAzTUIzMjc4OzY6Sm5GWnFkVVBiVlRTUWlkcU41WEJlQm85a0ps?= =?utf-8?B?SmMyNCtLUGFacWZFem03OFVQYW4yK3VDeHVveVhBb2k1RHVnd21SU0piWC96?= =?utf-8?B?cG1SbVJkSjVRdFQ4NmZHMkNQTEJsR2tNNjlaaUp5SzgxemNpZkhJT2ZCTURx?= =?utf-8?B?c055UUVaSW9ic0J5dGU0azJ6bWFGeGk5dmVSWmRNUzhoZ0VnL0VnQlJwc09z?= =?utf-8?B?aE12cUVPWEFmc1lTMUh2anorMmUxU1BVR1hoa3lEMzBFMlZYMVdpckMzbTJz?= =?utf-8?B?Y29uZzBDN1JjWitjWnB0L1RKdmdHVFF0a2FvVmoyQTUwVUtqQldLUEVtR01t?= =?utf-8?B?UjBvNXlEY3g3cDNHMkhJaGxyTHJCREI5ZWxFcm54ZTVraTJUZGo1OUhtcVFV?= =?utf-8?B?WW14Ry9CSnZWbFRnTWdSRU9xamNLTm55K1F6SnZEZUorL1VnOWxoYkRNMWN5?= =?utf-8?B?T3pWOFpHVXFOY1phamhjdWtrbzZzMm1MN3R4c3N4TkFLZTdsZXcyWjNwMFNi?= =?utf-8?B?VFhwb3BkN083Wmx0bGtKdnY1VkQzWHkwL2x0ZVNBUkRvUmVxbEV1U0cxdUQr?= =?utf-8?B?bnJrMjVCbkw0c2F5RmRtUFYzYjhwdHpPekdjbmZIQ085ei9OUkdkbHZDQmFO?= =?utf-8?B?YkJWK25VNDN6U2VUcEVjeWU1WGl1UFVOamRibjErazhSQWdlSHB3VjFrbDZx?= =?utf-8?B?b3JQV0d1dGpqTERJdGVSdGU5dTV1VkdxZmtYVEhSM3NVOGV0eUZ3WWZvT2Qy?= =?utf-8?B?QU0vUVNLMjhKOWRvZFdDakNaVUpLK1JPS2NnR21LdHhrb2h3QU1VT1hUSEtI?= =?utf-8?B?YnNFaFIza1J2YU9NYktrWGs1N0QrTzRkZWo5eEs2RmkxVE9KR1lhYy91blBs?= =?utf-8?B?cXI3MnVXM0N0Ni9QRkliNEJFekZRczNPS29Ga1laK0tGQ2hNYmV0d3FDaE9P?= =?utf-8?B?NGZGTFBlMkd0SlFtRFNzMS9IRmE2dHZQTFNmejl6MVYzNXhld1FBT3FKUXh3?= =?utf-8?B?RDMrK0c5QlplZ3FDME1XdG1mMHRzNUJ0dnZiWWhldlF1K2VZMmVWRVRscHlH?= =?utf-8?B?aE9Kd1JzZkpJTU9kcitxZm5ZVDI4b2gxa0VWYXVIVkhqUWtDUysxQ0hHVUtK?= =?utf-8?B?RG9SV2txVTk3NzBhbUJONlJURjgwSEZTMEZMSk5JRmE5ZlhSRVJRVjVVQlhG?= =?utf-8?B?RTFueWtKcDIzT3lESVBkeFJEeHVKWjB2b0pteS9FTis2dHFHT0xKcERYcks0?= =?utf-8?B?cTF5K0NySzJINFZKTWZpMDU0bnV5WkQ5d0YweWMrZzVmUjNPM0p4ZFpxa0NO?= =?utf-8?B?WHp3YWFrSmNkb04xOGlQL0xvL25oMlVROWxSOUJLSTZIUlh5ZzFqZmZ1NnQz?= =?utf-8?Q?mGcF5+T?= X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3278; 5:HWavrevxL9UBaUzhnkP5u/mebatVDCT0Salp+6XW8zAht44SsuBy+JqCQ1UIOmWnZprO1DsBPLlh5TjtvqxxvM2CVs2cNgvnnTguF03wuFbpjY0Qrv1hhEdEjlly1zTT+Ei/hW+/jWpMp0GubJZUJXTY6RpUpaJfMjLQDeGrNX4IIzwIapFWMsFgdGMk2m+CzocuQkjdeX8Fkxrz3EyyZC4XuLGcX8yLNsejTalffyercefr+9L39n1mLDKcpHyLlrQCG0A0Sb3kxDv/X0bGDpYt7ZSU/Noy5qAXwG6gJdHgQuUdwzcvVbr9iLg8hYCUe5Ut9X4jG6ytwiC7D+2a7orn7uLP8qK3bCgTzOpt5lC7+8EoRZPKtIO1WEjYi3w/3VfPQrHSh+OeT8CjfTAHIIkf7Yjm7iFT1a9m9wsUGi5Whz489AsDywraeqF0zsyEzI/ZdcJSlbi+jw0pVUIKoK/vANIZPBg/5kySNMCG9ebjLDPYP/JzOSquW3qf3hYX; 24:Cx3rjibBoBK8wga23Crqcplk47L/QVKBuGc3/ljlE4nGhtFfTdZxCcSCv7j1EkmJsQeO3wOMPcYMu+U/CBXRjMhNNmuV0N2uH3QR4Xv3EWI= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3278; 7:nzzu7tmnVOOU2DwYqRdcDkXfiP6sWWt5zi+weCAYJJsmZ6Z9xi74D6qolnJxKn2BqCLYPm2KRwN9o3qRi3Q9miCb9m4lTU5Kcf5yPTmKxSgUnge8VTlJj/r2L7TN2elAr/AFHg3J8Y2NvXXlHEuOUihBeyV3NbSRQorlRY+W7kTqCc3RgFFOWFfVrvQJELibGz27D5ZIcWY/Phy+ZfNVSktys7SUl/KU3u5V9sy+svi94qhZ2fddj1ksuybLkrfFWN9kHcbfKdgfPYWqNVzmBmLiiacu74hB168evQm9e+cCNlamQlRtC57N8z3KaojgIheDT8VZhIVcTduy0rANVIl8XU8mMNBogG2JVwzoOsA/ZFcFyySwzzuZPUBcV9JJJhyEK4ruF9cZr0wky10MiO40ubVos+MMcVtf8SaD+JVjIsZIwGgcegRK1Wjj5U9ilPmIG24Ocsn9+STrGrBnQuFEnaCCeaOfuktsyd0doHgaZ99c9hRhJA9vTVDeHecECNLLp+lPNDEQkL/f2YLdYkOwTCmJTeq2+cABohcNWqukpHiCZ2NFhGGWqaA1qnQytlCAnNSAlOlU+VvVFRY5BjcwU7pqEh077YAyKoUaMuSoDDmGg1R+HxXnu209X7bvUdAbroXwZXftaTte5VV7u0WaV+C+L4NjIWzkubVo8ptqxzss5LTuNuksp2DTqVo94IGSVd6QcjJldqDOHhf+v9lpmsl3xQcVbL8P5HpEJUNfLRo3Xyw//xumDVmXEgT09T3UD0osHn2uxZ0X4ghVGQpFv2V6fimyJ0L0+UGWY6I= X-OriginatorOrg: ca.weatherford.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jun 2017 16:58:35.5618 (UTC) X-MS-Exchange-CrossTenant-Id: dd63fb60-07f6-4d96-8d40-ebeca61a524e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=dd63fb60-07f6-4d96-8d40-ebeca61a524e; Ip=[23.103.226.20]; Helo=[032-smtp-out.weatherford.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR03MB3278 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 16:58:41 -0000 PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBTdGV2ZW4gSGFydGxhbmQgW21h aWx0bzpraWxsaW5nQG11bHRpcGxheS5jby51a10NCj4gU2VudDogTW9uZGF5LCBKdW5lIDE5LCAy MDE3IDc6MzIgUE0NCj4gVG86IGZyZWVic2QtZnNAZnJlZWJzZC5vcmcNCj4gU3ViamVjdDogUmU6 IEZyZWVCU0QgMTEuMSBCZXRhIDIgWkZTIHBlcmZvcm1hbmNlIGRlZ3JhZGF0aW9uIG9uIFNTRHMN Cj4NCj4gT24gMjAvMDYvMjAxNyAwMTo1NywgQ2F6YSwgQWFyb24gd3JvdGU6DQo+ID4+IHZmcy56 ZnMubWluX2F1dG9fYXNoaWZ0IGlzIGEgc3lzY3RsIG9ubHkgaXRzIG5vdCBhIHR1bmVhYmxlLCBz byBzZXR0aW5nIGl0IGluIC9ib290L2xvYWRlci5jb25mIHdvbid0IGhhdmUgYW55IGVmZmVjdC4N Cj4gPj4NCj4gPj4gVGhlcmUncyBubyBuZWVkIGZvciBpdCB0byBiZSBhIHR1bmVhYmxlIGFzIGl0 IG9ubHkgZWZmZWN0cyB2ZGV2cyB3aGVuIHRoZXkgYXJlIGNyZWF0ZWQsIHdoaWNoIGFuIG9ubHkg YmUgZG9uZSBvbmNlIHRoZSBzeXN0ZW0gaXMgcnVubmluZy4NCj4gPj4NCj4gPiBUaGUgYnNkaW5z dGFsbCBpbnN0YWxsIHNjcmlwdCBpdHNlbGYgc2V0IHZmcy56ZnMubWluX2F1dG9fc2hpZnQ9MTIg aW4gL2Jvb3QvbG9hZGVyLmNvbmYgeWV0LCBhcyB5b3Ugc2F5LCB0aGlzIGRvZXNuJ3QgZG8gYW55 dGhpbmcuICBBcyBhIHVzZXIsIHRoaXMgaXMgYSBiaXQgY29uZnVzaW5nIHRvIHNlZSBpdCBpbiAv Ym9vdC9sb2FkZXIuY29uZiBidXQgZG8gYSAnc3lzY3RsIC1hIHwgZ3JlcCBtaW5fYXV0b19hc2hp ZnQnIGFuZCBzZWUgJ3Zmcy56ZnMubWluX2F1dG9fYXNoaWZ0OiA5JyBzbyBmZWx0IGl0IHdhcyB3 b3J0aCBtZW50aW9uaW5nLg0KPiBBYnNvbHV0ZWx5LCBwYXRjaCBpcyBpbiByZXZpZXcgaGVyZTog aHR0cHM6Ly9yZXZpZXdzLmZyZWVic2Qub3JnL0QxMTI3OA0KDQpUaGFua3MgZm9yIHRha2luZyBj YXJlIG9mIHRoaXMgU3RldmUgLSBhcHByZWNpYXRlZC4NCg0KPiA+DQo+ID4+IFlvdSBkb24ndCBl eHBsYWluIHdoeSB5b3UgYmVsaWV2ZSB0aGVyZSBpcyBkZWdyYWRpbmcgcGVyZm9ybWFuY2U/DQo+ ID4gQXMgSSByZWxhdGVkIGluIG15IHBvc3QsIG15IHByZXZpb3VzIEZyZWVCU0QgMTEtU3RhYmxl IHNldHVwIHVzaW5nIHRoaXMgc2FtZSBoYXJkd2FyZSwgSSB3YXMgc2VlaW5nIDk1ME1CL3MgYWZ0 ZXIgYm9vdHVwLiAgSSd2ZSBiZWVuIHBvc3RpbmcgdG8gdGhlIGZyZWVic2QtaGFja2VycyBsaXN0 LCBidXQgaGF2ZSBtb3ZlZCB0byBmcmVlYnNkLWZzIGxpc3QgYXMgdGhpcyBzZWVtaW5nbHkgaGFz IHNvbWV0aGluZyB0byBkbyB3aXRoIEZyZWVCU0QrWkZTIGJlaGF2aW9yIGFuZCB1c2VyIEpvdiBo YWQgcHJldmlvdXNseSBjcm9zcy1wb3N0ZWQgdG8gdGhpcyBsaXN0IGZvciBtZToNCj4gPiBodHRw czovL2RvY3MuZnJlZWJzZC5vcmcvY2dpL2dldG1zZy5jZ2k/ZmV0Y2g9MjkwNSswK2FyY2hpdmUv MjAxNy9mcmVlDQo+ID4gYnNkLWZzLzIwMTcwNjE4LmZyZWVic2QtZnMNCj4gPg0KPiA+IEkndmUg YmVlbiB1c2luZyBGcmVlQlNEK1pGUyBldmVyIHNpbmNlIEZyZWVCU0QgOS4wLCBhZG1pdHRlZGx5 LCB3aXRoIGEgZGlmZmVyZW50IHpwb29sIGxheW91dCB3aGljaCBpcyBlc3NlbnRpYWxseSBhcyBm b2xsb3dzOg0KPiA+ICAgICAgYWRhWHAxIC0gZ3B0Ym9vdCBsb2FkZXINCj4gPiAgICAgIGFkYVhw MiAtIDFHQiBVRlMgcGFydGl0aW9uDQo+ID4gICAgICBhZGFYcDMgLSBVRlMgd2l0aCBVVUlEIGxh YmVsZWQgcGFydGl0aW9uIGhvc3RpbmcgYSBHRU9NIEVMSSBsYXllcg0KPiA+IHVzaW5nIE5VTEwg ZW5jcnlwdGlvbiB0byBlbXVsYXRlIDRrIHNlY3RvcnMgKGRvbmUgYmVmb3JlIGFzaGlmdCB3YXMg YW4NCj4gPiBvcHRpb24pDQo+ID4NCj4gPiBTbywgYWRhWHAzIHdvdWxkIHNob3cgdXAgYXMgc29t ZXRoaW5nIGxpa2UgdGhlIGZvbGxvd2luZzoNCj4gPg0KPiA+ICAgIC9kZXYvZ3B0L2I2MmZlYjIw LTU1NGItMTFlNy05ODliLTAwMGJhYjMzMmVlOA0KPiA+ICAgIC9kZXYvZ3B0L2I2MmZlYjIwLTU1 NGItMTFlNy05ODliLTAwMGJhYjMzMmVlOC5lbGkNCj4gPg0KPiA+IFRoZW4sIHRoZSB6cG9vbCBt aXJyb3JlZCBwYWlyIHdvdWxkIGJlIHNvbWV0aGluZyBsaWtlIHRoZSBmb2xsb3dpbmc6DQo+ID4N Cj4gPiAgICBwb29sOiB3d2Jhc2UNCj4gPiAgIHN0YXRlOiBPTkxJTkUNCj4gPiAgICBzY2FuOiBu b25lIHJlcXVlc3RlZA0KPiA+IGNvbmZpZzoNCj4gPg0KPiA+ICAgICAgICAgIE5BTUUgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgU1RBVEUgICAgIFJFQUQgV1JJ VEUgQ0tTVU0NCj4gPiAgICAgICAgICB3d2Jhc2UgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIE9OTElORSAgICAgICAwICAgICAwICAgICAwDQo+ID4gICAgICAgICAg ICBtaXJyb3ItMCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBPTkxJTkUg ICAgICAgMCAgICAgMCAgICAgMA0KPiA+ICAgICAgICAgICAgICBncHQvYjYyZmViMjAtNTU0Yi0x MWU3LTk4OWItMDAwYmFiMzMyZWU4LmVsaSAgT05MSU5FICAgICAgIDAgICAgIDAgICAgIDANCj4g PiAgICAgICAgICAgICAgZ3B0LzRjNTk2ZDQwLTU1NGMtMTFlNy1iZWIxLTAwMjU5MDc2NmI0MS5l bGkgIE9OTElORSAgICAgICAwICAgICAwICAgICAwDQo+ID4NCj4gPiBVc2luZyB0aGUgYWJvdmUg enBvb2wgY29uZmlndXJhdGlvbiBvbiB0aGlzIHNhbWUgaGFyZHdhcmUgb24gRnJlZUJTRCAxMS1T dGFibGUsIEkgd2FzIHNlZWluZyByZWFkIHNwZWVkcyBvZiA5NTBNQi9zIHVzaW5nIGRkIChkZCBp Zj0vdGVzdGRiL3Rlc3Qgb2Y9L2Rldi9udWxsIGJzPTFtKS4gIEhvd2V2ZXIsIGFmdGVyIGFueXdo ZXJlIGZyb20gNSB0byAyNCBob3VycywgcGVyZm9ybWFuY2Ugd291bGQgZGVncmFkZSBkb3duIHRv IGxlc3MgdGhhbiAxMDBNQi9zIGZvciB1bmtub3duIHJlYXNvbnMgLSBzZXJ2ZXIgd2FzIGVzc2Vu dGlhbGx5IGlkbGUgc28gaXQncyBhDQo+ICBteXN0ZXJ5IHRvIG1lIHdoeSB0aGlzIG9jY3Vycy4g IEknbSBzZWVpbmcgdGhpcyBiZWhhdmlvciBvbiBGcmVlQlNEIDEwLjNSIGFtZDY0IHVwIHRocm91 Z2ggRnJlZUJTRDExLjAgU3RhYmxlLiAgQXMgSSB3YXNuJ3QgbWFraW5nIGFueSBoZWFkd2F5IGlu IHJlc29sdmluZyB0aGlzLCBJIG9wdGVkIHRvZGF5IHRvIHVzZSB0aGUgRnJlZUJTRDExLjEgQmV0 YSAyIG1lbXN0aWNrIGltYWdlIHRvIGNyZWF0ZSBhIGJhc2ljIEZyZWVCU0QgMTEuMSBCZXRhIDIg YW1kNjQgQXV0byhaRlMpIGluc3RhbGxhdGlvbiB0byBzZWUgaWYgdGhpcyB3b3VsZCByZXNvbHZl IHRoZSBvcmlnaW5hbA0KPiAgaXNzdWUgSSB3YXMgaGF2aW5nIGFzIEkgd291bGQgYmUgdXNpbmcg WkZTLW9uLXJvb3QgYW5kIHZmcy56ZnMubWluX2F1dG9fYXNoaWZ0PTEyIGluc3RlYWQgb2YgbXkg b3duIGVtdWxhdGlvbiBhcyBkZXNjcmliZWQgYWJvdmUuICBIb3dldmVyLCBpbnN0ZWFkIG9mIHNl ZWluZyB0aGUgOTUwTUIvcyB0aGF0IEkgZXhwZWN0ZWQgLSB3aGljaCBpdCB3aGF0IEkgc2VlIGl0 IHdpdGggbXkgYWx0ZXJuYXRpdmUgZW11bGF0aW9uIC0gSSdtIHNlZWluZyA0NTBNQi9zLiAgSSd2 ZSB5ZXQgdG8gZGV0ZXJtaW5lIGlmIHRoaXMgenBvb2wgc2V0dXAgYXMgZG9uZSBieSB0aGUgYnNk aW5zdGFsbCBzY3JpcHQgPiB3aWxsIHN1ZmZlciBmcm9tIHRoZSBvcmlnaW5hbCBwZXJmb3JtYW5j ZSBkZWdyYWRhdGlvbiBJIG9ic2VydmVkLg0KPiA+DQo+ID4+IFdoYXQgaXMgdGhlIGV4YWN0IGRk IGNvbW1hbmQgeW91ciBydW5uaW5nIGFzIHRoYXQgY2FuIGhhdmUgYSBodWdlIGltcGFjdCBvbiBw ZXJmb3JtYW5jZS4NCj4gPiBkZCBpZj0vdGVzdGRiL3Rlc3Qgb2Y9L2Rldi9udWxsIGJzPTFtDQo+ ID4NCj4gPiBOb3RlIHRoYXQgZmlsZSAvdGVzdGRiL3Rlc3QgaXMgMTZHQiwgdHdpY2UgdGhlIHNp emUgb2YgcmFtIGF2YWlsYWJsZSBpbiB0aGlzIHN5c3RlbS4gIFRoZSAvdGVzdGRiIGRpcmVjdG9y eSBpcyBhIFpGUyBmaWxlIHN5c3RlbSB3aXRoIHJlY29yZHNpemU9OGssIGNob3NlbiBhcyB1bHRp bWF0ZWx5IGl0J3MgaW50ZW5kZWQgdG8gaG9zdCBhIFBvc3RncmVTUUwgZGF0YWJhc2Ugd2hpY2gg dXNlcyBhbiA4ayBwYWdlIHNpemUuDQo+ID4NCj4gPiBNeSB1bmRlcnN0YW5kaW5nIGlzIHRoYXQg YSBaRlMgbWlycm9yZWQgcG9vbCB3aXRoIHR3byBkcml2ZXMgY2FuIHJlYWQgZnJvbSBib3RoIGRy aXZlcyBhdCB0aGUgc2FtZSB0aW1lIGhlbmNlIGRvdWJsZSB0aGUgc3BlZWQuICBUaGlzIGlzIHdo YXQgSSd2ZSBhY3R1YWxseSBvYnNlcnZlZCBldmVyIHNpbmNlIEkgZmlyc3Qgc3RhcnRlZCB1c2lu ZyB0aGlzIGluIEZyZWVCU0QgOS4wIHdpdGggdGhlIEdFT00gRUxJIDRrIHNlY3RvciBlbXVsYXRp b24uICBUaGlzIGlzIGFjdHVhbGx5IG15IGZpcnN0IHRpbWUgdXNpbmcgRnJlZUJTRCdzIG5hdGl2 ZSBpbnN0YWxsZXIncyBBdXRvKFpGUykgc2V0dXAgPiB3aXRoIDRrIHNlY3RvcnMgZW11bGF0ZWQg dXNpbmcgdmZzLnpmcy5taW5fYXV0b19hc2hpZnQ9MTIuICBBcyBpdCdzIGEgWkZTIG1pcnJvcmVk IHBvb2wsIEkgc3RpbGwgZXhwZWN0ZWQgaXQgdG8gYmUgYWJsZSB0byByZWFkIGF0IGRvdWJsZS1z cGVlZCBhcyBpdCBkb2VzIHdpdGggdGhlIEdFT00gRUxJIDRrIHNlY3RvciBlbXVsYXRpb247IGhv d2V2ZXIsIGl0IGRvZXMgbm90Lg0KPiA+DQo+PiBPbiAxOS8wNi8yMDE3IDIzOjE0LCBDYXphLCBB YXJvbiB3cm90ZToNCj4+PiBJJ3ZlIGJlZW4gIGhhdmluZyBhIHByb2JsZW0gd2l0aCBGcmVlQlNE IFpGUyBTU0QgcGVyZm9ybWFuY2UgaW5leHBsaWNhYmx5IGRlZ3JhZGluZyBhZnRlciA8IDI0ICBo b3VycyB1cHRpbWUgYXMgZGVzY3JpYmVkIGluIGEgc2VwYXJhdGUgZS1tYWlsIHRocmVhZC4gIElu IGFuIGVmZm9ydCB0byBnZXQgZG93biB0byBiYXNpY3MsIEkndmUgbm93IHBlcmZvcm1lZCBhIFpG Uy1vbi1Sb290IGluc3RhbGwgb2YgRnJlZUJTRCAxMS4xIEJldGEgMiBhbWQ2NCB1c2luZyB0aGUg ZGVmYXVsdCBBdXRvKFpGUykgaW5zdGFsbCB1c2luZyB0aGUgZGVmYXVsdCA0ayBzZWN0b3IgZW11 bGF0aW9uICh2ZnMuemZzLm1pbl9hdXRvX2FzaGlmdD0zRDEyKSBzZXR0aW5nIChubyBzd2FwLCBu b3QgZW5jcnlwdGVkKS4NCj4+Pg0KPj4+IEZpcnN0bHksIHRoZSB2ZnMuemZzLm1pbl9hdXRvX2Fz aGlmdD0zRDEyIGlzIHNldCBjb3JyZWN0bHkgaW4gdGhlIC9ib290PS9sb2FkZXIuY29uZiBmaWxl LCBidXQgZG9lc24ndCBhcHBlYXIgdG8gd29yayBiZWNhdXNlIHdoZW4gSSBsb2cgaW4gYW5kIGRv ICJzeXN0Y3RsIC1hIHwgZ3JlcCBtaW5fYXV0b19hc2hpZnQiIGl0J3Mgc2V0IHRvIDkgYW5kIG5v dCAxMiBhcyBleHBlY3RlZC4gIEkgdHJpZWQgc2V0dGluZyBpdCB0byB2ZnMuemZzLm1pbl9hdXRv X2FzaGlmdD0zRCIxMiIgaW4gL2Jvb3QvbG9hZGVyLmNvbmYgYnV0IHRoYXQgZGlkbid0IG1ha2Ug YW55IGRpZmZlcmVuY2Ugc28gSSBmaW5hbGx5IGp1c3QgYWRkZWQgaXQgdG8gL2V0Yy9zeXNjdGwu Y29uZiB3aGVyZSBpdCBzZWVtcyB0byB3b3JrLiAgU28sIHNvbWV0aGluZyBuZWVkcyB0byBiZSBj aGFuZ2VkIHRvIG1ha2UgdGhpcyBmdW5jdGlvbmFseSB3b3JrIGNvcnJlY3RseS4NCj4+Pg0KPj4+ IE5leHQsIGFmdGVyIHJlYm9vdCBJIHdhcyBleHBlY3Rpbmcgc29tZXdoZXJlIGluIHRoZSBuZWln aGJvcmhvb2Qgb2YgOTUwTUIvcyBmcm9tIHRoZSBaRlMgbWlycm9yZWQgenBvb2wgb2YgMiBTYW1z dW5nIDg1MCBQcm8gMjU2R0IgU1NEcyB0aGF0IEknbSB1c2luZyBhcyBJIHdhcyBwcmV2aW91c2x5 IHNlZWluZyB0aGlzIGJlZm9yZSB3aXRoIG15IHByZXZpb3VzIEZyZWVCU0QgMTEtU3RhYmxlIHNl dHVwIHdoaWNoLCBhZG1pdHRlZGx5LCBpcyBhIGRpZmZlcmVudCBmcm9tIHRoZSB3YXkgdGhlIGJz ZGluc3RhbGwgc2NyaXB0IGRvZXMgaXQuICBIb3dldmVyLCBJJ20gc2VlaW5nIGhhbGYgdGhhdCBv biBib290dXAuDQo+Pj4NCj4+PiBQZXJmb3JtYW5jZSByZXN1bHQ6DQo+Pj4gU3RhcnRpbmcgJ2Rk JyB0ZXN0IG9mIGxhcmdlIGZpbGUuLi5wbGVhc2Ugd2FpdA0KPj4+IDE2MDAwKzAgcmVjb3JkcyBp bg0KPj4+IDE2MDAwKzAgcmVjb3JkcyBvdXQNCj4+PiAxNjc3NzIxNjAwMCBieXRlcyB0cmFuc2Zl cnJlZCBpbiAzNy40MDcwNDMgc2VjcyAoNDQ4NTA0MjA3DQo+Pj4gYnl0ZXMvc2VjKQ0KPiBDYW4g eW91IHNob3cgdGhlIG91dHB1dCBmcm9tIGdzdGF0IC1wZCBkdXJpbmcgdGhpcyBERCBwbGVhc2Uu DQoNCmRUOiAxLjAwMXMgIHc6IDEuMDAwcw0KIEwocSkgIG9wcy9zICAgIHIvcyAgIGtCcHMgICBt cy9yICAgIHcvcyAgIGtCcHMgICBtcy93ICAgIGQvcyAgIGtCcHMgICBtcy9kICAgJWJ1c3kgTmFt ZQ0KICAgIDAgICA0MzE4ICAgNDMxOCAgMzQ4NjUgICAgMC4wICAgICAgMCAgICAgIDAgICAgMC4w ICAgICAgMCAgICAgIDAgICAgMC4wICAgMTQuMnwgYWRhMA0KICAgIDAgICA0NDAyICAgNDQwMiAg MzUyMTMgICAgMC4wICAgICAgMCAgICAgIDAgICAgMC4wICAgICAgMCAgICAgIDAgICAgMC4wICAg MTQuNHwgYWRhMQ0KDQpkVDogMS4wMDJzICB3OiAxLjAwMHMNCiBMKHEpICBvcHMvcyAgICByL3Mg ICBrQnBzICAgbXMvciAgICB3L3MgICBrQnBzICAgbXMvdyAgICBkL3MgICBrQnBzICAgbXMvZCAg ICVidXN5IE5hbWUNCiAgICAxICAgNDI0OSAgIDQyNDkgIDM0MTM2ICAgIDAuMCAgICAgIDAgICAg ICAwICAgIDAuMCAgICAgIDAgICAgICAwICAgIDAuMCAgIDE0LjF8IGFkYTANCiAgICAwICAgNDM5 MyAgIDQzOTMgIDM1Mjg3ICAgIDAuMCAgICAgIDAgICAgICAwICAgIDAuMCAgICAgIDAgICAgICAw ICAgIDAuMCAgIDE0LjV8IGFkYTENCg0KRXZlcnkgbm93IGFuZCBhZ2FpbiwgSSB3YXMgc2VlaW5n IGQvcyBoaXQsIHdoaWNoIEkgdW5kZXJzdGFuZCB0byBiZSBUUklNIG9wZXJhdGlvbnMgLSBpdCB3 b3VsZCBicmllZmx5IHNob3cgMTYgdGhlbiBnbyBiYWNrIHRvIDAuDQoNCnRlc3RAZjExMWJldGEy On4gIyBkZCBpZj0vdGVzdGRiL3Rlc3Qgb2Y9L2Rldi9udWxsIGJzPTFtDQoxNjAwMCswIHJlY29y ZHMgaW4NCjE2MDAwKzAgcmVjb3JkcyBvdXQNCjE2Nzc3MjE2MDAwIGJ5dGVzIHRyYW5zZmVycmVk IGluIDQzLjQ0NzM0MyBzZWNzICgzODYxNTA1NjEgYnl0ZXMvc2VjKQ0KdGVzdEBmMTExYmV0YTI6 fiAjIHVwdGltZQ0KIDk6NTRBTSAgdXAgMTk6MzgsIDIgdXNlcnMsIGxvYWQgYXZlcmFnZXM6IDIu OTIsIDEuMDEsIDAuNDQNCnJvb3RAZjExMWJldGEyOn4gIyBkZCBpZj0vdGVzdGRiL3Rlc3Qgb2Y9 L2Rldi9udWxsIGJzPTFtDQoxNjAwMCswIHJlY29yZHMgaW4NCjE2MDAwKzAgcmVjb3JkcyBvdXQN CjE2Nzc3MjE2MDAwIGJ5dGVzIHRyYW5zZmVycmVkIGluIDIzNi4wOTcwMTEgc2VjcyAoNzEwNjA2 ODggYnl0ZXMvc2VjKQ0KdGVzdEBmMTExYmV0YTI6fiAjIHVwdGltZQ0KMTA6MzZBTSAgdXAgMjA6 MjAsIDIgdXNlcnMsIGxvYWQgYXZlcmFnZXM6IDAuOTAsIDAuNjIsIDAuMzYNCg0KQXMgY2FuIGJl IHNlZW4gaW4gdGhlIGFib3ZlICdkZCcgdGVzdCByZXN1bHRzLCBJJ20gYmFjayB0byBzZWVpbmcg dGhlIG9yaWdpbmFsIGlzc3VlIEkgcmVwb3J0ZWQgb24gZnJlZWJzZC1oYWNrZXJzIC0gcGVyZm9y bWFuY2UgZGVncmFkaW5nIGFmdGVyIDwgMjQgaG91cnMgb2YgdXB0aW1lIGdvaW5nIGZyb20gfjM4 Nk1CL3NlYyB0byB+NzFNQi9zZWMgaW5leHBpY2FibHkgLSB0aGlzIHNlcnZlciBpc24ndCBkb2lu ZyBhbnl0aGluZyBvdGhlciB0aGFuIHJ1bm5pbmcgdGhpcyB0ZXN0IGhvdXJseS4NCg0KUGxlYXNl IG5vdGUgaW4gdGhlIGdzdGF0IC1wZCBvdXRwdXQgYWJvdmUsIHRoaXMgd2FzIGFmdGVyIHRoZSBw ZXJmb3JtYW5jZSBkZWdyYWRhdGlvbiBoaXQuICBQcmlvciB0byB0aGlzLCBJIHdhcyBzZWVpbmcg JWJ1c3kgb2YgfjYwJS4gIEluIHRoaXMgcGFydGljdWxhciBpbnN0YW5jZSwgdGhlIHBlcmZvcm1h bmNlIGRlZ3JhZGF0aW9uIGhpdCB+MjBocnMgaW50byB0aGUgdGVzdCBidXQgSSd2ZSBzZWUgaXQg aGl0IGFzIHNvb24gYXMgfjVocnMuDQoNClByZXZpb3VzbHksIEFsbGFuIEp1ZGUgaGFkIGFkdmlz ZWQgemZzLnZmcy50cmltLmVuYWJsZWQ9MCB0byBzZWUgaWYgdGhpcyBjaGFuZ2VkIHRoZSBiZWhh dmlvci4gIEkgZGlkIHRoaXM7IGhvd2V2ZXIsIGl0IGhhZCBubyBpbXBhY3QgLSBidXQgdGhhdCB3 YXMgd2hlbiBJIHdhcyB1c2luZyB0aGUgR0VPTSBFTEkgNGsgc2VjdG9yIGVtdWxhdGlvbiBhbmQg bm90IHRoZSBhc2hpZnQgNGsgc2VjdG9yIGVtdWxhdGlvbi4gIFRoZSBHRU9NIEVMSSA0ayBzZWN0 b3IgZW11bGF0aW9uIGRvZXMgbm90IGFwcGVhciB0byB3b3JrIHdpdGggVFJJTSBvcGVyYXRpb25z IGFzIGdzdGF0IC1kIGluIHRoYXQgY2FzZSBhbHdheXMgc3RheWVkIGF0IDAgb3BzL3MuICBJIGNh biB0cnkgZGlzYWJsaW5nIHRyaW0sIGJ1dCBkaWQgbm90IHdhbnQgdG8gcmVib290IHRoZSBzZXJ2 ZXIgdG8gcmVzdGFydCB0aGUgdGVzdCBpbiBjYXNlIHRoZXJlIHdhcyBzb21lIGFkZGl0aW9uYWwg aW5mbyB3b3J0aCBjYXB0dXJpbmcuDQoNCkkgaGF2ZSBjYXB0dXJlZCBhbiBob3VybHkgbG9nIHRo YXQgY2FuIGJlIHByb3ZpZGVkIGNvbnRhaW5pbmcgdGhlIGluaXRpYWwgZG1zZywgenBvb2wgc3Rh dHVzLCB6ZnMgbGlzdCwgemZzIGdldCBhbGwgYWxvbmcgd2l0aCBhbiBob3VybHkgY2FwdHVyZSBv ZiB0aGUgcmVzdWx0cyBvZiBydW5uaW5nIHRoZSBhYm92ZSAnZGQnIHRlc3Qgd2l0aCBhc3NvY2lh dGVkIHpmcy1zdGF0cyAtYSBhbmQgc3lzY3RsIC1hIG91dHB1dCB0aG91Z2ggaXQncyBjdXJyZW50 bHkgMi44TUIgaGVuY2UgdG9vIGxhcmdlIHRvIHBvc3QgdG8gdGhpcyBsaXN0Lg0KDQpBbHNvLCB0 aGVyZSBzZWVtcyB0byBiZSBhIHByb2JsZW0gd2l0aCBteSBmcmVlYnNkLWZzIHN1YnNjcmlwdGlv biBhcyBJJ20gbm90IGdldHRpbmcgZS1tYWlsIG5vdGlmaWNhdGlvbnMgZGVzcGl0ZSBoYXZpbmcg c3VibWl0dGVkIGEgc3Vic2NyaXB0aW9uIHJlcXVlc3Qgc28gYXBvbG9naWVzIGZvciBteSBzbG93 IHJlc3BvbnNlcy4NCg0KLS0NCkFhcm9uDQoNCg0KVGhpcyBtZXNzYWdlIG1heSBjb250YWluIGNv bmZpZGVudGlhbCBhbmQgcHJpdmlsZWdlZCBpbmZvcm1hdGlvbi4gSWYgaXQgaGFzIGJlZW4gc2Vu dCB0byB5b3UgaW4gZXJyb3IsIHBsZWFzZSByZXBseSB0byBhZHZpc2UgdGhlIHNlbmRlciBvZiB0 aGUgZXJyb3IgYW5kIHRoZW4gaW1tZWRpYXRlbHkgZGVsZXRlIGl0LiBJZiB5b3UgYXJlIG5vdCB0 aGUgaW50ZW5kZWQgcmVjaXBpZW50LCBkbyBub3QgcmVhZCwgY29weSwgZGlzY2xvc2Ugb3Igb3Ro ZXJ3aXNlIHVzZSB0aGlzIG1lc3NhZ2UuIFRoZSBzZW5kZXIgZGlzY2xhaW1zIGFueSBsaWFiaWxp dHkgZm9yIHN1Y2ggdW5hdXRob3JpemVkIHVzZS4gUExFQVNFIE5PVEUgdGhhdCBhbGwgaW5jb21p bmcgZS1tYWlscyBzZW50IHRvIFdlYXRoZXJmb3JkIGUtbWFpbCBhY2NvdW50cyB3aWxsIGJlIGFy Y2hpdmVkIGFuZCBtYXkgYmUgc2Nhbm5lZCBieSB1cyBhbmQvb3IgYnkgZXh0ZXJuYWwgc2Vydmlj ZSBwcm92aWRlcnMgdG8gZGV0ZWN0IGFuZCBwcmV2ZW50IHRocmVhdHMgdG8gb3VyIHN5c3RlbXMs IGludmVzdGlnYXRlIGlsbGVnYWwgb3IgaW5hcHByb3ByaWF0ZSBiZWhhdmlvciwgYW5kL29yIGVs aW1pbmF0ZSB1bnNvbGljaXRlZCBwcm9tb3Rpb25hbCBlLW1haWxzIChzcGFtKS4gVGhpcyBwcm9j ZXNzIGNvdWxkIHJlc3VsdCBpbiBkZWxldGlvbiBvZiBhIGxlZ2l0aW1hdGUgZS1tYWlsIGJlZm9y ZSBpdCBpcyByZWFkIGJ5IGl0cyBpbnRlbmRlZCByZWNpcGllbnQgYXQgb3VyIG9yZ2FuaXphdGlv bi4gTW9yZW92ZXIsIGJhc2VkIG9uIHRoZSBzY2FubmluZyByZXN1bHRzLCB0aGUgZnVsbCB0ZXh0 IG9mIGUtbWFpbHMgYW5kIGF0dGFjaG1lbnRzIG1heSBiZSBtYWRlIGF2YWlsYWJsZSB0byBXZWF0 aGVyZm9yZCBzZWN1cml0eSBhbmQgb3RoZXIgcGVyc29ubmVsIGZvciByZXZpZXcgYW5kIGFwcHJv cHJpYXRlIGFjdGlvbi4gSWYgeW91IGhhdmUgYW55IGNvbmNlcm5zIGFib3V0IHRoaXMgcHJvY2Vz cywgcGxlYXNlIGNvbnRhY3QgdXMgYXQgZGF0YXByaXZhY3lAd2VhdGhlcmZvcmQuY29tLg0K From owner-freebsd-fs@freebsd.org Tue Jun 20 17:16:35 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EA321D9D5C7 for ; Tue, 20 Jun 2017 17:16:35 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) Received: from NAM01-BN3-obe.outbound.protection.outlook.com (mail-bn3nam01on0085.outbound.protection.outlook.com [104.47.33.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2E75A670C4 for ; Tue, 20 Jun 2017 17:16:34 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weatherford.onmicrosoft.com; s=selector1-ca-weatherford-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=toPZ4M6HGVap0fEigG2yG+UVU+5CY7tJwEMtbuq7Xz0=; b=IGOe0CbScQrwGTyB06D3HoMBUwzzPXjSgXpwudOJMVYB5m2Trr8HStQEeRysudOzLe9x8a/mSdujicGBlw7jC1O5h8k8f4dsm28HHVBdK3dSpnGvc6fGfp16cIcbhO3fs0aTwkXHEnno0JSuPNz73NL5nIBbG6RC8Q/PZ8OVb7s= Received: from BLUPR0301CA0010.namprd03.prod.outlook.com (10.162.113.148) by MWHPR03MB3166.namprd03.prod.outlook.com (10.174.174.35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 17:16:31 +0000 Received: from BY2FFO11FD028.protection.gbl (2a01:111:f400:7c0c::120) by BLUPR0301CA0010.outlook.office365.com (2a01:111:e400:5259::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1199.15 via Frontend Transport; Tue, 20 Jun 2017 17:16:31 +0000 Authentication-Results: spf=pass (sender IP is 23.103.226.20) smtp.mailfrom=ca.weatherford.com; freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=bestguesspass action=none header.from=ca.weatherford.com; Received-SPF: Pass (protection.outlook.com: domain of ca.weatherford.com designates 23.103.226.20 as permitted sender) receiver=protection.outlook.com; client-ip=23.103.226.20; helo=032-smtp-out.weatherford.com; Received: from 032-smtp-out.weatherford.com (23.103.226.20) by BY2FFO11FD028.mail.protection.outlook.com (10.1.15.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1157.12 via Frontend Transport; Tue, 20 Jun 2017 17:16:30 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net (141.251.110.81) by DM2PR58MB015.032d.mgd.msft.net (141.251.110.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 17:16:29 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) by DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) with mapi id 15.01.1178.018; Tue, 20 Jun 2017 17:16:29 +0000 From: "Caza, Aaron" To: "freebsd-fs@freebsd.org" Subject: RE: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Topic: RE: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Index: AdLp6PSQo+GaVjnpTieZ/TkgLGTt0Q== Date: Tue, 20 Jun 2017 17:16:29 +0000 Message-ID: <36ec1fb476e647a78c19463eea493859@DM2PR58MB013.032d.mgd.msft.net> Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [141.251.205.196] x-ms-publictraffictype: Email X-MS-TrafficTypeDiagnostic: DM2PR58MB015:|MWHPR03MB3166: X-MS-Office365-Filtering-Correlation-Id: 6c2d23ae-64f3-4235-aec1-08d4b800181e MIME-Version: 1.0 X-OrganizationHeadersPreserved: DM2PR58MB015.032d.mgd.msft.net X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: =?us-ascii?Q?CIP:23.103.226.20; IPV:NLI; CTRY:; EFV:NLI; SFV:NSPM; SFS:(100090?= =?us-ascii?Q?20)(39850400002)(39400400002)(39840400002)(39450400003)(3986?= =?us-ascii?Q?0400002)(39410400002)(2980300002)(438002)(199003)(189002)(37?= =?us-ascii?Q?7454003)(13464003)(24454002)(9170700003)(45080400002)(773600?= =?us-ascii?Q?2)(102836003)(356003)(3846002)(6116002)(189998001)(108616004?= =?us-ascii?Q?)(260700001)(106466001)(790700001)(19618635001)(72206003)(47?= =?us-ascii?Q?8600001)(966005)(1680700002)(15974865002)(5890100001)(247360?= =?us-ascii?Q?03)(84326002)(2900100001)(2501003)(86146001)(861006)(6692600?= =?us-ascii?Q?2)(86362001)(7696004)(54356999)(22756006)(5660300001)(678660?= =?us-ascii?Q?02)(6916009)(7906003)(5640700003)(606005)(99936001)(733005)(?= =?us-ascii?Q?54896002)(55016002)(54556002)(38730400002)(8936002)(6306002)?= =?us-ascii?Q?(236005)(512954002)(8676002)(81166006)(9686003)(53946003)(11?= =?us-ascii?Q?0136004)(53936002)(6246003)(2351001)(2906002)(53546009)(6959?= =?us-ascii?Q?6002)(42882006)(229853002)(66066001)(33646002)(50986999)(562?= =?us-ascii?Q?774006)(299355004);DIR:OUT;SFP:1101;SCL:1;SRVR:MWHPR03MB3166?= =?us-ascii?Q?;H:032-smtp-out.weatherford.com;FPR:;SPF:Pass;MLV:ovrnspm;A:?= =?us-ascii?Q?0;MX:1;PTR:InfoDomainNonexistent;LANG:en;?= X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD028; 1:daGLGdk0yxliVuXU9ExqUlBf9qZwXnrUa8XrarlwQzW8MFIlwJODFAzHAnkr22suGxnMTjX20Y8mLUbNdgLCD55Bz53IZvYwrxRxKD6R8in4HOukNuEmaHnsjIgLozZtxNL0B/l/zJ+ZG6b8X1gPgMrs41vPKnyx2sk+YLpkNzrnISS+uFdwvl1oaZ0jy12me/446jHNWDXQD9OBgHLv2qh+I/wZMzO8VxBQmL3HCOBezeg1u+5LrKpjoH0NUVpab2q0jofPRTkzyHTgxSS6Gq5LccWlvwoNOW1aK6rpWNzMAbXchzNg6zImNMvSy2/wAWVQSiEjwp7Qt1tNT1wOGmbwWAk0aVa9jpBvPG/gprYZc2aXuP4hX2YaQEJXMRC4TBa57HUA9LwyKEyB2BMrIE5eSpeVgdJNEpNcDUaMS4lmWOiIuXP+6K/OdUycqX6WOIHC4iMZqzJalPYp6hnzU1XiQqwRxrM16tM0e4ny5DuV4lAg+xyw4Y5zs2sg7l5v666d14TYQIjKOBx6uhLd5p7LOlmvWXvEujfE/U1P/6kwrBGorQ14mkYrNbmPei0UiZ3/OpNeBke/6FLNwRsdZWESzyVrv6EMvkb46biibvZpyaz3q625LOrmaK/EZ8hA+zv+Txt54TCbgkf60qBQBRwVEeWyJ7GaIMb5hlcX789sxTAPqEnIdWZlESi78UgqQ55I8GfzN7GBfUKSsMDJ9IC/ezRMdFEz0Kz3ya24W96zGAtNYxkqzd2Obo0KtJhXrAtKNEP3PGh8h7MjijRKxV0ry4TAsgMC2PNrcUAV4YQ+806Bwz32wGEB2oaL1kDL8RxZy+DkKwDi6Gx4Q/sK0N36Q99Vla9ZQtP0KuxPJI2V8nmetbx3/mwaBs/7TZdZ X-CrossPremisesHeadersPromoted: BY2FFO11FD028.protection.gbl X-CrossPremisesHeadersFiltered: BY2FFO11FD028.protection.gbl X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500055)(300135000095)(300000501055)(300135300095)(300000502055)(300135100095)(22001)(8251501002)(2017030254075)(300000503055)(300135400095)(49563074)(201703131423075)(201703031133081)(201702281549075)(300000504055)(300135200095)(300000505055)(300135600095)(300000506048)(300135500095); SRVR:MWHPR03MB3166; X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 3:999/iWb2t1u4OBhtvd4my13GP5nQx7PeXZS+z2oPnXjfES5m2sdf/q1J3CLy8b2zgC9jzDWu1Kifp7r7WErgHvf3IONBdA6b7v6UgChSgm2l9rwcq0L6rOqPm5fJ8VBl3GYQHhyguqeL4Cngz7vPgm9CmisNvA0kzuim8YHWB7fDI8g4F5aIXXOFZ+B34iA+XYWyq3ZX38Sp0CpQXZI53bTUPSVBFLoc20e7vSWrxVJvt0DNQ2Tq0PxRiFWh6DWu5G9YFB9ru6WjyTsFHZvtM65Flo2Acy1SKpd9TkwMEY6JxV6ELvNuAcaTP0zH3Tc12kyQf6SJe169OiqFCIoRTCMdRrLjo0+0mlxRcdzBPvtjyZWG7UoirpinHcjd0V3TkpZfs+L+EXkE2LHEqkI0XSX9HZLwTZ+3h2UG+4Co6j1VxDWedzOShVWiiyow05KUaBUVVA2pysF+rdeBlM7/NAatB+Y9MA3pVaWdpFbnowSv6dJ4zjNdM/xd+S0pGqZpXoeNSJWsWeSGoSdfKXPLQfX5pG4CPVpDTwJIeUb5Qxc6Q39tBzNhkmpbGrMWP8N9j00obF2KIKBhdd+DUK9DJFRS/Rm4ojMZq/vY+oQaADpQhF3OWSNGOUmGKmUi1/rrkXYIJuxVMnDM8tQgBgN0Z+KmtXmq+xsZgoY3yggxCc/Br9/pT6I9rtFTp4+A7ColfG+BBEBBY/7r2Zd7sf7bxrMwvdF0tLpn9Dp6OWpgVAP8GRA35jzhEjTfljMaEkQk9cS29UvGzpN5n92kQoFaUuWGJ21zRluue3X/yS/Utsm5mlUx5INRPhZF+QbX1cbMU8CBXWWXa7h4m+NdSlZXnKjMu3puaYwBUCjTowAOZv/DNI4xVqTkoMfAarYVIkDmIVb+UIyw1EP1oiprAlBkq9O4NUIPKbvjRZrRSg1KOhzkspClmoLP6keAV+KMJeh2 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3166; 25:WYi1Bs1uQxzg8v1t69IYQaLRqNQ+OdHtlQmKp4UOt?= =?us-ascii?Q?b/+jnIGnH0uxt7Q/9jwQAiYvKiZx2uVxO+72cac2rd7G3sSwZZdrV/lDNFWv?= =?us-ascii?Q?FWZTg5juIkjat1a0y215oVaHvzv7HwaEg5iOaxGeVXTjA7ldIQf7qTqwP4vq?= =?us-ascii?Q?5q6LSAiKWFJ3sKzS5tDwCfV//DfYl6phD37Bw+fAYNEljkCqxyjxpaxvTYkr?= =?us-ascii?Q?qG96C1+kOkzwobAgFv45AhtkkfWBhPyJPPhqijiuy5VcpbJl3Lga16dd1g1f?= =?us-ascii?Q?my5XIujNe+QoibnjvIk009LJKV3WhUrxAjHlH+dmVt5E2Fr2zd+WtHRVrsLD?= =?us-ascii?Q?fFfa+KW+fcd1/yZArWKVuj+m3C5ZNNFchESm4HPKlbe3x2gKJvsQeu4TuhNE?= =?us-ascii?Q?jWXLCnEm0C5YIhb5M9MPGwc7qwTNrX+lb4MLA6/TFmLIMwavGFBbYz9L6OHM?= =?us-ascii?Q?9IX5+Kry6eNwW19L2MogHxXoZ6vNaL4yEZYpDPm7yQgZK1TUqnzZNwFQXKIJ?= =?us-ascii?Q?5AIlA4FO2KCxT3LW+MENclMbPprHDfMMRDxle0/6oM4ZDXz8OVWGa6fXii9m?= =?us-ascii?Q?x2x41zzlhfHyOS9YNcW2t86WAMkD+axOiAFVT0V3J6Bc5Zfo7nNjSEJGa0jS?= =?us-ascii?Q?WAbAsPYxTR48z6ywiy1AffQqbEsXXOHSejfitOUaoMls3Vg8M3rZqgxE1It4?= =?us-ascii?Q?O/Heh4O/OtMUhqgegEdRgPqf5WO+puSKZgwmb11wlTx4SLnSDkGGN1SsQdsG?= =?us-ascii?Q?iKPXNbGc2AYqYZ8StkXaIlkTEPABZ5FOlMCRnpxHDOEq0X9dNycG23modcnj?= =?us-ascii?Q?NSn+n1R1nSLRi1M3HQGO2gJg6xcSLgEBDp22050uww050gvxr50F4TdXkRtE?= =?us-ascii?Q?IWT+Qury8SWFEGzOkOo0T7cIOW1tDunzxA15jvY7Phwk1DZSIIGVb5mFvS3r?= =?us-ascii?Q?+bKmZfEmEK1ZV57N0J/2k1XSJzqUC3UDJyhIc1cGg=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 31:KXOCy+WxsgPROjhD+9pEN3VzM5GY8fRgDBWUmfYHOZZfCqI7mYln9DFNgY0CbviqqX+saSnmyslSVZk+RqpeqPSE3MUUo44gb4JTxhsHE159iZt3SfwGEQKWB5udVt4uKcEGKJdiNBw0DAjG1snkvm2Z7SPciH9U1sFx1mUTpYpS+k4CyIvW6sdRYsPMLxMmIpTN9eBTA6RLviqGTRO0FlP0yszzbCIpZz2F0Q6iAvobEiJ/SdMbDbxE0IeonCirMsA6AoR18rEXldxupjmg2a5Vq6K6YflvcQQkCagkljpeugHx/hydkX95JkoFMHjhtzmqrpvssQdpO5SFECqiGsgON9NzIOk3UYYQA70lLge9BjpOuzbAMyruhUKhK9U5IJzuvjBK2Rf5LcnVHzgmfL/xoFoDhDyeuJhjPHwZBQ7XRCUxlCxfSPDr1FRjwEqebAHZAmyPO+Elxtc2UOlZpoixud0ZTjXN+A1gtE2R794GPOQpmOoUSuDuTEvO5gNIigSsQIS2fzXav9hrkKoojaj48Sej33xCyi7Q/djzTL66U1QPya2c4X5eLH6u9eXttNd6fjJS2k/nXKtDuu6CkK+Y0PS4k0XU9p58IQcIjpeVA/1VPQ42kLNVXcfGypHpA7zrBTMsnejAMsOjPvDNi5Fe96jcYvqMJeh2Av9MFtHWkfZRY7pHuVDyDltwMo2zmCcIh0voU0WfbDFOYCqvIFAqDfcr+L1Q79xwB3i+T9UjirUX9NbdLt8cnar/JyCH X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 20:x0IY2DWHFZ0k4aCMJdIGqfzHXHMxyJBcdDblqBePUgSar9O1EjDK/yfjZeIKki2xYf1fLIw7or7EwOT3KIrXnmbegxY6j8poVGANjqwznzRLUc8pg2122MkvGVlUm5ch6jNh2ykEw8e2jehTYlB3MP8Obh1unr5S38oFWEdgiTvJkgM4Pv7vW+nVuTTlHXkFrBMO14pZqtkASFv9ja58jkgPuqVS20K4WXa9vI14SMdyumHlmHEfU0UNHHENtKct1wGg3HN3zkztYlD/gHMqE7VTXKvzpVzhnP4kXLiXHqxoJspwijV0Z5dCipxy/gMrXZWwjKL9SCRSaFFYkQh79U0ThEW1pARrcZUdd8svLzYq3lmDbnzeKaib9YQ7GpdrUktsGUU4D/+vc0U6W5v5kdoU5mwWkL+w4hqkqgib2HTc/TIfhnD7kSjkS3akOwBZLyTcN2gXabv8mZ6P/cKkuY77ShNK7KyPDwwFQ3kbXAXBxulpxdyf6PnT6OzoIzsL X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(158342451672863)(84480959824636)(209352067349851)(192374486261705)(31418570063057)(59266950573212)(128460861657000)(254730959083279)(86561027422486)(21748063052155); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(102415395)(11241501159)(6040450)(601004)(2401047)(13018025)(8121501046)(5005006)(13016025)(10201501046)(100000703101)(100105400095)(93006095)(93004095)(3002001)(6055026)(6041248)(20161123562025)(20161123564025)(20161123558100)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(20161123555025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:MWHPR03MB3166; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:MWHPR03MB3166; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3166; 4:/+PaV3mKbP69u3VkD9HUjx59AzUViaX/Z0D8m8aq/U?= =?us-ascii?Q?YnHInMzRpfQAKfexK/uo+J7dtaA22ByuIKMOoQbHq/SNzplRIR5OEh91hMox?= =?us-ascii?Q?WFJACVWDbqobre/Rp7KKfHLhUBdhmJvEMl3+UMTsEweQtKophc17g3TzWUjw?= =?us-ascii?Q?dmeeYxInhzmWCbEAyAErVSWkAKBNS5UE9LBOebc94TfRNjVEMRBSRSCUspyp?= =?us-ascii?Q?gwyq1or5pfYEEiZHOXODn4rSIE1UczjGFUgqLYAT7S/C2veonTiknDTLAC7L?= =?us-ascii?Q?mccHgol6isIhmcBEKCajnPZ2igXiucFmQnhDKawkVpGuqLb6An6zmCHxlqHX?= =?us-ascii?Q?KuezPPfSzjmn3kQ33z9fy3MWaFndV/FRbSVZUpGxZztqO5uJcpD+pOPjz3xq?= =?us-ascii?Q?JmlPfoAVMSR2Xz9mmuK7mxDE8rADzl4eWfxSoE2fHu8BYA9s8aoRS0b9jnW7?= =?us-ascii?Q?TxZJWI9sHRVqJZZ79ags+lar+K8uyyg6yVliWm0Ddkvf25xWmDkLrgfpArf9?= =?us-ascii?Q?ytDZ4z1a/9Yq6fp0DISth3limYr4OF6IYmGEd197ZSn4gx7Cm9Jchm2Cdvdx?= =?us-ascii?Q?WaI5JbNG9SVnYAGvh90ovkIlq7aC0UAVx6dzTbOm4DhBuJ2t/sKwh3/HgT5h?= =?us-ascii?Q?WRtXNi5dAtqrItb0bs4MG4Y7j+J65Zxk0e3rrWUA5QuWXRJQKh2ii1KEF8jM?= =?us-ascii?Q?n0BerccdGMsLAof0fo4Ff7NVnw6y/d1eeWfvDGhBJ5wDesFl9B3aLOO1cgH3?= =?us-ascii?Q?TWJ8Jp8dXGebKI3XDGEyQW9IbQqLMiDQe61Knt9Bu56RrsSbeqVP5fdCGRin?= =?us-ascii?Q?Seql23kG0Nv4zFCPwd8jKUS1xvAtUodlSvI8wG4xvWTfcGIL5QaNX3pGBg9R?= =?us-ascii?Q?XqTWqAIIuJLKjvcMCGEaOVPy+BeSTIQ45nDrSG2JVUXHlGaAnS0+C1SDzUjK?= =?us-ascii?Q?CpslktEi3t6BArOXxa+AQuOZZVLJgxxSi8VS3f+ZKQJLPo4w2orOipLT+s4s?= =?us-ascii?Q?4Y/praU3HYYebz57wgILTAXxWG+7dSGUpBpSE3UPk02oSAcybiqW+4Ud0orw?= =?us-ascii?Q?5SAXb6bHS5xp0q3ae9A86bcqa2ubHXC1vkZMtn+2PRVeCvGjAHZaL2ZwatBg?= =?us-ascii?Q?6CCq9sz3Y++DmV7N205/6WcLmo78dvrOECEGWVlNH6gwU3a6BjCLSZhM3Vyg?= =?us-ascii?Q?4Lkq9VA1sO+aC+OMp1RA1v8QdZLiO3bDUKASE/Ff2LSjYMj1X3JEFvNc7e41?= =?us-ascii?Q?BRQdCX4ns6tVZDY6oXVOepP6A5iRCHLZOgImyjVunGXl9J5BrJa9paVCS3pk?= =?us-ascii?Q?wWMtn0yDZHsbEOam1ROdYLfQIDV6CLCrw72m8Zsd87lhiFeM5d//SVSizEZz?= =?us-ascii?Q?clXihfuAPgniQ5BDoYr4ATsnts3tTX6xpF1HJF3iUA3OlnERkeMIfIJQ7/0v?= =?us-ascii?Q?bURfOd/IMDSfDtp/MIc+cYKT4jNPOeyMavc21oct/GXOhcr8JDleZ2A3BTCL?= =?us-ascii?Q?A/wZy1Dmqujmy902uhjOd5BPYmeEYqRynNGdjtpmHUFklcGk5dk68+lBRY2m?= =?us-ascii?Q?QzIV9n2l5gQpaCS96RwSemcTKftemMM7MXCcWZiiMCmb9kd918y3aeJFjtFQ?= =?us-ascii?Q?uVxRHZ5k++rvhuJxBg2g=3D=3D?= X-Forefront-PRVS: 03449D5DD1 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3166; 23:keRT6SWjoQenjVGt4oSBTxCjlvp986sYl+eMTuTG+?= =?us-ascii?Q?Mp1EeazzUjVW77maj4sUrfoBpuMtk0uYfeDRJ1g2yw8bdlHoATy1pyZaH8Bn?= =?us-ascii?Q?enfu8L88gyGE3QzIRjYvIlxQJbm9wM/+ANI2+Kftlgt1vIZlDAfF5j6BKr5N?= =?us-ascii?Q?YVvnKOTgauSP2th3SbIrj+Ufa/h1QNU3REqhYT95OCotghM0aZpRWgRjS34z?= =?us-ascii?Q?vjQ669dazk6SKg8Z9Yn7ELV1RWPnGpo8EiQAoXLhpc28Q0kYkyDw0YeFww/Z?= =?us-ascii?Q?ep3CHerL8tNyYyorSDfbcKbolEcmNVPMbMlh2X2ql8hyO50lo4KguwsXLzz9?= =?us-ascii?Q?XKvHYhWocVGNfkkEa49gGvdlGRPVcyufNJif3ZpjZYYW2LjtWsOiYuuSQ79T?= =?us-ascii?Q?/itlbO+qfMvB1hxE05TwvSZNKGIc+59Oot7GtUayk5KL9sm89QM9jx6eroGp?= =?us-ascii?Q?7kKemg7F7E4bQSYHz4fmhewbgtOoS1UmiLyzbwS376O9d9VYfay0h6/RIMTF?= =?us-ascii?Q?fwp9KWMpSJsbR784bcSFROaTw4zHfPTSCIzFj41hobDgFOuqu02lQrU7kynv?= =?us-ascii?Q?SPGMnD20QebY1OPWfKibnmlA+DUR4HMMWKtUqVnK5QaAshgf8F00k5HYsPCG?= =?us-ascii?Q?CLh5NcFN24MjKgQeEyc41XnlO6TDYy4xlXMkRjS6bK0B7wV8WmPpgL2AesKa?= =?us-ascii?Q?nuZQUYrDgAZJINFY5ckBZx3uZiTqtMEI3eIWqRNIIDRd+M30TNmP//mHXbhb?= =?us-ascii?Q?SnNiGhU0CSUbVJGvmKqLZEC7+OzVCQcTAivZRDHFAJ8ehmZLam01mzWoLze2?= =?us-ascii?Q?uii95cKvqy45jP/YOcQ7fhWl4G5HxuaFpddhuQMEQ6aWC8RyA6GU61urbYfx?= =?us-ascii?Q?ZM2vgep+LAdxf6zORWPhHl5y0bpZoO8bP5OQjzJg2LEWNqJMMri8amfxkmj8?= =?us-ascii?Q?JDrO54usRBa3PSUQhAnGrA8qcdQk0FbmPShHHWsBuS+4YO9FxY7MEut9ikKQ?= =?us-ascii?Q?iK5kVTzyqW21G0JZKhYhvI5aMkp0UJaxeghZyJ1JyU/KAMwmJihQI9sAQtdp?= =?us-ascii?Q?uJxQKbq2zerR7lL0PjJn/MFrI+MQCgyny8rWO//l6/laR4GcM8yUysu1HDmx?= =?us-ascii?Q?L3C15H+GqIJVawKd5En8LrT23c3eyfMMZGg1jMo7NPJhTFuqBZhyY6OsSmD7?= =?us-ascii?Q?eOU31EuwqVF3RIA8RsrO2tFOfuKmjMEZeKP626dTQFeLmeR98rP81gSceUsg?= =?us-ascii?Q?m192kiNO5wYPcldMLrE0OgBqAn9FL8CkKJy0XmoKRXAeA1X7xbTeZbWGhATb?= =?us-ascii?Q?quc41rpW1RowzVH5W21YvNLQXo/Cd44t3L5XHs0iMjUnqhNViCaUa3nl+G4F?= =?us-ascii?Q?wezjCtQwr2skDTpFyJkpg1ocFJuqlqh7XKQs+MG66iomgN0jacaAPvHsZmOI?= =?us-ascii?Q?7ZOLUfCFH/lBNx7NuOq/I7ainYcj/+6IyVBEgkWiLcmMF0jJO6Fu7nK51ytC?= =?us-ascii?Q?W9qpGLaAujFh6z8IPLH2QCTuzdkOLwvvVXytzEfvwTRpkxQnN2SOBc1jkIpQ?= =?us-ascii?Q?jxvZ0eIUzj0F3NQks26PtA7on7164jEiLPO3D2agwTkE0HZbVnLYbknDbke5?= =?us-ascii?Q?67KUTMwOQG9+MNkM5AV7AOA52P5yEzV9C7PAb/yux4OkCNgj8uV9BrO6y+kb?= =?us-ascii?Q?pvqOZ4JbbNWzMWdkWs0n8MjEAiZRmmSgwA6u46xMUu/4Lp+96g9BG/9kDQjw?= =?us-ascii?Q?c5dP1Hf4dvJQ7e07AvUVs6EOCoaXdip47LBUQjBjsUcWWaxRuz9Wyiuzu1WR?= =?us-ascii?Q?3Vu6nJ1sPyeWZooS703u+upOFr2Bf9D2g=3D=3D?= X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3166; 6:sZU1vRPVJcl1hf8T+ZOnEay5ScVKcCE+RtQ9USmLRo?= =?us-ascii?Q?ZB5YtsB+f+6YLv0IVRYCyMzknITR+MwrJt6buWuOo3jUww8fOLknajT5GD1c?= =?us-ascii?Q?KjLgnIEq8VmCBMN/W61QAQkZjTtdnRWDjuqk+VCqOupy2pXpWo2L9FlJhSYo?= =?us-ascii?Q?9rltGbIBn2iIxFxWQpe5aubSkIt9GBIXqbdo6P4cH6FHdYyfj1wgZo4gA3dr?= =?us-ascii?Q?RZ079jc7q6vXcR6OXmfyJMzvz4odv2n1EI6ZY3ntOq+l5PRnRT+pn9KqxOOT?= =?us-ascii?Q?z6x1hw42Rb8aMRnsbt4cMizxg5/WUdh8xbnnjdXXBGRetiAxD81WcSUc5jcd?= =?us-ascii?Q?JtiiLY64BegrlEIC4BbcViUp2bnG2k3cPlHu3RWNemdEZFlXl//z1thCqEa/?= =?us-ascii?Q?LUlZPLfm1M4odtczUdtsmb3rLleGKweNno60yYm3P4hWHgh4wyLDQZV+POXC?= =?us-ascii?Q?SCNOnDmcbE7H4EhoxU5nIZtXXzuJcIOpH7cxzFkyE98My/gWrlE7keaeX/M3?= =?us-ascii?Q?73Hqjm+ZsCpEvIkGDFcfKjqgPuRZ1sfHekwtkk6KQAKmZ2KbYz2EtVcmiGrF?= =?us-ascii?Q?rlQOS8uXjjk028sRYTtodjhb5JO5GyTXtcIE2PNOygH0FtT0xuxhCVgKjD4L?= =?us-ascii?Q?esrWCe6svZ7lueRjxIn45G1QZ1qn7I9CUbQnAbNEg8UofIFcLy8vfDMuutZW?= =?us-ascii?Q?q0tbP84GDEz9QQQaFVr95DKS+cKlxYUA6vAtntzbKi+iXLo6SgUGvUJ/QLxe?= =?us-ascii?Q?IrGfjxflWBggPp9umj3R3WTcrcao7vWfbt8tnctH9dXPhzZxglEb3n+F0YCX?= =?us-ascii?Q?yH8LYcQaysM9W7C0sLHsJQDC5Z8tXHObA/K77IoD0dmmibPJW9/8uhCV8Jt5?= =?us-ascii?Q?uVFfBnsSYrCpVQ3yIQaejARPIyFRhQRMP6eVkZzDX1xJUwc0JUG1vav4lSYR?= =?us-ascii?Q?CZonXtZp0PRWGN+7dKiQYW3T5LfiOe65/72Me1iBSOPuElESRHMwde1JdMfs?= =?us-ascii?Q?WZT/qBmhMgYaWsZuxHaebm?= X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 5:cYVwUNY/eiNe2Mh4tHTAaAhoP91ftXQuv0yykhJpQeJzKzDqz1G57HEjPgS6YOEAxyB2AqZFito8YBYCrAO42QB+EiYCGovgag0za5uXIXX8WyiwqrAqSCME1bhI9+NWflle1wGHaIRGJZkgEi2Zi2m6SSwK9MbFxIwNQDz9ALNBHFsp7kzhJGLtad2+rRvjYCDtEIkolj6PcbJWoq+soS6Ojk9EgLXXb3bfz2omhPYq1izrSAO4dGUaEP4Z7EGwlNJYLUTlvNBNJ7gh7iUM51C93ZW6/D4/3L8AQxrT0qUTofalgzSGG43f8Nc2ZOX98BzeC14hyjCDiyC6edOf26UKHXxgdsOQuO2gv+K4w2znj4E0u11bXQHfHVhpq2bOImUIifihVLluU7Ai/gEtcBrwPL4u4OUyn0A2KmibC5YQiSLlr0o83RMR4Nkgh7OkHdWn5+yAygQ2PGUMPSdQBB82/V2aeNUNKoUWP65YjRjL1Pfls97fhkx4FaZM22S1; 24:+w431Io4i0FNMPWAC2BaU1ptsQbGSKv5qQao4K4YnMJw7ckeJBpbUE/GzSo6O3uHWTFX1uqjgqapqAsx/MnPhgTNfeTa8rVHUtRtTHpoq1o= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 7:Wb2l//aVe5wO3Gjta0+7rvbpG8/p/wfQPaibrtJ5RvzQJIo7Vd9oBzdNODBFFypccNfLZR6ZdVP5Aw4raroc/X43zdTG06TnV0BOokWDGkMQcpNHmOXY5ZeOrMAWagJP17s6P4iTMWB8fVm+SR3ZCfGk2XNa3KCr7tk8F9HUvIbWkaMIp79uKIUOPkXHY6m04MOLoiiU9VcCjIrRve3sAEOCV+NouUbuUZfUEwvkhNgyotfdlg7C5FoayEPH9+5vebc/1ve8RE1XDQqfhd9X9wKCC9JA5fPsEVp6opkFridlff6CVNlxiidlNBxwllcF8ntxx5I+3uHTpsWKMdDygTn/c76SCyDUy+LGhQS1CneSeuFySnidkyCqQod3kU4HomVTHWk4R5bqsgOKIYgZXlhUWQt7KsCSd/vN61NdRbQ4Dui79avCdnRLj/VQvkbwEMAyXkqT/wXfPYjkCxY3m8gmBBGusCFUWKAkw3zx0sl99V+8lkvViq9SM1Ym3kpoFFQs+Kkki07s+bFpkPGDspeYW07g7KwxCA+UO6Brng32NuXlYG4BIaWmT5fif+YbNEIuIpC411XQNLKeiOj8blJbF/v5GzgWnNkWz3og3mdUlVXG7fG8Yo7JK8vZYHayhwivMlpMT7k8Wcuu7GsMAsBmGPLuBDhiv4Sqbv+rV0xQvtlXL3vibeKUwsiOrc7MM+Hp2R+dTv9055kT2/OJeOSstforjMIy54yTBoHqU2TvEAHbXAhwFRJaCG358g3yuHZ+JaggiuUye9lpo4hYpneTgqMH4qBWK5uMr5xwdO8= X-OriginatorOrg: ca.weatherford.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jun 2017 17:16:30.9321 (UTC) X-MS-Exchange-CrossTenant-Id: dd63fb60-07f6-4d96-8d40-ebeca61a524e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=dd63fb60-07f6-4d96-8d40-ebeca61a524e; Ip=[23.103.226.20]; Helo=[032-smtp-out.weatherford.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR03MB3166 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 17:16:36 -0000 > -----Original Message----- > From: Steven Hartland [mailto:killing@multiplay.co.uk] > Sent: Monday, June 19, 2017 7:32 PM > To: freebsd-fs@freebsd.org > Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs > > On 20/06/2017 01:57, Caza, Aaron wrote: > >> vfs.zfs.min_auto_ashift is a sysctl only its not a tuneable, so settin= g it in /boot/loader.conf won't have any effect. > >> > >> There's no need for it to be a tuneable as it only effects vdevs when = they are created, which an only be done once the system is running. > >> > > The bsdinstall install script itself set vfs.zfs.min_auto_shift=3D12 in= /boot/loader.conf yet, as you say, this doesn't do anything. As a user, t= his is a bit confusing to see it in /boot/loader.conf but do a 'sysctl -a |= grep min_auto_ashift' and see 'vfs.zfs.min_auto_ashift: 9' so felt it was = worth mentioning. > Absolutely, patch is in review here: > https://reviews.freebsd.org/D11278 Thanks for taking care of this Steve - appreciated. > > > >> You don't explain why you believe there is degrading performance? > > As I related in my post, my previous FreeBSD 11-Stable setup using this= same hardware, I was seeing 950MB/s after bootup. I've been posting to th= e freebsd-hackers list, but have moved to freebsd-fs list as this seemingly= has something to do with FreeBSD+ZFS behavior and user Jov had previously = cross-posted to this list for me: > > https://docs.freebsd.org/cgi/getmsg.cgi?fetch=3D2905+0+archive/2017/fr > > ee > > bsd-fs/20170618.freebsd-fs > > > > I've been using FreeBSD+ZFS ever since FreeBSD 9.0, admittedly, with a = different zpool layout which is essentially as follows: > > adaXp1 - gptboot loader > > adaXp2 - 1GB UFS partition > > adaXp3 - UFS with UUID labeled partition hosting a GEOM ELI > > layer using NULL encryption to emulate 4k sectors (done before > > ashift was an > > option) > > > > So, adaXp3 would show up as something like the following: > > > > /dev/gpt/b62feb20-554b-11e7-989b-000bab332ee8 > > /dev/gpt/b62feb20-554b-11e7-989b-000bab332ee8.eli > > > > Then, the zpool mirrored pair would be something like the following: > > > > pool: wwbase > > state: ONLINE > > scan: none requested > > config: > > > > NAME STATE RE= AD WRITE CKSUM > > wwbase ONLINE = 0 0 0 > > mirror-0 ONLINE = 0 0 0 > > gpt/b62feb20-554b-11e7-989b-000bab332ee8.eli ONLINE = 0 0 0 > > gpt/4c596d40-554c-11e7-beb1-002590766b41.eli ONLINE = 0 0 0 > > > > Using the above zpool configuration on this same hardware on FreeBSD > > 11-Stable, I was seeing read speeds of 950MB/s using dd (dd > > if=3D/testdb/test of=3D/dev/null bs=3D1m). However, after anywhere fro= m 5 > > to 24 hours, performance would degrade down to less than 100MB/s for > > unknown reasons - server was essentially idle so it's a > mystery to me why this occurs. I'm seeing this behavior on FreeBSD > 10.3R amd64 up through FreeBSD11.0 Stable. As I wasn't making any headwa= y in resolving this, I opted today to use the FreeBSD11.1 Beta 2 memstick i= mage to create a basic FreeBSD 11.1 Beta 2 amd64 Auto(ZFS) installation to = see if this would resolve the original issue I was having as I would be us= ing ZFS-on-root and vfs.zfs.min_auto_ashift=3D12 instead of my own emulatio= n as described above. However, instead of seeing the 950MB/s that I expect= ed - which it what I see it with my alternative emulation - I'm seeing 450M= B/s. I've yet to determine if this zpool setup as done by the bsdinstall s= cript > will suffer from the original performance degradation I observed. > > > >> What is the exact dd command your running as that can have a huge impa= ct on performance. > > dd if=3D/testdb/test of=3D/dev/null bs=3D1m > > > > Note that file /testdb/test is 16GB, twice the size of ram available in= this system. The /testdb directory is a ZFS file system with recordsize= =3D8k, chosen as ultimately it's intended to host a PostgreSQL database whi= ch uses an 8k page size. > > > > My understanding is that a ZFS mirrored pool with two drives can read f= rom both drives at the same time hence double the speed. This is what I've= actually observed ever since I first started using this in FreeBSD 9.0 wit= h the GEOM ELI 4k sector emulation. This is actually my first time using F= reeBSD's native installer's Auto(ZFS) setup > with 4k sectors emulated usin= g vfs.zfs.min_auto_ashift=3D12. As it's a ZFS mirrored pool, I still expec= ted it to be able to read at double-speed as it does with the GEOM ELI 4k s= ector emulation; however, it does not. > > >> On 19/06/2017 23:14, Caza, Aaron wrote: >>> I've been having a problem with FreeBSD ZFS SSD performance inexplicab= ly degrading after < 24 hours uptime as described in a separate e-mail thr= ead. In an effort to get down to basics, I've now performed a ZFS-on-Root = install of FreeBSD 11.1 Beta 2 amd64 using the default Auto(ZFS) install us= ing the default 4k sector emulation (vfs.zfs.min_auto_ashift=3D3D12) settin= g (no swap, not encrypted). >>> >>> Firstly, the vfs.zfs.min_auto_ashift=3D3D12 is set correctly in the /bo= ot=3D/loader.conf file, but doesn't appear to work because when I log in an= d do "systctl -a | grep min_auto_ashift" it's set to 9 and not 12 as expect= ed. I tried setting it to vfs.zfs.min_auto_ashift=3D3D"12" in /boot/loader= .conf but that didn't make any difference so I finally just added it to /et= c/sysctl.conf where it seems to work. So, something needs to be changed to= make this functionaly work correctly. >>> >>> Next, after reboot I was expecting somewhere in the neighborhood of 950= MB/s from the ZFS mirrored zpool of 2 Samsung 850 Pro 256GB SSDs that I'm u= sing as I was previously seeing this before with my previous FreeBSD 11-Sta= ble setup which, admittedly, is a different from the way the bsdinstall scr= ipt does it. However, I'm seeing half that on bootup. >>> >>> Performance result: >>> Starting 'dd' test of large file...please wait >>> 16000+0 records in >>> 16000+0 records out >>> 16777216000 bytes transferred in 37.407043 secs (448504207 >>> bytes/sec) > Can you show the output from gstat -pd during this DD please. dT: 1.001s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d = %busy Name 0 4318 4318 34865 0.0 0 0 0.0 0 0 0.0= 14.2| ada0 0 4402 4402 35213 0.0 0 0 0.0 0 0 0.0= 14.4| ada1 dT: 1.002s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d = %busy Name 1 4249 4249 34136 0.0 0 0 0.0 0 0 0.0= 14.1| ada0 0 4393 4393 35287 0.0 0 0 0.0 0 0 0.0= 14.5| ada1 Every now and again, I was seeing d/s hit, which I understand to be TRIM op= erations - it would briefly show 16 then go back to 0. test@f111beta2:~ # dd if=3D/testdb/test of=3D/dev/null bs=3D1m 16000+0 records in 16000+0 records out 16777216000 bytes transferred in 43.447343 secs (386150561 bytes/sec) test@= f111beta2:~ # uptime 9:54AM up 19:38, 2 users, load averages: 2.92, 1.01,= 0.44 root@f111beta2:~ # dd if=3D/testdb/test of=3D/dev/null bs=3D1m 16000+0 records in 16000+0 records out 16777216000 bytes transferred in 236.097011 secs (71060688 bytes/sec) test@= f111beta2:~ # uptime 10:36AM up 20:20, 2 users, load averages: 0.90, 0.62,= 0.36 As can be seen in the above 'dd' test results, I'm back to seeing the origi= nal issue I reported on freebsd-hackers - performance degrading after < 24 = hours of uptime going from ~386MB/sec to ~71MB/sec inexpicably - this serve= r isn't doing anything other than running this test hourly. Please note in the gstat -pd output above, this was after the performance d= egradation hit. Prior to this, I was seeing %busy of ~60%. In this partic= ular instance, the performance degradation hit ~20hrs into the test but I'v= e see it hit as soon as ~5hrs. Previously, Allan Jude had advised zfs.vfs.trim.enabled=3D0 to see if this = changed the behavior. I did this; however, it had no impact - but that was= when I was using the GEOM ELI 4k sector emulation and not the ashift 4k se= ctor emulation. The GEOM ELI 4k sector emulation does not appear to work w= ith TRIM operations as gstat -d in that case always stayed at 0 ops/s. I c= an try disabling trim, but did not want to reboot the server to restart the= test in case there was some additional info worth capturing. I have captured an hourly log that can be provided containing the initial d= msg, zpool status, zfs list, zfs get all along with an hourly capture of th= e results of running the above 'dd' test with associated zfs-stats -a and s= ysctl -a output though it's currently 2.8MB hence too large to post to this= list. Also, there seems to be a problem with my freebsd-fs subscription as I'm no= t getting e-mail notifications despite having submitted a subscription requ= est so apologies for my slow responses. -- Aaron Aaron Caza Senior Server Developer Weatherford SLS Canada R&D Group Weatherford | 1620 27 Ave NE | #124B | Calgary | AB | T2E 8W4 Direct +1 (403) 693-7773 Aaron.Caza@ca.weatherford.com | www.w= eatherford.com [cid:image001.jpg@01D27566.E8E4ABE0] [cid:image002.jpg@01D27566.E8E4ABE0] [cid:image003.jpg@01D27566.E8E4ABE0] [cid:image004.jpg@01D27566.E8E4ABE0] [cid:image005.jpg@01D27566.E8E4ABE0] This message may contain confidential and privileged information. If it ha= s been sent to you in error, please reply to advise the sender of the error= and then immediately delete it. If you are not the intended recipient, do= not read, copy, disclose or otherwise use this message. The sender discla= ims any liability for such unauthorized use. PLEASE NOTE that all incoming= e-mails sent to Weatherford e-mail accounts will be archived and may be sc= anned by us and/or by external service providers to detect and prevent thre= ats to our systems, investigate illegal or inappropriate behavior, and/or e= liminate unsolicited promotional e-mails("spam"). This process could resul= t in deletion of a legitimate e-mail before it is read by its intended reci= pient at our organization. Moreover, based on the scanning results, the fu= ll text of e-mails and attachments may be made available to Weatherford sec= urity and other personnel for review and appropriate action. If you have a= ny concerns about this process, please contact us at dataprivacy@weatherfor= d.com. This message may contain confidential and privileged information. If it has= been sent to you in error, please reply to advise the sender of the error = and then immediately delete it. If you are not the intended recipient, do n= ot read, copy, disclose or otherwise use this message. The sender disclaims= any liability for such unauthorized use. PLEASE NOTE that all incoming e-m= ails sent to Weatherford e-mail accounts will be archived and may be scanne= d by us and/or by external service providers to detect and prevent threats = to our systems, investigate illegal or inappropriate behavior, and/or elimi= nate unsolicited promotional e-mails (spam). This process could result in d= eletion of a legitimate e-mail before it is read by its intended recipient = at our organization. Moreover, based on the scanning results, the full text= of e-mails and attachments may be made available to Weatherford security a= nd other personnel for review and appropriate action. If you have any conce= rns about this process, please contact us at dataprivacy@weatherford.com. From owner-freebsd-fs@freebsd.org Tue Jun 20 17:27:38 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A7E0D9D8E9 for ; Tue, 20 Jun 2017 17:27:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0B69C67676 for ; Tue, 20 Jun 2017 17:27:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5KHRb9k036181 for ; Tue, 20 Jun 2017 17:27:37 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219224] [zfs] sync hanging in _cv_wait zil_commit zfs_sync sys_sync Date: Tue, 20 Jun 2017 17:27:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: t@sdf.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Overcome By Events X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 17:27:38 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219224 Tino Reinhardt changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Closed Resolution|--- |Overcome By Events --- Comment #3 from Tino Reinhardt --- I'm pretty sure I haven't seen this issue on 10.3-RELEASE and now I fail to reproduce it on 11.1-PRERELEASE, etc., all on the same hardware as before. Working versions: n-3-b% uname -a FreeBSD n-3-b 11.1-PRERELEASE FreeBSD 11.1-PRERELEASE #0 r319653: Wed Jun 7 15:17:47 CEST 2017 root@n-3-b:/usr/obj/usr/src/sys/DEBUG amd64 tools-1-b% uname -a FreeBSD tools-1-b 11.1-BETA1 FreeBSD 11.1-BETA1 #0 r319852: Mon Jun 12 15:1= 2:38 CEST 2017 root@tools-1-b:/usr/obj/usr/src/sys/DEBUG amd64 nagios-2-b% uname -a FreeBSD nagios-2-b 11.1-BETA2 FreeBSD 11.1-BETA2 #0 r320148: Tue Jun 20 18:07:08 CEST 2017 root@tools-2-b:/usr/obj/usr/src/sys/GENERIC amd64 DEBUG kernel config is GENERIC plus options INVARIANTS options INVARIANT_SUPPORT options WITNESS options WITNESS_SKIPSPIN options DEBUG_LOCKS options DEBUG_VFS_LOCKS I've had to remove DIAGNOSTIC for performance reasons. With 11.1-RELEASE upcoming I consider this problem solved, although I fail = to spot the changes in svn log so far. Thank you. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 20 17:29:49 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C05DAD9D967 for ; Tue, 20 Jun 2017 17:29:49 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) Received: from NAM01-BN3-obe.outbound.protection.outlook.com (mail-bn3nam01on0043.outbound.protection.outlook.com [104.47.33.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3619F67722 for ; Tue, 20 Jun 2017 17:29:48 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weatherford.onmicrosoft.com; s=selector1-ca-weatherford-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=idzRNoQReN2sA2GI37518VdEw4SRBwa1raXgKosCFD4=; b=Yv4/BcEWs5XMFLzfO5fzoQWD8xUsRm6pF57jNDxQAI/CZ8wYq6bRhrebY1r0uPRq5K2rlCVJW4ll8AksvaEUj4y9niltfA0Rs9edOE743wzYy9kUSSF1D35inTT5MpVNscZLk5qXGV9Zx1iofPTnQ1/NcvhePUKtTQPShz83xmE= Received: from BN3PR03CA0111.namprd03.prod.outlook.com (10.174.66.29) by DM5PR03MB3164.namprd03.prod.outlook.com (10.174.190.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 17:29:47 +0000 Received: from BN1BFFO11OLC002.protection.gbl (2a01:111:f400:7c10::1:162) by BN3PR03CA0111.outlook.office365.com (2603:10b6:400:4::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14 via Frontend Transport; Tue, 20 Jun 2017 17:29:47 +0000 Authentication-Results: spf=pass (sender IP is 23.103.226.20) smtp.mailfrom=ca.weatherford.com; freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=bestguesspass action=none header.from=ca.weatherford.com; Received-SPF: Pass (protection.outlook.com: domain of ca.weatherford.com designates 23.103.226.20 as permitted sender) receiver=protection.outlook.com; client-ip=23.103.226.20; helo=032-smtp-out.weatherford.com; Received: from 032-smtp-out.weatherford.com (23.103.226.20) by BN1BFFO11OLC002.mail.protection.outlook.com (10.58.145.13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14 via Frontend Transport; Tue, 20 Jun 2017 17:29:45 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net (141.251.110.81) by DM2PR58MB016.032d.mgd.msft.net (141.251.110.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 17:29:44 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) by DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) with mapi id 15.01.1178.018; Tue, 20 Jun 2017 17:29:44 +0000 From: "Caza, Aaron" To: "freebsd-fs@freebsd.org" , Karl Denninger Subject: RE: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Topic: RE: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Index: AdLp6XXjvHQ23GL/ReyOIakk8di/hw== Date: Tue, 20 Jun 2017 17:29:44 +0000 Message-ID: <2d8a525e5355406aa7b8bf6690101008@DM2PR58MB013.032d.mgd.msft.net> Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [141.251.205.196] x-ms-publictraffictype: Email X-MS-TrafficTypeDiagnostic: DM2PR58MB016:|DM5PR03MB3164: X-MS-Office365-Filtering-Correlation-Id: 46495f4a-6261-404c-7591-08d4b801f24d Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OrganizationHeadersPreserved: DM2PR58MB016.032d.mgd.msft.net X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:23.103.226.20; IPV:NLI; CTRY:; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(39410400002)(39400400002)(39850400002)(39860400002)(39450400003)(39840400002)(2980300002)(438002)(24454002)(199003)(42174003)(13464003)(51914003)(189002)(377454003)(9170700003)(478600001)(46406003)(24736003)(22756006)(305945005)(5660300001)(50466002)(66066001)(47776003)(189998001)(86146001)(72206003)(86362001)(50986999)(53546009)(5890100001)(2501003)(54356999)(97756001)(6116002)(38730400002)(3846002)(102836003)(6246003)(7736002)(7696004)(53936002)(23726003)(2906002)(106466001)(42882006)(356003)(33646002)(6306002)(108616004)(229853002)(2900100001)(55016002)(9686003)(8936002)(8676002)(8746002)(81166006); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR03MB3164; H:032-smtp-out.weatherford.com; FPR:; SPF:Pass; MLV:sfv; A:0; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BN1BFFO11OLC002; 1:w0W4vlSqElpTOEq3eLUV0kLs2DXCnSbV++KVuJc9mVzqPHpzIbbRiZJni7wPMzBo+bpVHsdhSPL3PNfCZMyJDU8VPolb10NW67e79Oq6fkalb7oeyqj3lz8pn+vJimKGI/3uHIYvhDsPDh403nScPrxq3TYXvFIEkL3DBh8UyhOqFwoN+c1Gsa6U+QWmhPwoGcUoNg/pOVcpYwydqSXUCa6C2s3uwDkvoRj3sUebkx14AfNyi8N6gSpmmqMO+iAeEJJM3lzBeElPSjrl5ogQBFKaKGjnOElqXu3/NJY/u3Jm9UT1y9wFluDusXfVNQFC4Ctqvp94Qltnq+VdLTudduAQ1mSLjlgUIgmb1U51Ymo/gdT/Q76xNOieVO0zIu9BRqyvLH3/AE7rZzGkhBKVx+z3p03W0jX/ZcCYyDjR+8Lc46efgeUkd6aFySngK0kVmQs3SKB2Ma7oBp4blwtXhq9njhGVvmdIMqBU/BEpuZVruZELAbgEspi6mpXZXBCo/nQQBcwGaDhID/hxpdu2+Q== X-CrossPremisesHeadersPromoted: BN1BFFO11OLC002.protection.gbl X-CrossPremisesHeadersFiltered: BN1BFFO11OLC002.protection.gbl X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(8251501002)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:DM5PR03MB3164; X-Microsoft-Exchange-Diagnostics: 1; DM5PR03MB3164; 3:4/IPKR1GJxIzZFDcb0vc9tku0FLUkSu/5Ex2qXSWqIRNemW9EIgy4QCGZKutpJmOVgZmg5rOmNiQwcBte94vb1kvMSnVGO2CinfoRv+DPJekfYqMDRXq7Z05USE5CEbRYogXx1G/1ZyqiDnpCfahaW83Zcd6g4MykkOyyh2d7C5o7dvG55O4i29aJQvi7ofpxLBNZ1X1qY8oTv93SL2+UVcO85OTCUhEhtlT1tzAM3qZGf5EOoOORh6EAv8NkU6X0RkGFbfAImDqJrVjsTkp/xcuyQ7BYmD3pC6rPLHjL7XSYkKfuKS8hZygFVShomE7JYp10KoE5ZqQn8aEGVkdvFSBJFahZcmZmWgeapC19rksekBAcuEnfWGPJ27E2HjxQ38f/TRFPElt/vShYZk7w89JQSKIp2mo7aztI7W7mjyfKTOURBB1q0eEUu7b+Qiy424SP5ZffkuE1wzzvPpNvPFVV6quf6R4jDW1Dvfk+bYKkdivkevL+DaoL9Yz0hgJ3QFdtR7+iUeUXxbwjB/vaw== X-Microsoft-Exchange-Diagnostics: 1; DM5PR03MB3164; 25:+zSRoQqrVWdfKP/9zvWhEqjz9jOA5fL45zPYKfTNpYJvZ3mjXozHoKhVYYd7NM1C9MFO8EM6nV81rk1SnNGhjCl1l7p7eO02w6aMIjx2uRtaxsdgXWfRj2jEwN2fZ5UgkunSYkzni/RZOadtkDTyaBFkn8m4etfY/0bofce6T/oYOVAmAb9mb2hlW/1FjSpluDjSZBOl6KryOs519pRdn5zusUEeA2EBusELtYSppJZVnQ9wm9yyY3KwU4XLCrN+9tumH5LejnKAqdwlD39AriL9PLJ02EIXgsCxxKGkEdK06PomshsUFcxdPtE5BFwoNnRkmDaBCML7LJLZgjvH+CW7vgCVIDit6RnVl17NOCsot54bJTQlIPwKaNuTTiiBbdC6I6emKHyns91bpDfDClJMFhMQyPgL5SNGjbjPXwmkLmKxb6idWRJQWDSCWc2oGBlnb1QlzsBLzn2TTcdBOawChrU2qLsECCC2RPrtiO4dn7w13Gg6IRgu4l896H9kXQi4NbWnkvGbhw3bkj5RXQ421CU3GBEWehXzGl+YjUo0r3rvC+3FI9B1JxRl4tQd0gdRFp2jJ2/c+SaiUUhTkLWezUNd6ao+7XNSZ4szQio1D+eHdxg+lbguOr5e3GWx9ge5glKjFr0z2yAQeXgL4jy54V2mHU5T4TVXd7rMGXErHA8qPiB9gKICCg/MafHluxH9nFnBP+6Qd+R0Q04evHBA5riKNPXWneFFM/e3I1Zaykbze0YtuVeNkpyJd6IvtS8RWqi6JrT1NIauO+jbQzFKkEM8Xqo7LVaTdYc5gsxdwtKbou6PR9aIR0yqCsDsbLXYQ/U2AeoHieX6/Z1bFQUAngeSv6D/MdvKteADFwv95Wr0x6po+gVkSKmUh4oHckzae8K06hZfOxtF2EZWmLabmxZgmeG1uQNtX27MXto= X-Microsoft-Exchange-Diagnostics: 1; DM5PR03MB3164; 31:m/nFVFkNeF2Fj9BOmgl8RWUydjshJyREx8V5C9bzX6yFLvsI6Zw+6Dmp1nsGzAORevgkm/hrqlZk9UkcT0Wuhy+vJjRY0k8Dp8DOlDwySCsZaz1bPLS07TzW8VxNy5U4Z6P76aYECTchDQLlWpgV4JbRGHnZ7vtRZI5A0EpxLk5MhsfIUrDm2h+80gpAGCYT4Xo2qaJnEvvYGQ06WoKLgkgaJgKU1QcQMnu5C+IGrOq3DgoOA51YGYa2Wfy2fqc4hS3DKSCl3dtuAbrp0Cwf9O2j+J6nuGpf5K+F+yROv+y6OQqJ7eBV0WLWYkfuoE6ZoISSnFs039yE11A0jwviu0iljF28as55rQYVgAVmzFNnY7FHd2bbrBmUCHUK63wX7ZVcSpYKMuud6TTL/TyZ0tNjwJ9KmTs0MRZamdg/6EJ5vxXyCPQ8V8BR1FXk3VoTpOdXcUUGfe8GBJCt2IAyHCZX2quUD+7dtdVPGCj7jTsW60OkkB+CD8Q6Flwmotx3JP19SnIZnB/iPlo3VBzDWOQKedRvSYC2K/BmlFiCtKdJjcZzxWumIlDjq7RPcUM0/GTONIQVGLEYT6TzeVRKO3qJnvYesm1mwqEhw1R7Hc9uga1vrE2U//VN6jTd7Lquj4hOJy95a+EAQ5EbynLLw4z/9L7nV9Y8gjT+7qzwzR5oDUhAtspScIvpAq9hWLkAGPvAPWVtLgyiFQQLooHVDg== X-Microsoft-Exchange-Diagnostics: 1; DM5PR03MB3164; 20:3n1mFrGz0uVG2icHC31QU5heJPjZQOfIDLVxr3Kq5hVHyuw48t1MF+TxWxcWFZVFCkG+DMiGWR7CsIvi2S89nJ61Ey8cplHUvFtWtFu9Zb7crK4fsko8SVb2H8YmZAtlQKgdFTX0ojpngSE67svgIsFBShTZktbfjemvA8Tt7BAdNpPyNyzX84xFt5l7E0IozeZo55MskUfZsJFJ2w26GOIH1XKRtXvKNWdkP2NpnF32mU7s3tpQxVeK7OB7dGoTj8vpc3NcipFDGtLyq44focjtY+kgUKhvp4AcwK5cz3C7vELG+V4DdLynL+bwCKgYBHStsIFSMybDahh2gttgy90WFrAlrsLUkilSP0KIPVVsFdb/PF+m7jbVvNsHFUfEJCBFW9oTBqlWIiRY61OZEIssd9hr3cIvbZcSdpDkHtMqxWy56+xkicSggLOg69L4t8AvLesNrhrxULXF9jt/n9FBydCTtk2kFSxtv2JweAWWlyHAYmHBPHN6vlf0+wY3 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(192374486261705); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(11241501159)(6040450)(601004)(2401047)(13016025)(5005006)(8121501046)(13018025)(93006095)(93004095)(3002001)(10201501046)(100000703101)(100105400095)(6055026)(6041248)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123562025)(20161123560025)(20161123564025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:DM5PR03MB3164; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:DM5PR03MB3164; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DM5PR03MB3164; 4:ph8xr4Lg/jyKRDgoABEJFvSqG6Z5qrjWx6IgvtCxH7?= =?us-ascii?Q?Xm1o1sm5/royIVgILYi/s+UFepKBdJbOm6o/NOupsZDY3UCLpa+vb2JVMAf0?= =?us-ascii?Q?aEWosOIWfxqmxmha4AGIqYtT0DSZCHJDF2/qB4mheqItHFZisWkiH7RGVQbG?= =?us-ascii?Q?jJ3zVqz6+FU5rGleokw3739+djo+Hp88wxQBimJn9A8n+V7q5yNiJZ7Z7N4+?= =?us-ascii?Q?Oen6STHHCpOKal7K8JlJndStgzmKE+0eV6MThuVe8gQLxzInY1VyjQy0CRBn?= =?us-ascii?Q?qIfKnZvRwupyezM821q34lknuKrYaE6JQ38sBqUyVLZuAkC9ckT2oXJiW9fz?= =?us-ascii?Q?rzwHpDkHobBpM3WWUk4VByOYHMoaZ2teglwOaPK7zZ0RhoFixlvlyAGMa/An?= =?us-ascii?Q?4++BfhiYT04T1LFBBURDJVQsz0D5Ok0AJyfD4TWGCZBA5FtVB8sBS1IwF+F9?= =?us-ascii?Q?lbmlBVrpzcG27tXe/KQN8/qu3PqKBeNWJYVsx13peH94Ki/cs6qaZ4AkcK3+?= =?us-ascii?Q?j7jiQ+Ln1Tm9lfJBxiFrpHcVVTxqOYiaBDj75klmUZBO2s1k1s1C9JMMAIP3?= =?us-ascii?Q?Kk6CGfjHZgzXlMD32K24QIyYyehIEw/N+4JGzTu1p99xQRp2yaT9enYqB9KF?= =?us-ascii?Q?bfHS6k2mCIKl1rtx8ECDKceS3qSWB9f1olYYGfmJQX2rkvizoJvgk8hL8Rvv?= =?us-ascii?Q?Q4ccvNi8VA7l7XTSSASGp6EFmbRBidvtB8R3zBaD4QvAmRnw+9uk7KuMWcbX?= =?us-ascii?Q?KpRsQbO9FbEPA2pDHlbad03OL2/WKrkCxQhV5waI+leAAR9QMoXNKjdloR7o?= =?us-ascii?Q?/b79ltk07QkRa2IpZOk+GUZk9NnZadz2PgdlwJ7m9dD9um9rUD49trzex1WT?= =?us-ascii?Q?r9KMuajkP4hM6n01SBSsXuXf6gkUIAhceFpgOyPsmp5elmVesX+hcWEaHlcX?= =?us-ascii?Q?gGzMT+heLINcii3rkn4sqvs0ElvJ4IwsyyNHskMpBGYwYAvM4PBvAONZS2kV?= =?us-ascii?Q?Mua2dN/Lw+F3d8jA+ABrolviXvaExQR5LKnc/L8GdAk6hwM2a6LQXc7r4fQP?= =?us-ascii?Q?gU7o4TSS4s9jhXgPH7JWIiuYxSMI5mz4eGxnbRhFmIx3E2UK1GMJ32viXXh6?= =?us-ascii?Q?t6xdF0SlTP0NOImemgTDa4+6WH0HNm3/Oj/4V7RrZM4WrmVg9qcFLTFKWopq?= =?us-ascii?Q?p8Ht7DTXlXOkEv/T3Z0wdfia+aFWWInbZKsJgW6FD/gkUzQN1bJPIYqqmZTu?= =?us-ascii?Q?Cdaw20XxYE20DT7H+qOwi5fL5H5y5MfjoOwpnu?= X-Forefront-PRVS: 03449D5DD1 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DM5PR03MB3164; 23:WGMQJRmg2QT7PP0TO99dAGbweW63ugCsBW6W+hsVG?= =?us-ascii?Q?oi/tqT0vVEFc3HVYxuLEuCG8Efu60doPwTCDHMkp7ZLZE3VhTz+fcYhmIQKK?= =?us-ascii?Q?lMqDHlz/556Z+PiV76dR7/IQEAaEwfTKnHRY9i6JEGWzGu7qLIzs6u7+GPLL?= =?us-ascii?Q?v2ERnItMHh16YSae2cYElHMlROQ4HfB3lLQT7dn1qkSsKBMeBxqXZBBhHZFr?= =?us-ascii?Q?OTd9IAC6Dtg2w8apeijM1iHn7YzFJFMnMtbxAzOo5j5GolfWAHrnwQMwmNb/?= =?us-ascii?Q?L9SkswMaaAoRaoVVvT6iccn1KqelOq7QmamvH2EGry6T/zQQ/yzwqi60oTd4?= =?us-ascii?Q?MeVkPNffmF9ij3zclbswYhNR63U+WwWTw4GovfcR5pqLsTTVxOc8S7OLMWY2?= =?us-ascii?Q?UFWynQ8U4LG6rX93ROBbtRc+f0aGQB84PYgEv4zOp8z/GmUVtwJd+4KnVPHF?= =?us-ascii?Q?t9LNcwAAWRpJ7xY/F35PIAVlNkNs0oTVrxxtjUuJYLXY6CiVgFSPUFptUYAU?= =?us-ascii?Q?CXB7jz8E6AfiDlDu4UoolePA8958r8a9KETfzcLGCjMSEbjz+K6A31Dw65hn?= =?us-ascii?Q?hAGa3rtaLxq74V+17Ryop8mumaMQD/ukJ9294FMlZAcThGrR/zRi3/BNnXWI?= =?us-ascii?Q?HTlFH/YVZcPkhgr5XZ911OKDlmkltIMqErdKEyVR7PmduTFLu1oouP1ihhl8?= =?us-ascii?Q?Nz7Ya8SO3bkJzv4mmLprxNuvMlW+cbthfX5dawQHuQHkBF0pzYBM1dPbiDcf?= =?us-ascii?Q?jMdOoP9M38L4TPWuMQ+HpwQV+3WeUHxYHh/QBC/H38MDwUhnwn6/bPsia2Wk?= =?us-ascii?Q?QzyB+/UIrZ1+eUuWbRMbZXEyAU8NLStRHm5VX63zbXq0e2Qe+1uC96iaLXYZ?= =?us-ascii?Q?vN1thPkJnZplVt1ZeTabnc40SRpnqBGG11Pp8rt3uw5RCjLpKllTgZ7p4NJw?= =?us-ascii?Q?lpOvFOAVW7ciBNwceNNMxbMfVmj2aGXFr6/xJIDu2evo7Huv3xKs5C78IaSy?= =?us-ascii?Q?O56FwjKaQLz/GflTuU4EZgHJGzi7b1rBJN710tByxnXLsGQ0ldSiEEgf9VFe?= =?us-ascii?Q?i1msyihrRIkBxUJg55qj3psMV463xoCE3ZTpHxEQeFPhVCiM6s5Gh3Yph2VX?= =?us-ascii?Q?Pp63BvknZQydDRKP/EqTdUO/eE0iWLKRsxIElmrh8SojYaUS/RZbnGToXTIa?= =?us-ascii?Q?K6fjKNHNb0PXZcE3kJxV07ksMSXjtSq3d1dxx91N/RPxIZ6jcpL0ejkrQe14?= =?us-ascii?Q?wwK2xkU3wCXoNDmtksaZO0a4fhB3VGo2V/pa0w0lu8NtwmVIqoWxCuP5rimR?= =?us-ascii?Q?S9d5Wn9KOELdDxIrH2Fj3MWruiYk28ptfJBBzYUEgIJKSZgtrP7qqeERUpqg?= =?us-ascii?Q?V1TEtMQPkRJqC4daO+i7ReLsJBjVA6QbXYlhF5wDVhD+SzYMxdfm9q/EoI3u?= =?us-ascii?Q?/+XhPBJjQxEctQQfrphwxGw7EQIRbo=3D?= X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DM5PR03MB3164; 6:+n0t02Z3W/gMUMzPzjOzmuswAB1k0b10jmmppdLZKN?= =?us-ascii?Q?ZioAY/pSTG2MqaMEofBkY1kSE07H65QH/fNmi/8mMXgbuGLu56gAz0Qc3hb1?= =?us-ascii?Q?vhk1P37QY6GpeNB9wAEEZPbgaNONAObGL+UmZYnA+ynK6tPhpztNBMK8rwPD?= =?us-ascii?Q?1ITJAx1gbS1JzoXzPKaGVZT/dsUTRAtdCijlI+pAzHCgGIzpN5ktyuMeMtyT?= =?us-ascii?Q?XFvP/nnWKLSS64hYfRO2JrRuDqGPBQrqkX3pxcJd/+BxETM3EVP2++YmMp4t?= =?us-ascii?Q?NtIwu9cTsADvdEE97NlnJc5ZK0qpPdP4NV2hOUovs2SMA2EeyWgC5HgG60ez?= =?us-ascii?Q?jwybFU+GV0t7Ud25JmUd0hHQtcc4+o5qojpnxydkRyXZrxwc8rFkrZjKKEqq?= =?us-ascii?Q?U5Uk9hupPMKTBEme8DDI207sw7yyNovQQQSnO3Bmi/bVKMeRM+n2Wja+NcIv?= =?us-ascii?Q?XuFfTzM9gfp9fwOL6TrN1dGZqC/v67nWjS7B9nCgBDNopaDzd5iBrTSfLnrZ?= =?us-ascii?Q?c/SPkIdFB8L3/D0mwCcidD0q6X7nvk/J4RB/2u2vpUdMa/LnHBBicauR9ZGA?= =?us-ascii?Q?TY0qfVHEBBGz9rdJmnM5Psq4sZ7hrUT8eTZbTtFgO0J6CHmGhvmb/D+JIyJo?= =?us-ascii?Q?rt0o5EzXrxGzt25ZSvQpyjAJ2wamV9ZFJkQuIFBhAKVoesDdNRyLkMBfG7mc?= =?us-ascii?Q?tyzhomq/dKp6pjp109RrhEnwRKf8Tj1rOdjicsj4L+8UgCT8dYJ8KHY0da0H?= =?us-ascii?Q?EzTwoXI+Fz+rTqZnqtPHY2TNLZIV8fLtw3iCCh7V01mQd1M8z6STqkpYdwS7?= =?us-ascii?Q?lnF9+p3PpjWSZOYAwBTpkoj+cHQXReSZ3SMUwEJ4t8TlTOR6dzJSNuxD/Vle?= =?us-ascii?Q?/fpIIto9ZDJd2UzotYSt1BSpbTiwbrHHwOuiBBFi4B5If20ypJIOfJrT23sM?= =?us-ascii?Q?4NCkQgYdi/87/GMIerostEoJkCpeYq6c30gIiIROKY0wPn/i0g9Z2FoCfcMO?= =?us-ascii?Q?ol2TFSVu8CFT7kCDRgRNUz?= X-Microsoft-Exchange-Diagnostics: 1; DM5PR03MB3164; 5:JtvvFe5UoYP85lmtUJu1G8VofEXGqjKxI2N5gx0kHuu+ipFIVFcY+afVPaMh1KevGx1QZvexNTkYBOxkH1lwMZjzGg6yWhO16yzmHLW/CUylnQeEwFx3QIdM3RGqq03aWUKBtlp1T3bZQbzQohywC9LvyQFRCwatOBz16bmAMnYFJeTueguu6AwtvwbgTJpKbP7tsKZ00PDJIDgZOdGgjdc0oyiNfplbdB6wAWV0MSoMjFWh1MGIdQ9/1z+id93tpeKJhrCdhlPM1V4Z2ZugLo9zfmmB6bK5tNWq7QgKejKSVl1n3HJxVdT4oiWZuPu0WPJt7yvz50ApQvjIcEsGR7QIrwsaday9hNGZd95rF3k3hn6RCuoDi+hdzu5PhwM1sj1fGTDu/QkELGFBrS0O3LFAVyfzWmmU/jtdSyndXiKJIS6mP7TD7H4MjgsxP9QAmGOou62x2yCgAvlQs1neCyYQ5/FJr3hB3mqlVG9xs80vbf9XGBgF+LzGBgAh7UfG; 24:OwGmXMWzzlknePLBUBOMbND3NISDnlAvNwWY1gas1yO/dhZCIEWw/VFQAnw5OEmGA9/xb+CI4gT3tec4LYgO6mcBK4/wAcVj95mJrr/DFAs= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; DM5PR03MB3164; 7:1RIq2lVD7BkFzXws6qMk9+SYpbYuXxSQ4L5TVuRjUtBiY3SVx5VBYi5j2tq1pOgdkhj41Fqzf7MRVqUV6wIQ8zlQiPKsOFBRo6HiJYxHyYv+P3UHjX99tVbOnHD6A+vA4JVvr+fLDRxbPohaSm3osxCT8m35g4mkxV5jDVeY08oi7H55KNWGQpjEeQTspgBROLEqd1Hj2SgmOUhRYogKugpdBUklmBj2qJao0VOgBhbnRedjDNJYpSjuZ4cPB8U/ums7n0WMZ2A0YzJl3rg2fnPhI1J1szM9utL2utwI9WmO/MxQWYvX7t+l1ZvMTCn3iTKUPukoAxaKCACLAXjmupmvum7hyMaOkr0RPTWu+6r3XxlLECLx0xjoBWDxSJHzYxkdStNwGA8Z/kpStnebPu3V5jOvaos+Kf0GlnBtGuoWewNM3fceaJDw3oAhlkT1pL51HzKsLJwYvToWdr1+bQhIkpuK0t0+ryZhUvEe68N6hHJKN1GB4EH3wkt4Z8c2N/qb+l5GxymHfz24+OzySK2WKvPQ9MMwq+MrnGdUFmQt6j6lCNM9fap1aCW0Nmhp/ohuVmjO1fYgTV3+XHbu/lajQBNRv+Bm+/lg5f91zcok0e7DrKzJBsYf1rNND6LJjfhJ8bwQaAOpxoCd4NGOETN8Z3zLvbkG+W9OkC1suSiNnJyCW9GpLGwm6p219s1OfOYcGPl2/09h//6m5l3+6CbhSCVH5LgQvclpzi0VIp7vGEL1Zpyy65gKQCBXU/7j2sTRBaKpVjenWUVyf5m4wC44c6M2TDmMrwnx/3jQ3PM= X-OriginatorOrg: ca.weatherford.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jun 2017 17:29:45.8235 (UTC) X-MS-Exchange-CrossTenant-Id: dd63fb60-07f6-4d96-8d40-ebeca61a524e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=dd63fb60-07f6-4d96-8d40-ebeca61a524e; Ip=[23.103.226.20]; Helo=[032-smtp-out.weatherford.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR03MB3164 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 17:29:49 -0000 > -----Original Message----- > From: Karl Denninger [mailto:karl@denninger.net] > Sent: Monday, June 19, 2017 7:28 PM > To: freebsd-fs@freebsd.org > Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs > > Just one note below... > > On 6/19/2017 19:57, Caza, Aaron wrote: > > > > Note that file /testdb/test is 16GB, twice the size of ram available in= this system. The /testdb directory is a ZFS file system with recordsize= =3D8k, chosen as ultimately it's intended to host a PostgreSQL database whi= ch uses an 8k page size. > Do not make this assumption blindly. Yes, I know the docs say to set > recordsize=3D8k but this is something you need to benchmark against your > actual working data set. > > MANY Postgres workloads are MUCH faster (2x or more!) if you use a > default page size and lz4 compression -- including one I have in > production and have extensively benchmarked. The difference is NOT small= . >.... > > zroot/ticker compressratio 1.53x - > zroot/ticker mounted yes - > zroot/ticker quota none default > zroot/ticker reservation none default > zroot/ticker recordsize 128K default > zroot/ticker mountpoint /usr/local/pgsql/data-ticker local > zroot/ticker sharenfs off default > zroot/ticker checksum fletcher4 > inherited from zroot > zroot/ticker compression lz4 > inherited from zroot > zroot/ticker atime off > inherited from zroot > > You may also want to consider setting logbias=3Dthroughput. In some case= s > the improvement there can be quite material as well -- depending on the > insert/update traffic to the database in question. > > -- > Karl Denninger > karl@denninger.net > /The Market Ticker/ > /[S/MIME encrypted email preferred]/ Thanks for the suggestions Karl. I'll investigate further after I resolve = this performance degradation issue I'm experiencing. I recently read anoth= er FreeBSD+ZFS+PostgreSQL user's Scale15x presentation, PostgreZFS, Sean Ch= ittenden if I recall correctly, who also advised lz4 compression + 16K page= size rather than 8K with PostgreZFS. With regards to my performance woes, I was originally using PostgreSQL in m= y posts to freebsd-hackers@freebsd.org but started using 'dd' to remove it = as a point of contention. In attempting to resolve this issue, I tried usi= ng your patch to PR 187594 (https://bugs.freebsd.org/bugzilla/show_bug.cgi?= id=3D187594). Took a bit of effort to find a revision of FreeBSD 10 Stable= to which your FreeBSD10 patch would both apply and compile cleanly; howeve= r, it didn't resolve the issue I'm experiencing. -- Aaron This message may contain confidential and privileged information. If it has= been sent to you in error, please reply to advise the sender of the error = and then immediately delete it. If you are not the intended recipient, do n= ot read, copy, disclose or otherwise use this message. The sender disclaims= any liability for such unauthorized use. PLEASE NOTE that all incoming e-m= ails sent to Weatherford e-mail accounts will be archived and may be scanne= d by us and/or by external service providers to detect and prevent threats = to our systems, investigate illegal or inappropriate behavior, and/or elimi= nate unsolicited promotional e-mails (spam). This process could result in d= eletion of a legitimate e-mail before it is read by its intended recipient = at our organization. Moreover, based on the scanning results, the full text= of e-mails and attachments may be made available to Weatherford security a= nd other personnel for review and appropriate action. If you have any conce= rns about this process, please contact us at dataprivacy@weatherford.com. From owner-freebsd-fs@freebsd.org Tue Jun 20 17:59:00 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 45727D9E159 for ; Tue, 20 Jun 2017 17:59:00 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [67.205.158.196]) by mx1.freebsd.org (Postfix) with ESMTP id 1C0516884E for ; Tue, 20 Jun 2017 17:58:59 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id 65E16274BB for ; Tue, 20 Jun 2017 13:58:30 -0400 (EDT) Received: from [192.168.10.20] (D10.Denninger.Net [192.168.10.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id 737713D0B for ; Tue, 20 Jun 2017 12:58:28 -0500 (CDT) Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs To: "freebsd-fs@freebsd.org" References: <2d8a525e5355406aa7b8bf6690101008@DM2PR58MB013.032d.mgd.msft.net> From: Karl Denninger Message-ID: <4276a57b-5974-0d4b-c535-ea11c491b46f@denninger.net> Date: Tue, 20 Jun 2017 12:58:21 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: <2d8a525e5355406aa7b8bf6690101008@DM2PR58MB013.032d.mgd.msft.net> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms080006030407040800070301" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 17:59:00 -0000 This is a cryptographically signed message in MIME format. --------------ms080006030407040800070301 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 6/20/2017 12:29, Caza, Aaron wrote: >> -----Original Message----- >> From: Karl Denninger [mailto:karl@denninger.net] >> Sent: Monday, June 19, 2017 7:28 PM >> To: freebsd-fs@freebsd.org >> Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs >> >> Just one note below... >> >> On 6/19/2017 19:57, Caza, Aaron wrote: >>> Note that file /testdb/test is 16GB, twice the size of ram available = in this system. The /testdb directory is a ZFS file system with recordsi= ze=3D8k, chosen as ultimately it's intended to host a PostgreSQL database= which uses an 8k page size. >> Do not make this assumption blindly. Yes, I know the docs say to set >> recordsize=3D8k but this is something you need to benchmark against yo= ur >> actual working data set. >> >> MANY Postgres workloads are MUCH faster (2x or more!) if you use a >> default page size and lz4 compression -- including one I have in >> production and have extensively benchmarked. The difference is NOT sm= all.. >> .... >> >> zroot/ticker compressratio 1.53x - >> zroot/ticker mounted yes - >> zroot/ticker quota none defa= ult >> zroot/ticker reservation none defa= ult >> zroot/ticker recordsize 128K defa= ult >> zroot/ticker mountpoint /usr/local/pgsql/data-ticker loca= l >> zroot/ticker sharenfs off defa= ult >> zroot/ticker checksum fletcher4 >> inherited from zroot >> zroot/ticker compression lz4 >> inherited from zroot >> zroot/ticker atime off >> inherited from zroot >> >> You may also want to consider setting logbias=3Dthroughput. In some c= ases >> the improvement there can be quite material as well -- depending on th= e >> insert/update traffic to the database in question. >> >> -- >> Karl Denninger >> karl@denninger.net >> /The Market Ticker/ >> /[S/MIME encrypted email preferred]/ > Thanks for the suggestions Karl. I'll investigate further after I reso= lve this performance degradation issue I'm experiencing. I recently read= another FreeBSD+ZFS+PostgreSQL user's Scale15x presentation, PostgreZFS,= Sean Chittenden if I recall correctly, who also advised lz4 compression = + 16K page size rather than 8K with PostgreZFS. > > With regards to my performance woes, I was originally using PostgreSQL = in my posts to freebsd-hackers@freebsd.org but started using 'dd' to remo= ve it as a point of contention. In attempting to resolve this issue, I t= ried using your patch to PR 187594 (https://bugs.freebsd.org/bugzilla/sho= w_bug.cgi?id=3D187594). Took a bit of effort to find a revision of FreeB= SD 10 Stable to which your FreeBSD10 patch would both apply and compile c= leanly; however, it didn't resolve the issue I'm experiencing. I would not have expected my PR to impact this issue. I suspicious of a drive firmware interaction with your I/O pattern; SSDs are somewhat-notorious for having that come up under certain workloads that involve a lot of writes. --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms080006030407040800070301 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC BlwwggZYMIIEQKADAgECAgE9MA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE2MTIxODE5NDUzNVoXDTIxMTIxNzE5NDUzNVowVzEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxGzAZBgNVBAMUEmthcmxAZGVubmluZ2VyLm5ldDCCAiIwDQYJKoZIhvcNAQEBBQADggIP ADCCAgoCggIBAM2N5maxs7NkoY9g5NMxFWll0TYiO7gXrGZTo3q25ZJgNdPMwrntLz/5ewE9 07TEbwJ3ah/Ep9BfZm7JF9vTtE1HkgKtXNKi0pawNGm1Yn26Dz5AbUr1byby6dFtDJr14E07 trzDCtRRvTkOVSBj6PQPal0fAnDtkIYQBVcuMkXkuMCtyfE95pjm8g4K9l7lAcKii3T1/3rE hCc1o2nBnb7EN1/XwBeCDGB+I2SN/ftZDbKQqGAF5q9dUn+iXU7Z/CVSfUWmhVh6cVZA4Ftv TglUqj410OuPx+cUQch3h1kFgsuhQR63HiJc3HbRJllHsV0rihvL1CjeARQkhnA6uY9NLFST p5I/PfzBzW2MSmtN/tGZvmfKKnmtbfUNgkzbIR1K3lsum+yEL71kB93Xtz/4f1demEx5c8TJ RBIniDHjDeLGK1aoBu8nfnvXAvgthFNTWBOEoR49AHEPjC3kZj0l8JQml1Y8bTQD5gtC5txl klO60WV0EufU7Hy9CmynMuFtjiA2v71pm097rXeCdrAKgisdYeEESB+SFrlY65rLiLv4n8o1 PX7DqRfqKkOYIakZ0ug/yHVKcq2EM3RiJxwzls5gT70CoOBlKbrC98O8TA6teON0Jq30M06t NTI2HhvNbJDLbBH+Awf4h1UKB+0ufENwjVvF5Jfz8Ww/FaSDAgMBAAGjgfQwgfEwNwYIKwYB BQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgwCQYD VR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIBDQQf Fh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUpfAI3y+751pp9A0w 6vJHx8RoR/MwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYwFIES a2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBiB6MlugxYJdccD8boZ/u8 d8VxmLkJCtbfyYHRjYdyoABLW5hE3k3xSpYCM9L7vzWyV/UWwDYKi4ZzxHo4g+jG/GQZfKhx v38BQjL2G9xD0Hn2d+cygOq3UPjVYlbbfQoew6JbyCFXrrZ7/0jvRMLAN2+bRC7ynaFUixPH Whnj9JSH7ieYdzak8KN+G2coIC2t2iyfXVKehzi5gdNQ0vJ7+ypbGsRm4gE8Mdo9N/WgFPvZ HPFqR9Dwas7Z+aHwOabpk5r/336SyjOaZsn3MqKJQZL6GqDKusVOCWt+9uFAD8kadg7FetZe atIoD9I+zbp59oVoMnkMDMx7Hi85faU03csusqMGsjSsAzWSI1N8PJytZlchLiykokLKc3OL G87QKlErotlou7cfPX2BbEAH5wmkj9oiqZhxIL/wwAUA+PkiTbEmksKBNompSjUq/6UsR8EA s74gnu17lmijv8mrg2qMlwRirE7qG8pnE8egLtCDxcjd0Of9WMi2NJskn0/ovC7P+J60Napl m3ZIgPJst1piYSE0Zc1FIat4fFphMfK5v4iLblo1tFSlkdx1UNDGdg/U+LaXkNVXlMp8fyPm R80V6cIrCAlEWnBJNxG1UyfbbsvNMCCZBM4faGGsR/hhQOiydlruxhjL6P8J2WV8p11DdeGx KymWoil2s1J5WTGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxv cmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRww GgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5c3Rl bXMgTExDIENBAgE9MA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcN AQcBMBwGCSqGSIb3DQEJBTEPFw0xNzA2MjAxNzU4MjFaME8GCSqGSIb3DQEJBDFCBECUyv9M wX6pcPihyfH7yS/VQif4zKEwwBc7+3W0Y3J/TEfuI6aEAkJxg00lchib+uT3uL45nltSjVCS fChiCMVJMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggq hkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZI hvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNVBAgT B0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExM QzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3VkYSBT eXN0ZW1zIExMQyBDQQIBPTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYTAlVT MRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEg U3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0B CQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECAT0wDQYJKoZIhvcNAQEBBQAEggIABu52BojYATqQ 39/cDBA1Ve2tUHd64HEkGmsO53xYdwgyE2FsK3fN5c7TqsbkivzUEw0EfIiXDIXtdCsOwGlF eOBUd9IBjaeanz5nzC/szpwD0aXLOXtT2KQYR0Hvq6M7Wh2b8HChzORInNq7grDIUFdO2KSl PlmSuGc4yTI5wowAZDj2hc6V0A944dobWbQ7Ur5TXrth1Mba81FAPM64Lp/jYc+rZ0K2obpM wMiyuNezl8H28aKxzE77pi97MOn6WvPl3UlZZgsllwPcNYmpOUW1oybEl7gu/n0RmnSRI3yY htIP0kkW/v33wBC1TAv3sCjxCj3Quyg3S+NZc0fTg3ElitgSa7aUK7C1Y3iJMfxWroqSsrB1 vdO9zraYEk6RVFv+oaGsNjRdBO5REU059z2yxJLfYPrllAKroG6oIOOaeVj+r5XXJtNs869H V3nFGCQHl0kYUDodEJ0YsLULgsBdyW2+1apYq2vfZfQBfo1hQaxkYO6apURLdMxyQZ21VIva qylRwNRWl9YPAEFbnPe4URO96R0unPLi61KEmpLBYpyEgPx2JvOuazkIub+QqOKF9M2ViDPJ s/0/WiZc+N6i99uDNZBkEG2JqXWTyhgUgh+uFa9AjtMK4dXAU+4F/vIO2IjXY3vv+VK+frEF xPJXN9gf3TxqiqHaGUdrEYoAAAAAAAA= --------------ms080006030407040800070301-- From owner-freebsd-fs@freebsd.org Tue Jun 20 18:15:20 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 96E4BD9E845 for ; Tue, 20 Jun 2017 18:15:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 853EF6A31D for ; Tue, 20 Jun 2017 18:15:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5KIFKsw089927 for ; Tue, 20 Jun 2017 18:15:20 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220163] Allowed characters in UFS volume labels using /sbin/newfs Date: Tue, 20 Jun 2017 18:15:20 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 18:15:20 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220163 Conrad Meyer changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 20 18:50:32 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A766FD9F4E4 for ; Tue, 20 Jun 2017 18:50:32 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-bl2nam02on0069.outbound.protection.outlook.com [104.47.38.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 041F96E86B for ; Tue, 20 Jun 2017 18:50:31 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weatherford.onmicrosoft.com; s=selector1-ca-weatherford-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=sT6O56LXHDB+Ckj0yAzcA7LZwcbIZQ1nZP+5HPkD5mo=; b=t2Pi6+GkvyK/HJtGbDz4Lr2MVvjq8valLiDXJGdJjLe+rh9oaeQM/MJk50070rh9FWISq9hR5lpxVeltZoED+S3MIqNLyKEkIlxZ108HjwurYToYBONOLj/w9A4Y5aWgDY+4DhntsMLP9ibTGo66DEkNlUfMv7t05ONqdBppUVU= Received: from MWHPR03CA0018.namprd03.prod.outlook.com (10.175.133.156) by MWHPR03MB3166.namprd03.prod.outlook.com (10.174.174.35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 18:50:29 +0000 Received: from BN1AFFO11FD009.protection.gbl (2a01:111:f400:7c10::176) by MWHPR03CA0018.outlook.office365.com (2603:10b6:300:117::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14 via Frontend Transport; Tue, 20 Jun 2017 18:50:29 +0000 Authentication-Results: spf=pass (sender IP is 23.103.226.20) smtp.mailfrom=ca.weatherford.com; denninger.net; dkim=none (message not signed) header.d=none;denninger.net; dmarc=bestguesspass action=none header.from=ca.weatherford.com; Received-SPF: Pass (protection.outlook.com: domain of ca.weatherford.com designates 23.103.226.20 as permitted sender) receiver=protection.outlook.com; client-ip=23.103.226.20; helo=032-smtp-out.weatherford.com; Received: from 032-smtp-out.weatherford.com (23.103.226.20) by BN1AFFO11FD009.mail.protection.outlook.com (10.58.52.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14 via Frontend Transport; Tue, 20 Jun 2017 18:50:28 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net (141.251.110.81) by DM2PR58MB014.032d.mgd.msft.net (141.251.110.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 18:50:27 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) by DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) with mapi id 15.01.1178.018; Tue, 20 Jun 2017 18:50:27 +0000 From: "Caza, Aaron" To: Karl Denninger , "freebsd-fs@freebsd.org" Subject: RE: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Topic: RE: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Index: AdLp8ugzXKKDkz6vS/yx+crAeAsw7A== Date: Tue, 20 Jun 2017 18:50:27 +0000 Message-ID: Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [141.251.205.196] x-ms-publictraffictype: Email X-MS-TrafficTypeDiagnostic: DM2PR58MB014:|MWHPR03MB3166: X-MS-Office365-Filtering-Correlation-Id: 7e2e0221-15fa-44f0-2543-08d4b80d3881 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OrganizationHeadersPreserved: DM2PR58MB014.032d.mgd.msft.net X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:23.103.226.20; IPV:NLI; CTRY:; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(39410400002)(39850400002)(39840400002)(39450400003)(39400400002)(39860400002)(2980300002)(438002)(24454002)(13464003)(42174003)(51914003)(189002)(199003)(377454003)(9170700003)(38730400002)(8936002)(6306002)(55016002)(9686003)(81166006)(8676002)(5660300001)(97756001)(54356999)(22756006)(50466002)(33646002)(50986999)(66066001)(229853002)(6246003)(2906002)(53936002)(305945005)(8746002)(47776003)(42882006)(53546009)(106466001)(478600001)(72206003)(6116002)(3846002)(189998001)(102836003)(356003)(7736002)(108616004)(46406003)(2900100001)(86146001)(2501003)(7696004)(23726003)(86362001)(24736003)(5890100001); DIR:OUT; SFP:1101; SCL:1; SRVR:MWHPR03MB3166; H:032-smtp-out.weatherford.com; FPR:; SPF:Pass; MLV:sfv; MX:1; A:0; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BN1AFFO11FD009; 1:T8ENabrBw0ApL6knpcg8ax/ErTlSDYO+X+mcCd1+ki2Dffd+6RRXtHZNZK1p9Dgz94OEVSpGCQGthyfhlc8f1Z3zt+ZSeGUoiBhVrPYWs9WMCcjL2QqFy7pSW8tjoh8pvPv2Rkn8afT+Pjnu6LOqhCe8MHhf7KOK2P8GfsnKjL+P0qGTMDPLfBhSn8EN8aOZAzmK49Qpq+P9X1pBPCXqPsAZZa23GRmttNiHR3fbaRPBpWQvc27DMxekWwAkDQuiEjNwE+tpsyfIE2IEWYeuSOqaiS0Fo+HFwfH2nhCkRug7b1t1PTs+KtUfhzv9NzFfVd1Jl45lTpingflRr/piphmAaWZYKO/jCXtQsNe3ft7cp7dkSEomFdj4eRgco2OlR7Q1I8s0lfmmwss/AFE5SlP6xdjt2q4m1VC4N1/J4LmNwTDNpcThNEUEhhTwYxp0Jc9TxfYzxVXe5fogRamIE0AYANrHulkAXEOoU0rkMuF1UT5HXZWdaBHSiJN40ujkZe+R2OoQ3sDDw2t/k3AQVQ== X-CrossPremisesHeadersPromoted: BN1AFFO11FD009.protection.gbl X-CrossPremisesHeadersFiltered: BN1AFFO11FD009.protection.gbl X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500055)(300135000095)(300000501055)(300135300095)(300000502055)(300135100095)(22001)(8251501002)(2017030254075)(300000503055)(300135400095)(201703131423075)(201703031133081)(300000504055)(300135200095)(300000505055)(300135600095)(300000506048)(300135500095); SRVR:MWHPR03MB3166; X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 3:2W0P9Z/G4E2qeNSjwDmlCBl6RH1/a3MhBciIYL0rilefQ/OPdqau5gQCMdqt0t48sVkdWjiHgofeuuyq6VrcsgdnbjfUpFGFD9a1a3xs62UHUCzpdMjKyq9doTB2uDRubR1EnnTZvB0lQmCagU2dSqCz4fkRDrXHHla5rX0sFW2bpCZzkxePmg9kCPs4WWwnTYj3bsJx9rBiJRKbiL/eJQie+buJt0VUv8c/CXZuHPLMX0Mi9n8FflRXy9bGrX8UhXARlC/Mw+KPtXz5QE4PceIUT6MZXhpP1JrB0tQhi1xy2jATHDnFnOsB4+wFNvOVNysYxAxHbSHzYGYxSwMQV/Hxd90tuyAfLnJvPf6JHU78lHlI/MkSqq228nCSYfajNwCo7agkgrPAzy2O/kgBx/SQvMNc3506i+WFYS77PeE6Il9rTXLcZLOcoQBrPdPuk1YQ2cXiL0iPMI8sli6saRQA43MZkKYmLjD3LTGYSb0FrGbCEfCetTgmJEOxdK85QMayui7xDSQeMI+TOSIs/PNAIzUVAbeYD6nJM0In7lp2rAv2/lOrnIqZR9nT7IFjK++47KHlnZgCOH9z4ydOTgwU5MdbdQIcRgSrMtm5SDFWm1k7ZpRJx8h+3i0TMCHtgmNNUHt086QB62O2TRBvVhyPtbu4AcC5uTEebS+/hAOmh79PhUl5pXVOSl8mUYJgHZX4p9GzeHVpm/VHsP7gpN8DNSdn8Ar8oGj9jaDiK9kfr9dpZ5iSwh5kDSLwJSJEG+IdF0718nhHHlhu3Br0oFzbnf2n/2XxNYklPw5crs1BY8G1C37j6DvNMSybwdVmEtLlUmrWJTzv5Rrwu0N2AykuWq+R67oCi4JVDJVGzHuJr8MVZOfuoLZUmvIVwfxDw+X1NXw2QVEL1Xg/YRpmzQ== X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 25:MFvI+kFXLtk9l7v8I8Ix/zuKLpWj3ELMdpNZPyGmfbba1n7cTgX4qiyc96XKabVvRPq0PQbkjvbCyTmbrAj5q8Av8JDrrPjj/q6ffRtFto9nfutlQv1aYosKQRTpv15mg13XZ4Co0osC6l8et0NgOZBg2hzJpkukEItevT8yShQYcTm+cD/4vsJappQJ14TXq0Y4+0u4CeVMoR6gx+ETiYc+vMCFFRxSU6pN/Tuu6/jxr+0EEiQ9ThsO6LFWKOQkm4ZW5GHyePeR0GvpIxQcrBA/x1qZeNKsdW2kJ3tQZjsfuOnCLuiPNXHgsMdiwftK9cD/ju8MpnIcEaZmCT4++s9U1t7g3ZQ8AgH6PUu6kgQamE2yjzy9no8Q9Yj+jTRaJ0XQ5ebAnWtUicOQT8CHs/DzQo9pKX6BgAIuE9GwpnyyO9QgjXWLS/mubNEiyjGKc93y3VYTa48ED8EzkDDDs+Hi/wm4wEx7zq1EyVd9GC2vs/m3C8h9m1LZFMbZtn398F1eEWGir5v/Mk6cMWyFbKF48mL+iSboLxRmfJYjwT8to0825V4s5Ihbj6aKBReiNDuugSIDHm7pCN2tzNIRJMPtz+5TZT1jHaYL6pUiAELJ7FXWTZdLMg0nxu9O3Uv6y8DwADaAYdRtMUHd4eY4u8HryhTR30N4wBmmWUcya4d+YQKdNql7sw6Scg8TtwSk66i9gxM6WTiyduRHqwkJhB7w7UcEivMVmG63krQGRdfknpwPbc/Tuz2qMClGVxowkEWIheyd3mAXA7NZPV3tnwk2X1zmGft6gB6nP2Os4xwcp2slCGu3KtrCpAKQYYRlSj1Fv4WJuAp0AjrfiULv4WgG3PQuJsH6uIPoocfI59NQnV8pBbrKJOY2Xy6iJ5L4vLv2RCCXAsAZGXLP6nXARvAsjNlx/SN6xM0yjiwYwao= X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 31:sEOayVR9HMPlWyuky19TNqDQi54ODJ/GoaZRUg5CF7/RxPtXQeisXKqiX4MQrbjh0rvzsNbQxQIqB1PA+rK5BOpb9yvcrJ686Finjx0/d1HwUmoX0sJtcINnwD4UA8BGI4zmzW/UCs5b7/MfX3si/AmxwVUboggrRZ9HUAMaVkUxgwl9iCTMktXKQ2g1/JXkVXgBiNtCo6JfIgestkI3Ed8hmjh9VBdjdEHfVNav+eNGe2D8fN361RJQ0aCw7EuIaHwywMs8IVgMrg+mtCb5VZA1Neqy43jzQn6uVasBzlOV3IvG80oqUqaF8LwpowzOQx5IE3uPTgy/jUx0tZ6KSHAJj9PoFHdTVLyefVpWGjfDLHhF6NYq+B26ekn0MFxpCbmUAcvHZU4qDOh4Fi6nXEy8U2dOnhi/9rNtDeT5U5UXf4v0iIPuE3FVAHB4n0fWH/xKT+1pGtsuxxrGkkPOKYrihLyU7KQMz1KiNQ8d1VE1asflMCEZv7dSTgd1Jhl2gFyjIxldmRbymL1HzpBtC6Xai9d0a8orJaOQIx0Ut+rRDFpd5N95gEx/xScFAsxYK2TF1REgbnJa2NojH3Pb+BsoYzgZb93tpiAsVoP3TeD4Ze8O9K8dw6WWrI9Vjqj9xbW/f6kV7/fBIMYNkBsTJB2xUpm4FjHJSAg0eMJCg46twyGfZavdTzIVZCzhgl3R7mgjx9ityLrQhS4qMlEbsQ== X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 20:dnDKvTSucW7JPXpm5n7QLm5ShCps69fuZT4e/OVynfzbAIB06b9fnToja4GtQQSMU8PhoDL6Z4yKGpSHrAKwAY5xkXpntx9d1t01OnBpozuutsj6363V5IfBvm3R3p26sgydxfpYmDgp4oqhUWV/ENQgProNtASIN5kaQOVY7yAnxTzD5MZwEqNeZB/4dFix+LcAGcbgsgHsqQoYQqHR/MDUvRRyg/v2+6nTTFp+hsYeuXpNdn/bvz0rB0zaajLvw0uaD3FPvI6X4aB9khLEpPuvUhZg9GOTTlmT4VMNUs6ZKHHmSAbwVQuV2wyzOwk7QAdTTWWBGQ4BXD+1G8cNo8Fc6mYgmHHvqF8z/dyTaOIg1NbrQiTfoigVubHnFu1xp+Yg1aiy2Vg6iBTFGn378xfyxFARc5mXFOkOCx1ziSOsBJQ7f3h3CE0bzEd+CZmm2S4p3gaCAi10DrKP6zGUya84BqnSrTJjT0pA05wTyZARKXvRCA+X0sfxtI+q+IAS X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(158342451672863)(192374486261705); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(11241501159)(6040450)(601004)(2401047)(8121501046)(5005006)(13016025)(13018025)(93006095)(93004095)(3002001)(10201501046)(100000703101)(100105400095)(6055026)(6041248)(20161123560025)(20161123555025)(20161123558100)(20161123562025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:MWHPR03MB3166; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:MWHPR03MB3166; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3166; 4:etbsJ+XWPKt6AxfCgMN3z8Xvoz3m/t8X/M9wG0OFa1?= =?us-ascii?Q?4Ypt/V4a79P81a9WadFkz7TSgwIqF9us2ED8reNU8bAYrsRcyEJX1wXW7sae?= =?us-ascii?Q?zOWRGCrRzZQ9S0S+uvJQjurWrF5nNMzhYI2iTg3RKzoO3EnpVrW13jfM1pJM?= =?us-ascii?Q?4WuAmqVOS9iCI21mFZ0IHrG0Ast6Kp97mkobzI3jni27IMDTmP0cj+P+HW1l?= =?us-ascii?Q?WWfRjxR5UnAHkE+e834k5lX2vF82q5s22Sxzln9wBAS1pG6X/hEZgi1bllxB?= =?us-ascii?Q?j6LijK1d14FYr9fo3slKAgf8IW8n8woscyHoiYM6TyvplGi+b3JffbpHcQ/X?= =?us-ascii?Q?flsMsmDcF0nnnFohcQCUnjT1mxE9M1KkULjFYdPN2U/47nCdZh+aWP/qv5x3?= =?us-ascii?Q?Q4gc+zuhc1zMWKZqeK79F6RVOwUh9IpDKT+S9zldblkY5mvxuGFTMqLlvFxA?= =?us-ascii?Q?EsFOKEuvU7aH6gX/EOcj98r80ZlZbJGzEgjkAibh71yFCbxiVoca1aBxMIZj?= =?us-ascii?Q?kRI9FJgCKoImblYLBNVEJu2+KCu9e1+dnRKE+IhFO0AGHxa2RxGodYGPAmbq?= =?us-ascii?Q?Mar1+dXJ6LuXm0+xjKkaLymKXoXESNaq9SwgAKnpJmtqrW9XrinigAHFFZcn?= =?us-ascii?Q?WLYwQNEBNSUYJF1ANJm54xg/gCSmuSruxA9eeUCExxNpJv+Qqifh45HLv1WC?= =?us-ascii?Q?IfkUM0AyX1BOowmw2QRLrM2UEfTG79XoJKsELrFx1/M/jwfI+ZE7oeI1RI3p?= =?us-ascii?Q?3XCPez4CLnNV+wFsEDed0M9q+EqmfPHFsGGOGTBE9Ql9hUAcOVMWuCyc3ZZ9?= =?us-ascii?Q?7olVtHnpM1B1ta8RqUTod9E4DmSZXnb6dxh3+WeCcJNTka8WFALVDnHeRmX7?= =?us-ascii?Q?0X+UUfVQt0rVytsZXZ3JGvYHfCnx1/tiAK/h6x43zwkNd8Dxy75vpdlDhX5T?= =?us-ascii?Q?zSLV1w0cd/GMIOuDBgvKh3kZ8iZAi6n8HqQJEhCYRbGtLnqrAaftDTC4zt6K?= =?us-ascii?Q?OFawYU7zw72pbaMIu3FlR9Tk6MbJR3KRD0snJV6t4ynMNQs3bDCoHzZgFYBV?= =?us-ascii?Q?fGP5CznX+Pr45S63odusX7rNAB2p8Z7r6TCjhnOqc/VUN6CXqXtq9i4011ZY?= =?us-ascii?Q?GiJ2b9C6N1ypAtyCI9jDkzlUrAyxdelWe84tTJN56Zmf1D3zMzF5UnsZNuxY?= =?us-ascii?Q?0A6BdQI78QZpmtyB8OdOHUyYCSeIBmhXEDlAMu74QAvARqrL4hkoCJJga4fj?= =?us-ascii?Q?vWtZfuAYsRjYZ8Nd6KwjK0f9BqB785X/mY7oFvPOwZdcIQHyIacOjrXwzVy6?= =?us-ascii?Q?mMKhaOrMXqUOnTGH3Lt9w=3D?= X-Forefront-PRVS: 03449D5DD1 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3166; 23:qfwOJKhxOaQBAwU75qmfLU+31QsV+NiAxA8ol5uYx?= =?us-ascii?Q?qK+cgNyELQYw9j3OT1WfzuuomXk/d3R6Sl9G5e2HOaKoFwvJT4SOQoNr0aal?= =?us-ascii?Q?bXrZbrImLoJIKOVXWLfdIdX8NDvJbk6KNN8ZJ4oZVbn6jSf9Iqa8lgyQu9Uf?= =?us-ascii?Q?R6DskBqGDfu/pTyBB+kGu2xP+mlL1Y1cxn28KhX4Ed2mI11SOlnBUnUwFrNS?= =?us-ascii?Q?peTHE2AqcVUMq90AXgi5UfXEVuUabnu7eTGer03/yRm0gH83kH0N1kPHE71F?= =?us-ascii?Q?V79pvo9rShdQrhffHnuqJmNF6AZqpZBDsk7BZa21qP3XZ+6P6N0MyL8i8rEv?= =?us-ascii?Q?oBwt2qU26t0pfaSDAezczRvUFZqalrUrUz+UdYEhzY0Qw2gyEN0x/73wha1q?= =?us-ascii?Q?FgDYoXRGqmwNeWp38XZYu5Vn5+DkLt6hztGW0VKNb496HR6QPOMc4sZsNj8L?= =?us-ascii?Q?Us7O1k2H9HlUbZBI1gcyIIZCBkyhwFake7LaTUDCN6oCAY9QmQ6b7Ll7Gy0x?= =?us-ascii?Q?HTP9FFBkjQePLV6D08RMrdDKmZ90NS001NgRQ4hPiO3GRHqaUYTir2uXrAU1?= =?us-ascii?Q?G2NMOXq4tHSKVvqQT9Pga3/mW0WO6R/AnS0UJZS0pS5Xi2WXaPZaogzzgFj9?= =?us-ascii?Q?cT2ra2sYdlkS6WVQc64G7Z6POnvbynTL58+sDrl8zHH0bmN7Uiool6MikLVS?= =?us-ascii?Q?uTHgb0c9ODF+EA5r5b3MW1IQ1rN5KK0plBTp+HM8eB6ymKzueAtjzvivftVs?= =?us-ascii?Q?Y+e6EBfNqRktrdfDHs0KwqpM8U3hjdTKu2lvcBWpiv4kpO7pyw9mv/lKG9Wp?= =?us-ascii?Q?vW0AZOxpR6BtZ3W1l4Jn+EKAveGxSaXS+ZJ7TnGd3Z/0H+JIGCVq+2mhning?= =?us-ascii?Q?v060Mog5mWGuRRpFfAlz3KcUiUi2pXKkurzEHijlh/2taeNil8htu7L/xciM?= =?us-ascii?Q?SB3cvEN3rUo3tl50ARFmSfe9mMiYpKIMbKMR7IIfP3DGX1R1QqdAoOVrR50x?= =?us-ascii?Q?2SXsigvjcKrFS5VWSlHCFvRaJO2CrIa/wzmz4tlAgVNZfeNsrJkh1xLrlbr5?= =?us-ascii?Q?bKbH2NtKmNiJgbS87rGEW0g0cFtVA67kY5JMkJHKcPEXIhHrHxHRBMRVu5en?= =?us-ascii?Q?SaAspy/bv55SN6wTddMIYshXXUJ33lEcgGF6RonHPREHi46osOQDk0BYnOzX?= =?us-ascii?Q?mHuI1KxOBM3L7f+T7csgDZhuZKw7EB5rcCCRukgCdOx//QNIrDtfNWVhXwMw?= =?us-ascii?Q?Kbwb/X7kwXfs2bwO+cJqPmqTEUycSzgI2pKPHEdoCi4nM1dMpHL5e6UYpDbB?= =?us-ascii?Q?KUI6t08rnslBOfWUqVe45pw18ZXPlAZNQDEZjCP9iB05FONYtQf3yAE4/I57?= =?us-ascii?Q?NkySe/9tnzyhLCEC9CLFwsD0tJ01a1JiimiHRvP+fheAT4g9piwaIc8BKV08?= =?us-ascii?Q?RjFXTwtUI5zpuSvk+GHiRLsY5XWrMI=3D?= X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3166; 6:8vw75vBXynj0VBV89ysRnM7muU/tCtto79j/WzpXsL?= =?us-ascii?Q?zZudVAfyGjubAigxsBDu4bQNcW5P1jIPK1JnMKO3izVhtDaa7US/avgUFTQT?= =?us-ascii?Q?L4TfIz7f3q4xj51RpUTbsXeJZjEH37Lf6fF47EOHVv3K4sUiBykmrDYHBjgS?= =?us-ascii?Q?2Dkgpb8Tz9QtGRKrhXk5l90eTT8flcPhRHzIdCupn5ZqI8yfK3y2F/VszMOM?= =?us-ascii?Q?1K7GxY1te8W+yAzetB5tkbZDzwJGzHmuJEYGokmGttvjkqxzHr/vtYSKwMip?= =?us-ascii?Q?72xMa6NpIHdadbrKtkHQsXqEnpQ6ORBYSu7ID4a6b1vPMlqv3OSSS0nBq56O?= =?us-ascii?Q?G+OgQPPetTkx96TTV+vtXlrUsS341xc78bWj5kd0ieciMBHnAgwbqwzSJpdN?= =?us-ascii?Q?ZNE6GOn2YAMwAS+kvYbbLASfb5LTXZLsSPKZQmV6ewWk4c46nmf74wY0Q3gw?= =?us-ascii?Q?Q6s9O4wydxEQ/SaXZGjxPLdIuJhafY4hoBvyp0huZB7xONfHMyhyhoUwBD7W?= =?us-ascii?Q?O8os75YHeGJHAqD+QVyROQtLRK7V26v3YOYanQG5uId4a0a3iAhhsB99F9Va?= =?us-ascii?Q?OQJnmLo2ADxfkysqTE87fwZO+YcwdqAGbGCLPv47k5vcVJ1SiOz/x8zzGZg+?= =?us-ascii?Q?XEwoONX1OJraS+gnWpaZW8+6bfEFXXzYG8end/2TZyRTHxiENZHN+/j7BRLm?= =?us-ascii?Q?qSPl2+tk3gpFb+FP8j5gc6EiIK5a87PNkamfUFoPhGAwqV4V9pgp3H1/a4vn?= =?us-ascii?Q?YftNfaFGAoEVZ8W4KoMlZkwF1TIZMkHXvCxdCA2k9OKY3DwZSidx7A8k4W9x?= =?us-ascii?Q?TXZjuaNIztwWhlyXo/1DkY+rhHRfHfhwb6wuQp25vFt6rhyyR5YA961JjSYy?= =?us-ascii?Q?g38gNDI+7XyjpghpDUNsCNVCD5BZazVPOqYZcP8or+CY0u+95PCKiptiFVkZ?= =?us-ascii?Q?rXu8HgUnpHUk2jf+9mefvNWE4TMrkvL2H+8/T3Ygd5aXrAOiWy61YTtaGdKV?= =?us-ascii?Q?g5Sds/eZPZv1Vb/uvkLEyx?= X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 5:q9n7tAfvYw2eIz6ZwuvaNBvCWevIvRjxJJ88eh6kn5DL79rQogNvhpsJZYH7tAfATKrpkagyRsRrCGvNqEN5TlMH32ZTrz4SpmajFJbTyDJH1QbLRZ+X6fszm9Bmjr/HUbWBPw1lVc4hrXFQIZ+hatmBNC9xjpoYmwSqiElThR8AWWtH95Wgqx6JhF/+C/JQFOC4NOse8iepVlU55VS5h/ZhyY0WBXZZWJ+hWYcHHRfc4taqNw9YNXGJRQowc61sDGRJgiDHxjcheCL+2d3QbDEZv6s8S/0GWMXsE4S0X7lGhKbHNWfHASH7hE+pZrVtGwMCvzKEy3ANFdVii64L3LBUqupPAmv8d1Yu7UmkUnqdjeDCm7OCFvyDafL4O2r2UJI3nYYT0GbKhXdCOojF45cALVecdmo0tBn8sD5Tnqji9IupIZCC+bp3RZI+sj/4dF+SrsYqFTXTNMoJpcxGckm5ZQRgvaSwm7GdMBz2HwddwNuj/TSSkk1nyHB4QMzm; 24:9iz5HcdOy/oF9APTxRzUHsDjYT5gvCFHVnncTwgv7wmLCnYIyipIj1Oerlb8XvW58dSJSTJn+EIzrtan2dBYY/APIfSs0HgNRpLK0lY9gVA= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3166; 7:THgvPC+HRTeKOnl43WnTkCH1mbxllQD+aA3dQYeNyFdP4r/0KwFZCO9MVivyBVlUSuhAUEzEoehTPsLiJMmtrCh7eMNJAA2jmPfVZBr4UmkUUx6Au1Lm/lDV1ic4A1VjNt+/d9WhlvpCtJNTLEpfgMxQRSqUB47YNNKDvSwD3eDyfvu91jfcSvaIbvW2RDv0dYjPzDxsvodgSpMMm2ExyLGAI06g3YDoLoVfksitmirFcXrO1QUFM9v6LF62iR9MAO+pHLrCLuInIq7ZaMi+mRIjY+2sh/wwY5YC4ml2HiXsyhM3QAWZddSpce1NbpE1vEbNDTnmvwLKx2FhDmGkEwqQUTRhKQBQr1EeUu9SxRcNX2oJSM5jjIGkc5kHHAdvA63CxOq1XlWSwxcO6wvTJHOXbhWj2p0qx/JkQa9NDwNE+JWQaWIBOECBgpBz3eo5kWupSB/jNB9nK2Cq8MYXYtf7m0QiNhWWMgGnhhIoO/ZZCG/Y4RepO8ZmaMUIiwO2Q8VyVqjvnZV4sJywnweVCytNElfvexmwwRtJO3esSCKQMW+o7/h4wAYpyS0VBKt65rK/JazVCKSiLfgPeqFChfFa7ab3hbh3WIGoAfA3hk1jkqjO5To1cHf7yEZJZoCPvMgPhysOyxHhoPoBXjHVvhr9q7X/CDtMlw1/t6QuE9Hu/I3LMXcO8Y094DimhyjPdhrFwakC5r6rtGMaouxzcaeb79jODVnVzc0vsFTsxErHjPdarXKlSXax0RfEMp9gS9JuVDEVu6tglnKLaacimLsfMXTHDaWxY5zPtvP0Mew= X-OriginatorOrg: ca.weatherford.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jun 2017 18:50:28.7375 (UTC) X-MS-Exchange-CrossTenant-Id: dd63fb60-07f6-4d96-8d40-ebeca61a524e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=dd63fb60-07f6-4d96-8d40-ebeca61a524e; Ip=[23.103.226.20]; Helo=[032-smtp-out.weatherford.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR03MB3166 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 18:50:32 -0000 > -----Original Message----- > From: Karl Denninger [mailto:karl@denninger.net] > Sent: Tuesday, June 20, 2017 11:58 AM > To: freebsd-fs@freebsd.org > Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs > > On 6/20/2017 12:29, Caza, Aaron wrote: > >> -----Original Message----- > >> From: Karl Denninger [mailto:karl@denninger.net] > >> Sent: Monday, June 19, 2017 7:28 PM > >> To: freebsd-fs@freebsd.org > >> Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs > >> > >> Just one note below... > >> > >> On 6/19/2017 19:57, Caza, Aaron wrote: > >>> Note that file /testdb/test is 16GB, twice the size of ram available = =3Dn this system. The /testdb directory is a ZFS file system with recordsi= =3De=3D8k, chosen as ultimately it's intended to host a PostgreSQL database= =3Dwhich uses an 8k page size. > >> Do not make this assumption blindly. Yes, I know the docs say to set > >> recordsize=3D8k but this is something you need to benchmark against > >> yo=3Dr actual working data set. > >> > >> MANY Postgres workloads are MUCH faster (2x or more!) if you use a > >> default page size and lz4 compression -- including one I have in > >> production and have extensively benchmarked. The difference is NOT sm= =3Dll.. > >> .... > >> > >> zroot/ticker compressratio 1.53x - > >> zroot/ticker mounted yes - > >> zroot/ticker quota none defa= =3Dlt > >> zroot/ticker reservation none defa= =3Dlt > >> zroot/ticker recordsize 128K defa= =3Dlt > >> zroot/ticker mountpoint /usr/local/pgsql/data-ticker loca= =3D > >> zroot/ticker sharenfs off defa= =3Dlt > >> zroot/ticker checksum fletcher4 > >> inherited from zroot > >> zroot/ticker compression lz4 > >> inherited from zroot > >> zroot/ticker atime off > >> inherited from zroot > >> > >> You may also want to consider setting logbias=3Dthroughput. In some > >> c=3Dses the improvement there can be quite material as well -- > >> depending on th=3D insert/update traffic to the database in question. > >> > >> -- > >> Karl Denninger > >> karl@denninger.net /The Market Ticker/ > >> /[S/MIME encrypted email preferred]/ > > Thanks for the suggestions Karl. I'll investigate further after I reso= =3Dve this performance degradation issue I'm experiencing. I recently read= =3Danother FreeBSD+ZFS+PostgreSQL user's Scale15x presentation, PostgreZFS,= =3DSean Chittenden if I recall correctly, who also advised lz4 compression = =3D 16K page size rather than 8K with PostgreZFS. > > > > With regards to my performance woes, I was originally using PostgreSQL = =3Dn my posts to freebsd-hackers@freebsd.org but started using 'dd' to remo= =3De it as a point of contention. In attempting to resolve this issue, I t= =3Died using your patch to PR 187594 (https://bugs.freebsd.org/bugzilla/sho= =3D_bug.cgi?id=3D187594). Took a bit of effort to > > find a revision of = FreeB=3DD 10 Stable to which your FreeBSD10 patch would both apply and comp= ile c=3Deanly; however, it didn't resolve the issue I'm experiencing. > I would not have expected my PR to impact this issue. > > I suspicious of a drive firmware interaction with your I/O pattern; SSDs > are somewhat-notorious for having that come up under certain workloads > that involve a lot of writes. > I've observed this performance degradation on 6 different hardware systems = using 4 differents SSDS (2x Intel 510 120GB, 2x Intel 520 120GB, 2x Intel 5= 40 120GB, 2x Samsung 850 Pro SSDs) on FreeBSD10.3 RELEASE, FreeBSD 10.3 REL= EASEp6, FreeBSD 10.3RELEASEp19, FreeBSD 10-Stable, FreeBSD11.0 RELEASE, Fre= eBSD 11-Stable and now FreeBSD11.1 Beta 2. This latest testing I'm not doi= ng much in the way of writing - only logging the output of the 'dd' command= along with 'zfs-stats -a' and 'uptime' to go along with it once an hour. = Ran for ~20hrs before performance drop kicked in though why it happens is = inexplicable as this server isn't doing anything other than running this te= st hourly. I have a FreeBSD9.0 system using 2x Intel 520 120GB SSDs that doesn't exhib= it this performance degradation, maintaining ~400MB/s speeds even after man= y days of uptime. This is using the GEOM ELI layer to provide 4k sector em= ulation for the mirrored zpool as I previously described. Interestingly, using the GEOM ELI layering, I was seeing the following - FreeBSD 10.3 RELEASE : performance ~750MB/s when dd'ing 16GB file - FreeBSD 10 Stable : performance ~850MB/s when dd'ing 16GB file - FreeBSD 11 Stable : performance ~950MB/s when dd'ing 16GB file During the above testing, which was all done after reboot, gstat would show= %busy of 90-95%. When performance degradation hits, %busy drops to ~15%. Switching to FreeBSD 11.1 Beta 2 with Auto(ZFS) ashift-based 4k emulation o= f ZFS mirrored pool: - FreeBSD 11.1 Beta 2 : performance ~450MB/s when dd'ing 16GB file wit= h gstat %busy of ~60%. When performance degradation hits, %busy drops to ~= 15%. Now, I expected that removing the GEOM ELI layer and just using vfs.zfs.min= _auto_ashift=3D12 to do the 4k sector emulation would provide even better p= erformance. It's seems strange to me that it doesn't. > -- > Karl Denninger > karl@denninger.net > /The Market Ticker -- Aaron This message may contain confidential and privileged information. If it has= been sent to you in error, please reply to advise the sender of the error = and then immediately delete it. If you are not the intended recipient, do n= ot read, copy, disclose or otherwise use this message. The sender disclaims= any liability for such unauthorized use. PLEASE NOTE that all incoming e-m= ails sent to Weatherford e-mail accounts will be archived and may be scanne= d by us and/or by external service providers to detect and prevent threats = to our systems, investigate illegal or inappropriate behavior, and/or elimi= nate unsolicited promotional e-mails (spam). This process could result in d= eletion of a legitimate e-mail before it is read by its intended recipient = at our organization. Moreover, based on the scanning results, the full text= of e-mails and attachments may be made available to Weatherford security a= nd other personnel for review and appropriate action. If you have any conce= rns about this process, please contact us at dataprivacy@weatherford.com. From owner-freebsd-fs@freebsd.org Tue Jun 20 18:53:16 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B842BD9F6F4 for ; Tue, 20 Jun 2017 18:53:16 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost2.sentex.ca (smarthost2.sentex.ca [205.211.164.50]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "smarthost.sentex.ca", Issuer "smarthost.sentex.ca" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id CC65F6EC9E for ; Tue, 20 Jun 2017 18:53:14 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (lava.sentex.ca [IPv6:2607:f3e0:0:5::11]) by smarthost2.sentex.ca (8.15.2/8.15.2) with ESMTPS id v5KIr2ea016935 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Tue, 20 Jun 2017 14:53:02 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [192.168.43.26] (saphire3.sentex.ca [192.168.43.26]) by lava.sentex.ca (8.15.2/8.15.2) with ESMTP id v5KIr0hu025114; Tue, 20 Jun 2017 14:53:00 -0400 (EDT) (envelope-from mike@sentex.net) Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs To: Karl Denninger , "freebsd-fs@freebsd.org" References: <2d8a525e5355406aa7b8bf6690101008@DM2PR58MB013.032d.mgd.msft.net> <4276a57b-5974-0d4b-c535-ea11c491b46f@denninger.net> From: Mike Tancsa Organization: Sentex Communications Message-ID: Date: Tue, 20 Jun 2017 14:52:59 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: <4276a57b-5974-0d4b-c535-ea11c491b46f@denninger.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.78 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 18:53:16 -0000 On 6/20/2017 1:58 PM, Karl Denninger wrote: > > I suspicious of a drive firmware interaction with your I/O pattern; SSDs > are somewhat-notorious for having that come up under certain workloads > that involve a lot of writes. I wonder repeating the tests with plain old hard drives would show the same behavior ? ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-fs@freebsd.org Tue Jun 20 19:06:31 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0FA3ED9FAD1 for ; Tue, 20 Jun 2017 19:06:31 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-qt0-x232.google.com (mail-qt0-x232.google.com [IPv6:2607:f8b0:400d:c0d::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BBE706F55C for ; Tue, 20 Jun 2017 19:06:30 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mail-qt0-x232.google.com with SMTP id v20so24925980qtg.1 for ; Tue, 20 Jun 2017 12:06:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=BRaoUcV9CeWwiCmvNxpzBWvgXUxud6DxwAWF4UKBtA8=; b=t7TDseXxwNXBt2t6KWtwk/XsXYi7ncysMj7bEf/gXCAn2V5beUoImmKg8VGfMvkxnS pBcTFTS4LeM5R5ZBgNqx3y6xf5DImzzMLRuSWaqvGtzdwtXp+79CDO8iiqwlStOPQhje rV4H8SOYEQcyPlUrzMbcgfWZI+0Mc8NJg1u3mZB3OIuwtTMtpdxC8GwoHXZgbc6rlVAl slhfstn35bDYml0PL2THOs96XM+rdHcZph9dX5UTIaTwNJkD03zSCM8Yc7l21UayWNLq Pn9GGeh0u9P2lhKCTtloy0EFpjYQ3Xhpd2f+5ZJOZmzpsO+v0q+uOkVO/IIj4Q0Xvvrb fieg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=BRaoUcV9CeWwiCmvNxpzBWvgXUxud6DxwAWF4UKBtA8=; b=YW/L3r0ZV6AJTpHkmcM/Rf8N9wxiXyMEiScKHJlUZDC7FFYXQiui+GIQ/vE/ia4L7N xshf2l2yeDBoRfsQpubf1znCIcochVCt3qrlNDxiVg57WKozTOYj4JioCBuT06AnU910 uJxaA3nOx+Yz4xhRVLHrtlX+OJp3JWQeoj+8DEhZdRjfpsPlHMOIPAjY9nopGPMFFD7U FT0uu5Xvq8u+LwHm2p1RWKvl8Drn6lVPGv6p8dUAOqEI18YbTDX54lrh8oEE2/08GEuR R2ZQ9BAbzgcg9d/Fe7lqyUhXE1UoajHCppqAKSyNibV267hv2N5aefYVtp4BTrmThfpH EkZw== X-Gm-Message-State: AKS2vOxKAnGbKrXjLPXa/UtJkUQqOVHlkEBPFJiE4ptgzqjaGAshcPG0 fmqAPw1UXI1g7i6h/xziclBj8Anx7A== X-Received: by 10.200.33.235 with SMTP id 40mr38984003qtz.189.1497985589945; Tue, 20 Jun 2017 12:06:29 -0700 (PDT) MIME-Version: 1.0 Received: by 10.140.20.151 with HTTP; Tue, 20 Jun 2017 12:06:29 -0700 (PDT) In-Reply-To: References: From: Freddie Cash Date: Tue, 20 Jun 2017 12:06:29 -0700 Message-ID: Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs To: "Caza, Aaron" Cc: Karl Denninger , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 19:06:31 -0000 On Tue, Jun 20, 2017 at 11:50 AM, Caza, Aaron wrote: > I've observed this performance degradation on 6 different hardware system= s > using 4 differents SSDS (2x Intel 510 120GB, 2x Intel 520 120GB, 2x Intel > 540 120GB, 2x Samsung 850 Pro SSDs) on FreeBSD10.3 RELEASE, FreeBSD 10.3 > RELEASEp6, FreeBSD 10.3RELEASEp19, FreeBSD 10-Stable, FreeBSD11.0 RELEASE= , > FreeBSD 11-Stable and now FreeBSD11.1 Beta 2. This latest testing I'm no= t > doing much in the way of writing - only logging the output of the 'dd' > command along with 'zfs-stats -a' and 'uptime' to go along with it once a= n > hour. Ran for ~20hrs before performance drop kicked in though why it > happens is inexplicable as this server isn't doing anything other than > running this test hourly. > > I have a FreeBSD9.0 system using 2x Intel 520 120GB SSDs that doesn't > exhibit this performance degradation, maintaining ~400MB/s speeds even > after many days of uptime. This is using the GEOM ELI layer to provide 4= k > sector emulation for the mirrored zpool as I previously described. > =E2=80=8BI don't remember if this has been mentioned yet in either of your = threads on this, but what is the output of this command on all your poorly performing systems: sysctl =E2=80=8Bvfs.zfs.trim.enabled If it's set to 1 (the default), set it to 0 and re-run your tests. ZFS Trim support for SSDs was added to 10.0, so any system running FreeBSD 10+ will show a performance drop after awhile when the trim function kicks in to clear out deleted/unused blocks. Especially if it's an SSD that can't run Trim commands in parallel. You can look at the various ZFS trim-related stats to see what it's doing: sysctl vfs.zfs | grep trim --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@freebsd.org Tue Jun 20 19:23:29 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C2719DA02E4 for ; Tue, 20 Jun 2017 19:23:29 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [67.205.158.196]) by mx1.freebsd.org (Postfix) with ESMTP id 9903A70013 for ; Tue, 20 Jun 2017 19:23:29 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id AD59F27386 for ; Tue, 20 Jun 2017 15:23:29 -0400 (EDT) Received: from [192.168.10.20] (D10.Denninger.Net [192.168.10.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id 8CFFA3F3C for ; Tue, 20 Jun 2017 14:23:27 -0500 (CDT) Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs To: "freebsd-fs@freebsd.org" References: From: Karl Denninger Message-ID: <54684f63-81c4-053c-7253-8aa07433e8fa@denninger.net> Date: Tue, 20 Jun 2017 14:23:20 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms050708040103040502030205" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 19:23:29 -0000 This is a cryptographically signed message in MIME format. --------------ms050708040103040502030205 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 6/20/2017 13:50, Caza, Aaron wrote: > > I've observed this performance degradation on 6 different hardware syst= ems using 4 differents SSDS (2x Intel 510 120GB, 2x Intel 520 120GB, 2x I= ntel 540 120GB, 2x Samsung 850 Pro SSDs) on FreeBSD10.3 RELEASE, FreeBSD = 10.3 RELEASEp6, FreeBSD 10.3RELEASEp19, FreeBSD 10-Stable, FreeBSD11.0 RE= LEASE, FreeBSD 11-Stable and now FreeBSD11.1 Beta 2. This latest testing= I'm not doing much in the way of writing - only logging the output of th= e 'dd' command along with 'zfs-stats -a' and 'uptime' to go along with it= once an hour. Ran for ~20hrs before performance drop kicked in though = why it happens is inexplicable as this server isn't doing anything other = than running this test hourly. > > I have a FreeBSD9.0 system using 2x Intel 520 120GB SSDs that doesn't e= xhibit this performance degradation, maintaining ~400MB/s speeds even aft= er many days of uptime. This is using the GEOM ELI layer to provide 4k s= ector emulation for the mirrored zpool as I previously described. > > Interestingly, using the GEOM ELI layering, I was seeing the following > - FreeBSD 10.3 RELEASE : performance ~750MB/s when dd'ing 16GB file > - FreeBSD 10 Stable : performance ~850MB/s when dd'ing 16GB fi= le > - FreeBSD 11 Stable : performance ~950MB/s when dd'ing 16GB fi= le > > During the above testing, which was all done after reboot, gstat would = show %busy of 90-95%. When performance degradation hits, %busy drops to = ~15%. > > Switching to FreeBSD 11.1 Beta 2 with Auto(ZFS) ashift-based 4k emulati= on of ZFS mirrored pool: > - FreeBSD 11.1 Beta 2 : performance ~450MB/s when dd'ing 16GB file= with gstat %busy of ~60%. When performance degradation hits, %busy drop= s to ~15%. > > Now, I expected that removing the GEOM ELI layer and just using vfs.zfs= =2Emin_auto_ashift=3D12 to do the 4k sector emulation would provide even = better performance. It's seems strange to me that it doesn't. On one of my production systems (albeit in "hot spare" mode) here with ~20 days of uptime (plenty to saturate whatever, and this system DOES have my patch in it.) [\u@NewFS /dbms]# ls -al total 65580101 drwxr-xr-x 4 root wheel 5 Jun 20 14:06 . drwxr-xr-x 45 root wheel 55 Jun 1 10:58 .. -rw-r----- 1 root wheel 33554432000 Jun 20 14:13 test drwxr-xr-x 2 root wheel 2 Feb 4 2016 ticker-9.5 drwx------ 19 pgsql wheel 29 Apr 29 16:51 ticker-9.6 [\u@NewFS /dbms]# dd if=3Dtest of=3D/dev/null bs=3D1m 32000+0 records in 32000+0 records out 33554432000 bytes transferred in 43.023505 secs (779909306 bytes/sec) [\u@NewFS /dbms]# uname -v FreeBSD 11.0-STABLE #15 r312669M: Mon Jan 23 14:01:03 CST 2017 =20 karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP ~780 Mbps, more or less; the test file size is ~3x RAM and was created using a dd off /dev/random, so it should not benefit from compression (which IS on.) This is with 128kbps recordsize and lz4 on that particular zfs dataset.=20 The physical pool is a mirrored pair of Intel 730s and while this system is reasonably quiet right now it's not quiescent. ARC target and fill at present is 8.43Gb (out of ~12Gb physical) --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms050708040103040502030205 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC BlwwggZYMIIEQKADAgECAgE9MA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE2MTIxODE5NDUzNVoXDTIxMTIxNzE5NDUzNVowVzEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxGzAZBgNVBAMUEmthcmxAZGVubmluZ2VyLm5ldDCCAiIwDQYJKoZIhvcNAQEBBQADggIP ADCCAgoCggIBAM2N5maxs7NkoY9g5NMxFWll0TYiO7gXrGZTo3q25ZJgNdPMwrntLz/5ewE9 07TEbwJ3ah/Ep9BfZm7JF9vTtE1HkgKtXNKi0pawNGm1Yn26Dz5AbUr1byby6dFtDJr14E07 trzDCtRRvTkOVSBj6PQPal0fAnDtkIYQBVcuMkXkuMCtyfE95pjm8g4K9l7lAcKii3T1/3rE hCc1o2nBnb7EN1/XwBeCDGB+I2SN/ftZDbKQqGAF5q9dUn+iXU7Z/CVSfUWmhVh6cVZA4Ftv TglUqj410OuPx+cUQch3h1kFgsuhQR63HiJc3HbRJllHsV0rihvL1CjeARQkhnA6uY9NLFST p5I/PfzBzW2MSmtN/tGZvmfKKnmtbfUNgkzbIR1K3lsum+yEL71kB93Xtz/4f1demEx5c8TJ RBIniDHjDeLGK1aoBu8nfnvXAvgthFNTWBOEoR49AHEPjC3kZj0l8JQml1Y8bTQD5gtC5txl klO60WV0EufU7Hy9CmynMuFtjiA2v71pm097rXeCdrAKgisdYeEESB+SFrlY65rLiLv4n8o1 PX7DqRfqKkOYIakZ0ug/yHVKcq2EM3RiJxwzls5gT70CoOBlKbrC98O8TA6teON0Jq30M06t NTI2HhvNbJDLbBH+Awf4h1UKB+0ufENwjVvF5Jfz8Ww/FaSDAgMBAAGjgfQwgfEwNwYIKwYB BQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgwCQYD VR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIBDQQf Fh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUpfAI3y+751pp9A0w 6vJHx8RoR/MwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYwFIES a2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBiB6MlugxYJdccD8boZ/u8 d8VxmLkJCtbfyYHRjYdyoABLW5hE3k3xSpYCM9L7vzWyV/UWwDYKi4ZzxHo4g+jG/GQZfKhx v38BQjL2G9xD0Hn2d+cygOq3UPjVYlbbfQoew6JbyCFXrrZ7/0jvRMLAN2+bRC7ynaFUixPH Whnj9JSH7ieYdzak8KN+G2coIC2t2iyfXVKehzi5gdNQ0vJ7+ypbGsRm4gE8Mdo9N/WgFPvZ HPFqR9Dwas7Z+aHwOabpk5r/336SyjOaZsn3MqKJQZL6GqDKusVOCWt+9uFAD8kadg7FetZe atIoD9I+zbp59oVoMnkMDMx7Hi85faU03csusqMGsjSsAzWSI1N8PJytZlchLiykokLKc3OL G87QKlErotlou7cfPX2BbEAH5wmkj9oiqZhxIL/wwAUA+PkiTbEmksKBNompSjUq/6UsR8EA s74gnu17lmijv8mrg2qMlwRirE7qG8pnE8egLtCDxcjd0Of9WMi2NJskn0/ovC7P+J60Napl m3ZIgPJst1piYSE0Zc1FIat4fFphMfK5v4iLblo1tFSlkdx1UNDGdg/U+LaXkNVXlMp8fyPm R80V6cIrCAlEWnBJNxG1UyfbbsvNMCCZBM4faGGsR/hhQOiydlruxhjL6P8J2WV8p11DdeGx KymWoil2s1J5WTGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHRmxv cmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExDMRww GgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5c3Rl bXMgTExDIENBAgE9MA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcN AQcBMBwGCSqGSIb3DQEJBTEPFw0xNzA2MjAxOTIzMjBaME8GCSqGSIb3DQEJBDFCBEAIAPEz c4rdkKOywgcEczoaTsBr9qB5PWEbgFRI8uU8n0UGfSroTnG5h1XeATa4qnXCStQoPdzaDeOE Zcl2fxRlMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggq hkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZI hvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNVBAgT B0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExM QzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3VkYSBT eXN0ZW1zIExMQyBDQQIBPTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYTAlVT MRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEg U3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0B CQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECAT0wDQYJKoZIhvcNAQEBBQAEggIAHgol0gzU2C5H rjntgfZ3W/KW2VXwWkxO3GONl9aqhr7mCuNDCBu2uUDqzuhuyYzXs+/uVVJAwWK7mMVo05eq 0/1gfbSIuRAt2BBsUXvTG2AHqNOYb6nEiLnoJmXOUvcxBTauqBXjZiO6P7B/ytzP4VRbUegz p9LrLtj/Xx6kHt3MEmiW+1Qh3s4EhnMfAuO0q3X7Q6ui2D6FwBXKzcwndZmyNO1AlDt6VvQ8 4W9L3ybnAVf4N0HC7IYPOUzy/E8IuQs6Byg8p7DaFOmdPZAtw7Y3byNwPsTT4fRFt9BiQAE9 W1iWNq6U9VV9QRNbBPQyBPRZgSlQrT0yfD9G2xUpxvLZL92BJu0fs587Lx5fGFUMgTug6fxD MJv7F5HI0IpfoU2fPwcZ0uMs5U4jGS7dSwXYkabnXlpUBsX0FYvDzXRtnzTw+00D29fuqE25 h7XVCZVRqJ2J+baamtkNFt9izXQ4c4n3taUBhh7LE+KDCzeGgfn2sDWJMQf14OjOciN0dtyC utPqvaxCs+lrZRv5ESXPbA0iLGbsSDWncv39ilbhz3tjqRQObIGXfTxiI9l4h/ki4WW8nhUu XOqKI6lkzTYHwvcqn9IjQXcP8h4YsyeL8+eaR8zUmYsCMflsprQDVpwIeXcQYXbCuiXPAemP l/nwqHBxhvXAdPbi3sdgcAAAAAAAAAA= --------------ms050708040103040502030205-- From owner-freebsd-fs@freebsd.org Tue Jun 20 19:49:10 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 11CEFDA0C71 for ; Tue, 20 Jun 2017 19:49:10 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) Received: from NAM03-CO1-obe.outbound.protection.outlook.com (mail-co1nam03on0083.outbound.protection.outlook.com [104.47.40.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9A5FE7108D for ; Tue, 20 Jun 2017 19:49:09 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weatherford.onmicrosoft.com; s=selector1-ca-weatherford-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=Y8zZoSN2OeG8ZyhqLnd5ublbILHj6Dm4uwCwLMFqQJU=; b=ZnmepqMJ0696qaOp2viyGCGx/qLfJSFgg/dS+by4/TGFdhJKQZ0KmGFZOj+Cv25BeVFtzqLMVWUX9PYzxM/ZufTbgS7/inoFKMaT45m+PKYTrPBEQS9Pxzp/GUino2qgopCEd076Q6zEKlRrgOIxzkLC1BhIgAE9rgmu4Gi+Jx8= Received: from BN3PR03CA0102.namprd03.prod.outlook.com (10.174.66.20) by MWHPR03MB3165.namprd03.prod.outlook.com (10.174.174.34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 19:49:06 +0000 Received: from BL2FFO11FD013.protection.gbl (2a01:111:f400:7c09::193) by BN3PR03CA0102.outlook.office365.com (2603:10b6:400:4::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1199.15 via Frontend Transport; Tue, 20 Jun 2017 19:49:06 +0000 Authentication-Results: spf=pass (sender IP is 23.103.226.20) smtp.mailfrom=ca.weatherford.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=bestguesspass action=none header.from=ca.weatherford.com; Received-SPF: Pass (protection.outlook.com: domain of ca.weatherford.com designates 23.103.226.20 as permitted sender) receiver=protection.outlook.com; client-ip=23.103.226.20; helo=032-smtp-out.weatherford.com; Received: from 032-smtp-out.weatherford.com (23.103.226.20) by BL2FFO11FD013.mail.protection.outlook.com (10.173.160.221) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1157.12 via Frontend Transport; Tue, 20 Jun 2017 19:49:04 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net (141.251.110.81) by DM2PR58MB015.032d.mgd.msft.net (141.251.110.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 19:49:03 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) by DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) with mapi id 15.01.1178.018; Tue, 20 Jun 2017 19:49:03 +0000 From: "Caza, Aaron" To: Freddie Cash CC: Karl Denninger , "freebsd-fs@freebsd.org" Subject: RE: [EXTERNAL] Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Topic: [EXTERNAL] Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Index: AdLp8ugzXKKDkz6vS/yx+crAeAsw7AABWomAAAAKKOA= Date: Tue, 20 Jun 2017 19:49:03 +0000 Message-ID: <5d5f83de0a1a4bce8d01bf5d239b24b0@DM2PR58MB013.032d.mgd.msft.net> References: In-Reply-To: Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [141.251.136.68] x-ms-publictraffictype: Email X-MS-TrafficTypeDiagnostic: DM2PR58MB015:|MWHPR03MB3165: X-MS-Office365-Filtering-Correlation-Id: b1294c5c-ca41-4f5b-8e2d-08d4b81568d5 MIME-Version: 1.0 X-OrganizationHeadersPreserved: DM2PR58MB015.032d.mgd.msft.net X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:23.103.226.20; IPV:NLI; CTRY:; EFV:NLI; SFV:NSPM; SFS:(10009020)(39860400002)(39840400002)(39450400003)(39410400002)(39400400002)(39850400002)(2980300002)(438002)(199003)(189002)(24454002)(377454003)(9170700003)(86146001)(5890100001)(53546009)(84326002)(66066001)(229853002)(54906002)(55016002)(42882006)(22756006)(6916009)(86362001)(6306002)(54896002)(9686003)(236005)(7736002)(189998001)(356003)(6116002)(478600001)(512874002)(3846002)(102836003)(790700001)(2950100002)(2900100001)(24736003)(53936002)(4326008)(81166006)(50986999)(76176999)(108616004)(54356999)(2906002)(7696004)(8676002)(106466001)(39060400002)(33646002)(8936002)(110136004)(38730400002)(72206003)(1411001)(6246003)(5660300001); DIR:OUT; SFP:1101; SCL:1; SRVR:MWHPR03MB3165; H:032-smtp-out.weatherford.com; FPR:; SPF:Pass; MLV:ovrnspm; A:0; MX:1; PTR:InfoDomainNonexistent; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BL2FFO11FD013; 1:CocSLNcn5tqnqo60W8dZwaFeXlScg9yzWjekSTN2+N5DCTbK4VEHFXTvZ8jJ92VCZ1e0FuKXxhvuutqamzwUWMrxm1n7Q3Jf5UoeHaWU7laSKYC4hTRmEOfqYOvmgFI82DOixb/NN4gF7+7V4rUkh7KFsiFV9V0Jq0c9t5Xa/3EV1cQuB8MrjNWLX3V4MkmJhDT4beuHjheFEWqXfn8JgwHrip+R1YWda7E7uU5KRzjGZ5oBkCjfSGK4CQzwmBWBGtdIr2cQ4B7fjRxBq25ggGw+bctv/PZ2Kkpv5fqGLDkk5ZcsbMEqb3Ufefcg0Y6le/qvvcth7fhaOL3HYhY0oCcgLvGOy+h+VbK/zX2qb92LhHxsHq5XQoyusAZ66jPJM7lZ5SmLVXrKGvO6nXl/FSRLDZETd8LqMl/y5GHGJgEgQaBcj2xhHuXOy/yyzQZvLTmY7SsH1kQb23YYHqZnJUsE0bUgKtWhmZ0hxNXPjWkHPnMSRw4no7M1AcvniqYP3b5j2qU7ywKP62BsQO8NbsbS/uO4W+poy77lGxOCokcerjDKUJx9DP152D2PGnyTT1OlcCwYjWGUP4qVfRX0VXPDaswiTYsuzmQwTmpC00jkTM4ba5SiZviH4BO67W2NPxITv7xUl+mHvdWCbFoubKpncdu5rHQ9FNplnNpRgyrv+tQH/3X+5KEq0+kGZo7vst+3YJt61x0hYx1poFeo+DIPZoqd67pN68MIwMzqybyypp5khFifXqFlsRDKQkq/z3FkDY1J2n6/nqz2QMHTUonBWMsUJ2fGsXyHu8VPJ6DIa7seS8UuR4ugA4E6sfUBx+UDlYtu3kC8djkBK54a8y9kTqBZmmlo+r5ftm+bTQ3UFGiyMMMmWLjpAa7jhnxJ X-CrossPremisesHeadersPromoted: BL2FFO11FD013.protection.gbl X-CrossPremisesHeadersFiltered: BL2FFO11FD013.protection.gbl X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(8251501002)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:MWHPR03MB3165; X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3165; 3:TYGfozvCJDNJPb0f8Qdw8PC5b55BnphAKiTlYn4PSnbu9TkSzXE4SZ/F9phrBNZdn60oZ6dylG30ojN9NrrB+lr/bbbTXE1Z/VDBuNye/wPf8ttuY5Cjtf4/qprbq5jZCvJKtupplbrSvP7cC3L4eGxqMa+qcAdZ1Grq2EowRBSBYU0XUC7MNTRIUZkyQG655AJjldyBv8tXH2m12XyBwL2TAjySOwr75A3Ky3W/co57WRdsk/OqYW5UpRdyiURouwTlhR50BPFuncarDXlUxO7ELY5JhnOuqhnQqaQ4ixUNZryUOYNbzoJ9f2MJXdxzSWrNL5BreCTo8N22j5wIsJHUFQQuh822C6mjlZUTxuQ28rqsyjc600BokQkiUKlU7qMJm6ORIjYCwYHu0/K/+wevfTXskTKC3JaKtBqax34Wo8Uyd4RxH1i6pAvk/K7GQLoX6WA6iB6wknTgluqiOI0QiYuvoGNw81drYsnBa5fcOi95HifVywYz5b6vROa6kGA5CyIV85IqJvBZp8Hwxw== X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3165; 25:fiYPC7graZ9unW+KK1fHjsZ4lC6pxKJBBvfcqOXS0bNVHzb0ux6i4mlKiFZYNzZHpZi5ZY13DuB/+Y/A6lg/4ZZmtXaW+pLww9TgO3xBywaDAmirvlZtD0fmp4L1vYisH8l9NO+ZV2w9jdExowdQmdDvRLxiPKfIvDAAS4n68aaagSeI6OPTYfdQ3tD2/fh4eLWaVFyFg3pXhWs76hLAIdkKyIhELV5LIEo+FfzTdSPvXH76a32+KZZF6c7enta2MXRZtFFUcOWGWM6PVeKtUhVwnW1JBkjSCqvpwHzOFaTjByKjh+uILWyZNc9d3g1laLmR/edKoOiVwfrQbV7MZFgo6IaQFnXOvorc96BmqhutFtsmovXEGxW750/Yayr+neT6CrNMnqaa6hPpMJKvuCXgNPN9Wd4NtWQdvZKj1R/QmNUK9ztWtoccxVkPMMH/ITNu11R0+GlN3d1QwdzflpzGkxhIZlxQmdUw+GPAyCM=; 31:J//P682tmZBCUDEJ/AWlGTNkxkKWcvg2wjwfe6q92QGvFu6ti+cTJ1aE8WSypSVKPTjx9JG/ahVMytMBLQ31ZdF3AYQ94SF+vD5dfjuFOOmtW/zQds7fcEjRi8nr4rXAitDPQmEzOE9zBxel1aVkK/utAgm3gq5oozwnRAsygTjKUThBfbZ+EG61vtaO622TVBzhbVTOUPHXaJBjkUkB1h/MX+0kVWwim+clsUKHzlq3TuSle6yGwhEJT7Tk2TJ16Ddvv5wiq0E3MvCgxGI/KA== X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3165; 20:dIezVME9Nk7D4MEgL9AjnU3nabsr5EHUEHskcH/vzEzeeSmM2eLkHn5XuQOETLHKB7HCzDAj+D83HzZ2X7tbH5sFvsuIumzLvsgW0Hl9OBSe428r3y/h/NBIHYf/Je3y+cEohBCMOZ0dFjc2RidAzHKgMoULS2mEeA9pr0A9CH3GazihB1H52DVoPDAa89U8MK/CK5CjpbkYfgUaJijbijPHmL133qHvzsNb6mRRPdz5NzeZ40520hBTMv/9IwLJysLaWnrSLLDkeyHqZNjjh8dSxN30QKMjorpW4tHWtH8lLtGu2RdmoKUyRt/9ydmGrFWCyp+UllItNDD+SGjCaSHJdFxAy8fzSSPDJwN1iwXL4oopavlbJhDelhSjVzCppiytUDojJ4JAg8UAdLvyLLwbHysUHjTj42uQYz4aB5fK2v3j2ZUAmhNR7NfCgsMzhxabP/S5OQuSyq60cpXq4kVvlSbeTsZfai7YpxhWGAHGj/+EfwYxOrnLIOETI/jW X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(158342451672863)(84480959824636)(192374486261705)(21748063052155); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(11241501159)(6040450)(601004)(2401047)(13018025)(8121501046)(5005006)(13016025)(10201501046)(100000703101)(100105400095)(3002001)(93006095)(93004095)(6055026)(6041248)(20161123564025)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(20161123560025)(20161123558100)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:MWHPR03MB3165; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:MWHPR03MB3165; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3165; 4:cE0M2YDUAuo2QyAZkLLVGtkyO/ozzBlQq9CYlTeBLH?= =?us-ascii?Q?XCqIaiSoBaqPX0/BowNi1RgGHeBzP2chl+VQIyi2TMBv/J7RWaVthzj22bB5?= =?us-ascii?Q?c4BRsfgQNNuH8yV1BTjpQgpjwgZnkC4WBU4sz70lqh+aGOt4OcTuVDY8dt/Q?= =?us-ascii?Q?Za9VvXWHsMG+JKBhZbEXLmQR5OWb+LjRMUeHqVZjSirCTErJmiP2hWwv2XOu?= =?us-ascii?Q?Lsf5VD1fBH7erITG7XdvK4/XnnSZFb6x2PTImVMfvgkSeYqnyfd65y7puPxA?= =?us-ascii?Q?DaSByaxB36gy6A7Quav7T5Rrr9w6j14e2ULEkCF19z9g23FfhjqlINiA4kFH?= =?us-ascii?Q?rmIgYvTWRZe9VBLHo02pyM3UtFX2VIqMU43To1NSnJAi3iIcz+Cki5UnmPGh?= =?us-ascii?Q?H660iekVHH7R186vY5fD56TrAmwsyohe1ZzW/L8CU+48BPGuQdJO6n4HWkiE?= =?us-ascii?Q?qGZAcTGv3HsrvhWd3VBRCmfl+o7YJLb1L5OPiney0NZWii88ZiJxrBQF6C0B?= =?us-ascii?Q?1gfJjVpU4N4102Os6BILDbyjBsPWvUyLC9ALHbXTPNTpluqk4qTNX886UEnJ?= =?us-ascii?Q?svEvUtPhgGZkp17kjrMgozM+fASynpXoDD3sdXPHovmsxfFwqPNUViysCxxA?= =?us-ascii?Q?pf/PqbjQStwrZ9LkMZP6c77BX4MpUCe0YHX56cCl+1K3M0rHKnjbG1bHzvsx?= =?us-ascii?Q?V1FSNE85Sp2GTS8qymyje+WrWIWbDTVP+97uRVtSbloHz2in78+8vuBEgWjl?= =?us-ascii?Q?idD6fhuOPUf8Mvoj5j7Pkc7P1GDauF37LiPECBMugPbtqFzMQqgoU1zoZQv1?= =?us-ascii?Q?jrPkJ374JU/TMVThdqiWnSaruj7KsgRaYcr9XWuyftQXRF4Xr2NDAR7SWyWa?= =?us-ascii?Q?Cw/grPSdd6a+p1BZeGuhc6/x4V99PGFxzefCYISjMfM6MTW7UJ/e+kwtJoJE?= =?us-ascii?Q?v4XiKE76a1YjQwtX/gjDPt4ckdF5hGtGsyKlyoK6yI6nZtEfeVGjqQ/a2OYx?= =?us-ascii?Q?P/YdxQiDQw+mDXjymd2mEE5rS0lZcTOZghZ45sBAtj14U5zMvtI3cQ42Psyk?= =?us-ascii?Q?H4wgcdiVvT7kH6iHayNBgZBoHYBOi4AJClx8TNUnwNwgphR/sqHFf4kLJ8Aw?= =?us-ascii?Q?gzonYBugUv7POZYIp47CPv8tV8lmXrSZ03BzHXVW4iDz6KR+MNxqBDk2FX+Y?= =?us-ascii?Q?BPab+eWKtYS0olD+0pY27r6jqRGAcOYYldjxbdDbbgMQ/c1bdv3UdMENTQg8?= =?us-ascii?Q?XrSbILB46hKZ4tthyAEZwwXUFPPMp9/GR9zmRgdp3RWawTZQq1aU1xbWbREh?= =?us-ascii?Q?rJo9x4rTOs/u/F7/6ugKrfpCrbIskr72iCMZ/Z1FmNJS3zgPYY5wAmv7s606?= =?us-ascii?Q?QL7F3diHDN4ZLvTkFxf2Ds/54=3D?= X-Forefront-PRVS: 03449D5DD1 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3165; 23:io348s0hkaJil4w4dEGT3gbAGV87SZfpEYnqFkE+U?= =?us-ascii?Q?jS66qhKxiyio4UeSDOXSB5wYUXlS5CAvqkk0oFRZ2kFC1B2JJpFyVBEseAoE?= =?us-ascii?Q?DehWmS7i1dLftl4YWr8bVQm/sBtk2/cboY0C9akiUScs1qK3aLURZDTnNKRI?= =?us-ascii?Q?SrrlA24g8tT8X9LQu8C33YIuNR3TkpgdzOa0lqtH+7ZjSmTEgJROHahhp9k1?= =?us-ascii?Q?IXLqay4KpBNExohkHbVxyz9dtaiuGItcAdDFHJrIzL6IxJIhSvrtxb9Yx4l3?= =?us-ascii?Q?NRm6PjaYfGfBWqMJR03HogCMMXXWEfoTz3rWbWIYnbxVixRDgfAUnUpMohqE?= =?us-ascii?Q?dnys4I5RjyT4HYTxcJWZwDggh2jCvZ1r1ghy8hI2DKGa3fdGIEl05HShdyqL?= =?us-ascii?Q?CBKfgHN16N8TmWEgTpi3DCE3FguUDbaqRg096xOZFzn61b1sFt2QYD9QL6PX?= =?us-ascii?Q?Tl2xt7Mdgp0ZmD3cFt+rDgu1OjWnnTSquk79yTXowyy+O9CMENeH36/a15iV?= =?us-ascii?Q?DukpV+nAO+3kiCxV3bbw9jTDD7WmQFVbGstIqUfkQ8QpOVF+kMTcyM6bNMW1?= =?us-ascii?Q?E01KPhy5VRlasZLLh07crVtJDLV/hHx1uvNrjYSgDYMrIAUdQiR8gjnFpi25?= =?us-ascii?Q?P0erxMXRRdGjLG/VGbAmx+WOOaF7KeD2omyEXwdrsFWoWxnRJI1vaSYQC1fk?= =?us-ascii?Q?EfC/xCz75VQq1DdUhuz5JMjPLyqyZmWie4D147FUI8qLLyBVaiIQ8jOMEnvN?= =?us-ascii?Q?vv5Y2GekYLg+Ni4zYY2ksNjFMYBIp4JC0BVLeFlc1xvnS3GmsUUchgiylKGa?= =?us-ascii?Q?cZVJ9gZN++o3de60pv9hQ2vtj6/er3uZ4ngPK6KOWPH+h2hqtJrovZoOaFhB?= =?us-ascii?Q?wcSr/cmgGERZeehc5pt5+FmkEWIDNNHpt6VV62yMPNqg6b8qnFYTGYgurg+u?= =?us-ascii?Q?wVlCxJHDWal+CChF7YsfzjohCooh7Cs7FecDnGD3PVVX9xDNb5l/662FDECe?= =?us-ascii?Q?F/Xr2CL2RHGBGboMYR3qDhHESf9bcWP7tp9XOfw00uHzT+/azVEnZiTjxF3N?= =?us-ascii?Q?rCPF1Xx2YX6gAMMDFkELt2NunLhoqncApj9ss5Silu7ilGnriwJXPQeGiLGB?= =?us-ascii?Q?pj7UAMJ6OfBR2WTviU5JXnW+u80VUJLx6QnUMYb83oltR5pEvtLZf1+4I6BC?= =?us-ascii?Q?9iJyu8Laq7O9x80bLjoX0Wzupl/2Ynn/9myTdBfXSa2PXWG5R/jKLrn7safI?= =?us-ascii?Q?WamZyr8XYDC4RLvP1z/9feugKGJgRNSkmUpmzifHhGCK8xEYhnAqPPLUDCRl?= =?us-ascii?Q?l+LTPC7LLwXKpLxy5+j29AW8pGitnMsfR3qeqXzKZoG7T/EjJSlu+O0QFuEU?= =?us-ascii?Q?5rOUjccB1IhT5B62kTiOsDXsQqi98GIAKBRhKvIJIJ+OhPzmd47eHBGOuAKx?= =?us-ascii?Q?XWL5NxPQxMiIMdp6QdENlPDps+M1+nHHd34RHM6X6a0yBzArLHl?= X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3165; 6:WSQ18UOEPOHc15INko5VljCA8+ZNRtdhz/BhIo9npS?= =?us-ascii?Q?q/7f2kvW9FSrb3MD/36qAtqzBMh+pIgkWlkzXmdeKMrEba/rNbaMwc8KDGfA?= =?us-ascii?Q?Tl5cFo3EUz2ApBoM/3ppM+7CToKSDUNNtkv03sj8HvbJwcf2CrNwwS5IEMIT?= =?us-ascii?Q?inRPAkyw9r5Mq4HkXOa9ysnpwRlSTGkCTD6EKckUZM4klvINhkP02KDW/8uw?= =?us-ascii?Q?LeoOwMWCER+V/aLsaE3htFeFVilw068uqtmr5X87lYndxl/KOd1P5fLHU0ZV?= =?us-ascii?Q?PQ9qnbagP6whaYOfm8un+a6NH+thKCg+rWk4wY1AXSNEd3pEI0NmOtf4Df5k?= =?us-ascii?Q?+IBom0zCkJjxLvuOKm/d/XDN7aiWaYZNrd4BPVpWXEenodB1Z6rUA4z14Tw/?= =?us-ascii?Q?cuHM23GTysUKhqegEpQDVXnTO3nrlud8xpdZl5AmbCdUg0KPpyX4tFe9+nu5?= =?us-ascii?Q?J7VQuofzBde8DsuSZwhjv9Ir+HqweSB/4fhKClwNlG5fgJS6YwQRGgC54PH/?= =?us-ascii?Q?ncbn0RIU2p1oYrOr+/JEOXovh/yazu70Pni6GEe7EOQts9OQjBKNwGGm9pj5?= =?us-ascii?Q?rbcLG1e0HafjJqlISRPCQM3TWFX/cF/zfdclbmrLdO1XuG5y2tECZBaGLX34?= =?us-ascii?Q?IF9sMbFJLsCiP+iHUgKWfUlEYbrTv3HEuWXQXP5leuRCFpHJaRDf/k0Thkr5?= =?us-ascii?Q?02umjWpS34vzDpCVToK/WbQP+OI83OONj2YSvbekR/LMThnfbM+ObPuyx3cW?= =?us-ascii?Q?5h9F53TcRzVMUehRi13oxqIr5x+LDVdMbnRIE6GC8ZZmdSYCKHhEDefHP9dS?= =?us-ascii?Q?ERuMYj0y1I2S3i0m3N5Xk1Rl27ZfbL6w97QYfLFD+Y1SWYnzC240KrJneFYp?= =?us-ascii?Q?A50lG7I68VTsfDR3/MuYKSu63LZiPAS8An4e4IxbhFVa/J4WUg53hoz1aWi3?= =?us-ascii?Q?5IRSTjkHJi1/X2qCWccGoWku+pdDNJhz8vC5CQk5CiOr5QB7ggpLZBelS0H4?= =?us-ascii?Q?ZyxrV7Y8aJAdjg1PvUgYNS?= X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3165; 5:eadtb4PIWlCx68tHIq4IgPbYzql036gnbh0GzrHW3seAcv2WJJd864I6XY1uRhuhq/yrq5NOFR+9dh5V6BpMNnucVkLoguHkVKLIHjrpw9XONiQsdzN8zX0xlVYknM2pz8r7vdbV083eiLPrjlOuH45P3lT9/Hg2LxzVsJQ6EsOfr6HEY54BtK/p5BHj+fIS+Ky3+86A+lBIrYllXhcWbm1SH4WNT0POMgl/VswWv66JOv3MJC+/I8EHae1BS6HnvX+u9vun3cxGVRMqM2YOqdpZC0b69ZTyu8Z8sTUBD1n0uE9OzVtdkkHO0dgtdbZXEG20fD8DDq3R9a2q4zaQV0p0HeNZi9fou772tYo3oX9e0KDXu+R8uJSF5tI9iSpEPXPsXu/4eji3Xd0OHAB7pFTFNstcxflPzXW6kThQm+CNki8ViZXNlN07u0MjarF49FtGLIGopPpyBnTXhXZnnpPFINGGSQUnxLu/KwXFHmdOumPxljPobmRq1a1/anh2; 24:O9q2d70ppwUwA1WhXwqySdUC+zuWB0rZ1EI4VjfGAPf83JUG70YOLHPbRMgspTpN780l04jOBX4Zp5019KwTkQLbo6oxBstST4Ou/4plOYM= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3165; 7:MUM3XIVpc6JZj4GcBZQxo6jqoLmw9UTNo9cVw5asjCNwpWkgGzAggQKl+9436W1xXj+Kk8pzM/Wncz2AgSZTLT9dwIGkQa24sweO+T3RH8pLd1ejtxKORFTrsFCyKpWjHOO2AuC+JuHAh+7mhZPWfkWT8hHgn1R9dD1RRXQRxdAn/ocTXyBXWKSnRRszxnj3zn5sQmB8v14UFYgBlQc23r8hIROFKEX+xmdY/BCaNGBPmnB4bMXvuLSwzOxIY9+Ciy4U2EDf/oyz5hYQxouXAbRcCN7U9nsH9FC/yRzbEihsZ0BhGYaxR5pk+sZwiJ2FMg/IWTbiGYWywQW6s9pS63THOqjnrGamz4Dn0iuU9N5Rd8lpaPf5SDwOcnzWcwckDt2IR4W7dm0bH7T5V2d/YoOvZq5T3SfOrUxr4xAJa44aHO8I91zTYaqqHyf7hbG88w+2D3sTx5PoMhaypME7znyTzJgjoU9kAYAcGFvnQrpftEkOYoigoemwbjC0VSI9IkGDYrnf9/iTestz/AjYMqE5qOHqzDkJXMe2XTydIa6hDkDpV+AVFGpjlS9VhrDztGl1OXrGDv02Qf2sdnmfMiOA6KpU86fc9NhUER0+tVxXYeb0KPqxMEn1+SQlvYe2apXzB/XLRjUEAQFnEAwCjLKFTh9rcm8dOpEhSWpeBqsDBcP3Oj7T8nYvoQCGnFHRdXmc6wcCrrT29UZu2diB4TsmVfd5PeVpke+6tnLSDmYIOJRflzcvabmDqvKpAvXhc4RlyMGAfhSWvozsxbPmjVQm0uMMhGXslSbMLuRsJ4Y= X-OriginatorOrg: ca.weatherford.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jun 2017 19:49:04.7656 (UTC) X-MS-Exchange-CrossTenant-Id: dd63fb60-07f6-4d96-8d40-ebeca61a524e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=dd63fb60-07f6-4d96-8d40-ebeca61a524e; Ip=[23.103.226.20]; Helo=[032-smtp-out.weatherford.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR03MB3165 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 19:49:10 -0000 Pg0KPiBGcm9tOiBGcmVkZGllIENhc2ggW21haWx0bzpmandjYXNoQGdtYWlsLmNvbV0NCj4gU2Vu dDogVHVlc2RheSwgSnVuZSAyMCwgMjAxNyAxOjA2IFBNDQo+IFRvOiBDYXphLCBBYXJvbg0KPiBD YzogS2FybCBEZW5uaW5nZXI7IGZyZWVic2QtZnNAZnJlZWJzZC5vcmcNCj4gU3ViamVjdDogW0VY VEVSTkFMXSBSZTogRnJlZUJTRCAxMS4xIEJldGEgMiBaRlMgcGVyZm9ybWFuY2UgZGVncmFkYXRp b24gb24gU1NEcw0KPg0KPiBPbiBUdWUsIEp1biAyMCwgMjAxNyBhdCAxMTo1MCBBTSwgQ2F6YSwg QWFyb24gPEFhcm9uLkNhemFAY2Eud2VhdGhlcmZvcmQuY29tPG1haWx0bzpBYXJvbi5DYXphQGNh LndlYXRoZXJmb3JkLmNvbT4+IHdyb3RlOg0KSSd2ZSBvYnNlcnZlZCB0aGlzIHBlcmZvcm1hbmNl IGRlZ3JhZGF0aW9uIG9uIDYgZGlmZmVyZW50IGhhcmR3YXJlIHN5c3RlbXMgdXNpbmcgNCBkaWZm ZXJlbnRzIFNTRFMgKDJ4IEludGVsIDUxMCAxMjBHQiwgMnggSW50ZWwgNTIwIDEyMEdCLCAyeCBJ bnRlbCA1NDAgMTIwR0IsIDJ4IFNhbXN1bmcgODUwIFBybyBTU0RzKSBvbiBGcmVlQlNEMTAuMyBS RUxFQVNFLCBGcmVlQlNEIDEwLjMgUkVMRUFTRXA2LCBGcmVlQlNEIDEwLjNSRUxFQVNFcDE5LCBG cmVlQlNEIDEwLVN0YWJsZSwgRnJlZUJTRDExLjAgUkVMRUFTRSwgRnJlZUJTRCAxMS1TdGFibGUg YW5kIG5vdyBGcmVlQlNEMTEuMSBCZXRhIDIuICBUaGlzIGxhdGVzdCB0ZXN0aW5nIEknbSBub3Qg ZG9pbmcgbXVjaCBpbiB0aGUgd2F5IG9mIHdyaXRpbmcgLSBvbmx5IGxvZ2dpbmcgdGhlIG91dHB1 dCBvZiB0aGUgJ2RkJyBjb21tYW5kIGFsb25nIHdpdGggJ3pmcy1zdGF0cyAtYScgYW5kICd1cHRp bWUnIHRvIGdvIGFsb25nIHdpdGggaXQgb25jZSBhbiBob3VyLiAgIFJhbiBmb3IgfjIwaHJzIGJl Zm9yZSBwZXJmb3JtYW5jZSBkcm9wIGtpY2tlZCBpbiB0aG91Z2ggd2h5IGl0IGhhcHBlbnMgaXMg aW5leHBsaWNhYmxlIGFzIHRoaXMgc2VydmVyIGlzbid0IGRvaW5nIGFueXRoaW5nIG90aGVyIHRo YW4gcnVubmluZyB0aGlzIHRlc3QgaG91cmx5Lg0KDQpJIGhhdmUgYSBGcmVlQlNEOS4wIHN5c3Rl bSB1c2luZyAyeCBJbnRlbCA1MjAgMTIwR0IgU1NEcyB0aGF0IGRvZXNuJ3QgZXhoaWJpdCB0aGlz IHBlcmZvcm1hbmNlIGRlZ3JhZGF0aW9uLCBtYWludGFpbmluZyB+NDAwTUIvcyBzcGVlZHMgZXZl biBhZnRlciBtYW55IGRheXMgb2YgdXB0aW1lLiAgVGhpcyBpcyB1c2luZyB0aGUgR0VPTSBFTEkg bGF5ZXIgdG8gcHJvdmlkZSA0ayBzZWN0b3IgZW11bGF0aW9uIGZvciB0aGUgbWlycm9yZWQgenBv b2wgYXMgSSBwcmV2aW91c2x5IGRlc2NyaWJlZC4NCj4NCuKAiz4gSSBkb24ndCByZW1lbWJlciBp ZiB0aGlzIGhhcyBiZWVuIG1lbnRpb25lZCB5ZXQgaW4gZWl0aGVyIG9mIHlvdXIgdGhyZWFkcyBv biB0aGlzLCBidXQgd2hhdCBpcyB0aGUgb3V0cHV0IG9mIHRoaXMgY29tbWFuZCBvbiBhbGwgeW91 ciBwb29ybHkgcGVyZm9ybWluZyBzeXN0ZW1zOg0KPg0KPiBzeXNjdGwg4oCLdmZzLnpmcy50cmlt LmVuYWJsZWQNCj4NCj4gSWYgaXQncyBzZXQgdG8gMSAodGhlIGRlZmF1bHQpLCBzZXQgaXQgdG8g MCBhbmQgcmUtcnVuIHlvdXIgdGVzdHMuDQo+DQo+IFpGUyBUcmltIHN1cHBvcnQgZm9yIFNTRHMg d2FzIGFkZGVkIHRvIDEwLjAsIHNvIGFueSBzeXN0ZW0gcnVubmluZyBGcmVlQlNEIDEwKyB3aWxs IHNob3cgYSBwZXJmb3JtYW5jZSBkcm9wIGFmdGVyIGF3aGlsZSB3aGVuIHRoZSB0cmltIGZ1bmN0 aW9uIGtpY2tzIGluIHRvIGNsZWFyIG91dCBkZWxldGVkL3VudXNlZCBibG9ja3MuICBFc3BlY2lh bGx5IGlmIGl0J3MgPiBhbiBTU0QgdGhhdCBjYW4ndCBydW4gVHJpbSBjb21tYW5kcyBpbiBwYXJh bGxlbC4NCj4NCj4gWW91IGNhbiBsb29rIGF0IHRoZSB2YXJpb3VzIFpGUyB0cmltLXJlbGF0ZWQg c3RhdHMgdG8gc2VlIHdoYXQgaXQncyBkb2luZzoNCj4NCj4gc3lzY3RsIHZmcy56ZnMgfCBncmVw IHRyaW0NCj4NCj4gLS0NCj4gRnJlZGRpZSBDYXNoDQo+IGZqd2Nhc2hAZ21haWwuY29tPG1haWx0 bzpmandjYXNoQGdtYWlsLmNvbT4NCg0KV2hlbiB1c2luZyB0aGUgR0VPTSBFTEkgbGF5ZXIgdG8g cHJvdmlkZSB0aGUgNGsgc2VjdG9yIGVtdWxhdGlvbiBmb3IgdGhlIG1pcnJvcmVkIHpwb29sLCBJ IHRyaWVkIGRpc2FibGluZyBUcmltOyBob3dldmVyLCBpdCBoYWQgbm8gZWZmZWN0LiAgSeKAmXZl IG5vdyBtb2RpZmllZCBteSBGcmVlQlNEIDExLjEgQmV0YSAyIHRlc3Qgc2VydmVyIHRvIHJlbW92 ZSB0aGUgdmZzLnpmcy5hcmNfbWluIGFuZCB2ZnMuemZzLmFyY19tYXggc2V0dGluZ3Mgc28gdGhh dCB0aGUgc3lzdGVtIHVzZXMgdGhlIGRlZmF1bHRzIGFzIHdlbGwgYXMgc2V0IHZmcy56ZnMudHJp bS5lbmFibGVkPeKAnTDigJ0uICBJbml0aWFsIHJlc3VsdHMgYXJlIG1vcmUgaW4gbGluZSB3aXRo IHdoYXQgSSB3YXMgc2VlaW5nIHVzaW5nIEdFT00gRUxJIGxheWVyIHRvIHByb3ZpZGUgNGsgc2Vj dG9yIGVtdWxhdGlvbjoNCg0KdGVzdEBmMTExYmV0YTI6fiAjIGRkIGlmPS90ZXN0ZGIvdGVzdCBv Zj0vZGV2L251bGwgYnM9MW0NCjE2MDAwKzAgcmVjb3JkcyBpbg0KMTYwMDArMCByZWNvcmRzIG91 dA0KMTY3NzcyMTYwMDAgYnl0ZXMgdHJhbnNmZXJyZWQgaW4gMTkuNjMxNDg1IHNlY3MgKDg1NDYw NzU2NyBieXRlcy9zZWMpDQoNClBlcmhhcHMgdGhlIGZvcmNlZCB2ZnMuemZzLmFyY19taW4gYW5k IHZmcy56ZnMuYXJjX21heCBvZiA1ME1CLCB3aGlsZSBmaW5lIGZvciBGcmVlQlNEIDkuMCAod2hp Y2ggbmV2ZXIgc3VwcG9ydGVkIFRyaW0pLCBuZWVkIG1vcmUgbGliZXJhbCBzZXR0aW5ncyBmb3Ig RnJlZUJTRCAxMCBhbmQgbGF0ZXI/ICBJIGhhdmUgYSBudW1iZXIgb2Ygc3lzdGVtcyB3aXRoIG9u bHkgMkdCIHJhbSBydW5uaW5nIFBvc3RncmVTUUwgYWxvbmcgd2l0aCBhIG51bWJlciBvZiBvdGhl ciBkYWVtb25zIHdoaWNoIGlzIHdoeSBJIGNob3NlIHN1Y2ggbG93IEFSQyBzZXR0aW5ncyBpbiB0 aGUgZmlyc3QgcGxhY2UuDQoNClN0aWxsLCBpdCBzZWVtcyBvZGQgdGhhdCB0aGUgc3lzdGVtIHJ1 bnMgZmluZSBmb3Igc2V2ZXJhbCBob3VycyBiZWZvcmUgcGVyZm9ybWFuY2UgZGVncmFkZXMuICBX aXRoIHRoZSBvcmlnaW5hbCBHRU9NIEVMSSBsYXllcmluZyBJIHV0aWxpemVkLCBpbXBsZW1lbnRl ZCBiZWZvcmUgYXNoaWZ0IHdhcyBhdmFpbGFibGUgYW5kIHJldGFpbmVkLCBUcmltIHdhcyBub3Qg YW4gaXNzdWUgYXMgaXQgd2FzIG5ldmVyIHBhc3NlZCB0byB0aGUgZHJpdmVzLg0KDQpUaGlzIHRl c3Qgd2lsbCBwcm9iYWJseSB0YWtlIGEgd2hpbGUgdG8gY29tcGxldGUuDQpUaGlzIG1lc3NhZ2Ug bWF5IGNvbnRhaW4gY29uZmlkZW50aWFsIGFuZCBwcml2aWxlZ2VkIGluZm9ybWF0aW9uLiBJZiBp dCBoYXMgYmVlbiBzZW50IHRvIHlvdSBpbiBlcnJvciwgcGxlYXNlIHJlcGx5IHRvIGFkdmlzZSB0 aGUgc2VuZGVyIG9mIHRoZSBlcnJvciBhbmQgdGhlbiBpbW1lZGlhdGVseSBkZWxldGUgaXQuIElm IHlvdSBhcmUgbm90IHRoZSBpbnRlbmRlZCByZWNpcGllbnQsIGRvIG5vdCByZWFkLCBjb3B5LCBk aXNjbG9zZSBvciBvdGhlcndpc2UgdXNlIHRoaXMgbWVzc2FnZS4gVGhlIHNlbmRlciBkaXNjbGFp bXMgYW55IGxpYWJpbGl0eSBmb3Igc3VjaCB1bmF1dGhvcml6ZWQgdXNlLiBQTEVBU0UgTk9URSB0 aGF0IGFsbCBpbmNvbWluZyBlLW1haWxzIHNlbnQgdG8gV2VhdGhlcmZvcmQgZS1tYWlsIGFjY291 bnRzIHdpbGwgYmUgYXJjaGl2ZWQgYW5kIG1heSBiZSBzY2FubmVkIGJ5IHVzIGFuZC9vciBieSBl eHRlcm5hbCBzZXJ2aWNlIHByb3ZpZGVycyB0byBkZXRlY3QgYW5kIHByZXZlbnQgdGhyZWF0cyB0 byBvdXIgc3lzdGVtcywgaW52ZXN0aWdhdGUgaWxsZWdhbCBvciBpbmFwcHJvcHJpYXRlIGJlaGF2 aW9yLCBhbmQvb3IgZWxpbWluYXRlIHVuc29saWNpdGVkIHByb21vdGlvbmFsIGUtbWFpbHMgKHNw YW0pLiBUaGlzIHByb2Nlc3MgY291bGQgcmVzdWx0IGluIGRlbGV0aW9uIG9mIGEgbGVnaXRpbWF0 ZSBlLW1haWwgYmVmb3JlIGl0IGlzIHJlYWQgYnkgaXRzIGludGVuZGVkIHJlY2lwaWVudCBhdCBv dXIgb3JnYW5pemF0aW9uLiBNb3Jlb3ZlciwgYmFzZWQgb24gdGhlIHNjYW5uaW5nIHJlc3VsdHMs IHRoZSBmdWxsIHRleHQgb2YgZS1tYWlscyBhbmQgYXR0YWNobWVudHMgbWF5IGJlIG1hZGUgYXZh aWxhYmxlIHRvIFdlYXRoZXJmb3JkIHNlY3VyaXR5IGFuZCBvdGhlciBwZXJzb25uZWwgZm9yIHJl dmlldyBhbmQgYXBwcm9wcmlhdGUgYWN0aW9uLiBJZiB5b3UgaGF2ZSBhbnkgY29uY2VybnMgYWJv dXQgdGhpcyBwcm9jZXNzLCBwbGVhc2UgY29udGFjdCB1cyBhdCBkYXRhcHJpdmFjeUB3ZWF0aGVy Zm9yZC5jb20uDQo= From owner-freebsd-fs@freebsd.org Tue Jun 20 19:52:50 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0FC8CDA0EC6 for ; Tue, 20 Jun 2017 19:52:50 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x235.google.com (mail-wr0-x235.google.com [IPv6:2a00:1450:400c:c0c::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 99062713E1 for ; Tue, 20 Jun 2017 19:52:49 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x235.google.com with SMTP id c11so49732887wrc.3 for ; Tue, 20 Jun 2017 12:52:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=q8c5PqP+eTHC/+rERyrMX3TUUkTb63FJD1lQTBimLdk=; b=GVgLmHTjHIhjzWiRKAn+AuXzmS9iyzJswbyblOrOZJ/ID7VLX5ml/9Dde16M0m/O12 briB49yBzZDKsCUYIp5G4j36qK7N/6RLlBjSBshCGRWBae/kCJ3Md3qDHCMC3td9qjAT KADeN3Or/tusUv/NlJYkk5Wm6XOdSD5acf7GgiWJgW0nobcbPb08mv2mzvV8cBX2Y5fs RQcMwpEXp+uQjVoegBWT4G3ssesBLSRSEBlOqvxu9lUHJDXw62qaDElGD1AgFwEiULsL OFXXkDaBxIBNZyn71MZnffi0tZqRZAAYiqUKG7A9E8dpHM4dLbf+sQ3IbGyQsiYi5l6V LI+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=q8c5PqP+eTHC/+rERyrMX3TUUkTb63FJD1lQTBimLdk=; b=Wbs4PFAhCyZcPxHmTY49ttsXdMlBKI8/jKVO48vpAwSmoeVh5qnNi5Z9KDEgqdkbCt ZMy9Sk84dRww9FySFIn6KMuCY8mfUAiBw0SDxwZdGsbPv2r7KklMuT8iMEfXgDKCRJ1l m2BAQClMyuOjjtQDhCGfHf+SuXOJL5jqBAc2Uzgm9AD/rVstZm8dUVzxIZoPHCugeHvt hRBQ1JcU/gkfiK8G53HpYOAxGHFSBn4s85MgRUmJe0hFKd66AYc1M/0HWmi7YgnTByvI XX/KP2pGEGHqPw2sBXL48u++og2TqcFk9xTerwutgJIQGav6pQItKiQlJauAHZA/Wp5R 1fgA== X-Gm-Message-State: AKS2vOwchPEOHBnvBH830qC6HAfjMTQnLUlbntLFEe6Dmp+SD1ooE1YK w8x+txAeky/nURHu9JvI8g== X-Received: by 10.223.144.69 with SMTP id h63mr20259206wrh.187.1497988367249; Tue, 20 Jun 2017 12:52:47 -0700 (PDT) Received: from [10.10.1.111] ([185.97.61.15]) by smtp.gmail.com with ESMTPSA id k86sm13475604wmi.16.2017.06.20.12.52.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 20 Jun 2017 12:52:46 -0700 (PDT) Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs To: "Caza, Aaron" , "freebsd-fs@freebsd.org" References: <431d958c658a408d8bfd4c574a565439@DM2PR58MB013.032d.mgd.msft.net> <990886ae-b7c1-8630-1cef-f6678f0b5b63@multiplay.co.uk> From: Steven Hartland Message-ID: Date: Tue, 20 Jun 2017 20:52:45 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 19:52:50 -0000 On 20/06/2017 17:58, Caza, Aaron wrote: > >> Can you show the output from gstat -pd during this DD please. > dT: 1.001s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > 0 4318 4318 34865 0.0 0 0 0.0 0 0 0.0 14.2| ada0 > 0 4402 4402 35213 0.0 0 0 0.0 0 0 0.0 14.4| ada1 > > dT: 1.002s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > 1 4249 4249 34136 0.0 0 0 0.0 0 0 0.0 14.1| ada0 > 0 4393 4393 35287 0.0 0 0 0.0 0 0 0.0 14.5| ada1 You %busy is very low, so sounds like the bottleneck isn't in raw disk performance but somewhere else. Might be interesting to see if anything stands out in top -Sz and then press h for threads. You also mention ada, could you share: sysctl kern.cam And: sysctl vfs.zfs Finally when its performing well can you repeat the gstat -pd > Every now and again, I was seeing d/s hit, which I understand to be TRIM operations - it would briefly show 16 then go back to 0. > > test@f111beta2:~ # dd if=/testdb/test of=/dev/null bs=1m > 16000+0 records in > 16000+0 records out > 16777216000 bytes transferred in 43.447343 secs (386150561 bytes/sec) > test@f111beta2:~ # uptime > 9:54AM up 19:38, 2 users, load averages: 2.92, 1.01, 0.44 > root@f111beta2:~ # dd if=/testdb/test of=/dev/null bs=1m > 16000+0 records in > 16000+0 records out > 16777216000 bytes transferred in 236.097011 secs (71060688 bytes/sec) > test@f111beta2:~ # uptime > 10:36AM up 20:20, 2 users, load averages: 0.90, 0.62, 0.36 > > As can be seen in the above 'dd' test results, I'm back to seeing the original issue I reported on freebsd-hackers - performance degrading after < 24 hours of uptime going from ~386MB/sec to ~71MB/sec inexpicably - this server isn't doing anything other than running this test hourly. > > Please note in the gstat -pd output above, this was after the performance degradation hit. Prior to this, I was seeing %busy of ~60%. In this particular instance, the performance degradation hit ~20hrs into the test but I've see it hit as soon as ~5hrs. > > Previously, Allan Jude had advised zfs.vfs.trim.enabled=0 to see if this changed the behavior. I did this; however, it had no impact - but that was when I was using the GEOM ELI 4k sector emulation and not the ashift 4k sector emulation. The GEOM ELI 4k sector emulation does not appear to work with TRIM operations as gstat -d in that case always stayed at 0 ops/s. I can try disabling trim, but did not want to reboot the server to restart the test in case there was some additional info worth capturing. > > I have captured an hourly log that can be provided containing the initial dmsg, zpool status, zfs list, zfs get all along with an hourly capture of the results of running the above 'dd' test with associated zfs-stats -a and sysctl -a output though it's currently 2.8MB hence too large to post to this list. > > Also, there seems to be a problem with my freebsd-fs subscription as I'm not getting e-mail notifications despite having submitted a subscription request so apologies for my slow responses. > From owner-freebsd-fs@freebsd.org Tue Jun 20 20:26:12 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37EB7DA17CC for ; Tue, 20 Jun 2017 20:26:12 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on0076.outbound.protection.outlook.com [104.47.36.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B164D7281E for ; Tue, 20 Jun 2017 20:26:10 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weatherford.onmicrosoft.com; s=selector1-ca-weatherford-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=yrcYeL+dFL0DhOy1gDOkjoOtcqZTCt0l4mb+arkWmec=; b=V3ce5KdRqeSGrgYTARkGtMvLpR+Zjky3ZBHQrbZiulGmIkrj28WKBCpsP3nf89W8yshIu92DjJSVQCZ7S9piZcJDw69Jcsuahnov5IHH7Npo4rDQifwlOAWQ5bRs3rS5IrOaQ9ZoAkYVPCzbs/eWM6PAqPa8oQI/K9CNNwm3/oU= Received: from DM5PR03CA0018.namprd03.prod.outlook.com (10.175.104.28) by MWHPR03MB3165.namprd03.prod.outlook.com (10.174.174.34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 20:26:09 +0000 Received: from BY2FFO11FD037.protection.gbl (2a01:111:f400:7c0c::186) by DM5PR03CA0018.outlook.office365.com (2603:10b6:3:118::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14 via Frontend Transport; Tue, 20 Jun 2017 20:26:08 +0000 Authentication-Results: spf=pass (sender IP is 23.103.226.20) smtp.mailfrom=ca.weatherford.com; freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=bestguesspass action=none header.from=ca.weatherford.com; Received-SPF: Pass (protection.outlook.com: domain of ca.weatherford.com designates 23.103.226.20 as permitted sender) receiver=protection.outlook.com; client-ip=23.103.226.20; helo=032-smtp-out.weatherford.com; Received: from 032-smtp-out.weatherford.com (23.103.226.20) by BY2FFO11FD037.mail.protection.outlook.com (10.1.14.222) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14 via Frontend Transport; Tue, 20 Jun 2017 20:26:08 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net (141.251.110.81) by DM2PR58MB015.032d.mgd.msft.net (141.251.110.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 20:26:07 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) by DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) with mapi id 15.01.1178.018; Tue, 20 Jun 2017 20:26:07 +0000 From: "Caza, Aaron" To: "freebsd-fs@freebsd.org" , Steven Hartland Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Topic: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Index: AdLqAtGh3ZrvWZ+7QKOnTYE8YOBOpQ== Date: Tue, 20 Jun 2017 20:26:06 +0000 Message-ID: Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [141.251.136.68] x-ms-publictraffictype: Email X-MS-TrafficTypeDiagnostic: DM2PR58MB015:|MWHPR03MB3165: X-MS-Office365-Filtering-Correlation-Id: 7313e09c-4377-4a74-5ffd-08d4b81a95a8 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OrganizationHeadersPreserved: DM2PR58MB015.032d.mgd.msft.net X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:23.103.226.20; IPV:NLI; CTRY:; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(39410400002)(39400400002)(39850400002)(39450400003)(39840400002)(39860400002)(2980300002)(438002)(189002)(24454002)(51234002)(377454003)(199003)(9170700003)(81166006)(108616004)(50986999)(8676002)(2906002)(7696004)(54356999)(97756001)(24736003)(53936002)(2900100001)(50466002)(72206003)(38730400002)(5660300001)(8746002)(305945005)(106466001)(33646002)(8936002)(46406003)(47776003)(229853002)(55016002)(2501003)(53546009)(5890100001)(86146001)(66066001)(189998001)(23726003)(356003)(102836003)(3846002)(6116002)(478600001)(86362001)(42882006)(7736002)(9686003); DIR:OUT; SFP:1101; SCL:1; SRVR:MWHPR03MB3165; H:032-smtp-out.weatherford.com; FPR:; SPF:Pass; MLV:sfv; MX:1; A:0; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD037; 1:pl6f8gwIsE4avyVGNFMIMobEZRGgHfJ9mgiHr/Vq4+ikeAxvJ7oiPydLn0wxmO5hQWDBuksiHEP7AtDOtr0X3ngX+6w4C4IyyhhlyOl/Z6KJ3Z/KUR4CrKW1rzRRKgbQGNHUBaJARIFm7c903E95mPjkBehbn9Mxsu7YCPxU7fu1iUUhUYshjUbh/fYLB36VtXctQ3gAlh3jgtkE3uK6ENR3x8hYi81UvXLNs3hEwhttq2k9s4pMW7I32JjHM0MTvOnHrNlARkahhQNI6htzXboBrfjpFwY2EkawRmHWnacVXOdfJufBU5RBUGcXHmsPxiBfHu5OW1r8ev8X4wfpc10zC2fITmBZws5xuprxpdK+Z6kUqbYu2cw94rIN9Nh7d02UwFTX5xwZUtsa4ahdP7FeqtqMW+LKEI2vxitglJ/w9pbLOJXmDPGLsHwLLr+IGxkn+gKiF9Kmk69u1ikIRamL3yTTLSGGg9Vs68xSM+J9NCL0MFsMT4uCH5IV/P+EfIfUUIzVfJSKDX05bLQORA== X-CrossPremisesHeadersPromoted: BY2FFO11FD037.protection.gbl X-CrossPremisesHeadersFiltered: BY2FFO11FD037.protection.gbl X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(8251501002)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:MWHPR03MB3165; X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3165; 3:uHbeCOPHRIz3wo8XYT389LEV5GNeqDI5OFZWqi0eJIdGZrETje3rn1tPoJiD4c6g5/4ESOd9bVAoOBPg0v+4wGNI5uOc8eY7wWt916wOZhpBZaUqJd5+qm0CZyBPaYRME825x8BBwOyxM7EtbtawtUxtRq/L1qpWKdMXNHuTA+9SOxEv7xeiIuH/5Vzh/R9mpr6O2m2LiBmvgXG0m1Lad3UXQ1BmE/Qr1bSNadk9f2nNeQBIlY32t+uh99k4v/tCnBpyQawrIekAaH7Esul6gyiQISJtvoJIoNYkRQzPpiSBxDDB64bTu105NdYMoZXC2W/GvOy+0UQ49uExYBl0q0WJz0FL12UPJeHkyT6b8JS5c+C1tJQI2ouTA7ZPQwYpB+OITMIc05I0pOWHY5lq7GCYxg6aJuHSUBODK5qVq8Jq1Df1Wg0KM+8x6fMDXBMGO01e8xfzdRUOeRpXGIv/45dQrkIPRvftaM0BSG8KVc8kDvpp0GMVO0FkCRg3I8UEFHdkU+CowYhdHlRhlKd2CA== X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3165; 25:oMUvizg3+tAKjNwoIuLptovzWLkyzPMuUdK9Xg7eOdy4xcUfCTFAcuGal6hIHqAqBLgMZ6CWbuWsrih+yFQa+aIJ0Q76kYp4oVfweFxotIkdZzN0GMFD7RbmaOUVtaF3FnVbSmgQ01z/Kpakv+A81VgFfnoP14Ep8uWaMaoiv9rOVlgJ87UvmLbZU1rnRcXZ3JBCHTY5ySs3yEcNgUXuZprcFs+AmaNfsNC873mf2lbApt6UzbRLrMVTmLOc8tMf+5onZfLweLuMqwHGUc6YbyU8600iwoSBrVhT5RPrGQGoJgW8kJzG1UqZ2bS2aXb9DcmAp7QoN6fwmC+eJ+kdMbymoQOj8tSgzaYOzxDf+fHvDpcPP8pU5njgvrqZD2ABtN/bhAjgILiWE45R5Rmbzov+85tTNjjZKrF0QNHkoCfTd99PA2c+3ZyDBlxhrT9t28OdZyNbx54Wb8OOvBbohSFfoFltHtcdnWgbVTf6sCQ=; 31:0yDJ+g/Fi7a2wmZdmBHS05EKE5GaZYUpPAfwJ2ATu5VQDPlJvuNTjs6cGGZZHVsh+06ALfLFCqMMdcPgN32KCKNoZKo7jBFswNwz8sKtoAZvyzAp+H1t4wnXXfTZsmbo8gcEp2Akgiy2NtBzxzlRnI8t/bKhJiFB9IEm8ISxeRjS5pG5ZTPFV8rvgbeCh0vKEjRa8Q83MWMv7ni7ccSlFbGGJbe4UqYxlO0JdWQJ7X/8QABFSh/34QMZMWFgRWkj1Dybl7cMtIJLDrKym9VKkg== X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3165; 20:zLH8hgvtdiN2Te/NoOvvtf0LvHz7P1wxY4Qo5i+hU1mVrlX36H6bjAXh09TDPNtZBASephDv9/7ciYwmkBVF44w+kJJnMSeqb1h0ZMZ/QNMtIGXMeDGx4taJpjbUDXQ20jx6eyQKUiA/zDCbDfK5aeXK2xf4Ku1j5g3KSf2+pV5+lVcpwgxyCz2BZhDHIgK89ejCbpshhGqqimCA96yIlJtc/DKJch57grvwzpWmt8lQcx/VDPN/Dc0Nj3R7cJSb/c8waHgnFmdFa9x494YDnxMrmSeMGGeyGN5ed9uZxWjWL0FInL6yU9Hgdgb5UanqyF1cLE+lULFRonR89zzvNgEB/x39dkEE2TqHjq1ZaZDkPuzt8tuPtYX80slCXOIMimedheooAjtPVr8+NF5/CBzta/y52o8Hmqv1Q/dyI4dY9XrWymBZrPsPGQSYTQp0NSlWHQ1EbMglD3sgLtQeC42HeIf2H4m8tfcM8Pw3utA5+44ANWM4GwLOh75QDIOS X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(192374486261705); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(11241501159)(6040450)(601004)(2401047)(13018025)(8121501046)(13016025)(5005006)(10201501046)(100000703101)(100105400095)(93006095)(93004095)(3002001)(6055026)(6041248)(20161123562025)(20161123560025)(20161123555025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:MWHPR03MB3165; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:MWHPR03MB3165; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3165; 4:MBvvX7pm68dIhPX08CwXXaoNMF9y58lMF5eCSpvDWj?= =?us-ascii?Q?hSRv+gjbBdEuWrEDfKfLVnEQ+kta9psFsvTO/z2QOCZ/P/1JMFCXEARErDZa?= =?us-ascii?Q?M73LgW88oMvlIX74fOb2T0y1njXJHsXf5KmxOSQClNjkEK9GfmbgWfNvdo55?= =?us-ascii?Q?6zPZ/kqqwsRIayiEzEqwzXOFu2BMquU1V3Jgqqv5/bbNWIqIDTgHXCm8zbhB?= =?us-ascii?Q?naNnNPRM73kNB6fqkF7gK3XfAFKUtX7JpsHhjSdxQAQBeHO3P82/iJVp5cWV?= =?us-ascii?Q?IPeGbAOazsI2DLHAFyzm0YDj5US5E7OC62ZBRxe+fzA+iRluYWRzrnsWD44A?= =?us-ascii?Q?FQQh+v3QA73IP16jqSKqdUn403qrDvLTTmDMgXd92HSVW+OXOier8s372x3q?= =?us-ascii?Q?qAj/8gOYj82eEm7SMzAEQk7C1yrH9e2NgEvBc5v2Yq0hKxJznhIIE8AxIkXy?= =?us-ascii?Q?UEsWM8YNxKMlBD2FB+mxfcTW5S8mRz8pPbXUsGaLa5Fvwh8XEsHC1ZAggjhZ?= =?us-ascii?Q?lCxyeO+9avoZed5yfqv0rx0LlHFYxE64444xA+j7KfnAzPw6/g0F28JJROLL?= =?us-ascii?Q?U+Q3ZDp6NNl3y85tgZaHbLa0ni4+gTbcoo0f5uRVwNRJ8QCVkLMdm+4NiMfe?= =?us-ascii?Q?4R/XBuD8JL9SrJsAdU3Scj2Fs/wbE2J1RCNsLbs8lUpbu9t962cfGhHWEnXR?= =?us-ascii?Q?vors958TC+DZmkaURuXYZPoRUz48q1VsN5f3H+kmkO9RiwEXYi6LErx9Wd1l?= =?us-ascii?Q?q6w8YTv4mkQVrGc+DmaI7lYopMMyc95AdvuxsTpuHq3D8ZLsuC6VLg3YSWu1?= =?us-ascii?Q?rrvB3s7RYQMjDH2Jj2pILqljILQgCdYiONHH5uO6Goci2FLUQKBwgZiUK3/p?= =?us-ascii?Q?9nxUiV8zw6td4l3VKzKoX+n1QrVeNqmH9ye0b+FAyWZyE7tqpnBqKxih+AhN?= =?us-ascii?Q?TJ88y8ViMtNntD4XjVqaAMFEfKvbysHWXQ8jTrzNdL8Uzp7UJ/X/dx0KYJu2?= =?us-ascii?Q?ci3Zmjc9BSC9JGOrkLvx6SJlHWbaK6lvXQABQPNZp2gcd4wdevUqSfSlLYrV?= =?us-ascii?Q?6PokTOlbbvmtAVJTeExXLfY4e/wTTmbNfyQAaynE/tgZOXKHx28Id+V6VzgJ?= =?us-ascii?Q?qf8dep8D+88ayHwC1ZIGDm5773bjcInfg9Vb5Bxn6dOsMl7E7kwagVPdxSwP?= =?us-ascii?Q?KgOPjvsNEJwvTQ/gsb59lcVBIar49AQQnFlLoiGtszQUiKVUfr4ypUgKhIzm?= =?us-ascii?Q?Fbe34zlzflKkt15sUsfIFOLkbnqPvzfSRj4EZJ?= X-Forefront-PRVS: 03449D5DD1 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3165; 23:6qAXPCGhIB/kxb6xLfwlFZLzmWUM2RL6A06PIpySa?= =?us-ascii?Q?jF6EAyZqIp2sUU2GGCATRFKN7jDMSm47yp+fSsWBTdFIPgWd1QEDHAKs8/kr?= =?us-ascii?Q?jNPoy/jaP4Do0Bv4DRC6zc9kYj0BOBGOsmOEFw6UDkZWs5jthDIFtlLIhYmU?= =?us-ascii?Q?ObXauPse2GumxfArOHrHaTKO29nshN3nMNAeSC1I+zNuC+mjjSjaBB3JKCgU?= =?us-ascii?Q?WBdVAX4TYbqtd3S6rbDh+1LgQGzr5o+MXPWSsio3S12B7CCyVnKsZtPBmm46?= =?us-ascii?Q?TyK04NEF+t1vNPBkumSSR5cyA4NXFuc+cvYfsRnDaGeCaPVQZuQx/kYmmkW+?= =?us-ascii?Q?ZSwH70m4ejimrocysTGkTSrOIGia7AH1pLjtPgfQL6G2W2WTxe1TScC9+In4?= =?us-ascii?Q?PPaqgkZakUfrKWcvW2TdWHl2f9JeFe7p2vNOlE6mRtmGPiH7LXnh6CGBKnGp?= =?us-ascii?Q?VR4q/j3l956HrfW3eT0CgmOnBjKrv09crVnEQsnGITtMN3m5Rvy4lhtxsPkJ?= =?us-ascii?Q?Me1gZM07E7Z0JyP8bhutAIGDTTS4bKAaE0vtwKAakhRoxgemy7daiNRs9RFu?= =?us-ascii?Q?mq0VV9eveiSOOBvR11tS+TUW0IHgjPavrXErDLzBN/UTzE3JWQMnrrZkOGEC?= =?us-ascii?Q?xNbhqF4Vv9yhSEknznxpR8us/zStDBOzEtNIQQM9Eo4aLZaWXMzXtCOonzay?= =?us-ascii?Q?HwOwzjKZxq/U4F3JwvyVh+OJRtEOFTVlSQQAZfnE43Xbij/ApElf44OJaHYF?= =?us-ascii?Q?YgpbhQN87vAXF2CBV47IXt8izH0thiiy19rvwoQj0jdgHyxNX/0wvTxDfEph?= =?us-ascii?Q?xsEpC3D8TFf0WzIfPnL24ILqW1IkT8G374XevD8+NVNOtRh3zivgedEUU+3G?= =?us-ascii?Q?XZqty9IvCWazQHpVOGBJ0RzChPL9Pq7Z9FTrFIywsKd03/CT1XnyoeoL6zoR?= =?us-ascii?Q?z5UQk84BbDf5EP/eXgGZzN+nk+P5h5HqdGM54Sm72Bzdj4iUY7emEq/4mQOD?= =?us-ascii?Q?3m3egs/RUyuPheEaoF2br1vr+3w/oFaOu39No1XLdsnetYn9NdEbKE5/LmwC?= =?us-ascii?Q?RBPhxP81vxIlZy1welmLUAYC5ZTXL7J0s/Jrvai0aEeNIhMi24pxj/5pg9Qw?= =?us-ascii?Q?SgJ+SM3ShCXb+beOXzwhQZfXKXTqH62oVlgB9qocdFSggOUM0mZsmhjCeL1y?= =?us-ascii?Q?vfpS8XC2d7Zp2YW2Cgs1EbrWfNKnLXtTBfcj0bf/WFjhCUbkxOoY7LaFq1Td?= =?us-ascii?Q?KigKFPFkq4/paF8WMtsiIih2tstaEPp/Q5Cesouzl0o7VD1/eA3iMmpbaF1h?= =?us-ascii?Q?/jSpm4dUl5Rj0R4ZN3EC/joCjjPpki8MdZsnXGg1rOW?= X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3165; 6:bTLYAKOl4PmUmgvMvQl7KMMDs31gROgEHwAaDHr+E8?= =?us-ascii?Q?DE+DoaYx2wL3rYHQm/yaN+cfkmhjfxPQqh5N3NqfeNCrGuDx/h1rtdRGz5Fx?= =?us-ascii?Q?hxSfynqbbbLdSox7EYOIzguWo067lAjFZdXhruCnp7VbHBTaeg1ibbIDxGKu?= =?us-ascii?Q?VEi0eQ2KnpNLl7TpF5r43EXO4RltjWWAonHTkHJwta+KYO6LRJxFjol76Law?= =?us-ascii?Q?a2OYXqsh3okog/tD68d4YTKA0CfAutmYd9cvqpdylFeq40xIhJPLUTG8Bz4H?= =?us-ascii?Q?9O7PBjx3g4cKXxS49lYSI7imuRJ0CQYv3ajdRweizip7voEOe/4doctn7AaU?= =?us-ascii?Q?HvgxUGzhbVGhtG3CsFaeK2DllW2K0yPwJCsN82YnOzFL5AAiNLbbcyFpJm2A?= =?us-ascii?Q?vq0DUvHuBWx3rsl7ukF0O198UKHzhS07k+vdNPvwP3LrC+th+fLFAXZAyXrC?= =?us-ascii?Q?N7YZyvo+x36zZ6DBkjGUXLm71nA+6fWm2F/jiEml8jtkAI17a0zeIpX+/bWb?= =?us-ascii?Q?9nDABfY9mOMFH+8xduxft9UJ0BSRczLmS4jGiSouh4P1H7fn4U7wMi4WrYyM?= =?us-ascii?Q?bzmqM6SSh/iTAGjIQQXj/UXXHFyAtnUNmq7qblDptM+W/2gZA8B2PGC1f4Z/?= =?us-ascii?Q?mL7OsidFOywYjON8lWXQ4BORTQSOJa0K+4qsXvHUWz2tUp5WcUfPHsEc4rcc?= =?us-ascii?Q?H/K2dJGMRNyxZJ9g6Me9o8ueryP40YdPlnCbFtD19ifgD0mlKKOOHmYLft2G?= =?us-ascii?Q?oOP/qzNS02wTfXmEiWcZ5x0o1oew85KYfWgz5FvcTzyif2MnF0Rfp2T/hx53?= =?us-ascii?Q?5CFeSifvFq+oFQ+S0sgrGfBM9Fy917RsQrojzdvWr826yEb2l0KVflGQmv1h?= =?us-ascii?Q?YlPdQT8k4XQ3MHydBts8Mh0EbLl6GUdBGjyv/HPkTfaMLixJnUK70H416dA1?= =?us-ascii?Q?yrqgEihAz28G4RRU23hMt9lz1iPjC3XZ5dJggr0lANkdz8zXhAAug7WcgbG7?= =?us-ascii?Q?Gz0M1iKHz8GkjeEimjlorJ?= X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3165; 5:DJrR4hqLIH6czwOrEAUpfVS8fongVetWy2MD0EdvPwqASCis5u6gsw1lLfxsAwTPtkBrH1C1KAzDf7MtHuE8BjBBBe0R1+DUX1i8wPWD5lzjprl7c1Vwng6Vjlmc74DXd4V4r+wsxAzAfPDdT/Z9/3LMsGfanh1+PFBWZIosdRPMPfD1ir1B3jEjxooj1vDaed5CHJGoSGru3NwNSKhdrazIbIyiloTonF3jnk+9FPIXzp8aVzEGqS89g7RSJwzv4uqw1kdbQnsMAuO+pgMovei9w90zbGxGjKbDZpDYyQfG1lz8xwZHr+EVf3IZ7PwEy+xuPmXL0a8T7P/M1YsZGpLJSrnzTI2WDrQgwf4l7KutyIZ2QRxtV+OvZPY6jr0MuNjgBVG4oS6QkbIactB/ASq3AKOECjexq3t4UKrC9Po25aZ9DqLmeLfBlH6YObJzJeWBYvjKGg7wGDLxYTnVnLMveu+YtXwwyZTck1V9ftHapZhe4GG4mMMMrukaky6h; 24:1jj6O9d/6WneSpKQlvGYFmAcwirLo6Rl7Ul0CwZfnhhvsHdGb8RHKqPei7Gt8KMH8muYVmWaODkxWSQQNzQh4vi0CTbks78J0JibehOVXKc= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3165; 7:JK3W48yA7q74PxGctbGd5j59KKeZbrqoQhr1+UaroOFjAIcUArDvxNOVbSfbul6RWjJlx/B+FAUiNM7Thu+eMRJWwnRK0MTDc/FdGZKGtNXHfkDlCcbCFRWSkpOys9m/KFzZrrCAkCL6xf6P6cZ/ODBcHV8SMC422BWN/8QKKXVw+He1+Xa/OSxv4YZvQs/km9zgtMANUZFktKMMNFfENwz/rSIVFsbfecNqFF+/d+S+QDoueQ5gqXYdlY4I+uEMouR10VRgj1MEBfKyHqDqvFrILTFfFUpGWNTrP/cewwujQWu96frkOIe3LylC1TA+iItFxkG/2QGNumHoL+W4I9CK25CSArG+Le0cZeHlOkWUQNlOywPYaqxw4J9PycPcdwLIea3jLUSLGoM8vkli+otlpVPcR07g8URHZN16sXGnjNyLi+ggDGMU532b67xLlXR5ligl+GJnVR75MCnTMR/mbrwGI4bZVwrLwiNotJ2/pDx8C7PKlijwTXKew4madt4oqALzAMmYABwuHdijZLOEWkbKkEEH2PWLFDgbRHFHyENqjBmFOJWZzN8hdxgkm61ZPnx7VZfC34rwMEaj2QCpR5QkdddawQ13vFfQZf+ZOQkxw9j+OadAHekXDrXHSt8As+k/2dk5Hu4AUtAYjwpCQ/mJERTqkp6LbocQWibtbaS1jl18OSfGxPGxg+a9jGfHyAblOD3F6Owi+YPsw9j9sEvA29hPD9vPQr8znFid5DWZINpa9voOU8qfceV+dhp99hx4BtsEx2lrETCACRX98mGMkHE7OEckcCz3UBc= X-OriginatorOrg: ca.weatherford.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jun 2017 20:26:08.4802 (UTC) X-MS-Exchange-CrossTenant-Id: dd63fb60-07f6-4d96-8d40-ebeca61a524e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=dd63fb60-07f6-4d96-8d40-ebeca61a524e; Ip=[23.103.226.20]; Helo=[032-smtp-out.weatherford.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR03MB3165 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 20:26:12 -0000 > -----Original Message----- > From: Steven Hartland [mailto:killing@multiplay.co.uk] > Sent: Tuesday, June 20, 2017 1:53 PM > To: Caza, Aaron; freebsd-fs@freebsd.org > Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs > > > > On 20/06/2017 17:58, Caza, Aaron wrote: > > > >> Can you show the output from gstat -pd during this DD please. > > dT: 1.001s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name > > 0 4318 4318 34865 0.0 0 0 0.0 0 0 = 0.0 14.2| ada0 > > 0 4402 4402 35213 0.0 0 0 0.0 0 0 = 0.0 14.4| ada1 > > > > dT: 1.002s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name > > 1 4249 4249 34136 0.0 0 0 0.0 0 0 = 0.0 14.1| ada0 > > 0 4393 4393 35287 0.0 0 0 0.0 0 0 = 0.0 14.5| ada1 > You %busy is very low, so sounds like the bottleneck isn't in raw disk pe= rformance but somewhere else. > > Might be interesting to see if anything stands out in top -Sz and then pr= ess h for threads. > I rebooted the system to disable Trim so currently not degraded. > You also mention ada, could you share: > sysctl kern.cam > kern.cam.enc.emulate_array_devices: 1 kern.cam.sa.allow_io_split: 0 kern.cam.da.default_softtimeout: 0 kern.cam.da.send_ordered: 1 kern.cam.da.default_timeout: 60 kern.cam.da.retry_count: 4 kern.cam.da.poll_period: 3 kern.cam.ada.1.sort_io_queue: 0 kern.cam.ada.1.max_seq_zones: 0 kern.cam.ada.1.optimal_nonseq_zones: 0 kern.cam.ada.1.optimal_seq_zones: 0 kern.cam.ada.1.zone_support: None kern.cam.ada.1.zone_mode: Not Zoned kern.cam.ada.1.rotating: 0 kern.cam.ada.1.unmapped_io: 1 kern.cam.ada.1.write_cache: -1 kern.cam.ada.1.read_ahead: -1 kern.cam.ada.1.delete_method: DSM_TRIM kern.cam.ada.0.sort_io_queue: 0 kern.cam.ada.0.max_seq_zones: 0 kern.cam.ada.0.optimal_nonseq_zones: 0 kern.cam.ada.0.optimal_seq_zones: 0 kern.cam.ada.0.zone_support: None kern.cam.ada.0.zone_mode: Not Zoned kern.cam.ada.0.rotating: 0 kern.cam.ada.0.unmapped_io: 1 kern.cam.ada.0.write_cache: -1 kern.cam.ada.0.read_ahead: -1 kern.cam.ada.0.delete_method: DSM_TRIM kern.cam.ada.write_cache: 1 kern.cam.ada.read_ahead: 1 kern.cam.ada.spindown_suspend: 1 kern.cam.ada.spindown_shutdown: 1 kern.cam.ada.send_ordered: 1 kern.cam.ada.default_timeout: 30 kern.cam.ada.retry_count: 4 kern.cam.cd.timeout: 30000 kern.cam.cd.retry_count: 4 kern.cam.cd.poll_period: 3 kern.cam.scsi_delay: 5000 kern.cam.cam_srch_hi: 0 kern.cam.pmp.hide_special: 1 kern.cam.pmp.default_timeout: 30 kern.cam.pmp.retry_count: 1 kern.cam.debug_delay: 0 kern.cam.dflags: 0 kern.cam.num_doneqs: 2 kern.cam.xpt_generation: 13 kern.cam.boot_delay: 0 kern.cam.sort_io_queues: 1 > And: > sysctl vfs.zfs > vfs.zfs.trim.max_interval: 1 vfs.zfs.trim.timeout: 30 vfs.zfs.trim.txg_delay: 32 vfs.zfs.trim.enabled: 0 vfs.zfs.vol.immediate_write_sz: 32768 vfs.zfs.vol.unmap_enabled: 1 vfs.zfs.vol.recursive: 0 vfs.zfs.vol.mode: 1 vfs.zfs.version.zpl: 5 vfs.zfs.version.spa: 5000 vfs.zfs.version.acl: 1 vfs.zfs.version.ioctl: 7 vfs.zfs.debug: 0 vfs.zfs.super_owner: 0 vfs.zfs.immediate_write_sz: 32768 vfs.zfs.sync_pass_rewrite: 2 vfs.zfs.sync_pass_dont_compress: 5 vfs.zfs.sync_pass_deferred_free: 2 vfs.zfs.zio.dva_throttle_enabled: 1 vfs.zfs.zio.exclude_metadata: 0 vfs.zfs.zio.use_uma: 1 vfs.zfs.zil_slog_limit: 786432 vfs.zfs.cache_flush_disable: 0 vfs.zfs.zil_replay_disable: 0 vfs.zfs.min_auto_ashift: 12 vfs.zfs.max_auto_ashift: 13 vfs.zfs.vdev.trim_max_pending: 10000 vfs.zfs.vdev.bio_delete_disable: 0 vfs.zfs.vdev.bio_flush_disable: 0 vfs.zfs.vdev.queue_depth_pct: 1000 vfs.zfs.vdev.write_gap_limit: 4096 vfs.zfs.vdev.read_gap_limit: 32768 vfs.zfs.vdev.aggregation_limit: 131072 vfs.zfs.vdev.trim_max_active: 64 vfs.zfs.vdev.trim_min_active: 1 vfs.zfs.vdev.scrub_max_active: 2 vfs.zfs.vdev.scrub_min_active: 1 vfs.zfs.vdev.async_write_max_active: 10 vfs.zfs.vdev.async_write_min_active: 1 vfs.zfs.vdev.async_read_max_active: 3 vfs.zfs.vdev.async_read_min_active: 1 vfs.zfs.vdev.sync_write_max_active: 10 vfs.zfs.vdev.sync_write_min_active: 10 vfs.zfs.vdev.sync_read_max_active: 10 vfs.zfs.vdev.sync_read_min_active: 10 vfs.zfs.vdev.max_active: 1000 vfs.zfs.vdev.async_write_active_max_dirty_percent: 60 vfs.zfs.vdev.async_write_active_min_dirty_percent: 30 vfs.zfs.vdev.mirror.non_rotating_seek_inc: 1 vfs.zfs.vdev.mirror.non_rotating_inc: 0 vfs.zfs.vdev.mirror.rotating_seek_offset: 1048576 vfs.zfs.vdev.mirror.rotating_seek_inc: 5 vfs.zfs.vdev.mirror.rotating_inc: 0 vfs.zfs.vdev.trim_on_init: 1 vfs.zfs.vdev.cache.bshift: 16 vfs.zfs.vdev.cache.size: 0 vfs.zfs.vdev.cache.max: 16384 vfs.zfs.vdev.metaslabs_per_vdev: 200 vfs.zfs.txg.timeout: 5 vfs.zfs.space_map_blksz: 4096 vfs.zfs.spa_min_slop: 134217728 vfs.zfs.spa_slop_shift: 5 vfs.zfs.spa_asize_inflation: 24 vfs.zfs.deadman_enabled: 1 vfs.zfs.deadman_checktime_ms: 5000 vfs.zfs.deadman_synctime_ms: 1000000 vfs.zfs.debug_flags: 0 vfs.zfs.debugflags: 0 vfs.zfs.recover: 0 vfs.zfs.spa_load_verify_data: 1 vfs.zfs.spa_load_verify_metadata: 1 vfs.zfs.spa_load_verify_maxinflight: 10000 vfs.zfs.ccw_retry_interval: 300 vfs.zfs.check_hostid: 1 vfs.zfs.mg_fragmentation_threshold: 85 vfs.zfs.mg_noalloc_threshold: 0 vfs.zfs.condense_pct: 200 vfs.zfs.metaslab.bias_enabled: 1 vfs.zfs.metaslab.lba_weighting_enabled: 1 vfs.zfs.metaslab.fragmentation_factor_enabled: 1 vfs.zfs.metaslab.preload_enabled: 1 vfs.zfs.metaslab.preload_limit: 3 vfs.zfs.metaslab.unload_delay: 8 vfs.zfs.metaslab.load_pct: 50 vfs.zfs.metaslab.min_alloc_size: 33554432 vfs.zfs.metaslab.df_free_pct: 4 vfs.zfs.metaslab.df_alloc_threshold: 131072 vfs.zfs.metaslab.debug_unload: 0 vfs.zfs.metaslab.debug_load: 0 vfs.zfs.metaslab.fragmentation_threshold: 70 vfs.zfs.metaslab.gang_bang: 16777217 vfs.zfs.free_bpobj_enabled: 1 vfs.zfs.free_max_blocks: 18446744073709551615 vfs.zfs.no_scrub_prefetch: 0 vfs.zfs.no_scrub_io: 0 vfs.zfs.resilver_min_time_ms: 3000 vfs.zfs.free_min_time_ms: 1000 vfs.zfs.scan_min_time_ms: 1000 vfs.zfs.scan_idle: 50 vfs.zfs.scrub_delay: 4 vfs.zfs.resilver_delay: 2 vfs.zfs.top_maxinflight: 32 vfs.zfs.zfetch.array_rd_sz: 1048576 vfs.zfs.zfetch.max_idistance: 67108864 vfs.zfs.zfetch.max_distance: 8388608 vfs.zfs.zfetch.min_sec_reap: 2 vfs.zfs.zfetch.max_streams: 8 vfs.zfs.prefetch_disable: 0 vfs.zfs.delay_scale: 500000 vfs.zfs.delay_min_dirty_percent: 60 vfs.zfs.dirty_data_sync: 67108864 vfs.zfs.dirty_data_max_percent: 10 vfs.zfs.dirty_data_max_max: 4294967296 vfs.zfs.dirty_data_max: 853010841 vfs.zfs.max_recordsize: 1048576 vfs.zfs.send_holes_without_birth_time: 1 vfs.zfs.mdcomp_disable: 0 vfs.zfs.nopwrite_enabled: 1 vfs.zfs.dedup.prefetch: 1 vfs.zfs.l2c_only_size: 0 vfs.zfs.mfu_ghost_data_esize: 393216 vfs.zfs.mfu_ghost_metadata_esize: 224032256 vfs.zfs.mfu_ghost_size: 224425472 vfs.zfs.mfu_data_esize: 0 vfs.zfs.mfu_metadata_esize: 2973696 vfs.zfs.mfu_size: 8696320 vfs.zfs.mru_ghost_data_esize: 261177344 vfs.zfs.mru_ghost_metadata_esize: 0 vfs.zfs.mru_ghost_size: 261177344 vfs.zfs.mru_data_esize: 6841171968 vfs.zfs.mru_metadata_esize: 9506816 vfs.zfs.mru_size: 6954673152 vfs.zfs.anon_data_esize: 0 vfs.zfs.anon_metadata_esize: 0 vfs.zfs.anon_size: 22016 vfs.zfs.l2arc_norw: 1 vfs.zfs.l2arc_feed_again: 1 vfs.zfs.l2arc_noprefetch: 1 vfs.zfs.l2arc_feed_min_ms: 200 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_headroom: 2 vfs.zfs.l2arc_write_boost: 8388608 vfs.zfs.l2arc_write_max: 8388608 vfs.zfs.arc_meta_limit: 1803964416 vfs.zfs.arc_free_target: 14080 vfs.zfs.compressed_arc_enabled: 1 vfs.zfs.arc_shrink_shift: 7 vfs.zfs.arc_average_blocksize: 8192 vfs.zfs.arc_min: 901982208 vfs.zfs.arc_max: 7215857664 > Finally when its performing well can you repeat the gstat -pd > dT: 1.001s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d= %busy Name 3 3887 3887 426514 0.7 0 0 0.0 0 0 0.0= 90.7| ada0 3 3987 3987 434702 0.7 0 0 0.0 0 0 0.0= 92.0| ada1 dT: 1.002s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d= %busy Name 3 3958 3958 433563 0.7 0 0 0.0 0 0 0.0= 91.6| ada0 3 3989 3989 438417 0.7 0 0 0.0 0 0 0.0= 93.0| ada1 test@f111beta2:~ # dd if=3D/testdb/test of=3D/dev/null bs=3D1m 16000+0 records in 16000+0 records out 16777216000 bytes transferred in 19.385855 secs (865435959 bytes/sec) Note that this is with vfs.zfs.arc_min and vfs.zfs.arc_max unconstrained. = In previous testing on FreeBSD 10.3 Stable, I had tried a setting of "4000M= " for vfs.zfs.arc_min and vfs.zfs.arc_max and had still experienced the per= formance degradation though that was using the GEOM ELI layering so this la= test testing using ashift instead on FreeBSD 11.1 Beta 2 may exhibit a diff= erent behavior. -- Aaron This message may contain confidential and privileged information. If it has= been sent to you in error, please reply to advise the sender of the error = and then immediately delete it. If you are not the intended recipient, do n= ot read, copy, disclose or otherwise use this message. The sender disclaims= any liability for such unauthorized use. PLEASE NOTE that all incoming e-m= ails sent to Weatherford e-mail accounts will be archived and may be scanne= d by us and/or by external service providers to detect and prevent threats = to our systems, investigate illegal or inappropriate behavior, and/or elimi= nate unsolicited promotional e-mails (spam). This process could result in d= eletion of a legitimate e-mail before it is read by its intended recipient = at our organization. Moreover, based on the scanning results, the full text= of e-mails and attachments may be made available to Weatherford security a= nd other personnel for review and appropriate action. If you have any conce= rns about this process, please contact us at dataprivacy@weatherford.com. From owner-freebsd-fs@freebsd.org Tue Jun 20 21:26:54 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 36E56DA2B68 for ; Tue, 20 Jun 2017 21:26:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2239074F9D for ; Tue, 20 Jun 2017 21:26:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5KLQrHM007152 for ; Tue, 20 Jun 2017 21:26:53 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220163] Allowed characters in UFS volume labels using /sbin/newfs Date: Tue, 20 Jun 2017 21:26:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 21:26:54 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220163 --- Comment #1 from commit-hook@freebsd.org --- A commit references this bug: Author: mckusick Date: Tue Jun 20 21:26:42 UTC 2017 New revision: 320176 URL: https://svnweb.freebsd.org/changeset/base/320176 Log: Allow '_' in labels when specifying -L to newfs. Reported by: Keve Nagy Reviewed by: kib PR: 220163 MFC after: 5 days Changes: head/sbin/newfs/newfs.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 20 21:30:45 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1AD40DA2CC6 for ; Tue, 20 Jun 2017 21:30:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 08EB77534A for ; Tue, 20 Jun 2017 21:30:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5KLUidW013054 for ; Tue, 20 Jun 2017 21:30:44 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220163] Allowed characters in UFS volume labels using /sbin/newfs Date: Tue, 20 Jun 2017 21:30:45 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mckusick@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 21:30:45 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220163 Kirk McKusick changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mckusick@FreeBSD.org Status|New |In Progress --- Comment #2 from Kirk McKusick --- Change made to allow '_' in labels when specifying -L to newfs. Will be MFC'ed to 10 in five days and to 11 when allowed by re@. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 20 22:02:22 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31A55DA37A6 for ; Tue, 20 Jun 2017 22:02:22 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) Received: from NAM03-BY2-obe.outbound.protection.outlook.com (mail-by2nam03on0051.outbound.protection.outlook.com [104.47.42.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BEDAE763A1 for ; Tue, 20 Jun 2017 22:02:20 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weatherford.onmicrosoft.com; s=selector1-ca-weatherford-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=HyMNcZDGi0FLdu0fZw4g6ySiFjK9JXydVR5pk/wc6nQ=; b=OQTtrd8jnsudHG+giK4RAjq77U+yziOQqZV3wjYzEXdOwL2SeHZsVgvcCuKZKUiFivsBZnn+dDTFHPgMUjcvV5YqpWfq1n+oS3x+l3XjZlwWF4WOOPEqEeQARUtFSPZ8C6AnSDrBo82TuKCugiY3Dk2zcltu+1P7La8z+oV29pQ= Received: from DM5PR03CA0046.namprd03.prod.outlook.com (10.174.189.163) by BY2PR0301MB1621.namprd03.prod.outlook.com (10.163.28.27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 22:02:19 +0000 Received: from BN1AFFO11FD019.protection.gbl (2a01:111:f400:7c10::178) by DM5PR03CA0046.outlook.office365.com (2603:10b6:4:3b::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14 via Frontend Transport; Tue, 20 Jun 2017 22:02:19 +0000 Authentication-Results: spf=pass (sender IP is 23.103.226.20) smtp.mailfrom=ca.weatherford.com; freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=bestguesspass action=none header.from=ca.weatherford.com; Received-SPF: Pass (protection.outlook.com: domain of ca.weatherford.com designates 23.103.226.20 as permitted sender) receiver=protection.outlook.com; client-ip=23.103.226.20; helo=032-smtp-out.weatherford.com; Received: from 032-smtp-out.weatherford.com (23.103.226.20) by BN1AFFO11FD019.mail.protection.outlook.com (10.58.52.79) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14 via Frontend Transport; Tue, 20 Jun 2017 22:02:19 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net (141.251.110.81) by DM2PR58MB014.032d.mgd.msft.net (141.251.110.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 22:02:18 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) by DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) with mapi id 15.01.1178.018; Tue, 20 Jun 2017 22:02:17 +0000 From: "Caza, Aaron" To: "freebsd-fs@freebsd.org" Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Topic: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Index: AdLqD7Yl8kqAk+eARZ6a4Ow2ixKi2A== Date: Tue, 20 Jun 2017 22:02:17 +0000 Message-ID: <47a7d5714e3c4db4aacb6a65d82d8077@DM2PR58MB013.032d.mgd.msft.net> Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [141.251.136.4] x-ms-publictraffictype: Email X-MS-TrafficTypeDiagnostic: DM2PR58MB014:|BY2PR0301MB1621: X-MS-Office365-Filtering-Correlation-Id: 4048c3d2-170a-46e6-f234-08d4b828052a MIME-Version: 1.0 X-OrganizationHeadersPreserved: DM2PR58MB014.032d.mgd.msft.net X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:23.103.226.20; IPV:NLI; CTRY:; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(39400400002)(39850400002)(39840400002)(39450400003)(39410400002)(39860400002)(2980300002)(438002)(51914003)(189002)(199003)(9170700003)(7696004)(84326002)(2351001)(8676002)(86362001)(81166006)(8936002)(24736003)(7736002)(189998001)(478600001)(86146001)(5890100001)(356003)(54356999)(33646002)(2906002)(106466001)(2501003)(50986999)(72206003)(42882006)(6916009)(5640700003)(53936002)(54896002)(229853002)(9686003)(512954002)(2900100001)(55016002)(6306002)(260700001)(790700001)(102836003)(5660300001)(66066001)(110136004)(3846002)(6116002)(108616004)(38730400002); DIR:OUT; SFP:1101; SCL:1; SRVR:BY2PR0301MB1621; H:032-smtp-out.weatherford.com; FPR:; SPF:Pass; MLV:sfv; MX:1; A:0; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BN1AFFO11FD019; 1:vDE7XU4nPswWqOWovCm4oJwCnbOMVQIEwSkLiizIT8HgO7jmH7lA3+DgKqC/IMS7UJxjEvpxh5nFltD3uFAiW4IRNmjYf2zyeJS2tEWGkQN+MfOXREng2PXe33noiyIGKMNrlGnfPds1EeCAnl06qRev7C5rkCxx4ug0woNSaDR/Qqvo4pADMDrkxcSLIeyCYpdG6uQ6z47AMTlZaJoC7NgYuOLsHOoArZ9LO7jxMW3i2C+kCIyuQG2r8SLupmBnIbLFXz0J1d9IxO5y6MBNYrmaVy3BvSm2IpfAFR4EpL+7Vq+gdRwnk6Na34rEXu8QxoCEV7t9luVI6qVi5D9sxwWWDMHT15QsWwflcBJBqEoWOMPtwKicZVMuvUaF2QosG5iJwv4xw/JVf0S8o2AUnelhpLXXCfl2TQ7sqS5tv6/nTfa8pXNlsCfYPWinNuzeW9cpZxMOQmoSrNXsm3fOUuEs9kEvYhyOHFsMzcdvTVI1BHsulfA02HSCJNXAWLTTXFTwOc8M/wS7VN+yPvforhRKbxMT7tTcYQc+cH4OojK1AvS1V4WmC52vMahag6t4nN5kefNS08MFWlq9AkN22V/tN1gfQsulLCtrI1josLhQ+VatY12xT9hPmeGCP/5KlMuC2JZDPmNdcE2wfjEEMc9VBmn1deP586913qStwruwnVe/wpkGVSoN698IBIt2ENBTFEnPD9RA4uOd006MxQ== X-CrossPremisesHeadersPromoted: BN1AFFO11FD019.protection.gbl X-CrossPremisesHeadersFiltered: BN1AFFO11FD019.protection.gbl X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(8251501002)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:BY2PR0301MB1621; X-Microsoft-Exchange-Diagnostics: 1; BY2PR0301MB1621; 3:7WRLOX/XMP5tUSFBsD5gz8jazuHpO549UXkT+G3gxzrk6RFE4bVWc96/n7ZSJ3gJOGTwZTRDPdkutaDQVpzZZzxOBYPupLVXIn2LxTS/2q/AsHmgvj8bnpMnJc4GXxYU0hTwfWEUNLIl7RG3O+QU8Pc0P2eR00j7hXOTLJCQ8z5i9AjY83aGlg632+BqjZ5TB4WJuIrKxj33nu9JJHLZ+VnDmkvQwgz/W4L3CW7HDaDYP6d8Qx0buTdpwvnyV/m5RTjSCkX1kpjYCxl7mLPISWW6y2RJzp+QDpL4b2+jcWWRxAOQTFLyokQkFkcEl7xmivDe+yqADtQxfjnJua555qA+XVtuGJhMnEXe/NhWf3evFF5UwI42TagQRq4tBrWwhJwBdSkOpSdLXK6JiykK2ighiysveNNEmoub4XZ/SfZQBYS8Jea3j+uinooiiRihDOl+FN2kaP+Ra8BjP8xeBJ+RrxhLSCYgrAA/Daw1kKX48RqNh0elojFJKV1x9LdQ9gFSwVBnY7qbTVhJZJGDng== X-Microsoft-Exchange-Diagnostics: 1; BY2PR0301MB1621; 25:KR1DYtJDiqJU+u4Q9DLcgdjyE7vfTNqnnU9fjUboHJ4TVNKQ+VYkPB+xm3yo12guRIOaiwusYtOypaLoGz3pqk5mNfOVvttEvfIGzbvyNtDveduNBgAQNojhsJNausf9wULrO3SlovYQb34DHbSYpU4MBE2+d29SvmaP6336qjLYVJU1LCDcgEyoONdaENN42qo59WcMxiWQms6U7eQCJgu2VGhRdU6dAUhGmu94wgEa1K8C5M2MdXKnRJxggvCRpCMCB/qWPoLOXzqm4VXaDfLa3cSyT/qNB+S/P3qJ11KAkloDuA9OSnDD9kIv4egedPiVW9JHvMxKcYm/kUToNBU+YY9Z1+2TspIddJw6szfBS2ehkQs5tMVkEfWA4DKOfm28P2EO71DIzj2glVU6U3RgTUgl0Qjvbb/Iy1AWpXP7Qhz3AKAvrTOlkUf/jmrxjNFb8XdIces0uc24wow5cSDMeV3j5bjFL6N62lV8MxA=; 31:GOmu2A4JlA349u8U82e2g2IWMPDT4s/n8xfnWV/1hjvWEjej+BDr9luCvEmJCitjj5/K0szQk1ACNpfMxYlw1+73xsZ8jCOltkRxPCCfHXjS2i57fnssyUbqtqXh1/IARjHuCTgdEiNfb1smenfRo16bixvXn+1g4JyHRTBIdENJGOckzo93qVza9E+peOBeA+IZU0Wi5tfgac250QPBG/jpJSkvVaNLAlFWVJpDdBIOB+ip/ES7On/oxmfBUfzDC3j6mETY7XSJrVvgNcqXjw== X-Microsoft-Exchange-Diagnostics: 1; BY2PR0301MB1621; 20:5ycyjWfsj73DrjUA1du/fHaDCbS3/WEqbadsUFxyD3NOhOo0FQ32pBWgjn99zz2lhlEeiRPX4QrtX3F52gT/0Nt/3JfxeXmxdbUuybeCIuahvSD3pOfgXDAjF/WXqpW7TMusRFW0CXHlihWubLg5dvUgAyeEKQLvgClKayW4t9mQNIcpWuALQIcpemzKvYxHyc29HIkL6BXUt9yhRBt7Vn+cd654yGkpFLBP55UxPJsAjXMLvTn1Qpl9Zaul6E/5uwx78KQhfgoPo8L0UeQA62ptgdckO87+yM+1R0VuUG1ldmpalAdj+cGr+KgpYuqjquTK4hd2l7RjBwTJTGbJ8/bX679gXUcEv+YHteRPRp4v3wbDJTtebeYha1Hu98qzVreavvGgG6lao8vS7utztuGbEMVyk0DQ9jKdJQVrb1G7WZ574vALxM8wYCi0ZYTXzbmbtCDOW41Iwv6OKdxLn4mpUueBXegkke1EqUl6FF4QA6GL5T70/YUB5cvnKusr X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(158342451672863)(192374486261705)(21748063052155); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(11241501159)(6040450)(601004)(2401047)(13018025)(8121501046)(5005006)(13016025)(10201501046)(100000703101)(100105400095)(3002001)(93006095)(93004095)(6055026)(6041248)(20161123564025)(20161123562025)(20161123560025)(20161123558100)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:BY2PR0301MB1621; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:BY2PR0301MB1621; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BY2PR0301MB1621; 4:j8u8rkUKxrdvESVvYqhDqRhIN9y+dTxHI6pfZKn6?= =?us-ascii?Q?I6e6uK5MdobX44AnLkmPDjXKGcKmq6sTx9HrQjmPu/L6uzM+zEJ2jy8wht8j?= =?us-ascii?Q?GFZIls06iHyOUIPfAvPFFpzaN3BV/+NL9oI9g/mJJQc+e6b/sx5KNO2a38ID?= =?us-ascii?Q?8e3TxIGbWfTJujtzL8OnKTwAzCAb4Wu8C+C8GQXngjDTBtmxFP+uoBZ6B5YQ?= =?us-ascii?Q?2K6yrJz73tlBSXhNTGmpaYqIFquhCBChSGw/XjMAQeeTgmbHgl3TDalubddO?= =?us-ascii?Q?EeY86W0Ywui4k0H/BhDmDKCi1xCFNeVuG8/taVKGqqLmILK+6jKuXYFGo0+x?= =?us-ascii?Q?08mIrRoRz7zd62Ya3E3NFRpA9MyryJg0IK99jlLB0Qy8U8bpqugiPGUoSOGl?= =?us-ascii?Q?Zot1WTUH0x9nKvvhs9wz5EjwmkGFKFJXIvos2/5UsjG/fJtpiamquq8mPnC6?= =?us-ascii?Q?ApkNf9l9CdDGnbmGTaSHa/921VoMYnAp2EMG6OGKl1MNcKreG+mUklb+WyDj?= =?us-ascii?Q?ntNKzSuiRnSFfRRXEF+ZDRZ6sacX0fCCe1IUVKJRMfbtTBQRNvjDvqc6jq3I?= =?us-ascii?Q?EPPLBsB2BP895uftevYnQwP+j8XZA6CuRY+IwZPv5MPgdXMLqs+GuZeH9Z9C?= =?us-ascii?Q?Q4BmzCnhkpK2zi0MAKPwZuTyleF8AH96DWHRfet8uBTUqOfvzEfbQPhZ5lUT?= =?us-ascii?Q?ebrjHGZj6RwqCsTW3UWHu4pmdi9E4UOeAiDb44EszEM/muSNon3glcV0IlcH?= =?us-ascii?Q?KKl6ZiuNkZ7G5tSM5jRRtl/QIkUFqPBD4EmpWrr2iZEkpTUdcBoz0bU8mCJr?= =?us-ascii?Q?7CtpWFYs4DLtL4bL0FtQtToBz0nKRCmGpFLwFoisMZXeYYwiAIn3sLqXTZeJ?= =?us-ascii?Q?WsjH6xIrzUYenAadQwnyUI3VezkRIqidjQVjetbNIB6J8Mkw+XeBOt6Z55T7?= =?us-ascii?Q?+JnmSBTG12GfoTfxqbJm3Ym/eTKkks5CCYoUImQiEw3kOOFfshvnvZSr6PWO?= =?us-ascii?Q?rSFH1KnAbeQxBZ2pwOhGihPtxM9F3H/d5gbm62kUl75ZjGPsf7aQ38zdU6hU?= =?us-ascii?Q?pIeIFDFVuvyC5Kp4cfLiin3rWfHSoQ6BdfguKS3np5AWCf6M2TYfgLDFnlOq?= =?us-ascii?Q?dVOsKst6QQJZk7LS+BUKYw+f49MAN4PMLASSvxiyeiJeI+4ypbyr9bOl112R?= =?us-ascii?Q?oJvwDVCHy17xCCoBVxVOFFz5jiRJ+sAc9k9eoyFFrDMQXZw7rJmSX28SDJCj?= =?us-ascii?Q?cqETW8fkGTA6pqoeEa3MptemU0NpZogIjufrm3TJOwfD88Rgkq0AHRtjzJwS?= =?us-ascii?Q?NijXZzOP+u9T5P4kgzM2biul6Bp02iJQbNQlw0XdBcNu?= X-Forefront-PRVS: 03449D5DD1 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BY2PR0301MB1621; 23:4mDtqlOCuAfFM4fYz/iQ73l/RPN+wDbgos6/TMR?= =?us-ascii?Q?I6y5EaivOoXvQgJ+H8CMLEzZYZeecsRPQvKMkVOmYPRm/Qy0Y+F0KeBMoobB?= =?us-ascii?Q?U2y0DHvftgkpVTS67JFLl7IoEvgTxV96eb380vTpp+sHwjv8TkYcW3puRyfL?= =?us-ascii?Q?UFiIR0PxXTtzj67IakDEAWM+Ok4wtTrP1T9c0MIl3isWMiDpfd3rc6fnAJp0?= =?us-ascii?Q?Bg1I2WYf/TcdjME6GIA5+aKi/kTh8egDNkfxP5MuyPUOwSHfPRwBaBtaCTMM?= =?us-ascii?Q?sUKGESDtMfkNKXVDxJarCqFIuIU4SSccNdrEaXdoFmET6Dz2hjjp3dBxq5az?= =?us-ascii?Q?M0wO7oxG35wxCnCFiM3ZN9gYfnnS89ipT12cxcGA25/VWYFJMZ00BOa5OC8W?= =?us-ascii?Q?Yte5HPanhU6DLlYMrs9YxAWMywPAfGuFNC8g4lzrE4mJKHWKLfNFDrPxTchf?= =?us-ascii?Q?y2YZyUHN5O1UTrgxsF/myO5wUZPq4nj66gJ/91t3Y4odYjIeMdTf+53vt/mE?= =?us-ascii?Q?ZeEu7cwExkytYqVEgmscpn3Yh2rW+Jb/FCgiq6ww7uniSMyOz/tIFGOmfqSt?= =?us-ascii?Q?v3iB/D+tDerHUBk35lKbfcBsO0K20QniWGZfY+Ned4z7OmivFK+0+zZrw7Z5?= =?us-ascii?Q?G5uslwlljGgvH12QxFTH1naxV2nzbournlAqyaymmGX1b7JK7KSYB0LpbJIb?= =?us-ascii?Q?QwSVFibpGfsSwDFvd+ctOiar7dRmVQqOzCVk+al0YBEbsj40mkiyYNalkv41?= =?us-ascii?Q?Bubjs5aLOUGe84Pf9zwSz8QeVWM7ixh36QPSdkfqhMDxje7+As6trp2EruZF?= =?us-ascii?Q?vkONdIjx8/gFwb9jRNXfqv1ZqqFedDorxW2Zl+e8236Jj5fAOlhxud5d/xgX?= =?us-ascii?Q?AcBMRAXaqMSm1ISlMFoMzUzvCcBHsv3VhyJwtkTlAP4w2Lr75hFquPP15djP?= =?us-ascii?Q?/p55ZEZTwiOB8bXXJH5QR6zEmUAbw3hkzOREAaEoepAriPHsTTQ0bGDZ5Y5N?= =?us-ascii?Q?eiF5ok5tXpKpslq3SZuCyr5mm5oxR3zTOgbo6UEacY1tjRzjFcgGE5swYTZX?= =?us-ascii?Q?nopigxyWep7K7m6us5zOL2jQcc+g330Y8IuWYHBIl7q0DABUnko0oYEj2ltV?= =?us-ascii?Q?yHPc3rCcKsyeWylMWZSrmbMfml6g1yEGtkAOlzHy/gfheaUki1vefFATUCgR?= =?us-ascii?Q?ilF4RXeC/GIqftdgBLLB3L9W2e1M+pwMBFuynw0L9b38ce6ngMTuENbWeNRV?= =?us-ascii?Q?o0J+dxsyELM5TVmIcnO3RoprtmV1YMKUbAqFLs1fHTRZM6UjUh4wqd0I0MeF?= =?us-ascii?Q?jRiYbrCLYape9cLU/KWDoMnBXkVI7J3Id9ssy/n2F4Nr6?= X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BY2PR0301MB1621; 6:9Lq9avE/B3PoCXOAM4vmeVx/LlJQ6It7v6OvRYxB?= =?us-ascii?Q?HWcifuTabeS71PcVusrvaTrXj/qJwoDGFS1fjh8/TKORn3oHF6fj41dG0U3j?= =?us-ascii?Q?6bQdnkTTymyHXeQ+pTTWIhwTHXE0OhfUvATQLgLpz3EXj/4b9ThuCcp2St8R?= =?us-ascii?Q?YI0rn9IcOS8hpiOqAsbxxldtk2dSweMslfVc034qtL13zJu1H0Ma4j76YwSR?= =?us-ascii?Q?bku/I3+oRlOa7ESStpSfBL/yiRD4a228kXWVZeyqaPZ0osVOAcN9EKWHmVGn?= =?us-ascii?Q?KN4klR46uWNL5ggUHq9CHZ8euXlx8I7dQUg/w3en5D4wil/MsJaszefgWGKn?= =?us-ascii?Q?FZKvnFeQfNIZnnnN5QljnywsQLCC4tYGYzkoRVXbeQqNoiMHgji1D7WmKmUK?= =?us-ascii?Q?sU2OdKBkoHMm1LKfqTCsaHGoFiD0ufic8Dia6/xD+K5R2kQ3AZ9u5lHWSFEi?= =?us-ascii?Q?MkE1Mk0VwjSa+Jc0zJEiR04oJ5ooBqa5GM93LpzsmdBH8ZTb74C5yVOQBDuv?= =?us-ascii?Q?lqAbNtyIRkLLo4s15E/jg8FkTXgMtko2smfrh/814Ysxrhn3P1HAVt8cK6Nw?= =?us-ascii?Q?U+JZ7ZpVM/fhDv1uHBxw9oc7CH2zBJPZczsbf0gr63DADq2whE9rlBtCHnvQ?= =?us-ascii?Q?UwDHPUa9hI1YanHx6mVbtunZEZRKCoGUvGZJnMHfIxzVI415skhA1ouiRso1?= =?us-ascii?Q?n5B+nMqcftrHrXnAMrRIDeh7DufVIYPDEgcJWxEpn7qTMFYhB6cRvIUAmEgF?= =?us-ascii?Q?1ROmJVu62QsQRimCiTrN1t+9BCY2zLlKdrxv1M0k7DygSzF71374GV662hAc?= =?us-ascii?Q?zBXp66AR054d9lobcPJamarYHh8rLhsbPUfiJAU3n49d/SiX+BH+bBYs5iV4?= =?us-ascii?Q?y2xhgiF45pIyfKicDlEa7znJwQ5NutkVnnO+R9Cu2QXEKnl4yVuoxOerT/zm?= =?us-ascii?Q?BvtZ9G4/guYKp5zygzM5DfGactC4D45li++7elrNlw//wUeScrQWJE05sRoP?= =?us-ascii?Q?zPI00souULvokMMx2QysooGP?= X-Microsoft-Exchange-Diagnostics: 1; BY2PR0301MB1621; 5:69caCGt23Tk6KoMb/Ms45mRuL7QJYSUqSqvrHwuQmyE/qnX/b/VD/kcvU0IQbElLsVBAaaPwbxe2t0tiDSpyDSst1u0O9cUIeuF4YEKTaXCs6oOEg3nH5xbs07UQnUMtLs6NhOVmHG2BnouD+iI55n8nhKDMDZUhvVsgV63tmOo/xUcH3YPKyuNdc0SMY2uu8uZFsLea4aZFrASd40ZhZu0sq77qwv4a/QTrXktD0idnL1dfJ0LDL2FhwhFlw5duIBwxzHYjz5RjZRSlknFr5lu87cby8MBebnbS9L6CSgi1WBQUlRERC2c1JTN/VYTcOgV/FIo/nuBsbQNWtRXidWtkRcn11qX8ElFcQKUy2/0uSIBJROa7ua1Iy/d5Hmd1pc5XAALJbtgwRjpzJXAHf16O0QClOqNDYY5rAMHBS5iI1/hxociZMr+KeQ+Yq11d/Nt1E+vNFEpH8YAnWWD82n3gMRm3xieOSAZ7TA2GsT4u3uC06680QhpXxbUWL6wI; 24:5iTuOPKKsYYTeBY8WVOpXHuzmOGHI2aECxN+GiMo033hKiUpn6F7HuFX/Z47XZpk3EMbipM6ERlrLXtQQNXn7SOVWnEtDuQDqtqE8MMhuJw= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; BY2PR0301MB1621; 7:0F6+1H0jPh+fYEs1qrhFh7tp1Do9pO/y7Smq5eZP/qSoHzu9rLpWOJQvnLFRQGLhHmIM5FCFb8yx7dVwj9cyaIEEh08wswgVL2y486kkLg0iyzom5GgJjVF2vvR9fI2hHB0kURQAags+X2rNd91NWKBLK6PCQpGH+YVXMKGwJim9c5lkqH7sGCNgSxp74nhYOZZuShcCSmlwS1b/rz/RQUFW1VL94h7ymoNdZzQ67QK1Xo7v5eFaSHqz4egE/6gJroAoM+sx0mcPt0BlUscu6Zpn3dbONGUSvNttkZfAtZwFen5p7ZtStjbU7L22jz4gnE/wg8vXq5sISzJYMj/gTalmC6pkDFYaniB9lmLaUZUz8WCqYPz3P/qa7afjeyEzl6KSVYDZzCLbsWZDs8Tsrql6zZMYVr/tR/m/quhIVJ9LYjddUA4le6Xvvw9HXZn5q1KU0ednMFOZ1Kb67wpi9hMZt3Db6H4abw59q6B5MaiSHPPTfrkDk9Ak6vFjDvAXD7JWBBE/YWvDwrKlsoMktf2pvAXv39kN31zOXzg7x4q24g1v/wmYoaZMYE7/c2lvrCPim7I8ngKbxeS0RnOuqndAERrzlUHPkChRxuTvjCcFaNZ+vEA0yiIOLpLP6o2FcJzaeC+vnJ9Q9oa9w6w6NtXdKPNp8zVwVVHRtEgpIb6FF2khwJrA8DJsHNvf80PyOsLeIH9dQkpKwcK1viuTALLsUh2/pZy6Z/D9cBhce0So/BfGGGun4RuqrxGnAk/PD42IasrzMn4rCdI+shRqKRLDP9JQksSh2xe45EDFDnk= X-OriginatorOrg: ca.weatherford.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jun 2017 22:02:19.0205 (UTC) X-MS-Exchange-CrossTenant-Id: dd63fb60-07f6-4d96-8d40-ebeca61a524e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=dd63fb60-07f6-4d96-8d40-ebeca61a524e; Ip=[23.103.226.20]; Helo=[032-smtp-out.weatherford.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR0301MB1621 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 22:02:22 -0000 With regards to this issue, it's a case of the old FreeBSD 9.0 settings not= working the same as in FreeBSD 10 & 11: the ZFS ARC min & max settings I w= as utilizing were too aggressive. On a separate server with 18 days of uptime, bumping vfs.zfs.arc_min and vf= s.zfs.arc_max up to something more reasonable sans reboot restored performa= nce. Apologies for the noise and thanks for the suggestions. -- Aaron This message may contain confidential and privileged information. If it has= been sent to you in error, please reply to advise the sender of the error = and then immediately delete it. If you are not the intended recipient, do n= ot read, copy, disclose or otherwise use this message. The sender disclaims= any liability for such unauthorized use. PLEASE NOTE that all incoming e-m= ails sent to Weatherford e-mail accounts will be archived and may be scanne= d by us and/or by external service providers to detect and prevent threats = to our systems, investigate illegal or inappropriate behavior, and/or elimi= nate unsolicited promotional e-mails (spam). This process could result in d= eletion of a legitimate e-mail before it is read by its intended recipient = at our organization. Moreover, based on the scanning results, the full text= of e-mails and attachments may be made available to Weatherford security a= nd other personnel for review and appropriate action. If you have any conce= rns about this process, please contact us at dataprivacy@weatherford.com. From owner-freebsd-fs@freebsd.org Tue Jun 20 23:44:47 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A75DDA4C33 for ; Tue, 20 Jun 2017 23:44:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 481E2785B2 for ; Tue, 20 Jun 2017 23:44:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5KNiluq072729 for ; Tue, 20 Jun 2017 23:44:47 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220163] Allowed characters in UFS volume labels using /sbin/newfs Date: Tue, 20 Jun 2017 23:44:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: freebsd.bug@nagykeve.e4ward.com X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 23:44:47 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220163 --- Comment #3 from Keve Nagy --- Looks like the issue I reported has already been identified and fixed. We are just waiting for the patch to make its way to 10.x and 11.x releases. It seems like I can change the status to closed. I just don't know if I am allowed to do so. I'd better let somebody else do it, as there may be more administration to be performed. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 20 23:49:24 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D245CDA4D73 for ; Tue, 20 Jun 2017 23:49:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C091A78720 for ; Tue, 20 Jun 2017 23:49:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5KNnOdG079194 for ; Tue, 20 Jun 2017 23:49:24 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220163] Allowed characters in UFS volume labels using /sbin/newfs Date: Tue, 20 Jun 2017 23:49:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 23:49:24 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220163 --- Comment #4 from Conrad Meyer --- I think Kirk will close the PR after he MFCs it to 10 and 11. Thanks for t= he report, Keve. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Jun 21 07:42:05 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7BC3DD88C5E for ; Wed, 21 Jun 2017 07:42:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 69F0183F90 for ; Wed, 21 Jun 2017 07:42:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5L7g4Za057693 for ; Wed, 21 Jun 2017 07:42:05 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220163] Allowed characters in UFS volume labels using /sbin/newfs Date: Wed, 21 Jun 2017 07:42:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mckusick@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jun 2017 07:42:05 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220163 --- Comment #5 from Kirk McKusick --- Indeed, I will close it once the MFCs have happened. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Jun 21 07:58:46 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E916BD890C4 for ; Wed, 21 Jun 2017 07:58:46 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x232.google.com (mail-wr0-x232.google.com [IPv6:2a00:1450:400c:c0c::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9F01584495 for ; Wed, 21 Jun 2017 07:58:46 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x232.google.com with SMTP id 77so123399415wrb.1 for ; Wed, 21 Jun 2017 00:58:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=FYEb1py5PyE2GmULcmiqio4Oj5i79I7DeaHwQfg6wQA=; b=vJftco/IXaPsv/s5s4RMuQt3iUhugQBeuJKGtpcEf4MoCXeyqmowxbw37o2Bb/MloY agKzQrGHRA59CVOeyPElo5bV196auRZCERc5bg56z9h38vadNbpQEQhGagSfdha69KTW X+IekGGf8q+Py+ryLkQ5KjyKkWndc4h1e/Jg9Rk6rO7p3HljeE8UUL7mqEt2zYsZ7PVn JDNJXDVfgWsYaiNYGXV4TacHY+7IwMkwdvtGyaxSz1gEF86KURw+rpxYZZFPqN14H4gx Uts3CTwgGzRcYMgW8DqtPhPnkjPWQNkV8YWPAzEmJgNsjjZp/CuNGZ9uOJQ1c3mwzgXC JQtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=FYEb1py5PyE2GmULcmiqio4Oj5i79I7DeaHwQfg6wQA=; b=l6rJPXTb/C1uj7wdC5MLqArWpJRX6sgGX8G4xgEx10Sy/R2dQCjLZjQVqDK8s0kTiq JFNk6+dRUE0IpmC3RhWoIr2qA1yjfD5YeYpcBDx2zht28XaDN80kUeoRe2j2kVOvbeqr Xc0gNNBG22CKQkf+Sjg4ns87YSzqp44o1w8BagwLUSvZikvDxwTr+8rkRFlcb3kQGHLW 3NVZsDIMSFrNSBNcPbTaENZQ52gKHMx3aq7rDkDWRIrNKsd4ZHnLuhJNgCGIHMC7Se/7 wDxOvRXiU1I+9CRy/FT+PhZPgop1ww2jWQLm7mQC9mShsbSN9XWclaNkwmhcutVAQDkc V7pA== X-Gm-Message-State: AKS2vOz4xb5scbasi2npNXROLu5ZgNM9lnDmOiUvBswoA0IDlW3+V5o0 Y7Mr5CesSvaeKX2BzgZuCw== X-Received: by 10.28.153.75 with SMTP id b72mr5615604wme.46.1498031924345; Wed, 21 Jun 2017 00:58:44 -0700 (PDT) Received: from [10.10.1.111] ([185.97.61.15]) by smtp.gmail.com with ESMTPSA id k45sm1966636wrk.45.2017.06.21.00.58.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Jun 2017 00:58:43 -0700 (PDT) Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs To: freebsd-fs@freebsd.org References: <47a7d5714e3c4db4aacb6a65d82d8077@DM2PR58MB013.032d.mgd.msft.net> From: Steven Hartland Message-ID: Date: Wed, 21 Jun 2017 08:58:42 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: <47a7d5714e3c4db4aacb6a65d82d8077@DM2PR58MB013.032d.mgd.msft.net> Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jun 2017 07:58:47 -0000 While that may be the fix the behavior is interesting none the less, see my other reply about IOPs sizing. On 20/06/2017 23:02, Caza, Aaron wrote: > With regards to this issue, it's a case of the old FreeBSD 9.0 settings not working the same as in FreeBSD 10 & 11: the ZFS ARC min & max settings I was utilizing were too aggressive. > > On a separate server with 18 days of uptime, bumping vfs.zfs.arc_min and vfs.zfs.arc_max up to something more reasonable sans reboot restored performance. > > Apologies for the noise and thanks for the suggestions. > From owner-freebsd-fs@freebsd.org Wed Jun 21 08:01:04 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D0168D894E7 for ; Wed, 21 Jun 2017 08:01:04 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x22c.google.com (mail-wr0-x22c.google.com [IPv6:2a00:1450:400c:c0c::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 608B8846F0 for ; Wed, 21 Jun 2017 08:01:04 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x22c.google.com with SMTP id c11so68019771wrc.3 for ; Wed, 21 Jun 2017 01:01:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=DxsK+0Lpt2vUcHLUsgHVvhwprB9gCsjEVFo7v7sZtd8=; b=x+X+ZgaMebDzYhEZbrQdsuZ3+QJitCwtNcE/2W4atcowdVWrxSJ/tu8XRobLREpPmm KM5NYa28cdVhkypAS0uzHwv6P/I3zCG/gSYN3xyP5mX7r2X2Tb2yAgVWF9Cgi6qfSRH5 O4NzaKVFhnfmYxMwb/j1mqiBoRH/mTGpnVqkgNV3+xeJT1PlH60swtRWinQBP7AlU8E/ IR/eSHcznns0x7lrezQYYYYIkcPv8LEjf9uOCD/5mj2wi5yqqflGB+DeiusP1SaP5AiR P6DhqrZuo6RctNm+8h1uYs5ASYIsiAxAeZWy5CFJKMEsepTP1anURrqrFCS6DuCGjWzR vmHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=DxsK+0Lpt2vUcHLUsgHVvhwprB9gCsjEVFo7v7sZtd8=; b=ZLJAFTqnFMb8wZmL/2cJW8mHP0xqkDCcYW3mW0eA7FIbWxwEUjgmgXXJ7UXBEPdqpe 4q4Ya1Yy8G13sS07hqvxDJl4tAdyWXwBCEuAzKpLXT81nwi18lzrAmqVqy9ZfBFAwlzI qknPuyVUZzKpU+ChA8f/lRtRLLYCwKL4KBMuPqwABWtQWXKBbW+ZZMq7oG0M+aeA8RQM l5AS2UPPI8iQu50S5uDhXT1HNmPPVjQaf5AnDum9Pe7EBATaUfGHexUQOFc8Be15WcGd ZOtsrmhODlU9PrWPmMLVdhaQ+Ex0+SR1enbFK7GJwHb5AjM5TjdWA9gqCArIsMykm7cj 1Row== X-Gm-Message-State: AKS2vOzhX+tw/uYOqEKu9Jt80uRQ7WUCt37n80rYn2qCN1zu0gCIIvL4 2XC5P0nG4vEueLH0ZN3IUg== X-Received: by 10.28.63.209 with SMTP id m200mr5704550wma.40.1498032062296; Wed, 21 Jun 2017 01:01:02 -0700 (PDT) Received: from [10.10.1.111] ([185.97.61.15]) by smtp.gmail.com with ESMTPSA id e24sm7459417wrc.35.2017.06.21.01.01.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Jun 2017 01:01:01 -0700 (PDT) Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs To: "Caza, Aaron" , "freebsd-fs@freebsd.org" References: From: Steven Hartland Message-ID: <86bf6fad-977a-b096-46b9-e9099a57a1f4@multiplay.co.uk> Date: Wed, 21 Jun 2017 09:01:01 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jun 2017 08:01:04 -0000 On 20/06/2017 21:26, Caza, Aaron wrote: >> On 20/06/2017 17:58, Caza, Aaron wrote: >> dT: 1.001s w: 1.000s >> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name >> 0 4318 4318 34865 0.0 0 0 0.0 0 0 0.0 14.2| ada0 >> 0 4402 4402 35213 0.0 0 0 0.0 0 0 0.0 14.4| ada1 >> >> dT: 1.002s w: 1.000s >> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name >> 1 4249 4249 34136 0.0 0 0 0.0 0 0 0.0 14.1| ada0 >> 0 4393 4393 35287 0.0 0 0 0.0 0 0 0.0 14.5| ada1 >> You %busy is very low, so sounds like the bottleneck isn't in raw disk performance but somewhere else. >> >> Might be interesting to see if anything stands out in top -Sz and then press h for threads. >> > I rebooted the system to disable Trim so currently not degraded. > > dT: 1.001s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > 3 3887 3887 426514 0.7 0 0 0.0 0 0 0.0 90.7| ada0 > 3 3987 3987 434702 0.7 0 0 0.0 0 0 0.0 92.0| ada1 > > dT: 1.002s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > 3 3958 3958 433563 0.7 0 0 0.0 0 0 0.0 91.6| ada0 > 3 3989 3989 438417 0.7 0 0 0.0 0 0 0.0 93.0| ada1 > > test@f111beta2:~ # dd if=/testdb/test of=/dev/null bs=1m > 16000+0 records in > 16000+0 records out > 16777216000 bytes transferred in 19.385855 secs (865435959 bytes/sec) Now that is interesting, as your getting smaller number ops/s but much higher throughput. In the normal case you're seeing ~108Kb per read where in the degraded case you're seeing 8Kb per read. Given this and knowing the application level isn't effecting it, we need to identify where in the IO stack the reads are getting limited to 8Kb? With your additional information about ARC, it could be that the limited memory is the cause. Regards Steve From owner-freebsd-fs@freebsd.org Wed Jun 21 12:32:01 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ABF24D8E64E for ; Wed, 21 Jun 2017 12:32:01 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on0047.outbound.protection.outlook.com [104.47.36.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2DC0A674B4 for ; Wed, 21 Jun 2017 12:32:00 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weatherford.onmicrosoft.com; s=selector1-ca-weatherford-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=NvZtp+RuvVR+Oxlp4dOmQ4LhgqSNkdhGE5n+XH+ATxE=; b=oVzlYMwkppYOw28oPSaBzqtVKlhGHND8KHGmeQWCY4A6wzUTfjNiydGno1GpiUB+q8a/3IJCOIFCqUaBz8tE6pEn9fR1TumAqZ21Crbdm9MakQZmwuqgnLrw2LJ91rpbBhC9ZkVwxrA9XHYmKqzajOD2AlPdQvao02KZLrBm/HA= Received: from MWHPR03CA0011.namprd03.prod.outlook.com (10.175.133.149) by MWHPR03MB3278.namprd03.prod.outlook.com (10.174.249.34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Wed, 21 Jun 2017 12:31:58 +0000 Received: from BY2FFO11FD042.protection.gbl (2a01:111:f400:7c0c::197) by MWHPR03CA0011.outlook.office365.com (2603:10b6:300:117::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14 via Frontend Transport; Wed, 21 Jun 2017 12:31:58 +0000 Authentication-Results: spf=pass (sender IP is 23.103.226.20) smtp.mailfrom=ca.weatherford.com; multiplay.co.uk; dkim=none (message not signed) header.d=none;multiplay.co.uk; dmarc=bestguesspass action=none header.from=ca.weatherford.com; Received-SPF: Pass (protection.outlook.com: domain of ca.weatherford.com designates 23.103.226.20 as permitted sender) receiver=protection.outlook.com; client-ip=23.103.226.20; helo=032-smtp-out.weatherford.com; Received: from 032-smtp-out.weatherford.com (23.103.226.20) by BY2FFO11FD042.mail.protection.outlook.com (10.1.14.227) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14 via Frontend Transport; Wed, 21 Jun 2017 12:31:57 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net (141.251.110.81) by DM2PR58MB016.032d.mgd.msft.net (141.251.110.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Wed, 21 Jun 2017 12:31:56 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) by DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) with mapi id 15.01.1178.018; Wed, 21 Jun 2017 12:31:56 +0000 From: "Caza, Aaron" To: Steven Hartland , "freebsd-fs@freebsd.org" Subject: RE: [EXTERNAL] Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Topic: [EXTERNAL] Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Index: AdLqAtGh3ZrvWZ+7QKOnTYE8YOBOpQAYbQyAAAdhXYA= Date: Wed, 21 Jun 2017 12:31:56 +0000 Message-ID: <7dbee74c0071463db4fbb0a5de2e7244@DM2PR58MB013.032d.mgd.msft.net> References: <86bf6fad-977a-b096-46b9-e9099a57a1f4@multiplay.co.uk> In-Reply-To: <86bf6fad-977a-b096-46b9-e9099a57a1f4@multiplay.co.uk> Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [68.145.247.165] x-ms-publictraffictype: Email X-MS-TrafficTypeDiagnostic: DM2PR58MB016:|MWHPR03MB3278: X-MS-Office365-Filtering-Correlation-Id: d7d0261d-697c-4519-df12-08d4b8a1823b MIME-Version: 1.0 X-OrganizationHeadersPreserved: DM2PR58MB016.032d.mgd.msft.net X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:23.103.226.20; IPV:NLI; CTRY:; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(39860400002)(39840400002)(39450400003)(39400400002)(39410400002)(39850400002)(2980300002)(438002)(199003)(189002)(24454002)(377454003)(9170700003)(108616004)(86362001)(7696004)(54356999)(512874002)(76176999)(42882006)(2950100002)(2906002)(22756006)(5660300001)(229853002)(50986999)(84326002)(102836003)(3846002)(6116002)(790700001)(86146001)(8676002)(81166006)(8936002)(2900100001)(24736003)(33646002)(106466001)(38730400002)(6246003)(5890100001)(7736002)(2501003)(9686003)(55016002)(53936002)(189998001)(54896002)(6306002)(72206003)(66066001)(478600001)(356003)(53546010)(219204002); DIR:OUT; SFP:1101; SCL:1; SRVR:MWHPR03MB3278; H:032-smtp-out.weatherford.com; FPR:; SPF:Pass; MLV:sfv; A:0; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD042; 1:FNTihlw9HXAjw+RO/iHPVhrbD8WI+9w+jWq/pemHTckZNpGSSBrSk9wHKU6iJrsbf/7qbrAqQvsszyKAEguZDt2L8NQalkE0E4hCwnH9rxww/vdFkzQmr7x3jLa2lsb6Z765pSdVm+W3YNioo7wNpiSaFu7UVjOWdCLpDd3abUy0GYlYIKpaI479DETQYfoYosjl0t1Wr6NNW+KKt87Yn8pITRLXKeu7UQDfgolVY+vvNazDb8wQ3Xo6P3W3y9XJrhm4V4YoMQQD2rTQ2HsOaSBfr5d9JW/BZdEeBf+oJbDlGnz5jtp8qwjG7ZjNv3sMFeT36YheyPmfg6Zd/AECRCIdL1s3kVy+OLUEt8dWT4AP/f+lozAZUR1p88cqO4CdMTT/DUPeUueB+PdnJVQcvn18ZFZ5yjQIznZUZWXwdQKtL4Qrc7DcMAYlWfMGnRPUiI4utgYLD0zjLhRITiKx5ukW9HH8j3zyp+cpibsScmBF/Pc11rGvZ/+VhAiSBp3bqk3two0xUU/KSMIrJVO4qg== X-CrossPremisesHeadersPromoted: BY2FFO11FD042.protection.gbl X-CrossPremisesHeadersFiltered: BY2FFO11FD042.protection.gbl X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(8251501002)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:MWHPR03MB3278; X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3278; 3:mebe2eLBbEqCMo9QRIfpSOzdwjFCTNpCdN0FwoWC4v7T7UK6Fc9h3vvI3mNJrQU/XEY+8K/Y/lUwIVLCqVIQazM1YIr4tf2AfYBLKZPgDnNMcpwNJiFxYLq9hnPli2wOnU257VIcqVr69hjKgOrdU9BuVcJHEOg7bvrc2sy1p2RuWZeu9U0eBGkvJr4/gdIyV/Bk/ixoQ+4/DFyuWhqdAJYX4GkhhmqDjH+z8TitprT6EUjVX8CiDj2V1EwOR1gHlDNXynkeM1jDYo3n4sFf4tqDvW67Uw74bb3NR/hrjxJgImw4SjDyySn6MbIVwhruVwqMTkdmcTjniLEWna1gJVJEZ+/vb59K7nKY9wAYBIar7jrriY1gLlj7PyDzVBlujfM+F2tlk3MDLY7ggEhvfNGOwoxycqxzV7uTxJxddQcGU2R11TOsJHPnnpVtV1GIYajiAqYRszDT1Wh9R8HZammqkxV25Vmvu4i3m5Kp7+TenqvSzd5sFa84F0N3XTR0BkrJuxGNgKOBSh+l+0rrbA== X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3278; 25:KcFn5gJVblsFyZ97LtKb7H7Wsc9xFG5/E5EUysDI0RqZVvpBuXt2+n2ILq653nO2Z/uYOTmlckcdohAnv5ZLWa4RgMWPgjXwe9Qr8p4lNmfVTlU7qsmbHIELtY6mZWKyomdGZCHA7xob9iuwAkVhDirV3IlK25qvOUycrnsBfy0V+vmLHtq+0vpDJe6HzT49iywWuxZT9BLW45fqVbSnr+e/JPDM8KQjYO/dkMaVc/58wvvbm3PSqqEsmc85g/IbRMMYq2paoheI7AlOYZrqJcnGcg5ctaYunqKHAl2qbvn5WtkbZuUJWcylM/RF/actu40avgz58EN5fqNOCFySCuq9ECaLtcT5oPd3FyZOurHnABGZwIxl4d4b2oWbAson9q7tYI/z8142mcO3CMU/Lg64+RvUmY7FRUadescBFGVObXFztxCM2XvacXEPdYvGWcuxXDYkPQl8RVBiLU0oS4Fkfbx+fqrBny1LFE5Ha9g=; 31:LlbuAh94FQF8oJ2qNsk1p2Ri05cL1AxBnEOU2JoQo5l1dfJt5dgY5PnJzqT69Kw4YMGfAh1NVUcVVvdNgE63BjEYlAS9ZQ0Qqa4+3fLzUjGESpU4WCqIUHwxR+4WlrLMVsU+fYgz2s70P1FyALnZxjyVEAWy+xaX/SQPE9jDLVyf/bFxoU2JZfAM3Na7TaciPZShqt09DxvbaddVetc06AAqGdkwhTsvevo7SL8yb+J/mDUaaUNMghjq/tAqvqnJq1PHjPG3y+FsvXYRHe6jjw== X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3278; 20:jwHkSrxjXaDgd0zIjBsxAzAjzB+1h/gUrFOYd646HhKhhAAOgFpDhaBvqNEabuu4qmZ6KI7MlG/lmChD2EsJAUXceNwV8eEbwYdQRHWWjJ2xr4KsBWxqs13Vamu9A5H6O16N36SOVpr7tjV/FnOBnhyxhFTpScvvimJONpHBqNamzKW42k5y5kBOjsFyZ8BIf5IPsAg0meZFikBeNiWKayEReRIcEZ39D1TgdjMr5aQ9D+3cPewDZBBagrvHMZNpJ+9C4iLCG0hyd78sU70YNNDRZEY8eCf2m2A7TdCC8NDscxyVIEQCKKDQWqeqqvaUqM5guAnV/iw6E7lg7Cx1WtudWo+GpjleIIgOwMnz/K0p7vPwkmK3suvf+60p0vpfr540ItzDTgzMXYWq8mbIpxOj5iquBAjgKOLRWzZ79RZOpYRnwoFvVYUhrE8K/xUnkk8W+ySTT+ALUToVCbGW8KCMMCUaXOXLxr+2h95XAB1PPLctG+vsXqlNhkYWYYDr X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(192374486261705)(21748063052155); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(11241501159)(6040450)(601004)(2401047)(8121501046)(5005006)(13016025)(13018025)(93006095)(93004095)(3002001)(10201501046)(100000703101)(100105400095)(6055026)(6041248)(20161123555025)(20161123560025)(20161123558100)(20161123562025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:MWHPR03MB3278; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:MWHPR03MB3278; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3278; 4:ywPFW5koWYYAKXoEtdLWDD2YDi1E1wLDREnkP+XbUs?= =?us-ascii?Q?Z+scTd37KKeBOYyPx/XndL8HKT8QKrWqN9zbcISwqBiWwWQMPCT/O9hzgDhu?= =?us-ascii?Q?4sCSaDaJED0M8xHRbrAiwgTmQfr7bOTFmBHudtfmjT+7CbmmXE6x+9vPNIae?= =?us-ascii?Q?1cjN2vpFDK1Md2iZYlqY4Sb+f4aSYnUdiH+BNgVWp1LWwvIVjM+HQp9mrsgO?= =?us-ascii?Q?AUGuK7qUrECxwZMtFuZLeY8ANlherFBIyRGdxA3jzPQEg9VTNq3YluyOEhrl?= =?us-ascii?Q?Crt85MeXJgfxFBiwy4XyzU9qRYt15JY0TEi852VdE3VFOIoW0SzsUFdvJfPM?= =?us-ascii?Q?q7Qwer3jJmrhWMCibl5XyyxJoNe5xUAFWbiJ+qXjOr5ERe7zNu3cd614CMq1?= =?us-ascii?Q?FfbbC9qKuQXQtOAvUz24abQ2NLXB6W5m3Km+zHudaRohD1PYK3eSrLGDwNzu?= =?us-ascii?Q?zzlsK/QwSJDvdV16E6y4krdGB2QoN7I4c/TPeG12EQgvHd4AElEdJtTGS7Xh?= =?us-ascii?Q?arTUSYA0aWVgl7PYTm1kjpNUZgVzzDAC5v/keXq6qPkMk+8RAAaYBiCYSWv1?= =?us-ascii?Q?uOrz/F0kLcABl5Ocekixx68usrERZk7ZY/sL8Q44Y+aQisC3O3Yrpaoau96T?= =?us-ascii?Q?HkZXpFCP/uhX38FI+WT7zmoJdOe21vSzhcdhsnpN3nGYqGyJDozs1p0/Ff7t?= =?us-ascii?Q?TnnpIQaWAHzHofz+c37tGXYml2r+SYZjlTdx1LqNqlFMod3+v/Yegjt+mE1u?= =?us-ascii?Q?Ko1K7IrgKdexFOYhDxOEnbqh1c2G2TXU1iJdv+94Tl2HYF0hPgHh8fXhkPE8?= =?us-ascii?Q?j4MoqQjGfU0baIbarJHnWQb/V0N1/lNS9xxgZKSuNsc4W46EoXWVxudMQhOL?= =?us-ascii?Q?4yAQ9UJjXafv6cs+O20f+Y7BFML5iR4b/KTZ42pbnPP0J7LFFUyhNfAqW5kq?= =?us-ascii?Q?ll/qQbwHKEFMDGWNXNW2LtdfKTZDFGfYmjXIWsWHvj5/LnXuP4vw3ohU4Ebr?= =?us-ascii?Q?x3axrX/W5oclMDoEqAMpA7BJBy8XH/ysy7xyIFuwF4AFtm4LGlbJh0N8Neqb?= =?us-ascii?Q?5E+jlbmgO9AYZr/b4t0uogacamaF/p8kIiOXVBxmEZJyw8EHcfKWp0sbzUxJ?= =?us-ascii?Q?5PrSExL4YBn1asSt3mjGq2K/GLAq66d52EfO86Zpb+EzfW2kD9bleqck4aAQ?= =?us-ascii?Q?5Rtg2TFzecOgyhFkwBoGtfke0MSY1QO6sYAx0/9rCO3Oynbmnjqvb3/BA4Pq?= =?us-ascii?Q?glVTdsOhXQ/HSfZW1Vk+6nmAExcWeaEvkvlMMWWUlGpN53J8u23dqUUriHyS?= =?us-ascii?Q?9lhvxBgguIfYfFRg3X1Ek=3D?= X-Forefront-PRVS: 0345CFD558 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3278; 23:c3dRwBwGfwzE9r08+hrCCnbSYraEsB64BoJUh7u/n?= =?us-ascii?Q?T6VqmH5bB0QBDZ6FswEf/JxFSGNO85W586iV4ysoHaNvSK0n4vhWVloWnLQH?= =?us-ascii?Q?JMl+CyAPxGCTXvZTO+fhAnHrT4gp7xCjpX7dgK2BcYDpaecS1j5JqYGlrhUt?= =?us-ascii?Q?1F7gM06LWWixaKJHJwvL6zV0bHCI/zadMLrmFalmVWC4MKGLoWMn1LVuV9rw?= =?us-ascii?Q?69kLPH/oKRCgvFayXmoicB3RTkefanFMv7fuL2odViHok5PUcZboJ4CZH1aX?= =?us-ascii?Q?+fQ7x53vWx9SGASITOH3x481x/uzriCwUUAHQMH7v4HEQPX7+kaWuGt7jHap?= =?us-ascii?Q?A8/kVBdpJ+Pkmy35LyZ4IsZqLawra2jycJ1FX8PyXAAMeUg1t3d2yQA21sPv?= =?us-ascii?Q?r8K5dtvcJN2JonEFVATuSi5BKhveNliaGlqthqrGt2a3Ymu9E4MUdiGybdTy?= =?us-ascii?Q?WubceM1ps/+i1iy2b2T+JfheAJ9q+Ljvieru6JbWlw5RcuWw1yKV7xxUbGIn?= =?us-ascii?Q?K4Q78d3Ps0Pqhm2N1A2TGC8PxnCPHQPHHhTPVWsYZoE8IdC6fvP9fqqxwwPy?= =?us-ascii?Q?2QWyu7G2cnojSfSm9/zRgAW6pTdwFO8B05XCxAkfzq0eb6eN1R7m1TgajIXK?= =?us-ascii?Q?0OyhOrQIpsuZ3VGwQRGQmvA5qD43WBqqtxggy6kkqIkqaDTr7IaJStfPxMK9?= =?us-ascii?Q?D5e3NQ7Ls2P48w88GG2Vh6Mzk3rYMIzDq+PUEplOoLY/vE+Xz9cdGgl2jG6M?= =?us-ascii?Q?TUVz6KacouFXxfOODpsxZlIAQwLbl4jzW9j6q3alRm7hUo+o2qhxR/TFUypa?= =?us-ascii?Q?PGYnjxykME3PHhmHLJ4TEx3jyECCMgHzlVwCSstn9NgHO73jhRGm7bsTrIl3?= =?us-ascii?Q?T8GW+KePXP+e7jmmbuk+aDPK7xM9SB70mwqwAmyTjmL7lmjtK6UmenNjXFt4?= =?us-ascii?Q?+bHUhAeVa/Sig4hMjCKodoN42OLVwotWIXiV0JwGRYikCVnr32kna2kosGJi?= =?us-ascii?Q?3xTkhAJFZmIHvy6OdCGq0tjiT4rjvE4H76cfNWD/ZucS48HdEEmR8jupsNiq?= =?us-ascii?Q?NPwes1bA5K5PUPplrBISr/LMlZykMrcFVJ6zcHolNItDTavMeRLkyDhAMxfM?= =?us-ascii?Q?/l/xoS8k1lb/8d1eD/ZhA9rhJWgZIMYEyI83XEYCuQb6H9WeDrHo29fYzTFI?= =?us-ascii?Q?qxVS91Bw01RUj8grpvqVHWzeUJCQznq6m1Xm5o2R6fDxtCxMEPrGCB6qMZtj?= =?us-ascii?Q?1rvt/Y45jJpsxmGTsp50qfFr0ImaQhwPiclhBL6la39yL9hjTfWqslzgtMVl?= =?us-ascii?Q?TudftwvCwga+t7sQH4t1guubDcCTPgtEJBz0v+kKbb7IlFbArjFfjl2sBB0t?= =?us-ascii?Q?vs06SOqqJmhZ4/3JSqRdB/kSOY=3D?= X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3278; 6:b5vfmh6OVtZOKOTqjlO8StaFNzbhuriNJjZPKe9gFY?= =?us-ascii?Q?Kvi73yKw75RuGl3bKk74G5zTUEjl1jU2PUdBmic2UkFTcX3L6TTedeKN/Q5x?= =?us-ascii?Q?R9oWVEgNSV4zDKtVbOmLWR/olOg+xKeAhylZhcvh77JfCjfL71fzI6yL4but?= =?us-ascii?Q?Mh+VnQwTi9KWyGbBXMyotf5sfJEuApEJ/5VA4KdvgQ92cAnXomMzOEWls/kH?= =?us-ascii?Q?Ep+bXNUywqRomQJvhJ7tC7GE+0kY2j0jRRpkXTiOL5ZRxNRC5OFx+oRvF/tL?= =?us-ascii?Q?Ew8llBfMEAkrmhvkU1E2xxnzmtLm24WqKDRAF3GFRyYS3qa7vXKEuko/t0PG?= =?us-ascii?Q?wE2O2+hT6zPyytsMggAKy/2pb9obhgHlxokOoT4/KCczFlu4XI+oEDY1T6CC?= =?us-ascii?Q?L+hSTwDT+lBbx+KJBjHnLQY6Agj3DT7AY9BTl09TlS/UvHUrlrcAew1sMpnZ?= =?us-ascii?Q?wwh6frPvnovTZ8jQnWg1fUKMhWWwz2jT3NFHYfx0pv01oTqr/Zsww6S48dKo?= =?us-ascii?Q?Ru2Olnc7HLhNF/kYnN2RoT9KrIilF7/fEa4dAA6EXxbf8ffkSIUlgNuAOSk8?= =?us-ascii?Q?vlCELql/0IkKu4GemD3OHYP6X7TQnVVQy+h2wIqGEQFbMJMTPP3kJKX76sEN?= =?us-ascii?Q?3OVqFkJ6ufV621yuVnpEGvhUBvUv1ufgPxNfpN5oLhFlNhe+vMDIZA4bhNmX?= =?us-ascii?Q?yCsMoK0+Jr9HyiJ1nZCMaPSufjquqCIxpCwBd2X4nrRXA+tElQfkLparPMXs?= =?us-ascii?Q?+ZOs8NYKWZy7TRZpXoz5GGIJJAEjISKP2EEtt7DBLoJNq0zKz/TnMIqrOTow?= =?us-ascii?Q?dXi6DfobhSw5LN1jfRWV/G5WwbTYxWbes4ITL2hkYtmIEWDWct5wF4n64ZVI?= =?us-ascii?Q?KacKfUEUdjZunsfjmnFBYDst77s06azqb9mzYggXW8pfbmZMULmE1XBw3ASi?= =?us-ascii?Q?mOrkESRne/sBH0U2hJFPHD1GGVgoTJELB+2MP31QWt1JxfROuIuHvBNpWqGw?= =?us-ascii?Q?mD2obEHe0LBocK3v0WfSlt?= X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3278; 5:LuuzSpCaeHxeBsdh81nW94+qWnLlGAH1ShOMQvil69ko0VqQKErh6Rv4jnsXcQI16ifIePr3JL0Q8SjUoYTSMu6Rr3GFfbnuW7ceqgskizjaTG39qNRd0eWA5Og78rdi8mPdXVPSofJsUhwCNuJnVgsvqnEW7pH23EDnCeiub0pCcDv7t/0DmO88JbsRF2C2GXv8pVI140IE6tMQ1neZrV4s6ZNErhue02UJP1s3ZSr5kom7Ns/41FGuZmusz8UHtrLNBF4w1toptwWqblepDaNcnQelke8zRVk7EQYlm4dTKCn2t5eHgSeBbd+g1kk4LcARjp4E4DLvsv2OoBlKhh+i7EKUZDEgdnm4ZCJCNYXM4KrLDI6TwmTZxaIe2xYcnpAX8dGd3R5rKnSU5Qqx4oKyzKJB+berbkgGcKVL7qhCLtKm3lO7DMopAbFruEdlfkcmaFYAZ02PBGcbFrsfNln0bXcRaIAxmV96ErX62j0hWA1Qme67hPPqQtyEDTJo; 24:dQRa3kMVJj2N8C2SScsHVzo0uTPhkwiabHa+hRgQYVs/sJqLJtASxYhedehuixzM/87JkaMth3hHAPX0AUwrZJdz90zSyunO6zEPJEMnxRI= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3278; 7:E0rs1Ss6rx3Z2cS9dpRRYQqjuEA1eZCbhnUqpEFEt5mUb3XYfosw2HzHsPbTt8klzV/Sqo49dDJIpVMm8EIIB2x6VbhCQcW0KGrHx5L/EKjAXOytvQowcF8D6Z1KyE5uV2v+OH2hFn3DRYaVBkFZI0FdntU0sFZs4NPWkgIIydLsCuSitRy4SGBw/nR0g4vEGRcBSYvjfOuIzw6RTMbl2T8Kd0RRc55eihBN9IoEcYRWBwhPDZ63kID4NtBRZCd2An59iozh2ropoYU//odsC9+Dkq/ZjG6TpjB8PfBoiloTxDsjUf2Vgs4+xrgyVH0sTTLT0QH7fl1pARXaxhosVUV4amQ5t/e49Tcwocreh6KKQtEueQ+CjmqNI79wZqBODGnv5/EWsr/IjBB5FCli0Owizf52v/sF9vMw/8aNNHjAouVUSbnobznQ2eOrqu/DFdZDdoVABM4YGdMF9sWkKtqmDOpK4mF/PVn1QhQCW9OcpsWaa7pvuY3UAfGM75eXdamdLNdvOzYTJfLWmVr8rxag+EgMbSdEbPmBS6OEuF0dMcKKdV5ZGGKuAoy3NqFx54tInjrIwI9Mbwctp3R8beeg0/nAuVBJnQP6pwnSE+WnutY2VJkPzcl0atiHlJO4uiUiCWkNcTSdl7zuxZSD07f4RmrzsnLWH1e0CO9o+xCfjVicFg91IKoVFhhznAId1ZkDUATC60RQPyC4+M1WuzjXoKu2CpRzFpogDj4gKJNh91sZUuPN5107hTCk8FUmgPJCGsIKxu60e1gDinTpAKYv3o6Z88wMqNvJx/hq9yY= X-OriginatorOrg: ca.weatherford.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jun 2017 12:31:57.9353 (UTC) X-MS-Exchange-CrossTenant-Id: dd63fb60-07f6-4d96-8d40-ebeca61a524e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=dd63fb60-07f6-4d96-8d40-ebeca61a524e; Ip=[23.103.226.20]; Helo=[032-smtp-out.weatherford.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR03MB3278 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jun 2017 12:32:01 -0000 Pg0KPiBGcm9tOiBTdGV2ZW4gSGFydGxhbmQgW21haWx0bzpraWxsaW5nQG11bHRpcGxheS5jby51 a10NCj4gU2VudDogV2VkbmVzZGF5LCBKdW5lIDIxLCAyMDE3IDI6MDEgQU0NCj4gVG86IENhemEs IEFhcm9uOyBmcmVlYnNkLWZzQGZyZWVic2Qub3JnDQo+IFN1YmplY3Q6IFtFWFRFUk5BTF0gUmU6 IEZyZWVCU0QgMTEuMSBCZXRhIDIgWkZTIHBlcmZvcm1hbmNlIGRlZ3JhZGF0aW9uIG9uIFNTRHMN Cj4NCj4gT24gMjAvMDYvMjAxNyAyMToyNiwgQ2F6YSwgQWFyb24gd3JvdGU6DQoNCiBPbiAyMC8w Ni8yMDE3IDE3OjU4LCBDYXphLCBBYXJvbiB3cm90ZToNCg0KZFQ6IDEuMDAxcyAgdzogMS4wMDBz DQoNCiAgTChxKSAgb3BzL3MgICAgci9zICAga0JwcyAgIG1zL3IgICAgdy9zICAga0JwcyAgIG1z L3cgICAgZC9zICAga0JwcyAgIG1zL2QgICAlYnVzeSBOYW1lDQoNCiAgICAgMCAgIDQzMTggICA0 MzE4ICAzNDg2NSAgICAwLjAgICAgICAwICAgICAgMCAgICAwLjAgICAgICAwICAgICAgMCAgICAw LjAgICAxNC4yfCBhZGEwDQoNCiAgICAgMCAgIDQ0MDIgICA0NDAyICAzNTIxMyAgICAwLjAgICAg ICAwICAgICAgMCAgICAwLjAgICAgICAwICAgICAgMCAgICAwLjAgICAxNC40fCBhZGExDQoNCg0K DQpkVDogMS4wMDJzICB3OiAxLjAwMHMNCg0KICBMKHEpICBvcHMvcyAgICByL3MgICBrQnBzICAg bXMvciAgICB3L3MgICBrQnBzICAgbXMvdyAgICBkL3MgICBrQnBzICAgbXMvZCAgICVidXN5IE5h bWUNCg0KICAgICAxICAgNDI0OSAgIDQyNDkgIDM0MTM2ICAgIDAuMCAgICAgIDAgICAgICAwICAg IDAuMCAgICAgIDAgICAgICAwICAgIDAuMCAgIDE0LjF8IGFkYTANCg0KICAgICAwICAgNDM5MyAg IDQzOTMgIDM1Mjg3ICAgIDAuMCAgICAgIDAgICAgICAwICAgIDAuMCAgICAgIDAgICAgICAwICAg IDAuMCAgIDE0LjV8IGFkYTENCg0KWW91ICVidXN5IGlzIHZlcnkgbG93LCBzbyBzb3VuZHMgbGlr ZSB0aGUgYm90dGxlbmVjayBpc24ndCBpbiByYXcgZGlzayBwZXJmb3JtYW5jZSBidXQgc29tZXdo ZXJlIGVsc2UuDQoNCg0KDQpNaWdodCBiZSBpbnRlcmVzdGluZyB0byBzZWUgaWYgYW55dGhpbmcg c3RhbmRzIG91dCBpbiB0b3AgLVN6IGFuZCB0aGVuIHByZXNzIGggZm9yIHRocmVhZHMuDQoNCg0K DQoNCg0KSSByZWJvb3RlZCB0aGUgc3lzdGVtIHRvIGRpc2FibGUgVHJpbSBzbyBjdXJyZW50bHkg bm90IGRlZ3JhZGVkLg0KDQoNCg0KZFQ6IDEuMDAxcyAgdzogMS4wMDBzDQoNCiBMKHEpICBvcHMv cyAgICByL3MgICBrQnBzICAgbXMvciAgICB3L3MgICBrQnBzICAgbXMvdyAgICBkL3MgICBrQnBz ICAgbXMvZCAgICVidXN5IE5hbWUNCg0KICAgIDMgICAzODg3ICAgMzg4NyA0MjY1MTQgICAgMC43 ICAgICAgMCAgICAgIDAgICAgMC4wICAgICAgMCAgICAgIDAgICAgMC4wICAgOTAuN3wgYWRhMA0K DQogICAgMyAgIDM5ODcgICAzOTg3IDQzNDcwMiAgICAwLjcgICAgICAwICAgICAgMCAgICAwLjAg ICAgICAwICAgICAgMCAgICAwLjAgICA5Mi4wfCBhZGExDQoNCg0KDQpkVDogMS4wMDJzICB3OiAx LjAwMHMNCg0KIEwocSkgIG9wcy9zICAgIHIvcyAgIGtCcHMgICBtcy9yICAgIHcvcyAgIGtCcHMg ICBtcy93ICAgIGQvcyAgIGtCcHMgICBtcy9kICAgJWJ1c3kgTmFtZQ0KDQogICAgMyAgIDM5NTgg ICAzOTU4IDQzMzU2MyAgICAwLjcgICAgICAwICAgICAgMCAgICAwLjAgICAgICAwICAgICAgMCAg ICAwLjAgICA5MS42fCBhZGEwDQoNCiAgICAzICAgMzk4OSAgIDM5ODkgNDM4NDE3ICAgIDAuNyAg ICAgIDAgICAgICAwICAgIDAuMCAgICAgIDAgICAgICAwICAgIDAuMCAgIDkzLjB8IGFkYTENCg0K DQoNCnRlc3RAZjExMWJldGEyOn4gIyBkZCBpZj0vdGVzdGRiL3Rlc3Qgb2Y9L2Rldi9udWxsIGJz PTFtDQoNCjE2MDAwKzAgcmVjb3JkcyBpbg0KDQoxNjAwMCswIHJlY29yZHMgb3V0DQoNCjE2Nzc3 MjE2MDAwIGJ5dGVzIHRyYW5zZmVycmVkIGluIDE5LjM4NTg1NSBzZWNzICg4NjU0MzU5NTkgYnl0 ZXMvc2VjKQ0KPiBOb3cgdGhhdCBpcyBpbnRlcmVzdGluZywgYXMgeW91ciBnZXR0aW5nIHNtYWxs ZXIgbnVtYmVyIG9wcy9zIGJ1dCBtdWNoIGhpZ2hlciB0aHJvdWdocHV0Lg0KPg0KPiBJbiB0aGUg bm9ybWFsIGNhc2UgeW91J3JlIHNlZWluZyB+MTA4S2IgcGVyIHJlYWQgd2hlcmUgaW4gdGhlIGRl Z3JhZGVkIGNhc2UgeW91J3JlIHNlZWluZyA4S2IgcGVyIHJlYWQuDQo+DQo+IEdpdmVuIHRoaXMg YW5kIGtub3dpbmcgdGhlIGFwcGxpY2F0aW9uIGxldmVsIGlzbid0IGVmZmVjdGluZyBpdCwgd2Ug bmVlZCB0byBpZGVudGlmeSB3aGVyZSBpbiB0aGUgSU8gc3RhY2sgdGhlIHJlYWRzIGFyZSBnZXR0 aW5nIGxpbWl0ZWQgdG8gOEtiPw0KPg0KPiBXaXRoIHlvdXIgYWRkaXRpb25hbCBpbmZvcm1hdGlv biBhYm91dCBBUkMsIGl0IGNvdWxkIGJlIHRoYXQgdGhlIGxpbWl0ZWQgbWVtb3J5IGlzIHRoZSBj YXVzZS4NCj4NCj4gICAgUmVnYXJkcw0KPiAgICBTdGV2ZQ0KDQpJ4oCZbSBnbGFkIHRvIGxlYXJu IHRoYXQgdGhlIGFib3ZlIGluZm8gaXMgb2Ygc29tZSB1c2UuICBUaGUgNTBNIGxpbWl0IEkgcHJl dmlvdXNseSB1c2VkIGZvciB0aGUgWkZTIEFDUyBzZXJ2ZWQgbWUgd2VsbCBmb3IgdGhlIHBhc3Qg c2V2ZXJhbCB5ZWFycy4gIEFuZCwgaW4gZmFjdCwgSSB0aG91Z2h0IGl0IHdhcyBzdGlsbCB3b3Jr aW5nIHdlbGwgYW5kIG9ubHkgYWNjaWRlbnRhbGx5IHN0dW1ibGVkIG92ZXIgdGhlIHBlcmZvcm1h bmNlIGRyb3Agd2hlbiB0ZXN0aW5nIHNvbWUgSW50ZWwgNTQwIFNTRHMgd2hpY2ggd2VyZSB3b3Jr aW5nIHN1cnByaXNpbmdseSBzbmFwcGlseSBkZXNwaXRlIHVzaW5nIFRMQyBOQU5EIGZsYXNoLiAg SW5pdGlhbGx5LCBJIHNhdyBhIHNpbXBsZSBTUUwgcXVlcnkgaW4gUG9zdGdyZXMgZ28gZnJvbSB+ MzUgc2Vjb25kcyB0byB+NjM1IHNlY29uZHMgYW5kIHN1c3BlY3RlZCB0aGUgSW50ZWwgNTQwcyB3 ZXJlIHRoZSBjYXVzZS4gIFR1cm5zIG91dCBpdCB3YXMgbWUgaGFtc3RyaW5naW5nIHRoZSBBUkMg dGhhdCB3YXMgdG8gYmxhbWUuICBUaGF0IHNhaWQsIGl04oCZcyBpbnRlcmVzdGluZyB0aGF0LCB1 c2luZyB0aGUgR0VPTSBFTEkgbGF5ZXIgZm9yIDRrIHNlY3RvciBlbXVsYXRpb24sIGl0IHJ1bnMg ZmluZSBmb3Igc2V2ZXJhbCBob3VycyBiZWZvcmUgcGVyZm9ybWFuY2UgZHJvcHMgb2ZmIHdoZW4g dGhlIEFSQyBpcyBzZXQgdG9vIHNtYWxsIGdvaW5nIGZyb20gODUwTUIvcyBkb3duIHRvIDgwTUIv cy4gIEluIHRoZSBjYXNlIG9mIGFzaGlmdD0xMiwgdGhlIGluaXRpYWwgcGVyZm9ybWFuY2UgaW1w YWN0IGlzIGltbWVkaWF0ZSBvbiBib290dXAgZ29pbmcgZnJvbSA4NTBNQi9zIHdpdGggZGVmYXVs dCBBUkMgc2V0dGluZ3MgZG93biB0byA0NTBNQi9zIHdpdGggQVJDIHNldCB0byA1ME0uICBUaGVu LCBzb21lIGhvdXJzIGxhdGVyLCBkcm9wcGluZyBkb3duIHRvIH43ME1CL3MuDQoNCldpdGggcmVn YXJkcyB0byBUcmltLCB0aGVyZSB3ZXJlIGEgbnVtYmVyIG9mIHN1Z2dlc3Rpb25zIHRvIGRpc2Fi bGUgaXQuICBNeSB1bmRlcnN0YW5kaW5nIGlzIHRoYXQgVHJpbSBzdXBwb3J0IGlzIGhpZ2hseSBk ZXNpcmFibGUgdG8gbWFpbnRhaW4gcGVhayBwZXJmb3JtYW5jZSBidXQgaXQgc2VlbXMgaXTigJlz IHRoZSAjMSBzdXNwZWN0IHdoZW4gdGhlcmXigJlzIGEgcGVyZm9ybWFuY2UgZHJvcC4gIElzIGl0 IHRoYXQgcHJvYmxlbWF0aWM/ICBJ4oCZbSBjb25zaWRlcmluZyBzd2l0Y2hpbmcgZnJvbSBHRU9N IEVMSSB0byBhc2hpZnQ9MTIgZm9yIG15IDRrIHNlY3RvciBlbXVsYXRpb24gaW4gb3JkZXIgdG8g Z2V0IFRyaW0gYnV0IGlmIGl04oCZcyBvZiBubyBiZW5lZml0IHRoZW4gdGhlcmXigJlzIG5vdCBt dWNoIHBvaW50Lg0KDQpSZWdhcmRzLA0KQWFyb24NClRoaXMgbWVzc2FnZSBtYXkgY29udGFpbiBj b25maWRlbnRpYWwgYW5kIHByaXZpbGVnZWQgaW5mb3JtYXRpb24uIElmIGl0IGhhcyBiZWVuIHNl bnQgdG8geW91IGluIGVycm9yLCBwbGVhc2UgcmVwbHkgdG8gYWR2aXNlIHRoZSBzZW5kZXIgb2Yg dGhlIGVycm9yIGFuZCB0aGVuIGltbWVkaWF0ZWx5IGRlbGV0ZSBpdC4gSWYgeW91IGFyZSBub3Qg dGhlIGludGVuZGVkIHJlY2lwaWVudCwgZG8gbm90IHJlYWQsIGNvcHksIGRpc2Nsb3NlIG9yIG90 aGVyd2lzZSB1c2UgdGhpcyBtZXNzYWdlLiBUaGUgc2VuZGVyIGRpc2NsYWltcyBhbnkgbGlhYmls aXR5IGZvciBzdWNoIHVuYXV0aG9yaXplZCB1c2UuIFBMRUFTRSBOT1RFIHRoYXQgYWxsIGluY29t aW5nIGUtbWFpbHMgc2VudCB0byBXZWF0aGVyZm9yZCBlLW1haWwgYWNjb3VudHMgd2lsbCBiZSBh cmNoaXZlZCBhbmQgbWF5IGJlIHNjYW5uZWQgYnkgdXMgYW5kL29yIGJ5IGV4dGVybmFsIHNlcnZp Y2UgcHJvdmlkZXJzIHRvIGRldGVjdCBhbmQgcHJldmVudCB0aHJlYXRzIHRvIG91ciBzeXN0ZW1z LCBpbnZlc3RpZ2F0ZSBpbGxlZ2FsIG9yIGluYXBwcm9wcmlhdGUgYmVoYXZpb3IsIGFuZC9vciBl bGltaW5hdGUgdW5zb2xpY2l0ZWQgcHJvbW90aW9uYWwgZS1tYWlscyAoc3BhbSkuIFRoaXMgcHJv Y2VzcyBjb3VsZCByZXN1bHQgaW4gZGVsZXRpb24gb2YgYSBsZWdpdGltYXRlIGUtbWFpbCBiZWZv cmUgaXQgaXMgcmVhZCBieSBpdHMgaW50ZW5kZWQgcmVjaXBpZW50IGF0IG91ciBvcmdhbml6YXRp b24uIE1vcmVvdmVyLCBiYXNlZCBvbiB0aGUgc2Nhbm5pbmcgcmVzdWx0cywgdGhlIGZ1bGwgdGV4 dCBvZiBlLW1haWxzIGFuZCBhdHRhY2htZW50cyBtYXkgYmUgbWFkZSBhdmFpbGFibGUgdG8gV2Vh dGhlcmZvcmQgc2VjdXJpdHkgYW5kIG90aGVyIHBlcnNvbm5lbCBmb3IgcmV2aWV3IGFuZCBhcHBy b3ByaWF0ZSBhY3Rpb24uIElmIHlvdSBoYXZlIGFueSBjb25jZXJucyBhYm91dCB0aGlzIHByb2Nl c3MsIHBsZWFzZSBjb250YWN0IHVzIGF0IGRhdGFwcml2YWN5QHdlYXRoZXJmb3JkLmNvbS4NCg== From owner-freebsd-fs@freebsd.org Wed Jun 21 14:59:29 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE1FFD91569; Wed, 21 Jun 2017 14:59:29 +0000 (UTC) (envelope-from madpilot@FreeBSD.org) Received: from mail.madpilot.net (grunt.madpilot.net [78.47.145.38]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 810E06FCB6; Wed, 21 Jun 2017 14:59:28 +0000 (UTC) (envelope-from madpilot@FreeBSD.org) Received: from mail (mail [192.168.254.3]) by mail.madpilot.net (Postfix) with ESMTP id 3wt7CX5DQkzZqp; Wed, 21 Jun 2017 16:59:20 +0200 (CEST) Received: from mail.madpilot.net ([192.168.254.3]) by mail (mail.madpilot.net [192.168.254.3]) (amavisd-new, port 10024) with ESMTP id jmuBgyeqZgba; Wed, 21 Jun 2017 16:59:15 +0200 (CEST) Received: from marvin.madpilot.net (micro.madpilot.net [88.149.173.206]) by mail.madpilot.net (Postfix) with ESMTPSA; Wed, 21 Jun 2017 16:59:15 +0200 (CEST) Subject: vnode_pager_generic_getpages_done: I/O read error 5 caused by r318394 (was Re: FreeBSD 11.1-BETA1 Now Available) To: Peter Blok , Ed Maste , FreeBSD-STABLE Mailing List Cc: Glen Barber , Jonathan Chen , FreeBSD FS References: <20170610123435.GB69235@FreeBSD.org> <10A08FC1-C84E-4D06-9360-B7C3848F4680@bsd4all.org> From: Guido Falsi Message-ID: Date: Wed, 21 Jun 2017 16:59:14 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: <10A08FC1-C84E-4D06-9360-B7C3848F4680@bsd4all.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jun 2017 14:59:29 -0000 On 06/13/17 13:44, Peter Blok wrote: > Hi, > > For a while now, I’m not able to build a RPI1-B image from -stable. I have narrowed it dow to fix 318394, which adds a refresh option to geom_label. If I undo this fix in today’s stable it works ok. If I don’t I’m getting continuously: > > vm_fault: pager read error, pid 1 (init) > vnode_pager_generic_getpages_done: I/O read error 5 > > I have looked at the fix and I can’t figure out why it breaks the code. > > And yes I have tried various other SD cards - they all have the same issue. > Hi, I'm seeing similar symptoms with NanoBSD images on PCEngines ALIX and APU2 boards, using compactflash and SD card storage respectively. The problem has appeared as soon as I started testing 11.1-BETA1 from the stable branch. Problem appears when I update the image, using a slightly modified version of the standard nanobsd update and updatep[12] scripts. My changes are not in the dd/gpart commands though, which are the same. gpart seems the most likely candidate though. I have just discovered this thread and I will test reverting r318394 soon. Thanks to Peter for narrowing it down! Maybe this is related to having the disks mounted read-only? Should I open a bug report to properly track this down? P.S. CCing freebsd-fs@ -- Guido Falsi From owner-freebsd-fs@freebsd.org Wed Jun 21 15:11:05 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C6251D91A9D for ; Wed, 21 Jun 2017 15:11:05 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x235.google.com (mail-io0-x235.google.com [IPv6:2607:f8b0:4001:c06::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8B317704AA for ; Wed, 21 Jun 2017 15:11:05 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x235.google.com with SMTP id k93so5533814ioi.2 for ; Wed, 21 Jun 2017 08:11:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=IbuUg/rCLudVHHP8Af5vByu0e5vJKiGciUPuIh/eBLk=; b=YEYEA6cYAeA6P0pEi5T7yKlyd6baH1Sd4oQQ1D9jqEt24KQyAWdnllpiMXZ2Cq5uoK akfCe+1y4wjcxnbMhI2bXCcA/VS5QrRD1zXNEkOL20jb8oUnq6GqEqePah4KsiilrX+F L5X9XIOfuC+Ics3BYdiDUNY89vanatV3GzIXIvmMWMJA+SCwAb/mCOiH1/+oWJ0lcfMH ImzC+0V6fsn7kCvrP5nV7wW7Y+He4rMwDBTlP8FDQp8aRpE01LPfK5ZCAgetHsVCMvgl RRkVPwx9hSqU/heljjFhEmNegnAt4jRqVrlPiGNdH19C0GRv77xZg0FomuTlfib1F5Rr Hncg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=IbuUg/rCLudVHHP8Af5vByu0e5vJKiGciUPuIh/eBLk=; b=oHHBxsZZvew35Jol//Xmlsue08n1buI+8pNA5+bWo1SWB2a+OLQSaD83QmeGxg9sdQ DR+XChDQLJsJBHY9wZ7iElNwe4h9Xq3Y4bMx4bQbv03tGtb1Ovwe4Razbc9Mj9Pgb/PF 7B3PbudN6Lhr/YWBw9nhUG0x0dem1fRqWq/ikfSgcQn1FYvAcN/VeSBhYujMIR6KzieL 2JunejZQ3VdWpS+AcY5tSadZewwFL20Cnh8u91oESopnO9bzu2OMADQ7tQCTwJfwK/8i rNzt3rGsqqoAg2z33ObcsiZOjiuhhZiXzUYTIM/4zDAlPOCBdmL9dC3NISzgFFRJ0eCx ClJg== X-Gm-Message-State: AKS2vOzQlUp/egBIqj5Oe9clKxg9mBxRlmV8vLb4/rp6zcnp0uq2j9aD HY2SK4l46DDCfYXmXGX+dc6GcLOK+zNi X-Received: by 10.107.170.213 with SMTP id g82mr36489845ioj.148.1498057864791; Wed, 21 Jun 2017 08:11:04 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.192.69 with HTTP; Wed, 21 Jun 2017 08:11:04 -0700 (PDT) X-Originating-IP: [12.27.65.223] In-Reply-To: <7dbee74c0071463db4fbb0a5de2e7244@DM2PR58MB013.032d.mgd.msft.net> References: <86bf6fad-977a-b096-46b9-e9099a57a1f4@multiplay.co.uk> <7dbee74c0071463db4fbb0a5de2e7244@DM2PR58MB013.032d.mgd.msft.net> From: Warner Losh Date: Wed, 21 Jun 2017 09:11:04 -0600 X-Google-Sender-Auth: YuojNNKdLWKc_Np5NzjfSZvaH-4 Message-ID: Subject: Re: [EXTERNAL] Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs To: "Caza, Aaron" Cc: Steven Hartland , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jun 2017 15:11:05 -0000 On Wed, Jun 21, 2017 at 6:31 AM, Caza, Aaron wrote: > > > > From: Steven Hartland [mailto:killing@multiplay.co.uk] > > Sent: Wednesday, June 21, 2017 2:01 AM > > To: Caza, Aaron; freebsd-fs@freebsd.org > > Subject: [EXTERNAL] Re: FreeBSD 11.1 Beta 2 ZFS performance degradation > on SSDs > > > > On 20/06/2017 21:26, Caza, Aaron wrote: > > On 20/06/2017 17:58, Caza, Aaron wrote: > > dT: 1.001s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps > ms/d %busy Name > > 0 4318 4318 34865 0.0 0 0 0.0 0 0 > 0.0 14.2| ada0 > > 0 4402 4402 35213 0.0 0 0 0.0 0 0 > 0.0 14.4| ada1 > > > > dT: 1.002s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps > ms/d %busy Name > > 1 4249 4249 34136 0.0 0 0 0.0 0 0 > 0.0 14.1| ada0 > > 0 4393 4393 35287 0.0 0 0 0.0 0 0 > 0.0 14.5| ada1 > > You %busy is very low, so sounds like the bottleneck isn't in raw disk > performance but somewhere else. > > > > Might be interesting to see if anything stands out in top -Sz and then > press h for threads. > > > > > > I rebooted the system to disable Trim so currently not degraded. > > > > dT: 1.001s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps > ms/d %busy Name > > 3 3887 3887 426514 0.7 0 0 0.0 0 0 > 0.0 90.7| ada0 > > 3 3987 3987 434702 0.7 0 0 0.0 0 0 > 0.0 92.0| ada1 > > > > dT: 1.002s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps > ms/d %busy Name > > 3 3958 3958 433563 0.7 0 0 0.0 0 0 > 0.0 91.6| ada0 > > 3 3989 3989 438417 0.7 0 0 0.0 0 0 > 0.0 93.0| ada1 > > > > test@f111beta2:~ # dd if=3D/testdb/test of=3D/dev/null bs=3D1m > > 16000+0 records in > > 16000+0 records out > > 16777216000 bytes transferred in 19.385855 secs (865435959 bytes/sec) > > Now that is interesting, as your getting smaller number ops/s but much > higher throughput. > > > > In the normal case you're seeing ~108Kb per read where in the degraded > case you're seeing 8Kb per read. > > > > Given this and knowing the application level isn't effecting it, we nee= d > to identify where in the IO stack the reads are getting limited to 8Kb? > > > > With your additional information about ARC, it could be that the limite= d > memory is the cause. > > > > Regards > > Steve > > I=E2=80=99m glad to learn that the above info is of some use. The 50M li= mit I > previously used for the ZFS ACS served me well for the past several years= . > And, in fact, I thought it was still working well and only accidentally > stumbled over the performance drop when testing some Intel 540 SSDs which > were working surprisingly snappily despite using TLC NAND flash. > Initially, I saw a simple SQL query in Postgres go from ~35 seconds to ~6= 35 > seconds and suspected the Intel 540s were the cause. Turns out it was me > hamstringing the ARC that was to blame. That said, it=E2=80=99s interest= ing that, > using the GEOM ELI layer for 4k sector emulation, it runs fine for severa= l > hours before performance drops off when the ARC is set too small going fr= om > 850MB/s down to 80MB/s. In the case of ashift=3D12, the initial performa= nce > impact is immediate on bootup going from 850MB/s with default ARC setting= s > down to 450MB/s with ARC set to 50M. Then, some hours later, dropping do= wn > to ~70MB/s. > Yes. TLC drives typically have some amount, maybe 3% of their NAND configured as SLC NAND. for a 1TB drive, this would be 30GB. This SLC cache is super fast compared to the TLC parts. What's typically done is that the writes land in the SLC part and are then moved to the TLC parts as device bandwidth allows. If you overrun this landing pad, you are stuck with TLC write performance, which is going to be about ~1/10th that of SLC. > With regards to Trim, there were a number of suggestions to disable it. > My understanding is that Trim support is highly desirable to maintain pea= k > performance but it seems it=E2=80=99s the #1 suspect when there=E2=80=99s= a performance > drop. Is it that problematic? I=E2=80=99m considering switching from GE= OM ELI to > ashift=3D12 for my 4k sector emulation in order to get Trim but if it=E2= =80=99s of no > benefit then there=E2=80=99s not much point. > TRIM is highly desired to keep the write amplification on the drives down, which is often more of a longevity thing than a performance thing (though if you are write constrained, it may help a little). There's some drives that don't handle the TRIM quickly enough, which is why it's often implicated in performance issues. TRIM generally helps the drive make better decisions and copy less data around (thus reducing the number of non-user initiated writes, which drives write amp), though for some work loads it doesn't help much (especially ones where any files deleted are immediately replaced). The benefits from it are that your drives don't wear as quickly as a result. Warner From owner-freebsd-fs@freebsd.org Wed Jun 21 15:22:31 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B2477D9222B for ; Wed, 21 Jun 2017 15:22:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A06B170D88 for ; Wed, 21 Jun 2017 15:22:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5LFMVHj050677 for ; Wed, 21 Jun 2017 15:22:31 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220185] >> operator does not append in Fuse mounts Date: Wed, 21 Jun 2017 15:22:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jun 2017 15:22:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220185 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Jun 21 15:58:59 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9E92D92AFC for ; Wed, 21 Jun 2017 15:58:59 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wr0-x22a.google.com (mail-wr0-x22a.google.com [IPv6:2a00:1450:400c:c0c::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1C61971C39 for ; Wed, 21 Jun 2017 15:58:58 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wr0-x22a.google.com with SMTP id c11so85309218wrc.3 for ; Wed, 21 Jun 2017 08:58:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=UerZMAbp4fjfluopAJ7M9kd85pZ+HXuHrDXfLCHAjI8=; b=xdjUECRIgsXfEYvT0TWAX5qjKR/PbzShlrynWlIlSrsFWo+30z+lBZz05/bG4sLgXN 1O9Ugw609tAqxPngzBatuCOg7Tcdk1BYH5ebRP1SzOlIW7Lz0qoCiafC4G2QjsT57cjz 2nol6H9hw99WQgvvOCmJlHHw9jDGtBVNxAVRLTh+osc/2433Fbz2E4Gy2Yu7kEXLyATz eHE+D2Yg4THISZdHGHbQ9iPDaJlCou+xlmp2BcxOYE7Jy/OzQnIdknpcHzuhUVLWi/v+ KNwwQUepSUB9sGpH2KkkzKLpEEvfzB57vTR9KJ9CgjwlaVsURBJTNo0RpdfneeylHgLz Crnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=UerZMAbp4fjfluopAJ7M9kd85pZ+HXuHrDXfLCHAjI8=; b=kefuaqIc4emmO25ZxMitspRstxpZFexOClqe7+IBfLcjgdFv5gvXisiNnFttaSKeRh w9XPo0YDfZ+Wi0KKTt87aeLjYNlwRrvrUUoiyfUYAf8asdN06mjJvGDFGPQJ6UIz+QsX n6XdsMIKdHFPAHfan8QiE2JU9//caimVnDMU/aBnQgzv2tCRtEfmuh3BjEevJOU4DPoJ iYLTQCA5H6Foyrrl+7bkFS8zjUo80wmyMmfwHxFw/Vrujx2KfNqgP3KGtjaDcMtsi+MN EPKouocRo6AGWB/uONxL6ZpPlt6P4iS/wnou8T64KyuoU1Q6+/GectN6o03cxWfiYXO2 7AZA== X-Gm-Message-State: AKS2vOwaccuFjyF01meC27l15ddueN0c4f2lyV5SOZR5vFz76+CUvmHi 80vFcxyECYXKg8scb+jUMQ== X-Received: by 10.28.66.196 with SMTP id k65mr6959604wmi.55.1498060736644; Wed, 21 Jun 2017 08:58:56 -0700 (PDT) Received: from [10.10.1.111] ([185.97.61.15]) by smtp.gmail.com with ESMTPSA id w96sm1023387wrc.33.2017.06.21.08.58.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Jun 2017 08:58:55 -0700 (PDT) Subject: Re: [EXTERNAL] Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs To: "Caza, Aaron" , "freebsd-fs@freebsd.org" References: <86bf6fad-977a-b096-46b9-e9099a57a1f4@multiplay.co.uk> <7dbee74c0071463db4fbb0a5de2e7244@DM2PR58MB013.032d.mgd.msft.net> From: Steven Hartland Message-ID: Date: Wed, 21 Jun 2017 16:58:55 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: <7dbee74c0071463db4fbb0a5de2e7244@DM2PR58MB013.032d.mgd.msft.net> Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jun 2017 15:58:59 -0000 On 21/06/2017 13:31, Caza, Aaron wrote: > > > > > > *From:*Steven Hartland [mailto:killing@multiplay.co.uk] > > *Sent:*Wednesday, June 21, 2017 2:01 AM > > *To:*Caza, Aaron; freebsd-fs@freebsd.org > > *Subject:*[EXTERNAL] Re: FreeBSD 11.1 Beta 2 ZFS performance > degradation on SSDs > > > > > > On 20/06/2017 21:26, Caza, Aaron wrote: > > On 20/06/2017 17:58, Caza, Aaron wrote: > > dT: 1.001s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > > 0 4318 4318 34865 0.0 0 0 0.0 0 0 0.0 14.2| ada0 > > 0 4402 4402 35213 0.0 0 0 0.0 0 0 0.0 14.4| ada1 > > dT: 1.002s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > > 1 4249 4249 34136 0.0 0 0 0.0 0 0 0.0 14.1| ada0 > > 0 4393 4393 35287 0.0 0 0 0.0 0 0 0.0 14.5| ada1 > > You %busy is very low, so sounds like the bottleneck isn't in raw disk performance but somewhere else. > > Might be interesting to see if anything stands out in top -Sz and then press h for threads. > > I rebooted the system to disable Trim so currently not degraded. > > dT: 1.001s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > > 3 3887 3887 426514 0.7 0 0 0.0 0 0 0.0 90.7| ada0 > > 3 3987 3987 434702 0.7 0 0 0.0 0 0 0.0 92.0| ada1 > > dT: 1.002s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name > > 3 3958 3958 433563 0.7 0 0 0.0 0 0 0.0 91.6| ada0 > > 3 3989 3989 438417 0.7 0 0 0.0 0 0 0.0 93.0| ada1 > > test@f111beta2:~ # dd if=/testdb/test of=/dev/null bs=1m > > 16000+0 records in > > 16000+0 records out > > 16777216000 bytes transferred in 19.385855 secs (865435959 bytes/sec) > > > Now that is interesting, as your getting smaller number ops/s but > much higher throughput. > > > > In the normal case you're seeing ~108Kb per read where in the > degraded case you're seeing 8Kb per read. > > > > Given this and knowing the application level isn't effecting it, we > need to identify where in the IO stack the reads are getting limited > to 8Kb? > > > > With your additional information about ARC, it could be that the > limited memory is the cause. > > > > Regards > > Steve > > I’m glad to learn that the above info is of some use. The 50M limit I > previously used for the ZFS ACS served me well for the past several > years. And, in fact, I thought it was still working well and only > accidentally stumbled over the performance drop when testing some > Intel 540 SSDs which were working surprisingly snappily despite using > TLC NAND flash. Initially, I saw a simple SQL query in Postgres go > from ~35 seconds to ~635 seconds and suspected the Intel 540s were the > cause. Turns out it was me hamstringing the ARC that was to blame. > That said, it’s interesting that, using the GEOM ELI layer for 4k > sector emulation, it runs fine for several hours before performance > drops off when the ARC is set too small going from 850MB/s down to > 80MB/s. In the case of ashift=12, the initial performance impact is > immediate on bootup going from 850MB/s with default ARC settings down > to 450MB/s with ARC set to 50M. Then, some hours later, dropping down > to ~70MB/s. > > With regards to Trim, there were a number of suggestions to disable > it. My understanding is that Trim support is highly desirable to > maintain peak performance but it seems it’s the #1 suspect when > there’s a performance drop. Is it that problematic? I’m considering > switching from GEOM ELI to ashift=12 for my 4k sector emulation in > order to get Trim but if it’s of no benefit then there’s not much point. > The GEOM hack is just that, you can actually remove the device once you've created the pool and it will have no effect. From owner-freebsd-fs@freebsd.org Thu Jun 22 00:33:40 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BBC93D9C189 for ; Thu, 22 Jun 2017 00:33:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A9BE183F31 for ; Thu, 22 Jun 2017 00:33:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5M0XeSh096307 for ; Thu, 22 Jun 2017 00:33:40 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219972] Unable to zpool export following some zfs recv Date: Thu, 22 Jun 2017 00:33:40 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: pfribeiro@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jun 2017 00:33:40 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219972 --- Comment #3 from pfribeiro@gmail.com --- It seems the ability to reproduce this bug relies on there being more than = one CPU core, relatively unused, on the same system. In the case of my original system, if I run 'dd if=3D/dev/zero of=3D/dev/null' while I try the steps m= entioned in comment #1, then the bug is not reproducible. However, as soon as this command is killed, the bug is reproducible. On the forum thread linked to in comment #2, another user tried to reproduce the problem, but without success. I then configured two FreeBSD VMs on ESXi (host is a Xeon-D 1518, 4 core, with HyperThreading enabled), one running t= he VMDK image provided on the official FreeBSD download page, and another installed from the ISO image (with ZFS as root filesystem) to better mirror= my installation on my original system, an Intel NUC5CPYH. Initially, I also co= uld not reproduce the bug in the VMs, no matter how many times I tried. However, having observed the behaviour on the NUC installation, I then proceeded to change the CPU affinity on ESXi, so that the VM is allocated logical cores 0,2, and has minimum 1000MHz reserved. By running the import/export of 'slave' multiple times (after the respective zfs send/recv= ), I was eventually able to trigger this on the 56th run of import/export. Regarding the NUC installation, I can see that killing 'dd if=3D/dev/zero of=3D/dev/null' (ie, making the other core widely available) is only releva= nt for the import/export of 'slave' after the 'zfs send | recv' has taken place, w= hich suggests that there is a race-condition of sorts in the zpool export ioctl = code (which somehow relies on a previous recv). I would be happy to provide more diagnostics, however I would need further guidance from you, as I am not very familiar with the synchronization primitives of FreeBSD. I believe this bug will be hard to track down. I would also like to add that I have successfully (following the above cave= ats) reproduced this bug under more than one platform, with the following versio= ns, all on amd64: 11.0-RELEASE-p1 #0 r306420 11.0-RELEASE-p9 #0: Tue Apr 11 08:48:40 UTC 2017 11.1-BETA2 #0 r320072: Sun Jun 18 18:45:14 BST 2017 (I compiled my own kern= el with debugging on) Finally, for completeness my rudimentary test bash script is available at: https://pastebin.com/YcKSU1LA. You're welcome to check the related forum po= st from comment #2 as well. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Jun 22 08:26:28 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AAA20D86348; Thu, 22 Jun 2017 08:26:28 +0000 (UTC) (envelope-from madpilot@FreeBSD.org) Received: from mail.madpilot.net (grunt.madpilot.net [78.47.145.38]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6CA3972D19; Thu, 22 Jun 2017 08:26:28 +0000 (UTC) (envelope-from madpilot@FreeBSD.org) Received: from mail (mail [192.168.254.3]) by mail.madpilot.net (Postfix) with ESMTP id 3wtZRj3DcgzZqg; Thu, 22 Jun 2017 10:26:25 +0200 (CEST) Received: from mail.madpilot.net ([192.168.254.3]) by mail (mail.madpilot.net [192.168.254.3]) (amavisd-new, port 10024) with ESMTP id C_j4GNIDJruJ; Thu, 22 Jun 2017 10:26:23 +0200 (CEST) Received: from marvin.madpilot.net (micro.madpilot.net [88.149.173.206]) by mail.madpilot.net (Postfix) with ESMTPSA; Thu, 22 Jun 2017 10:26:23 +0200 (CEST) Subject: Re: vnode_pager_generic_getpages_done: I/O read error 5 caused by r318394 (was Re: FreeBSD 11.1-BETA1 Now Available) From: Guido Falsi To: Peter Blok , Ed Maste , FreeBSD-STABLE Mailing List Cc: FreeBSD FS , Glen Barber , Jonathan Chen References: <20170610123435.GB69235@FreeBSD.org> <10A08FC1-C84E-4D06-9360-B7C3848F4680@bsd4all.org> Message-ID: <05e6bd02-3582-aea1-5fc3-19caa4073f94@FreeBSD.org> Date: Thu, 22 Jun 2017 10:26:23 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jun 2017 08:26:28 -0000 On 06/21/17 16:59, Guido Falsi wrote: > On 06/13/17 13:44, Peter Blok wrote: >> Hi, >> >> For a while now, I’m not able to build a RPI1-B image from -stable. I have narrowed it dow to fix 318394, which adds a refresh option to geom_label. If I undo this fix in today’s stable it works ok. If I don’t I’m getting continuously: >> >> vm_fault: pager read error, pid 1 (init) >> vnode_pager_generic_getpages_done: I/O read error 5 >> >> I have looked at the fix and I can’t figure out why it breaks the code. >> >> And yes I have tried various other SD cards - they all have the same issue. >> > > Hi, > > I'm seeing similar symptoms with NanoBSD images on PCEngines ALIX and > APU2 boards, using compactflash and SD card storage respectively. The > problem has appeared as soon as I started testing 11.1-BETA1 from the > stable branch. > > Problem appears when I update the image, using a slightly modified > version of the standard nanobsd update and updatep[12] scripts. My > changes are not in the dd/gpart commands though, which are the same. > gpart seems the most likely candidate though. > > I have just discovered this thread and I will test reverting r318394 > soon. Thanks to Peter for narrowing it down! > > Maybe this is related to having the disks mounted read-only? > I noticed that after the problem appears many commands, including shutdown, start failing telling "device not configured" for all mounted FSes. I'm even unable to "ls /dev". Looks like the geom refresh changes devices from below the system in a way which triggers this reaction. I don't know the geom code and have been unable to find an immediate problem in the commit mentioned above. I'd really like some help to know where to look, or what kind of debugging information is needed. This is quite a bad bug for people running NanoBSD and should be fixed before the release. Thanks in advance! -- Guido Falsi From owner-freebsd-fs@freebsd.org Thu Jun 22 16:38:46 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C3F07D901D4 for ; Thu, 22 Jun 2017 16:38:46 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it0-x22b.google.com (mail-it0-x22b.google.com [IPv6:2607:f8b0:4001:c0b::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8041F81534 for ; Thu, 22 Jun 2017 16:38:46 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it0-x22b.google.com with SMTP id m62so53542017itc.0 for ; Thu, 22 Jun 2017 09:38:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=/AU5XL78bUnAcFlDsAhdINqsfKaIO3apC/Ma1Cc6HIQ=; b=oDv87gWIE8IeisZ7kGQ+I77ffkMl2HiH4pFgSQBUmYGICIg6cjWsRwaAkXlhk+YzG9 2dCWbjQD6lLrz7ZKNwh/E8Iaz94CnP9EpMaGyrG3SuZqhQ28XD+muRYFfLJfxY1lnS+d LhXbndDr/dK4pb3MC/rOXRMlVIshatCBCfQx3SatjLM8oVPcbFCNdk1kAp9mWy7eIs2P v+iqblQgYBdiWx/yt5titAcFOJcRQTFOvKZOqdj8B5Ubp0GN/vV6ZM/Oi2GN6WrxfH8+ T6xzmzkc3xn4kNHWpQO+OgTf3/4n/rSnlAtmjCQqCjVa3/V24p7sVvKNUPdrT6hadkFL 7jPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=/AU5XL78bUnAcFlDsAhdINqsfKaIO3apC/Ma1Cc6HIQ=; b=AmckUaAKpFkwZ+dKaRoAmqZKkp8JrHaL6ZjF422BfunUPnpcxMLI5zUAEFyqJ6fRWo G6xgm084eShxDThqSFkw9ETu3JtUJKSbxr3EYMD1A3tgmBndpjyzeo74y3/kt9l/Fojt CBnlG787IswSHQU4mZEYIiZVHaS14qEUpBPgkOPJLAEDLfoQtNBqbx+CFA5SXyZZ76MI pnr5JgFCTlpfoGtjmRy2u6dneCV7u1Istdzo1Ya3sgx6TzDF64/PWdt4wJvR8QI5Lg/Y Tx+oA2K0X+U6qm7cJSiA/sUep09qLMaLC6xNvy/D0EY8JwV6kBm5Sv5iPZUWiQiRRBoq hxhQ== X-Gm-Message-State: AKS2vOz9Vn5nCs5alQebd4DlszpmC0PjkAmIEX1XKwN3IyHNrxY5EYWC 7zCZdqogABrAQPx+Wa1J57vSO6yNv5pB X-Received: by 10.36.20.211 with SMTP id 202mr2965173itg.103.1498149525938; Thu, 22 Jun 2017 09:38:45 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.192.69 with HTTP; Thu, 22 Jun 2017 09:38:45 -0700 (PDT) X-Originating-IP: [2603:300b:6:5100:8171:44d0:c310:f5a9] In-Reply-To: <05e6bd02-3582-aea1-5fc3-19caa4073f94@FreeBSD.org> References: <20170610123435.GB69235@FreeBSD.org> <10A08FC1-C84E-4D06-9360-B7C3848F4680@bsd4all.org> <05e6bd02-3582-aea1-5fc3-19caa4073f94@FreeBSD.org> From: Warner Losh Date: Thu, 22 Jun 2017 10:38:45 -0600 X-Google-Sender-Auth: SGukOdEb9ptB2VDlvejvWsReULc Message-ID: Subject: Re: vnode_pager_generic_getpages_done: I/O read error 5 caused by r318394 (was Re: FreeBSD 11.1-BETA1 Now Available) To: Guido Falsi Cc: Peter Blok , Ed Maste , FreeBSD-STABLE Mailing List , FreeBSD FS , Glen Barber , Jonathan Chen Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jun 2017 16:38:46 -0000 On Thu, Jun 22, 2017 at 2:26 AM, Guido Falsi wrote: > On 06/21/17 16:59, Guido Falsi wrote: > > On 06/13/17 13:44, Peter Blok wrote: > >> Hi, > >> > >> For a while now, I=E2=80=99m not able to build a RPI1-B image from -st= able. I > have narrowed it dow to fix 318394, which adds a refresh option to > geom_label. If I undo this fix in today=E2=80=99s stable it works ok. If = I don=E2=80=99t > I=E2=80=99m getting continuously: > >> > >> vm_fault: pager read error, pid 1 (init) > >> vnode_pager_generic_getpages_done: I/O read error 5 > >> > >> I have looked at the fix and I can=E2=80=99t figure out why it breaks = the code. > >> > >> And yes I have tried various other SD cards - they all have the same > issue. > >> > > > > Hi, > > > > I'm seeing similar symptoms with NanoBSD images on PCEngines ALIX and > > APU2 boards, using compactflash and SD card storage respectively. The > > problem has appeared as soon as I started testing 11.1-BETA1 from the > > stable branch. > > > > Problem appears when I update the image, using a slightly modified > > version of the standard nanobsd update and updatep[12] scripts. My > > changes are not in the dd/gpart commands though, which are the same. > > gpart seems the most likely candidate though. > > > > I have just discovered this thread and I will test reverting r318394 > > soon. Thanks to Peter for narrowing it down! > > > > Maybe this is related to having the disks mounted read-only? > > > > I noticed that after the problem appears many commands, including > shutdown, start failing telling "device not configured" for all mounted > FSes. I'm even unable to "ls /dev". > > Looks like the geom refresh changes devices from below the system in a > way which triggers this reaction. > > I don't know the geom code and have been unable to find an immediate > problem in the commit mentioned above. I'd really like some help to know > where to look, or what kind of debugging information is needed. > > This is quite a bad bug for people running NanoBSD and should be fixed > before the release. > So can I recreate this with the embedded-type NanoBSD image? If this change breaks NanoBSD, it will need to be reverted... Warner From owner-freebsd-fs@freebsd.org Thu Jun 22 17:06:20 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31593D90B4B; Thu, 22 Jun 2017 17:06:20 +0000 (UTC) (envelope-from madpilot@FreeBSD.org) Received: from mail.madpilot.net (grunt.madpilot.net [78.47.145.38]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B906982642; Thu, 22 Jun 2017 17:06:19 +0000 (UTC) (envelope-from madpilot@FreeBSD.org) Received: from mail (mail [192.168.254.3]) by mail.madpilot.net (Postfix) with ESMTP id 3wtnzR1sVgzb56; Thu, 22 Jun 2017 19:06:11 +0200 (CEST) Received: from mail.madpilot.net ([192.168.254.3]) by mail (mail.madpilot.net [192.168.254.3]) (amavisd-new, port 10024) with ESMTP id FE8vaIaHIRzM; Thu, 22 Jun 2017 19:06:09 +0200 (CEST) Received: from tommy.madpilot.net (micro.madpilot.net [88.149.173.206]) by mail.madpilot.net (Postfix) with ESMTPSA; Thu, 22 Jun 2017 19:06:09 +0200 (CEST) Subject: Re: vnode_pager_generic_getpages_done: I/O read error 5 caused by r318394 (was Re: FreeBSD 11.1-BETA1 Now Available) To: Warner Losh Cc: Peter Blok , Ed Maste , FreeBSD-STABLE Mailing List , FreeBSD FS , Glen Barber , Jonathan Chen References: <20170610123435.GB69235@FreeBSD.org> <10A08FC1-C84E-4D06-9360-B7C3848F4680@bsd4all.org> <05e6bd02-3582-aea1-5fc3-19caa4073f94@FreeBSD.org> From: Guido Falsi Message-ID: <0c4c58d6-f34c-a358-3bda-122913110c6d@FreeBSD.org> Date: Thu, 22 Jun 2017 19:06:08 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jun 2017 17:06:20 -0000 On 06/22/17 18:38, Warner Losh wrote: > > > On Thu, Jun 22, 2017 at 2:26 AM, Guido Falsi > wrote: > > On 06/21/17 16:59, Guido Falsi wrote: > > On 06/13/17 13:44, Peter Blok wrote: > >> Hi, > >> > >> For a while now, I’m not able to build a RPI1-B image from -stable. I have narrowed it dow to fix 318394, which adds a refresh option to geom_label. If I undo this fix in today’s stable it works ok. If I don’t I’m getting continuously: > >> > >> vm_fault: pager read error, pid 1 (init) > >> vnode_pager_generic_getpages_done: I/O read error 5 > >> > >> I have looked at the fix and I can’t figure out why it breaks the code. > >> > >> And yes I have tried various other SD cards - they all have the same issue. > >> > > > > Hi, > > > > I'm seeing similar symptoms with NanoBSD images on PCEngines ALIX and > > APU2 boards, using compactflash and SD card storage respectively. The > > problem has appeared as soon as I started testing 11.1-BETA1 from the > > stable branch. > > > > Problem appears when I update the image, using a slightly modified > > version of the standard nanobsd update and updatep[12] scripts. My > > changes are not in the dd/gpart commands though, which are the same. > > gpart seems the most likely candidate though. > > > > I have just discovered this thread and I will test reverting r318394 > > soon. Thanks to Peter for narrowing it down! > > > > Maybe this is related to having the disks mounted read-only? > > > > I noticed that after the problem appears many commands, including > shutdown, start failing telling "device not configured" for all mounted > FSes. I'm even unable to "ls /dev". > > Looks like the geom refresh changes devices from below the system in a > way which triggers this reaction. > > I don't know the geom code and have been unable to find an immediate > problem in the commit mentioned above. I'd really like some help to know > where to look, or what kind of debugging information is needed. > > This is quite a bad bug for people running NanoBSD and should be fixed > before the release. > > > So can I recreate this with the embedded-type NanoBSD image? If this > change breaks NanoBSD, it will need to be reverted... > You should be able to reproduce it with a nanobsd image, then updating it using the standard script which dumps the new image in the "other" partition and uses gpart to configure the new partition as bootable. I'm using a slightly modified update script which also mounts the new partition in /mnt and performs some operations there. Then it dismounts the partition and launches the "gpart set -a active -i ${_to} ${NANO_DRIVE}" command (which I suspect is exactly where the actual problem is happening). I also tested reverting the change and can confirm that it makes the problem go away. I'm sure it can be triggered by other gpart operations. I'm trying to understand exactly which operations. I'll followup as soon as I have easier use case to reproduce it. I first need to revert to an image affected by the problem. Thanks for your feedback! -- Guido Falsi From owner-freebsd-fs@freebsd.org Thu Jun 22 19:36:16 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 40AAAD93331 for ; Thu, 22 Jun 2017 19:36:16 +0000 (UTC) (envelope-from theunusualmatt@gmail.com) Received: from mail-yw0-x241.google.com (mail-yw0-x241.google.com [IPv6:2607:f8b0:4002:c05::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EADC42DBC for ; Thu, 22 Jun 2017 19:36:15 +0000 (UTC) (envelope-from theunusualmatt@gmail.com) Received: by mail-yw0-x241.google.com with SMTP id s127so1590434ywg.3 for ; Thu, 22 Jun 2017 12:36:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=U1dL32fARACwJf4MH9lG9gOa1MpyLW7Q9MMkuKzjhN4=; b=YJMZVaXjRShF0DbLni4t05WmUIp30GNhJzTzWmwoS04SyLSNTIp36ttapdH8zX1j67 GYFYgjA3My8Xn4yETjYET+TxyiSYsUxSKMl/afQU/gcidoUDhEb5cDmRC/HvrDldlUBt Wgcd79UxkYCC1ub6I5b1zh5zmY33bYDOqp6wjVjGrw6negDOqxYJOZM0JRAjoqjq84CR ktFU+S/ozyFka3eeZg/biGDyjshAOp4g0oXwe5Y16/OqMS07TwdxhVgOmf93kb3WqpJ9 b7pvnJhn6m012whpUHvYWYANva8NIHReaXWgHm6zi2OA0GfxK5Wmx0bXz1VFI64e2Z56 L75A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=U1dL32fARACwJf4MH9lG9gOa1MpyLW7Q9MMkuKzjhN4=; b=Kb1nTD5R57GCK4jinaMqZN8jW6E58tJQABieRUlUJaN4S3mgcnHSiX7M6TLYhafGy/ ePakj8soBH5+KSISaMdVx3SkbB70ZSSTIux17lJ6hDnplSFOkICANEm+VMZ7R7dyMlQj J+EUZX6keMO+VS0KBYpHkaU+Jb48zjWOLUoq1EUR5I3WEe5gtxK38JfgGLqCPKMnNRzb LYuqya0O5eoQDytAFlcsVokaav7Ytk13lu6gsljf4TWVPsWnVLxLHeK8JnF9UK3oEQpg uZ+U3XSvv8p3P+MgGFMK15nDNcF99tpvBloRhajZ9SAXDzBrH2ojh6fmNyjq9cp55eZz OXjw== X-Gm-Message-State: AKS2vOzMzgDJ70LnDyv4CkqgKNNU0Ec9EeQY0Zyk6D8y53HeUqN+uoM0 jTqrg+yrRBe3zbMNpiRJmaIZ5SkAtI8DJ1A= X-Received: by 10.129.147.5 with SMTP id k5mr3090453ywg.128.1498160174869; Thu, 22 Jun 2017 12:36:14 -0700 (PDT) MIME-Version: 1.0 Received: by 10.129.103.70 with HTTP; Thu, 22 Jun 2017 12:36:14 -0700 (PDT) From: Matt B Date: Thu, 22 Jun 2017 15:36:14 -0400 Message-ID: Subject: SMBv1 Deprecation To: freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jun 2017 19:36:16 -0000 Long time user of FreeBSD here. I have been happily using the mount_smbfs binary and in my fstab to mount Windows Shares on boot to be used by various network services house on multiple FreeBSD systems. Sadly, it appears these connections all use SMBv1 NT1 security to perform the mount operation. With the new security landscape, post-WannaCry ransomware, in a mixed-mode environment where all the shares live in Windows, that just won't do. This has been discussed many times before in the past but there hasn't been any headway AFAIK. Every other piece of software I have encountered has moved away from this deprecated network protocol to the far more secure versions of SMB to perform Windows share operations. As a stop gap, I have implemented a very rudimentary NFS server advertising shares, but configuring a Kerberos infrastructure and setting new accounts for each and every service (not to mention the new permissions nightmares even with Active Directory) on multiple BSD systems is arduous. Rather, I am wondering why FreeBSD is behind the ball on the development? The other Linux based systems I run required a simple addition of the vers=SMB2 flag to the fstab entry to successfully mount. I understand the code base is very old for the mount_smbfs, but what is the way forward here? NFS is simply a workaround as far as I am concerned and every other *nix style distro seems to play nice with SMB. Is there an ETR on this greatly needed and long overdue update to mount newer style SMB shares? From owner-freebsd-fs@freebsd.org Thu Jun 22 20:02:31 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E077BD9384C; Thu, 22 Jun 2017 20:02:31 +0000 (UTC) (envelope-from madpilot@FreeBSD.org) Received: from mail.madpilot.net (grunt.madpilot.net [78.47.145.38]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A1AFA37EF; Thu, 22 Jun 2017 20:02:31 +0000 (UTC) (envelope-from madpilot@FreeBSD.org) Received: from mail (mail [192.168.254.3]) by mail.madpilot.net (Postfix) with ESMTP id 3wtsts4FmBzb56; Thu, 22 Jun 2017 22:02:29 +0200 (CEST) Received: from mail.madpilot.net ([192.168.254.3]) by mail (mail.madpilot.net [192.168.254.3]) (amavisd-new, port 10024) with ESMTP id 2YAnvQsr0yva; Thu, 22 Jun 2017 22:02:27 +0200 (CEST) Received: from tommy.madpilot.net (micro.madpilot.net [88.149.173.206]) by mail.madpilot.net (Postfix) with ESMTPSA; Thu, 22 Jun 2017 22:02:26 +0200 (CEST) Subject: Re: vnode_pager_generic_getpages_done: I/O read error 5 caused by r318394 (was Re: FreeBSD 11.1-BETA1 Now Available) From: Guido Falsi To: Warner Losh Cc: FreeBSD-STABLE Mailing List , FreeBSD FS , Glen Barber , Jonathan Chen , Peter Blok References: <20170610123435.GB69235@FreeBSD.org> <10A08FC1-C84E-4D06-9360-B7C3848F4680@bsd4all.org> <05e6bd02-3582-aea1-5fc3-19caa4073f94@FreeBSD.org> <0c4c58d6-f34c-a358-3bda-122913110c6d@FreeBSD.org> Message-ID: Date: Thu, 22 Jun 2017 22:02:26 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: <0c4c58d6-f34c-a358-3bda-122913110c6d@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jun 2017 20:02:32 -0000 On 06/22/17 19:06, Guido Falsi wrote: > On 06/22/17 18:38, Warner Losh wrote: > I'll followup as soon as I have easier use case to reproduce it. I first > need to revert to an image affected by the problem. I have made a few more tests. I am able to trigger this bug easily by running gpart. I'm testing on a PCEngines APU2 board with SD memory card. # gpart set -a active -i 1 mmcsd0 active set on mmcsd0s1 # fsck_ffs -n /dev/mmcsd0s1a ** /dev/mmcsd0s1a (NO WRITE) ** Last Mounted on /mnt ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames Segmentation fault # shutdown -r now /sbin/shutdown: Device not configured also, if I open another shell I can't perform many other operations which are not failing in the previous root shell: > tail /var/log/messages /usr/bin/tail: Device not configured. BTW while testing this multiple times I also had the root shell segfault while browsing history, so it should be quite easy to reproduce on your side too. running the gpart set command triggers it every time, with slightly different bu always disruptive symptoms. There is a chance it only shows with these embedded systems storage controllers though. -- Guido Falsi From owner-freebsd-fs@freebsd.org Thu Jun 22 20:19:01 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ED39FD93B9E for ; Thu, 22 Jun 2017 20:19:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DB4233F0F for ; Thu, 22 Jun 2017 20:19:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5MKJ1Q8074097 for ; Thu, 22 Jun 2017 20:19:01 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220203] [zfs] [panic] in dmu_objset_do_userquota_updates on mount Date: Thu, 22 Jun 2017 20:19:01 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jun 2017 20:19:02 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220203 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Jun 22 21:30:23 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 151D7D94B87 for ; Thu, 22 Jun 2017 21:30:23 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670067.outbound.protection.outlook.com [40.107.67.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B97E466239 for ; Thu, 22 Jun 2017 21:30:22 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0192.CANPRD01.PROD.OUTLOOK.COM (10.165.218.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Thu, 22 Jun 2017 21:30:20 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.1178.023; Thu, 22 Jun 2017 21:30:19 +0000 From: Rick Macklem To: Matt B , "freebsd-fs@freebsd.org" Subject: Re: SMBv1 Deprecation Thread-Topic: SMBv1 Deprecation Thread-Index: AQHS647UYCj1ThwsvUCkJ0Hkf0cfhqIxY+HF Date: Thu, 22 Jun 2017 21:30:19 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTXPR01MB0192; 7:Zxy7229RgV+IncAvUe1MSKRZNEoB3gfaTeQzkC6Y1ewI5utbdSmNrwMYyYXL+ebQM5f1klA/d2FdlTyvgI8qWQ0SxLZ/rCnx6SNlK/WKyXQnzB8T33150oWcIyMrs4iVUkJcsQ4W5nAS7U7k9NzXzzq6YdSiY7W4v1kq2aEzZBonlTDjx34HN+xTXH2/zbOuhqNMlsA+fJzOEC31klI70QtL3OwTEpS8WesVX0MQA+5ris1hQ0lfJh/GdMO9WTlBSKTgmYH5DyudWSNLPutOY+XG6sKI6m1MIsp31PqHm8TbJSSTJm1hfX03fEzub2Is6F09wpcLMhyYXJZGLDLaMGVf9WnZSugFJNhXzazcGr3K52sx1iAAR4oRKwfkiFlptOcgKvEs2y7p/NN0HMTzNuB5H8rZKGJjNVw3xZN2IpWcthXIC1cOh0T15t75JhwSdWJRImz2lRVRYbywctPH4ebq5i18w79P/7U0AQdsIms7kRXyp6HOvAqejAmc7mPc0QaQYfdQc9xBfb3n2y8/5ub5j6h2gN3HDRccbhWAxpeJoAf7zFuSTGVtTJNFpJCIORCGgwki1yEgeQ40ybyV0doKQUAzGl0axtQ1tPtlNcB2DKNIswvZs3/2FEZDD966LFp0uOXQj+zWsHH5IFBN1ku4xcbo6ceJ/pCzXCQjFiCko5uzTh/+GvdGNU6/V9Se9AN6SOneHX7HiaGgKPDUEfHb/wgVbhJSfxVryS0I2bMNcOAps27gmrbzRA/A12TfZzxSTUqoJ8wpj5nwoxI/zAPMJm9Ni/DVbJCoVeQhq2c= x-ms-office365-filtering-correlation-id: 5f720b68-2cdb-4a17-7273-08d4b9b5e204 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:YTXPR01MB0192; x-ms-traffictypediagnostic: YTXPR01MB0192: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(192374486261705)(75325880899374); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(3002001)(93006095)(93001095)(100000703101)(100105400095)(10201501046)(6041248)(20161123564025)(20161123562025)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123555025)(20161123560025)(20161123558100)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:YTXPR01MB0192; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:YTXPR01MB0192; x-forefront-prvs: 03468CBA43 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(39410400002)(39400400002)(39840400002)(39850400002)(39450400003)(377454003)(2906002)(33656002)(7116003)(122556002)(5660300001)(966005)(189998001)(6506006)(305945005)(53546010)(2950100002)(2501003)(14454004)(86362001)(478600001)(229853002)(39060400002)(6246003)(53936002)(7696004)(54356999)(3660700001)(76176999)(38730400002)(50986999)(77096006)(6306002)(55016002)(25786009)(6436002)(102836003)(74316002)(81166006)(2900100001)(8676002)(74482002)(8936002)(9686003)(3280700002); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0192; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; MLV:ovrnspm; PTR:InfoNoRecords; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Jun 2017 21:30:19.7931 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0192 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jun 2017 21:30:23 -0000 Well, the short answer is...somebody has to do it. (At this time, I believe that there are two people employed by the FreeBSD Foundation to do FreeBSD kernel work.) The rest of FreeBSD's development is done by volunteers (some of which do the work for an employer and get permission from the employer to upstream the work). I, for example, do NFS as a hobby and always have, but to be honest, there aren't many out there as stupid as I am and willing to do this;-) So, if you have the skills and time, feel free to do an implementation and, so long it is appropriately licensed (no GPL or similar), I suspect someone would be willing to work with you to get it into FreeBSD. If there is an SMBv2 implementation in one of the other BSDen (NetBSD, OpenBSD,...) the port wouldn't be an immense amount of work, but there are differences in the VFS and similar that will need to be dealt with. Otherwise, you are pretty much implementing it from scratch, using the SMBv1 code as a starting point. rick ________________________________________ From: owner-freebsd-fs@freebsd.org on behalf= of Matt B Sent: Thursday, June 22, 2017 3:36:14 PM To: freebsd-fs@freebsd.org Subject: SMBv1 Deprecation Long time user of FreeBSD here. I have been happily using the mount_smbfs binary and in my fstab to mount Windows Shares on boot to be used by various network services house on multiple FreeBSD systems. Sadly, it appears these connections all use SMBv1 NT1 security to perform the mount operation. With the new security landscape, post-WannaCry ransomware, in a mixed-mode environment where all the shares live in Windows, that just won't do. This has been discussed many times before in the past but there hasn't been any headway AFAIK. Every other piece of software I have encountered has moved away from this deprecated network protocol to the far more secure versions of SMB to perform Windows share operations. As a stop gap, I have implemented a very rudimentary NFS server advertising shares, but configuring a Kerberos infrastructure and setting new accounts for each and every service (not to mention the new permissions nightmares even with Active Directory) on multiple BSD systems is arduous. Rather, I am wondering why FreeBSD is behind the ball on the development? The other Linux based systems I run required a simple addition of the vers=3DSMB2 fla= g to the fstab entry to successfully mount. I understand the code base is very old for the mount_smbfs, but what is the way forward here? NFS is simply a workaround as far as I am concerned and every other *nix style distro seems to play nice with SMB. Is there an ETR on this greatly needed and long overdue update to mount newer style SMB shares? _______________________________________________ freebsd-fs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Thu Jun 22 22:24:07 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 42FFCD95879 for ; Thu, 22 Jun 2017 22:24:07 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660068.outbound.protection.outlook.com [40.107.66.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BD79B67A34 for ; Thu, 22 Jun 2017 22:24:06 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0190.CANPRD01.PROD.OUTLOOK.COM (10.165.218.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Thu, 22 Jun 2017 22:24:03 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.1178.023; Thu, 22 Jun 2017 22:24:03 +0000 From: Rick Macklem To: "freebsd-fs@freebsd.org" Subject: patch that adds two additional NFSv4.1 compound RPCs to the client Thread-Topic: patch that adds two additional NFSv4.1 compound RPCs to the client Thread-Index: AQHS66XK/ka2oKFZiUmSwrXkmlPFhw== Date: Thu, 22 Jun 2017 22:24:03 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: authentication-results: freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTXPR01MB0190; 7:nzFJ3XfO06TX2djaEeO6pE49jAsbu3ESNQL3xX1gEZZ/OD1OYwx7MJ1xfclp56fPRJVMwCWgaZHelh8uLk7Wi+lABDm3B9oQtG9Qr7elSfgEbXVpzkKMxGwRyerUzMQw9maxP+vVV799IW6n090HZfUYjQX6g1V8XWxVq8hDv9QwFA5A5I36fVycSm0L70sX9eEf9wP7PCBCslcDAsAibpc9vWaWe0uROr5UD12o84XW+16xKagtNYkEfzSnDqOcDorhtl551+nDB7mkzWoKAlUp7+5FdC+LvyIHXFPOjbADuBKrmw9pJ+lGroY4b+CSp/1Utx5t6On4k8P/DZ9ra7kK2y7VvViw47tJxBvhiz6kTxvVOYSqQcbPpoDDMq/3LbsvoNSy1QhWbewi8rj902tFbA69vyUury0mAM02YJyBlQNmQHnitvtl8ljUE1385r762pHDMDWHpowJabFDw2agQi4GFvYsdFSeMFULbTDxGL/DV8BUMAkHvakHvkyXtoBPTmdh2tVo4h7bBO7aMO4TUVyo32GJBMrrklyhb5AwhxXEXZtCUfM0JXzxjIMu1Upkf2QnYMuuO4iMnm2tkE0eIBMtzXRjj9k3L2hKAeahHbIX83QIGJTwme2zSCOHsOSxHo7UaKMZJtXJYexXYChJB0tenxVebkqodO+5+ITE8yJpTJ6DW5MJB9jZmD0M2Bz98MZpJRlrXinxdzo8IMJ1VSsBNTiQ0XzEPEMPbLokWikCH/Zn2n1XdaM95YJngvtTpNcTAeC/FG2dACfxgJ7s71mQf9A9ENOYZEI9dsg= x-ms-office365-filtering-correlation-id: 114b4b7f-5af0-4bd7-a387-08d4b9bd636f x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254075)(49563074)(201703131423075)(201703031133081)(201702281549075); SRVR:YTXPR01MB0190; x-ms-traffictypediagnostic: YTXPR01MB0190: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(5213294742642); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(102415395)(6040450)(2401047)(8121501046)(5005006)(10201501046)(100000703101)(100105400095)(93006095)(93001095)(3002001)(6041248)(20161123560025)(20161123558100)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(20161123555025)(20161123564025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:YTXPR01MB0190; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:YTXPR01MB0190; x-forefront-prvs: 03468CBA43 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(39400400002)(39850400002)(39410400002)(39450400003)(39840400002)(478600001)(5890100001)(2501003)(9686003)(2351001)(305945005)(2900100001)(74482002)(81166006)(2906002)(8936002)(77096006)(122556002)(3280700002)(25786009)(55016002)(53936002)(3660700001)(74316002)(5640700003)(102836003)(6506006)(38730400002)(8676002)(6916009)(6436002)(86362001)(189998001)(99936001)(110136004)(54356999)(5660300001)(7696004)(50986999)(14454004)(33656002); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0190; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; MLV:sfv; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/mixed; boundary="_002_YTXPR01MB018901BE4BA7E9780F4B430BDDDB0YTXPR01MB0189CANP_" MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Jun 2017 22:24:03.3050 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0190 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jun 2017 22:24:07 -0000 --_002_YTXPR01MB018901BE4BA7E9780F4B430BDDDB0YTXPR01MB0189CANP_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable The attached patch (you can also find it in PR#219550), adds two new compou= nd RPCs to the NFSv4.1/pNFS client. These compounds combine the Open with the LayoutGet, avoiding the need to do additional RPCs for pNFS. The patch is fairly large, but only affects the NFSv4.1 client with the "pn= fs" option. It factors most of the LayoutGet RPC code into separate functions and then = uses those to do RPCs that follow an Open with a LayoutGet. This code has been tested a fair amount by me using my pNFS server (in projects/pnfs-planb-server). Would anyone like to review this? rick= --_002_YTXPR01MB018901BE4BA7E9780F4B430BDDDB0YTXPR01MB0189CANP_ Content-Type: application/octet-stream; name="openlayout.patch" Content-Description: openlayout.patch Content-Disposition: attachment; filename="openlayout.patch"; size=38817; creation-date="Thu, 22 Jun 2017 22:23:56 GMT"; modification-date="Thu, 22 Jun 2017 22:23:56 GMT" Content-Transfer-Encoding: base64 LS0tIHN5cy9mcy9uZnMvbmZzcG9ydC5oLm9yaWcJMjAxNy0wNC0yNyAxMTo1MTo1MS4wMDAwMDAw MDAgLTA0MDAKKysrIHN5cy9mcy9uZnMvbmZzcG9ydC5oCTIwMTctMDUtMTkgMTM6MjI6NDUuNjA0 MjE5MDAwIC0wNDAwCkBAIC0zNTcsMTEgKzM1NywxMyBAQAogI2RlZmluZQlORlNQUk9DX1dSSVRF RFMJCTUxCiAjZGVmaW5lCU5GU1BST0NfUkVBRERTCQk1MgogI2RlZmluZQlORlNQUk9DX0NPTU1J VERTCTUzCisjZGVmaW5lCU5GU1BST0NfT1BFTkxBWUdFVAk1NAorI2RlZmluZQlORlNQUk9DX0NS RUFURUxBWUdFVAk1NQogCiAvKgogICogTXVzdCBiZSBkZWZpbmVkIGFzIG9uZSBoaWdoZXIgdGhh biB0aGUgbGFzdCBORlN2NC4xIFByb2MjIGFib3ZlLgogICovCi0jZGVmaW5lCU5GU1Y0MV9OUFJP Q1MJCTU0CisjZGVmaW5lCU5GU1Y0MV9OUFJPQ1MJCTU2CiAKICNlbmRpZgkvKiBORlNfVjNOUFJP Q1MgKi8KIApAQCAtMzkwLDcgKzM5Miw3IEBAIHN0cnVjdCBuZnNzdGF0c3YxIHsKIAl1aW50NjRf dAlyZWFkbGlua19iaW9zOwogCXVpbnQ2NF90CWJpb2NhY2hlX3JlYWRkaXJzOwogCXVpbnQ2NF90 CXJlYWRkaXJfYmlvczsKLQl1aW50NjRfdAlycGNjbnRbTkZTVjQxX05QUk9DUyArIDE1XTsKKwl1 aW50NjRfdAlycGNjbnRbTkZTVjQxX05QUk9DUyArIDEzXTsKIAl1aW50NjRfdAlycGNyZXRyaWVz OwogCXVpbnQ2NF90CXNydnJwY2NudFtORlNWNDJfTk9QUyArIE5GU1Y0T1BfRkFLRU5PUFNdOwog CXVpbnQ2NF90CXNydnJwY19lcnJzOwotLS0gc3lzL2ZzL25mcy9uZnNwcm90by5oLm9yaWcJMjAx Ny0wNS0xMSAxMTowODowMS42ODEyNTEwMDAgLTA0MDAKKysrIHN5cy9mcy9uZnMvbmZzcHJvdG8u aAkyMDE3LTA1LTE5IDEzOjIzOjIxLjk5MTI5MDAwMCAtMDQwMApAQCAtMzQ2LDExICszNDYsMTMg QEAKICNkZWZpbmUJTkZTUFJPQ19XUklURURTCQk1MQogI2RlZmluZQlORlNQUk9DX1JFQUREUwkJ NTIKICNkZWZpbmUJTkZTUFJPQ19DT01NSVREUwk1MworI2RlZmluZQlORlNQUk9DX09QRU5MQVlH RVQJNTQKKyNkZWZpbmUJTkZTUFJPQ19DUkVBVEVMQVlHRVQJNTUKIAogLyoKICAqIE11c3QgYmUg ZGVmaW5lZCBhcyBvbmUgaGlnaGVyIHRoYW4gdGhlIGxhc3QgTkZTdjQuMSBQcm9jIyBhYm92ZS4K ICAqLwotI2RlZmluZQlORlNWNDFfTlBST0NTCQk1NAorI2RlZmluZQlORlNWNDFfTlBST0NTCQk1 NgogCiAjZW5kaWYJLyogTkZTX1YzTlBST0NTICovCiAKLS0tIHN5cy9mcy9uZnMvbmZzX2NvbW1v bnN1YnMuYy5vcmlnCTIwMTctMDUtMTYgMDk6Mzg6MjEuNzIzMTYzMDAwIC0wNDAwCisrKyBzeXMv ZnMvbmZzL25mc19jb21tb25zdWJzLmMJMjAxNy0wNS0xOSAxMzoyMToyNy44Njc3NzgwMDAgLTA0 MDAKQEAgLTE3Nyw3ICsxNzcsNyBAQCBzdGF0aWMgc3RydWN0IG5mc3J2X2x1Z2hhc2gJKm5mc2dy b3VwbmFtCiAgKi8KIGludCBuZnNfYmlncmVwbHlbTkZTVjQxX05QUk9DU10gPSB7IDAsIDAsIDAs IDEsIDAsIDEsIDEsIDAsIDAsIDAsIDAsCiAgICAgMCwgMCwgMCwgMCwgMCwgMSwgMSwgMCwgMCwg MCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwKLSAgICAwLCAw LCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAwLCAxLCAwIH07CisgICAg MCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMSwgMCwgMCwg MCB9OwogCiAvKiBsb2NhbCBmdW5jdGlvbnMgKi8KIHN0YXRpYyBpbnQgbmZzcnZfc2tpcGFjZShz dHJ1Y3QgbmZzcnZfZGVzY3JpcHQgKm5kLCBpbnQgKmFjZXNpemVwKTsKLS0tIHN5cy9mcy9uZnNj bGllbnQvbmZzX2NsY29tc3Vicy5jLm9yaWcJMjAxNy0wNS0xNiAxMjozNjoxNi4wOTg4NzgwMDAg LTA0MDAKKysrIHN5cy9mcy9uZnNjbGllbnQvbmZzX2NsY29tc3Vicy5jCTIwMTctMDUtMTkgMTM6 MjU6NTkuODIzMDkyMDAwIC0wNDAwCkBAIC0xMTIsNiArMTEyLDggQEAgc3RhdGljIHN0cnVjdCB7 CiAJeyBORlNWNE9QX1dSSVRFLCAxLCAiV3JpdGVEUyIsIDcsIH0sCiAJeyBORlNWNE9QX1JFQUQs IDEsICJSZWFkRFMiLCA2LCB9LAogCXsgTkZTVjRPUF9DT01NSVQsIDEsICJDb21taXREUyIsIDgs IH0sCisJeyBORlNWNE9QX09QRU4sIDMsICJPcGVuTGF5b3V0R2V0IiwgMTMsIH0sCisJeyBORlNW NE9QX09QRU4sIDgsICJDcmVhdGVMYXlHZXQiLCAxMiwgfSwKIH07CiAKIC8qCkBAIC0xMjAsNyAr MTIyLDcgQEAgc3RhdGljIHN0cnVjdCB7CiBzdGF0aWMgaW50IG5mc19iaWdyZXF1ZXN0W05GU1Y0 MV9OUFJPQ1NdID0gewogCTAsIDAsIDAsIDAsIDAsIDAsIDAsIDEsIDAsIDAsIDEsIDAsIDAsIDAs IDAsIDAsIDAsIDAsIDAsIDAsIDAsIDAsCiAJMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwg MCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwgMCwKLQkwLCAwLCAwLCAwLCAwLCAw LCAxLCAwLCAwCisJMCwgMCwgMCwgMCwgMCwgMCwgMSwgMCwgMCwgMCwgMAogfTsKIAogLyoKLS0t IHN5cy9mcy9uZnNjbGllbnQvbmZzX2NscnBjb3BzLmMub3JpZwkyMDE3LTA1LTExIDEwOjUyOjQy LjE2ODcxMTAwMCAtMDQwMAorKysgc3lzL2ZzL25mc2NsaWVudC9uZnNfY2xycGNvcHMuYwkyMDE3 LTA1LTE5IDEzOjI5OjA4LjQwOTA5ODAwMCAtMDQwMApAQCAtMTI3LDYgKzEyNywyOSBAQCBzdGF0 aWMgZW51bSBuZnNjbGRzX3N0YXRlIG5mc2NsX2dldHNhbWVzCiBzdGF0aWMgaW50IG5mc3JwY19j b21taXRkcyh2bm9kZV90LCB1aW50NjRfdCwgaW50LCBzdHJ1Y3QgbmZzY2xkcyAqLAogICAgIHN0 cnVjdCBuZnNmaCAqLCBzdHJ1Y3QgdWNyZWQgKiwgTkZTUFJPQ19UICosIHZvaWQgKik7CiAjZW5k aWYKK3N0YXRpYyB2b2lkIG5mc3J2X3NldHVwbGF5b3V0Z2V0KHN0cnVjdCBuZnNydl9kZXNjcmlw dCAqLCBpbnQsIHVpbnQ2NF90LAorICAgIHVpbnQ2NF90LCB1aW50NjRfdCwgbmZzdjRzdGF0ZWlk X3QgKiwgaW50LCBpbnQpOworc3RhdGljIGludCBuZnNydl9wYXJzZWxheW91dGdldChzdHJ1Y3Qg bmZzcnZfZGVzY3JpcHQgKiwgbmZzdjRzdGF0ZWlkX3QgKiwKKyAgICBpbnQgKiwgc3RydWN0IG5m c2NsZmxheW91dGhlYWQgKik7CitzdGF0aWMgaW50IG5mc3JwY19nZXRvcGVubGF5b3V0KHN0cnVj dCBuZnNtb3VudCAqLCB2bm9kZV90LCB1X2ludDhfdCAqLAorICAgIGludCwgdWludDhfdCAqLCBp bnQsIHVpbnQzMl90LCBzdHJ1Y3QgbmZzY2xvcGVuICosIHVpbnQ4X3QgKiwgaW50LAorICAgIHN0 cnVjdCBuZnNjbGRlbGVnICoqLCBzdHJ1Y3QgdWNyZWQgKiwgTkZTUFJPQ19UICopOworc3RhdGlj IGludCBuZnNycGNfZ2V0Y3JlYXRlbGF5b3V0KHZub2RlX3QsIGNoYXIgKiwgaW50LCBzdHJ1Y3Qg dmF0dHIgKiwKKyAgICBuZnNxdWFkX3QsIGludCwgc3RydWN0IG5mc2Nsb3duZXIgKiwgc3RydWN0 IG5mc2NsZGVsZWcgKiosCisgICAgc3RydWN0IHVjcmVkICosIE5GU1BST0NfVCAqLCBzdHJ1Y3Qg bmZzdmF0dHIgKiwgc3RydWN0IG5mc3ZhdHRyICosCisgICAgc3RydWN0IG5mc2ZoICoqLCBpbnQg KiwgaW50ICosIHZvaWQgKiwgaW50ICopOworc3RhdGljIGludCBuZnNycGNfb3BlbmxheW91dHJw YyhzdHJ1Y3QgbmZzbW91bnQgKiwgdm5vZGVfdCwgdV9pbnQ4X3QgKiwKKyAgICBpbnQsIHVpbnQ4 X3QgKiwgaW50LCB1aW50MzJfdCwgc3RydWN0IG5mc2Nsb3BlbiAqLCB1aW50OF90ICosIGludCwK KyAgICBzdHJ1Y3QgbmZzY2xkZWxlZyAqKiwgbmZzdjRzdGF0ZWlkX3QgKiwgaW50LCBpbnQsIGlu dCAqLAorICAgIHN0cnVjdCBuZnNjbGZsYXlvdXRoZWFkICosIGludCAqLCBzdHJ1Y3QgdWNyZWQg KiwgTkZTUFJPQ19UICopOworc3RhdGljIGludCBuZnNycGNfY3JlYXRlbGF5b3V0KHZub2RlX3Qs IGNoYXIgKiwgaW50LCBzdHJ1Y3QgdmF0dHIgKiwKKyAgICBuZnNxdWFkX3QsIGludCwgc3RydWN0 IG5mc2Nsb3duZXIgKiwgc3RydWN0IG5mc2NsZGVsZWcgKiosCisgICAgc3RydWN0IHVjcmVkICos IE5GU1BST0NfVCAqLCBzdHJ1Y3QgbmZzdmF0dHIgKiwgc3RydWN0IG5mc3ZhdHRyICosCisgICAg c3RydWN0IG5mc2ZoICoqLCBpbnQgKiwgaW50ICosIHZvaWQgKiwgaW50ICosIG5mc3Y0c3RhdGVp ZF90ICosCisgICAgaW50LCBpbnQsIGludCAqLCBzdHJ1Y3QgbmZzY2xmbGF5b3V0aGVhZCAqLCBp bnQgKik7CitzdGF0aWMgaW50IG5mc3JwY19sYXlvdXRnZXRyZXMoc3RydWN0IG5mc21vdW50ICos IHZub2RlX3QsIHVpbnQ4X3QgKiwKKyAgICBpbnQsIG5mc3Y0c3RhdGVpZF90ICosIGludCwgdWlu dDMyX3QgKiwgc3RydWN0IG5mc2NsbGF5b3V0ICoqLAorICAgIHN0cnVjdCBuZnNjbGZsYXlvdXRo ZWFkICosIGludCwgaW50ICosIHN0cnVjdCB1Y3JlZCAqLCBORlNQUk9DX1QgKik7CiAKIC8qCiAg KiBuZnMgbnVsbCBjYWxsIGZyb20gdmZzLgpAQCAtMzAxLDExICszMjQsMjcgQEAgZWxzZSBwcmlu dGYoIiBmaGw9MFxuIik7CiAJCWNsaWRyZXYgPSAwOwogCSAgICBpZiAocmV0ID09IE5GU0NMT1BF Tl9ET09QRU4pIHsKIAkJaWYgKG5wLT5uX3Y0ICE9IE5VTEwpIHsKLQkJCWVycm9yID0gbmZzcnBj X29wZW5ycGMobm1wLCB2cCwgbnAtPm5fdjQtPm40X2RhdGEsCi0JCQkgICBucC0+bl92NC0+bjRf ZmhsZW4sIG5wLT5uX2ZocC0+bmZoX2ZoLAotCQkJICAgbnAtPm5fZmhwLT5uZmhfbGVuLCBtb2Rl LCBvcCwKLQkJCSAgIE5GUzROT0RFTkFNRShucC0+bl92NCksIG5wLT5uX3Y0LT5uNF9uYW1lbGVu LCAmZHAsCi0JCQkgICAwLCAweDAsIGNyZWQsIHAsIDAsIDApOworCQkJLyoKKwkJCSAqIEZvciB0 aGUgZmlyc3QgYXR0ZW1wdCwgdHJ5IGFuZCBnZXQgYSBsYXlvdXQsIGlmCisJCQkgKiBwTkZTIGlz IGVuYWJsZWQgZm9yIHRoZSBtb3VudC4KKwkJCSAqLworCQkJaWYgKCFORlNIQVNQTkZTKG5tcCkg fHwgbmZzY2xfZW5hYmxlY2FsbGIgPT0gMCB8fAorCQkJICAgIG5mc19udW1uZnNjYmQgPT0gMCB8 fAorCQkJICAgIChucC0+bl9mbGFnICYgTk5PTEFZT1VUKSAhPSAwIHx8IHJldHJ5Y250ID4gMCkK KwkJCQllcnJvciA9IG5mc3JwY19vcGVucnBjKG5tcCwgdnAsCisJCQkJICAgIG5wLT5uX3Y0LT5u NF9kYXRhLAorCQkJCSAgICBucC0+bl92NC0+bjRfZmhsZW4sIG5wLT5uX2ZocC0+bmZoX2ZoLAor CQkJCSAgICBucC0+bl9maHAtPm5maF9sZW4sIG1vZGUsIG9wLAorCQkJCSAgICBORlM0Tk9ERU5B TUUobnAtPm5fdjQpLAorCQkJCSAgICBucC0+bl92NC0+bjRfbmFtZWxlbiwKKwkJCQkgICAgJmRw LCAwLCAweDAsIGNyZWQsIHAsIDAsIDApOworCQkJZWxzZQorCQkJCWVycm9yID0gbmZzcnBjX2dl dG9wZW5sYXlvdXQobm1wLCB2cCwKKwkJCQkgICAgbnAtPm5fdjQtPm40X2RhdGEsCisJCQkJICAg IG5wLT5uX3Y0LT5uNF9maGxlbiwgbnAtPm5fZmhwLT5uZmhfZmgsCisJCQkJICAgIG5wLT5uX2Zo cC0+bmZoX2xlbiwgbW9kZSwgb3AsCisJCQkJICAgIE5GUzROT0RFTkFNRShucC0+bl92NCksCisJ CQkJICAgIG5wLT5uX3Y0LT5uNF9uYW1lbGVuLCAmZHAsIGNyZWQsIHApOwogCQkJaWYgKGRwICE9 IE5VTEwpIHsKICNpZmRlZiBBUFBMRQogCQkJCU9TQml0QW5kQXRvbWljKChpbnQzMl90KX5OREVM RUdNT0QsIChVSW50MzIgKikmbnAtPm5fZmxhZyk7CkBAIC0xODk0LDkgKzE5MzMsMTUgQEAgbmZz cnBjX2NyZWF0ZSh2bm9kZV90IGR2cCwgY2hhciAqbmFtZSwgaQogCQkJY2xpZHJldiA9IG5tcC0+ bm1fY2xwLT5uZnNjX2NsaWVudGlkcmV2OwogCQllbHNlCiAJCQljbGlkcmV2ID0gMDsKLQkJZXJy b3IgPSBuZnNycGNfY3JlYXRldjQoZHZwLCBuYW1lLCBuYW1lbGVuLCB2YXAsIGN2ZXJmLCBmbW9k ZSwKLQkJICBvd3AsICZkcCwgY3JlZCwgcCwgZG5hcCwgbm5hcCwgbmZocHAsIGF0dHJmbGFncCwg ZGF0dHJmbGFncCwKLQkJICBkc3R1ZmYsICZ1bmxvY2tlZCk7CisJCWlmICghTkZTSEFTUE5GUyhu bXApIHx8IG5mc2NsX2VuYWJsZWNhbGxiID09IDAgfHwKKwkJICAgIG5mc19udW1uZnNjYmQgPT0g MCB8fCByZXRyeWNudCA+IDApCisJCQllcnJvciA9IG5mc3JwY19jcmVhdGV2NChkdnAsIG5hbWUs IG5hbWVsZW4sIHZhcCwgY3ZlcmYsCisJCQkgIGZtb2RlLCBvd3AsICZkcCwgY3JlZCwgcCwgZG5h cCwgbm5hcCwgbmZocHAsCisJCQkgIGF0dHJmbGFncCwgZGF0dHJmbGFncCwgZHN0dWZmLCAmdW5s b2NrZWQpOworCQllbHNlCisJCQllcnJvciA9IG5mc3JwY19nZXRjcmVhdGVsYXlvdXQoZHZwLCBu YW1lLCBuYW1lbGVuLCB2YXAsCisJCQkgIGN2ZXJmLCBmbW9kZSwgb3dwLCAmZHAsIGNyZWQsIHAs IGRuYXAsIG5uYXAsIG5maHBwLAorCQkJICBhdHRyZmxhZ3AsIGRhdHRyZmxhZ3AsIGRzdHVmZiwg JnVubG9ja2VkKTsKIAkJLyoKIAkJICogVGhlcmUgaXMgbm8gbmVlZCB0byBpbnZhbGlkYXRlIGNh Y2hlZCBhdHRyaWJ1dGVzIGhlcmUsCiAJCSAqIHNpbmNlIG5ldyBwb3N0LWRlbGVnYXRpb24gaXNz dWUgYXR0cmlidXRlcyBhcmUgYWx3YXlzCkBAIC00NzY2LDE0OSArNDgxMSwyMiBAQCBuZnNycGNf bGF5b3V0Z2V0KHN0cnVjdCBuZnNtb3VudCAqbm1wLCB1CiAgICAgbmZzdjRzdGF0ZWlkX3QgKnN0 YXRlaWRwLCBpbnQgKnJldG9uY2xvc2VwLCBzdHJ1Y3QgbmZzY2xmbGF5b3V0aGVhZCAqZmxocCwK ICAgICBzdHJ1Y3QgdWNyZWQgKmNyZWQsIE5GU1BST0NfVCAqcCwgdm9pZCAqc3R1ZmYpCiB7Ci0J dWludDMyX3QgKnRsOwogCXN0cnVjdCBuZnNydl9kZXNjcmlwdCBuZnNkLCAqbmQgPSAmbmZzZDsK LQlzdHJ1Y3QgbmZzZmggKm5maHA7Ci0Jc3RydWN0IG5mc2NsZmxheW91dCAqZmxwLCAqcHJldmZs cCwgKnRmbHA7Ci0JaW50IGNudCwgZXJyb3IsIGdvdGlvbW9kZSwgZmhjbnQsIG5maGxlbiwgaSwg ajsKLQl1aW50OF90ICpjcDsKLQl1aW50NjRfdCByZXRsZW47CisJaW50IGVycm9yOwogCi0JZmxw ID0gTlVMTDsKLQlnb3Rpb21vZGUgPSAtMTsKIAluZnNjbF9yZXFzdGFydChuZCwgTkZTUFJPQ19M QVlPVVRHRVQsIG5tcCwgZmhwLCBmaGxlbiwgTlVMTCwgTlVMTCk7Ci0JTkZTTV9CVUlMRCh0bCwg dWludDMyX3QgKiwgNCAqIE5GU1hfVU5TSUdORUQgKyAzICogTkZTWF9IWVBFUiArCi0JICAgIE5G U1hfU1RBVEVJRCk7Ci0JKnRsKysgPSBuZXduZnNfZmFsc2U7CQkvKiBEb24ndCBzaWduYWwgYXZh aWxhYmlsaXR5LiAqLwotCSp0bCsrID0gdHhkcl91bnNpZ25lZChORlNMQVlPVVRfTkZTVjRfMV9G SUxFUyk7Ci0JKnRsKysgPSB0eGRyX3Vuc2lnbmVkKGlvbW9kZSk7Ci0JdHhkcl9oeXBlcihvZmZz ZXQsIHRsKTsKLQl0bCArPSAyOwotCXR4ZHJfaHlwZXIobGVuLCB0bCk7Ci0JdGwgKz0gMjsKLQl0 eGRyX2h5cGVyKG1pbmxlbiwgdGwpOwotCXRsICs9IDI7Ci0JKnRsKysgPSB0eGRyX3Vuc2lnbmVk KHN0YXRlaWRwLT5zZXFpZCk7Ci0JTkZTQ0xfREVCVUcoNCwgImxheWdldCBzZXE9JWRcbiIsIChp bnQpc3RhdGVpZHAtPnNlcWlkKTsKLQkqdGwrKyA9IHN0YXRlaWRwLT5vdGhlclswXTsKLQkqdGwr KyA9IHN0YXRlaWRwLT5vdGhlclsxXTsKLQkqdGwrKyA9IHN0YXRlaWRwLT5vdGhlclsyXTsKLQkq dGwgPSB0eGRyX3Vuc2lnbmVkKGxheW91dGxlbik7CisJbmZzcnZfc2V0dXBsYXlvdXRnZXQobmQs IGlvbW9kZSwgb2Zmc2V0LCBsZW4sIG1pbmxlbiwgc3RhdGVpZHAsCisJICAgIGxheW91dGxlbiwg MCk7CiAJbmQtPm5kX2ZsYWcgfD0gTkRfVVNFR1NTTkFNRTsKIAllcnJvciA9IG5ld25mc19yZXF1 ZXN0KG5kLCBubXAsIE5VTEwsICZubXAtPm5tX3NvY2tyZXEsIE5VTEwsIHAsIGNyZWQsCiAJICAg IE5GU19QUk9HLCBORlNfVkVSNCwgTlVMTCwgMSwgTlVMTCwgTlVMTCk7CisJTkZTQ0xfREVCVUco NCwgImxheWdldCBlcnI9JWQgc3Q9JWRcbiIsIGVycm9yLCBuZC0+bmRfcmVwc3RhdCk7CiAJaWYg KGVycm9yICE9IDApCiAJCXJldHVybiAoZXJyb3IpOwotCWlmIChuZC0+bmRfcmVwc3RhdCA9PSAw KSB7Ci0JCU5GU01fRElTU0VDVCh0bCwgdWludDMyX3QgKiwgMiAqIE5GU1hfVU5TSUdORUQgKyBO RlNYX1NUQVRFSUQpOwotCQlpZiAoKnRsKysgIT0gMCkKLQkJCSpyZXRvbmNsb3NlcCA9IDE7Ci0J CWVsc2UKLQkJCSpyZXRvbmNsb3NlcCA9IDA7Ci0JCXN0YXRlaWRwLT5zZXFpZCA9IGZ4ZHJfdW5z aWduZWQodWludDMyX3QsICp0bCsrKTsKLQkJTkZTQ0xfREVCVUcoNCwgInJldG9uY2xzPSVkIHN0 c2VxPSVkXG4iLCAqcmV0b25jbG9zZXAsCi0JCSAgICAoaW50KXN0YXRlaWRwLT5zZXFpZCk7Ci0J CXN0YXRlaWRwLT5vdGhlclswXSA9ICp0bCsrOwotCQlzdGF0ZWlkcC0+b3RoZXJbMV0gPSAqdGwr KzsKLQkJc3RhdGVpZHAtPm90aGVyWzJdID0gKnRsKys7Ci0JCWNudCA9IGZ4ZHJfdW5zaWduZWQo aW50LCAqdGwpOwotCQlORlNDTF9ERUJVRyg0LCAibGF5ZyBjbnQ9JWRcbiIsIGNudCk7Ci0JCWlm IChjbnQgPD0gMCB8fCBjbnQgPiAxMDAwMCkgewotCQkJLyogRG9uJ3QgYWNjZXB0IG1vcmUgdGhh biAxMDAwMCBsYXlvdXRzIGluIHJlcGx5LiAqLwotCQkJZXJyb3IgPSBORlNFUlJfQkFEWERSOwot CQkJZ290byBuZnNtb3V0OwotCQl9Ci0JCWZvciAoaSA9IDA7IGkgPCBjbnQ7IGkrKykgewotCQkJ LyogRGlzc2VjdCBhbGwgdGhlIHdheSB0byB0aGUgZmlsZSBoYW5kbGUgY250LiAqLwotCQkJTkZT TV9ESVNTRUNUKHRsLCB1aW50MzJfdCAqLCAzICogTkZTWF9IWVBFUiArCi0JCQkgICAgNiAqIE5G U1hfVU5TSUdORUQgKyBORlNYX1Y0REVWSUNFSUQpOwotCQkJZmhjbnQgPSBmeGRyX3Vuc2lnbmVk KGludCwgKih0bCArIDExICsKLQkJCSAgICBORlNYX1Y0REVWSUNFSUQgLyBORlNYX1VOU0lHTkVE KSk7Ci0JCQlORlNDTF9ERUJVRyg0LCAiZmhjbnQ9JWRcbiIsIGZoY250KTsKLQkJCWlmIChmaGNu dCA8IDAgfHwgZmhjbnQgPiAxMDApIHsKLQkJCQkvKiBEb24ndCBhY2NlcHQgbW9yZSB0aGFuIDEw MCBmaWxlIGhhbmRsZXMuICovCi0JCQkJZXJyb3IgPSBORlNFUlJfQkFEWERSOwotCQkJCWdvdG8g bmZzbW91dDsKLQkJCX0KLQkJCWlmIChmaGNudCA+IDEpCi0JCQkJZmxwID0gbWFsbG9jKHNpemVv ZigqZmxwKSArIChmaGNudCAtIDEpICoKLQkJCQkgICAgc2l6ZW9mKHN0cnVjdCBuZnNmaCAqKSwK LQkJCQkgICAgTV9ORlNGTEFZT1VULCBNX1dBSVRPSyk7Ci0JCQllbHNlCi0JCQkJZmxwID0gbWFs bG9jKHNpemVvZigqZmxwKSwKLQkJCQkgICAgTV9ORlNGTEFZT1VULCBNX1dBSVRPSyk7Ci0JCQlm bHAtPm5mc2ZsX2ZsYWdzID0gMDsKLQkJCWZscC0+bmZzZmxfZmhjbnQgPSAwOwotCQkJZmxwLT5u ZnNmbF9kZXZwID0gTlVMTDsKLQkJCWZscC0+bmZzZmxfb2ZmID0gZnhkcl9oeXBlcih0bCk7IHRs ICs9IDI7Ci0JCQlyZXRsZW4gPSBmeGRyX2h5cGVyKHRsKTsgdGwgKz0gMjsKLQkJCWlmIChmbHAt Pm5mc2ZsX29mZiArIHJldGxlbiA8IGZscC0+bmZzZmxfb2ZmKQotCQkJCWZscC0+bmZzZmxfZW5k ID0gVUlOVDY0X01BWCAtIGZscC0+bmZzZmxfb2ZmOwotCQkJZWxzZQotCQkJCWZscC0+bmZzZmxf ZW5kID0gZmxwLT5uZnNmbF9vZmYgKyByZXRsZW47Ci0JCQlmbHAtPm5mc2ZsX2lvbW9kZSA9IGZ4 ZHJfdW5zaWduZWQoaW50LCAqdGwrKyk7Ci0JCQlpZiAoZ290aW9tb2RlID09IC0xKQotCQkJCWdv dGlvbW9kZSA9IGZscC0+bmZzZmxfaW9tb2RlOwotCQkJTkZTQ0xfREVCVUcoNCwgImxheWcgcmVx aW9tPSVkIHJldGlvbT0lZFxuIiwgaW9tb2RlLAotCQkJICAgIChpbnQpZmxwLT5uZnNmbF9pb21v ZGUpOwotCQkJaWYgKGZ4ZHJfdW5zaWduZWQoaW50LCAqdGwrKykgIT0KLQkJCSAgICBORlNMQVlP VVRfTkZTVjRfMV9GSUxFUykgewotCQkJCXByaW50ZigiTkZTdjQuMTogZ290IG5vbi1maWxlcyBs YXlvdXRcbiIpOwotCQkJCWVycm9yID0gTkZTRVJSX0JBRFhEUjsKLQkJCQlnb3RvIG5mc21vdXQ7 Ci0JCQl9Ci0JCQlORlNCQ09QWSgrK3RsLCBmbHAtPm5mc2ZsX2RldiwgTkZTWF9WNERFVklDRUlE KTsKLQkJCXRsICs9IChORlNYX1Y0REVWSUNFSUQgLyBORlNYX1VOU0lHTkVEKTsKLQkJCWZscC0+ bmZzZmxfdXRpbCA9IGZ4ZHJfdW5zaWduZWQodWludDMyX3QsICp0bCsrKTsKLQkJCU5GU0NMX0RF QlVHKDQsICJmbHV0aWw9MHgleFxuIiwgZmxwLT5uZnNmbF91dGlsKTsKLQkJCWZscC0+bmZzZmxf c3RyaXBlMSA9IGZ4ZHJfdW5zaWduZWQodWludDMyX3QsICp0bCsrKTsKLQkJCWZscC0+bmZzZmxf cGF0b2ZmID0gZnhkcl9oeXBlcih0bCk7IHRsICs9IDI7Ci0JCQlpZiAoZnhkcl91bnNpZ25lZChp bnQsICp0bCkgIT0gZmhjbnQpIHsKLQkJCQlwcmludGYoIkVFSyEgYmFkIGZoY250XG4iKTsKLQkJ CQllcnJvciA9IE5GU0VSUl9CQURYRFI7Ci0JCQkJZ290byBuZnNtb3V0OwotCQkJfQotCQkJZm9y IChqID0gMDsgaiA8IGZoY250OyBqKyspIHsKLQkJCQlORlNNX0RJU1NFQ1QodGwsIHVpbnQzMl90 ICosIE5GU1hfVU5TSUdORUQpOwotCQkJCW5maGxlbiA9IGZ4ZHJfdW5zaWduZWQoaW50LCAqdGwp OwotCQkJCWlmIChuZmhsZW4gPD0gMCB8fCBuZmhsZW4gPiBORlNYX1Y0RkhNQVgpIHsKLQkJCQkJ ZXJyb3IgPSBORlNFUlJfQkFEWERSOwotCQkJCQlnb3RvIG5mc21vdXQ7Ci0JCQkJfQotCQkJCW5m aHAgPSBtYWxsb2Moc2l6ZW9mKCpuZmhwKSArIG5maGxlbiAtIDEsCi0JCQkJICAgIE1fTkZTRkgs IE1fV0FJVE9LKTsKLQkJCQlmbHAtPm5mc2ZsX2ZoW2pdID0gbmZocDsKLQkJCQlmbHAtPm5mc2Zs X2ZoY250Kys7Ci0JCQkJbmZocC0+bmZoX2xlbiA9IG5maGxlbjsKLQkJCQlORlNNX0RJU1NFQ1Qo Y3AsIHVpbnQ4X3QgKiwgTkZTTV9STkRVUChuZmhsZW4pKTsKLQkJCQlORlNCQ09QWShjcCwgbmZo cC0+bmZoX2ZoLCBuZmhsZW4pOwotCQkJfQotCQkJaWYgKGZscC0+bmZzZmxfaW9tb2RlID09IGdv dGlvbW9kZSkgewotCQkJCS8qIEtlZXAgdGhlIGxpc3QgaW4gaW5jcmVhc2luZyBvZmZzZXQgb3Jk ZXIuICovCi0JCQkJdGZscCA9IExJU1RfRklSU1QoZmxocCk7Ci0JCQkJcHJldmZscCA9IE5VTEw7 Ci0JCQkJd2hpbGUgKHRmbHAgIT0gTlVMTCAmJgotCQkJCSAgICB0ZmxwLT5uZnNmbF9vZmYgPCBm bHAtPm5mc2ZsX29mZikgewotCQkJCQlwcmV2ZmxwID0gdGZscDsKLQkJCQkJdGZscCA9IExJU1Rf TkVYVCh0ZmxwLCBuZnNmbF9saXN0KTsKLQkJCQl9Ci0JCQkJaWYgKHByZXZmbHAgPT0gTlVMTCkK LQkJCQkJTElTVF9JTlNFUlRfSEVBRChmbGhwLCBmbHAsIG5mc2ZsX2xpc3QpOwotCQkJCWVsc2UK LQkJCQkJTElTVF9JTlNFUlRfQUZURVIocHJldmZscCwgZmxwLAotCQkJCQkgICAgbmZzZmxfbGlz dCk7Ci0JCQl9IGVsc2UgewotCQkJCXByaW50ZigibmZzY2xfbGF5b3V0Z2V0KCk6IGdvdCB3cm9u ZyBpb21vZGVcbiIpOwotCQkJCW5mc2NsX2ZyZWVmbGF5b3V0KGZscCk7Ci0JCQl9Ci0JCQlmbHAg PSBOVUxMOwotCQl9Ci0JfQotCWlmIChuZC0+bmRfcmVwc3RhdCAhPSAwICYmIGVycm9yID09IDAp CisJaWYgKG5kLT5uZF9yZXBzdGF0ID09IDApCisJCWVycm9yID0gbmZzcnZfcGFyc2VsYXlvdXRn ZXQobmQsIHN0YXRlaWRwLCByZXRvbmNsb3NlcCwgZmxocCk7CisJaWYgKGVycm9yID09IDAgJiYg bmQtPm5kX3JlcHN0YXQgIT0gMCkKIAkJZXJyb3IgPSBuZC0+bmRfcmVwc3RhdDsKLW5mc21vdXQ6 Ci0JaWYgKGVycm9yICE9IDAgJiYgZmxwICE9IE5VTEwpCi0JCW5mc2NsX2ZyZWVmbGF5b3V0KGZs cCk7CiAJbWJ1Zl9mcmVlbShuZC0+bmRfbXJlcCk7CiAJcmV0dXJuIChlcnJvcik7CiB9CkBAIC01 MjA5LDggKzUxMjcsNyBAQCBuZnNycGNfZ2V0bGF5b3V0KHN0cnVjdCBuZnNtb3VudCAqbm1wLCB2 CiAgICAgc3RydWN0IG5mc2NsbGF5b3V0ICoqbHlwcCwgc3RydWN0IHVjcmVkICpjcmVkLCBORlNQ Uk9DX1QgKnApCiB7CiAJc3RydWN0IG5mc2NsbGF5b3V0ICpseXA7Ci0Jc3RydWN0IG5mc2NsZmxh eW91dCAqZmxwLCAqdGZscDsKLQlzdHJ1Y3QgbmZzY2xkZXZpbmZvICpkaXA7CisJc3RydWN0IG5m c2NsZmxheW91dCAqZmxwOwogCXN0cnVjdCBuZnNjbGZsYXlvdXRoZWFkIGZsaDsKIAlpbnQgZXJy b3IgPSAwLCBpc2xvY2tlZCwgbGF5b3V0bGVuLCByZWNhbGxlZCwgcmV0b25jbG9zZTsKIAluZnN2 NHN0YXRlaWRfdCBzdGF0ZWlkOwpAQCAtNTI1MiwzNSArNTE2OSwxMyBAQCBuZnNycGNfZ2V0bGF5 b3V0KHN0cnVjdCBuZnNtb3VudCAqbm1wLCB2CiAJCQkgICAgKHVpbnQ2NF90KTAsIGxheW91dGxl biwgJnN0YXRlaWQsICZyZXRvbmNsb3NlLAogCQkJICAgICZmbGgsIGNyZWQsIHAsIE5VTEwpOwog CQl9CisJCWVycm9yID0gbmZzcnBjX2xheW91dGdldHJlcyhubXAsIHZwLCBuZmhwLT5uZmhfZmgs CisJCSAgICBuZmhwLT5uZmhfbGVuLCAmc3RhdGVpZCwgcmV0b25jbG9zZSwgbm90aWZ5Yml0c3As ICZseXAsCisJCSAgICAmZmxoLCBlcnJvciwgTlVMTCwgY3JlZCwgcCk7CiAJCWlmIChlcnJvciA9 PSAwKQotCQkJTElTVF9GT1JFQUNIKHRmbHAsICZmbGgsIG5mc2ZsX2xpc3QpIHsKLQkJCQllcnJv ciA9IG5mc2NsX2FkZGRldmluZm8obm1wLCBOVUxMLCB0ZmxwKTsKLQkJCQlpZiAoZXJyb3IgIT0g MCkgewotCQkJCQllcnJvciA9IG5mc3JwY19nZXRkZXZpY2VpbmZvKG5tcCwKLQkJCQkJICAgIHRm bHAtPm5mc2ZsX2RldiwKLQkJCQkJICAgIE5GU0xBWU9VVF9ORlNWNF8xX0ZJTEVTLAotCQkJCQkg ICAgbm90aWZ5Yml0c3AsICZkaXAsIGNyZWQsIHApOwotCQkJCQlpZiAoZXJyb3IgIT0gMCkKLQkJ CQkJCWJyZWFrOwotCQkJCQllcnJvciA9IG5mc2NsX2FkZGRldmluZm8obm1wLCBkaXAsCi0JCQkJ CSAgICB0ZmxwKTsKLQkJCQkJaWYgKGVycm9yICE9IDApCi0JCQkJCQlwcmludGYoCi0JCQkJCQkg ICAgImdldGxheW91dDogY2Fubm90IGFkZFxuIik7Ci0JCQkJfQotCQkJfQotCQlpZiAoZXJyb3Ig PT0gMCkgewotCQkJLyoKLQkJCSAqIG5mc2NsX2xheW91dCgpIGFsd2F5cyByZXR1cm5zIHdpdGgg dGhlIG5mc2x5X2xvY2sKLQkJCSAqIHNldCB0byBhIHJlZmNudCAoc2hhcmVkIGxvY2spLgotCQkJ ICovCi0JCQllcnJvciA9IG5mc2NsX2xheW91dChubXAsIHZwLCBuZmhwLT5uZmhfZmgsCi0JCQkg ICAgbmZocC0+bmZoX2xlbiwgJnN0YXRlaWQsIHJldG9uY2xvc2UsICZmbGgsICZseXAsCi0JCQkg ICAgY3JlZCwgcCk7Ci0JCQlpZiAoZXJyb3IgPT0gMCkKLQkJCQkqbHlwcCA9IGx5cDsKLQkJfSBl bHNlIGlmIChpc2xvY2tlZCAhPSAwKQotCQkJbmZzdjRfdW5sb2NrKCZseXAtPm5mc2x5X2xvY2ss IDApOworCQkJKmx5cHAgPSBseXA7CisJCWVsc2UgaWYgKGlzbG9ja2VkICE9IDApCisJCQluZnNj bF9yZWxsYXlvdXQobHlwLCAxKTsKIAl9IGVsc2UKIAkJKmx5cHAgPSBseXA7CiAJcmV0dXJuIChl cnJvcik7CkBAIC02MDAzLDMgKzU4OTgsNzU1IEBAIG5mc21vdXQ6CiB9CiAjZW5kaWYKIAorLyoK KyAqIFNldCB1cCB0aGUgWERSIGFyZ3VtZW50cyBmb3IgdGhlIExheW91dEdldCBvcGVyYXRpb24u CisgKi8KK3N0YXRpYyB2b2lkCituZnNydl9zZXR1cGxheW91dGdldChzdHJ1Y3QgbmZzcnZfZGVz Y3JpcHQgKm5kLCBpbnQgaW9tb2RlLCB1aW50NjRfdCBvZmZzZXQsCisgICAgdWludDY0X3QgbGVu LCB1aW50NjRfdCBtaW5sZW4sIG5mc3Y0c3RhdGVpZF90ICpzdGF0ZWlkcCwgaW50IGxheW91dGxl biwKKyAgICBpbnQgdXNlY3Vyc3RhdGVpZCkKK3sKKwl1aW50MzJfdCAqdGw7CisKKwlORlNNX0JV SUxEKHRsLCB1aW50MzJfdCAqLCA0ICogTkZTWF9VTlNJR05FRCArIDMgKiBORlNYX0hZUEVSICsK KwkgICAgTkZTWF9TVEFURUlEKTsKKwkqdGwrKyA9IG5ld25mc19mYWxzZTsJCS8qIERvbid0IHNp Z25hbCBhdmFpbGFiaWxpdHkuICovCisJKnRsKysgPSB0eGRyX3Vuc2lnbmVkKE5GU0xBWU9VVF9O RlNWNF8xX0ZJTEVTKTsKKwkqdGwrKyA9IHR4ZHJfdW5zaWduZWQoaW9tb2RlKTsKKwl0eGRyX2h5 cGVyKG9mZnNldCwgdGwpOworCXRsICs9IDI7CisJdHhkcl9oeXBlcihsZW4sIHRsKTsKKwl0bCAr PSAyOworCXR4ZHJfaHlwZXIobWlubGVuLCB0bCk7CisJdGwgKz0gMjsKKwlpZiAodXNlY3Vyc3Rh dGVpZCAhPSAwKSB7CisJCS8qIFNwZWNpYWwgc3RhdGVpZCBmb3IgQ3VycmVudCBzdGF0ZWlkLiAq LworCQkqdGwrKyA9IHR4ZHJfdW5zaWduZWQoMSk7CisJCSp0bCsrID0gMDsKKwkJKnRsKysgPSAw OworCQkqdGwrKyA9IDA7CisJfSBlbHNlIHsKKwkJKnRsKysgPSB0eGRyX3Vuc2lnbmVkKHN0YXRl aWRwLT5zZXFpZCk7CisJCU5GU0NMX0RFQlVHKDQsICJsYXlnZXQgc2VxPSVkXG4iLCAoaW50KXN0 YXRlaWRwLT5zZXFpZCk7CisJCSp0bCsrID0gc3RhdGVpZHAtPm90aGVyWzBdOworCQkqdGwrKyA9 IHN0YXRlaWRwLT5vdGhlclsxXTsKKwkJKnRsKysgPSBzdGF0ZWlkcC0+b3RoZXJbMl07CisJfQor CSp0bCA9IHR4ZHJfdW5zaWduZWQobGF5b3V0bGVuKTsKK30KKworLyoKKyAqIFBhcnNlIHRoZSBy ZXBseSBmb3IgYSBzdWNjZXNzZnVsIExheW91dEdldCBvcGVyYXRpb24uCisgKi8KK3N0YXRpYyBp bnQKK25mc3J2X3BhcnNlbGF5b3V0Z2V0KHN0cnVjdCBuZnNydl9kZXNjcmlwdCAqbmQsIG5mc3Y0 c3RhdGVpZF90ICpzdGF0ZWlkcCwKKyAgICBpbnQgKnJldG9uY2xvc2VwLCBzdHJ1Y3QgbmZzY2xm bGF5b3V0aGVhZCAqZmxocCkKK3sKKwl1aW50MzJfdCAqdGw7CisJc3RydWN0IG5mc2NsZmxheW91 dCAqZmxwLCAqcHJldmZscCwgKnRmbHA7CisJaW50IGNudCwgZXJyb3IsIGdvdGlvbW9kZSwgZmhj bnQsIG5maGxlbiwgaSwgajsKKwl1aW50NjRfdCByZXRsZW47CisJc3RydWN0IG5mc2ZoICpuZmhw OworCXVpbnQ4X3QgKmNwOworCisJZXJyb3IgPSAwOworCWZscCA9IE5VTEw7CisJZ290aW9tb2Rl ID0gLTE7CisJTkZTTV9ESVNTRUNUKHRsLCB1aW50MzJfdCAqLCAyICogTkZTWF9VTlNJR05FRCAr IE5GU1hfU1RBVEVJRCk7CisJaWYgKCp0bCsrICE9IDApCisJCSpyZXRvbmNsb3NlcCA9IDE7CisJ ZWxzZQorCQkqcmV0b25jbG9zZXAgPSAwOworCXN0YXRlaWRwLT5zZXFpZCA9IGZ4ZHJfdW5zaWdu ZWQodWludDMyX3QsICp0bCsrKTsKKwlORlNDTF9ERUJVRyg0LCAicmV0b25jbHM9JWQgc3RzZXE9 JWRcbiIsICpyZXRvbmNsb3NlcCwKKwkgICAgKGludClzdGF0ZWlkcC0+c2VxaWQpOworCXN0YXRl aWRwLT5vdGhlclswXSA9ICp0bCsrOworCXN0YXRlaWRwLT5vdGhlclsxXSA9ICp0bCsrOworCXN0 YXRlaWRwLT5vdGhlclsyXSA9ICp0bCsrOworCWNudCA9IGZ4ZHJfdW5zaWduZWQoaW50LCAqdGwp OworCU5GU0NMX0RFQlVHKDQsICJsYXlnIGNudD0lZFxuIiwgY250KTsKKwlpZiAoY250IDw9IDAg fHwgY250ID4gMTAwMDApIHsKKwkJLyogRG9uJ3QgYWNjZXB0IG1vcmUgdGhhbiAxMDAwMCBsYXlv dXRzIGluIHJlcGx5LiAqLworCQllcnJvciA9IE5GU0VSUl9CQURYRFI7CisJCWdvdG8gbmZzbW91 dDsKKwl9CisJZm9yIChpID0gMDsgaSA8IGNudDsgaSsrKSB7CisJCS8qIERpc3NlY3QgYWxsIHRo ZSB3YXkgdG8gdGhlIGZpbGUgaGFuZGxlIGNudC4gKi8KKwkJTkZTTV9ESVNTRUNUKHRsLCB1aW50 MzJfdCAqLCAzICogTkZTWF9IWVBFUiArCisJCSAgICA2ICogTkZTWF9VTlNJR05FRCArIE5GU1hf VjRERVZJQ0VJRCk7CisJCWZoY250ID0gZnhkcl91bnNpZ25lZChpbnQsICoodGwgKyAxMSArCisJ CSAgICBORlNYX1Y0REVWSUNFSUQgLyBORlNYX1VOU0lHTkVEKSk7CisJCU5GU0NMX0RFQlVHKDQs ICJmaGNudD0lZFxuIiwgZmhjbnQpOworCQlpZiAoZmhjbnQgPCAwIHx8IGZoY250ID4gMTAwKSB7 CisJCQkvKiBEb24ndCBhY2NlcHQgbW9yZSB0aGFuIDEwMCBmaWxlIGhhbmRsZXMuICovCisJCQll cnJvciA9IE5GU0VSUl9CQURYRFI7CisJCQlnb3RvIG5mc21vdXQ7CisJCX0KKwkJaWYgKGZoY250 ID4gMSkKKwkJCWZscCA9IG1hbGxvYyhzaXplb2YoKmZscCkgKyAoZmhjbnQgLSAxKSAqCisJCQkg ICAgc2l6ZW9mKHN0cnVjdCBuZnNmaCAqKSwgTV9ORlNGTEFZT1VULCBNX1dBSVRPSyk7CisJCWVs c2UKKwkJCWZscCA9IG1hbGxvYyhzaXplb2YoKmZscCksIE1fTkZTRkxBWU9VVCwgTV9XQUlUT0sp OworCQlmbHAtPm5mc2ZsX2ZsYWdzID0gMDsKKwkJZmxwLT5uZnNmbF9maGNudCA9IDA7CisJCWZs cC0+bmZzZmxfZGV2cCA9IE5VTEw7CisJCWZscC0+bmZzZmxfb2ZmID0gZnhkcl9oeXBlcih0bCk7 IHRsICs9IDI7CisJCXJldGxlbiA9IGZ4ZHJfaHlwZXIodGwpOyB0bCArPSAyOworCQlpZiAoZmxw LT5uZnNmbF9vZmYgKyByZXRsZW4gPCBmbHAtPm5mc2ZsX29mZikKKwkJCWZscC0+bmZzZmxfZW5k ID0gVUlOVDY0X01BWCAtIGZscC0+bmZzZmxfb2ZmOworCQllbHNlCisJCQlmbHAtPm5mc2ZsX2Vu ZCA9IGZscC0+bmZzZmxfb2ZmICsgcmV0bGVuOworCQlmbHAtPm5mc2ZsX2lvbW9kZSA9IGZ4ZHJf dW5zaWduZWQoaW50LCAqdGwrKyk7CisJCWlmIChnb3Rpb21vZGUgPT0gLTEpCisJCQlnb3Rpb21v ZGUgPSBmbHAtPm5mc2ZsX2lvbW9kZTsKKwkJaWYgKGZ4ZHJfdW5zaWduZWQoaW50LCAqdGwrKykg IT0gTkZTTEFZT1VUX05GU1Y0XzFfRklMRVMpIHsKKwkJCXByaW50ZigiTkZTdjQuMTogZ290IG5v bi1maWxlcyBsYXlvdXRcbiIpOworCQkJZXJyb3IgPSBORlNFUlJfQkFEWERSOworCQkJZ290byBu ZnNtb3V0OworCQl9CisJCU5GU0JDT1BZKCsrdGwsIGZscC0+bmZzZmxfZGV2LCBORlNYX1Y0REVW SUNFSUQpOworCQl0bCArPSAoTkZTWF9WNERFVklDRUlEIC8gTkZTWF9VTlNJR05FRCk7CisJCWZs cC0+bmZzZmxfdXRpbCA9IGZ4ZHJfdW5zaWduZWQodWludDMyX3QsICp0bCsrKTsKKwkJTkZTQ0xf REVCVUcoNCwgImZsdXRpbD0weCV4XG4iLCBmbHAtPm5mc2ZsX3V0aWwpOworCQlmbHAtPm5mc2Zs X3N0cmlwZTEgPSBmeGRyX3Vuc2lnbmVkKHVpbnQzMl90LCAqdGwrKyk7CisJCWZscC0+bmZzZmxf cGF0b2ZmID0gZnhkcl9oeXBlcih0bCk7IHRsICs9IDI7CisJCWlmIChmeGRyX3Vuc2lnbmVkKGlu dCwgKnRsKSAhPSBmaGNudCkgeworCQkJcHJpbnRmKCJFRUshIGJhZCBmaGNudFxuIik7CisJCQll cnJvciA9IE5GU0VSUl9CQURYRFI7CisJCQlnb3RvIG5mc21vdXQ7CisJCX0KKwkJZm9yIChqID0g MDsgaiA8IGZoY250OyBqKyspIHsKKwkJCU5GU01fRElTU0VDVCh0bCwgdWludDMyX3QgKiwgTkZT WF9VTlNJR05FRCk7CisJCQluZmhsZW4gPSBmeGRyX3Vuc2lnbmVkKGludCwgKnRsKTsKKwkJCWlm IChuZmhsZW4gPD0gMCB8fCBuZmhsZW4gPiBORlNYX1Y0RkhNQVgpIHsKKwkJCQllcnJvciA9IE5G U0VSUl9CQURYRFI7CisJCQkJZ290byBuZnNtb3V0OworCQkJfQorCQkJbmZocCA9IG1hbGxvYyhz aXplb2YoKm5maHApICsgbmZobGVuIC0gMSwgTV9ORlNGSCwKKwkJCSAgICBNX1dBSVRPSyk7CisJ CQlmbHAtPm5mc2ZsX2ZoW2pdID0gbmZocDsKKwkJCWZscC0+bmZzZmxfZmhjbnQrKzsKKwkJCW5m aHAtPm5maF9sZW4gPSBuZmhsZW47CisJCQlORlNNX0RJU1NFQ1QoY3AsIHVpbnQ4X3QgKiwgTkZT TV9STkRVUChuZmhsZW4pKTsKKwkJCU5GU0JDT1BZKGNwLCBuZmhwLT5uZmhfZmgsIG5maGxlbik7 CisJCX0KKwkJaWYgKGZscC0+bmZzZmxfaW9tb2RlID09IGdvdGlvbW9kZSkgeworCQkJLyogS2Vl cCB0aGUgbGlzdCBpbiBpbmNyZWFzaW5nIG9mZnNldCBvcmRlci4gKi8KKwkJCXRmbHAgPSBMSVNU X0ZJUlNUKGZsaHApOworCQkJcHJldmZscCA9IE5VTEw7CisJCQl3aGlsZSAodGZscCAhPSBOVUxM ICYmCisJCQkgICAgdGZscC0+bmZzZmxfb2ZmIDwgZmxwLT5uZnNmbF9vZmYpIHsKKwkJCQlwcmV2 ZmxwID0gdGZscDsKKwkJCQl0ZmxwID0gTElTVF9ORVhUKHRmbHAsIG5mc2ZsX2xpc3QpOworCQkJ fQorCQkJaWYgKHByZXZmbHAgPT0gTlVMTCkKKwkJCQlMSVNUX0lOU0VSVF9IRUFEKGZsaHAsIGZs cCwgbmZzZmxfbGlzdCk7CisJCQllbHNlCisJCQkJTElTVF9JTlNFUlRfQUZURVIocHJldmZscCwg ZmxwLAorCQkJCSAgICBuZnNmbF9saXN0KTsKKwkJfSBlbHNlIHsKKwkJCXByaW50ZigibmZzY2xf bGF5b3V0Z2V0KCk6IGdvdCB3cm9uZyBpb21vZGVcbiIpOworCQkJbmZzY2xfZnJlZWZsYXlvdXQo ZmxwKTsKKwkJfQorCQlmbHAgPSBOVUxMOworCX0KK25mc21vdXQ6CisJaWYgKGVycm9yICE9IDAg JiYgZmxwICE9IE5VTEwpCisJCW5mc2NsX2ZyZWVmbGF5b3V0KGZscCk7CisJcmV0dXJuIChlcnJv cik7Cit9CisKKy8qCisgKiBTaW1pbGFyIHRvIG5mc3JwY19nZXRsYXlvdXQoKSwgZXhjZXB0IHRo YXQgaXQgdXNlcyBuZnNycGNfb3BlbmxheWdldCgpLAorICogc28gdGhhdCBpdCBkb2VzIGJvdGgg YW4gT3BlbiBhbmQgYSBMYXlvdXRnZXQuCisgKi8KK3N0YXRpYyBpbnQKK25mc3JwY19nZXRvcGVu bGF5b3V0KHN0cnVjdCBuZnNtb3VudCAqbm1wLCB2bm9kZV90IHZwLCB1X2ludDhfdCAqbmZocCwK KyAgICBpbnQgZmhsZW4sIHVpbnQ4X3QgKm5ld2ZocCwgaW50IG5ld2ZobGVuLCB1aW50MzJfdCBt b2RlLAorICAgIHN0cnVjdCBuZnNjbG9wZW4gKm9wLCB1aW50OF90ICpuYW1lLCBpbnQgbmFtZWxl biwgc3RydWN0IG5mc2NsZGVsZWcgKipkcHAsCisgICAgc3RydWN0IHVjcmVkICpjcmVkLCBORlNQ Uk9DX1QgKnApCit7CisJc3RydWN0IG5mc2NsbGF5b3V0ICpseXA7CisJc3RydWN0IG5mc2NsZmxh eW91dCAqZmxwOworCXN0cnVjdCBuZnNjbGZsYXlvdXRoZWFkIGZsaDsKKwlpbnQgZXJyb3IsIGlz bG9ja2VkLCBsYXlvdXRsZW4sIHJlY2FsbGVkLCByZXRvbmNsb3NlLCB1c2VjdXJzdGF0ZWlkOwor CWludCBsYXlzdGF0OworCW5mc3Y0c3RhdGVpZF90IHN0YXRlaWQ7CisJc3RydWN0IG5mc2Nsc2Vz c2lvbiAqdHNlcDsKKworCWVycm9yID0gMDsKKwkvKgorCSAqIElmIGx5cCBpcyByZXR1cm5lZCBu b24tTlVMTCwgdGhlcmUgd2lsbCBiZSBhIHJlZmNudCAoc2hhcmVkIGxvY2spCisJICogb24gaXQs IGlmZiBmbHAgIT0gTlVMTCBvciBhIGxvY2sgKGV4Y2x1c2l2ZSBsb2NrKSBvbiBpdCBpZmYKKwkg KiBmbHAgPT0gTlVMTC4KKwkgKi8KKwlseXAgPSBuZnNjbF9nZXRsYXlvdXQobm1wLT5ubV9jbHAs IG5ld2ZocCwgbmV3ZmhsZW4sIDAsICZmbHAsCisJICAgICZyZWNhbGxlZCk7CisJTkZTQ0xfREVC VUcoNCwgIm5mc3JwY19nZXRvcGVubGF5b3V0IG5mc2NsX2dldGxheW91dCBseXA9JXBcbiIsIGx5 cCk7CisJaWYgKGx5cCA9PSBOVUxMKQorCQlpc2xvY2tlZCA9IDA7CisJZWxzZSBpZiAoZmxwICE9 IE5VTEwpCisJCWlzbG9ja2VkID0gMTsKKwllbHNlCisJCWlzbG9ja2VkID0gMjsKKwlpZiAoKGx5 cCA9PSBOVUxMIHx8IGZscCA9PSBOVUxMKSAmJiByZWNhbGxlZCA9PSAwKSB7CisJCUxJU1RfSU5J VCgmZmxoKTsKKwkJdHNlcCA9IG5mc21udF9tZHNzZXNzaW9uKG5tcCk7CisJCWxheW91dGxlbiA9 IHRzZXAtPm5mc2Vzc19tYXhjYWNoZSAtIChORlNYX1NUQVRFSUQgKworCQkgICAgMyAqIE5GU1hf VU5TSUdORUQpOworCQlpZiAobHlwID09IE5VTEwpCisJCQl1c2VjdXJzdGF0ZWlkID0gMTsKKwkJ ZWxzZSB7CisJCQl1c2VjdXJzdGF0ZWlkID0gMDsKKwkJCXN0YXRlaWQuc2VxaWQgPSBseXAtPm5m c2x5X3N0YXRlaWQuc2VxaWQ7CisJCQlzdGF0ZWlkLm90aGVyWzBdID0gbHlwLT5uZnNseV9zdGF0 ZWlkLm90aGVyWzBdOworCQkJc3RhdGVpZC5vdGhlclsxXSA9IGx5cC0+bmZzbHlfc3RhdGVpZC5v dGhlclsxXTsKKwkJCXN0YXRlaWQub3RoZXJbMl0gPSBseXAtPm5mc2x5X3N0YXRlaWQub3RoZXJb Ml07CisJCX0KKwkJZXJyb3IgPSBuZnNycGNfb3BlbmxheW91dHJwYyhubXAsIHZwLCBuZmhwLCBm aGxlbiwKKwkJICAgIG5ld2ZocCwgbmV3ZmhsZW4sIG1vZGUsIG9wLCBuYW1lLCBuYW1lbGVuLAor CQkgICAgZHBwLCAmc3RhdGVpZCwgdXNlY3Vyc3RhdGVpZCwgbGF5b3V0bGVuLAorCQkgICAgJnJl dG9uY2xvc2UsICZmbGgsICZsYXlzdGF0LCBjcmVkLCBwKTsKKwkJTkZTQ0xfREVCVUcoNCwgImFm dCBuZnNycGNfb3BlbmxheW91dHJwYyBsYXlzdGF0PSVkIGVycj0lZFxuIiwKKwkJICAgIGxheXN0 YXQsIGVycm9yKTsKKwkJbGF5c3RhdCA9IG5mc3JwY19sYXlvdXRnZXRyZXMobm1wLCB2cCwgbmV3 ZmhwLCBuZXdmaGxlbiwKKwkJICAgICZzdGF0ZWlkLCByZXRvbmNsb3NlLCBOVUxMLCAmbHlwLCAm ZmxoLCBsYXlzdGF0LCAmaXNsb2NrZWQsCisJCSAgICBjcmVkLCBwKTsKKwl9IGVsc2UKKwkJZXJy b3IgPSBuZnNycGNfb3BlbnJwYyhubXAsIHZwLCBuZmhwLCBmaGxlbiwgbmV3ZmhwLCBuZXdmaGxl biwKKwkJICAgIG1vZGUsIG9wLCBuYW1lLCBuYW1lbGVuLCBkcHAsIDAsIDAsIGNyZWQsIHAsIDAs IDApOworCWlmIChpc2xvY2tlZCA9PSAyKQorCQluZnNjbF9yZWxsYXlvdXQobHlwLCAxKTsKKwll bHNlIGlmIChpc2xvY2tlZCA9PSAxKQorCQluZnNjbF9yZWxsYXlvdXQobHlwLCAwKTsKKwlyZXR1 cm4gKGVycm9yKTsKK30KKworLyoKKyAqIFRoaXMgZnVuY3Rpb24gZG9lcyBhbiBPcGVuK0xheW91 dEdldCBmb3IgYW4gTkZTdjQuMSBtb3VudCB3aXRoIHBORlMKKyAqIGVuYWJsZWQsIG9ubHkgZm9y IHRoZSBDTEFJTV9OVUxMIGNhc2UuICBBbGwgb3RoZXIgTkZTdjQgT3BlbnMgYXJlCisgKiBoYW5k bGVkIGJ5IG5mc3JwY19vcGVucnBjKCkuCisgKiBGb3IgdGhlIGNhc2Ugd2hlcmUgb3AgPT0gTlVM TCwgZHZwIGlzIHRoZSBkaXJlY3RvcnkuICBXaGVuIG9wICE9IE5VTEwsIGl0CisgKiBjYW4gYmUg TlVMTC4KKyAqLworc3RhdGljIGludAorbmZzcnBjX29wZW5sYXlvdXRycGMoc3RydWN0IG5mc21v dW50ICpubXAsIHZub2RlX3QgdnAsIHVfaW50OF90ICpuZmhwLAorICAgIGludCBmaGxlbiwgdWlu dDhfdCAqbmV3ZmhwLCBpbnQgbmV3ZmhsZW4sIHVpbnQzMl90IG1vZGUsCisgICAgc3RydWN0IG5m c2Nsb3BlbiAqb3AsIHVpbnQ4X3QgKm5hbWUsIGludCBuYW1lbGVuLCBzdHJ1Y3QgbmZzY2xkZWxl ZyAqKmRwcCwKKyAgICBuZnN2NHN0YXRlaWRfdCAqc3RhdGVpZHAsIGludCB1c2VjdXJzdGF0ZWlk LAorICAgIGludCBsYXlvdXRsZW4sIGludCAqcmV0b25jbG9zZXAsIHN0cnVjdCBuZnNjbGZsYXlv dXRoZWFkICpmbGhwLAorICAgIGludCAqbGF5c3RhdHAsIHN0cnVjdCB1Y3JlZCAqY3JlZCwgTkZT UFJPQ19UICpwKQoreworCXVpbnQzMl90ICp0bDsKKwlzdHJ1Y3QgbmZzcnZfZGVzY3JpcHQgbmZz ZCwgKm5kID0gJm5mc2Q7CisJc3RydWN0IG5mc2NsZGVsZWcgKm5kcCA9IE5VTEw7CisJc3RydWN0 IG5mc3ZhdHRyIG5mc3ZhOworCXN0cnVjdCBuZnNjbHNlc3Npb24gKnRzZXA7CisJdWludDMyX3Qg cmZsYWdzLCBkZWxlZzsKKwluZnNhdHRyYml0X3QgYXR0cmJpdHM7CisJaW50IGVycm9yLCByZXQs IGFjZXNpemUsIGxpbWl0YnksIGlvbW9kZTsKKworCSpkcHAgPSBOVUxMOworCSpsYXlzdGF0cCA9 IEVOWElPOworCW5mc2NsX3JlcXN0YXJ0KG5kLCBORlNQUk9DX09QRU5MQVlHRVQsIG5tcCwgbmZo cCwgZmhsZW4sIE5VTEwsIE5VTEwpOworCU5GU01fQlVJTEQodGwsIHVpbnQzMl90ICosIDUgKiBO RlNYX1VOU0lHTkVEKTsKKwkqdGwrKyA9IHR4ZHJfdW5zaWduZWQob3AtPm5mc29fb3duLT5uZnNv d19zZXFpZCk7CisJKnRsKysgPSB0eGRyX3Vuc2lnbmVkKG1vZGUgJiBORlNWNE9QRU5fQUNDRVNT Qk9USCk7CisJKnRsKysgPSB0eGRyX3Vuc2lnbmVkKChtb2RlID4+IE5GU0xDS19TSElGVCkgJiBO RlNWNE9QRU5fREVOWUJPVEgpOworCXRzZXAgPSBuZnNtbnRfbWRzc2Vzc2lvbihubXApOworCSp0 bCsrID0gdHNlcC0+bmZzZXNzX2NsaWVudGlkLmx2YWxbMF07CisJKnRsID0gdHNlcC0+bmZzZXNz X2NsaWVudGlkLmx2YWxbMV07CisJbmZzbV9zdHJ0b20obmQsIG9wLT5uZnNvX293bi0+bmZzb3df b3duZXIsIE5GU1Y0Q0xfTE9DS05BTUVMRU4pOworCU5GU01fQlVJTEQodGwsIHVpbnQzMl90ICos IDIgKiBORlNYX1VOU0lHTkVEKTsKKwkqdGwrKyA9IHR4ZHJfdW5zaWduZWQoTkZTVjRPUEVOX05P Q1JFQVRFKTsKKwkqdGwgPSB0eGRyX3Vuc2lnbmVkKE5GU1Y0T1BFTl9DTEFJTU5VTEwpOworCW5m c21fc3RydG9tKG5kLCBuYW1lLCBuYW1lbGVuKTsKKwlORlNNX0JVSUxEKHRsLCB1aW50MzJfdCAq LCBORlNYX1VOU0lHTkVEKTsKKwkqdGwgPSB0eGRyX3Vuc2lnbmVkKE5GU1Y0T1BfR0VUQVRUUik7 CisJTkZTWkVST19BVFRSQklUKCZhdHRyYml0cyk7CisJTkZTU0VUQklUX0FUVFJCSVQoJmF0dHJi aXRzLCBORlNBVFRSQklUX0NIQU5HRSk7CisJTkZTU0VUQklUX0FUVFJCSVQoJmF0dHJiaXRzLCBO RlNBVFRSQklUX1RJTUVNT0RJRlkpOworCW5mc3J2X3B1dGF0dHJiaXQobmQsICZhdHRyYml0cyk7 CisJTkZTTV9CVUlMRCh0bCwgdWludDMyX3QgKiwgTkZTWF9VTlNJR05FRCk7CisJKnRsID0gdHhk cl91bnNpZ25lZChORlNWNE9QX0xBWU9VVEdFVCk7CisJaWYgKChtb2RlICYgTkZTVjRPUEVOX0FD Q0VTU1dSSVRFKSAhPSAwKQorCQlpb21vZGUgPSBORlNMQVlPVVRJT01PREVfUlc7CisJZWxzZQor CQlpb21vZGUgPSBORlNMQVlPVVRJT01PREVfUkVBRDsKKwluZnNydl9zZXR1cGxheW91dGdldChu ZCwgaW9tb2RlLCAwLCBVSU5UNjRfTUFYLCAwLCBzdGF0ZWlkcCwKKwkgICAgbGF5b3V0bGVuLCB1 c2VjdXJzdGF0ZWlkKTsKKwllcnJvciA9IG5ld25mc19yZXF1ZXN0KG5kLCBubXAsIE5VTEwsICZu bXAtPm5tX3NvY2tyZXEsIHZwLCBwLCBjcmVkLAorCSAgICBORlNfUFJPRywgTkZTX1ZFUjQsIE5V TEwsIDEsIE5VTEwsIE5VTEwpOworCWlmIChlcnJvciAhPSAwKQorCQlyZXR1cm4gKGVycm9yKTsK KwlORlNDTF9JTkNSU0VRSUQob3AtPm5mc29fb3duLT5uZnNvd19zZXFpZCwgbmQpOworCWlmIChu ZC0+bmRfcmVwc3RhdCAhPSAwKQorCQkqbGF5c3RhdHAgPSBuZC0+bmRfcmVwc3RhdDsKKwlpZiAo KG5kLT5uZF9mbGFnICYgTkRfTk9NT1JFREFUQSkgPT0gMCkgeworCQkvKiBORF9OT01PUkVEQVRB IHdpbGwgYmUgc2V0IGlmIHRoZSBPcGVuIG9wZXJhdGlvbiBmYWlsZWQuICovCisJCU5GU01fRElT U0VDVCh0bCwgdV9pbnQzMl90ICosIE5GU1hfU1RBVEVJRCArCisJCSAgICA2ICogTkZTWF9VTlNJ R05FRCk7CisJCW9wLT5uZnNvX3N0YXRlaWQuc2VxaWQgPSAqdGwrKzsKKwkJb3AtPm5mc29fc3Rh dGVpZC5vdGhlclswXSA9ICp0bCsrOworCQlvcC0+bmZzb19zdGF0ZWlkLm90aGVyWzFdID0gKnRs Kys7CisJCW9wLT5uZnNvX3N0YXRlaWQub3RoZXJbMl0gPSAqdGw7CisJCXJmbGFncyA9IGZ4ZHJf dW5zaWduZWQodV9pbnQzMl90LCAqKHRsICsgNikpOworCQllcnJvciA9IG5mc3J2X2dldGF0dHJi aXRzKG5kLCAmYXR0cmJpdHMsIE5VTEwsIE5VTEwpOworCQlpZiAoZXJyb3IgIT0gMCkKKwkJCWdv dG8gbmZzbW91dDsKKwkJTkZTTV9ESVNTRUNUKHRsLCB1X2ludDMyX3QgKiwgTkZTWF9VTlNJR05F RCk7CisJCWRlbGVnID0gZnhkcl91bnNpZ25lZCh1X2ludDMyX3QsICp0bCk7CisJCWlmIChkZWxl ZyA9PSBORlNWNE9QRU5fREVMRUdBVEVSRUFEIHx8CisJCSAgICBkZWxlZyA9PSBORlNWNE9QRU5f REVMRUdBVEVXUklURSkgeworCQkJaWYgKCEob3AtPm5mc29fb3duLT5uZnNvd19jbHAtPm5mc2Nf ZmxhZ3MgJgorCQkJICAgICAgTkZTQ0xGTEFHU19GSVJTVERFTEVHKSkKKwkJCQlvcC0+bmZzb19v d24tPm5mc293X2NscC0+bmZzY19mbGFncyB8PQorCQkJCSAgKE5GU0NMRkxBR1NfRklSU1RERUxF RyB8IE5GU0NMRkxBR1NfR09UREVMRUcpOworCQkJbmRwID0gbWFsbG9jKHNpemVvZihzdHJ1Y3Qg bmZzY2xkZWxlZykgKyBuZXdmaGxlbiwKKwkJCSAgICBNX05GU0NMREVMRUcsIE1fV0FJVE9LKTsK KwkJCUxJU1RfSU5JVCgmbmRwLT5uZnNkbF9vd25lcik7CisJCQlMSVNUX0lOSVQoJm5kcC0+bmZz ZGxfbG9jayk7CisJCQluZHAtPm5mc2RsX2NscCA9IG9wLT5uZnNvX293bi0+bmZzb3dfY2xwOwor CQkJbmRwLT5uZnNkbF9maGxlbiA9IG5ld2ZobGVuOworCQkJTkZTQkNPUFkobmV3ZmhwLCBuZHAt Pm5mc2RsX2ZoLCBuZXdmaGxlbik7CisJCQluZXduZnNfY29weWluY3JlZChjcmVkLCAmbmRwLT5u ZnNkbF9jcmVkKTsKKwkJCW5mc2NsX2xvY2tpbml0KCZuZHAtPm5mc2RsX3J3bG9jayk7CisJCQlO RlNNX0RJU1NFQ1QodGwsIHVfaW50MzJfdCAqLCBORlNYX1NUQVRFSUQgKworCQkJICAgIE5GU1hf VU5TSUdORUQpOworCQkJbmRwLT5uZnNkbF9zdGF0ZWlkLnNlcWlkID0gKnRsKys7CisJCQluZHAt Pm5mc2RsX3N0YXRlaWQub3RoZXJbMF0gPSAqdGwrKzsKKwkJCW5kcC0+bmZzZGxfc3RhdGVpZC5v dGhlclsxXSA9ICp0bCsrOworCQkJbmRwLT5uZnNkbF9zdGF0ZWlkLm90aGVyWzJdID0gKnRsKys7 CisJCQlyZXQgPSBmeGRyX3Vuc2lnbmVkKGludCwgKnRsKTsKKwkJCWlmIChkZWxlZyA9PSBORlNW NE9QRU5fREVMRUdBVEVXUklURSkgeworCQkJCW5kcC0+bmZzZGxfZmxhZ3MgPSBORlNDTERMX1dS SVRFOworCQkJCS8qCisJCQkJICogSW5kaWNhdGVzIGhvdyBtdWNoIHRoZSBmaWxlIGNhbiBncm93 LgorCQkJCSAqLworCQkJCU5GU01fRElTU0VDVCh0bCwgdV9pbnQzMl90ICosCisJCQkJICAgIDMg KiBORlNYX1VOU0lHTkVEKTsKKwkJCQlsaW1pdGJ5ID0gZnhkcl91bnNpZ25lZChpbnQsICp0bCsr KTsKKwkJCQlzd2l0Y2ggKGxpbWl0YnkpIHsKKwkJCQljYXNlIE5GU1Y0T1BFTl9MSU1JVFNJWkU6 CisJCQkJCW5kcC0+bmZzZGxfc2l6ZWxpbWl0ID0gZnhkcl9oeXBlcih0bCk7CisJCQkJCWJyZWFr OworCQkJCWNhc2UgTkZTVjRPUEVOX0xJTUlUQkxPQ0tTOgorCQkJCQluZHAtPm5mc2RsX3NpemVs aW1pdCA9CisJCQkJCSAgICBmeGRyX3Vuc2lnbmVkKHVfaW50NjRfdCwgKnRsKyspOworCQkJCQlu ZHAtPm5mc2RsX3NpemVsaW1pdCAqPQorCQkJCQkgICAgZnhkcl91bnNpZ25lZCh1X2ludDY0X3Qs ICp0bCk7CisJCQkJCWJyZWFrOworCQkJCWRlZmF1bHQ6CisJCQkJCWVycm9yID0gTkZTRVJSX0JB RFhEUjsKKwkJCQkJZ290byBuZnNtb3V0OworCQkJCX07CisJCQl9IGVsc2UKKwkJCQluZHAtPm5m c2RsX2ZsYWdzID0gTkZTQ0xETF9SRUFEOworCQkJaWYgKHJldCAhPSAwKQorCQkJCW5kcC0+bmZz ZGxfZmxhZ3MgfD0gTkZTQ0xETF9SRUNBTEw7CisJCQllcnJvciA9IG5mc3J2X2Rpc3NlY3RhY2Uo bmQsICZuZHAtPm5mc2RsX2FjZSwgJnJldCwKKwkJCSAgICAmYWNlc2l6ZSwgcCk7CisJCQlpZiAo ZXJyb3IgIT0gMCkKKwkJCQlnb3RvIG5mc21vdXQ7CisJCX0gZWxzZSBpZiAoZGVsZWcgIT0gTkZT VjRPUEVOX0RFTEVHQVRFTk9ORSkgeworCQkJZXJyb3IgPSBORlNFUlJfQkFEWERSOworCQkJZ290 byBuZnNtb3V0OworCQl9CisJCWlmICgocmZsYWdzICYgTkZTVjRPUEVOX0xPQ0tUWVBFUE9TSVgp ICE9IDAgfHwKKwkJICAgIG5mc2NsX2Fzc3VtZXBvc2l4bG9ja3MpCisJCQlvcC0+bmZzb19wb3Np eGxvY2sgPSAxOworCQllbHNlCisJCQlvcC0+bmZzb19wb3NpeGxvY2sgPSAwOworCQlORlNNX0RJ U1NFQ1QodGwsIHVfaW50MzJfdCAqLCAyICogTkZTWF9VTlNJR05FRCk7CisJCS8qIElmIHRoZSAy bmQgZWxlbWVudCA9PSBORlNfT0ssIHRoZSBHZXRhdHRyIHN1Y2NlZWRlZC4gKi8KKwkJaWYgKCor K3RsID09IDApIHsKKwkJCWVycm9yID0gbmZzdjRfbG9hZGF0dHIobmQsIE5VTEwsICZuZnN2YSwg TlVMTCwKKwkJCSAgICBOVUxMLCAwLCBOVUxMLCBOVUxMLCBOVUxMLCBOVUxMLCBOVUxMLCAwLAor CQkJICAgIE5VTEwsIE5VTEwsIE5VTEwsIHAsIGNyZWQpOworCQkJaWYgKGVycm9yICE9IDApCisJ CQkJZ290byBuZnNtb3V0OworCQkJaWYgKG5kcCAhPSBOVUxMKSB7CisJCQkJbmRwLT5uZnNkbF9j aGFuZ2UgPSBuZnN2YS5uYV9maWxlcmV2OworCQkJCW5kcC0+bmZzZGxfbW9kdGltZSA9IG5mc3Zh Lm5hX210aW1lOworCQkJCW5kcC0+bmZzZGxfZmxhZ3MgfD0gTkZTQ0xETF9NT0RUSU1FU0VUOwor CQkJCSpkcHAgPSBuZHA7CisJCQkJbmRwID0gTlVMTDsKKwkJCX0KKwkJCS8qCisJCQkgKiBBdCB0 aGlzIHBvaW50LCB0aGUgT3BlbiBoYXMgc3VjY2VlZGVkLCBzbyBzZXQKKwkJCSAqIG5kX3JlcHN0 YXQgPSBORlNfT0suICBJZiB0aGUgTGF5b3V0Z2V0IGZhaWxlZCwKKwkJCSAqIHRoaXMgZnVuY3Rp b24ganVzdCB3b24ndCByZXR1cm4gYSBsYXlvdXQuCisJCQkgKi8KKwkJCWlmIChuZC0+bmRfcmVw c3RhdCA9PSAwKSB7CisJCQkJTkZTTV9ESVNTRUNUKHRsLCB1aW50MzJfdCAqLCAyICogTkZTWF9V TlNJR05FRCk7CisJCQkJKmxheXN0YXRwID0gZnhkcl91bnNpZ25lZChpbnQsICorK3RsKTsKKwkJ CQlpZiAoKmxheXN0YXRwID09IDApIHsKKwkJCQkJZXJyb3IgPSBuZnNydl9wYXJzZWxheW91dGdl dChuZCwKKwkJCQkJICAgIHN0YXRlaWRwLCByZXRvbmNsb3NlcCwgZmxocCk7CisJCQkJCWlmIChl cnJvciAhPSAwKQorCQkJCQkJKmxheXN0YXRwID0gZXJyb3I7CisJCQkJfQorCQkJfSBlbHNlCisJ CQkJbmQtPm5kX3JlcHN0YXQgPSAwOwkvKiBSZXR1cm4gMCBmb3IgT3Blbi4gKi8KKwkJfQorCX0K KwlpZiAobmQtPm5kX3JlcHN0YXQgIT0gMCAmJiBlcnJvciA9PSAwKQorCQllcnJvciA9IG5kLT5u ZF9yZXBzdGF0OworbmZzbW91dDoKKwlmcmVlKG5kcCwgTV9ORlNDTERFTEVHKTsKKwltYnVmX2Zy ZWVtKG5kLT5uZF9tcmVwKTsKKwlyZXR1cm4gKGVycm9yKTsKK30KKworLyoKKyAqIFNpbWlsYXIg bmZzcnBjX2NyZWF0ZXY0KCksIGJ1dCBhbHNvIGRvZXMgdGhlIExheW91dEdldCBvcGVyYXRpb24u CisgKiBVc2VkIG9ubHkgZm9yIG1vdW50cyB3aXRoIHBORlMgZW5hYmxlZC4KKyAqLworc3RhdGlj IGludAorbmZzcnBjX2NyZWF0ZWxheW91dCh2bm9kZV90IGR2cCwgY2hhciAqbmFtZSwgaW50IG5h bWVsZW4sIHN0cnVjdCB2YXR0ciAqdmFwLAorICAgIG5mc3F1YWRfdCBjdmVyZiwgaW50IGZtb2Rl LCBzdHJ1Y3QgbmZzY2xvd25lciAqb3dwLCBzdHJ1Y3QgbmZzY2xkZWxlZyAqKmRwcCwKKyAgICBz dHJ1Y3QgdWNyZWQgKmNyZWQsIE5GU1BST0NfVCAqcCwgc3RydWN0IG5mc3ZhdHRyICpkbmFwLAor ICAgIHN0cnVjdCBuZnN2YXR0ciAqbm5hcCwgc3RydWN0IG5mc2ZoICoqbmZocHAsIGludCAqYXR0 cmZsYWdwLAorICAgIGludCAqZGF0dHJmbGFncCwgdm9pZCAqZHN0dWZmLCBpbnQgKnVubG9ja2Vk cCwgbmZzdjRzdGF0ZWlkX3QgKnN0YXRlaWRwLAorICAgIGludCB1c2VjdXJzdGF0ZWlkLCBpbnQg bGF5b3V0bGVuLCBpbnQgKnJldG9uY2xvc2VwLAorICAgIHN0cnVjdCBuZnNjbGZsYXlvdXRoZWFk ICpmbGhwLCBpbnQgKmxheXN0YXRwKQoreworCXVpbnQzMl90ICp0bDsKKwlpbnQgZXJyb3IgPSAw LCBkZWxlZywgbmV3b25lLCByZXQsIGFjZXNpemUsIGxpbWl0Ynk7CisJc3RydWN0IG5mc3J2X2Rl c2NyaXB0IG5mc2QsICpuZCA9ICZuZnNkOworCXN0cnVjdCBuZnNjbG9wZW4gKm9wOworCXN0cnVj dCBuZnNjbGRlbGVnICpkcCA9IE5VTEw7CisJc3RydWN0IG5mc25vZGUgKm5wOworCXN0cnVjdCBu ZnNmaCAqbmZocDsKKwlzdHJ1Y3QgbmZzY2xzZXNzaW9uICp0c2VwOworCW5mc2F0dHJiaXRfdCBh dHRyYml0czsKKwluZnN2NHN0YXRlaWRfdCBzdGF0ZWlkOworCXVpbnQzMl90IHJmbGFnczsKKwlz dHJ1Y3QgbmZzbW91bnQgKm5tcDsKKworCW5tcCA9IFZGU1RPTkZTKGR2cC0+dl9tb3VudCk7CisJ bnAgPSBWVE9ORlMoZHZwKTsKKwkqbGF5c3RhdHAgPSBFTlhJTzsKKwkqdW5sb2NrZWRwID0gMDsK KwkqbmZocHAgPSBOVUxMOworCSpkcHAgPSBOVUxMOworCSphdHRyZmxhZ3AgPSAwOworCSpkYXR0 cmZsYWdwID0gMDsKKwlpZiAobmFtZWxlbiA+IE5GU19NQVhOQU1MRU4pCisJCXJldHVybiAoRU5B TUVUT09MT05HKTsKKwlORlNDTF9SRVFTVEFSVChuZCwgTkZTUFJPQ19DUkVBVEVMQVlHRVQsIGR2 cCk7CisJLyoKKwkgKiBGb3IgVjQsIHRoaXMgaXMgYWN0dWFsbHkgYW4gT3BlbiBvcC4KKwkgKi8K KwlORlNNX0JVSUxEKHRsLCB1X2ludDMyX3QgKiwgNSAqIE5GU1hfVU5TSUdORUQpOworCSp0bCsr ID0gdHhkcl91bnNpZ25lZChvd3AtPm5mc293X3NlcWlkKTsKKwkqdGwrKyA9IHR4ZHJfdW5zaWdu ZWQoTkZTVjRPUEVOX0FDQ0VTU1dSSVRFIHwKKwkgICAgTkZTVjRPUEVOX0FDQ0VTU1JFQUQpOwor CSp0bCsrID0gdHhkcl91bnNpZ25lZChORlNWNE9QRU5fREVOWU5PTkUpOworCXRzZXAgPSBuZnNt bnRfbWRzc2Vzc2lvbihubXApOworCSp0bCsrID0gdHNlcC0+bmZzZXNzX2NsaWVudGlkLmx2YWxb MF07CisJKnRsID0gdHNlcC0+bmZzZXNzX2NsaWVudGlkLmx2YWxbMV07CisJbmZzbV9zdHJ0b20o bmQsIG93cC0+bmZzb3dfb3duZXIsIE5GU1Y0Q0xfTE9DS05BTUVMRU4pOworCU5GU01fQlVJTEQo dGwsIHVfaW50MzJfdCAqLCAyICogTkZTWF9VTlNJR05FRCk7CisJKnRsKysgPSB0eGRyX3Vuc2ln bmVkKE5GU1Y0T1BFTl9DUkVBVEUpOworCWlmICgoZm1vZGUgJiBPX0VYQ0wpICE9IDApIHsKKwkJ aWYgKE5GU0hBU1NFU1NQRVJTSVNUKG5tcCkpIHsKKwkJCS8qIFVzZSBHVUFSREVEIGZvciBwZXJz aXN0ZW50IHNlc3Npb25zLiAqLworCQkJKnRsID0gdHhkcl91bnNpZ25lZChORlNDUkVBVEVfR1VB UkRFRCk7CisJCQluZnNjbF9maWxsc2F0dHIobmQsIHZhcCwgZHZwLCAwLCAwKTsKKwkJfSBlbHNl IHsKKwkJCS8qIE90aGVyd2lzZSwgdXNlIEVYQ0xVU0lWRTRfMS4gKi8KKwkJCSp0bCA9IHR4ZHJf dW5zaWduZWQoTkZTQ1JFQVRFX0VYQ0xVU0lWRTQxKTsKKwkJCU5GU01fQlVJTEQodGwsIHVfaW50 MzJfdCAqLCBORlNYX1ZFUkYpOworCQkJKnRsKysgPSBjdmVyZi5sdmFsWzBdOworCQkJKnRsID0g Y3ZlcmYubHZhbFsxXTsKKwkJCW5mc2NsX2ZpbGxzYXR0cihuZCwgdmFwLCBkdnAsIDAsIDApOwor CQl9CisJfSBlbHNlIHsKKwkJKnRsID0gdHhkcl91bnNpZ25lZChORlNDUkVBVEVfVU5DSEVDS0VE KTsKKwkJbmZzY2xfZmlsbHNhdHRyKG5kLCB2YXAsIGR2cCwgMCwgMCk7CisJfQorCU5GU01fQlVJ TEQodGwsIHVfaW50MzJfdCAqLCBORlNYX1VOU0lHTkVEKTsKKwkqdGwgPSB0eGRyX3Vuc2lnbmVk KE5GU1Y0T1BFTl9DTEFJTU5VTEwpOworCW5mc21fc3RydG9tKG5kLCBuYW1lLCBuYW1lbGVuKTsK KwkvKiBHZXQgdGhlIG5ldyBmaWxlJ3MgaGFuZGxlIGFuZCBhdHRyaWJ1dGVzLCBwbHVzIHNhdmUg dGhlIEZILiAqLworCU5GU01fQlVJTEQodGwsIHVfaW50MzJfdCAqLCAzICogTkZTWF9VTlNJR05F RCk7CisJKnRsKysgPSB0eGRyX3Vuc2lnbmVkKE5GU1Y0T1BfU0FWRUZIKTsKKwkqdGwrKyA9IHR4 ZHJfdW5zaWduZWQoTkZTVjRPUF9HRVRGSCk7CisJKnRsID0gdHhkcl91bnNpZ25lZChORlNWNE9Q X0dFVEFUVFIpOworCU5GU0dFVEFUVFJfQVRUUkJJVCgmYXR0cmJpdHMpOworCW5mc3J2X3B1dGF0 dHJiaXQobmQsICZhdHRyYml0cyk7CisJLyogR2V0IHRoZSBkaXJlY3RvcnkncyBwb3N0LW9wIGF0 dHJpYnV0ZXMuICovCisJTkZTTV9CVUlMRCh0bCwgdV9pbnQzMl90ICosIE5GU1hfVU5TSUdORUQp OworCSp0bCA9IHR4ZHJfdW5zaWduZWQoTkZTVjRPUF9QVVRGSCk7CisJbmZzbV9maHRvbShuZCwg bnAtPm5fZmhwLT5uZmhfZmgsIG5wLT5uX2ZocC0+bmZoX2xlbiwgMCk7CisJTkZTTV9CVUlMRCh0 bCwgdV9pbnQzMl90ICosIE5GU1hfVU5TSUdORUQpOworCSp0bCA9IHR4ZHJfdW5zaWduZWQoTkZT VjRPUF9HRVRBVFRSKTsKKwluZnNydl9wdXRhdHRyYml0KG5kLCAmYXR0cmJpdHMpOworCU5GU01f QlVJTEQodGwsIHVfaW50MzJfdCAqLCAyICogTkZTWF9VTlNJR05FRCk7CisJKnRsKysgPSB0eGRy X3Vuc2lnbmVkKE5GU1Y0T1BfUkVTVE9SRUZIKTsKKwkqdGwgPSB0eGRyX3Vuc2lnbmVkKE5GU1Y0 T1BfTEFZT1VUR0VUKTsKKwluZnNydl9zZXR1cGxheW91dGdldChuZCwgTkZTTEFZT1VUSU9NT0RF X1JXLCAwLCBVSU5UNjRfTUFYLCAwLCBzdGF0ZWlkcCwKKwkgICAgbGF5b3V0bGVuLCB1c2VjdXJz dGF0ZWlkKTsKKwllcnJvciA9IG5mc2NsX3JlcXVlc3QobmQsIGR2cCwgcCwgY3JlZCwgZHN0dWZm KTsKKwlpZiAoZXJyb3IgIT0gMCkKKwkJcmV0dXJuIChlcnJvcik7CisJTkZTQ0xfREVCVUcoNCwg Im5mc3JwY19jcmVhdGVsYXlvdXQgc3RhdD0lZCBlcnI9JWRcbiIsIG5kLT5uZF9yZXBzdGF0LAor CSAgICBlcnJvcik7CisJaWYgKG5kLT5uZF9yZXBzdGF0ICE9IDApCisJCSpsYXlzdGF0cCA9IG5k LT5uZF9yZXBzdGF0OworCU5GU0NMX0lOQ1JTRVFJRChvd3AtPm5mc293X3NlcWlkLCBuZCk7CisJ aWYgKChuZC0+bmRfZmxhZyAmIE5EX05PTU9SRURBVEEpID09IDApIHsKKwkJTkZTQ0xfREVCVUco NCwgIm5mc3JwY19jcmVhdGVsYXlvdXQgb3BlbiBzdWNjZWVkZWRcbiIpOworCQlORlNNX0RJU1NF Q1QodGwsIHVfaW50MzJfdCAqLCBORlNYX1NUQVRFSUQgKworCQkgICAgNiAqIE5GU1hfVU5TSUdO RUQpOworCQlzdGF0ZWlkLnNlcWlkID0gKnRsKys7CisJCXN0YXRlaWQub3RoZXJbMF0gPSAqdGwr KzsKKwkJc3RhdGVpZC5vdGhlclsxXSA9ICp0bCsrOworCQlzdGF0ZWlkLm90aGVyWzJdID0gKnRs OworCQlyZmxhZ3MgPSBmeGRyX3Vuc2lnbmVkKHVfaW50MzJfdCwgKih0bCArIDYpKTsKKwkJbmZz cnZfZ2V0YXR0cmJpdHMobmQsICZhdHRyYml0cywgTlVMTCwgTlVMTCk7CisJCU5GU01fRElTU0VD VCh0bCwgdV9pbnQzMl90ICosIE5GU1hfVU5TSUdORUQpOworCQlkZWxlZyA9IGZ4ZHJfdW5zaWdu ZWQoaW50LCAqdGwpOworCQlpZiAoZGVsZWcgPT0gTkZTVjRPUEVOX0RFTEVHQVRFUkVBRCB8fAor CQkgICAgZGVsZWcgPT0gTkZTVjRPUEVOX0RFTEVHQVRFV1JJVEUpIHsKKwkJCWlmICghKG93cC0+ bmZzb3dfY2xwLT5uZnNjX2ZsYWdzICYKKwkJCSAgICAgIE5GU0NMRkxBR1NfRklSU1RERUxFRykp CisJCQkJb3dwLT5uZnNvd19jbHAtPm5mc2NfZmxhZ3MgfD0KKwkJCQkgIChORlNDTEZMQUdTX0ZJ UlNUREVMRUcgfCBORlNDTEZMQUdTX0dPVERFTEVHKTsKKwkJCWRwID0gbWFsbG9jKHNpemVvZihz dHJ1Y3QgbmZzY2xkZWxlZykgKyBORlNYX1Y0RkhNQVgsCisJCQkgICAgTV9ORlNDTERFTEVHLCBN X1dBSVRPSyk7CisJCQlMSVNUX0lOSVQoJmRwLT5uZnNkbF9vd25lcik7CisJCQlMSVNUX0lOSVQo JmRwLT5uZnNkbF9sb2NrKTsKKwkJCWRwLT5uZnNkbF9jbHAgPSBvd3AtPm5mc293X2NscDsKKwkJ CW5ld25mc19jb3B5aW5jcmVkKGNyZWQsICZkcC0+bmZzZGxfY3JlZCk7CisJCQluZnNjbF9sb2Nr aW5pdCgmZHAtPm5mc2RsX3J3bG9jayk7CisJCQlORlNNX0RJU1NFQ1QodGwsIHVfaW50MzJfdCAq LCBORlNYX1NUQVRFSUQgKworCQkJICAgIE5GU1hfVU5TSUdORUQpOworCQkJZHAtPm5mc2RsX3N0 YXRlaWQuc2VxaWQgPSAqdGwrKzsKKwkJCWRwLT5uZnNkbF9zdGF0ZWlkLm90aGVyWzBdID0gKnRs Kys7CisJCQlkcC0+bmZzZGxfc3RhdGVpZC5vdGhlclsxXSA9ICp0bCsrOworCQkJZHAtPm5mc2Rs X3N0YXRlaWQub3RoZXJbMl0gPSAqdGwrKzsKKwkJCXJldCA9IGZ4ZHJfdW5zaWduZWQoaW50LCAq dGwpOworCQkJaWYgKGRlbGVnID09IE5GU1Y0T1BFTl9ERUxFR0FURVdSSVRFKSB7CisJCQkJZHAt Pm5mc2RsX2ZsYWdzID0gTkZTQ0xETF9XUklURTsKKwkJCQkvKgorCQkJCSAqIEluZGljYXRlcyBo b3cgbXVjaCB0aGUgZmlsZSBjYW4gZ3Jvdy4KKwkJCQkgKi8KKwkJCQlORlNNX0RJU1NFQ1QodGws IHVfaW50MzJfdCAqLAorCQkJCSAgICAzICogTkZTWF9VTlNJR05FRCk7CisJCQkJbGltaXRieSA9 IGZ4ZHJfdW5zaWduZWQoaW50LCAqdGwrKyk7CisJCQkJc3dpdGNoIChsaW1pdGJ5KSB7CisJCQkJ Y2FzZSBORlNWNE9QRU5fTElNSVRTSVpFOgorCQkJCQlkcC0+bmZzZGxfc2l6ZWxpbWl0ID0gZnhk cl9oeXBlcih0bCk7CisJCQkJCWJyZWFrOworCQkJCWNhc2UgTkZTVjRPUEVOX0xJTUlUQkxPQ0tT OgorCQkJCQlkcC0+bmZzZGxfc2l6ZWxpbWl0ID0KKwkJCQkJICAgIGZ4ZHJfdW5zaWduZWQodV9p bnQ2NF90LCAqdGwrKyk7CisJCQkJCWRwLT5uZnNkbF9zaXplbGltaXQgKj0KKwkJCQkJICAgIGZ4 ZHJfdW5zaWduZWQodV9pbnQ2NF90LCAqdGwpOworCQkJCQlicmVhazsKKwkJCQlkZWZhdWx0Ogor CQkJCQllcnJvciA9IE5GU0VSUl9CQURYRFI7CisJCQkJCWdvdG8gbmZzbW91dDsKKwkJCQl9Owor CQkJfSBlbHNlIHsKKwkJCQlkcC0+bmZzZGxfZmxhZ3MgPSBORlNDTERMX1JFQUQ7CisJCQl9CisJ CQlpZiAocmV0ICE9IDApCisJCQkJZHAtPm5mc2RsX2ZsYWdzIHw9IE5GU0NMRExfUkVDQUxMOwor CQkJZXJyb3IgPSBuZnNydl9kaXNzZWN0YWNlKG5kLCAmZHAtPm5mc2RsX2FjZSwgJnJldCwKKwkJ CSAgICAmYWNlc2l6ZSwgcCk7CisJCQlpZiAoZXJyb3IgIT0gMCkKKwkJCQlnb3RvIG5mc21vdXQ7 CisJCX0gZWxzZSBpZiAoZGVsZWcgIT0gTkZTVjRPUEVOX0RFTEVHQVRFTk9ORSkgeworCQkJZXJy b3IgPSBORlNFUlJfQkFEWERSOworCQkJZ290byBuZnNtb3V0OworCQl9CisKKwkJLyogTm93LCB3 ZSBzaG91bGQgaGF2ZSB0aGUgc3RhdHVzIGZvciB0aGUgU2F2ZUZILiAqLworCQlORlNNX0RJU1NF Q1QodGwsIHVpbnQzMl90ICosIDIgKiBORlNYX1VOU0lHTkVEKTsKKwkJaWYgKCorK3RsID09IDAp IHsKKwkJCU5GU0NMX0RFQlVHKDQsICJuZnNycGNfY3JlYXRlbGF5b3V0IFNhdmVGSCBva1xuIik7 CisJCQkvKgorCQkJICogTm93LCBwcm9jZXNzIHRoZSBHZXRGSCBhbmQgR2V0YXR0ciBmb3IgdGhl IG5ld2x5CisJCQkgKiBjcmVhdGVkIGZpbGUuIG5mc2NsX210b2ZoKCkgd2lsbCBzZXQKKwkJCSAq IE5EX05PTU9SRURBVEEgaWYgdGhlc2Ugd2VyZW4ndCBzdWNjZXNzZnVsLgorCQkJICovCisJCQll cnJvciA9IG5mc2NsX210b2ZoKG5kLCBuZmhwcCwgbm5hcCwgYXR0cmZsYWdwKTsKKwkJCU5GU0NM X0RFQlVHKDQsICJhZnQgbmZzY2xfbXRvZmggZXJyPSVkXG4iLCBlcnJvcik7CisJCQlpZiAoZXJy b3IgIT0gMCkKKwkJCQlnb3RvIG5mc21vdXQ7CisJCX0gZWxzZQorCQkJbmQtPm5kX2ZsYWcgfD0g TkRfTk9NT1JFREFUQTsKKwkJLyogTm93IHdlIGhhdmUgdGhlIFB1dEZIIGFuZCBHZXRhdHRyIGZv ciB0aGUgZGlyZWN0b3J5LiAqLworCQlpZiAoKG5kLT5uZF9mbGFnICYgTkRfTk9NT1JFREFUQSkg PT0gMCkgeworCQkJTkZTTV9ESVNTRUNUKHRsLCB1aW50MzJfdCAqLCAyICogTkZTWF9VTlNJR05F RCk7CisJCQlpZiAoKisrdGwgIT0gMCkKKwkJCQluZC0+bmRfZmxhZyB8PSBORF9OT01PUkVEQVRB OworCQkJZWxzZSB7CisJCQkJTkZTTV9ESVNTRUNUKHRsLCB1aW50MzJfdCAqLCAyICoKKwkJCQkg ICAgTkZTWF9VTlNJR05FRCk7CisJCQkJaWYgKCorK3RsICE9IDApCisJCQkJCW5kLT5uZF9mbGFn IHw9IE5EX05PTU9SRURBVEE7CisJCQl9CisJCX0KKwkJaWYgKChuZC0+bmRfZmxhZyAmIE5EX05P TU9SRURBVEEpID09IDApIHsKKwkJCS8qIExvYWQgdGhlIGRpcmVjdG9yeSBhdHRyaWJ1dGVzLiAq LworCQkJZXJyb3IgPSBuZnNtX2xvYWRhdHRyKG5kLCBkbmFwKTsKKwkJCU5GU0NMX0RFQlVHKDQs ICJhZnQgbmZzbV9sb2FkYXR0ciBlcnI9JWRcbiIsIGVycm9yKTsKKwkJCWlmIChlcnJvciAhPSAw KQorCQkJCWdvdG8gbmZzbW91dDsKKwkJCSpkYXR0cmZsYWdwID0gMTsKKwkJCWlmIChkcCAhPSBO VUxMICYmICphdHRyZmxhZ3AgIT0gMCkgeworCQkJCWRwLT5uZnNkbF9jaGFuZ2UgPSBubmFwLT5u YV9maWxlcmV2OworCQkJCWRwLT5uZnNkbF9tb2R0aW1lID0gbm5hcC0+bmFfbXRpbWU7CisJCQkJ ZHAtPm5mc2RsX2ZsYWdzIHw9IE5GU0NMRExfTU9EVElNRVNFVDsKKwkJCX0KKwkJCS8qCisJCQkg KiBXZSBjYW4gbm93IGNvbXBsZXRlIHRoZSBPcGVuIHN0YXRlLgorCQkJICovCisJCQluZmhwID0g Km5maHBwOworCQkJaWYgKGRwICE9IE5VTEwpIHsKKwkJCQlkcC0+bmZzZGxfZmhsZW4gPSBuZmhw LT5uZmhfbGVuOworCQkJCU5GU0JDT1BZKG5maHAtPm5maF9maCwgZHAtPm5mc2RsX2ZoLAorCQkJ CSAgICBuZmhwLT5uZmhfbGVuKTsKKwkJCX0KKwkJCS8qCisJCQkgKiBHZXQgYW4gT3BlbiBzdHJ1 Y3R1cmUgdGhhdCB3aWxsIGJlCisJCQkgKiBhdHRhY2hlZCB0byB0aGUgT3Blbk93bmVyLCBhY3F1 aXJlZCBhbHJlYWR5LgorCQkJICovCisJCQllcnJvciA9IG5mc2NsX29wZW4oZHZwLCBuZmhwLT5u ZmhfZmgsIG5maHAtPm5maF9sZW4sIAorCQkJICAgIChORlNWNE9QRU5fQUNDRVNTV1JJVEUgfCBO RlNWNE9QRU5fQUNDRVNTUkVBRCksIDAsCisJCQkgICAgY3JlZCwgcCwgTlVMTCwgJm9wLCAmbmV3 b25lLCBOVUxMLCAwKTsKKwkJCWlmIChlcnJvciAhPSAwKQorCQkJCWdvdG8gbmZzbW91dDsKKwkJ CW9wLT5uZnNvX3N0YXRlaWQgPSBzdGF0ZWlkOworCQkJbmV3bmZzX2NvcHlpbmNyZWQoY3JlZCwg Jm9wLT5uZnNvX2NyZWQpOworCQorCQkJbmZzY2xfb3BlbnJlbGVhc2Uobm1wLCBvcCwgZXJyb3Is IG5ld29uZSk7CisJCQkqdW5sb2NrZWRwID0gMTsKKworCQkJLyogTm93LCBoYW5kbGUgdGhlIFJl c3RvcmVGSCBhbmQgTGF5b3V0R2V0LiAqLworCQkJaWYgKG5kLT5uZF9yZXBzdGF0ID09IDApIHsK KwkJCQlORlNNX0RJU1NFQ1QodGwsIHVpbnQzMl90ICosIDQgKiBORlNYX1VOU0lHTkVEKTsKKwkJ CQkqbGF5c3RhdHAgPSBmeGRyX3Vuc2lnbmVkKGludCwgKih0bCArIDMpKTsKKwkJCQlpZiAoKmxh eXN0YXRwID09IDApIHsKKwkJCQkJZXJyb3IgPSBuZnNydl9wYXJzZWxheW91dGdldChuZCwKKwkJ CQkJICAgIHN0YXRlaWRwLCByZXRvbmNsb3NlcCwgZmxocCk7CisJCQkJCWlmIChlcnJvciAhPSAw KQorCQkJCQkJKmxheXN0YXRwID0gZXJyb3I7CisJCQkJfQorCQkJCU5GU0NMX0RFQlVHKDQsICJh ZnQgbmZzcnZfcGFyc2VsYXlvdXQgZXJyPSVkXG4iLAorCQkJCSAgICBlcnJvcik7CisJCQl9IGVs c2UKKwkJCQluZC0+bmRfcmVwc3RhdCA9IDA7CisJCX0KKwl9CisJaWYgKG5kLT5uZF9yZXBzdGF0 ICE9IDAgJiYgZXJyb3IgPT0gMCkKKwkJZXJyb3IgPSBuZC0+bmRfcmVwc3RhdDsKKwlpZiAoZXJy b3IgPT0gTkZTRVJSX1NUQUxFQ0xJRU5USUQgfHwgZXJyb3IgPT0gTkZTRVJSX0JBRFNFU1NJT04p CisJCW5mc2NsX2luaXRpYXRlX3JlY292ZXJ5KG93cC0+bmZzb3dfY2xwKTsKK25mc21vdXQ6CisJ TkZTQ0xfREVCVUcoNCwgImVvIG5mc3JwY19jcmVhdGVsYXlvdXQgZXJyPSVkXG4iLCBlcnJvcik7 CisJaWYgKGVycm9yID09IDApCisJCSpkcHAgPSBkcDsKKwllbHNlCisJCWZyZWUoZHAsIE1fTkZT Q0xERUxFRyk7CisJbWJ1Zl9mcmVlbShuZC0+bmRfbXJlcCk7CisJcmV0dXJuIChlcnJvcik7Cit9 CisKKy8qCisgKiBTaW1pbGFyIHRvIG5mc3JwY19nZXRvcGVubGF5b3V0KCksIGV4Y2VwdCB0aGF0 IGl0IHVzZWQgZm9yIHRoZSBDcmVhdGUgY2FzZS4KKyAqLworc3RhdGljIGludAorbmZzcnBjX2dl dGNyZWF0ZWxheW91dCh2bm9kZV90IGR2cCwgY2hhciAqbmFtZSwgaW50IG5hbWVsZW4sIHN0cnVj dCB2YXR0ciAqdmFwLAorICAgIG5mc3F1YWRfdCBjdmVyZiwgaW50IGZtb2RlLCBzdHJ1Y3QgbmZz Y2xvd25lciAqb3dwLCBzdHJ1Y3QgbmZzY2xkZWxlZyAqKmRwcCwKKyAgICBzdHJ1Y3QgdWNyZWQg KmNyZWQsIE5GU1BST0NfVCAqcCwgc3RydWN0IG5mc3ZhdHRyICpkbmFwLAorICAgIHN0cnVjdCBu ZnN2YXR0ciAqbm5hcCwgc3RydWN0IG5mc2ZoICoqbmZocHAsIGludCAqYXR0cmZsYWdwLAorICAg IGludCAqZGF0dHJmbGFncCwgdm9pZCAqZHN0dWZmLCBpbnQgKnVubG9ja2VkcCkKK3sKKwlzdHJ1 Y3QgbmZzY2xsYXlvdXQgKmx5cDsKKwlzdHJ1Y3QgbmZzY2xmbGF5b3V0aGVhZCBmbGg7CisJc3Ry dWN0IG5mc2ZoICpuZmhwOworCXN0cnVjdCBuZnNjbHNlc3Npb24gKnRzZXA7CisJc3RydWN0IG5m c21vdW50ICpubXA7CisJbmZzdjRzdGF0ZWlkX3Qgc3RhdGVpZDsKKwlpbnQgZXJyb3IsIGxheW91 dGxlbiwgcmV0b25jbG9zZSwgbGF5c3RhdDsKKworCWVycm9yID0gMDsKKwlubXAgPSBWRlNUT05G UyhkdnAtPnZfbW91bnQpOworCUxJU1RfSU5JVCgmZmxoKTsKKwl0c2VwID0gbmZzbW50X21kc3Nl c3Npb24obm1wKTsKKwlsYXlvdXRsZW4gPSB0c2VwLT5uZnNlc3NfbWF4Y2FjaGUgLSAoTkZTWF9T VEFURUlEICsgMyAqIE5GU1hfVU5TSUdORUQpOworCWVycm9yID0gbmZzcnBjX2NyZWF0ZWxheW91 dChkdnAsIG5hbWUsIG5hbWVsZW4sIHZhcCwgY3ZlcmYsIGZtb2RlLAorCSAgICBvd3AsIGRwcCwg Y3JlZCwgcCwgZG5hcCwgbm5hcCwgbmZocHAsIGF0dHJmbGFncCwgZGF0dHJmbGFncCwKKwkgICAg ZHN0dWZmLCB1bmxvY2tlZHAsICZzdGF0ZWlkLCAxLCBsYXlvdXRsZW4sICZyZXRvbmNsb3NlLCAm ZmxoLAorCSAgICAmbGF5c3RhdCk7CisJTkZTQ0xfREVCVUcoNCwgImFmdCBuZnNycGNfY3JlYXRl bGF5b3V0cnBjIGxheXN0YXQ9JWQgZXJyPSVkXG4iLAorCSAgICBsYXlzdGF0LCBlcnJvcik7CisJ bHlwID0gTlVMTDsKKwluZmhwID0gKm5maHBwOworCWxheXN0YXQgPSBuZnNycGNfbGF5b3V0Z2V0 cmVzKG5tcCwgZHZwLCBuZmhwLT5uZmhfZmgsIG5maHAtPm5maF9sZW4sCisJICAgICZzdGF0ZWlk LCByZXRvbmNsb3NlLCBOVUxMLCAmbHlwLCAmZmxoLCBsYXlzdGF0LCBOVUxMLCBjcmVkLCBwKTsK KwlpZiAobGF5c3RhdCA9PSAwKQorCQluZnNjbF9yZWxsYXlvdXQobHlwLCAwKTsKKwlyZXR1cm4g KGVycm9yKTsKK30KKworLyoKKyAqIFByb2Nlc3MgdGhlIHJlc3VsdHMgb2YgYSBsYXlvdXRnZXQo KSBvcGVyYXRpb24uCisgKi8KK3N0YXRpYyBpbnQKK25mc3JwY19sYXlvdXRnZXRyZXMoc3RydWN0 IG5mc21vdW50ICpubXAsIHZub2RlX3QgdnAsIHVpbnQ4X3QgKm5ld2ZocCwKKyAgICBpbnQgbmV3 ZmhsZW4sIG5mc3Y0c3RhdGVpZF90ICpzdGF0ZWlkcCwgaW50IHJldG9uY2xvc2UsIHVpbnQzMl90 ICpub3RpZnliaXQsCisgICAgc3RydWN0IG5mc2NsbGF5b3V0ICoqbHlwcCwgc3RydWN0IG5mc2Ns ZmxheW91dGhlYWQgKmZsaHAsCisgICAgaW50IGxheXN0YXQsIGludCAqaXNsb2NrZWRwLCBzdHJ1 Y3QgdWNyZWQgKmNyZWQsIE5GU1BST0NfVCAqcCkKK3sKKwlzdHJ1Y3QgbmZzY2xmbGF5b3V0ICp0 ZmxwOworCXN0cnVjdCBuZnNjbGRldmluZm8gKmRpcDsKKworCWlmIChsYXlzdGF0ID09IE5GU0VS Ul9VTktOTEFZT1VUVFlQRSkgeworCQkvKiBEaXNhYmxlIFBORlMuICovCisJCU5GU0NMX0RFQlVH KDEsICJkaXNhYmxlIFBORlNcbiIpOworCQlORlNMT0NLTU5UKG5tcCk7CisJCW5tcC0+bm1fc3Rh dGUgJj0gfk5GU1NUQV9QTkZTOworCQlORlNVTkxPQ0tNTlQobm1wKTsKKwl9CisJaWYgKGxheXN0 YXQgPT0gMCkgeworCQlORlNDTF9ERUJVRyg0LCAibmZzcnBjX2xheW91dGdldHJlcyBhdCBGT1JF QUNIXG4iKTsKKwkJTElTVF9GT1JFQUNIKHRmbHAsIGZsaHAsIG5mc2ZsX2xpc3QpIHsKKwkJCWxh eXN0YXQgPSBuZnNjbF9hZGRkZXZpbmZvKG5tcCwgTlVMTCwgdGZscCk7CisJCQlORlNDTF9ERUJV Ryg0LCAiYWZ0IGFkZGRldj0lZFxuIiwgbGF5c3RhdCk7CisJCQlpZiAobGF5c3RhdCAhPSAwKSB7 CisJCQkJbGF5c3RhdCA9IG5mc3JwY19nZXRkZXZpY2VpbmZvKG5tcCwKKwkJCQkgICAgdGZscC0+ bmZzZmxfZGV2LCBORlNMQVlPVVRfTkZTVjRfMV9GSUxFUywKKwkJCQkgICAgbm90aWZ5Yml0LCAm ZGlwLCBjcmVkLCBwKTsKKwkJCQlORlNDTF9ERUJVRyg0LCAiYWZ0IG5mc3JwY19nZGk9JWRcbiIs CisJCQkJICAgIGxheXN0YXQpOworCQkJCWlmIChsYXlzdGF0ICE9IDApCisJCQkJCWJyZWFrOwor CQkJCWxheXN0YXQgPSBuZnNjbF9hZGRkZXZpbmZvKG5tcCwgZGlwLCB0ZmxwKTsKKwkJCQlpZiAo bGF5c3RhdCAhPSAwKQorCQkJCQlwcmludGYoImdldGxheW91dDogY2Fubm90IGFkZFxuIik7CisJ CQl9CisJCX0KKwl9CisJaWYgKGxheXN0YXQgPT0gMCkgeworCQkvKgorCQkgKiBuZnNjbF9sYXlv dXQoKSBhbHdheXMgcmV0dXJucyB3aXRoIHRoZSBuZnNseV9sb2NrCisJCSAqIHNldCB0byBhIHJl ZmNudCAoc2hhcmVkIGxvY2spLgorCQkgKiBQYXNzaW5nIGluIGR2cCBpcyBzdWZmaWNpZW50LCBz aW5jZSBpdCBpcyBvbmx5IHVzZWQgdG8KKwkJICogZ2V0IHRoZSBmc2lkIGZvciB0aGUgZmlsZSBz eXN0ZW0uCisJCSAqLworCQlsYXlzdGF0ID0gbmZzY2xfbGF5b3V0KG5tcCwgdnAsIG5ld2ZocCwg bmV3ZmhsZW4sIHN0YXRlaWRwLAorCQkgICAgcmV0b25jbG9zZSwgZmxocCwgbHlwcCwgY3JlZCwg cCk7CisJCU5GU0NMX0RFQlVHKDQsICJuZnNycGNfbGF5b3V0Z2V0cmVzOiBhZnQgbmZzY2xfbGF5 b3V0PSVkXG4iLAorCQkgICAgbGF5c3RhdCk7CisJCWlmIChsYXlzdGF0ID09IDAgJiYgaXNsb2Nr ZWRwICE9IE5VTEwpCisJCQkqaXNsb2NrZWRwID0gMTsKKwl9CisJcmV0dXJuIChsYXlzdGF0KTsK K30KKwotLS0gdXNyLmJpbi9uZnNzdGF0L25mc3N0YXQuYy5vcmlnCTIwMTctMDUtMTkgMTQ6MDI6 MjIuMzM2MzU4MDAwIC0wNDAwCisrKyB1c3IuYmluL25mc3N0YXQvbmZzc3RhdC5jCTIwMTctMDUt MTkgMTQ6MDI6MjIuMzM3MDQ1MDAwIC0wNDAwCkBAIC05MjYsNiArOTI2LDEzIEBAIGV4cDQxX2lu dHByKGludCBjbGllbnRPbmx5LCBpbnQgc2VydmVyT24KIAkJICAgICh1aW50bWF4X3QpZXh0X25m c3N0YXRzLnJwY2NudFtORlNQUk9DX0NPTU1JVERTXSk7CiAJCWlmIChwcmludHRpdGxlKQogCQkJ cHJpbnRmKAorCQkJICAgICIlMTIuMTJzICUxMi4xMnNcbiIsCisJCQkgICAgIk9wZW5MYXlvdXQi LCAiQ3JlYXRlTGF5b3V0Iik7CisJCXByaW50ZigiJTEyanUgJTEyanVcbiIsCisJCSAgICAodWlu dG1heF90KWV4dF9uZnNzdGF0cy5ycGNjbnRbTkZTUFJPQ19PUEVOTEFZR0VUXSwKKwkJICAgICh1 aW50bWF4X3QpZXh0X25mc3N0YXRzLnJwY2NudFtORlNQUk9DX0NSRUFURUxBWUdFVF0pOworCQlp ZiAocHJpbnR0aXRsZSkKKwkJCXByaW50ZigKIAkJCSAgICAiJTEyLjEycyAlMTIuMTJzICUxMi4x MnMgJTEyLjEycyAlMTIuMTJzICUxMi4xMnNcbiIsCiAJCQkgICAgIk9wZW5Pd25lciIsICJPcGVu cyIsICJMb2NrT3duZXIiLCAiTG9ja3MiLAogCQkJICAgICJEZWxlZ3MiLCAiTG9jYWxPd24iKTsK --_002_YTXPR01MB018901BE4BA7E9780F4B430BDDDB0YTXPR01MB0189CANP_-- From owner-freebsd-fs@freebsd.org Fri Jun 23 00:14:10 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB595D98094 for ; Fri, 23 Jun 2017 00:14:10 +0000 (UTC) (envelope-from theunusualmatt@gmail.com) Received: from mail-yw0-x236.google.com (mail-yw0-x236.google.com [IPv6:2607:f8b0:4002:c05::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A812170863 for ; Fri, 23 Jun 2017 00:14:10 +0000 (UTC) (envelope-from theunusualmatt@gmail.com) Received: by mail-yw0-x236.google.com with SMTP id 63so11894783ywr.0 for ; Thu, 22 Jun 2017 17:14:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=xmviNx1s4N9DgiBCuvFIW2OHbNNpBSmCPi6D7HoY6X8=; b=EXqnL5EsCqgeiplt9DMBw25XBIBfNFwDol/dy67o60Rcy2Vr2h6t/DHZDN9akKqUZl vyDzxv2krMoDKfl4Z4OIvD2hxAQobmAqnMk+oW8kHYLNEJ020B+OdiJMUgtjJER1PiAQ RqTf0tS4GNntE/OnQVTykaoQCON915gFWFwwnafdex1AisZo3VRiJnWwghSrEJ/t2dUR wXrIicAVZzIrqvVIhD5jGt+S7msfhSQsWhZfo35P4QTi7h/sm6A2kjEgYdBhMR/sfJvQ hNIL07I/q2PrbkofUQR7eWPf8PGW0YRxe3y39q1xRflfNP0o3OKuGkGa6/+ykYBj0G7r +6bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=xmviNx1s4N9DgiBCuvFIW2OHbNNpBSmCPi6D7HoY6X8=; b=Xb2O9Jf1iz/ydR+ZBqqqBqMEl3Fb37H/uodx1IGYXydxDQut3w/N984iVP3UFChKCb v6DPAB36h2qiaHTr95usTdhryFbUP5+iqD5+bdGL8vbIUeIaTAEumR061x8quuob3Wu6 Dy8vAvqnaNqWPKqIxiKh7eiF5enX1fys/O5xOlXiV6HdMhLAkpBK98YEauEDfT1IbGCX +uQdhk+/StsYHDG2GeVkI10AjKYIJpFXacqwoc3zowOxg6bDGxQTv2S+0G5RoH6mJXnj emsH2wbGkclE3oIAGRWHrlBPIqvE7cXI1YsBQ5OxpRFI5bCC3cGACn9ZR4qY/wD5Dg45 UZ9w== X-Gm-Message-State: AKS2vOwhbPwuz7B8+W6n8Tjtk8PpaeZsKCeai6dIJutWt1uYjMfzAxb/ TKamFQ4tJvDL66ycyqbxSrMGihqxcw== X-Received: by 10.129.156.71 with SMTP id t68mr4331297ywg.257.1498176849811; Thu, 22 Jun 2017 17:14:09 -0700 (PDT) MIME-Version: 1.0 Received: by 10.129.103.70 with HTTP; Thu, 22 Jun 2017 17:14:09 -0700 (PDT) In-Reply-To: References: From: Matt B Date: Thu, 22 Jun 2017 20:14:09 -0400 Message-ID: Subject: Re: SMBv1 Deprecation To: Rick Macklem Cc: "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jun 2017 00:14:11 -0000 I totally understand. I try to support the FreeBSD Foundation with donations as often as I can as well as reporting bugs promptly as I am sure resources are spread thin. My skill set isn't really that of a programmer though. I am working right now at checking the Darwin/OS X code for mount_smbfs and other modules associated with smbfs in the hopes of possibly getting something viable for BSD, even if it has to be a port due to license issues. Progress is slow just due to lack of knowledge in the programming arena. On Thu, Jun 22, 2017 at 5:30 PM, Rick Macklem wrote: > Well, the short answer is...somebody has to do it. > (At this time, I believe that there are two people employed by > the FreeBSD Foundation to do FreeBSD kernel work.) > The rest of FreeBSD's development is done by volunteers > (some of which do the work for an employer and get permission > from the employer to upstream the work). > I, for example, do NFS as a hobby and always have, but to be honest, > there aren't many out there as stupid as I am and willing to do this;-) > > So, if you have the skills and time, feel free to do an implementation > and, so long it is appropriately licensed (no GPL or similar), I suspect > someone would be willing to work with you to get it into FreeBSD. > > If there is an SMBv2 implementation in one of the other BSDen > (NetBSD, OpenBSD,...) the port wouldn't be an immense amount > of work, but there are differences in the VFS and similar that will > need to be dealt with. > Otherwise, you are pretty much implementing it from scratch, using > the SMBv1 code as a starting point. > > rick > ________________________________________ > From: owner-freebsd-fs@freebsd.org on > behalf of Matt B > Sent: Thursday, June 22, 2017 3:36:14 PM > To: freebsd-fs@freebsd.org > Subject: SMBv1 Deprecation > > Long time user of FreeBSD here. I have been happily using the mount_smbfs > binary and in my fstab to mount Windows Shares on boot to be used by > various network services house on multiple FreeBSD systems. Sadly, it > appears these connections all use SMBv1 NT1 security to perform the mount > operation. With the new security landscape, post-WannaCry ransomware, in a > mixed-mode environment where all the shares live in Windows, that just > won't do. This has been discussed many times before in the past but there > hasn't been any headway AFAIK. Every other piece of software I have > encountered has moved away from this deprecated network protocol to the far > more secure versions of SMB to perform Windows share operations. As a stop > gap, I have implemented a very rudimentary NFS server advertising shares, > but configuring a Kerberos infrastructure and setting new accounts for each > and every service (not to mention the new permissions nightmares even with > Active Directory) on multiple BSD systems is arduous. Rather, I am > wondering why FreeBSD is behind the ball on the development? The other > Linux based systems I run required a simple addition of the vers=SMB2 flag > to the fstab entry to successfully mount. I understand the code base is > very old for the mount_smbfs, but what is the way forward here? NFS is > simply a workaround as far as I am concerned and every other *nix style > distro seems to play nice with SMB. Is there an ETR on this greatly needed > and long overdue update to mount newer style SMB shares? > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Fri Jun 23 00:43:07 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C978D9884E for ; Fri, 23 Jun 2017 00:43:07 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670046.outbound.protection.outlook.com [40.107.67.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 238D371667 for ; Fri, 23 Jun 2017 00:43:06 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0190.CANPRD01.PROD.OUTLOOK.COM (10.165.218.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Fri, 23 Jun 2017 00:43:03 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.1178.023; Fri, 23 Jun 2017 00:43:03 +0000 From: Rick Macklem To: Matt B CC: "freebsd-fs@freebsd.org" Subject: Re: SMBv1 Deprecation Thread-Topic: SMBv1 Deprecation Thread-Index: AQHS647UYCj1ThwsvUCkJ0Hkf0cfhqIxY+HFgAAwRYCAAAc7aA== Date: Fri, 23 Jun 2017 00:43:03 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTXPR01MB0190; 7:TkblkZJUMONTPhGOMg7AVy2iz3x0zNDcE/pWREtX/9cULL9rXl/VIgOE1bts7eMR8Qr1nw0ereO33jTs71cFhFNk5jETWM2rWRkeZmJNZAlzQjaMjtWg+wwlA3JVIk3fHgqV1d65EHxTZxgNker3K6xgQ6CGE32bM3cR/Uss9+p20wJofHK57vLsWP6UHZysocVcJMYuRl67sdao88ecKF8tjj08uAEzT4kwQmY2U5O7DZvKJsxKNJ1kq1FNmgigy3UNHCwbZZCB31f3ng2sava1y8e8DxFpic8k90CdOaS1UWu1AtlSTV4q2ckrOFUH4WbNnEHWGm0vAZEut/5MZWgcGTkolctOFyqyaXTO/sJFqFPiHH+189U7PjUCUcTvAJiET8ZJu1+ionJrMzfOkXXyl8MoxUeYEKlk0ozax5cQuI47Kj8K2dHVv2mJEyNkZ3TznypgL1IzaIe4Y0eCyZVQY1ksSKxBIAhLlZquuI++avF/gPgDr4h7APuE5W70gIgrqWxupUw2i/hFvSNT1cEGiVGGPCstFwGVQzANx5qBoc24qyHOTkleDvdYi1L8l0a0xcSwHKb8c1zFcVKWGSeLPscErmRAKQ9lcqlfCXSpOpZ4MmwqTc53jCiI7nxXfpA83dvYe9z1sF0ekeLMlxtAen4r+3EGnqC8Lx3S02nno/OCHvTd8q7WGw4FK7+wzq6qzkacLU9BTvyIzXaoE+jY7E5W7LxxVzMA+NAK0ymbhrqMpBdArocT7khGF9TpxMhqFGirHrQV2uboYASWlBloV4Z6DYsoajfFEESPVU0= x-ms-office365-filtering-correlation-id: d72f64bf-10c4-4269-e209-08d4b9d0ce87 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:YTXPR01MB0190; x-ms-traffictypediagnostic: YTXPR01MB0190: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(192374486261705)(75325880899374)(211171220733660); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(10201501046)(100000703101)(100105400095)(3002001)(93006095)(93001095)(6041248)(20161123564025)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123562025)(20161123555025)(20161123560025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:YTXPR01MB0190; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:YTXPR01MB0190; x-forefront-prvs: 0347410860 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(39410400002)(39850400002)(39400400002)(39450400003)(39840400002)(24454002)(377454003)(478600001)(6306002)(966005)(9686003)(4326008)(305945005)(2900100001)(53546010)(74482002)(81166006)(68736007)(2906002)(8936002)(77096006)(3280700002)(55016002)(25786009)(53936002)(122556002)(3660700001)(74316002)(39060400002)(7116003)(102836003)(38730400002)(6506006)(8676002)(6916009)(2950100002)(6436002)(189998001)(86362001)(1411001)(54356999)(7696004)(229853002)(6246003)(5660300001)(110136004)(50986999)(76176999)(14454004)(33656002); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0190; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; MLV:ovrnspm; PTR:InfoNoRecords; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Jun 2017 00:43:03.4961 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0190 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jun 2017 00:43:07 -0000 Mac OS X uses a very different VFS than FreeBSD (unless you go back to 10.3 Panther). As such, I'm afraid it won't be a straightforward port. Good luck with it. Maybe someone else reading this will be interested in helping out, rick ________________________________________ From: Matt B Sent: Thursday, June 22, 2017 8:14:09 PM To: Rick Macklem Cc: freebsd-fs@freebsd.org Subject: Re: SMBv1 Deprecation I totally understand. I try to support the FreeBSD Foundation with donation= s as often as I can as well as reporting bugs promptly as I am sure resourc= es are spread thin. My skill set isn't really that of a programmer though. = I am working right now at checking the Darwin/OS X code for mount_smbfs and= other modules associated with smbfs in the hopes of possibly getting somet= hing viable for BSD, even if it has to be a port due to license issues. Pro= gress is slow just due to lack of knowledge in the programming arena. On Thu, Jun 22, 2017 at 5:30 PM, Rick Macklem > wrote: Well, the short answer is...somebody has to do it. (At this time, I believe that there are two people employed by the FreeBSD Foundation to do FreeBSD kernel work.) The rest of FreeBSD's development is done by volunteers (some of which do the work for an employer and get permission from the employer to upstream the work). I, for example, do NFS as a hobby and always have, but to be honest, there aren't many out there as stupid as I am and willing to do this;-) So, if you have the skills and time, feel free to do an implementation and, so long it is appropriately licensed (no GPL or similar), I suspect someone would be willing to work with you to get it into FreeBSD. If there is an SMBv2 implementation in one of the other BSDen (NetBSD, OpenBSD,...) the port wouldn't be an immense amount of work, but there are differences in the VFS and similar that will need to be dealt with. Otherwise, you are pretty much implementing it from scratch, using the SMBv1 code as a starting point. rick ________________________________________ From: owner-freebsd-fs@freebsd.org > on behalf = of Matt B > Sent: Thursday, June 22, 2017 3:36:14 PM To: freebsd-fs@freebsd.org Subject: SMBv1 Deprecation Long time user of FreeBSD here. I have been happily using the mount_smbfs binary and in my fstab to mount Windows Shares on boot to be used by various network services house on multiple FreeBSD systems. Sadly, it appears these connections all use SMBv1 NT1 security to perform the mount operation. With the new security landscape, post-WannaCry ransomware, in a mixed-mode environment where all the shares live in Windows, that just won't do. This has been discussed many times before in the past but there hasn't been any headway AFAIK. Every other piece of software I have encountered has moved away from this deprecated network protocol to the far more secure versions of SMB to perform Windows share operations. As a stop gap, I have implemented a very rudimentary NFS server advertising shares, but configuring a Kerberos infrastructure and setting new accounts for each and every service (not to mention the new permissions nightmares even with Active Directory) on multiple BSD systems is arduous. Rather, I am wondering why FreeBSD is behind the ball on the development? The other Linux based systems I run required a simple addition of the vers=3DSMB2 fla= g to the fstab entry to successfully mount. I understand the code base is very old for the mount_smbfs, but what is the way forward here? NFS is simply a workaround as far as I am concerned and every other *nix style distro seems to play nice with SMB. Is there an ETR on this greatly needed and long overdue update to mount newer style SMB shares? _______________________________________________ freebsd-fs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Fri Jun 23 05:03:43 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E394CD9C883 for ; Fri, 23 Jun 2017 05:03:43 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B67DE780EC for ; Fri, 23 Jun 2017 05:03:43 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (124-148-108-84.dyn.iinet.net.au [124.148.108.84]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id v5N53bew062278 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Thu, 22 Jun 2017 22:03:41 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: SMBv1 Deprecation To: Matt B , Rick Macklem Cc: "freebsd-fs@freebsd.org" References: From: Julian Elischer Message-ID: <9b556cbe-f9f3-ab15-6fcd-71397d18c126@freebsd.org> Date: Fri, 23 Jun 2017 13:03:31 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jun 2017 05:03:44 -0000 On 23/6/17 8:14 am, Matt B wrote: > I totally understand. I try to support the FreeBSD Foundation with > donations as often as I can as well as reporting bugs promptly as I am sure > resources are spread thin. My skill set isn't really that of a programmer > though. I am working right now at checking the Darwin/OS X code for > mount_smbfs and other modules associated with smbfs in the hopes of > possibly getting something viable for BSD, even if it has to be a port due > to license issues. Progress is slow just due to lack of knowledge in the > programming arena. That's how we all started out.. some personal itch that had to be scratched. You do some work on it. From that you build up an expertise in that field, and then you start answering questions when people ask about that area, and then you find you've a commit bit and are spending serious time on it, and then a company offers you serious money to fix something (*) and before you know it... (*) seriously that happens. Companies have itches too but instead of spare time, they have cash. > > On Thu, Jun 22, 2017 at 5:30 PM, Rick Macklem wrote: > >> Well, the short answer is...somebody has to do it. >> (At this time, I believe that there are two people employed by >> the FreeBSD Foundation to do FreeBSD kernel work.) >> The rest of FreeBSD's development is done by volunteers >> (some of which do the work for an employer and get permission >> from the employer to upstream the work). >> I, for example, do NFS as a hobby and always have, but to be honest, >> there aren't many out there as stupid as I am and willing to do this;-) >> >> So, if you have the skills and time, feel free to do an implementation >> and, so long it is appropriately licensed (no GPL or similar), I suspect >> someone would be willing to work with you to get it into FreeBSD. >> >> If there is an SMBv2 implementation in one of the other BSDen >> (NetBSD, OpenBSD,...) the port wouldn't be an immense amount >> of work, but there are differences in the VFS and similar that will >> need to be dealt with. >> Otherwise, you are pretty much implementing it from scratch, using >> the SMBv1 code as a starting point. >> >> rick >> ________________________________________ >> From: owner-freebsd-fs@freebsd.org on >> behalf of Matt B >> Sent: Thursday, June 22, 2017 3:36:14 PM >> To: freebsd-fs@freebsd.org >> Subject: SMBv1 Deprecation >> >> Long time user of FreeBSD here. I have been happily using the mount_smbfs >> binary and in my fstab to mount Windows Shares on boot to be used by >> various network services house on multiple FreeBSD systems. Sadly, it >> appears these connections all use SMBv1 NT1 security to perform the mount >> operation. With the new security landscape, post-WannaCry ransomware, in a >> mixed-mode environment where all the shares live in Windows, that just >> won't do. This has been discussed many times before in the past but there >> hasn't been any headway AFAIK. Every other piece of software I have >> encountered has moved away from this deprecated network protocol to the far >> more secure versions of SMB to perform Windows share operations. As a stop >> gap, I have implemented a very rudimentary NFS server advertising shares, >> but configuring a Kerberos infrastructure and setting new accounts for each >> and every service (not to mention the new permissions nightmares even with >> Active Directory) on multiple BSD systems is arduous. Rather, I am >> wondering why FreeBSD is behind the ball on the development? The other >> Linux based systems I run required a simple addition of the vers=SMB2 flag >> to the fstab entry to successfully mount. I understand the code base is >> very old for the mount_smbfs, but what is the way forward here? NFS is >> simply a workaround as far as I am concerned and every other *nix style >> distro seems to play nice with SMB. Is there an ETR on this greatly needed >> and long overdue update to mount newer style SMB shares? >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Fri Jun 23 08:47:17 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 520FBD9F94B for ; Fri, 23 Jun 2017 08:47:17 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: from mail-wr0-x242.google.com (mail-wr0-x242.google.com [IPv6:2a00:1450:400c:c0c::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CAE247D324; Fri, 23 Jun 2017 08:47:16 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: by mail-wr0-x242.google.com with SMTP id k67so10802381wrc.1; Fri, 23 Jun 2017 01:47:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=FPsb4ZPS4PSuqp3YDONQbJKzhtKzCvov3N4Y7+mGlgk=; b=Tdi/zQ1lcSXbDB29BoC9roxDji6iW3vUSyY+CsQ+VClGeWvN0jxi9M9uJGq/DhyoOg yMZ8YIwytmf8h8OG78vUyVk1uujg/16Vgi3/lu2QL7JHT9KIub5yIguRHazrxpAkU6Jr Pi8c5VZui6ssBHI5p+TZoJUQo70Q4dahcaceqoUsdKhqur95XrBXYy73+kzBj0syMuZI FWG5FV5zd2n4XQdeCFOK4SMaQk7a7FIvL7myh0qXArt2T26yELtlgYUkz1m2dbgHLIpk 0W5ywOcDXOB3OcU4JY5scYdl1Al+eyt6cj9A2OzF3AXBLgxPr64TmJwxb19vI+X9CSiG z4pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=FPsb4ZPS4PSuqp3YDONQbJKzhtKzCvov3N4Y7+mGlgk=; b=pYJja2aOUFVEXsbA2kx+OkzlXZUZPVmP0I/lsRKhUYNDXQJOMLBKhiaYJDbiBGeGSY yluHxUWM6aQQM/ZuC36sbIhAvJmzIMnHIfDU/ISKODFFoHVxgihdcVsY6tiel2PM57SH pq+6KfsASpGYTXi2cGx9Iku52kEYeADhls8ngEVSyJtf/bR3jelXFXay4vV/lakuG9rk D+/s7xL0abhfz6wxROJyzqAvwDBgWXEZHPyXi46Bma6ej1aqiRsQaqmf/S0RFiM4nHkA FdwvclOWitATthgfTCcMEtgvsIisofiLSZ7ojLBGssSLiM7HLxHcgWYkYTg2vdzcy8si 6/8A== X-Gm-Message-State: AKS2vOyDKhqUIe2y85l3HkT8TLIMEg8kSUevQqCI/VuOcKHiz9JhetDz WZ11MD+qzEBYtZ0T X-Received: by 10.223.148.129 with SMTP id 1mr5417263wrr.28.1498207635031; Fri, 23 Jun 2017 01:47:15 -0700 (PDT) Received: from ernst.home (p578E14F3.dip0.t-ipconnect.de. [87.142.20.243]) by smtp.gmail.com with ESMTPSA id r5sm2068801wmr.9.2017.06.23.01.47.00 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 23 Jun 2017 01:47:00 -0700 (PDT) Date: Fri, 23 Jun 2017 10:46:54 +0200 From: Gary Jennejohn To: Julian Elischer Cc: Matt B , Rick Macklem , "freebsd-fs@freebsd.org" Subject: Re: SMBv1 Deprecation Message-ID: <20170623104654.07e5a3e0@ernst.home> In-Reply-To: <9b556cbe-f9f3-ab15-6fcd-71397d18c126@freebsd.org> References: <9b556cbe-f9f3-ab15-6fcd-71397d18c126@freebsd.org> Reply-To: gljennjohn@gmail.com X-Mailer: Claws Mail 3.15.0 (GTK+ 2.24.31; amd64-portbld-freebsd12.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jun 2017 08:47:17 -0000 On Fri, 23 Jun 2017 13:03:31 +0800 Julian Elischer wrote: > On 23/6/17 8:14 am, Matt B wrote: > > I totally understand. I try to support the FreeBSD Foundation with > > donations as often as I can as well as reporting bugs promptly as I am sure > > resources are spread thin. My skill set isn't really that of a programmer > > though. I am working right now at checking the Darwin/OS X code for > > mount_smbfs and other modules associated with smbfs in the hopes of > > possibly getting something viable for BSD, even if it has to be a port due > > to license issues. Progress is slow just due to lack of knowledge in the > > programming arena. > > That's how we all started out.. some personal itch that had to be scratched. > You do some work on it. > From that you build up an expertise in that field, and then you start answering > questions when people ask about that area, and then you find you've > a commit bit and are spending serious time on it, and then a company offers you serious > money to fix something (*) and before you know it... > > (*) seriously that happens. > Companies have itches too but instead of spare time, they have cash. > You might consider trying /usr/ports/net/samba46, which is at version 4.6.4 and appears to support SMB2 out of the box. This opinion is based on looking at the source, which has lots of smb2-specific code. I don't use Windows Shares, so I have no persoanl experience with this port. -- Gary Jennejohn From owner-freebsd-fs@freebsd.org Fri Jun 23 09:20:04 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 22C0CDA0508 for ; Fri, 23 Jun 2017 09:20:04 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8F5E67E4D5 for ; Fri, 23 Jun 2017 09:20:02 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id CA93667924 for ; Fri, 23 Jun 2017 11:14:49 +0200 (CEST) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id 5bn4lKAyuVeB for ; Fri, 23 Jun 2017 11:14:44 +0200 (CEST) Received: from mail.local.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id 1FE62678F7 for ; Fri, 23 Jun 2017 11:14:44 +0200 (CEST) Received: from bsdlo.incore (bsdlo.incore [192.168.0.84]) by mail.local.incore (Postfix) with ESMTP id 03960508A1 for ; Fri, 23 Jun 2017 11:14:44 +0200 (CEST) Message-ID: <594CDC03.3010309@incore.de> Date: Fri, 23 Jun 2017 11:14:43 +0200 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: ufs snapshot is sometimes corrupt on gjourneled partition Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jun 2017 09:20:04 -0000 I try to understand the cause for the "free inode" problem described in https://lists.freebsd.org/pipermail/freebsd-fs/2013-November/018610.html. I have setup a test server (FreeBSD 10.3-STABLE #4 r317936) with a gjournaled partition for /home: mount -t ufs | grep /home --> /dev/mirror/gmsvt7p10.journal on /home (ufs, asynchronous, local, noatime, gjournal) My test creates one snapshot of /home (gets alway inode 4) and removes this snapshot: for i in 1 2 3 4 5 6 7 8; do echo "starting snaptest $i" >/dev/console mount -u -o snapshot -o noatime -o async /home/.snap/fscktest /home echo $(ls -ils /home/.snap/fscktest) >/dev/console rm -f /home/.snap/fscktest /home done I never have more than this one snapshot at work and during the test I never have any other user processes working on /home. A typical output looks like this: Jun 21 15:59:52 root: starting snaptest 1 Jun 21 15:59:52 root: 4 26592 -r-------- 1 root operator 90762970240 21 Jun 15:59 /home/.snap/fscktest Jun 21 15:59:53 root: starting snaptest 2 Jun 21 15:59:53 root: 4 26592 -r-------- 1 root operator 90762970152 21 Jun 15:59 /home/.snap/fscktest Jun 21 15:59:54 kernel: freeing inode /home/4 with 704 blocks Jun 21 15:59:54 root: starting snaptest 3 Jun 21 15:59:54 kernel: free inode /home/4 had 704 blocks Jun 21 15:59:54 root: 4 26592 -r-------- 1 root operator 90762969976 21 Jun 15:59 /home/.snap/fscktest Jun 21 15:59:56 kernel: freeing inode /home/4 with 2112 blocks Jun 21 15:59:56 root: starting snaptest 4 Jun 21 15:59:56 kernel: free inode /home/4 had 2112 blocks Jun 21 15:59:56 root: 4 26592 -r-------- 1 root operator 90762970240 21 Jun 15:59 /home/.snap/fscktest Jun 21 15:59:57 root: starting snaptest 5 Jun 21 15:59:57 root: 4 26592 -r-------- 1 root operator 90762970240 21 Jun 15:59 /home/.snap/fscktest Jun 21 15:59:58 root: starting snaptest 6 Jun 21 15:59:58 root: 4 26592 -r-------- 1 root operator 90762970216 21 Jun 15:59 /home/.snap/fscktest Jun 21 15:59:59 kernel: freeing inode /home/4 with 192 blocks Jun 21 15:59:59 root: starting snaptest 7 Jun 21 15:59:59 kernel: free inode /home/4 had 192 blocks Jun 21 15:59:59 root: 4 26592 -r-------- 1 root operator 90762970240 21 Jun 16:00 /home/.snap/fscktest Jun 21 16:00:00 root: starting snaptest 8 Jun 21 16:00:00 root: 4 26592 -r-------- 1 root operator 90762970240 21 Jun 16:00 /home/.snap/fscktest The "free inode /home/4 had NNN blocks" message during run of the mount command is output of ffs_valloc(), because ffs_load_inode() has load the disk inode 4 with a non zero i_blocks field. The corresponding "freeing inode /home/4 with NNN blocks" message during the previous rm command is output of my following diagnostic patch in function ffs_truncate(): --- ffs_inode.c.1st 2016-06-08 17:25:21.000000000 +0200 +++ ffs_inode.c 2017-06-19 10:02:07.145360000 +0200 @@ -551,6 +551,9 @@ DIP_SET(ip, i_blocks, DIP(ip, i_blocks) - blocksreleased); else /* sanity */ DIP_SET(ip, i_blocks, 0); + if (bootverbose == 2 && DIP(ip, i_blocks) > 0) + printf("freeing inode %s/%lu with %ld blocks\n", + fs->fs_fsmnt, (u_long)ip->i_number, (long)DIP(ip, i_blocks)); ip->i_flag |= IN_CHANGE; #ifdef QUOTA (void) chkdq(ip, -blocksreleased, NOCRED, 0); The rm command can only free all the blocks of the snapshotfile (means i_blocks for inode 4 ends with zero) , if this file has the "correct" size: ls -ils /home/.snap/fscktest --> 4 53184 -r-------- 1 root operator 90762970240 Jun 17 06:15 /home/.snap/fscktest The size of the /home partition is given by diskinfo /dev/mirror/gmsvt7p10.journal --> /dev/mirror/gmsvt7p10.journal 512 90762954240 177271395 So we have 2769865 full 32kB blocks with size 90631864320. During creating a snapshot a "last block" (32kB) is written at this offset ending at 90762969088. Finally the snapblklist is written with VOP_WRITE: "Write out the list of allocated blocks to the end of the snapshot". In all my correct tests the table snapblklist is 1152 bytes in size giving the correct size of the snapshot file : 90762970240. In this case the table snapblklist has 144 entries of length 8: one lenght entry and 143 logical block numbers recorded in mapacct_ufs2(): if (acctit && expungetype == BLK_SNAP && blkno != BLK_SNAP) *ip->i_snapblklist++ = lblkno; The console output above shows three error situations with block counters 704, 2112 and 192. Dividing these values by 8 gives exactly the reduced size of the snapblocklist at the end of the snapshotfile, so in these cases the snapshotfile is corrupt. I use a test kernel with some extra counters ct_* in mapacct_ufs2(): ++ct_blkno_all; if (blkno == 0) ++ct_blkno_0; if (blkno == BLK_NOCOPY) ++ct_blkno_nocopy; if (blkno == BLK_SNAP) ++ct_blkno_snap; if (blkno == 0 || blkno == BLK_NOCOPY) continue; if (acctit && expungetype == BLK_SNAP && blkno != BLK_SNAP) { *ip->i_snapblklist++ = lblkno; ++ct_snapblklist; } if (blkno == BLK_SNAP) blkno = blkstofrags(fs, lblkno); ++ct_blkfree; ffs_blkfree(ip->i_ump, fs, vp, blkno, fs->fs_bsize, inum, vp->v_type, NULL); and for the 8 test runs shown above I can see these results using DTrace at probe expunge_ufs2:return (blkno_snap is always 0): test blkno_all blkno_0 blkno_nocopy snapblklist blkfree cg_nocopy ------------------------------------------------------------------- 1 ok 2770545 353320 2416404 143 821 2416404 2 bad 2770545 587860 2181875 132 810 2416404 3 bad 2770545 956582 1813175 110 788 2416393 4 ok 2770545 353364 2416360 143 821 2416360 5 ok 2770545 353364 2416360 143 821 2416360 6 bad 2770545 418376 2351351 140 818 2416360 7 ok 2770545 353367 2416357 143 821 2416357 8 ok 2770545 353367 2416357 143 821 2416357 For correct tests the sum of blkno_0 and blkno_nocopy is always the same (2769724), for bad tests especially the counter for blkno_nocopy is significant lower. In the test table I give one more column cg_nocopy for a counter I have added in cgaccount() to see how many entries are set to BLK_NOCOPY during copy of cylinder group maps: if (ffs_isblock(fs, cg_blksfree(cgp), loc)) { ++ct_cg_nocopy; DIP_SET(ip, i_db[loc], BLK_NOCOPY); } ... if (ffs_isblock(fs, cg_blksfree(cgp), loc)) { ++ct_cg_nocopy; ((ufs2_daddr_t *)(ibp->b_data))[indiroff] = BLK_NOCOPY; } For correct tests all the BLK_NOCOPY's which are set in cgaccount() can later be seen in mapacct_ufs2(), for bad tests many of the BLK_NOCOPY's have changed to 0. I looks like the rm command removing the previous snapshot in some way runs "in the background" simultan to expunge_ufs2() and changes some of the BLK_NOCOPY's to zero. So this may be a buffer management problem which only exists on gjourneled partitions, maybe getblk/readblock used in indiracct_ufs2() is not compatibel with gjournel in the special case of creating or removing a spapshot. A hint in this direction is the fact, that the first test after cleaning the partition with umount /home; fsck -y /home; mount /home always succeeds. The following modified test procedure never fails: for i in 1 2 3 4 5 6 7 8; do echo "starting snaptest $i" >/dev/console mount -u -o snapshot -o noatime -o async /home/.snap/fscktest /home echo $(ls -ils /home/.snap/fscktest) >/dev/console rm -f /home/.snap/fscktest /home umount /home mount /home done Another proof that the snapshot file is corrupt when the snapblklist is shortend is the fact that the rm command sporadically panics in a kernel routine that is known to be correct: nread portion of the kernel message buffer: dev = mirror/gmsvt7p10.journal, block = 19727560, fs = /home panic: ffs_blkfree_cg: freeing free block cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0857e3b1c0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0857e3b270 vpanic() at vpanic+0x126/frame 0xfffffe0857e3b2b0 panic() at panic+0x43/frame 0xfffffe0857e3b310 ffs_blkfree_cg() at ffs_blkfree_cg+0x5c6/frame 0xfffffe0857e3b3d0 ffs_blkfree() at ffs_blkfree+0x99/frame 0xfffffe0857e3b430 ffs_indirtrunc() at ffs_indirtrunc+0x474/frame 0xfffffe0857e3b510 ffs_indirtrunc() at ffs_indirtrunc+0x423/frame 0xfffffe0857e3b5f0 ffs_truncate() at ffs_truncate+0x10b4/frame 0xfffffe0857e3b7d0 ufs_inactive() at ufs_inactive+0x16b/frame 0xfffffe0857e3b810 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xf7/frame 0xfffffe0857e3b840 vinactive() at vinactive+0xc6/frame 0xfffffe0857e3b890 vputx() at vputx+0x27a/frame 0xfffffe0857e3b8f0 kern_unlinkat() at kern_unlinkat+0x243/frame 0xfffffe0857e3bae0 amd64_syscall() at amd64_syscall+0x2c6/frame 0xfffffe0857e3bbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0857e3bbf0 --- syscall (10, FreeBSD ELF64, sys_unlink), rip = 0x80095425a, rsp = 0x7fffffffe988, rbp = 0x7fffffffea20 --- Any hints solving this problem are welcome. -- Andreas Longwitz From owner-freebsd-fs@freebsd.org Fri Jun 23 10:38:00 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D395DA194C for ; Fri, 23 Jun 2017 10:38:00 +0000 (UTC) (envelope-from se@freebsd.org) Received: from mailout11.t-online.de (mailout11.t-online.de [194.25.134.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mailout00.t-online.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 11490808E8 for ; Fri, 23 Jun 2017 10:37:59 +0000 (UTC) (envelope-from se@freebsd.org) Received: from fwd18.aul.t-online.de (fwd18.aul.t-online.de [172.20.26.244]) by mailout11.t-online.de (Postfix) with SMTP id D3ACD4212569; Fri, 23 Jun 2017 12:37:50 +0200 (CEST) Received: from Stefans-MBP-2.fritz.box (r1lg70ZBoh4xDvEGS5Uc0EJtuWOppLuBuFzQb3X+fkSdC9gvNsCKUMjAXwdevNGZFF@[84.154.108.252]) by fwd18.t-online.de with (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384 encrypted) esmtp id 1dOLyC-1DGj0S0; Fri, 23 Jun 2017 12:37:44 +0200 Subject: Re: SMBv1 Deprecation To: theunusualmatt@gmail.com References: <9b556cbe-f9f3-ab15-6fcd-71397d18c126@freebsd.org> <20170623104654.07e5a3e0@ernst.home> From: Stefan Esser Cc: freebsd-fs@freebsd.org Message-ID: <45b0864b-680c-8fe0-f5a5-353b6373d069@freebsd.org> Date: Fri, 23 Jun 2017 12:37:44 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: <20170623104654.07e5a3e0@ernst.home> Content-Type: text/plain; charset=windows-1252 Content-Language: en-GB Content-Transfer-Encoding: 7bit X-ID: r1lg70ZBoh4xDvEGS5Uc0EJtuWOppLuBuFzQb3X+fkSdC9gvNsCKUMjAXwdevNGZFF X-TOI-MSGID: 90e43e2f-0b83-43b6-8c93-ff5043439130 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jun 2017 10:38:00 -0000 Am 23.06.17 um 10:46 schrieb Gary Jennejohn: > On Fri, 23 Jun 2017 13:03:31 +0800 > Julian Elischer wrote: > >> On 23/6/17 8:14 am, Matt B wrote: >>> I totally understand. I try to support the FreeBSD Foundation with >>> donations as often as I can as well as reporting bugs promptly as I am sure >>> resources are spread thin. My skill set isn't really that of a programmer >>> though. I am working right now at checking the Darwin/OS X code for >>> mount_smbfs and other modules associated with smbfs in the hopes of >>> possibly getting something viable for BSD, even if it has to be a port due >>> to license issues. Progress is slow just due to lack of knowledge in the >>> programming arena. >> >> That's how we all started out.. some personal itch that had to be scratched. >> You do some work on it. >> From that you build up an expertise in that field, and then you start answering >> questions when people ask about that area, and then you find you've >> a commit bit and are spending serious time on it, and then a company offers you serious >> money to fix something (*) and before you know it... >> >> (*) seriously that happens. >> Companies have itches too but instead of spare time, they have cash. >> > > You might consider trying /usr/ports/net/samba46, which is at > version 4.6.4 and appears to support SMB2 out of the box. As you probably know, that is the SMB *server* side, and Samba supports the even newer SMB3 - not what you were looking for ... I doubt that there are many users of smbfs, which allows *client* access to Windows servers. You may want to have a look at FuseSMB, which might be easier to port to FreeBSD than teaching smbfs newer SMB protocols. Windows servers (at least 2012 and 2016) support NFS upto version 4.1, and if you can configure the servers to provide NFS access to the relevant data, that might be the easiest route for you. Regards, STefan From owner-freebsd-fs@freebsd.org Fri Jun 23 12:10:09 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7DE4ADA3476 for ; Fri, 23 Jun 2017 12:10:09 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660046.outbound.protection.outlook.com [40.107.66.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 20CB282FEB; Fri, 23 Jun 2017 12:10:08 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Fri, 23 Jun 2017 12:10:06 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.1178.023; Fri, 23 Jun 2017 12:10:06 +0000 From: Rick Macklem To: Stefan Esser , "theunusualmatt@gmail.com" CC: "freebsd-fs@freebsd.org" Subject: Re: SMBv1 Deprecation Thread-Topic: SMBv1 Deprecation Thread-Index: AQHS647UYCj1ThwsvUCkJ0Hkf0cfhqIxY+HFgACBNe2AAD5TAIAAHvcAgAAX5XY= Date: Fri, 23 Jun 2017 12:10:05 +0000 Message-ID: References: <9b556cbe-f9f3-ab15-6fcd-71397d18c126@freebsd.org> <20170623104654.07e5a3e0@ernst.home>, <45b0864b-680c-8fe0-f5a5-353b6373d069@freebsd.org> In-Reply-To: <45b0864b-680c-8fe0-f5a5-353b6373d069@freebsd.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTXPR01MB0189; 7:TXG3qvTrHP73kl+24NRfT4Kd3GiY2vUs8DlR3FtnoDq+jvp52hdM7J6/z6Gxm5v42ZlNn1RPDB9HiyficKMPWXYFqt0r7Zbbx5tU8z/pGD6bs3XhMpYW7fCWu6Id0exZD+ofwInlrGJb2MgHiesdgBaHPvutrTCCX4k/6RoBaKvEEa/Rlmvt8/x77iE3I/hqpLvrmZj7dud4UMd9/qBxGHMdC8zfC0BOUuTZDvEZ+R1mOIICILZzK/0ztW9zjAt5wv+O+qbL14/6+Z4PnhjIXYjFpM4PeLqQ0Qhlp9wPn3sBqR0/xdiHsQ28W90zZx5YqqgSZAYFZotCsnk1fRp+9Z9WhaEyIUrPzchACvySCUsQZAPqIFUuEBWfeWOAf7gshiYjn20Ujd1DtDcaPnqOqMcIJN1SJ5rgTRkI5ioxgRP0arEhXQb/lt7w9N/puRmKME6Kv7D7ylWKCNVUlvokNYqOC9Eblc9gFL39OkQCVAszaYJOeBgkGXfBa/XXvZk9wxR7UV5T0baD4ceO21uzwq34/XWXWZtWWIMepkDA7RqRrABCTFNfQMyID1mHEh41uK3NZCeocOVDUmMrQc9x9jreZCzRvUF+tAhOMqiLiLHupSXAJOFLXwVoTLE3s8nfKXgsDoz2XjeOX+kA/uqdlpmptESFFLa0vZ7SNbUgZy8QXCZbT0nQrYgthWH0n/8D2jqBDCHTm2eve/03zDfDWPFZUrxPKWcj0oxy9JwW6n6TuvaL/1tUr14Ds4qehIkXTHgAkrQIp1tLLqlFtH169aTyu80dx0PRemtTxU3hscU= x-ms-office365-filtering-correlation-id: c5bf2aae-7553-43ee-f63b-08d4ba30c90c x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:YTXPR01MB0189; x-ms-traffictypediagnostic: YTXPR01MB0189: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(10201501046)(100000703101)(100105400095)(3002001)(93006095)(93001095)(6041248)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123564025)(20161123562025)(20161123558100)(20161123560025)(20161123555025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:YTXPR01MB0189; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:YTXPR01MB0189; x-forefront-prvs: 0347410860 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(39840400002)(39450400003)(39400400002)(39850400002)(39410400002)(24454002)(199003)(189002)(3280700002)(7116003)(2950100002)(305945005)(2501003)(7696004)(8936002)(6246003)(39060400002)(6506006)(4326008)(25786009)(102836003)(189998001)(86362001)(229853002)(5660300001)(55016002)(14454004)(9686003)(93886004)(8676002)(77096006)(81166006)(74316002)(122556002)(2906002)(3660700001)(76176999)(2900100001)(54356999)(74482002)(33656002)(38730400002)(6436002)(53936002)(478600001)(50986999); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0189; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; MLV:ovrnspm; PTR:InfoNoRecords; A:1; MX:1; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Jun 2017 12:10:05.9255 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0189 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jun 2017 12:10:09 -0000 Stefan Esser wrote: [lots of stuff snipped] > You may want to have a look at FuseSMB, which might be easier to port to > FreeBSD than teaching smbfs newer SMB protocols. Yes, if there is a fuse module, that shouldn't be too hard to get working. If there is something missing in the FreeBSD fuse interface it needs, I mig= ht be able to help with that, since I have done a few fuse patches (for the ke= rnel interface that uses the module, not the module itself). > Windows servers (at least 2012 and 2016) support NFS upto version 4.1, > and if you can configure the servers to provide NFS access to the > relevant data, that might be the easiest route for you. I've never tested the FreeBSD NFSv4.1 client against a Windows server (to be honest, I didn't know they supported one until now;-), but I might be able to help if go this route and have problems with the mounts. Good luck with it, rick From owner-freebsd-fs@freebsd.org Fri Jun 23 13:42:32 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1C539DA4F54 for ; Fri, 23 Jun 2017 13:42:32 +0000 (UTC) (envelope-from theunusualmatt@gmail.com) Received: from mail-yw0-x236.google.com (mail-yw0-x236.google.com [IPv6:2607:f8b0:4002:c05::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BB3251088; Fri, 23 Jun 2017 13:42:31 +0000 (UTC) (envelope-from theunusualmatt@gmail.com) Received: by mail-yw0-x236.google.com with SMTP id 63so17185029ywr.0; Fri, 23 Jun 2017 06:42:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=Div+2/YZb05mhJUWBRFoDA6hX0Oqbi+F7aw7J2BEmnc=; b=dQbnblQ2KUg4yG0srdWwYfnnp/JmtiJFEeKDYdlanhJljQNMimVBlpu93SD4G9p7ea 7M2I8tvQD5zWpb3hb9EJzjUI7eHNi7xL+Uy66P5RMe9TU1QrREmMq/CGR1WXDkjs09Lv LoPgptdeaoW7yzjaUslBPpsJ6rg4VXVwdKOhtLEJhouaLPg8ayMoPKAng4y2LBYG4LOd KrXUoW2T+UE2YKiElx3IbdTZ6asNvmHqweN4ybnjYFS7NrGilcL4Mj19FY6aGCkDzz+/ zqcqON030a8wicFko2PXsrq/vj/r1GMmzkrzs2SMOqGOkGWJzxVJRr6i65eBN0o8dnJL jp0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=Div+2/YZb05mhJUWBRFoDA6hX0Oqbi+F7aw7J2BEmnc=; b=byflntHjQOkMktpclJQmf2ZB8hipTSE+R+zlG/JB1XFIx7XxBnp7i7OOzm8MPQn8Lo 9mOAXzw2Q+mCc/O0MUGWWkZdKezT2SdmqPCg9jpsr4NWh9VeIEivt9ujVFxVWx5S+DBY pq8fjRHOH+Yl9CLEo+P21ggouaFi9qQI+I+V9rezXiULzsznTo3qOv4HIWqoZRrj3DRE nStPH3VK6EbKBxWw9v3+Szv13/RMZhvV5Ax8VVtLd53AlEsy+aoXiU20GNXvgO4Ij+JA RYQ5qT7Gr/UoPDE1g1xflQm6NZbB5dkLLjCRAPlHBNFi5BxxNx/s1E78mD53J4mKFRhM j+cQ== X-Gm-Message-State: AKS2vOxgxyO09PqS4RkQYBEpKbax5PZtBmC72cBH522ebcLQi94GcpGn 0Cg9xdwsXy+5YKu9H3U4qTBM+alraVIa6cs= X-Received: by 10.129.156.71 with SMTP id t68mr6532525ywg.257.1498225350994; Fri, 23 Jun 2017 06:42:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.129.103.70 with HTTP; Fri, 23 Jun 2017 06:42:30 -0700 (PDT) In-Reply-To: References: <9b556cbe-f9f3-ab15-6fcd-71397d18c126@freebsd.org> <20170623104654.07e5a3e0@ernst.home> <45b0864b-680c-8fe0-f5a5-353b6373d069@freebsd.org> From: Matt B Date: Fri, 23 Jun 2017 09:42:30 -0400 Message-ID: Subject: Re: SMBv1 Deprecation To: Rick Macklem Cc: Stefan Esser , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jun 2017 13:42:32 -0000 I am currently using the Win implementation of NFS 4.1 to provide share access in the interim. NFS does work, and it works well, but due to spread out local service accounts on the BSD systems, permissions has become a bit of a challenge. I would have to set up idmapping in the Win environment and then configure all shares with these new perms that Windows can understand. Right now, when the scripts and programs run, they plop down files/folders that have the perms of the user running the script/program. Windows loses its mind and I have to force grab ownership of the files and folders and re-inherit perms from the parent directory. Windows doesn't like that and thus it is a slow process to cascade down the NTFS ACLs. The other prong to the NFS approach is Kerberos. I would have to generate keytabs for all of these systems, some of them live in a DMZ and navigate to the shares through a firewall, which means I need to open up more ports from the DMZ back to the core for Kerberos to work. Not something I want to do. I have used the netsmb fuse module. It doesn't like being mounted via fstab. I had to modify the source code to get it to even try to mount from fstab, and even then it was clunky. I think the best way forward is to get mount_smbfs working with SMBv2 or higher. I'd love to get this working properly. I just don't know where to start here. Should I focus on getting smbfs updated? Is it even necessary to do that? Is the problem with just how mount_smbfs communicates with the share? Any ideas would be great. On Fri, Jun 23, 2017 at 8:10 AM, Rick Macklem wrote: > Stefan Esser wrote: > [lots of stuff snipped] > > You may want to have a look at FuseSMB, which might be easier to port to > > FreeBSD than teaching smbfs newer SMB protocols. > Yes, if there is a fuse module, that shouldn't be too hard to get working. > If there is something missing in the FreeBSD fuse interface it needs, I > might > be able to help with that, since I have done a few fuse patches (for the > kernel > interface that uses the module, not the module itself). > > > Windows servers (at least 2012 and 2016) support NFS upto version 4.1, > > and if you can configure the servers to provide NFS access to the > > relevant data, that might be the easiest route for you. > I've never tested the FreeBSD NFSv4.1 client against a Windows server > (to be honest, I didn't know they supported one until now;-), but I might > be able to help if go this route and have problems with the mounts. > > Good luck with it, rick > From owner-freebsd-fs@freebsd.org Fri Jun 23 13:48:31 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 789AEDA5014 for ; Fri, 23 Jun 2017 13:48:31 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3D5DB12B4 for ; Fri, 23 Jun 2017 13:48:30 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id 3A2FA679A3 for ; Fri, 23 Jun 2017 15:48:28 +0200 (CEST) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id V7MUEBXC43yx for ; Fri, 23 Jun 2017 15:48:27 +0200 (CEST) Received: from mail.local.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id EED936797C for ; Fri, 23 Jun 2017 15:48:26 +0200 (CEST) Received: from bsdlo.incore (bsdlo.incore [192.168.0.84]) by mail.local.incore (Postfix) with ESMTP id D891F508A2 for ; Fri, 23 Jun 2017 15:48:26 +0200 (CEST) Message-ID: <594D1C2A.1090306@incore.de> Date: Fri, 23 Jun 2017 15:48:26 +0200 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition References: <594CDC03.3010309@incore.de> In-Reply-To: <594CDC03.3010309@incore.de> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jun 2017 13:48:31 -0000 Oh excuse me, but there was a superfluous point at the end of the link, it should be: https://lists.freebsd.org/pipermail/freebsd-fs/2013-November/018610.html And of course I did not want to remove /home, the correct rm command is rm -f /home/.snap/fscktest Andreas Longwitz From owner-freebsd-fs@freebsd.org Fri Jun 23 16:30:29 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 76FB5DA79C5 for ; Fri, 23 Jun 2017 16:30:29 +0000 (UTC) (envelope-from se@freebsd.org) Received: from mailout11.t-online.de (mailout11.t-online.de [194.25.134.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mailout00.t-online.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 389BC6657E for ; Fri, 23 Jun 2017 16:30:28 +0000 (UTC) (envelope-from se@freebsd.org) Received: from fwd40.aul.t-online.de (fwd40.aul.t-online.de [172.20.26.139]) by mailout11.t-online.de (Postfix) with SMTP id 92D1F426480B; Fri, 23 Jun 2017 18:30:25 +0200 (CEST) Received: from Stefans-MBP-2.fritz.box (V+6lcrZSohttEn4-z8ek25wNYgPBNa40-6INFtcUJNAIiFwNpbQwoH67o6SsB7WQlB@[84.154.108.252]) by fwd40.t-online.de with (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384 encrypted) esmtp id 1dORTT-0h3Fk80; Fri, 23 Jun 2017 18:30:23 +0200 Subject: Re: SMBv1 Deprecation To: Matt B Cc: Stefan Esser , "freebsd-fs@freebsd.org" References: <9b556cbe-f9f3-ab15-6fcd-71397d18c126@freebsd.org> <20170623104654.07e5a3e0@ernst.home> <45b0864b-680c-8fe0-f5a5-353b6373d069@freebsd.org> From: Stefan Esser Message-ID: <3e73e276-0e8d-ec88-5d25-58fdff2d27a6@freebsd.org> Date: Fri, 23 Jun 2017 18:30:23 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Language: en-GB Content-Transfer-Encoding: 7bit X-ID: V+6lcrZSohttEn4-z8ek25wNYgPBNa40-6INFtcUJNAIiFwNpbQwoH67o6SsB7WQlB X-TOI-MSGID: 06f944e4-7b09-45a3-a9c1-eba200c86d5d X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jun 2017 16:30:29 -0000 Am 23.06.17 um 15:42 schrieb Matt B: > I have used the netsmb fuse module. It doesn't like being mounted via > fstab. I had to modify the source code to get it to even try to mount > from fstab, and even then it was clunky. I think the best way forward is > to get mount_smbfs working with SMBv2 or higher. I'd love to get this > working properly. I just don't know where to start here. Should I focus > on getting smbfs updated? Is it even necessary to do that? Is the > problem with just how mount_smbfs communicates with the share? Any ideas > would be great. The smbfs code in the kernel is responsible for all communication with the SMB server. You need to make that support SMB2 (or SMB3) client operations. A specification of the SMB2/3 protocol can be downloaded from Microsoft: http://download.microsoft.com/download/9/5/E/95EF66AF-9026-4BB0-A41D-A4F81802D92C/%5BMS-SMB2%5D.pdf Implementing the client is a lot simpler compared to a server, since you can rely on the server working according to the spec and only have to implement the subset of functionality you need to provide open/close/read/write/delete for the protocol version that appears most appropriate (and you can ignore all optional features that are not strictly necessary to get access to the server). But it'd be quite and undertaking, I assume ... Regards, STefan From owner-freebsd-fs@freebsd.org Fri Jun 23 17:51:23 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 07673D8650C for ; Fri, 23 Jun 2017 17:51:23 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: from connect.ultra-secure.de (connect.ultra-secure.de [88.198.71.201]) by mx1.freebsd.org (Postfix) with ESMTP id C86B36AD23; Fri, 23 Jun 2017 17:51:21 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: (Haraka outbound); Fri, 23 Jun 2017 19:50:08 +0200 Authentication-Results: connect.ultra-secure.de; iprev=pass; auth=pass (plain); spf=none smtp.mailfrom=ultra-secure.de Received-SPF: None (connect.ultra-secure.de: domain of ultra-secure.de does not designate 217.71.83.52 as permitted sender) receiver=connect.ultra-secure.de; identity=mailfrom; client-ip=217.71.83.52; helo=[192.168.1.200]; envelope-from= Received: from [192.168.1.200] (217-071-083-052.ip-tech.ch [217.71.83.52]) by connect.ultra-secure.de (Haraka/2.6.2-toaster) with ESMTPSA id A32C5F79-9FDC-4F30-BB7D-04523DABC834.1 envelope-from (authenticated bits=0) (version=TLSv1/SSLv3 cipher=AES256-SHA verify=NO); Fri, 23 Jun 2017 19:50:06 +0200 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: SMBv1 Deprecation From: Rainer Duffner In-Reply-To: <3e73e276-0e8d-ec88-5d25-58fdff2d27a6@freebsd.org> Date: Fri, 23 Jun 2017 19:50:04 +0200 Cc: Matt B , "freebsd-fs@freebsd.org" , Stefan Esser Message-Id: References: <9b556cbe-f9f3-ab15-6fcd-71397d18c126@freebsd.org> <20170623104654.07e5a3e0@ernst.home> <45b0864b-680c-8fe0-f5a5-353b6373d069@freebsd.org> <3e73e276-0e8d-ec88-5d25-58fdff2d27a6@freebsd.org> To: Stefan Esser X-Mailer: Apple Mail (2.3124) X-Haraka-GeoIP: EU, CH, 451km X-Haraka-ASN: 24951 X-Haraka-GeoIP-Received: X-Haraka-ASN: 24951 217.71.80.0/20 X-Haraka-ASN-CYMRU: asn=24951 net=217.71.80.0/20 country=CH assignor=ripencc date=2003-08-07 X-Haraka-FCrDNS: 217-071-083-052.ip-tech.ch X-Haraka-p0f: os="Mac OS X " link_type="DSL" distance=14 total_conn=1 shared_ip=N X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on spamassassin X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, HTML_MESSAGE autolearn=ham autolearn_force=no version=3.4.1 X-Haraka-Karma: score: 6, good: 3387, bad: 1, connections: 3803, history: 3386, asn_score: 928, asn_connections: 948, asn_good: 928, asn_bad: 0, pass:asn, asn_all_good, relaying Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jun 2017 17:51:23 -0000 > Am 23.06.2017 um 18:30 schrieb Stefan Esser : >=20 > But it'd be quite and undertaking, I assume ... AFAIK, SMB is far from trivial. There=E2=80=99s a list of 3rd-party products on Microsofts website that = rely on SMBv1 or they either stop working completely or are crippled = severely. Our Sophos UTM at work is on that list, too, incidentally. Also, the SMB-clients of the various Linux distributions of my coworkers = and myself currently refuse to mount our home directories from the = Windows fileserver. (Other shares work, just not the home directories - thank god we=E2=80=99r= e not dependent on that functionality) It stopped working previously, then started working again and it=E2=80=99s= now non-functional again (with an error-message that doesn=E2=80=99t = really say anything). So, whoever feels brave enough to through him/herself into this has my = deepest, heartfelt sympathy. From owner-freebsd-fs@freebsd.org Fri Jun 23 21:54:03 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EC4B9D8AFA3 for ; Fri, 23 Jun 2017 21:54:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DA2BE76333 for ; Fri, 23 Jun 2017 21:54:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5NLs384092856 for ; Fri, 23 Jun 2017 21:54:03 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219972] Unable to zpool export following some zfs recv Date: Fri, 23 Jun 2017 21:54:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: pfribeiro@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_severity Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jun 2017 21:54:04 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219972 pfribeiro@gmail.com changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|Affects Only Me |Affects Some People --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Jun 24 04:55:57 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C7D3D9795B for ; Sat, 24 Jun 2017 04:55:57 +0000 (UTC) (envelope-from kaduk@mit.edu) Received: from dmz-mailsec-scanner-4.mit.edu (dmz-mailsec-scanner-4.mit.edu [18.9.25.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AB30D83326 for ; Sat, 24 Jun 2017 04:55:56 +0000 (UTC) (envelope-from kaduk@mit.edu) X-AuditID: 1209190f-4b3ff70000004b5a-c2-594df0d4aaa6 Received: from mailhub-auth-1.mit.edu ( [18.9.21.35]) (using TLS with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by dmz-mailsec-scanner-4.mit.edu (Symantec Messaging Gateway) with SMTP id 15.16.19290.4D0FD495; Sat, 24 Jun 2017 00:55:48 -0400 (EDT) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id v5O4tlxU019472; Sat, 24 Jun 2017 00:55:47 -0400 Received: from kduck.kaduk.org (24-107-191-124.dhcp.stls.mo.charter.com [24.107.191.124]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id v5O4th4x030710 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sat, 24 Jun 2017 00:55:46 -0400 Date: Fri, 23 Jun 2017 23:55:44 -0500 From: Benjamin Kaduk To: Matt B Cc: "freebsd-fs@freebsd.org" Subject: Re: SMBv1 Deprecation Message-ID: <20170624045543.GY39245@kduck.kaduk.org> References: <9b556cbe-f9f3-ab15-6fcd-71397d18c126@freebsd.org> <20170623104654.07e5a3e0@ernst.home> <45b0864b-680c-8fe0-f5a5-353b6373d069@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.7.1 (2016-10-04) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrJIsWRmVeSWpSXmKPExsUixCmqrHvlg2+kQd9lbYtjj3+yWbxvUHFg 8pjxaT6Lx85Zd9kDmKK4bFJSczLLUov07RK4Mu6vfchW8Je/4sLTB8wNjM95uhg5OSQETCRO zzrF0sXIxSEksJhJ4uH2g+wgCSGBjYwSPZekIBJXmSTuNixg62Lk4GARUJXY/8IFpIZNQEWi ofsyM4gtAhSeOe0qG4jNLGAqsbn/NhOILSwgJ/Hg1lGwmbxAyz4tfsAKMXMqi8Ss58/YIBKC EidnPmGBaNaSuPHvJRPILmYBaYnl/zhAwpwCgRJb7qwB2yUqoCzx9/A9lgmMArOQdM9C0j0L oXsBI/MqRtmU3Crd3MTMnOLUZN3i5MS8vNQiXRO93MwSvdSU0k2M4BCV5N/BOKfB+xCjAAej Eg9vhrdvpBBrYllxZe4hRkkOJiVR3tgzPpFCfEn5KZUZicUZ8UWlOanFhxglOJiVRHgPPAAq 501JrKxKLcqHSUlzsCiJ84prNEYICaQnlqRmp6YWpBbBZGU4OJQkeJmAsSgkWJSanlqRlplT gpBm4uAEGc4DNJzxMsjw4oLE3OLMdIj8KUZFKXHeXe+BEgIgiYzSPLheUAqRyN5f84pRHOgV YV5rkCoeYPqB634FNJgJaPCMNT4gg0sSEVJSDYzmtf+ie5czvr5bftLr5O/DmzctX2QhqC4n eFx+yfkbL1TenOEWibC8fWVzCq/NkiXt0YFLl5//oRW7cU9Hy4sAJePbmy/svXvP4Ln+o1dr jJQ+a85QefNlm/bPkwo8LZ1Vy/Su9nOk3nkxlZXtkJ7kC6ndl8SN3xSX5xc4uv6p6ry9LcJw 40RTJZbijERDLeai4kQAnSaWivwCAAA= X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Jun 2017 04:55:57 -0000 On Fri, Jun 23, 2017 at 09:42:30AM -0400, Matt B wrote: > I am currently using the Win implementation of NFS 4.1 to provide share > access in the interim. NFS does work, and it works well, but due to spread > out local service accounts on the BSD systems, permissions has become a bit > of a challenge. I would have to set up idmapping in the Win environment and > then configure all shares with these new perms that Windows can understand. > Right now, when the scripts and programs run, they plop down files/folders > that have the perms of the user running the script/program. Windows loses > its mind and I have to force grab ownership of the files and folders and > re-inherit perms from the parent directory. Windows doesn't like that and > thus it is a slow process to cascade down the NTFS ACLs. The other prong to > the NFS approach is Kerberos. I would have to generate keytabs for all of > these systems, some of them live in a DMZ and navigate to the shares > through a firewall, which means I need to open up more ports from the DMZ > back to the core for Kerberos to work. Not something I want to do. What follows is a digression from the core point of the thread, but as one of the (upstream) developers for security/krb5, I would really like to know more about why you are reluctant ot open up ports for Kerberos traffic. Of course there is the sheer mundane work of actually changing the configuration to effect the opening of the ports, but it sounds like perhaps you are unhappy for some deeper reason, like perhaps a desire to reduce the overall number of open ports or a particular distrust of Kerberos. With respect to the latter, the Kerberos protocol is explicitly designed to run over a hostile network, and both the Heimdal and MIT implementations are mature projects that have many production deployments exposed to the internet. From my (presumably biased) perspective, switching to Kerberos+NFS would be a security win over SMBv1, even with the extra open ports. Thanks, Ben From owner-freebsd-fs@freebsd.org Sat Jun 24 14:35:50 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 931CDD9FBE2 for ; Sat, 24 Jun 2017 14:35:50 +0000 (UTC) (envelope-from theunusualmatt@gmail.com) Received: from mail-yb0-x22c.google.com (mail-yb0-x22c.google.com [IPv6:2607:f8b0:4002:c09::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C8F26E77D for ; Sat, 24 Jun 2017 14:35:50 +0000 (UTC) (envelope-from theunusualmatt@gmail.com) Received: by mail-yb0-x22c.google.com with SMTP id b81so908200yba.2 for ; Sat, 24 Jun 2017 07:35:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=Iopf1wB6JWwIGY7lG58PQGKNVRHylq20reFHVEyFSsY=; b=kbvzN+FKZoKppRVAIWa3q/ZKeztpvAabBZBCn/JZdbbMROVE8M5pDXa7bcOwxlUPsN eSPTNm/XhrRdwC2f3hLebdO0lbLvoHm9ir2Pb99EW/Gp/PHvLUvcE4WnLhuIQRWgJOmi Q1QNorSsdz2VmVe2IzFYq+2bnRAr5VQa90MqvonNgtARzpEWVHDwfunZ2EkyV0uyKD38 SG5C4ywgU8Ox2KE08AE9B7UXVoQ/xSctCiJgL/91U3UN8HbDU6k2Mo+A8wpld0XDKm/8 bLzEUIaoqTUPgK10nIl5KB5AA5r2nUOK/PSearZASeLPrp0GPiTGf6GYH0hslUm+F3cK uq2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=Iopf1wB6JWwIGY7lG58PQGKNVRHylq20reFHVEyFSsY=; b=rUtvLo6Ayi90Y9OII+SvEOFoy6Q1nI+ycu5OB+mueJLUVrlWTaYXsiywsFPcqcQBfA 61rKSoD5VStIgLwBS5YfWKgLvsPS6IVlr+6UmAmntyo9L8GcjzBLNGlYxJj1nzTHxZu4 t8MVsCNtGFP08/xwWvpiUQJCjCdQnO7kmNqWKl5zAcVaiRqrmTihz3+UTKMHuI81IMww b20Pn07kloNWHE35JGN9pzVpMGrKuLnKxsvG3CKTzyTR3OGAiFK2tJmhnLJCbQCMtsjx yTvMxPh0WDMPw43z/Kl22iR9JqKQyJi9QCWofchWEqjOPkbfkhSFLgJD0SJbM1+oGH+u 6b4Q== X-Gm-Message-State: AKS2vOwKvTQL+aX7X3djc0OBKpGzZXT0BeTeBqS5nV4ub+UyMs+fny9x 3FZ3cEJE4Q+2B6zIw/EZ9GW2dUJxcA== X-Received: by 10.37.43.199 with SMTP id r190mr9597134ybr.118.1498314949323; Sat, 24 Jun 2017 07:35:49 -0700 (PDT) MIME-Version: 1.0 Received: by 10.129.103.70 with HTTP; Sat, 24 Jun 2017 07:35:48 -0700 (PDT) In-Reply-To: <20170624045543.GY39245@kduck.kaduk.org> References: <9b556cbe-f9f3-ab15-6fcd-71397d18c126@freebsd.org> <20170623104654.07e5a3e0@ernst.home> <45b0864b-680c-8fe0-f5a5-353b6373d069@freebsd.org> <20170624045543.GY39245@kduck.kaduk.org> From: Matt B Date: Sat, 24 Jun 2017 10:35:48 -0400 Message-ID: Subject: Re: SMBv1 Deprecation To: Benjamin Kaduk Cc: "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Jun 2017 14:35:50 -0000 It is about decreasing the attack surface. I certainly trust the level of security and validation the Kerberos provides. The physical act of going into the security gateways and opening ports is quite the menial task. The main problem I have with the implementation is the deployment of keytabs to the physical systems, which is a bit of a process to actually get the key over there, then configuring idmapping in Windows, which brings another round of issues regarding AD structure and permissions on the shares. More ports open between the DMZ and the core is just one more negative reason (to me) to not go forward with an NFS Kerberos deployment. Kerberos and NFS are definitely a great combination when the configuration suites the situation. I am looking into figuring out how to just implement SMBv2 for BSD as I believe that is the best solution for my network architecture. On Sat, Jun 24, 2017 at 12:55 AM, Benjamin Kaduk wrote: > On Fri, Jun 23, 2017 at 09:42:30AM -0400, Matt B wrote: > > I am currently using the Win implementation of NFS 4.1 to provide share > > access in the interim. NFS does work, and it works well, but due to > spread > > out local service accounts on the BSD systems, permissions has become a > bit > > of a challenge. I would have to set up idmapping in the Win environment > and > > then configure all shares with these new perms that Windows can > understand. > > Right now, when the scripts and programs run, they plop down > files/folders > > that have the perms of the user running the script/program. Windows loses > > its mind and I have to force grab ownership of the files and folders and > > re-inherit perms from the parent directory. Windows doesn't like that and > > thus it is a slow process to cascade down the NTFS ACLs. The other prong > to > > the NFS approach is Kerberos. I would have to generate keytabs for all of > > these systems, some of them live in a DMZ and navigate to the shares > > through a firewall, which means I need to open up more ports from the DMZ > > back to the core for Kerberos to work. Not something I want to do. > > What follows is a digression from the core point of the thread, but > as one of the (upstream) developers for security/krb5, I would > really like to know more about why you are reluctant ot open up > ports for Kerberos traffic. Of course there is the sheer mundane > work of actually changing the configuration to effect the opening of > the ports, but it sounds like perhaps you are unhappy for some > deeper reason, like perhaps a desire to reduce the overall number of > open ports or a particular distrust of Kerberos. > > With respect to the latter, the Kerberos protocol is explicitly > designed to run over a hostile network, and both the Heimdal and MIT > implementations are mature projects that have many production > deployments exposed to the internet. From my (presumably biased) > perspective, switching to Kerberos+NFS would be a security win over > SMBv1, even with the extra open ports. > > Thanks, > > Ben > From owner-freebsd-fs@freebsd.org Sat Jun 24 15:30:25 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5B48DA09FA for ; Sat, 24 Jun 2017 15:30:25 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5C59F6FCB9 for ; Sat, 24 Jun 2017 15:30:25 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v5OFUIHt040964 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 24 Jun 2017 18:30:18 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v5OFUIHt040964 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v5OFUHCJ040956; Sat, 24 Jun 2017 18:30:17 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 24 Jun 2017 18:30:17 +0300 From: Konstantin Belousov To: Andreas Longwitz Cc: freebsd-fs@freebsd.org, Kirk McKusick Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition Message-ID: <20170624153017.GP3437@kib.kiev.ua> References: <594CDC03.3010309@incore.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <594CDC03.3010309@incore.de> User-Agent: Mutt/1.8.3 (2017-05-23) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Jun 2017 15:30:25 -0000 On Fri, Jun 23, 2017 at 11:14:43AM +0200, Andreas Longwitz wrote: > I try to understand the cause for the "free inode" problem described in > https://lists.freebsd.org/pipermail/freebsd-fs/2013-November/018610.html. > > I have setup a test server (FreeBSD 10.3-STABLE #4 r317936) with a > gjournaled partition for /home: > mount -t ufs | grep /home --> > /dev/mirror/gmsvt7p10.journal on /home (ufs, asynchronous, local, > noatime, gjournal) As the first thing to try, if you perform your tests on the raw partition without gjournal, does the problem stay around ? > > My test creates one snapshot of /home (gets alway inode 4) and removes > this snapshot: > > for i in 1 2 3 4 5 6 7 8; do > echo "starting snaptest $i" >/dev/console > mount -u -o snapshot -o noatime -o async /home/.snap/fscktest /home > echo $(ls -ils /home/.snap/fscktest) >/dev/console > rm -f /home/.snap/fscktest /home > done > > I never have more than this one snapshot at work and during the test I > never have any other > user processes working on /home. A typical output looks like this: > > Jun 21 15:59:52 root: starting snaptest 1 > Jun 21 15:59:52 root: 4 26592 -r-------- 1 root operator 90762970240 > 21 Jun 15:59 /home/.snap/fscktest > Jun 21 15:59:53 root: starting snaptest 2 > Jun 21 15:59:53 root: 4 26592 -r-------- 1 root operator 90762970152 > 21 Jun 15:59 /home/.snap/fscktest > Jun 21 15:59:54 kernel: freeing inode /home/4 with 704 blocks > Jun 21 15:59:54 root: starting snaptest 3 > Jun 21 15:59:54 kernel: free inode /home/4 had 704 blocks > Jun 21 15:59:54 root: 4 26592 -r-------- 1 root operator 90762969976 > 21 Jun 15:59 /home/.snap/fscktest > Jun 21 15:59:56 kernel: freeing inode /home/4 with 2112 blocks > Jun 21 15:59:56 root: starting snaptest 4 > Jun 21 15:59:56 kernel: free inode /home/4 had 2112 blocks > Jun 21 15:59:56 root: 4 26592 -r-------- 1 root operator 90762970240 > 21 Jun 15:59 /home/.snap/fscktest > Jun 21 15:59:57 root: starting snaptest 5 > Jun 21 15:59:57 root: 4 26592 -r-------- 1 root operator 90762970240 > 21 Jun 15:59 /home/.snap/fscktest > Jun 21 15:59:58 root: starting snaptest 6 > Jun 21 15:59:58 root: 4 26592 -r-------- 1 root operator 90762970216 > 21 Jun 15:59 /home/.snap/fscktest > Jun 21 15:59:59 kernel: freeing inode /home/4 with 192 blocks > Jun 21 15:59:59 root: starting snaptest 7 > Jun 21 15:59:59 kernel: free inode /home/4 had 192 blocks > Jun 21 15:59:59 root: 4 26592 -r-------- 1 root operator 90762970240 > 21 Jun 16:00 /home/.snap/fscktest > Jun 21 16:00:00 root: starting snaptest 8 > Jun 21 16:00:00 root: 4 26592 -r-------- 1 root operator 90762970240 > 21 Jun 16:00 /home/.snap/fscktest > > The "free inode /home/4 had NNN blocks" message during run of the mount > command is output of ffs_valloc(), because ffs_load_inode() has load the > disk inode 4 with a non zero i_blocks field. The corresponding "freeing > inode /home/4 with NNN blocks" message during the previous rm command > is output of my following diagnostic patch in function ffs_truncate(): > > --- ffs_inode.c.1st 2016-06-08 17:25:21.000000000 +0200 > +++ ffs_inode.c 2017-06-19 10:02:07.145360000 +0200 > @@ -551,6 +551,9 @@ > DIP_SET(ip, i_blocks, DIP(ip, i_blocks) - blocksreleased); > else /* sanity */ > DIP_SET(ip, i_blocks, 0); > + if (bootverbose == 2 && DIP(ip, i_blocks) > 0) > + printf("freeing inode %s/%lu with %ld blocks\n", > + fs->fs_fsmnt, (u_long)ip->i_number, > (long)DIP(ip, i_blocks)); > ip->i_flag |= IN_CHANGE; > #ifdef QUOTA > (void) chkdq(ip, -blocksreleased, NOCRED, 0); > > The rm command can only free all the blocks of the snapshotfile (means > i_blocks for inode 4 ends with zero) , if this file has the "correct" size: > > ls -ils /home/.snap/fscktest --> > 4 53184 -r-------- 1 root operator 90762970240 Jun 17 06:15 > /home/.snap/fscktest > > The size of the /home partition is given by > diskinfo /dev/mirror/gmsvt7p10.journal --> > /dev/mirror/gmsvt7p10.journal 512 90762954240 177271395 > > So we have 2769865 full 32kB blocks with size 90631864320. During > creating a snapshot a "last block" (32kB) is written at this offset > ending at 90762969088. Finally the snapblklist is written with > VOP_WRITE: "Write out the list of allocated blocks to the end of the > snapshot". In all my correct tests the table snapblklist is 1152 bytes > in size giving the correct size of the snapshot file : 90762970240. In > this case the table snapblklist has 144 entries of length 8: one lenght > entry and 143 logical block numbers recorded in mapacct_ufs2(): > > if (acctit && expungetype == BLK_SNAP && blkno != BLK_SNAP) > *ip->i_snapblklist++ = lblkno; > > The console output above shows three error situations with block > counters 704, 2112 and 192. Dividing these values by 8 gives exactly the > reduced size of the snapblocklist at the end of the snapshotfile, so in > these cases the snapshotfile is corrupt. I am not sure what do you mean by 'match' there. Could you explicitely mention what relations between snapshot size and leaked blocks of the free snapshot inode did you noted ? > > I use a test kernel with some extra counters ct_* in mapacct_ufs2(): > > ++ct_blkno_all; > if (blkno == 0) > ++ct_blkno_0; > if (blkno == BLK_NOCOPY) > ++ct_blkno_nocopy; > if (blkno == BLK_SNAP) > ++ct_blkno_snap; > if (blkno == 0 || blkno == BLK_NOCOPY) > continue; > if (acctit && expungetype == BLK_SNAP && blkno != BLK_SNAP) { > *ip->i_snapblklist++ = lblkno; > ++ct_snapblklist; > } > if (blkno == BLK_SNAP) > blkno = blkstofrags(fs, lblkno); > ++ct_blkfree; > ffs_blkfree(ip->i_ump, fs, vp, blkno, fs->fs_bsize, inum, > vp->v_type, NULL); > > and for the 8 test runs shown above I can see these results using DTrace > at probe expunge_ufs2:return (blkno_snap is always 0): > > test blkno_all blkno_0 blkno_nocopy snapblklist blkfree cg_nocopy > ------------------------------------------------------------------- > 1 ok 2770545 353320 2416404 143 821 2416404 > 2 bad 2770545 587860 2181875 132 810 2416404 > 3 bad 2770545 956582 1813175 110 788 2416393 > 4 ok 2770545 353364 2416360 143 821 2416360 > 5 ok 2770545 353364 2416360 143 821 2416360 > 6 bad 2770545 418376 2351351 140 818 2416360 > 7 ok 2770545 353367 2416357 143 821 2416357 > 8 ok 2770545 353367 2416357 143 821 2416357 > > For correct tests the sum of blkno_0 and blkno_nocopy is always the same > (2769724), for bad tests especially the counter for blkno_nocopy is > significant lower. In the test table I give one more column cg_nocopy > for a counter I have added in cgaccount() to see how many entries are > set to BLK_NOCOPY during copy of cylinder group maps: > > if (ffs_isblock(fs, cg_blksfree(cgp), loc)) { > ++ct_cg_nocopy; > DIP_SET(ip, i_db[loc], BLK_NOCOPY); > } > ... > if (ffs_isblock(fs, cg_blksfree(cgp), loc)) { > ++ct_cg_nocopy; > ((ufs2_daddr_t *)(ibp->b_data))[indiroff] = > BLK_NOCOPY; > } > > For correct tests all the BLK_NOCOPY's which are set in cgaccount() can > later be seen in mapacct_ufs2(), for bad tests many of the BLK_NOCOPY's > have changed to 0. > > I looks like the rm command removing the previous snapshot in some way > runs "in the background" simultan to expunge_ufs2() and changes some of > the BLK_NOCOPY's to zero. So this may be a buffer management problem > which only exists on gjourneled partitions, maybe getblk/readblock used > in indiracct_ufs2() is not compatibel with gjournel in the special case > of creating or removing a spapshot. A hint in this direction is the > fact, that the first test after cleaning the partition with > umount /home; fsck -y /home; mount /home > always succeeds. The following modified test procedure never fails: > > for i in 1 2 3 4 5 6 7 8; do > echo "starting snaptest $i" >/dev/console > mount -u -o snapshot -o noatime -o async /home/.snap/fscktest /home > echo $(ls -ils /home/.snap/fscktest) >/dev/console > rm -f /home/.snap/fscktest /home > umount /home > mount /home > done After the allocations of required blocks for the snapshot inode are finished, the filesystem is suspended. You can see the call to vfs_write_suspend() in the ffs_snapshot() where the suspension is enforced. As part of the suspension, all soft-update workitems are flushed, this is done by vfs_write_suspend() calling VFS_SYNC(MNT_SUSPEND). UFS classifies writers into primary and secondary. Primary are mostly the writes initiated by the top-level syscall entries, like direct calls to write(2) or metadata-changing ops mkdir(), create() and so on. Secondary are writes performed when system initiates metadata updates during inactivation, quota updates, softdep background processing and similar. Primary modifications are blocked outright on suspension, while secondary are waited to finish in the mentioned VFS_SYNC(MNT_SUSPEND) call. If you can provide a proof that some SU-related activity happens after the suspension is established, this would be interesting to see. Might be it is something different and much simpler, but I do not see an obvious mistake in the code, after reading your observations. > > Another proof that the snapshot file is corrupt when the snapblklist is > shortend is the fact that the rm command sporadically panics in a kernel > routine that is known to be correct: > > nread portion of the kernel message buffer: > dev = mirror/gmsvt7p10.journal, block = 19727560, fs = /home > panic: ffs_blkfree_cg: freeing free block > cpuid = 1 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfffffe0857e3b1c0 > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0857e3b270 > vpanic() at vpanic+0x126/frame 0xfffffe0857e3b2b0 > panic() at panic+0x43/frame 0xfffffe0857e3b310 > ffs_blkfree_cg() at ffs_blkfree_cg+0x5c6/frame 0xfffffe0857e3b3d0 > ffs_blkfree() at ffs_blkfree+0x99/frame 0xfffffe0857e3b430 > ffs_indirtrunc() at ffs_indirtrunc+0x474/frame 0xfffffe0857e3b510 > ffs_indirtrunc() at ffs_indirtrunc+0x423/frame 0xfffffe0857e3b5f0 > ffs_truncate() at ffs_truncate+0x10b4/frame 0xfffffe0857e3b7d0 > ufs_inactive() at ufs_inactive+0x16b/frame 0xfffffe0857e3b810 > VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xf7/frame 0xfffffe0857e3b840 > vinactive() at vinactive+0xc6/frame 0xfffffe0857e3b890 > vputx() at vputx+0x27a/frame 0xfffffe0857e3b8f0 > kern_unlinkat() at kern_unlinkat+0x243/frame 0xfffffe0857e3bae0 > amd64_syscall() at amd64_syscall+0x2c6/frame 0xfffffe0857e3bbf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0857e3bbf0 > --- syscall (10, FreeBSD ELF64, sys_unlink), rip = 0x80095425a, rsp = > 0x7fffffffe988, rbp = 0x7fffffffea20 --- > > Any hints solving this problem are welcome. From owner-freebsd-fs@freebsd.org Sat Jun 24 22:00:43 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7ACA7DA66CA for ; Sat, 24 Jun 2017 22:00:43 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 59AF37A595 for ; Sat, 24 Jun 2017 22:00:43 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id v5OM3U2Q096283; Sat, 24 Jun 2017 15:03:30 -0700 (PDT) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201706242203.v5OM3U2Q096283@chez.mckusick.com> From: Kirk McKusick To: Andreas Longwitz Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition cc: Konstantin Belousov , freebsd-fs@freebsd.org In-reply-to: <20170624153017.GP3437@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <96281.1498341810.1@chez.mckusick.com> Content-Transfer-Encoding: quoted-printable Date: Sat, 24 Jun 2017 15:03:30 -0700 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Jun 2017 22:00:43 -0000 > Date: Sat, 24 Jun 2017 18:30:17 +0300 > From: Konstantin Belousov > To: Andreas Longwitz > Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition > = > On Fri, Jun 23, 2017 at 11:14:43AM +0200, Andreas Longwitz wrote: >> I try to understand the cause for the "free inode" problem described in >> https://lists.freebsd.org/pipermail/freebsd-fs/2013-November/018610.h= tml >> = >> I have setup a test server (FreeBSD 10.3-STABLE #4 r317936) with a >> gjournaled partition for /home: >> mount -t ufs | grep /home --> >> /dev/mirror/gmsvt7p10.journal on /home (ufs, asynchronous, local, >> noatime, gjournal) > > As the first thing to try, if you perform your tests on the raw > partition without gjournal, does the problem stay around ? I concur that running your test without gjournal is the next test to try. I think that your suspicion that there is a race condition with gjournal is likely correct. And if that is true, then the problem will go away when gjournal is taken out of the stack. >> My test creates one snapshot of /home (gets alway inode 4) and removes >> this snapshot: >> = >> for i in 1 2 3 4 5 6 7 8; do >> echo "starting snaptest $i" >/dev/console >> mount -u -o snapshot -o noatime -o async /home/.snap/fscktest /hom= e >> echo $(ls -ils /home/.snap/fscktest) >/dev/console >> rm -f /home/.snap/fscktest >> done >> = >> I never have more than this one snapshot at work and during the test I >> never have any other >> user processes working on /home. A typical output looks like this: >> = >> Jun 21 15:59:52 root: starting snaptest 1 >> Jun 21 15:59:52 root: 4 26592 -r-------- 1 root operator 90762970240 >> 21 Jun 15:59 /home/.snap/fscktest >> Jun 21 15:59:53 root: starting snaptest 2 >> Jun 21 15:59:53 root: 4 26592 -r-------- 1 root operator 90762970152 >> 21 Jun 15:59 /home/.snap/fscktest >> Jun 21 15:59:54 kernel: freeing inode /home/4 with 704 blocks >> Jun 21 15:59:54 root: starting snaptest 3 >> Jun 21 15:59:54 kernel: free inode /home/4 had 704 blocks >> Jun 21 15:59:54 root: 4 26592 -r-------- 1 root operator 90762969976 >> 21 Jun 15:59 /home/.snap/fscktest >> Jun 21 15:59:56 kernel: freeing inode /home/4 with 2112 blocks >> Jun 21 15:59:56 root: starting snaptest 4 >> Jun 21 15:59:56 kernel: free inode /home/4 had 2112 blocks >> Jun 21 15:59:56 root: 4 26592 -r-------- 1 root operator 90762970240 >> 21 Jun 15:59 /home/.snap/fscktest >> Jun 21 15:59:57 root: starting snaptest 5 >> Jun 21 15:59:57 root: 4 26592 -r-------- 1 root operator 90762970240 >> 21 Jun 15:59 /home/.snap/fscktest >> Jun 21 15:59:58 root: starting snaptest 6 >> Jun 21 15:59:58 root: 4 26592 -r-------- 1 root operator 90762970216 >> 21 Jun 15:59 /home/.snap/fscktest >> Jun 21 15:59:59 kernel: freeing inode /home/4 with 192 blocks >> Jun 21 15:59:59 root: starting snaptest 7 >> Jun 21 15:59:59 kernel: free inode /home/4 had 192 blocks >> Jun 21 15:59:59 root: 4 26592 -r-------- 1 root operator 90762970240 >> 21 Jun 16:00 /home/.snap/fscktest >> Jun 21 16:00:00 root: starting snaptest 8 >> Jun 21 16:00:00 root: 4 26592 -r-------- 1 root operator 90762970240 >> 21 Jun 16:00 /home/.snap/fscktest >> = >> The "free inode /home/4 had NNN blocks" message during run of the mount >> command is output of ffs_valloc(), because ffs_load_inode() has load th= e >> disk inode 4 with a non zero i_blocks field. The corresponding "freeing >> inode /home/4 with NNN blocks" message during the previous rm command >> is output of my following diagnostic patch in function ffs_truncate(): >> = >> --- ffs_inode.c.1st 2016-06-08 17:25:21.000000000 +0200 >> +++ ffs_inode.c 2017-06-19 10:02:07.145360000 +0200 >> @@ -551,6 +551,9 @@ >> DIP_SET(ip, i_blocks, DIP(ip, i_blocks) - blocksrelease= d); >> else /* sanity */ >> DIP_SET(ip, i_blocks, 0); >> + if (bootverbose =3D=3D 2 && DIP(ip, i_blocks) > 0) >> + printf("freeing inode %s/%lu with %ld blocks\n", >> + fs->fs_fsmnt, (u_long)ip->i_number, >> (long)DIP(ip, i_blocks)); >> ip->i_flag |=3D IN_CHANGE; >> #ifdef QUOTA >> (void) chkdq(ip, -blocksreleased, NOCRED, 0); >> = >> The rm command can only free all the blocks of the snapshotfile (means >> i_blocks for inode 4 ends with zero) , if this file has the "correct" s= ize: >> = >> ls -ils /home/.snap/fscktest --> >> 4 53184 -r-------- 1 root operator 90762970240 Jun 17 06:15 >> /home/.snap/fscktest >> = >> The size of the /home partition is given by >> diskinfo /dev/mirror/gmsvt7p10.journal --> >> /dev/mirror/gmsvt7p10.journal 512 90762954240 177271395 >> = >> So we have 2769865 full 32kB blocks with size 90631864320. During >> creating a snapshot a "last block" (32kB) is written at this offset >> ending at 90762969088. Finally the snapblklist is written with >> VOP_WRITE: "Write out the list of allocated blocks to the end of the >> snapshot". In all my correct tests the table snapblklist is 1152 bytes >> in size giving the correct size of the snapshot file : 90762970240. In >> this case the table snapblklist has 144 entries of length 8: one lenght >> entry and 143 logical block numbers recorded in mapacct_ufs2(): >> = >> if (acctit && expungetype =3D=3D BLK_SNAP && blkno !=3D BLK_S= NAP) >> *ip->i_snapblklist++ =3D lblkno; >> = >> The console output above shows three error situations with block >> counters 704, 2112 and 192. Dividing these values by 8 gives exactly th= e >> reduced size of the snapblocklist at the end of the snapshotfile, so in >> these cases the snapshotfile is corrupt. > > I am not sure what do you mean by 'match' there. Could you explicitely > mention what relations between snapshot size and leaked blocks of the > free snapshot inode did you noted ? I too am confused here. Are you saying for example that 192 / 8 =3D=3D 24 and that the snapblocklist is short by 24 blocks? Because from the table below, it appears to be short by only 3 blocks. >> I use a test kernel with some extra counters ct_* in mapacct_ufs2(): >> = >> ++ct_blkno_all; >> if (blkno =3D=3D 0) >> ++ct_blkno_0; >> if (blkno =3D=3D BLK_NOCOPY) >> ++ct_blkno_nocopy; >> if (blkno =3D=3D BLK_SNAP) >> ++ct_blkno_snap; >> if (blkno =3D=3D 0 || blkno =3D=3D BLK_NOCOPY) >> continue; >> if (acctit && expungetype =3D=3D BLK_SNAP && blkno !=3D BLK_= SNAP) { >> *ip->i_snapblklist++ =3D lblkno; >> ++ct_snapblklist; >> } >> if (blkno =3D=3D BLK_SNAP) >> blkno =3D blkstofrags(fs, lblkno); >> ++ct_blkfree; >> ffs_blkfree(ip->i_ump, fs, vp, blkno, fs->fs_bsize, inum, >> vp->v_type, NULL); >> = >> and for the 8 test runs shown above I can see these results using DTrac= e >> at probe expunge_ufs2:return (blkno_snap is always 0): >> = >> test blkno_all blkno_0 blkno_nocopy snapblklist blkfree cg_nocopy >> ------------------------------------------------------------------- >> 1 ok 2770545 353320 2416404 143 821 2416404 >> 2 bad 2770545 587860 2181875 132 810 2416404 >> 3 bad 2770545 956582 1813175 110 788 2416393 >> 4 ok 2770545 353364 2416360 143 821 2416360 >> 5 ok 2770545 353364 2416360 143 821 2416360 >> 6 bad 2770545 418376 2351351 140 818 2416360 >> 7 ok 2770545 353367 2416357 143 821 2416357 >> 8 ok 2770545 353367 2416357 143 821 2416357 >> = >> For correct tests the sum of blkno_0 and blkno_nocopy is always the sam= e >> (2769724), for bad tests especially the counter for blkno_nocopy is >> significant lower. In the test table I give one more column cg_nocopy >> for a counter I have added in cgaccount() to see how many entries are >> set to BLK_NOCOPY during copy of cylinder group maps: >> = >> if (ffs_isblock(fs, cg_blksfree(cgp), loc)) { >> ++ct_cg_nocopy; >> DIP_SET(ip, i_db[loc], BLK_NOCOPY); >> } >> ... >> if (ffs_isblock(fs, cg_blksfree(cgp), loc)) { >> ++ct_cg_nocopy; >> ((ufs2_daddr_t *)(ibp->b_data))[indiroff] =3D >> BLK_NOCOPY; >> } >> = >> For correct tests all the BLK_NOCOPY's which are set in cgaccount() can >> later be seen in mapacct_ufs2(), for bad tests many of the BLK_NOCOPY's >> have changed to 0. >> = >> I looks like the rm command removing the previous snapshot in some way >> runs "in the background" simultan to expunge_ufs2() and changes some of >> the BLK_NOCOPY's to zero. So this may be a buffer management problem >> which only exists on gjourneled partitions, maybe getblk/readblock used >> in indiracct_ufs2() is not compatibel with gjournel in the special case >> of creating or removing a spapshot. A hint in this direction is the >> fact, that the first test after cleaning the partition with >> umount /home; fsck -y /home; mount /home >> always succeeds. The following modified test procedure never fails: >> = >> for i in 1 2 3 4 5 6 7 8; do >> echo "starting snaptest $i" >/dev/console >> mount -u -o snapshot -o noatime -o async /home/.snap/fscktest /hom= e >> echo $(ls -ils /home/.snap/fscktest) >/dev/console >> rm -f /home/.snap/fscktest /home >> umount /home >> mount /home >> done > > After the allocations of required blocks for the snapshot inode > are finished, the filesystem is suspended. You can see the > call to vfs_write_suspend() in the ffs_snapshot() where the > suspension is enforced. As part of the suspension, all soft-update > workitems are flushed, this is done by vfs_write_suspend() calling > VFS_SYNC(MNT_SUSPEND). > = > UFS classifies writers into primary and secondary. Primary are mostly > the writes initiated by the top-level syscall entries, like direct > calls to write(2) or metadata-changing ops mkdir(), create() and so on. > Secondary are writes performed when system initiates metadata updates > during inactivation, quota updates, softdep background processing and > similar. Primary modifications are blocked outright on suspension, while > secondary are waited to finish in the mentioned VFS_SYNC(MNT_SUSPEND) > call. > = > If you can provide a proof that some SU-related activity happens after t= he > suspension is established, this would be interesting to see. > Might be it is something different and much simpler, but I do not see > an obvious mistake in the code, after reading your observations. The mount information at the top shows: /dev/mirror/gmsvt7p10.journal on /home (ufs, asynchronous, local, noatime,= gjournal) Thus no soft updates are being used. We are running on just a basic UFS filesystem. So, soft update processing has no relevance here. >> Another proof that the snapshot file is corrupt when the snapblklist is >> shortend is the fact that the rm command sporadically panics in a kerne= l >> routine that is known to be correct: >> = >> nread portion of the kernel message buffer: >> dev =3D mirror/gmsvt7p10.journal, block =3D 19727560, fs =3D /home >> panic: ffs_blkfree_cg: freeing free block >> cpuid =3D 1 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe0857e3b1c0 >> kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0857e3b270 >> vpanic() at vpanic+0x126/frame 0xfffffe0857e3b2b0 >> panic() at panic+0x43/frame 0xfffffe0857e3b310 >> ffs_blkfree_cg() at ffs_blkfree_cg+0x5c6/frame 0xfffffe0857e3b3d0 >> ffs_blkfree() at ffs_blkfree+0x99/frame 0xfffffe0857e3b430 >> ffs_indirtrunc() at ffs_indirtrunc+0x474/frame 0xfffffe0857e3b510 >> ffs_indirtrunc() at ffs_indirtrunc+0x423/frame 0xfffffe0857e3b5f0 >> ffs_truncate() at ffs_truncate+0x10b4/frame 0xfffffe0857e3b7d0 >> ufs_inactive() at ufs_inactive+0x16b/frame 0xfffffe0857e3b810 >> VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xf7/frame 0xfffffe0857e3b840 >> vinactive() at vinactive+0xc6/frame 0xfffffe0857e3b890 >> vputx() at vputx+0x27a/frame 0xfffffe0857e3b8f0 >> kern_unlinkat() at kern_unlinkat+0x243/frame 0xfffffe0857e3bae0 >> amd64_syscall() at amd64_syscall+0x2c6/frame 0xfffffe0857e3bbf0 >> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0857e3bbf0 >> --- syscall (10, FreeBSD ELF64, sys_unlink), rip =3D 0x80095425a, rsp =3D >> 0x7fffffffe988, rbp =3D 0x7fffffffea20 --- >> = >> Any hints solving this problem are welcome. Per the suggestion at the top, I recommend trying your test without gjournal present to see if the bug goes away. If that is true, then we can focus our attention on a possible race during rm in the gjournal code. Kirk McKusick