From owner-freebsd-fs@freebsd.org Tue Jun 20 20:26:12 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37EB7DA17CC for ; Tue, 20 Jun 2017 20:26:12 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on0076.outbound.protection.outlook.com [104.47.36.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B164D7281E for ; Tue, 20 Jun 2017 20:26:10 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weatherford.onmicrosoft.com; s=selector1-ca-weatherford-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=yrcYeL+dFL0DhOy1gDOkjoOtcqZTCt0l4mb+arkWmec=; b=V3ce5KdRqeSGrgYTARkGtMvLpR+Zjky3ZBHQrbZiulGmIkrj28WKBCpsP3nf89W8yshIu92DjJSVQCZ7S9piZcJDw69Jcsuahnov5IHH7Npo4rDQifwlOAWQ5bRs3rS5IrOaQ9ZoAkYVPCzbs/eWM6PAqPa8oQI/K9CNNwm3/oU= Received: from DM5PR03CA0018.namprd03.prod.outlook.com (10.175.104.28) by MWHPR03MB3165.namprd03.prod.outlook.com (10.174.174.34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 20:26:09 +0000 Received: from BY2FFO11FD037.protection.gbl (2a01:111:f400:7c0c::186) by DM5PR03CA0018.outlook.office365.com (2603:10b6:3:118::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14 via Frontend Transport; Tue, 20 Jun 2017 20:26:08 +0000 Authentication-Results: spf=pass (sender IP is 23.103.226.20) smtp.mailfrom=ca.weatherford.com; freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=bestguesspass action=none header.from=ca.weatherford.com; Received-SPF: Pass (protection.outlook.com: domain of ca.weatherford.com designates 23.103.226.20 as permitted sender) receiver=protection.outlook.com; client-ip=23.103.226.20; helo=032-smtp-out.weatherford.com; Received: from 032-smtp-out.weatherford.com (23.103.226.20) by BY2FFO11FD037.mail.protection.outlook.com (10.1.14.222) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14 via Frontend Transport; Tue, 20 Jun 2017 20:26:08 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net (141.251.110.81) by DM2PR58MB015.032d.mgd.msft.net (141.251.110.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1178.14; Tue, 20 Jun 2017 20:26:07 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) by DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) with mapi id 15.01.1178.018; Tue, 20 Jun 2017 20:26:07 +0000 From: "Caza, Aaron" To: "freebsd-fs@freebsd.org" , Steven Hartland Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Topic: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs Thread-Index: AdLqAtGh3ZrvWZ+7QKOnTYE8YOBOpQ== Date: Tue, 20 Jun 2017 20:26:06 +0000 Message-ID: Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [141.251.136.68] x-ms-publictraffictype: Email X-MS-TrafficTypeDiagnostic: DM2PR58MB015:|MWHPR03MB3165: X-MS-Office365-Filtering-Correlation-Id: 7313e09c-4377-4a74-5ffd-08d4b81a95a8 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OrganizationHeadersPreserved: DM2PR58MB015.032d.mgd.msft.net X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:23.103.226.20; IPV:NLI; CTRY:; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(39410400002)(39400400002)(39850400002)(39450400003)(39840400002)(39860400002)(2980300002)(438002)(189002)(24454002)(51234002)(377454003)(199003)(9170700003)(81166006)(108616004)(50986999)(8676002)(2906002)(7696004)(54356999)(97756001)(24736003)(53936002)(2900100001)(50466002)(72206003)(38730400002)(5660300001)(8746002)(305945005)(106466001)(33646002)(8936002)(46406003)(47776003)(229853002)(55016002)(2501003)(53546009)(5890100001)(86146001)(66066001)(189998001)(23726003)(356003)(102836003)(3846002)(6116002)(478600001)(86362001)(42882006)(7736002)(9686003); DIR:OUT; SFP:1101; SCL:1; SRVR:MWHPR03MB3165; H:032-smtp-out.weatherford.com; FPR:; SPF:Pass; MLV:sfv; MX:1; A:0; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD037; 1:pl6f8gwIsE4avyVGNFMIMobEZRGgHfJ9mgiHr/Vq4+ikeAxvJ7oiPydLn0wxmO5hQWDBuksiHEP7AtDOtr0X3ngX+6w4C4IyyhhlyOl/Z6KJ3Z/KUR4CrKW1rzRRKgbQGNHUBaJARIFm7c903E95mPjkBehbn9Mxsu7YCPxU7fu1iUUhUYshjUbh/fYLB36VtXctQ3gAlh3jgtkE3uK6ENR3x8hYi81UvXLNs3hEwhttq2k9s4pMW7I32JjHM0MTvOnHrNlARkahhQNI6htzXboBrfjpFwY2EkawRmHWnacVXOdfJufBU5RBUGcXHmsPxiBfHu5OW1r8ev8X4wfpc10zC2fITmBZws5xuprxpdK+Z6kUqbYu2cw94rIN9Nh7d02UwFTX5xwZUtsa4ahdP7FeqtqMW+LKEI2vxitglJ/w9pbLOJXmDPGLsHwLLr+IGxkn+gKiF9Kmk69u1ikIRamL3yTTLSGGg9Vs68xSM+J9NCL0MFsMT4uCH5IV/P+EfIfUUIzVfJSKDX05bLQORA== X-CrossPremisesHeadersPromoted: BY2FFO11FD037.protection.gbl X-CrossPremisesHeadersFiltered: BY2FFO11FD037.protection.gbl X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(8251501002)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:MWHPR03MB3165; X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3165; 3:uHbeCOPHRIz3wo8XYT389LEV5GNeqDI5OFZWqi0eJIdGZrETje3rn1tPoJiD4c6g5/4ESOd9bVAoOBPg0v+4wGNI5uOc8eY7wWt916wOZhpBZaUqJd5+qm0CZyBPaYRME825x8BBwOyxM7EtbtawtUxtRq/L1qpWKdMXNHuTA+9SOxEv7xeiIuH/5Vzh/R9mpr6O2m2LiBmvgXG0m1Lad3UXQ1BmE/Qr1bSNadk9f2nNeQBIlY32t+uh99k4v/tCnBpyQawrIekAaH7Esul6gyiQISJtvoJIoNYkRQzPpiSBxDDB64bTu105NdYMoZXC2W/GvOy+0UQ49uExYBl0q0WJz0FL12UPJeHkyT6b8JS5c+C1tJQI2ouTA7ZPQwYpB+OITMIc05I0pOWHY5lq7GCYxg6aJuHSUBODK5qVq8Jq1Df1Wg0KM+8x6fMDXBMGO01e8xfzdRUOeRpXGIv/45dQrkIPRvftaM0BSG8KVc8kDvpp0GMVO0FkCRg3I8UEFHdkU+CowYhdHlRhlKd2CA== X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3165; 25:oMUvizg3+tAKjNwoIuLptovzWLkyzPMuUdK9Xg7eOdy4xcUfCTFAcuGal6hIHqAqBLgMZ6CWbuWsrih+yFQa+aIJ0Q76kYp4oVfweFxotIkdZzN0GMFD7RbmaOUVtaF3FnVbSmgQ01z/Kpakv+A81VgFfnoP14Ep8uWaMaoiv9rOVlgJ87UvmLbZU1rnRcXZ3JBCHTY5ySs3yEcNgUXuZprcFs+AmaNfsNC873mf2lbApt6UzbRLrMVTmLOc8tMf+5onZfLweLuMqwHGUc6YbyU8600iwoSBrVhT5RPrGQGoJgW8kJzG1UqZ2bS2aXb9DcmAp7QoN6fwmC+eJ+kdMbymoQOj8tSgzaYOzxDf+fHvDpcPP8pU5njgvrqZD2ABtN/bhAjgILiWE45R5Rmbzov+85tTNjjZKrF0QNHkoCfTd99PA2c+3ZyDBlxhrT9t28OdZyNbx54Wb8OOvBbohSFfoFltHtcdnWgbVTf6sCQ=; 31:0yDJ+g/Fi7a2wmZdmBHS05EKE5GaZYUpPAfwJ2ATu5VQDPlJvuNTjs6cGGZZHVsh+06ALfLFCqMMdcPgN32KCKNoZKo7jBFswNwz8sKtoAZvyzAp+H1t4wnXXfTZsmbo8gcEp2Akgiy2NtBzxzlRnI8t/bKhJiFB9IEm8ISxeRjS5pG5ZTPFV8rvgbeCh0vKEjRa8Q83MWMv7ni7ccSlFbGGJbe4UqYxlO0JdWQJ7X/8QABFSh/34QMZMWFgRWkj1Dybl7cMtIJLDrKym9VKkg== X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3165; 20:zLH8hgvtdiN2Te/NoOvvtf0LvHz7P1wxY4Qo5i+hU1mVrlX36H6bjAXh09TDPNtZBASephDv9/7ciYwmkBVF44w+kJJnMSeqb1h0ZMZ/QNMtIGXMeDGx4taJpjbUDXQ20jx6eyQKUiA/zDCbDfK5aeXK2xf4Ku1j5g3KSf2+pV5+lVcpwgxyCz2BZhDHIgK89ejCbpshhGqqimCA96yIlJtc/DKJch57grvwzpWmt8lQcx/VDPN/Dc0Nj3R7cJSb/c8waHgnFmdFa9x494YDnxMrmSeMGGeyGN5ed9uZxWjWL0FInL6yU9Hgdgb5UanqyF1cLE+lULFRonR89zzvNgEB/x39dkEE2TqHjq1ZaZDkPuzt8tuPtYX80slCXOIMimedheooAjtPVr8+NF5/CBzta/y52o8Hmqv1Q/dyI4dY9XrWymBZrPsPGQSYTQp0NSlWHQ1EbMglD3sgLtQeC42HeIf2H4m8tfcM8Pw3utA5+44ANWM4GwLOh75QDIOS X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(192374486261705); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(11241501159)(6040450)(601004)(2401047)(13018025)(8121501046)(13016025)(5005006)(10201501046)(100000703101)(100105400095)(93006095)(93004095)(3002001)(6055026)(6041248)(20161123562025)(20161123560025)(20161123555025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:MWHPR03MB3165; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:MWHPR03MB3165; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3165; 4:MBvvX7pm68dIhPX08CwXXaoNMF9y58lMF5eCSpvDWj?= =?us-ascii?Q?hSRv+gjbBdEuWrEDfKfLVnEQ+kta9psFsvTO/z2QOCZ/P/1JMFCXEARErDZa?= =?us-ascii?Q?M73LgW88oMvlIX74fOb2T0y1njXJHsXf5KmxOSQClNjkEK9GfmbgWfNvdo55?= =?us-ascii?Q?6zPZ/kqqwsRIayiEzEqwzXOFu2BMquU1V3Jgqqv5/bbNWIqIDTgHXCm8zbhB?= =?us-ascii?Q?naNnNPRM73kNB6fqkF7gK3XfAFKUtX7JpsHhjSdxQAQBeHO3P82/iJVp5cWV?= =?us-ascii?Q?IPeGbAOazsI2DLHAFyzm0YDj5US5E7OC62ZBRxe+fzA+iRluYWRzrnsWD44A?= =?us-ascii?Q?FQQh+v3QA73IP16jqSKqdUn403qrDvLTTmDMgXd92HSVW+OXOier8s372x3q?= =?us-ascii?Q?qAj/8gOYj82eEm7SMzAEQk7C1yrH9e2NgEvBc5v2Yq0hKxJznhIIE8AxIkXy?= =?us-ascii?Q?UEsWM8YNxKMlBD2FB+mxfcTW5S8mRz8pPbXUsGaLa5Fvwh8XEsHC1ZAggjhZ?= =?us-ascii?Q?lCxyeO+9avoZed5yfqv0rx0LlHFYxE64444xA+j7KfnAzPw6/g0F28JJROLL?= =?us-ascii?Q?U+Q3ZDp6NNl3y85tgZaHbLa0ni4+gTbcoo0f5uRVwNRJ8QCVkLMdm+4NiMfe?= =?us-ascii?Q?4R/XBuD8JL9SrJsAdU3Scj2Fs/wbE2J1RCNsLbs8lUpbu9t962cfGhHWEnXR?= =?us-ascii?Q?vors958TC+DZmkaURuXYZPoRUz48q1VsN5f3H+kmkO9RiwEXYi6LErx9Wd1l?= =?us-ascii?Q?q6w8YTv4mkQVrGc+DmaI7lYopMMyc95AdvuxsTpuHq3D8ZLsuC6VLg3YSWu1?= =?us-ascii?Q?rrvB3s7RYQMjDH2Jj2pILqljILQgCdYiONHH5uO6Goci2FLUQKBwgZiUK3/p?= =?us-ascii?Q?9nxUiV8zw6td4l3VKzKoX+n1QrVeNqmH9ye0b+FAyWZyE7tqpnBqKxih+AhN?= =?us-ascii?Q?TJ88y8ViMtNntD4XjVqaAMFEfKvbysHWXQ8jTrzNdL8Uzp7UJ/X/dx0KYJu2?= =?us-ascii?Q?ci3Zmjc9BSC9JGOrkLvx6SJlHWbaK6lvXQABQPNZp2gcd4wdevUqSfSlLYrV?= =?us-ascii?Q?6PokTOlbbvmtAVJTeExXLfY4e/wTTmbNfyQAaynE/tgZOXKHx28Id+V6VzgJ?= =?us-ascii?Q?qf8dep8D+88ayHwC1ZIGDm5773bjcInfg9Vb5Bxn6dOsMl7E7kwagVPdxSwP?= =?us-ascii?Q?KgOPjvsNEJwvTQ/gsb59lcVBIar49AQQnFlLoiGtszQUiKVUfr4ypUgKhIzm?= =?us-ascii?Q?Fbe34zlzflKkt15sUsfIFOLkbnqPvzfSRj4EZJ?= X-Forefront-PRVS: 03449D5DD1 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3165; 23:6qAXPCGhIB/kxb6xLfwlFZLzmWUM2RL6A06PIpySa?= =?us-ascii?Q?jF6EAyZqIp2sUU2GGCATRFKN7jDMSm47yp+fSsWBTdFIPgWd1QEDHAKs8/kr?= =?us-ascii?Q?jNPoy/jaP4Do0Bv4DRC6zc9kYj0BOBGOsmOEFw6UDkZWs5jthDIFtlLIhYmU?= =?us-ascii?Q?ObXauPse2GumxfArOHrHaTKO29nshN3nMNAeSC1I+zNuC+mjjSjaBB3JKCgU?= =?us-ascii?Q?WBdVAX4TYbqtd3S6rbDh+1LgQGzr5o+MXPWSsio3S12B7CCyVnKsZtPBmm46?= =?us-ascii?Q?TyK04NEF+t1vNPBkumSSR5cyA4NXFuc+cvYfsRnDaGeCaPVQZuQx/kYmmkW+?= =?us-ascii?Q?ZSwH70m4ejimrocysTGkTSrOIGia7AH1pLjtPgfQL6G2W2WTxe1TScC9+In4?= =?us-ascii?Q?PPaqgkZakUfrKWcvW2TdWHl2f9JeFe7p2vNOlE6mRtmGPiH7LXnh6CGBKnGp?= =?us-ascii?Q?VR4q/j3l956HrfW3eT0CgmOnBjKrv09crVnEQsnGITtMN3m5Rvy4lhtxsPkJ?= =?us-ascii?Q?Me1gZM07E7Z0JyP8bhutAIGDTTS4bKAaE0vtwKAakhRoxgemy7daiNRs9RFu?= =?us-ascii?Q?mq0VV9eveiSOOBvR11tS+TUW0IHgjPavrXErDLzBN/UTzE3JWQMnrrZkOGEC?= =?us-ascii?Q?xNbhqF4Vv9yhSEknznxpR8us/zStDBOzEtNIQQM9Eo4aLZaWXMzXtCOonzay?= =?us-ascii?Q?HwOwzjKZxq/U4F3JwvyVh+OJRtEOFTVlSQQAZfnE43Xbij/ApElf44OJaHYF?= =?us-ascii?Q?YgpbhQN87vAXF2CBV47IXt8izH0thiiy19rvwoQj0jdgHyxNX/0wvTxDfEph?= =?us-ascii?Q?xsEpC3D8TFf0WzIfPnL24ILqW1IkT8G374XevD8+NVNOtRh3zivgedEUU+3G?= =?us-ascii?Q?XZqty9IvCWazQHpVOGBJ0RzChPL9Pq7Z9FTrFIywsKd03/CT1XnyoeoL6zoR?= =?us-ascii?Q?z5UQk84BbDf5EP/eXgGZzN+nk+P5h5HqdGM54Sm72Bzdj4iUY7emEq/4mQOD?= =?us-ascii?Q?3m3egs/RUyuPheEaoF2br1vr+3w/oFaOu39No1XLdsnetYn9NdEbKE5/LmwC?= =?us-ascii?Q?RBPhxP81vxIlZy1welmLUAYC5ZTXL7J0s/Jrvai0aEeNIhMi24pxj/5pg9Qw?= =?us-ascii?Q?SgJ+SM3ShCXb+beOXzwhQZfXKXTqH62oVlgB9qocdFSggOUM0mZsmhjCeL1y?= =?us-ascii?Q?vfpS8XC2d7Zp2YW2Cgs1EbrWfNKnLXtTBfcj0bf/WFjhCUbkxOoY7LaFq1Td?= =?us-ascii?Q?KigKFPFkq4/paF8WMtsiIih2tstaEPp/Q5Cesouzl0o7VD1/eA3iMmpbaF1h?= =?us-ascii?Q?/jSpm4dUl5Rj0R4ZN3EC/joCjjPpki8MdZsnXGg1rOW?= X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; MWHPR03MB3165; 6:bTLYAKOl4PmUmgvMvQl7KMMDs31gROgEHwAaDHr+E8?= =?us-ascii?Q?DE+DoaYx2wL3rYHQm/yaN+cfkmhjfxPQqh5N3NqfeNCrGuDx/h1rtdRGz5Fx?= =?us-ascii?Q?hxSfynqbbbLdSox7EYOIzguWo067lAjFZdXhruCnp7VbHBTaeg1ibbIDxGKu?= =?us-ascii?Q?VEi0eQ2KnpNLl7TpF5r43EXO4RltjWWAonHTkHJwta+KYO6LRJxFjol76Law?= =?us-ascii?Q?a2OYXqsh3okog/tD68d4YTKA0CfAutmYd9cvqpdylFeq40xIhJPLUTG8Bz4H?= =?us-ascii?Q?9O7PBjx3g4cKXxS49lYSI7imuRJ0CQYv3ajdRweizip7voEOe/4doctn7AaU?= =?us-ascii?Q?HvgxUGzhbVGhtG3CsFaeK2DllW2K0yPwJCsN82YnOzFL5AAiNLbbcyFpJm2A?= =?us-ascii?Q?vq0DUvHuBWx3rsl7ukF0O198UKHzhS07k+vdNPvwP3LrC+th+fLFAXZAyXrC?= =?us-ascii?Q?N7YZyvo+x36zZ6DBkjGUXLm71nA+6fWm2F/jiEml8jtkAI17a0zeIpX+/bWb?= =?us-ascii?Q?9nDABfY9mOMFH+8xduxft9UJ0BSRczLmS4jGiSouh4P1H7fn4U7wMi4WrYyM?= =?us-ascii?Q?bzmqM6SSh/iTAGjIQQXj/UXXHFyAtnUNmq7qblDptM+W/2gZA8B2PGC1f4Z/?= =?us-ascii?Q?mL7OsidFOywYjON8lWXQ4BORTQSOJa0K+4qsXvHUWz2tUp5WcUfPHsEc4rcc?= =?us-ascii?Q?H/K2dJGMRNyxZJ9g6Me9o8ueryP40YdPlnCbFtD19ifgD0mlKKOOHmYLft2G?= =?us-ascii?Q?oOP/qzNS02wTfXmEiWcZ5x0o1oew85KYfWgz5FvcTzyif2MnF0Rfp2T/hx53?= =?us-ascii?Q?5CFeSifvFq+oFQ+S0sgrGfBM9Fy917RsQrojzdvWr826yEb2l0KVflGQmv1h?= =?us-ascii?Q?YlPdQT8k4XQ3MHydBts8Mh0EbLl6GUdBGjyv/HPkTfaMLixJnUK70H416dA1?= =?us-ascii?Q?yrqgEihAz28G4RRU23hMt9lz1iPjC3XZ5dJggr0lANkdz8zXhAAug7WcgbG7?= =?us-ascii?Q?Gz0M1iKHz8GkjeEimjlorJ?= X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3165; 5:DJrR4hqLIH6czwOrEAUpfVS8fongVetWy2MD0EdvPwqASCis5u6gsw1lLfxsAwTPtkBrH1C1KAzDf7MtHuE8BjBBBe0R1+DUX1i8wPWD5lzjprl7c1Vwng6Vjlmc74DXd4V4r+wsxAzAfPDdT/Z9/3LMsGfanh1+PFBWZIosdRPMPfD1ir1B3jEjxooj1vDaed5CHJGoSGru3NwNSKhdrazIbIyiloTonF3jnk+9FPIXzp8aVzEGqS89g7RSJwzv4uqw1kdbQnsMAuO+pgMovei9w90zbGxGjKbDZpDYyQfG1lz8xwZHr+EVf3IZ7PwEy+xuPmXL0a8T7P/M1YsZGpLJSrnzTI2WDrQgwf4l7KutyIZ2QRxtV+OvZPY6jr0MuNjgBVG4oS6QkbIactB/ASq3AKOECjexq3t4UKrC9Po25aZ9DqLmeLfBlH6YObJzJeWBYvjKGg7wGDLxYTnVnLMveu+YtXwwyZTck1V9ftHapZhe4GG4mMMMrukaky6h; 24:1jj6O9d/6WneSpKQlvGYFmAcwirLo6Rl7Ul0CwZfnhhvsHdGb8RHKqPei7Gt8KMH8muYVmWaODkxWSQQNzQh4vi0CTbks78J0JibehOVXKc= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; MWHPR03MB3165; 7:JK3W48yA7q74PxGctbGd5j59KKeZbrqoQhr1+UaroOFjAIcUArDvxNOVbSfbul6RWjJlx/B+FAUiNM7Thu+eMRJWwnRK0MTDc/FdGZKGtNXHfkDlCcbCFRWSkpOys9m/KFzZrrCAkCL6xf6P6cZ/ODBcHV8SMC422BWN/8QKKXVw+He1+Xa/OSxv4YZvQs/km9zgtMANUZFktKMMNFfENwz/rSIVFsbfecNqFF+/d+S+QDoueQ5gqXYdlY4I+uEMouR10VRgj1MEBfKyHqDqvFrILTFfFUpGWNTrP/cewwujQWu96frkOIe3LylC1TA+iItFxkG/2QGNumHoL+W4I9CK25CSArG+Le0cZeHlOkWUQNlOywPYaqxw4J9PycPcdwLIea3jLUSLGoM8vkli+otlpVPcR07g8URHZN16sXGnjNyLi+ggDGMU532b67xLlXR5ligl+GJnVR75MCnTMR/mbrwGI4bZVwrLwiNotJ2/pDx8C7PKlijwTXKew4madt4oqALzAMmYABwuHdijZLOEWkbKkEEH2PWLFDgbRHFHyENqjBmFOJWZzN8hdxgkm61ZPnx7VZfC34rwMEaj2QCpR5QkdddawQ13vFfQZf+ZOQkxw9j+OadAHekXDrXHSt8As+k/2dk5Hu4AUtAYjwpCQ/mJERTqkp6LbocQWibtbaS1jl18OSfGxPGxg+a9jGfHyAblOD3F6Owi+YPsw9j9sEvA29hPD9vPQr8znFid5DWZINpa9voOU8qfceV+dhp99hx4BtsEx2lrETCACRX98mGMkHE7OEckcCz3UBc= X-OriginatorOrg: ca.weatherford.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jun 2017 20:26:08.4802 (UTC) X-MS-Exchange-CrossTenant-Id: dd63fb60-07f6-4d96-8d40-ebeca61a524e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=dd63fb60-07f6-4d96-8d40-ebeca61a524e; Ip=[23.103.226.20]; Helo=[032-smtp-out.weatherford.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR03MB3165 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2017 20:26:12 -0000 > -----Original Message----- > From: Steven Hartland [mailto:killing@multiplay.co.uk] > Sent: Tuesday, June 20, 2017 1:53 PM > To: Caza, Aaron; freebsd-fs@freebsd.org > Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs > > > > On 20/06/2017 17:58, Caza, Aaron wrote: > > > >> Can you show the output from gstat -pd during this DD please. > > dT: 1.001s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name > > 0 4318 4318 34865 0.0 0 0 0.0 0 0 = 0.0 14.2| ada0 > > 0 4402 4402 35213 0.0 0 0 0.0 0 0 = 0.0 14.4| ada1 > > > > dT: 1.002s w: 1.000s > > L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name > > 1 4249 4249 34136 0.0 0 0 0.0 0 0 = 0.0 14.1| ada0 > > 0 4393 4393 35287 0.0 0 0 0.0 0 0 = 0.0 14.5| ada1 > You %busy is very low, so sounds like the bottleneck isn't in raw disk pe= rformance but somewhere else. > > Might be interesting to see if anything stands out in top -Sz and then pr= ess h for threads. > I rebooted the system to disable Trim so currently not degraded. > You also mention ada, could you share: > sysctl kern.cam > kern.cam.enc.emulate_array_devices: 1 kern.cam.sa.allow_io_split: 0 kern.cam.da.default_softtimeout: 0 kern.cam.da.send_ordered: 1 kern.cam.da.default_timeout: 60 kern.cam.da.retry_count: 4 kern.cam.da.poll_period: 3 kern.cam.ada.1.sort_io_queue: 0 kern.cam.ada.1.max_seq_zones: 0 kern.cam.ada.1.optimal_nonseq_zones: 0 kern.cam.ada.1.optimal_seq_zones: 0 kern.cam.ada.1.zone_support: None kern.cam.ada.1.zone_mode: Not Zoned kern.cam.ada.1.rotating: 0 kern.cam.ada.1.unmapped_io: 1 kern.cam.ada.1.write_cache: -1 kern.cam.ada.1.read_ahead: -1 kern.cam.ada.1.delete_method: DSM_TRIM kern.cam.ada.0.sort_io_queue: 0 kern.cam.ada.0.max_seq_zones: 0 kern.cam.ada.0.optimal_nonseq_zones: 0 kern.cam.ada.0.optimal_seq_zones: 0 kern.cam.ada.0.zone_support: None kern.cam.ada.0.zone_mode: Not Zoned kern.cam.ada.0.rotating: 0 kern.cam.ada.0.unmapped_io: 1 kern.cam.ada.0.write_cache: -1 kern.cam.ada.0.read_ahead: -1 kern.cam.ada.0.delete_method: DSM_TRIM kern.cam.ada.write_cache: 1 kern.cam.ada.read_ahead: 1 kern.cam.ada.spindown_suspend: 1 kern.cam.ada.spindown_shutdown: 1 kern.cam.ada.send_ordered: 1 kern.cam.ada.default_timeout: 30 kern.cam.ada.retry_count: 4 kern.cam.cd.timeout: 30000 kern.cam.cd.retry_count: 4 kern.cam.cd.poll_period: 3 kern.cam.scsi_delay: 5000 kern.cam.cam_srch_hi: 0 kern.cam.pmp.hide_special: 1 kern.cam.pmp.default_timeout: 30 kern.cam.pmp.retry_count: 1 kern.cam.debug_delay: 0 kern.cam.dflags: 0 kern.cam.num_doneqs: 2 kern.cam.xpt_generation: 13 kern.cam.boot_delay: 0 kern.cam.sort_io_queues: 1 > And: > sysctl vfs.zfs > vfs.zfs.trim.max_interval: 1 vfs.zfs.trim.timeout: 30 vfs.zfs.trim.txg_delay: 32 vfs.zfs.trim.enabled: 0 vfs.zfs.vol.immediate_write_sz: 32768 vfs.zfs.vol.unmap_enabled: 1 vfs.zfs.vol.recursive: 0 vfs.zfs.vol.mode: 1 vfs.zfs.version.zpl: 5 vfs.zfs.version.spa: 5000 vfs.zfs.version.acl: 1 vfs.zfs.version.ioctl: 7 vfs.zfs.debug: 0 vfs.zfs.super_owner: 0 vfs.zfs.immediate_write_sz: 32768 vfs.zfs.sync_pass_rewrite: 2 vfs.zfs.sync_pass_dont_compress: 5 vfs.zfs.sync_pass_deferred_free: 2 vfs.zfs.zio.dva_throttle_enabled: 1 vfs.zfs.zio.exclude_metadata: 0 vfs.zfs.zio.use_uma: 1 vfs.zfs.zil_slog_limit: 786432 vfs.zfs.cache_flush_disable: 0 vfs.zfs.zil_replay_disable: 0 vfs.zfs.min_auto_ashift: 12 vfs.zfs.max_auto_ashift: 13 vfs.zfs.vdev.trim_max_pending: 10000 vfs.zfs.vdev.bio_delete_disable: 0 vfs.zfs.vdev.bio_flush_disable: 0 vfs.zfs.vdev.queue_depth_pct: 1000 vfs.zfs.vdev.write_gap_limit: 4096 vfs.zfs.vdev.read_gap_limit: 32768 vfs.zfs.vdev.aggregation_limit: 131072 vfs.zfs.vdev.trim_max_active: 64 vfs.zfs.vdev.trim_min_active: 1 vfs.zfs.vdev.scrub_max_active: 2 vfs.zfs.vdev.scrub_min_active: 1 vfs.zfs.vdev.async_write_max_active: 10 vfs.zfs.vdev.async_write_min_active: 1 vfs.zfs.vdev.async_read_max_active: 3 vfs.zfs.vdev.async_read_min_active: 1 vfs.zfs.vdev.sync_write_max_active: 10 vfs.zfs.vdev.sync_write_min_active: 10 vfs.zfs.vdev.sync_read_max_active: 10 vfs.zfs.vdev.sync_read_min_active: 10 vfs.zfs.vdev.max_active: 1000 vfs.zfs.vdev.async_write_active_max_dirty_percent: 60 vfs.zfs.vdev.async_write_active_min_dirty_percent: 30 vfs.zfs.vdev.mirror.non_rotating_seek_inc: 1 vfs.zfs.vdev.mirror.non_rotating_inc: 0 vfs.zfs.vdev.mirror.rotating_seek_offset: 1048576 vfs.zfs.vdev.mirror.rotating_seek_inc: 5 vfs.zfs.vdev.mirror.rotating_inc: 0 vfs.zfs.vdev.trim_on_init: 1 vfs.zfs.vdev.cache.bshift: 16 vfs.zfs.vdev.cache.size: 0 vfs.zfs.vdev.cache.max: 16384 vfs.zfs.vdev.metaslabs_per_vdev: 200 vfs.zfs.txg.timeout: 5 vfs.zfs.space_map_blksz: 4096 vfs.zfs.spa_min_slop: 134217728 vfs.zfs.spa_slop_shift: 5 vfs.zfs.spa_asize_inflation: 24 vfs.zfs.deadman_enabled: 1 vfs.zfs.deadman_checktime_ms: 5000 vfs.zfs.deadman_synctime_ms: 1000000 vfs.zfs.debug_flags: 0 vfs.zfs.debugflags: 0 vfs.zfs.recover: 0 vfs.zfs.spa_load_verify_data: 1 vfs.zfs.spa_load_verify_metadata: 1 vfs.zfs.spa_load_verify_maxinflight: 10000 vfs.zfs.ccw_retry_interval: 300 vfs.zfs.check_hostid: 1 vfs.zfs.mg_fragmentation_threshold: 85 vfs.zfs.mg_noalloc_threshold: 0 vfs.zfs.condense_pct: 200 vfs.zfs.metaslab.bias_enabled: 1 vfs.zfs.metaslab.lba_weighting_enabled: 1 vfs.zfs.metaslab.fragmentation_factor_enabled: 1 vfs.zfs.metaslab.preload_enabled: 1 vfs.zfs.metaslab.preload_limit: 3 vfs.zfs.metaslab.unload_delay: 8 vfs.zfs.metaslab.load_pct: 50 vfs.zfs.metaslab.min_alloc_size: 33554432 vfs.zfs.metaslab.df_free_pct: 4 vfs.zfs.metaslab.df_alloc_threshold: 131072 vfs.zfs.metaslab.debug_unload: 0 vfs.zfs.metaslab.debug_load: 0 vfs.zfs.metaslab.fragmentation_threshold: 70 vfs.zfs.metaslab.gang_bang: 16777217 vfs.zfs.free_bpobj_enabled: 1 vfs.zfs.free_max_blocks: 18446744073709551615 vfs.zfs.no_scrub_prefetch: 0 vfs.zfs.no_scrub_io: 0 vfs.zfs.resilver_min_time_ms: 3000 vfs.zfs.free_min_time_ms: 1000 vfs.zfs.scan_min_time_ms: 1000 vfs.zfs.scan_idle: 50 vfs.zfs.scrub_delay: 4 vfs.zfs.resilver_delay: 2 vfs.zfs.top_maxinflight: 32 vfs.zfs.zfetch.array_rd_sz: 1048576 vfs.zfs.zfetch.max_idistance: 67108864 vfs.zfs.zfetch.max_distance: 8388608 vfs.zfs.zfetch.min_sec_reap: 2 vfs.zfs.zfetch.max_streams: 8 vfs.zfs.prefetch_disable: 0 vfs.zfs.delay_scale: 500000 vfs.zfs.delay_min_dirty_percent: 60 vfs.zfs.dirty_data_sync: 67108864 vfs.zfs.dirty_data_max_percent: 10 vfs.zfs.dirty_data_max_max: 4294967296 vfs.zfs.dirty_data_max: 853010841 vfs.zfs.max_recordsize: 1048576 vfs.zfs.send_holes_without_birth_time: 1 vfs.zfs.mdcomp_disable: 0 vfs.zfs.nopwrite_enabled: 1 vfs.zfs.dedup.prefetch: 1 vfs.zfs.l2c_only_size: 0 vfs.zfs.mfu_ghost_data_esize: 393216 vfs.zfs.mfu_ghost_metadata_esize: 224032256 vfs.zfs.mfu_ghost_size: 224425472 vfs.zfs.mfu_data_esize: 0 vfs.zfs.mfu_metadata_esize: 2973696 vfs.zfs.mfu_size: 8696320 vfs.zfs.mru_ghost_data_esize: 261177344 vfs.zfs.mru_ghost_metadata_esize: 0 vfs.zfs.mru_ghost_size: 261177344 vfs.zfs.mru_data_esize: 6841171968 vfs.zfs.mru_metadata_esize: 9506816 vfs.zfs.mru_size: 6954673152 vfs.zfs.anon_data_esize: 0 vfs.zfs.anon_metadata_esize: 0 vfs.zfs.anon_size: 22016 vfs.zfs.l2arc_norw: 1 vfs.zfs.l2arc_feed_again: 1 vfs.zfs.l2arc_noprefetch: 1 vfs.zfs.l2arc_feed_min_ms: 200 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_headroom: 2 vfs.zfs.l2arc_write_boost: 8388608 vfs.zfs.l2arc_write_max: 8388608 vfs.zfs.arc_meta_limit: 1803964416 vfs.zfs.arc_free_target: 14080 vfs.zfs.compressed_arc_enabled: 1 vfs.zfs.arc_shrink_shift: 7 vfs.zfs.arc_average_blocksize: 8192 vfs.zfs.arc_min: 901982208 vfs.zfs.arc_max: 7215857664 > Finally when its performing well can you repeat the gstat -pd > dT: 1.001s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d= %busy Name 3 3887 3887 426514 0.7 0 0 0.0 0 0 0.0= 90.7| ada0 3 3987 3987 434702 0.7 0 0 0.0 0 0 0.0= 92.0| ada1 dT: 1.002s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d= %busy Name 3 3958 3958 433563 0.7 0 0 0.0 0 0 0.0= 91.6| ada0 3 3989 3989 438417 0.7 0 0 0.0 0 0 0.0= 93.0| ada1 test@f111beta2:~ # dd if=3D/testdb/test of=3D/dev/null bs=3D1m 16000+0 records in 16000+0 records out 16777216000 bytes transferred in 19.385855 secs (865435959 bytes/sec) Note that this is with vfs.zfs.arc_min and vfs.zfs.arc_max unconstrained. = In previous testing on FreeBSD 10.3 Stable, I had tried a setting of "4000M= " for vfs.zfs.arc_min and vfs.zfs.arc_max and had still experienced the per= formance degradation though that was using the GEOM ELI layering so this la= test testing using ashift instead on FreeBSD 11.1 Beta 2 may exhibit a diff= erent behavior. -- Aaron This message may contain confidential and privileged information. If it has= been sent to you in error, please reply to advise the sender of the error = and then immediately delete it. If you are not the intended recipient, do n= ot read, copy, disclose or otherwise use this message. The sender disclaims= any liability for such unauthorized use. PLEASE NOTE that all incoming e-m= ails sent to Weatherford e-mail accounts will be archived and may be scanne= d by us and/or by external service providers to detect and prevent threats = to our systems, investigate illegal or inappropriate behavior, and/or elimi= nate unsolicited promotional e-mails (spam). This process could result in d= eletion of a legitimate e-mail before it is read by its intended recipient = at our organization. Moreover, based on the scanning results, the full text= of e-mails and attachments may be made available to Weatherford security a= nd other personnel for review and appropriate action. If you have any conce= rns about this process, please contact us at dataprivacy@weatherford.com.