From owner-freebsd-hackers@freebsd.org Sun Jun 11 16:51:18 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9CE13D8A8B7 for ; Sun, 11 Jun 2017 16:51:18 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) Received: from NAM03-BY2-obe.outbound.protection.outlook.com (mail-by2nam03on0079.outbound.protection.outlook.com [104.47.42.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3BC286F431; Sun, 11 Jun 2017 16:51:17 +0000 (UTC) (envelope-from Aaron.Caza@ca.weatherford.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weatherford.onmicrosoft.com; s=selector1-ca-weatherford-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=f10qQohDYboXn0G1k/EqtP3PxWMKvTnPc9VsMUJrtG0=; b=vbcZVgOCYJ4ttD08fG+rGGNfcUrzY3uSZnvSdZ6RJUKyqN01Ppq4bxeUjj32GLDtnNtos0H5eE+k3Zz9TVge9bVzSUDRLIoLfscRqYEMaMkZv7YgsEJqcqv9fwmfpuuPaFh8kSVHIG8OdmeKPTrMiSpAzsGAOfRRhQ0GEv4ehL0= Received: from CY1PR03CA0023.namprd03.prod.outlook.com (10.174.128.33) by BY2PR0301MB0743.namprd03.prod.outlook.com (10.160.63.21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1157.12; Sun, 11 Jun 2017 16:51:16 +0000 Received: from BY2FFO11FD033.protection.gbl (2a01:111:f400:7c0c::101) by CY1PR03CA0023.outlook.office365.com (2603:10b6:600::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1157.12 via Frontend Transport; Sun, 11 Jun 2017 16:51:16 +0000 Authentication-Results: spf=pass (sender IP is 23.103.226.20) smtp.mailfrom=ca.weatherford.com; freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=bestguesspass action=none header.from=ca.weatherford.com; Received-SPF: Pass (protection.outlook.com: domain of ca.weatherford.com designates 23.103.226.20 as permitted sender) receiver=protection.outlook.com; client-ip=23.103.226.20; helo=032-smtp-out.weatherford.com; Received: from 032-smtp-out.weatherford.com (23.103.226.20) by BY2FFO11FD033.mail.protection.outlook.com (10.1.14.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1143.11 via Frontend Transport; Sun, 11 Jun 2017 16:51:15 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net (141.251.110.81) by DM2PR58MB013.032d.mgd.msft.net (141.251.110.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1157.12; Sun, 11 Jun 2017 16:51:14 +0000 Received: from DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) by DM2PR58MB013.032d.mgd.msft.net ([141.251.110.81]) with mapi id 15.01.1157.017; Sun, 11 Jun 2017 16:51:14 +0000 From: "Caza, Aaron" To: Allan Jude , "freebsd-hackers@freebsd.org" Subject: Re: FreeBSD10 Stable + ZFS + PostgreSQL + SSD performance drop < 24 hours Thread-Topic: FreeBSD10 Stable + ZFS + PostgreSQL + SSD performance drop < 24 hours Thread-Index: AdLi0u6ENydglNrDQ4GNB7pMA3Leng== Date: Sun, 11 Jun 2017 16:51:13 +0000 Message-ID: Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [141.251.110.68] x-ms-publictraffictype: Email X-MS-TrafficTypeDiagnostic: DM2PR58MB013:|BY2PR0301MB0743: X-MS-Office365-Filtering-Correlation-Id: 7e34c11b-4227-4cbf-4cf9-08d4b0ea1307 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OrganizationHeadersPreserved: DM2PR58MB013.032d.mgd.msft.net X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:23.103.226.20; IPV:NLI; CTRY:; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(39450400003)(39850400002)(39410400002)(39400400002)(39840400002)(39860400002)(2980300002)(438002)(13464003)(24454002)(377454003)(199003)(189002)(9170700003)(966005)(55016002)(86146001)(38730400002)(72206003)(2906002)(42882006)(189998001)(551544002)(5660300001)(478600001)(8676002)(97756001)(9686003)(6306002)(450100002)(53936002)(23726003)(86362001)(6246003)(102836003)(53546009)(3846002)(81166006)(46406003)(8746002)(356003)(8936002)(108616004)(47776003)(24736003)(66066001)(7696004)(33646002)(22756006)(5890100001)(2501003)(54356999)(50986999)(2900100001)(305945005)(7736002)(229853002); DIR:OUT; SFP:1101; SCL:1; SRVR:BY2PR0301MB0743; H:032-smtp-out.weatherford.com; FPR:; SPF:Pass; MLV:sfv; A:0; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD033; 1:lP2lrgj9AAO4oUdTb37sudYm/WHbUloo2KxNFYDUtlKNUKZdHDqnLecDZBdTZMernUAcegCihWCYvCqAlez0oTwZqINkj5C/z9lPlJkXJifRDDbcYanMoGIqFKxpjJVkrDdD4kUw3JFmkjYqXqNOUOsbddPUQUYVC5P/Rrd5BhrsgOp75lw70tKHtQCyy9x9daxAQ4DTembAuHXpF9nWnHcoUN6J+33PfU+BuFCLvKy37uJ7Jn058oZ7EKq757n6kSNX13jOcW6v/YmCVvyf/M9WxSCHc69FBCYSJt3WxGSSx0kVeHz0YB27X4JBvHqWVxDtR0Tc9sGn4VH1KK7I1mbSHInI+4cgl6lYIPlQF9cNSLN4lSeLM+5CWLDIoWZee4gqkjfsIcROVND2ZXieifTaIahnsx+vvEqoCt6o8HCv+3vzHLERl6PFhIhbIfPu+RTyd6X7JyQY8mJlXJG2TJ+0K0x1c+fA/37zkovmKBp/IzIuT8VTYofMX+wnHz7hotPo5ECnxvcizuBirXA91w== X-CrossPremisesHeadersPromoted: BY2FFO11FD033.protection.gbl X-CrossPremisesHeadersFiltered: BY2FFO11FD033.protection.gbl X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(8251501002)(2017030254075)(201703131423075)(201703031133081); SRVR:BY2PR0301MB0743; X-Microsoft-Exchange-Diagnostics: 1; BY2PR0301MB0743; 3:8jR6RJB0ugLz1LnafpxQ6KIOpGmk4AegnguSsa2n0kndoC9YGhj47heGkg+mgVSDZTe9+CQyM0mhBUU4mm4e1lbUpngbYTK+VFcW1r0D7JqYGZKhc+q2+TbmBZsefCQSnp2ybkOkJtMogx9x89XR1CoH1aTuH7dHhE/Z10yDBz7atM6zygIMykBKILvsWbwztvet7h/HsiDYSz67dr8WpObkO329LKbc3Pvy4Uee2AziSbY/rMdU80wtVShjiDJCRInQCmEMq39KAgd3aLjmP3xx9nLvqZpYAF4VgZ2ciKoSn9DtkPDev1dKmWuxkinZlpu++USiKKJVmSZfkwA5CU04nQiKNE8C6pTeRn7Isdg5LjykqtrvsQVS9/d6hFnuACbG+XV/lmE74rT7nkuKOVFX5BrwlDU9cAqKMXV5zQCIlgezDAnrqmYjd3DUK6QmUxILu2R8jz+LV7DOgYkVygMq3LpH8eBsCevgIf0c00s= X-Microsoft-Exchange-Diagnostics: 1; BY2PR0301MB0743; 25:XgdUfeERWxrCHURXjTS7vEjxtUq5zYzJyLDKJ7E+xBZ3H4EP91uJXapaYmPd/ofi/YQJagCh65G8kDlpRWk1cyXaa33duU7Fkeqro/+Jm8AqaFvz3V+yBu50F1KvpVNb5sdn2Y8F28PdTL11Ipk6RvdSU+FtdQf026HpBEXkkSRnsTACAWXRe5N3ImeqROSTc5LK0vDC4tUsr7etB5t5pbiGLyFipJg3+pPH8tMNikhNNC1crbkm5KZaP7XBjLl/83RgnQeBgzjLX7ktClCzIxsrBBRK7hzbieQJMQFGwQr1m+4tZnTaUs3jpUK46hOnrXF81pw+SwQfMhkk8bjULgOBYxdAaq+KqkZMQU6y60e+TKQhFL8bJ84m591duS8JGi5HA3lCK0eAfmys5P5lWtdM/nR0Sy0Od78m56a3VL0v5vv9Aj85z9y/xAxI8ZNmh8dfG6ptSTp26qeEjbMLJbX+0cC7zmaBEveiBzcbarM=; 31:Xl5TQD0Rfay72gQoS9os461jO/TzGcVNjOv0TttfKrPVXt9UwjFe8gaGY3mYbKZ2e9VhbGApHxK4tCJAWLPN2gVDRtQKxrhILpcCQT0dEyTq5B6AFbA10dG1AwbVMroT68LFSnSZOT3p28TXMTGevOzKhKQZodCDsDxCyava/x9OcGhmPhMqW7HXTbe6C9R3EHJ0Z+P8IF9m1PQHrgqM0B2cNFA6mgk2KQVhwyNiAgMYac5kVO+duwAtSa1a8rgORzziDFEccUmhZLTXGhmkMg== X-Microsoft-Exchange-Diagnostics: 1; BY2PR0301MB0743; 20:xYEdzOpFlYeIOG9XFP1vIlzxHB2v3RejCmp0324qiAlpUh1CpA64Z1umS9GDMWSneqM3LIk/tzBov/9kp7NH6kpFOmvM8DdwCbGoUK4GLuAnpRwsOqmGq6lGJ3GTcyS11EJpTO6A/F64Pjb0pscm77eXwHTA9HApk2yNSVYPHisxVS+/gwoC5UVoZCOFFKzUJ6zeVbfTk8mfBX/3iydAuT9o/FoeeFGW3bgwlbj+s+KpMHS1RswMf3KU3Yhv/u6S0pixzeNzuMacbssm/p/woFh0W0T+C8UNIMrbTO4IbStIJPv0t09GDpy/LLvDUbWK7fBfxmkhAbZDmGLulcbYJ93GW7QIPcoxSv8esFzxIhVnntU1BDjrYh2zEyeUz6jhZToEUOcsJt5Lj+teBAI2asNbk7v7Kw1zJvcsNUMHoDvOLbf3dL4c3D+5Yv5vBIETyz+duQL0qzv1RNAmX3g6ZuMhj8kickw56kZAwNtnfvQTSq5YlBGVKGVQUb835uOX X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(158342451672863)(192374486261705)(75325880899374); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(11241501159)(6040450)(601004)(2401047)(13018025)(13016025)(8121501046)(5005006)(93006095)(93004095)(3002001)(100000703101)(100105400095)(10201501046)(6055026)(6041248)(20161123555025)(20161123558100)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123564025)(20161123560025)(20161123562025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:BY2PR0301MB0743; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:BY2PR0301MB0743; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BY2PR0301MB0743; 4:sDJ14HVy6YwIlm7QMNh05GG4eMXVzicK9PyizGRs?= =?us-ascii?Q?IlSvMOWbMtRUcXRtzfrtzFMnj+4TtgEYnWAke89iigZaxgJCjPo17q6dBjCr?= =?us-ascii?Q?lcsim0KkyFrV9J5pBHnQiK5EcQjtPAr4ay2sO917dIYjqkx6M6KUqXZ62fuZ?= =?us-ascii?Q?gnQXlfchkY7z0Ke724NAl5+Ez0JxjaCcPRLwOUFLIdrXUvzyv+ZvHRdN3laH?= =?us-ascii?Q?G+4I0gRtGhOKlwQO7o1dH4uyojOmG/piNlFoCD+avMWLObMADq7mmWL4ozLY?= =?us-ascii?Q?bJXxZMRvMAXaS5byplaDI9r2TFKeHlkoTIzZUu41x1YGndTDsNIlmCFSRpjD?= =?us-ascii?Q?i33uH4ZH3SVRmxvuQ3wG/nOmstA4Wyb/a+tPz34KccgEtX9Yd2YSL23GAG1F?= =?us-ascii?Q?k2WVyCmiG9vDHaHfg2bB7kU30bXntZr7B2Lr5p+z82C/caqoRjI0acRvDVQi?= =?us-ascii?Q?F7crd2EG2D5MujQMqrFDADnnJ+63gr6FfUHLueo494VRBCf25+0XRwmTXcwm?= =?us-ascii?Q?ifRJRbxdIHQwIJJqmMG5tggnujt96SEuZLCs1+w/payLQoKRg8Xv8uEpL5pQ?= =?us-ascii?Q?xML9Pagfxd99GiVfltsV+p0KKJTwX5B0nYa0o186Ox0qvdCkDdYrKYwH3Jd0?= =?us-ascii?Q?i7rcUMgRfBSElWPwoQIC1cFku6/q2ALdWYZDLsV/Q9lB8vZwk6VzVgJh3o4n?= =?us-ascii?Q?w3ZQEJwCRgopDC5DvsE6vWrZCLDR2EE7RMJORtKT14y60GbX4r4k7l4F51Qn?= =?us-ascii?Q?VFmBKy4ZuLPPJL+GK36OTQccn3Y5ODegBtgoXriHhvShIqcTj+m3ADnQee1e?= =?us-ascii?Q?/NGIFFjDxhZzayM6l84y5tYjbV2FUuRTkJw3sfUSkRtcSTRR/ECBT/LywWCD?= =?us-ascii?Q?F/4NiqhPyoVZQYiQ0zHKnuF36SdT5sYkAzHuPIZz+kUuXwa41LYAqZBYCjU0?= =?us-ascii?Q?3aiBWfBCZibeYqKADqHUNQMSQsTYjg78/Uh3b6cvMfa28BJoywLs0tzF4liL?= =?us-ascii?Q?s5sF9Jnxw4wyKfratj4kT4/n5FlsehnyD+AOUBal6b2bO1xtdDnggRb94I7u?= =?us-ascii?Q?WPUxMsL408oAvPmKgCaCOyUD/YXOERmEfjGi9i7D4dx3DKavOOAEw9idL5x+?= =?us-ascii?Q?0acOEGSM68ek8bl6m7bT044jNtt39yOUGny9VVfL6RX2Lm8/vJUehPkMra3M?= =?us-ascii?Q?tzce+Hl7gzG8jxSaC4EA76s4r5/GDO2GhLuww0vpmK6Y0sMJ+KRMWzxBLUFY?= =?us-ascii?Q?lMKFlCm2Xd6kNk1578eKIcfh8u9oLlKB5ciTtxTAjX0mW9AR18zbxFH5X0O5?= =?us-ascii?Q?vT83UJ/7qLSyHKL1DTeVLIXCBgHntoDs9QsMFGzwW2vF?= X-Forefront-PRVS: 03355EE97E X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BY2PR0301MB0743; 23:G7Nz2h/OKiiZ/XAjDfgOScNMLBDoRYLxCuPLbYD?= =?us-ascii?Q?wunwM4giXcYMXpOY/wOiuZfB0V8cKg9R0qd73JTz3o93SXpslJcLsY70P0W2?= =?us-ascii?Q?K8Iesx+bo1HaDdW/2gu+NEVyefMNOk8R2/Yk+ApS/k4Pdk5yNIvzYEr9mUXT?= =?us-ascii?Q?4bilvEMVsbaOfa5U5/emQBm/1UrrwotFqwR+JDeXIxXh2Q2edIlvGT2CQFKD?= =?us-ascii?Q?+BcA8oKP0KL3sgjQxTIpnFVgVmZelCdfn8GBEioOA2NTOwE5xN+Etx0NZAC4?= =?us-ascii?Q?1QARnV/CSkj7J2MwgkB3nFxjs5m8sSz1JUzi72/DqWfgWnvR1d4xnPuMdkqn?= =?us-ascii?Q?Cnbv7AWvJsAuH8oKqckI+L991m1HzeCPUFNpcG63mP+Fnyrr7ni9QQki/lO0?= =?us-ascii?Q?1YFuPUD6WfxUPX09HV7AlNt7ALq0hIhRSUmqZaawLZl4yDMWKzBstVWmkxv/?= =?us-ascii?Q?7/BS7oh/Kg/zS/IJFkS+9DR9SUDL1yO2JhXASr5cwmZulzhZoDF4ksnAGEeb?= =?us-ascii?Q?S05UidUkZSD6TntKyj4sLNgw8I9fB3RF2l7QdFDgBpYv0R3dHmpgCU0Zrg43?= =?us-ascii?Q?Ke6XS5H6jGzUVb6pS22NspgciZaMUrGaOXQKTiS0MTmMQrZQ+EvtUjpmjrCH?= =?us-ascii?Q?OC71G4/RRpx888YqmYt0qWxVIrT1lLWEGhaNFj+V5kh0VmCrPZuuKsW3vYN6?= =?us-ascii?Q?OyrdUdwoKK1+jUu5DStiD61TqqHj7XTZf/K/z1IFL8TP0sPKlmsv5J/mU3AU?= =?us-ascii?Q?uBnqwiG8trHa2G4Hb5aESZ46xm07WtwqMMPUc+BipPvvf43ShthW4BZouI1K?= =?us-ascii?Q?F5uaqtBS47yzet+gL1oJojioskzTDkehbU18i7AAhcZYMnGt7xMl1Dd12vzr?= =?us-ascii?Q?TU+NMVnMvFVpkarjfXdDgfP4QDbhInc5uBz1ldaJQirir7pX8a8S9+mww5MI?= =?us-ascii?Q?8BkJVVpOZ71DN4+LJ5+nuAC5JFqq+4JSnxPovbOp4MkyJAz9WMh5yuR7rUHj?= =?us-ascii?Q?d3oOxWrtqRuFM3FTKi679sp8SaKPjcuc/WZpYAkc3Mb/U7Qo1D1cbbB5VZxA?= =?us-ascii?Q?AiA2rSCWyCJHjTtWfrGmx7iAfSkKWQKiXPptzR9l6KiuO2LMvABDX8/Bn86K?= =?us-ascii?Q?1D3NfLmelVECOufCgBpLI4rmDJauR/MBwGrbK9dE8LEUsa8qX1gipAndnm36?= =?us-ascii?Q?MixtlYxhbSMmGhDJRcm0Ts5gYLNb/UUEMZKsgmQU+7QMTrskB7H6jQyOx9Se?= =?us-ascii?Q?744c1b/hEHCjD//ijrTBTT+yooLb2beiZK8Tic5sBGNA++R+2x3CLSOyFVHM?= =?us-ascii?Q?FvY4cX9oBE+nGF6urpxY6KGbapuGbN61WWuGpCHnruW0jMRiLaGsFBS7Or6X?= =?us-ascii?Q?yoPyV6djaLtdkMVBtyPOHCN8f8ktluTzlJRvoRe3EYvfDVdvd?= X-Microsoft-Exchange-Diagnostics: 1; BY2PR0301MB0743; 6:4SX7C7HX0+7VjFJUt05++8IdkQm5IWd9xHqTCPOa5YQL6tC72NH7cx048exlgJ1e8aPFSA222SwJHiyPaFxbJMqkPfhwEA8btO22krZW4AFDNHonC0N0f5SFKgIDBW18ATJ32JkPM1Q0DCxz/JVzEtSfAu9ecV1zgugb4NPekJa32h2eW0rTjLBVBa5424hbfoJrMdy9vh5qgq2kf6s/Gf0pQJJ4EOxsE0Pi0k6pXVw+JSIDfIyZncbNo7D0Em2f2WNod4Fv7a0yhBH4iMBhhMFX3dn5mK43Kgqsin/yq1sIgUJVr8N5sKXvcWcqKV7G6ZnowvotYRbNnmpGyJtZ6ejywQRvA1s8Okp5wHMthd220uHndmpmIqEOVONYrGVscwXUT4AEWS8jHQLQF+erUJJaECTukZoCaZrLMU+Sqdxhqr+PwCYyvOYpwoqLDi3G7EqglloepeYbu3+nH6/FXi9BzjPumCcCPV4p4GQ0ejR2gya6dKpEHPa2/jYZj5CjaYyWibwhwUVRNgBfqoGongOr8iG3fp9bY6hk2tE3SfM= X-Microsoft-Exchange-Diagnostics: 1; BY2PR0301MB0743; 5:9O1GM1YSknxnxdhsF9/EwZzCPLFTY0fI2Nyo6x+2q1GulUuUVKzt1mSigzOtnd6GVA6u/kmZ5BKlAzipGu+kKoyTk7O8GUzmxCcEr3fMDrw7nbQaPbjO9tU2rIEZyieycutj2jD4Hk9UtpQvHcp3j+UOqL+kOt8r4zhcPoAlgu2njxEpiGjicbcG4Xe6Ye15Cqwi/JJL44uJEXF5StkmS88jdtfAEhcdJPdg59qn6qPF0OcJtsJZ2P5AZgpbzhDUIZtPV8DzCqEIZgOj2sdfbxxSp000blFeHvelvL5e+XYASt1viXUbKgJa5ualViBfDse5D3OglFId7m+CITHz3ciV8ICoIiehlH20kvugUTQJO+hsKLMXCebwK+ASOug8/SLOqDBdFNWEr/g262vMQo+ryCQ03ba/PVpmYEeZaY2BPP3RbtieTA6dTDqZVMvUTNXS1V9YZRfGsk+QC8qmtde8NcO/pQmX/PPCsCqrJ6990KVHi06vIP4Ng9RHkOqg; 24:932RW/7TjqG80W/hwbzx5vfx26qPbwFWHhgfcNNph3QAxSQNCStM6jvIvGIGrzThFlJR24S7Swjsb+GaSo72+aTgYLyqjPzax/BELifHxmc= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; BY2PR0301MB0743; 7:PTwCoUS1GHaOR8MMlGhYVsf1h2hsTe8LG0ZrHKMr48Xo+kmaAu2o+Dxs4ngm+s7C9SFgNHnvKxMmc5jkfxXCZw3YLGxLUwLV0/52BzN3SyES05NFV2XGLf4rsFmWEYB675ss4w3Ee2rN9XZVa54th639bxqUNRO+9XpnAzsAz7O8BmMzJWh3C6Z1qdBuOEBYNP6h9EzcBGebkuoLp/Wv5ZCbeHOBsIPI2+YNZOIIZEfXsvhqh3nyQ6kJHMGu9ez2WamfoiNlJh64IzrHRvVlbuZZz398A+xGlYTBCBgQn+qN6mJXicL6oIDO9QMRrjshyAn1TpkrmAEiW+zIQ9MPag== X-OriginatorOrg: ca.weatherford.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jun 2017 16:51:15.3227 (UTC) X-MS-Exchange-CrossTenant-Id: dd63fb60-07f6-4d96-8d40-ebeca61a524e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=dd63fb60-07f6-4d96-8d40-ebeca61a524e; Ip=[23.103.226.20]; Helo=[032-smtp-out.weatherford.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR0301MB0743 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jun 2017 16:51:18 -0000 Thanks Allan for the suggestions. I tried gstat -d but deletes (d/s) doesn= 't seem to be it as it stays at 0 despite vfs.zfs.trim.enabled=3D1. This is most likely due to the "layering" I use as, for historical reasons,= I have GEOM ELI set up to essentially emulate 4k sectors regardless of the= underlying media. I do my own alignment and partition sizing as well as h= ave the ZFS record size set to 8k for Postgres. In gstat, the SSDs %busy is 90-100% on startup after reboot. Once the perf= ormance degradation hits (<24 hours later), I'm seeing %busy at ~10%. #!/bin/sh psql --username=3Dtest --password=3Dsupersecret -h /db -d test << EOL \timing on select count(*) from test; \q EOL Sample run of above script after reboot (before degradation hits) (Samsung = 850 Pros in ZFS mirror): Timing is on. count ---------- 21568508 (1 row) Time: 57029.262 ms Sample run of above script after degradation (Samsung 850 Pros in ZFS mirro= r): Timing is on. count ---------- 21568508 (1 row) Time: 583595.239 ms (Uptime ~1 day in this particular case.) Any other suggestions? Regards, A -----Original Message----- From: owner-freebsd-hackers@freebsd.org [mailto:owner-freebsd-hackers@freeb= sd.org] On Behalf Of Allan Jude Sent: Saturday, June 10, 2017 9:40 PM To: freebsd-hackers@freebsd.org Subject: [EXTERNAL] Re: FreeBSD10 Stable + ZFS + PostgreSQL + SSD performan= ce drop < 24 hours On 06/10/2017 12:36, Slawa Olhovchenkov wrote: > On Sat, Jun 10, 2017 at 04:25:59PM +0000, Caza, Aaron wrote: > >> Gents, >> >> I'm experiencing an issue where iterating over a PostgreSQL table of ~21= .5 million rows (select count(*)) goes from ~35 seconds to ~635 seconds on = Intel 540 SSDs. This is using a FreeBSD 10 amd64 stable kernel back from J= an 2017. SSDs are basically 2 drives in a ZFS mirrored zpool. I'm using P= ostgreSQL 9.5.7. >> >> I've tried: >> >> * Using the FreeBSD10 amd64 stable kernel snapshot of May 25, 2017= . >> >> * Tested on half a dozen machines with different models of SSDs: >> >> o Intel 510s (120GB) in ZFS mirrored pair >> >> o Intel 520s (120GB) in ZFS mirrored pair >> >> o Intel 540s (120GB) in ZFS mirrored pair >> >> o Samsung 850 Pros (256GB) in ZFS mirrored pair >> >> * Using bonnie++ to remove Postgres from the equation and performa= nce does indeed drop. >> >> * Rebooting server and immediately re-running test and performance= is back to original. >> >> * Tried using Karl Denninger's patch from PR187594 (which took som= e work to find a kernel that the FreeBSD10 patch would both apply and compi= le cleanly against). >> >> * Tried disabling ZFS lz4 compression. >> >> * Ran the same test on a FreeBSD9.0 amd64 system using PostgreSQL = 9.1.3 with 2 Intel 520s in ZFS mirrored pair. System had 165 days uptime a= nd test took ~80 seconds after which I rebooted and re-ran test and was sti= ll at ~80 seconds (older processor and memory in this system). >> >> I realize that there's a whole lot of info I'm not including (dmesg, zfs= -stats -a, gstat, et cetera): I'm hoping some enlightened individual will b= e able to point me to a solution with only the above to go on. > > Just a random guess: can you try r307264 (I am mean regression in > r307266)? > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " > This sounds a bit like an issue I investigated for a customer a few months = ago. Look at gstat -d (includes DELETE operations like TRIM) If you see a lot of that happening, but try: vfs.zfs.trim.enabled=3D0 in /b= oot/loader.conf and see if your issues go away. the FreeBSD TRIM code for ZFS basicallys waits until the sector has been fr= ee for a while (to avoid doing a TRIM on a block we'll immediately reuse), = so your benchmark will run file for a little while, then suddenly the TRIM = will kick in. For postgres, fio, bonnie++ etc, make sure the ZFS dataset you are storing = the data on / benchmarking has a recordsize that matches the workload. If you are doing a write-only benchmark, and you see lots of reads in gstat= , you know you are having to do read/modify/write's, and that is why your p= erformance is so bad. -- Allan Jude _______________________________________________ freebsd-hackers@freebsd.org mailing list https://lists.freebsd.org/mailman/= listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" This message may contain confidential and privileged information. If it has= been sent to you in error, please reply to advise the sender of the error = and then immediately delete it. If you are not the intended recipient, do n= ot read, copy, disclose or otherwise use this message. The sender disclaims= any liability for such unauthorized use. PLEASE NOTE that all incoming e-m= ails sent to Weatherford e-mail accounts will be archived and may be scanne= d by us and/or by external service providers to detect and prevent threats = to our systems, investigate illegal or inappropriate behavior, and/or elimi= nate unsolicited promotional e-mails (spam). This process could result in d= eletion of a legitimate e-mail before it is read by its intended recipient = at our organization. Moreover, based on the scanning results, the full text= of e-mails and attachments may be made available to Weatherford security a= nd other personnel for review and appropriate action. If you have any conce= rns about this process, please contact us at dataprivacy@weatherford.com.