From owner-freebsd-fs@freebsd.org Tue Aug 11 03:46:03 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 12449378988 for ; Tue, 11 Aug 2020 03:46:03 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [209.51.186.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BQdzG4xNpz3xhr for ; Tue, 11 Aug 2020 03:46:02 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id 0994F20FEB; Tue, 11 Aug 2020 03:45:56 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net 0994F20FEB From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> <327f4b10-9727-331e-2dc9-641dad96dd2a@freebsd.org> Autocrypt: addr=allanjude@freebsd.org; prefer-encrypt=mutual; keydata= xsFNBFVwZcYBEADwrZDH0xe0ZVjc9ORCc6PcBLwS/RTXA6NkvpD6ea02pZ8lPOVgteuuugFc D34LdDbiWr+479vfrKBh+Y38GL0oZ0/13j10tIlDMHSa5BU0y6ACtnhupFvVlQ57+XaJAb/q 7qkfSiuxVwQ3FY3PL3cl1RrIP5eGHLA9hu4eVbu+FOX/q/XVKz49HaeIaxzo2Q54572VzIo6 C28McX9m65UL5fXMUGJDDLCItLmehZlHsQQ+uBxvODLFpVV2lUgDR/0rDa0B9zHZX8jY8qQ7 ZdCSy7CwClXI054CkXZCaBzgxYh/CotdI8ezmaw7NLs5vWNTxaDEFXaFMQtMVhvqQBpHkfOD 7rjjOmFw00nJL4FuPE5Yut0CPyx8vLjVmNJSt/Y8WxxmhutsqJYFgYfWl/vaWkrFLur/Zcmz IklwLw35HLsCZytCN5A3rGKdRbQjD6QPXOTJu0JPrJF6t2xFkWAT7oxnSV0ELhl2g+JfMMz2 Z1PDmS3NRnyEdqEm7NoRGXJJ7bgxDbN+9SXTyOletqGNXj/bSrBvhvZ0RQrzdHAPwQUfVSU2 qBhQEi2apSZstgVNMan0GUPqCdbE2zpysg+zT7Yhvf9EUQbzPL4LpdK1llT9fZbrdMzEXvEF oSvwJFdV3sqKmZc7b+E3PuxK6GTsKqaukd/3Cj8aLHG1T1im1QARAQABzSJBbGxhbiBKdWRl IDxhbGxhbmp1ZGVAZnJlZWJzZC5vcmc+wsF/BBMBAgApBQJVcGXGAhsjBQkSzAMABwsJCAcD AgEGFQgCCQoLBBYCAwECHgECF4AACgkQGZU1PhKYC34Muw/+JOKpSfhhysWFYiRXynGRDe07 Z6pVsn7DzrPUMRNZfHu8Uujmmy3p2nx9FelIY9yjd2UKHhug+whM54MiIFs90eCRVa4XEsPR 4FFAm0DAWrrb7qhZFcE/GhHdRWpZ341WAElWf6Puj2devtRjfYbikvj5+1V1QmDbju7cEw5D mEET44pTuD2VMRJpu2yZZzkM0i+wKFuPxlhqreufA1VNkZXI/rIfkYWK+nkXd9Efw3YdCyCQ zUgTUCb88ttSqcyhik/li1CDbXBpkzDCKI6I/8fAb7jjOC9LAtrZJrdgONywcVFoyK9ZN7EN AVA+xvYCmuYhR/3zHWH1g4hAm1v1+gIsufhajhfo8/wY1SetlzPaYkSkVQLqD8T6zZyhf+AN bC7ci44UsiKGAplB3phAXrtSPUEqM86kbnHg3fSx37kWKUiYNOnx4AC2VXvEiKsOBlpyt3dw WQbOtOYM+vkfbBwDtoGOOPYAKxc4LOIt9r+J8aD+gTooi9Eo5tvphATf9WkCpl9+aaGbSixB tUpvQMRnSMqTqq4Z7DeiG6VMRQIjsXDSLJEUqcfhnLFo0Ko/RiaHd5xyAQ4DhQ9QpkyQjjNf /3f/dYG7JAtoD30txaQ5V8uHrz210/77DRRX+HJjEj6xCxWUGvQgvEZf5XXyxeePvqZ+zQyT DX61bYw6w6bOwU0EVXBlxgEQAMy7YVnCCLN4oAOBVLZ5nUbVPvpUhsdA94/0/P+uqCIh28Cz ar56OCX0X19N/nAWecxL4H32zFbIRyDB2V/MEh4p9Qvyu/j4i1r3Ex5GhOT2hnit43Ng46z5 29Es4TijrHJP4/l/rB2VOqMKBS7Cq8zk1cWqaI9XZ59imxDNjtLLPPM+zQ1yE3OAMb475QwN UgWxTMw8rkA7CEaqeIn4sqpTSD5C7kT1Bh26+rbgJDZ77D6Uv1LaCZZOaW52okW3bFbdozV8 yM2u+xz2Qs8bHz67p+s+BlygryiOyYytpkiK6Iy4N7FTolyj5EIwCuqzfk0SaRHeOKX2ZRjC qatkgoD/t13PNT38V9tw3qZVOJDS0W6WM8VSg+F+bkM9LgJ8CmKV+Hj0k3pfGfYPOZJ/v18i +SmZmL/Uw2RghnwDWGAsPCKu4uZR777iw7n9Io6Vfxndw2dcS0e9klvFYoaGS6H2F13Asygr WBzFNGFQscN4mUW+ZYBzpTOcHkdT7w8WS55BmXYLna+dYer9/HaAuUrONjujukN4SPS1fMJ2 /CS/idAUKyyVVX5vozoNK2JVC1h1zUAVsdnmhEzNPsvBoqcVNfyqBFROEVLIPwq+lQMGNVjH ekLTKRWf59MEhUC2ztjSKkGmwdg73d6xSXMuq45EgIJV2wPvOgWQonoHH/kxABEBAAHCwWUE GAECAA8FAlVwZcYCGwwFCRLMAwAACgkQGZU1PhKYC34w5A//YViBtZyDV5O+SJT9FFO3lb9x Zdxf0trA3ooCt7gdBkdnBM6T5EmjgVZ3KYYyFfwXZVkteuCCycMF/zVw5eE9FL1+zz9gg663 nY9q2F77TZTKXVWOLlOV2bY+xaK94U4ytogOGhh9b4UnQ/Ct3+6aviCF78Go608BXbmF/GVT 7uhddemk7ItxM1gE5Hscx3saxGKlayaOsdPKeGTVJCDEtHDuOc7/+jGh5Zxpk/Hpi+DUt1ot 8e6hPYLIQa4uVx4f1xxxV858PQ7QysSLr9pTV7FAQ18JclCaMc7JWIa3homZQL/MNKOfST0S 2e+msuRwQo7AnnfFKBUtb02KwpA4GhWryhkjUh/kbVc1wmGxaU3DgXYQ5GV5+Zf4kk/wqr/7 KG0dkTz6NLCVLyDlmAzuFhf66DJ3zzz4yIo3pbDYi3HB/BwJXVSKB3Ko0oUo+6/qMrOIS02L s++QE/z7K12CCcs7WwOjfCYHK7VtE0Sr/PfybBdTbuDncOuAyAIeIKxdI2nmQHzl035hhvQX s4CSghsP319jAOQiIolCeSbTMD4QWMK8RL/Pe1FI1jC3Nw9s+jq8Dudtbcj2UwAP/STUEbJ9 5rznzuuhPjE0e++EU/RpWmcaIMK/z1zZDMN+ce2v1qzgV936ZhJ3iaVzyqbEE81gDxg3P+IM kiYh4ZtPB4Q= Subject: Re: ZSTD Project Weekly Status Update Message-ID: Date: Mon, 10 Aug 2020 23:45:49 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="YS1XubqmixOoUKSoGiUVjrTm02zNNWNzk" X-Rspamd-Queue-Id: 4BQdzG4xNpz3xhr X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; ASN(0.00)[asn:6939, ipnet:209.51.160.0/19, country:US] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2020 03:46:03 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --YS1XubqmixOoUKSoGiUVjrTm02zNNWNzk Content-Type: multipart/mixed; boundary="JZRdp68DXAPxHE95DlB0LrH2lLIWkdpt3"; protected-headers="v1" From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer Message-ID: Subject: Re: ZSTD Project Weekly Status Update References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> <327f4b10-9727-331e-2dc9-641dad96dd2a@freebsd.org> In-Reply-To: --JZRdp68DXAPxHE95DlB0LrH2lLIWkdpt3 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable This is the eighth weekly status report on the project to complete the integration of ZSTD compression into OpenZFS. https://github.com/openzfs/zfs/pull/10692 - I created some new tests around the L2ARC to facilitate testing of ZSTD + L2ARC. These tests found an issue (even with just lz4 compression) where if the compressed_arc feature is disabled, the wrong size is used when calculating the checksum of the buffer read back from the L2ARC, resulting in a silent checksum failure. After the block from the L2ARC fails to checksum, it is re-read from the main pool. https://github.com/openzfs/zfs/pull/10693 - I have created a patch to fix the issue between L2ARC and compressed_arc. https://github.com/allanjude/zfs/commit/1f565ef0c6bd2e785fb3777c111184bb4= bc551c4 - A followup to the rewritten version of the ZSTD feature activation code. The handling of zfs_prop_set_special() was not actually setting the property, so we return -1 so that the normal property setting routine will be followed, in addition to the special handling. https://github.com/allanjude/zfs/commit/8eac845a221952b3c9c52b4caf9be4bdf= 401e2b9 - Fixed an issue where if compression failed (this can be triggered by "early abort", where the data is uncompressable and wont fit in the output buffer that is 12.5% smaller than the input), it would skip the encryption code block, which could result in data being written to the L2ARC uncompressed and unencrypted. Based on the above, I am considering that we might want to calculate the checksum of the block after we re-transform it, and make sure it matches the checksum in the blockpointer, if it does not, we just skip writing to the L2ARC as if the block was ineligible for one of the normal reasons. This would ensure we don't end up reading from the L2ARC only to re-read from the main pool because the block did not survive the trip.= That leaves just the future proofing bits left (L2ARC, nop-write, etc when newer ZSTD does not recompress the block in the same way), but that specific bit doesn't need to block merging ZSTD support. This project is sponsored by the FreeBSD Foundation. On 2020-08-05 22:49, Allan Jude wrote: > This is the seventh weekly status report on the project to integrate > ZSTD into OpenZFS. >=20 > The compatibility related changes I created last week were refined and > marged into the mainline branch. >=20 > Thanks to Brian Behlendorf for reviewing my proposed change for the zst= d > feature flag activation, and pointing out a better approach. I have > reworked the patch based on his suggestion and prototype: >=20 > https://github.com/allanjude/zfs/commit/2508dafcec0a05d61afc5fbd5da356e= 201afbe97 > - Activate the per-dataset ZSTD feature flag as soon as the property is= > set to ZSTD. Before, simply doing `zfs set compression=3Dzstd dataset` > would not activate the feature flag. The feature flag would be activate= d > when the first block that used ZSTD compression was written (see > dsl_dataset_block_born()). This meant that if you set the property, > exported the pool, the pool would import on systems with older versions= > of ZFS that did not support ZSTD, but would crash their userspace tools= , > because the property value was out of bounds. >=20 >=20 > https://github.com/allanjude/zfs/commit/b8bec3fd2a8feb3a4de572eb15515d3= 764f92a35 > - I created a test that ensures that the feature flag is activated by > `zfs set compression=3Dzstd` and also ensures that the feature flag > reverts to the 'enabled' state once the last dataset using zstd is > destroyed. >=20 >=20 > The next step is ensuring that ZSTD compression inter-operates properly= > with the L2ARC and Encryption etc. >=20 > I've also been discussing ideas with Brian about future-proofing, to > handle the case where a newer version of ZSTD might compression the sam= e > input differently (better ratio), and how that would impact L2ARC, > nop-write, etc. One idea (originally from Pawel Dawidek) is to do > something similar to what encryption does, and split the checksum field= =2E > Using half to checksum the original data, and half the compressed > version. This would allow ZFS to detect when the same content compresse= d > differently (combined with the ZSTD version header in the compressed > data), giving better compatibility as we upgrade ZSTD. >=20 >=20 > This project is sponsored by the FreeBSD Foundation. >=20 >=20 >=20 --=20 Allan Jude --JZRdp68DXAPxHE95DlB0LrH2lLIWkdpt3-- --YS1XubqmixOoUKSoGiUVjrTm02zNNWNzk Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIbBAEBAgAGBQJfMhRzAAoJEBmVNT4SmAt+TkIP+Inmj5M3TkOm4G1Rc2rgEEDz 0BXygiUCbrFHn3aeadgeImYIGmZfATH4USFGpBt9FcyETeUCp/9fTlHPnd2PIttY Y6TlVi9jqa8Gnb7M/HG3egArF3oYODvc/oJk+AurRNX+45mVWSkmfddc7YRwgZta /AQVVgAyYSukgASD8MgRyQV1FNe0OhFJWk3o5Xqc6qUhSpO3K3PfWgzkzBgVtLUa XD6T5EgxQo632dbBG3V/S8a6nhKht21rV35FyrUjxLWCWh5bibAJyY9CZNXdi+F5 sdEyoBuDWB1KwUJDR+OA4RO93cL4l6Ai3Km4UbeA12HTp7jmFwah/z32M5zPy0xg L3mQBVOCfn+IuFb2aNS/U/JAu9ydy/vnY0oX9YKD1kLoYEkRznnSlms/+EBryMUm TS6HRzOHoHNMjCVbuGfc0CNt1CVzEywJvqX2Mrj/HvTlsx89hfbxkdgE6XMJ8C57 U5WBKt3qLWNRTCiz65/CnY/ZB4TJucdBEfe3g3IEmyOSR6fFBMaSOlqe8oXYDmeT BoIcoR51CUf5TZLrBuTC0BBY/2q54M89Ad4wwO5/Se6EPCxngm8707tB/lDdHM9S Tn47VhhCsIq/IFVIae0/TqBWQ43bvqJjWyj30SBvSRlDvBYN8DdVMna85bV3oR+d nQXzKwfuDffLL+v1+lY= =uF1Q -----END PGP SIGNATURE----- --YS1XubqmixOoUKSoGiUVjrTm02zNNWNzk-- From owner-freebsd-fs@freebsd.org Sat Aug 15 10:47:54 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DCFD13B5313 for ; Sat, 15 Aug 2020 10:47:54 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BTH894V1qz3dpw for ; Sat, 15 Aug 2020 10:47:53 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from outgoing.leidinger.net (p508d5858.dip0.t-ipconnect.de [80.141.88.88]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256)) (Client did not present a certificate) by mailgate.Leidinger.net (Postfix) with ESMTPSA id E0C42D42A for ; Sat, 15 Aug 2020 12:47:41 +0200 (CEST) Received: from webmail.leidinger.net (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256) (Client did not present a certificate) by outgoing.leidinger.net (Postfix) with ESMTPS id 56BDE14E1 for ; Sat, 15 Aug 2020 12:47:23 +0200 (CEST) Date: Sat, 15 Aug 2020 12:47:23 +0200 Message-ID: <20200815124723.Horde.XErLXGD767bCee6vNVm5gBT@webmail.leidinger.net> From: Alexander Leidinger To: freebsd-fs@freebsd.org Subject: Re: zfs scrub enable by default References: In-Reply-To: Accept-Language: de,en Content-Type: multipart/signed; boundary="=_7LPHxqkS6ZS0wef0ywo1nZa"; protocol="application/pgp-signature"; micalg=pgp-sha1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 4BTH894V1qz3dpw X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.60 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[leidinger.net:s=outgoing-alex]; NEURAL_HAM_MEDIUM(-0.96)[-0.959]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.002]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[leidinger.net:+]; DMARC_POLICY_ALLOW(-0.50)[leidinger.net,quarantine]; NEURAL_HAM_SHORT(-0.54)[-0.535]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE]; RCVD_TLS_ALL(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[80.141.88.88:received] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Aug 2020 10:47:54 -0000 This message is in MIME format and has been PGP signed. --=_7LPHxqkS6ZS0wef0ywo1nZa Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Quoting Matthew Ahrens via freebsd-fs (from=20=20 Tue,=204 Aug 2020 08:54:41 -0700): > This question was raised elsewhere, and I agree with this reply from Geor= ge > Wilson, my colleague and an expert in the i/o subsystems of ZFS as well a= s > having lots of experience with customers: > > Having scrubs enabled by default is a great idea but at Sun (and Delphix > too) we found that the impact was often too much for some > workloads/customers. This is the challenge we faced and why there was nev= er > a policy to enable it everywhere. We did explore ideas to make the impact > less and to be able to always scrub. Some of those ideas included periodi= c > or continuous scrubs where the impact could be reduced by only scrubbing > portions of the pool at a time, at a reduced i/o rate. At Delphix, we hav= e > investigated similar concepts and one of our interns prototyped one of th= e > ideas.Much has changed since the early scrub days and revisiting some of > the earlier ideas and investigating new ones is probably a good topic for > the community. I do think that just enabling scrub by default without > further enhancements would still be too impactful for some customers but > the concept definitely has merit. Does the zpool man-page come from OpenZFS, or is it a FreeBSD addition? The reason for this question is, that I had checked the content for=20=20 any=20mention of the FreeBSD periodic scripts. There is nothing=20=20 mentioned=20and I think it would be good to mention the periodic=20=20 scrubbing=20functionality of FreeBSD in there (back then when I wrote=20=20 the=20scrubbing part for "periodic daily" I added docs to the=20=20 periodic.conf=20man-page, but failed to mention it in the zpool man-page). If the zpool man-page comes from OpenZFS, what would be a good way to=20=20 mention=20such kind of functionality (I think we should mention that as=20= =20 part=20of periodic daily there is the possibility to scrub and that more=20= =20 docs=20is available in periodic.conf (daily_scrub_zfs_* variables))? Bye, Alexander. --=20 http://www.Leidinger.net=20Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_7LPHxqkS6ZS0wef0ywo1nZa Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAABAgAGBQJfN706AAoJEBINsJsD+NiGyngP/33EDD8FrsnYqs7OYmwOFKRt 9mShRR/zOeAOwYVP+W7bZpnzyOjh2y83vGOX3QgAYVXnqaxm4/p/FKtvPf4aeYXI sgcX/Z/zAQzWTPZLZtqrdW/Lbrgmcsr9l7NhklVHwHLxd5+OXusAEbGF3kycRLvV 5Yt6T+PmBMcvtcUvJ9MDfFEe++xP2cPHYtKZe2sH8auiJ7mCXLZmCLvFbTUWHi6x DKqNw9so6yb5ZADbI2PenwTZ4rCpwPeujtWIwh4Snn9XRjYHoK792W8eemo74Mgn nRCTO+xWIn8XqKPBzKel49FVsJEDd4EPEUayxOQcPomvkyp8vJBM0Xv4xhAOpSjh PKi+0O+P8BaF5dyAZu2k6GQQJq40ScJ8ygWt6qMle9kSLuk+yRCwA3bCofFqfcKn qyqj20z94/AblCLajcxwf9ZAZyOZrkCo7sMvqdHRool59fzb/SMeQgu93ZflcdFV ZpRqErsgsklxlNs4uiYMzrGkhY5KO1cji2+Pn7IxhVXO73biMR/d4V650+9/tRKv ZA2ME9LRVSV6qxDrHQGd/GT2ofJLnvzPgrb4yk4ru0uxfx8Fmrv0mh9VuTG2NYJ/ YiKuxUJZR+/xvIcEQijnPuFnU+q0dbwn2ZctGP2smkI/2KZgWZEiV5GdNgj11+cJ +a2ISR1IcDDJuHl6bjoH =Htzy -----END PGP SIGNATURE----- --=_7LPHxqkS6ZS0wef0ywo1nZa-- From owner-freebsd-fs@freebsd.org Sat Aug 15 11:31:31 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 527CD3B6BFB for ; Sat, 15 Aug 2020 11:31:31 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: from connect.ultra-secure.de (connect.ultra-secure.de [88.198.71.201]) by mx1.freebsd.org (Postfix) with ESMTP id 4BTJ6V01vnz3yQF; Sat, 15 Aug 2020 11:31:29 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: (Haraka outbound); Sat, 15 Aug 2020 13:30:13 +0200 Received-SPF: SoftFail (connect.ultra-secure.de: domain of ultra-secure.de does not designate 217.71.83.52 as permitted sender) receiver=connect.ultra-secure.de; identity=mailfrom; client-ip=217.71.83.52; helo=[192.168.1.238]; envelope-from= Received: from [192.168.1.238] (217-071-083-052.ip-tech.ch [217.71.83.52]) by connect.ultra-secure.de (Haraka/2.6.2-toaster) with ESMTPSA id FFF4E6DB-AD45-44AB-801D-5BD7BD5724A9.1 envelope-from (authenticated bits=0) (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 verify=NO); Sat, 15 Aug 2020 13:30:11 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.15\)) Subject: Re: zfs scrub enable by default From: Rainer Duffner In-Reply-To: Date: Sat, 15 Aug 2020 13:31:18 +0200 Cc: freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <8512AB1C-B14D-4F7A-A6DF-DF93DD215977@ultra-secure.de> References: To: Steve Wills X-Mailer: Apple Mail (2.3445.104.15) X-Haraka-GeoIP: EU, CH, 451km X-Haraka-ASN: 24951 X-Haraka-GeoIP-Received: X-Haraka-ASN: 24951 217.71.80.0/20 X-Haraka-ASN-CYMRU: asn=24951 net=217.71.80.0/20 country=CH assignor=ripencc date=2003-08-07 X-Haraka-FCrDNS: 217-071-083-052.ip-tech.ch X-Haraka-p0f: os="Mac OS X " link_type="DSL" distance=14 total_conn=1 shared_ip=N X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on spamassassin X-Spam-Level: X-Spam-Status: No, score=-2.2 required=5.0 tests=ALL_TRUSTED,BAYES_00, SPF_SOFTFAIL autolearn=no autolearn_force=no version=3.4.1 X-Haraka-Karma: score: 6, good: 14095, bad: 290, connections: 15799, history: 13805, asn_score: 128, asn_connections: 188, asn_good: 145, asn_bad: 17, pass:asn, relaying X-Rspamd-Queue-Id: 4BTJ6V01vnz3yQF X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rainer@ultra-secure.de designates 88.198.71.201 as permitted sender) smtp.mailfrom=rainer@ultra-secure.de X-Spamd-Result: default: False [-2.15 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.94)[-0.941]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+mx]; DMARC_NA(0.00)[ultra-secure.de]; NEURAL_HAM_LONG(-1.02)[-1.016]; NEURAL_HAM_SHORT(-0.49)[-0.491]; RCPT_COUNT_TWO(0.00)[2]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:88.198.0.0/16, country:DE]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Aug 2020 11:31:31 -0000 > Am 03.08.2020 um 18:10 schrieb Steve Wills : >=20 > Hi, >=20 > I wonder why we don't enable zfs periodic scrub by default? >=20 > = https://svnweb.freebsd.org/base/head/usr.sbin/periodic/periodic.conf?view=3D= markup#l162 >=20 > Anyone happen to know? >=20 > Thanks, > Steve >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" I have two HP Gen10 servers with a lot of drives (and an enclosure) that = crashes during the scrub. It may be the firmware, or a bug in the driver. Or a bug in the = hardware. I=E2=80=99ve disabled scrub there and enable it every time I update the = firmware or FreeBSD. Then it crashes again and I disable it, so the co-worker who is on-call = doesn=E2=80=99t get mad at me.=20 Rainer From owner-freebsd-fs@freebsd.org Sat Aug 15 19:30:23 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 55B9B37C04B; Sat, 15 Aug 2020 19:30:23 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BTVl31fmWz4dCX; Sat, 15 Aug 2020 19:30:23 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id 16CCF2CAB6; Sat, 15 Aug 2020 19:30:23 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-lf1-f51.google.com with SMTP id b30so6427224lfj.12; Sat, 15 Aug 2020 12:30:23 -0700 (PDT) X-Gm-Message-State: AOAM533EWLk8oqkQAASaWwoWIZf+o/SsQSbD5HDdUf11LHYtlIugpAtM PE7mNyDUzVeabwLSASWPun1bneWs95yhUhoeKDY= X-Google-Smtp-Source: ABdhPJxOrSmfJbJf/TGBvMgiRyCIsYfty5VK5vXpvADcYLaXpZB+m5OhuURb9jEhJzPb4xWkX0cnYxaOab96l925QUY= X-Received: by 2002:a19:189:: with SMTP id 131mr4003208lfb.128.1597519821598; Sat, 15 Aug 2020 12:30:21 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Matthew Macy Date: Sat, 15 Aug 2020 12:30:09 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: CFT for vendor openzfs - week 5 reminder + memdisk images To: Daniel Nebdal Cc: freebsd-current , freebsd-fs , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Aug 2020 19:30:23 -0000 > > Hi. I have some problems downloading the amd64 image: > > baymax /home/djn > fetch -a > https://people.freebsd.org/~freqlabs/freebsd-openzfs/latest/freebsd-openzfs-amd64-2020081100-memstick.img.xz > freebsd-openzfs-amd64-2020081100-memstick.img. 19% of 655 MB 2179 kBps 04m07s > fetch: freebsd-openzfs-amd64-2020081100-memstick.img.xz appears to be > truncated: 134152192/687158140 bytes > baymax /home/djn > fetch -ar > https://people.freebsd.org/~freqlabs/freebsd-openzfs/latest/freebsd-openzfs-amd64-2020081100-memstick.img.xz > freebsd-openzfs-amd64-2020081100-memstick.img. 20% of 655 MB 882 kBps 09m10s > fetch: freebsd-openzfs-amd64-2020081100-memstick.img.xz appears to be > truncated: 139132928/687158140 bytes > baymax /home/djn > fetch -ar > https://people.freebsd.org/~freqlabs/freebsd-openzfs/latest/freebsd-openzfs-amd64-2020081100-memstick.img.xz > freebsd-openzfs-amd64-2020081100-memstick.img. 20% of 655 MB 647 kBps 12m23s > fetch: freebsd-openzfs-amd64-2020081100-memstick.img.xz appears to be > truncated: 142065664/687158140 bytes > baymax /home/djn > > > It also fails using Firefox on windows on a different machine. (It's > also much slower from that machine, about 200 kB/sec. I have no idea > if that's relevant.) Yes, this appears to have been going on for at least the last week. The FreeBSD infrastructure directly available to developers appears to be unreliable for serving large files. Individuals with accounts on freefall have been able to scp the files. It's possible that we may just end up sharing images more widely by way of releng generated images after commit. I'll see if there's an alternative for the last week of the CFT. Cheers. -M > > -- > Daniel Nebdal