From owner-freebsd-hackers@freebsd.org Fri Oct 2 15:47:39 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DF1A443070D for ; Fri, 2 Oct 2020 15:47:39 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670063.outbound.protection.outlook.com [40.107.67.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C2vWt74lkz4Ljf; Fri, 2 Oct 2020 15:47:38 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iYSO+IArlT68K3Cpg9Enj11qo3aCCvBMB+KcB8l2sREwHjHZWQihWAfZwkMh57P9iQMw8tfrVMC9242bNtVFbm0xTNTys2ziRrCYvpbfoI8bnMTri/UsXmlh/5XBBQ2n9hYySPMS6/TPSznaRkN6qqhwTyHXevMAfccX0zeTKuipb8NS1uBTLKDCq7e2zg4xJ2IYrE9Dhu3X/YCU2poNF/m1XCiepm0peuIMYyRGfCe4dz5VfK9i7iv/EFOTZuyv7Hq7dC+NYqQ0FtRLFm/Pgm26h7glrrnmmnr0XAPfhY5HNlf5WsuVqWjFs8oM8EGT7rGxEpwTmjRjeNB7F1LyNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=I72CMT3rdqblECxvIlmi3awhJJJb7DCiXuW4ce1iaRw=; b=HuUmhKbtcbTWvgOnN/RJ5rwp4XJRU0hrvcoPicX1piBQ7wNMKp1zGAqMSUofyAU+OgINPTJFIGX8DhW0Gh2bRRd0RtU7ZgWUM+RS7e6Roe5nUh2mwQgEgseuULiBbhZIdlaW16VyMWMnPdkssNab9O3UxpmXCvMkykN5PaGOv3lu1ZSo8AiqCAhJBg05a075cK2uB5/h7Sa2gV7GyH9m8IsEHyOWYPYjo1mZzFrPPjg5o6nALYPZYHdBu0aEWkVfXiJhWdgpDSfpgAexkJ+54mY3vsvxBYM0qXeF11wv0K/ABaqavElmko/nRfFJ+Ywv/+5HzggAXygZPFbPMcNjtw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=I72CMT3rdqblECxvIlmi3awhJJJb7DCiXuW4ce1iaRw=; b=mBc76zjRadVKPos75S7U+HzhI8oTPPoHXaZVdsDISKvh+zGkElvumgrAFGpAtKZgQnfE7ZxOLIHXk4mRzRjN5wlVkPlJ0AJV1F4lFL+DBt9rBcEMAz6DdHJcHpJirE1YMapXd8uaQVEli+GItLFE3fSVg/3KyzXFYkWlfiGeGOP1oSpfbjbhZ+1+QB5/mOT41ko7UNt+yP74N1paD0ed6IylNtkcaR29p/oV7+SCe4KhoRNfi4X0pjWfpO+j5TQpt0aQWIT85lvdb4DtixdihQcRmbvo2ppb2KyxeKeAUCbtHW9IXDut52+RrycxX2QcN0DbDBha6T2W6dRq8ULWXg== Received: from YTBPR01MB3966.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:24::27) by YT1PR01MB2426.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:3::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3433.38; Fri, 2 Oct 2020 15:47:37 +0000 Received: from YTBPR01MB3966.CANPRD01.PROD.OUTLOOK.COM ([fe80::687f:d85a:a0a3:bd20]) by YTBPR01MB3966.CANPRD01.PROD.OUTLOOK.COM ([fe80::687f:d85a:a0a3:bd20%6]) with mapi id 15.20.3433.039; Fri, 2 Oct 2020 15:47:37 +0000 From: Rick Macklem To: Chris Stephan , Alan Somers CC: FreeBSD Hackers Subject: Re: RFC: copy_file_range(3) Thread-Topic: RFC: copy_file_range(3) Thread-Index: AQHWj2Uep8NVOqCTP0KuS7h/nlgLPqlxrARqgAAEfICAAHAflYAAMOYAgAApba+ACJ95gIAAfXJMgAG51XyABzQMEA== Date: Fri, 2 Oct 2020 15:47:37 +0000 Message-ID: References: , , , , , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3c4a12f2-caf6-4995-bb49-08d866ea7c8e x-ms-traffictypediagnostic: YT1PR01MB2426: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2958; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: pIAajzTJsRuLSbE+5/0Ley6sPje6HdVaybGT3dTH1Xd7eeQBxJWNvca5JoJp+yg2KP4gmJrzrdaMPg/iQzDqeSZ6y5XD0BwPPSD9ap8BzYX9s0nAyrVKNZs7YT+rimexOvqgiSUHr8kEonFO2em64m2g9mMekjSvo9j7BCnESNAjeVW7Jk1M9ZeWyULink3UdSb+NyrOi1hWM34ZaaiJf97buE5n5V4X3PEoX5AL8WkAMlnnlsdSP3BeOwXf5U9lDlo2kN9XCtO4ANEiyzLTPSzk/6XgWz8TVmgDUlNHZwH6+1PewuW4spCytEOJBgsB9t71wyQv8vkyeEmJwswmObmpf415MclkjJTOoGGTvAc9laxVoY/M8XLcELlOLaHwdSLUjYImJyx6lZ+1vrVFwA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YTBPR01MB3966.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(396003)(39860400002)(376002)(366004)(136003)(346002)(91956017)(66556008)(83380400001)(7696005)(66446008)(76116006)(66946007)(5660300002)(52536014)(30864003)(53546011)(6506007)(83080400001)(966005)(110136005)(2906002)(55016002)(9686003)(66476007)(71200400001)(45080400002)(786003)(478600001)(8676002)(8936002)(64756008)(186003)(33656002)(4326008)(86362001)(316002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: z6Vzo15TQEz16jwpIYCBKvsmyne/buLNKqWgTC2xZeBq4Cl3CYcjn8SNMf2+rITO0hw22vLYP3HSx7ROb3DrjKvM6gqHGkqwErOFyJKzOgwCLkFvfYNZWuHjiKDF6vY8lRT7bF3zXdq2v/cl4ze+4g01AzU5h1lZNrRRebHmsmuUGf8kB+xeGFBIZa/eLT/orSrzA+u3fX5ubqpQzZhreelD/zEX22YGgLQeIvVvh8FW4wXe3HamxsqSpalEZuuZ1GuLC7vxxiNFJnJWSbVBKfPVIjySsZKRwArorEncghSNuRDP7SVAQ/nFhW8BW9MYYdwwn9o/wb6ikZMmLDMafSAj30Y7dfPydIw8AwtFH0QKg1wjC7wHngSusNiiTfRHYrn82BdKNby1Th1XghYAU31begkFDADfWl2cXLiswgB8xgg1eSnGg+R/6N8PaELZKNl3Uj8t/MjdAJHI/VQ+txAel6o7zjQf2xgxEHXDeGAc+vE0+YXVpBxdITJCqS1wvc2d08q80HkxPVgi/4P/Co5OVEGq7FQIV+HtdYivVaN7/p6JsMY9deUlx4CGClOFk4RwkqUF9mdD5uwdwranOCo1esNabOaSq4z1NrK85XVuCKNbkM7PcvvL2XcstHLu1HlTP755DGki7PK+w1oaRw6leHRFmEqTYJ0ocDLQuNsEJC71ySYv7hM5lmGS007M/UlS9m+JWNPUW3i1I47B6Q== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YTBPR01MB3966.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 3c4a12f2-caf6-4995-bb49-08d866ea7c8e X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Oct 2020 15:47:37.0494 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: DsZZAXqb2pxT9b9Jqiq6roLD5djkgpyyCdS0qsTNa42/cSurwxSLkm7D6hdczRUFv4uuoz0bKlmb8jOp/X0Rnw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YT1PR01MB2426 X-Rspamd-Queue-Id: 4C2vWt74lkz4Ljf X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=mBc76zjR; dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.63 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-5.97 / 15.00]; NEURAL_HAM_MEDIUM(-1.02)[-1.018]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-0.98)[-0.982]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-0.87)[-0.870]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.63:from]; FREEMAIL_TO(0.00)[live.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-hackers]; RCVD_IN_DNSWL_LOW(-0.10)[40.107.67.63:from] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2020 15:47:39 -0000 [stuff snipped]=0A= Rick Macklem wrote:=0A= >Chris Stephan wrote:=0A= >> New to the list and Late to the discussion. I am thinking increasing the= Len could=0A= >> cause possible degradation of performance when used on slower or legacy= =0A= >> systems. On the other hand just picking a len and cutting it off at a ha= rd max=0A= >> seems crude even with a tunable. Admittedly my naive opinion in this mat= ter=0A= >> ponders, could there be a sysctl tunable that just sets an estimate on t= imeframe=0A= >> instead of size. As you said 100M is roughly a sec on modem hardware. IO= PS are=0A= >> already tracked. The inverse of the avg IOPS for the filesystem in quest= ion could=0A= >> be used against this tunable to derive the estimated size limit of the n= ext=0A= >> read/write. This would allow the max len within the syscall to both hono= r a=0A= >> timeframe before a signal would be handled and maximize efficiency acros= s a=0A= >> large breadth of systems varying in performance. I=92m sure it is more c= omplicated=0A= >> than I suggest... just tossing my 2c in.=0A= >Yes. Using time will work for the generic copy case and I think that's a g= ood idea.=0A= >Then we can leave the file system specific cases up to the implementors.= =0A= >(For NFSv4.2, once you send the RPC to the server, the client has no contr= ol over=0A= > how long it takes to reply. The current sysctl that sets a size is still = about all the=0A= > NFSv4.2 code can do.)=0A= When I looked at a wireshark packet trace, it turned out that the Copy RPC= =0A= happened quickly and it was the subsequent Commit RPC that could take=0A= 1sec or more.=0A= As such, setting a time limit on Copy was not useful.=0A= Testing shows that 16Mbytes/Copy is small enough to keep the Commit RPC=0A= well below 1sec even on really slow server hardware (Pentium 4 with IDE dis= k).=0A= There was also no appreciable performance improvement for Copy sizes=0A= greater than 16Mbytes for the testing I did.=0A= As such, I think the vfs.nfs.maxcopyrange sysctl with a default of 16Mbytes= =0A= is all that can be done for NFSv4.2.=0A= =0A= For local file systems, a patch that detects pending signals is in progress= .=0A= =0A= rick=0A= =0A= Thanks for the suggestion, rick=0A= =0A= Chris=0A= =0A= Sent from FreeBSD=0A= ________________________________=0A= From: owner-freebsd-hackers@freebsd.org = on behalf of Rick Macklem =0A= Sent: Sunday, September 20, 2020 11:28:21 PM=0A= To: Alan Somers =0A= Cc: FreeBSD Hackers =0A= Subject: Re: RFC: copy_file_range(3)=0A= =0A= [I have only indented your most recent comments]=0A= Alan Somers wrote:=0A= On Sun, Sep 20, 2020 at 5:14 PM Rick Macklem > wrote:=0A= Alan Somers wrote:=0A= >On Sun, Sep 20, 2020 at 9:58 AM Rick Macklem >> wrote:=0A= >>Alan Somers wrote:=0A= >>>copy_file_range(2) is nifty, but it has a few sharp edges:=0A= >>>1) Certain file systems don't support it, necessitating a write/read bas= ed=0A= >>>fallback=0A= >>>2) It doesn't handle sparse files as well as SEEK_HOLE/SEEK_DATA=0A= >>>3) It's slightly tricky to both efficiently deal with holes and also=0A= >>>promptly respond to signals=0A= >>>=0A= >>>These problems aren't terribly hard, but it seems to me like most=0A= >>>applications that use copy_file_range would share the exact same=0A= >>>solutions. In particular, I'm thinking about cp(1), dd(1), and=0A= >>>install(8). Those three could benefit from sharing a userland wrapper t= hat=0A= >>>handles the above problems.=0A= >>>=0A= >>>Should we add such a wrapper to libc? If so, what should it be called, = and=0A= >>>should it be public or just private to /usr/src ?=0A= >>There has been a discussion on src-committers which I suggested should=0A= >>be taken to a public mailing list.=0A= >>=0A= >>The basic question is...=0A= >>Whether or not the copy_file_range(2) syscall should be compatible with= =0A= >>the Linux one.=0A= >>When I did the syscall, I tried to make it Linux-compatible, arguing that= =0A= >>Linux is now a de-facto standard.=0A= >>The Linux syscall only works on regular files, which is why Alan's patch = for=0A= >>cp required a "fallback to the old way" for VCHR files like /dev/null.=0A= >>=0A= >>He is considering a wrapper in libc to provide FreeBSD specific semantics= ,=0A= >>which I have no problem with, so long as the naming and man page make=0A= >>it clear that it is not compatible with the Linux syscall.=0A= >>(Personally, I'd prefer a wrapper in libc to making the actual syscall no= n-Linux=0A= >> compatible, but that is just mho.)=0A= >>=0A= >>Hopefully this helps clarify what Alan is asking, rick=0A= >>=0A= >>I don't think the two questions are equivalent. I think that copy_file_r= ange(2) >>ought to work on character devices. Separately, even it does, I = think a userland >>wrapper would still be useful. It would still be able t= o handle sparse files more >>efficiently than the kernel-based vn_generic_c= opy_file_range.=0A= I saw this also stated in your #2 above, but wonder why you think a wrapper= =0A= would handle holes more efficiently.=0A= vn_generic_copy_file_range() does look for holes via SEEK_DATA/SEEK_HOLE=0A= just like a wrapper would and retains them as far as possible. It also look= s=0A= for blocks of all zero bytes for file systems that do not support SEEK_DATA= /=0A= SEEK_HOLE (like NFS versions prior to 4.2) and creates holes for these in= =0A= the output file.=0A= --> The only cases that I am aware of where the holes are not retained are:= =0A= - When the min holesize for the output file is larger than that of the= =0A= input file.=0A= - When the hole straddles the byte range specified for the syscall.=0A= (Or when the hole straddles two copy_file_range(2) syscalls, if you= =0A= prefer.)=0A= =0A= If you are copying the entire file and do not care how long the syscall=0A= takes (which also implies how long it will take for a termination signal=0A= like C to be handled), the most efficient usage is to specify=0A= a "len" argument equal to UINT64_MAX.=0A= --> This will usually copy the whole file in one gulp, although it is not= =0A= guaranteed to copy everything, given the Linux semantics definition= =0A= of it (an NFSv4.2 server can simply choose to copy less, for example= ).=0A= --> This allows the kernel to use whatever block size works efficien= tly=0A= and does not require an allocation of a large userspace buffer= for=0A= the date, nor that the data be copied to/from userspace.=0A= =0A= The problem with doing the whole file in one gulp are:=0A= - A large file can take quite a while and any signal won't be processed unt= il=0A= the gulp is done.=0A= --> If you wrote a program that allocated a 100Gbyte buffer and then=0A= copied a file using read(2)/write(2) with a size of 100Gbytes in a = loop,=0A= you'd end up with the same result.=0A= - As kib@ noted, if the input file never reports EOF (as /dev/zero does),= =0A= then the "one gulp" wouldn't end until storage is exhausted on the=0A= output file(s) device and C wouldn't stop it (since it is one b= ig=0A= syscall).=0A= --> As such, I suggested that, if the syscall is extended to allow VCH= R,=0A= that the "len" argument be clipped at "K Mbytes" for that case t= o=0A= avoid filling the storage device before being able to C ou= t=0A= of it, for this case.=0A= I suppose the answer for #3 is...=0A= - smaller "len" allows for quicker response to signals=0A= but=0A= - smaller "len" results in less efficient use of the syscall.=0A= =0A= Your patch for "cp" seemed fine, but used a small "len" and, as such,=0A= made the use of copy_file_range(2) less efficient.=0A= =0A= All I see the wrapper dong is handling the VCHR case (if the syscall remain= s=0A= as it is now and returns EINVAL to be compatible with Linux) and making=0A= some rather arbitrary choice w.r.t. how big "len" should be.=0A= --> Choosing an appropriate "len" might better be left to the specific use= =0A= case, I think?=0A= =0A= In summary, it's mostly whether VCHR gets handled by the syscall or a=0A= wrapper?=0A= =0A= > 1) In order to quickly respond to a signal, a program must use a modest l= en with > copy_file_range=0A= Does this matter? Or put another way, is a 1-2sec delay in response to C=0A= an issue for "cp".=0A= When kib@ reviewed the syscall, he did not see the delay in signal handling= =0A= a significant problem, noting that it is no different than a large process = core=0A= dumping.=0A= =0A= > 2) If a hole is larger than len, that will cause vn_generic_copy_file_ran= ge to=0A= > truncate the output file to the middle of the hole. Then, in the next in= vocation,=0A= > truncate it again to a larger size.=0A= > 3) The result is a file that is not as sparse as the original.=0A= >=0A= > For example, on UFS:=0A= > $ truncate -s 1g sparsefile=0A= > $ cp sparsefile sparsefile2=0A= > $ du -sh sparsefile*=0A= > 96K sparsefile=0A= > 32M sparsefile2=0A= If you care about maintaining sparseness, a "len" of 100Mbytes or more woul= d=0A= be a reasonable choice. Since "cp" has never maintained sparseness, I didn'= t=0A= suggest such a size when I reviewed your patch for "cp".=0A= --> I/O subsystem performance varies widely, but I think 100Mbytes will lim= it=0A= the delay in signal handling to about 1sec. Isn't that quick enough?= =0A= =0A= > My idea for a userland wrapper would solve this problem by using=0A= > SEEK_HOLE/SEEK_DATA to copy holes in their entirety, and use copy_file_ra= nge for=0A= > everything else with a modest len. Alternatively, we could eliminate the= need for=0A= > the wrapper by enabling copy_file_range for every file system, and making= =0A= > vn_generic_copy_file_range interruptible, so copy_file_range can be calle= d with=0A= > large len without penalizing signal handling performance.=0A= The problem with doing this is it largely defeats the purpose of copy_file_= range().=0A= 1 - What about file systems that do not support SEEK_DATA/SEEK_HOLE.=0A= (All NFS mounts except NFSv4.2 ones against servers that support the= =0A= NFSv4.2 Seek operation are in this category.)=0A= 2 - For NFSv4.2 with servers that support Seek, the copy of an entire file= =0A= can be done via a few (or only one) RPC if you make "len" large and=0A= don't use Seek.=0A= If you combine using Seek with len =3D=3D2Mbytes, then you do a lot mo= re RPCs=0A= with associated overheads and RPC RTT delays. You still avoid moving a= ll=0A= the data across the wire, but you do lose a lot of the performance adv= antage.=0A= =0A= I could have made copy_file_range(2) a lot simpler if the generic code didn= 't=0A= try and maintain holes, but I wanted it to work well for file systems that = did=0A= not support SEEK_DATA/SEEK_HOLE.=0A= =0A= I'd suggest you try patching "cp" to use a 100Mbyte "len" for copy_file_ran= ge()=0A= and test that.=0A= You should fine the sparseness is mostly maintained and that you can = C=0A= out of a large file copy without undue delay.=0A= Then try it over NFS mounts (both v4.2 and v3) for the same large sparse fi= le.=0A= =0A= You can also code up a patched "cp" using SEEK_DATA/SEEK_HOLE and see=0A= how they compare.=0A= =0A= rick=0A= =0A= =0A= -Alan=0A= _______________________________________________=0A= freebsd-hackers@freebsd.org mailing list=0A= https://nam11.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists.f= reebsd.org%2Fmailman%2Flistinfo%2Ffreebsd-hackers&data=3D02%7C01%7C%7C2= 7ea5166cf99415d3bba08d85de6d259%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%= 7C637362593231297450&sdata=3DSfm9MxjQ6MVHgG%2Fw3sghn0hebSFjZo%2FSaUyZ9H= Pyws8%3D&reserved=3D0=0A= To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"= =0A= _______________________________________________=0A= freebsd-hackers@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-hackers=0A= To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"= =0A= _______________________________________________=0A= freebsd-hackers@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-hackers=0A= To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"= =0A=