Date: Fri, 7 Aug 2026 18:47:50 -0400 From: Mark Johnston <markj@freebsd.org> To: Mark Millard <marklmi@yahoo.com> Cc: Alan Somers <asomers@freebsd.org>, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org, Jitendra Bhati <bhatijitendra2022@gmail.com> Subject: Re: git: 4bd01d6ae016 - main - fts: refactor to use fd-relative operations internally Message-ID: <anZgls-TALdU7CY_@nuc> In-Reply-To: <8c09ef72-bac2-414e-8670-9fa54a3d06c2@yahoo.com> References: <6a70e844.251cb.6995429f@gitrepo.freebsd.org> <anI2QoB7hj7CSMRq@nuc> <CAOtMX2g_U_mspHXdf3HL%2BFr-kh6XSbxR6JSTmVQ9OHDTqfckYw@mail.gmail.com> <CAOtMX2i9NxYuu=LZwLLP7NErLf4i5A7K=wLC%2B-GywchbwUZacw@mail.gmail.com> <anJiX_r4gThay5bk@nuc> <CAOtMX2j0yN9EF5fbMZ3YH7CgPS-KfNVDtgauDLva0uekkXiYDQ@mail.gmail.com> <anX6LVmcB1H5s_DY@nuc> <8c09ef72-bac2-414e-8670-9fa54a3d06c2@yahoo.com>
index | next in thread | previous in thread | raw e-mail
On Fri, Aug 07, 2026 at 02:05:17PM -0700, Mark Millard wrote: > On 8/7/26 08:30, Mark Johnston wrote: > > On Thu, Aug 06, 2026 at 04:52:55PM -0600, Alan Somers wrote: > >> On Tue, Aug 4, 2026 at 4:06 PM Mark Johnston <markj@freebsd.org> wrote: > >>> On Tue, Aug 04, 2026 at 01:33:40PM -0600, Alan Somers wrote: > >>>> On Tue, Aug 4, 2026 at 1:00 PM Alan Somers <asomers@freebsd.org> wrote: > >>>>> > >>>>> On Tue, Aug 4, 2026 at 12:58 PM Mark Johnston <markj@freebsd.org> wrote: > >>>>>> > >>>>>> On Mon, Aug 03, 2026 at 07:13:08PM +0000, Alan Somers wrote: > >>>>>>> The branch main has been updated by asomers: > >>>>>>> > >>>>>>> URL: https://cgit.FreeBSD.org/src/commit/?id=4bd01d6ae01632501b63438b8d9a401db9744a78 > >>>>>>> > >>>>>>> commit 4bd01d6ae01632501b63438b8d9a401db9744a78 > >>>>>>> Author: Jitendra Bhati <bhatijitendra2022@gmail.com> > >>>>>>> AuthorDate: 2026-06-12 17:07:55 +0000 > >>>>>>> Commit: Alan Somers <asomers@FreeBSD.org> > >>>>>>> CommitDate: 2026-08-03 19:12:28 +0000 > >>>>>>> > >>>>>>> fts: refactor to use fd-relative operations internally > >>>>>>> > >>>>>>> Replace all _open() calls with _openat() in __fts_open(), fts_read(), > >>>>>>> and fts_children(). Replace statfs() with _fstatfs(). > >>>>>>> > >>>>>>> Add fts_dirfd to struct _ftsent, set to the file descriptor of the > >>>>>>> parent directory. Callers can use openat(ent->fts_dirfd, ent->fts_name, > >>>>>>> ...) to access files safely without relying on fts_accpath, which > >>>>>>> enables programs in capability mode to open the files described by > >>>>>>> _ftsent. > >>>>>>> > >>>>>>> This is a preparatory change for fts_openat() which will allow callers > >>>>>>> to provide a pre-opened directory fd, enabling fts(3) traversal inside > >>>>>>> Capsicum capability mode. > >>>>>>> > >>>>>>> Mirror all fts_open() changes to fts_open_b(). > >>>>>>> > >>>>>>> As a result of expanding _ftsend, publish new ELF symbol versions for > >>>>>>> fts_openat and related functions. > >>>>>>> > >>>>>>> Sponsored by: Google LLC (GSoC 2026) > >>>>>>> Reviewed by: asomers > >>>>>>> Pull Request: https://github.com/freebsd/freebsd-src/pull/2303 > >>>>>> > >>>>>> Quite a few regression tests seem to be failing after this commit, e.g., > >>>>>> some of the makefs tests: https://ci.freebsd.org/job/FreeBSD-main-amd64-test/29201/ > >>>>> > >>>>> I'm testing a fix now. I should have something ready for you soon. > >> > >> The fix is committed now, as 74bd6fb1eb02 . Sorry for the trouble. > > > > Thanks, but I think there are still some problems. I'm still seeing > > vnode exhaustion while running the test suite. > > > > First, fts_build() appears to dup() the directory entry into every > > single(!) child FTS entry. That's slow and means that we'll potentially > > acquire a large number of fds when traversing a wide directory, enough > > to hit per-process fd limits. Second, the return value from dup() is > > not checked, and it's not clear to me whether that's handled elsewhere. > > Finally, I'm fairly sure these dup()ed fds are leaked in various places; > > for instance, isn't fts_lfree() potentially leaking the dirfd of each > > entry it frees? > > > > The first and last issues are probably straightforward to fix, but I'm > > not sure about the second one. I don't think FTS can reasonably consume > > an fd for every entry in a directory, that's going to cause a lot of > > problems. > > > > > > A question showed up on Discord's area for FreeBSD ports : > > QUOTE > Ooo could this also be related to why I’m getting “too many open files” > errors from poudriere when building things like go and rust? > END QUOTE Yes that seems plausible. See also https://github.com/freebsd/freebsd-src/pull/2360 > > This was from someone that had progressed to/past 74bd6fb1eb02 in order > to avoid the prior poudriere(-devel) problems they were having. > > > (I'm still synchronized to before the problems started.)home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?anZgls-TALdU7CY_>