ommits-src-main@FreeBSD.org> List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 References: <6a70e844.251cb.6995429f@gitrepo.freebsd.org> <6870a20e-5366-486e-9405-44ad57a89bf4@FreeBSD.org> In-Reply-To: <6870a20e-5366-486e-9405-44ad57a89bf4@FreeBSD.org> From: Alan Somers Date: Mon, 10 Aug 2026 08:58:04 -0600 X-Gm-Features: AUfX_myjSFe910xMi8xvuJ_xM7Eie1yEaaJBV3z7EeYYqmuNNKkRXSs-G6ivnZw Message-ID: Subject: Re: git: 4bd01d6ae016 - main - fts: refactor to use fd-relative operations internally To: Kyle Evans Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org, Jitendra Bhati , Mark Johnston Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4hJdDF6y0jz3fxL X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Spamd-Bar: ---- On Mon, Aug 10, 2026 at 8:26=E2=80=AFAM Kyle Evans wro= te: > > On 8/7/26 10:30, Mark Johnston wrote: > > On Thu, Aug 06, 2026 at 04:52:55PM -0600, Alan Somers wrote: > >> On Tue, Aug 4, 2026 at 4:06=E2=80=AFPM Mark Johnston wrote: > >>> On Tue, Aug 04, 2026 at 01:33:40PM -0600, Alan Somers wrote: > >>>> On Tue, Aug 4, 2026 at 1:00=E2=80=AFPM Alan Somers wrote: > >>>>> > >>>>> On Tue, Aug 4, 2026 at 12:58=E2=80=AFPM Mark Johnston wrote: > >>>>>> > >>>>>> On Mon, Aug 03, 2026 at 07:13:08PM +0000, Alan Somers wrote: > >>>>>>> The branch main has been updated by asomers: > >>>>>>> > >>>>>>> URL: https://cgit.FreeBSD.org/src/commit/?id=3D4bd01d6ae01632501b= 63438b8d9a401db9744a78 > >>>>>>> > >>>>>>> commit 4bd01d6ae01632501b63438b8d9a401db9744a78 > >>>>>>> Author: Jitendra Bhati > >>>>>>> AuthorDate: 2026-06-12 17:07:55 +0000 > >>>>>>> Commit: Alan Somers > >>>>>>> CommitDate: 2026-08-03 19:12:28 +0000 > >>>>>>> > >>>>>>> fts: refactor to use fd-relative operations internally > >>>>>>> > >>>>>>> Replace all _open() calls with _openat() in __fts_open(), ft= s_read(), > >>>>>>> and fts_children(). Replace statfs() with _fstatfs(). > >>>>>>> > >>>>>>> Add fts_dirfd to struct _ftsent, set to the file descriptor = of the > >>>>>>> parent directory. Callers can use openat(ent->fts_dirfd, ent= ->fts_name, > >>>>>>> ...) to access files safely without relying on fts_accpath, = which > >>>>>>> enables programs in capability mode to open the files descri= bed by > >>>>>>> _ftsent. > >>>>>>> > >>>>>>> This is a preparatory change for fts_openat() which will all= ow callers > >>>>>>> to provide a pre-opened directory fd, enabling fts(3) traver= sal inside > >>>>>>> Capsicum capability mode. > >>>>>>> > >>>>>>> Mirror all fts_open() changes to fts_open_b(). > >>>>>>> > >>>>>>> As a result of expanding _ftsend, publish new ELF symbol ver= sions for > >>>>>>> fts_openat and related functions. > >>>>>>> > >>>>>>> Sponsored by: Google LLC (GSoC 2026) > >>>>>>> Reviewed by: asomers > >>>>>>> Pull Request: https://github.com/freebsd/freebsd-src/pull/= 2303 > >>>>>> > >>>>>> Quite a few regression tests seem to be failing after this commit,= e.g., > >>>>>> some of the makefs tests: https://ci.freebsd.org/job/FreeBSD-main-= amd64-test/29201/ > >>>>> > >>>>> I'm testing a fix now. I should have something ready for you soon. > >> > >> The fix is committed now, as 74bd6fb1eb02 . Sorry for the trouble. > > > > Thanks, but I think there are still some problems. I'm still seeing > > vnode exhaustion while running the test suite. > > > > First, fts_build() appears to dup() the directory entry into every > > single(!) child FTS entry. That's slow and means that we'll potentiall= y > > acquire a large number of fds when traversing a wide directory, enough > > to hit per-process fd limits. Second, the return value from dup() is > > not checked, and it's not clear to me whether that's handled elsewhere. > > Finally, I'm fairly sure these dup()ed fds are leaked in various places= ; > > for instance, isn't fts_lfree() potentially leaking the dirfd of each > > entry it frees? > > > > The first and last issues are probably straightforward to fix, but I'm > > not sure about the second one. I don't think FTS can reasonably consum= e > > an fd for every entry in a directory, that's going to cause a lot of > > problems. > > > > Hi, > > Can we go ahead and back this out until the outstanding issues are > resolved, please? > > Thanks, > > Kyle Evans The "too many open file descriptors" problem should be fixed by https://github.com/freebsd/freebsd-src/pull/2360 , and I'm doing the final pre-commit testing of that PR right now.