ld and
reinstall poudriere-devel so that it picks up the new fts API, I start
seeing jexec errors again.

I added some debug fprintfs to jail exec and its inputs look OK:
[00:00:02] Loading MOVED for /var/poudriere/data/.m/CURRENTamd64-default/ref/usr/ports
[00:00:03] Gathering ports metadata
[00:00:03] Warning: (ports-mgmt/poudriere-devel): jexec CURRENTamd64-default jid=11 uid=0 euid=0
[00:00:03] Warning: (misc/freebsd-release-manifests): jexec CURRENTamd64-default jid=11 uid=0 euid=0
[00:00:03] Warning: (misc/freebsd-release-manifests): jexec jail_attach() errno=1
[00:00:03] Warning: (misc/freebsd-release-manifests): jexec: jail_attach(11): Operation not permitted
[00:00:03] Warning: (ports-mgmt/pkg): jexec CURRENTamd64-default jid=11 uid=0 euid=0
[00:00:03] Warning: (misc/freebsd-release-manifests): Error: Error looking up dependencies for misc/freebsd-release-manifests
[00:00:03] Warning: (ports-mgmt/pkg): jexec jail_attach() errno=1
[00:00:03] Warning: (ports-mgmt/portconfig): jexec CURRENTamd64-default jid=11 uid=0 euid=0
[00:00:03] Warning: (ports-mgmt/pkg): jexec: jail_attach(11): Operation not permitted
[00:00:03] Warning: (ports-mgmt/portconfig): jexec jail_attach() errno=1
[00:00:03] Warning: (ports-mgmt/portconfig): jexec: jail_attach(11): Operation not permitted
[00:00:03] Warning: (ports-mgmt/pkg): Error: Error looking up dependencies for ports-mgmt/pkg
[00:00:03] Warning: (ports-mgmt/portconfig): Error: Error looking up dependencies for ports-mgmt/portconfig
[00:00:03] Error: /usr/local/share/poudriere/bulk.sh:gather_port_vars:184:Fatal errors encountered gathering ports metadata

but the call fo jail_attach() fails.

I then added some kernel printf calls and found that chroot_refuse_vdir_fds()
is failing with EPERM.  This is not documented in the jail_attach()
man page, but it will fail if the calling process has any file
descriptors that point to directories, since they would open security
holes.  That made me suspect a file descriptor leak in FTS, possibly a
missing CLOEXEC.

Without any deep analysis, I sprinkled some into the code and it seemed
to fix poudriere-devel:

--3346832139-145962118-1786338523=:5014
Content-Type: TEXT/X-DIFF; CHARSET=us-ascii; NAME=fts.diff
Content-Disposition: ATTACHMENT; FILENAME=fts.diff

diff --git a/lib/libc/gen/fts.c b/lib/libc/gen/fts.c
index 204340e4b87f..396e484cd953 100644
--- a/lib/libc/gen/fts.c
+++ b/lib/libc/gen/fts.c
@@ -444,7 +444,7 @@ fts_read(FTS *sp)
 		p->fts_info = fts_stat(sp, p, 1, -1);
 		if (p->fts_info == FTS_D && !ISSET(FTS_NOCHDIR)) {
 			if ((p->fts_symfd = p->fts_dirfd >= 0 ?
-			    _dup(p->fts_dirfd) :
+			    _fcntl(p->fts_dirfd, F_DUPFD_CLOEXEC) :
 			    _open(".", O_RDONLY | O_CLOEXEC, 0)) < 0) {
 				p->fts_errno = errno;
 				p->fts_info = FTS_ERR;
@@ -539,7 +539,7 @@ next:	tmp = p;
 			if (p->fts_info == FTS_D && !ISSET(FTS_NOCHDIR)) {
 				if ((p->fts_symfd =
 				    p->fts_dirfd >= 0 ?
-				    _dup(p->fts_dirfd) :
+				    _fcntl(p->fts_dirfd, F_DUPFD_CLOEXEC) :
 				    _open(".", O_RDONLY | O_CLOEXEC, 0)) < 0) {
 					p->fts_errno = errno;
 					p->fts_info = FTS_ERR;
@@ -682,7 +682,7 @@ fts_children(FTS *sp, int instr)
 		return (sp->fts_child = fts_build(sp, instr));
 
 	if ((fd = sp->fts_cur->fts_dirfd >= 0 ?
-	    _dup(sp->fts_cur->fts_dirfd) :
+	    _fcntl(sp->fts_cur->fts_dirfd, F_DUPFD_CLOEXEC) :
 	    _open(".", O_RDONLY | O_CLOEXEC, 0)) < 0)
 		return (NULL);
 	sp->fts_child = fts_build(sp, instr);
@@ -919,7 +919,7 @@ mem1:				saved_errno = errno;
 		}
 
 		p->fts_level = level;
-		p->fts_dirfd = _dup(_dirfd(dirp));
+		p->fts_dirfd = _fcntl(_dirfd(dirp), F_DUPFD_CLOEXEC);
 		p->fts_parent = sp->fts_cur;
 		p->fts_pathlen = len + dnamlen;
 

--3346832139-145962118-1786338523=:5014--