From owner-freebsd-current@FreeBSD.ORG Sun Feb 15 14:33:02 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3747516A4CE; Sun, 15 Feb 2004 14:33:02 -0800 (PST) Received: from sarevok.webteckies.org (node123e0.a2000.nl [24.132.35.224]) by mx1.FreeBSD.org (Postfix) with ESMTP id C9C4F43D1F; Sun, 15 Feb 2004 14:33:01 -0800 (PST) (envelope-from mdev@sarevok.webteckies.org) Received: by sarevok.webteckies.org (Postfix, from userid 100) id D97DBB83A; Sun, 15 Feb 2004 23:33:00 +0100 (CET) From: Melvyn Sopacua Organization: WebTeckies.org To: current@FreeBSD.org Date: Sun, 15 Feb 2004 23:32:55 +0100 User-Agent: KMail/1.5.94 References: In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg=pgp-sha1; boundary="Boundary-02=_cO/LACtSa0DXzlk"; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200402152333.00565.freebsd-current@webteckies.org> cc: Robert Watson Subject: Re: Jails that keep hanging around X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Feb 2004 22:33:02 -0000 --Boundary-02=_cO/LACtSa0DXzlk Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Sunday 15 February 2004 19:18, Robert Watson wrote: > On Sun, 15 Feb 2004, Melvyn Sopacua wrote: > > I have yet to figure out what triggers the bug, but I end up with > > 'running' jails, without any processes. So I thought I'd create 'jld' to > > remove a jail. However - prison_find isn't exported to userland. > > Probably for good reason. > [ snip very informative analysis ] > The only real exception to this is the tcpcb > -- TCP connection remenants can last for quite a time after their socket > exits, since they follow the TCP state machine (which has long waits). So > this could be it -- check netstat and see if there are any largely closed > TCP sessions from the jails. FYI, this is not a complete list of > credential references, but it accounts for most of them. > > Are you using any services that pass references to sockets or other file > descriptor objects in and out of the jail using UNIX domain sockets? If > so, that could also be it. Mostly this happens when setting up the jails. The first one (10) with the= =20 jail-${IP}.idg.nl hostname, is created by a script 'makejail.sh', which is= =20 largely based upon the make steps descriped in jail(8). After the steps=20 listed there, the jail is fired up to execute 'ports-jail.sh' inside the=20 jail, which basically sets up some standard ports (portupgrade, shells, vim= ,=20 postfix). I copy over distfiles, but it is possible, that fetch(1) came int= o=20 play. After this script, all is done - the jail should exit. The number 15 with the correct hostname, is me logging in and doing some=20 additional work, mostly installing and configuring ports. The use of fetch(= 1)=20 is very likely, as these ports are used inside jails only and typically don= 't=20 exist in the host system's distfiles. I've logged out, without having stuff= =20 in the background. I've tried to get the refcount, but get stuck on the proc structure - getti= ng=20 a bus error a few lines down. I couldn't find a faster road to this - if=20 anybody can, I'd be happy to run some diagnostics. Probably the most=20 interesting thing is to see whether the prison refcount is really 0. My attempt below, was compiled with -D_KERNEL -lkvm. =2D-=20 Melvyn =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D =46reeBSD sarevok.webteckies.org 5.2-CURRENT FreeBSD 5.2-CURRENT #1: Sat Fe= b 14=20 02:48:30 CET 2004 =20 root@sarevok.webteckies.org:/usr/obj/usr/src/sys/SAREVOK_NOAPM_NODEBUG i386 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D /* vim600: sw=3D4 ts=3D4 ai */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include static struct prison * prison_find(int prid) { struct prison *pr; pid_t pid; char errbuf[_POSIX2_LINE_MAX]; kvm_t *kern; struct kinfo_proc *p_procinfo; struct proc *p_proc; struct ucred *p_uc; int num_procs, i, status; pid =3D fork(); pr =3D NULL; switch (pid) { case -1 : perror("fork()"); break; case 0 : setsid(); jail_attach(prid); sleep(60); exit(0); default: sleep(1); /* wait for jail_attach() */ kern =3D kvm_openfiles(NULL, NULL, NULL, O_RDONLY, errbuf); if( kern =3D=3D NULL ) { fprintf(stderr, "Cannot open kernel memory: %s", errbuf); break; } p_procinfo =3D kvm_getprocs(kern, KERN_PROC_PID, pid, &num_procs); printf("Getting %d processes for pid %d and my pid is %d\n", num_procs,= =20 pid, getpid()); for( i=3Dnum_procs; --i >=3D 0 ; ++p_procinfo ) { p_proc =3D p_procinfo->ki_paddr; if( p_proc =3D=3D NULL ) { fprintf(stderr, "Cannot get proc struct from proinfo %d.\n", i); break; } p_uc =3D (struct ucred *)p_proc->p_ucred; if( p_uc =3D=3D NULL ) { fprintf(stderr, "Cannot get ucred struct from proc (pi %d).\n", i); break; } pr =3D (struct prison *)p_uc->cr_prison; if( pr !=3D NULL ) break; } kvm_close(kern); waitpid(pid, &status, WNOHANG); break; } return pr; } int main(int argc, char *argv[]) { struct prison *p; int jid; if( argc =3D=3D 2 ) { jid =3D (int)strtol(argv[1], NULL, 10); if( jid > 0 ) { p =3D prison_find(jid); if( p =3D=3D NULL ) { fprintf(stderr, "Cannot find the prison!"); } else { printf("Refcount is: %d\n", p->pr_ref); } } else { printf("Invalid jid: %d", jid); } } else { printf("No jid\n"); } =09 return 0; } --Boundary-02=_cO/LACtSa0DXzlk Content-Type: application/pgp-signature Content-Description: signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQBAL/OcOv9JNmfFN5URAk0lAJoCiXBcNebYhGXsBoPBc2aOuAiclwCeKWkk IRSyFk1DVrDORoL9+73oOGI= =Frs0 -----END PGP SIGNATURE----- --Boundary-02=_cO/LACtSa0DXzlk--