Date: Tue, 4 Mar 2008 19:43:00 +0200 From: Kostik Belousov <kostikbel@gmail.com> To: Pietro Cerutti <gahr@freebsd.org> Cc: freebsd-current <freebsd-current@freebsd.org> Subject: Re: kernel panic at giant_poll (kern_conf.c:385) Message-ID: <20080304174300.GD57756@deviant.kiev.zoral.com.ua> In-Reply-To: <47CD3208.90708@FreeBSD.org> References: <47CD3208.90708@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--BDQ1BLzfIZJT+sgm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Mar 04, 2008 at 12:27:04PM +0100, Pietro Cerutti wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 >=20 > Dear all, >=20 > FreeBSD gahrtop.localhost 8.0-CURRENT FreeBSD 8.0-CURRENT #8: Mon Mar 3 > 14:07:47 CET 2008 > root@gahrtop.localhost:/usr/obj/usr/src/sys/MSI1034 i386 >=20 > vmcore and kernel.symbols available at: > http://www.gahr.ch/FreeBSD/misc/crash17/vmcore.17 > http://www.gahr.ch/FreeBSD/misc/crash17/kernel.symbols >=20 > (if you get a page not found, 's|www.gahr.ch|213.142.182.66|', DNS are > still refreshing after relocation...) >=20 > Fatal trap 12: page fault while in kernel mode > cpuid =3D 1; apic id =3D 01 > fault virtual address =3D 0x60 > fault code =3D supervisor read, page not present > instruction pointer =3D 0x20:0xc055d43b > stack pointer =3D 0x28:0xe757ca68 > frame pointer =3D 0x28:0xe757ca84 > code segment =3D base 0x0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, def32 1, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 399 (moused) > trap number =3D 12 > panic: page fault > cpuid =3D 1 > Uptime: 54m21s > Physical memory: 2031 MB > Dumping 83 MB: 68 52 36 20 4 >=20 > (kgdb) bt > #0 doadump () at pcpu.h:195 > #1 0xc0592a86 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c= :417 > #2 0xc0592ec8 in panic (fmt=3D0x104 <Address 0x104 out of bounds>) at > /usr/src/sys/kern/kern_shutdown.c:571 > #3 0xc07f5b8a in trap_fatal (frame=3D0xe757ca28, eva=3D40) at > /usr/src/sys/i386/i386/trap.c:898 > #4 0xc07f5ef9 in trap_pfault (frame=3D0xe757ca28, usermode=3D0, eva=3D96= ) at > /usr/src/sys/i386/i386/trap.c:811 > #5 0xc07f68db in trap (frame=3D0xe757ca28) at > /usr/src/sys/i386/i386/trap.c:489 > #6 0xc07dd1fb in calltrap () at /usr/src/sys/i386/i386/exception.s:146 > #7 0xc055d43b in giant_poll (dev=3D0xc4c09e00, events=3D64, td=3D0xc4e6b= 220) > at /usr/src/sys/kern/kern_conf.c:385 > #8 0xc05161d5 in devfs_poll_f (fp=3D0xc4e715e4, events=3D64, > cred=3D0xc4ace500, td=3D0xc4e6b220) at /usr/src/sys/fs/devfs/devfs_vnops.= c:842 > #9 0xc05c9fca in kern_select (td=3D0xc4e6b220, nd=3D20, fd_in=3D0xbfbfea= 70, > fd_ou=3D0x0, fd_ex=3D0x0, tvp=3D0xe757cc6c) at file.h:265 > #10 0xc05ca41c in select (td=3D0xc4e6b220, uap=3D0xe757ccf8) at > /usr/src/sys/kern/sys_generic.c:758 > #11 0xc07f6142 in syscall (frame=3D0xe757cd38) at > /usr/src/sys/i386/i386/trap.c:1034 > #12 0xc07dd260 in Xint0x80_syscall () at > /usr/src/sys/i386/i386/exception.s:203 > #13 0x00000033 in ?? () > (kgdb) up 7 > #7 0xc055d43b in giant_poll (dev=3D0xc4c09e00, events=3D64, td=3D0xc4e6b= 220) > at /usr/src/sys/kern/kern_conf.c:385 > 385 retval =3D dev->si_devsw->d_gianttrick-> > (kgdb) list > 380 giant_poll(struct cdev *dev, int events, struct thread *td) > 381 { > 382 int retval; > 383 > 384 mtx_lock(&Giant); > 385 retval =3D dev->si_devsw->d_gianttrick-> > 386 d_poll(dev, events, td); > 387 mtx_unlock(&Giant); > 388 return (retval); > 389 } > (kgdb) set print pretty > (kgdb) print *dev > $2 =3D { > si_priv =3D 0xc4c09e00, > si_flags =3D 0, > si_atime =3D { > tv_sec =3D 1204568256, > tv_nsec =3D 0 > }, > si_ctime =3D { > tv_sec =3D 0, > tv_nsec =3D 0 > }, > si_mtime =3D { > tv_sec =3D 0, > tv_nsec =3D 0 > }, > si_uid =3D 0, > si_gid =3D 5, > si_mode =3D 420, > si_cred =3D 0x0, > si_drv0 =3D 0, > si_refcount =3D 3, > si_list =3D { > le_next =3D 0x0, > le_prev =3D 0xc0871a58 > }, > si_clone =3D { > le_next =3D 0x0, > le_prev =3D 0x0 > }, > si_children =3D { > lh_first =3D 0x0 > }, > si_siblings =3D { > le_next =3D 0x0, > le_prev =3D 0x0 > }, > si_parent =3D 0x0, > si_name =3D 0xc4c09e78 "ums0", > si_drv1 =3D 0x0, > si_drv2 =3D 0x0, > si_devsw =3D 0x0, > si_iosize_max =3D 65536, > si_usecount =3D 1, > si_threadcount =3D 1, > __si_u =3D { > - ---Type <return> to continue, or q <return> to quit--- > __sit_tty =3D 0x0, > __sid_snapdata =3D 0x0 > }, > __si_namebuf =3D "ums0", '\0' <repeats 59 times> > } >=20 > Please note that dev->si_devsw is NULL, and kern_conf.c:385 tries to > dereference it. I remember this issue was already reported by M. Warner Losh. The reason for the problem is that si_refcount does not protect struct cdev, it only keeps the cdev structure from going away (as well as protecting against finalizing the cdevsw). The callers of devvn_refthread() are assumed to only use the returned dsw instead of dereferencing dev again. Giant_trick explicitely breaks the rule. Besides the problem I described above, there is another race in make_dev_credv(), where cdevsw may be finalized just before new device is created, since dev_mtx is dropped between prep_cdevsw and newdev() and later. This requires rapid device creation/destruction, and was discovered by Peter Holm. You may try the following patch. Besides removing giant_trick and directly checking the Giant, it puts the prep_cdevsw() into the common dev_mtx-protected region. diff --git a/sys/fs/devfs/devfs_vnops.c b/sys/fs/devfs/devfs_vnops.c index d9565ee..58e943a 100644 --- a/sys/fs/devfs/devfs_vnops.c +++ b/sys/fs/devfs/devfs_vnops.c @@ -369,7 +369,9 @@ devfs_close(struct vop_close_args *ap) error =3D dsw->d_close(dev, ap->a_fflag, S_IFCHR, td); PICKUP_GIANT(); } else { + mtx_lock(&Giant); error =3D dsw->d_close(dev, ap->a_fflag, S_IFCHR, td); + mtx_unlock(&Giant); } dev_relthread(dev); vn_lock(vp, vp_locked | LK_RETRY); @@ -491,7 +493,11 @@ devfs_ioctl_f(struct file *fp, u_long com, void *data,= struct ucred *cred, struc dev_relthread(dev); return (error); } + if (dsw->d_flags & D_NEEDGIANT) + mtx_lock(&Giant); error =3D dsw->d_ioctl(dev, com, data, fp->f_flag, td); + if (dsw->d_flags & D_NEEDGIANT) + mtx_unlock(&Giant); dev_relthread(dev); if (error =3D=3D ENOIOCTL) error =3D ENOTTY; @@ -534,7 +540,11 @@ devfs_kqfilter_f(struct file *fp, struct knote *kn) error =3D devfs_fp_check(fp, &dev, &dsw); if (error) return (error); + if (dsw->d_flags & D_NEEDGIANT) + mtx_lock(&Giant); error =3D dsw->d_kqfilter(dev, kn); + if (dsw->d_flags & D_NEEDGIANT) + mtx_unlock(&Giant); dev_relthread(dev); return (error); } @@ -780,10 +790,12 @@ devfs_open(struct vop_open_args *ap) error =3D dsw->d_open(dev, ap->a_mode, S_IFCHR, td); PICKUP_GIANT(); } else { + mtx_lock(&Giant); if (dsw->d_fdopen !=3D NULL) error =3D dsw->d_fdopen(dev, ap->a_mode, td, fp); else error =3D dsw->d_open(dev, ap->a_mode, S_IFCHR, td); + mtx_unlock(&Giant); } =20 vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); @@ -839,7 +851,11 @@ devfs_poll_f(struct file *fp, int events, struct ucred= *cred, struct thread *td) error =3D devfs_fp_check(fp, &dev, &dsw); if (error) return (error); + if (dsw->d_flags & D_NEEDGIANT) + mtx_lock(&Giant); error =3D dsw->d_poll(dev, events, td); + if (dsw->d_flags & D_NEEDGIANT) + mtx_unlock(&Giant); dev_relthread(dev); return(error); } @@ -874,7 +890,11 @@ devfs_read_f(struct file *fp, struct uio *uio, struct = ucred *cred, int flags, st if ((flags & FOF_OFFSET) =3D=3D 0) uio->uio_offset =3D fp->f_offset; =20 + if (dsw->d_flags & D_NEEDGIANT) + mtx_lock(&Giant); error =3D dsw->d_read(dev, uio, ioflag); + if (dsw->d_flags & D_NEEDGIANT) + mtx_unlock(&Giant); if (uio->uio_resid !=3D resid || (error =3D=3D 0 && resid !=3D 0)) vfs_timestamp(&dev->si_atime); dev_relthread(dev); @@ -1308,7 +1328,11 @@ devfs_write_f(struct file *fp, struct uio *uio, stru= ct ucred *cred, int flags, s =20 resid =3D uio->uio_resid; =20 + if (dsw->d_flags & D_NEEDGIANT) + mtx_lock(&Giant); error =3D dsw->d_write(dev, uio, ioflag); + if (dsw->d_flags & D_NEEDGIANT) + mtx_unlock(&Giant); if (uio->uio_resid !=3D resid || (error =3D=3D 0 && resid !=3D 0)) { vfs_timestamp(&dev->si_ctime); dev->si_mtime =3D dev->si_ctime; diff --git a/sys/kern/kern_conf.c b/sys/kern/kern_conf.c index 33285b4..910a41a 100644 --- a/sys/kern/kern_conf.c +++ b/sys/kern/kern_conf.c @@ -294,125 +294,6 @@ no_poll(struct cdev *dev __unused, int events, struct= thread *td __unused) =20 #define no_dump (dumper_t *)enodev =20 -static int -giant_open(struct cdev *dev, int oflags, int devtype, struct thread *td) -{ - int retval; - - mtx_lock(&Giant); - retval =3D dev->si_devsw->d_gianttrick-> - d_open(dev, oflags, devtype, td); - mtx_unlock(&Giant); - return (retval); -} - -static int -giant_fdopen(struct cdev *dev, int oflags, struct thread *td, struct file = *fp) -{ - int retval; - - mtx_lock(&Giant); - retval =3D dev->si_devsw->d_gianttrick-> - d_fdopen(dev, oflags, td, fp); - mtx_unlock(&Giant); - return (retval); -} - -static int -giant_close(struct cdev *dev, int fflag, int devtype, struct thread *td) -{ - int retval; - - mtx_lock(&Giant); - retval =3D dev->si_devsw->d_gianttrick-> - d_close(dev, fflag, devtype, td); - mtx_unlock(&Giant); - return (retval); -} - -static void -giant_strategy(struct bio *bp) -{ - - mtx_lock(&Giant); - bp->bio_dev->si_devsw->d_gianttrick-> - d_strategy(bp); - mtx_unlock(&Giant); -} - -static int -giant_ioctl(struct cdev *dev, u_long cmd, caddr_t data, int fflag, struct = thread *td) -{ - int retval; - - mtx_lock(&Giant); - retval =3D dev->si_devsw->d_gianttrick-> - d_ioctl(dev, cmd, data, fflag, td); - mtx_unlock(&Giant); - return (retval); -} - =20 -static int -giant_read(struct cdev *dev, struct uio *uio, int ioflag) -{ - int retval; - - mtx_lock(&Giant); - retval =3D dev->si_devsw->d_gianttrick-> - d_read(dev, uio, ioflag); - mtx_unlock(&Giant); - return (retval); -} - -static int -giant_write(struct cdev *dev, struct uio *uio, int ioflag) -{ - int retval; - - mtx_lock(&Giant); - retval =3D dev->si_devsw->d_gianttrick-> - d_write(dev, uio, ioflag); - mtx_unlock(&Giant); - return (retval); -} - -static int -giant_poll(struct cdev *dev, int events, struct thread *td) -{ - int retval; - - mtx_lock(&Giant); - retval =3D dev->si_devsw->d_gianttrick-> - d_poll(dev, events, td); - mtx_unlock(&Giant); - return (retval); -} - -static int -giant_kqfilter(struct cdev *dev, struct knote *kn) -{ - int retval; - - mtx_lock(&Giant); - retval =3D dev->si_devsw->d_gianttrick-> - d_kqfilter(dev, kn); - mtx_unlock(&Giant); - return (retval); -} - -static int -giant_mmap(struct cdev *dev, vm_offset_t offset, vm_paddr_t *paddr, int np= rot) -{ - int retval; - - mtx_lock(&Giant); - retval =3D dev->si_devsw->d_gianttrick-> - d_mmap(dev, offset, paddr, nprot); - mtx_unlock(&Giant); - return (retval); -} - - /* * struct cdev * and u_dev_t primitives */ @@ -485,27 +366,16 @@ umajor(dev_t dev) static void fini_cdevsw(struct cdevsw *devsw) { - struct cdevsw *gt; + mtx_assert(&devmtx, MA_OWNED); =20 - if (devsw->d_gianttrick !=3D NULL) { - gt =3D devsw->d_gianttrick; - memcpy(devsw, gt, sizeof *devsw); - free(gt, M_DEVT); - devsw->d_gianttrick =3D NULL; - } devsw->d_flags &=3D ~D_INIT; } =20 static void prep_cdevsw(struct cdevsw *devsw) { - struct cdevsw *dsw2; =20 - if (devsw->d_flags & D_NEEDGIANT) - dsw2 =3D malloc(sizeof *dsw2, M_DEVT, M_WAITOK); - else - dsw2 =3D NULL; - dev_lock(); + mtx_assert(&devmtx, MA_OWNED); =20 if (devsw->d_version !=3D D_VERSION_01) { printf( @@ -532,41 +402,29 @@ prep_cdevsw(struct cdevsw *devsw) if (devsw->d_poll =3D=3D NULL) devsw->d_poll =3D ttypoll; } =20 - if (devsw->d_flags & D_NEEDGIANT) { - if (devsw->d_gianttrick =3D=3D NULL) { - memcpy(dsw2, devsw, sizeof *dsw2); - devsw->d_gianttrick =3D dsw2; - } else - free(dsw2, M_DEVT); - } - -#define FIXUP(member, noop, giant) \ +#define FIXUP(member, noop) \ do { \ if (devsw->member =3D=3D NULL) { \ devsw->member =3D noop; \ - } else if (devsw->d_flags & D_NEEDGIANT) \ - devsw->member =3D giant; \ } \ - while (0) - - FIXUP(d_open, null_open, giant_open); - FIXUP(d_fdopen, NULL, giant_fdopen); - FIXUP(d_close, null_close, giant_close); - FIXUP(d_read, no_read, giant_read); - FIXUP(d_write, no_write, giant_write); - FIXUP(d_ioctl, no_ioctl, giant_ioctl); - FIXUP(d_poll, no_poll, giant_poll); - FIXUP(d_mmap, no_mmap, giant_mmap); - FIXUP(d_strategy, no_strategy, giant_strategy); - FIXUP(d_kqfilter, no_kqfilter, giant_kqfilter); + } while (0) + + FIXUP(d_open, null_open); + FIXUP(d_fdopen, NULL); + FIXUP(d_close, null_close); + FIXUP(d_read, no_read); + FIXUP(d_write, no_write); + FIXUP(d_ioctl, no_ioctl); + FIXUP(d_poll, no_poll); + FIXUP(d_mmap, no_mmap); + FIXUP(d_strategy, no_strategy); + FIXUP(d_kqfilter, no_kqfilter); =20 if (devsw->d_dump =3D=3D NULL) devsw->d_dump =3D no_dump; =20 LIST_INIT(&devsw->d_devs); =20 devsw->d_flags |=3D D_INIT; - - dev_unlock(); } =20 struct cdev * @@ -580,10 +438,10 @@ make_dev_credv(int flags, struct cdevsw *devsw, int m= inornr, KASSERT((minornr & ~MAXMINOR) =3D=3D 0, ("Invalid minor (0x%x) in make_dev", minornr)); =20 - if (!(devsw->d_flags & D_INIT))=20 - prep_cdevsw(devsw); dev =3D devfs_alloc(); dev_lock(); + if (!(devsw->d_flags & D_INIT))=20 + prep_cdevsw(devsw); dev =3D newdev(devsw, minornr, dev); if (flags & MAKEDEV_REF) dev_refl(dev); @@ -884,9 +742,6 @@ clone_create(struct clonedevs **cdp, struct cdevsw *csw= , int *up, struct cdev ** KASSERT(*up <=3D CLONE_UNITMASK, ("Too high unit (0x%x) in clone_create", *up)); =20 - if (!(csw->d_flags & D_INIT)) - prep_cdevsw(csw); - /* * Search the list for a lot of things in one go: * A preexisting match is returned immediately. @@ -898,6 +753,8 @@ clone_create(struct clonedevs **cdp, struct cdevsw *csw= , int *up, struct cdev ** unit =3D *up; ndev =3D devfs_alloc(); dev_lock(); + if (!(csw->d_flags & D_INIT)) + prep_cdevsw(csw); low =3D extra; de =3D dl =3D NULL; cd =3D *cdp; diff --git a/sys/kern/tty_cons.c b/sys/kern/tty_cons.c index 35638c4..758d225 100644 --- a/sys/kern/tty_cons.c +++ b/sys/kern/tty_cons.c @@ -393,7 +393,11 @@ cn_devopen(struct cn_device *cnd, struct thread *td, i= nt forceopen) csw =3D dev_refthread(dev); if (csw =3D=3D NULL) return (ENXIO); + if (csw->d_flags & D_NEEDGIANT) + mtx_lock(&Giant); error =3D (*csw->d_open)(dev, openflag, 0, td); + if (csw->d_flags & D_NEEDGIANT) + mtx_unlock(&Giant); dev_relthread(dev); return (error); } @@ -458,7 +462,11 @@ cnread(struct cdev *dev, struct uio *uio, int flag) csw =3D dev_refthread(dev); if (csw =3D=3D NULL) return (ENXIO); + if (csw->d_flags & D_NEEDGIANT) + mtx_lock(&Giant); error =3D (csw->d_read)(dev, uio, flag); + if (csw->d_flags & D_NEEDGIANT) + mtx_unlock(&Giant); dev_relthread(dev); return (error); } @@ -482,7 +490,11 @@ cnwrite(struct cdev *dev, struct uio *uio, int flag) csw =3D dev_refthread(dev); if (csw =3D=3D NULL) return (ENXIO); + if (csw->d_flags & D_NEEDGIANT) + mtx_lock(&Giant); error =3D (csw->d_write)(dev, uio, flag); + if (csw->d_flags & D_NEEDGIANT) + mtx_unlock(&Giant); dev_relthread(dev); return (error); } @@ -518,7 +530,11 @@ cnioctl(struct cdev *dev, u_long cmd, caddr_t data, in= t flag, struct thread *td) csw =3D dev_refthread(dev); if (csw =3D=3D NULL) return (ENXIO); + if (csw->d_flags & D_NEEDGIANT) + mtx_lock(&Giant); error =3D (csw->d_ioctl)(dev, cmd, data, flag, td); + if (csw->d_flags & D_NEEDGIANT) + mtx_unlock(&Giant); dev_relthread(dev); return (error); } @@ -543,7 +559,11 @@ cnpoll(struct cdev *dev, int events, struct thread *td) csw =3D dev_refthread(dev); if (csw =3D=3D NULL) return (ENXIO); + if (csw->d_flags & D_NEEDGIANT) + mtx_lock(&Giant); error =3D (csw->d_poll)(dev, events, td); + if (csw->d_flags & D_NEEDGIANT) + mtx_unlock(&Giant); dev_relthread(dev); return (error); } @@ -564,7 +584,11 @@ cnkqfilter(struct cdev *dev, struct knote *kn) csw =3D dev_refthread(dev); if (csw =3D=3D NULL) return (ENXIO); + if (csw->d_flags & D_NEEDGIANT) + mtx_lock(&Giant); error =3D (csw->d_kqfilter)(dev, kn); + if (csw->d_flags & D_NEEDGIANT) + mtx_unlock(&Giant); dev_relthread(dev); return (error); } diff --git a/sys/kern/vfs_bio.c b/sys/kern/vfs_bio.c index 00ff023..94f5986 100644 --- a/sys/kern/vfs_bio.c +++ b/sys/kern/vfs_bio.c @@ -3118,7 +3118,11 @@ dev_strategy(struct cdev *dev, struct buf *bp) bufdone(bp); return; } + if (csw->d_flags & D_NEEDGIANT) + mtx_lock(&Giant); (*csw->d_strategy)(bip); + if (csw->d_flags & D_NEEDGIANT) + mtx_unlock(&Giant); dev_relthread(dev); } =20 diff --git a/sys/sys/conf.h b/sys/sys/conf.h index c36ea8c..fdf9d4f 100644 --- a/sys/sys/conf.h +++ b/sys/sys/conf.h @@ -213,7 +213,7 @@ struct cdevsw { LIST_ENTRY(cdevsw) d_list; LIST_HEAD(, cdev) d_devs; int d_spare3; - struct cdevsw *d_gianttrick; + void *d_spare4; }; =20 #define NUMCDEVSW 256 diff --git a/sys/vm/device_pager.c b/sys/vm/device_pager.c index f0d661f..4f67112 100644 --- a/sys/vm/device_pager.c +++ b/sys/vm/device_pager.c @@ -131,11 +131,17 @@ dev_pager_alloc(void *handle, vm_ooffset_t size, vm_p= rot_t prot, vm_ooffset_t fo * XXX assumes VM_PROT_* =3D=3D PROT_* */ npages =3D OFF_TO_IDX(size); + if (csw->d_flags & D_NEEDGIANT) + mtx_lock(&Giant); for (off =3D foff; npages--; off +=3D PAGE_SIZE) if ((*csw->d_mmap)(dev, off, &paddr, (int)prot) !=3D 0) { + if (csw->d_flags & D_NEEDGIANT) + mtx_unlock(&Giant); dev_relthread(dev); return (NULL); } + if (csw->d_flags & D_NEEDGIANT) + mtx_unlock(&Giant); =20 mtx_lock(&dev_pager_mtx); =20 @@ -220,7 +226,11 @@ dev_pager_getpages(object, m, count, reqpage) panic("dev_pager_getpage: no cdevsw"); prot =3D PROT_READ; /* XXX should pass in? */ =20 + if (csw->d_flags & D_NEEDGIANT) + mtx_lock(&Giant); ret =3D (*csw->d_mmap)(dev, (vm_offset_t)offset << PAGE_SHIFT, &paddr, pr= ot); + if (csw->d_flags & D_NEEDGIANT) + mtx_unlock(&Giant); KASSERT(ret =3D=3D 0, ("dev_pager_getpage: map function returns error")); dev_relthread(dev); =20 --BDQ1BLzfIZJT+sgm Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEARECAAYFAkfNiiMACgkQC3+MBN1Mb4gz/ACgoE3gXiCoFPbA/tPTWkXMJpUZ iOQAoMA+afp3wTBGbTn0flLoOfc3n8KY =E3la -----END PGP SIGNATURE----- --BDQ1BLzfIZJT+sgm--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080304174300.GD57756>