Date: Wed, 23 Jul 2014 17:11:22 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Bryan Drewery <bdrewery@FreeBSD.org> Cc: current@FreeBSD.org Subject: Re: r268621: panic: shadowed tmpfs v_object [with dump] Message-ID: <20140723141122.GG93733@kib.kiev.ua> In-Reply-To: <53CEDD74.9070804@FreeBSD.org> References: <53CED27C.4080306@FreeBSD.org> <53CED29F.1090809@FreeBSD.org> <53CED718.2090108@FreeBSD.org> <53CEDD74.9070804@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--pZNecdF2o6CbdTLS
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Tue, Jul 22, 2014 at 02:53:56PM -0700, Bryan Drewery wrote:
> On 7/22/14, 2:26 PM, Bryan Drewery wrote:
> > On 7/22/14, 2:07 PM, Bryan Drewery wrote:
> >> Meant to send to current@, moving there.
> >>
> >> On 7/22/14, 2:07 PM, Bryan Drewery wrote:
> >>> On r268621:
> >>>
> >>>> panic: shadowed tmpfs v_object 0xfffff807a7f96600
> >>>> cpuid =3D 0
> >>>> KDB: stack backtrace:
> >>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> >>>> 0xfffffe1247d67390
> >>>> kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe1247d67440
> >>>> vpanic() at vpanic+0x126/frame 0xfffffe1247d67480
> >>>> kassert_panic() at kassert_panic+0x139/frame 0xfffffe1247d674f0
> >>>> vm_object_deallocate() at vm_object_deallocate+0x236/frame
> >>>> 0xfffffe1247d67550
> >>>> tmpfs_free_node() at tmpfs_free_node+0x138/frame 0xfffffe1247d67580
> >>>> tmpfs_reclaim() at tmpfs_reclaim+0x17d/frame 0xfffffe1247d675c0
> >>>> VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xf7/frame 0xfffffe1247d675f0
> >>>> vgonel() at vgonel+0x1a1/frame 0xfffffe1247d67660
> >>>> vrecycle() at vrecycle+0x3e/frame 0xfffffe1247d67690
> >>>> tmpfs_inactive() at tmpfs_inactive+0x4c/frame 0xfffffe1247d676b0
> >>>> VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xf7/frame 0xfffffe1247d676e0
> >>>> vinactive() at vinactive+0xc6/frame 0xfffffe1247d67730
> >>>> vputx() at vputx+0x27a/frame 0xfffffe1247d67790
> >>>> tmpfs_rename() at tmpfs_rename+0xf5/frame 0xfffffe1247d67860
> >>>> VOP_RENAME_APV() at VOP_RENAME_APV+0xfc/frame 0xfffffe1247d67890
> >>>> kern_renameat() at kern_renameat+0x3ef/frame 0xfffffe1247d67ae0
> >>>> amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe1247d67bf0
> >>>> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe1247d67bf0
> >>>> --- syscall (128, FreeBSD ELF64, sys_rename), rip =3D 0x80088b74a, r=
sp =3D
> >>>> 0x7fffffffe238, rbp =3D 0x7fffffffe710 ---
> >>>> Uptime: 6d4h0m3s
> >>>>
> >>>> Dump failed. Partition too small.
> >>>
> >>> Unfortunately I have no dump to debug.
> >>>
> >>
> > Running poudriere again after boot hit the issue right away:
> >
> >
> >> (kgdb) bt
> >> #0 doadump (textdump=3D1) at pcpu.h:219
> >> #1 0xffffffff809122a7 in kern_reboot (howto=3D260) at
> >> /usr/src/sys/kern/kern_shutdown.c:445
> >> #2 0xffffffff809127e5 in vpanic (fmt=3D<value optimized out>, ap=3D<v=
alue
> >> optimized out>) at /usr/src/sys/kern/kern_shutdown.c:744
> >> #3 0xffffffff80912679 in kassert_panic (fmt=3D<value optimized out>) =
at
> >> /usr/src/sys/kern/kern_shutdown.c:632
> >> #4 0xffffffff80ba7996 in vm_object_deallocate (object=3D<value
> >> optimized out>) at /usr/src/sys/vm/vm_object.c:562
> >> #5 0xffffffff820a75a8 in tmpfs_free_node (tmp=3D0xfffff800b5155980,
> >> node=3D0xfffff802716ba740) at
> >> /usr/src/sys/modules/tmpfs/../../fs/tmpfs/tmpfs_subr.c:335
> >> #6 0xffffffff820a363d in tmpfs_reclaim (v=3D<value optimized out>) at
> >> /usr/src/sys/modules/tmpfs/../../fs/tmpfs/tmpfs_vnops.c:1276
> >> #7 0xffffffff80e48717 in VOP_RECLAIM_APV (vop=3D<value optimized out>,
> >> a=3D<value optimized out>) at vnode_if.c:2017
> >> #8 0xffffffff809c1381 in vgonel (vp=3D0xfffff802716b61d8) at
> >> vnode_if.h:830
> >> #9 0xffffffff809c18be in vrecycle (vp=3D0xfffff802716b61d8) at
> >> /usr/src/sys/kern/vfs_subr.c:2655
> >> #10 0xffffffff820a61cc in tmpfs_inactive (v=3D<value optimized out>) at
> >> /usr/src/sys/modules/tmpfs/../../fs/tmpfs/tmpfs_vnops.c:1242
> >> #11 0xffffffff80e485b7 in VOP_INACTIVE_APV (vop=3D<value optimized out=
>,
> >> a=3D<value optimized out>) at vnode_if.c:1951
> >> #12 0xffffffff809bfd36 in vinactive (vp=3D0xfffff802716b61d8,
> >> td=3D0xfffff80187e29920) at vnode_if.h:807
> >> #13 0xffffffff809c012a in vputx (vp=3D0xfffff802716b61d8, func=3D2) at
> >> /usr/src/sys/kern/vfs_subr.c:2267
> >> #14 0xffffffff820a47c5 in tmpfs_rename (v=3D<value optimized out>) at
> >> /usr/src/sys/modules/tmpfs/../../fs/tmpfs/tmpfs_vnops.c:1023
> >> #15 0xffffffff80e47d3c in VOP_RENAME_APV (vop=3D<value optimized out>,
> >> a=3D<value optimized out>) at vnode_if.c:1544
> >> #16 0xffffffff809cc77f in kern_renameat (td=3D<value optimized out>,
> >> oldfd=3D<value optimized out>, old=3D<value optimized out>, newfd=3D<v=
alue
> >> optimized out>, new=3D<value optimized out>,
> >> pathseg=3D<value optimized out>) at vnode_if.h:636
> >> #17 0xffffffff80d280fa in amd64_syscall (td=3D0xfffff80187e29920,
> >> traced=3D0) at subr_syscall.c:133
> >> #18 0xffffffff80d0a64b in Xfast_syscall () at
> >> /usr/src/sys/amd64/amd64/exception.S:407
> >> (kgdb) p *(vm_object_t)0xfffff8027169f500
> >> $1 =3D {lock =3D {lock_object =3D {lo_name =3D 0xffffffff80fe89f6 "vm =
object",
> >> lo_flags =3D 90374144, lo_data =3D 0, lo_witness =3D 0xfffffe00006e768=
0},
> >> rw_lock =3D 18446735284191271200}, object_list =3D {
> >> tqe_next =3D 0xfffff8027169f400, tqe_prev =3D 0xfffff8027169f620},
> >> shadow_head =3D {lh_first =3D 0xfffff801b8489e00}, shadow_list =3D {le=
_next
> >> =3D 0x0, le_prev =3D 0x0}, memq =3D {tqh_first =3D 0xfffff811d966bc08,
> >> tqh_last =3D 0xfffff811d966bc18}, rtree =3D {rt_root =3D
> >> 18446735354278362121, rt_flags =3D 0 '\0'}, size =3D 1, generation =3D=
1,
> >> ref_count =3D 1, shadow_count =3D 1, memattr =3D 6 '\006', type =3D 1 =
'\001',
> >> flags =3D 528, pg_color =3D 0, paging_in_progress =3D 0,
> >> resident_page_count =3D 1, backing_object =3D 0x0, backing_object_offs=
et =3D
> >> 0, pager_object_list =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, rvq =3D=
{
> >> lh_first =3D 0x0}, cache =3D {rt_root =3D 0, rt_flags =3D 0 '\0'},=
handle
> >> =3D 0x0, un_pager =3D {vnp =3D {vnp_size =3D 0, writemappings =3D 0}, =
devp =3D
> >> {devp_pglist =3D {tqh_first =3D 0x0, tqh_last =3D 0x0}, ops =3D 0x0,
> >> dev =3D 0x0}, sgp =3D {sgp_pglist =3D {tqh_first =3D 0x0, tqh_la=
st =3D
> >> 0x0}}, swp =3D {swp_tmpfs =3D 0x0, swp_bcount =3D 0}}, cred =3D 0x0, c=
harge =3D 0}
> >> (kgdb) frame 8
> >> #8 0xffffffff809c1381 in vgonel (vp=3D0xfffff802716b61d8) at
> >> vnode_if.h:830
> >> 830 return (VOP_RECLAIM_APV(vp->v_op, &a));
> >> (kgdb) p *vp
> >> $2 =3D {v_tag =3D 0xffffffff820abf96 "tmpfs", v_op =3D 0xffffffff820ac=
938,
> >> v_data =3D 0x0, v_mount =3D 0xfffff8004733a000, v_nmntvnodes =3D {tqe_=
next =3D
> >> 0xfffff802716b6000, tqe_prev =3D 0xfffff802716b63d0}, v_un =3D {
> >> vu_mount =3D 0x0, vu_socket =3D 0x0, vu_cdev =3D 0x0, vu_fifoinfo =
=3D
> >> 0x0}, v_hashlist =3D {le_next =3D 0x0, le_prev =3D 0x0}, v_cache_src =
=3D
> >> {lh_first =3D 0x0}, v_cache_dst =3D {tqh_first =3D 0x0,
> >> tqh_last =3D 0xfffff802716b6228}, v_cache_dd =3D 0x0, v_lock =3D
> >> {lock_object =3D {lo_name =3D 0xffffffff820abf96 "tmpfs", lo_flags =3D
> >> 116588544, lo_data =3D 0, lo_witness =3D 0xfffffe0000711980},
> >> lk_lock =3D 18446735284191271200, lk_exslpfail =3D 0, lk_timo =3D =
51,
> >> lk_pri =3D 96}, v_interlock =3D {lock_object =3D {lo_name =3D
> >> 0xffffffff80fafc26 "vnode interlock", lo_flags =3D 16973824, lo_data =
=3D 0,
> >> lo_witness =3D 0xfffffe00006e7500}, mtx_lock =3D 4}, v_vnlock =3D
> >> 0xfffff802716b6240, v_actfreelist =3D {tqe_next =3D 0xfffff80271898588,
> >> tqe_prev =3D 0xfffff8004733a078}, v_bufobj =3D {bo_lock =3D {
> >> lock_object =3D {lo_name =3D 0xffffffff80fb8084 "bufobj interloc=
k",
> >> lo_flags =3D 86179840, lo_data =3D 0, lo_witness =3D 0xfffffe00006ef38=
0},
> >> rw_lock =3D 1}, bo_ops =3D 0xffffffff814942a0, bo_object =3D 0x0,
> >> bo_synclist =3D {le_next =3D 0x0, le_prev =3D 0x0}, bo_private =3D
> >> 0xfffff802716b61d8, __bo_vnode =3D 0xfffff802716b61d8, bo_clean =3D {b=
v_hd
> >> =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff802716b62f8}, bv_root =3D {
> >> pt_root =3D 0}, bv_cnt =3D 0}, bo_dirty =3D {bv_hd =3D {tqh_fi=
rst =3D
> >> 0x0, tqh_last =3D 0xfffff802716b6318}, bv_root =3D {pt_root =3D 0}, bv=
_cnt =3D
> >> 0}, bo_numoutput =3D 0, bo_flag =3D 0, bo_bsize =3D 4096},
> >> v_pollinfo =3D 0x0, v_label =3D 0x0, v_lockf =3D 0x0, v_rl =3D {rl_w=
aiters =3D
> >> {tqh_first =3D 0x0, tqh_last =3D 0xfffff802716b6360}, rl_currdep =3D 0=
x0},
> >> v_cstart =3D 0, v_lasta =3D 0, v_lastw =3D 0, v_clen =3D 0,
> >> v_holdcnt =3D 2, v_usecount =3D 0, v_iflag =3D 2688, v_vflag =3D 0,
> >> v_writecount =3D 0, v_hash =3D 40987489, v_type =3D VREG}
> >> (kgdb) info locals
> >> mp =3D (struct mount *) 0xfffff8004733a000
> >> fromnd =3D {ni_dirp =3D 0x801006080 <Address 0x801006080 out of bounds=
>,
> >> ni_segflg =3D UIO_USERSPACE, ni_rightsneeded =3D {cr_rights =3D
> >> {144115188142965760, 288230376151711744}},
> >> ni_startdir =3D 0xfffff802716b63b0, ni_rootdir =3D 0xfffff8026b01a76=
0,
> >> ni_topdir =3D 0xfffff8026b01a760, ni_dirfd =3D -100, ni_strictrelative=
=3D
> >> 0, ni_filecaps =3D {fc_rights =3D {cr_rights =3D {0, 0}},
> >> fc_ioctls =3D 0x0, fc_nioctls =3D -1, fc_fcntls =3D 0}, ni_vp =3D
> >> 0xfffff80271898588, ni_dvp =3D 0xfffff802716b63b0, ni_pathlen =3D 1,
> >> ni_next =3D 0xfffff80061ea501f "", ni_loopcnt =3D 0, ni_cnd =3D {cn_na=
meiop
> >> =3D 2,
> >> cn_flags =3D 67148812, cn_thread =3D 0xfffff80187e29920, cn_cred =
=3D
> >> 0xfffff80038911800, cn_lkflags =3D 524288, cn_pnbuf =3D 0xfffff80061ea=
5000
> >> "/var/run/ld-elf.so.hints.HTjP6A",
> >> cn_nameptr =3D 0xfffff80061ea5009 "ld-elf.so.hints.HTjP6A",
> >> cn_namelen =3D 22, cn_consume =3D 0}}
> >> tond =3D {ni_dirp =3D 0x403e66 <Address 0x403e66 out of bounds>, ni_se=
gflg
> >> =3D UIO_USERSPACE, ni_rightsneeded =3D {cr_rights =3D {144115188080051=
200,
> >> 288230376151711744}}, ni_startdir =3D 0xfffff802716b63b0,
> >> ni_rootdir =3D 0xfffff8026b01a760, ni_topdir =3D 0xfffff8026b01a760,
> >> ni_dirfd =3D -100, ni_strictrelative =3D 0, ni_filecaps =3D {fc_rights=
=3D
> >> {cr_rights =3D {0, 0}}, fc_ioctls =3D 0x0, fc_nioctls =3D -1,
> >> fc_fcntls =3D 0}, ni_vp =3D 0xfffff802716b61d8, ni_dvp =3D
> >> 0xfffff802716b63b0, ni_pathlen =3D 1, ni_next =3D 0xfffff80038d69418 "=
",
> >> ni_loopcnt =3D 0, ni_cnd =3D {cn_nameiop =3D 3, cn_flags =3D 134257708,
> >> cn_thread =3D 0xfffff80187e29920, cn_cred =3D 0xfffff80038911800,
> >> cn_lkflags =3D 524288, cn_pnbuf =3D 0xfffff80038d69400
> >> "/var/run/ld-elf.so.hints", cn_nameptr =3D 0xfffff80038d69409
> >> "ld-elf.so.hints",
> >> cn_namelen =3D 15, cn_consume =3D 0}}
> >> rights =3D {cr_rights =3D {144115188080051200, 288230376151711744}}
> >> mp =3D (struct mount *) 0xfffff8004733a000
> >> error =3D <value optimized out>
> >> fvp =3D <value optimized out>
> >> tvp =3D <value optimized out>
> >> tdvp =3D <value optimized out>
> >> (kgdb) p *mp
> >> $9 =3D {mnt_mtx =3D {lock_object =3D {lo_name =3D 0xffffffff80f8fcec "=
struct
> >> mount mtx", lo_flags =3D 16973824, lo_data =3D 0, lo_witness =3D
> >> 0xfffffe00006e7a00}, mtx_lock =3D 4}, mnt_gen =3D 1, mnt_list =3D {
> >> tqe_next =3D 0xfffff80038fa9cc0, tqe_prev =3D 0xfffff80187b74ce8},
> >> mnt_op =3D 0xffffffff820ace60, mnt_vfc =3D 0xffffffff820acf80,
> >> mnt_vnodecovered =3D 0xfffff801b853e760, mnt_syncer =3D 0xfffff8026b01=
a588,
> >> mnt_ref =3D 13206, mnt_nvnodelist =3D {tqh_first =3D 0xfffff8026b01a=
760,
> >> tqh_last =3D 0xfffff802718985a8}, mnt_nvnodelistsize =3D 13205,
> >> mnt_activevnodelist =3D {tqh_first =3D 0xfffff802716b61d8,
> >> tqh_last =3D 0xfffff8026b01a648}, mnt_activevnodelistsize =3D 730,
> >> mnt_writeopcount =3D 1, mnt_kern_flag =3D 0, mnt_flag =3D 4096, mnt_op=
t =3D
> >> 0xfffff8000e59cc30, mnt_optnew =3D 0xfffff8001b9ea050,
> >> mnt_maxsymlinklen =3D 0, mnt_stat =3D {f_version =3D 537068824, f_ty=
pe =3D
> >> 135, f_flags =3D 4096, f_bsize =3D 4096, f_iosize =3D 4096, f_blocks =
=3D
> >> 1835008, f_bfree =3D 1738991, f_bavail =3D 1738991, f_files =3D 256901=
12,
> >> f_ffree =3D 25676911, f_syncwrites =3D 0, f_asyncwrites =3D 0,
> >> f_syncreads =3D 0, f_asyncreads =3D 0, f_spare =3D {0, 0, 0, 0, 0, 0, =
0, 0,
> >> 0, 0}, f_namemax =3D 255, f_owner =3D 0, f_fsid =3D {val =3D {-2029977=
843,
> >> 135}}, f_charspare =3D '\0' <repeats 79 times>, f_fstypename =
=3D
> >> "tmpfs\000\000\000\000\000\000\000\000\000\000", f_mntfromname =3D
> >> "tmpfs", '\0' <repeats 82 times>,
> >> f_mntonname =3D "/poudriere/data/.m/exp-10amd64-commit-test/01",
> >> '\0' <repeats 42 times>}, mnt_cred =3D 0xfffff80047478700, mnt_data =3D
> >> 0xfffff800b5155980, mnt_time =3D 0, mnt_iosize_max =3D 65536,
> >> mnt_export =3D 0x0, mnt_label =3D 0x0, mnt_hashseed =3D 1147308587,
> >> mnt_lockref =3D 0, mnt_secondary_writes =3D 0, mnt_secondary_accwrites=
=3D
> >> 0, mnt_susp_owner =3D 0x0, mnt_gjprovider =3D 0x0, mnt_explock =3D {
> >> lock_object =3D {lo_name =3D 0xffffffff80f8fd0f "explock", lo_flag=
s =3D
> >> 108199936, lo_data =3D 0, lo_witness =3D 0xfffffe000070ef80}, lk_lock =
=3D 1,
> >> lk_exslpfail =3D 0, lk_timo =3D 0, lk_pri =3D 96},
> >> mnt_upper_link =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, mnt_uppers =
=3D
> >> {tqh_first =3D 0x0, tqh_last =3D 0xfffff8004733a320}}
> >
>=20
> Shadowed object:
>=20
> > (kgdb) p *$1->shadow_head->lh_first
> > $3 =3D {lock =3D {lock_object =3D {lo_name =3D 0xffffffff80fe89f6 "vm o=
bject", lo_flags =3D 90374144, lo_data =3D 0, lo_witness =3D 0xfffffe00006e=
7680}, rw_lock =3D 1}, object_list =3D {tqe_next =3D 0xfffff801b8b3ae00,
> > tqe_prev =3D 0xfffff802717bb120}, shadow_head =3D {lh_first =3D 0x0=
}, shadow_list =3D {le_next =3D 0x0, le_prev =3D 0xfffff8027169f530}, memq =
=3D {tqh_first =3D 0xfffff811da2c75f8, tqh_last =3D 0xfffff811da2c7608},
> > rtree =3D {rt_root =3D 18446735354291320313, rt_flags =3D 0 '\0'}, si=
ze =3D 1, generation =3D 1, ref_count =3D 1, shadow_count =3D 0, memattr =
=3D 6 '\006', type =3D 0 '\0', flags =3D 12288, pg_color =3D 1598,
> > paging_in_progress =3D 0, resident_page_count =3D 1, backing_object =
=3D 0xfffff8027169f500, backing_object_offset =3D 0, pager_object_list =3D =
{tqe_next =3D 0x0, tqe_prev =3D 0x0}, rvq =3D {lh_first =3D 0x0}, cache =3D=
{
> > rt_root =3D 0, rt_flags =3D 0 '\0'}, handle =3D 0x0, un_pager =3D {=
vnp =3D {vnp_size =3D 0, writemappings =3D 0}, devp =3D {devp_pglist =3D {t=
qh_first =3D 0x0, tqh_last =3D 0x0}, ops =3D 0x0, dev =3D 0x0}, sgp =3D {
> > sgp_pglist =3D {tqh_first =3D 0x0, tqh_last =3D 0x0}}, swp =3D {s=
wp_tmpfs =3D 0x0, swp_bcount =3D 0}}, cred =3D 0xfffff80038911800, charge =
=3D 4096}
>=20
Try this
diff --git a/sys/vm/vm_object.c b/sys/vm/vm_object.c
index 1b97bdf..bb01f00 100644
--- a/sys/vm/vm_object.c
+++ b/sys/vm/vm_object.c
@@ -559,8 +559,6 @@ vm_object_deallocate(vm_object_t object)
(object->handle =3D=3D NULL) &&
(object->type =3D=3D OBJT_DEFAULT ||
object->type =3D=3D OBJT_SWAP)) {
- KASSERT((object->flags & OBJ_TMPFS_NODE) =3D=3D 0,
- ("shadowed tmpfs v_object %p", object));
vm_object_t robject;
=20
robject =3D LIST_FIRST(&object->shadow_head);
@@ -568,6 +566,8 @@ vm_object_deallocate(vm_object_t object)
("vm_object_deallocate: ref_count: %d, shadow_count: %d",
object->ref_count,
object->shadow_count));
+ KASSERT((robject->flags & OBJ_TMPFS_NODE) =3D=3D 0,
+ ("shadowed tmpfs v_object %p", object));
if (!VM_OBJECT_TRYWLOCK(robject)) {
/*
* Avoid a potential deadlock.
@@ -637,6 +637,8 @@ retry:
doterm:
temp =3D object->backing_object;
if (temp !=3D NULL) {
+ KASSERT((object->flags & OBJ_TMPFS_NODE) =3D=3D 0,
+ ("shadowed tmpfs v_object 2 %p", object));
VM_OBJECT_WLOCK(temp);
LIST_REMOVE(object, shadow_list);
temp->shadow_count--;
--pZNecdF2o6CbdTLS
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBAgAGBQJTz8KKAAoJEJDCuSvBvK1BvugP+wR3VKgEftuq+LpUthzIVkqd
HmDeWO9HFd84I1ewg3z+Am0nJvkbVKULSw6uGEu6oHVIS6q+rU8zj2iYTU5N0VtX
bzL5yXtX+UR2pLpPCJGSHLfqim8RIDPdWD+ZR2Dt028BUtD8Y1v7ejwIMqD8ZvSt
i36ZyD/WfGW3+5RvXkm60xGxgaA5xcjM6qwBQQGLgoLjXmwf3i6VB2wFVvs53ppS
C7zdCXG4FsO/8YMDNAsf3EtZaFG4QV7wIWHlm9Inys6fCs203URzSPPNl5trRIO7
bffq3YJMTgJoa1K/zp1URIEE4VRWxb8j0fxMgak+xjp3y0WNC/Jv022DJ56UYdfN
o18O6e9oLRiTa2k5ELThBOUxuaEjyZ5dZ43zuyScxGq84HLy64ZjTq0n24Kh0CjB
8Embnkth/lsV/RczQ7eVTwe/Y8Io4o+vPYUYYAXPwywJjVc/A5P9H1CTbEl3ovQq
0ZL5+WzE4vsmy36vs6TYL8ir5H3mGQb/coK52kqDPrqS+Z8uGzs6+Tmo9f/T+pmp
Q4lIi/4OsePYhpdzl6+oIZmn4YL0JK+iF65pG+mnJvYpdo3xFVeuaHPLAA02kYe5
BnUyI5lyexTc2TTn2h43zoSfyaN5s8W51BVpPvPefcklfJfDFKwVj+BpEVBvNxg4
rRS5jcJgY6gLAa3CQ3gL
=EOQL
-----END PGP SIGNATURE-----
--pZNecdF2o6CbdTLS--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140723141122.GG93733>