Date: Mon, 31 Dec 2012 19:27:44 +0200 From: Konstantin Belousov <kostikbel@gmail.com> To: Ronald Klop <ronald-freebsd8@klop.yi.org> Cc: cognet@freebsd.org, fs@freebsd.org, gber@freebsd.org Subject: Re: Nandfs use of MNT_VNODE_FOREACH Message-ID: <20121231172744.GQ82219@kib.kiev.ua> In-Reply-To: <op.wp6w2zg98527sy@212-182-167-131.ip.telfort.nl> References: <20121227230223.GN82219@kib.kiev.ua> <op.wp6d7vdo8527sy@212-182-167-131.ip.telfort.nl> <20121231162145.GA29588@dft-labs.eu> <20121231162540.GB29588@dft-labs.eu> <20121231163039.GN82219@kib.kiev.ua> <op.wp6w2zg98527sy@212-182-167-131.ip.telfort.nl>
next in thread | previous in thread | raw e-mail | index | archive | help
--1xJJiECQTZ8+YlWm
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Mon, Dec 31, 2012 at 05:50:48PM +0100, Ronald Klop wrote:
> On Mon, 31 Dec 2012 17:30:39 +0100, Konstantin Belousov =20
> <kostikbel@gmail.com> wrote:
>=20
> > On Mon, Dec 31, 2012 at 05:25:40PM +0100, Mateusz Guzik wrote:
> >> On Mon, Dec 31, 2012 at 05:21:45PM +0100, Mateusz Guzik wrote:
> >> > Can you adjust this function so that 'if (mp->mnt_syncer =3D=3D vp)'
> >> > performs VI_UNLOCK as well?
> >> >
> >> > Something like:
> >> > if (mp->mnt_syncer =3D=3D vp || VOP_ISLOCKED(vp)) {
> >> > VI_UNLOCK(mp);
> >> > continue;
> >> > }
> >> >
> >> > I will have time to dig into this next week.
> >>
> >> Err.. I meant VI_UNLOCK(vp).
> >
> > This is definitely a bug, thank you for noticing. Updated patch below.
> >
> > diff --git a/sys/fs/nandfs/nandfs_segment.c =20
> > b/sys/fs/nandfs/nandfs_segment.c
> > index 836bead..7433e77 100644
> > --- a/sys/fs/nandfs/nandfs_segment.c
> > +++ b/sys/fs/nandfs/nandfs_segment.c
> > @@ -478,39 +478,19 @@ nandfs_iterate_dirty_vnodes(struct mount *mp, =20
> > struct nandfs_seginfo *seginfo)
> > struct nandfs_node *nandfs_node;
> > struct vnode *vp, *mvp;
> > struct thread *td;
> > - int error, lockreq, update;
> > + int error, update;
> > td =3D curthread;
> > - lockreq =3D LK_EXCLUSIVE | LK_INTERLOCK | LK_RETRY;
> > - MNT_ILOCK(mp);
> > -
> > - MNT_VNODE_FOREACH(vp, mp, mvp) {
> > + MNT_VNODE_FOREACH_ACTIVE(vp, mp, mvp) {
> > update =3D 0;
> > - if (mp->mnt_syncer =3D=3D vp)
> > - continue;
> > - if (VOP_ISLOCKED(vp))
> > - continue;
> > -
> > - VI_LOCK(vp);
> > - MNT_IUNLOCK(mp);
> > - if (vp->v_iflag & VI_DOOMED) {
> > + if (mp->mnt_syncer =3D=3D vp || VOP_ISLOCKED(vp)) {
> > VI_UNLOCK(vp);
> > - MNT_ILOCK(mp);
> > - continue;
> > - }
> > -
> > - if ((error =3D vget(vp, lockreq, td)) !=3D 0) {
> > - MNT_ILOCK(mp);
> > continue;
> > }
> > -
> > - if (vp->v_iflag & VI_DOOMED) {
> > - vput(vp);
> > - MNT_ILOCK(mp);
> > + if (vget(vp, LK_EXCLUSIVE | LK_INTERLOCK | LK_NOWAIT, td) !=3D 0)
> > continue;
> > - }
> > nandfs_node =3D VTON(vp);
> > if (nandfs_node->nn_flags & IN_MODIFIED) {
> > @@ -532,12 +512,8 @@ nandfs_iterate_dirty_vnodes(struct mount *mp, =20
> > struct nandfs_seginfo *seginfo)
> > if (update)
> > nandfs_node_update(nandfs_node);
> > -
> > - MNT_ILOCK(mp);
> > }
> > - MNT_IUNLOCK(mp);
> > -
> > return (0);
> > }
> >
>=20
> This boots fine. I added some options from the kernel debugging handbook.=
=20
> Some LORs appear. I will copy-paste them here.
Yes, the LORs are genuine nandfs bugs, not caused by the change I posted.
Thank you for the testing.
>=20
> Trying to mount root from nandfs:/dev/gnand0s.root []...
> WARNING: NANDFS is considered to be a highly experimental feature in =20
> FreeBSD.
> (probe0:umass-sim0:0:0:0): Down reving Protocol Version from 2 to 0?
> pass0 at umass-sim0 bus 0 scbus0 target 0 lun 0
> pass0: <USB 2.0 USB Flash Drive 1100> Removable Direct Access SCSI-0 devi=
ce
> pass0: Serial Number AA00000000002542
> pass0: 40.000MB/s transfers
> da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
> da0: <USB 2.0 USB Flash Drive 1100> Removable Direct Access SCSI-0 device
> da0: Serial Number AA00000000002542
> da0: 40.000MB/s transfers
> da0: 3894MB (7975296 512 byte sectors: 255H 63S/T 496C)
> GEOM: new disk da0
> lock order reversal:
> 1st 0xcc334d40 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:1898
> 2nd 0xc3ef4a30 snandfs (snandfs) @ =20
> /usr/src/sys/fs/nandfs/nandfs_subr.c:444
> 3rd 0xcc334860 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2631
> KDB: stack backtrace:
> db_trace_self() at db_trace_self+0x10
> scp=3D0xc0bb462c rlv=3D0xc09334d8 (X_db_sym_numargs+0x1bc)
> rsp=3D0xd35ae3ec rfp=3D0xd35ae508
> X_db_sym_numargs() at X_db_sym_numargs+0x198
> scp=3D0xc09334b4 rlv=3D0xc0a6ed5c (kdb_backtrace+0x3c)
> rsp=3D0xd35ae50c rfp=3D0xd35ae51c
> r4=3D0xc0cc7838
> kdb_backtrace() at kdb_backtrace+0x10
> scp=3D0xc0a6ed30 rlv=3D0xc0a853e8 (witness_display_spinlock+0x80)
> rsp=3D0xd35ae520 rfp=3D0xd35ae534
> r4=3D0x00000001
> witness_display_spinlock() at witness_display_spinlock+0x60
> scp=3D0xc0a853c8 rlv=3D0xc0a861c4 (witness_checkorder+0x8b4)
> rsp=3D0xd35ae538 rfp=3D0xd35ae588
> r5=3D0xc3ef4a30 r4=3D0xc0c27690
> witness_checkorder() at witness_checkorder+0x10
> scp=3D0xc0a85920 rlv=3D0xc0a253bc (__lockmgr_args+0x828)
> rsp=3D0xd35ae58c rfp=3D0xd35ae5e8
> r10=3D0x00000000 r9=3D0xc3ef4ad0
> r8=3D0x00081900 r7=3D0xc3685000 r6=3D0xcc334860 r5=3D0x00000100
> r4=3D0x00080000
> __lockmgr_args() at __lockmgr_args+0x10
> scp=3D0xc0a24ba4 rlv=3D0xc0ab9ff0 (getblk+0x128)
> rsp=3D0xd35ae5ec rfp=3D0xd35ae630
> r10=3D0x00000000 r9=3D0x00000000
> r8=3D0x000002a4 r7=3D0xc3ef4ad0 r6=3D0xcc334800 r5=3D0x00000000
> r4=3D0x00000000
> getblk() at getblk+0x10
> scp=3D0xc0ab9ed8 rlv=3D0xc0aba918 (breadn_flags+0x30)
> rsp=3D0xd35ae634 rfp=3D0xd35ae658
> r10=3D0xd35ae6e4 r9=3D0x0001500a
> r8=3D0xc3fe8000 r7=3D0x00000000 r6=3D0xc3ef49d8 r5=3D0x00000000
> r4=3D0x00000000
> breadn_flags() at breadn_flags+0x10
> scp=3D0xc0aba8f8 rlv=3D0xc098ea58 (nandfs_bread+0x80)
> rsp=3D0xd35ae65c rfp=3D0xd35ae6a0
> r6=3D0x000002a4 r5=3D0xc3fe8000
> r4=3D0x00000000
> nandfs_bread() at nandfs_bread+0x10
> scp=3D0xc098e9e8 rlv=3D0xc098efb0 (nandfs_vtop+0xfc)
> rsp=3D0xd35ae6a4 rfp=3D0xd35ae710
> r10=3D0x00000000 r8=3D0xc3fe8000
> r7=3D0x00000000 r6=3D0xd35ae72c r5=3D0xc3fe7db0 r4=3D0x00000000
> nandfs_vtop() at nandfs_vtop+0x10
> scp=3D0xc098eec4 rlv=3D0xc099551c (nandfs_itimes+0x304)
> rsp=3D0xd35ae714 rfp=3D0xd35ae764
> r10=3D0x00001000 r9=3D0xc3fc5000
> r8=3D0xc3fe7db0 r7=3D0x00000000 r6=3D0x00000000 r5=3D0xcc334ce0
> r4=3D0x00001000
> nandfs_itimes() at nandfs_itimes+0x228
> scp=3D0xc0995440 rlv=3D0xc0be0284 (VOP_STRATEGY_APV+0xbc)
> rsp=3D0xd35ae768 rfp=3D0xd35ae780
> r10=3D0xd35ae9ec r9=3D0x00000000
> r8=3D0xc36e25c0 r7=3D0x00000000 r6=3D0xd35ae788 r5=3D0xc0c91a00
> r4=3D0xc0c42434
> VOP_STRATEGY_APV() at VOP_STRATEGY_APV+0x10
> scp=3D0xc0be01d8 rlv=3D0xc0ab47a8 (bufstrategy+0x68)
> rsp=3D0xd35ae784 rfp=3D0xd35ae7a8
> r6=3D0xc3ef4870 r5=3D0xcc334ce0
> r4=3D0xc3ef4870
> bufstrategy() at bufstrategy+0x10
> scp=3D0xc0ab4750 rlv=3D0xc0abaa34 (breadn_flags+0x14c)
> rsp=3D0xd35ae7ac rfp=3D0xd35ae7d0
> r5=3D0x00000000 r4=3D0xcc334ce0
> breadn_flags() at breadn_flags+0x10
> scp=3D0xc0aba8f8 rlv=3D0xc098ea58 (nandfs_bread+0x80)
> rsp=3D0xd35ae7d4 rfp=3D0xd35ae818
> r6=3D0x00000000 r5=3D0xc3fe7db0
> r4=3D0x00000000
> nandfs_bread() at nandfs_bread+0x10
> scp=3D0xc098e9e8 rlv=3D0xc0993afc (nandfs_wakeup_wait_sync+0x2e0c)
> rsp=3D0xd35ae81c rfp=3D0xd35aea9c
> r10=3D0xc3fe7db0 r8=3D0xc36e25c0
> r7=3D0x00000000 r6=3D0xd35ae9bc r5=3D0x00000000 r4=3D0xc3fda360
> nandfs_wakeup_wait_sync() at nandfs_wakeup_wait_sync+0xc9c
> scp=3D0xc099198c rlv=3D0xc0ac8390 (vfs_donmount+0x9c8)
> rsp=3D0xd35aeaa0 rfp=3D0xd35aec80
> r10=3D0xd35aec84 r9=3D0x00000000
> r8=3D0xc3fe0bc0 r7=3D0x00000000 r6=3D0x00004001 r5=3D0xd35aebbc
> r4=3D0x00000000
> vfs_donmount() at vfs_donmount+0x10
> scp=3D0xc0ac79d8 rlv=3D0xc0ac8ec4 (kernel_mount+0x90)
> rsp=3D0xd35aec84 rfp=3D0xd35aecc4
> r10=3D0xc3fe5a00 r9=3D0x00000000
> r8=3D0x00000000 r7=3D0x00004000 r6=3D0x00000000 r5=3D0xc3fda400
> r4=3D0xffffffff
> kernel_mount() at kernel_mount+0x10
> scp=3D0xc0ac8e44 rlv=3D0xc0ac99a0 (root_mount_hold+0x684)
> rsp=3D0xd35aecc8 rfp=3D0xd35aecfc
> r8=3D0xc36e2427 r7=3D0xc36e2420
> r6=3D0xc3fda400 r5=3D0xd35aecd0 r4=3D0xffffffff
> root_mount_hold() at root_mount_hold+0x3a0
> scp=3D0xc0ac96bc rlv=3D0xc0aca734 (vfs_mountroot+0xca0)
> rsp=3D0xd35aed00 rfp=3D0xd35aee2c
> r10=3D0xc36e23c0 r9=3D0xc3683000
> r8=3D0xd35aeeac r7=3D0xc3fe1000 r6=3D0xc0c29240 r5=3D0x00000000
> r4=3D0x00000000
> vfs_mountroot() at vfs_mountroot+0x10
> scp=3D0xc0ac9aa4 rlv=3D0xc09f8754 (exec_shell_imgact+0x7ac)
> rsp=3D0xd35aee30 rfp=3D0xd35aee84
> r10=3D0x00000000 r9=3D0xc3683000
> r8=3D0xd35aeeac r7=3D0x00000000 r6=3D0xc09f86fc r5=3D0xc0c1167c
> r4=3D0xc0cb9448
> exec_shell_imgact() at exec_shell_imgact+0x764
> scp=3D0xc09f870c rlv=3D0xc0a14fd4 (fork_exit+0x98)
> rsp=3D0xd35aee88 rfp=3D0xd35aeea8
> r10=3D0x00000000 r9=3D0x00000000
> r8=3D0xd35aeeac r7=3D0x00000000 r6=3D0xc09f86fc r5=3D0xc3683000
> r4=3D0xc3685000
> fork_exit() at fork_exit+0x10
> scp=3D0xc0a14f4c rlv=3D0xc0bc4ad4 (fork_trampoline+0x14)
> rsp=3D0xd35aeeac rfp=3D0x00000000
> r8=3D0x00000000 r7=3D0x00000000
> r6=3D0x00000000 r5=3D0x00000000 r4=3D0xc09f86fc
> after vfs_mountroot()
> start_init: trying /sbin/init
> sys_execve returned 0
> (da0:umass-sim0:0:0:0): PREVENT ALLOW MEDIUM REMOVAL. CDB: 1e 0 0 0 1 0
> (da0:umass-sim0:0:0:0): CAM status: Auto-Sense Retrieval Failed
> (da0:umass-sim0:0:0:0): Error 5, Unretryable error
> Dec 31 16:42:47 init: init started
> Setting hostuuid: 64f53bc5-bfde-11d3-902f-005043016d4c.
> Setting hostid: 0x2afd1481.
> No suitable dump device was found.
> Entropy harvesting: interrupts ethernet point_to_pointsha256: /kernel: No=
=20
> such file or directory
> kickstart.
> Starting file system checks:
> /dev/da0s2: FILE SYSTEM CLEAN; SKIPPING CHECKS
> /dev/da0s2: clean, 894115 free (2531 frags, 111448 blocks, 0.3% =20
> fragmentation)
> /dev/da0s1: FILESYSTEM CLEAN; SKIPPING CHECKS
> lock order reversal:
> 1st 0xc3ef4a30 snandfs (snandfs) @ =20
> /usr/src/sys/fs/nandfs/nandfs_segment.c:1199
> 2nd 0xc3ef4b98 devfs (devfs) @ /usr/src/sys/fs/nandfs/nandfs_segment.c:=
793
> KDB: stack backtrace:
> db_trace_self() at db_trace_self+0x10
> scp=3D0xc0bb462c rlv=3D0xc09334d8 (X_db_sym_numargs+0x1bc)
> rsp=3D0xd5bf8b1c rfp=3D0xd5bf8c38
> X_db_sym_numargs() at X_db_sym_numargs+0x198
> scp=3D0xc09334b4 rlv=3D0xc0a6ed5c (kdb_backtrace+0x3c)
> rsp=3D0xd5bf8c3c rfp=3D0xd5bf8c4c
> r4=3D0xc0cc7838
> kdb_backtrace() at kdb_backtrace+0x10
> scp=3D0xc0a6ed30 rlv=3D0xc0a853e8 (witness_display_spinlock+0x80)
> rsp=3D0xd5bf8c50 rfp=3D0xd5bf8c64
> r4=3D0x00000001
> witness_display_spinlock() at witness_display_spinlock+0x60
> scp=3D0xc0a853c8 rlv=3D0xc0a861c4 (witness_checkorder+0x8b4)
> rsp=3D0xd5bf8c68 rfp=3D0xd5bf8cb8
> r5=3D0xc3ef4a30 r4=3D0xc0c14ad8
> witness_checkorder() at witness_checkorder+0x10
> scp=3D0xc0a85920 rlv=3D0xc0a253bc (__lockmgr_args+0x828)
> rsp=3D0xd5bf8cbc rfp=3D0xd5bf8d18
> r10=3D0xc36e2680 r9=3D0xc3ef4c04
> r8=3D0x00080000 r7=3D0xc400dc00 r6=3D0xc3ef4b98 r5=3D0x00000000
> r4=3D0x00080000
> __lockmgr_args() at __lockmgr_args+0x10
> scp=3D0xc0a24ba4 rlv=3D0xc0abfc30 (vop_stdlock+0x48)
> rsp=3D0xd5bf8d1c rfp=3D0xd5bf8d38
> r10=3D0xc36e2680 r9=3D0xc4011880
> r8=3D0xc3fc5000 r7=3D0x00000001 r6=3D0x00000000 r5=3D0xd5bf8d68
> r4=3D0xc0ca4dfc
> vop_stdlock() at vop_stdlock+0x10
> scp=3D0xc0abfbf8 rlv=3D0xc0be0500 (VOP_LOCK1_APV+0xa0)
> rsp=3D0xd5bf8d3c rfp=3D0xd5bf8d50
> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x10
> scp=3D0xc0be0470 rlv=3D0xc098b824 (nandfs_node_create+0x1994)
> rsp=3D0xd5bf8d54 rfp=3D0xd5bf8da4
> r5=3D0xc4011880 r4=3D0xd5bf8e14
> nandfs_node_create() at nandfs_node_create+0x194c
> scp=3D0xc098b7dc rlv=3D0xc098bb90 (nandfs_node_create+0x1d00)
> rsp=3D0xd5bf8da8 rfp=3D0xd5bf8dcc
> r10=3D0xc36e2680 r9=3D0xc3fc5000
> r8=3D0xc3fc5000 r7=3D0x00000001 r6=3D0x00000000 r5=3D0xc4011880
> r4=3D0xd5bf8e14
> nandfs_node_create() at nandfs_node_create+0x1c48
> scp=3D0xc098bad8 rlv=3D0xc098c354 (nandfs_segment_constructor+0x740)
> rsp=3D0xd5bf8dd0 rfp=3D0xd5bf8e64
> r10=3D0xc36e2680 r8=3D0xc3fc5000
> r7=3D0xc36e2680 r6=3D0x00000000 r5=3D0x0003c9ca r4=3D0xd5bf8e14
> nandfs_segment_constructor() at nandfs_segment_constructor+0x10
> scp=3D0xc098bc24 rlv=3D0xc0990bc4 (nandfs_get_seg_stat+0x518)
> rsp=3D0xd5bf8e68 rfp=3D0xd5bf8e84
> r10=3D0x00000104 r9=3D0x00000000
> r8=3D0xd5bf8eac r7=3D0xc36e25c0 r6=3D0xc3fc5000 r5=3D0xc36e25c0
> r4=3D0x00000000
> nandfs_get_seg_stat() at nandfs_get_seg_stat+0x49c
> scp=3D0xc0990b48 rlv=3D0xc0a14fd4 (fork_exit+0x98)
> rsp=3D0xd5bf8e88 rfp=3D0xd5bf8ea8
> r6=3D0xc0990b38 r5=3D0xc0cb74c4
> r4=3D0xc400dc00
> fork_exit() at fork_exit+0x10
> scp=3D0xc0a14f4c rlv=3D0xc0bc4ad4 (fork_trampoline+0x14)
> rsp=3D0xd5bf8eac rfp=3D0x00000000
> r8=3D0xc0cb77b0 r7=3D0x00000000
> r6=3D0x00000000 r5=3D0xc36e25c0 r4=3D0xc0990b38
> Mounting local file systems:.
> Writing entropy file:.
> Setting hostname: sh10.klop.ws.
> Starting Network: lo0 mge0.
> lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=3D600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet 127.0.0.1 netmask 0xff000000
> mge0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric mge0: l=
ink =20
> state changed to UP
> 0 mtu 1500
> options=3D8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
> ether 00:50:43:01:6d:4c
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
> Starting devd.
> Starting dhclient.
> DHCPDISCOVER on mge0 to 255.255.255.255 port 67 interval 6
> DHCPOFFER from 192.168.1.254
> DHCPREQUEST on mge0 to 255.255.255.255 port 67
> DHCPACK from 192.168.1.254
> bound to 192.168.1.35 -- renewal in 43200 seconds.
> Creating and/or trimming log files.
> Starting syslogd.
> realpath: /dev/dumpdev: No such file or directory
> /etc/rc: WARNING: Dump device does not exist. Savecore not run.
> ELF ldconfig path: /lib /usr/lib /usr/lib/compat
> Clearing /tmp (X related).
> Updating motd:.
> Starting ntpd.
> Performing sanity check on sshd configuration.
> Starting sshd.
> Starting cron.
>=20
> Mon Dec 31 16:43:48 UTC 2012
>=20
> FreeBSD/arm (sh10.klop.ws) (ttyu0)
>=20
> login:
>=20
>=20
> Regards,
> Ronald.
--1xJJiECQTZ8+YlWm
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (FreeBSD)
iQIcBAEBAgAGBQJQ4csQAAoJEJDCuSvBvK1BdbUQAKAHLHj8AzpQrebU9JlZbGLG
5LbIus9Gytz/sdayIAdYU5+GStYFagbr0c9gvY4d+SmZtRyQ7gcM08Jtdq+pYlWI
ANHM2wHWIVAiDEQeyNIygjUCOxwC77A+di/mNRvuQHo6rX7LCPfEYi7wUwEF3Hbv
H2VUG7SgnOsIcIEbyHDW4IOVi+HMrESEuZ/ROlBOpz0MVNESblbBdFahyaoslQrk
zls5K5vZPYNgzx9/9b0/0v42MlBZ9TkLopk6zKXtVGd/3nn2IRdoUhw1b9/pc5y2
pUmCUovVA4uV6oaZzFUAYNv6eV1lChA3o5lQwNfFJBDmjUp3KIvF5VaWj27Q/6eb
/XQ1J4Q6ylvO1Va/j5/YAbFjoVWRZlJkIEyE7TPz5SanptBlVCAd2F03nIwkk558
+/NmIxCsg6pf3IGrg5mOXasbbjYbwA1ftsrl8czwa2rOcmUNY388VsmUIs+O97Wu
0Rb/xvNhmwZH8aLd5SvqVmzefYcoAX7IG9pRoB4If07pt01hC/HaCpY2DuSzLPjA
O4OXcJ8BK+7sNJfMJdlweHHclPgLKvQKNllgRom4jEJfyUZNLPazGPqGIrRAVgEF
qZElnVfSzI5COBGUT/NJ5crcO2PgwpMWa+vDtfn/+zlm9UL2a8Omi2FYpSIRzzbn
xONMIUJBnSO/8rKCvpU1
=y5WL
-----END PGP SIGNATURE-----
--1xJJiECQTZ8+YlWm--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20121231172744.GQ82219>