Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 21 Sep 2013 08:08:37 -0400
From:      Randall Stewart <rrs@lakerest.net>
To:        David Cheney <david.cheney@canonical.com>
Cc:        freebsd-arm@freebsd.org, Ronald Klop <ronald-freebsd8@klop.yi.org>
Subject:   Re: freebsd on beaglebone black, lock reversal panic shortly after first boot
Message-ID:  <C6DAB5EB-9C3B-4AD3-B31F-2616626A3DC9@lakerest.net>
In-Reply-To: <CAHPQsETr9tXQ99atJBFyv1epE-cC9fn6EkfpC0zMt2u%2B1UkpVQ@mail.gmail.com>
References:  <CAHPQsES33YNYVGFF=dFLinaNQex2wrAm%2B8e6qHip1-HdaAg26Q@mail.gmail.com> <op.w3q9gg2g8527sy@212-182-167-131.ip.telfort.nl> <CAHPQsETr9tXQ99atJBFyv1epE-cC9fn6EkfpC0zMt2u%2B1UkpVQ@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
David:

I believe you are seeing a reported Lock Order Reversal since you
are running with a WITNESS kernel.

This particular one I recognize and see on other platforms as well.
Its one of those ones that has never been either fixed or marked as
"blessed".

If you want to get rid of these (and increase performance) go in to your
kernel configuration and comment out the two lines that mention "option =
WITNESS*"
(just put a # in front of them).

That will disable witness and speed up the kernel ;-)

Of course if you are doing kernel debugging you probably want WITNESS on =
since it
does give lots of good information.. that particular one is just one we =
have to ignore ;-D

Best wishes

R
On Sep 21, 2013, at 5:53 AM, David Cheney wrote:

> Ronald,
>=20
> Thanks for your reply. I'm sorry, I am inexperienced with this sort of
> thing, the machine has not crash, in fact it still building a port
> now. =46rom my linux background, if I saw that much output, the host
> would be kerput.
>=20
> Cheers
>=20
> Dave
>=20
>=20
> On Sat, Sep 21, 2013 at 7:46 PM, Ronald Klop
> <ronald-freebsd8@klop.yi.org> wrote:
>> On Sat, 21 Sep 2013 00:45:48 +0200, David Cheney
>> <david.cheney@canonical.com> wrote:
>>=20
>>> Hello,
>>>=20
>>> I've been trying to replace my old RPi freebsd system, which I =
haven't
>>> been able to get a working build for months on, with a Beaglebone
>>> Black. This was the message I received after booting the card for =
the
>>> first time and trying to portsnap.
>>>=20
>>> Cheers
>>>=20
>>> Dave
>>=20
>>=20
>> In the subject you mention it is a panic, but that does not show from =
the
>> output.
>> Is your machine hanging? Or does it reboot after the panic? Or can =
you just
>> work on.
>> There are a lot of lock order reversals (LORs) which are harmless for =
the
>> working of the machine. I don't know about this one.
>>=20
>> Ronald.
>>=20
>>>=20
>>> ---
>>>=20
>>>=20
>>> FreeBSD/arm (beaglebone) (ttyu0)
>>>=20
>>> login: root
>>> Sep 20 21:57:34 beaglebone login: ROOT LOGIN (root) ON ttyu0
>>> FreeBSD 10.0-ALPHA2 (BEAGLEBONE) #0 r255662: Fri Sep 20 23:04:55 EST =
2013
>>>=20
>>> Welcome to FreeBSD!
>>>=20
>>> Before seeking technical support, please use the following =
resources:
>>>=20
>>> o  Security advisories and updated errata information for all =
releases are
>>>   at http://www.FreeBSD.org/releases/ - always consult the ERRATA =
section
>>>   for your release first as it's updated frequently.
>>>=20
>>> o  The Handbook and FAQ documents are at http://www.FreeBSD.org/ =
and,
>>>   along with the mailing lists, can be searched by going to
>>>   http://www.FreeBSD.org/search/.  If the doc package has been =
installed
>>>   (or fetched via pkg install lang-freebsd-doc, where lang is the
>>>   2-letter language code, e.g. en), they are also available =
formatted
>>>   in /usr/local/share/doc/freebsd.
>>>=20
>>> If you still have a question or problem, please take the output of
>>> `uname -a', along with any relevant error messages, and email it
>>> as a question to the questions@FreeBSD.org mailing list.  If you are
>>> unfamiliar with FreeBSD's directory layout, please refer to the =
hier(7)
>>> manual page.  If you are not familiar with manual pages, type `man =
man'.
>>>=20
>>> Edit /etc/motd to change this login announcement.
>>>=20
>>> root@beaglebone:~ # df -h
>>> Filesystem        Size    Used   Avail Capacity  Mounted on
>>> /dev/mmcsd0s2a    3.5G    396M    2.9G    12%    /
>>> devfs             1.0k    1.0k      0B   100%    /dev
>>> /dev/mmcsd0s1       2M    693k    1.3M    34%    /boot/msdos
>>> /dev/md0           28M     24k     26M     0%    /tmp
>>> /dev/md1           14M     60k     13M     0%    /var/log
>>> /dev/md2          4.4M    8.0k    4.0M     0%    /var/tmp
>>> root@beaglebone:~ # portsnap fetch && portsnap extract
>>> Looking up portsnap.FreeBSD.org mirrors... 7 mirrors found.
>>> Fetching public key from ec2-ap-southeast-2.portsnap.freebsd.org... =
done.
>>> Fetching snapshot tag from =
ec2-ap-southeast-2.portsnap.freebsd.org...
>>> done.
>>> Fetching snapshot metadata... done.
>>> Fetching snapshot generated at Fri Sep 20 00:08:00 UTC 2013:
>>> f88e945869b296b518253a080d33b8c988a382fd82586c100% of   69 MB 1473 =
kBps
>>> 00m48s
>>> Extracting snapshot... lock order reversal:
>>> 1st 0xcd1299d0 bufwait (bufwait) @
>>> /root/crochet-freebsd/src/sys/kern/vfs_bio.c:3059
>>> 2nd 0xc2a8f600 dirhash (dirhash) @
>>> /root/crochet-freebsd/src/sys/ufs/ufs/ufs_dirhash.c:284
>>> KDB: stack backtrace:
>>> db_trace_self() at db_trace_self
>>>         pc =3D 0xc052ecf4  lr =3D 0xc022dcd0 =
(db_trace_self_wrapper+0x30)
>>>         sp =3D 0xdd6ff7e0  fp =3D 0xdd6ff8f8
>>>        r10 =3D 0xcd1299d0
>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x30
>>>         pc =3D 0xc022dcd0  lr =3D 0xc038e9e0 (kdb_backtrace+0x38)
>>>         sp =3D 0xdd6ff900  fp =3D 0xdd6ff908
>>>         r4 =3D 0xc066a484  r5 =3D 0xc05ae078
>>>         r6 =3D 0xc0592f71  r7 =3D 0xc0598518
>>> kdb_backtrace() at kdb_backtrace+0x38
>>>         pc =3D 0xc038e9e0  lr =3D 0xc03a8b5c =
(witness_checkorder+0xda0)
>>>         sp =3D 0xdd6ff910  fp =3D 0xdd6ff960
>>>         r4 =3D 0xc05ae43b
>>> witness_checkorder() at witness_checkorder+0xda0
>>>         pc =3D 0xc03a8b5c  lr =3D 0xc035ff8c (_sx_xlock+0x84)
>>>         sp =3D 0xdd6ff968  fp =3D 0xdd6ff990
>>>         r4 =3D 0x0000011c  r5 =3D 0xc05ae075
>>>         r6 =3D 0xc2a8f610  r7 =3D 0xc2a8f600
>>>         r8 =3D 0x00000000  r9 =3D 0xc2c067e0
>>>        r10 =3D 0x000001b4
>>> _sx_xlock() at _sx_xlock+0x84
>>>         pc =3D 0xc035ff8c  lr =3D 0xc04ebfa8 (ufsdirhash_add+0x34)
>>>         sp =3D 0xdd6ff998  fp =3D 0xdd6ff9b0
>>>         r4 =3D 0xc2a8f600  r5 =3D 0xc2ab1e80
>>>         r6 =3D 0xdd6ffa48  r7 =3D 0x00000001
>>>         r8 =3D 0x0000084c
>>> ufsdirhash_add() at ufsdirhash_add+0x34
>>>         pc =3D 0xc04ebfa8  lr =3D 0xc04ee88c (ufs_direnter+0x45c)
>>>         sp =3D 0xdd6ff9b8  fp =3D 0xdd6ffa20
>>>         r4 =3D 0xc2ab1e80  r5 =3D 0xcd3af84c
>>>         r6 =3D 0xdd6ffa48  r7 =3D 0x00000001
>>>         r8 =3D 0x0000004c  r9 =3D 0xc2c067e0
>>> ufs_direnter() at ufs_direnter+0x45c
>>>         pc =3D 0xc04ee88c  lr =3D 0xc04f7e04 (ufs_makeinode+0x3f4)
>>>         sp =3D 0xdd6ffa28  fp =3D 0xdd6ffb88
>>>         r4 =3D 0xdd6ffa48  r5 =3D 0xc2ab1180
>>>         r6 =3D 0xdd6ffd48  r7 =3D 0x00000000
>>>         r8 =3D 0xc2c067e0  r9 =3D 0xdd6ffd30
>>>        r10 =3D 0x000081a4
>>> ufs_makeinode() at ufs_makeinode+0x3f4
>>>         pc =3D 0xc04f7e04  lr =3D 0xc04f4268 (ufs_create+0x24)
>>>         sp =3D 0xdd6ffb90  fp =3D 0xdd6ffb90
>>>         r4 =3D 0xdd6ffc60  r5 =3D 0xc064eba0
>>>         r6 =3D 0x00000000  r7 =3D 0x00100a02
>>>         r8 =3D 0x00000000  r9 =3D 0xdd6ffd30
>>>        r10 =3D 0xdd6ffd50
>>> ufs_create() at ufs_create+0x24
>>>         pc =3D 0xc04f4268  lr =3D 0xc0552268 (VOP_CREATE_APV+0xd0)
>>>         sp =3D 0xdd6ffb98  fp =3D 0xdd6ffba8
>>> VOP_CREATE_APV() at VOP_CREATE_APV+0xd0
>>>         pc =3D 0xc0552268  lr =3D 0xc0408540 (vn_open_cred+0x278)
>>>         sp =3D 0xdd6ffbb0  fp =3D 0xdd6ffc90
>>>         r4 =3D 0xdd6ffce0  r5 =3D 0xdd6ffd30
>>>         r6 =3D 0xc2c067e0
>>> vn_open_cred() at vn_open_cred+0x278
>>>         pc =3D 0xc0408540  lr =3D 0xc04082c0 (vn_open+0x24)
>>>         sp =3D 0xdd6ffc98  fp =3D 0xdd6ffca0
>>>         r4 =3D 0xc2a2d640  r5 =3D 0xc2a76000
>>>         r6 =3D 0xdd6ffce0  r7 =3D 0x00000012
>>>         r8 =3D 0x00000000  r9 =3D 0x000500cf
>>>        r10 =3D 0xdd6ffcd0
>>> vn_open() at vn_open+0x24
>>>         pc =3D 0xc04082c0  lr =3D 0xc040187c (kern_openat+0x24c)
>>>         sp =3D 0xdd6ffca8  fp =3D 0xdd6ffda8
>>> kern_openat() at kern_openat+0x24c
>>>         pc =3D 0xc040187c  lr =3D 0xc04015c4 (sys_open+0x28)
>>>         sp =3D 0xdd6ffdb0  fp =3D 0xdd6ffdb8
>>>         r4 =3D 0xc2a2d640  r5 =3D 0x00000000
>>>         r6 =3D 0x20cc1198  r7 =3D 0x00000000
>>>         r8 =3D 0xdd6ffe10  r9 =3D 0xc29f6c80
>>>        r10 =3D 0x20c24080
>>> sys_open() at sys_open+0x28
>>>         pc =3D 0xc04015c4  lr =3D 0xc0540764 (swi_handler+0x284)
>>>         sp =3D 0xdd6ffdc0  fp =3D 0xdd6ffe58
>>> swi_handler() at swi_handler+0x284
>>>         pc =3D 0xc0540764  lr =3D 0xc0530350 (swi_entry+0x2c)
>>>         sp =3D 0xdd6ffe60  fp =3D 0xbfffe960
>>>         r4 =3D 0x20c0a200  r5 =3D 0x000001a4
>>>         r6 =3D 0x20cc1198  r7 =3D 0x00000005
>>>         r8 =3D 0x20c0a200  r9 =3D 0x00000001
>>> swi_entry() at swi_entry+0x2c
>>>         pc =3D 0xc0530350  lr =3D 0xc0530350 (swi_entry+0x2c)
>>>         sp =3D 0xdd6ffe60  fp =3D 0xbfffe960
>>> Unable to unwind further
>>> lock order reversal:
>>> 1st 0xc2c06814 ufs (ufs) @
>>> /root/crochet-freebsd/src/sys/kern/vfs_subr.c:2099
>>> 2nd 0xcd1299d0 bufwait (bufwait) @
>>> /root/crochet-freebsd/src/sys/ufs/ffs/ffs_vnops.c:262
>>> 3rd 0xc2cb9154 ufs (ufs) @
>>> /root/crochet-freebsd/src/sys/kern/vfs_subr.c:2099
>>> KDB: stack backtrace:
>>> db_trace_self() at db_trace_self
>>>         pc =3D 0xc052ecf4  lr =3D 0xc022dcd0 =
(db_trace_self_wrapper+0x30)
>>>         sp =3D 0xdd6ff578  fp =3D 0xdd6ff690
>>>        r10 =3D 0xc05adb30
>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x30
>>>         pc =3D 0xc022dcd0  lr =3D 0xc038e9e0 (kdb_backtrace+0x38)
>>>         sp =3D 0xdd6ff698  fp =3D 0xdd6ff6a0
>>>         r4 =3D 0xc066a484  r5 =3D 0xc0592f71
>>>         r6 =3D 0xc059b3d0  r7 =3D 0xc05adb33
>>> kdb_backtrace() at kdb_backtrace+0x38
>>>         pc =3D 0xc038e9e0  lr =3D 0xc03a8b5c =
(witness_checkorder+0xda0)
>>>         sp =3D 0xdd6ff6a8  fp =3D 0xdd6ff6f8
>>>         r4 =3D 0xc0581d1c
>>> witness_checkorder() at witness_checkorder+0xda0
>>>         pc =3D 0xc03a8b5c  lr =3D 0xc033c5fc (__lockmgr_args+0x698)
>>>         sp =3D 0xdd6ff700  fp =3D 0xdd6ff768
>>>         r4 =3D 0x00080100  r5 =3D 0x00000833
>>>         r6 =3D 0xc2cb9154  r7 =3D 0x00080000
>>>         r8 =3D 0x00000100  r9 =3D 0x00000000
>>>        r10 =3D 0xc059b3cd
>>> __lockmgr_args() at __lockmgr_args+0x698
>>>         pc =3D 0xc033c5fc  lr =3D 0xc04e678c (ffs_lock+0x80)
>>>         sp =3D 0xdd6ff770  fp =3D 0xdd6ff798
>>>         r4 =3D 0xdd6ff7b8  r5 =3D 0x00080100
>>>         r6 =3D 0xc2cb9120  r7 =3D 0xc2cb9154
>>>         r8 =3D 0xc2cb9174  r9 =3D 0x00000000
>>>        r10 =3D 0xc059b3cd
>>> ffs_lock() at ffs_lock+0x80
>>>         pc =3D 0xc04e678c  lr =3D 0xc0554118 (VOP_LOCK1_APV+0xd8)
>>>         sp =3D 0xdd6ff7a0  fp =3D 0xdd6ff7b0
>>>         r4 =3D 0xdd6ff7b8  r5 =3D 0xc064e690
>>>         r6 =3D 0x00000000  r7 =3D 0x00000000
>>>         r8 =3D 0xc2a2d640  r9 =3D 0x00000833
>>> VOP_LOCK1_APV() at VOP_LOCK1_APV+0xd8
>>>         pc =3D 0xc0554118  lr =3D 0xc0408cbc (_vn_lock+0x78)
>>>         sp =3D 0xdd6ff7b8  fp =3D 0xdd6ff7e8
>>>         r4 =3D 0xc2cb9120  r5 =3D 0x00080100
>>>         r6 =3D 0x00080100
>>> _vn_lock() at _vn_lock+0x78
>>>         pc =3D 0xc0408cbc  lr =3D 0xc03f97e8 (vget+0x60)
>>>         sp =3D 0xdd6ff7f0  fp =3D 0xdd6ff810
>>>         r4 =3D 0xc2cb9120  r5 =3D 0x00000000
>>>         r6 =3D 0x00080100  r7 =3D 0x00000000
>>>         r8 =3D 0xc2a2d640  r9 =3D 0xc059a607
>>>        r10 =3D 0x00000000
>>> vget() at vget+0x60
>>>         pc =3D 0xc03f97e8  lr =3D 0xc03ede5c (vfs_hash_get+0xe4)
>>>         sp =3D 0xdd6ff818  fp =3D 0xdd6ff848
>>>         r4 =3D 0xc2a0b2b0  r5 =3D 0x00000000
>>>         r6 =3D 0x0002787d  r7 =3D 0x00000000
>>>         r8 =3D 0xc2cb9120
>>> vfs_hash_get() at vfs_hash_get+0xe4
>>>         pc =3D 0xc03ede5c  lr =3D 0xc04e17f4 (ffs_vgetf+0x3c)
>>>         sp =3D 0xdd6ff850  fp =3D 0xdd6ff8a0
>>>         r4 =3D 0x0002787d  r5 =3D 0xc05aa74b
>>>         r6 =3D 0xc2bb13c0  r7 =3D 0xdd6ff908
>>>         r8 =3D 0xc2bb13c0  r9 =3D 0x00080000
>>>        r10 =3D 0xc2a0b2b0
>>> ffs_vgetf() at ffs_vgetf+0x3c
>>>         pc =3D 0xc04e17f4  lr =3D 0xc04da468 =
(softdep_sync_buf+0x974)
>>>         sp =3D 0xdd6ff8a8  fp =3D 0xdd6ff928
>>>         r4 =3D 0x0002787d  r5 =3D 0xc05aa74b
>>>         r6 =3D 0xc2bb13c0  r7 =3D 0xc05aa74b
>>>         r8 =3D 0xc2bb13c0  r9 =3D 0x0000088b
>>>        r10 =3D 0x0002787d
>>> softdep_sync_buf() at softdep_sync_buf+0x974
>>>         pc =3D 0xc04da468  lr =3D 0xc04e7560 (ffs_syncvnode+0x2bc)
>>>         sp =3D 0xdd6ff930  fp =3D 0xdd6ff980
>>>         r4 =3D 0x00000000  r5 =3D 0xcd188d48
>>>         r6 =3D 0x00060200  r7 =3D 0xc2c067e0
>>>         r8 =3D 0xc05adb30  r9 =3D 0xcd1299d0
>>>        r10 =3D 0xcd129978
>>> ffs_syncvnode() at ffs_syncvnode+0x2bc
>>>         pc =3D 0xc04e7560  lr =3D 0xc04e6650 (ffs_fsync+0x20)
>>>         sp =3D 0xdd6ff988  fp =3D 0xdd6ff998
>>>         r4 =3D 0xdd6ff9f0  r5 =3D 0xc2c067e0
>>>         r6 =3D 0x00000000  r7 =3D 0x00000200
>>>         r8 =3D 0xc2a2d640  r9 =3D 0xc2c067e0
>>>        r10 =3D 0xc33445a0
>>> ffs_fsync() at ffs_fsync+0x20
>>>         pc =3D 0xc04e6650  lr =3D 0xc05534b8 (VOP_FSYNC_APV+0xd0)
>>>         sp =3D 0xdd6ff9a0  fp =3D 0xdd6ff9b0
>>>         r4 =3D 0xdd6ff9f0  r5 =3D 0xc064e690
>>>         r6 =3D 0x00000000  r7 =3D 0x00000200
>>> VOP_FSYNC_APV() at VOP_FSYNC_APV+0xd0
>>>         pc =3D 0xc05534b8  lr =3D 0xc04eeb54 (ufs_direnter+0x724)
>>>         sp =3D 0xdd6ff9b8  fp =3D 0xdd6ffa20
>>>         r4 =3D 0xc33445a0  r5 =3D 0x00000000
>>>         r6 =3D 0xcd356e00
>>> ufs_direnter() at ufs_direnter+0x724
>>>         pc =3D 0xc04eeb54  lr =3D 0xc04f7e04 (ufs_makeinode+0x3f4)
>>>         sp =3D 0xdd6ffa28  fp =3D 0xdd6ffb88
>>>         r4 =3D 0xdd6ffa48  r5 =3D 0xc332fa00
>>>         r6 =3D 0xdd6ffd48  r7 =3D 0x00000000
>>>         r8 =3D 0xc2c067e0  r9 =3D 0xdd6ffd30
>>>        r10 =3D 0x000081a4
>>> ufs_makeinode() at ufs_makeinode+0x3f4
>>>         pc =3D 0xc04f7e04  lr =3D 0xc04f4268 (ufs_create+0x24)
>>>         sp =3D 0xdd6ffb90  fp =3D 0xdd6ffb90
>>>         r4 =3D 0xdd6ffc60  r5 =3D 0xc064eba0
>>>         r6 =3D 0x00000000  r7 =3D 0x00100a02
>>>         r8 =3D 0x00000000  r9 =3D 0xdd6ffd30
>>>        r10 =3D 0xdd6ffd50
>>> ufs_create() at ufs_create+0x24
>>>         pc =3D 0xc04f4268  lr =3D 0xc0552268 (VOP_CREATE_APV+0xd0)
>>>         sp =3D 0xdd6ffb98  fp =3D 0xdd6ffba8
>>> VOP_CREATE_APV() at VOP_CREATE_APV+0xd0
>>>         pc =3D 0xc0552268  lr =3D 0xc0408540 (vn_open_cred+0x278)
>>>         sp =3D 0xdd6ffbb0  fp =3D 0xdd6ffc90
>>>         r4 =3D 0xdd6ffce0  r5 =3D 0xdd6ffd30
>>>         r6 =3D 0xc2c067e0
>>> vn_open_cred() at vn_open_cred+0x278
>>>         pc =3D 0xc0408540  lr =3D 0xc04082c0 (vn_open+0x24)
>>>         sp =3D 0xdd6ffc98  fp =3D 0xdd6ffca0
>>>         r4 =3D 0xc2a2d640  r5 =3D 0xc2a76000
>>>         r6 =3D 0xdd6ffce0  r7 =3D 0x00000012
>>>         r8 =3D 0x00000000  r9 =3D 0x000500cf
>>>        r10 =3D 0xdd6ffcd0
>>> vn_open() at vn_open+0x24
>>>         pc =3D 0xc04082c0  lr =3D 0xc040187c (kern_openat+0x24c)
>>>         sp =3D 0xdd6ffca8  fp =3D 0xdd6ffda8
>>> kern_openat() at kern_openat+0x24c
>>>         pc =3D 0xc040187c  lr =3D 0xc04015c4 (sys_open+0x28)
>>>         sp =3D 0xdd6ffdb0  fp =3D 0xdd6ffdb8
>>>         r4 =3D 0xc2a2d640  r5 =3D 0x00000000
>>>         r6 =3D 0x20cc1198  r7 =3D 0x00000000
>>>         r8 =3D 0xdd6ffe10  r9 =3D 0xc29f6c80
>>>        r10 =3D 0x20c24080
>>> sys_open() at sys_open+0x28
>>>         pc =3D 0xc04015c4  lr =3D 0xc0540764 (swi_handler+0x284)
>>>         sp =3D 0xdd6ffdc0  fp =3D 0xdd6ffe58
>>> swi_handler() at swi_handler+0x284
>>>         pc =3D 0xc0540764  lr =3D 0xc0530350 (swi_entry+0x2c)
>>>         sp =3D 0xdd6ffe60  fp =3D 0xbfffe960
>>>         r4 =3D 0x20c0a200  r5 =3D 0x000001a4
>>>         r6 =3D 0x20cc1198  r7 =3D 0x00000
>>> _______________________________________________
>>> freebsd-arm@freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-arm
>>> To unsubscribe, send any mail to =
"freebsd-arm-unsubscribe@freebsd.org"
>>=20
>> _______________________________________________
>> freebsd-arm@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-arm
>> To unsubscribe, send any mail to =
"freebsd-arm-unsubscribe@freebsd.org"
> _______________________________________________
> freebsd-arm@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-arm
> To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org"
>=20

------------------------------
Randall Stewart
803-317-4952 (cell)




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C6DAB5EB-9C3B-4AD3-B31F-2616626A3DC9>