From owner-freebsd-arm@FreeBSD.ORG Sat Sep 21 12:08:43 2013 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id CCA8F1E8 for ; Sat, 21 Sep 2013 12:08:43 +0000 (UTC) (envelope-from rrs@lakerest.net) Received: from lakerest.net (lakerest.net [70.155.160.98]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id DBD892BF2 for ; Sat, 21 Sep 2013 12:08:42 +0000 (UTC) Received: from [10.1.1.103] (bsd4.lakerest.net [70.155.160.102]) (authenticated bits=0) by lakerest.net (8.14.4/8.14.3) with ESMTP id r8LC8bHD081753 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sat, 21 Sep 2013 08:08:38 -0400 (EDT) (envelope-from rrs@lakerest.net) Subject: Re: freebsd on beaglebone black, lock reversal panic shortly after first boot Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Randall Stewart In-Reply-To: Date: Sat, 21 Sep 2013 08:08:37 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: David Cheney X-Mailer: Apple Mail (2.1283) Cc: freebsd-arm@freebsd.org, Ronald Klop X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Sep 2013 12:08:43 -0000 David: I believe you are seeing a reported Lock Order Reversal since you are running with a WITNESS kernel. This particular one I recognize and see on other platforms as well. Its one of those ones that has never been either fixed or marked as "blessed". If you want to get rid of these (and increase performance) go in to your kernel configuration and comment out the two lines that mention "option = WITNESS*" (just put a # in front of them). That will disable witness and speed up the kernel ;-) Of course if you are doing kernel debugging you probably want WITNESS on = since it does give lots of good information.. that particular one is just one we = have to ignore ;-D Best wishes R On Sep 21, 2013, at 5:53 AM, David Cheney wrote: > Ronald, >=20 > Thanks for your reply. I'm sorry, I am inexperienced with this sort of > thing, the machine has not crash, in fact it still building a port > now. =46rom my linux background, if I saw that much output, the host > would be kerput. >=20 > Cheers >=20 > Dave >=20 >=20 > On Sat, Sep 21, 2013 at 7:46 PM, Ronald Klop > wrote: >> On Sat, 21 Sep 2013 00:45:48 +0200, David Cheney >> wrote: >>=20 >>> Hello, >>>=20 >>> I've been trying to replace my old RPi freebsd system, which I = haven't >>> been able to get a working build for months on, with a Beaglebone >>> Black. This was the message I received after booting the card for = the >>> first time and trying to portsnap. >>>=20 >>> Cheers >>>=20 >>> Dave >>=20 >>=20 >> In the subject you mention it is a panic, but that does not show from = the >> output. >> Is your machine hanging? Or does it reboot after the panic? Or can = you just >> work on. >> There are a lot of lock order reversals (LORs) which are harmless for = the >> working of the machine. I don't know about this one. >>=20 >> Ronald. >>=20 >>>=20 >>> --- >>>=20 >>>=20 >>> FreeBSD/arm (beaglebone) (ttyu0) >>>=20 >>> login: root >>> Sep 20 21:57:34 beaglebone login: ROOT LOGIN (root) ON ttyu0 >>> FreeBSD 10.0-ALPHA2 (BEAGLEBONE) #0 r255662: Fri Sep 20 23:04:55 EST = 2013 >>>=20 >>> Welcome to FreeBSD! >>>=20 >>> Before seeking technical support, please use the following = resources: >>>=20 >>> o Security advisories and updated errata information for all = releases are >>> at http://www.FreeBSD.org/releases/ - always consult the ERRATA = section >>> for your release first as it's updated frequently. >>>=20 >>> o The Handbook and FAQ documents are at http://www.FreeBSD.org/ = and, >>> along with the mailing lists, can be searched by going to >>> http://www.FreeBSD.org/search/. If the doc package has been = installed >>> (or fetched via pkg install lang-freebsd-doc, where lang is the >>> 2-letter language code, e.g. en), they are also available = formatted >>> in /usr/local/share/doc/freebsd. >>>=20 >>> If you still have a question or problem, please take the output of >>> `uname -a', along with any relevant error messages, and email it >>> as a question to the questions@FreeBSD.org mailing list. If you are >>> unfamiliar with FreeBSD's directory layout, please refer to the = hier(7) >>> manual page. If you are not familiar with manual pages, type `man = man'. >>>=20 >>> Edit /etc/motd to change this login announcement. >>>=20 >>> root@beaglebone:~ # df -h >>> Filesystem Size Used Avail Capacity Mounted on >>> /dev/mmcsd0s2a 3.5G 396M 2.9G 12% / >>> devfs 1.0k 1.0k 0B 100% /dev >>> /dev/mmcsd0s1 2M 693k 1.3M 34% /boot/msdos >>> /dev/md0 28M 24k 26M 0% /tmp >>> /dev/md1 14M 60k 13M 0% /var/log >>> /dev/md2 4.4M 8.0k 4.0M 0% /var/tmp >>> root@beaglebone:~ # portsnap fetch && portsnap extract >>> Looking up portsnap.FreeBSD.org mirrors... 7 mirrors found. >>> Fetching public key from ec2-ap-southeast-2.portsnap.freebsd.org... = done. >>> Fetching snapshot tag from = ec2-ap-southeast-2.portsnap.freebsd.org... >>> done. >>> Fetching snapshot metadata... done. >>> Fetching snapshot generated at Fri Sep 20 00:08:00 UTC 2013: >>> f88e945869b296b518253a080d33b8c988a382fd82586c100% of 69 MB 1473 = kBps >>> 00m48s >>> Extracting snapshot... lock order reversal: >>> 1st 0xcd1299d0 bufwait (bufwait) @ >>> /root/crochet-freebsd/src/sys/kern/vfs_bio.c:3059 >>> 2nd 0xc2a8f600 dirhash (dirhash) @ >>> /root/crochet-freebsd/src/sys/ufs/ufs/ufs_dirhash.c:284 >>> KDB: stack backtrace: >>> db_trace_self() at db_trace_self >>> pc =3D 0xc052ecf4 lr =3D 0xc022dcd0 = (db_trace_self_wrapper+0x30) >>> sp =3D 0xdd6ff7e0 fp =3D 0xdd6ff8f8 >>> r10 =3D 0xcd1299d0 >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x30 >>> pc =3D 0xc022dcd0 lr =3D 0xc038e9e0 (kdb_backtrace+0x38) >>> sp =3D 0xdd6ff900 fp =3D 0xdd6ff908 >>> r4 =3D 0xc066a484 r5 =3D 0xc05ae078 >>> r6 =3D 0xc0592f71 r7 =3D 0xc0598518 >>> kdb_backtrace() at kdb_backtrace+0x38 >>> pc =3D 0xc038e9e0 lr =3D 0xc03a8b5c = (witness_checkorder+0xda0) >>> sp =3D 0xdd6ff910 fp =3D 0xdd6ff960 >>> r4 =3D 0xc05ae43b >>> witness_checkorder() at witness_checkorder+0xda0 >>> pc =3D 0xc03a8b5c lr =3D 0xc035ff8c (_sx_xlock+0x84) >>> sp =3D 0xdd6ff968 fp =3D 0xdd6ff990 >>> r4 =3D 0x0000011c r5 =3D 0xc05ae075 >>> r6 =3D 0xc2a8f610 r7 =3D 0xc2a8f600 >>> r8 =3D 0x00000000 r9 =3D 0xc2c067e0 >>> r10 =3D 0x000001b4 >>> _sx_xlock() at _sx_xlock+0x84 >>> pc =3D 0xc035ff8c lr =3D 0xc04ebfa8 (ufsdirhash_add+0x34) >>> sp =3D 0xdd6ff998 fp =3D 0xdd6ff9b0 >>> r4 =3D 0xc2a8f600 r5 =3D 0xc2ab1e80 >>> r6 =3D 0xdd6ffa48 r7 =3D 0x00000001 >>> r8 =3D 0x0000084c >>> ufsdirhash_add() at ufsdirhash_add+0x34 >>> pc =3D 0xc04ebfa8 lr =3D 0xc04ee88c (ufs_direnter+0x45c) >>> sp =3D 0xdd6ff9b8 fp =3D 0xdd6ffa20 >>> r4 =3D 0xc2ab1e80 r5 =3D 0xcd3af84c >>> r6 =3D 0xdd6ffa48 r7 =3D 0x00000001 >>> r8 =3D 0x0000004c r9 =3D 0xc2c067e0 >>> ufs_direnter() at ufs_direnter+0x45c >>> pc =3D 0xc04ee88c lr =3D 0xc04f7e04 (ufs_makeinode+0x3f4) >>> sp =3D 0xdd6ffa28 fp =3D 0xdd6ffb88 >>> r4 =3D 0xdd6ffa48 r5 =3D 0xc2ab1180 >>> r6 =3D 0xdd6ffd48 r7 =3D 0x00000000 >>> r8 =3D 0xc2c067e0 r9 =3D 0xdd6ffd30 >>> r10 =3D 0x000081a4 >>> ufs_makeinode() at ufs_makeinode+0x3f4 >>> pc =3D 0xc04f7e04 lr =3D 0xc04f4268 (ufs_create+0x24) >>> sp =3D 0xdd6ffb90 fp =3D 0xdd6ffb90 >>> r4 =3D 0xdd6ffc60 r5 =3D 0xc064eba0 >>> r6 =3D 0x00000000 r7 =3D 0x00100a02 >>> r8 =3D 0x00000000 r9 =3D 0xdd6ffd30 >>> r10 =3D 0xdd6ffd50 >>> ufs_create() at ufs_create+0x24 >>> pc =3D 0xc04f4268 lr =3D 0xc0552268 (VOP_CREATE_APV+0xd0) >>> sp =3D 0xdd6ffb98 fp =3D 0xdd6ffba8 >>> VOP_CREATE_APV() at VOP_CREATE_APV+0xd0 >>> pc =3D 0xc0552268 lr =3D 0xc0408540 (vn_open_cred+0x278) >>> sp =3D 0xdd6ffbb0 fp =3D 0xdd6ffc90 >>> r4 =3D 0xdd6ffce0 r5 =3D 0xdd6ffd30 >>> r6 =3D 0xc2c067e0 >>> vn_open_cred() at vn_open_cred+0x278 >>> pc =3D 0xc0408540 lr =3D 0xc04082c0 (vn_open+0x24) >>> sp =3D 0xdd6ffc98 fp =3D 0xdd6ffca0 >>> r4 =3D 0xc2a2d640 r5 =3D 0xc2a76000 >>> r6 =3D 0xdd6ffce0 r7 =3D 0x00000012 >>> r8 =3D 0x00000000 r9 =3D 0x000500cf >>> r10 =3D 0xdd6ffcd0 >>> vn_open() at vn_open+0x24 >>> pc =3D 0xc04082c0 lr =3D 0xc040187c (kern_openat+0x24c) >>> sp =3D 0xdd6ffca8 fp =3D 0xdd6ffda8 >>> kern_openat() at kern_openat+0x24c >>> pc =3D 0xc040187c lr =3D 0xc04015c4 (sys_open+0x28) >>> sp =3D 0xdd6ffdb0 fp =3D 0xdd6ffdb8 >>> r4 =3D 0xc2a2d640 r5 =3D 0x00000000 >>> r6 =3D 0x20cc1198 r7 =3D 0x00000000 >>> r8 =3D 0xdd6ffe10 r9 =3D 0xc29f6c80 >>> r10 =3D 0x20c24080 >>> sys_open() at sys_open+0x28 >>> pc =3D 0xc04015c4 lr =3D 0xc0540764 (swi_handler+0x284) >>> sp =3D 0xdd6ffdc0 fp =3D 0xdd6ffe58 >>> swi_handler() at swi_handler+0x284 >>> pc =3D 0xc0540764 lr =3D 0xc0530350 (swi_entry+0x2c) >>> sp =3D 0xdd6ffe60 fp =3D 0xbfffe960 >>> r4 =3D 0x20c0a200 r5 =3D 0x000001a4 >>> r6 =3D 0x20cc1198 r7 =3D 0x00000005 >>> r8 =3D 0x20c0a200 r9 =3D 0x00000001 >>> swi_entry() at swi_entry+0x2c >>> pc =3D 0xc0530350 lr =3D 0xc0530350 (swi_entry+0x2c) >>> sp =3D 0xdd6ffe60 fp =3D 0xbfffe960 >>> Unable to unwind further >>> lock order reversal: >>> 1st 0xc2c06814 ufs (ufs) @ >>> /root/crochet-freebsd/src/sys/kern/vfs_subr.c:2099 >>> 2nd 0xcd1299d0 bufwait (bufwait) @ >>> /root/crochet-freebsd/src/sys/ufs/ffs/ffs_vnops.c:262 >>> 3rd 0xc2cb9154 ufs (ufs) @ >>> /root/crochet-freebsd/src/sys/kern/vfs_subr.c:2099 >>> KDB: stack backtrace: >>> db_trace_self() at db_trace_self >>> pc =3D 0xc052ecf4 lr =3D 0xc022dcd0 = (db_trace_self_wrapper+0x30) >>> sp =3D 0xdd6ff578 fp =3D 0xdd6ff690 >>> r10 =3D 0xc05adb30 >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x30 >>> pc =3D 0xc022dcd0 lr =3D 0xc038e9e0 (kdb_backtrace+0x38) >>> sp =3D 0xdd6ff698 fp =3D 0xdd6ff6a0 >>> r4 =3D 0xc066a484 r5 =3D 0xc0592f71 >>> r6 =3D 0xc059b3d0 r7 =3D 0xc05adb33 >>> kdb_backtrace() at kdb_backtrace+0x38 >>> pc =3D 0xc038e9e0 lr =3D 0xc03a8b5c = (witness_checkorder+0xda0) >>> sp =3D 0xdd6ff6a8 fp =3D 0xdd6ff6f8 >>> r4 =3D 0xc0581d1c >>> witness_checkorder() at witness_checkorder+0xda0 >>> pc =3D 0xc03a8b5c lr =3D 0xc033c5fc (__lockmgr_args+0x698) >>> sp =3D 0xdd6ff700 fp =3D 0xdd6ff768 >>> r4 =3D 0x00080100 r5 =3D 0x00000833 >>> r6 =3D 0xc2cb9154 r7 =3D 0x00080000 >>> r8 =3D 0x00000100 r9 =3D 0x00000000 >>> r10 =3D 0xc059b3cd >>> __lockmgr_args() at __lockmgr_args+0x698 >>> pc =3D 0xc033c5fc lr =3D 0xc04e678c (ffs_lock+0x80) >>> sp =3D 0xdd6ff770 fp =3D 0xdd6ff798 >>> r4 =3D 0xdd6ff7b8 r5 =3D 0x00080100 >>> r6 =3D 0xc2cb9120 r7 =3D 0xc2cb9154 >>> r8 =3D 0xc2cb9174 r9 =3D 0x00000000 >>> r10 =3D 0xc059b3cd >>> ffs_lock() at ffs_lock+0x80 >>> pc =3D 0xc04e678c lr =3D 0xc0554118 (VOP_LOCK1_APV+0xd8) >>> sp =3D 0xdd6ff7a0 fp =3D 0xdd6ff7b0 >>> r4 =3D 0xdd6ff7b8 r5 =3D 0xc064e690 >>> r6 =3D 0x00000000 r7 =3D 0x00000000 >>> r8 =3D 0xc2a2d640 r9 =3D 0x00000833 >>> VOP_LOCK1_APV() at VOP_LOCK1_APV+0xd8 >>> pc =3D 0xc0554118 lr =3D 0xc0408cbc (_vn_lock+0x78) >>> sp =3D 0xdd6ff7b8 fp =3D 0xdd6ff7e8 >>> r4 =3D 0xc2cb9120 r5 =3D 0x00080100 >>> r6 =3D 0x00080100 >>> _vn_lock() at _vn_lock+0x78 >>> pc =3D 0xc0408cbc lr =3D 0xc03f97e8 (vget+0x60) >>> sp =3D 0xdd6ff7f0 fp =3D 0xdd6ff810 >>> r4 =3D 0xc2cb9120 r5 =3D 0x00000000 >>> r6 =3D 0x00080100 r7 =3D 0x00000000 >>> r8 =3D 0xc2a2d640 r9 =3D 0xc059a607 >>> r10 =3D 0x00000000 >>> vget() at vget+0x60 >>> pc =3D 0xc03f97e8 lr =3D 0xc03ede5c (vfs_hash_get+0xe4) >>> sp =3D 0xdd6ff818 fp =3D 0xdd6ff848 >>> r4 =3D 0xc2a0b2b0 r5 =3D 0x00000000 >>> r6 =3D 0x0002787d r7 =3D 0x00000000 >>> r8 =3D 0xc2cb9120 >>> vfs_hash_get() at vfs_hash_get+0xe4 >>> pc =3D 0xc03ede5c lr =3D 0xc04e17f4 (ffs_vgetf+0x3c) >>> sp =3D 0xdd6ff850 fp =3D 0xdd6ff8a0 >>> r4 =3D 0x0002787d r5 =3D 0xc05aa74b >>> r6 =3D 0xc2bb13c0 r7 =3D 0xdd6ff908 >>> r8 =3D 0xc2bb13c0 r9 =3D 0x00080000 >>> r10 =3D 0xc2a0b2b0 >>> ffs_vgetf() at ffs_vgetf+0x3c >>> pc =3D 0xc04e17f4 lr =3D 0xc04da468 = (softdep_sync_buf+0x974) >>> sp =3D 0xdd6ff8a8 fp =3D 0xdd6ff928 >>> r4 =3D 0x0002787d r5 =3D 0xc05aa74b >>> r6 =3D 0xc2bb13c0 r7 =3D 0xc05aa74b >>> r8 =3D 0xc2bb13c0 r9 =3D 0x0000088b >>> r10 =3D 0x0002787d >>> softdep_sync_buf() at softdep_sync_buf+0x974 >>> pc =3D 0xc04da468 lr =3D 0xc04e7560 (ffs_syncvnode+0x2bc) >>> sp =3D 0xdd6ff930 fp =3D 0xdd6ff980 >>> r4 =3D 0x00000000 r5 =3D 0xcd188d48 >>> r6 =3D 0x00060200 r7 =3D 0xc2c067e0 >>> r8 =3D 0xc05adb30 r9 =3D 0xcd1299d0 >>> r10 =3D 0xcd129978 >>> ffs_syncvnode() at ffs_syncvnode+0x2bc >>> pc =3D 0xc04e7560 lr =3D 0xc04e6650 (ffs_fsync+0x20) >>> sp =3D 0xdd6ff988 fp =3D 0xdd6ff998 >>> r4 =3D 0xdd6ff9f0 r5 =3D 0xc2c067e0 >>> r6 =3D 0x00000000 r7 =3D 0x00000200 >>> r8 =3D 0xc2a2d640 r9 =3D 0xc2c067e0 >>> r10 =3D 0xc33445a0 >>> ffs_fsync() at ffs_fsync+0x20 >>> pc =3D 0xc04e6650 lr =3D 0xc05534b8 (VOP_FSYNC_APV+0xd0) >>> sp =3D 0xdd6ff9a0 fp =3D 0xdd6ff9b0 >>> r4 =3D 0xdd6ff9f0 r5 =3D 0xc064e690 >>> r6 =3D 0x00000000 r7 =3D 0x00000200 >>> VOP_FSYNC_APV() at VOP_FSYNC_APV+0xd0 >>> pc =3D 0xc05534b8 lr =3D 0xc04eeb54 (ufs_direnter+0x724) >>> sp =3D 0xdd6ff9b8 fp =3D 0xdd6ffa20 >>> r4 =3D 0xc33445a0 r5 =3D 0x00000000 >>> r6 =3D 0xcd356e00 >>> ufs_direnter() at ufs_direnter+0x724 >>> pc =3D 0xc04eeb54 lr =3D 0xc04f7e04 (ufs_makeinode+0x3f4) >>> sp =3D 0xdd6ffa28 fp =3D 0xdd6ffb88 >>> r4 =3D 0xdd6ffa48 r5 =3D 0xc332fa00 >>> r6 =3D 0xdd6ffd48 r7 =3D 0x00000000 >>> r8 =3D 0xc2c067e0 r9 =3D 0xdd6ffd30 >>> r10 =3D 0x000081a4 >>> ufs_makeinode() at ufs_makeinode+0x3f4 >>> pc =3D 0xc04f7e04 lr =3D 0xc04f4268 (ufs_create+0x24) >>> sp =3D 0xdd6ffb90 fp =3D 0xdd6ffb90 >>> r4 =3D 0xdd6ffc60 r5 =3D 0xc064eba0 >>> r6 =3D 0x00000000 r7 =3D 0x00100a02 >>> r8 =3D 0x00000000 r9 =3D 0xdd6ffd30 >>> r10 =3D 0xdd6ffd50 >>> ufs_create() at ufs_create+0x24 >>> pc =3D 0xc04f4268 lr =3D 0xc0552268 (VOP_CREATE_APV+0xd0) >>> sp =3D 0xdd6ffb98 fp =3D 0xdd6ffba8 >>> VOP_CREATE_APV() at VOP_CREATE_APV+0xd0 >>> pc =3D 0xc0552268 lr =3D 0xc0408540 (vn_open_cred+0x278) >>> sp =3D 0xdd6ffbb0 fp =3D 0xdd6ffc90 >>> r4 =3D 0xdd6ffce0 r5 =3D 0xdd6ffd30 >>> r6 =3D 0xc2c067e0 >>> vn_open_cred() at vn_open_cred+0x278 >>> pc =3D 0xc0408540 lr =3D 0xc04082c0 (vn_open+0x24) >>> sp =3D 0xdd6ffc98 fp =3D 0xdd6ffca0 >>> r4 =3D 0xc2a2d640 r5 =3D 0xc2a76000 >>> r6 =3D 0xdd6ffce0 r7 =3D 0x00000012 >>> r8 =3D 0x00000000 r9 =3D 0x000500cf >>> r10 =3D 0xdd6ffcd0 >>> vn_open() at vn_open+0x24 >>> pc =3D 0xc04082c0 lr =3D 0xc040187c (kern_openat+0x24c) >>> sp =3D 0xdd6ffca8 fp =3D 0xdd6ffda8 >>> kern_openat() at kern_openat+0x24c >>> pc =3D 0xc040187c lr =3D 0xc04015c4 (sys_open+0x28) >>> sp =3D 0xdd6ffdb0 fp =3D 0xdd6ffdb8 >>> r4 =3D 0xc2a2d640 r5 =3D 0x00000000 >>> r6 =3D 0x20cc1198 r7 =3D 0x00000000 >>> r8 =3D 0xdd6ffe10 r9 =3D 0xc29f6c80 >>> r10 =3D 0x20c24080 >>> sys_open() at sys_open+0x28 >>> pc =3D 0xc04015c4 lr =3D 0xc0540764 (swi_handler+0x284) >>> sp =3D 0xdd6ffdc0 fp =3D 0xdd6ffe58 >>> swi_handler() at swi_handler+0x284 >>> pc =3D 0xc0540764 lr =3D 0xc0530350 (swi_entry+0x2c) >>> sp =3D 0xdd6ffe60 fp =3D 0xbfffe960 >>> r4 =3D 0x20c0a200 r5 =3D 0x000001a4 >>> r6 =3D 0x20cc1198 r7 =3D 0x00000 >>> _______________________________________________ >>> freebsd-arm@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-arm >>> To unsubscribe, send any mail to = "freebsd-arm-unsubscribe@freebsd.org" >>=20 >> _______________________________________________ >> freebsd-arm@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-arm >> To unsubscribe, send any mail to = "freebsd-arm-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-arm@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org" >=20 ------------------------------ Randall Stewart 803-317-4952 (cell)