Date: Thu, 7 Jul 2005 11:55:20 +0200 From: =?ISO-8859-1?Q?C=E9dric?= Jonas <cedric@virtual-globe.net> To: freebsd-smp@freebsd.org Subject: Panic in ath driver with SMP Message-ID: <20050707115520.522ad6bc@ganymed.decemplex.loc>
next in thread | raw e-mail | index | archive | help
--Multipart_Thu__7_Jul_2005_11_55_20_+0200_UaB138Fe/nSaCl6A
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Hello,
Since 1 or 2 months, I experienced panics when I use the ath interface of o=
ur server (configured as hostap).=20
Also, the system hangs often (without panic) during the boot process at the=
"Setting hostname: icarus."=20
level. Because I had absolutely no time, I couldn't learn to use the kernel=
debugger before.=20
icarus# uname -a
FreeBSD icarus 5.4-STABLE FreeBSD 5.4-STABLE #0: Sun Jul 3 15:14:13 CEST =
2005 cedric@icarus:/usr/obj/usr/src/sys/ICARUS i386
Sources are from July, 1
debug.mpsafenet: 1
After compiling the kernel with debug options, I get the following during t=
he boot process:
Setting hostname: icarus.
lock order reversal
1st 0xc2cd4c6c ath0 (network driver) @ /usr/src/sys/dev/ath/if_ath.c:893
2nd 0xc30583c8 user map (user map) @ /usr/src/sys/vm/vm_map.c:2998
I suppose this explains the occasional hangups during the boot process?
The following was obtained after a panic (which occured only when I use the=
ath interface):
icarus# kgdb kernel.debug /var/crash/vmcore.86
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so=
: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you a=
re
welcome to change it and/or distribute copies of it under certain conditio=
ns.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
#0 doadump () at pcpu.h:160
160 __asm __volatile("movl %%fs:0,%0" : "=3Dr" (td));
(kgdb) list *0xc0695dfd
0xc0695dfd is in bus_dmamap_load_mbuf (pmap.h:201).
196
197 if ((pa =3D PTD[va >> PDRSHIFT]) & PG_PS) {
198 pa =3D (pa & ~(NBPDR - 1)) | (va & (NBPDR - 1));
199 } else {
200 pa =3D *vtopte(va);
201 pa =3D (pa & PG_FRAME) | (va & PAGE_MASK);
202 }
203 return pa;
204 }
205
(kgdb) backtrace
#0 doadump () at pcpu.h:160
#1 0xc05241dc in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:=
410
#2 0xc05244f1 in panic (fmt=3D0xc06f0b1f "%s") at /usr/src/sys/kern/kern_=
shutdown.c:566
#3 0xc06a96ec in trap_fatal (frame=3D0xe5ba3b80, eva=3D3221110944) at /us=
r/src/sys/i386/i386/trap.c:817
#4 0xc06a941f in trap_pfault (frame=3D0xe5ba3b80, usermode=3D0, eva=3D322=
1110944) at /usr/src/sys/i386/i386/trap.c:735
#5 0xc06a9085 in trap (frame=3D
{tf_fs =3D -440795112, tf_es =3D -1068433392, tf_ds =3D -1027473392, =
tf_edi =3D -1026674344, tf_esi =3D -1026496768, tf_ebp =3D -440779748, tf_i=
sp =3D -440779860, tf_ebx =3D 0, tf_edx =3D 0, tf_ecx =3D -117273402, tf_ea=
x =3D 1019944, tf_trapno =3D 12, tf_err =3D 0, tf_eip =3D -1066836483, tf_c=
s =3D 8, tf_eflags =3D 66055, tf_esp =3D -440779804, tf_ss =3D -1068382496}=
) at /usr/src/sys/i386/i386/trap.c:425
#6 0xc069796a in calltrap () at /usr/src/sys/i386/i386/exception.s:140
#7 0xe5ba0018 in ?? ()
#8 0xc0510010 in exit1 (td=3D0xc2cba680, rv=3D-440779764) at /usr/src/sys=
/kern/kern_exit.c:394
#9 0xc0494122 in ath_rxbuf_init (sc=3D0xc2cd4000, bf=3D0xc2ce3158) at /us=
r/src/sys/dev/ath/if_ath.c:1601
#10 0xc0494560 in ath_rx_proc (arg=3D0xc2cd4000, npending=3D1) at /usr/src=
/sys/dev/ath/if_ath.c:1818
#11 0xc0540c57 in taskqueue_run (queue=3D0xc2c94dc0) at /usr/src/sys/kern/=
subr_taskqueue.c:191
#12 0xc0540cbe in taskqueue_swi_run (dummy=3D0x0) at /usr/src/sys/kern/sub=
r_taskqueue.c:213
#13 0xc0512ad8 in ithread_loop (arg=3D0xc2c28300) at /usr/src/sys/kern/ker=
n_intr.c:547
#14 0xc0511ed4 in fork_exit (callout=3D0xc05129b4 <ithread_loop>, arg=3D0x=
c2c28300, frame=3D0xe5ba3d38) at /usr/src/sys/kern/kern_fork.c:791
#15 0xc06979cc in fork_trampoline () at /usr/src/sys/i386/i386/exception.s=
:209
(kgdb)=20
Kernel config is attached.
Another important detail could be that I use a dual Xeon box, with HT enabl=
ed, which results
in 4 logical CPU's (SMP issue?) and explain why anybody reported this probl=
em before?!
I tried to disable PREEMPTION, the hyperthreading functionality (=3D> 2 log=
ical CPU), and I set
debug.mpsafenet to 0, but the LOR (+ Panic when ath0 is used) continue to o=
ccure...
I tried also to downgrade to 5.3-RELEASE, where the problems do _not_ occur=
... but that's not
a solution.
I hope somebody could help, if you need additional informations, let me kno=
w.
Thank you.
--=20
Best regards,=20
C=E9dric Jonas
--Multipart_Thu__7_Jul_2005_11_55_20_+0200_UaB138Fe/nSaCl6A
Content-Type: application/octet-stream; name=ICARUS
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename=ICARUS
bWFjaGluZQkJaTM4NgpjcHUJCQlJNjg2X0NQVQppZGVudAkJSUNBUlVTCgojIFRvIHN0YXRpY2Fs
bHkgY29tcGlsZSBpbiBkZXZpY2Ugd2lyaW5nIGluc3RlYWQgb2YgL2Jvb3QvZGV2aWNlLmhpbnRz
CiNoaW50cwkJIkdFTkVSSUMuaGludHMiCQkjIERlZmF1bHQgcGxhY2VzIHRvIGxvb2sgZm9yIGRl
dmljZXMuCgptYWtlb3B0aW9ucwlERUJVRz0tZwkJIyBCdWlsZCBrZXJuZWwgd2l0aCBnZGIoMSkg
ZGVidWcgc3ltYm9scwoKb3B0aW9ucyAJS0RCCQkJIyBFbmFibGUga2VybmVsIGRlYnVnZ2VyIHN1
cHBvcnQuCm9wdGlvbnMgCUdEQgkJCSMgU3VwcG9ydCByZW1vdGUgR0RCLgpvcHRpb25zIAlJTlZB
UklBTlRTCQkjIEVuYWJsZSBjYWxscyBvZiBleHRyYSBzYW5pdHkgY2hlY2tpbmcKb3B0aW9ucyAJ
SU5WQVJJQU5UX1NVUFBPUlQJIyBFeHRyYSBzYW5pdHkgY2hlY2tzIG9mIGludGVybmFsIHN0cnVj
dHVyZXMsIHJlcXVpcmVkIGJ5IElOVkFSSUFOVFMKb3B0aW9ucyAJV0lUTkVTUwkJCSMgRW5hYmxl
IGNoZWNrcyB0byBkZXRlY3QgZGVhZGxvY2tzIGFuZCBjeWNsZXMKb3B0aW9ucyAJV0lUTkVTU19T
S0lQU1BJTgkjIERvbid0IHJ1biB3aXRuZXNzIG9uIHNwaW5sb2NrcyBmb3Igc3BlZWQKCm9wdGlv
bnMgCVNDSEVEXzRCU0QJCSMgNEJTRCBzY2hlZHVsZXIKb3B0aW9ucyAJSU5FVAkJCSMgSW50ZXJO
RVR3b3JraW5nCm9wdGlvbnMgCUlORVQ2CQkJIyBJUHY2IGNvbW11bmljYXRpb25zIHByb3RvY29s
cwpvcHRpb25zIAlGRlMJCQkjIEJlcmtlbGV5IEZhc3QgRmlsZXN5c3RlbQpvcHRpb25zIAlTT0ZU
VVBEQVRFUwkJIyBFbmFibGUgRkZTIHNvZnQgdXBkYXRlcyBzdXBwb3J0Cm9wdGlvbnMgCVVGU19B
Q0wJCQkjIFN1cHBvcnQgZm9yIGFjY2VzcyBjb250cm9sIGxpc3RzCm9wdGlvbnMgCVVGU19ESVJI
QVNICQkjIEltcHJvdmUgcGVyZm9ybWFuY2Ugb24gYmlnIGRpcmVjdG9yaWVzCm9wdGlvbnMgCU1E
X1JPT1QJCQkjIE1EIGlzIGEgcG90ZW50aWFsIHJvb3QgZGV2aWNlCm9wdGlvbnMgCU5GU0NMSUVO
VAkJIyBOZXR3b3JrIEZpbGVzeXN0ZW0gQ2xpZW50Cm9wdGlvbnMgCU5GU1NFUlZFUgkJIyBOZXR3
b3JrIEZpbGVzeXN0ZW0gU2VydmVyCm9wdGlvbnMgCUNEOTY2MAkJCSMgSVNPIDk2NjAgRmlsZXN5
c3RlbQpvcHRpb25zIAlQUk9DRlMJCQkjIFByb2Nlc3MgZmlsZXN5c3RlbSAocmVxdWlyZXMgUFNF
VURPRlMpCm9wdGlvbnMgCVBTRVVET0ZTCQkjIFBzZXVkby1maWxlc3lzdGVtIGZyYW1ld29yawpv
cHRpb25zIAlHRU9NX0dQVAkJIyBHVUlEIFBhcnRpdGlvbiBUYWJsZXMuCm9wdGlvbnMgCUNPTVBB
VF80MwkJIyBDb21wYXRpYmxlIHdpdGggQlNEIDQuMyBbS0VFUCBUSElTIV0Kb3B0aW9ucyAJQ09N
UEFUX0ZSRUVCU0Q0CQkjIENvbXBhdGlibGUgd2l0aCBGcmVlQlNENApvcHRpb25zIAlTQ1NJX0RF
TEFZPTE1MDAwCSMgRGVsYXkgKGluIG1zKSBiZWZvcmUgcHJvYmluZyBTQ1NJCm9wdGlvbnMgCUtU
UkFDRQkJCSMga3RyYWNlKDEpIHN1cHBvcnQKb3B0aW9ucyAJU1lTVlNITQkJCSMgU1lTVi1zdHls
ZSBzaGFyZWQgbWVtb3J5Cm9wdGlvbnMgCVNZU1ZNU0cJCQkjIFNZU1Ytc3R5bGUgbWVzc2FnZSBx
dWV1ZXMKb3B0aW9ucyAJU1lTVlNFTQkJCSMgU1lTVi1zdHlsZSBzZW1hcGhvcmVzCm9wdGlvbnMg
CV9LUE9TSVhfUFJJT1JJVFlfU0NIRURVTElORyAjIFBPU0lYIFAxMDAzXzFCIHJlYWwtdGltZSBl
eHRlbnNpb25zCm9wdGlvbnMgCUtCRF9JTlNUQUxMX0NERVYJIyBpbnN0YWxsIGEgQ0RFViBlbnRy
eSBpbiAvZGV2Cm9wdGlvbnMgCUFIQ19SRUdfUFJFVFRZX1BSSU5UCSMgUHJpbnQgcmVnaXN0ZXIg
Yml0ZmllbGRzIGluIGRlYnVnCgkJCQkJIyBvdXRwdXQuICBBZGRzIH4xMjhrIHRvIGRyaXZlci4K
b3B0aW9ucyAJQUhEX1JFR19QUkVUVFlfUFJJTlQJIyBQcmludCByZWdpc3RlciBiaXRmaWVsZHMg
aW4gZGVidWcKCQkJCQkjIG91dHB1dC4gIEFkZHMgfjIxNWsgdG8gZHJpdmVyLgpvcHRpb25zIAlB
REFQVElWRV9HSUFOVAkJIyBHaWFudCBtdXRleCBpcyBhZGFwdGl2ZS4KCmRldmljZQkJYXBpYwkJ
IyBJL08gQVBJQwoKIyBCdXMgc3VwcG9ydC4gIERvIG5vdCByZW1vdmUgaXNhLCBldmVuIGlmIHlv
dSBoYXZlIG5vIGlzYSBzbG90cwpkZXZpY2UJCWlzYQpkZXZpY2UJCXBjaQoKIyBGbG9wcHkgZHJp
dmVzCmRldmljZQkJZmRjCgojIEFUQSBhbmQgQVRBUEkgZGV2aWNlcwpkZXZpY2UJCWF0YQpkZXZp
Y2UJCWF0YWRpc2sJCSMgQVRBIGRpc2sgZHJpdmVzCmRldmljZQkJYXRhcmFpZAkJIyBBVEEgUkFJ
RCBkcml2ZXMKZGV2aWNlCQlhdGFwaWNkCQkjIEFUQVBJIENEUk9NIGRyaXZlcwpkZXZpY2UJCWF0
YXBpZmQJCSMgQVRBUEkgZmxvcHB5IGRyaXZlcwpkZXZpY2UJCWF0YXBpc3QJCSMgQVRBUEkgdGFw
ZSBkcml2ZXMKb3B0aW9ucyAJQVRBX1NUQVRJQ19JRAkjIFN0YXRpYyBkZXZpY2UgbnVtYmVyaW5n
CgojIFNDU0kgcGVyaXBoZXJhbHMKZGV2aWNlCQlzY2J1cwkJIyBTQ1NJIGJ1cyAocmVxdWlyZWQg
Zm9yIFNDU0kpCmRldmljZQkJY2gJCSMgU0NTSSBtZWRpYSBjaGFuZ2VycwpkZXZpY2UJCWRhCQkj
IERpcmVjdCBBY2Nlc3MgKGRpc2tzKQpkZXZpY2UJCXNhCQkjIFNlcXVlbnRpYWwgQWNjZXNzICh0
YXBlIGV0YykKZGV2aWNlCQljZAkJIyBDRApkZXZpY2UJCXBhc3MJCSMgUGFzc3Rocm91Z2ggZGV2
aWNlIChkaXJlY3QgU0NTSSBhY2Nlc3MpCmRldmljZQkJc2VzCQkjIFNDU0kgRW52aXJvbm1lbnRh
bCBTZXJ2aWNlcyAoYW5kIFNBRi1URSkKCiMgU0NTSSBjb250cm9sbGVyCmRldmljZQkJc3ltCgoj
IGF0a2JkYzAgY29udHJvbHMgYm90aCB0aGUga2V5Ym9hcmQgYW5kIHRoZSBQUy8yIG1vdXNlCmRl
dmljZQkJYXRrYmRjCQkjIEFUIGtleWJvYXJkIGNvbnRyb2xsZXIKZGV2aWNlCQlhdGtiZAkJIyBB
VCBrZXlib2FyZApkZXZpY2UJCXBzbQkJIyBQUy8yIG1vdXNlCgpkZXZpY2UJCXZnYQkJIyBWR0Eg
dmlkZW8gY2FyZCBkcml2ZXIKCmRldmljZQkJc3BsYXNoCmRldmljZQkJZ3JlZW5fc2F2ZXIKCiMg
c3lzY29ucyBpcyB0aGUgZGVmYXVsdCBjb25zb2xlIGRyaXZlciwgcmVzZW1ibGluZyBhbiBTQ08g
Y29uc29sZQpkZXZpY2UJCXNjCgpkZXZpY2UJCWFncAkJIyBzdXBwb3J0IHNldmVyYWwgQUdQIGNo
aXBzZXRzCgojIEZsb2F0aW5nIHBvaW50IHN1cHBvcnQgLSBkbyBub3QgZGlzYWJsZS4KZGV2aWNl
CQlucHgKCiMgUG93ZXIgbWFuYWdlbWVudCBzdXBwb3J0IChzZWUgTk9URVMgZm9yIG1vcmUgb3B0
aW9ucykKZGV2aWNlCQlhcG0KIyBBZGQgc3VzcGVuZC9yZXN1bWUgc3VwcG9ydCBmb3IgdGhlIGk4
MjU0LgpkZXZpY2UJCXBtdGltZXIKCiMgU2VyaWFsIChDT00pIHBvcnRzCiNkZXZpY2UJCXNpbwkJ
IyA4MjUwLCAxNls0NV01MCBiYXNlZCBzZXJpYWwgcG9ydHMKCiMgUGFyYWxsZWwgcG9ydApkZXZp
Y2UJCXBwYwpkZXZpY2UJCXBwYnVzCQkjIFBhcmFsbGVsIHBvcnQgYnVzIChyZXF1aXJlZCkKZGV2
aWNlCQlscHQJCSMgUHJpbnRlcgpkZXZpY2UJCXBsaXAJCSMgVENQL0lQIG92ZXIgcGFyYWxsZWwK
ZGV2aWNlCQlwcGkJCSMgUGFyYWxsZWwgcG9ydCBpbnRlcmZhY2UgZGV2aWNlCmRldmljZQkJdnBv
CQkjIFJlcXVpcmVzIHNjYnVzIGFuZCBkYQoKIyBQQ0kgRXRoZXJuZXQgTklDcyB0aGF0IHVzZSB0
aGUgY29tbW9uIE1JSSBidXMgY29udHJvbGxlciBjb2RlLgojIE5PVEU6IEJlIHN1cmUgdG8ga2Vl
cCB0aGUgJ2RldmljZSBtaWlidXMnIGxpbmUgaW4gb3JkZXIgdG8gdXNlIHRoZXNlIE5JQ3MhCmRl
dmljZQkJbWlpYnVzCQkjIE1JSSBidXMgc3VwcG9ydApkZXZpY2UJCWZ4cAkJIyBJbnRlbCBFdGhl
ckV4cHJlc3MgUFJPLzEwMEIgKDgyNTU3LCA4MjU1OCkKZGV2aWNlCQlybAkJIyBSZWFsVGVrIDgx
MjkvODEzOQoKIyAnZGV2aWNlIGVkJyByZXF1aXJlcyAnZGV2aWNlIG1paWJ1cycKZGV2aWNlCQll
ZAkJIyBORVsxMl0wMDAsIFNNQyBVbHRyYSwgM2M1MDMsIERTODM5MCBjYXJkcwoKIyBXaXJlbGVz
cyBOSUMgY2FyZHMKZGV2aWNlCQl3bGFuCQkjIDgwMi4xMSBzdXBwb3J0CmRldmljZQkJYXRoCmRl
dmljZQkJYXRoX2hhbAkJIyBBdGhlcm9zIEhBTCAoaW5jbHVkZXMgYmluYXJ5IGNvbXBvbmVudCkK
CiMgUHNldWRvIGRldmljZXMuCmRldmljZQkJbG9vcAkJIyBOZXR3b3JrIGxvb3BiYWNrCmRldmlj
ZQkJbWVtCQkjIE1lbW9yeSBhbmQga2VybmVsIG1lbW9yeSBkZXZpY2VzCmRldmljZQkJaW8JCSMg
SS9PIGRldmljZQpkZXZpY2UJCXJhbmRvbQkJIyBFbnRyb3B5IGRldmljZQpkZXZpY2UJCWV0aGVy
CQkjIEV0aGVybmV0IHN1cHBvcnQKZGV2aWNlCQlzbAkJIyBLZXJuZWwgU0xJUApkZXZpY2UJCXBw
cAkJIyBLZXJuZWwgUFBQCmRldmljZQkJdHVuCQkjIFBhY2tldCB0dW5uZWwuCmRldmljZQkJcHR5
CQkjIFBzZXVkby10dHlzICh0ZWxuZXQgZXRjKQpkZXZpY2UJCW1kCQkjIE1lbW9yeSAiZGlza3Mi
CmRldmljZQkJZ2lmCQkjIElQdjYgYW5kIElQdjQgdHVubmVsaW5nCmRldmljZQkJZmFpdGgJCSMg
SVB2Ni10by1JUHY0IHJlbGF5aW5nICh0cmFuc2xhdGlvbikKCiMgVGhlIGBicGYnIGRldmljZSBl
bmFibGVzIHRoZSBCZXJrZWxleSBQYWNrZXQgRmlsdGVyLgojIEJlIGF3YXJlIG9mIHRoZSBhZG1p
bmlzdHJhdGl2ZSBjb25zZXF1ZW5jZXMgb2YgZW5hYmxpbmcgdGhpcyEKZGV2aWNlCQlicGYJCSMg
QmVya2VsZXkgcGFja2V0IGZpbHRlcgoKIyBVU0Igc3VwcG9ydApkZXZpY2UJCXVoY2kJCSMgVUhD
SSBQQ0ktPlVTQiBpbnRlcmZhY2UKZGV2aWNlCQlvaGNpCQkjIE9IQ0kgUENJLT5VU0IgaW50ZXJm
YWNlCmRldmljZQkJdXNiCQkjIFVTQiBCdXMgKHJlcXVpcmVkKQpkZXZpY2UJCXVkYnAJCSMgVVNC
IERvdWJsZSBCdWxrIFBpcGUgZGV2aWNlcwpkZXZpY2UJCXVnZW4JCSMgR2VuZXJpYwpkZXZpY2UJ
CXVoaWQJCSMgIkh1bWFuIEludGVyZmFjZSBEZXZpY2VzIgpkZXZpY2UJCXVrYmQJCSMgS2V5Ym9h
cmQKZGV2aWNlCQl1bHB0CQkjIFByaW50ZXIKZGV2aWNlCQl1bWFzcwkJIyBEaXNrcy9NYXNzIHN0
b3JhZ2UgLSBSZXF1aXJlcyBzY2J1cyBhbmQgZGEKZGV2aWNlCQl1bXMJCSMgTW91c2UKZGV2aWNl
CQl1cmlvCQkjIERpYW1vbmQgUmlvIDUwMCBNUDMgcGxheWVyCmRldmljZQkJdXNjYW5uZXIJIyBT
Y2FubmVycwoKIyBQRgpkZXZpY2UJCXBmCQkJI1BGIE9wZW5CU0QgcGFja2V0LWZpbHRlciBmaXJl
d2FsbApkZXZpY2UJCXBmbG9nCQkJI2xvZ2dpbmcgc3VwcG9ydCBpbnRlcmZhY2UgZm9yIFBGCmRl
dmljZQkJcGZzeW5jCQkJI3N5bmNocm9uaXphdGlvbiBpbnRlcmZhY2UgZm9yIFBGCm9wdGlvbnMg
CUFMVFEKb3B0aW9ucyAJQUxUUV9QUklRCSMgUHJpb3JpdHkgUXVldWVpbmcKb3B0aW9ucyAJQUxU
UV9OT1BDQwkjIFJlcXVpcmVkIGZvciBTTVAgYnVpbGQKCiMgUFBQb0UgRFNMCm9wdGlvbnMJCU5F
VEdSQVBICm9wdGlvbnMJCU5FVEdSQVBIX1NPQ0tFVApvcHRpb25zCQlORVRHUkFQSF9QUFBPRQpv
cHRpb25zCQlORVRHUkFQSF9FVEhFUgoKIyBTTVAKb3B0aW9ucyAJU01QCQkJIyBTeW1tZXRyaWMg
TXVsdGlQcm9jZXNzb3IgS2VybmVsCm9wdGlvbnMJCU1QVEFCTEVfRk9SQ0VfSFRUCSMgRW5hYmxl
IEhUVCBDUFVzIHdpdGggdGhlIE1QIFRhYmxlCm9wdGlvbnMJCVBSRUVNUFRJT04KCm9wdGlvbnMg
CUNQVV9FTkFCTEVfVENDCm9wdGlvbnMgCUNQVV9GQVNURVJfNVg4Nl9GUFUKb3B0aW9ucyAJQ1BV
X1NVU1BfSExUCgojIFNhdmUga2VybmVsIGNvbmZpZyBpbiBiaW5hcnkncwpvcHRpb25zIAlJTkNM
VURFX0NPTkZJR19GSUxFCgojIERpc2tzCm9wdGlvbnMgCVFVT1RBCQkJI2VuYWJsZSBkaXNrIHF1
b3RhcwoKIyBQb3NpeCAxMDAzLjFiCm9wdGlvbnMgCVAxMDAzXzFCX1NFTUFQSE9SRVMKCiMgU291
bmQKIyBkZXZpY2UJCXNvdW5kCiMgZGV2aWNlCQlzbmRfaWNoCiMgZGV2aWNlCQlzcGVha2VyCQkj
UGxheSBJQk0gQkFTSUMtc3R5bGUgbm9pc2VzIG91dCB5b3VyIHNwZWFrZXIKCiMgU01CdXMKZGV2
aWNlCQlzbWJ1cwkJIyBCdXMgc3VwcG9ydCwgcmVxdWlyZWQgZm9yIHNtYiBiZWxvdy4KZGV2aWNl
CQlpY2hzbWIKZGV2aWNlCQlzbWIKCiMgUmVib290IGFmdGVyIHBhbmljCm9wdGlvbnMgCVBBTklD
X1JFQk9PVF9XQUlUX1RJTUU9NjAKCiMgRW11bGF0aW9uCm9wdGlvbnMgCUNPTVBBVF9MSU5VWApv
cHRpb25zIAlDT01QQVRfQU9VVApvcHRpb25zIAlMSU5QUk9DRlMKCiMgRmlsZXN5c3RlbXMKb3B0
aW9ucyAJTlVMTEZTCQkJI05VTEwgZmlsZXN5c3RlTQo=
--Multipart_Thu__7_Jul_2005_11_55_20_+0200_UaB138Fe/nSaCl6A--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20050707115520.522ad6bc>