Date: Tue, 11 May 2010 21:24:30 -0700 From: David Wolfskill <david@catwhisker.org> To: "K. Macy" <kmacy@freebsd.org> Cc: Ed Schouten <ed@80386.nl>, current@freebsd.org Subject: Re: Panic @r207844; current process: flowcleaner "Fatal trap 12: page fault while in kernel mode" Message-ID: <20100512042430.GH685@bunrab.catwhisker.org> In-Reply-To: <AANLkTinz1-66W90-Lpi_4pP_9XABmj46yWfn_IoT0TtL@mail.gmail.com> References: <20100510140722.GD49209@bunrab.catwhisker.org> <20100510182243.GV56080@hoeg.nl> <20100510182625.GF49209@bunrab.catwhisker.org> <AANLkTilOTxkU-DEJWdoa1xFNPwJ-J634Oe5zDEuo4Him@mail.gmail.com> <20100511131817.GB685@bunrab.catwhisker.org> <AANLkTinz1-66W90-Lpi_4pP_9XABmj46yWfn_IoT0TtL@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--NzX0AQGjRQPusK/O
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Tue, May 11, 2010 at 08:30:09PM -0700, K. Macy wrote:
> Please try 207949 ....
> ...
The panic (this time) didn't show up until about 10 seconds after the
login: prompt showed up on the serial console. Here's what it looks
like:
=2E..
3 Select option, [Enter] for default 3
3 or [Space] to pause timer 9 3
@DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDY
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-CURRENT #157 r207911M: Tue May 11 20:54:25 PDT 2010
root@freebeast.catwhisker.org:/usr/obj/usr/src/sys/GENERIC i386
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.60GHz (3614.55-MHz 686-class CPU)
Origin =3D "GenuineIntel" Id =3D 0xf41 Family =3D f Model =3D 4 Stepp=
ing =3D 1
Features=3D0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PG=
E,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=3D0x659d<SSE3,DTES64,MON,DS_CPL,EST,TM2,CNXT-ID,CX16,xTPR>
AMD Features=3D0x20100000<NX,LM>
TSC: P-state invariant
real memory =3D 2147483648 (2048 MB)
avail memory =3D 2086129664 (1989 MB)
ACPI APIC Table: <PTLTD APIC >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 2 package(s) x 1 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 6
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
kbd1 at kbdmux0
acpi0: <PTLTD RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <unknown> at device 0.1 (no driver attached)
pci0: <base peripheral> at device 1.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
aac0: <Adaptec SCSI RAID 2200S> mem 0xdc000000-0xdfffffff irq 24 at device =
1.0 on pci2
aac0: Enable Raw I/O
aac0: New comm. interface enabled
aac0: [ITHREAD]
aac0: Adaptec 2200S, aac driver 2.1.9-1
aacp0: <SCSI Passthrough Bus> on aac0
aacp1: <SCSI Passthrough Bus> on aac0
pcib3: <ACPI PCI-PCI bridge> at device 0.2 on pci1
pci3: <ACPI PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.1> port 0x2000-0x203f=
mem 0xd8200000-0xd821ffff irq 54 at device 2.0 on pci3
em0: [FILTER]
em0: Ethernet address: 00:30:48:2d:32:6a
em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.1> port 0x2040-0x207f=
mem 0xd8220000-0xd823ffff irq 55 at device 2.1 on pci3
em1: [FILTER]
em1: Ethernet address: 00:30:48:2d:32:6b
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 6.0 on pci0
pci5: <ACPI PCI bus> on pcib5
uhci0: <Intel 82801EB (ICH5) USB controller USB-A> port 0x1400-0x141f irq 1=
6 at device 29.0 on pci0
uhci0: [ITHREAD]
usbus0: <Intel 82801EB (ICH5) USB controller USB-A> on uhci0
uhci1: <Intel 82801EB (ICH5) USB controller USB-B> port 0x1420-0x143f irq 1=
9 at device 29.1 on pci0
uhci1: [ITHREAD]
usbus1: <Intel 82801EB (ICH5) USB controller USB-B> on uhci1
uhci2: <Intel 82801EB (ICH5) USB controller USB-C> port 0x1440-0x145f irq 1=
8 at device 29.2 on pci0
uhci2: [ITHREAD]
usbus2: <Intel 82801EB (ICH5) USB controller USB-C> on uhci2
uhci3: <Intel 82801EB (ICH5) USB controller USB-D> port 0x1460-0x147f irq 1=
6 at device 29.3 on pci0
uhci3: [ITHREAD]
usbus3: <Intel 82801EB (ICH5) USB controller USB-D> on uhci3
ehci0: <Intel 82801EB/R (ICH5) USB 2.0 controller> mem 0xd8001000-0xd80013f=
f irq 23 at device 29.7 on pci0
ehci0: [ITHREAD]
usbus4: EHCI version 1.0
usbus4: <Intel 82801EB/R (ICH5) USB 2.0 controller> on ehci0
pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci6: <ACPI PCI bus> on pcib6
vgapci0: <VGA-compatible display> port 0x3000-0x30ff mem 0xd9000000-0xd9fff=
fff,0xd8300000-0xd8300fff irq 17 at device 1.0 on pci6
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH5 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177=
,0x376,0x14a0-0x14af at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model Generic PS/2 mouse, device ID 0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff,0xc9000-0x=
c9fff,0xca000-0xcafff,0xcb000-0xcf7ff pnpid ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=3D0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: parallel port not found.
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 122d00000e24
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 122d00000e24
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 480Mbps High Speed USB v2.0
ata1: DMA limited to UDMA33, controller found non-ATA66 cable
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4
acd0: DVDROM <MATSHITADVD-ROM SR-8178/PZ16> at ata1-slave UDMA33=20
aacd0: <RAID 1 (Mirror)> on aac0
aacd0: 34970MB (71619584 sectors)
aacd1: <RAID 1 (Mirror)> on aac0
aacd1: 69974MB (143307008 sectors)
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub4: 8 ports with 8 removable, self powered
ses0 at aacp0 bus 0 scbus0 target 6 lun 0
ses0: <SUPER GEM318 0> Fixed Uninstalled SCSI-2 device=20
ses0: 3.300MB/s transfers
ses0: SAF-TE Compliant Device
pass0 at aacp0 bus 0 scbus0 target 0 lun 0
pass0: <SEAGATE ST336754LC 0003> Fixed Uninstalled SCSI-3 device=20
pass0: 3.300MB/s transfers
pass1 at aacp0 bus 0 scbus0 target 1 lun 0
pass1: <SEAGATE ST336754LC 0003> Fixed Uninstalled SCSI-3 device=20
pass1: 3.300MB/s transfers
pass2 at aacp0 bus 0 scbus0 target 2 lun 0
pass2: <SEAGATE ST373454LC 0005> Fixed Uninstalled SCSI-3 device=20
pass2: 3.300MB/s transfers
pass3 at aacp0 bus 0 scbus0 target 3 lun 0
pass3: <SEAGATE ST373454LC 0005> Fixed Uninstalled SCSI-3 device=20
pass3: 3.300MB/s transfers
SMP: AP CPU #1 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/aacd0s4a
Setting hostuuid: 80f1e964-dc63-0010-89d5-0030482d326a.
Setting hostid: 0xc74551dd.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
/dev/aacd0s4a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/aacd0s4a: clean, 267641 free (2161 frags, 33185 blocks, 0.3% fragmenta=
tion)
/dev/aacd1s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/aacd1s1e: clean, 4751807 free (262119 frags, 561211 blocks, 2.0% fragm=
entation)
/dev/aacd0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/aacd0s1a: clean, 422042 free (682 frags, 52670 blocks, 0.1% fragmentat=
ion)
/dev/aacd0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/aacd0s1d: clean, 1076253 free (237 frags, 134502 blocks, 0.0% fragment=
ation)
/dev/aacd1s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/aacd1s1d: clean, 11714564 free (199148 frags, 1439427 blocks, 1.2% fra=
gmentation)
/dev/aacd0s2a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/aacd0s2a: clean, 596264 free (1144 frags, 74390 blocks, 0.2% fragmenta=
tion)
/dev/aacd0s2d: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/aacd0s2d: clean, 1108504 free (24176 frags, 135541 blocks, 1.3% fragme=
ntation)
/dev/aacd0s3a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/aacd0s3a: clean, 578034 free (1186 frags, 72106 blocks, 0.2% fragmenta=
tion)
/dev/aacd0s3d: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/aacd0s3d: clean, 1070824 free (41840 frags, 128623 blocks, 2.3% fragme=
ntation)
/dev/aacd0s4d: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/aacd0s4d: clean, 1075779 free (60523 frags, 126907 blocks, 3.4% fragme=
ntation)
/dev/aacd0s4f: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/aacd0s4f: clean, 2071045 free (357 frags, 258836 blocks, 0.0% fragment=
ation)
Mounting local file systems:.
Setting hostname: freebeast.catwhisker.org.
lock order reversal:
1st 0xc68406b8 ufs (ufs) @ /usr/src/sys/kern/vfs_lookup.c:502
2nd 0xd94fe304 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_softdep.c:11363
3rd 0xc6867af8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2091
KDB: stack backtrace:
db_trace_self_wrapper(c0cc73e5,ebe22300,c08ea0d5,c08da44b,c0cca3d8,...) at =
db_trace_self_wrapper+0x26
kdb_backtrace(c08da44b,c0cca3d8,c553d030,c5540568,ebe2235c,...) at kdb_back=
trace+0x29
_witness_debugger(c0cca3d8,c6867af8,c0cbc6fd,c5540568,c0cd14f1,...) at _wit=
ness_debugger+0x25
witness_checkorder(c6867af8,9,c0cd14f1,82b,0,...) at witness_checkorder+0x8=
39
__lockmgr_args(c6867af8,80100,c6867b18,0,0,...) at __lockmgr_args+0x7f9
ffs_lock(ebe22480,c08e9e7b,c0cd0996,80100,c6867aa0,...) at ffs_lock+0x8a
VOP_LOCK1_APV(c0dd5020,ebe22480,c638e0a4,c0defa60,c6867aa0,...) at VOP_LOCK=
1_APV+0xb5
_vn_lock(c6867aa0,80100,c0cd14f1,82b,4,...) at _vn_lock+0x5e
vget(c6867aa0,80100,c638e000,50,0,...) at vget+0xb9
vfs_hash_get(c6848ca8,78c22,80000,c638e000,ebe225d0,...) at vfs_hash_get+0x=
e6
ffs_vgetf(c6848ca8,78c22,80000,ebe225d0,1,...) at ffs_vgetf+0x49
softdep_sync_metadata(c6840660,0,c0ced738,144,0,...) at softdep_sync_metada=
ta+0xc92
ffs_syncvnode(c6840660,1,c638e000,ebe22690,246,...) at ffs_syncvnode+0x3e2
ffs_truncate(c6840660,200,0,880,c5588480,...) at ffs_truncate+0x862
ufs_direnter(c6840660,c6867aa0,ebe22944,ebe22bd4,0,...) at ufs_direnter+0x8=
d4
ufs_makeinode(ebe22bd4,0,ebe22b30,ebe22a8c,c0c03425,...) at ufs_makeinode+0=
x557
ufs_create(ebe22b30,ebe22b48,0,0,ebe22ba8,...) at ufs_create+0x30
VOP_CREATE_APV(c0dd5020,ebe22b30,ebe22bd4,ebe22ac8,0,...) at VOP_CREATE_APV=
+0xa5
vn_open_cred(ebe22ba8,ebe22c5c,1a4,0,c5588480,...) at vn_open_cred+0x215
vn_open(ebe22ba8,ebe22c5c,1a4,c637e770,0,...) at vn_open+0x3b
kern_openat(c638e000,ffffff9c,804c5e8,0,602,...) at kern_openat+0x125
kern_open(c638e000,804c5e8,0,601,21b6,...) at kern_open+0x35
open(c638e000,ebe22cf8,c0cfff3b,c0caa814,c6866550,...) at open+0x30
syscall(ebe22d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (5, FreeBSD ELF32, open), eip =3D 0x281744b3, esp =3D 0xbfbfec4=
c, ebp =3D 0xbfbfecb8 ---
Starting Network: lo0 em0 em1.
lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3D3<RXCSUM,TXCSUM>
inet 127.0.0.1 netmask 0xff000000=20
inet6 ::1 prefixlen 128=20
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3=20
nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=3D289b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WO=
L_UCAST,WOL_MAGIC>
ether 00:30:48:2d:32:6a
inet 172.16.8.10 netmask 0xffffff00 broadcast 172.16.8.255
inet6 fe80::230:48ff:fe2d:326a%em0 prefixlen 64 tentative scopeid 0=
x1=20
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
em1: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=3D389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WO=
L_UCAST,WOL_MCAST,WOL_MAGIC>
ether 00:30:48:2d:32:6b
media: Ethernet autoselect
status: no carrier
Starting devd.
Starting Network: em1.
em1: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=3D389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WO=
L_UCAST,WOL_MCAST,WOL_MAGIC>
ether 00:30:48:2d:32:6b
media: Ethernet autoselect
status: no carrier
add net default: gateway 172.16.8.1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
add net fe80::: gateway ::1
add net ff02::: gateway ::1
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/=
lib/compat
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout
Creating and/or trimming log files.
Starting syslogd.
No core dumps found.
Starting rpcbind.
NFS access cache time=3D60
Setting NIS domain: lmdhw.com.
Starting ypbind.
Starting amd.
Clearing /tmp (X related).
Starting mountd.
Starting nfsd.
Starting lpd.
Updating motd:.
Starting ntpd.
Starting powerd.
Starting rsyncd.
Starting cvsupd.
Configuring syscons: blanktime.
Starting sshd.
Starting cron.
Starting background file system checks in 60 seconds.
Tue May 11 21:20:35 PDT 2010
FreeBSD/i386 (freebeast.catwhisker.org) (ttyu0)
login:=20
Fatal trap 12: page fault while in kernel mode
cpuid =3D 0; apic id =3D 00
fault virtual address =3D 0x68
fault code =3D supervisor read, page not present
instruction pointer =3D 0x20:0xc0894306
stack pointer =3D 0x28:0xe9bc6c5c
frame pointer =3D 0x28:0xe9bc6c7c
code segment =3D base 0x0, limit 0xfffff, type 0x1b
=3D DPL 0, pres 1, def32 1, gran 1
processor eflags =3D interrupt enabled, resume, IOPL =3D 0
current process =3D 20 (flowcleaner)
[ thread pid 20 tid 100067 ]
Stopped at _mtx_lock_flags+0x46: movl 0x10(%ebx),%eax
db> bt
Tracing pid 20 tid 100067 td 0xc5a19000
_mtx_lock_flags(58,0,c0cd2efb,570,0,...) at _mtx_lock_flags+0x46
flowtable_free_stale(c0e28ac0,0,c0cd2efb,600,0,...) at flowtable_free_stale=
+0x2fb
flowtable_cleaner(0,e9bc6d38,c0cbf1f8,343,c63172a8,...) at flowtable_cleane=
r+0xd0
fork_exit(c094fa90,0,e9bc6d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip =3D 0, esp =3D 0xe9bc6d70, ebp =3D 0 ---
db>=20
I wish I had better news; sorry.
Peace,
david
--=20
David H. Wolfskill david@catwhisker.org
Depriving a girl or boy of an opportunity for education is evil.
See http://www.catwhisker.org/~david/publickey.gpg for my public key.
--NzX0AQGjRQPusK/O
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (FreeBSD)
iEYEARECAAYFAkvqLXoACgkQmprOCmdXAD1QUwCeOTvc8Tn7yNx2OS4gMlBhLzio
90YAnjNn7nUrAAkcXkE0d53jgpGcrUak
=Yt/K
-----END PGP SIGNATURE-----
--NzX0AQGjRQPusK/O--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100512042430.GH685>