Date: Tue, 17 Oct 2006 12:01:53 GMT From: Marcin Koziej<lolownia@gmail.com> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/104489: dhclient re0 -- panic: mutex Giant not owned at ../../../net/bpf.c:1432 Message-ID: <200610171201.k9HC1rSl064515@www.freebsd.org> Resent-Message-ID: <200610171210.k9HCAKLR067195@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 104489 >Category: kern >Synopsis: dhclient re0 -- panic: mutex Giant not owned at ../../../net/bpf.c:1432 >Confidential: no >Severity: serious >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Oct 17 12:10:20 GMT 2006 >Closed-Date: >Last-Modified: >Originator: Marcin Koziej >Release: FreeBSD 6.2-PRERELEASE #0: Tue Oct 17 00:30:40 CEST 2006 >Organization: >Environment: FreeBSD carnivore 6.2-PRERELEASE FreeBSD 6.2-PRERELEASE #0: Tue Oct 17 00:30:40 CEST 2006 creep@carnivore:/home/src/sys/i386/compile/CRASH i386 >Description: Unread portion of the kernel message buffer: Copyright (c) 1992-2006 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-PRERELEASE #0: Tue Oct 17 12:40:30 CEST 2006 root@carnivore:/home/src/sys/i386/compile/CRASH WARNING: WITNESS option enabled, expect reduced performance. WARNING: debug.mpsafenet forced to 0 as ipsec requires Giant WARNING: MPSAFE network stack disabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Athlon(tm) 64 Processor 3700+ (2401.38-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0xf4a Stepping = 10 Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2> AMD Features=0xe0500800<SYSCALL,NX,MMX+,LM,3DNow+,3DNow> real memory = 536281088 (511 MB) avail memory = 506388480 (482 MB) WITNESS: spin lock hptlock not in order list acpi0: <PTLTD RSDT> on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 acpi_ec0: <Embedded Controller: GPE 0xb> port 0x62,0x66 on acpi0 cpu0: <ACPI CPU> on acpi0 powernow0: <PowerNow! K8> on cpu0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci_link4: BIOS IRQ 11 for 0.17.INTC is invalid pci_link6: BIOS IRQ 10 for 0.10.INTA is invalid pci_link7: BIOS IRQ 10 does not match initial IRQ 11 pci_link4: BIOS IRQ 11 for 0.11.INTB is invalid pci_link5: BIOS IRQ 11 for 0.11.INTC is invalid pci0: <ACPI PCI bus> on pcib0 pcib0: no PRT entry for 0.12.INTA agp0: <VIA 8380 host to PCI bridge> mem 0xd0000000-0xdfffffff at device 0.0 on pci0 pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 nvidia0: <GeForce FX Go5700> mem 0xc1000000-0xc1ffffff,0xe0000000-0xefffffff irq 9 at device 0.0 on pci1 WITNESS: spin lock ctl.mtx_rm not in order list nvidia0: [GIANT-LOCKED] WITNESS: spin lock dev.mtx_rm not in order list ndis0: <INPROCOMM IPN2220 Wireless LAN Card> port 0x1c00-0x1c1f mem 0xc0006000-0xc000601f,0xc0005000-0xc00057ff irq 11 at device 10.0 on pci0 ndis0: [GIANT-LOCKED] ndis0: NDIS API version: 5.1 ndis0: Ethernet address: 00:0e:9b:99:ee:a8 cbb0: <PCI-CardBus Bridge> irq 10 at device 11.0 on pci0 cardbus0: <CardBus bus> on cbb0 pccard0: <16-bit PCCard bus> on cbb0 cbb1: <PCI-CardBus Bridge> irq 9 at device 11.1 on pci0 cardbus1: <CardBus bus> on cbb1 pccard1: <16-bit PCCard bus> on cbb1 fwohci0: <1394 Open Host Controller Interface> mem 0xc0005800-0xc0005fff,0xc0000000-0xc0003fff irq 10 at device 11.2 on pci0 fwohci0: [GIANT-LOCKED] fwohci0: OHCI version 1.10 (ROM=0) fwohci0: No. of Isochronous channels is 4. fwohci0: EUI64 00:0a:e4:05:10:10:5b:ee fwohci0: Phy 1394a available S400, 2 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: <IEEE1394(FireWire) bus> on fwohci0 fwe0: <Ethernet over FireWire> on firewire0 if_fwe0: Fake Ethernet address: 02:0a:e4:10:5b:ee fwe0: Ethernet address: 02:0a:e4:10:5b:ee sbp0: <SBP-2/SCSI over FireWire> on firewire0 fwohci0: Initiate bus reset fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me) firewire0: bus manager 0 (me) re0: <RealTek 8169SB/8110SB Single-chip Gigabit Ethernet> port 0x1000-0x10ff mem 0xc0006400-0xc00064ff irq 11 at device 12.0 on pci0 miibus0: <MII bus> on re0 rgephy0: <RTL8169S/8110S media interface> on miibus0 rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto re0: Ethernet address: 00:0a:e4:a7:d3:4a re0: [FAST] uhci0: <VIA 83C572 USB controller> port 0x1c20-0x1c3f at device 16.0 on pci0 uhci0: [GIANT-LOCKED] usb0: <VIA 83C572 USB controller> on uhci0 usb0: USB revision 1.0 uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1: <VIA 83C572 USB controller> port 0x1c40-0x1c5f at device 16.1 on pci0 uhci1: [GIANT-LOCKED] usb1: <VIA 83C572 USB controller> on uhci1 usb1: USB revision 1.0 uhub1: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered uhci2: <VIA 83C572 USB controller> port 0x1c60-0x1c7f at device 16.2 on pci0 uhci2: [GIANT-LOCKED] usb2: <VIA 83C572 USB controller> on uhci2 usb2: USB revision 1.0 uhub2: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub2: 2 ports with 2 removable, self powered ehci0: <VIA VT6202 USB 2.0 controller> mem 0xc0006800-0xc00068ff at device 16.3 on pci0 ehci0: [GIANT-LOCKED] usb3: EHCI version 1.0 usb3: companion controllers, 2 ports each: usb0 usb1 usb2 usb3: <VIA VT6202 USB 2.0 controller> on ehci0 usb3: USB revision 2.0 uhub3: VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 uhub3: 6 ports with 6 removable, self powered isab0: <PCI-ISA bridge> at device 17.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <VIA 8235 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1c80-0x1c8f at device 17.1 on pci0 ata0: <ATA channel 0> on atapci0 ata1: <ATA channel 1> on atapci0 pci0: <multimedia, audio> at device 17.5 (no driver attached) pci0: <simple comms> at device 17.6 (no driver attached) acpi_acad0: <AC Adapter> on acpi0 battery0: <ACPI Control Method Battery> on acpi0 acpi_lid0: <Control Method Lid Switch> on acpi0 acpi_button0: <Sleep Button> on acpi0 acpi_tz0: <Thermal Zone> on acpi0 acpi_tz1: <Thermal Zone> on acpi0 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: <PS/2 Mouse> irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model Synaptics Touchpad, device ID 0 ppc0: <ECP parallel printer port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/8 bytes threshold ppbus0: <Parallel port bus> on ppc0 plip0: <PLIP network interface> on ppbus0 lpt0: <Printer> on ppbus0 lpt0: Interrupt-driven port ppi0: <Parallel I/O> on ppbus0 sio0 port 0x2f8-0x2ff irq 3 drq 1 flags 0x10 on acpi0 sio0: type 16550A pmtimer0 on isa0 orm0: <ISA Option ROMs> at iomem 0xc0000-0xcffff,0xd8000-0xdbfff,0xdc000-0xdffff on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounter "TSC" frequency 2401379397 Hz quality 800 Timecounters tick every 1.000 msec IPsec: Initialized Security Association Processing. ad0: 95396MB <Seagate ST9100822A 3.01> at ata0-master UDMA100 acd0: DVDR <TSSTcorpCD/DVDW TS-L532A/TI50> at ata1-master UDMA33 cd0 at ata1 bus 0 target 0 lun 0 cd0: <TSSTcorp CD/DVDW TS-L532A TI50> Removable CD-ROM SCSI-0 device cd0: 33.000MB/s transfers cd0: Attempt to query device size failed: NOT READY, Medium not present Trying to mount root from ufs:/dev/ad0s1a <118>Loading configuration files. <118>kernel dumps on /dev/ad0s1b <118>Entropy harvesting: <118> interrupts <118> ethernet <118> point_to_point <118> kickstart <118>. <118>swapon: adding /dev/ad0s1b as swap device <118>Starting powerd. <118>powerd: <118>cannot open pid file <118>: <118>Read-only file system <118>Starting file system checks: <118>/dev/ad0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS <118>/dev/ad0s1a: clean, 85086 free (16558 frags, 8566 blocks, 3.3% fragmentation) <118>/dev/ad0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS <118>/dev/ad0s1e: clean, 820045 free (162205 frags, 82230 blocks, 3.6% fragmentation) <118>/dev/ad0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS <118>/dev/ad0s1d: clean, 807986 free (65442 frags, 92818 blocks, 1.4% fragmentation) <118>/dev/ad0s2e: FILE SYSTEM CLEAN; SKIPPING CHECKS <118>/dev/ad0s2e: clean, 2512288 free (3664 frags, 313578 blocks, 0.0% fragmentation) <118>/dev/ad0s2a: FILE SYSTEM CLEAN; SKIPPING CHECKS <118>/dev/ad0s2a: clean, 355511 free (2383 frags, 44141 blocks, 0.5% fragmentation) <118>/dev/ad0s2b: FILE SYSTEM CLEAN; SKIPPING CHECKS <118>/dev/ad0s2b: clean, 3413627 free (2283 frags, 426418 blocks, 0.0% fragmentation) <118>/dev/ad0s2d: FILE SYSTEM CLEAN; SKIPPING CHECKS <118>/dev/ad0s2d: clean, 292818 free (114 frags, 36588 blocks, 0.0% fragmentation) <118>Mounting local file systems: <118>. <118>Setting hostname: carnivore. <118>hw.syscons.bell: <118>1 <118> -> <118>0 <118> <118>net.inet6.ip6.v6only: <118>1 <118> -> <118>0 <118> <118>hw.psm.synaptics.low_speed_threshold: <118>20 <118> -> <118>10 <118> <118>vfs.usermount: <118>0 <118> -> <118>1 <118> <118>hw.psm.tap_timeout: <118>125000 <118> -> <118>120000 <118> <118>hw.acpi.thermal.polling_rate: <118>10 <118> -> <118>2 <118> <118>re0: no link ... <118>. <118> got link <118>DHCPREQUEST on re0 to 255.255.255.255 port 67 <118> panic: mutex Giant not owned at ../../../net/bpf.c:1432 KDB: enter: panic Dumping 511 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 511MB (130672 pages) 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc04a9bd9 in db_fncall (dummy1=0, dummy2=0, dummy3=1999, dummy4=0xd49d7964 "`j§Ŕ") at ../../../ddb/db_command.c:492 #2 0xc04a9952 in db_command (last_cmdp=0xc0a76164, cmd_table=0x0, aux_cmd_tablep=0xc09eba20, aux_cmd_tablep_end=0xc09eba3c) at ../../../ddb/db_command.c:350 #3 0xc04a9a5a in db_command_loop () at ../../../ddb/db_command.c:458 #4 0xc04abbba in db_trap (type=3, code=0) at ../../../ddb/db_main.c:222 #5 0xc07153f7 in kdb_trap (type=0, code=0, tf=0xd49d7ab0) at ../../../kern/subr_kdb.c:473 #6 0xc093e3da in trap (frame= {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = 1, tf_esi = -1063553714, tf_ebp = -727876872, tf_isp = -727876900, tf_ebx = -727876816, tf_edx = 1, tf_ecx = -1048489984, tf_eax = 18, tf_trapno = 3, tf_err = 0, tf_eip = -1066315520, tf_cs = 32, tf_eflags = 524934, tf_esp = -1063540168, tf_ss = -1063550787}) at ../../../i386/i386/trap.c:594 #7 0xc092a09a in calltrap () at ../../../i386/i386/exception.s:139 #8 0xc0715100 in kdb_enter (msg=0x12 <Address 0x12 out of bounds>) at cpufunc.h:60 #9 0xc06f5e9c in panic (fmt=0xc09b754e "mutex %s not owned at %s:%d") at ../../../kern/kern_shutdown.c:549 #10 0xc06ebc77 in _mtx_assert (m=0xc0a9c980, what=-1048489984, file=0xc09c2ea4 "../../../net/bpf.c", line=1432) at ../../../kern/kern_mutex.c:768 #11 0xc077579f in catchpacket (d=0xc3e56800, pkt=0x12 <Address 0x12 out of bounds>, pktlen=343, snaplen=4294967295, cpfn=0xc07753b0 <bpf_mcopy>, tv=0x12) at ../../../net/bpf.c:1432 #12 0xc0775516 in bpf_mtap (bp=0xc3ce1ac0, m=0xc3de1900) at ../../../net/bpf.c:1343 #13 0xc077c982 in ether_input (ifp=0xc3bdc000, m=0xc3de1900) at ../../../net/if_ethersubr.c:574 #14 0xc0626a22 in re_rxeof (sc=0xc3b84000) at ../../../dev/re/if_re.c:1732 #15 0xc0626e35 in re_int_task (arg=0xc3b84000, npending=1) at ../../../dev/re/if_re.c:1962 #16 0xc071def4 in taskqueue_run (queue=0xc3a57580) at ../../../kern/subr_taskqueue.c:257 #17 0xc071e6f3 in taskqueue_fast_run (dummy=0x0) at ../../../kern/subr_taskqueue.c:435 ---Type <return> to continue, or q <return> to quit--- #18 0xc06dc97b in ithread_execute_handlers (p=0xc3a5f430, ie=0xc3a57500) at ../../../kern/kern_intr.c:682 #19 0xc06dcb33 in ithread_loop (arg=0xc3b29270) at ../../../kern/kern_intr.c:765 #20 0xc06db384 in fork_exit (callout=0xc06dca60 <ithread_loop>, arg=0x12, frame=0x12) at ../../../kern/kern_fork.c:821 #21 0xc092a0fc in fork_trampoline () at ../../../i386/i386/exception.s:208 (kgdb) ps pid proc uid ppid pgrp flag stat comm wchan 408 c3e54c90 0 407 49 000002 1 dhclient select c0ae8224 407 c3e54218 65 398 49 004102 1 dhclient nanslp c0a9d26c 398 c3e54000 0 217 49 004002 1 sh wait c3e54000 217 c3bd8648 0 49 49 000002 1 sh wait c3bd8648 183 c3e54648 0 1 183 000000 1 adjkerntz pause c3e5467c 92 c3d16430 0 1 92 000000 1 powerd nanslp c0a9d26c 49 c3bd8860 0 1 49 004002 1 sh wait c3bd8860 48 c3bd8a78 0 0 0 000204 1 softdepflush sdflush c0afedf4 47 c3bd8c90 0 0 0 000204 1 vnlru vlruwt c3bd8c90 46 c3d15000 0 0 0 000204 1 syncer syncer c0a9cfdc 45 c3d15218 0 0 0 000204 1 bufdaemon psleep c0ae878c 44 c3d15430 0 0 0 00020c 1 pagezero pgzero c0affd64 43 c3d15648 0 0 0 000204 1 vmdaemon psleep c0aff8b4 42 c3d15860 0 0 0 000204 1 pagedaemon psleep c0aff870 41 c3d15a78 0 0 0 000204 1 swi0: sio 40 c3d15c90 0 0 0 000204 1 irq7: ppc0 39 c3d16000 0 0 0 000204 1 irq12: psm0 38 c3b4cc90 0 0 0 000204 1 irq1: atkbd0 37 c3bd7000 0 0 0 000204 1 acpi_cooling0 cooling c3cf9cd4 36 c3bd7218 0 0 0 000204 1 acpi_thermal tzpoll c1182e18 35 c3bd7430 0 0 0 000204 1 irq15: ata1 34 c3bd7648 0 0 0 000204 1 irq14: ata0 33 c3bd7860 0 0 0 000204 1 usb3 usbevt c3b83a10 32 c3bd7a78 0 0 0 000204 1 usb2 usbevt c3ced210 31 c3bd7c90 0 0 0 000204 1 usb1 usbevt c3cce210 30 c3bd8000 0 0 0 000204 1 usbtask usbtsk c0a8cea4 29 c3bd8218 0 0 0 000204 1 usb0 usbevt c3ccf210 28 c3bd8430 0 0 0 000204 1 cbb1 cbb cv c3b823f0 ---Type <return> to continue, or q <return> to quit--- 27 c3aa7648 0 0 0 000204 1 cbb0 cbb cv c3b863f0 26 c3aa7860 0 0 0 000204 1 irq10: cbb0 uhci1* 25 c3aa7a78 0 0 0 000204 1 irq11: ndis0 re0+ 24 c3aa7c90 0 0 0 000204 1 irq9: nvidia0 cbb* 23 c3b4c000 0 0 0 000204 1 swi6: Giant taskq 22 c3b4c218 0 0 0 000204 1 swi6: task queue 9 c3b4c430 0 0 0 000204 1 kqueue taskq - c3b4b480 8 c3b4c648 0 0 0 000204 1 acpi_task_2 - c3b4b500 7 c3b4c860 0 0 0 000204 1 acpi_task_1 - c3b4b500 6 c3b4ca78 0 0 0 000204 1 acpi_task_0 - c3b4b500 21 c3a5f218 0 0 0 000204 1 swi2: cambio 20 c3a5f430 0 0 0 000204 1 swi5: + 5 c3a5f648 0 0 0 000204 1 thread taskq - c3a57600 19 c3a5f860 0 0 0 000204 1 Windows Workitem 3 KeWFS d5be2c7c 18 c3a5fa78 0 0 0 000204 1 Windows Workitem 2 KeWFS d5bd9c7c 17 c3a5fc90 0 0 0 000204 1 Windows Workitem 1 KeWFS d5bd0c7c 16 c3aa7000 0 0 0 000204 1 Windows Workitem 0 KeWFS d5bc7c7c 15 c3aa7218 0 0 0 000204 1 Windows DPC 0 KeWFS d5bbec78 14 c3aa7430 0 0 0 000204 1 yarrow - c0a8abc0 4 c3a58000 0 0 0 000204 1 g_down - c0a8d6e8 3 c3a58218 0 0 0 000204 1 g_up - c0a8d6e4 2 c3a58430 0 0 0 000204 1 g_event - c0a8d6dc 13 c3a58648 0 0 0 000204 1 swi3: vm 12 c3a58860 0 0 0 00020c 1 swi4: clock sio 11 c3a58a78 0 0 0 000204 1 swi1: net 10 c3a58c90 0 0 0 00020c 1 idle 1 c3a5f000 0 0 1 004200 1 init wait c3a5f000 0 c0a8d7e0 0 0 0 000200 1 swapper (kgdb) kldstat Id Refs Address Size Name 1 23 0xc0400000 811390 kernel 2 3 0xc0c12000 1f074 linux.ko 3 1 0xc0c32000 40fc powernow.ko 4 1 0xc0c37000 4a628c nvidia.ko 5 1 0xc10de000 29790 i2220ntx_sys.ko 6 3 0xc1108000 1a81c ndis.ko 7 2 0xc1123000 d918 if_ndis.ko 8 1 0xc1131000 649a4 acpi.ko 9 1 0xc3ea3000 7000 linprocfs.ko Kernel is almost a GENERIC: --- /usr/src/sys/i386/conf/GENERIC Sat Oct 14 12:14:38 2006 +++ /usr/src/sys/i386/conf/CRASH Tue Oct 17 12:34:59 2006 @@ -1,35 +1,15 @@ machine i386 -cpu I486_CPU -cpu I586_CPU cpu I686_CPU -ident GENERIC +ident CRASH # To statically compile in device wiring instead of /boot/device.hints #hints "GENERIC.hints" # Default places to look for devices. makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols -options SCHED_4BSD # 4BSD scheduler +options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options INET6 # IPv6 communications protocols @@ -56,8 +36,27 @@ +options IPSEC +options IPSEC_ESP + +options KDB +options DDB +options GDB +options WITNESS +options INVARIANT_SUPPORT +options INVARIANTS +makeoptions DEBUG=-g +options KTR + device apic # I/O APIC # Bus support. @@ -138,8 +132,6 @@ device atkbd # AT keyboard device psm # PS/2 mouse -device kbdmux # keyboard multiplexer - device vga # VGA video card driver device splash # Splash screen and screen saver support @@ -191,7 +183,6 @@ +device vlan # Pseudo devices. device loop # Network loopback +device mem # Memory and kernel memory devices +device io # I/O device device random # Entropy device device ether # Ethernet support device sl # Kernel SLIP @@ -258,7 +248,13 @@ # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter +device pf +device pflog >How-To-Repeat: Just use dhclient to setup a re0 card. >Fix: >Release-Note: >Audit-Trail: >Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200610171201.k9HC1rSl064515>