From owner-freebsd-stable@FreeBSD.ORG Mon Nov 21 23:54:43 2005 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 158EF16A42C for ; Mon, 21 Nov 2005 23:54:43 +0000 (GMT) (envelope-from gcr+freebsd-stable@tharned.org) Received: from mx13.sac.fedex.com (mx13.sac.fedex.com [199.81.197.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id C5DC243D7B for ; Mon, 21 Nov 2005 23:54:24 +0000 (GMT) (envelope-from gcr+freebsd-stable@tharned.org) Received: from inet04.prod.fedex.com (inet04.prod.fedex.com [199.81.10.44]) by mx13.sac.fedex.com (8.12.9p2/8.12.9) with ESMTP id jALNsBKZ043841 for ; Mon, 21 Nov 2005 17:54:19 -0600 (CST) (envelope-from gcr+freebsd-stable@tharned.org) Received: from w10.sac.fedex.com (w10.sac.fedex.com [161.135.204.136]) by inet04.prod.fedex.com (8.12.11/8.12.11) with ESMTP id jALNsAxI010919 for ; Mon, 21 Nov 2005 17:54:11 -0600 (CST) Received: from w10.sac.fedex.com (gcr@localhost [127.0.0.1]) by w10.sac.fedex.com (8.13.4/8.13.4) with ESMTP id jALNsAXJ097724 for ; Mon, 21 Nov 2005 17:54:10 -0600 (CST) (envelope-from gcr+freebsd-stable@tharned.org) Received: from localhost (gcr@localhost) by w10.sac.fedex.com (8.13.4/8.13.4/Submit) with ESMTP id jALNs9Qs097721 for ; Mon, 21 Nov 2005 17:54:09 -0600 (CST) (envelope-from gcr+freebsd-stable@tharned.org) Date: Mon, 21 Nov 2005 17:54:09 -0600 (CST) From: Greg Rivers Sender: gcr@fedex.com To: freebsd-stable@freebsd.org Message-ID: <20051121164139.T48994@w10.sac.fedex.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Subject: Recurring problem: processes block accessing UFS file system X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2005 23:54:43 -0000 I've recently put up three busy email relay hosts running 6.0-STABLE. Performance is excellent except for a nagging critical issue that keeps cropping up. /var/spool is its own file system mounted on a geom stripe of four BSD partitions (details below). Once every two or three days all the processes accessing /var/spool block forever in disk wait. All three machines suffer this problem. No diagnostic messages are generated and the machines continue running fine otherwise, but a reboot is required to clear the condition. This problem occurs during normal operation, but is particularly likely to occur during a backup when dump makes a snapshot. There doesn't appear to be a problem with gstripe, as gstripe status is "UP" and I can read the raw device just fine while processes continue to block on the file system. I tried running a kernel with WITNESS and DIAGNOSTIC, but these options shed no light. If I catch the problem early enough I can break successfully into kdb; otherwise, if too many processes stack up, the machine hangs going into kdb and must be power-cycled. I'd appreciate any insight anyone may have into this problem or advise on turning this report into a coherent PR. -- Greg Rivers $ uname -a FreeBSD mx17.sac.fedex.com 6.0-STABLE FreeBSD 6.0-STABLE #0: Fri Nov 18 23:34:59 CST 2005 root@mx17.sac.fedex.com:/usr/obj/usr/src/sys/MX i386 (kernel built from sources current at time of build) $ df -h /var/spool Filesystem Size Used Avail Capacity Mounted on /dev/stripe/gs0 260G 2.2G 237G 1% /var/spool $ mount /dev/da0a on / (ufs, local, soft-updates) devfs on /dev (devfs, local) /dev/md0 on /tmp (ufs, asynchronous, local, noatime) /dev/mirror/gm0 on /var (ufs, local, soft-updates) /dev/mirror/gm1d on /var/log (ufs, local, soft-updates) /dev/stripe/gs0 on /var/spool (ufs, local, noatime, soft-updates) /dev/md1 on /var/spool/mqueue/.hoststat (ufs, asynchronous, local, noatime) /dev/mirror/gm1a on /usr (ufs, local, soft-updates) /dev/mirror/gm2 on /home (ufs, local, soft-updates) devfs on /var/named/dev (devfs, local) $ gstripe list Geom name: gs0 State: UP Status: Total=4, Online=4 Type: AUTOMATIC Stripesize: 131072 ID: 3224140609 Providers: 1. Name: stripe/gs0 Mediasize: 283460501504 (264G) Sectorsize: 512 Mode: r1w1e1 Consumers: 1. Name: da0d Mediasize: 70865144320 (66G) Sectorsize: 512 Mode: r1w1e2 Number: 0 2. Name: da1d Mediasize: 70865144320 (66G) Sectorsize: 512 Mode: r1w1e2 Number: 2 3. Name: da2d Mediasize: 70865144320 (66G) Sectorsize: 512 Mode: r1w1e2 Number: 1 4. Name: da3d Mediasize: 70865144320 (66G) Sectorsize: 512 Mode: r1w1e2 Number: 3 (Host is a HP/Compaq DL380 G4) $ dmesg Copyright (c) 1992-2005 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 6.0-STABLE #0: Fri Nov 18 23:34:59 CST 2005 root@mx17.sac.fedex.com:/usr/obj/usr/src/sys/MX ACPI APIC Table: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 3.40GHz (3400.14-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf41 Stepping = 1 Features=0xbfebfbff Features2=0x649d> AMD Features=0x20000000 Hyperthreading: 2 logical CPUs real memory = 3758043136 (3583 MB) avail memory = 3683098624 (3512 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard ioapic3 irqs 72-95 on motherboard ioapic4 irqs 96-119 on motherboard npx0: [FAST] npx0: on motherboard npx0: INT 16 interface acpi0: on motherboard acpi0: Power Button (fixed) pci_link0: irq 7 on acpi0 pci_link1: irq 5 on acpi0 pci_link2: irq 5 on acpi0 pci_link3: irq 5 on acpi0 pci_link4: irq 0 on acpi0 pci_link5: irq 5 on acpi0 pci_link6: irq 5 on acpi0 pci_link7: irq 7 on acpi0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 pcib0: on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci2: on pcib1 pcib2: at device 0.0 on pci2 pci3: on pcib2 bge0: mem 0xfddf0000-0xfddfffff irq 25 at device 1.0 on pci3 miibus0: on bge0 brgphy0: on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto bge0: Ethernet address: 00:13:21:09:89:79 bge1: mem 0xfdde0000-0xfddeffff irq 26 at device 1.1 on pci3 miibus1: on bge1 brgphy1: on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto bge1: Ethernet address: 00:13:21:09:89:78 pcib3: at device 0.2 on pci2 pci4: on pcib3 ciss0: port 0x4000-0x40ff mem 0xfdef0000-0xfdef1fff,0xfde80000-0xfdebffff irq 51 at device 3.0 on pci4 ciss0: [GIANT-LOCKED] pcib4: at device 6.0 on pci0 pci5: on pcib4 pcib5: at device 0.0 on pci5 pci6: on pcib5 pcib6: at device 0.2 on pci5 pci10: on pcib6 em0: port 0x5000-0x503f mem 0xfdfe0000-0xfdffffff,0xfdf80000-0xfdfbffff irq 97 at device 1.0 on pci10 em0: Ethernet address: 00:11:0a:56:07:2e em1: port 0x5040-0x507f mem 0xfdf60000-0xfdf7ffff irq 98 at device 1.1 on pci10 em1: Ethernet address: 00:11:0a:56:07:2f pcib7: at device 30.0 on pci0 pci1: on pcib7 pci1: at device 3.0 (no driver attached) pci1: at device 4.0 (no driver attached) pci1: at device 4.2 (no driver attached) isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x500-0x50f at device 31.1 on pci0 ata0: on atapci0 ata1: on atapci0 acpi_tz0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] sio0: port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A, console orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xcbfff,0xcc000-0xcd7ff,0xcd800-0xcefff,0xee000-0xeffff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x100> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 sio1 at port 0x2f8-0x2ff irq 3 on isa0 sio1: type 16550A Timecounters tick every 1.000 msec acd0: DVDROM at ata0-master PIO4 da0 at ciss0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-0 device da0: 135.168MB/s transfers da0: 69459MB (142253280 512 byte sectors: 255H 32S/T 17433C) da1 at ciss0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-0 device da1: 135.168MB/s transfers da1: 69459MB (142253280 512 byte sectors: 255H 32S/T 17433C) da2 at ciss0 bus 0 target 2 lun 0 da2: Fixed Direct Access SCSI-0 device da2: 135.168MB/s transfers da2: 69459MB (142253280 512 byte sectors: 255H 32S/T 17433C) da3 at ciss0 bus 0 target 3 lun 0 da3: Fixed Direct Access SCSI-0 device da3: 135.168MB/s transfers da3: 69459MB (142253280 512 byte sectors: 255H 32S/T 17433C) da4 at ciss0 bus 0 target 4 lun 0 da4: Fixed Direct Access SCSI-0 device da4: 135.168MB/s transfers da4: 69459MB (142253280 512 byte sectors: 255H 32S/T 17433C) da5 at ciss0 bus 0 target 5 lun 0 da5: Fixed Direct Access SCSI-0 device da5: 135.168MB/s transfers da5: 69459MB (142253280 512 byte sectors: 255H 32S/T 17433C) SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! GEOM_STRIPE: Device gs0 created (id=3224140609). GEOM_STRIPE: Disk da0d attached to gs0. GEOM_MIRROR: Device gm2 created (id=3517057742). GEOM_MIRROR: Device gm2: provider da1a detected. GEOM_STRIPE: Disk da1d attached to gs0. GEOM_STRIPE: Disk da2d attached to gs0. GEOM_MIRROR: Device gm2: provider da3a detected. GEOM_MIRROR: Device gm2: provider da3a activated. GEOM_MIRROR: Device gm2: provider da1a activated. GEOM_MIRROR: Device gm2: provider mirror/gm2 launched. GEOM_STRIPE: Disk da3d attached to gs0. GEOM_STRIPE: Device gs0 activated. GEOM_MIRROR: Device gm0 created (id=3319228670). GEOM_MIRROR: Device gm0: provider da4a detected. GEOM_MIRROR: Device gm1 created (id=283379314). GEOM_MIRROR: Device gm1: provider da4d detected. GEOM_MIRROR: Device gm0: provider da5a detected. GEOM_MIRROR: Device gm0: provider da5a activated. GEOM_MIRROR: Device gm0: provider da4a activated. GEOM_MIRROR: Device gm0: provider mirror/gm0 launched. GEOM_MIRROR: Device gm1: provider da5d detected. GEOM_MIRROR: Device gm1: provider da5d activated. GEOM_MIRROR: Device gm1: provider da4d activated. GEOM_MIRROR: Device gm1: provider mirror/gm1 launched. Trying to mount root from ufs:/dev/da0a em1: link state changed to UP I obtained the following process listing and traces from kdb. I traced mksnap_ffs which was blocked in "ufs", and two random sendmail processes that were blocked in "ufs" and "suspfs" respectively. ... Expensive timeout(9) function: 0xc04c9082(0xc4c97d80) 0.002908205 s ... # set -o vi # sync;sync # KDB: enter: Line break on console timeout stopping cpus [thread pid 11 tid 100005 ] Stopped at kdb_enter+0x30: leave db> ps pid proc uid ppid pgrp flag stat wmesg wchan cmd 13273 c9301a3c 11521 13271 13270 0004000 [SLPQ piperd 0xc56897f8][SLP] awk 13272 c9301c48 11521 13271 13270 0004100 [SLPQ ufs 0xc502ce28][SLP] sendmail 13271 c9266a3c 11521 13270 13270 0004000 [SLPQ piperd 0xc4ff3cc0][SLP] ksh 13270 c53d8418 11521 13269 13270 0004000 [SLPQ wait 0xc53d8418][SLP] sh 13269 c92cca3c 0 689 689 0000000 [SLPQ piperd 0xc530bcc0][SLP] cron 13267 c9302000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13266 c92ccc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13265 c92ce000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13264 c92ce20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13263 c92ce418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13262 c92ce624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13261 c930220c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13260 c92ce830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13259 c9302418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13258 c926a000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13257 c9302624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13256 c926a20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13255 c9302830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13254 c926a418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13253 c926a624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13252 c926a830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13251 c9302a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13250 c8feec48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13249 c60fdc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13248 c926aa3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13247 c92cea3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13246 c92cec48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13245 c926ac48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13244 c926d000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13243 c92cf000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13242 c926d20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13241 c92cf20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13240 c91a8000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13239 c60f6418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13238 c9119418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13237 c926d418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13235 c926d624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13234 c9190a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13233 c926d830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13232 c90a4830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13231 c9190624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13230 c926da3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13229 c586ca3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13228 c918a000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13227 c90a4c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13226 c926dc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13225 c918a20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13224 c918a418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13223 c918a624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13222 c4fa2a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13221 c9193830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13220 c926e000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13219 c918a830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13218 c9193a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13217 c9193c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13216 c918aa3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13214 c926e418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13213 c90a4418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13212 c926e624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13211 c91a6000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13210 c926e830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13209 c9190830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13208 c501da3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13207 c91a620c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13206 c91aa000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13205 c90a4624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13204 c91a6418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13203 c918ac48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13202 c91a6624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13201 c9190000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13199 c91a6830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13198 c919020c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13197 c9190418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13196 c91a6a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13195 c90a4a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13194 c91a6c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13186 c501d000 11521 13181 13178 0004000 [SLPQ piperd 0xc4ff0000][SLP] awk 13185 c587420c 11521 13181 13178 0004100 [SLPQ ufs 0xc502ce28][SLP] sendmail 13181 c9191000 11521 13178 13178 0004000 [SLPQ piperd 0xc8b44990][SLP] ksh 13178 c9191418 11521 13175 13178 0004000 [SLPQ wait 0xc9191418][SLP] sh 13175 c91a820c 0 689 689 0000000 [SLPQ piperd 0xc4fef990][SLP] cron 13174 c91a8418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13173 c9191624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13172 c9119624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13171 c91a8624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13170 c91a8830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13169 c91a8a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13168 c91a8c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13167 c91a9000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13166 c91a920c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13165 c91a9418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13164 c9119830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13163 c91a9624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13162 c9191830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13161 c9119a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13160 c9119c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13159 c91a9830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13158 c9191a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13157 c9122000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13156 c91a9a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13155 c91a9c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13153 c91aa20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13152 c9191c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13151 c91aa418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13150 c91aa624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13149 c91aa830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13148 c9192000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13147 c912220c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13146 c9122418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13145 c919220c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13144 c9122624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13143 c9192418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13142 c9122830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13141 c9192624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13140 c91aaa3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13139 c9122a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13138 c91aac48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13137 c9192830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13136 c9192a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13135 c9192c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13134 c91af000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13133 c9050c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13132 c9122c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13131 c9193000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13130 c909c000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13129 c919320c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13128 c9193418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13127 c909c20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13126 c9193624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13125 c8ff6418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13124 c8ff6624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13123 c8ff6830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13122 c9123000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13121 c909c418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13120 c8ff6a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13119 c912320c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13118 c909c624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13117 c8ff6c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13116 c909c830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13115 c909ca3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13114 c9123418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13113 c9123624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13112 c9123830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13111 c909cc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13110 c9123a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13109 c9123c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13108 c909e000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13107 c9124000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13106 c9045000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13105 c909e20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13104 c909e418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13103 c912420c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13102 c904520c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13101 c909e624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13100 c909e830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13099 c9124418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13098 c909ea3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13097 c909ec48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13096 c90a1000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13095 c9124624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13094 c90a120c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13093 c90a1418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13092 c90a1624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13091 c9045418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13090 c9045624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13089 c9045830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13088 c9045a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13087 c9124830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13086 c9124a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13085 c9124c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13084 c90a1830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13083 c90a1a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13082 c8fa8a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13081 c90a1c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13080 c90a2000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13079 c90a220c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13078 c8fa8c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13077 c90a2418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13076 c8fee000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13075 c9045c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13074 c90a2624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13073 c8fee20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13072 c8fee418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13071 c904c000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13070 c90a2830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13069 c8fee624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13068 c90a2a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13067 c90a2c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13066 c90a4000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13065 c90a420c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13064 c8fee830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13063 c904c20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13062 c904c418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13061 c904c624 11521 13059 13057 0004000 [SLPQ piperd 0xc5689990][SLP] awk 13060 c8feea3c 11521 13059 13057 0004100 [SLPQ ufs 0xc502ce28][SLP] sendmail 13059 c904c830 11521 13057 13057 0004000 [SLPQ piperd 0xc8b44198][SLP] ksh 13057 c904ca3c 11521 13055 13057 0004000 [SLPQ wait 0xc904ca3c][SLP] sh 13055 c8ff2000 0 689 689 0000000 [SLPQ piperd 0xc4ff3b28][SLP] cron 13053 c904cc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13052 c8ff220c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13051 c904d000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13050 c904d20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13049 c533520c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13048 c8ff2418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13047 c8ef6418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13046 c904d418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13045 c53d1830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13044 c8ef6624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13043 c904d624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13042 c8ff2624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13041 c8ff2830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13040 c904d830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13039 c904da3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13038 c904dc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13037 c8ef6830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13036 c8ef6a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13035 c8ff2a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13034 c8ff2c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13033 c9050000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13032 c8ef6c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13031 c8fa1000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13030 c8fa120c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13029 c8fa1418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13028 c8ff3000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13027 c905020c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13026 c9050418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13025 c8fa1624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13024 c8ff320c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13023 c8fa1830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13022 c8ff3418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13021 c8ff3624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13020 c8fa1a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13019 c60fc830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13018 c9050624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13017 c8ff3830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13016 c8fa1c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13015 c8fa4000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13014 c8ff3a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13013 c9050830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13012 c8ff3c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13011 c8ff5000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13010 c502320c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13009 c8ff520c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13008 c9050a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13007 c8fa420c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13006 c8fa4418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13005 c8ff5418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13004 c8fa4624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13003 c5023830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13002 c8ff5624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13001 c8ff5830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 13000 c8fa4830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12999 c8fa4a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12998 c4fed830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12997 c8ff5a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12996 c8fa4c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12995 c8ff5c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12994 c8ff6000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12993 c586f000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12992 c8fa5000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12991 c8fa520c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12990 c586c000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12989 c8fa5418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12988 c4fa1418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12987 c8fa5624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12986 c8ff620c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12985 c8fa5830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12984 c8eee000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12983 c8eee20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12982 c8eee418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12981 c5550624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12980 c8fa5a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12979 c60fea3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12978 c8fa5c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12977 c8fa8000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12976 c8fa820c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12975 c5588418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12974 c8eee624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12973 c8fa8418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12972 c8fa8624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12971 c5335c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12970 c8fa8830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12969 c60fec48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12968 c8eee830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12967 c8eeea3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12966 c8eac000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12965 c8eeec48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12964 c4d7ac48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12963 c4fa1000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12962 c8ef3000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12961 c8ef320c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12960 c8ef3418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12959 c8eac20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12958 c8ef3624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12957 c501d20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12956 c8ef3830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12955 c8eac418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12954 c8ef3a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12953 c8eac624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12952 c5024000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12951 c8ef3c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12950 c586c830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12949 c8ef4000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12948 c8ef420c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12947 c8eac830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12946 c8eaca3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12945 c8eacc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12944 c8eaf000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12943 c8eaf20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12942 c8eaf418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12941 c8ef4418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12940 c8eaf624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12939 c8ef4624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12938 c8ef4830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12937 c4f9d624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12936 c8eaf830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12935 c8eafa3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12934 c8eafc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12933 c8eb1000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12932 c4fa220c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12931 c8eb120c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12930 c559020c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12929 c8eb1418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12928 c8eb1624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12927 c8ef4a3c 11521 12925 12924 0004000 [SLPQ piperd 0xc4ff4cc0][SLP] awk 12926 c8ef4c48 11521 12925 12924 0004100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12925 c554b000 11521 12924 12924 0004000 [SLPQ piperd 0xc4ff3990][SLP] ksh 12924 c586cc48 11521 12923 12924 0004000 [SLPQ wait 0xc586cc48][SLP] sh 12923 c8ef5000 0 689 689 0000000 [SLPQ piperd 0xc501ccc0][SLP] cron 12922 c532e830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12921 c4fa1a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12920 c53d9830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12919 c53d1418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12918 c8ef520c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12917 c8ef5418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12916 c8ef5624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12915 c53d9418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12914 c8ef5830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12913 c8ef5a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12912 c8ef5c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12911 c8eb1830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12910 c8ef6000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12909 c8eb1a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12908 c8ef620c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12907 c586fc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12906 c60f6000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12905 c8eb1c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12904 c4f9d000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12903 c5873c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12902 c5023a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12901 c4f9da3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12900 c5024c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12899 c8eb3000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12898 c4f9d20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12897 c5586418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12896 c8eb320c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12895 c8eb3418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12894 c8eb3624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12893 c8eb3830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12892 c8eb3a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12891 c8eb3c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12890 c4d7a418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12889 c5873418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12888 c5551830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12887 c501d418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12886 c53d820c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12885 c586fa3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12884 c5023c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12883 c532e000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12882 c5335830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12881 c53d8c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12880 c4fec20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12879 c60fd20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12878 c586c418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12877 c558620c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12876 c5550418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12875 c555020c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12874 c4d76000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12873 c532ea3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12872 c5874418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12871 c532ec48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12870 c5339c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12869 c4fec624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12868 c5023418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12867 c60fe830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12866 c4d7aa3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12865 c60fd830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12864 c586f20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12863 c5551418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12862 c53d1c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12861 c60fe000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12860 c53d1a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12859 c60fe624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12858 c532e418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12857 c5873000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12856 c5024418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12855 c554b20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12854 c5550c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12853 c5874000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12852 c4fed418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12851 c554bc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12850 c4fa1c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12849 c5335000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12847 c5874830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12846 c4d7a624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12845 c53d1624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12844 c5586624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12843 c60f6a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12842 c4fec830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12841 c5588c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12840 c4fa1624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12839 c5550830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12838 c53d8624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12837 c60fc000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12836 c60fc20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12835 c5590c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12834 c5588624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12833 c60fe20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12832 c5335418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12831 c5590418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12830 c5335624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12829 c60f620c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12828 c5586000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12827 c586c20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12826 c4fec000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12825 c5873624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12824 c60fcc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12823 c5588830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12822 c4fa2624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12821 c60fd624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12820 c5586c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12819 c53d8830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12818 c587320c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12817 c53d120c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12816 c60fda3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12815 c60fc624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12814 c5551624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12813 c4f9dc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12812 c53d9624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12811 c4fa2418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12810 c60f6624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12809 c60f6c48 100 673 673 0000100 [SLPQ ufs 0xc53db5a8][SLP] sendmail 12808 c53d8a3c 100 673 673 0000100 [SLPQ ufs 0xc53e6388][SLP] sendmail 12807 c554b624 100 673 673 0000100 [SLPQ ufs 0xc53db388][SLP] sendmail 12806 c5588000 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail 12805 c533920c 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail 12804 c5339000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12803 c5339624 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail 12801 c554ba3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12800 c60fd418 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail 12799 c4fa2c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12798 c558820c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12797 c5551a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12788 c5024624 11521 12784 12781 0004000 [SLPQ piperd 0xc501c4c8][SLP] awk 12787 c5590830 11521 12784 12781 0004100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail 12784 c4fe8c48 11521 12781 12781 0004000 [SLPQ piperd 0xc69167f8][SLP] ksh 12781 c5551000 11521 12778 12781 0004000 [SLPQ wait 0xc5551000][SLP] sh 12778 c532e624 0 689 689 0000000 [SLPQ piperd 0xc56894c8][SLP] cron 12777 c5550a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12776 c60fe418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12775 c5874c48 100 1 12775 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail 12773 c5023624 100 673 673 0000100 [SLPQ ufs 0xc53f1e28][SLP] sendmail 12772 c5590624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12771 c5586830 100 1 12771 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12769 c4d7a830 100 1 12769 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail 12765 c53d9a3c 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail 12762 c4f9d418 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail 12761 c5586a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12760 c554b830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12759 c4fe8a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12758 c53d1000 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail 12752 c4fe8418 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail 12750 c53d9c48 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail 12732 c5874a3c 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail 12656 c5339830 100 1 12656 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12559 c5588a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12542 c554b418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12500 c586f830 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail 12463 c4fed20c 100 1 12463 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12454 c586f418 100 673 673 0000100 [SLPQ select 0xc067dac4][SLP] sendmail 12384 c555120c 100 1 12384 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 12079 c4fa1830 100 1 12079 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 11910 c586c624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 11707 c4fec418 11008 11706 11151 0004102 [SLPQ ufs 0xc502ce28][SLP] mksnap_ffs 11706 c5024a3c 11008 11151 11151 0004002 [SLPQ wait 0xc5024a3c][SLP] sh 11654 c60fd000 100 673 673 0000100 [SLPQ select 0xc067dac4][SLP] sendmail 11646 c5873a3c 100 673 673 0000100 [SLPQ select 0xc067dac4][SLP] sendmail 11161 c60fc418 11008 11107 11161 0004002 [SLPQ ttyin 0xc8bf1410][SLP] ksh 11151 c501dc48 11008 11110 11151 0004002 [SLPQ wait 0xc501dc48][SLP] dump 11110 c60f6830 11008 11107 11110 0004002 [SLPQ pause 0xc60f6864][SLP] ksh 11107 c4d7620c 11008 11106 11107 0000100 [SLPQ select 0xc067dac4][SLP] screen 11106 c501d830 11008 5725 11106 0004102 [SLPQ pause 0xc501d864][SLP] screen 10540 c4fed624 100 1 10540 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 9815 c4fa2830 100 673 673 0000100 [SLPQ select 0xc067dac4][SLP] sendmail 9623 c4f9d830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 8980 c4fecc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 8719 c5873830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 7925 c53d9000 100 673 673 0000100 [SLPQ select 0xc067dac4][SLP] sendmail 7149 c4fedc48 100 673 673 0000100 [SLPQ select 0xc067dac4][SLP] sendmail 6972 c586f624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 5725 c5024830 11008 5724 5725 0004002 [SLPQ pause 0xc5024864][SLP] ksh 5724 c60fca3c 11008 5719 5719 0000100 [SLPQ select 0xc067dac4][SLP] sshd 5719 c4fa120c 0 668 5719 0004100 [SLPQ sbwait 0xc588d900][SLP] sshd 5077 c5335a3c 100 673 673 0000100 [SLPQ select 0xc067dac4][SLP] sendmail 2513 c502420c 0 2512 2513 0004002 [SLPQ ttyin 0xc4da1810][SLP] ksh 2512 c4fe8000 0 1 2512 0004102 [SLPQ wait 0xc4fe8000][SLP] login 2511 c5550000 0 1 2511 0004002 [SLPQ ttyin 0xc4d97010][SLP] getty 2510 c5590000 0 1 2510 0004002 [SLPQ ttyin 0xc4da1010][SLP] getty 2509 c4feda3c 0 1 2509 0004002 [SLPQ ttyin 0xc4da3810][SLP] getty 2508 c53d920c 0 1 2508 0004002 [SLPQ ttyin 0xc4da1410][SLP] getty 2497 c5590a3c 0 1 2497 0000000 [SLPQ select 0xc067dac4][SLP] inetd 689 c4fed000 0 1 689 0000000 [SLPQ nanslp 0xc06307ac][SLP] cron 677 c53d8000 25 1 677 0000100 [SLPQ pause 0xc53d8034][SLP] sendmail 676 c5339a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 673 c5339418 0 1 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail 668 c532e20c 0 1 668 0000100 [SLPQ select 0xc067dac4][SLP] sshd 650 c4fe8830 0 1 650 0000000 [SLPQ select 0xc067dac4][SLP] ntpd 558 c5023000 53 1 558 0000100 [SLPQ select 0xc067dac4][SLP] named 497 c4fe820c 0 1 497 0000000 [SLPQ select 0xc067dac4][SLP] syslogd 460 c4feca3c 0 1 460 0000000 [SLPQ select 0xc067dac4][SLP] devd 223 c4fe8624 0 0 0 0000204 [SLPQ mdwait 0xc5017000][SLP] md1 214 c501d624 0 0 0 0000204 [SLPQ mdwait 0xc4fc5000][SLP] md0 155 c4fa2000 0 0 0 0000204 [SLPQ m:w1 0xc4f43400][SLP] g_mirror gm1 154 c4d76418 0 0 0 0000204 [SLPQ m:w2 0xc4f9ce00][SLP] g_mirror gm0 153 c4d76624 0 0 0 0000204 [SLPQ m:w2 0xc4dc1600][SLP] g_mirror gm2 152 c4d76830 0 0 0 0000204 [SLPQ - 0xe8ccecfc][SLP] schedcpu 151 c4d76a3c 0 0 0 0000204 [SLPQ vlruwt 0xc4d76a3c][SLP] vnlru 150 c4d76c48 0 0 0 0000204 [SLPQ syncer 0xc063051c][SLP] syncer 149 c4d79000 0 0 0 0000204 [SLPQ psleep 0xc067e00c][SLP] bufdaemon 148 c4d7920c 0 0 0 0000204 [SLPQ pollid 0xc062fb60][SLP] idlepoll 147 c4d79418 0 0 0 000020c [SLPQ pgzero 0xc06817a4][SLP] pagezero 146 c4d79624 0 0 0 0000204 [SLPQ psleep 0xc06812f4][SLP] vmdaemon 145 c4d79830 0 0 0 0000204 [SLPQ psleep 0xc06812b0][SLP] pagedaemon 144 c4d79a3c 0 0 0 0000204 [IWAIT] swi0: sio 143 c4d79c48 0 0 0 0000204 [SLPQ cooling 0xc4d828d4][SLP] acpi_cooling0 142 c4d7a000 0 0 0 0000204 [SLPQ tzpoll 0xc07b14d4][SLP] acpi_thermal 141 c4d7a20c 0 0 0 0000204 [SLPQ idle 0xc4d37c74][SLP] ciss_notify0 9 c4c9aa3c 0 0 0 0000204 [SLPQ - 0xc4cba500][SLP] thread taskq 140 c4c9ac48 0 0 0 0000204 [IWAIT] swi6:+ 8 c4cad000 0 0 0 0000204 [SLPQ - 0xc4cba680][SLP] acpi_task2 7 c4cad20c 0 0 0 0000204 [SLPQ - 0xc4cba680][SLP] acpi_task1 6 c4cad418 0 0 0 0000204 [SLPQ - 0xc4cba680][SLP] acpi_task0 139 c4cad624 0 0 0 0000204 [IWAIT] swi6: task queue 138 c4cad830 0 0 0 0000204 [IWAIT] swi2: cambio 5 c4cada3c 0 0 0 0000204 [SLPQ - 0xc4cba900][SLP] kqueue taskq 137 c4cadc48 0 0 0 0000204 [IWAIT] swi5:+ 136 c4cb1000 0 0 0 0000204 [SLPQ - 0xc062cee0][SLP] yarrow 4 c4cb120c 0 0 0 0000204 [SLPQ - 0xc062d848][SLP] g_down 3 c4cb1418 0 0 0 0000204 [SLPQ - 0xc062d844][SLP] g_up 2 c4cb1624 0 0 0 0000204 [SLPQ - 0xc062d83c][SLP] g_event 135 c4cb1830 0 0 0 0000204 [IWAIT] swi3: vm 134 c4cb1a3c 0 0 0 000020c [IWAIT] swi4: clock sio 133 c4cb1c48 0 0 0 0000204 [IWAIT] swi1: net 132 c4c83624 0 0 0 0000204 [IWAIT] irq119: 131 c4c83830 0 0 0 0000204 [IWAIT] irq118: 130 c4c83a3c 0 0 0 0000204 [IWAIT] irq117: 129 c4c83c48 0 0 0 0000204 [IWAIT] irq116: 128 c4c96000 0 0 0 0000204 [IWAIT] irq115: 127 c4c9620c 0 0 0 0000204 [IWAIT] irq114: 126 c4c96418 0 0 0 0000204 [IWAIT] irq113: 125 c4c96624 0 0 0 0000204 [IWAIT] irq112: 124 c4c96830 0 0 0 0000204 [IWAIT] irq111: 123 c4c96a3c 0 0 0 0000204 [IWAIT] irq110: 122 c4c96c48 0 0 0 0000204 [IWAIT] irq109: 121 c4c9a000 0 0 0 0000204 [IWAIT] irq108: 120 c4c9a20c 0 0 0 0000204 [IWAIT] irq107: 119 c4c9a418 0 0 0 0000204 [IWAIT] irq106: 118 c4c9a624 0 0 0 0000204 [IWAIT] irq105: 117 c4c9a830 0 0 0 0000204 [IWAIT] irq104: 116 c4c6f20c 0 0 0 0000204 [IWAIT] irq103: 115 c4c6f418 0 0 0 0000204 [IWAIT] irq102: 114 c4c6f624 0 0 0 0000204 [IWAIT] irq101: 113 c4c6f830 0 0 0 0000204 [IWAIT] irq100: 112 c4c6fa3c 0 0 0 0000204 [IWAIT] irq99: 111 c4c6fc48 0 0 0 0000204 [IWAIT] irq98: em1 110 c4c82000 0 0 0 0000204 [IWAIT] irq97: em0 109 c4c8220c 0 0 0 0000204 [IWAIT] irq96: 108 c4c82418 0 0 0 0000204 [IWAIT] irq95: 107 c4c82624 0 0 0 0000204 [IWAIT] irq94: 106 c4c82830 0 0 0 0000204 [IWAIT] irq93: 105 c4c82a3c 0 0 0 0000204 [IWAIT] irq92: 104 c4c82c48 0 0 0 0000204 [IWAIT] irq91: 103 c4c83000 0 0 0 0000204 [IWAIT] irq90: 102 c4c8320c 0 0 0 0000204 [IWAIT] irq89: 101 c4c83418 0 0 0 0000204 [IWAIT] irq88: 100 c4c6b000 0 0 0 0000204 [IWAIT] irq87: 99 c4c6b20c 0 0 0 0000204 [IWAIT] irq86: 98 c4c6b418 0 0 0 0000204 [IWAIT] irq85: 97 c4c6b624 0 0 0 0000204 [IWAIT] irq84: 96 c4c6b830 0 0 0 0000204 [IWAIT] irq83: 95 c4c6ba3c 0 0 0 0000204 [IWAIT] irq82: 94 c4c6bc48 0 0 0 0000204 [IWAIT] irq81: 93 c4c6e000 0 0 0 0000204 [IWAIT] irq80: 92 c4c6e20c 0 0 0 0000204 [IWAIT] irq79: 91 c4c6e418 0 0 0 0000204 [IWAIT] irq78: 90 c4c6e624 0 0 0 0000204 [IWAIT] irq77: 89 c4c6e830 0 0 0 0000204 [IWAIT] irq76: 88 c4c6ea3c 0 0 0 0000204 [IWAIT] irq75: 87 c4c6ec48 0 0 0 0000204 [IWAIT] irq74: 86 c4c6f000 0 0 0 0000204 [IWAIT] irq73: 85 c4c57000 0 0 0 0000204 [IWAIT] irq72: 84 c4c5720c 0 0 0 0000204 [IWAIT] irq71: 83 c4c57418 0 0 0 0000204 [IWAIT] irq70: 82 c4c57624 0 0 0 0000204 [IWAIT] irq69: 81 c4c57830 0 0 0 0000204 [IWAIT] irq68: 80 c4c57a3c 0 0 0 0000204 [IWAIT] irq67: 79 c4c57c48 0 0 0 0000204 [IWAIT] irq66: 78 c4c5a000 0 0 0 0000204 [IWAIT] irq65: 77 c4c5a20c 0 0 0 0000204 [IWAIT] irq64: 76 c4c5a418 0 0 0 0000204 [IWAIT] irq63: 75 c4c5a624 0 0 0 0000204 [IWAIT] irq62: 74 c4c5a830 0 0 0 0000204 [IWAIT] irq61: 73 c4c5aa3c 0 0 0 0000204 [IWAIT] irq60: 72 c4c5ac48 0 0 0 0000204 [IWAIT] irq59: 71 c4c2c20c 0 0 0 0000204 [IWAIT] irq58: 70 c4c2c418 0 0 0 0000204 [IWAIT] irq57: 69 c4c2c624 0 0 0 0000204 [IWAIT] irq56: 68 c4c2c830 0 0 0 0000204 [IWAIT] irq55: 67 c4c2ca3c 0 0 0 0000204 [IWAIT] irq54: 66 c4c2cc48 0 0 0 0000204 [IWAIT] irq53: 65 c4c48000 0 0 0 0000204 [IWAIT] irq52: 64 c4c4820c 0 0 0 0000204 [IWAIT] irq51: ciss0 63 c4c48418 0 0 0 0000204 [IWAIT] irq50: 62 c4c48624 0 0 0 0000204 [IWAIT] irq49: 61 c4c48830 0 0 0 0000204 [IWAIT] irq48: 60 c4c48a3c 0 0 0 0000204 [IWAIT] irq47: 59 c4c48c48 0 0 0 0000204 [IWAIT] irq46: 58 c4c1d624 0 0 0 0000204 [IWAIT] irq45: 57 c4c1d830 0 0 0 0000204 [IWAIT] irq44: 56 c4c1da3c 0 0 0 0000204 [IWAIT] irq43: 55 c4c1dc48 0 0 0 0000204 [IWAIT] irq42: 54 c4c2b000 0 0 0 0000204 [IWAIT] irq41: 53 c4c2b20c 0 0 0 0000204 [IWAIT] irq40: 52 c4c2b418 0 0 0 0000204 [IWAIT] irq39: 51 c4c2b624 0 0 0 0000204 [IWAIT] irq38: 50 c4c2b830 0 0 0 0000204 [IWAIT] irq37: 49 c4c2ba3c 0 0 0 0000204 [IWAIT] irq36: 48 c4c2bc48 0 0 0 0000204 [IWAIT] irq35: 47 c4c2c000 0 0 0 0000204 [IWAIT] irq34: 46 c4c10c48 0 0 0 0000204 [IWAIT] irq33: 45 c4c1c000 0 0 0 0000204 [IWAIT] irq32: 44 c4c1c20c 0 0 0 0000204 [IWAIT] irq31: 43 c4c1c418 0 0 0 0000204 [IWAIT] irq30: 42 c4c1c624 0 0 0 0000204 [IWAIT] irq29: 41 c4c1c830 0 0 0 0000204 [IWAIT] irq28: 40 c4c1ca3c 0 0 0 0000204 [IWAIT] irq27: 39 c4c1cc48 0 0 0 0000204 [IWAIT] irq26: bge1 38 c4c1d000 0 0 0 0000204 [IWAIT] irq25: bge0 37 c4c1d20c 0 0 0 0000204 [IWAIT] irq24: 36 c4c1d418 0 0 0 0000204 [IWAIT] irq23: 35 c4c02624 0 0 0 0000204 [IWAIT] irq22: 34 c4c02830 0 0 0 0000204 [IWAIT] irq21: 33 c4c02a3c 0 0 0 0000204 [IWAIT] irq20: 32 c4c02c48 0 0 0 0000204 [IWAIT] irq19: 31 c4c10000 0 0 0 0000204 [IWAIT] irq18: 30 c4c1020c 0 0 0 0000204 [IWAIT] irq17: 29 c4c10418 0 0 0 0000204 [IWAIT] irq16: 28 c4c10624 0 0 0 0000204 [IWAIT] irq15: ata1 27 c4c10830 0 0 0 0000204 [IWAIT] irq14: ata0 26 c4c10a3c 0 0 0 0000204 [IWAIT] irq13: 25 c4bb020c 0 0 0 0000204 [IWAIT] irq12: 24 c4bb0418 0 0 0 0000204 [IWAIT] irq11: 23 c4bb0624 0 0 0 0000204 [IWAIT] irq10: 22 c4bb0830 0 0 0 0000204 [IWAIT] irq9: acpi0 21 c4bb0a3c 0 0 0 0000204 [IWAIT] irq8: 20 c4bb0c48 0 0 0 0000204 [IWAIT] irq7: 19 c4c02000 0 0 0 0000204 [IWAIT] irq6: 18 c4c0220c 0 0 0 0000204 [IWAIT] irq5: 17 c4c02418 0 0 0 0000204 [IWAIT] irq4: sio0 16 c4bab000 0 0 0 0000204 [IWAIT] irq3: sio1 15 c4bab20c 0 0 0 0000204 [IWAIT] irq0: 14 c4bab418 0 0 0 0000204 [IWAIT] irq1: atkbd0 13 c4bab624 0 0 0 000020c [CPU 0] idle: cpu0 12 c4bab830 0 0 0 000020c [CPU 1] idle: cpu1 11 c4baba3c 0 0 0 000020c [CPU 2] idle: cpu2 10 c4babc48 0 0 0 000020c [CPU 3] idle: cpu3 1 c4bb0000 0 0 1 0004200 [SLPQ wait 0xc4bb0000][SLP] init 0 c062d940 0 0 0 0000200 [IWAIT] swapper db> trace 11707 Tracing pid 11707 tid 100191 td 0xc4fe9900 sched_switch(c4fe9900,0,1,c067d8a0,e3d78fa4) at sched_switch+0x158 mi_switch(1,0,c05ea3ba,1a7) at mi_switch+0x1d1 sleepq_switch(c502ce28,c05e85df,da,2,eb6f8548) at sleepq_switch+0xe4 sleepq_wait(c502ce28,0,c05e85df,da,0) at sleepq_wait+0x11 msleep(c502ce28,c062f98c,50,c05ed658,0) at msleep+0x32e acquire(eb6f85a4,40,60000,b6,c4fe9900) at acquire+0x88 lockmgr(c502ce28,2002,c502ce4c,c4fe9900,eb6f85cc) at lockmgr+0x426 vop_stdlock(eb6f861c,c05eaa4f,c061dda0,eb6f861c,eb6f85dc) at vop_stdlock+0x2f VOP_LOCK_APV(c061e300,eb6f861c,eb6f85f4,c05cc673,eb6f861c) at VOP_LOCK_APV+0x44 ffs_lock(eb6f861c,246,2002,c502cdd0,eb6f8638) at ffs_lock+0x19 VOP_LOCK_APV(c061dda0,eb6f861c,c060f004,c4d69444,1b2) at VOP_LOCK_APV+0x44 vn_lock(c502cdd0,2002,c4fe9900,1b2,c4ba9300) at vn_lock+0xd0 ffs_snapshot(c4d69400,c4f36900,eb6f89b0,6c,0) at ffs_snapshot+0x14a9 ffs_mount(c4d69400,c4fe9900,c05ee869,29b,eb6f8ab0) at ffs_mount+0x9a9 vfs_domount(c4fe9900,c4d9e4d0,c4d30b40,11211000,c575c7c0) at vfs_domount+0x63e vfs_donmount(c4fe9900,11211000,eb6f8bf4,c5481080,e) at vfs_donmount+0x108 kernel_mount(c59ba6a0,11211000,eb6f8c38,6c,bfbfe8c8) at kernel_mount+0x46 ffs_cmount(c59ba6a0,bfbfe110,11211000,c4fe9900,c061dac0) at ffs_cmount+0x85 mount(c4fe9900,eb6f8d04,c05fc65c,3c6,4) at mount+0x1d7 syscall(3b,3b,3b,8814819c,bfbfe0b0) at syscall+0x2a6 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (21, FreeBSD ELF32, mount), eip = 0x880b8b03, esp = 0xbfbfe07c, ebp = 0xbfbfe778 --- db> trace 13267 Tracing pid 13267 tid 100735 td 0xc92fc900 sched_switch(c92fc900,0,1,c067d600,c5bb72dc) at sched_switch+0x158 mi_switch(1,0,c05ea3ba,1a7) at mi_switch+0x1d1 sleepq_switch(c50088d8,c05e85df,da,1,ebfcc6b4) at sleepq_switch+0xe4 sleepq_wait(c50088d8,0,c05e85df,da,0) at sleepq_wait+0x11 msleep(c50088d8,c062e7d4,50,c05ed658,0) at msleep+0x32e acquire(ebfcc710,40,60000,b6,c92fc900) at acquire+0x88 lockmgr(c50088d8,3002,c50088fc,c92fc900,ebfcc738) at lockmgr+0x426 vop_stdlock(ebfcc788,c50088fc,c061dda0,ebfcc788,ebfcc748) at vop_stdlock+0x2f VOP_LOCK_APV(c061e300,ebfcc788,ebfcc760,c05cc673,ebfcc788) at VOP_LOCK_APV+0x44 ffs_lock(ebfcc788,1002,1002,c5008880,ebfcc7a4) at ffs_lock+0x19 VOP_LOCK_APV(c061dda0,ebfcc788,c05ef151,31f,c92fc900) at VOP_LOCK_APV+0x44 vn_lock(c5008880,1002,c92fc900,c92fc900,0) at vn_lock+0xd0 lookup(ebfcc898,0,c05ee77c,b6,38) at lookup+0xc3 namei(ebfcc898,0,c05ef041,f9,8140af0) at namei+0x37a kern_statfs(c92fc900,8140af0,0,ebfccaec,c92fc900) at kern_statfs+0x6d statfs(c92fc900,ebfccd04,8,c05eb70b,2) at statfs+0x35 syscall(3b,3b,bfbf003b,0,80ef6a0) at syscall+0x2a6 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (396, FreeBSD ELF32, statfs), eip = 0x882966e3, esp = 0xbfbfae4c, ebp = 0xbfbfb048 --- db> trace 12800 Tracing pid 12800 tid 100352 td 0xc60f7c00 sched_switch(c60f7c00,0,1,c067d328,77790a88) at sched_switch+0x158 mi_switch(1,0,c05ea3ba,1a7) at mi_switch+0x1d1 sleepq_switch(c4d6946c,c05e85df,da,0,eb9c3944) at sleepq_switch+0xe4 sleepq_wait(c4d6946c,0,c05e85df,da,0) at sleepq_wait+0x11 msleep(c4d6946c,c4d69444,29f,c05ef16f,0,c062a820,c53f0880,eb9c3980,c062f8b4) at msleep+0x32e vn_write_suspend_wait(c53f0880,c4d69400,1,887,c05eec46) at vn_write_suspend_wait+0x11b ufs_inactive(eb9c39cc,eb9c39e4,c0503c6f,c061dda0,eb9c39cc) at ufs_inactive+0x1b4 VOP_INACTIVE_APV(c061dda0,eb9c39cc,c05eec46,887,c062a6a0) at VOP_INACTIVE_APV+0x3a vinactive(c53f0880,c60f7c00,c05eec46,819,c4e49000) at vinactive+0x62 vput(c53f0880,ffffffdf,2,c60f7c00,0) at vput+0x159 vn_open_cred(eb9c3bc4,eb9c3cc4,180,c8e22280,5) at vn_open_cred+0xe7 vn_open(eb9c3bc4,eb9c3cc4,180,5,eb9c3b50) at vn_open+0x33 kern_open(c60f7c00,8141480,0,a03,180) at kern_open+0xca open(c60f7c00,eb9c3d04,c,c05eb482,3) at open+0x36 syscall(3b,bfbf003b,bfbf003b,0,a02) at syscall+0x2a6 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0x88305967, esp = 0xbfbfa5ac, ebp = 0xbfbfa638 --- db> continue spin lock sio held by 0xc4bac780 for > 5 seconds kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 07 fault virtual address = 0x144 fault code = supervisor read, page not present instruction pointer = 0x20:0xc04ce557 stack pointer = 0x28:0xe7134bc4 frame pointer = 0x28:0xe7134bcc code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 10 (idle: cpu3) trap number = 12 panic: page fault cpuid = 3 KDB: stack backtrace: kdb_backtrace(c05fb775,3,c05defdf,e7134b18,c4bac600) at kdb_backtrace+0x2f panic(c05defdf,c05fc499,c4babdd0,1,1) at panic+0x129 trap_fatal(e7134b84,144,c4bac600,c4c2da80,144) at trap_fatal+0x31e trap(e7130008,c0490028,c05f0028,c4bac600,c06836c0) at trap+0x119 calltrap() at calltrap+0x5 --- trap 0xc, eip = 0xc04ce557, esp = 0xe7134bc4, ebp = 0xe7134bcc --- witness_display_spinlock(c06836c0,0,c4bac780,c4bac600,c06836c0) at witness_display_spinlock+0x9 _mtx_lock_spin(c06836c0,c4bac600,0,c05f8952,56e) at _mtx_lock_spin+0xb2 _mtx_lock_spin_flags(c06836c0,0,c05f8952,56e,e7134c54) at _mtx_lock_spin_flags+0x82 siointr(c4da1c00,c062f9c0,2,4,c4bac600) at siointr+0x2a intr_execute_handlers(c4ba4490,e7134c6c,e7134cb0,c05af203,34) at intr_execute_handlers+0x8c lapic_handle_intr(34) at lapic_handle_intr+0x3b Xapic_isr1() at Xapic_isr1+0x33 --- interrupt, eip = 0xc07a3272, esp = 0xe7134cb0, ebp = 0xe7134cb0 --- acpi_cpu_c1(c4bac600,e7134cc8,246,c062f9c0,e7134ce4) at acpi_cpu_c1+0x5 acpi_cpu_idle(e7134d04,c0493346,c062f9c0,0,c05e5dac) at acpi_cpu_idle+0x18a cpu_idle(c062f9c0,0,c05e5dac,7d,0) at cpu_idle+0x29 idle_proc(0,e7134d38,c05e5cb0,30d,0) at idle_proc+0x6e fork_exit(c04932d8,0,e7134d38) at fork_exit+0x78 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe7134d6c, ebp = 0 --- Uptime: 26m8s