From owner-freebsd-stable Thu Aug 9 20: 7:53 2001 Delivered-To: freebsd-stable@freebsd.org Received: from ns.yogotech.com (ns.yogotech.com [206.127.123.66]) by hub.freebsd.org (Postfix) with ESMTP id ECFE237B401 for ; Thu, 9 Aug 2001 20:07:45 -0700 (PDT) (envelope-from nate@yogotech.com) Received: from nomad.yogotech.com (nomad.yogotech.com [206.127.123.131]) by ns.yogotech.com (8.9.3/8.9.3) with ESMTP id PAA08294 for ; Thu, 9 Aug 2001 15:19:43 -0600 (MDT) (envelope-from nate@nomad.yogotech.com) Received: (from nate@localhost) by nomad.yogotech.com (8.8.8/8.8.8) id PAA03049; Thu, 9 Aug 2001 15:19:43 -0600 (MDT) (envelope-from nate) From: Nate Williams MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15218.65134.426946.295368@nomad.yogotech.com> Date: Thu, 9 Aug 2001 15:19:42 -0600 To: stable@FreeBSD.org Subject: Strange I/O behavior X-Mailer: VM 6.95 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Reply-To: nate@yogotech.com (Nate Williams) Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I've got a dual-CPU box that's acting very strangely. At various times throughout the day, the box starts to crawl. In analyzing it, the only thing that jumps out at me is that the first fixed disk is running a huge number of I/O transactions, to the point that it's almost saturating the disk. However, the amount transferred is almost nil, but since this disk contains / and /usr (which means all the files are on it), interactive performance goes to the dump during these sessions. The box is an all SCSI system, with the first disk being a 17GB Seagate. The dmesg is below. Here's the output of iostat 2, which shows alot of transactions going on. However, there are no active users doing anything as far as I can see, and top shows the system as being mostly idle. Any ideas how I can determine which application(s) are causing the huge I/O loads? Nate turret:~ % iostat 2 tty da0 da1 cd0 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 14 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0 0 1 0 99 0 19 7.82 111 0.85 0.00 0 0.00 0.00 0 0.00 0 0 4 1 95 0 19 7.92 58 0.45 0.00 0 0.00 0.00 0 0.00 0 0 0 1 99 0 19 7.83 103 0.79 0.00 0 0.00 0.00 0 0.00 0 0 1 0 98 0 19 8.20 108 0.86 0.00 0 0.00 0.00 0 0.00 0 0 2 2 96 0 19 7.63 165 1.23 0.00 0 0.00 0.00 0 0.00 0 0 2 2 97 0 19 6.85 90 0.60 0.00 0 0.00 0.00 0 0.00 0 0 3 1 96 0 19 12.06 75 0.88 0.00 0 0.00 0.00 0 0.00 0 0 4 2 93 0 19 5.85 23 0.13 0.00 0 0.00 0.00 0 0.00 0 0 1 1 99 0 19 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0 0 0 0100 0 19 8.12 129 1.02 0.00 0 0.00 0.00 0 0.00 0 0 1 0 98 0 19 10.15 33 0.33 0.00 0 0.00 0.00 0 0.00 0 0 1 1 98 0 19 11.28 40 0.44 0.00 0 0.00 0.00 0 0.00 0 0 2 1 97 0 19 7.11 69 0.48 0.00 0 0.00 0.00 0 0.00 0 0 3 1 97 0 19 10.89 116 1.24 0.00 0 0.00 0.00 0 0.00 0 0 1 0 98 0 19 8.84 22 0.19 0.00 0 0.00 0.00 0 0.00 0 0 2 1 97 0 19 8.18 83 0.66 0.00 0 0.00 0.00 0 0.00 0 0 1 1 98 0 19 7.76 14 0.11 0.00 0 0.00 0.00 0 0.00 0 0 2 0 98 0 19 7.95 37 0.29 0.00 0 0.00 0.00 0 0.00 0 0 1 0 99 0 19 8.24 20 0.16 0.00 0 0.00 0.00 0 0.00 0 0 1 1 98 Copyright (c) 1992-2001 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.3-RELEASE #1: Mon Jul 23 14:52:32 PDT 2001 nate@turret.americas.nokia.com:/usr/src/sys/compile/TURRET Timecounter "i8254" frequency 1193182 Hz CPU: Pentium III/Pentium III Xeon/Celeron (601.37-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x673 Stepping = 3 Features=0x383fbff real memory = 268423168 (262132K bytes) avail memory = 258195456 (252144K bytes) Programming 24 pins in IOAPIC #0 IOAPIC #0 intpin 2 -> irq 0 IOAPIC #0 intpin 16 -> irq 11 IOAPIC #0 intpin 17 -> irq 10 IOAPIC #0 intpin 18 -> irq 12 IOAPIC #0 intpin 19 -> irq 5 FreeBSD/SMP: Multiprocessor motherboard cpu0 (BSP): apic id: 1, version: 0x00040011, at 0xfee00000 cpu1 (AP): apic id: 0, version: 0x00040011, at 0xfee00000 io0 (APIC): apic id: 2, version: 0x00170011, at 0xfec00000 Preloaded elf kernel "kernel" at 0xc0321000. Pentium Pro MTRR support enabled md0: Malloc disk npx0: on motherboard npx0: INT 16 interface pcib0: on motherboard pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 pci1: at 0.0 irq 11 isab0: at device 4.0 on pci0 isa0: on isab0 pci0: at 4.1 pci0: at 4.2 irq 5 Timecounter "PIIX" frequency 3579545 Hz chip1: port 0xe800-0xe80f at device 4.3 on pci0 fxp0: port 0xd000-0xd03f mem 0xe0800000-0xe081ffff,0xe1000000-0xe1000fff irq 5 at device 9.0 on pci0 fxp0: Ethernet address 00:02:b3:36:b1:93 fxp1: port 0xb800-0xb83f mem 0xdf800000-0xdf8fffff,0xe0000000-0xe0000fff irq 12 at device 10.0 on pci0 fxp1: Ethernet address 00:90:27:a5:68:11 ahc0: port 0xb400-0xb4ff mem 0xdf000000-0xdf0 00fff irq 10 at device 11.0 on pci0 aic7890/91: Wide Channel A, SCSI Id=7, 32/255 SCBs fdc0: at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0 fdc0: FIFO enabled, 8 bytes threshold fd0: <1440-KB 3.5" drive> on fdc0 drive 0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: flags 0x1 irq 1 on atkbdc0 kbd0 at atkbd0 vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 sio0: type 16550A sio1 at port 0x2f8-0x2ff irq 3 on isa0 sio1: type 16550A ppc0: at port 0x378-0x37f irq 7 on isa0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/9 bytes threshold plip0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 APIC_IO: Testing 8254 interrupt delivery APIC_IO: routing 8254 via IOAPIC #0 intpin 2 Waiting 2 seconds for SCSI devices to settle SMP: AP CPU #1 Launched! sa0 at ahc0 bus 0 target 4 lun 0 sa0: Removable Sequential Access SCSI-2 device sa0: 10.000MB/s transfers (10.000MHz, offset 15) da0 at ahc0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device da0: 40.000MB/s transfers (20.000MHz, offset 15, 16bit), Tagged Queueing Enabled da0: 17366MB (35566480 512 byte sectors: 255H 63S/T 2213C) Mounting root from ufs:/dev/da0s1a ch0 at ahc0 bus 0 target 6 lun 0 ch0: Removable Changer SCSI-2 device ch0: 10.000MB/s transfers (10.000MHz, offset 15) ch0: 10 slots, 1 drive, 1 picker, 0 portals cd0 at ahc0 bus 0 target 1 lun 0 cd0: Removable CD-ROM SCSI-2 device cd0: 20.000MB/s transfers (20.000MHz, offset 16) cd0: Attempt to query device size failed: NOT READY, Medium not present da1 at ahc0 bus 0 target 3 lun 0 da1: Fixed Direct Access SCSI-2 device da1: 40.000MB/s transfers (20.000MHz, offset 31, 16bit) da1: 17461MB (35761710 512 byte sectors: 255H 63S/T 2226C) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message