From owner-freebsd-stable@FreeBSD.ORG Tue Nov 18 15:02:58 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 54A9810656DB for ; Tue, 18 Nov 2008 15:02:58 +0000 (UTC) (envelope-from igor.lyapin@gmail.com) Received: from mail-qy0-f18.google.com (mail-qy0-f18.google.com [209.85.221.18]) by mx1.freebsd.org (Postfix) with ESMTP id BB8D78FC1F for ; Tue, 18 Nov 2008 15:02:57 +0000 (UTC) (envelope-from igor.lyapin@gmail.com) Received: by qyk11 with SMTP id 11so929066qyk.19 for ; Tue, 18 Nov 2008 07:02:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type; bh=VnGJ1m/nSdpZVyRY+Jq0fqrodp01K5TCbQditfDTjDw=; b=FyFi+GjM8dzlMkavpcYiTYjICT5oteU0DcdpG3P/JGjh4/bmgFtUw2AqaO4O/qrBcM QD9jxNPrxtaXobzPPEzAoxhyzyNHHWQjwPm8WdOm0aD5OehOxLZurPlnsEC2eI6nEWfV fiBuxhQwU4kz8QlbYOkdVeTDCY9p7dpE4G4Wg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=dFTLgWTFNyapc7jQx6ntOQfWSNQepS03Wk1pPNZR+5jxa1jnY+YuE085pj8jiiXLHU h8rxYQx7TJdyngAb1TDBkzgw2eQEICSQegLNjPhmXQTPnJVfQ9BdyVjezuBYZiCoOmUk iUGXV0bGJtOtphTIbvmCYhfTFpVdIZKlFwPiI= Received: by 10.215.66.8 with SMTP id t8mr3886352qak.160.1227019055140; Tue, 18 Nov 2008 06:37:35 -0800 (PST) Received: by 10.214.43.11 with HTTP; Tue, 18 Nov 2008 06:37:35 -0800 (PST) Message-ID: Date: Tue, 18 Nov 2008 17:37:35 +0300 From: "Igor Lyapin" To: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: High system in %system load . X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Nov 2008 15:02:58 -0000 Hello Got strange problem with high system "%system load" and very slow user level programs (apache+php+mysql) behavior gstat shows 1.5-4% hard disk busy load but system shows about 20-30% load while user load is max 5%. vmstat shows from 2 to 35 process in "b" state. Now use 7.0-RELEASE-p5 , but the same problem was with 7.0-RELEASE. And have no idea what to do with this. ------------------------------------------------------------------- #uname -a FreeBSD i.x00.net 7.0-RELEASE-p5 FreeBSD 7.0-RELEASE-p5 #0: Mon Nov 10 19:49:07 MSK 2008 root@i.x00.net:/usr/obj/usr/src/sys/KERNEL-4BSD amd64 ------------------------------------------------------------------- #top last pid: 47964; load averages: 1.26, 1.62, 1.75 up 0+19:17:13 17:11:06 287 processes: 10 running, 277 sleeping CPU states: 2.2% user, 0.0% nice, 28.3% system, 0.2% interrupt, 69.3% idle Mem: 1286M Active, 1729M Inact, 478M Wired, 131M Cache, 214M Buf, 302M Free Swap: 8192M Total, 8192M Free ------------------------------------------------------------------- #vmstat 5 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us sy id 1 24 0 4357672 419648 10598 1 0 0 10648 34 0 0 558 25212 9072 6 15 80 1 29 0 4365116 414892 3060 0 0 0 2780 0 0 0 975 12270 11501 3 29 68 1 25 0 4392488 408596 14142 0 0 0 13903 0 0 0 526 16696 10389 6 30 64 ------------------------------------------------------------------- #ps ax | grep " [DLW] " 45585 ?? D 0:04.68 /usr/local/sbin/httpd 45975 ?? D 0:04.92 /usr/local/sbin/httpd 45981 ?? D 0:29.91 /usr/local/sbin/httpd 45995 ?? D 0:23.99 /usr/local/sbin/httpd 46019 ?? D 0:26.76 /usr/local/sbin/httpd 46160 ?? D 0:18.89 /usr/local/sbin/httpd 46287 ?? D 0:19.88 /usr/local/sbin/httpd 46303 ?? D 0:05.14 /usr/local/sbin/httpd 46307 ?? D 0:29.15 /usr/local/sbin/httpd 46398 ?? D 0:09.18 /usr/local/sbin/httpd 46617 ?? D 0:02.00 /usr/local/sbin/httpd 46767 ?? D 0:05.26 /usr/local/sbin/httpd 46771 ?? D 0:02.07 /usr/local/sbin/httpd 46803 ?? D 0:30.51 /usr/local/sbin/httpd 46807 ?? D 0:01.34 /usr/local/sbin/httpd 46889 ?? D 0:01.35 /usr/local/sbin/httpd 47659 ?? D 0:01.54 /usr/local/sbin/httpd 47711 ?? D 0:00.92 /usr/local/sbin/httpd 47849 ?? D 0:01.00 /usr/local/sbin/httpd 47877 ?? D 0:00.40 /usr/local/sbin/httpd ------------------------------------------------------------------- #dmesg All buffers synced. Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-RELEASE-p5 #0: Mon Nov 10 19:49:07 MSK 2008 root@i.x00.net:/usr/obj/usr/src/sys/KERNEL-4BSD Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (2400.10-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6fb Stepping = 11 Features=0xbfebfbff Features2=0xe3bd AMD Features=0x20100800 AMD Features2=0x1 Cores per package: 4 usable memory = 4283449344 (4085 MB) avail memory = 4134977536 (3943 MB) ACPI APIC Table: <022108 APIC2247> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 hptrr: HPT RocketRAID controller driver v1.1 (Nov 10 2008 19:48:39) acpi0: <022108 RSDT2247> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, eff00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 928092806000928 device_attach: est0 attach returned 6 p4tcc0: on cpu0 cpu1: on acpi0 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 928092806000928 device_attach: est1 attach returned 6 p4tcc1: on cpu1 cpu2: on acpi0 est2: on cpu2 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 928092806000928 device_attach: est2 attach returned 6 p4tcc2: on cpu2 cpu3: on acpi0 est3: on cpu3 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 928092806000928 device_attach: est3 attach returned 6 p4tcc3: on cpu3 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci5: on pcib1 pcib2: irq 16 at device 28.0 on pci0 pci2: on pcib2 pcib3: irq 16 at device 28.4 on pci0 pci3: on pcib3 bge0: mem 0xfeaf0000-0xfeafffff irq 16 at device 0.0 on pci3 miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge0: Ethernet address: 00:1f:c6:48:c8:9b bge0: [ITHREAD] pcib4: irq 17 at device 28.5 on pci0 pci4: on pcib4 bge1: mem 0xfebf0000-0xfebfffff irq 17 at device 0.0 on pci4 miibus1: on bge1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge1: Ethernet address: 00:1f:c6:48:ca:b7 bge1: [ITHREAD] pcib5: at device 30.0 on pci0 pci1: on pcib5 vgapci0: port 0xec00-0xec7f mem 0xf8000000-0xfbffffff,0xfe9c0000-0xfe9fffff at device 4.0 on pci1 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] atapci1: port 0xdc00-0xdc07,0xd880-0xd883,0xd800-0xd807,0xd480-0xd483,0xd400-0xd40f mem 0xfe8ffc00-0xfe8fffff irq 19 at device 31.2 on pci0 atapci1: [ITHREAD] atapci1: AHCI called from vendor specific driver atapci1: AHCI Version 01.10 controller with 4 ports detected ata2: on atapci1 ata2: [ITHREAD] ata3: on atapci1 ata3: [ITHREAD] ata4: on atapci1 ata4: [ITHREAD] ata5: on atapci1 ata5: [ITHREAD] pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 orm0: at iomem 0xc0000-0xc7fff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 sio0: type 8250 or not responding sio0: [FILTER] sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding disabled, default to accept, logging limited to 5 packets/entry by default hptrr: no controller detected. acd0: DVDR at ata0-master UDMA33 ad4: 476940MB at ata2-master SATA300 ad6: 476940MB at ata3-master SATA300 ad10: 476940MB at ata5-master SATA300 ar0: 476937MB status: READY ar0: disk0 READY (master) using ad4 at ata2-master ar0: disk1 READY (mirror) using ad6 at ata3-master SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! Trying to mount root from ufs:/dev/ar0s1a bge0: link state changed to UP ------------------------------------------------------------------- cat ./KERNEL-4BSD | grep -v "^#" cpu HAMMER ident KERNEL-4BSD options SCHED_4BSD # 4BSD scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options INET6 # IPv6 communications protocols options SCTP # Stream Control Transmission Protocol options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling options MD_ROOT # MD is a potential root device options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!] options COMPAT_IA32 # Compatible with i386 binaries options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_FREEBSD5 # Compatible with FreeBSD5 options COMPAT_FREEBSD6 # Compatible with FreeBSD6 options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options KBD_INSTALL_CDEV # install a CDEV entry in /dev options ADAPTIVE_GIANT # Giant mutex is adaptive. options STOP_NMI # Stop CPUS using NMI instead of IPI options AUDIT # Security event auditing options SMP # Symmetric MultiProcessor Kernel options IPFIREWALL options IPFIREWALL_VERBOSE options IPFIREWALL_VERBOSE_LIMIT=5 options IPFIREWALL_DEFAULT_TO_ACCEPT options QUOTA device cpufreq device acpi device pci device fdc device ata device atadisk # ATA disk drives device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives options ATA_STATIC_ID # Static device numbering device ahc # AHA2940 and onboard AIC7xxx devices options AHC_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~128k to driver. device ahd # AHA39320/29320 and onboard AIC79xx devices options AHD_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~215k to driver. device amd # AMD 53C974 (Tekram DC-390(T)) device hptiop # Highpoint RocketRaid 3xxx series device isp # Qlogic family device mpt # LSI-Logic MPT-Fusion device sym # NCR/Symbios Logic (newer chipsets + those of `ncr') device trm # Tekram DC395U/UW/F DC315U adapters device adv # Advansys SCSI adapters device adw # Advansys wide SCSI adapters device aic # Adaptec 15[012]x SCSI adapters, AIC-6[23]60. device bt # Buslogic/Mylex MultiMaster SCSI adapters device scbus # SCSI bus (required for SCSI) device ch # SCSI media changers device da # Direct Access (disks) device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct SCSI access) device ses # SCSI Environmental Services (and SAF-TE) device amr # AMI MegaRAID device arcmsr # Areca SATA II RAID device ciss # Compaq Smart RAID 5* device dpt # DPT Smartcache III, IV - See NOTES for options device hptmv # Highpoint RocketRAID 182x device hptrr # Highpoint RocketRAID 17xx, 22xx, 23xx, 25xx device iir # Intel Integrated RAID device ips # IBM (Adaptec) ServeRAID device mly # Mylex AcceleRAID/eXtremeRAID device twa # 3ware 9000 series PATA/SATA RAID device aac # Adaptec FSA RAID device aacp # SCSI passthrough for aac (requires CAM) device ida # Compaq Smart RAID device mfi # LSI MegaRAID SAS device mlx # Mylex DAC960 family device twe # 3ware ATA RAID device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer device vga # VGA video card driver device splash # Splash screen and screen saver support device sc device agp # support several AGP chipsets device sio # 8250, 16[45]50 based serial ports device uart # Generic UART driver device de # DEC/Intel DC21x4x (``Tulip'') device em # Intel PRO/1000 adapter Gigabit Ethernet Card device ixgb # Intel PRO/10GbE Ethernet Card device le # AMD Am7900 LANCE and Am79C9xx PCnet device txp # 3Com 3cR990 (``Typhoon'') device vx # 3Com 3c590, 3c595 (``Vortex'') device miibus # MII bus support device bce # Broadcom BCM5706/BCM5708 Gigabit Ethernet device bfe # Broadcom BCM440x 10/100 Ethernet device bge # Broadcom BCM570xx Gigabit Ethernet device dc # DEC/Intel 21143 and various workalikes device fxp # Intel EtherExpress PRO/100B (82557, 82558) device lge # Level 1 LXT1001 gigabit Ethernet device msk # Marvell/SysKonnect Yukon II Gigabit Ethernet device nfe # nVidia nForce MCP on-board Ethernet device nge # NatSemi DP83820 gigabit Ethernet device pcn # AMD Am79C97x PCI 10/100 (precedence over 'le') device re # RealTek 8139C+/8169/8169S/8110S device rl # RealTek 8129/8139 device sf # Adaptec AIC-6915 (``Starfire'') device sis # Silicon Integrated Systems SiS 900/SiS 7016 device sk # SysKonnect SK-984x & SK-982x gigabit Ethernet device ste # Sundance ST201 (D-Link DFE-550TX) device ti # Alteon Networks Tigon I/II gigabit Ethernet device tl # Texas Instruments ThunderLAN device tx # SMC EtherPower II (83c170 ``EPIC'') device vge # VIA VT612x gigabit Ethernet device vr # VIA Rhine, Rhine II device wb # Winbond W89C840F device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'') device loop # Network loopback device random # Entropy device device ether # Ethernet support device sl # Kernel SLIP device ppp # Kernel PPP device tun # Packet tunnel. device pty # Pseudo-ttys (telnet etc) device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) device firmware # firmware assist module device bpf # Berkeley packet filter device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device usb # USB Bus (required) device ugen # Generic device uhid # "Human Interface Devices" device ukbd # Keyboard device umass # Disks/Mass storage - Requires scbus and da device ums # Mouse -- The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.