Date: Tue, 18 Nov 2008 17:37:35 +0300 From: "Igor Lyapin" <igor.lyapin@gmail.com> To: freebsd-stable@freebsd.org Subject: High system in %system load . Message-ID: <fd2a4c1f0811180637m6ee44d4fv4cf97d18e322ca5a@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello
Got strange problem with high system "%system load" and very slow user level
programs (apache+php+mysql) behavior
gstat shows 1.5-4% hard disk busy load but system shows about 20-30% load
while user load is max 5%.
vmstat shows from 2 to 35 process in "b" state.
Now use 7.0-RELEASE-p5 , but the same problem was with 7.0-RELEASE.
And have no idea what to do with this.
-------------------------------------------------------------------
#uname -a
FreeBSD i.x00.net 7.0-RELEASE-p5 FreeBSD 7.0-RELEASE-p5 #0: Mon Nov 10
19:49:07 MSK 2008 root@i.x00.net:/usr/obj/usr/src/sys/KERNEL-4BSD amd64
-------------------------------------------------------------------
#top
last pid: 47964; load averages: 1.26, 1.62, 1.75 up
0+19:17:13 17:11:06
287 processes: 10 running, 277 sleeping
CPU states: 2.2% user, 0.0% nice, 28.3% system, 0.2% interrupt, 69.3%
idle
Mem: 1286M Active, 1729M Inact, 478M Wired, 131M Cache, 214M Buf, 302M Free
Swap: 8192M Total, 8192M Free
-------------------------------------------------------------------
#vmstat 5
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us
sy id
1 24 0 4357672 419648 10598 1 0 0 10648 34 0 0 558 25212 9072
6 15 80
1 29 0 4365116 414892 3060 0 0 0 2780 0 0 0 975 12270 11501
3 29 68
1 25 0 4392488 408596 14142 0 0 0 13903 0 0 0 526 16696 10389
6 30 64
-------------------------------------------------------------------
#ps ax | grep " [DLW] "
45585 ?? D 0:04.68 /usr/local/sbin/httpd
45975 ?? D 0:04.92 /usr/local/sbin/httpd
45981 ?? D 0:29.91 /usr/local/sbin/httpd
45995 ?? D 0:23.99 /usr/local/sbin/httpd
46019 ?? D 0:26.76 /usr/local/sbin/httpd
46160 ?? D 0:18.89 /usr/local/sbin/httpd
46287 ?? D 0:19.88 /usr/local/sbin/httpd
46303 ?? D 0:05.14 /usr/local/sbin/httpd
46307 ?? D 0:29.15 /usr/local/sbin/httpd
46398 ?? D 0:09.18 /usr/local/sbin/httpd
46617 ?? D 0:02.00 /usr/local/sbin/httpd
46767 ?? D 0:05.26 /usr/local/sbin/httpd
46771 ?? D 0:02.07 /usr/local/sbin/httpd
46803 ?? D 0:30.51 /usr/local/sbin/httpd
46807 ?? D 0:01.34 /usr/local/sbin/httpd
46889 ?? D 0:01.35 /usr/local/sbin/httpd
47659 ?? D 0:01.54 /usr/local/sbin/httpd
47711 ?? D 0:00.92 /usr/local/sbin/httpd
47849 ?? D 0:01.00 /usr/local/sbin/httpd
47877 ?? D 0:00.40 /usr/local/sbin/httpd
-------------------------------------------------------------------
#dmesg
All buffers synced.
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-RELEASE-p5 #0: Mon Nov 10 19:49:07 MSK 2008
root@i.x00.net:/usr/obj/usr/src/sys/KERNEL-4BSD
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (2400.10-MHz K8-class
CPU)
Origin = "GenuineIntel" Id = 0x6fb Stepping = 11
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xe3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 4
usable memory = 4283449344 (4085 MB)
avail memory = 4134977536 (3943 MB)
ACPI APIC Table: <022108 APIC2247>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
hptrr: HPT RocketRAID controller driver v1.1 (Nov 10 2008 19:48:39)
acpi0: <022108 RSDT2247> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, eff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 928092806000928
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 928092806000928
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 928092806000928
device_attach: est2 attach returned 6
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
est3: <Enhanced SpeedStep Frequency Control> on cpu3
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 928092806000928
device_attach: est3 attach returned 6
p4tcc3: <CPU Frequency Thermal Control> on cpu3
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci5: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci3: <ACPI PCI bus> on pcib3
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x4201> mem
0xfeaf0000-0xfeafffff irq 16 at device 0.0 on pci3
miibus0: <MII bus> on bge0
brgphy0: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bge0: Ethernet address: 00:1f:c6:48:c8:9b
bge0: [ITHREAD]
pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci4: <ACPI PCI bus> on pcib4
bge1: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x4201> mem
0xfebf0000-0xfebfffff irq 17 at device 0.0 on pci4
miibus1: <MII bus> on bge1
brgphy1: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bge1: Ethernet address: 00:1f:c6:48:ca:b7
bge1: [ITHREAD]
pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib5
vgapci0: <VGA-compatible display> port 0xec00-0xec7f mem
0xf8000000-0xfbffffff,0xfe9c0000-0xfe9fffff at device 4.0 on pci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH7 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <Intel ICH7 SATA300 controller> port
0xdc00-0xdc07,0xd880-0xd883,0xd800-0xd807,0xd480-0xd483,0xd400-0xd40f mem
0xfe8ffc00-0xfe8fffff irq 19 at device 31.2 on pci0
atapci1: [ITHREAD]
atapci1: AHCI called from vendor specific driver
atapci1: AHCI Version 01.10 controller with 4 ports detected
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
ata4: <ATA channel 2> on atapci1
ata4: [ITHREAD]
ata5: <ATA channel 3> on atapci1
ata5: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 8250 or not responding
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding disabled,
default to accept, logging limited to 5 packets/entry by default
hptrr: no controller detected.
acd0: DVDR <MATSHITADVD-RAM UJ-860S/1.00> at ata0-master UDMA33
ad4: 476940MB <Seagate ST3500320NS SN05> at ata2-master SATA300
ad6: 476940MB <Seagate ST3500320NS SN05> at ata3-master SATA300
ad10: 476940MB <Seagate ST3500320NS SN05> at ata5-master SATA300
ar0: 476937MB <Intel MatrixRAID RAID1> status: READY
ar0: disk0 READY (master) using ad4 at ata2-master
ar0: disk1 READY (mirror) using ad6 at ata3-master
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Trying to mount root from ufs:/dev/ar0s1a
bge0: link state changed to UP
-------------------------------------------------------------------
cat ./KERNEL-4BSD | grep -v "^#"
cpu HAMMER
ident KERNEL-4BSD
options SCHED_4BSD # 4BSD scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
options INET6 # IPv6 communications protocols
options SCTP # Stream Control Transmission
Protocol
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big
directories
options UFS_GJOURNAL # Enable gjournal-based UFS
journaling
options MD_ROOT # MD is a potential root device
options MSDOSFS # MSDOS Filesystem
options CD9660 # ISO 9660 Filesystem
options PROCFS # Process filesystem (requires
PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options GEOM_PART_GPT # GUID Partition Tables.
options GEOM_LABEL # Provides labelization
options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!]
options COMPAT_IA32 # Compatible with i386 binaries
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options COMPAT_FREEBSD5 # Compatible with FreeBSD5
options COMPAT_FREEBSD6 # Compatible with FreeBSD6
options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
options KTRACE # ktrace(1) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time
extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options ADAPTIVE_GIANT # Giant mutex is adaptive.
options STOP_NMI # Stop CPUS using NMI instead of IPI
options AUDIT # Security event auditing
options SMP # Symmetric MultiProcessor Kernel
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_VERBOSE_LIMIT=5
options IPFIREWALL_DEFAULT_TO_ACCEPT
options QUOTA
device cpufreq
device acpi
device pci
device fdc
device ata
device atadisk # ATA disk drives
device ataraid # ATA RAID drives
device atapicd # ATAPI CDROM drives
options ATA_STATIC_ID # Static device numbering
device ahc # AHA2940 and onboard AIC7xxx devices
options AHC_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~128k to driver.
device ahd # AHA39320/29320 and onboard AIC79xx devices
options AHD_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~215k to driver.
device amd # AMD 53C974 (Tekram DC-390(T))
device hptiop # Highpoint RocketRaid 3xxx series
device isp # Qlogic family
device mpt # LSI-Logic MPT-Fusion
device sym # NCR/Symbios Logic (newer chipsets + those
of `ncr')
device trm # Tekram DC395U/UW/F DC315U adapters
device adv # Advansys SCSI adapters
device adw # Advansys wide SCSI adapters
device aic # Adaptec 15[012]x SCSI adapters,
AIC-6[23]60.
device bt # Buslogic/Mylex MultiMaster SCSI adapters
device scbus # SCSI bus (required for SCSI)
device ch # SCSI media changers
device da # Direct Access (disks)
device sa # Sequential Access (tape etc)
device cd # CD
device pass # Passthrough device (direct SCSI access)
device ses # SCSI Environmental Services (and SAF-TE)
device amr # AMI MegaRAID
device arcmsr # Areca SATA II RAID
device ciss # Compaq Smart RAID 5*
device dpt # DPT Smartcache III, IV - See NOTES for
options
device hptmv # Highpoint RocketRAID 182x
device hptrr # Highpoint RocketRAID 17xx, 22xx, 23xx,
25xx
device iir # Intel Integrated RAID
device ips # IBM (Adaptec) ServeRAID
device mly # Mylex AcceleRAID/eXtremeRAID
device twa # 3ware 9000 series PATA/SATA RAID
device aac # Adaptec FSA RAID
device aacp # SCSI passthrough for aac (requires CAM)
device ida # Compaq Smart RAID
device mfi # LSI MegaRAID SAS
device mlx # Mylex DAC960 family
device twe # 3ware ATA RAID
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device kbdmux # keyboard multiplexer
device vga # VGA video card driver
device splash # Splash screen and screen saver support
device sc
device agp # support several AGP chipsets
device sio # 8250, 16[45]50 based serial ports
device uart # Generic UART driver
device de # DEC/Intel DC21x4x (``Tulip'')
device em # Intel PRO/1000 adapter Gigabit Ethernet
Card
device ixgb # Intel PRO/10GbE Ethernet Card
device le # AMD Am7900 LANCE and Am79C9xx PCnet
device txp # 3Com 3cR990 (``Typhoon'')
device vx # 3Com 3c590, 3c595 (``Vortex'')
device miibus # MII bus support
device bce # Broadcom BCM5706/BCM5708 Gigabit Ethernet
device bfe # Broadcom BCM440x 10/100 Ethernet
device bge # Broadcom BCM570xx Gigabit Ethernet
device dc # DEC/Intel 21143 and various workalikes
device fxp # Intel EtherExpress PRO/100B (82557, 82558)
device lge # Level 1 LXT1001 gigabit Ethernet
device msk # Marvell/SysKonnect Yukon II Gigabit
Ethernet
device nfe # nVidia nForce MCP on-board Ethernet
device nge # NatSemi DP83820 gigabit Ethernet
device pcn # AMD Am79C97x PCI 10/100 (precedence over
'le')
device re # RealTek 8139C+/8169/8169S/8110S
device rl # RealTek 8129/8139
device sf # Adaptec AIC-6915 (``Starfire'')
device sis # Silicon Integrated Systems SiS 900/SiS
7016
device sk # SysKonnect SK-984x & SK-982x gigabit
Ethernet
device ste # Sundance ST201 (D-Link DFE-550TX)
device ti # Alteon Networks Tigon I/II gigabit
Ethernet
device tl # Texas Instruments ThunderLAN
device tx # SMC EtherPower II (83c170 ``EPIC'')
device vge # VIA VT612x gigabit Ethernet
device vr # VIA Rhine, Rhine II
device wb # Winbond W89C840F
device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'')
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device sl # Kernel SLIP
device ppp # Kernel PPP
device tun # Packet tunnel.
device pty # Pseudo-ttys (telnet etc)
device md # Memory "disks"
device gif # IPv6 and IPv4 tunneling
device faith # IPv6-to-IPv4 relaying (translation)
device firmware # firmware assist module
device bpf # Berkeley packet filter
device uhci # UHCI PCI->USB interface
device ohci # OHCI PCI->USB interface
device ehci # EHCI PCI->USB interface (USB 2.0)
device usb # USB Bus (required)
device ugen # Generic
device uhid # "Human Interface Devices"
device ukbd # Keyboard
device umass # Disks/Mass storage - Requires scbus and da
device ums # Mouse
--
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?fd2a4c1f0811180637m6ee44d4fv4cf97d18e322ca5a>