Date: Sat, 6 Nov 2004 16:41:50 +1100 From: "Steven Adams" <steve@drifthost.com> To: <questions@freebsd.org> Subject: Big Problem, Load Avg Very High Message-ID: <20041106054206.82A8143D86@mx1.FreeBSD.org>
next in thread | raw e-mail | index | archive | help
Hi,
I run FreeBSD 5.2.1 with the following hardware
Acer Server: Altos
LSI Megaraid with 5x 36 gig scsi on raid 5.
Dual xeon cpu's
1GIG ECC Ram
2x 250GIG IDE Drives
We host a couple or sites on this server (not very big sites). My server
load is always around 0.90 - 3.40. Sometimes it will jump up to 10-15.
At random it will jump up to 30-40 load and I wont even be able to get to
the server, typing commands on the remote ip based kvm is VERY slow
sometimes missing letters. As soon as im able to get top running it shows
====================================
last pid: 52614; load averages: 6.82, 15.75, 15.18
up 1+03:07:12 16:32:22
462 processes: 1 running, 460 sleeping, 1 zombie
CPU states: 0.0% user, 0.0% nice, 0.6% system, 0.6% interrupt, 98.7%
idle
Mem: 615M Active, 68M Inact, 288M Wired, 29M Cache, 112M Buf, 1844K Free
Swap: 1536M Total, 555M Used, 981M Free, 36% Inuse, 12K In
===================================
I am trying to figure out why the load is so high for a server that should
be able to handle it easily..
I did notice once when running systat -vmstat the amr0(scsi raid) jumps up
to 99% busy copying 2-3MB/s for a few moments then goes back down..
After one of the times it went to 50load it got this on the console screen.
FYI: amrd0s1h is /home partition
==================================================
Swap_pager: indefinite wait buffer: device: amrd0s1h, blkno: 103776, size:
32768
Swap_pager: indefinite wait buffer: device: amrd0s1h, blkno: 130801, size:
4096
==================================================
I wanted to check the logs but for some weird reason syslogd is not logging
to /var/log/security and /var/log/blah
I tried to do a killall -HUP syslog and killall -9 syslogd ; syslogd
It logs to the other log files but wont log to the /var/log/blah and
/var/log/security so I cant see if there Is a problem with the server.
My config is below.
================================================================
*.* /var/log/blah
*.err;kern.debug;auth.notice;mail.crit /dev/console
*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err
/var/log/messages
security.* /var/log/security
auth.info;authpriv.info /var/log/auth.log
local0.notice;local0.debug;mail.*;mail.none;mail.info;local0.info
/var/log/maillog
lpr.info /var/log/lpd-errs
ftp.info /var/log/xferlog
cron.* /var/log/cron
*.=debug /var/log/debug.log
*.emerg *
# uncomment this to log all writes to /dev/console to /var/log/console.log
#console.info /var/log/console.log
# uncomment this to enable logging of all log messages to /var/log/all.log
# touch /var/log/all.log and chmod it to mode 600 before it will work
#*.* /var/log/all.log
# uncomment this to enable logging to a remote loghost named loghost
#*.* @loghost
# uncomment these if you're running inn
# news.crit /var/log/news/news.crit
# news.err /var/log/news/news.err
# news.notice /var/log/news/news.notice
!startslip
*.* /var/log/slip.log
!ppp
*.* /var/log/ppp.log
================================================================
I am really confussed what else I can do to find out what is causing this
problem as the server completely locks up when it goes to 50+ load.
It seems that its copying a lot of information to the swap drive and is
running out of ram, which I don't know why it seems apache is taking up all
of the ram for some weird reason?
Here is my demsg, I hope you guys can help!
==============================================
Copyright (c) 1992-2004 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 5.2.1-RELEASE #3: Wed Sep 29 20:50:16 EST 2004
root@inertia.drifthost.com:/usr/src/sys/i386/compile/DRIFTHOST
Preloaded elf kernel "/boot/kernel/kernel" at 0xc070a000.
Preloaded elf module "/boot/kernel/acpi.ko" at 0xc070a2bc.
ACPI APIC Table: <RCC GCSL >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2392.05-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf29 Stepping = 9
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA
,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,>
Hyperthreading: 2 logical CPUs
real memory = 1073676288 (1023 MB)
avail memory = 1037717504 (989 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
ioapic0 <Version 1.1> irqs 0-15 on motherboard
ioapic1 <Version 1.1> irqs 16-31 on motherboard
ioapic2 <Version 1.1> irqs 32-47 on motherboard
Pentium Pro MTRR support enabled
ACPI-0660: *** Warning: Type override - [DEB_] had invalid type
(Integer) for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [MLIB] had invalid type
(Integer) for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [IO__] had invalid type
(Integer) for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [DATA] had invalid type (String)
for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [SIO_] had invalid type (String)
for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [SB__] had invalid type (String)
for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [PM__] had invalid type (String)
for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [ICNT] had invalid type (String)
for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [ACPI] had invalid type (String)
for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [IORG] had invalid type (String)
for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [SB__] had invalid type (String)
for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [PM__] had invalid type (String)
for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [SIO_] had invalid type (String)
for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [PM__] had invalid type (String)
for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [BIOS] had invalid type
(Integer) for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [CMOS] had invalid type
(Integer) for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [KBC_] had invalid type
(Integer) for Scope operator, changed to (Scope)
ACPI-0660: *** Warning: Type override - [OEM_] had invalid type
(Integer) for Scope operator, changed to (Scope)
npx0: [FAST]
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <RCC GCSL > on motherboard
Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000
pcibios: BIOS version 2.10
Using $PIR table, 10 entries at 0xc00f4b00
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x508-0x50b on acpi0
acpi_cpu0: <CPU> on acpi0
acpi_cpu1: <CPU> on acpi0
acpi_cpu2: <CPU> on acpi0
acpi_cpu3: <CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
bge0: <Broadcom BCM5702 Gigabit Ethernet, ASIC rev. 0x1002> mem
0xfe9e0000-0xfe9effff irq 16 at device 2.0 on pci0
bge0: Ethernet address: 00:c0:9f:25:6a:ad
miibus0: <MII bus> on bge0
brgphy0: <BCM5703 10/100/1000baseTX PHY> on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX,
1000baseTX-FDX, auto
pci0: <display, VGA> at device 14.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 15.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <ServerWorks CSB5 UDMA100 controller> port
0xffa0-0xffaf,0x374-0x377,0x170-0x177,0x3f4-0x3f7,0x1f0-0x1f7 at device 0
ata0: at 0x1f0 irq 14 on atapci0
ata0: [MPSAFE]
ata1: at 0x170 irq 15 on atapci0
ata1: [MPSAFE]
pcib1: <ACPI Host-PCI bridge> on acpi0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI Host-PCI bridge> on acpi0
pci2: <ACPI PCI bus> on pcib2
amr0: <LSILogic MegaRAID> mem 0xfc8f0000-0xfc8fffff irq 24 at device 4.0 on
pci2
amr0: <LSILogic MegaRAID SCSI 320-1> Firmware 1L26, BIOS G112, 64MB RAM
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: model Generic PS/2 mouse, device ID 0
fdc0: ready for input in output
fdc0: cmd 3 failed at out byte 1 of 3
fdc0: ready for input in output
fdc0: cmd 3 failed at out byte 1 of 3
orm0: <Option ROMs> at iomem 0xc9800-0xcafff,0xc0000-0xc7fff on isa0
pmtimer0 on isa0
fdc0: <Enhanced floppy controller (i82077, NE72065 or clone)> at port
0x3f7,0x3f0-0x3f5 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 10.000 msec
ipfw2 initialized, divert disabled, rule-based forwarding enabled, default
to deny, logging limited to 10 packets/entry by dt
GEOM: create disk ad2 dp=0xc6549c60
ad2: 190782MB <WDC WD2000JB-00EVA0> [387621/16/63] at ata1-master UDMA100
amrd0: <LSILogic MegaRAID logical drive> on amr0
amrd0: 140008MB (286736384 sectors) RAID 5 (optimal)
GEOM: create disk amrd0 dp=0xc66e030c
ses0 at amr0 bus 0 target 6 lun 0
ses0: <SDR GEM318P 1> Fixed Processor SCSI-2 device
ses0: SAF-TE Compliant Device
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
===========================================================
Steven Adams steve@drifthost.com
DriftNet Web Services http://www.drifthost.com
Home: +61 2 94274857
Fax: +61 2 94274857
Mobile +61 (0) 404 085644
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20041106054206.82A8143D86>