Date: Tue, 16 Oct 2007 11:51:33 +0300 From: Krassimir Slavchev <krassi@bulinfo.net> To: Alexey Popov <lol@chistydom.ru> Cc: freebsd-hackers@freebsd.org, Kris Kennaway <kris@FreeBSD.org>, freebsd-stable@freebsd.org Subject: Re: amrd disk performance drop after running under high load Message-ID: <47147B95.9040400@bulinfo.net> In-Reply-To: <47146FB4.6040306@chistydom.ru> References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> <47146FB4.6040306@chistydom.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Alexey Popov wrote: > Hi. > > Kris Kennaway wrote: > >>> After some time of running under high load disk performance become >>> expremely poor. At that periods 'systat -vm 1' shows something like >>> this: >> What does "high load" mean? You need to explain the system workload >> more. > This web service is similiar to YouTube. This server is video store. I > have around 200G of *.flv (flash video) files on the server. > > I run lighttpd as a web server. Disk load is usually around 50%, network > output 100Mbit/s, 100 simultaneous connections. CPU is mostly idle. > > As you can see it is a trivial service - sending files to network via HTTP. > >> >>> Disks amrd0 >>> KB/t 85.39 >>> tps 5 >>> MB/s 0.38 >>> % busy 99 >> >>> Apart of all, I tried to make mutex profiling and here's the results >>> (sorted by the total number of acquisitions): >>> >>> Bad case: >>> >>> 102 223514 273977 0 14689 1651568 /usr/src/sys/vm/uma_core.c:2349 (512) >>> 950 263099 273968 0 15004 14427 /usr/src/sys/vm/uma_core.c:2450 (512) >>> 108 150422 175840 0 10978 22988519 /usr/src/sys/vm/uma_core.c:1888 >>> (mbuf) >> >> > Here you can see that high UMA activity happens in periods of low disk >> > performance. But I'm not sure whether this is a root of the >> problem, not >> > a consequence. >> >> The extremely high contention there does seem to say you have a mbuf >> starvation problem and not a disk problem. I don't know why this >> would be happening off-hand. > But there's no mbuf shortage in `netstat -m`. > > What else can I try to track down the source of the problem? > >> Can you also provide more details about the system hardware and >> configuration? > This is Dell 2850 2 x Xeon 3.2, 4Gb RAM, 6x300Gb SCSI RAID5. I'll attach > details. > > With best regards, > Alexey Popov > > > ------------------------------------------------------------------------ > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" last pid: 11008; load averages: 0.07, 0.10, 0.08 up 47+08:32:50 11:46:15 38 processes: 1 running, 37 sleeping Mem: 46M Active, 3443M Inact, 246M Wired, 144M Cache, 208M Buf, 5596K Free Swap: 2048M Total, 4K Used, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 56386 root 1 4 0 19856K 10000K kqread 1 115:19 2.88% lighttpd 636 root 1 96 0 18292K 4212K select 0 25:39 0.00% snmpd 784 root 1 96 0 19668K 2072K select 1 2:31 0.00% sshd 680 root 1 96 0 7732K 1384K select 0 1:59 0.00% ntpd 1540 root 1 96 0 35092K 6496K select 0 1:30 0.00% httpd 769 root 4 20 0 14148K 2632K kserel 0 1:04 0.00% bacula-fd 755 root 1 96 0 3852K 1060K select 1 0:22 0.00% master 568 root 1 96 0 3648K 908K select 0 0:18 0.00% syslogd 80663 root 1 8 0 3688K 1016K nanslp 1 0:05 0.00% cron 760 postfix 1 96 0 3944K 1160K select 0 0:04 0.00% qmgr 89776 www 1 20 0 35180K 6684K lockf 0 0:04 0.00% httpd 89763 www 1 20 0 35180K 6684K lockf 0 0:04 0.00% httpd 89774 www 1 20 0 35180K 6684K lockf 0 0:04 0.00% httpd 89775 www 1 96 0 35180K 6684K select 0 0:04 0.00% httpd 699 root 1 20 0 7732K 1388K pause 0 0:03 0.00% ntpd 484 root 1 4 0 652K 220K select 0 0:00 0.00% devd 10904 llp 1 96 0 30616K 3564K select 0 0:00 0.00% sshd 10915 root 1 20 0 3912K 2340K pause 1 0:00 0.00% csh You run apache with mod_perl or php too. How many clients handle this apache server? Also in this light load you have locked files! Check script execution times (/server-status may be useful). When you have hight load check swap usage and haw many processes are in lockf state. Best Regards -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFHFHuVxJBWvpalMpkRAnqTAJ9FgURNk98dtD0HYX6xIz17R6sLpQCgh5nJ XBtfOyzJJbkjzVzSF/WfmHc= =oTHZ -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?47147B95.9040400>