Date: Wed, 21 Jan 2004 11:39:58 -0200 From: Patrick Tracanelli <eksffa@freebsdbrasil.com.br> To: current@freebsd.org Cc: Jean Milanez Melo <jmelo@FreeBSDBrasil.COM.BR> Subject: Strange Behaviour with Jail Message-ID: <400E812E.5000006@freebsdbrasil.com.br>
next in thread | raw e-mail | index | archive | help
Hello people, good morning.
In around two weeks i have noticed some strange behaviour with freebsd
releng_5 and Jails. I know we may be wrong at first, 'cos our enviroment
is not the most recent as it could. Actually our jail systems is build
on 5.1-RELEASE-p10. Sure we plan rebuild it on top of RELENG_5_2, but
now please, check the enviroment.
Both, jails and the main system were built on 5.1-RELEASE-p10. The
actual testing enviroment is made up of 4 jails. My choice to limit disk
space usage was to set it based on vnodes, currently it is up as
/dev/md38 on /usr/jail/69.59.167.38 (ufs, local)
devfs on /usr/jail/69.59.167.38/dev (devfs, local)
procfs on /usr/jail/69.59.167.38/proc (procfs, local)
/dev/md34 on /usr/jail/69.59.167.34 (ufs, local)
devfs on /usr/jail/69.59.167.34/dev (devfs, local)
procfs on /usr/jail/69.59.167.34/proc (procfs, local)
/dev/md41 on /usr/jail/69.59.167.41 (ufs, local)
devfs on /usr/jail/69.59.167.41/dev (devfs, local)
procfs on /usr/jail/69.59.167.41/proc (procfs, local)
/dev/md33 on /usr/jail/69.59.167.33 (ufs, local)
devfs on /usr/jail/69.59.167.33/dev (devfs, local)
procfs on /usr/jail/69.59.167.33/proc (procfs, local)
The mounting point as well as the vnode image files are under
/dev/ar0s1h on /usr/jail (ufs, local, soft-updates)
which is UFS2 FS, built with default newfs args (16K block, 2K fragment
size), and the vnode images are 1024k made built from dd(1) -- dd
if=/dev/zero of=default.vnode bs=1024k...).
The enviroment used to run better on 4.8-STABLE, at least with the same
configuration as above, and about 4 to 12 jails. The annoying behaviour
is that, suddenly (ie, when a jail may have a few mail load, as about 2k
simultaneous messages) the whole system gets unavailable. It does not
crash, but get really slow; the load avg, in a known slow circunstance,
that ill put the data now, is almost about 0.09, 0.03, 0.01
vmstat -w1 gives me:
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us
sy id
1 95 0 586800 41540 2258 0 0 0 218 16 0 0 361 0 433
1 4 95
0 95 0 586800 41540 2 0 0 0 2 0 0 0 340 0 256
0 3 97
0 95 0 586800 41540 0 0 0 0 0 0 0 0 337 0 254
0 2 98
0 95 0 587916 41396 29 0 0 0 13 0 0 0 340 0 261
0 3 97
0 95 0 587916 41396 0 0 0 0 0 0 0 0 340 0 258
0 3 97
It is the most significant behaviour i have fount. Under low load
circunstances, the "blocked for resources" column is at least about 20.
Under a little bit more load, it gets easily up to 40, 50, and when it
gets ~ 90, system almost stops responding to user/application commands
and requests.
systat -vm gives me:
2 users Load 0.08 0.03 0.00 Jan 21 11:06
Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 97212 15228 454132 39168 54964 count
All 1021520 29528 776344 64828 pages
23 zfod Interrupts
Proc:r p d s w Csw Trp Sys Int Sof Flt 12 cow 244 total
25153 352 51 286 363 5 46 128016 wire
stray 0
120712 act
stray 6
0.3%Sys 0.2%Intr 0.2%User 0.0%Nice 99.4%Idl 720028 inact
stray 7
| | | | | | | | | | 53156 cache
npx0 13
1808 free 16
fxp0 9
daefr
ata0 14
Namei Name-cache Dir-cache 21 prcfr
ata1 15
Calls hits % hits % react
atapci1 1
1157 1148 99 pdwak
atkbd0 1
pdpgs
ppc0 7
Disks ad0 ad1 ad4 ad6 ar0 fd0 intrn 100 clk 0
KB/t 0.00 0.00 0.00 0.00 24.00 0.00 114160 buf 128 rtc 8
tps 0 0 0 0 0 0 87 dirtybuf
MB/s 0.00 0.00 0.00 0.00 0.01 0.00 70150 desiredvnodes
% busy 0 0 0 0 0 0 15102 numvnodes
where i dont find anything suspect to be wrong (even interrupts), and
top(1) shows CPU states: 0.0% user, 0.0% nice, 0.8% system, 5.6%
interrupt, 93.7% idle
There are about 277 jailed processes. Under, what i would call, denial
of service circunstances (where everything gets as slow as possible) the
disk access is still ridiculously low:
serv1001# iostat -w1
tty ad0 ad1 ad4 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id
0 26 114.02 8 0.92 18.33 1 0.02 0.00 0 0.00 0 0 1
0 98
0 230 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1 0 1 0 98
0 76 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 7 0 3 0 90
0 76 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0 0 0
0 100
About the hardware:
Disks:
ad0: 76319MB <WDC WD800BB-00DKA0> [155061/16/63] at ata0-master UDMA33
ad1: 76319MB <WDC WD800BB-00CAA1> [155061/16/63] at ata0-slave UDMA33
ad4: 76319MB <WDC WD800BB-32CCB0> [155061/16/63] at ata2-master UDMA100
ad6: 76319MB <WDC WD800BB-32CCB0> [155061/16/63] at ata3-master UDMA100
ar0: 152638MB <ATA SPAN array> [19458/255/63] status: READY subdisks:
(ad0 and ad1 are backup only)
NIC is an Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet with about 20
IP addresses (aliases) for jails (currently only 4 is used);
real memory = 1073741824 (1024 MB);
4G swap (2 per disk);
CPU: Intel(R) Pentium(R) III CPU family 1133MHz (1132.79-MHz
686-class CPU)
The custom kernel has device polling, HZ frequency on "1000" (i have
already lowed it at the default 100, but the behaviour did not change),
no TCP/IP tuning, customized at most on hardware support (only what is
needed) and the only performance customization besides device polling is
maxuser set to 1024. No debug.
The jail systems run essencially qmail/vpopmail, Apache13+SSL+PHP, MySQL
and pure-ftpd.
I have read -- and it was not only once -- that setting jail as vnode is
not recomended. I consider having one partition per jail, but i am not
sure at the moment if it will solve the main problem. Sure, still need
to test under 5.2-R, what will soon be done, but are those changes
relevant to an enviroment with only 4 jails?
Any ideas on what i could set better, and what might be wrong? Any other
data I should send about system behaviour?
BTW, the jailstatemod third party kernel module by Bjoern A. Zeeb is
been used (it just does not allow the jail enviroment to "see" the other
disk devices, slices and such -- only its own).
Thanks for yer attention.
--
Patrick Tracanelli
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?400E812E.5000006>