Date: Sat, 28 Mar 2015 23:22:52 +0300 From: Artem Kuchin <artem@artem.ru> To: freebsd-fs@freebsd.org Subject: Little research how rm -rf and tar kill server Message-ID: <55170D9C.1070107@artem.ru>
next in thread | raw e-mail | index | archive | help
I had a freebsd 9 server updated via buildkernel/buildworld last time
in june 2014 and
everything was fine.
The server had 100+ web sites apache only, mysql database (40+
databases), imap server.
2x2TB seagate disks in geom mirror config, SU+J UFS.
Every night incremental or full backup was done into tar.gz (gtar -czf)
It never generated any problems. Sometimes i had to delete full sites
with tone of files
using rm -rf and it never caused any problem too.
In february 2015 i migrated to a new server.
Hardware. CPU is the same (some xeon 4 core + HT), ram is the same 32GB,
disks now 2x3TB TOSHIBA, same geom mirror, same SU+J USF
Freebsd is now 10-STABLE (updated via buildkernel/buildworld)
I changed fronend to NGINX and it now serves all static files, back end
is still Apache,
switch fro mysql to MariaDB (it seems a lot faster)
And in the first week i discovered two things:
1) untaring (tar -xf) a backup from tar.gz overloads the server
2) rm -rf overloads the server
How overload looks like:
Here is typical picture or normal operation:
TOP:
last pid: 60506; load averages: 1.10, 0.93, 0.86 up 30+16:00:32 18:25:22
563 processes: 2 running, 560 sleeping, 1 stopped
CPU: 13.0% user, 0.0% nice, 2.4% system, 0.2% interrupt, 84.3% idle
Mem: 2034M Active, 25G Inact, 3412M Wired, 20M Cache, 1656M Buf, 943M Free
Swap: 4096M Total, 51M Used, 4045M Free, 1% Inuse
systat -io
/0% /10 /20 /30 /40 /50 /60 /70 /80 /90 /100
ada0 MB/s
tps|XXXXXX
ada1 MB/s
tps|XXXXX
mysql:
show processlist
26 connection
all in "sleep" (many request are executed, but hard to catch, too fast)
Now, when i started
tar -xf backup.tgz
after about 5 minutes number of processed rise to 800, many in ufs state,
mysql show processlist show about 200 requests in opening tables state,
sysstat -io show tps over 1000
sites which use mysql stop responding, static sites work but slow,
ssh loging takes very login time.
I used pv to limit tgz reading bandwidth to 5MB/sec - did not help, the
end of
the world just started a little later.
Then i accidentally did sync from shell and situation became better.
So, i run the same untar command and added a script in parallel which
did sync
every 120 seconds - all problems went away.
This is VERY strange, because man 2 sync says sync is done every 30 seconds
anyway. Apparently not.
Now, next i did cp -Rp a huge tree with tons of files - no such problems
like with untar
now i
rm -rf test1
test1 has 4 levels of subdirs with tons reltivelly small files.
Here is what i get after 3 minutes
systat -io
/0% /10 /20 /30 /40 /50 /60 /70 /80 /90 /100
ada0 MB/sXXXX
tps|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX298.54
ada1 MB/sXXXX
tps|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX298.94
top
last pid: 69540; load averages: 2.48, 1.48, 1.16 up 30+16:55:02 19:19:52
767 processes: 2 running, 764 sleeping, 1 stopped
CPU: 0.9% user, 0.0% nice, 0.3% system, 0.2% interrupt, 98.6% idle
Mem: 8129M Active, 14G Inact, 3548M Wired, 333M Cache, 1655M Buf, 5722M Free
Swap: 4096M Total, 51M Used, 4045M Free, 1% Inuse
mysql
show processlist
205 rows
in
opening table
creating table
Web servers are dead.
I CTRL-C rm command
sync
took 230 seconds
did not help, now 1000 processes, over 300 stuck sql requests
killall -TERM httpd
wait, not help
sync
200 seconds
mysql query queue is empty
sysstat jumpt to 1000
then return to idle
restart httpd
everything is ok
TEST 2 for rm
rm -rf test1
sync every 60 seconds in parallel
All the same problem, sync every 60 s did not help.
TEST 3 for rm
/usr/bin/nice -n20 rm -rf test1
All the same
TEST 4 for rm
/usr/bin/nice -n20 rm -rf test1
fsync every 60 seconds in parallel
All the same
So, questions and thoughts:
1) Why i had no problem such this in fbsd 9? I think the reason for the
problem is in kernel, not in hardware or mariadb+nginx because server
load did not
increase at all, even decreases a little.
2) I consider it a sever bug, because even normal used (and i have
plenty of them using ssh) can eventually do rm -rf and kill all sites.
Which means there are
must be some way to limit io usage per user
Artem
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55170D9C.1070107>