Date: Wed, 19 Dec 2007 17:09:48 -0200 From: "Alexandre Biancalana" <biancalana@gmail.com> To: freebsd-performance@freebsd.org Subject: Bad performance when accessing a lot of small files Message-ID: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi List,
I have a backup server running FreeBSD 7-BETA3. The cpu is CPU:
Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz, 3GB Ram, 10x 500GB
SATA, Areca 1231-ML, the filesystem used to backup my other servers
locally is build on top of ARC-1231, 4TB (32k stripe) zfs filesystem
with gzip compression.
This machine receive backups from ~30 servers, (of all kinds and
sizes, databases, fileservers, image servers, webservers, etc) all
night, write the last day in LTO-3 tapes and store some days older
days in disk.
The behavior that I'm observing and that want your help is when the
system is accessing some directory with many small files ( directories
with ~ 1 million of ~30kb files), the performance is very poor.
Here is the output of iostat -x 1 when the system is accessing (and
writing tapes) with normal (no many small files) directories:
extended device statistics
device r/s w/s kr/s kw/s wait svc_t b
ad0 0.0 0.0 0.0 0.0 0 0.0 0
da0 429.8 0.0 19774.1 0.0 6 12.1 81
sa0 0.0 3800.7 0.0 38007.3 1 0.2 88
sa1 0.0 2637.6 0.0 26375.8 1 0.3 92
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
pass4 0.0 0.0 0.0 0.0 0 0.0 0
ch0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t b
ad0 0.0 0.0 0.0 0.0 0 0.0 0
da0 1196.0 0.0 58474.9 0.0 3 4.3 78
sa0 0.0 3021.1 0.0 30210.7 1 0.3 78
sa1 0.0 3029.8 0.0 30298.0 1 0.3 79
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
pass4 0.0 0.0 0.0 0.0 0 0.0 0
ch0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t b
ad0 0.0 8.1 0.0 104.9 0 1.8 0
da0 1141.4 0.0 63241.1 0.0 0 2.5 39
sa0 0.0 1660.5 0.0 16605.5 0 0.3 42
sa1 0.0 1701.2 0.0 17012.5 1 0.2 42
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
pass4 0.0 0.0 0.0 0.0 0 0.0 0
ch0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t b
ad0 0.0 0.0 0.0 0.0 0 0.0 0
da0 1400.8 140.1 77701.0 4185.3 0 2.0 44
sa0 0.0 2762.7 0.0 27626.7 1 0.3 72
sa1 0.0 2775.3 0.0 27753.1 1 0.3 72
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
pass4 0.0 0.0 0.0 0.0 0 0.0 0
ch0 0.0 0.0 0.0 0.0 0 0.0 0
Here is the output when accessing some directory with many small files:
extended device statistics
device r/s w/s kr/s kw/s wait svc_t b
ad0 0.0 0.0 0.0 0.0 0 0.0 0
da0 738.6 0.0 38628.1 0.0 7 10.0 114
sa0 0.0 6001.2 0.0 60011.5 0 0.1 88
sa1 0.0 0.0 0.0 0.0 0 0.0 0
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
pass4 0.0 0.0 0.0 0.0 0 0.0 0
ch0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t b
ad0 0.0 0.0 0.0 0.0 0 0.0 0
da0 539.9 75.6 26989.5 1808.7 15 9.2 78
sa0 0.0 4842.5 0.0 48425.0 0 0.1 72
sa1 0.0 0.0 0.0 0.0 0 0.0 0
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
pass4 0.0 0.0 0.0 0.0 0 0.0 0
ch0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t b
ad0 0.0 0.0 0.0 0.0 0 0.0 0
da0 550.5 40.4 25794.9 321.5 6 12.0 99
sa0 0.0 5928.0 0.0 59280.1 1 0.1 89
sa1 0.0 0.0 0.0 0.0 0 0.0 0
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
pass4 0.0 0.0 0.0 0.0 0 0.0 0
ch0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t b
ad0 0.0 0.0 0.0 0.0 0 0.0 0
da0 972.2 0.0 47698.7 0.0 8 7.6 93
sa0 0.0 3840.6 0.0 38406.2 1 0.2 63
sa1 0.0 2015.9 0.0 20158.9 1 0.2 36
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
pass4 0.0 0.0 0.0 0.0 0 0.0 0
ch0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t b
ad0 0.0 10.6 0.0 169.1 0 1.9 0
da0 357.4 0.0 18103.7 0.0 6 18.7 97
sa0 0.0 3476.1 0.0 34761.3 1 0.3 94
sa1 0.0 3476.1 0.0 34761.3 1 0.3 94
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0
pass2 0.0 0.0 0.0 0.0 0 0.0 0
pass3 0.0 0.0 0.0 0.0 0 0.0 0
pass4 0.0 0.0 0.0 0.0 0 0.0 0
ch0 0.0 0.0 0.0 0.0 0 0.0 0
The bigger difference can be seen in the times elapsed to write some
directories to tape, here are some examples, following the number of
files, directory size and time to write this:
Normal directories (without a lot of small files):
3369 files, 169 GB, 1h 50 min
7049 files, 14 GB, 14 min
2764 files, 6 GB, 8 min
Directories with small files:
1125796 files, 30 GB, 1h 31 min
1325589 files, 36 GB, 1h, 40 min
861632 files, 80 GB, 10h 40 min
1860893 files, 654 GB, 10h 54min
I can provide any other necessary information.
Any help is *very* appreciated !
Best Regards,
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce>