Date: Mon, 05 Oct 2009 10:55:45 PDT From: Dieter <freebsd@sopwith.solgatos.com> To: freebsd-performance@freebsd.org Subject: tuning FFS for large files Re: A specific example of a disk i/o problem Message-ID: <200910051755.RAA11047@sopwith.solgatos.com> In-Reply-To: Your message of "Thu, 01 Oct 2009 09:10:43 EDT." <20091001091043.477f4b9b.wmoran@collaborativefusion.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I found a clue! The problem occurs with my big data partitions, which are newfs-ed with options intended to improve things. Reading a large file from the normal ad4s5b partition only delays other commands slightly, as expected. Reading a large file from the tuned ad4s11 partition yields the delay of minutes for other i/o. # tunefs -p /dev/ad4s5b tunefs: ACLs: (-a) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: gjournal: (-J) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 1024 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: optimization preference: (-o) time tunefs: volume label: (-L) # tunefs -p /dev/ad4s11 tunefs: ACLs: (-a) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: gjournal: (-J) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 57984 tunefs: average file size: (-f) 67108864 tunefs: average number of files in a directory: (-s) 16 tunefs: minimum percentage of free space: (-m) 8% tunefs: optimization preference: (-o) time tunefs: volume label: (-L) Here is the newfs command used for creating large data partitions: newfs -e 57984 -b 65536 -f 8192 -g 67108864 -h 16 -i 67108864 -U -o time $partition Even this isn't tuned the way I wanted to. -g * -h must be less than 4 G due to 32 bit problem (system panics). Note the 32 bit problem is in the filesystem code, I'm running amd64. IIRC there is a PR about this. (I'm assuming the bug hasn't been fixed yet) Result is that I must specify -g and -h smaller than they should be. And they have way more inodes than needed. (IIRC it doesn't actually use -i 67108864) Partition sizes range from ~200 GB to 1.5 TB. A small number of directories, perhaps a dozen. About 1/2 the files are small, a few KB, the other half are large, 500MB-25GB. Goals are to minimize seeking and use space efficiently. Files are written sequentially, read mostly (99.99%) sequentially, no editing. I still think it may have something to do with a common resource such as disk cache, as it can affect other disks, including swap. Hmmm, does swap use the disk cache? But now it appears that FFS tuning is the cause of the bottleneck. Do these tuning parameters increase the number of some data structure needed? Can I crank that up? Do I have to change the FFS tuning? Suggestions?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200910051755.RAA11047>