From owner-freebsd-fs  Fri Aug 13  6:13:20 1999
Delivered-To: freebsd-fs@freebsd.org
Received: from godzilla.zeta.org.au (godzilla.zeta.org.au [203.26.10.9])
	by hub.freebsd.org (Postfix) with ESMTP id 975C414C25
	for <freebsd-fs@FreeBSD.ORG>; Fri, 13 Aug 1999 06:13:12 -0700 (PDT)
	(envelope-from bde@godzilla.zeta.org.au)
Received: (from bde@localhost)
	by godzilla.zeta.org.au (8.8.7/8.8.7) id XAA14647;
	Fri, 13 Aug 1999 23:13:21 +1000
Date: Fri, 13 Aug 1999 23:13:21 +1000
From: Bruce Evans <bde@zeta.org.au>
Message-Id: <199908131313.XAA14647@godzilla.zeta.org.au>
To: phk@critter.freebsd.dk, vernick@bell-labs.com
Subject: Re: Help with understand file system performance
Cc: freebsd-fs@FreeBSD.ORG
Sender: owner-freebsd-fs@FreeBSD.ORG
Precedence: bulk
X-Loop: FreeBSD.org

>>3. On a per process basis, performance increases when the number of
>>files per directory increases/number of subdirs decreases.  Is this
>>because there is a better chance the directory information about the
>>file could be in memory?
>
>Yes.  The minimum directory size is the fragsize of the filesystem,
>filling the directories better means better performance.
>
>>Our goal, of course, it to maximize performance.  So any help in the
>>tuning of our system (i.e. reading lots of ~15KB files) would be
>>appreciated

Try increasing nbuf.  I think effective caching of directories still
requires 1 buffer per directory.  

>Try fiddling the newfs parameters.  I see 17% speedup using:
>
>	newfs -b 16384 -f 4096 -c 100

I see a 48% speedup using

	linux
	mkfs.ext2 -b 4096 $device $size_of_device_in_4k_units

:->.  This is despite (or because of) ext2fs's block allocator being
broken (it essentially ignores cylinder groups).

The following times are for `tar zxpf linux-2.2.9.tar.gz', unmount
(to sync), and `tar cf /dev/null linux lost+found' on a new filesystem
on a Quantum KA disk on an overclocked Celeron-366 system:

ffs-4096-512:
       41.82 real         3.24 user         3.34 sys
        3.05 real         0.00 user         0.07 sys
       16.53 real         0.08 user         1.34 sys
ffs-4096-1024:
       35.89 real         3.35 user         3.70 sys
        2.11 real         0.00 user         0.07 sys
       15.53 real         0.13 user         1.39 sys
ffs-4096-2048:
       29.32 real         3.24 user         4.36 sys
        1.17 real         0.00 user         0.07 sys
       12.20 real         0.14 user         1.42 sys
ffs-4096-4096:
       28.85 real         3.34 user         4.51 sys
        1.12 real         0.00 user         0.07 sys
       11.24 real         0.10 user         1.59 sys
ffs-8192-1024:
       33.39 real         3.26 user         5.44 sys
        2.94 real         0.00 user         0.07 sys
       13.40 real         0.12 user         1.18 sys
ffs-8192-2048:
       28.08 real         3.29 user         3.01 sys
        2.32 real         0.00 user         0.07 sys
       11.21 real         0.06 user         1.26 sys
ffs-8192-4096:
       25.05 real         3.27 user         2.99 sys
        1.87 real         0.00 user         0.07 sys
        9.17 real         0.09 user         1.21 sys
ffs-8192-8192:
       23.27 real         3.27 user         2.82 sys
        1.53 real         0.00 user         0.07 sys
        8.94 real         0.10 user         1.23 sys
ffs-16384-2048:
       28.22 real         3.43 user         4.78 sys
        2.52 real         0.00 user         0.07 sys
       12.01 real         0.10 user         1.55 sys
ffs-16384-4096:
       24.32 real         3.41 user         3.51 sys
        1.97 real         0.00 user         0.07 sys
       10.56 real         0.11 user         1.37 sys
ffs-16384-8192:
       23.63 real         3.33 user         3.35 sys
        2.35 real         0.00 user         0.07 sys
        8.66 real         0.09 user         1.15 sys
ffs-16384-16384:
       85.41 real         3.33 user         3.28 sys
        2.00 real         0.00 user         0.08 sys
        9.51 real         0.10 user         1.17 sys
ext2fs-1024-1024:
       36.33 real         3.33 user         3.67 sys
        1.42 real         0.00 user         0.07 sys
       14.49 real         0.10 user         2.28 sys
ext2fs-4096-4096:
       20.81 real         3.38 user         3.54 sys
        1.01 real         0.00 user         0.07 sys
        6.96 real         0.12 user         1.57 sys

Note the anomalously slow times for ffs-16384-16384.

I analyzed why ffs was slow and ext2fs was fast for the `tar cf'
part a year or two ago.  It was because ffs handles fragments poorly
and needs many more small (but not small enough to be in the drive's
cache) backwards seeks.  Not using fragments reduced ext2fs's advantage
significantly but not completely.  ffs-4K-4K is only slightly faster
than ffs-8K-1K now, presumably because drive caches are larger and
command overheads are relatively higher (the KA acts like a slow
SCSI drive in wanting a block size of at least 8K to keep up with
the disk).

This output was produced by the following program:

---
#!/bin/sh

for b in 4096 8192 16384
do
	for f in $(($b / 8)) $(($b / 4)) $(($b / 2)) $b
	do
		echo ffs-$b-$f: >>/tmp/ztimes
		newfs -b $b -f $f /dev/rwd2s2a
		mount /dev/wd2s2a /d
		cd /d
		sync
		time tar zxpf $loc/z/dist/*2.2.9.tar.gz 2>>/tmp/ztimes
		cd /tmp
		time umount /d 2>>/tmp/ztimes
		mount /dev/wd2s2a /d
		cd /d
		sync
		time tar cf /dev/null * 2>>/tmp/ztimes
		cd /tmp
		umount /d
	done
done

for b in 1024 4096
do
	for f in $b
	do
		echo ext2fs-$b-$f: >>/tmp/ztimes
		# linux
		mkfs.ext2 -b $b /dev/rwd2s2a $((4819437 / ($b / 512)))
		# fsck.ext2 /dev/wd2s2a
		mount -t ext2fs /dev/wd2s2a /d
		cd /d
		sync
		time tar zxpf $loc/z/dist/*2.2.9.tar.gz 2>>/tmp/ztimes
		cd /tmp
		time umount /d 2>>/tmp/ztimes
		mount -t ext2fs /dev/wd2s2a /d
		cd /d
		sync
		time tar cf /dev/null * 2>>/tmp/ztimes
		cd /tmp
		umount /d
	done
done
---

Bruce


To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-fs" in the body of the message