Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 2 Sep 1998 18:52:35 -0400 (EDT)
From:      Carl Mascott <cmascott@world.std.com>
To:        hackers@FreeBSD.ORG
Subject:   Re: Reading/writing /usr/ports is VERY slow
Message-ID:  <199809022252.SAA00868@europa.local>

next in thread | raw e-mail | index | archive | help
To go along with the discussion of FFS directory allocation
policy and its effect on performance, here is my posting to
comp.unix.bsd.freebsd.misc prior to the one that
Luigi Rizzo <luigi@labinfo.iet.unipi.it> quoted from on this
list.  This earlier posting of mine contains my test data,
which people might want to look at.

Most of my discussion of softupdates isn't relevant here.
I was refuting another posting to the newsgroup that
was trying to suggest that softupdates was the cure for
this performance problem.

I have subscribed to hackers@freebsd.org to follow this,
so it's no longer necessary to Cc me.

-------------

Thanks to all who confirmed that the slow filesystem throughput
on /usr/ports is a generic problem.

I performed some experiments to try to pin down the cause of
the slow read throughput, and I think I succeeded.  I measured
read throughput on /usr/ports and /var/tmp/ports, a subset
of the ports collection that I constructed for these tests.
(I didn't have room on /var for the whole collection.)
The test consisted of tarring the directory tree to stdout and
redirecting stdout to /dev/null.  The reason for testing read
throughput rather than write throughput is that reading is a
simpler case: there are no allocations, and, with noatime,
there are no metadata updates.

				/usr/ports	/var/tmp/ports
				----------	--------------
disk usage (du)			58243		8753
files				32575		4328
directories			12361		1647
avg files per directory		2.64		2.63


/usr filesystem: 512 MB, 8K/1K, 16 x 32 MB cylinder groups, 50% full,
			Seagate Barracuda 2LP

/var filesystem: 30 MB, 8K/1K, 1 x 30 MB cylinder group, 46% full,
			Seagate Hawk 2LP


		FILESYSTEM READ THROUGHPUT, KB/S

/usr/ports			142.4
/usr/ports, noatime		192.2
/var/tmp/ports			336.7
/var/tmp/ports, noatime		564.7


The throughput on /var/tmp/ports is 2.36x or 2.94x (noatime)
greater than on /usr/ports.  The composition of the two
trees is similar.  The difference is that /var/tmp/ports is
contained entirely in one cylinder group (because that's all
the filesystem has), while /usr/ports is spread over 16
cylinder groups.  Remember, FFS places every new directory
in a different cylinder group than that of its parent.

This test actually understates the performance advantage of
having the entire tree in one cylinder group because /var
is on a slower disk than is /usr.

This test shows that metadata updates are not the main
performance problem when reading /usr/ports.  Eliminating
them entirely gives a read speedup of only 1.35x.  This
is an upper bound on the improvement that softupdates
could make in read throughput.

I don't have a feeling for the effect of metadata updates
or the efficacy of the softupdates modifications where
write throughput is concerned.  I see write throughput
of 9 KB/s in /usr/ports.  Perhaps the softupdates mods
would help to get the write throughput closer to the
read throughput (i.e., improve it from abysmal to poor).

In any event softupdates is not an option for me until
it appears in -stable.

To improve the read and write throughput from poor to
fair, which is the best you'll ever do with something
like /usr/ports, the seeks have got to be reduced a lot.

Where to go from here?  A tree like /usr/ports is really
atypical (2.6 small files per directory).  One wouldn't
want to speed up FFS on /usr/ports if the result was to
slow it down on more typical file I/O.  That said, I still
have to wonder if FFS's directory placement policy can't
be improved upon.  One definitely doesn't want to let a
cylinder group fill up while there is still plenty of
space in the filesystem.  But it might be possible to
accomplish this without spreading new directories all
over the filesystem.

By the way, another thing that suffers from FFS's
directory placement policy is the find command, and not
just when it is run in /usr/ports.  Try running it in
/usr/src if you have a moderate amount of source
installed and just listen to your disk drive.

I'm really curious about the directory placement policy in
vxfs (Veritas), which also has cylinder groups (called
allocation units).  Does anybody happen to know what it is?

-------------

To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199809022252.SAA00868>