Date: Wed, 2 Sep 1998 18:52:35 -0400 (EDT) From: Carl Mascott <cmascott@world.std.com> To: hackers@FreeBSD.ORG Subject: Re: Reading/writing /usr/ports is VERY slow Message-ID: <199809022252.SAA00868@europa.local>
next in thread | raw e-mail | index | archive | help
To go along with the discussion of FFS directory allocation policy and its effect on performance, here is my posting to comp.unix.bsd.freebsd.misc prior to the one that Luigi Rizzo <luigi@labinfo.iet.unipi.it> quoted from on this list. This earlier posting of mine contains my test data, which people might want to look at. Most of my discussion of softupdates isn't relevant here. I was refuting another posting to the newsgroup that was trying to suggest that softupdates was the cure for this performance problem. I have subscribed to hackers@freebsd.org to follow this, so it's no longer necessary to Cc me. ------------- Thanks to all who confirmed that the slow filesystem throughput on /usr/ports is a generic problem. I performed some experiments to try to pin down the cause of the slow read throughput, and I think I succeeded. I measured read throughput on /usr/ports and /var/tmp/ports, a subset of the ports collection that I constructed for these tests. (I didn't have room on /var for the whole collection.) The test consisted of tarring the directory tree to stdout and redirecting stdout to /dev/null. The reason for testing read throughput rather than write throughput is that reading is a simpler case: there are no allocations, and, with noatime, there are no metadata updates. /usr/ports /var/tmp/ports ---------- -------------- disk usage (du) 58243 8753 files 32575 4328 directories 12361 1647 avg files per directory 2.64 2.63 /usr filesystem: 512 MB, 8K/1K, 16 x 32 MB cylinder groups, 50% full, Seagate Barracuda 2LP /var filesystem: 30 MB, 8K/1K, 1 x 30 MB cylinder group, 46% full, Seagate Hawk 2LP FILESYSTEM READ THROUGHPUT, KB/S /usr/ports 142.4 /usr/ports, noatime 192.2 /var/tmp/ports 336.7 /var/tmp/ports, noatime 564.7 The throughput on /var/tmp/ports is 2.36x or 2.94x (noatime) greater than on /usr/ports. The composition of the two trees is similar. The difference is that /var/tmp/ports is contained entirely in one cylinder group (because that's all the filesystem has), while /usr/ports is spread over 16 cylinder groups. Remember, FFS places every new directory in a different cylinder group than that of its parent. This test actually understates the performance advantage of having the entire tree in one cylinder group because /var is on a slower disk than is /usr. This test shows that metadata updates are not the main performance problem when reading /usr/ports. Eliminating them entirely gives a read speedup of only 1.35x. This is an upper bound on the improvement that softupdates could make in read throughput. I don't have a feeling for the effect of metadata updates or the efficacy of the softupdates modifications where write throughput is concerned. I see write throughput of 9 KB/s in /usr/ports. Perhaps the softupdates mods would help to get the write throughput closer to the read throughput (i.e., improve it from abysmal to poor). In any event softupdates is not an option for me until it appears in -stable. To improve the read and write throughput from poor to fair, which is the best you'll ever do with something like /usr/ports, the seeks have got to be reduced a lot. Where to go from here? A tree like /usr/ports is really atypical (2.6 small files per directory). One wouldn't want to speed up FFS on /usr/ports if the result was to slow it down on more typical file I/O. That said, I still have to wonder if FFS's directory placement policy can't be improved upon. One definitely doesn't want to let a cylinder group fill up while there is still plenty of space in the filesystem. But it might be possible to accomplish this without spreading new directories all over the filesystem. By the way, another thing that suffers from FFS's directory placement policy is the find command, and not just when it is run in /usr/ports. Try running it in /usr/src if you have a moderate amount of source installed and just listen to your disk drive. I'm really curious about the directory placement policy in vxfs (Veritas), which also has cylinder groups (called allocation units). Does anybody happen to know what it is? ------------- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199809022252.SAA00868>