Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 22 Oct 2003 12:57:53 -0700 (PDT)
From:      julian@vicor.com (Julian Elischer)
To:        freebsd-fs@freebsd.org, kmarx@vicor.com, mckusick@mckusick.com
Cc:        davep@vicor.com
Subject:   Re: 4.8 ffs_dirpref problem
Message-ID:  <20031022195753.27C707A49F@mail.vicor-nb.com>
In-Reply-To: <3F95D3F3.2050203@vicor.com>

next in thread | previous in thread | raw e-mail | index | archive | help
Kirk?, 
I'm away in easwtern europe on the end of a wet bit of string....


>From kmarx@vicor.com  Tue Oct 21 17:53:42 2003
	X-Original-To: julian@vicor-nb.com
	Delivered-To: julian@vicor-nb.com
	Date: Tue, 21 Oct 2003 17:48:51 -0700
	From: Ken Marx <kmarx@vicor.com>
	User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.3b) Gecko/20030402
	X-Accept-Language: en-us, en
	MIME-Version: 1.0
	To: freebsd-fs@freebsd.org
	Cc: Julian Elischer <julian@vicor-nb.com>,
		John Lynch <jpl@vicor.com>, Dave Parker Smith <davep@vicor.com>,
		Cayford Burrell <cburrell@vicor.com>,
		victor elischer <VicPE@aol.com>, Josh Howard <jrh@vicor.com>,
		Ken Marx <kmarx@vicor.com>
	Subject: 4.8 ffs_dirpref problem
	Content-Type: text/plain; charset=us-ascii; format=flowed
	Content-Transfer-Encoding: 7bit

	Hi,

	We have 560GB raids that were sometimes bogging down heavily
	in our production systems. Under 4.8-RELEASE (recently upgrated from 4.4)
	we find that when:

		o the raid file system grows to over 85% capacity (with only
		  30% inode usage)
		o we create ~1500 or so 2-6kb files in a given dir
		o (note: soft updates NOT enabled)

	We see:

		o 100% cpu utilization, all in system
		o I/O transfer rates of ~200kb/sec, down from normal of 15-30MB/s

	We profiled the kernel and found a large number of calls to ffs_alloc().
	After many twisty pasages, we finally diff'd 4.4 with 4.8 ffs_alloc.c,
	and found a major difference in the ffs_dirpref() call. Hacking the
	4.4 logic back in 'fixed' the problem: We can now fill the /raid
	entirely with no real noticeable performance degradation.

	The nice comments for 4.4/4.8 versions of ffs_dirpref() seem to explain
	things fairly clearly:

	4.4 -  ffs_alloc.c,v 1.64.2.1 2000/03/16 08:15:53 ps:
	--------------------------------------
	 * The policy implemented by this algorithm is to select from
	 * among those cylinder groups with above the average number of
	 * free inodes, the one with the smallest number of directories.

	4.8 - ffs_alloc.c,v 1.64.2.2 2001/09/21 19:15:21 dillon:
	-----------------------------------------
	 * The policy implemented by this algorithm is to allocate a
	 * directory inode in the same cylinder group as its parent
	 * directory, but also to reserve space for its files inodes
	 * and data. Restrict the number of directories which may be
	 * allocated one after another in the same cylinder group
	 * without intervening allocation of files.
	 *
	 * If we allocate a first level directory then force allocation
	 * in another cylinder group.

	For us, the 4.4 policy seems far superior, at least when the file system
	approches capacity.

	We'd like to avoid local kernel hacks and keep with main line
	FreeBSD code. Is there some way that the old policy can be supported,
	perhaps via a tunefs or sysctl type option?

	Actually, if the new policy can be fixed up to avoid the problem, that
	would of course be just as dandy.

	Thanks very much,
	k
	-- 
	Ken Marx, kmarx@vicor-nb.com
	We need to hit the nail on the head and set the agenda regarding total 
	quality.
			- http://www.bigshed.com/cgi-bin/speak.cgi




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20031022195753.27C707A49F>