From owner-freebsd-fs@FreeBSD.ORG Wed Nov 2 13:13:13 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 98CDE106566C for ; Wed, 2 Nov 2011 13:13:13 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta08.emeryville.ca.mail.comcast.net (qmta08.emeryville.ca.mail.comcast.net [76.96.30.80]) by mx1.freebsd.org (Postfix) with ESMTP id 8092C8FC08 for ; Wed, 2 Nov 2011 13:13:13 +0000 (UTC) Received: from omta11.emeryville.ca.mail.comcast.net ([76.96.30.36]) by qmta08.emeryville.ca.mail.comcast.net with comcast id sD7K1h0010mlR8UA8DD5hT; Wed, 02 Nov 2011 13:13:05 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta11.emeryville.ca.mail.comcast.net with comcast id sDA91h00g1t3BNj8XDAA4Y; Wed, 02 Nov 2011 13:10:10 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id BA7B8102C19; Wed, 2 Nov 2011 06:13:11 -0700 (PDT) Date: Wed, 2 Nov 2011 06:13:11 -0700 From: Jeremy Chadwick To: Borja Marcos Message-ID: <20111102131311.GA56941@icarus.home.lan> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: Default inode number too low in FFS nowadays? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Nov 2011 13:13:13 -0000 On Wed, Nov 02, 2011 at 12:57:33PM +0100, Borja Marcos wrote: > Today I?ve come across an issue long ago forgotten :) Running out of i-nodes. The last time I dealt with this was about 3 months ago, on a Netapp filer; hardly forgotten. :-) > The condition was triggered on a 8 GB /usr filesystem which includes the ports tree, on which I have compiled several ports and their dependencies. I know, maybe not the best practices, it's a machine being used to test a couple of thingies, but I wonder how many newbies can run into such a problem. I guess many. > > Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on > /dev/ad1s1e 8245660 5308960 2277048 70% 507099 61219 89% /usr > > Now i-nodes aren't completely full, as I have deleted /usr/obj (turns out I had made a make world as well). > > Looking at the number of i-nodes per /usr subdirectory, I have noticed that, wow! /usr/ports consumes A LOT of them. > > freebsd9-borja# find . -print | wc -l > 405481 "find -x /usr/ports | wc -l" on my system returns 146420. "find -x /usr | wc -l" on my system returns 264901 (and this includes a populated /usr/obj buildworld/buildkernel tree. /usr/obj returns 37126, just for the record). $ df -k -i /usr Filesystem 1024-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ada0s1f 38110116 3425114 31636194 10% 258980 4686938 5% /usr Your 8GB filesystem is only ~21% the capacity of mine. So if we do some simple math, your /usr should really be showing (assuming yours is as "bare bones" as mine) around 26% inodes used. Yet what you have is 89%. You probably just have a lot of other software on /usr that "abuses" lots of individual files. You list off what ports you built from source, which means you have lots of "build cruft" laying around en masse (work/ directories). > I know the /usr isn't that large, and I know that I have compiled a > lot on this battered /usr filesystem, but with the increasing > complexity of many programs and the increased number of dependences in > ports, I would: > > 1) At least double the default number of i-nodes per filesystem > > 2) Suggest strongly in sysinstall to create a /usr/ports filesystem > for the ports. But probably this won't be popular and many people will > tend to create a single large root to rule them all.... Really? That isn't what the default installer does (re: single filesystem), and hasn't since... well, as long as I can remember (2.2.8). If someone chooses to create a single / for all of their stuff, so be it. It's their choice. This is very much a "Linux-ism"; I used to do the same thing back when I used Linux in the early 90s and was quickly taught why it's both useful *and* bad. > What do you think? Here's what I think: I wouldn't recommend a /usr/ports filesystem. It solves nothing. Imagine: /usr/ports can now run out of inodes while /usr won't. What would this address? It would decrease impact to applications creating new files on /usr. How often do you have that happening on your system? For me it's extremely rare. For comparison, /var gets a lot more activity. Adding a /usr/ports filesystem also provides zero scalability. Meaning: say someone decides "8GB for /usr/ports is enough" during the installer and so on. Then down the road, we run out of space, or inodes, or whatever else. Oops, what now? Symlinks all over the place? Yeah, not going to happen. We were already in this boat regarding the root filesystem size, which for quite some time defaulted to 512MBytes, which was not enough. Today, the default size is 1GBytes, which is enough for a kernel built with debugging symbols and so on, but tends to reach ~55-58% used. There's a reason /usr on FreeBSD defaults to "all the remaining space on the disk" if you pick the defaults/auto. Surprise. :-) The summarised version is: 1. You have control over this yourself: newfs(8) -i flag. You can even input this flag during sysinstall when building a new system. 2. When you built the system, you chose to assume /usr should be 8GBytes. Your usage habits may have changed, software gotten larger, etc.. It may be time for you to re-evaluate how you create your filesystems and what sizes you choose. You chose something that didn't scale. :-) 3. Get in the habit of cleaning up after yourself. Don't leave work/ directories laying around inside your ports ("make install clean" solves this problem, but the clean phase only happens at the end of whatever is being built, including dependencies). 4. For a limited-space (not just size but inodes!) filesystem, consider using packages (pkg_add -r) for anything that does not require customisation during the build/configure phase. For example, for quite some time now on my systems, I've always installed the package versions of perl, python27, cmake, gettext, pcre, etc. solely because they are "key softwares" that I hate building from source. 5. Most FreeBSD Ports folks are aware and respectful as much as possible of limited-space environments, especially due to those on embedded platforms. A very hot topic/debate many years ago was whether or not there should be a new file introduced into each ports' directory to define some dependency bits. Obviously it did not happen. I'm not sure all committers are aware of this though. For example, a recent (new) port I submitted contained 2 files in files/ (two patches). The committer chose to split this up into 14 files (which not only wastes more inodes, but also makes my life as port maintainer a bigger PITA given where the patches come from). You get the idea I'm sure. > I can't imagine what it might have been if I had compiled Gnome or KDE... :D I imagine you would have experienced that your filesystem ran out of inodes. Oh wait, you said it did... :-) "What" was compiled has no bearing on the situation; failure is failure, whether you were building Office.org or compiling a 4-file C program. > For the record, the ports I had compiled are: > {snip} -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |