From owner-freebsd-fs@FreeBSD.ORG Fri Mar 15 13:12:57 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D7DDDC9C for ; Fri, 15 Mar 2013 13:12:57 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Received: from melon.pingpong.net (melon.pingpong.net [79.136.116.200]) by mx1.freebsd.org (Postfix) with ESMTP id 77AE8DFC for ; Fri, 15 Mar 2013 13:12:57 +0000 (UTC) Received: from girgBook.local (citron2.pingpong.net [195.178.173.68]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by melon.pingpong.net (Postfix) with ESMTPSA id 7A20F14D05; Fri, 15 Mar 2013 14:12:49 +0100 (CET) Message-ID: <51431E50.2020109@FreeBSD.org> Date: Fri, 15 Mar 2013 14:12:48 +0100 From: Palle Girgensohn User-Agent: Postbox 3.0.7 (Macintosh/20130119) MIME-Version: 1.0 To: Kirk McKusick Subject: Re: leaking lots of unreferenced inodes (pg_xlog files?), maybe after moving tables and indexes to tablespace on different volume References: <201303131652.r2DGqSr4051899@chez.mckusick.com> In-Reply-To: <201303131652.r2DGqSr4051899@chez.mckusick.com> X-Enigmail-Version: 1.2.3 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Jeff Roberson X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Mar 2013 13:12:57 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Kirk, Thanks for your reply! Kirk McKusick skrev: > Thanks for your report. It is certainly unlike anything that we have > seen reported before. > > Are you running your /usr filesystem with (the default) journalled > soft updates? You can check this by running the `mount' command with > no arguments. yes, plain vanilla: /dev/da0s1f on /usr (ufs, local, soft-updates) > > Rather than rebooting your system, it would be most helpful if you > could instead shut it down to single user. Then do the following: OK, we have planned downtime this evening. I will run the suggested commands in a script session before rebooting. Best regards, Palle > > Create a transcript of your session by running `script'. Once running > in the session run these commands: > > Run `mount' to show your filesystem configuration. Run `df -hi /usr' > to see whether the inodes are still missing. Verify that you can > cleanly unmount /usr (e.g., that the unmount does not hang and does > not complain). Remount /usr and run `df -hi' to see whether the > inodes are still missing. Unmount /usr again and run `fsck_ffs -p -f > -d /usr'. If the fsck_ffs fails with an unexpected inconsistency, you > can run `fsck_ffs -y -d /usr' to force it to clean up. When you have > the filesystem successfully cleaned up, type `exit' to get out of the > script session and mail me the transcript of the session > (typescript). > > Thanks for your help in tracking this down. > > Kirk McKusick > > ----- Original Message: > > Date: Wed, 13 Mar 2013 11:23:13 +0100 From: Palle Girgensohn > To: freebsd-fs@freebsd.org Subject: leaking lots > of unreferenced inodes (pg_xlog files?), maybe after moving tables > and indexes to tablespace on different volume > > Hi! > > Running postgresql-9.2.2 on FreeBSD 9.1 amd64 using vanilla ufs file > system. > > I have the postgresql base/ on the /usr disk, and a separate volume > /opt where the default tablespace resides. This means that the amount > of data on the /usr disk sould be stable. This is not the case, the > disk usage grows linearly (it seems to leave many inodes > unreferenced). > > The the discrepancy between df and du is now huge: > > # du -sxh /usr; df -h /usr 4,6G /usr Filesystem Size Used > Avail Capacity Mounted on /dev/da0s1f 104G 88G 8.0G 92% > /usr > > 4,6G vs 88GB, that must be more than a rounding error? > > Strange thing is I cannot find any open files among the missing. > > # lsof /usr| awk '{print $9}'|xargs ls -l > /dev/null > > returns no errors (a missing file would render an error with ls). If > there where open files not referenced in any directory, they should > be found. > > Next thing is fsck, and yes, there are plenty of unreferenced files. > > I ran fsck while system is running (i.e. read only) to get a grip > oif the amount of lost inodes: > > fsck /usr | awk '{print $1}'|cut -f 2 -d=| perl -e '$i = 0; while > (<>) { $i += $_;}; print $i / 1024 / 1024; print "\n";' > 85223.3530330658 > > ~85 GB gone, that's 80% of the disk, and it accounts fo all the > missing space. > > MTIME for the inodes are pretty evenly spread over time since the > machine was updated to FreeBSD 9.1, rebooted, and PostgreSQL was > updated to 9.2. All was done at the same time, so I can't really tell > who's to blaim, but this is the only server, out of a dozen that > where updated to exactly the same versions, that has this problem. > All other servers have their /usr disk usage stable (since all data > resides on a separate tablespace). > > The unreferenced inodes are almost exclusively around 16 MB in size, > so they most certainly all are postgresql pg_xlog files. This means > all files are lost from the same portion of code in the database > engine. > > How could it possibly be able to leave unreferenced inodes around > like this at such a scale? Is the culprit a combination of postgresql > and file system code? Both where updated. > > pg_xlog checkpoints seems to happen approximately every three > minutes: > > Mar 13 00:39:08 dbserver postgres[5298]: [48-1] db=,user= LOG: > checkpoint starting: time Mar 13 00:41:38 dbserver postgres[5298]: > [49-1] db=,user= LOG: checkpoint complete: wrote 2542 buffers (0.3%); > 0 transaction log file(s) added, 0 removed, 1 recycled; write=149.667 > s, sync=0.101 s, total=149.770 s; sync files=628, longest=0.021 s, > average=0.000 s Mar 13 00:44:08 dbserver postgres[5298]: [50-1] > db=,user= LOG: checkpoint starting: time Mar 13 00:46:38 dbserver > postgres[5298]: [51-1] db=,user= LOG: checkpoint complete: wrote 3996 > buffers (0.4%); 0 transaction log file(s) added, 0 removed, 1 > recycled; write=149.438 s, sync=0.111 s, total=149.551 s; sync > files=823, longest=0.006 s, average=0.000 s Mar 13 00:49:08 dbserver > postgres[5298]: [52-1] db=,user= LOG: checkpoint starting: time Mar > 13 00:51:38 dbserver postgres[5298]: [53-1] db=,user= LOG: checkpoint > complete: wrote 13736 buffers (1.4%); 0 transaction log file(s) > added, 0 removed, 2 recycled; write=149.958 s, sync=0.311 s, > total=150.271 s; sync files=1335, longest=0.079 s, average=0.000 s > Mar 13 00:54:08 dbserver postgres[5298]: [54-1] db=,user= LOG: > checkpoint starting: time Mar 13 00:56:38 dbserver postgres[5298]: > [55-1] db=,user= LOG: checkpoint complete: wrote 14638 buffers > (1.5%); 0 transaction log file(s) added, 0 removed, 17 recycled; > write=149.330 s, sync=0.271 s, total=149.603 s; sync files=1363, > longest=0.017 s, average=0.000 s Mar 13 00:59:08 dbserver > postgres[5298]: [56-1] db=,user= LOG: checkpoint starting: time Mar > 13 01:01:38 dbserver postgres[5298]: [57-1] db=,user= LOG: checkpoint > complete: wrote 8035 buffers (0.8%); 0 transaction log file(s) added, > 0 removed, 21 recycled; write=149.285 s, sync=0.146 s, total=149.433 > s; sync files=1160, longest=0.003 s, average=0.000 s Mar 13 01:04:08 > dbserver postgres[5298]: [58-1] db=,user= LOG: checkpoint starting: > time Mar 13 01:06:37 dbserver postgres[5298]: [59-1] db=,user= LOG: > checkpoint complete: wrote 2156 buffers (0.2%); 0 transaction log > file(s) added, 0 removed, 9 recycled; write=149.402 s, sync=0.057 s, > total=149.461 s; sync files=610, longest=0.000 s, average=0.000 s Mar > 13 01:09:08 dbserver postgres[5298]: [60-1] db=,user= LOG: checkpoint > starting: time > > > I'm pretty certain that unmounting the file system and running fsck > will regain the lost space, but will it stop there? > > Stopping postgresql briefly did not help, I tried that. That would > have helped if the files where open, but they're not. It seems to > postgresql did the right thing, and FreeBSD failed to unreference the > files. > > The server has about 30 databases and ~127 concurrent connections > (not all beeing active simultaneously, though), so it is fair to say > it is pretty active, but nothing extreme. > > Hardware is HP DL360, using their HT Smart Array P410i. > > Any ideas how to debug this? Or shall I just reboot, fsck, hope the > problem will go away, and when it does, forget about it? > > Thanks, Palle -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 > v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: > Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iQEcBAEBAgAGBQJRQFORAAoJEIhV+7FrxBJDzVUIAJHU011JDxLxj8/xg05Gwhgq > XK3xB+0N0NSUQ50yhcRKLINz/j/XfeS0ZxlH+MstaPA9y0r1JUXMxkb/uTUvGBiy > jutk3eVe0cati9cVZbJkRU5FxEgmQ0fg0GOMl3RQAErkh5achj+klWvN7PnwGjTs > O3L9RgckKuxTJffk52GAS05qY/TKR6f08kdX3I2cFtqw3tyTyrXU0JPdk2snuPhv > H40xV46zgtWMFDvZLt61MryQ7/JotVQwU78scUB+zxrf8KKM9V0mM7pk0pIbG4Qw > NJBpZJ5gjbl4x+dkQrtZdL65yq88hACYwo9D+83Ct4ig8tgcQ7ViNHWxJqknK7Q= > =3ZZs -----END PGP SIGNATURE----- > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, > send any mail to "freebsd-fs-unsubscribe@freebsd.org" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJRQx5QAAoJEIhV+7FrxBJDAcYIALvj4hiWoAN/wchrJIbfiXbY XcPNIuqKFT1sRYWgdLZQ7e34zmvtmPfa0WW6/OFHbI5q+G/xciuZLhTl7EZ98IvD jbBUR4SLLcrOvFNe35b43eOqr12okIboLg2fQx/jUbWQM19V/2/YaLobBDl2iv/v gbD5ErL3yd0YBU1EFETho3hsL9fzbmSczQqhWWs0glD+aiHDQbtIAFVkC3IZSaLl MNhqrzKsv4kEHXSylYRU2RbHYKNg55jQ1JHA5HKinqZbe7qLmyqr4dFVtbYEgGE9 DCh4/buO0/UIg+Te7WuD2XxMhfutgbGN6kOaTXk3NQhtgd5a/8I/yqfv6zGtglY= =UFMm -----END PGP SIGNATURE-----