From owner-freebsd-fs@FreeBSD.ORG Wed Mar 13 16:52:34 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 037127ED; Wed, 13 Mar 2013 16:52:34 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [IPv6:2001:5a8:4:7e72:4a5b:39ff:fe12:452]) by mx1.freebsd.org (Postfix) with ESMTP id A38E874F; Wed, 13 Mar 2013 16:52:33 +0000 (UTC) Received: from chez.mckusick.com (localhost [127.0.0.1]) by chez.mckusick.com (8.14.3/8.14.3) with ESMTP id r2DGqSr4051899; Wed, 13 Mar 2013 09:52:29 -0700 (PDT) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201303131652.r2DGqSr4051899@chez.mckusick.com> To: Palle Girgensohn Subject: Re: leaking lots of unreferenced inodes (pg_xlog files?), maybe after moving tables and indexes to tablespace on different volume In-reply-to: <51405391.1020006@FreeBSD.org> Date: Wed, 13 Mar 2013 09:52:28 -0700 From: Kirk McKusick X-Spam-Status: No, score=0.0 required=5.0 tests=MISSING_MID, UNPARSEABLE_RELAY autolearn=failed version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on chez.mckusick.com Cc: freebsd-fs@freebsd.org, Jeff Roberson X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Mar 2013 16:52:34 -0000 Thanks for your report. It is certainly unlike anything that we have seen reported before. Are you running your /usr filesystem with (the default) journalled soft updates? You can check this by running the `mount' command with no arguments. Rather than rebooting your system, it would be most helpful if you could instead shut it down to single user. Then do the following: Create a transcript of your session by running `script'. Once running in the session run these commands: Run `mount' to show your filesystem configuration. Run `df -hi /usr' to see whether the inodes are still missing. Verify that you can cleanly unmount /usr (e.g., that the unmount does not hang and does not complain). Remount /usr and run `df -hi' to see whether the inodes are still missing. Unmount /usr again and run `fsck_ffs -p -f -d /usr'. If the fsck_ffs fails with an unexpected inconsistency, you can run `fsck_ffs -y -d /usr' to force it to clean up. When you have the filesystem successfully cleaned up, type `exit' to get out of the script session and mail me the transcript of the session (typescript). Thanks for your help in tracking this down. Kirk McKusick ----- Original Message: Date: Wed, 13 Mar 2013 11:23:13 +0100 From: Palle Girgensohn To: freebsd-fs@freebsd.org Subject: leaking lots of unreferenced inodes (pg_xlog files?), maybe after moving tables and indexes to tablespace on different volume Hi! Running postgresql-9.2.2 on FreeBSD 9.1 amd64 using vanilla ufs file system. I have the postgresql base/ on the /usr disk, and a separate volume /opt where the default tablespace resides. This means that the amount of data on the /usr disk sould be stable. This is not the case, the disk usage grows linearly (it seems to leave many inodes unreferenced). The the discrepancy between df and du is now huge: # du -sxh /usr; df -h /usr 4,6G /usr Filesystem Size Used Avail Capacity Mounted on /dev/da0s1f 104G 88G 8.0G 92% /usr 4,6G vs 88GB, that must be more than a rounding error? Strange thing is I cannot find any open files among the missing. # lsof /usr| awk '{print $9}'|xargs ls -l > /dev/null returns no errors (a missing file would render an error with ls). If there where open files not referenced in any directory, they should be found. Next thing is fsck, and yes, there are plenty of unreferenced files. I ran fsck while system is running (i.e. read only) to get a grip oif the amount of lost inodes: fsck /usr | awk '{print $1}'|cut -f 2 -d=| perl -e '$i = 0; while (<>) { $i += $_;}; print $i / 1024 / 1024; print "\n";' 85223.3530330658 ~85 GB gone, that's 80% of the disk, and it accounts fo all the missing space. MTIME for the inodes are pretty evenly spread over time since the machine was updated to FreeBSD 9.1, rebooted, and PostgreSQL was updated to 9.2. All was done at the same time, so I can't really tell who's to blaim, but this is the only server, out of a dozen that where updated to exactly the same versions, that has this problem. All other servers have their /usr disk usage stable (since all data resides on a separate tablespace). The unreferenced inodes are almost exclusively around 16 MB in size, so they most certainly all are postgresql pg_xlog files. This means all files are lost from the same portion of code in the database engine. How could it possibly be able to leave unreferenced inodes around like this at such a scale? Is the culprit a combination of postgresql and file system code? Both where updated. pg_xlog checkpoints seems to happen approximately every three minutes: Mar 13 00:39:08 dbserver postgres[5298]: [48-1] db=,user= LOG: checkpoint starting: time Mar 13 00:41:38 dbserver postgres[5298]: [49-1] db=,user= LOG: checkpoint complete: wrote 2542 buffers (0.3%); 0 transaction log file(s) added, 0 removed, 1 recycled; write=149.667 s, sync=0.101 s, total=149.770 s; sync files=628, longest=0.021 s, average=0.000 s Mar 13 00:44:08 dbserver postgres[5298]: [50-1] db=,user= LOG: checkpoint starting: time Mar 13 00:46:38 dbserver postgres[5298]: [51-1] db=,user= LOG: checkpoint complete: wrote 3996 buffers (0.4%); 0 transaction log file(s) added, 0 removed, 1 recycled; write=149.438 s, sync=0.111 s, total=149.551 s; sync files=823, longest=0.006 s, average=0.000 s Mar 13 00:49:08 dbserver postgres[5298]: [52-1] db=,user= LOG: checkpoint starting: time Mar 13 00:51:38 dbserver postgres[5298]: [53-1] db=,user= LOG: checkpoint complete: wrote 13736 buffers (1.4%); 0 transaction log file(s) added, 0 removed, 2 recycled; write=149.958 s, sync=0.311 s, total=150.271 s; sync files=1335, longest=0.079 s, average=0.000 s Mar 13 00:54:08 dbserver postgres[5298]: [54-1] db=,user= LOG: checkpoint starting: time Mar 13 00:56:38 dbserver postgres[5298]: [55-1] db=,user= LOG: checkpoint complete: wrote 14638 buffers (1.5%); 0 transaction log file(s) added, 0 removed, 17 recycled; write=149.330 s, sync=0.271 s, total=149.603 s; sync files=1363, longest=0.017 s, average=0.000 s Mar 13 00:59:08 dbserver postgres[5298]: [56-1] db=,user= LOG: checkpoint starting: time Mar 13 01:01:38 dbserver postgres[5298]: [57-1] db=,user= LOG: checkpoint complete: wrote 8035 buffers (0.8%); 0 transaction log file(s) added, 0 removed, 21 recycled; write=149.285 s, sync=0.146 s, total=149.433 s; sync files=1160, longest=0.003 s, average=0.000 s Mar 13 01:04:08 dbserver postgres[5298]: [58-1] db=,user= LOG: checkpoint starting: time Mar 13 01:06:37 dbserver postgres[5298]: [59-1] db=,user= LOG: checkpoint complete: wrote 2156 buffers (0.2%); 0 transaction log file(s) added, 0 removed, 9 recycled; write=149.402 s, sync=0.057 s, total=149.461 s; sync files=610, longest=0.000 s, average=0.000 s Mar 13 01:09:08 dbserver postgres[5298]: [60-1] db=,user= LOG: checkpoint starting: time I'm pretty certain that unmounting the file system and running fsck will regain the lost space, but will it stop there? Stopping postgresql briefly did not help, I tried that. That would have helped if the files where open, but they're not. It seems to postgresql did the right thing, and FreeBSD failed to unreference the files. The server has about 30 databases and ~127 concurrent connections (not all beeing active simultaneously, though), so it is fair to say it is pretty active, but nothing extreme. Hardware is HP DL360, using their HT Smart Array P410i. Any ideas how to debug this? Or shall I just reboot, fsck, hope the problem will go away, and when it does, forget about it? Thanks, Palle -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJRQFORAAoJEIhV+7FrxBJDzVUIAJHU011JDxLxj8/xg05Gwhgq XK3xB+0N0NSUQ50yhcRKLINz/j/XfeS0ZxlH+MstaPA9y0r1JUXMxkb/uTUvGBiy jutk3eVe0cati9cVZbJkRU5FxEgmQ0fg0GOMl3RQAErkh5achj+klWvN7PnwGjTs O3L9RgckKuxTJffk52GAS05qY/TKR6f08kdX3I2cFtqw3tyTyrXU0JPdk2snuPhv H40xV46zgtWMFDvZLt61MryQ7/JotVQwU78scUB+zxrf8KKM9V0mM7pk0pIbG4Qw NJBpZJ5gjbl4x+dkQrtZdL65yq88hACYwo9D+83Ct4ig8tgcQ7ViNHWxJqknK7Q= =3ZZs -----END PGP SIGNATURE----- _______________________________________________ freebsd-fs@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"