From owner-freebsd-questions@FreeBSD.ORG Fri Mar 22 12:59:51 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id ECD7172 for ; Fri, 22 Mar 2013 12:59:51 +0000 (UTC) (envelope-from emorrasg@yahoo.es) Received: from nm20.bullet.mail.ird.yahoo.com (nm20.bullet.mail.ird.yahoo.com [77.238.189.77]) by mx1.freebsd.org (Postfix) with SMTP id 4982C777 for ; Fri, 22 Mar 2013 12:59:50 +0000 (UTC) Received: from [77.238.189.231] by nm20.bullet.mail.ird.yahoo.com with NNFMP; 22 Mar 2013 12:59:44 -0000 Received: from [46.228.39.123] by tm12.bullet.mail.ird.yahoo.com with NNFMP; 22 Mar 2013 12:59:44 -0000 Received: from [127.0.0.1] by smtp160.mail.ir2.yahoo.com with NNFMP; 22 Mar 2013 12:59:44 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.es; s=s1024; t=1363957184; bh=GJjPIXyrkMpkvEcYKDAAi4rAphzS7wlNniWnkR3FVrc=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:Date:From:To:Subject:Message-Id:In-Reply-To:References:X-Mailer:Mime-Version:Content-Type:Content-Transfer-Encoding; b=p0MhpvARkPDsVn3TAjSdE0zujXTYJ7KqkKyiMvRAM4iWveaQunxewmywb9gDxlIke0fJ5m6j0wRGpIVB9JagqQG4BwXJWCVsrcWysqk0pStx+UMjCK6rXU5F0Yc+JHn+vte+BwPNRCNafyVer29UvS2nItxmGQRyhAAOk2KAiQo= X-Yahoo-Newman-Id: 385588.39509.bm@smtp160.mail.ir2.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: Zpmihe4VM1mRS__qdvk3P8SOjEgAMiSt7G4L6n4xjMYaMiE uLiVyk5wPxwHgheEE4oyOHkp1yuKgyDpoFA7NJwM5hHrn3XTm4LdLMg.c.lM F70X2VXpUD9AurF0TS0A5Ryqu0IVU_4Bo0E13c9y7AHhy6UnDWo_vvZJFkOG X51wfOgJ_nKdyTTq0YkOcJKt0FiCbjhqk_rzUw5ORdav31vBmLsEa6wKsPh8 xMlV5MoMduIWWB7Te0lBYd2r3L7ZIi5qb.pTKROSLD.8qDpT1D8lSWlZZUVn G_yFER3AOZey3VTDUxU2Sbtmz_gQgfMtv2naLDMkURSq7y_jrRiml8.c5wdn 5faMkpXWUw6crDQlWPFJA7Ile2It47KYXzlq9yWoiVThTQHfCPP82c7hZklI 3d2znvmbRf1Z_2B5yfQ38q3RkCr.B3a66DfOLrOb9aK1ZKIbgYJlRvBTE4WT FgVa3tuOzULm2sR2f53SYkEyf4fO344NXNoSYBYyYp.PcYTrGgkAekNhn2MW cDadkxEE_bQSo X-Yahoo-SMTP: mX392iiswBAeJNdO_s.EW62LZDJR X-Rocket-Received: from camibar.emorras.eu (emorrasg@85.219.46.40 with plain) by smtp160.mail.ir2.yahoo.com with SMTP; 22 Mar 2013 12:59:44 +0000 UTC Date: Fri, 22 Mar 2013 13:59:45 +0100 From: Eduardo Morras To: freebsd-questions@freebsd.org Subject: Re: "Leaking" disk space Message-Id: <20130322135945.3bf78de167d56d8019072626@yahoo.es> In-Reply-To: References: <20130320170241.278d4fb515536cb6669ae0bf@yahoo.es> X-Mailer: Sylpheed 3.3.0 (GTK+ 2.24.17; amd64-portbld-freebsd9.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2013 12:59:52 -0000 On Wed, 20 Mar 2013 16:55:34 +0000 Dan Thomas wrote: > > a) Where do you have the wal files? > > pg_xlog is symlinked to /usr/local/pglog/pg_xlog (ie, out of the > partition mounted as /usr/local/pgsql which is exhibiting this > behaviour). > As Matthew Seaman says in other answer, this is the problem. Check http://lists.freebsd.org/pipermail/freebsd-fs/2013-March/016702.html and next thread messages. It seems that writing file follow the symlink but makes a shadow/ghost file entry in original directory/disk. I see that you don't have trim enabled on the postgres fs, tunefs -p /usr/local/pgsql/ shows option t disabled. Is trim enabled on the fs where the symlink points? (Show the output of tunefs -p /dev/_don't_know_the_dev_entry_name) > b) Are you sure that unused/old wal files are erased? > > As above, but yes they seem to be being deleted properly > > c) Do you have any postgres log level activated (like the ones used > for long queries)? > > Yes we have slow query logging enabled. pg_log is symlinked out of > that partition to /usr/local/pglog/pg_log as well. > > d) Does your queries have GROUP BY on very big data sets? Those create > big temporal data files. > > Yes we do a lot of that! However there are definitely no unlinked > files, and the problem doesn't go away when pg is shut down. However a > reboot does fix it. Those questions were only to check and be sure is not a "normal" temp files problem. What does dmesg show about filesystem check? Does it mark dirty filesystem? # WARNING!! Make a backup first!!! If you stop postgres, and shoot #fsck_ffs -E /dev/mfid1s1d , does the problem solve? # END WARNING!! Please post the output of the fsck_ffs. If the fsck_ffs doesn't solve the problem, check if there exist a lost+found directory on /usr/local/pgsql/ and it's content. > > e) With question a) and b), do you use streaming replication? > > Yes we do. This problem is not present on the warm standby servers > that are being streamed to. We have failed over to the warm standbys > previously (we're currently doing this regularly to work around the > problem without too much downtime). Once we switch the warm standby to > primary, it begins leaking space. It may store old&all wal files, but it seems a bug at filesystem level. Trim support in ufs was added to 9.0 and backported to 8 and may be a candidate to watch. --- --- Eduardo Morras