From owner-freebsd-fs@FreeBSD.ORG Tue Nov 18 19:00:39 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 57874987 for ; Tue, 18 Nov 2014 19:00:39 +0000 (UTC) Received: from anubis.delphij.net (anubis.delphij.net [IPv6:2001:470:1:117::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anubis.delphij.net", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 37423335 for ; Tue, 18 Nov 2014 19:00:39 +0000 (UTC) Received: from zeta.ixsystems.com (unknown [12.229.62.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id 7631820357; Tue, 18 Nov 2014 11:00:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphij.net; s=anubis; t=1416337238; x=1416351638; bh=bVXijf5ZAK1ol69Qo+VV4btz0myLoIpPxnWR8AVK3jA=; h=Date:From:Reply-To:To:Subject:References:In-Reply-To; b=Q28FNDv0j8uE52KNimlsZHTJulBqbySjHkspVsQa5NxmU7g1xs8v/IyUVmr4Fl5bF YID4/ur+RU9JXYwVs6AdixywjdtyBeWJbv4dJ5MAQDLMaZAh4xAcWKondu9xLn3D0g jm01kvSTyAm2gcAS8GhvTLUN/Lm3HqT82xG4MTDM= Message-ID: <546B9754.4060906@delphij.net> Date: Tue, 18 Nov 2014 11:00:36 -0800 From: Xin Li Reply-To: d@delphij.net Organization: The FreeBSD Project MIME-Version: 1.0 To: Adam Nowacki , freebsd-fs@freebsd.org Subject: Re: No more free space after upgrading to 10.1 and zpool upgrade References: <20141116080128.GA20042@exhan.dylanleigh.net> <20141118054443.GA40514@core.summit> <546B8203.5040607@platinum.linux.pl> In-Reply-To: <546B8203.5040607@platinum.linux.pl> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Nov 2014 19:00:39 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 11/18/14 09:29, Adam Nowacki wrote: > On 2014-11-18 06:44, Emil Mikulic wrote: >> On Sun, Nov 16, 2014 at 04:10:28PM +0100, Olivier Cochard-Labb? >> wrote: >>> On Sun, Nov 16, 2014 at 9:01 AM, Dylan Leigh >> dylanleigh.net> wrote: >>> >>>> >>>> Could you provide some other details about the pool >>>> structure/config, including the output of "zpool status"? >>>> >>>> >>> It's a raidz1 pool build with 5 SATA 2TB drives, and there are >>> 5 zvolumes without advanced features (no compression, no >>> snapshot, no de-dup, etc...). Because it's a raidz1 pool, I >>> know that FREE space reported by a "zpool list" include >>> redundancy overhead and is bigger than AVAIL space reported by >>> a "zfs list". >>> >>> I've moved about 100GB (on hundred GigaByte) of files and after >>> this step there were only 2GB (two GigaByte) of Free space >>> only: How is it possible ? >> >> I had the same problem. Very old pool: >> >> History for 'jupiter': 2010-01-20.20:46:00 zpool create jupiter >> raidz /dev/ad10 /dev/ad12 /dev/ad14 >> >> I upgraded FreeBSD 8.3 to 9.0, which I think went fine, but when >> I upgraded to 10.1, I had 0B AVAIL according to "zfs list" and >> df(1), even though there was free space according to "zpool >> list" >> >> # zpool list -p jupiter NAME SIZE ALLOC FREE FRAG >> EXPANDSZ CAP DEDUP HEALTH ALTROOT jupiter 4466765987840 >> 4330587288576 136178699264 30% - 96 1.00x >> ONLINE - >> >> # zfs list -p jupiter NAME USED >> AVAIL REFER MOUNTPOINT jupiter >> 2884237136220 0 46376 /jupiter >> >> Deleting files, snapshots, and child filesystems didn't help, >> AVAIL stayed at zero bytes... until I deleted enough: >> >> NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP >> HEALTH ALTROOT jupiter 4466765987840 4320649953280 >> 146116034560 30% - 96 1.00x ONLINE - >> >> NAME USED AVAIL REFER MOUNTPOINT jupiter >> 2877618732010 4350460950 46376 /jupiter >> >> Apparently, the above happened somewhere between 96.0% and 96.9% >> used. >> >> Any ideas what happened here? It's almost like 100+GB of free >> space is somehow reserved by the system (and I don't mean "zfs >> set reservation", those are all "none") > > This commit is to blame: > http://svnweb.freebsd.org/base?view=revision&revision=268455 > > 3.125% of disk space is reserved. Note that the reserved space is so that one can always delete files, etc. to get the pool back to a usable state. I've added a new tunable/sysctl in r274674, but note that tuning is not recommended: by using too much space the pool would become read-only permanently and one will have to dump data and recreate the pool. Cheers, - -- Xin LI https://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0 iQIcBAEBCgAGBQJUa5dUAAoJEJW2GBstM+nsyiUP/0KuNeTrocsQPrZ8YnsDGHyd QXFDdZ9B9RTD3GygUwLZIAX0st1pCy28sTfF1Ph54rfq2DkIJaJwUzlOeOTceNup hppXcYah5kX4YnnVek73+W6JZWxUV9MbSpOYn6bhyItyRtbv2dDpytJa6D6uKggq rkpoU1tyIQLZJPZ5m9pL7h3XvxZpHRJLSqD7JlYr9aXzqFDoXxq5vvD6tZkpkx7f sFhcSDEPb7wKbPA+UbQ+YvycMJyEqKDgdOWvqC1puSGPqRzN8WZcM8Qw/Rs9wpsl QiCK1OJQwO1RBIJUJq9SVyCE08lDDvMrG+3kEemCac8p066/15Vpxoqu818mskfS 0MA6CUQMAepjHoyntd6vokWGu6O9Lx92pRa11/RfQ5xql29hmOz3dXBtcIX7ApJQ Wxcvip+2yLaeDMw0bc0M1nxUpuQUPbf4Rob0li8T0S7g26Dll84FUBAWV/1F5hh2 +7i3Tt385opZgautZDEiGk0MZFLb+2EdqXxOWi479vJ3rS1Q2oGsgiPH8PQ4Lrog QmQ9KplmxIOyrIVdMUrq+ywHWY6nMA5buKsXTEaLKqghCM2mJK8n9LpwZoAkfG9y Ueko1GssfYhuJ++VQrOAFYcf7voxSrlj4XPdzofS+lrDQ7FluF8b+iFWjz/IEQug b9PAF4KyBdPfdWMqQgW9 =e2r4 -----END PGP SIGNATURE-----