From owner-freebsd-fs@FreeBSD.ORG Tue Nov 18 05:44:51 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9F84243E for ; Tue, 18 Nov 2014 05:44:51 +0000 (UTC) Received: from mail-pa0-x22b.google.com (mail-pa0-x22b.google.com [IPv6:2607:f8b0:400e:c03::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 69951FD1 for ; Tue, 18 Nov 2014 05:44:51 +0000 (UTC) Received: by mail-pa0-f43.google.com with SMTP id kx10so3584888pab.16 for ; Mon, 17 Nov 2014 21:44:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=02mHDdS7+pGKCuTxZ8qwhWPw+CNkYUagmz3IBKOFkdc=; b=nlBsSGZaxA0HAsOEnPAn3zVVQhyv60GsAh/jE9Uenn4ZJANh/7aHVFvJ6sUad8ejv6 KCxcX+HBXKeJyWJZwIjTZo1FB/wFZRydh1iVNESNNgP+iULhElh+8hm12bmdcrO2XXrv COpNJy6oVUa7mrpsB96oePtwcSjTLmphAFDQMraNnEoxr1OoQ4IFxA5o2ElohxqgDtT5 IA0+9CPZqd4FO3/4HK0miCRNgTyNtDSyUMoayESxRF0NzsQn9ePD0vo5zeEl+8N7TMPY nulXBl/xKYd+Q7o+iVEajRJXWoT/iS0+sIhp4t3kBxbugveAHXcFVyB/1myhUYs8DWOr 76Eg== X-Received: by 10.70.44.208 with SMTP id g16mr9335441pdm.130.1416289490993; Mon, 17 Nov 2014 21:44:50 -0800 (PST) Received: from core.summit (2001-44b8-31c7-e401-4a5b-39ff-fe76-f665.static.ipv6.internode.on.net. [2001:44b8:31c7:e401:4a5b:39ff:fe76:f665]) by mx.google.com with ESMTPSA id pc10sm36679038pbb.21.2014.11.17.21.44.48 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Nov 2014 21:44:50 -0800 (PST) Date: Tue, 18 Nov 2014 16:44:43 +1100 From: Emil Mikulic To: freebsd-fs@freebsd.org Subject: Re: No more free space after upgrading to 10.1 and zpool upgrade Message-ID: <20141118054443.GA40514@core.summit> References: <20141116080128.GA20042@exhan.dylanleigh.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: fbsd@dylanleigh.net X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Nov 2014 05:44:51 -0000 On Sun, Nov 16, 2014 at 04:10:28PM +0100, Olivier Cochard-Labb? wrote: > On Sun, Nov 16, 2014 at 9:01 AM, Dylan Leigh wrote: > > > > > Could you provide some other details about the pool structure/config, > > including the output of "zpool status"? > > > > > It's a raidz1 pool build with 5 SATA 2TB drives, and there are 5 zvolumes > without advanced features (no compression, no snapshot, no de-dup, etc...). > Because it's a raidz1 pool, I know that FREE space reported by a "zpool > list" include redundancy overhead and is bigger than AVAIL space reported > by a "zfs list". > > I've moved about 100GB (on hundred GigaByte) of files and after this step > there were only 2GB (two GigaByte) of Free space only: How is it possible ? I had the same problem. Very old pool: History for 'jupiter': 2010-01-20.20:46:00 zpool create jupiter raidz /dev/ad10 /dev/ad12 /dev/ad14 I upgraded FreeBSD 8.3 to 9.0, which I think went fine, but when I upgraded to 10.1, I had 0B AVAIL according to "zfs list" and df(1), even though there was free space according to "zpool list" # zpool list -p jupiter NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT jupiter 4466765987840 4330587288576 136178699264 30% - 96 1.00x ONLINE - # zfs list -p jupiter NAME USED AVAIL REFER MOUNTPOINT jupiter 2884237136220 0 46376 /jupiter Deleting files, snapshots, and child filesystems didn't help, AVAIL stayed at zero bytes... until I deleted enough: NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT jupiter 4466765987840 4320649953280 146116034560 30% - 96 1.00x ONLINE - NAME USED AVAIL REFER MOUNTPOINT jupiter 2877618732010 4350460950 46376 /jupiter Apparently, the above happened somewhere between 96.0% and 96.9% used. Any ideas what happened here? It's almost like 100+GB of free space is somehow reserved by the system (and I don't mean "zfs set reservation", those are all "none")