From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 08:15:07 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 982E7BF4; Tue, 14 Oct 2014 08:15:07 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id E4C7B8F9; Tue, 14 Oct 2014 08:15:06 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 1C80A20E70935; Tue, 14 Oct 2014 08:14:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 42F1120E70934; Tue, 14 Oct 2014 08:14:57 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "K. Macy" References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Tue, 14 Oct 2014 09:14:54 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: "freebsd-fs@FreeBSD.org" , mark , FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 08:15:07 -0000 ----- Original Message ----- From: "K. Macy" >>> Thank you both for analysis and effort! >>> >>> I can't rule out the possibility that my main system pool >>> on a SSD was low on space at some point in time, but the >>> three 4 GiB cloned pools (sys1boot and its brothers) were all >>> created as a zfs send / receive copies of the main / (root) >>> file system and I haven't noticed anything unusual during >>> syncing. This syncing was done manually (using zxfer) and >>> independently from the upgrade on the system - on a steady/quiet >>> system, when the source file system definitely had sufficient >>> free space. >>> >>> The source file system now shows 1.2 GiB of usage shown >>> by df: >>> shiny/ROOT 61758388 1271620 60486768 2% / >>> Seems unlikely that the 1.2 GiB has grown to 4 GiB space >>> on a cloned filesystem. >>> >>> Will try to import the main two pools after re-creating >>> a sane boot pool... >> >> >> Yer zfs list only shows around 2-3GB used too but zpool list >> shows the pool is out of space. Cant rule out an accounting >> issue though. >> > > What is using the extra space in the pool? Is there an unmounted > dataset or snapshot? Do you know how to easily tell? Unlike txg and > zio processing I don't have the luxury of having just read that part > of the codebase. Its not clear but I believe it could just be fragmention even though its ashift=9. I sent the last snapshot to another pool of the same size and it resulted in: NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - sys1copy 3.97G 3.47G 512M 72% - 87% 1.00x ONLINE - I believe FRAG is 0% as the feature wasn't enabled for the lifetime of the pool hence its simply not showing a valid value. zfs list -t all -r sys1boot NAME USED AVAIL REFER MOUNTPOINT sys1boot 1.76G 2.08G 11K /sys1boot sys1boot/ROOT 1.72G 2.08G 1.20G /sys1boot/ROOT sys1boot/ROOT@auto-2014-08-16_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-08-17_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-08-19_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-08-20_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-08-21_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-08-22_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-08-23_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-08-24_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-08-26_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-08-27_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-08-28_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-08-29_04.00 128K - 1.19G - sys1boot/ROOT@auto-2014-08-31_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-01_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-02_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-09-03_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-04_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-05_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-09-07_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-08_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-09_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-09-10_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-10_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-10_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-10_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-10_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-10_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-11_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-11_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-11_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-11_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-11_16.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-09-11_20.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-09-12_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-12_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-12_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-12_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-12_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-12_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_20.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-16_00.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-16_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-16_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-16_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-16_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-16_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-17_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-17_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-17_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-17_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-17_16.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-17_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_23.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_01.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_02.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_03.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_05.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_06.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_07.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_09.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_10.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_11.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_13.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_14.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_15.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_16.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-19_17.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-19_18.00 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_18.40 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_18.50 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.00 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.10 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.20 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.30 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.40 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.50 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.00 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.10 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.20 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.30 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.40 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.50 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.00 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.10 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.20 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.30 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.40 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.50 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_22.00 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_22.10 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_22.20 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_22.30 0 - 1.20G - Regards Steve