From owner-freebsd-fs@FreeBSD.ORG Tue Jan 14 16:31:27 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8B41B3E5 for ; Tue, 14 Jan 2014 16:31:27 +0000 (UTC) Received: from mail-ob0-x233.google.com (mail-ob0-x233.google.com [IPv6:2607:f8b0:4003:c01::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 585AC1736 for ; Tue, 14 Jan 2014 16:31:27 +0000 (UTC) Received: by mail-ob0-f179.google.com with SMTP id wp4so3957010obc.24 for ; Tue, 14 Jan 2014 08:31:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=LnwPDVBlUEa7YCiaax4ptTnSDGdndg3J0C+ZCEMekEw=; b=r5XlKF1EnyEnfMFdxnUKSQDMA3Mg2xfdbnFNCXCRlcXMsVhiIaDfrbWZWo/CazNXiW pCq/OjKVx485qwHLscbz1kyuYIaFDtJ2WDGNQq9MKzwiAi1NAL0pAx4HjIkQ11ec6H+l S5j+Gkdfmlh3faQfVPGE1ghlP2IekpLC9jvFREfWGRfyURek3SHOvB02h0u1EduzAi2U 2b+wppAVbu7Y25P1Wvd6j3ET6rVi7oJJCWWnulR9s0EG0GZl0lcQKIuONydU7B4QCJpG eD4n7io9eUKi6YjzPyFjiY6qA3MHAKMstN5E29oo2E1mmGeBPnWs32J32BySOcUPPs0i aq6A== MIME-Version: 1.0 X-Received: by 10.182.143.103 with SMTP id sd7mr1807784obb.70.1389717086556; Tue, 14 Jan 2014 08:31:26 -0800 (PST) Received: by 10.60.171.145 with HTTP; Tue, 14 Jan 2014 08:31:26 -0800 (PST) Date: Tue, 14 Jan 2014 17:31:26 +0100 Message-ID: Subject: zpool import taking weeks ... From: Ulysse 31 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 16:31:27 -0000 Hi all, Hope that someone may have some advices or tips to try solve my actual problem : A few weeks ago, our backup server, a freebsd 9.1 with zfs v28 on a raidz of 20TB (15 used) 64Gb RAM, had a serious hook up : a periodical sync (zfs send/recv) between a distant server and the backup server went wrong, which leaded to a "zfs recv" runing on the server and a "zfs rollback" on the same dataset. which lead to hang partially the machine, so machine was rebooted. At boot time, the machine took long time to import, and out of swap messages had come on the screen after 48 hours. We decided to reboot again and boot on a livecd (zfsguru), on which we add a usb drive as swap storage, and then when have launched the import of the pool with : zpool import -N -F on the first 48h hours the import took about all memory plus 779Mo of swap, since it is a zfsguru cd, i only have two terminals available (ALT+F1 and ALT+F2). The import was running on the second and on the first I could monitor the mem/cpu usage. On the first terminal, for some reason, when i launch top, it was quitting right after the first screen refresh. so I was firstly checking the machine with "top | head -n 24". After some days, I just write the following command on the first terminal "while true; do top | head -n 24; sleep 5; done". And it was working ... for 2 mins, after that, the terminal hung ... I can still check the import is running by using "CTRL+T" on the second terminal where zpool import is running, but the infos are not really helpfull unless telling that it still running. I get something like : load: 0.00 cmd: zpool 20299 [tx->tx_sync_done_cv)] 612062,13r 0.04u 0.29s 0% 48k sometimes, the "load: 0.00" goes arround 0.20 then comes back to 0.00 most of the time. it has been now running for more than a week, from what i read arround the only thing i can do is wait ... if someone as tips or ideas I would be really happy ^^' The storage is using one zpool with multiple dataset with dedup on (i know it EATS RAM :s ) On the live cd dmesg i could read at begining of the import something like : "Warning: can't open objset " followed by the dataset name that was crashed. I don't mind loosing this particular dataset, but some other are ... well, important. It is the first time I use zfsguru livecd, so at boot, i set the root password, in order to log into via ssh if needed, after loosing one of the terminal i realize that zfsguru is configured to use only "ssh" user to log into ssh, not root (stupid me ...). Thanks all for your kind help. Cheers. -- Ulysse31