From owner-freebsd-fs@FreeBSD.ORG Tue Jan 14 18:07:30 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1D91B6A for ; Tue, 14 Jan 2014 18:07:30 +0000 (UTC) Received: from mail-ob0-x234.google.com (mail-ob0-x234.google.com [IPv6:2607:f8b0:4003:c01::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D5CD91F95 for ; Tue, 14 Jan 2014 18:07:29 +0000 (UTC) Received: by mail-ob0-f180.google.com with SMTP id wm4so3266292obc.39 for ; Tue, 14 Jan 2014 10:07:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=j+ST2vr4RN7RxM4lbM1YVhb9gGcZbuERntMkxkTJg1Y=; b=eOrEdB4ppImN5aeTnMuqzINC1DMcsQcW5LOVK5EK3odb4227ChEGJWVolAgFvSpT9A iG6b9Yp1ppCZbOLdMbdMmn01+tHDPWkSigy/sUTxSa1GmXCXw7rfO6fqib75JKW7j8jv 3/DZHM+d2gxKLjpF6XaHlT2oPiDyElWy/bBHL7rlb4UfyMQRvELtpJsmHdjzJagyvJfj Ob5Z67ahJIYsp0vlo3vrrdDH+cstxIMXFJEsA3BHrglzuPBdq47+etyB0FAsvG/1cWf4 ykiO4IPYcLYHavA2naSeCH/cP+2KcZaVbfqwCIsg+4hryrl1U2EJ7L7Q7SqgoUlidgU8 DJ7Q== MIME-Version: 1.0 X-Received: by 10.182.29.33 with SMTP id g1mr2206002obh.59.1389722849172; Tue, 14 Jan 2014 10:07:29 -0800 (PST) Received: by 10.76.132.9 with HTTP; Tue, 14 Jan 2014 10:07:29 -0800 (PST) In-Reply-To: References: Date: Tue, 14 Jan 2014 10:07:29 -0800 Message-ID: Subject: Re: zpool import taking weeks ... From: Freddie Cash To: Thomas Hoffmann Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 18:07:30 -0000 On Tue, Jan 14, 2014 at 8:58 AM, Thomas Hoffmann wrote: > On Tue, Jan 14, 2014 at 11:52 AM, Freddie Cash wrote: > >> Dedupe enabled on this server? >> >> >> -- >> Freddie Cash >> fjwcash@gmail.com > > > Buried in his rather lengthy text was: > > "The storage is using one zpool with multiple dataset with dedup on (i > know it EATS RAM :s )" > > So the answer is yes. > =E2=80=8BWhoops, missed that. In that case, it's "normal". The pool was interrupted in the middle of a send and a rollback. The import will now do the necessary cleanup to return the pool to the correct state, during the import process. To do so, it has to remove all the blocks from the send, remove all the blocks from the rollback, and update the DDT entries for all those blocks. It will load the DDT into ARC as needed, and eventually blow out all the RAM in the system and lockup. Reboot, re-do the import, it will continue from where it left off, load the DDT into ARC, and eventually blow out all the RAM in the system and lockup. Rinse, repeat, reboot, blah blah blah. After several iterations, it will eventually finish the cleanup, and the pool will import. This is a known limitation of dedupe and interrupted operations. Similar to the "destroy lots of filesystems" and then lockup issue. Basically, dedupe makes a lot of things very slow in recovery. I once had to spend 2 weeks doing the reboot/import/wait/lockup dance to get through a similar situation.=E2=80=8B =E2=80=8B-=E2=80=8B - Freddie Cash fjwcash@gmail.com