From owner-freebsd-stable@freebsd.org Sat Aug 5 17:49:41 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E4BADC0B7E; Sat, 5 Aug 2017 17:49:41 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-qt0-x22c.google.com (mail-qt0-x22c.google.com [IPv6:2607:f8b0:400d:c0d::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BBA65818F2; Sat, 5 Aug 2017 17:49:40 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mail-qt0-x22c.google.com with SMTP id t37so23413542qtg.5; Sat, 05 Aug 2017 10:49:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=0R6LJ9fGoWE4JTkpSXDZaeltyd+X4nKwBC0ysqsqVK0=; b=Bi8oS1icEWVGaNit04OMI3AbGS2BPw3dWAjqAciodvdMn7E85C6+0w0cylbzM8aCMh dvyNFjhlUDHcjdJXvXnonI8H6N9SYS8F8/qJDDhz+nseXlitfjdK75uePGulQjgqJbJn e6JSlgX0XALN+CtHKEX3MlJy8sXhxgNWsPSH5diOsA+DppcNoBCeHJnecg8Y/AOZzPJu OGV3eiPvBVx4ERy6A86YgOwY4lxHPGdn/W2Lrn+/mdoEwQQhzfJuJa0eZUSD7BKJwR6y Lm+ChhR96o6UxmClIKY17Bik8oS+XFE0aC95QC84HQ3wz4RJoQz3dbW0zGrYtZ1TbKDG TPPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=0R6LJ9fGoWE4JTkpSXDZaeltyd+X4nKwBC0ysqsqVK0=; b=mkIxkBg4TKzREANqFmOkGSc19gNYxdSUnDfmz30CuUKmxALRaNRBlWB13ThqtC1GhZ RmsQFS6kp9ArWrKd38ZugTKO2eMcAhGx/nDWMI/Ymsc0H4H8J/HSJ+pTdlO/QPxVUaFC /QbnoCjoahMcYhF3H0ehZz9DJLhr/ssikwjChCGK4Dm+H172NT3yJgYf4+qe8+z592mj 7SwmV8vm+7ZqP0iecEXX3h5gWLPKdsJscy3UlgikFs5mQMAzGS1fLjFmiixQgvekJZYr voyQtiY/HIF9SRGqq26yMfJqa4fNuW5ckMsPRgG9RDjBby7yWRuQivGvWNSRMl9waxud 1TiA== X-Gm-Message-State: AHYfb5ikPTW5AbjkGWntSB/2iBrDTJS/WM/lZuJajy+qy6HZYYbk3+9o V4xj5bmSBfIL3YGMVPTqIa8Nn77WXA== X-Received: by 10.237.42.88 with SMTP id k24mr8782754qtf.58.1501955379916; Sat, 05 Aug 2017 10:49:39 -0700 (PDT) MIME-Version: 1.0 Received: by 10.140.32.136 with HTTP; Sat, 5 Aug 2017 10:49:39 -0700 (PDT) Received: by 10.140.32.136 with HTTP; Sat, 5 Aug 2017 10:49:39 -0700 (PDT) In-Reply-To: References: <1bd10b1e-0583-6f44-297e-3147f6daddc5@norma.perm.ru> From: Freddie Cash Date: Sat, 5 Aug 2017 10:49:39 -0700 Message-ID: Subject: Re: a strange and terrible saga of the cursed iSCSI ZFS SAN To: "Eugene M. Zheganin" Cc: FreeBSD Stable , FreeBSD Filesystems Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Aug 2017 17:49:41 -0000 On Aug 5, 2017 10:09 AM, "Eugene M. Zheganin" wrote: And I want to also ask - what happens when the system's memory isn't enough for deduplication - does it crash, or does the problem of mounting the pool appear, like some articles mention ? Can't really help with the other issues, but can speak to this one. "Insufficient" RAM for dedupe won't be an issue during normal operation, reading and writing to the pool. Performance will slow down over time as the DDT increases in size taking up ARC/L2ARC space, possibly even spilling over into raw disk reads. Where the issues crop up is during snapshot deletion and filesystem destruction. Those operations can run the system out of usable kmem and lock things up. (Deletions cause a lot of churn in the DDT.) On reboot, the pool import process will continue the deletion process, which will run the system out of kmen locking it up. Rinse and repeat for a week until the deletion process completes. L2ARC helps, but not nearly as much as adding more RAM! Device replacement will also be an issue as the resilver process will be extremely slow (at least, it is for us with a very full, very fragmented pool; equally full pools without d we dedupe resilver very quickly). And don't try to replace two drives in separate vdevs unless you want to increase the resilver time by a huge factor. This is from experience with a 24-drive system with only 48 GB of RAM, and a 90-disk system with 128 GB of RAM. Both with dedupe enabled. Both running at about 90% full, very fragmented as they create and delete snapshots of all filesystems on a daily basis. They've been running file about 5 years now, possibly longer. Cheers, Freddie