From owner-freebsd-current@FreeBSD.ORG Fri Jan 3 18:17:29 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BDF4D25C for ; Fri, 3 Jan 2014 18:17:29 +0000 (UTC) Received: from email2.allantgroup.com (email2.emsphone.com [199.67.51.116]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 87F5E153A for ; Fri, 3 Jan 2014 18:17:28 +0000 (UTC) Received: from dan.emsphone.com (dan.emsphone.com [172.17.17.101]) by email2.allantgroup.com (8.14.7/8.14.7) with ESMTP id s03IGMh3097269 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 3 Jan 2014 12:16:22 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1]) by dan.emsphone.com (8.14.7/8.14.6) with ESMTP id s03IGMCo006991 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 3 Jan 2014 12:16:22 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.7/8.14.7/Submit) id s03IGMWV006990; Fri, 3 Jan 2014 12:16:22 -0600 (CST) (envelope-from dan) Date: Fri, 3 Jan 2014 12:16:22 -0600 From: Dan Nelson To: "O. Hartmann" Subject: Re: ZFS command can block the whole ZFS subsystem! Message-ID: <20140103181622.GA61275@dan.emsphone.com> References: <20140103130021.30569db4@thor.walstatt.dyndns.org> <20140103171457.0fbf0cd4@telesto> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140103171457.0fbf0cd4@telesto> X-OS: FreeBSD 9.1-STABLE User-Agent: Mutt/1.5.22 (2013-10-16) X-Virus-Scanned: clamav-milter 0.98 at email2.allantgroup.com X-Virus-Status: Clean X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (email2.allantgroup.com [172.17.19.78]); Fri, 03 Jan 2014 12:16:22 -0600 (CST) X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, RP_MATCHES_RCVD autolearn=ham version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on email2.allantgroup.com X-Scanned-By: MIMEDefang 2.73 Cc: FreeBSD CURRENT , Steven Hartland X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jan 2014 18:17:29 -0000 In the last episode (Jan 03), O. Hartmann said: > On Fri, 3 Jan 2014 14:38:03 -0000 "Steven Hartland" wrote: > > From: "O. Hartmann" > > > For some security reasons, I dumped via "dd" a large file onto a 3TB > > > disk. The systems is 11.0-CURRENT #1 r259667: Fri Dec 20 22:43:56 CET > > > 2013 amd64. Filesystem in question is a single ZFS pool. > > > > > > Issuing the command > > > > > > "rm dumpfile.txt" > > > > > > and then hitting Ctrl-Z to bring the rm command into background via > > > fg" (I use FreeBSD's csh in that console) locks up the entire command > > > and even worse - it seems to wind up the pool in question for being > > > exported! > > > > You can check that gstat -d > > command report 100% acticity on the drive. I exported the pool in > question in single user mode and now try to import it back while in > miltiuser mode. Did you happen to have enabled deduplication on the filesystem in question? That's the only thing I can think of that would make file deletions run slow. I have deleted files up to 10GB on regular filesystems with no noticable delay at the commandline. If you have deduplication enabled, however, each block's hash has to be looked up in the dedupe table, and if you don't have enough RAM for it to be loaded completely into memory, that will be very very slow :) There are varying recommendations on how much RAM you need for a given pool size, since the DDT has to hold an entry for each block written, and blocksize depends on whether you wrote your files sequentially (128K blocks) or randomly (8k or smaller). Each DDT entry takes 320 bytes of RAM, so a full 3TB ZFS pool would need at minimum 320*(3TB/128K) ~= 7GB of RAM to hold the DDT, and much more than that if your averge blocksize is less than 128K. So, if your system has less than 8GB of RAM in it, there's no way the DDT will be able to stay in memory, so you're probably going to have to do at least one disk seek (probably more, since you're writing to the DDT as well) per block in the file you're deleting. You should probably have 16GB or more RAM, and use an SSD as a L2ARC device as well. -- Dan Nelson dnelson@allantgroup.com