From owner-freebsd-fs@freebsd.org Sun Jun 19 21:08:58 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DABD3A77540 for ; Sun, 19 Jun 2016 21:08:58 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: from mail-wm0-x22e.google.com (mail-wm0-x22e.google.com [IPv6:2a00:1450:400c:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7A085131C for ; Sun, 19 Jun 2016 21:08:58 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: by mail-wm0-x22e.google.com with SMTP id r201so36771638wme.1 for ; Sun, 19 Jun 2016 14:08:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=+Ut9zSQCpJGxEkobF3odWKwUrfZDLb0gSBldIxOBmjg=; b=A/97cRQlFyNaJZrkTaoByb6fRI9R785rl/B1E+JraTAgx2gTRwpLTpLUVuKs7Fdhco yj15Dp++f8j/4cxepbWsZUjptzrEan1m6OXlHUuEezHQ3czgXkBwrqvZPPru/5rhbszo saeJYNbTkC1AjpOPuzpESKMdv1/KgmFPXlf7BM9aUR8MIwQF8sLUbJ8nRaYOjIpsrj3D JTGsabBn82yFZ/HIXkVmgXe4TyHU8aqvxFdIapyEgu4kat33WHfPHTthQjT+tkVk1I+V Ch80bG1BUzreOYXJjz80AtNmS76aoQjAp9F+ZAFjHghLHelk3AgyPHhX1POFEYPdzJHq 5DFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=+Ut9zSQCpJGxEkobF3odWKwUrfZDLb0gSBldIxOBmjg=; b=GJqAEWAAaTaCQI1Pz6V/vXYBtYPdpWJ19tgH93zCgofpDXGp5Q05SVozomf6ej7UOA 0GRhum9To2+MMAOly0hFMyhzpqtjQz2VKclblgAu2xdz/mfHoju0rFG6gqJQ/fRMKgSQ 32HT1H+dGVfjOmLv/j/INSCKRs/2vfwW5v5kUXivqDtq9XWG86xV4gia0ZX4lNEx2z3p yHuPEv8VTPZYizjHyQiRyqivWfXEJVT+tTZzw4b8K1m4mhXtGqoJfuSmVX2HigVZvn9d 6oosBWX5q/gi9GBh1tfpIFy8lXuI13mmt7Rx8+ypefatp2YizRkM+jCTNYoQ6q4olhhh LEmA== X-Gm-Message-State: ALyK8tLZbaJ/VOQUKdo91m+a7iOxgkK63U0GVbEkcUAxLGZRPjY1AGwbfekFYWASBao+gA== X-Received: by 10.28.41.134 with SMTP id p128mr2876764wmp.20.1466370536329; Sun, 19 Jun 2016 14:08:56 -0700 (PDT) Received: from [192.168.20.30] (optiplexnetworks.plus.com. [212.159.80.17]) by smtp.googlemail.com with ESMTPSA id ej9sm26377632wjd.7.2016.06.19.14.08.55 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 19 Jun 2016 14:08:55 -0700 (PDT) Subject: Re: High CPU Interrupt using ZFS To: Paul Kraus References: <57cfcda4-6ff7-0c2e-4f58-ad09ce7cab28@gmail.com> <2F83F199-80C1-4B98-A18D-C5343EE4F783@kraus-haus.org> Cc: FreeBSD Filesystems From: Kaya Saman Message-ID: <71559f37-9439-f7b4-9d1f-091875c098ab@gmail.com> Date: Sun, 19 Jun 2016 22:08:54 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: <2F83F199-80C1-4B98-A18D-C5343EE4F783@kraus-haus.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Jun 2016 21:08:58 -0000 On 06/19/2016 09:45 PM, Paul Kraus wrote: >> On Jun 19, 2016, at 3:38 PM, Kaya Saman wrote: > > >> # zpool list >> NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT >> ZPOOL_2 27.2T 26.3T 884G - 41% 96% 1.00x ONLINE - >> ZPOOL_3 298G 248G 50.2G - 34% 83% 1.00x ONLINE - >> ZPOOL_4 1.81T 1.75T 66.4G - 25% 96% 1.00x ONLINE - >> ZPOOL_5 186G 171G 14.9G - 62% 92% 1.00x ONLINE - >> workspaces 119G 77.7G 41.3G - 56% 65% 1.00x ONLINE - >> zroot 111G 88.9G 22.1G - 70% 80% 1.00x ONLINE - > Are you aware that ZFS performance drops substantially once a pool exceeds a certain % full, the threshold for which varies with pool type and work load. It is generally considered a bad idea to run pools more than 80% full with any configuration or workload. ZFS is designed first and foremost for data integrity, not performance and running pools too full causes _huge_ write performance penalties. Does your system hang correspond to a write request to any of the pools that are more than 80% full ? The pool that is at 92% capacity and 62% fragmented is especially at risk for this behavior. > > The underlying reason for this behavior is that as a pool get more and more full it takes more and more time to find an appropriate available slab to write new data to, since _all_ writes are treated as new data (that is the whole point of the Copy on Write design) _any_ write to a close to full pool incurs the huge performance penalty. > > This means that if you write the data and _never_ modify it and that you can stand the write penalty as you add data to the mostly full zpools, then you may be able to use ZFS like this, otherwise just don’t. > > On my virtual hosts, running FreeBSD 10.x and VirtualBox, a pool more than 80% full will make the VMs unacceptably unresponsive, I strive to keep the pools at less than 60% capacity. Disk storage is (relatively) cheap these days. > Thanks for this! Yep I was aware that things would "slow down" but not render the system totally unresponsive. This maybe the reason.... as once the CPU interrupt goes to unacceptably high levels even the NICs start flapping as they're all bound to an LACP LAGG interface. I will probably need to get a few JBOD chassis for expansion as though my chassis can take 22 disks they are all full :-( Regards, Kaya