From owner-freebsd-stable@freebsd.org Sun Aug 13 10:09:31 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 85A04DD5134; Sun, 13 Aug 2017 10:09:31 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id EFD45675B5; Sun, 13 Aug 2017 10:09:30 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from [IPv6:2a02:2698:26:ba6d:b8c2:5ea7:b69b:1849] (dynamic-2a02-2698-26-0-0.perm.ertelecom.ru [IPv6:2a02:2698:26:ba6d:b8c2:5ea7:b69b:1849] (may be forged)) (authenticated bits=0) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPSA id v7DA9Psd036500 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Sun, 13 Aug 2017 15:09:25 +0500 (YEKT) (envelope-from emz@norma.perm.ru) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=norma.perm.ru; s=key; t=1502618965; bh=VySA4zhZ8cIn8iDCAv7T+QPVKuUplKPP7uddFkMqSME=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=GVQIjTlavkVVDMRs5ECS4zaIyDyYO1H23k0y9ZxhMRh8olUlZ14HQ8cIGsTJLDeAi fx5Cycf9RrGGL2hMH5qps1GJwM2nMlvcbEmnF7ZZLEMFWsAoNnTHBpoMm280d9BPj9 N/y1NFgmNDgMKt/Ky/WsEQsOzrk9TiMGW+oipalo= Subject: Re: zfs listing and CPU To: freebsd-fs@freebsd.org Cc: freebsd-stable References: From: "Eugene M. Zheganin" Message-ID: <399c4309-c7a6-a7e9-299f-9675224c090d@norma.perm.ru> Date: Sun, 13 Aug 2017 15:09:23 +0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Aug 2017 10:09:31 -0000 Hi, On 12.08.2017 20:50, Paul Kraus wrote: >> On Aug 11, 2017, at 2:28 AM, Eugene M. Zheganin wrote: >> >> Why does the zfs listing eat so much of the CPU ? >> 47114 root 1 20 0 40432K 3840K db->db 4 0:05 26.84% zfs >> 47099 root 1 20 0 40432K 3840K zio->i 17 0:05 26.83% zfs >> 47106 root 1 20 0 40432K 3840K db->db 21 0:05 26.81% zfs >> 47150 root 1 20 0 40432K 3428K db->db 13 0:03 26.31% zfs >> 47141 root 1 20 0 40432K 3428K zio->i 28 0:03 26.31% zfs >> 47135 root 1 20 0 40432K 3312K g_wait 9 0:03 25.51% zfs >> This is from winter 2017 11-STABLE (r310734), one of the 'zfs'es is cloning, and all the others are 'zfs list -t all'. I have like 25 gigs of free RAM, do I have any chance of speeding this up using may be some caching or some sysctl tuning ? We are using a simple ZFS web API that may issue concurrent or sequential listing requests, so as you can see they sometimes do stack. > How many snapshots do you have ? I have only seen this behavior with LOTS (not hundreds, but thousands) of snapshots. [root@san1:~]# zfs list -t snapshot | wc -l 88 > What does your `iostat -x 1` look like ? I expect that you are probably saturating your drives with random I/O. Well, it's really long, and the disks are busy with random i/o indeed, but byst only for 20-30%. As about iostat - it's really long, because I have hundreds (not thousands) of zvols, and they do show up in iostat -x. But nothing unusual besides that. Thanks. Eugene.