From owner-freebsd-stable@freebsd.org Sun Aug 13 11:13:24 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 35D0ADD9174; Sun, 13 Aug 2017 11:13:24 +0000 (UTC) (envelope-from tenzin.lhakhang@gmail.com) Received: from mail-wm0-x22d.google.com (mail-wm0-x22d.google.com [IPv6:2a00:1450:400c:c09::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DB08669DCD; Sun, 13 Aug 2017 11:13:23 +0000 (UTC) (envelope-from tenzin.lhakhang@gmail.com) Received: by mail-wm0-x22d.google.com with SMTP id k20so11309051wmg.0; Sun, 13 Aug 2017 04:13:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=sb/SJRa2EAKbfvjv+9OyiiuIyzHrgAXqAYd9b/GNhQ4=; b=X+u+ry9ujs4blAdvjWWf63TW9+wYwKYqo2afC1m5OuuFAZQkqwgnTmAcyJjlT5Kj9a gxXzBi1agZjPwUgi0C98cLl6SkmYHN95+VNFTBzwwkMlZ0hX0LzpmzDgJoEtExMeSN6n 0cSI9X8Y/gryhpPi75HVXdTLCe+QwrgW9YqIJbgtT1JIf+STAJNLj6dxiMFHmZURQ9cq NYYksF4BBzXSNh3zyMKLU6plJ2xmutkG2lgrDzA2fMxeOh+jAlWnsZY/Bc25P9QBPBnt Xo0LgWDYlEq5aGYj1pQFFvFE2OK0IV9evLyts2mxgyEYSGEk0flPMnD6JFwAiivFTuE8 shjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=sb/SJRa2EAKbfvjv+9OyiiuIyzHrgAXqAYd9b/GNhQ4=; b=gv+PCBSZWHk0+eAkdvA6WxBphBi93D0j05smp48er/Q1uCh5FV8eJ2v2arFtsDMnIr Z81Fcnsb0v+eSwzNBwDjMSUFKgrp5mHLAHTbryGjej+OmlwXelfAjyYE30DF33yUkFdK VGgzwS3nRVwJ+mCaUO9HKBMdaJgiXLoHBUDPwmiHgSNI1kv7TLemODHHKCNBx/otAYHr jaOggGdmtkOfTpMLiuohHXdH1byLHQbV87oZEMCeONirSFl5cwkTE+vGWjFGW4yYAJgs DOvxuOE/PFRaTwGTEx8Gxtt+BICC31RASrdEdqX9Y5ZigHNAZz91hw6yJZu30sA7xBpL wxqg== X-Gm-Message-State: AHYfb5jwcKtFy8+fPRyCdFeSyudJ+R8FT88fuA0jE72rV4sCYZIDUoO3 nz5CjpX8cUmHNTm/eiSLERAoM26myS+8 X-Received: by 10.28.12.201 with SMTP id 192mr2274527wmm.45.1502622801545; Sun, 13 Aug 2017 04:13:21 -0700 (PDT) MIME-Version: 1.0 Received: by 10.223.167.2 with HTTP; Sun, 13 Aug 2017 04:13:20 -0700 (PDT) In-Reply-To: <399c4309-c7a6-a7e9-299f-9675224c090d@norma.perm.ru> References: <399c4309-c7a6-a7e9-299f-9675224c090d@norma.perm.ru> From: Tenzin Lhakhang Date: Sun, 13 Aug 2017 07:13:20 -0400 Message-ID: Subject: Re: zfs listing and CPU To: "Eugene M. Zheganin" Cc: "freebsd-fs@FreeBSD.org" , freebsd-stable Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Aug 2017 11:13:24 -0000 You may want to have an async zfs-get program/script that regularly does a zfs get -Ho and stores then in a local cache (redis or your own program) at a set interval and then the api can hit the cache instead of directly running get or list. - Some silly person will try to benchmark your zfs web-API and overload your server with zfs processes. - Example: let me run [ ab -c 10 -n 10000 http://yourserver/zfs-api/list ] -- Let me run 10 concurrent connection with a total of 10k requests to your api (it's a simple one liner -- people will be tempted to benchmark like this). Example: https://github.com/tlhakhan/ideal-potato/blob/master/zdux/routers/zfs/service.js#L9 - This is a JS example, but you can easily script it or another language (golang) for cache separation and another program for the API. Also, zfs does have a -c property to get cached values -- these values are stored in an internal zfs process cache. The -c doesn't help if you have 1000(0)s of filesystems, a single list can still take minutes. Sending the list is also several megabytes. zfs list -Hrpc -o space zfs get -Hrpc space all - H= no headers - r = recursive - p = machine parseable - c = hit cached entries Fixes: if ok, it may be good to stop the API, kill slowly the zfs list -t all. @ Eugene: - I have seen single zfs list -Ho space -rt all take about 4-5 minutes, on an 8000+ zfs_dataset zpool. --- Notes: my knowledge is from the illumos-zfs man pages but should apply here. On Sun, Aug 13, 2017 at 6:09 AM, Eugene M. Zheganin wrote: > Hi, > > On 12.08.2017 20:50, Paul Kraus wrote: > >> On Aug 11, 2017, at 2:28 AM, Eugene M. Zheganin >>> wrote: >>> >>> Why does the zfs listing eat so much of the CPU ? >>> 47114 root 1 20 0 40432K 3840K db->db 4 0:05 26.84% zfs >>> 47099 root 1 20 0 40432K 3840K zio->i 17 0:05 26.83% zfs >>> 47106 root 1 20 0 40432K 3840K db->db 21 0:05 26.81% zfs >>> 47150 root 1 20 0 40432K 3428K db->db 13 0:03 26.31% zfs >>> 47141 root 1 20 0 40432K 3428K zio->i 28 0:03 26.31% zfs >>> 47135 root 1 20 0 40432K 3312K g_wait 9 0:03 25.51% zfs >>> This is from winter 2017 11-STABLE (r310734), one of the 'zfs'es is >>> cloning, and all the others are 'zfs list -t all'. I have like 25 gigs of >>> free RAM, do I have any chance of speeding this up using may be some >>> caching or some sysctl tuning ? We are using a simple ZFS web API that may >>> issue concurrent or sequential listing requests, so as you can see they >>> sometimes do stack. >>> >> How many snapshots do you have ? I have only seen this behavior with LOTS >> (not hundreds, but thousands) of snapshots. >> > [root@san1:~]# zfs list -t snapshot | wc -l > 88 > >> What does your `iostat -x 1` look like ? I expect that you are probably >> saturating your drives with random I/O. >> > > Well, it's really long, and the disks are busy with random i/o indeed, but > byst only for 20-30%. As about iostat - it's really long, because I have > hundreds (not thousands) of zvols, and they do show up in iostat -x. But > nothing unusual besides that. > > > Thanks. > > Eugene. > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >