From owner-freebsd-stable@freebsd.org Mon Aug 14 11:15:11 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C0321DE3B24; Mon, 14 Aug 2017 11:15:11 +0000 (UTC) (envelope-from tenzin.lhakhang@gmail.com) Received: from mail-wr0-x22e.google.com (mail-wr0-x22e.google.com [IPv6:2a00:1450:400c:c0c::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5C51177DAD; Mon, 14 Aug 2017 11:15:11 +0000 (UTC) (envelope-from tenzin.lhakhang@gmail.com) Received: by mail-wr0-x22e.google.com with SMTP id y41so8774914wrd.3; Mon, 14 Aug 2017 04:15:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=oX3ioggygsDcsNUW6QE0FYMIBCQkF6IxPhYSCCQoaGA=; b=sRmBm5fQn7YGyz+581g3//62tgjCsjH9Fgo4GLgD1QnwxoNQnSssT7bbPGmKsf5StB 1HQd1c6hS9XSWBtr5yIZ1A6f3M5uIeMBtK5ohHRYm+Y/KvCWjX0WchGyv3tGhbYyP8De XsPVP0iLW5sfapxB7HEZsMY3Ev4642qw54twCmxME+BMP+m8fLtTZWSvyJpqx3Xuk8d4 N1pLFMTHWpczxnixoMGhtSQYeiADRsiqz3Q2IclMpK2sV8i8HJAUq7xNIQcDYg8RyK9h bk30igYQHUgfAvF00GxQHeE5LK61sspC8kRVz8T0ZjAjX5NCdsgjioWyf1T8TjjyMcU/ yYbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=oX3ioggygsDcsNUW6QE0FYMIBCQkF6IxPhYSCCQoaGA=; b=g6boV2I/u7pUeUrmUBnVbD2vc8Qz0rfWqf+80lNqP8uddHxxPFWntrsemz5C1Ma/wM Nt4Vyc9nC+/GwNHmGZRXE3moPF4DuZO4ED4ReCdd2CaFns4KC9GRqpflUIiAaFTSGn9l 58to7Butk04T4KgAf2fLU8/+eMcJHpnn5+ZZqfhcadZz3q+Vw1z1ka40s1Lrov+/gjoV Rr2+AZjolEK9t0smtti2wFNNTdizTGwG+uFfaGnhZVGZo99byQJKwcVJH9gtV51zVV2z FMVNEOPwgByTEdl50UUU0rRNF2ejlrpP2iqG3eTAIFxi+Y2qfS9LaQ+dIxdk573QtBLg GheQ== X-Gm-Message-State: AHYfb5jX4i8yN5/EISE063AgxQrl/cMDQoVhVHmq51+qjg3EYhjNQrcl u4R5DemaVBbBseBq1gYsquxZGEpxRND0 X-Received: by 10.223.186.130 with SMTP id p2mr1598413wrg.151.1502709308743; Mon, 14 Aug 2017 04:15:08 -0700 (PDT) MIME-Version: 1.0 Received: by 10.223.167.2 with HTTP; Mon, 14 Aug 2017 04:15:07 -0700 (PDT) In-Reply-To: References: <399c4309-c7a6-a7e9-299f-9675224c090d@norma.perm.ru> From: Tenzin Lhakhang Date: Mon, 14 Aug 2017 07:15:07 -0400 Message-ID: Subject: Re: zfs listing and CPU To: "Eugene M. Zheganin" Cc: "freebsd-fs@FreeBSD.org" , freebsd-stable Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Aug 2017 11:15:11 -0000 Interesting, even the illumos man page doesn't have the -c defined. I remember reading about it at one point. I can only get `zfs get -Hrpc all` working, even though -c is not documented. So, I retract my statement on the zfs list with cached. # zfs get -*Hpc* -o property,value name,used,avail zones name zones used 1219313598464 available 761203195904 # I use the above way to create easily parseable zfs_dataset objects. As for needing immediate/latest version of zfs list on entity creation -- you ideally want a way to do cache invalidation or use some sort of message/event driven system between the client that creates the entity and the server. You can perform the cache invalidation by monitoring the 'creation' events that come to your API, or by tailing zpool history [pool] for changes. - I recently read this article and they show a nice way to decouple the storage events and the api, architecture is not entirely new but can be adapted to your case. - https://medium.freecodecamp.org/million-websockets-and-go-cc58418460bb Thanks, Tenzin On Sun, Aug 13, 2017 at 2:25 PM, Eugene M. Zheganin wrote: > On 13.08.2017 16:13, Tenzin Lhakhang wrote: > >> You may want to have an async zfs-get program/script that regularly does >> a zfs get -Ho and stores then in a local cache (redis or your own program) >> at a set interval and then the api can hit the cache instead of directly >> running get or list. >> > I cannot because the cache will become stale on first new entity creation, > which happens all the time. > > - Some silly person will try to benchmark your zfs web-API and overload >> your server with zfs processes. >> - Example: let me run [ ab -c 10 -n 10000 http://yourserver/zfs-api/list >> ] >> -- Let me run 10 concurrent connection with a total of 10k requests to >> your api (it's a simple one liner -- people will be tempted to benchmark >> like this). >> >> Example: >> https://github.com/tlhakhan/ideal-potato/blob/master/zdux/ro >> uters/zfs/service.js#L9 >> - This is a JS example, but you can easily script it or another language >> (golang) for cache separation and another program for the API. >> >> Also, zfs does have a -c property to get cached values -- these values >> are stored in an internal zfs process cache. The -c doesn't help if you >> have 1000(0)s of filesystems, a single list can still take minutes. >> Sending the list is also several megabytes. >> > Doesn't have on FreeBSD. > > > > Thanks. > Eugene. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >