From owner-svn-src-head@freebsd.org Sun Sep 8 20:27:43 2019 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0A4D1E09A2; Sun, 8 Sep 2019 20:27:43 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-io1-xd42.google.com (mail-io1-xd42.google.com [IPv6:2607:f8b0:4864:20::d42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46RNC20RDrz43Kb; Sun, 8 Sep 2019 20:27:41 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-io1-xd42.google.com with SMTP id h144so24130085iof.7; Sun, 08 Sep 2019 13:27:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=bL823FPjBR5qUjIupXjojBD3rJVDnaCjLe9xvBx1KJk=; b=kDva9FqXT+OUkCdsQwRVdqmA3NsrESsxSF8WW1YG7Fu1xUHcbwDwrzzD7NiMDGsuMx 5sbc3e7VdE5EQ3SzGR0DQtO6vW7wS0zNV/XZT/lMsn3HWy9e1Ok7VqgCWz0gs7xazojN BubUXHgjUNEgrBGWSP9PO/duVrM9wieF+Sbp4MYon97KWtmxf8bL8CkoEHreL2me3l5A SWQfRhjbdTBFDX+rr9ThKtN0GnYR7WDQ7EuCno2NTe1oPcXtdtvsUwr25Fy/EJryMdu2 fkcPzFXc7geP97o5a+/diqGAI5U0wr1G7g/Q4bR8hrfGAxEzAR9KNry/OjDBNS/LswBU RyIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=bL823FPjBR5qUjIupXjojBD3rJVDnaCjLe9xvBx1KJk=; b=Ovsj6aRH+xNEdN0mbQeP5bUnEVyRWSK8zqop2xqkEtosXEAtvAaGiGgyfrLCh9l8pB ZZtTZpltZDfiJTbiHLjUm99vqpQ03ZPen2y2ciVcuMTx0TepvnHeqPsvHgiwSB+5xDUY 7cAfStq8azSj0ixVle2Ak+H1jJxOd3c0aDRgFK4To8uO3ddlqOVBFDOx1Sb3bF7k0kqT pu3fIXyQXvDQ48apWZnkp3fKCC+/kKWS2nEE6sMybZgPXu1JmvSE+5N+0hn/JQFR1APQ P2HZktbtaxsBLzA55/zOKFIHwOjkvogwyPKltfH/x2m1IwF+zowZauGIc1ZerEu5lKFy ykdQ== X-Gm-Message-State: APjAAAUgZ15S7y/8YhQshJKvglcrMhICzqftHyn+lo18R3bi3leDo1vz Hd7baxivOjhgYungr8c3v5usQz1G3Dg= X-Google-Smtp-Source: APXvYqysphZqFTGZxgZg6IKPwDnmOTZke1JRc3zTFIF+Gdfa3jtG9TI9iUFZNfVu3tOFRa7rBLvLcA== X-Received: by 2002:a05:6638:692:: with SMTP id i18mr21559519jab.108.1567974460724; Sun, 08 Sep 2019 13:27:40 -0700 (PDT) Received: from raichu (toroon0560w-lp140-01-69-159-39-167.dsl.bell.ca. [69.159.39.167]) by smtp.gmail.com with ESMTPSA id s201sm32893839ios.83.2019.09.08.13.27.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Sep 2019 13:27:39 -0700 (PDT) Sender: Mark Johnston Date: Sun, 8 Sep 2019 16:18:19 -0400 From: Mark Johnston To: Slawa Olhovchenkov Cc: Andriy Gapon , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: Re: svn commit: r351673 - in head: lib/libmemstat share/man/man9 sys/cddl/compat/opensolaris/kern sys/kern sys/vm Message-ID: <20190908201819.GA49837@raichu> References: <201909012222.x81MMh0F022462@repo.freebsd.org> <79c74018-1329-ee69-3480-e2f99821fa93@FreeBSD.org> <20190903161427.GA38096@zxy.spb.ru> <20190903220106.GB26733@raichu> <20190904144524.GD3953@zxy.spb.ru> <20190907145034.GB6523@spy> <20190907153110.GG3953@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190907153110.GG3953@zxy.spb.ru> User-Agent: Mutt/1.12.1 (2019-06-15) X-Rspamd-Queue-Id: 46RNC20RDrz43Kb X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=kDva9FqX; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::d42 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-2.25 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_IN_DNSWL_NONE(0.00)[2.4.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.55)[ip: (2.32), ipnet: 2607:f8b0::/32(-2.75), asn: 15169(-2.27), country: US(-0.05)]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; RCVD_TLS_ALL(0.00)[] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Sep 2019 20:27:43 -0000 On Sat, Sep 07, 2019 at 06:31:10PM +0300, Slawa Olhovchenkov wrote: > On Sat, Sep 07, 2019 at 10:50:34AM -0400, Mark Johnston wrote: > > > On Wed, Sep 04, 2019 at 05:45:24PM +0300, Slawa Olhovchenkov wrote: > > > On Tue, Sep 03, 2019 at 06:01:06PM -0400, Mark Johnston wrote: > > > > > Mostly problem I am see at this > > > > > work -- very slowly vm_page_free(). May be currenly this is more > > > > > speedy... > > > > > > > > How did you determine this? > > > > > > This is you guess: > > > > So was the guess correct? > > I am just trust to you. > How to check this guess? You can try to measure time spent in pmap_remove() relative to the rest of _kmem_unback(). > > If so, IMO the real solution is to avoid kmem_* > > for data buffers and use ABD instead. > > What problem resolve this? To allocate buffers larger than PAGE_SIZE the kernel must allocate a number of physical pages and map them using the page tables. The step of creating and destroying mappings is expensive and doesn't scale well to many CPUs. With ABD, ZFS avoids this expense when its caches are shrunk. > ABD any way is slowly vs kmem_*. Can we solve this problem instead? > > > ====== > > > > while ((slab = SLIST_FIRST(&freeslabs)) != NULL) { > > > > SLIST_REMOVE(&freeslabs, slab, uma_slab, us_hlink); > > > > keg_free_slab(keg, slab, keg->uk_ipers); > > > > } > > > > 2019 Feb 2 19:49:54.800524364 zio_data_buf_1048576 1032605 cache_reclaim limit 100 dom 0 nitems 1672 imin 298 > > > > 2019 Feb 2 19:49:54.800524364 zio_data_buf_1048576 1033736 cache_reclaim recla 149 dom 0 nitems 1672 imin 298 > > > > 2019 Feb 2 19:49:54.802524468 zio_data_buf_1048576 3119710 cache_reclaim limit 100 dom 1 nitems 1 imin 0 > > > > 2019 Feb 2 19:49:54.802524468 zio_data_buf_1048576 3127550 keg_drain2 > > > > 2019 Feb 2 19:49:54.803524487 zio_data_buf_1048576 4444219 keg_drain3 > > > > 2019 Feb 2 19:49:54.838524634 zio_data_buf_1048576 39553705 keg_drain4 > > > > 2019 Feb 2 19:49:54.838524634 zio_data_buf_1048576 39565323 zone_reclaim:return > > > > > > > > 35109.486 ms for last loop, 149 items to freed. > > > > > > 35ms to free 149MB (38144 4KB pages), so roughly 1us per page. That > > > does seem like a lot, but freeing a page (vm_page_free(m)) is much > > > more expensive than freeing an item to UMA (i.e., uma_zfree()). > > > Most of that time will be spent in _kmem_unback(). > > > ======