From owner-svn-src-all@freebsd.org Fri Feb 15 07:53:57 2019 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EC46C14F33CC; Fri, 15 Feb 2019 07:53:56 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail106.syd.optusnet.com.au (mail106.syd.optusnet.com.au [211.29.132.42]) by mx1.freebsd.org (Postfix) with ESMTP id 45607839B8; Fri, 15 Feb 2019 07:53:55 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from [192.168.0.102] (c110-21-101-228.carlnfd1.nsw.optusnet.com.au [110.21.101.228]) by mail106.syd.optusnet.com.au (Postfix) with ESMTPS id 532643D4313; Fri, 15 Feb 2019 18:53:53 +1100 (AEDT) Date: Fri, 15 Feb 2019 18:53:52 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Mark Johnston cc: Justin Hibbits , Gleb Smirnoff , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: Re: svn commit: r343030 - in head/sys: cam conf dev/md dev/nvme fs/fuse fs/nfsclient fs/smbfs kern sys ufs/ffs vm In-Reply-To: <20190214161153.GA50900@raichu> Message-ID: <20190215182303.A1446@besplex.bde.org> References: <201901150102.x0F12Hlt025856@repo.freebsd.org> <20190213192450.32343d6a@ralga.knownspace> <20190214153345.C1404@besplex.bde.org> <20190214161153.GA50900@raichu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.2 cv=P6RKvmIu c=1 sm=1 tr=0 a=PalzARQSbocsUSjMRkwAPg==:117 a=PalzARQSbocsUSjMRkwAPg==:17 a=kj9zAlcOel0A:10 a=iUrt-v8bpbI0hJeD-q4A:9 a=CjuIK1q_8ugA:10 X-Rspamd-Queue-Id: 45607839B8 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.97 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.97)[-0.969,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Feb 2019 07:53:57 -0000 On Thu, 14 Feb 2019, Mark Johnston wrote: > On Thu, Feb 14, 2019 at 06:56:42PM +1100, Bruce Evans wrote: >* ... >> The only relevant commit between the good and bad versions seems to be >> r343453. This fixes uma_prealloc() to actually work. But it is a feature >> for it to not work when its caller asks for too much. > > I guess you meant r343353. In any case, the pbuf keg is _NOFREE, so > even without preallocation the large pbuf zone limits may become > problematic if there are bursts of allocation requests. Oops. >* ... >> I don't understand how pbuf_preallocate() allocates for the other >> pbuf pools. When I debugged this for clpbufs, the preallocation was >> not used. pbuf types other than clpbufs seem to be unused in my >> configurations. I thought that pbufs were used during initialization, >> since they end up with a nonzero FREE count, but their only use seems >> to be to preallocate them. > > All of the pbuf zones share a common slab allocator. The zones have > individual limits but can tap in to the shared preallocation. It seems to be working as intended now (except the allocation count is 3 higher than expected): XX ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP XX XX swrbuf: 336, 128, 0, 0, 0, 0, 0 XX swwbuf: 336, 64, 0, 0, 0, 0, 0 XX nfspbuf: 336, 128, 0, 0, 0, 0, 0 XX mdpbuf: 336, 25, 0, 0, 0, 0, 0 XX clpbuf: 336, 128, 0, 35, 2918, 0, 0 XX vnpbuf: 336, 2048, 0, 0, 0, 0, 0 XX pbuf: 336, 16, 0, 2505, 0, 0, 0 pbuf should har 2537 preallocated and FREE initially, but seems to actually have 2540. pbufs were only used for clustering, and 35 of them were moved from pbuf to clpbuf. In the buggy version, the preallocations stopped after 4. Then clustering presumably moved these 4 to clpbuf. After that, clustering presumably used non-preallocated buffers until it reached its limit, and then recycled its own buffers. What should happen to recover the old overcommit behaviour with better debugging is 256 preallocated buffers (a few more for large systems) in pbuf and moving these to other pools, but never allocating from other pools (keep buffers in other pools only as an optimization and release them to the main pool under pressure). Also allow dynamic tuning of the pool[s] size[s]. The vnode cache does essentially this by using 1 overcommitted pool with unlimited size in uma and external management of the size. The separate pools correspond to separate file systems. These are too hard to manage, so the vnode cache throws everything into the main pool and depends on locality for the overcommit to not be too large. Bruce