From owner-svn-src-head@freebsd.org Thu Feb 27 08:23:12 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A0532258D5C; Thu, 27 Feb 2020 08:23:12 +0000 (UTC) (envelope-from jeff@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48Slzg0nnxz4GVf; Thu, 27 Feb 2020 08:23:10 +0000 (UTC) (envelope-from jeff@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id BDE35DCED; Thu, 27 Feb 2020 08:23:10 +0000 (UTC) (envelope-from jeff@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 01R8NAVt032672; Thu, 27 Feb 2020 08:23:10 GMT (envelope-from jeff@FreeBSD.org) Received: (from jeff@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 01R8NAAQ032671; Thu, 27 Feb 2020 08:23:10 GMT (envelope-from jeff@FreeBSD.org) Message-Id: <202002270823.01R8NAAQ032671@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jeff set sender to jeff@FreeBSD.org using -f From: Jeff Roberson Date: Thu, 27 Feb 2020 08:23:10 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r358377 - head/sys/vm X-SVN-Group: head X-SVN-Commit-Author: jeff X-SVN-Commit-Paths: head/sys/vm X-SVN-Commit-Revision: 358377 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Feb 2020 08:23:12 -0000 Author: jeff Date: Thu Feb 27 08:23:10 2020 New Revision: 358377 URL: https://svnweb.freebsd.org/changeset/base/358377 Log: A pair of performance improvements. Swap buckets on free as well as alloc so that alloc is always the most cache-hot data. When selecting a zone domain for the round-robin bucket cache use the local domain unless there is a severe imbalance. This does not affinitize memory, only locks and queues. Reviewed by: markj, rlibby Differential Revision: https://reviews.freebsd.org/D23824 Modified: head/sys/vm/uma_core.c Modified: head/sys/vm/uma_core.c ============================================================================== --- head/sys/vm/uma_core.c Thu Feb 27 07:02:33 2020 (r358376) +++ head/sys/vm/uma_core.c Thu Feb 27 08:23:10 2020 (r358377) @@ -564,26 +564,29 @@ zone_domain_lock(uma_zone_t zone, int domain) } /* - * Search for the domain with the least cached items and return it, breaking - * ties with a preferred domain by returning it. + * Search for the domain with the least cached items and return it if it + * is out of balance with the preferred domain. */ static __noinline int zone_domain_lowest(uma_zone_t zone, int pref) { - long least, nitems; + long least, nitems, prefitems; int domain; int i; - least = LONG_MAX; + prefitems = least = LONG_MAX; domain = 0; for (i = 0; i < vm_ndomains; i++) { nitems = ZDOM_GET(zone, i)->uzd_nitems; if (nitems < least) { domain = i; least = nitems; - } else if (nitems == least && (i == pref || domain == pref)) - domain = pref; + } + if (domain == pref) + prefitems = nitems; } + if (prefitems < least * 2) + return (pref); return (domain); } @@ -4102,8 +4105,11 @@ uma_zfree_arg(uma_zone_t zone, void *item, void *udata bucket = &cache->uc_crossbucket; } else #endif - if (bucket->ucb_cnt >= bucket->ucb_entries) - bucket = &cache->uc_freebucket; + if (bucket->ucb_cnt == bucket->ucb_entries && + cache->uc_freebucket.ucb_cnt < + cache->uc_freebucket.ucb_entries) + cache_bucket_swap(&cache->uc_freebucket, + &cache->uc_allocbucket); if (__predict_true(bucket->ucb_cnt < bucket->ucb_entries)) { cache_bucket_push(cache, bucket, item); critical_exit();