From owner-svn-src-all@freebsd.org Tue Feb 18 01:21:57 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 078F924CEB6; Tue, 18 Feb 2020 01:21:57 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48M33m63Kcz4C5f; Tue, 18 Feb 2020 01:21:56 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id BE1501825A; Tue, 18 Feb 2020 01:21:56 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 01I1Lusv056636; Tue, 18 Feb 2020 01:21:56 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 01I1LuB2056635; Tue, 18 Feb 2020 01:21:56 GMT (envelope-from mav@FreeBSD.org) Message-Id: <202002180121.01I1LuB2056635@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 18 Feb 2020 01:21:56 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-12@freebsd.org Subject: svn commit: r358060 - in stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Group: stable-12 X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 358060 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Feb 2020 01:21:57 -0000 Author: mav Date: Tue Feb 18 01:21:56 2020 New Revision: 358060 URL: https://svnweb.freebsd.org/changeset/base/358060 Log: MFC r357502: Few microoptimizations to dbuf layer. Move db_link into the same cache line as db_blkid and db_level. It allows significantly reduce avl_add() time in dbuf_create() on systems with large RAM and huge number of dbufs per dnode. Avoid few accesses to dbuf_caches[].size, which is highly congested under high IOPS and never stays in cache for a long time. Use local value we are receiving from zfs_refcount_add_many() any way. Remove cache_size_bytes_max bump from dbuf_evict_one(). I don't see a point to do it on dbuf eviction after we done it on insertion in dbuf_rele_and_unlock(). Modified: stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dbuf.h Directory Properties: stable/12/ (props changed) Modified: stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c ============================================================================== --- stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Tue Feb 18 01:19:32 2020 (r358059) +++ stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c Tue Feb 18 01:21:56 2020 (r358060) @@ -671,13 +671,6 @@ dbuf_cache_lowater_bytes(void) } static inline boolean_t -dbuf_cache_above_hiwater(void) -{ - return (zfs_refcount_count(&dbuf_caches[DB_DBUF_CACHE].size) > - dbuf_cache_hiwater_bytes()); -} - -static inline boolean_t dbuf_cache_above_lowater(void) { return (zfs_refcount_count(&dbuf_caches[DB_DBUF_CACHE].size) > @@ -717,8 +710,6 @@ dbuf_evict_one(void) ASSERT3U(db->db_caching_status, ==, DB_DBUF_CACHE); db->db_caching_status = DB_NO_CACHE; dbuf_destroy(db); - DBUF_STAT_MAX(cache_size_bytes_max, - zfs_refcount_count(&dbuf_caches[DB_DBUF_CACHE].size)); DBUF_STAT_BUMP(cache_total_evicts); } else { multilist_sublist_unlock(mls); @@ -778,16 +769,15 @@ dbuf_evict_thread(void *unused __unused) * dbuf cache using the callers context. */ static void -dbuf_evict_notify(void) +dbuf_evict_notify(uint64_t size) { /* * We check if we should evict without holding the dbuf_evict_lock, * because it's OK to occasionally make the wrong decision here, * and grabbing the lock results in massive lock contention. */ - if (zfs_refcount_count(&dbuf_caches[DB_DBUF_CACHE].size) > - dbuf_cache_max_bytes) { - if (dbuf_cache_above_hiwater()) + if (size > dbuf_cache_max_bytes) { + if (size > dbuf_cache_hiwater_bytes()) dbuf_evict_one(); cv_signal(&dbuf_evict_cv); } @@ -3186,6 +3176,7 @@ void dbuf_rele_and_unlock(dmu_buf_impl_t *db, void *tag, boolean_t evicting) { int64_t holds; + uint64_t size; ASSERT(MUTEX_HELD(&db->db_mtx)); DBUF_VERIFY(db); @@ -3282,15 +3273,14 @@ dbuf_rele_and_unlock(dmu_buf_impl_t *db, void *tag, bo db->db_caching_status = dcs; multilist_insert(dbuf_caches[dcs].cache, db); - (void) zfs_refcount_add_many( + size = zfs_refcount_add_many( &dbuf_caches[dcs].size, db->db.db_size, db); if (dcs == DB_DBUF_METADATA_CACHE) { DBUF_STAT_BUMP(metadata_cache_count); DBUF_STAT_MAX( metadata_cache_size_bytes_max, - zfs_refcount_count( - &dbuf_caches[dcs].size)); + size); } else { DBUF_STAT_BUMP( cache_levels[db->db_level]); @@ -3299,15 +3289,12 @@ dbuf_rele_and_unlock(dmu_buf_impl_t *db, void *tag, bo cache_levels_bytes[db->db_level], db->db.db_size); DBUF_STAT_MAX(cache_size_bytes_max, - zfs_refcount_count( - &dbuf_caches[dcs].size)); + size); } mutex_exit(&db->db_mtx); - if (db->db_caching_status == DB_DBUF_CACHE && - !evicting) { - dbuf_evict_notify(); - } + if (dcs == DB_DBUF_CACHE && !evicting) + dbuf_evict_notify(size); } if (do_arc_evict) Modified: stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dbuf.h ============================================================================== --- stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dbuf.h Tue Feb 18 01:19:32 2020 (r358059) +++ stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dbuf.h Tue Feb 18 01:21:56 2020 (r358060) @@ -189,6 +189,13 @@ typedef struct dmu_buf_impl { */ struct dmu_buf_impl *db_hash_next; + /* + * Our link on the owner dnodes's dn_dbufs list. + * Protected by its dn_dbufs_mtx. Should be on the same cache line + * as db_level and db_blkid for the best avl_add() performance. + */ + avl_node_t db_link; + /* our block number */ uint64_t db_blkid; @@ -229,12 +236,6 @@ typedef struct dmu_buf_impl { /* pointer to most recent dirty record for this buffer */ dbuf_dirty_record_t *db_last_dirty; - - /* - * Our link on the owner dnodes's dn_dbufs list. - * Protected by its dn_dbufs_mtx. - */ - avl_node_t db_link; /* Link in dbuf_cache or dbuf_metadata_cache */ multilist_node_t db_cache_link;