From owner-svn-src-all@freebsd.org Mon Nov 30 16:18:33 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EB2A34A2115; Mon, 30 Nov 2020 16:18:33 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cl9QK6L0jz3s1q; Mon, 30 Nov 2020 16:18:33 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CC333262C6; Mon, 30 Nov 2020 16:18:33 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 0AUGIXd3053159; Mon, 30 Nov 2020 16:18:33 GMT (envelope-from markj@FreeBSD.org) Received: (from markj@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 0AUGIXP0053158; Mon, 30 Nov 2020 16:18:33 GMT (envelope-from markj@FreeBSD.org) Message-Id: <202011301618.0AUGIXP0053158@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: markj set sender to markj@FreeBSD.org using -f From: Mark Johnston Date: Mon, 30 Nov 2020 16:18:33 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r368189 - head/sys/vm X-SVN-Group: head X-SVN-Commit-Author: markj X-SVN-Commit-Paths: head/sys/vm X-SVN-Commit-Revision: 368189 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Nov 2020 16:18:34 -0000 Author: markj Date: Mon Nov 30 16:18:33 2020 New Revision: 368189 URL: https://svnweb.freebsd.org/changeset/base/368189 Log: uma: Avoid allocating buckets with the cross-domain lock held Allocation of a bucket can trigger a cross-domain free in the bucket zone, e.g., if the per-CPU alloc bucket is empty, we free it and get migrated to a remote domain. This can lead to deadlocks since a bucket zone may allocate buckets from itself or a pair of bucket zones could be allocating from each other. Fix the problem by dropping the cross-domain lock before allocating a new bucket and handling refill races. Use a list of empty buckets to ensure that we can make forward progress. Reported by: imp, mjg (witness(9) warnings) Discussed with: jeff Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D27341 Modified: head/sys/vm/uma_core.c Modified: head/sys/vm/uma_core.c ============================================================================== --- head/sys/vm/uma_core.c Mon Nov 30 15:04:35 2020 (r368188) +++ head/sys/vm/uma_core.c Mon Nov 30 16:18:33 2020 (r368189) @@ -4250,7 +4250,7 @@ zfree_item: static void zone_free_cross(uma_zone_t zone, uma_bucket_t bucket, void *udata) { - struct uma_bucketlist fullbuckets; + struct uma_bucketlist emptybuckets, fullbuckets; uma_zone_domain_t zdom; uma_bucket_t b; smr_seq_t seq; @@ -4274,31 +4274,57 @@ zone_free_cross(uma_zone_t zone, uma_bucket_t bucket, * lock on the current crossfree bucket. A full matrix with * per-domain locking could be used if necessary. */ + STAILQ_INIT(&emptybuckets); STAILQ_INIT(&fullbuckets); ZONE_CROSS_LOCK(zone); - while (bucket->ub_cnt > 0) { + for (; bucket->ub_cnt > 0; bucket->ub_cnt--) { item = bucket->ub_bucket[bucket->ub_cnt - 1]; domain = item_domain(item); zdom = ZDOM_GET(zone, domain); if (zdom->uzd_cross == NULL) { - zdom->uzd_cross = bucket_alloc(zone, udata, M_NOWAIT); - if (zdom->uzd_cross == NULL) - break; + if ((b = STAILQ_FIRST(&emptybuckets)) != NULL) { + STAILQ_REMOVE_HEAD(&emptybuckets, ub_link); + zdom->uzd_cross = b; + } else { + /* + * Avoid allocating a bucket with the cross lock + * held, since allocation can trigger a + * cross-domain free and bucket zones may + * allocate from each other. + */ + ZONE_CROSS_UNLOCK(zone); + b = bucket_alloc(zone, udata, M_NOWAIT); + if (b == NULL) + goto out; + ZONE_CROSS_LOCK(zone); + if (zdom->uzd_cross != NULL) { + STAILQ_INSERT_HEAD(&emptybuckets, b, + ub_link); + } else { + zdom->uzd_cross = b; + } + } } b = zdom->uzd_cross; b->ub_bucket[b->ub_cnt++] = item; b->ub_seq = seq; if (b->ub_cnt == b->ub_entries) { STAILQ_INSERT_HEAD(&fullbuckets, b, ub_link); - zdom->uzd_cross = NULL; + if ((b = STAILQ_FIRST(&emptybuckets)) != NULL) + STAILQ_REMOVE_HEAD(&emptybuckets, ub_link); + zdom->uzd_cross = b; } - bucket->ub_cnt--; } ZONE_CROSS_UNLOCK(zone); +out: if (bucket->ub_cnt == 0) bucket->ub_seq = SMR_SEQ_INVALID; bucket_free(zone, bucket, udata); + while ((b = STAILQ_FIRST(&emptybuckets)) != NULL) { + STAILQ_REMOVE_HEAD(&emptybuckets, ub_link); + bucket_free(zone, b, udata); + } while ((b = STAILQ_FIRST(&fullbuckets)) != NULL) { STAILQ_REMOVE_HEAD(&fullbuckets, ub_link); domain = item_domain(b->ub_bucket[0]);