From nobody Fri Jan 31 18:40:29 2025 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Yl4TL40Dwz5mnBN for ; Fri, 31 Jan 2025 18:40:38 +0000 (UTC) (envelope-from jrtc27@jrtc27.com) Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Yl4TL082Hz3DrB for ; Fri, 31 Jan 2025 18:40:38 +0000 (UTC) (envelope-from jrtc27@jrtc27.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-wr1-f46.google.com with SMTP id ffacd0b85a97d-385e1fcb0e1so1183546f8f.2 for ; Fri, 31 Jan 2025 10:40:37 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738348836; x=1738953636; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rh3NIVKmUmc3L17V2z3yKBnaM3FaTZ4SR6gm7a4gqug=; b=YiLDI42Q+6s5jUax1PlmlBZYadyLAa++GxHvOSNq3Z8cMIucEiqjOosmWsKxgeR4l9 VcshTDABcSvODnxjWlo3ulUFhm/ghJw7I2R1a4rcgoLb4e3DFCwpp3f3SutYHnz0d/jr v8+6clgr8HkFaV4LuG6W/V4Cvh2ER1zTC0DwDAGaglyhAMAxZBDWpchf5/JY2a6dMrt+ t0oEmUcPAyocgICoASy71EuEwuXub12XPdhoAXh+w3OUTl2ccoMgFOk4SEXYQ6hiHzzY aWD31qO51BVjmyIlaZNnkgJAzh8glyyIrURv8dfA3EKeCkfJExGW8ic5U3t0VQq0fJHM dBEw== X-Forwarded-Encrypted: i=1; AJvYcCVeTf/dLgAa3PBEK2xo3aqm9R13KKQF2aUnTjWKr4ufRd5HaSDJDJ/ea0lP/JMSdoZ5G9NySm+VmXP1uVOFZMLYS9hDpA==@freebsd.org X-Gm-Message-State: AOJu0YzHstM+LzgNBV7izShpZ4IKss3tg88Yvr7+pthKhmbouDwCxyrP Bkz3swjTXrexSgsu931kkFEGtf1sYrJ5pfDisAJf+i9fzGAxwOlVYbjSR6cnASKJcIvxe8g9Cah B X-Gm-Gg: ASbGncu1M86beDqP6Hoxwq/3yoe9SgUXiTm+gR/5YV8qNbDmArfQXVxidIU1Kav/9mU wRPue9VEDtHGkp7NHv2nKMuoZ4iCWR2bwODfNvMiWFaXKFNqALAPGQC4u2CwYqpry0DfxBUztAe su2A4X7SdM76U7Ra32SjFxHe4U3pGiD16KmHWsyysKAEPJMVwQEDlwzFbUE5qX24s0FUbgXOr+d sX/G6aMItZl/Tv+0x5/mte/CxuNbcVmLu1y0ug7iz0CivzEwfwujySJg6UBWDCgDQsB/P/B8ymx ItF2bGoxXe26be2X30ggM6pGFhyHQQ== X-Google-Smtp-Source: AGHT+IF1BQzFc9DAg4NQptVP2YRPJVL/dN/QXvajeK+RIOKMmJ1Mq/1V0ikbynGLWxSo31oS73Owyw== X-Received: by 2002:adf:e386:0:b0:385:faf5:eb9f with SMTP id ffacd0b85a97d-38c520af4d3mr9486299f8f.48.1738348836088; Fri, 31 Jan 2025 10:40:36 -0800 (PST) Received: from smtpclient.apple ([131.111.5.201]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38c5c102e19sm5423624f8f.36.2025.01.31.10.40.35 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 31 Jan 2025 10:40:35 -0800 (PST) Content-Type: text/plain; charset=us-ascii List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.300.87.4.3\)) Subject: Re: git: 0078df5f0258 - main - vm_phys: reduce touching of page->pool fields From: Jessica Clarke In-Reply-To: Date: Fri, 31 Jan 2025 18:40:29 +0000 Cc: Doug Moore , src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <202501290914.50T9EcNn077539@gitrepo.freebsd.org> To: Ruslan Bukin X-Mailer: Apple Mail (2.3826.300.87.4.3) X-Rspamd-Queue-Id: 4Yl4TL082Hz3DrB X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] On 30 Jan 2025, at 22:11, Ruslan Bukin wrote: >=20 > Hi Doug, >=20 > riscv seems to panic after this commit. See the message > https://people.freebsd.org/~br/panic.txt >=20 > I think I hit the same panic around a week ago with the previous (now = reverted) commit. Does 4015ff43cbbe fix this for you? Jess > On Wed, Jan 29, 2025 at 09:14:38AM +0000, Doug Moore wrote: >> The branch main has been updated by dougm: >>=20 >> URL: = https://cgit.FreeBSD.org/src/commit/?id=3D0078df5f025854600595462e56fa95d3= 4e732141 >>=20 >> commit 0078df5f025854600595462e56fa95d34e732141 >> Author: Doug Moore >> AuthorDate: 2025-01-29 09:13:17 +0000 >> Commit: Doug Moore >> CommitDate: 2025-01-29 09:13:17 +0000 >>=20 >> vm_phys: reduce touching of page->pool fields >>=20 >> Change the usage of the pool field in vm_page structs. >>=20 >> Currently, every page belongs to a pool, and the pool field = identifies >> that pool, whether the page is allocated or free. >>=20 >> With this change, the pool field of the first page of a free block = is >> used by the buddy allocator to identify its pool, but the buddy >> allocator makes no guarantees about the pool field value for = allocated >> pages. The buddy allocator requires that a pool parameter be = passed as >> part of freeing memory. A function that allocates memory may use = the >> pool field of a page to record what pool to pass as that parameter >> when the memory is freed, but might not need to do so for every >> allocated page. >>=20 >> Suggested by: alc >> Reviewed by: markj (previous version) >> Tested by: pho >> Differential Revision: https://reviews.freebsd.org/D45409 >> --- >> sys/vm/vm_kern.c | 2 +- >> sys/vm/vm_page.c | 97 ++++++++++++++++++++++-------------------- >> sys/vm/vm_phys.c | 123 = ++++++++++++++++++++++++++++------------------------- >> sys/vm/vm_phys.h | 6 +-- >> sys/vm/vm_reserv.c | 31 +++++++------- >> 5 files changed, 136 insertions(+), 123 deletions(-) >>=20 >> diff --git a/sys/vm/vm_kern.c b/sys/vm/vm_kern.c >> index 86ab2529e27f..2b85dbde1dd6 100644 >> --- a/sys/vm/vm_kern.c >> +++ b/sys/vm/vm_kern.c >> @@ -953,7 +953,7 @@ kmem_bootstrap_free(vm_offset_t start, vm_size_t = size) >>=20 >> vmd =3D vm_pagequeue_domain(m); >> vm_domain_free_lock(vmd); >> - vm_phys_free_pages(m, 0); >> + vm_phys_free_pages(m, m->pool, 0); >> vm_domain_free_unlock(vmd); >>=20 >> vm_domain_freecnt_inc(vmd, 1); >> diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c >> index ba22c7f97f2f..961b32da6599 100644 >> --- a/sys/vm/vm_page.c >> +++ b/sys/vm/vm_page.c >> @@ -572,6 +572,7 @@ vm_page_startup(vm_offset_t vaddr) >> #if defined(__i386__) && defined(VM_PHYSSEG_DENSE) >> long ii; >> #endif >> + int pool; >> #ifdef VM_FREEPOOL_LAZYINIT >> int lazyinit; >> #endif >> @@ -651,6 +652,8 @@ vm_page_startup(vm_offset_t vaddr) >> dump_add_page(pa); >> pa +=3D PAGE_SIZE; >> } >> +#else >> + (void)pa; >> #endif >> /* >> * Compute the number of pages of memory that will be available for >> @@ -755,9 +758,12 @@ vm_page_startup(vm_offset_t vaddr) >> */ >> vm_phys_init(); >>=20 >> + pool =3D VM_FREEPOOL_DEFAULT; >> #ifdef VM_FREEPOOL_LAZYINIT >> lazyinit =3D 1; >> TUNABLE_INT_FETCH("debug.vm.lazy_page_init", &lazyinit); >> + if (lazyinit) >> + pool =3D VM_FREEPOOL_LAZYINIT; >> #endif >>=20 >> /* >> @@ -777,48 +783,27 @@ vm_page_startup(vm_offset_t vaddr) >> seg =3D &vm_phys_segs[segind]; >>=20 >> /* >> - * If lazy vm_page initialization is not enabled, simply >> - * initialize all of the pages in the segment. Otherwise, we >> - * only initialize: >> - * 1. Pages not covered by phys_avail[], since they might be >> - * freed to the allocator at some future point, e.g., by >> - * kmem_bootstrap_free(). >> - * 2. The first page of each run of free pages handed to the >> - * vm_phys allocator, which in turn defers initialization >> - * of pages until they are needed. >> - * This avoids blocking the boot process for long periods, which >> - * may be relevant for VMs (which ought to boot as quickly as >> - * possible) and/or systems with large amounts of physical >> - * memory. >> + * Initialize pages not covered by phys_avail[], since they >> + * might be freed to the allocator at some future point, e.g., >> + * by kmem_bootstrap_free(). >> */ >> -#ifdef VM_FREEPOOL_LAZYINIT >> - if (lazyinit) { >> - startp =3D seg->start; >> - for (i =3D 0; phys_avail[i + 1] !=3D 0; i +=3D 2) { >> - if (startp >=3D seg->end) >> - break; >> - >> - if (phys_avail[i + 1] < startp) >> - continue; >> - if (phys_avail[i] <=3D startp) { >> - startp =3D phys_avail[i + 1]; >> - continue; >> - } >> - >> - m =3D vm_phys_seg_paddr_to_vm_page(seg, startp); >> - for (endp =3D MIN(phys_avail[i], seg->end); >> - startp < endp; startp +=3D PAGE_SIZE, m++) { >> - vm_page_init_page(m, startp, segind, >> - VM_FREEPOOL_DEFAULT); >> - } >> + startp =3D seg->start; >> + for (i =3D 0; phys_avail[i + 1] !=3D 0; i +=3D 2) { >> + if (startp >=3D seg->end) >> + break; >> + if (phys_avail[i + 1] < startp) >> + continue; >> + if (phys_avail[i] <=3D startp) { >> + startp =3D phys_avail[i + 1]; >> + continue; >> } >> - } else >> -#endif >> - for (m =3D seg->first_page, pa =3D seg->start; >> - pa < seg->end; m++, pa +=3D PAGE_SIZE) { >> - vm_page_init_page(m, pa, segind, >> + m =3D vm_phys_seg_paddr_to_vm_page(seg, startp); >> + for (endp =3D MIN(phys_avail[i], seg->end); >> + startp < endp; startp +=3D PAGE_SIZE, m++) { >> + vm_page_init_page(m, startp, segind, >> VM_FREEPOOL_DEFAULT); >> } >> + } >>=20 >> /* >> * Add the segment's pages that are covered by one of >> @@ -835,16 +820,30 @@ vm_page_startup(vm_offset_t vaddr) >> if (pagecount =3D=3D 0) >> continue; >>=20 >> + /* >> + * If lazy vm_page initialization is not enabled, simply >> + * initialize all of the pages in the segment covered by >> + * phys_avail. Otherwise, initialize only the first >> + * page of each run of free pages handed to the vm_phys >> + * allocator, which in turn defers initialization of >> + * pages until they are needed. >> + * >> + * This avoids blocking the boot process for long >> + * periods, which may be relevant for VMs (which ought >> + * to boot as quickly as possible) and/or systems with >> + * large amounts of physical memory. >> + */ >> m =3D vm_phys_seg_paddr_to_vm_page(seg, startp); >> -#ifdef VM_FREEPOOL_LAZYINIT >> - if (lazyinit) { >> - vm_page_init_page(m, startp, segind, >> - VM_FREEPOOL_LAZYINIT); >> + vm_page_init_page(m, startp, segind, pool); >> + if (pool =3D=3D VM_FREEPOOL_DEFAULT) { >> + for (int j =3D 1; j < pagecount; j++) { >> + vm_page_init_page(&m[j], >> + startp + ptoa(j), segind, pool); >> + } >> } >> -#endif >> vmd =3D VM_DOMAIN(seg->domain); >> vm_domain_free_lock(vmd); >> - vm_phys_enqueue_contig(m, pagecount); >> + vm_phys_enqueue_contig(m, pool, pagecount); >> vm_domain_free_unlock(vmd); >> vm_domain_freecnt_inc(vmd, pagecount); >> vm_cnt.v_page_count +=3D (u_int)pagecount; >> @@ -2341,6 +2340,7 @@ found: >> m->flags =3D flags; >> m->a.flags =3D 0; >> m->oflags =3D (object->flags & OBJ_UNMANAGED) !=3D 0 ? VPO_UNMANAGED = : 0; >> + m->pool =3D VM_FREEPOOL_DEFAULT; >> if ((req & (VM_ALLOC_NOBUSY | VM_ALLOC_SBUSY)) =3D=3D 0) >> m->busy_lock =3D VPB_CURTHREAD_EXCLUSIVE; >> else if ((req & VM_ALLOC_SBUSY) !=3D 0) >> @@ -2558,6 +2558,7 @@ vm_page_alloc_contig_domain(vm_object_t object, = vm_pindex_t pindex, int domain, >> m->ref_count =3D 1; >> m->a.act_count =3D 0; >> m->oflags =3D oflags; >> + m->pool =3D VM_FREEPOOL_DEFAULT; >> if (vm_page_iter_insert(&pages, m, object, pindex, mpred)) { >> if ((req & VM_ALLOC_WIRED) !=3D 0) >> vm_wire_sub(npages); >> @@ -2655,6 +2656,7 @@ found: >> m->flags =3D (m->flags & PG_ZERO) | flags; >> m->a.flags =3D 0; >> m->oflags =3D VPO_UNMANAGED; >> + m->pool =3D VM_FREEPOOL_DIRECT; >> m->busy_lock =3D VPB_UNBUSIED; >> if ((req & VM_ALLOC_WIRED) !=3D 0) { >> vm_wire_add(1); >> @@ -2803,6 +2805,7 @@ vm_page_alloc_noobj_contig_domain(int domain, = int req, u_long npages, >> m->ref_count =3D 1; >> m->a.act_count =3D 0; >> m->oflags =3D VPO_UNMANAGED; >> + m->pool =3D VM_FREEPOOL_DIRECT; >>=20 >> /* >> * Zero the page before updating any mappings since the page is >> @@ -2881,7 +2884,7 @@ vm_page_zone_release(void *arg, void **store, = int cnt) >> vm_domain_free_lock(vmd); >> for (i =3D 0; i < cnt; i++) { >> m =3D (vm_page_t)store[i]; >> - vm_phys_free_pages(m, 0); >> + vm_phys_free_pages(m, pgcache->pool, 0); >> } >> vm_domain_free_unlock(vmd); >> vm_domain_freecnt_inc(vmd, cnt); >> @@ -3266,7 +3269,7 @@ unlock: >> do { >> MPASS(vm_page_domain(m) =3D=3D domain); >> SLIST_REMOVE_HEAD(&free, plinks.s.ss); >> - vm_phys_free_pages(m, 0); >> + vm_phys_free_pages(m, m->pool, 0); >> cnt++; >> } while ((m =3D SLIST_FIRST(&free)) !=3D NULL); >> vm_domain_free_unlock(vmd); >> @@ -4271,7 +4274,7 @@ vm_page_free_toq(vm_page_t m) >> return; >> } >> vm_domain_free_lock(vmd); >> - vm_phys_free_pages(m, 0); >> + vm_phys_free_pages(m, m->pool, 0); >> vm_domain_free_unlock(vmd); >> vm_domain_freecnt_inc(vmd, 1); >> } >> diff --git a/sys/vm/vm_phys.c b/sys/vm/vm_phys.c >> index cf1ed5818b2f..0b7de1d34255 100644 >> --- a/sys/vm/vm_phys.c >> +++ b/sys/vm/vm_phys.c >> @@ -187,7 +187,7 @@ SYSCTL_INT(_vm, OID_AUTO, ndomains, CTLFLAG_RD, >> static void _vm_phys_create_seg(vm_paddr_t start, vm_paddr_t end, int = domain); >> static void vm_phys_create_seg(vm_paddr_t start, vm_paddr_t end); >> static void vm_phys_split_pages(vm_page_t m, int oind, struct = vm_freelist *fl, >> - int order, int tail); >> + int order, int pool, int tail); >>=20 >> static bool __diagused >> vm_phys_pool_valid(int pool) >> @@ -391,10 +391,12 @@ sysctl_vm_phys_locality(SYSCTL_HANDLER_ARGS) >> #endif >>=20 >> static void >> -vm_freelist_add(struct vm_freelist *fl, vm_page_t m, int order, int = tail) >> +vm_freelist_add(struct vm_freelist *fl, vm_page_t m, int order, int = pool, >> + int tail) >> { >>=20 >> m->order =3D order; >> + m->pool =3D pool; >> if (tail) >> TAILQ_INSERT_TAIL(&fl[order].pl, m, listq); >> else >> @@ -687,7 +689,7 @@ vm_phys_register_domains(int ndomains = __numa_used, >> */ >> static __inline void >> vm_phys_split_pages(vm_page_t m, int oind, struct vm_freelist *fl, = int order, >> - int tail) >> + int pool, int tail) >> { >> vm_page_t m_buddy; >>=20 >> @@ -697,19 +699,20 @@ vm_phys_split_pages(vm_page_t m, int oind, = struct vm_freelist *fl, int order, >> KASSERT(m_buddy->order =3D=3D VM_NFREEORDER, >> ("vm_phys_split_pages: page %p has unexpected order %d", >> m_buddy, m_buddy->order)); >> - vm_freelist_add(fl, m_buddy, oind, tail); >> + vm_freelist_add(fl, m_buddy, oind, pool, tail); >> } >> } >>=20 >> static void >> -vm_phys_enq_chunk(struct vm_freelist *fl, vm_page_t m, int order, = int tail) >> +vm_phys_enq_chunk(struct vm_freelist *fl, vm_page_t m, int order, = int pool, >> + int tail) >> { >> KASSERT(order >=3D 0 && order < VM_NFREEORDER, >> ("%s: invalid order %d", __func__, order)); >>=20 >> - vm_freelist_add(fl, m, order, tail); >> + vm_freelist_add(fl, m, order, pool, tail); >> #ifdef VM_FREEPOOL_LAZYINIT >> - if (__predict_false(m->pool =3D=3D VM_FREEPOOL_LAZYINIT)) { >> + if (__predict_false(pool =3D=3D VM_FREEPOOL_LAZYINIT)) { >> vm_page_t m_next; >> vm_paddr_t pa; >> int npages; >> @@ -738,7 +741,8 @@ vm_phys_enq_chunk(struct vm_freelist *fl, = vm_page_t m, int order, int tail) >> * The physical page m's buddy must not be free. >> */ >> static void >> -vm_phys_enq_beg(vm_page_t m, u_int npages, struct vm_freelist *fl, = int tail) >> +vm_phys_enq_beg(vm_page_t m, u_int npages, struct vm_freelist *fl, = int pool, >> + int tail) >> { >> int order; >>=20 >> @@ -754,7 +758,7 @@ vm_phys_enq_beg(vm_page_t m, u_int npages, struct = vm_freelist *fl, int tail) >> order =3D ilog2(npages); >> KASSERT(order < VM_NFREEORDER, >> ("%s: order %d is out of range", __func__, order)); >> - vm_phys_enq_chunk(fl, m, order, tail); >> + vm_phys_enq_chunk(fl, m, order, pool, tail); >> m +=3D 1 << order; >> npages -=3D 1 << order; >> } >> @@ -774,7 +778,8 @@ vm_phys_enq_beg(vm_page_t m, u_int npages, struct = vm_freelist *fl, int tail) >> * parameter m. Otherwise, the physical page m's buddy must not be = free. >> */ >> static vm_page_t >> -vm_phys_enq_range(vm_page_t m, u_int npages, struct vm_freelist *fl, = int tail) >> +vm_phys_enq_range(vm_page_t m, u_int npages, struct vm_freelist *fl, = int pool, >> + int tail) >> { >> int order; >>=20 >> @@ -788,7 +793,7 @@ vm_phys_enq_range(vm_page_t m, u_int npages, = struct vm_freelist *fl, int tail) >> ("vm_phys_enq_range: page %p has unexpected order %d", >> m, m->order)); >> order =3D ffs(npages) - 1; >> - vm_phys_enq_chunk(fl, m, order, tail); >> + vm_phys_enq_chunk(fl, m, order, pool, tail); >> m +=3D 1 << order; >> npages -=3D 1 << order; >> } >> @@ -796,33 +801,30 @@ vm_phys_enq_range(vm_page_t m, u_int npages, = struct vm_freelist *fl, int tail) >> } >>=20 >> /* >> - * Set the pool for a contiguous, power of two-sized set of physical = pages. >> + * Complete initialization a contiguous, power of two-sized set of = physical >> + * pages. >> * >> * If the pages currently belong to the lazy init pool, then the = corresponding >> * page structures must be initialized. In this case it is assumed = that the >> * first page in the run has already been initialized. >> */ >> static void >> -vm_phys_set_pool(int pool, vm_page_t m, int order) >> +vm_phys_finish_init(vm_page_t m, int order) >> { >> #ifdef VM_FREEPOOL_LAZYINIT >> if (__predict_false(m->pool =3D=3D VM_FREEPOOL_LAZYINIT)) { >> vm_paddr_t pa; >> int segind; >>=20 >> - m->pool =3D pool; >> - >> TSENTER(); >> pa =3D m->phys_addr + PAGE_SIZE; >> segind =3D m->segind; >> for (vm_page_t m_tmp =3D m + 1; m_tmp < &m[1 << order]; >> m_tmp++, pa +=3D PAGE_SIZE) >> - vm_page_init_page(m_tmp, pa, segind, pool); >> + vm_page_init_page(m_tmp, pa, segind, VM_NFREEPOOL); >> TSEXIT(); >> - } else >> + } >> #endif >> - for (vm_page_t m_tmp =3D m; m_tmp < &m[1 << order]; m_tmp++) >> - m_tmp->pool =3D pool; >> } >>=20 >> /* >> @@ -833,7 +835,8 @@ vm_phys_set_pool(int pool, vm_page_t m, int = order) >> * The returned pages may not be physically contiguous. However, in = contrast >> * to performing multiple, back-to-back calls to = vm_phys_alloc_pages(..., 0), >> * calling this function once to allocate the desired number of pages = will >> - * avoid wasted time in vm_phys_split_pages(). >> + * avoid wasted time in vm_phys_split_pages(). The allocated pages = have no >> + * valid pool field set. >> * >> * The free page queues for the specified domain must be locked. >> */ >> @@ -869,7 +872,8 @@ vm_phys_alloc_npages(int domain, int pool, int = npages, vm_page_t ma[]) >> * Return excess pages to fl. Its order >> * [0, oind) queues are empty. >> */ >> - vm_phys_enq_range(m, avail - i, fl, 1); >> + vm_phys_enq_range(m, avail - i, fl, >> + pool, 1); >> return (npages); >> } >> } >> @@ -881,7 +885,7 @@ vm_phys_alloc_npages(int domain, int pool, int = npages, vm_page_t ma[]) >> while ((m =3D TAILQ_FIRST(&alt[oind].pl)) !=3D >> NULL) { >> vm_freelist_rem(alt, m, oind); >> - vm_phys_set_pool(pool, m, oind); >> + vm_phys_finish_init(m, oind); >> avail =3D i + (1 << oind); >> end =3D imin(npages, avail); >> while (i < end) >> @@ -893,7 +897,7 @@ vm_phys_alloc_npages(int domain, int pool, int = npages, vm_page_t ma[]) >> * are empty. >> */ >> vm_phys_enq_range(m, avail - i, >> - fl, 1); >> + fl, pool, 1); >> return (npages); >> } >> } >> @@ -940,7 +944,7 @@ vm_phys_alloc_freelist_pages(int domain, int = freelist, int pool, int order) >> if (m !=3D NULL) { >> vm_freelist_rem(fl, m, oind); >> /* The order [order, oind) queues are empty. */ >> - vm_phys_split_pages(m, oind, fl, order, 1); >> + vm_phys_split_pages(m, oind, fl, order, pool, 1); >> return (m); >> } >> } >> @@ -957,9 +961,9 @@ vm_phys_alloc_freelist_pages(int domain, int = freelist, int pool, int order) >> m =3D TAILQ_FIRST(&alt[oind].pl); >> if (m !=3D NULL) { >> vm_freelist_rem(alt, m, oind); >> - vm_phys_set_pool(pool, m, oind); >> + vm_phys_finish_init(m, oind); >> /* The order [order, oind) queues are empty. */ >> - vm_phys_split_pages(m, oind, fl, order, 1); >> + vm_phys_split_pages(m, oind, fl, order, pool, 1); >> return (m); >> } >> } >> @@ -1198,11 +1202,12 @@ vm_phys_fictitious_unreg_range(vm_paddr_t = start, vm_paddr_t end) >>=20 >> /* >> * Free a contiguous, power of two-sized set of physical pages. >> + * The pool field in the first page determines the destination pool. >> * >> * The free page queues must be locked. >> */ >> void >> -vm_phys_free_pages(vm_page_t m, int order) >> +vm_phys_free_pages(vm_page_t m, int pool, int order) >> { >> struct vm_freelist *fl; >> struct vm_phys_seg *seg; >> @@ -1210,13 +1215,12 @@ vm_phys_free_pages(vm_page_t m, int order) >> vm_page_t m_buddy; >>=20 >> KASSERT(m->order =3D=3D VM_NFREEORDER, >> - ("vm_phys_free_pages: page %p has unexpected order %d", >> - m, m->order)); >> - KASSERT(vm_phys_pool_valid(m->pool), >> - ("vm_phys_free_pages: page %p has unexpected pool %d", >> - m, m->pool)); >> + ("%s: page %p has unexpected order %d", >> + __func__, m, m->order)); >> + KASSERT(vm_phys_pool_valid(pool), >> + ("%s: unexpected pool param %d", __func__, pool)); >> KASSERT(order < VM_NFREEORDER, >> - ("vm_phys_free_pages: order %d is out of range", order)); >> + ("%s: order %d is out of range", __func__, order)); >> seg =3D &vm_phys_segs[m->segind]; >> vm_domain_free_assert_locked(VM_DOMAIN(seg->domain)); >> if (order < VM_NFREEORDER - 1) { >> @@ -1230,15 +1234,14 @@ vm_phys_free_pages(vm_page_t m, int order) >> break; >> fl =3D (*seg->free_queues)[m_buddy->pool]; >> vm_freelist_rem(fl, m_buddy, order); >> - if (m_buddy->pool !=3D m->pool) >> - vm_phys_set_pool(m->pool, m_buddy, order); >> + vm_phys_finish_init(m_buddy, order); >> order++; >> pa &=3D ~(((vm_paddr_t)1 << (PAGE_SHIFT + order)) - 1); >> m =3D vm_phys_seg_paddr_to_vm_page(seg, pa); >> } while (order < VM_NFREEORDER - 1); >> } >> - fl =3D (*seg->free_queues)[m->pool]; >> - vm_freelist_add(fl, m, order, 1); >> + fl =3D (*seg->free_queues)[pool]; >> + vm_freelist_add(fl, m, order, pool, 1); >> } >>=20 >> #ifdef VM_FREEPOOL_LAZYINIT >> @@ -1290,12 +1293,13 @@ vm_phys_lazy_init_domain(int domain, bool = locked) >> VM_ALLOC_NORMAL, 1 << oind); >> if (unlocked) >> vm_domain_free_unlock(vmd); >> - vm_phys_set_pool(VM_FREEPOOL_DEFAULT, m, oind); >> + vm_phys_finish_init(m, oind); >> if (unlocked) { >> vm_domain_freecnt_inc(vmd, 1 << oind); >> vm_domain_free_lock(vmd); >> } >> - vm_phys_free_pages(m, oind); >> + vm_phys_free_pages(m, VM_FREEPOOL_DEFAULT, >> + oind); >> } >> } >> } >> @@ -1344,12 +1348,12 @@ SYSINIT(vm_phys_lazy_init, SI_SUB_SMP, = SI_ORDER_ANY, vm_phys_lazy_sysinit, >>=20 >> /* >> * Free a contiguous, arbitrarily sized set of physical pages, = without >> - * merging across set boundaries. >> + * merging across set boundaries. Assumes no pages have a valid = pool field. >> * >> * The free page queues must be locked. >> */ >> void >> -vm_phys_enqueue_contig(vm_page_t m, u_long npages) >> +vm_phys_enqueue_contig(vm_page_t m, int pool, u_long npages) >> { >> struct vm_freelist *fl; >> struct vm_phys_seg *seg; >> @@ -1363,14 +1367,15 @@ vm_phys_enqueue_contig(vm_page_t m, u_long = npages) >> */ >> vm_domain_free_assert_locked(vm_pagequeue_domain(m)); >> seg =3D &vm_phys_segs[m->segind]; >> - fl =3D (*seg->free_queues)[m->pool]; >> + fl =3D (*seg->free_queues)[pool]; >> m_end =3D m + npages; >> /* Free blocks of increasing size. */ >> lo =3D atop(VM_PAGE_TO_PHYS(m)); >> if (m < m_end && >> (diff =3D lo ^ (lo + npages - 1)) !=3D 0) { >> order =3D min(ilog2(diff), VM_NFREEORDER - 1); >> - m =3D vm_phys_enq_range(m, roundup2(lo, 1 << order) - lo, fl, 1); >> + m =3D vm_phys_enq_range(m, roundup2(lo, 1 << order) - lo, fl, >> + pool, 1); >> } >>=20 >> /* Free blocks of maximum size. */ >> @@ -1379,20 +1384,22 @@ vm_phys_enqueue_contig(vm_page_t m, u_long = npages) >> KASSERT(seg =3D=3D &vm_phys_segs[m->segind], >> ("%s: page range [%p,%p) spans multiple segments", >> __func__, m_end - npages, m)); >> - vm_phys_enq_chunk(fl, m, order, 1); >> + vm_phys_enq_chunk(fl, m, order, pool, 1); >> m +=3D 1 << order; >> } >> /* Free blocks of diminishing size. */ >> - vm_phys_enq_beg(m, m_end - m, fl, 1); >> + vm_phys_enq_beg(m, m_end - m, fl, pool, 1); >> } >>=20 >> /* >> * Free a contiguous, arbitrarily sized set of physical pages. >> + * Assumes that every page but the first has no valid pool field. >> + * Uses the pool value in the first page if valid, otherwise = default. >> * >> * The free page queues must be locked. >> */ >> void >> -vm_phys_free_contig(vm_page_t m, u_long npages) >> +vm_phys_free_contig(vm_page_t m, int pool, u_long npages) >> { >> vm_paddr_t lo; >> vm_page_t m_start, m_end; >> @@ -1416,11 +1423,11 @@ vm_phys_free_contig(vm_page_t m, u_long = npages) >> * end of the range last. >> */ >> if (m_start < m_end) >> - vm_phys_enqueue_contig(m_start, m_end - m_start); >> + vm_phys_enqueue_contig(m_start, pool, m_end - m_start); >> if (order_start < max_order) >> - vm_phys_free_pages(m, order_start); >> + vm_phys_free_pages(m, pool, order_start); >> if (order_end < max_order) >> - vm_phys_free_pages(m_end, order_end); >> + vm_phys_free_pages(m_end, pool, order_end); >> } >>=20 >> /* >> @@ -1474,15 +1481,15 @@ vm_phys_unfree_page(vm_paddr_t pa) >> struct vm_phys_seg *seg; >> vm_paddr_t pa_half; >> vm_page_t m, m_set, m_tmp; >> - int order; >> + int order, pool; >>=20 >> seg =3D vm_phys_paddr_to_seg(pa); >> vm_domain_free_assert_locked(VM_DOMAIN(seg->domain)); >>=20 >> +#ifdef VM_FREEPOOL_LAZYINIT >> /* >> * The pages on the free lists must be initialized. >> */ >> -#ifdef VM_FREEPOOL_LAZYINIT >> vm_phys_lazy_init_domain(seg->domain, true); >> #endif >>=20 >> @@ -1515,7 +1522,8 @@ vm_phys_unfree_page(vm_paddr_t pa) >> * is larger than a page, shrink "m_set" by returning the half >> * of "m_set" that does not contain "m" to the free lists. >> */ >> - fl =3D (*seg->free_queues)[m_set->pool]; >> + pool =3D m_set->pool; >> + fl =3D (*seg->free_queues)[pool]; >> order =3D m_set->order; >> vm_freelist_rem(fl, m_set, order); >> while (order > 0) { >> @@ -1527,7 +1535,7 @@ vm_phys_unfree_page(vm_paddr_t pa) >> m_tmp =3D m_set; >> m_set =3D vm_phys_seg_paddr_to_vm_page(seg, pa_half); >> } >> - vm_freelist_add(fl, m_tmp, order, 0); >> + vm_freelist_add(fl, m_tmp, order, pool, 0); >> } >> KASSERT(m_set =3D=3D m, ("vm_phys_unfree_page: fatal = inconsistency")); >> return (true); >> @@ -1668,7 +1676,8 @@ vm_phys_find_queues_contig( >> * alignment of the first physical page in the set. If the given = value >> * "boundary" is non-zero, then the set of physical pages cannot = cross >> * any physical address boundary that is a multiple of that value. = Both >> - * "alignment" and "boundary" must be a power of two. >> + * "alignment" and "boundary" must be a power of two. Sets the pool >> + * field to DEFAULT in the first allocated page. >> */ >> vm_page_t >> vm_phys_alloc_contig(int domain, u_long npages, vm_paddr_t low, = vm_paddr_t high, >> @@ -1727,12 +1736,12 @@ vm_phys_alloc_contig(int domain, u_long = npages, vm_paddr_t low, vm_paddr_t high, >> fl =3D (*queues)[m->pool]; >> oind =3D m->order; >> vm_freelist_rem(fl, m, oind); >> - if (m->pool !=3D VM_FREEPOOL_DEFAULT) >> - vm_phys_set_pool(VM_FREEPOOL_DEFAULT, m, oind); >> + vm_phys_finish_init(m, oind); >> } >> /* Return excess pages to the free lists. */ >> fl =3D (*queues)[VM_FREEPOOL_DEFAULT]; >> - vm_phys_enq_range(&m_run[npages], m - &m_run[npages], fl, 0); >> + vm_phys_enq_range(&m_run[npages], m - &m_run[npages], fl, >> + VM_FREEPOOL_DEFAULT, 0); >>=20 >> /* Return page verified to satisfy conditions of request. */ >> pa_start =3D VM_PAGE_TO_PHYS(m_run); >> diff --git a/sys/vm/vm_phys.h b/sys/vm/vm_phys.h >> index 43d94a9420f2..046fe26b476d 100644 >> --- a/sys/vm/vm_phys.h >> +++ b/sys/vm/vm_phys.h >> @@ -64,15 +64,15 @@ vm_page_t vm_phys_alloc_contig(int domain, u_long = npages, vm_paddr_t low, >> int vm_phys_alloc_npages(int domain, int pool, int npages, vm_page_t = ma[]); >> vm_page_t vm_phys_alloc_pages(int domain, int pool, int order); >> int vm_phys_domain_match(int prefer, vm_paddr_t low, vm_paddr_t = high); >> -void vm_phys_enqueue_contig(vm_page_t m, u_long npages); >> +void vm_phys_enqueue_contig(vm_page_t m, int pool, u_long npages); >> int vm_phys_fictitious_reg_range(vm_paddr_t start, vm_paddr_t end, >> vm_memattr_t memattr); >> void vm_phys_fictitious_unreg_range(vm_paddr_t start, vm_paddr_t = end); >> vm_page_t vm_phys_fictitious_to_vm_page(vm_paddr_t pa); >> int vm_phys_find_range(vm_page_t bounds[], int segind, int domain, >> u_long npages, vm_paddr_t low, vm_paddr_t high); >> -void vm_phys_free_contig(vm_page_t m, u_long npages); >> -void vm_phys_free_pages(vm_page_t m, int order); >> +void vm_phys_free_contig(vm_page_t m, int pool, u_long npages); >> +void vm_phys_free_pages(vm_page_t m, int pool, int order); >> void vm_phys_init(void); >> vm_page_t vm_phys_paddr_to_vm_page(vm_paddr_t pa); >> vm_page_t vm_phys_seg_paddr_to_vm_page(struct vm_phys_seg *seg, = vm_paddr_t pa); >> diff --git a/sys/vm/vm_reserv.c b/sys/vm/vm_reserv.c >> index ad513962d50d..04453e050a38 100644 >> --- a/sys/vm/vm_reserv.c >> +++ b/sys/vm/vm_reserv.c >> @@ -480,7 +480,7 @@ vm_reserv_depopulate(vm_reserv_t rv, int index) >> if (rv->popcnt =3D=3D 0) { >> vm_reserv_remove(rv); >> vm_domain_free_lock(vmd); >> - vm_phys_free_pages(rv->pages, VM_LEVEL_0_ORDER); >> + vm_phys_free_pages(rv->pages, rv->pages->pool, VM_LEVEL_0_ORDER); >> vm_domain_free_unlock(vmd); >> counter_u64_add(vm_reserv_freed, 1); >> } >> @@ -943,7 +943,7 @@ static void >> vm_reserv_break(vm_reserv_t rv) >> { >> vm_page_t m; >> - int hi, lo, pos; >> + int pos, pos0, pos1; >>=20 >> vm_reserv_assert_locked(rv); >> CTR5(KTR_VM, "%s: rv %p object %p popcnt %d inpartpop %d", >> @@ -954,23 +954,24 @@ vm_reserv_break(vm_reserv_t rv) >> for (; m < rv->pages + VM_LEVEL_0_NPAGES; m +=3D = VM_SUBLEVEL_0_NPAGES) >> #endif >> m->psind =3D 0; >> - hi =3D lo =3D -1; >> - pos =3D 0; >> - for (;;) { >> - bit_ff_at(rv->popmap, pos, VM_LEVEL_0_NPAGES, lo !=3D hi, &pos); >> - if (lo =3D=3D hi) { >> - if (pos =3D=3D -1) >> - break; >> - lo =3D pos; >> - continue; >> - } >> + pos0 =3D bit_test(rv->popmap, 0) ? -1 : 0; >> + pos1 =3D -1 - pos0; >> + for (pos =3D 0; pos < VM_LEVEL_0_NPAGES; ) { >> + /* Find the first different bit after pos. */ >> + bit_ff_at(rv->popmap, pos + 1, VM_LEVEL_0_NPAGES, >> + pos1 < pos0, &pos); >> if (pos =3D=3D -1) >> pos =3D VM_LEVEL_0_NPAGES; >> - hi =3D pos; >> + if (pos0 < pos1) { >> + pos0 =3D pos; >> + continue; >> + } >> + /* Free unused pages from pos0 to pos. */ >> + pos1 =3D pos; >> vm_domain_free_lock(VM_DOMAIN(rv->domain)); >> - vm_phys_enqueue_contig(&rv->pages[lo], hi - lo); >> + vm_phys_enqueue_contig(&rv->pages[pos0], VM_FREEPOOL_DEFAULT, >> + pos1 - pos0); >> vm_domain_free_unlock(VM_DOMAIN(rv->domain)); >> - lo =3D hi; >> } >> bit_nclear(rv->popmap, 0, VM_LEVEL_0_NPAGES - 1); >> rv->popcnt =3D 0; >=20 Jess