From owner-svn-src-all@FreeBSD.ORG Sat May 9 20:08:38 2015 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0FEECE57; Sat, 9 May 2015 20:08:38 +0000 (UTC) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F281B1D62; Sat, 9 May 2015 20:08:37 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id t49K8bbx096503; Sat, 9 May 2015 20:08:37 GMT (envelope-from kib@FreeBSD.org) Received: (from kib@localhost) by svn.freebsd.org (8.14.9/8.14.9/Submit) id t49K8a5B096499; Sat, 9 May 2015 20:08:36 GMT (envelope-from kib@FreeBSD.org) Message-Id: <201505092008.t49K8a5B096499@svn.freebsd.org> X-Authentication-Warning: svn.freebsd.org: kib set sender to kib@FreeBSD.org using -f From: Konstantin Belousov Date: Sat, 9 May 2015 20:08:36 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r282690 - in head/sys: kern vm X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 May 2015 20:08:38 -0000 Author: kib Date: Sat May 9 20:08:36 2015 New Revision: 282690 URL: https://svnweb.freebsd.org/changeset/base/282690 Log: The vmem callback to reclaim kmem arena address space on low or fragmented conditions currently just wakes up the pagedaemon. The kmem arena is significantly smaller then the total available physical memory, which means that there are loads where kmem arena space could be exhausted, while there is a lot of pages available still. The woken up pagedaemon sees vm_pages_needed != 0, verifies the condition vm_paging_needed() which is false, clears the pass and returns back to sleep, not calling neither uma_reclaim() nor lowmem handler. To handle low kmem arena conditions, create additional pagedaemon thread which calls uma_reclaim() directly. The thread sleeps on the dedicated channel and kmem_reclaim() wakes the thread in addition to the pagedaemon. Reported and tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Modified: head/sys/kern/kern_malloc.c head/sys/vm/uma.h head/sys/vm/uma_core.c head/sys/vm/vm_pageout.c Modified: head/sys/kern/kern_malloc.c ============================================================================== --- head/sys/kern/kern_malloc.c Sat May 9 19:43:48 2015 (r282689) +++ head/sys/kern/kern_malloc.c Sat May 9 20:08:36 2015 (r282690) @@ -665,13 +665,15 @@ reallocf(void *addr, unsigned long size, } /* - * Wake the page daemon when we exhaust KVA. It will call the lowmem handler - * and uma_reclaim() callbacks in a context that is safe. + * Wake the uma reclamation pagedaemon thread when we exhaust KVA. It + * will call the lowmem handler and uma_reclaim() callbacks in a + * context that is safe. */ static void kmem_reclaim(vmem_t *vm, int flags) { + uma_reclaim_wakeup(); pagedaemon_wakeup(); } Modified: head/sys/vm/uma.h ============================================================================== --- head/sys/vm/uma.h Sat May 9 19:43:48 2015 (r282689) +++ head/sys/vm/uma.h Sat May 9 20:08:36 2015 (r282690) @@ -690,4 +690,7 @@ struct uma_percpu_stat { uint64_t _ups_reserved[5]; /* Reserved. */ }; +void uma_reclaim_wakeup(void); +void uma_reclaim_worker(void *); + #endif /* _VM_UMA_H_ */ Modified: head/sys/vm/uma_core.c ============================================================================== --- head/sys/vm/uma_core.c Sat May 9 19:43:48 2015 (r282689) +++ head/sys/vm/uma_core.c Sat May 9 20:08:36 2015 (r282690) @@ -3222,16 +3222,17 @@ uma_find_refcnt(uma_zone_t zone, void *i } /* See uma.h */ -void -uma_reclaim(void) +static void +uma_reclaim_locked(bool kmem_danger) { + #ifdef UMA_DEBUG printf("UMA: vm asked us to release pages!\n"); #endif - sx_xlock(&uma_drain_lock); + sx_assert(&uma_drain_lock, SA_XLOCKED); bucket_enable(); zone_foreach(zone_drain); - if (vm_page_count_min()) { + if (vm_page_count_min() || kmem_danger) { cache_drain_safe(NULL); zone_foreach(zone_drain); } @@ -3243,9 +3244,42 @@ uma_reclaim(void) zone_drain(slabzone); zone_drain(slabrefzone); bucket_zone_drain(); +} + +void +uma_reclaim(void) +{ + + sx_xlock(&uma_drain_lock); + uma_reclaim_locked(false); sx_xunlock(&uma_drain_lock); } +static int uma_reclaim_needed; + +void +uma_reclaim_wakeup(void) +{ + + uma_reclaim_needed = 1; + wakeup(&uma_reclaim_needed); +} + +void +uma_reclaim_worker(void *arg __unused) +{ + + sx_xlock(&uma_drain_lock); + for (;;) { + sx_sleep(&uma_reclaim_needed, &uma_drain_lock, PVM, + "umarcl", 0); + if (uma_reclaim_needed) { + uma_reclaim_needed = 0; + uma_reclaim_locked(true); + } + } +} + /* See uma.h */ int uma_zone_exhausted(uma_zone_t zone) Modified: head/sys/vm/vm_pageout.c ============================================================================== --- head/sys/vm/vm_pageout.c Sat May 9 19:43:48 2015 (r282689) +++ head/sys/vm/vm_pageout.c Sat May 9 20:08:36 2015 (r282690) @@ -1724,8 +1724,9 @@ vm_pageout_init(void) static void vm_pageout(void) { + int error; #if MAXMEMDOM > 1 - int error, i; + int i; #endif swap_pager_swap_init(); @@ -1739,6 +1740,10 @@ vm_pageout(void) } } #endif + error = kthread_add(uma_reclaim_worker, NULL, curproc, NULL, + 0, 0, "uma"); + if (error != 0) + panic("starting uma_reclaim helper, error %d\n", error); vm_pageout_worker((void *)(uintptr_t)0); }