Date: Mon, 23 May 2016 06:04:38 +0000 (UTC) From: Mark Johnston <markj@FreeBSD.org> To: src-committers@freebsd.org, svn-src-user@freebsd.org Subject: svn commit: r300476 - in user/alc/PQ_LAUNDRY/sys: sys vm Message-ID: <201605230604.u4N64cOG001232@repo.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: markj Date: Mon May 23 06:04:38 2016 New Revision: 300476 URL: https://svnweb.freebsd.org/changeset/base/300476 Log: Add a basic laundering policy. This policy stems from the notion that there are two reasons to launder pages: 1. Shortfall, in which the inactive and free queues are depleted, and the system must launder dirty pages in order to reclaim memory. 2. Fairness: the system should periodically launder dirty pages to ensure that applications cannot excessively influence the system's memory reclaimation behaviour. Note that this does not imply that clear and dirty pages must be treated equally: page laundering is an expensive operation. However, the relative costs of reclaiming a clean vs. dirty page should be bounded in some well-defined way, and in particular, it should not be possible to force the system to reclaim only clean pages indefinitely. Under memory pressure the system should eventually launder some dirty pages, even when inactive clean pages are plentiful. Thus, laundering targets are chosen based on the current state of the paging queues. In shortfall, the laundry thread attempts to meet the shortfall within 0.5s, the pagedaemon sleep period. Because it is the sole source of clean pages, no attempts are made to limit the laundering rate: the laundry thread goes all-out. If the system is not in shortfall, the laundry thread may elect to launder some dirty pages in an attempt to satisfy the fairness policy. This is referred to as background laundering. Several conditions must be met for background laundering to occur: a) The laundry queue must contain a significant amount of the system's inactive memory: if the number of dirty pages is miniscule, nothing is gained by laundering them. Moreover, write clustering works better if the number of dirty pages is allowed to grow to some threshold before any laundering is performed. The ratio of clean to dirty pages serves as a threshold here, controlled by bkgrd_launder_ratio. By default, dirty pages must constitute at least 1% of inactive memory for background laundering to occur. b) The number of free pages must be low. If there is plentiful free memory, there's no reason to launder pages. The number of free pages must be smaller than bkgrd_launder_thresh for background laundering to occur. By default, this is chosen to be the max of half the free target and 3/2s of the pagedaemon wakeup threshold. The idea is to start laundering before the pagedaemon wakes up. c) The pagedaemon thread(s) must be active. If the number of free pages is low but the system is not under memory pressure, we should not continue background laundering indefinitely. We use vm_cnt.v_pdwakeups as a proxy for pagedaemon activity: when a background laundering run begins, the pdwakeups value is recorded; a second run cannot begin until pdwakeups has been incremented at least once. When the conditions for background laundering are met, the laundry thread determines the target number of pages and begins laundering. It attempts to meet the target within one second unless the corresponding laundering rate would exceed bkgrd_launder_max (32MB/s by default). The target is given by 0.5*l(L)*FT/l(I), where FT is the free page threshold used by the pagedaemon. In particular, the number of pages laundered is proportional to the ratio of dirty to clean inactive pages. When performing background laundering, the pagedaemon counts reactivated pages towards its target. Reviewed by: alc Modified: user/alc/PQ_LAUNDRY/sys/sys/vmmeter.h user/alc/PQ_LAUNDRY/sys/vm/vm_pageout.c Modified: user/alc/PQ_LAUNDRY/sys/sys/vmmeter.h ============================================================================== --- user/alc/PQ_LAUNDRY/sys/sys/vmmeter.h Mon May 23 06:01:04 2016 (r300475) +++ user/alc/PQ_LAUNDRY/sys/sys/vmmeter.h Mon May 23 06:04:38 2016 (r300476) @@ -185,6 +185,18 @@ vm_paging_needed(void) (u_int)vm_pageout_wakeup_thresh); } +/* + * Return the number of pages we need to launder. + * A positive number indicates that we have a shortfall of clean pages. + */ +static inline int +vm_laundry_target(void) +{ + + return (vm_cnt.v_inactive_target - vm_cnt.v_inactive_count + + vm_paging_target()); +} + #endif /* systemwide totals computed every five seconds */ Modified: user/alc/PQ_LAUNDRY/sys/vm/vm_pageout.c ============================================================================== --- user/alc/PQ_LAUNDRY/sys/vm/vm_pageout.c Mon May 23 06:01:04 2016 (r300475) +++ user/alc/PQ_LAUNDRY/sys/vm/vm_pageout.c Mon May 23 06:04:38 2016 (r300476) @@ -231,6 +231,21 @@ SYSCTL_INT(_vm, OID_AUTO, act_scan_laund CTLFLAG_RW, &act_scan_laundry_weight, 0, "weight given to clean vs. dirty pages in active queue scans"); +static u_int bkgrd_launder_ratio = 100; +SYSCTL_UINT(_vm, OID_AUTO, bkgrd_launder_ratio, + CTLFLAG_RW, &bkgrd_launder_ratio, 0, + "ratio of inactive to laundry pages to trigger background laundering"); + +static u_int bkgrd_launder_max = 32768; +SYSCTL_UINT(_vm, OID_AUTO, bkgrd_launder_max, + CTLFLAG_RW, &bkgrd_launder_max, 0, + "maximum background laundering rate, in pages per second"); + +static u_int bkgrd_launder_thresh; +SYSCTL_UINT(_vm, OID_AUTO, bkgrd_launder_thresh, + CTLFLAG_RW, &bkgrd_launder_thresh, 0, + "free page threshold below which background laundering may be started"); + #define VM_PAGEOUT_PAGE_COUNT 16 int vm_pageout_page_count = VM_PAGEOUT_PAGE_COUNT; @@ -239,7 +254,7 @@ SYSCTL_INT(_vm, OID_AUTO, max_wired, CTLFLAG_RW, &vm_page_max_wired, 0, "System-wide limit to wired page count"); static boolean_t vm_pageout_fallback_object_lock(vm_page_t, vm_page_t *); -static void vm_pageout_launder(struct vm_domain *vmd); +static int vm_pageout_launder(struct vm_domain *vmd, int launder); static void vm_pageout_laundry_worker(void *arg); #if !defined(NO_SWAPPING) static void vm_pageout_map_deactivate_pages(vm_map_t, long); @@ -877,32 +892,21 @@ unlock_mp: } /* - * XXX + * Attempt to launder the specified number of pages. + * + * Returns the number of pages successfully laundered. */ -static void -vm_pageout_launder(struct vm_domain *vmd) +static int +vm_pageout_launder(struct vm_domain *vmd, int launder) { vm_page_t m, next; struct vm_pagequeue *pq; vm_object_t object; - int act_delta, error, launder, maxscan, numpagedout, vnodes_skipped; - boolean_t pageout_ok, queue_locked; - - /* - * Compute the number of pages we want to move from the laundry queue to - * the inactive queue. If there is no shortage of clean, inactive - * pages, we allow laundering to proceed at a trickle to ensure that - * dirty pages will eventually be reused. Otherwise, the inactive queue - * target is scaled by the ratio of the sleep intervals of the laundry - * queue and inactive queue worker threads. - */ - launder = vm_cnt.v_inactive_target - vm_cnt.v_inactive_count + - vm_paging_target() + vm_pageout_deficit; - if (launder < 0) - launder = 1; - else - launder /= VM_LAUNDER_RATE; + int act_delta, error, maxscan, numpagedout, starting_target; + int vnodes_skipped; + boolean_t pageout_ok, queue_locked, shortfall; + starting_target = launder; vnodes_skipped = 0; /* @@ -917,6 +921,8 @@ vm_pageout_launder(struct vm_domain *vmd */ pq = &vmd->vmd_pagequeues[PQ_LAUNDRY]; maxscan = pq->pq_cnt; + shortfall = vm_laundry_target() > 0; + vm_pagequeue_lock(pq); queue_locked = TRUE; for (m = TAILQ_FIRST(&pq->pq_pl); @@ -991,6 +997,17 @@ vm_pageout_launder(struct vm_domain *vmd * queue. */ m->act_count += act_delta + ACT_ADVANCE; + + /* + * If this was a background laundering, count + * activated pages towards our target. The + * purpose of background laundering is to ensure + * that pages are eventually cycled through the + * laundry queue, and an activation is a valid + * way out. + */ + if (!shortfall) + launder--; goto drop_page; } else if ((object->flags & OBJ_DEAD) == 0) goto requeue_page; @@ -1064,29 +1081,121 @@ relock_queue: */ if (vnodes_skipped > 0 && launder > 0) (void)speedup_syncer(); + + return (starting_target - launder); } /* - * XXX + * Perform the work of the laundry thread: periodically wake up and determine + * whether any pages need to be laundered. If so, determine the number of pages + * that need to be laundered, and launder them. */ static void vm_pageout_laundry_worker(void *arg) { struct vm_domain *domain; - int domidx; + uint64_t ninact, nlaundry; + int cycle, tcycle, domidx, gen, launder, laundered; + int shortfall, prev_shortfall, target; domidx = (uintptr_t)arg; domain = &vm_dom[domidx]; KASSERT(domain->vmd_segs != 0, ("domain without segments")); vm_pageout_init_marker(&domain->vmd_laundry_marker, PQ_LAUNDRY); + cycle = tcycle = 0; + gen = -1; + shortfall = prev_shortfall = 0; + target = 0; + + if (bkgrd_launder_thresh == 0) + bkgrd_launder_thresh = max(vm_cnt.v_free_target / 2, + 3 * vm_pageout_wakeup_thresh / 2); + /* * The pageout laundry worker is never done, so loop forever. */ for (;;) { + KASSERT(target >= 0, ("negative target %d", target)); + launder = 0; + + /* + * First determine whether we're in shortfall. If so, there's + * an impending need for clean pages. We attempt to launder the + * target within one pagedaemon sleep period. + */ + shortfall = vm_laundry_target() + vm_pageout_deficit; + if (shortfall > 0) { + /* + * If the shortfall has grown since the last cycle or + * we're still in shortfall despite a previous + * laundering run, start a new run. + */ + if (shortfall > prev_shortfall || cycle == tcycle) { + target = shortfall; + cycle = 0; + tcycle = VM_LAUNDER_RATE; + } + prev_shortfall = shortfall; + launder = target / (tcycle - (cycle % tcycle)); + goto launder; + } else { + if (prev_shortfall > 0) + /* We're out of shortfall; the target is met. */ + target = 0; + shortfall = prev_shortfall = 0; + } + + /* + * There's no immediate need to launder any pages; see if we + * meet the conditions to perform background laundering: + * + * 1. we haven't yet reached the target of the current + * background laundering run, or + * 2. the ratio of dirty to clean inactive pages exceeds the + * background laundering threshold and the free page count is + * low. + * + * We don't start a new background laundering run unless the + * pagedaemon has been woken up at least once since the previous + * run. + */ + if (target > 0 && cycle != tcycle) { + /* Continue an ongoing background run. */ + launder = target / (tcycle - (cycle % tcycle)); + goto launder; + } + + ninact = vm_cnt.v_inactive_count; + nlaundry = vm_cnt.v_laundry_count; + if (ninact > 0 && + vm_cnt.v_pdwakeups != gen && + vm_cnt.v_free_count < bkgrd_launder_thresh && + nlaundry * bkgrd_launder_ratio >= ninact) { + cycle = 0; + tcycle = VM_LAUNDER_INTERVAL; + gen = vm_cnt.v_pdwakeups; + if (nlaundry >= ninact) + target = vm_cnt.v_free_target; + else + target = (nlaundry * vm_cnt.v_free_target << 16) / + ninact >> 16; + target /= 2; + if (target > bkgrd_launder_max) + tcycle = target * VM_LAUNDER_INTERVAL / + bkgrd_launder_max; + launder = target / (tcycle - (cycle % tcycle)); + } + +launder: + if (launder > 0) { + laundered = vm_pageout_launder(domain, launder); + target -= min(laundered, target); + } + tsleep(&vm_cnt.v_laundry_count, PVM, "laundr", hz / VM_LAUNDER_INTERVAL); - vm_pageout_launder(domain); + cycle++; } }
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201605230604.u4N64cOG001232>