From owner-freebsd-bugs@FreeBSD.ORG Sat Dec 22 19:00:00 2012 Return-Path: Delivered-To: freebsd-bugs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EDD96F7C for ; Sat, 22 Dec 2012 19:00:00 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id B9B448FC12 for ; Sat, 22 Dec 2012 19:00:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id qBMJ00rc043768 for ; Sat, 22 Dec 2012 19:00:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id qBMJ00g6043767; Sat, 22 Dec 2012 19:00:00 GMT (envelope-from gnats) Resent-Date: Sat, 22 Dec 2012 19:00:00 GMT Resent-Message-Id: <201212221900.qBMJ00g6043767@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Ian Lepore Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5BCC4D2B for ; Sat, 22 Dec 2012 18:57:05 +0000 (UTC) (envelope-from ilepore@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 28D468FC0C for ; Sat, 22 Dec 2012 18:57:04 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id qBMIv4mT052217 for ; Sat, 22 Dec 2012 11:57:04 -0700 (MST) (envelope-from ilepore@damnhippie.dyndns.org) Received: from revolution.hippie.lan (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id qBMIv1rZ068992 for ; Sat, 22 Dec 2012 11:57:02 -0700 (MST) (envelope-from ilepore@damnhippie.dyndns.org) Received: (from ilepore@localhost) by revolution.hippie.lan (8.14.5/8.14.4/Submit) id qBMIv1MC047004; Sat, 22 Dec 2012 11:57:01 -0700 (MST) (envelope-from ilepore) Message-Id: <201212221857.qBMIv1MC047004@revolution.hippie.lan> Date: Sat, 22 Dec 2012 11:57:01 -0700 (MST) From: Ian Lepore To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Subject: bin/174641: [patch] jemalloc enhancement: allow MI request for min-sized chunks X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Ian Lepore List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Dec 2012 19:00:01 -0000 >Number: 174641 >Category: bin >Synopsis: [patch] jemalloc enhancement: allow MI request for min-sized chunks >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: change-request >Submitter-Id: current-users >Arrival-Date: Sat Dec 22 19:00:00 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Ian Lepore >Release: FreeBSD 10.0-CURRENT arm >Organization: Symmetricom, Inc. >Environment: FreeBSD dpcur 10.0-CURRENT FreeBSD 10.0-CURRENT #2 r244488M: Fri Dec 21 11:57:58 MST 2012 root@revolution.hippie.lan:/local/build/staging/freebsd/dp10/obj/arm.arm/local/build/staging/freebsd/dp10/src/sys/DP-1001N arm >Description: Programs which use mlockall(2), such as watchdogd, wire down physical pages for all allocated vmspace. jemalloc allocates large chunks of vmspace by default for efficiency. A daemon such as watchdogd can accidentally use up all available ram on a small embedded system unless it tunes the lg_chunk value down as small as possible. A problem arises because the minimum size that can be configured for the lg_chunk option is machine-dependent (it's based on PAGE_SIZE) and it also must be large enough to satisfy some of jemalloc's internal requirements which can't be known by the program. The tuning must be done before the first allocation (before entry to main()), precluding the possibilty of using runtime logic to determine the smallest legal value. The attached patch enhances jemalloc's handling of the lg_chunk option: - Requesting a value of zero asks jemalloc to quietly set the chunk size to the smallest value allowable for the hardware it's running on. - Requesting a non-zero size smaller than the minimum will result in using the minimum size and emitting a warning about clipping the value. It also changes watchdogd to use this new feature. >How-To-Repeat: >Fix: --- jemalloc_lgchunk.diff begins here --- diff -r 28b12492cd58 usr.sbin/watchdogd/watchdogd.c --- a/usr.sbin/watchdogd/watchdogd.c Fri Dec 21 11:20:58 2012 -0700 +++ b/usr.sbin/watchdogd/watchdogd.c Sat Dec 22 11:07:13 2012 -0700 @@ -71,6 +71,14 @@ static int nap = 1; static char *test_cmd = NULL; /* + * Ask malloc to map minimum-sized chunks of virtual address space at a time, so + * that mlockall() won't needlessly wire megabytes of unused memory into the + * process. This must be done using the malloc_conf string so that it gets set + * up before the first allocation (before entry to main()). + */ +const char * malloc_conf = "lg_chunk:0"; + +/* * Periodically pat the watchdog, preventing it from firing. */ int diff -r 28b12492cd58 contrib/jemalloc/src/jemalloc.c --- a/contrib/jemalloc/src/jemalloc.c Fri Dec 21 11:20:58 2012 -0700 +++ b/contrib/jemalloc/src/jemalloc.c Sat Dec 22 11:07:27 2012 -0700 @@ -474,9 +474,10 @@ malloc_conf_init(void) while (*opts != '\0' && malloc_conf_next(&opts, &k, &klen, &v, &vlen) == false) { +#define CONF_KEY_MATCHES(n) \ + (sizeof(n)-1 == klen && strncmp(n, k, klen) == 0) #define CONF_HANDLE_BOOL_HIT(o, n, hit) \ - if (sizeof(n)-1 == klen && strncmp(n, k, \ - klen) == 0) { \ + if (CONF_KEY_MATCHES(n)) { \ if (strncmp("true", v, vlen) == 0 && \ vlen == sizeof("true")-1) \ o = true; \ @@ -498,8 +499,7 @@ malloc_conf_init(void) continue; \ } #define CONF_HANDLE_SIZE_T(o, n, min, max) \ - if (sizeof(n)-1 == klen && strncmp(n, k, \ - klen) == 0) { \ + if (CONF_KEY_MATCHES(n)) { \ uintmax_t um; \ char *end; \ \ @@ -519,8 +519,7 @@ malloc_conf_init(void) continue; \ } #define CONF_HANDLE_SSIZE_T(o, n, min, max) \ - if (sizeof(n)-1 == klen && strncmp(n, k, \ - klen) == 0) { \ + if (CONF_KEY_MATCHES(n)) { \ long l; \ char *end; \ \ @@ -541,8 +540,7 @@ malloc_conf_init(void) continue; \ } #define CONF_HANDLE_CHAR_P(o, n, d) \ - if (sizeof(n)-1 == klen && strncmp(n, k, \ - klen) == 0) { \ + if (CONF_KEY_MATCHES(n)) { \ size_t cpylen = (vlen <= \ sizeof(o)-1) ? vlen : \ sizeof(o)-1; \ @@ -557,11 +555,41 @@ malloc_conf_init(void) * one data page in the absence of redzones, or three * pages in the presence of redzones. In order to * simplify options processing, fix the limit based on - * config_fill. + * config_fill. The limit is based on page size and our + * private internal requirements, so be liberal: If the + * value is zero, quietly use the minimum; if it's + * non-zero but less than the minimum, warn and clip the + * value to the minimum. */ - CONF_HANDLE_SIZE_T(opt_lg_chunk, "lg_chunk", LG_PAGE + - (config_fill ? 2 : 1), (sizeof(size_t) << 3) - 1) - if (strncmp("dss", k, klen) == 0) { + if (CONF_KEY_MATCHES("lg_chunk")) { + uintmax_t min, max, um; + char *end; + + min = LG_PAGE + (config_fill ? 2 : 1); + max = (sizeof(size_t) << 3) - 1; + set_errno(0); + um = malloc_strtoumax(v, &end, 0); + if (get_errno() != 0 || (uintptr_t)end - + (uintptr_t)v != vlen) { + malloc_conf_error( + "Invalid conf value", + k, klen, v, vlen); + } else if (um > max) { + malloc_conf_error( + "Out-of-range conf value", + k, klen, v, vlen); + } else if (um < min) { + if (um != 0) + malloc_conf_error( + "Out-of-range conf value " + "clipped to min", + k, klen, v, vlen); + opt_lg_chunk = min; + } else + opt_lg_chunk = um; + continue; + } + if (CONF_KEY_MATCHES("dss")) { int i; bool match = false; for (i = 0; i < dss_prec_limit; i++) { diff -r 28b12492cd58 contrib/jemalloc/doc/jemalloc.3 --- a/contrib/jemalloc/doc/jemalloc.3 Fri Dec 21 11:20:58 2012 -0700 +++ b/contrib/jemalloc/doc/jemalloc.3 Sat Dec 22 11:07:33 2012 -0700 @@ -653,6 +653,9 @@ is specified during configuration, in wh "opt\&.lg_chunk" (\fBsize_t\fR) r\- .RS 4 Virtual memory chunk size (log base 2)\&. The default chunk size is 4 MiB (2^22)\&. +The minimum size is based on the host's page size and various internal requirements\&. +If the requested size is zero, the minimum size is quietly used, otherwise if the requested +size is smaller than the minimum it is clipped to the minimum and a warning is printed\&. .RE .PP "opt\&.dss" (\fBconst char *\fR) r\- --- jemalloc_lgchunk.diff ends here --- >Release-Note: >Audit-Trail: >Unformatted: