Date: Sun, 5 Apr 2020 02:08:17 +0000 (UTC) From: Xin LI <delphij@FreeBSD.org> To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-12@freebsd.org Subject: svn commit: r359636 - in stable/12/contrib/xz/src: common xz Message-ID: <202004050208.03528HYH050368@repo.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: delphij Date: Sun Apr 5 02:08:17 2020 New Revision: 359636 URL: https://svnweb.freebsd.org/changeset/base/359636 Log: MFC r357609: MFV r357608: Limit memory usage in xz(1) instead of in tuklib. Modified: stable/12/contrib/xz/src/common/tuklib_physmem.c stable/12/contrib/xz/src/xz/hardware.c stable/12/contrib/xz/src/xz/xz.1 Directory Properties: stable/12/ (props changed) Modified: stable/12/contrib/xz/src/common/tuklib_physmem.c ============================================================================== --- stable/12/contrib/xz/src/common/tuklib_physmem.c Sun Apr 5 01:53:35 2020 (r359635) +++ stable/12/contrib/xz/src/common/tuklib_physmem.c Sun Apr 5 02:08:17 2020 (r359636) @@ -45,7 +45,6 @@ # include <sys/systemcfg.h> #elif defined(TUKLIB_PHYSMEM_SYSCONF) -# include <limits.h> # include <unistd.h> #elif defined(TUKLIB_PHYSMEM_SYSCTL) @@ -146,16 +145,13 @@ tuklib_physmem(void) #elif defined(TUKLIB_PHYSMEM_SYSCONF) const long pagesize = sysconf(_SC_PAGESIZE); const long pages = sysconf(_SC_PHYS_PAGES); - if (pagesize != -1 && pages != -1) { + if (pagesize != -1 && pages != -1) // According to docs, pagesize * pages can overflow. // Simple case is 32-bit box with 4 GiB or more RAM, // which may report exactly 4 GiB of RAM, and "long" // being 32-bit will overflow. Casting to uint64_t // hopefully avoids overflows in the near future. ret = (uint64_t)pagesize * (uint64_t)pages; - if (ret > SIZE_T_MAX) - ret = SIZE_T_MAX; - } #elif defined(TUKLIB_PHYSMEM_SYSCTL) int name[2] = { Modified: stable/12/contrib/xz/src/xz/hardware.c ============================================================================== --- stable/12/contrib/xz/src/xz/hardware.c Sun Apr 5 01:53:35 2020 (r359635) +++ stable/12/contrib/xz/src/xz/hardware.c Sun Apr 5 02:08:17 2020 (r359636) @@ -68,8 +68,38 @@ hardware_memlimit_set(uint64_t new_memlimit, new_memlimit = (uint32_t)new_memlimit * total_ram / 100; } - if (set_compress) + if (set_compress) { memlimit_compress = new_memlimit; + +#if SIZE_MAX == UINT32_MAX + // FIXME? + // + // When running a 32-bit xz on a system with a lot of RAM and + // using a percentage-based memory limit, the result can be + // bigger than the 32-bit address space. Limiting the limit + // below SIZE_MAX for compression (not decompression) makes + // xz lower the compression settings (or number of threads) + // to a level that *might* work. In practice it has worked + // when using a 64-bit kernel that gives full 4 GiB address + // space to 32-bit programs. In other situations this might + // still be too high, like 32-bit kernels that may give much + // less than 4 GiB to a single application. + // + // So this is an ugly hack but I will keep it here while + // it does more good than bad. + // + // Use a value less than SIZE_MAX so that there's some room + // for the xz program and so on. Don't use 4000 MiB because + // it could look like someone mixed up base-2 and base-10. + const uint64_t limit_max = UINT64_C(4020) << 20; + + // UINT64_MAX is a special case for the string "max" so + // that has to be handled specially. + if (memlimit_compress != UINT64_MAX + && memlimit_compress > limit_max) + memlimit_compress = limit_max; +#endif + } if (set_decompress) memlimit_decompress = new_memlimit; Modified: stable/12/contrib/xz/src/xz/xz.1 ============================================================================== --- stable/12/contrib/xz/src/xz/xz.1 Sun Apr 5 01:53:35 2020 (r359635) +++ stable/12/contrib/xz/src/xz/xz.1 Sun Apr 5 02:08:17 2020 (r359636) @@ -1005,6 +1005,25 @@ instead of until the details have been decided. .RE .IP "" +For 32-bit +.BR xz +there is a special case: if the +.I limit +would be over +.BR "4020\ MiB" , +the +.I limit +is set to +.BR "4020\ MiB" . +(The values +.B 0 +and +.B max +aren't affected by this. +A similar feature doesn't exist for decompression.) +This can be helpful when a 32-bit executable has access +to 4\ GiB address space while hopefully doing no harm in other situations. +.IP "" See also the section .BR "Memory usage" . .TP
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?202004050208.03528HYH050368>