From owner-svn-src-all@freebsd.org Thu Feb 6 07:45:08 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 02A38245909; Thu, 6 Feb 2020 07:45:08 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48Cr7R646lz3GPv; Thu, 6 Feb 2020 07:45:07 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C74771864B; Thu, 6 Feb 2020 07:45:07 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 0167j7HG011559; Thu, 6 Feb 2020 07:45:07 GMT (envelope-from delphij@FreeBSD.org) Received: (from delphij@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 0167j7aW011558; Thu, 6 Feb 2020 07:45:07 GMT (envelope-from delphij@FreeBSD.org) Message-Id: <202002060745.0167j7aW011558@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: delphij set sender to delphij@FreeBSD.org using -f From: Xin LI Date: Thu, 6 Feb 2020 07:45:07 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r357608 - vendor/xz/dist/src/xz X-SVN-Group: vendor X-SVN-Commit-Author: delphij X-SVN-Commit-Paths: vendor/xz/dist/src/xz X-SVN-Commit-Revision: 357608 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Feb 2020 07:45:08 -0000 Author: delphij Date: Thu Feb 6 07:45:07 2020 New Revision: 357608 URL: https://svnweb.freebsd.org/changeset/base/357608 Log: Apply a reduced version of upstream 353970510895f6a80adfe60cf71b70a95adfa8bc which implements memory limit in xz(1) when running in 32-bit mode. The change is applied directly in vendor area to ease future import. Modified: vendor/xz/dist/src/xz/hardware.c vendor/xz/dist/src/xz/xz.1 Modified: vendor/xz/dist/src/xz/hardware.c ============================================================================== --- vendor/xz/dist/src/xz/hardware.c Thu Feb 6 01:25:30 2020 (r357607) +++ vendor/xz/dist/src/xz/hardware.c Thu Feb 6 07:45:07 2020 (r357608) @@ -68,8 +68,38 @@ hardware_memlimit_set(uint64_t new_memlimit, new_memlimit = (uint32_t)new_memlimit * total_ram / 100; } - if (set_compress) + if (set_compress) { memlimit_compress = new_memlimit; + +#if SIZE_MAX == UINT32_MAX + // FIXME? + // + // When running a 32-bit xz on a system with a lot of RAM and + // using a percentage-based memory limit, the result can be + // bigger than the 32-bit address space. Limiting the limit + // below SIZE_MAX for compression (not decompression) makes + // xz lower the compression settings (or number of threads) + // to a level that *might* work. In practice it has worked + // when using a 64-bit kernel that gives full 4 GiB address + // space to 32-bit programs. In other situations this might + // still be too high, like 32-bit kernels that may give much + // less than 4 GiB to a single application. + // + // So this is an ugly hack but I will keep it here while + // it does more good than bad. + // + // Use a value less than SIZE_MAX so that there's some room + // for the xz program and so on. Don't use 4000 MiB because + // it could look like someone mixed up base-2 and base-10. + const uint64_t limit_max = UINT64_C(4020) << 20; + + // UINT64_MAX is a special case for the string "max" so + // that has to be handled specially. + if (memlimit_compress != UINT64_MAX + && memlimit_compress > limit_max) + memlimit_compress = limit_max; +#endif + } if (set_decompress) memlimit_decompress = new_memlimit; Modified: vendor/xz/dist/src/xz/xz.1 ============================================================================== --- vendor/xz/dist/src/xz/xz.1 Thu Feb 6 01:25:30 2020 (r357607) +++ vendor/xz/dist/src/xz/xz.1 Thu Feb 6 07:45:07 2020 (r357608) @@ -1005,6 +1005,25 @@ instead of until the details have been decided. .RE .IP "" +For 32-bit +.BR xz +there is a special case: if the +.I limit +would be over +.BR "4020\ MiB" , +the +.I limit +is set to +.BR "4020\ MiB" . +(The values +.B 0 +and +.B max +aren't affected by this. +A similar feature doesn't exist for decompression.) +This can be helpful when a 32-bit executable has access +to 4\ GiB address space while hopefully doing no harm in other situations. +.IP "" See also the section .BR "Memory usage" . .TP