From owner-freebsd-ports@FreeBSD.ORG Sat Jun 19 00:29:40 2010 Return-Path: Delivered-To: ports@FreeBSD.org Received: from mandree.no-ip.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by hub.freebsd.org (Postfix) with ESMTP id 85C8C106566C; Sat, 19 Jun 2010 00:29:40 +0000 (UTC) (envelope-from mandree@FreeBSD.org) Received: from [IPv6:::1] (localhost [IPv6:::1]) by merlin.emma.line.org (Postfix) with ESMTP id 6189C9440F; Fri, 18 Jun 2010 18:54:47 +0200 (CEST) Message-ID: <4C1BA4D4.9000205@FreeBSD.org> Date: Fri, 18 Jun 2010 18:54:44 +0200 From: Matthias Andree Organization: FreeBSD developers User-Agent: Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.1.9) Gecko/20100317 SUSE/3.0.4-1.1.1 Thunderbird/3.0.4 MIME-Version: 1.0 To: ports@FreeBSD.org, portmgr@freebsd.org X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit Cc: Christian Weisgerber , Lasse Collin Subject: FreeBSD ports USE_XZ critical issue on low-RAM computers X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Jun 2010 00:29:40 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 (blind carbon copy to Lasse Collin, maintainer of xzutils upstream) Greetings, I've just had xz break my devel/libtool22 FreeBSD port build on a low memory computer (128 MB). Reason is that xz by default caps memory use at ~40% of physical RAM (this is documented), and skips decompressing files if that doesn't suffice. Options to override this are provided, but I question this behaviour: - - This xz feature may seem reasonable if we do have a choice about memory use, meaning, during compression. There we can reduce dictionary size, search depth, and all that, and reduce memory use at the expense of compression factor. No big deal. - - This feature is, in my perception, INADEQUATE during decompression. If I have a .xz file (downloaded from the Internet) that needs 90 MB RAM to decompress, then I need to use those 90 MB no matter if that's nice or not, it's just critical. I am proposing to Lasse Collin to drop memory capping functionality in xz/lzma in decompress mode, and in lzmadec/xzdec. This is what happens in practice: > # cd /usr/ports/devel/libtool22 > # make > ===> Vulnerability check disabled, database not found > ===> License check disabled, port has not defined LICENSE > ===> Extracting for libtool-2.2.6b > => MD5 Checksum OK for libtool-2.2.6b.tar.lzma. > => SHA256 Checksum OK for libtool-2.2.6b.tar.lzma. > ===> libtool-2.2.6b depends on file: /usr/local/bin/xz - found > /usr/local/bin/xz: /usr/ports/distfiles//libtool-2.2.6b.tar.lzma: Memory usage limit reached > /usr/local/bin/xz: Limit was 46 MiB, but 65 MiB would have been needed > ===> Patching for libtool-2.2.6b > ===> Applying FreeBSD patches for libtool-2.2.6b > patch: **** can't cd to /usr/ports/devel/libtool22/work/libtool-2.2.6b: No such file or directory > => Patch patch-libltdl-Makefile.in failed to apply cleanly. > *** Error code 1 > > Stop in /usr/ports/devel/libtool22. Investigating this (on FreeBSD 6.4-RELEASE-p10 i386): - - The ports system generates this command line and executes it: > for file in libtool-2.2.6b.tar.lzma; do if ! (cd /usr/ports/devel/libtool22/work && /usr/local/bin/xz -dc /usr/ports/distfiles//$file | /usr/bin/tar -xf - --no-same-owner --no-same-permissions); then exit 1; fi; done - - xz fails with error message as above, exits with return code 1, after not having printed any bytes on stdout. - - tar decompresses a 0-byte file, which succeeds. - - The FreeBSD ports system does not detect the tar input was empty, and proceeds to the patch phase, which fails because tar didn't extract files - so patch fails. This misleads end users about the actual error. 2. xz port or its use in the framework: we need to disable the memory capping for decompressing. We simply must try to decompress no matter what. I'll wait a couple of days hoping for Lasse's response, but generally, we have two alternatives to proceed: a. directly patch archivers/xz so that the memory capping is not applied during decompression. Should be considered upstream -- because during decompression you cannot tweak parameters to reduce memory usage, you need to proceed with what's needed by the file to build the dictionaries and whatnot to decompress. (CC'ing port maintainer Christian Weisgerber) b. ports framework: pass command line arguments or environment variables to defeat capping with the -M option. Note that -M 0 sets the default cap to 40% of physical RAM. Example (the large figure is 2^31-1 i. e. INT_MAX): > # export XZ_OPT=-M2147483647 > # xz -dc >/dev/null /usr/ports/distfiles/libtool*.tar.lzma > # echo $? > 0 I'd propose using XT_OPT+=-M$(getconf LLONG_MAX) for now. We can't use unsigned types because FreeBSD getconf, as of 6.4, would return those as negative numbers (due to internal use of intmax_t even where variables are of unsigned type and might be in range (INTMAX_MAX, UINTMAX_MAX], which causes (justified!) xz complaints, and INTMAX_MAX or INT64_MAX cannot be queried via getconf(1) on FreeBSD (undefined). The getconf fix is non-trivial, so I expect we better not rely on that now. So for FreeBSD, I propose to patch this Mk/bsd.port.mk line #2382 (as of CVS rev. 1.642) EXTRACT_CMD?= ${XZ_CMD} to EXTRACT_CMD?= ${XZ_CMD} $$(getconf LLONG_MAX) Other proposals? Best regards Matthias - -- Matthias Andree (ports/ committer) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkwbpNMACgkQvmGDOQUufZXWJQCgmSOmo5a0rnFR3aNa7/4NEwGk pQ8AoJ2liNx8o62j5Z6ON1Lh22n60hia =pYpJ -----END PGP SIGNATURE-----