From owner-freebsd-ports@FreeBSD.ORG Sat Jun 19 13:41:25 2010 Return-Path: Delivered-To: ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DBB55106577D; Sat, 19 Jun 2010 13:41:24 +0000 (UTC) (envelope-from lasse.collin@tukaani.org) Received: from mailfw02.zoner.fi (mailfw02.zoner.fi [84.34.147.249]) by mx1.freebsd.org (Postfix) with ESMTP id 8C3FE8FC1D; Sat, 19 Jun 2010 13:41:23 +0000 (UTC) Received: from www25.zoner.fi ([84.34.147.45]) by wwwsmtp02.zoner.fi with ESMTP; 19 Jun 2010 16:41:21 +0300 Received: from 86-60-146-209-dyn-dsl.ssp.fi ([86.60.146.209] helo=kaneli.localnet) by www25.zoner.fi with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69) (envelope-from ) id 1OPyIT-0007LU-QJ; Sat, 19 Jun 2010 16:41:21 +0300 From: Lasse Collin To: Matthias Andree Date: Sat, 19 Jun 2010 16:41:26 +0300 User-Agent: KMail/1.13.3 (Linux/2.6.33-ARCH; KDE/4.4.4; x86_64; ; ) References: <4C1BA4D4.9000205@FreeBSD.org> <201006190855.05061.lasse.collin@tukaani.org> <4C1C9F9F.8090808@FreeBSD.org> In-Reply-To: <4C1C9F9F.8090808@FreeBSD.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Message-Id: <201006191641.26301.lasse.collin@tukaani.org> X-Antivirus-Scanner: Clean mail though you should still use an Antivirus Cc: ports@freebsd.org, Christian Weisgerber , portmgr@freebsd.org Subject: Re: FreeBSD ports USE_XZ critical issue on low-RAM computers X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Jun 2010 13:41:25 -0000 On 2010-06-19 Matthias Andree wrote: > We have system facilities for limiting resources, including those > that limit virtual memory. Limiting virtual memory with "ulimit -v" is generally not so great. It cripples mmap(), which can use a lot of virtual memory while using little actual RAM. If I made xz use mmap() for handling input files when possible, limiting virtual memory would have little to do with limiting the actual memory usage of xz: if xz mmapped a 280 MiB file that needs 65 MiB of memory to decompress, xz would run out memory if virtual memory was capped to 300 MiB. Luckily for you I don't plan to use mmap() in xz. :-) Perhaps FreeBSD provides a good working way to limit the amount of memory that a process actually can use. I don't see such a way e.g. in Linux, so having some method in the application to limit memory usage is definitely nice. It's even more useful in the compression library, because a virtual-memory-hog application on a busy server doesn't necessarily want to use tons of RAM for decompressing data from untrusted sources. > For compression, it's less critical because service is degraded, not > denied, but I'd still think -M max would be the better default. I can > always put "export XZ_OPT=-3" in /etc/profile.d/local.sh or wherever > it belongs on the OS of the day. If a script has "xz -9", it overrides XZ_OPT=-3. > I still think utilities and applications should /not/ impose > arbitrarily lower limits by default though. There's no multithreading in xz yet, but when there is, do you want xz to use as many threads as there are CPU cores _by default_? If so, do you mind if compressing with "xz -9" used around 3.5 GiB of memory on a four-core system no matter how much RAM it has? I think it is quite obvious that you want the number of threads to be limited so that xz won't accidentally exceed the total amount of physical RAM, because then it is much slower than using fewer threads. Being faster is the whole point of threading anyway. Naturally doing unusual things is sometimes wanted so a limit can be overriden. This is all about the default behavior only. In most cases, lowering the compression settings automatically is friendly towards the user. People easily write "xz -9" to scripts without thinking if they actually want that, because they are used to -9 with gzip and bzip2. I would find it dumb to annoy users of slightly older hardware with _default behavior_ that puts their system to swap whenever such a script is ran. They can still get the swap-till-the- morning behavior if they really want it by disabling the limit when compressing by using XZ_OPT. > > Disabling the limiter completely by default doesn't seem like an > > option, because it would only change who will be annoyed. Comments > > are very welcome. Thanks. > > It is a necessity to change it. In short, some people find a default limit annoying and some other people would find lack of default limit annoying. (And most people probably don't care.) So the question is, which group will complain more; obviously I cannot make everyone happy. At this point it starts to look like that your group is winning. ;-) I will have to discuss with people in the other group before making decisions. -- Lasse Collin | IRC: Larhzu @ IRCnet & Freenode