From owner-freebsd-arch Tue May 15 10:24:47 2001 Delivered-To: freebsd-arch@freebsd.org Received: from earth.backplane.com (earth-nat-cw.backplane.com [208.161.114.67]) by hub.freebsd.org (Postfix) with ESMTP id 210DD37B424 for ; Tue, 15 May 2001 10:24:44 -0700 (PDT) (envelope-from dillon@earth.backplane.com) Received: (from dillon@localhost) by earth.backplane.com (8.11.3/8.11.2) id f4FHOYt54576; Tue, 15 May 2001 10:24:34 -0700 (PDT) (envelope-from dillon) Date: Tue, 15 May 2001 10:24:34 -0700 (PDT) From: Matt Dillon Message-Id: <200105151724.f4FHOYt54576@earth.backplane.com> To: Terry Lambert Cc: Rik van Riel , arch@FreeBSD.ORG, linux-mm@kvack.org, sfkaplan@cs.amherst.edu Subject: Re: on load control / process swapping References: <3B00CECF.9A3DEEFA@mindspring.com> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG :Rik van Riel wrote: :> So we should not allow just one single large job to take all :> of memory, but we should allow some small jobs in memory too. : :Historically, this problem is solved with a "working set :quota". We have a process-wide working set quota. It's called the 'memoryuse' resource. :... :> 5) have a better chance of getting out of the overload situation :> sooner :> :> I realise this would make the scheduling algorithm slightly :> more complex and I'm not convinced doing this would be worth :> it myself, but we may want to do some brainstorming over this ;) : :A per vnode working set quota with a per use count adjust :would resolve most load thrashing issues. Programs with It most certainly would not. Limiting the number of pages you allow to be 'cached' on a vnode by vnode basis would be a disaster. It has absolutely nothing whatsoever to do with thrashing or thrash-management. It would simply be an artificial limitation based on artificial assumptions that are as likely to be wrong as right. If I've learned anything working on the FreeBSD VM system, it's that the number of assumptions you make in regards to what programs do, how they do it, how much data they should be able to cache, and so forth is directly proportional to how badly you fuck up the paging algorithms. I implemented a special page-recycling algorithm in 4.1/4.2 (which is still there in 4.3). Basically it tries predict when it is possible to throw away pages 'behind' a sequentially accessed file, so as not to allow that file to blow away your cache. E.G. if you have 128M of ram and you are sequentially accessing a 200MB file, obviously there is not much point in trying to cache the data as you read it. But being able to predict something like this is extremely difficult. In fact, nearly impossible. And without being able to make the prediction accurately you simply cannot determine how much data you should try to cache before you begin recycling it. I wound up having to change the algorithm to act more like a heuristic -- it does a rough prediction but doesn't hold the system to it, then allows the page priority mechanism to refine the prediction. But it can take several passes (or non-passes) on the file before the page recycling stabilizes. So the jist of the matter is that FreeBSD (1) already has process-wide working set limitations which are activated when the system is under load, and (2) already has a heuristic that attempts to predict when not to cache pages. Actually several heuristics (a number of which were in place in the original CSRG code). -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message