From owner-freebsd-current@FreeBSD.ORG Mon Nov 18 12:10:22 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 37258F17; Mon, 18 Nov 2013 12:10:22 +0000 (UTC) Received: from mail-qa0-x236.google.com (mail-qa0-x236.google.com [IPv6:2607:f8b0:400d:c00::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CD2962567; Mon, 18 Nov 2013 12:10:21 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id f11so1145119qae.6 for ; Mon, 18 Nov 2013 04:10:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=v5eD+yxyx4RXPz2/rlhOv350IlwQtIUrqZeXMtBJX+M=; b=hoim1jtWE5vY0rmrIH7vvklPIGGwUy5FP0dkfGP0a9H2TrM7rzzzzc1M7fTbsoGyWv IEdTmxnMLXV54aDdIoqkU3I0zPFviKXznZchnNpqQYq9TOQnqSafccoZFhxAc3FtumL9 tol6BelQOa/F6kfdqCC4nNsS720BbLG5PB7Q9ufJTfQej9K4moaS01IJvFcM/9DRFZPy W2Z3AEryFvSkEHYP0jpMzJalB7a3n+7XQVgkVDaJn2jmcTcNUR0lZjf9K0fmCdtFZfoD al1JCYap8/ucOrRJVZcOl3+vif/suFvhikdWvqPuKplnitKW0wb1UUBUoKWvy+oO6rBz Ddrw== MIME-Version: 1.0 X-Received: by 10.224.98.200 with SMTP id r8mr33352927qan.26.1384776619952; Mon, 18 Nov 2013 04:10:19 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.224.207.66 with HTTP; Mon, 18 Nov 2013 04:10:19 -0800 (PST) In-Reply-To: <5289DBF9.80004@FreeBSD.org> References: <52894C92.60905@FreeBSD.org> <5289DBF9.80004@FreeBSD.org> Date: Mon, 18 Nov 2013 04:10:19 -0800 X-Google-Sender-Auth: FWiYWYHm8mpA44UH0srn0ozYp3g Message-ID: Subject: Re: UMA cache back pressure From: Adrian Chadd To: Alexander Motin Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-hackers@freebsd.org" , "freebsd-current@freebsd.org" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Nov 2013 12:10:22 -0000 On 18 November 2013 01:20, Alexander Motin wrote: > On 18.11.2013 10:41, Adrian Chadd wrote: >> >> Your patch does three things: >> >> * adds a couple new buckets; > > > These new buckets make bucket size self-tuning more soft and precise. > Without them there are buckets for 1, 5, 13, 29, ... items. While at bigger > sizes difference about 2x is fine, at smallest ones it is 5x and 2.6x > respectively. New buckets make that line look like 1, 3, 5, 9, 13, 29, > reducing jumps between steps, making algorithm work softer, allocating and > freeing memory in better fitting chunks. Otherwise there is quite a big gap > between allocating 128K and 5x128K of RAM at once. Right. That makes sense, but your initial email didn't say "oh, I'm adding more buckets." :-) > >> * reduces some lock contention > > > More precisely patch adds check for congestion on free to grow bucket sizes > same as on allocation. As consequence that indeed should reduce lock > congestion, but I don't have specific numbers. All I see is that VM and UMA > mutexes no longer appear in profiling top after all these changes. Sure. But again, you don't say that in your commit message. :) > * does soft back pressure > > In this list you have missed mentioning small but major point of the patch > -- we should prevent problems, not just solve them. As I have written in > original email, this specific change shown me 1.5x performance improvement > in low-memory condition. As I understand, that happened because VM no longer > have to repeatedly allocate and free hugely oversized buckets of 10-15 * > 128K. yup, sorry I missed this. It's a sneaky two lines. :) > >> * does the aggressive backpressure. > > > After all above that is mostly just a safety belt. With 40GB RAM that code > was triggered only couple times during full hour of testing with debug > logging inserted there. On machine with 2GB RAM it is triggered quite > regularly and probably that is unavoidable since even with lowest bucket > size of one item 24 CPUs mean 48 cache buckets, i.e. up to 6MB of otherwise > unreleasable memory for single 128K zone. > > >> So, do you get any benefits from just the first one, or first two? > > > I don't see much reason to handle that in pieces. As I have described above, > each part has own goal, but they much better work together. Well, with changes like this, having them broken up and committed in small pieces make it easier for people to do regression testing with. If you introduce some regression in a particular workload then the user or developer is only going to find that it's this patch and won't necessarily know how to break it down into pieces to see which piece actually introduced the regression in their specific workload. I totally agree that this should be done! It just does seem to be something that could be committed in smaller pieces quite easily so to make potential debugging later on down the road much easier. Each commit builds on the previous commit. So, something like (in order): * add two new buckets, here's why * fix locking, here's why * soft back pressure * aggressive backpressure Did you get profiling traces from the VM free paths? Is it because it's churning the physical pages through the VM physical allocator? or? -adrian