From owner-freebsd-hackers@FreeBSD.ORG Sun Nov 17 23:09:12 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ADB3CC9B; Sun, 17 Nov 2013 23:09:12 +0000 (UTC) Received: from mail-ea0-x234.google.com (mail-ea0-x234.google.com [IPv6:2a00:1450:4013:c01::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EDE892B56; Sun, 17 Nov 2013 23:09:11 +0000 (UTC) Received: by mail-ea0-f180.google.com with SMTP id f15so268946eak.25 for ; Sun, 17 Nov 2013 15:09:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=9JhQdvAs5+jD0Qs/gm3vGXoKD2I5ED1Rr63BFPP9G4Y=; b=jA+duaKysm+lYhos+xqy6q7TTtjxwWUNK27Tvv0L9z1x2oVrOqabcb7r9dvfFrtE/T 9pNAQsggWNMMzYyHtQ9wnnQtPjQRVSN4QNDNoYOD8SaUo2W0Ee9EADPNRwfgEyUmRbtg +I/v3XDUOiHvFdHmYNq58tdiwkIN9vlAJYLUTtdH2IyN+QE7lPv7buRn1sRibDmei8VA lGkkOnQ2liEFW9hvXI2RGChoKHWLCASs8t8s0BKD7Lbg41r+BoQ+mIUknIwWD1XEs9Q9 PoUjt7b+6yQ4QQnnxwfSSgA91s02wTaxRxm68QnfEUMVzGmtC5f/gqL9qe/KJOUuX3aW WQ4g== X-Received: by 10.15.65.11 with SMTP id p11mr345117eex.49.1384729749539; Sun, 17 Nov 2013 15:09:09 -0800 (PST) Received: from mavbook.mavhome.dp.ua ([178.137.150.35]) by mx.google.com with ESMTPSA id o47sm31544449eem.21.2013.11.17.15.09.07 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 17 Nov 2013 15:09:08 -0800 (PST) Sender: Alexander Motin Message-ID: <52894C92.60905@FreeBSD.org> Date: Mon, 18 Nov 2013 01:09:06 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.1.0 MIME-Version: 1.0 To: "freebsd-hackers@freebsd.org" , "freebsd-current@freebsd.org" Subject: UMA cache back pressure Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Nov 2013 23:09:12 -0000 Hi. I've created patch, based on earlier work of avg@, to add back pressure to UMA allocation caches. The problem of physical memory or KVA exhaustion existed there for many years and it is quite critical now for improving systems performance while keeping stability. Changes done in memory allocation last years improved situation. but haven't fixed completely. My patch solves remaining problems from two sides: a) reducing bucket sizes every time system detects low memory condition; and b) as last-resort mechanism for very low memory condition, it cycling over all CPUs to purge their per-CPU UMA caches. Benefit of this approach is in absence of any additional hard-coded limits on cache sizes -- they are self-tuned, based on load and memory pressure. With this change I believe it should be safe enough to enable UMA allocation caches in ZFS via vfs.zfs.zio.use_uma tunable (at least for amd64). I did many tests on machine with 24 logical cores (and as result strong allocation cache effects), and can say that with 40GB RAM using UMA caches, allowed by this change, by two times increases results of SPEC NFS benchmark on ZFS pool of several SSDs. To test system stability I've run the same test with physical memory limited to just 2GB and system successfully survived that, and even showed results 1.5 times better then with just last resort measures of b). In both cases tools/umastat no longer shows unbound UMA cache growth, that makes me believe in viability of this approach for longer runs. I would like to hear some comments about that: http://people.freebsd.org/~mav/uma_pressure.patch Thank you. -- Alexander Motin