From owner-freebsd-fs@FreeBSD.ORG Sat Oct 3 00:59:37 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 559661065692; Sat, 3 Oct 2009 00:59:37 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-yw0-f197.google.com (mail-yw0-f197.google.com [209.85.211.197]) by mx1.freebsd.org (Postfix) with ESMTP id EA4898FC0C; Sat, 3 Oct 2009 00:59:36 +0000 (UTC) Received: by ywh35 with SMTP id 35so4377598ywh.7 for ; Fri, 02 Oct 2009 17:59:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type; bh=4VyeEIcrzDi+lOMGrICjduOOHbXu7zcRYqPuIAgsYfE=; b=bp1su8DuHqpHGnaO+AqTD49Hm7cuO+41On64lcMNm/NGmUleox+O7QxyA4v9F2SCt5 IOb5cLMAjIBA9GqTlPtua6rDsNusrLAkQyEN6nKKl14i3gSh8qBm8QyJ+L73seQsxNVT lWanMz1Qov+BATINT/yLgCFTOOVj56cvX4OaY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=wlrG20OcDE5GTEKaPEXFgMKYK7UlrKlvfZclGg+Mzji/YB/vwUN0TzLUokS0Xe8vJQ 6bjZoMEmXvQRz9DwzqvX8gEFANHICuDUKGAFBvei6TBNgv7iBXvksv93AVSKPcTPh6Qi bNaEv5COCBEPOjv5DWIw3UUpC77W6nT1z8MVk= MIME-Version: 1.0 Sender: artemb@gmail.com Received: by 10.91.178.19 with SMTP id f19mr1811136agp.33.1254531576337; Fri, 02 Oct 2009 17:59:36 -0700 (PDT) In-Reply-To: <20091003000909.GD1660@garage.freebsd.pl> References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu> <20091002184526.GA1660@garage.freebsd.pl> <20091003000909.GD1660@garage.freebsd.pl> Date: Fri, 2 Oct 2009 17:59:36 -0700 X-Google-Sender-Auth: 321a7f472bea9057 Message-ID: From: Artem Belevich To: Pawel Jakub Dawidek Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org Subject: Re: ARC size constantly shrinks, then ZFS slows down extremely X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2009 00:59:37 -0000 > Before we start debuging pathological cases, could you try the patch > with defaul settings? Eventually with vm.kmem_size set to the amount of > RAM you have. System runs stable/8 r197716 on 4-core amd64 with 8G of RAM. With default /boot/loader.conf. Kernel comes up with following parameters: vm.kmem_size: 2764533760 vfs.zfs.arc_min: 215979200 vfs.zfs.arc_max: 1727833600 Under load ARC size reaches ~1.7G. At that time top reports: Mem: 47M Active, 11M Inact, 2158M Wired, 268K Cache, 21M Buf, 5693M Free However, as the FS load continues, ARC size, stays at 1.7G for couple of minutes, then shrinks down to 1.2G, then slowly grows to 1.7G, stays there for a little and then the shrink/grow cycle repeats. Throughout the test there's always ~5G of *free* memory. =============================================================== Now, the same experiment, with vm.kmem_size=8G vm.kmem_size: 8589934592 vfs.zfs.arc_min: 939524096 vfs.zfs.arc_max: 7516192768 ARC grows to 6.2G: Mem: 47M Active, 13M Inact, 7376M Wired, 31M Buf, 473M Free Then it quickly shrinks to 4.6G and grows to 6.2G again, shrinks again, etc.. What's different from the previous case is that after a while ZFS adjusts target size (kstat.zfs.misc.arcstats.c) down to ~5.8G and after that ZFS size oscillates between 4.2G and 5.6G. Another observation -- ARC shrinking happens when system is left with ~512M of free memory. Yet another observation is that even with ARC peak of ~5.8G, system has about 7.5G wired. Where did almost 2G of difference go? Fragmentation? I've tried both experiments with and without L2ARC -- behavior seems to be the same. --Artem