From owner-freebsd-fs@FreeBSD.ORG Wed Aug 9 00:44:58 2006 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E313116A4E0 for ; Wed, 9 Aug 2006 00:44:57 +0000 (UTC) (envelope-from qdolan@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.180]) by mx1.FreeBSD.org (Postfix) with ESMTP id C1B3843D62 for ; Wed, 9 Aug 2006 00:44:37 +0000 (GMT) (envelope-from qdolan@gmail.com) Received: by py-out-1112.google.com with SMTP id w49so93370pyg for ; Tue, 08 Aug 2006 17:44:36 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:in-reply-to:references:mime-version:content-type:message-id:cc:content-transfer-encoding:from:subject:date:to:x-mailer; b=PQT6XyAZIGFB+Iis08KZ7O5pwu9qk2gLogxMB35mQi30O31WfRbMfNotLY0tXOQTzG4gUVH7lmAQeSdfsWNdqkICCfxevfQ7arm9WRY0SGkeswiavbksNx9YyjtgsoYb2a2SYzqyAM+y08evx/PfTRMNrSe7k3MGaw9otrACKrQ= Received: by 10.35.126.7 with SMTP id d7mr454452pyn; Tue, 08 Aug 2006 17:44:36 -0700 (PDT) Received: from ?172.22.1.30? ( [203.13.70.60]) by mx.gmail.com with ESMTP id 12sm529840nzn.2006.08.08.17.44.35; Tue, 08 Aug 2006 17:44:36 -0700 (PDT) In-Reply-To: <44D8B6D6.6070209@centtech.com> References: <06530464-7491-4BC1-97C7-87285B03F2EB@gmail.com> <44D8B6D6.6070209@centtech.com> Mime-Version: 1.0 (Apple Message framework v752.2) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Q Date: Wed, 9 Aug 2006 10:44:31 +1000 To: Eric Anderson X-Mailer: Apple Mail (2.752.2) Cc: freebsd-fs@freebsd.org Subject: Re: Problem with ufs not releasing vm_pages on busy volume. (soft updates related) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Aug 2006 00:44:58 -0000 On 09/08/2006, at 2:07 AM, Eric Anderson wrote: > On 08/08/06 00:14, Q wrote: >> On 02/08/2006, at 8:10 PM, Q wrote: >>> I have a problem that seems to be related to something ufs >>> related not releasing some vm_pages on busy filesystems. I have >>> two servers running PostgreSQL, one running 6.0-RELEASE, the >>> other 6.1-RELEASE. Both are under the same (fairly heavy) load, >>> performing the same operations in bursts every five minutes. The >>> filesystems in question are 450-500Gig, each server using a >>> different brand of RAID card, they both have soft-updates enabled. >>> >>> The problem is that both servers are seeing an accumulation of >>> about 100Mb of active pages per day (looking at >>> vm.stats.vm.v_active_count) that never get released. The only >>> way to release these pages is to unmount the filesystem and >>> remount it. Failing to do this results in the server eventually >>> locking up. >>> >>> If someone could provide me with some direction on how to go >>> about tracking down what might be causing this to happen it >>> would be much appreciated. >> I have narrowed the cause of this issue down further to something >> to do with soft updates. If I turn off soft updates for the >> filesystem hosting the database the system no longer accumulates >> active vm_pages constantly. Instead for accumulating 100Mb a day >> of active vm pages until all memory is consumed, it will hover >> around 50-60Mb with soft updates disabled. >> If someone familiar with the softupdates code is willing to help >> me pinpoint the cause of this problem it would be much appreciated. > > > Is it possible for you to upgrade to the latest 6-STABLE branch, > just to make sure that the issue hasn't been fixed already? I did a buildworld on one the machine running 6.0-RELEASE last night. I just have to schedule some downtime to do the upgrade sometime today. Having two identical servers has it's advantages. > Is there any way to reproduce this on another box for testing? (I > assume not, due to the nature of these things) I think this bug is very circumstance specific, and my database design just happens to exercise the bug. The fact that I have two identical servers experiencing exactly the same problem does helps narrow the field of possibilities, but the servers are my no means "expendable". > Also - I wonder if doing a snapshot on the filesystem would flush > out the pages - is that something you can try? Possibly. I will see what I can do. -- Seeya...Q -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- _____ / Quinton Dolan - qdolan@gmail.com __ __/ / / __/ / / / __ / _/ / / Gold Coast, QLD, Australia __/ __/ __/ ____/ / - / Ph: +61 419 729 806 _______ / _\