From owner-freebsd-stable Thu Dec 30 16:49:15 1999 Delivered-To: freebsd-stable@freebsd.org Received: from overcee.netplex.com.au (overcee.netplex.com.au [202.12.86.7]) by hub.freebsd.org (Postfix) with ESMTP id 7FC251525B; Thu, 30 Dec 1999 16:49:04 -0800 (PST) (envelope-from peter@netplex.com.au) Received: from netplex.com.au (localhost [127.0.0.1]) by overcee.netplex.com.au (Postfix) with ESMTP id 01D721CA0; Fri, 31 Dec 1999 08:49:02 +0800 (WST) (envelope-from peter@netplex.com.au) X-Mailer: exmh version 2.1.1 10/15/1999 To: Tom Cc: Matthew Dillon , freebsd-stable@FreeBSD.ORG, freebsd-hackers@FreeBSD.ORG Subject: Re: softupdates and debug.max_softdeps In-Reply-To: Message from Tom of "Thu, 30 Dec 1999 11:14:08 PST." Date: Fri, 31 Dec 1999 08:49:01 +0800 From: Peter Wemm Message-Id: <19991231004902.01D721CA0@overcee.netplex.com.au> Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG FYI: On hub.freebsd.org (the freebsd mailing list server), if we activate softupdates on the disk containing the postfix spool, the machine reboots (silently if I recall correctly) within 5 minutes of postfix starting up. This is a much smaller system of course, with smaller memory and filesystem working set. (postfix spool of ~50-80MB, 256MB ram). I thought I'd post this as a real-use datapoint. Tom wrote: > Yes, postmark operates on the same file set. I used the following > postmark settings: > > set number 30000 > set transactions 4000000 > set size 1500 200000 > > which uses a set of 30,000 files, and does a 4,000,000 transactions them > (random mix of various operations), and size between 1,500 and 200,000 > bytes. BTW, I hacked my version of postmark to use unsigned ints in > various places. > > I guess by having a very large filesystem (80GB), and mostly empty, the > softupdate code is able to queue an enormous amount of metadata updates > over time. > > I tried forcing max_softdeps down to 50,000, and within a couple of > hours all processes accessing that filesystem hung! > > Also, postmark is filesytem benchmarking and stress tester utility. > Adding fsync() would defeat the purpose a bit! > > So in summary, if max_softdeps is left at the default, the system will > reboot in 24 to 36 hours. If max_softdeps is set down, filesystem access > will eventually hang within 12 hours. > > > On Thu, 30 Dec 1999, Matthew Dillon wrote: > > > Well, in general I would not mess with max_softdeps - softupdates gets > > very inefficient if it hits its limits. I think you may have found a > > flaw in the code, though. Softupdates reschedules its vnode sync whene ver > > it does something to the vnode. Postmark must be operating on the same > > set of files for very long periods of time, including truncating and > > extending them, for softupdates to get that far behind! Kirk may have > > to modify the vnode scheduling to not reschedule the vnode beyond a > > certain aggregate delay in order to ensure that things get synchronized > > in a reasonable period of time. > > > > Softupdates biggest problem are with overly-long delays in block > > reclamation - several people have commented on it. I think what you > > are seeing is a special case of this problem that causes it to be much > > worse then normal. > > > > In the mean time you have a couple of choices. You can try running > > 'sync' every so often, or you can write a small C program to fsync() > > the files postfix messes with every so often. > > > > -Matt > > Matthew Dillon > > > > > > : I'm trying to find some information on reasonable settings for > > :debug.max_softdeps on a recent FreeBSD-stable system. > > : > > : It seems that if you have a machine that is able to generate disk IO > > :much faster than can be handled, has a large amount of RAM (and therefore > > :debug.max_softdeps is large), and the filesystem is very large (about > > :80GB), filesystem metadata updates can get _very_ far behind. > > : > > : For instance, on a test system running 4 instances of postmark > > :continuously for 24 hours, "df" reports that 40 GB of disk space is being > > :used, even though only about 5 GB is actually used. If I kill the > > :postmark processes, the metadata is eventually dribbled out and "df" > > :reports 5GB in use. It takes about 20 minutes for the metadata to be > > :updated on a completely ideal system. > > : > > : On this particular system, it doesn't seem to stabilize either. If the > > :4 postmark instances are allowed to run, disk usage seems to climb > > :indefinitely (at 40GB it was still climbing), until eventually the machine > > :silently reboots. > > : > > : debug.max_softdeps is by default set to 523,712 (1 GB of RAM). Is that > > :a resonable value? I see some tests in the docs with max_softdeps set to > > :4000 or so. > > : > > : > > :Tom > > > > > > > > > Tom > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-hackers" in the body of the message > > Cheers, -Peter -- Peter Wemm - peter@FreeBSD.org; peter@yahoo-inc.com; peter@netplex.com.au To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message