Date: Thu, 28 Jan 1999 12:34:26 -0500 (EST) From: "John S. Dyson" <dyson@iquest.net> To: dillon@apollo.backplane.com (Matthew Dillon) Cc: dyson@iquest.net, wes@softweyr.com, toasty@home.dragondata.com, hackers@FreeBSD.ORG Subject: Re: High Load cron patches - comments? Message-ID: <199901281734.MAA21561@y.dyson.net> In-Reply-To: <199901281709.JAA07773@apollo.backplane.com> from Matthew Dillon at "Jan 28, 99 09:09:30 am"
next in thread | previous in thread | raw e-mail | index | archive | help
Matthew Dillon said: > :> > :> Especially as we start diving more into SMP and threaded applications; > :> which will need some effective means of throttling themselves. The > :> problem with Matt's comment above is he doesn't offer any useful > :> alternative, and couting child processes just isn't an effective means > :> of throttling the overall load on a machine. > :> > :It is *sometimes* appropriate to criticize, even when alternatives aren't > :provided. The kind of technique that I have successfully experimented with > :is a scheme that has two phases: A costing mechanism and a stats mechanism. > : > :The costing mechanism is a direct call from when the resource is attempted > :to be allocated. It checks immediately if the cost (and recent incurred > > Well, actually I would put forth that limiting the number of processes > any one subsystem is allowed to fork is perfectly acceptable and generally > produces better results then trying to dynamically balance the load > between subsystems on any given machine. > I don't agree about on "any" given machine. > > For the first two years of BEST's existance, I literally spent day and > night trying to balance things on overloaded machines: The web server, > sendmail, news, user load, popper, and so forth. Load-relating balancing > never worked well. It took a year before I realized that it wouldn't > work at all. After we started slapping absolute limits on things, the > machines stopped crashing due to multi-subsysem fork cascade failures. > There is value in both absolute and rate limits (control theory and systems are your friend.) It all applies, but your experience at Best is one datapoint with one specific developer and not "any" developer. I suspect that this technique that I have developed had not been used at Best. It might not be applicable for what you were trying to do, but it is indeed applicable to what I am trying to do. Paging rate limitation is a very valuable load balancing tool, but if you don't design your technology correctly, it is reasonble to get odd or non-productive results. Throttling fork rate is also a valuable tool, and maybe a hard limit is good also. It is all about how creative you are (or want to be) in your solution :-). -- John | Never try to teach a pig to sing, dyson@iquest.net | it makes one look stupid jdyson@nc.com | and it irritates the pig. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199901281734.MAA21561>