From owner-freebsd-hackers Wed Feb 19 19:33:14 2003 Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2FEDF37B401 for ; Wed, 19 Feb 2003 19:33:12 -0800 (PST) Received: from mail.iskon.hr (mail.iskon.hr [213.191.128.4]) by mx1.FreeBSD.org (Postfix) with SMTP id D1A4043F75 for ; Wed, 19 Feb 2003 19:33:09 -0800 (PST) (envelope-from zec@tel.fer.hr) Received: (qmail 22819 invoked from network); 19 Feb 2003 23:50:36 +0100 Received: from zg07-152.dialin.iskon.hr (HELO tel.fer.hr) (213.191.150.153) by mail.iskon.hr with SMTP; 19 Feb 2003 23:50:36 +0100 Message-ID: <3E540A43.215CDECD@tel.fer.hr> Date: Wed, 19 Feb 2003 23:50:43 +0100 From: Marko Zec X-Mailer: Mozilla 4.8 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Mooneer Salem Cc: FreeBSD Hackers Subject: Re: Per-jail CPU limits? References: Content-Type: text/plain; charset=iso-8859-2 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Mooneer Salem wrote: > Hello, > > I've been looking at the kernel source, in particular the scheduler > in the past few weeks. I found a place in kern_switch.c where per-jail > CPU controls could be added (in particular, in the kse_reassign() function). > >From looking at that function, I could loop through td_runq until I either: > > 1. Found a thread that isn't jailed, > 2. Found a jailed thread, but determine it's safe to let it run because > it does not go over sysctl-defined limits, or > 3. Find no usable thread, in which case the KSE would theoretically switch > over to the idle process until it's time to repeat the process again. > > This should allow the use of the standard FreeBSD scheduler, except for > the jail limits. The question is, how do we determine the total CPU used > by the jail? I found the kg_estcpu entry in struct ksegrp, which the thread > has a pointer to, but would that be enough? Is there a different approach we > could take that would solve this problem? A rudimentary CPU usage limiting on per virtual image basis (virtual image can be considered a jailed environment with its own independent network stack instance) was implemented using an algorithm very similar to what you proposed, so you might check the original patch against 4.7-RELEASE kernel at http://www.tel.fer.hr/zec/BSD/vimage/index.html As I didn't have enough time yet to make a usable port to 5.0, my assumptions regarding programming in -CURRENT might be slightly wrong, but I guess you'll have to: 1) extend the jail structure to hold CPU usage accounting information on per-jail basis; 2) update this field when doing normal per-process CPU accounting in kern/kern_clock.c / statclock(); 3) do some decay filtering to ensure stability and "smoothness" of the acquired per-jail CPU usage data; 4) in kern/kern_switch.c / chooseproc() implement the steps you originally defined as 1. to 3. 5) on each HZ tick in kern/kern_synch.c / schedclock() check the current process/jail hasn't consumed more CPU time than it was allowed, and if it has, reschedule a new process. This is necessary to ensure acceptable interactive response for processes/jails running with administratively restricted CPU resources, otherwise the process could consume the entire time quantum (10 ms by default), and would than have to wait annoyingly long in order for the average per-jail CPU usage to drop under the defined threshold. 6) optionally, extend procrunnable() in kern/kern_switch.c to return 0 in case there are over-the-CPU-limit processes in active run queue, in order for idle loop to be able to execute the halt instruction, instead of unnecessarily looping endlessly through chooseproc() until the next clock tick. This can be especially useful on laptops where you don't want a process with CPU usage limit to actually burn the battery power in idle loop, and also burn your lap at the same time :) Note: everything I wrote is based on my experience with 4.7-R kernel, in 5.0 many things have changed replacing process with threads as the atomic entities for scheduling, so probably the function naming and some logic has changed also... Marko To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message