Date: Wed, 20 May 2009 19:24:42 +0400 From: =?UTF-8?B?0JzQtdC90YzRiNC40LrQvtCyINCa0L7QvdGB0YLQsNC90YLQuNC9?= <kostjn@peterhost.ru> To: freebsd-jail@freebsd.org Subject: Re: CPU limit for Jails(patch for ULE scheduler) Message-ID: <4A1420BA.5050406@peterhost.ru> In-Reply-To: <49ED55FF.5080306@peterhost.ru> References: <49ED55FF.5080306@peterhost.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
Меньшиков Константин wrote: > Hello all! > Many users want have limits on resourse for jail, for examle cpu and > memory limit. > I`m rewrire original cdjones patch for cpu limit for jail under ULE > scheduler. > So, this work simple. Hi. I`m rewrite jail limit patch under CURRENT. New patch limited CPU, memory, filedesc, process. And allow change limit on the fly You can download tar.gz from http://kostjn.spb.ru/patch-jail-limit-8CURRENT.tar.gz =========================================================================== How to use. =========================================================================== Build cvsup CURRENT cd /usr/src patch -p0 < patch-jail-limit-8CURRENT make buildkernel make buildworld make installkernel reboot make installworld Create new entry in login.conf, for example class jail128 jail128:\ :cputime=10:\ :memoryuse=128M:\ :maxproc=256:\ :openfiles=1024:\ :tc=default: Cputime is percent on 1 core. Openfiles is sum filedesc for all proc in jail. Create new jail. ... Add in /etc/rc.conf jail_test_flags="-Ljail128" Run new jail /etc/rc.d/jail start test =========================================================================== Sysctl =========================================================================== Added sysctl [root@book ~]# sysctl security.jail.limit security.jail.limit.enable: 1 security.jail.limit.memory_exceed_kill: 0 [root@book ~]# sysctl -d security.jail.limit security.jail.limit: Jail limit security.jail.limit.enable: Enable jail limit security.jail.limit.memory_exceed_kill: Kill biggest proc in jail, if jail excee d memory limit =========================================================================== Jset and Jget =========================================================================== jset and jget is program for set new jail limit and get current limit Example [root@book ~]# cat /etc/rc.conf | grep jail2 jail_list="jail1 jail2 jail3 jail4 jail5 jail6 jail7 jail8 jail9 jail10" jail_jail2_rootdir="/usr/jails/jail2/" jail_jail2_hostname="jail2.book.pht" jail_jail2_interface="re0" jail_jail2_ip="192.168.200.22" jail_jail2_flags="-Ljail64" [root@book ~]# /etc/rc.d/jail start jail2 Configuring jails:. Starting jails: jail2.book.pht. [root@book ~]# cd ~kostjn/ [root@book /home/kostjn]# ./jget.o 1 Jail limits and rusage, jid = 1 Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512 Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65 [root@book /home/kostjn]# ./jset.o 1 jail2048 Set new jail limits, jid = 1 Limits: CPU 30, MEM 2048M, NPROC 1024, NOFILE 2048 [root@book /home/kostjn]# ./jget.o 1 Jail limits and rusage, jid = 1 Limits: CPU 30, MEM 2048M, NPROC 1024, NOFILE 2048 Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65 You see that new limit is set. =========================================================================== Test =========================================================================== Cpu limit <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Script [root@book /home/kostjn]# cat test.sh #!/bin/sh for i in `jot 8 1`; do cpuset -l0 jexec 1 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 2 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 3 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 4 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 5 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 6 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 7 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 8 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 9 /a.out & done cpuset -l0 jexec 10 /a.out & Set class for all jail. [root@book /home/kostjn]# for i in `jot 10 1`; do ./jset.o $i jail128 ;done Set new jail limits, jid = 1 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 2 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 3 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 4 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 5 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 6 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 7 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 8 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 9 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 10 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 [root@book /home/kostjn]# jexec 1 bash [root@jail1 /]# cat cpu.c #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <math.h> int main(int argc,char *argv[]){ int64_t i,j=0; char *s; for (;;){ } } Run test.sh Result top last pid: 3513; load averages: 70.87, 37.58, 16.40 up 0+00:44:02 14:19:46 185 processes: 74 running, 111 sleeping CPU: 49.9% user, 0.0% nice, 0.0% system, 0.2% interrupt, 49.9% idle Mem: 139M Active, 24M Inact, 47M Wired, 192K Cache, 29M Buf, 1785M Free Swap: 4044M Total, 4044M Free PID JID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAN 3502 10 root 1 97 0 1480K 1244K CPU0 0 0:13 8.79% a.out 3474 6 root 1 97 0 1480K 1244K RUN 0 0:04 4.69% a.out 3431 2 root 1 96 0 1480K 1244K RUN 0 0:03 4.30% a.out 3454 4 root 1 97 0 1480K 1244K RUN 0 0:03 4.05% a.out 3422 1 root 1 96 0 1480K 1244K RUN 0 0:04 3.86% a.out 3482 7 root 1 97 0 1480K 1244K RUN 0 0:03 3.86% a.out 3447 3 root 1 97 0 1480K 1244K RUN 0 0:03 3.86% a.out 3429 1 root 1 96 0 1480K 1244K RUN 0 0:03 3.66% a.out 3485 8 root 1 97 0 1480K 1244K RUN 0 0:05 3.56% a.out 3424 1 root 1 96 0 1480K 1244K RUN 0 0:04 3.56% a.out 3464 5 root 1 97 0 1480K 1244K RUN 0 0:02 3.56% a.out 3438 2 root 1 96 0 1480K 1244K RUN 0 0:03 3.47% a.out 3494 9 root 1 96 0 1480K 1244K RUN 0 0:03 3.27% a.out 3497 9 root 1 97 0 1480K 1244K RUN 0 0:05 3.17% a.out 3433 2 root 1 96 0 1480K 1244K RUN 0 0:03 2.88% a.out 3428 1 root 1 96 0 1480K 1244K RUN 0 0:02 2.88% a.out 3487 8 root 1 97 0 1480K 1244K RUN 0 0:04 2.78% a.out ps auxwwww -ojid | more root 3502 9.0 0.1 1480 1244 v2 RJ 2:15PM 0:07.40 /a.out 10 root 3476 4.4 0.1 1480 1244 v2 RJ 2:15PM 0:04.38 /a.out 7 root 3480 4.1 0.1 1480 1244 v2 RJ 2:15PM 0:03.02 /a.out 7 root 3498 3.9 0.1 1480 1244 v2 RJ 2:15PM 0:04.00 /a.out 9 root 3429 3.7 0.1 1480 1244 v2 RJ 2:15PM 0:01.38 /a.out 1 root 3487 3.6 0.1 1480 1244 v2 RJ 2:15PM 0:03.32 /a.out 8 root 3452 3.5 0.1 1480 1244 v2 RJ 2:15PM 0:01.37 /a.out 4 root 3463 3.5 0.1 1480 1244 v2 RJ 2:15PM 0:01.65 /a.out 5 root 3472 3.3 0.1 1480 1244 v2 RJ 2:15PM 0:02.63 /a.out 6 root 3437 3.2 0.1 1480 1244 v2 RJ 2:15PM 0:01.93 /a.out 2 root 3494 3.0 0.1 1480 1244 v2 RJ 2:15PM 0:02.92 /a.out 9 root 3500 3.0 0.1 1480 1244 v2 RJ 2:15PM 0:03.63 /a.out 9 We see that jail 10 (1 thread), used ~10 % cpu under heavy load. <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Resourse compute [root@book /home/kostjn]# ./jset.o 1 jail64 Set new jail limits, jid = 1 Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512 [root@book /home/kostjn]# ./jget.o 1 Jail limits and rusage, jid = 1 Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512 Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65 [root@book /home/kostjn]# [root@book /home/kostjn]# jexec 1 bash [root@jail1 /]# apachectl stop /usr/local/sbin/apachectl stop: httpd stopped [root@jail1 /]# exit [root@book /home/kostjn]# ./jget.o 1 Jail limits and rusage, jid = 1 Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512 Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Resource limit [root@book /home/kostjn]# ./jget.o 1 Jail limits and rusage, jid = 1 Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512 Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24 [root@book /home/kostjn]# jexec 1 bash [root@jail1 /]# cat mem.c #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <math.h> int main(int argc,char *argv[]){ int64_t i,j=0; char *s; for (i=0; i < 1000 ;i++){ s = malloc(100000 * sizeof(char)); } sleep(1000); } [root@jail1 /]# cc mem.c && ./a.out & [1] 1320 [root@jail1 /]# ls bash: fork: Cannot allocate memory [root@jail1 /]# exit [root@book /home/kostjn]# ./jget.o 1 Jail limits and rusage, jid = 1 Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512 Usage: CPU 1, MEM 103M, NPROC 5, NOFILE 31 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< We see that jail exceed memory limit. And new fork, mmap syscall not permitted. If you set sysctl [root@book /home/kostjn]# sysctl security.jail.limit.memory_exceed_kill=1 security.jail.limit.memory_exceed_kill: 1 -> 1 [root@book /home/kostjn]# ./jget.o 1 Jail limits and rusage, jid = 1 Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512 Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24 [root@book /home/kostjn]# jexec 1 bash [root@jail1 /]# ./a.out Killed: 9 [root@jail1 /]# exit [root@book /home/kostjn]# tail -n 1 /var/log/messages May 20 14:10:17 book kernel: pid 1337 (a.out), uid 0, jid 1 was killed: Prison e xceed memory limit <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< If you attempt set nonexisten class, limit set to infinity. [root@book /home/kostjn]# ./jset.o 1 jail123 Set new jail limits, jid = 1 Limits: CPU 9223372036854775807, MEM 20M, NPROC 9223372036854775807, NOFILE 9223 372036854775807 =========================================================================== Problem =========================================================================== If you have problem in this patch. Add to kernel config options KTR options KTR_ENTRIES=1024 options KTR_COMPILE=(KTR_PROC|KTR_JAIL|KTR_SCHED|KTR_RUNQ|KTR_LOCK|KTR_CONTENTIO N) options KTR_MASK=KTR_JAIL options KTR_CPUMASK=0x3 options KTR_VERBOSE options PRINTF_BUFR_SIZE=128 Rebuild kernel. Reboot. Set sysctl sysctl debug.ktr.mask=65536 and check /var/log/messages
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4A1420BA.5050406>