From owner-freebsd-hackers@FreeBSD.ORG Wed May 20 15:51:45 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1877B106566B for ; Wed, 20 May 2009 15:51:45 +0000 (UTC) (envelope-from kostjn@peterhost.ru) Received: from fb0.z8.ru (fb0.z8.ru [80.93.58.95]) by mx1.freebsd.org (Postfix) with ESMTP id 813338FC1B for ; Wed, 20 May 2009 15:51:44 +0000 (UTC) (envelope-from kostjn@peterhost.ru) Received: from mail.z8.ru ([80.93.58.56]) by fb0.z8.ru with esmtp (Exim 4.69 (FreeBSD)) (envelope-from ) id 1M6nqF-000Idz-6Z for freebsd-hackers@freebsd.org; Wed, 20 May 2009 19:36:27 +0400 Received: from [85.235.196.139] (helo=kostjn.pht) by mail.z8.ru with esmtpa (Exim 4.67 (FreeBSD)) (envelope-from ) id 1M6nq2-000E9o-V6 for freebsd-hackers@freebsd.org; Wed, 20 May 2009 19:36:15 +0400 Message-ID: <4A1423D9.30105@peterhost.ru> Date: Wed, 20 May 2009 19:38:01 +0400 From: =?UTF-8?B?0JzQtdC90YzRiNC40LrQvtCyINCa0L7QvdGB0YLQsNC90YLQuNC9?= User-Agent: Thunderbird 2.0.0.18 (X11/20090328) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org References: <49ED55FF.5080306@peterhost.ru> In-Reply-To: <49ED55FF.5080306@peterhost.ru> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Jail limits under CURRENT X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2009 15:51:45 -0000 Меньшиков Константин wrote: Hi. I`m rewrite jail limit patch under CURRENT. New patch limited CPU, memory, filedesc, process. And allow change limit on the fly You can download tar.gz from http://kostjn.spb.ru/patch-jail-limit-8CURRENT.tar.gz =========================================================================== How to use. =========================================================================== Build cvsup CURRENT cd /usr/src patch -p0 < patch-jail-limit-8CURRENT make buildkernel make buildworld make installkernel reboot make installworld Create new entry in login.conf, for example class jail128 jail128:\ :cputime=10:\ :memoryuse=128M:\ :maxproc=256:\ :openfiles=1024:\ :tc=default: Cputime is percent on 1 core. Openfiles is sum filedesc for all proc in jail. Create new jail. ... Add in /etc/rc.conf jail_test_flags="-Ljail128" Run new jail /etc/rc.d/jail start test =========================================================================== Sysctl =========================================================================== Added sysctl [root@book ~]# sysctl security.jail.limit security.jail.limit.enable: 1 security.jail.limit.memory_exceed_kill: 0 [root@book ~]# sysctl -d security.jail.limit security.jail.limit: Jail limit security.jail.limit.enable: Enable jail limit security.jail.limit.memory_exceed_kill: Kill biggest proc in jail, if jail excee d memory limit =========================================================================== Jset and Jget =========================================================================== jset and jget is program for set new jail limit and get current limit Example [root@book ~]# cat /etc/rc.conf | grep jail2 jail_list="jail1 jail2 jail3 jail4 jail5 jail6 jail7 jail8 jail9 jail10" jail_jail2_rootdir="/usr/jails/jail2/" jail_jail2_hostname="jail2.book.pht" jail_jail2_interface="re0" jail_jail2_ip="192.168.200.22" jail_jail2_flags="-Ljail64" [root@book ~]# /etc/rc.d/jail start jail2 Configuring jails:. Starting jails: jail2.book.pht. [root@book ~]# cd ~kostjn/ [root@book /home/kostjn]# ./jget.o 1 Jail limits and rusage, jid = 1 Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512 Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65 [root@book /home/kostjn]# ./jset.o 1 jail2048 Set new jail limits, jid = 1 Limits: CPU 30, MEM 2048M, NPROC 1024, NOFILE 2048 [root@book /home/kostjn]# ./jget.o 1 Jail limits and rusage, jid = 1 Limits: CPU 30, MEM 2048M, NPROC 1024, NOFILE 2048 Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65 You see that new limit is set. =========================================================================== Test =========================================================================== Cpu limit <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Script [root@book /home/kostjn]# cat test.sh #!/bin/sh for i in `jot 8 1`; do cpuset -l0 jexec 1 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 2 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 3 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 4 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 5 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 6 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 7 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 8 /a.out & done for i in `jot 8 1`; do cpuset -l0 jexec 9 /a.out & done cpuset -l0 jexec 10 /a.out & Set class for all jail. [root@book /home/kostjn]# for i in `jot 10 1`; do ./jset.o $i jail128 ;done Set new jail limits, jid = 1 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 2 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 3 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 4 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 5 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 6 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 7 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 8 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 9 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 Set new jail limits, jid = 10 Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024 [root@book /home/kostjn]# jexec 1 bash [root@jail1 /]# cat cpu.c #include #include #include #include int main(int argc,char *argv[]){ int64_t i,j=0; char *s; for (;;){ } } Run test.sh Result top last pid: 3513; load averages: 70.87, 37.58, 16.40 up 0+00:44:02 14:19:46 185 processes: 74 running, 111 sleeping CPU: 49.9% user, 0.0% nice, 0.0% system, 0.2% interrupt, 49.9% idle Mem: 139M Active, 24M Inact, 47M Wired, 192K Cache, 29M Buf, 1785M Free Swap: 4044M Total, 4044M Free PID JID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAN 3502 10 root 1 97 0 1480K 1244K CPU0 0 0:13 8.79% a.out 3474 6 root 1 97 0 1480K 1244K RUN 0 0:04 4.69% a.out 3431 2 root 1 96 0 1480K 1244K RUN 0 0:03 4.30% a.out 3454 4 root 1 97 0 1480K 1244K RUN 0 0:03 4.05% a.out 3422 1 root 1 96 0 1480K 1244K RUN 0 0:04 3.86% a.out 3482 7 root 1 97 0 1480K 1244K RUN 0 0:03 3.86% a.out 3447 3 root 1 97 0 1480K 1244K RUN 0 0:03 3.86% a.out 3429 1 root 1 96 0 1480K 1244K RUN 0 0:03 3.66% a.out 3485 8 root 1 97 0 1480K 1244K RUN 0 0:05 3.56% a.out 3424 1 root 1 96 0 1480K 1244K RUN 0 0:04 3.56% a.out 3464 5 root 1 97 0 1480K 1244K RUN 0 0:02 3.56% a.out 3438 2 root 1 96 0 1480K 1244K RUN 0 0:03 3.47% a.out 3494 9 root 1 96 0 1480K 1244K RUN 0 0:03 3.27% a.out 3497 9 root 1 97 0 1480K 1244K RUN 0 0:05 3.17% a.out 3433 2 root 1 96 0 1480K 1244K RUN 0 0:03 2.88% a.out 3428 1 root 1 96 0 1480K 1244K RUN 0 0:02 2.88% a.out 3487 8 root 1 97 0 1480K 1244K RUN 0 0:04 2.78% a.out ps auxwwww -ojid | more root 3502 9.0 0.1 1480 1244 v2 RJ 2:15PM 0:07.40 /a.out 10 root 3476 4.4 0.1 1480 1244 v2 RJ 2:15PM 0:04.38 /a.out 7 root 3480 4.1 0.1 1480 1244 v2 RJ 2:15PM 0:03.02 /a.out 7 root 3498 3.9 0.1 1480 1244 v2 RJ 2:15PM 0:04.00 /a.out 9 root 3429 3.7 0.1 1480 1244 v2 RJ 2:15PM 0:01.38 /a.out 1 root 3487 3.6 0.1 1480 1244 v2 RJ 2:15PM 0:03.32 /a.out 8 root 3452 3.5 0.1 1480 1244 v2 RJ 2:15PM 0:01.37 /a.out 4 root 3463 3.5 0.1 1480 1244 v2 RJ 2:15PM 0:01.65 /a.out 5 root 3472 3.3 0.1 1480 1244 v2 RJ 2:15PM 0:02.63 /a.out 6 root 3437 3.2 0.1 1480 1244 v2 RJ 2:15PM 0:01.93 /a.out 2 root 3494 3.0 0.1 1480 1244 v2 RJ 2:15PM 0:02.92 /a.out 9 root 3500 3.0 0.1 1480 1244 v2 RJ 2:15PM 0:03.63 /a.out 9 We see that jail 10 (1 thread), used ~10 % cpu under heavy load. <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Resourse compute [root@book /home/kostjn]# ./jset.o 1 jail64 Set new jail limits, jid = 1 Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512 [root@book /home/kostjn]# ./jget.o 1 Jail limits and rusage, jid = 1 Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512 Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65 [root@book /home/kostjn]# [root@book /home/kostjn]# jexec 1 bash [root@jail1 /]# apachectl stop /usr/local/sbin/apachectl stop: httpd stopped [root@jail1 /]# exit [root@book /home/kostjn]# ./jget.o 1 Jail limits and rusage, jid = 1 Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512 Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Resource limit [root@book /home/kostjn]# ./jget.o 1 Jail limits and rusage, jid = 1 Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512 Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24 [root@book /home/kostjn]# jexec 1 bash [root@jail1 /]# cat mem.c #include #include #include #include int main(int argc,char *argv[]){ int64_t i,j=0; char *s; for (i=0; i < 1000 ;i++){ s = malloc(100000 * sizeof(char)); } sleep(1000); } [root@jail1 /]# cc mem.c && ./a.out & [1] 1320 [root@jail1 /]# ls bash: fork: Cannot allocate memory [root@jail1 /]# exit [root@book /home/kostjn]# ./jget.o 1 Jail limits and rusage, jid = 1 Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512 Usage: CPU 1, MEM 103M, NPROC 5, NOFILE 31 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< We see that jail exceed memory limit. And new fork, mmap syscall not permitted. If you set sysctl [root@book /home/kostjn]# sysctl security.jail.limit.memory_exceed_kill=1 security.jail.limit.memory_exceed_kill: 1 -> 1 [root@book /home/kostjn]# ./jget.o 1 Jail limits and rusage, jid = 1 Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512 Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24 [root@book /home/kostjn]# jexec 1 bash [root@jail1 /]# ./a.out Killed: 9 [root@jail1 /]# exit [root@book /home/kostjn]# tail -n 1 /var/log/messages May 20 14:10:17 book kernel: pid 1337 (a.out), uid 0, jid 1 was killed: Prison e xceed memory limit <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< If you attempt set nonexisten class, limit set to infinity. [root@book /home/kostjn]# ./jset.o 1 jail123 Set new jail limits, jid = 1 Limits: CPU 9223372036854775807, MEM 20M, NPROC 9223372036854775807, NOFILE 9223 372036854775807 =========================================================================== Problem =========================================================================== If you have problem in this patch. Add to kernel config options KTR options KTR_ENTRIES=1024 options KTR_COMPILE=(KTR_PROC|KTR_JAIL|KTR_SCHED|KTR_RUNQ|KTR_LOCK|KTR_CONTENTIO N) options KTR_MASK=KTR_JAIL options KTR_CPUMASK=0x3 options KTR_VERBOSE options PRINTF_BUFR_SIZE=128 Rebuild kernel. Reboot. Set sysctl sysctl debug.ktr.mask=65536 and check /var/log/messages