From owner-freebsd-hackers@freebsd.org Fri Apr 19 19:58:52 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5FE041577256 for ; Fri, 19 Apr 2019 19:58:52 +0000 (UTC) (envelope-from andreas.sommer87@googlemail.com) Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 43B548F730; Fri, 19 Apr 2019 19:58:51 +0000 (UTC) (envelope-from andreas.sommer87@googlemail.com) Received: by mail-wr1-x430.google.com with SMTP id t17so8024211wrw.13; Fri, 19 Apr 2019 12:58:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:cc:message-id:date:user-agent :mime-version:content-language:content-transfer-encoding; bh=PE8RH0dPIgnAURvsPbbe9tqgjhKqyzZxyZ+Gxsh3H0s=; b=GN5n43hNdgNP55CDE94W17nSYKOG1x/kaAG3iCDDcVC+dJhuNnXjtJOLMAWfcZwcgg qhpr2vLJ291c+5sQVAtqpjsWF4nWJYsmfABK4pzaH1RiA9zglNM4/BWD/Dg1L5aXsj1B zNp+sXiY09BHfNc1/qRYT8Ko1iGgjqBtPiIQVOOpDOTQn+nRlPjo7M196kqAIN9/hRWW c2jKTCmWHiMXJw0dMjEnIcNylWz8zpRb3EGye+901kAeNjg4SKCEb0Nsvy/Ncw2HoVsg 5pivlfI8Uz+WCySOFHbyf6lhpcHJXmcm0QefBRs7uNh4+0IjImUT7RVUMyCM+bjkInAT zOWw== X-Gm-Message-State: APjAAAU5oOnI6bW8rofWoIWhxWeNMB2FpoWpmx+g7g5TXvklqGOu9DDO lYN6t9/PMgC6MoXsHhO5tA4oelxurkY= X-Google-Smtp-Source: APXvYqytExNq+bTW8SIChmwrH+BYCpIxof4/L9O/GoQeZcj0k8xOGytXHtWAGl+ukAfgDhZdHLe9bQ== X-Received: by 2002:adf:f58f:: with SMTP id f15mr4095110wro.183.1555703929762; Fri, 19 Apr 2019 12:58:49 -0700 (PDT) Received: from asommer-mac.local ([2a02:2455:41f:4e00:f5ea:abaa:e09a:4d33]) by smtp.googlemail.com with ESMTPSA id j9sm6358383wrr.93.2019.04.19.12.58.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Apr 2019 12:58:48 -0700 (PDT) To: freebsd-hackers@freebsd.org From: Andreas Sommer Subject: Poudriere nested jail configuration - want explanation for each setting Cc: Bryan Drewery , Baptiste Daroussin Message-ID: <1c310e2a-0f3f-f455-71d5-2d03213c4773@googlemail.com> Date: Fri, 19 Apr 2019 21:58:46 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 43B548F730 X-Spamd-Bar: ------ X-Spamd-Result: default: False [-6.77 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[googlemail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[googlemail.com]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[googlemail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[0.3.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; IP_SCORE(-2.82)[ip: (-9.40), ipnet: 2a00:1450::/32(-2.38), asn: 15169(-2.25), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.95)[-0.946,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 19:58:52 -0000 Hi all (main poudriere authors in CC, hoping this is etiquette-friendly), I'm currently writing a tutorial that includes setting up Poudriere within a jail (nested). While the appropriate settings are nicely listed at https://github.com/freebsd/poudriere/wiki/poudriere_in_jail, I'd like to get clarity which ones are actually needed and *why*. Of course I did some research in freebsd/poudriere code which resulted in the write-up I have until now (feel free to improve grammar/text as well): > Jail config: > ip4.addr += "lo0|127.0.0.3"; > exec.poststart = "/sbin/zfs jail buildbot-worker0 zroot/pdr/w0"; > allow.chflags; > allow.mount; > allow.mount.devfs; > allow.mount.nullfs; > allow.mount.procfs; > allow.mount.tmpfs; > allow.mount.zfs; # only needed if you use ZFS > allow.raw_sockets; # optional > allow.socket_af; # optional > allow.sysvipc; # optional > children.max=16; > enforce_statfs=1; > [...] > - `ip4.addr += "lo0|127.0.0.3"` adds another IPv4 address to the jail. You will later configure Poudriere's `LOIP4` variable in order to assign this loopback address to build jails which are not supposed to talk to the Internet or other machines in your network, such as during the `build` phase. If you ever have a build which requires Internet access during build, Poudriere supports a variable `ALLOW_NETWORKING_PACKAGES` as workaround. However, you should prefer the clean way, i.e. performing downloads and other Internet-facing tasks earlier, in the `fetch` phase which is permitted Internet access. > - `allow.chflags` allows Poudriere to render certain system files in the build jail immutable, since builds are not supposed to overwrite e.g. `/bin/sh` > - `allow.mount` and the other `allow.mount.*` options enable Poudriere to mount certain required filesystems into the build jails > - `allow.raw_sockets` (permit use of raw sockets) and `allow.socket_af` (use of any socket address family) are applied to the Internet-capable build jails. This is mostly only helpful so that you can run tools like `ping` in interactive mode i.e. when entering a build jail to debug problems. You could disable these permissions. > - `allow.sysvipc` is actually deprecated in favor of three separate settings `sysvmsg`/`sysvsem`/`sysvshm` to restrict jails to only see their own shared memory objects (via "SYS V" IPC primitives). However, Poudriere can only pass on `allow.sysvipc` to build jails because it cannot read the relevant sysctl information for the three separate parameters (as of FreeBSD 11.2). With this deprecated configuration, the jail could read shared memory of processes outside the jail. This is only relevant for certain software that depends on IPC features, like PostgreSQL, so chances are small for this to affect security. You can remove this configuration unless you depend on a port that requires it during build. > - `children.max=16` allows 16 sub-jails below the worker jail. You can raise this number later if you have a lot of CPUs and Poudriere tries to create more build jails than permitted. Note that each Poudriere build will try to create a reference jail and 2 build jails per "job", and its default is to use the number of CPUs (as output by `sysctl -n hw.ncpu`) as job count. > - `enforce_statfs=1` is required together with `allow.mount` in order to mount certain filesystems. Can you confirm the descriptions? Especially for the `allow.sysvipc` topic, I digged a little more: There's `security.jail.sysvipc_allowed={0,1}` which is passed on to build jails by Poudriere. And for the 3 non-obsolete jail settings `sysvmsg`/`sysvsem`/`sysvshm`, sysctl has to offer `kern.features.sysv_{shm,sem,msg}` and `security.jail.param.sysv{shm,sem,msg}.` (with a dubious dot at the end!). And here's the problem: no matter whether I configure the jail with `{sysvmsg,sysvsem,sysvshm} = "new";` or leave out any SYSVIPC setting (= disallow altogether), the sysctls stay the same at: > $ sudo jexec myjail sysctl -a | grep .sysv > kern.features.sysv_shm: 1 <---- these are always 1 > kern.features.sysv_sem: 1 > kern.features.sysv_msg: 1 > security.jail.param.sysvshm.: 0 <---- these are always 0 > security.jail.param.sysvsem.: 0 > security.jail.param.sysvmsg.: 0 > security.jail.param.allow.sysvipc: 0 <---- these are unrelated to the 3 separate settings > security.jail.sysvipc_allowed: 0 On the other hand, I confirmed the `{sysvmsg,sysvsem,sysvshm} = "new";` settings do what they should (using FreeBSD 11.2 and PostgreSQL) – prohibit the jail from seeing outer shared objects. Therefore I concluded that Poudriere simply has no visibility whether the "poudriere jail" has those features, and hence cannot pass those on to its nested build jails. It does that for `allow.sysvipc`, though, since the sysctl `security.jail.sysvipc_allowed` shows the correct value within the jail. Reference: poudriere/src/share/poudriere/include/common.sh.freebsd > if [ ${JAILED} -eq 0 ] || \ > [ $(sysctl -n security.jail.sysvipc_allowed) -eq 1 ]; then > JAIL_PARAMS="${JAIL_PARAMS:+${JAIL_PARAMS} }allow.sysvipc" > fi Thanks in advance, Andreas