From owner-freebsd-jail@freebsd.org  Wed Jul  6 12:13:38 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id A3B94B75437
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Wed,  6 Jul 2016 12:13:38 +0000 (UTC)
 (envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org
 [IPv6:2001:1900:2254:206a::16:76])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 92F131517
 for <freebsd-jail@FreeBSD.org>; Wed,  6 Jul 2016 12:13:38 +0000 (UTC)
 (envelope-from bugzilla-noreply@freebsd.org)
Received: from bugs.freebsd.org ([127.0.1.118])
 by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u66CDbYR011899
 for <freebsd-jail@FreeBSD.org>; Wed, 6 Jul 2016 12:13:38 GMT
 (envelope-from bugzilla-noreply@freebsd.org)
From: bugzilla-noreply@freebsd.org
To: freebsd-jail@FreeBSD.org
Subject: [Bug 209112] /usr/sbin/jail jails fail to launch with possible race
 when jails mount common dir with nullfs
Date: Wed, 06 Jul 2016 12:13:37 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: bin
X-Bugzilla-Version: 10.3-RELEASE
X-Bugzilla-Keywords: 
X-Bugzilla-Severity: Affects Some People
X-Bugzilla-Who: djn@araxis.com
X-Bugzilla-Status: New
X-Bugzilla-Resolution: 
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: freebsd-jail@FreeBSD.org
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: cc
Message-ID: <bug-209112-9824-wWooAWbppG@https.bugs.freebsd.org/bugzilla/>
In-Reply-To: <bug-209112-9824@https.bugs.freebsd.org/bugzilla/>
References: <bug-209112-9824@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 06 Jul 2016 12:13:38 -0000

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209112

djn@araxis.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djn@araxis.com

--- Comment #4 from djn@araxis.com ---
I am also seeing this issue with FreeBSD 10.3-Release-p4 =E2=80=93 I have s=
ix jails,
and only two or three start on boot (two fail due to a dependency on another
jail that fails to start). I encounter no problems when starting the jails
post-boot.

I added logging to /etc/rc.d/jail in a similar manner to the original poste=
r,
and see this output:

----
mount_nullfs: /jails/pgsql/data: mount_nullfs: /jails/hg/data: Operation not
supported by device
Operation not supported by device
jail: pgsql: /sbin/mount -t nullfs -o rw /data/pgsql /jails/pgsql/data: fai=
led
jail: hg: /sbin/mount -t nullfs -o rw /data/hg /jails/hg/data: failed
jail: support: skipped
jail: logic: skipped
www-bhs-0: created
backup-bhs-0: created

...
----

Here is my jail.conf:

----
# START BLOCK: Araxis jail defaults -- DO NOT EDIT
exec.start =3D "/bin/sh /etc/rc";
exec.stop =3D "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
mount.fdescfs;
allow.noset_hostname;
host =3D new;

# temporarily allow during development/testing
allow.raw_sockets;
# END BLOCK: Araxis jail defaults
# START BLOCK: Araxis jail settings for jail www-bhs-0.araxis.net -- DO NOT
EDIT
www-bhs-0 {
  mount.fstab =3D "/jails/.fstabs/www-bhs-0";
  path =3D "/jails/www-bhs-0";
  host.hostname =3D www-bhs-0.araxis.net;
  ip4.addr =3D "lo1|10.11.11.2/32";
  ip6.addr =3D "ix0|2607:5300:60:9b9c::2/64";
}
# END BLOCK: Araxis jail settings for jail www-bhs-0.araxis.net
# START BLOCK: Araxis jail settings for jail backup-bhs-0.araxis.net -- DO =
NOT
EDIT
backup-bhs-0 {
  mount.fstab =3D "/jails/.fstabs/backup-bhs-0";
  path =3D "/jails/backup-bhs-0";
  host.hostname =3D backup-bhs-0.araxis.net;
  ip4.addr =3D "lo2|10.12.12.8/32";
  enforce_statfs =3D 1;
  allow.mount;
  allow.mount.zfs;
  exec.poststart =3D "/sbin/zfs set jailed=3Don zroot/bkup && /sbin/zfs jail
${name} zroot/bkup && jexec ${name} /sbin/zfs mount -a";
  exec.prestop =3D "/sbin/zfs unjail ${name} zroot/bkup && /sbin/zfs set
jailed=3Doff zroot/bkup";
}
# END BLOCK: Araxis jail settings for jail backup-bhs-0.araxis.net
# START BLOCK: Araxis jail settings for jail pgsql.araxis.net -- DO NOT EDIT
pgsql {
  mount.fstab =3D "/jails/.fstabs/pgsql";
  path =3D "/jails/pgsql";
  host.hostname =3D pgsql.araxis.net;
  ip4.addr =3D "lo2|10.12.12.4/32";
  allow.sysvipc;
}
# END BLOCK: Araxis jail settings for jail pgsql.araxis.net
# START BLOCK: Araxis jail settings for jail support.araxis.net -- DO NOT E=
DIT
support {
  mount.fstab =3D "/jails/.fstabs/support";
  path =3D "/jails/support";
  host.hostname =3D support.araxis.net;
  ip4.addr =3D "lo2|10.12.12.5/32";
  depend =3D pgsql;
}
# END BLOCK: Araxis jail settings for jail support.araxis.net
# START BLOCK: Araxis jail settings for jail logic.araxis.com -- DO NOT EDIT
logic {
  mount.fstab =3D "/jails/.fstabs/logic";
  path =3D "/jails/logic";
  host.hostname =3D logic.araxis.com;
  ip4.addr =3D "lo2|10.12.12.6/32";
  depend =3D pgsql;
}
# END BLOCK: Araxis jail settings for jail logic.araxis.com
# START BLOCK: Araxis jail settings for jail hg.araxis.net -- DO NOT EDIT
hg {
  mount.fstab =3D "/jails/.fstabs/hg";
  path =3D "/jails/hg";
  host.hostname =3D hg.araxis.net;
  ip4.addr =3D "lo2|10.12.12.3/32";
}
# END BLOCK: Araxis jail settings for jail hg.araxis.net
----

The various fstab files are nearly identical. Here=E2=80=99s the one
(/jails/.fstabs/pgsql) for the pgsql jail, which always fails to start on b=
oot:

----
# Device    Mountpoint    FStype    Options   Dump    Pass#
/data/pgsql   /jails/pgsql/data    nullfs    rw    0   0
----

--=20
You are receiving this mail because:
You are the assignee for the bug.=

From owner-freebsd-jail@freebsd.org  Wed Jul  6 12:20:03 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id B1272B7565B
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Wed,  6 Jul 2016 12:20:03 +0000 (UTC)
 (envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org
 [IPv6:2001:1900:2254:206a::16:76])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id A025A17F5
 for <freebsd-jail@FreeBSD.org>; Wed,  6 Jul 2016 12:20:03 +0000 (UTC)
 (envelope-from bugzilla-noreply@freebsd.org)
Received: from bugs.freebsd.org ([127.0.1.118])
 by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u66CK3Ve093049
 for <freebsd-jail@FreeBSD.org>; Wed, 6 Jul 2016 12:20:03 GMT
 (envelope-from bugzilla-noreply@freebsd.org)
From: bugzilla-noreply@freebsd.org
To: freebsd-jail@FreeBSD.org
Subject: [Bug 209112] /usr/sbin/jail jails fail to launch with possible race
 when jails mount common dir with nullfs
Date: Wed, 06 Jul 2016 12:20:03 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: bin
X-Bugzilla-Version: 10.3-RELEASE
X-Bugzilla-Keywords: 
X-Bugzilla-Severity: Affects Some People
X-Bugzilla-Who: djn@araxis.com
X-Bugzilla-Status: New
X-Bugzilla-Resolution: 
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: freebsd-jail@FreeBSD.org
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: 
Message-ID: <bug-209112-9824-AmdmK6ryOL@https.bugs.freebsd.org/bugzilla/>
In-Reply-To: <bug-209112-9824@https.bugs.freebsd.org/bugzilla/>
References: <bug-209112-9824@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 06 Jul 2016 12:20:03 -0000

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209112

--- Comment #5 from djn@araxis.com ---
I should also like to add my voice to those requesting logging of boot-time
jail startup problems. The lack of any diagnostics cost me several hours of
time in tracking down this problem.

--=20
You are receiving this mail because:
You are the assignee for the bug.=

From owner-freebsd-jail@freebsd.org  Wed Jul  6 12:57:08 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id A4B2BB75DC0
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Wed,  6 Jul 2016 12:57:08 +0000 (UTC)
 (envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org
 [IPv6:2001:1900:2254:206a::16:76])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 865851885
 for <freebsd-jail@FreeBSD.org>; Wed,  6 Jul 2016 12:57:08 +0000 (UTC)
 (envelope-from bugzilla-noreply@freebsd.org)
Received: from bugs.freebsd.org ([127.0.1.118])
 by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u66Cv8RJ070501
 for <freebsd-jail@FreeBSD.org>; Wed, 6 Jul 2016 12:57:08 GMT
 (envelope-from bugzilla-noreply@freebsd.org)
From: bugzilla-noreply@freebsd.org
To: freebsd-jail@FreeBSD.org
Subject: [Bug 209112] /usr/sbin/jail jails fail to launch with possible race
 when jails mount common dir with nullfs
Date: Wed, 06 Jul 2016 12:57:08 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: bin
X-Bugzilla-Version: 10.3-RELEASE
X-Bugzilla-Keywords: 
X-Bugzilla-Severity: Affects Some People
X-Bugzilla-Who: djn@araxis.com
X-Bugzilla-Status: New
X-Bugzilla-Resolution: 
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: freebsd-jail@FreeBSD.org
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: 
Message-ID: <bug-209112-9824-XsRuL1qQiK@https.bugs.freebsd.org/bugzilla/>
In-Reply-To: <bug-209112-9824@https.bugs.freebsd.org/bugzilla/>
References: <bug-209112-9824@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 06 Jul 2016 12:57:08 -0000

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209112

--- Comment #6 from djn@araxis.com ---
I have what seems (on first try, at least) to be a viable alternative
workaround that is (somewhat) less icky than adding artificial dependencies
between jails. Simply add the following two lines to /etc/rc.conf (or
/etc/rc.conf.d/jail):

jail_parallel_start=3D"NO"
jail_list=3D"list of all jails to start"

Specifying the jail_list explicitly means that the jail_parallel_start sett=
ing
can take effect, since the default _ALL case in jail_start() (which ignores
jail_parallel_start) is then bypassed.

--=20
You are receiving this mail because:
You are the assignee for the bug.=

From owner-freebsd-jail@freebsd.org  Wed Jul  6 16:50:02 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id E7031B75287
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Wed,  6 Jul 2016 16:50:02 +0000 (UTC)
 (envelope-from list1@gjunka.com)
Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client CN "msa1.earth.yoonka.com",
 Issuer "msa1.earth.yoonka.com" (not verified))
 by mx1.freebsd.org (Postfix) with ESMTPS id 792FC1AA2
 for <freebsd-jail@freebsd.org>; Wed,  6 Jul 2016 16:50:01 +0000 (UTC)
 (envelope-from list1@gjunka.com)
Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20])
 (authenticated bits=0)
 by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u66Gnqkg030411
 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO)
 for <freebsd-jail@freebsd.org>; Wed, 6 Jul 2016 16:49:52 GMT
 (envelope-from list1@gjunka.com)
To: freebsd-jail@freebsd.org
From: Grzegorz Junka <list1@gjunka.com>
Subject: Effective rule sets in a jail?
Message-ID: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com>
Date: Wed, 6 Jul 2016 16:49:52 +0000
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101
 Thunderbird/45.1.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 06 Jul 2016 16:50:03 -0000

I have the following in my jail.conf:

devfs_ruleset = 4;

vpn1 {
   ip4.addr = 10.70.5.254;
   ip4.addr += "tun0|10.70.5.1 10.70.5.254 mtu 1500 netmask 
255.255.255.255";
   interface = lagg0;
   devfs_ruleset = 5;
}

I expect that in the jail both rules 4 and 5 are active. How can I check 
that?

Grzegorz


From owner-freebsd-jail@freebsd.org  Thu Jul  7 04:04:44 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id BF55DB758DD
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Thu,  7 Jul 2016 04:04:44 +0000 (UTC)
 (envelope-from ultima1252@gmail.com)
Received: from mail-yw0-x22d.google.com (mail-yw0-x22d.google.com
 [IPv6:2607:f8b0:4002:c05::22d])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (Client CN "smtp.gmail.com",
 Issuer "Google Internet Authority G2" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 6C4A013AD
 for <freebsd-jail@freebsd.org>; Thu,  7 Jul 2016 04:04:44 +0000 (UTC)
 (envelope-from ultima1252@gmail.com)
Received: by mail-yw0-x22d.google.com with SMTP id i12so5050559ywa.1
 for <freebsd-jail@freebsd.org>; Wed, 06 Jul 2016 21:04:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=mime-version:in-reply-to:references:from:date:message-id:subject:to
 :cc; bh=zS1OQyP+Fho3y4s2969wxqZTa2NPIBuspy0iaKe9Ffs=;
 b=AZUF9GFtyGCH0W52gG9AI9ph3xK2JnRwa0ndbZ+AOEFPU7ROis0m/1a6GXy/qrNzEA
 9KB09LFBTeTBqlih4HoqjtGNqLrle4Ev4pbfs3NfEwWHY2a/0bLzgg6O7evhrPeMP3dy
 SZ0JJqvl/9aoEVRDxuoMx31y5pnNP5s7RZ7S2jpgPtQMNNn2fP6ol16zHlrkZ7IgMCeV
 qmQr2EEEKz1qXif363hdVZZe4xwW2gr44yL/+P5vKifpGVyjto7vH47BM5dgG+qnsRgH
 HKRVCU9BRY/AU2St74T0up5JH/8jTklaplQwK41ZB4J/mx9eS8FFpe3LSm5xHNKKs0Gp
 FfMw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:in-reply-to:references:from:date
 :message-id:subject:to:cc;
 bh=zS1OQyP+Fho3y4s2969wxqZTa2NPIBuspy0iaKe9Ffs=;
 b=VlQSh/Cvq0Hwsi+WQePCUIzAeKcEyJZEUtomhGTRR2kwucoStI0K0xOf83bIBJWHAx
 ArCmHgJxhvBDgPKP7hrPTNlrgch8C+lxJHgKTqZHnQHr7I+8QzXZ3bO6V7bmTev0hQUJ
 mKh3iU3OyjZ59syQ4lqBXc1nUosV4eCevHY6HDoND49GD4Hdy10/0EGLlkASMm1wmmtA
 V8pDDNYM+8JTeSmf6dz5lO41fFzEI89fpzh/w0EWDQPdYBmGWykf17D+IjL1bqlH7Fcy
 pPuQmjTxmj/HRte0iEIVIP6tnLqeFj8qWA7/46id6QELN75M1yf2CGMsFQ/b+lMeZmnJ
 tfew==
X-Gm-Message-State: ALyK8tKDd9vO5t0FLoXkiKDqouM29ncEZBAW00JoxPv9BjPBsJ5xcokDaifFed306KpjTLJZLXUssXLodZCngA==
X-Received: by 10.37.11.129 with SMTP id 123mr16568486ybl.55.1467864283535;
 Wed, 06 Jul 2016 21:04:43 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.129.5.216 with HTTP; Wed, 6 Jul 2016 21:04:43 -0700 (PDT)
In-Reply-To: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com>
References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com>
From: Ultima <ultima1252@gmail.com>
Date: Thu, 7 Jul 2016 00:04:43 -0400
Message-ID: <CANJ8om5R-BT=heC+giMTXFH8YQXhuPQZjQ_S-P1bQ1XBGS16uQ@mail.gmail.com>
Subject: Re: Effective rule sets in a jail?
To: Grzegorz Junka <list1@gjunka.com>
Cc: freebsd-jail@freebsd.org
Content-Type: text/plain; charset=UTF-8
X-Content-Filtered-By: Mailman/MimeDel 2.1.22
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 07 Jul 2016 04:04:44 -0000

Not so. The top variable, devfs_ruleset = 4 is being set as the default for
all jails. The devfs_ruleset = 5 inside the brackets is changing the
default value.

How to check what ruleset is mounted? That is a great question. I'm not
sure of an easy way to check other than verifying the /dev directory inside
the jail.

Ultima

From owner-freebsd-jail@freebsd.org  Thu Jul  7 07:53:39 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 45D4DB21F54
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Thu,  7 Jul 2016 07:53:39 +0000 (UTC)
 (envelope-from 000.fbsd@quip.cz)
Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 09D5D1845
 for <freebsd-jail@freebsd.org>; Thu,  7 Jul 2016 07:53:38 +0000 (UTC)
 (envelope-from 000.fbsd@quip.cz)
Received: from elsa.codelab.cz (localhost [127.0.0.1])
 by elsa.codelab.cz (Postfix) with ESMTP id B1A9C284E7;
 Thu,  7 Jul 2016 09:53:29 +0200 (CEST)
Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz
 [86.49.16.209])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (No client certificate requested)
 by elsa.codelab.cz (Postfix) with ESMTPSA id 00B5328454;
 Thu,  7 Jul 2016 09:53:28 +0200 (CEST)
Message-ID: <577E0A78.1040600@quip.cz>
Date: Thu, 07 Jul 2016 09:53:28 +0200
From: Miroslav Lachman <000.fbsd@quip.cz>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32
MIME-Version: 1.0
To: Ultima <ultima1252@gmail.com>, Grzegorz Junka <list1@gjunka.com>
CC: freebsd-jail@freebsd.org
Subject: Re: Effective rule sets in a jail?
References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com>
 <CANJ8om5R-BT=heC+giMTXFH8YQXhuPQZjQ_S-P1bQ1XBGS16uQ@mail.gmail.com>
In-Reply-To: <CANJ8om5R-BT=heC+giMTXFH8YQXhuPQZjQ_S-P1bQ1XBGS16uQ@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 07 Jul 2016 07:53:39 -0000

Ultima wrote on 07/07/2016 06:04:
> Not so. The top variable, devfs_ruleset = 4 is being set as the default for
> all jails. The devfs_ruleset = 5 inside the brackets is changing the
> default value.
>
> How to check what ruleset is mounted? That is a great question. I'm not
> sure of an easy way to check other than verifying the /dev directory inside
> the jail.

There is no way to set more than one devfs rule to jail AFAIK.
You can see the rule number in output of jls -s or jls -n.

Miroslav Lachman


From owner-freebsd-jail@freebsd.org  Thu Jul  7 08:41:36 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5FCDEB769FA
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Thu,  7 Jul 2016 08:41:36 +0000 (UTC)
 (envelope-from list1@gjunka.com)
Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client CN "msa1.earth.yoonka.com",
 Issuer "msa1.earth.yoonka.com" (not verified))
 by mx1.freebsd.org (Postfix) with ESMTPS id 092331D08
 for <freebsd-jail@freebsd.org>; Thu,  7 Jul 2016 08:41:35 +0000 (UTC)
 (envelope-from list1@gjunka.com)
Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20])
 (authenticated bits=0)
 by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u678fW6v053455
 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO)
 for <freebsd-jail@freebsd.org>; Thu, 7 Jul 2016 08:41:33 GMT
 (envelope-from list1@gjunka.com)
Subject: Re: Effective rule sets in a jail?
References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com>
 <CANJ8om5R-BT=heC+giMTXFH8YQXhuPQZjQ_S-P1bQ1XBGS16uQ@mail.gmail.com>
 <577E0A78.1040600@quip.cz>
To: freebsd-jail@freebsd.org
From: Grzegorz Junka <list1@gjunka.com>
Message-ID: <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com>
Date: Thu, 7 Jul 2016 08:41:32 +0000
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101
 Thunderbird/45.1.1
MIME-Version: 1.0
In-Reply-To: <577E0A78.1040600@quip.cz>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 07 Jul 2016 08:41:36 -0000


On 07/07/2016 07:53, Miroslav Lachman wrote:
> Ultima wrote on 07/07/2016 06:04:
>> Not so. The top variable, devfs_ruleset = 4 is being set as the 
>> default for
>> all jails. The devfs_ruleset = 5 inside the brackets is changing the
>> default value.
>>
>> How to check what ruleset is mounted? That is a great question. I'm not
>> sure of an easy way to check other than verifying the /dev directory 
>> inside
>> the jail.
>
> There is no way to set more than one devfs rule to jail AFAIK.
> You can see the rule number in output of jls -s or jls -n.
>
> Miroslav Lachman
>

I was referring to this clause in the man document:

Descendant jails inherit the parent jail's devfs ruleset enforcement.

I thought that the outside rule is combined with the inside rule in the 
jail definition. But thanks for the hint about jls -s, it does shows the 
(single) active rule set (however without referring to the specific 
rules defined in devfs.rules or a combination of it).

Grzegorz

From owner-freebsd-jail@freebsd.org  Thu Jul  7 09:04:01 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8CA9CB74164
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Thu,  7 Jul 2016 09:04:01 +0000 (UTC)
 (envelope-from 000.fbsd@quip.cz)
Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 50FD81CF2
 for <freebsd-jail@freebsd.org>; Thu,  7 Jul 2016 09:04:00 +0000 (UTC)
 (envelope-from 000.fbsd@quip.cz)
Received: from elsa.codelab.cz (localhost [127.0.0.1])
 by elsa.codelab.cz (Postfix) with ESMTP id E4936284AE;
 Thu,  7 Jul 2016 11:03:56 +0200 (CEST)
Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz
 [86.49.16.209])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (No client certificate requested)
 by elsa.codelab.cz (Postfix) with ESMTPSA id C875D2848E;
 Thu,  7 Jul 2016 11:03:55 +0200 (CEST)
Message-ID: <577E1AFB.90100@quip.cz>
Date: Thu, 07 Jul 2016 11:03:55 +0200
From: Miroslav Lachman <000.fbsd@quip.cz>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32
MIME-Version: 1.0
To: Grzegorz Junka <list1@gjunka.com>, freebsd-jail@freebsd.org
Subject: Re: Effective rule sets in a jail?
References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com>
 <CANJ8om5R-BT=heC+giMTXFH8YQXhuPQZjQ_S-P1bQ1XBGS16uQ@mail.gmail.com>
 <577E0A78.1040600@quip.cz> <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com>
In-Reply-To: <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 07 Jul 2016 09:04:01 -0000

Grzegorz Junka wrote on 07/07/2016 10:41:


> I was referring to this clause in the man document:
>
> Descendant jails inherit the parent jail's devfs ruleset enforcement.

This is true for hierarchical "nested" jails = jail inside jail.
And inheriting doesn't mean merging.
You can't allow devices in descendant jail which are not allowed on parent.

> I thought that the outside rule is combined with the inside rule in the
> jail definition. But thanks for the hint about jls -s, it does shows the
> (single) active rule set (however without referring to the specific
> rules defined in devfs.rules or a combination of it).

You are mixing nested jails context with jail.conf context where 
"outside" definitions are the defaults for all jails which are not 
overriding those values with own values.

Miroslav Lachman

From owner-freebsd-jail@freebsd.org  Thu Jul  7 09:42:07 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7733AB74C5B
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Thu,  7 Jul 2016 09:42:07 +0000 (UTC)
 (envelope-from list1@gjunka.com)
Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client CN "msa1.earth.yoonka.com",
 Issuer "msa1.earth.yoonka.com" (not verified))
 by mx1.freebsd.org (Postfix) with ESMTPS id 1E83D1236
 for <freebsd-jail@freebsd.org>; Thu,  7 Jul 2016 09:42:06 +0000 (UTC)
 (envelope-from list1@gjunka.com)
Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20])
 (authenticated bits=0)
 by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u679g4Ff054596
 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO)
 for <freebsd-jail@freebsd.org>; Thu, 7 Jul 2016 09:42:05 GMT
 (envelope-from list1@gjunka.com)
Subject: Re: Effective rule sets in a jail?
To: freebsd-jail@freebsd.org
References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com>
 <CANJ8om5R-BT=heC+giMTXFH8YQXhuPQZjQ_S-P1bQ1XBGS16uQ@mail.gmail.com>
 <577E0A78.1040600@quip.cz> <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com>
 <577E1AFB.90100@quip.cz>
From: Grzegorz Junka <list1@gjunka.com>
Message-ID: <6ccead58-a38a-80a4-b5b8-a509c4271b8f@gjunka.com>
Date: Thu, 7 Jul 2016 09:42:04 +0000
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101
 Thunderbird/45.1.1
MIME-Version: 1.0
In-Reply-To: <577E1AFB.90100@quip.cz>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 07 Jul 2016 09:42:07 -0000


On 07/07/2016 09:03, Miroslav Lachman wrote:
> Grzegorz Junka wrote on 07/07/2016 10:41:
>
>
>> I was referring to this clause in the man document:
>>
>> Descendant jails inherit the parent jail's devfs ruleset enforcement.
>
> This is true for hierarchical "nested" jails = jail inside jail.
> And inheriting doesn't mean merging.
> You can't allow devices in descendant jail which are not allowed on 
> parent.
>
>> I thought that the outside rule is combined with the inside rule in the
>> jail definition. But thanks for the hint about jls -s, it does shows the
>> (single) active rule set (however without referring to the specific
>> rules defined in devfs.rules or a combination of it).
>
> You are mixing nested jails context with jail.conf context where 
> "outside" definitions are the defaults for all jails which are not 
> overriding those values with own values.
>
> Miroslav Lachman

OK, I am just an user, not very familiar with the terminology. For me 
(as a programmer) inheriting means overriding, so merging the more 
specific to the less specific declarations.

Does it mean that the "inheriting" works in nested declarations but 
doesn't take into account the default value? In other words, the default 
is just default unless it re-defined in a jail declaration. If that's 
the case then wouldn't be more clear to name the "outside" default 
declaration as default, e.g. "default_devfs_ruleset"? Then it would be 
more difficult to confuse the default with the one that can be inherited.

Grzegorz

From owner-freebsd-jail@freebsd.org  Thu Jul  7 10:06:43 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4EEC8B75193
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Thu,  7 Jul 2016 10:06:43 +0000 (UTC)
 (envelope-from 000.fbsd@quip.cz)
Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id AF4231AD3
 for <freebsd-jail@freebsd.org>; Thu,  7 Jul 2016 10:06:42 +0000 (UTC)
 (envelope-from 000.fbsd@quip.cz)
Received: from elsa.codelab.cz (localhost [127.0.0.1])
 by elsa.codelab.cz (Postfix) with ESMTP id 361322848C;
 Thu,  7 Jul 2016 12:06:34 +0200 (CEST)
Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz
 [86.49.16.209])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (No client certificate requested)
 by elsa.codelab.cz (Postfix) with ESMTPSA id 30B3B28412;
 Thu,  7 Jul 2016 12:06:33 +0200 (CEST)
Message-ID: <577E29A8.5000504@quip.cz>
Date: Thu, 07 Jul 2016 12:06:32 +0200
From: Miroslav Lachman <000.fbsd@quip.cz>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32
MIME-Version: 1.0
To: Grzegorz Junka <list1@gjunka.com>, freebsd-jail@freebsd.org
Subject: Re: Effective rule sets in a jail?
References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com>
 <CANJ8om5R-BT=heC+giMTXFH8YQXhuPQZjQ_S-P1bQ1XBGS16uQ@mail.gmail.com>
 <577E0A78.1040600@quip.cz> <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com>
 <577E1AFB.90100@quip.cz> <6ccead58-a38a-80a4-b5b8-a509c4271b8f@gjunka.com>
In-Reply-To: <6ccead58-a38a-80a4-b5b8-a509c4271b8f@gjunka.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 07 Jul 2016 10:06:43 -0000

Grzegorz Junka wrote on 07/07/2016 11:42:

> OK, I am just an user, not very familiar with the terminology. For me
> (as a programmer) inheriting means overriding, so merging the more
> specific to the less specific declarations.
>
> Does it mean that the "inheriting" works in nested declarations but
> doesn't take into account the default value? In other words, the default
> is just default unless it re-defined in a jail declaration. If that's
> the case then wouldn't be more clear to name the "outside" default
> declaration as default, e.g. "default_devfs_ruleset"? Then it would be
> more difficult to confuse the default with the one that can be inherited.

I think it is simple in current form. (And I am not sys developer, I was 
web application programmer before I became sysadmin)
I started with jails long time before jail2 with jail.conf. Current 
jail.conf is soooo simpler in comparision with rc.conf style variables.

Naming each default variable with different name will be harder to code, 
harder to write in jail.conf, harder to document in manpages.

Almost all programming languages works the same in this context - later 
variable definition wins.

So you can easily define all variables needed to run jails and then set 
just those specific to one jail - IPs and hostname:

## Typical static defaults:
## Use the rc scripts to start and stop jails.  Mount jail's /dev.
exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;
exec.system_user   = "root";
exec.jail_user     = "root";
mount.devfs;
devfs_ruleset      = 4;
enforce_statfs     = 1;
#allow.set_hostname = false;
#allow.mount;
allow.set_hostname = 0;
allow.sysvipc      = 0;
allow.raw_sockets  = 0;

## Dynamic wildcard parameter:
path            = "/vol1/jail/$name";
exec.consolelog = "/var/log/jail/$name.console";
mount.fstab     = "/etc/fstab.$name";

## Jail myjail0
myjail0 {
         host.hostname = "myjail0.example.conf";
         ip4.addr      = 10.20.30.40;
}

## Jail myjail1
myjail1 {
         host.hostname = "myjail1.example.conf";
         ip4.addr      = 10.20.30.41;
}


devfs_ruleset is the same as the other variables - you can't (and I hope 
nobody expect) to merge global default value of e.g. exec.system_user or 
allow.sysvipc with variables defined in specific jail context. Those 
variables can have only one value (bool, or string, or number; not an 
array). It is the same for devfs_rules. Can't have more than one numeric 
value, can't combine two together.

I think you will be familiar with this very soon.

Miroslav Lachman

From owner-freebsd-jail@freebsd.org  Thu Jul  7 11:17:45 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 950FEB76142
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Thu,  7 Jul 2016 11:17:45 +0000 (UTC)
 (envelope-from list1@gjunka.com)
Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client CN "msa1.earth.yoonka.com",
 Issuer "msa1.earth.yoonka.com" (not verified))
 by mx1.freebsd.org (Postfix) with ESMTPS id 4B24A17B0
 for <freebsd-jail@freebsd.org>; Thu,  7 Jul 2016 11:17:44 +0000 (UTC)
 (envelope-from list1@gjunka.com)
Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20])
 (authenticated bits=0)
 by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u67BHfSP056496
 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO)
 for <freebsd-jail@freebsd.org>; Thu, 7 Jul 2016 11:17:41 GMT
 (envelope-from list1@gjunka.com)
Subject: Re: Effective rule sets in a jail?
To: freebsd-jail@freebsd.org
References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com>
 <CANJ8om5R-BT=heC+giMTXFH8YQXhuPQZjQ_S-P1bQ1XBGS16uQ@mail.gmail.com>
 <577E0A78.1040600@quip.cz> <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com>
 <577E1AFB.90100@quip.cz> <6ccead58-a38a-80a4-b5b8-a509c4271b8f@gjunka.com>
 <577E29A8.5000504@quip.cz>
From: Grzegorz Junka <list1@gjunka.com>
Message-ID: <4d3f5584-7dd1-e6fc-540f-9ed3f1fb63f4@gjunka.com>
Date: Thu, 7 Jul 2016 11:17:41 +0000
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101
 Thunderbird/45.1.1
MIME-Version: 1.0
In-Reply-To: <577E29A8.5000504@quip.cz>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 07 Jul 2016 11:17:45 -0000


On 07/07/2016 10:06, Miroslav Lachman wrote:
> Grzegorz Junka wrote on 07/07/2016 11:42:
>
>> OK, I am just an user, not very familiar with the terminology. For me
>> (as a programmer) inheriting means overriding, so merging the more
>> specific to the less specific declarations.
>>
>> Does it mean that the "inheriting" works in nested declarations but
>> doesn't take into account the default value? In other words, the default
>> is just default unless it re-defined in a jail declaration. If that's
>> the case then wouldn't be more clear to name the "outside" default
>> declaration as default, e.g. "default_devfs_ruleset"? Then it would be
>> more difficult to confuse the default with the one that can be 
>> inherited.
>
> I think it is simple in current form. (And I am not sys developer, I 
> was web application programmer before I became sysadmin)
> I started with jails long time before jail2 with jail.conf. Current 
> jail.conf is soooo simpler in comparision with rc.conf style variables.
>
> Naming each default variable with different name will be harder to 
> code, harder to write in jail.conf, harder to document in manpages.
>
> Almost all programming languages works the same in this context - 
> later variable definition wins.
>
> So you can easily define all variables needed to run jails and then 
> set just those specific to one jail - IPs and hostname:
>
> ## Typical static defaults:
> ## Use the rc scripts to start and stop jails.  Mount jail's /dev.
> exec.start = "/bin/sh /etc/rc";
> exec.stop  = "/bin/sh /etc/rc.shutdown";
> exec.clean;
> exec.system_user   = "root";
> exec.jail_user     = "root";
> mount.devfs;
> devfs_ruleset      = 4;
> enforce_statfs     = 1;
> #allow.set_hostname = false;
> #allow.mount;
> allow.set_hostname = 0;
> allow.sysvipc      = 0;
> allow.raw_sockets  = 0;
>
> ## Dynamic wildcard parameter:
> path            = "/vol1/jail/$name";
> exec.consolelog = "/var/log/jail/$name.console";
> mount.fstab     = "/etc/fstab.$name";
>
> ## Jail myjail0
> myjail0 {
>         host.hostname = "myjail0.example.conf";
>         ip4.addr      = 10.20.30.40;
> }
>
> ## Jail myjail1
> myjail1 {
>         host.hostname = "myjail1.example.conf";
>         ip4.addr      = 10.20.30.41;
> }
>
>
> devfs_ruleset is the same as the other variables - you can't (and I 
> hope nobody expect) to merge global default value of e.g. 
> exec.system_user or allow.sysvipc with variables defined in specific 
> jail context. Those variables can have only one value (bool, or 
> string, or number; not an array). It is the same for devfs_rules. 
> Can't have more than one numeric value, can't combine two together.
>
> I think you will be familiar with this very soon.
>
> Miroslav Lachman

OK, so my confusion steams from the fact that the devfs rules are 
defined somewhere else and the jail.conf is simply taking into account 
the rule number, not its content. In that context it indeed makes sense.

It could be simply a matter of adding a clarification that each jail can 
have only one devfs ruleset assigned to it (which then would be 
calculated according to the standard rules defined in jail.conf), for 
example:

Descendant jails inherit the parent jail's devfs ruleset. Devfs rules 
enforced in the jail are defined by the single calculated ruleset.

What do you think?

Grzegorz

From owner-freebsd-jail@freebsd.org  Thu Jul  7 14:03:17 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 82A43B747EB
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Thu,  7 Jul 2016 14:03:17 +0000 (UTC)
 (envelope-from eto.freebsd@ethome.sk)
Received: from smtpout6.dnsserver.eu (smtpout6.dnsserver.eu [92.240.253.144])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256
 bits)) (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 438CF1A08
 for <freebsd-jail@freebsd.org>; Thu,  7 Jul 2016 14:03:16 +0000 (UTC)
 (envelope-from eto.freebsd@ethome.sk)
Received: from [92.240.253.67] (helo=smtp3s109.dnsserver.eu)
 by smtpout6.dnsserver.eu with esmtp (Exim 4.84 (FreeBSD))
 (envelope-from <eto.freebsd@ethome.sk>) id 1bL9WO-00070o-20
 for freebsd-jail@freebsd.org; Thu, 07 Jul 2016 15:39:16 +0200
Received: from [80.242.44.220] (helo=eto-mona.office.smartweb.sk)
 by smtp3s109.dnsserver.eu with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128)
 (Exim 4.83 (FreeBSD)) (envelope-from <eto.freebsd@ethome.sk>)
 id 1bL9WP-000Ak0-DQ
 for freebsd-jail@freebsd.org; Thu, 07 Jul 2016 15:39:17 +0200
Date: Thu, 7 Jul 2016 15:31:48 +0200
From: "Martin \"eto\" Misuth" <eto.freebsd@ethome.sk >
To: freebsd-jail@freebsd.org
Subject: Re: Effective rule sets in a jail?
Message-ID: <20160707153148.7da31609@eto-mona.office.smartweb.sk>
In-Reply-To: <4d3f5584-7dd1-e6fc-540f-9ed3f1fb63f4@gjunka.com>
References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com>
 <CANJ8om5R-BT=heC+giMTXFH8YQXhuPQZjQ_S-P1bQ1XBGS16uQ@mail.gmail.com>
 <577E0A78.1040600@quip.cz>
 <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com>
 <577E1AFB.90100@quip.cz>
 <6ccead58-a38a-80a4-b5b8-a509c4271b8f@gjunka.com>
 <577E29A8.5000504@quip.cz>
 <4d3f5584-7dd1-e6fc-540f-9ed3f1fb63f4@gjunka.com>
Organization: ethome.sk
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-SA-Exim-Connect-IP: 80.242.44.220
X-SA-Exim-Mail-From: eto.freebsd@ethome.sk
X-SA-Exim-Scanned: No (on smtp3s109.dnsserver.eu);
 SAEximRunCond expanded to false
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 07 Jul 2016 14:03:17 -0000

On Thu, 7 Jul 2016 11:17:41 +0000
Grzegorz Junka <list1@gjunka.com> wrote:
> 
> Descendant jails inherit the parent jail's devfs ruleset. Devfs rules 
> enforced in the jail are defined by the single calculated ruleset.
> 
> What do you think?
> 

IMHO, regarding jails, better mental model would be like this:
 - any single jail can have one and only one devfs ruleset number assigned
   - however, different standalone jails can have different devfs ruleset
     number assigned
 - nested jails inherit ruleset number from their parent jail

Regarding rulesets "inheritance"/"merging" you are probably looking into the
wrong place. devfs ruleset system is completely orthogonal to jails, as it
is used for other things as well.

You can "merge" devfs rulesets in devfs /etc/devfs.rules. 

Look into /etc/defaults/devfs.rules how initial rulesets are built.

First everything is hidden by ruleset 1 aka "devfsrules_hide_all". This is "by
default deny" policy, which should, according to me, used whenever one can.

Then, new rulesets are created by unhiding various groups of devices. 
Like for example you have minimal sub-ruleset 2 aka "devfsrules_unhide_basic".
That one is required to get minimal working /dev. Otherwise most programs break.

Finally ruleset 4 aka "devfsrules_jail" is built, which can be used by jails.

I personally "classify" jail types into groups. Let's call such group a jail
"class" (for the purpose of classification).

Thus to get what you want, you should create custom ruleset per jail "class" and
assign it to your jails based on their "class".

[devfsrules_jail_class_no_zfs=16]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login

Class might be not good word for this, as it is quite "loaded" by now, but I am
using it that way.

Some jails might end up so special, they require completely fine tuned ruleset. 
Those cannot be completely "classified" at all like this for example:

[devfsrules_jail_proxy=333]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_unhide_jail_proxy_tuns 

"devfsrules_unhide_jail_proxy_tuns" sub-rule in this case unhides
several tun interfaces used solely by this jail only.

devfs.conf files are "parsed" by /etc/rc.d/devfs rc script which is run quite
early after boot. If you look at it you will see it is using /etc/rc.subr 
devfs_* subroutines of rc.d framework which invoke /sbin/devfs helper program.

Theoretically if /etc/rc.d/devfs and /etc/rc.subr are not enough for
you, you could write helper script to invoke /sbin/devfs to setup most
convoluted rule ids directly by hand.

  eto

From owner-freebsd-jail@freebsd.org  Thu Jul  7 18:27:00 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5401BB75577
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Thu,  7 Jul 2016 18:27:00 +0000 (UTC)
 (envelope-from list1@gjunka.com)
Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client CN "msa1.earth.yoonka.com",
 Issuer "msa1.earth.yoonka.com" (not verified))
 by mx1.freebsd.org (Postfix) with ESMTPS id 0AC761F2B
 for <freebsd-jail@freebsd.org>; Thu,  7 Jul 2016 18:26:59 +0000 (UTC)
 (envelope-from list1@gjunka.com)
Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20])
 (authenticated bits=0)
 by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u67IQoDA064384
 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO)
 for <freebsd-jail@freebsd.org>; Thu, 7 Jul 2016 18:26:50 GMT
 (envelope-from list1@gjunka.com)
Subject: Re: Effective rule sets in a jail?
To: freebsd-jail@freebsd.org
References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com>
 <CANJ8om5R-BT=heC+giMTXFH8YQXhuPQZjQ_S-P1bQ1XBGS16uQ@mail.gmail.com>
 <577E0A78.1040600@quip.cz> <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com>
 <577E1AFB.90100@quip.cz> <6ccead58-a38a-80a4-b5b8-a509c4271b8f@gjunka.com>
 <577E29A8.5000504@quip.cz> <4d3f5584-7dd1-e6fc-540f-9ed3f1fb63f4@gjunka.com>
 <20160707153148.7da31609@eto-mona.office.smartweb.sk>
From: Grzegorz Junka <list1@gjunka.com>
Message-ID: <8d24b89f-3a83-ac6b-5051-c50accb24e9d@gjunka.com>
Date: Thu, 7 Jul 2016 18:26:50 +0000
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101
 Thunderbird/45.1.1
MIME-Version: 1.0
In-Reply-To: <20160707153148.7da31609@eto-mona.office.smartweb.sk>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 07 Jul 2016 18:27:00 -0000


On 07/07/2016 13:31, Martin "eto" Misuth wrote:
> IMHO, regarding jails, better mental model would be like this:
>   - any single jail can have one and only one devfs ruleset number assigned
>     - however, different standalone jails can have different devfs ruleset
>       number assigned
>   - nested jails inherit ruleset number from their parent jail
>
> Regarding rulesets "inheritance"/"merging" you are probably looking into the
> wrong place. devfs ruleset system is completely orthogonal to jails, as it
> is used for other things as well.
>
> You can "merge" devfs rulesets in devfs /etc/devfs.rules.
>
> Look into /etc/defaults/devfs.rules how initial rulesets are built.
>
> First everything is hidden by ruleset 1 aka "devfsrules_hide_all". This is "by
> default deny" policy, which should, according to me, used whenever one can.
>
> Then, new rulesets are created by unhiding various groups of devices.
> Like for example you have minimal sub-ruleset 2 aka "devfsrules_unhide_basic".
> That one is required to get minimal working /dev. Otherwise most programs break.
>
> Finally ruleset 4 aka "devfsrules_jail" is built, which can be used by jails.
>
> I personally "classify" jail types into groups. Let's call such group a jail
> "class" (for the purpose of classification).
>
> Thus to get what you want, you should create custom ruleset per jail "class" and
> assign it to your jails based on their "class".
>
> [devfsrules_jail_class_no_zfs=16]
> add include $devfsrules_hide_all
> add include $devfsrules_unhide_basic
> add include $devfsrules_unhide_login
>
> Class might be not good word for this, as it is quite "loaded" by now, but I am
> using it that way.
>
> Some jails might end up so special, they require completely fine tuned ruleset.
> Those cannot be completely "classified" at all like this for example:
>
> [devfsrules_jail_proxy=333]
> add include $devfsrules_hide_all
> add include $devfsrules_unhide_basic
> add include $devfsrules_unhide_login
> add include $devfsrules_unhide_jail_proxy_tuns
>
> "devfsrules_unhide_jail_proxy_tuns" sub-rule in this case unhides
> several tun interfaces used solely by this jail only.
>
> devfs.conf files are "parsed" by /etc/rc.d/devfs rc script which is run quite
> early after boot. If you look at it you will see it is using /etc/rc.subr
> devfs_* subroutines of rc.d framework which invoke /sbin/devfs helper program.
>
> Theoretically if /etc/rc.d/devfs and /etc/rc.subr are not enough for
> you, you could write helper script to invoke /sbin/devfs to setup most
> convoluted rule ids directly by hand.
>
>    eto
> _______________________________________________
> freebsd-jail@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-jail
> To unsubscribe, send any mail to "freebsd-jail-unsubscribe@freebsd.org"

That's great! Thanks for the comprehensive explanation, I wish it was in 
the man already so I wouldn't need to enquiry additionally here. It 
makes sense, as I mentioned in my previous email, I got confused and 
messed jail inheritance with the inheritance of devfs rule sets, they 
are orthogonal as you stated. I amended my rules to include the basic 
ones from rule 4 to the more specific one for one particular jail and it 
works. Thanks again!
Grzegorz


From owner-freebsd-jail@freebsd.org  Fri Jul  8 18:28:55 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 586FFB85B9D
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Fri,  8 Jul 2016 18:28:55 +0000 (UTC)
 (envelope-from tommyj27@gmail.com)
Received: from mail-vk0-x22a.google.com (mail-vk0-x22a.google.com
 [IPv6:2607:f8b0:400c:c05::22a])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (Client CN "smtp.gmail.com",
 Issuer "Google Internet Authority G2" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 1508E1744
 for <freebsd-jail@freebsd.org>; Fri,  8 Jul 2016 18:28:55 +0000 (UTC)
 (envelope-from tommyj27@gmail.com)
Received: by mail-vk0-x22a.google.com with SMTP id v6so65154868vkb.2
 for <freebsd-jail@freebsd.org>; Fri, 08 Jul 2016 11:28:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=mime-version:from:date:message-id:subject:to;
 bh=RUaLTzswdADaJlJvtoai3ksxkQWFKe1tXTg1tLJ36a8=;
 b=shgWxQWCSFP38Djlp11KReuVANggaa9rRT+qro7f8A5A6HbicPpuhyncN8gRUbyjfU
 5sGOUwYOqwH/bvRrc/mLgos2IqevN8nueZa9JW6h3Og5rDhhyqurZ+4o5TqYk9fQeoEH
 onO/5hP7Ynozn2GOjoDFcZMRcjXY6iFo/gsHXKMcUuaBClWXnVLugkyahRwrawJ28TY4
 T6eicFFmJg08429vmfUfAC4HaDUp2F61+1bM8fnA3Q32tXv5l3sTU3OGEkjXl3cewo2d
 u8AKcpSVA1kQv9tnVBAz4KSgN0v5y9cqwXUoSx2RGDv4vUpRSVTbzidaudahtHuFnzoV
 eNyA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
 bh=RUaLTzswdADaJlJvtoai3ksxkQWFKe1tXTg1tLJ36a8=;
 b=jMyNxE78EjScGMSAELfMtEB1yOpoRh+Vsoi5hOfIpLNg/ADLzlOByXFgLY5vqgY0QS
 9x9WsDjrnaNuhdm/ve2A6FH73tQWqI2O1wxVqLzm1kCmoR7ic09wo49i2wgkM0QbStCK
 7GT0cdnbvHn4cCgaVtCVo95Ecg0W5yzb7jCB35SsyFnbjra3vzuYWLOoqh6e2lUgug3X
 Y1au1CLferi/kDnBVl6hVEjsWwMetqnGKpLSYkLbDI3m7zNg1RIFutJ7dFkVB++03PXf
 IFS7H4F/j54of5VzYCG2nD9zOR8Shzd88BVD4a3t02lOy+6YbuutIwrQz8lHM9kshlyk
 9dfw==
X-Gm-Message-State: ALyK8tLx2iDCWp1L4H3U4mCvfDyQxUvk21R2z9EWlI0pZs+rZu6Ia0q4SRs7b4wU40CfIoL8QOOZ72wALlgyRA==
X-Received: by 10.159.41.164 with SMTP id s33mr3324906uas.146.1468002533977;
 Fri, 08 Jul 2016 11:28:53 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.159.33.136 with HTTP; Fri, 8 Jul 2016 11:28:53 -0700 (PDT)
From: Thomas Johnson <tommyj27@gmail.com>
Date: Fri, 8 Jul 2016 13:28:53 -0500
Message-ID: <CAMwYC7ZWH0s3h0AV8KxSLQk_NipvOkaMzkf2jF2Ty9Rx5x1H3w@mail.gmail.com>
Subject: NFS + nullfs + jail = zombies?
To: freebsd-jail@freebsd.org
Content-Type: text/plain; charset=UTF-8
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 08 Jul 2016 18:28:55 -0000

I am working on developing a clustered application utilizing jails and
running into problems that seem to be NFS-related. I'm hoping that
someone can point out my error.

The jail images and my application data are served via NFS. The host
mounts NFS at boot, and then uses nullfs mounts to assemble the jail
tree when the jail is created (fstab files and jail.conf are below).
This seems to work fine, the jail starts and is usable. The problem
comes when I remove/restart the jail. Frequently (but not
consistently), the jail gets stuck in a dying state, causing the
unmount of the jail root (nullfs) to fail with a "device busy" error.

# jail -f /var/local/jail.conf -r wds1-1a
Stopping cron.
Waiting for PIDS: 1361.
.
Terminated
wds1-1a: removed
umount: unmount of /var/jail/wds1-1a failed: Device busy
# jls -av
   JID  Hostname                      Path
        Name                          State
        CPUSetID
        IP Address(es)
     1  wds1-1a                       /var/jail/wds1-1a
        wds1-1a                       DYING
        2
        2620:1:1:1:1a::1

Through trial-and-error I have determined that forcing an unmount of
the root works, but subsequent mounts to that mount point will fail to
unmount with the same error. Deleting and recreating the mountpoint
fixes the mounting issue, but the dying jail remains permanently.

I have also found that if I copy the jail root to local storage and
update the jail's fstab to nullfs mount this, the problem seems to go
away. This leads me to believe that the issue is related to the NFS
source for the nullfs mount. statd and lockd are both running on the
host.

My relevant configurations are below. I can provide any other
information desired.

# Host fstab line for jail root.
#
10.219.212.1:/vol/dev/wds/jail_base  /jail/base nfs ro    0    0


# Jail fstab file (mount.fstab)
#
/jail/base /var/jail/wds1-1a nullfs ro 0 0
# writable (UFS-backed) /var
/var/jail-vars/wds1-1a /var/jail/wds1-1a/var nullfs rw 0 0


# jail.conf file
#
* {
    devfs_ruleset = "4";
    mount.devfs;
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    interface = "vmx1";
    allow.dying = 1;
    exec.prestart = "/usr/local/bin/rsync -avC --delete
/jail/${image}/var/ /var/jail-vars/${host.hostname}/";
    }

# JMANAGE wds1-1a
wds1-1a {
    path = "/var/jail/wds1-1a";
    ip6.addr = "2620:1:1:1:1a::1";
    host.hostname = "wds1-1a";
    host.domainname = "dev";
    mount.fstab = "/var/local/fstab.wds1-1a";
    $image = "base";
}

From owner-freebsd-jail@freebsd.org  Fri Jul  8 21:07:30 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id E44D5B858AE
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Fri,  8 Jul 2016 21:07:30 +0000 (UTC)
 (envelope-from luzar722@gmail.com)
Received: from mail-it0-x229.google.com (mail-it0-x229.google.com
 [IPv6:2607:f8b0:4001:c0b::229])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (Client CN "smtp.gmail.com",
 Issuer "Google Internet Authority G2" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id AC58719C3
 for <freebsd-jail@freebsd.org>; Fri,  8 Jul 2016 21:07:30 +0000 (UTC)
 (envelope-from luzar722@gmail.com)
Received: by mail-it0-x229.google.com with SMTP id h190so19286989ith.1
 for <freebsd-jail@freebsd.org>; Fri, 08 Jul 2016 14:07:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=message-id:date:from:user-agent:mime-version:to:cc:subject
 :references:in-reply-to:content-transfer-encoding;
 bh=PNA7aC7MBpWej+rTAtpn9mPYY+hWCrZS2ZXk4Oj2gqA=;
 b=Ycd0hldEqvDt9bOzq9nnrHxZ+EEoI9Y1jO6vPlBxumTZbPJ7BUVKhb2t66ADuC8/Ro
 mKDD/vqktCtOxDnpGww5je4zJxoxCPEqv9dNnwvXBGlFENdeR/Ej77emfq3TI2PCTyTL
 R1U6IpzieBBQ983AIzcFM+hAovq4wp+ESk6YqUT9Bi6r98HGNCavHd+6rZAJNPSAk/T2
 csTa90sohbODx2a16sZpPND38/X7+uzyy+J3msX5UjkHPJ3KH3BLpgEMN5EXxpCgecBY
 w2Yck0Ocd3cjNiGH4uvodtalJNNWxiz5/BlVAMrEWuhfIi6G/Hd5rfJ2RNG/NcgpRbue
 xvPw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to
 :cc:subject:references:in-reply-to:content-transfer-encoding;
 bh=PNA7aC7MBpWej+rTAtpn9mPYY+hWCrZS2ZXk4Oj2gqA=;
 b=jr2bk9+gkXDdEpLU3rO/tT+KcaR3Xe8ovX0M7YY5WMUfnnUDII0zDZ+WuvMFcn5v+j
 cw8szxhL0oHbatcQ/af18N4zi8xXP78CAHrzfe6rfsBYd0BLIoz3ukFruOtBUxcDI63D
 wI+Al44l9GLRDUvapm5Bw6+Hua3AQvXBg3/KsN1LLesVcnDlA/IG+p0WOzcS3rER/WEx
 DaBDijVY330Llo+mEuo6jhIxnVEbvNuCJYtx3qeDbtqyF0xXdWymaejCqWr/MkPmRxdi
 /o5lrCQ+rPEdbX7Adyv3pnbjiqTJRLh4iVth/OBnQlFKVvcEIgojZf0hK8fgSU488rVV
 nLzg==
X-Gm-Message-State: ALyK8tLBxDwahnbGIv/RjjOu6huDZxCh4MP+pYbhnUv4JfLDtazzUT7J+avmCu+0t/Mkzw==
X-Received: by 10.36.117.200 with SMTP id y191mr526187itc.35.1468012050040;
 Fri, 08 Jul 2016 14:07:30 -0700 (PDT)
Received: from [10.0.10.3] (cpe-184-56-210-236.neo.res.rr.com.
 [184.56.210.236])
 by smtp.googlemail.com with ESMTPSA id v129sm1998920itg.10.2016.07.08.14.07.29
 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128);
 Fri, 08 Jul 2016 14:07:29 -0700 (PDT)
Message-ID: <57801614.3000205@gmail.com>
Date: Fri, 08 Jul 2016 17:07:32 -0400
From: Ernie Luzar <luzar722@gmail.com>
User-Agent: Thunderbird 2.0.0.24 (Windows/20100228)
MIME-Version: 1.0
To: Thomas Johnson <tommyj27@gmail.com>
CC: freebsd-jail@freebsd.org
Subject: Re: NFS + nullfs + jail = zombies?
References: <CAMwYC7ZWH0s3h0AV8KxSLQk_NipvOkaMzkf2jF2Ty9Rx5x1H3w@mail.gmail.com>
In-Reply-To: <CAMwYC7ZWH0s3h0AV8KxSLQk_NipvOkaMzkf2jF2Ty9Rx5x1H3w@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 08 Jul 2016 21:07:31 -0000

Thomas Johnson wrote:
> I am working on developing a clustered application utilizing jails and
> running into problems that seem to be NFS-related. I'm hoping that
> someone can point out my error.
> 
> The jail images and my application data are served via NFS. The host
> mounts NFS at boot, and then uses nullfs mounts to assemble the jail
> tree when the jail is created (fstab files and jail.conf are below).
> This seems to work fine, the jail starts and is usable. The problem
> comes when I remove/restart the jail. Frequently (but not
> consistently), the jail gets stuck in a dying state, causing the
> unmount of the jail root (nullfs) to fail with a "device busy" error.
> 
> # jail -f /var/local/jail.conf -r wds1-1a
> Stopping cron.
> Waiting for PIDS: 1361.
> .
> Terminated
> wds1-1a: removed
> umount: unmount of /var/jail/wds1-1a failed: Device busy
> # jls -av
>    JID  Hostname                      Path
>         Name                          State
>         CPUSetID
>         IP Address(es)
>      1  wds1-1a                       /var/jail/wds1-1a
>         wds1-1a                       DYING
>         2
>         2620:1:1:1:1a::1
> 
> Through trial-and-error I have determined that forcing an unmount of
> the root works, but subsequent mounts to that mount point will fail to
> unmount with the same error. Deleting and recreating the mountpoint
> fixes the mounting issue, but the dying jail remains permanently.
> 
> I have also found that if I copy the jail root to local storage and
> update the jail's fstab to nullfs mount this, the problem seems to go
> away. This leads me to believe that the issue is related to the NFS
> source for the nullfs mount. statd and lockd are both running on the
> host.
> 
> My relevant configurations are below. I can provide any other
> information desired.
> 
> # Host fstab line for jail root.
> #
> 10.219.212.1:/vol/dev/wds/jail_base  /jail/base nfs ro    0    0
> 
> 
> # Jail fstab file (mount.fstab)
> #
> /jail/base /var/jail/wds1-1a nullfs ro 0 0
> # writable (UFS-backed) /var
> /var/jail-vars/wds1-1a /var/jail/wds1-1a/var nullfs rw 0 0
> 
> 
> # jail.conf file
> #
> * {
>     devfs_ruleset = "4";
>     mount.devfs;
>     exec.start = "/bin/sh /etc/rc";
>     exec.stop = "/bin/sh /etc/rc.shutdown";
>     interface = "vmx1";
>     allow.dying = 1;
>     exec.prestart = "/usr/local/bin/rsync -avC --delete
> /jail/${image}/var/ /var/jail-vars/${host.hostname}/";
>     }
> 
> # JMANAGE wds1-1a
> wds1-1a {
>     path = "/var/jail/wds1-1a";
>     ip6.addr = "2620:1:1:1:1a::1";
>     host.hostname = "wds1-1a";
>     host.domainname = "dev";
>     mount.fstab = "/var/local/fstab.wds1-1a";
>     $image = "base";
> }


If I remember correctly from past posts about NFS and jails, NFS does 
not play well with jails. The jails directory tree has to be on the host 
running the jail. Remove NFS from the picture and it should just work. 
Your beating a dead horse.


From owner-freebsd-jail@freebsd.org  Sat Jul  9 14:23:08 2016
Return-Path: <owner-freebsd-jail@freebsd.org>
Delivered-To: freebsd-jail@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8EEEBB8367B
 for <freebsd-jail@mailman.ysv.freebsd.org>;
 Sat,  9 Jul 2016 14:23:08 +0000 (UTC)
 (envelope-from jamie@gritton.org)
Received: from gritton.org (gritton.org [162.220.209.3])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client CN "www.gritton.org",
 Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified))
 by mx1.freebsd.org (Postfix) with ESMTPS id 778AE18A5
 for <freebsd-jail@freebsd.org>; Sat,  9 Jul 2016 14:23:08 +0000 (UTC)
 (envelope-from jamie@gritton.org)
Received: from gritton.org (gritton.org [162.220.209.3])
 by gritton.org (8.15.2/8.15.2) with ESMTPS id u69EMPDx005353
 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
 Sat, 9 Jul 2016 08:22:25 -0600 (MDT)
 (envelope-from jamie@gritton.org)
Received: (from www@localhost)
 by gritton.org (8.15.2/8.15.2/Submit) id u69EMPPF005352;
 Sat, 9 Jul 2016 08:22:25 -0600 (MDT)
 (envelope-from jamie@gritton.org)
X-Authentication-Warning: gritton.org: www set sender to jamie@gritton.org
 using -f
To: freebsd-jail@freebsd.org
Subject: Re: NFS + nullfs + jail = zombies?
X-PHP-Originating-Script: 0:rcube.php
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Sat, 09 Jul 2016 08:22:25 -0600
From: James Gritton <jamie@gritton.org>
In-Reply-To: <CAMwYC7ZWH0s3h0AV8KxSLQk_NipvOkaMzkf2jF2Ty9Rx5x1H3w@mail.gmail.com>
References: <CAMwYC7ZWH0s3h0AV8KxSLQk_NipvOkaMzkf2jF2Ty9Rx5x1H3w@mail.gmail.com>
Message-ID: <596319b8dce811ea6e332c48d3542451@gritton.org>
X-Sender: jamie@gritton.org
User-Agent: Roundcube Webmail/1.2.0
X-BeenThere: freebsd-jail@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "Discussion about FreeBSD jail\(8\)" <freebsd-jail.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-jail/>
List-Post: <mailto:freebsd-jail@freebsd.org>
List-Help: <mailto:freebsd-jail-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-jail>,
 <mailto:freebsd-jail-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 09 Jul 2016 14:23:08 -0000

On 2016-07-08 12:28, Thomas Johnson wrote:
> I am working on developing a clustered application utilizing jails and
> running into problems that seem to be NFS-related. I'm hoping that
> someone can point out my error.
> 
> The jail images and my application data are served via NFS. The host
> mounts NFS at boot, and then uses nullfs mounts to assemble the jail
> tree when the jail is created (fstab files and jail.conf are below).
> This seems to work fine, the jail starts and is usable. The problem
> comes when I remove/restart the jail. Frequently (but not
> consistently), the jail gets stuck in a dying state, causing the
> unmount of the jail root (nullfs) to fail with a "device busy" error.
> 
> # jail -f /var/local/jail.conf -r wds1-1a
> Stopping cron.
> Waiting for PIDS: 1361.
> .
> Terminated
> wds1-1a: removed
> umount: unmount of /var/jail/wds1-1a failed: Device busy
> # jls -av
>    JID  Hostname                      Path
>         Name                          State
>         CPUSetID
>         IP Address(es)
>      1  wds1-1a                       /var/jail/wds1-1a
>         wds1-1a                       DYING
>         2
>         2620:1:1:1:1a::1
> 
> Through trial-and-error I have determined that forcing an unmount of
> the root works, but subsequent mounts to that mount point will fail to
> unmount with the same error. Deleting and recreating the mountpoint
> fixes the mounting issue, but the dying jail remains permanently.
> 
> I have also found that if I copy the jail root to local storage and
> update the jail's fstab to nullfs mount this, the problem seems to go
> away. This leads me to believe that the issue is related to the NFS
> source for the nullfs mount. statd and lockd are both running on the
> host.
> 
> My relevant configurations are below. I can provide any other
> information desired.
> 
> # Host fstab line for jail root.
> #
> 10.219.212.1:/vol/dev/wds/jail_base  /jail/base nfs ro    0    0
> 
> 
> # Jail fstab file (mount.fstab)
> #
> /jail/base /var/jail/wds1-1a nullfs ro 0 0
> # writable (UFS-backed) /var
> /var/jail-vars/wds1-1a /var/jail/wds1-1a/var nullfs rw 0 0
> 
> 
> # jail.conf file
> #
> * {
>     devfs_ruleset = "4";
>     mount.devfs;
>     exec.start = "/bin/sh /etc/rc";
>     exec.stop = "/bin/sh /etc/rc.shutdown";
>     interface = "vmx1";
>     allow.dying = 1;
>     exec.prestart = "/usr/local/bin/rsync -avC --delete
> /jail/${image}/var/ /var/jail-vars/${host.hostname}/";
>     }
> 
> # JMANAGE wds1-1a
> wds1-1a {
>     path = "/var/jail/wds1-1a";
>     ip6.addr = "2620:1:1:1:1a::1";
>     host.hostname = "wds1-1a";
>     host.domainname = "dev";
>     mount.fstab = "/var/local/fstab.wds1-1a";
>     $image = "base";
> }

What happens if you take jails out of the equation?  I know this isn't 
entirely a non-jail issue, but I wonder if a jail is required for the 
mount point to be un-re-mountable.  I've heard before of NFS-related 
problems where a jail remains dying forever, but this has been more of 
an annoyance than a real problem.

It's not so much that I want to absolve jails, as I want to see where 
the main fight exists.  It's tricky enough fixing an interface between 
two systems, but we've got three here.

- Jamie