From nobody Wed Nov 30 00:21:28 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NMKdL0DzLz4hyh7 for ; Wed, 30 Nov 2022 00:21:42 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pf1-x430.google.com (mail-pf1-x430.google.com [IPv6:2607:f8b0:4864:20::430]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NMKdK07tsz44Lf; Wed, 30 Nov 2022 00:21:41 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=a94iGDIV; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::430 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-x430.google.com with SMTP id 21so2173876pfw.4; Tue, 29 Nov 2022 16:21:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=uWP0laZ14Jnym5vhvrGwrmkRsZWD7JneHuGU4tC7Beo=; b=a94iGDIVGByJSjsBOvaZ1yZ2tt4o68DnD31s+hvDbQZH/8oiac8JHVoAOwt5qp9umy VbxFs8XEudYvbdlAmAw38gXJ8JHg4P+btTO4KuhRIU7cbiYlCpsu1xmSYaE9kJe9AgaN c8Wrssd7nOOTkMH6UIoNpuvpuXLe1VGmAFx/TL9jrXrQgsyVFFrNB8wxfDZ166ZLMW+A lthpO2/2U63Us0Zz/R9Cq0Kpt0IsLuInsg4XOoX1fSIwLnTt1FpzHqPAkgtgUjnJ6vo2 JjRBNMUGCjtcrhn4GOGbFVibgk/iUERbn6TW9aiUmqHIC/uDtLbaGd1ezSSzhQw5xt/7 /dxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=uWP0laZ14Jnym5vhvrGwrmkRsZWD7JneHuGU4tC7Beo=; b=ZSDu251bkw8lYaFwPphsD1171YlnaJexG4N2OEYBBKQmbkbDn4xVqj9J1pEo82Zw/D /QSK2v1LpusPi4jPWeRurOr91qUnMWm38FrvpVF9oWWJOf9PrbIwbYkepFo11+QMHzL4 rM8RKs6LU+KS9lBFQ4zwFAjko/I0H8B7J9XuXHrtlLs4vInZxnMdj0ZuOtOD5P7IAkdJ geNx+PewvGcq6bde62cKLUsW0voF0JuqMbHMS7r5vChjhWMqV6Zp+2ykotZySgh6W3AL bcGm18/pYoKDCpFf0+FbXx5JWRGjLqArMkqPKcNgPZut2EVxYK4dIlNk203VIH0Ho1JD n/qw== X-Gm-Message-State: ANoB5pnn5RTYrJyG8LJGJDBgDbORNa/D6/ZXZhQyZWf6XWS3oDQ0qZAW fFt9oTyz012aKT++P/aJymjgr4HfZ5qvMD/Zy1LK+2w= X-Google-Smtp-Source: AA0mqf6/r9OAuXMJRzHb2lAo6ssFWi1jihxgupmRfewotv1od/2oc6Mbk2yoJTMarZbO3GIIDSiMioIh8xx1VKi2r1w= X-Received: by 2002:a05:6a00:21c8:b0:562:e0fb:3c79 with SMTP id t8-20020a056a0021c800b00562e0fb3c79mr39583679pfj.39.1669767699552; Tue, 29 Nov 2022 16:21:39 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <82103A1E-9D39-47B0-9520-205583C8B680@lysator.liu.se> In-Reply-To: <82103A1E-9D39-47B0-9520-205583C8B680@lysator.liu.se> From: Rick Macklem Date: Tue, 29 Nov 2022 16:21:28 -0800 Message-ID: Subject: Re: RFC: nfsd in a vnet jail To: Peter Eriksson Cc: FreeBSD CURRENT , "Bjoern A. Zeeb" , Alan Somers Content-Type: multipart/alternative; boundary="00000000000096ddc205eea51614" X-Spamd-Result: default: False [-3.92 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.92)[-0.922]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::430:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_ALL(0.00)[]; TAGGED_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_THREE(0.00)[4]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4NMKdK07tsz44Lf X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --00000000000096ddc205eea51614 Content-Type: text/plain; charset="UTF-8" On Sun, Nov 27, 2022 at 10:04 AM Peter Eriksson wrote: > Keep the global variables as defaults that apply to all nfsds and allow > (at least some subset) to be overridden inside the net jails if some things > need to be changed from the defaults? > > This is pretty much a reply to one of the posts selected at random, but I thought that better than starting a new email thread. bz@ and asomers@ have both asked about running mountd within a vnet prison (one via offlist email and the other on phabricator). I think it is worth discussing here... mountd (rightly or wrongly) does two distinctly different things: 1 - It pushes the exports into the kernel via nmount() so they can be hung off of the "struct mount" for a file system's mount point. --> This can only work for file system mount points and can only be done once for any given file system mount point. At this time, I have it done once globally outside of the prisons. The alternative I can see is doing it within each prison, but I think that would require that each prison have its own file system(s). (ie. The prison's root would always be a file system mount point.) 2 - It handles RPC Mount protocol requests from NFSv3 clients. This one is NFSv3 specific, which is why I have done this NFSv4 only at this time. To do this, it must be able to register with rpcbind, and I have no idea if running rpcbind in a vnet jail is practical. Enforcing the use for separate file systems for each jail also makes things safer, since the exports are enforced by the kernel. Without this, a malicious NFSv4 client could "guess" a file handle for a file outside the jail and gain access to that file. Put another way, without a separate file system, there is no way to stop a malicious client from finding files above the Root file handle. (Normal clients will use PutRootFH and LookupParent and these won't be able to go above the top of the jail.) So, what do others think of enforcing the requirement that each jail have its own file systems for this? rick > - Peter > > > On Fri, Nov 25, 2022, 4:24 PM Rick Macklem wrote: > >> Hi, >> >> bz@ has encouraged me to fiddle with the nfsd >> so that it works in a vnet jail. >> I have now basically done so, specifically for >> NFSv4, since NFSv3 presents various issues. >> >> What I have not yet done is put global variables >> in the vnet. This needs to be done so that the nfsd >> can be run in multiple jail instances and/or in and >> outside of a jail. >> The problem is that there are 100s of global variables. >> >> I can see two approaches: >> 1 - Move them all into the vnet jail. This would imply >> that all the sysctls need to somehow be changed, >> which would seem to be a POLA violation. >> It also implies a lot of stuff in the vnet. >> 2 - Just move the global variables that will always >> differ from one nfsd to another (this would make >> the sysctls global and apply to all nfsds). >> This will keep the number of globals in the vnet >> smaller. >> >> I am currently leaning towards #2, put what do others >> think? >> >> rick >> ps: Personally, I don't know what use there is of >> running the nfsd inside a vnet jail, but bz@ has >> some use case. >> > > --00000000000096ddc205eea51614 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Sun, Nov 27, 2022 at 10:04 AM Peter Erikss= on <pen@lysator.liu.se> wro= te:
Keep th= e global variables as defaults that apply to all nfsds and allow (at least = some subset) to be overridden inside the net jails if some things need to b= e changed from the defaults?

This is pretty much a= reply to one of the posts selected at random,
but I thought that bette= r than starting a new email thread.

bz@ and asomers@ have bo= th asked about running mountd within a vnet prison
(one via offlist ema= il and the other on phabricator).

I think it is worth discussi= ng here...
mountd (rightly or wrongly) does two distinctly different th= ings:
1 - It pushes the exports into the kernel via nmount() so they
=C2=A0 =C2=A0 can be hung off of the "struct mount" for a file = system's
=C2=A0 =C2=A0 mount point.
=C2=A0 =C2=A0 --> This c= an only work for file system mount points and can
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 only be done once for any given file system mount point.

=C2=A0 =C2=A0 At this time, I have it done once globally outside of the p= risons.
=C2=A0 =C2=A0 The alternative I can see is doing it within each= prison, but I
=C2=A0 =C2=A0 think that would require that each prison = have its own file system(s).
=C2=A0 =C2=A0 (ie. The prison's root w= ould always be a file system mount point.)

2 - It handles RPC = Mount protocol requests from NFSv3 clients.=C2=A0 This one
=C2=A0 =C2= =A0 is NFSv3 specific, which is why I have done this NFSv4 only at
=C2= =A0 =C2=A0 this time.=C2=A0 To do this, it must be able to register with rp= cbind,
=C2=A0 =C2=A0 and I have no idea if running rpcbind in a vnet ja= il is practical.

Enforcing the use for separate file systems f= or each jail also makes
things safer, since the exports are enforced by= the kernel. Without
this, a malicious NFSv4 client could "guess= " a file handle for a file
outside the jail and gain access to tha= t file. Put another way, without
a separate file system, there is no wa= y to stop a malicious client from
finding files above the Root file han= dle. (Normal clients will use
PutRootFH and LookupParent and these won&= #39;t be able to go above the top
of the jail.)

So, what d= o others think of enforcing the requirement that each jail
have its own= file systems for this?

rick
=C2=A0
- Peter
<= div>

On Fri, Nov 25, 2022, 4:24 PM= Rick Macklem <rick.macklem@gmail.com> wrote:
Hi,

bz@ has encouraged me to fiddle with the nfsd
so that it works in a vnet jail.
I have now basically done so, specifical= ly for
NFSv4, since NFSv3 present= s various issues.

What I have not yet done is put global vari= ables
in the vnet. This needs to = be done so that the nfsd
can be r= un in multiple jail instances and/or in and
outside of a jail.
The= problem is that there are 100s of global variables.

I can se= e two approaches:
1 - Move them a= ll into the vnet jail. This would imply
=C2=A0 =C2=A0 that all the sysctls need to somehow be changed,
=
=C2=A0 =C2=A0 which would seem to be a= POLA violation.
=C2=A0 =C2=A0 It= also implies a lot of stuff in the vnet.
2 - Just move the global variables that will always
=C2=A0 =C2=A0 differ from one nfsd to another (= this would make
=C2=A0 =C2=A0 the= sysctls global and apply to all nfsds).
=C2=A0 =C2=A0 This will keep the number of globals in the vnet
=C2=A0 =C2=A0 smaller.

= I am currently leaning towards #2, put what do others
think?

<= /div>
rick
ps: Personally, I don't know what use there is of
=C2=A0 =C2=A0 running the nfsd inside a = vnet jail, but bz@ has
=C2=A0 =C2= =A0 some use case.

--00000000000096ddc205eea51614--