Date: Thu, 11 Oct 2018 13:49:59 +0200 From: Karli =?ISO-8859-1?Q?Sj=F6berg?= <karli@inparadise.se> To: Peter Eriksson <peter@ifm.liu.se>, Rick Macklem <rmacklem@uoguelph.ca>, Felix Winterhalter <felix@audiofair.de> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: NFSv4 Kerberos mount from Linux Message-ID: <054a82958e9c1bed12c3a159d69db36b4cfcb5ec.camel@inparadise.se> In-Reply-To: <33A0F0BC-4AD8-4DE3-B484-42B7FB208B6A@ifm.liu.se> References: <30f6446c-6fed-4b1e-9cae-9c417974ec46@audiofair.de> <YTOPR0101MB1820A5756D172342AF441C25DDEA0@YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM> <c1ffda48-3809-bb4c-6d97-451765b0e25e@audiofair.de> <YTOPR0101MB18207F35A3973F26C6A58F6ADDE00@YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM> <33A0F0BC-4AD8-4DE3-B484-42B7FB208B6A@ifm.liu.se>
next in thread | previous in thread | raw e-mail | index | archive | help
--=-6HWtWOXsHgjO/mRWq1SE Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, 2018-10-11 at 10:36 +0200, Peter Eriksson wrote: > Just a few comments (random brain-dump) in case someone else is > having problems with NFS & Kerberos.=20 "Just a few comments", hot dog, that was a great write-up! It would be great, I think, to have this on the FreeBSD wiki or in the Handbook even, because I agree not many people use Kerberized NFS since it=C2=B4s such a hassle and not that much info about it. /K >=20 >=20 > We=E2=80=99ve been using NFSv4 with Kerberos from Linux clients here for = many > years (with Solaris-based NFS servers and MIT Kerberos) and lately > using FreeBSD as the NFS server OS (in an Microsoft AD Kerberos > environment).=20 >=20 > There are a few differences in behaviour (from a Solaris > perspective), mainly regarding the =E2=80=9Cpseudo=E2=80=9D NFSv4 filesys= tem but not > something that can=E2=80=99t be handled. In the process of moving to Free= BSD > based servers I=E2=80=99ve also been testing lots of different clients in > order to make sure everything works as expected. Here are some notes: >=20 > General stuff: >=20 > 1. Have a _kerberos.my.zone.com DNS TXT record containing the > Kerberos realm (nice to have) > 2. Have a _nfsv4domain.my.zone.com <http://nfsv4domain.my.zone.com/>; > DNS TXT record containing the NFSv4 =E2=80=9Cdomain=E2=80=9D (not all cli= ents use it, > but it=E2=80=99s nice to have) >=20 >=20 > * FreeBSD server (with ZFS filesystems), things below /export is NFS- > shared as (for example) server:/staff/user1 >=20 > 1. /etc/exports (we _only_ allow sec=3Dkrb<various flavours> for home > directories): > V4: /export -sec=3Dkrb5:krb5i:krb5p >=20 > Or (on a server also serving some (read-only) sec=3Dsys filesystems > below /exports) > V4: /export -sec=3Dkrb5:krb5i:krb5p:sys >=20 >=20 > 2. /etc/zfs/exports (generated from =E2=80=9Cset sharenfs=3Dxxx=E2=80=9D = on the ZFS > filesystems) > Home-server: > /export/staff -sec=3Dkrb5:krb5i:krb5p=20 > /export/staff/user1 -sec=3Dkrb5:krb5i:krb5p=20 > /export/staff/user2 -sec=3Dkrb5:krb5i:krb5p=20 > /export/students -sec=3Dkrb5:krb5i:krb5p=20 > /export/students/user3 -sec=3Dkrb5:krb5i:krb5p=20 >=20 > Software-server: > /export/db/icons -sec=3Dsys -ro=20 > /export/old/ifm/solaris-x86 -sec=3Dkrb5:krb5i:krb5p:sys -ro=20 > /export/old/ifm/windows-86 -sec=3Dkrb5:krb5i:krb5p:sys -ro=20 > /export/software -sec=3Dkrb5:krb5i:krb5p -ro=20 >=20 >=20 > 3. Make sure you have host/FQDN@REALM and nfs/FQDN@REALM Kerberos > principals in /etc/krb5.keytab and that FQDN is listed in /etc/hosts > like: >=20 > 1.2.3.4 foo.bar.com <http://foo.bar.com/>; foo >=20 > 4. rc.conf (we have a lot of users in our AD so we have to use a > large number for usermax, replace =E2=80=9Cliu.se <http://liu.se/>=E2=80= =9D with your > NFSv4 =E2=80=9Cdomain=E2=80=9D for nfsuserd_flags) >=20 > gssd_enable=3D"YES" > nfs_server_enable=3D"YES" > nfsv4_server_enable=3D"YES" > nfscbd_enable=3D"YES" > mountd_enable=3D"YES" > nfsuserd_enable=3D"YES" > nfsuserd_flags=3D"-manage-gids -domain liu.se -usertimeout 10 -usermax > 100000 16" >=20 > 5. Make sure you use NTPD so the clock is correct.=20 >=20 >=20 > * All clients (Solaris 10, OmniOS, MacOS 10.12-10.14, FreeBSD 11.0- > 11.2, CentOS 7, Debian 9, Ubuntu 17-18 tested): >=20 > 1. Make sure FQDN is in /etc/hosts >=20 > 2. Make sure you use NTPD so the clock is correct. >=20 > 3. Have a =E2=80=9Chost/FQDN@REALM=E2=80=9D Kerberos host principal in > /etc/krb5.keytab (nfs or root is not needed for NFS-mounting to work) >=20 > 4. We use a fairly default /etc/krb5.conf, sort of like: >=20 > > [libdefaults] > > default_realm =3D REALM > > dns_lookup_realm =3D true > >=20 > > ticket_lifetime =3D 24h > > renew_lifetime =3D 7d > > forwardable =3D true > >=20 > > default_ccache_name =3D KEYRING:persistent:%{uid} >=20 > KEYRING probably only works on Linux and there are some problems with > KEYRING in Debian & Ubuntu since not everything in it supports it due > to them using Heimdal instead of MIT (like for smbclient) but it > mostly works. Works fine in CentOS 7 though - in general CentOS 7 > feels more =E2=80=9Centerprise=E2=80=9D-ready than Debian & Ubuntu. The o= ld classic > FILE-ccaches should work fine though. >=20 > For mounting we use the automounter and a "executable map=E2=80=9D (perl > script) that looks up records in DNS (Hesiod-style) since the built- > in Hesiod support in most automounters is a bit.. lacking. Works > quite well. You can find the scripts we use here: >=20 > http://www.grebo.net/~peter/nfs < > http://www.grebo.net/~peter/nfs>; >=20 > (The dns-update scripts use data from an SQL database so probably > isn=E2=80=99t directly usable to anybody else. We use the same SQL databa= se > to populate a locally developed BerkeleyDB-based NSS-database on each > FreeBSD server in order to speed things up since AD/LDAP-looks with > ~90k users and silly amounts of AD groups takes forever, even with > cacheing). >=20 > Some Linux-specific stuff:=20 >=20 > Packages needed: >=20 > CentOS: > - nfs-utils > - libnfsidmap > - nfs4-acl-tools > - autofs >=20 > Debian: > - keyutils > - nfs-kernel-server # rpc.idmapd needs this due to a bug in Debian >=20 > Ubuntu: > - keyutils >=20 > Other nice-to have packages: > - hesiod > - autofs-hesiod >=20 > Some settings to check for: >=20 > /etc/default/nfs-common: > NEED_IDMAPD=3Dyes > NEED_GSSD=3Dyes >=20 > /etc/idmapd.conf (replace =E2=80=9Cliu.se <http://liu.se/>=E2=80=9D wit= h your NFSv4 > =E2=80=9Cdomain=E2=80=9D): > Domain=3Dliu.se >=20 > /etc/request-key.d/id_resolver.conf (should be there already if > using a modern Linux and you=E2=80=99ve added the packages above): > create id_resolver * * /usr/sbin/nfsidmap %k > %d >=20 >=20 > MacOS: >=20 > Basically require the latest - 10.14 (Mojave) - for things to work > smoothly. In theory 10.12 & 10.13 should work but there is some bug > in them that causes the OS to panic when you try to use NFS & > Kerberos. 10.11 and earlier doesn=E2=80=99t support good enough encryptio= n > for us=E2=80=A6 But with 10.14 you just need to get a Kerberos ticket an= d > then you can mount things just fine. >=20 > /etc/nfs.conf should contain (replace =E2=80=9Cliu.se <http://liu.se/>=E2= =80=9D with > your NFSv4 =E2=80=9Cdomain=E2=80=9D): > nfs.client.default_nfs4domain=3Dliu.se <http://liu.se/>; >=20 >=20 >=20 > (There are a lot of problems you can run into with Microsofts AD > implementation of Kerberos too that we=E2=80=99ve had to be fighting with= , > but that=E2=80=99s a whole other topic) >=20 > - Peter >=20 >=20 > > On 10 Oct 2018, at 23:47, Rick Macklem <rmacklem@uoguelph.ca> > > wrote: > >=20 > > Felix Winterhalter wrote: > > On 10/4/18 5:21 PM, Rick Macklem wrote: > > [stuff snipped] > > > > > I am now trying to mount this directory as root first without > > > > > having to > > > > > deal with user keytabs or tickets. > > > > >=20 > > > > > This works fine with -sec=3Dsys and nfsv4.1 and nfsv3 and > > > > > -sec=3Dkrb5p. > > > > > This does not however work with nfsv4 and krb5p or any other > > > > > krb5 flavor. > > > >=20 > > > > Sorry, I'm not sure what you are saying here. Is it > > > > 1 - no version of NFS works for krb5p or > > > > 2 - NFSv4.1 works for krb5p, but NFSv4.0 does not or > > > > 3 - only nfsv3 works for krb5p > >=20 > > [snipped lots of text] > > >=20 > > > #3 is indeed what was happening. I could mount with krb5p for > > > nfsv3 > > > (which I was not aware was even doable) however nfsv4 would > > > stubbornly > > > refuse to do any mounting. > >=20 > > Yes, RPCSEC_GSS was done by Sun for NFSv3 and it was a good fit, > > since NFSv3 > > does not have any server state to maintain. As such, all RPCs are > > atomic operations > > done by users (which for Kerberized mounts must have a TGT in a > > credential cache). > >=20 > > NFSv4 wasn't really a good fit for the model, because the NFSv4 > > server maintains > > lock state (NFSv4 Opens are a form of lock used by Windows at file > > open time). > > There are "state maintenance" operations that must be done by the > > user doing > > the mount (usually root), where they typically don't have a TGT in > > a credential > > cache. > > --> The ugly solution for this is typically a host-based client > > credential in a keytab > > on the client. (Usually a principal like " > > root/client-host.domain@REALM" or > > "host/client-host.domain@REALM" or " > > nfs/client-host.domain@REALM" > > in the default keytab on the client.) > >=20 > > > I have now after a lot of try and error figured out what I need > > > to do in > > > order to make it work. > > >=20 > > > To start with I have kerberos credentials with both host/ and > > > nfs/ on > > > both client and server. Mounting nfsv4 shares with krb5p from a > > > linux > > > server has also worked in this context. > >=20 > > Yes, I'm assuming that satisfied the host-based client credential > > as I described > > above. > >=20 > > > I leave you to judge whether what I found out is intended > > > behaviour or > > > if something weird is going on. > >=20 > > Yes, sounds like intended behaviour, since the client must have a > > Kerberos > > credential to use for the "state maintenance" operations that are > > not done on > > behalf of a user. > >=20 > > > My exports file originally looked something like this: > > >=20 > > > /nfsTests/ /nfsTests/testexport /nfsTests/otherexport > > > -maproot=3Droot > > > -sec=3Dkrb5p clients > > >=20 > > > V4: /nfsTests -sec=3Dkrb5p clients > > >=20 > > > Which allowed me to do nfsv3 krb5p mounts but not nfsv4 krb5p > > > mounts. > > >=20 > > > Changing the exports file to this: > > >=20 > > > /nfsTests/ /nfsTests/testexport /nfsTests/otherexport > > > -maproot=3Droot > > > -sec=3Dkrb5p clients > > >=20 > > > V4: /nfsTests -sec=3Dkrb5p,krb5i clients > >=20 > > This suggests that there is a bug in the client, where it uses > > krb5i instead of krb5p > > at some point in the mounting process. (I have also seen cases > > where the client > > erroneously falls back on using sys at some point in the mounting > > process.) > > (You did mention before you were using the Linux client. If you are > > using a FreeBSD > > client, I would be interested in looking at this.) > >=20 > > > Allows nfsv4 krb5p mounts to work for some reason I do not > > > understand. > > > Not setting the -sec option on the V4 line apparently defaults to > > > -sec=3Dsys and doesn't allow any krb5 mounts. I'm not sure that > > > this is a > > > good default as I wasn't even aware that the -sec option needed > > > to be > > > set on this line. > >=20 > > In FreeBSD, defaults are meant to maintain backwards compatibility. > > This means that > > AUTH_SYS should work by default. Also, AUTH_SYS is what 99.9999% of > > FreeBSD > > NFS mounts still use, from what I've seen.) > >=20 > > > I've got packet traces of the nfsv3 krb5 and krb5i mounts and > > > I'll make > > > traces of the two nfsv4 mount attempts and send them to you if > > > you're > > > interested. I'm still not sure what exactly is happening here. > >=20 > > The successful one for NFSv4 might be interesting. If you look at > > it in > > wireshark, I suspect you'd find somewhere during the mount that it > > did RPCs which were not krb5p and that would show why the addition > > of krb5i made it work. > >=20 > > I did suggest you start with -sec=3Dsys:krb5:krb5i:krb5p and, once > > that works, > > remove the security flavours one at a time until the mount doesn't > > work. > > (Then you capture packets for the minimal case that does work and > > look at > > what security flavours the client is using for all RPCs done during > > the mount.) > >=20 > > You now know why almost no one uses Kerberized NFSv4 mounts. > > Unfortunately, the NFSv4 working group has never gotten around to > > a better solution. Discussion of a host based encryption technique > > using > > something like SSL has happened, but no one has gone beyond that. > >=20 > > rick > > _______________________________________________ > > freebsd-fs@freebsd.org <mailto:freebsd-fs@freebsd.org> mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs < > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs>; > > To unsubscribe, send any mail to " > > freebsd-fs-unsubscribe@freebsd.org <mailto: > > freebsd-fs-unsubscribe@freebsd.org>" >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --=-6HWtWOXsHgjO/mRWq1SE Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iQEzBAABCgAdFiEEw0f5gWr+/IyirHlmGmjXrg3Zy5EFAlu/OOcACgkQGmjXrg3Z y5FJYAf+JNciD69KSLg0WtgA4ZRxLw2VHiA27e4ROK/5SFdi3kyIZPYpze2KQC+V fOHjwJv6cakRUFr3m5uJoNOwEmoXfr0RWRhoSC4b/3k+SmCLEaeLXz1o0GA2IU2q gF+L74i09sY3nZYZh4BX17gTRCxmpF/Mo3mx1uOiM7teiZBSQ6zXT6zGXK04xgA1 tn9Utk+XDjt7ILTpgX7mAPwepywVb0h7j+8xXfzAupKJpi/Q8fQU0BUoKlrZhg6h UVkFqKl5uQtLMyBqwn1tm2uALj8T3h4H9um1L02xr3kTeJkbpbNZpFt2aQ7mX21F SbrMImalBRg9C6C8J2kGNsk9EW18yA== =HKAY -----END PGP SIGNATURE----- --=-6HWtWOXsHgjO/mRWq1SE--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?054a82958e9c1bed12c3a159d69db36b4cfcb5ec.camel>