Date: Thu, 11 Oct 2018 10:36:27 +0200 From: Peter Eriksson <peter@ifm.liu.se> To: Rick Macklem <rmacklem@uoguelph.ca>, Felix Winterhalter <felix@audiofair.de> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: NFSv4 Kerberos mount from Linux Message-ID: <33A0F0BC-4AD8-4DE3-B484-42B7FB208B6A@ifm.liu.se> In-Reply-To: <YTOPR0101MB18207F35A3973F26C6A58F6ADDE00@YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM> References: <30f6446c-6fed-4b1e-9cae-9c417974ec46@audiofair.de> <YTOPR0101MB1820A5756D172342AF441C25DDEA0@YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM> <c1ffda48-3809-bb4c-6d97-451765b0e25e@audiofair.de> <YTOPR0101MB18207F35A3973F26C6A58F6ADDE00@YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
Just a few comments (random brain-dump) in case someone else is having = problems with NFS & Kerberos.=20 We=E2=80=99ve been using NFSv4 with Kerberos from Linux clients here for = many years (with Solaris-based NFS servers and MIT Kerberos) and lately = using FreeBSD as the NFS server OS (in an Microsoft AD Kerberos = environment).=20 There are a few differences in behaviour (from a Solaris perspective), = mainly regarding the =E2=80=9Cpseudo=E2=80=9D NFSv4 filesystem but not = something that can=E2=80=99t be handled. In the process of moving to = FreeBSD based servers I=E2=80=99ve also been testing lots of different = clients in order to make sure everything works as expected. Here are = some notes: General stuff: 1. Have a _kerberos.my.zone.com DNS TXT record containing the Kerberos = realm (nice to have) 2. Have a _nfsv4domain.my.zone.com <http://nfsv4domain.my.zone.com/>; DNS = TXT record containing the NFSv4 =E2=80=9Cdomain=E2=80=9D (not all = clients use it, but it=E2=80=99s nice to have) * FreeBSD server (with ZFS filesystems), things below /export is = NFS-shared as (for example) server:/staff/user1 1. /etc/exports (we _only_ allow sec=3Dkrb<various flavours> for home = directories): V4: /export -sec=3Dkrb5:krb5i:krb5p Or (on a server also serving some (read-only) sec=3Dsys filesystems = below /exports) V4: /export -sec=3Dkrb5:krb5i:krb5p:sys 2. /etc/zfs/exports (generated from =E2=80=9Cset sharenfs=3Dxxx=E2=80=9D = on the ZFS filesystems) Home-server: /export/staff -sec=3Dkrb5:krb5i:krb5p=20= /export/staff/user1 -sec=3Dkrb5:krb5i:krb5p=20 /export/staff/user2 -sec=3Dkrb5:krb5i:krb5p=20 /export/students -sec=3Dkrb5:krb5i:krb5p=20 /export/students/user3 -sec=3Dkrb5:krb5i:krb5p=20 Software-server: /export/db/icons -sec=3Dsys -ro=20 /export/old/ifm/solaris-x86 -sec=3Dkrb5:krb5i:krb5p:sys -ro=20= /export/old/ifm/windows-86 -sec=3Dkrb5:krb5i:krb5p:sys -ro=20 /export/software -sec=3Dkrb5:krb5i:krb5p -ro=20 3. Make sure you have host/FQDN@REALM and nfs/FQDN@REALM Kerberos = principals in /etc/krb5.keytab and that FQDN is listed in /etc/hosts = like: 1.2.3.4 foo.bar.com <http://foo.bar.com/>; foo 4. rc.conf (we have a lot of users in our AD so we have to use a large = number for usermax, replace =E2=80=9Cliu.se <http://liu.se/>=E2=80=9D = with your NFSv4 =E2=80=9Cdomain=E2=80=9D for nfsuserd_flags) gssd_enable=3D"YES" nfs_server_enable=3D"YES" nfsv4_server_enable=3D"YES" nfscbd_enable=3D"YES" mountd_enable=3D"YES" nfsuserd_enable=3D"YES" nfsuserd_flags=3D"-manage-gids -domain liu.se -usertimeout 10 -usermax = 100000 16" 5. Make sure you use NTPD so the clock is correct.=20 * All clients (Solaris 10, OmniOS, MacOS 10.12-10.14, FreeBSD 11.0-11.2, = CentOS 7, Debian 9, Ubuntu 17-18 tested): 1. Make sure FQDN is in /etc/hosts 2. Make sure you use NTPD so the clock is correct. 3. Have a =E2=80=9Chost/FQDN@REALM=E2=80=9D Kerberos host principal in = /etc/krb5.keytab (nfs or root is not needed for NFS-mounting to work) 4. We use a fairly default /etc/krb5.conf, sort of like: > [libdefaults] > default_realm =3D REALM > dns_lookup_realm =3D true >=20 > ticket_lifetime =3D 24h > renew_lifetime =3D 7d > forwardable =3D true >=20 > default_ccache_name =3D KEYRING:persistent:%{uid} KEYRING probably only works on Linux and there are some problems with = KEYRING in Debian & Ubuntu since not everything in it supports it due to = them using Heimdal instead of MIT (like for smbclient) but it mostly = works. Works fine in CentOS 7 though - in general CentOS 7 feels more = =E2=80=9Centerprise=E2=80=9D-ready than Debian & Ubuntu. The old classic = FILE-ccaches should work fine though. For mounting we use the automounter and a "executable map=E2=80=9D (perl = script) that looks up records in DNS (Hesiod-style) since the built-in = Hesiod support in most automounters is a bit.. lacking. Works quite = well. You can find the scripts we use here: http://www.grebo.net/~peter/nfs = <http://www.grebo.net/~peter/nfs>; (The dns-update scripts use data from an SQL database so probably = isn=E2=80=99t directly usable to anybody else. We use the same SQL = database to populate a locally developed BerkeleyDB-based NSS-database = on each FreeBSD server in order to speed things up since AD/LDAP-looks = with ~90k users and silly amounts of AD groups takes forever, even with = cacheing). Some Linux-specific stuff:=20 Packages needed: CentOS: - nfs-utils - libnfsidmap - nfs4-acl-tools - autofs Debian: - keyutils - nfs-kernel-server # rpc.idmapd needs this due to a bug in Debian Ubuntu: - keyutils Other nice-to have packages: - hesiod - autofs-hesiod Some settings to check for: /etc/default/nfs-common: NEED_IDMAPD=3Dyes NEED_GSSD=3Dyes /etc/idmapd.conf (replace =E2=80=9Cliu.se <http://liu.se/>=E2=80=9D = with your NFSv4 =E2=80=9Cdomain=E2=80=9D): Domain=3Dliu.se /etc/request-key.d/id_resolver.conf (should be there already if using = a modern Linux and you=E2=80=99ve added the packages above): create id_resolver * * /usr/sbin/nfsidmap %k %d MacOS: Basically require the latest - 10.14 (Mojave) - for things to work = smoothly. In theory 10.12 & 10.13 should work but there is some bug in = them that causes the OS to panic when you try to use NFS & Kerberos. = 10.11 and earlier doesn=E2=80=99t support good enough encryption for = us=E2=80=A6 But with 10.14 you just need to get a Kerberos ticket and = then you can mount things just fine. /etc/nfs.conf should contain (replace =E2=80=9Cliu.se = <http://liu.se/>=E2=80=9D with your NFSv4 =E2=80=9Cdomain=E2=80=9D): nfs.client.default_nfs4domain=3Dliu.se <http://liu.se/>; (There are a lot of problems you can run into with Microsofts AD = implementation of Kerberos too that we=E2=80=99ve had to be fighting = with, but that=E2=80=99s a whole other topic) - Peter > On 10 Oct 2018, at 23:47, Rick Macklem <rmacklem@uoguelph.ca> wrote: >=20 > Felix Winterhalter wrote: > On 10/4/18 5:21 PM, Rick Macklem wrote: > [stuff snipped] >>>> I am now trying to mount this directory as root first without = having to >>>> deal with user keytabs or tickets. >>>>=20 >>>> This works fine with -sec=3Dsys and nfsv4.1 and nfsv3 and = -sec=3Dkrb5p. >>>> This does not however work with nfsv4 and krb5p or any other krb5 = flavor. >>> Sorry, I'm not sure what you are saying here. Is it >>> 1 - no version of NFS works for krb5p or >>> 2 - NFSv4.1 works for krb5p, but NFSv4.0 does not or >>> 3 - only nfsv3 works for krb5p > [snipped lots of text] >>=20 >> #3 is indeed what was happening. I could mount with krb5p for nfsv3 >> (which I was not aware was even doable) however nfsv4 would = stubbornly >> refuse to do any mounting. > Yes, RPCSEC_GSS was done by Sun for NFSv3 and it was a good fit, since = NFSv3 > does not have any server state to maintain. As such, all RPCs are = atomic operations > done by users (which for Kerberized mounts must have a TGT in a = credential cache). >=20 > NFSv4 wasn't really a good fit for the model, because the NFSv4 server = maintains > lock state (NFSv4 Opens are a form of lock used by Windows at file = open time). > There are "state maintenance" operations that must be done by the user = doing > the mount (usually root), where they typically don't have a TGT in a = credential > cache. > --> The ugly solution for this is typically a host-based client = credential in a keytab > on the client. (Usually a principal like = "root/client-host.domain@REALM" or > "host/client-host.domain@REALM" or "nfs/client-host.domain@REALM" > in the default keytab on the client.) >=20 >> I have now after a lot of try and error figured out what I need to do = in >> order to make it work. >>=20 >> To start with I have kerberos credentials with both host/ and nfs/ on >> both client and server. Mounting nfsv4 shares with krb5p from a linux >> server has also worked in this context. > Yes, I'm assuming that satisfied the host-based client credential as I = described > above. >=20 >> I leave you to judge whether what I found out is intended behaviour = or >> if something weird is going on. > Yes, sounds like intended behaviour, since the client must have a = Kerberos > credential to use for the "state maintenance" operations that are not = done on > behalf of a user. >=20 >> My exports file originally looked something like this: >>=20 >> /nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=3Droot >> -sec=3Dkrb5p clients >>=20 >> V4: /nfsTests -sec=3Dkrb5p clients >>=20 >> Which allowed me to do nfsv3 krb5p mounts but not nfsv4 krb5p mounts. >>=20 >> Changing the exports file to this: >>=20 >> /nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=3Droot >> -sec=3Dkrb5p clients >>=20 >> V4: /nfsTests -sec=3Dkrb5p,krb5i clients > This suggests that there is a bug in the client, where it uses krb5i = instead of krb5p > at some point in the mounting process. (I have also seen cases where = the client > erroneously falls back on using sys at some point in the mounting = process.) > (You did mention before you were using the Linux client. If you are = using a FreeBSD > client, I would be interested in looking at this.) >=20 >> Allows nfsv4 krb5p mounts to work for some reason I do not = understand. >> Not setting the -sec option on the V4 line apparently defaults to >> -sec=3Dsys and doesn't allow any krb5 mounts. I'm not sure that this = is a >> good default as I wasn't even aware that the -sec option needed to be >> set on this line. > In FreeBSD, defaults are meant to maintain backwards compatibility. = This means that > AUTH_SYS should work by default. Also, AUTH_SYS is what 99.9999% of = FreeBSD > NFS mounts still use, from what I've seen.) >=20 >> I've got packet traces of the nfsv3 krb5 and krb5i mounts and I'll = make >> traces of the two nfsv4 mount attempts and send them to you if you're >> interested. I'm still not sure what exactly is happening here. > The successful one for NFSv4 might be interesting. If you look at it = in > wireshark, I suspect you'd find somewhere during the mount that it > did RPCs which were not krb5p and that would show why the addition > of krb5i made it work. >=20 > I did suggest you start with -sec=3Dsys:krb5:krb5i:krb5p and, once = that works, > remove the security flavours one at a time until the mount doesn't = work. > (Then you capture packets for the minimal case that does work and look = at > what security flavours the client is using for all RPCs done during = the mount.) >=20 > You now know why almost no one uses Kerberized NFSv4 mounts. > Unfortunately, the NFSv4 working group has never gotten around to > a better solution. Discussion of a host based encryption technique = using > something like SSL has happened, but no one has gone beyond that. >=20 > rick > _______________________________________________ > freebsd-fs@freebsd.org <mailto:freebsd-fs@freebsd.org> mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs = <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>; > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org = <mailto:freebsd-fs-unsubscribe@freebsd.org>"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?33A0F0BC-4AD8-4DE3-B484-42B7FB208B6A>