Date: Fri, 12 Oct 2018 00:37:55 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Peter Eriksson <peter@ifm.liu.se>, Felix Winterhalter <felix@audiofair.de> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: NFSv4 Kerberos mount from Linux Message-ID: <YTOPR0101MB1820439E0BFBF57DB2572E92DDE20@YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <33A0F0BC-4AD8-4DE3-B484-42B7FB208B6A@ifm.liu.se> References: <30f6446c-6fed-4b1e-9cae-9c417974ec46@audiofair.de> <YTOPR0101MB1820A5756D172342AF441C25DDEA0@YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM> <c1ffda48-3809-bb4c-6d97-451765b0e25e@audiofair.de> <YTOPR0101MB18207F35A3973F26C6A58F6ADDE00@YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM>, <33A0F0BC-4AD8-4DE3-B484-42B7FB208B6A@ifm.liu.se>
next in thread | previous in thread | raw e-mail | index | archive | help
From: Peter Eriksson wrote: >Just a few comments (random brain-dump) in case someone else is having problems >with NFS & Kerberos. > > >We’ve been using NFSv4 with Kerberos from Linux clients here for many years (with >Solaris-based NFS servers and MIT Kerberos) and lately using FreeBSD as the NFS >server OS (in an Microsoft AD Kerberos environment). I agree with the other post that this is a very useful document and it would be nice to keep it somewhere where it is easier to find than in the mailing list archive. I don't know where that could be, so hopefully someone else in FreeBSD-land can suggest a good place? The one area you don't discuss (and maybe isn't really a problem?) is what ticket encryption type(s) you use. Kerberized NFS still uses DES (someday this may change, but I think that requires implementation of RPCSEC_GSS V3), so it needs an 8byte session key. (I have never seen a documented way to convert a session key of greater than 8bytes into an 8byte session key for RPCSEC_GSS to use. As such, I have no idea what happens if you choose a ticket encryption type that results in a greater than 8byte key.) Here's a couple of minor comments: [lots of good stuff snipped] >4. rc.conf (we have a lot of users in our AD so we have to use a large number for >usermax, replace “liu.se<http://liu.se>” with your NFSv4 “domain” for >nfsuserd_flags) > >gssd_enable="YES" >nfs_server_enable="YES" >nfsv4_server_enable="YES" >nfscbd_enable="YES" Since the nfscbd is only used by the client (and only when delegations or pNFS is enabled in the server), I believe this is harmless, but unnecessary for the server. >mountd_enable="YES" >nfsuserd_enable="YES" >nfsuserd_flags="-manage-gids -domain liu.se<http://liu.se>; -usertimeout 10 -usermax 100000 16" > Btw, if you have many clients doing NFSv4.0 mounts, tuning of your DRC is advised. (The defaults are for extremely small NFS servers.) NFSv4.1 mounts don't use the DRC, so if that is what your clients are doing, this isn't necessary. It's a little off topic, but for NFSv4.0 (and/or NFSv3) mounts and a reasonable sized NFS server, reasonable settings in /etc/sysctl.conf might be: vfs.nfsd.tcpcachetimeo=600 vfs.nfsd.tcphighwater=100000 vfs.nfsd.v4statelimit=10000000 vfs.nfsd.clienthashsize=10000 vfs.nfsd.statehashsize=10000 - If most/all of your mounts are NFSv4.1, then instead of the above, you might want: vfs.nfsd.sessionhashsize=10000 >5. Make sure you use NTPD so the clock is correct. > > >* All clients (Solaris 10, OmniOS, MacOS 10.12-10.14, FreeBSD 11.0-11.2, CentOS 7, >Debian 9, Ubuntu 17-18 tested): > >1. Make sure FQDN is in /etc/hosts > >2. Make sure you use NTPD so the clock is correct. > >3. Have a “host/FQDN@REALM” Kerberos host principal in /etc/krb5.keytab (nfs or >root is not needed for NFS-mounting to work) I'm guessing that you use the "gssname=host" mount option for your FreeBSD clients? (The name after "gssname=" is whatever name you have before "/FQDN@REALM" for this principal name.) This is what I referred to as the host-based initiator credential. Thanks for posting this, rick 4. We use a fairly default /etc/krb5.conf, sort of like: [libdefaults] default_realm = REALM dns_lookup_realm = true ticket_lifetime = 24h renew_lifetime = 7d forwardable = true default_ccache_name = KEYRING:persistent:%{uid} KEYRING probably only works on Linux and there are some problems with KEYRING in Debian & Ubuntu since not everything in it supports it due to them using Heimdal instead of MIT (like for smbclient) but it mostly works. Works fine in CentOS 7 though - in general CentOS 7 feels more “enterprise”-ready than Debian & Ubuntu. The old classic FILE-ccaches should work fine though. For mounting we use the automounter and a "executable map” (perl script) that looks up records in DNS (Hesiod-style) since the built-in Hesiod support in most automounters is a bit.. lacking. Works quite well. You can find the scripts we use here: http://www.grebo.net/~peter/nfs (The dns-update scripts use data from an SQL database so probably isn’t directly usable to anybody else. We use the same SQL database to populate a locally developed BerkeleyDB-based NSS-database on each FreeBSD server in order to speed things up since AD/LDAP-looks with ~90k users and silly amounts of AD groups takes forever, even with cacheing). Some Linux-specific stuff: Packages needed: CentOS: - nfs-utils - libnfsidmap - nfs4-acl-tools - autofs Debian: - keyutils - nfs-kernel-server # rpc.idmapd needs this due to a bug in Debian Ubuntu: - keyutils Other nice-to have packages: - hesiod - autofs-hesiod Some settings to check for: /etc/default/nfs-common: NEED_IDMAPD=yes NEED_GSSD=yes /etc/idmapd.conf (replace “liu.se<http://liu.se>” with your NFSv4 “domain”): Domain=liu.se<http://liu.se>; /etc/request-key.d/id_resolver.conf (should be there already if using a modern Linux and you’ve added the packages above): create id_resolver * * /usr/sbin/nfsidmap %k %d MacOS: Basically require the latest - 10.14 (Mojave) - for things to work smoothly. In theory 10.12 & 10.13 should work but there is some bug in them that causes the OS to panic when you try to use NFS & Kerberos. 10.11 and earlier doesn’t support good enough encryption for us… But with 10.14 you just need to get a Kerberos ticket and then you can mount things just fine. /etc/nfs.conf should contain (replace “liu.se<http://liu.se>” with your NFSv4 “domain”): nfs.client.default_nfs4domain=liu.se<http://liu.se>; (There are a lot of problems you can run into with Microsofts AD implementation of Kerberos too that we’ve had to be fighting with, but that’s a whole other topic) - Peter On 10 Oct 2018, at 23:47, Rick Macklem <rmacklem@uoguelph.ca<mailto:rmacklem@uoguelph.ca>> wrote: Felix Winterhalter wrote: On 10/4/18 5:21 PM, Rick Macklem wrote: [stuff snipped] I am now trying to mount this directory as root first without having to deal with user keytabs or tickets. This works fine with -sec=sys and nfsv4.1 and nfsv3 and -sec=krb5p. This does not however work with nfsv4 and krb5p or any other krb5 flavor. Sorry, I'm not sure what you are saying here. Is it 1 - no version of NFS works for krb5p or 2 - NFSv4.1 works for krb5p, but NFSv4.0 does not or 3 - only nfsv3 works for krb5p [snipped lots of text] #3 is indeed what was happening. I could mount with krb5p for nfsv3 (which I was not aware was even doable) however nfsv4 would stubbornly refuse to do any mounting. Yes, RPCSEC_GSS was done by Sun for NFSv3 and it was a good fit, since NFSv3 does not have any server state to maintain. As such, all RPCs are atomic operations done by users (which for Kerberized mounts must have a TGT in a credential cache). NFSv4 wasn't really a good fit for the model, because the NFSv4 server maintains lock state (NFSv4 Opens are a form of lock used by Windows at file open time). There are "state maintenance" operations that must be done by the user doing the mount (usually root), where they typically don't have a TGT in a credential cache. --> The ugly solution for this is typically a host-based client credential in a keytab on the client. (Usually a principal like "root/client-host.domain@REALM" or "host/client-host.domain@REALM" or "nfs/client-host.domain@REALM" in the default keytab on the client.) I have now after a lot of try and error figured out what I need to do in order to make it work. To start with I have kerberos credentials with both host/ and nfs/ on both client and server. Mounting nfsv4 shares with krb5p from a linux server has also worked in this context. Yes, I'm assuming that satisfied the host-based client credential as I described above. I leave you to judge whether what I found out is intended behaviour or if something weird is going on. Yes, sounds like intended behaviour, since the client must have a Kerberos credential to use for the "state maintenance" operations that are not done on behalf of a user. My exports file originally looked something like this: /nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=root -sec=krb5p clients V4: /nfsTests -sec=krb5p clients Which allowed me to do nfsv3 krb5p mounts but not nfsv4 krb5p mounts. Changing the exports file to this: /nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=root -sec=krb5p clients V4: /nfsTests -sec=krb5p,krb5i clients This suggests that there is a bug in the client, where it uses krb5i instead of krb5p at some point in the mounting process. (I have also seen cases where the client erroneously falls back on using sys at some point in the mounting process.) (You did mention before you were using the Linux client. If you are using a FreeBSD client, I would be interested in looking at this.) Allows nfsv4 krb5p mounts to work for some reason I do not understand. Not setting the -sec option on the V4 line apparently defaults to -sec=sys and doesn't allow any krb5 mounts. I'm not sure that this is a good default as I wasn't even aware that the -sec option needed to be set on this line. In FreeBSD, defaults are meant to maintain backwards compatibility. This means that AUTH_SYS should work by default. Also, AUTH_SYS is what 99.9999% of FreeBSD NFS mounts still use, from what I've seen.) I've got packet traces of the nfsv3 krb5 and krb5i mounts and I'll make traces of the two nfsv4 mount attempts and send them to you if you're interested. I'm still not sure what exactly is happening here. The successful one for NFSv4 might be interesting. If you look at it in wireshark, I suspect you'd find somewhere during the mount that it did RPCs which were not krb5p and that would show why the addition of krb5i made it work. I did suggest you start with -sec=sys:krb5:krb5i:krb5p and, once that works, remove the security flavours one at a time until the mount doesn't work. (Then you capture packets for the minimal case that does work and look at what security flavours the client is using for all RPCs done during the mount.) You now know why almost no one uses Kerberized NFSv4 mounts. Unfortunately, the NFSv4 working group has never gotten around to a better solution. Discussion of a host based encryption technique using something like SSL has happened, but no one has gone beyond that. rick _______________________________________________ freebsd-fs@freebsd.org<mailto:freebsd-fs@freebsd.org> mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org<mailto:freebsd-fs-unsubscribe@freebsd.org>"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YTOPR0101MB1820439E0BFBF57DB2572E92DDE20>