Date: Thu, 10 Dec 2020 21:00:07 +0100 From: Peter Eriksson <pen@lysator.liu.se> To: J David <j.david.lists@gmail.com> Cc: FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: Major issues with nfsv4 Message-ID: <976EC1BD-AB37-478C-B567-E8013E80F071@lysator.liu.se> In-Reply-To: <CABXB=RRB2nUk0pPDisBQPdicUA3ooHpg8QvBwjG_nFU4cHvCYw@mail.gmail.com> References: <CABXB=RRB2nUk0pPDisBQPdicUA3ooHpg8QvBwjG_nFU4cHvCYw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Any particular reason you choose to use NFSv4.0 and not NFSv4.1? Also, it might be useful information if you could show the configuration = your are using on the server and the clients. Are the client FreeBSD = 12.2 also or (more common) some Linux variant? We are using NFS v4.0 and v4.1 with great success here from our FreeBSD = 12.1, 12.2 and 11.3 servers from various Linux (and some OmniOS clients = - only 4.0 on those) with=20 Kerberos. With NFSv4 there are some additional things you need to set up compared = to NFSv3. For example the NFS-Domain name which must be the same on = servers & clients, and you must run the nfsuserd daemon, and have the V4 = export line. Our NFS server setup: > root:/etc # egrep 'nfs|gss|sec' rc.conf rc.conf.d/* /boot/loader.conf = /etc/sysctl.conf exports zfs/exports >=20 > rc.conf:gssd_enable=3D"YES" > rc.conf:nfs_server_enable=3D"YES" > rc.conf:nfsv4_server_enable=3D"YES" > rc.conf:nfscbd_enable=3D"YES" >=20 > rc.conf.d/nfsuserd:nfsuserd_enable=3D"YES" > rc.conf.d/nfsuserd:nfsuserd_flags=3D"-manage-gids -domain = your.nfs.domain.id 16" >=20 > exports:V4: /export -sec=3Dkrb5:krb5i:krb5p=09 >=20 > zfs/exports:/export/staff -sec=3Dkrb5:krb5i:krb5p=20 On a Linux client (Debian for example) you need to configure the = NFS-domain, make sure the idmap/gssd stuff is running and make sure you = nfsmount correctly=E2=80=A6 /etc/default/nfs-common NEED_IDMAPD=3Dyes NEED_GSSD=3Dyes /etc/idmapd.conf [general] Domain =3D your.nfs.domain.id Local-Realms =3D YOUR-KRB5-REALM /etc/nfsmount.conf [NFSMount_Global_Options] Defaultvers =3D 4.1 Packages need on Linux clients:=20 keyutils nfs-kernel-server (on Debian 9) Nfs-utils libnfsidmap nfs4-acl-tools rpcgssd (CentOS 7) We use =E2=80=9Cfstype=3Dnfs4,sec=3Dkrb5=E2=80=9D when mounting on the = Linux clients. At least on CentOS 7 if you use = =E2=80=9Cfstype=3Dnfs,vers=3D4,sec=3Dkrb5=E2=80=9D then iit will use 4.0 = instead of the highest supported NFS version=E2=80=A6 - Peter > On 10 Dec 2020, at 17:15, J David <j.david.lists@gmail.com> wrote: >=20 > Recently, we attempted to get with the 2000's and try switching from > NFSv3 to NFSv4 on our 12.2 servers. This has not gone well. >=20 > Any system we switch to NFSv4 mounts is functionally unusable, pegged > at 100% system CPU usage, load average 70+, largely from nfscl threads > and client processes using NFS. >=20 > Dmesg shows NFS-related messages: >=20 > $ dmesg | fgrep -i nfs | sort | uniq -c | sort -n > 1 nfsv4 err=3D10010 > 4 nfsv4 client/server protocol prob err=3D10026 > 29 nfscl: never fnd open >=20 > Nfsstat shows no client activity; "nfsstat -e -c 1" and "nfsstat -c 1" > both report: >=20 > GtAttr Lookup Rdlink Read Write Rename Access Rddir > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 >=20 > Meanwhile, tcpdump on the client shows an endless stream of getattr > requests at the exact same time nfsstat -c says nothing is happening: >=20 > $ sudo tcpdump -n -i net1 -c 10 port 2049 and src 172.20.200.39 > 14:47:27.037974 IP 172.20.200.39.727 > 172.20.20.161.2049: Flags [.], > ack 72561, win 545, options [nop,nop,TS val 234259249 ecr 4155804100], > length 0 > 14:47:27.046282 IP 172.20.200.39.727 > 172.20.20.161.2049: Flags [P.], > seq 139940:140092, ack 72561, win 545, options [nop,nop,TS val > 234259259 ecr 4155804100], length 152: NFS request xid 1544756021 148 > getattr fh 0,5/0 > 14:47:27.051260 IP 172.20.200.39.727 > 172.20.20.161.2049: Flags [P.], > seq 140092:140248, ack 72641, win 545, options [nop,nop,TS val > 234259269 ecr 4155804104], length 156: NFS request xid 1544756022 152 > getattr fh 0,5/0 > 14:47:27.063372 IP 172.20.200.39.727 > 172.20.20.161.2049: Flags [P.], > seq 140248:140404, ack 72721, win 545, options [nop,nop,TS val > 234259279 ecr 4155804106], length 156: NFS request xid 1544756023 152 > getattr fh 0,5/0 > 14:47:27.068646 IP 172.20.200.39.727 > 172.20.20.161.2049: Flags [P.], > seq 140404:140556, ack 72801, win 545, options [nop,nop,TS val > 234259279 ecr 4155804108], length 152: NFS request xid 1544756024 148 > getattr fh 0,5/0 > 14:47:27.080627 IP 172.20.200.39.727 > 172.20.20.161.2049: Flags [P.], > seq 140556:140712, ack 72881, win 545, options [nop,nop,TS val > 234259299 ecr 4155804110], length 156: NFS request xid 1544756025 152 > getattr fh 0,5/0 > 14:47:27.085224 IP 172.20.200.39.727 > 172.20.20.161.2049: Flags [P.], > seq 140712:140868, ack 72961, win 545, options [nop,nop,TS val > 234259299 ecr 4155804112], length 156: NFS request xid 1544756026 152 > getattr fh 0,5/0 > 14:47:27.096802 IP 172.20.200.39.727 > 172.20.20.161.2049: Flags [P.], > seq 140868:141024, ack 73041, win 545, options [nop,nop,TS val > 234259309 ecr 4155804114], length 156: NFS request xid 1544756027 152 > getattr fh 0,5/0 > 14:47:27.101849 IP 172.20.200.39.727 > 172.20.20.161.2049: Flags [P.], > seq 141024:141180, ack 73121, win 545, options [nop,nop,TS val > 234259319 ecr 4155804116], length 156: NFS request xid 1544756028 152 > getattr fh 0,5/0 > 14:47:27.112905 IP 172.20.200.39.727 > 172.20.20.161.2049: Flags [P.], > seq 141180:141336, ack 73201, win 545, options [nop,nop,TS val > 234259329 ecr 4155804118], length 156: NFS request xid 1544756029 152 > getattr fh 0,5/0 >=20 > Only 10 shown here for brevity, but: >=20 > $ sudo tcpdump -n -i net1 -c 10000 port 2049 and src 172.20.200.39 | > fgrep getattr | wc -l > tcpdump: verbose output suppressed, use -v or -vv for full protocol = decode > listening on net1, link-type EN10MB (Ethernet), capture size 262144 = bytes > 10000 packets captured > 20060 packets received by filter > 0 packets dropped by kernel > 9759 >=20 > There are no dropped packets or network problems: >=20 > $ netstat -in -I net1 > Name Mtu Network Address Ipkts Ierrs Idrop > Opkts Oerrs Coll > net1 1500 <Link#2> 12:33:df:5f:79:d7 40988832 0 0 > 48760307 0 0 > net1 - 172.20.0.0/16 172.20.200.39 40942065 - - > 48756241 - - >=20 > The mount flags in fstab are: >=20 > ro,nfsv4,nosuid >=20 > The mount flags as reported by "nfsstat -m" are: >=20 > = nfsv4,minorversion=3D0,tcp,resvport,hard,cto,sec=3Dsys,acdirmin=3D3,acdirm= ax=3D60,acregmin=3D5,acregmax=3D60,nametimeo=3D60,negnametimeo=3D60,rsize=3D= 65536,wsize=3D65536,readdirsize=3D65536,readahead=3D1,wcommitsize=3D167772= 16,timeout=3D120,retrans=3D2147483647 >=20 > Today, I managed to kill everything down to one user process that was > exhibiting this behavior. After a kill -9 on that process, it went to > "REsJ" but continued to burn the same amount of CPU (all system). > Oddly the run state / wait channel was just "CPU1." Running "ktrace" > did not produce any trace records. Probably that is predictable for a > process in E state; if the process had crossed the user/kernel > boundary in a way ktrace could detect, it would have exited. >=20 > At that point, I started unmounting filesystems. Everything but the > NFS filesystem used by that process unmounted cleanly. The umount for > that filesystem went to D state for about a minute and then kicked > back "Device busy." That's fair, if awfully slow. >=20 > Meanwhile, that user process continued burning system CPU with the E > flag set, not doing anything whatsoever in userspace, still producing > 300+ "getattr fh 0,5/0" per second according to tcpdump and 0 > according to nfsstat. >=20 > Eventually, I rebooted with fstab set back to nfsv3. >=20 > This feels like the user process is in a system call that is stuck in > an endless loop repeating some operation that generates that getattr > request. But that is a feeling, not a fact. >=20 > This is fairly easy to reproduce; it seems pretty consistent within a > few hours (a day at most) any time I switch the relevant mounts to > nfsv4. Reverting to nfsv3 makes this issue completely disappear. >=20 > What on earth could be going on here? What other information can I > provide that would help track this down? >=20 > Thanks for any advice! > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?976EC1BD-AB37-478C-B567-E8013E80F071>