From owner-freebsd-bugs@freebsd.org Tue Nov 24 18:24:03 2020 Return-Path: Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5E1F92EDC69 for ; Tue, 24 Nov 2020 18:24:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4CgXTv21hhz53Qg for ; Tue, 24 Nov 2020 18:24:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 438822EDC67; Tue, 24 Nov 2020 18:24:03 +0000 (UTC) Delivered-To: bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 434EC2EDA9D for ; Tue, 24 Nov 2020 18:24:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CgXTv0ythz53f9 for ; Tue, 24 Nov 2020 18:24:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0A7EF23666 for ; Tue, 24 Nov 2020 18:24:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 0AOIO2fd035822 for ; Tue, 24 Nov 2020 18:24:02 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 0AOIO2Da035821 for bugs@FreeBSD.org; Tue, 24 Nov 2020 18:24:02 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 251347] NFS hangs on client side when mounted from outside in Jail Tree (BROKEN NFS SERVER OR MIDDLEWARE) Date: Tue, 24 Nov 2020 18:24:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: arne@Steinkamm.COM X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Nov 2020 18:24:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D251347 Bug ID: 251347 Summary: NFS hangs on client side when mounted from outside in Jail Tree (BROKEN NFS SERVER OR MIDDLEWARE) Product: Base System Version: 12.1-STABLE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: arne@Steinkamm.COM Created attachment 219934 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D219934&action= =3Dedit tar file with all configs and command outputs of server, client and jail Hi folks, I ran into an really nasty problem after introducing FreeBSD to a really big company. All machines are running FreeBSD 12.1 Release and use 10GBit/s ethernet lin= ks with Cisco Nexus 55xx switches. Additional 1GBit/s networks for admin- und security related issues exists. I have this problem on all my NFS client machine running FreeBSD. NFS Server (FreeBSD 12.1) exports a ZFS dataset with ca. 1,5 TB. Linux clients mounting this dataset with nfsv3 TCP mounts have no problems. FreeBSD hosts mount this dataset with nfs3 TCP. Scenario 1) FreeBSD client mounts share. No problems. Scenario 2) FreeBSD client mounts share and relay it using nullfs mounts in VNET (if_bridge) Jails. Scenario 3) FreeBSD client mounts share twice. One mountpoint is located in his primary file system tree and the other one is inside the jail's sub tree. Accessing the share from inside the jail hangs the NFS mount after only a f= ew minutes of usage. In Scenario 3 only the mount inside the jails directory sub tree is blocked. The other one is working. Nevertheless df & friends on the main host is blo= cked too because they touch the Jails subtree o9n their traversal down. Same in Scenario 2. There the only NFS mount is blocked. There are no error messages at this point of time. No console text about st= ale NFS links etc. Nothing in the Jail's console file. umount of the mount blocks too but the blocking mount line vanished in the mount(8) output.=20 The only way to unfreeze is to stop the jail in which sub tree the NULLFS or NFS mount is located in. Stopping the jail immediately frees the hanging df and ls etc. processes. On the console I can read after umount -f / jail stop: newnfs: server '192.168.67.38' error: fileid changed. fsid 0:0: expected fi= leid 0x4, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE) I read the comments about this error message in /usr/src/sys/fs/nfsclient/nfs_clport.c. They are not matching this setup. T= here is no broken middleware... only cisco switches, a HP Proliant Gen9 FreeBSD = NFS Server and a few Oracle Sunfire-something x86_64 FreeBSD clients. The NFS Server is running FreeBSD 12.1 and the network is as simple as a 10 GBit/s network can be. Playing with enforce_statfs seems to have no effect. What's going wrong here? Some details: I use the jib script from /usr/src/share/exampes/jails after adding two lin= es to add MTU support to handle the Jumbo 10 Gig network. Firewall configurations are enabled in the kernel. To understand the problem here all is disabled or in pass through configurations. There are no other visible problems.=20 I add this files as attachment: adm01: total 240 -rw-r--r-- 1 arne ego 21833 Nov 24 18:24 NFS-Client-ADM001 -rw------- 1 arne ego 3254 Nov 24 00:07 NFS-Client-devfs.rules -rw-r----- 1 arne ego 17288 Nov 24 18:57 NFS-Client-df -rw-r--r-- 1 arne ego 19051 Nov 24 18:24 NFS-Client-dmesg.boot -rw-r--r-- 1 arne ego 636 Nov 24 14:41 NFS-Client-fstab -rw-r----- 1 arne ego 25965 Nov 24 18:24 NFS-Client-ifconfig-a -rw-r--r-- 1 arne ego 3600 Nov 24 18:33 NFS-Client-jail.conf -rw-r--r-- 1 arne ego 1239 Nov 23 23:31 NFS-Client-loader.conf -rw-r----- 1 arne ego 522 Nov 24 18:58 NFS-Client-nfsstat-m -rw-r----- 1 arne ego 76519 Nov 24 03:23 NFS-Client-pf.conf -rw-r--r-- 1 arne ego 25053 Nov 24 18:21 NFS-Client-rc.conf -rw-r--r-- 1 arne ego 1084 Nov 23 23:33 NFS-Client-sysctl.conf fs001: total 124 -rw-r--r-- 1 arne ego 21767 Nov 24 19:05 Fileserver-FS001 -rw-r--r-- 1 arne ego 27395 Nov 24 19:06 Fileserver-dmesg.boot -rw-r----- 1 arne ego 14781 Nov 24 19:03 Fileserver-ifconfig-a -rw-r--r-- 1 arne ego 1020 Nov 16 21:40 Fileserver-loader.conf -rw-r--r-- 1 arne ego 14356 Nov 24 19:04 Fileserver-rc.conf -rw-r--r-- 1 arne ego 1106 Nov 16 21:38 Fileserver-sysctl.conf -rw-r----- 1 arne ego 6398 Nov 24 18:14 Fileserver-zfs_properties jail: total 28 -rw-r----- 1 arne ego 1671 Nov 24 19:01 inside-jail-ifconfig-a -r-------- 1 arne ego 9 Nov 24 00:22 inside-jail-pf.conf -rw-r--r-- 1 arne ego 636 Nov 24 19:00 inside-jail-rc.conf -rw-r--r-- 1 arne ego 709 Nov 24 00:17 inside-jail-sysctl.conf Thanks in advance, .//. Arne --=20 You are receiving this mail because: You are the assignee for the bug.=