From owner-freebsd-questions@freebsd.org Fri May 22 18:43:08 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EA3462D9290 for ; Fri, 22 May 2020 18:43:08 +0000 (UTC) (envelope-from Norman.Gray@glasgow.ac.uk) Received: from plockton.cent.gla.ac.uk (plockton.cent.gla.ac.uk [130.209.16.75]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49TFjm05tSz3SKT for ; Fri, 22 May 2020 18:43:07 +0000 (UTC) (envelope-from Norman.Gray@glasgow.ac.uk) Received: from cas07.campus.gla.ac.uk ([130.209.14.164]) by plockton.cent.gla.ac.uk with esmtp (Exim 4.72) (envelope-from ) id 1jcCdN-0000aD-MP; Fri, 22 May 2020 19:43:05 +0100 Received: from CAS08.campus.gla.ac.uk (130.209.14.165) by cas07.campus.gla.ac.uk (130.209.14.164) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 19:43:05 +0100 Received: from GBR01-LO2-obe.outbound.protection.outlook.com (104.47.21.55) by CAS08.campus.gla.ac.uk (130.209.14.165) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Fri, 22 May 2020 19:43:05 +0100 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hw7XTam0e3keVQn72LEEx98tofE0xpI0N2vuhRlKGlE3zZiQKqhZC+a2YrJrOHhj3sF3KFC6EzHv8K9JENLq1cm8RIDDz5+cJHRcxDM+k4o14PWBXHGijLgWaDp+LsNu3WzP4Vx/B8PdJIccyqvj+Z1bDYqMF1h79mjD/kWuf6GesHHYvuoK9EI5CTjjBCMvkmt2oL9KGIQzaVjlGsGwfUhLdo0jmbRAXaRomrseoQ3m7guumrj/wRfdqY0EH2DrsoChaBNSbAnWchlcm3KjZjup+vyz4Y0X2dVMNAFblz9/4hdMTyUmyxvq+LdfsQvvaLX/aRDuvbIu7maKo50KKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=O+J75hyft3HCoatXQyoJybiSYeo25rqkFcxXVUkQIao=; b=HAWm5+PwgSaERoOgdIVgXlcLSFGo9ZWRQ6XgRZg/A+7spQRgc2nFPsA5c7Gci8aBj3D4bQji4QgPCzCZaL4NTBYa995xg2bNVabDTkQ5l+VMofNEedszKmSrqcMwz8xZWRILaEFSJ/tuo0p67RgMS2mdv+HBmeLeVYneaQTOCCoFssBfm8L4uhYtLO7TG+JcfUf4ufsMzekj5RvztIO+iPIr91AcTYK4/V+QSToDs4KFN0AMpo8OCLAtv8Fbyb/jf8Ik/X+51jgkTUl69G563Avx0udUCETDKjAAwZIOl4CiEVvR489ykMAIVrYSsunKPWiNIcyCnt1OLpW1Rfvn5g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=glasgow.ac.uk; dmarc=pass action=none header.from=glasgow.ac.uk; dkim=pass header.d=glasgow.ac.uk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gla.onmicrosoft.com; s=selector2-gla-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=O+J75hyft3HCoatXQyoJybiSYeo25rqkFcxXVUkQIao=; b=XFKE10vb88pJys/W/fM4UMhg5YBZRLkFIFmJATcC0AMMiDD9acpOR72d8ODY9zGmRV8S8TkmGFE27flcIjIh7BQPWKbVOn4RvANe9pG7w9DLHasCt/DjubwmQj67ambKLgqgxWvqbgujP/yrhqZCn1tk0dD4KTNmfo2jupZ5yL8= Received: from CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM (2603:10a6:401:8::19) by CWXP265MB0997.GBRP265.PROD.OUTLOOK.COM (2603:10a6:401:4a::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.23; Fri, 22 May 2020 18:43:04 +0000 Received: from CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM ([fe80::40d7:744b:8734:b8dd]) by CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM ([fe80::40d7:744b:8734:b8dd%6]) with mapi id 15.20.3021.027; Fri, 22 May 2020 18:43:04 +0000 From: "Norman Gray" To: Greg Veldman CC: FreeBSD Questions Subject: Re: Documentation and debugging for NFSv4 Date: Fri, 22 May 2020 19:43:02 +0100 X-Mailer: MailMate (1.13.1r5671) Message-ID: In-Reply-To: <20200522162937.GJ1068@aurora.gregv.net> References: <20200522162937.GJ1068@aurora.gregv.net> Content-Type: text/plain; format=flowed Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: LO2P265CA0441.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:e::21) To CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM (2603:10a6:401:8::19) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [130.209.33.134] (81.2.70.164) by LO2P265CA0441.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:e::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.23 via Frontend Transport; Fri, 22 May 2020 18:43:04 +0000 X-Mailer: MailMate (1.13.1r5671) X-Originating-IP: [81.2.70.164] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a34bc29d-a762-484c-f439-08d7fe7ff65b X-MS-TrafficTypeDiagnostic: CWXP265MB0997: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-Forefront-PRVS: 04111BAC64 X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: uhlQqrj+ysKpDhYgA25SyzQBtjr7XoCtGZPL1fY9qeXV0Wht9mwuxDCO8nfYgRSY5DTujUI26o+o5zIbkjsJk55T3i+M4kDe0EvIt2YrQWVcrSU3188bN1xHu1oRSq9SPENZY0roARg1z8B2UD4upST3mGvR6dnBcikPgXShDWnfgRL1afE/HTHZON89F6po1GUyiSN7N6QLmZLygu01pTEgc1fFJH3m9Q9gMsmELCLKnRZM3ZAM0MvLVJ1r4OM2AcHkBoqnnE1wEB6DYrKB5nO5XfygdEPXJNyBR6wEIU3VifYg1UHoIbLdG3oaMt4F1pAaCZThZoQftdFyfiP1T2jWaVTs/UejB0YGax5XyYcbiwY5IvfSXwfGZzcr8Me9WYHow4HgkPPD1ut/TQ3UuE0q4XdUBlAhulPzB9eyVZsck5PD5Ofp7e8I+IlZSNz2h2UjI5qQTuP628Zr3WywoBNySm6Ln23Je/6QvBLZaR0KZ11gfJZPW+OhS9D7qQR1 X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(136003)(366004)(376002)(396003)(39860400002)(346002)(956004)(786003)(53546011)(186003)(16576012)(66556008)(66946007)(478600001)(52116002)(5660300002)(86362001)(316002)(6706004)(16526019)(33656002)(26005)(66476007)(966005)(2906002)(6486002)(8936002)(36756003)(6916009)(8676002)(2616005)(4326008)(78286006); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: h0lIPqqARcM8OBH3APpV/xrlO0NKQnX6l6dKEF+ojwXrQvszFKyIec/uaxECTnCSmTOLRXpp9eUaEI5RfZfB9DbWvZ/GnFqYEIYWQU0ZYZPrI2L2q2sKyxTsYAcpMtbw1xRM0zxEw0OVr4GKGoZHUWA6yE9clkHNko5sRgRbArkyUy+YbxbzV2MkZgQz+r/WMCJCyHKyZIo1Oap5sWR816ow5b4eax8JFnyWkI7QVVNedR1HBl/0/Jgi89gEpRTa+aztfn9qkQbMY5y6mxaJPeDgIuuGR6ajS3E8D1nsLhRb5LRyGja8Oekyn7rnPu8krKKeDtxqDfTIf1ucGpwBRO79AC7AkQPl+3oEYPeqHDR9JlJLyvzY2FvgYvFIYmbw8RiL76sJ4FgserA4Ust87QRFFbMp+DLT0k6w5IU2gLUm2S/t9Zc7n2jUcw6oTZf8e19RpYNaCqE78ggZwwkKxKxv2tMWXVDnDVjjZGq7I4M= X-MS-Exchange-CrossTenant-Network-Message-Id: a34bc29d-a762-484c-f439-08d7fe7ff65b X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 May 2020 18:43:04.5596 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 6e725c29-763a-4f50-81f2-2e254f0133c8 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: o5i5LEULRxXbhdBvk7lI0PxTZYdjCz6NH+UMMyxhwOPMh2vm3OcqoJD7t/YdJhFrVMlxO1QfESzklwBYbox2v5074pkaSjinbeR16kpwIMg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CWXP265MB0997 X-OriginatorOrg: glasgow.ac.uk X-Rspamd-Queue-Id: 49TFjm05tSz3SKT X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gla.onmicrosoft.com header.s=selector2-gla-onmicrosoft-com header.b=XFKE10vb; dmarc=none; spf=none (mx1.freebsd.org: domain of Norman.Gray@glasgow.ac.uk has no SPF policy when checking 130.209.16.75) smtp.mailfrom=Norman.Gray@glasgow.ac.uk X-Spamd-Result: default: False [-1.89 / 15.00]; NEURAL_HAM_MEDIUM(-1.05)[-1.046]; R_DKIM_ALLOW(-0.20)[gla.onmicrosoft.com:s=selector2-gla-onmicrosoft-com]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[130.209.16.75:from]; R_MISSING_CHARSET(2.50)[]; NEURAL_HAM_LONG(-0.97)[-0.974]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[glasgow.ac.uk]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[130.209.16.75:from]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[gla.onmicrosoft.com:+]; R_SPF_NA(0.00)[no SPF record]; NEURAL_HAM_SHORT(-0.37)[-0.367]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:786, ipnet:130.209.0.0/16, country:GB]; RCVD_COUNT_SEVEN(0.00)[7]; MID_RHS_MATCH_FROM(0.00)[]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2020 18:43:09 -0000 Greg, hello. Thanks for your questions. I didn't describe the problem in detail in the first message, to keep it shorter. On 22 May 2020, at 17:29, Greg Veldman wrote: First things first: > FYI NFS problems in general usually come down to either firewall > settings or MTU mismatches. If there are any firewalls between > the client or server you may wish to temporarily disable them to > test (or add an allow all rule). Also check that the MTU is > the same for the whole path. There are no firewalls in between, and indeed on one pair of machines they're on the same subnet. I'd be surprised it's an MTU problem (though I'm not ruling anything = out) because I can do some mounts, so I _think_ that rules that out? > What symptoms are you seeing? Does it mount at all? If not > does it give you an error message? Does it mount but is it > not usable? Does NFSv3 work or is any version broken? The following is a description of a sequence of things more-or-less working as expected, followed by the now-clearer puzzle below '----'. server:/etc/exports /tank/home -maproot=3Dnobody -network=3D172.16.45.0/24 /tank/home/astro -maproot=3Dnobody -network=3D172.16.45.0/24 /tank/home/astro/norman -maproot=3Dnobody = -network=3D172.16.45.0/24 V4: /tank/home -network=3D172.16.45.0/24 There are numerous other lines in the exports file, and each of the above lines is almost duplicated, to refer to the networks for the machines `client1`, `client2` and `client3` below. I don't _particularly_ want /tank/home and /tank/home/astro exported, but I learn in exports(5) that if I'm exporting ZFS filesystems (which these are) then intermediate FSs must be exported as well. The V4 line 'turns on' the NFSv4 server, and marks /tank/home as the 'root' for this purpose. server:/etc/rc.conf mountd_enable=3D"YES" nfs_server_enable=3D"YES" nfsv4_server_enable=3D"YES" rpcbind_enable=3D"YES" server# rpcinfo -s program version(s) netid(s) service = owner 100000 2,3,4 local,udp6,tcp6,udp,tcp rpcbind = superuser 100005 3,1 tcp,udp,tcp6,udp6 mountd = superuser 100003 3,2 tcp6,tcp,udp6,udp nfs = superuser Ie, no version 4 for the 'nfs' service. On the client (CentOS 7.8, Linux automount version 5.0.7-109.el7): client1# mount -tnfs server:/astro/norman /mnt client1# mount | grep norman server:/astro/norman on /mnt type nfs4 = (rw,relatime,vers=3D4.1,rsize=3D131072,wsize=3D131072,namlen=3D255,hard,p= roto=3Dtcp,timeo=3D600,retrans=3D2,sec=3Dsys,clientaddr=3D172.20.45.184,l= ocal_lock=3Dnone,addr=3D130.209.45.61) (note 'absolute' path relative to NFS root), and norman@client1$ flock /mnt/try.lock echo hello hello That is, this is mounted as NFSv4, and file-locking works. Good. The same is true with an Ubuntu12 machine ('client2') also running = automount v5.0.6 (yes, Ubuntu 12; this work is part of a sequence of steps to let that machine out of the dungeon it's currently in for its own good). client# mount -tnfs server:/astro/norman /mnt The same thing works with the mounted filesystem given as `server:astro/norman` (ie, without the leading slash). This was the result of a hint on some other forum post (and the exports(5) page doesn't make it clear which one is correct). This also seems to work in this particular case, but an anomaly is that the mount command shows this as server:/astro/norman as if it were an absolute path. If, in contrast, we do client1# mount -tnfs server:/tank/home/astro/norman /mnt client1# mount | grep norman server:/tank/home/astro/norman on /mnt type nfs = (rw,relatime,vers=3D3,rsize=3D131072,wsize=3D131072,namlen=3D255,hard,pro= to=3Dtcp,timeo=3D600,retrans=3D2,sec=3Dsys,mountaddr=3D130.209.45.61,moun= tvers=3D3,mountport=3D780,mountproto=3Dudp,local_lock=3Dnone,addr=3D130.2= 09.45.61) and norman@client1$ flock /mnt/try.lock echo hello =2E..hangs for a 4 minutes (what timeout is this?). Also, this appears to be a NFS3 mount. I do not think I would have been able to predict that would happen. nfsv4(4) says "setting [rootdir] to anything other than ``/'' will result in clients being required to use different mount paths for NFSv4 than for NFS Version 2 or 3.", but it doesn't say why, or what the different paths might be, which is frustrating. So what's happening here is something like the client, in the first case, trying a v4 mount of /astro/norman with success, and then, in the second case, trying a v4 mount of /tank/home/astro/norman, which fails, so the client falls back to a v3 mount of that path (which doesn't work for me, because of the different flock semantics in NFSv4). Again, I don't know if this is a sane heuristic, or if the manpages are telling me I should have predicted that. So far so good. Client1 and client2 are two different Linux distros, on two different subnets (albeit with almost the same automount version, slightly surprisingly), and both can mount the server filesystem. For what it's worth, the same thing works when using an automount map appropriately configured via LDAP. I haven't tried any FreeBSD clients here: I'm serving almost exclusively CentOS clients (plus occasional legacy rogues, of course). ---- I now go to another CentOS 7.8 machine client3# mount -tnfs server:/astro/norman /mnt client3# mount|grep /mnt server:/astro/norman on /mnt type nfs4 = (rw,relatime,vers=3D4.1,rsize=3D131072,wsize=3D131072,namlen=3D255,hard,p= roto=3Dtcp,timeo=3D600,retrans=3D2,sec=3Dsys,clientaddr=3D130.209.202.212= ,local_lock=3Dnone,addr=3D130.209.45.61) client3# ls /mnt ls: reading directory /mnt: Input/output error Nothing! The same thing -- an apparently successful mount, and then an I/O error -- happens when I let the automounter do the work. If I try client3# mount -tnfs server:/tank/home/astro/norman /mnt mount.nfs: Stale file handle that is, the server path that should end up with a v3 mount, I get a separate NFS problem. So: My real problem is that at this point I have run out of techniques to diagnose this. The server looks... OK (modulo the 100003 3/4 version puzzle I've mentioned above), and some mounts work (including on another CentOS machine at the same OS version, which I've not knowingly configured significantly differently). There's nothing relevant in the logs. I have a suspicion that if I simply rebooted everything, I'd clear out some caches and this might start working as I expect (as it happens, client3 is scheduled for a reboot tonight anyway, so we'll find out), but that's never been a fully satisfactory solution even if it worked. I've always felt uncomfortable with the (perceived) lack of NFS diagnostics for this sort of situation, but in a good number of years, this is the first time it's really bitten me. > NFSv4 introduces a couple of new things that can also be issues. > The first is the v4 tree root, and the second is the concept > of ID mapping to avoid the need to sync UID/GID numbers. If > you can make it work with v3 but not v4, I'd start looking there. > Specifically make sure you have a "V4" line in /etc/exports on > the FreeBSD server, and make sure nfsuserd is running and > configured with the same domain name as in /etc/idmapd.conf on > the Linux side (and make sure rpc.idmapd is running on Linux). I haven't done anything clever with user mapping, because I don't _think_ I need to (do I?). All of the machines in question refer to the same LDAP directory for account and mount-map information. Is this something I should care about nonetheless, whether or not it's related to this mount problem? Thanks for reading this far. Best wishes, Norman -- = Norman Gray : http://www.astro.gla.ac.uk/users/norman/it/ Research IT Coordinator : School of Physics and Astronomy // My current template week for IT tasks is: Monday, Tuesday, and Friday