From owner-freebsd-questions@freebsd.org Sat May 23 16:22:51 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 38C2B2CF63D for ; Sat, 23 May 2020 16:22:51 +0000 (UTC) (envelope-from Norman.Gray@glasgow.ac.uk) Received: from plockton.cent.gla.ac.uk (plockton.cent.gla.ac.uk [130.209.16.75]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49TpYQ2GMTz4G3G for ; Sat, 23 May 2020 16:22:49 +0000 (UTC) (envelope-from Norman.Gray@glasgow.ac.uk) Received: from cas07.campus.gla.ac.uk ([130.209.14.164]) by plockton.cent.gla.ac.uk with esmtp (Exim 4.72) (envelope-from ) id 1jcWv9-00056Q-SG; Sat, 23 May 2020 17:22:47 +0100 Received: from cas07.campus.gla.ac.uk (130.209.14.164) by cas07.campus.gla.ac.uk (130.209.14.164) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Sat, 23 May 2020 17:22:47 +0100 Received: from GBR01-LO2-obe.outbound.protection.outlook.com (104.47.21.55) by cas07.campus.gla.ac.uk (130.209.14.164) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Sat, 23 May 2020 17:22:47 +0100 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CGic174I87HSuJ1CdcgquFOFKMxsk4ZXE2FTm4mT6d3sKt8qvlrDZg6sXeY2kqAcmqxOlHnNhl6L22IagyBWpDNIg7Ump3jVDQSL3hI3KrP3hGx8GlIZZ5lhqzxddoDaGOCAJ2z0U+1XAPKtGilvwuky/QMDpE+ZRYZhTk8kusoB3/ftzCj03WySbXmN0bEu/AD5PLme78lvlTNzYVABw4CJ0aTp9cXdDQjF7yrZh61WDiHHwt9TeyjTV7K+jjnb+KB8/OsBtPlUneWB1HETHH6GwzqGm103nBXFORdvR9cj4fvtMftcVzamTo+XGb+I52seiBJBIma3cA1P88kCxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uTHOu9uMkgiuLoZWoelhrTLCh695Pp3jDPilwwrncsY=; b=Q3hXM6Kev1B7YYi5yrLCcGVPJBQVfDMDNG6PpZEKADJIV3kxFEFHpixIOS8k8u7lPe6tT/A4kztn5+eaS73SR4JajA0kdBcHtyzwPq2ddqxN5HV8OQUcpkcYlfWcm/yM3KNhuYpHKkhdPin1Qpxj/NsNjact4mQ3ae6Q1lbQlfccduPjWL7NBmFcYP5udUZM/3cGrN5fQ509n6rWLyJB6v2BQJ/yZBzZr286DTvw+P6lW08pELL3iLgwNLI7vGiwGyD4KnDWxPUUv1+qca5nr4U6AZ/i/25RZ7THRajMIQYxmxu+33iAza++HEiecJFi0wzxxYlIhXdUa3xC3bLtdw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=glasgow.ac.uk; dmarc=pass action=none header.from=glasgow.ac.uk; dkim=pass header.d=glasgow.ac.uk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gla.onmicrosoft.com; s=selector2-gla-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uTHOu9uMkgiuLoZWoelhrTLCh695Pp3jDPilwwrncsY=; b=UqzXoFRxiIrXEZRM5G6nLf3w8+RB0xh3jB0FJkYHFIsgi7786GY0BvxpaQ2PKTpo/eOALNJRp2T7+6p8wBYlN8HE0+0rA+fAgftiX870oh4Vbk4mjhaagZ+qU4m2CiYawYv31Z1J5hjD+hPv7VpkKRIdN9aPcp33NdlGh9m9Hjg= Received: from CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM (2603:10a6:401:8::19) by CWXP265MB0903.GBRP265.PROD.OUTLOOK.COM (2603:10a6:401:3d::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.23; Sat, 23 May 2020 16:22:46 +0000 Received: from CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM ([fe80::40d7:744b:8734:b8dd]) by CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM ([fe80::40d7:744b:8734:b8dd%6]) with mapi id 15.20.3021.027; Sat, 23 May 2020 16:22:46 +0000 From: "Norman Gray" To: Doug McIntyre , Remy Zandwijk CC: FreeBSD Questions Subject: Re: Documentation and debugging for NFSv4 Date: Sat, 23 May 2020 17:22:44 +0100 X-Mailer: MailMate (1.13.1r5671) Message-ID: <057FFF2E-CB29-4A74-8F27-8ADF0AE4C202@glasgow.ac.uk> In-Reply-To: <20200522182635.GA4515@geeks.org> References: <20200522182635.GA4515@geeks.org> Content-Type: text/plain; format=flowed X-ClientProxiedBy: LO2P123CA0047.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600::35) To CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM (2603:10a6:401:8::19) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [130.209.33.135] (81.2.70.164) by LO2P123CA0047.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.24 via Frontend Transport; Sat, 23 May 2020 16:22:45 +0000 X-Mailer: MailMate (1.13.1r5671) X-Originating-IP: [81.2.70.164] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 223e6164-8b5d-4e79-d79d-08d7ff3586f5 X-MS-TrafficTypeDiagnostic: CWXP265MB0903: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-Forefront-PRVS: 0412A98A59 X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: JJkWFU3knzOo/OZO7f8y2n0hPTKvwUC55JauYKdTo7boRRg6AVpNXX3qmuhlQsFrGDcmH5whJIu5veZ4yK1OyAx18YdkTkgWXYgOJuXakQAC+1REPmuBAzKhY8vYRmpjdGNDpugF76G/sqI7eVajuPs1XYfb9nm+xm8E014Z0NzAQTaYCSDTqns1Q6rdm5lLYT18ZvtBijK76HQcJZLYrYP7Iwlwdiel7kE5VIOipj4saryX/SSFeg3pwK+9mCRu4dvoYylKR7hJ5eO+B2kmXzqKXN2WHbWz/ucsscObjF9AHCYhFgzVidl31nlCCVdPyQQgGv22dUM6G00eX639Q6jJ2XwQ+o9twSgcay6wcp/Cen0qGforGEsn+K1u4rfY9cqRfOxKO1DRajgRI0xxXGqC97oaJ6rcnZFBjz8cpTf/1RuF381GkHoHpMGmoWSe7kXeZLAoxeKPYqhqQFQtZMoTbPWkYgWRquKsrxqBA0fNtq6cjbphawFcoV8/rnIi X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CWXP265MB0149.GBRP265.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(346002)(136003)(396003)(366004)(39860400002)(376002)(26005)(186003)(52116002)(2906002)(6486002)(66476007)(66556008)(66946007)(966005)(4326008)(6706004)(16526019)(33656002)(86362001)(478600001)(956004)(2616005)(110136005)(53546011)(786003)(16576012)(316002)(5660300002)(8676002)(8936002)(36756003)(78286006); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: Hws+ydqbeIVGBaSZ7IpaAE4Rx17V0S0YBLdWit0fgJS+C2QMHrGScJu0kkowBOGfmbdNJWDXPOvhfhz20ykpEOEQ3acJMfMwMpSs/0pGG8mZPzSFc0iq4Vn8EvnvxzCiv5z7VE//y7fuoF7rFG7vc9gQBZqR8HpdjXfttjinh3JJPEAoG2EEz9FqYopNefPL/+RrO/15hQuJdXtIPdMdUZrSJnfpJ8UKcSUEnpbCrlaHba9OaA8VgKeeEOtq2un3Fry7NpC+VgGKnIfKqEH4dckssLxd4UBLBhtSnt/AGQqjavIQ6sCCo1R3S6mLi+UhxhGivLtPO21NujAKDk4JRi/r7NG5v3Kma9iOcL6hbq5LcQ+gWQ3SkaTWY09c/P/xFUg5e0UYDQpCdfit+QSUvTvIjuTVG2CxfAGV7sJ9URPdd/e/bHkFX4xYPR4uLHLvDS9VM5ij9qVWz5/ucjJ35I2QO7JNNd+bJAfNU4egb3w= X-MS-Exchange-CrossTenant-Network-Message-Id: 223e6164-8b5d-4e79-d79d-08d7ff3586f5 X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 May 2020 16:22:46.1830 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 6e725c29-763a-4f50-81f2-2e254f0133c8 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8S8vn+QxwHgdRJi07O+pLFgaPt3f1EqQwtYIT5PzqSXPNPoZqW/o50sFqKnIo1KJXsNQML+eM9lpkkaxpl2LaFa/WGOq0hSXA5Pk9ZDSAF4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CWXP265MB0903 X-OriginatorOrg: glasgow.ac.uk X-Rspamd-Queue-Id: 49TpYQ2GMTz4G3G X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gla.onmicrosoft.com header.s=selector2-gla-onmicrosoft-com header.b=UqzXoFRx; dmarc=none; spf=none (mx1.freebsd.org: domain of Norman.Gray@glasgow.ac.uk has no SPF policy when checking 130.209.16.75) smtp.mailfrom=Norman.Gray@glasgow.ac.uk X-Spamd-Result: default: False [-4.11 / 15.00]; NEURAL_HAM_MEDIUM(-1.01)[-1.010]; R_DKIM_ALLOW(-0.20)[gla.onmicrosoft.com:s=selector2-gla-onmicrosoft-com]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; RWL_MAILSPIKE_GOOD(0.00)[130.209.16.75:from]; NEURAL_HAM_LONG(-0.98)[-0.982]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[glasgow.ac.uk]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[130.209.16.75:from]; DKIM_TRACE(0.00)[gla.onmicrosoft.com:+]; NEURAL_HAM_SHORT(-0.12)[-0.115]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:786, ipnet:130.209.0.0/16, country:GB]; RCVD_COUNT_SEVEN(0.00)[7]; MID_RHS_MATCH_FROM(0.00)[]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 May 2020 16:22:51 -0000 Doug and Remy, hello. Thanks for your additional observations. On 22 May 2020, at 19:26, Doug McIntyre wrote: > On Fri, May 22, 2020 at 03:15:01PM +0100, Norman Gray wrote: >> I'm having difficulty finding consistent documentation and debugging >> tools >> for NFSv4. Is there some handbook-like source that I'm missing? Or >> some >> layer of documentation for configuration or debugging that I've >> failed to >> find? > > I think in general, that NFSv4 is not widely deployed outside of > hetrogenous linux environments. Given the state of things, I'd imagine > it is downgraded to NFSv3 more often than not in other use cases of > mixed > OSes. That doesn't seem to be the case with me. Even an Ubuntu12 client seems happy to make an NFSv4 mount from this server (though I think it's 4.0), and I have a CentOS 7.8 machine similarly happy with 4.1. But it's another CentOS 7.8 client which refuses to make the connection. (there's an aha below...) It also turns out that a FreeBSD 11.3 client can't mount this unless -overs=4 is explicitly provided on the mount command (this is actually explained in the mount_nfs(8) manpage (!), which says that the default strategy is to try 3 then 2). Succeeding: ubuntu12# mount -tnfs server:/astro/home /mnt centos78@a# mount -tnfs server:/astro/home /mnt freebsd113# mount -tnfs -overs=4 server:/astro/norman /mnt Aha... Failing: centos78@b and Ubuntu14@b.... aha. I have FINALLY found some consistency to the machines which fail: they're all in a different DNS (sub)domain to the server, though in the same netblocks as the machines which succeed. Specifically, they fail during the ls: the client sends a PUTFH and a READDIR opcode, and the PUTFH succeeds but the READDIR fails, with a NFS4ERR_NOFILEHANDLE, which seems to suggest that the FH that the PUTFH sent wasn't saved (if I'm understanding RFC 3530 for NFSv4.0 and RFC 5661 for v4.1, correctly). In the case of the machines which succeed, in subdomain @a, the corresponding NFS request looks pretty much identical, but the response is successful. (An odd thing is that in the _successful_ cases, Wireshark shows a lot of 'TCP ACKed unseen segment' warnings, but I can't see how this might be relevant) This is the only consistency I can see, but I can't see how this is relevant to anything. * NFS works at the TCP layer, after resolution of DNS names * There are no domain names that I can see in the tcpdump traffic * The only mentions of domains in RFC 5661 are irrelevant to this. The domain names in the context of owners and groups are not, I think, relevant. >> Normally some combination of netstat and tcpdump would make some >> headway, >> but SunRPC is blacker magic than that. > > NFSv4 is a big change, most implementations I've seen operate over TCP > instead of UDP > whereas TCP was optional in v2 and v3. As I read RFC 3530, NFSv4 is TCP-only (well, TCP and SCTP), and doesn't use UDP at all. > NFSv4 doesn't need rpc portmapper, nor > other helper daemons. The IDmapper is a big change as well, no more > UID passed > through, but all UIDs have to be mapped back and forth on both sides. Also from Remy: > Is nfsuserd running? According to the man page, it is needed for NFSv4 > to work properly. I think the string-based user and group information is strictly optional, though usual/recommended. nfsv4(4) indicates that that the wire protocol can have strings or numbers-containing-strings (and cf RFC 5661 Sect.5.9). At any rate, I see the _same_ behaviour both with and without nfsuserd running on the server. When it's running, I see domain names in the responses to GETATTR requests, but nowhere in the requests that are failing. This is however the best clue so far, since I can see that `nfsidmap -d` produces different results on the two sets of client machines. I wonder if this worked by accident -- due to a default configuration -- without nfsuserd before. It appears that I'll need to learn more about what role that has in the protocol. I wonder if the relevant id is somehow encoded into the (I thought opaque) filehandles that are passed back and forth. > Make sure you use V4 definitions in /etc/exports. From what I > remember even connecting as a client needed 'V4: /' in there to > connect right to a linux NFSv4 server, but I could be misremembering. That's right. The presence of the 'V4' in the /etc/exports appears to be what enables NFSv4 service (exports(5) seems to vaguely suggest this without being explicit). Thanks for your thoughts; any others most welcome. Best wishes, Norman -- Norman Gray : http://www.astro.gla.ac.uk/users/norman/it/ Research IT Coordinator : School of Physics and Astronomy // My current template week for IT tasks is: Monday, Tuesday, and Friday