From owner-freebsd-stable@freebsd.org Sat May 22 00:56:10 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 74A9F63FCDB for ; Sat, 22 May 2021 00:56:10 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670062.outbound.protection.outlook.com [40.107.67.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Fn4m936lrz3hVf for ; Sat, 22 May 2021 00:56:09 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SXr+sHxfbzjg7HMG+pAqNg1AtzOx1d0Uw2RySioJszpLH7zp2J/O1u0reYi5sb6DqmlIwGE0vKFCf9OJaEEPCUDUWSt977a4PlBggvEnS4vREherSaNy9CDIJhYVdIPOyydN3QcycwuQpZ4B4hgmndVN2ZQ6x6yeVdUxbbnwvzvkHt2I6drSYaPnufhhfJ/zeDX5z5Q0DOaKbqvWNRh+ilgRHMq0pvxROCFH7ugNl/bbzTPwZrO7NS65AVn4iP0woK2rggraOu2qnJQ1mwr0le16v5bkx76czgyJA4C4dcNkXByT9FJEXHGEdMc1sDcYaN2mJrwg+e4PxDtUm6zaFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fYVwXEh/WADn3J03vAh4b3qN2TqO225iDocWs8UMqHw=; b=Fru5cpyTQ6x1ZV9xlkcEiWd4bIA1dzimAaQI73YXpVO49KL+hOBTQXl3xetjFjrHUurRNG8VZ94ZRSt6JlxszlFgai3I/wLIBLum+Ezp4m1Y2+nHRO47P/+oIVKwlA/svEhnlb0mGJUAaNmsxyEPVCcMiajVWRRnra7Yb0bxfZpZ4aG0q/neZNbE1K6DdgbtjMNYTKbSAhN1S0Ka4CweB9/oWIRq8So+njn+W/1rlEwTvl8ioeKKC3v75cYCGXLorNZ/WRJDj/uYj3/QJFdDzKFjebEE/Hbr0LPD1HVYAxNn3oTSc2WzX84g/aIZ5RYY0TwixCkdQcfce3Q3+/LIgg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fYVwXEh/WADn3J03vAh4b3qN2TqO225iDocWs8UMqHw=; b=VE4Kp+8QgO7TW5lxFzUU+hsor5b8NQowyzdtlgRrqo3r0dt1NPApnK6Qdvz/yyCG2xxceKmt6w+fB3CwtZaIpp1OA+TYjOs3lHxm8l9uVU6v4DLPY2uZ6Hn3x6B68XeJzr02ZoSP93lUBWDAUrSTOlLRFu2RZIhYlHNECSsCkW/iWssONWfwP0GQxRQcTb4E/8oQfjCTJRQtSoIlFLoObKnUztM3hcVJ1uWbypRFw2DuDHiTyj2w+V85IwesAAysjCsQM2kkVW4S6L6u52FCwBSoBjrPREdX9BF+ig6tQfoTtEaEoULGjJ6pG3vmsUXVgcI8l69MnnU23WE1DpDiDQ== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR01MB2485.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:42::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4129.28; Sat, 22 May 2021 00:56:07 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::1c05:585a:132a:f08e%4]) with mapi id 15.20.4129.035; Sat, 22 May 2021 00:56:07 +0000 From: Rick Macklem To: Mark Millard CC: FreeBSD-STABLE Mailing List Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context) Thread-Topic: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context) Thread-Index: AQHXTc2kjmx42SEu2k+yBbZoJcd6XartEcEhgAAMYPOAABsogIAAEDYAgAARtACAAAf8xoAADUaAgAClJlqAAEnAAIAARWni Date: Sat, 22 May 2021 00:56:07 +0000 Message-ID: References: <623369D9-5EE5-4FEF-B9AD-56499E8F1C09.ref@yahoo.com> <623369D9-5EE5-4FEF-B9AD-56499E8F1C09@yahoo.com> <04D7264A-206B-4281-B452-779B01EA3327@yahoo.com> <34E915B3-30DF-408C-A931-C39188F3EB0F@yahoo.com> , <508C3B05-79E5-49ED-8032-DA7DF249E154@yahoo.com> In-Reply-To: <508C3B05-79E5-49ED-8032-DA7DF249E154@yahoo.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 07a4fd84-d359-462e-1e31-08d91cbc6212 x-ms-traffictypediagnostic: YQXPR01MB2485: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: hbTtBlzEhTbZyK+CL1tFAfDdXKQx6YbnO+L2YSSQAz7EHvTpO+1Z35dXqjhWnzEX+CGAvIrcAnS/3QRTEkwJ5ZT6aJOhKzefVjn7IZcvBqjmVTyPN4FY72Gi2dVchFJwTG05hysn3tQJnuXx21CF1Ljt3I+nejxCGXJfMfMYxp0w3Z3NaXwPdfNqBaFK2WFOqxMeUZTNNA/7nX6WyLnTTrQHBedtRKq373OBErDR9kiKnxXGaKv6q/3UqKhFLZI3fxHqZPedssmFOS2LyDQFiz4rcmRy6+XNxzGt8wxLEOGAw8AxNPpOyn4x/zVGM77rid9f0q+3IgF7Kk2VwCLbSIrNyYo6UmTTRape46whC6qgyWhkgTcAJp32BMRjO9l1yDHglMdvGBnVGIwHPTR7fG5C8jbDFpiBCieU5TrDtAYq3+rLI86KofzcTyrfTifVZMQNax93+ZUrTCjaEoiqn4iItb1BF4bh1p/QxYSFk30mwYILgSs6iG2BCD42hMvGDoIVN+vQ3cbHKI3IzXov9eeuywstyhGo5KG17OtM3CML8D0KcIntHXBTlMkrq1B7Apj36XRKlm2IPNkK/s0NzMZlJgkhpa8nleqMtB6MXpaKpdGN2lRk8hW+TN8lQc7EKrd7G+o5BYKfw14qDqE2iAQ8n1PzrFHRrVtMzo36F4j3MINUSjyzrMux5/3Pipzpbat2PLkQ4l1Ge1+r9QipuNt7GMPS6XKJkACMPZyZ164= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(346002)(396003)(136003)(366004)(39860400002)(376002)(9686003)(38100700002)(55016002)(33656002)(66476007)(122000001)(64756008)(66446008)(66556008)(4326008)(71200400001)(8936002)(8676002)(86362001)(66946007)(5660300002)(76116006)(7696005)(966005)(786003)(478600001)(6506007)(6916009)(316002)(186003)(52536014)(2906002)(83380400001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?qU+TFz2cspxcVST7p0/NLTCy78fwkpqVvR4Yub2eICXDpqjaaai+hOcy1q?= =?iso-8859-1?Q?yW01BD04IIA5R8aBF7Hd8f/cGf7bvSLpTX/+neKxelWD+PzoLg8vZWpU05?= =?iso-8859-1?Q?bGKhNRFhLQH28RomriWkY/QnYK9WTq/T39BAlEnqtjOTLdOa0HUKCb27GH?= =?iso-8859-1?Q?OpwFDF018r8Moc7+CQEsPoOnXshT/NZYZDK2vgH6BlDkBRBcB3tcR95pOR?= =?iso-8859-1?Q?gaDpNctnrzu2bvpUYNM7qFPvP+5kW3UEV2DwPioOw93yuBtQR5P/B92TB4?= =?iso-8859-1?Q?9NHtVxs8Adar9E7PNeulnW4+RTjPI+m0oUJUI95RkSLWh9/MFmJbTAxbFT?= =?iso-8859-1?Q?w/4DMaGdenhCtHIyx8oQE11oXB42zW2pKOkhAS5v0+bteFeZ8gV7xOJJW9?= =?iso-8859-1?Q?yM/v6m7bixyeIE3K6hBl/Z53wEWAID7TBh+6nqo3Ja4gz5+FMLwYtVF7Aa?= =?iso-8859-1?Q?+n8sLi49X43M0bR8S8udHjAojnqyqaqlGo5Zdv+XtNa4O+C1cH+VKN0WpA?= =?iso-8859-1?Q?S5X04mrKteMYOWGqyIualqJ61TeIza9tppwzZVGEIYoERDwMdz9BVy1k13?= =?iso-8859-1?Q?lgXHuNq9KOAh4NSjUSd0WH6DuunuqNGW5qTo2IFxHus6o9k4PrwtMH15GZ?= =?iso-8859-1?Q?1rdapu1fff5h/ZlaRLjPW34niq8M3WRNLhHByg8Ub8d8lDjnnm6O8Zjz57?= =?iso-8859-1?Q?xQOJXCclEbRdq2jYO0mtzq5zynWPdPIJKT89gxa2y5KUh9VY7WUraxU6Bx?= =?iso-8859-1?Q?f5QwVORkt7cuHwaeaXAmbx0LlGsNwSBcaK0pH7KXb5HC0hfjiVvP7TlIhr?= =?iso-8859-1?Q?hG2wahg5LC5E9qpIkkDu1L1/6Nmgh+OlpLBBg8V7IFTc24wVMmMhhTD3Og?= =?iso-8859-1?Q?bZROiK86S8/pI4eQrUdVOJ3yWCgMz9Y0BggXmJlYC37RTwWXCxoM4WB7gl?= =?iso-8859-1?Q?UjacjVLpdn9GsLd7JZuY/JIWoMeG9R0rJrnTiOHIfS7EQKZKvpARqY4YuA?= =?iso-8859-1?Q?daeLaU3eJEkniUfuVNLjaukKfZ0T5d4H2XtZ2BdkiwTo5833aF1IYC5G9B?= =?iso-8859-1?Q?d6S+K36pOmY1KDcbubCKV9UUcLJfWu2tSIov/FM1xsKAJfGksdKV3wGq7h?= =?iso-8859-1?Q?FNR7VCYuPhXbJ0ElgFpjd2auhfJVBlIrE33xFBYua8UUFc5z+qjdcPgkla?= =?iso-8859-1?Q?W+m7LXwE+WQMud39uoqxY4yyopi1UCBoAv2dccQvB2VWDhce+L3bhHt5Tj?= =?iso-8859-1?Q?RKIQUaG2VXwUaMHmxti3HywpCxGAXbSDGFmWcQ8+WshxJ69kXQgQ0etsrw?= =?iso-8859-1?Q?CyMJ+Df3a3a1ZRjUdjJy+HGJXs3hcnDR6VVb9zg/LKa8rDmFg7BruqEQzZ?= =?iso-8859-1?Q?rZD/82R7TpnCIYtV0OftY1huWBgxv79bWc6IE7FV6yVlIyH9+rjMg5EPuT?= =?iso-8859-1?Q?tSLqG/n4O6wMMQmw?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 07a4fd84-d359-462e-1e31-08d91cbc6212 X-MS-Exchange-CrossTenant-originalarrivaltime: 22 May 2021 00:56:07.4169 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: MXukGmuHWijV8hgonljzGihcMqaKv9IZS4RaFGzDznx3lHfoyRzl5MJXOUlV4zoornGuFdrYfxRTcZIhTcKHKQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB2485 X-Rspamd-Queue-Id: 4Fn4m936lrz3hVf X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=VE4Kp+8Q; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.62 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.10 / 15.00]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[yahoo.com]; RCVD_IN_DNSWL_LOW(-0.10)[40.107.67.62:from]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.67.62:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; SPAMHAUS_ZRD(0.00)[40.107.67.62:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.62:from]; MAILMAN_DEST(0.00)[freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 May 2021 00:56:10 -0000 Mark Millard wrote:=0A= [stuff snipped]=0A= >Well, why is it that ls -R, find, and diff -r all get file=0A= >name problems via genet0 but diff -r gets no problems=0A= >comparing the content of files that it does match up (the=0A= >vast majority)? Any clue how could the problems possibly=0A= >be unique to the handling of file names/paths? Does it=0A= >suggest anything else to look into for getting some more=0A= >potentially useful evidence?=0A= Well, all I can do is describe the most common TSO related=0A= failure:=0A= - When a read RPC reply (including NFS/RPC/TCP/IP headers)=0A= is slightly less than 64K bytes (many TSO implementations are=0A= limited to 64K or 32 discontiguous segments, think 32 2K=0A= mbuf clusters), the driver decides it is ok, but when the MAC=0A= header is added it exceeds what the hardware can handle correctly...=0A= --> This will happen when reading a regular file that is slightly less=0A= than a multiple of 64K in size.=0A= or=0A= --> This will happen when reading just about any large directory,=0A= since the directory reply for a 64K request is converted to Sun XDR= =0A= format and clipped at the last full directory entry that will fit wit= hin 64K.=0A= For ports, where most files are small, I think you can tell which is more= =0A= likely to happen.=0A= --> If TSO is disabled, I have no idea how this might matter, but??=0A= =0A= >I'll note that netstat -I ue0 -d and netstat -I genet0 -d=0A= >do not report changes in Ierrs or Idrop in a before vs.=0A= >after failures comparison. (There may be better figures=0A= >to look at for all I know.)=0A= >=0A= >I tried "ifconfig genet0 -rxcsum -rxcsum -rxcsum6 -txcsum6"=0A= >and got no obvious change in behavior.=0A= All we know is that the data is getting corrupted somehow.=0A= =0A= NFS traffic looks very different than typical TCP traffic. It is=0A= mostly small messages travelling in both directions concurrently,=0A= with some large messages thrown in the mix.=0A= All I'm saying is that, testing a net interface with something like=0A= bulk data transfer in one direction doesn't verify it works for NFS=0A= traffic.=0A= =0A= Also, the large RPC messages are a chain of about 33 mbufs of=0A= various lengths, including a mix of partial clusters and regular=0A= data mbufs, whereas a bulk send on a socket will typically=0A= result in an mbuf chain of a lot of full 2K clusters.=0A= --> As such, NFS can be good at tickling subtle bugs it the=0A= net driver related to mbuf handling.=0A= =0A= rick=0A= =0A= > W.r.t. reverting r367492...the patch to replace r367492 was just=0A= > committed to "main" by rscheff@ with a two week MFC, so it=0A= > should be in stable/13 soon. Not sure if an errata can be done=0A= > for it for releng13.0?=0A= =0A= That update is reported to be causing "rack" related panics:=0A= =0A= https://lists.freebsd.org/pipermail/dev-commits-src-main/2021-May/004440.ht= ml=0A= =0A= reports (via links):=0A= =0A= panic: _mtx_lock_sleep: recursed on non-recursive mutex so_snd @ /syzkaller= /managers/i386/kernel/sys/modules/tcp/rack/../../../netinet/tcp_stacks/rack= .c:10632=0A= =0A= Still, I have a non-debug update to main building and will=0A= likely do a debug build as well. llvm is rebuilding, so=0A= the builds will take a notable time.=0A= =0A= > Thanks for isolating this, rick=0A= > ps: Co-incidentally, I've been thinking of buying an RBPi4 as a toy.=0A= =0A= I'll warn that the primary "small arm" development/support=0A= folk(s) do not work on the RPi*'s these days, beyond=0A= committing what others provide and the like.=0A= =0A= =0A= =0A= =0A= =3D=3D=3D=0A= Mark Millard=0A= marklmi at yahoo.com=0A= ( dsl-only.net went=0A= away in early 2018-Mar)=0A= =0A= =0A=