From owner-freebsd-hackers@freebsd.org Sat Sep 5 02:58:54 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 128D43DA2F7 for ; Sat, 5 Sep 2020 02:58:54 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670041.outbound.protection.outlook.com [40.107.67.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BjzlK0MKWz3VB4; Sat, 5 Sep 2020 02:58:52 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=f3vUF6SvIpyvgUXr+ekGiRR+EE4HCWojDZNtCOFjUvth8guAwCMyt1B965JTELReTzeCw1K+0f6ZerLHalM4w2PYaXN5n/ErKFoibRl7O5Q/Gsiu0xuBposQ0/zItJguu6a17fKEdQNAixRZUX6K6MOTASNmTrtGI96Olymldc/J7Ab488JSO6u+rPuS+iuB6vRAJla8YlvvFpeaoPlvf8scQPGsPVmgrOGANJkhrRnoWRgKVm7OvSJIkbFXUVlP6XRYiBfewhVCnz2Lbx7ZN8I7ibKPJh1BPfwdir75fxhq1o4KppwUJumjOKjds8jZMXrbtiR9+dPkirBJ+g9ybQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=83xbzpEXJ9t39zy1cPDP3mM8r7Ne2XkZ1BQv095UHJM=; b=l0oLhH0s2CXz236RdiwI2OU7jMvz3xubBLbCkwkwLGxWaliRQzkW6zSwAGw6ce7NMdKGPPV7Cnv1qxJ5eSaKTaKfn3QTVaGnFfmO7tdwo39zPVSqNDn78oyeYkJclYejjvfaEewywy5Ofw6Jck/jO8BVxninFHKnq8x5QYF+P4spQEF5OLyztbKpAZ9YEcDengLgYeRtzhy+nRZx5kVvZwOHJh0CnXqtTyG3yNWyjdvIF0tUf2r1QwKy+FKa07UCfLotb8/3Rtty1K0Dz0VvrB3FSNCsFK6hdvdyHm7fa3i33KT4oeHDPhsQ939+hXRxaRir7o8L1Mt1S3V7XCUIWw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=83xbzpEXJ9t39zy1cPDP3mM8r7Ne2XkZ1BQv095UHJM=; b=J+DNI4BIsTjDaYtfTK4YOKOc8QMV1PwgNpyc0Fu4te6Ldw9XidLULXeZbr873PQOuQWRZ89hR95ZpAw0XDUyewFakdBOGJfnQUawx3Zznlry9oD4/Annz3c1xmpnoFr6+LceFU5pLjUeQKghInkYkV2BwiI0Bd7Q4t56ubzR8gnB1JjYRPviDdvtJt6IiwafTl7xqtSGWT4NetzrZ9m5xuChplAbs68nXesmVM5fS0gD+CzWHLded4X/8kuDfC8FLOdpcWNAhIHFYU3u0YT3YAt92L20jELNMIxC9EYEMkgBPpQKraqA9Me21k4dsVmuWOz9GLLxvGcQaWuk7zlwTA== Received: from YTBPR01MB3966.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:24::27) by YTOPR0101MB0713.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:25::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.15; Sat, 5 Sep 2020 02:58:51 +0000 Received: from YTBPR01MB3966.CANPRD01.PROD.OUTLOOK.COM ([fe80::687f:d85a:a0a3:bd20]) by YTBPR01MB3966.CANPRD01.PROD.OUTLOOK.COM ([fe80::687f:d85a:a0a3:bd20%6]) with mapi id 15.20.3348.017; Sat, 5 Sep 2020 02:58:51 +0000 From: Rick Macklem To: Alan Somers , FreeBSD Hackers Subject: Re: panic!("docallb") in nfsrv_docallback Thread-Topic: panic!("docallb") in nfsrv_docallback Thread-Index: AQHWgxtspJUVGlzLXkCTJBO3gNcFCalZVDgu Date: Sat, 5 Sep 2020 02:58:51 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 29c26bb6-aaa4-4d06-beb7-08d851479e6f x-ms-traffictypediagnostic: YTOPR0101MB0713: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: tJ8Z3gtvj3iXilZCogArUsNKiV8ENW+r8eH5gnXQ11kuOqWb+vaMVrW1gG5TuDl+tp94QWx2anygceyUeCGL1B2+Zk2I+2j39p+ZdEhhYW6mhkC5tvH0zW04t3QfNry2OLnMuLxGu0erVk4GeEMjc6sZ7SagEGW89kwN8PPv2JlNxjFPlTZjAa3qKOOE2JtfjDwlWpgfg+Y4ZNj5cOqJYHuVpoD7nPw0EEJhh0lmARLam1vB0lCVFr/hdmTQPF8MXTYMgztrqEBWwQ4ypz/tI/Rtg4sS5YVjTa/Iy24ov3ZkFsrH20QPtaMENKbXryRKCtFp1kvSPSoUS+juvjS1F14MqmRpzJBxa7s1sBMhhXWoTtfsRmWctjcxJaF8XG72kWcncfdt7Ir7/IXlxFERQg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YTBPR01MB3966.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(136003)(346002)(366004)(39860400002)(396003)(376002)(83380400001)(9686003)(55016002)(478600001)(5660300002)(786003)(316002)(186003)(33656002)(91956017)(76116006)(966005)(64756008)(8936002)(66946007)(66446008)(110136005)(8676002)(86362001)(71200400001)(66476007)(66556008)(7696005)(450100002)(2906002)(6506007)(52536014); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: tXX8sb7vGajP/X6aH9uWBR5NfZh3n+8KVcs+PcFilJL9/1WyfYhBQgrXXFG2F4s7YB0YO48dci+R6Rmyw2JgJNdeQ2j8cPc93J7fkEbkyiOsgmgnH3vuJ54pkBj5LjySUIL+CgvF2R9UiYGoZ9mzBn1Xq2oyS41XO4a6ZyiaNXjP5maBETzkMvpSdFWY4nddKWiUKddPPv1HIoNpng7wloHIygjuRSaGFndF+/3i/uPI6cAenugilw3c4A1SUoUxl79oe44tF9tDLoxQmBIt4Vv/IBxuv7zNjhHrWz2En5tFLN8vlKsSDoF7TowJqrMJEZm10ruUxfRS+lUz9nWDIVESWwer2arimO1CTuG8vHbeote9S/6i/XdZzjB+batueuAPcWJaWQqv4ba9Vq4beDhEA+GmSwscThvM9w5lbI1N9nG3gcOkmHm6rCK1KT158OkO47/DV90vbMz/cs7w2d1ANtfGKoyA811MNiN1XnGkkifTx4ZyHkOgQx+tnlysfBVWktiuTTZ2mOGBnocZEF4XMi1RRFF8OT+zEQe/vE/HwK65pM4Qu2icKUh9qQt104vpZEuKgzzWNsLaL3ig4Mg59Ho2LM0vQhVZeviBpiNHE7+CcorDEQXhrO1VC/KQoWukztMyH7TTvAces+497MZF0DtwMpvhLsxRJ2uEh0oJRnLmSPJa/tnsKjkDclWf20wEgpnD2MT+KnaTfVszBQ== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YTBPR01MB3966.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 29c26bb6-aaa4-4d06-beb7-08d851479e6f X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Sep 2020 02:58:51.4687 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: JjVOivyvYLD4EGUl2lUaFPIzz+JN/czOqfhsqEzyAx5TW/YfrKaB1XsFCyTyeWWKiclwZt4ZenZOMBCfMKBQag== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB0713 X-Rspamd-Queue-Id: 4BjzlK0MKWz3VB4 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=J+DNI4BI; dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.41 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.46 / 15.00]; NEURAL_HAM_MEDIUM(-1.04)[-1.039]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.02)[-1.020]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.67.41:from]; SUBJECT_HAS_EXCLAIM(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.40)[-1.405]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-hackers]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.41:from] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Sep 2020 02:58:54 -0000 Alan Somers wrote:=0A= >I just saw this panic on a 12-stable machine. Unfortunately, I don't have= =0A= >a core dump, just a stack trace. It was serving NFS v4.0, with delegation= s=0A= >enabled. The clients were all Debian, with Linux 3.16.0.=0A= I will generically note that I believe the Linux NFS client developers most= ly=0A= test NFSv4.1, 4.2, so if the clients support NFSv4.1, it might be worth upg= rading?=0A= =0A= Also, delegations aren't enabled by default for a couple of reasons.=0A= 1 - For a long time, Linux only knew how to use read delegations and I felt= =0A= (and still feel) they are pretty useless.=0A= 2 - They are complex to get right.=0A= 3 - Although they should reduce the number of Open operations against the= =0A= server, I haven't observed dramatic performance improvements because= =0A= of them.=0A= =0A= >The proximal cause of the panic seems to be that the file had a write=0A= >delegation issued to an unconfirmed client. Root cause is harder to=0A= >determine. Did the kernel previously issue a delegation to an unconfirmed= =0A= >client? Or did the client somehow change to an unconfirmed state after th= e=0A= >delegation was issued, perhaps due to a race?=0A= I think the first case is more likely. Since client confirmation happens im= mediately=0A= for NFSv4.1 (the ExchangeID and Createsession must occur before anything=0A= else can happen), I wouldn;t be surprised if the Linux client tries to do a= n Open=0A= before the SetClientIDConfirm has completed for NFSv4.0.=0A= =0A= >It's hard to tell, but I don't see any checks for lc_flags &=0A= >LCL_NEEDSCONFIRM in nfsrv_openctrl (which issues the delegations), so I'm= =0A= >guessing that that's the problem.=0A= The server should definitely check for a confirmed ClientID during Open and= =0A= fail any Open attempt where that is not the case.=0A= --> I'll take a look at the code. I wrote it about 20years ago, but I can p= robably=0A= figure out how it works.;-)=0A= =0A= > If so, then the event trace would look=0A= >like this:=0A= >=0A= >1) Client Alice sends SETCLIENTID. The server creates a client state=0A= >structure=0A= > for her.=0A= >_) Client Alice should've sent SETCLIENTID_CONFIRM, but doesn't. Bad Alic= e!=0A= >2) Client Alice sends OPEN for some file, and is issued a write delegation= .=0A= > The server shouldn't have issued it, because Alice's client ID is=0A= > unconfirmed. Bad server!=0A= >3) Client Bob tries to do a GETATTR on that same file.=0A= >4) In nfsrv_checkgetattr, the kernel finds a write delegation for that fil= e,=0A= > owned by client Alice.=0A= >5) The kernel tries to send a NFSV4OP_CBGETATTR callback to Alice, to see= =0A= >if the=0A= > file's attributes have changed.=0A= >6) But Alice's client ID is unconfirmed. Oh no! Panic!=0A= >=0A= >Does this sound plausible? Should there be a check for LCL_NEEDSCONFIRM= =0A= >somewhere around line 3166 in nfs_nfsdstate.c? Grateful for any help.=0A= Yes, it does. I would have thought that I'd have checked for the unconfirme= d=0A= ClientID, but maybe not.=0A= =0A= It is also possible that the client somehow did a SetClientID after the Ope= n=0A= that issued the delegation, putting it back in "unconfirmed" state.=0A= It that was the case, maybe the panic(), intended to catch corrupted data= =0A= structures, was overkill.=0A= =0A= >-Alan=0A= >=0A= >P.S.: stack trace=0A= >=0A= >kdb_backtrace=0A= >vpanic=0A= >panic=0A= >nfsrv_docallback=0A= >nfsrv_checkgetattr=0A= nfsrv_checkgetattr() should probably check for the case of an unconfirmed= =0A= clientid and then return ignoring any delegations hanging off it instead= =0A= of attempting a callback.=0A= --> This would handle the case where the client did a SetClientID after the= =0A= Open that acquired the delegation, leaving the ClientID unconfirmed.= =0A= - The two RPCs doing SetClientID and SetClientIDConfirm are normally= =0A= done only upon mounting or when the client thinks it has lost the= =0A= ClientID due to a lease expiry, but there is also the case where it= is=0A= changing the callback address. (This could explain the SetClientID= =0A= happening after the Open that acquired the delegation.)=0A= --> Hint. Can you now see why NFSv4.1 chose to do things differently?=0A= =0A= nfsrvd_getattr=0A= nfsrvd_dorpc=0A= nfssvc_program=0A= svc_run_internal=0A= svc_thread_start=0A= fork_exit=0A= fork_trampoline=0A= =0A= Thanks for reporting it. I'll take a look, rick=0A= =0A= _______________________________________________=0A= freebsd-hackers@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-hackers=0A= To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"= =0A= =0A=