From nobody Sat Mar 2 14:52:58 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tn7Hf2qqdz5CNBc for ; Sat, 2 Mar 2024 14:53:18 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tn7Hd6nZmz4vdB; Sat, 2 Mar 2024 14:53:17 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x1032.google.com with SMTP id 98e67ed59e1d1-29a2545a1e7so2285803a91.2; Sat, 02 Mar 2024 06:53:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709391196; x=1709995996; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=iwILxCpl00VkleF46SMQTU7NI1hQzhRkNzAq2ZC8OCc=; b=lxFmtXCHVZvxE+h9WO2Iw2zEUDMM+LXT/K4GYLEG2fyTPEpa6GVzmbzE1QXBUPU5+7 +6ZnkK9ql3oXlyoGZQuaRLrRk8dwYS2vpwVaSPtDqo5snAtcQhoVZY6324kaLzDVFBhW Axp1ROliUg0oLiwkDvs2epnadL4+U23lC9tt0Re1PLkBKOzkgRVJO8lfjtNEVwZnL0mX J+BzoiQ4zCCvRRr3QrQq07JsP+zc2aGRD8GQ1mY6UZ/uUCULIHM2jg2Vxy+sWq8FsyE1 qG4iwJLLXHGKEvMoJxI/bv+taRvrGLBi2JLUvu+J42uZVThyLLtaLyI/TlAeNyWizqdn Erfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709391196; x=1709995996; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iwILxCpl00VkleF46SMQTU7NI1hQzhRkNzAq2ZC8OCc=; b=SP6b7+xX/29XBYVjEE7IBkFipzNjzn3LAsbsh9McA9E6hwwbo6BNiKc+BM8gP91Fiy P7YRO9j6thVB6IzyW1tUnsZ9dXGExZp6TFoXnij0FvvOOhW7PMbLHk1YFKq2TpsVw9yL e+TkLZRxO37mqFhWEuKRly0ww4CCRGpogzWX5aVEQqTGubCjao3Ng/LD7U/X8l2NnfWP QKOL0vJgV2JYfBQQOLvbJIwkb9whGhMDX7i4KdEd/EbIAbm6XD3WPPWcpejSMqADyxW+ 7fFCijc3Mv62kUVi2xRcgy5iiVN+0+7O5hfnBAMJ4oaaRG1hrIaUs4dnh0zDCoaajkrB EIZA== X-Forwarded-Encrypted: i=1; AJvYcCVodbznGNy69D7FIHUnBmTeXeQEY2g5w9mY4YaYarbGt00sbkG+nXQHNv3QBtg9aw/nIqsJYGzXUV14DvWsg+MU4UxZRuWGLo8qV5TxgZ3J/FIOMAJGM7yC X-Gm-Message-State: AOJu0Yzr2jyRpOPWhTnG/c34OYMedwm+SotNvTWeV6EFdEYnFfPM17BU 5wZ4funzRXRakfiyzgRgbu0MddkDPjqCd3YenHXU2SOx8hjrRcU8q3taOTR1yF8ZeHT7n41Jt/Q 3vpJ8ZXPkK7+QTaPwpHdxomz5vVlRImk= X-Google-Smtp-Source: AGHT+IFT2eGovaUyeUCDHdgTmeLosPF3mQae1OBTtJxTsdedjK/Axv/ryRxH0xYSXjKniYE9aQdpd0WvpASPWs7GDyk= X-Received: by 2002:a17:90a:4211:b0:29a:a1cd:da65 with SMTP id o17-20020a17090a421100b0029aa1cdda65mr3806220pjg.43.1709391195794; Sat, 02 Mar 2024 06:53:15 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <1020651467.1592.1709280020993@localhost> In-Reply-To: From: Rick Macklem Date: Sat, 2 Mar 2024 06:52:58 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Konstantin Belousov Cc: Ronald Klop , Garrett Wollman , stable@freebsd.org, rmacklem@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4Tn7Hd6nZmz4vdB On Sat, Mar 2, 2024 at 6:13=E2=80=AFAM Konstantin Belousov wrote: > > CAUTION: This email originated from outside of the University of Guelph. = Do not click links or open attachments unless you recognize the sender and = know the content is safe. If in doubt, forward suspicious emails to IThelp@= uoguelph.ca. > > > On Sat, Mar 02, 2024 at 05:40:08AM -0800, Rick Macklem wrote: > > On Fri, Mar 1, 2024 at 10:51=E2=80=AFPM Konstantin Belousov wrote: > > > > > > CAUTION: This email originated from outside of the University of Guel= ph. Do not click links or open attachments unless you recognize the sender = and know the content is safe. If in doubt, forward suspicious emails to ITh= elp@uoguelph.ca. > > > > > > > > > On Fri, Mar 01, 2024 at 06:23:56AM -0800, Rick Macklem wrote: > > > > On Fri, Mar 1, 2024 at 12:00=E2=80=AFAM Ronald Klop wrote: > > > > > > > > > > Interesting read. > > > > > > > > > > Would it be possible to separate locking for admin actions like = a client mounting an fs from traffic flowing for file operations? > > > > Well, the NFS server does not really have any concept of a mount. > > > > What I am referring to is the ClientID maintained for NFSv4 mounts, > > > > which all the open/lock/session/layout state hangs off of. > > > > > > > > For most cases, this state information can safely be accessed/modif= ied > > > > via a mutex, but there are three exceptions: > > > > - creating a new ClientID (which is done by the ExchangeID operatio= n) > > > > and typically happens when a NFS client does a mount. > > > > - delegation Recall (which only happens when delegations are enable= d) > > > > One of the reasons delegations are not enabled by default on the > > > > FreeBSD server. > > > > - the DestroyClientID which is typically done by a NFS client durin= g dismount. > > > > For these cases, it is just too difficult to do them without sleepi= ng. > > > > As such, there is a sleep lock which the nfsd threads normally acqu= ire shared > > > > when doing NFSv4 operations, but for the above cases the lock is aq= uired > > > > exclusive. > > > > - I had to give the exclusive lock priority over shared lock > > > > acquisition (it is a > > > > custom locking mechanism with assorted weirdnesses) because witho= ut > > > > that someone reported that new mounts took up to 1/2hr to occur. > > > > (The exclusive locker waited for 30min before all the other nfsd = threads > > > > were not busy.) > > > > Because of this priority, once a nfsd thread requests the exclusi= ve lock, > > > > all other nfsd threads executing NFSv4 RPCs block after releasing= their > > > > shared lock, until the exclusive locker releases the exclusive lo= ck. > > > Normal lockmgr locks + TDP_DEADLKTREAT private thread flag provide th= e > > > property of pref. exclusive waiters in presence of the shared waiters= . > > > I think this is what you described above. > > It also has some weird properties, like if there are multiple requestor= s > > for the exclusive lock, once one thread gets it (the threads are nfsd w= orker > > threads and indistinct), the others that requested an exclusive thread = are > > unblocked without the lock being issued to them. > This sounds to me as LK_SLEEPFAIL feature of lockmgr. > Do not underestimate the amount of weird features in it. Yep, sounds like it. I should take a look to see if lockmgr will work instead of the "rolled my own". I should also take another look at new client creation, to see if there is = a way to do it that doesn't require the exclusive lock (a lot of that code is 20years old now). rick > > > They then check if the exclusive lock is still needed (usually not, sin= ce > > the other thread has dealt with the case where it was needed) and > > then they can acquire a shared lock. > > Without this, there were cases where several threads would acquire > > the exclusive lock and then discover that the lock was not needed and > > just release it again. > > > > It also uses an assortment of weird flags/call args. > > > > rick > >