From owner-freebsd-current@freebsd.org Mon Apr 19 08:30:20 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 098875E82AF for ; Mon, 19 Apr 2021 08:30:20 +0000 (UTC) (envelope-from olav@backupbay.com) Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FP0PR0nDHz3N14 for ; Mon, 19 Apr 2021 08:30:18 +0000 (UTC) (envelope-from olav@backupbay.com) Received: by mail-ed1-x532.google.com with SMTP id i3so13922153edt.1 for ; Mon, 19 Apr 2021 01:30:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=backupbay-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=QUPziVF6dsO6L7KSKJxgYOABBBW+MjAeY0SU80olPCc=; b=ir0tNY5Yg2WSJOT6jlWRrcuGuH8FmnvTbkLKgwsSbvxsNhaGbOZ2I/POGPPg58tySM OArB1QY+36JtNF8ds9sq0NG0upkAEdb0neCAaq91lJkmOmbx/84u2bd+e24FgDK25Q6U HkHam7DtpokSGfxTnusAYaa5yUBfvFUmJmgzKAto7ZjX2S7MOBEIye360QUGQ8BNnCEd toNBZopnln6YZbsPyQhdtmW97eiQnBgwLgQhvfb+pPiGdr8IaVhFl2IVFm04lAxZJ1I3 hkLQ9mj1+7X1k1rSxIiQ9WjCGPHZewfrhptHMKPb+mUGMccV6Nq4Aljym2o/xuILe9yU 2kRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QUPziVF6dsO6L7KSKJxgYOABBBW+MjAeY0SU80olPCc=; b=GpQNv5QqnFK1vodMQJSxiYXw/FtSCCirIyrT4bFp020nWapdOrgGAHgJsX78OWrVjA MG91Zm567v6kmDAjs9XiNpOdUWNgLtCLmlthiQoQNgk0a9xLkp3nTaN4aEfH8Ad7pMxn Psvg0H6fUFoguKxlep++lRzs7TqaFEJgk2YHnHpzCHUEELnSZbC5KW+181zO/0jMt3re 3UNKSpRoEU+8i6tK+S560f0+M2b3zZMvI66Jm+TvHU7hXocgDWFDHf7cEtGPgeGqD2gV VrJfenyKyzUFzb/Ju9toz8WUCr201uZsgeUqIJwfS7jiPfmhAWBzPI2iHs8P1YsKG1et Y2Yg== X-Gm-Message-State: AOAM532ZUGuliYYrJ+sxs9zxMpJ9q/G7ulBJXSC6k/mnXQM/eUCt5X94 AmayRdcirPJK0/1FC5OSZGwK7XjNVv90YfWjU3St7g== X-Google-Smtp-Source: ABdhPJzJFpfKjHk+VuCkLzSFA6HyqTB/syo9tKPQaHxBVJ1rENTmy8z4eiW2HE//6PTTA6yyPocCEW9CcitqcwcYPxc= X-Received: by 2002:a50:9b12:: with SMTP id o18mr24791288edi.376.1618821016597; Mon, 19 Apr 2021 01:30:16 -0700 (PDT) MIME-Version: 1.0 References: <902a3c81-2ce8-49c0-b163-5ffa4b90afe5@www.fastmail.com> In-Reply-To: From: Olav Gjerde Date: Mon, 19 Apr 2021 10:30:05 +0200 Message-ID: Subject: Re: NFS issues since upgrading to 13-RELEASE To: Rick Macklem Cc: Allan Jude , "freebsd-current@freebsd.org" X-Rspamd-Queue-Id: 4FP0PR0nDHz3N14 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=backupbay-com.20150623.gappssmtp.com header.s=20150623 header.b=ir0tNY5Y; dmarc=none; spf=softfail (mx1.freebsd.org: 2a00:1450:4864:20::532 is neither permitted nor denied by domain of olav@backupbay.com) smtp.mailfrom=olav@backupbay.com X-Spamd-Result: default: False [-3.30 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[backupbay-com.20150623.gappssmtp.com:s=20150623]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[backupbay.com]; R_SPF_SOFTFAIL(0.00)[~all:c]; RBL_SENDERSCORE_FAIL(0.00)[2a00:1450:4864:20::532:server fail]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::532:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[backupbay-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::532:from]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::532:from]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-current] X-Mailman-Approved-At: Mon, 19 Apr 2021 14:17:43 +0000 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Apr 2021 08:30:20 -0000 I have tried D29690 patch and reverting back to r367492 this weekend. Neither made any difference for my system. There is also a reddit thread about this https://www.reddit.com/r/freebsd/comments/mqol4o/nfs_issues_since_upgrading= _to_13release/ On Sat, Apr 17, 2021 at 1:10 AM Rick Macklem wrote: > Just fyi, I just got a "recursed on non-recursed mutex" panic in > socantrcvmore() with the D29690 patch, so you might not > want to test with that one yet. > > rick > > ________________________________________ > From: owner-freebsd-current@freebsd.org > on behalf of Olav Gjerde > Sent: Thursday, April 15, 2021 3:21 PM > To: Allan Jude > Cc: freebsd-current@freebsd.org > Subject: Re: NFS issues since upgrading to 13-RELEASE > > CAUTION: This email originated from outside of the University of Guelph. > Do not click links or open attachments unless you recognize the sender an= d > know the content is safe. If in doubt, forward suspicious emails to > IThelp@uoguelph.ca > > > Well something do happen if I restart NFS Service on FreeBSD , it works f= or > like 10 seconds then it gets unresponsive again. > > This is my output from `nfsstat -d 1` > > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 8.00 1025 8.00 8.02 17170 134.54 2.01 72716 142.54 0.07 51 34 > 8.00 2273 17.76 7.99 31273 244.07 2.01 133267 261.83 0.14 20 82 > 8.03 4889 38.33 7.99 25885 202.07 2.06 119340 240.40 0.13 21 81 > [=3D=3D=3D=3D=3D Read =3D=3D=3D=3D=3D] [=3D=3D=3D=3D=3D Write =3D=3D=3D= =3D] [=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Total =3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D] > KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s ms ql %b > 7.98 8811 68.64 8.00 12997 101.54 2.22 78396 170.18 0.15 1 80 > 7.99 922 7.20 8.00 3798 29.68 2.10 17965 36.87 0.09 0 11 > 8.07 2959 23.31 0.00 0 0.00 2.67 8938 23.31 0.86 32 72 > 7.97 7088 55.18 0.00 0 0.00 2.66 21233 55.18 1.05 16 98 > 7.98 4666 36.38 0.00 0 0.00 2.66 13986 36.38 0.36 9 29 > 8.00 4513 35.24 8.00 7662 59.86 2.20 44188 95.10 0.27 10 49 > 7.98 4799 37.40 8.00 11422 89.23 2.16 60076 126.63 0.19 0 51 > 8.00 4322 33.76 0.00 0 0.00 2.67 12967 33.76 0.89 0 42 > 8.02 4839 37.91 0.00 0 0.00 2.67 14550 37.91 0.54 17 41 > 8.01 4516 35.32 0.00 0 0.00 2.67 13569 35.32 0.57 27 38 > 7.95 4459 34.62 8.00 1195 9.34 2.49 18109 43.96 0.55 0 45 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 > > > > On Thu, Apr 15, 2021 at 9:07 PM Olav Gjerde wrote: > > > I have the same issue, using Ubuntu 20.10 with Linux 5.8 kernel. The > Linux > > NFS client will get unresponsive and it does not recover in my case, ev= en > > if I restart NFS on FreeBSD. I upgraded from FreeBSD 12.1-RELEASE thoug= h. > > > > On Thu, Apr 15, 2021 at 8:36 PM Allan Jude > wrote: > > > >> On 4/15/2021 9:22 AM, Chris Roose wrote: > >> > I posted this in -questions and someone suggested I post here as wel= l. > >> > > >> > I'm having NFS availability issues between my Proxmox client and > >> FreeBSD server (10G link) since upgrading to 13-RELEASE. And > unfortunately > >> I upgraded my ZFS pool to v2.0.0 before I noticed the issue, so I'm > kind of > >> stuck. > >> > > >> > Periodically, the NFS server (I've tried both v3 and v4.2 clients) > will > >> go unresponsive for several minutes. I never had this problem on 12.2, > and > >> as far as I can tell it's not a disk or network I/O issue. I'll get > several > >> "nfs: server not responding, still trying" messages on the client and = a > few > >> minutes later it usually recovers. It's not clear to me yet what's > causing > >> the block. Restarting nfsd on the server will resolve the issue if it > >> doesn't clear itself. > >> > > >> > Any pointers for troubleshooting this? I've been looking through > >> vmstat, gstat, top, etc. when the problem occurs, but I haven't been > able > >> to pinpoint the issue. I can get pcap, but it would be from the hosts, > >> because I don't have a 10G tap or managed switch. > >> > > >> > >> run `nfsstat -d 1` and try to capture a few lines from before, during, > >> and after the stall, and that may provide some insight. > >> > >> Specifically, does the queue length grow, suggesting it is waiting on > >> the I/O subsystem, or does it just stop getting traffic all together. > >> > >> > >> -- > >> Allan Jude > >> _______________________________________________ > >> freebsd-current@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-current > >> To unsubscribe, send any mail to " > freebsd-current-unsubscribe@freebsd.org > >> " > >> > > > > > > -- > > Kind Regards / Med Vennlig Hilsen > > > > Olav Gr=C3=B8n=C3=A5s Gjerde > > > > BackupBay Gjerde > > Madlaforen 35 > > 4042 HAFRSFJORD > > Norway > > Phone: +47 918 000 59 > > > > > -- > Kind Regards / Med Vennlig Hilsen > > Olav Gr=C3=B8n=C3=A5s Gjerde > > BackupBay Gjerde > Madlaforen 35 > 4042 HAFRSFJORD > Norway > Phone: +47 918 000 59 > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org= " > --=20 Kind Regards / Med Vennlig Hilsen Olav Gr=C3=B8n=C3=A5s Gjerde BackupBay Gjerde Madlaforen 35 4042 HAFRSFJORD Norway Phone: +47 918 000 59