From owner-freebsd-current@freebsd.org Thu Apr 15 19:21:21 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0D64E5D3FB7 for ; Thu, 15 Apr 2021 19:21:21 +0000 (UTC) (envelope-from olav@backupbay.com) Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [IPv6:2a00:1450:4864:20::634]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FLq2S15Ktz4l5P for ; Thu, 15 Apr 2021 19:21:19 +0000 (UTC) (envelope-from olav@backupbay.com) Received: by mail-ej1-x634.google.com with SMTP id w3so38586102ejc.4 for ; Thu, 15 Apr 2021 12:21:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=backupbay-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=5ddY6AYHSCwed8jKa1+tLBula6R7FLK81JzYEHNZWWs=; b=ZkEG1xlzGxkNsRdBHGjzWKp0XgVnRhCeZsr5Q58/aXbXQu7KciH39urohnSBMXtvr3 eZ53pXxPFJzLWnJoPWgTdeb2uIavmvwG2nWPpG3KE4fUVAs+65ZWI4klIrmbQnGQlGJN 2SvkTCt0UbsGfgu5oc97icOHIzXwMgLtpTGNlzCtooENNJcLYNBrWhQhIcPqIHySCqOt QmL97RAXxtZDfPJXuUyiHooLDMZwO+RZZhD2EoiyKyK+4nYmh3SfFGmi6NcwGqZvM3cD OHgbNhbmPINxv/8ZPslIYGsGlCRGBMWja/MK4kw+Krb4+briz+e49wJssYJms5HLSHEP yxkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=5ddY6AYHSCwed8jKa1+tLBula6R7FLK81JzYEHNZWWs=; b=lqV0ljWRaOwc7yK12EO5H/Jg5e1evdTepl7WH8oMsjw7L0ZfpR4ALvsdSR4OMpMaRg n8xKCMW/z8V3vpeSOsjoldaB/l2mIAmvT5tJTJRE/0xsOXTNHQwyjzyISJ6hCDDVp/ZQ yRHdtm+79WvbCG6zLREeBc5ZkbIJWxCvbv4A/lbw+8D9M6D+PVf0CCXn9PQaPwPYFddX XgQ2pkaJbJBfSAAFpzSzM58nTgCICD168x6d5r5TCRStRh0f4kW4tiosUvtc26GbG8xv hFwow9I1tuFgIl4iwayI2GYGh/MboDDdUVcMNJjxlcrn7qssk0rktrUumAGu3QrQqkcs clJg== X-Gm-Message-State: AOAM530hZJIK+kKWkGxBR38XnsJRXLT6sN5abr7WJx/EQhzbQIGCfd7E GXlyum+zP8FxYRNFQ1sjkM8wNT/kdMy7jvmlKpqBb/bTv4BEp2HH X-Google-Smtp-Source: ABdhPJxSnnW2NGTcRPzW17XttXMirjZF76lDLMv7qCBzKi0ugKUXSSi/b1RcBlOJaeEpEzx2DRd5SJlRZtrLfEzcNcc= X-Received: by 2002:a17:906:3e89:: with SMTP id a9mr4858541ejj.405.1618514478589; Thu, 15 Apr 2021 12:21:18 -0700 (PDT) MIME-Version: 1.0 References: <902a3c81-2ce8-49c0-b163-5ffa4b90afe5@www.fastmail.com> In-Reply-To: From: Olav Gjerde Date: Thu, 15 Apr 2021 21:21:07 +0200 Message-ID: Subject: Re: NFS issues since upgrading to 13-RELEASE To: Allan Jude Cc: freebsd-current@freebsd.org X-Rspamd-Queue-Id: 4FLq2S15Ktz4l5P X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=backupbay-com.20150623.gappssmtp.com header.s=20150623 header.b=ZkEG1xlz; dmarc=none; spf=softfail (mx1.freebsd.org: 2a00:1450:4864:20::634 is neither permitted nor denied by domain of olav@backupbay.com) smtp.mailfrom=olav@backupbay.com X-Spamd-Result: default: False [-3.20 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[backupbay-com.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[backupbay.com]; R_SPF_SOFTFAIL(0.00)[~all]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::634:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[backupbay-com.20150623.gappssmtp.com:+]; MIME_BASE64_TEXT(0.10)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::634:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::634:from]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-current] X-Mailman-Approved-At: Fri, 16 Apr 2021 06:22:01 +0000 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Apr 2021 19:21:21 -0000 Well something do happen if I restart NFS Service on FreeBSD , it works for like 10 seconds then it gets unresponsive again. This is my output from `nfsstat -d 1` 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 8.00 1025 8.00 8.02 17170 134.54 2.01 72716 142.54 0.07 51 34 8.00 2273 17.76 7.99 31273 244.07 2.01 133267 261.83 0.14 20 82 8.03 4889 38.33 7.99 25885 202.07 2.06 119340 240.40 0.13 21 81 [=3D=3D=3D=3D=3D Read =3D=3D=3D=3D=3D] [=3D=3D=3D=3D=3D Write =3D=3D=3D=3D= ] [=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Total =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D] KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s ms ql %b 7.98 8811 68.64 8.00 12997 101.54 2.22 78396 170.18 0.15 1 80 7.99 922 7.20 8.00 3798 29.68 2.10 17965 36.87 0.09 0 11 8.07 2959 23.31 0.00 0 0.00 2.67 8938 23.31 0.86 32 72 7.97 7088 55.18 0.00 0 0.00 2.66 21233 55.18 1.05 16 98 7.98 4666 36.38 0.00 0 0.00 2.66 13986 36.38 0.36 9 29 8.00 4513 35.24 8.00 7662 59.86 2.20 44188 95.10 0.27 10 49 7.98 4799 37.40 8.00 11422 89.23 2.16 60076 126.63 0.19 0 51 8.00 4322 33.76 0.00 0 0.00 2.67 12967 33.76 0.89 0 42 8.02 4839 37.91 0.00 0 0.00 2.67 14550 37.91 0.54 17 41 8.01 4516 35.32 0.00 0 0.00 2.67 13569 35.32 0.57 27 38 7.95 4459 34.62 8.00 1195 9.34 2.49 18109 43.96 0.55 0 45 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0 On Thu, Apr 15, 2021 at 9:07 PM Olav Gjerde wrote: > I have the same issue, using Ubuntu 20.10 with Linux 5.8 kernel. The Linu= x > NFS client will get unresponsive and it does not recover in my case, even > if I restart NFS on FreeBSD. I upgraded from FreeBSD 12.1-RELEASE though. > > On Thu, Apr 15, 2021 at 8:36 PM Allan Jude wrote: > >> On 4/15/2021 9:22 AM, Chris Roose wrote: >> > I posted this in -questions and someone suggested I post here as well. >> > >> > I'm having NFS availability issues between my Proxmox client and >> FreeBSD server (10G link) since upgrading to 13-RELEASE. And unfortunate= ly >> I upgraded my ZFS pool to v2.0.0 before I noticed the issue, so I'm kind= of >> stuck. >> > >> > Periodically, the NFS server (I've tried both v3 and v4.2 clients) wil= l >> go unresponsive for several minutes. I never had this problem on 12.2, a= nd >> as far as I can tell it's not a disk or network I/O issue. I'll get seve= ral >> "nfs: server not responding, still trying" messages on the client and a = few >> minutes later it usually recovers. It's not clear to me yet what's causi= ng >> the block. Restarting nfsd on the server will resolve the issue if it >> doesn't clear itself. >> > >> > Any pointers for troubleshooting this? I've been looking through >> vmstat, gstat, top, etc. when the problem occurs, but I haven't been abl= e >> to pinpoint the issue. I can get pcap, but it would be from the hosts, >> because I don't have a 10G tap or managed switch. >> > >> >> run `nfsstat -d 1` and try to capture a few lines from before, during, >> and after the stall, and that may provide some insight. >> >> Specifically, does the queue length grow, suggesting it is waiting on >> the I/O subsystem, or does it just stop getting traffic all together. >> >> >> -- >> Allan Jude >> _______________________________________________ >> freebsd-current@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.or= g >> " >> > > > -- > Kind Regards / Med Vennlig Hilsen > > Olav Gr=C3=B8n=C3=A5s Gjerde > > BackupBay Gjerde > Madlaforen 35 > 4042 HAFRSFJORD > Norway > Phone: +47 918 000 59 > --=20 Kind Regards / Med Vennlig Hilsen Olav Gr=C3=B8n=C3=A5s Gjerde BackupBay Gjerde Madlaforen 35 4042 HAFRSFJORD Norway Phone: +47 918 000 59