From owner-freebsd-current@freebsd.org Sat Apr 17 13:23:49 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1BD515D6429 for ; Sat, 17 Apr 2021 13:23:49 +0000 (UTC) (envelope-from jason.unovitch@gmail.com) Received: from mail-io1-xd2d.google.com (mail-io1-xd2d.google.com [IPv6:2607:f8b0:4864:20::d2d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FMv103M4Cz4hTy; Sat, 17 Apr 2021 13:23:48 +0000 (UTC) (envelope-from jason.unovitch@gmail.com) Received: by mail-io1-xd2d.google.com with SMTP id b9so672163iod.13; Sat, 17 Apr 2021 06:23:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=C4GQv4KXEjvVoy+3vqrqqBGNFJdWmbDcYEMVShxDAbs=; b=cOOXG5BEGm2T0VNI1QgXvBNOOvuhSdjgd6doH+neR2lh0CUI8ynIGbsbAsILITjB8p QQwff0fXwWV8/h3DGj0z+5pKfAnvhTYJ4Gf7glmj/JSgLIv96YmKXW+y8KPVz+5RJKZL gbtZ3Dlk4aXqdfJWet0q+WmFwrcXFD3QIGNE4ZouJHGeqDiya/vwP8Mbs1cgpJbRfVWy pbnOjpZZXkksAwh+IL4Tbz+LmGY8kspCYBOODp/y2stQT0WT49LH1qmiZDpt5DewgdSd AGguO+QegVxCz664z4xdQmtrXql/AmkZK02WWpJDiodpNMiTrk4Ym4xPm7DMQmIpJmuz cBiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=C4GQv4KXEjvVoy+3vqrqqBGNFJdWmbDcYEMVShxDAbs=; b=bhiE/5XzS20Df2/CBkPjymKV4uH/wMA80GZc6qUu/aC2jn2QtLImLYTuvCrLPZ5C+Q qB/5ZEMQu0bvmFzy3RSU/dU/3ScF3jwjd3FxOzPZpU3Sr0h1ELFAY35vpBo4/0uuGSme 0JktMb7er5BvjRinvS9Pb80NrW0ZY7aVx6fa5Ux7voldKmF3pdH6hHSBmx4Dx5Mz2kSD JkzuMwJklLtF1hL0bEbg+sng6/NfxN2/XM2OCoN4kpYrj0j287IhOfXY/e2zTB6Kmmfx leH2B2CWFc4EZ+humr+1MX8rb3DoljJzY/tvW8yz0YfhZhbhnUX8GBZKGR0aVdEAsN1S kGuA== X-Gm-Message-State: AOAM5337VDgTJ4eAjm8bOSp50AqOu980M9DYbSWqVJXWldOmhBZBJlsc OaFjSLcbJnNKMtiNbMK9hXEMWUzCZAGyGDngJmHT6requJioMA== X-Google-Smtp-Source: ABdhPJzmo06tSBRf8ItNArauPBOIlBneaTXwX0UAcQRmBWBoaO6gNAxfyFyu5k9IvkFkfxxA/XV7ExEMz2BIcilvoLM= X-Received: by 2002:a6b:f802:: with SMTP id o2mr7345552ioh.197.1618665827107; Sat, 17 Apr 2021 06:23:47 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Jason Unovitch Date: Sat, 17 Apr 2021 09:23:35 -0400 Message-ID: Subject: Re: NFS issues since upgrading to 13-RELEASE To: FreeBSD Current , Olav Gjerde X-Rspamd-Queue-Id: 4FMv103M4Cz4hTy X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=cOOXG5BE; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jasonunovitch@gmail.com designates 2607:f8b0:4864:20::d2d as permitted sender) smtp.mailfrom=jasonunovitch@gmail.com X-Spamd-Result: default: False [-0.00 / 15.00]; RBL_SENDERSCORE_FAIL(0.00)[2607:f8b0:4864:20::d2d:server fail]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::d2d:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(1.00)[1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::d2d:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::d2d:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-current] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Apr 2021 13:23:49 -0000 Olav, Does anything change if you set -tso -lro on the serving NIC on your FreeBSD server side? Do the Linux clients remain responsive then? I had seen something similar with a Ubuntu 20.04 client going to a 13.0-CURRENT/STABLE server but chalked it up to it starting around the time of the last Chelsio firmware update. I haven't had the time to dig into it and the temporary triage of setting '-tso -lro' at boot hasn't been as temporary as I hoped. That did stable it up on my side. Curious what you see and if that helps add another data point to understanding the issue. Jason On Fri, Apr 16, 2021 at 1:30 AM wrot > Date: Thu, 15 Apr 2021 21:07:18 +0200 > From: Olav Gjerde > To: Allan Jude > Cc: freebsd-current@freebsd.org > Subject: Re: NFS issues since upgrading to 13-RELEASE > Message-ID: > irDNg2mOA@mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > I have the same issue, using Ubuntu 20.10 with Linux 5.8 kernel. The Linux > NFS client will get unresponsive and it does not recover in my case, even > if I restart NFS on FreeBSD. I upgraded from FreeBSD 12.1-RELEASE though. > > On Thu, Apr 15, 2021 at 8:36 PM Allan Jude wrote: > > > On 4/15/2021 9:22 AM, Chris Roose wrote: > > > I posted this in -questions and someone suggested I post here as well. > > > > > > I'm having NFS availability issues between my Proxmox client and > FreeBSD > > server (10G link) since upgrading to 13-RELEASE. And unfortunately I > > upgraded my ZFS pool to v2.0.0 before I noticed the issue, so I'm kind of > > stuck. > > > > > > Periodically, the NFS server (I've tried both v3 and v4.2 clients) will > > go unresponsive for several minutes. I never had this problem on 12.2, > and > > as far as I can tell it's not a disk or network I/O issue. I'll get > several > > "nfs: server not responding, still trying" messages on the client and a > few > > minutes later it usually recovers. It's not clear to me yet what's > causing > > the block. Restarting nfsd on the server will resolve the issue if it > > doesn't clear itself. > > > > > > Any pointers for troubleshooting this? I've been looking through > vmstat, > > gstat, top, etc. when the problem occurs, but I haven't been able to > > pinpoint the issue. I can get pcap, but it would be from the hosts, > because > > I don't have a 10G tap or managed switch. > > > > > > > run `nfsstat -d 1` and try to capture a few lines from before, during, > > and after the stall, and that may provide some insight. > > > > Specifically, does the queue length grow, suggesting it is waiting on > > the I/O subsystem, or does it just stop getting traffic all together. > > > > > > -- > > Allan Jude > > _______________________________________________ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to " > freebsd-current-unsubscribe@freebsd.org" > > > > > -- > Kind Regards / Med Vennlig Hilsen > > Olav Gr?n?s Gjerde > > BackupBay Gjerde > Madlaforen 35 > 4042 HAFRSFJORD > Norway > Phone: +47 918 000 59 >