From owner-freebsd-stable@freebsd.org Thu Dec 19 21:43:57 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 569EF1EBCC8 for ; Thu, 19 Dec 2019 21:43:57 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 47f53x0pXvz48YD for ; Thu, 19 Dec 2019 21:43:57 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 19F1A1EBCC7; Thu, 19 Dec 2019 21:43:57 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 19B101EBCC6 for ; Thu, 19 Dec 2019 21:43:57 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47f53w1MqXz48YC for ; Thu, 19 Dec 2019 21:43:55 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: by mail-wm1-x335.google.com with SMTP id p17so7212354wma.1 for ; Thu, 19 Dec 2019 13:43:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=zG6Y6t3Wkfd9Kl9ilev35H+aUElZY+JKuN64YpXxE2A=; b=HDgltDIa3TRPVI1HmA+TSx8MsYJabUccdlx6JDr+tckHvuGIxZ6FSYMvZl3d0FweE0 KrWlwxv45J3w5bAl+iASwrNQ0qOrMf9pwbxoPC7iFGWE2q3s2SON/ip2MfOn/lKr6sDn rNXTyu4PoKIgqtFYBqLFXPTCai5HgDRyRy8XAD0nfHifMbTbhP7Ohb+haIMVhmhOOKq6 7pe24D81o/HowZdWpX/3PMIQqYXPPDhgBqLSD2s+KJFxWW3PwbXJY6xMtfhHXbZNdVNN Q8YnHyoErTqF7d87PBZ//zsYn7qBQd0NkPt3Gh2xKIJfQp336tXekFq+ua2VFpFEfLKc AJ2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=zG6Y6t3Wkfd9Kl9ilev35H+aUElZY+JKuN64YpXxE2A=; b=DUA1UUQ8Tfw5m+uX8U5TcFZ6AJ7GXZwy+F7yt0kzitdGlwnhJqDMN7gZqgeaqozpm0 2JxXcJF2AiJw8exmpS/5pt4AU48jS290EdZrkZ1ESbyeJ8Ak94vjTZCicmd0mVU21q7X W2wReBfJMlHyMSu4r+ASnBi88jIx+CCDOQ0Y1JT5TXj0Y5jalMqu+MkSm/9fwnIxKiOa dQLpXXKU7yRDHmSUZAYrwRqequZmfllmpp4jdWiqGqtMhmvC3XBYYcHpSA8tAoDzkHLX hRnf+MpIU3P/W5yiZvinKwwMBIpMOsIC3eQ1vTa3wntNJayPZfsgEhepfGsHToOuYdlB tmvQ== X-Gm-Message-State: APjAAAUC1141clQkKOhO0Afo67QsByynbd4eO22/I+RyHkv0DAA7YRl+ pzvu/WfU8Yjs00tMp6qDnyIFm29VMW9SWy0sAKw= X-Google-Smtp-Source: APXvYqzK/woX27Ahv9NMS3dyn3qM+omDRO7LUUO/tFPxQLiS4D698q3LiwVfv+dYlBhsrDHTRJHR3vjijPul4q8mtLY= X-Received: by 2002:a7b:c342:: with SMTP id l2mr13193708wmj.159.1576791832534; Thu, 19 Dec 2019 13:43:52 -0800 (PST) MIME-Version: 1.0 References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> In-Reply-To: <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> From: Richard P Mackerras Date: Thu, 19 Dec 2019 21:43:42 +0000 Message-ID: Subject: Re: nfs lockd errors after NetApp software upgrade. To: Daniel Braniss Cc: Rick Macklem , "stable@freebsd.org" X-Rspamd-Queue-Id: 47f53w1MqXz48YC X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=HDgltDIa; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mack63richard@gmail.com designates 2a00:1450:4864:20::335 as permitted sender) smtp.mailfrom=mack63richard@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (-9.19), ipnet: 2a00:1450::/32(-2.65), asn: 15169(-1.90), country: US(-0.05)]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[5.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2019 21:43:57 -0000 Hi, At ONTAP 9.3P6 there is a possible LACP group issue after upgrade. Have you checked any LACP groups, These should not be a problem but I assume network interfaces are at the home ports, not on slower ports or something silly. It is marginally better if the traffic goes direct to the node where the volume is but the difference should nothing. Have you looked at the NetApp performance data? If you are going to do wireshark tcpdumps then you might want to run them from the NetApp as well. https://kb.netapp.com/app/answers/answer_view/a_id/1029833/~/how-to-capture-packet-traces-%28tcpdump%29-on-ontap-9.2%2B-systems- ::> network tcpdump start -node -port e0a -buffer-size 2097151 Let us know how you go, Richard