From owner-freebsd-stable@freebsd.org Mon Dec 16 15:30:19 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8BE9E1E6EBC for ; Mon, 16 Dec 2019 15:30:19 +0000 (UTC) (envelope-from marcnarc@gmail.com) Received: from mail-qk1-x742.google.com (mail-qk1-x742.google.com [IPv6:2607:f8b0:4864:20::742]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47c4w9316Hz3RFL for ; Mon, 16 Dec 2019 15:30:17 +0000 (UTC) (envelope-from marcnarc@gmail.com) Received: by mail-qk1-x742.google.com with SMTP id c16so1759185qko.6 for ; Mon, 16 Dec 2019 07:30:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=JQuhdFCH5jcb6KXv0qeCT4vBnxVQ1+6AEXRcvoyQUGI=; b=Mm5SnOWDIVVI0EUJfhIs73EIex8IJ+VYZNMtwFtenzUR/Q+odgL0rbr4suFa6gVSdu CD/4P8EvW2uQYNMb27KK4JLd23xASzb1U3q9f355MPzfjyh4RXMBNmK2LreYIQX2pb5p jkA7aZygCQsndMLVp36JtPXqsHyfpgNY1dP2+H69P5WKqXMpxtlOAoTUChdyP8FLFBMZ 9RsM3BsjU5cYJYYhPmf2Gye/EVrz/XjjZ+5MMy2mHwBpAN8J/vRyXu7e9Ddn3q+xp24a ZR2JJLhXC30aaBcW9lu1XE0EX2Qn0zdeYm4n5v9C6GzHtX7c9uTPzle/PVNeGueJcgBn 1UjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=JQuhdFCH5jcb6KXv0qeCT4vBnxVQ1+6AEXRcvoyQUGI=; b=oYQAwTNbB2ogteyNqvllCUPAjuTdALmbTDsZcuE7bkrxf54tcUhqM7CCw2/Jy+k8xg 92HU9tHyW4LEKkhSmo4K7juQ1O4Ty7OhZxTsJjJucqkaBJ8C1uLqHVr2uFL1tT44BsGo HD0iOYNeox5ceO4QzGqf73G/h82XrumOHZLh5UR5KNI9FaILBjpgm5AQaxHuj8OuzROx 2fhK1GeGZKkkuIBJT8CtqAqrgAG3cohd9sfOM5+0g/d5Q+XgC01EuwnOXchCAnkscTqk 45Zm6z1t+QpeU8d+S9mb7melOMnwO2gKaNazN5cqICfJPhfTkE8XTYq3VSovoZYMlNjY +sgA== X-Gm-Message-State: APjAAAUx+pgfftspt4OV86I4NYr0ydFfLZn4WC6MhqwNmc59LPFRdgca e+XDrKI/l1PXmr3Upkwyl8M= X-Google-Smtp-Source: APXvYqypGFqIRdZLAHHtQ/+E5s3J5KOzq9f5pSMYUq04BXLzTUuDGPQbXf6UoqDNQRyCZFICNvkivg== X-Received: by 2002:a05:620a:14a2:: with SMTP id x2mr27883780qkj.36.1576510216294; Mon, 16 Dec 2019 07:30:16 -0800 (PST) Received: from [10.10.1.32] ([192.252.130.194]) by smtp.gmail.com with ESMTPSA id w21sm7213191qth.17.2019.12.16.07.30.13 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 16 Dec 2019 07:30:14 -0800 (PST) Subject: Re: Boot loader stuck after first stage upgrading 11.2 to 12.0-RC2 To: sashk , Eugene Grosbein , Mark Martinec , "freebsd-stable@freebsd.org" References: <22f5b92a09ea4d62ac3feb74457067f7@ijs.si> <5EEBAFC0-4FA3-4219-A918-7376F4223656@me.com> <0F5FCC70-EADB-4F9E-A391-F1A73BE5608F@me.com> <1543954753.1860.243.camel@freebsd.org> <53ceda24-fa1b-8546-3511-bd500b440dfe@digiware.nl> <4c4019102b63054f8de93324dba0e776@ijs.si> <998f886a-7498-7268-75cf-4e767835db82@grosbein.net> <12248491576286986@iva8-03ad76494624.qloud-c.yandex.net> From: Marc Branchaud Message-ID: Date: Mon, 16 Dec 2019 10:29:06 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.2.1 MIME-Version: 1.0 In-Reply-To: <12248491576286986@iva8-03ad76494624.qloud-c.yandex.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 47c4w9316Hz3RFL X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Mm5SnOWD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marcnarc@gmail.com designates 2607:f8b0:4864:20::742 as permitted sender) smtp.mailfrom=marcnarc@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (2.87), ipnet: 2607:f8b0::/32(-2.19), asn: 15169(-1.90), country: US(-0.05)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[freebsd]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2.4.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Dec 2019 15:30:19 -0000 On 2019-12-13 8:29 p.m., sashk wrote: > I hit this issue about year ago and Toomas Soome was able to figure out what was causing this and fixed in the https://svnweb.freebsd.org/base?view=revision&revision=342151 > > I believe this got into 12.1. It did, along with a later fix for UEFI: https://svnweb.freebsd.org/base?view=revision&revision=342618 Thanks all! M. > -Aleks > > > 13.12.2019, 15:59, "Marc Branchaud" : >> On 2019-12-10 10:16 p.m., Eugene Grosbein wrote: >>>  10.12.2019 23:08, Mark Martinec пишет: >>>>  2019-12-10 16:35, Marc Branchaud wrote: >>>> >>>>>  On 2019-12-10 9:18 a.m., Mark Martinec wrote: >>>>>>  Commenting on a thread from 2018-12 and from 2019-09-20, with my solution >>>>>>  to the boot problem at the end, in case anyone is still interested. >>>>> >>>>>  Thank you very much for this. A couple of questions: >>>>> >>>>>  (1) Why do you say "raw devices for historical reasons"? Glancing >>>>>  through the zpool man page and the Handbook, I see nothing >>>>>  recommending or requiring GPT partitions. >>>> >>>>  Apparently using raw devices for zpool is now discouraged, >>>>  although I don't think it has ever become officially unsupported. >>> >>>  Loader bugs do not mean that "using raw devices for zpool is now discouraged". >> >> Glad to hear that! >> >> Should there be a PR for this issue? (A quick Bugzilla search for >> "zpool gpt boot" found nothing...) >> >>                 M. >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@freebsd.org Mon Dec 16 18:53:54 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4CC8E1CBE60; Mon, 16 Dec 2019 18:53:54 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47c9R61M0Wz4CB9; Mon, 16 Dec 2019 18:53:54 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 1129) id 2780E12F1D; Mon, 16 Dec 2019 18:53:54 +0000 (UTC) Date: Mon, 16 Dec 2019 18:53:54 +0000 From: Li-Wen Hsu To: freebsd-testing@freebsd.org Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: FreeBSD CI Weekly Report 2019-12-15 Message-ID: <20191216185354.GA25613@freefall.freebsd.org> Reply-To: freebsd-testing@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.11.4 (2019-03-13) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Dec 2019 18:53:54 -0000 (Please send the followup to freebsd-testing@ and note Reply-To is set.) FreeBSD CI Weekly Report 2019-12-15 =================================== Here is a summary of the FreeBSD Continuous Integration results for the period from 2019-12-09 to 2019-12-15. During this period, we have: * 2566 builds (96.3% (+1.3) passed, 3.7% (-1.3) failed) of buildworld and buildkernel (GENERIC and LINT) were executed on aarch64, amd64, armv6, armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, riscv64, sparc64 architectures for head, stable/12, stable/11 branches. * 400 test runs (96.4% (-1.6) passed, 1.8% (+0.1) unstable, 1.8% (+1.5) exception) were executed on amd64, i386, riscv64 architectures for head, stable/12, stable/11 branches. * 43 doc builds (93% (-7) passed, 7% (+7) failed) Test case status (on 2019-12-15 23:59): | Branch/Architecture | Total | Pass | Fail | Skipped | | ------------------- | --------- | --------- | ----- | ------- | | head/amd64 | 7635 (+3) | 7566 (+3) | 0 (0) | 69 (0) | | head/i386 | 7633 (+3) | 7560 (+3) | 0 (0) | 73 (0) | | 12-STABLE/amd64 | 7488 (0) | 7440 (+3) | 0 (0) | 48 (-3) | | 12-STABLE/i386 | 7486 (0) | 7428 (0) | 0 (0) | 58 (0) | | 11-STABLE/amd64 | 6858 (0) | 6811 (0) | 0 (0) | 47 (0) | | 11-STABLE/i386 | 6856 (0) | 6807 (+3) | 0 (0) | 49 (-3) | (The statistics from experimental jobs are omitted) If any of the issues found by CI are in your area of interest or expertise please investigate the PRs listed below. The latest web version of this report is available at https://hackmd.io/@FreeBSD-CI/report-20191215 and archive is available at https://hackmd.io/@FreeBSD-CI/ , any help is welcome. ## News * Experimental "Hardware test lab" result is available at: https://ci.freebsd.org/hwlab/ , more hardware support is welcomed! ## Failing and Flaky Tests (from experimental jobs) * https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/ * cddl.usr.sbin.dtrace.common.misc.t_dtrace_contrib.tst_dynopt_d * https://bugs.freebsd.org/237641 * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/ * There are ~13 failing and ~109 skipped cases, including flakey ones, see https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/lastCompletedBuild/testReport/ for more details * Work for cleaning these failing cass are in progress ## Disabled Tests * sys.fs.tmpfs.mount_test.large https://bugs.freebsd.org/212862 * sys.fs.tmpfs.link_test.kqueue https://bugs.freebsd.org/213662 * sys.kqueue.libkqueue.kqueue_test.main https://bugs.freebsd.org/233586 * sys.kern.ptrace_test.ptrace__PT_KILL_competing_stop https://bugs.freebsd.org/220841 * lib.libc.regex.exhaust_test.regcomp_too_big (i386 only) https://bugs.freebsd.org/237450 * sys.netinet.socket_afinet.socket_afinet_bind_zero https://bugs.freebsd.org/238781 * sys.netpfil.pf.names.names * sys.netpfil.pf.synproxy.synproxy https://bugs.freebsd.org/238870 * sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger https://bugs.freebsd.org/239292 * sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger https://bugs.freebsd.org/239397 * sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger https://bugs.freebsd.org/239399 * sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger https://bugs.freebsd.org/239425 * lib.libc.gen.getmntinfo_test.getmntinfo_test https://bugs.freebsd.org/240049 * sys.sys.qmath_test.qdivq_s64q https://bugs.freebsd.org/240219 * sys.kern.ptrace_test.ptrace__getppid https://bugs.freebsd.org/240510 * lib.libc.sys.stat_test.stat_socket https://bugs.freebsd.org/240621 * lib.libarchive.functional_test.test_write_filter_zstd https://bugs.freebsd.org/240683 * lib.libcasper.services.cap_dns.dns_test.main https://bugs.freebsd.org/241435 * local.kyua.* (31 cases) & local.lutok.* (3 cases) on 11-i386 https://ci.freebsd.org/job/FreeBSD-stable-11-i386-test/2278/testReport/ ## Issues ### Cause build fails * https://bugs.freebsd.org/233735 Possible build race: genoffset.o /usr/src/sys/sys/types.h: error: machine/endian.h: No such file or directory * https://bugs.freebsd.org/233769 Possible build race: ld: error: unable to find library -lgcc_s ### Cause kernel panics * https://bugs.freebsd.org/238870 sys.netpfil.pf.names.names and sys.netpfil.pf.synproxy.synproxy cause panic Patch exists: * https://reviews.freebsd.org/D20868 * https://reviews.freebsd.org/D20869 ### Open * https://bugs.freebsd.org/237403 Tests in sys/opencrypto should be converted to Python3 * https://bugs.freebsd.org/237641 Flakey test case: common.misc.t_dtrace_contrib.tst_dynopt_d * https://bugs.freebsd.org/237656 "Freed UMA keg (rtentry) was not empty (18 items). Lost 1 pages of memory." seen when running sys/netipsec tests * https://bugs.freebsd.org/238781 sys.netinet.socket_afinet.socket_afinet_bind_zero does not work when mac_portacl(4) loaded * https://bugs.freebsd.org/239292 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger * https://bugs.freebsd.org/239397 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger * https://bugs.freebsd.org/239399 Flakey test case: sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger * https://bugs.freebsd.org/239425 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger * https://bugs.freebsd.org/241662 Flakey test case: lib.libarchive.functional_test.test_fuzz_iso9660 ### Others * [Tickets related to testing@](https://preview.tinyurl.com/y9maauwg) From owner-freebsd-stable@freebsd.org Wed Dec 18 14:05:03 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7F58B1DFA73 for ; Wed, 18 Dec 2019 14:05:03 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 47dGwt4ypzz4d9r for ; Wed, 18 Dec 2019 14:05:02 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: by mailman.nyi.freebsd.org (Postfix) id A87491DFA72; Wed, 18 Dec 2019 14:05:02 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A834E1DFA71 for ; Wed, 18 Dec 2019 14:05:02 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47dGws0WLJz4d9q for ; Wed, 18 Dec 2019 14:05:00 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cs.huji.ac.il; s=57791128; h=To:Date:Message-Id:Subject:Mime-Version:Content-Transfer-Encoding:Content-Type:From; bh=012z0TF0k7+89nXq3vGs4cL36goRK99Ql+ibx4DE9Ug=; b=ehgS6pe85NdQBy7uFLKCea0xBCOaIRxpb9PZyHVEUvl8B886db/kwKFP7AXyCG5ZPeMORX3ljHlDWn+eaMamfF+f/8Z0vGOleVId9YOs9d9qyVmc3PkvmqcCQUn0aRNDybprraX/LMUd6SAz6vtWQJsW8LtQ43BU2qvfQ+b8YcsD8HwVVvCTtylFbPZHrGkPQs2dC0dFBD5VIeYXyxPKQJwOhbvJHAmQU2jZQx3NkmOU8yV4tK/F+WMR3etqyuP5lFr/8oX5gQOogIW4ashnI4Wf3RkfaPgl06yOw4RIUocwyciKhsevsCunglPtgt/4dJsEaRo+5Ei8cSuSS8tCqA==; Received: from macmini.bk.cs.huji.ac.il ([132.65.179.19]) by kabab.cs.huji.ac.il with esmtp id 1ihZwf-0008Mi-Ur for stable@freebsd.org; Wed, 18 Dec 2019 16:04:57 +0200 From: Daniel Braniss Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.40.2.2.4\)) Subject: nfs lockd errors after NetApp software upgrade. Message-Id: Date: Wed, 18 Dec 2019 16:04:57 +0200 To: stable@freebsd.org X-Mailer: Apple Mail (2.3608.40.2.2.4) X-Rspamd-Queue-Id: 47dGws0WLJz4d9q X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.huji.ac.il header.s=57791128 header.b=ehgS6pe8; dmarc=pass (policy=none) header.from=huji.ac.il; spf=none (mx1.freebsd.org: domain of danny@cs.huji.ac.il has no SPF policy when checking 132.65.116.210) smtp.mailfrom=danny@cs.huji.ac.il X-Spamd-Result: default: False [-3.30 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[cs.huji.ac.il:s=57791128]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-1.00)[ip: (-2.25), ipnet: 132.64.0.0/13(-1.57), asn: 378(-1.25), country: IL(0.05)]; DKIM_TRACE(0.00)[cs.huji.ac.il:+]; DMARC_POLICY_ALLOW(-0.50)[huji.ac.il,none]; RCVD_IN_DNSWL_NONE(0.00)[210.116.65.132.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:378, ipnet:132.64.0.0/13, country:IL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Dec 2019 14:05:03 -0000 Hi, The server with the problems is running FreeBSD 11.1 stable, it was = working fine for several months, but after a software upgrade of our NetAPP server it=E2=80=99s reporting = many lockd errors and becomes catatonic, ... Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not = responding Dec 18 13:11:45 moo-09 last message repeated 7 times Dec 18 13:12:55 moo-09 last message repeated 8 times Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is alive = again Dec 18 13:13:10 moo-09 last message repeated 8 times Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen = queue overflow: 194 already in queue awaiting acceptance (1 occurrences) Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen = queue overflow: 193 already in queue awaiting acceptance (3957 = occurrences) Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen = queue overflow: 193 already in queue awaiting acceptance (3404 = occurrences) Dec 18 13:16:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen = queue overflow: 196 already in queue awaiting acceptance (3553 = occurrences) Dec 18 13:17:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen = queue overflow: 194 already in queue awaiting acceptance (3661 = occurrences) Dec 18 13:18:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen = queue overflow: 197 already in queue awaiting acceptance (4030 = occurrences) Dec 18 13:19:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen = queue overflow: 193 already in queue awaiting acceptance (2560 = occurrences) Dec 18 13:20:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen = queue overflow: 193 already in queue awaiting acceptance (1495 = occurrences) Dec 18 13:21:32 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen = queue overflow: 193 already in queue awaiting acceptance (817 = occurrences) Dec 18 14:54:43 moo-09 kernel: nfs server fr-06:/mdlbck: lockd not = responding Dec 18 14:55:19 moo-09 last message repeated 2 times Dec 18 14:55:34 moo-09 kernel: nfs server fr-06:/mdlbck: lockd is alive = again =E2=80=A6 any ideas? thanks, danny From owner-freebsd-stable@freebsd.org Wed Dec 18 14:55:21 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 40F891E0933 for ; Wed, 18 Dec 2019 14:55:21 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 47dJ2w70Zwz4gLR for ; Wed, 18 Dec 2019 14:55:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: by mailman.nyi.freebsd.org (Postfix) id EE7731E0932; Wed, 18 Dec 2019 14:55:20 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EE3451E0930 for ; Wed, 18 Dec 2019 14:55:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660054.outbound.protection.outlook.com [40.107.66.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47dJ2v4K7Nz4gLQ for ; Wed, 18 Dec 2019 14:55:19 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZpxeILJ8+6feMWnamfpvvKlQ268jvb3C0/UTLShpNjQ/HYY5rzNpyT37eXK4PZNT1ui9eWpxKr1bcUbs0HSbdgES5+7UfSYTvZx4gjEH2VyJlgyOhh48nXSDlD/qoI22qhG0GMT2GbmkkQlUZolC9KYGYcFKDj9TS3Rw6sSg7iCVqz0PxqvOjav1NrezqYSPF5Ke9x7Ft2fhLOLLAwBUYJud9eACaJZ8M+vCHV/yGyo36CHhGSBzv5UfbOBSgP3tpotF81JjyxMo3bTQicdST5EM8zeyNqno2w/gjXm/GbmS0uOKCWow+9iZivwRmpZtz75K2kkzGfmfaGgbAY/NKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mMvLfLfLXCHPMrVg9i5fK09BNAb+EG+DQ/CJg2loocg=; b=AKTKQlv6urInaBZI/bqHoZTK7wwo2kv4TagxhZ5O3M7dDaOQRAC9WR/LjS62KPnRYTQeCQrUf1y9Tp+9lWTtPRPVLzDga6KdEvJeaGY+IpOBZ8eyFHDE3h6F9jvv8XE/QLefk3SVmhBV0ejio7oW65K0E18b9vZnRG1de7i3Gsacrrfih7Wx5rwmJhG81Pq8Mqlbrqcx8e+EJIOOCwxAdfO6j0cQli2YJNwnocgPkav4wfaSnRT9WVOsAuGQ5hsTZYvhR4bJD7VoSA5LAtB7f4Tv/w+33SMGOjw3Hv7cmqcBKsTqh4qH/hPs8whSyvJmY8LCdC/W87AwDYOt71Xbeg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM (52.132.69.153) by YQBPR0101MB1106.CANPRD01.PROD.OUTLOOK.COM (52.132.72.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2538.15; Wed, 18 Dec 2019 14:55:16 +0000 Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75]) by YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75%5]) with mapi id 15.20.2538.019; Wed, 18 Dec 2019 14:55:16 +0000 From: Rick Macklem To: Daniel Braniss , "stable@freebsd.org" Subject: Re: nfs lockd errors after NetApp software upgrade. Thread-Topic: nfs lockd errors after NetApp software upgrade. Thread-Index: AQHVtawq+ga5QLcdVkqBDG/GW9zFg6e/+Am+ Date: Wed, 18 Dec 2019 14:55:16 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 24b97e81-01b6-4f69-b0b9-08d783ca4b30 x-ms-traffictypediagnostic: YQBPR0101MB1106: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 0255DF69B9 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(189003)(199004)(6506007)(76116006)(2906002)(86362001)(66946007)(66446008)(64756008)(66556008)(8936002)(55016002)(966005)(66476007)(26005)(7696005)(186003)(5660300002)(33656002)(498600001)(71200400001)(8676002)(9686003)(110136005)(81156014)(81166006)(52536014); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB1106; H:YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: oSiqKDUAGphsvTq3NixmBcWnfEC0DD4zsh8XVyHzrWIZhJiDmitKG/J9NiziQMqF24yylqAnHFXsON2MvZnPm++SWDmp4Nu665Uh2afQUN/B1u9O0603lRm+EjZsyJQcvq2dlvs9nrvDl5EsTFDqthmkuiv1S7VymI1QJ5xPysbbuz18JyVAbv3FUuMHHjxLmu+ogru3OIXeM9IT0Jux/mKlAp6Dsp8am8RHOoVANqWk/oyaiuwZHwA+qVgaXcxyK+3W9ZSU2x2C2ug+WY++mRVVsQQ1SoKC6EYbZkinMMzp/63MTsuGEs+sTSpPZVqvwoCTGVRaAkEALGQDTowfai1TdfwiQM9emRth+OcT43DRmKD5dE+iKXcbg+LDTRXyXhz6d5hklA8bQVx4qszFYhmAScVtF2M6HrYBkv20iAH4iyzPt/SugLJVBewRjCe3KbIawfz+1+SDw44GN8xDq0hywMHCSR/KgieXpObqP8A= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 24b97e81-01b6-4f69-b0b9-08d783ca4b30 X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Dec 2019 14:55:16.3239 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: bmAw5PNx7+pnporYTLXWdI5lRh05jiO2FWK+xRBYGSasacwAD3Bp8YCIpF+Z9VqF8gJLlZb9iAfV7+V7J4TjsA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1106 X-Rspamd-Queue-Id: 47dJ2v4K7Nz4gLQ X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.54 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.66 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[uoguelph.ca]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[54.66.107.40.list.dnswl.org : 127.0.3.0]; IP_SCORE(-1.36)[ipnet: 40.64.0.0/10(-3.83), asn: 8075(-2.91), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Dec 2019 14:55:21 -0000 Daniel Braniss wrote:=0A= =0A= >Hi,=0A= >The server with the problems is running FreeBSD 11.1 stable, it was workin= g fine for >several months,=0A= >but after a software upgrade of our NetAPP server it=92s reporting many lo= ckd errors >and becomes catatonic,=0A= >...=0A= >Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not respon= ding=0A= >Dec 18 13:11:45 moo-09 last message repeated 7 times=0A= >Dec 18 13:12:55 moo-09 last message repeated 8 times=0A= >Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is alive a= gain=0A= >Dec 18 13:13:10 moo-09 last message repeated 8 times=0A= >Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen q= ueue >overflow: 194 already in queue awaiting acceptance (1 occurrences)=0A= >Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen q= ueue >overflow: 193 already in queue awaiting acceptance (3957 occurrences)= =0A= >Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen q= ueue >overflow: 193 already in queue awaiting acceptance =85=0A= Seems like their software upgrade didn't improve handling of NLM RPCs?=0A= Appears to be handling RPCs slowly and/or intermittently. Note that no one= =0A= tests it with IPv6, so at least make sure you are still using IPv4 for the = mounts and=0A= try and make sure IP broadcast works between client and Netapp. I think the= NLM=0A= and NSM (rpc.statd) still use IP broadcast sometimes.=0A= =0A= Maybe the network guys can suggest more w.r.t. why, but as I've stated befo= re,=0A= the NLM is a fundamentally broken protocol which was never published by Sun= ,=0A= so I suggest you avoid using it if at all possible.=0A= =0A= - If the locks don't need to be seen by other clients, you can just use the= "nolockd"=0A= mount option.=0A= or=0A= - If locks need to be seen by other clients, try NFSv4 mounts. Netapp filer= s=0A= should support NFSv4.1, which is a much better protocol that NFSv4.0.=0A= =0A= Good luck with it, rick=0A= =85=0A= any ideas?=0A= =0A= thanks,=0A= danny=0A= =0A= _______________________________________________=0A= freebsd-stable@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"= =0A= From owner-freebsd-stable@freebsd.org Wed Dec 18 15:46:45 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 388361E19CD for ; Wed, 18 Dec 2019 15:46:45 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 47dKBD346gz4jcb for ; Wed, 18 Dec 2019 15:46:44 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: by mailman.nyi.freebsd.org (Postfix) id 675961E19CC; Wed, 18 Dec 2019 15:46:44 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 671401E19CB for ; Wed, 18 Dec 2019 15:46:44 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47dKBC1ZZhz4jcZ for ; Wed, 18 Dec 2019 15:46:43 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cs.huji.ac.il; s=57791128; h=To:References:Message-Id:Content-Transfer-Encoding:Cc:Date:In-Reply-To:From:Subject:Mime-Version:Content-Type; bh=cstAlQaFrlEalhGR8BqQQxR9mVRRElGsvpWy9ZBU6sU=; b=0VU5TF0T96t/hZ96Aj70xcjIcbj8DBaE5fA2YQc31Pe2iOjd3VkRUEuNiTcQwhhPMKyFClzWp2DMzq58GvXXuV4lAiPnoHiDJ9uMdrlDLSdwqvMCYoY4ZPeRwb6BWJu0br7fJ5J2WcvpVn/a6FXZ3rvXBCzRxA83eiuwx2M2+0NSGLqpHrYCOdcVeTQyp8Do3SPvSTfPPJneyA+8Wv+CTJO9j6J5IwnPgR+mEZasHPkWLX9plWR9Cz2pbowF/jwCvviuKvIyLU/y5K3+fjBTPli1Nh9R1NYdU6dNOP6BA5/HzHIn1H7iP2TGse7f0ztGfzwzGbk1oaY7+xupNMHNpg==; Received: from macmini.bk.cs.huji.ac.il ([132.65.179.19]) by kabab.cs.huji.ac.il with esmtp id 1ihbX6-000EWt-99; Wed, 18 Dec 2019 17:46:40 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.40.2.2.4\)) Subject: Re: nfs lockd errors after NetApp software upgrade. From: Daniel Braniss In-Reply-To: Date: Wed, 18 Dec 2019 17:46:40 +0200 Cc: "stable@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> References: To: Rick Macklem X-Mailer: Apple Mail (2.3608.40.2.2.4) X-Rspamd-Queue-Id: 47dKBC1ZZhz4jcZ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.huji.ac.il header.s=57791128 header.b=0VU5TF0T; dmarc=pass (policy=none) header.from=huji.ac.il; spf=none (mx1.freebsd.org: domain of danny@cs.huji.ac.il has no SPF policy when checking 132.65.116.210) smtp.mailfrom=danny@cs.huji.ac.il X-Spamd-Result: default: False [-3.43 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[cs.huji.ac.il:s=57791128]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-1.13)[ip: (-2.61), ipnet: 132.64.0.0/13(-1.72), asn: 378(-1.38), country: IL(0.05)]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[cs.huji.ac.il:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[210.116.65.132.list.dnswl.org : 127.0.10.0]; DMARC_POLICY_ALLOW(-0.50)[huji.ac.il,none]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:378, ipnet:132.64.0.0/13, country:IL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Dec 2019 15:46:45 -0000 > On 18 Dec 2019, at 16:55, Rick Macklem wrote: >=20 > Daniel Braniss wrote: >=20 >> Hi, >> The server with the problems is running FreeBSD 11.1 stable, it was = working fine for >several months, >> but after a software upgrade of our NetAPP server it=E2=80=99s = reporting many lockd errors >and becomes catatonic, >> ... >> Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not = responding >> Dec 18 13:11:45 moo-09 last message repeated 7 times >> Dec 18 13:12:55 moo-09 last message repeated 8 times >> Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is = alive again >> Dec 18 13:13:10 moo-09 last message repeated 8 times >> Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 194 already in queue awaiting acceptance (1 = occurrences) >> Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 193 already in queue awaiting acceptance (3957 = occurrences) >> Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 193 already in queue awaiting acceptance =E2=80=A6= > Seems like their software upgrade didn't improve handling of NLM RPCs? > Appears to be handling RPCs slowly and/or intermittently. Note that no = one > tests it with IPv6, so at least make sure you are still using IPv4 for = the mounts and > try and make sure IP broadcast works between client and Netapp. I = think the NLM > and NSM (rpc.statd) still use IP broadcast sometimes. >=20 we are ipv4 - we have our own class c :-) > Maybe the network guys can suggest more w.r.t. why, but as I've stated = before, > the NLM is a fundamentally broken protocol which was never published = by Sun, > so I suggest you avoid using it if at all possible. well, at the moment the ball is on NetAPP court, and switching to NFSv4 = at the moment is out of the question, it=E2=80=99s a production server used by several thousand students. >=20 > - If the locks don't need to be seen by other clients, you can just = use the "nolockd" > mount option. > or > - If locks need to be seen by other clients, try NFSv4 mounts. Netapp = filers > should support NFSv4.1, which is a much better protocol that = NFSv4.0. >=20 > Good luck with it, rick thanks danny > =E2=80=A6 > any ideas? >=20 > thanks, > danny >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@freebsd.org Wed Dec 18 15:58:39 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 09CDE1E1D13 for ; Wed, 18 Dec 2019 15:58:39 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 47dKRy4VGRz4k4n for ; Wed, 18 Dec 2019 15:58:38 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 976061E1D12; Wed, 18 Dec 2019 15:58:38 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 971811E1D11 for ; Wed, 18 Dec 2019 15:58:38 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47dKRx4hRXz4k4m for ; Wed, 18 Dec 2019 15:58:37 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: by mail-wr1-x435.google.com with SMTP id b6so2881236wrq.0 for ; Wed, 18 Dec 2019 07:58:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=rcCGtHgoWDc8yxmnUqPTMVUKIxwZHpXEEoEb08ieNIs=; b=i2B7q097fzQBMi7GdhUALoeBKBRT9H0PQaOuNoky4FNVXU65iAVJ3ngz2XRBauXg8M PJH+w8n3dmO5bd6o5VQfBc1JwMPHDTGI/8Q1UKM5/LECuCdJBE2QPqfqzLgBRw+xNou3 /4AUu4GcVfZ81et0HtHDSobSxxABMVO7BM0MsmvvGGVqrpd4KRjFQkg+orvt5ZVt+2Z/ hG785CSmz/0uTt2cSs3wuCamJdMnqLb0nKMjwWALWI6fIh1Wk99pzR3keNpLul3D/oBH 6G5xfkG0291LIPlk8qbd6ugdZhsZ4AS/Lgk1lKGQuffnsDcaOQFSGl4nt6VYQwynLVqB z4MQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=rcCGtHgoWDc8yxmnUqPTMVUKIxwZHpXEEoEb08ieNIs=; b=KAJdXOg1l7NI2FFGHqF8K2B5OBdZUDqOPTliYObuJ2jTrBQj0fk68SHcrItf6jufHv hDauM9tSOGHqFq7fkEaL5Kj1gbW7jEdy0exxviWEb+NfARkfbqWEgN+MmHXUkw0WowT6 x+kZxEYcvJuQlpSFxRfIPfVBdzqRrm8hVyzimOfFxjaPprVpae4s5G6fCSpmX++Xt3V9 G32MHAv/ETcgJ8tYQ1KtXNFBIrUNRS0HGStqPbyalqxT1mAvtXu08ACr6/IKJjUjUCxK aLl5lxMXuX7/4sjK3wXP7RTr7jxOXYkR7evG3bVhcprSpkM+0+01roE3cMj6/CH/xUZj dxkA== X-Gm-Message-State: APjAAAV6ZpoveZUT2o2NjJDdWhhQjvky5GCSDrj4OvqTldXJ6dpqrsL9 wzbFOuV7sj4O4FckFQpuPZ4itzymb+/dbfgnA6Bn9g== X-Google-Smtp-Source: APXvYqz/cnxM9P5EGVXGj4Y7NK43w585u25fkkNaJpEB+rwdk2aQH5n17BSJr6/CnfI81ccrMle/A60FEBMUe65MB8o= X-Received: by 2002:a5d:6144:: with SMTP id y4mr3694322wrt.367.1576684715421; Wed, 18 Dec 2019 07:58:35 -0800 (PST) MIME-Version: 1.0 References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> In-Reply-To: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> From: Richard P Mackerras Date: Wed, 18 Dec 2019 15:58:24 +0000 Message-ID: Subject: Re: nfs lockd errors after NetApp software upgrade. To: Daniel Braniss Cc: Rick Macklem , "stable@freebsd.org" X-Rspamd-Queue-Id: 47dKRx4hRXz4k4m X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=i2B7q097; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mack63richard@gmail.com designates 2a00:1450:4864:20::435 as permitted sender) smtp.mailfrom=mack63richard@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[5.3.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.00)[ip: (-9.44), ipnet: 2a00:1450::/32(-2.65), asn: 15169(-1.90), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Dec 2019 15:58:39 -0000 Hi, What software version is the NetApp using? Is the exported volume big? Is the vserver configured for 64bit identifiers? If you enable NFS V4.0 or 4.1 other NFS clients using defaults might mount NFSv4.x unexpectedly after a reboot so you need to watch that. Cheers Richard (NetApp admin) On Wed, 18 Dec 2019 at 15:46, Daniel Braniss wrote: > > > > On 18 Dec 2019, at 16:55, Rick Macklem wrote: > > > > Daniel Braniss wrote: > > > >> Hi, > >> The server with the problems is running FreeBSD 11.1 stable, it was > working fine for >several months, > >> but after a software upgrade of our NetAPP server it=E2=80=99s reporti= ng many > lockd errors >and becomes catatonic, > >> ... > >> Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not > responding > >> Dec 18 13:11:45 moo-09 last message repeated 7 times > >> Dec 18 13:12:55 moo-09 last message repeated 8 times > >> Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is > alive again > >> Dec 18 13:13:10 moo-09 last message repeated 8 times > >> Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: > Listen queue >overflow: 194 already in queue awaiting acceptance (1 > occurrences) > >> Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: > Listen queue >overflow: 193 already in queue awaiting acceptance (3957 > occurrences) > >> Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: > Listen queue >overflow: 193 already in queue awaiting acceptance =E2=80= =A6 > > Seems like their software upgrade didn't improve handling of NLM RPCs? > > Appears to be handling RPCs slowly and/or intermittently. Note that no > one > > tests it with IPv6, so at least make sure you are still using IPv4 for > the mounts and > > try and make sure IP broadcast works between client and Netapp. I think > the NLM > > and NSM (rpc.statd) still use IP broadcast sometimes. > > > we are ipv4 - we have our own class c :-) > > Maybe the network guys can suggest more w.r.t. why, but as I've stated > before, > > the NLM is a fundamentally broken protocol which was never published by > Sun, > > so I suggest you avoid using it if at all possible. > well, at the moment the ball is on NetAPP court, and switching to NFSv4 a= t > the moment is out of the question, it=E2=80=99s > a production server used by several thousand students. > > > > > - If the locks don't need to be seen by other clients, you can just use > the "nolockd" > > mount option. > > or > > - If locks need to be seen by other clients, try NFSv4 mounts. Netapp > filers > > should support NFSv4.1, which is a much better protocol that NFSv4.0. > > > > Good luck with it, rick > thanks > danny > > > =E2=80=A6 > > any ideas? > > > > thanks, > > danny > > > > _______________________________________________ > > freebsd-stable@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.or= g > " > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@freebsd.org Wed Dec 18 16:06:34 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D416F1E20B6 for ; Wed, 18 Dec 2019 16:06:34 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 47dKd63v0kz4kfF for ; Wed, 18 Dec 2019 16:06:34 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: by mailman.nyi.freebsd.org (Postfix) id 83A591E20B5; Wed, 18 Dec 2019 16:06:34 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 836641E20B4 for ; Wed, 18 Dec 2019 16:06:34 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47dKd55Gznz4kfD for ; Wed, 18 Dec 2019 16:06:33 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cs.huji.ac.il; s=57791128; h=References:To:Cc:In-Reply-To:Date:Subject:Mime-Version:Content-Type:Message-Id:From; bh=R2PPf7p/waSs1s183gJiCLcMmSYK2ZhtcMcpuY+A72w=; b=SP2TwWilRQ4s+yV2PuozE7W8tmmfdfpX1/U1VIUyi38x9Wxp+ziKRkfRDr7Lkmdley4gFWEvVqXFVtNueMXgH+B84HFX1s0eVBDcfcV5z2KjAQeTmEcywwalp4W6tfjnOhcaIn4ObP0kD9ooOFqVyTrMNH0EQjFofhAcXqQBXso2DW0kuLkP19MI0vsBJoyv7HEzKpPwjPvS/69YMkiunJ0jHPc+cRWXTNIIMp2k+kjLo1j5LuFVOnT+CPsqVHBMgB3q9NBgRJ2q7gYZ53iARA6nYJJR4GYxkXvq5+E/7/dj35JABV7nWrysdZkEtSqRfkhND1CzE70FSHzznkZ5sg==; Received: from macmini.bk.cs.huji.ac.il ([132.65.179.19]) by kabab.cs.huji.ac.il with esmtp id 1ihbqJ-000F2h-O5; Wed, 18 Dec 2019 18:06:31 +0200 From: Daniel Braniss Message-Id: <06DD810E-3945-40AD-A1A2-578DF4FB1150@cs.huji.ac.il> Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.40.2.2.4\)) Subject: Re: nfs lockd errors after NetApp software upgrade. Date: Wed, 18 Dec 2019 18:06:31 +0200 In-Reply-To: Cc: Rick Macklem , "stable@freebsd.org" To: Richard P Mackerras References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> X-Mailer: Apple Mail (2.3608.40.2.2.4) X-Rspamd-Queue-Id: 47dKd55Gznz4kfD X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.huji.ac.il header.s=57791128 header.b=SP2TwWil; dmarc=pass (policy=none) header.from=huji.ac.il; spf=none (mx1.freebsd.org: domain of danny@cs.huji.ac.il has no SPF policy when checking 132.65.116.210) smtp.mailfrom=danny@cs.huji.ac.il X-Spamd-Result: default: False [-3.55 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[cs.huji.ac.il:s=57791128]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; IP_SCORE(-1.25)[ip: (-2.96), ipnet: 132.64.0.0/13(-1.86), asn: 378(-1.49), country: IL(0.05)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[cs.huji.ac.il:+]; DMARC_POLICY_ALLOW(-0.50)[huji.ac.il,none]; RCVD_IN_DNSWL_NONE(0.00)[210.116.65.132.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:378, ipnet:132.64.0.0/13, country:IL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Dec 2019 16:06:34 -0000 > On 18 Dec 2019, at 17:58, Richard P Mackerras = wrote: >=20 > Hi, > What software version is the NetApp using? the very latest :-), but will try and find out later. > Is the exported volume big? about 500G, but many files as far as I know, only accessed by one host running the web app - = moodle. > Is the vserver configured for 64bit identifiers what the issue here? > ? >=20 > If you enable NFS V4.0 or 4.1 other NFS clients using defaults might = mount NFSv4.x unexpectedly after a reboot so you need to watch that.=20 >=20 > Cheers=20 >=20 > Richard=20 > (NetApp admin) >=20 > On Wed, 18 Dec 2019 at 15:46, Daniel Braniss > wrote: >=20 >=20 > > On 18 Dec 2019, at 16:55, Rick Macklem > wrote: > >=20 > > Daniel Braniss wrote: > >=20 > >> Hi, > >> The server with the problems is running FreeBSD 11.1 stable, it was = working fine for >several months, > >> but after a software upgrade of our NetAPP server it=E2=80=99s = reporting many lockd errors >and becomes catatonic, > >> ... > >> Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not = responding > >> Dec 18 13:11:45 moo-09 last message repeated 7 times > >> Dec 18 13:12:55 moo-09 last message repeated 8 times > >> Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is = alive again > >> Dec 18 13:13:10 moo-09 last message repeated 8 times > >> Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 194 already in queue awaiting acceptance (1 = occurrences) > >> Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 193 already in queue awaiting acceptance (3957 = occurrences) > >> Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 193 already in queue awaiting acceptance =E2=80=A6= > > Seems like their software upgrade didn't improve handling of NLM = RPCs? > > Appears to be handling RPCs slowly and/or intermittently. Note that = no one > > tests it with IPv6, so at least make sure you are still using IPv4 = for the mounts and > > try and make sure IP broadcast works between client and Netapp. I = think the NLM > > and NSM (rpc.statd) still use IP broadcast sometimes. > >=20 > we are ipv4 - we have our own class c :-) > > Maybe the network guys can suggest more w.r.t. why, but as I've = stated before, > > the NLM is a fundamentally broken protocol which was never published = by Sun, > > so I suggest you avoid using it if at all possible. > well, at the moment the ball is on NetAPP court, and switching to = NFSv4 at the moment is out of the question, it=E2=80=99s > a production server used by several thousand students. >=20 > >=20 > > - If the locks don't need to be seen by other clients, you can just = use the "nolockd" > > mount option. > > or > > - If locks need to be seen by other clients, try NFSv4 mounts. = Netapp filers > > should support NFSv4.1, which is a much better protocol that = NFSv4.0. > >=20 > > Good luck with it, rick > thanks > danny >=20 > > =E2=80=A6 > > any ideas? > >=20 > > thanks, > > danny > >=20 > > _______________________________________________ > > freebsd-stable@freebsd.org = mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-stable = > > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org = " >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing = list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable = > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org = " From owner-freebsd-stable@freebsd.org Wed Dec 18 16:22:23 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 79B4C1E2850 for ; Wed, 18 Dec 2019 16:22:23 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [104.236.120.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47dKzL3z81z4lcT for ; Wed, 18 Dec 2019 16:22:22 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id A134E2110E1 for ; Wed, 18 Dec 2019 11:22:15 -0500 (EST) Received: from [192.168.10.25] (D15.Denninger.Net [192.168.10.25]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id 5776E1F157B for ; Wed, 18 Dec 2019 10:22:15 -0600 (CST) To: freebsd-stable@freebsd.org From: Karl Denninger Subject: ZFS and power management Autocrypt: addr=karl@denninger.net; prefer-encrypt=mutual; keydata= xsFNBF1Rd+gBEACmLAH7SAzdQq57ZN56QQEy0jDFfH5BvGOMZgCaP+Y5lJQ5u9WphCoCALMs Rg0o1Q9DRNWgUmy/cgsxioXAEzZFXXzOHPJhwplVOgfjxnoByD5KQhWG8Owm9QmATdtiZPSV 4UYVNUIbZv7btSnnAXysG2OUHajYS5PVeFQxFbhNFq/SS8VaXr1WEVTFa8NFKp2W3/KY1A+U KKDUlYwnOauK3fnY9chF2IRSoxAbBJFrJ4lPGz04HtzNos4Q9CBfTphKcdFjcPntNS9wrqs3 sm+7hLNTH9B2Kj6aekG5UhD03eyP+gevTgBy51RL6ULzI13Kc4aeyOByuBXrA8D2m2Ee67iy 4+ZSxM9Wn1gQce5624OWzCYIGBH2r75Bshp1KHKu36N2rN//kyKYnwl/z6UZB/S9cMUFKZgL gFx7QxpFX/HvSiBcPfcGS0meModpg6qma7/2jRoQAXacslpiT+uOfRGspNbnglkbw435RzX/ kMUclJQNZBBBUpPiGjVCjeBTiAfN8TyjS+pWzwxNCUZWbYO5xVaS0gbIhgVNoBOGn1rdTsdA PP65SRjaoL5KY6bzkkzrXLB2Djx8/p4vr0qIqxIQWbewJq3xKyKGiqI46ae77BF7k0B++Ndx g9K9UeWKl/iJ0eoI0ftR+xH3aIHTU1Or3j/tj4j8Z0tnVSyt1wARAQABzSNLYXJsIERlbm5p bmdlciA8a2FybEBkZW5uaW5nZXIubmV0PsLBfwQTAQgAKQUCXVF36AIbIwUJCWYBgAcLCQgH AwIBBhUIAgkKCwQWAgMBAh4BAheAAAoJEG8twBXrj1l4swkP/3uOzRxW16K6H4JIEIRMUEbt nxDhmk+gR/7H9phg7HtvR7i22QejZX1N1NHcGRNmBwLshWVjJkHKhCE/AM8Cf9XyaV2ft6qn g1xK6NuhapxVuaaMeCVPUzsPkTcR+JMl72ZR4Q+mJMVQButCITekmr7aIzIZ80fF0t86rnq+ O74ZGt0SAMsLV/GAKlIw8fGMi9Xj4OKDgqmxTnIoV4+0mpo26W957pnlOrjN3/6VqWUyAdHH DkyqsuP/9jx2f5pZCcD7X04+93GI+sGb1s6BOFRHq2oJgs6W0z0nPx5Ks9MDDgSQlxXAryje 17WphTR7DWn1BeF3Y8AhRkzc2+Mgc5s1i2fPe6YwvksDNOEyNXIvFV7chwDQYb0Q3I8XsoHu 2WUjXp0kVokobJPdVdY55nbY+brezweRJMiEpFtGOmoUekQWlI5KS1kE8+Xuqpm+MSxEpqY8 5ncPt0lekOrICGajlOotkUK86iVemlW1rMzMc5Xwp9j8oxa+bRtGD6u1rYz4i+qIdE+GSCBy 1nnHN/my0nefhQyHXr8wGVEbyiMZCten9fm1iXpBr0jY+tvtbo8XqZQG7Lr+3kSO6VUgc8kW IPf2HxIV7AnGUN+ddZGCcPPhb2mY/Yy7si54wJFj6YoG+/+rNjF9F5d8WeLoeUWczgHTvZmS o6F7UhjjuwzgzsFNBF1Rd+gBEADNVFS8nQ+kpKOpgtP+f3bCVxHAm7eHMbX6oew5yZiQwfD+ 1RWNWLVOMeTt7G2e5HsHpJOUwFUJhbDb0omB0r38xTSVSAig9kmUfb7tTMJG2bG7WfWykBOM WIZ4OhCf+ISv9dUkjNgx4ionWotFxwDiPRwWumVQ7WYZmRZlhDWMiaHgKvBrjJ7Y6GKPRbQc 5/0Qz9xGhXKlFxDQrrSMkyRThIOxXqdfD9z3rEsV3ZwOojzNsnkIImnQMKyIAR0FBQop34G9 wDQi7fxk8wGIfDszwfR4oAdDdPGq4gcAvE7Fd3xKyNpGyjSED5szoaFjldaZSXQIffquSUvy sFCTTLRIso5Dn9uQgi57gIv+5mnyKBfm2Z2P6pEQPSt073TED9rS0+JpniJL7rKRVpO5niqw sQJS6ht+JF88rXro+SiwxD/KeDpTuuJ10+ohLVi1Y+X82X7BIQEhqtFp9FVJSds4o/eNyaHd SoqfoeWMy3EV+rdJ3DneXcPS1BgxO57Rko5Hx3NUSVK83ovFb+Ofes9SLNdqNu3xAUcfpRdS DyxzpVbCq6Y2CIojiaweiYe5BOBhmR9OPGhqP8YD7GukYmQufAVuOrIVyctBlVPHgMBb+UX+ ItYXuX4weSJWLOsmM45xd/EYvBq2DWFpKlyihoktNzTGqxGsNeG7gCOEUTAnUwARAQABwsFl BBgBCAAPBQJdUXfoAhsMBQkJZgGAAAoJEG8twBXrj1l4Dm0P/iEx2gIHSOnvgpG799Vf2RM0 7gPbDWzDaw8YTV49H+VTOqq7RlT52aO0QfNAmtppX0V1/5f30fuSCF46NWnYGu35P/LvOAPb sLbeWCyJy4GOPN4cjsBMbgmooGdl24RdcvGMmY177o7oOSWBqXfhAj+YA6r+hEar1qxqLgwB Gy8wAId4qYSQhN/FxiQbyUs2tPAI6Wn/41pI7Hu6WgmRGpZrBv8HhVV9Gl7jallSsS/g+fhu WRbDKCknUS5SX3+w2AUFr4kf62gSSxXBxd075KnViV9c0sraAPI31XbM5QUc0Xssfaqs6Srr z4MjKaLhb7GD8C1JwI23PuGdFvk9WK996UvIyjdWIE99VSlg/5gEKkXzwx7oysrSG9BqkfGf I4addK55xRQPul0V3s2LtDoQTxg3VHrL6wrvGhYUcTHLmlsvNx1EOb5a3xBT+SUK/Ltq08LW YcmNbU/G217MlfvDJYHCb0uOtxqJFm8RiZGj2eEcLgvyWnlWCD2rfP4EqCxmpr3Ic725FiQR cBbdTV3clTgclhBG3TA9dxVjfZDcatz5cFBwXP8k5Yn9tNl90T2r79V4SNh1mCHtGTSEf449 qz9tm7EguLchjmoirJTuiipZKcalcHAHtz4VPUykdXsrfEJTzdEcujzqF6v/9CY+DjpAd3et Z0vw7xC5tS+b Message-ID: <57da15d4-0944-982b-7d7e-d7b2571e869c@denninger.net> Date: Wed, 18 Dec 2019 10:22:16 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.3.0 MIME-Version: 1.0 Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms000200040008000805060605" X-Rspamd-Queue-Id: 47dKzL3z81z4lcT X-Spamd-Bar: ------- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=denninger.net; spf=pass (mx1.freebsd.org: domain of karl@denninger.net designates 104.236.120.189 as permitted sender) smtp.mailfrom=karl@denninger.net X-Spamd-Result: default: False [-7.40 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; HAS_ATTACHMENT(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[denninger.net,none]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~]; ASN(0.00)[asn:14061, ipnet:104.236.64.0/18, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-2.50)[ip: (-9.84), ipnet: 104.236.64.0/18(-4.39), asn: 14061(1.78), country: US(-0.05)]; RECEIVED_SPAMHAUS_PBL(0.00)[197.57.1.68.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11] X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Dec 2019 16:22:23 -0000 This is a cryptographically signed message in MIME format. --------------ms000200040008000805060605 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable I'm curious if anyone has come up with a way to do this... I have a system here that has two pools -- one comprised of SSD disks that are the "most commonly used" things including user home directories and mailboxes, and another that is comprised of very large things that are far less-commonly used (e.g. video data files, media, build environments for various devices, etc.) The second pool has perhaps two dozen filesystems that are mounted, but again, rarely accessed.=C2=A0 However, despite them being rarely accessed= ZFS performs various maintenance checkpoint functions on a nearly-continuous basis (it appears) because there's a low level, but not zero, amount of I/O traffic to and from them.=C2=A0 Thus if I set power control (e.g. spi= n down after 5 minutes of inactivity) they never do.=C2=A0 I could simply export the pool but I prefer (greatly) to not do that because some of the data on that pool (e.g. backups from PCs) is information that if a user wants to get to it it ought to "just work." Well, one disk is no big deal.=C2=A0 A rack full of them is another matte= r.=C2=A0 I could materially cut the power consumption of this box down (likely by a third or more) if those disks were spun down during 95% of the time the box is up, but with the "standard" way ZFS does things that doesn't appear to be possible. Has anyone taken a crack at changing the paradigm (e.g. using the automounter, perhaps?) to get around this? --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms000200040008000805060605 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkxMjE4MTYyMjE2 WjBPBgkqhkiG9w0BCQQxQgRA2eV7Xa1FIcs8Fg/5RcSLeo84CrSY20BBBomke8WIE6beTetd XUk++dU4gEoe7Nl2+c0srBPIf8regPcaF+VeYDBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgCoWSwojEQcBkRejVqDs0F0c6+8IJk9TIYOyr/SXq45lGeDyZVJInEplBUGXG7z+z4j lses3d/By0PHv6xjfi0GaFSL2dOP9LCN2N7MR16EL/zi8/3eJ488zPSAdYSCR8JvgfsgRDNy tL9o9+w7U6xmICDD05Kl7pbhNj0RqapQk8FF45hMlu2gGfbD6UOmjonSDnW4sFMmAvVcHu31 y8YIkaqxuFx6h/tuxVbiG3GDDL2wPTyF7AX7EagV8Y6sTEFoMTEEc0MCxZWKejeZbZdM5MqZ PVMgADH4sgnSVmNjLWAv/AmCsGaISvU4V63O6SC6xhVkp1YhZyS+Wn5qPn8LUwhu4OHSL9M3 Tz7Yyi/2TQ4vPigo5o/nj6PypM/emyzbJZ7nI7LcU+pAjqybDkSwj+BKY9bxr5WLdI/SkPiO wXPaOL9SqtLa5dt+UYWZcULqh/w2cD1IgwYuRxchIUicPvYvuQxW6ZAX9859YWgVEJGLLL8T R/b+e0MuOsW5y43td4nvxzhWEq2mi7/wiSKDm2uLe2mVeQL+fd+717Jtymsk80FjurvFb21k dmCDNNZophRmuszfx0azrteWW1Lh1gabyPpNY/AE2kOW9skfZo1aVrPLdq0VhA2yzUbxWzZ2 Pti5esdq5Y9JA9o8fLgw42KTaDgI6SZpfWSrPNXERgAAAAAAAA== --------------ms000200040008000805060605-- From owner-freebsd-stable@freebsd.org Wed Dec 18 16:31:43 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7EF8C1E2B73 for ; Wed, 18 Dec 2019 16:31:43 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f42.google.com (mail-ot1-f42.google.com [209.85.210.42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47dLB651Q3z4mBr for ; Wed, 18 Dec 2019 16:31:42 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f42.google.com with SMTP id p8so3156090oth.10 for ; Wed, 18 Dec 2019 08:31:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=1/1OuL9VgkeMBhPQB7QzUBWGrZOYhHDLv4u/JFVe7V0=; b=V9aM4qwoH8Wx6q3sDns9BKxSynv65a88kFOKYLlroasO98h1XUn9WkSpbdBnOy6ODR JloVBHNtGea7VcibUFDuoUh2c52lx40dwydBwdS8usCkDem4OAM2aV+TQxWDie7yyZ3w 5AoG9loruIoLCyJjc/C4wA3OY4ZTCdwnZQbqL6YVxd6BlPCc3OD4BVnFjvKR8HzeeoE2 k52gXbFRNq9AJlX0TuUJU7wg+RGpv5gquMYvDIujNP8QAZiSz3sSIp5YUGbtmb3/UNzC xK+xiPrgVBUCkf6dHFCbP8L3vh5dH3fI0FStxSicsq1STZYk6DqLLXmxiNOrgueMKlAa gpFg== X-Gm-Message-State: APjAAAXUKClOLTrerQl5igLNoYaMF5oeSr/Se6CzcJljD1pxXlm6e9+7 7WbWzc2+4AJjfVnzh3qhHrLNvya9jVxHUeNBpmnD+Q== X-Google-Smtp-Source: APXvYqw4RpSmkgWITBLg9DedngsEKrAumdTkC8Pv5eWUzcYTI1buyhZ+UuBYa5xSP9mZXeumJvTCN0PmCqc8I55fmww= X-Received: by 2002:a05:6830:12cf:: with SMTP id a15mr3529879otq.222.1576686700948; Wed, 18 Dec 2019 08:31:40 -0800 (PST) MIME-Version: 1.0 References: <57da15d4-0944-982b-7d7e-d7b2571e869c@denninger.net> In-Reply-To: <57da15d4-0944-982b-7d7e-d7b2571e869c@denninger.net> From: Alan Somers Date: Wed, 18 Dec 2019 09:31:29 -0700 Message-ID: Subject: Re: ZFS and power management To: Karl Denninger Cc: FreeBSD X-Rspamd-Queue-Id: 47dLB651Q3z4mBr X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.210.42 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-3.07 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[42.210.85.209.rep.mailspike.net : 127.0.0.18]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[42.210.85.209.list.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-1.07)[ip: (-0.28), ipnet: 209.85.128.0/17(-3.12), asn: 15169(-1.90), country: US(-0.05)]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Dec 2019 16:31:43 -0000 On Wed, Dec 18, 2019 at 9:22 AM Karl Denninger wrote: > I'm curious if anyone has come up with a way to do this... > > I have a system here that has two pools -- one comprised of SSD disks > that are the "most commonly used" things including user home directories > and mailboxes, and another that is comprised of very large things that > are far less-commonly used (e.g. video data files, media, build > environments for various devices, etc.) > > The second pool has perhaps two dozen filesystems that are mounted, but > again, rarely accessed. However, despite them being rarely accessed ZFS > performs various maintenance checkpoint functions on a nearly-continuous > basis (it appears) because there's a low level, but not zero, amount of > I/O traffic to and from them. Thus if I set power control (e.g. spin > down after 5 minutes of inactivity) they never do. I could simply > export the pool but I prefer (greatly) to not do that because some of > the data on that pool (e.g. backups from PCs) is information that if a > user wants to get to it it ought to "just work." > > Well, one disk is no big deal. A rack full of them is another matter. > I could materially cut the power consumption of this box down (likely by > a third or more) if those disks were spun down during 95% of the time > the box is up, but with the "standard" way ZFS does things that doesn't > appear to be possible. > > Has anyone taken a crack at changing the paradigm (e.g. using the > automounter, perhaps?) to get around this? > > -- > Karl Denninger > karl@denninger.net > /The Market Ticker/ > /[S/MIME encrypted email preferred]/ > I have, and I found that it wasn't actually ZFS's fault. By itself ZFS wasn't initiating any background I/O whatsoever. I used a combination of fstat and dtrace to track down the culprit processes. Once I had shutdown/patched/reconfigured each of those processes, the disks stayed idle indefinitely. You might have success using the same strategy. I suspect that the automounter wouldn't help you, because any access that ought to "just work" for a normal user would likewise "just work" for whatever background process is hitting your disks right now. -Alan From owner-freebsd-stable@freebsd.org Wed Dec 18 23:54:29 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 33C131C9895 for ; Wed, 18 Dec 2019 23:54:29 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 47dX106KJRz3QJt for ; Wed, 18 Dec 2019 23:54:28 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id D4B7F1C9894; Wed, 18 Dec 2019 23:54:28 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D36081C9892 for ; Wed, 18 Dec 2019 23:54:28 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47dX0z4KhQz3QJs for ; Wed, 18 Dec 2019 23:54:27 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: by mail-wm1-x333.google.com with SMTP id p17so3803063wma.1 for ; Wed, 18 Dec 2019 15:54:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=EDcV6cBC2gY9Ob+46cXAsCGmAXxCReFWujRTgInKfXw=; b=UCdeZ60JxkzustPnqUXZpQmbn4xbk05iKEYQCotBDgdZxEg51Z7VjJgS7EucLhV27Y XyiG8GCTkrCPUcg6ynGUlq/o3qTI5CeGT/NrOmlQeQwo2rJeQ3nvdkPk6rlNqeHx55am koelfax+cGP2zZf6iSIfJvIZGo1bRETqv8UC1KRhDi2YmA3Wna3zYYjXElct/35ItMhl R405s6/7c5u9/qalTf3inIYoSpjUuQocu+QcTIqEOTjD8Soyh4bwwmIjN6XYvCkQ40nO NSnmFYpETt58URExG6FNdZtdZbuAIJQ8kjknIRBKH7nhE3Z4zGpwJIGph5ETzmfCbgz2 Pw6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=EDcV6cBC2gY9Ob+46cXAsCGmAXxCReFWujRTgInKfXw=; b=Qm6FX3HCPdOSqrI5KRmidJ7ZYnOLjFfdsR8wQSfDfnYD5TJhuCInd++e6b9HV39eD5 N/Gz5OPT+rAic09PHI5zyoM/Pm7RVpJHM+vGncYB2cV5v25SauDif0h5P51DiHnSGRI+ Uou8HcgfudrggCNrHuRn6gph/pmFRerq9xZedyyGE7oNBoLTeTLcZ/s+GHa5EA6QXOp/ BEAGzrmawWUpClTaNI49plRhSAUTSS/IwDUcpN8S+twvMo9gnzkwO1TnK1fZtB2rvG3u EomdIjJwZr4F1wzA6M8WDmjwQWO3PRyS9FOVj4a7HZm6vreIVtrzjGG9i1bTWUtGSfRP s7Jg== X-Gm-Message-State: APjAAAWtd4/y3qUK7/TDk+jrjo0Mk5j5vrz/imqo2GzkRr4F29gFVoKN TEsMvFF+IMJNEbRNmdEHpKavnVQwBDT5O7LKXzs= X-Google-Smtp-Source: APXvYqziKYct/ppkF0vbneE6biNQYW9xmwdCaH5ORzNBhQ3+OtcchJnTB1MvoG8EoP0EIPZb/LATDI7/HFgDxZfCbBE= X-Received: by 2002:a7b:c750:: with SMTP id w16mr6698873wmk.46.1576713265231; Wed, 18 Dec 2019 15:54:25 -0800 (PST) MIME-Version: 1.0 References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <06DD810E-3945-40AD-A1A2-578DF4FB1150@cs.huji.ac.il> In-Reply-To: <06DD810E-3945-40AD-A1A2-578DF4FB1150@cs.huji.ac.il> From: Richard P Mackerras Date: Wed, 18 Dec 2019 23:54:14 +0000 Message-ID: Subject: Re: nfs lockd errors after NetApp software upgrade. To: Daniel Braniss Cc: Rick Macklem , "stable@freebsd.org" X-Rspamd-Queue-Id: 47dX0z4KhQz3QJs X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=UCdeZ60J; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mack63richard@gmail.com designates 2a00:1450:4864:20::333 as permitted sender) smtp.mailfrom=mack63richard@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[3.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.00)[ip: (-8.71), ipnet: 2a00:1450::/32(-2.65), asn: 15169(-1.90), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Dec 2019 23:54:29 -0000 Hi, I=E2=80=99m sure the 64 bit identifiers isn=E2=80=99t an issue. Your export= isn=E2=80=99t vast. I assume you have restarted statd and lockd on FreeBSD. I did search on the NetApp site earlier and nothing lept out then. Sorry, Richard On Wed, 18 Dec 2019 at 16:06, Daniel Braniss wrote: > > > On 18 Dec 2019, at 17:58, Richard P Mackerras > wrote: > > Hi, > What software version is the NetApp using? > > the very latest :-), but will try and find out later. > > Is the exported volume big? > > about 500G, but many files > as far as I know, only accessed by one host running the web app - moodle. > > Is the vserver configured for 64bit identifiers > > what the issue here? > > ? > > If you enable NFS V4.0 or 4.1 other NFS clients using defaults might moun= t > NFSv4.x unexpectedly after a reboot so you need to watch that. > > Cheers > > Richard > (NetApp admin) > > On Wed, 18 Dec 2019 at 15:46, Daniel Braniss wrote: > >> >> >> > On 18 Dec 2019, at 16:55, Rick Macklem wrote: >> > >> > Daniel Braniss wrote: >> > >> >> Hi, >> >> The server with the problems is running FreeBSD 11.1 stable, it was >> working fine for >several months, >> >> but after a software upgrade of our NetAPP server it=E2=80=99s report= ing many >> lockd errors >and becomes catatonic, >> >> ... >> >> Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not >> responding >> >> Dec 18 13:11:45 moo-09 last message repeated 7 times >> >> Dec 18 13:12:55 moo-09 last message repeated 8 times >> >> Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is >> alive again >> >> Dec 18 13:13:10 moo-09 last message repeated 8 times >> >> Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: >> Listen queue >overflow: 194 already in queue awaiting acceptance (1 >> occurrences) >> >> Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: >> Listen queue >overflow: 193 already in queue awaiting acceptance (3957 >> occurrences) >> >> Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: >> Listen queue >overflow: 193 already in queue awaiting acceptance =E2=80= =A6 >> > Seems like their software upgrade didn't improve handling of NLM RPCs? >> > Appears to be handling RPCs slowly and/or intermittently. Note that no >> one >> > tests it with IPv6, so at least make sure you are still using IPv4 for >> the mounts and >> > try and make sure IP broadcast works between client and Netapp. I thin= k >> the NLM >> > and NSM (rpc.statd) still use IP broadcast sometimes. >> > >> we are ipv4 - we have our own class c :-) >> > Maybe the network guys can suggest more w.r.t. why, but as I've stated >> before, >> > the NLM is a fundamentally broken protocol which was never published b= y >> Sun, >> > so I suggest you avoid using it if at all possible. >> well, at the moment the ball is on NetAPP court, and switching to NFSv4 >> at the moment is out of the question, it=E2=80=99s >> a production server used by several thousand students. >> >> > >> > - If the locks don't need to be seen by other clients, you can just us= e >> the "nolockd" >> > mount option. >> > or >> > - If locks need to be seen by other clients, try NFSv4 mounts. Netapp >> filers >> > should support NFSv4.1, which is a much better protocol that NFSv4.0= . >> > >> > Good luck with it, rick >> thanks >> danny >> >> > =E2=80=A6 >> > any ideas? >> > >> > thanks, >> > danny >> > >> > _______________________________________________ >> > freebsd-stable@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> > To unsubscribe, send any mail to " >> freebsd-stable-unsubscribe@freebsd.org" >> >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org= " >> > > From owner-freebsd-stable@freebsd.org Thu Dec 19 00:22:56 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 731B01CAAD0 for ; Thu, 19 Dec 2019 00:22:56 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 47dXdr0Xd4z3wxv for ; Thu, 19 Dec 2019 00:22:56 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: by mailman.nyi.freebsd.org (Postfix) id 126C21CAACF; Thu, 19 Dec 2019 00:22:56 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 122A21CAACE for ; Thu, 19 Dec 2019 00:22:56 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670044.outbound.protection.outlook.com [40.107.67.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47dXdp1nJdz3wxt for ; Thu, 19 Dec 2019 00:22:53 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mMZdbdlqHPDIDoGpBrJRzjOtbIphzO4roTVOQMjJ98zNeun/9QuK/Sv4JM9FhaRI1sMoDdgo57lChAKec76132f1aGtDzFMTWEwwbdU/4t6mYx9HKIn+4p8cyjvH+cnFZrrTCqIJP66fLj+PfqPOjpZNM/5osOqplZrokqlktERjrIPMvJ9WBVtO3Ot7s5LEtywFxm1Tsz8X621Ee2t5uPv8fUeUi4WNi5c8XXGtWuIIPvoHJzZ5xMwxvd95LWZVyJonfDiRAgCKrLzklhmXKsinXI5qPDLqnVAwgUhIHUldWfyU0/bGyE++BprfDp7w3jkLdPZlLHMTE3AAwV3l1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3cSr3J7q4zdTHJlmr91dtI1CVJST62X4ZjuO8uH6xPE=; b=fr1MujSRVcg0XY56Bm4ubiGFBSNsCL1yMaFJ4gfA3IggHt3Pk3jArQiJlQR8qxc9+k/+DmABYJxTCLU0KQDyEtDFIK1V+iyJOyP9R9a8+o+jw6QPgAn8d/j5aeHrJOqYbdMKR8VxInJlNuw1iZvkx0vRAerP4gPmFv2f8NXtekYILdQTU85TsqLyk+xo9LUUYbXdDj+RUGD+yz1pwZ8JsYCdZoEMtg3Fn5AIuxxA8SB6IVPLmu/qElBpsB5/1K3UXa9ReFnImC2/DtPB5pFa2JhDdj+X8K37HBUFg6iam9lrjFZnqbg+8EYXEek4kqe/skFLLgFkpOxOug4ThqkkJg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM (52.132.69.153) by YQBPR0101MB1332.CANPRD01.PROD.OUTLOOK.COM (52.132.68.141) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2538.16; Thu, 19 Dec 2019 00:22:52 +0000 Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75]) by YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75%5]) with mapi id 15.20.2538.019; Thu, 19 Dec 2019 00:22:52 +0000 From: Rick Macklem To: Richard P Mackerras , Daniel Braniss CC: "stable@freebsd.org" Subject: Re: nfs lockd errors after NetApp software upgrade. Thread-Topic: nfs lockd errors after NetApp software upgrade. Thread-Index: AQHVtawq+ga5QLcdVkqBDG/GW9zFg6e/+Am+gAARTACAAANHAIAAi7Y3 Date: Thu, 19 Dec 2019 00:22:52 +0000 Message-ID: References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 672e2b26-d2c6-45db-8aff-08d7841995f1 x-ms-traffictypediagnostic: YQBPR0101MB1332: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 0256C18696 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(136003)(39860400002)(396003)(376002)(346002)(199004)(189003)(52536014)(26005)(66946007)(4326008)(53546011)(966005)(86362001)(2906002)(6506007)(316002)(478600001)(110136005)(786003)(71200400001)(186003)(33656002)(81166006)(7696005)(8936002)(81156014)(66446008)(64756008)(66556008)(66476007)(55016002)(9686003)(5660300002)(8676002)(76116006); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB1332; H:YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 2xzC5CR7NHJDgvR/iqpyjWe/FGZj9/WePKhqWpsVAX7VtFZsUUvQ8Fm2cxYHYADGSRfLxeoI3RCu7QkmOG8v4j56gqK6pJHphDBr35OdtM1MDYXC/Jg56N7pMvQM13BkL2sECglAXIJ5xIxLbHmVSM1SUd0oHyN7g09tM5IHPy8uCjJDl0CRxlUxOJamYDfAVw8KWuOivTynReP4bly2l9m1OJpHBdYvNNfWiyemXlophKRNuK0dUjcD7esv1zCEv86zqRKCsQLewMureefuFX7KCtSrlVTN3kJ1M/7Bd2uCUs1rCwUmY6D0RR4nQ9ZH56tF2K5WDlZc+P4qhGfp4jjWmEjxoO1x5q9taSnq5Hxbgvk6bWeYArPxtMWLEt37fuPlhxq3M9J0KKGv5O+jpUDkkjGG1QQPL500PeH44MIcWVdYQaevWdd3DfiD6ivaau9elxS2BCmyHrII0HuLKAhayPnsKHheP8rWGlZZSLo= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 672e2b26-d2c6-45db-8aff-08d7841995f1 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Dec 2019 00:22:52.0288 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: JwzlVuNveRiyM++32qPL5MaHxJ4lMNMNSEuQ1NyO0ApVxUTkJg4cxx+1Q2fWUm6GF/wLNAzKxl57920Lhmglvw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1332 X-Rspamd-Queue-Id: 47dXdp1nJdz3wxt X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.44 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.66 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[uoguelph.ca]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[44.67.107.40.list.dnswl.org : 127.0.3.0]; IP_SCORE(-1.36)[ipnet: 40.64.0.0/10(-3.83), asn: 8075(-2.91), country: US(-0.05)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2019 00:22:56 -0000 Richard P Mackerras wrote:=0A= =0A= >Hi,=0A= >What software version is the NetApp using?=0A= >Is the exported volume big?=0A= >Is the vserver configured for 64bit identifiers?=0A= >=0A= >If you enable NFS V4.0 or 4.1 other NFS clients using defaults might mount= NFSv4.x >unexpectedly after a reboot so you need to watch that.=0A= The FreeBSD client always uses NFSv3 mounts by default. To get NFSv4 you mu= st=0A= explicitly specify the "nfsv4" or "vers=3D4" mount option. For NFSv4.1, you= must=0A= also specify "minorversion=3D1".=0A= =0A= The Linux distros I am familiar with will use the highest NFS version suppo= rted by=0A= the server by default. (I suspect some are using NFSv4.1 without realizing = it,=0A= which isn't necessarily bad.)=0A= =0A= nfsstat -m=0A= will show you which version is actually in use for both FreeBSD and Linux.= =0A= =0A= rick=0A= =0A= Cheers=0A= =0A= Richard=0A= (NetApp admin)=0A= =0A= On Wed, 18 Dec 2019 at 15:46, Daniel Braniss > wrote:=0A= =0A= =0A= > On 18 Dec 2019, at 16:55, Rick Macklem > wrote:=0A= >=0A= > Daniel Braniss wrote:=0A= >=0A= >> Hi,=0A= >> The server with the problems is running FreeBSD 11.1 stable, it was work= ing fine for >several months,=0A= >> but after a software upgrade of our NetAPP server it=92s reporting many = lockd errors >and becomes catatonic,=0A= >> ...=0A= >> Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not resp= onding=0A= >> Dec 18 13:11:45 moo-09 last message repeated 7 times=0A= >> Dec 18 13:12:55 moo-09 last message repeated 8 times=0A= >> Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is alive= again=0A= >> Dec 18 13:13:10 moo-09 last message repeated 8 times=0A= >> Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen= queue >overflow: 194 already in queue awaiting acceptance (1 occurrences)= =0A= >> Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen= queue >overflow: 193 already in queue awaiting acceptance (3957 occurrence= s)=0A= >> Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen= queue >overflow: 193 already in queue awaiting acceptance =85=0A= > Seems like their software upgrade didn't improve handling of NLM RPCs?=0A= > Appears to be handling RPCs slowly and/or intermittently. Note that no on= e=0A= > tests it with IPv6, so at least make sure you are still using IPv4 for th= e mounts and=0A= > try and make sure IP broadcast works between client and Netapp. I think t= he NLM=0A= > and NSM (rpc.statd) still use IP broadcast sometimes.=0A= >=0A= we are ipv4 - we have our own class c :-)=0A= > Maybe the network guys can suggest more w.r.t. why, but as I've stated be= fore,=0A= > the NLM is a fundamentally broken protocol which was never published by S= un,=0A= > so I suggest you avoid using it if at all possible.=0A= well, at the moment the ball is on NetAPP court, and switching to NFSv4 at = the moment is out of the question, it=92s=0A= a production server used by several thousand students.=0A= =0A= >=0A= > - If the locks don't need to be seen by other clients, you can just use t= he "nolockd"=0A= > mount option.=0A= > or=0A= > - If locks need to be seen by other clients, try NFSv4 mounts. Netapp fil= ers=0A= > should support NFSv4.1, which is a much better protocol that NFSv4.0.= =0A= >=0A= > Good luck with it, rick=0A= thanks=0A= danny=0A= =0A= > =85=0A= > any ideas?=0A= >=0A= > thanks,=0A= > danny=0A= >=0A= > _______________________________________________=0A= > freebsd-stable@freebsd.org mailing lis= t=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org<= mailto:freebsd-stable-unsubscribe@freebsd.org>"=0A= =0A= _______________________________________________=0A= freebsd-stable@freebsd.org mailing list= =0A= https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"=0A= From owner-freebsd-stable@freebsd.org Thu Dec 19 04:21:56 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6F1611D4327 for ; Thu, 19 Dec 2019 04:21:56 +0000 (UTC) (envelope-from ggm@algebras.org) Received: from mail-io1-xd35.google.com (mail-io1-xd35.google.com [IPv6:2607:f8b0:4864:20::d35]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47ddxb1Kjpz4DTq for ; Thu, 19 Dec 2019 04:21:54 +0000 (UTC) (envelope-from ggm@algebras.org) Received: by mail-io1-xd35.google.com with SMTP id z193so4375685iof.1 for ; Wed, 18 Dec 2019 20:21:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=algebras-org.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=1W/N+HJcT4V4Dc1X9YAbZ9q3rXSUR/H0McoS9RUKAXU=; b=PfuAbhihQvv9Ny6o6Pys/dCwSAD1tlLGkV3zCD6909W/M/YriNGaH+q5UBd36aUNgS meMVG5ypCTZbh2/SM878gr37i/tq/u/W0N5C5kfaBM/q+8qdJMtQQ5TJxl2tL6zdI89W YQKKK11FgDMuiOYmjw4mxJ7wFQifY/wLey0sVPDwgxWRLI2ayD3CuC0svvtv/XeHsJaT t/V/gTi5ajj4l0nncGTQDh8PjOOKpgvF8G0qlhfITWzksYiZGN89mDxKF3YXoXHHMiuH 6SOsAWyCKXE3CUbF3Pgah4+qxJRNatJ9+oU0MB1zCx7c9zzg+JW4HDoxF/Hk0Ncf0sAq PRRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=1W/N+HJcT4V4Dc1X9YAbZ9q3rXSUR/H0McoS9RUKAXU=; b=fxG81Xx+7XM9ftKqLcsr91iq6G50dv3KMSlYcvGEjcpKArAI37u3cBthBEMjOV6LgR sXwKNp2p+PJnwTDOcONe2mzfW8Vft9dUoX4qWtZ0xxgNGmTL0hKwTwyIWUWnnn9ztnJK NYvgVkQAv0WdPvSKgixYSs/5zlKlGFaNrib0rfE8sVZiXDFv99IGmOHVgfZQHMrMxy9o rdx7FJsoNqxfJEW49kpzzsU+6aCko7Q4Mebz4MF5HzOTJsCV2zLbETl6UroW0s0IzUfm r3kYCVRWYfVhommgc4JfYC19K4byqcGk8kXPgpVYdlDLPebrOtXyBC9NjlobtPa0x1MH /M4w== X-Gm-Message-State: APjAAAVHnWgjHKI0mVeOQmU1ZZSbj2QAAbAvS6/vfoxwg4gTr2FDhwt+ aSemiR42Rj2ecE1IqTFDrv9Nu+efjyOfS2yHNRRSuwdJ3l8= X-Google-Smtp-Source: APXvYqxdeNBUJpEu0z0rcj8oy4aTmlewFzQ9fdu3BwM1UPs3009oG1sPjZvCuHDpxW8neaTkHhymw17lQ+k0VlGxvDM= X-Received: by 2002:a6b:5801:: with SMTP id m1mr4110899iob.65.1576729312695; Wed, 18 Dec 2019 20:21:52 -0800 (PST) MIME-Version: 1.0 From: George Michaelson Date: Thu, 19 Dec 2019 14:21:41 +1000 Message-ID: Subject: 12.1, Dell 740 and the M2 SSD as boot media doesn't work To: FreeBSD Stable Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 47ddxb1Kjpz4DTq X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=algebras-org.20150623.gappssmtp.com header.s=20150623 header.b=PfuAbhih; dmarc=none; spf=pass (mx1.freebsd.org: domain of ggm@algebras.org designates 2607:f8b0:4864:20::d35 as permitted sender) smtp.mailfrom=ggm@algebras.org X-Spamd-Result: default: False [-4.66 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[algebras-org.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[algebras.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[algebras-org.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[5.3.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-2.16)[ip: (-6.64), ipnet: 2607:f8b0::/32(-2.19), asn: 15169(-1.90), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2019 04:21:56 -0000 I tried an install on a new Dell 740, which has dual M2 256GB internal SSD on a "Boss" controller. It exposes as AHCI. It wouldn't recognize these as installable drives as raw devices. If you use the Boss to define a virtual drive in RAID-1 it exposes fine. Tried BIOS mode (not UEFI), tried FB13. No go. But, as a virtual, worked fine in UEFI boot mode. Dell. I mean, its not like we haven't been here before with their special SAS firmware, but still if anyone else has seen this and knows magic which unlocks the real device on AHCI I'd love to know. (I actually considered using them as ARC and Cache for ZFS on the real SAS drives but since the host doesn't "see" them as attached devices, I went with a h/w mirror pair to install the OS) -G From owner-freebsd-stable@freebsd.org Thu Dec 19 04:53:48 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3B7091D4E09 for ; Thu, 19 Dec 2019 04:53:48 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47dffL3kW6z4G3G for ; Thu, 19 Dec 2019 04:53:46 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id xBJ4rcOW084397 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 19 Dec 2019 04:53:39 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: ggm@algebras.org Received: from [10.58.0.4] (dadv@[10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id xBJ4rWlV036895 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 19 Dec 2019 11:53:32 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: 12.1, Dell 740 and the M2 SSD as boot media doesn't work To: George Michaelson , FreeBSD Stable References: From: Eugene Grosbein Message-ID: <983b9569-f968-156c-a773-0cc245569744@grosbein.net> Date: Thu, 19 Dec 2019 11:53:27 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 47dffL3kW6z4G3G X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.81 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-1.71)[ip: (-4.54), ipnet: 2a01:4f8::/29(-2.42), asn: 24940(-1.55), country: DE(-0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2019 04:53:48 -0000 19.12.2019 11:21, George Michaelson wrote: > I tried an install on a new Dell 740, which has dual M2 256GB internal > SSD on a "Boss" controller. It exposes as AHCI. > > It wouldn't recognize these as installable drives as raw devices. If > you use the Boss to define a virtual drive in RAID-1 it exposes fine. > > Tried BIOS mode (not UEFI), tried FB13. No go. But, as a virtual, > worked fine in UEFI boot mode. > > Dell. I mean, its not like we haven't been here before with their > special SAS firmware, but still if anyone else has seen this and knows > magic which unlocks the real device on AHCI I'd love to know. You either switch the controller to "HBA" mode using its setup utility, or keep it in current "RAID" mode but configure your SSD as "Non-RAID" drive. Also, you must use mrsas(4) driver instead of default mfi(4) to get TRIM support for SSD. mfi(4) has no TRIM support at all. You may switch to mrsas(4) after intallation, though. From owner-freebsd-stable@freebsd.org Thu Dec 19 05:35:51 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 161F51D5B0F for ; Thu, 19 Dec 2019 05:35:51 +0000 (UTC) (envelope-from ggm@algebras.org) Received: from mail-io1-xd2b.google.com (mail-io1-xd2b.google.com [IPv6:2607:f8b0:4864:20::d2b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47dgZt0WG3z4Hbv for ; Thu, 19 Dec 2019 05:35:49 +0000 (UTC) (envelope-from ggm@algebras.org) Received: by mail-io1-xd2b.google.com with SMTP id x1so4476308iop.7 for ; Wed, 18 Dec 2019 21:35:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=algebras-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ZLhSeOJgcVfO06t4wU9IELwiFePtn6IeZIIQ48T3w6s=; b=rF3ayWC9YKka20y3/VZcyMJvZtdJceFOkwzkhOvCmECI3uk56UuLECTOCIJdcvE3ss V7WI7i6BekmngTVy/g8geMNJHNSWa6UDaLzMV7iRWs1OStxNCKozib/2EnAiRJZidarp Np/ZCreOm1l0W0YPGe5FOKLRoUWcD9wnyTY+JR/YtBrSy0CzlzpuDmr5nH4kW68MeUdT fBvxCDmaaXph4Zq8Nbs77ojcAN00ER1dXPlKoHKxDf/cBi9MWRS3aUwCjlyU2rGujIom CK/s2KnaLXke1tWJ24T75eNOrZOkkuzTrne+dlzb9DRzY3b5ANI2B0mZ2xYIobpqFeSO BtEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ZLhSeOJgcVfO06t4wU9IELwiFePtn6IeZIIQ48T3w6s=; b=FDL1qofdsPYVGVM9iyGPL2fCSp7MHNhy51LHbTIiK/doG2lzaIz+FJgusMYxUmitRe fjmdMOST1p1A9/AAEXMu1DWAb+SWDr1GG86hS3e43EQUj2ekLyq+/ezf87j5L1gudVjZ kpMHo8Nt/1K8N1EyZzxKUqhhDtvPWMKHbsZ616aLyTZA0MwFG4twHYStCCXuPP4B09lM mz+L9tQogS2ptFRraF07EU+DFg7gLAyAh5To8ibnTxKcz/p503nNYfbYMZHH7/JrXygw eJl9Sy4ZxDM87x7PbV9+VZAipX2rkc+BEh0pFhl6XV6Yl87wGCv26vVu1SdLpg5koouL O3ig== X-Gm-Message-State: APjAAAViEdskk5LQ0uqPn233UIcaTw0WyrJj5NymJm7V5lUGjS0jiZa7 KaSNKJtlXMfs/+AGRNBpNVpbSg1UwIrgjIstc8dUik+O0dc= X-Google-Smtp-Source: APXvYqzTU75YBzHl0S0HS1UtSeLPdcwjcTv8O+GFCa1+OwDWMl7YVRf/IUGvccQQyGkuJUwON+dLIHTyrNpa/4P8TME= X-Received: by 2002:a02:b385:: with SMTP id p5mr5550537jan.43.1576733748617; Wed, 18 Dec 2019 21:35:48 -0800 (PST) MIME-Version: 1.0 References: <983b9569-f968-156c-a773-0cc245569744@grosbein.net> In-Reply-To: <983b9569-f968-156c-a773-0cc245569744@grosbein.net> From: George Michaelson Date: Thu, 19 Dec 2019 15:35:37 +1000 Message-ID: Subject: Re: 12.1, Dell 740 and the M2 SSD as boot media doesn't work To: Eugene Grosbein Cc: FreeBSD Stable Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 47dgZt0WG3z4Hbv X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=algebras-org.20150623.gappssmtp.com header.s=20150623 header.b=rF3ayWC9; dmarc=none; spf=pass (mx1.freebsd.org: domain of ggm@algebras.org designates 2607:f8b0:4864:20::d2b as permitted sender) smtp.mailfrom=ggm@algebras.org X-Spamd-Result: default: False [-4.51 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[algebras-org.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[algebras.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[algebras-org.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[b.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-2.01)[ip: (-5.92), ipnet: 2607:f8b0::/32(-2.19), asn: 15169(-1.90), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2019 05:35:51 -0000 On Thu, Dec 19, 2019 at 2:53 PM Eugene Grosbein wrote: > You either switch the controller to "HBA" mode using its setup utility, > or keep it in current "RAID" mode but configure your SSD as "Non-RAID" drive. Tried HBA mode. Devices attached as AHCI were not recognized. Tried leaving in RAID mode: cannot have two non_RAID devices. Its either one (Raid) and one (non-Raid) or one RAID only. Only one VD_ defined mirror pair in RAID mode was recognized after boot by the ISO boot media. Is it possible interrupting second stage boot to set hw.x.y.z= sysctl type settings would help? Something about device recognition? > > Also, you must use mrsas(4) driver instead of default mfi(4) to get TRIM support for SSD. > mfi(4) has no TRIM support at all. You may switch to mrsas(4) after intallation, though. Thanks for the tip! -G From owner-freebsd-stable@freebsd.org Thu Dec 19 09:50:46 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E10C61DB675 for ; Thu, 19 Dec 2019 09:50:46 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 47dnF20YVmz4TdR for ; Thu, 19 Dec 2019 09:50:46 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: by mailman.nyi.freebsd.org (Postfix) id 0D3931DB674; Thu, 19 Dec 2019 09:50:46 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0D0131DB673 for ; Thu, 19 Dec 2019 09:50:46 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47dnF04fk9z4TdG for ; Thu, 19 Dec 2019 09:50:43 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cs.huji.ac.il; s=57791128; h=To:References:Message-Id:Content-Transfer-Encoding:Cc:Date:In-Reply-To:From:Subject:Mime-Version:Content-Type; bh=aDThAax3PE8mzGIUS3o3WR1MoQkbKkHWRt1c5rQJG/s=; b=dcz90JnOXBxBN/7PAutgRxx877QKZmGj2Veolzt0PwsUJbiDK8D4kJ+K+Yo/sm4bddUWOhuccEzaqfuVK/ssCQC27fxQkd2pXjMn1V8sc81t9EJSju64MJgl6Wat78B1q9lZzkbKCsoEaQ68NuKe/lIyWVvU4eLNYXwQtrXPXBRFGOLiDBtvvMhGvQh9p6vTaKG0i6YpC0XL22uIe2MgsIC/VCJdMmhOcA8rNozzakjzxLMSjWT20VYAju2F0F88wc4F9bisqdZWI7DZb2LeNw6RFVUVMx5Hs7MwZej4+JpsuKKn+jAe4B4EL4+2hn1zXWOS17MtcwPkqR63PqenBA==; Received: from bach.cs.huji.ac.il ([132.65.80.20]) by kabab.cs.huji.ac.il with esmtp id 1ihsS8-0009FR-1Z; Thu, 19 Dec 2019 11:50:40 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: nfs lockd errors after NetApp software upgrade. From: Daniel Braniss In-Reply-To: Date: Thu, 19 Dec 2019 11:50:39 +0200 Cc: Richard P Mackerras , "stable@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> To: Rick Macklem X-Mailer: Apple Mail (2.3445.9.1) X-Rspamd-Queue-Id: 47dnF04fk9z4TdG X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.huji.ac.il header.s=57791128 header.b=dcz90JnO; dmarc=pass (policy=none) header.from=huji.ac.il; spf=none (mx1.freebsd.org: domain of danny@cs.huji.ac.il has no SPF policy when checking 132.65.116.210) smtp.mailfrom=danny@cs.huji.ac.il X-Spamd-Result: default: False [-3.66 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[cs.huji.ac.il:s=57791128]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-1.36)[ip: (-3.28), ipnet: 132.64.0.0/13(-1.99), asn: 378(-1.60), country: IL(0.05)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[cs.huji.ac.il:+]; DMARC_POLICY_ALLOW(-0.50)[huji.ac.il,none]; RCVD_IN_DNSWL_NONE(0.00)[210.116.65.132.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:378, ipnet:132.64.0.0/13, country:IL]; FREEMAIL_CC(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2019 09:50:46 -0000 > On 19 Dec 2019, at 02:22, Rick Macklem wrote: >=20 > Richard P Mackerras wrote: >=20 >> Hi, >> What software version is the NetApp using? >> Is the exported volume big? >> Is the vserver configured for 64bit identifiers? >>=20 >> If you enable NFS V4.0 or 4.1 other NFS clients using defaults might = mount NFSv4.x >unexpectedly after a reboot so you need to watch that. > The FreeBSD client always uses NFSv3 mounts by default. To get NFSv4 = you must > explicitly specify the "nfsv4" or "vers=3D4" mount option. For = NFSv4.1, you must > also specify "minorversion=3D1=E2=80=9D. >=20 > The Linux distros I am familiar with will use the highest NFS version = supported by > the server by default. (I suspect some are using NFSv4.1 without = realizing it, > which isn't necessarily bad.) >=20 > nfsstat -m > will show you which version is actually in use for both FreeBSD and = Linux. >=20 all mounts are nfsv3/tcp the error is also appearing on freebsd-11.2-stable, I=E2=80=99m now = checking if it=E2=80=99s also happening on 12.1 btw, the NetApp version is 9.3P17 cheers, danny > rick >=20 > Cheers >=20 > Richard > (NetApp admin) >=20 > On Wed, 18 Dec 2019 at 15:46, Daniel Braniss = > wrote: >=20 >=20 >> On 18 Dec 2019, at 16:55, Rick Macklem = > wrote: >>=20 >> Daniel Braniss wrote: >>=20 >>> Hi, >>> The server with the problems is running FreeBSD 11.1 stable, it was = working fine for >several months, >>> but after a software upgrade of our NetAPP server it=E2=80=99s = reporting many lockd errors >and becomes catatonic, >>> ... >>> Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not = responding >>> Dec 18 13:11:45 moo-09 last message repeated 7 times >>> Dec 18 13:12:55 moo-09 last message repeated 8 times >>> Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is = alive again >>> Dec 18 13:13:10 moo-09 last message repeated 8 times >>> Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 194 already in queue awaiting acceptance (1 = occurrences) >>> Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 193 already in queue awaiting acceptance (3957 = occurrences) >>> Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 193 already in queue awaiting acceptance =E2=80=A6= >> Seems like their software upgrade didn't improve handling of NLM = RPCs? >> Appears to be handling RPCs slowly and/or intermittently. Note that = no one >> tests it with IPv6, so at least make sure you are still using IPv4 = for the mounts and >> try and make sure IP broadcast works between client and Netapp. I = think the NLM >> and NSM (rpc.statd) still use IP broadcast sometimes. >>=20 > we are ipv4 - we have our own class c :-) >> Maybe the network guys can suggest more w.r.t. why, but as I've = stated before, >> the NLM is a fundamentally broken protocol which was never published = by Sun, >> so I suggest you avoid using it if at all possible. > well, at the moment the ball is on NetAPP court, and switching to = NFSv4 at the moment is out of the question, it=E2=80=99s > a production server used by several thousand students. >=20 >>=20 >> - If the locks don't need to be seen by other clients, you can just = use the "nolockd" >> mount option. >> or >> - If locks need to be seen by other clients, try NFSv4 mounts. Netapp = filers >> should support NFSv4.1, which is a much better protocol that = NFSv4.0. >>=20 >> Good luck with it, rick > thanks > danny >=20 >> =E2=80=A6 >> any ideas? >>=20 >> thanks, >> danny >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org mailing = list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing = list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@freebsd.org Thu Dec 19 14:09:40 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 76CC01E17E7 for ; Thu, 19 Dec 2019 14:09:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 47dtzm1dWnz3Dl7 for ; Thu, 19 Dec 2019 14:09:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: by mailman.nyi.freebsd.org (Postfix) id 361061E17E6; Thu, 19 Dec 2019 14:09:40 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 35CEE1E17E5 for ; Thu, 19 Dec 2019 14:09:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660089.outbound.protection.outlook.com [40.107.66.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47dtzl1sdsz3Dl6 for ; Thu, 19 Dec 2019 14:09:38 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AfSqKAUG+NG3/FiBUm2nqMZtnU5ZJdd5UmgOSe17bOCYdF24bdwl6FuumUa2ABO7MfCqQ2d454JEbYeIwWWdF9lqxpuCMjW410JtM72LC+5sfv36SVBb2x4ATqIgMeHwUwk0FHOG+q1QFdrHEHOnUnB9pPelS1UVkuFeWkfPhMyjGY31+YWcjVSiJR4dxLnOsdLhQCPyhELl4QyoFI5LEJMP+nWSuvBjC6jC1dR//L/4GIO3gGy4R/RuCqBMyEsKrgpIexYfkFM16K2f2H2jqeUowv/qqH7FxWlAZkj/Srqhy2A8NE6RobdD2yExh+qZVzfsZXs80KOSTkbC+Sqb0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7za0d1ilKDjMG1D4C+8cdHbVkjsnzgYE4A6BSWIUX2Y=; b=Td+hiFSa5Guy51/8cz08wDkXm/uP90VSm1J7whGtwv9866RXNJANrNv98BKpIys+J/6RmMglq1Dlol8lRsDRR1+HnartRkTGVd8aLUNr08cBI8JOtNGHWc2IWvSkuRpVSfuiLe6C6yEh+UUhKC8kpr64JRb966AXTTKq+adFwZJ7kVaLZRb2Tx/Iojo4IC/Wonz9p8OyuHJz6NCYEUHyucmh+FsNVrhj6uQ6OWT7j5iE18/ZCf++CF3vRpHvu4/+y8+npUZlez1bOvI1tNruTxUObIOaJhxlBnzbMLUkJNisvrLhw7lBNfqXn9XCHywhcDileYHIxMzk9J96beKPSg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM (52.132.69.153) by YQBPR0101MB0835.CANPRD01.PROD.OUTLOOK.COM (52.132.71.150) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.16; Thu, 19 Dec 2019 14:09:37 +0000 Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75]) by YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75%5]) with mapi id 15.20.2538.019; Thu, 19 Dec 2019 14:09:37 +0000 From: Rick Macklem To: Daniel Braniss CC: Richard P Mackerras , "stable@freebsd.org" Subject: Re: nfs lockd errors after NetApp software upgrade. Thread-Topic: nfs lockd errors after NetApp software upgrade. Thread-Index: AQHVtawq+ga5QLcdVkqBDG/GW9zFg6e/+Am+gAARTACAAANHAIAAi7Y3gACf34CAAEVO6A== Date: Thu, 19 Dec 2019 14:09:37 +0000 Message-ID: References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> , <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> In-Reply-To: <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: fa84a4dd-d9ea-4bea-502e-08d7848d152a x-ms-traffictypediagnostic: YQBPR0101MB0835: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:7691; x-forefront-prvs: 0256C18696 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(136003)(396003)(366004)(39860400002)(346002)(376002)(189003)(199004)(26005)(186003)(53546011)(7696005)(6506007)(55016002)(6916009)(9686003)(71200400001)(2906002)(786003)(54906003)(8936002)(316002)(86362001)(81166006)(81156014)(8676002)(5660300002)(52536014)(966005)(478600001)(4326008)(33656002)(66946007)(66556008)(64756008)(66446008)(66476007)(76116006); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB0835; H:YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: yEqKfGg3diXckIOo78VB9S65yu8tUgQs6JHl36OkVF89B4JBL0Wc/pyHiuot3gkJoi2Tr17aRo5ZvufoezGb+rqJh2oh5cikhNErHUK76zGlU7EOQLMryYmE7xYvaIMPE6+rbGLP0OheRTooYVxP1Mt/YaSSM2UwglFaULFWoJQMb9aVLPZEoxOoXdS8+dl5KSjKjk9+Tep39fbqOPQoxhbYP6JdcTOqfLKBVhJQZpzMv9IZfZPYDJbFY+UmghhDIyJoH9LfaXakrvsxZWB91wyWkBOwj3PSqw/x+0NRI+MaKXvi3J4fkxxmkRnCtTZmo464PK6gMYe6L/BpK+boYs7wvc2nxs02NeVkt7chTChXmvH5n+mqoEm04reSeeMYVo4wZc+Srt8oB2LGl8nrnz1oGQyfsqlgTIIp/5Lt/R7/Qg80YQWF8BBs4+1cWcYvYk02hecaGbdSMc8w1NJyFKaxxRlkZCMiA0nGgtGqzvs= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: fa84a4dd-d9ea-4bea-502e-08d7848d152a X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Dec 2019 14:09:37.5654 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: O35IXmwn9kcPjIdsBA/xSBC5M0gi3AM5W53uplF49ldFTaPulGXjMEI/1xgnhxXbzjLGPaJmRTkz5q7weug7AQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB0835 X-Rspamd-Queue-Id: 47dtzl1sdsz3Dl6 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.89 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.66 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[uoguelph.ca]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[89.66.107.40.list.dnswl.org : 127.0.3.0]; IP_SCORE(-1.36)[ipnet: 40.64.0.0/10(-3.83), asn: 8075(-2.92), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1]; FREEMAIL_CC(0.00)[gmail.com] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2019 14:09:40 -0000 Daniel Braniss wrote:=0A= [stuff snipped]=0A= >all mounts are nfsv3/tcp=0A= This doesn't affect what the NLM code (rpc.lockd) uses. I honestly don't kn= ow when=0A= the NLM uses tcp vs udp. I think rpc.statd still uses IP broadcast at times= .=0A= =0A= To me, it looks like a network configuration issue.=0A= You could capture packets (maybe when a client first starts rpc.statd and r= pc.lockd)=0A= and then look at them in wireshark. I'd disable statup of rpc.lockd and rpc= .statd=0A= at boot for a test client and then run something like:=0A= # tcpdump -s 0 -s out.pcap host =0A= - and then start rpc.statd and rpc.lockd=0A= Then I'd look at out.pcap in wireshark (much better at decoding this stuff = than=0A= tcpdump). I'd look for things like different reply IP addresses from the Ne= tapp,=0A= which might confuse this tired old NLM protocol Sun devised in the mid-1980= s.=0A= =0A= >the error is also appearing on freebsd-11.2-stable, I=92m now checking if = it=92s also=0A= >happening on 12.1=0A= >btw, the NetApp version is 9.3P17=0A= Yes. I wasn't the author of the NSM and NLM code (long ago I refused to eve= n=0A= try to implement it, because I knew the protocol was badly broken) and I av= oid=0A= fiddling with. As such, it won't have change much since around FreeBSD7.=0A= =0A= rick=0A= =0A= cheers,=0A= danny=0A= =0A= > rick=0A= >=0A= > Cheers=0A= >=0A= > Richard=0A= > (NetApp admin)=0A= >=0A= > On Wed, 18 Dec 2019 at 15:46, Daniel Braniss > wrote:=0A= >=0A= >=0A= >> On 18 Dec 2019, at 16:55, Rick Macklem > wrote:=0A= >>=0A= >> Daniel Braniss wrote:=0A= >>=0A= >>> Hi,=0A= >>> The server with the problems is running FreeBSD 11.1 stable, it was wor= king fine for >several months,=0A= >>> but after a software upgrade of our NetAPP server it=92s reporting many= lockd errors >and becomes catatonic,=0A= >>> ...=0A= >>> Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not res= ponding=0A= >>> Dec 18 13:11:45 moo-09 last message repeated 7 times=0A= >>> Dec 18 13:12:55 moo-09 last message repeated 8 times=0A= >>> Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is aliv= e again=0A= >>> Dec 18 13:13:10 moo-09 last message repeated 8 times=0A= >>> Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Liste= n queue >overflow: 194 already in queue awaiting acceptance (1 occurrences)= =0A= >>> Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Liste= n queue >overflow: 193 already in queue awaiting acceptance (3957 occurrenc= es)=0A= >>> Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Liste= n queue >overflow: 193 already in queue awaiting acceptance =85=0A= >> Seems like their software upgrade didn't improve handling of NLM RPCs?= =0A= >> Appears to be handling RPCs slowly and/or intermittently. Note that no o= ne=0A= >> tests it with IPv6, so at least make sure you are still using IPv4 for t= he mounts and=0A= >> try and make sure IP broadcast works between client and Netapp. I think = the NLM=0A= >> and NSM (rpc.statd) still use IP broadcast sometimes.=0A= >>=0A= > we are ipv4 - we have our own class c :-)=0A= >> Maybe the network guys can suggest more w.r.t. why, but as I've stated b= efore,=0A= >> the NLM is a fundamentally broken protocol which was never published by = Sun,=0A= >> so I suggest you avoid using it if at all possible.=0A= > well, at the moment the ball is on NetAPP court, and switching to NFSv4 a= t the moment is out of the question, it=92s=0A= > a production server used by several thousand students.=0A= >=0A= >>=0A= >> - If the locks don't need to be seen by other clients, you can just use = the "nolockd"=0A= >> mount option.=0A= >> or=0A= >> - If locks need to be seen by other clients, try NFSv4 mounts. Netapp fi= lers=0A= >> should support NFSv4.1, which is a much better protocol that NFSv4.0.= =0A= >>=0A= >> Good luck with it, rick=0A= > thanks=0A= > danny=0A= >=0A= >> =85=0A= >> any ideas?=0A= >>=0A= >> thanks,=0A= >> danny=0A= >>=0A= >> _______________________________________________=0A= >> freebsd-stable@freebsd.org mailing li= st=0A= >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org= "=0A= >=0A= > _______________________________________________=0A= > freebsd-stable@freebsd.org mailing lis= t=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org<= mailto:freebsd-stable-unsubscribe@freebsd.org>"=0A= =0A= From owner-freebsd-stable@freebsd.org Thu Dec 19 14:21:22 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5ACC21E1D4D for ; Thu, 19 Dec 2019 14:21:22 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 47dvFF5N9Yz3FYm for ; Thu, 19 Dec 2019 14:21:21 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: by mailman.nyi.freebsd.org (Postfix) id B88A01E1D4B; Thu, 19 Dec 2019 14:21:21 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B84F41E1D4A for ; Thu, 19 Dec 2019 14:21:21 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47dvFD1k8lz3FYD for ; Thu, 19 Dec 2019 14:21:19 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cs.huji.ac.il; s=57791128; h=To:References:Message-Id:Content-Transfer-Encoding:Cc:Date:In-Reply-To:From:Subject:Mime-Version:Content-Type; bh=0X53b4uadxmJclmab2aTrjy8jfXbJ68JCiPGfa/nT4c=; b=iwbX9+vviB48IzG5c2x/S+jPj5C87A/APQ3WoowJBwImGiUR+FmLSd4fh9MAdbNE6Vg9vz1DpwOSjQlUBZGPn0ogyDi8V7K+7hrOP2KPxEkKfj4++q01veuGl199ROP6reGCKRbBwuKr1XrK+XTC1E5wXa6RtxKAqHz+hkBgPxoHrFozR9tOG+MUpm2Do7pd64nm1Tur3c+W9CJ90ri43UUIcQNgmQmpuofy4uicCeoApX2/LPZCNwBCWGfgVFoMdHr8/HxKstFHHSLlboAqKC8PV7etBsRumNMSqI1107j2ppCVOLzLWwp6ibLih7zGaeoouCnylyuasKWP3XJhdQ==; Received: from macmini.bk.cs.huji.ac.il ([132.65.179.19]) by kabab.cs.huji.ac.il with esmtp id 1ihwg0-000NH7-6Y; Thu, 19 Dec 2019 16:21:16 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.40.2.2.4\)) Subject: Re: nfs lockd errors after NetApp software upgrade. From: Daniel Braniss In-Reply-To: Date: Thu, 19 Dec 2019 16:21:16 +0200 Cc: Richard P Mackerras , "stable@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> To: Rick Macklem X-Mailer: Apple Mail (2.3608.40.2.2.4) X-Rspamd-Queue-Id: 47dvFD1k8lz3FYD X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.huji.ac.il header.s=57791128 header.b=iwbX9+vv; dmarc=pass (policy=none) header.from=huji.ac.il; spf=none (mx1.freebsd.org: domain of danny@cs.huji.ac.il has no SPF policy when checking 132.65.116.210) smtp.mailfrom=danny@cs.huji.ac.il X-Spamd-Result: default: False [-3.77 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[cs.huji.ac.il:s=57791128]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-1.47)[ip: (-3.57), ipnet: 132.64.0.0/13(-2.12), asn: 378(-1.69), country: IL(0.05)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[cs.huji.ac.il:+]; DMARC_POLICY_ALLOW(-0.50)[huji.ac.il,none]; RCVD_IN_DNSWL_NONE(0.00)[210.116.65.132.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:378, ipnet:132.64.0.0/13, country:IL]; FREEMAIL_CC(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2019 14:21:22 -0000 > On 19 Dec 2019, at 16:09, Rick Macklem wrote: >=20 > Daniel Braniss wrote: > [stuff snipped] >> all mounts are nfsv3/tcp > This doesn't affect what the NLM code (rpc.lockd) uses. I honestly = don't know when > the NLM uses tcp vs udp. I think rpc.statd still uses IP broadcast at = times. can the replay cache have any influence here? I tend to remember way = back issues with it, >=20 > To me, it looks like a network configuration issue. that was/is my gut feelings too, but, as far as we can tell, nothing has = changed in the network infrastructure, the problems appeared after the NetAPP=E2=80=99s software was updated, = it was working fine till then. the problems are also happening on freebsd 12.1 > You could capture packets (maybe when a client first starts rpc.statd = and rpc.lockd) > and then look at them in wireshark. I'd disable statup of rpc.lockd = and rpc.statd > at boot for a test client and then run something like: > # tcpdump -s 0 -s out.pcap host > - and then start rpc.statd and rpc.lockd > Then I'd look at out.pcap in wireshark (much better at decoding this = stuff than > tcpdump). I'd look for things like different reply IP addresses from = the Netapp, > which might confuse this tired old NLM protocol Sun devised in the = mid-1980s. >=20 it=E2=80=99s going to be an interesting week end :-( =20 >> the error is also appearing on freebsd-11.2-stable, I=E2=80=99m now = checking if it=E2=80=99s also >> happening on 12.1 >> btw, the NetApp version is 9.3P17 > Yes. I wasn't the author of the NSM and NLM code (long ago I refused = to even > try to implement it, because I knew the protocol was badly broken) and = I avoid > fiddling with. As such, it won't have change much since around = FreeBSD7. and we haven=E2=80=99t had any issues with it for years, so you must = have done something good cheers, danny >=20 > rick >=20 > cheers, > danny >=20 >> rick >>=20 >> Cheers >>=20 >> Richard >> (NetApp admin) >>=20 >> On Wed, 18 Dec 2019 at 15:46, Daniel Braniss = > wrote: >>=20 >>=20 >>> On 18 Dec 2019, at 16:55, Rick Macklem = > wrote: >>>=20 >>> Daniel Braniss wrote: >>>=20 >>>> Hi, >>>> The server with the problems is running FreeBSD 11.1 stable, it was = working fine for >several months, >>>> but after a software upgrade of our NetAPP server it=E2=80=99s = reporting many lockd errors >and becomes catatonic, >>>> ... >>>> Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not = responding >>>> Dec 18 13:11:45 moo-09 last message repeated 7 times >>>> Dec 18 13:12:55 moo-09 last message repeated 8 times >>>> Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is = alive again >>>> Dec 18 13:13:10 moo-09 last message repeated 8 times >>>> Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 194 already in queue awaiting acceptance (1 = occurrences) >>>> Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 193 already in queue awaiting acceptance (3957 = occurrences) >>>> Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 193 already in queue awaiting acceptance =E2=80=A6= >>> Seems like their software upgrade didn't improve handling of NLM = RPCs? >>> Appears to be handling RPCs slowly and/or intermittently. Note that = no one >>> tests it with IPv6, so at least make sure you are still using IPv4 = for the mounts and >>> try and make sure IP broadcast works between client and Netapp. I = think the NLM >>> and NSM (rpc.statd) still use IP broadcast sometimes. >>>=20 >> we are ipv4 - we have our own class c :-) >>> Maybe the network guys can suggest more w.r.t. why, but as I've = stated before, >>> the NLM is a fundamentally broken protocol which was never published = by Sun, >>> so I suggest you avoid using it if at all possible. >> well, at the moment the ball is on NetAPP court, and switching to = NFSv4 at the moment is out of the question, it=E2=80=99s >> a production server used by several thousand students. >>=20 >>>=20 >>> - If the locks don't need to be seen by other clients, you can just = use the "nolockd" >>> mount option. >>> or >>> - If locks need to be seen by other clients, try NFSv4 mounts. = Netapp filers >>> should support NFSv4.1, which is a much better protocol that = NFSv4.0. >>>=20 >>> Good luck with it, rick >> thanks >> danny >>=20 >>> =E2=80=A6 >>> any ideas? >>>=20 >>> thanks, >>> danny >>>=20 >>> _______________________________________________ >>> freebsd-stable@freebsd.org = mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org mailing = list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >=20 From owner-freebsd-stable@freebsd.org Thu Dec 19 21:43:57 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 569EF1EBCC8 for ; Thu, 19 Dec 2019 21:43:57 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 47f53x0pXvz48YD for ; Thu, 19 Dec 2019 21:43:57 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 19F1A1EBCC7; Thu, 19 Dec 2019 21:43:57 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 19B101EBCC6 for ; Thu, 19 Dec 2019 21:43:57 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47f53w1MqXz48YC for ; Thu, 19 Dec 2019 21:43:55 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: by mail-wm1-x335.google.com with SMTP id p17so7212354wma.1 for ; Thu, 19 Dec 2019 13:43:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=zG6Y6t3Wkfd9Kl9ilev35H+aUElZY+JKuN64YpXxE2A=; b=HDgltDIa3TRPVI1HmA+TSx8MsYJabUccdlx6JDr+tckHvuGIxZ6FSYMvZl3d0FweE0 KrWlwxv45J3w5bAl+iASwrNQ0qOrMf9pwbxoPC7iFGWE2q3s2SON/ip2MfOn/lKr6sDn rNXTyu4PoKIgqtFYBqLFXPTCai5HgDRyRy8XAD0nfHifMbTbhP7Ohb+haIMVhmhOOKq6 7pe24D81o/HowZdWpX/3PMIQqYXPPDhgBqLSD2s+KJFxWW3PwbXJY6xMtfhHXbZNdVNN Q8YnHyoErTqF7d87PBZ//zsYn7qBQd0NkPt3Gh2xKIJfQp336tXekFq+ua2VFpFEfLKc AJ2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=zG6Y6t3Wkfd9Kl9ilev35H+aUElZY+JKuN64YpXxE2A=; b=DUA1UUQ8Tfw5m+uX8U5TcFZ6AJ7GXZwy+F7yt0kzitdGlwnhJqDMN7gZqgeaqozpm0 2JxXcJF2AiJw8exmpS/5pt4AU48jS290EdZrkZ1ESbyeJ8Ak94vjTZCicmd0mVU21q7X W2wReBfJMlHyMSu4r+ASnBi88jIx+CCDOQ0Y1JT5TXj0Y5jalMqu+MkSm/9fwnIxKiOa dQLpXXKU7yRDHmSUZAYrwRqequZmfllmpp4jdWiqGqtMhmvC3XBYYcHpSA8tAoDzkHLX hRnf+MpIU3P/W5yiZvinKwwMBIpMOsIC3eQ1vTa3wntNJayPZfsgEhepfGsHToOuYdlB tmvQ== X-Gm-Message-State: APjAAAUC1141clQkKOhO0Afo67QsByynbd4eO22/I+RyHkv0DAA7YRl+ pzvu/WfU8Yjs00tMp6qDnyIFm29VMW9SWy0sAKw= X-Google-Smtp-Source: APXvYqzK/woX27Ahv9NMS3dyn3qM+omDRO7LUUO/tFPxQLiS4D698q3LiwVfv+dYlBhsrDHTRJHR3vjijPul4q8mtLY= X-Received: by 2002:a7b:c342:: with SMTP id l2mr13193708wmj.159.1576791832534; Thu, 19 Dec 2019 13:43:52 -0800 (PST) MIME-Version: 1.0 References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> In-Reply-To: <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> From: Richard P Mackerras Date: Thu, 19 Dec 2019 21:43:42 +0000 Message-ID: Subject: Re: nfs lockd errors after NetApp software upgrade. To: Daniel Braniss Cc: Rick Macklem , "stable@freebsd.org" X-Rspamd-Queue-Id: 47f53w1MqXz48YC X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=HDgltDIa; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mack63richard@gmail.com designates 2a00:1450:4864:20::335 as permitted sender) smtp.mailfrom=mack63richard@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (-9.19), ipnet: 2a00:1450::/32(-2.65), asn: 15169(-1.90), country: US(-0.05)]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[5.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2019 21:43:57 -0000 Hi, At ONTAP 9.3P6 there is a possible LACP group issue after upgrade. Have you checked any LACP groups, These should not be a problem but I assume network interfaces are at the home ports, not on slower ports or something silly. It is marginally better if the traffic goes direct to the node where the volume is but the difference should nothing. Have you looked at the NetApp performance data? If you are going to do wireshark tcpdumps then you might want to run them from the NetApp as well. https://kb.netapp.com/app/answers/answer_view/a_id/1029833/~/how-to-capture-packet-traces-%28tcpdump%29-on-ontap-9.2%2B-systems- ::> network tcpdump start -node -port e0a -buffer-size 2097151 Let us know how you go, Richard From owner-freebsd-stable@freebsd.org Fri Dec 20 03:07:37 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 041C51CA2AD for ; Fri, 20 Dec 2019 03:07:37 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from mail.egr.msu.edu (boomhauer.egr.msu.edu [35.9.37.164]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47fDFM63cTz4NRf for ; Fri, 20 Dec 2019 03:07:35 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from boomhauer (localhost [127.0.0.1]) by mail.egr.msu.edu (Postfix) with ESMTP id 4E8EDE59AE for ; Thu, 19 Dec 2019 22:07:34 -0500 (EST) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mail.egr.msu.edu ([127.0.0.1]) by boomhauer (boomhauer.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7qTnXS3lirez for ; Thu, 19 Dec 2019 22:07:33 -0500 (EST) Received: from EGR authenticated sender mcdouga9 Subject: Re: nfs lockd errors after NetApp software upgrade. To: freebsd-stable@freebsd.org References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> From: Adam McDougall Message-ID: Date: Thu, 19 Dec 2019 22:07:32 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.3.0 MIME-Version: 1.0 In-Reply-To: <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 47fDFM63cTz4NRf X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=msu.edu; spf=pass (mx1.freebsd.org: domain of mcdouga9@egr.msu.edu designates 35.9.37.164 as permitted sender) smtp.mailfrom=mcdouga9@egr.msu.edu X-Spamd-Result: default: False [-3.04 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:boomhauer.egr.msu.edu]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[164.37.9.35.list.dnswl.org : 127.0.11.2]; DMARC_POLICY_ALLOW(-0.50)[msu.edu,none]; IP_SCORE(-0.04)[asn: 237(-0.13), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:237, ipnet:35.0.0.0/10, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Dec 2019 03:07:37 -0000 Try changing bool_t do_tcp = FALSE; to TRUE in /usr/src/sys/nlm/nlm_prot_impl.c, recompile the kernel and try again. I think this makes it match Linux client behavior. I suspect I ran into the same issue as you. I do think I used nolockd is a workaround temporarily. I can provide some more details if it works. On 12/19/19 9:21 AM, Daniel Braniss wrote: > > >> On 19 Dec 2019, at 16:09, Rick Macklem wrote: >> >> Daniel Braniss wrote: >> [stuff snipped] >>> all mounts are nfsv3/tcp >> This doesn't affect what the NLM code (rpc.lockd) uses. I honestly don't know when >> the NLM uses tcp vs udp. I think rpc.statd still uses IP broadcast at times. > can the replay cache have any influence here? I tend to remember way back issues > with it, >> >> To me, it looks like a network configuration issue. > that was/is my gut feelings too, but, as far as we can tell, nothing has changed in the network infrastructure, > the problems appeared after the NetAPP’s software was updated, it was working fine till then. > > the problems are also happening on freebsd 12.1 > >> You could capture packets (maybe when a client first starts rpc.statd and rpc.lockd) >> and then look at them in wireshark. I'd disable statup of rpc.lockd and rpc.statd >> at boot for a test client and then run something like: >> # tcpdump -s 0 -s out.pcap host >> - and then start rpc.statd and rpc.lockd >> Then I'd look at out.pcap in wireshark (much better at decoding this stuff than >> tcpdump). I'd look for things like different reply IP addresses from the Netapp, >> which might confuse this tired old NLM protocol Sun devised in the mid-1980s. >> > it’s going to be an interesting week end :-( > >>> the error is also appearing on freebsd-11.2-stable, I’m now checking if it’s also >>> happening on 12.1 >>> btw, the NetApp version is 9.3P17 >> Yes. I wasn't the author of the NSM and NLM code (long ago I refused to even >> try to implement it, because I knew the protocol was badly broken) and I avoid >> fiddling with. As such, it won't have change much since around FreeBSD7. > and we haven’t had any issues with it for years, so you must have done something good > > cheers, > danny > >> >> rick >> >> cheers, >> danny >> >>> rick >>> >>> Cheers >>> >>> Richard >>> (NetApp admin) >>> >>> On Wed, 18 Dec 2019 at 15:46, Daniel Braniss > wrote: >>> >>> >>>> On 18 Dec 2019, at 16:55, Rick Macklem > wrote: >>>> >>>> Daniel Braniss wrote: >>>> >>>>> Hi, >>>>> The server with the problems is running FreeBSD 11.1 stable, it was working fine for >several months, >>>>> but after a software upgrade of our NetAPP server it’s reporting many lockd errors >and becomes catatonic, >>>>> ... >>>>> Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not responding >>>>> Dec 18 13:11:45 moo-09 last message repeated 7 times >>>>> Dec 18 13:12:55 moo-09 last message repeated 8 times >>>>> Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is alive again >>>>> Dec 18 13:13:10 moo-09 last message repeated 8 times >>>>> Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen queue >overflow: 194 already in queue awaiting acceptance (1 occurrences) >>>>> Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen queue >overflow: 193 already in queue awaiting acceptance (3957 occurrences) >>>>> Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen queue >overflow: 193 already in queue awaiting acceptance … >>>> Seems like their software upgrade didn't improve handling of NLM RPCs? >>>> Appears to be handling RPCs slowly and/or intermittently. Note that no one >>>> tests it with IPv6, so at least make sure you are still using IPv4 for the mounts and >>>> try and make sure IP broadcast works between client and Netapp. I think the NLM >>>> and NSM (rpc.statd) still use IP broadcast sometimes. >>>> >>> we are ipv4 - we have our own class c :-) >>>> Maybe the network guys can suggest more w.r.t. why, but as I've stated before, >>>> the NLM is a fundamentally broken protocol which was never published by Sun, >>>> so I suggest you avoid using it if at all possible. >>> well, at the moment the ball is on NetAPP court, and switching to NFSv4 at the moment is out of the question, it’s >>> a production server used by several thousand students. >>> >>>> >>>> - If the locks don't need to be seen by other clients, you can just use the "nolockd" >>>> mount option. >>>> or >>>> - If locks need to be seen by other clients, try NFSv4 mounts. Netapp filers >>>> should support NFSv4.1, which is a much better protocol that NFSv4.0. >>>> >>>> Good luck with it, rick >>> thanks >>> danny >>> >>>> … >>>> any ideas? >>>> >>>> thanks, >>>> danny >>>> >>>> _______________________________________________ >>>> freebsd-stable@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >>> >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >> > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@freebsd.org Fri Dec 20 17:19:40 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CD01F1DEDF2 for ; Fri, 20 Dec 2019 17:19:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660089.outbound.protection.outlook.com [40.107.66.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47fb8W1cNFz4Bkm for ; Fri, 20 Dec 2019 17:19:38 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lEKZfZNTZ+oFAYkQqCmwDq8M7sMtMazYspLYXLLY15V7G/ThnDLskDtblBfqXhXiq35OeVaFGKEwJChi1217fWv0ymOL3ZAaXEbIXRaSeMvAUkRtbE33Qqi9yexR9msI7FJx0ahe2cLrLI6xsy/xAEqr6WoXu5rRjYobs0hAYX025uzHGuLckWWgFoL4I/syfyfKkNEiJqsYjrdgWmPK792hR9DxTqnvuCtF0vnSgBNOboFax2ZhNxDNNzuBTeP5yqwVTybv25CkLfIKllT1K8pGhoLQcug0ZGyu9AcqgrC7ZkkFr8otcFoxDoAhZ0/YH2DcwRL+SZO3ZZtWTCVvng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u1B2mtitw2pvbX37wt6FFJH/eme4RqeapeCZj7zB8cA=; b=bole+D7rcAWCW8UCoqclOVc2bWKjLkNohMuYU0zk1iNT3gRueI1wKHtJYsqfTum9Q2x8P+xdQuXG0gniSEPyI/QQBRj5916D3E1XtdpMkmWxgcjk8zvpHFHGGjSe/ybBHx/p7nxKiCi/1MGTUceegHFPZsuUr+2Xzcz53pl7Mk4A3uAJrcT4j2n8iB4btQla67HxZYwooiPCIUPa7MD3SeOT1ctaAZWKGfP+Ourai+AZlxCob/gkfcFFh+3xyRK4crraHv8ccfWocg3bH7JDvlutO0lIK+4fbOizgqO8HJWQ7//git0eLoStnBLqDrPZ3hkPdW2mL0TjID4wuNdR+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM (52.132.69.153) by YQBPR0101MB2193.CANPRD01.PROD.OUTLOOK.COM (52.132.68.30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.16; Fri, 20 Dec 2019 17:19:37 +0000 Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75]) by YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75%5]) with mapi id 15.20.2538.019; Fri, 20 Dec 2019 17:19:37 +0000 From: Rick Macklem To: Adam McDougall , "freebsd-stable@freebsd.org" Subject: Re: nfs lockd errors after NetApp software upgrade. Thread-Topic: nfs lockd errors after NetApp software upgrade. Thread-Index: AQHVtawq+ga5QLcdVkqBDG/GW9zFg6e/+Am+gAARTACAAANHAIAAi7Y3gACf34CAAEVO6IAABk4AgADWGACAAO1eZQ== Date: Fri, 20 Dec 2019 17:19:37 +0000 Message-ID: References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 7a2bf692-74f6-43c8-5b40-08d78570ca45 x-ms-traffictypediagnostic: YQBPR0101MB2193: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:5236; x-forefront-prvs: 025796F161 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(136003)(396003)(39860400002)(366004)(376002)(346002)(189003)(199004)(5660300002)(81166006)(8676002)(86362001)(81156014)(9686003)(8936002)(2906002)(110136005)(76116006)(478600001)(71200400001)(296002)(316002)(186003)(33656002)(7696005)(66446008)(66946007)(53546011)(52536014)(966005)(786003)(64756008)(66476007)(55016002)(66556008)(6506007); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB2193; H:YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: dbSwzA04qAg6rYH2x10XdxTLpjEQdccSyqxxU71iLoJjOj5lwKM5U4NjnWnZfj1AdnwApGwHLgnVqE0WGCFORQNoruLzGhaoSd/KG14UJeJYxiOoAxeS6N1qosBRSjKLU566eGCc2oYuAwTb8Wg1g7iM71lng8MgEONHOtSSqWgyaLzkYll57cZww9oIyILWIcG3qPZftQoexOQA48adbsHxCvQwZcYLgHO8+XGYN8F9H946VGivcIusBAtOKmlHc1Uv4rnWK1UyvxtNPCF2PeTB4J7XlwxvaPTV4QW5mKp3NM73W4KUGFxDc1Kow+ZE6fEELhYMYMN00wfv9LWThMmvYjHhQ/ujfW3mFTajaC9yBbu5V1mQLuSUd9nPiUYcHWDDk8cCHXsUKdbKmy7Zl/86oCo8hPP1Ndv+bQ4bbCBqy+64XUm1M9YEofYCnsWxaxEZYpF7YxQ5Ff2mkPl/2H9lSv+FyiIUsN+Gy8fQMG8= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 7a2bf692-74f6-43c8-5b40-08d78570ca45 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Dec 2019 17:19:37.1289 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: YbWFXxgOPxs7BBKWCpc/P09AxZF6epOOvGTQFKofhPdw8sPXsRQaQFgkZeKnnwnygYnrJApYd+j4hO+OAz+MnA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB2193 X-Rspamd-Queue-Id: 47fb8W1cNFz4Bkm X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.89 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.66 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[uoguelph.ca]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[89.66.107.40.list.dnswl.org : 127.0.3.0]; IP_SCORE(-1.36)[ipnet: 40.64.0.0/10(-3.84), asn: 8075(-2.92), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Dec 2019 17:19:40 -0000 Adam McDougall wrote:=0A= >Try changing bool_t do_tcp =3D FALSE; to TRUE in=0A= >/usr/src/sys/nlm/nlm_prot_impl.c, recompile the kernel and try again. I=0A= >think this makes it match Linux client behavior. I suspect I ran into=0A= >the same issue as you. I do think I used nolockd is a workaround=0A= >temporarily. I can provide some more details if it works.=0A= If this fixes the problem, please let me know.=0A= =0A= I'm not sure I'd want to change the default, since it might break things fo= r=0A= others, but I can definitely make it a tunable, so that people don't need t= o=0A= recompile a kernel to deal with it.=0A= =0A= rick=0A= =0A= On 12/19/19 9:21 AM, Daniel Braniss wrote:=0A= >=0A= >=0A= >> On 19 Dec 2019, at 16:09, Rick Macklem wrote:=0A= >>=0A= >> Daniel Braniss wrote:=0A= >> [stuff snipped]=0A= >>> all mounts are nfsv3/tcp=0A= >> This doesn't affect what the NLM code (rpc.lockd) uses. I honestly don't= know when=0A= >> the NLM uses tcp vs udp. I think rpc.statd still uses IP broadcast at ti= mes.=0A= > can the replay cache have any influence here? I tend to remember way back= issues=0A= > with it,=0A= >>=0A= >> To me, it looks like a network configuration issue.=0A= > that was/is my gut feelings too, but, as far as we can tell, nothing has = changed in the network infrastructure,=0A= > the problems appeared after the NetAPP=92s software was updated, it was w= orking fine till then.=0A= >=0A= > the problems are also happening on freebsd 12.1=0A= >=0A= >> You could capture packets (maybe when a client first starts rpc.statd an= d rpc.lockd)=0A= >> and then look at them in wireshark. I'd disable statup of rpc.lockd and = rpc.statd=0A= >> at boot for a test client and then run something like:=0A= >> # tcpdump -s 0 -s out.pcap host =0A= >> - and then start rpc.statd and rpc.lockd=0A= >> Then I'd look at out.pcap in wireshark (much better at decoding this stu= ff than=0A= >> tcpdump). I'd look for things like different reply IP addresses from the= Netapp,=0A= >> which might confuse this tired old NLM protocol Sun devised in the mid-1= 980s.=0A= >>=0A= > it=92s going to be an interesting week end :-(=0A= >=0A= >>> the error is also appearing on freebsd-11.2-stable, I=92m now checking = if it=92s also=0A= >>> happening on 12.1=0A= >>> btw, the NetApp version is 9.3P17=0A= >> Yes. I wasn't the author of the NSM and NLM code (long ago I refused to = even=0A= >> try to implement it, because I knew the protocol was badly broken) and I= avoid=0A= >> fiddling with. As such, it won't have change much since around FreeBSD7.= =0A= > and we haven=92t had any issues with it for years, so you must have done = something good=0A= >=0A= > cheers,=0A= > danny=0A= >=0A= >>=0A= >> rick=0A= >>=0A= >> cheers,=0A= >> danny=0A= >>=0A= >>> rick=0A= >>>=0A= >>> Cheers=0A= >>>=0A= >>> Richard=0A= >>> (NetApp admin)=0A= >>>=0A= >>> On Wed, 18 Dec 2019 at 15:46, Daniel Braniss > wrote:=0A= >>>=0A= >>>=0A= >>>> On 18 Dec 2019, at 16:55, Rick Macklem > wrote:=0A= >>>>=0A= >>>> Daniel Braniss wrote:=0A= >>>>=0A= >>>>> Hi,=0A= >>>>> The server with the problems is running FreeBSD 11.1 stable, it was w= orking fine for >several months,=0A= >>>>> but after a software upgrade of our NetAPP server it=92s reporting ma= ny lockd errors >and becomes catatonic,=0A= >>>>> ...=0A= >>>>> Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not r= esponding=0A= >>>>> Dec 18 13:11:45 moo-09 last message repeated 7 times=0A= >>>>> Dec 18 13:12:55 moo-09 last message repeated 8 times=0A= >>>>> Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is al= ive again=0A= >>>>> Dec 18 13:13:10 moo-09 last message repeated 8 times=0A= >>>>> Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Lis= ten queue >overflow: 194 already in queue awaiting acceptance (1 occurrence= s)=0A= >>>>> Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Lis= ten queue >overflow: 193 already in queue awaiting acceptance (3957 occurre= nces)=0A= >>>>> Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Lis= ten queue >overflow: 193 already in queue awaiting acceptance =85=0A= >>>> Seems like their software upgrade didn't improve handling of NLM RPCs?= =0A= >>>> Appears to be handling RPCs slowly and/or intermittently. Note that no= one=0A= >>>> tests it with IPv6, so at least make sure you are still using IPv4 for= the mounts and=0A= >>>> try and make sure IP broadcast works between client and Netapp. I thin= k the NLM=0A= >>>> and NSM (rpc.statd) still use IP broadcast sometimes.=0A= >>>>=0A= >>> we are ipv4 - we have our own class c :-)=0A= >>>> Maybe the network guys can suggest more w.r.t. why, but as I've stated= before,=0A= >>>> the NLM is a fundamentally broken protocol which was never published b= y Sun,=0A= >>>> so I suggest you avoid using it if at all possible.=0A= >>> well, at the moment the ball is on NetAPP court, and switching to NFSv4= at the moment is out of the question, it=92s=0A= >>> a production server used by several thousand students.=0A= >>>=0A= >>>>=0A= >>>> - If the locks don't need to be seen by other clients, you can just us= e the "nolockd"=0A= >>>> mount option.=0A= >>>> or=0A= >>>> - If locks need to be seen by other clients, try NFSv4 mounts. Netapp = filers=0A= >>>> should support NFSv4.1, which is a much better protocol that NFSv4.0.= =0A= >>>>=0A= >>>> Good luck with it, rick=0A= >>> thanks=0A= >>> danny=0A= >>>=0A= >>>> =85=0A= >>>> any ideas?=0A= >>>>=0A= >>>> thanks,=0A= >>>> danny=0A= >>>>=0A= >>>> _______________________________________________=0A= >>>> freebsd-stable@freebsd.org mailing = list=0A= >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= >>>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.o= rg"=0A= >>>=0A= >>> _______________________________________________=0A= >>> freebsd-stable@freebsd.org mailing l= ist=0A= >>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= >>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.or= g"=0A= >>=0A= >=0A= > _______________________________________________=0A= > freebsd-stable@freebsd.org mailing list=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"= =0A= >=0A= =0A= _______________________________________________=0A= freebsd-stable@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"= =0A= From owner-freebsd-stable@freebsd.org Sat Dec 21 07:32:12 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A61861CA315 for ; Sat, 21 Dec 2019 07:32:12 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47fy4C1YZzz3yc1 for ; Sat, 21 Dec 2019 07:32:10 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cs.huji.ac.il; s=57791128; h=References:To:Cc:In-Reply-To:Date:Subject:Mime-Version:Content-Type:Message-Id:From; bh=PYsCHz9/hVNGO4IBklUJOVUriTIX/rq5baeVj7IIAFw=; b=bKoufoifMfHtAa7je020pdtJFAyL3CCUjZP/LMWfRYChxzm2szziewQvPe7/18Z3WseSShSVaSSaajMTsA9XtfOy+ypS3Vji8HP1ISEiFKvSUD+27Vbc0x/Ps1hNUxcUnt2tUm//oPHXoOosPNeU3xFSUBhAq3E4MT26Qj7Sg0myFsHz91U9lCusbTv3OOgGMQw1hszjsSSRCLSq3HHWVwG8PfZxe8B8oJB7QNRA6fv/qO/ctFr3MWw8mx8NVaNXKix+dJxFnRBTCTzyUNgHo43ll7LFHZyV/CiCc0ChqhSZ7W7s0Ea15we2rmCiYvzyiswcIjql0C/NRYSqIvBhtA==; Received: from macmini.bk.cs.huji.ac.il ([132.65.179.19]) by kabab.cs.huji.ac.il with esmtp id 1iiZF7-000HCt-Og; Sat, 21 Dec 2019 09:32:05 +0200 From: Daniel Braniss Message-Id: <8A78F67B-C244-45CF-B9BF-D7062669B33B@cs.huji.ac.il> Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.40.2.2.4\)) Subject: Re: nfs lockd errors after NetApp software upgrade. Date: Sat, 21 Dec 2019 09:32:05 +0200 In-Reply-To: Cc: Adam McDougall , "freebsd-stable@freebsd.org" To: Rick Macklem References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> X-Mailer: Apple Mail (2.3608.40.2.2.4) X-Rspamd-Queue-Id: 47fy4C1YZzz3yc1 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.huji.ac.il header.s=57791128 header.b=bKoufoif; dmarc=pass (policy=none) header.from=huji.ac.il; spf=none (mx1.freebsd.org: domain of danny@cs.huji.ac.il has no SPF policy when checking 132.65.116.210) smtp.mailfrom=danny@cs.huji.ac.il X-Spamd-Result: default: False [-3.86 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[cs.huji.ac.il:s=57791128]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; IP_SCORE(-1.56)[ip: (-3.85), ipnet: 132.64.0.0/13(-2.23), asn: 378(-1.79), country: IL(0.05)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[cs.huji.ac.il:+]; DMARC_POLICY_ALLOW(-0.50)[huji.ac.il,none]; RCVD_IN_DNSWL_NONE(0.00)[210.116.65.132.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:378, ipnet:132.64.0.0/13, country:IL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Dec 2019 07:32:12 -0000 > On 20 Dec 2019, at 19:19, Rick Macklem wrote: >=20 > Adam McDougall wrote: >> Try changing bool_t do_tcp =3D FALSE; to TRUE in >> /usr/src/sys/nlm/nlm_prot_impl.c, recompile the kernel and try again. = I >> think this makes it match Linux client behavior. I suspect I ran into >> the same issue as you. I do think I used nolockd is a workaround >> temporarily. I can provide some more details if it works. > If this fixes the problem, please let me know. >=20 > I'm not sure I'd want to change the default, since it might break = things for > others, but I can definitely make it a tunable, so that people don't = need to > recompile a kernel to deal with it. >=20 great! I was just about to see how it can be done(tunable) but need to = check if it can be done at any time, or just at boot time. thanks. btw, currently, from several hours of analysing the traffic, it seems = that nlm is UDP. danny > rick >=20 > On 12/19/19 9:21 AM, Daniel Braniss wrote: >>=20 >>=20 >>> On 19 Dec 2019, at 16:09, Rick Macklem wrote: >>>=20 >>> Daniel Braniss wrote: >>> [stuff snipped] >>>> all mounts are nfsv3/tcp >>> This doesn't affect what the NLM code (rpc.lockd) uses. I honestly = don't know when >>> the NLM uses tcp vs udp. I think rpc.statd still uses IP broadcast = at times. >> can the replay cache have any influence here? I tend to remember way = back issues >> with it, >>>=20 >>> To me, it looks like a network configuration issue. >> that was/is my gut feelings too, but, as far as we can tell, nothing = has changed in the network infrastructure, >> the problems appeared after the NetAPP=E2=80=99s software was = updated, it was working fine till then. >>=20 >> the problems are also happening on freebsd 12.1 >>=20 >>> You could capture packets (maybe when a client first starts = rpc.statd and rpc.lockd) >>> and then look at them in wireshark. I'd disable statup of rpc.lockd = and rpc.statd >>> at boot for a test client and then run something like: >>> # tcpdump -s 0 -s out.pcap host >>> - and then start rpc.statd and rpc.lockd >>> Then I'd look at out.pcap in wireshark (much better at decoding this = stuff than >>> tcpdump). I'd look for things like different reply IP addresses from = the Netapp, >>> which might confuse this tired old NLM protocol Sun devised in the = mid-1980s. >>>=20 >> it=E2=80=99s going to be an interesting week end :-( >>=20 >>>> the error is also appearing on freebsd-11.2-stable, I=E2=80=99m now = checking if it=E2=80=99s also >>>> happening on 12.1 >>>> btw, the NetApp version is 9.3P17 >>> Yes. I wasn't the author of the NSM and NLM code (long ago I refused = to even >>> try to implement it, because I knew the protocol was badly broken) = and I avoid >>> fiddling with. As such, it won't have change much since around = FreeBSD7. >> and we haven=E2=80=99t had any issues with it for years, so you must = have done something good >>=20 >> cheers, >> danny >>=20 >>>=20 >>> rick >>>=20 >>> cheers, >>> danny >>>=20 >>>> rick >>>>=20 >>>> Cheers >>>>=20 >>>> Richard >>>> (NetApp admin) >>>>=20 >>>> On Wed, 18 Dec 2019 at 15:46, Daniel Braniss = > wrote: >>>>=20 >>>>=20 >>>>> On 18 Dec 2019, at 16:55, Rick Macklem = > wrote: >>>>>=20 >>>>> Daniel Braniss wrote: >>>>>=20 >>>>>> Hi, >>>>>> The server with the problems is running FreeBSD 11.1 stable, it = was working fine for >several months, >>>>>> but after a software upgrade of our NetAPP server it=E2=80=99s = reporting many lockd errors >and becomes catatonic, >>>>>> ... >>>>>> Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd = not responding >>>>>> Dec 18 13:11:45 moo-09 last message repeated 7 times >>>>>> Dec 18 13:12:55 moo-09 last message repeated 8 times >>>>>> Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd = is alive again >>>>>> Dec 18 13:13:10 moo-09 last message repeated 8 times >>>>>> Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 194 already in queue awaiting acceptance (1 = occurrences) >>>>>> Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 193 already in queue awaiting acceptance (3957 = occurrences) >>>>>> Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 193 already in queue awaiting acceptance =E2=80=A6= >>>>> Seems like their software upgrade didn't improve handling of NLM = RPCs? >>>>> Appears to be handling RPCs slowly and/or intermittently. Note = that no one >>>>> tests it with IPv6, so at least make sure you are still using IPv4 = for the mounts and >>>>> try and make sure IP broadcast works between client and Netapp. I = think the NLM >>>>> and NSM (rpc.statd) still use IP broadcast sometimes. >>>>>=20 >>>> we are ipv4 - we have our own class c :-) >>>>> Maybe the network guys can suggest more w.r.t. why, but as I've = stated before, >>>>> the NLM is a fundamentally broken protocol which was never = published by Sun, >>>>> so I suggest you avoid using it if at all possible. >>>> well, at the moment the ball is on NetAPP court, and switching to = NFSv4 at the moment is out of the question, it=E2=80=99s >>>> a production server used by several thousand students. >>>>=20 >>>>>=20 >>>>> - If the locks don't need to be seen by other clients, you can = just use the "nolockd" >>>>> mount option. >>>>> or >>>>> - If locks need to be seen by other clients, try NFSv4 mounts. = Netapp filers >>>>> should support NFSv4.1, which is a much better protocol that = NFSv4.0. >>>>>=20 >>>>> Good luck with it, rick >>>> thanks >>>> danny >>>>=20 >>>>> =E2=80=A6 >>>>> any ideas? >>>>>=20 >>>>> thanks, >>>>> danny >>>>>=20 >>>>> _______________________________________________ >>>>> freebsd-stable@freebsd.org = mailing list >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >>>>=20 >>>> _______________________________________________ >>>> freebsd-stable@freebsd.org = mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >>>=20 >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >>=20 >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-stable@freebsd.org mailing = list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable = > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org = " From owner-freebsd-stable@freebsd.org Sat Dec 21 17:32:19 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3716A1D8AF0 for ; Sat, 21 Dec 2019 17:32:19 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660077.outbound.protection.outlook.com [40.107.66.77]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47gCNd6xFQz4Swf for ; Sat, 21 Dec 2019 17:32:17 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SG8yW+mqBCqr/uEccUQZlTwFeN13NN90RtOJKkqfaloz5Q6A1MT2lLMTogfDAbgtINB5mz6T4wGeL+jtAOXad2ptpXhlV3F1uZQVOd+n1pVytZfjSDP1AZhgHTdIPStY8/uRK4UlGykhzoVgCwwr189fSfvg722mq+c/PwtF2DHoMbe2pyXvsq6C0OcxN7llOzc44jq+TYUjfZ+Tz9vR83kh4XD1fimk5LfStphy3VbWM8ikQF0toE8lXHXt3N3YxA2tH9SfQqrjf9USYgBVE2/2Q6uL8qFsA+bV3OWMOcLtn5ZiRQxu3yu8pbk6xHY4GnnQ/nkZ9qygjWOl+oM5HA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hke4OU7Fvr9JZyzyscup2t+HULyvo6jYKOpP4USVOQs=; b=ll5SF5dRJPk9inR0P1O8qCwbWQ5Qj4WX25NOINo46bpaxlrX0IL/K7IJOGWGTcVJAC7VBuHvMVuBpp0AoDaCCcW+kHIG/J93AepTEeas0M0KVSkHD7WVKUXTkPqpoLapICAPau30qINN4JcVHA+Fy8WQ46EXbwJ7MX5Uta6OtK5PaIHwFao3n0Jgy7Bh+9GQ+Iahhksvn29KVrJMQTdlURJTaDHVwv72pOoOVJV9SyJEjghuqO8YhkDqOzIFE9Z3KSoHriAf1XrBQCxF5Ea3SnLRHCWg/5ecISjzqIU81zWLxpUBx1TvmT6uU9Tl9BOJiZ0nJTvMBo1nVLsN4X0z+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM (52.132.69.153) by YQBPR0101MB1012.CANPRD01.PROD.OUTLOOK.COM (52.132.66.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.15; Sat, 21 Dec 2019 17:32:16 +0000 Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75]) by YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75%5]) with mapi id 15.20.2538.023; Sat, 21 Dec 2019 17:32:15 +0000 From: Rick Macklem To: Daniel Braniss CC: Adam McDougall , "freebsd-stable@freebsd.org" Subject: Re: nfs lockd errors after NetApp software upgrade. Thread-Topic: nfs lockd errors after NetApp software upgrade. Thread-Index: AQHVtawq+ga5QLcdVkqBDG/GW9zFg6e/+Am+gAARTACAAANHAIAAi7Y3gACf34CAAEVO6IAABk4AgADWGACAAO1eZYAA7uGAgACmPw0= Date: Sat, 21 Dec 2019 17:32:15 +0000 Message-ID: References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> , <8A78F67B-C244-45CF-B9BF-D7062669B33B@cs.huji.ac.il> In-Reply-To: <8A78F67B-C244-45CF-B9BF-D7062669B33B@cs.huji.ac.il> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a1c7e0ec-816c-4713-b9a1-08d7863bb8da x-ms-traffictypediagnostic: YQBPR0101MB1012: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:5236; x-forefront-prvs: 0258E7CCD4 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(39860400002)(346002)(366004)(376002)(136003)(189003)(199004)(86362001)(33656002)(66446008)(66476007)(64756008)(66556008)(71200400001)(478600001)(66946007)(8936002)(52536014)(966005)(5660300002)(76116006)(6916009)(54906003)(2906002)(4326008)(186003)(55016002)(6506007)(81166006)(9686003)(53546011)(316002)(7696005)(786003)(8676002)(81156014); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB1012; H:YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: B2Wwqf1FhMUVRi+wsZvqVRMdhj3qHiUza4odjCGvrroRw/MBKH2I7M0Vo4zN7hs4e6fmodM3r3K2+Y24jvRTf/kOgegO/fxCmcOs5hFBxzt1f6k7Nw8cQDBYcQmaLcbp5RDcRlKjfDkXb3LDKH8lSQ9VxEKln58Fz2274zN7XWgIeUuzTGR/ixuj1NfnVjQ7h2jjaHnCMAU/CFWNGzF8I3KFOZujEBow+jFJnHLQYZ8LJJrCeOAzgtQBbjLF/81jfiJFckwzKpiQyBxGxF3OremnqV9U9oWfErnUjg80E9gIpEj4sCo4nKJArARkrcHtk+3cwdjYAUZZUsmBDF/Gxc1Rn7X/HvEsFgN67zQxYSI1iLhnOFBN5LTbH3EazibUynLg58vAcA5Q4xlta0uXX1Ig0QQ60gmDU1nmadEK5gxife/DDz2EwoctOmh06PpI7MGgwnS3qI65hjalCPZDXZY0CGsG0b6DpTo0Er0Vpwc= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: a1c7e0ec-816c-4713-b9a1-08d7863bb8da X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Dec 2019 17:32:15.7150 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Njl4Kw1Vs428k5da44LKrNH6FisoTnTvxhkUUOSDvU2yJEBlG+rcxQU5vijlYMshotn2jecMZrBESzbVqvqDHg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1012 X-Rspamd-Queue-Id: 47gCNd6xFQz4Swf X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.77 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.66 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[uoguelph.ca]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[77.66.107.40.list.dnswl.org : 127.0.3.0]; IP_SCORE(-1.36)[ipnet: 40.64.0.0/10(-3.84), asn: 8075(-2.92), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Dec 2019 17:32:19 -0000 Daniel Braniss wrote:=0A= >>On 20 Dec 2019, at 19:19, Rick Macklem >>> wrote:=0A= >>=0A= >>Adam McDougall wrote:=0A= >>>Try changing bool_t do_tcp =3D FALSE; to TRUE in=0A= >>>/usr/src/sys/nlm/nlm_prot_impl.c, recompile the kernel and try again. I= =0A= >>>think this makes it match Linux client behavior. I suspect I ran into=0A= >>>the same issue as you. I do think I used nolockd is a workaround=0A= >>>temporarily. I can provide some more details if it works.=0A= >>If this fixes the problem, please let me know.=0A= >>=0A= >>I'm not sure I'd want to change the default, since it might break things = for=0A= >>others, but I can definitely make it a tunable, so that people don't need= to=0A= >>recompile a kernel to deal with it.=0A= >>=0A= >>=0A= >great! I was just about to see how it can be done(tunable) but need to che= ck if it can >be done=0A= >at any time, or just at boot time.=0A= I haven't looked at the code, but I suspect changing it on the fly could ca= use problems,=0A= so I am inclined to make it a tunable (boot time only).=0A= =0A= >thanks.=0A= >btw, currently, from several hours of analysing the traffic, it seems that= nlm is UDP.=0A= I assume that means you haven't tried flipping it to TCP yet.=0A= =0A= Please let us know how it goes, rick=0A= =0A= danny=0A= =0A= =0A= rick=0A= =0A= On 12/19/19 9:21 AM, Daniel Braniss wrote:=0A= =0A= =0A= On 19 Dec 2019, at 16:09, Rick Macklem > wrote:=0A= =0A= Daniel Braniss wrote:=0A= [stuff snipped]=0A= all mounts are nfsv3/tcp=0A= This doesn't affect what the NLM code (rpc.lockd) uses. I honestly don't kn= ow when=0A= the NLM uses tcp vs udp. I think rpc.statd still uses IP broadcast at times= .=0A= can the replay cache have any influence here? I tend to remember way back i= ssues=0A= with it,=0A= =0A= To me, it looks like a network configuration issue.=0A= that was/is my gut feelings too, but, as far as we can tell, nothing has ch= anged in the network infrastructure,=0A= the problems appeared after the NetAPP=92s software was updated, it was wor= king fine till then.=0A= =0A= the problems are also happening on freebsd 12.1=0A= =0A= You could capture packets (maybe when a client first starts rpc.statd and r= pc.lockd)=0A= and then look at them in wireshark. I'd disable statup of rpc.lockd and rpc= .statd=0A= at boot for a test client and then run something like:=0A= # tcpdump -s 0 -s out.pcap host =0A= - and then start rpc.statd and rpc.lockd=0A= Then I'd look at out.pcap in wireshark (much better at decoding this stuff = than=0A= tcpdump). I'd look for things like different reply IP addresses from the Ne= tapp,=0A= which might confuse this tired old NLM protocol Sun devised in the mid-1980= s.=0A= =0A= it=92s going to be an interesting week end :-(=0A= =0A= the error is also appearing on freebsd-11.2-stable, I=92m now checking if i= t=92s also=0A= happening on 12.1=0A= btw, the NetApp version is 9.3P17=0A= Yes. I wasn't the author of the NSM and NLM code (long ago I refused to eve= n=0A= try to implement it, because I knew the protocol was badly broken) and I av= oid=0A= fiddling with. As such, it won't have change much since around FreeBSD7.=0A= and we haven=92t had any issues with it for years, so you must have done so= mething good=0A= =0A= cheers,=0A= danny=0A= =0A= =0A= rick=0A= =0A= cheers,=0A= danny=0A= =0A= rick=0A= =0A= Cheers=0A= =0A= Richard=0A= (NetApp admin)=0A= =0A= On Wed, 18 Dec 2019 at 15:46, Daniel Braniss > wrote:=0A= =0A= =0A= On 18 Dec 2019, at 16:55, Rick Macklem > wrote:=0A= =0A= Daniel Braniss wrote:=0A= =0A= Hi,=0A= The server with the problems is running FreeBSD 11.1 stable, it was working= fine for >several months,=0A= but after a software upgrade of our NetAPP server it=92s reporting many loc= kd errors >and becomes catatonic,=0A= ...=0A= Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not respond= ing=0A= Dec 18 13:11:45 moo-09 last message repeated 7 times=0A= Dec 18 13:12:55 moo-09 last message repeated 8 times=0A= Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is alive ag= ain=0A= Dec 18 13:13:10 moo-09 last message repeated 8 times=0A= Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen qu= eue >overflow: 194 already in queue awaiting acceptance (1 occurrences)=0A= Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen qu= eue >overflow: 193 already in queue awaiting acceptance (3957 occurrences)= =0A= Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen qu= eue >overflow: 193 already in queue awaiting acceptance =85=0A= Seems like their software upgrade didn't improve handling of NLM RPCs?=0A= Appears to be handling RPCs slowly and/or intermittently. Note that no one= =0A= tests it with IPv6, so at least make sure you are still using IPv4 for the = mounts and=0A= try and make sure IP broadcast works between client and Netapp. I think the= NLM=0A= and NSM (rpc.statd) still use IP broadcast sometimes.=0A= =0A= we are ipv4 - we have our own class c :-)=0A= Maybe the network guys can suggest more w.r.t. why, but as I've stated befo= re,=0A= the NLM is a fundamentally broken protocol which was never published by Sun= ,=0A= so I suggest you avoid using it if at all possible.=0A= well, at the moment the ball is on NetAPP court, and switching to NFSv4 at = the moment is out of the question, it=92s=0A= a production server used by several thousand students.=0A= =0A= =0A= - If the locks don't need to be seen by other clients, you can just use the= "nolockd"=0A= mount option.=0A= or=0A= - If locks need to be seen by other clients, try NFSv4 mounts. Netapp filer= s=0A= should support NFSv4.1, which is a much better protocol that NFSv4.0.=0A= =0A= Good luck with it, rick=0A= thanks=0A= danny=0A= =0A= =85=0A= any ideas?=0A= =0A= thanks,=0A= danny=0A= =0A= _______________________________________________=0A= freebsd-stable@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"=0A= =0A= _______________________________________________=0A= freebsd-stable@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"=0A= =0A= =0A= _______________________________________________=0A= freebsd-stable@freebsd.org mailing list= =0A= https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"= =0A= =0A= =0A= _______________________________________________=0A= freebsd-stable@freebsd.org mailing list= =0A= https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"=0A= _______________________________________________=0A= freebsd-stable@freebsd.org mailing list= =0A= https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"=0A= =0A=