From owner-freebsd-current@freebsd.org Sun May 17 05:35:07 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7A77413E2AB for ; Sun, 17 May 2020 05:35:07 +0000 (UTC) (envelope-from bsd-lists@BSDforge.com) Received: from udns.ultimatedns.net (static-24-113-41-81.wavecable.com [24.113.41.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "ultimatedns.net", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 49PrSp6Pxmz3gps for ; Sun, 17 May 2020 05:35:06 +0000 (UTC) (envelope-from bsd-lists@BSDforge.com) Received: from udns.ultimatedns.net (localhost [IPv6:0:0:0:0:0:0:0:1]) by udns.ultimatedns.net (8.15.2/8.15.2) with ESMTPS id 04H5ZQxn018946 (version=TLSv1.2 cipher=DHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 16 May 2020 22:35:32 -0700 (PDT) (envelope-from bsd-lists@BSDforge.com) X-Mailer: Cypht MIME-Version: 1.0 From: Chris Reply-To: bsd-lists@BSDforge.com To: freebsd-current Subject: installworld failure on r361107 (curs_addch.3.filt) Date: Sat, 16 May 2020 22:35:32 -0700 Message-Id: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 49PrSp6Pxmz3gps X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; ASN(0.00)[asn:11404, ipnet:24.113.0.0/16, country:US]; local_wl_ip(0.00)[24.113.41.81] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 May 2020 05:35:07 -0000 On a recent install from FreeBSD-13=2E0-CURRENT-amd64-20200514-r361019-disc1=2E= iso I intitated a build/install world/kernel from src @r361107 The build of world and kernel went as anticipated, as did the kernel install=2E After booting to single user for the installworld=2E The install failed with: =2E=2E=2E dll=2Eh /usr/include/ install -l s -o root -g wheel -m 755 curses=2Eh /usr/include/ncurses=2Eh install -o root -g wheel -m 444 curs_addch=2E3=2Efilt /usr/share/man/man3/cur= s_add ch=2E3 install: curs_addch=2E3=2Efilt: No such file or directory *** Error code 71 Stop=2E make[6]: stopped in /usr/src/lib/ncurses/ncursesw *** Error code 1 Stop=2E make[5]: stopped in /usr/src/lib/ncurses *** Error code 1 Stop=2E make[4]: stopped in /usr/src/lib *** Error code 1 Stop=2E make[3]: stopped in /usr/src *** Error code 1 There nothing interesting/unusual in src=2Econf(5) and make=2Econf doesn't exist=2E Any chance for recovery? Or will I need initiate a new install, and hope for the best checking out a newer revision of src=2E Thanks in advance=2E --Chris From owner-freebsd-current@freebsd.org Sun May 17 06:29:32 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 81FD313FF20 for ; Sun, 17 May 2020 06:29:32 +0000 (UTC) (envelope-from bsd-lists@BSDforge.com) Received: from udns.ultimatedns.net (static-24-113-41-81.wavecable.com [24.113.41.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "ultimatedns.net", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 49Psgc2B7Tz411w for ; Sun, 17 May 2020 06:29:31 +0000 (UTC) (envelope-from bsd-lists@BSDforge.com) Received: from udns.ultimatedns.net (localhost [IPv6:0:0:0:0:0:0:0:1]) by udns.ultimatedns.net (8.15.2/8.15.2) with ESMTPS id 04H6Tqbc019720 (version=TLSv1.2 cipher=DHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 16 May 2020 23:29:59 -0700 (PDT) (envelope-from bsd-lists@BSDforge.com) X-Mailer: Cypht MIME-Version: 1.0 In-Reply-To: From: Chris Reply-To: bsd-lists@BSDforge.com To: freebsd-current Subject: Re: installworld failure on r361107 (curs_addch.3.filt) Date: Sat, 16 May 2020 23:29:58 -0700 Message-Id: <39e78c2189af9987846c18170dac4eea@udns.ultimatedns.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 49Psgc2B7Tz411w X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; ASN(0.00)[asn:11404, ipnet:24.113.0.0/16, country:US]; local_wl_ip(0.00)[24.113.41.81] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 May 2020 06:29:32 -0000 On Sat, 16 May 2020 22:35:32 -0700 bsd-lists@BSDforge=2Ecom said Pilot error=2E=2E=2E > On a recent install from > FreeBSD-13=2E0-CURRENT-amd64-20200514-r361019-disc1=2Eiso > I intitated a build/install world/kernel from src @r361107 >=20 > The build of world and kernel went as anticipated, as did > the kernel install=2E After booting to single user for the > installworld=2E The install failed with: > =2E=2E=2E > dll=2Eh /usr/include/ > install -l s -o root -g wheel -m 755 curses=2Eh /usr/include/ncurses=2Eh > install -o root -g wheel -m 444 curs_addch=2E3=2Efilt=20 > /usr/share/man/man3/curs_add > ch=2E3 > install: curs_addch=2E3=2Efilt: No such file or directory > *** Error code 71 >=20 > Stop=2E > make[6]: stopped in /usr/src/lib/ncurses/ncursesw > *** Error code 1 >=20 > Stop=2E > make[5]: stopped in /usr/src/lib/ncurses > *** Error code 1 >=20 > Stop=2E > make[4]: stopped in /usr/src/lib > *** Error code 1 >=20 > Stop=2E > make[3]: stopped in /usr/src > *** Error code 1 >=20 > There nothing interesting/unusual in src=2Econf(5) > and make=2Econf doesn't exist=2E OK This was a case of pilot error=2E After the buildworld I realized that I hadn't added WITHOUT_MANCOMPRESS=2E Which I wanted=2E So being of the belief that both non-compressed, as well as compressed pages had been created during the buildworld=2E I simply added it before installworld=2E Knowing it would pick up the change=2E Well, as it turns out, both copies do *not* exist=2E End of story, and sorry for the noise=2E --Chris >=20 > Any chance for recovery? Or will I need initiate a new > install, and hope for the best checking out a newer revision > of src=2E >=20 > Thanks in advance=2E >=20 > --Chris >=20 >=20 > _______________________________________________ > freebsd-current@freebsd=2Eorg mailing list > https://lists=2Efreebsd=2Eorg/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd=2Eorg= " From owner-freebsd-current@freebsd.org Tue May 19 01:06:59 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 939272F1F4E; Tue, 19 May 2020 01:06:59 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49QyQW3H06z4cxY; Tue, 19 May 2020 01:06:59 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 1129) id 593E119B71; Tue, 19 May 2020 01:06:59 +0000 (UTC) Date: Tue, 19 May 2020 01:06:59 +0000 From: Li-Wen Hsu To: freebsd-testing@freebsd.org Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: FreeBSD CI Weekly Report 2020-05-10 Message-ID: <20200519010659.GA95306@freefall.freebsd.org> Reply-To: freebsd-testing@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 May 2020 01:06:59 -0000 (Please send the followup to freebsd-testing@ and note Reply-To is set.) FreeBSD CI Weekly Report 2020-05-10 =================================== Here is a summary of the FreeBSD Continuous Integration results for the period from 2020-05-04 to 2020-05-10. During this period, we have: * 2003 builds (92.1% (-1.5) passed, 7.9% (+1.5) failed) of buildworld and buildkernel (GENERIC and LINT) were executed on aarch64, amd64, armv6, armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, riscv64, sparc64 architectures for head, stable/12, stable/11 branches. * 283 test runs (23.7% (-44.5) passed, 35.0% (+6.9) unstable, 41.3% (+37.6) exception) were executed on amd64, i386, riscv64 architectures for head, stable/12, stable/11 branches. * 68 doc and www builds (79.4% (-5.2) passed, 20.6% (+5.2) failed) Test case status (on 2020-05-10 23:59): | Branch/Architecture | Total | Pass | Fail | Skipped | | ------------------- | --------- | ---------- | -------- | ------- | | head/amd64 | 7782 (+4) | 7692 (+7) | 0 (0) | 90 (-3) | | head/i386 | 7780 (+4) | 7682 (+6) | 0 (0) | 98 (-2) | | 12-STABLE/amd64 | 7521 (0) | 7460 (-3) | 0 (-2) | 61 (+5) | | 12-STABLE/i386 | 7519 (0) | 7453 (0) | 0 (-2) | 66 (+2) | | 11-STABLE/amd64 | 6883 (0) | 6833 (-1) | 1 (+1) | 49 (0) | | 11-STABLE/i386 | 6881 (0) | 6816 (-14) | 14 (+14) | 51 (0) | (The statistics from experimental jobs are omitted) If any of the issues found by CI are in your area of interest or expertise please investigate the PRs listed below. The latest web version of this report is available at https://hackmd.io/@FreeBSD-CI/report-20200510 and archive is available at https://hackmd.io/@FreeBSD-CI/ , any help is welcome. ## News * Several non-x86 jobs for -head are added: * https://ci.freebsd.org/job/FreeBSD-head-aarch64-test/ * https://ci.freebsd.org/job/FreeBSD-head-armv7-test/ * https://ci.freebsd.org/job/FreeBSD-head-mips64-test/ * https://ci.freebsd.org/job/FreeBSD-head-powerpc64-test/ Most of the work are done by trasz@ These jobs will be scheduled to run at least once a day after the dedicated VM host has been setup. ## Fixed test cases * lib.libproc.proc_test.symbol_lookup Fixed in https://svnweb.freebsd.org/changeset/base/360979 ## Fixed and fixed test cases * lib.atf.* and libexec.atf.* on stable/11 Failed after llvm9 merged, fixed after build https://ci.freebsd.org/job/FreeBSD-stable-11-i386-test/2662/ with fixeds merged. ## Failing jobs * https://ci.freebsd.org/job/FreeBSD-head-amd64-gcc6_build/ ``` /usr/local/bin/x86_64-unknown-freebsd12.1-ld: /tmp/obj/workspace/src/amd64.amd64/lib/clang/liblldb/liblldb.a(IOHandlerCursesGUI.o): in function `curses::Window::Box(unsigned int, unsigned int)': /workspace/src/contrib/llvm-project/lldb/source/Core/IOHandlerCursesGUI.cpp:361: undefined reference to `box' /usr/local/bin/x86_64-unknown-freebsd12.1-ld: /workspace/src/contrib/llvm-project/lldb/source/Core/IOHandlerCursesGUI.cpp:361: undefined reference to `box' collect2: error: ld returned 1 exit status ``` ## Regressions * (head, stable/12, stable/11) 3 tests start failing after llvm10 import * lib.msun.ctrig_test.test_inf_inputs https://bugs.freebsd.org/244732 * (DTrace) common.pid.t_dtrace_contrib.err_D_PROC_OFF_toobig_d https://bugs.freebsd.org/244823 * Lock-order reversals triggered by tests under sys.net.if_lagg_test.* on i386 https://bugs.freebsd.org/244163 Discovered by newly endabled sys.net.* tests. ([r357857](https://svnweb.freebsd.org/changeset/base/357857)) * sys.net.if_lagg_test.lacp_linkstate_destroy_stress panics i386 kernel https://bugs.freebsd.org/244168 Discovered by newly endabled sys.net.* tests. ([r357857](https://svnweb.freebsd.org/changeset/base/357857)) * (test case) sys.geom.class.multipath.misc.fail_on_error (on 12-STABLE) https://bugs.freebsd.org/244158 Will be fixed after r360807 merged to stable/12. ## Failing and Flaky tests (from experimental jobs) * https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/ * cddl.usr.sbin.dtrace.common.misc.t_dtrace_contrib.tst_dynopt_d * https://bugs.freebsd.org/237641 * cddl.usr.sbin.dtrace.common.pid.t_dtrace_contrib.err_D_PROC_OFF_toobig_d * https://bugs.freebsd.org/244823 * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/ * There are ~13 failing and ~109 skipped cases, including flakey ones, see https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/lastCompletedBuild/testReport/ for more details * Work for cleaning these failing cass are in progress * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_ltp/ * Total 3670 tests (0), 2285 success (0), 579 failures (0), 806 skipped (0) ## Disabled Tests * sys.fs.tmpfs.mount_test.large https://bugs.freebsd.org/212862 * sys.fs.tmpfs.link_test.kqueue https://bugs.freebsd.org/213662 * sys.kqueue.libkqueue.kqueue_test.main https://bugs.freebsd.org/233586 * sys.kern.ptrace_test.ptrace__PT_KILL_competing_stop https://bugs.freebsd.org/220841 * lib.libc.regex.exhaust_test.regcomp_too_big (i386 only) https://bugs.freebsd.org/237450 * sys.netinet.socket_afinet.socket_afinet_bind_zero https://bugs.freebsd.org/238781 * sys.netpfil.pf.names.names * sys.netpfil.pf.synproxy.synproxy https://bugs.freebsd.org/238870 * sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger https://bugs.freebsd.org/239292 * sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger https://bugs.freebsd.org/239397 * sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger https://bugs.freebsd.org/239399 * sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger https://bugs.freebsd.org/239425 * lib.libc.gen.getmntinfo_test.getmntinfo_test https://bugs.freebsd.org/240049 * sys.sys.qmath_test.qdivq_s64q https://bugs.freebsd.org/240219 * sys.kern.ptrace_test.ptrace__getppid https://bugs.freebsd.org/240510 * lib.libc.sys.stat_test.stat_socket https://bugs.freebsd.org/240621 * lib.libarchive.functional_test.test_write_filter_zstd https://bugs.freebsd.org/240683 * lib.libcasper.services.cap_dns.dns_test.main https://bugs.freebsd.org/241435 * local.kyua.* (31 cases) & local.lutok.* (3 cases) on 11-i386 https://ci.freebsd.org/job/FreeBSD-stable-11-i386-test/2278/testReport/ * sys.geom.class.multipath.failloop.failloop https://bugs.freebsd.org/242689 * sys.kern.ptrace_test.ptrace__procdesc_reparent_wait_child https://bugs.freebsd.org/243605 * sys.kern.ptrace_test.ptrace__parent_wait_after_attach https://bugs.freebsd.org/244055 * sys.kern.ptrace_test.ptrace__parent_exits_before_child https://bugs.freebsd.org/244056 * sys.geom.class.multipath.misc.fail_on_error (12-STABLE) https://bugs.freebsd.org/244158 * sys.net.if_lagg_test.witness (i386) https://bugs.freebsd.org/244163 * PipePdfork.WildcardWait in sys.capsicum.capsicum-test.main https://bugs.freebsd.org/244165 * sys.net.if_lagg_test.lacp_linkstate_destroy_stress (i386) https://bugs.freebsd.org/244168 * sys.netinet6.frag6.frag6_07.frag6_07 https://bugs.freebsd.org/244170 * sys.netinet.fibs_test.udp_dontroute6 https://bugs.freebsd.org/244172 * sys.netpfil.pf.nat.exhaust https://bugs.freebsd.org/244703 * sys.geom.class.gate.ggate_test.ggated https://bugs.freebsd.org/244737 * sys.netipsec.tunnel.empty.v{4,6} https://bugs.freebsd.org/245832 ## Issues ### Cause build fails * https://bugs.freebsd.org/233735 Possible build race: genoffset.o /usr/src/sys/sys/types.h: error: machine/endian.h: No such file or directory * https://bugs.freebsd.org/233769 Possible build race: ld: error: unable to find library -lgcc_s ### Cause kernel panics * https://bugs.freebsd.org/238870 sys.netpfil.pf.names.names and sys.netpfil.pf.synproxy.synproxy cause panic Patch exists: * https://reviews.freebsd.org/D20868 * https://reviews.freebsd.org/D20869 ### Open * https://bugs.freebsd.org/237403 Tests in sys/opencrypto should be converted to Python3 * https://bugs.freebsd.org/237641 Flakey test case: common.misc.t_dtrace_contrib.tst_dynopt_d * https://bugs.freebsd.org/237656 "Freed UMA keg (rtentry) was not empty (18 items). Lost 1 pages of memory." seen when running sys/netipsec tests * https://bugs.freebsd.org/238781 sys.netinet.socket_afinet.socket_afinet_bind_zero does not work when mac_portacl(4) loaded * https://bugs.freebsd.org/239292 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger * https://bugs.freebsd.org/239397 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger * https://bugs.freebsd.org/239399 Flakey test case: sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger * https://bugs.freebsd.org/239425 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger * https://bugs.freebsd.org/241662 Flakey test case: lib.libarchive.functional_test.test_fuzz_iso9660 * https://bugs.freebsd.org/246443 (new) sys.net.if_clone_test.epair_stress sometimes exceeds timeout limit but not caught by kyua ### Others * [Tickets related to testing@](https://preview.tinyurl.com/y9maauwg) From owner-freebsd-current@freebsd.org Tue May 19 01:08:18 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8C5A82F278F; Tue, 19 May 2020 01:08:18 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49QyS22qZ9z4dFs; Tue, 19 May 2020 01:08:18 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 1129) id 5509119E56; Tue, 19 May 2020 01:08:18 +0000 (UTC) Date: Tue, 19 May 2020 01:08:18 +0000 From: Li-Wen Hsu To: freebsd-testing@freebsd.org Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: FreeBSD CI Weekly Report 2020-05-17 Message-ID: <20200519010818.GB95306@freefall.freebsd.org> Reply-To: freebsd-testing@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 May 2020 01:08:18 -0000 (Please send the followup to freebsd-testing@ and note Reply-To is set.) FreeBSD CI Weekly Report 2020-05-17 =================================== Here is a summary of the FreeBSD Continuous Integration results for the period from 2020-05-11 to 2020-05-17. During this period, we have: * 1977 builds (94.6% (+2.5) passed, 5.4% (-2.5) failed) of buildworld and buildkernel (GENERIC and LINT) were executed on aarch64, amd64, armv6, armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, riscv64, sparc64 architectures for head, stable/12, stable/11 branches. * 236 test runs (41.1% (+17.4) passed, 33.9% (-1.1) unstable, 25.0% (-16.3) exception) were executed on amd64, i386, riscv64 architectures for head, stable/12, stable/11 branches. * 33 doc and www builds (97.0% (+17.6) passed, 3.0% (-17.6) failed) Test case status (on 2020-05-17 23:59): | Branch/Architecture | Total | Pass | Fail | Skipped | | ------------------- | ---------- | ---------- | ------- | -------- | | head/amd64 | 7828 (+46) | 7737 (+45) | 1 (+1) | 90 (0) | | head/i386 | 7826 (+46) | 7725 (+43) | 0 (0) | 101 (+3) | | 12-STABLE/amd64 | 7542 (+21) | 7484 (+24) | 0 (0) | 58 (-3) | | 12-STABLE/i386 | 7540 (+21) | 7471 (+18) | 0 (0) | 69 (+3) | | 11-STABLE/amd64 | 6883 (0) | 6830 (-3) | 0 (-1) | 53 (+4) | | 11-STABLE/i386 | 6881 (0) | 6829 (+13) | 0 (-14) | 52 (+1) | (The statistics from experimental jobs are omitted) If any of the issues found by CI are in your area of interest or expertise please investigate the PRs listed below. The latest web version of this report is available at https://hackmd.io/@FreeBSD-CI/report-20200517 and archive is available at https://hackmd.io/@FreeBSD-CI/ , any help is welcome. ## Failing jobs * https://ci.freebsd.org/job/FreeBSD-head-amd64-gcc6_build/ ``` /usr/local/bin/x86_64-unknown-freebsd12.1-ld: /tmp/obj/workspace/src/amd64.amd64/lib/clang/liblldb/liblldb.a(IOHandlerCursesGUI.o): in function `curses::Window::Box(unsigned int, unsigned int)': /workspace/src/contrib/llvm-project/lldb/source/Core/IOHandlerCursesGUI.cpp:361: undefined reference to `box' /usr/local/bin/x86_64-unknown-freebsd12.1-ld: /workspace/src/contrib/llvm-project/lldb/source/Core/IOHandlerCursesGUI.cpp:361: undefined reference to `box' collect2: error: ld returned 1 exit status ``` ## Regressions * (new) lib.libexecinfo.backtrace_test.backtrace_fmt_basic starts failing on amd64 after r360915 https://bugs.freebsd.org/246537 * (head, stable/12, stable/11) 2 tests start failing after llvm10 import * lib.msun.ctrig_test.test_inf_inputs https://bugs.freebsd.org/244732 * (DTrace) common.pid.t_dtrace_contrib.err_D_PROC_OFF_toobig_d https://bugs.freebsd.org/244823 * Lock-order reversals triggered by tests under sys.net.if_lagg_test.* on i386 https://bugs.freebsd.org/244163 Discovered by newly endabled sys.net.* tests. ([r357857](https://svnweb.freebsd.org/changeset/base/357857)) * sys.net.if_lagg_test.lacp_linkstate_destroy_stress panics i386 kernel https://bugs.freebsd.org/244168 Discovered by newly endabled sys.net.* tests. ([r357857](https://svnweb.freebsd.org/changeset/base/357857)) * (test case) sys.geom.class.multipath.misc.fail_on_error (on 12-STABLE) https://bugs.freebsd.org/244158 Will be fixed after r360807 merged to stable/12. ## Failing and Flaky tests (from experimental jobs) * https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/ * cddl.usr.sbin.dtrace.common.misc.t_dtrace_contrib.tst_dynopt_d * https://bugs.freebsd.org/237641 * cddl.usr.sbin.dtrace.common.pid.t_dtrace_contrib.err_D_PROC_OFF_toobig_d * https://bugs.freebsd.org/244823 * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/ * There are ~13 failing and ~109 skipped cases, including flakey ones, see https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/lastCompletedBuild/testReport/ for more details * Work for cleaning these failing cass are in progress * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_ltp/ * Total 3670 tests (0), 2285 success (0), 579 failures (0), 806 skipped (0) ## Disabled Tests * sys.fs.tmpfs.mount_test.large https://bugs.freebsd.org/212862 * sys.fs.tmpfs.link_test.kqueue https://bugs.freebsd.org/213662 * sys.kqueue.libkqueue.kqueue_test.main https://bugs.freebsd.org/233586 * sys.kern.ptrace_test.ptrace__PT_KILL_competing_stop https://bugs.freebsd.org/220841 * lib.libc.regex.exhaust_test.regcomp_too_big (i386 only) https://bugs.freebsd.org/237450 * sys.netinet.socket_afinet.socket_afinet_bind_zero https://bugs.freebsd.org/238781 * sys.netpfil.pf.names.names * sys.netpfil.pf.synproxy.synproxy https://bugs.freebsd.org/238870 * sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger https://bugs.freebsd.org/239292 * sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger https://bugs.freebsd.org/239397 * sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger https://bugs.freebsd.org/239399 * sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger https://bugs.freebsd.org/239425 * lib.libc.gen.getmntinfo_test.getmntinfo_test https://bugs.freebsd.org/240049 * sys.sys.qmath_test.qdivq_s64q https://bugs.freebsd.org/240219 * sys.kern.ptrace_test.ptrace__getppid https://bugs.freebsd.org/240510 * lib.libc.sys.stat_test.stat_socket https://bugs.freebsd.org/240621 * lib.libarchive.functional_test.test_write_filter_zstd https://bugs.freebsd.org/240683 * lib.libcasper.services.cap_dns.dns_test.main https://bugs.freebsd.org/241435 * local.kyua.* (31 cases) & local.lutok.* (3 cases) on 11-i386 https://ci.freebsd.org/job/FreeBSD-stable-11-i386-test/2278/testReport/ * sys.geom.class.multipath.failloop.failloop https://bugs.freebsd.org/242689 * sys.kern.ptrace_test.ptrace__procdesc_reparent_wait_child https://bugs.freebsd.org/243605 * sys.kern.ptrace_test.ptrace__parent_wait_after_attach https://bugs.freebsd.org/244055 * sys.kern.ptrace_test.ptrace__parent_exits_before_child https://bugs.freebsd.org/244056 * sys.geom.class.multipath.misc.fail_on_error (12-STABLE) https://bugs.freebsd.org/244158 * sys.net.if_lagg_test.witness (i386) https://bugs.freebsd.org/244163 * PipePdfork.WildcardWait in sys.capsicum.capsicum-test.main https://bugs.freebsd.org/244165 * sys.net.if_lagg_test.lacp_linkstate_destroy_stress (i386) https://bugs.freebsd.org/244168 * sys.netinet6.frag6.frag6_07.frag6_07 https://bugs.freebsd.org/244170 * sys.netinet.fibs_test.udp_dontroute6 https://bugs.freebsd.org/244172 * sys.netpfil.pf.nat.exhaust https://bugs.freebsd.org/244703 * sys.geom.class.gate.ggate_test.ggated https://bugs.freebsd.org/244737 * sys.netipsec.tunnel.empty.v{4,6} https://bugs.freebsd.org/245832 ## Issues ### Cause build fails * https://bugs.freebsd.org/233735 Possible build race: genoffset.o /usr/src/sys/sys/types.h: error: machine/endian.h: No such file or directory * https://bugs.freebsd.org/233769 Possible build race: ld: error: unable to find library -lgcc_s ### Cause kernel panics * https://bugs.freebsd.org/238870 sys.netpfil.pf.names.names and sys.netpfil.pf.synproxy.synproxy cause panic Patch exists: * https://reviews.freebsd.org/D20868 * https://reviews.freebsd.org/D20869 ### Open * https://bugs.freebsd.org/237403 Tests in sys/opencrypto should be converted to Python3 * https://bugs.freebsd.org/237641 Flakey test case: common.misc.t_dtrace_contrib.tst_dynopt_d * https://bugs.freebsd.org/237656 "Freed UMA keg (rtentry) was not empty (18 items). Lost 1 pages of memory." seen when running sys/netipsec tests * https://bugs.freebsd.org/238781 sys.netinet.socket_afinet.socket_afinet_bind_zero does not work when mac_portacl(4) loaded * https://bugs.freebsd.org/239292 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger * https://bugs.freebsd.org/239397 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger * https://bugs.freebsd.org/239399 Flakey test case: sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger * https://bugs.freebsd.org/239425 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger * https://bugs.freebsd.org/241662 Flakey test case: lib.libarchive.functional_test.test_fuzz_iso9660 * https://bugs.freebsd.org/246443 sys.net.if_clone_test.epair_stress sometimes exceeds timeout limit but not caught by kyua ### Others * [Tickets related to testing@](https://preview.tinyurl.com/y9maauwg) From owner-freebsd-current@freebsd.org Thu May 21 01:04:23 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D51ED2F97B0 for ; Thu, 21 May 2020 01:04:23 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670041.outbound.protection.outlook.com [40.107.67.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49SBGY4bd4z43mj; Thu, 21 May 2020 01:04:18 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jy2uqJLh6ezSoOWd3riEmfwrVqS9yBp1w3T5LuUeCaAHFfczrLe4RFhQjFoIm1ydP7YObkJmHIkjpOvwZPxGNziYFWver0WmupH0lchNc4guabZwlF+exn47XdhSmivDlCRG+e/hGp0Q/PhLQlfSiFE/Bp9RdaYkW84ZLyEInVyFQ7l8E3lQ9DH4n+KzXuI/nVRiUGc0h7GSH6wFflwiKBPzRm6VIG5WJyCixvQEkOqbNvKXQkMsaHWxdfmPGZrW8r9OUJWn3mgOf8g8lWQ6tukHbMlb/qXylYM+vhi/t2jRdCJ4JmhxIv6MX1TsupPd5QV5jRDU9oudGmLvbIosJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bAANsJStp2sEDL01xJArR9RRZrF1IanPQX+bfFyfxqM=; b=JkBRDCTdVyiTPgmSYXDLNbqbB4tWT8kQWqQZB/B5mFDEV7ooTdcOxOYHwg4fS47Y2aH4TpNrbGW7Bg3pefW8KL4wkTRHZzvO9bcc3Jwfuha8X2Qi1GMV6S9DjS7JAjzpIFMH4S6rI7ONMxw9bJo3G5gmb5jvpZJHhH+wgJP0pyuYHU1Xj0IjRtSLSsd4QD8LhYiwwtGyxjIziiQZ4l6Fj7hY9sTfzSPhfDMoibe0QZBRcmTMBtq0VuTjcsflr/E44tBX2E/tEq8+zn57AljjgKdjn2VwSjVXs84OxDUfDnImBqirit1umubzqbjeOKjXlJ8IuHUwGXnhgvAP2/deDw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bAANsJStp2sEDL01xJArR9RRZrF1IanPQX+bfFyfxqM=; b=h6XVFO35lk/SnlgG5P/249By+DmSq3tNqMdNu1m0nN1OeKKi7vwQENuw/yilvjF261nIseziboi1X5NNIx748V7yX528Jll4gPhhOEQ3oChgGbdNBuC81JBscxBfEpOWa7LQrlX7XD8jwucetLvfwpr8mFUm9R1WjDW8xmVpxEI/TU3dcT6e8uzv9FVBZNpL+D7bvPUR1hhGbbz4xb4FZ1ZiF7i4aytvSGjmwebbfmt6Dyuw5f1u6fqX3GMxbQLO6JDCWTF8ge2ZUAqMticSG6tiTzxTMkvMkZrV9Z+XGa43G+2bSaogp1Bmgzle76H/GFrFrNE9jaCsI5XoG3bCYQ== Received: from QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:32::26) by QB1PR01MB3955.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:3f::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3000.20; Thu, 21 May 2020 01:04:17 +0000 Received: from QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM ([fe80::dd96:945c:b6ee:ffa2]) by QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM ([fe80::dd96:945c:b6ee:ffa2%6]) with mapi id 15.20.3000.034; Thu, 21 May 2020 01:04:17 +0000 From: Rick Macklem To: "freebsd-current@FreeBSD.org" CC: Konstantin Belousov , "rlibby@freebsd.org" Subject: r358252 causes intermittent hangs where processes are stuck sleeping on btalloc Thread-Topic: r358252 causes intermittent hangs where processes are stuck sleeping on btalloc Thread-Index: AQHWLwsKoH8lF1xuI0WPckyCNZ/JGQ== Date: Thu, 21 May 2020 01:04:16 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 7211535c-1627-4fb0-6ebd-08d7fd22e2b3 x-ms-traffictypediagnostic: QB1PR01MB3955: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 041032FF37 x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: xT+kom4WaQbn2lDg0IdQ5oxMc5rcdvgHUFcmaRZ3eks+bjcpl7mvb0aCa083ogFxmOfB1ndh7aUQchwEh8E/p49hpzfYB26N7+4qkU6JNa/BxH4+xooFgCJEUcFeDfAjvrOeOWKjhbSCpMvBTnvcAWrVN+gL63/66SpdSE/IIW5OEO8f58r3h8PFk88KTkxnD8aiC9QZAcmwFxy4ovYrktqW9654XEQvctS5pY85kXLbF6rDkrir/NgcvHBCafvX/My/CFLoGVMDh07Qk6GxwXDwNzcSLFXi5Jjlk0K3MnKnTUeGHDPLvjelhhK+Xj/+VgvWr0VlXnK9UJUfmtv4E6jic7FGoOn/vhdUbUfJrY5E2Gc5ogYfguZa/azEhqxM/8rFgzQCVrrijCwOUakOa6quHT8KLGKlbtZK+NrboAlqUb3cV9uyBAWSjiJPcZ8P x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(39860400002)(136003)(366004)(346002)(396003)(376002)(66476007)(52536014)(71200400001)(66946007)(6506007)(8936002)(55016002)(4326008)(316002)(2906002)(186003)(450100002)(7696005)(9686003)(786003)(5660300002)(66446008)(66556008)(64756008)(76116006)(478600001)(6916009)(33656002)(54906003)(8676002)(86362001)(4744005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: WtiPVqQrPbE0+Bt5IdkoatOm3yCfQ3w481TQkM+AUIY+bkgACXNz7d14X/yGmk1Pa++k0H13WNDwvpNpJ42nmnZtXFs9YijTHoxQ82HJXLws6HSBUMaXBtjRU3g3ZLafd+XiXQjlVWEFmLlhV0tBhOJ/y7/Y0Ql8QrsOnToB7LorV7PwjbYy4cZk41ECQMNtKN9aOrtoOGoMIH8YXbCd06v4UMleqfLd1Ol1Qhlik0BuxfVDjns0nJwNrUab032iW5S5GoKqjrfvcGNUMqy9shQnxQfN8aBkhXPSsdrxBeNrBRw03dZK1zIsptMJ4YxnIn2pgUArieghYIMVeoN8gskLIqF7DKD4Me7SQIKfDF67inCeWXs1qmnqGYK5BszJ3TMaGSBWCfHUksnaXRZNUshZUZxItGztuRJv0Df65GSn81QNdhSTuc2Ui47CMXd2AjGY5wSJ6uXszoHQZz19eHVw7VLa62C63eK8U+y37clGkojxLxVxv5XuYZky1ZMmBn6nJjPvW67Bn4+698xmRXoZz82505ceDXyzijp62JDi6xMbSjLgELzmOATgFuwy x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 7211535c-1627-4fb0-6ebd-08d7fd22e2b3 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 May 2020 01:04:16.8930 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 2vlv7/zvtjBVwz29a2L0QIgIqEFUUobDthpqK63dNBts0EPs7cRPMj8hH7qmivq4R9+37HgylL6DFa3kIyRBlg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: QB1PR01MB3955 X-Rspamd-Queue-Id: 49SBGY4bd4z43mj X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=h6XVFO35; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.41 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.14 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.980]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.41:from]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; DMARC_NA(0.00)[uoguelph.ca]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; NEURAL_HAM_SHORT(-1.53)[-1.530]; NEURAL_HAM_LONG(-1.04)[-1.035]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; RCVD_IN_DNSWL_LOW(-0.10)[40.107.67.41:from] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 May 2020 01:04:23 -0000 Hi,=0A= =0A= Since I hadn't upgraded a kernel through the winter, it took me a while=0A= to bisect this, but r358252 seems to be the culprit.=0A= =0A= If I do a kernel build over NFS using my not so big Pentium 4 (single core,= =0A= 1.25Gbytes RAM, i386), about every second attempt will hang.=0A= When I do a "ps" in the debugger, I see processes sleeping on btalloc.=0A= If I revert to r358251, I cannot reproduce this.=0A= =0A= Any ideas?=0A= =0A= I can easily test any change you might suggest to see if it fixes the=0A= problem.=0A= =0A= If you want more debug info, let me know, since I can easily=0A= reproduce it.=0A= =0A= Thanks, rick=0A= From owner-freebsd-current@freebsd.org Thu May 21 06:59:04 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BC58832B87D for ; Thu, 21 May 2020 06:59:04 +0000 (UTC) (envelope-from rlibby@gmail.com) Received: from mail-qt1-f178.google.com (mail-qt1-f178.google.com [209.85.160.178]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49SL7q739Pz4V8c; Thu, 21 May 2020 06:59:03 +0000 (UTC) (envelope-from rlibby@gmail.com) Received: by mail-qt1-f178.google.com with SMTP id p12so4682004qtn.13; Wed, 20 May 2020 23:59:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=5U4BHeNFicaawlgNtbUTsTB4ZHJ2JKZAhV6iPalujVk=; b=Kywmtwn3RmMRwFtaNbzT2sbXHFpWnJybQ5k+SEArOmt34Vvqz9GOUEPURKZ8z4vS0c bylcruVdYJCnx5l1QsdsSIjUYAzptY7Q4iFovvpAtjaS4tSGLwvqbJmBcwCSCHo2uv91 O5E8QoHj3IwvptWjZzMEL2Mg019QdRzBbYq/m0GWWtwj/uiKpajHqBvcSqL86VyT9cnN viYsZG7ALF74fTw0Njw4D6TyWDfxseesDLQc/2cboWQOVzZyB7oxgKGtisjlIgwD3jf8 dkfdg9y0+koWrSLEbEWuH/MYWnt+VxNhlxLfaKvQ3Hk4S/Gq4ArFVQKdUC6zmKcN1wSm sDkw== X-Gm-Message-State: AOAM53019VZylOTdWwNV8xJS8lQlaTXxoiAbPcb2CBTBWVlwtSSbbBld /pU5t7krqf9lUT/KcGQ2wA2TqTwm X-Google-Smtp-Source: ABdhPJz825IW8yntPy7b102j0rBupwcOfDM2Ga/WEVrsCiGqiKzyeT0KbZjYswE2lugrRgRHIOYWAw== X-Received: by 2002:ac8:1af3:: with SMTP id h48mr8780296qtk.371.1590044342386; Wed, 20 May 2020 23:59:02 -0700 (PDT) Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com. [209.85.160.174]) by smtp.gmail.com with ESMTPSA id f7sm4206253qtg.96.2020.05.20.23.59.01 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 20 May 2020 23:59:01 -0700 (PDT) Received: by mail-qt1-f174.google.com with SMTP id m64so4730027qtd.4; Wed, 20 May 2020 23:59:01 -0700 (PDT) X-Received: by 2002:ac8:3968:: with SMTP id t37mr9053586qtb.174.1590044341606; Wed, 20 May 2020 23:59:01 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ryan Libby Date: Wed, 20 May 2020 23:58:50 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: r358252 causes intermittent hangs where processes are stuck sleeping on btalloc To: Rick Macklem Cc: "freebsd-current@FreeBSD.org" , Konstantin Belousov Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 49SL7q739Pz4V8c X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rlibby@gmail.com designates 209.85.160.178 as permitted sender) smtp.mailfrom=rlibby@gmail.com X-Spamd-Result: default: False [-1.71 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; ARC_NA(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[209.85.160.178:from]; NEURAL_HAM_LONG(-0.30)[-0.302]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.47)[-0.472]; RCVD_IN_DNSWL_NONE(0.00)[209.85.160.178:from]; NEURAL_HAM_MEDIUM(-0.94)[-0.938]; FORGED_SENDER(0.30)[rlibby@freebsd.org,rlibby@gmail.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[rlibby@freebsd.org,rlibby@gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 May 2020 06:59:04 -0000 On Wed, May 20, 2020 at 6:04 PM Rick Macklem wrote: > > Hi, > > Since I hadn't upgraded a kernel through the winter, it took me a while > to bisect this, but r358252 seems to be the culprit. > > If I do a kernel build over NFS using my not so big Pentium 4 (single core, > 1.25Gbytes RAM, i386), about every second attempt will hang. > When I do a "ps" in the debugger, I see processes sleeping on btalloc. > If I revert to r358251, I cannot reproduce this. > > Any ideas? > > I can easily test any change you might suggest to see if it fixes the > problem. > > If you want more debug info, let me know, since I can easily > reproduce it. > > Thanks, rick Nothing obvious to me. I can maybe try a repro on a VM... ddb ps, acttrace, alltrace, show all vmem, show page would be welcome. "btalloc" is "We're either out of address space or lost a fill race." From owner-freebsd-current@freebsd.org Thu May 21 10:14:37 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AAF0032FE42 for ; Thu, 21 May 2020 10:14:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49SQTT27sQz4hfN; Thu, 21 May 2020 10:14:36 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id 04LAETP2027910 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 21 May 2020 13:14:32 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 04LAETP2027910 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id 04LAESks027909; Thu, 21 May 2020 13:14:28 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 21 May 2020 13:14:28 +0300 From: Konstantin Belousov To: Ryan Libby Cc: Rick Macklem , "freebsd-current@FreeBSD.org" , Konstantin Belousov Subject: Re: r358252 causes intermittent hangs where processes are stuck sleeping on btalloc Message-ID: <20200521101428.GC64045@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 49SQTT27sQz4hfN X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 May 2020 10:14:37 -0000 On Wed, May 20, 2020 at 11:58:50PM -0700, Ryan Libby wrote: > On Wed, May 20, 2020 at 6:04 PM Rick Macklem wrote: > > > > Hi, > > > > Since I hadn't upgraded a kernel through the winter, it took me a while > > to bisect this, but r358252 seems to be the culprit. > > > > If I do a kernel build over NFS using my not so big Pentium 4 (single core, > > 1.25Gbytes RAM, i386), about every second attempt will hang. > > When I do a "ps" in the debugger, I see processes sleeping on btalloc. > > If I revert to r358251, I cannot reproduce this. > > > > Any ideas? > > > > I can easily test any change you might suggest to see if it fixes the > > problem. > > > > If you want more debug info, let me know, since I can easily > > reproduce it. > > > > Thanks, rick > > Nothing obvious to me. I can maybe try a repro on a VM... > > ddb ps, acttrace, alltrace, show all vmem, show page would be welcome. > > "btalloc" is "We're either out of address space or lost a fill race." Yes, I would be not surprised to be out of something on 1G i386 machine. Please also add 'show alllocks'. From owner-freebsd-current@freebsd.org Thu May 21 21:02:00 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9A3382F50E2 for ; Thu, 21 May 2020 21:02:00 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670068.outbound.protection.outlook.com [40.107.67.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49ShrR3jNLz4LWj for ; Thu, 21 May 2020 21:01:59 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Vu+bwUkAhm+QZBAp5Zi1WVc41fH8gi2kuCk1dVGAypyFX9IdZ31RUOZrLjFlTrMaFsVg+xg0RQNM05dcHrROT8lLursd2rzrpsNtT3PEAX3UXUOgkKPj8OY5gxSrgvnoiD1lsY9YSYDD+AhtVO0krIlfuXhRMHVDPSscX6nL6XPYENRI8uJck9pUOYGMWyAJkfUiW81kBv/Rk1F1RsC3MAczLRV0iYk1yJItnyhqTy3m7R4nNC8XpjaK/0aB4FTtd7TGkHKnEZMjkGr22R4iabHMC2lWZHuhTWBhQTgAIJu82+p0nFVJUpCfo4Oc/Khms/lnZOpc4Yu1PZUf/QE4tw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nZxj1B306/WYaagKlHxlDfjvDOrB9nVxRteCBEEhjYI=; b=k23Jp1DFquwJMr9YAsiGT2sbpW8oQxGUdNCf4Ox06saIKElgnk5XbjcvmUbjiV+ZKf6q8aVXyl4KYPWlXJSM3S1IzKBPp3AESRFIQMwV8Xl7fG3AtmSLs3ypSCJmjsBtfv/qAyOY/XbwUBbI3OB3m4QiNaonCT114kFhhcTCm/N41UYzwkn0L9/gfPc5eIryUHK+5H6MjQlm1hHAw+WqrttMz/bcHgoOhzaYPnwfOLApCTVlalQpyvK4l4gipD46VorSel+abofpSuhL1N/ea/5zcZEsqM7N+CtHejbh9q+93iq6y3Gf5BR5MQNpa86WSCB7yllOa8eXi1ziapTIAQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nZxj1B306/WYaagKlHxlDfjvDOrB9nVxRteCBEEhjYI=; b=Dl6Pw6Z6vmqMch0ACl2kG5fchncHqV3sz+twR+XuewcZrP4RZdY/dKQxCabbnEtufSRwMrCZB39OH7IcFsmYiTRucdWaIWH84B7lDXyQniHSqMEwld+Cqn/LD0n40zJi/urd6Jlhpq4aPdJqKA1tRrcHOnTbmXkZ37sLhzc8G9TfGQJvyLEC7yB7YVpdyG8yZAkJzGHF3K32bNr305sGtlZQNEJN6+2oK4xhjuHyyHgPRxPlORxIPN+/hF/8gHyunQE3gEbLmy3YQWZiBjRbQwZSCp5nWAdScMd37pMq1xh7JlfenO0fZ05QfZofYXuIFzSmDI49nIZCAfNNVFw8jA== Received: from QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:32::26) by QB1PR01MB3026.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:3b::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.23; Thu, 21 May 2020 21:01:57 +0000 Received: from QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM ([fe80::dd96:945c:b6ee:ffa2]) by QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM ([fe80::dd96:945c:b6ee:ffa2%6]) with mapi id 15.20.3021.027; Thu, 21 May 2020 21:01:48 +0000 From: Rick Macklem To: "freebsd-current@FreeBSD.org" Subject: RFC: merging nfs-over-tls changes into head/sys Thread-Topic: RFC: merging nfs-over-tls changes into head/sys Thread-Index: AQHWL7DSVvlRzqq+nkOycfyzzN7vIA== Date: Thu, 21 May 2020 21:01:48 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: fdc7e953-7bff-46e4-f5da-08d7fdca2d4a x-ms-traffictypediagnostic: QB1PR01MB3026: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:7691; x-forefront-prvs: 041032FF37 x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 8AECIcIezLXlbWMfiw/jPde7k16+EVFpbRgnAF/DCAl1iGOr/r/2ukRJFGLmXejGwtPLc/TcSpoAoS/3uxX4ACkAhQSVpXgRBHEanf9xJDUl3QDbSs6bpoO4Ql9z5lax/hRiejYhEbiWs6l4twQjiFLImBX/WFEXCNoTNaaRpUKT5dqmFMUy+OUqiaBl+/NZzd5Nyg1KZjER55qo0Bp/+9Bh1oRrzIVOT8U9tImHLvVoUJvk2ltqD3R/yHuAFmqGJ2F90ZtLnDI6+agbj/zyJsxF/m6EJh2qBK+Tmo8/5T4lrRgnCTc7TZu+OKfvISQoTY6M9HGaBu8pvb7ykQ9rqg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(396003)(346002)(136003)(39860400002)(376002)(366004)(64756008)(66574014)(8676002)(8936002)(5660300002)(86362001)(71200400001)(9686003)(55016002)(66446008)(6916009)(66476007)(6506007)(478600001)(52536014)(66946007)(186003)(76116006)(2906002)(786003)(316002)(66556008)(91956017)(33656002)(7696005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: VLDmbfXFRF1p67YZ7p4oJm72WqvomGO5v6LUkP7Y5JWG1apbU2zya1tPVT15EmHxR/YqUB7VyBBeFxFaS9h6Ue8iA7iLur0lPFJf8jCgn7n3O7sLeeEjJKsAJgzLH0X39tgeqXFPMFxdtlrGLHktdTYZNZtFRI73IZ6gNfVSUiXZvMo5tV+2v4Hcg2fhxY2bXLpJQuQ+J7F5d2/cr7w89IJ4WxMQRs0PLmDqTL39IFQEO675xStB4/545g3JNsNuT2+pGd+c1shSmV9TFDocdeJJywsAJnFrugnSPHUCxjPTJ3RXEgcHALH/5KNzJEHOgrpK3ChTn5MlFNfcGwq5sSOf/X5NPcVjnwcdMWoRvTjZ+eLuigFARy24XzhYq+Ramg2H7bmOk5o2ifOiidhZdis1fvdOJ9xZekCE0itmXmakekaDWUGYRiCxu5qagFFlKjXZAiYx0NT6Gl1WW2/KsPEkGExR7ut5fwJj+ytQqpFADAluVeknPTgw9aQ3xBpLMyPMZGU0KuOSUku8A5hbIidSd/snkGRt/3HkYblrhDQ= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: fdc7e953-7bff-46e4-f5da-08d7fdca2d4a X-MS-Exchange-CrossTenant-originalarrivaltime: 21 May 2020 21:01:48.0563 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: IHFk9HwhGR2JcgOnHoxDXuqWxc57iFTNWxCbODiQ/HMAdaZbtfM8hf9wbyZ8AOZVW4ICQsK2PN8atG8J0/fP5w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: QB1PR01MB3026 X-Rspamd-Queue-Id: 49ShrR3jNLz4LWj X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=Dl6Pw6Z6; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.68 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-5.52 / 15.00]; NEURAL_HAM_MEDIUM(-1.05)[-1.049]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; RCVD_IN_DNSWL_LOW(-0.10)[40.107.67.68:from]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[uoguelph.ca]; NEURAL_HAM_LONG(-1.01)[-1.013]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; NEURAL_HAM_SHORT(-0.86)[-0.858]; TO_DN_EQ_ADDR_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.68:from] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 May 2020 21:02:00 -0000 Hi,=0A= =0A= I have now completed changes to the code in projects/nfs-over-tls, which=0A= implements TLS encryption of NFS RPC messages. (This roughly conforms=0A= to the internet draft "Towards Remote Procedure Call Encryption By Default"= ,=0A= which should soon become an RFC. For now, TLS1.2 is used instead of TLS1.3,= =0A= since FreeBSD's KERN_TLS does not yet implement TLS1.3.)=0A= =0A= I'd like to start merging some of the kernel changes into head/sys.=0A= =0A= The first of these would be creation of the syscall used by the daemons.=0A= (The code in projects/nfs-over-tls cheats and uses the syscall for the gssd= ,=0A= but it needs to have its own syscall so that the gssd daemon can run concu= rrently=0A= with it. I didn't want testers to need to build userland just to get a sys= call stub=0A= in libc.)=0A= =0A= After this, there are a bunch of changes to the NFS code to add support for= =0A= ext_pgs mbufs (these are significant patches, but should not affect the=0A= non-ext_pgs mbuf case, since they'll be conditional on ND_EXTPGS/M_EXTPGS).= =0A= =0A= Does this sound ok to do?=0A= =0A= Please let me know if you see problems with me doing this?=0A= =0A= Thanks, rick=0A= From owner-freebsd-current@freebsd.org Thu May 21 23:57:14 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8274F2F8CA7 for ; Thu, 21 May 2020 23:57:14 +0000 (UTC) (envelope-from db@db.net) Received: from tfm.com (mtbaker.tfm.com [192.231.224.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.tfm.com", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49Smkd2cdSz4Wct; Thu, 21 May 2020 23:57:12 +0000 (UTC) (envelope-from db@db.net) Received: from night.db.net (DB-DSL.ServerNorth.com [98.124.61.131]) (authenticated bits=0) by tfm.com (8.14.4/8.14.4) with ESMTP id 04LNv1ve004063 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 21 May 2020 16:57:04 -0700 (PDT) Date: Thu, 21 May 2020 19:56:59 -0400 From: Diane Bruce To: Andreas Nilsson Cc: cem@freebsd.org, "Andrey V. Elsukov" , Current FreeBSD Subject: Re: hwpstate_intel hangs kernel Message-ID: <20200521235659.GA14173@night.db.net> References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 49Smkd2cdSz4Wct X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of db@db.net has no SPF policy when checking 192.231.224.2) smtp.mailfrom=db@db.net X-Spamd-Result: default: False [0.65 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.32)[-0.316]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.03)[-0.028]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[db.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_SHORT(0.10)[0.097]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_NA(0.00)[no SPF record]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:10488, ipnet:192.231.224.0/22, country:US]; FREEMAIL_CC(0.00)[freebsd.org,yandex.ru] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 May 2020 23:57:14 -0000 On Wed, Feb 05, 2020 at 02:45:50PM +0100, Andreas Nilsson wrote: Ok I am going to respond to this old email from February.. > Hello, > > I upgraded to a newer version, git 87d669d3863-c266265, and I do not > experience the random hang anymore. The machine still hangs on boot on > "hwpstate_intel0: on cpu0" unless I set > 'hint.hwpstate_intel.0.disabled="1"' in loader.conf. > As a few others know on IRC I ran into exactly this same problem on a brand new Lenovo Carbon. I missed this thread somehow. I also had to bisect the commit. Would it be possible to put a note into UPDATING and default to disabled=1 for now? ;) ... > > Best regards > Andreas > > On Sat, Feb 1, 2020 at 11:26 PM Andreas Nilsson wrote: > > > Hello Conrad, > > > > thank you Andrey for bisecting! I'll try with that hint and see how it > > works for me. > > > > Best regards > > Andreas > > > > On Sat, Feb 1, 2020, 18:18 Conrad Meyer wrote: > > > >> Hi Andrey, > >> > >> Please try 'hint.hwpstate_intel.0.disabled="1"' as a workaround for now. > >> > >> I think I have identified at least one problematic piece of code, > >> although I don't know if it's the root cause. I will go ahead and fix > >> that, which may not fix the hang, and also add some debug printfs that > >> can be enabled to help identify the real issue. > >> > >> Thanks for the report and bisect. > >> > >> Best, > >> Conrad > >> > >> On Sat, Feb 1, 2020 at 6:06 AM Andrey V. Elsukov > >> wrote: > >> > > >> > 31.01.2020 18:11, Andrey V. Elsukov пишет: > >> > > On 24.01.2020 19:52, Andreas Nilsson wrote: > >> > >> It hangs during kernel boot and the last message printed on console > >> is: > >> > >> hwpstate_intel0: on cpu0 > >> > > > >> > > Hi, > >> > > > >> > > Did you find the cause of this hang? > >> > > I also tried to update today from r350816 to r357330. But my Lenovo X1 > >> > > Carbon 4th hangs on the same message. > >> > > > >> > > >> > Hi, > >> > > >> > I have bisected the bad commit, it is r357002. Yep. I also had to bisect this from what is now some 5 months ago :-( Diane -- - db@FreeBSD.org db@db.net http://www.db.net/~db From owner-freebsd-current@freebsd.org Fri May 22 21:58:36 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A006B2DE4B3 for ; Fri, 22 May 2020 21:58:36 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49TL3J3jvsz3yBd; Fri, 22 May 2020 21:58:36 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from John-Baldwins-MacBook-Pro-274.local (unknown [IPv6:2601:648:8203:2990:112f:d17b:daab:c6ca]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id 36A7A270D8; Fri, 22 May 2020 21:58:36 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Subject: Re: RFC: merging nfs-over-tls changes into head/sys To: Rick Macklem , "freebsd-current@FreeBSD.org" References: From: John Baldwin Autocrypt: addr=jhb@FreeBSD.org; keydata= mQGiBETQ+XcRBADMFybiq69u+fJRy/0wzqTNS8jFfWaBTs5/OfcV7wWezVmf9sgwn8TW0Dk0 c9MBl0pz+H01dA2ZSGZ5fXlmFIsee1WEzqeJzpiwd/pejPgSzXB9ijbLHZ2/E0jhGBcVy5Yo /Tw5+U/+laeYKu2xb0XPvM0zMNls1ah5OnP9a6Ql6wCgupaoMySb7DXm2LHD1Z9jTsHcAQMD /1jzh2BoHriy/Q2s4KzzjVp/mQO5DSm2z14BvbQRcXU48oAosHA1u3Wrov6LfPY+0U1tG47X 1BGfnQH+rNAaH0livoSBQ0IPI/8WfIW7ub4qV6HYwWKVqkDkqwcpmGNDbz3gfaDht6nsie5Z pcuCcul4M9CW7Md6zzyvktjnbz61BADGDCopfZC4of0Z3Ka0u8Wik6UJOuqShBt1WcFS8ya1 oB4rc4tXfSHyMF63aPUBMxHR5DXeH+EO2edoSwViDMqWk1jTnYza51rbGY+pebLQOVOxAY7k do5Ordl3wklBPMVEPWoZ61SdbcjhHVwaC5zfiskcxj5wwXd2E9qYlBqRg7QeSm9obiBCYWxk d2luIDxqaGJARnJlZUJTRC5vcmc+iGAEExECACAFAkTQ+awCGwMGCwkIBwMCBBUCCAMEFgID AQIeAQIXgAAKCRBy3lIGd+N/BI6RAJ9S97fvbME+3hxzE3JUyUZ6vTewDACdE1stFuSfqMvM jomvZdYxIYyTUpC5Ag0ERND5ghAIAPwsO0B7BL+bz8sLlLoQktGxXwXQfS5cInvL17Dsgnr3 1AKa94j9EnXQyPEj7u0d+LmEe6CGEGDh1OcGFTMVrof2ZzkSy4+FkZwMKJpTiqeaShMh+Goj XlwIMDxyADYvBIg3eN5YdFKaPQpfgSqhT+7El7w+wSZZD8pPQuLAnie5iz9C8iKy4/cMSOrH YUK/tO+Nhw8Jjlw94Ik0T80iEhI2t+XBVjwdfjbq3HrJ0ehqdBwukyeJRYKmbn298KOFQVHO EVbHA4rF/37jzaMadK43FgJ0SAhPPF5l4l89z5oPu0b/+5e2inA3b8J3iGZxywjM+Csq1tqz hltEc7Q+E08AAwUIAL+15XH8bPbjNJdVyg2CMl10JNW2wWg2Q6qdljeaRqeR6zFus7EZTwtX sNzs5bP8y51PSUDJbeiy2RNCNKWFMndM22TZnk3GNG45nQd4OwYK0RZVrikalmJY5Q6m7Z16 4yrZgIXFdKj2t8F+x613/SJW1lIr9/bDp4U9tw0V1g3l2dFtD3p3ZrQ3hpoDtoK70ioIAjjH aIXIAcm3FGZFXy503DOA0KaTWwvOVdYCFLm3zWuSOmrX/GsEc7ovasOWwjPn878qVjbUKWwx Q4QkF4OhUV9zPtf9tDSAZ3x7QSwoKbCoRCZ/xbyTUPyQ1VvNy/mYrBcYlzHodsaqUDjHuW+I SQQYEQIACQUCRND5ggIbDAAKCRBy3lIGd+N/BCO8AJ9j1dWVQWxw/YdTbEyrRKOY8YZNwwCf afMAg8QvmOWnHx3wl8WslCaXaE8= Message-ID: <6387cc78-c483-6271-7108-bf19a935dc01@FreeBSD.org> Date: Fri, 22 May 2020 14:58:34 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2020 21:58:36 -0000 On 5/21/20 2:01 PM, Rick Macklem wrote: > Hi, > > I have now completed changes to the code in projects/nfs-over-tls, which > implements TLS encryption of NFS RPC messages. (This roughly conforms > to the internet draft "Towards Remote Procedure Call Encryption By Default", > which should soon become an RFC. For now, TLS1.2 is used instead of TLS1.3, > since FreeBSD's KERN_TLS does not yet implement TLS1.3.) > > I'd like to start merging some of the kernel changes into head/sys. > > The first of these would be creation of the syscall used by the daemons. > (The code in projects/nfs-over-tls cheats and uses the syscall for the gssd, > but it needs to have its own syscall so that the gssd daemon can run concurrently > with it. I didn't want testers to need to build userland just to get a syscall stub > in libc.) > > After this, there are a bunch of changes to the NFS code to add support for > ext_pgs mbufs (these are significant patches, but should not affect the > non-ext_pgs mbuf case, since they'll be conditional on ND_EXTPGS/M_EXTPGS). > > Does this sound ok to do? > > Please let me know if you see problems with me doing this? I don't see any problems, per se, but I still need to do some changes on my end for software KTLS RX before it's ready to merge (I'm hoping to kill the iovecs in the kthreads entirely). -- John Baldwin From owner-freebsd-current@freebsd.org Fri May 22 22:17:02 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7FBEE2DF80F for ; Fri, 22 May 2020 22:17:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670063.outbound.protection.outlook.com [40.107.67.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49TLSZ0fGWz40Kk; Fri, 22 May 2020 22:17:01 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PQ22jH6ZqphD6iuhA5gJL8m/yVtEJ98YswKgMIWoR5HLS3+ZuYMbJM43+4JISuKxtuqzRU5h6e17F8E+3VFWijA1OGQK54FBoxQnGS6XDvE6+Ewbc/xrNZEBCq7esEDCaXMjDuzDn+1KFxVqt/0XXmjeC8jhChdOhHtDuiqncpdq8LELpWqkrqx8ia+PX9+REvB8NDNujAzz/yK+LTHAJjnYTLXbeOLJmCGBPLsoKDCuslcv04Gl67nAEmdKvFPn2O75AOKdmLs02IbTmWTtjTyM12p1mruvRWncV3tFNsgDFZE/F+6w5Pw9yHKBcFP9jRYCaffQvyENXeQKCRWVrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0EdHY+oNpjrYOYOONZ8Lzwsjzv6RYgUyB6k2NQkZRxk=; b=copVEdo/qe08IbLe3yBENarU4RxAhQmQgAlWruwgBGj7nH6yQcNNdy+MkNP+poTIjsZViQKN8Nr0jbywH3w53deYSsM5tJXlMQsUJ7huVPXLMI2C+OVxGLS8NVvKl6fyC4GSLyoGt5wf9TrhlicttKO9GF7sG2dqViPtExUULdvaLJsOpCl+hxH9YUNCBAlVfYtZpbKZuCy/3m/T3PrxLmKB9Y0BQnzwzhzkaXUTctI1OIVNJAOp/cbTLDrl0nkOg0+gBCruPeGKrE/3kTaNlSS1EHdzCgXUkiPUsiCdB7Qlr5eWnWUwyjaXswFdo51COKpY4TthfGrWNXkJBz4GTQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0EdHY+oNpjrYOYOONZ8Lzwsjzv6RYgUyB6k2NQkZRxk=; b=Z5o97lPoGBgZek6m9XZAWe4wNHwGyJTAzFqvbFfGwsMyVTb3QErMAIdgOcwf2lem6oUS5gzDr393NO1gc/eJJsU97WSb54+wgR6qBzkWQ1kMSiF98fzcx+dA6xjsva03W7gnSjl3UEsp7Td7EK+pu26hEZ8xua5D8tyfWGB/7+Yqr01j1uC72V3fsGdImXfsRkXrJ/bHkOeY1LTU2rDuDew+RgfN5LDSnaDvQmSIZfnnzSiY0UVr8DUMNdpj34rl1EbmmAghb8ETTmkK6Hh4KhUWgdwsKi0Gy1Kxp5HfaFNDq9qWTCJeu5rY44u7XQk7tCCmYBP2iXldelI2cottQg== Received: from QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:32::26) by QB1PR01MB2721.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:31::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.24; Fri, 22 May 2020 22:17:00 +0000 Received: from QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM ([fe80::dd96:945c:b6ee:ffa2]) by QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM ([fe80::dd96:945c:b6ee:ffa2%6]) with mapi id 15.20.3021.027; Fri, 22 May 2020 22:16:51 +0000 From: Rick Macklem To: John Baldwin , "freebsd-current@FreeBSD.org" Subject: Re: RFC: merging nfs-over-tls changes into head/sys Thread-Topic: RFC: merging nfs-over-tls changes into head/sys Thread-Index: AQHWL7DSVvlRzqq+nkOycfyzzN7vIKi0qOQAgAABtaU= Date: Fri, 22 May 2020 22:16:51 +0000 Message-ID: References: , <6387cc78-c483-6271-7108-bf19a935dc01@FreeBSD.org> In-Reply-To: <6387cc78-c483-6271-7108-bf19a935dc01@FreeBSD.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 636e1046-ccdb-42c2-f991-08d7fe9dd3ef x-ms-traffictypediagnostic: QB1PR01MB2721: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-forefront-prvs: 04111BAC64 x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: DuGzY24jfLBITOWHfBGvTRsYm6HCYfhc4IlnJMkI+cYg8mdFJadHvRHHw9Gc2S1WbCfLJ8rEBDLPkzlXScULJPAwfhcKQ4+Fh/d+ocz3p9SBESESrqA/bhDlhLz07LFXld7yVgUlsL2O7B8UJMfJi8kLs/z+jHY5VapUQkWz9CZ7OZOha2reXNar8DBrX3u/YyvXWQtJq8MWvjTHMpGD+9MHzi4tgbQvDhedw1xZYUld1WuDq6m4uUcnC6Npm78e/E39RAOTo1fwWVbcm8ZrjRZ+PUn22mC+q/amNzX+phbq9cuTvfoZOGVkf6fZdrup0HgE+oHdQ4hyaRUU2eXxEw== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(376002)(346002)(396003)(136003)(39860400002)(366004)(7696005)(66574014)(33656002)(8676002)(8936002)(6506007)(86362001)(186003)(316002)(786003)(5660300002)(110136005)(71200400001)(66476007)(478600001)(2906002)(450100002)(55016002)(66556008)(64756008)(66446008)(52536014)(9686003)(66946007)(76116006); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: sE3dYr4xPaXyzHh0rTeBo0G7cc7qdU8LgAMmLFrNYKIIXaupGZzmc03suKuHMF65Y42SyXsYqLYoL2ksTIxYm06ChhmqRM2TNsxilLVrRkUmSI2SC2t16rBsIIaC0pkcJNq4FTPnRQh1NNICdf4VENqXDj/NiszNrFy5SJK+mDVG6sUt4gFcGiqRF+3NAQrn5FHmShf4rIin+r+9qKAJEkFc9181izBroHAOd4RrI9pctbUWLoahhCOl6hrxeY9euqYyOgV7WfQPlpPcX3nDqapvkPjVUZVqgk9ZukPFXzmlHzDrtmPuZ/RQAYuUYD8HQaycywyx1RUw7ISDJw6aYGMBfJYauPH+drcZChFf2gi4Wk+i4hA6/ztsd5SdbUYV+97/ltbOGSF45QbWy6RKoSMdTBwJ1GgMES4Gt1l8y3cXPKPb4LWOkn9hj5yURlD8aap+7unLeX9KVxZ7frjrRf0NycVSLJCbVbj3oSGGXNF+sL9EPbce3Ezkw7/5gI3gplrrCnwjoWomITK27hvgihreNoRqO1HpDRrYey/7V/k= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 636e1046-ccdb-42c2-f991-08d7fe9dd3ef X-MS-Exchange-CrossTenant-originalarrivaltime: 22 May 2020 22:16:51.4165 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ZFnVa2tuIrv4PS3jgMZn3GT5Egn51JCVr+qXlOOIPrpn/1e9c2wyddVSaO75UwGrmgCcmfNZLadZYI5LzvAOmg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: QB1PR01MB2721 X-Rspamd-Queue-Id: 49TLSZ0fGWz40Kk X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2020 22:17:02 -0000 John Baldwin wrote:=0A= >On 5/21/20 2:01 PM, Rick Macklem wrote:=0A= >> Hi,=0A= >>=0A= >> I have now completed changes to the code in projects/nfs-over-tls, which= =0A= >> implements TLS encryption of NFS RPC messages. (This roughly conforms=0A= >> to the internet draft "Towards Remote Procedure Call Encryption By Defau= lt",=0A= >> which should soon become an RFC. For now, TLS1.2 is used instead of TLS1= .3,=0A= >> since FreeBSD's KERN_TLS does not yet implement TLS1.3.)=0A= >>=0A= >> I'd like to start merging some of the kernel changes into head/sys.=0A= >>=0A= >> The first of these would be creation of the syscall used by the daemons.= =0A= >> (The code in projects/nfs-over-tls cheats and uses the syscall for the g= ssd,=0A= >> but it needs to have its own syscall so that the gssd daemon can run co= ncurrently=0A= >> with it. I didn't want testers to need to build userland just to get a = syscall stub=0A= >> in libc.)=0A= >>=0A= >> After this, there are a bunch of changes to the NFS code to add support = for=0A= >> ext_pgs mbufs (these are significant patches, but should not affect the= =0A= >> non-ext_pgs mbuf case, since they'll be conditional on ND_EXTPGS/M_EXTPG= S).=0A= >>=0A= >> Does this sound ok to do?=0A= >>=0A= >> Please let me know if you see problems with me doing this?=0A= >=0A= >I don't see any problems, per se, but I still need to do some changes on m= y=0A= >end for software KTLS RX before it's ready to merge (I'm hoping to kill=0A= >the iovecs in the kthreads entirely).=0A= Sure. My plan is to merge bits and pieces, because some of it involves part= s=0A= of the system like mount exports or changes to soreceive_generic(),=0A= that will require reviews.=0A= =0A= To be honest, most of the changes are not specifically nfs-over-tls (or=0A= krpc-over-tls, although NFS is currently the only consumer).=0A= They are things like generating ext_pgs mbuf lists (which can be used for= =0A= non-TLS connections, although I'm not sure they are useful for other cases?= )=0A= or a better way of handling the krpc client side receive.=0A= =0A= I think it will be quite a while before all the kernel bits are in head, bu= t having=0A= the syscall in head (mainly the syscall stub in libc) will make it easier f= or=0A= testers to set systems up. They may not be FreeBSD types.=0A= =0A= No rush on the TLS changes from my perspective. (It would be nice to get=0A= the kernel bits in FreeBSD13. The userland stuff could probably become a=0A= package/port, I think?=0A= =0A= Thanks yet again, for your help with this, rick=0A= =0A= =0A= --=0A= John Baldwin=0A= From owner-freebsd-current@freebsd.org Fri May 22 23:46:30 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7E6822F10D9 for ; Fri, 22 May 2020 23:46:30 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660086.outbound.protection.outlook.com [40.107.66.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49TNRm740Lz44DM; Fri, 22 May 2020 23:46:28 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nIy6wrsfLTrMuQtHT9thI/U2LIRvMbHEIEkQja996ahNCa3Qd1ipXm6imClsbsVz9ImNt3gXwsfrHAE9UNgnRwGVe6Sroc2ii6J7Kox18EP6mu4bvMYEIw406tdbp125+G8ouVHG6yDbiM4Oiva1u2w1cyLiyQkaf+/uU7PTYFQWTO49kOfhKTAlqSf/2t97TI9GTkWVyRk490hgbRVNK2JTEKoyWrixLAhO8Jd6vG9fDiQGqNgkQtUAXm85RwIp7J00uT/dxdBXLo2dGw1FQ5HA9B0a+bYjEUqEIDi6oLrL7LzaTIvsSwDapQTvHE7b1isGYPRdNkoPVTY7T75YIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Z+OKVYPA6GaJpCTmnIxItoA3P/I0iVSarNhzD0Fjwg4=; b=TWm800ZHX12hi0DHY84ZGj2QRDl/KTpmCiO3yV2D95UCwvd3n8U7ot+4OA7RFuoNMSHAQ0l9jiJh8Q7X8USDfejQv+Q+Dt6E0qcy/TzLp/WDhBQBJGUllnUuoQHTKmt/5JxCS24HdxTCpgX4vNmQqvP9FcCIiDjUQjdbkfeadBO959LrLSxe6mGrGMug1Sf2m1jRZBfciFxQMyvtLg1R0wSZTFORAu4ZXInp5ikxQf5LXgTH83oeX3LaNV5YJ6VPSBaMdFUcivB0kRoVwOcQiSnGQ7G6FBcJM/a/be0rczxMCAt8Gwg/c9PF3sAmj4QnUCBoRD0mdj501eRpPgvR7A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Z+OKVYPA6GaJpCTmnIxItoA3P/I0iVSarNhzD0Fjwg4=; b=bPWDL5d1mz3jO/eDi067wczj7DADmSDpsyr+DJVKnI9C9hIzgALbDz9qE2AtyQAHTvxJXJTGApl0aYuJM1kIj4ST+JDTyhQGnn46S/Yd9XjpkBlX3QbG4r9GIeIYlaWxMx/0EFU+PEhJ96cY1kFycqbvDiFnYrrRr+gZoCA407SvQ7mFucil6v+6eeR/fCe4Wb/3DQD0bEk0PWEO/rEfCnEIl/NJgcxJaVcXdi31U62IFq4V17Y65lURRH/HA99cqO18noPN3aN5hcqUptvOKW7HIHY8Jzq9dhCtEGUjtZlLME7LOmPF0jcpFaWhJYA4E/88UzSWwSqwnh2zbuN9dw== Received: from QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:32::26) by QB1PR01MB3761.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:32::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.23; Fri, 22 May 2020 23:46:26 +0000 Received: from QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM ([fe80::dd96:945c:b6ee:ffa2]) by QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM ([fe80::dd96:945c:b6ee:ffa2%6]) with mapi id 15.20.3021.027; Fri, 22 May 2020 23:46:26 +0000 From: Rick Macklem To: Konstantin Belousov , Ryan Libby CC: "freebsd-current@FreeBSD.org" , Konstantin Belousov Subject: Re: r358252 causes intermittent hangs where processes are stuck sleeping on btalloc Thread-Topic: r358252 causes intermittent hangs where processes are stuck sleeping on btalloc Thread-Index: AQHWLwsKoH8lF1xuI0WPckyCNZ/JGaiyHHkAgAA2qQCAAlx9bA== Date: Fri, 22 May 2020 23:46:26 +0000 Message-ID: References: , <20200521101428.GC64045@kib.kiev.ua> In-Reply-To: <20200521101428.GC64045@kib.kiev.ua> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: dd4bd9f5-9131-4432-c815-08d7feaa57c9 x-ms-traffictypediagnostic: QB1PR01MB3761: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 04111BAC64 x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 9uhzkR95RMIl7kHZQYTb16TnNT5b2qD7vzmAHYfrIuCi/yi1LjwnT83DfFs7IGBXOyKXBSHOt+No3tFgHbERqYrsLniVl+rvuwSdmpbH5PPDHhHEDCh7VB3muDmjAXczPahLfHCHCi1Fp/7W37Uf4nPSzVWMiBxrbMI933z2PhvIeBXl5JeOZ/tOlj7NEsEwjVVdDF5EEW1T4vrKXBPon5axuen73RkZBrcSJ98i7iC13k/9WN3P9yspEPBjp6ZxWGI7jhCdumCjtaGtK/A5LYnMTuS7YKLut1oZYf78dFFw6kQ3NAHpJKtacl5twDyg x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:QB1PR01MB3649.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(396003)(376002)(136003)(39860400002)(346002)(366004)(66476007)(5660300002)(478600001)(2906002)(33656002)(86362001)(71200400001)(66946007)(76116006)(8676002)(6506007)(186003)(53546011)(8936002)(7696005)(4326008)(52536014)(786003)(66556008)(9686003)(66446008)(55016002)(316002)(54906003)(110136005)(64756008); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: RoRzeKUSTzoyYkjAdB1+ND0C7UfsYN7ibrUvRCDe9FzoHS1YdKNhc1UgIiw68nhgZZwMHx9x3zHAMtSck0z3Av2wIn1MzNcbSEUeTTWZxYBNWlT5nZiiv0xgwU42ylZl6p1Eixcn6Dnw10aAco6aFF8dj0EEcLNE9H1wdA4xEHrnTnsbyKX8LIJrvRrjBeGw1YoYOZW0SXBhgfspRI/PunHu15fRvf+WW1Vwt6pijJ7G9BnGwa3rn/otE1gz3npmdAyHItzeRn1cZu1F3cDjUGrsEWVl5SU8lttg2SHEXY38mFXk2W0jLLtY6qnByW5sCnfJUxrSSqVDGM02b92n/rQMLNs4kFs1ohW3SJ8z+WyeRVBZEozpnTmLp+FfuIr1PUiWYe3KEcpOc39IgKsUlXMWN9NMEWvyCxv/5UTPNZX+tJDhKttwfpMXVRAdobiaYCyX0+gPMWcVOJj1d9cXkbnd4wL/KU0GR+En1NU8AEoQbSkP3P1U9UGBK70BlRHTOO6dvX8FWaoC+BJYxEKIzK7QPlOEL0dpvt38lVtCqL4= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: dd4bd9f5-9131-4432-c815-08d7feaa57c9 X-MS-Exchange-CrossTenant-originalarrivaltime: 22 May 2020 23:46:26.5462 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 7K3XcsELMOnE4iLATLr2y/nX8nZhMfVGK+MqcusV9cVkjUeQu+7j5nvMMLWqwU6q8zwpdR7wdz3ObeNWjdUn1A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: QB1PR01MB3761 X-Rspamd-Queue-Id: 49TNRm740Lz44DM X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=bPWDL5d1; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.86 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-5.18 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.994]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[uoguelph.ca]; NEURAL_HAM_LONG(-0.99)[-0.987]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; NEURAL_HAM_SHORT(-0.70)[-0.702]; RCVD_IN_DNSWL_NONE(0.00)[40.107.66.86:from]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.66.86:from] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2020 23:46:30 -0000 Konstantin Belousov wrote:=0A= >On Wed, May 20, 2020 at 11:58:50PM -0700, Ryan Libby wrote:=0A= >> On Wed, May 20, 2020 at 6:04 PM Rick Macklem wrot= e:=0A= >> >=0A= >> > Hi,=0A= >> >=0A= >> > Since I hadn't upgraded a kernel through the winter, it took me a whil= e=0A= >> > to bisect this, but r358252 seems to be the culprit.=0A= >> >=0A= >> > If I do a kernel build over NFS using my not so big Pentium 4 (single = core,=0A= >> > 1.25Gbytes RAM, i386), about every second attempt will hang.=0A= >> > When I do a "ps" in the debugger, I see processes sleeping on btalloc.= =0A= >> > If I revert to r358251, I cannot reproduce this.=0A= >> >=0A= >> > Any ideas?=0A= >> >=0A= >> > I can easily test any change you might suggest to see if it fixes the= =0A= >> > problem.=0A= >> >=0A= >> > If you want more debug info, let me know, since I can easily=0A= >> > reproduce it.=0A= >> >=0A= >> > Thanks, rick=0A= >>=0A= >> Nothing obvious to me. I can maybe try a repro on a VM...=0A= >>=0A= >> ddb ps, acttrace, alltrace, show all vmem, show page would be welcome.= =0A= >>=0A= >> "btalloc" is "We're either out of address space or lost a fill race."=0A= >=0A= >Yes, I would be not surprised to be out of something on 1G i386 machine.= =0A= >Please also add 'show alllocks'.=0A= Ok, I used an up to date head kernel and it took longer to reproduce a hang= .=0A= This time, none of the processes are stuck on "btalloc".=0A= I'll try and give you most of the above, but since I have to type it in by = hand=0A= from the screen, I might not get it all. (I'm no real typist;-)=0A= > show alllocks=0A= exclusive lockmgr ufs (ufs) r =3D 0 locked @ kern/vfs_subr.c: 3259=0A= exclusive lockmgr nfs (nfs) r =3D 0 locked @ kern/vfs_lookup.c:737=0A= exclusive sleep mutex kernel area domain (kernel arena domain) r =3D 0 lock= ed @ kern/subr_vmem.c:1343=0A= exclusive lockmgr bufwait (bufwait) r =3D 0 locked @ kern/vfs_bio.c:1663=0A= exclusive lockmgr ufs (ufs) r =3D 0 locked @ kern/vfs_subr.c:2930=0A= exclusive lockmgr syncer (syncer) r =3D 0 locked @ kern/vfs_subr.c:2474=0A= Process 12 (intr) thread 0x.. (1000008)=0A= exclusive sleep mutex Giant (Giant) r =3D 0 locked @ kern/kern_intr.c:1152= =0A= =0A= > ps=0A= - Not going to list them all, but here are the ones that seem interesting..= .=0A= 18 0 0 0 DL vlruwt 0x11d939cc [vnlru]=0A= 16 0 0 0 DL (threaded) [bufdaemon]=0A= 100069 D qsleep [bufdaemon]=0A= 100074 D - [bufspacedaemon-0]=0A= 100084 D sdflush 0x11923284 [/ worker]=0A= - and more of these for the other UFS file systems=0A= 9 0 0 0 DL psleep 0x1e2f830 [vmdaemon]=0A= 8 0 0 0 DL (threaded) [pagedaemon]=0A= 100067 D psleep 0x1e2e95c [dom0]=0A= 100072 D launds 0x1e2e968 [laundry: dom0]=0A= 100073 D umarcl 0x12cc720 [uma]=0A= =85 a bunch of usb and cam ones=0A= 100025 D - 0x1b2ee40 [doneq0]=0A= =85=0A= 12 0 0 0 RL (threaded) [intr]=0A= 100007 I [swi6: task queue]=0A= 100008 Run CPU 0 [swi6: Giant taskq]=0A= =85=0A= 100000 D swapin 0x1d96dfc [swapper]=0A= - and a bunch more in D state.=0A= Does this mean the swapper was trying to swap in?=0A= =0A= > acttrace=0A= - just shows the keyboard=0A= kdb_enter() at kdb_enter+0x35/frame=0A= vt_kbdevent() at vt_kdbevent+0x329/frame=0A= kdbmux_intr() at kbdmux_intr+0x19/frame=0A= taskqueue_run_locked() at taskqueue_run_locked+0x175/frame=0A= taskqueue_run() at taskqueue_run+0x44/frame=0A= taskqueue_swi_giant_run(0) at taskqueue_swi_giant_run+0xe/frame=0A= ithread_loop() at ithread_loop+0x237/frame=0A= fork_exit() at fork_exit+0x6c/frame=0A= fork_trampoline() at 0x../frame=0A= =0A= > show all vmem=0A= vmem 0x.. 'transient arena'=0A= quantum: 4096=0A= size: 23592960=0A= inuse: 0=0A= free: 23592960=0A= busy tags: 0=0A= free tags: 2=0A= inuse size free size=0A= 16777216 0 0 1 23592960=0A= vmem 0x.. 'buffer arena'=0A= quantum: 4096=0A= size: 94683136=0A= inuse: 94502912=0A= free: 180224=0A= busy tags: 1463=0A= free tags: 3=0A= inuse size free size=0A= 16384 2 32768 1 16384=0A= 32768 39 1277952 1 32768=0A= 65536 1422 93192192 0 0=0A= 131072 0 0 1 131072=0A= vmem 0x.. 'i386trampoline'=0A= quantum: 1=0A= size: 24576=0A= inuse: 20860=0A= free: 3716=0A= busy tags: 9=0A= free tags: 3=0A= inuse size free size=0A= 32 1 48 1 52=0A= 64 2 208 0 0=0A= 128 2 280 0 0=0A= 2048 1 2048 1 3664=0A= 4096 2 8192 0 0=0A= 8192 1 10084 0 0=0A= vmem 0x.. 'kernel rwx arena'=0A= quantum: 4096=0A= size: 0=0A= inuse: 0=0A= free: 0=0A= busy tags: 0=0A= free tags: 0=0A= vmem 0x.. 'kernel area dom'=0A= quantum: 4096=0A= size: 56623104=0A= inuse: 56582144=0A= free: 40960=0A= busy tags: 11224=0A= free tags: 3=0A= inuse size free size=0A= 4096 11091 45428736 0 0=0A= 8192 63 516096 0 0=0A= 16384 12 196608 0 0=0A= 32768 6 196608 0 0=0A= 40960 2 81920 1 40960=0A= 65536 16 1048576 0 0=0A= 94208 1 94208 0 0=0A= 110592 1 110592 0 0=0A= 131072 15 2441216 0 0=0A= 262144 15 3997696 0 0=0A= 524288 1 524288 0 0=0A= 1048576 1 1945600 0 0=0A= vmem 0x.. 'kernel arena'=0A= quantum: 4096=0A= size: 390070272=0A= inuse: 386613248=0A= free: 3457024=0A= busy tags: 873=0A= free tags: 3=0A= inuse size free size=0A= 4096 35 143360 1 4096=0A= 8192 2 16384 2 16384=0A= 12288 1 12288 0 0=0A= 16384 30 491520 0 0=0A= 20480 140 2867200 0 0=0A= 65536 1 65536 0 0=0A= 131072 631 82706432 0 0=0A= 1048576 0 0 1 1339392=0A= 2097152 27 56623104 1 2097152=0A= 8388608 1 13774848 0 0=0A= 16777216 3 74883072 0 0=0A= 33554432 1 36753408 0 0=0A= 67108864 1 118276096 0 0=0A= =0A= > alltrace=0A= - I can't face typing too much more, but I'll put a few=0A= here that look interesting=0A= =0A= - for csh=0A= sched_switch()=0A= mi_switch()=0A= kern_yield()=0A= getblkx()=0A= breadn_flags()=0A= ffs_update()=0A= ufs_inactive()=0A= VOP_INACTIVE()=0A= vinactivef()=0A= vput_final()=0A= vm_object_deallocate()=0A= vm_map_process_deferred()=0A= kern_munmap()=0A= sys_munmap()=0A= =0A= - For cc=0A= sched_switch()=0A= mi_switch()=0A= sleepq_switch()=0A= sleepq_timedwait()=0A= _sleep()=0A= pause_sbt()=0A= vmem_bt_alloc()=0A= keg_alloc_slab()=0A= zone_import()=0A= cache_alloc()=0A= cache_alloc_retry()=0A= uma_zalloc_arg()=0A= bt_fill()=0A= vmem_xalloc()=0A= vmem_alloc()=0A= kmem_alloc()=0A= kmem_malloc_domainset()=0A= page_alloc()=0A= keg_alloc_slab()=0A= zone_import()=0A= cache_alloc()=0A= cache_alloc_retry()=0A= uma_zalloc_arg()=0A= nfscl_nget()=0A= nfs_create()=0A= vop_sigdefer()=0A= nfs_vnodeops_bypass()=0A= VOP_CREATE_APV()=0A= vn_open_cred()=0A= vn_open()=0A= kern_openat()=0A= sys_openat()=0A= =0A= Then there are a bunch of these... for cc, make=0A= sched_switch()=0A= mi_switch()=0A= sleepq_switch()=0A= sleepq_catch_signals()=0A= sleepq_wait_sig()=0A= kern_wait6()=0A= sys_wait4()=0A= =0A= - for vnlru=0A= sched_switch()=0A= mi_switch()=0A= sleepq_switch()=0A= sleepq_timedwait()=0A= _sleep()=0A= vnlru_proc()=0A= fork_exit()=0A= fork_trampoline()=0A= =0A= - for syncer=0A= sched_switch()=0A= mi_switch()=0A= critical_exit_preempt()=0A= intr_event_handle()=0A= intr_execute_handlers()=0A= lapic_handle_intr()=0A= Xapic_isr1()=0A= - interrupt=0A= memset()=0A= cache_alloc()=0A= cache_alloc_retry()=0A= uma_zalloc_arg()=0A= vmem_xalloc()=0A= vmem_bt_alloc()=0A= keg_alloc_slab()=0A= zone_import()=0A= cache_alloc()=0A= cache_alloc_retry()=0A= uma_zalloc_arg()=0A= bt_fill()=0A= vmem_xalloc()=0A= vmem_alloc()=0A= bufkva_alloc()=0A= getnewbuf()=0A= getblkx()=0A= breadn_flags()=0A= ffs_update()=0A= ffs_sync()=0A= sync_fsync()=0A= VOP_FSYNC_APV()=0A= sched_sync()=0A= fork_exit()=0A= fork_trampoline()=0A= =0A= - For bufdaemon (a bunch of these)=0A= sched_switch()=0A= mi_switch()=0A= sleepq_switch()=0A= sleepq_timedwait()=0A= _sleep()=0A= buf_daemon()=0A= fork_exit()=0A= fork_trampoline()=0A= =0A= vmdaemon and pagedaemon are basically just like above,=0A= sleeping in=0A= vm_daemon()=0A= or=0A= vm_pageout_worker()=0A= or=0A= vm_pageout_laundry_worker()=0A= or=0A= uma_reclaim_worker()=0A= =0A= That's all the typing I can take right now.=0A= I can probably make this happen again if you want more specific stuff.=0A= =0A= rick=0A= =0A= =0A= =0A= =0A= From owner-freebsd-current@freebsd.org Sat May 23 10:56:15 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7DC232FE679 for ; Sat, 23 May 2020 10:56:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49TgJZ2p3Gz3VVt; Sat, 23 May 2020 10:56:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id 04NAu29M054220 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 23 May 2020 13:56:05 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 04NAu29M054220 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id 04NAu11J054219; Sat, 23 May 2020 13:56:01 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 23 May 2020 13:56:01 +0300 From: Konstantin Belousov To: Rick Macklem Cc: Ryan Libby , "freebsd-current@FreeBSD.org" Subject: Re: r358252 causes intermittent hangs where processes are stuck sleeping on btalloc Message-ID: <20200523105601.GN64045@kib.kiev.ua> References: <20200521101428.GC64045@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 49TgJZ2p3Gz3VVt X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-0.82 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-0.32)[-0.325]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_MEDIUM(-0.27)[-0.270]; R_SPF_SOFTFAIL(0.00)[~all:c]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.23)[-0.228]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 May 2020 10:56:15 -0000 On Fri, May 22, 2020 at 11:46:26PM +0000, Rick Macklem wrote: > Konstantin Belousov wrote: > >On Wed, May 20, 2020 at 11:58:50PM -0700, Ryan Libby wrote: > >> On Wed, May 20, 2020 at 6:04 PM Rick Macklem wrote: > >> > > >> > Hi, > >> > > >> > Since I hadn't upgraded a kernel through the winter, it took me a while > >> > to bisect this, but r358252 seems to be the culprit. > >> > > >> > If I do a kernel build over NFS using my not so big Pentium 4 (single core, > >> > 1.25Gbytes RAM, i386), about every second attempt will hang. > >> > When I do a "ps" in the debugger, I see processes sleeping on btalloc. > >> > If I revert to r358251, I cannot reproduce this. > >> > > >> > Any ideas? > >> > > >> > I can easily test any change you might suggest to see if it fixes the > >> > problem. > >> > > >> > If you want more debug info, let me know, since I can easily > >> > reproduce it. > >> > > >> > Thanks, rick > >> > >> Nothing obvious to me. I can maybe try a repro on a VM... > >> > >> ddb ps, acttrace, alltrace, show all vmem, show page would be welcome. > >> > >> "btalloc" is "We're either out of address space or lost a fill race." > > > >Yes, I would be not surprised to be out of something on 1G i386 machine. > >Please also add 'show alllocks'. > Ok, I used an up to date head kernel and it took longer to reproduce a hang. > This time, none of the processes are stuck on "btalloc". > I'll try and give you most of the above, but since I have to type it in by hand > from the screen, I might not get it all. (I'm no real typist;-) > > show alllocks > exclusive lockmgr ufs (ufs) r = 0 locked @ kern/vfs_subr.c: 3259 > exclusive lockmgr nfs (nfs) r = 0 locked @ kern/vfs_lookup.c:737 > exclusive sleep mutex kernel area domain (kernel arena domain) r = 0 locked @ kern/subr_vmem.c:1343 > exclusive lockmgr bufwait (bufwait) r = 0 locked @ kern/vfs_bio.c:1663 > exclusive lockmgr ufs (ufs) r = 0 locked @ kern/vfs_subr.c:2930 > exclusive lockmgr syncer (syncer) r = 0 locked @ kern/vfs_subr.c:2474 > Process 12 (intr) thread 0x.. (1000008) > exclusive sleep mutex Giant (Giant) r = 0 locked @ kern/kern_intr.c:1152 > > > ps > - Not going to list them all, but here are the ones that seem interesting... > 18 0 0 0 DL vlruwt 0x11d939cc [vnlru] > 16 0 0 0 DL (threaded) [bufdaemon] > 100069 D qsleep [bufdaemon] > 100074 D - [bufspacedaemon-0] > 100084 D sdflush 0x11923284 [/ worker] > - and more of these for the other UFS file systems > 9 0 0 0 DL psleep 0x1e2f830 [vmdaemon] > 8 0 0 0 DL (threaded) [pagedaemon] > 100067 D psleep 0x1e2e95c [dom0] > 100072 D launds 0x1e2e968 [laundry: dom0] > 100073 D umarcl 0x12cc720 [uma] > … a bunch of usb and cam ones > 100025 D - 0x1b2ee40 [doneq0] > … > 12 0 0 0 RL (threaded) [intr] > 100007 I [swi6: task queue] > 100008 Run CPU 0 [swi6: Giant taskq] > … > 100000 D swapin 0x1d96dfc [swapper] > - and a bunch more in D state. > Does this mean the swapper was trying to swap in? > > > acttrace > - just shows the keyboard > kdb_enter() at kdb_enter+0x35/frame > vt_kbdevent() at vt_kdbevent+0x329/frame > kdbmux_intr() at kbdmux_intr+0x19/frame > taskqueue_run_locked() at taskqueue_run_locked+0x175/frame > taskqueue_run() at taskqueue_run+0x44/frame > taskqueue_swi_giant_run(0) at taskqueue_swi_giant_run+0xe/frame > ithread_loop() at ithread_loop+0x237/frame > fork_exit() at fork_exit+0x6c/frame > fork_trampoline() at 0x../frame > > > show all vmem > vmem 0x.. 'transient arena' > quantum: 4096 > size: 23592960 > inuse: 0 > free: 23592960 > busy tags: 0 > free tags: 2 > inuse size free size > 16777216 0 0 1 23592960 > vmem 0x.. 'buffer arena' > quantum: 4096 > size: 94683136 > inuse: 94502912 > free: 180224 > busy tags: 1463 > free tags: 3 > inuse size free size > 16384 2 32768 1 16384 > 32768 39 1277952 1 32768 > 65536 1422 93192192 0 0 > 131072 0 0 1 131072 > vmem 0x.. 'i386trampoline' > quantum: 1 > size: 24576 > inuse: 20860 > free: 3716 > busy tags: 9 > free tags: 3 > inuse size free size > 32 1 48 1 52 > 64 2 208 0 0 > 128 2 280 0 0 > 2048 1 2048 1 3664 > 4096 2 8192 0 0 > 8192 1 10084 0 0 > vmem 0x.. 'kernel rwx arena' > quantum: 4096 > size: 0 > inuse: 0 > free: 0 > busy tags: 0 > free tags: 0 > vmem 0x.. 'kernel area dom' > quantum: 4096 > size: 56623104 > inuse: 56582144 > free: 40960 > busy tags: 11224 > free tags: 3 I think this is the trouble. Did you tried to reduce kern.maxvnodes ? What is the default value for the knob on your machine ? We scaled maxvnodes for ZFS and UFS, might be NFS is even more demanding, having larger node size. > inuse size free size > 4096 11091 45428736 0 0 > 8192 63 516096 0 0 > 16384 12 196608 0 0 > 32768 6 196608 0 0 > 40960 2 81920 1 40960 > 65536 16 1048576 0 0 > 94208 1 94208 0 0 > 110592 1 110592 0 0 > 131072 15 2441216 0 0 > 262144 15 3997696 0 0 > 524288 1 524288 0 0 > 1048576 1 1945600 0 0 > vmem 0x.. 'kernel arena' > quantum: 4096 > size: 390070272 > inuse: 386613248 > free: 3457024 > busy tags: 873 > free tags: 3 > inuse size free size > 4096 35 143360 1 4096 > 8192 2 16384 2 16384 > 12288 1 12288 0 0 > 16384 30 491520 0 0 > 20480 140 2867200 0 0 > 65536 1 65536 0 0 > 131072 631 82706432 0 0 > 1048576 0 0 1 1339392 > 2097152 27 56623104 1 2097152 > 8388608 1 13774848 0 0 > 16777216 3 74883072 0 0 > 33554432 1 36753408 0 0 > 67108864 1 118276096 0 0 > > > alltrace > - I can't face typing too much more, but I'll put a few > here that look interesting > > - for csh > sched_switch() > mi_switch() > kern_yield() > getblkx() > breadn_flags() > ffs_update() > ufs_inactive() > VOP_INACTIVE() > vinactivef() > vput_final() > vm_object_deallocate() > vm_map_process_deferred() > kern_munmap() > sys_munmap() > > - For cc > sched_switch() > mi_switch() > sleepq_switch() > sleepq_timedwait() > _sleep() > pause_sbt() > vmem_bt_alloc() > keg_alloc_slab() > zone_import() > cache_alloc() > cache_alloc_retry() > uma_zalloc_arg() > bt_fill() > vmem_xalloc() > vmem_alloc() > kmem_alloc() > kmem_malloc_domainset() > page_alloc() > keg_alloc_slab() > zone_import() > cache_alloc() > cache_alloc_retry() > uma_zalloc_arg() > nfscl_nget() > nfs_create() > vop_sigdefer() > nfs_vnodeops_bypass() > VOP_CREATE_APV() > vn_open_cred() > vn_open() > kern_openat() > sys_openat() > > Then there are a bunch of these... for cc, make > sched_switch() > mi_switch() > sleepq_switch() > sleepq_catch_signals() > sleepq_wait_sig() > kern_wait6() > sys_wait4() > > - for vnlru > sched_switch() > mi_switch() > sleepq_switch() > sleepq_timedwait() > _sleep() > vnlru_proc() > fork_exit() > fork_trampoline() > > - for syncer > sched_switch() > mi_switch() > critical_exit_preempt() > intr_event_handle() > intr_execute_handlers() > lapic_handle_intr() > Xapic_isr1() > - interrupt > memset() > cache_alloc() > cache_alloc_retry() > uma_zalloc_arg() > vmem_xalloc() > vmem_bt_alloc() > keg_alloc_slab() > zone_import() > cache_alloc() > cache_alloc_retry() > uma_zalloc_arg() > bt_fill() > vmem_xalloc() > vmem_alloc() > bufkva_alloc() > getnewbuf() > getblkx() > breadn_flags() > ffs_update() > ffs_sync() > sync_fsync() > VOP_FSYNC_APV() > sched_sync() > fork_exit() > fork_trampoline() > > - For bufdaemon (a bunch of these) > sched_switch() > mi_switch() > sleepq_switch() > sleepq_timedwait() > _sleep() > buf_daemon() > fork_exit() > fork_trampoline() > > vmdaemon and pagedaemon are basically just like above, > sleeping in > vm_daemon() > or > vm_pageout_worker() > or > vm_pageout_laundry_worker() > or > uma_reclaim_worker() > > That's all the typing I can take right now. > I can probably make this happen again if you want more specific stuff. > > rick > > > >