From owner-freebsd-hackers@freebsd.org Mon Oct 5 00:41:43 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 32AEB3F1051 for ; Mon, 5 Oct 2020 00:41:43 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670072.outbound.protection.outlook.com [40.107.67.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C4MH974GMz4ShY for ; Mon, 5 Oct 2020 00:41:41 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=L2bmZTt7OGD36RFTMLG/7MtioIjF6II8Nk++0WEULwZ+btFZ6lrKBMZLWdVVbZHQEqS76BPF3mV4mHzvHTcJC6WW17wMKlR6SXgD0bzsSwNCgQS3TJzF/47fI/yubJX/jBRdlMP/zv6Kx5fMQ4cIxYP1afhvaYmPTTo+1hYkjHNB+Sa7s3jJAou0ogSw4BTAtSRPSqifZkmBwhPAEwjcMKGRzhXY5TLz2DESHkPdHO4wOBpUuG4d3aKUt4kbyLlcCgVuR7KxiQOCaKeRsBnYBtX64u6rUHr655Zh5Vzl1fypYihal+8SkBD91IB3xxSOZfR3vNbFN1KYfSBX0wkx+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UqEpeCeHfZP4e/CP2dwwglkVdlYF/lOCpeRwM0vxGFM=; b=Mo9pWWbCmuOGpSIfq5Yt7fgyDzVpP6gasGu7giLCVf/RiXfCbArOeHC9FSzKPr4xYHy7RWpIx07YH357Wy8MBMQM6yaaNLSNuB1Cxg2BBZP1s76Z8NjcCo6LuRwT5ffNPxa6LZcTIdlDSXGPDTVQsEt4Wwp0OBzL6WaV14xkYL8X8BdWiI/ddoMAh1XHJiYOGDfqBEI6b2nClCQntKWcvdwkOSnMDhyrovxkPkL/T7o4sq9XR/R9vf15MgG6HuBSnhhl8ChJ/LlzO/0MPyMIXstHAAUGLzRjHIkEo0CkpAMXcjutaZFGTopbva6vNLzlIqtVM3wGgDmvGh8v5+IlMQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UqEpeCeHfZP4e/CP2dwwglkVdlYF/lOCpeRwM0vxGFM=; b=GDUr6ReENPLMujiXmZjT0vaG/lefV1H5y6rN0kzBlMeDY5W81EjOuTdYOaShRxPIVksxwnfM6ZUNxw/Geu3xRde7JarF+STqUTx1qi1iRpbMBC+cUYWA31Or5SzSmCSQNR79DBqRqFw3MEJpo0FcRUrhdQvoNFir4Xvj2qbEwaPJ3c5sxRKS4yLcalaZT0SgL7x7ouGpPgzyQcW0x4IaDgzccJi7AxyesquhDIn6MxkgEyfXjTD/O4Xeo+tZH7oHP6yGaTLcCqBrNn5+irWlgLzOpXSDC9yUp5zxzfGUHFHYwdtDljX8vA04Vl/AqBhZMNIJ2wPwWulRGrPggWUa5Q== Received: from YTBPR01MB3966.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:24::27) by YTOPR0101MB2121.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:1a::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3433.36; Mon, 5 Oct 2020 00:41:39 +0000 Received: from YTBPR01MB3966.CANPRD01.PROD.OUTLOOK.COM ([fe80::687f:d85a:a0a3:bd20]) by YTBPR01MB3966.CANPRD01.PROD.OUTLOOK.COM ([fe80::687f:d85a:a0a3:bd20%6]) with mapi id 15.20.3433.044; Mon, 5 Oct 2020 00:41:39 +0000 From: Rick Macklem To: Meowthink , FreeBSD Hackers Subject: Re: How this wired boot timing bug comes, init rc scripts or zfs? Thread-Topic: How this wired boot timing bug comes, init rc scripts or zfs? Thread-Index: AQHWmfkl7m9uvwZgSESfOu7kxuLuh6mH8/LQgAAFTgCAACiqgIAAB7lA Date: Mon, 5 Oct 2020 00:41:39 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: bd81c107-25fa-4856-4633-08d868c76c56 x-ms-traffictypediagnostic: YTOPR0101MB2121: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: soy5LLE99TdupxBPhvCZVxi4IqGy1xAvCCOfO6KTFZhbJL8B3bi4EbDDF0uiAaJu2ZytQ987RF/wPIZIak2zqsQHI8a6XHfykSVvfPU15PnMPRbIyeAy8MRxenUg7iht00LylI4mmSdqyyJS83Sv9+xdIlgUvyOqV6Rz+6B+XuTcgPGlO/wENRhSkVWhy+JfEn/yFNUzFhhRnVASK0XyV5YqN/iqzeW+8G+vMlCPi0Qrq91cJMSwvU17SbFWUd/zd77H8k1MNMZzBlYKVOgq6MUfm2Op5JvewIHIwPrICTeYg6YvSbQczU+QvF4kxWcObdDG+kNBLmi6qttbeMH8wmXy18wmxMl4NYyM3E9pbiWpQU5iqlUSWRTSzcBum8y500VGpbeKs+ubtOHwrBiqaQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YTBPR01MB3966.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(396003)(376002)(39850400004)(366004)(136003)(346002)(966005)(66446008)(2906002)(186003)(83080400001)(83380400001)(110136005)(6506007)(64756008)(8936002)(478600001)(5660300002)(86362001)(316002)(71200400001)(786003)(8676002)(66946007)(66556008)(66476007)(76116006)(52536014)(7696005)(9686003)(55016002)(33656002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: 4gxQe7EbcXZYPi8ruDepVgRadPTGk6eZJOeipp6VJlnvXniHfTbRlQkLlFucFSeNP0Uz+alRwQHTeQMVs1FLhNO9yghjsWVDtz41rHOJzzjHsp9i2kWEM+QNELR4+DnJjjzxTE7gDHWygxN8rnwo6qIqX+bOvHfkv4FDaOTKnIAxI8m6eJk/XYCIGrW8wpT+aWe8B0irnsX7L/uUZPpWme4xQnI2X74FbQMPl2RDC5lnSpVBcRbP42gpa59LEXNiFKqKsVuXAx+f8HRW7FuUhsaeyBU6WJrLLwBzvA76O+T0MpbNKY/q57hAbaBTxe0aJSfv5IdBLS1plyV6JxUuESSY2RlI+AAzmy+yJcnKo+/6LLFzJOieH+rDMQ6zoF3IcZgM8bYr4aHmIMdlTvXXDZrsPKTICh2NMCRhqPy39TVGA17xZClL64Z4L/E/e3Ms7dgy2lO0tlRsZEnGhOUgr20V/kfM2DvvE57jTXCYIYozIZezN/gNagicy4gDltgTYp6RVmQrnALb6xXIA0LAsodMX0+MNpCnH8O1CASwIb+O+OqfRYWtWq8AfFDL0lIbD6PwNcBAoJ4mMTsL2u6lhF8YcdSlvFlPlhZNz7yNAF8I533t4YJ8P4eQyO4e9VwD9txPwtCTxtK1IdWIIDlDIUaJinQaOjyvOx1wYXjRHfW8KRxjFObjNmAGaG5PkRpY6lgd90S/w3NhRmhKRPfQRw== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YTBPR01MB3966.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: bd81c107-25fa-4856-4633-08d868c76c56 X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Oct 2020 00:41:39.7779 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: zY8DjEPT1MLxNM7zKWdDAwByEag3kxTtSydptgCCnnHSqbGvpN3BKOfSFxv6FihdSCcF0uAfwgAdwyb8abVEyw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB2121 X-Rspamd-Queue-Id: 4C4MH974GMz4ShY X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=GDUr6ReE; dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.72 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-5.29 / 15.00]; NEURAL_HAM_MEDIUM(-0.99)[-0.991]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.02)[-1.019]; MIME_GOOD(-0.10)[text/plain]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-1.18)[-1.184]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.72:from]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers]; RCVD_IN_DNSWL_LOW(-0.10)[40.107.67.72:from] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2020 00:41:43 -0000 Meowthink wrote:=0A= >On Mon, Oct 5, 2020 at 5:37 AM Warner Losh wrote:=0A= >>=0A= >>=0A= >>=0A= >> On Sun, Oct 4, 2020, 3:28 PM Rick Macklem wrote:= =0A= >>>=0A= >>> Meowthink wrote:=0A= >>> >Hello hackers,=0A= >>> >Recently I installworld and rebooted a server, seems working, but my= =0A= >>> >kerberized nfsd, precisely gssd, is not functional.=0A= >>> >At first I thought it may be a bug from stable, so I did some trivial= =0A= >>> >tests, replacing the kernel with releng one, then the whole world, but= =0A= >>> >found this is nothing related to the kernel, and triggers randomly=0A= >>> >when rebooting some recent stable/11 world (releng/11.4 seems fine).= =0A= >>> >To dig it deeper, here is what the console showed when failing:=0A= >>> >```=0A= >>> >Oct 3 20:23:59 r kernel: Starting file system checks:=0A= >>> >Oct 3 20:23:59 r kernel: /etc/rc: WARNING: run_rc_command: cannot run= =0A= >>> >/usr/sbin/gssd=0A= >>> >Oct 3 20:23:59 r kernel: Mounting local filesystems:.=0A= >>> >Oct 3 20:23:59 r kernel: Updating CPU Microcode...=0A= >>> >Oct 3 20:23:59 r kernel: Done.=0A= >>> >Oct 3 20:23:59 r kernel: Starting ctld.=0A= >>> >Oct 3 20:23:59 r kernel: ctld: bind(2) failed for [::]: Can't assign= =0A= >>> >requested address=0A= >>> >Oct 3 20:23:59 r kernel: ctld: bind(2) failed for 0.0.0.0: Can't=0A= >>> >assign requested address=0A= >>> >Oct 3 20:23:59 r kernel: ctld: failed to apply configuration; exiting= =0A= >>> >Oct 3 20:23:59 r kernel: /etc/rc: WARNING: failed to start ctld=0A= >>> >Oct 3 20:23:59 r kernel: ELF ldconfig path: /lib /usr/lib=0A= >>> >/usr/lib/compat /usr/local/lib /usr/dt/lib /usr/local/lib/compat=0A= >>> >/usr/local/lib/gcc9 /usr/local/lib/graphviz /usr/local/lib/nss=0A= >>> >/usr/local/lib/perl5/5.28/mach/CORE /usr/local/lib/pth=0A= >>> >/usr/local/lib/qt4 /usr/local/lib/qt5 /usr/local/lib/samba4=0A= >>> >/usr/local/llvm10/lib /usr/local/share/chromium=0A= >>> >Oct 3 20:23:59 r kernel: 32-bit compatibility ldconfig path:=0A= >>> >/usr/lib32 /usr/local/lib32/compat=0A= >>> >Oct 3 20:23:59 r kernel: Setting hostname: r.domain.net.=0A= >>> >Oct 3 20:23:59 r kernel: Setting up harvesting:=0A= >>> >[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD= ,ATTACH,CACHED=0A= >>> >Oct 3 20:23:59 r kernel: Feeding entropy: .=0A= >>> >Oct 3 20:23:59 r kernel: Starting Network: lo0 bge0 bge1.=0A= >>> >```=0A= >>> >Then I realized I have / and /usr in separated zfs(same zpool). It may= =0A= >>> >be that / mounted but /usr not. Thus I changed my /etc/rc.d/gssd line= =0A= >>> >7 to # REQUIRE: mountcritlocal.=0A= >>> I don't know what has changed post-11.4 to cause this.=0A= >>> However, the problem with adding "mountcritlocal" is that it assumes=0A= >>> /usr is a locally mounted file system and not NFS mounted nor a subtree= =0A= >>> of "/".=0A= >>> --> I think a better solution might be to move gssd to /sbin, which sho= uld=0A= >>> always be a part of the root fs. (/etc/rc.d/gssd already has "roo= t" as=0A= >>> REQUIRED.)=0A= >>=0A= >>=0A= >> Make the move. It is an early utility in reality.=0A= >>=0A= >> Warner=0A= >=0A= >But take care :)=0A= >$ ldd /usr/sbin/gssd=0A= >/usr/sbin/gssd:=0A= > libgssapi.so.10 =3D> /usr/lib/libgssapi.so.10 (0x80082b000)=0A= > libkrb5.so.11 =3D> /usr/lib/libkrb5.so.11 (0x800a35000)=0A= > libroken.so.11 =3D> /usr/lib/libroken.so.11 (0x800cb4000)=0A= > libc.so.7 =3D> /lib/libc.so.7 (0x800ec7000)=0A= > libasn1.so.11 =3D> /usr/lib/libasn1.so.11 (0x80127e000)=0A= > libcom_err.so.5 =3D> /usr/lib/libcom_err.so.5 (0x801521000)=0A= > libcrypt.so.5 =3D> /lib/libcrypt.so.5 (0x801723000)=0A= > libcrypto.so.8 =3D> /lib/libcrypto.so.8 (0x801a00000)=0A= > libhx509.so.11 =3D> /usr/lib/libhx509.so.11 (0x801e75000)=0A= > libwind.so.11 =3D> /usr/lib/libwind.so.11 (0x8020c2000)=0A= > libheimbase.so.11 =3D> /usr/lib/libheimbase.so.11 (0x8022ea000)=0A= > libprivateheimipcc.so.11 =3D> /usr/lib/libprivateheimipcc.so.11=0A= >(0x8024ee000)=0A= > libthr.so.3 =3D> /lib/libthr.so.3 (0x8026f1000)=0A= Good point. Moving all those libraries isn't worth the effort, imho.=0A= =0A= I suppose "mountcritlocal" is harmless and fixes the case where /usr=0A= is a separately mounted local FS.=0A= (I can't resist pointing out that we no longer need to worry about the=0A= size limitation of a 2.5Mbyte RK05 disk, so having /usr on a separate=0A= file system may no longer be a critical requirement, but...;-)=0A= =0A= rick=0A= =0A= >>=0A= >>=0A= >> I have a similar problem with the rpc.tlsclntd daemon I have developed= =0A= >> for NFS-over-TLS.=0A= >> - Neither sec=3Dkrb5[ip] nor tls can be used for an NFS mounted root,=0A= >> but I think we just have to live with that?=0A= >>=0A= >> Do others have any suggestions? rick=0A= >>=0A= >> By the way, /etc/rc.d/ctld to #=0A= >> REQUIRE: netif. Everything works fine, even rebooting several times.=0A= >> What I am confused is how this happens. It seems that /etc/rc.d/gssd=0A= >> (in addition, /etc/rc.d/ctld) hasn't been changed since 2016. Both=0A= >> gssd and init in stable/11 have no functional changes since=0A= >> releng/11.4. Maybe zfs? but it's in the kernel, and kernel r366306=0A= >> with releng/11.4 world works(though I only tested few times since=0A= >> rebooting is too boring).=0A= >> Any ideas?=0A= >>=0A= >> Cheers,=0A= >> meowthink=0A= >> _______________________________________________=0A= >> freebsd-hackers@freebsd.org mailing list=0A= >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers=0A= >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.or= g"=0A= >>=0A= >> _______________________________________________=0A= >> freebsd-hackers@freebsd.org mailing list=0A= >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers=0A= >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.or= g"=0A= _______________________________________________=0A= freebsd-hackers@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-hackers=0A= To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"= =0A= =0A=