From nobody Thu May 12 02:08:44 2022 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id C66251ADAA78 for ; Thu, 12 May 2022 02:08:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KzFZ43mfZz4X5Z for ; Thu, 12 May 2022 02:08:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 5EF9724E63 for ; Thu, 12 May 2022 02:08:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 24C28iLu041815 for ; Thu, 12 May 2022 02:08:44 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 24C28iPB041814 for bugs@FreeBSD.org; Thu, 12 May 2022 02:08:44 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 263908] Something spawning many "sh" process, system no longer boots, in single user /var/log empty Date: Thu, 12 May 2022 02:08:44 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 13.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: greg@teamworkweb.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1652321324; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SZMx9ykPsCKoRQgYeubND+U3bxONu8jOk7S61ZT7xCI=; b=eeue7jd8QY+MmTn1Ij7538dMUWyhl5DiyZzmCP7sPxsxWV6TFJWkVyJpuZXhg+6SQCtWcj eGLjbVanaud/QTDb9+gX0vjQudRtMArkDj92QHjwkDncfgSoKpGvyM4wzdRqEXppK5yy+y +DBKF+P/ma4PYv2EjdLaeu33FEVYV1XBJoDok4wx2wxs+eDX9lLe+yWFqNEWork5Aoz6Jz +7wqzmZtxlwutKxyzauln2AEzYOzXu/xO7AlM/m/Qo0+B2TKLODdEZ6qevIQd69Nj/J/59 CE+uv5NWGL6HYah9bMyfBTLHtazAS/s8eYWgalUw5lCqMyhe0ue1oj/6x1xVbA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1652321324; a=rsa-sha256; cv=none; b=V64qgtS96txVEhaD7jwswM8nGutfpbslBec3/JwN2cGoNyVCRSOe9lMnvE3Vtl4H5KA1ce qDy6fJTHqw3M6Fr0f/mODQh+KEM9htS1cJwe5GvacrzTQ2LQF9lRG06jYIY950xEtvuZyn 1TCwgH/acGiF2muQCnZlimftfkZRHQT+KilI1VbfJtvIJnLE4FIEpw3/tfibm1IU+k1j2r SyH6fDdqmcs0BEMF9dBm3gJv29h75BdInBMAp+U+HsFXyvl+wW+4HLRr0SjMGdvH402Kky FofbemvQMw59HWRC2I26U1uQ6IUmbMWpN1LLzNvn/zhry2CUSLwCvUUgz+a/GQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D263908 --- Comment #1 from Greg --- I have made some progress, and I have to report that something is up with z= fsd! Cannot say for sure that is what was spawning all the "sh" processes, but I suspect that to be the case. After commenting out the following from /etc/rc.conf I am now able to boot normally: #zfsd_enable=3D"YES" #service zfsd start This was after figuring out how to zfs set readonly=3Doff and mount -a my z= root pool in single user mode. And trying all of this first: - Comment out everything in /etc/crontab - Remove all the sysctl and other tunable tweaks I had customized Now it is complaining about my dRAID test pool not being available, and that appears to still be listed in zpool.cache. But it was having this issue with not booting, prior to pulling one of the HBAs (LSI 9361 mentioned previousl= y), so this issue with zfsd existed while that pool was still available. I will double check, but I am fairly sure there was nothing wrong with that pool. Regardless, I cannot imagine it is intended behavior for zfsd to prev= ent a system from booting, regardless of the state any zpools are in? Beyond perhaps serious issues with zroot, which doesn't appear to be the case here= (it passes a scrub no issues). If anyone is interested in getting more debugging, while I still have the t= est case and hardware setup for this, please let me know. Willing to put a litt= le more effort into figuring this out. Again, same setup under 13.0 was not having this issue. Same benchmarks run back to back for days on end. Same dRAID design. Same used of zfsd. Thanks! -Greg- --=20 You are receiving this mail because: You are the assignee for the bug.=