From nobody Mon Jul 10 09:39:35 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QzzVl61YMz2trx1 for ; Mon, 10 Jul 2023 09:39:43 +0000 (UTC) (envelope-from jfc@mit.edu) Received: from outgoing-exchange-7.mit.edu (outgoing-exchange-7.mit.edu [18.9.28.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.outgoing-exchange.mit.edu", Issuer "InCommon RSA Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QzzVl49vQz3CQZ for ; Mon, 10 Jul 2023 09:39:43 +0000 (UTC) (envelope-from jfc@mit.edu) Authentication-Results: mx1.freebsd.org; none Received: from w92exedge4.exchange.mit.edu (W92EXEDGE4.EXCHANGE.MIT.EDU [18.7.73.16]) by outgoing-exchange-7.mit.edu (8.14.7/8.12.4) with ESMTP id 36A9dd0l001338; Mon, 10 Jul 2023 05:39:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mit.edu; s=outgoing; t=1688981980; bh=f8UVnyqCxomvFx86F1+JLePkkCdNwg/db0OtFqcsL0o=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=eYcYsvaUYShFBOKj+/rFnGSmvA27zCpFmdSj53dY0kifk5CZ2Uod5ATOLs2F3SJLU XvJV4/kGv1fZgPF2DWnObuYoDmfDeMOzGSS0GOvIJpgb0dLdS5i1K34IziNTKIxmdZ Rl7SFMejGqqzuNJ5IX/ERpiI3LMoe9fEo+faKKZdv5DXehqJefHHe1qofK8PYRlbtX XGmi1RVMTrL+/rHBZjxvu0kK9L9bzb1doIqzSWzs2/x3e1hHgKfHUFZ5B9NyFgcF19 edjq8Z/l6tP38Wwd1O3BWdN0T5pawKn3fURNri5/pH/h3KGgAH2IvdkjO/amfL+c12 0y6AAFoM+ZrHQ== Received: from w92expo19.exchange.mit.edu (18.7.74.73) by w92exedge4.exchange.mit.edu (18.7.73.16) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Mon, 10 Jul 2023 05:38:57 -0400 Received: from oc11exhyb7.exchange.mit.edu (18.9.1.112) by w92expo19.exchange.mit.edu (18.7.74.73) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Mon, 10 Jul 2023 05:39:38 -0400 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.172) by oc11exhyb7.exchange.mit.edu (18.9.1.112) with Microsoft SMTP Server (TLS) id 15.0.1497.48 via Frontend Transport; Mon, 10 Jul 2023 05:39:38 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EtKFCcL60o64DCdxMralRVfmOKwJDwGRgIWVEuj8mr14JO4AaJgnW4DpIUPmD16eplTjv3NA3jdmJWmshoLNhDqFY4kHqdkqNDd8l0EvdCsmWOPwZYFXUNNalUU8OTxsBy4vVaX2ZwxnY4Yo8+skqadD4MTBsGZvdv3sNSIZwViCAi1FLdrPPtVXqhscsGl4SmTDOinjrb7GxPG4rD3H3z42sz1jZYNIGVKzXsZbW1ay8E6kWIwaOujoXEL1vnsiRLbJvPNhck9y38oBoRlTXV/AOVtr1gigEQwSKm1QVRSNiOuuiCswPmVZCs7rWUCYXNW0RQysFX57k2RR2AC5kw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=f8UVnyqCxomvFx86F1+JLePkkCdNwg/db0OtFqcsL0o=; b=BZdzPXIu+2nNskpYGt8U7nxWA7Kb1ZV/JGxorw6v4MdYnOFTJAeUdRoSCXywcN4Ei8PLQR7wesrdA+pOR4spmlGRJSOK5Z7fR7YjkDeRDnRuAN0tPjCWtvofBzXT0nYR4WFjsrhY/2hPoCKwS+WJ6OyHa4mvr6WJdIM9FhkTeg6+0unsZzV+Za3CVL/b8iN6xbOp/f9zvjghmDa60FtIo1+q+NAyNaWNneBp68mrGE7y5zDYXb7hlRh+Jci0o1Gxv9PnGf58Ytia3xW+U839rnbLIsxTBUozuRF6/ld6YqotZEs9Aio0+s5qZMwsdPhrawojM9wn3vPvRCuE9sSqmA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mit.edu; dmarc=pass action=none header.from=mit.edu; dkim=pass header.d=mit.edu; arc=none Received: from SA3PR01MB8450.prod.exchangelabs.com (2603:10b6:806:382::17) by MW4PR01MB6162.prod.exchangelabs.com (2603:10b6:303:79::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.31; Mon, 10 Jul 2023 09:39:35 +0000 Received: from SA3PR01MB8450.prod.exchangelabs.com ([fe80::19cc:d2bf:6e20:ea9e]) by SA3PR01MB8450.prod.exchangelabs.com ([fe80::19cc:d2bf:6e20:ea9e%5]) with mapi id 15.20.6565.028; Mon, 10 Jul 2023 09:39:35 +0000 From: John F Carr To: Konstantin Belousov CC: Current FreeBSD Subject: Re: shell hung in fork system call Thread-Topic: shell hung in fork system call Thread-Index: AQHZsrZ/ESS/lm7aiEed00HTHSgdc6+yE+cAgAAC3gCAAAaggIAAogAA Date: Mon, 10 Jul 2023 09:39:35 +0000 Message-ID: <1684E4FD-8C43-4D2C-BC34-659A263BBBAB@mit.edu> References: <909E2C96-3BFA-41AD-8EE7-0902231C2B95@mit.edu> <52A8F775-17D9-4240-A444-98AD5339622F@mit.edu> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SA3PR01MB8450:EE_|MW4PR01MB6162:EE_ x-ms-office365-filtering-correlation-id: e2f649d4-0a44-4345-bd5b-08db81299266 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: nKTyOcUplQ4xKgrZguNZXI8Ci5Zor6RxdbVR9p3SbmBYYXgfytu+RKs7I8QzilAUc7ba9W1eWyHCwp3HtYSH8NlD6iXdIhyiNRsCOt5TIXEmjcOvFC6VERCOlJrm/GzGgq/mBfDHoFRA+dF4lo3xpuwi7AVhO0e2nFAHphMdq/iVbiQ/4BJUR5V9FSzBZ3Yp+sqZSwIL4/JQhCDcxwIavAjEiEA3XSPmE+1UUuATCTyQF0bQzj3zriJQM1SHOlCAB5a93XlDN46g5AXDrEFyvROE0kM5YAYVvRqhpaeTqhgI4guSfppcATy3rEJyxj63+vCHfGueOtun93ZACBVpMRbil21AnIdWvOG03Gkf3hUW8ysKlPUX1/HUrkoRpbRBSEyg0xZl3VuklYkJIXrvb5bV7FuLFnNwClooprjkEIJuyslpcXXbk5zmj8YRGidly20NOhyiMuELG4wO3pvAHUhdgHIPc3BbXp/rCWigSLQ3uP8H63IsTav6GMxDOQYuEe995RK3pRgWvdKUEzm7cv43axPa7R5OzWDy9P9NDaXzTadzxHFQlMC8wAqPeO5jkcUzl4fXEVqH+dGn06KGW2JxOGDrUnDibzIrEFfVq+/29l0FupFfCT2nbhsRFQHo x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SA3PR01MB8450.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(366004)(396003)(136003)(39860400002)(346002)(376002)(451199021)(38070700005)(36756003)(33656002)(86362001)(38100700002)(75432002)(122000001)(478600001)(71200400001)(6486002)(6512007)(8676002)(8936002)(5660300002)(2906002)(786003)(316002)(91956017)(6916009)(4326008)(76116006)(66946007)(66556008)(66476007)(66446008)(64756008)(41300700001)(2616005)(26005)(6506007)(53546011)(186003)(83380400001);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?lOICi041H/ujw5uj+rRfFVApXIcGIhwoV75eoleKK6jahsJ6enhxom4xUT+F?= =?us-ascii?Q?od6PnEBz9gh1HjvHB6M3R4auuUafNzGFQYVCLJUy+RDmUqkiUojKcJryHhD5?= =?us-ascii?Q?irq6LRdcH7AqrAWUHgRJGvCnxylFaQk/70ZlS+P/w3FQnh8ZXX55RodTF+BD?= =?us-ascii?Q?5dh4obZWMKfnU+amlA+plR9BABFAiyZjHbHGdTFvwXg6xcleLRISsbb2t/dO?= =?us-ascii?Q?7yx94t8vwP97X+DtaEhji4PfGM8LF2ryQ2iSJPh5k3kSy9RENy0UQpeTIOsg?= =?us-ascii?Q?4uvUdEslxz6XRKS5AFqHBtU+hB6luu7yRMALN4LxiMsekN87MEYPftrn5b2k?= =?us-ascii?Q?Rh2+yx03Tj1BDh54GH2Uzzor8JV3IrOidsHmJPOOIMsME3p/IAulfn3BEDpM?= =?us-ascii?Q?3Nbd5/5laiZb/ijWmDMNNyrN6+EbR3gZEqi6XN/47I6kczso/h7AqwoHLf9/?= =?us-ascii?Q?aqQSn5hJnuUXoh9BXpB6gl4MmfDzWYrbqEH+1UrvCPKSxWca86Q/Uxuc6OAk?= =?us-ascii?Q?3dcNoxuZ900YABSqTd/BhMj6mUWW9fWkstGAls+TAmhpxUDyoZnfEkgRrO3w?= =?us-ascii?Q?8xi0yrFE820+x0dnX5F6270uNYZkl4OlnjC6G4yTatPfdmC85Uk9UmIuLht9?= =?us-ascii?Q?PwZ+rNjY7YghSXEdNo7VpTpAsqisBPmc4BnZufUt6B75zdRpfkwUX99S6Sjb?= =?us-ascii?Q?LlPRAmZ5zAGexcmpXmRe5H9WN2IOz+pC9k+PPKqn4Xb0JIwf20D4tqGA/OTN?= =?us-ascii?Q?+EqcchxFb0wH4OpShOZjnmnz+FpTQnqKc4gZeq6v7mj9GoXl0MxnL2apBfae?= =?us-ascii?Q?5SMaVEE+vbgv5t+beXnPLTkB2+MHDWf9FjdFzUsFHgXTLvoUL5OcZhOk1QJj?= =?us-ascii?Q?EecX6J2iZFvJxEKB9WnL0E0TWXYBeC878CeZEpl9JF2+skGMweMfPgs4kVdG?= =?us-ascii?Q?8XSRH2lnZeU3JFsmy+Ss9vKYQxScxi2l8pBh6aEweOZhTnk+vtJQiY09hcu/?= =?us-ascii?Q?dhIr06Z42wI5ru+4JwfMz6w3M+oJH/G+D6UhOghQYPGA0AC/TAc7zURSwYZ8?= =?us-ascii?Q?2OXL3S2Vr/gSt3v60QPrUytVaH6Esg11NBWZ5N20J2TG6ETGz5rnxAwf9B46?= =?us-ascii?Q?zfzmBut+1siILm/vZUSA1gG4DMF6OY8nodMJ+aDZdbDOOqjf17MzVGHA9iIT?= =?us-ascii?Q?ZpWdAL8rl1udnWE/+qjN/RNDJgsvOFilWDa45BsyualPOVJd2xLHqxHmNtT9?= =?us-ascii?Q?3DHAudTMnPeWxylEO62tv8b/1fdSuJLK+rWXuHaZbXG4pDgO7iIPiK1hu3/D?= =?us-ascii?Q?RSaaFv5QzRyhK+GRFIgiDMiHPgsalBqWQ0yXS59ZNT8Bx52q8mg4dZoT24NV?= =?us-ascii?Q?onhn3xV3JuZdPyJ27V6aGJxz8/rCzSnRtRBBLbxqcsIhLvA3SO8EzPHSmqug?= =?us-ascii?Q?CtL6QP7BM9N+j1zy/HuDsaeRZtCxMSdz6UixnZzHlwSWBWVoTdh1HHHO/bzX?= =?us-ascii?Q?7jR4Yk05iaMrOPeGzKeD3ocT5KZbeo4rpCXTbN6rfQ31rScvXFyLAtFyl70/?= =?us-ascii?Q?dR9WRSpg1T74pmLIYlg=3D?= Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SA3PR01MB8450.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: e2f649d4-0a44-4345-bd5b-08db81299266 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Jul 2023 09:39:35.2347 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 64afd9ba-0ecf-4acf-bc36-935f6235ba8b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: wNpVO6eDkj4jrltbzBjNUoaymLBH55ObB2pV/Zl1MaSmvU99PwOFAjhDLV4mfMNX X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR01MB6162 X-OriginatorOrg: mit.edu X-Rspamd-Queue-Id: 4QzzVl49vQz3CQZ X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:3, ipnet:18.9.0.0/16, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N > On Jul 9, 2023, at 19:59, Konstantin Belousov wrote= : >=20 > On Sun, Jul 09, 2023 at 11:36:03PM +0000, John F Carr wrote: >>=20 >>=20 >>> On Jul 9, 2023, at 19:25, Konstantin Belousov wro= te: >>>=20 >>> On Sun, Jul 09, 2023 at 10:41:27PM +0000, John F Carr wrote: >>>> Kernel and system at a146207d66f320ed239c1059de9df854b66b55b7 plus som= e irrelevant local changes, four 64 bit ARM processors, make.conf sets CPUT= YPE?=3Dcortex-a57. >>>>=20 >>>> I typed ^C while /bin/sh was starting a pipeline and my shell got hung= in the middle of fork(). >>>>=20 >>>>> From the terminal: >>>>=20 >>>> # git log --oneline --|more >>>> ^C^C^C >>>> load: 3.26 cmd: sh 95505 [fork] 5308.67r 0.00u 0.03s 0% 2860k >>>> mi_switch+0x198 sleepq_switch+0xfc sleepq_timedwait+0x40 _sleep+0x264 = fork1+0x67c sys_fork+0x34 do_el0_sync+0x4c8 handle_el0_sync+0x44=20 >>>> load: 3.16 cmd: sh 95505 [fork] 5311.75r 0.00u 0.03s 0% 2860k >>>> mi_switch+0x198 sleepq_switch+0xfc sleepq_timedwait+0x40 _sleep+0x264 = fork1+0x67c sys_fork+0x34 do_el0_sync+0x4c8 handle_el0_sync+0x44=20 >>>>=20 >>>> According to ps -d on another terminal the shell has no children: >>>>=20 >>>> PID TT STAT TIME COMMAND >>>> [...] >>>> 873 u0 IWs 0:00.00 `-- login [pam] (login) >>>> 874 u0 I 0:00.17 `-- -sh (sh) >>>> 95504 u0 I 0:00.01 `-- su - >>>> 95505 u0 D+ 0:00.05 `-- -su (sh) >>>> [...] >>>>=20 >>>> Nothing on the (115200 bps serial) console. No change in system perfo= rmance. >>>>=20 >>>> The system is busy copying a large amount of data from the network to = a ZFS pool on spinning disks. The git|more pipeline could have taken some = time to get going while I/O requests worked their way through the queue. I= t would not have touched the busy pool, only the zroot pool on an SSD. >>>>=20 >>>> Has anything changed recently that might cause this? >>>=20 >>> There was some change around fork, but your sleep seems to be not from >>> that change. Can you show the wait channel for the process? Do someth= ing >>> like >>> $ ps alxww >>>=20 >>=20 >> UID PID PPID C PRI NI VSZ RSS MWCHAN STAT TT TIME COMMA= ND >> 0 95505 95504 2 20 0 13508 2876 fork D+ u0 0:00.13 -su (= sh) >>=20 >> This is probably the same information displayed as [fork] in the output = from ^T. >>=20 >> Does it correspond to the source line >>=20 >> pause("fork", hz / 2); >>=20 >> ? >=20 > Yes, it is rate-limiting code. Still it is interesting to see the whole > ps output. >=20 > Do you have 7a70f17ac4bd64dc1a5020f in your source? No, I do not have that commit. The comment mentions livelock. CPU use as reported by iostat did not chang= e after the process hung.