From nobody Fri Oct 7 00:26:51 2022 X-Original-To: apache@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mk8JD17lHz4ds8Z for ; Fri, 7 Oct 2022 00:26:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Mk8JC6Jbqz43lZ for ; Fri, 7 Oct 2022 00:26:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Mk8JC5BbjzQS2 for ; Fri, 7 Oct 2022 00:26:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 2970QpsN097216 for ; Fri, 7 Oct 2022 00:26:51 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 2970QpRL097215 for apache@FreeBSD.org; Fri, 7 Oct 2022 00:26:51 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" From: bugzilla-noreply@freebsd.org To: apache@FreeBSD.org Subject: maintainer-feedback requested: [Bug 266873] www/apache24: creating forked processes at 100% CPU spinning in sched_yield() Date: Fri, 07 Oct 2022 00:26:51 +0000 X-Bugzilla-Type: request X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: apache@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? Message-ID: In-Reply-To: References: X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Support of apache-related ports List-Archive: https://lists.freebsd.org/archives/freebsd-apache List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-apache@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1665102411; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4T6AHovIdm7aQa1xQj9DdQP+w7Ng9CACDec7ybaUTzs=; b=Ofg7MbTP63/XzZZ6xjpvD0lQNdiz9KjO2UbSRSPSDAzuUPTeeP/hqpuDn9CBG3sEwU7LoM Brqj/l30HDIa7asuLsoDWrLfq/Ezx5riHRu8QNA+tdc/p0XMctO6zcHEoMEd+QKAvt+de4 5oeNry//W9Dggtn97xW+FGxnczvzreMO6fv5BQh/nZ1EkerfYb6AtgzUuORJcsQXemmWDP GMlfmYV0JlMiQ1YCdbkMNI1cfAH1/PPV245PD1TLuwRiys8h6KhTGSOh57qoPCUTHDK7xS t4KaVsoo1JCTRuzP4e78pdRBDfZ6SyzcV29oNI89BgZi3pFZZsMh31WXv4pQQw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1665102411; a=rsa-sha256; cv=none; b=dleAjuwh+76j17lro6nRyEUZsEJ6cpO7holKqGPljJGaYjjc50zcIpZWJgeyfYBjDKCGrv MuSj7BdYpFSGuSPtpq0KODVD1xEKm4R7eHxCErd5WviPiWJFlJYRGg2rUjfLennzLH6SkD 5j5rrMenQrdSQdnggHXc1mNOJYKPDaOdEcmc3mNw9V/qwRmH2AuMoXuFl4Vyl8sOGfS78G DaZiPL0iIDSpdjHEWCXK1P1LtNeG/U4Vz653zrw6rciNRRoOCKJ7TCR0yfUwTKQzU2aTSL WUNLZqSx11XAomeVCnI0HPVoSunMJHNqCuC0SWOpVG/++koas5/uoC89JDD8/Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N Bugzilla Automation has asked freebsd-apache (Nobody) for maintainer-feedback: Bug 266873: www/apache24: creating forked processes at 100% CPU spinning in sched_yield() https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D266873 --- Description --- Apache version: apache24-2.4.54 - build with poudriere FreeBSD: 12.3-STABLE FreeBSD 12.3-STABLE stable/12-3a9e60169ee GENERIC amd64 Since updating OS and apache a couple of weeks ago, I have noticed that my system is getting filled up with httpd processes consuming 100% CPU.=20 Here is an example with one rogue process, but after a day of running the system will be eventually full of httpd processes spinning at 100%.=20 root 9947 100.0 0.0 207000 41380 - RJ 10:59 18:42.98 /usr/local/sbin/httpd -DNOHTTPACCEPT www 10243 1.4 0.1 310180 138488 - SJ 11:17 0:01.44 /usr/local/sbin/httpd -DNOHTTPACCEPT www 10059 0.4 0.1 242068 71156 - SJ 11:06 0:03.63 /usr/local/sbin/httpd -DNOHTTPACCEPT root 9738 0.0 0.0 207000 41388 - SsJ 10:37 0:00.35 /usr/local/sbin/httpd -DNOHTTPACCEPT www 9853 0.0 0.1 244096 70872 - SJ 10:47 0:06.49 /usr/local/sbin/httpd -DNOHTTPACCEPT www 9887 0.0 0.1 242500 69808 - IJ 10:51 0:08.48 /usr/local/sbin/httpd -DNOHTTPACCEPT www 10058 0.0 0.1 307876 136996 - IJ 11:06 0:04.76 /usr/local/sbin/httpd -DNOHTTPACCEPT www 10136 0.0 0.1 242592 71452 - IJ 11:12 0:02.50 /usr/local/sbin/httpd -DNOHTTPACCEPT www 10144 0.0 0.1 244644 71328 - IJ 11:14 0:01.31 /usr/local/sbin/httpd -DNOHTTPACCEPT www 10166 0.0 0.1 244648 73028 - SJ 11:15 0:00.99 /usr/local/sbin/httpd -DNOHTTPACCEPT www 10239 0.0 0.1 209800 61568 - IJ 11:16 0:00.20 /usr/local/sbin/httpd -DNOHTTPACCEPT Usual behavior with prefork will be an initial process running as root, whi= ch then forks additional processes and sets them to run as www user. What appe= ars to be happening here is that when the fork occurs, something gets stuck bef= ore it is able to switch to www.=20 In the case of the rogue process above, output of truss -p 9947 is simply a continuous loop of: sched_yield() =3D 0 (0x0) For some extra context, I am running this inside a jail, hosting nextcloud = with mod_php80, mysql, redis. None of my other web apps are having this issue bu= t it could be something which only appears under load and nextcloud is certainly= the heaviest application here.=20 Also, I tried switching over to php-fpm with mpm_event and the same issue w= as encountered there, so it certainly feels like something core to apache's forking code at play here.