From owner-freebsd-questions@freebsd.org Sat Oct 24 12:28:10 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 257AC448091 for ; Sat, 24 Oct 2020 12:28:10 +0000 (UTC) (envelope-from chris@cretaforce.gr) Received: from relay1.cretaforce.gr (relay1.cretaforce.gr [195.201.253.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.cretaforce.gr", Issuer "RapidSSL RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CJL3Y1N4Dz4VFy for ; Sat, 24 Oct 2020 12:28:09 +0000 (UTC) (envelope-from chris@cretaforce.gr) Received: from server1.cretaforce.gr (server1.cretaforce.gr [138.201.248.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.cretaforce.gr", Issuer "RapidSSL RSA CA 2018" (verified OK)) by smtp1.cretaforce.gr (Postfix) with ESMTPS id A29761FA06 for ; Sat, 24 Oct 2020 15:28:07 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cretaforce.gr; s=cretaforce; t=1603542487; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=WlKeSC23dGvU9I6VNQ7eut8WiRo6CSLAxOohyeX38uY=; b=v6DUnCgoDudlW/D8RgJ5HwjgLT4SDiNLgjfADvta/rs1e9semmmKdktBB/dX01iZgq6c5V oToDTxmScQ/uOCdBG2uJb0R7wSIQSnrz/BQi+tJzmopuYOg8svcrPcZ7xXrcjU0unOOMig VL1e7j+FnL71jAhi0RIUoPoF83R6Bjk= Received: from macbook-air.fritz.box (athedsl-172484.home.otenet.gr [85.75.217.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: chris@cretaforce.gr) by server1.cretaforce.gr (Postfix) with ESMTPSA id 4808327330 for ; Sat, 24 Oct 2020 15:28:07 +0300 (EEST) From: Christos Chatzaras Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: PHP-FPM processes hang in UFS-state Message-Id: <675E2CFF-C45D-48C2-8E55-4E9876D5098A@cretaforce.gr> Date: Sat, 24 Oct 2020 15:28:07 +0300 To: freebsd-questions@freebsd.org X-Mailer: Apple Mail (2.3608.120.23.2.4) X-Rspamd-Queue-Id: 4CJL3Y1N4Dz4VFy X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cretaforce.gr header.s=cretaforce header.b=v6DUnCgo; dmarc=none; spf=pass (mx1.freebsd.org: domain of chris@cretaforce.gr designates 195.201.253.145 as permitted sender) smtp.mailfrom=chris@cretaforce.gr X-Spamd-Result: default: False [-4.04 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:195.201.253.145:c]; MV_CASE(0.50)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[cretaforce.gr:+]; NEURAL_HAM_SHORT(-0.92)[-0.925]; RECEIVED_SPAMHAUS_PBL(0.00)[85.75.217.98:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:195.201.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[195.201.253.145:from]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.05)[-1.047]; R_DKIM_ALLOW(-0.20)[cretaforce.gr:s=cretaforce]; FREEFALL_USER(0.00)[chris]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.97)[-0.971]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; DMARC_NA(0.00)[cretaforce.gr]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_LOW(-1.00)[cretaforce.gr:dkim]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Oct 2020 12:28:10 -0000 This issue happened in 3 different servers the last months. Some PHP-FPM processes hang in UFS-state. The websites that cause the hang are all Prestashop and it's related to = their var/cache files (I believe it happens when they clear the cache = and the files are regenerated). This is a php-fpm process that hang: procstat -kk 73548 PID TID COMM TDNAME KSTACK 73548 100630 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_xlock_hard+0x1f5 = __lockmgr_args+0x2bf ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a vfs_hash_get+0xb0 ffs_vgetf+0x3f softdep_sync_buf+0xbee = ffs_syncvnode+0x26f ffs_truncate+0x71b ufs_direnter+0x5f8 = ufs_makeinode+0x5fb ufs_create+0x34 VOP_CREATE_APV+0x76=20 procstat -kk -a : https://www.coderz.gr/ufs/procstat.txt mount | grep home : https://www.coderz.gr/ufs/mount.txt dumpfs /dev/mirror/gm0p7 : https://www.coderz.gr/ufs/dumpfs.txt I have to force reboot the server as /home is not possible to clean = unmount. Then I run fsck which fixes some issues because of the unclean = reboot. Then I run fsck again that shows no errors. Then I mount /home = and first rename the cache folders and finally delete var/cache and = allow it to regenerate prestashop files. But the same issue happens = again after few days. Any idea how to troubleshoot it?=