Date: Sat, 24 Oct 2020 20:06:20 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Christos Chatzaras <chris@cretaforce.gr> Cc: freebsd-fs@freebsd.org Subject: Re: PHP-FPM processes hang in UFS-state Message-ID: <20201024170620.GC2643@kib.kiev.ua> In-Reply-To: <988121C5-4545-4720-8D07-814E9D02DDCA@cretaforce.gr> References: <988121C5-4545-4720-8D07-814E9D02DDCA@cretaforce.gr>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Oct 24, 2020 at 03:27:14PM +0300, Christos Chatzaras wrote: > This issue happened in 3 different servers the last months. > > Some PHP-FPM processes hang in UFS-state. > > The websites that cause the hang are all Prestashop and it's related to their var/cache files (I believe it happens when they clear the cache and the files are regenerated). > > > This is a php-fpm process that hang: > > procstat -kk 73548 > > PID TID COMM TDNAME KSTACK > 73548 100630 php-fpm - mi_switch+0xe2 sleepq_wait+0x2c sleeplk+0x185 lockmgr_xlock_hard+0x1f5 __lockmgr_args+0x2bf ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 vget+0x4a vfs_hash_get+0xb0 ffs_vgetf+0x3f softdep_sync_buf+0xbee ffs_syncvnode+0x26f ffs_truncate+0x71b ufs_direnter+0x5f8 ufs_makeinode+0x5fb ufs_create+0x34 VOP_CREATE_APV+0x76 > > > procstat -kk -a : > > https://www.coderz.gr/ufs/procstat.txt <https://pastebin.com/raw/Wd8s3AT1>; > > > mount | grep home : > > https://www.coderz.gr/ufs/mount.txt <https://www.coderz.gr/ufs/mount.txt>; > > > dumpfs /dev/mirror/gm0p7 : > > https://www.coderz.gr/ufs/dumpfs.txt <https://www.coderz.gr/ufs/dumpfs.txt>; > > > I have to force reboot the server as /home is not possible to clean unmount. Then I run fsck which fixes some issues because of the unclean reboot. Then I run fsck again that shows no errors. Then I mount /home and first rename the cache folders and finally delete var/cache and allow it to regenerate prestashop files. But the same issue happens again after few days. > > > Any idea how to troubleshoot it? Do you have quotas enabled ? Does the problem disappear with quotas turned off ? Regardless of the quotas question, If the issue is reproducable, first upgrade to 12.2 at least. Then follow instructions at https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20201024170620.GC2643>