Date: Sat, 24 Oct 2020 21:40:09 +0300 From: Christos Chatzaras <chris@cretaforce.gr> To: Konstantin Belousov <kostikbel@gmail.com> Cc: freebsd-fs@freebsd.org Subject: Re: PHP-FPM processes hang in UFS-state Message-ID: <1B608C97-944A-4FE6-A867-42ECFB5A1E8C@cretaforce.gr> In-Reply-To: <20201024181319.GE2643@kib.kiev.ua> References: <988121C5-4545-4720-8D07-814E9D02DDCA@cretaforce.gr> <20201024170620.GC2643@kib.kiev.ua> <74B608B9-B36E-41D2-B298-4F829A727403@cretaforce.gr> <20201024181319.GE2643@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
> On 24 Oct 2020, at 21:13, Konstantin Belousov <kostikbel@gmail.com> = wrote: > I do not remember about 12.1. I certainly not aware of anything in = 12.2. > Quota is less tested than non-quota. One of the backtraces in your = dump > indicates that a quota file lock was preventing other thread to make a > progress. Can you tell me where I see this file lock about quota? I search = https://www.coderz.gr/ufs/procstat.txt = <https://www.coderz.gr/ufs/procstat.txt>; and didn't find it. Also "10.6. Debugging Deadlocks" write "The recipe below is especially = useful if you suspect that a deadlock occurs in the VFS layer. Add these = options to the kernel configuration file.". Do I have to compile new = kernel with these options enabled or I can try with my GENERIC kernel? If I understand correctly because the system doesn't panic but keeps = running I have to make it panic using "sysctl debug.kdb.panic=3D1" to = create the core dump. Then after the system boots I run "kgdb = /boot/kernel/kernel /var/crash/vmcore.0". Then I run the commands = described in "10.6. Debugging Deadlocks". Am I right? Here is procstat --k for all stuck processes: procstat -kk 18239 39430 54990 63240 66176 73548 83476 84493 = 92011 95629 PID TID COMM TDNAME KSTACK 18239 100333 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101 39430 100747 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101 54990 100232 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101 63240 100218 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101 66176 100256 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101 73548 100630 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_xlock_hard+0x1f5 = __lockmgr_args+0x2bf ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a vfs_hash_get+0xb0 ffs_vgetf+0x3f softdep_sync_buf+0xbee = ffs_syncvnode+0x26f ffs_truncate+0x71b ufs_direnter+0x5f8 = ufs_makeinode+0x5fb ufs_create+0x34 VOP_CREATE_APV+0x76 83476 100710 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101 84493 100835 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_accessat+0x108 amd64_syscall+0x364 = fast_syscall_common+0x101 92011 100586 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101 95629 101591 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101=20=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1B608C97-944A-4FE6-A867-42ECFB5A1E8C>