From owner-freebsd-fs@freebsd.org Sat Oct 24 18:40:15 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 524D5451384 for ; Sat, 24 Oct 2020 18:40:15 +0000 (UTC) (envelope-from chris@cretaforce.gr) Received: from relay2.cretaforce.gr (relay2.cretaforce.gr [195.201.253.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.cretaforce.gr", Issuer "RapidSSL RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CJVJt2zpHz3g5p for ; Sat, 24 Oct 2020 18:40:14 +0000 (UTC) (envelope-from chris@cretaforce.gr) Received: from server1.cretaforce.gr (server1.cretaforce.gr [138.201.248.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.cretaforce.gr", Issuer "RapidSSL RSA CA 2018" (verified OK)) by smtp2.cretaforce.gr (Postfix) with ESMTPS id 644DB1FA71 for ; Sat, 24 Oct 2020 21:40:12 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cretaforce.gr; s=cretaforce; t=1603564812; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=WO/iDwFtLGYB6jfPrhEllVB+LP7WR7PELObuj332hqk=; b=tpFAaRoRIpsi4Yg5buPDLaAL8D0cv4e/wXqPSlRHjrHNXKAdXZkyQV3MMuVDYvKmrzqciE zKimnM8H5hNRtWUSl1rL125ZVLP7kgYixUQXZmdMmAtoSEQUX9aKVhxiqsyi3fnygUsWJc bP+WHGIMMlZMcII46PT84z8FU47sRoI= Received: from macbook-air.fritz.box (athedsl-172484.home.otenet.gr [85.75.217.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: chris@cretaforce.gr) by server1.cretaforce.gr (Postfix) with ESMTPSA id AA52327330; Sat, 24 Oct 2020 21:40:11 +0300 (EEST) From: Christos Chatzaras Message-Id: <1B608C97-944A-4FE6-A867-42ECFB5A1E8C@cretaforce.gr> Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: PHP-FPM processes hang in UFS-state Date: Sat, 24 Oct 2020 21:40:09 +0300 In-Reply-To: <20201024181319.GE2643@kib.kiev.ua> Cc: freebsd-fs@freebsd.org To: Konstantin Belousov References: <988121C5-4545-4720-8D07-814E9D02DDCA@cretaforce.gr> <20201024170620.GC2643@kib.kiev.ua> <74B608B9-B36E-41D2-B298-4F829A727403@cretaforce.gr> <20201024181319.GE2643@kib.kiev.ua> X-Mailer: Apple Mail (2.3608.120.23.2.4) X-Rspamd-Queue-Id: 4CJVJt2zpHz3g5p X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cretaforce.gr header.s=cretaforce header.b=tpFAaRoR; dmarc=none; spf=pass (mx1.freebsd.org: domain of chris@cretaforce.gr designates 195.201.253.149 as permitted sender) smtp.mailfrom=chris@cretaforce.gr X-Spamd-Result: default: False [-4.12 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:195.201.253.149]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[cretaforce.gr:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.99)[-0.986]; FREEMAIL_TO(0.00)[gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[85.75.217.98:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:24940, ipnet:195.201.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[195.201.253.149:from]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.07)[-1.067]; R_DKIM_ALLOW(-0.20)[cretaforce.gr:s=cretaforce]; FREEFALL_USER(0.00)[chris]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.97)[-0.967]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[cretaforce.gr]; DWL_DNSWL_LOW(-1.00)[cretaforce.gr:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Oct 2020 18:40:15 -0000 > On 24 Oct 2020, at 21:13, Konstantin Belousov = wrote: > I do not remember about 12.1. I certainly not aware of anything in = 12.2. > Quota is less tested than non-quota. One of the backtraces in your = dump > indicates that a quota file lock was preventing other thread to make a > progress. Can you tell me where I see this file lock about quota? I search = https://www.coderz.gr/ufs/procstat.txt = and didn't find it. Also "10.6. Debugging Deadlocks" write "The recipe below is especially = useful if you suspect that a deadlock occurs in the VFS layer. Add these = options to the kernel configuration file.". Do I have to compile new = kernel with these options enabled or I can try with my GENERIC kernel? If I understand correctly because the system doesn't panic but keeps = running I have to make it panic using "sysctl debug.kdb.panic=3D1" to = create the core dump. Then after the system boots I run "kgdb = /boot/kernel/kernel /var/crash/vmcore.0". Then I run the commands = described in "10.6. Debugging Deadlocks". Am I right? Here is procstat --k for all stuck processes: procstat -kk 18239 39430 54990 63240 66176 73548 83476 84493 = 92011 95629 PID TID COMM TDNAME KSTACK 18239 100333 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101 39430 100747 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101 54990 100232 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101 63240 100218 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101 66176 100256 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101 73548 100630 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_xlock_hard+0x1f5 = __lockmgr_args+0x2bf ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a vfs_hash_get+0xb0 ffs_vgetf+0x3f softdep_sync_buf+0xbee = ffs_syncvnode+0x26f ffs_truncate+0x71b ufs_direnter+0x5f8 = ufs_makeinode+0x5fb ufs_create+0x34 VOP_CREATE_APV+0x76 83476 100710 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101 84493 100835 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_accessat+0x108 amd64_syscall+0x364 = fast_syscall_common+0x101 92011 100586 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101 95629 101591 php-fpm - mi_switch+0xe2 = sleepq_wait+0x2c sleeplk+0x185 lockmgr_slock_hard+0x1ae = __lockmgr_args+0xe7 ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 = vget+0x4a cache_lookup+0x9e7 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x76 = lookup+0x6d1 namei+0x437 kern_statat+0x72 sys_fstatat+0x2f = amd64_syscall+0x364 fast_syscall_common+0x101=20=