From owner-freebsd-fs@freebsd.org Sat Oct 24 19:16:24 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 37FAE451F11 for ; Sat, 24 Oct 2020 19:16:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CJW6Z5HTbz3yx3 for ; Sat, 24 Oct 2020 19:16:22 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 09OJGCvD070547 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 24 Oct 2020 22:16:15 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 09OJGCvD070547 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 09OJGCUM070546; Sat, 24 Oct 2020 22:16:12 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 24 Oct 2020 22:16:12 +0300 From: Konstantin Belousov To: Christos Chatzaras Cc: freebsd-fs@freebsd.org Subject: Re: PHP-FPM processes hang in UFS-state Message-ID: <20201024191612.GF2643@kib.kiev.ua> References: <988121C5-4545-4720-8D07-814E9D02DDCA@cretaforce.gr> <20201024170620.GC2643@kib.kiev.ua> <74B608B9-B36E-41D2-B298-4F829A727403@cretaforce.gr> <20201024181319.GE2643@kib.kiev.ua> <1B608C97-944A-4FE6-A867-42ECFB5A1E8C@cretaforce.gr> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1B608C97-944A-4FE6-A867-42ECFB5A1E8C@cretaforce.gr> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4CJW6Z5HTbz3yx3 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-1.08 / 15.00]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_MEDIUM(-0.71)[-0.707]; R_SPF_SOFTFAIL(0.00)[~all:c]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.46)[-0.456]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.08)[0.084]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-fs]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Oct 2020 19:16:24 -0000 On Sat, Oct 24, 2020 at 09:40:09PM +0300, Christos Chatzaras wrote: > > > > On 24 Oct 2020, at 21:13, Konstantin Belousov wrote: > > I do not remember about 12.1. I certainly not aware of anything in 12.2. > > Quota is less tested than non-quota. One of the backtraces in your dump > > indicates that a quota file lock was preventing other thread to make a > > progress. > > Can you tell me where I see this file lock about quota? I search https://www.coderz.gr/ufs/procstat.txt and didn't find it. 30 100097 syncer - mi_switch+0xe2 sleepq_wait+0x2c sleeplk+0x185 lockmgr_xlock_hard+0x1f5 __lockmgr_args+0x2bf ffs_lock+0xa5 VOP_LOCK1_APV+0x7c _vn_lock+0x44 vget+0x4a qsync+0x99 ffs_sync+0x7d5 sync_fsync+0x10f VOP_FSYNC_APV+0x7c sched_sync+0x379 fork_exit+0x83 fork_trampoline+0xe > > > Also "10.6. Debugging Deadlocks" write "The recipe below is especially useful if you suspect that a deadlock occurs in the VFS layer. Add these options to the kernel configuration file.". Do I have to compile new kernel with these options enabled or I can try with my GENERIC kernel? > > If I understand correctly because the system doesn't panic but keeps running I have to make it panic using "sysctl debug.kdb.panic=1" to create the core dump. Then after the system boots I run "kgdb /boot/kernel/kernel /var/crash/vmcore.0". Then I run the commands described in "10.6. Debugging Deadlocks". Am I right? No, the commands are for ddb.