From owner-svn-src-all@freebsd.org Tue Nov 28 20:37:58 2017 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 74889DEE248; Tue, 28 Nov 2017 20:37:58 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lf0-x231.google.com (mail-lf0-x231.google.com [IPv6:2a00:1450:4010:c07::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E801677677; Tue, 28 Nov 2017 20:37:57 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lf0-x231.google.com with SMTP id t197so1315407lfe.7; Tue, 28 Nov 2017 12:37:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=yiNZ9sLWfbEopR8ioOpk+t53mu0W1eh7DOf04syE36U=; b=CjJvsDXQQWH9ADZbB40JyMKK8PhpISan2qCBGGrLV4J6GTUv78pA+PVGfO8Xk0JLGk Rw12j3ROJ9/b2MX+FyGqxBqWqwFYEkLOJnHp4JpQHSVEZ0on+XrXIqNprFp75ZD/duAs plSaqn2t8xQ9rDwUmK64GB5oDAMQs+TbKKNU8R61iMnZAqcwVQVjhIHjUUdYLjOtVmP+ m0Q4DWZVuBx33XofiaApF9dK+z+HCsqyG7z6HX7RoevXe6aGf30detco7ylkWFlmdxpt 2rpf8ezF94qzs58RNbwgo44M1GMOBnHZcifobp3FqfmXOlN6n2Qf5NChOwFaBnh6Sftn X/qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=yiNZ9sLWfbEopR8ioOpk+t53mu0W1eh7DOf04syE36U=; b=g5cWKpUOETtjA21psblMp9j3mJvlEIOcDFfnqIzYzQnVT5Qa6L+AAVt3k/UHsDa4hY KpssjpfEZFuZIpazuSYO2dEp5+amnTAY7gxpMyDWa+M++rHnVtSWV1DQbPFXF1hD9a67 aBr1dTWElaPqPiKojWMkv5SroCkOK08FaR+8Mk4pcuPuWOidkhHxBImYk4BU+KikszLT xNBiAuyMn59NtgmKmkHNpXE88ztUQ7B0zYnoyU4Qn3mv6U/HJwpuBXILEfS/9mcbi12/ l6bQsdwfn6M+bVkrTb2G0Hou2rt5+kRc8sP+B88RhS8a2AijwY7l4y4AJvbakc98zsJi 8ZEg== X-Gm-Message-State: AJaThX6msVWkZbUXDP7ehrz+FJtOCKIN4/wBpSmPPLQBu5Ha4JQ3d0BU dc1E67UYFw7Bfb9ed6ZJfXuqA8FP+UiW+vhNh78= X-Google-Smtp-Source: AGs4zMb3UdOGejonY8d801+Ck+Iedu5yj4/re1ku2e/Q1wFeA+sIk+lQQbyqy2EBUTiTB9rQluUPODngey6GFYxgagM= X-Received: by 10.25.121.10 with SMTP id u10mr151336lfc.50.1511901475709; Tue, 28 Nov 2017 12:37:55 -0800 (PST) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 10.179.76.19 with HTTP; Tue, 28 Nov 2017 12:37:55 -0800 (PST) In-Reply-To: <20171127064940.GA91390@x2.osted.lan> References: <201711252049.vAPKnRaA026691@repo.freebsd.org> <20171126064927.GA68986@x2.osted.lan> <20171127064940.GA91390@x2.osted.lan> From: Alan Somers Date: Tue, 28 Nov 2017 13:37:55 -0700 X-Google-Sender-Auth: CqJpNXyk97sCv3OP99Cknl6B5v8 Message-ID: Subject: Re: svn commit: r326200 - head/sys/kern To: Peter Holm Cc: Mateusz Guzik , Mateusz Guzik , "src-committers@freebsd.org" , "svn-src-all@freebsd.org" , "svn-src-head@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 20:37:58 -0000 On Sun, Nov 26, 2017 at 11:49 PM, Peter Holm wrote: > On Sun, Nov 26, 2017 at 10:11:21PM +0100, Mateusz Guzik wrote: >> On Sun, Nov 26, 2017 at 7:49 AM, Peter Holm wrote: >> >> > On Sat, Nov 25, 2017 at 08:49:27PM +0000, Mateusz Guzik wrote: >> > > Author: mjg >> > > Date: Sat Nov 25 20:49:27 2017 >> > > New Revision: 326200 >> > > URL: https://svnweb.freebsd.org/changeset/base/326200 >> > > >> > > Log: >> > > Add the missing lockstat check for thread lock. >> > > >> > > Modified: >> > > head/sys/kern/kern_mutex.c >> > > >> > > Modified: head/sys/kern/kern_mutex.c >> > > ============================================================ >> > ================== >> > > --- head/sys/kern/kern_mutex.c Sat Nov 25 20:37:13 2017 >> > (r326199) >> > >> > Hi Mateusz, >> > >> > Could this be yours? >> > >> > spin lock 0xffffffff81ddc100 (turnstile chain) held by 0xfffff80014e57000 >> > (tid 100277) too long >> > panic: spin lock held too long >> > cpuid = 20 >> > time = 1511677687 >> > KDB: stack backtrace: >> > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> > 0xfffffe104f86ea40 >> > vpanic() at vpanic+0x268/frame 0xfffffe104f86eb10 >> > kproc_shutdown() at kproc_shutdown/frame 0xfffffe104f86eb70 >> > _mtx_lock_spin_failed() at _mtx_lock_spin_failed+0x87/frame >> > 0xfffffe104f86eba0 >> > _mtx_lock_spin_cookie() at _mtx_lock_spin_cookie+0x4ba/frame >> > 0xfffffe104f86ed10 >> > __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x21f/frame >> > 0xfffffe104f86edb0 >> > turnstile_trywait() at turnstile_trywait+0x32/frame 0xfffffe104f86edd0 >> > __rw_rlock_hard() at __rw_rlock_hard+0x118/frame 0xfffffe104f86ee60 >> > __rw_rlock_int() at __rw_rlock_int+0xd5/frame 0xfffffe104f86ee90 >> > getblk() at getblk+0xdb/frame 0xfffffe104f86ef40 >> > breadn_flags() at breadn_flags+0x6e/frame 0xfffffe104f86eff0 >> > ffs_getcg() at ffs_getcg+0x20d/frame 0xfffffe104f86f0d0 >> > ffs_nodealloccg() at ffs_nodealloccg+0xd0/frame 0xfffffe104f86f210 >> > ffs_hashalloc() at ffs_hashalloc+0xb7/frame 0xfffffe104f86f290 >> > ffs_valloc() at ffs_valloc+0x1c1/frame 0xfffffe104f86f390 >> > ufs_makeinode() at ufs_makeinode+0xba/frame 0xfffffe104f86f520 >> > ufs_create() at ufs_create+0x34/frame 0xfffffe104f86f540 >> > VOP_CREATE_APV() at VOP_CREATE_APV+0x241/frame 0xfffffe104f86f5c0 >> > VOP_CREATE() at VOP_CREATE+0x59/frame 0xfffffe104f86f620 >> > vn_open_cred() at vn_open_cred+0x311/frame 0xfffffe104f86f770 >> > vn_open() at vn_open+0x4c/frame 0xfffffe104f86f7b0 >> > kern_openat() at kern_openat+0x212/frame 0xfffffe104f86f920 >> > syscallenter() at syscallenter+0x4e4/frame 0xfffffe104f86f9f0 >> > amd64_syscall() at amd64_syscall+0x4d/frame 0xfffffe104f86fab0 >> > >> > https://people.freebsd.org/~pho/stress/log/mateusz004.txt >> > >> > - Peter >> > >> >> Fixed in r326237. >> > > Yes, that fixed the issue reported. > After some more tests I got this one: > > 20171127 01:19:57 all (31/561): mkfifo2c.sh > spin lock 0xffffffff81edace0 (smp rendezvous) held by 0xfffff80d2ae31000 (tid 100529) too long > panic: spin lock held too long > cpuid = 14 > time = 1511742163 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe105096c0b0 > vpanic() at vpanic+0x19c/frame 0xfffffe105096c130 > panic() at panic+0x43/frame 0xfffffe105096c190 > _mtx_lock_spin_cookie() at _mtx_lock_spin_cookie+0x352/frame 0xfffffe105096c200 > __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0xd8/frame 0xfffffe105096c240 > smp_targeted_tlb_shootdown() at smp_targeted_tlb_shootdown+0xd7/frame 0xfffffe105096c2c0 > smp_masked_invlpg_range() at smp_masked_invlpg_range+0x50/frame 0xfffffe105096c2f0 > pmap_invalidate_range() at pmap_invalidate_range+0x1f4/frame 0xfffffe105096c340 > allocbuf() at allocbuf+0x107/frame 0xfffffe105096c3b0 > brelse() at brelse+0x219/frame 0xfffffe105096c400 > trunc_dependencies() at trunc_dependencies+0x745/frame 0xfffffe105096c4a0 > softdep_setup_freeblocks() at softdep_setup_freeblocks+0x7b4/frame 0xfffffe105096c510 > ffs_truncate() at ffs_truncate+0x6b8/frame 0xfffffe105096c700 > ufs_inactive() at ufs_inactive+0x1cb/frame 0xfffffe105096c740 > VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xfd/frame 0xfffffe105096c770 > vinactive() at vinactive+0xe3/frame 0xfffffe105096c7c0 > vputx() at vputx+0x31e/frame 0xfffffe105096c820 > kern_rmdirat() at kern_rmdirat+0x1c9/frame 0xfffffe105096c980 > amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe105096cab0 > > Not so much more info here, I'm afraid: > https://people.freebsd.org/~pho/stress/log/mateusz005.txt > > -- > Peter I'm seeing a similar panic, at r326314: panic: spin lock held too long cpuid = 3 time = 1511900768 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0b58bcef00 vpanic() at vpanic+0x19c/frame 0xfffffe0b58bcef80 panic() at panic+0x43/frame 0xfffffe0b58bcefe0 _mtx_lock_spin_cookie() at _mtx_lock_spin_cookie+0x352/frame 0xfffffe0b58bcf050 __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0xd8/frame 0xfffffe0b58bcf090 smp_targeted_tlb_shootdown() at smp_targeted_tlb_shootdown+0xd7/frame 0xfffffe0b58bcf110 smp_masked_invlpg_range() at smp_masked_invlpg_range+0x50/frame 0xfffffe0b58bcf140 pmap_invalidate_range() at pmap_invalidate_range+0x1f4/frame 0xfffffe0b58bcf190 vm_thread_stack_dispose() at vm_thread_stack_dispose+0x2f/frame 0xfffffe0b58bcf1e0 thread_free() at thread_free+0x39/frame 0xfffffe0b58bcf200 thread_reap() at thread_reap+0x9e/frame 0xfffffe0b58bcf220 thread_alloc() at thread_alloc+0xf/frame 0xfffffe0b58bcf240 kthread_add() at kthread_add+0x43/frame 0xfffffe0b58bcf2e0 _taskqueue_start_threads() at _taskqueue_start_threads+0xde/frame 0xfffffe0b58bcf370 taskqueue_start_threads() at taskqueue_start_threads+0x3f/frame 0xfffffe0b58bcf3d0 taskq_create_proc() at taskq_create_proc+0x92/frame 0xfffffe0b58bcf400 spa_activate() at spa_activate+0x372/frame 0xfffffe0b58bcf4a0 spa_open_common() at spa_open_common+0xd2/frame 0xfffffe0b58bcf510 spa_get_stats() at spa_get_stats+0x4f/frame 0xfffffe0b58bcf6f0 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x25/frame 0xfffffe0b58bcf730 zfsdev_ioctl() at zfsdev_ioctl+0x766/frame 0xfffffe0b58bcf7e0 devfs_ioctl() at devfs_ioctl+0xcb/frame 0xfffffe0b58bcf830 VOP_IOCTL_APV() at VOP_IOCTL_APV+0xd9/frame 0xfffffe0b58bcf860 vn_ioctl() at vn_ioctl+0x124/frame 0xfffffe0b58bcf970 devfs_ioctl_f() at devfs_ioctl_f+0x1f/frame 0xfffffe0b58bcf990 kern_ioctl() at kern_ioctl+0x2c4/frame 0xfffffe0b58bcf9f0 sys_ioctl() at sys_ioctl+0x15c/frame 0xfffffe0b58bcfac0 amd64_syscall() at amd64_syscall+0x79b/frame 0xfffffe0b58bcfbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0b58bcfbf0