From owner-freebsd-hackers@freebsd.org Wed Aug 10 17:24:02 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6D4CBB5DC2 for ; Wed, 10 Aug 2016 17:24:02 +0000 (UTC) (envelope-from hoomanfazaeli@gmail.com) Received: from mail-wm0-x22a.google.com (mail-wm0-x22a.google.com [IPv6:2a00:1450:400c:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 25D841EAD for ; Wed, 10 Aug 2016 17:24:02 +0000 (UTC) (envelope-from hoomanfazaeli@gmail.com) Received: by mail-wm0-x22a.google.com with SMTP id q128so107051459wma.1 for ; Wed, 10 Aug 2016 10:24:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to; bh=w9qA69b5PmjssCKhFLpjS1HvCcGMr99t+jHQfU6iVps=; b=0SXx4a/RWAswN8A9ozLjSFhcTImfZfcbq7WhfKlwlUFMQrgVZG5T+ua180cLfwTS50 W5o50nrOz2msGnTnfMGXE67ZbkHvxL/OS/8dexUc5aPT7pUBX0rDlJqwYoEMDqbrQ+TG 4NFvVH+Ts8wDyDPVH0LU808W0Ukp1WfHqm3f07FKiNqLAnvRjU5IXsUfsxSBUitfMurB xPdhzCJilHw33kTKqiiNDmFrGMzjgQELwdvzIEVa21KxejeSiXSnLcAUTUE0+LOzx+hk dVZ548AfGb7bohp8ZawdrOoUSKbgGmtEAh2hE9EJEdZJtnRPovLVQaJnUJHQFd/4+reZ gaUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to; bh=w9qA69b5PmjssCKhFLpjS1HvCcGMr99t+jHQfU6iVps=; b=Jj2MqbUyPmDMH167hHElsRT72ptHEejxmiI6UT1SJKxn3c6e12j+iIDH4XQj08kd04 hNfWc77mIx43qOjwrSHEFShqdvkMrc4YUKhgtyIskmBYF7SdZ9sJjtu9cRvt57maCxNI 7jh73p4U8vcEGWAiTJPp11ps6nF7k7LkOE28kYKlGgbLW4kFgSO3SLXnfDOqm1x+zybk /OxReX0EdbzDb1nsCkTSrUpRqQUgx3DrrEMdxNPA7JohNa9sWMOITxjpn/W0H9x5ugcw N7zdAHRgR6047fJzBV11IDPy+qAWZcjFuh/gGXg9eEXhvSIymMGJYsRUFvMx3mo9S+tj G4XQ== X-Gm-Message-State: AEkoousuFWE7aOs0nx5wRLJAEPgVcYiqT4TA1GSwNFkGFBz6afsIrcDS8GWSwbISmNKvww== X-Received: by 10.194.104.106 with SMTP id gd10mr5871783wjb.55.1470849840690; Wed, 10 Aug 2016 10:24:00 -0700 (PDT) Received: from [192.168.2.30] ([2.190.216.101]) by smtp.googlemail.com with ESMTPSA id m81sm9362872wmf.1.2016.08.10.10.23.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Aug 2016 10:24:00 -0700 (PDT) Message-ID: <57AB632D.4000501@gmail.com> Date: Wed, 10 Aug 2016 21:53:57 +0430 From: Hooman Fazaeli User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130215 Thunderbird/17.0.3 MIME-Version: 1.0 To: Ryan Stone CC: Konstantin Belousov , FreeBSD Hackers Subject: Re: 9.3-RELEASE panic: spin lock held too long References: <57AB349B.2010805@gmail.com> <20160810141948.GP83214@kib.kiev.ua> <57AB462A.2080608@gmail.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2016 17:24:02 -0000 On 2016-08-10 21:28, Ryan Stone wrote: > On Wed, Aug 10, 2016 at 11:20 AM, Hooman Fazaeli > wrote: > > > kgdb /boot/kernel/kernel /var/crash/vmcore.14 > ... > ... > (kgdb) bt > #0 doadump (textdump=1) at pcpu.h:250 > #1 0xc0ade835 in kern_reboot (howto=260) at ../../../kern/kern_shutdown.c:454 > #2 0xc0adeb32 in panic (fmt=) at ../../../kern/kern_shutdown.c:642 > #3 0xc0ac9cff in _mtx_lock_spin_failed (m=0x0) at ../../../kern/kern_mutex.c:515 > #4 0xc0ac9e75 in _mtx_lock_spin (m=0xc140a4c0, tid=3384060112, opts=0, file=0x0, line=0) at ../../../kern/kern_mutex.c:557 > #5 0xc0b096c5 in sched_add (td=0xc9b00bc0, flags=0) at ../../../kern/sched_ule.c:1153 > #6 0xc0b09890 in sched_wakeup (td=0xc9b00bc0) at ../../../kern/sched_ule.c:1991 > #7 0xc0ae8968 in setrunnable (td=0xc9b00bc0) at ../../../kern/kern_synch.c:537 > #8 0xc0b2227e in sleepq_resume_thread (sq=0xc869fd40, td=0xc9b00bc0, pri=104) at ../../../kern/subr_sleepqueue.c:763 > #9 0xc0b22fd3 in sleepq_broadcast (wchan=0xc95741e4, flags=1, pri=104, queue=0) at ../../../kern/subr_sleepqueue.c:865 > #10 0xc0a8c4cd in cv_broadcastpri (cvp=0xc95741e4, pri=104) at ../../../kern/kern_condvar.c:448 > #11 0xc0b2a406 in doselwakeup (sip=0xc963faac, pri=104) at ../../../kern/sys_generic.c:1683 > #12 0xc0b2a4be in selwakeuppri (sip=0xc963faac, pri=104) at ../../../kern/sys_generic.c:1651 > #13 0xc0a9fa59 in knote_enqueue (kn=) at ../../../kern/kern_event.c:1786 > #14 0xc0aa073f in kqueue_register (kq=0xc963fa80, kev=0xf0e07b20, td=0xc9b4a8d0, waitok=1) at ../../../kern/kern_event.c:1154 > #15 0xc0aa09f3 in kern_kevent (td=0xc9b4a8d0, fd=152, nchanges=2, nevents=0, k_ops=0xf0e07c20, timeout=0x0) at ../../../kern/kern_event.c:850 > #16 0xc0aa16ce in sys_kevent (td=0xc9b4a8d0, uap=0xf0e07ccc) at ../../../kern/kern_event.c:771 > #17 0xc0fcc8c3 in syscall (frame=0xf0e07d08) at subr_syscall.c:135 > #18 0xc0fb60f1 in Xint0x80_syscall () at ../../../i386/i386/exception.s:270 > #19 0x00000033 in ?? () > Previous frame inner to this frame (corrupt stack?) > > (kgdb) up 4 > #4 0xc0ac9e75 in _mtx_lock_spin (m=0xc140a4c0, tid=3384060112, opts=0, file=0x0, line=0) at ../../../kern/kern_mutex.c:557 > 557 ../../../kern/kern_mutex.c: No such file or directory. > in ../../../kern/kern_mutex.c > > (kgdb) p *m > $1 = {lock_object = {lo_name = 0xc140ab08 "sched lock 0", lo_flags = 720896, lo_data = 0, lo_witness = 0x0}, mtx_lock = 3355943664} > > ------------ > > As you see, the mtx_lock is 3355943664 (0xc807a2f0), the same TID reported in panic string. > > (kgdb) info threads > ... > 34 Thread 100045 (PID=12: intr/irq267: igb0:que 0) sched_switch (td=0xc807a2f0, newtd=0xc7da18d0, flags=265) at ../../../kern/sched_ule.c:1904 > ... > > > This sounds somewhat familiar. Is it always 'sched lock 0' that is ultimately leaked? Could you try applying this patch and seeing whether the new KASSERT triggers? > > https://people.freebsd.org/~rstone/patches/sched_balance_kassert.diff > No. I have panics involving 'turnstile lock' (see the original post) and 'sched lock 2' too. -- Best regards Hooman Fazaeli