From owner-freebsd-hackers@freebsd.org Wed Aug 10 16:58:31 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 511EABB5374 for ; Wed, 10 Aug 2016 16:58:31 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-io0-x22e.google.com (mail-io0-x22e.google.com [IPv6:2607:f8b0:4001:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1A02A1C8C for ; Wed, 10 Aug 2016 16:58:31 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-io0-x22e.google.com with SMTP id m101so47200175ioi.2 for ; Wed, 10 Aug 2016 09:58:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=GFBa4K8nsn5zwdQTAdMlZTzslbxJSXNDAmznDdfzhvw=; b=mi/k6eqrqUBkH4zSLv0gtQvh+zbxy0/s1ZmuD30IRB0n/xnw4hAjLbDUU2IjiFATnA yY3K0zm/jSaCbPMTZyQFjbWe+mNs/yGbwfK67q5Uyn+kMUH1QXlWJv7wMKIRaXmPl+WP U1GFk3A23ck6yXKjBRb92P9hH/FcIbZ5qapSQ/a92sli71BBKg1+lcc+cp4i2A27Gt6A tDKONG9DfRh/hbn+lsg6AMSx2rrFX2aqPFuA5W0vXZLwCLHzy8I8bKxgChI5wdH+uW0T PSW9lMX3BQ5R8QFPVlXDFXGIXmyZazpcSPe4zvtv22nDJRkUL4FwLkZyTaNojOWgY/C8 YgFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=GFBa4K8nsn5zwdQTAdMlZTzslbxJSXNDAmznDdfzhvw=; b=HMbwRrDtGyylIlZhfHjaxTpPz0cTrNNi4bMdlrTuNX7StxRD4Aa4nzI+L1ZajcXhYR Yq4oH/ks6tIEwgBMn1e+H2SSgXrR7vt/2t11admTn6KIFTnmPnU2Vb+WKQpgzp6omVUq T26U3a5+RzKfgaMquvh/d7j0U4c0Sbn4FQIHr4RKLCh0bWvLdUjz0YLtq4Cr1x3NuFFP 2U2jl91ztN67T6GV7AK8/RvD28dbHavpcGCuhlrebWqvzQCFq0lZjaBGrH4dhr3PsHOK tKird4hxdwQXOn7+hAI/CkcRQ7mZ1SdMh6wIhG5oV+7TnZDLGGi/9IjnuZDLkqeWgDHd X28Q== X-Gm-Message-State: AEkoouuk0HBub5k26ZRIk637xSqQCUyJLcye+xhnwfrk59jOr9xwzXxCz6aq+tYWoswJsMv3UEYgZXrwI6Mr0Q== X-Received: by 10.107.129.97 with SMTP id c94mr6568140iod.102.1470848310519; Wed, 10 Aug 2016 09:58:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.200.71 with HTTP; Wed, 10 Aug 2016 09:58:29 -0700 (PDT) In-Reply-To: <57AB462A.2080608@gmail.com> References: <57AB349B.2010805@gmail.com> <20160810141948.GP83214@kib.kiev.ua> <57AB462A.2080608@gmail.com> From: Ryan Stone Date: Wed, 10 Aug 2016 12:58:29 -0400 Message-ID: Subject: Re: 9.3-RELEASE panic: spin lock held too long To: Hooman Fazaeli Cc: Konstantin Belousov , FreeBSD Hackers Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2016 16:58:31 -0000 On Wed, Aug 10, 2016 at 11:20 AM, Hooman Fazaeli wrote: > > kgdb /boot/kernel/kernel /var/crash/vmcore.14 > ... > ... > (kgdb) bt > #0 doadump (textdump=1) at pcpu.h:250 > #1 0xc0ade835 in kern_reboot (howto=260) at ../../../kern/kern_shutdown.c: > 454 > #2 0xc0adeb32 in panic (fmt=) at > ../../../kern/kern_shutdown.c:642 > #3 0xc0ac9cff in _mtx_lock_spin_failed (m=0x0) at > ../../../kern/kern_mutex.c:515 > #4 0xc0ac9e75 in _mtx_lock_spin (m=0xc140a4c0, tid=3384060112, opts=0, > file=0x0, line=0) at ../../../kern/kern_mutex.c:557 > #5 0xc0b096c5 in sched_add (td=0xc9b00bc0, flags=0) at > ../../../kern/sched_ule.c:1153 > #6 0xc0b09890 in sched_wakeup (td=0xc9b00bc0) at > ../../../kern/sched_ule.c:1991 > #7 0xc0ae8968 in setrunnable (td=0xc9b00bc0) at > ../../../kern/kern_synch.c:537 > #8 0xc0b2227e in sleepq_resume_thread (sq=0xc869fd40, td=0xc9b00bc0, > pri=104) at ../../../kern/subr_sleepqueue.c:763 > #9 0xc0b22fd3 in sleepq_broadcast (wchan=0xc95741e4, flags=1, pri=104, > queue=0) at ../../../kern/subr_sleepqueue.c:865 > #10 0xc0a8c4cd in cv_broadcastpri (cvp=0xc95741e4, pri=104) at > ../../../kern/kern_condvar.c:448 > #11 0xc0b2a406 in doselwakeup (sip=0xc963faac, pri=104) at > ../../../kern/sys_generic.c:1683 > #12 0xc0b2a4be in selwakeuppri (sip=0xc963faac, pri=104) at > ../../../kern/sys_generic.c:1651 > #13 0xc0a9fa59 in knote_enqueue (kn=) at > ../../../kern/kern_event.c:1786 > #14 0xc0aa073f in kqueue_register (kq=0xc963fa80, kev=0xf0e07b20, > td=0xc9b4a8d0, waitok=1) at ../../../kern/kern_event.c:1154 > #15 0xc0aa09f3 in kern_kevent (td=0xc9b4a8d0, fd=152, nchanges=2, > nevents=0, k_ops=0xf0e07c20, timeout=0x0) at ../../../kern/kern_event.c:850 > #16 0xc0aa16ce in sys_kevent (td=0xc9b4a8d0, uap=0xf0e07ccc) at > ../../../kern/kern_event.c:771 > #17 0xc0fcc8c3 in syscall (frame=0xf0e07d08) at subr_syscall.c:135 > #18 0xc0fb60f1 in Xint0x80_syscall () at ../../../i386/i386/exception.s > :270 > #19 0x00000033 in ?? () > Previous frame inner to this frame (corrupt stack?) > > (kgdb) up 4 > #4 0xc0ac9e75 in _mtx_lock_spin (m=0xc140a4c0, tid=3384060112, opts=0, > file=0x0, line=0) at ../../../kern/kern_mutex.c:557 > 557 ../../../kern/kern_mutex.c: No such file or directory. > in ../../../kern/kern_mutex.c > > (kgdb) p *m > $1 = {lock_object = {lo_name = 0xc140ab08 "sched lock 0", lo_flags = > 720896, lo_data = 0, lo_witness = 0x0}, mtx_lock = 3355943664} > > ------------ > > As you see, the mtx_lock is 3355943664 (0xc807a2f0), the same TID reported > in panic string. > > (kgdb) info threads > ... > 34 Thread 100045 (PID=12: intr/irq267: igb0:que 0) sched_switch > (td=0xc807a2f0, newtd=0xc7da18d0, flags=265) at > ../../../kern/sched_ule.c:1904 > ... This sounds somewhat familiar. Is it always 'sched lock 0' that is ultimately leaked? Could you try applying this patch and seeing whether the new KASSERT triggers? https://people.freebsd.org/~rstone/patches/sched_balance_kassert.diff