From owner-freebsd-current@FreeBSD.ORG Mon Jul 25 21:44:05 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CB2231065672 for ; Mon, 25 Jul 2011 21:44:05 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 893658FC08 for ; Mon, 25 Jul 2011 21:44:05 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap4EALviLU6DaFvO/2dsb2JhbAA0AQEFKVwdDgoCAg0lAhZRBQKEbaN8uG6RFIErhAWBDwSScJApUw X-IronPort-AV: E=Sophos;i="4.67,264,1309752000"; d="scan'208";a="128527713" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 25 Jul 2011 17:44:04 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id BCC47B3F0C; Mon, 25 Jul 2011 17:44:04 -0400 (EDT) Date: Mon, 25 Jul 2011 17:44:04 -0400 (EDT) From: Rick Macklem To: Steve Kargl Message-ID: <254152988.989678.1311630244759.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <20110725162323.GA27425@troutmask.apl.washington.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-current@FreeBSD.org, Andriy Gapon Subject: Re: Heavy I/O blocks FreeBSD box for several seconds X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jul 2011 21:44:05 -0000 Steve Kargl wrote: > On Mon, Jul 25, 2011 at 01:00:27PM +0300, Andriy Gapon wrote: > > on 12/07/2011 11:05 Andriy Gapon said the following: > > > I think that the best thing you can further provide (as objective > > > evidence for > > > the problem at hand) is ktr(4) traces for at least KTR_SCHED mask. > > > Perhaps you > > > even already have them from your previous sessions with Jeff. > > > > > > P.S. This is not a promise to actually debug this issue based on > > > the traces :-) > > > > So do you have an opportunity to provide this kind of information? > > Actually I would like KTR_SCHED|KTR_INTR|KTR_PROC|KTR_SYSC mask. > > Also, sysctl kern.sched output would be useful too. > > This is for the ULE case, of course. > > > > I won't have time until next week to investigate. > hrs sent me this panic. I'm wondering if it might be relevant to this? spin lock 0xffffffff80cb52c0 (sched lock 1) held by 0xffffff0012c7f8c0 (tid 100317) too long panic: spin lock held too long cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 panic() at panic+0x187 _mtx_lock_spin_failed() at _mtx_lock_spin_failed+0x39 _mtx_lock_spin() at _mtx_lock_spin+0x9e sched_add() at sched_add+0x117 setrunnable() at setrunnable+0x78 sleepq_signal() at sleepq_signal+0x7a cv_signal() at cv_signal+0x3b xprt_active() at xprt_active+0xe3 svc_vc_soupcall() at svc_vc_soupcall+0xc sowakeup() at sowakeup+0x69 tcp_do_segment() at tcp_do_segment+0x2cbd tcp_input() at tcp_input+0xcdd ip_input() at ip_input+0xac netisr_dispatch_src() at netisr_dispatch_src+0x7e ether_demux() at ether_demux+0x14d ether_input() at ether_input+0x17d em_rxeof() at em_rxeof+0x1ca em_handle_que() at em_handle_que+0x5b taskqueue_run_locked() at taskqueue_run_locked+0x85 taskqueue_thread_loop() at taskqueue_thread_loop+0x4e fork_exit() at fork_exit+0x11f fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000160d00, rbp = 0 --- KDB: enter: panic [thread pid 0 tid 100033 ] Stopped at kdb_enter+0x3b: movq $0,0x6b4e62(%rip) db> ps