From owner-freebsd-bugs@FreeBSD.ORG Tue May 13 02:20:00 2014 Return-Path: Delivered-To: freebsd-bugs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 91FDBBEA for ; Tue, 13 May 2014 02:20:00 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6B4112E85 for ; Tue, 13 May 2014 02:20:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s4D2K08u044572 for ; Tue, 13 May 2014 02:20:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s4D2K0Wp044571; Tue, 13 May 2014 02:20:00 GMT (envelope-from gnats) Resent-Date: Tue, 13 May 2014 02:20:00 GMT Resent-Message-Id: <201405130220.s4D2K0Wp044571@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Colin Percival Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B5DE7A63 for ; Tue, 13 May 2014 02:11:12 +0000 (UTC) Received: from o3.shared.sendgrid.net (o3.shared.sendgrid.net [208.117.48.85]) by mx1.freebsd.org (Postfix) with SMTP id 5304E2E3A for ; Tue, 13 May 2014 02:11:11 +0000 (UTC) Received: by mf262.sendgrid.net with SMTP id mf262.38845.53717DEBA Tue, 13 May 2014 02:05:31 +0000 (UTC) Received: from mail.tarsnap.com (unknown [10.100.60.108]) by ismtpd-010 (SG) with ESMTP id 145f353de87.5129.17dfce for ; Tue, 13 May 2014 02:05:31 +0000 (GMT) Received: (qmail 5216 invoked from network); 13 May 2014 02:05:29 -0000 Received: from unknown (HELO clamshell.daemonology.net) (127.0.0.1) by ec2-107-20-205-189.compute-1.amazonaws.com with ESMTP; 13 May 2014 02:05:29 -0000 Received: (qmail 10782 invoked by uid 1001); 13 May 2014 02:03:22 -0000 Message-Id: <20140513020322.10781.qmail@clamshell.daemonology.net> Date: 13 May 2014 02:03:22 -0000 From: Colin Percival Reply-To: Colin Percival To: FreeBSD-gnats-submit@freebsd.org X-Send-Pr-Version: 3.114 Subject: kern/189745: ns8250 uart spin lock held too long X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2014 02:20:00 -0000 >Number: 189745 >Category: kern >Synopsis: ns8250 uart spin lock held too long >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue May 13 02:20:00 UTC 2014 >Closed-Date: >Last-Modified: >Originator: Colin Percival >Release: FreeBSD 10.0-RELEASE >Organization: >Environment: FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014 root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC 6 identical panics reported via panicmail; all afflicted systems were in EC2, i.e., with hw.broken_txfifo=1. >Description: panic: spin lock held too long #2 0xffffffff808af8f4 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:754 #3 0xffffffff8089cb71 in _mtx_lock_spin_cookie (c=, tid=, opts=, file=, line=) at /usr/src/sys/kern/kern_mutex.c:554 #4 0xffffffff80725223 in ns8250_bus_ipend (sc=0xfffff800083fc400) at uart_cpu.h:92 #5 0xffffffff80723fe7 in uart_intr (arg=0xfffff800083fc400) at uart_if.h:87 #6 0xffffffff80883e5b in intr_event_handle (ie=0xfffff8000819c200, frame=0xfffffe03a5ff8870) at /usr/src/sys/kern/kern_intr.c:1437 #7 0xffffffff80d8d1c8 in intr_execute_handlers (isrc=0xfffff800081bf168, frame=0xfffffe03a5ff8870) at /usr/src/sys/x86/x86/intr_machdep.c:269 In some panic reports there were _mtx_lock_spin_cookie -> printf -> vprintf -> kvprintf -> putchar -> cnputs -> cnputc -> uart_cnputc cycles as a result of the bug being re-trigerred by attempting to print the "spin lock ... held ... too long" warning. (Which may be something worth fixing separately, by detecting the loop and skipping straight to the panic call.) No console output is available (panicmail does not submit it) so I don't know which thread was holding the spin lock for so long; but it clearly unlocked eventually since the "spinning too long -> printf -> spinning too long" cycles were not infinite. >How-To-Repeat: No idea. This panic is strongly correlated with residing in EC2, but it may merely be caused by the broken_txfifo option since EC2 is where that option is most often used. >Fix: >Release-Note: >Audit-Trail: >Unformatted: