From owner-freebsd-current@FreeBSD.ORG Fri Jul 13 20:46:24 2007 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4D92416A40A for ; Fri, 13 Jul 2007 20:46:24 +0000 (UTC) (envelope-from mattjreimer@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.242]) by mx1.freebsd.org (Postfix) with ESMTP id EF13213C4AC for ; Fri, 13 Jul 2007 20:46:23 +0000 (UTC) (envelope-from mattjreimer@gmail.com) Received: by an-out-0708.google.com with SMTP id c14so149969anc for ; Fri, 13 Jul 2007 13:46:23 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=WW3Wg50jZxIIUv8hCmsmQlTwKf35/GA9AgTb0vM7Xtg9pWeJkLaCf4F28meTdk/5DvUlb3Zs7v88h1ghxnDca3kW/beOUKBf6afHHiQSxbXMAhrAy7Jk8APuhxQvKGEBL2ZINIXrXBcHl22zejeNRjjF2VcaMV+bv22fn2Dk4sg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=iLfNp60afIPTBotoVxgkHjP1Lin4mg0hxo5YVhLjVVeodvAzakxqf6dl0Jxhs8kGxACTRhHIl9Jkjq0GWSZsMzsIO914toHtk1Pgt5xFBSnrHKhYmhEZJPpU45OQTC067t/eH2LWYodUxhPmB8qwUzwFFgpLe1jdpC4Fuk9kAEg= Received: by 10.100.166.14 with SMTP id o14mr1121869ane.1184359583421; Fri, 13 Jul 2007 13:46:23 -0700 (PDT) Received: by 10.100.142.3 with HTTP; Fri, 13 Jul 2007 13:46:23 -0700 (PDT) Message-ID: Date: Fri, 13 Jul 2007 13:46:23 -0700 From: "Matt Reimer" To: "Scott Long" In-Reply-To: <4697DE41.7090100@samsco.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <200707131528.51396.jhb@freebsd.org> <4697DE41.7090100@samsco.org> Cc: erich , freebsd-current@freebsd.org Subject: Re: arcmsr crash X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jul 2007 20:46:24 -0000 On 7/13/07, Scott Long wrote: > John Baldwin wrote: > > On Tuesday 05 June 2007 05:22:38 pm Matt Reimer wrote: > >> Once a week or so we're seeing a panic with a -current kernel built > >> just before the gcc 4.2 import (maybe three weeks ago). The box has a > >> Supermicro X7DBE/X7DBE+ motherboard with two Xeon 5160s, 16G RAM, and > >> an Areca 1220 controller with eight 500G disks connected. > >> > >> Does this indicate that the arcmsr driver is at fault: > >> > >> Tracing command irq16: arcmsr0 pid 26 tid 100018 td 0xffffff040fc5b000 > >> cpustop_handler() at cpustop_handler+0x35 > >> ipi_nmi_handler() at ipi_nmi_handler+0x2e > >> trap() at trap+0x365 > >> nmi_calltrap() at nmi_calltrap+0x8 > >> --- trap 0x13, rip = 0xffffffff8041ab11, rsp = 0xffffffffab59eff0, rbp > >> = 0xffffffffac0a37d0 --- > >> siocnclose() at siocnclose+0x21 > >> sio_cnputc() at sio_cnputc+0x89 > >> cnputc() at cnputc+0x6a > >> putchar() at putchar+0x5f > >> kvprintf() at kvprintf+0xd45 > >> printf() at printf+0xe1 > >> panic() at panic+0x145 > >> xpt_done() at xpt_done+0x14a > >> arcmsr_interrupt() at arcmsr_interrupt+0x2df > >> ithread_loop() at ithread_loop+0x108 > >> fork_exit() at fork_exit+0xaa > >> fork_trampoline() at fork_trampoline+0xe > >> --- trap 0, rip = 0, rsp = 0xffffffffac0a3d30, rbp = 0 --- > > > > Looks like it has panic'd here: > > > > switch (done_ccb->ccb_h.path->periph->type) { > > case CAM_PERIPH_BIO: > > mtx_lock(&cam_bioq_lock); > > TAILQ_INSERT_TAIL(&cam_bioq, &done_ccb->ccb_h, > > sim_links.tqe); > > done_ccb->ccb_h.pinfo.index = CAM_DONEQ_INDEX; > > mtx_unlock(&cam_bioq_lock); > > swi_sched(cambio_ih, 0); > > break; > > default: > > panic("unknown periph type %d", > > done_ccb->ccb_h.path->periph->type); > > } > > > > which should seem to indicate that, yes, it is a driver bug. > > > > The doneq has gotten corrupted somehow. The only real way that this > could happen is if xpt_done() was called twice on the same ccb. Whether > this is a hardware bug (hardware completing the same command twice) or > a driver bug is unknown. I'll try to add some seatbelts to CAM to > detect this kind of condition. But yes, it's ultimately something in > the arcmsr subsystem that is at fault. Do you have any suggestions of instrumentation printfs I could add to zero in on what part of the driver is at fault? Matt