From owner-freebsd-stable@FreeBSD.ORG Mon Sep 14 22:08:52 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1773210656A6 for ; Mon, 14 Sep 2009 22:08:52 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-yw0-f179.google.com (mail-yw0-f179.google.com [209.85.211.179]) by mx1.freebsd.org (Postfix) with ESMTP id C6C8E8FC25 for ; Mon, 14 Sep 2009 22:08:51 +0000 (UTC) Received: by ywh9 with SMTP id 9so4870875ywh.32 for ; Mon, 14 Sep 2009 15:08:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=tvlUUMJElrlwdMp/1b5xcXaUAZD7aqoSAavO79JJmw0=; b=U42+AsPJf6rJVOyBdB84cZTgwa2sQb8zNqz8LE2YeHWXyvhT/t99Q7iQZEFniGErRr OF6b0YDVSNoCYCUqkB2f9zqJPxPK+Q4b7O5vaRyISorCPR1p0nWrLIlCCGdpupLW4fyN svKvSRjgTrKEb2xCD921V68EdEXZ3Ya+TYGw8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=Kp4QGlm0R08PwFrJUvmFBfqNoTooVXbwt4nIvRdXzMDdGEOsm0UMehpx7Cooi7r6S3 vRH/2dIdPaR4fJvxSLVtO4reN+yP1Xc+IpEkMf3oZmfQMdQ2ilvOlpU4WvPQc+rPyP1O X/OCUrAVqilqreVb8+xE1emoL2z4pH9rn6ibg= MIME-Version: 1.0 Received: by 10.150.210.17 with SMTP id i17mr10821767ybg.290.1252964831847; Mon, 14 Sep 2009 14:47:11 -0700 (PDT) In-Reply-To: <06D5F9F6F655AD4C92E28B662F7F853E030C6646@seaxch09.desktop.isilon.com> References: <06D5F9F6F655AD4C92E28B662F7F853E030C6646@seaxch09.desktop.isilon.com> Date: Mon, 14 Sep 2009 14:47:11 -0700 Message-ID: From: Navdeep Parhar To: Matthew Fleming Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: cxgb LOR X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Sep 2009 22:08:52 -0000 A lot of these LORs were fixed in cxgb in FreeBSD 8. You can look at cxgb_main.c in 8 for details. I'll also try and figure out if those changes are easily MFC'able. Regards, Navdeep On Mon, Sep 14, 2009 at 2:29 PM, Matthew Fleming wrote: > We got a cxgb LOR report of: > > 1st 0xffffff8001e37be0 vlan_global (vlan_global) @ > /build/mnt/src/sys/modules/if_vlan/../../net/if_vlan.c:1310 > =A02nd 0xffffff80010892f0 cxgb port lock (cxgb port lock) @ > /build/mnt/src/sys/modules/cxgb/../../dev/cxgb/cxgb_main.c:1956 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > witness_checkorder() at witness_checkorder+0x9e2 > _sx_xlock() at _sx_xlock+0x55 > cxgb_ioctl() at cxgb_ioctl+0x1e8 > vlan_ioctl() at vlan_ioctl+0x359 > ifhwioctl() at ifhwioctl+0xb1 > ifioctl() at ifioctl+0xb1 > kern_ioctl() at kern_ioctl+0xa3 > ioctl() at ioctl+0xf1 > freebsd32_ioctl() at freebsd32_ioctl+0x13e > isi_syscall() at isi_syscall+0x94 > ia32_syscall() at ia32_syscall+0x1a3 > Xint0x80_syscall() at Xint0x80_syscall+0x60 > --- syscall (54, FreeBSD ELF32, freebsd32_ioctl), rip =3D 0x2868db1b, rsp > =3D 0xffffd4bc, rbp =3D 0xffffda38 --- > > > So we tried changing cxgb to not USE_SX. =A0This resulted in a different > LOR: > > lock order reversal: (sleepable after non-sleepable) > =A01st 0xffffff8000f9d508 cxgb controller lock 0 (cxgb controller lock 0) > @ > /build/mnt/src/sys/modules/cxgb/cxgb/../../../dev/cxgb/cxgb_main.c:1889 > =A02nd 0xffffffff806064e0 ACPI root bus (ACPI root bus) @ > /build/mnt/src/sys/dev/acpica/acpi.c:1040 > KDB: stack backtrace: > [ffffffff8018e9fa] db_trace_self_wrapper+0x2a > [ffffffff80298e89] witness_checkorder+0x719 > [ffffffff8025bf75] _sx_xlock+0x55 > [ffffffff8019678a] acpi_alloc_resource+0x9a > [ffffffff80281714] resource_list_alloc+0x184 > [ffffffff801d9f98] pci_alloc_resource+0x158 > [ffffffff802814b9] bus_alloc_resource+0x89 > [ffffffff81804201] cxgb_setup_interrupts+0x51 > [ffffffff81807f33] cxgb_up+0xa3 > [ffffffff818083c0] cxgb_init_locked+0x1b0 > [ffffffff81808539] cxgb_init+0x39 > [ffffffff81808758] cxgb_ioctl+0x1f8 > [ffffffff8031e9e1] ifhwioctl+0xb1 > [ffffffff8031f720] ifioctl+0xb0 > [ffffffff8029a873] kern_ioctl+0xa3 > [ffffffff8029aad1] ioctl+0xf1 > [ffffffff8041eb93] freebsd32_ioctl+0xb3 > [ffffffff8025d963] isi_syscall+0x83 > [ffffffff8041de63] ia32_syscall+0x1a3 > [ffffffff803efc60] Xint0x80_syscall+0x60 > --- syscall (54, FreeBSD ELF32, freebsd32_ioctl), rip =3D 0x2826ea67, rsp > =3D 0xffffd8ac, rbp =3D 0xffffd928 --- > > (we modified cxgb_ioctl to call cxgb_init because otherwise the cxgb > interface would require an ifconfig up before it detected a link, which > was different behaviour from the em driver. =A0Since the locks in questio= n > are acquired inside cxgb_init() I don't think the rest of the stack is > relevant, but network stack isn't my area of expertise). > > So it seems that with cxgb we're damned if we do, damned if we don't. > Any advice on which LOR is "worse" or if one is harmless, or how to make > it go away? > > Note also that if cxgb uses a mtx then it will do malloc while holding > the mtx in this stack: > > [ffffffff803cc58a] uma_zalloc_arg+0x2da > [ffffffff80241ef9] malloc+0x89 > [ffffffff8023115f] intr_event_add_handler+0x5f > [ffffffff803f26d2] intr_add_handler+0x72 > [ffffffff801dc171] pci_setup_intr+0x41 > [ffffffff801dc171] pci_setup_intr+0x41 > [ffffffff802807e6] bus_setup_intr+0x96 > [ffffffff8180423c] cxgb_setup_interrupts+0x8c > [ffffffff81807f33] cxgb_up+0xa3 > [ffffffff818083c0] cxgb_init_locked+0x1b0 > [ffffffff81808539] cxgb_init+0x39 > > Thanks, > matthew > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >