From owner-freebsd-stable@FreeBSD.ORG Mon Sep 14 21:28:56 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34B58106566B for ; Mon, 14 Sep 2009 21:28:56 +0000 (UTC) (envelope-from matthew.fleming@isilon.com) Received: from seaxch09.isilon.com (seaxch09.isilon.com [74.85.160.25]) by mx1.freebsd.org (Postfix) with ESMTP id 1E88A8FC08 for ; Mon, 14 Sep 2009 21:28:55 +0000 (UTC) x-mimeole: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Date: Mon, 14 Sep 2009 14:29:19 -0700 Message-ID: <06D5F9F6F655AD4C92E28B662F7F853E030C6646@seaxch09.desktop.isilon.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: cxgb LOR Thread-Index: Aco1fx5vwbpBDFrnTtyydf7rwzkkkwAAp1gw From: "Matthew Fleming" To: Subject: cxgb LOR X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Sep 2009 21:28:56 -0000 We got a cxgb LOR report of: 1st 0xffffff8001e37be0 vlan_global (vlan_global) @ /build/mnt/src/sys/modules/if_vlan/../../net/if_vlan.c:1310 2nd 0xffffff80010892f0 cxgb port lock (cxgb port lock) @ /build/mnt/src/sys/modules/cxgb/../../dev/cxgb/cxgb_main.c:1956 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a witness_checkorder() at witness_checkorder+0x9e2 _sx_xlock() at _sx_xlock+0x55 cxgb_ioctl() at cxgb_ioctl+0x1e8 vlan_ioctl() at vlan_ioctl+0x359 ifhwioctl() at ifhwioctl+0xb1 ifioctl() at ifioctl+0xb1 kern_ioctl() at kern_ioctl+0xa3 ioctl() at ioctl+0xf1 freebsd32_ioctl() at freebsd32_ioctl+0x13e isi_syscall() at isi_syscall+0x94 ia32_syscall() at ia32_syscall+0x1a3 Xint0x80_syscall() at Xint0x80_syscall+0x60 --- syscall (54, FreeBSD ELF32, freebsd32_ioctl), rip =3D 0x2868db1b, = rsp =3D 0xffffd4bc, rbp =3D 0xffffda38 --- So we tried changing cxgb to not USE_SX. This resulted in a different LOR: lock order reversal: (sleepable after non-sleepable) 1st 0xffffff8000f9d508 cxgb controller lock 0 (cxgb controller lock 0) @ /build/mnt/src/sys/modules/cxgb/cxgb/../../../dev/cxgb/cxgb_main.c:1889 2nd 0xffffffff806064e0 ACPI root bus (ACPI root bus) @ /build/mnt/src/sys/dev/acpica/acpi.c:1040 KDB: stack backtrace: [ffffffff8018e9fa] db_trace_self_wrapper+0x2a [ffffffff80298e89] witness_checkorder+0x719 [ffffffff8025bf75] _sx_xlock+0x55 [ffffffff8019678a] acpi_alloc_resource+0x9a [ffffffff80281714] resource_list_alloc+0x184 [ffffffff801d9f98] pci_alloc_resource+0x158 [ffffffff802814b9] bus_alloc_resource+0x89 [ffffffff81804201] cxgb_setup_interrupts+0x51 [ffffffff81807f33] cxgb_up+0xa3 [ffffffff818083c0] cxgb_init_locked+0x1b0 [ffffffff81808539] cxgb_init+0x39 [ffffffff81808758] cxgb_ioctl+0x1f8 [ffffffff8031e9e1] ifhwioctl+0xb1 [ffffffff8031f720] ifioctl+0xb0 [ffffffff8029a873] kern_ioctl+0xa3 [ffffffff8029aad1] ioctl+0xf1 [ffffffff8041eb93] freebsd32_ioctl+0xb3 [ffffffff8025d963] isi_syscall+0x83 [ffffffff8041de63] ia32_syscall+0x1a3 [ffffffff803efc60] Xint0x80_syscall+0x60 --- syscall (54, FreeBSD ELF32, freebsd32_ioctl), rip =3D 0x2826ea67, = rsp =3D 0xffffd8ac, rbp =3D 0xffffd928 --- (we modified cxgb_ioctl to call cxgb_init because otherwise the cxgb interface would require an ifconfig up before it detected a link, which was different behaviour from the em driver. Since the locks in question are acquired inside cxgb_init() I don't think the rest of the stack is relevant, but network stack isn't my area of expertise). So it seems that with cxgb we're damned if we do, damned if we don't. Any advice on which LOR is "worse" or if one is harmless, or how to make it go away? Note also that if cxgb uses a mtx then it will do malloc while holding the mtx in this stack: [ffffffff803cc58a] uma_zalloc_arg+0x2da [ffffffff80241ef9] malloc+0x89 [ffffffff8023115f] intr_event_add_handler+0x5f [ffffffff803f26d2] intr_add_handler+0x72 [ffffffff801dc171] pci_setup_intr+0x41 [ffffffff801dc171] pci_setup_intr+0x41 [ffffffff802807e6] bus_setup_intr+0x96 [ffffffff8180423c] cxgb_setup_interrupts+0x8c [ffffffff81807f33] cxgb_up+0xa3 [ffffffff818083c0] cxgb_init_locked+0x1b0 [ffffffff81808539] cxgb_init+0x39 Thanks, matthew