Date: Mon, 14 Sep 2009 14:29:19 -0700 From: "Matthew Fleming" <matthew.fleming@isilon.com> To: <freebsd-stable@freebsd.org> Subject: cxgb LOR Message-ID: <06D5F9F6F655AD4C92E28B662F7F853E030C6646@seaxch09.desktop.isilon.com>
next in thread | raw e-mail | index | archive | help
We got a cxgb LOR report of: 1st 0xffffff8001e37be0 vlan_global (vlan_global) @ /build/mnt/src/sys/modules/if_vlan/../../net/if_vlan.c:1310 2nd 0xffffff80010892f0 cxgb port lock (cxgb port lock) @ /build/mnt/src/sys/modules/cxgb/../../dev/cxgb/cxgb_main.c:1956 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a witness_checkorder() at witness_checkorder+0x9e2 _sx_xlock() at _sx_xlock+0x55 cxgb_ioctl() at cxgb_ioctl+0x1e8 vlan_ioctl() at vlan_ioctl+0x359 ifhwioctl() at ifhwioctl+0xb1 ifioctl() at ifioctl+0xb1 kern_ioctl() at kern_ioctl+0xa3 ioctl() at ioctl+0xf1 freebsd32_ioctl() at freebsd32_ioctl+0x13e isi_syscall() at isi_syscall+0x94 ia32_syscall() at ia32_syscall+0x1a3 Xint0x80_syscall() at Xint0x80_syscall+0x60 --- syscall (54, FreeBSD ELF32, freebsd32_ioctl), rip =3D 0x2868db1b, = rsp =3D 0xffffd4bc, rbp =3D 0xffffda38 --- So we tried changing cxgb to not USE_SX. This resulted in a different LOR: lock order reversal: (sleepable after non-sleepable) 1st 0xffffff8000f9d508 cxgb controller lock 0 (cxgb controller lock 0) @ /build/mnt/src/sys/modules/cxgb/cxgb/../../../dev/cxgb/cxgb_main.c:1889 2nd 0xffffffff806064e0 ACPI root bus (ACPI root bus) @ /build/mnt/src/sys/dev/acpica/acpi.c:1040 KDB: stack backtrace: [ffffffff8018e9fa] db_trace_self_wrapper+0x2a [ffffffff80298e89] witness_checkorder+0x719 [ffffffff8025bf75] _sx_xlock+0x55 [ffffffff8019678a] acpi_alloc_resource+0x9a [ffffffff80281714] resource_list_alloc+0x184 [ffffffff801d9f98] pci_alloc_resource+0x158 [ffffffff802814b9] bus_alloc_resource+0x89 [ffffffff81804201] cxgb_setup_interrupts+0x51 [ffffffff81807f33] cxgb_up+0xa3 [ffffffff818083c0] cxgb_init_locked+0x1b0 [ffffffff81808539] cxgb_init+0x39 [ffffffff81808758] cxgb_ioctl+0x1f8 [ffffffff8031e9e1] ifhwioctl+0xb1 [ffffffff8031f720] ifioctl+0xb0 [ffffffff8029a873] kern_ioctl+0xa3 [ffffffff8029aad1] ioctl+0xf1 [ffffffff8041eb93] freebsd32_ioctl+0xb3 [ffffffff8025d963] isi_syscall+0x83 [ffffffff8041de63] ia32_syscall+0x1a3 [ffffffff803efc60] Xint0x80_syscall+0x60 --- syscall (54, FreeBSD ELF32, freebsd32_ioctl), rip =3D 0x2826ea67, = rsp =3D 0xffffd8ac, rbp =3D 0xffffd928 --- (we modified cxgb_ioctl to call cxgb_init because otherwise the cxgb interface would require an ifconfig up before it detected a link, which was different behaviour from the em driver. Since the locks in question are acquired inside cxgb_init() I don't think the rest of the stack is relevant, but network stack isn't my area of expertise). So it seems that with cxgb we're damned if we do, damned if we don't. Any advice on which LOR is "worse" or if one is harmless, or how to make it go away? Note also that if cxgb uses a mtx then it will do malloc while holding the mtx in this stack: [ffffffff803cc58a] uma_zalloc_arg+0x2da [ffffffff80241ef9] malloc+0x89 [ffffffff8023115f] intr_event_add_handler+0x5f [ffffffff803f26d2] intr_add_handler+0x72 [ffffffff801dc171] pci_setup_intr+0x41 [ffffffff801dc171] pci_setup_intr+0x41 [ffffffff802807e6] bus_setup_intr+0x96 [ffffffff8180423c] cxgb_setup_interrupts+0x8c [ffffffff81807f33] cxgb_up+0xa3 [ffffffff818083c0] cxgb_init_locked+0x1b0 [ffffffff81808539] cxgb_init+0x39 Thanks, matthew
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?06D5F9F6F655AD4C92E28B662F7F853E030C6646>