From owner-freebsd-current@FreeBSD.ORG Sat Aug 27 17:20:26 2005 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EF8B416A41F for ; Sat, 27 Aug 2005 17:20:25 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 97ABC43D49 for ; Sat, 27 Aug 2005 17:20:25 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id E11CA46B0C; Sat, 27 Aug 2005 13:20:24 -0400 (EDT) Date: Sat, 27 Aug 2005 18:20:24 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: "M. Warner Losh" In-Reply-To: <20050827.104631.10908351.imp@bsdimp.com> Message-ID: <20050827181827.O24510@fledge.watson.org> References: <20050826115503.D4B0C4E704@pipa.profix.cz> <20050827.104631.10908351.imp@bsdimp.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: bzeeb-lists@lists.zabbadoz.net, freebsd-current@freebsd.org, dandee@volny.cz Subject: Re: LOR route vr0 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Aug 2005 17:20:26 -0000 On Sat, 27 Aug 2005, M. Warner Losh wrote: > In message: > "Bjoern A. Zeeb" writes: > : > lock order reversal > : > 1st 0xc17621ec rtentry (rtentry) @ /usr/src/sys/net/route.c:1269 > : > 2nd 0xc15ec938 vr0 (network driver) @ /usr/src/sys/pci/if_vr.c:1391 > : > : added with ID 140: http://sources.zabbadoz.net/freebsd/lor.html#140 > > I've noticed a *HUGE* number of LORs that look like this: > > ock order reversal > 1st 0xc17490e4 rtentry (rtentry) @ sys/netinet/if_ether.c:445 > 2nd 0xc15c94b0 rl1 (network driver) @ sys/pci/if_rl.c:1451 Generally speaking, network interface device driver locks follow network stack locks in the lock order. However, I've not really looked much at the route table locking so can't speak to whether that is the case specifically for routing locks. If it is, the below traces reflect the correct order, and you might want to add a hard-coded entry to witness in order to catch the reverse order. Lock order reversals between the network stack and device drivers tend to occur as a result of the device driver calling into the network stack while holding the device driver mutex. Someone (tm) should work out if the right order is route locks -> device driver locks, as it's likely a common calss of bugs across many drivers. Robert N M Watson > KDB: stack backtrace: > kdb_backtrace(c07dcab1,c15c94b0,c160a1b0,c07c54d7,c07f401e) at kdb_backtrace+0x2e > witness_checkorder(c15c94b0,9,c07f401e,5ab,c07e32bd) at witness_checkorder+0x6c3 > _mtx_lock_flags(c15c94b0,0,c07f401e,5ab,c152cc00) at _mtx_lock_flags+0x8a > rl_start(c152cc00,1,c07e2eae,836) at rl_start+0x37 > if_start(c152cc00,0,c07e32bd,195,202) at if_start+0x99 > ether_output_frame(c152cc00,c169c100,6,c0562c28,c169c100) > at ether_output_frame+0x218 > ether_output(c152cc00,c169c100,cbfe79f0,0,2,c1740001,2302,c07e66e8,1bd,519) > at ether_output+0x47e > arprequest(c152cc00,c16cfcc8,cbfe7ae4,c15fa6ab,c05998a6) at arprequest+0x109 > arpresolve(c152cc00,c1749084,c169a400,cbfe7ae0,cbfe7a64) at arpresolve+0x32d > ether_output(c152cc00,c169a400,cbfe7ae0,c1749084,0) at ether_output+0x7b > ip_output(c169a400,0,cbfe7adc,0,0) at ip_output+0xb7a > > and am seeing the following in my newly locked ed driver: > > lock order reversal > 1st 0xc1cb3588 rtentry (rtentry) @ net/route.c:1269 > 2nd 0xc1fd3420 ed1 (network driver) @ /dell/imp/p4/newcard/src/sys/modules/ed/../../dev/ed/if_ed.c:697 > KDB: stack backtrace: > kdb_backtrace(0,ffffffff,c0680950,c067f5a0,c064bd44) at kdb_backtrace+0x29 > witness_checkorder(c1fd3420,9,c201ff8b,2b9) at witness_checkorder+0x52c > _mtx_lock_flags(c1fd3420,0,c201ff8b,2b9,c1a86c00) at _mtx_lock_flags+0x5b > ed_start(c1a86c00) at ed_start+0x1f > if_start(c1a86c00) at if_start+0x7b > ether_output_frame(c1a86c00,c1bbeb00,c04c0920,ffffffff,0) at ether_output_frame+0x1dc > ether_output(c1a86c00,c1bbeb00,e5832a38,0,2) at ether_output+0x3e4 > arprequest(c1a86c00,c1d77ac8,e5832b08,c20236ab) at arprequest+0xd8 > arpresolve(c1a86c00,c1cb3528,c1bbed00,e5832b04,e5832aa8) at arpresolve+0x30b > ether_output(c1a86c00,c1bbed00,e5832b04,c1cb3528,c1d77a00) at ether_output+0x6b > ip_output(c1bbed00,0,e5832b00,0,0) at ip_output+0x78c > udp_output(c1cb1168,c1bbed00,0,0,c1a8d600) at udp_output+0x4a7 > udp_send(c1d59c84,0,c1bbed00,0,0) at udp_send+0x1a > sosend(c1d59c84,0,e5832c3c,c1bbed00,0) at sosend+0x5e3 > kern_sendit(c1a8d600,4,e5832cbc,0,0) at kern_sendit+0x104 > sendit(c1a8d600,4,e5832cbc,0,807a023) at sendit+0x163 > sendto(c1a8d600,e5832d04,6,0,206) at sendto+0x4d > syscall(3b,3b,3b,805a000,28219fa4) at syscall+0x22f > > and was wondering if there's a common cause. > > Warner > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >