Date: Sat, 14 Mar 2009 18:01:32 +0000 (GMT) From: Robert Watson <rwatson@FreeBSD.org> To: Nick Withers <nick@nickwithers.com> Cc: freebsd-stable@freebsd.org Subject: Re: NICs locking up, "*tcp_sc_h" Message-ID: <alpine.BSF.2.00.0903141756100.27597@fledge.watson.org> In-Reply-To: <1237020646.1532.24.camel@localhost> References: <1236920519.1490.30.camel@localhost> <alpine.BSF.2.00.0903130935290.61873@fledge.watson.org> <alpine.BSF.2.00.0903130949210.61873@fledge.watson.org> <1237020646.1532.24.camel@localhost>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 14 Mar 2009, Nick Withers wrote: > Right, here we go! ... Turns out that the problem is a lock cycle triggered by the syncache calling, indirectly, the firewall during output, and the firewall trying to look up the connection for the packet. Thread one: > Tracing PID 31 tid 100030 td 0xffffff00012016e0 > sched_switch() at sched_switch+0xdf > mi_switch() at mi_switch+0x18b > turnstile_wait() at turnstile_wait+0x1c4 > _mtx_lock_sleep() at _mtx_lock_sleep+0x76 > _mtx_lock_flags() at _mtx_lock_flags+0x95 > syncache_lookup() at syncache_lookup+0xee > syncache_expand() at syncache_expand+0x38 > tcp_input() at tcp_input+0x99b > ip_input() at ip_input+0xaf > ether_demux() at ether_demux+0x1b9 > ether_input() at ether_input+0x1bb > fxp_intr() at fxp_intr+0x224 > ithread_loop() at ithread_loop+0xe9 > fork_exit() at fork_exit+0x112 > fork_trampoline() at fork_trampoline+0xe > --- trap 0, rip = 0, rsp = 0xfffffffe80174d30, rbp = 0 --- This thread holds TCP locks and is trying to acquire the syncache lock. Thread two: > sched_switch() at sched_switch+0xdf > mi_switch() at mi_switch+0x18b > turnstile_wait() at turnstile_wait+0x1c4 > _rw_rlock() at _rw_rlock+0x9c > ipfw_chk() at ipfw_chk+0x3ac1 > ipfw_check_out() at ipfw_check_out+0xb1 > pfil_run_hooks() at pfil_run_hooks+0xac > ip_output() at ip_output+0x357 > syncache_respond() at syncache_respond+0x2fd > syncache_timer() at syncache_timer+0x15a > softclock() at softclock+0x270 > ithread_loop() at ithread_loop+0xe9 > fork_exit() at fork_exit+0x112 > fork_trampoline() at fork_trampoline+0xe This is the syncache timer holding syncache locks, calling IP output, and IPFW trying to acquire TCP locks. Am I right in thinking that you are using uid/gid/jail firewall rules? They suffer from a fundamental architectural problem in that they require reaching "up" to a higher level of the stack at times when it's not always a good idea to do so. In general we solve the problem by passing "down" the inpcb for a connection in the output path so that TCP doesn't have to look it up -- however, in the case of the syncache we actually don't have the inpcb easily in hand (or at least, we have it, but we can't just lock it because syncache locks are after TCP locks in the lock order...). It transpires that what the firewall really wants is not the inpcb, but the credential, but those are interfaces we can't change right now. I'll need to think a bit about a proper fix for this, but you'll find the problem likely goes away if you eliminate all uid/gid/jail rules from your firewall. You could also tweak the syncache logic not to use a retransmit timer, which might slightly extend the time it takes for systems to connect to your host in the presence of packet loss, but would eliminate this transmission path entirely. We'll need a real and more general fix, however, to commit, and I'll look and see what I can come up with. Robert N M Watson Computer Laboratory University of Cambridge
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.0903141756100.27597>