Date: Mon, 7 Nov 2011 18:24:14 +0000 (UTC) From: "Bjoern A. Zeeb" <bz@FreeBSD.ORG> To: Maxim Sobolev <sobomax@FreeBSD.org> Cc: freebsd-net@FreeBSD.ORG, Robert Watson <rwatson@FreeBSD.ORG>, "current@freebsd.org" <current@FreeBSD.ORG> Subject: Re: Panic in the udp_input() under heavy load Message-ID: <alpine.BSF.2.00.1111071818250.4603@ai.fobar.qr> In-Reply-To: <4EB804D2.2090101@FreeBSD.org> References: <4EB804D2.2090101@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 7 Nov 2011, Maxim Sobolev wrote:
> Hi Gang,
>
> We are seeing repeatable panics under high PPS load on our production
> systems. It happens when the traffic gets into the range or 200MBps and
> 150-200K PPS. We have been managed to track it down to the following piece of
> code:
>
> (gdb) l *udp_input+0x5d2
> 0xffffffff806f6202 is in udp_input (/usr/src/sys/netinet/udp_usrreq.c:628).
> 623 if (inp->inp_ip_minttl && inp->inp_ip_minttl > ip->ip_ttl) {
> 624 INP_RUNLOCK(inp);
> 625 goto badunlocked;
> 626 }
> 627 up = intoudpcb(inp);
> 628 if (up->u_tun_func == NULL) {
> 629 udp_append(inp, ip, m, iphlen + sizeof(struct
> udphdr), &udp_in);
> 630 } else {
> 631 /*
> 632 * Engage the tunneling protocol.
>
> The faulty line appears to be 628, with up value is being NULL, attempt to
> deference it causes NULL pointer exception. I believe this particular piece
> of code has been introduced here:
Unlikely; the inp is properly locked there and the udp info attach
better still be valid there; your problem is most likely elsewhere;
try to see if you have other threads and see what they do at the same
time, etc. You would need to race with udp_detach(); you also want
to make sure that the inp still looks sane from either ddb or a dump
and we are not talking about random memory corruption here.
/bz
--
Bjoern A. Zeeb You have to have visions!
Stop bit received. Insert coin for new address family.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.1111071818250.4603>