From owner-freebsd-net@FreeBSD.ORG Mon Nov 7 18:42:07 2011 Return-Path: Delivered-To: freebsd-net@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D621106566B; Mon, 7 Nov 2011 18:42:07 +0000 (UTC) (envelope-from bz@FreeBSD.ORG) Received: from mx1.sbone.de (bird.sbone.de [46.4.1.90]) by mx1.freebsd.org (Postfix) with ESMTP id C11798FC14; Mon, 7 Nov 2011 18:42:06 +0000 (UTC) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id B640525D3893; Mon, 7 Nov 2011 18:24:16 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id DA056BD465E; Mon, 7 Nov 2011 18:24:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id dcCsXAtP3PFX; Mon, 7 Nov 2011 18:24:14 +0000 (UTC) Received: from nv.sbone.de (nv.sbone.de [IPv6:fde9:577b:c1a9:31::2013:138]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 9C035BD4660; Mon, 7 Nov 2011 18:24:14 +0000 (UTC) Date: Mon, 7 Nov 2011 18:24:14 +0000 (UTC) From: "Bjoern A. Zeeb" To: Maxim Sobolev In-Reply-To: <4EB804D2.2090101@FreeBSD.org> Message-ID: References: <4EB804D2.2090101@FreeBSD.org> X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@FreeBSD.ORG, Robert Watson , "current@freebsd.org" Subject: Re: Panic in the udp_input() under heavy load X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-net@freebsd.org List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Nov 2011 18:42:07 -0000 On Mon, 7 Nov 2011, Maxim Sobolev wrote: > Hi Gang, > > We are seeing repeatable panics under high PPS load on our production > systems. It happens when the traffic gets into the range or 200MBps and > 150-200K PPS. We have been managed to track it down to the following piece of > code: > > (gdb) l *udp_input+0x5d2 > 0xffffffff806f6202 is in udp_input (/usr/src/sys/netinet/udp_usrreq.c:628). > 623 if (inp->inp_ip_minttl && inp->inp_ip_minttl > ip->ip_ttl) { > 624 INP_RUNLOCK(inp); > 625 goto badunlocked; > 626 } > 627 up = intoudpcb(inp); > 628 if (up->u_tun_func == NULL) { > 629 udp_append(inp, ip, m, iphlen + sizeof(struct > udphdr), &udp_in); > 630 } else { > 631 /* > 632 * Engage the tunneling protocol. > > The faulty line appears to be 628, with up value is being NULL, attempt to > deference it causes NULL pointer exception. I believe this particular piece > of code has been introduced here: Unlikely; the inp is properly locked there and the udp info attach better still be valid there; your problem is most likely elsewhere; try to see if you have other threads and see what they do at the same time, etc. You would need to race with udp_detach(); you also want to make sure that the inp still looks sane from either ddb or a dump and we are not talking about random memory corruption here. /bz -- Bjoern A. Zeeb You have to have visions! Stop bit received. Insert coin for new address family.