From owner-freebsd-net@FreeBSD.ORG Fri Aug 17 21:06:54 2007 Return-Path: <owner-freebsd-net@FreeBSD.ORG> Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3352016A41A for <freebsd-net@freebsd.org>; Fri, 17 Aug 2007 21:06:54 +0000 (UTC) (envelope-from sullrich@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.189]) by mx1.freebsd.org (Postfix) with ESMTP id ED84213C48E for <freebsd-net@freebsd.org>; Fri, 17 Aug 2007 21:06:53 +0000 (UTC) (envelope-from sullrich@gmail.com) Received: by rv-out-0910.google.com with SMTP id l15so512605rvb for <freebsd-net@freebsd.org>; Fri, 17 Aug 2007 14:06:53 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=XlLphA1QQmZXJnKcpeVlfRzOSTC++n3Ms2QdjhQpfH8PEqs/Bkyy1ePR8o+WVSo25z5Uf7MgD50Ib7JNYe+5IHoSXwQVo/93a7umQ3qu8R9abg6Y/gn1iKmMnbsiEL7zKoXK7JHD3kZJWy6QtiVaa5HtITEY4s1kux8QUqIt9os= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=oRFO5T1hYvd9bjm6zkmeKu1LF0ymQfjzB05Iv4TJwoH1vzKQDQnYIupgK+Dfl4NbyBn/gFYBogM1VXo1rq0V4Fo6P7jF0amDukiGC8YbgiVs9Lk1k7XIkOCDWvUnDKBA1OH32sbWfSY1O5WtnVGiurBpUWkp7cm+s1onX8fW7Ys= Received: by 10.114.61.1 with SMTP id j1mr2113691waa.1187384405801; Fri, 17 Aug 2007 14:00:05 -0700 (PDT) Received: by 10.114.76.7 with HTTP; Fri, 17 Aug 2007 14:00:05 -0700 (PDT) Message-ID: <d5992baf0708171400ibdb0a3w2c352f1ec8cb1eeb@mail.gmail.com> Date: Fri, 17 Aug 2007 17:00:05 -0400 From: "Scott Ullrich" <sullrich@gmail.com> To: FreeBSD_Net <freebsd-net@freebsd.org> In-Reply-To: <d5992baf0708171353j7e7563a8y8dca4475779bc410@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <d5992baf0708171353j7e7563a8y8dca4475779bc410@mail.gmail.com> Subject: Re: Racoon(ipsec-tools) enters sbwait state or 100% CPU utilization quite often on RELENG_1_2 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net> List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Fri, 17 Aug 2007 21:06:54 -0000 On 8/17/07, Scott Ullrich <sullrich@gmail.com> wrote: > Hello! > > We are trying to track down a problem that involves a large number of > ipsec tunnels (in this case 80). Frequently racoon (ipsec-tools > 0.7rc1 and also 0.6) will deadlock into the sbwait state or will enter > a 100% cpu usage state and will not recover without killing the > process and restarting. > > # uname -a > FreeBSD pfsense.geekgod.com 6.2-RELEASE-p7 FreeBSD 6.2-RELEASE-p7 #0: > Sat Aug 4 18:35:24 EDT 2007 > sullrich@builder6.pfsense.com:/usr/obj.pfSense/usr/src/sys/pfSense.6 > i386 > > Kernel configuration file: > http://cvs.pfsense.com/cgi-bin/cvsweb.cgi/tools/builder_scripts/conf/pfSense.6?rev=1.65;content-type=text%2Fplain > > We where able to obtain a backtrace of what happens when the process > enters a 100% tail-spin: > > $ more /root/racoon.20070817.2112.txt > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-marcel-freebsd"... > Attaching to program: /usr/local/sbin/racoon, process 9144 > Reading symbols from /lib/libutil.so.5...done. > Loaded symbols for /lib/libutil.so.5 > Reading symbols from /lib/libcrypto.so.4...done. > Loaded symbols for /lib/libcrypto.so.4 > Reading symbols from /lib/libreadline.so.6...done. > Loaded symbols for /lib/libreadline.so.6 > Reading symbols from /lib/libc.so.6...done. > Loaded symbols for /lib/libc.so.6 > Reading symbols from /lib/libncurses.so.6...done. > Loaded symbols for /lib/libncurses.so.6 > Reading symbols from /libexec/ld-elf.so.1...done. > Loaded symbols for /libexec/ld-elf.so.1 > 0x2827a187 in recvfrom () from /lib/libc.so.6 > #0 0x2827a187 in recvfrom () from /lib/libc.so.6 > #1 0x28225904 in recv () from /lib/libc.so.6 > #2 0x0805f4f5 in pk_recv (so=11, lenp=0xbfbfe558) at pfkey.c:2826 > #3 0x0805f622 in pfkey_dump_sadb (satype=3) at pfkey.c:314 > #4 0x0805ac3d in purge_ipsec_spi (dst0=0x81b1080, proto=3, spi=0x8188140, n=1) > at isakmp_inf.c:1173 > #5 0x0805ba5c in isakmp_info_recv (iph1=0x81c1e00, msg0=0x1) > at isakmp_inf.c:565 > #6 0x0804ec49 in isakmp_main (msg=0x8218240, remote=0xbfbfe7f0, > local=0xbfbfe770) at isakmp.c:671 > #7 0x0805003e in isakmp_handler (so_isakmp=24) at isakmp.c:395 > #8 0x0804bf88 in session () at session.c:223 > #9 0x0804b901 in main (ac=0, av=0xbfbfee4c) at main.c:264 > #0 0x2827a187 in recvfrom () from /lib/libc.so.6 > #1 0x28225904 in recv () from /lib/libc.so.6 > #2 0x0805f4f5 in pk_recv (so=11, lenp=0xbfbfe558) at pfkey.c:2826 > #3 0x0805f622 in pfkey_dump_sadb (satype=3) at pfkey.c:314 > #4 0x0805ac3d in purge_ipsec_spi (dst0=0x81b1080, proto=3, spi=0x8188140, n=1) > at isakmp_inf.c:1173 > #5 0x0805ba5c in isakmp_info_recv (iph1=0x81c1e00, msg0=0x1) > at isakmp_inf.c:565 > #6 0x0804ec49 in isakmp_main (msg=0x8218240, remote=0xbfbfe7f0, > local=0xbfbfe770) at isakmp.c:671 > #7 0x0805003e in isakmp_handler (so_isakmp=24) at isakmp.c:395 > #8 0x0804bf88 in session () at session.c:223 > #9 0x0804b901 in main (ac=0, av=0xbfbfee4c) at main.c:264 > > Does anyone know what we can look at further to try and eliminate the > problem or does anyone have suggestions on how we can debug further? > Sorry, that title should have read RELENG_6_2. Freudian slip. Any help is appreciated. Scott