From owner-freebsd-net@FreeBSD.ORG Fri Aug 17 21:20:00 2007 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7CBF216A418 for ; Fri, 17 Aug 2007 21:20:00 +0000 (UTC) (envelope-from sullrich@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.179]) by mx1.freebsd.org (Postfix) with ESMTP id 500EB13C4A8 for ; Fri, 17 Aug 2007 21:20:00 +0000 (UTC) (envelope-from sullrich@gmail.com) Received: by wa-out-1112.google.com with SMTP id m33so333153wag for ; Fri, 17 Aug 2007 14:20:00 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=CAWrDHrz+ZdsGXgBRMthNYZdBWi3fGn6pMA5l4I3KNNBHz4pHzTwipFy0kxMB/m5bGc2LYyyUdLVNN4ALvNan0R23KKenz3LhacCTgnAkLP3EHCdhXtZd70LzAaypx2SwQ8UyXMtdGkddXxdHKzVPmbZ8jzqs40F6g0Ycid3csg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=DLzCKDkCFiHxE3/1ihZrFg+Qb6N2dxYjbBBcKlYtxpfKtvyLUFWg62uLCWURXKXGt4fRfu7qCioiu6S/V4IZaHwSrtZSEL1WvN8+Isa9mhf7sWO9GJesrl+j6KurskT16Af8oEA6H0ht+PYTYP983fFkR8WQpmWoxEc08qyD4lU= Received: by 10.115.108.1 with SMTP id k1mr209175wam.1187384036592; Fri, 17 Aug 2007 13:53:56 -0700 (PDT) Received: by 10.114.76.7 with HTTP; Fri, 17 Aug 2007 13:53:56 -0700 (PDT) Message-ID: Date: Fri, 17 Aug 2007 16:53:56 -0400 From: "Scott Ullrich" To: FreeBSD_Net MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: Racoon(ipsec-tools) enters sbwait state or 100% CPU utilization quite often on RELENG_1_2 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2007 21:20:00 -0000 Hello! We are trying to track down a problem that involves a large number of ipsec tunnels (in this case 80). Frequently racoon (ipsec-tools 0.7rc1 and also 0.6) will deadlock into the sbwait state or will enter a 100% cpu usage state and will not recover without killing the process and restarting. # uname -a FreeBSD pfsense.geekgod.com 6.2-RELEASE-p7 FreeBSD 6.2-RELEASE-p7 #0: Sat Aug 4 18:35:24 EDT 2007 sullrich@builder6.pfsense.com:/usr/obj.pfSense/usr/src/sys/pfSense.6 i386 Kernel configuration file: http://cvs.pfsense.com/cgi-bin/cvsweb.cgi/tools/builder_scripts/conf/pfSense.6?rev=1.65;content-type=text%2Fplain We where able to obtain a backtrace of what happens when the process enters a 100% tail-spin: $ more /root/racoon.20070817.2112.txt GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Attaching to program: /usr/local/sbin/racoon, process 9144 Reading symbols from /lib/libutil.so.5...done. Loaded symbols for /lib/libutil.so.5 Reading symbols from /lib/libcrypto.so.4...done. Loaded symbols for /lib/libcrypto.so.4 Reading symbols from /lib/libreadline.so.6...done. Loaded symbols for /lib/libreadline.so.6 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/libncurses.so.6...done. Loaded symbols for /lib/libncurses.so.6 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 0x2827a187 in recvfrom () from /lib/libc.so.6 #0 0x2827a187 in recvfrom () from /lib/libc.so.6 #1 0x28225904 in recv () from /lib/libc.so.6 #2 0x0805f4f5 in pk_recv (so=11, lenp=0xbfbfe558) at pfkey.c:2826 #3 0x0805f622 in pfkey_dump_sadb (satype=3) at pfkey.c:314 #4 0x0805ac3d in purge_ipsec_spi (dst0=0x81b1080, proto=3, spi=0x8188140, n=1) at isakmp_inf.c:1173 #5 0x0805ba5c in isakmp_info_recv (iph1=0x81c1e00, msg0=0x1) at isakmp_inf.c:565 #6 0x0804ec49 in isakmp_main (msg=0x8218240, remote=0xbfbfe7f0, local=0xbfbfe770) at isakmp.c:671 #7 0x0805003e in isakmp_handler (so_isakmp=24) at isakmp.c:395 #8 0x0804bf88 in session () at session.c:223 #9 0x0804b901 in main (ac=0, av=0xbfbfee4c) at main.c:264 #0 0x2827a187 in recvfrom () from /lib/libc.so.6 #1 0x28225904 in recv () from /lib/libc.so.6 #2 0x0805f4f5 in pk_recv (so=11, lenp=0xbfbfe558) at pfkey.c:2826 #3 0x0805f622 in pfkey_dump_sadb (satype=3) at pfkey.c:314 #4 0x0805ac3d in purge_ipsec_spi (dst0=0x81b1080, proto=3, spi=0x8188140, n=1) at isakmp_inf.c:1173 #5 0x0805ba5c in isakmp_info_recv (iph1=0x81c1e00, msg0=0x1) at isakmp_inf.c:565 #6 0x0804ec49 in isakmp_main (msg=0x8218240, remote=0xbfbfe7f0, local=0xbfbfe770) at isakmp.c:671 #7 0x0805003e in isakmp_handler (so_isakmp=24) at isakmp.c:395 #8 0x0804bf88 in session () at session.c:223 #9 0x0804b901 in main (ac=0, av=0xbfbfee4c) at main.c:264 Does anyone know what we can look at further to try and eliminate the problem or does anyone have suggestions on how we can debug further? Thanks Scott