From owner-freebsd-net@FreeBSD.ORG Thu Oct 4 14:07:07 2007 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E1B7B16A419 for ; Thu, 4 Oct 2007 14:07:07 +0000 (UTC) (envelope-from seth.mos@xs4all.nl) Received: from smtp-vbr5.xs4all.nl (smtp-vbr5.xs4all.nl [194.109.24.25]) by mx1.freebsd.org (Postfix) with ESMTP id 6B58A13C458 for ; Thu, 4 Oct 2007 14:07:07 +0000 (UTC) (envelope-from seth.mos@xs4all.nl) Received: from [127.0.0.1] ([194.151.164.164]) (authenticated bits=0) by smtp-vbr5.xs4all.nl (8.13.8/8.13.8) with ESMTP id l94E75Th044782 for ; Thu, 4 Oct 2007 16:07:06 +0200 (CEST) (envelope-from seth.mos@xs4all.nl) Message-ID: <4704F388.5080506@xs4all.nl> Date: Thu, 04 Oct 2007 16:07:04 +0200 From: Seth Mos User-Agent: Thunderbird 2.0.0.6 (Windows/20070728) MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <46FBB7BC.3010301@xs4all.nl> In-Reply-To: <46FBB7BC.3010301@xs4all.nl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by XS4ALL Virus Scanner Subject: Re: Racoon 0.7 on FreeBSD 6 with a lot of VPN tunnels X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Oct 2007 14:07:08 -0000 Hello Yvan, I have tested the suggested workaorund in pfkey.c by raising the ceiling to 768kbytes from 128. I also raised the limit in the socketvar.h in FreeBSD 6 Stable from the default 128kbytes to 768kbytes. Any higher values then 768kbytes result in a integer overflow and prevents a build world of FreeBSD 6. The critical hit is around 120~130 active tunnels out of a configuration of 390. After his critical ceiling the racoon keeps getting stuck in sbwait after about 1 or 2 minutes after starting. A good way to test this with less tunnels is sending reload signals to the racoon processes which forces a lot of pfkey traffic. Which will eventually trigger the truncated pfkey socket buffer problem. The real problem is that racoon 0.6.7 and racoon 0.7 do not correctly handle a truncated socket. Instead of retrying the operation they either idle in sbwait and waiting for something that will never arrive or run away on a buffer which does not exist anymore. Any further suggestions? Kind regards, Seth Mos pfSense Developer Seth Mos schreef: > Hello there, > > I have problems with racoon hanging in sbwait state with ipsec-tools 0.6.7 > or getting into a tailspin on ipsec-tools 0.7. > > The problem is that the pfkey interface breaks down with a lot of VPN > tunnels and spd entries. The FreeBSd PR is here. > http://www.freebsd.org/cgi/query-pr.cgi?pr=115651 > > I have 390 discrete IPSEC VPN tunnels and endpoints. I have loaded this > all up into one config. I am using pfSense as the platform of choice. > pfSense 1.2-RC2 specifically which is based on FreeBSD 6.2 Stable p7. Note > that I am also a pfSense developer. > > At this current time I have exactly 112 IPSEC tunnels active. I am using > 3des-sha1 with a 3600 lifetime for phase 1 and aes128-sha1 with a 28800 > lifetime for phase2. > > On ipsec-tools 0.6.7 I can go by for several days before the racoon > process wedges itself into a hanging sbwait state (0% cpu). On ipsec-tools > 0.7 the situation is significantly worse and it starts churning 100% cpu > somewhere every 1-4 hours. > > Basically where 0.6.7 was difficult 0.7 has become unworkable. > > The hardware in question is a Dell PE860 with 6 gigabit nics (about 2mbps > ipsec traffic at most) with a DualCore Xeon 3050 2.13Ghz. With 1GB ram. > > In the pfSense kernel we use this patch. > http://cvs.pfsense.com/cgi-bin/cvsweb.cgi/tools/patches/RELENG_6_2/socketvar.h.diff > > Which helps significantly. Without this patch the situation is the same as > I have described above but will be reached at about 30-40 active tunnels > instead of the 112 I have now. > > I really need a reliable solution to this problem. > > Kind regards, > > Seth Mos > pfSense Developer >