From owner-freebsd-net@FreeBSD.ORG Thu Nov 6 12:30:51 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 91B9C106564A for ; Thu, 6 Nov 2008 12:30:51 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from el-out-1112.google.com (el-out-1112.google.com [209.85.162.181]) by mx1.freebsd.org (Postfix) with ESMTP id 48EFF8FC26 for ; Thu, 6 Nov 2008 12:30:50 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: by el-out-1112.google.com with SMTP id v27so282242ele.13 for ; Thu, 06 Nov 2008 04:30:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=Kqw75D/i7wEk8GRFxDs3owldEFdr53J6sIm+yLVyNSY=; b=owJDRG5tmU7hjCsxKEBxALB/+4rQ2lDSrzRPcj4aEXtHDgNfUk24vd/4hQz/0b/MX7 IwxkVoGZeR177Tfp9sBfaRl6RujCgQ5KqlRJQ81/HhMr72klvtG6R8CZjdHBqVOExCyz /Ls5NEJ5rSbJ1x3Hfu2wdo2D++9h2sWnXhpRg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=pMsMbnwDP5qD77eQUn/7F2YZnL1Qh9UVfg8UCOChxhujnkROrgyUC6mZ5UAhT6hyjj erSUyw5jklbthFvPKM0uhJuRUNAB2OCzD0LoGZvcwoOQJhLfZkHaxoICJZeePtcU3VMt hXghraHjTWKSkPzNAodr9n0eRQhOcGV0s14+k= Received: by 10.90.31.8 with SMTP id e8mr1772192age.37.1225974649704; Thu, 06 Nov 2008 04:30:49 -0800 (PST) Received: by 10.90.81.10 with HTTP; Thu, 6 Nov 2008 04:30:49 -0800 (PST) Message-ID: Date: Thu, 6 Nov 2008 15:30:49 +0300 From: pluknet To: "Peter Jeremy" In-Reply-To: <20081106104307.GC51239@server.vk2pj.dyndns.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20081106104307.GC51239@server.vk2pj.dyndns.org> Cc: FreeBSD Net Subject: Re: CARP performance tuning question. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Nov 2008 12:30:51 -0000 2008/11/6 Peter Jeremy : > Whilst I don't doubt that you have a problem, your comments don't > correlate particularly well with the data you have provided and > this makes it difficult to immediately suggest a solution. > > On 2008-Nov-05 16:40:32 +0300, pluknet wrote: >>AT work we use device carp(4) under high load: > > carp(4) is solely a failover mechanism. It either generates or receives > somewhat under 1pps per carp interface and the state it maintains is > basically 'master' or 'backup'. I suspect the 'load' is being caused > by pf(4), possibly in conjunction with pfsync(4). > >>The problem is that the server experiences a bad interactivity (from >>70k states and very bad from 120-150k) >>i.e. when a network workload (and interrupts count) begin to increase. >> >>>From top(1): >>CPU states: 0.0% user, 0.0% nice, 0.4% system, 76.3% interrupt, 23.3% idle >> PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND >> 13 root 1 -44 -163 0K 8K WAIT 407:43 57.86% swi1: net > > I agree that swi1 is using a significant amount of CPU but top is > still reporting >23% idle so you shouldn't be getting poor interactive > performance. > >>ATM pfctl -s info shows such numbers: >> >>State Table Total Rate >> current entries 153972 >> searches 6052078938 4800.8/s >> inserts 120373545 95.5/s >> removals 120219573 95.4/s > > That shows the load on pf(4) but doesn't really reflect what the > system is doing as a whole. > >>It works currently under UP, but could be rebuilt to work under SMP >>(Xeon 5130) if that helps. > > Unfortunately, I don't know if this will help or not because I'm not > sure what bottleneck you are hitting. > >>Can someone give hints to decrease interrupt count and to help with >>the server stability at all? > > Well, you haven't actually reported what the interrupt count or > what instability you are seeing so this is a bit difficult. > > Can you please provide some more information: > - output from 'uname -a' > - output from 'vmstat -i; sleep 10; vmstat -i' under load > - output from 'netstat -i' > - 10-15 seconds of output from 'netstat -i 1' under load > - What is the box doing? Is it a straight filtering router? Does it > handle NAT? Is it running apps itself (eg web, ftp, mail)? > - What speed are the interface(s) running at? > - What instability problems are you seeing? > - Please provide more details on what you mean by 'bad interactivity'. > - How complex is your pf ruleset? How many rules? Anything unusual? > - What scheduler are you using? > - What is the full output of 'pfctl -s info'? > Thanks for your answer and, please, ignore this premature mail. It would need a bit more analysis. -- wbr, pluknet