From owner-freebsd-net@FreeBSD.ORG Thu Nov 6 10:53:46 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65BB410657A1 for ; Thu, 6 Nov 2008 10:53:46 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from fallbackmx10.syd.optusnet.com.au (fallbackmx10.syd.optusnet.com.au [211.29.132.251]) by mx1.freebsd.org (Postfix) with ESMTP id EA1748FC0C for ; Thu, 6 Nov 2008 10:53:45 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail36.syd.optusnet.com.au (mail36.syd.optusnet.com.au [211.29.133.76]) by fallbackmx10.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id mA6AhBJB028562 for ; Thu, 6 Nov 2008 21:43:11 +1100 Received: from server.vk2pj.dyndns.org (c122-106-215-175.belrs3.nsw.optusnet.com.au [122.106.215.175]) by mail36.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id mA6Ah8MZ018156 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 6 Nov 2008 21:43:09 +1100 X-Bogosity: Ham, spamicity=0.000000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.3/8.14.3) with ESMTP id mA6Ah8oE054886; Thu, 6 Nov 2008 21:43:08 +1100 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.3/8.14.3/Submit) id mA6Ah8SQ054885; Thu, 6 Nov 2008 21:43:08 +1100 (EST) (envelope-from peter) Date: Thu, 6 Nov 2008 21:43:07 +1100 From: Peter Jeremy To: pluknet Message-ID: <20081106104307.GC51239@server.vk2pj.dyndns.org> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="/Uq4LBwYP4y1W6pO" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.18 (2008-05-17) Cc: FreeBSD Net Subject: Re: CARP performance tuning question. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Nov 2008 10:53:46 -0000 --/Uq4LBwYP4y1W6pO Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Whilst I don't doubt that you have a problem, your comments don't correlate particularly well with the data you have provided and this makes it difficult to immediately suggest a solution. On 2008-Nov-05 16:40:32 +0300, pluknet wrote: >AT work we use device carp(4) under high load: carp(4) is solely a failover mechanism. It either generates or receives somewhat under 1pps per carp interface and the state it maintains is basically 'master' or 'backup'. I suspect the 'load' is being caused by pf(4), possibly in conjunction with pfsync(4). >The problem is that the server experiences a bad interactivity (from >70k states and very bad from 120-150k) >i.e. when a network workload (and interrupts count) begin to increase. > >>From top(1): >CPU states: 0.0% user, 0.0% nice, 0.4% system, 76.3% interrupt, 23.3% i= dle > PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMM= AND > 13 root 1 -44 -163 0K 8K WAIT 407:43 57.86% swi1:= net I agree that swi1 is using a significant amount of CPU but top is still reporting >23% idle so you shouldn't be getting poor interactive performance. >ATM pfctl -s info shows such numbers: > >State Table Total Rate > current entries 153972 > searches 6052078938 4800.8/s > inserts 120373545 95.5/s > removals 120219573 95.4/s That shows the load on pf(4) but doesn't really reflect what the system is doing as a whole. >It works currently under UP, but could be rebuilt to work under SMP >(Xeon 5130) if that helps. Unfortunately, I don't know if this will help or not because I'm not sure what bottleneck you are hitting. >Can someone give hints to decrease interrupt count and to help with >the server stability at all? Well, you haven't actually reported what the interrupt count or what instability you are seeing so this is a bit difficult. Can you please provide some more information: - output from 'uname -a' - output from 'vmstat -i; sleep 10; vmstat -i' under load - output from 'netstat -i' - 10-15 seconds of output from 'netstat -i 1' under load - What is the box doing? Is it a straight filtering router? Does it handle NAT? Is it running apps itself (eg web, ftp, mail)? - What speed are the interface(s) running at? - What instability problems are you seeing? - Please provide more details on what you mean by 'bad interactivity'. - How complex is your pf ruleset? How many rules? Anything unusual? - What scheduler are you using? - What is the full output of 'pfctl -s info'? --=20 Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. --/Uq4LBwYP4y1W6pO Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkkSyjsACgkQ/opHv/APuIdoiQCgsTHYbDRYx+VnitKkbpy1OsmJ TEoAn0ZxKbz0Hy2BRiBTbVjzjEVVJD6G =Ef3M -----END PGP SIGNATURE----- --/Uq4LBwYP4y1W6pO--