From owner-freebsd-net@FreeBSD.ORG Tue Jan 4 19:19:11 2005 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0362416A4CE; Tue, 4 Jan 2005 19:19:11 +0000 (GMT) Received: from mail.trippynames.com (mail.trippynames.com [38.113.223.19]) by mx1.FreeBSD.org (Postfix) with ESMTP id A59A643D55; Tue, 4 Jan 2005 19:19:10 +0000 (GMT) (envelope-from sean@chittenden.org) Received: from localhost (localhost [127.0.0.1]) by mail.trippynames.com (Postfix) with ESMTP id C4CAAA8121; Tue, 4 Jan 2005 11:18:04 -0800 (PST) Received: from mail.trippynames.com ([127.0.0.1]) by localhost (rand.nxad.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 33568-01; Tue, 4 Jan 2005 11:17:47 -0800 (PST) Received: from [192.168.1.5] (dsl081-069-073.sfo1.dsl.speakeasy.net [64.81.69.73]) by mail.trippynames.com (Postfix) with ESMTP id B0480A81B2; Tue, 4 Jan 2005 11:17:27 -0800 (PST) In-Reply-To: <7632915A8F000C4FAEFCF272A880344165164F@Ehost067.exch005intermedia.net> References: <7632915A8F000C4FAEFCF272A880344165164F@Ehost067.exch005intermedia.net> Mime-Version: 1.0 (Apple Message framework v619) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <69A7A6D4-5E85-11D9-ACFF-000A95C705DC@chittenden.org> Content-Transfer-Encoding: 7bit From: Sean Chittenden Date: Tue, 4 Jan 2005 11:18:28 -0800 To: "Jeff Behl" X-Mailer: Apple Mail (2.619) cc: freebsd-net@freebsd.org cc: freebsd-performance@freebsd.org Subject: Re: %cpu in system - squid performance in FreeBSD 5.3 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jan 2005 19:19:11 -0000 > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU > COMMAND > 474 squid 96 0 68276K 62480K select 0 53:38 16.80% 16.80% > squid > 311 bind 20 0 10628K 6016K kserel 0 12:28 0.00% 0.00% > named > > It's actually so good that one machine can now handle all traffic > (around 180 Mb/s) at < %50 cpu utilization. Seems like something in > the > network stack is responsible for the high %system cpu util... OH!!!! Wow, I should've noticed that earlier. Your hint about someone on Linux having the same problem tipped me off to look at the process state again. Anyway, you nearly answered your own question, save you probably aren't familiar with select(2)'s lack of scalability. Read these: http://www.kegel.com/c10k.html Specifically the four methods mentioned here: http://www.kegel.com/c10k.html#nb.select Then look at the benchmarks done using libevent(3): http://www.monkey.org/~provos/libevent/ Dime to dollar you're spending all of your time copying file descriptor arrays in and out of the kernel because squid uses select(2) instead of kqueue(2). Might be an interesting project for someone to take up to convert that to kqueue(2). Until then, any local TCP load balancer that uses kqueue(2) would also solve your problem (I'm not aware of any off the top of my head... pound(8) does, but it is only used for HTTP and is not a reverse proxy) and would likely prevent you from having your problems. -sc -- Sean Chittenden