From owner-freebsd-hackers  Thu Nov  6 02:14:55 1997
Return-Path: <owner-freebsd-hackers>
Received: (from root@localhost)
          by hub.freebsd.org (8.8.7/8.8.7) id CAA05285
          for hackers-outgoing; Thu, 6 Nov 1997 02:14:55 -0800 (PST)
          (envelope-from owner-freebsd-hackers)
Received: from implode.root.com (implode.root.com [198.145.90.17])
          by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id CAA05276
          for <hackers@FreeBSD.ORG>; Thu, 6 Nov 1997 02:14:52 -0800 (PST)
          (envelope-from root@implode.root.com)
Received: from implode.root.com (localhost [127.0.0.1])
	by implode.root.com (8.8.5/8.8.5) with ESMTP id CAA10471;
	Thu, 6 Nov 1997 02:17:25 -0800 (PST)
Message-Id: <199711061017.CAA10471@implode.root.com>
To: Joe McGuckin <joe@via.net>
cc: hackers@FreeBSD.ORG
Subject: Re: 'sys' is consuming %50 of cpu 
In-reply-to: Your message of "Wed, 05 Nov 1997 12:11:32 PST."
             <199711052011.MAA27359@monk.via.net> 
From: David Greenman <dg@root.com>
Reply-To: dg@root.com
Date: Thu, 06 Nov 1997 02:17:25 -0800
Sender: owner-freebsd-hackers@FreeBSD.ORG
X-Loop: FreeBSD.org
Precedence: bulk

>I've got a web server that was just upgraded to a 300Mhz Pentium II.
>
>Now the load average is down to a more manageable '20'.
>
>This machine has around 3K tcp connections to it according to netstat.
>Here they are listed by type:
>
>CLOSE_WAIT                5
>LAST_ACK                 25
>ESTABLISHED             443
>SYN_RCVD                 59
>FIN_WAIT_1              341
>LISTEN                    1
>TIME_WAIT               723
>FIN_WAIT_2             1716
>CLOSING                  29
>
>One thing I've noticed is that vmstat reports that the kernel
>is using over 50% of the cpu:
...
>Is there anything I can do to reduce the amount of cpu the kernel is using?
>With this many network connections, would I gain anything by making the
> TCBHASHSIZE
>larger?

   Yes, increasing TCBHASHSIZE would help - I use 2048 on wcarchive. Be sure
to keep it a power of two, however. You didn't mention which version of
FreeBSD this was with, but there are some general things I can say...first,
the suggestion made by others to upgrade the ethernet card to a Intel Pro/100B
is a good one. This should trim 10-15% off the CPU time (some from system,
some from interrupt). There is, however, a scalability problem in FreeBSD with
handling many thousands of connections like you have on your machine. I've
been working on this issue on & off for nearly 3 years (I implemented the PCB
hashing in FreeBSD more than 2 1/2 years ago), but just started looking at it
again recently about a week ago. There are many things to do, and hopefully
I'll have some major improvements for this problem in the 3.0 release.

-DG

David Greenman
Core-team/Principal Architect, The FreeBSD Project