From owner-freebsd-hackers Mon Jun 24 14:56:43 2002 Delivered-To: freebsd-hackers@freebsd.org Received: from utility.clubscholarship.com (utility.clubscholarship.com [198.78.70.175]) by hub.freebsd.org (Postfix) with ESMTP id 662B537B405 for ; Mon, 24 Jun 2002 14:56:18 -0700 (PDT) Received: from localhost (root@localhost) by utility.clubscholarship.com (8.11.6/8.11.6) with ESMTP id g5OLqLd46015; Mon, 24 Jun 2002 14:52:21 -0700 (PDT) (envelope-from root@utility.clubscholarship.com) Date: Mon, 24 Jun 2002 14:52:21 -0700 (PDT) From: Patrick Thomas To: Terry Lambert Cc: Nielsen , Subject: Re: (jail) problem and a (possible) solution ? In-Reply-To: <3D178E89.AB893E72@mindspring.com> Message-ID: <20020624143448.J68572-100000@utility.clubscholarship.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > It's obvious that you are running a large number of httpd's; the Yes, we are running a lot of httpd's: ps auxw | grep httpd | wc -l = 288 > The way to cross-check this would be to run a continuous "netstat -m", > e.g.: Funny you should ask :) I was already doing that. Here is the output from a `netstat -m` run once per minute - the machine crashed sometime in the next 30-60 seconds after I got this output: 524/2576/34816 mbufs in use (current/peak/max): 500 mbufs allocated to data 24 mbufs allocated to packet headers 273/2254/8704 mbuf clusters in use (current/peak/max) 5152 Kbytes allocated to network (19% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines > Basically, if you have any denials, or if the number of mbuf > clusters gets really large, then you could have a problem. Do you think it is reasonable that the above netstat -m output could, within 30 or so seconds, ramp up to the bad situation you are describing ? Because it looks fairly benign to me... I have three questions: 1. Forgetting about my paticular problem for a moment, let's say you have to tune a machine to run 200+ httpd servers along with another 800 misc. processes, etc. What do you suggest setting, just to be safe (again, as a precaution - forgetting that in reality I am tryig to fix a sick machine) So far I have only tuned: In my kernel: maxusers=256 (was 512, change to 256 didn't help) options SHMMAXPGS=16384 options SHMMAX="(SHMMAXPGS*PAGE_SIZE+1)" options SHMSEG=256 options SEMMNI=384 options SEMMNS=768 options SEMMNU=384 options SEMMAP=384 (all this SHM and SEM stuff is to run multiple postgres') and at boot time: sysctl -w jail.sysvipc_allowed=1 sysctl -w kern.ipc.shmall=65535 sysctl -w kern.ipc.shmmax=134217728 sysctl -w net.inet.tcp.syncookies=0 So anything obvious I am missing that you would tune for a 200+ http + 800 other processes machine? 2. Let's say I was being targeted by that effective attack you spoke of...any way to immunize myself ? 3. You spoke of: > # sysctl -a | grep tcp | grep space > net.inet.tcp.sendspace: 32768 > net.inet.tcp.recvspace: 65536 > > I guess the best way to deal with this would be to drop the size > of the send or receive queues, until it didn't consume all your > memory. In general, the size of these queues is supposed to be > a *maximum*, not a *mean*, so the number of sockets possible, > times the maximum total of both, will often exceed the amount of > available mbuf space. a) are you saying to collect these sysctls regularly and try to see their values right at the crash ? b) where do I "drop the size of the send or receive queues" ? (sysctl or kernel setting?) thank you very much. I will try to get a full `ps` tonight when it crashes again :( --PT > > An interesting attack that is moderately effective on FreeBSD > boxes is to send with a very large size, and not send one of > the fragments (e.g. the second one) to prevent fragment > reassembly, and therefore saturate the reassembly queue. The > Linux UDP NFS client code does this unintentionally, but you > could believe that someone might be doing it intentionally, > as well, which would also work against TCP. It's doubtful that > you are being hit by a FreeBSD targetted attack, however. > > -- Terry > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message