From owner-freebsd-hackers Mon Jun 24 15:31: 9 2002 Delivered-To: freebsd-hackers@freebsd.org Received: from utility.clubscholarship.com (utility.clubscholarship.com [198.78.70.175]) by hub.freebsd.org (Postfix) with ESMTP id 7BBA537B665 for ; Mon, 24 Jun 2002 15:29:03 -0700 (PDT) Received: from localhost (root@localhost) by utility.clubscholarship.com (8.11.6/8.11.6) with ESMTP id g5OMPbh47566 for ; Mon, 24 Jun 2002 15:25:42 -0700 (PDT) (envelope-from root@utility.clubscholarship.com) Date: Mon, 24 Jun 2002 15:25:37 -0700 (PDT) From: Patrick Thomas To: Subject: tunings for many httpds... Message-ID: <20020624151650.I68572-100000@utility.clubscholarship.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG As a splinter to the ongoing KVA/crash/memory discussion, I am wondering: - given a machine that will run 250+ httpds and another ~800 misc. processes, what system tunings would any of you suggest other than the ones I have done: In my kernel: maxusers=256 (was 512, change to 256 didn't help) options SHMMAXPGS=16384 options SHMMAX="(SHMMAXPGS*PAGE_SIZE+1)" options SHMSEG=256 options SEMMNI=384 options SEMMNS=768 options SEMMNU=384 options SEMMAP=384 (all this SHM and SEM stuff is to run multiple postgres') and at boot time: sysctl -w jail.sysvipc_allowed=1 sysctl -w kern.ipc.shmall=65535 sysctl -w kern.ipc.shmmax=134217728 sysctl -w net.inet.tcp.syncookies=0 Anything obvious I am missing ? Terry seems to think: It's obvious that you are running a large number of httpd's; the sbwait in this case could be reasonably assumed to be waits based on "sendfile" for a change in so->so_snd->sb_cc; if that's the case, then it may be that you are simply running out of mbufs, and are deadlocking. This can happen if you have enough data in the pipe that you can not receive more data (e.g. the m_pullup() in tcp_input() could fail before other things would fail). Two things about this interested me: a) watching `top` output anytime of the day, i see several httpd processes in "sbwait" - granted I can only see 40 lines of processes or so in `top`, but usually at least two show "sbwait". Worrisome ? b) As I showed him, the netstat -m output 30-60 seconds before the crash looks very benign: 524/2576/34816 mbufs in use (current/peak/max): 500 mbufs allocated to data 24 mbufs allocated to packet headers 273/2254/8704 mbuf clusters in use (current/peak/max) 5152 Kbytes allocated to network (19% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Is it possible that within 30 seconds or so current mbufs would skyrocket and my percentage of mb_map in use would skyrocket and I would start to see requests for memory denied ? All comments appreciated. --PT To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message