From owner-freebsd-questions@FreeBSD.ORG Mon Jan 2 19:54:35 2012 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D2FB10657E0 for ; Mon, 2 Jan 2012 19:54:35 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id 306ED8FC1C for ; Mon, 2 Jan 2012 19:54:35 +0000 (UTC) Received: from pps.filterd (ltcfislmsgpa04 [127.0.0.1]) by ltcfislmsgpa04.fnfis.com (8.14.4/8.14.4) with SMTP id q02JQd4j005317; Mon, 2 Jan 2012 13:54:34 -0600 Received: from smtp.fisglobal.com ([10.132.206.16]) by ltcfislmsgpa04.fnfis.com with ESMTP id 123hwm88gy-1 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Mon, 02 Jan 2012 13:54:34 -0600 Received: from dtwin (10.14.152.15) by smtp.fisglobal.com (10.132.206.16) with Microsoft SMTP Server (TLS) id 14.1.323.3; Mon, 2 Jan 2012 13:54:33 -0600 From: Devin Teske To: "'Muhammet S. AYDIN'" , References: In-Reply-To: Date: Mon, 2 Jan 2012 11:54:36 -0800 Message-ID: <046f01ccc988$5b710310$12530930$@fisglobal.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQD6WO/1aBJN+GJ6EC0xVuIXUuk2apeeWwyw Content-Language: en-us X-Originating-IP: [10.14.152.15] X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.5.7110, 1.0.211, 0.0.0000 definitions=2012-01-02_06:2012-01-02, 2012-01-02, 1970-01-01 signatures=0 Cc: Dave Robison , devin.teske@fisglobal.com Subject: RE: freebsd server limits question X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jan 2012 19:54:35 -0000 > -----Original Message----- > From: owner-freebsd-questions@freebsd.org [mailto:owner-freebsd- > questions@freebsd.org] On Behalf Of Muhammet S. AYDIN > Sent: Monday, January 02, 2012 11:13 AM > To: freebsd-questions@freebsd.org > Subject: freebsd server limits question > > Hello everyone. > > My first post here and I'd like to thank everyone who's involved within the > FreeBSD project. We are using FreeBSD on our web servers and we are very > happy with it. > > We have an online messaging application that is using mongodb. Our members > send messages to "the voice" show's (turkish version) contestants. Our two > mongodb instances ended up in two centos6 servers. We have failed. So hard. > There were announcements and calls made live on tv. We had +30K/sec visitors > to the app. > > When I looked at the mongodb errors, I had thousands of these: > http://pastie.org/private/nd681sndos0bednzjea0g. You may be wondering why > I'm telling you about centos. Well, we are making the switch from centos to > freebsd FreeBSD. I would like to know what are our limits? How we can set it up > so our FreeBSD servers can handle min 20K connections (mongodb's connection > limit)? > > Our two servers have 24 core CPUs and 32 GBs of RAM. We are also very open to > suggestions. Please help me out here so we don't fail deadly, again. We have similar hardware (24x core CPUs but 48GB of RAM instead of 32). NOTE: The machine has 2x igb(4) interfaces and we're negotiating at 1000baseTX "Gigabit" full-duplex link-speed. We had similar problems, but have had zero problems in the past 2 months with high-load (read below). ASIDE: We're using FreeBSD 8.1-RELEASE-p6 We found that the following tweaks had to be made in /etc/sysctl.conf : ### Network Tuning ### # Increase TCP maximum segment lifetime net.inet.tcp.msl=15000 # Increase TCP time before keepalive probes again net.inet.tcp.keepidle=300000 # Increase maximum number of mbuf clusters allowed (174808 => 32768) kern.ipc.nmbclusters=32768 # Increase by 8-times the maximum socket buffer size (262144 => 2097152) kern.ipc.maxsockbuf=2097152 # Increase by 64-times the max pending socket conn. queue size (128 => 8192) kern.ipc.somaxconn=8192 # Increase by ~8-times the maximum number of [open] files (8232 => 65536) kern.maxfiles=65536 # Increase by ~4-times the max files allowed open per process (7408 => 32768) kern.maxfilesperproc=32768 # Disable delay of ACK to try and piggyback it onto a data packet (1 => 0) net.inet.tcp.delayed_ack=0 # Increase by ~2-times the maximum outgoing TCP datagram size (32768 => 65535) net.inet.tcp.sendspace=65535 # Increase maximum space for incoming UDP datagrams (41600 => 65535) net.inet.udp.recvspace=65535 # Increase by ~6-times the maximum outgoing UDP datagram size (9216 => 57344) net.inet.udp.maxdgram=57344 # Increase by ~8-times the default stream receive space (8192 => 65535) net.local.stream.recvspace=65535 # Increase by ~8-times the default stream send space (8192 => 65535) net.local.stream.sendspace=65535 Meanwhile, yet more tweaks go into /boot/loader.conf : ### Process/Memory Tuning ### # Increase by 4-times the maximum data size (536870912 => 2147483648) kern.maxdsiz="2147483648" # Increase by 4-times the maximum stack size (67108864 => 268435456) kern.maxssiz="268435456" ### Network Tuning ### # Increase maximum outgoing Netgraph datagram size (20480 => 45000) net.graph.maxdgram="45000" # Increase maximum space for incoming Netgraph datagrams (20480 => 45000) net.graph.recvspace="45000" # Increase by 128-times max num of data queue items to allocate (512 => 65536) net.graph.maxdata=65536 With the above tweaks in-place for both sysctl.conf(5) and loader.conf(5), all our problems are gone. Your mileage may vary, but I suspect that the above collection of tweaks will work well for you. They should be safe for both 32-bit (both regular and PAE) and 64 (all tested). However, if you are the cautious type, I would recommend adding one optimizer at a time, rebooting after each tweak. -- Devin _____________ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.