Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 2 Jan 2012 11:54:36 -0800
From:      Devin Teske <devin.teske@fisglobal.com>
To:        "'Muhammet S. AYDIN'" <whalberg@gmail.com>, <freebsd-questions@freebsd.org>
Cc:        Dave Robison <daver@vicor.com>, devin.teske@fisglobal.com
Subject:   RE: freebsd server limits question
Message-ID:  <046f01ccc988$5b710310$12530930$@fisglobal.com>
In-Reply-To: <CAP28s1DhhsSV%2Bz8BuRDVjHypD%2BpECuXQEH5BKjJRKMorcWL0rw@mail.gmail.com>
References:  <CAP28s1DhhsSV%2Bz8BuRDVjHypD%2BpECuXQEH5BKjJRKMorcWL0rw@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help


> -----Original Message-----
> From: owner-freebsd-questions@freebsd.org [mailto:owner-freebsd-
> questions@freebsd.org] On Behalf Of Muhammet S. AYDIN
> Sent: Monday, January 02, 2012 11:13 AM
> To: freebsd-questions@freebsd.org
> Subject: freebsd server limits question
> 
> Hello everyone.
> 
> My first post here and I'd like to thank everyone who's involved within the
> FreeBSD project. We are using FreeBSD on our web servers and we are very
> happy with it.
> 
> We have an online messaging application that is using mongodb. Our members
> send messages to "the voice" show's (turkish version) contestants. Our two
> mongodb instances ended up in two centos6 servers. We have failed. So hard.
> There were announcements and calls made live on tv. We had +30K/sec visitors
> to the app.
> 
> When I looked at the mongodb errors, I had thousands of these:
> http://pastie.org/private/nd681sndos0bednzjea0g. You may be wondering why
> I'm telling you about centos. Well, we are making the switch from centos to
> freebsd FreeBSD. I would like to know what are our limits? How we can set it
up
> so our FreeBSD servers can handle min 20K connections (mongodb's connection
> limit)?
> 
> Our two servers have 24 core CPUs and 32 GBs of RAM. We are also very open to
> suggestions. Please help me out here so we don't fail deadly, again.

We have similar hardware (24x core CPUs but 48GB of RAM instead of 32).

NOTE: The machine has 2x igb(4) interfaces and we're negotiating at 1000baseTX
"Gigabit" full-duplex link-speed.

We had similar problems, but have had zero problems in the past 2 months with
high-load (read below).

ASIDE: We're using FreeBSD 8.1-RELEASE-p6

We found that the following tweaks had to be made in /etc/sysctl.conf :

### Network Tuning ###
# Increase TCP maximum segment lifetime
net.inet.tcp.msl=15000
# Increase TCP time before keepalive probes again
net.inet.tcp.keepidle=300000
# Increase maximum number of mbuf clusters allowed (174808 => 32768)
kern.ipc.nmbclusters=32768
# Increase by 8-times the maximum socket buffer size (262144 => 2097152)
kern.ipc.maxsockbuf=2097152
# Increase by 64-times the max pending socket conn. queue size (128 => 8192)
kern.ipc.somaxconn=8192
# Increase by ~8-times the maximum number of [open] files (8232 => 65536)
kern.maxfiles=65536
# Increase by ~4-times the max files allowed open per process (7408 => 32768)
kern.maxfilesperproc=32768
# Disable delay of ACK to try and piggyback it onto a data packet (1 => 0)
net.inet.tcp.delayed_ack=0
# Increase by ~2-times the maximum outgoing TCP datagram size (32768 => 65535)
net.inet.tcp.sendspace=65535
# Increase maximum space for incoming UDP datagrams (41600 => 65535)
net.inet.udp.recvspace=65535
# Increase by ~6-times the maximum outgoing UDP datagram size (9216 => 57344)
net.inet.udp.maxdgram=57344
# Increase by ~8-times the default stream receive space (8192 => 65535)
net.local.stream.recvspace=65535
# Increase by ~8-times the default stream send space (8192 => 65535)
net.local.stream.sendspace=65535

Meanwhile, yet more tweaks go into /boot/loader.conf :

### Process/Memory Tuning ###
# Increase by 4-times the maximum data size (536870912 => 2147483648)
kern.maxdsiz="2147483648"
# Increase by 4-times the maximum stack size (67108864 => 268435456)
kern.maxssiz="268435456"
### Network Tuning ###
# Increase maximum outgoing Netgraph datagram size (20480 => 45000)
net.graph.maxdgram="45000"
# Increase maximum space for incoming Netgraph datagrams (20480 => 45000)
net.graph.recvspace="45000"
# Increase by 128-times max num of data queue items to allocate (512 => 65536)
net.graph.maxdata=65536


With the above tweaks in-place for both sysctl.conf(5) and loader.conf(5), all
our problems are gone.

Your mileage may vary, but I suspect that the above collection of tweaks will
work well for you. They should be safe for both 32-bit (both regular and PAE)
and 64 (all tested). However, if you are the cautious type, I would recommend
adding one optimizer at a time, rebooting after each tweak.
-- 
Devin

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?046f01ccc988$5b710310$12530930$>