From owner-freebsd-current@FreeBSD.ORG Wed Oct 3 20:01:40 2012 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 48D3F1065674; Wed, 3 Oct 2012 20:01:40 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx1.freebsd.org (Postfix) with ESMTP id E97AC8FC1B; Wed, 3 Oct 2012 20:01:39 +0000 (UTC) Received: by oagn9 with SMTP id n9so8432303oag.13 for ; Wed, 03 Oct 2012 13:01:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=EZFDkdXYfASTgzGisFEhsYsaVriToztG9OWZGvRAoOE=; b=NoWnS46D5wduDE+cJF1gCG6TQbcrjlNNHuYKy99lAgOHqj6RKER41Cdn+wreWJZDmE s6YJC22AvFBxg4G5/6nBwLBc+8r/WTEduzk+fWTSJsuOzZXdFkrOTYTbS6IZPuAqOjDC vfn8zAB0mFMXmBaalbinoLylK+1P0iYF0JsoU2O3pszFiqn3u4AUPBfgPaShEQ1ggrjj N1It6dN9VmHLjvZcoSrq1sEN3kBySABrhShqdp8voFqw+m7xnvXY2VB7HhLelWMhTZyX GBDw7F49Q4NmRsJ8EEru/FIRCSmYBX6aneYk8nYbAgBLDaVF6wqOEjWk3C+mt5djIXei wNDw== MIME-Version: 1.0 Received: by 10.182.218.37 with SMTP id pd5mr2490917obc.24.1349294498995; Wed, 03 Oct 2012 13:01:38 -0700 (PDT) Received: by 10.76.142.201 with HTTP; Wed, 3 Oct 2012 13:01:38 -0700 (PDT) In-Reply-To: <03e101cda197$326dc240$974946c0$@org> References: <03e101cda197$326dc240$974946c0$@org> Date: Wed, 3 Oct 2012 13:01:38 -0700 Message-ID: From: Garrett Cooper To: freebsd@chrysalisnet.org Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-current@freebsd.org, Gleb Smirnoff Subject: Re: sysctl kern.ipc.somaxconn limit 65535 why? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Oct 2012 20:01:40 -0000 On Wed, Oct 3, 2012 at 11:45 AM, wrote: > Hi everyone. > > Actually 65k sockets is incredibly easy to reach. > > I manage some servers for a very large website, it currently has several > http servers clustered to handle daily traffic and this is only dynamic > content, static has its own servers, databases also have own servers. > > We recently started using memcached to cache some queries and we found on > default server limits sockets were saturated extremely quickly, on the linux > servers bumping it to a few hundred K solved the issue, wasnt possible on > bsd. We did manage to get it down to only needing about 20k by setting > extremely low timeout values. > > In addition we had to migrate all our mysql servers from freebsd to debian > because they were hitting some arbitary OS limit but I could never figure > out what, sys% usage went through the roof when this limit was hit, issue > didnt occur on debian. I feel recently freebsd is more focused on desktop's > and as such developer's never develop for a heavy server usage scenario, and > I keep coming across hardcoded low limits. As rightly pointed out default > values now days are useless 128 for somaxconn? maybe ok for a desktop. > > Freebsd also has issues with high numbers of cpu cores, very high locking > overheads, the problem is the modern hardware is focusing on more and more > cores and as such much more work on a server so a OS has to handle more and > more processing and data, this includes more bigger connection queues and so > on. FreeBSD just isnt scaling with more powerful hardware. Also forgetting > as well DDOS attack scenarios where a high queue size is useful. > > I have used FreeBSD since the 4.x days and have been forced to migrate OS on > several servers as FreeBSD to be blunt couldnt handle the load on those > servers. It does seem as if developers of the OS are getting out of touch > in respect to modern hosting enviromnents. I cant tell app developers to > fix their apps to work on FreeBSD, they dont care, if it works fine on > windows and linux then the app isnt broken as far as they are concerned. > The more FreeBSD dev's have this aatitude of not adapting the less machines > will run the OS. I hate many things about debian but at the end of the day > I have to use what can handle the load or I lose my job. > > I do accept its entirely possible some tunables could fix any issues I have > come across but believe me I have spent 100s of man hours on issues, and > tuned everything I could find. > > To me this 32767 limit on somaxconn seems restrive and I vote for it been > bumped to at least 8x that amount as a minimum. A more suitable default > would probably be around 1024 as well instead of 128. Dead lingering > connections in timewait etc. all use one of these, and that can very easily > get into the 1000s. Here's where it's being held at 65535 (sys/kern/kern_uipc.c): 3276 static int 3277 sysctl_somaxconn(SYSCTL_HANDLER_ARGS) 3278 { 3279 int error; 3280 int val; 3281 3282 val = somaxconn; 3283 error = sysctl_handle_int(oidp, &val, 0, req); 3284 if (error || !req->newptr ) 3285 return (error); 3286 3287 if (val < 1 || val > USHRT_MAX) 3288 return (EINVAL); 3289 More details about the need for this limit are in http://svnweb.freebsd.org/base?view=revision&revision=140730 . It looks like this needs to be enhanced to support more than USHRT_MAX, which will require socket structure changes and other fun things, but should be possible... I've CCed glebius for comment on this. Thanks! -Garrett