From owner-freebsd-questions@FreeBSD.ORG Fri Jul 8 07:20:42 2011 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F1B461065672 for ; Fri, 8 Jul 2011 07:20:42 +0000 (UTC) (envelope-from wooh@wooh.hu) Received: from mail.netidea.hu (netwarehouse.netidea.hu [195.228.254.126]) by mx1.freebsd.org (Postfix) with ESMTP id 81E1F8FC12 for ; Fri, 8 Jul 2011 07:20:42 +0000 (UTC) Received: from radon (localhost [127.0.0.1]) by mail.netidea.hu (Postfix) with ESMTP id C62BFC89902 for ; Fri, 8 Jul 2011 08:55:39 +0200 (CEST) X-Virus-Scanned: amavisd-new at netidea.hu Received: from mail.netidea.hu ([127.0.0.1]) by radon (mail.netidea.hu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z-EctDdcyJKf for ; Fri, 8 Jul 2011 08:55:37 +0200 (CEST) Received: from helium-2.local (catv-80-99-86-247.catv.broadband.hu [80.99.86.247]) by mail.netidea.hu (Postfix) with ESMTPA id E7AA6C8984E for ; Fri, 8 Jul 2011 08:55:36 +0200 (CEST) Message-ID: <4E16AB2F.6050109@wooh.hu> Date: Fri, 08 Jul 2011 09:01:03 +0200 From: Adam PAPAI User-Agent: Postbox 2.5.0 (Macintosh/20110628) MIME-Version: 1.0 To: freebsd-questions@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: RELEASE-8.1 with heavy network activity (nginx + php-fpm) X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2011 07:20:43 -0000 I have a FreeBSD box with 8.1. It seems we hit the limits for the maximum network connections, because I always see this message in the nginx-error.log: 2011/07/08 08:48:40 [error] 40438#0: *30564045 kevent() reported that connect() failed (54: Connection reset by peer) while connecting to upstream, client: 188.36.171.27, server: netadclick.com, request: "GET /hirdetes/148 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.netadclick.com", referrer: "http://www.netadclick.com/hirdetes/148" 2011/07/08 08:48:41 [error] 40438#0: *30564021 writev() failed (54: Connection reset by peer) while sending request to upstream, client: 95.171.73.72, server: netadclick.com, request: "GET /hirdetes/1051/0505 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.netadclick.com", referrer: "http://www.freespirit.hu/szex/durva_ezert_ne_hordj_tangat" The Box has approximately: 35,000 - 65,000 connections at the same time: In the early morning it has 45,564 connections: netstat -n | grep '^tcp4' | wc -l 45564 The system runs correctly for a few minutes, and after it's reaching "some" limits, all connections are reset (Connection reset by peer, the error log is filled with thousands of Connection reset by peer message) and it starts working correctly again for another few minutes. I've tuned some variables like: kern.ipc.shmall=32768 kern.ipc.shmmax=134217728 kern.ipc.semmap=256 net.inet.ip.maxfragpackets=1600 kern.threads.max_threads_per_proc=4096 kern.maxfiles=204800 kern.maxfilesperproc=200000 kern.maxvnodes=200000 net.inet.tcp.maxtcptw=200000 kern.ipc.nmbjumbo9=24000 kern.ipc.nmbjumbo16=10240 kern.ipc.maxsockets=65536 net.inet.tcp.fast_finwait2_recycle=1 net.inet.tcp.recvbuf_auto=0 net.inet.tcp.recvspace=65535 net.inet.ip.portrange.randomized=0 net.inet.ip.intr_queue_maxlen=4096 net.inet.tcp.finwait2_timeout=30000 kern.ipc.somaxconn=2048 /boot/loader.conf kern.ipc.semmni=1024 kern.ipc.semmns=2048 kern.ipc.semmnu=1024 kern.ipc.maxsockets=65536 kern.maxproc=12328 kern.ipc.shmmax=2147483648 kern.ipc.somaxconn=4096 net.inet.tcp.hostcache.cachelimit=65532 What should I do? Any ideas? Any tuning tips? Thanks in advance, -- Adam PAPAI