From owner-freebsd-net@FreeBSD.ORG Mon Feb 15 14:50:28 2010 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C406710656A4; Mon, 15 Feb 2010 14:50:28 +0000 (UTC) (envelope-from babkin@verizon.net) Received: from vms173005pub.verizon.net (vms173005pub.verizon.net [206.46.173.5]) by mx1.freebsd.org (Postfix) with ESMTP id A31D18FC22; Mon, 15 Feb 2010 14:50:28 +0000 (UTC) Received: from verizon.net ([unknown] [173.54.27.21]) by vms173005.mailsrvcs.net (Sun Java(tm) System Messaging Server 7u2-7.02 32bit (built Apr 16 2009)) with ESMTPA id <0KXV00L0DYFDI671@vms173005.mailsrvcs.net>; Mon, 15 Feb 2010 07:50:02 -0600 (CST) Sender: root Message-id: <4B79205B.619A0A1A@verizon.net> Date: Mon, 15 Feb 2010 05:22:19 -0500 From: Sergey Babkin X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 4.7-RELEASE i386) X-Accept-Language: en, ru MIME-version: 1.0 To: Maxim Sobolev References: <4B79297D.9080403@FreeBSD.org> Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7bit Cc: freebsd-net@FreeBSD.org, Jack Vogel , FreeBSD Hackers Subject: Re: Sudden mbuf demand increase and shortage under the load X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Feb 2010 14:50:28 -0000 Maxim Sobolev wrote: > > Hi, > > Our company have a FreeBSD based product that consists of the numerous > interconnected processes and it does some high-PPS UDP processing > (30-50K PPS is not uncommon). We are seeing some strange periodic > failures under the load in several such systems, which usually evidences > itself in IPC (even through unix domain sockets) suddenly either > breaking down or pausing and restoring only some time later (like 5-10 > minutes). The only sign of failure I managed to find was the increase of > the "requests for mbufs denied" in the netstat -m and number of total > mbuf clusters (nmbclusters) raising up to the limit. As a simple idea: UDP is not flow-controlled. So potentially nothing stops an application from sending the packets as fast as it can. If it's faster than the network card can process, they would start collecting. So this might be worth a try as a way to reproduce the problem and see if the system has a safeguard against it or not. Another possibility: what happens if a process is bound to an UDP socket but doesn't actually read the data from it? FreeBSD used to be pretty good at it, just throwing away the data beyond a certain limit, SVR4 was running out of network memory. But it might have changed, so might be worth a look too. -SB