Date: Mon, 15 Feb 2010 05:22:19 -0500 From: Sergey Babkin <babkin@verizon.net> To: Maxim Sobolev <sobomax@FreeBSD.org> Cc: freebsd-net@FreeBSD.org, Jack Vogel <jfvogel@gmail.com>, FreeBSD Hackers <freebsd-hackers@FreeBSD.org> Subject: Re: Sudden mbuf demand increase and shortage under the load Message-ID: <4B79205B.619A0A1A@verizon.net> References: <4B79297D.9080403@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Maxim Sobolev wrote: > > Hi, > > Our company have a FreeBSD based product that consists of the numerous > interconnected processes and it does some high-PPS UDP processing > (30-50K PPS is not uncommon). We are seeing some strange periodic > failures under the load in several such systems, which usually evidences > itself in IPC (even through unix domain sockets) suddenly either > breaking down or pausing and restoring only some time later (like 5-10 > minutes). The only sign of failure I managed to find was the increase of > the "requests for mbufs denied" in the netstat -m and number of total > mbuf clusters (nmbclusters) raising up to the limit. As a simple idea: UDP is not flow-controlled. So potentially nothing stops an application from sending the packets as fast as it can. If it's faster than the network card can process, they would start collecting. So this might be worth a try as a way to reproduce the problem and see if the system has a safeguard against it or not. Another possibility: what happens if a process is bound to an UDP socket but doesn't actually read the data from it? FreeBSD used to be pretty good at it, just throwing away the data beyond a certain limit, SVR4 was running out of network memory. But it might have changed, so might be worth a look too. -SB
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4B79205B.619A0A1A>