From owner-freebsd-net@FreeBSD.ORG Fri Dec 19 14:27:58 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CB890388; Fri, 19 Dec 2014 14:27:58 +0000 (UTC) Received: from relay1.speechpro.com (relay1.speechpro.com [79.134.214.22]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 226B22124; Fri, 19 Dec 2014 14:27:58 +0000 (UTC) Received: from mail.stc ([192.168.8.64] helo=spb2.speechpro.com) by relay1.speechpro.com with esmtp (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1Y1yED-000MSX-6t; Fri, 19 Dec 2014 17:08:25 +0300 Received: from [192.168.6.123] by mail.stc with esmtp (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1Y1yEC-000FDC-TU; Fri, 19 Dec 2014 17:08:24 +0300 Message-ID: <54943158.90307@speechpro.com> Date: Fri, 19 Dec 2014 17:08:24 +0300 From: Yuriy Tabolin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: Jack Vogel , "Alexander V. Chernikov" , FreeBSD Net Subject: Re: problem with 9k jumbo clusters References: <54803C75.2020300@speechpro.com> <5480AF38.3070100@FreeBSD.org> In-Reply-To: X-Archived: Yes Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Dec 2014 14:27:59 -0000 I tried to understand the code ixgbe, but I did not succeed. If anybody could, make patch for me, please. 05.12.2014 00:49, Jack Vogel пишет: > I had wanted to remove the larger cluster sizes from the driver a > while back, but for reasons > I don't remember that code change didn't happen. The new 40G ixl > driver does this, it only > uses standard clusters for anything under 2K, and above that > everything uses 4K. > > I would be curious to see if this change would resolve your problem, > would you like a patch, > or are you able to hack the code yourself to do this? > > Jack > > > On Thu, Dec 4, 2014 at 11:00 AM, Alexander V. Chernikov > > wrote: > > On 04.12.2014 13:50, Yuriy Tabolin wrote: > > Hi All. > I have a server with two Intel 10G NIC. OS FreeBSD > 10.1-Release amd64. Server works like NFS, samba-server and > iSCSI target. Both NICs aggregated into lagg device and set > MTU 9014 to them. There are some tuning sysctl.conf: > kern.maxfiles=6289601 > kern.maxfilesperproc=5660640 > kern.maxvnodes=3339565 > kern.ipc.nmbclusters=12255588 > kern.ipc.nmbjumbop=6127794 > kern.ipc.nmbufs=78435780 > kern.ipc.maxsockbuf=16777216 > kern.ipc.maxsockets=6289600 > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.sendspace=65536 > net.inet.tcp.recvbuf_max=16777216 > net.inet.tcp.recvspace=65536 > > After some days of working, the errors are appearing: > ix1: Interface stopped DISTRIBUTING, possible flapping > ix0: Interface stopped DISTRIBUTING, possible flapping > ix0: Could not setup receive structures > ix1: Could not setup receive structures > > Hello. It looks like > https://lists.freebsd.org/pipermail/freebsd-net/2014-May/038630.html > is relevant here. > > > After that errors the NICs stoped working. netstat -m shows: > 32881/33854/66735 mbufs in use (current/cache/total) > 16370/8198/24568/12255588 mbuf clusters in use > (current/cache/total/max) > 16370/4807 mbuf+clusters out of packet secondary zone in use > (current/cache) > 0/873/873/6127794 4k (page size) jumbo clusters in use > (current/cache/total/max) > 16383/21517/37900/1815641 9k jumbo clusters in use > (current/cache/total/max) > 0/0/0/1021298 16k jumbo clusters in use (current/cache/total/max) > 188407K/222004K/410411K bytes allocated to network > (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) > 0/101414306/0 requests for jumbo clusters denied (4k/9k/16k) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > > 9k jumbo clusters max is too big, but looks like system cannot > allocate them. There are huge number of "9k requests for jumbo > clusters denied". ifconfig ix down/up don't helped, reboot is > needed. Thanks for any help! > > > -- > Best regards, > Tabolin Yuriy > System administrator > Speech Technology Center > _______________________________________________ > freebsd-net@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org > " > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org > " > > -- Best regards, Tabolin Yuriy System administrator Speech Technology Center