From owner-freebsd-net@FreeBSD.ORG Thu Dec 4 21:49:37 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5E8521CC; Thu, 4 Dec 2014 21:49:37 +0000 (UTC) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EA36DABE; Thu, 4 Dec 2014 21:49:36 +0000 (UTC) Received: by mail-wi0-f177.google.com with SMTP id l15so29416662wiw.16 for ; Thu, 04 Dec 2014 13:49:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=pRsp0DV9ons8UKz0d4C/LlqkOxqbcpB8lHoEOiEcPBs=; b=p0uDVthCbgbtgBy1MT2LQxCSNr0U69HQK6YJCAYKLXv4xK6eHjNBCYZ2XOAw2kfpqe 3Jfft5vZBDoBL7i6nGgTcfJWnEJw72WKE37G+Gd4V0AYU1xQzK2fBRT7b1PprAxKKe5J Gx8758hHQ3nfqaA35r508b4iXG1mVRrtj1BHcjwp+xzG9YVaWJSTtfPSYAHkVGyUr2OX wP6dlluOd9fY5FuqdDSSZWGz8gHOxUrDc8uSuELLeXM3pO1Hn1jYj+6qNcJXdybbtl7a 5dXGKhz/82uVbwGvP4UTtLhoG3PQKynEAKP0tnhJ31b9RDtZs0IGUU4x1hGdSQC8/W9g NSWA== MIME-Version: 1.0 X-Received: by 10.180.75.199 with SMTP id e7mr377300wiw.21.1417729775421; Thu, 04 Dec 2014 13:49:35 -0800 (PST) Received: by 10.194.81.233 with HTTP; Thu, 4 Dec 2014 13:49:35 -0800 (PST) In-Reply-To: <5480AF38.3070100@FreeBSD.org> References: <54803C75.2020300@speechpro.com> <5480AF38.3070100@FreeBSD.org> Date: Thu, 4 Dec 2014 13:49:35 -0800 Message-ID: Subject: Re: problem with 9k jumbo clusters From: Jack Vogel To: "Alexander V. Chernikov" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: FreeBSD Net , Yuriy Tabolin X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Dec 2014 21:49:37 -0000 I had wanted to remove the larger cluster sizes from the driver a while back, but for reasons I don't remember that code change didn't happen. The new 40G ixl driver does this, it only uses standard clusters for anything under 2K, and above that everything uses 4K. I would be curious to see if this change would resolve your problem, would you like a patch, or are you able to hack the code yourself to do this? Jack On Thu, Dec 4, 2014 at 11:00 AM, Alexander V. Chernikov < melifaro@freebsd.org> wrote: > On 04.12.2014 13:50, Yuriy Tabolin wrote: > >> Hi All. >> I have a server with two Intel 10G NIC. OS FreeBSD 10.1-Release amd64. >> Server works like NFS, samba-server and iSCSI target. Both NICs aggregated >> into lagg device and set MTU 9014 to them. There are some tuning >> sysctl.conf: >> kern.maxfiles=6289601 >> kern.maxfilesperproc=5660640 >> kern.maxvnodes=3339565 >> kern.ipc.nmbclusters=12255588 >> kern.ipc.nmbjumbop=6127794 >> kern.ipc.nmbufs=78435780 >> kern.ipc.maxsockbuf=16777216 >> kern.ipc.maxsockets=6289600 >> net.inet.tcp.sendbuf_max=16777216 >> net.inet.tcp.sendspace=65536 >> net.inet.tcp.recvbuf_max=16777216 >> net.inet.tcp.recvspace=65536 >> >> After some days of working, the errors are appearing: >> ix1: Interface stopped DISTRIBUTING, possible flapping >> ix0: Interface stopped DISTRIBUTING, possible flapping >> ix0: Could not setup receive structures >> ix1: Could not setup receive structures >> > Hello. It looks like > https://lists.freebsd.org/pipermail/freebsd-net/2014-May/038630.html > is relevant here. > > >> After that errors the NICs stoped working. netstat -m shows: >> 32881/33854/66735 mbufs in use (current/cache/total) >> 16370/8198/24568/12255588 mbuf clusters in use (current/cache/total/max) >> 16370/4807 mbuf+clusters out of packet secondary zone in use >> (current/cache) >> 0/873/873/6127794 4k (page size) jumbo clusters in use >> (current/cache/total/max) >> 16383/21517/37900/1815641 9k jumbo clusters in use >> (current/cache/total/max) >> 0/0/0/1021298 16k jumbo clusters in use (current/cache/total/max) >> 188407K/222004K/410411K bytes allocated to network (current/cache/total) >> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) >> 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) >> 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) >> 0/101414306/0 requests for jumbo clusters denied (4k/9k/16k) >> 0 requests for sfbufs denied >> 0 requests for sfbufs delayed >> 0 requests for I/O initiated by sendfile >> >> 9k jumbo clusters max is too big, but looks like system cannot allocate >> them. There are huge number of "9k requests for jumbo clusters denied". >> ifconfig ix down/up don't helped, reboot is needed. Thanks for any help! >> >> >> -- >> Best regards, >> Tabolin Yuriy >> System administrator >> Speech Technology Center >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> >> > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >