From owner-freebsd-net@FreeBSD.ORG Sat Oct 26 18:55:20 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4F34E159 for ; Sat, 26 Oct 2013 18:55:20 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-ve0-x229.google.com (mail-ve0-x229.google.com [IPv6:2607:f8b0:400c:c01::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0E9992D50 for ; Sat, 26 Oct 2013 18:55:19 +0000 (UTC) Received: by mail-ve0-f169.google.com with SMTP id c14so2363459vea.0 for ; Sat, 26 Oct 2013 11:55:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=yNdU9k4G49LfaZJ6BtlEvU8UMxbwTl0DV8qcdC/BObQ=; b=fKFW156PbsJVjuipT3XnP5ye8ZsNjIl64WmRKmaA7Y/Dj1Lg/aYeP/ZJrcrlM8RDYo kh99ukZO16KKzsWOmtKxWdua0ULgKqQDx68+6noYeRC4w6gG1k8nHSCpUtsEdM6IGzzX AE9kk+P5TTWtSxUynipAJvdXGVVgYuQbLLMXeDIlCPB6zQEZQk5dEAtLldDyE4Q4M42+ +2/5Zcr0f87oiCd6mx8pWuxfYGzcyRDtC5jXFMyOD04amXgfsOaMUgHgbpKy8c1LsXL2 CKViZ/xWdBIq2v4WDCF4+zfC9zr0EcZqHueZU5E3Hl0tVaQMrh3J1kUoyrcfaHQ75Chn RlsA== MIME-Version: 1.0 X-Received: by 10.52.118.73 with SMTP id kk9mr6893426vdb.13.1382813719122; Sat, 26 Oct 2013 11:55:19 -0700 (PDT) Received: by 10.220.30.130 with HTTP; Sat, 26 Oct 2013 11:55:19 -0700 (PDT) In-Reply-To: <201310260552.r9Q5qcIv092748@hergotha.csail.mit.edu> References: <201310260552.r9Q5qcIv092748@hergotha.csail.mit.edu> Date: Sat, 26 Oct 2013 14:55:19 -0400 Message-ID: Subject: Re: Or it could be ZFS memory starvation and 9k packets (was Re: istgt causes massive jumbo nmbclusters loss) From: Zaphod Beeblebrox To: Garrett Wollman Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Oct 2013 18:55:20 -0000 To be clear, I made just this patch: Index: if_em.c =================================================================== --- if_em.c (revision 256870) +++ if_em.c (working copy) @@ -1343,10 +1343,10 @@ */ if (adapter->hw.mac.max_frame_size <= 2048) adapter->rx_mbuf_sz = MCLBYTES; - else if (adapter->hw.mac.max_frame_size <= 4096) + else /*if (adapter->hw.mac.max_frame_size <= 4096) */ adapter->rx_mbuf_sz = MJUMPAGESIZE; - else - adapter->rx_mbuf_sz = MJUM9BYTES; + /* else + adapter->rx_mbuf_sz = MJUM9BYTES; */ /* Prepare receive descriptors and buffers */ if (em_setup_receive_structures(adapter)) { (which is against 9.2-STABLE if you're looking). The result is that no 9k clusters appear to be allocated. I'm still running the system as before, but so far the problem has not recurred. Of note, given your comment, is that this patch doesn't appear to break anything, either. Should I send-pr it? On Sat, Oct 26, 2013 at 1:52 AM, Garrett Wollman < wollman@hergotha.csail.mit.edu> wrote: > In article < > CACpH0MfEy50Y5QOZCdn2co_JmY_QPfVRxYwK-73W0WYsHB-Fqw@mail.gmail.com> you > write: > > >Now... below the netstat -m shows 1399 9k bufs with 376 available. When > >the network gets busy, I've seen 4k or even 5k bufs in total... never near > >the 77k max. After some time of lesser activity, the number of 9k buffers > >returns to this level. > > The network interface (driver) almost certainly should not be using 9k > mbufs. These buffers are physically contiguous, and after not too > much activity, it will be nearly impossible to allocate three > physically contiguous buffers. > > >> That has an em0 with jumbo packets enabled: > >> > >> em0: flags=8843 metric 0 mtu > 9014 > > I don't know for certain about em(4), but it very likely should not be > using 9k mbufs. Intel network hardware has done scatter-gather since > nearly the year dot. (Seriously, I wrote a network driver for the > i82586 back at the very beginning of FreeBSD's existence, and *that* > part had scatter-gather. No jumbo frames, though!) > > The entire existence of 9k and 16k mbufs is probably a mistake. There > should not be any network interfaces that are modern enough to do > jumbo frames but ancient enough to require physically contiguous pages > for each frame. I don't know if the em(4) driver is written such that > you can just disable the use of those mbufs, though. You could try > making this change, though. Look for this code in if_em.c: > > /* > ** Figure out the desired mbuf > ** pool for doing jumbos > */ > if (adapter->max_frame_size <= 2048) > adapter->rx_mbuf_sz = MCLBYTES; > else if (adapter->max_frame_size <= 4096) > adapter->rx_mbuf_sz = MJUMPAGESIZE; > else > adapter->rx_mbuf_sz = MJUM9BYTES; > > Comment out the last two lines and change the else if (...) to else. > It's not obvious that the rest of the code can cope with this, but it > does work that way on other Intel hardware so it seems like it may be > worth a shot. > > -GAWollman >