From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 07:32:27 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F984106564A; Wed, 9 Feb 2011 07:32:27 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id DA66B8FC0C; Wed, 9 Feb 2011 07:32:26 +0000 (UTC) Received: by gyf3 with SMTP id 3so2756878gyf.13 for ; Tue, 08 Feb 2011 23:32:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=EXf4gLDmSS8e0pHwwLjDOE6oDUDiAbxuttVqInZTNvk=; b=dgeUbUAdxkgrlODcYfWVG6NnGfZ+lqZeNqDw1SvfF3HycLZon6PdUUdaaJUFDwlanY 4H9U5sKnf685tvsbN1xsniUuN03tTlKrf8FswlQyQcZ4hjQYN5nTdTJAHHLp0bfQ6Gg3 VL1MEvl2tE7yK5/NRnniCA47+x4GXM2BskodE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=dDBazXiZAaQv05K7laDMV/eUajr+VPQbfcsmbnmZDn8NKBFy3Ee/EcnMtaffelytB3 +803/YP3g6atWoUsI0xNATPnh1u4jTHtWIYjom4h2CbBC0LDFYVzNFdlSx5dd70V5j9s 8RtNbGA+fI0l9SAYm6ik1xMVRn4h9cKAMlEEk= MIME-Version: 1.0 Received: by 10.236.103.41 with SMTP id e29mr3876311yhg.11.1297236745789; Tue, 08 Feb 2011 23:32:25 -0800 (PST) Received: by 10.147.171.17 with HTTP; Tue, 8 Feb 2011 23:32:25 -0800 (PST) In-Reply-To: References: Date: Tue, 8 Feb 2011 23:32:25 -0800 Message-ID: From: Jack Vogel To: Karim Fodil-Lemelin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Pyun YongHyeon , beezarliu , Michael Tuexen , freebsd-net@freebsd.org Subject: Re: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 07:32:27 -0000 Hmmm, well so much for that theory :) Jack On Tue, Feb 8, 2011 at 4:06 PM, Karim Fodil-Lemelin < fodillemlinkarim@gmail.com> wrote: > > > 2011/2/8 Jack Vogel > > >> I have been following this, and thinking about it. I still am working from >> a theoretical >> standpoint, but based on a patch I got quite a long time back and never >> quite groked, >> I believe now that I might have a solution. >> >> The original PR and patch was kern/150516 from Beezar Liu, I was never >> quite comfortable >> with the code changes, nor convinced that it was a real issue and not a >> misunderstanding. >> However I think now that this very report might be behind what we are >> seeing today. I have >> a slightly different approach to solving it, of course it remains to be >> seen if it handles it >> properly. >> >> Please try the patch I've attached, I'm open to further correction or >> polishing of the >> changes. And thanks to Beezar for his original report and changes, this is >> not for em, >> but if this eliminates the problem its clearly needed in all drivers. >> >> Jack >> >> >> Hi Jack, > > Thanks for your help. I tried your patch and it didn't work so I added a > couple of printf to see if the added code was getting hit: > > --- a/freebsd/sys/dev/e1000/if_igb.c > --More--(byte 1253)+++ b/freebsd/sys/dev/e1000/if_igb.c > @@ -612,7 +612,7 @@ igb_attach(device_t dev) > device_get_nameunit(dev)); > > INIT_DEBUGOUT("igb_attach: end"); > - > + printf("this driver has a patch from Jack Vogel\n"); > return (0); > > err_late: > @@ -4131,6 +4131,7 @@ igb_rxeof(struct igb_queue *que, int count, int > *done) > struct mbuf *sendmp, *mh, *mp; > struct igb_rx_buf *rxbuf; > u16 hlen, plen, hdr, vtag; > + int commit; > bool eop = FALSE; > > cur = &rxr->rx_base[i]; > @@ -4255,10 +4256,23 @@ next_desc: > bus_dmamap_sync(rxr->rxdma.dma_tag, rxr->rxdma.dma_map, > BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); > > + commit = i; /* capture the old index */ > + > /* Advance our pointers to the next descriptor. */ > if (++i == adapter->num_rx_desc) > i = 0; > /* > + ** Sanity test for ring full, if this > + ** happens we need to refresh immediately > + ** or refresh may deadlock. > + */ > + if (i == rxr->next_to_refresh) { > + igb_refresh_mbufs(rxr, commit); > + printf("igb_refresh_mbufs called with commit %d\n", > commit); > + processed = 0; > + } > + > + /* > ** Send to the stack or LRO > */ > if (sendmp != NULL) { > > Here is the results: > > # dmesg | grep Vogel > this driver has a patch from Jack Vogel > this driver has a patch from Jack Vogel > > # netstat -m > 60453/52707/113160 mbufs in use (current/cache/total) > 48416/51584/100000/100000 mbuf clusters in use (current/cache/total/max) > 2894/690 mbuf+clusters out of packet secondary zone in use (current/cache) > 11946/854/12800/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 164834K/119760K/284595K bytes allocated to network (current/cache/total) > 0/339/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/4/6656 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > # dmesg | grep commit > > At this point RX has hung. > > Somehow the check (i == rxr->next_to_refresh) is never true in this case. > Also, I did read kern/150516 and couldn't wrap my head around the patch for > the em driver that Beezar Liu suggested. > > Regards, > > Karim. > >