From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 17:24:04 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F3E0106564A for ; Mon, 7 Feb 2011 17:24:04 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-px0-f182.google.com (mail-px0-f182.google.com [209.85.212.182]) by mx1.freebsd.org (Postfix) with ESMTP id EB8DE8FC15 for ; Mon, 7 Feb 2011 17:24:03 +0000 (UTC) Received: by pxi1 with SMTP id 1so1106137pxi.13 for ; Mon, 07 Feb 2011 09:24:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=FqS1EVS3oeKwS1D/xf8qUX3jZft0MwbMBoB8+a4sSKo=; b=RnwtbZEsUYtHEB1yJKlneMYgE7uXuimC3cfhaFVyGcVQZy+TV6HV8HQi/8Kh++ejJu j80MGCtBEwoHZGuqS/xUDy6ZXLvI0sLj0mIaJs4HDkLDstGSehq494i+cYJjUZ01IkG4 m+Z08VTGwEYsPa1Pedjj6wZuo2IgRvoM9vppQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=Jw+7BDSbH8GBXmV8eLl+Eh9ftbwjmgdbZLWBL3Nm3B5TQDKXh1gTf2pEL9oyj9p0OC bLQccTR8GhsdfSfH+dX6j+gUbZgX09nuzjWDNGkvfX6V9tZIpeg9+Gue7CO6P1siCi/D u2lZxGNRnMpBafNKnr8RFISuDSHVI9Q06csRA= MIME-Version: 1.0 Received: by 10.142.241.16 with SMTP id o16mr15648508wfh.86.1297097884598; Mon, 07 Feb 2011 08:58:04 -0800 (PST) Received: by 10.142.246.3 with HTTP; Mon, 7 Feb 2011 08:58:04 -0800 (PST) Date: Mon, 7 Feb 2011 11:58:04 -0500 Message-ID: From: Karim Fodil-Lemelin To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: igb driver tx hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 17:24:04 -0000 Hi List, The issue is with the igb driver from 7.4 RC3 r218406. If the driver runs out of mbuf clusters it simply stops receiving even after the clusters have been freed. igb0@pci0:7:0:0: class=0x020000 card=0x00008086 chip=0x10a78086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet Here is how to reproduce: 1) Set mbuf cluster count very low (4096) 2) do an intensive UDP or TCP transfer Some pointers: - After few seconds or minutes maximum, the traffic does not flow any more across the machine. - At this point we see the InErrors jump in a burst on network iterface igb0. Then, all packets in are counted as In Errors. - Their number is exactly equal to dev.igb.0.mac_stats.missed_packets, and not far from dev.igb.0.mac_stats.xoff_txd (Note that one is on receive the other on send). - Also we see this counter goes non-null: dev.igb.0.mac_stats.recv_no_buff - We tried to enable/disable TSO, Checksum on receive and send, flow control and aim. The problem still shows up, only the time to get there is different. - Forcing the driver to use one queue (static int igb_num_queues = 1;) did not help. - Using igb_header_split=TRUE does help a bit but not unlike disabling hardware acceleration it just takes more time (probably because its using mbufs more instead of clusters and there is 4times more mbufs then clusters). - Using larger values for mbufs cluster also delays the problem We think the code in "igb_refresh_mbufs()" does not handle well running out of mbufs but that is how far we can get at the moment. We are seeking your help and wisdom, and are willing to test patches or suggested settings. Thanks in advance, Karim.