From owner-freebsd-net@FreeBSD.ORG Tue Jan 27 16:56:10 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A1A71065785 for ; Tue, 27 Jan 2009 16:56:10 +0000 (UTC) (envelope-from yonyossef.lists@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.170]) by mx1.freebsd.org (Postfix) with ESMTP id 578218FC12 for ; Tue, 27 Jan 2009 16:56:09 +0000 (UTC) (envelope-from yonyossef.lists@gmail.com) Received: by ug-out-1314.google.com with SMTP id 30so216756ugs.39 for ; Tue, 27 Jan 2009 08:56:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:references:subject :date:message-id:mime-version:content-type:content-transfer-encoding :x-mailer:thread-index:x-mimeole:in-reply-to; bh=Thd6iqs5e/8jFUV20L3dnRNNDBQ4gvDyc8fK1hGthvk=; b=YAfK6NMKHmxRV2xMeoB0ZapYIkHYTk/CIMoZkyS1b6Jacs+4ogrtYbyWI1fYb3Bqs9 Jxz/EEAv2UUDRiiMatBDm69vDK4if3zw4aiGhENx+zBu6AhN8erPHjbSAndCqbobfz9U mALb2fPsxdR1TYPo2P0l7UXQsE01dGVwTOdKM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:references:subject:date:message-id:mime-version :content-type:content-transfer-encoding:x-mailer:thread-index :x-mimeole:in-reply-to; b=ijUiX/MLaRxu7RaqiMYKsvGX1bLwmI4tNlCZ+WjjDjRCDIVB4Xwbv6T6KHih69hCR/ HMipRp/Qu6oqXtUwQKh1uA9ykBWibcJXKHLnbuUmlLJakwwVjr+96XMxbG0/azykP5Rv SbDpqbH2fzrjsiwci8NwN6YH4zvN8uCtzd6R0= Received: by 10.67.123.8 with SMTP id a8mr1888270ugn.74.1233075368397; Tue, 27 Jan 2009 08:56:08 -0800 (PST) Received: from mtllpt03 (DSL212-235-20-133.bb.netvision.net.il [212.235.20.133]) by mx.google.com with ESMTPS id a1sm12215896ugf.42.2009.01.27.08.56.05 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 27 Jan 2009 08:56:07 -0800 (PST) From: "Yony Yossef" To: "'Kip Macy'" , "Yony Yossef" References: <000001c97e23$0d81df20$39ed1aac@mtl.com> <3c1674c90901261212l5432b99fpd77bcff96883bd21@mail.gmail.com> Date: Tue, 27 Jan 2009 18:56:02 +0200 Message-ID: <002201c980a0$25903420$220f000a@mtl.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 11 Thread-Index: Acl/8m9FmPRM11MOTJegl6/LJM/lvgAqFJaw X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3350 In-Reply-To: <3c1674c90901261212l5432b99fpd77bcff96883bd21@mail.gmail.com> Cc: johan@nocrew.org, freebsd-net@freebsd.org, Eitan Shefi , Amit Krig , Liran Liss Subject: RE: freebsd 7.0-RELEASE BUG ping: sendto: No buffer space available X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jan 2009 16:56:10 -0000 > "No buffer space available" is ENOBUFS. This error is > returned when the ifq is full - see IFQ_ENQUEUE in if_var.h. > You'll almost certainly want to set ifq_len to larger than > the default for a 10 gigabit driver. This most likely to be a > combination of ifq_len being too small and inadequate txq > overflow handling. However, without the code to look at I can > only guess. > > Cheers, > Kip ---- my Tx loop is pretty normal (I've removed debug calls and irrelevant parts): while (!IFQ_DRV_IS_EMPTY(&dev->if_snd)) { IFQ_DRV_DEQUEUE(&dev->if_snd, m_head); if (m_head == NULL) break; if ( (err = mtnic_xmit_encap(priv, &m_head, &used)) != 0) { priv->ifp->if_drv_flags |= IFF_DRV_OACTIVE; IFQ_LOCK(&dev->if_snd); IFQ_DRV_PREPEND(&dev->if_snd, m_head); IFQ_UNLOCK(&dev->if_snd); break; } ETHER_BPF_MTAP(dev, m_head); } ---- my ifq_max_len is set to 4096, I believe it's bigger than the other 10GigE drivers (cxgb is using 1K, for instance): dev->if_snd.ifq_drv_maxlen = MTNIC_TX_IFQ_MAXLEN; // = 4096 IFQ_SET_MAXLEN(&dev->if_snd, dev->if_snd.ifq_drv_maxlen); IFQ_SET_READY(&dev->if_snd); ---- there's another strange symptom to this hang: after the 20 minutes I've mentioned (caused by the sender full IFQ) I have to ping the sender from the receiver side in order to retain traffic. During the 20 minutes it will not work, but after it passes a single ping will make the sender IFQ go back to life. Yony