From owner-freebsd-net@FreeBSD.ORG Sun Jul 21 13:03:22 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 52A96BFA for ; Sun, 21 Jul 2013 13:03:22 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm16-vm6.bullet.mail.ne1.yahoo.com (nm16-vm6.bullet.mail.ne1.yahoo.com [98.138.91.109]) by mx1.freebsd.org (Postfix) with ESMTP id 05FD7A9E for ; Sun, 21 Jul 2013 13:03:21 +0000 (UTC) Received: from [98.138.90.57] by nm16.bullet.mail.ne1.yahoo.com with NNFMP; 21 Jul 2013 13:03:21 -0000 Received: from [98.138.101.182] by tm10.bullet.mail.ne1.yahoo.com with NNFMP; 21 Jul 2013 13:03:21 -0000 Received: from [127.0.0.1] by omp1093.mail.ne1.yahoo.com with NNFMP; 21 Jul 2013 13:03:21 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 120982.92203.bm@omp1093.mail.ne1.yahoo.com Received: (qmail 21099 invoked by uid 60001); 21 Jul 2013 13:03:21 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1374411801; bh=1qwfArlKpLQXh33e59K9D+kJb381hVxONr4Z6FPKk0M=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=1BEZ4RRAV1XPxSoaD7Fo0G5EeOJM5DRc71nIbfZzKBnR76l6gsRP0kDlV2alOk4Vmadd9Mhul0TKntscptUwFLdDiWD9/SknptLXC4vNqi5IvWapuT1F4I10SivcbckOvQzTApbXuEkXbV9StPlNfvm/i/U/VGGurGkPiV+ycek= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=k1qzilOuF6hUpdd4KoFQOXWQXbGS/duTY2WHr1jzQRkHwO2V7loQDZN9g6GGpqhHfBh+a32FYpC6p+0Dk34KLO4IdNjRsuV35kTn96vbqCH0REL0xDo4V/37IvS5B71jL/DW04Gn26pUZuZY+ft4f0JH6cBtV2uPAad3ypU7rDQ=; X-YMail-OSG: vvRcqIsVM1kXVD4dVWWNRJiyM3z_8zFWVHaD.2XrXHM4fEv QSASVGbZ1LrIczzbn5YRlWRmqmTYsUWC0mLAuAemzn9KSD_mijD.GOIRY_xt 2gHOb4KakOeeLXidx849o3UEdjnj0BAEyapP1t_6m0VkxhSwjeG2QYoT7L4h 4VxEws2ha5uQb0Mrc7yoH92kaq.ezif2bpkv2SYxMFzVREXoO7pEaQOhHN5F 6A3ktrb6psN00DxUNFtvDIOg1q1krDh5HvdeO.qgRNVWbjBDZFZZVV3CHKb0 Q6WZreng8GlSeA.u910m27x3jOFKflVXQvGX6_MV3iQ6Bx1k0paZtG9I9wCW gHmdbYZjmy5vwlNjBWW1KDBuZ0MfX08yeK_pq9JPjYFGHAw9t8Y0B369vCRE ZFr9knhCWDACq1tx4GTiu_aYnJkM7AELmWVevpulexW5bOwnvzdUBEGHxaAt VzNyxKQmRDcCdNFJo0nHCoICw6c6bupz6S72VRAzlC7JsiSYUVy.so0V63Hp np0InfcrhKTlqCdYs6cvbdv9Gv69Ydzpxe5MmU.nORNmMHJfXjiR_E9mMv_k 66dDA3ueCZ.oMVA-- Received: from [98.203.118.124] by web121604.mail.ne1.yahoo.com via HTTP; Sun, 21 Jul 2013 06:03:20 PDT X-Rocket-MIMEInfo: 002.001, DQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KT24gU2F0LCA3LzIwLzEzLCBpc3AgPG1saW5lQHVrci5uZXQ.IHdyb3RlOg0KDQogU3ViamVjdDogTEFDUCBMQUdHIGRldmljZSBwcm9ibGVtcw0KIFRvOiBmcmVlYnNkLW5ldEBmcmVlYnNkLm9yZw0KIERhdGU6IFNhdHVyZGF5LCBKdWx5IDIwLCAyMDEzLCAxMDowNCBBTQ0KIA0KIA0KIA0KIA0KIEhpISBDYW4gYW55Ym9keSB0ZWxsIG1lLCBpcyB0aGVyZSBhbnkgcGxhbnMgdG8gaW1wcm92ZQ0KIExBR0coODAyLjNhZCkNCiABMAEBAQE- X-Mailer: YahooMailClassic/268 YahooMailWebService/0.8.149.560 Message-ID: <1374411800.11157.YahooMailBasic@web121604.mail.ne1.yahoo.com> Date: Sun, 21 Jul 2013 06:03:20 -0700 (PDT) From: Barney Cordoba Subject: Re: LACP LAGG device problems To: freebsd-net@freebsd.org, isp In-Reply-To: <53315.1374329071.4500971523820617728@ffe15.ukr.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jul 2013 13:03:22 -0000 -------------------------------------------- On Sat, 7/20/13, isp wrote: Subject: LACP LAGG device problems To: freebsd-net@freebsd.org Date: Saturday, July 20, 2013, 10:04 AM Hi! Can anybody tell me, is there any plans to improve LAGG(802.3ad) device driver in FreeBSD? It will be greate to have a possibility to set LACP mode (active/passive) and system priority. Also there is no way to set hashing algorithm and master interface (port). And we can't see any information about our neighbor. The same function in Linux is named Bonding and it is much more better. I realy can donate some money to those who can make this improvements. Best regards. > _______________________________________________ Why are you using LAGG when 10g cards are like $350? It's not a peering protocol nor it is PTP; can you see your "peer" info on an ethernet? Bonding is a late 90s concept designed to connect 2 slow links to get higher speeds, back in the day when 100Mb/s was ambitious. The point of LAGG is that it's transparent; you can load balance traffic to multiple hosts or create a redundant link without having to have equipment running some special applications, or any special logic above the LAGG device. Describing how you are using LAGG (and why) might be better than just asking for "improvements". BC From owner-freebsd-net@FreeBSD.ORG Sun Jul 21 13:49:31 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 5F7CCBA3 for ; Sun, 21 Jul 2013 13:49:31 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-x22f.google.com (mail-wg0-x22f.google.com [IPv6:2a00:1450:400c:c00::22f]) by mx1.freebsd.org (Postfix) with ESMTP id EFACAD82 for ; Sun, 21 Jul 2013 13:49:30 +0000 (UTC) Received: by mail-wg0-f47.google.com with SMTP id j13so273079wgh.2 for ; Sun, 21 Jul 2013 06:49:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=Fishv2GrqTtglBjcsQ0ZuQB11fmIKed7wZ4MFbM4QrE=; b=e4dxGdU4vqoahli9cv+xKyhI8rMWRxSFg30jyO2sfadNbG2+7Gg4i505Kg+ZsOGE3e dlG8ARE5ZY4OxFXMEeQKvs+avMHr6RFbEIrwnx8PLBn0fDsdQeNxMaAQSbHeGYzc+RV+ H8/ABgbrjMOiDkmEjIeDfcL//2fhskVNSCPrRuMMErbo/klBcVRmlRJtQNqCz7StPnRC lFbH5H+k5QJQum0q6YZzWIQTlhsQTDzwH6Exvx3t/vjwxsVeOvoTwzLrqjHG1J0W8Rix m2ydYJJCQZWGonFzBpDxqGvQlVMI+UohTj3wlUPXGZ/K3LKNuNVGpcAJyVFdc2DBaC0+ xC7w== MIME-Version: 1.0 X-Received: by 10.180.160.165 with SMTP id xl5mr27039676wib.46.1374414570114; Sun, 21 Jul 2013 06:49:30 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Sun, 21 Jul 2013 06:49:30 -0700 (PDT) In-Reply-To: <1374411800.11157.YahooMailBasic@web121604.mail.ne1.yahoo.com> References: <53315.1374329071.4500971523820617728@ffe15.ukr.net> <1374411800.11157.YahooMailBasic@web121604.mail.ne1.yahoo.com> Date: Sun, 21 Jul 2013 06:49:30 -0700 X-Google-Sender-Auth: 2IELYfvlLb9sLFSM6Sy0NrKIc4Y Message-ID: Subject: Re: LACP LAGG device problems From: Adrian Chadd To: Barney Cordoba Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org, isp X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jul 2013 13:49:31 -0000 Hah! I'm pushing 20GE out using lagg right now (and fixing the er, amusing behaviour of doing so.) I'm aiming to hit 40 once I get hardware that doesn't get upset pushing that many bits. The netops people at ${JOB} also point out that even today switches occasionally get confused and "crash" a switchport. Ew. So yes, there are people using lagg, both for failover and throughput reasons. I'm working on debugging/statistics right now as part of general "why are things behaving crappy" debugging. I'll see about improving some of the peer reporting at the same time. -adrian On 21 July 2013 06:03, Barney Cordoba wrote: > > -------------------------------------------- > On Sat, 7/20/13, isp wrote: > > Subject: LACP LAGG device problems > To: freebsd-net@freebsd.org > Date: Saturday, July 20, 2013, 10:04 AM > > > > > Hi! Can anybody tell me, is there any plans to improve > LAGG(802.3ad) > device driver in FreeBSD? > It will be greate to have a possibility to set LACP mode > (active/passive) > and system priority. > Also there is no way to set hashing algorithm and master > interface > (port). > And we can't see any information about our neighbor. > The same function in Linux is named Bonding and it is much > more better. > I realy can donate some money to those who can make this > improvements. > Best regards. > > > > _______________________________________________ > > Why are you using LAGG when 10g cards are like $350? It's not > a peering protocol nor it is PTP; can you see your "peer" info on > an ethernet? > > Bonding is a late 90s concept designed to connect 2 slow links to > get higher speeds, back in the day when 100Mb/s was ambitious. > The point of LAGG is that it's transparent; you can load balance > traffic to multiple hosts or create a redundant link without having > to have equipment running some special applications, or any special > logic above the LAGG device. > > Describing how you are using LAGG (and why) might be better > than just asking for "improvements". > > BC > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Sun Jul 21 14:40:40 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A90D217F; Sun, 21 Jul 2013 14:40:40 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-ob0-x22f.google.com (mail-ob0-x22f.google.com [IPv6:2607:f8b0:4003:c01::22f]) by mx1.freebsd.org (Postfix) with ESMTP id 6A4BC150; Sun, 21 Jul 2013 14:40:40 +0000 (UTC) Received: by mail-ob0-f175.google.com with SMTP id xn12so7112586obc.20 for ; Sun, 21 Jul 2013 07:40:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=RMbM7maYRHx2zrnIyfSJc6fK/1FHS73H+dRlP2ZVhkk=; b=DW2XbbyBy6cHJ9d1GZgliNVcgAv5sWes8Z5UayFZwNoQ46ILguaqKGcXeJVV0VVT4B W1doK4d5xFNf/QVYvcpSkZzk87AlRSIe6NLGCthh4GUGqP7DaUFku6Q0XDoP+NFcLoPg KfkLjnHzhILPchYTReISuQ8fgjUHcIEac3jMdGu9zoAADbK4Ly2QAvUCmqmxO6fV2bxF o5CI5KH9IGM3EAVqkbS8/RMJ1wqH/3mn9dx9oCYbNMl412o2PN0eCYRYIK5LAdgGe1yw JtOOfsOrNW+VoNMSdfKVZ7DCJxzVWthzOAWypciHDaxz/xakAMlUYhJJ4splEIh90PvC I0sw== MIME-Version: 1.0 X-Received: by 10.60.52.165 with SMTP id u5mr23670422oeo.15.1374417640014; Sun, 21 Jul 2013 07:40:40 -0700 (PDT) Sender: kob6558@gmail.com Received: by 10.76.112.212 with HTTP; Sun, 21 Jul 2013 07:40:39 -0700 (PDT) In-Reply-To: References: <53315.1374329071.4500971523820617728@ffe15.ukr.net> <1374411800.11157.YahooMailBasic@web121604.mail.ne1.yahoo.com> Date: Sun, 21 Jul 2013 07:40:39 -0700 X-Google-Sender-Auth: SADtlXrLRg43Tk_JkkJzEBcBSuM Message-ID: Subject: Re: LACP LAGG device problems From: Kevin Oberman To: Adrian Chadd Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Barney Cordoba , isp , freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jul 2013 14:40:40 -0000 On Sun, Jul 21, 2013 at 6:49 AM, Adrian Chadd wrote: > Hah! > > I'm pushing 20GE out using lagg right now (and fixing the er, amusing > behaviour of doing so.) I'm aiming to hit 40 once I get hardware that > doesn't get upset pushing that many bits. The netops people at ${JOB} > also point out that even today switches occasionally get confused and > "crash" a switchport. Ew. > > So yes, there are people using lagg, both for failover and throughput > reasons. > > I'm working on debugging/statistics right now as part of general "why > are things behaving crappy" debugging. I'll see about improving some > of the peer reporting at the same time. > > > > -adrian > > On 21 July 2013 06:03, Barney Cordoba wrote: > > > > -------------------------------------------- > > On Sat, 7/20/13, isp wrote: > > > > Subject: LACP LAGG device problems > > To: freebsd-net@freebsd.org > > Date: Saturday, July 20, 2013, 10:04 AM > > > > > > > > > > Hi! Can anybody tell me, is there any plans to improve > > LAGG(802.3ad) > > device driver in FreeBSD? > > It will be greate to have a possibility to set LACP mode > > (active/passive) > > and system priority. > > Also there is no way to set hashing algorithm and master > > interface > > (port). > > And we can't see any information about our neighbor. > > The same function in Linux is named Bonding and it is much > > more better. > > I realy can donate some money to those who can make this > > improvements. > > Best regards. > > > > > > > _______________________________________________ > > > > Why are you using LAGG when 10g cards are like $350? It's not > > a peering protocol nor it is PTP; can you see your "peer" info on > > an ethernet? > > > > Bonding is a late 90s concept designed to connect 2 slow links to > > get higher speeds, back in the day when 100Mb/s was ambitious. > > The point of LAGG is that it's transparent; you can load balance > > traffic to multiple hosts or create a redundant link without having > > to have equipment running some special applications, or any special > > logic above the LAGG device. > > > > Describing how you are using LAGG (and why) might be better > > than just asking for "improvements". > > > > BC > I am aware of at least one case where 100G WAN links are being LAGGed today. Only two ATM, but 4x100G is on the horizon. I suspect 4x100G or even more is already in place in the data center, but I have no actual knowledge. and 100G i still quite a bit more than $320 per port. And that is ignoring the cost of 100G routing, switching, and optical gear. LAGG is not going away any time soon. I'm sure we will see Nx400G as soon as 400G Ethernet is available. -- R. Kevin Oberman, Network Engineer E-mail: rkoberman@gmail.com From owner-freebsd-net@FreeBSD.ORG Sun Jul 21 14:59:44 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DBA2184C for ; Sun, 21 Jul 2013 14:59:44 +0000 (UTC) (envelope-from mline@ukr.net) Received: from ffe10.ukr.net (ffe10.ukr.net [195.214.192.60]) by mx1.freebsd.org (Postfix) with ESMTP id 91C9C26C for ; Sun, 21 Jul 2013 14:59:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Date:Message-Id:From:To:References:In-Reply-To:Subject:Cc:Content-Type:Content-Transfer-Encoding:MIME-Version; bh=+HlehALKRs3gsVuVgc0ZWsaTw/2AhlEnRcV5xd5q86c=; b=w2QslR8UWviYtkUb/ypms02GK3YQZGtmYQELTrDnJ97RHqTzSl3Ejvs7F6pXiJUcL0IgS2oGTB9tvYohuDU72yw3D3wwRBrW7f7fgh+GTPyydd6AoJ8lHKSrtZZ2NjbYB1BLknHMB1biTAJc1+Gs0xDl/LnRm34+Z002HSuHUnY=; Received: from mail by ffe10.ukr.net with local ID 1V0uqU-000IjY-88 ; Sun, 21 Jul 2013 17:42:46 +0300 MIME-Version: 1.0 Subject: Re[2]: LACP LAGG device problems In-Reply-To: <1374411800.11157.YahooMailBasic@web121604.mail.ne1.yahoo.com> References: <1374411800.11157.YahooMailBasic@web121604.mail.ne1.yahoo.com> To: "Barney Cordoba" From: "isp" X-Mailer: freemail.ukr.net 4.0 Message-Id: <55306.1374417766.4134591960826904576@ffe10.ukr.net> Date: Sun, 21 Jul 2013 17:42:46 +0300 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: binary Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jul 2013 14:59:44 -0000 I'm using LACP becasue: 1. 10G cards are much more expensive than 350$. They cost nearly 700$-800$. 2. I don't have switches with 10G interfaces. They are also very expensive. --- Message --- From: "Barney Cordoba" Date: 21 July 2013, 16:03:23 -------------------------------------------- On Sat, 7/20/13, isp < mline@ukr.net > wrote: Subject: LACP LAGG device problems To: freebsd-net@freebsd.org Date: Saturday, July 20, 2013, 10:04 AM Hi! Can anybody tell me, is there any plans to improve LAGG(802.3ad) device driver in FreeBSD? It will be greate to have a possibility to set LACP mode (active/passive) and system priority. Also there is no way to set hashing algorithm and master interface (port). And we can't see any information about our neighbor. The same function in Linux is named Bonding and it is much more better. I realy can donate some money to those who can make this improvements. Best regards. > _______________________________________________ Why are you using LAGG when 10g cards are like $350? It's not a peering protocol nor it is PTP; can you see your "peer" info on an ethernet? Bonding is a late 90s concept designed to connect 2 slow links to get higher speeds, back in the day when 100Mb/s was ambitious. The point of LAGG is that it's transparent; you can load balance traffic to multiple hosts or create a redundant link without having to have equipment running some special applications, or any special logic above the LAGG device. Describing how you are using LAGG (and why) might be better than just asking for "improvements". BC From owner-freebsd-net@FreeBSD.ORG Sun Jul 21 16:23:49 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B8497D77 for ; Sun, 21 Jul 2013 16:23:49 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm19-vm1.bullet.mail.ne1.yahoo.com (nm19-vm1.bullet.mail.ne1.yahoo.com [98.138.91.56]) by mx1.freebsd.org (Postfix) with ESMTP id 4D2548FA for ; Sun, 21 Jul 2013 16:23:49 +0000 (UTC) Received: from [98.138.101.129] by nm19.bullet.mail.ne1.yahoo.com with NNFMP; 21 Jul 2013 16:23:42 -0000 Received: from [98.138.87.1] by tm17.bullet.mail.ne1.yahoo.com with NNFMP; 21 Jul 2013 16:23:42 -0000 Received: from [127.0.0.1] by omp1001.mail.ne1.yahoo.com with NNFMP; 21 Jul 2013 16:23:42 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 208830.11913.bm@omp1001.mail.ne1.yahoo.com Received: (qmail 84742 invoked by uid 60001); 21 Jul 2013 16:23:42 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1374423822; bh=QdS+SDtez+8N0+YASxnAsQV2CfqUisW6izDGYCBMbsQ=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=GpJPA68lEemEnjuH0MAW1jYJBpqO1hy5SohxnfZNNt4641VF60VuILdgvOqokoTUMbI9uF5loR80JqCyYGS2Z180awl6Q+PGdiQyXfqL293aYDKznmLb4Jk4An2OKHLrc2kPQekmuX4zeu2jRb1pqMJUGszd3aabmyoqQO0ar/M= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=e04/scy4Z3+aSoxNgwB5l3EtUVjaUGviziZQ5rvlFZPczpkTIo+97OA0IZw8CpjeUnSOBH3k1paS9WU4dJLjZMHmWze1kh1jNhTteBUfpT3UGI6nm0fblLtBu1XwelTzAoZF7ciovh/FXMbMwGU99jk5/P+klITjIuQwoyfUuTA=; X-YMail-OSG: iluIogAVM1nKyULvWAUJaVeyeTSZ7LWVBeMdKwL02ShCkxd BV9nco4uVX2orzPA- Received: from [98.203.118.124] by web121604.mail.ne1.yahoo.com via HTTP; Sun, 21 Jul 2013 09:23:41 PDT X-Rocket-MIMEInfo: 002.001, SSB3YXNuJ3QgcmVmZXJyaW5nIHRvIHNjaWVuY2UgcHJvamVjdHMuIE5vciBkaWQgSSBzYXkgaXQgd2Fzbid0IHVzZWZ1bC4KT25seSB0aGF0IDEwZyBpcyBjaGVhcCBub3cgYW5kIHF1aXRlIGEgYml0IGJldHRlci4gTEFHRyBpc24ndCBwZXJmZWN0LgoKCi0tLS0tIE9yaWdpbmFsIE1lc3NhZ2UgLS0tLS0KRnJvbTogQWRyaWFuIENoYWRkIDxhZHJpYW5AZnJlZWJzZC5vcmc.ClRvOiBCYXJuZXkgQ29yZG9iYSA8YmFybmV5X2NvcmRvYmFAeWFob28uY29tPgpDYzogZnJlZWJzZC1uZXRAZnJlZWJzZC5vcmc7IGkBMAEBAQE- X-Mailer: YahooMailWebService/0.8.149.560 References: <53315.1374329071.4500971523820617728@ffe15.ukr.net> <1374411800.11157.YahooMailBasic@web121604.mail.ne1.yahoo.com> Message-ID: <1374423821.78377.YahooMailNeo@web121604.mail.ne1.yahoo.com> Date: Sun, 21 Jul 2013 09:23:41 -0700 (PDT) From: Barney Cordoba Subject: Re: LACP LAGG device problems To: Adrian Chadd In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Barney Cordoba List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jul 2013 16:23:49 -0000 I wasn't referring to science projects. Nor did I say it wasn't useful.=0AO= nly that 10g is cheap now and quite a bit better. LAGG isn't perfect.=0A=0A= =0A----- Original Message -----=0AFrom: Adrian Chadd = =0ATo: Barney Cordoba =0ACc: freebsd-net@freebsd.= org; isp =0ASent: Sunday, July 21, 2013 9:49 AM=0ASubject: R= e: LACP LAGG device problems=0A=0AHah!=0A=0AI'm pushing 20GE out using lagg= right now (and fixing the er, amusing=0Abehaviour of doing so.) I'm aiming= to hit 40 once I get hardware that=0Adoesn't get upset pushing that many b= its. The netops people at ${JOB}=0Aalso point out that even today switches = occasionally get confused and=0A"crash" a switchport. Ew.=0A=0ASo yes, ther= e are people using lagg, both for failover and throughput reasons.=0A=0AI'm= working on debugging/statistics right now as part of general "why=0Aare th= ings behaving crappy" debugging. I'll see about improving some=0Aof the pee= r reporting at the same time.=0A=0A=0A=0A-adrian=0A=0AOn 21 July 2013 06:03= , Barney Cordoba wrote:=0A>=0A> ----------------= ----------------------------=0A> On Sat, 7/20/13, isp wrote= :=0A>=0A>=A0 Subject: LACP LAGG device problems=0A>=A0 To: freebsd-net@free= bsd.org=0A>=A0 Date: Saturday, July 20, 2013, 10:04 AM=0A>=0A>=0A>=0A>=0A>= =A0 Hi! Can anybody tell me, is there any plans to improve=0A>=A0 LAGG(802.= 3ad)=0A>=A0 device driver in FreeBSD?=0A>=A0 It will be greate to have a po= ssibility to set LACP mode=0A>=A0 (active/passive)=0A>=A0 and system priori= ty.=0A>=A0 Also there is no way to set hashing algorithm and master=0A>=A0 = interface=0A>=A0 (port).=0A>=A0 And we can't see any information about our = neighbor.=0A>=A0 The same function in Linux is named Bonding and it is much= =0A>=A0 more better.=0A>=A0 I realy can donate some money to those who can = make this=0A>=A0 improvements.=0A>=A0 Best regards.=0A>=0A>=A0 >=0A>=A0 ___= ____________________________________________=0A>=0A> Why are you using LAGG= when 10g cards are like $350? It's not=0A> a peering protocol nor it is PT= P; can you see your "peer" info on=0A> an ethernet?=0A>=0A> Bonding is a la= te 90s concept designed to connect 2 slow links to=0A> get higher speeds, b= ack in the day when 100Mb/s was ambitious.=0A> The point of LAGG is that it= 's transparent; you can load balance=0A> traffic to multiple hosts or creat= e a redundant link without having=0A> to have equipment running some specia= l applications, or any special=0A> logic above the LAGG device.=0A>=0A> Des= cribing how you are using LAGG (and why) might be better=0A> than just aski= ng for "improvements".=0A>=0A> BC=0A>=0A> _________________________________= ______________=0A> freebsd-net@freebsd.org mailing list=0A> http://lists.fr= eebsd.org/mailman/listinfo/freebsd-net=0A> To unsubscribe, send any mail to= "freebsd-net-unsubscribe@freebsd.org"=0A From owner-freebsd-net@FreeBSD.ORG Sun Jul 21 18:31:38 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 159F2D2A for ; Sun, 21 Jul 2013 18:31:38 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-we0-x236.google.com (mail-we0-x236.google.com [IPv6:2a00:1450:400c:c03::236]) by mx1.freebsd.org (Postfix) with ESMTP id A3A28C87 for ; Sun, 21 Jul 2013 18:31:37 +0000 (UTC) Received: by mail-we0-f182.google.com with SMTP id u55so510939wes.13 for ; Sun, 21 Jul 2013 11:31:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=LzuU8tmd2HO/uC/+D3GsaHFFWo7884ROuqmVwqh9SPY=; b=Ou+siQMSn2nzNOtUd7AY/FTOv1e/M3M+M1+SZkIP65JRC3I3A6j8LRO4mo0g0TdjgW R0JD21pP979XacUclspOHQbC1B3QsheZg4wUkwNT7XZjM0NrtxHm5W8Z8KFDhkZZL6Uw p/u5Rs4mpLVmZPuSM3fRU0uWYNrNDJDV3dPHKyv6iO8kgwDG927ccl9+yq6xgsN+WkpU QKGaMIcSZXJJDrfGc4SKUPcwcS3cl1xs8753XrB+vldMO/rGk2pKyisH6AtSJWcUJnQC JJM4Ye6t7y6NZzg93IPhcLUhXGrY/cNOec7Y/b60/Zk4srNzxCNQYFVuu+o33w3MLnBt WFEg== MIME-Version: 1.0 X-Received: by 10.194.63.229 with SMTP id j5mr16868408wjs.79.1374431496865; Sun, 21 Jul 2013 11:31:36 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Sun, 21 Jul 2013 11:31:36 -0700 (PDT) In-Reply-To: <1374423821.78377.YahooMailNeo@web121604.mail.ne1.yahoo.com> References: <53315.1374329071.4500971523820617728@ffe15.ukr.net> <1374411800.11157.YahooMailBasic@web121604.mail.ne1.yahoo.com> <1374423821.78377.YahooMailNeo@web121604.mail.ne1.yahoo.com> Date: Sun, 21 Jul 2013 11:31:36 -0700 X-Google-Sender-Auth: rypFG0nHiwk7ZsaolBwuklNbP74 Message-ID: Subject: Re: LACP LAGG device problems From: Adrian Chadd To: Barney Cordoba Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jul 2013 18:31:38 -0000 Barney, I now work at netflix. We push >10gig per box. I'm working on making that much, much more than 10gig. It's not a "science project." sheesh. :-) -adrian On 21 July 2013 09:23, Barney Cordoba wrote: > I wasn't referring to science projects. Nor did I say it wasn't useful. > Only that 10g is cheap now and quite a bit better. LAGG isn't perfect. > > > ----- Original Message ----- > From: Adrian Chadd > To: Barney Cordoba > Cc: freebsd-net@freebsd.org; isp > Sent: Sunday, July 21, 2013 9:49 AM > Subject: Re: LACP LAGG device problems > > Hah! > > I'm pushing 20GE out using lagg right now (and fixing the er, amusing > behaviour of doing so.) I'm aiming to hit 40 once I get hardware that > doesn't get upset pushing that many bits. The netops people at ${JOB} > also point out that even today switches occasionally get confused and > "crash" a switchport. Ew. > > So yes, there are people using lagg, both for failover and throughput reasons. > > I'm working on debugging/statistics right now as part of general "why > are things behaving crappy" debugging. I'll see about improving some > of the peer reporting at the same time. > > > > -adrian > > On 21 July 2013 06:03, Barney Cordoba wrote: >> >> -------------------------------------------- >> On Sat, 7/20/13, isp wrote: >> >> Subject: LACP LAGG device problems >> To: freebsd-net@freebsd.org >> Date: Saturday, July 20, 2013, 10:04 AM >> >> >> >> >> Hi! Can anybody tell me, is there any plans to improve >> LAGG(802.3ad) >> device driver in FreeBSD? >> It will be greate to have a possibility to set LACP mode >> (active/passive) >> and system priority. >> Also there is no way to set hashing algorithm and master >> interface >> (port). >> And we can't see any information about our neighbor. >> The same function in Linux is named Bonding and it is much >> more better. >> I realy can donate some money to those who can make this >> improvements. >> Best regards. >> >> > >> _______________________________________________ >> >> Why are you using LAGG when 10g cards are like $350? It's not >> a peering protocol nor it is PTP; can you see your "peer" info on >> an ethernet? >> >> Bonding is a late 90s concept designed to connect 2 slow links to >> get higher speeds, back in the day when 100Mb/s was ambitious. >> The point of LAGG is that it's transparent; you can load balance >> traffic to multiple hosts or create a redundant link without having >> to have equipment running some special applications, or any special >> logic above the LAGG device. >> >> Describing how you are using LAGG (and why) might be better >> than just asking for "improvements". >> >> BC >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Sun Jul 21 18:53:01 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 83FA4420 for ; Sun, 21 Jul 2013 18:53:01 +0000 (UTC) (envelope-from scottl@netflix.com) Received: from mail-ye0-x22a.google.com (mail-ye0-x22a.google.com [IPv6:2607:f8b0:4002:c04::22a]) by mx1.freebsd.org (Postfix) with ESMTP id 414F1D56 for ; Sun, 21 Jul 2013 18:53:01 +0000 (UTC) Received: by mail-ye0-f170.google.com with SMTP id q3so1901012yen.1 for ; Sun, 21 Jul 2013 11:53:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netflix.com; s=google; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:x-mailer:from:subject:date :to; bh=uaQXAlR6X810zICDpzN3NuZa+USk1J6QNaKYFcc+bNI=; b=fno7eZvWDGfgTpQJ3MsIonPeJk8YQDjubCWOGrftShpRYG6ZT+aJ+zSXqlmDUjnAwt 4QWBs70VofM2gjWQ43ewtE0Twf727zsUWuVKRQnkkRtg9Ss+jUoaY2Ij7V8Jvuu4rlIi rttlc7AWhm598k9eEapdTqg+UELhXwGT+EItQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:x-mailer:from:subject:date :to:x-gm-message-state; bh=uaQXAlR6X810zICDpzN3NuZa+USk1J6QNaKYFcc+bNI=; b=V+T1mACiVOPOvr8kJnDgEapAzG1UC9nA4fLLvoXQgzQAgvaWBOL6+fVh+EARu6bsFR MKxYNdqv60b9tncudAcU7jRyikP1HjbEfZ9RlM+x6dLqSZmsGXp2u0DRAS7zQgszFR23 i2HQLKjiiiLEIcuT6E0BXbTqbUEmp+vheXMt5YBLkUCGE1UiHRjPSUP3caHI6N7vuAiJ b8UxhkBJRoZprnMIdpOtvAR41GTTZzsmVWANPuq/AG62mV3yOHee/P2n0+ISddkarVYQ /fcTlsiKGcbq5T3jLkia7Izg62KmysO7SCoD/AQrc1c2DZdYk5B/dJwkIiVqpIxz6nQa hv4Q== X-Received: by 10.236.78.1 with SMTP id f1mr12471019yhe.29.1374432780790; Sun, 21 Jul 2013 11:53:00 -0700 (PDT) Received: from ?IPv6:2600:1008:b01a:17dc:e5b9:1236:3f68:67a7? ([2600:1008:b01a:17dc:e5b9:1236:3f68:67a7]) by mx.google.com with ESMTPSA id s47sm34695200yhl.19.2013.07.21.11.52.59 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 21 Jul 2013 11:53:00 -0700 (PDT) References: <53315.1374329071.4500971523820617728@ffe15.ukr.net> <1374411800.11157.YahooMailBasic@web121604.mail.ne1.yahoo.com> <1374423821.78377.YahooMailNeo@web121604.mail.ne1.yahoo.com> Mime-Version: 1.0 (1.0) In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: <541D7415-70A4-4374-8B8D-FD80A2F03208@netflix.com> X-Mailer: iPhone Mail (10B350) From: Scott Long Subject: Re: LACP LAGG device problems Date: Sun, 21 Jul 2013 13:52:57 -0500 To: Adrian Chadd X-Gm-Message-State: ALoCoQlG+5xkybArGdJRGYRnSbh6YXqeHq2yg0yL1GBRj69op1ermLrHDCTrZqyluW1+2bhf8JKw Cc: Barney Cordoba , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jul 2013 18:53:01 -0000 Adrian, you're killing my spam filter! But yes, our use of FreeBSD at Netfl= ix is hardly a science project. Http://openconnect.netflix.com Scott On Jul 21, 2013, at 1:31 PM, Adrian Chadd wrote: > Barney, >=20 > I now work at netflix. We push >10gig per box. I'm working on making > that much, much more than 10gig. It's not a "science project." >=20 > sheesh. :-) >=20 >=20 >=20 > -adrian >=20 > On 21 July 2013 09:23, Barney Cordoba wrote: >> I wasn't referring to science projects. Nor did I say it wasn't useful. >> Only that 10g is cheap now and quite a bit better. LAGG isn't perfect. >>=20 >>=20 >> ----- Original Message ----- >> From: Adrian Chadd >> To: Barney Cordoba >> Cc: freebsd-net@freebsd.org; isp >> Sent: Sunday, July 21, 2013 9:49 AM >> Subject: Re: LACP LAGG device problems >>=20 >> Hah! >>=20 >> I'm pushing 20GE out using lagg right now (and fixing the er, amusing >> behaviour of doing so.) I'm aiming to hit 40 once I get hardware that >> doesn't get upset pushing that many bits. The netops people at ${JOB} >> also point out that even today switches occasionally get confused and >> "crash" a switchport. Ew. >>=20 >> So yes, there are people using lagg, both for failover and throughput rea= sons. >>=20 >> I'm working on debugging/statistics right now as part of general "why >> are things behaving crappy" debugging. I'll see about improving some >> of the peer reporting at the same time. >>=20 >>=20 >>=20 >> -adrian >>=20 >> On 21 July 2013 06:03, Barney Cordoba wrote: >>>=20 >>> -------------------------------------------- >>> On Sat, 7/20/13, isp wrote: >>>=20 >>> Subject: LACP LAGG device problems >>> To: freebsd-net@freebsd.org >>> Date: Saturday, July 20, 2013, 10:04 AM >>>=20 >>>=20 >>>=20 >>>=20 >>> Hi! Can anybody tell me, is there any plans to improve >>> LAGG(802.3ad) >>> device driver in FreeBSD? >>> It will be greate to have a possibility to set LACP mode >>> (active/passive) >>> and system priority. >>> Also there is no way to set hashing algorithm and master >>> interface >>> (port). >>> And we can't see any information about our neighbor. >>> The same function in Linux is named Bonding and it is much >>> more better. >>> I realy can donate some money to those who can make this >>> improvements. >>> Best regards. >>>=20 >>> _______________________________________________ >>>=20 >>> Why are you using LAGG when 10g cards are like $350? It's not >>> a peering protocol nor it is PTP; can you see your "peer" info on >>> an ethernet? >>>=20 >>> Bonding is a late 90s concept designed to connect 2 slow links to >>> get higher speeds, back in the day when 100Mb/s was ambitious. >>> The point of LAGG is that it's transparent; you can load balance >>> traffic to multiple hosts or create a redundant link without having >>> to have equipment running some special applications, or any special >>> logic above the LAGG device. >>>=20 >>> Describing how you are using LAGG (and why) might be better >>> than just asking for "improvements". >>>=20 >>> BC >>>=20 >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Sun Jul 21 19:15:13 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 848DACBF for ; Sun, 21 Jul 2013 19:15:13 +0000 (UTC) (envelope-from sthaug@nethelp.no) Received: from bizet.nethelp.no (bizet.nethelp.no [195.1.209.33]) by mx1.freebsd.org (Postfix) with SMTP id CCE85E2C for ; Sun, 21 Jul 2013 19:15:12 +0000 (UTC) Received: (qmail 18234 invoked from network); 21 Jul 2013 19:08:31 -0000 Received: from bizet.nethelp.no (HELO localhost) (195.1.209.33) by bizet.nethelp.no with SMTP; 21 Jul 2013 19:08:31 -0000 Date: Sun, 21 Jul 2013 21:08:31 +0200 (CEST) Message-Id: <20130721.210831.74736231.sthaug@nethelp.no> To: scottl@netflix.com Subject: Re: LACP LAGG device problems From: sthaug@nethelp.no In-Reply-To: <541D7415-70A4-4374-8B8D-FD80A2F03208@netflix.com> References: <1374423821.78377.YahooMailNeo@web121604.mail.ne1.yahoo.com> <541D7415-70A4-4374-8B8D-FD80A2F03208@netflix.com> X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: barney_cordoba@yahoo.com, adrian@freebsd.org, freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jul 2013 19:15:13 -0000 > Adrian, you're killing my spam filter! But yes, our use of FreeBSD at Netflix is hardly a science project. Http://openconnect.netflix.com With my ISP hat on, I expect us to continue using LAGG with 10G member links for many years - simply because 100G is so expensive. One can hope that CFP2 and CFP4 will improve matters somewhat (though it won't help much for the DWDM transport side). Steinar Haug, Nethelp consulting, sthaug@nethelp.no From owner-freebsd-net@FreeBSD.ORG Mon Jul 22 05:22:36 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id F1EFE96 for ; Mon, 22 Jul 2013 05:22:35 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-vb0-x22a.google.com (mail-vb0-x22a.google.com [IPv6:2607:f8b0:400c:c02::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B4C3A226B for ; Mon, 22 Jul 2013 05:22:35 +0000 (UTC) Received: by mail-vb0-f42.google.com with SMTP id i3so4492123vbh.15 for ; Sun, 21 Jul 2013 22:22:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Ct2VfUDf4kYk44plOhlKU8Nf8Yjdt1YiZwgA0T5/yOw=; b=wXT2Cf4Y6CsybR0opZEuTvWV+p69Ut6Ucpg2HertOnEQJaNQpHT5hoOrLAC17ZiUT6 zAceQmhyhmQjR5OA2wJ4HYDxHdWt/7QRgWjXUpK/goGc+l1Xrto4Ths/n7TRd20H0oq4 YSfBmXUv4r9KYQs0eHakJMp0tEzvfXna2XIxg7EcC0OlNUmXqaObS8deTR1p0Q/06OjN xvpe3bSj/eQpDLWJjOwtEP6Kce34xLClVtNeOtm3Rfyih0NfsCtuocbHihmXUOEfzH6/ ZV3wL0s/ermkcpPeaEOHiane56/osNYbE3kKf9sKizQPN2ZmWFUcbw63LoJSamUxhqVX oAwg== MIME-Version: 1.0 X-Received: by 10.58.67.9 with SMTP id j9mr9012160vet.22.1374470554772; Sun, 21 Jul 2013 22:22:34 -0700 (PDT) Received: by 10.221.22.199 with HTTP; Sun, 21 Jul 2013 22:22:34 -0700 (PDT) In-Reply-To: References: <06BA4BD5-BE4E-4184-AFBB-D7FD4B2597D9@your.org> Date: Mon, 22 Jul 2013 01:22:34 -0400 Message-ID: Subject: Re: Duplicate Address Detection misfire? From: Zaphod Beeblebrox To: Kevin Day Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 05:22:36 -0000 I've further narrowed this down. According to the output: em0 DAD detected duplicate IPv6 address fe80:2::250:56ff:fe2e:45fd: NS in/out=2/1, NA in=0 ... FreeBSD *thinks* it has transmitted one and received 2 solicitations. The packet dump shows two solicitations (which would, if it were not bogus, indicate that another machine was booting at the exact same time trying to use the same link-local address). The question becomes: is vmware duplicating the packet, or is FreeBSD? IE: driver problem with em0 and vmware? I'm not completely sure how to debug this. On Thu, Jul 18, 2013 at 9:21 PM, Zaphod Beeblebrox wrote: > > > > On Sun, Jun 30, 2013 at 10:39 PM, Kevin Day wrote: > >> >> On Jun 30, 2013, at 6:48 PM, Zaphod Beeblebrox wrote: >> >> > I have a FreeBSD 9.1-RELEASE vmware guest running. It is using the >> > "bridged" type of networking with VMWare. It gets it's IPv4 address >> from >> > DHCP (successfully) and then fails to initialize IPv6. The relevant >> > rc.conf is: >> > >> > ipv6_activate_all_interfaces="YES" >> > ifconfig_em0_ipv6="inet6 accept_rtadv" >> > ip6addrctl_verbose="YES" >> > >> > The console output says: >> > >> > em0: DAD detected duplicate IPv6 address fe80:2::20c:29ff:fe0a:3989: NS >> > in/out=2/1, NA in=0 >> > em0: DAD complete for fe80:2::20c:29ff:fe0a:3989 - duplicate found >> > em0: manual intervention required >> > em0: possible hardware address duplication deteted, disable IPv6 >> > >> > And subsequently, em0's nd6 has "IFDISABLED" in it. >> > >> > With wireshark, I see two ICMPv6 neighbor solicitations that are >> identical >> > --- is this the problem? >> > >> > How do I fix this? >> >> Did you copy this VM and have both copies running at once? If so, it >> assigned the same MAC address to each VM. >> >> VMware is suppose to detect this and ask if you "copied" or "moved" the >> VM, and if you say "copied" it will randomly assign a new MAC to the VM. If >> this didn't happen or if you said "moved" when you actually copied it, just >> go in and delete/re-create the network interface in the VM's settings to >> create a new MAC for it. >> >> If that's not the issue, we'd probably need more details about your >> configuration. >> > > To further diagnose, there is only one VM running. To ensure that there > were no duplicates, I reassigned the MAC address in the VMWare > configuration dialogue. Additionally, I tried stopping rtadvd on my router > (no effect) and I tried putting the guest on a "host-only network" > (basically isolated it) --- this clears the problem --- both the link-local > and the static address are assigned. > > Frustrated, I dumped the windows interface that is bridged to the VMWare > guest. When it boots, I see the following: > > 2461 19:24:16.376027000 Vmware_2e:46:fd Broadcast ARP 42 > Gratuitous ARP for 66.96.20.42 (Request) > 2462 19:24:16.388241000 :: ff02::1:ff00:42 ICMPv6 78 > Neighbor Solicitation for 2001:1928:1::42 > 2463 19:24:16.389065000 :: ff02::1:ff00:42 ICMPv6 78 > Neighbor Solicitation for 2001:1928:1::42 > 2464 19:24:16.444130000 :: ff02::16 ICMPv6 130 Multicast > Listener Report Message v2 > 2465 19:24:16.444605000 :: ff02::16 ICMPv6 130 Multicast > Listener Report Message v2 > 2466 19:24:16.594663000 :: ff02::1:ff2e:46fd ICMPv6 78 > Neighbor Solicitation for fe80::250:56ff:fe2e:46fd > 2467 19:24:16.595179000 :: ff02::1:ff2e:46fd ICMPv6 78 > Neighbor Solicitation for fe80::250:56ff:fe2e:46fd > 2753 19:24:22.274728000 Vmware_2e:46:fd Broadcast ARP 42 > Who has 66.96.20.33? Tell 66.96.20.42 > 2754 19:24:22.274902000 Intel_bc:6f:87 Vmware_2e:46:fd ARP 60 > 66.96.20.33 is at 00:0e:0c:bc:6f:87 > > ... and then it goes on to chatter ipv4-wise as expected. Note that there > are two of each packet. Is that normal? The ethernet source of all these > packets is my vmware guest (save the who-has reply that I copied in). > > From owner-freebsd-net@FreeBSD.ORG Mon Jul 22 11:06:48 2013 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id BD1DB496 for ; Mon, 22 Jul 2013 11:06:48 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AD66324A3 for ; Mon, 22 Jul 2013 11:06:48 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6MB6m3d053770 for ; Mon, 22 Jul 2013 11:06:48 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6MB6mqR053768 for freebsd-net@FreeBSD.org; Mon, 22 Jul 2013 11:06:48 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 22 Jul 2013 11:06:48 GMT Message-Id: <201307221106.r6MB6mqR053768@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-net@FreeBSD.org Subject: Current problem reports assigned to freebsd-net@FreeBSD.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 11:06:48 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- s kern/180468 net [request] LOCAL_PEERCRED support for PF_INET o kern/180430 net [ofed] [patch] Bad UDP checksum calc for fragmented pa o kern/180065 net [netinet6] [patch] Multicast loopback to own host brok p kern/179975 net [igb] igb(4) fails to do polling(4) o kern/179926 net [lacp] [patch] active aggregator selection bug o kern/179824 net [ixgbe] System (9.1-p4) hangs on heavy ixgbe network t o kern/179733 net [lagg] [patch] interface loses capabilities when proto o kern/179429 net [tap] STP enabled tap bridge o kern/179299 net [igb] Intel X540-T2 - unstable driver a kern/179264 net [vimage] [pf] Core dump with Packet filter and VIMAGE o kern/178947 net [arp] arp rejecting not working o kern/178782 net [ixgbe] 82599EB SFP does not work with passthrough und o kern/178612 net [run] kernel panic due the problems with run driver o kern/178472 net [ip6] [patch] make return code consistent with IPv4 co o kern/178079 net [tcp] Switching TCP CC algorithm panics on sparc64 wit s kern/178071 net FreeBSD unable to recongize Kontron (Industrial Comput o kern/177905 net [xl] [panic] ifmedia_set when pluging CardBus LAN card o kern/177618 net [bridge] Problem with bridge firewall with trunk ports o kern/177417 net [ip6] Invalid protocol value in ipsec6_common_input_cb o kern/177402 net [igb] [pf] problem with ethernet driver igb + pf / alt o kern/177400 net [jme] JMC25x 1000baseT establishment issues o kern/177366 net [ieee80211] negative malloc(9) statistics for 80211nod f kern/177362 net [netinet] [patch] Wrong control used to return TOS o kern/177194 net [netgraph] Unnamed netgraph nodes for vlan interfaces o kern/177139 net [igb] igb drops ethernet ports 2 and 3 o kern/176884 net [re] re0 flapping up/down o kern/176671 net [epair] MAC address for epair device not unique o kern/176484 net [ipsec] [enc] [patch] panic: IPsec + enc(4); device na o kern/176446 net [netinet] [patch] Concurrency in ixgbe driving out-of- o kern/176420 net [kernel] [patch] incorrect errno for LOCAL_PEERCRED o kern/176419 net [kernel] [patch] socketpair support for LOCAL_PEERCRED o kern/176401 net [netgraph] page fault in netgraph o kern/176167 net [ipsec][lagg] using lagg and ipsec causes immediate pa o kern/176097 net [lagg] [patch] lagg/lacp broken when aggregated interf o kern/176027 net [em] [patch] flow control systcl consistency for em dr o kern/176026 net [tcp] [patch] TCP wrappers caused quite a lot of warni o bin/175974 net ppp(8): logic issue o kern/175864 net [re] Intel MB D510MO, onboard ethernet not working aft o kern/175852 net [amd64] [patch] in_cksum_hdr() behaves differently on o kern/175734 net no ethernet detected on system with EG20T PCH chipset o kern/175267 net [pf] [tap] pf + tap keep state problem o kern/175236 net [epair] [gif] epair and gif Devices On Bridge o kern/175182 net [panic] kernel panic on RADIX_MPATH when deleting rout o kern/175153 net [tcp] will there miss a FIN when do TSO? o kern/174959 net [net] [patch] rnh_walktree_from visits spurious nodes o kern/174958 net [net] [patch] rnh_walktree_from makes unreasonable ass o kern/174897 net [route] Interface routes are broken o kern/174851 net [bxe] [patch] UDP checksum offload is wrong in bxe dri o kern/174850 net [bxe] [patch] bxe driver does not receive multicasts o kern/174849 net [bxe] [patch] bxe driver can hang kernel when reset o kern/174822 net [tcp] Page fault in tcp_discardcb under high traffic o kern/174602 net [gif] [ipsec] traceroute issue on gif tunnel with ipse o kern/174535 net [tcp] TCP fast retransmit feature works strange o kern/173871 net [gif] process of 'ifconfig gif0 create hangs' when if_ o kern/173475 net [tun] tun(4) stays opened by PID after process is term o kern/173201 net [ixgbe] [patch] Missing / broken ixgbe sysctl's and tu o kern/173137 net [em] em(4) unable to run at gigabit with 9.1-RC2 o kern/173002 net [patch] data type size problem in if_spppsubr.c o kern/172895 net [ixgb] [ixgbe] do not properly determine link-state o kern/172683 net [ip6] Duplicate IPv6 Link Local Addresses o kern/172675 net [netinet] [patch] sysctl_tcp_hc_list (net.inet.tcp.hos o kern/172113 net [panic] [e1000] [patch] 9.1-RC1/amd64 panices in igb(4 o kern/171840 net [ip6] IPv6 packets transmitting only on queue 0 o kern/171739 net [bce] [panic] bce related kernel panic o kern/171711 net [dummynet] [panic] Kernel panic in dummynet o kern/171532 net [ndis] ndis(4) driver includes 'pccard'-specific code, o kern/171531 net [ndis] undocumented dependency for ndis(4) o kern/171524 net [ipmi] ipmi driver crashes kernel by reboot or shutdow s kern/171508 net [epair] [request] Add the ability to name epair device o kern/171228 net [re] [patch] if_re - eeprom write issues o kern/170701 net [ppp] killl ppp or reboot with active ppp connection c o kern/170267 net [ixgbe] IXGBE_LE32_TO_CPUS is probably an unintentiona o kern/170081 net [fxp] pf/nat/jails not working if checksum offloading o kern/169898 net ifconfig(8) fails to set MTU on multiple interfaces. o kern/169676 net [bge] [hang] system hangs, fully or partially after re o kern/169664 net [bgp] Wrongful replacement of interface connected net o kern/169620 net [ng] [pf] ng_l2tp incoming packet bypass pf firewall o kern/169459 net [ppp] umodem/ppp/3g stopped working after update from o kern/169438 net [ipsec] ipv4-in-ipv6 tunnel mode IPsec does not work p kern/168294 net [ixgbe] [patch] ixgbe driver compiled in kernel has no o kern/168246 net [em] Multiple em(4) not working with qemu o kern/168245 net [arp] [regression] Permanent ARP entry not deleted on o kern/168244 net [arp] [regression] Unable to manually remove permanent o kern/168183 net [bce] bce driver hang system o kern/167947 net [setfib] [patch] arpresolve checks only the default FI o kern/167603 net [ip] IP fragment reassembly's broken: file transfer ov o kern/167500 net [em] [panic] Kernel panics in em driver o kern/167325 net [netinet] [patch] sosend sometimes return EINVAL with o kern/167202 net [igmp]: Sending multiple IGMP packets crashes kernel o kern/166462 net [gre] gre(4) when using a tunnel source address from c o kern/166285 net [arp] FreeBSD v8.1 REL p8 arp: unknown hardware addres o kern/166255 net [net] [patch] It should be possible to disable "promis p kern/165903 net mbuf leak o kern/165622 net [ndis][panic][patch] Unregistered use of FPU in kernel s kern/165562 net [request] add support for Intel i350 in FreeBSD 7.4 o kern/165526 net [bxe] UDP packets checksum calculation whithin if_bxe o kern/165488 net [ppp] [panic] Fatal trap 12 jails and ppp , kernel wit o kern/165305 net [ip6] [request] Feature parity between IP_TOS and IPV6 o kern/165296 net [vlan] [patch] Fix EVL_APPLY_VLID, update EVL_APPLY_PR o kern/165181 net [igb] igb freezes after about 2 weeks of uptime o kern/165174 net [patch] [tap] allow tap(4) to keep its address on clos o kern/165152 net [ip6] Does not work through the issue of ipv6 addresse o kern/164495 net [igb] connect double head igb to switch cause system t o kern/164490 net [pfil] Incorrect IP checksum on pfil pass from ip_outp o kern/164475 net [gre] gre misses RUNNING flag after a reboot o kern/164265 net [netinet] [patch] tcp_lro_rx computes wrong checksum i o kern/163903 net [igb] "igb0:tx(0)","bpf interface lock" v2.2.5 9-STABL o kern/163481 net freebsd do not add itself to ping route packet o kern/162927 net [tun] Modem-PPP error ppp[1538]: tun0: Phase: Clearing o kern/162558 net [dummynet] [panic] seldom dummynet panics o kern/162153 net [em] intel em driver 7.2.4 don't compile o kern/162110 net [igb] [panic] RELENG_9 panics on boot in IGB driver - o kern/162028 net [ixgbe] [patch] misplaced #endif in ixgbe.c o kern/161277 net [em] [patch] BMC cannot receive IPMI traffic after loa o kern/160873 net [igb] igb(4) from HEAD fails to build on 7-STABLE o kern/160750 net Intel PRO/1000 connection breaks under load until rebo o kern/160693 net [gif] [em] Multicast packet are not passed from GIF0 t o kern/160293 net [ieee80211] ppanic] kernel panic during network setup o kern/160206 net [gif] gifX stops working after a while (IPv6 tunnel) o kern/159817 net [udp] write UDPv4: No buffer space available (code=55) o kern/159629 net [ipsec] [panic] kernel panic with IPsec in transport m o kern/159621 net [tcp] [panic] panic: soabort: so_count o kern/159603 net [netinet] [patch] in_ifscrubprefix() - network route c o kern/159601 net [netinet] [patch] in_scrubprefix() - loopback route re o kern/159294 net [em] em watchdog timeouts o kern/159203 net [wpi] Intel 3945ABG Wireless LAN not support IBSS o kern/158930 net [bpf] BPF element leak in ifp->bpf_if->bif_dlist o kern/158726 net [ip6] [patch] ICMPv6 Router Announcement flooding limi o kern/158694 net [ix] [lagg] ix0 is not working within lagg(4) o kern/158665 net [ip6] [panic] kernel pagefault in in6_setscope() o kern/158635 net [em] TSO breaks BPF packet captures with em driver f kern/157802 net [dummynet] [panic] kernel panic in dummynet o kern/157785 net amd64 + jail + ipfw + natd = very slow outbound traffi o kern/157418 net [em] em driver lockup during boot on Supermicro X9SCM- o kern/157410 net [ip6] IPv6 Router Advertisements Cause Excessive CPU U o kern/157287 net [re] [panic] INVARIANTS panic (Memory modified after f o kern/157200 net [network.subr] [patch] stf(4) can not communicate betw o kern/157182 net [lagg] lagg interface not working together with epair o kern/156877 net [dummynet] [panic] dummynet move_pkt() null ptr derefe o kern/156667 net [em] em0 fails to init on CURRENT after March 17 o kern/156408 net [vlan] Routing failure when using VLANs vs. Physical e o kern/156328 net [icmp]: host can ping other subnet but no have IP from o kern/156317 net [ip6] Wrong order of IPv6 NS DAD/MLD Report o kern/156283 net [ip6] [patch] nd6_ns_input - rtalloc_mpath does not re o kern/156279 net [if_bridge][divert][ipfw] unable to correctly re-injec o kern/156226 net [lagg]: failover does not announce the failover to swi o kern/156030 net [ip6] [panic] Crash in nd6_dad_start() due to null ptr o kern/155680 net [multicast] problems with multicast s kern/155642 net [new driver] [request] Add driver for Realtek RTL8191S o kern/155597 net [panic] Kernel panics with "sbdrop" message o kern/155420 net [vlan] adding vlan break existent vlan o kern/155177 net [route] [panic] Panic when inject routes in kernel o kern/155010 net [msk] ntfs-3g via iscsi using msk driver cause kernel o kern/154943 net [gif] ifconfig gifX create on existing gifX clears IP s kern/154851 net [new driver] [request]: Port brcm80211 driver from Lin o kern/154850 net [netgraph] [patch] ng_ether fails to name nodes when t o kern/154679 net [em] Fatal trap 12: "em1 taskq" only at startup (8.1-R o kern/154600 net [tcp] [panic] Random kernel panics on tcp_output o kern/154557 net [tcp] Freeze tcp-session of the clients, if in the gat o kern/154443 net [if_bridge] Kernel module bridgestp.ko missing after u o kern/154286 net [netgraph] [panic] 8.2-PRERELEASE panic in netgraph o kern/154255 net [nfs] NFS not responding o kern/154214 net [stf] [panic] Panic when creating stf interface o kern/154185 net race condition in mb_dupcl p kern/154169 net [multicast] [ip6] Node Information Query multicast add o kern/154134 net [ip6] stuck kernel state in LISTEN on ipv6 daemon whic o kern/154091 net [netgraph] [panic] netgraph, unaligned mbuf? o conf/154062 net [vlan] [patch] change to way of auto-generatation of v o kern/153937 net [ral] ralink panics the system (amd64 freeBSDD 8.X) wh o kern/153936 net [ixgbe] [patch] MPRC workaround incorrectly applied to o kern/153816 net [ixgbe] ixgbe doesn't work properly with the Intel 10g o kern/153772 net [ixgbe] [patch] sysctls reference wrong XON/XOFF varia o kern/153497 net [netgraph] netgraph panic due to race conditions o kern/153454 net [patch] [wlan] [urtw] Support ad-hoc and hostap modes o kern/153308 net [em] em interface use 100% cpu o kern/153244 net [em] em(4) fails to send UDP to port 0xffff o kern/152893 net [netgraph] [panic] 8.2-PRERELEASE panic in netgraph o kern/152853 net [em] tftpd (and likely other udp traffic) fails over e o kern/152828 net [em] poor performance on 8.1, 8.2-PRE o kern/152569 net [net]: Multiple ppp connections and routing table prob o kern/152235 net [arp] Permanent local ARP entries are not properly upd o kern/152141 net [vlan] [patch] encapsulate vlan in ng_ether before out o kern/152036 net [libc] getifaddrs(3) returns truncated sockaddrs for n o kern/151690 net [ep] network connectivity won't work until dhclient is o kern/151681 net [nfs] NFS mount via IPv6 leads to hang on client with o kern/151593 net [igb] [panic] Kernel panic when bringing up igb networ o kern/150920 net [ixgbe][igb] Panic when packets are dropped with heade o kern/150557 net [igb] igb0: Watchdog timeout -- resetting o kern/150251 net [patch] [ixgbe] Late cable insertion broken o kern/150249 net [ixgbe] Media type detection broken o bin/150224 net ppp(8) does not reassign static IP after kill -KILL co f kern/149969 net [wlan] [ral] ralink rt2661 fails to maintain connectio o kern/149643 net [rum] device not sending proper beacon frames in ap mo o kern/149609 net [panic] reboot after adding second default route o kern/149117 net [inet] [patch] in_pcbbind: redundant test o kern/149086 net [multicast] Generic multicast join failure in 8.1 o kern/148018 net [flowtable] flowtable crashes on ia64 o kern/147912 net [boot] FreeBSD 8 Beta won't boot on Thinkpad i1300 11 o kern/147894 net [ipsec] IPv6-in-IPv4 does not work inside an ESP-only o kern/147155 net [ip6] setfb not work with ipv6 o kern/146845 net [libc] close(2) returns error 54 (connection reset by f kern/146792 net [flowtable] flowcleaner 100% cpu's core load o kern/146719 net [pf] [panic] PF or dumynet kernel panic o kern/146534 net [icmp6] wrong source address in echo reply o kern/146427 net [mwl] Additional virtual access points don't work on m f kern/146394 net [vlan] IP source address for outgoing connections o bin/146377 net [ppp] [tun] Interface doesn't clear addresses when PPP o kern/146358 net [vlan] wrong destination MAC address o kern/146165 net [wlan] [panic] Setting bssid in adhoc mode causes pani o kern/146082 net [ng_l2tp] a false invaliant check was performed in ng_ o kern/146037 net [panic] mpd + CoA = kernel panic o kern/145825 net [panic] panic: soabort: so_count o kern/145728 net [lagg] Stops working lagg between two servers. p kern/145600 net TCP/ECN behaves different to CE/CWR than ns2 reference f kern/144917 net [flowtable] [panic] flowtable crashes system [regressi o kern/144882 net MacBookPro =>4.1 does not connect to BSD in hostap wit o kern/144874 net [if_bridge] [patch] if_bridge frees mbuf after pfil ho o conf/144700 net [rc.d] async dhclient breaks stuff for too many people o kern/144616 net [nat] [panic] ip_nat panic FreeBSD 7.2 f kern/144315 net [ipfw] [panic] freebsd 8-stable reboot after add ipfw o kern/144231 net bind/connect/sendto too strict about sockaddr length o kern/143846 net [gif] bringing gif3 tunnel down causes gif0 tunnel to s kern/143673 net [stf] [request] there should be a way to support multi s kern/143666 net [ip6] [request] PMTU black hole detection not implemen o kern/143622 net [pfil] [patch] unlock pfil lock while calling firewall o kern/143593 net [ipsec] When using IPSec, tcpdump doesn't show outgoin o kern/143591 net [ral] RT2561C-based DLink card (DWL-510) fails to work o kern/143208 net [ipsec] [gif] IPSec over gif interface not working o kern/143034 net [panic] system reboots itself in tcp code [regression] o kern/142877 net [hang] network-related repeatable 8.0-STABLE hard hang o kern/142774 net Problem with outgoing connections on interface with mu o kern/142772 net [libc] lla_lookup: new lle malloc failed f kern/142518 net [em] [lagg] Problem on 8.0-STABLE with em and lagg o kern/142018 net [iwi] [patch] Possibly wrong interpretation of beacon- o kern/141861 net [wi] data garbled with WEP and wi(4) with Prism 2.5 f kern/141741 net Etherlink III NIC won't work after upgrade to FBSD 8, o kern/140742 net rum(4) Two asus-WL167G adapters cannot talk to each ot o kern/140682 net [netgraph] [panic] random panic in netgraph f kern/140634 net [vlan] destroying if_lagg interface with if_vlan membe o kern/140619 net [ifnet] [patch] refine obsolete if_var.h comments desc o kern/140346 net [wlan] High bandwidth use causes loss of wlan connecti o kern/140142 net [ip6] [panic] FreeBSD 7.2-amd64 panic w/IPv6 o kern/140066 net [bwi] install report for 8.0 RC 2 (multiple problems) o kern/139387 net [ipsec] Wrong lenth of PF_KEY messages in promiscuous o bin/139346 net [patch] arp(8) add option to remove static entries lis o kern/139268 net [if_bridge] [patch] allow if_bridge to forward just VL p kern/139204 net [arp] DHCP server replies rejected, ARP entry lost bef o kern/139117 net [lagg] + wlan boot timing (EBUSY) o kern/138850 net [dummynet] dummynet doesn't work correctly on a bridge o kern/138782 net [panic] sbflush_internal: cc 0 || mb 0xffffff004127b00 o kern/138688 net [rum] possibly broken on 8 Beta 4 amd64: able to wpa a o kern/138678 net [lo] FreeBSD does not assign linklocal address to loop o kern/138407 net [gre] gre(4) interface does not come up after reboot o kern/138332 net [tun] [lor] ifconfig tun0 destroy causes LOR if_adata/ o kern/138266 net [panic] kernel panic when udp benchmark test used as r f kern/138029 net [bpf] [panic] periodically kernel panic and reboot o kern/137881 net [netgraph] [panic] ng_pppoe fatal trap 12 p bin/137841 net [patch] wpa_supplicant(8) cannot verify SHA256 signed p kern/137776 net [rum] panic in rum(4) driver on 8.0-BETA2 o bin/137641 net ifconfig(8): various problems with "vlan_device.vlan_i o kern/137392 net [ip] [panic] crash in ip_nat.c line 2577 o kern/137372 net [ral] FreeBSD doesn't support wireless interface from o kern/137089 net [lagg] lagg falsely triggers IPv6 duplicate address de o kern/136911 net [netgraph] [panic] system panic on kldload ng_bpf.ko t o kern/136618 net [pf][stf] panic on cloning interface without unit numb o kern/135502 net [periodic] Warning message raised by rtfree function i o kern/134583 net [hang] Machine with jail freezes after random amount o o kern/134531 net [route] [panic] kernel crash related to routes/zebra o kern/134157 net [dummynet] dummynet loads cpu for 100% and make a syst o kern/133969 net [dummynet] [panic] Fatal trap 12: page fault while in o kern/133968 net [dummynet] [panic] dummynet kernel panic o kern/133736 net [udp] ip_id not protected ... o kern/133595 net [panic] Kernel Panic at pcpu.h:195 o kern/133572 net [ppp] [hang] incoming PPTP connection hangs the system o kern/133490 net [bpf] [panic] 'kmem_map too small' panic on Dell r900 o kern/133235 net [netinet] [patch] Process SIOCDLIFADDR command incorre f kern/133213 net arp and sshd errors on 7.1-PRERELEASE o kern/133060 net [ipsec] [pfsync] [panic] Kernel panic with ipsec + pfs o kern/132889 net [ndis] [panic] NDIS kernel crash on load BCM4321 AGN d o conf/132851 net [patch] rc.conf(5): allow to setfib(1) for service run o kern/132734 net [ifmib] [panic] panic in net/if_mib.c o kern/132705 net [libwrap] [patch] libwrap - infinite loop if hosts.all o kern/132672 net [ndis] [panic] ndis with rt2860.sys causes kernel pani o kern/132354 net [nat] Getting some packages to ipnat(8) causes crash o kern/132277 net [crypto] [ipsec] poor performance using cryptodevice f o kern/131781 net [ndis] ndis keeps dropping the link o kern/131776 net [wi] driver fails to init o kern/131753 net [altq] [panic] kernel panic in hfsc_dequeue o bin/131365 net route(8): route add changes interpretation of network f kern/130820 net [ndis] wpa_supplicant(8) returns 'no space on device' o kern/130628 net [nfs] NFS / rpc.lockd deadlock on 7.1-R o kern/130525 net [ndis] [panic] 64 bit ar5008 ndisgen-erated driver cau o kern/130311 net [wlan_xauth] [panic] hostapd restart causing kernel pa o kern/130109 net [ipfw] Can not set fib for packets originated from loc f kern/130059 net [panic] Leaking 50k mbufs/hour f kern/129719 net [nfs] [panic] Panic during shutdown, tcp_ctloutput: in o kern/129517 net [ipsec] [panic] double fault / stack overflow f kern/129508 net [carp] [panic] Kernel panic with EtherIP (may be relat o kern/129219 net [ppp] Kernel panic when using kernel mode ppp o kern/129197 net [panic] 7.0 IP stack related panic o bin/128954 net ifconfig(8) deletes valid routes o bin/128602 net [an] wpa_supplicant(8) crashes with an(4) o kern/128448 net [nfs] 6.4-RC1 Boot Fails if NFS Hostname cannot be res o bin/128295 net [patch] ifconfig(8) does not print TOE4 or TOE6 capabi o bin/128001 net wpa_supplicant(8), wlan(4), and wi(4) issues o kern/127826 net [iwi] iwi0 driver has reduced performance and connecti o kern/127815 net [gif] [patch] if_gif does not set vlan attributes from o kern/127724 net [rtalloc] rtfree: 0xc5a8f870 has 1 refs f bin/127719 net [arp] arp: Segmentation fault (core dumped) f kern/127528 net [icmp]: icmp socket receives icmp replies not owned by p kern/127360 net [socket] TOE socket options missing from sosetopt() o bin/127192 net routed(8) removes the secondary alias IP of interface f kern/127145 net [wi]: prism (wi) driver crash at bigger traffic o kern/126895 net [patch] [ral] Add antenna selection (marked as TBD) o kern/126874 net [vlan]: Zebra problem if ifconfig vlanX destroy o kern/126695 net rtfree messages and network disruption upon use of if_ o kern/126339 net [ipw] ipw driver drops the connection o kern/126075 net [inet] [patch] internet control accesses beyond end of o bin/125922 net [patch] Deadlock in arp(8) o kern/125920 net [arp] Kernel Routing Table loses Ethernet Link status o kern/125845 net [netinet] [patch] tcp_lro_rx() should make use of hard o kern/125258 net [socket] socket's SO_REUSEADDR option does not work o kern/125239 net [gre] kernel crash when using gre o kern/124341 net [ral] promiscuous mode for wireless device ral0 looses o kern/124225 net [ndis] [patch] ndis network driver sometimes loses net o kern/124160 net [libc] connect(2) function loops indefinitely o kern/124021 net [ip6] [panic] page fault in nd6_output() o kern/123968 net [rum] [panic] rum driver causes kernel panic with WPA. o kern/123892 net [tap] [patch] No buffer space available o kern/123890 net [ppp] [panic] crash & reboot on work with PPP low-spee o kern/123858 net [stf] [patch] stf not usable behind a NAT o kern/123758 net [panic] panic while restarting net/freenet6 o bin/123633 net ifconfig(8) doesn't set inet and ether address in one o kern/123559 net [iwi] iwi periodically disassociates/associates [regre o bin/123465 net [ip6] route(8): route add -inet6 -interfac o kern/123463 net [ipsec] [panic] repeatable crash related to ipsec-tool o conf/123330 net [nsswitch.conf] Enabling samba wins in nsswitch.conf c o kern/123160 net [ip] Panic and reboot at sysctl kern.polling.enable=0 o kern/122989 net [swi] [panic] 6.3 kernel panic in swi1: net o kern/122954 net [lagg] IPv6 EUI64 incorrectly chosen for lagg devices f kern/122780 net [lagg] tcpdump on lagg interface during high pps wedge o kern/122685 net It is not visible passing packets in tcpdump(1) o kern/122319 net [wi] imposible to enable ad-hoc demo mode with Orinoco o kern/122290 net [netgraph] [panic] Netgraph related "kmem_map too smal o kern/122252 net [ipmi] [bge] IPMI problem with BCM5704 (does not work o kern/122033 net [ral] [lor] Lock order reversal in ral0 at bootup ieee o bin/121895 net [patch] rtsol(8)/rtsold(8) doesn't handle managed netw s kern/121774 net [swi] [panic] 6.3 kernel panic in swi1: net o kern/121555 net [panic] Fatal trap 12: current process = 12 (swi1: net o kern/121534 net [ipl] [nat] FreeBSD Release 6.3 Kernel Trap 12: o kern/121443 net [gif] [lor] icmp6_input/nd6_lookup o kern/121437 net [vlan] Routing to layer-2 address does not work on VLA o bin/121359 net [patch] [security] ppp(8): fix local stack overflow in o kern/121257 net [tcp] TSO + natd -> slow outgoing tcp traffic o kern/121181 net [panic] Fatal trap 3: breakpoint instruction fault whi o kern/120966 net [rum] kernel panic with if_rum and WPA encryption o kern/120566 net [request]: ifconfig(8) make order of arguments more fr o kern/120304 net [netgraph] [patch] netgraph source assumes 32-bit time o kern/120266 net [udp] [panic] gnugk causes kernel panic when closing U o bin/120060 net routed(8) deletes link-level routes in the presence of o kern/119945 net [rum] [panic] rum device in hostap mode, cause kernel o kern/119791 net [nfs] UDP NFS mount of aliased IP addresses from a Sol o kern/119617 net [nfs] nfs error on wpa network when reseting/shutdown f kern/119516 net [ip6] [panic] _mtx_lock_sleep: recursed on non-recursi o kern/119432 net [arp] route add -host -iface causes arp e o kern/119225 net [wi] 7.0-RC1 no carrier with Prism 2.5 wifi card [regr o kern/118727 net [netgraph] [patch] [request] add new ng_pf module o kern/117423 net [vlan] Duplicate IP on different interfaces o bin/117339 net [patch] route(8): loading routing management commands o bin/116643 net [patch] [request] fstat(1): add INET/INET6 socket deta o kern/116185 net [iwi] if_iwi driver leads system to reboot o kern/115239 net [ipnat] panic with 'kmem_map too small' using ipnat o kern/115019 net [netgraph] ng_ether upper hook packet flow stops on ad o kern/115002 net [wi] if_wi timeout. failed allocation (busy bit). ifco o kern/114915 net [patch] [pcn] pcn (sys/pci/if_pcn.c) ethernet driver f o kern/113432 net [ucom] WARNING: attempt to net_add_domain(netgraph) af o kern/112722 net [ipsec] [udp] IP v4 udp fragmented packet reject o kern/112686 net [patm] patm driver freezes System (FreeBSD 6.2-p4) i38 o bin/112557 net [patch] ppp(8) lock file should not use symlink name o kern/112528 net [nfs] NFS over TCP under load hangs with "impossible p o kern/111537 net [inet6] [patch] ip6_input() treats mbuf cluster wrong o kern/111457 net [ral] ral(4) freeze o kern/110284 net [if_ethersubr] Invalid Assumption in SIOCSIFADDR in et o kern/110249 net [kernel] [regression] [patch] setsockopt() error regre o kern/109470 net [wi] Orinoco Classic Gold PC Card Can't Channel Hop o bin/108895 net pppd(8): PPPoE dead connections on 6.2 [regression] f kern/108197 net [panic] [gif] [ip6] if_delmulti reference counting pan o kern/107944 net [wi] [patch] Forget to unlock mutex-locks o conf/107035 net [patch] bridge(8): bridge interface given in rc.conf n o kern/106444 net [netgraph] [panic] Kernel Panic on Binding to an ip to o kern/106316 net [dummynet] dummynet with multipass ipfw drops packets o kern/105945 net Address can disappear from network interface s kern/105943 net Network stack may modify read-only mbuf chain copies o bin/105925 net problems with ifconfig(8) and vlan(4) [regression] o kern/104851 net [inet6] [patch] On link routes not configured when usi o kern/104751 net [netgraph] kernel panic, when getting info about my tr o kern/104738 net [inet] [patch] Reentrant problem with inet_ntoa in the o kern/103191 net Unpredictable reboot o kern/103135 net [ipsec] ipsec with ipfw divert (not NAT) encodes a pac o kern/102540 net [netgraph] [patch] supporting vlan(4) by ng_fec(4) o conf/102502 net [netgraph] [patch] ifconfig name does't rename netgrap o kern/102035 net [plip] plip networking disables parallel port printing o kern/100709 net [libc] getaddrinfo(3) should return TTL info o kern/100519 net [netisr] suggestion to fix suboptimal network polling o kern/98597 net [inet6] Bug in FreeBSD 6.1 IPv6 link-local DAD procedu o bin/98218 net wpa_supplicant(8) blacklist not working o kern/97306 net [netgraph] NG_L2TP locks after connection with failed o conf/97014 net [gif] gifconfig_gif? in rc.conf does not recognize IPv f kern/96268 net [socket] TCP socket performance drops by 3000% if pack o kern/95519 net [ral] ral0 could not map mbuf o kern/95288 net [pppd] [tty] [panic] if_ppp panic in sys/kern/tty_subr o kern/95277 net [netinet] [patch] IP Encapsulation mask_match() return o kern/95267 net packet drops periodically appear f kern/93378 net [tcp] Slow data transfer in Postfix and Cyrus IMAP (wo o kern/93019 net [ppp] ppp and tunX problems: no traffic after restarti o kern/92880 net [libc] [patch] almost rewritten inet_network(3) functi s kern/92279 net [dc] Core faults everytime I reboot, possible NIC issu o kern/91859 net [ndis] if_ndis does not work with Asus WL-138 o kern/91364 net [ral] [wep] WF-511 RT2500 Card PCI and WEP o kern/91311 net [aue] aue interface hanging o kern/87421 net [netgraph] [panic]: ng_ether + ng_eiface + if_bridge o kern/86871 net [tcp] [patch] allocation logic for PCBs in TIME_WAIT s o kern/86427 net [lor] Deadlock with FASTIPSEC and nat o kern/85780 net 'panic: bogus refcnt 0' in routing/ipv6 o bin/85445 net ifconfig(8): deprecated keyword to ifconfig inoperativ o bin/82975 net route change does not parse classfull network as given o kern/82881 net [netgraph] [panic] ng_fec(4) causes kernel panic after o kern/82468 net Using 64MB tcp send/recv buffers, trafficflow stops, i o bin/82185 net [patch] ndp(8) can delete the incorrect entry o kern/81095 net IPsec connection stops working if associated network i o kern/78968 net FreeBSD freezes on mbufs exhaustion (network interface o kern/78090 net [ipf] ipf filtering on bridged packets doesn't work if o kern/77341 net [ip6] problems with IPV6 implementation o kern/75873 net Usability problem with non-RFC-compliant IP spoof prot s kern/75407 net [an] an(4): no carrier after short time a kern/71474 net [route] route lookup does not skip interfaces marked d o kern/71469 net default route to internet magically disappears with mu o kern/68889 net [panic] m_copym, length > size of mbuf chain o kern/66225 net [netgraph] [patch] extend ng_eiface(4) control message o kern/65616 net IPSEC can't detunnel GRE packets after real ESP encryp s kern/60293 net [patch] FreeBSD arp poison patch a kern/56233 net IPsec tunnel (ESP) over IPv6: MTU computation is wrong s bin/41647 net ifconfig(8) doesn't accept lladdr along with inet addr o kern/39937 net ipstealth issue a kern/38554 net [patch] changing interface ipaddress doesn't seem to w o kern/31940 net ip queue length too short for >500kpps o kern/31647 net [libc] socket calls can return undocumented EINVAL o kern/30186 net [libc] getaddrinfo(3) does not handle incorrect servna f kern/24959 net [patch] proper TCP_NOPUSH/TCP_CORK compatibility o conf/23063 net [arp] [patch] for static ARP tables in rc.network o kern/21998 net [socket] [patch] ident only for outgoing connections o kern/5877 net [socket] sb_cc counts control data as well as data dat 452 problems total. From owner-freebsd-net@FreeBSD.ORG Mon Jul 22 14:20:02 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 115B8161 for ; Mon, 22 Jul 2013 14:20:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E70052111 for ; Mon, 22 Jul 2013 14:20:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6MEK1CE094831 for ; Mon, 22 Jul 2013 14:20:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6MEK1YE094830; Mon, 22 Jul 2013 14:20:01 GMT (envelope-from gnats) Date: Mon, 22 Jul 2013 14:20:01 GMT Message-Id: <201307221420.r6MEK1YE094830@freefall.freebsd.org> To: freebsd-net@FreeBSD.org Cc: From: Meny Yossefi Subject: RE: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragmented packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Meny Yossefi List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 14:20:02 -0000 The following reply was made to PR kern/180430; it has been noted by GNATS. From: Meny Yossefi To: John Baldwin , "bug-followup@freebsd.org" Cc: Subject: RE: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragmented packets Date: Mon, 22 Jul 2013 14:11:51 +0000 --_000_F2E47A38E4D0B9499D76F2AB8901571A73A0C0DAMTLDAG01mtlcom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi John, The problem is that the HW will only calculate csum for parts of the payloa= d, one fragment at a time, while the receiver side, in our case the tcp/ip stack, will expect to valid= ate the packet's payload as a whole. I agree with the change you offered, though one thing bothers me. This change will add two conditions to the send flow which will probably ha= ve an effect on performance. Maybe 'likely' can be useful here ? BTW, I'm not entirely sure, but I think the CSUM_IP flag is always set, so = maybe the first condition is not necessary. What do you think ? -Meny -----Original Message----- From: John Baldwin [mailto:jhb@freebsd.org] Sent: Friday, July 19, 2013 6:29 PM To: bug-followup@freebsd.org; Meny Yossefi Subject: Re: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragment= ed packets Oops, my previous reply didn't make it to the PR itself. Is the problem that the hardware checksum overwrites arbitrary data in the = packet (at the location where the UDP header would be)? Also, I've looked at other drivers, and they all look at the CSUM_* flags t= o determine the right settings. IP fragments will not have CSUM_UDP or CSU= M_TCP set, so I think the more correct fix is this: Index: en_tx.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- en_tx.c (revision 253470) +++ en_tx.c (working copy) @@ -780,8 +780,12 @@ retry: tx_desc->ctrl.srcrb_flags =3D cpu_to_be32(MLX4_WQE_CTRL_CQ_U= PDATE | = MLX4_WQE_CTRL_SOLICITED); if (mb->m_pkthdr.csum_flags & (CSUM_IP|CSUM_TCP|CSUM_UDP)) { - tx_desc->ctrl.srcrb_flags |=3D cpu_to_be32(M= LX4_WQE_CTRL_IP_CSUM | - = MLX4_WQE_CTRL_TCP_UDP_CSUM); + if (mb->m_pkthdr.csum_flags & CSUM_IP) + tx_desc->ctrl.srcrb_flags |= =3D + cpu_to_be32(MLX4_WQE_CTRL= _IP_CSUM); + if (mb->m_pkthdr.csum_flags & (CSUM_TCP|CSUM_= UDP)) + tx_desc->ctrl.srcrb_flags |= =3D + cpu_to_be32(MLX4_WQE_CTRL= _TCP_UDP_CSUM); priv->port_stats.tx_chksum_offload++; } -- John Baldwin --_000_F2E47A38E4D0B9499D76F2AB8901571A73A0C0DAMTLDAG01mtlcom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi John,

 

The problem is that= the HW will only calculate csum for parts of the payload, one fragment at = a time,

while the receiver = side, in our case the tcp/ip stack, will expect to validate the packet̵= 7;s payload as a whole.

 

I agree with the ch= ange you offered, though one thing bothers me.

This change will ad= d two conditions to the send flow which will probably have an effect on per= formance.

Maybe ‘likely= ’ can be useful here ?

 

BTW, I’m not = entirely sure, but I think the CSUM_IP flag is always set, so maybe the fir= st condition is not necessary.

What do you think ?=

 

-Meny

 

 

-----Original Message-----
From: John Baldwin [mailto:jhb@freebsd.org]
Sent: Friday, July 19, 2013 6:29 PM
To: bug-followup@freebsd.org; Meny Yossefi
Subject: Re: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragment= ed packets

 

Oops, my previous reply didn't make it to the PR = itself.

 

Is the problem that the hardware checksum overwri= tes arbitrary data in the packet (at the location where the UDP header woul= d be)?

 

Also, I've looked at other drivers, and they all = look at the CSUM_* flags to determine the right settings.  IP fragment= s will not have CSUM_UDP or CSUM_TCP set, so I think the more correct fix i= s this:

 

Index: en_tx.c

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D

--- en_tx.c      &n= bsp;    (revision 253470)

+++ en_tx.c    &n= bsp;   (working copy)

@@ -780,8 +780,12 @@ retry:

        &= nbsp;      tx_desc->ctrl.srcrb_flags =3D cpu_to= _be32(MLX4_WQE_CTRL_CQ_UPDATE |

        &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           MLX4_WQE_= CTRL_SOLICITED);

        &= nbsp;      if (mb->m_pkthdr.csum_flags & (C= SUM_IP|CSUM_TCP|CSUM_UDP)) {

-        =             &nb= sp;         tx_desc->ctrl.srcrb_= flags |=3D cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM |

-        =             &nb= sp;            =             &nb= sp;            =             &nb= sp;            =             &nb= sp;            =   MLX4_WQE_CTRL_TCP_UDP_CSUM);

+       &n= bsp;            = ;         if (mb->m_pkthdr.csum_= flags & CSUM_IP)

+       &n= bsp;            = ;            &n= bsp;            tx_d= esc->ctrl.srcrb_flags |=3D

+       &n= bsp;            = ;            &n= bsp;            &nbs= p;   cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM);

+       &n= bsp;            = ;         if (mb->m_pkthdr.csum_= flags & (CSUM_TCP|CSUM_UDP))

+       &n= bsp;            = ;            &n= bsp;            tx_d= esc->ctrl.srcrb_flags |=3D

+       &n= bsp;            = ;            &n= bsp;            &nbs= p;   cpu_to_be32(MLX4_WQE_CTRL_TCP_UDP_CSUM);

        &= nbsp;           &nbs= p;          priv->port_stat= s.tx_chksum_offload++;

        &= nbsp;      }

 

--

John Baldwin

--_000_F2E47A38E4D0B9499D76F2AB8901571A73A0C0DAMTLDAG01mtlcom_-- From owner-freebsd-net@FreeBSD.ORG Mon Jul 22 16:10:01 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9365D164 for ; Mon, 22 Jul 2013 16:10:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 82CB3265C for ; Mon, 22 Jul 2013 16:10:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6MGA1Ms017361 for ; Mon, 22 Jul 2013 16:10:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6MGA1Rr017360; Mon, 22 Jul 2013 16:10:01 GMT (envelope-from gnats) Date: Mon, 22 Jul 2013 16:10:01 GMT Message-Id: <201307221610.r6MGA1Rr017360@freefall.freebsd.org> To: freebsd-net@FreeBSD.org Cc: From: John Baldwin Subject: Re: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragmented packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: John Baldwin List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 16:10:01 -0000 The following reply was made to PR kern/180430; it has been noted by GNATS. From: John Baldwin To: Meny Yossefi Cc: "bug-followup@freebsd.org" Subject: Re: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragmented packets Date: Mon, 22 Jul 2013 11:40:08 -0400 On Monday, July 22, 2013 10:11:51 am Meny Yossefi wrote: > Hi John, > > > > The problem is that the HW will only calculate csum for parts of the payload, one fragment at a time, > > while the receiver side, in our case the tcp/ip stack, will expect to validate the packet's payload as a whole. Ok. > I agree with the change you offered, though one thing bothers me. > > This change will add two conditions to the send flow which will probably have an effect on performance. > > Maybe 'likely' can be useful here ? FreeBSD tends to not use likely/unlikely unless there is a demonstrable gain via benchmark measurements. However, if the OFED code regularly uses it and you feel this is a case where you would normally use it, it can be added. > BTW, I'm not entirely sure, but I think the CSUM_IP flag is always set, so maybe the first condition is not necessary. > > What do you think ? If the user uses ifconfig to disable checksum offload and force software checksums the flag will not be set. > -Meny > > > > > > -----Original Message----- > From: John Baldwin [mailto:jhb@freebsd.org] > Sent: Friday, July 19, 2013 6:29 PM > To: bug-followup@freebsd.org; Meny Yossefi > Subject: Re: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragmented packets > > > > Oops, my previous reply didn't make it to the PR itself. > > > > Is the problem that the hardware checksum overwrites arbitrary data in the packet (at the location where the UDP header would be)? > > > > Also, I've looked at other drivers, and they all look at the CSUM_* flags to determine the right settings. IP fragments will not have CSUM_UDP or CSUM_TCP set, so I think the more correct fix is this: > > > > Index: en_tx.c > > =================================================================== > > --- en_tx.c (revision 253470) > > +++ en_tx.c (working copy) > > @@ -780,8 +780,12 @@ retry: > > tx_desc->ctrl.srcrb_flags = cpu_to_be32(MLX4_WQE_CTRL_CQ_UPDATE | > > MLX4_WQE_CTRL_SOLICITED); > > if (mb->m_pkthdr.csum_flags & (CSUM_IP|CSUM_TCP|CSUM_UDP)) { > > - tx_desc->ctrl.srcrb_flags |= cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM | > > - MLX4_WQE_CTRL_TCP_UDP_CSUM); > > + if (mb->m_pkthdr.csum_flags & CSUM_IP) > > + tx_desc->ctrl.srcrb_flags |= > > + cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM); > > + if (mb->m_pkthdr.csum_flags & (CSUM_TCP|CSUM_UDP)) > > + tx_desc->ctrl.srcrb_flags |= > > + cpu_to_be32(MLX4_WQE_CTRL_TCP_UDP_CSUM); > > priv->port_stats.tx_chksum_offload++; > > } > > > > -- > > John Baldwin > -- John Baldwin From owner-freebsd-net@FreeBSD.ORG Mon Jul 22 16:20:03 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B67E3506 for ; Mon, 22 Jul 2013 16:20:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8E3C5270C for ; Mon, 22 Jul 2013 16:20:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6MGK2vx020054 for ; Mon, 22 Jul 2013 16:20:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6MGK2X1020053; Mon, 22 Jul 2013 16:20:02 GMT (envelope-from gnats) Date: Mon, 22 Jul 2013 16:20:02 GMT Message-Id: <201307221620.r6MGK2X1020053@freefall.freebsd.org> To: freebsd-net@FreeBSD.org Cc: From: Tiago Daniel Jacobs Subject: Re: kern/179429: [tap] STP enabled tap bridge X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Tiago Daniel Jacobs List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 16:20:03 -0000 The following reply was made to PR kern/179429; it has been noted by GNATS. From: Tiago Daniel Jacobs To: bug-followup@FreeBSD.org, os@tdj.cc Cc: Subject: Re: kern/179429: [tap] STP enabled tap bridge Date: Mon, 22 Jul 2013 13:07:56 -0300 It's a bug because STP don't work as expected when using TAP interfaces. From owner-freebsd-net@FreeBSD.ORG Mon Jul 22 19:33:00 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8C878B73 for ; Mon, 22 Jul 2013 19:33:00 +0000 (UTC) (envelope-from apache@server.energica.com.pt) Received: from server.energica.com.pt (server.energica.com.pt [195.22.21.3]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0F2C62039 for ; Mon, 22 Jul 2013 19:32:59 +0000 (UTC) Received: from server.energica.com.pt (localhost [127.0.0.1]) by server.energica.com.pt (8.13.1/8.13.1) with ESMTP id r6MJWs41032218 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 22 Jul 2013 20:32:55 +0100 Received: (from apache@localhost) by server.energica.com.pt (8.13.1/8.13.1/Submit) id r6MJWrl2032205; Mon, 22 Jul 2013 20:32:53 +0100 Date: Mon, 22 Jul 2013 20:32:53 +0100 To: net@freebsd.org Subject: online e-statement From: ANZ Online Banking Message-Id: <1307461111.483@anz.com.au> MIME-Version: 1.0 Content-Type: text/plain X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 19:33:00 -0000 A new statement is available. To view, click on the [1]"ACCOUNTS" References 1. http://www.theimperium.be/language/www.anz.com/nz/inetbanking/bankmain.html From owner-freebsd-net@FreeBSD.ORG Mon Jul 22 20:02:13 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 077E8497; Mon, 22 Jul 2013 20:02:13 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D908B21AE; Mon, 22 Jul 2013 20:02:12 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r6MK26Cd007072 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 22 Jul 2013 13:02:06 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r6MK25SH007071; Mon, 22 Jul 2013 13:02:05 -0700 (PDT) (envelope-from jmg) Date: Mon, 22 Jul 2013 13:02:05 -0700 From: John-Mark Gurney To: trafdev Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour Message-ID: <20130722200205.GO26412@funkthat.com> Mail-Followup-To: trafdev , Adrian Chadd , Sepherosa Ziehau , freebsd-net@freebsd.org References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <51E455D5.2090403@mail.ru> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Mon, 22 Jul 2013 13:02:06 -0700 (PDT) Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, Adrian Chadd X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 20:02:13 -0000 trafdev wrote this message on Mon, Jul 15, 2013 at 13:04 -0700: > Yep I think it's wasting of resources, poll manager should somehow be > configured to update only one process/thread. > Anyone know how to do that? This isn't currently possible w/o a shared kqueue, since the event is level triggered, not edge.. You could do it w/ a shared kqueue using _ONESHOT (but then you'd also have a shared listen fd which obviously isn't what the OP wants)... I guess it wouldn't be too hard to do a wake one style thing, where kqueue only deliveres the event once per "item/level", but right now kqueue doesn't know anything about the format of data (which would be number of listeners waiting)... Even if it did, there would be this dangerous contract that if an event is returned that the user land process would handle it... How is kqueue suppose to handle a server that crashes/dies between getting the event and accepting a connection? How is userland suppose to know that an event wasn't handled, or is just taking a long time? Sadly, if you want to avoid the thundering heards problem, I think blocking on accept is the best method, or using a fd passing scheme where only on process accept's connections... > On Mon Jul 15 12:53:55 2013, Adrian Chadd wrote: > >i've noticed this when doing this stuff in a threaded program with > >each thread listening on the same port. > > > >All threads wake up on each accepted connection, one thread wins and > >the other threads get EAGAIN. > > > > > > > >-adrian > > > >On 15 July 2013 12:31, trafdev wrote: > >>Thanks for reply. > >> > >>This approach produces lot of "resource temporary unavailable" (eagain) on > >>accept-ing connections in N-1 processes. > >>Is this possible to avoid this by e.g. tweaking kqueue? > >> > >> > >>On Sun Jul 14 19:37:59 2013, Sepherosa Ziehau wrote: > >>> > >>>On Sat, Jul 13, 2013 at 1:16 PM, trafdev wrote: > >>>> > >>>>Hello. > >>>> > >>>>Could someone help with following problem of SO_REUSEPORT. > >>> > >>> > >>>The most portable "load balance" between processes listening on the > >>>same TCP addr/port probably is: > >>> > >>>s=socket(); > >>>bind(s); > >>>listen(s); > >>>/* various socketopt and fcntl as you needed */ > >>>pid=fork(); > >>>if (pid==0) { > >>> server_loop(s); > >>> exit(1); > >>>} > >>>server_loop(s); > >>>exit(1); > >>> > >>>Even in Linux or DragonFly SO_REUSEPORT "load balance" between > >>>processes listening on the same TCP addr/port was introduced recently, > >>>so you probably won't want to rely on it. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-net@FreeBSD.ORG Mon Jul 22 21:26:35 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D8ADCC62; Mon, 22 Jul 2013 21:26:35 +0000 (UTC) (envelope-from trafdev@mail.ru) Received: from fallback3.mail.ru (fallback3.mail.ru [94.100.176.58]) by mx1.freebsd.org (Postfix) with ESMTP id 5883C25CC; Mon, 22 Jul 2013 21:26:35 +0000 (UTC) Received: from smtp49.i.mail.ru (smtp49.i.mail.ru [94.100.177.109]) by fallback3.mail.ru (mPOP.Fallback_MX) with ESMTP id C9552EF6DF60; Tue, 23 Jul 2013 01:26:33 +0400 (MSK) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:Subject:CC:To:MIME-Version:From:Date:Message-ID; bh=6q1Le/jIPEZy2+u861Ht2Yr3ZGWfeDMWoCfzZlduIe0=; b=ETy1BJV90GQGKeMfJwiECgDvSop82UdlarLqqWlJqr0y7Hcoow/0R4UWJvduQyuJG5V1kkUAFpAOZFwTgw5xGzJfN3t+PVmDgP1fKoKL75vjb8hI0lKBuMWGTjSbwagNmJXsC3KmHoL80LmuMyPU2T4j4+dSvyrdPQJx44v3RmE=; Received: from [50.156.108.197] (port=42334 helo=[192.168.1.116]) by smtp49.i.mail.ru with esmtpa (envelope-from ) id 1V1Ncf-0006io-5F; Tue, 23 Jul 2013 01:26:25 +0400 Message-ID: <51EDA37A.9040200@mail.ru> Date: Mon, 22 Jul 2013 14:26:18 -0700 From: trafdev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130628 Thunderbird/17.0.7 MIME-Version: 1.0 To: John-Mark Gurney Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> <20130722200205.GO26412@funkthat.com> In-Reply-To: <20130722200205.GO26412@funkthat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, Adrian Chadd X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2013 21:26:36 -0000 Actually overhead is almost zero, the real problem is in non-equivalent load distribution between processes. As https://lwn.net/Articles/542629/ mentions - "At Google, they have seen a factor-of-three difference between the thread accepting the most connections and the thread accepting the fewest connections;" I'm getting almost same results On Mon Jul 22 13:02:05 2013, John-Mark Gurney wrote: > trafdev wrote this message on Mon, Jul 15, 2013 at 13:04 -0700: >> Yep I think it's wasting of resources, poll manager should somehow be >> configured to update only one process/thread. >> Anyone know how to do that? > > This isn't currently possible w/o a shared kqueue, since the event is > level triggered, not edge.. You could do it w/ a shared kqueue using > _ONESHOT (but then you'd also have a shared listen fd which obviously > isn't what the OP wants)... > > I guess it wouldn't be too hard to do a wake one style thing, where > kqueue only deliveres the event once per "item/level", but right now > kqueue doesn't know anything about the format of data (which would be > number of listeners waiting)... Even if it did, there would be this > dangerous contract that if an event is returned that the user land > process would handle it... How is kqueue suppose to handle a server > that crashes/dies between getting the event and accepting a connection? > How is userland suppose to know that an event wasn't handled, or is > just taking a long time? > > Sadly, if you want to avoid the thundering heards problem, I think > blocking on accept is the best method, or using a fd passing scheme > where only on process accept's connections... > >> On Mon Jul 15 12:53:55 2013, Adrian Chadd wrote: >>> i've noticed this when doing this stuff in a threaded program with >>> each thread listening on the same port. >>> >>> All threads wake up on each accepted connection, one thread wins and >>> the other threads get EAGAIN. >>> >>> >>> >>> -adrian >>> >>> On 15 July 2013 12:31, trafdev wrote: >>>> Thanks for reply. >>>> >>>> This approach produces lot of "resource temporary unavailable" (eagain) on >>>> accept-ing connections in N-1 processes. >>>> Is this possible to avoid this by e.g. tweaking kqueue? >>>> >>>> >>>> On Sun Jul 14 19:37:59 2013, Sepherosa Ziehau wrote: >>>>> >>>>> On Sat, Jul 13, 2013 at 1:16 PM, trafdev wrote: >>>>>> >>>>>> Hello. >>>>>> >>>>>> Could someone help with following problem of SO_REUSEPORT. >>>>> >>>>> >>>>> The most portable "load balance" between processes listening on the >>>>> same TCP addr/port probably is: >>>>> >>>>> s=socket(); >>>>> bind(s); >>>>> listen(s); >>>>> /* various socketopt and fcntl as you needed */ >>>>> pid=fork(); >>>>> if (pid==0) { >>>>> server_loop(s); >>>>> exit(1); >>>>> } >>>>> server_loop(s); >>>>> exit(1); >>>>> >>>>> Even in Linux or DragonFly SO_REUSEPORT "load balance" between >>>>> processes listening on the same TCP addr/port was introduced recently, >>>>> so you probably won't want to rely on it. > From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 02:54:18 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 05E0BB53; Tue, 23 Jul 2013 02:54:18 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-x229.google.com (mail-wg0-x229.google.com [IPv6:2a00:1450:400c:c00::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4795D20E8; Tue, 23 Jul 2013 02:54:17 +0000 (UTC) Received: by mail-wg0-f41.google.com with SMTP id y10so3087857wgg.0 for ; Mon, 22 Jul 2013 19:54:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; bh=ArOW21+syw7+7BVO9YgDepBRilKBR4O020i1X3s5hUE=; b=WZo7uboShIpeGuoV5d59CV6m7ZLEXeuO7iGX4LyvNkAAt9vrnNuKPbhtCfDXt++0rC xDOOTzVKpgOLuWAvA8WlcfT7gBIMtzqqP6gTb1RDAIkcAgG0QWsXMVdsFmjwVgpQOA4t ATDasA3Y4vdT/OIH91UGApXI9X9jOrRXaN64x6etyKzIPqEKAUGFrdawwY5Ua/GIysaZ vNTxa79GQjtUdYNAiqK/qNmRd9okQl/Xoaw15qa+nyfR3PhvF6zkBV8LtgQ4URm2/zfW cb93BdsTIfFpWR9ZPOW9HmCTFcdhBPQMywA9vpIPIaJ7Ty8nYTmEZeZUv3bk0xDQ27zu hH8A== MIME-Version: 1.0 X-Received: by 10.194.11.72 with SMTP id o8mr21941742wjb.0.1374548055644; Mon, 22 Jul 2013 19:54:15 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Mon, 22 Jul 2013 19:54:15 -0700 (PDT) Date: Mon, 22 Jul 2013 19:54:15 -0700 X-Google-Sender-Auth: 2QvaLZkA24TsxikzWYQD5dd6BtE Message-ID: Subject: [ixgbe] Register txd/rxd sysctl values From: Adrian Chadd To: Jack F Vogel , FreeBSD Net , freebsd-current Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 02:54:18 -0000 Hi, This patch adds and hw.ixgbe tree and adds rxd/txd. These are already tunables but it wasn't easy to see if these were being set. I'd like to commit this soon. Thanks! -adrian adrian@freebsd-10-hack2:~/work/freebsd/head/src/sys/dev/ixgbe % svn diff . Index: ixgbe.c =================================================================== --- ixgbe.c (revision 253553) +++ ixgbe.c (working copy) @@ -234,6 +234,8 @@ ** TUNEABLE PARAMETERS: */ +static SYSCTL_NODE(_hw, OID_AUTO, ixgbe, CTLFLAG_RD, 0, "IXGBE driver parameters"); + /* ** AIM: Adaptive Interrupt Moderation ** which means that the interrupt rate @@ -291,6 +293,11 @@ static int ixgbe_rxd = PERFORM_RXD; TUNABLE_INT("hw.ixgbe.rxd", &ixgbe_rxd); +SYSCTL_INT(_hw_ixgbe, OID_AUTO, rxd, CTLFLAG_RDTUN, &ixgbe_rxd, 0, + "Number of receive descriptors per queue"); +SYSCTL_INT(_hw_ixgbe, OID_AUTO, txd, CTLFLAG_RDTUN, &ixgbe_txd, 0, + "Number of transmit descriptors per queue"); + /* ** Defining this on will allow the use ** of unsupported SFP+ modules, note that adrian@freebsd-10-hack2:~/work/freebsd/head/src/sys/dev/ixgbe % From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 04:05:57 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2AB9FBB4 for ; Tue, 23 Jul 2013 04:05:57 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qc0-x22b.google.com (mail-qc0-x22b.google.com [IPv6:2607:f8b0:400d:c01::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E3ED82475 for ; Tue, 23 Jul 2013 04:05:56 +0000 (UTC) Received: by mail-qc0-f171.google.com with SMTP id n1so4039044qcw.2 for ; Mon, 22 Jul 2013 21:05:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=yRR0CtI8dfK++c14hZZBjR5s2mGmBJK29XjDtJ58opI=; b=t0utmy1RKEHNKhm3DWX9Q/t+9G09uffPYoZPAlB/FHcPqbdlPNl9H+D8oF82iNhTKO D6ET+GuQGhX7FQ8kOoWOl5RV9VzmFCRftJb/iRUeN51fe8+ZxC5m0UpB9fKpGKu974VX v5eI1zQdsnM2IG/9+W/OFEQgEIANQ15vjPL2tMSZXUNfPc2jGLXD7IypELAbEZr2KuXs h/njYQ6BsuJybR/x3Z8uEv0CbBtc2MXxqJgxuZuWNekf/YdvzIhPxmFCFPS0VVMHfE5p S4+OT2eEv9XttK8G53A7ng9iWnHR7WZZchmKEtEXtpIlGRGuA5AHNeqjR1rFy6EkEHbv Gb7w== MIME-Version: 1.0 X-Received: by 10.49.58.70 with SMTP id o6mr35650783qeq.1.1374552355987; Mon, 22 Jul 2013 21:05:55 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.195.72 with HTTP; Mon, 22 Jul 2013 21:05:55 -0700 (PDT) In-Reply-To: <51EDA37A.9040200@mail.ru> References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> <20130722200205.GO26412@funkthat.com> <51EDA37A.9040200@mail.ru> Date: Mon, 22 Jul 2013 21:05:55 -0700 X-Google-Sender-Auth: C8yt9vWBVQBYb5LDgfaKkbR5cgU Message-ID: Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour From: Adrian Chadd To: trafdev Content-Type: text/plain; charset=ISO-8859-1 Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, John-Mark Gurney X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 04:05:57 -0000 On 22 July 2013 14:26, trafdev wrote: > Actually overhead is almost zero, the real problem is in non-equivalent load > distribution between processes. > As https://lwn.net/Articles/542629/ mentions - > "At Google, they have seen a factor-of-three difference between the thread > accepting the most connections and the thread accepting the fewest > connections;" > I'm getting almost same results On FreeBSD? -adrian From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 05:44:24 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 74104A1F for ; Tue, 23 Jul 2013 05:44:24 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-vc0-x22c.google.com (mail-vc0-x22c.google.com [IPv6:2607:f8b0:400c:c03::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 358BB27C9 for ; Tue, 23 Jul 2013 05:44:24 +0000 (UTC) Received: by mail-vc0-f172.google.com with SMTP id m17so3757059vca.17 for ; Mon, 22 Jul 2013 22:44:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=cISGNvoK951GGniZZ/3sEy9n54yP/FUFrX0Y6zAYLF8=; b=azZ16USVUI1S9cbcIcz8858XWE0njxOVnMJ4Y7Bs/FxchDusGIAZrE0E5eYdqtOnXQ sfW+vWqxZ0cIuTwuNRqExX+H16dvFJKoq4cabs31ZauhzFpg0sd10VD/dq/M+CLecOIK 2guuZbCv/MDpiSGRgEy40HeMtedoDGQNN41chSbZXUIZ0Or2um2V99UJsSyQqa14s2r9 n6QC9teVhNWDQ72zvJ78c0KTxMQaZrdBJ2YNLFDJpD8fEXuwjHMANb647VXWwJ+y3bVK BwzrdcQ4emqtZZtD7ta2Y3lKE5oj2PFHl1jXshk5mvoohOhz3F9jw1OFPKadiyFEIwTX 4BWQ== MIME-Version: 1.0 X-Received: by 10.220.42.140 with SMTP id s12mr2876264vce.4.1374558263288; Mon, 22 Jul 2013 22:44:23 -0700 (PDT) Received: by 10.221.22.199 with HTTP; Mon, 22 Jul 2013 22:44:23 -0700 (PDT) In-Reply-To: References: <06BA4BD5-BE4E-4184-AFBB-D7FD4B2597D9@your.org> Date: Tue, 23 Jul 2013 01:44:23 -0400 Message-ID: Subject: Re: Duplicate Address Detection misfire? From: Zaphod Beeblebrox To: Kevin Day Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 05:44:24 -0000 What to do when you don't trust the interface? VMWare is obviously emulating the hardware and their interpretation of what the hardware "is" is possibly different from ours. If I boot single-user and tcpdump the interface, I see two transmitted solicitations. The kernel claims to have sent one. My concern: is the vmware interface reflecting the solicitation packet because it is a broadcast packet? To determine this, I've gone over the tcpdump and pcap-filter man pages to look for a way to only dump packets leaving from or arriving at an interface. Can this be done? If VMWare is reflecting the packet back, I'm curious as to how we can fix this. On Mon, Jul 22, 2013 at 1:22 AM, Zaphod Beeblebrox wrote: > I've further narrowed this down. According to the output: > > em0 DAD detected duplicate IPv6 address fe80:2::250:56ff:fe2e:45fd: NS > in/out=2/1, NA in=0 > > ... FreeBSD *thinks* it has transmitted one and received 2 solicitations. > The packet dump shows two solicitations (which would, if it were not bogus, > indicate that another machine was booting at the exact same time trying to > use the same link-local address). > > The question becomes: is vmware duplicating the packet, or is FreeBSD? > IE: driver problem with em0 and vmware? > > I'm not completely sure how to debug this. > > > > On Thu, Jul 18, 2013 at 9:21 PM, Zaphod Beeblebrox wrote: > >> >> >> >> On Sun, Jun 30, 2013 at 10:39 PM, Kevin Day wrote: >> >>> >>> On Jun 30, 2013, at 6:48 PM, Zaphod Beeblebrox >>> wrote: >>> >>> > I have a FreeBSD 9.1-RELEASE vmware guest running. It is using the >>> > "bridged" type of networking with VMWare. It gets it's IPv4 address >>> from >>> > DHCP (successfully) and then fails to initialize IPv6. The relevant >>> > rc.conf is: >>> > >>> > ipv6_activate_all_interfaces="YES" >>> > ifconfig_em0_ipv6="inet6 accept_rtadv" >>> > ip6addrctl_verbose="YES" >>> > >>> > The console output says: >>> > >>> > em0: DAD detected duplicate IPv6 address fe80:2::20c:29ff:fe0a:3989: NS >>> > in/out=2/1, NA in=0 >>> > em0: DAD complete for fe80:2::20c:29ff:fe0a:3989 - duplicate found >>> > em0: manual intervention required >>> > em0: possible hardware address duplication deteted, disable IPv6 >>> > >>> > And subsequently, em0's nd6 has "IFDISABLED" in it. >>> > >>> > With wireshark, I see two ICMPv6 neighbor solicitations that are >>> identical >>> > --- is this the problem? >>> > >>> > How do I fix this? >>> >>> Did you copy this VM and have both copies running at once? If so, it >>> assigned the same MAC address to each VM. >>> >>> VMware is suppose to detect this and ask if you "copied" or "moved" the >>> VM, and if you say "copied" it will randomly assign a new MAC to the VM. If >>> this didn't happen or if you said "moved" when you actually copied it, just >>> go in and delete/re-create the network interface in the VM's settings to >>> create a new MAC for it. >>> >>> If that's not the issue, we'd probably need more details about your >>> configuration. >>> >> >> To further diagnose, there is only one VM running. To ensure that there >> were no duplicates, I reassigned the MAC address in the VMWare >> configuration dialogue. Additionally, I tried stopping rtadvd on my router >> (no effect) and I tried putting the guest on a "host-only network" >> (basically isolated it) --- this clears the problem --- both the link-local >> and the static address are assigned. >> >> Frustrated, I dumped the windows interface that is bridged to the VMWare >> guest. When it boots, I see the following: >> >> 2461 19:24:16.376027000 Vmware_2e:46:fd Broadcast ARP >> 42 Gratuitous ARP for 66.96.20.42 (Request) >> 2462 19:24:16.388241000 :: ff02::1:ff00:42 ICMPv6 78 >> Neighbor Solicitation for 2001:1928:1::42 >> 2463 19:24:16.389065000 :: ff02::1:ff00:42 ICMPv6 78 >> Neighbor Solicitation for 2001:1928:1::42 >> 2464 19:24:16.444130000 :: ff02::16 ICMPv6 130 >> Multicast Listener Report Message v2 >> 2465 19:24:16.444605000 :: ff02::16 ICMPv6 130 >> Multicast Listener Report Message v2 >> 2466 19:24:16.594663000 :: ff02::1:ff2e:46fd ICMPv6 78 >> Neighbor Solicitation for fe80::250:56ff:fe2e:46fd >> 2467 19:24:16.595179000 :: ff02::1:ff2e:46fd ICMPv6 78 >> Neighbor Solicitation for fe80::250:56ff:fe2e:46fd >> 2753 19:24:22.274728000 Vmware_2e:46:fd Broadcast ARP >> 42 Who has 66.96.20.33? Tell 66.96.20.42 >> 2754 19:24:22.274902000 Intel_bc:6f:87 Vmware_2e:46:fd ARP 60 >> 66.96.20.33 is at 00:0e:0c:bc:6f:87 >> >> ... and then it goes on to chatter ipv4-wise as expected. Note that >> there are two of each packet. Is that normal? The ethernet source of all >> these packets is my vmware guest (save the who-has reply that I copied in). >> >> > From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 05:51:02 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 22047B83; Tue, 23 Jul 2013 05:51:02 +0000 (UTC) (envelope-from trafdev@mail.ru) Received: from fallback3.mail.ru (fallback3.mail.ru [94.100.176.58]) by mx1.freebsd.org (Postfix) with ESMTP id C9919281C; Tue, 23 Jul 2013 05:51:01 +0000 (UTC) Received: from smtp37.i.mail.ru (smtp37.i.mail.ru [94.100.177.97]) by fallback3.mail.ru (mPOP.Fallback_MX) with ESMTP id D64A7EF74CB3; Tue, 23 Jul 2013 09:50:15 +0400 (MSK) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:Subject:CC:To:MIME-Version:From:Date:Message-ID; bh=wa1+BW5odCevwmqORacUOSEh7tRiDGeFgtw7/kh4JZM=; b=oOj2lyr27/o+LGVIhSefBHuzEjXTJW718OpldzDmTqPfQBePWHI6RYvGUUNgC0nCCzl8O0P39j8CfsqXXv48bytmDXwiS7T0rJc0xuRiF7JepyQNAJAk6eXLuazjYLhw9QSsQUIsVCiSz9vhp060jhVdEptF0IvQLQx9GtUarJw=; Received: from [50.156.108.197] (port=48064 helo=[192.168.1.116]) by smtp37.i.mail.ru with esmtpa (envelope-from ) id 1V1VU6-00043s-UZ; Tue, 23 Jul 2013 09:50:07 +0400 Message-ID: <51EE198B.7040509@mail.ru> Date: Mon, 22 Jul 2013 22:50:03 -0700 From: trafdev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130628 Thunderbird/17.0.7 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> <20130722200205.GO26412@funkthat.com> <51EDA37A.9040200@mail.ru> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, John-Mark Gurney X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 05:51:02 -0000 yep, FreeBSD 9.1-RELEASE-p3 On Mon Jul 22 21:05:55 2013, Adrian Chadd wrote: > On 22 July 2013 14:26, trafdev wrote: >> Actually overhead is almost zero, the real problem is in non-equivalent load >> distribution between processes. >> As https://lwn.net/Articles/542629/ mentions - >> "At Google, they have seen a factor-of-three difference between the thread >> accepting the most connections and the thread accepting the fewest >> connections;" >> I'm getting almost same results > > On FreeBSD? > > > -adrian > > From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 06:15:12 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C52A9126 for ; Tue, 23 Jul 2013 06:15:12 +0000 (UTC) (envelope-from kpaasial@gmail.com) Received: from mail-qa0-x236.google.com (mail-qa0-x236.google.com [IPv6:2607:f8b0:400d:c00::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8B5C82929 for ; Tue, 23 Jul 2013 06:15:12 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id bv4so1622446qab.13 for ; Mon, 22 Jul 2013 23:15:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=JLORlYBqMRki0FIbpkqvTACjIv11gil5JS/QOP1osVc=; b=hOrUa6GnPwNmuu184po1RQms3+yxNlppV8qX2/aWUs+P0o5Unm+xRFLRSXKlfL/Q0/ +2Jpz6pthfZx+Sfrwh+dnhCYZAaOlBpvXjv3q7Rfjl1UGzJcYNP3+bEW+GSbMYUklGDP 0T+S5qEfVXHRycjAuFBOy1AXvG/0vWDBOYuBf4X5BW1IIMV2JrhLjJ9DPf0VK1vquCg2 YTx9L7C9pj2vaxGdqHs3L6EVS/WIbT/EsQ0yLbbo4Bu+a++Xdb5LBCm5EHAcKWTmYXUt KVRGhCUy+PYUs4QLlWGLU67+hdV5XtTohX/GFkVBqSVEERwjfK1u18xfMYzkGjr1Imj7 lLZw== MIME-Version: 1.0 X-Received: by 10.49.127.4 with SMTP id nc4mr36725444qeb.41.1374560111681; Mon, 22 Jul 2013 23:15:11 -0700 (PDT) Received: by 10.224.78.194 with HTTP; Mon, 22 Jul 2013 23:15:11 -0700 (PDT) In-Reply-To: References: <06BA4BD5-BE4E-4184-AFBB-D7FD4B2597D9@your.org> Date: Tue, 23 Jul 2013 09:15:11 +0300 Message-ID: Subject: Re: Duplicate Address Detection misfire? From: Kimmo Paasiala To: Zaphod Beeblebrox Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD Net , Kevin Day X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 06:15:12 -0000 On Tue, Jul 23, 2013 at 8:44 AM, Zaphod Beeblebrox wrote: > What to do when you don't trust the interface? VMWare is obviously > emulating the hardware and their interpretation of what the hardware "is" > is possibly different from ours. > > If I boot single-user and tcpdump the interface, I see two transmitted > solicitations. The kernel claims to have sent one. > > My concern: is the vmware interface reflecting the solicitation packet > because it is a broadcast packet? > > To determine this, I've gone over the tcpdump and pcap-filter man pages to > look for a way to only dump packets leaving from or arriving at an > interface. Can this be done? > > If VMWare is reflecting the packet back, I'm curious as to how we can fix > this. > > That sounds exactly like my experience with DAD misbehaving on my Zyxel WAP3205 access point. It is reflecting the multicasts (I hope that's the right term) so that any IPv6 equipment on the wireless network will think that its address is already in use. For the record, the client machines in my case are all OS X. Nice to know I'm not the only one with such problems. -Kimmo From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 06:37:02 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DB38736E for ; Tue, 23 Jul 2013 06:37:02 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qe0-x230.google.com (mail-qe0-x230.google.com [IPv6:2607:f8b0:400d:c02::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9FDA129DB for ; Tue, 23 Jul 2013 06:37:02 +0000 (UTC) Received: by mail-qe0-f48.google.com with SMTP id 2so4202838qea.21 for ; Mon, 22 Jul 2013 23:37:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=InzdxXPnySkthfEl10UivxP5yiVmriS02doDtT+teHI=; b=x8ly3bGu4J2Rz9/PbKSffV6aF5vgmKr+K7aQZRoqlNHAQyjRgp+dC/ZqLz72YDhwY5 gP2u+g9WBFGLOGGfIGHd0xaxZPb5BieXlHGS96IJM4mP+uMR/6EWsA9/LHXu1Xkmumko 2WqOIiS0fsDbPpdSNO0l0E6RfBLn1NKoi7StSTmpMdDH5Doz3/xESdIbdbpyO5m6kS55 IJdiApzzifY/ETW+vxdFhmYY3zlbpEFmzxRXodEDA+jhQ0SN2mzaEGHT3vGNpOVIGsf9 UwDS8P7RJHe/vhXa00TheGagD0fOiCkSfOey5PB6FOzN3HKNN3jX9TVs1AA280xIgmxg ODqw== MIME-Version: 1.0 X-Received: by 10.49.58.70 with SMTP id o6mr36076831qeq.1.1374561421676; Mon, 22 Jul 2013 23:37:01 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.195.72 with HTTP; Mon, 22 Jul 2013 23:37:01 -0700 (PDT) In-Reply-To: <51EE198B.7040509@mail.ru> References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> <20130722200205.GO26412@funkthat.com> <51EDA37A.9040200@mail.ru> <51EE198B.7040509@mail.ru> Date: Mon, 22 Jul 2013 23:37:01 -0700 X-Google-Sender-Auth: 7f7KqpYLi8ngH_EqjEg4VX0YXlA Message-ID: Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour From: Adrian Chadd To: trafdev Content-Type: text/plain; charset=ISO-8859-1 Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, John-Mark Gurney X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 06:37:02 -0000 Sweet, have any code you can share that can elicit this? I'm writing a network / disk application layer right now so i can model/test this stuff out. I'd love to see some pointers/example code. Thanks! -adrian From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 06:37:58 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8F2A441F for ; Tue, 23 Jul 2013 06:37:58 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qc0-x22f.google.com (mail-qc0-x22f.google.com [IPv6:2607:f8b0:400d:c01::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 52FC729F1 for ; Tue, 23 Jul 2013 06:37:58 +0000 (UTC) Received: by mail-qc0-f175.google.com with SMTP id k14so4098511qcv.20 for ; Mon, 22 Jul 2013 23:37:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=dsqzoP4tm0aa+xFLVklWN5Bu0JRe/iasFfHZvC06Tzw=; b=IPTQaQ4Wq/id1X7nRb3549GFxHGOk+mpKqUcYfSi3BBtlOs58Z2bztVJQy5ovnV0+E dxkTB5ngILNGYIIyZR5j7WiFc10GQY1a6QEIjgvdsI8dj+x7yTLo5BjdMaLTqzJv2dgp xw//puWqpUcdlm+3xp2wkV3fNeyExmVdPHpIifGJ2NPHjDibp5uiB03HKu9dWh9IpNW7 y6GXsYuDR26Durr8CWgRDAMTZXejUtNnhua7LcQptCXhqVYZJ9oE/AjhrfdYCuPL+DDx 7AVZ6oPX1PM9KhbKlL7tTvhXw39qwkBmJjgF8wqKj5VXEKjCwImj9W/0w4ZV3PA52tqL kgBQ== MIME-Version: 1.0 X-Received: by 10.49.117.195 with SMTP id kg3mr36538602qeb.68.1374561477425; Mon, 22 Jul 2013 23:37:57 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.195.72 with HTTP; Mon, 22 Jul 2013 23:37:57 -0700 (PDT) In-Reply-To: References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> <20130722200205.GO26412@funkthat.com> <51EDA37A.9040200@mail.ru> <51EE198B.7040509@mail.ru> Date: Mon, 22 Jul 2013 23:37:57 -0700 X-Google-Sender-Auth: TPxBrXG239fmEnAIkVgsBd1aNq0 Message-ID: Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour From: Adrian Chadd To: trafdev Content-Type: text/plain; charset=ISO-8859-1 Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, John-Mark Gurney X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 06:37:58 -0000 eg: * one process, one listen thread, multiple dispatch threads? * n processes, one listen FD per process, all listening on the same IP:port? * one process, each thread listening on the same IP:port? * something else? Thanks, -adrian From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 07:09:45 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E989ABE0; Tue, 23 Jul 2013 07:09:44 +0000 (UTC) (envelope-from trafdev@mail.ru) Received: from smtp47.i.mail.ru (smtp47.i.mail.ru [94.100.177.107]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6598F2B02; Tue, 23 Jul 2013 07:09:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:Subject:CC:To:MIME-Version:From:Date:Message-ID; bh=TqOcck8PeG6L1MCeZW1/SzPqbuR3Ey8z390dZgAPZxw=; b=ZaLjEzUSRDGs6GCWEb1KdUEulym9P9WPZhd7hG/gpLVyDVbROhEjLtOeYoaZK5YJAZQuiDqB2m5UQjiuCdzluhRWA6Furrnm7lez3rAj0E2sZgFfXThorwxnTvUyOx/GZAT+CxwAiBjWQF5a7AN2ayYZaijK5WRad3aVslHCE0I=; Received: from [50.156.108.197] (port=25693 helo=[192.168.1.116]) by smtp47.i.mail.ru with esmtpa (envelope-from ) id 1V1Wj1-0000Pu-Ff; Tue, 23 Jul 2013 11:09:35 +0400 Message-ID: <51EE2C2B.4020800@mail.ru> Date: Tue, 23 Jul 2013 00:09:31 -0700 From: trafdev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130628 Thunderbird/17.0.7 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> <20130722200205.GO26412@funkthat.com> <51EDA37A.9040200@mail.ru> <51EE198B.7040509@mail.ru> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, John-Mark Gurney X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 07:09:45 -0000 It's like shared acceptor FD and N processes: Listening proc: bool HttpServer::Listen(unsigned short port, uint listen_backlog) { LOG4CXX_TRACE(kLogger, "Listen"); if ((sockd_acceptor_ = socket(AF_INET, SOCK_STREAM, 0)) == -1) { LOG4CXX_ERROR_ERRNO(kLogger, "socket"); return false; } struct sockaddr_in sa_in; memset(&sa_in, 0, sizeof(sa_in)); sa_in.sin_family = AF_INET; sa_in.sin_port = htons(port); sa_in.sin_addr.s_addr = htonl(INADDR_ANY); int yes = 1; if (setsockopt(sockd_acceptor_, SOL_SOCKET, SO_REUSEPORT, &yes, sizeof (yes)) == -1) { LOG4CXX_ERROR_ERRNO(kLogger, "setsockopt"); return false; } if (bind(sockd_acceptor_, (const struct sockaddr*)&sa_in, sizeof(sa_in)) == -1) { LOG4CXX_ERROR_ERRNO(kLogger, "bind"); return false; } if (listen(sockd_acceptor_, listen_backlog) == -1) { LOG4CXX_ERROR_ERRNO(kLogger, "socket listen"); return false; } if (!fcntl_set(sockd_acceptor_, O_NONBLOCK)) return false; sockd_acceptor_watcher_.set (this); sockd_acceptor_watcher_.start(sockd_acceptor_, ev::READ); int wrk = thread::hardware_concurrency(); while (--wrk) { pid_t pid = fork(); if (pid == 0) { LOG4CXX_INFO(kLogger, "process " << wrk << " started"); ev::default_loop().post_fork(); ev::default_loop().run(); return true; } } ev::default_loop().run(); return true; } Accept conn callback: void HttpServer::AcceptConnCb(ev::io &w, int revents) { LOG4CXX_TRACE(kLogger, "AcceptConnCb"); CHECK_LIBEV_ERROR struct sockaddr_in addr; socklen_t addr_len = sizeof (addr); int sockd; if ((sockd = accept(sockd_acceptor_, (struct sockaddr *) &addr, &addr_len)) == -1) { if (errno == EAGAIN || errno == EWOULDBLOCK) { return; } else { LOG4CXX_ERROR_ERRNO(kLogger, "accept"); return; } } if (!fcntl_set(sockd, O_NONBLOCK)) return; char str[INET_ADDRSTRLEN]; if (!inet_ntop(AF_INET, &(addr.sin_addr), str, INET_ADDRSTRLEN)) { LOG4CXX_ERROR(kLogger, "inet_ntop"); return; } // Internal logic follows... ConnIn* c = ConnPool::Instance().Add(sockd, str); // c->Recv(this); return; } Accept conn callback is called in N processes on each connection, only one wins, others exit by errno == EAGAIN case. Overhead is almost zero. Problem is that "wins" distribution is far from equal. On Mon Jul 22 23:37:57 2013, Adrian Chadd wrote: > eg: > > * one process, one listen thread, multiple dispatch threads? > * n processes, one listen FD per process, all listening on the same IP:port? > * one process, each thread listening on the same IP:port? > * something else? > > Thanks, > > > > -adrian > > From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 11:39:14 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 776E2CD1 for ; Tue, 23 Jul 2013 11:39:14 +0000 (UTC) (envelope-from sr@swisscenter.com) Received: from mail.swisslink.ch (mail.swisslink.ch [94.103.96.90]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EBF42294C for ; Tue, 23 Jul 2013 11:39:13 +0000 (UTC) Received: from [10.8.0.14] (gate.swisslink.ch [62.2.195.10]) (authenticated bits=0) by mail.swisslink.ch (8.14.3/8.14.3/Debian-9.4) with ESMTP id r6NBSkxU016434 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 23 Jul 2013 13:28:46 +0200 Message-ID: <51EE68E9.5010805@swisscenter.com> Date: Tue, 23 Jul 2013 13:28:41 +0200 From: =?ISO-8859-1?Q?S=E9bastien_RICCIO?= User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:23.0) Gecko/20100101 Thunderbird/23.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: FreeBSD 9.1 and BCM57711 issues (broadcom 10ge ethernet card) Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: clamav-milter 0.97.8 at mail X-Virus-Status: Clean X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 11:39:14 -0000 Hi freebsd-net! We recently installed FreeBSD 9.1 64bit on a Dell PowerEdge R510 system in which we have two BCM57711 (for a total of four 10Gbit interfaces.) We're planning to use it as a storage filer using ZFS/NFS. Actually in test, the filer is connected with two 10gigs interfaces to a 10ge Dell PowerConnect switch that serves some linux clients using 10ge cards too. We get into a lot of troubles trying to get something working out of this setup. -- First issue: Without any special tweaking, when we're reading or writing to the NFS server from a client, the network card crashes and become. In the logs I can see: Jul 19 11:49:26 filer-01-a kernel: bxe0: ---------- Begin crash dump ---------- Jul 19 11:49:26 filer-01-a kernel: bxe0: ------------------------------ Idle Check ------------------------------ Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CFC: AC > 1 - LCID 39 CID_CAM 0x7 Value is 0xc Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: VOQ_0, VOQ credit is not equal to initial credit. Values are 0xf8 0x140 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: P0 Byte credit is not equal to initial credit. Values are 0x5a1c 0x8000 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING CCM: XX protection CAM is not empty. Value is 0x1 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING XCM: XX protection CAM is not empty. Value is 0x1 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING BRB1: BRB is not empty. Value is 0x3 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING TCM: FIC0_INIT_CRD is not 64. Value is 0x30 Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR TSEM: interrupt status 0 is not 0. Value is 0x10000 Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CSEM: interrupt status 0 is not 0. Value is 0x10000 Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR XSEM: interrupt status 0 is not 0. Value is 0x10000 Jul 19 11:49:26 filer-01-a kernel: bxe0: bxe_idle_chk(): Failed with 4 error(s) and 0 warning(s)! Jul 19 11:49:26 filer-01-a kernel: bxe0: ------------------------------------------------------------------------ Jul 19 11:49:26 filer-01-a kernel: bxe0: ------------------------------ Idle Check ------------------------------ Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CFC: AC > 1 - LCID 39 CID_CAM 0x7 Value is 0xc Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: VOQ_0, VOQ credit is not equal to initial credit. Values are 0xf8 0x140 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: P0 Byte credit is not equal to initial credit. Values are 0x5a1c 0x8000 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING CCM: XX protection CAM is not empty. Value is 0x1 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING XCM: XX protection CAM is not empty. Value is 0x1 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING BRB1: BRB is not empty. Value is 0x4 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING TCM: FIC0_INIT_CRD is not 64. Value is 0x30 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING PRS: TCM current credit is not 0. Value is 0x10 Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR TSEM: interrupt status 0 is not 0. Value is 0x10000 Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CSEM: interrupt status 0 is not 0. Value is 0x10000 Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR XSEM: interrupt status 0 is not 0. Value is 0x10000 Jul 19 11:49:26 filer-01-a kernel: bxe0: bxe_idle_chk(): Failed with 4 error(s) and 0 warning(s)! Jul 19 11:49:26 filer-01-a kernel: bxe0: ------------------------------------------------------------------------ Jul 19 11:49:26 filer-01-a kernel: bxe0: ---------- End crash dump ---------- A reboot of the system is even not enough. After rebooting the system, I can't even ping any hosts on the network. It seems that it leaves the card in a bogus state that requires a complete power cycle to get the cards back in business. We found out that disabling: tso4 txcsum rxcsum on the cards prevent this from happening. So although I think it's not, let's say we have a fix for this setting in rc.conf something like this: ifconfig_bxe0="inet 10.50.50.11 netmask 255.255.255.0 mtu 9000 -tso4 -txcsum -rxcsum" -- Second issue, Issuing an ifconfig mtu 9000 on the interfaces randomly produce this error: Jul 19 09:47:03 filer-01-a kernel: bxe0: /usr/src/sys/dev/bxe/if_bxe.c(10934): Memory allocation failure! Cannot fill fp[04] RX chain. Jul 19 09:47:03 filer-01-a kernel: bxe0: /usr/src/sys/dev/bxe/if_bxe.c(3921): NIC initialization failed, aborting! Jul 19 09:47:12 filer-01-a kernel: bxe3: /usr/src/sys/dev/bxe/if_bxe.c(10934): Memory allocation failure! Cannot fill fp[04] RX chain. Jul 19 09:47:12 filer-01-a kernel: bxe3: /usr/src/sys/dev/bxe/if_bxe.c(3921): NIC initialization failed, aborting! That sounds quite bad and, I can't reproduce it with mtu 1500 setting. (But does it makes sens to use a MTU of 1500 on a 10gig local network...?) -- Third issue, part 1) We've tried two interfaces (each interface with an mtu of 9000) using lagg, like this: ifconfig bxe0 up -tso4 -txcsum -rxcsum mtu 9000 ifconfig bxe2 up -tso4 -txcsum -rxcsum mtu 9000 ifconfig lagg0 create ifconfig lagg0 up laggproto failover laggport bxe0 laggport bxe2 10.50.50.11/24 This instantanely crashes the kernel and cause a machine reboot. The log says: Jul 19 09:47:12 filer-01-a kernel: Jul 19 09:47:12 filer-01-a kernel: Jul 19 09:47:12 filer-01-a kernel: Fatal trap 12: page fault while in kernel mode Jul 19 09:47:12 filer-01-a kernel: cpuid = 0; apic id = 20 Jul 19 09:47:12 filer-01-a kernel: fault virtual address = 0x6d Jul 19 09:47:12 filer-01-a kernel: fault code = supervisor read data, page not present Jul 19 09:47:12 filer-01-a kernel: instruction pointer = 0x20:0xffffffff808d5879 Jul 19 09:47:12 filer-01-a kernel: stack pointer = 0x28:0xffffff80003227f0 --*** BOOOM REBOOT ***-- Jul 19 09:49:49 filer-01-a syslogd: kernel boot file is /boot/kernel/kernel /var/crash/core.txt.0 returns: Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 5; apic id = 33 fault virtual address = 0x6d fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff808d5879 stack pointer = 0x28:0xffffff80003227f0 frame pointer = 0x28:0xffffff8000322820 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (swi6: task queue) trap number = 12 panic: page fault cpuid = 5 KDB: stack backtrace: #0 0xffffffff809208a6 at kdb_backtrace+0x66 #1 0xffffffff808ea8be at panic+0x1ce #2 0xffffffff80bd8240 at trap_fatal+0x290 #3 0xffffffff80bd857d at trap_pfault+0x1ed #4 0xffffffff80bd8b9e at trap+0x3ce #5 0xffffffff80bc315f at calltrap+0x8 #6 0xffffffff8045da8c at bxe_free_buf_rings+0x4c #7 0xffffffff8046c0d5 at bxe_init_locked+0x125 #8 0xffffffff80470cfe at bxe_ioctl+0x4fe #9 0xffffffff8099d08f at if_setlladdr+0x1ff #10 0xffffffff8174c94a at lagg_port_setlladdr+0x8a #11 0xffffffff8092cf55 at taskqueue_run_locked+0x85 #12 0xffffffff8092d0da at taskqueue_run+0x3a #13 0xffffffff808be8d4 at intr_event_execute_handlers+0x104 #14 0xffffffff808c0076 at ithread_loop+0xa6 #15 0xffffffff808bb9ef at fork_exit+0x11f #16 0xffffffff80bc368e at fork_trampoline+0xe Uptime: 39m41s Dumping 1505 out of 32735 MB:..2%..11%..21%..31%..41%..52%..61%..71%..81%..91% Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/if_lagg.ko...Reading symbols from /boot/kernel/if_lagg.ko.symbols...done. done. Loaded symbols for /boot/kernel/if_lagg.ko #0 doadump (textdump=Variable "textdump" is not available. ) at pcpu.h:224 224 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=Variable "textdump" is not available. ) at pcpu.h:224 #1 0xffffffff808ea3a1 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:448 #2 0xffffffff808ea897 in panic (fmt=0x1
) at /usr/src/sys/kern/kern_shutdown.c:636 #3 0xffffffff80bd8240 in trap_fatal (frame=0xc, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:857 #4 0xffffffff80bd857d in trap_pfault (frame=0xffffff8000322740, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:773 #5 0xffffffff80bd8b9e in trap (frame=0xffffff8000322740) at /usr/src/sys/amd64/amd64/trap.c:456 #6 0xffffffff80bc315f in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228 #7 0xffffffff808d5879 in free (addr=0xffffff80083e5000, mtp=0xffffffff81198ba0) at uma_int.h:413 #8 0xffffffff8045da8c in bxe_free_buf_rings (sc=0xffffff8000c1c000) at /usr/src/sys/dev/bxe/if_bxe.c:3787 #9 0xffffffff8046c0d5 in bxe_init_locked (sc=0x0, load_mode=0) at /usr/src/sys/dev/bxe/if_bxe.c:4063 #10 0xffffffff80470cfe in bxe_ioctl (ifp=0xfffffe000ec59000, command=Variable "command" is not available. ) at /usr/src/sys/dev/bxe/if_bxe.c:9668 #11 0xffffffff8099d08f in if_setlladdr (ifp=0xfffffe000ec59000, lladdr=0xfffffe00125da4c8 "", len=6) at /usr/src/sys/net/if.c:3304 #12 0xffffffff8174c94a in lagg_port_setlladdr (arg=Variable "arg" is not available. ) at /usr/src/sys/modules/if_lagg/../../net/if_lagg.c:495 #13 0xffffffff8092cf55 in taskqueue_run_locked (queue=0xfffffe000e833980) at /usr/src/sys/kern/subr_taskqueue.c:308 #14 0xffffffff8092d0da in taskqueue_run (queue=0xfffffe000e833980) at /usr/src/sys/kern/subr_taskqueue.c:322 #15 0xffffffff808be8d4 in intr_event_execute_handlers (p=Variable "p" is not available. ) at /usr/src/sys/kern/kern_intr.c:1262 #16 0xffffffff808c0076 in ithread_loop (arg=0xfffffe000e66c140) at /usr/src/sys/kern/kern_intr.c:1275 #17 0xffffffff808bb9ef in fork_exit ( callout=0xffffffff808bffd0 , arg=0xfffffe000e66c140, frame=0xffffff8000322c40) at /usr/src/sys/kern/kern_fork.c:992 #18 0xffffffff80bc368e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:602 #19 0x0000000000000000 in ?? () #20 0x0000000000000000 in ?? () #21 0x0000000000000001 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000000000000 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () #37 0x0000000000000000 in ?? () #38 0x0000000000000000 in ?? () #39 0x0000000000000000 in ?? () #40 0x0000000000000000 in ?? () #41 0x0000000000000000 in ?? () #42 0x0000000000000000 in ?? () #43 0x0000000000000005 in ?? () #44 0xffffffff81244180 in tdq_cpu () #45 0xfffffe000e698000 in ?? () #46 0x0000000000000000 in ?? () #47 0xffffff8000322b30 in ?? () #48 0xffffff8000322ad8 in ?? () #49 0xfffffe000e6728e0 in ?? () #50 0xffffffff8091352e in sched_switch (td=0x0, newtd=0xfffffe000e66c140, flags=Variable "flags" is not available. ) at /usr/src/sys/kern/sched_ule.c:1921 Previous frame inner to this frame (corrupt stack?) (kgdb) Okay guess it has something to do again with the MTU 9000 but this time it does completly panic the kernel. This is no good. Part 2) Trying bonding with normal MTU 1500 ifconfig bxe0 up -tso4 -txcsum -rxcsum mtu 1500 ifconfig bxe2 up -tso4 -txcsum -rxcsum mtu 1500 ifconfig lagg0 create ifconfig lagg0 up laggproto failover laggport bxe0 laggport bxe2 10.50.50.11/24 This time. No error messages, no crash. Yiha! But no. Even everything seems to be correct, the bonding is not working. We can't ping any host on the network. Also the lagg0 says: No carrier see: bxe0: flags=8843 metric 0 mtu 1500 options=b8 ether 00:10:18:98:35:f8 inet6 fe80::210:18ff:fe98:35f8%bxe0 prefixlen 64 scopeid 0x3 nd6 options=29 media: Ethernet autoselect (10Gbase-SR ) status: active bxe2: flags=8843 metric 0 mtu 1500 options=b8 ether 00:10:18:98:35:f8 inet6 fe80::210:18ff:fe95:eaa0%bxe2 prefixlen 64 scopeid 0x5 nd6 options=29 media: Ethernet autoselect (10Gbase-SR ) status: active lagg0: flags=8843 metric 0 mtu 1500 options=b8 ether 00:10:18:98:35:f8 inet6 fe80::7a2b:cbff:fe1a:eab1%lagg0 prefixlen 64 scopeid 0x14 inet 10.50.50.11 netmask 0xffffff00 broadcast 10.50.50.255 nd6 options=21 media: Ethernet autoselect status: no carrier laggproto failover lagghash l2,l3,l4 laggport: bxe2 flags=0<> laggport: bxe0 flags=1 Please note that priore to installing freebsd, the machine was running a Debian 7 GNU/Linux 64 bit OS where we had the cards bonded and MTU'ed to 9000 without any crash or stability issue. So it looks to me that there is something really wrong with the broadcom driver on freebsd 9.1, at least with the NIC's used in Dell servers. Provided that broadcom themselves doesn't supply drivers for freebsd Is there any possible fix ? Thanks for your attention and your help. Cheers, Sébastien From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 11:55:45 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id C46BCF5D for ; Tue, 23 Jul 2013 11:55:45 +0000 (UTC) (envelope-from prvs=1916be8aae=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2FDAA29F4 for ; Tue, 23 Jul 2013 11:55:44 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50005115231.msg for ; Tue, 23 Jul 2013 12:55:41 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Tue, 23 Jul 2013 12:55:41 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=1916be8aae=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-net@freebsd.org Message-ID: <7D8CE344ACD04EC8B8C7DC813D1EA3F6@multiplay.co.uk> From: "Steven Hartland" To: =?iso-8859-1?Q?S=E9bastien_RICCIO?= , References: <51EE68E9.5010805@swisscenter.com> Subject: Re: FreeBSD 9.1 and BCM57711 issues (broadcom 10ge ethernet card) Date: Tue, 23 Jul 2013 12:56:12 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 11:55:45 -0000 Have you tried a more recent version e.g. 9.2-PRERELEASE or 9/stable? Regards Steve ----- Original Message ----- From: "Sébastien RICCIO" To: Sent: Tuesday, July 23, 2013 12:28 PM Subject: FreeBSD 9.1 and BCM57711 issues (broadcom 10ge ethernet card) Hi freebsd-net! We recently installed FreeBSD 9.1 64bit on a Dell PowerEdge R510 system in which we have two BCM57711 (for a total of four 10Gbit interfaces.) We're planning to use it as a storage filer using ZFS/NFS. Actually in test, the filer is connected with two 10gigs interfaces to a 10ge Dell PowerConnect switch that serves some linux clients using 10ge cards too. We get into a lot of troubles trying to get something working out of this setup. -- First issue: Without any special tweaking, when we're reading or writing to the NFS server from a client, the network card crashes and become. In the logs I can see: Jul 19 11:49:26 filer-01-a kernel: bxe0: ---------- Begin crash dump ---------- Jul 19 11:49:26 filer-01-a kernel: bxe0: ------------------------------ Idle Check ------------------------------ Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CFC: AC > 1 - LCID 39 CID_CAM 0x7 Value is 0xc Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: VOQ_0, VOQ credit is not equal to initial credit. Values are 0xf8 0x140 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: P0 Byte credit is not equal to initial credit. Values are 0x5a1c 0x8000 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING CCM: XX protection CAM is not empty. Value is 0x1 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING XCM: XX protection CAM is not empty. Value is 0x1 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING BRB1: BRB is not empty. Value is 0x3 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING TCM: FIC0_INIT_CRD is not 64. Value is 0x30 Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR TSEM: interrupt status 0 is not 0. Value is 0x10000 Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CSEM: interrupt status 0 is not 0. Value is 0x10000 Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR XSEM: interrupt status 0 is not 0. Value is 0x10000 Jul 19 11:49:26 filer-01-a kernel: bxe0: bxe_idle_chk(): Failed with 4 error(s) and 0 warning(s)! Jul 19 11:49:26 filer-01-a kernel: bxe0: ------------------------------------------------------------------------ Jul 19 11:49:26 filer-01-a kernel: bxe0: ------------------------------ Idle Check ------------------------------ Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CFC: AC > 1 - LCID 39 CID_CAM 0x7 Value is 0xc Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: VOQ_0, VOQ credit is not equal to initial credit. Values are 0xf8 0x140 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: P0 Byte credit is not equal to initial credit. Values are 0x5a1c 0x8000 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING CCM: XX protection CAM is not empty. Value is 0x1 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING XCM: XX protection CAM is not empty. Value is 0x1 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING BRB1: BRB is not empty. Value is 0x4 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING TCM: FIC0_INIT_CRD is not 64. Value is 0x30 Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING PRS: TCM current credit is not 0. Value is 0x10 Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR TSEM: interrupt status 0 is not 0. Value is 0x10000 Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CSEM: interrupt status 0 is not 0. Value is 0x10000 Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR XSEM: interrupt status 0 is not 0. Value is 0x10000 Jul 19 11:49:26 filer-01-a kernel: bxe0: bxe_idle_chk(): Failed with 4 error(s) and 0 warning(s)! Jul 19 11:49:26 filer-01-a kernel: bxe0: ------------------------------------------------------------------------ Jul 19 11:49:26 filer-01-a kernel: bxe0: ---------- End crash dump ---------- A reboot of the system is even not enough. After rebooting the system, I can't even ping any hosts on the network. It seems that it leaves the card in a bogus state that requires a complete power cycle to get the cards back in business. We found out that disabling: tso4 txcsum rxcsum on the cards prevent this from happening. So although I think it's not, let's say we have a fix for this setting in rc.conf something like this: ifconfig_bxe0="inet 10.50.50.11 netmask 255.255.255.0 mtu 9000 -tso4 -txcsum -rxcsum" -- Second issue, Issuing an ifconfig mtu 9000 on the interfaces randomly produce this error: Jul 19 09:47:03 filer-01-a kernel: bxe0: /usr/src/sys/dev/bxe/if_bxe.c(10934): Memory allocation failure! Cannot fill fp[04] RX chain. Jul 19 09:47:03 filer-01-a kernel: bxe0: /usr/src/sys/dev/bxe/if_bxe.c(3921): NIC initialization failed, aborting! Jul 19 09:47:12 filer-01-a kernel: bxe3: /usr/src/sys/dev/bxe/if_bxe.c(10934): Memory allocation failure! Cannot fill fp[04] RX chain. Jul 19 09:47:12 filer-01-a kernel: bxe3: /usr/src/sys/dev/bxe/if_bxe.c(3921): NIC initialization failed, aborting! That sounds quite bad and, I can't reproduce it with mtu 1500 setting. (But does it makes sens to use a MTU of 1500 on a 10gig local network...?) -- Third issue, part 1) We've tried two interfaces (each interface with an mtu of 9000) using lagg, like this: ifconfig bxe0 up -tso4 -txcsum -rxcsum mtu 9000 ifconfig bxe2 up -tso4 -txcsum -rxcsum mtu 9000 ifconfig lagg0 create ifconfig lagg0 up laggproto failover laggport bxe0 laggport bxe2 10.50.50.11/24 This instantanely crashes the kernel and cause a machine reboot. The log says: Jul 19 09:47:12 filer-01-a kernel: Jul 19 09:47:12 filer-01-a kernel: Jul 19 09:47:12 filer-01-a kernel: Fatal trap 12: page fault while in kernel mode Jul 19 09:47:12 filer-01-a kernel: cpuid = 0; apic id = 20 Jul 19 09:47:12 filer-01-a kernel: fault virtual address = 0x6d Jul 19 09:47:12 filer-01-a kernel: fault code = supervisor read data, page not present Jul 19 09:47:12 filer-01-a kernel: instruction pointer = 0x20:0xffffffff808d5879 Jul 19 09:47:12 filer-01-a kernel: stack pointer = 0x28:0xffffff80003227f0 --*** BOOOM REBOOT ***-- Jul 19 09:49:49 filer-01-a syslogd: kernel boot file is /boot/kernel/kernel /var/crash/core.txt.0 returns: Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 5; apic id = 33 fault virtual address = 0x6d fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff808d5879 stack pointer = 0x28:0xffffff80003227f0 frame pointer = 0x28:0xffffff8000322820 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (swi6: task queue) trap number = 12 panic: page fault cpuid = 5 KDB: stack backtrace: #0 0xffffffff809208a6 at kdb_backtrace+0x66 #1 0xffffffff808ea8be at panic+0x1ce #2 0xffffffff80bd8240 at trap_fatal+0x290 #3 0xffffffff80bd857d at trap_pfault+0x1ed #4 0xffffffff80bd8b9e at trap+0x3ce #5 0xffffffff80bc315f at calltrap+0x8 #6 0xffffffff8045da8c at bxe_free_buf_rings+0x4c #7 0xffffffff8046c0d5 at bxe_init_locked+0x125 #8 0xffffffff80470cfe at bxe_ioctl+0x4fe #9 0xffffffff8099d08f at if_setlladdr+0x1ff #10 0xffffffff8174c94a at lagg_port_setlladdr+0x8a #11 0xffffffff8092cf55 at taskqueue_run_locked+0x85 #12 0xffffffff8092d0da at taskqueue_run+0x3a #13 0xffffffff808be8d4 at intr_event_execute_handlers+0x104 #14 0xffffffff808c0076 at ithread_loop+0xa6 #15 0xffffffff808bb9ef at fork_exit+0x11f #16 0xffffffff80bc368e at fork_trampoline+0xe Uptime: 39m41s Dumping 1505 out of 32735 MB:..2%..11%..21%..31%..41%..52%..61%..71%..81%..91% Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/if_lagg.ko...Reading symbols from /boot/kernel/if_lagg.ko.symbols...done. done. Loaded symbols for /boot/kernel/if_lagg.ko #0 doadump (textdump=Variable "textdump" is not available. ) at pcpu.h:224 224 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=Variable "textdump" is not available. ) at pcpu.h:224 #1 0xffffffff808ea3a1 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:448 #2 0xffffffff808ea897 in panic (fmt=0x1
) at /usr/src/sys/kern/kern_shutdown.c:636 #3 0xffffffff80bd8240 in trap_fatal (frame=0xc, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:857 #4 0xffffffff80bd857d in trap_pfault (frame=0xffffff8000322740, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:773 #5 0xffffffff80bd8b9e in trap (frame=0xffffff8000322740) at /usr/src/sys/amd64/amd64/trap.c:456 #6 0xffffffff80bc315f in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228 #7 0xffffffff808d5879 in free (addr=0xffffff80083e5000, mtp=0xffffffff81198ba0) at uma_int.h:413 #8 0xffffffff8045da8c in bxe_free_buf_rings (sc=0xffffff8000c1c000) at /usr/src/sys/dev/bxe/if_bxe.c:3787 #9 0xffffffff8046c0d5 in bxe_init_locked (sc=0x0, load_mode=0) at /usr/src/sys/dev/bxe/if_bxe.c:4063 #10 0xffffffff80470cfe in bxe_ioctl (ifp=0xfffffe000ec59000, command=Variable "command" is not available. ) at /usr/src/sys/dev/bxe/if_bxe.c:9668 #11 0xffffffff8099d08f in if_setlladdr (ifp=0xfffffe000ec59000, lladdr=0xfffffe00125da4c8 "", len=6) at /usr/src/sys/net/if.c:3304 #12 0xffffffff8174c94a in lagg_port_setlladdr (arg=Variable "arg" is not available. ) at /usr/src/sys/modules/if_lagg/../../net/if_lagg.c:495 #13 0xffffffff8092cf55 in taskqueue_run_locked (queue=0xfffffe000e833980) at /usr/src/sys/kern/subr_taskqueue.c:308 #14 0xffffffff8092d0da in taskqueue_run (queue=0xfffffe000e833980) at /usr/src/sys/kern/subr_taskqueue.c:322 #15 0xffffffff808be8d4 in intr_event_execute_handlers (p=Variable "p" is not available. ) at /usr/src/sys/kern/kern_intr.c:1262 #16 0xffffffff808c0076 in ithread_loop (arg=0xfffffe000e66c140) at /usr/src/sys/kern/kern_intr.c:1275 #17 0xffffffff808bb9ef in fork_exit ( callout=0xffffffff808bffd0 , arg=0xfffffe000e66c140, frame=0xffffff8000322c40) at /usr/src/sys/kern/kern_fork.c:992 #18 0xffffffff80bc368e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:602 #19 0x0000000000000000 in ?? () #20 0x0000000000000000 in ?? () #21 0x0000000000000001 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000000000000 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () #37 0x0000000000000000 in ?? () #38 0x0000000000000000 in ?? () #39 0x0000000000000000 in ?? () #40 0x0000000000000000 in ?? () #41 0x0000000000000000 in ?? () #42 0x0000000000000000 in ?? () #43 0x0000000000000005 in ?? () #44 0xffffffff81244180 in tdq_cpu () #45 0xfffffe000e698000 in ?? () #46 0x0000000000000000 in ?? () #47 0xffffff8000322b30 in ?? () #48 0xffffff8000322ad8 in ?? () #49 0xfffffe000e6728e0 in ?? () #50 0xffffffff8091352e in sched_switch (td=0x0, newtd=0xfffffe000e66c140, flags=Variable "flags" is not available. ) at /usr/src/sys/kern/sched_ule.c:1921 Previous frame inner to this frame (corrupt stack?) (kgdb) Okay guess it has something to do again with the MTU 9000 but this time it does completly panic the kernel. This is no good. Part 2) Trying bonding with normal MTU 1500 ifconfig bxe0 up -tso4 -txcsum -rxcsum mtu 1500 ifconfig bxe2 up -tso4 -txcsum -rxcsum mtu 1500 ifconfig lagg0 create ifconfig lagg0 up laggproto failover laggport bxe0 laggport bxe2 10.50.50.11/24 This time. No error messages, no crash. Yiha! But no. Even everything seems to be correct, the bonding is not working. We can't ping any host on the network. Also the lagg0 says: No carrier see: bxe0: flags=8843 metric 0 mtu 1500 options=b8 ether 00:10:18:98:35:f8 inet6 fe80::210:18ff:fe98:35f8%bxe0 prefixlen 64 scopeid 0x3 nd6 options=29 media: Ethernet autoselect (10Gbase-SR ) status: active bxe2: flags=8843 metric 0 mtu 1500 options=b8 ether 00:10:18:98:35:f8 inet6 fe80::210:18ff:fe95:eaa0%bxe2 prefixlen 64 scopeid 0x5 nd6 options=29 media: Ethernet autoselect (10Gbase-SR ) status: active lagg0: flags=8843 metric 0 mtu 1500 options=b8 ether 00:10:18:98:35:f8 inet6 fe80::7a2b:cbff:fe1a:eab1%lagg0 prefixlen 64 scopeid 0x14 inet 10.50.50.11 netmask 0xffffff00 broadcast 10.50.50.255 nd6 options=21 media: Ethernet autoselect status: no carrier laggproto failover lagghash l2,l3,l4 laggport: bxe2 flags=0<> laggport: bxe0 flags=1 Please note that priore to installing freebsd, the machine was running a Debian 7 GNU/Linux 64 bit OS where we had the cards bonded and MTU'ed to 9000 without any crash or stability issue. So it looks to me that there is something really wrong with the broadcom driver on freebsd 9.1, at least with the NIC's used in Dell servers. Provided that broadcom themselves doesn't supply drivers for freebsd Is there any possible fix ? Thanks for your attention and your help. Cheers, Sébastien _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 12:10:59 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id C4D32A83 for ; Tue, 23 Jul 2013 12:10:59 +0000 (UTC) (envelope-from sr@swisscenter.com) Received: from mail.swisslink.ch (mail.swisslink.ch [94.103.96.90]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2AFE82AF6 for ; Tue, 23 Jul 2013 12:10:58 +0000 (UTC) Received: from [10.8.0.14] (gate.swisslink.ch [62.2.195.10]) (authenticated bits=0) by mail.swisslink.ch (8.14.3/8.14.3/Debian-9.4) with ESMTP id r6NCAYNc019699 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 23 Jul 2013 14:10:34 +0200 Message-ID: <51EE72B6.6020905@swisscenter.com> Date: Tue, 23 Jul 2013 14:10:30 +0200 From: =?ISO-8859-1?Q?S=E9bastien_RICCIO?= User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:23.0) Gecko/20100101 Thunderbird/23.0 MIME-Version: 1.0 To: Steven Hartland , freebsd-net@freebsd.org Subject: Re: FreeBSD 9.1 and BCM57711 issues (broadcom 10ge ethernet card) References: <51EE68E9.5010805@swisscenter.com> <7D8CE344ACD04EC8B8C7DC813D1EA3F6@multiplay.co.uk> In-Reply-To: <7D8CE344ACD04EC8B8C7DC813D1EA3F6@multiplay.co.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: clamav-milter 0.97.8 at mail X-Virus-Status: Clean X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 12:10:59 -0000 Hi Steve, Not yet. As the gol was to prepare a system to be production ready, I went with the latest stable... well at least I thought that 9.1 was the latest stable. I'm going to try it. But that would mean that the driver in the 9.2-prerelease kernel has been modified. Going to take a look at the change logs if I can find it :) Thanks for your help. Cheers, Sébastien On 23.07.2013 13:56, Steven Hartland wrote: > Have you tried a more recent version e.g. 9.2-PRERELEASE or 9/stable? > > Regards > Steve > > ----- Original Message ----- From: "Sébastien RICCIO" > > To: > Sent: Tuesday, July 23, 2013 12:28 PM > Subject: FreeBSD 9.1 and BCM57711 issues (broadcom 10ge ethernet card) > > > Hi freebsd-net! > > We recently installed FreeBSD 9.1 64bit on a Dell PowerEdge R510 system > in which we have two BCM57711 (for a total of four 10Gbit interfaces.) > > We're planning to use it as a storage filer using ZFS/NFS. > > Actually in test, the filer is connected with two 10gigs interfaces to a > 10ge Dell PowerConnect switch that serves some linux clients using 10ge > cards too. > > We get into a lot of troubles trying to get something working out of > this setup. > > -- > > First issue: > > Without any special tweaking, when we're reading or writing to the NFS > server from a client, the network card crashes and become. In the logs I > can see: > > Jul 19 11:49:26 filer-01-a kernel: bxe0: ---------- Begin crash dump > ---------- > Jul 19 11:49:26 filer-01-a kernel: bxe0: > ------------------------------ Idle Check ------------------------------ > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CFC: AC > 1 - LCID 39 > CID_CAM 0x7 Value is 0xc > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: VOQ_0, VOQ credit > is not equal to initial credit. Values are 0xf8 0x140 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: P0 Byte credit is > not equal to initial credit. Values are 0x5a1c 0x8000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING CCM: XX protection CAM > is not empty. Value is 0x1 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING XCM: XX protection CAM > is not empty. Value is 0x1 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING BRB1: BRB is not empty. > Value is 0x3 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING TCM: FIC0_INIT_CRD is > not 64. Value is 0x30 > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR TSEM: interrupt status 0 > is not 0. Value is 0x10000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CSEM: interrupt status 0 > is not 0. Value is 0x10000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR XSEM: interrupt status 0 > is not 0. Value is 0x10000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: bxe_idle_chk(): Failed with 4 > error(s) and 0 warning(s)! > Jul 19 11:49:26 filer-01-a kernel: bxe0: > ------------------------------------------------------------------------ > Jul 19 11:49:26 filer-01-a kernel: bxe0: > ------------------------------ Idle Check ------------------------------ > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CFC: AC > 1 - LCID 39 > CID_CAM 0x7 Value is 0xc > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: VOQ_0, VOQ credit > is not equal to initial credit. Values are 0xf8 0x140 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: P0 Byte credit is > not equal to initial credit. Values are 0x5a1c 0x8000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING CCM: XX protection CAM > is not empty. Value is 0x1 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING XCM: XX protection CAM > is not empty. Value is 0x1 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING BRB1: BRB is not empty. > Value is 0x4 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING TCM: FIC0_INIT_CRD is > not 64. Value is 0x30 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING PRS: TCM current credit > is not 0. Value is 0x10 > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR TSEM: interrupt status 0 > is not 0. Value is 0x10000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CSEM: interrupt status 0 > is not 0. Value is 0x10000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR XSEM: interrupt status 0 > is not 0. Value is 0x10000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: bxe_idle_chk(): Failed with 4 > error(s) and 0 warning(s)! > Jul 19 11:49:26 filer-01-a kernel: bxe0: > ------------------------------------------------------------------------ > Jul 19 11:49:26 filer-01-a kernel: bxe0: ---------- End crash dump > ---------- > > A reboot of the system is even not enough. After rebooting the system, I > can't even ping any hosts on the network. It seems that it leaves the > card in a bogus state that requires a complete power cycle to get the > cards back in business. > > We found out that disabling: tso4 txcsum rxcsum on the cards prevent > this from happening. > > So although I think it's not, let's say we have a fix for this setting > in rc.conf something like this: > ifconfig_bxe0="inet 10.50.50.11 netmask 255.255.255.0 mtu 9000 -tso4 > -txcsum -rxcsum" > > -- > > Second issue, > > Issuing an ifconfig mtu 9000 on the interfaces randomly produce this > error: > > Jul 19 09:47:03 filer-01-a kernel: bxe0: > /usr/src/sys/dev/bxe/if_bxe.c(10934): Memory allocation failure! Cannot > fill fp[04] RX chain. > Jul 19 09:47:03 filer-01-a kernel: bxe0: > /usr/src/sys/dev/bxe/if_bxe.c(3921): NIC initialization failed, aborting! > Jul 19 09:47:12 filer-01-a kernel: bxe3: > /usr/src/sys/dev/bxe/if_bxe.c(10934): Memory allocation failure! Cannot > fill fp[04] RX chain. > Jul 19 09:47:12 filer-01-a kernel: bxe3: > /usr/src/sys/dev/bxe/if_bxe.c(3921): NIC initialization failed, aborting! > > That sounds quite bad and, I can't reproduce it with mtu 1500 setting. > (But does it makes sens to use a MTU of 1500 on a 10gig local > network...?) > > -- > > Third issue, > > part 1) > > We've tried two interfaces (each interface with an mtu of 9000) using > lagg, like this: > > ifconfig bxe0 up -tso4 -txcsum -rxcsum mtu 9000 > ifconfig bxe2 up -tso4 -txcsum -rxcsum mtu 9000 > ifconfig lagg0 create > ifconfig lagg0 up laggproto failover laggport bxe0 laggport bxe2 > 10.50.50.11/24 > > This instantanely crashes the kernel and cause a machine reboot. The log > says: > > Jul 19 09:47:12 filer-01-a kernel: > Jul 19 09:47:12 filer-01-a kernel: > Jul 19 09:47:12 filer-01-a kernel: Fatal trap 12: page fault while in > kernel mode > Jul 19 09:47:12 filer-01-a kernel: cpuid = 0; apic id = 20 > Jul 19 09:47:12 filer-01-a kernel: fault virtual address = 0x6d > Jul 19 09:47:12 filer-01-a kernel: fault code = supervisor > read data, page not present > Jul 19 09:47:12 filer-01-a kernel: instruction pointer = > 0x20:0xffffffff808d5879 > Jul 19 09:47:12 filer-01-a kernel: stack pointer = > 0x28:0xffffff80003227f0 > --*** BOOOM REBOOT ***-- > Jul 19 09:49:49 filer-01-a syslogd: kernel boot file is > /boot/kernel/kernel > > /var/crash/core.txt.0 returns: > > Unread portion of the kernel message buffer: > Fatal trap 12: page fault while in kernel mode > cpuid = 5; apic id = 33 > fault virtual address = 0x6d > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff808d5879 > stack pointer = 0x28:0xffffff80003227f0 > frame pointer = 0x28:0xffffff8000322820 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 12 (swi6: task queue) > trap number = 12 > panic: page fault > cpuid = 5 > KDB: stack backtrace: > #0 0xffffffff809208a6 at kdb_backtrace+0x66 > #1 0xffffffff808ea8be at panic+0x1ce > #2 0xffffffff80bd8240 at trap_fatal+0x290 > #3 0xffffffff80bd857d at trap_pfault+0x1ed > #4 0xffffffff80bd8b9e at trap+0x3ce > #5 0xffffffff80bc315f at calltrap+0x8 > #6 0xffffffff8045da8c at bxe_free_buf_rings+0x4c > #7 0xffffffff8046c0d5 at bxe_init_locked+0x125 > #8 0xffffffff80470cfe at bxe_ioctl+0x4fe > #9 0xffffffff8099d08f at if_setlladdr+0x1ff > #10 0xffffffff8174c94a at lagg_port_setlladdr+0x8a > #11 0xffffffff8092cf55 at taskqueue_run_locked+0x85 > #12 0xffffffff8092d0da at taskqueue_run+0x3a > #13 0xffffffff808be8d4 at intr_event_execute_handlers+0x104 > #14 0xffffffff808c0076 at ithread_loop+0xa6 > #15 0xffffffff808bb9ef at fork_exit+0x11f > #16 0xffffffff80bc368e at fork_trampoline+0xe > Uptime: 39m41s > Dumping 1505 out of 32735 > MB:..2%..11%..21%..31%..41%..52%..61%..71%..81%..91% > > Reading symbols from /boot/kernel/zfs.ko...Reading symbols from > /boot/kernel/zfs.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/zfs.ko > Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from > /boot/kernel/opensolaris.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/opensolaris.ko > Reading symbols from /boot/kernel/if_lagg.ko...Reading symbols from > /boot/kernel/if_lagg.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/if_lagg.ko > #0 doadump (textdump=Variable "textdump" is not available. > ) at pcpu.h:224 > 224 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) #0 doadump (textdump=Variable "textdump" is not available. > ) at pcpu.h:224 > #1 0xffffffff808ea3a1 in kern_reboot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:448 > #2 0xffffffff808ea897 in panic (fmt=0x1
) > at /usr/src/sys/kern/kern_shutdown.c:636 > #3 0xffffffff80bd8240 in trap_fatal (frame=0xc, eva=Variable "eva" is > not available. > ) > at /usr/src/sys/amd64/amd64/trap.c:857 > #4 0xffffffff80bd857d in trap_pfault (frame=0xffffff8000322740, > usermode=0) > at /usr/src/sys/amd64/amd64/trap.c:773 > #5 0xffffffff80bd8b9e in trap (frame=0xffffff8000322740) > at /usr/src/sys/amd64/amd64/trap.c:456 > #6 0xffffffff80bc315f in calltrap () > at /usr/src/sys/amd64/amd64/exception.S:228 > #7 0xffffffff808d5879 in free (addr=0xffffff80083e5000, > mtp=0xffffffff81198ba0) at uma_int.h:413 > #8 0xffffffff8045da8c in bxe_free_buf_rings (sc=0xffffff8000c1c000) > at /usr/src/sys/dev/bxe/if_bxe.c:3787 > #9 0xffffffff8046c0d5 in bxe_init_locked (sc=0x0, load_mode=0) > at /usr/src/sys/dev/bxe/if_bxe.c:4063 > #10 0xffffffff80470cfe in bxe_ioctl (ifp=0xfffffe000ec59000, > command=Variable "command" is not available. > ) > at /usr/src/sys/dev/bxe/if_bxe.c:9668 > #11 0xffffffff8099d08f in if_setlladdr (ifp=0xfffffe000ec59000, > lladdr=0xfffffe00125da4c8 "", len=6) at /usr/src/sys/net/if.c:3304 > #12 0xffffffff8174c94a in lagg_port_setlladdr (arg=Variable "arg" is not > available. > ) > at /usr/src/sys/modules/if_lagg/../../net/if_lagg.c:495 > #13 0xffffffff8092cf55 in taskqueue_run_locked (queue=0xfffffe000e833980) > at /usr/src/sys/kern/subr_taskqueue.c:308 > #14 0xffffffff8092d0da in taskqueue_run (queue=0xfffffe000e833980) > at /usr/src/sys/kern/subr_taskqueue.c:322 > #15 0xffffffff808be8d4 in intr_event_execute_handlers (p=Variable "p" is > not available. > ) > at /usr/src/sys/kern/kern_intr.c:1262 > #16 0xffffffff808c0076 in ithread_loop (arg=0xfffffe000e66c140) > at /usr/src/sys/kern/kern_intr.c:1275 > #17 0xffffffff808bb9ef in fork_exit ( > callout=0xffffffff808bffd0 , arg=0xfffffe000e66c140, > frame=0xffffff8000322c40) at /usr/src/sys/kern/kern_fork.c:992 > #18 0xffffffff80bc368e in fork_trampoline () > at /usr/src/sys/amd64/amd64/exception.S:602 > #19 0x0000000000000000 in ?? () > #20 0x0000000000000000 in ?? () > #21 0x0000000000000001 in ?? () > #22 0x0000000000000000 in ?? () > #23 0x0000000000000000 in ?? () > #24 0x0000000000000000 in ?? () > #25 0x0000000000000000 in ?? () > #26 0x0000000000000000 in ?? () > #27 0x0000000000000000 in ?? () > #28 0x0000000000000000 in ?? () > #29 0x0000000000000000 in ?? () > #30 0x0000000000000000 in ?? () > #31 0x0000000000000000 in ?? () > #32 0x0000000000000000 in ?? () > #33 0x0000000000000000 in ?? () > #34 0x0000000000000000 in ?? () > #35 0x0000000000000000 in ?? () > #36 0x0000000000000000 in ?? () > #37 0x0000000000000000 in ?? () > #38 0x0000000000000000 in ?? () > #39 0x0000000000000000 in ?? () > #40 0x0000000000000000 in ?? () > #41 0x0000000000000000 in ?? () > #42 0x0000000000000000 in ?? () > #43 0x0000000000000005 in ?? () > #44 0xffffffff81244180 in tdq_cpu () > #45 0xfffffe000e698000 in ?? () > #46 0x0000000000000000 in ?? () > #47 0xffffff8000322b30 in ?? () > #48 0xffffff8000322ad8 in ?? () > #49 0xfffffe000e6728e0 in ?? () > #50 0xffffffff8091352e in sched_switch (td=0x0, newtd=0xfffffe000e66c140, > flags=Variable "flags" is not available. > ) at /usr/src/sys/kern/sched_ule.c:1921 > Previous frame inner to this frame (corrupt stack?) > (kgdb) > > Okay guess it has something to do again with the MTU 9000 but this time > it does completly panic the kernel. This is no good. > > > Part 2) Trying bonding with normal MTU 1500 > > ifconfig bxe0 up -tso4 -txcsum -rxcsum mtu 1500 > ifconfig bxe2 up -tso4 -txcsum -rxcsum mtu 1500 > ifconfig lagg0 create > ifconfig lagg0 up laggproto failover laggport bxe0 laggport bxe2 > 10.50.50.11/24 > > This time. No error messages, no crash. Yiha! > > But no. Even everything seems to be correct, the bonding is not working. > We can't ping any host on the network. > Also the lagg0 says: No carrier > > see: > > bxe0: flags=8843 metric 0 mtu > 1500 > options=b8 > ether 00:10:18:98:35:f8 > inet6 fe80::210:18ff:fe98:35f8%bxe0 prefixlen 64 scopeid 0x3 > nd6 options=29 > media: Ethernet autoselect (10Gbase-SR ) > status: active > bxe2: flags=8843 metric 0 mtu > 1500 > options=b8 > ether 00:10:18:98:35:f8 > inet6 fe80::210:18ff:fe95:eaa0%bxe2 prefixlen 64 scopeid 0x5 > nd6 options=29 > media: Ethernet autoselect (10Gbase-SR ) > status: active > lagg0: flags=8843 metric 0 mtu > 1500 > options=b8 > ether 00:10:18:98:35:f8 > inet6 fe80::7a2b:cbff:fe1a:eab1%lagg0 prefixlen 64 scopeid 0x14 > inet 10.50.50.11 netmask 0xffffff00 broadcast 10.50.50.255 > nd6 options=21 > media: Ethernet autoselect > status: no carrier > laggproto failover lagghash l2,l3,l4 > laggport: bxe2 flags=0<> > laggport: bxe0 flags=1 > > Please note that priore to installing freebsd, the machine was running a > Debian 7 GNU/Linux 64 bit OS where we had the cards bonded and MTU'ed to > 9000 without any crash or stability issue. > So it looks to me that there is something really wrong with the broadcom > driver on freebsd 9.1, at least with the NIC's used in Dell servers. > > Provided that broadcom themselves doesn't supply drivers for freebsd Is > there any possible fix ? > > Thanks for your attention and your help. > > Cheers, > Sébastien > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > ================================================ > This e.mail is private and confidential between Multiplay (UK) Ltd. > and the person or entity to whom it is addressed. In the event of > misdirection, the recipient is prohibited from using, copying, > printing or otherwise disseminating it or any information contained in > it. > In the event of misdirection, illegible or incomplete transmission > please telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. > From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 12:50:06 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id F3AD3677 for ; Tue, 23 Jul 2013 12:50:05 +0000 (UTC) (envelope-from sr@swisscenter.com) Received: from mail.swisslink.ch (mail.swisslink.ch [94.103.96.90]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5D7322CC8 for ; Tue, 23 Jul 2013 12:50:04 +0000 (UTC) Received: from [10.8.0.14] (gate.swisslink.ch [62.2.195.10]) (authenticated bits=0) by mail.swisslink.ch (8.14.3/8.14.3/Debian-9.4) with ESMTP id r6NCnbrr023061 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 23 Jul 2013 14:49:37 +0200 Message-ID: <51EE7BDD.2050700@swisscenter.com> Date: Tue, 23 Jul 2013 14:49:33 +0200 From: =?ISO-8859-1?Q?S=E9bastien_RICCIO?= User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:23.0) Gecko/20100101 Thunderbird/23.0 MIME-Version: 1.0 To: Steven Hartland , freebsd-net@freebsd.org Subject: Re: FreeBSD 9.1 and BCM57711 issues (broadcom 10ge ethernet card) References: <51EE68E9.5010805@swisscenter.com> <7D8CE344ACD04EC8B8C7DC813D1EA3F6@multiplay.co.uk> In-Reply-To: <7D8CE344ACD04EC8B8C7DC813D1EA3F6@multiplay.co.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: clamav-milter 0.97.8 at mail X-Virus-Status: Clean X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 12:50:06 -0000 root@filer-01-a:/# freebsd-update -r 9.2-BETA1 upgrade Looking up update.FreeBSD.org mirrors... 4 mirrors found. Fetching metadata signature for 9.1-RELEASE from update4.freebsd.org... done. Fetching metadata index... done. Inspecting system... done. The following components of FreeBSD seem to be installed: kernel/generic src/src world/base world/lib32 The following components of FreeBSD do not seem to be installed: world/doc world/games Does this look reasonable (y/n)? y Fetching metadata signature for 9.2-BETA1 from update4.freebsd.org... done. Fetching metadata index... done. The update metadata is correctly signed, but failed an integrity check. Cowardly refusing to proceed any further. Cowardly ? :) On 23.07.2013 13:56, Steven Hartland wrote: > Have you tried a more recent version e.g. 9.2-PRERELEASE or 9/stable? > > Regards > Steve > > ----- Original Message ----- From: "Sébastien RICCIO" > > To: > Sent: Tuesday, July 23, 2013 12:28 PM > Subject: FreeBSD 9.1 and BCM57711 issues (broadcom 10ge ethernet card) > > > Hi freebsd-net! > > We recently installed FreeBSD 9.1 64bit on a Dell PowerEdge R510 system > in which we have two BCM57711 (for a total of four 10Gbit interfaces.) > > We're planning to use it as a storage filer using ZFS/NFS. > > Actually in test, the filer is connected with two 10gigs interfaces to a > 10ge Dell PowerConnect switch that serves some linux clients using 10ge > cards too. > > We get into a lot of troubles trying to get something working out of > this setup. > > -- > > First issue: > > Without any special tweaking, when we're reading or writing to the NFS > server from a client, the network card crashes and become. In the logs I > can see: > > Jul 19 11:49:26 filer-01-a kernel: bxe0: ---------- Begin crash dump > ---------- > Jul 19 11:49:26 filer-01-a kernel: bxe0: > ------------------------------ Idle Check ------------------------------ > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CFC: AC > 1 - LCID 39 > CID_CAM 0x7 Value is 0xc > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: VOQ_0, VOQ credit > is not equal to initial credit. Values are 0xf8 0x140 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: P0 Byte credit is > not equal to initial credit. Values are 0x5a1c 0x8000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING CCM: XX protection CAM > is not empty. Value is 0x1 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING XCM: XX protection CAM > is not empty. Value is 0x1 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING BRB1: BRB is not empty. > Value is 0x3 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING TCM: FIC0_INIT_CRD is > not 64. Value is 0x30 > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR TSEM: interrupt status 0 > is not 0. Value is 0x10000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CSEM: interrupt status 0 > is not 0. Value is 0x10000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR XSEM: interrupt status 0 > is not 0. Value is 0x10000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: bxe_idle_chk(): Failed with 4 > error(s) and 0 warning(s)! > Jul 19 11:49:26 filer-01-a kernel: bxe0: > ------------------------------------------------------------------------ > Jul 19 11:49:26 filer-01-a kernel: bxe0: > ------------------------------ Idle Check ------------------------------ > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CFC: AC > 1 - LCID 39 > CID_CAM 0x7 Value is 0xc > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: VOQ_0, VOQ credit > is not equal to initial credit. Values are 0xf8 0x140 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING QM: P0 Byte credit is > not equal to initial credit. Values are 0x5a1c 0x8000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING CCM: XX protection CAM > is not empty. Value is 0x1 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING XCM: XX protection CAM > is not empty. Value is 0x1 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING BRB1: BRB is not empty. > Value is 0x4 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING TCM: FIC0_INIT_CRD is > not 64. Value is 0x30 > Jul 19 11:49:26 filer-01-a kernel: bxe0: WARNING PRS: TCM current credit > is not 0. Value is 0x10 > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR TSEM: interrupt status 0 > is not 0. Value is 0x10000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR CSEM: interrupt status 0 > is not 0. Value is 0x10000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: ERROR XSEM: interrupt status 0 > is not 0. Value is 0x10000 > Jul 19 11:49:26 filer-01-a kernel: bxe0: bxe_idle_chk(): Failed with 4 > error(s) and 0 warning(s)! > Jul 19 11:49:26 filer-01-a kernel: bxe0: > ------------------------------------------------------------------------ > Jul 19 11:49:26 filer-01-a kernel: bxe0: ---------- End crash dump > ---------- > > A reboot of the system is even not enough. After rebooting the system, I > can't even ping any hosts on the network. It seems that it leaves the > card in a bogus state that requires a complete power cycle to get the > cards back in business. > > We found out that disabling: tso4 txcsum rxcsum on the cards prevent > this from happening. > > So although I think it's not, let's say we have a fix for this setting > in rc.conf something like this: > ifconfig_bxe0="inet 10.50.50.11 netmask 255.255.255.0 mtu 9000 -tso4 > -txcsum -rxcsum" > > -- > > Second issue, > > Issuing an ifconfig mtu 9000 on the interfaces randomly produce this > error: > > Jul 19 09:47:03 filer-01-a kernel: bxe0: > /usr/src/sys/dev/bxe/if_bxe.c(10934): Memory allocation failure! Cannot > fill fp[04] RX chain. > Jul 19 09:47:03 filer-01-a kernel: bxe0: > /usr/src/sys/dev/bxe/if_bxe.c(3921): NIC initialization failed, aborting! > Jul 19 09:47:12 filer-01-a kernel: bxe3: > /usr/src/sys/dev/bxe/if_bxe.c(10934): Memory allocation failure! Cannot > fill fp[04] RX chain. > Jul 19 09:47:12 filer-01-a kernel: bxe3: > /usr/src/sys/dev/bxe/if_bxe.c(3921): NIC initialization failed, aborting! > > That sounds quite bad and, I can't reproduce it with mtu 1500 setting. > (But does it makes sens to use a MTU of 1500 on a 10gig local > network...?) > > -- > > Third issue, > > part 1) > > We've tried two interfaces (each interface with an mtu of 9000) using > lagg, like this: > > ifconfig bxe0 up -tso4 -txcsum -rxcsum mtu 9000 > ifconfig bxe2 up -tso4 -txcsum -rxcsum mtu 9000 > ifconfig lagg0 create > ifconfig lagg0 up laggproto failover laggport bxe0 laggport bxe2 > 10.50.50.11/24 > > This instantanely crashes the kernel and cause a machine reboot. The log > says: > > Jul 19 09:47:12 filer-01-a kernel: > Jul 19 09:47:12 filer-01-a kernel: > Jul 19 09:47:12 filer-01-a kernel: Fatal trap 12: page fault while in > kernel mode > Jul 19 09:47:12 filer-01-a kernel: cpuid = 0; apic id = 20 > Jul 19 09:47:12 filer-01-a kernel: fault virtual address = 0x6d > Jul 19 09:47:12 filer-01-a kernel: fault code = supervisor > read data, page not present > Jul 19 09:47:12 filer-01-a kernel: instruction pointer = > 0x20:0xffffffff808d5879 > Jul 19 09:47:12 filer-01-a kernel: stack pointer = > 0x28:0xffffff80003227f0 > --*** BOOOM REBOOT ***-- > Jul 19 09:49:49 filer-01-a syslogd: kernel boot file is > /boot/kernel/kernel > > /var/crash/core.txt.0 returns: > > Unread portion of the kernel message buffer: > Fatal trap 12: page fault while in kernel mode > cpuid = 5; apic id = 33 > fault virtual address = 0x6d > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff808d5879 > stack pointer = 0x28:0xffffff80003227f0 > frame pointer = 0x28:0xffffff8000322820 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 12 (swi6: task queue) > trap number = 12 > panic: page fault > cpuid = 5 > KDB: stack backtrace: > #0 0xffffffff809208a6 at kdb_backtrace+0x66 > #1 0xffffffff808ea8be at panic+0x1ce > #2 0xffffffff80bd8240 at trap_fatal+0x290 > #3 0xffffffff80bd857d at trap_pfault+0x1ed > #4 0xffffffff80bd8b9e at trap+0x3ce > #5 0xffffffff80bc315f at calltrap+0x8 > #6 0xffffffff8045da8c at bxe_free_buf_rings+0x4c > #7 0xffffffff8046c0d5 at bxe_init_locked+0x125 > #8 0xffffffff80470cfe at bxe_ioctl+0x4fe > #9 0xffffffff8099d08f at if_setlladdr+0x1ff > #10 0xffffffff8174c94a at lagg_port_setlladdr+0x8a > #11 0xffffffff8092cf55 at taskqueue_run_locked+0x85 > #12 0xffffffff8092d0da at taskqueue_run+0x3a > #13 0xffffffff808be8d4 at intr_event_execute_handlers+0x104 > #14 0xffffffff808c0076 at ithread_loop+0xa6 > #15 0xffffffff808bb9ef at fork_exit+0x11f > #16 0xffffffff80bc368e at fork_trampoline+0xe > Uptime: 39m41s > Dumping 1505 out of 32735 > MB:..2%..11%..21%..31%..41%..52%..61%..71%..81%..91% > > Reading symbols from /boot/kernel/zfs.ko...Reading symbols from > /boot/kernel/zfs.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/zfs.ko > Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from > /boot/kernel/opensolaris.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/opensolaris.ko > Reading symbols from /boot/kernel/if_lagg.ko...Reading symbols from > /boot/kernel/if_lagg.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/if_lagg.ko > #0 doadump (textdump=Variable "textdump" is not available. > ) at pcpu.h:224 > 224 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) #0 doadump (textdump=Variable "textdump" is not available. > ) at pcpu.h:224 > #1 0xffffffff808ea3a1 in kern_reboot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:448 > #2 0xffffffff808ea897 in panic (fmt=0x1
) > at /usr/src/sys/kern/kern_shutdown.c:636 > #3 0xffffffff80bd8240 in trap_fatal (frame=0xc, eva=Variable "eva" is > not available. > ) > at /usr/src/sys/amd64/amd64/trap.c:857 > #4 0xffffffff80bd857d in trap_pfault (frame=0xffffff8000322740, > usermode=0) > at /usr/src/sys/amd64/amd64/trap.c:773 > #5 0xffffffff80bd8b9e in trap (frame=0xffffff8000322740) > at /usr/src/sys/amd64/amd64/trap.c:456 > #6 0xffffffff80bc315f in calltrap () > at /usr/src/sys/amd64/amd64/exception.S:228 > #7 0xffffffff808d5879 in free (addr=0xffffff80083e5000, > mtp=0xffffffff81198ba0) at uma_int.h:413 > #8 0xffffffff8045da8c in bxe_free_buf_rings (sc=0xffffff8000c1c000) > at /usr/src/sys/dev/bxe/if_bxe.c:3787 > #9 0xffffffff8046c0d5 in bxe_init_locked (sc=0x0, load_mode=0) > at /usr/src/sys/dev/bxe/if_bxe.c:4063 > #10 0xffffffff80470cfe in bxe_ioctl (ifp=0xfffffe000ec59000, > command=Variable "command" is not available. > ) > at /usr/src/sys/dev/bxe/if_bxe.c:9668 > #11 0xffffffff8099d08f in if_setlladdr (ifp=0xfffffe000ec59000, > lladdr=0xfffffe00125da4c8 "", len=6) at /usr/src/sys/net/if.c:3304 > #12 0xffffffff8174c94a in lagg_port_setlladdr (arg=Variable "arg" is not > available. > ) > at /usr/src/sys/modules/if_lagg/../../net/if_lagg.c:495 > #13 0xffffffff8092cf55 in taskqueue_run_locked (queue=0xfffffe000e833980) > at /usr/src/sys/kern/subr_taskqueue.c:308 > #14 0xffffffff8092d0da in taskqueue_run (queue=0xfffffe000e833980) > at /usr/src/sys/kern/subr_taskqueue.c:322 > #15 0xffffffff808be8d4 in intr_event_execute_handlers (p=Variable "p" is > not available. > ) > at /usr/src/sys/kern/kern_intr.c:1262 > #16 0xffffffff808c0076 in ithread_loop (arg=0xfffffe000e66c140) > at /usr/src/sys/kern/kern_intr.c:1275 > #17 0xffffffff808bb9ef in fork_exit ( > callout=0xffffffff808bffd0 , arg=0xfffffe000e66c140, > frame=0xffffff8000322c40) at /usr/src/sys/kern/kern_fork.c:992 > #18 0xffffffff80bc368e in fork_trampoline () > at /usr/src/sys/amd64/amd64/exception.S:602 > #19 0x0000000000000000 in ?? () > #20 0x0000000000000000 in ?? () > #21 0x0000000000000001 in ?? () > #22 0x0000000000000000 in ?? () > #23 0x0000000000000000 in ?? () > #24 0x0000000000000000 in ?? () > #25 0x0000000000000000 in ?? () > #26 0x0000000000000000 in ?? () > #27 0x0000000000000000 in ?? () > #28 0x0000000000000000 in ?? () > #29 0x0000000000000000 in ?? () > #30 0x0000000000000000 in ?? () > #31 0x0000000000000000 in ?? () > #32 0x0000000000000000 in ?? () > #33 0x0000000000000000 in ?? () > #34 0x0000000000000000 in ?? () > #35 0x0000000000000000 in ?? () > #36 0x0000000000000000 in ?? () > #37 0x0000000000000000 in ?? () > #38 0x0000000000000000 in ?? () > #39 0x0000000000000000 in ?? () > #40 0x0000000000000000 in ?? () > #41 0x0000000000000000 in ?? () > #42 0x0000000000000000 in ?? () > #43 0x0000000000000005 in ?? () > #44 0xffffffff81244180 in tdq_cpu () > #45 0xfffffe000e698000 in ?? () > #46 0x0000000000000000 in ?? () > #47 0xffffff8000322b30 in ?? () > #48 0xffffff8000322ad8 in ?? () > #49 0xfffffe000e6728e0 in ?? () > #50 0xffffffff8091352e in sched_switch (td=0x0, newtd=0xfffffe000e66c140, > flags=Variable "flags" is not available. > ) at /usr/src/sys/kern/sched_ule.c:1921 > Previous frame inner to this frame (corrupt stack?) > (kgdb) > > Okay guess it has something to do again with the MTU 9000 but this time > it does completly panic the kernel. This is no good. > > > Part 2) Trying bonding with normal MTU 1500 > > ifconfig bxe0 up -tso4 -txcsum -rxcsum mtu 1500 > ifconfig bxe2 up -tso4 -txcsum -rxcsum mtu 1500 > ifconfig lagg0 create > ifconfig lagg0 up laggproto failover laggport bxe0 laggport bxe2 > 10.50.50.11/24 > > This time. No error messages, no crash. Yiha! > > But no. Even everything seems to be correct, the bonding is not working. > We can't ping any host on the network. > Also the lagg0 says: No carrier > > see: > > bxe0: flags=8843 metric 0 mtu > 1500 > options=b8 > ether 00:10:18:98:35:f8 > inet6 fe80::210:18ff:fe98:35f8%bxe0 prefixlen 64 scopeid 0x3 > nd6 options=29 > media: Ethernet autoselect (10Gbase-SR ) > status: active > bxe2: flags=8843 metric 0 mtu > 1500 > options=b8 > ether 00:10:18:98:35:f8 > inet6 fe80::210:18ff:fe95:eaa0%bxe2 prefixlen 64 scopeid 0x5 > nd6 options=29 > media: Ethernet autoselect (10Gbase-SR ) > status: active > lagg0: flags=8843 metric 0 mtu > 1500 > options=b8 > ether 00:10:18:98:35:f8 > inet6 fe80::7a2b:cbff:fe1a:eab1%lagg0 prefixlen 64 scopeid 0x14 > inet 10.50.50.11 netmask 0xffffff00 broadcast 10.50.50.255 > nd6 options=21 > media: Ethernet autoselect > status: no carrier > laggproto failover lagghash l2,l3,l4 > laggport: bxe2 flags=0<> > laggport: bxe0 flags=1 > > Please note that priore to installing freebsd, the machine was running a > Debian 7 GNU/Linux 64 bit OS where we had the cards bonded and MTU'ed to > 9000 without any crash or stability issue. > So it looks to me that there is something really wrong with the broadcom > driver on freebsd 9.1, at least with the NIC's used in Dell servers. > > Provided that broadcom themselves doesn't supply drivers for freebsd Is > there any possible fix ? > > Thanks for your attention and your help. > > Cheers, > Sébastien > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > ================================================ > This e.mail is private and confidential between Multiplay (UK) Ltd. > and the person or entity to whom it is addressed. In the event of > misdirection, the recipient is prohibited from using, copying, > printing or otherwise disseminating it or any information contained in > it. > In the event of misdirection, illegible or incomplete transmission > please telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 12:55:15 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 82E4C904 for ; Tue, 23 Jul 2013 12:55:15 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-pb0-x22d.google.com (mail-pb0-x22d.google.com [IPv6:2607:f8b0:400e:c01::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5E6AB2CF7 for ; Tue, 23 Jul 2013 12:55:15 +0000 (UTC) Received: by mail-pb0-f45.google.com with SMTP id mc8so8358337pbc.4 for ; Tue, 23 Jul 2013 05:55:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=yuKNlNFm3Gn6O41XckH5koZb0eJ5+w1oFE9Lzix0n2w=; b=A0SYU/Nm1k7d/eEkReBRUxge1FWuZ+FKw3ZoN5V+WvPXj3DdUu1RN0Gz5/HT1WqtIe hHVI1wNv7SZXPrc788rYZrTK1ieNcwEdU2ATz7uNHsnKuJouGZRX1/LdpEPEYJ6MtFt5 8fB57fpPnZDCyp4zX+OCprLmgAc32RbJnHjORYb7zWlk9TqHIP3zOKUgoiz7svFG/aD2 XN9FdSJYps31QKmiKjWefbmNEE9xw23o1WqWCZAodUJQQ26ybTj1APOTY9TCPb4RGqZb 0afxO5j+dl1kVbo3CaYDmkafp05iT3EcAfCQDcWZqlFvd4ee6w7q9E91duKSQfXBAaPo f8+g== MIME-Version: 1.0 X-Received: by 10.66.159.105 with SMTP id xb9mr36867316pab.146.1374584114931; Tue, 23 Jul 2013 05:55:14 -0700 (PDT) Received: by 10.70.88.74 with HTTP; Tue, 23 Jul 2013 05:55:14 -0700 (PDT) In-Reply-To: <51EE7BDD.2050700@swisscenter.com> References: <51EE68E9.5010805@swisscenter.com> <7D8CE344ACD04EC8B8C7DC813D1EA3F6@multiplay.co.uk> <51EE7BDD.2050700@swisscenter.com> Date: Tue, 23 Jul 2013 07:55:14 -0500 Message-ID: Subject: Re: FreeBSD 9.1 and BCM57711 issues (broadcom 10ge ethernet card) From: Adam Vande More To: =?ISO-8859-1?Q?S=E9bastien_RICCIO?= Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-net , Steven Hartland X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 12:55:15 -0000 On Tue, Jul 23, 2013 at 7:49 AM, S=E9bastien RICCIO wro= te: > root@filer-01-a:/# freebsd-update -r 9.2-BETA1 upgrade > > Looking up update.FreeBSD.org mirrors... 4 mirrors found. > Fetching metadata signature for 9.1-RELEASE from update4.freebsd.org... > done. > Fetching metadata index... done. > Inspecting system... done. > > The following components of FreeBSD seem to be installed: > kernel/generic src/src world/base world/lib32 > > The following components of FreeBSD do not seem to be installed: > world/doc world/games > > Does this look reasonable (y/n)? y > > Fetching metadata signature for 9.2-BETA1 from update4.freebsd.org... don= e. > Fetching metadata index... done. > > The update metadata is correctly signed, but > failed an integrity check. > Cowardly refusing to proceed any further. > > > Cowardly ? :) http://lists.freebsd.org/pipermail/freebsd-stable/2013-July/074377.html --=20 Adam Vande More From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 13:09:42 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 64C72D31 for ; Tue, 23 Jul 2013 13:09:42 +0000 (UTC) (envelope-from sr@swisscenter.com) Received: from mail.swisslink.ch (mail.swisslink.ch [94.103.96.90]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EB1EA2DA0 for ; Tue, 23 Jul 2013 13:09:41 +0000 (UTC) Received: from [10.8.0.14] (gate.swisslink.ch [62.2.195.10]) (authenticated bits=0) by mail.swisslink.ch (8.14.3/8.14.3/Debian-9.4) with ESMTP id r6ND9Iwr024616 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 23 Jul 2013 15:09:18 +0200 Message-ID: <51EE807A.6020106@swisscenter.com> Date: Tue, 23 Jul 2013 15:09:14 +0200 From: =?ISO-8859-1?Q?S=E9bastien_RICCIO?= User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:23.0) Gecko/20100101 Thunderbird/23.0 MIME-Version: 1.0 To: Adam Vande More Subject: Re: FreeBSD 9.1 and BCM57711 issues (broadcom 10ge ethernet card) References: <51EE68E9.5010805@swisscenter.com> <7D8CE344ACD04EC8B8C7DC813D1EA3F6@multiplay.co.uk> <51EE7BDD.2050700@swisscenter.com> In-Reply-To: X-Virus-Scanned: clamav-milter 0.97.8 at mail X-Virus-Status: Clean Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-net , Steven Hartland X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 13:09:42 -0000 Thanks for the information Adam. Going to update it with svn. Cheers, Sébastien On 23.07.2013 14:55, Adam Vande More wrote: > On Tue, Jul 23, 2013 at 7:49 AM, Sébastien RICCIO > wrote: > > root@filer-01-a:/# freebsd-update -r 9.2-BETA1 upgrade > > Looking up update.FreeBSD.org > mirrors... 4 mirrors found. > Fetching metadata signature for 9.1-RELEASE from > update4.freebsd.org... done. > Fetching metadata index... done. > Inspecting system... done. > > The following components of FreeBSD seem to be installed: > kernel/generic src/src world/base world/lib32 > > The following components of FreeBSD do not seem to be installed: > world/doc world/games > > Does this look reasonable (y/n)? y > > Fetching metadata signature for 9.2-BETA1 from > update4.freebsd.org... done. > Fetching metadata index... done. > > The update metadata is correctly signed, but > failed an integrity check. > Cowardly refusing to proceed any further. > > > Cowardly ? :) > > > http://lists.freebsd.org/pipermail/freebsd-stable/2013-July/074377.html > > -- > Adam Vande More From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 14:10:01 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id C4B02F6D for ; Tue, 23 Jul 2013 14:10:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B2693206B for ; Tue, 23 Jul 2013 14:10:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6NEA1jV009091 for ; Tue, 23 Jul 2013 14:10:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6NEA1Ap009090; Tue, 23 Jul 2013 14:10:01 GMT (envelope-from gnats) Date: Tue, 23 Jul 2013 14:10:01 GMT Message-Id: <201307231410.r6NEA1Ap009090@freefall.freebsd.org> To: freebsd-net@FreeBSD.org Cc: From: Meny Yossefi Subject: RE: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragmented packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Meny Yossefi List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 14:10:01 -0000 The following reply was made to PR kern/180430; it has been noted by GNATS. From: Meny Yossefi To: John Baldwin Cc: "bug-followup@freebsd.org" Subject: RE: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragmented packets Date: Tue, 23 Jul 2013 14:01:23 +0000 --_000_F2E47A38E4D0B9499D76F2AB8901571A73A0D893MTLDAG01mtlcom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Let's stick with the FreeBSD standards (without the likely/unlikely) Let me just double check the CSUM flags work as expected. I'll get back to you as soon as I'm done. -Meny -----Original Message----- From: John Baldwin [mailto:jhb@freebsd.org] Sent: Monday, July 22, 2013 7:04 PM To: Meny Yossefi Cc: bug-followup@freebsd.org Subject: Re: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragment= ed packets On Monday, July 22, 2013 10:11:51 am Meny Yossefi wrote: > Hi John, > > > > The problem is that the HW will only calculate csum for parts of the > payload, one fragment at a time, > > while the receiver side, in our case the tcp/ip stack, will expect to val= idate the packet's payload as a whole. Ok. > I agree with the change you offered, though one thing bothers me. > > This change will add two conditions to the send flow which will probably = have an effect on performance. > > Maybe 'likely' can be useful here ? FreeBSD tends to not use likely/unlikely unless there is a demonstrable gai= n via benchmark measurements. However, if the OFED code regularly uses it = and you feel this is a case where you would normally use it, it can be adde= d. > BTW, I'm not entirely sure, but I think the CSUM_IP flag is always set, s= o maybe the first condition is not necessary. > > What do you think ? If the user uses ifconfig to disable checksum offload and force software ch= ecksums the flag will not be set. > -Meny > > > > > > -----Original Message----- > From: John Baldwin [mailto:jhb@freebsd.org] > Sent: Friday, July 19, 2013 6:29 PM > To: bug-followup@freebsd.org; Meny Yosse= fi > Subject: Re: kern/180430: [ofed] [patch] Bad UDP checksum calc for > fragmented packets > > > > Oops, my previous reply didn't make it to the PR itself. > > > > Is the problem that the hardware checksum overwrites arbitrary data in th= e packet (at the location where the UDP header would be)? > > > > Also, I've looked at other drivers, and they all look at the CSUM_* > flags to determine the right settings. IP fragments will not have > CSUM_UDP or CSUM_TCP set, so I think the more correct fix is this: > > > > Index: en_tx.c > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > --- en_tx.c (revision 253470) > > +++ en_tx.c (working copy) > > @@ -780,8 +780,12 @@ retry: > > tx_desc->ctrl.srcrb_flags =3D > cpu_to_be32(MLX4_WQE_CTRL_CQ_UPDATE | > > > MLX4_WQE_CTRL_SOLICITED); > > if (mb->m_pkthdr.csum_flags & > (CSUM_IP|CSUM_TCP|CSUM_UDP)) { > > - tx_desc->ctrl.srcrb_flags |=3D cpu_to_be32= (MLX4_WQE_CTRL_IP_CSUM | > > - = MLX4_WQE_CTRL_TCP_UDP_CSUM); > > + if (mb->m_pkthdr.csum_flags & CSUM_IP) > > + > + tx_desc->ctrl.srcrb_flags |=3D > > + > + cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM); > > + if (mb->m_pkthdr.csum_flags & > + (CSUM_TCP|CSUM_UDP)) > > + > + tx_desc->ctrl.srcrb_flags |=3D > > + > + cpu_to_be32(MLX4_WQE_CTRL_TCP_UDP_CSUM); > > priv->port_stats.tx_chksum_offload++; > > } > > > > -- > > John Baldwin > -- John Baldwin --_000_F2E47A38E4D0B9499D76F2AB8901571A73A0D893MTLDAG01mtlcom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Let's stick with th= e FreeBSD standards (without the likely/unlikely)

 

Let me just double = check the CSUM flags work as expected.

I’ll get back= to you as soon as I’m done.

 

-Meny

 

 

-----Original Message-----
From: John Baldwin [mailto:jhb@freebsd.org]
Sent: Monday, July 22, 2013 7:04 PM
To: Meny Yossefi
Cc: bug-followup@freebsd.org
Subject: Re: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragment= ed packets

 

On Monday, July 22, 2013 10:11:51 am Meny Yossefi= wrote:

> Hi John,

>

>

>

> The problem is that the HW will only calcula= te csum for parts of the

> payload, one fragment at a time,<= /p>

>

> while the receiver side, in our case the tcp= /ip stack, will expect to validate the packet's payload as a whole.

 

Ok.

 

> I agree with the change you offered, though = one thing bothers me.

>

> This change will add two conditions to the s= end flow which will probably have an effect on performance.

>

> Maybe 'likely' can be useful here ?

 

FreeBSD tends to not use likely/unlikely unless t= here is a demonstrable gain via benchmark measurements.  However, if t= he OFED code regularly uses it and you feel this is a case where you would = normally use it, it can be added.

 

> BTW, I'm not entirely sure, but I think the = CSUM_IP flag is always set, so maybe the first condition is not necessary.<= o:p>

>

> What do you think ?

 

If the user uses ifconfig to disable checksum off= load and force software checksums the flag will not be set.

 

> -Meny

>

>

>

>

>

> -----Original Message-----

> From: John Baldwin [mailto:jhb= @freebsd.org]

> Sent: Friday, July 19, 2013 6:29 PM

> To: bug-followup@free= bsd.org; Meny Yossefi

> Subject: Re: kern/180430: [ofed] [patch] Bad= UDP checksum calc for

> fragmented packets

>

>

>

> Oops, my previous reply didn't make it to th= e PR itself.

>

>

>

> Is the problem that the hardware checksum ov= erwrites arbitrary data in the packet (at the location where the UDP header= would be)?

>

>

>

> Also, I've looked at other drivers, and they= all look at the CSUM_*

> flags to determine the right settings. = IP fragments will not have

> CSUM_UDP or

CSUM_TCP set, so I think the more correct fix is = this:

>

>

>

> Index: en_tx.c

>

> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D

>

> --- en_tx.c     &nb= sp;     (revision 253470)

>

> +++ en_tx.c   &nb= sp;    (working copy)

>

> @@ -780,8 +780,12 @@ retry:

>

>       &nb= sp;        tx_desc->ctrl.srcrb_flags = =3D

> cpu_to_be32(MLX4_WQE_CTRL_CQ_UPDATE |

>

>       &nb= sp;            =             &nb= sp;            =             &nb= sp;            =             &nb= sp;            

> MLX4_WQE_CTRL_SOLICITED);

>

>       &nb= sp;        if (mb->m_pkthdr.csum_flag= s &

> (CSUM_IP|CSUM_TCP|CSUM_UDP)) {

>

> -       &= nbsp;           &nbs= p;          tx_desc->ctrl.s= rcrb_flags |=3D cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM |

>

> -       &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;  MLX4_WQE_CTRL_TCP_UDP_CSUM);

>

> +       &n= bsp;            = ;         if (mb->m_pkthdr.= csum_flags & CSUM_IP)

>

> +      &nb= sp;            =             &nb= sp;            =

> + tx_desc->ctrl.srcrb_flags |=3D=

>

> +      &nb= sp;            =             &nb= sp;            =     

> + cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM);

>

> +      &nb= sp;             = ;         if (mb->m_pkthdr.= csum_flags &

> + (CSUM_TCP|CSUM_UDP))

>

> +      &nb= sp;            =             &nb= sp;            =

> + tx_desc->ctrl.srcrb_flags |=3D=

>

> +      &nb= sp;            =             &nb= sp;            =     

> + cpu_to_be32(MLX4_WQE_CTRL_TCP_UDP_CSUM= );

>

>       &nb= sp;            =             priv->= ;port_stats.tx_chksum_offload++;

>

>       &nb= sp;        }

>

>

>

> --

>

> John Baldwin

>

 

--

John Baldwin

--_000_F2E47A38E4D0B9499D76F2AB8901571A73A0D893MTLDAG01mtlcom_-- From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 14:17:03 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 0AA8D4A1 for ; Tue, 23 Jul 2013 14:17:03 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-vb0-x235.google.com (mail-vb0-x235.google.com [IPv6:2607:f8b0:400c:c02::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BD95C20E4 for ; Tue, 23 Jul 2013 14:17:02 +0000 (UTC) Received: by mail-vb0-f53.google.com with SMTP id p12so5465151vbe.40 for ; Tue, 23 Jul 2013 07:17:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=s+Ua4XyjU0GkOG3ZtYgw3vIdY1PVrVeFEj5CdwoeJhc=; b=sVY5uuwWxNOmGFqylJEM4MhXgSt9OPg7AqO9PhUj3/gP7Ctb1L9i/XaG0x2tnZOWNX bzRJZVpub+96xHUrrlDq/EbVtDvR19hZLNgiiv5vHnnP3k/MOQ+WnYn/79OsZMK5KWa9 QFhHL/HWrdkD9Ng6CckCmJA9bu5dJm4FP0erqIxh5uW7RjL8IkhWIBMscquppKadWoP4 zAJM6HqxVdWArtp8C768wJwmTPsKBBRpcS8SpjQoy1TT0ddY4yMTz281lpAr3rOtbTxn EywV5bMbcYWHh9FRFSFMZ2HwJByTDO8bWrzo4AEl8dKfochHhTaKpvnyfqXZ+koe94eN SPmQ== MIME-Version: 1.0 X-Received: by 10.58.40.16 with SMTP id t16mr11805146vek.64.1374589021862; Tue, 23 Jul 2013 07:17:01 -0700 (PDT) Received: by 10.221.22.199 with HTTP; Tue, 23 Jul 2013 07:17:01 -0700 (PDT) In-Reply-To: References: <06BA4BD5-BE4E-4184-AFBB-D7FD4B2597D9@your.org> Date: Tue, 23 Jul 2013 10:17:01 -0400 Message-ID: Subject: Re: Duplicate Address Detection misfire? From: Zaphod Beeblebrox To: Kimmo Paasiala Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net , Kevin Day X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 14:17:03 -0000 Does that mean, then, that the only fix open to some people is to turn off DAD? I have another idea: Require DAD to inspect the sending MAC address. If the sending MAC address is _our_ MAC address, then the packet is not an indication of a duplicate address? On Tue, Jul 23, 2013 at 2:15 AM, Kimmo Paasiala wrote: > On Tue, Jul 23, 2013 at 8:44 AM, Zaphod Beeblebrox > wrote: > > What to do when you don't trust the interface? VMWare is obviously > > emulating the hardware and their interpretation of what the hardware "is" > > is possibly different from ours. > > > > If I boot single-user and tcpdump the interface, I see two transmitted > > solicitations. The kernel claims to have sent one. > > > > My concern: is the vmware interface reflecting the solicitation packet > > because it is a broadcast packet? > > > > To determine this, I've gone over the tcpdump and pcap-filter man pages > to > > look for a way to only dump packets leaving from or arriving at an > > interface. Can this be done? > > > > If VMWare is reflecting the packet back, I'm curious as to how we can fix > > this. > > > > > > That sounds exactly like my experience with DAD misbehaving on my > Zyxel WAP3205 access point. It is reflecting the multicasts (I hope > that's the right term) so that any IPv6 equipment on the wireless > network will think that its address is already in use. For the record, > the client machines in my case are all OS X. Nice to know I'm not the > only one with such problems. > > -Kimmo > From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 14:22:19 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A41196BC for ; Tue, 23 Jul 2013 14:22:19 +0000 (UTC) (envelope-from kevin@your.org) Received: from mail.your.org (mail.your.org [IPv6:2001:4978:1:2::cc09:3717]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5E91C212E for ; Tue, 23 Jul 2013 14:22:19 +0000 (UTC) Received: from mail.your.org (chi02.mail.your.org [204.9.55.23]) by mail.your.org (Postfix) with ESMTP id 875FE84397; Tue, 23 Jul 2013 14:22:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=your.org; h=content-type :mime-version:subject:from:in-reply-to:date:cc:message-id :references:to; s=selector1; bh=U33RfFyVvjzEFKvdUtMtHTSsKtE=; b= mhLgG/XspznFY3vXWWzU8/IrjOT6X5wJpFJRrtG99g3wFljNDoL3ehjvFtUJ4NAa s/khsvhpQZhzIoNXE5PDhtGtrj70JwhIf+n2F0Xi48ojfPRx82bMSDJTuKSnKg4b 3MOC2+iQPuvAgVge/NDfIylC96weZEjMlEI/yfVzaGQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=your.org; h=content-type :mime-version:subject:from:in-reply-to:date:cc:message-id :references:to; q=dns; s=selector1; b=ZmRB65aaktzqDhyKdyje+bb5Uc Xp/tSMcMV/qe3KS1lZ1i9Q9bf44XLxGOM15X3B/O1Ts8UazrAv7UdahsjANgtDYA o2IS1CHlcbS+UNoRA5M2ikgHqO4IVtxvGrPg7unsMvE/xx2EV/OVUIn2dYKDl+8v /3E2DgD8C0Mx6fnoM= Received: from vpn132.rw1.your.org (vpn132.rw1.your.org [204.9.51.132]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.your.org (Postfix) with ESMTPSA id 3054284396; Tue, 23 Jul 2013 14:22:17 +0000 (UTC) Content-Type: multipart/signed; boundary="Apple-Mail=_8415871D-F9FE-4784-94D1-C94BC2AC3DBD"; protocol="application/pkcs7-signature"; micalg=sha1 Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: Duplicate Address Detection misfire? From: Kevin Day In-Reply-To: Date: Tue, 23 Jul 2013 09:22:16 -0500 Message-Id: <3341F665-B179-4A99-B48A-81EE3D0478A6@your.org> References: <06BA4BD5-BE4E-4184-AFBB-D7FD4B2597D9@your.org> To: Zaphod Beeblebrox X-Mailer: Apple Mail (2.1508) Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 14:22:19 -0000 --Apple-Mail=_8415871D-F9FE-4784-94D1-C94BC2AC3DBD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 Sorry for the slow reply. We run mainly ESXi (the bare metal version of VMware, not the desktop = versions) but I do remember seeing something sort of similar on a = desktop version ages ago. Basically we narrowed it down to a Windows = driver bug on the host system's ethernet card. It was basically = reflecting back everything it transmitted into the receive queue. This = was before IPv6 was in use here, but I remember it breaking a file = sharing protocol (SMB? AFP?) that also didn't like seeing its own = multicasts/broadcasts echoed back to itself. By any chance is this on a system using WiFi rather than wired ethernet? = Many routers/access points/wifi adapters have problems with the idea of = VMware's "bridged mode" networking - they expect only one MAC per = station and do not do the right thing in some places. I don't have an answer for you, but I'd look at the physical networking = card/adapter on the host OS first if I were troubleshooting this. = Updated driver/replace with something else/etc. -- Kevin On Jul 23, 2013, at 12:44 AM, Zaphod Beeblebrox = wrote: > What to do when you don't trust the interface? VMWare is obviously = emulating the hardware and their interpretation of what the hardware = "is" is possibly different from ours. >=20 > If I boot single-user and tcpdump the interface, I see two transmitted = solicitations. The kernel claims to have sent one. >=20 > My concern: is the vmware interface reflecting the solicitation packet = because it is a broadcast packet? >=20 > To determine this, I've gone over the tcpdump and pcap-filter man = pages to look for a way to only dump packets leaving from or arriving at = an interface. Can this be done? >=20 > If VMWare is reflecting the packet back, I'm curious as to how we can = fix this. >=20 --Apple-Mail=_8415871D-F9FE-4784-94D1-C94BC2AC3DBD Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPITCCBN0w ggPFoAMCAQICEHGS++YZX6xNEoV0cTSiGKcwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0Ix GzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwR Q29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0w NDAxMDEwMDAwMDBaFw0yODEyMzEyMzU5NTlaMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQx FzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsx ITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJz dC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAsjmFpPJ9q0E7YkY3rs3BYHW8OWX5ShpHornMSMxqmNVNNRm5pELlzkniii8efNIx B8dOtINknS4p1aJkxIW9hVE1eaROaJB7HHqkkqgX8pgV8pPMyaQylbsMTzC9mKALi+VuG6JG+ni8 om+rWV6lL8/K2m2qL+usobNqqrcuZzWLeeEeaYji5kbNoKXqvgvOdjp6Dpvq/NonWz1zHyLmSGHG TPNpsaguG7bUMSAsvIKKjqQOpdeJQ/wWWq8dcdcRWdq6hw2v+vPhwvCkxWeM1tZUOt4KpLoDd7Nl yP0e03RiqhjKaJMeoYV+9Udly/hNVyh00jT/MLbu9mIwFIws6wIDAQABo4IBJzCCASMwHwYDVR0j BBgwFoAUoBEKIz6W8Qfs4q8p74Klf9AwpLQwHQYDVR0OBBYEFImCZ33EnSZwAEu0UEh83j2uBG59 MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdJQQWMBQGCCsGAQUFBwMCBggr BgEFBQcDBDARBgNVHSAECjAIMAYGBFUdIAAwewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5j b21vZG9jYS5jb20vQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwu Y29tb2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDARBglghkgBhvhCAQEEBAMCAQYw DQYJKoZIhvcNAQEFBQADggEBAJ2Vyzy4fqUJxB6/C8LHdo45PJTGEKpPDMngq4RdiVTgZTvzbRx8 NywlVF+WIfw3hJGdFdwUT4HPVB1rbEVgxy35l1FM+WbKPKCCjKbI8OLp1Er57D9Wyd12jMOCAU9s APMeGmF0BEcDqcZAV5G8ZSLFJ2dPV9tkWtmNH7qGL/QGrpxp7en0zykX2OBKnxogL5dMUbtGB8SK N04g4wkxaMeexIud6H4RvDJoEJYRmETYKlFgTYjrdDrfQwYyyDlWjDoRUtNBpEMD9O3vMyfbOeAU TibJ2PU54om4k123KSZB6rObroP8d3XK6Mq1/uJlSmM+RMTQw16Hc6mYHK9/FX8wggUaMIIEAqAD AgECAhBtGeqnGU9qMyLmIjJ6qnHeMA0GCSqGSIb3DQEBBQUAMIGuMQswCQYDVQQGEwJVUzELMAkG A1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNU IE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVRO LVVTRVJGaXJzdC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsMB4XDTExMDQyODAwMDAw MFoXDTIwMDUzMDEwNDgzOFowgZMxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNo ZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMTkwNwYD VQQDEzBDT01PRE8gQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwggEi MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCShIRbS1eY1F4vi6ThQMijU1hfZmXxMk73nzJ9 VdB4TFW3QpTg+SdxB8XGaaS5MsTxQBqQzCdWYn8XtXFpruUgG+TLY15gyqJB9mrho/+43x9IbWVD jCouK2M4d9+xF6zC2oIC1tQyatRnbyATj1w1+uVUgK/YcQodNwoCUFNslR2pEBS0mZVZEjH/CaLS TNxS297iQAFbSGjdxUq04O0kHzqvcV8H46y/FDuwJXFoPfQP1hdYRhWBPGiLi4MPbXohV+Y0sNsy fuNK4aVScmQmkU6lkg//4LFg/RpvaFGZY40ai6XMQpubfSJj06mg/M6ekN9EGfRcWzW6FvOnm//B AgMBAAGjggFLMIIBRzAfBgNVHSMEGDAWgBSJgmd9xJ0mcABLtFBIfN49rgRufTAdBgNVHQ4EFgQU ehNOAHRbxnhjZCfBL+KgW7x5xXswDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQAw EQYDVR0gBAowCDAGBgRVHSAAMFgGA1UdHwRRME8wTaBLoEmGR2h0dHA6Ly9jcmwudXNlcnRydXN0 LmNvbS9VVE4tVVNFUkZpcnN0LUNsaWVudEF1dGhlbnRpY2F0aW9uYW5kRW1haWwuY3JsMHQGCCsG AQUFBwEBBGgwZjA9BggrBgEFBQcwAoYxaHR0cDovL2NydC51c2VydHJ1c3QuY29tL1VUTkFkZFRy dXN0Q2xpZW50X0NBLmNydDAlBggrBgEFBQcwAYYZaHR0cDovL29jc3AudXNlcnRydXN0LmNvbTAN BgkqhkiG9w0BAQUFAAOCAQEAhda+eFdVbTN/RFL+QtUGqAEDgIr7DbL9Sr/2r0FJ9RtaxdKtG3Nu PukmfOZMmMEwKN/L+0I8oSU+CnXW0D05hmbRoZu1TZtvryhsHa/l6nRaqNqxwPF1ei+eupN5yv7i kR5WdLL4jdPgQ3Ib7Y/9YDkgR/uLrzplSDyYPaUlv73vYOBJ5RbI6z9Dg/Dg7g3B080zX5vQvWBq szv++tTJOjwf7Zv/m0kzvkIpOYPuM2kugp1FTahp2oAbHj3SGl18R5mlmwhtEpmG1l1XBxunML5L SUS4kH7K0Xk467Qz+qA6XSZYnmFVGLQh1ZnV4ENAQjC+6qXnlNKw/vN1+X9u5zCCBR4wggQGoAMC AQICEQDX7payXlJleRO8EofThBYfMA0GCSqGSIb3DQEBBQUAMIGTMQswCQYDVQQGEwJHQjEbMBkG A1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01P RE8gQ0EgTGltaXRlZDE5MDcGA1UEAxMwQ09NT0RPIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQg U2VjdXJlIEVtYWlsIENBMB4XDTEzMDYxNjAwMDAwMFoXDTE0MDYxNjIzNTk1OVowHzEdMBsGCSqG SIb3DQEJARYOa2V2aW5AeW91ci5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+ +8pMP5JQB9JNtJMBToSs0n8raC/nLnXvgLO1W2ab2h5z/9tmUhIaYF5GXsPTcstp3rWGY85yMgC6 fFVzgo8eiCz0jbiuwH5VKry7WSdUh3ZLGkVfMYaSOh1BjAN7AUEC/k4F5XAmPr8OfEUi1sEY/7Ot wWMankm0N2QdKyffrcFkiRlxHb4uoNcY+a0R8RSv0ILDH5xlRDx67XpuJ1hg6YCz4Xv/NdEe4H/Y Ek/ulaFICLTOxf4KvOI6Tq4ZaV+j/rzkBkmjRAtI522Uml4uwYaXvKzxj+l6Ezm5wLvf1zaQL/8l e7JgwK2eV3wuDqO7gNDpx9lpSDACioolMy55AgMBAAGjggHeMIIB2jAfBgNVHSMEGDAWgBR6E04A dFvGeGNkJ8Ev4qBbvHnFezAdBgNVHQ4EFgQUv94xWfNBA4IaXtCd/3dmO3GpZoowDgYDVR0PAQH/ BAQDAgWgMAwGA1UdEwEB/wQCMAAwIAYDVR0lBBkwFwYIKwYBBQUHAwQGCysGAQQBsjEBAwUCMBEG CWCGSAGG+EIBAQQEAwIFIDBGBgNVHSAEPzA9MDsGDCsGAQQBsjEBAgEBATArMCkGCCsGAQUFBwIB Fh1odHRwczovL3NlY3VyZS5jb21vZG8ubmV0L0NQUzBXBgNVHR8EUDBOMEygSqBIhkZodHRwOi8v Y3JsLmNvbW9kb2NhLmNvbS9DT01PRE9DbGllbnRBdXRoZW50aWNhdGlvbmFuZFNlY3VyZUVtYWls Q0EuY3JsMIGIBggrBgEFBQcBAQR8MHowUgYIKwYBBQUHMAKGRmh0dHA6Ly9jcnQuY29tb2RvY2Eu Y29tL0NPTU9ET0NsaWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcnQwJAYIKwYB BQUHMAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTAZBgNVHREEEjAQgQ5rZXZpbkB5b3VyLm9y ZzANBgkqhkiG9w0BAQUFAAOCAQEAFs2L7FssxGKC4+3j40bFG0FIp1LIZwczT4b16Y3zprZZ09/T ojYAsVQfNCbvSNcGAoiYRpr5/lrbRq+OYHwb4CYlt6k0XFhlRccDqYadmzVSEgiGUpGn9GDtjwRL QvmAVYFj/up2uythSNmIM3J/2kS5U9MX+lbzHhX7GVhtWB8Y8q6F4L3rr4CW3RE4vE+hUY1LXW0t sQkWpM8XCyrAiTeb3O5IqdqrfHG77iEfbPWM8VF3XMbjEp/iitripx7xci44BXaUEYk0M6bN80rf s+INL5Fo5DzEAbOuST4rgLM1ZkzljpCESGOm5A3JxMTcXEdjWCuAUlTZEp3QmcIqxjGCA64wggOq AgEBMIGpMIGTMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYD VQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDE5MDcGA1UEAxMwQ09NT0RP IENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEA1+6Wsl5SZXkTvBKH 04QWHzAJBgUrDgMCGgUAoIIB2TAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJ BTEPFw0xMzA3MjMxNDIyMTdaMCMGCSqGSIb3DQEJBDEWBBTC6MPa/E59qUmjImYOEz3GzUNdJzCB ugYJKwYBBAGCNxAEMYGsMIGpMIGTMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5j aGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDE5MDcG A1UEAxMwQ09NT0RPIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEA 1+6Wsl5SZXkTvBKH04QWHzCBvAYLKoZIhvcNAQkQAgsxgayggakwgZMxCzAJBgNVBAYTAkdCMRsw GQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNP TU9ETyBDQSBMaW1pdGVkMTkwNwYDVQQDEzBDT01PRE8gQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFu ZCBTZWN1cmUgRW1haWwgQ0ECEQDX7payXlJleRO8EofThBYfMA0GCSqGSIb3DQEBAQUABIIBACH3 9ZK/71qEwcHZK4laEqMTzuEBe9Q2kf/kydqp7CvsKhRLFjY8sw+xPZCWbu4Vj6Y5ZoJF3zB+TRUE FKYrXbLrfsMOmME9rQsTTME5rlrikOnynhjHjOwFSvgX/2iI3cTSo8ydd5EWjeLCqCttucYAlQ25 E2S7GEOLrnXXhOIjgOxaovUDBaxUCYE/Oz+fknJt9OYRzNoUu44M+2ApgjfVqOT3FhPGQU1UPApP f4V+tH9A5sTgBSi8t94helga5OIO9L5xzwV98W4+MFTfH42V3bTBnxU1EqstRk5Ucqef6h3ZAWRp Mf3ZwkvUluxZSwuK12m4STf4b66s9W0g6mkAAAAAAAA= --Apple-Mail=_8415871D-F9FE-4784-94D1-C94BC2AC3DBD-- From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 14:39:42 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2E20FB74 for ; Tue, 23 Jul 2013 14:39:42 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-x22d.google.com (mail-wi0-x22d.google.com [IPv6:2a00:1450:400c:c05::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B7BC121F2 for ; Tue, 23 Jul 2013 14:39:41 +0000 (UTC) Received: by mail-wi0-f173.google.com with SMTP id hq4so3097399wib.12 for ; Tue, 23 Jul 2013 07:39:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=iCuAuG/rjE4dfxajZkiEIQDOUBj8FLES5RcIEX5jNKI=; b=gxHRpbBGxqyOtB7AlbAfDj3bN7zblwgGpFYrOp07v9wK8xAMprIlafmpxg4G8Bf95d X30NGUTApUezRs2BTiVCavQbI1wFXbEy5yBCI/SWXC2HXDXaWCjSEQiCASI0Eri6/AHH arYkdO3erGRyrny9A2Z6VhOoS6wZq8IvqP4yHY7lwY5uumAj6VmfFatR2YitAARwPMY6 S24j8XHSy+9gmGbkwX8m7Hw+Cvm3w9giaTJ9KqChexwSyT8VL+DW2xjBjnbdrOhHNwd2 wfLx6HmJNF1q39AvUdnhjID3yt0KGMJ4FkhkoPdQlQkg0FBhrbQD4R1yHB0Vpem17N8g Q/SQ== MIME-Version: 1.0 X-Received: by 10.180.160.165 with SMTP id xl5mr33225492wib.46.1374590379987; Tue, 23 Jul 2013 07:39:39 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Tue, 23 Jul 2013 07:39:39 -0700 (PDT) In-Reply-To: <51EE2C2B.4020800@mail.ru> References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> <20130722200205.GO26412@funkthat.com> <51EDA37A.9040200@mail.ru> <51EE198B.7040509@mail.ru> <51EE2C2B.4020800@mail.ru> Date: Tue, 23 Jul 2013 07:39:39 -0700 X-Google-Sender-Auth: NwNRzoSr_De_ua_qQkP5cgHqYEw Message-ID: Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour From: Adrian Chadd To: trafdev Content-Type: text/plain; charset=ISO-8859-1 Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, John-Mark Gurney X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 14:39:42 -0000 On 23 July 2013 00:09, trafdev wrote: > It's like shared acceptor FD and N processes: [snip] looks like mine, but I use threads. > Accept conn callback is called in N processes on each connection, only one > wins, > others exit by errno == EAGAIN case. Overhead is almost zero. > Problem is that "wins" distribution is far from equal. Right. I'm not at that stage yet, but I can totally see that happening. Ok. Time to hit up the TCP stack people to weigh in on the recent lkml posts about this: http://lwn.net/Articles/542629/ With SO_REUSEPORT, we should create one listen FD per thread, and let the OS balance how that gets distributed. Rather than one listen socket that is shared between all processes/threads. I'll try that locally and see if that works right. Would you mind trying it locally and see if it improves the distribution of work? Thanks, -adrian From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 17:18:09 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id AE29C783 for ; Tue, 23 Jul 2013 17:18:09 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-we0-x22a.google.com (mail-we0-x22a.google.com [IPv6:2a00:1450:400c:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 426ED2A82 for ; Tue, 23 Jul 2013 17:18:09 +0000 (UTC) Received: by mail-we0-f170.google.com with SMTP id w60so1229836wes.15 for ; Tue, 23 Jul 2013 10:18:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=VhodCoUe6rFJASOWuGre4MLzwlm5UacbDteFmW79Z2M=; b=bxE6PIkGXpzRw2poak/tHJNzdRnrddsIhw2mfbGsMWYmystm3abbJaVz+QIjAIj/1D W8KG8rrqwM72zHqHG4N7w3PtifIG8IrTrxZ1UmmanRVm9NStAsPC6fjDRYZ/IYXafzjV zGqhhKZBddLMpoYWawKvAxFL1cAUpw8e5sT7gPTmy2kACq/wVlKONg2ZqnltMjLx+Jbd B+aZ6vYHBLZlsI5lFBefRPfhQUChAwzM6rbWA8sX0bEBtMSQOkTKCYuLJgMwx+fMaj66 yMRCyV5tzTaURpa71QLSSa+usAa6c+Arym2BKSSmC4su4r4+//8s59mlc6dEPKZn2oaX jjZA== MIME-Version: 1.0 X-Received: by 10.194.63.229 with SMTP id j5mr23328764wjs.79.1374599887571; Tue, 23 Jul 2013 10:18:07 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Tue, 23 Jul 2013 10:18:07 -0700 (PDT) In-Reply-To: References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> <20130722200205.GO26412@funkthat.com> <51EDA37A.9040200@mail.ru> <51EE198B.7040509@mail.ru> <51EE2C2B.4020800@mail.ru> Date: Tue, 23 Jul 2013 10:18:07 -0700 X-Google-Sender-Auth: PymJ1vcNJrOpSpOPLkLSWp03yMI Message-ID: Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour From: Adrian Chadd To: trafdev Content-Type: text/plain; charset=ISO-8859-1 Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, John-Mark Gurney X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 17:18:09 -0000 Answering my own email: SO_REUSEPORT on FreeBSD doesn't load balance incoming connections. Test case: * 8 threads * each creates a TCP socket, listening on port 1667, with SO_REUSEPORT * only the first thread ever sees incoming requests. I think this load distribution feature is useful to implement, but it shouldn't be called SO_REUSEPORT. (Silly Linux, why would you do that too..) -adrian On 23 July 2013 07:39, Adrian Chadd wrote: > On 23 July 2013 00:09, trafdev wrote: >> It's like shared acceptor FD and N processes: > > [snip] looks like mine, but I use threads. > >> Accept conn callback is called in N processes on each connection, only one >> wins, >> others exit by errno == EAGAIN case. Overhead is almost zero. >> Problem is that "wins" distribution is far from equal. > > Right. I'm not at that stage yet, but I can totally see that happening. > > Ok. Time to hit up the TCP stack people to weigh in on the recent lkml > posts about this: > > http://lwn.net/Articles/542629/ > > With SO_REUSEPORT, we should create one listen FD per thread, and let > the OS balance how that gets distributed. Rather than one listen > socket that is shared between all processes/threads. I'll try that > locally and see if that works right. Would you mind trying it locally > and see if it improves the distribution of work? > > Thanks, > > > -adrian From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 17:25:00 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2E1D39B7 for ; Tue, 23 Jul 2013 17:25:00 +0000 (UTC) (envelope-from sr@swisscenter.com) Received: from mail.swisslink.ch (mail.swisslink.ch [94.103.96.90]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A78272ADE for ; Tue, 23 Jul 2013 17:24:59 +0000 (UTC) Received: from [10.8.0.14] (gate.swisslink.ch [62.2.195.10]) (authenticated bits=0) by mail.swisslink.ch (8.14.3/8.14.3/Debian-9.4) with ESMTP id r6NHOoTd010861 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 23 Jul 2013 19:24:50 +0200 Message-ID: <51EEBC5E.3060606@swisscenter.com> Date: Tue, 23 Jul 2013 19:24:46 +0200 From: =?ISO-8859-1?Q?S=E9bastien_RICCIO?= User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:23.0) Gecko/20100101 Thunderbird/23.0 MIME-Version: 1.0 To: Adam Vande More Subject: Re: FreeBSD 9.1 and BCM57711 issues (broadcom 10ge ethernet card) References: <51EE68E9.5010805@swisscenter.com> <7D8CE344ACD04EC8B8C7DC813D1EA3F6@multiplay.co.uk> <51EE7BDD.2050700@swisscenter.com> <51EE807A.6020106@swisscenter.com> In-Reply-To: <51EE807A.6020106@swisscenter.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: clamav-milter 0.97.8 at mail X-Virus-Status: Clean Cc: freebsd-net , Steven Hartland X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 17:25:00 -0000 Hi, So I've upgraded the system with th stable/9 subversion tree, but the same problems occurs with these network cards. IMHO this driver is broken ... Do you know who I should contact to report the issue ? Thanks again. Sébastien On 23.07.2013 15:09, Sébastien RICCIO wrote: > Thanks for the information Adam. Going to update it with svn. > > Cheers, > Sébastien > > On 23.07.2013 14:55, Adam Vande More wrote: >> On Tue, Jul 23, 2013 at 7:49 AM, Sébastien RICCIO > > wrote: >> >> root@filer-01-a:/# freebsd-update -r 9.2-BETA1 upgrade >> >> Looking up update.FreeBSD.org >> mirrors... 4 mirrors found. >> Fetching metadata signature for 9.1-RELEASE from >> update4.freebsd.org... done. >> Fetching metadata index... done. >> Inspecting system... done. >> >> The following components of FreeBSD seem to be installed: >> kernel/generic src/src world/base world/lib32 >> >> The following components of FreeBSD do not seem to be installed: >> world/doc world/games >> >> Does this look reasonable (y/n)? y >> >> Fetching metadata signature for 9.2-BETA1 from >> update4.freebsd.org... done. >> Fetching metadata index... done. >> >> The update metadata is correctly signed, but >> failed an integrity check. >> Cowardly refusing to proceed any further. >> >> >> Cowardly ? :) >> >> >> http://lists.freebsd.org/pipermail/freebsd-stable/2013-July/074377.html >> >> -- >> Adam Vande More > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 17:54:50 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1907D3AA; Tue, 23 Jul 2013 17:54:50 +0000 (UTC) (envelope-from trafdev@mail.ru) Received: from smtp49.i.mail.ru (smtp49.i.mail.ru [94.100.177.109]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 872772C08; Tue, 23 Jul 2013 17:54:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:Subject:CC:To:MIME-Version:From:Date:Message-ID; bh=TgxRVOR/HDk3pslwDTt+0/Fgj6xZ4bxuMvhzNhezyfU=; b=TUtE/y/mGkko3A/mLq0bFZ4ZRq3ZxrxOQfeeAs535ba6g1oWOUp0/zX1lhTtwyxuMC/ba6dCv5RsnihmhBVGzCPwuQcynNLh4cQO8DLL6yX+Y8wGtKyAe+TGIjmnYwlf/0bXJQq0K+mErqtSUB0QQCnM0cL22CkIl/+xVedCp2I=; Received: from [50.156.108.197] (port=50336 helo=[192.168.1.116]) by smtp49.i.mail.ru with esmtpa (envelope-from ) id 1V1gnN-0006Dp-TX; Tue, 23 Jul 2013 21:54:46 +0400 Message-ID: <51EEC361.9050806@mail.ru> Date: Tue, 23 Jul 2013 10:54:41 -0700 From: trafdev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130628 Thunderbird/17.0.7 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> <20130722200205.GO26412@funkthat.com> <51EDA37A.9040200@mail.ru> <51EE198B.7040509@mail.ru> <51EE2C2B.4020800@mail.ru> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, John-Mark Gurney X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 17:54:50 -0000 Yes, and if you kill this first thread - second thread will start to receive connections and so on. That's why I've used one processes-shared acceptor socket which behaves better (load is balancing between processes but equality of distribution is far from ideal). Btw as per https://lwn.net/Articles/542629/ Linux 3.1 SOLVES all these problems via SO_REUSEPORT. On Tue Jul 23 10:18:07 2013, Adrian Chadd wrote: > Answering my own email: > > SO_REUSEPORT on FreeBSD doesn't load balance incoming connections. > > Test case: > > * 8 threads > * each creates a TCP socket, listening on port 1667, with SO_REUSEPORT > * only the first thread ever sees incoming requests. > > I think this load distribution feature is useful to implement, but it > shouldn't be called SO_REUSEPORT. > > (Silly Linux, why would you do that too..) > > > > -adrian > > On 23 July 2013 07:39, Adrian Chadd wrote: >> On 23 July 2013 00:09, trafdev wrote: >>> It's like shared acceptor FD and N processes: >> >> [snip] looks like mine, but I use threads. >> >>> Accept conn callback is called in N processes on each connection, only one >>> wins, >>> others exit by errno == EAGAIN case. Overhead is almost zero. >>> Problem is that "wins" distribution is far from equal. >> >> Right. I'm not at that stage yet, but I can totally see that happening. >> >> Ok. Time to hit up the TCP stack people to weigh in on the recent lkml >> posts about this: >> >> http://lwn.net/Articles/542629/ >> >> With SO_REUSEPORT, we should create one listen FD per thread, and let >> the OS balance how that gets distributed. Rather than one listen >> socket that is shared between all processes/threads. I'll try that >> locally and see if that works right. Would you mind trying it locally >> and see if it improves the distribution of work? >> >> Thanks, >> >> >> -adrian > > From owner-freebsd-net@FreeBSD.ORG Tue Jul 23 19:38:55 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 58D4C698 for ; Tue, 23 Jul 2013 19:38:55 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-x22f.google.com (mail-wi0-x22f.google.com [IPv6:2a00:1450:400c:c05::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E10C02142 for ; Tue, 23 Jul 2013 19:38:54 +0000 (UTC) Received: by mail-wi0-f175.google.com with SMTP id m6so3444486wiv.8 for ; Tue, 23 Jul 2013 12:38:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=ezqOc6NI+WAZxsayIH4buImAKzlGLQda/lbydjD6X6I=; b=iLHMH5ZTZV0IK0H1BTAvnf+kSaaDLye8sVdTfBsMMMcRTsR0vQ1eRKsCDhNK7SpxIE nxqyslK24XCIwR5GrkCVrU131iyesAJf/HK9mxt1b6/3B2zo+pAGJ/LACC9n8amwi0tE 0FXSXj40dyTaBg+glbSiLWqijhXmoxBRMf1ztymQ/OWzz7iQhWLxvDwG/mNk5KfFjQ2U YjsVD1HyglIioxDr5wQWLmhRjUvzpdDPxdEgHtGnttyaHG0iV6vUyADjokmEYYC0cNxf MkajV4HN66tO+O3eiRFwDnhoPTZuBkoTOjyfAOE3v3fhRkowi9Ys7iI2k2ucVlhkQxM/ C59A== MIME-Version: 1.0 X-Received: by 10.194.92.6 with SMTP id ci6mr139175wjb.79.1374608333136; Tue, 23 Jul 2013 12:38:53 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Tue, 23 Jul 2013 12:38:53 -0700 (PDT) In-Reply-To: <51EEC361.9050806@mail.ru> References: <51E0E2AF.7090404@mail.ru> <51E44E2F.8060700@mail.ru> <51E455D5.2090403@mail.ru> <20130722200205.GO26412@funkthat.com> <51EDA37A.9040200@mail.ru> <51EE198B.7040509@mail.ru> <51EE2C2B.4020800@mail.ru> <51EEC361.9050806@mail.ru> Date: Tue, 23 Jul 2013 12:38:53 -0700 X-Google-Sender-Auth: m4LUfvW5Q1GRRww1bk979bHcDbs Message-ID: Subject: Re: SO_REUSEPORT: strange kernel balancer behaviour From: Adrian Chadd To: trafdev Content-Type: text/plain; charset=ISO-8859-1 Cc: Sepherosa Ziehau , freebsd-net@freebsd.org, John-Mark Gurney X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 19:38:55 -0000 On 23 July 2013 10:54, trafdev wrote: > Yes, and if you kill this first thread - second thread will start to receive > connections and so on. > That's why I've used one processes-shared acceptor socket which behaves > better (load is balancing > between processes but equality of distribution is far from ideal). > > Btw as per https://lwn.net/Articles/542629/ Linux 3.1 SOLVES all these > problems via SO_REUSEPORT. Right, but the patch totally rewrites what SO_REUSEPORT means. I'd like to see similar behaviour in FreeBSD, but under a different socket option name. -adrain From owner-freebsd-net@FreeBSD.ORG Wed Jul 24 00:53:29 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9BCDE590 for ; Wed, 24 Jul 2013 00:53:29 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-ve0-x22d.google.com (mail-ve0-x22d.google.com [IPv6:2607:f8b0:400c:c01::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5D8392FA9 for ; Wed, 24 Jul 2013 00:53:29 +0000 (UTC) Received: by mail-ve0-f173.google.com with SMTP id jw11so6513658veb.4 for ; Tue, 23 Jul 2013 17:53:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=txK2H165mbwo0R3vOb9ZESKZu4UeKN/SoSXbBBWMris=; b=umYjYN+Rvp88y7VM1+2yeNLwehle/viYsP8jr9bSAo44ARESVGQbxJfw6KJMkkghCV Fv43HOFCuoU0KIzGnjz62AzEGCflCpJ+sRUMXthg+6/gbOWvJPAGmmiW7gASL40Mn7HD vGkiLbqnp/odGiTRz4C+FoE4N0+hci9CSsuOo/WxmSC0W3x4fPg12M8v0LPn2gltk5x2 /OLb+MxhLfoYlaiQa5xm9RNWu67IU3tOLZZKbCxPLd/h7q0X1nH/53AA80p8wBAVlxR8 TD6CD+ntB4g6XY7Vxoz01VH3FwRqUgIp79kBQQ0A0dnkQq0a7s4/PoQyjAZttQn4CF5V V1YQ== MIME-Version: 1.0 X-Received: by 10.220.249.67 with SMTP id mj3mr4143559vcb.23.1374627208455; Tue, 23 Jul 2013 17:53:28 -0700 (PDT) Received: by 10.221.22.199 with HTTP; Tue, 23 Jul 2013 17:53:28 -0700 (PDT) In-Reply-To: <3341F665-B179-4A99-B48A-81EE3D0478A6@your.org> References: <06BA4BD5-BE4E-4184-AFBB-D7FD4B2597D9@your.org> <3341F665-B179-4A99-B48A-81EE3D0478A6@your.org> Date: Tue, 23 Jul 2013 20:53:28 -0400 Message-ID: Subject: Re: Duplicate Address Detection misfire? From: Zaphod Beeblebrox To: Kevin Day Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jul 2013 00:53:29 -0000 Ironically, the win 7 host is using an "em" series card. Specifically, an Intel "Gigabit CT". It is configured in vlan mode and the VMWare bridged connection is connected to a VLAN. Regardless, tho, Intel's own drivers are usually not this bad --- and the driver is up-to-date. On Tue, Jul 23, 2013 at 10:22 AM, Kevin Day wrote: > Sorry for the slow reply. > > We run mainly ESXi (the bare metal version of VMware, not the desktop > versions) but I do remember seeing something sort of similar on a desktop > version ages ago. Basically we narrowed it down to a Windows driver bug on > the host system's ethernet card. It was basically reflecting back > everything it transmitted into the receive queue. This was before IPv6 was > in use here, but I remember it breaking a file sharing protocol (SMB? AFP?) > that also didn't like seeing its own multicasts/broadcasts echoed back to > itself. > > By any chance is this on a system using WiFi rather than wired ethernet? > Many routers/access points/wifi adapters have problems with the idea of > VMware's "bridged mode" networking - they expect only one MAC per station > and do not do the right thing in some places. > > I don't have an answer for you, but I'd look at the physical networking > card/adapter on the host OS first if I were troubleshooting this. Updated > driver/replace with something else/etc. > > -- Kevin > > On Jul 23, 2013, at 12:44 AM, Zaphod Beeblebrox wrote: > > > What to do when you don't trust the interface? VMWare is obviously > emulating the hardware and their interpretation of what the hardware "is" > is possibly different from ours. > > > > If I boot single-user and tcpdump the interface, I see two transmitted > solicitations. The kernel claims to have sent one. > > > > My concern: is the vmware interface reflecting the solicitation packet > because it is a broadcast packet? > > > > To determine this, I've gone over the tcpdump and pcap-filter man pages > to look for a way to only dump packets leaving from or arriving at an > interface. Can this be done? > > > > If VMWare is reflecting the packet back, I'm curious as to how we can > fix this. > > > > From owner-freebsd-net@FreeBSD.ORG Wed Jul 24 00:58:06 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3DBC57C4; Wed, 24 Jul 2013 00:58:06 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 163412FEF; Wed, 24 Jul 2013 00:58:06 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6O0w518046061; Wed, 24 Jul 2013 00:58:05 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6O0w5IY046060; Wed, 24 Jul 2013 00:58:05 GMT (envelope-from linimon) Date: Wed, 24 Jul 2013 00:58:05 GMT Message-Id: <201307240058.r6O0w5IY046060@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/180775: [bxe] if_bxe driver broken with Broadcom BCM57711 cards X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jul 2013 00:58:06 -0000 Old Synopsis: if_bxe driver broken with Broadcom BCM57711 cards New Synopsis: [bxe] if_bxe driver broken with Broadcom BCM57711 cards Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Wed Jul 24 00:57:52 UTC 2013 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=180775 From owner-freebsd-net@FreeBSD.ORG Wed Jul 24 01:00:54 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 4743E919 for ; Wed, 24 Jul 2013 01:00:54 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-ve0-x22d.google.com (mail-ve0-x22d.google.com [IPv6:2607:f8b0:400c:c01::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 089B9201F for ; Wed, 24 Jul 2013 01:00:53 +0000 (UTC) Received: by mail-ve0-f173.google.com with SMTP id jw11so6516415veb.4 for ; Tue, 23 Jul 2013 18:00:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=VbvbNBey4brmVDpgSAZbgDX/cPkvho6o0nWMt5Jlunc=; b=aNgUP5jLdeXq7d6jvVqpTZ9JxO71QtI4xA9GnVogtPoEyyE83E2GUiYOi+0Ux5s+bf Cy6gMSbTF6BBk1vaZUvyM2LAQ610Uc2pseUiJD2S3MEmfx7N1D62Tg6GXkzzfY9s8HjR 8BaCHP7aTZdq3lDKJQ1Fk3p94qIEd/CytS6bhm1pVD5TSkbhwmo3ohblrebAjpJHI0AM pEA70x9fljAcfEKW3k5NHRC90VHAsFUgzbMFDtE/FotkTxY+zyv4X7Z1FEEsfS9E9BhU gngRoYw/EQ7weyCmT4KE7HdnZDIKWCWfxVGZaqbt4Kl0GvpQnTmYLIcDQT6tLCC3VC3A R4Rw== MIME-Version: 1.0 X-Received: by 10.58.234.161 with SMTP id uf1mr12701626vec.57.1374627653244; Tue, 23 Jul 2013 18:00:53 -0700 (PDT) Received: by 10.221.22.199 with HTTP; Tue, 23 Jul 2013 18:00:53 -0700 (PDT) In-Reply-To: References: <06BA4BD5-BE4E-4184-AFBB-D7FD4B2597D9@your.org> <3341F665-B179-4A99-B48A-81EE3D0478A6@your.org> Date: Tue, 23 Jul 2013 21:00:53 -0400 Message-ID: Subject: Re: Duplicate Address Detection misfire? From: Zaphod Beeblebrox To: Kevin Day Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jul 2013 01:00:54 -0000 Doesn't my original suggestion still stand... regardless of how this particular problem is fixed? That is: if the sending MAC address is _our_ MAC address, then the address is not duplicate. It seems a simple change (unless the function that processes the packet would have difficulty determining the MAC address) and it seems to be infallible logic. In fact, if we receive a packet from our own MAC address, shouldn't we drop and log it? From owner-freebsd-net@FreeBSD.ORG Wed Jul 24 09:14:55 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id A88006BF; Wed, 24 Jul 2013 09:14:55 +0000 (UTC) (envelope-from alexl@mellanox.com) Received: from eu1sys200aog122.obsmtp.com (eu1sys200aog122.obsmtp.com [207.126.144.153]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 806F323F5; Wed, 24 Jul 2013 09:14:53 +0000 (UTC) Received: from MTLCAS01.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob122.postini.com ([207.126.147.11]) with SMTP ID DSNKUe+a8XXTsOguv4h+zAnqF9s630ASbSMY@postini.com; Wed, 24 Jul 2013 09:14:54 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS01.mtl.com ([10.0.8.71]) with mapi id 14.03.0123.003; Wed, 24 Jul 2013 12:14:24 +0300 From: Alex Liptsin To: "freebsd-questions@freebsd.org" , "freebsd-net@freebsd.org" Subject: How can I remove one interface from lagg, without destroying all lagg? Thread-Topic: How can I remove one interface from lagg, without destroying all lagg? Thread-Index: Ac6ITfoh9RO11JA5Sw+l5hpPWhRMLg== Date: Wed, 24 Jul 2013 09:14:23 +0000 Message-ID: <64DAB3164E410447932305F50F896D8D6AFAF99F@MTLDAG01.mtl.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Oded Shanoon , Regev Lev X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jul 2013 09:14:55 -0000 Hi. I have lagg interface created on my server: [root@h-qa-094 ~]$ ifconfig lagg0 lagg0: flags=3D8802 metric 0 mtu 1500 options=3D401bb ether 00:02:c9:19:82:80 nd6 options=3D21 media: Ethernet autoselect status: active laggproto failover lagghash l2,l3,l4 laggport: igb1 flags=3D0<> laggport: mlxen1 flags=3D0<> laggport: mlxen0 flags=3D5 Now, I want to removr igb1 interface from that lag. How can I do it? Regards, Alex Liptsin Software Quality Assurance Engineer | Mellanox Technologies Ltd. Office: +972 (74) 7236141 Mobile: +972(54) 7833986 Fax: +972(74) 7236161 Email: alexl@mellanox.com Mellanox, Tel-Hai Industrial Park. Building 7, M.P. Upper Galilee 12100 Isr= ael From owner-freebsd-net@FreeBSD.ORG Wed Jul 24 09:21:37 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 13ABFC23; Wed, 24 Jul 2013 09:21:37 +0000 (UTC) (envelope-from milu@dat.pl) Received: from jab.dat.pl (dat.pl [80.51.155.34]) by mx1.freebsd.org (Postfix) with ESMTP id 61EC9249B; Wed, 24 Jul 2013 09:21:35 +0000 (UTC) Received: from jab.dat.pl (jsrv.dat.pl [127.0.0.1]) by jab.dat.pl (Postfix) with ESMTP id B1EBAC2; Wed, 24 Jul 2013 11:21:34 +0200 (CEST) X-Virus-Scanned: amavisd-new at dat.pl Received: from jab.dat.pl ([127.0.0.1]) by jab.dat.pl (jab.dat.pl [127.0.0.1]) (amavisd-new, port 10024) with LMTP id GgKC8TRIQxv7; Wed, 24 Jul 2013 11:21:32 +0200 (CEST) Received: from [10.0.6.80] (unknown [212.69.68.42]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by jab.dat.pl (Postfix) with ESMTPSA id EC74F51; Wed, 24 Jul 2013 11:21:31 +0200 (CEST) Message-ID: <51EF9CB0.3000701@dat.pl> Date: Wed, 24 Jul 2013 11:21:52 +0200 From: Maciej Milewski User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130625 Thunderbird/17.0.7 MIME-Version: 1.0 To: Alex Liptsin Subject: Re: How can I remove one interface from lagg, without destroying all lagg? References: <64DAB3164E410447932305F50F896D8D6AFAF99F@MTLDAG01.mtl.com> In-Reply-To: <64DAB3164E410447932305F50F896D8D6AFAF99F@MTLDAG01.mtl.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" , Oded Shanoon , "freebsd-questions@freebsd.org" , Regev Lev X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jul 2013 09:21:37 -0000 On 24.07.2013 11:14, Alex Liptsin wrote: > Hi. > > I have lagg interface created on my server: > > [root@h-qa-094 ~]$ ifconfig lagg0 > lagg0: flags=8802 metric 0 mtu 1500 > options=401bb > ether 00:02:c9:19:82:80 > nd6 options=21 > media: Ethernet autoselect > status: active > laggproto failover lagghash l2,l3,l4 > laggport: igb1 flags=0<> > laggport: mlxen1 flags=0<> > laggport: mlxen0 flags=5 > > Now, I want to removr igb1 interface from that lag. > How can I do it? man lagg: Child interfaces can be added using the laggport child-iface option and removed using the -laggport child-iface option. so |ifconfig lagg0 -laggport /igb1/| should be working. > Regards, > Alex Liptsin > Software Quality Assurance Engineer | Mellanox Technologies Ltd. > Office: +972 (74) 7236141 > Mobile: +972(54) 7833986 > Fax: +972(74) 7236161 > Email: alexl@mellanox.com > Mellanox, Tel-Hai Industrial Park. Building 7, M.P. Upper Galilee 12100 Israel > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" -- Pozdrawiam, Maciej Milewski From owner-freebsd-net@FreeBSD.ORG Wed Jul 24 09:21:38 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E6623C24 for ; Wed, 24 Jul 2013 09:21:38 +0000 (UTC) (envelope-from jhellenthal@dataix.net) Received: from mail-oa0-x233.google.com (mail-oa0-x233.google.com [IPv6:2607:f8b0:4003:c02::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A5645249C for ; Wed, 24 Jul 2013 09:21:38 +0000 (UTC) Received: by mail-oa0-f51.google.com with SMTP id i4so361446oah.24 for ; Wed, 24 Jul 2013 02:21:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dataix.net; s=rsa; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:x-mailer:from:subject:date :to; bh=l4xhXSNtr3Aadf78FWp9UyY0zeqogz9HoMI6I9+XFAM=; b=YRyzY5HdOk/5+Jm5+Upb3uRhXIbAX5YPfSd87j90TSxBi8yn9/Ae91O0DdJhWEClda 5DmRGYJO4cIz931EyC/rPuGoit3dqezj5ENeYddLsACRV+jdpPyPKPkJcr6oGBz3Xb8Z LjD8Xpz+eaY9/wwUt1oITM8bdE6SW5SzDWxxs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:x-mailer:from:subject:date :to:x-gm-message-state; bh=l4xhXSNtr3Aadf78FWp9UyY0zeqogz9HoMI6I9+XFAM=; b=cnsnFNWfnoQLwpO+EnmCVH1Qr1yruYun236TL0l6gD5zkiAS6OUDBlHclJ+LhsTuOZ Xg85TXyB5A1OWyaS60iPxGVRbB37R94tStaLYL7WplrGrRrajhUmPxmJEo5Zol5AwApN eLviQJZVtvPUuEGPBWCAzaBG6mSnsJ9dafMtSOOlxsg8rznMyTh2jyXJ6fGoDirpjy0S NfpJ042w+E7AOXtvpBrZ7Flq+cECOYEekWo9G3irQl4fX16xhWreFUCCar85LD1L6C+3 Y5vV+HJP2kz3PSYIyAtetPyGt+BVgiXEqEOpJNM82NcS+4uN4a3j3oeoD+7ioToyb/Ip xZUw== X-Received: by 10.182.225.134 with SMTP id rk6mr29316385obc.40.1374657697989; Wed, 24 Jul 2013 02:21:37 -0700 (PDT) Received: from [192.168.31.185] (68-188-148-148.dhcp.aldl.mi.charter.com. [68.188.148.148]) by mx.google.com with ESMTPSA id z10sm47654379obl.13.2013.07.24.02.21.33 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 24 Jul 2013 02:21:36 -0700 (PDT) References: <64DAB3164E410447932305F50F896D8D6AFAF99F@MTLDAG01.mtl.com> Mime-Version: 1.0 (1.0) In-Reply-To: <64DAB3164E410447932305F50F896D8D6AFAF99F@MTLDAG01.mtl.com> Content-Type: multipart/signed; micalg=sha1; boundary=Apple-Mail-789E3631-BC1C-4573-8F13-1896F40B6978; protocol="application/pkcs7-signature" Content-Transfer-Encoding: 7bit Message-Id: <3FE0FA6C-D48C-44DD-A5D9-1FAD31D40B23@dataix.net> X-Mailer: iPhone Mail (10B350) From: Jason Hellenthal Subject: Re: How can I remove one interface from lagg, without destroying all lagg? Date: Wed, 24 Jul 2013 05:21:28 -0400 To: Alex Liptsin X-Gm-Message-State: ALoCoQlqKqBjgsqOy+Ykag0I8lzYr5DKkftCSngeQLTABHVOj07pwjBze+J4uzqbXVPwqttpniLo X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" , Oded Shanoon , "freebsd-questions@freebsd.org" , Regev Lev X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jul 2013 09:21:39 -0000 --Apple-Mail-789E3631-BC1C-4573-8F13-1896F40B6978 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Use -laggport portN --=20 Jason Hellenthal Inbox: jhellenthal@DataIX.net Voice: +1 (616) 953-0176 JJH48-ARIN On Jul 24, 2013, at 5:14, Alex Liptsin wrote: > Hi. >=20 > I have lagg interface created on my server: >=20 > [root@h-qa-094 ~]$ ifconfig lagg0 > lagg0: flags=3D8802 metric 0 mtu 1500 > options=3D401bb > ether 00:02:c9:19:82:80 > nd6 options=3D21 > media: Ethernet autoselect > status: active > laggproto failover lagghash l2,l3,l4 > laggport: igb1 flags=3D0<> > laggport: mlxen1 flags=3D0<> > laggport: mlxen0 flags=3D5 >=20 > Now, I want to removr igb1 interface from that lag. > How can I do it? >=20 >=20 >=20 >=20 > Regards, > Alex Liptsin > Software Quality Assurance Engineer | Mellanox Technologies Ltd. > Office: +972 (74) 7236141 > Mobile: +972(54) 7833986 > Fax: +972(74) 7236161 > Email: alexl@mellanox.com > Mellanox, Tel-Hai Industrial Park. Building 7, M.P. Upper Galilee 12100 Is= rael >=20 > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" --Apple-Mail-789E3631-BC1C-4573-8F13-1896F40B6978 Content-Type: application/pkcs7-signature; name=smime.p7s Content-Disposition: attachment; filename=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIUOTCCBjAw ggUYoAMCAQICAwaijjANBgkqhkiG9w0BAQsFADCBjDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0 YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcx ODA2BgNVBAMTL1N0YXJ0Q29tIENsYXNzIDEgUHJpbWFyeSBJbnRlcm1lZGlhdGUgQ2xpZW50IENB MB4XDTEzMDUxODA4NTA0OFoXDTE0MDUxOTIyMDk0N1owSDEfMB0GA1UEAwwWamhlbGxlbnRoYWxA ZGF0YWl4Lm5ldDElMCMGCSqGSIb3DQEJARYWamhlbGxlbnRoYWxAZGF0YWl4Lm5ldDCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBALgnYFS1bWZr3KhKBzWAdRwrY+En+RRV8nCaYubqrMG+ YJbuenaIKSbIuFiDWipW4RHYTpE28pKaSnaVTG9WtAZvsWj0gYN9g2fYCnCOUceES2Yvi3RavxpB hsuzKIfsHb8iNNSEuczLu6gn4mQyaHwE4x6xSUKmbK8njR+YoF522F60wjsnq5dlOJdTrhDfObE5 5P23279WbRp8azgZX1VRB66wdKRDuSI1vBts4Nsha2paXd6HUUduHrPACBQREJTGXN8XtEKVwo63 aKUhRgtUwHNEuSWck/xwVl7PBUWH2dORAWTCqHjNuCKNOQ1/0LMiyMj7FdsBjN4dgL4YZpsCAwEA AaOCAtwwggLYMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggr BgEFBQcDBDAdBgNVHQ4EFgQU29qUrmZtgQ7ZVoDKogfpJOSfk+YwHwYDVR0jBBgwFoAUU3Ltkpzg 2ssBXHx+ljVO8tS4UYIwIQYDVR0RBBowGIEWamhlbGxlbnRoYWxAZGF0YWl4Lm5ldDCCAUwGA1Ud IASCAUMwggE/MIIBOwYLKwYBBAGBtTcBAgMwggEqMC4GCCsGAQUFBwIBFiJodHRwOi8vd3d3LnN0 YXJ0c3NsLmNvbS9wb2xpY3kucGRmMIH3BggrBgEFBQcCAjCB6jAnFiBTdGFydENvbSBDZXJ0aWZp Y2F0aW9uIEF1dGhvcml0eTADAgEBGoG+VGhpcyBjZXJ0aWZpY2F0ZSB3YXMgaXNzdWVkIGFjY29y ZGluZyB0byB0aGUgQ2xhc3MgMSBWYWxpZGF0aW9uIHJlcXVpcmVtZW50cyBvZiB0aGUgU3RhcnRD b20gQ0EgcG9saWN5LCByZWxpYW5jZSBvbmx5IGZvciB0aGUgaW50ZW5kZWQgcHVycG9zZSBpbiBj b21wbGlhbmNlIG9mIHRoZSByZWx5aW5nIHBhcnR5IG9ibGlnYXRpb25zLjA2BgNVHR8ELzAtMCug KaAnhiVodHRwOi8vY3JsLnN0YXJ0c3NsLmNvbS9jcnR1MS1jcmwuY3JsMIGOBggrBgEFBQcBAQSB gTB/MDkGCCsGAQUFBzABhi1odHRwOi8vb2NzcC5zdGFydHNzbC5jb20vc3ViL2NsYXNzMS9jbGll bnQvY2EwQgYIKwYBBQUHMAKGNmh0dHA6Ly9haWEuc3RhcnRzc2wuY29tL2NlcnRzL3N1Yi5jbGFz czEuY2xpZW50LmNhLmNydDAjBgNVHRIEHDAahhhodHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS8wDQYJ KoZIhvcNAQELBQADggEBAHsw8/Hw07gsNTKYnld74NBFtHnQOPkXYuccWx3j0PGQe9nqNxeingBf 2yvx+xBQzBoi4J1u84Jbrbe8Ii3+LLD/QMW9cN0SBIgRStPQLVee4STdjeabGmpXQa7omC02wYYO 83qh6CgJEIbmrsBSZH8ZSVrjkC4UmZS8wAQMS3qTWAPF0ZQGWx2+Gks2fXuacyt2LpNR+p9ogjAZ 1/rmUKjNhQZLswytaLRUdwAwSfQ3+TNs68h6Kv1LC3bNGBT3NEtr2q/nzzb5MzuFcDE6f9exroAC 4BHmokAprhna/vZdb6BrPjpXgRAlWAh3wEMxw75M9S/Nbzj/jNp+I+lvUJYwggY0MIIEHKADAgEC AgEeMA0GCSqGSIb3DQEBBQUAMH0xCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQu MSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMSkwJwYDVQQDEyBT dGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0wNzEwMjQyMTAxNTVaFw0xNzEwMjQy MTAxNTVaMIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMi U2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMvU3RhcnRDb20gQ2xh c3MgMSBQcmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IB DwAwggEKAoIBAQDHCYPMzi3YGrEppC4Tq5a+ijKDjKaIQZZVR63UbxIP6uq/I0fhCu+cQhoUfE6E RKKnu8zPf1Jwuk0tsvVCk6U9b+0UjM0dLep3ZdE1gblK/1FwYT5Pipsu2yOMluLqwvsuz9/9f1+1 PKHG/FaR/wpbfuIqu54qzHDYeqiUfsYzoVflR80DAC7hmJ+SmZnNTWyUGHJbBpA8Q89lGxahNvur yGaC/o2/ceD2uYDX9U8Eg5DpIpGQdcbQeGarV04WgAUjjXX5r/2dabmtxWMZwhZna//jdiSyrrSM TGKkDiXm6/3/4ebfeZuCYKzN2P8O2F/Xe2AC/Y7zeEsnR7FOp+uXAgMBAAGjggGtMIIBqTAPBgNV HRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQUU3Ltkpzg2ssBXHx+ljVO8tS4 UYIwHwYDVR0jBBgwFoAUTgvvGqRAW6UXaYcwyjRoQ9BBrvIwZgYIKwYBBQUHAQEEWjBYMCcGCCsG AQUFBzABhhtodHRwOi8vb2NzcC5zdGFydHNzbC5jb20vY2EwLQYIKwYBBQUHMAKGIWh0dHA6Ly93 d3cuc3RhcnRzc2wuY29tL3Nmc2NhLmNydDBbBgNVHR8EVDBSMCegJaAjhiFodHRwOi8vd3d3LnN0 YXJ0c3NsLmNvbS9zZnNjYS5jcmwwJ6AloCOGIWh0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL3Nmc2Nh LmNybDCBgAYDVR0gBHkwdzB1BgsrBgEEAYG1NwECATBmMC4GCCsGAQUFBwIBFiJodHRwOi8vd3d3 LnN0YXJ0c3NsLmNvbS9wb2xpY3kucGRmMDQGCCsGAQUFBwIBFihodHRwOi8vd3d3LnN0YXJ0c3Ns LmNvbS9pbnRlcm1lZGlhdGUucGRmMA0GCSqGSIb3DQEBBQUAA4ICAQAKgwh9eKssBly4Y4xerhy5 I3dNoXHYfYa8PlVLL/qtXnkFgdtY1o95CfegFJTwqBBmf8pyTUnFsukDFUI22zF5bVHzuJ+GxhnS qN2sD1qetbYwBYK2iyYA5Pg7Er1A+hKMIzEzcduRkIMmCeUTyMyikfbUFvIBivtvkR8ZFAk22BZy +pJfAoedO61HTz4qSfQoCRcLN5A0t4DkuVhTMXIzuQ8CnykhExD6x4e6ebIbrjZLb7L+ocR0y4Yj Cl/Pd4MXU91y0vTipgr/O75CDUHDRHCCKBVmz/Rzkc/b970MEeHt5LC3NiWTgBSvrLEuVzBKM586 YoRD9Dy3OHQgWI270g+5MYA8GfgI/EPT5G7xPbCDz+zjdH89PeR3U4So4lSXur6H6vp+m9TQXPF3 a0LwZrp8MQ+Z77U1uL7TelWO5lApsbAonrqASfTpaprFVkL4nyGH+NHST2ZJPWIBk81i6Vw0ny0q ZW2Niy/QvVNKbb43A43ny076khXO7cNbBIRdJ/6qQNq9Bqb5C0Q5nEsFcj75oxQRqlKf6TcvGbjx kJh8BYtv9ePsXklAxtm8J7GCUBthHSQgepbkOexhJ0wP8imUkyiPHQ0GvEnd83129fZjoEhdGwXV 27ioRKbj/cIq7JRXun0NbeY+UdMYu9jGfIpDLtUUGSgsg2zMGs5R4jCCB8kwggWxoAMCAQICAQEw DQYJKoZIhvcNAQEFBQAwfTELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzAp BgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcxKTAnBgNVBAMTIFN0YXJ0 Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTA2MDkxNzE5NDYzNloXDTM2MDkxNzE5NDYz NlowfTELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3Vy ZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcxKTAnBgNVBAMTIFN0YXJ0Q29tIENlcnRpZmlj YXRpb24gQXV0aG9yaXR5MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAwYjbCbxsRnx4 n5V7tTOQ8nJi1sE2ICIkXs7pd/JDCqIGZKTMjjb4OOYj8G5tsTzdcqOFHKHTPbQzK9Mvr/7qsEFZ Z7bEBn0KnnSF1nlMgDd63zkFUln39BtGQ6TShYXSw3HzdWI0uiyKfx6P7u000BHHls1SPboz1t1N 3gs7SkufwiYv+rUWHHI1d8o8XebK4SaLGjZ2XAHbdBQl/u21oIgP3XjKLR8HlzABLXJ5+kbWEyqo uaarg0kd5fLv3eQBjhgKj2NTFoViqQ4ZOsy1ZqbCa3QH5Cvhdj60bdj2ROFzYh87xL6gU1YlbFEJ 96qryr92/W2b853bvz1mvAxWqq+YSJU6S9+nWFDZOHWpW+pDDAL/mevobE1wWyllnN2qXcyvATHs DOvSjejqnHvmbvcnZgwaSNduQuM/3iE+e+ENcPtjqqhsGlS0XCV6yaLJixamuyx+F14FTVhuEh0B 7hIQDcYyfxj//PT6zW6R6DZJvhpIaYvClk0aErJpF8EKkNb6eSJIv7p7afhwx/p6N9jYDdJ2T1f/ kLfjkdLd78Jgt2c63f6qnPDUi39yIs7Gn5e2+K+KoBCo2fsYxra1XFI8ibYZKnMBCg8DsxJg8nov gdujbv8mMJf1i92JV7atPbOvK8W3dgLwpdYrmoYUKnL24zOMXQlLE9+7jHQTUksCAwEAAaOCAlIw ggJOMAwGA1UdEwQFMAMBAf8wCwYDVR0PBAQDAgGuMB0GA1UdDgQWBBROC+8apEBbpRdphzDKNGhD 0EGu8jBkBgNVHR8EXTBbMCygKqAohiZodHRwOi8vY2VydC5zdGFydGNvbS5vcmcvc2ZzY2EtY3Js LmNybDAroCmgJ4YlaHR0cDovL2NybC5zdGFydGNvbS5vcmcvc2ZzY2EtY3JsLmNybDCCAV0GA1Ud IASCAVQwggFQMIIBTAYLKwYBBAGBtTcBAQEwggE7MC8GCCsGAQUFBwIBFiNodHRwOi8vY2VydC5z dGFydGNvbS5vcmcvcG9saWN5LnBkZjA1BggrBgEFBQcCARYpaHR0cDovL2NlcnQuc3RhcnRjb20u b3JnL2ludGVybWVkaWF0ZS5wZGYwgdAGCCsGAQUFBwICMIHDMCcWIFN0YXJ0IENvbW1lcmNpYWwg KFN0YXJ0Q29tKSBMdGQuMAMCAQEagZdMaW1pdGVkIExpYWJpbGl0eSwgcmVhZCB0aGUgc2VjdGlv biAqTGVnYWwgTGltaXRhdGlvbnMqIG9mIHRoZSBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhv cml0eSBQb2xpY3kgYXZhaWxhYmxlIGF0IGh0dHA6Ly9jZXJ0LnN0YXJ0Y29tLm9yZy9wb2xpY3ku cGRmMBEGCWCGSAGG+EIBAQQEAwIABzA4BglghkgBhvhCAQ0EKxYpU3RhcnRDb20gRnJlZSBTU0wg Q2VydGlmaWNhdGlvbiBBdXRob3JpdHkwDQYJKoZIhvcNAQEFBQADggIBABZsmfRmDDT10IVefQrs 2hBOOBxe36YlBUuRMsHoO/E93UQJWwdJiinLZgK3sZr3JZgJPI4b4d02hytLu2jTOWY9oCbH8jmR HVGrgnt+1c5a5OIDV3Bplwj5XlimCt+MBppFFhY4Cl5X9mLHegIF5rwetfKe9Kkpg/iyFONuKIdE w5Aa3jipPKxDTWRFzt0oqVzyc3sE+Bfoq7HzLlxkbnMxOhK4vLMR5H2PgVGaO42J9E2TZns8A+3T mh2a82VQ9aDQdZ8vr/DqgkOY+GmciXnEQ45GcuNkNhKv9yUeOImQd37Da2q5w8tES6x4kIvnxywe SxFEyDRSJ80KXZ+FwYnVGnjylRBTMt2AhGZ12bVoKPthLr6EqDjAmRKGpR5nZK0GLi+pcIXHlg98 iWX1jkNUDqvdpYA5lGDANMmWcCyjEvUfSHu9HH5rt52Q9CI7rvj8Ksr6glKg769LVZPrwbXwIous NE4mIgShhyx1SrflfRPXuAxkwDbSyS+GEowjCcEbgjtzSaNqV4eU5dZ4xZlDY+NN4Hct4WWZcmkE GkcJ5g8BViT7H78OealYLrnECQF+lbptAAY+supKEDnY0Cv1v+x1v5cCxQkbCNxVN+KB+zeEQ2Ig yudWS2Xq/mzBJJMkoTTrBf+aIq6bfT/xZVEKpjBqs/SIHIAN/HKK6INeMYIDbzCCA2sCAQEwgZQw gYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUg RGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFBy aW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQQIDBqKOMAkGBSsOAwIaBQCgggGvMBgGCSqGSIb3 DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEzMDcyNDA5MjEzMlowIwYJKoZIhvcN AQkEMRYEFOFVtV0xPOji3AQv/+r51bhDySeoMIGlBgkrBgEEAYI3EAQxgZcwgZQwgYwxCzAJBgNV BAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBD ZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkgSW50 ZXJtZWRpYXRlIENsaWVudCBDQQIDBqKOMIGnBgsqhkiG9w0BCRACCzGBl6CBlDCBjDELMAkGA1UE BhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBEaWdpdGFsIENl cnRpZmljYXRlIFNpZ25pbmcxODA2BgNVBAMTL1N0YXJ0Q29tIENsYXNzIDEgUHJpbWFyeSBJbnRl cm1lZGlhdGUgQ2xpZW50IENBAgMGoo4wDQYJKoZIhvcNAQEBBQAEggEAeZELEzEMWNNFwYLp3G0n 8cYtEOWaUZi7sDYTJip6rJ6p7cMl1a8xRwtu5ER5ms/JSVsZwYu+SXsBpAXBvju9mtgUa5EUrvK7 gaftHwHPURFxY7ZhBjHendStLPk36jHUAHlkpp1AtTTO79d+MKoFp0InHg7V9ZWZEma7n1RyXFeI 3X1CqjeNLfmeT0/t7hCyiS9x26NX2EEx6P1KMAAj+rjvILf+7j2RwtDHe8V/hcwZSFlHUPAz7Dh2 WJdYbYd5UsIcz0vSxh+dLL8X6VHFTjx/XMh50blEgmfT4M4+XgCZ/qpUyYHmj4X7OTmkq9ZOa08J q43HaEXNS+w5cf9DHgAAAAAAAA== --Apple-Mail-789E3631-BC1C-4573-8F13-1896F40B6978-- From owner-freebsd-net@FreeBSD.ORG Wed Jul 24 13:20:01 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5EEF6624 for ; Wed, 24 Jul 2013 13:20:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3F2712F6E for ; Wed, 24 Jul 2013 13:20:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6ODK0HO013593 for ; Wed, 24 Jul 2013 13:20:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6ODK0mf013592; Wed, 24 Jul 2013 13:20:00 GMT (envelope-from gnats) Date: Wed, 24 Jul 2013 13:20:00 GMT Message-Id: <201307241320.r6ODK0mf013592@freefall.freebsd.org> To: freebsd-net@FreeBSD.org Cc: From: Meny Yossefi Subject: RE: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragmented packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Meny Yossefi List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jul 2013 13:20:01 -0000 The following reply was made to PR kern/180430; it has been noted by GNATS. From: Meny Yossefi To: John Baldwin Cc: "bug-followup@freebsd.org" Subject: RE: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragmented packets Date: Wed, 24 Jul 2013 13:14:53 +0000 --_000_F2E47A38E4D0B9499D76F2AB8901571A73A0EC13MTLDAG01mtlcom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi John, Looks good. -Meny From: Meny Yossefi Sent: Tuesday, July 23, 2013 5:01 PM To: 'John Baldwin' Cc: 'bug-followup@freebsd.org' Subject: RE: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragment= ed packets Let's stick with the FreeBSD standards (without the likely/unlikely) Let me just double check the CSUM flags work as expected. I'll get back to you as soon as I'm done. -Meny -----Original Message----- From: John Baldwin [mailto:jhb@freebsd.org] Sent: Monday, July 22, 2013 7:04 PM To: Meny Yossefi Cc: bug-followup@freebsd.org Subject: Re: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragment= ed packets On Monday, July 22, 2013 10:11:51 am Meny Yossefi wrote: > Hi John, > > > > The problem is that the HW will only calculate csum for parts of the > payload, one fragment at a time, > > while the receiver side, in our case the tcp/ip stack, will expect to val= idate the packet's payload as a whole. Ok. > I agree with the change you offered, though one thing bothers me. > > This change will add two conditions to the send flow which will probably = have an effect on performance. > > Maybe 'likely' can be useful here ? FreeBSD tends to not use likely/unlikely unless there is a demonstrable gai= n via benchmark measurements. However, if the OFED code regularly uses it = and you feel this is a case where you would normally use it, it can be adde= d. > BTW, I'm not entirely sure, but I think the CSUM_IP flag is always set, s= o maybe the first condition is not necessary. > > What do you think ? If the user uses ifconfig to disable checksum offload and force software ch= ecksums the flag will not be set. > -Meny > > > > > > -----Original Message----- > From: John Baldwin [mailto:jhb@freebsd.org] > Sent: Friday, July 19, 2013 6:29 PM > To: bug-followup@freebsd.org; Meny Yosse= fi > Subject: Re: kern/180430: [ofed] [patch] Bad UDP checksum calc for > fragmented packets > > > > Oops, my previous reply didn't make it to the PR itself. > > > > Is the problem that the hardware checksum overwrites arbitrary data in th= e packet (at the location where the UDP header would be)? > > > > Also, I've looked at other drivers, and they all look at the CSUM_* > flags to determine the right settings. IP fragments will not have > CSUM_UDP or CSUM_TCP set, so I think the more correct fix is this: > > > > Index: en_tx.c > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > --- en_tx.c (revision 253470) > > +++ en_tx.c (working copy) > > @@ -780,8 +780,12 @@ retry: > > tx_desc->ctrl.srcrb_flags =3D > cpu_to_be32(MLX4_WQE_CTRL_CQ_UPDATE | > > > MLX4_WQE_CTRL_SOLICITED); > > if (mb->m_pkthdr.csum_flags & > (CSUM_IP|CSUM_TCP|CSUM_UDP)) { > > - tx_desc->ctrl.srcrb_flags |=3D cpu_to_be32= (MLX4_WQE_CTRL_IP_CSUM | > > - = MLX4_WQE_CTRL_TCP_UDP_CSUM); > > + if (mb->m_pkthdr.csum_flags & CSUM_IP) > > + > + tx_desc->ctrl.srcrb_flags |=3D > > + > + cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM); > > + if (mb->m_pkthdr.csum_flags & > + (CSUM_TCP|CSUM_UDP)) > > + > + tx_desc->ctrl.srcrb_flags |=3D > > + > + cpu_to_be32(MLX4_WQE_CTRL_TCP_UDP_CSUM); > > priv->port_stats.tx_chksum_offload++; > > } > > > > -- > > John Baldwin > -- John Baldwin --_000_F2E47A38E4D0B9499D76F2AB8901571A73A0EC13MTLDAG01mtlcom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi John,

 

Looks good.

 

-Meny

 

From: Meny Yos= sefi
Sent: Tuesday, July 23, 2013 5:01 PM
To: 'John Baldwin'
Cc: 'bug-followup@freebsd.org'
Subject: RE: kern/180430: [ofed] [patch] Bad UDP checksum calc for f= ragmented packets

 

Let's stick with th= e FreeBSD standards (without the likely/unlikely)

 

Let me just double = check the CSUM flags work as expected.

I’ll get back= to you as soon as I’m done.

 

-Meny

 

 

-----Original Message-----
From: John Baldwin [mailto:jhb@freebsd.o= rg]
Sent: Monday, July 22, 2013 7:04 PM
To: Meny Yossefi
Cc: bug-followup@freebsd.org
Subject: Re: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragment= ed packets

 

On Monday, July 22, 2013 10:11:51 am Meny Yossefi= wrote:

> Hi John,

>

>

>

> The problem is that the HW will only calcula= te csum for parts of the

> payload, one fragment at a time,<= /p>

>

> while the receiver side, in our case the tcp= /ip stack, will expect to validate the packet's payload as a whole.

 

Ok.

 

> I agree with the change you offered, though = one thing bothers me.

>

> This change will add two conditions to the s= end flow which will probably have an effect on performance.

>

> Maybe 'likely' can be useful here ?

 

FreeBSD tends to not use likely/unlikely unless t= here is a demonstrable gain via benchmark measurements.  However, if t= he OFED code regularly uses it and you feel this is a case where you would = normally use it, it can be added.

 

> BTW, I'm not entirely sure, but I think the = CSUM_IP flag is always set, so maybe the first condition is not necessary.<= o:p>

>

> What do you think ?

 

If the user uses ifconfig to disable checksum off= load and force software checksums the flag will not be set.

 

> -Meny

>

>

>

>

>

> -----Original Message-----

> From: John Baldwin [mailto:jhb= @freebsd.org]

> Sent: Friday, July 19, 2013 6:29 PM

> To: bug-followup@free= bsd.org; Meny Yossefi

> Subject: Re: kern/180430: [ofed] [patch] Bad= UDP checksum calc for

> fragmented packets

>

>

>

> Oops, my previous reply didn't make it to th= e PR itself.

>

>

>

> Is the problem that the hardware checksum ov= erwrites arbitrary data in the packet (at the location where the UDP header= would be)?

>

>

>

> Also, I've looked at other drivers, and they= all look at the CSUM_*

> flags to determine the right settings. = IP fragments will not have

> CSUM_UDP or

CSUM_TCP set, so I think the more correct fix is = this:

>

>

>

> Index: en_tx.c

>

> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D

>

> --- en_tx.c     &nb= sp;     (revision 253470)

>

> +++ en_tx.c   &nb= sp;    (working copy)

>

> @@ -780,8 +780,12 @@ retry:

>

>       &nb= sp;        tx_desc->ctrl.srcrb_flags = =3D

> cpu_to_be32(MLX4_WQE_CTRL_CQ_UPDATE |

>

>       &nb= sp;            =             &nb= sp;            =             &nb= sp;            =             &nb= sp;            

> MLX4_WQE_CTRL_SOLICITED);

>

>       &nb= sp;        if (mb->m_pkthdr.csum_flag= s &

> (CSUM_IP|CSUM_TCP|CSUM_UDP)) {

>

> -       &= nbsp;           &nbs= p;          tx_desc->ctrl.s= rcrb_flags |=3D cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM |

>

> -       &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;  MLX4_WQE_CTRL_TCP_UDP_CSUM);

>

> +       &n= bsp;            = ;         if (mb->m_pkthdr.= csum_flags & CSUM_IP)

>

> +      &nb= sp;            =             &nb= sp;            =

> + tx_desc->ctrl.srcrb_flags |=3D=

>

> +      &nb= sp;            =             &nb= sp;            =     

> + cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM);

>

> +      &nb= sp;             = ;         if (mb->m_pkthdr.= csum_flags &

> + (CSUM_TCP|CSUM_UDP))

>

> +      &nb= sp;            =             &nb= sp;            =

> + tx_desc->ctrl.srcrb_flags |=3D=

>

> +      &nb= sp;            =             &nb= sp;            =     

> + cpu_to_be32(MLX4_WQE_CTRL_TCP_UDP_CSUM= );

>

>       &nb= sp;            =             priv->= ;port_stats.tx_chksum_offload++;

>

>       &nb= sp;        }

>

>

>

> --

>

> John Baldwin

>

 

--

John Baldwin

--_000_F2E47A38E4D0B9499D76F2AB8901571A73A0EC13MTLDAG01mtlcom_-- From owner-freebsd-net@FreeBSD.ORG Wed Jul 24 20:36:24 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 91F0E18E for ; Wed, 24 Jul 2013 20:36:24 +0000 (UTC) (envelope-from spawk@acm.poly.edu) Received: from acm.poly.edu (acm.poly.edu [64.147.113.26]) by mx1.freebsd.org (Postfix) with ESMTP id 6E63A2AE4 for ; Wed, 24 Jul 2013 20:36:24 +0000 (UTC) Received: from acm.poly.edu (localhost [127.0.0.1]) by acm.poly.edu (Postfix) with ESMTP id 9F1681F13AB for ; Wed, 24 Jul 2013 16:27:37 -0400 (EDT) Received: (qmail 28843 invoked from network); 24 Jul 2013 20:27:37 -0000 Received: from unknown (HELO ?10.50.50.204?) (spawk@64.147.100.2) by acm.poly.edu with CAMELLIA256-SHA encrypted SMTP; 24 Jul 2013 20:27:37 -0000 Message-ID: <51F0386D.2000709@acm.poly.edu> Date: Wed, 24 Jul 2013 16:26:21 -0400 From: Boris Kochergin User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:17.0) Gecko/20130416 Thunderbird/17.0.5 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Recommendations for 10gbps NIC Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jul 2013 20:36:24 -0000 Hi. I am looking for recommendations for a 10gbps NIC from someone who has successfully used it on FreeBSD. It will be used on FreeBSD 9.1-R/amd64 to capture packets. Some desired features are: - PCIe - LC connectors - 10GBASE-SR - Either single- or dual-port - Multiqueue Specific part numbers would be appreciated. Thank you. -Boris From owner-freebsd-net@FreeBSD.ORG Wed Jul 24 21:12:53 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7EFB1BB1 for ; Wed, 24 Jul 2013 21:12:53 +0000 (UTC) (envelope-from sean_bruno@yahoo.com) Received: from nm36-vm8.bullet.mail.gq1.yahoo.com (nm36-vm8.bullet.mail.gq1.yahoo.com [98.136.216.223]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 32B172C5A for ; Wed, 24 Jul 2013 21:12:52 +0000 (UTC) Received: from [98.137.12.190] by nm36.bullet.mail.gq1.yahoo.com with NNFMP; 24 Jul 2013 21:07:25 -0000 Received: from [98.136.164.76] by tm11.bullet.mail.gq1.yahoo.com with NNFMP; 24 Jul 2013 21:07:25 -0000 Received: from [127.0.0.1] by smtp238.mail.gq1.yahoo.com with NNFMP; 24 Jul 2013 21:07:25 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1374700045; bh=wbj4tWuo2zfL2xRxUrml8UAMsIRMuqQnswo4TNmIhLs=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:Subject:From:Reply-To:To:Content-Type:Date:Message-ID:Mime-Version:X-Mailer; b=cq5Abby9WPUsn50EobQWbSGQEaOA9M3B/AqoQ6mVXuboqAW3Rf5TkDvY9gYESBv1guR+S8ORwl/IlwaIfTPmJ5zRkznGNe+Rm1JbzlZxB0yMQ2fohLz5NXyqSYuf8gY/lxXsZgtnkdmggMcRJPflLfepilUM2II03cs6y1ukQwg= X-Yahoo-Newman-Id: 749798.86900.bm@smtp238.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: HUnlBSUVM1k8fJ0AZaAKAJgqZGDhNwQqio.qdQ8WAj2xvJq Oba1MnwUIVuB00wFzC4aom.VLnsu5Opb20IOUMDR1i9SymEzFbQ7Me.cPfhp S7MVnD32QjTNW_k0XRPjuCl5yUUNs96KWvcfwHUCNwod7Snctx9TMxNrIiL8 bH8VobvZX0KY_BU_5_TtI8cXZlqOWtw6rkKE7VtPuWFIChbN2_KT23fHwUh8 lJ8zFKgTApJzDRvmS8J_fdAg8cfz2RuD_5Efuk_yvJN6nfdkmrNe2opPJ3_D LlKIVi_f3p4O5z9T4jFYDbqSs6uYL7oX1vLoJr.NbKArUbLld.SmsWCE9pT_ YXGPR9uL_TAdZnSkXyp1M5Gbr1iFhZFXgb3mzKCs1yhLJT2V3UJ3c8N9TEfa phoKOVphSxptlwB77SFfaPiowYfEhhSO_X8Sd6dE3plWSQLqDWSk0lJyO8A8 cXEygxivourQI08Vn.SPD3zWzRGS1pmvPuL1cAIqGmodBbpH6ekV01lkfkgL RmGz0o6qCJTURtyFQlQ6ZKwiDPuT3n5_TlmK9g60BCQRd8thpD1cjhOVofcU _Ew5_5kLJ5npoHSa81hieyWsn9aCwsPYOpzrg_ZYxG2lWRHmGCjzyFFmbRKQ e4nzPsORQrRqppd4- X-Yahoo-SMTP: u5BKR6OswBC_iZJVfGRoMkTIpc8pEA4- X-Rocket-Received: from [192.168.43.121] (sean_bruno@70.197.25.149 with ) by smtp238.mail.gq1.yahoo.com with SMTP; 24 Jul 2013 21:07:24 +0000 UTC Subject: bce(4) panics, 9.2rc1 From: Sean Bruno To: "freebsd-net@freebsd.org" Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-6jvZXs+m5HP/cvZnZ+vb" Date: Wed, 24 Jul 2013 14:07:22 -0700 Message-ID: <1374700042.1493.14.camel@localhost> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: sbruno@freebsd.org List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jul 2013 21:12:53 -0000 --=-6jvZXs+m5HP/cvZnZ+vb Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Running 9.2 in production load mail servers. We're hitting the "watchdog" message and crashing with the stable/9 version. We're reverting the change from 2 weeks ago and seeing if it still happens. We didn't see this from stable/9 from about a month ago. Sean ref:=20 http://svnweb.freebsd.org/base?view=3Drevision&revision=3D253128 bce0: ../../../dev/bce/if_bce.c(7871): Watchdog timeout occurred, resetting! Jul 24 20:22:15=20 nm4 7 syslogd: sendtFatal trap 12: page fault while in kernel mode o: No route to hcpuid =3D 13; ost Jul 24 20:2apic id =3D 13 f2:15 nm47 kernel: rtual addressbce0: link state =3D 0x10 changed to DOWNfau Jul 24 20:22:1lt 5 ncode =3D supervisor read m47 syslogd: sendadto: No route tota, page no host t present instruction pointer =3D 0x20:0xffffffff803a1931 stack pointer =3D 0x28:0xffffff8c5be0aab0 frame pointer =3D 0x28:0xffffff8c5be0ab60 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 12 (irq256: bce0) trap number =3D 12 panic: page fault cpuid =3D 13 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff8c5be0a540 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8c5be0a600 panic() at panic+0x1d8/frame 0xffffff8c5be0a700 trap_fatal() at trap_fatal+0x290/frame 0xffffff8c5be0a760 trap_pfault() at trap_pfault+0x1d2/frame 0xffffff8c5be0a7f0 trap() at trap+0x43a/frame 0xffffff8c5be0a9f0 calltrap() at calltrap+0x8/frame 0xffffff8c5be0a9f0 --- trap 0xc, rip =3D 0xffffffff803a1931, rsp =3D 0xffffff8c5be0aab0, rbp = =3D 0xffffff8c5be0ab60 --- bce_intr() at bce_intr+0x301/frame 0xffffff8c5be0ab60 intr_event_execute_handlers() at intr_event_execute_handlers+0x6a/frame 0xffffff8c5be0ab90 ithread_loop() at ithread_loop+0xac/frame 0xffffff8c5be0abe0 fork_exit() at fork_exit+0x135/frame 0xffffff8c5be0ac30 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8c5be0ac30 --- trap 0, rip =3D 0, rsp =3D 0xffffff8c5be0acf0, rbp =3D 0 --- Uptime: 38m18s Dumping 2120 out of 24538 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Dump complete --=-6jvZXs+m5HP/cvZnZ+vb Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (FreeBSD) iQEcBAABAgAGBQJR8EIAAAoJEBkJRdwI6BaHApwH/ilEk3JMb0EkM3niWu4XJJDN SZ6qD6XJasQM8xI7dtTp813d0Cxv8dLzxlhjNbSC44Sam8IPm7heFCKZInjbOPg7 0IpdmEhwesVw7GfRMgbhRhedXjWM5W1hTFOp0cWXc2KUNDvqDjbmw/3KXcGr/dCi 50knEUkuIUuG153y8EycRclMAK/q36AJoIvV5OY921H8B48wHNbBgskSaUZJDbrD IUt8SrewcKNG9+Iju/3qpinDJJkp3sglr+P1Y0uFjmFOIEDjYw4TYdBoALU/XMc1 TQcI55S6DUOLXiImSmWWTxcMMQJWdDdGI411cHjLH6i8xuUTyPZTlYiA9IfnZL0= =KOPg -----END PGP SIGNATURE----- --=-6jvZXs+m5HP/cvZnZ+vb-- From owner-freebsd-net@FreeBSD.ORG Wed Jul 24 21:23:48 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 74E5AD59 for ; Wed, 24 Jul 2013 21:23:48 +0000 (UTC) (envelope-from sean_bruno@yahoo.com) Received: from nm24-vm0.bullet.mail.bf1.yahoo.com (nm24-vm0.bullet.mail.bf1.yahoo.com [98.139.213.161]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E1F4D2CA4 for ; Wed, 24 Jul 2013 21:23:47 +0000 (UTC) Received: from [98.139.215.141] by nm24.bullet.mail.bf1.yahoo.com with NNFMP; 24 Jul 2013 21:23:46 -0000 Received: from [68.142.230.74] by tm12.bullet.mail.bf1.yahoo.com with NNFMP; 24 Jul 2013 21:23:46 -0000 Received: from [127.0.0.1] by smtp231.mail.bf1.yahoo.com with NNFMP; 24 Jul 2013 21:23:46 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1374701026; bh=j8nzMI0RO9GSVwOLHEt1+tCgjvtgFestDo07GIQqRhU=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:Subject:From:Reply-To:To:Cc:In-Reply-To:References:Content-Type:Date:Message-ID:Mime-Version:X-Mailer; b=Ik5/H3hE6aTyj0dOiuizquXha+HpAwD+eiLnK8GtAUPP95UPiYsO4bBF+fEX8rnkV20rNldbIJfQW04F2sOaWe++Xq+mrWCG2c0Fmd5tJGllPolFx+3Eiuo1Q2GsV2b/gX3UquofJIrJgvzlt4Xg2ffMiEA4nNXxeHlO3A5uPvw= X-Yahoo-Newman-Id: 306575.20483.bm@smtp231.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: qZEJj2YVM1nByP9Qp97LzWSN.UxVsaPj8S0y7Z2t5SqquzJ HgSRqrA8f5kdtP49wFnQDq0800SczDFoPIEl_Xk2m6zmgXwlJYGjw3gxHL3b o15gg1ptMkRXl7yo_Qh186XZIn2NB6gBmQTazT3SrAkC_jT3REKIIC7BByKp 2zC9S09cyBEueymQjhIwH.ZesjrhbiSgb4FSfjcXuHmIT7SkDwDcDLOYC99O IJDaSQXrfE8hCJlDNd8xmNAdIKhXN2Vc1B0x2jgpVKUGnk5.P8KL0q6oStQ5 Yvr30d1t7EUcvbFu7N9BKTEZTx6LalZbCd9zOBxB_BEeLSWkmxRCSEoSidZV 7xIgtZzV2Di76HDYxoejC7taAfgXWTRqfTMA4JpxL1XR6EDBhwEi36XaEFT7 w7hNQGR8kgHBj8wg94O_dI35CZc4.3wX7ms4e5HbWSoEHh541W_OCdCQzTEr G5S5VMewisTckw0.rFiVkNd_05ysUATY6oeOWriqbmgU3aVe.FC_1ECFAOCa R3MA.9MLGi8B6RatoDXkOfi15XWab9gQTP2FyCy0IRRc2bIYOxSw1pQjIWZU ykoiIa6y9s5qt2O5Sji_HC_zSsioSHCPoPlaoBX4bvMQU77.wpTEWO6kAGD3 .Vgc8uMDsz3EujzY- X-Yahoo-SMTP: u5BKR6OswBC_iZJVfGRoMkTIpc8pEA4- X-Rocket-Received: from [192.168.43.121] (sean_bruno@70.197.25.149 with ) by smtp231.mail.bf1.yahoo.com with SMTP; 24 Jul 2013 21:23:46 +0000 UTC Subject: Re: bce(4) panics, 9.2rc1 From: Sean Bruno To: sbruno@freebsd.org In-Reply-To: <1374700042.1493.14.camel@localhost> References: <1374700042.1493.14.camel@localhost> Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-ZH2ePxxmHc69O0Q0uBpo" Date: Wed, 24 Jul 2013 14:23:42 -0700 Message-ID: <1374701022.1964.0.camel@localhost> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: sbruno@freebsd.org List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jul 2013 21:23:48 -0000 --=-ZH2ePxxmHc69O0Q0uBpo Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable On Wed, 2013-07-24 at 14:07 -0700, Sean Bruno wrote: > Running 9.2 in production load mail servers. We're hitting the > "watchdog" message and crashing with the stable/9 version. We're > reverting the change from 2 weeks ago and seeing if it still happens. > We didn't see this from stable/9 from about a month ago. >=20 >=20 > Sean >=20 > ref:=20 > http://svnweb.freebsd.org/base?view=3Drevision&revision=3D253128 These panics are happening on: bce0: mem 0xda000000-0xdbffffff irq 36 at device 0.0 on pci1 miibus0: on bce0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce0: Ethernet address: d4:ae:52:8d:42:fc bce0: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.11) Coal (RX:6,6,18,18; TX:20,20,80,80) bce1: mem 0xdc000000-0xddffffff irq 48 at device 0.1 on pci1 miibus1: on bce1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce1: Ethernet address: d4:ae:52:8d:42:fd bce1: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.11) Coal (RX:6,6,18,18; TX:20,20,80,80) --=-ZH2ePxxmHc69O0Q0uBpo Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (FreeBSD) iQEcBAABAgAGBQJR8EXVAAoJEBkJRdwI6BaHEK4H/2KziMI8yT+iFweng0fyLMdZ eD0UC+XnzeQ+eJb+kB9PWtCKrh/fUDNaXv7iIapOHURVD90hm8QGlAY+Js1fA2n8 5dMU8JTKJJQV2jr3oO+2fl4dKDLsCx1nP4eyoCga7Ck8uc+W6vTxHAPM1TAeoFnk 8urGDTgSyE1OO3fdzuAms+EZTyJ2uPsoKm5UzwXHklx3zYX3zPqxqjpaiuCqZjg6 QeqZY2MWWPhtNcXTgmkFIbqfmV5+zHa95uIWQpNA3Yu83sx5aCjZNbQU6D5PlZQ2 3PUNmrhzII1zNDuxy6WpRQrCoS/bIPaQFf2h5TOJ/Jt4DFJGdB85vskwH05lWz4= =YaKg -----END PGP SIGNATURE----- --=-ZH2ePxxmHc69O0Q0uBpo-- From owner-freebsd-net@FreeBSD.ORG Wed Jul 24 21:29:45 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 755E3FAA; Wed, 24 Jul 2013 21:29:45 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ea0-x235.google.com (mail-ea0-x235.google.com [IPv6:2a00:1450:4013:c01::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D8AEB2CF2; Wed, 24 Jul 2013 21:29:44 +0000 (UTC) Received: by mail-ea0-f181.google.com with SMTP id a15so520511eae.12 for ; Wed, 24 Jul 2013 14:29:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=TjNS5sXuELYS10R5qghJhAYGAKjWNPR4wdYTHb3gBhU=; b=nQAU+GoKBJBoMnOXLPKlAMKxu1Kta0Kaoa4i4i+2fRmknwtCsYvfVVLXuf+GpFuH+Z kgXjfzf9tPhT2x3+pcbusq1HjodXtjvdRS78FbW3sUnUFPiJqRMLjspLi2mPMdoklgce qdJrk/AW0eM2/uoHyLZQZuiwc01tvJ54Q1CL0VLemAHEmLlMZmdxpz47G0qeVJ0PNLva oZgHA4zQQ4RtpF15Ea9nwHp1U7Ht+z9RCM0IJ8XrjMmvaCdD/HpymkZhiTPOGcOkYz+e h9K9QI4SJTu5UF3Rpv4Uo2C/bE8gYijMXhy+vq9okpAXRC6KT7tSbLRBxRi62dgVZW2L UPDQ== MIME-Version: 1.0 X-Received: by 10.14.69.206 with SMTP id n54mr38815065eed.154.1374701383114; Wed, 24 Jul 2013 14:29:43 -0700 (PDT) Sender: hiren.panchasara@gmail.com Received: by 10.14.105.137 with HTTP; Wed, 24 Jul 2013 14:29:43 -0700 (PDT) In-Reply-To: <1374701022.1964.0.camel@localhost> References: <1374700042.1493.14.camel@localhost> <1374701022.1964.0.camel@localhost> Date: Wed, 24 Jul 2013 14:29:43 -0700 X-Google-Sender-Auth: IPGxHYmuH2aVRmkeeJnhGI_0fD4 Message-ID: Subject: Re: bce(4) panics, 9.2rc1 From: hiren panchasara To: sbruno@freebsd.org Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jul 2013 21:29:45 -0000 On Wed, Jul 24, 2013 at 2:23 PM, Sean Bruno wrote: > On Wed, 2013-07-24 at 14:07 -0700, Sean Bruno wrote: >> Running 9.2 in production load mail servers. We're hitting the >> "watchdog" message and crashing with the stable/9 version. We're >> reverting the change from 2 weeks ago and seeing if it still happens. >> We didn't see this from stable/9 from about a month ago. >> pciconf -lvb: http://people.freebsd.org/~hiren/pciconf.txt Thanks, Hiren From owner-freebsd-net@FreeBSD.ORG Wed Jul 24 21:49:35 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 897F7668; Wed, 24 Jul 2013 21:49:35 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5DCE62E19; Wed, 24 Jul 2013 21:49:35 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6OLnZ6v021603; Wed, 24 Jul 2013 21:49:35 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6OLnZj4021602; Wed, 24 Jul 2013 21:49:35 GMT (envelope-from linimon) Date: Wed, 24 Jul 2013 21:49:35 GMT Message-Id: <201307242149.r6OLnZj4021602@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-amd64@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/180791: [ofed] [patch] Kernel crash on ifdown and kldunload mlxen X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jul 2013 21:49:35 -0000 Old Synopsis: Kernel crash on ifdown and kldunload mlxen New Synopsis: [ofed] [patch] Kernel crash on ifdown and kldunload mlxen Responsible-Changed-From-To: freebsd-amd64->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Wed Jul 24 21:48:47 UTC 2013 Responsible-Changed-Why: relcassify. http://www.freebsd.org/cgi/query-pr.cgi?pr=180791 From owner-freebsd-net@FreeBSD.ORG Wed Jul 24 22:35:53 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C7421615; Wed, 24 Jul 2013 22:35:53 +0000 (UTC) (envelope-from prvs=1917699425=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 462CC2FCE; Wed, 24 Jul 2013 22:35:52 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50005139356.msg; Wed, 24 Jul 2013 23:35:48 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 24 Jul 2013 23:35:48 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=1917699425=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <23BE0BA3B9BF436BA4EC2628914723F3@multiplay.co.uk> From: "Steven Hartland" To: , References: <1374700042.1493.14.camel@localhost> Subject: Re: bce(4) panics, 9.2rc1 Date: Wed, 24 Jul 2013 23:36:19 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jul 2013 22:35:53 -0000 ----- Original Message ----- From: "Sean Bruno" As a guess its likely the interrupt handler is triggering while the watchdog timeout handler is re-initialising the card so you inconsitent state resulting in the crash. In from /var/crash should help determine the cause and confirm / deny that. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-net@FreeBSD.ORG Thu Jul 25 06:54:53 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 745D6A3F; Thu, 25 Jul 2013 06:54:53 +0000 (UTC) (envelope-from olivier2553@gmail.com) Received: from mail-qa0-x234.google.com (mail-qa0-x234.google.com [IPv6:2607:f8b0:400d:c00::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 250002089; Thu, 25 Jul 2013 06:54:52 +0000 (UTC) Received: by mail-qa0-f52.google.com with SMTP id bq6so402944qab.4 for ; Wed, 24 Jul 2013 23:54:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=SjqXS2Gtw7bYQGdJbeN3Hasiff+i2FlD5UbF48XuSGk=; b=J5dyCOfoh4mGtgAK1A8rA1A2sxjH2gqWl0CoNvrn+4AV2wCdWKbrhaJnBTGMeuSU9M TX7PY4YYZHPvZ+Zq1lY9RVxte7Np9gza27VEJnZUCiyHFFlYlhBNVEboN2iYykzHPsYn 8HB16+Lp1BT0WiA1+30AKxNoflYXHX7t1XAa0HITbG6mqR1JWyK2dZ++FFfWMR2uuzKy 5vG/MXU3u1N4KZ5QUrgVTtxBa3ofQp52oTIgq7YncdnN+Gs8prwKUo5jxJZyiiryUnee 6J5JrkFKdw0MbQn/wTGedi8+XovI2M9cTvl9Rc5jlUkXTXGheHFwo7NMkwZ763v+TZCX mjhA== MIME-Version: 1.0 X-Received: by 10.49.132.41 with SMTP id or9mr47224974qeb.18.1374735292256; Wed, 24 Jul 2013 23:54:52 -0700 (PDT) Sender: olivier2553@gmail.com Received: by 10.49.0.235 with HTTP; Wed, 24 Jul 2013 23:54:52 -0700 (PDT) In-Reply-To: References: Date: Thu, 25 Jul 2013 13:54:52 +0700 X-Google-Sender-Auth: 4W3gw6_RjX5QntGRsHoivPCv5sc Message-ID: Subject: Re: Same MAC address in 2 different VLANs From: Olivier Nicole To: krad Content-Type: text/plain; charset=ISO-8859-1 Cc: Olivier Nicole , freebsd-net@freebsd.org, "freebsd-questions@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 06:54:53 -0000 > I think you maybe ok. Ive just looked at my esx config and the esx > management interfaces use their own generated macs, not the physical > interfaces ones. All the vms obviously use generated macs as well. > > However I only looked over it at a superficial level. > > Have you considered using a tap or spare phyical interface on your flex box > and not linking it to the network? Thank you, that was a brilliant idea: Flex only needs that one interface, with the specific MAC, exists on the host, it does not specifically try to use that interface for managing licenses, so a tap hanging to nowhere is the solution. Best regards, Olivier > > > On 19 July 2013 10:29, Olivier Nicole wrote: > >> Hello, >> >> Could any one comment about the use of the same MAC address in 2 >> separate VLANs? >> >> All my machines are connected to 2 VLANs (one public and one private) >> with no routing in between the VLANs. >> >> I used to run a FLEX license manager to a physical machine. When I >> virtualized that service, I had to use the MAC address of that physical >> machine for the virtual machine (FLEX is linked to the MAc address and I >> coul dnot issue new license as licensed the pproduct is not supported >> anymore). The virtual NIC that has the old MAC address is connected to >> the public VLAN. >> >> Now I want to reuse the physical machine as a VMware server. Dell nor >> VMware offer a solution to change the MAC address (like >> ifconfig em0 link xx:xx:xx:xx:xx:xx would do). So I plan to connect the >> NIC with the incriminated MAC to the private VLAN. >> >> Most (if not all) my servers are FreeBSD. Most will access the virtual >> machine running FLEX and may access the VMware server also. The servers >> are not VLAN aware. >> >> Will this be an issue? >> >> Best regars, >> >> Olivier >> >> -- >> _______________________________________________ >> freebsd-questions@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-questions >> To unsubscribe, send any mail to " >> freebsd-questions-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Thu Jul 25 10:46:29 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 1C63A54A for ; Thu, 25 Jul 2013 10:46:29 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id D4A2C2FB0 for ; Thu, 25 Jul 2013 10:46:28 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:ad72:2c3d:f785:a64c]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPSA id 8CEC54AC57 for ; Thu, 25 Jul 2013 14:46:27 +0400 (MSK) Date: Thu, 25 Jul 2013 14:46:22 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <932442845.20130725144622@serebryakov.spb.ru> To: "freebsd-net@freebsd.org" Subject: A huge amount of "sonewconn: pcb 0xfffffe0053916dc8: Listen queue overflow: 193 already in queue awaiting acceptance" in logs recently (9-STABLE) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 10:46:29 -0000 Hello, Freebsd-net. I have 9.1-STABLE r253105 system, which started to flood logs with "sonewconn: pcb 0xfffffe0053916dc8: Listen queue overflow: 193 already in queue awaiting acceptance" messages (there are thousnds of it, if you take "last message was repeated 600 times" in account). Nothing was changed in settings for long time. How could I determine, which connections (listen port, at least) cause these messages? -- // Black Lion AKA Lev Serebryakov From owner-freebsd-net@FreeBSD.ORG Thu Jul 25 11:24:43 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id F16411D7; Thu, 25 Jul 2013 11:24:43 +0000 (UTC) (envelope-from flo@smeets.im) Received: from mail.solomo.de (mail.solomo.de [5.9.87.18]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A95952281; Thu, 25 Jul 2013 11:24:43 +0000 (UTC) Received: from cpos1.nexxtmobile.de (localhost [127.0.0.1]) by mail.solomo.de (Postfix) with ESMTP id AF94B1D015; Thu, 25 Jul 2013 13:24:35 +0200 (CEST) X-Virus-Scanned: amavisd-new at nexxtmobile.de Received: from mail.solomo.de ([127.0.0.1]) by cpos1.nexxtmobile.de (cpos1.nexxtmobile.de [127.0.0.1]) (amavisd-new, port 10024) with LMTP id bKghyQPjBd1I; Thu, 25 Jul 2013 13:24:34 +0200 (CEST) Received: from nibbler-osx.local (85-22-131-80.ip.dokom21.de [85.22.131.80]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.solomo.de (Postfix) with ESMTPSA id 9A3B61E000; Thu, 25 Jul 2013 13:24:33 +0200 (CEST) Message-ID: <51F10AED.4060407@smeets.im> Date: Thu, 25 Jul 2013 13:24:29 +0200 From: Florian Smeets User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Thunderbird/25.0a1 MIME-Version: 1.0 To: lev@FreeBSD.org, "freebsd-net@freebsd.org" Subject: Re: A huge amount of "sonewconn: pcb 0xfffffe0053916dc8: Listen queue overflow: 193 already in queue awaiting acceptance" in logs recently (9-STABLE) References: <932442845.20130725144622@serebryakov.spb.ru> In-Reply-To: <932442845.20130725144622@serebryakov.spb.ru> X-Enigmail-Version: 1.6a1pre Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ACCRvDdr46oMbLaUpSiIOB7TovxSttFxm" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 11:24:44 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --ACCRvDdr46oMbLaUpSiIOB7TovxSttFxm Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 25.07.13 12:46, Lev Serebryakov wrote: > Hello, Freebsd-net. >=20 > I have 9.1-STABLE r253105 system, which started to flood logs with > "sonewconn: pcb 0xfffffe0053916dc8: Listen queue overflow: 193 already= in > queue awaiting acceptance" messages (there are thousnds of it, if you t= ake > "last message was repeated 600 times" in account). >=20 > Nothing was changed in settings for long time. >=20 > How could I determine, which connections (listen port, at least) cause= > these messages? >=20 netstat -A more details are in the commit log of r242306. MFC to stable/9 was r252782. Florian --ACCRvDdr46oMbLaUpSiIOB7TovxSttFxm Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAlHxCu8ACgkQapo8P8lCvwmDewCgoGy9wHXSqplLwLymRupOapkk TBEAoK7Pco8aurtv7mOkLq7y4i+OgteS =vr+c -----END PGP SIGNATURE----- --ACCRvDdr46oMbLaUpSiIOB7TovxSttFxm-- From owner-freebsd-net@FreeBSD.ORG Thu Jul 25 12:51:31 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 3F185E4D for ; Thu, 25 Jul 2013 12:51:31 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A60902708 for ; Thu, 25 Jul 2013 12:51:30 +0000 (UTC) Received: (qmail 54684 invoked from network); 25 Jul 2013 13:39:30 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 25 Jul 2013 13:39:30 -0000 Message-ID: <51F11F47.2000008@freebsd.org> Date: Thu, 25 Jul 2013 14:51:19 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: lev@FreeBSD.org Subject: Re: A huge amount of "sonewconn: pcb 0xfffffe0053916dc8: Listen queue overflow: 193 already in queue awaiting acceptance" in logs recently (9-STABLE) References: <932442845.20130725144622@serebryakov.spb.ru> In-Reply-To: <932442845.20130725144622@serebryakov.spb.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 12:51:31 -0000 On 25.07.2013 12:46, Lev Serebryakov wrote: > Hello, Freebsd-net. > > I have 9.1-STABLE r253105 system, which started to flood logs with > "sonewconn: pcb 0xfffffe0053916dc8: Listen queue overflow: 193 already in > queue awaiting acceptance" messages (there are thousnds of it, if you take > "last message was repeated 600 times" in account). > > Nothing was changed in settings for long time. > > How could I determine, which connections (listen port, at least) cause > these messages? This means the rate of incoming connection attempts is grater than the speed of the application accepting them. Typically you either suffer from a DoS attack or your server is undersized for the amount of traffic it is receiving. A change to rate-limit the number of these messages is in the works to prevent it filling from the logs too fast. -- Andre From owner-freebsd-net@FreeBSD.ORG Thu Jul 25 12:55:23 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6C0BDF64 for ; Thu, 25 Jul 2013 12:55:23 +0000 (UTC) (envelope-from peterxu@cyphy.net) Received: from atl4mhob04.myregisteredsite.com (atl4mhob04.myregisteredsite.com [209.17.115.42]) by mx1.freebsd.org (Postfix) with ESMTP id 2035B274A for ; Thu, 25 Jul 2013 12:55:22 +0000 (UTC) Received: from mailpod.hostingplatform.com ([10.30.71.204]) by atl4mhob04.myregisteredsite.com (8.14.4/8.14.4) with ESMTP id r6PCtGDa026619 for ; Thu, 25 Jul 2013 08:55:16 -0400 Received: (qmail 17598 invoked by uid 0); 25 Jul 2013 12:55:16 -0000 X-TCPREMOTEIP: 27.154.58.234 X-Authenticated-UID: peterxu@cyphy.net Received: from unknown (HELO Peters-MacAir.local) (peterxu@cyphy.net@27.154.58.234) by 0 with ESMTPA; 25 Jul 2013 12:55:15 -0000 Message-ID: <51F1202C.2050406@cyphy.net> Date: Thu, 25 Jul 2013 20:55:08 +0800 From: Xu Zhe User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: freebsd-questions@FreeBSD.org Subject: How to create vlan (four NIC into one) using lagg Content-Type: text/plain; charset=GB2312 Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 12:55:23 -0000 Hi, all, I am trying to use lagg to bind four 1Gb NIC into 4Gb one. I was testing this using two machines running FreeBSD 8.2, each of the machine has four 1Gb ethernet card, and connected correspondingly, means: MACHINE1 MACHINE2 em0 <--------------------->em0 em1 <--------------------->em1 em2 <--------------------->em2 em3 <--------------------->em3 Then I created vlan called 'lagg0' on each machine using: ifconfig lagg0 create ifconfig lagg0 laggproto lacp laggport em0 laggport em1 laggport em2 laggport em3 ifconfig lagg0 1.1.1.1/24 ifconfig lagg0 up And do this on MACH2 too, only change IP from 1.1.1.1 to 1.1.1.2. But I cannot ping each other, since none of the link is both active: MACHINE1 # ifconfig lagg0 lagg0: flags=8843 metric 0 mtu 1500 options=219b ether 00:08:9b:d4:91:64 inet 1.1.1.1 netmask 0xffffff00 broadcast 1.1.1.255 media: Ethernet autoselect status: active laggproto lacp laggport: em3 flags=1c laggport: em2 flags=18 laggport: em1 flags=18 laggport: em0 flags=18 MACHINE2 # ifconfig lagg0 lagg0: flags=8843 metric 0 mtu 1500 options=219b ether 00:08:9b:d3:72:60 inet 1.1.1.2 netmask 0xffffff00 broadcast 1.1.1.255 media: Ethernet autoselect status: active laggproto lacp laggport: em3 flags=18 laggport: em2 flags=1c laggport: em1 flags=1c laggport: em0 flags=1c So, em3 is active on MACHINE1 but not active on MACH2, while em0-em2 are active on MACH2 but not on MACHI1. What might be the problem? Thanks! Peter From owner-freebsd-net@FreeBSD.ORG Thu Jul 25 14:05:59 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id BA76FE76 for ; Thu, 25 Jul 2013 14:05:59 +0000 (UTC) (envelope-from laurie_jennings_1977@yahoo.com) Received: from nm38-vm0.bullet.mail.ne1.yahoo.com (nm38-vm0.bullet.mail.ne1.yahoo.com [98.138.229.144]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 624932A93 for ; Thu, 25 Jul 2013 14:05:58 +0000 (UTC) Received: from [98.138.90.48] by nm38.bullet.mail.ne1.yahoo.com with NNFMP; 25 Jul 2013 14:03:26 -0000 Received: from [98.138.87.1] by tm1.bullet.mail.ne1.yahoo.com with NNFMP; 25 Jul 2013 14:03:26 -0000 Received: from [127.0.0.1] by omp1001.mail.ne1.yahoo.com with NNFMP; 25 Jul 2013 14:03:26 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 685471.12924.bm@omp1001.mail.ne1.yahoo.com Received: (qmail 2837 invoked by uid 60001); 25 Jul 2013 14:03:26 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1374761006; bh=Kxw8+Lw7bwQnJfMEOggGVXcbKVEI36Jfi+g+FSW0Q7g=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:MIME-Version:Content-Type; b=eRIWagZmFe9qJeEM/F3oujoqKcAXzbNdg6TB2Ay4es0l5aE71YZaWlyA9X3eEzpZlhTVmKVnCRtOrLOT3m4KcKdi/JUgFdrhrYy5wmR3llsjNhtCP4BmO60SLeiAmMN4Kz9qsVJYB4FaSUxmmrFrZsCcpLGqyH2kTVeFozQIKGU= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:MIME-Version:Content-Type; b=S8NomuEJXoKny1fUNSlQD4I1UyPhtMQxxK6Vh2OcWu/ZQJjueruMtGMRAMFP7QBQmAMf+Q59nOu6f0BHORMJgph5uYzqOMfgVCsvzma4MAzudEgLiI6kY8NGedyidcFX9f5KSETSBpMOMMyUoiCsRLVOw2UYBiOzQDWqnPPl0y8=; X-YMail-OSG: wq75WxAVM1lmMju0Ba6Cp9tXP8zvY0HdFXDiFhtTKRIyJYL Hy0TCn.22rsSlAAxnvvIUxA04wL6_WHqIkSWob1sBFDBAc.bY2eEW7qezDHh FLfawgyUCpsjLusB6_8lS.mw28UnsdOg78PfDhs6arPGt3sYP9xku7Dak5xU ztk6kABvJgmD7RDvL1enzSAy2q_lFoJBLn8I0mCji1IazTV1uvewHkiaSRRk mvCcLrMOC_zjA2Wxb7aFaIu9Gw5j7AINkOdDN7HdRNiBzCFPwvHYvFoLR1jr gYF7BQvK9NjgD7uoN_st26edmjXolpcHTvP4bok2hQckM1_8iC_T24KQi6Kj krj_2RbOyEBxXIpTyXhsqe8HPCkhMJjkAIeLYEWgjeXcDnhv9rIRKlpLZ4Rh b_D4S0xGrtMH2zao2j6zju9ss6NVZYoLzLqwHYm2u1H36f4uz_mcyG_2XMpy sBI0sisEhQp_4PCDaAvO4k0uqrzknI__Z1gRI5RvC68Iy49ILCJ3jabnHVnT Iv5jjBkbo Received: from [98.203.118.124] by web125804.mail.ne1.yahoo.com via HTTP; Thu, 25 Jul 2013 07:03:26 PDT X-Rocket-MIMEInfo: 002.001, SW0gdHJ5aW5nIHRvIGdldCBhIHVzZXIgc3BhY2UgcHRocmVhZCB0byBtb25pdG9yIHNvbWUga2VybmVsIGJ1ZmZlcnMgYnV0IEkgY2FuJ3QgZ2V0IGl0IHRvIHJ1biBhdA0KYW55IHNvcnQgb3IgcmVsaWFibGUgcHJpb3JpdHkuIFRoZSBkb2NzIG9uIHJ0cHJpbyBhcmUgcHJldHR5IHdlYWsuIEkndmUgdHJpZWQgdG8gY2hhbmdlIHRoZSBydHByaW8gb2YgDQp0aGUgcHJvY2VzcywgYnV0IGl0IGp1c3QgbG9ja3MgdXAgdGhlIHN5c3RlbS4gSSBjYW4ndCBmaWd1cmUgaXQgb3V0Lg0KDQp3aGF0IGFyZSB0aGUgd2EBMAEBAQE- X-Mailer: YahooMailClassic/280 YahooMailWebService/0.8.150.561 Message-ID: <1374761006.92547.YahooMailBasic@web125804.mail.ne1.yahoo.com> Date: Thu, 25 Jul 2013 07:03:26 -0700 (PDT) From: Laurie Jennings Subject: rtprio threads To: freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 14:05:59 -0000 Im trying to get a user space pthread to monitor some kernel buffers but I can't get it to run at any sort or reliable priority. The docs on rtprio are pretty weak. I've tried to change the rtprio of the process, but it just locks up the system. I can't figure it out. what are the ways to give a user process the top users pace priority? renice doesn't seem to help much. Thanks! Laurie From owner-freebsd-net@FreeBSD.ORG Thu Jul 25 16:34:58 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C0D403F4; Thu, 25 Jul 2013 16:34:58 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 93CC622C9; Thu, 25 Jul 2013 16:34:58 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6PGYwaM069591; Thu, 25 Jul 2013 16:34:58 GMT (envelope-from jhb@freefall.freebsd.org) Received: (from jhb@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6PGYwQN069590; Thu, 25 Jul 2013 16:34:58 GMT (envelope-from jhb) Date: Thu, 25 Jul 2013 16:34:58 GMT Message-Id: <201307251634.r6PGYwQN069590@freefall.freebsd.org> To: menyy@mellanox.com, jhb@FreeBSD.org, freebsd-net@FreeBSD.org, jhb@FreeBSD.org From: jhb@FreeBSD.org Subject: Re: kern/180430: [ofed] [patch] Bad UDP checksum calc for fragmented packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 16:34:58 -0000 Synopsis: [ofed] [patch] Bad UDP checksum calc for fragmented packets State-Changed-From-To: open->patched State-Changed-By: jhb State-Changed-When: Thu Jul 25 16:34:38 UTC 2013 State-Changed-Why: Fix committed to HEAD, thanks! Responsible-Changed-From-To: freebsd-net->jhb Responsible-Changed-By: jhb Responsible-Changed-When: Thu Jul 25 16:34:38 UTC 2013 Responsible-Changed-Why: Fix committed to HEAD, thanks! http://www.freebsd.org/cgi/query-pr.cgi?pr=180430 From owner-freebsd-net@FreeBSD.ORG Thu Jul 25 17:47:31 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 8473D8A9 for ; Thu, 25 Jul 2013 17:47:31 +0000 (UTC) (envelope-from sean_bruno@yahoo.com) Received: from nm32-vm7.bullet.mail.ne1.yahoo.com (nm32-vm7.bullet.mail.ne1.yahoo.com [98.138.229.55]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3277B2633 for ; Thu, 25 Jul 2013 17:47:30 +0000 (UTC) Received: from [98.138.226.180] by nm32.bullet.mail.ne1.yahoo.com with NNFMP; 25 Jul 2013 17:47:24 -0000 Received: from [98.138.84.39] by tm15.bullet.mail.ne1.yahoo.com with NNFMP; 25 Jul 2013 17:47:24 -0000 Received: from [127.0.0.1] by smtp107.mail.ne1.yahoo.com with NNFMP; 25 Jul 2013 17:47:24 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1374774444; bh=hVDDH8NtBsQxI3Y+8jWTfgweK+poCNQBWxwtgT+CX64=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:Subject:From:Reply-To:To:Cc:In-Reply-To:References:Content-Type:Date:Message-ID:Mime-Version:X-Mailer; b=bpPVR8GoqN0TfnxZ3+V2M/ll74aV4Ougy2RcgJKio3JWF5ZyJRSA2Rphv9obN2EvuErAeWF+oPVGG8o+KWU4JzBUGtsE5tqFyQzqikaDAUzVssaTmZeb9d4bvtKon9Q4xmKhroJsK1/+n/jEJCxGxxyLCvz7/cXgdAvwBMke8ng= X-Yahoo-Newman-Id: 152954.82246.bm@smtp107.mail.ne1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: 5wn1VTgVM1kqTBRwEmGTT_1Hh.hiU0rSsGO7R1JS4qiQVpi bfnkmPc6b3VQNcjqbKHF9upSX_7PrIPChqkj5whAICmkgWjBEJfcMVGSaX6l Io3gcpRLp.gDpmYvSQXQBYScx4SKwopqJa9sSqfSOGD5PJSkN8vhx4q4E_qf DJivZvHR5fvwHK9NceOrw5ZmjyqD35R4tGXp6hlUtrwHuIHyottBf0WgWhkq PLoJAxUOgF0m8xhTA2NJWTLdwVzS3DPcCiWGofWPtkwRpp1096u5mJhKB4eT HalGov.ZwapU6rT.7Zd.t1OSznnb9EH3JF2YRDUgU9dC8zbkVL1yhbvg_dfE 7lct3uA9UlrsQ._m32xdwQKI3hdUDTvyjjZgMFr1gdVOBLrRz_f91hnI.P7c sOPmpXpEkyAcpsVfGRWX026BuSjGaKWiFTqcyKMCoVStJRy6mFN29ULJcOyl EAdJSEH8WxhE6HjNCek_jSSJMsJIB6Zrmxi0WK3jzqJwfqqlLDLc_Ii4Wi7C l4F9lSsS19DYyRqqoaTLyniID3aew2gbSq0XDBpdoA0V.4Kr1SKfovCdU7_Y Wj8HLTYlgS0EgYlliHq2C0.tINs2ZAG8v1QLcxgpHBQ4eXiMM84E2dt8Z62W N2kb8dYaoW5O_hQ-- X-Yahoo-SMTP: u5BKR6OswBC_iZJVfGRoMkTIpc8pEA4- X-Rocket-Received: from [192.168.43.121] (sean_bruno@70.197.7.255 with ) by smtp107.mail.ne1.yahoo.com with SMTP; 25 Jul 2013 10:47:24 -0700 PDT Subject: Re: bce(4) panics, 9.2rc1 From: Sean Bruno To: sbruno@freebsd.org In-Reply-To: <1374701022.1964.0.camel@localhost> References: <1374700042.1493.14.camel@localhost> <1374701022.1964.0.camel@localhost> Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-HWrkwmn8stVuRXHIJuNV" Date: Thu, 25 Jul 2013 10:47:21 -0700 Message-ID: <1374774441.1438.12.camel@localhost> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: sbruno@freebsd.org List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 17:47:31 -0000 --=-HWrkwmn8stVuRXHIJuNV Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable On Wed, 2013-07-24 at 14:23 -0700, Sean Bruno wrote: > On Wed, 2013-07-24 at 14:07 -0700, Sean Bruno wrote: > > Running 9.2 in production load mail servers. We're hitting the > > "watchdog" message and crashing with the stable/9 version. We're > > reverting the change from 2 weeks ago and seeing if it still happens. > > We didn't see this from stable/9 from about a month ago. > >=20 > >=20 > > Sean > >=20 > > ref:=20 > > http://svnweb.freebsd.org/base?view=3Drevision&revision=3D253128 >=20 > These panics are happening on: >=20 > bce0: mem > 0xda000000-0xdbffffff irq 36 at device 0.0 on pci1 > miibus0: on bce0 > brgphy0: PHY 1 on miibus0 > brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > bce0: Ethernet address: d4:ae:52:8d:42:fc > bce0: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); > Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.11) > Coal (RX:6,6,18,18; TX:20,20,80,80) > bce1: mem > 0xdc000000-0xddffffff irq 48 at device 0.1 on pci1 > miibus1: on bce1 > brgphy1: PHY 1 on miibus1 > brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > bce1: Ethernet address: d4:ae:52:8d:42:fd > bce1: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); > Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.11) > Coal (RX:6,6,18,18; TX:20,20,80,80) >=20 These machines are using IPMI for management (Dell R410) and seem to be unable to attach to /dev/ipmi0: ipmi0: port 0xca8,0xcac on acpi0 ipmi0: KCS mode found at io 0xca8 on acpi .... ipmi1: on isa0 device_attach: ipmi1 attach returned 16 ipmi1: on isa0 device_attach: ipmi1 attach returned 16 ... ipmi0: Timed out waiting for GET_DEVICE_ID Sean --=-HWrkwmn8stVuRXHIJuNV Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (FreeBSD) iQEcBAABAgAGBQJR8WSoAAoJEBkJRdwI6BaHeMMH/REfTEQG2nbVk9iqZ/xgi11T LrRYle6N0LUuP1NQwgP55r2+9wmM8g36T342YbXn/pKG+IOMZ9FwD2r6pI5kFmis HINStYr+xNlFM1GCPbAMca1SdO5s2ahmXIjbukJXJZ8aIHrlRUiTZLd3LpvuTAJu eLrxfHoCCJRf7gGGCNASjh/zLIWb/Xgmrh7B3PgoYOi7Q18gTJ5mPCzKgQEqbhXM H47bUXlrhC7fY5+Pks5Nt7e3Fz4zYSyGfBGXp4UtYiZVUVVPSOS0EwU09CNgh0+p 8h/OclCXQsJOhvQrNhriaW2J9g2ZNnG7JfT0SAN7ZVES43wJIcNcz6GqJ25JBVU= =AEY4 -----END PGP SIGNATURE----- --=-HWrkwmn8stVuRXHIJuNV-- From owner-freebsd-net@FreeBSD.ORG Thu Jul 25 18:10:53 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4A697EF for ; Thu, 25 Jul 2013 18:10:53 +0000 (UTC) (envelope-from melifaro@FreeBSD.org) Received: from mail.ipfw.ru (unknown [IPv6:2a01:4f8:120:6141::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0E7A027BA for ; Thu, 25 Jul 2013 18:10:53 +0000 (UTC) Received: from v6.mpls.in ([2a02:978:2::5] helo=ws.su29.net) by mail.ipfw.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.76 (FreeBSD)) (envelope-from ) id 1V2Q37-000Kc4-Mj; Thu, 25 Jul 2013 22:14:01 +0400 Message-ID: <51F16A07.9030505@FreeBSD.org> Date: Thu, 25 Jul 2013 22:10:15 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20120121 Thunderbird/9.0 MIME-Version: 1.0 To: Boris Kochergin Subject: Re: Recommendations for 10gbps NIC References: <51F0386D.2000709@acm.poly.edu> In-Reply-To: <51F0386D.2000709@acm.poly.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 18:10:53 -0000 On 25.07.2013 00:26, Boris Kochergin wrote: > Hi. Hello. > > I am looking for recommendations for a 10gbps NIC from someone who has > successfully used it on FreeBSD. It will be used on FreeBSD 9.1-R/amd64 > to capture packets. Some desired features are: > > - PCIe > - LC connectors > - 10GBASE-SR > - Either single- or dual-port > - Multiqueue Intel 82598/99/X520 Emulex OCe10102-NM Mellanox ConnectX Chelsio T4 > > Specific part numbers would be appreciated. Thank you. > > -Boris > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Thu Jul 25 18:22:49 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 1BD47458; Thu, 25 Jul 2013 18:22:49 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-la0-x22d.google.com (mail-la0-x22d.google.com [IPv6:2a00:1450:4010:c03::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 666C6284C; Thu, 25 Jul 2013 18:22:48 +0000 (UTC) Received: by mail-la0-f45.google.com with SMTP id ev20so1640555lab.4 for ; Thu, 25 Jul 2013 11:22:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=54TXFqdUxXqrOPzm/xbrPT1xzJPKp/zNdm7zHiSy35s=; b=Wme3Zoo/7rcvviewoN51KhJRSNHNQ0zkC6LCKIdHvlt04ENxyBRh75OUESiYb/zCuN FG/jVgXdKiQvQPzeqjpVA3BDqR0t1SBkSFCLX3/CrHtXSReSeyqRuFP+qr3flNuonjsJ Y91et1TvXgW2YfqLnZzlMn2cuJgk13PEneOJCuFL4glNGuJEogvy51w6oDnB7CHOQFgz rNjlX1HLmIXMUxpNBDUHHt/0Vu7EZnu7Q2eShXsLmw5rFc0V/e77Q8vi1JjukaMOagrW G1mziBMKCFSv2szdgI/odv7lYOvQHFJnUGRI0351Tk2YbPQu3m80cwE0z1zxLL81lhF0 r2mw== MIME-Version: 1.0 X-Received: by 10.152.29.227 with SMTP id n3mr19607310lah.43.1374776566205; Thu, 25 Jul 2013 11:22:46 -0700 (PDT) Sender: rizzo.unipi@gmail.com Received: by 10.114.200.15 with HTTP; Thu, 25 Jul 2013 11:22:46 -0700 (PDT) In-Reply-To: <51F16A07.9030505@FreeBSD.org> References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> Date: Thu, 25 Jul 2013 20:22:46 +0200 X-Google-Sender-Auth: nTwQokoct89x8Xu6Z5vtWkwolkQ Message-ID: Subject: Re: Recommendations for 10gbps NIC From: Luigi Rizzo To: "Alexander V. Chernikov" Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org, Boris Kochergin X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 18:22:49 -0000 On Thu, Jul 25, 2013 at 8:10 PM, Alexander V. Chernikov wrote: > On 25.07.2013 00:26, Boris Kochergin wrote: >> >> Hi. > > Hello. >> >> >> I am looking for recommendations for a 10gbps NIC from someone who has >> successfully used it on FreeBSD. It will be used on FreeBSD 9.1-R/amd64 >> to capture packets. Some desired features are: >> if you are into high speed packet capture i'd really suggest the intel, is the only one that is supported by netmap and can give you true line rate cheers luigi >> - PCIe >> - LC connectors >> - 10GBASE-SR >> - Either single- or dual-port >> - Multiqueue > > Intel 82598/99/X520 > Emulex OCe10102-NM > Mellanox ConnectX > Chelsio T4 >> >> >> Specific part numbers would be appreciated. Thank you. >> >> -Boris >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2211611 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+------------------------------- From owner-freebsd-net@FreeBSD.ORG Thu Jul 25 18:23:59 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 61EBC501 for ; Thu, 25 Jul 2013 18:23:59 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: from nm12-vm7.bullet.mail.gq1.yahoo.com (nm12-vm7.bullet.mail.gq1.yahoo.com [98.136.218.206]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id F36A82860 for ; Thu, 25 Jul 2013 18:23:58 +0000 (UTC) Received: from [98.137.12.62] by nm12.bullet.mail.gq1.yahoo.com with NNFMP; 25 Jul 2013 18:17:29 -0000 Received: from [208.71.42.195] by tm7.bullet.mail.gq1.yahoo.com with NNFMP; 25 Jul 2013 18:17:29 -0000 Received: from [127.0.0.1] by smtp206.mail.gq1.yahoo.com with NNFMP; 25 Jul 2013 18:17:29 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1374776249; bh=u8SAmT1m81lonIGngvO9STI+vtHjJelMEMOo6mAaglM=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:References:Mime-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding:Message-Id:Cc:X-Mailer:From:Subject:Date:To; b=ZDUXJHuiwR/QkwQjog5cq0zvlp2+zxUUn6Wam565ZqPEqUjqyDDgtnBbiI7/ccbmEcGjhF9mjbq/BguKUvjCxH7PN+tWA9qpx0VMKf19pcuXqnVjQqqlnv2GG+uFVdjqfRTQYQQvEirMj4xsvUjkcwvKAltkfeEirZpn6FPBun4= X-Yahoo-Newman-Id: 845828.57665.bm@smtp206.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: xaabmisVM1kPaAgSNCX22ODSajya.6lwBqmGOnp1iWpEkv. _bDCdWOo6VdDPs5fZQRNCjsWyRp0uIhmMhXqCW9ShmprQt7eQ84a5ALjQy5y Ip93oPiBcyMSvQ1.bpoA9otyCYIgWZTIe_KTaEzc40QEc.bOeU4.acaD_wxa DGT0q02Gak2XbdPWd4ZCYK0BaP6I1nMrdw2lV6bCujZtT8fu1jI_lz_GS33O f8sqZS6opqSfrW_mu.2WxTAztILY5ivjKMHx.3dTUDKUH0_Zu.QYwN23i1hv tQVYEAmc39ZNF5s5QG35ky9R3T1dPxOKl9iVg.QnCzMvVPy.wK.w9g.oZOSD 1gNR0ZaV4U..SboM2R1wcXUrhJ1GWMbaWRIOuNYjnUBuSnVoXSXqhRh6nTKj V65O_UVOV8KMNsY2iBW78vzVsRNkTA9bZG_kyMVmk_ARwrhK0m_v7bHdZPPf ogaSFcCsradBUuI6UXBrck3PD89fI95FfDdt41OZaFORAZ_EveZnZOVnaEya T9LlwswVfZrX3BuvvTCnyPM37fMB6s2bqW0g- X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- X-Rocket-Received: from [10.191.224.211] (scott4long@70.209.18.182 with ) by smtp206.mail.gq1.yahoo.com with SMTP; 25 Jul 2013 11:17:29 -0700 PDT References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> Mime-Version: 1.0 (1.0) In-Reply-To: <51F16A07.9030505@FreeBSD.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: X-Mailer: iPhone Mail (10B350) From: Scott Long Subject: Re: Recommendations for 10gbps NIC Date: Thu, 25 Jul 2013 14:17:28 -0400 To: "Alexander V. Chernikov" Cc: "freebsd-net@freebsd.org" , Boris Kochergin X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 18:23:59 -0000 We use both the chelsio and intel offerings Scott On Jul 25, 2013, at 2:10 PM, "Alexander V. Chernikov" = wrote: > On 25.07.2013 00:26, Boris Kochergin wrote: >> Hi. > Hello. >>=20 >> I am looking for recommendations for a 10gbps NIC from someone who has >> successfully used it on FreeBSD. It will be used on FreeBSD 9.1-R/amd64 >> to capture packets. Some desired features are: >>=20 >> - PCIe >> - LC connectors >> - 10GBASE-SR >> - Either single- or dual-port >> - Multiqueue > Intel 82598/99/X520 > Emulex OCe10102-NM > Mellanox ConnectX > Chelsio T4 >>=20 >> Specific part numbers would be appreciated. Thank you. >>=20 >> -Boris >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >=20 > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Thu Jul 25 19:40:03 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 3843F96A for ; Thu, 25 Jul 2013 19:40:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 153F12C0B for ; Thu, 25 Jul 2013 19:40:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6PJe2FO005834 for ; Thu, 25 Jul 2013 19:40:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6PJe2pJ005833; Thu, 25 Jul 2013 19:40:02 GMT (envelope-from gnats) Date: Thu, 25 Jul 2013 19:40:02 GMT Message-Id: <201307251940.r6PJe2pJ005833@freefall.freebsd.org> To: freebsd-net@FreeBSD.org Cc: From: John Baldwin Subject: Re: kern/180791: [ofed] [patch] Kernel crash on ifdown and kldunload mlxen X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: John Baldwin List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 19:40:03 -0000 The following reply was made to PR kern/180791; it has been noted by GNATS. From: John Baldwin To: bug-followup@freebsd.org, shahark@mellanox.com Cc: Subject: Re: kern/180791: [ofed] [patch] Kernel crash on ifdown and kldunload mlxen Date: Thu, 25 Jul 2013 15:18:06 -0400 Thanks. One note is that it seems like the state_lock should be held when stop is called. Also, the callout used for stats should be drained during detach. (If you use callot_init_mtx() instead of callout_init() then callout_stop() under the lock is race-free, though I'd still do the callout_drain() during detach to be safe.) Also, I had some other changes I made to this file to make the locking more consistent with other NIC drivers in the tree. Can you look at this and test this patch please? Index: en_netdev.c =================================================================== --- en_netdev.c (revision 253547) +++ en_netdev.c (working copy) @@ -495,11 +495,6 @@ static void mlx4_en_do_get_stats(struct work_struc queue_delayed_work(mdev->workqueue, &priv->stats_task, STATS_DELAY); } - if (mdev->mac_removed[MLX4_MAX_PORTS + 1 - priv->port]) { - panic("mlx4_en_do_get_stats: Unexpected mac removed for %d\n", - priv->port); - mdev->mac_removed[MLX4_MAX_PORTS + 1 - priv->port] = 0; - } mutex_unlock(&mdev->state_lock); } @@ -688,8 +683,8 @@ int mlx4_en_start_port(struct net_device *dev) mlx4_en_set_multicast(dev); /* Enable the queues. */ - atomic_clear_int(&dev->if_drv_flags, IFF_DRV_OACTIVE); - atomic_set_int(&dev->if_drv_flags, IFF_DRV_RUNNING); + dev->if_drv_flags &= ~IFF_DRV_OACTIVE; + dev->if_drv_flags |= IFF_DRV_RUNNING; callout_reset(&priv->watchdog_timer, MLX4_EN_WATCHDOG_TIMEOUT, mlx4_en_watchdog_timeout, priv); @@ -761,7 +756,7 @@ void mlx4_en_stop_port(struct net_device *dev) callout_stop(&priv->watchdog_timer); - atomic_clear_int(&dev->if_drv_flags, IFF_DRV_RUNNING); + dev->if_drv_flags &= ~(IFF_DRV_RUNNING | IFF_DRV_OACTIVE); } static void mlx4_en_restart(struct work_struct *work) @@ -802,19 +797,30 @@ mlx4_en_init(void *arg) { struct mlx4_en_priv *priv; struct mlx4_en_dev *mdev; + + priv = arg; + mdev = priv->mdev; + mutex_lock(&mdev->state_lock); + mlx4_en_init_locked(priv); + mutex_unlock(&mdev->state_lock); +} + +static void +mlx4_en_init_locked(struct mlx4_en_priv *priv) +{ + + struct mlx4_en_dev *mdev; struct ifnet *dev; int i; - priv = arg; dev = priv->dev; mdev = priv->mdev; - mutex_lock(&mdev->state_lock); if (dev->if_drv_flags & IFF_DRV_RUNNING) mlx4_en_stop_port(dev); if (!mdev->device_up) { en_err(priv, "Cannot open - device down/disabled\n"); - goto out; + return; } /* Reset HW statistics and performance counters */ @@ -835,9 +841,6 @@ mlx4_en_init(void *arg) mlx4_en_set_default_moderation(priv); if (mlx4_en_start_port(dev)) en_err(priv, "Failed starting port:%d\n", priv->port); - -out: - mutex_unlock(&mdev->state_lock); } void mlx4_en_free_resources(struct mlx4_en_priv *priv) @@ -927,9 +930,14 @@ void mlx4_en_destroy_netdev(struct net_device *dev if (priv->sysctl) sysctl_ctx_free(&priv->conf_ctx); + mutex_lock(&mdev->state_lock); + mlx4_en_stop_port(dev); + mutex_unlock(&mdev->state_lock); + cancel_delayed_work(&priv->stats_task); /* flush any pending task for this netdev */ flush_workqueue(mdev->workqueue); + callout_drain(&priv->watchdog_timer); /* Detach the netdev so tasks would not attempt to access it */ mutex_lock(&mdev->state_lock); @@ -1091,25 +1099,25 @@ static int mlx4_en_ioctl(struct ifnet *dev, u_long error = -mlx4_en_change_mtu(dev, ifr->ifr_mtu); break; case SIOCSIFFLAGS: + mutex_lock(&mdev->state_lock); if (dev->if_flags & IFF_UP) { - if ((dev->if_drv_flags & IFF_DRV_RUNNING) == 0) { - mutex_lock(&mdev->state_lock); + if ((dev->if_drv_flags & IFF_DRV_RUNNING) == 0) mlx4_en_start_port(dev); - mutex_unlock(&mdev->state_lock); - } else + else mlx4_en_set_multicast(dev); } else { - mutex_lock(&mdev->state_lock); if (dev->if_drv_flags & IFF_DRV_RUNNING) { mlx4_en_stop_port(dev); if_link_state_change(dev, LINK_STATE_DOWN); } - mutex_unlock(&mdev->state_lock); } + mutex_unlock(&mdev->state_lock); break; case SIOCADDMULTI: case SIOCDELMULTI: + mutex_lock(&mdev->state_lock); mlx4_en_set_multicast(dev); + mutex_unlock(&mdev->state_lock); break; case SIOCSIFMEDIA: case SIOCGIFMEDIA: @@ -1116,6 +1124,7 @@ static int mlx4_en_ioctl(struct ifnet *dev, u_long error = ifmedia_ioctl(dev, ifr, &priv->media, command); break; case SIOCSIFCAP: + mutex_lock(&mdev->state_lock); mask = ifr->ifr_reqcap ^ dev->if_capenable; if (mask & IFCAP_HWCSUM) dev->if_capenable ^= IFCAP_HWCSUM; @@ -1130,7 +1139,8 @@ static int mlx4_en_ioctl(struct ifnet *dev, u_long if (mask & IFCAP_WOL_MAGIC) dev->if_capenable ^= IFCAP_WOL_MAGIC; if (dev->if_drv_flags & IFF_DRV_RUNNING) - mlx4_en_init(priv); + mlx4_en_init_locked(priv); + mutex_unlock(&mdev->state_lock); VLAN_CAPABILITIES(dev); break; default: -- John Baldwin From owner-freebsd-net@FreeBSD.ORG Thu Jul 25 21:26:47 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2ED799C8 for ; Thu, 25 Jul 2013 21:26:47 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-x229.google.com (mail-wi0-x229.google.com [IPv6:2a00:1450:400c:c05::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BC53A210C for ; Thu, 25 Jul 2013 21:26:46 +0000 (UTC) Received: by mail-wi0-f169.google.com with SMTP id c10so149220wiw.4 for ; Thu, 25 Jul 2013 14:26:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; bh=l6YPgvYTB/cr8B9yokKSjYciW1MwDUhiZFcvWdFbiys=; b=rI1Ko8rhkMueZOj5X19Md+NZTVE63dWta5eNDl3IYGWE4mFT/wzqieziO0bWFcp3jQ jJF0dyHqS4HF26G1eU0qN85JZGxhdaTLmxTwH6DIal0u0Cgt8bP70s1lvywQNAq1ZzEd O/QQO1Z5mNYyCkXD7E/WD4gNQOLXBmBYNGzfqWQ4hwky8a/V4ylSK3K0KjCT1KkTcl8e dthsQgSFITxttNu1PIWbUQW4Dd1VAi+H1kRNtop9Oa2DsxtUXiplGp4mjL64dPpFHGz4 XLDrbNqD5xoWtQkekbrH4x9RcQrxrDiUZg3Jh3STTRcq7ROm7ZgoXh7RPl+qi0tQ4YEv 2B5g== MIME-Version: 1.0 X-Received: by 10.194.92.6 with SMTP id ci6mr8086252wjb.79.1374787605031; Thu, 25 Jul 2013 14:26:45 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Thu, 25 Jul 2013 14:26:44 -0700 (PDT) Date: Thu, 25 Jul 2013 14:26:44 -0700 X-Google-Sender-Auth: ZlwQQ-DW0ofVBm5u4KZEb6GDyQw Message-ID: Subject: [rfc] lacp sysctl fixes From: Adrian Chadd To: FreeBSD Net Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 21:26:47 -0000 Hi, This patch introduces a few things: * there's now an lacp node, under net.link.lagg.{unit} * There's a strict mode parameter now * There's now a debug node * .. and rx_test/tx_test sit under that. http://people.freebsd.org/~adrian/netflix/20130725-lacp-debugging-1.diff It's a pre-requisite for shifting the LACP debug/trace calls into a per-lacp instance and extending things to be slightly more useful. Right now it's almost an all-or-nothing thing. Thanks! -adrian From owner-freebsd-net@FreeBSD.ORG Fri Jul 26 02:04:33 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 5721B41B; Fri, 26 Jul 2013 02:04:33 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2C06F2BFF; Fri, 26 Jul 2013 02:04:33 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6Q24XWr089692; Fri, 26 Jul 2013 02:04:33 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6Q24WVi089691; Fri, 26 Jul 2013 02:04:32 GMT (envelope-from linimon) Date: Fri, 26 Jul 2013 02:04:32 GMT Message-Id: <201307260204.r6Q24WVi089691@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/180844: [panic] [re] Intermittent panic (re driver?) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 02:04:33 -0000 Synopsis: [panic] [re] Intermittent panic (re driver?) Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Fri Jul 26 02:04:25 UTC 2013 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=180844 From owner-freebsd-net@FreeBSD.ORG Fri Jul 26 08:41:19 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id DFD3EA00; Fri, 26 Jul 2013 08:41:19 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: from mail-bk0-x235.google.com (mail-bk0-x235.google.com [IPv6:2a00:1450:4008:c01::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 307062C95; Fri, 26 Jul 2013 08:41:19 +0000 (UTC) Received: by mail-bk0-f53.google.com with SMTP id e11so1013064bkh.40 for ; Fri, 26 Jul 2013 01:41:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=PC2yBLD4sdxpXx3zpWvuw6oUQ/D95mDFZXguj+tUOhA=; b=EdKXSW287baKs12vg/zFSgUaNvdB9bc98pHOGNW7BPZ8CfyXzFX4awH0IUy5F29Rjm B7mmJYdSP1XKQWYkqbGOv2XjcRKiREKuaxl/B99hfI7JQYmllkjagAQBO/JCRfBsnJZt n+Fl2CB66z2xe97kwzw9hxdpreoFfV7rzrXr8zSQdbBPlIFfMmdMI2qR5px6NUV3FgHv tqbEuZeXy4FekPjj5RTKkxnaeGz3te9rrfjKbSxtLWjX/GkDJ0KwFrWExM1Nut1ag06a ET2HiLcJqyLm/Hki9CHIYpDFmVthMDfFskKsbJRzdQQ4Q4cy3a0oKtvE1bKLlzAskvAg PuWQ== MIME-Version: 1.0 X-Received: by 10.204.186.141 with SMTP id cs13mr7138105bkb.34.1374828077329; Fri, 26 Jul 2013 01:41:17 -0700 (PDT) Received: by 10.205.113.134 with HTTP; Fri, 26 Jul 2013 01:41:17 -0700 (PDT) In-Reply-To: <51F1202C.2050406@cyphy.net> References: <51F1202C.2050406@cyphy.net> Date: Fri, 26 Jul 2013 11:41:17 +0300 Message-ID: Subject: Re: How to create vlan (four NIC into one) using lagg From: Boris Astardzhiev To: Xu Zhe Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-net@freebsd.org, freebsd-questions@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 08:41:19 -0000 Hi Xu Zhe, If I were you I would first of all check cables. They might be the cause. Secondly, if cables are good, to me this report very much resembles a PR I reported a few weeks ago - http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/179926 Check its set and look at the patch I submitted. It's a pity there's no response to it. Greetings, Boris On Thu, Jul 25, 2013 at 3:55 PM, Xu Zhe wrote: > Hi, all, > > I am trying to use lagg to bind four 1Gb NIC into 4Gb one. I was testing > this using two machines running FreeBSD 8.2, each of the machine has > four 1Gb ethernet card, and connected correspondingly, means: > > MACHINE1 MACHINE2 > em0 <--------------------->em0 > em1 <--------------------->em1 > em2 <--------------------->em2 > em3 <--------------------->em3 > > Then I created vlan called 'lagg0' on each machine using: > > ifconfig lagg0 create > ifconfig lagg0 laggproto lacp laggport em0 laggport em1 laggport em2 > laggport em3 > ifconfig lagg0 1.1.1.1/24 > ifconfig lagg0 up > > And do this on MACH2 too, only change IP from 1.1.1.1 to 1.1.1.2. > > But I cannot ping each other, since none of the link is both active: > > MACHINE1 > # ifconfig lagg0 > lagg0: flags=8843 metric 0 mtu 1500 > > options=219b > ether 00:08:9b:d4:91:64 > inet 1.1.1.1 netmask 0xffffff00 broadcast 1.1.1.255 > media: Ethernet autoselect > status: active > laggproto lacp > laggport: em3 flags=1c > laggport: em2 flags=18 > laggport: em1 flags=18 > laggport: em0 flags=18 > > MACHINE2 > # ifconfig lagg0 > lagg0: flags=8843 metric 0 mtu 1500 > > options=219b > ether 00:08:9b:d3:72:60 > inet 1.1.1.2 netmask 0xffffff00 broadcast 1.1.1.255 > media: Ethernet autoselect > status: active > laggproto lacp > laggport: em3 flags=18 > laggport: em2 flags=1c > laggport: em1 flags=1c > laggport: em0 flags=1c > > So, em3 is active on MACHINE1 but not active on MACH2, while em0-em2 are > active on MACH2 but not on MACHI1. > > What might be the problem? > > Thanks! > Peter > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Fri Jul 26 13:23:08 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id A24E47C2 for ; Fri, 26 Jul 2013 13:23:08 +0000 (UTC) (envelope-from sean_bruno@yahoo.com) Received: from nm1-vm1.bullet.mail.bf1.yahoo.com (nm1-vm1.bullet.mail.bf1.yahoo.com [98.139.213.163]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 529BD2B4D for ; Fri, 26 Jul 2013 13:23:08 +0000 (UTC) Received: from [98.139.215.142] by nm1.bullet.mail.bf1.yahoo.com with NNFMP; 26 Jul 2013 13:23:01 -0000 Received: from [98.139.211.202] by tm13.bullet.mail.bf1.yahoo.com with NNFMP; 26 Jul 2013 13:23:01 -0000 Received: from [127.0.0.1] by smtp211.mail.bf1.yahoo.com with NNFMP; 26 Jul 2013 13:23:01 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1374844981; bh=I2IU7ojlz6ChBBwObLYi/kE7SO4iUJPWz7tDh4mLCDs=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:Subject:From:Reply-To:To:In-Reply-To:References:Content-Type:Date:Message-ID:Mime-Version:X-Mailer; b=Y55Nu49gSlU9/AoUhVUM77nf1O3R/VrOE3/v59sDwgwrlKjw7bgasomQjtOL5V5MevnSakmHuzt5JlyqW8kVkLtUP7BYKIL6l8orQeU/S8lFEz3BxYpXebUSDljrZZbwHJsEFRLuhX4ni1mKj34wf8khID11yMPOXdkljTpGaA0= X-Yahoo-Newman-Id: 80649.62018.bm@smtp211.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: BYhMEv0VM1lqHPdnTvaRJ6l2lZpyrZ2Kg7ZezIcb6s_1Xwk NqLrwvNqkkeVpeBBVvo5I3.abouiX44sJ1MuHoY0daTz4DlfFr.Zb_EclzY6 OgGGX813NkSQQq5WqXrIESX_NkcG46ztBze.VIRDP6s2Ly.qbYZQJyy6J.ey YALK_Ix6miknpsYsWiQ0cG37Q98zT6Rmnk.paC6E.8FKWhwvysS.yEfnj61B .V5jWC4_a9quHR.gAKnPR99hIMx8qMtkZ87kTHjVFXy1ytiiYngLV1GZUo6I AflYO7CHIt28y4Gk4v_QlrnxkPae_fAGtlBK1i4L0D_AFGc_rJIrTTkJ9vjU i5J0HRSUZE.jc1xzBYAvgWtvHb.Yhj95RF6qIHMCRMTxXl5mFQA.OB_WOSrj nqnAIlw5hLrv1xuq5ogFzd.wvr5Vnz2zK9sIsBywNDr.LscQE0tAEPQUm5q2 i.JZsvFTDw.Z0rQMw9rJkJrutWQ3S.DjNIlxmb9VQwgVYcA.3MhTTmGoz1Np IR_xiFVQrKlVzcv2hEo1NO2n6hOyda93KiR3A0W2gfyWN7BBVdIKfITcc6RD U2Avq2DVQozGX5ThHFKOS7XpxBl8OyeAcQvXhx0ibj4YZ2Z8DjEOmFkxicQ- - X-Yahoo-SMTP: u5BKR6OswBC_iZJVfGRoMkTIpc8pEA4- X-Rocket-Received: from [192.168.1.210] (sean_bruno@71.202.40.63 with ) by smtp211.mail.bf1.yahoo.com with SMTP; 26 Jul 2013 06:23:00 -0700 PDT Subject: Re: bce(4) panics, 9.2rc1, IPMI related? From: Sean Bruno To: "freebsd-net@freebsd.org" In-Reply-To: <1374774441.1438.12.camel@localhost> References: <1374700042.1493.14.camel@localhost> <1374701022.1964.0.camel@localhost> <1374774441.1438.12.camel@localhost> Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-II4MRHQ0nn4a6qVoRST0" Date: Fri, 26 Jul 2013 06:22:59 -0700 Message-ID: <1374844979.1477.14.camel@localhost> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: sbruno@freebsd.org List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 13:23:08 -0000 --=-II4MRHQ0nn4a6qVoRST0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable > >=20 > > bce0: mem > > 0xda000000-0xdbffffff irq 36 at device 0.0 on pci1 > > miibus0: on bce0 > > brgphy0: PHY 1 on miibus0 > > brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > > 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > > bce0: Ethernet address: d4:ae:52:8d:42:fc > > bce0: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); > > Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.11) > > Coal (RX:6,6,18,18; TX:20,20,80,80) > > bce1: mem > > 0xdc000000-0xddffffff irq 48 at device 0.1 on pci1 > > miibus1: on bce1 > > brgphy1: PHY 1 on miibus1 > > brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > > 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > > bce1: Ethernet address: d4:ae:52:8d:42:fd > > bce1: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); > > Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.11) > > Coal (RX:6,6,18,18; TX:20,20,80,80) > >=20 >=20 There was no change reverting r253128. I don't think that this affects what I'm seeing. However ... see below >=20 > These machines are using IPMI for management (Dell R410) and seem to be > unable to attach to /dev/ipmi0: >=20 > ipmi0: port 0xca8,0xcac on acpi0 > ipmi0: KCS mode found at io 0xca8 on acpi > .... > ipmi1: on isa0 > device_attach: ipmi1 attach returned 16 > ipmi1: on isa0 > device_attach: ipmi1 attach returned 16 > ... > ipmi0: Timed out waiting for GET_DEVICE_ID >=20 >=20 > Sean >=20 tl;dr need a review of http://people.freebsd.org/~sbruno/ipmi_fixes.txt I don't understand why, but ipmi_isa.c attach isn't seeing that ipmi_acpi.c is attached at all. Moreover, it takes slightly more that 3 seconds for the BMC on a Dell R410 to respond to GET_DEVICE_ID while using the SOL console at 9600. :-( So, I've adjusted ipmivars.h::MAX_TIMEOUT to be (6 * hz) to properly attach to ipmi0. I've also done something gross, but I can't really see any way around it. I've added detection to ipmi_isa.c to see if the acpi IPMI interface is enabled/disabled via "acpi_disabled" which means I have to include ACPI header files in acpi_isa.c ... amusing, but it seems to work. I've been able to return the IPMI controller to the same behavior that it appears to have in stable/7 now with the attached patch. --- ipmi0: port 0xca8,0xcac on acpi0 ipmi0: KCS mode found at io 0xca8 on acpi ipmi0: IPMI device rev. 0, firmware rev. 1.90, version 2.0 ipmi0: Number of channels 5 ipmi0: Attached watchdog --- It looks like our implementation of IPMI somehow tries to attach TWICE to the IPMI controller, once via ACPI and once via ISA. This is really confusing the hell out of the Broadcom management firmware even though the second attachment fails. Sean --=-II4MRHQ0nn4a6qVoRST0 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (FreeBSD) iQEcBAABAgAGBQJR8ngvAAoJEBkJRdwI6BaH4G4IAJFVhxAOiiYrYWfZNG/cGrhs P+7j0SGKWk1C5CF2VxV5a6V8ma/2LkwNOyteem98dEIeV+Yj9YvPsZ9U/M8s8JJb hskulFV7fPnDKXkX6R9SoaB60a+5C4UirWSqw8OpOx0kBzYg8OtBKQfqeZs4JWMR f0Dlcw6QM+un7hDRWC+z45OfyiR8MzP68Zs6IRqiSS3BEOaiSpe1eg3f5Es1XJxc kVTosH/meOqhzqbhAoMYC6hYbObtCOq0FRs+BDMgbHXXtVH6oXkEA1Vhd5tSH8cO bhOmodMlD4tdFa4baH13ND1MngR6vOHZYI1DxOeISH0bxPkfX5BFJDNLyn9xLm4= =hR/H -----END PGP SIGNATURE----- --=-II4MRHQ0nn4a6qVoRST0-- From owner-freebsd-net@FreeBSD.ORG Fri Jul 26 14:51:38 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 1C97677D for ; Fri, 26 Jul 2013 14:51:38 +0000 (UTC) (envelope-from melifaro@yandex-team.ru) Received: from forward-corp1g.mail.yandex.net (forward-corp1g.mail.yandex.net [IPv6:2a02:6b8:0:1402::10]) by mx1.freebsd.org (Postfix) with ESMTP id A39302F81 for ; Fri, 26 Jul 2013 14:51:37 +0000 (UTC) Received: from smtpcorp4.mail.yandex.net (smtpcorp4.mail.yandex.net [95.108.252.2]) by forward-corp1g.mail.yandex.net (Yandex) with ESMTP id 8F2DA3660149 for ; Fri, 26 Jul 2013 18:51:33 +0400 (MSK) Received: from smtpcorp4.mail.yandex.net (localhost [127.0.0.1]) by smtpcorp4.mail.yandex.net (Yandex) with ESMTP id 7679D2C00D9 for ; Fri, 26 Jul 2013 18:51:33 +0400 (MSK) Received: from dhcp170-36-red.yandex.net (dhcp170-36-red.yandex.net [95.108.170.36]) by smtpcorp4.mail.yandex.net (nwsmtp/Yandex) with ESMTP id Fjh5ceiw6T-pXxeCXI7; Fri, 26 Jul 2013 18:51:33 +0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1374850293; bh=PG3tAwxHVveTSdodvC0hlowbSpA+lu1cfQwEKXzD5fg=; h=Message-ID:Date:From:User-Agent:MIME-Version:To:Subject: Content-Type; b=Ex9D2eRRMLHYuAlTXoRCu6Wr66PTF7LJlmeJCThDwMzwKgUZdK+1ncWEeVnIsk8MS fUecmAH+J6woLU+4bxHzPvlSAaIHyty5WiCyuHJk9t8GAQv8sKl6+91noyXLuJQL7k GWoh1nMN/GvJcPh6BT8UOW+bVLjdnuRV+VKQZoRo= Authentication-Results: smtpcorp4.mail.yandex.net; dkim=pass header.i=@yandex-team.ru Message-ID: <51F28CBD.8040501@yandex-team.ru> Date: Fri, 26 Jul 2013 18:50:37 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130418 Thunderbird/17.0.5 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: [patch] pxeloader sometimes "freezes" if called via pxechain Content-Type: multipart/mixed; boundary="------------060004010402060108050100" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 14:51:38 -0000 This is a multi-part message in MIME format. --------------060004010402060108050100 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hello list! Currently our pxeloader prefer to get NFS server information by its own instead of using cached one. This is how it is done: pxe_open() indirectly opens socket (PXENV_UDP_OPEN, with cached IP from BOOTPLAYER). After that, bootp() resolve is done, resulting in (probably) more "valid" server address. Current code incorrectly assumes that DHCP server returns the same IP address for given MAC regardless of other parameters. If this is not true, outgoing packets are send via cached IP (since the only way to change source IP is to reopen pxe handle), but receive filter is programmed to use "new" one. This leads to random timeouts in some DHCP configurations. What we can probably do: 1) try cached server address if any without doing DHCP (possibly breaking some users setups) 2) try to reopen PXE handle with new address (netif_close() / netif_open(), which is complicated due to many open/close calls reading 'old' address) 3) save address/gw and restore them (if differs) before/after bootp(). 4) probably a bunch of others Patch implementing (3) is attached. I'll commit it on the end of next week if there will be no objections. --------------060004010402060108050100 Content-Type: text/plain; charset=UTF-8; name="pxechain_fix.diff" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="pxechain_fix.diff" Index: sys/boot/i386/libi386/pxe.c =================================================================== --- sys/boot/i386/libi386/pxe.c (revision 239353) +++ sys/boot/i386/libi386/pxe.c (working copy) @@ -255,11 +255,17 @@ pxe_open(struct open_file *f, ...) char temp[FNAME_SIZE]; int error = 0; int i; + struct in_addr _myip, _gateip, _rootip; + struct iodesc *d; va_start(args, f); devname = va_arg(args, char*); va_end(args); + _myip.s_addr = INADDR_ANY; + _gateip.s_addr = INADDR_ANY; + _rootip.s_addr = INADDR_ANY; + /* On first open, do netif open, mount, etc. */ if (pxe_opens == 0) { /* Find network interface. */ @@ -271,6 +277,16 @@ pxe_open(struct open_file *f, ...) } if (pxe_debug) printf("pxe_open: netif_open() succeeded\n"); + if (bootplayer.yip != INADDR_ANY) { + _myip.s_addr = bootplayer.yip; + _rootip.s_addr = bootplayer.sip; + _gateip.s_addr = bootplayer.gip; + if (pxe_debug) { + printf("pxe_open: cached myip: %s\n", inet_ntoa(_myip)); + printf("pxe_open: cached gw: %s\n", inet_ntoa(_gateip)); + printf("pxe_open: cached server: %s\n", inet_ntoa(_rootip)); + } + } } if (rootip.s_addr == 0) { /* @@ -280,9 +296,24 @@ pxe_open(struct open_file *f, ...) * which brought us to life and a default rootpath. */ bootp(pxe_sock, BOOTP_PXE); + if (_myip.s_addr != INADDR_ANY && _myip.s_addr != myip.s_addr) { + if (!(d = socktodesc(pxe_sock))) { + printf("pxe_open: bad socket. %d\n", pxe_sock); + return (ENXIO); + } + if (pxe_debug) { + printf("pxe_open: reverting IP back to %s\n", + inet_ntoa(_myip)); + printf("pxe_open: (unused)DHCP IP is %s\n", + inet_ntoa(myip)); + } + myip = _myip; + gateip = _gateip; + d->myip = myip; + } if (rootip.s_addr == 0) rootip.s_addr = bootplayer.sip; - if (!rootpath[0]) + if ((!rootpath[0]) || (!strcmp(rootpath, "/"))) strcpy(rootpath, PXENFSROOTPATH); for (i = 0; rootpath[i] != '\0' && i < FNAME_SIZE; i++) @@ -297,6 +328,7 @@ pxe_open(struct open_file *f, ...) } printf("pxe_open: server addr: %s\n", inet_ntoa(rootip)); printf("pxe_open: server path: %s\n", rootpath); + printf("pxe_open: local addr: %s\n", inet_ntoa(myip)); printf("pxe_open: gateway ip: %s\n", inet_ntoa(gateip)); setenv("boot.netif.ip", inet_ntoa(myip), 1); @@ -590,6 +622,9 @@ pxe_netif_probe(struct netif *nif, void *machdep_h bzero(udpopen_p, sizeof(*udpopen_p)); udpopen_p->src_ip = bootplayer.yip; +#ifdef PXE_DEBUG + printf("pxe_netif_probe: IP=%s\n", inet_ntoa(udpopen->src_ip)); +#endif pxe_call(PXENV_UDP_OPEN); if (udpopen_p->status != 0) { --------------060004010402060108050100-- From owner-freebsd-net@FreeBSD.ORG Fri Jul 26 15:31:05 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0F972228 for ; Fri, 26 Jul 2013 15:31:05 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm2-vm5.bullet.mail.ne1.yahoo.com (nm2-vm5.bullet.mail.ne1.yahoo.com [98.138.91.224]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BD63421F9 for ; Fri, 26 Jul 2013 15:31:04 +0000 (UTC) Received: from [98.138.226.179] by nm2.bullet.mail.ne1.yahoo.com with NNFMP; 26 Jul 2013 15:30:58 -0000 Received: from [98.138.89.170] by tm14.bullet.mail.ne1.yahoo.com with NNFMP; 26 Jul 2013 15:30:58 -0000 Received: from [127.0.0.1] by omp1026.mail.ne1.yahoo.com with NNFMP; 26 Jul 2013 15:30:58 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 455354.40765.bm@omp1026.mail.ne1.yahoo.com Received: (qmail 90245 invoked by uid 60001); 26 Jul 2013 15:30:58 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1374852658; bh=Xr0QMD8agYb8yGuC+h/bsYcrnAGJQdudxcQfZqgbYUY=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=c6ToKezncXSvpc/k2rD6fwvmQnbJcc/yXEAlmwrPBeeRw2Q3heBvyPpI2vX1FhGG9Q9iYw5s5a5oDrJrThBS+XKJq3rWn2lhu0vRADQwhNhbIoQWereMjGTkb+70wQ8Neawj1nb28gkZ1KqD8O0tzpwE/FUp3sI9AJfqc5dvFqM= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=2bNU1flqby28BahFBQL887XYp9YHENcOjI4BcYD6ERKsi6F2pIJhpK6Y46NVN0LvLKdTcKwRjO5Fgqgb6gzE69/23+n64A5C8Ta7stPPFsk6byCKwP+ZMXqvbtfUIie48juL9YQJwQPybInTCq25WDv27BjbKUbOH2zycn9ajNk=; X-YMail-OSG: hQ6kJC4VM1nzhCJAc5683Ns6097dDVDpic1fYkUk8YrgfkM JkLG0PUCV.cPRz0tuQJrnVkjWR1Gl1eCLqpO8KapuPxL2i97QmYRnZuAYS1x DB92xZpdOfesacyNGWZ5MVOwuoe8IZAf19v2W1XV7TKa9Sa65ISnG.j9YP.X ISvfYn6NfWdEeAAGgPzJhVKpVjM.u6Hha8_tMI82TKI2uQPzEUsz73ZF9zSW gDwUnAhkS9cBul0C5xs645XqeLCdT2mUADSTmLPvnLI3yHXPTabuXxOIxNcL h4hUDsUyxcwiidPrBYaDQVlvLBIw8DVpQPFoRzgzt1apQdEBoGh148jx1QfC .29Dle0gGSGGty6RLoMpzO5li1u8Tq1aLuBh1WMQwomRRtvld4av6BWMwhPo BS9hfxHWfSIdHNrqMy3YlIlrzSiOhSm5Oa44DR0PUys1n_h0IJXDUukK58iX v0DD0D3GSaFjMq2b1dr26L.jZovBPH3FNkEiWURaBlYTGMVa4unLo7f4cM23 j6mz12ogIhc6BtYXI3GNgA0sQuwC2toiVeuk9KHLjlgB7GF48AeUlDMDydne QInp8r.j1mu4PqEia34QPcQ-- Received: from [98.203.118.124] by web121601.mail.ne1.yahoo.com via HTTP; Fri, 26 Jul 2013 08:30:58 PDT X-Rocket-MIMEInfo: 002.001, CgoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCiBGcm9tOiBBbGV4YW5kZXIgVi4gQ2hlcm5pa292IDxtZWxpZmFyb0BGcmVlQlNELm9yZz4KVG86IEJvcmlzIEtvY2hlcmdpbiA8c3Bhd2tAYWNtLnBvbHkuZWR1PiAKQ2M6IGZyZWVic2QtbmV0QGZyZWVic2Qub3JnIApTZW50OiBUaHVyc2RheSwgSnVseSAyNSwgMjAxMyAyOjEwIFBNClN1YmplY3Q6IFJlOiBSZWNvbW1lbmRhdGlvbnMgZm9yIDEwZ2JwcyBOSUMKIAoKT24gMjUuMDcuMjAxMyAwMDoyNiwgQm9yaXMgS29jaGVyZ2luIHdyb3RlOgoBMAEBAQE- X-Mailer: YahooMailWebService/0.8.150.561 References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> Message-ID: <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> Date: Fri, 26 Jul 2013 08:30:58 -0700 (PDT) From: Barney Cordoba Subject: Re: Recommendations for 10gbps NIC To: "Alexander V. Chernikov" In-Reply-To: <51F16A07.9030505@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Barney Cordoba List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 15:31:05 -0000 ________________________________ From: Alexander V. Chernikov To: Boris Kochergin Cc: freebsd-net@freebsd.org Sent: Thursday, July 25, 2013 2:10 PM Subject: Re: Recommendations for 10gbps NIC On 25.07.2013 00:26, Boris Kochergin wrote: > Hi. Hello. > > I am looking for recommendations for a 10gbps NIC from someone who has > successfully used it on FreeBSD. It will be used on FreeBSD 9.1-R/amd64 > to capture packets. Some desired features are: > > - PCIe > - LC connectors > - 10GBASE-SR > - Either single- or dual-port > - Multiqueue Intel 82598/99/X520 Emulex OCe10102-NM Mellanox ConnectX Chelsio T4 Do they all cost the same, have the exact same features and have equally well-written drivers? Which do you recommend and why? BC From owner-freebsd-net@FreeBSD.ORG Fri Jul 26 16:26:54 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id F19DD36D for ; Fri, 26 Jul 2013 16:26:54 +0000 (UTC) (envelope-from melifaro@FreeBSD.org) Received: from mail.ipfw.ru (unknown [IPv6:2a01:4f8:120:6141::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 882DD24F4 for ; Fri, 26 Jul 2013 16:26:54 +0000 (UTC) Received: from dhcp170-36-red.yandex.net ([95.108.170.36]) by mail.ipfw.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.76 (FreeBSD)) (envelope-from ) id 1V2ku4-0005aC-5Z; Fri, 26 Jul 2013 20:30:04 +0400 Message-ID: <51F2A313.9070105@FreeBSD.org> Date: Fri, 26 Jul 2013 20:25:55 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130418 Thunderbird/17.0.5 MIME-Version: 1.0 To: Barney Cordoba Subject: Re: Recommendations for 10gbps NIC References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> In-Reply-To: <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 16:26:55 -0000 On 26.07.2013 19:30, Barney Cordoba wrote: > > > ------------------------------------------------------------------------ > *From:* Alexander V. Chernikov > *To:* Boris Kochergin > *Cc:* freebsd-net@freebsd.org > *Sent:* Thursday, July 25, 2013 2:10 PM > *Subject:* Re: Recommendations for 10gbps NIC > > On 25.07.2013 00:26, Boris Kochergin wrote: > > Hi. > Hello. > > > > I am looking for recommendations for a 10gbps NIC from someone who has > > successfully used it on FreeBSD. It will be used on FreeBSD 9.1-R/amd64 > > to capture packets. Some desired features are: > > > > - PCIe > > - LC connectors > > - 10GBASE-SR > > - Either single- or dual-port > > - Multiqueue > Intel 82598/99/X520 > Emulex OCe10102-NM > Mellanox ConnectX > Chelsio T4 > > Do they all cost the same, have the exact same features and have > equally well-written drivers? Which do you recommend > and why? Well, Intel/Chelsio/Mellanox costs $500-600 / 2 ports, Emulex costs a bit higher (of course, YMMV) Regarding features: it depends on your needs mostly (e.g. forwarding / TCP server / IB ). for forwarding: Each one is capable of doing at least 8 rx/tx queues per port with adjustable rx/tx ring size. One important thing to note is that netmap is currently not available for any non-Intel NIC. General hw offload features: LRO/TSO4/vlans/jumbo are supported by all of them (not sure about TSO6 in Mellanox) however I can't say how good it works. SFP+/Twinax: Intel was very restrictive on supported spf/cables (however, Jack changed this a moth ago). Chelsio is much better here. Advanced features: Intel (82599-based NICs) has hw firewall called FlowDirector which can be programmed to redirect particular IPv4/IPv6 TCP/UDP streams to given queue (or dropped). However it is currently not supported on FreeBSD. (Intel also has IPSEC offload functionality, which, afaik, is currently not supported) On the other side, Chelsio has similar functionality which is supported via cxgbetool binary (available in tools/) Usability: IMHO Chelsio/Intel are the best (a lot of things are runtime-configurable, lots of counters, etc..) > > BC From owner-freebsd-net@FreeBSD.ORG Fri Jul 26 16:28:25 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 370C7464 for ; Fri, 26 Jul 2013 16:28:25 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0B8F42514 for ; Fri, 26 Jul 2013 16:28:24 +0000 (UTC) Received: from compute6.internal (compute6.nyi.mail.srv.osa [10.202.2.46]) by gateway1.nyi.mail.srv.osa (Postfix) with ESMTP id BCC1820CAC for ; Fri, 26 Jul 2013 12:28:23 -0400 (EDT) Received: from web3 ([10.202.2.213]) by compute6.internal (MEProxy); Fri, 26 Jul 2013 12:28:23 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=message-id:from:to:mime-version :content-transfer-encoding:content-type:in-reply-to:references :subject:date; s=smtpout; bh=dFv3obzWsg3KZpwYnX9Yv+qsAH8=; b=Cvf lfh2IpQg2lHg+643PoZjgpeiQECO7avyB6905CWvi45iSizP36BlZ4+gWzctvAfc HKJ+KeidcL9F8wMEGueVQPonPvxCSiX4/6NvR+oxBGEP75U8nOIHoexdWCpD49xR icuppSWZh18wxMZb69hiZZB0jNgNH3E0N34Ee72Y= Received: by web3.nyi.mail.srv.osa (Postfix, from userid 99) id A17A8B01D88; Fri, 26 Jul 2013 12:28:23 -0400 (EDT) Message-Id: <1374856103.13825.1940671.2C599B3B@webmail.messagingengine.com> X-Sasl-Enc: Nc9X4TR8jtyiALsIwa635mY6slN9MNQVfQCzNX3ISwBx 1374856103 From: Mark Felder To: freebsd-net@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Mailer: MessagingEngine.com Webmail Interface - ajax-23e62cd3 In-Reply-To: <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> Subject: Re: Recommendations for 10gbps NIC Date: Fri, 26 Jul 2013 11:28:23 -0500 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 16:28:25 -0000 My only experience with 10gbps on FreeBSD is on my friend's server -- Solarstorm SFN5161T 10GBASE-T Server Adapter I guess the model you'd want is the SFN5162F and then you put in any SFP+ adapters you want, so this would cover your 10GBASE-SR requirement. The driver is new as of 9.1-RELEASE. It was surprising how well it performed out of the box. Without tuning we had something like 8.5Gbit/s in iperf. I have no idea if it provides your multiqueue requirement. From owner-freebsd-net@FreeBSD.ORG Fri Jul 26 21:00:02 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 158C5D6D; Fri, 26 Jul 2013 21:00:02 +0000 (UTC) (envelope-from feenberg@nber.org) Received: from mail2.nber.org (mail2.nber.org [66.251.72.79]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C70E12FEE; Fri, 26 Jul 2013 21:00:01 +0000 (UTC) Received: from nber3.nber.org (nber3.nber.org [66.251.72.73]) by mail2.nber.org (8.14.4/8.14.4) with ESMTP id r6QKxpAQ066313 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Fri, 26 Jul 2013 16:59:52 -0400 (EDT) (envelope-from feenberg@nber.org) Date: Fri, 26 Jul 2013 16:59:51 -0400 (EDT) From: Daniel Feenberg To: "Alexander V. Chernikov" Subject: Re: Recommendations for 10gbps NIC In-Reply-To: <51F2A313.9070105@FreeBSD.org> Message-ID: References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> <51F2A313.9070105@FreeBSD.org> User-Agent: Alpine 2.03 (LRH 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Anti-Virus: Kaspersky Anti-Virus for Linux Mail Server 5.6.39/RELEASE, bases: 20130726 #10689482, check: 20130726 clean Cc: Barney Cordoba , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 21:00:02 -0000 On Fri, 26 Jul 2013, Alexander V. Chernikov wrote: > On 26.07.2013 19:30, Barney Cordoba wrote: >> >> >> ------------------------------------------------------------------------ >> *From:* Alexander V. Chernikov >> *To:* Boris Kochergin >> *Cc:* freebsd-net@freebsd.org >> *Sent:* Thursday, July 25, 2013 2:10 PM >> *Subject:* Re: Recommendations for 10gbps NIC >> >> On 25.07.2013 00:26, Boris Kochergin wrote: >> > Hi. >> Hello. >> > >> > I am looking for recommendations for a 10gbps NIC from someone who has >> > successfully used it on FreeBSD. It will be used on FreeBSD 9.1-R/amd64 >> > to capture packets. Some desired features are: >> > We have experience with HP NC523SFP and Chelsio N320E. The key difference among 10GBE cards for us is how they treat foreign DACs. The HP would PXE boot with several brands and generic DACs, but the Chelsio required a Chelsio brand DAC to PXE boot. There was firmware on the NIC to check the brand of cable. Both worked fine once booted. The Chelsio cables were hard to find, which became a problem. Also, when used with diskless Unix clients the Chelsio cards seemed to hang from time to time. Otherwise packet loss was one in a million for both cards, even with 7 meter cables. We liked the fact that the Chelsio cards were single-port and cheaper. I don't really understand why nearly all 10GBE cards are dual-port. Surely there is a market for NICs between 1 gigabit and 20 gigabit. The NIC heatsinks are too hot to touch during use unless specially cooled. Daniel Feenberg NBER From owner-freebsd-net@FreeBSD.ORG Fri Jul 26 21:45:12 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D4413E2B; Fri, 26 Jul 2013 21:45:12 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AA2B22224; Fri, 26 Jul 2013 21:45:12 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6QLjCai038314; Fri, 26 Jul 2013 21:45:12 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6QLjC6U038313; Fri, 26 Jul 2013 21:45:12 GMT (envelope-from linimon) Date: Fri, 26 Jul 2013 21:45:12 GMT Message-Id: <201307262145.r6QLjC6U038313@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/180873: [sctp] SCTP connection hangs on COOKIE_ECHOED X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 21:45:12 -0000 Old Synopsis: SCTP connection hangs on COOKIE_ECHOED New Synopsis: [sctp] SCTP connection hangs on COOKIE_ECHOED Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Fri Jul 26 21:45:01 UTC 2013 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=180873 From owner-freebsd-net@FreeBSD.ORG Fri Jul 26 21:46:42 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2D696F39; Fri, 26 Jul 2013 21:46:42 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 01845225A; Fri, 26 Jul 2013 21:46:42 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6QLkf7P038374; Fri, 26 Jul 2013 21:46:41 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6QLkfSA038373; Fri, 26 Jul 2013 21:46:41 GMT (envelope-from linimon) Date: Fri, 26 Jul 2013 21:46:41 GMT Message-Id: <201307262146.r6QLkfSA038373@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/180722: [bluetooth] bluetooth takes 30-50 attempts to pair to keyboard and will not re-connect X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 21:46:42 -0000 Old Synopsis: bluetooth takes 30-50 attempts to pair to keyboard and will not re-connect New Synopsis: [bluetooth] bluetooth takes 30-50 attempts to pair to keyboard and will not re-connect Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Fri Jul 26 21:46:03 UTC 2013 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=180722 From owner-freebsd-net@FreeBSD.ORG Fri Jul 26 22:14:19 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C4927B36 for ; Fri, 26 Jul 2013 22:14:19 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm7-vm0.bullet.mail.ne1.yahoo.com (nm7-vm0.bullet.mail.ne1.yahoo.com [98.138.91.66]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7AF232380 for ; Fri, 26 Jul 2013 22:14:19 +0000 (UTC) Received: from [98.138.101.129] by nm7.bullet.mail.ne1.yahoo.com with NNFMP; 26 Jul 2013 22:14:18 -0000 Received: from [98.138.89.240] by tm17.bullet.mail.ne1.yahoo.com with NNFMP; 26 Jul 2013 22:14:18 -0000 Received: from [127.0.0.1] by omp1013.mail.ne1.yahoo.com with NNFMP; 26 Jul 2013 22:14:18 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 999721.17298.bm@omp1013.mail.ne1.yahoo.com Received: (qmail 43147 invoked by uid 60001); 26 Jul 2013 22:14:17 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1374876857; bh=Fn7U9xa/nf7rJ1PA7qEZie1WLiWJ4BhOv8CfGoj0l0g=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=ozKZRm+Eg21Z487jUmjIKfkvssT78fl9qA/G3BeHxKHnCCyoAM1EWPcmcrEjYtlXHlLEwSbTLXy1lngN6mHhghRRPFGDmL/ERBN7j9OY+yne1G46lneQxB1sUMKGlztoAz2prdGfD/M0SGc5rIErzue6olmlVhwqJzYXj0bMawc= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=s0R0XSM00cmykEZhDyHpLW34Xt8OCTMKNOG3+RVYViO5EB5S0vOswymzDx03kw2XjssX6GJT4Lf77I02tsBz62kG2D6fCrsj76/U+lv5WVm8/hQXFelkw9EC3Xr3Ft9hAB8r/RUb1lbiyMgs2cr6amF+TX3tkAFgRKS+oBnP3xw=; X-YMail-OSG: QyQqr8oVM1lz8.dAEktyq6OCVDD53D5XT81tDknRaP9eO2s ExzwsDx.6FREC1Hjn7MIZCrXmoSjX.Y6CkdHsLn4tdoMGEuLTIVELpOXBO9I 2WH1YN_6.t3juNosUIhJ8CYCW59YzrnYXhLAXgHyUaxAMDeSlnKI9PhBmdUE m28i3dAvfvAzFBSGYHGwb_EXF6quuVvb4wuuU3UkJM1rwkhOei6DZ3CRExKP otAr.rXhG6lN72N6nhNEYKJnJFnRmk5id2Od.dyCKI916BUlNonCDCrkw8nu 6OZBETMuYD9YNONDR3Lf_u8_h6qTn9KcaHletbZSEcGS5rQDaMQHaxkJQdIB Gyt9HpAcAYsk7ZbyFO0gFlPb2Ak2_prvxTxZTJ7jZjNMOn7ujbOneSplCJug 8V3Wf.XQ2aoLt4NQ4WoL5SEg7SMUSozPrJVuxRqnJgKPENce71jAvA0WMMlR JfQF1Cb394fcx8XKYFZoLVKUsMon7gghkIQEsMMhQUA.Lg2jLD25Zw3LqJNn Zb.g3uB_MADPRRa_XrEPg1yAF0xBa_R87LJ.ZrWcpbAMZQKqLkPlVc42ZKpU _6TkSy1KscdX_g3k- Received: from [98.203.118.124] by web121603.mail.ne1.yahoo.com via HTTP; Fri, 26 Jul 2013 15:14:17 PDT X-Rocket-MIMEInfo: 002.001, CgoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCiBGcm9tOiBEYW5pZWwgRmVlbmJlcmcgPGZlZW5iZXJnQG5iZXIub3JnPgpUbzogQWxleGFuZGVyIFYuIENoZXJuaWtvdiA8bWVsaWZhcm9ARnJlZUJTRC5vcmc.IApDYzogQmFybmV5IENvcmRvYmEgPGJhcm5leV9jb3Jkb2JhQHlhaG9vLmNvbT47ICJmcmVlYnNkLW5ldEBmcmVlYnNkLm9yZyIgPGZyZWVic2QtbmV0QGZyZWVic2Qub3JnPiAKU2VudDogRnJpZGF5LCBKdWx5IDI2LCAyMDEzIDQ6NTkgUE0KU3ViamVjdDogUmU6IFJlY29tbWVuZGEBMAEBAQE- X-Mailer: YahooMailWebService/0.8.150.561 References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> <51F2A313.9070105@FreeBSD.org> Message-ID: <1374876857.42890.YahooMailNeo@web121603.mail.ne1.yahoo.com> Date: Fri, 26 Jul 2013 15:14:17 -0700 (PDT) From: Barney Cordoba Subject: Re: Recommendations for 10gbps NIC To: Daniel Feenberg , "Alexander V. Chernikov" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Barney Cordoba List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 22:14:19 -0000 =0A=0A=0A=0A________________________________=0A From: Daniel Feenberg =0ATo: Alexander V. Chernikov =0ACc: B= arney Cordoba ; "freebsd-net@freebsd.org" =0ASent: Friday, July 26, 2013 4:59 PM=0ASubject: Re: Re= commendations for 10gbps NIC=0A =0A=0A=0AOn Fri, 26 Jul 2013, Alexander V. = Chernikov wrote:=0A=0A> On 26.07.2013 19:30, Barney Cordoba wrote:=0A>> =0A= >> =0A>> ------------------------------------------------------------------= ------=0A>> *From:* Alexander V. Chernikov =0A>> *To:= * Boris Kochergin =0A>> *Cc:* freebsd-net@freebsd.org= =0A>> *Sent:* Thursday, July 25, 2013 2:10 PM=0A>> *Subject:* Re: Recommend= ations for 10gbps NIC=0A>> =0A>> On 25.07.2013 00:26, Boris Kochergin wrote= :=0A>> > Hi.=0A>> Hello.=0A>> >=0A>> > I am looking for recommendations for= a 10gbps NIC from someone who has=0A>> > successfully used it on FreeBSD. = It will be used on FreeBSD 9.1-R/amd64=0A>> > to capture packets. Some desi= red features are:=0A>> >=0A=0AWe have experience with HP NC523SFP and Chels= io N320E. The key difference =0Aamong 10GBE cards for us is how they treat = foreign DACs. The HP would PXE =0Aboot with several brands and generic DACs= , but the Chelsio required a =0AChelsio brand DAC to PXE boot.=A0 There was= firmware on the NIC to check the =0Abrand of cable. Both worked fine once = booted. The Chelsio cables were hard =0Ato find, which became a problem. Al= so, when used with diskless Unix =0Aclients the Chelsio cards seemed to han= g from time to time. Otherwise =0Apacket loss was one in a million for both= cards, even with 7 meter cables.=0A=0AWe liked the fact that the Chelsio c= ards were single-port and cheaper. I =0Adon't really understand why nearly = all 10GBE cards are dual-port. Surely =0Athere is a market for NICs between= 1 gigabit and 20 gigabit.=0A=0AThe NIC heatsinks are too hot to touch duri= ng use unless specially cooled.=0A=0ADaniel Feenberg=0ANBER=0A=0A=0A=0A----= -----------------=0AThe same reason that they don't make single core cpus a= nymore. It costs about the=0Asame to make a 1 port chip as a 2 port chip.= =0A=0AI find it interesting =A0how so many talk about "the cards", when mos= t often the=0Adifferences are with "the drivers". Luigi made the most usefu= l comment; if you ever=0Awant to use netmap, =A0you need to buy a card comp= atible with netmap. Although=0Ayou don't =A0need netmap just to capture 10G= b/s. Forwarding, Maybe.=A0=0A=0AI also find it interesting that nobody seem= s to have a handle on the performance=0Adifferences. Obviously they're all = different. Maybe substantially different.=0A=0AThe x540 with RJ45 has the o= bvious advantage of being compatible with regular gigabit cards,=A0=0Aand s= ingle port adapters are about $325 in the US.=A0=0A=0AWhen cheap(er) 10g RJ= 45 switches become available, it will start to be used more and more.=0AVer= y soon.=0A=0ABC=0A From owner-freebsd-net@FreeBSD.ORG Sat Jul 27 08:02:40 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id B36AA2FA for ; Sat, 27 Jul 2013 08:02:40 +0000 (UTC) (envelope-from melifaro@FreeBSD.org) Received: from mail.ipfw.ru (unknown [IPv6:2a01:4f8:120:6141::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 742A92BAC for ; Sat, 27 Jul 2013 08:02:40 +0000 (UTC) Received: from v6.mpls.in ([2a02:978:2::5] helo=ws.su29.net) by mail.ipfw.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.76 (FreeBSD)) (envelope-from ) id 1V2zVc-000CyT-8n; Sat, 27 Jul 2013 12:05:48 +0400 Message-ID: <51F37E97.3090203@FreeBSD.org> Date: Sat, 27 Jul 2013 12:02:31 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20120121 Thunderbird/9.0 MIME-Version: 1.0 To: Barney Cordoba Subject: Re: Recommendations for 10gbps NIC References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> <51F2A313.9070105@FreeBSD.org> <1374876857.42890.YahooMailNeo@web121603.mail.ne1.yahoo.com> In-Reply-To: <1374876857.42890.YahooMailNeo@web121603.mail.ne1.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-net@freebsd.org" , Daniel Feenberg X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2013 08:02:40 -0000 On 27.07.2013 02:14, Barney Cordoba wrote: > > > ------------------------------------------------------------------------ > *From:* Daniel Feenberg > *To:* Alexander V. Chernikov > *Cc:* Barney Cordoba ; > "freebsd-net@freebsd.org" > *Sent:* Friday, July 26, 2013 4:59 PM > *Subject:* Re: Recommendations for 10gbps NIC > > > On Fri, 26 Jul 2013, Alexander V. Chernikov wrote: > > > On 26.07.2013 19:30, Barney Cordoba wrote: > >> > >> > >> ------------------------------------------------------------------------ > >> *From:* Alexander V. Chernikov > > >> *To:* Boris Kochergin > > >> *Cc:* freebsd-net@freebsd.org > >> *Sent:* Thursday, July 25, 2013 2:10 PM > >> *Subject:* Re: Recommendations for 10gbps NIC > >> > >> On 25.07.2013 00:26, Boris Kochergin wrote: > >> > Hi. > >> Hello. > >> > > >> > I am looking for recommendations for a 10gbps NIC from someone who has > >> > successfully used it on FreeBSD. It will be used on FreeBSD > 9.1-R/amd64 > >> > to capture packets. Some desired features are: > >> > > > We have experience with HP NC523SFP and Chelsio N320E. The key difference > among 10GBE cards for us is how they treat foreign DACs. The HP would PXE > boot with several brands and generic DACs, but the Chelsio required a > Chelsio brand DAC to PXE boot. There was firmware on the NIC to check the > brand of cable. Both worked fine once booted. The Chelsio cables were hard > to find, which became a problem. Also, when used with diskless Unix > clients the Chelsio cards seemed to hang from time to time. Otherwise > packet loss was one in a million for both cards, even with 7 meter cables. > > We liked the fact that the Chelsio cards were single-port and cheaper. I > don't really understand why nearly all 10GBE cards are dual-port. Surely > there is a market for NICs between 1 gigabit and 20 gigabit. > > The NIC heatsinks are too hot to touch during use unless specially cooled. > > Daniel Feenberg > NBER > > > --------------------- > The same reason that they don't make single core cpus anymore. It costs > about the > same to make a 1 port chip as a 2 port chip. > > I find it interesting how so many talk about "the cards", when most > often the > differences are with "the drivers". Luigi made the most useful comment; > if you ever > want to use netmap, you need to buy a card compatible with netmap. Although > you don't need netmap just to capture 10Gb/s. Forwarding, Maybe. > > I also find it interesting that nobody seems to have a handle on the > performance > differences. Obviously they're all different. Maybe substantially different. It depends on what kind of performance you are talking about. All NICs are capable of doing linerate RX/TX for both small/big packets. The only notable exception I;m aware of are Intel 82598-based NICs which advertise PCI-E X8 gen2 with _2.5GT_ link speed, giving you maximum ~14Gbit/s bw for 2 ports instead of 20. > > The x540 with RJ45 has the obvious advantage of being compatible with > regular gigabit cards, > and single port adapters are about $325 in the US. > > When cheap(er) 10g RJ45 switches become available, it will start to be > used more and more. > Very soon. > > BC > From owner-freebsd-net@FreeBSD.ORG Sat Jul 27 08:15:54 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 65AF0456; Sat, 27 Jul 2013 08:15:54 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-la0-x22b.google.com (mail-la0-x22b.google.com [IPv6:2a00:1450:4010:c03::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AEA772BF3; Sat, 27 Jul 2013 08:15:53 +0000 (UTC) Received: by mail-la0-f43.google.com with SMTP id fs13so2883883lab.2 for ; Sat, 27 Jul 2013 01:15:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=WU0jGTI9zPgNW1T0MKJ6mzLhEhhhah49cegKkNvRFzA=; b=eRS3Cy7A6HWqmgmX54AnIizpQQ0aH0g6Rn92F3+L2T3ZVl6iAqvqdsa8j8XPial/pD 3K3zENEKeaVRZgvbS1tn9PuxCC7ZiegQHaaNri7dnI3ZzfFEvsrKCpgIKLb4AHG+nMtY TQEaigXP8QFdu7Nw+NbcQaaGzQbXxkVuMDLHVNyzlGUcyVjo/Au2ekK7IdWDjpwMRKTK 7U8RmP48pem+SYnNYAk1T+GtduGn/J/pfzxW8Tvpr1PlKckfqP0Exbhq3jAESdc4grCE 5n6ZFBFZ+l2K8YlJeFHBssQDGCK+/C0nErZDhCx3/LyTj1fQmfeQ0O6pRe2KjKPhX7ME F+XQ== MIME-Version: 1.0 X-Received: by 10.112.20.137 with SMTP id n9mr2073184lbe.20.1374912951640; Sat, 27 Jul 2013 01:15:51 -0700 (PDT) Sender: rizzo.unipi@gmail.com Received: by 10.114.200.15 with HTTP; Sat, 27 Jul 2013 01:15:51 -0700 (PDT) In-Reply-To: <51F37E97.3090203@FreeBSD.org> References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> <51F2A313.9070105@FreeBSD.org> <1374876857.42890.YahooMailNeo@web121603.mail.ne1.yahoo.com> <51F37E97.3090203@FreeBSD.org> Date: Sat, 27 Jul 2013 10:15:51 +0200 X-Google-Sender-Auth: mFuvXtYcC12uyUfcZXNJ23XjciM Message-ID: Subject: Re: Recommendations for 10gbps NIC From: Luigi Rizzo To: "Alexander V. Chernikov" Content-Type: text/plain; charset=ISO-8859-1 Cc: Barney Cordoba , Daniel Feenberg , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2013 08:15:54 -0000 On Sat, Jul 27, 2013 at 10:02 AM, Alexander V. Chernikov wrote: > On 27.07.2013 02:14, Barney Cordoba wrote: >> >> >> >> ------------------------------------------------------------------------ >> *From:* Daniel Feenberg >> *To:* Alexander V. Chernikov >> *Cc:* Barney Cordoba ; >> "freebsd-net@freebsd.org" >> *Sent:* Friday, July 26, 2013 4:59 PM >> >> *Subject:* Re: Recommendations for 10gbps NIC >> >> >> On Fri, 26 Jul 2013, Alexander V. Chernikov wrote: >> >> > On 26.07.2013 19:30, Barney Cordoba wrote: >> >> >> >> >> >> >> ------------------------------------------------------------------------ >> >> *From:* Alexander V. Chernikov > > >> >> *To:* Boris Kochergin > >> >> *Cc:* freebsd-net@freebsd.org >> >> >> *Sent:* Thursday, July 25, 2013 2:10 PM >> >> *Subject:* Re: Recommendations for 10gbps NIC >> >> >> >> On 25.07.2013 00:26, Boris Kochergin wrote: >> >> > Hi. >> >> Hello. >> >> > >> >> > I am looking for recommendations for a 10gbps NIC from someone who >> has >> >> > successfully used it on FreeBSD. It will be used on FreeBSD >> 9.1-R/amd64 >> >> > to capture packets. Some desired features are: >> >> > >> >> We have experience with HP NC523SFP and Chelsio N320E. The key difference >> among 10GBE cards for us is how they treat foreign DACs. The HP would PXE >> boot with several brands and generic DACs, but the Chelsio required a >> Chelsio brand DAC to PXE boot. There was firmware on the NIC to check the >> brand of cable. Both worked fine once booted. The Chelsio cables were hard >> to find, which became a problem. Also, when used with diskless Unix >> clients the Chelsio cards seemed to hang from time to time. Otherwise >> packet loss was one in a million for both cards, even with 7 meter cables. >> >> We liked the fact that the Chelsio cards were single-port and cheaper. I >> don't really understand why nearly all 10GBE cards are dual-port. Surely >> there is a market for NICs between 1 gigabit and 20 gigabit. >> >> The NIC heatsinks are too hot to touch during use unless specially cooled. >> >> Daniel Feenberg >> NBER >> >> >> --------------------- >> The same reason that they don't make single core cpus anymore. It costs >> about the >> same to make a 1 port chip as a 2 port chip. >> >> I find it interesting how so many talk about "the cards", when most >> often the >> differences are with "the drivers". Luigi made the most useful comment; >> if you ever >> want to use netmap, you need to buy a card compatible with netmap. >> Although >> you don't need netmap just to capture 10Gb/s. Forwarding, Maybe. >> >> I also find it interesting that nobody seems to have a handle on the >> performance >> differences. Obviously they're all different. Maybe substantially >> different. > > It depends on what kind of performance you are talking about. > All NICs are capable of doing linerate RX/TX for both small/big packets. this is actually not true. I have direct experience with Intel, Mellanox and Broadcom, and small packets are a problem across the board even with 1 port. >From my experience only intel can do line rate (14.88Mpps) with 64-byte frames, but suffers a bit with sizes that are not multiple of 64. Mellanox peaks at around 7Mpps. Broadcom is limited to some 2.5Mpps. This is all with netmap, using the regular stack you are going to see much much less. Large frames (1400+) are probably not a problem for anyone, but since the original post asked for packet capture, i thought the small-frame case is a relevant one. > The only notable exception I;m aware of are Intel 82598-based NICs which > advertise PCI-E X8 gen2 with _2.5GT_ link speed, giving you maximum > ~14Gbit/s bw for 2 ports instead of 20. This makes me curious because i believe people have used netmap with the 82598 and achieved close to line rate even with 64-byte frames/one port, and i thought (maybe I am wrong ?) the various 2-port NICs use 4 lanes per port. So the number i remember does not match with your quote of 2.5Gt/s. Are all 82598 using 2.5GT/s (which is a gen.1 speed) instead of 5 ? cheers luigi From owner-freebsd-net@FreeBSD.ORG Sat Jul 27 08:42:26 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 665969AA for ; Sat, 27 Jul 2013 08:42:26 +0000 (UTC) (envelope-from melifaro@FreeBSD.org) Received: from mail.ipfw.ru (unknown [IPv6:2a01:4f8:120:6141::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id F05232C9A for ; Sat, 27 Jul 2013 08:42:25 +0000 (UTC) Received: from v6.mpls.in ([2a02:978:2::5] helo=ws.su29.net) by mail.ipfw.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.76 (FreeBSD)) (envelope-from ) id 1V3086-000DH9-4b; Sat, 27 Jul 2013 12:45:34 +0400 Message-ID: <51F387E8.6090704@FreeBSD.org> Date: Sat, 27 Jul 2013 12:42:16 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20120121 Thunderbird/9.0 MIME-Version: 1.0 To: Luigi Rizzo Subject: Re: Recommendations for 10gbps NIC References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> <51F2A313.9070105@FreeBSD.org> <1374876857.42890.YahooMailNeo@web121603.mail.ne1.yahoo.com> <51F37E97.3090203@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Barney Cordoba , Daniel Feenberg , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2013 08:42:26 -0000 On 27.07.2013 12:15, Luigi Rizzo wrote: > On Sat, Jul 27, 2013 at 10:02 AM, Alexander V. Chernikov > wrote: >> On 27.07.2013 02:14, Barney Cordoba wrote: >>> >>> >>> >>> ------------------------------------------------------------------------ >>> *From:* Daniel Feenberg >>> *To:* Alexander V. Chernikov >>> *Cc:* Barney Cordoba; >>> "freebsd-net@freebsd.org" >>> *Sent:* Friday, July 26, 2013 4:59 PM >>> >>> *Subject:* Re: Recommendations for 10gbps NIC >>> >>> >>> On Fri, 26 Jul 2013, Alexander V. Chernikov wrote: >>> >>> > On 26.07.2013 19:30, Barney Cordoba wrote: >>> >> >>> >> >>> >> >>> ------------------------------------------------------------------------ >>> >> *From:* Alexander V. Chernikov>> > >>> >> *To:* Boris Kochergin> >>> >> *Cc:* freebsd-net@freebsd.org >>> >>> >> *Sent:* Thursday, July 25, 2013 2:10 PM >>> >> *Subject:* Re: Recommendations for 10gbps NIC >>> >> >>> >> On 25.07.2013 00:26, Boris Kochergin wrote: >>> >> > Hi. >>> >> Hello. >>> >> > >>> >> > I am looking for recommendations for a 10gbps NIC from someone who >>> has >>> >> > successfully used it on FreeBSD. It will be used on FreeBSD >>> 9.1-R/amd64 >>> >> > to capture packets. Some desired features are: >>> >> > >>> >>> We have experience with HP NC523SFP and Chelsio N320E. The key difference >>> among 10GBE cards for us is how they treat foreign DACs. The HP would PXE >>> boot with several brands and generic DACs, but the Chelsio required a >>> Chelsio brand DAC to PXE boot. There was firmware on the NIC to check the >>> brand of cable. Both worked fine once booted. The Chelsio cables were hard >>> to find, which became a problem. Also, when used with diskless Unix >>> clients the Chelsio cards seemed to hang from time to time. Otherwise >>> packet loss was one in a million for both cards, even with 7 meter cables. >>> >>> We liked the fact that the Chelsio cards were single-port and cheaper. I >>> don't really understand why nearly all 10GBE cards are dual-port. Surely >>> there is a market for NICs between 1 gigabit and 20 gigabit. >>> >>> The NIC heatsinks are too hot to touch during use unless specially cooled. >>> >>> Daniel Feenberg >>> NBER >>> >>> >>> --------------------- >>> The same reason that they don't make single core cpus anymore. It costs >>> about the >>> same to make a 1 port chip as a 2 port chip. >>> >>> I find it interesting how so many talk about "the cards", when most >>> often the >>> differences are with "the drivers". Luigi made the most useful comment; >>> if you ever >>> want to use netmap, you need to buy a card compatible with netmap. >>> Although >>> you don't need netmap just to capture 10Gb/s. Forwarding, Maybe. >>> >>> I also find it interesting that nobody seems to have a handle on the >>> performance >>> differences. Obviously they're all different. Maybe substantially >>> different. >> >> It depends on what kind of performance you are talking about. >> All NICs are capable of doing linerate RX/TX for both small/big packets. > > this is actually not true. I have direct experience with Intel, > Mellanox and Broadcom, > and small packets are a problem across the board even with 1 port. > > From my experience only intel can do line rate (14.88Mpps) with 64-byte frames, > but suffers a bit with sizes that are not multiple of 64. > Mellanox peaks at around 7Mpps. > Broadcom is limited to some 2.5Mpps. Wow. So I'm wrong. However, Chelsio T4 can do at least ~8/port. I'll check for full linerate and report. > This is all with netmap, using the regular stack you are going to see > much much less. > > Large frames (1400+) are probably not a problem for anyone, but since the > original post asked for packet capture, i thought the small-frame case > is a relevant one. > >> The only notable exception I;m aware of are Intel 82598-based NICs which >> advertise PCI-E X8 gen2 with _2.5GT_ link speed, giving you maximum >> ~14Gbit/s bw for 2 ports instead of 20. > > This makes me curious because i believe people have used netmap with > the 82598 and achieved close to line rate even with 64-byte frames/one port, > and i thought (maybe I am wrong ?) the various 2-port NICs use 4 lanes per port. > So the number i remember does not match with your quote of 2.5Gt/s. > Are all 82598 using 2.5GT/s (which is a gen.1 speed) instead of 5 ? Quoting 82598EB datasheet: The PCIe v2.0 (2.5 GT/s) interface is used by the 82598EB as a host interface. It supports x8, x4, x2 and x1 configurations at a speed of 2.5 GHz. The maximum aggregated raw ban.. Actually I discovered this exactly with netmap and 82598*-DA2 NIC :) > > cheers > luigi > From owner-freebsd-net@FreeBSD.ORG Sat Jul 27 13:30:02 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C4756FC5 for ; Sat, 27 Jul 2013 13:30:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 97A4F263C for ; Sat, 27 Jul 2013 13:30:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6RDU1bb048791 for ; Sat, 27 Jul 2013 13:30:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6RDU08v048790; Sat, 27 Jul 2013 13:30:00 GMT (envelope-from gnats) Date: Sat, 27 Jul 2013 13:30:00 GMT Message-Id: <201307271330.r6RDU08v048790@freefall.freebsd.org> To: freebsd-net@FreeBSD.org Cc: From: Michael Tuexen Subject: Re: kern/180873: [sctp] SCTP connection hangs on COOKIE_ECHOED X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Michael Tuexen List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2013 13:30:02 -0000 The following reply was made to PR kern/180873; it has been noted by GNATS. From: Michael Tuexen To: bug-followup@FreeBSD.org, jau@iki.fi Cc: Subject: Re: kern/180873: [sctp] SCTP connection hangs on COOKIE_ECHOED Date: Sat, 27 Jul 2013 15:23:41 +0200 Which addresses are you binding? Are the programs you use available? I'm pretty sure the problem is related to the address scopes in IPv6. I haven't done testing link local addresses at all, I think. Best regards Michael From owner-freebsd-net@FreeBSD.ORG Sat Jul 27 16:22:42 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2F0131A2 for ; Sat, 27 Jul 2013 16:22:42 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 91EC02B6F for ; Sat, 27 Jul 2013 16:22:41 +0000 (UTC) Received: (qmail 69107 invoked from network); 27 Jul 2013 17:10:12 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 27 Jul 2013 17:10:12 -0000 Message-ID: <51F3F3BA.20604@freebsd.org> Date: Sat, 27 Jul 2013 18:22:18 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: "Alexander V. Chernikov" Subject: Re: Recommendations for 10gbps NIC References: <51F0386D.2000709@acm.poly.edu> <51F16A07.9030505@FreeBSD.org> <1374852658.90079.YahooMailNeo@web121601.mail.ne1.yahoo.com> <51F2A313.9070105@FreeBSD.org> <1374876857.42890.YahooMailNeo@web121603.mail.ne1.yahoo.com> <51F37E97.3090203@FreeBSD.org> <51F387E8.6090704@FreeBSD.org> In-Reply-To: <51F387E8.6090704@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-net@freebsd.org" , Luigi Rizzo X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2013 16:22:42 -0000 On 27.07.2013 10:42, Alexander V. Chernikov wrote: > On 27.07.2013 12:15, Luigi Rizzo wrote: >> On Sat, Jul 27, 2013 at 10:02 AM, Alexander V. Chernikov >> wrote: >> This makes me curious because i believe people have used netmap with >> the 82598 and achieved close to line rate even with 64-byte frames/one port, >> and i thought (maybe I am wrong ?) the various 2-port NICs use 4 lanes per port. >> So the number i remember does not match with your quote of 2.5Gt/s. >> Are all 82598 using 2.5GT/s (which is a gen.1 speed) instead of 5 ? > > Quoting 82598EB datasheet: > The PCIe v2.0 (2.5 GT/s) interface is used by the 82598EB as a host interface. It supports x8, x4, > x2 and x1 configurations at a speed of 2.5 GHz. The maximum aggregated raw ban.. > > Actually I discovered this exactly with netmap and 82598*-DA2 NIC :) Discussing the 82598 is moot because it has been replaced with the 82599 which supports x1-x8 at 5 GT/s. AFAIK you can't event buy the 82598 anymore. -- Andre From owner-freebsd-net@FreeBSD.ORG Sat Jul 27 20:49:36 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0CCEA452 for ; Sat, 27 Jul 2013 20:49:36 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-vc0-x233.google.com (mail-vc0-x233.google.com [IPv6:2607:f8b0:400c:c03::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C2DB5246C for ; Sat, 27 Jul 2013 20:49:35 +0000 (UTC) Received: by mail-vc0-f179.google.com with SMTP id ht10so1268056vcb.24 for ; Sat, 27 Jul 2013 13:49:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=+GG5LsLPu8s6xuqXkJheeyKyxgiGzfnAcJe5ktfRvy4=; b=ZpY0L151MKEpMtjaEkIN34w18pCL+/jw13rA7MedOiEPmU+PDlC7+Zl6S50XwuzsxR OZnhDvq8rsW9INuQI4+6utOTtTS7YBbmIv0JGZcBOOjxrk6hi3+83hKcew/MgMvlUabJ csseuSkpHokEfQU+wD68S9DaZkIeYZEneVM575i43BZ2Q+KM+ZRL8Uu9BMm6j0yIzR3V A/tQ9E97WaNU817fKxe9xWWtcbEMiJ6UIJ+guRtYE2C5tEAX4dH88s9OU5BEbnVxsLAV doGtHTmoA13KMM84YLkxptaJx1e3FTDuNjZcWIxOTMYRRdgMJqfEHdG0FNrRdS8Mws3A Ys5g== MIME-Version: 1.0 X-Received: by 10.58.67.9 with SMTP id j9mr23520705vet.22.1374958174869; Sat, 27 Jul 2013 13:49:34 -0700 (PDT) Received: by 10.221.22.199 with HTTP; Sat, 27 Jul 2013 13:49:34 -0700 (PDT) Date: Sat, 27 Jul 2013 16:49:34 -0400 Message-ID: Subject: Please implement patch in PR180893 From: Zaphod Beeblebrox To: FreeBSD Net Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2013 20:49:36 -0000 I'd like to advocate implementing http://www.freebsd.org/cgi/query-pr.cgi?pr=180893 Quoting the PR: Some errant network equipment (including the simulation of a network by VMware, as an example) will reflect back multicast packets to the sender. This breaks protocols such as DAD and makes IPv6 nearly impossible to use on these networks. Now, the argument could be made to fix these network elements, but there is an elegant solution that improves the quality of FreeBSD: To refuse packets that have a source ethernet address of the receiving interface. If you consider this notion, you can quickly and easily accept that an interface should never "receive" a packet from it's own MAC address. This behaviour mirrors Linux behavior and I assume Windows behavior. I won't claim to be experienced in kernel matters, but I chose the location for this modification to allow BPF to "see" the packets (for network diagnosis). This test, however, could be moved within this function or even given a sysctl knob. From owner-freebsd-net@FreeBSD.ORG Sat Jul 27 21:55:31 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D6342575 for ; Sat, 27 Jul 2013 21:55:31 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-x233.google.com (mail-wg0-x233.google.com [IPv6:2a00:1450:400c:c00::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6DAE32698 for ; Sat, 27 Jul 2013 21:55:31 +0000 (UTC) Received: by mail-wg0-f51.google.com with SMTP id a12so65330wgh.6 for ; Sat, 27 Jul 2013 14:55:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=vUWoCAF+mEhvswJmQE6KhleIVkyYHiX6GTDvFtKvNoc=; b=FULsxOn/qCPuXzSoviBPINSg3TePhx14pBjHqvDzLfK0CsO7/feX0YyNTczbl5yWIZ c3VwBK6nmQS3jgSS8c6pCMTX+L39gCVjMORibmI8OuabhFIXDJzbXszvff1SL3mgDdAl bFLrBVeFBp3PEWjJEnBxA6Ti8mh1PUKsxbqE/CeB21fRZV7M1nnitFU5T/IxNoSxlfDg tfA86gQP/PZELQm6UOfRdc0PHTLr4kioUYxgeU8KUxvlRSk0R3y8xL01fmOkwXRlIAaz P4rsbZNuIzK+afGrEyr+pkXdJ4tf7YWfKmOoUA0wQ4/NcsMMz9l1sshBuat7c52VUdhL E72A== MIME-Version: 1.0 X-Received: by 10.180.160.165 with SMTP id xl5mr2800612wib.46.1374962129832; Sat, 27 Jul 2013 14:55:29 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Sat, 27 Jul 2013 14:55:29 -0700 (PDT) In-Reply-To: References: Date: Sat, 27 Jul 2013 14:55:29 -0700 X-Google-Sender-Auth: L8ZMcO3JwGXGsq7nfsUL73pxY1g Message-ID: Subject: Re: Please implement patch in PR180893 From: Adrian Chadd To: Zaphod Beeblebrox Content-Type: text/plain; charset=ISO-8859-1 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jul 2013 21:55:31 -0000 Sure, but it would be nice to file bugs with VMware and such to ensure they fix their bugs. Anyone have any issue with this? The issue I have is the if_printf(), it should be rate limited at the very least. It would also be nice to have a different counter to reflect that kind of dropped packet.. 2c, adrian On 27 July 2013 13:49, Zaphod Beeblebrox wrote: > I'd like to advocate implementing > http://www.freebsd.org/cgi/query-pr.cgi?pr=180893 > > Quoting the PR: > > Some errant network equipment (including the simulation of a network > by VMware, as an example) will reflect back multicast packets to the sender. > This breaks protocols such as DAD and makes IPv6 nearly impossible to use > on these networks. > > Now, the argument could be made to fix these network elements, but > there is an elegant solution that improves the quality of FreeBSD: To refuse > packets that have a source ethernet address of the receiving interface. If > you consider this notion, you can quickly and easily accept that an > interface > should never "receive" a packet from it's own MAC address. > > This behaviour mirrors Linux behavior and I assume Windows behavior. > > I won't claim to be experienced in kernel matters, but I chose the > location for this modification to allow BPF to "see" the packets (for > network diagnosis). This test, however, could be moved within this function > or even given a sysctl knob. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"