From owner-freebsd-net@FreeBSD.ORG Tue Jul 12 20:23:40 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 263BF1065673; Tue, 12 Jul 2011 20:23:40 +0000 (UTC) (envelope-from davidch@broadcom.com) Received: from mms2.broadcom.com (mms2.broadcom.com [216.31.210.18]) by mx1.freebsd.org (Postfix) with ESMTP id E689A8FC12; Tue, 12 Jul 2011 20:23:39 +0000 (UTC) Received: from [10.9.200.131] by mms2.broadcom.com with ESMTP (Broadcom SMTP Relay (Email Firewall v6.3.2)); Tue, 12 Jul 2011 13:27:15 -0700 X-Server-Uuid: D3C04415-6FA8-4F2C-93C1-920E106A2031 Received: from IRVEXCHCCR01.corp.ad.broadcom.com ([10.252.49.31]) by IRVEXCHHUB01.corp.ad.broadcom.com ([10.9.200.131]) with mapi; Tue, 12 Jul 2011 13:22:25 -0700 From: "David Christensen" To: "Charles Sprickman" Date: Tue, 12 Jul 2011 13:22:14 -0700 Thread-Topic: bce packet loss Thread-Index: AcxASZhXV0vuHBD2R1y0D3Uzm2S4AwAhBZiA Message-ID: <5D267A3F22FD854F8F48B3D2B523819385F12FEBE6@IRVEXCHCCR01.corp.ad.broadcom.com> References: <20110706201509.GA5559@michelle.cdnetworks.com> <20110707174233.GB8702@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B523819385C32D96B7@IRVEXCHCCR01.corp.ad.broadcom.com> <5D267A3F22FD854F8F48B3D2B523819385F12FE86B@IRVEXCHCCR01.corp.ad.broadcom.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-cr-hashedpuzzle: EbwW H0fU JQNt KQo+ Lnx5 NgeI N3gT RPsy UwvG Wwhu adPn csZA g5wE h5Vj kSVv pOKG; 4; ZABhAHYAaQBkAGMAaABAAGYAcgBlAGUAYgBzAGQALgBvAHIAZwA7AGYAcgBlAGUAYgBzAGQALQBuAGUAdABAAGYAcgBlAGUAYgBzAGQALgBvAHIAZwA7AHAAeQB1AG4AeQBoAEAAZwBtAGEAaQBsAC4AYwBvAG0AOwBzAHAAbwByAGsAQABiAHcAYQB5AC4AbgBlAHQA; Sosha1_v1; 7; {9DCC3B7E-B2CE-4D59-AA8E-B8F479F269FD}; ZABhAHYAaQBkAGMAaABAAGIAcgBvAGEAZABjAG8AbQAuAGMAbwBtAA==; Tue, 12 Jul 2011 20:22:14 GMT;UgBFADoAIABiAGMAZQAgAHAAYQBjAGsAZQB0ACAAbABvAHMAcwA= x-cr-puzzleid: {9DCC3B7E-B2CE-4D59-AA8E-B8F479F269FD} acceptlanguage: en-US MIME-Version: 1.0 X-WSS-ID: 620271A962O24003635-01-01 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Cc: YongHyeon PYUN , "freebsd-net@freebsd.org" , David Christensen Subject: RE: bce packet loss X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jul 2011 20:23:40 -0000 > > There won't be any indication in the driver since flow control > > is managed in hardware. You'd need a wire capture to see that > > bce(4) has stopped sending frames in response to receiving an > > XOFF flow control frame or started sending frames in response > > to receiving an XON flow control frame. >=20 > Ah. I was hoping for something in the ifconfig output. I'll see if > tcpdump and wireshark can tell me anything about this host. >=20 Flow control frames are consumed by the MAC and aren't passed to the OS stack so you won't see them with a host based packet capture application like tcpdump. You'd need something like a JDSU Xgig protocol analyzer to see them on the wire. > One the one host (w/bce) I just set to full auto, the switch claims to > have negotiated 1000FD w/flow control (this specifically shows as > "auto+enabled" on the switch side). >=20 > I see that the "sysctl dev.bce.1" tree has some info, and I can see that > the NIC is receiving flow control frames: >=20 > dev.bce.1.stat_XonPauseFramesReceived: 16638 > dev.bce.1.stat_XoffPauseFramesReceived: 17239 >=20 > These lines are a bit puzzling though: >=20 > dev.bce.1.stat_FlowControlDone: 0 > dev.bce.1.stat_XoffStateEntered: 0 Auto-negotiation occurs between PHYs and then the link comes up. When link is up bce(4) will configure the MAC based on the auto-neg results returned by brgphy(4) (the switch should follow a similar procedure on its side). Flow control occurs at the MAC level. It looks like the MAC is not configured to=20 support flow control as negotiated by the PHY during auto-neg, resulting in flow control frames being ignored. The driver will set this in the MAC based on PHY results: http://fxr.watson.org/fxr/source/dev/bce/if_bce.c#L2046 Either the PHY results are coming back incorrectly or it may be firmware on bce(4) is disabling the behavior even though bce(4) correctly enables it. Dave