From owner-freebsd-bugs@FreeBSD.ORG Thu Jul 10 17:03:01 2014 Return-Path: Delivered-To: freebsd-bugs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 26106F31 for ; Thu, 10 Jul 2014 17:03:01 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0CF632ED7 for ; Thu, 10 Jul 2014 17:03:01 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.8/8.14.8) with ESMTP id s6AH30OT083528 for ; Thu, 10 Jul 2014 17:03:00 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 191786] New: bce link state changes to same state are ignored by lagg Date: Thu, 10 Jul 2014 17:03:01 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: vegeta@tuxpowered.net X-Bugzilla-Status: Needs Triage X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jul 2014 17:03:01 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191786 Bug ID: 191786 Summary: bce link state changes to same state are ignored by lagg Product: Base System Version: 9.2-RELEASE Hardware: amd64 OS: Any Status: Needs Triage Severity: Affects Only Me Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: vegeta@tuxpowered.net I have a Dell PowerEdge m610 system with 6 bce NICs. BladeCenter is equipped with M6348 switches in fabric A and pass-through modules in fabrics B and C. bce[01] are connected to switch stack via backplane and form a lagg with LACP, bce[23] are connected via pass-through module to copper ports in the same switch stack and create another lagg with LACP. bce[45] are connected elsewhere without LAG. As far as I understand an interface can change its state in lagg when it goes down or up, thanks to if_link_state_change and do_link_state_change in sys/net/if.c. if_link_state_change is equipped with a check for "changing" into the same state. I've added following printfs to see what exactly happens: --- a/sys/dev/bce/if_bce.c +++ b/sys/dev/bce/if_bce.c @@ -6460,11 +6460,23 @@ bce_phy_intr(struct bce_softc *sc) if (new_link_state) { if (bootverbose) if_printf(sc->bce_ifp, "link UP\n"); + printf("%s: state change to UP; from 0x%x to 0x%x, calling if_link_state_change(0x%x, 0x%x)\n", + sc->bce_ifp->if_xname, + old_link_state, + new_link_state, + sc->bce_ifp->if_link_state, + LINK_STATE_UP); if_link_state_change(sc->bce_ifp, LINK_STATE_UP); } else { if (bootverbose) if_printf(sc->bce_ifp, "link DOWN\n"); + printf("%s: state change to DOWN; from 0x%x to 0x%x, calling if_link_state_change(0x%x, 0x%x)\n", + sc->bce_ifp->if_xname, + old_link_state, + new_link_state, + sc->bce_ifp->if_link_state, + LINK_STATE_DOWN); if_link_state_change(sc->bce_ifp, LINK_STATE_DOWN); } --- a/sys/net/if.c +++ b/sys/net/if.c @@ -1900,9 +1900,12 @@ void *(*vlan_cookie_p)(struct ifnet *); void if_link_state_change(struct ifnet *ifp, int link_state) { + log(LOG_NOTICE, "%s: link state changed to %s, scheduling do_link_state_change\n", ifp->if_xname, (link_state == LINK_STATE_UP) ? "UP" : "DOWN" ); /* Return if state hasn't changed. */ - if (ifp->if_link_state == link_state) + if (ifp->if_link_state == link_state) { + log(LOG_NOTICE, "%s: not really changed, skipping\n", ifp->if_xname); return; + } ifp->if_link_state = link_state; @@ -1921,6 +1924,8 @@ do_link_state_change(void *arg, int pending) if (ifp->if_vlantrunk != NULL) (*vlan_link_state_p)(ifp); + log(LOG_NOTICE, "%s: link state changed to %s, calling hooks\n", ifp->if_xname, (link_state == LINK_STATE_UP) ? "UP" : "DOWN" ); + if ((ifp->if_type == IFT_ETHER || ifp->if_type == IFT_L2VLAN) && IFP2AC(ifp)->ac_netgraph != NULL) (*ng_ether_link_state_p)(ifp, link_state); @@ -1928,8 +1933,10 @@ do_link_state_change(void *arg, int pending) (*carp_linkstate_p)(ifp); if (ifp->if_bridge) (*bridge_linkstate_p)(ifp); - if (ifp->if_lagg) + if (ifp->if_lagg) { + log(LOG_NOTICE, "%s: hook link_state\n", ifp->if_xname); (*lagg_linkstate_p)(ifp, link_state); + } if (IS_DEFAULT_VNET(curvnet)) devctl_notify("IFNET", ifp->if_xname, As howtos and manuals suggest, my rc.conf contains ifconfig_bce[01234]="up", this causes the following thing to be logged: Jul 10 17:18:55 aw19lb2 kernel: bce0: state change to UP; from 0x0 to 0x1, calling if_link_state_change(0x0, 0x2) Jul 10 17:18:55 aw19lb2 kernel: bce1: state change to UP; from 0x0 to 0x1, calling if_link_state_change(0x0, 0x2) Jul 10 17:18:55 aw19lb2 kernel: bce2: state change to UP; from 0x0 to 0x1, calling if_link_state_change(0x0, 0x2) Jul 10 17:18:55 aw19lb2 kernel: bce3: state change to UP; from 0x0 to 0x1, calling if_link_state_change(0x0, 0x2) Later on laggs get their ports added: ifconfig_lagg1_name="internal" ifconfig_internal="laggproto lacp laggport bce2 laggport bce3" This causes following lacp debug messages: Jul 10 17:18:55 aw19lb2 kernel: bce2: media changed 0x0 -> 0x22, ether = 1, fdx = 0, link = 1 Jul 10 17:18:55 aw19lb2 kernel: bce2: partner timeout changed Jul 10 17:18:55 aw19lb2 kernel: bce2: LACP_STATE_AGGREGATION==0, key is 8003 Jul 10 17:18:55 aw19lb2 kernel: bce2: -> UNSELECTED Jul 10 17:18:55 aw19lb2 kernel: bce3: media changed 0x0 -> 0x22, ether = 1, fdx = 0, link = 1 Jul 10 17:18:55 aw19lb2 kernel: bce3: partner timeout changed Jul 10 17:18:55 aw19lb2 kernel: bce3: LACP_STATE_AGGREGATION==0, key is 8004 Jul 10 17:18:55 aw19lb2 kernel: bce3: -> UNSELECTED Ports are added, but they are missing Full Duplex for some reason, so they don't participate in lagg yet. This gets fixed later on, but not for bce3: Jul 10 17:18:57 aw19lb2 kernel: bce3: state change to UP; from 0x0 to 0x1, calling if_link_state_change(0x2, 0x2) Jul 10 17:18:57 aw19lb2 kernel: bce3: link state changed to UP, scheduling do_link_state_change Jul 10 17:18:57 aw19lb2 kernel: bce3: not really changed, skipping bce3 performs change from down to up, but there was never change to down, so if_link_state_change ignores the change and lagg stuff is never called. For bce2 this looks different, there is a real change to down, and then up: Jul 10 17:18:59 aw19lb2 kernel: bce2: state change to DOWN; from 0x1 to 0x0, calling if_link_state_change(0x2, 0x1) Jul 10 17:18:59 aw19lb2 kernel: bce2: link state changed to DOWN, scheduling do_link_state_change Jul 10 17:18:59 aw19lb2 kernel: bce2: link state changed to DOWN, calling hooks Jul 10 17:18:59 aw19lb2 kernel: bce2: hook link_state Jul 10 17:18:59 aw19lb2 kernel: bce2: media changed 0x22 -> 0x100630, ether = 1, fdx = 1, link = 0 Jul 10 17:18:59 aw19lb2 kernel: bce2: LACP_STATE_AGGREGATION==0, key is 8003 Jul 10 17:18:59 aw19lb2 kernel: bce2: link state changed to DOWN # end of do_link_state_change Jul 10 17:18:59 aw19lb2 kernel: bce2: state change to UP; from 0x0 to 0x1, calling if_link_state_change(0x1, 0x2) Jul 10 17:18:59 aw19lb2 kernel: bce2: link state changed to UP, scheduling do_link_state_change Jul 10 17:18:59 aw19lb2 kernel: bce2: link state changed to UP, calling hooks Jul 10 17:18:59 aw19lb2 kernel: bce2: hook link_state Jul 10 17:18:59 aw19lb2 kernel: bce2: media changed 0x100630 -> 0x100630, ether = 1, fdx = 1, link = 1 Jul 10 17:18:59 aw19lb2 kernel: bce2: key is 20b Jul 10 17:18:59 aw19lb2 kernel: bce2: -> UNSELECTED Jul 10 17:18:59 aw19lb2 kernel: bce2: link state changed to UP Jul 10 17:18:59 aw19lb2 kernel: bce2: lacpdu receive Jul 10 17:18:59 aw19lb2 kernel: bce2: lacp_sm_rx_update_ntt: assert ntt Jul 10 17:18:59 aw19lb2 kernel: bce2: old pstate 38 Jul 10 17:18:59 aw19lb2 kernel: bce2: new pstate c5 Jul 10 17:18:59 aw19lb2 kernel: bce2: lacpdu transmit In the meantime there are multiple messages "bce[23]: lacpdu receive", so both links are really up and are receiving packets from switch. My system is patched with patch from http://lists.freebsd.org/pipermail/freebsd-net/2013-February/034649.html, otherwise there are even more problems, as bce[01] are show to be connected to different media, even though both of them are connected directly to switch via Blade Centre's backplane. Generally so strict checking for media type forbids creating a 4-port lagg for such network configuration as I have, where some ports would use direct backplane connection and some extra copper cables, but this is yet another topic... [root@aw19lb2 ~]# ifconfig -m bce0 bce0: flags=8843 metric 0 mtu 1500 options=800bb capabilities=800bb ether c8:1f:66:9d:1b:88 media: Ethernet autoselect (1000baseSX ) status: active supported media: media autoselect media 1000baseSX mediaopt full-duplex media 1000baseSX [root@aw19lb2 ~]# [root@aw19lb2 ~]# ifconfig -m bce1 bce1: flags=8843 metric 0 mtu 1500 options=800bb capabilities=800bb ether c8:1f:66:9d:1b:88 media: Ethernet autoselect (1000baseT ) status: active supported media: media autoselect media 1000baseSX mediaopt full-duplex media 1000baseSX -- You are receiving this mail because: You are the assignee for the bug.