From owner-freebsd-net@FreeBSD.ORG Thu Apr 2 11:01:54 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0D5DAA43 for ; Thu, 2 Apr 2015 11:01:54 +0000 (UTC) Received: from kerio.tuxis.nl (alcyone.saas.tuxis.net [31.3.111.19]) (using TLSv1.1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7E42CF83 for ; Thu, 2 Apr 2015 11:01:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=tuxis.nl; s=mail; h=from:subject:date:message-id:to:mime-version:content-type:in-reply-to; bh=bmskgIuzIhTFXKP7htiPQKUsetpDqFscS6GG7MUF+NM=; b=RYzEaqYZGtAaPI/FJlwxAwYXrImj6Pw8zzPV/s0xUWb8f0VGGoXRwoquoek6q4mXp3QcfAYMTmEBu v6KIjGjyxf/G1cMSzZZ38+0FTSk4iOAexGDD6C9h6U4K/Wmm8QZOEW4a0GVBQ7dGfwHutTQ2UfQpFx neIYNjPyUNK3/AOE4vcZF3TTgv7QO5Wt/iGO9D+YYbNyeqyRQ0SaH1Nrci6EpRsIRjpbZ+U+IpGW6H LHbcFVV/eEC00iW3HnBJDNp0Wrb51uJieg0VKc7D/IxtbTi9LOI+due/qYa1C1gJdFqHwCmQ2+1M3E R/3vVqI/qkx8kXAjQsU/a5EHIXtpTFA== X-Footer: dHV4aXMubmw= Received: from [31.3.104.222] ([31.3.104.222]) by kerio.tuxis.nl (Kerio Connect 8.5.0 beta 1) for freebsd-net@freebsd.org; Thu, 2 Apr 2015 13:01:42 +0200 Date: Thu, 2 Apr 2015 13:01:42 +0200 Subject: Re: Urgent matter: LACP trunk stops distributing X-Mailer: Kerio Connect 8.5.0 beta 1/Kerio Connect Client X-User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36 Message-ID: <2042783266-15365@kerio.tuxis.nl> X-Priority: 3 Importance: Normal In-Reply-To: <1953214074-10699@kerio.tuxis.nl> MIME-Version: 1.0 MIME-Version: 1.0 MIME-Version: 1.0 MIME-Version: 1.0 From: Mark Schouten To: freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg="sha1"; boundary="=-lq2bHpg3wDF3v+pWDHkm" X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Apr 2015 11:01:54 -0000 --=-lq2bHpg3wDF3v+pWDHkm Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, So I've changed the configuration last night, eliminating the possibility t= hat the issue was LACP. All traffic was handled by the single igb interface= . It didn't help, at all. This morning the interface stopped working, without any form of messages an= d started working again after a few minutes. Then, some minutes later it st= opped again, to resume after a few minutes. So now I plugged the switch into an em-port and I tried to move the vlan in= terfaces to the em-interface. That worked, partially. The server kept the o= ld mac-address in the arp-table. Even after I deleted the vlan interface. I rebooted the box, and so far so good. But still, any help is appreciated. Met vriendelijke groeten, --=C2=A0 Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK:=C2=A061527076=C2=A0| http://www.tuxis.nl/ T: 0318 200208 | info@tuxis.nl Van: Mark Schouten =20 Aan: =20 Verzonden: 1-4-2015 12:07=20 Onderwerp: Urgent matter: LACP trunk stops distributing=20 Hi,=20 =20 I've got an urgent matter on my hands, my FreeBSD iScsibox has dropped it's= LACP trunk twice in the last 24 hours, which causes a whole lot of issues = on my VPS cluster, using that storage. =20 =20 It's a FreeBSD box with two igb-interfaces in an LACP trunk with jumboframe= s and vlans. If the connection drops, the only thing I see is:=20 =20 Apr=C2=A0 1 10:37:18 zstore02 kernel: igb0: Interface stopped DISTRIBUTING,= possible flapping=20 Apr=C2=A0 1 10:37:28 zstore02 kernel: igb1: Interface stopped DISTRIBUTING,= possible flapping=20 =20 The switch has nothing to say, so it doesn't seem that there are acual link= flaps. I have a feeling that it's this LACP trunk that goes haywire somewhe= re on the FreeBSD side. If anyone can help me with this, I'd be grateful. I= f someone (preferably EU-based) wants to help out professionally as a consu= ltant, that would be fine too. Or if anyone knows someone in .nl that offer= s this level of expertise on consultingbase, let me know.=20 =20 I'm running 10.1-RELEASE on the box, see the (AFAICS) relevant config below= . The switch is a Brocade ICX6430.=20 =20 [root@zstore02 ~]# ifconfig lagg0=20 lagg0: flags=3D8843 metric 0 mtu 90= 00=20 =C2=A0=C2=A0=C2=A0 options=3D4019b=20 =C2=A0=C2=A0=C2=A0 ether 0c:c4:7a:12:13:dc=20 =C2=A0=C2=A0=C2=A0 nd6 options=3D29=20 =C2=A0=C2=A0=C2=A0 media: Ethernet autoselect=20 =C2=A0=C2=A0=C2=A0 status: active=20 =C2=A0=C2=A0=C2=A0 laggproto lacp lagghash l2,l3,l4=20 =C2=A0=C2=A0=C2=A0 laggport: em1 flags=3D0<>=20 =C2=A0=C2=A0=C2=A0 laggport: em0 flags=3D0<>=20 =C2=A0=C2=A0=C2=A0 laggport: igb1 flags=3D1c=20 =C2=A0=C2=A0=C2=A0 laggport: igb0 flags=3D1c=20 =20 [root@zstore02 ~]# sysctl -a | grep lacp=20 net.link.lagg.lacp.debug: 0=20 net.link.lagg.0.lacp.lacp_strict_mode: 1=20 net.link.lagg.0.lacp.debug.rx_test: 0=20 net.link.lagg.0.lacp.debug.tx_test: 0=20 =20 =20 =20 Met vriendelijke groeten,=20 =20 --=C2=A0=20 Kerio Operator in de Cloud? https://www.kerioindecloud.nl/=20 Mark Schouten =C2=A0| Tuxis Internet Engineering=20 KvK:=C2=A061527076=C2=A0| http://www.tuxis.nl/=20 T: 0318 200208 | info@tuxis.nl = --=-lq2bHpg3wDF3v+pWDHkm Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Disposition: attachment; filename="smime.p7s" Content-Description: Electronic Signature S/MIME Content-Transfer-Encoding: base64 MIIRwQYJKoZIhvcNAQcCoIIRsjCCEa4CAQExCzAJBgUrDgMCGgUAMAsGCSqGSIb3DQEHAaCCDt4w ggUbMIIEA6ADAgECAhAsv+VdGX6YsSHI/WRu2j2JMA0GCSqGSIb3DQEBBQUAMIGTMQswCQYDVQQG EwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYD VQQKExFDT01PRE8gQ0EgTGltaXRlZDE5MDcGA1UEAxMwQ09NT0RPIENsaWVudCBBdXRoZW50aWNh dGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBMB4XDTE0MDcwMTAwMDAwMFoXDTE1MDcwMTIzNTk1OVow HjEcMBoGCSqGSIb3DQEJARYNbWFya0B0dXhpcy5ubDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC AQoCggEBANHu3SxlMZOG5GA0/mqtRXR1QmWwhUXzmCIprI0IPtSBWSA31YBJ5qcmXRhLzaiTB3Fr UpIGkW5aAZnDms9DD64kasF3oZE00Fvfnj/BDGbw098px1PukKfg4hasbTaELAjQTSUj8xRSHzKk VVynvLA/YmyRT/+u3ueK4wdaxcej241xH6mNfZeiKMAvbkv6Tm9vdup0BtqqbRSKcnc01KKrspun Eh73jLIUhP21uJv8vuOTxS1I9zJSlhIcMCEapjBQ+26cQl+s+qBuAs/LP3UPVytSbxvicdhDxtqH npN2h3jJ/+J86zGfQi8bG7EamsULTGHkbgIJL9AdKZRgAKECAwEAAaOCAd0wggHZMB8GA1UdIwQY MBaAFHoTTgB0W8Z4Y2QnwS/ioFu8ecV7MB0GA1UdDgQWBBR+dMZE22X/SMCyTxYzIP+NMZBNXTAO BgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADAgBgNVHSUEGTAXBggrBgEFBQcDBAYLKwYBBAGy MQEDBQIwEQYJYIZIAYb4QgEBBAQDAgUgMEYGA1UdIAQ/MD0wOwYMKwYBBAGyMQECAQEBMCswKQYI KwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQvQ1BTMFcGA1UdHwRQME4wTKBKoEiG Rmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9ET0NsaWVudEF1dGhlbnRpY2F0aW9uYW5kU2Vj dXJlRW1haWxDQS5jcmwwgYgGCCsGAQUFBwEBBHwwejBSBggrBgEFBQcwAoZGaHR0cDovL2NydC5j b21vZG9jYS5jb20vQ09NT0RPQ2xpZW50QXV0aGVudGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNy dDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29tb2RvY2EuY29tMBgGA1UdEQQRMA+BDW1hcmtA dHV4aXMubmwwDQYJKoZIhvcNAQEFBQADggEBAIB8FhqaML1EzfvgNwwHDC3k0ICeMerOncgee6uJ KLxwU2mstttX5jtAmgK9RuDOu+TrMkkpF2yxYMTPpSM8nL7r+N/kdogu5Bustol8WTsW1e5vs+Nh hJYFORk113ouur1kSjXuHF8TWy+/PjFJBS/xm/H+/fkghppRU+4Dj2IReUBvlexAPYr4VDxjV7AD xPOXqTQkP15LWGvhTz2YVbJ3IAVOyUNkRhr9QwzToUxXa9k/QAOpXMuvS74AT2RBV/YCEEx7ebRD MAR6lZcbYiV8sXv1ASbnMdO3Fh2F98g+5rJn5PfFH8qLpapsZx0I2/axtSG09QMDJqXd3Ab6NpEw ggUaMIIEAqADAgECAhBtGeqnGU9qMyLmIjJ6qnHeMA0GCSqGSIb3DQEBBQUAMIGuMQswCQYDVQQG EwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUg VVNFUlRSVVNUIE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQG A1UEAxMtVVROLVVTRVJGaXJzdC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsMB4XDTEx MDQyODAwMDAwMFoXDTIwMDUzMDEwNDgzOFowgZMxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVh dGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1p dGVkMTkwNwYDVQQDEzBDT01PRE8gQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1h aWwgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCShIRbS1eY1F4vi6ThQMijU1hf ZmXxMk73nzJ9VdB4TFW3QpTg+SdxB8XGaaS5MsTxQBqQzCdWYn8XtXFpruUgG+TLY15gyqJB9mrh o/+43x9IbWVDjCouK2M4d9+xF6zC2oIC1tQyatRnbyATj1w1+uVUgK/YcQodNwoCUFNslR2pEBS0 mZVZEjH/CaLSTNxS297iQAFbSGjdxUq04O0kHzqvcV8H46y/FDuwJXFoPfQP1hdYRhWBPGiLi4MP bXohV+Y0sNsyfuNK4aVScmQmkU6lkg//4LFg/RpvaFGZY40ai6XMQpubfSJj06mg/M6ekN9EGfRc WzW6FvOnm//BAgMBAAGjggFLMIIBRzAfBgNVHSMEGDAWgBSJgmd9xJ0mcABLtFBIfN49rgRufTAd BgNVHQ4EFgQUehNOAHRbxnhjZCfBL+KgW7x5xXswDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQI MAYBAf8CAQAwEQYDVR0gBAowCDAGBgRVHSAAMFgGA1UdHwRRME8wTaBLoEmGR2h0dHA6Ly9jcmwu dXNlcnRydXN0LmNvbS9VVE4tVVNFUkZpcnN0LUNsaWVudEF1dGhlbnRpY2F0aW9uYW5kRW1haWwu Y3JsMHQGCCsGAQUFBwEBBGgwZjA9BggrBgEFBQcwAoYxaHR0cDovL2NydC51c2VydHJ1c3QuY29t L1VUTkFkZFRydXN0Q2xpZW50X0NBLmNydDAlBggrBgEFBQcwAYYZaHR0cDovL29jc3AudXNlcnRy dXN0LmNvbTANBgkqhkiG9w0BAQUFAAOCAQEAhda+eFdVbTN/RFL+QtUGqAEDgIr7DbL9Sr/2r0FJ 9RtaxdKtG3NuPukmfOZMmMEwKN/L+0I8oSU+CnXW0D05hmbRoZu1TZtvryhsHa/l6nRaqNqxwPF1 ei+eupN5yv7ikR5WdLL4jdPgQ3Ib7Y/9YDkgR/uLrzplSDyYPaUlv73vYOBJ5RbI6z9Dg/Dg7g3B 080zX5vQvWBqszv++tTJOjwf7Zv/m0kzvkIpOYPuM2kugp1FTahp2oAbHj3SGl18R5mlmwhtEpmG 1l1XBxunML5LSUS4kH7K0Xk467Qz+qA6XSZYnmFVGLQh1ZnV4ENAQjC+6qXnlNKw/vN1+X9u5zCC BJ0wggOFoAMCAQICEDQ96SusJzT/j8s0lPvMcFQwDQYJKoZIhvcNAQEFBQAwbzELMAkGA1UEBhMC U0UxFDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5hbCBUVFAg TmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9vdDAeFw0wNTA2MDcwODA5 MTBaFw0yMDA1MzAxMDQ4MzhaMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcT DlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxITAfBgNVBAsT GGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJzdC1DbGllbnQg QXV0aGVudGljYXRpb24gYW5kIEVtYWlsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA sjmFpPJ9q0E7YkY3rs3BYHW8OWX5ShpHornMSMxqmNVNNRm5pELlzkniii8efNIxB8dOtINknS4p 1aJkxIW9hVE1eaROaJB7HHqkkqgX8pgV8pPMyaQylbsMTzC9mKALi+VuG6JG+ni8om+rWV6lL8/K 2m2qL+usobNqqrcuZzWLeeEeaYji5kbNoKXqvgvOdjp6Dpvq/NonWz1zHyLmSGHGTPNpsaguG7bU MSAsvIKKjqQOpdeJQ/wWWq8dcdcRWdq6hw2v+vPhwvCkxWeM1tZUOt4KpLoDd7NlyP0e03RiqhjK aJMeoYV+9Udly/hNVyh00jT/MLbu9mIwFIws6wIDAQABo4H0MIHxMB8GA1UdIwQYMBaAFK29mHo0 tCb3+sQmVO8DveAky1QaMB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNVHQ8BAf8E BAMCAQYwDwYDVR0TAQH/BAUwAwEB/zARBgNVHSAECjAIMAYGBFUdIAAwRAYDVR0fBD0wOzA5oDeg NYYzaHR0cDovL2NybC51c2VydHJ1c3QuY29tL0FkZFRydXN0RXh0ZXJuYWxDQVJvb3QuY3JsMDUG CCsGAQUFBwEBBCkwJzAlBggrBgEFBQcwAYYZaHR0cDovL29jc3AudXNlcnRydXN0LmNvbTANBgkq hkiG9w0BAQUFAAOCAQEAAbyc42MosPMxAcLfe91ioAGdIzEPnJJzU1HqH0z61p/Eyi9nfngzD3QW uZGHkfWKJvpkcADYHvkLBGJQh5OB1Nr1I9s0u4VWtHA0bniDNx6FHMURFZJfhxe9rGr98cLRzIlf sXzwPlHyNfN87GCYazor4O/fs32G67Ub9VvsonyYE9cAULnRLXPeA3h04QWFMV7LmrmdlMa5lDd1 ctxE+2fo8PolHlKn2iXpR+CgxzygTrEKNvt3SJ/vl4r7tP7jlBSog7xcLT/SYHFg7sJxggzpiDbj 2iC0o6BsqpZLuICOdcpJB/Y7FLrf3AXZn9vgsuZNoHgm5+ctbn9fxh6IFTGCAqswggKnAgEBMIGo MIGTMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdT YWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDE5MDcGA1UEAxMwQ09NT0RPIENsaWVu dCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhAsv+VdGX6YsSHI/WRu2j2JMAkG BSsOAwIaBQCggdgwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTUw NDAyMTEwMTQyWjAjBgkqhkiG9w0BCQQxFgQUBzMjEFFutXYkJ4baYIyiV2MNDBwweQYJKoZIhvcN AQkPMWwwajALBglghkgBZQMEASowCwYJYIZIAWUDBAEWMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0D BzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgw DQYJKoZIhvcNAQEBBQAEggEASXS6OF4pnKqNvKEx9qZ6Z4er059azmik5Hv68oYbbD8Q+QGjY/Mp HubV4hHXIfElA1F2REuGbkZ0S0ZWtvGn/sBWiFuZakokaCKGFfs9nDAoUB1WEFUj3vVKr0przRBj emeyOWZczeeUOhKLdMIGg64RrZQ+CKOtm8byCn2TAPbYJJU8BpfKh9CkpXJy+Xtes+ZCF+znuops 7bGmPmpShoh7QPKhenFLqBbeToF8zzJkZbPF5LiQAjxAbxlP9pBmuIUCp0y4X7xtQ7OqkBhWb0Bk sK3g70hYUakcZDOOAszH46MYZM7KZ+zr0HlncRsaPJMoR8RZFDrhSBGkzWrZrg== --=-lq2bHpg3wDF3v+pWDHkm--