From owner-freebsd-net@FreeBSD.ORG Mon Mar 23 16:03:08 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 082D1EA6 for ; Mon, 23 Mar 2015 16:03:08 +0000 (UTC) Received: from nef2.ens.fr (nef2.ens.fr [129.199.96.40]) by mx1.freebsd.org (Postfix) with ESMTP id AE468242 for ; Mon, 23 Mar 2015 16:03:07 +0000 (UTC) Received: from biologie.ens.fr (milda.ens.fr [129.199.18.219]) by nef2.ens.fr (8.13.6/1.01.28121999) with ESMTP id t2NFWuZD056462 ; Mon, 23 Mar 2015 16:32:56 +0100 (CET) X-Envelope-To: freebsd-net@freebsd.org Received: from localhost (av3.biologie.ens.fr [129.199.21.124]) by biologie.ens.fr (Postfix) with ESMTP id 8BA285E3; Mon, 23 Mar 2015 16:32:56 +0100 (CET) X-Virus-Scanned: spam & virus filtering at av3.ens.fr X-Spam-Flag: NO X-Spam-Score: -10.91 X-Spam-Level: X-Spam-Status: No, score=-10.91 tagged_above=-9999 required=5 tests=[ALL_TRUSTED=-1, AUTHD_RELAY=-8, BAYES_00=-1.9, T_RP_MATCHES_RCVD=-0.01] autolearn=ham Received: from biologie.ens.fr ([IPv6:::ffff:129.199.18.219]) by localhost (av3.biologie.ens.fr [::ffff:129.199.21.124]) (amavisd-new, port 10024) with ESMTP id 1glhmty91dcj; Mon, 23 Mar 2015 16:32:56 +0100 (CET) Received: from [129.199.16.44] (hades.ens.fr [129.199.16.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: phnguyen) by biologie.ens.fr (Postfix) with ESMTPSA id 314404B; Mon, 23 Mar 2015 16:32:56 +0100 (CET) Message-ID: <55103227.1090005@biologie.ens.fr> Date: Mon, 23 Mar 2015 16:32:55 +0100 From: Phi-Phong NGUYEN User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Carp unpingable on vlan interfaces with lagg on emulex cards Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nef2.ens.fr [129.199.96.32]); Mon, 23 Mar 2015 16:32:56 +0100 (CET) Cc: =?UTF-8?B?ImNvY2ggPj4gT2xpdmllciBDb2NoYXJkLUxhYmLDqSI=?= , sysinfo , pierre vincens X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2015 16:03:08 -0000 I have two routers running freeBSD 10.1-RELEASE with 10gigaethernet Emulex cards (OCm14104-UX-D + OCe14102‑UX-D) On each router : ifconfig_lagg2="laggproto lacp laggport oce0 laggport oce1 laggport oce4 laggport oce6" Several vlans are build within the lagg : On router1 : [...] cloned_interfaces=".... vlan3101 ..." ifconfig_vlan3101="vlan 3101 vlandev lagg2 X.X.X.252/24" ifconfig_vlan3101_alias0="vhid 121 advskew 100 pass mypass X.X.X.254/32" [...] vlan3101: flags=8943 metric 0 mtu 1500 options=303 ether 00:90:fa:xx:xx:xx inet X.X.X.252 netmask 0xffffff00 broadcast X.X.X.255 inet X.X.X.254 netmask 0xffffffff broadcast X.X.X.254 vhid 121 nd6 options=29 media: Ethernet autoselect status: active vlan: 3101 parent interface: lagg2 carp: BACKUP vhid 121 advbase 1 advskew 100 On router2 : [...] cloned_interfaces=".... vlan3101 ..." ifconfig_vlan3101="vlan 3101 vlandev lagg2 X.X.X.252/24" ifconfig_vlan3101_alias0="vhid 121 pass mypass X.X.X.254/32" [...] vlan3101: flags=8943 metric 0 mtu 1500 options=303 ether 00:90:fa:xx:xx:xx inet X.X.X.253 netmask 0xffffff00 broadcast X.X.X.255 inet X.X.X.254 netmask 0xffffffff broadcast X.X.X.254 vhid 121 nd6 options=29 media: Ethernet autoselect status: active vlan: 3101 parent interface: lagg2 carp: MASTER vhid 121 advbase 1 advskew 0 So, router2 is the master VRRP for vlan3101, ang changing advskew to 150 on router2 leaves router1 become the master. BUT : Router1 can ping X.X.X.253 and router2 is able to ping X.X.X.252, alas, router1 can't ping X.X.X.254 (the virtual ip address) as a host belonging to the 3101 vlan. The arp request does work : router1 # ping X.X.X.254 <- no "Host is down", no unreachable router1 # arp -n X.X.X.254 ? (X.X.X.254) at 00:00:5e:00:01:79 on vlan3101 expires in 1129 seconds [vlan] Already try : 1) Reduce the lagg to one single interface -> Doesn't work 2) Create the vlans on a physical emulex interface -> Doesn't work 3) Downgrade to 9.3-RELEASE with old syntax carp -> Doesn't work 4) Upgrade all the firmwares including Emulex cards. 5) Create the lagg on a physical copper interface (Intel X540) -> Does work !! So, my questions are : 1) Is it related to the Emulex cards ? 2) Is there a workaround for this sort of problem ? 2) If I replace some Emulex cards with Intel ones (Say X520 items), is the result guaranteed ? At $ 700-$ 800 each, this is a crucial question. More information : router1 # sysctl -a |grep carp net.inet.carp.allow: 1 net.inet.carp.preempt: 1 net.inet.carp.log: 1 net.inet.carp.demotion: 0 net.inet.carp.senderr_demotion_factor: 0 net.inet.carp.ifdown_demotion_factor: 240 Thanks in advance. -- Phi-Phong NGUYEN Service informatique Institut de Biologie ENS 46 rue d'Ulm 75230 PARIS CEDEX 05 Tel: 01 44 32 36 34 Fax: 01 44 32 36 30