From owner-freebsd-net@FreeBSD.ORG Fri Jan 31 23:45:12 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9F37086D for ; Fri, 31 Jan 2014 23:45:12 +0000 (UTC) Received: from mail-ee0-x22a.google.com (mail-ee0-x22a.google.com [IPv6:2a00:1450:4013:c00::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3B66F15DC for ; Fri, 31 Jan 2014 23:45:12 +0000 (UTC) Received: by mail-ee0-f42.google.com with SMTP id b15so751833eek.29 for ; Fri, 31 Jan 2014 15:45:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=ItOilkexPfeiNLizJhiOTpVezO51thfPcHeAy8iIWDI=; b=YXT0x1D56gYZM2e4kupKHP3SaRNkKBNFireiCdS+DYXjKzlfLaEs3m+19qGJSSBqjn L7bGfL0IknU8V+ljJcRkUNnPw66VBGTIwNhTZMKqeGv3dafAIB8Kgj/MB0m2wAqdPZMM 262t4qMlV1J8pyMF0lJ7AXfjf96cmgHACB+/xzLmroREH7N39grArxa5giARYNzjb4fo RRDT5vpcHoVp4QjYQ7j1UxdS9mD4GjUC34flROaScMXluO6mfQZf8NH5EkzozRbdFmE8 lrIUbFfxedYojdfEcKmO9ztDdKFyMTcGr1GLm2S33+o/Ua4xfEj3tW/ChW03Vw0W8D5g 6dyA== MIME-Version: 1.0 X-Received: by 10.14.126.9 with SMTP id a9mr5850552eei.95.1391211910555; Fri, 31 Jan 2014 15:45:10 -0800 (PST) Received: by 10.14.65.4 with HTTP; Fri, 31 Jan 2014 15:45:10 -0800 (PST) Date: Fri, 31 Jan 2014 15:45:10 -0800 Message-ID: Subject: Errors using span interface on if_bridge(4) From: hiren panchasara To: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Jan 2014 23:45:12 -0000 Below is my setup: 11.0-CURRENT FreeBSD 11.0-CURRENT #1 r260789:260806M: Thu Jan 23 21:18:08 UTC 2014 (n/w stack is untouched) ix1: flags=8943 metric 0 mtu 1500 options=8400b8 ether 38:ea:a7:8b:af:c4 inet6 fe80::3aea:a7ff:fe8b:afc4%ix1 prefixlen 64 scopeid 0x6 inet 10.73.149.91 netmask 0xffffff00 broadcast 10.73.149.255 nd6 options=29 media: Ethernet autoselect (10Gbase-Twinax ) status: active ix2: flags=8943 metric 0 mtu 1500 options=8400b8 ether 90:e2:ba:30:73:40 inet6 fe80::92e2:baff:fe30:7340%ix2 prefixlen 64 scopeid 0x7 inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255 nd6 options=29 media: Ethernet autoselect (10Gbase-Twinax ) status: active ix3: flags=8943 metric 0 mtu 1500 options=8400b8 ether 90:e2:ba:30:73:41 inet6 fe80::92e2:baff:fe30:7341%ix3 prefixlen 64 scopeid 0x8 inet 192.168.0.3 netmask 0xffffff00 broadcast 192.168.0.255 nd6 options=29 media: Ethernet autoselect (autoselect ) status: active bridge0: flags=8843 metric 0 mtu 1500 ether 02:a1:25:9a:8f:00 nd6 options=9 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: ix1 flags=143 ifmaxaddr 0 port 6 priority 128 path cost 2000 member: ix2 flags=8 ifmaxaddr 0 port 7 priority 128 path cost 2000 ix2 and ix3 are connected back to back via a cable so that I can snoop any traffic arriving on bridge0 on to ix3. I have tcpdump going on all 3 interfaces. What I am seeing is interesting when I send data to ix1 via iperf3 (iperf3 -c 10.73.149.91) . I see packets coming to ix1, getting copied to ix2 but on ix3 I only see a few packets making it successfully, for rest I see: 23:30:01.308691 IP bad-hlen 0 23:30:01.308700 IP bad-hlen 0 23:30:01.308711 IP bad-hlen 0 Failure is intermittent. Some packets get through but I see this error for others. Looking at the packet carefully, for all those packets with errors, header length for ipv4 is being reported as 0. Only other indication I could see was: -bash-4.2$ sysctl -a | grep checksum_errs dev.ix.0.mac_stats.checksum_errs: 0 dev.ix.1.mac_stats.checksum_errs: 0 dev.ix.2.mac_stats.checksum_errs: 0 dev.ix.3.mac_stats.checksum_errs: 5686743 I also disabled tso and lro on all of them. Looking at the code: if_bridge.c has bridge_span() which does m_copypacket() to span interface. 2549 mc = m_copypacket(m, M_NOWAIT); 2550 if (mc == NULL) { 2551 sc->sc_ifp->if_oerrors++; 2552 continue; 2553 } 2554 2555 bridge_enqueue(sc, dst_if, mc); Now, I am not sure if its failing at m_copypacket() or after that in bridge_enqueue(). Not sure how do I look at if_oerrors count. Any further help in debugging would be great. cheers, Hiren