From owner-freebsd-net@freebsd.org Fri Apr 14 15:11:08 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0F360D3D15F for ; Fri, 14 Apr 2017 15:11:08 +0000 (UTC) (envelope-from Joe@stream-technologies.com) Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01on0083.outbound.protection.outlook.com [104.47.2.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 67B6AC74 for ; Fri, 14 Apr 2017 15:11:06 +0000 (UTC) (envelope-from Joe@stream-technologies.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=streamtechnologiesuk.onmicrosoft.com; s=selector1-streamtechnologies-com01e; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=bHlAdaMAXUeg9TjnxkkBFPjNlhZ2eh/yUS/MqbeD1v0=; b=nK3/NIiurj3O+8T7bbYzcv2ci9WY5y4rBR9tZDBEh4zUhcueR7o23GsZ171jQHa1gsHLouTuwwFUHncGMsAjPeoNDs8Kwup6vlfIXmQhdC+/k2x6FTi5vsVwBvWk4lNSP+wMvtvdM95TskmdF5WKvKSgAYFKIKIA6LKcrbGq5jw= Authentication-Results: freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=stream-technologies.com; Received: from [192.168.6.128] (212.20.240.118) by AM2PR07MB1028.eurprd07.prod.outlook.com (2a01:111:e400:8444::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1034.5; Fri, 14 Apr 2017 15:11:04 +0000 Subject: Re: cxgbe netmap promiscuous mode? To: Navdeep Parhar References: <58D3C6F4.6010500@stream-technologies.com> <58D521C0.1000804@stream-technologies.com> CC: "freebsd-net@freebsd.org" From: Joe Jones Message-ID: <58F0E683.7050806@stream-technologies.com> Date: Fri, 14 Apr 2017 16:10:59 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [212.20.240.118] X-ClientProxiedBy: DB6P192CA0019.EURP192.PROD.OUTLOOK.COM (2603:10a6:4:b8::29) To AM2PR07MB1028.eurprd07.prod.outlook.com (2a01:111:e400:8444::25) X-MS-Office365-Filtering-Correlation-Id: 98a19547-8965-4014-b2aa-08d483487862 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(201703131423075)(201703031133081); SRVR:AM2PR07MB1028; X-Microsoft-Exchange-Diagnostics: 1; AM2PR07MB1028; 3:yBlfTQA9mNvcPHvjfKKGqBt/G+TJ0xhtgcGb1PaPJy47wGaNOMRKPTlDPNrVUJDDU/L38dembcp73URuUwK+ooSY4YqimgyV6AwAss6I17zBXladPONGpfIZmVelK7AsmsbonIxNCdd8L2K7E52Qlacdr+m2j/bnquhWbAmuu+pC/+PhIk1vAfmC6FE/jHUeR/PiEpamgCvdHKXFWlv7nK2YoMQk5DqqjGfyRVuxHZbqkwFp9wi9O6ShAx+HX05TtmWyr8i0XE54C2yHvyhh1Llaw68KRYzDnjSjWpEAF2nHeAAsO6wsM0OlATIcycOq+ENr604gKPh2PJ3X/fS6TQ==; 25:gU7ujKpVpn86xTkrNZU8qCEa+ondJ5IwoNKYEtBO81oTVJJhxWZvVZS/ZA3q5dDyBN/tZh9pNW6rXah5lEQWBDmhZxyqL6fqXiJntdJqgHZ717moYqfITWrl0HWv79RaFtqYZdLhHZ3A7HtygWzfp8pkJ2eCpj06hSs/4oWWdUaDgUpFcELsU5K0llTKgc9CsuTnl2HuXW+K651PlZRrH0Gde8YQq+N4mzs/UEmFnST4rdDrMF7mM2lTL6G7TYyDvd5bjMWZrrotLsKCIYaLnNalgSY5Lvjz7ByQFXtudOxvjEWb8kh3p1dIkIYrSNUsovmfuYBKAm5AiBKhnEfJQQxjgNRgl2dpIo5rB9YX27U9kCQ+e+j+OwNwPZWFJ/DbIHpiB1j+ygo/VURgfTfTiDtq7T6EUsZklH6287ubKCsp+pOOSNydWY6l6PxW9rhk2pHJN5hDQnq4bzJ2Nyjpsg== X-Microsoft-Exchange-Diagnostics: 1; AM2PR07MB1028; 31:3V2oMHVjZSm6opaVAxUsvmr7YOyd8gEgT8Vo3BhS1qMg3L5fvWRJ37cc0c1nQOdVy/jrMLieUKGJw3X5AuWph8IOijy19NLiPqXrADHxW+q78aHNWQaeZtcerSaRdZ70547++FDO8lrWthUNUHp1FmsJju9wyhtNgIcJCmWqzJd7Bpvz2MC2NatkoaG5lu+f/dMc+YaKVHElC+eJvZiqFN5GPUnerk0BLGh0jFLpIoUWxfgV1OIz+VEuGUAbydfohdQJHbnCU4zzsHtoBzWFLCiYzQyKgk0yDyywKnPrS0I= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(158342451672863)(20558992708506); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040450)(601004)(2401047)(8121501046)(5005006)(93006095)(93001095)(10201501046)(3002001)(6041248)(201703131423075)(201702281528075)(201703061421075)(20161123560025)(20161123562025)(20161123555025)(20161123564025)(6072148); SRVR:AM2PR07MB1028; BCL:0; PCL:0; RULEID:; SRVR:AM2PR07MB1028; X-Microsoft-Exchange-Diagnostics: 1; AM2PR07MB1028; 4:/qnb6tPPPrOWjQfNFpv6kPlm7enITfWX0mh+oYkWpVBDq5KoSFMRYJdfQyqF4Bi7SXyTRwZmoe2hiWrkCW/SWB4+04RbQnUyimf6sRqWJrGkpG83+/dZagRAKPrVm5d+Pg0+rc3XaU1wktgZlxlLcsddpLYXkrEIwCVuzOUKhgDD6r5u4sxUGCR2QywCPX6eXCE9d9022kf/eGpAQV7H9gChGMStgwkYLx1IEE+0QBtzdYwaGSi42nWoFAsMN24CZzow3sKYSaJUHMLljrn6itQnlUH5EpqFQGuuQqWkxTej/FHYm/3y4citrqS80YQwcIZUdvE5ATIxPNUSnCNVqRvBaOXxyrQYqs19ko7c1ehW9fC2oborvc1b9169wPyczmb+drOCuKocXz7CHj8bblq4jc5geijulnpYIqTSgVbip9Uf2IdU8Sx2INCjVb0cwA84z3Qxj4GUU3k1JGPstbOegkJ0RZ3ntwXjDZzs2yi5027zund2jg4l9sIl5OiTuJwYfLbwy8k0dXSy+uL1r52XKcRrq7w9X2QuetI08tAVZJqVrJadj2K3ex3NAyKMC8GgtTBZLqiedAu//uRDR0Hm48Qqc6XORhGDjXunyy6tcHHPQ6U78yJtNuObuIt3v43jq5L2YgEZdkdxG1bBNpV5AlWhsTcqKQDRAEYtrUSnDwnFanVFwh6RTOpO3Zpu1aTJzmWifiPqUB3/bt6EUjToaUFlHk8uZNS5PUrUyarOOWO4cY41srz+sbrpbI9UrhOdAgQt8AkjemrhJkx1uDUuv91BmINtygrWYAU9LSQ= X-Forefront-PRVS: 02778BF158 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(6049001)(39400400002)(39450400003)(39410400002)(39830400002)(377454003)(24454002)(77096006)(3480700004)(6486002)(189998001)(229853002)(59896002)(33656002)(6666003)(3846002)(80792005)(4001350100001)(65816999)(6916009)(86362001)(2950100002)(36756003)(6116002)(23676002)(76176999)(54356999)(25786009)(230700001)(7736002)(42186005)(50986999)(8676002)(4326008)(305945005)(93886004)(2906002)(50466002)(117156002)(53546009)(5660300001)(1411001)(83506001)(110136004)(38730400002)(6246003)(47776003)(53936002)(81166006)(66066001); DIR:OUT; SFP:1101; SCL:1; SRVR:AM2PR07MB1028; H:[192.168.6.128]; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtBTTJQUjA3TUIxMDI4OzIzOllHbHVHOWhoaUlmanNFd1ZxTFZoNTR3Rjdn?= =?utf-8?B?OU1ndTJHMjd4dnBPVFp6dGlMYkZzckVxTUlaOWFoUG40TkV3TjJWUFVUcEx0?= =?utf-8?B?UjdkV1pSNXE0QkppM3RSSnRrTm5yRmJ0Tk0yblRjNFM5eWdNeEhPL0NYbFgr?= =?utf-8?B?WlRVNi9oVndDNWtUa3dGeURhMC9CdnRDaURGazl1VnlnUHhBSCswVzJqS1hL?= =?utf-8?B?d1hRVjEzREk5U1RhaitiMWNaNHM5YU9hcFR5LytzcElwL1NQZkt5MDJ4TTBi?= =?utf-8?B?NDR6dWRyQzM0UHVVRndxTGdhVTBabUkzVTNDbnErQTZETkR3QTE1OU45WHJL?= =?utf-8?B?djdLK01oUGRabHNFcmtmS0tvdWdRUDdSZDI2ZXZ0NGR4QzZRUldUNnVCcGJH?= =?utf-8?B?Zm95ZnVNUjlXMS9oWldSTWFISHc1ZW1BVW9EaG41OWRkRkhUY0VlVTZYNWwr?= =?utf-8?B?cXA5eWRhV25KelJxRGFLTHA1VlpNZ3VOV1cwbU9Mc1ZMR3NmNjh3VkVkU2E1?= =?utf-8?B?OU9VVHNoM3ptbHYvS0ZYRzlLNitia20yQmVWTUluSTNTV2VHTWptb0JBTmNY?= =?utf-8?B?QlBSdmxVU2o2RWdFSCtGUW80RWFOTFdzZElRZVRnRHhKOVRGZXcvM0pBa3I5?= =?utf-8?B?aXpVRVc1elpiaFBDUVdadmhUNmNkL0tBbGZmVlkwKzNVRXNCcUIwNFBoZDVY?= =?utf-8?B?dE91MmhhODZienBFdUlncWtjZkg5c2cyd2FrSjY5NjlDalM0d1c5bTZ2ZTFJ?= =?utf-8?B?cVdKZVQ4ZW5SeEVISkFDdFJPN0Vqb2xkaXVNYmJ3VS80NUdGaG0vbkRkNTA3?= =?utf-8?B?cXpIZ2hVUzdhVkFXMUtVLzA1U242eHhnQ1VBYktlS25RUkpQd2E1cG1NUksz?= =?utf-8?B?TVZNVFV2SFpYQ3BXMkw1RktkRzJ4OHdXSk9wR252anM0K2dJcDVEWGt5MXVk?= =?utf-8?B?anh5ZmRCRlVpTE0ySE1nOE9yZ3JKZDVFQmFOTjB4K3Rtb1AwQm8xdHFQejBW?= =?utf-8?B?KzBTdmd4VkxDeHhURG9ranZDdmk1OWdsdngwMHJKdWFTZCtIOGNkNEMySDB6?= =?utf-8?B?VTlDaVhEQ0NSM2VxWE10Mng5T2VvVzBwY0RrdmxLem8vb21mWlhuQ0hRZDR2?= =?utf-8?B?UndNQlBOV3ZRT1hnSTYyNFBJb2poQ2ExakJoamRSRVh4N0hLYkNiZDNuUlFC?= =?utf-8?B?VWZ6d1NQaURySEEyc252TFlOd0ZjVmZJWTlZcW9nQ3k2VzlUc2gvSDJ1RWZ2?= =?utf-8?B?WlhHSkhKR1ZpVk9KWnhuSlI1ajQ2NHFDNGc4RzV1V2JtdlBzVlc5Q3lyV2tw?= =?utf-8?B?aFhWYzRaL2lpUWVnekNLN3AwTFA5NXQ0RDRsYkE5QUNUREo5cG0xdjVCV2pv?= =?utf-8?B?Zld3NWg2WC9rV2FqUFJKMEpxV2gxWnBQallPZW9GOWpXd3haWi8rYXNvZnpx?= =?utf-8?B?eXd5QkNqVnpMWEs4SGNMMFlMVktlSC8zRFBDOGlTZUhPZ3JvdHNXM3FhWEp5?= =?utf-8?B?cXczME9uUnpwMFJ2cURMZmQxMExPU0dvc1BoakZ4clRXUk1KNlNiblFkbjhm?= =?utf-8?B?eHgyTHBhUXJnUG9GK1BON0JoZjFTMGtPaVd2K0ZlMTViQmRuNmNiOVphUWht?= =?utf-8?B?ZTQxODZNMGFsTTdTOWdDTGhjQkVmZ05haVBjbms3eks2eStPYlZIUDdaeXFa?= =?utf-8?Q?74CrmCsTMZ8EfHokabPQqcUhXwRaAdcdxOEeewG?= X-Microsoft-Exchange-Diagnostics: 1; AM2PR07MB1028; 6:KOAMKrcgVMTVfKF6K0aFOW0y8QBcfBv3hnC1qVUOpAs3WEvhmG6bLo7dIeNDCyQUiptDTmHrk1+7aLNGOQkXHG0knKVPH9MLMvfi+ui8IUrkZPDY41ru4wOb4Htb7689AkR5DTCAhkVD/e7fdLX2WVY5NZQblkWtsaBGw3kOQWnlyWRP07/JoDcyew61bgMtrQf718Nr8x2yf0n+UgGwnL3+HBh0CIWTCJAOMsiqee9Vipv9SnANwcho2MT4fww39JhYDblhOD3FhbVhL5SI49UNp2sDAsdrLhxR6K36tfG69cTRtUvePJ8aWAJKdWARd7QF9YcL7uOfuFDBn9N6tUnVNVp84bBYP+ld+DOXVmWY9PyBuACjMW6LPJKmADrFvJ13xvEl8AZ4fTsNU44Ii3i2ctskga5Vs5ZpdF/5BFEGFahumFZIGFHiRJujMp+4TsBtYuSrURMFwkZttYChGA==; 5:eNdt5poAbAul/kMuxG0rJrk+g1r6w1+680u9Q4+g/2O3lPp1bsTVFNfks09aVDC6DW+PFK6gABie3JffLlMoPOFJk7fO9gpHzCZp+xvj4jFa0soHHVqYx6Qcgbd/PktvESflu9PJM8+JmfizE7S5+LW61JJsddbmiwlixzauY6k=; 24:S75may02Zg3kXYc3TKuTjUCw0/+SyyieFmsbjOjQRGppyEJmR0p5fAHPNoL/Fsymh1wL0VuzGQv/SZgfRwIXxquTP8fNgfxBCagU3WkFa2w= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; AM2PR07MB1028; 7:cWkdIvLzFtyiuGVcHJD0Vf8J3bE4DNgfJSUqj9QmbJTDrp8uC8gyc+5WmjBp95So9K1THE/gvziNmzlbhlw82Iu6YfVFE4lZ14nuk29RklwnrgIrOuKtDQc/FyH0dk+zdpxG7fd2kgURNH9S6y7SQLDFa3PwLdBkKHf/2AzJMlmKQUoO6fiVFEt6GbBS4giBJUSfdwuMqwH4YNPniQWbn/sFwkiy7C+4cxoFn1cyHf6DNbzrox/OThA+GkNnc8FmxsIDDM4dCM9e3jjRaXE15JkzRfr4LmadlFYbugKR8vuf6Tia+6fmPmQclUiIL1aZV3K491QJzTWoGUpjQxBbGQ== X-OriginatorOrg: stream-technologies.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2017 15:11:04.3938 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM2PR07MB1028 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Apr 2017 15:11:08 -0000 Hi Navdeep, I think I have found a driver bug. Earlier today I set up the switch I'm using so that two of the ports mirror the traffic on one of the other ports. We are planning on using a similar setup to allow packet tracing without stressing the boxes our application is running on any more then they are already. I connected both ports to one of our cxgbe cards, My intention was to use tcpdump to check that my switch config was doing what I thought it should. I ran ifconfig cxl? promisc -vlanhwtag up on both interfaces, this is what the interfaces looked like cxl0: flags=28943 metric 0 mtu 1500 options=ec07ab ether 00:07:43:33:8a:20 nd6 options=29 media: Ethernet 10Gbase-Twinax status: active vcxl0: flags=8802 metric 0 mtu 1500 options=ec07bb ether 00:07:43:33:8a:22 nd6 options=29 media: Ethernet 10Gbase-Twinax status: active cxl1: flags=28943 metric 0 mtu 1500 options=ec07ab ether 00:07:43:33:8a:28 nd6 options=29 media: Ethernet 10Gbase-Twinax status: active vcxl1: flags=8802 metric 0 mtu 1500 options=ec07bb ether 00:07:43:33:8a:2a nd6 options=29 media: Ethernet 10Gbase-Twinax status: active The interesting thing is, a tcpdump on cxl0 showed all the traffic I expected to see, while tcpdump on cxl1 showed only broadcast traffic. After playing with the switch config to make sure the difference was not on the switch I pulled both patch cables out and into another server with the same card. On the second server I could use tcpdump and see all the traffic I expected on either interface. Then back on the original server, I reloaded the device driver and tried again. Now I got only broadcast on cxl0 and cxl1. Then finally I got all the traffic to show up by doing ifconfig cxl1 -promisc ifconfig cxl1 promisc It would appearer to me that the card can get into a state where ifconfig reports that it is in promiscuous mode when it is not. Joe Jones On 24/03/17 16:44, Navdeep Parhar wrote: > On Fri, Mar 24, 2017 at 6:40 AM, Joe Jones wrote: >> Hello Navdeep, >> >> ... >> >> We were using our own MACs, we can fix the problem by using the mac from the >> vcxl interface. Should we not be able to capture all traffic on the >> interface regardless of what destination MAC it has. > You should, but you'll need to put vcxl in promiscous mode for > that. The command that you posted had cxl in promiscuous mode. As > I said earlier they share the wire but operate as independent network > interfaces otherwise. > > Regards, > Navdeep