From owner-freebsd-net@freebsd.org Fri Mar 24 13:40:26 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0CFFD14664 for ; Fri, 24 Mar 2017 13:40:26 +0000 (UTC) (envelope-from Joe@stream-technologies.com) Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-eopbgr00089.outbound.protection.outlook.com [40.107.0.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 539D11E25 for ; Fri, 24 Mar 2017 13:40:25 +0000 (UTC) (envelope-from Joe@stream-technologies.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=streamtechnologiesuk.onmicrosoft.com; s=selector1-streamtechnologies-com01e; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=an50YkUhRRQogfOM4aoWApFjJyXhPMcaIqmWTexZHzU=; b=kqF2Qe5DhoJr3VlLk2UmmH1cmaa4Da11gexRi2pBq3dwtO+73evK4Nm3ki9tuYOf14xMqP8F+DGHbsxPYSA2x5HuT/RZI8Ex6pCwe/3ganXDLwMmbhtwPAefya/jahegQ2z7cMX0MacReQPNtt5O4pxDO+9OMpses94+1ObIOls= Authentication-Results: freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=stream-technologies.com; Received: from [192.168.6.128] (212.20.240.118) by DB5PR07MB1031.eurprd07.prod.outlook.com (2a01:111:e400:510d::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.991.4; Fri, 24 Mar 2017 13:40:21 +0000 Subject: Re: cxgbe netmap promiscuous mode? To: Navdeep Parhar References: <58D3C6F4.6010500@stream-technologies.com> CC: "freebsd-net@freebsd.org" From: Joe Jones Message-ID: <58D521C0.1000804@stream-technologies.com> Date: Fri, 24 Mar 2017 13:40:16 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [212.20.240.118] X-ClientProxiedBy: VI1PR0602CA0011.eurprd06.prod.outlook.com (2603:10a6:800:bc::21) To DB5PR07MB1031.eurprd07.prod.outlook.com (2a01:111:e400:510d::24) X-MS-Office365-Filtering-Correlation-Id: 0dba802f-2984-4c23-e74b-08d472bb515f X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:DB5PR07MB1031; X-Microsoft-Exchange-Diagnostics: 1; DB5PR07MB1031; 3:GMe6xcfCxqUqqMq4Wpgw3vR5afPH9dRlGFuIh7ie/y2jwjuhN24Mt50Hrw3jycYzMNL+4fMdRi2/zVgEEdetunGuweQ7vgeTr7TX4yQBr25ud85e6cdAx491eWuTV/wzobyCQzY3aS4OL2u4XY5vHyoHXJRIFgPoORBl/E6kP7JbNV/hzQqzx5Cs9kcUiPa/HhOMaf76h9hAcjEtssdYPmbJnPkRZD2e4LrFhmav9K79J+XY9ch4IxP2rgb/5c/o3Ad2C8c8BN+hbfMY71YRZg==; 25:MT9JdXshlS3TtFPaOfFp31G9giyWGUJMCQez+FefgakKIEXk2ntspe+CFobkqh5vt93gQcWHywoJ6iEkE+GFYGRnmjnb/zgj6dq1NsChWEylarnvxfxYHXHkhn0CbEPyGPW+dAEDcRD2X3oSXVOVuNBU2O2gMEknATCHm1yjCQuqZqk2tIYeP6ryz+qQ1c6KHP9ogmBTdu6J4sE016rLSyrhynJu7sRQJBfZyzkmMLqN3dDBIIvkqodcMzFTg8dWkvuZUKqHjjY68uT/yPm5okUXNKo5rV/rGYFPZcikxPBT0nZgYOKr9Avh5dMC5Sm7CyPuUl15jvxCpZvJbW5ZRa6Nz5n8Hik5p4polDUbkf5LPvZ3knCuYC2Q+5E1UDmqUFWmoscgU8c+bbdcPNWUF8PhsYxKLqFwB/vf5z+3/JBbZC2w57/A53dwLOKTmH4wqO0drDzRmRsxhjWWkl4ywg== X-Microsoft-Exchange-Diagnostics: 1; DB5PR07MB1031; 31:Fubd3cSCKk9S9slOLz1wyUW9hC8yO8f4QjCwA+BxrscS1vfiZWL/gJVkzjEz20DgEjuq6HJWYG/5IguBHxMzjE9Mcp8iUSu5k42abBn/5DTCXF30sxmqlXif0YXhPDAn+kOcYTBidV17wu8f2eDsTljJRTrvh2LE7tYxXg0K8OybjdYzXnlE/cpg3BroYfE/saptCHVxW583iB5a2FVuTIErzzs3QgsWk0yeaHUZDmXQQUFjBqTVaED2dBIwsAGeXMf5Soo+ddF9ZhSLGbZIq+6LeCDxOUT1TKTIIK/RclM= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(75325880899374); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6041248)(20161123558025)(20161123560025)(20161123555025)(20161123562025)(20161123564025)(6072148); SRVR:DB5PR07MB1031; BCL:0; PCL:0; RULEID:; SRVR:DB5PR07MB1031; X-Microsoft-Exchange-Diagnostics: 1; DB5PR07MB1031; 4:6IPyuk3DEUPd4QcZROIppqfDYIIxEPWA7/k90GRzuowcxRQoeqvxtW7BairGFN9R5+jZfvHNK/efIFy8HeQf8lI5xS/DqxNJcgsKKCOZAwhlQL9J/2cUcxzv6ZJzq9ds0kXi11fgZ8HeOSi+uiOhNQITrE9o/66+QIh79W4PNZmVl9yQ/7evWD3y3nMGZl6EOd3kvF6CRlCWJ2NSKZnG+d/Q8K8dFPWDxBmyfwY43TM7GwUM8yoeQmyfyDuIP0GaFKa+rOOAYe0UU4bs9fpi/gC71rcGp10u9gGDFF3hz6x2EvZw84Z8RTSxlx9Q6CU+ac3/rqsGln9NoqPawhxn1bnHBLY6+bky/XF3RBVwwt97yYLxz1haXXIwkT58JzFezWzcvfVYspE65YShma8jEsqzeFmDoI/6XS8S8BBwt5OH5qJFI8vVNHCAwjqlyfBZYtbnDaowtI7d/psZxgSQMWEz0T2cTx1zb1rW7F8L+NlhnpV6WtbZRUfj7dj+Ad7emhk9HysxXL+/Hjb61XFg/quCH4rT9MJAt2TGlklJjIfVxxlIwaV2k1DOwGruc9lWijHzvstF5nYufYktpokdWcJdMHvHZ9eQUNlBJGWHxK5+XB0C8s+n/oBA6/KE0LU/fxAISIBeUk7EQnEY3xDSsQ== X-Forefront-PRVS: 0256C18696 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(6049001)(39410400002)(39450400003)(39830400002)(24454002)(377454003)(66654002)(6116002)(80792005)(76176999)(3846002)(80316001)(86362001)(64126003)(50986999)(65816999)(54356999)(4001350100001)(38730400002)(6666003)(23676002)(50466002)(6916009)(33656002)(66066001)(65806001)(110136004)(6246003)(117156001)(65956001)(47776003)(4326008)(230700001)(5660300001)(8676002)(3480700004)(81166006)(305945005)(53936002)(7736002)(2906002)(25786009)(2950100002)(1411001)(77096006)(36756003)(6486002)(229853002)(42186005)(83506001)(6306002)(189998001)(59896002); DIR:OUT; SFP:1101; SCL:1; SRVR:DB5PR07MB1031; H:[192.168.6.128]; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtEQjVQUjA3TUIxMDMxOzIzOlZ4VjE1cGFxNUlDT1Ewd2Z6d0xzK3cwOEdQ?= =?utf-8?B?Q2JwV29tWHVDckRCTHhyWWQ4bU9PdXVjaVhEMEsva2svUWhCeU15aDFGOXcy?= =?utf-8?B?T0w5OHNwOVZDTy9URWZ6MXU1bXI3b29yb3dTSUpEekZ1VG8xTmFlNVdVVjdh?= =?utf-8?B?Ym5MRGJuOWV0amVBUlh4c1U3VXBlaU1XMDc1d1NZMDNjYU5KTWNtMXFhL1VK?= =?utf-8?B?K1JIT0x3V0pyREVSeFEwYzA0TVBDVmV2WFlUQXFraGdMYXhFbDBTV1VHa2dT?= =?utf-8?B?LzAyRldxRGlOUDlDbzk4R1pNREFKWEZ1TC9sY3llSGptYnVYN0Y2QWNCQUty?= =?utf-8?B?ekYrZXRaN1QrQ2lOdnpYa1UvTnlhME5ScVBUR24vWGh4UXhrWFo1WnloV2ZQ?= =?utf-8?B?K1UzaUdSdjVXT2J2czZXQ2lFcDMwTXpOeFhLc2R1cXFFM3VwWC93MlkvVEQw?= =?utf-8?B?V3RTQk1XODRFYjFmT1BNNXRuZHUra3BkZEQ5ZW0rWXlCamFvS3BnZlBjclRZ?= =?utf-8?B?eGFyL09GTWw4S1E2UGs2aFFvQjB5YXdsNjVmbnRhME5BcytwOWo5QTEzTFhK?= =?utf-8?B?eG1TZjRPWjlaZUlDM0pvbVJsdzFrWkIrL1FEaVFRQVhiTEVjY2J6U1Q2RHhL?= =?utf-8?B?S29kVUZYeDdtR2tnR1JETUE5SFRxQlZ6Zk1kRFdTcFJCdHFYN3pwNkdEaFBn?= =?utf-8?B?NS9NNEJZM0s3UWRXdURTYjc3OURtWHNSRVQvTW9yTFJyT0RDSlNRTG9lQVBi?= =?utf-8?B?RjFOVlM2UVArR2tNQnJiUjlGRmJ6d1QvcFRZdi9SV1k4TThWSk5ZZ1pldDNh?= =?utf-8?B?T3ptdDhwZ1czcVhLUlhBYUF4UjU3bjVYYlF4bS9pUTZhSHNLOWgyZC9ncjZE?= =?utf-8?B?bkVCOWhpZlhrcHNtRkZHaWhLU1ZzcFlNdUVpa2tXdmRRRWJyd0tMUXBzZ1ZK?= =?utf-8?B?c2J5emw4RlcwdUZ6N3gyYnM4SFlySE1GSGlqcUdsS2xRNGVrZ1F6WXFwVDFP?= =?utf-8?B?b2pjMlhRbVhJVjlieHBRand4ZEh1OXhCTVMvSEtIWmhGOEcvVS9vOWFiN2xT?= =?utf-8?B?UHkxOVpCOTN4NjZYMEdkSUNSMjVkUTF0WndEME9sR3lzeVd3MlZ3ZDJyT0Vi?= =?utf-8?B?MS82N2R1cEozZm0zYnVrUnlNOTVMdHM4TXcrOFNvV25aMjh6d1o0RDQ3NTlM?= =?utf-8?B?VnhGcUQ1Nytwb0lqMEF3VHFZbDRrTW90VXlFcWM3L2UyUWRQL1hzOGEzZHd0?= =?utf-8?B?dTdPMUE1K1oxRnI3RVFvSDZBQzh2MU1qWURWa0xFNGJXM3dNRVNXQVNuYkE3?= =?utf-8?B?OXlNam91bGMvWDQzRGpmYzJZZ1lFVFRpcDJ5OERub2llb2VxdjBkdXdBVUtB?= =?utf-8?B?eHdFeHpsdXB0ZTYxeWFUaUxPdGQxWTN2S2UxbnhNMXRld010WnRGNWE1L2xK?= =?utf-8?B?bDFGaDZnSnJPbDNleDRiRFpvVEYvUVV1SDFJMmJpQ2UwMGI4cWgzSnZNeWdR?= =?utf-8?B?TnJaTFBuc2FlUjVNTTllSWJLOTZRUFd5L2FORGwrRFhDbUVqRGw5dXVJbjA3?= =?utf-8?B?Vjc5aUNGM2JkRCtWSkNsL25GWnk4bjh6eWQ5NHhhUmpicEtYck1OQTdLdXph?= =?utf-8?B?Ny9EQzhhZTQ2R2w1TTBRTHJhK2FvNkZkZTdDUElsaFFKNnY5eXNUQUwxNTRq?= =?utf-8?B?RG5ZR2hnV1VLcTgzMjFqcWJFQzdST1BHbEZMSTJ0WHVlUGVicDJZYU9sZkM4?= =?utf-8?B?RG9VQ3lUT00zdUs4MXMyRDJDMXFGRFNESjJnMkVSZnQ2T1ZDY0Qxc0tkYXlJ?= =?utf-8?Q?lA56U6qUKJLR3?= X-Microsoft-Exchange-Diagnostics: 1; DB5PR07MB1031; 6:xxmaLFsmgvw6Od7F1PV2TNgRM+yId7xv9p2s/N27+l6c4drQB67+IvldPB8sL2jyLINarwtufDNO9V6PcERdQKYDkL0RgIgM+usxRR9MKIfe/PkzG0+BmMbxKoLhYBmWTyZ4ekvVqI+FEWh1eCpmHHATU88BvcbmwnJJRv5OycvAOTyIKuIRwzCwyuH+pCFLuKuTYLUEuoYbmiosha6IQ1xtPTNa21JLVVCmJG1IVai10dASZb4PleJtWxIDd/xiYxsFnLSUfqP6nUqXqwD65Y6OI3vsfb8ZTtpwQevmYIaLtBkG7+MCWZSm28Dmy0kDnKncuaPOwaLuXILdqlCRMdmETK8sl45HugGEYQhU1/0qFQUZlWOTpMmkozxxIM6x0emHgSzLI4OGt0rLYaAb5w==; 5:q8pECWFT+Z6WHHccM8KK8zZAsrlz8iuagDSUaqoKoPmcYMTSmRCCZzaFlhh1EsLE6MK6cZGncsemHlifPMz6gnNc20yl8NEB76LtBOT+qZWeBSiKvps9SQcrsuSQTQtDuHz9TGF0TPzssQbeD74Vrw==; 24:AMn0MrObvUM9gS7blwjS3P6XAqfi6fpFx9aDcZHwiSO1PzfZnzV/8j2u+rpaFzAq8ul1Q/m4aDoKCuGtWaU/wNpH0yQL9+lHbLX+iXRyZIc= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; DB5PR07MB1031; 7:z0IITXbF7QwOFprbzZJRedNEbkyI5u7EdPxMzYY6eaYl/REf5cbdjdCqjefxji28H6kEY/XzJGo2oyru9dSemI0Lj3JESF0lrPz9qYD+rz1eEYZRrmTh7qyFcQHJbNbZtXi4qtsbtVUzBdEx9/XrG1bZgu+VbyiUV+R7uuJrM/YPPbTVpT5sHVUUvkVOiM50X0V0tyQ1PAE2BFaNOqibXmJNSZ7QLfxdXqLrD3XBsoWoFWklCkndgWWa3P2aH22QJGmq/7DHcBlwK/v/LHcVIrRkTcjI7k+tepk7UWpN1d2fWoUrZU5kROMmWDPCD5K7wI7U0eqPAo0XIF0DL7Y8mw== X-OriginatorOrg: stream-technologies.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Mar 2017 13:40:21.2320 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5PR07MB1031 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Mar 2017 13:40:27 -0000 Hello Navdeep, This is the panic from 11.0-RELEASE-p8, I think p0 panicked in the same generic_netmap_txsync function. Fatal trap 12: page fault while in kernel mode cpuid = 5; apic id = 0a fault virtual address = 0x1 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff806e38aa stack pointer = 0x28:0xfffffe047ba18440 frame pointer = 0x28:0xfffffe047ba18490 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 2191 (main) trap number = 12 panic: page fault cpuid = 5 KDB: stack backtrace: #0 0xffffffff80b240f7 at kdb_backtrace+0x67 #1 0xffffffff80ad9462 at vpanic+0x182 #2 0xffffffff80ad92d3 at panic+0x43 #3 0xffffffff80fa1d51 at trap_fatal+0x351 #4 0xffffffff80fa1f43 at trap_pfault+0x1e3 #5 0xffffffff80fa14ec at trap+0x26c #6 0xffffffff80f841c1 at calltrap+0x8 #7 0xffffffff806e5a80 at generic_netmap_txsync+0x330 #8 0xffffffff806e06f9 at netmap_ioctl+0x279 #9 0xffffffff8098624f at devfs_ioctl_f+0x13f #10 0xffffffff80b41b34 at kern_ioctl+0x2d4 #11 0xffffffff80b417f1 at sys_ioctl+0x171 #12 0xffffffff80fa26ae at amd64_syscall+0x4ce #13 0xffffffff80f844ab at Xfast_syscall+0xfb We were using our own MACs, we can fix the problem by using the mac from the vcxl interface. Should we not be able to capture all traffic on the interface regardless of what destination MAC it has. Thanks Joe Jones On 23/03/17 18:38, Navdeep Parhar wrote: > Your netmap application should be using the 'vcxl' interface, not the > cxl interface. Even though they share a physical port they have > different MAC addresses and are totally autonomous. The peer should > use the vcxl interface's MAC if it wants to reach the netmap > application. > > Do you have the panic message and stack? I know of a couple of panics > that have been fixed in -STABLE -- one was one related to emulated > mode and the second one was an illegal lock acquisition. > > Regards, > Navdeep > > On Thu, Mar 23, 2017 at 6:00 AM, Joe Jones wrote: >> Hello, >> >> We have a T520-SO and have made a new install of 11.0, to begin with the box >> would panic every time we tried to switch the card into netmap mode. So we >> recompiled the kernel with netmap removed, then compiled the netmap kernel >> module from github, as this in our experience generally leads to a more >> stable netmap. >> >> we have >> >> uname -a >> FreeBSD goose2 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0: Wed Mar 22 >> 16:52:35 UTC 2017 joe@goose2:/usr/obj/usr/src/sys/GENERIC amd64 >> >> and the following in /boot/loader.conf >> >> t4fw_cfg_load="YES" >> t5fw_cfg_load="YES" >> if_cxgbe_load="YES" >> hw.cxgbe.fl_pktshift=0 >> hw.cxgbe.toecaps_allowed=0 >> hw.cxgbe.nnmtxq10g=8 >> hw.cxgbe.nnmrxq10g=8 >> hw.cxgbe.num_vis=2 >> >> Before I run our application I run >> >> ifconfig cxl1 promisc -vlanhwtag up >> >> Now our application can now start without panicking the kernel. Here is >> where it gets interesting, our application is able to announce it's self via >> ARP, I can see the ethernet switch learning which port it's on, and other >> hosts adding it to their ARP tables. When I try an ICMP ping it goes >> missing. After watching the TX packet graph for the connected port on the >> switch while starting and stopping a flood ping to the application, I'm sure >> the packets are getting sent to the card, however I don't see them in the >> netmap ring. If I kill our application, then use ifconfig to create and >> configure a vlan port I can confirm that the card is working and has >> connectivity. >> >> Here's what I think is happening. ARP requests are received because they are >> sent to the broadcast address. Our application then announces it's self. >> However traffic destined for the application is send to a MAC address which >> is neither the broadcast or the MAC programed into the hardware and is >> dropped. My understanding of promiscuous it that it informs the card that we >> want these dropped packets. It looks to me like, when the card is in netmap >> mode the promisc flag is being ignored. >> >> I have also tried using freebsd-update to update to p8. As with the p0 >> kernel we get a panic when we switch the card into netmap mode. >> >> We did previously have these cards working in netmap mode. We were using a >> pre 11 snapshot of the svn head though . >> >> Many Thanks >> >> Joe Jones >> Stream Technologies >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"