From owner-freebsd-net@freebsd.org Fri Mar 24 10:19:59 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DEEACA2771 for ; Fri, 24 Mar 2017 10:19:59 +0000 (UTC) (envelope-from Joe@stream-technologies.com) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-ve1eur01on0043.outbound.protection.outlook.com [104.47.1.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 61F3711D4 for ; Fri, 24 Mar 2017 10:19:57 +0000 (UTC) (envelope-from Joe@stream-technologies.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=streamtechnologiesuk.onmicrosoft.com; s=selector1-streamtechnologies-com01e; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=Hfz85j5731DZfOzCatMvO6LeHDkzLSmHuLO8FfG8rnc=; b=K3Nj999W0T8Q13ROhe70kn4jH95E8yXTZd/OeKvnUTXxUkJpjsMc/+20Ttv0uORnBncE9HiR+Ul/1n/BEHYzkXKyedP0AX4fLJEZaG6pN6ZcvjeIwuD0JnWP3f32/Vf+r9njfwo4Py0ftgnr+TOC+rytLSQ+Yl01nOnOTK6qBzU= Authentication-Results: freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=stream-technologies.com; Received: from [192.168.6.128] (212.20.240.118) by VI1PR07MB1039.eurprd07.prod.outlook.com (10.161.111.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1005.2; Fri, 24 Mar 2017 10:19:53 +0000 Subject: Re: cxgbe netmap promiscuous mode? To: Vincenzo Maffione References: <58D3C6F4.6010500@stream-technologies.com> CC: "freebsd-net@freebsd.org" From: Joe Jones Message-ID: <58D4F2C5.9020204@stream-technologies.com> Date: Fri, 24 Mar 2017 10:19:49 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [212.20.240.118] X-ClientProxiedBy: AM4PR0501CA0058.eurprd05.prod.outlook.com (10.172.222.154) To VI1PR07MB1039.eurprd07.prod.outlook.com (10.161.111.143) X-MS-Office365-Filtering-Correlation-Id: 2622253a-ad95-465e-c8eb-08d4729f5030 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:VI1PR07MB1039; X-Microsoft-Exchange-Diagnostics: 1; VI1PR07MB1039; 3:Dirm8ilRS2AgLyhJG5r+D3zw4bGJpOItjCSfS1QoLbWeiyU2j3jJPfPyKUgQYM5pPi6gQp8ImMxjCsQaKt1ncg8WvoNcbLmOxRvXHIqFroGoBRm6C6+UEJkJjjBo1tRX+q+Er0494RZiakBsgIvaDbJA4CNBm+rLQl1iib400rw6zUVtksCFEezJHmI+krf1YwxIzz8wGza84qrAsH37tkySch36YxxyIQZY5bv2ZGVl0sGflLHWUKBn7UVmbaCHMM+TkzF3/sCBL+gXiTC58Q==; 25:6TpSPziYXw5inCovglPwDenOfERsExDu4KKr3jbBXeT3RGwVM84bDu5PUbwbgTn4AdXzj3pXsHlx4RGL55+UuYs5gnmRaPuMPnYOumZq/duuwe9ED3ThdM6ja2iMxB8pvAMMelHySwpkmkgDxOPKwxPdiegmgSgNipoLrXumiWsrlxHQBaeqO5BHCJSchc27Wcc5GGsd1BFF5nsvIspGkR/Lipe+GuMQ96kHPVV8V0GCvCZ670SGsR4/fNVZR9WsgaE19gkuk1Dui6a9/aHb+nXkvHN6lGbfi30EHetsamFTUqc/ijPQ+GKW/WaadFFEovJ1JoH9KqZWPzKf4jJpvlAbgCfH4TJFqZ7eYnt4o5SBQjb7GHlmoIZikCuEIOBQVlux7T1WCJ/R+P1LzGgegxBWzQTfyuHEYz4oNvvbNnWgHr0F6xbOJ65CqPnjEiC50ikdnGKA0dxqcOiNb0eI0A== X-Microsoft-Exchange-Diagnostics: 1; VI1PR07MB1039; 31:6D6JcX2kgyNIStLt3Va22KgNBrylPPgkSNAXiYBeX5TZCJUM8w2uN81s0Fne8zV2my2Id73DeThJktR0NshJLpDUlITObZ48jjKDPTN7kyz8MO79pfVS3m1FxbeIyyolP5yprK+xuQ6uAxmPCxTbPZTrOX6u1Y+FSdsiE+7wbtgwt1CwYEWr1TZtpyL2BacVQHRnPn7mnCvv/pZhZsiShmKHA45FxhIKQWIGr6fsfcZj8PbnB0gx7Lfsx5dAEYyDJXsYfCc+2lJndFYFuGjKtJH1Q2/IhydFOqFGj4/Yi+U= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(75325880899374); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6041248)(20161123564025)(20161123562025)(20161123555025)(20161123560025)(20161123558025)(6072148); SRVR:VI1PR07MB1039; BCL:0; PCL:0; RULEID:; SRVR:VI1PR07MB1039; X-Microsoft-Exchange-Diagnostics: 1; VI1PR07MB1039; 4:26UaO05r0JNfs4iDF86fvK1/i0IJg4weW7/NiS1bJf5Ky5HJeP5CcElUCWbjkyi2yydyjl6uUIMktsoM3gcTcYH09l8Fe6jfPVKn64Kxfw34PUkT1f5iqr3aHjhOCaalgvlGQCVLWLYRK5dBCVWnh4vMfzAdLkWNi4N6WAN2B8IJfessCp/pjyJco+n4jR/xGVsGugLuBEPBBFfC9Jak+oN3EyS9sAQnxRTP9gTfehK6xvlMIvH2QjbIolurBSJWoKV4LJmlcI0lIxBmGpZhjH68eypmY/UFRFmQHasKldr3wuyCVxH4ljiitrRMo7OBzQ2CAFnf+joSx7IpZNyAYjuR1fjCNVCBHeVYzBHVYgJTs3LPBbO1jUy+AzKksnOF95IidIkF9uG653BY/P/meLFtEEzDL6aXYowleXN1CBLYpQSY8/HnQdqexFbE22cTZALO4cdJVLKW3P+oHlOe58NyvBu/u/9bW3m01jpAeqNQ2dhKbrLQ13H32dU/uknZe8zfQpgV0zCeL1SodhkFdOmhYw1zqCkeMwqxwTySb2xp+ljJG1NKQKuwDjEskUujGAgA5tr+gcTMYV2gBcS/lB7euQUxx0QE96WHqNLERuZJgVxmM6eu+buWeAsymwKAobqo1BIFeFzhB8oFFtXcKQ== X-Forefront-PRVS: 0256C18696 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(6049001)(39410400002)(39450400003)(39830400002)(66654002)(24454002)(377424004)(189998001)(65956001)(36756003)(230700001)(23676002)(66066001)(65806001)(5660300001)(6486002)(50986999)(87266999)(110136004)(86362001)(53546009)(65816999)(76176999)(6246003)(54356999)(53936002)(25786009)(83506001)(47776003)(229853002)(6916009)(33656002)(3846002)(38730400002)(2950100002)(77096006)(4326008)(6306002)(117156001)(90366009)(6666003)(6116002)(305945005)(64126003)(50466002)(42186005)(8676002)(3480700004)(80792005)(7736002); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR07MB1039; H:[192.168.6.128]; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtWSTFQUjA3TUIxMDM5OzIzOnVjSkc5NXhkZkQxVDdFQ2lScWJZTE1QZThH?= =?utf-8?B?L3hFTHhYU1NWNU5Cd2lxcWx2YjN0ZGVzalBMK0RjRWpzazRmUEpmaFRuUkxY?= =?utf-8?B?ZkxDY1Q5U1JXb0RDZEVMTFZHc29UWWRCOGxmK1lTaXBadkFXbitkVnFLazVF?= =?utf-8?B?VTNwek9CUmRiWHp2RzUrSnFhcm5lQVpRb0ZhZkdwMW5Oam1Oc3J3aEtGaFBN?= =?utf-8?B?Zk5kUE9HUjZycXg1TVJ6eTBudW9hUk9wZ2h5RjB2NGRRcTMrR1MxUTRTTmds?= =?utf-8?B?OUNBZCsxbGxNeDdoeTd0Q3g3Qk5oK05KWE90RDRJS2hScGYvd1ltdUhFSjJQ?= =?utf-8?B?aTllV1dIdzZJZ3JHNG1ZUlFKdllIZnU4ZUpTM2lvVWRaZjZQWVM5VG84TS9o?= =?utf-8?B?RmZvbmFZMmd2NkdQTzhKUUhNeDg2LzJoRGh0WW1uSldCbHF3RlhzUWRsUUlD?= =?utf-8?B?NnFaYTRydlFIVDFOVXNvaGJ4SmRrL3pSdDFyMmJKdnJRZ3dtelNpamt6RkFP?= =?utf-8?B?MGxxc0dKVjA2aGxsZERSYzZsUWE5OUN1N0x6emJ5My9Tc1hTTFRGcDk1VkVW?= =?utf-8?B?ekswTVpkZkxWUkhxQzBZRU01Y1hscUFPV0sxbmloa21EczFNL0xUeXE4N3JG?= =?utf-8?B?VHhpOU5FTFRwOWdqZk1sZWJhb3AzVHB2S0VqVzF1Zk8rK3p0UGZSYm43Zml0?= =?utf-8?B?R2ZHVzIxbndFL1gxVnUxYytUR0FrcFdlaHF1c3cyNUlTQ1lUUjdMM1llWjVO?= =?utf-8?B?WDZLRXBjM0lWUGd3Nk16RURzdnl0ZXcyaDZBZk1hV0I5cFJTZGhMNG5LTU9Z?= =?utf-8?B?c3RWR1haQllia2ZkaGczYXJhbE9nVlp0aS9Oc1lwb214Z0ZCZGowcXRmS2pQ?= =?utf-8?B?NVdMTzFUZ0hEOTRnRzJBU2FQYkVuVkZDb1NVcjN2SzNTOTdwUkFKVlJjZnk5?= =?utf-8?B?OEt6S2M4QlkzekJBV3g3WEdFUGZ2WFBxaTV1cGpBaGo1N0xIVVJlVkVWNkhR?= =?utf-8?B?MDVpeGlFOU45UGpodjhJYkJ5a3VBZm9wb1ZtakQ0QnIrcUw4QWZCYjNmOFRI?= =?utf-8?B?K0xpWXFzaGhSUk4xbmNpRTl6Qyt3NG5ZS1V1V1JveUxCNDdKenptZDdkRWlt?= =?utf-8?B?d2RRbVE5MGFFL2NjemNiOVMrSFA0Ky81N2ZObFkyWm03K2hBNzFRc0owREV6?= =?utf-8?B?UFRZcVBmTTR5TTVYdFI1dEwxamxjTmFUYndZcEl3NWo4UTlhRXIzeGFreGgr?= =?utf-8?B?SnRqNXRGckMvZHN4c0RPTVQxOU8vR1FFWFBiMHlUL3FKYjJBUHdyUFNPb2xM?= =?utf-8?B?M0VraTJJQy9sN2JoSVFyV0QwbFR1Q2FHekgrNnQzeEprNTUvYXAyUUUzWmRs?= =?utf-8?B?MzdJWU1YTDAyNENJcWFVejdQSklUeHpsTGJ5Umw2MFNVUkJ1ZUplKy9UbFI0?= =?utf-8?B?V1BmRk9Jd3pWMStnbGZTWUVleGlIN0tTSktOaHBieWdPWkl0bmhjMHczR25B?= =?utf-8?B?TUNCSWtsRFRRQitaenlnbVAzcDk4cDJEUi95bWtzLzJvOU1oQ3ZYa3puQXVW?= =?utf-8?B?VEVnSXFBYmFYOGdGc1RHdXk5OEZSU1N3aUIyS0d0RDdCaXdVMW8zd2dUQkJV?= =?utf-8?B?SGtPNkR4VG9xUFl3NWV3UkZjWVdtSTJZQktFQzNOelNoQi9GdFhUL0xJdVVy?= =?utf-8?Q?ElXOE0AIh9d8w2uOdHLUrPJEayPyW0lDlkcEd6M?= X-Microsoft-Exchange-Diagnostics: 1; VI1PR07MB1039; 6:UzGmr+zygZFZBywyqSQkqzdLU7yiGAshLsRujNiANc8eKvTAy7EsBWU2vX1zzbixdn0oDwTShEAlI+hoHK/SOzxrGNm31/XIkWDKappgvLxvDBOxoC49v6quTZ2xFh/6E7ueBpPP5wQ5E+YY+dwWue8ZWGSauPsvH9oVsTM05LCfxHXNKb8uc3o4yMiRnrC61uTrhJCS+zOPeA26ybIiglIQA/ENAsyFnrR1x9srHFk/NM/EKePhiPTsRqhRBvFyGwoDCwUCu2dBCc6aaittaUUy2ykTGXQOpqV7L1YD4XiG4f5XW0bp1FhzVMxtUXnUkMhf+f9qaOzt17CbSkrVSUfYAv7NxieoQxNuVXRar7gx2iCXUY/u8gDf3WIDTR1ObiVnNfxfwTRkO5Ss5vKdGg==; 5:KrYhaNZqy+7+ZS73gkJXlKwTE29wewZnVhc0P0xcHxhay9EOH0npy/G66kPcyoOsweC+13i6KY5LlC7TqNHTmnRLGubJe2CEAEYoD4p12DoiUaiBrD1PVjqPCo/Xy1G/4lnfaXnmX8qJxUZo+0Aj6w==; 24:95mGpagA4sA5VfAmnvcuRDVCLtLxOP6JGqx5/yLLzBzHQjte4lKH0PQrR8vdsOfXW5gcp3Brbf2P71EED8jw94NxBKVrzEeGEeMq4fSPttE= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; VI1PR07MB1039; 7:0HonQt3gAG6XVZvbHKYcdlRfa4vbzNgTfG8eCSgtljUHumi3lPbcqZsA3BTwYri1kChvoZkRw+hJBLoSe/3k8FtOEYSImp3bDc3iYQkCE7j+XKiUeAmzqIfnsnq0Xv3KYa5P2AGgy/9+ABNNe594wG2cyRcI4jOlKQH5PApTIqNZLAmLpCotVmo7B5XTrMBy5FBZDOjSqVFOEPi3uUL8ZgsIrWsR5SkKTChctFWuD1UXUuCHeTeONfL/VNHTX9AgJS6q/qJhT7bN6GZtWmSZBlBBmPZjfzQM6FFtR301lLcx7KPiXR01awfUQ+F5L7IifnMaO2EtXjWS4NjdRzjJDg== X-OriginatorOrg: stream-technologies.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Mar 2017 10:19:53.4882 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR07MB1039 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Mar 2017 10:19:59 -0000 Hi Vincenzo, I just tried with that sysctl set to 2, I get a similar looking panic to before Fatal trap 12: page fault while in kernel mode cpuid = 7; apic id = 0e fault virtual address = 0x1 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff806e38aa stack pointer = 0x28:0xfffffe047ba18440 frame pointer = 0x28:0xfffffe047ba18490 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 2205 (main) trap number = 12 panic: page fault cpuid = 7 KDB: stack backtrace: #0 0xffffffff80b240f7 at kdb_backtrace+0x67 #1 0xffffffff80ad9462 at vpanic+0x182 #2 0xffffffff80ad92d3 at panic+0x43 #3 0xffffffff80fa1d51 at trap_fatal+0x351 #4 0xffffffff80fa1f43 at trap_pfault+0x1e3 #5 0xffffffff80fa14ec at trap+0x26c #6 0xffffffff80f841c1 at calltrap+0x8 #7 0xffffffff806e5a80 at generic_netmap_txsync+0x330 #8 0xffffffff806e06f9 at netmap_ioctl+0x279 #9 0xffffffff8098624f at devfs_ioctl_f+0x13f #10 0xffffffff80b41b34 at kern_ioctl+0x2d4 #11 0xffffffff80b417f1 at sys_ioctl+0x171 #12 0xffffffff80fa26ae at amd64_syscall+0x4ce #13 0xffffffff80f844ab at Xfast_syscall+0xfb This is on 11.0-RELEASE-p8 Thanks, Joe Jones On 23/03/17 18:20, Vincenzo Maffione wrote: > Hi, > You could try to use netmap in emulated mode (sysctl > dev.netmap.admode=2). If this works, at least you know that the > problem is in the cxgbe netmap support and not in the netmap core itself. > > Cheers, > Vincenzo > > 2017-03-23 14:00 GMT+01:00 Joe Jones >: > > Hello, > > We have a T520-SO and have made a new install of 11.0, to begin > with the box would panic every time we tried to switch the card > into netmap mode. So we recompiled the kernel with netmap removed, > then compiled the netmap kernel module from github, as this in our > experience generally leads to a more stable netmap. > > we have > > uname -a > FreeBSD goose2 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0: Wed Mar > 22 16:52:35 UTC 2017 joe@goose2:/usr/obj/usr/src/sys/GENERIC amd64 > > and the following in /boot/loader.conf > > t4fw_cfg_load="YES" > t5fw_cfg_load="YES" > if_cxgbe_load="YES" > hw.cxgbe.fl_pktshift=0 > hw.cxgbe.toecaps_allowed=0 > hw.cxgbe.nnmtxq10g=8 > hw.cxgbe.nnmrxq10g=8 > hw.cxgbe.num_vis=2 > > Before I run our application I run > > ifconfig cxl1 promisc -vlanhwtag up > > Now our application can now start without panicking the kernel. > Here is where it gets interesting, our application is able to > announce it's self via ARP, I can see the ethernet switch learning > which port it's on, and other hosts adding it to their ARP tables. > When I try an ICMP ping it goes missing. After watching the TX > packet graph for the connected port on the switch while starting > and stopping a flood ping to the application, I'm sure the packets > are getting sent to the card, however I don't see them in the > netmap ring. If I kill our application, then use ifconfig to > create and configure a vlan port I can confirm that the card is > working and has connectivity. > > Here's what I think is happening. ARP requests are received > because they are sent to the broadcast address. Our application > then announces it's self. However traffic destined for the > application is send to a MAC address which is neither the > broadcast or the MAC programed into the hardware and is dropped. > My understanding of promiscuous it that it informs the card that > we want these dropped packets. It looks to me like, when the card > is in netmap mode the promisc flag is being ignored. > > I have also tried using freebsd-update to update to p8. As with > the p0 kernel we get a panic when we switch the card into netmap mode. > > We did previously have these cards working in netmap mode. We were > using a pre 11 snapshot of the svn head though . > > Many Thanks > > Joe Jones > Stream Technologies > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org > " > > > > > -- > Vincenzo Maffione