From owner-freebsd-net@freebsd.org Fri Mar 24 13:50:49 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4679AD14B17 for ; Fri, 24 Mar 2017 13:50:49 +0000 (UTC) (envelope-from Joe@stream-technologies.com) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-ve1eur01on0065.outbound.protection.outlook.com [104.47.1.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 83520803 for ; Fri, 24 Mar 2017 13:50:47 +0000 (UTC) (envelope-from Joe@stream-technologies.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=streamtechnologiesuk.onmicrosoft.com; s=selector1-streamtechnologies-com01e; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=4zbSTKNDGHcfGtJrN4MfqGm/xCrIMuhUGFgx3q+mCLk=; b=RMzdzgaKZ3naFpWDK1nIhGwt7dOLdwVDIxRWzIRX2K0ArMLEB+n1n9ywzFMCT+knlxV5mbJbp1jTEu4Avh9MaEhW/dkCsy8L1yH5kw/4Y/WHGC/Edoa5kbVdfhnf79lkWJ1i03UbhWDRmecwhlDoucXKFM0mpEvbB6oJ7KOOWY0= Authentication-Results: freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=stream-technologies.com; Received: from [192.168.6.128] (212.20.240.118) by HE1PR07MB1034.eurprd07.prod.outlook.com (10.162.27.26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.991.4; Fri, 24 Mar 2017 13:50:43 +0000 Subject: Re: cxgbe netmap promiscuous mode? To: Vincenzo Maffione References: <58D3C6F4.6010500@stream-technologies.com> <58D4F2C5.9020204@stream-technologies.com> CC: "freebsd-net@freebsd.org" From: Joe Jones Message-ID: <58D5242F.9040706@stream-technologies.com> Date: Fri, 24 Mar 2017 13:50:39 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [212.20.240.118] X-ClientProxiedBy: HE1PR0901CA0062.eurprd09.prod.outlook.com (10.168.89.158) To HE1PR07MB1034.eurprd07.prod.outlook.com (10.162.27.26) X-MS-Office365-Filtering-Correlation-Id: 654d5e32-07a7-4bea-6942-08d472bcc422 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:HE1PR07MB1034; X-Microsoft-Exchange-Diagnostics: 1; HE1PR07MB1034; 3:b1u7dHJLfaQ0KkfuO9RRqRm5LrLZyN0UNMkmL4xAzchVia5yCScqcXV0tp8ni5sbhkGtMbUtyXoXErZGjCRi18AELUKwVpg2cKqBORSAjW0/S0eKDcKPTKD4bZuiYrB4ccKeesDHV+L2zIayPFJjILsoKnGjGI4pwSTfcdUTUoyrXnd9CNiX3lC/WYDuDINonvXGVbfsQJ8QW7F/F+Vf0ibjP0yaJKk8ka1qJ41VCM/DQ5jkYRsOFcWIQjt/9aeduZN4EuMk+ZLMUg02is2Yxg==; 25:C0OsuKYVtvhbjXbLba3pKF6nZOLoMMFDj3crnJZ/itMfYmgyYA9Rv2BDHBf1Ijpj2IKbGFVZJwcnBWBHqVMFUV/RBOzO1R9ofMGkvYjBx31TWt8l7ifYvmFArhJjgzeeqa7NRZqJz9GvChGb4iEBvfUiklcXFelAPfgrcq+aVXFbzlkVddyb9L/ziz3sMNTy87iL8v0Yt++I3dXwyqHK5VHHGptBVJ1yxVBnRRsqetP0gI9Q8GizdozFEI1WercZHG432eks/Bl434xcQ0HyLqT81h1NjptNAZAGdRWGm0tR4030/W8gE1l9ct0nuc1KDOgia+2Q2DH+SxrgV7Dvty2wRMeSGzJbAcBsIZohsXcrx9cKeT8q0eiq92Ti1qaAgqUI6aX9a6rpR3OwA6+aNY9bS27vc3CQB0q7nIHQ5DLE6D2NfAT8+o+jJFVSEuae2Yg7ZSLXtuibbGkfBLcQgQ== X-Microsoft-Exchange-Diagnostics: 1; HE1PR07MB1034; 31:Z2PgTqWqgTelHOaK6OKV9FpSnZ5ss2Wz+t89u0Z1ZjEslElHNVN7oVekXW3mUuhFUnCEN/ZACBzjhHjRatCEhpUVqNvxTDxg5aBv/+HWOB3D1FY7iFtBm5iNv/GVZpj9tCXjAXpKTnHDrz805nXcDQbejBe2Ury/Aw3it9T5PCS5XnDncHNTn35GjECr4PTft0y/nMqwFZsTJQrCj5TajmL++UihCU75CWspMVdlIbC6F+tQ6lvbXwrBfsjkDe+fy9svF92LxS7fJQzZitv5yw== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(75325880899374); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040391)(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6041248)(20161123558025)(20161123562025)(20161123564025)(20161123555025)(201703131423016)(201702281528016)(201703061421016)(201703061406016)(20161123560025)(6072148); SRVR:HE1PR07MB1034; BCL:0; PCL:0; RULEID:; SRVR:HE1PR07MB1034; X-Microsoft-Exchange-Diagnostics: 1; HE1PR07MB1034; 4:QSwsGqmdUsHxeQU9NAhikxEk5tQssjYp0ugo67b28661MlFo5/1vbssVjaa6/GOfgz87momPfhozhjKrelijzLO+vIM1oYutV+PvmRaz2AT10IvOOHDQG5Nblib8pYpEixUA4UXTrAdmEVkrZ3SAL2ME29U9s7/CbYhMrrmPjP56zTXJCh0wa2dUESz8X/HhhU367c+4v0phgccV4H4gWpQjusXWKCBSa/K2Klqof2fX19aWT69ZAKVR8GyjOUzjWwATyeQUXTHAYHRL0AbCAdQTDy6O10eZcHyLjRoCBFcBYByFGOSqbeHABM9nV2xrIQD0bM0XseG7A9ZIiEqb29kflynF5EIJzWUsZMupq/HDziQkHAT7sQ1okU507yGEC76a+Dp+LXoHusZvQVyGXK9YUMwRYAfPz3XHLni+iP7c9FUMAfWLCAiSC4UZFBhh7Hkyb10SA/1elJhb6HQ7lYSxYkDs1jOCJ0pB8JM+vhgXmUvnHN0Msbj5rdibPZIfXPYnewzcn8o+pR8K6YoQEz3lhLG/7mBQ0/q+KX/pEMaNo09uCLJu591LersAaxFRDI3zaQMdi9tI7d/z0dr+o/basguAmBiC0gwLJVM9BJHuzcl9G+ffCkGukLagKKM+6UcnTMZlkDXTv0v9xJivQfQNsLuwNsniyhBMbnAOZG9DEBvzc40Y0LntExr+Ju755PQstB6xkJDyIVaHA6t3Fp0AWMxjrPwHVNBRua8MPA3dF/2s471a9ly5sH2UoEL7M76iZVXcBQnBJ5IUQ/jIupAvh82qS40PwaOO4rcO6xc= X-Forefront-PRVS: 0256C18696 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(6049001)(39830400002)(39410400002)(39450400003)(39400400002)(377424004)(24454002)(66654002)(25786009)(66066001)(23676002)(6666003)(2950100002)(47776003)(80316001)(65806001)(6916009)(53936002)(6306002)(64126003)(5660300001)(77096006)(3846002)(50986999)(80792005)(4001350100001)(65816999)(229853002)(189998001)(38730400002)(6246003)(110136004)(36756003)(76176999)(3480700004)(54356999)(6116002)(86362001)(83506001)(65956001)(42186005)(59896002)(93886004)(117156001)(8676002)(4326008)(2906002)(90366009)(6486002)(50466002)(81166006)(33656002)(230700001)(305945005)(7736002); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR07MB1034; H:[192.168.6.128]; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtIRTFQUjA3TUIxMDM0OzIzOitTMkIvU21hdTBZRDNJMkEyM3pIbFhsMmhN?= =?utf-8?B?ZU9JNmc2ajdBQXdlSnhTYWxnZzkrSlh1WlRtTk0wblBBSGVwU0Y5K1UvbUV0?= =?utf-8?B?NVZ6OFlpcUdlL3NNY2JXMWI0WU5lNnExQnNIY3RUS05YbWJ6L3R3UXE0V1l2?= =?utf-8?B?K29vSDRsWXcwZzlEMkZCZW9sTjBFY1p6TW51ZWFaNVhneTJuaWNsU3I4WHJB?= =?utf-8?B?TExheU1QdmtYMnRVcHdqenhwK0pVSkhjOTlETWsxWTUzWlFvazNjci91UlJD?= =?utf-8?B?L3BlK0Z3YlFIc3c1VTF2QmNrb2lsNmdyQkM5Q0JnQ3AyOXRrU0FXcWR0Zk4w?= =?utf-8?B?UzJYUTdnNkRmdlE1c0tpSWFVUW9GZCtNNkRjcURpRTBMd2Q4WnFjVE5OZFFG?= =?utf-8?B?U3lIcnBvVjFEZEx5b2NTK2oxdGZ0bm9KMVFTVFhTR2ZDRk43L2NwNFdKMnh2?= =?utf-8?B?UkQweE9tdy95eDIzMlpJUlJBU1dvQzE3M2tnUW4wTkVONVlkQXA2TS9KVXhH?= =?utf-8?B?RVIxNXBUa3E1K0xrRC9hUWhLY2x2SGpYVjlLakl6OWZCcS8ySHpyY3dOMFY3?= =?utf-8?B?dWtESzJwelkzckpRSWI0VWlNYkptdjkwQUxRdjQ4MXQ2TFJVVi9GdnY4R1Rv?= =?utf-8?B?RDNieDRRaFV0ZGM5ZTlzM1J0aU5xWDhXWnk4NEhSMDFYdUptUmZnVERoazFk?= =?utf-8?B?ZkkrMTNHVHRxdmJEL0JkRnM0UXpSeWZjNnEwMTJWTUFXdVVXTVFrMmdhcDlv?= =?utf-8?B?b1VvRmdvQUlvdTh6RXVkTmNZUkxib2FRcDMzZ2F3Nmt5ZW1MR0lSaGh0NEw5?= =?utf-8?B?VjlIczhKaEZyU0E0VFVuckJPMHlOSzh6Q2RSY3NmTEdLcGliT0NmRWVsS0dl?= =?utf-8?B?TExJVkx1NHhEWnNFazRLQ1ROam5ybTN0Zjgxb3pLaG92VktpWnJ3SnB4blJZ?= =?utf-8?B?QklnTVpwMzUxdzhFRnlNK3NVSjh3ejRwaFhra08wY2l6RVVUd2piZnBSZW5o?= =?utf-8?B?WGtJSU1wU3djUjhNTG96U28rUEVtWVJwbkNRTXoxSkNTU3hDRlNHZXp5N3R6?= =?utf-8?B?aTAwNlJKUDFBTVdSbXlOR1VxNnZiVFIrNkg3UmQ1TTJnN29GVUJ6VEFDbWtt?= =?utf-8?B?YU9OSDRUU0I2YVhZWGNGL2lIMUUzLzZ1N1d1YXZZRkRDV01USDRMVXZJV0xu?= =?utf-8?B?eHI3dDRqMkEwZ003bVhCOTViSWM4L3VNR3dOQnI4b0UxTVZxNVVDV1R4REpC?= =?utf-8?B?REVseVRtcFE5ejNnSTZ4ZUcyRlcwRENISzNZK1FQZmxTbG9BSXV6Z0QyaTE0?= =?utf-8?B?UkNmV3hxMVl3OG9KaGhyc2ZGZ2NFZG9QSExNOGh5ZWExN1lwYnh4bXBSa09m?= =?utf-8?B?bXBIOTkvVituSEdBMTBNdHQzbmhiMU5sY1lPSlBVRFBjTU9FZXlLUGxzcmxF?= =?utf-8?B?QXM0UURubW5NRU1qRkFQN3dhSXVsNk80STBFWXFXVXEvRVYrL1NYQXM4cEtL?= =?utf-8?B?YnlvemJyY2V5d1ptRVZsR0xVQ1ZkaGxsSUF5ZlZFbWRlUkNabS9BQWdINHV6?= =?utf-8?B?dERCNENWait3U0ZERUlwblkyU2pLNzhkV3JFeWU3cVk2N1grZTVzRlZwcmNI?= =?utf-8?B?UjhFSldGeUhlTUJSRE1BM2RiQnRVZnd4NmhsbW9LUmZiQlh4TWZISzJJWXRE?= =?utf-8?B?eG1lVXhXc0VUQ0RHQ1NOR29Mb3g3RnovZjNhaWVMQ3VUREtWRTVqVTdjbGJv?= =?utf-8?B?dGx5YUYwT2ZabkV6VHhsY3E2bEQ0S2xoQ1VQSHB2N3ErNlp4SjE3WEVtd3V1?= =?utf-8?B?cmQwb3ZKWEdDSkxhYmhxUTVKYVNCaExXZFRrK2ZnQW0vcWlFK0FNMmVyUldM?= =?utf-8?Q?i7s1DaTonh4=3D?= X-Microsoft-Exchange-Diagnostics: 1; HE1PR07MB1034; 6:zEBfwEDCtn12hWKgHdl6DGNLUycVaRPvuT502B2h+IaUWwzuWufILSOgUruLuwDKTyFO33kLfz/PL2tNOAujRJA7DPsoPx0LSKvIP6RHNn3EGye+mu5t7A16gJ2jbO7R57xRbgFrumb0MhrLCPy7+XIZwXiZEZToOb4o9rpuDl0PlnqGFXZQC6aRBQQ28HETH4MhHPDvWYejhGQA+Th9q9EPih0IgDzslWf5bhwJxOFj2rvXWZVG0Dnn3jwPU8aTiABZrtvCYaw7S4ehkywwJxfxZz7oIYe36Veb14ButN7UpCqAwseVSVd2JFVwYt5HehZr5UvfHAa6JoKEXR1v65b9gfjGor4s0PKjwBH7EVaJ6P5aqQDBoSvkZttAs9VJ2FQ9PSYHqwgXq+7RGBt/Lw==; 5:KTTtflmBWgteo2Tu27c+pOFQ3bKjnknXqPmgFofRfYahtmuBqPXXhNUK+94pGFRpknwYmgxhwkROj2vCwD54t+0U3d/OEeirgtdG/BIPifAbT6f1rLYdrIvdsq+A/I4e2kMUbpzJHeTCicTbJBP2sA==; 24:3f8Gw16SeOQ0QJL4uksH3IG5GPnoQucAz+b2S+et72ZVilO85X7RU8RrWiHHVHeFBq663QoYJnEy1zrriroFxuLVoBu8IZTkaNg5tyDOprA= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; HE1PR07MB1034; 7:3fosPaDSQxwW+6c764lilvu/ffXKNMLdVKphjoXjEvbAA0l1LIsIFCskeIPAd5pNpPLWI950FKI+trTBD7r/W9tRbLFlS2CWCGaw9PTHkri2R0RkOSsBa/tvQ8y34ES54BcVFZgmMaVG/8IUKK3TiCNiAuGGlIRqCaWmAW4MiF/Z6q85/XvNjtExeN8lDm3WTnUz3jHGDz5M9ooKnOZhu/9D3YviK5Ub0GLqSPIqsJwRW5M1oZJUiEIG+O/kK02EAtkYs4CtKGl5UlZoAkE4iq7V9es50mTzl9/b2TbWIDjCmhGRP2oaDIlgU2e3HK1Ju6JhzF8wJMxXzig6Fm3Fig== X-OriginatorOrg: stream-technologies.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Mar 2017 13:50:43.3478 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR07MB1034 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Mar 2017 13:50:49 -0000 Hi Vincenzo, we can get rid if the panic if we compile the kernel without netmap, then load an up to date netmap as a module. We can live with that for now. Some time last year before Free BSD 11 was branched from head, we compiled a checkout and it worked without problem on the same hardware setup. I may try and figure out what the regression is, I'm not familiar with the FreeBSD release process either though. Thanks, Joe Jones On 24/03/17 10:29, Vincenzo Maffione wrote: > Hi Joe, > There was a fix for a panic in emulated mode that was applied > stable/11 branch, so I guess it also ended up into FreeBSD-11.0-STABLE. > I don't know whether the same fix ended up into in 11.0-RELEASE-p8 > (I'm not familiar with FreeBSD releasing process, sorry!). > > Or maybe this panic happens with netmap upstream? > If this is a new bug, it would be nice to have the kernel with the > debug symbols enabled, so that we can get more detailed information > from the stack trace. > > Cheers, > Vincenzo > > 2017-03-24 11:19 GMT+01:00 Joe Jones >: > > Hi Vincenzo, > > I just tried with that sysctl set to 2, I get a similar looking > panic to before > > Fatal trap 12: page fault while in kernel mode > cpuid = 7; apic id = 0e > fault virtual address = 0x1 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff806e38aa > stack pointer = 0x28:0xfffffe047ba18440 > frame pointer = 0x28:0xfffffe047ba18490 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 2205 (main) > trap number = 12 > panic: page fault > cpuid = 7 > KDB: stack backtrace: > #0 0xffffffff80b240f7 at kdb_backtrace+0x67 > #1 0xffffffff80ad9462 at vpanic+0x182 > #2 0xffffffff80ad92d3 at panic+0x43 > #3 0xffffffff80fa1d51 at trap_fatal+0x351 > #4 0xffffffff80fa1f43 at trap_pfault+0x1e3 > #5 0xffffffff80fa14ec at trap+0x26c > #6 0xffffffff80f841c1 at calltrap+0x8 > #7 0xffffffff806e5a80 at generic_netmap_txsync+0x330 > #8 0xffffffff806e06f9 at netmap_ioctl+0x279 > #9 0xffffffff8098624f at devfs_ioctl_f+0x13f > #10 0xffffffff80b41b34 at kern_ioctl+0x2d4 > #11 0xffffffff80b417f1 at sys_ioctl+0x171 > #12 0xffffffff80fa26ae at amd64_syscall+0x4ce > #13 0xffffffff80f844ab at Xfast_syscall+0xfb > > This is on 11.0-RELEASE-p8 > > Thanks, > Joe Jones > > On 23/03/17 18:20, Vincenzo Maffione wrote: > > Hi, > You could try to use netmap in emulated mode (sysctl > dev.netmap.admode=2). If this works, at least you know that > the problem is in the cxgbe netmap support and not in the > netmap core itself. > > Cheers, > Vincenzo > > 2017-03-23 14:00 GMT+01:00 Joe Jones > > >>: > > Hello, > > We have a T520-SO and have made a new install of 11.0, to > begin > with the box would panic every time we tried to switch the > card > into netmap mode. So we recompiled the kernel with netmap > removed, > then compiled the netmap kernel module from github, as > this in our > experience generally leads to a more stable netmap. > > we have > > uname -a > FreeBSD goose2 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0: > Wed Mar > 22 16:52:35 UTC 2017 > joe@goose2:/usr/obj/usr/src/sys/GENERIC amd64 > > and the following in /boot/loader.conf > > t4fw_cfg_load="YES" > t5fw_cfg_load="YES" > if_cxgbe_load="YES" > hw.cxgbe.fl_pktshift=0 > hw.cxgbe.toecaps_allowed=0 > hw.cxgbe.nnmtxq10g=8 > hw.cxgbe.nnmrxq10g=8 > hw.cxgbe.num_vis=2 > > Before I run our application I run > > ifconfig cxl1 promisc -vlanhwtag up > > Now our application can now start without panicking the > kernel. > Here is where it gets interesting, our application is able to > announce it's self via ARP, I can see the ethernet switch > learning > which port it's on, and other hosts adding it to their ARP > tables. > When I try an ICMP ping it goes missing. After watching the TX > packet graph for the connected port on the switch while > starting > and stopping a flood ping to the application, I'm sure the > packets > are getting sent to the card, however I don't see them in the > netmap ring. If I kill our application, then use ifconfig to > create and configure a vlan port I can confirm that the > card is > working and has connectivity. > > Here's what I think is happening. ARP requests are received > because they are sent to the broadcast address. Our > application > then announces it's self. However traffic destined for the > application is send to a MAC address which is neither the > broadcast or the MAC programed into the hardware and is > dropped. > My understanding of promiscuous it that it informs the > card that > we want these dropped packets. It looks to me like, when > the card > is in netmap mode the promisc flag is being ignored. > > I have also tried using freebsd-update to update to p8. As > with > the p0 kernel we get a panic when we switch the card into > netmap mode. > > We did previously have these cards working in netmap mode. > We were > using a pre 11 snapshot of the svn head though . > > Many Thanks > > Joe Jones > Stream Technologies > > _______________________________________________ > freebsd-net@freebsd.org > > mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > > > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org > > >" > > > > > -- > Vincenzo Maffione > > > > > > -- > Vincenzo Maffione