From owner-freebsd-net@freebsd.org Fri Apr 23 08:21:42 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2F9795E811E for ; Fri, 23 Apr 2021 08:21:42 +0000 (UTC) (envelope-from mueller6722@twc.com) Received: from p-impout005.msg.pkvw.co.charter.net (p-impout005aa.msg.pkvw.co.charter.net [47.43.26.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FRS1d2Jswz3s2t for ; Fri, 23 Apr 2021 08:21:40 +0000 (UTC) (envelope-from mueller6722@twc.com) Received: from localhost ([96.28.177.163]) by cmsmtp with ESMTP id Zr49l2G8HeKjEZr49lRcpE; Fri, 23 Apr 2021 08:21:34 +0000 X-Authority-Analysis: v=2.3 cv=ALzgjvLx c=1 sm=1 tr=0 a=xqrt2BZAGHte7XHhrxJgbA==:117 a=xqrt2BZAGHte7XHhrxJgbA==:17 a=HpEJnUlJZJkA:10 a=m5xhijH0GzXBgENEFjQA:9 Date: Fri, 23 Apr 2021 08:21:22 +0000 From: "Thomas Mueller" To: freebsd-net@freebsd.org Subject: Re: Client Networking Issues / NIC Lab References: X-CMAE-Envelope: MS4wfIWygZqiKeKXb25DgtJRaDLHN1mzSZ+osUNkg0N7fdRIwP9EgGZysqVWkgtITCQwXzYOyrD6sadDdYKZ5ihje17/7d8LjKGS8yfdaaWJOU4oOwAiE3/N 3WG92t5RvonQfRLBSR7vAA0f7+a9jzdSL6v5HZrg8mhM/uIkPyOE1Raj X-Rspamd-Queue-Id: 4FRS1d2Jswz3s2t X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mueller6722@twc.com designates 47.43.26.136 as permitted sender) smtp.mailfrom=mueller6722@twc.com X-Spamd-Result: default: False [-0.72 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[47.43.26.136:from]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[47.43.26.136:from]; FREEMAIL_FROM(0.00)[twc.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[twc.com]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[47.43.26.136:from:127.0.2.255]; RECEIVED_SPAMHAUS_PBL(0.00)[96.28.177.163:received]; MISSING_MID(2.50)[]; R_SPF_ALLOW(-0.20)[+ip4:47.43.26.0/24]; NEURAL_HAM_LONG(-0.92)[-0.920]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[twc.com]; ASN(0.00)[asn:40294, ipnet:47.43.24.0/21, country:US]; MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-net]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Apr 2021 08:21:42 -0000 from Kevin Bowling: > I have been looking into client networking issues in FreeBSD lately. > To summarize the situation, common NICs like Intel gigabit (e1000 aka > lem(4)/em(4)/igb(4)), Realtek (re(4)), Aquantia, and Tehuti Networks > are unmaintained or not present on FreeBSD. The purpose of this > thread is to gauge whether that matters, and if it does what to do. I > believe it is important because we are losing out on a pipeline of new > contributors by not supporting client hardware well. We risk losing > NAS, firewall, and other embedded users which may not be large enough > to negotiate with these vendors for support or have the volume to do > custom BOMs to avoid risky parts. My opinion has been developed after > researching the drivers, Bugzilla, and various internet forums where > end users exchange advice or ask for help where FreeBSD is the > underlying cause. > e1000 is in the best shape, with recent vendor involvement, but covers > 20 years of silicon with over 100 chipsets (of which at least 60 are > significant variations). Datasheets are readily available for most of > them, as well as "specification updates" which list errata. There are > chipsets which have been completely broken for several years. More > common, there are cases that lead to user frustration, including with > the most recent hardware. All of the silicon tends to have > significant bugs around PCIe, TSO, NC-SI (IPMI sideband), arbitration > conflicts with ME and more. Intel doesn't patch the microcode on > these, but many of the issues can be worked around in software. > Performing an audit of the driver will take quite a while, and making > and testing changes gives me concern. When we (my previous employer > and team) converted these drivers to iflib, we fixed some of the > common cases for PCIe and TSO issues but only had a handful of chips > to test against, so the driver works better for some and worse or not > at all for others. I have started fixing some of the bugs in > Bugzilla, but I only have a few e1000 variants on hand to test, and I > have an unrelated full time job so this is just occupying limited > spare time as a hobby. > re(4) is in pretty abhorrent state. All of these chips require > runtime patching of the phy (which I believe is a DSP algorithm that > gets improved over time) and mcu code. That is totally absent in > FreeBSD. A vendor driver exists in net/realtek-re-kmod which contains > the fixups and works alright for many users. This driver cannot be > imported into FreeBSD as is. There is a strange use of the C > PreProcessor which blows up compile time and driver size needlessly. > The out of tree driver has a different set of supported adapters, so > some kind of meld is necessary. Realtek does not provide public chip > documentation, I am trying to see if they will grant NDA access to > contributors. Some re(4) chips work in FreeBSD, some don't. I gave up on FreeBSD 12.x because of re(4) deficiencies. Sad to have to seek NDA access for an open-source project like FreeBSD. NetBSD seems to work better. OpenBSD GPT support is in such condition as to render incompatible with my system. Haiku, maybe? My computer with on-motherboard AR9271 wireless chip, dating to 2013, is still waiting for FreeBSD support. > Aquantia has an out of tree driver in net/aquantia-atlantic-kmod. The > code is not currently in a place where I'd like to see it in the tree. > I am not really sure how common these are, the company was acquired by > Marvell which is still producing them as a client networking option > while they have other IP for higher end/speed. > Tehuti Networks seems to have gone out of business. Probably not > worth worrying about. > 1) Do nothing. This situation has gone on for a while. Users are > somewhat accustomed to purchasing FreeBSD-specific hardware for things > like SOHO gateways and NAS. A lot of people just revert back to Linux > for client use. OpenBSD seems to have more active contribution around > this kind of thing and works better for common cases so that may be > another exit ramp. > 2) Quantify usage data and beg the vendors for help. This might work > for Intel, however these devices have transferred to a client team at > intel that does not plan to support FreeBSD, and intel does not keep > test systems around long enough to meet FreeBSD user's needs. Realtek > is a similar story, I am unsure how long they hold on to test systems > and would probably need technical guidance to work with the FreeBSD > community. Unsure about Marvell, I've never worked with them. > 3) Build a NIC lab and focus on building community support. It would > also give the vendors a place to test hardware their labs have purged > (due to IT asset management policies or other bureaucratic blunders). > Set some boundaries like a 15 year window of chipsets which should > cover practical embedded use cases. There are backplane systems > and/or external PCI(e) expansion systems that could be assembled to > house a large number of NICs. It would probably be cheaper than this, > but say a budget of $15000USD is enough to purchase some expansions, a > couple managed switches, and a few dozen common NICs. Community > members may also send in NICs they wish to see supported or undergo > testing. For this to work out long term, there needs to be a quorum > of people interested in collaborating on the issue. There are some > risks around simply setting this up, depending on the configuration, > the bus topology may introduce problems unrelated to the NICs and we'd > probably need some semi-automated device.hints or devctl stuff to keep > from over provisioning system resources (work on a subset of cards at > a time). An interesting extension of this would be a semi-automated > validation setup for subsystem changes (significant driver changes, > iflib, lro, etc). > 4) ??? Tom