Date: Tue, 07 Nov 2023 11:31:09 +0000 From: Benoit Chesneau <benoitc@enki-multimedia.eu> To: "freebsd-net@FreeBSD.org" <freebsd-net@freebsd.org> Subject: modern NIC to use a mellanox connectx-4 Message-ID: <T0GcB0nYnLJLMd945cyHMAf8rvOKnELBprLqrIA2rKpW665-5jcXoQWzuuAEjQ3MY2wsgws1Uy_g4Gb5U_zY_zD8yiugFQNjjVKH00n7ALU=@enki-multimedia.eu>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Hi all, Reading the Freebsd 14.0 changes it seems that netfli xsponsors the removal of old drivers. Not sure what it means exactly, but as of todaty which kind of full featured NIC for servers we can use that bring at least 2x25G ? By full featured I mean good support of SR-IOV and netmap though I am not sure if netmap is considered stable. For now all my hardware has a mellanox connectx-4 Lx 2xx25G which has its problems: ``` mlx5_core0@pci0:17:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x1590 subdevice=0x00d4 vendor = 'Mellanox Technologies' device = 'MT27710 Family [ConnectX-4 Lx]' class = network subclass = ethernet PCI-e errors = Correctable Error Detected Unsupported Request Detected Corrected = Advisory Non-Fatal Error mlx5_core1@pci0:17:0:1: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x1590 subdevice=0x00d4 vendor = 'Mellanox Technologies' device = 'MT27710 Family [ConnectX-4 Lx]' class = network subclass = ethernet PCI-e errors = Correctable Error Detected Unsupported Request Detected Corrected = Advisory Non-Fatal Error ``` But it appears that SR-IOV barely works on it: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270859 Also speed with SR-IOV is half of the bandwidth I could get. Netmap works weirdly and ipperf3 makes the kernell panic when used with udp mode. Not sure what could be done for it. Any feedback is welcome :) Benoît Chesneau, Enki Multimedia — t. +33608655490 Sent with [Proton Mail](https://proton.me/) secure email. [-- Attachment #2 --] <div style="font-family: Arial, sans-serif; font-size: 14px;">Hi all,</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">Reading the Freebsd 14.0 changes it seems that netfli xsponsors the removal of old drivers. Not sure what it means exactly, but as of todaty which kind of full featured NIC for servers we can use that bring at least 2x25G ? By full featured I mean good support of SR-IOV and netmap though I am not sure if netmap is considered stable.</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">For now all my hardware has a <span>mellanox connectx-4 Lx 2xx25G which has its problems:</span></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span><br></span></div><div style="font-family: Arial, sans-serif; font-size: 14px;">```</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span>mlx5_core0@pci0:17:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x1590 subdevice=0x00d4</span><div><span> vendor = 'Mellanox Technologies'</span></div><div><span> device = 'MT27710 Family [ConnectX-4 Lx]'</span></div><div><span> class = network</span></div><div><span> subclass = ethernet</span></div><div><span> PCI-e errors = Correctable Error Detected</span></div><div><span> Unsupported Request Detected</span></div><div><span> Corrected = Advisory Non-Fatal Error</span></div><div><span>mlx5_core1@pci0:17:0:1: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x1590 subdevice=0x00d4</span></div><div><span> vendor = 'Mellanox Technologies'</span></div><div><span> device = 'MT27710 Family [ConnectX-4 Lx]'</span></div><div><span> class = network</span></div><div><span> subclass = ethernet</span></div><div><span> PCI-e errors = Correctable Error Detected</span></div><div><span> Unsupported Request Detected</span></div><span> Corrected = Advisory Non-Fatal Error</span><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">```</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">But it appears that SR-IOV barely works on it: <span><a target="_blank" rel="noreferrer nofollow noopener" href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270859">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270859</a></span></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">Also speed with SR-IOV is half of the bandwidth I could get. </div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">Netmap works weirdly and ipperf3 makes the kernell panic when used with udp mode. Not sure what could be done for it. </div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">Any feedback is welcome :)</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div> <div class="protonmail_signature_block" style="font-family: Arial, sans-serif; font-size: 14px;"> <div class="protonmail_signature_block-user"> <div style="font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Helvetica; font-size: 12px; color: rgb(0, 0, 0);">Benoît Chesneau, Enki Multimedia<br></div><div style="font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Helvetica; font-size: 12px; color: rgb(0, 0, 0);">—<br></div><div style="font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Helvetica; font-size: 12px; color: rgb(0, 0, 0);">t. +33608655490 <br></div> </div> <div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div> <div class="protonmail_signature_block-proton"> Sent with <a target="_blank" href="https://proton.me/" rel="noopener noreferrer">Proton Mail</a> secure email. </div> </div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?T0GcB0nYnLJLMd945cyHMAf8rvOKnELBprLqrIA2rKpW665-5jcXoQWzuuAEjQ3MY2wsgws1Uy_g4Gb5U_zY_zD8yiugFQNjjVKH00n7ALU=>