Date: Tue, 2 Dec 2025 08:49:29 -0800 From: Navdeep Parhar <nparhar@gmail.com> To: mike tancsa <mike@sentex.net>, FreeBSD Questions <freebsd-questions@freebsd.org> Subject: Re: Strange Chelsio performance (T6225-CR) Message-ID: <ff3621dd-5b5a-484c-9109-4b0519bf6020@gmail.com> In-Reply-To: <4503cca3-dd1d-4bd0-a82c-f9c2866f468a@sentex.net>
index | next in thread | previous in thread | raw e-mail
On 12/1/25 10:01 AM, mike tancsa wrote: > TL;DR. I can get 10G coming in and out on port 0, Port 1 can only take > traffic at ~ 2.5 Gb/s incoming, but can send out a full 10G. > > I have a t6 2 port NIC on a FreeBSD 13 stable server that gets a lot of > backups coming into it via ssh and zfs streams (zrepl). At this point, > I swapped out the NIC, and motherboard and even tried a second NIC but > the same strange 2.5Gb/s incoming results using iperf3 as a test. If I > run iperf3 -c -P5 I get a bit better performance, but still not 10G. On > port 0, its solid 10G in both directions, no dropped packets etc. > > port 0 is on a quanta switch > port 1 is a cisco 3850 switch with 4 10G ports. I setup 2 other test > boxes and on the test boxes I can get a full 10G on both ports in a test > environ through the cisco using the same nics and Supermicro MB (H12SSL > and H11SSL). When the H12SSL was my test board, all was good in my > tests. But when I swapped it for the H11SSL that was showing problems, I > had the same issue when I made it live. I have tried different GBICs as > well. Again, the GBICs in the test environment work great in the same > class T6 card > > in /boot/loader.conf I have > > > hw.cxgbe.rdmacaps_allowed="0" > hw.cxgbe.iscsicaps_allowed="0" > hw.cxgbe.fcoecaps_allowed="0" > hw.cxgbe.pause_settings="0" > hw.cxgbe.attack_filter="1" > hw.cxgbe.drop_pkts_with_l3_errors="1" > > hw.cxgbe.largest_rx_cluster="65536" # 64KB max buffers > hw.cxgbe.safest_rx_cluster="16384" # 16KB default buffers Have you tried your tests with these two left to their default values? We don't have a 64KB allocator in the kernel (16KB is the max) so the first setting isn't even valid. Please also provide the output of # sysctl dev.cc | egrep 'link|caps|pause' It is possible that the link on the good port has pause based flow control enabled and the other one doesn't. Regards, Navdeephome | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ff3621dd-5b5a-484c-9109-4b0519bf6020>