From nobody Wed Aug 10 19:09:03 2022 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4M2zxr2CGWz4Z3R5 for ; Wed, 10 Aug 2022 19:09:04 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4M2zxq3bl8z3bj4; Wed, 10 Aug 2022 19:09:03 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.16.1/8.16.1) with ESMTPS id 27AJ92wb061004 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 10 Aug 2022 15:09:02 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:50c8:2c61:7c85:d1a1] ([IPv6:2607:f3e0:0:4:50c8:2c61:7c85:d1a1]) by pyroxene2a.sentex.ca (8.16.1/8.15.2) with ESMTPS id 27AJ91aB078549 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Wed, 10 Aug 2022 15:09:01 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <415bed72-33b4-84ac-111c-f576d7a5fc47@sentex.net> Date: Wed, 10 Aug 2022 15:09:03 -0400 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.1.1 Subject: Re: igc problems with heavy traffic Content-Language: en-US From: mike tancsa To: "Pieper, Jeffrey E" , Jim King , "stable@freebsd.org" , "kbowling@FreeBSD.org" References: <59b9cec0-d8c2-ce72-b5e9-99d1a1e807f8@sentex.net> <86995d10-af63-d053-972e-dd233029f3bf@jimking.net> <3d874f65-8ce2-8f06-f19a-14cd550166e3@sentex.net> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 on 64.7.153.18 X-Rspamd-Queue-Id: 4M2zxq3bl8z3bj4 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-3.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROM(0.00)[]; FREEFALL_USER(0.00)[mike]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[sentex.net]; TO_DN_EQ_ADDR_SOME(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 8/10/2022 2:14 PM, mike tancsa wrote: > On 8/10/2022 1:47 PM, Pieper, Jeffrey E wrote: >> >> You could try disabling EEE (Energy Efficient Ethernet). Something >> like: sysctl dev.igc.0.eee_control=0. > > > Adding > > hw.igc.eee_setting=0 > > in /boot/loader.conf seems to take ok and the NIC works on bootup. I > will run some stress tests with it set. Thanks! > I tried with just that setting and not adjusting the FC and still got the link bounce fairly quickly on the 2.5G xover connection. I was using cat 5 but changed the cable to cat 6 to see if that makes a difference. Re-running the test now with cat 6 I setup 4 servers A----B----C----D A and D are just 2 random servers. B & C are the 2 machines with the 4 igc NICs. B &C and linked on igc1 via xover and A & D were on xover cable to igc2 of B &C 192.168.20.2   ---------- 192.168.20.1 192.168.21.1--------------192.168.21.2                                           10.1.255.209 -----10.1.255.168 Other odd thing I noticed was when I rebooted and set the eee settings to 0 I lost the link between 21.1 and 21.2 on igc2. Whats odd however is that if I change it so that they both go through a switch instead of xover cable, there does not seem to be a problem with the link. Same cable to the switch seems fine so not sure if something else is going on at the link layer. Only the link between the 2 igc nics is 2.5 G and thats still on xover.  I am running iperf3 between 192.168.20.2 and 192.168.21.2.     ---Mike