From owner-freebsd-net@freebsd.org Tue Nov 26 06:34:29 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D8C8E1AAFA2 for ; Tue, 26 Nov 2019 06:34:29 +0000 (UTC) (envelope-from bsd-lists@BSDforge.com) Received: from udns.ultimatedns.net (static-24-113-41-81.wavecable.com [24.113.41.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "ultimatedns.net", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47MYz82wVdz4S12 for ; Tue, 26 Nov 2019 06:34:27 +0000 (UTC) (envelope-from bsd-lists@BSDforge.com) Received: from udns.ultimatedns.net (localhost [IPv6:0:0:0:0:0:0:0:1]) by udns.ultimatedns.net (8.15.2/8.15.2) with ESMTPS id xAQ6ZCkI050963 (version=TLSv1.2 cipher=DHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 25 Nov 2019 22:35:19 -0800 (PST) (envelope-from bsd-lists@BSDforge.com) X-Mailer: Cypht MIME-Version: 1.0 Cc: In-Reply-To: From: Chris Reply-To: bsd-lists@BSDforge.com To: Artem Viklenko Subject: Re: How to remove watchdog? Date: Mon, 25 Nov 2019 22:35:18 -0800 Message-Id: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 47MYz82wVdz4S12 X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of bsd-lists@BSDforge.com has no SPF policy when checking 24.113.41.81) smtp.mailfrom=bsd-lists@BSDforge.com X-Spamd-Result: default: False [2.71 / 15.00]; ARC_NA(0.00)[]; HAS_REPLYTO(0.00)[bsd-lists@BSDforge.com]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; IP_SCORE(-0.16)[ip: (-1.37), ipnet: 24.113.0.0/16(-0.68), asn: 11404(1.30), country: US(-0.05)]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[BSDforge.com]; REPLYTO_ADDR_EQ_FROM(0.00)[]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.61)[0.608,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.35)[0.352,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:11404, ipnet:24.113.0.0/16, country:US]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2019 06:34:29 -0000 On Tue, 26 Nov 2019 07:55:11 +0200 Artem Viklenko artem@viklenko=2Enet said > Hi! >=20 > I have several small boxes with realtek nics acting as a router/firewall=2E > Also had same issues=2E FreeBSD driver didn't work at least for me so I=20 > switched to Realtek's driver=2E But after some time traffic stops passing= =20 > my routers=2E Did some investigation and found that the issue is 9k mbufs=2E > As far as I understand more traffic you push more issues with 9k mbufs > appears due to memory fragmentation=2E > You can check it with 'vmstat -z | grep mbuf'=2E Hah! I wondered about that=2E It *appeared* to be due to increaded load on the NIC -- hence our interest in upping to 10G NICs=2E >=20 > So I decided to do wery dirty hack - I've changed Jumbo_Frame_9k > to Jumbo_Frame_4k in the if_re=2Ec from Realtek's latest 1=2E95 driver=2E > It comiles and work on FreeBSD 10=2Ex and 11=2Ex and now it just works > (vendor says that driver is for older versions of the FreeBSD OS)=2E > And no more issues=2E >=20 > ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP >=20 > mbuf_packet: 256, 2362080, 2, 1263, 2054916, 0, 0 > mbuf: 256, 2362080, 514, 1776,3460790080, 0, = 0 > mbuf_cluster: 2048, 369076, 1265, 31, 154081, 0, 0 > mbuf_jumbo_page: 4096, 184537, 513, 294,1592339809, 0, = 0 > mbuf_jumbo_9k: 9216, 54677, 0, 0, 0, 0, 0 > mbuf_jumbo_16k: 16384, 30756, 0, 0, 0, 0, 0 >=20 > Now driver use mbuf_jumbo_page not mbuf_jumbo_9k and no fails=2E >=20 > I'm ok with mtu 1500 in my environment and I don't know if mtu 9000 will > work with this change=2E But at least it is stable now even after 100 days > of uptime (just rebooted after upgraded to 11=2E3-RELEASE-p5)=2E Thank you *very* much, Artem=2E I'll have a closer look=2E I'm thinking of taking your concept, and upping it to 7k=2E I'll post back, if anything good comes of it=2E :) >=20 > Hope this helps=2E It does=2E :) --Chris >=20 >=20 > 26=2E11=2E19 02:44, Chris =D0=BF=D0=B8=D1=88=D0=B5: > > Or at least make it non fatal=2E > > OK here's the story; I'm experimenting with a multiport NIC (re(4)) > > as we hope to start using multiport 10G NICs=2E > > Any of the re's we've used in the past have been very stable, which > > is why I picked the one I did for this experiment=2E This one has been > > performing rock solid for some 4 to 6 mos, under full time use=2E That > > is until the last week=2E Where we're seeing: > > watchdog timeout > > repeated frequently=2E Which is ultimately fatal=2E ifconfig up/down will > > not resuscitate it=2E Nor will service ifconfig restart, or plugging/ > > unplugging the cable(s)=2E Bouncing the server is the only cure=2E Which > > is unacceptable=2E Any, and All suggestions, or insight into the matter > > GREATLY appreciated=2E Note; while this is an old 11=2E1, we're not plannin= g > > to up this box until we can confirm this can be cured=2E :) > >=20 > > Details follow: > > 11=2E1-STABLE r327867 amd64 > >=20 > > watchdog timeout > > watchdog timeout > > watchdog timeout > > watchdog timeout > > watchdog timeout > > watchdog timeout > > watchdog timeout > > watchdog timeout > > watchdog timeout > > watchdog timeout > > watchdog timeout > > watchdog timeout > >=20 > > rc=2Econf(5) > > ifconfig_re0=3D"inet AA=2EBBB=2ECC=2EXX netmask 255=2E255=2E255=2E0 rxcsum txcsum t= so4" > > ifconfig_re1=3D"inet AA=2EBBB=2ECC=2EWW netmask 255=2E255=2E255=2E0 rxcsum txcsum t= so4" > > ifconfig_re1_alias0=3D"inet AA=2EBBB=2ECC=2EZZ netmask 255=2E255=2E255=2E0" > >=20 > > ifconfig(8) > > re0: flags=3D8843 metric 0 mtu = 1500 > >=20 > > =C2=A0=C2=A0=C2=A0=C2=A0options=3D8219b > > =C2=A0=C2=A0=C2=A0=C2=A0ether 00:13:3b:0f:13:44 > > =C2=A0=C2=A0=C2=A0=C2=A0hwaddr 00:13:3b:0f:13:44 > > =C2=A0=C2=A0=C2=A0=C2=A0inet6 fe80::213:3bff:fe0f:1344%re0 prefixlen 6= 4 scopeid 0x1=20 > > =C2=A0=C2=A0=C2=A0=C2=A0inet AA=2EBBB=2ECC=2EXX netmask 0xffffff00 broadcast= 24=2E113=2E41=2E255=20 > > =C2=A0=C2=A0=C2=A0=C2=A0nd6 options=3D23 > > =C2=A0=C2=A0=C2=A0=C2=A0media: Ethernet autoselect (1000baseT ) > > =C2=A0=C2=A0=C2=A0=C2=A0status: active > > re1: flags=3D8843 metric 0 mtu = 1500 > >=20 > > =C2=A0=C2=A0=C2=A0=C2=A0options=3D8219b > > =C2=A0=C2=A0=C2=A0=C2=A0ether 00:13:3b:0f:13:45 > > =C2=A0=C2=A0=C2=A0=C2=A0hwaddr 00:13:3b:0f:13:45 > > =C2=A0=C2=A0=C2=A0=C2=A0inet AA=2EBBB=2ECC=2EWW netmask 0xffffff00 broadcast= 24=2E113=2E41=2E255=20 > > =C2=A0=C2=A0=C2=A0=C2=A0inet AA=2EBBB=2ECC=2EZZ netmask 0xffffff00 broadcast= 24=2E113=2E41=2E255=20 > > =C2=A0=C2=A0=C2=A0=C2=A0inet6 fe80::213:3bff:fe0f:1345%re1 prefixlen 6= 4 scopeid 0x2 > > =C2=A0=C2=A0=C2=A0=C2=A0nd6=20 > > options=3D23 > > =C2=A0=C2=A0=C2=A0=C2=A0media: Ethernet autoselect (1000baseT ) > > =C2=A0=C2=A0=C2=A0=C2=A0status: active > > lo0: flags=3D8049 metric 0 mtu 16384 > > =C2=A0=C2=A0=C2=A0=C2=A0options=3D600003 > > =C2=A0=C2=A0=C2=A0=C2=A0inet6 ::1 prefixlen 128 =C2=A0=C2=A0=C2=A0= =C2=A0inet6 fe80::1%lo0 prefixlen 64 > > scopeid=20 > > 0x3 =C2=A0=C2=A0=C2=A0=C2=A0inet 127=2E0=2E0=2E1 netmask 0xff000000 =C2=A0= =C2=A0=C2=A0=C2=A0nd6=20 > > options=3D21 > > =C2=A0=C2=A0=C2=A0=C2=A0groups: lo > > pciconf(8) > > re0@pci0:5:0:0:=C2=A0=C2=A0=C2=A0 class=3D0x020000 card=3D0x012310ec ch= ip=3D0x816810ec=20 > > rev=3D0x07 hdr=3D0x00 > > =C2=A0=C2=A0 vendor=C2=A0=C2=A0=C2=A0=C2=A0 =3D 'Realtek Semiconductor= Co=2E, Ltd=2E' > > =C2=A0=C2=A0 device=C2=A0=C2=A0=C2=A0=C2=A0 =3D 'RTL8111/8168/8411 PCI= Express Gigabit Ethernet=20 > > Controller' > > =C2=A0=C2=A0 class=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D network > > =C2=A0=C2=A0 subclass=C2=A0=C2=A0 =3D ethernet > > re1@pci0:6:0:0:=C2=A0=C2=A0=C2=A0 class=3D0x020000 card=3D0x012310ec ch= ip=3D0x816810ec=20 > > rev=3D0x07 hdr=3D0x00 > > =C2=A0=C2=A0 vendor=C2=A0=C2=A0=C2=A0=C2=A0 =3D 'Realtek Semiconductor= Co=2E, Ltd=2E' > > =C2=A0=C2=A0 device=C2=A0=C2=A0=C2=A0=C2=A0 =3D 'RTL8111/8168/8411 PCI= Express Gigabit Ethernet=20 > > Controller' > > =C2=A0=C2=A0 class=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D network > > =C2=A0=C2=A0 subclass=C2=A0=C2=A0 =3D ethernet > >=20 > > Thanks again! > >=20 > > --Chris > >=20 > >=20 > > _______________________________________________ > > freebsd-net@freebsd=2Eorg mailing list > > https://lists=2Efreebsd=2Eorg/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd=2Eorg" >=20 > --=20 > Regards!