From owner-freebsd-net@freebsd.org Sun Mar 22 17:35:36 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C00F0267DB0 for ; Sun, 22 Mar 2020 17:35:36 +0000 (UTC) (envelope-from dan@langille.org) Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48ll5z3c6vz4f33 for ; Sun, 22 Mar 2020 17:35:35 +0000 (UTC) (envelope-from dan@langille.org) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.west.internal (Postfix) with ESMTP id 3D181528; Sun, 22 Mar 2020 13:35:33 -0400 (EDT) Received: from imap36 ([10.202.2.86]) by compute2.internal (MEProxy); Sun, 22 Mar 2020 13:35:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=langille.org; h= mime-version:message-id:in-reply-to:references:date:from:to :subject:content-type; s=fm3; bh=jHoCe98JPRwreAjLfcEK3kf0Fh+9Z3M jtKswncDQZpw=; b=AdFJ3z+jm8dMJDdH1S2YL3P+vHLOvJTexNJ7OhRPvc8TODV 5CxfWMJR+7Z4GBuQ6tOzNDZr+I1tjCUxJAHmka0ePtQ3EUpcQErEf3krdl2bZYOl xPGkkD6Mfh2PdgESz7rWaXLybIDWWoMhKaJxP5WhprCGveUPnZ80qBydWO3EqiGZ tUc3b1vtU6RV/AVrExnf15AifcgrqeA75yFW+KjLplxYRtPXwOv4CzUzn40OWNCl RubwuIK53KCn5IOQicze58wRPbBkrIwhuZLRUWnxqSn5osqmfJZkHSFe2a1dXX4+ JdxwqAx9d7HQGLhGRkE2se0tcQjcxtkiZQXIj0g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=jHoCe9 8JPRwreAjLfcEK3kf0Fh+9Z3MjtKswncDQZpw=; b=WsUWRMlJqlLBFUQS77svpn fb5C/fRQAMtXHJWdVKnFA4ezihzfsrMKtNbL4qplyNv65PBbUSYWWiRKFMqW0TMm SUa4VuGSQyp8RDOXEyDcWwBghIRFKAA5wbaX8i0bKeJqOaTWSoNScvFsxYtpC0Ja 2mOhWR59Y0MzhVN2OnevHE/W0Q4p6N/Rai55LIlOHCBhfznsZ+3x3SBLBm5sH7Yt lPSIHhKxlMg0kDN+WuaFq4FG+Rtt0mBICfzpiIGZaW1rzfTGSn7xRULWoetx2qpY 3JGQr88sb4bqKmqHU1lXORoVpQQxtKtubaB0k3Y10pCLTyCl1lIGQ5xrteIJmv5A == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrudegiedguddthecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefofgggkfgjfhffhffvufgtsehttdertderredtnecuhfhrohhmpedfffgr nhcunfgrnhhgihhllhgvfdcuoegurghnsehlrghnghhilhhlvgdrohhrgheqnecuffhomh grihhnpehsvghrvhgvthhhvghhohhmvgdrtghomhdpfhhrvggvsghsugdrohhrghenucev lhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegurghnsehlrg hnghhilhhlvgdrohhrgh X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 252891880062; Sun, 22 Mar 2020 13:35:32 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.7-1021-g152deaf-fmstable-20200319v1 Mime-Version: 1.0 Message-Id: In-Reply-To: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> Date: Sun, 22 Mar 2020 13:35:11 -0400 From: "Dan Langille" To: "Pieper, Jeffrey E" , "freebsd-net@freebsd.org" Subject: Re: SFP+ on PRO/10GbE Content-Type: text/plain X-Rspamd-Queue-Id: 48ll5z3c6vz4f33 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=langille.org header.s=fm3 header.b=AdFJ3z+j; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=WsUWRMlJ; dmarc=pass (policy=none) header.from=langille.org; spf=pass (mx1.freebsd.org: domain of dan@langille.org designates 64.147.123.20 as permitted sender) smtp.mailfrom=dan@langille.org X-Spamd-Result: default: False [-5.59 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[langille.org:s=fm3,messagingengine.com:s=fm2]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; URIBL_BLOCKED(0.00)[servethehome.com.multi.uribl.com,messagingengine.com.multi.uribl.com,langille.org.multi.uribl.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[langille.org:+,messagingengine.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[langille.org,none]; NEURAL_HAM_MEDIUM(-0.99)[-0.994,0]; IP_SCORE(-3.50)[ip: (-9.85), ipnet: 64.147.123.0/24(-4.92), asn: 11403(-2.69), country: US(-0.05)]; RCVD_IN_DNSWL_LOW(-0.10)[20.123.147.64.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; MID_RHS_WWW(0.50)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2020 17:35:36 -0000 Hello Jeff, Partial success. The card is now able to use an SFP+ optic. It warns me when the optic is installed: Mar 22 16:49:45 r720-01 kernel: WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules. I cannot use an SFP+ optic at the switch. The connection just does not happen. If I go back to the original SFP optic, the connection occurs, as expected at 1G. On the switch side, I've tried a known good optic from an existing connection. I could install an PRO/10GbE instead, that has a built-in transceiver. I have two of those in use now, both working on 10G. Ideas? Thank you. On Sat, Mar 21, 2020, at 7:20 PM, Pieper, Jeffrey E wrote: > Hi Dan, > > Intel(R) Ethernet Server Adapter X520-2 only supports Intel-branded > optics, however you can bypass that check in the driver by adding the > following to /boot/loader.conf: > > hw.ix.unsupported_sfp=1 > > I hope this helps, > Jeff > > -----Original Message----- > From: owner-freebsd-net@freebsd.org On > Behalf Of Dan Langille > Sent: Saturday, March 21, 2020 3:25 PM > To: freebsd-net@freebsd.org > Subject: SFP+ on PRO/10GbE > > Hello, > > I have an PRO/10GbE in a Dell R720, running FreeBSD 12.1, connected to > a Unifi US-16-X switch. At present it is using an SFP transceiver. I > wanted to move to SFP+. > > I am not sure which card this is. The iDRAC tells me: > > * Integrated NIC 1 - Intel(R) 2P X520/2P I350 rNDC > * the service tag claims this R720 was configured with a Intel X520 DP > 10Gb DA/SFP+ > with part number C63DV. Looking that up, the images I found are > consistent with > what I have in the box. The NIC has room for two transceivers and two > ethernet cables. > > When the SFP+ transceiver is inserted, this message appears: > > kernel: ix0: Unsupported SFP+ module type was detected. > > I don't know if this is: > > * the wrong transceiver - I've tried two [1] > * the card firmware rejects anything not on a whitelist [2] > * the transceiver is in the wrong slot [3] > * only does DAC on 10G [4] > > Ideas and suggestions please. Thank you. > > $ dmesg|grep ix0|head -n 1 > ix0: port 0xfcc0-0xfcdf > mem 0xd8d00000-0xd8dfffff,0xd8ff8000-0xd8ffbfff irq 36 at device 0.0 > numa-domain 0 on pci1 > > pciconf -lcbv provides this: > > ix0@pci0:1:0:0: class=0x020000 card=0x1f721028 chip=0x10fb8086 rev=0x01 > hdr=0x00 > vendor = 'Intel Corporation' > device = '82599ES 10-Gigabit SFI/SFP+ Network Connection' > class = network > subclass = ethernet > bar [10] = type Memory, range 64, base 0xd8d00000, size 1048576, > enabled > bar [18] = type I/O Port, range 32, base 0xfcc0, size 32, enabled > bar [20] = type Memory, range 64, base 0xd8ff8000, size 16384, > enabled > cap 01[40] = powerspec 3 supports D0 D3 current D0 > cap 05[50] = MSI supports 1 message, 64 bit, vector masks > cap 11[70] = MSI-X supports 64 messages, enabled > Table in map 0x20[0x0], PBA in map 0x20[0x2000] > cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS > link x8(x8) speed 5.0(5.0) ASPM disabled(L0s) > cap 03[e0] = VPD > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected > ecap 0003[140] = Serial 1 ecf4bbffffc0bda0 > ecap 000e[150] = ARI 1 > ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI > disabled > 0 VFs configured out of 64 supported > First VF RID Offset 0x0180, VF RID Stride 0x0002 > VF Device ID 0x10ed > Page Sizes: 4096 (enabled), 8192, 65536, 262144, > 1048576, 4194304 > > > > [1] - The working 1G SFP is a Fiberstore SFP1G-SX-85. I have tried a > Fiberstore SFP-10GSR-85 > and a Avago AFBR-703SDZ > > [2] - > https://forums.servethehome.com/index.php?threads/patching-intel-x520-eeprom-to-unlock-all-sfp-transceivers.24634/ > > [3] - there are two slots for transceiver in this unit - I thought I > read something > about one being different from the other, but I cannot locate > that now. > I tried inserting a second SFP+ into the second slot, but nothing > seemed to change. > I suspect it is just for lagg. > > [4] - https://forums.freebsd.org/threads/intel-x520-da2-or-x520-sr2.72173/ > > -- > Dan Langille > dan@langille.org > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > -- Dan Langille dan@langille.org From owner-freebsd-net@freebsd.org Sun Mar 22 18:30:53 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DFC0D2690DB for ; Sun, 22 Mar 2020 18:30:53 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-qv1-xf2e.google.com (mail-qv1-xf2e.google.com [IPv6:2607:f8b0:4864:20::f2e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48lmKn1Mbyz3G68 for ; Sun, 22 Mar 2020 18:30:52 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-qv1-xf2e.google.com with SMTP id p19so3424183qve.0 for ; Sun, 22 Mar 2020 11:30:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=XkGzGGWKgH48cg5FNE9OsKJusIOnyXMXympjMyVyq9U=; b=E7LvBwO9Ny4V6pV4KenX7uPuhHNIuOlu91C+3TciiPMJFETLfmyuZP6cu6F5Ziv3tj sO+6/wGrKy9jUj/t06tnFJ7yn9Nlpav+wiwNQvaAP/M045Fc3CBEdez8LJzSunLlEBD9 zr54oBUhhef0m4hAwq1yfUsQ/yCPMRrSKoV4pLwksJ2lRyg0Yu56JDsA1Tp6XIbkf8Fh h6P048D2zKiI1dFjMkVD+okPFDJqpFc8oTNg9iTbJ/1DXh2al8DQ/QzJvVmo2GimO5un 2dQ5f+6INRkbVG8NGDQooBOLlrLwo8b8+2tT56+1OECERbdPKWM3FhQbg4RT+snN9jvw fuSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=XkGzGGWKgH48cg5FNE9OsKJusIOnyXMXympjMyVyq9U=; b=n3YOdS1lJWOMpRW46gW2qvV1SYu/9tlL3r053MBVy3GKU09w8Wv7fie55yU+sBs7cq 07HpSL2s+AHuwzTYPorYi+b7yRktRqoLHdM3+EDOOO6fF5DEGXXHSpMMX7+ez9dF6iEB jxmNZOD37q6Vmyqm8spn3GxE+8ryP72C13eCehuo2VGf9Tg/0RQabZoDx0mGkKo7QFVY xuS+YQfGDrJi7hBXDzM86pPHEgIPp9Umh+kvqvcI/kykZl7ggJljMG6AHFXVTcWUxczC pk16jWeXdd2WOpvvC2fDS1E1gXn2maTdymSuvfsyD1qQNGuIdguVGG+tNmV3/nOGM5nn 23Pw== X-Gm-Message-State: ANhLgQ0KzjUlUjeg8sf+/FNGzr5TjbEt0XgT4Tro36NC/RojaZ2GGEDX Enu6dHxmeY9xkvnwhSkYu18Sn7HMLYRRj6l2cD7QxQ0q X-Google-Smtp-Source: ADFU+vvrCJZxTcsFGVbIjuU45JyXvxLtbXBgdHS28+xOfIPXEXPtNEP2qbYBLbLK4txStDNn5U/0S7vYn/TigwmQK3E= X-Received: by 2002:a0c:fd85:: with SMTP id p5mr1469705qvr.205.1584901851953; Sun, 22 Mar 2020 11:30:51 -0700 (PDT) MIME-Version: 1.0 From: Ryan Stone Date: Sun, 22 Mar 2020 14:30:41 -0400 Message-ID: Subject: Can net.iflib.min_tx_latency=0 introduce an unbounded amount of latency? To: freebsd-net Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 48lmKn1Mbyz3G68 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=E7LvBwO9; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rysto32@gmail.com designates 2607:f8b0:4864:20::f2e as permitted sender) smtp.mailfrom=rysto32@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_TWO(0.00)[2]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE_FREEMAIL(0.00)[]; IP_SCORE(0.00)[ip: (-5.05), ipnet: 2607:f8b0::/32(-0.54), asn: 15169(-0.66), country: US(-0.05)]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[e.2.f.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2020 18:30:54 -0000 I've been tracking down a bug at work that appears to be due to excessive latency introduced on the TX side of a TCP connection. In looking through the iflib code, I noticed the tunable net.iflib.min_tx_latency. My reading of the iflib code is that if a packet is enqueued to a tx ring but not posted (when min_tx_latency=0), then iflib can introduce an unbounded amount of latency if the ring is idle. Is my reading of the code correct? From owner-freebsd-net@freebsd.org Sun Mar 22 20:44:07 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C434326C228 for ; Sun, 22 Mar 2020 20:44:07 +0000 (UTC) (envelope-from sthaug@nethelp.no) Received: from bizet.nethelp.no (bizet.nethelp.no [IPv6:2001:8c0:9e04:500::1]) by mx1.freebsd.org (Postfix) with ESMTP id 48lqHS1gntz45L0 for ; Sun, 22 Mar 2020 20:44:03 +0000 (UTC) (envelope-from sthaug@nethelp.no) Received: from localhost (bizet.nethelp.no [IPv6:2001:8c0:9e04:500::1]) by bizet.nethelp.no (Postfix) with ESMTP id 4C496E607D; Sun, 22 Mar 2020 21:43:55 +0100 (CET) Date: Sun, 22 Mar 2020 21:43:55 +0100 (CET) Message-Id: <20200322.214355.415142200.sthaug@nethelp.no> To: dan@langille.org Cc: jeffrey.e.pieper@intel.com, freebsd-net@freebsd.org Subject: Re: SFP+ on PRO/10GbE From: sthaug@nethelp.no In-Reply-To: References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> X-Mailer: Mew version 6.7 on Emacs 26 / Mule 6.0 (HANACHIRUSATO) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 48lqHS1gntz45L0 X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of sthaug@nethelp.no designates 2001:8c0:9e04:500::1 as permitted sender) smtp.mailfrom=sthaug@nethelp.no X-Spamd-Result: default: False [3.06 / 15.00]; ARC_NA(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[nethelp.no]; NEURAL_SPAM_MEDIUM(0.68)[0.685,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.73)[0.734,0]; FROM_NO_DN(0.00)[]; MID_CONTAINS_FROM(1.00)[]; RCVD_COUNT_ONE(0.00)[1]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:2116, ipnet:2001:8c0::/29, country:NO]; IP_SCORE(0.24)[asn: 2116(1.23), country: NO(-0.01)]; ONCE_RECEIVED(0.10)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2020 20:44:07 -0000 > Partial success. The card is now able to use an SFP+ optic. It warns me > when the optic is installed: > > Mar 22 16:49:45 r720-01 kernel: WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules. > > I cannot use an SFP+ optic at the switch. The connection just does not happen. > > If I go back to the original SFP optic, the connection occurs, as expected at 1G. > > On the switch side, I've tried a known good optic from an existing connection. > > I could install an PRO/10GbE instead, that has a built-in transceiver. I have > two of those in use now, both working on 10G. Have you tried connected it to something other than the Unifi switch? I have two Dell R430 servers running 12.1-STABLE, using Intel X520 DP 10Gb DA/SFP+ cards with non-Intel branded LR optics. Works fine with hw.ix.unsupported_sfp=1 setting - one of them connected to a Juniper MX480 router, the other to an Extreme X670 switch. Steinar Haug, Nethelp consulting, sthaug@nethelp.no From owner-freebsd-net@freebsd.org Sun Mar 22 21:00:49 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5793326CBCB for ; Sun, 22 Mar 2020 21:00:49 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48lqfn0bPTz4Pm7 for ; Sun, 22 Mar 2020 21:00:49 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id 1137B26CBAB; Sun, 22 Mar 2020 21:00:49 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0FE2D26CBAA for ; Sun, 22 Mar 2020 21:00:49 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48lqfm5hBCz4Plp for ; Sun, 22 Mar 2020 21:00:48 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A99EF37F0 for ; Sun, 22 Mar 2020 21:00:48 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 02ML0mbM096189 for ; Sun, 22 Mar 2020 21:00:48 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 02ML0mS4096188 for net@FreeBSD.org; Sun, 22 Mar 2020 21:00:48 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202003222100.02ML0mS4096188@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: net@FreeBSD.org Subject: Problem reports for net@FreeBSD.org that need special attention Date: Sun, 22 Mar 2020 21:00:48 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2020 21:00:49 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- In Progress | 221146 | [ixgbe] Problem with second laggport In Progress | 235700 | oce(4) driver causes fatal trap 12 on boot with e New | 204438 | setsockopt() handling of kern.ipc.maxsockbuf limi New | 205592 | TCP processing in IPSec causes kernel panic New | 213410 | [carp] service netif restart causes hang only whe Open | 7556 | ppp: sl_compress_init() will fail if called anyth Open | 187835 | ngctl(8) strange behavior when adding more than 5 Open | 193452 | Dell PowerEdge 210 II -- Kernel panic bce (broadc Open | 194453 | dummynet(4): pipe config bw parameter limited to Open | 200319 | Bridge+CARP crashes/freezes Open | 202510 | [CARP] advertisements sourced from CARP IP cause Open | 207261 | netmap: Doesn't do TX sync with kqueue Open | 210726 | tcp connect() can return invalid EADDRINUSE (Eg: Open | 222273 | igb(4): Kernel panic (fatal trap 12) due to netwo Open | 225438 | panic in6_unlink_ifa() due to race Open | 225792 | ECMP is broken since tryforward() Open | 227720 | Kernel panic in ppp server Open | 230807 | if_alc(4): Driver not working for Killer Networki Open | 235524 | igb(4): Ethernet interface loses active link stat Open | 236888 | ppp daemon: Allow MTU to be overridden for PPPoE Open | 236983 | bnxt(4) VLAN not operational unless explicit "ifc Open | 237072 | netgraph(4): performance issue [on HardenedBSD]? Open | 237391 | route get returns no result for network addresses Open | 237840 | Removed dummynet dependency on ipfw Open | 238324 | Add XG-C100C/AQtion AQC107 10GbE NIC driver Open | 240530 | netgraph/ng_source: Allow ng_source to inject int Open | 240944 | em(4): Crash with Intel 82571EB NIC with AMD Pile Open | 240969 | netinet6: Neighbour reachability detection broken Open | 241106 | tun/ppp: panic: vm_fault: fault on nofault entry Open | 241162 | Panic in closefp() triggered by nginx (uwsgi with Open | 241191 | route flush panic with RADIX_MPATH Open | 243463 | ix0: Watchdog timeout Open | 244066 | divert: Add sysctls for divert socket send and re Open | 118111 | rc: network.subr Add MAC address based interface 34 problems total for which you should take action. From owner-freebsd-net@freebsd.org Sun Mar 22 21:11:51 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 14C6126DABF for ; Sun, 22 Mar 2020 21:11:51 +0000 (UTC) (envelope-from dan@langille.org) Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48lqvV1w0tz4dqL for ; Sun, 22 Mar 2020 21:11:50 +0000 (UTC) (envelope-from dan@langille.org) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.west.internal (Postfix) with ESMTP id 16FFA449; Sun, 22 Mar 2020 17:11:47 -0400 (EDT) Received: from imap36 ([10.202.2.86]) by compute2.internal (MEProxy); Sun, 22 Mar 2020 17:11:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=langille.org; h= mime-version:message-id:in-reply-to:references:date:from:to:cc :subject:content-type; s=fm3; bh=ACaBsq2Z0M+sSFHbQaq5DST5iFXJtmL LIULUc10tWOI=; b=tYhQGOe/pbtlqt7tGTKX6Fq9f2dXSVSYXBPA6JaOorq+WOo yXhkzArcOurMK3MbvJ4ccn/hAkdm/cqEJT0D3q7PmxupweloS9vW84krxSeQQswB hOoNFkZLWHKp/v7I37n2a8e/z9Fgb74G0tBm5NoLRIxn6MbL8JDZxEpwch/cGem7 8S2sIlO1v2uc0v2u9gwWImXTq4pk+N8IvfbKsG3iUto0O25EOAWhw6ivys+TrVXU 8MjP23Y0TF500Mcp87tQo0BOXUIRz1Ku2K/VTMTDV8O4PRI/EmOLbuhUgcAm4S1l fO9atXj+e79/yIB/ZErXwA9wjqvWNSVCRSbbMpw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=ACaBsq 2Z0M+sSFHbQaq5DST5iFXJtmLLIULUc10tWOI=; b=vf5NH5JqVse7ZOclDRSyDD AMA7AeANPQ/ZEh+8bfCTDrHHpRMl0uztD62Lo8Uyt0KrPl/yoJvtrI+G4Stf82Kk rirSLAXp4AyKadDutdnEJCuWH6WIko6TV/Tj5+UY7ITH+tCJN1JZ+iithDHjBnXV S8TaCsugYxwvzFfYO/PfEx4+RSPnFE954GOgeiMwyYe1dxgS7W9sQp9N7SqMRECd WtGaOzMmZWplV4BppYJeHTpeqMXDyBveyxqjlcld5w8Thm9RGR7cIxTnDRiPuxzb Szo38GUBM1HM2PqHfARt58k85op3XNjLkQ3QCwy5Sk0k0PL1YXjJelT0BaTj/YbA == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrudegiedgudegjecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefofgggkfgjfhffhffvufgtsehttdertderredtnecuhfhrohhmpedfffgr nhcunfgrnhhgihhllhgvfdcuoegurghnsehlrghnghhilhhlvgdrohhrgheqnecuvehluh hsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepuggrnheslhgrnhhg ihhllhgvrdhorhhg X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 9AAF71880062; Sun, 22 Mar 2020 17:11:45 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.7-1021-g152deaf-fmstable-20200319v1 Mime-Version: 1.0 Message-Id: In-Reply-To: <20200322.214355.415142200.sthaug@nethelp.no> References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> <20200322.214355.415142200.sthaug@nethelp.no> Date: Sun, 22 Mar 2020 17:11:20 -0400 From: "Dan Langille" To: sthaug@nethelp.no Cc: "Pieper, Jeffrey E" , freebsd-net@freebsd.org Subject: Re: SFP+ on PRO/10GbE Content-Type: text/plain X-Rspamd-Queue-Id: 48lqvV1w0tz4dqL X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=langille.org header.s=fm3 header.b=tYhQGOe/; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=vf5NH5Jq; dmarc=pass (policy=none) header.from=langille.org; spf=pass (mx1.freebsd.org: domain of dan@langille.org designates 64.147.123.24 as permitted sender) smtp.mailfrom=dan@langille.org X-Spamd-Result: default: False [-5.58 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.997,0]; R_DKIM_ALLOW(-0.20)[langille.org:s=fm3,messagingengine.com:s=fm2]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.24]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; URIBL_BLOCKED(0.00)[langille.org.multi.uribl.com,messagingengine.com.multi.uribl.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[langille.org:+,messagingengine.com:+]; DMARC_POLICY_ALLOW(-0.50)[langille.org,none]; IP_SCORE(-3.50)[ip: (-9.82), ipnet: 64.147.123.0/24(-4.92), asn: 11403(-2.69), country: US(-0.05)]; RCVD_IN_DNSWL_LOW(-0.10)[24.123.147.64.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; MID_RHS_WWW(0.50)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2020 21:11:51 -0000 On Sun, Mar 22, 2020, at 4:43 PM, sthaug@nethelp.no wrote: > > Partial success. The card is now able to use an SFP+ optic. It warns me > > when the optic is installed: > > > > Mar 22 16:49:45 r720-01 kernel: WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules. > > > > I cannot use an SFP+ optic at the switch. The connection just does not happen. > > > > If I go back to the original SFP optic, the connection occurs, as expected at 1G. > > > > On the switch side, I've tried a known good optic from an existing connection. > > > > I could install an PRO/10GbE instead, that has a built-in transceiver. I have > > two of those in use now, both working on 10G. > > Have you tried connected it to something other than the Unifi switch? The only SFP+ capable switches I have are Unifi. I just tried the other switch (US-48) which had one SFP+ port free. Same issues there. > I have two Dell R430 servers running 12.1-STABLE, using Intel X520 DP > 10Gb DA/SFP+ cards with non-Intel branded LR optics. Works fine with > hw.ix.unsupported_sfp=1 setting - one of them connected to a Juniper > MX480 router, the other to an Extreme X670 switch. I had Extreme switches here before. They are long gone now. Nice things though. Thank you. -- Dan Langille dan@langille.org From owner-freebsd-net@freebsd.org Sun Mar 22 23:03:50 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A27CC2704DA for ; Sun, 22 Mar 2020 23:03:50 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from mail.in-addr.com (mail.in-addr.com [IPv6:2a01:4f8:191:61e8::2525:2525]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48ltNk2SdZz3KKl for ; Sun, 22 Mar 2020 23:03:50 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by mail.in-addr.com with local (Exim 4.92.3 (FreeBSD)) (envelope-from ) id 1jG9d5-0002Gg-PI; Sun, 22 Mar 2020 23:03:39 +0000 Date: Sun, 22 Mar 2020 23:03:39 +0000 From: Gary Palmer To: Dan Langille Cc: sthaug@nethelp.no, "Pieper, Jeffrey E" , freebsd-net@freebsd.org Subject: Re: SFP+ on PRO/10GbE Message-ID: <20200322230339.GB5808@in-addr.com> References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> <20200322.214355.415142200.sthaug@nethelp.no> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gpalmer@freebsd.org X-SA-Exim-Scanned: No (on mail.in-addr.com); SAEximRunCond expanded to false X-Rspamd-Queue-Id: 48ltNk2SdZz3KKl X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [1.51 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_SPAM_MEDIUM(0.71)[0.711,0]; NEURAL_SPAM_LONG(0.80)[0.798,0]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2020 23:03:50 -0000 On Sun, Mar 22, 2020 at 05:11:20PM -0400, Dan Langille wrote: > On Sun, Mar 22, 2020, at 4:43 PM, sthaug@nethelp.no wrote: > > > Partial success. The card is now able to use an SFP+ optic. It warns me > > > when the optic is installed: > > > > > > Mar 22 16:49:45 r720-01 kernel: WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules. > > > > > > I cannot use an SFP+ optic at the switch. The connection just does not happen. > > > > > > If I go back to the original SFP optic, the connection occurs, as expected at 1G. > > > > > > On the switch side, I've tried a known good optic from an existing connection. > > > > > > I could install an PRO/10GbE instead, that has a built-in transceiver. I have > > > two of those in use now, both working on 10G. > > > > Have you tried connected it to something other than the Unifi switch? > > The only SFP+ capable switches I have are Unifi. > > I just tried the other switch (US-48) which had one SFP+ port free. Same issues there. Did the Unifi switch see the SFP+ optics on the switch end? What happens if you take the SFP+ module from the server and put it in the other Unifi switch and try establishing a link between the two switches? Gary From owner-freebsd-net@freebsd.org Sun Mar 22 23:24:28 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 63130270E88 for ; Sun, 22 Mar 2020 23:24:28 +0000 (UTC) (envelope-from dan@langille.org) Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48ltrV3B1hz4Gff; Sun, 22 Mar 2020 23:24:25 +0000 (UTC) (envelope-from dan@langille.org) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.west.internal (Postfix) with ESMTP id B4D153FE; Sun, 22 Mar 2020 19:24:22 -0400 (EDT) Received: from imap36 ([10.202.2.86]) by compute2.internal (MEProxy); Sun, 22 Mar 2020 19:24:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=langille.org; h= mime-version:message-id:in-reply-to:references:date:from:to:cc :subject:content-type; s=fm3; bh=gVKTcgZCApwTa7XrcTt17jSWlhAcEy0 Uq1/zXoJbnfE=; b=t3nU32ZVZa1XF3bdfqR9FlFGTRicsjC1+BKJxRzFIi/zSwM L9mrozAFhU8JSQrZE2VTq5g04gzMGtibcR1K/xtIQtx4IO6SCnHG7/4abQCIWhph 5rEexZdtAyT5B+o5HUO2Ak5UJ3B2QGe1eSfGsxPrEPUC+Flh9feFRrrDiycl/FNi fMBl8uhiDZiYpI3cjq4rfVmU4Xg6cvw4OT666tiLRI+e8PTqfcbLaXd97tmXHI4d kZBXe12HeIzdkWDDP8WKzIgo4reqICXUvd6J9s3fA63RrA433xfztom5Nf2Dlb3H T8+iTOP7E7Hkb12zV7Ld8fSEnD+/lRHn/kPjXXQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=gVKTcg ZCApwTa7XrcTt17jSWlhAcEy0Uq1/zXoJbnfE=; b=qYrYxdStqQDtRw7iovrJVI AOeq5nHhRX/QMh+JKl5HchXZvzX4kPKtfJ5dlSPLDlAoZieJAmPMv4TY4GYAmGgF TUSug08RLCiDQJlWgxJtwaJW1ukQOr+4ppclM3l2OW5bQGm7ACCevP0yLx6bCUiE mc+dp3YsO5FBrUoYn/X/81KEDeDh6AjDE8ufnmClkB1ch3BLglV6oAZDP7vsMV/J T+4oiw40xRVQW4A23B+hrmwnCU1ak3EXbFPjYrcG4Jhn7HaDXCkdzy+vRB851oY0 oTKyLiCfMiVF8xeH1hNpNT9t34amfiZyr3XK6faLzk0seyrQAlM44HOA8x9cAcyQ == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrudegiedgudejhecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefofgggkfgjfhffhffvufgtsehttdertderredtnecuhfhrohhmpedfffgr nhcunfgrnhhgihhllhgvfdcuoegurghnsehlrghnghhilhhlvgdrohhrgheqnecuvehluh hsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepuggrnheslhgrnhhg ihhllhgvrdhorhhg X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id BE7431880062; Sun, 22 Mar 2020 19:24:21 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.7-1021-g152deaf-fmstable-20200319v1 Mime-Version: 1.0 Message-Id: <8141b371-5cdf-4f82-8959-5fa1071aa6dd@www.fastmail.com> In-Reply-To: <20200322230339.GB5808@in-addr.com> References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> <20200322.214355.415142200.sthaug@nethelp.no> <20200322230339.GB5808@in-addr.com> Date: Sun, 22 Mar 2020 19:24:01 -0400 From: "Dan Langille" To: "Gary Palmer" Cc: sthaug@nethelp.no, "Pieper, Jeffrey E" , freebsd-net@freebsd.org Subject: Re: SFP+ on PRO/10GbE Content-Type: text/plain X-Rspamd-Queue-Id: 48ltrV3B1hz4Gff X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=langille.org header.s=fm3 header.b=t3nU32ZV; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=qYrYxdSt; dmarc=pass (policy=none) header.from=langille.org; spf=pass (mx1.freebsd.org: domain of dan@langille.org designates 64.147.123.24 as permitted sender) smtp.mailfrom=dan@langille.org X-Spamd-Result: default: False [-5.58 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(-0.20)[langille.org:s=fm3,messagingengine.com:s=fm2]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.24]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; URIBL_BLOCKED(0.00)[langille.org.multi.uribl.com,messagingengine.com.multi.uribl.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[langille.org:+,messagingengine.com:+]; DMARC_POLICY_ALLOW(-0.50)[langille.org,none]; IP_SCORE(-3.49)[ip: (-9.81), ipnet: 64.147.123.0/24(-4.92), asn: 11403(-2.69), country: US(-0.05)]; RCVD_IN_DNSWL_LOW(-0.10)[24.123.147.64.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; MID_RHS_WWW(0.50)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Mar 2020 23:24:28 -0000 On Sun, Mar 22, 2020, at 7:03 PM, Gary Palmer wrote: > On Sun, Mar 22, 2020 at 05:11:20PM -0400, Dan Langille wrote: > > On Sun, Mar 22, 2020, at 4:43 PM, sthaug@nethelp.no wrote: > > > > Partial success. The card is now able to use an SFP+ optic. It warns me > > > > when the optic is installed: > > > > > > > > Mar 22 16:49:45 r720-01 kernel: WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules. > > > > > > > > I cannot use an SFP+ optic at the switch. The connection just does not happen. > > > > > > > > If I go back to the original SFP optic, the connection occurs, as expected at 1G. > > > > > > > > On the switch side, I've tried a known good optic from an existing connection. > > > > > > > > I could install an PRO/10GbE instead, that has a built-in transceiver. I have > > > > two of those in use now, both working on 10G. > > > > > > Have you tried connected it to something other than the Unifi switch? > > > > The only SFP+ capable switches I have are Unifi. > > > > I just tried the other switch (US-48) which had one SFP+ port free. Same issues there. > > Did the Unifi switch see the SFP+ optics on the switch end? That port never lit up on the switch. I didn't check anything else, switch-related. > What happens if you take the SFP+ module from the server and put it in the > other Unifi switch and try establishing a link between the two switches? The two switches are already joined via SFP+ so I bet that would shut down the port in question. -- Dan Langille dan@langille.org From owner-freebsd-net@freebsd.org Mon Mar 23 04:50:23 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D140B27848E for ; Mon, 23 Mar 2020 04:50:23 +0000 (UTC) (envelope-from pkelsey@gmail.com) Received: from mail-io1-f54.google.com (mail-io1-f54.google.com [209.85.166.54]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48m24Z3DVvz4Lr5 for ; Mon, 23 Mar 2020 04:50:22 +0000 (UTC) (envelope-from pkelsey@gmail.com) Received: by mail-io1-f54.google.com with SMTP id h131so12857982iof.1 for ; Sun, 22 Mar 2020 21:50:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=iixK0l1EVdyhjj/FQyb0YkG8SnpaaTCcrU6sy1F7CdM=; b=FK9kSF+arncjvBb1hnvCxxbQOf2ete310JMutjNWLu8PIho0uPHHDjo8qJo0HqTymo l9XjbscUZBEKLndhsOL5oj5HyM85V+9T88Ub9GIARzcYfxkSg0/DZkoQeCvK86xaU4jv CNxIK7zDQo+ujnJKKPDDRsIt+DSBL80r2D1imgZFPIa9NUrusoH99ALidTTS59rzdXts gK2+cnU3PCdDM4oX3omaXktN6VZeglFXt78ixjuYr5X0o9+izjh3G2bXJ0McXqJPy6AJ EtMzMKp6HBpacJJquYSapdegcVLMYhJjawd5545ba/vvABQ4D16InVqUkbqVzc2jWzbV ilHw== X-Gm-Message-State: ANhLgQ1UYf+en4krNUPalObncNLfWhGhg6UYwMUgM6yPC8xn25XJA6eT y0HXe0dImewJBLjnkwKCU2o8sVWmehkBcRg+CwM= X-Google-Smtp-Source: ADFU+vumZadE6Q/28S1fG61pYcI+e92PlcQfrF2UDW5QtFd2WAHGGtVfJAjrr1Q26OE/e+/GQmzSDEB3TlKmc7sZkXI= X-Received: by 2002:a02:cc6f:: with SMTP id j15mr19145040jaq.80.1584939020855; Sun, 22 Mar 2020 21:50:20 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Patrick Kelsey Date: Mon, 23 Mar 2020 00:50:08 -0400 Message-ID: Subject: Re: Can net.iflib.min_tx_latency=0 introduce an unbounded amount of latency? To: Ryan Stone Cc: freebsd-net X-Rspamd-Queue-Id: 48m24Z3DVvz4Lr5 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pkelsey@gmail.com designates 209.85.166.54 as permitted sender) smtp.mailfrom=pkelsey@gmail.com X-Spamd-Result: default: False [-2.63 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[54.166.85.209.list.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-1.63)[ip: (-4.90), ipnet: 209.85.128.0/17(-2.60), asn: 15169(-0.60), country: US(-0.05)]; FORGED_SENDER(0.30)[pkelsey@freebsd.org,pkelsey@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[pkelsey@freebsd.org,pkelsey@gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2020 04:50:23 -0000 On Sun, Mar 22, 2020 at 2:31 PM Ryan Stone wrote: > I've been tracking down a bug at work that appears to be due to > excessive latency introduced on the TX side of a TCP connection. In > looking through the iflib code, I noticed the tunable > net.iflib.min_tx_latency. My reading of the iflib code is that if a > packet is enqueued to a tx ring but not posted (when > min_tx_latency=0), then iflib can introduce an unbounded amount of > latency if the ring is idle. Is my reading of the code correct? > > Let's see...in the case described above, the next iflib_timer() invocation (max 500ms later) should find that txq->ift_db_pending is non-zero, which will cause it to enqueue the TX task, which will invoke _task_fn_tx(), which will then enqueue the special marker (void **)&txq, and one way or another, iflib_txq_drain() will be called. Looking at iflib_txq_drain() it appears that the first call to iflib_txd_db_check() should post the pending packets either because a non-zero number of slots were just reclaimed, or the check for txq->ift_db_pending >= TXQ_MAX_DB_DEFERRED(...) is satisfied, unless there are too many posted packets in the hardware queue that aren't being drained by the hardware (this would result in the in_use parameter remaining elevated such that a small number of pending packets would not exceed the threshold, and also no slot reclaim occurring). -Patrick From owner-freebsd-net@freebsd.org Mon Mar 23 05:00:21 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B045A2788BC for ; Mon, 23 Mar 2020 05:00:21 +0000 (UTC) (envelope-from vas@sibptus.ru) Received: from admin.sibptus.ru (admin.sibptus.ru [IPv6:2001:19f0:5001:21dc::10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48m2J44dMwz4Y40 for ; Mon, 23 Mar 2020 05:00:20 +0000 (UTC) (envelope-from vas@sibptus.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sibptus.ru; s=20181118; h=Message-ID:Subject:To:From:Date:In-Reply-To; bh=VfEnOLRWTsu6LJnGc4G4/os8GgTKfLzV9yHjkwL9leo=; b=XXmqMN8MrINEBRtRn3F+m1xXOp KnVZrkpSYQYfOXVZ2OI957Xd1LU0RqiihhjepbhcS3fgqoeEMi2ggWkck3nGPQj3vTGuzPzaAe9JK Dqo//aMpCT5Xri4LgjOUM0GwL4LtZ6VL6WdIdHWuEpgyw2rP4MtEsMuv6CbFUjiZHD7A=; Received: from vas by admin.sibptus.ru with local (Exim 4.93.0.4 (FreeBSD)) (envelope-from ) id 1jGFC8-000DB6-Sa for freebsd-net@freebsd.org; Mon, 23 Mar 2020 12:00:12 +0700 Date: Mon, 23 Mar 2020 12:00:12 +0700 From: Victor Sudakov To: freebsd-net@freebsd.org Subject: IP MTU on gif and gre interfaces (with and without IPSec encryption) Message-ID: <20200323050012.GA50490@admin.sibptus.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="zYM0uCDKw75PZbzx" Content-Disposition: inline X-PGP-Key: http://admin.sibptus.ru/~vas/ X-PGP-Fingerprint: 10E3 1171 1273 E007 C2E9 3532 0DA4 F259 9B5E C634 X-Rspamd-Queue-Id: 48m2J44dMwz4Y40 X-Spamd-Bar: -------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=sibptus.ru header.s=20181118 header.b=XXmqMN8M; dmarc=pass (policy=none) header.from=sibptus.ru; spf=pass (mx1.freebsd.org: domain of vas@sibptus.ru designates 2001:19f0:5001:21dc::10 as permitted sender) smtp.mailfrom=vas@sibptus.ru X-Spamd-Result: default: False [-8.46 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.996,0]; R_DKIM_ALLOW(-0.20)[sibptus.ru:s=20181118]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-3.36)[ip: (-9.89), ipnet: 2001:19f0:5000::/38(-4.94), asn: 20473(-1.94), country: US(-0.05)]; DKIM_TRACE(0.00)[sibptus.ru:+]; DMARC_POLICY_ALLOW(-0.50)[sibptus.ru,none]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:20473, ipnet:2001:19f0:5000::/38, country:US]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2020 05:00:21 -0000 --zYM0uCDKw75PZbzx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Dear Colleagues, If the MTU on the external Ethernet interface is 1500, it should be safe to configure the gif(4) interfaces with mtu=3D1480 and gre(4) interfaces with mtu=3D1476, correct? I've noticed that a newly created gre0 interface has the expected "mtu 1476" value, but a newly created gif0 interface has "mtu 1280", why would the default be so low? A second question. If the gif and gre tunnels will be wrapped in IPSec transport mode, does it make sense to set the MTU on the corresponding gif and gre interfaces to some lower value? --=20 Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ --zYM0uCDKw75PZbzx Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJeeEJcAAoJEA2k8lmbXsY0L7gH/28dlW0TzJvrYCimZPgvq9UP X4/ur29r5M0IH4WldQfSIm1uoMjB7I4LallXU12K27RTsZcXwUBwHzjINaWMsNbi Jj1b1bAA0DdiDCLQeeJ1nWSMX/GYRq2cd9NR/AFJ8SQyZQxoiCk7OcRkwbc7eS9p FQ9Cqf/oBaEGcgtjUOcHs+Ahv1EM4nFF1qkgdXsGiCrugQ7nr9yt2ZI/3SCui+e8 H3bFqgYfE1El0skJ9kYNlv1vC5XpKvwXGdIy5l22ZTZ1igueajDr9V3L79ObIluI nVHiieTSaDewhVJgthhAGj4pF8BkeDyk0xwnHEMGlbDLm6trvKYgfDYPViALm68= =21Ty -----END PGP SIGNATURE----- --zYM0uCDKw75PZbzx-- From owner-freebsd-net@freebsd.org Mon Mar 23 07:17:17 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DA2A027ABB4 for ; Mon, 23 Mar 2020 07:17:17 +0000 (UTC) (envelope-from vas@sibptus.ru) Received: from admin.sibptus.ru (admin.sibptus.ru [IPv6:2001:19f0:5001:21dc::10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48m5L46FYnz4F6q for ; Mon, 23 Mar 2020 07:17:16 +0000 (UTC) (envelope-from vas@sibptus.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sibptus.ru; s=20181118; h=In-Reply-To:Message-ID:Subject:To:From:Date; bh=fjhJf3e8DgEOZsWpUkopH8f3vZ8BsDrK+dMWbaCP9qY=; b=ac794zKXS+FeZtTOex16iw72kn L1G3DPf/uV/rDhhpbXsrPaBhcuuL8EBkPWvN24eliiXw1XMIWSJO/WPdVI9hsdtucprAA06Dk8AYd w9fZRh878Ah5BLBzPeNlX4nnwnnIvzwCj9bQRXphQBu5i1M/DPZqdiOyWPFhUGZrjQdI=; Received: from vas by admin.sibptus.ru with local (Exim 4.93.0.4 (FreeBSD)) (envelope-from ) id 1jGHKl-000E9c-FL for freebsd-net@freebsd.org; Mon, 23 Mar 2020 14:17:15 +0700 Date: Mon, 23 Mar 2020 14:17:15 +0700 From: Victor Sudakov To: freebsd-net@freebsd.org Subject: Re: IP MTU on gif and gre interfaces (with and without IPSec encryption) Message-ID: <20200323071715.GB54231@admin.sibptus.ru> References: <20200323050012.GA50490@admin.sibptus.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="KsGdsel6WgEHnImy" Content-Disposition: inline In-Reply-To: <20200323050012.GA50490@admin.sibptus.ru> X-PGP-Key: http://admin.sibptus.ru/~vas/ X-PGP-Fingerprint: 10E3 1171 1273 E007 C2E9 3532 0DA4 F259 9B5E C634 X-Rspamd-Queue-Id: 48m5L46FYnz4F6q X-Spamd-Bar: -------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=sibptus.ru header.s=20181118 header.b=ac794zKX; dmarc=pass (policy=none) header.from=sibptus.ru; spf=pass (mx1.freebsd.org: domain of vas@sibptus.ru designates 2001:19f0:5001:21dc::10 as permitted sender) smtp.mailfrom=vas@sibptus.ru X-Spamd-Result: default: False [-8.46 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.996,0]; R_DKIM_ALLOW(-0.20)[sibptus.ru:s=20181118]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-3.36)[ip: (-9.89), ipnet: 2001:19f0:5000::/38(-4.94), asn: 20473(-1.94), country: US(-0.05)]; DKIM_TRACE(0.00)[sibptus.ru:+]; DMARC_POLICY_ALLOW(-0.50)[sibptus.ru,none]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:20473, ipnet:2001:19f0:5000::/38, country:US]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2020 07:17:17 -0000 --KsGdsel6WgEHnImy Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Victor Sudakov wrote: >=20 > If the MTU on the external Ethernet interface is 1500, it should be safe > to configure the gif(4) interfaces with mtu=3D1480 and gre(4) > interfaces with mtu=3D1476, correct? >=20 > I've noticed that a newly created gre0 interface has the expected "mtu 14= 76" > value, but a newly created gif0 interface has "mtu 1280", why would the > default be so low? >=20 > A second question. If the gif and gre tunnels will be wrapped in > IPSec transport mode, does it make sense to set the MTU on the > corresponding gif and gre interfaces to some lower value? And to make the matter even more mysterious, if_ipsec(4) which is essentially the same as gif, is created by default with mtu=3D1400 for some reason. --=20 Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ --KsGdsel6WgEHnImy Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJeeGJ7AAoJEA2k8lmbXsY0CyoH/3Rtm1q5B3WuthzkTX0/1+B0 SjHIAaUxUyajiIN7+E4DuqHRvnLoMwM5omnGFgek7hmbJFhpgIzSKuHjmsCNNHX6 BpmeRS2YFlnucq365nr/5yaRJh3VM0DT+K1LPJ9csa8jkl+SMoheEMPGlRQR5lMY 79rvqCYn4lZ3wZ6dAJw0PhY9nEbsXpIp0RuA87Wa0x+zdZg7pcnXEyGdJxc2u2qz 7D4G8Cw8RdN9lcUDeqvUCc7hG1pl9AP6iz7pTo5WGFbmYJ9c+qSYkb8smFcnx04G IdAN4kR4FBrttamrM/UqhuMkAjNCTAmWwmmVtBmETMftrZCkpp5R7KGQfuIi+ro= =PmYG -----END PGP SIGNATURE----- --KsGdsel6WgEHnImy-- From owner-freebsd-net@freebsd.org Mon Mar 23 08:17:29 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3BF0327C4A4 for ; Mon, 23 Mar 2020 08:17:29 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from mailgate2.uni-hannover.de (mailgate2.uni-hannover.de [130.75.2.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48m6gX28qGz4N7B for ; Mon, 23 Mar 2020 08:17:27 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mailgate2.uni-hannover.de (Postfix) with ESMTPS id B19E915B5 for ; Mon, 23 Mar 2020 09:17:25 +0100 (CET) Received: from comet2.terra.ger ([130.75.117.49]) by intranet.aei.uni-hannover.de (IBM Domino Release 9.0.1FP8) with ESMTP id 2020032309172475-126807 ; Mon, 23 Mar 2020 09:17:24 +0100 Date: Mon, 23 Mar 2020 09:17:24 +0100 From: Gerrit Kuehn To: freebsd-net@freebsd.org Subject: Re: SFP+ on PRO/10GbE Message-ID: <20200323091724.6d45245f@comet2.terra.ger> In-Reply-To: References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> <20200322.214355.415142200.sthaug@nethelp.no> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; amd64-portbld-freebsd11.2) MIME-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 9.0.1FP8|February 23, 2017) at 23/03/2020 09:17:24, Serialize by Router on intranet/aei-hannover(Release 9.0.1FP8|February 23, 2017) at 23/03/2020 09:17:24, Serialize complete at 23/03/2020 09:17:24 X-TNEFEvaluated: 1 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 48m6gX28qGz4N7B X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of gerrit.kuehn@aei.mpg.de has no SPF policy when checking 130.75.2.114) smtp.mailfrom=gerrit.kuehn@aei.mpg.de X-Spamd-Result: default: False [2.67 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_DN_NONE(0.00)[]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[mpg.de]; RCVD_IN_DNSWL_MED(-0.20)[114.2.75.130.list.dnswl.org : 127.0.11.2]; NEURAL_SPAM_LONG(1.00)[1.000,0]; NEURAL_SPAM_MEDIUM(0.96)[0.962,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:680, ipnet:130.75.0.0/16, country:DE]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(0.01)[asn: 680(0.07), country: DE(-0.02)] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2020 08:17:29 -0000 On Sun, 22 Mar 2020 17:11:20 -0400 "Dan Langille" wrote: > > > I cannot use an SFP+ optic at the switch. The connection just > > > does not happen. > I just tried the other switch (US-48) which had one SFP+ port free. > Same issues there. Does the switch allow to select the link capacity manually? I've seen switches failing to do auto-negotiation, falling through with no link at all. cu Gerrit From owner-freebsd-net@freebsd.org Mon Mar 23 08:24:57 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E2E5027C82C for ; Mon, 23 Mar 2020 08:24:57 +0000 (UTC) (envelope-from sergey@akhmatov.ru) Received: from dutch.chaklun.ru (dutch.chaklun.ru [IPv6:2a01:7c8:aab0:135::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48m6r842LLz4Yvr for ; Mon, 23 Mar 2020 08:24:56 +0000 (UTC) (envelope-from sergey@akhmatov.ru) Received: from [176.195.68.100] (helo=[192.168.1.3]) by dutch.chaklun.ru with esmtpsa (TLS1.3) tls TLS_AES_128_GCM_SHA256 (Exim 4.93.0.4 (FreeBSD)) (envelope-from ) id 1jGIO5-0001E6-RQ; Mon, 23 Mar 2020 11:24:45 +0300 Subject: Re: SFP+ on PRO/10GbE To: Dan Langille , "Pieper, Jeffrey E" , "freebsd-net@freebsd.org" References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> From: Sergey Akhmatov Message-ID: <88679204-4c23-7fce-681d-ab633b326fef@akhmatov.ru> Date: Mon, 23 Mar 2020 11:24:43 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 48m6r842LLz4Yvr X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of sergey@akhmatov.ru has no SPF policy when checking 2a01:7c8:aab0:135::1) smtp.mailfrom=sergey@akhmatov.ru X-Spamd-Result: default: False [-1.78 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; IP_SCORE(-1.11)[ipnet: 2a01:7c8::/32(-3.69), asn: 20857(-1.87), country: NL(0.03)]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[akhmatov.ru]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-0.76)[-0.759,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.82)[-0.818,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:20857, ipnet:2a01:7c8::/32, country:NL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2020 08:24:58 -0000 On 2020-03-22 8:35 PM, Dan Langille wrote: > Hello Jeff, > > Partial success. The card is now able to use an SFP+ optic. It warns me > when the optic is installed: > > Mar 22 16:49:45 r720-01 kernel: WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules. Do you see correct media type and signal levels running: $ ifconfig -vm ix0 ? From owner-freebsd-net@freebsd.org Mon Mar 23 09:08:56 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 28C0327D816 for ; Mon, 23 Mar 2020 09:08:56 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from mail.punkt.de (mail.punkt.de [217.29.41.227]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48m7pv0tyxz4SD4 for ; Mon, 23 Mar 2020 09:08:54 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from [IPv6:2003:a:d59:3800:5cc8:2c6c:7020:8519] (unknown [IPv6:2003:a:d59:3800:5cc8:2c6c:7020:8519]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.punkt.de (Postfix) with ESMTPSA id C9C272A365; Mon, 23 Mar 2020 10:08:46 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: IP MTU on gif and gre interfaces (with and without IPSec encryption) From: "Patrick M. Hausen" In-Reply-To: <20200323050012.GA50490@admin.sibptus.ru> Date: Mon, 23 Mar 2020 10:08:40 +0100 Cc: freebsd-net@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <7D560C6A-EB51-414A-A3A1-18587D40C218@punkt.de> References: <20200323050012.GA50490@admin.sibptus.ru> To: Victor Sudakov X-Mailer: Apple Mail (2.3445.104.11) X-Rspamd-Queue-Id: 48m7pv0tyxz4SD4 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.41.227 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-0.64 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.26)[-0.257,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[punkt.de]; NEURAL_HAM_LONG(-0.32)[-0.325,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-0.26)[ipnet: 217.29.32.0/20(-0.71), asn: 16188(-0.56), country: DE(-0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2020 09:08:56 -0000 Hi all, > Am 23.03.2020 um 06:00 schrieb Victor Sudakov : > I've noticed that a newly created gre0 interface has the expected "mtu = 1476" > value, but a newly created gif0 interface has "mtu 1280", why would = the > default be so low? gif is frequently used as the innermost encapsulation like in gif tunnel across host mode IPsec. Then there might be PPPoE, too. Possibly a VLAN tag ... So most probably the idea is to go as low as possible for arbitrary = additional encapsulations while still satisfying the minimum requirement for IPv6. For the others: no idea, I guess historical ;-) Patrick --=20 punkt.de GmbH Patrick M. Hausen .infrastructure Kaiserallee 13a 76133 Karlsruhe Tel. +49 721 9109500 https://infrastructure.punkt.de info@punkt.de AG Mannheim 108285 Gesch=C3=A4ftsf=C3=BChrer: J=C3=BCrgen Egeling, Daniel Lienert, Fabian = Stein From owner-freebsd-net@freebsd.org Mon Mar 23 09:21:08 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0285C27DC24 for ; Mon, 23 Mar 2020 09:21:08 +0000 (UTC) (envelope-from vas@sibptus.ru) Received: from admin.sibptus.ru (admin.sibptus.ru [IPv6:2001:19f0:5001:21dc::10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48m84y2W5Sz4fMb for ; Mon, 23 Mar 2020 09:21:06 +0000 (UTC) (envelope-from vas@sibptus.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sibptus.ru; s=20181118; h=In-Reply-To:Message-ID:Subject:To:From:Date; bh=470PASQIxss7+2+2ARdYLkHTX/bmE/X/lXJeCv2uGZc=; b=KMnY2WYWVLG8go9wkkKUGde6ef 3od3t4G5hxLwDAkho1xhwV/6J2a1UouSDk+TihT5mCSVI+KdLD81KyquwWkVjUs84FXIj2f/O4I/B rBNzI75hjOKJ0x98cYk304D+ZD1sBlDZrYH4RQDppadxVq3eRx+EYOsLOjAfFam0Ujfg=; Received: from vas by admin.sibptus.ru with local (Exim 4.93.0.4 (FreeBSD)) (envelope-from ) id 1jGJGa-000EpO-HK; Mon, 23 Mar 2020 16:21:04 +0700 Date: Mon, 23 Mar 2020 16:21:04 +0700 From: Victor Sudakov To: "Patrick M. Hausen" Cc: freebsd-net@freebsd.org Subject: Re: IP MTU on gif and gre interfaces (with and without IPSec encryption) Message-ID: <20200323092104.GA56721@admin.sibptus.ru> References: <20200323050012.GA50490@admin.sibptus.ru> <7D560C6A-EB51-414A-A3A1-18587D40C218@punkt.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="envbJBWh7q8WU6mo" Content-Disposition: inline In-Reply-To: <7D560C6A-EB51-414A-A3A1-18587D40C218@punkt.de> X-PGP-Key: http://admin.sibptus.ru/~vas/ X-PGP-Fingerprint: 10E3 1171 1273 E007 C2E9 3532 0DA4 F259 9B5E C634 X-Rspamd-Queue-Id: 48m84y2W5Sz4fMb X-Spamd-Bar: -------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=sibptus.ru header.s=20181118 header.b=KMnY2WYW; dmarc=pass (policy=none) header.from=sibptus.ru; spf=pass (mx1.freebsd.org: domain of vas@sibptus.ru designates 2001:19f0:5001:21dc::10 as permitted sender) smtp.mailfrom=vas@sibptus.ru X-Spamd-Result: default: False [-8.46 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; R_DKIM_ALLOW(-0.20)[sibptus.ru:s=20181118]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; IP_SCORE(-3.36)[ip: (-9.89), ipnet: 2001:19f0:5000::/38(-4.94), asn: 20473(-1.94), country: US(-0.05)]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[sibptus.ru:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[sibptus.ru,none]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:20473, ipnet:2001:19f0:5000::/38, country:US]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2020 09:21:08 -0000 --envbJBWh7q8WU6mo Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Patrick M. Hausen wrote: >=20 > > Am 23.03.2020 um 06:00 schrieb Victor Sudakov : > > I've noticed that a newly created gre0 interface has the expected "mtu = 1476" > > value, but a newly created gif0 interface has "mtu 1280", why would the > > default be so low? >=20 > gif is frequently used as the innermost encapsulation like in gif tunnel > across host mode IPsec. Then there might be PPPoE, too. Possibly a > VLAN tag ... Please correct me if I'm wrong: - ESP overhead - 40 bytes - UDP encapsulation of ESP (udp/4500): 8 bytes - PPPoE overhead - 8 bytes (?) - A VLAN tag just increases the max frame size, it does not reduce the IP M= TU. So we could keep the safe default for gif(4) at 1500-40-8-8=3D1444 bytes. OK, at 1400 as for if_ipsec. But not at 1280! > So most probably the idea is to go as low as possible for arbitrary addit= ional > encapsulations while still satisfying the minimum requirement for IPv6. >=20 > For the others: no idea, I guess historical ;-) Looks like it. But there are times to have a fresh look at things. --=20 Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ --envbJBWh7q8WU6mo Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJeeH+AAAoJEA2k8lmbXsY0CREIAIHSjKbQhoCqY7zZvJdYsaeb MlHhPJHdy0whqreC2Xr9nNVL2aEzux08znDGSk5Wai1fKmerbWaOsZOqEF/ZJibn GmlIpF/W3FG6aAxAORiMItBm6G1QrVkMvh0J1bDTZajh5i2Ywc9nMIdfR8d5jgoL awGr7X9YwaOteUv/dIqkDdwDjMFjEr+NZWnTDBy4+6puMX6CPHLgRBFA4mTbKJMO FTXV4crW/AnwkWeR4EAN+XQt7rsjUwaPlUoNM42ZMGwgxKHoaY54GkKm+bX3T1uU VXW4bBq/s4TMPP91kVAbdymH5ki9AL8VUl1E9jgRCJsaB9l/HA8Xr+96zAP+ry0= =Vvhi -----END PGP SIGNATURE----- --envbJBWh7q8WU6mo-- From owner-freebsd-net@freebsd.org Mon Mar 23 09:26:49 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 92E9827DD73 for ; Mon, 23 Mar 2020 09:26:49 +0000 (UTC) (envelope-from vas@sibptus.ru) Received: from admin.sibptus.ru (admin.sibptus.ru [IPv6:2001:19f0:5001:21dc::10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48m8CX646jz3JgF for ; Mon, 23 Mar 2020 09:26:48 +0000 (UTC) (envelope-from vas@sibptus.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sibptus.ru; s=20181118; h=In-Reply-To:Message-ID:Subject:To:From:Date; bh=SIWoAF3pRQ/BNPNx/1G1dqivtv2tl15Wbctrhrs7DU0=; b=NXqhfAr+TCLFbLw2wVXlOrRfAG G3e4mshBr1XlC4Q9Q47dZMxx23SJxwNOAN6lYqkWxLSnskCxUn1ZHw0hHB3sF5b/4tSzrXvSYAyh+ GQGLVQ8MRzpQA5fxEuWO+/9jRLX+aIGVxZA2ClneOKptRo8V2iKBWubTEbogWR4IUtUo=; Received: from vas by admin.sibptus.ru with local (Exim 4.93.0.4 (FreeBSD)) (envelope-from ) id 1jGJM7-000Err-9P; Mon, 23 Mar 2020 16:26:47 +0700 Date: Mon, 23 Mar 2020 16:26:47 +0700 From: Victor Sudakov To: "Patrick M. Hausen" Cc: freebsd-net@freebsd.org Subject: Re: IP MTU on gif and gre interfaces (with and without IPSec encryption) Message-ID: <20200323092647.GA57111@admin.sibptus.ru> References: <20200323050012.GA50490@admin.sibptus.ru> <7D560C6A-EB51-414A-A3A1-18587D40C218@punkt.de> <20200323092104.GA56721@admin.sibptus.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6TrnltStXW4iwmi0" Content-Disposition: inline In-Reply-To: <20200323092104.GA56721@admin.sibptus.ru> X-PGP-Key: http://admin.sibptus.ru/~vas/ X-PGP-Fingerprint: 10E3 1171 1273 E007 C2E9 3532 0DA4 F259 9B5E C634 X-Rspamd-Queue-Id: 48m8CX646jz3JgF X-Spamd-Bar: -------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=sibptus.ru header.s=20181118 header.b=NXqhfAr+; dmarc=pass (policy=none) header.from=sibptus.ru; spf=pass (mx1.freebsd.org: domain of vas@sibptus.ru designates 2001:19f0:5001:21dc::10 as permitted sender) smtp.mailfrom=vas@sibptus.ru X-Spamd-Result: default: False [-8.46 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; R_DKIM_ALLOW(-0.20)[sibptus.ru:s=20181118]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; IP_SCORE(-3.37)[ip: (-9.89), ipnet: 2001:19f0:5000::/38(-4.94), asn: 20473(-1.95), country: US(-0.05)]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[sibptus.ru:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[sibptus.ru,none]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:20473, ipnet:2001:19f0:5000::/38, country:US]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Mar 2020 09:26:49 -0000 --6TrnltStXW4iwmi0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Victor Sudakov wrote: > Patrick M. Hausen wrote: > >=20 > > > Am 23.03.2020 um 06:00 schrieb Victor Sudakov : > > > I've noticed that a newly created gre0 interface has the expected "mt= u 1476" > > > value, but a newly created gif0 interface has "mtu 1280", why would t= he > > > default be so low? > >=20 > > gif is frequently used as the innermost encapsulation like in gif tunnel > > across host mode IPsec. Then there might be PPPoE, too. Possibly a > > VLAN tag ... >=20 > Please correct me if I'm wrong: >=20 > - ESP overhead - 40 bytes > - UDP encapsulation of ESP (udp/4500): 8 bytes > - PPPoE overhead - 8 bytes (?) > - A VLAN tag just increases the max frame size, it does not reduce the IP= MTU. >=20 > So we could keep the safe default for gif(4) at 1500-40-8-8=3D1444 bytes. > OK, at 1400 as for if_ipsec. But not at 1280! I should probably have counted the 20 bytes of the additional IP header which results in 1500-40-8-8-20=3D1424. So 1400 is really safe. --=20 Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/ --6TrnltStXW4iwmi0 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJeeIDXAAoJEA2k8lmbXsY087YH+wVNzMozSWG8rUHqoqJLVAHw aHNKQh+mawIyVO2In+54ed3tQP06UxPYWcbtaH+zoBggdidQBw5MQKkynntSstKT 4LaYZoQO2dP6bO85l44zuR2iT69E1OXauW8jIx6B+avVB/4niedvEZfyCloHcaFg Mvvw5Gxss3OhMGX5/xggs/XPO0LWke5k8txVTrLywJ0ESv1K44TMTleSv2sXDp1X Jmwnu6/0xQVRtbSUeSPIFH3dCfHMeev+ft97iOmh4kamrCAmxa3VvavNhCmNc9FT 5N0QfdZNdKa6jS9pJyuu/rHkSBc30b/sPlZnrOsm1wytlcVQdvXggE1K/dgglKI= =Wpa6 -----END PGP SIGNATURE----- --6TrnltStXW4iwmi0-- From owner-freebsd-net@freebsd.org Tue Mar 24 11:37:50 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 31BCA2A29A1 for ; Tue, 24 Mar 2020 11:37:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48mq4C72T3z3Fhh for ; Tue, 24 Mar 2020 11:37:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 289C72A29A0; Tue, 24 Mar 2020 11:37:41 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1E99F2A299F for ; Tue, 24 Mar 2020 11:37:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48mq4271fjz3Ff5 for ; Tue, 24 Mar 2020 11:37:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 180D3269FC for ; Tue, 24 Mar 2020 11:37:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 02OBbDKw071928 for ; Tue, 24 Mar 2020 11:37:13 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 02OBbDrr071927 for net@FreeBSD.org; Tue, 24 Mar 2020 11:37:13 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 208205] re0 watchdog timeout Date: Tue, 24 Mar 2020 11:37:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: needs-qa X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: ml@netfence.it X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 11:37:50 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D208205 --- Comment #34 from ml@netfence.it --- (In reply to Chris Hutchinson from comment #33) Aren't those value's default proportional to system RAM? >From a quick survey on my systems I see: 4GB 11.3 123975 247952 16GB 11.3 507532 1015064 32GB 11.3 1017580 2035160 32GB 12.1 1019729 2039460 I guess the last two don't actually differ due to FreeBSD version differenc= e, but due to little difference in memory availability. Also I've seen this problem on the third system above, where the values are= 4 times the default. So increasing them a little is no solution. I've not seen the problem on the last machine (with 12.1), but that might be because re0 is a WAN port on a slow link. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Tue Mar 24 12:39:53 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5D5902A4FAE for ; Tue, 24 Mar 2020 12:39:53 +0000 (UTC) (envelope-from snar@snar.spb.ru) Received: from staff.retn.net (staff.retn.net [IPv6:2a02:2d8:0:20ff:232a::8]) by mx1.freebsd.org (Postfix) with ESMTP id 48mrRf1NgWz480M for ; Tue, 24 Mar 2020 12:39:40 +0000 (UTC) (envelope-from snar@snar.spb.ru) Received: from staff.retn.net (staff.retn.net [IPv6:2a02:2d8:0:20ff:232a::8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: snarspb@) by staff.retn.net (Postfix) with ESMTPSA id 31A4DC87 for ; Tue, 24 Mar 2020 15:37:45 +0300 (MSK) Date: Tue, 24 Mar 2020 15:37:36 +0300 From: Alexandre Snarskii To: freebsd-net@freebsd.org Subject: netmap/ixl and crc addition.. Message-ID: <20200324123721.GA26248@staff.retn.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 48mrRf1NgWz480M X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of snar@snar.spb.ru has no SPF policy when checking 2a02:2d8:0:20ff:232a::8) smtp.mailfrom=snar@snar.spb.ru X-Spamd-Result: default: False [2.63 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_SPAM_MEDIUM(0.72)[0.725,0]; DMARC_NA(0.00)[snar.spb.ru]; NEURAL_SPAM_LONG(0.90)[0.904,0]; R_SPF_NA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:9002, ipnet:2a02:2d8::/32, country:EU]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(-0.00)[country: EU(-0.01)]; ONCE_RECEIVED(0.10)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 12:39:53 -0000 Hi! Long story short: looks like intel x722 does not by default add CRC to outbound frames, so with FreeBSD 11-stable netmap-generated traffic is dropped on the next port.. Fix is simple, attached. The same behaviour of 'unconditionally ask card to compute crc' can be found in both if_ixl: https://svnweb.freebsd.org/base/stable/11/sys/dev/ixl/ixl_txrx.c?view=markup#l408 and in DPDK i40e driver: https://github.com/DPDK/dpdk/blob/master/drivers/net/i40e/i40e_rxtx.c#L1105 so, I guess, it's safe. PS: of course, first idea was to upgrade to FreeBSD 12-stable, but while this upgrade solved the crc problem, this version shows 'stalled tx queue' problem: after CTXSYNC tail == head == cur, 'no space in ring' condition. Attempts to ignore this condition led to continuous ring resets in txcsync: Mar 17 20:21:08 host kernel: 668.224836 [1679] nm_txsync_prologue ixl1 TX3: fail 'head < kring->rhead || head > kring->rtail' h 136 c 136 t 135 rh 135 rc 135 rt 135 hc 135 ht 135 Mar 17 20:21:08 host kernel: 668.238300 [1787] netmap_ring_reinit called for ixl1 TX3 PPS: hardware details: Dell VEP4600, based on Xeon D-2100 with two onboard X722 ports (actually, four, but two of them are not wired). CPU: Intel(R) Xeon(R) D-2187NT CPU @ 2.00GHz (2000.06-MHz K8-class CPU) Origin="GenuineIntel" Id=0x50654 Family=0x6 Model=0x55 Stepping=4 ixl0: mem 0xfa000000-0xfaffffff,0xfb018000-0xfb01ffff irq 11 at device 0.0 numa-domain 0 on pci12 ixl0: using 1024 tx descriptors and 1024 rx descriptors ixl0: fw 3.1.57069 api 1.5 nvm 3.33 etid 80001007 oem 1.263.0 ixl0: PF-ID[0]: VFs 32, MSIX 129, VF MSIX 5, QPs 384, I2C ixl0: Using MSIX interrupts with 9 vectors ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active ixl0: Ethernet address: 3c:2c:30:30:59:85 ixl0: SR-IOV ready ixl0: netmap queues/slots: TX 8/1024, RX 8/1024 ixl0@pci0:184:0:0: class=0x020000 card=0x00008086 chip=0x37d38086 rev=0x04 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection X722 for 10GbE SFP+' class = network subclass = ethernet From owner-freebsd-net@freebsd.org Tue Mar 24 14:10:53 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C009B2A6BE0 for ; Tue, 24 Mar 2020 14:10:53 +0000 (UTC) (envelope-from snar@snar.spb.ru) Received: from staff.retn.net (staff.retn.net [IPv6:2a02:2d8:0:20ff:232a::8]) by mx1.freebsd.org (Postfix) with ESMTP id 48mtSY08fFz3D8Y for ; Tue, 24 Mar 2020 14:10:36 +0000 (UTC) (envelope-from snar@snar.spb.ru) Received: from staff.retn.net (staff.retn.net [IPv6:2a02:2d8:0:20ff:232a::8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: snarspb@) by staff.retn.net (Postfix) with ESMTPSA id F1F82DF8; Tue, 24 Mar 2020 17:08:40 +0300 (MSK) Date: Tue, 24 Mar 2020 17:08:39 +0300 From: Alexandre Snarskii To: freebsd-net@freebsd.org Cc: Alexandre Snarskii Subject: Re: netmap/ixl and crc addition.. Message-ID: <20200324140839.GB26248@staff.retn.net> References: <20200324123721.GA26248@staff.retn.net> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="J2SCkAp4GZ/dPZZf" Content-Disposition: inline In-Reply-To: <20200324123721.GA26248@staff.retn.net> User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 48mtSY08fFz3D8Y X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=temperror (mx1.freebsd.org: error in processing during lookup of snar@snar.spb.ru: DNS error) smtp.mailfrom=snar@snar.spb.ru X-Spamd-Result: default: False [-1.58 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_FAIL(0.00)[8.0.0.0.0.0.0.0.0.0.0.0.a.2.3.2.f.f.0.2.0.0.0.0.8.d.2.0.2.0.a.2.list.dnswl.org:query timed out]; R_SPF_DNSFAIL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.79)[-0.793,0]; HAS_ATTACHMENT(0.00)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; DMARC_NA(0.00)[snar.spb.ru]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-0.79)[-0.788,0]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:9002, ipnet:2a02:2d8::/32, country:EU]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(-0.00)[country: EU(-0.01)]; ONCE_RECEIVED(0.10)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 14:10:54 -0000 --J2SCkAp4GZ/dPZZf Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Tue, Mar 24, 2020 at 03:37:36PM +0300, Alexandre Snarskii wrote: > > Hi! > > Long story short: looks like intel x722 does not by default add CRC to > outbound frames, so with FreeBSD 11-stable netmap-generated traffic is > dropped on the next port.. Fix is simple, attached. ... add missing attach :( > > The same behaviour of 'unconditionally ask card to compute crc' can > be found in both if_ixl: > https://svnweb.freebsd.org/base/stable/11/sys/dev/ixl/ixl_txrx.c?view=markup#l408 > and in DPDK i40e driver: > https://github.com/DPDK/dpdk/blob/master/drivers/net/i40e/i40e_rxtx.c#L1105 > so, I guess, it's safe. > > PS: of course, first idea was to upgrade to FreeBSD 12-stable, but while > this upgrade solved the crc problem, this version shows 'stalled tx queue' > problem: after CTXSYNC tail == head == cur, 'no space in ring' condition. > Attempts to ignore this condition led to continuous ring resets in txcsync: > > Mar 17 20:21:08 host kernel: 668.224836 [1679] nm_txsync_prologue ixl1 TX3: fail 'head < kring->rhead || head > kring->rtail' h 136 c 136 t 135 rh 135 rc 135 rt 135 hc 135 ht 135 > Mar 17 20:21:08 host kernel: 668.238300 [1787] netmap_ring_reinit called for ixl1 TX3 > > PPS: hardware details: Dell VEP4600, based on Xeon D-2100 with > two onboard X722 ports (actually, four, but two of them are not > wired). > > CPU: Intel(R) Xeon(R) D-2187NT CPU @ 2.00GHz (2000.06-MHz K8-class CPU) > Origin="GenuineIntel" Id=0x50654 Family=0x6 Model=0x55 Stepping=4 > > ixl0: mem 0xfa000000-0xfaffffff,0xfb018000-0xfb01ffff irq 11 at device 0.0 numa-domain 0 on pci12 > ixl0: using 1024 tx descriptors and 1024 rx descriptors > ixl0: fw 3.1.57069 api 1.5 nvm 3.33 etid 80001007 oem 1.263.0 > ixl0: PF-ID[0]: VFs 32, MSIX 129, VF MSIX 5, QPs 384, I2C > ixl0: Using MSIX interrupts with 9 vectors > ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active > ixl0: Ethernet address: 3c:2c:30:30:59:85 > ixl0: SR-IOV ready > ixl0: netmap queues/slots: TX 8/1024, RX 8/1024 > > ixl0@pci0:184:0:0: class=0x020000 card=0x00008086 chip=0x37d38086 rev=0x04 hdr=0x00 > vendor = 'Intel Corporation' > device = 'Ethernet Connection X722 for 10GbE SFP+' > class = network > subclass = ethernet > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" --J2SCkAp4GZ/dPZZf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="if_ixl_netmap.h.diff" Index: sys/dev/netmap/if_ixl_netmap.h =================================================================== --- sys/dev/netmap/if_ixl_netmap.h (revision 359243) +++ sys/dev/netmap/if_ixl_netmap.h (working copy) @@ -239,7 +239,8 @@ curr->cmd_type_offset_bsz = htole64( ((u64)len << I40E_TXD_QW1_TX_BUF_SZ_SHIFT) | flags | - ((u64)I40E_TX_DESC_CMD_EOP << I40E_TXD_QW1_CMD_SHIFT) + ((u64)I40E_TX_DESC_CMD_EOP << I40E_TXD_QW1_CMD_SHIFT) | + ((u64)I40E_TX_DESC_CMD_ICRC << I40E_TXD_QW1_CMD_SHIFT) ); // XXX more ? /* make sure changes to the buffer are synced */ --J2SCkAp4GZ/dPZZf-- From owner-freebsd-net@freebsd.org Tue Mar 24 14:17:33 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3F3FA2A6F9E for ; Tue, 24 Mar 2020 14:17:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48mtcX4MM2z3GV8 for ; Tue, 24 Mar 2020 14:17:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id AD7E32A6F9B; Tue, 24 Mar 2020 14:17:24 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AB8F62A6F9A for ; Tue, 24 Mar 2020 14:17:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48mtcM44j6z3GRx for ; Tue, 24 Mar 2020 14:17:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 32587B2B for ; Tue, 24 Mar 2020 14:17:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 02OEH14x073899 for ; Tue, 24 Mar 2020 14:17:01 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 02OEH1Er073898 for net@FreeBSD.org; Tue, 24 Mar 2020 14:17:01 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 208205] re0 watchdog timeout Date: Tue, 24 Mar 2020 14:16:55 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: needs-qa X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: portmaster@bsdforge.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 14:17:33 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D208205 --- Comment #35 from Chris Hutchinson --- (In reply to ml from comment #34) For *me*, doubling the values from their defaults fixed it. I only mentioned it as 1) disabling MSI-X (default solution) reduces performance -- even significantly 2) It worked. So felt it worth mentioning. :) --Chris --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Tue Mar 24 19:14:12 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 058C6267EAB for ; Tue, 24 Mar 2020 19:14:12 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-qv1-xf32.google.com (mail-qv1-xf32.google.com [IPv6:2607:f8b0:4864:20::f32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48n1Bd0KKBz4dqr; Tue, 24 Mar 2020 19:14:00 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-qv1-xf32.google.com with SMTP id g4so3135228qvo.12; Tue, 24 Mar 2020 12:14:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=kQpei7Qexd8vUpQ/2UdVMwkEyplf683x+1z/ERfqFNY=; b=hCgIpi+rbuctyHIo90fKCpCmVPq79xbCGvQWj1WrvzKs46hBKAc6jVKEnamP4uni7J 3FVm1a7X0GQnJtFVMegXhILOF9Gu2VKA+pckAdIRBqIUxRGw+SePkJqB2V31DEl5ug8q NuidHEnltwAzQn+AN7flAvj7phAtKehFHm8iwsTeCjvLVtGR7ZfTq68UXBwqRd7+TsMe ViYSF8oe9fjzz9tqCpFPVjgPtIIndFKbwaShqomJByumqi3q3a/CpLRYfjZJ8RGjJw8D wGvRlMAzWUmQlf4tSBQhIGVCNWcddU1CNNMGEWmFskMkP1bjfXFkDTvW+kcAb9weIMvz Ct6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=kQpei7Qexd8vUpQ/2UdVMwkEyplf683x+1z/ERfqFNY=; b=qxwwb6V2IfSAbsQ13ZtoxnlI61VWebWN9T6jfvmlkoTCVdQZ471M1qj5OK85502uSL aAgyHcyidUgZldSfmr5Simzgx+hvZljrLDTDVJk9l87fmhTp9qTuxQr48B2kx+Cb9cA+ ww2BO9/v2Te+o+TmrqTC4RRsyPpDfTmFfglSD7dux28gApGfhb4wxuZSlvC+1tXs3JGe XIW4v93Qqebp1SV3DT6jHGoJpDv6Rdl8wM23R/9xViJ4D787MTpiTwPTW427Tc4U4+Yj a3N2kzeVEtbYlBSoY/U13+TtEIWi9qryGnzwdByciWzsYptth8NY2Ui/vIfMBA0bmtG2 R1UA== X-Gm-Message-State: ANhLgQ0hnt8WvfGjeC26D08CHL666Ds9Or1MUs8d+XqV/RD69eaH2Zex cV7L8nbH4+nAk9ogcAs88oYbaI3uOuobGKLHrb22dJOk X-Google-Smtp-Source: ADFU+vu53+lFaFOnfBGQYh3mUNocDOvNzoBqP6RiH/kpJsRrzkDI2anmRF1etImwqrmgnx+UhiOSvzxrRlzkUpXLEes= X-Received: by 2002:a0c:aee5:: with SMTP id n37mr25958857qvd.173.1585077231245; Tue, 24 Mar 2020 12:13:51 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ryan Stone Date: Tue, 24 Mar 2020 15:13:40 -0400 Message-ID: Subject: Re: Can net.iflib.min_tx_latency=0 introduce an unbounded amount of latency? To: Patrick Kelsey Cc: freebsd-net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 48n1Bd0KKBz4dqr X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=hCgIpi+r; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rysto32@gmail.com designates 2607:f8b0:4864:20::f32 as permitted sender) smtp.mailfrom=rysto32@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE_FREEMAIL(0.00)[]; IP_SCORE(0.00)[ip: (-7.67), ipnet: 2607:f8b0::/32(-0.38), asn: 15169(-0.49), country: US(-0.05)]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2.3.f.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 19:14:13 -0000 Thanks for the analysis. I haven't had much of an opportunity to get into the guts of iflib and I'm not very successful at following the round trips through mpring yet. On Mon, Mar 23, 2020 at 12:50 AM Patrick Kelsey wrote= : > > > > On Sun, Mar 22, 2020 at 2:31 PM Ryan Stone wrote: >> >> I've been tracking down a bug at work that appears to be due to >> excessive latency introduced on the TX side of a TCP connection. In >> looking through the iflib code, I noticed the tunable >> net.iflib.min_tx_latency. My reading of the iflib code is that if a >> packet is enqueued to a tx ring but not posted (when >> min_tx_latency=3D0), then iflib can introduce an unbounded amount of >> latency if the ring is idle. Is my reading of the code correct? >> > > Let's see...in the case described above, the next iflib_timer() invocatio= n (max 500ms later) should find that txq->ift_db_pending is non-zero, which= will cause it to enqueue the TX task, which will invoke _task_fn_tx(), whi= ch will then enqueue the special marker (void **)&txq, and one way or anoth= er, iflib_txq_drain() will be called. > > Looking at iflib_txq_drain() it appears that the first call to iflib_txd_= db_check() should post the pending packets either because a non-zero number= of slots were just reclaimed, or the check for txq->ift_db_pending >=3D TX= Q_MAX_DB_DEFERRED(...) is satisfied, unless there are too many posted packe= ts in the hardware queue that aren't being drained by the hardware (this wo= uld result in the in_use parameter remaining elevated such that a small num= ber of pending packets would not exceed the threshold, and also no slot rec= laim occurring). > > -Patrick From owner-freebsd-net@freebsd.org Tue Mar 24 20:23:15 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 14D53269785 for ; Tue, 24 Mar 2020 20:23:15 +0000 (UTC) (envelope-from dan@langille.org) Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48n2kG3RLqz46Hn for ; Tue, 24 Mar 2020 20:23:02 +0000 (UTC) (envelope-from dan@langille.org) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 707F95C0227; Tue, 24 Mar 2020 16:22:53 -0400 (EDT) Received: from imap36 ([10.202.2.86]) by compute2.internal (MEProxy); Tue, 24 Mar 2020 16:22:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=langille.org; h= mime-version:message-id:in-reply-to:references:date:from:to :subject:content-type; s=fm3; bh=LFlB7+1mKFwTmTddRh8HdQ8jMqry6mP keW5jAnFogXE=; b=biG5yb7i+8HMtRVyrL+/8CtJjLEQAkt8N11eOtrTklkfA91 EN2PGmB0lHZS8/MCMdyIm8S3Ya4G10DYe6nyA8MIcShwDxIXaFy+k8lZ2GDUt8Zs aFhMNk36Sgc8Mp+EkaXOzHFfnsfp6KWY+54MNedRsijSDOwJmC1fUoBy7y2m4Ktv g1uw3ua6M0weiNXpxzF1nxK2DaGQlaUw0LtzmndzzwQdTp+ridF8hUl/+iiN3GOy EndrhTuW0nw6DjGFAazu4hnz83uN1HsNy3uQLhBSHwcTuLCXmkjTbsQi5QOvwkGZ ORDJZ4RGpcTw2NKv6a7dLa/kLsbOwSXgnTc2oGQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=LFlB7+ 1mKFwTmTddRh8HdQ8jMqry6mPkeW5jAnFogXE=; b=tb4QJL+QtmiijylsPRdaCh 4MYlyOsa0jcFH+5o+BlXf74NFm+rzP7TikuVS3h05v81VEoN5emBsVGQay0ap8AU 6ZcB69ZAcH+kopY23wqYqETQR/dgFf/N0t6DSv6Y59vB54CC4uq9mBNtc4ItWdxl MFaZCGSP/55Fsr3XDiudbnc45zn2jPgCxQwWrKhQHQA2zPk6t64+eS4VW1m/rcwM K3PuUqGzx4NrbG7as7M5VYSP5fvABCAMHjF42R0FpKlAUBULbdXSWCZZBhuDhxWg ibtFk+YG3wt0vivz/5nBv0ZpfsJm37vlmVIKg4vyGCJtiF0viyG/5pD9gjVBBKRA == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrudehuddguddthecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefofgggkfgjfhffhffvufgtsehttdertderredtnecuhfhrohhmpedfffgr nhcunfgrnhhgihhllhgvfdcuoegurghnsehlrghnghhilhhlvgdrohhrgheqnecuvehluh hsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepuggrnheslhgrnhhg ihhllhgvrdhorhhg X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id A328D1880062; Tue, 24 Mar 2020 16:22:52 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.7-1021-g152deaf-fmstable-20200319v1 Mime-Version: 1.0 Message-Id: In-Reply-To: <88679204-4c23-7fce-681d-ab633b326fef@akhmatov.ru> References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> <88679204-4c23-7fce-681d-ab633b326fef@akhmatov.ru> Date: Tue, 24 Mar 2020 16:22:32 -0400 From: "Dan Langille" To: "Sergey Akhmatov" , "Pieper, Jeffrey E" , "freebsd-net@freebsd.org" Subject: Re: SFP+ on PRO/10GbE Content-Type: text/plain X-Rspamd-Queue-Id: 48n2kG3RLqz46Hn X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=langille.org header.s=fm3 header.b=biG5yb7i; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=tb4QJL+Q; dmarc=pass (policy=none) header.from=langille.org; spf=pass (mx1.freebsd.org: domain of dan@langille.org designates 66.111.4.29 as permitted sender) smtp.mailfrom=dan@langille.org X-Spamd-Result: default: False [-5.58 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[langille.org:s=fm3,messagingengine.com:s=fm2]; XM_UA_NO_VERSION(0.01)[]; URIBL_BLOCKED(0.00)[messagingengine.com.multi.uribl.com,langille.org.multi.uribl.com]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.29]; RCPT_COUNT_THREE(0.00)[3]; MIME_GOOD(-0.10)[text/plain]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[langille.org:+,messagingengine.com:+]; DMARC_POLICY_ALLOW(-0.50)[langille.org,none]; IP_SCORE(-3.49)[ip: (-9.84), ipnet: 66.111.4.0/24(-4.89), asn: 11403(-2.69), country: US(-0.05)]; RBL_NIXSPAM_FAIL(0.00)[29.4.111.66.ix.dnsbl.manitu.net:query timed out]; RCVD_IN_DNSWL_LOW(-0.10)[29.4.111.66.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; MID_RHS_WWW(0.50)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 20:23:15 -0000 On Mon, Mar 23, 2020, at 4:24 AM, Sergey Akhmatov wrote: > > On 2020-03-22 8:35 PM, Dan Langille wrote: > > Hello Jeff, > > > > Partial success. The card is now able to use an SFP+ optic. It warns me > > when the optic is installed: > > > > Mar 22 16:49:45 r720-01 kernel: WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules. > > Do you see correct media type and signal levels running: > > $ ifconfig -vm ix0 > > ? [dan@r720-01:~] $ ifconfig -vm ix0 ix0: flags=8943 metric 0 mtu 1500 options=a538b9 capabilities=f53fbb ether ... [snip] media: Ethernet autoselect (1000baseSX ) status: active supported media: media autoselect media 10Gbase-SR nd6 options=23 plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: AVAGO PN: AFBR-703SDZ SN: AD0946A00YB DATE: 2009-11-10 module temperature: 38.09 C Voltage: 3.33 Volts RX: 0.31 mW (-4.95 dBm) TX: 0.56 mW (-2.48 dBm) [dan@r720-01:~] $ That looks right to me, but I have never inspected this before. -- Dan Langille dan@langille.org From owner-freebsd-net@freebsd.org Tue Mar 24 20:59:21 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A209126A416 for ; Tue, 24 Mar 2020 20:59:21 +0000 (UTC) (envelope-from sergey@akhmatov.ru) Received: from dutch.chaklun.ru (dutch.chaklun.ru [IPv6:2a01:7c8:aab0:135::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48n3Wx1Gw4z4KmW for ; Tue, 24 Mar 2020 20:59:08 +0000 (UTC) (envelope-from sergey@akhmatov.ru) Received: from [176.195.68.100] (helo=[192.168.1.3]) by dutch.chaklun.ru with esmtpsa (TLS1.3) tls TLS_AES_128_GCM_SHA256 (Exim 4.93.0.4 (FreeBSD)) (envelope-from ) id 1jGqdM-0003LL-IY; Tue, 24 Mar 2020 23:58:48 +0300 Subject: Re: SFP+ on PRO/10GbE To: Dan Langille , "Pieper, Jeffrey E" , "freebsd-net@freebsd.org" References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> <88679204-4c23-7fce-681d-ab633b326fef@akhmatov.ru> From: Sergey Akhmatov Message-ID: <3265c7ca-656c-9269-c889-fe5499958232@akhmatov.ru> Date: Tue, 24 Mar 2020 23:58:45 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 48n3Wx1Gw4z4KmW X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of sergey@akhmatov.ru has no SPF policy when checking 2a01:7c8:aab0:135::1) smtp.mailfrom=sergey@akhmatov.ru X-Spamd-Result: default: False [-2.19 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; IP_SCORE(-1.10)[ipnet: 2a01:7c8::/32(-3.68), asn: 20857(-1.87), country: NL(0.03)]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[akhmatov.ru]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-1.00)[-0.997,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.985,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:20857, ipnet:2a01:7c8::/32, country:NL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 20:59:22 -0000 On 2020-03-24 11:22 PM, Dan Langille wrote: > > media: Ethernet autoselect (1000baseSX ) > status: active > supported media: > media autoselect > media 10Gbase-SR > nd6 options=23 > plugged: SFP/SFP+/SFP28 10G Base-SR (LC) > vendor: AVAGO PN: AFBR-703SDZ SN: AD0946A00YB DATE: 2009-11-10 > module temperature: 38.09 C Voltage: 3.33 Volts > RX: 0.31 mW (-4.95 dBm) TX: 0.56 mW (-2.48 dBm) > [dan@r720-01:~] $ > > That looks right to me, but I have never inspected this before. Not right actually. The supported media is "10Gbase-SR" which is correct for this type of SFP,  but autodetected is "1000baseSX" and it's not even listed as supported. That doesn't make sense to me, maybe try to set media type manually? # ifconfig ix0 media 10gbase-sr From owner-freebsd-net@freebsd.org Tue Mar 24 21:30:34 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D71C826B2C5 for ; Tue, 24 Mar 2020 21:30:34 +0000 (UTC) (envelope-from jeffrey.e.pieper@intel.com) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "fmsmga106.fm.intel.com", Issuer "Sectigo RSA Organization Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48n4Cn4STRz4Wbn; Tue, 24 Mar 2020 21:30:12 +0000 (UTC) (envelope-from jeffrey.e.pieper@intel.com) IronPort-SDR: 1xCd5ouokHJpL7NNsSCGt9O6suB4Z0cUtGfYZMN899W6b5WjqN6ar4/1gbmb/IFMQ8YHc6IZu6 2g5uaXS3kboA== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2020 14:30:05 -0700 IronPort-SDR: KdxlQPE9uGAx1t1pE1ZPPDGIFkfGZq7rxcw7nRmQSfQMJ3LetE3PNpQpeA7oA7Zo5bQxd8lEpH p+RKODZxIIuw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,301,1580803200"; d="scan'208";a="281943471" Received: from orsmsx103.amr.corp.intel.com ([10.22.225.130]) by fmsmga002.fm.intel.com with ESMTP; 24 Mar 2020 14:30:04 -0700 Received: from orsmsx159.amr.corp.intel.com (10.22.240.24) by ORSMSX103.amr.corp.intel.com (10.22.225.130) with Microsoft SMTP Server (TLS) id 14.3.439.0; Tue, 24 Mar 2020 14:30:04 -0700 Received: from orsmsx111.amr.corp.intel.com ([169.254.12.226]) by ORSMSX159.amr.corp.intel.com ([169.254.11.47]) with mapi id 14.03.0439.000; Tue, 24 Mar 2020 14:30:03 -0700 From: "Pieper, Jeffrey E" To: Dan Langille , Gary Palmer CC: sthaug , "freebsd-net@freebsd.org" Subject: RE: SFP+ on PRO/10GbE Thread-Topic: SFP+ on PRO/10GbE Thread-Index: AQHV/8+iW1/7kfU5G02OZndwRC+br6hTrbfAgAGofICAADS7gIAAB6kAgAAfYYCAAAWxgIADAumA//+K+WA= Date: Tue, 24 Mar 2020 21:30:02 +0000 Message-ID: <2A35EA60C3C77D438915767F458D6568B9094CED@ORSMSX111.amr.corp.intel.com> References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> <20200322.214355.415142200.sthaug@nethelp.no> <20200322230339.GB5808@in-addr.com> <8141b371-5cdf-4f82-8959-5fa1071aa6dd@www.fastmail.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.2.0.6 dlp-reaction: no-action x-originating-ip: [10.22.254.138] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Rspamd-Queue-Id: 48n4Cn4STRz4Wbn X-Spamd-Bar: --------- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=intel.com; spf=pass (mx1.freebsd.org: domain of jeffrey.e.pieper@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=jeffrey.e.pieper@intel.com X-Spamd-Result: default: False [-9.78 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; HAS_XOIP(0.00)[]; URIBL_BLOCKED(0.00)[imgur.com.multi.uribl.com]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:192.55.52.136/32]; RCPT_COUNT_THREE(0.00)[4]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-3.78)[ip: (-9.91), ipnet: 192.55.52.0/24(-4.96), asn: 4983(-3.96), country: US(-0.05)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[136.52.55.192.list.dnswl.org : 127.0.9.2]; DMARC_POLICY_ALLOW(-0.50)[intel.com,none]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:4983, ipnet:192.55.52.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; WHITELIST_SPF_DKIM(-3.00)[intel.com:s:+] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 21:30:35 -0000 You cannot force speed with ix, only force autoneg. You need to use the adv= ertise_speed sysctl for this: sysctl -d dev.ix.0.advertise_speed dev.ix.0.advertise_speed: Control advertised link speed using these flags: 0x1 - advertise 100M 0x2 - advertise 1G 0x4 - advertise 10G 0x8 - advertise 10M 100M and 10M are only supported on certain adapters. Jeff -----Original Message----- From: Dan Langille =20 Sent: Tuesday, March 24, 2020 2:23 PM To: Gary Palmer Cc: sthaug ; Pieper, Jeffrey E ; freebsd-net@freebsd.org Subject: Re: SFP+ on PRO/10GbE On Sun, Mar 22, 2020, at 7:24 PM, Dan Langille wrote: > On Sun, Mar 22, 2020, at 7:03 PM, Gary Palmer wrote: > > On Sun, Mar 22, 2020 at 05:11:20PM -0400, Dan Langille wrote: > > > On Sun, Mar 22, 2020, at 4:43 PM, sthaug@nethelp.no wrote: > > > > > Partial success. The card is now able to use an SFP+ optic. =20 > > > > > It warns me when the optic is installed: > > > > >=20 > > > > > Mar 22 16:49:45 r720-01 kernel: WARNING: Intel (R) Network Connec= tions are quality tested using Intel (R) Ethernet Optics. Using untested mo= dules is not supported and may cause unstable operation or damage to the mo= dule or the adapter. Intel Corporation is not responsible for any harm caus= ed by using untested modules. > > > > >=20 > > > > > I cannot use an SFP+ optic at the switch. The connection just do= es not happen. > > > > >=20 > > > > > If I go back to the original SFP optic, the connection occurs, as= expected at 1G. > > > > >=20 > > > > > On the switch side, I've tried a known good optic from an existin= g connection. > > > > >=20 > > > > > I could install an PRO/10GbE instead, that has a built-in=20 > > > > > transceiver. I have two of those in use now, both working on 10G. > > > >=20 > > > > Have you tried connected it to something other than the Unifi switc= h? > > >=20 > > > The only SFP+ capable switches I have are Unifi. > > >=20 > > > I just tried the other switch (US-48) which had one SFP+ port free. = Same issues there. > >=20 > > Did the Unifi switch see the SFP+ optics on the switch end? >=20 > That port never lit up on the switch. I didn't check anything else,=20 > switch-related. I just inserted a 10G transceiver into port 8. It shows up as a 10GBase-SR Screen shots at https://imgur.com/a/SSEgyxN At the same time, on the server side, ifconfig indicates: status: no carrier supported media: media autoselect media 10Gbase-SR nd6 options=3D23 plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: AVAGO PN: AFBR-703SDZ SN: AD0946A00YB DATE: 2009-11-10 module temperature: 36.75 C Voltage: 3.32 Volts RX: 0.67 mW (-1.72 dBm) TX: 0.56 mW (-2.46 dBm) I have not yet tried moving from auto-negotiate on both ends. -- Dan Langille dan@langille.org From owner-freebsd-net@freebsd.org Tue Mar 24 21:31:52 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D4D3C26B4A3 for ; Tue, 24 Mar 2020 21:31:52 +0000 (UTC) (envelope-from dan@langille.org) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48n4FP2X9cz4XBR; Tue, 24 Mar 2020 21:31:36 +0000 (UTC) (envelope-from dan@langille.org) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 1BBE95C0224; Tue, 24 Mar 2020 17:23:35 -0400 (EDT) Received: from imap36 ([10.202.2.86]) by compute2.internal (MEProxy); Tue, 24 Mar 2020 17:23:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=langille.org; h= mime-version:message-id:in-reply-to:references:date:from:to:cc :subject:content-type; s=fm3; bh=iPVGC6dspOuBtWQOba9icHK7YUBWRuw Z8uUQK2GrmFs=; b=MWV6NhmYurDBOQb42nNdp2ZzFEN2mctWrhLYmlni5NsE+2f yuuHOQmyCtOPcano7lG3wunnxR7T1ld0JkmCxBqNkTnoAGHP4QWXLGcgwkZSSrG0 8eO+F2YLcGZSu5qF5Wsfrx26mfPbrCRSFhiQt+Z4SAZ8Be4D0MJTGwScCLI8DDiN qisuXXXbysuOgH0InwdcTVyzEwfThUT5aW6Cf5qIZFn8kkRLgbFoNFRRt/4pyIm8 yGICwJs5l++MJzUlwzshHCC7+Ukcx8f773sjDH2znyU4EyGpY6wZkFxyVjbgclTl rS+tNYVqzypG+w0GM707uKJ0a2WVXpqvQ/1HOHg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=iPVGC6 dspOuBtWQOba9icHK7YUBWRuwZ8uUQK2GrmFs=; b=AeYcWgER5Ez1+Q0ujx9FzC FvWAAbebPup6+nxs/FPP4/XGs8qMY2xIlWFWtMmPkDQs0d4K/JJHiTzm24XSgd49 VK6/S1Gej9g8ZZaiIP3mMNnp8QLvXWUbw96XI1hzIXGb7DZDAA7jS6w25NjYSRKf CTv30V9JaBk3rw188L2xgy4mXE9QRoE/KbXm0TsRxIUQ52RAOFfRFGR15Lvjkj2J 9u78xh2OcxLZbXUYs8EdK5KTHxU1Vgm4ucuB8KskDXKSbpeiJA9RoNLWCkr1qsLA sxBSU2SLFZx9OosqORWpxWbLoXiRM3lGRA3qLIoUX/x7Xmhq44PeUtHaZ9bgT44g == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrudehuddguddujecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefofgggkfgjfhffhffvufgtsehttdertderredtnecuhfhrohhmpedfffgr nhcunfgrnhhgihhllhgvfdcuoegurghnsehlrghnghhilhhlvgdrohhrgheqnecuffhomh grihhnpehimhhguhhrrdgtohhmnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghm pehmrghilhhfrhhomhepuggrnheslhgrnhhgihhllhgvrdhorhhg X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 550E61880062; Tue, 24 Mar 2020 17:23:34 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.7-1021-g152deaf-fmstable-20200319v1 Mime-Version: 1.0 Message-Id: In-Reply-To: <8141b371-5cdf-4f82-8959-5fa1071aa6dd@www.fastmail.com> References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> <20200322.214355.415142200.sthaug@nethelp.no> <20200322230339.GB5808@in-addr.com> <8141b371-5cdf-4f82-8959-5fa1071aa6dd@www.fastmail.com> Date: Tue, 24 Mar 2020 17:23:13 -0400 From: "Dan Langille" To: "Gary Palmer" Cc: sthaug , "Pieper, Jeffrey E" , freebsd-net@freebsd.org Subject: Re: SFP+ on PRO/10GbE Content-Type: text/plain X-Rspamd-Queue-Id: 48n4FP2X9cz4XBR X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=langille.org header.s=fm3 header.b=MWV6NhmY; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=AeYcWgER; dmarc=pass (policy=none) header.from=langille.org; spf=pass (mx1.freebsd.org: domain of dan@langille.org designates 66.111.4.27 as permitted sender) smtp.mailfrom=dan@langille.org X-Spamd-Result: default: False [-5.58 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[langille.org:s=fm3,messagingengine.com:s=fm2]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.27]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; URIBL_BLOCKED(0.00)[imgur.com.multi.uribl.com,messagingengine.com.multi.uribl.com,langille.org.multi.uribl.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DBL_FAIL(0.00)[query timed out]; DKIM_TRACE(0.00)[langille.org:+,messagingengine.com:+]; DMARC_POLICY_ALLOW(-0.50)[langille.org,none]; ZRD_FAIL(0.00)[query timed out]; IP_SCORE(-3.49)[ip: (-9.84), ipnet: 66.111.4.0/24(-4.89), asn: 11403(-2.69), country: US(-0.05)]; RCVD_IN_DNSWL_LOW(-0.10)[27.4.111.66.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; MID_RHS_WWW(0.50)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 21:31:53 -0000 On Sun, Mar 22, 2020, at 7:24 PM, Dan Langille wrote: > On Sun, Mar 22, 2020, at 7:03 PM, Gary Palmer wrote: > > On Sun, Mar 22, 2020 at 05:11:20PM -0400, Dan Langille wrote: > > > On Sun, Mar 22, 2020, at 4:43 PM, sthaug@nethelp.no wrote: > > > > > Partial success. The card is now able to use an SFP+ optic. It warns me > > > > > when the optic is installed: > > > > > > > > > > Mar 22 16:49:45 r720-01 kernel: WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules. > > > > > > > > > > I cannot use an SFP+ optic at the switch. The connection just does not happen. > > > > > > > > > > If I go back to the original SFP optic, the connection occurs, as expected at 1G. > > > > > > > > > > On the switch side, I've tried a known good optic from an existing connection. > > > > > > > > > > I could install an PRO/10GbE instead, that has a built-in transceiver. I have > > > > > two of those in use now, both working on 10G. > > > > > > > > Have you tried connected it to something other than the Unifi switch? > > > > > > The only SFP+ capable switches I have are Unifi. > > > > > > I just tried the other switch (US-48) which had one SFP+ port free. Same issues there. > > > > Did the Unifi switch see the SFP+ optics on the switch end? > > That port never lit up on the switch. I didn't check anything else, > switch-related. I just inserted a 10G transceiver into port 8. It shows up as a 10GBase-SR Screen shots at https://imgur.com/a/SSEgyxN At the same time, on the server side, ifconfig indicates: status: no carrier supported media: media autoselect media 10Gbase-SR nd6 options=23 plugged: SFP/SFP+/SFP28 10G Base-SR (LC) vendor: AVAGO PN: AFBR-703SDZ SN: AD0946A00YB DATE: 2009-11-10 module temperature: 36.75 C Voltage: 3.32 Volts RX: 0.67 mW (-1.72 dBm) TX: 0.56 mW (-2.46 dBm) I have not yet tried moving from auto-negotiate on both ends. -- Dan Langille dan@langille.org From owner-freebsd-net@freebsd.org Tue Mar 24 21:45:36 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BD6CA26BA8E for ; Tue, 24 Mar 2020 21:45:36 +0000 (UTC) (envelope-from dan@langille.org) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48n4YD3LGbz4cjK; Tue, 24 Mar 2020 21:45:20 +0000 (UTC) (envelope-from dan@langille.org) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 695DC5C02BD; Tue, 24 Mar 2020 17:45:12 -0400 (EDT) Received: from imap36 ([10.202.2.86]) by compute2.internal (MEProxy); Tue, 24 Mar 2020 17:45:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=langille.org; h= mime-version:message-id:in-reply-to:references:date:from:to:cc :subject:content-type; s=fm3; bh=+6A4QNGUXgWshF8SRD55aff8GLNYeON z7poQYW7E0ek=; b=us6gpUyFxEug6y31yFOF5/olCKAC1BWhLcfNLFOpyvh/IGw BnBVf1I3Dj7vaII40ytpihfsmUB2EtOF1j8H+Nf7Gn62Tl6DW7T8DHRe72Rk2Gho nxBtVTsPsbDmsPpGd/N9OFjvHPmX+2oZcy5qP3VQ0/yRsiINHU/VuCzMn/Cv39JE xN9jjdm21AOER90CxmpzCWyEJFWfIOb3sgqusems4XILoyAUvUa82oVS4pfPcu1+ yK48ptMjfW5AwFRRYNudMwIEtFJBkMqaF8bMk8/sRXwxtEZFTHXFxsINSUK8jK6J LJhr9bTFUXltgg1fRjEB61g8EoFFt8VMRibJ3pA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=+6A4QN GUXgWshF8SRD55aff8GLNYeONz7poQYW7E0ek=; b=Hn5J6fFbBu8ourKRYnDV7u 2Hld6MxIDe8Y26pJ3194s3/inmN4yFoPJnvTA/WrNxOxkNXWeLSxJ14Zj/Z2mpcF hiYMi30t0+cSfXUDZD4oMsqClq0KVr85kjxBoLGE3fJCRR4ApDpB+JrBIApneV1g gBSFK/hIs08GVjNsdT/wc+jG8C7+QLyPyTK8LROLXswdyvv9JSrVHtWHUzCdv/gz mi6iML3kSoIRKiuZNxwJzQ0Wg518+NFO65Nw47UZa3JPjxnkQtoL6eVsEscCutyt 7HPukpwlJ7FFdLFaVDDUyjT/8B7p+g7J6F5kjfCpEaAEmnzr4Sg7AmVbTSfJjxwg == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrudehuddguddvudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefofgggkfgjfhffhffvufgtsehttdertderredtnecuhfhrohhmpedfffgr nhcunfgrnhhgihhllhgvfdcuoegurghnsehlrghnghhilhhlvgdrohhrgheqnecuffhomh grihhnpehimhhguhhrrdgtohhmnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghm pehmrghilhhfrhhomhepuggrnheslhgrnhhgihhllhgvrdhorhhg X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 3BC3A1880062; Tue, 24 Mar 2020 17:45:11 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.7-1021-g152deaf-fmstable-20200319v1 Mime-Version: 1.0 Message-Id: In-Reply-To: References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> <20200322.214355.415142200.sthaug@nethelp.no> <20200322230339.GB5808@in-addr.com> <8141b371-5cdf-4f82-8959-5fa1071aa6dd@www.fastmail.com> Date: Tue, 24 Mar 2020 17:44:50 -0400 From: "Dan Langille" To: "Gary Palmer" Cc: sthaug , "Pieper, Jeffrey E" , freebsd-net@freebsd.org Subject: Re: SFP+ on PRO/10GbE Content-Type: text/plain X-Rspamd-Queue-Id: 48n4YD3LGbz4cjK X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=langille.org header.s=fm3 header.b=us6gpUyF; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=Hn5J6fFb; dmarc=pass (policy=none) header.from=langille.org; spf=pass (mx1.freebsd.org: domain of dan@langille.org designates 66.111.4.27 as permitted sender) smtp.mailfrom=dan@langille.org X-Spamd-Result: default: False [-5.58 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[langille.org:s=fm3,messagingengine.com:s=fm2]; XM_UA_NO_VERSION(0.01)[]; URIBL_BLOCKED(0.00)[imgur.com.multi.uribl.com,langille.org.multi.uribl.com,messagingengine.com.multi.uribl.com]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.27]; RCPT_COUNT_THREE(0.00)[4]; MIME_GOOD(-0.10)[text/plain]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[langille.org:+,messagingengine.com:+]; DMARC_POLICY_ALLOW(-0.50)[langille.org,none]; IP_SCORE(-3.49)[ip: (-9.84), ipnet: 66.111.4.0/24(-4.89), asn: 11403(-2.69), country: US(-0.05)]; RCVD_IN_DNSWL_LOW(-0.10)[27.4.111.66.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; MID_RHS_WWW(0.50)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 21:45:37 -0000 On Tue, Mar 24, 2020, at 5:23 PM, Dan Langille wrote: > On Sun, Mar 22, 2020, at 7:24 PM, Dan Langille wrote: > > On Sun, Mar 22, 2020, at 7:03 PM, Gary Palmer wrote: > > > On Sun, Mar 22, 2020 at 05:11:20PM -0400, Dan Langille wrote: > > > > On Sun, Mar 22, 2020, at 4:43 PM, sthaug@nethelp.no wrote: > > > > > > Partial success. The card is now able to use an SFP+ optic. It warns me > > > > > > when the optic is installed: > > > > > > > > > > > > Mar 22 16:49:45 r720-01 kernel: WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules. > > > > > > > > > > > > I cannot use an SFP+ optic at the switch. The connection just does not happen. > > > > > > > > > > > > If I go back to the original SFP optic, the connection occurs, as expected at 1G. > > > > > > > > > > > > On the switch side, I've tried a known good optic from an existing connection. > > > > > > > > > > > > I could install an PRO/10GbE instead, that has a built-in transceiver. I have > > > > > > two of those in use now, both working on 10G. > > > > > > > > > > Have you tried connected it to something other than the Unifi switch? > > > > > > > > The only SFP+ capable switches I have are Unifi. > > > > > > > > I just tried the other switch (US-48) which had one SFP+ port free. Same issues there. > > > > > > Did the Unifi switch see the SFP+ optics on the switch end? > > > > That port never lit up on the switch. I didn't check anything else, > > switch-related. > > I just inserted a 10G transceiver into port 8. It shows up as a 10GBase-SR > > Screen shots at https://imgur.com/a/SSEgyxN > > At the same time, on the server side, ifconfig indicates: > > status: no carrier > supported media: > media autoselect > media 10Gbase-SR > nd6 options=23 > plugged: SFP/SFP+/SFP28 10G Base-SR (LC) > vendor: AVAGO PN: AFBR-703SDZ SN: AD0946A00YB DATE: 2009-11-10 > module temperature: 36.75 C Voltage: 3.32 Volts > RX: 0.67 mW (-1.72 dBm) TX: 0.56 mW (-2.46 dBm) > > I have not yet tried moving from auto-negotiate on both ends. With the switch set to 10 Gbps FDX, this command on the server gives an error: # ifconfig ix0 media 10Gbase-SR ix0: Invalid media type! Then I tried: # ifconfig ix0 down # ifconfig ix0 up Then it worked. 10G. I moved the cable back to the original 10G transceiver where it was doing 1G. That went to 10G. Auto-negotiate is not needed. Just the up/down to reset the NIC. All good now. :) Thank you. -- Dan Langille dan@langille.org From owner-freebsd-net@freebsd.org Tue Mar 24 21:46:16 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 397A926BB0D for ; Tue, 24 Mar 2020 21:46:16 +0000 (UTC) (envelope-from dan@langille.org) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48n4ZD6lpJz4d2v for ; Tue, 24 Mar 2020 21:46:10 +0000 (UTC) (envelope-from dan@langille.org) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id B3F175C02CE; Tue, 24 Mar 2020 17:46:03 -0400 (EDT) Received: from imap36 ([10.202.2.86]) by compute2.internal (MEProxy); Tue, 24 Mar 2020 17:46:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=langille.org; h= mime-version:message-id:in-reply-to:references:date:from:to :subject:content-type:content-transfer-encoding; s=fm3; bh=0Ng0Y YxPmFmqdaGH+g3MlsVFmglPMnjW5Anf90as1hU=; b=oeqaGMovMuOqs+NNBl89m YgZj8i7+uTjI9E08gNeGBGZKWQ14/z8ADK77LU4JY31uwnay0U3X47oOboMNdCPk cyt8JJKkyfuPUgZZM07a3iAzgepnlVgPkry+RoaedR+81X/WBC87gnBSRs87v3qr rKgJOPZocCmU4MVJifG4wov3/vU3eJS3tqCPpi+Ggc24RKidyVPEPVRtC9Jl7c7w +wbEX/04BxSG957GB5XOEB1Tl12sMg/XDzwlJZNEoSvv667OQOQwkCR4WoGQxGfs 5F7H+78+rPrPHEl55RghiPwRiZOg+zCr8hZzUuxMXTkhxA4LilttAQ2BZgVYKVOF Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=0Ng0YYxPmFmqdaGH+g3MlsVFmglPMnjW5Anf90as1 hU=; b=Mxs7Pp5/durh4LWhmWgf3nfbBknVAG4nud9ZelXzbxnhjM9di+UdRwWzI jjcUkG7yc4Zk4P285rfDq7zcAuhdTHxSVMpfKEyYb+3q2p4Ytgsvoj5zn4Zf2bhH VdV1/hsdUiio3QOjHS2JvdvK3wVa1z7H9wup3XfgGRNolBDy/OS4z2AeeE6iYjDw pLWwQ0uo5EFX5yxJRv6zf0OBIP0GSKNK+LE7V3b2AYXmqJcNFoPRU8KS564cH9Le sqA/sqZZOpm1CmDooze9CR0Sg3vcPZy+Z543f/QUzeNsh6agJ4eHpZBqdtE2XZRc o16w22HgJaobHlhKynRDi5ENU4ocQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrudehuddguddvudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefofgggkfgjfhffhffvufgtgfesthhqredtreerjeenucfhrhhomhepfdff rghnucfnrghnghhilhhlvgdfuceouggrnheslhgrnhhgihhllhgvrdhorhhgqeenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegurghnsehlrghn ghhilhhlvgdrohhrgh X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id D287F1880067; Tue, 24 Mar 2020 17:46:02 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.7-1021-g152deaf-fmstable-20200319v1 Mime-Version: 1.0 Message-Id: <0afabbfe-ffd8-43f6-aeb8-647712232a0c@www.fastmail.com> In-Reply-To: <3265c7ca-656c-9269-c889-fe5499958232@akhmatov.ru> References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> <88679204-4c23-7fce-681d-ab633b326fef@akhmatov.ru> <3265c7ca-656c-9269-c889-fe5499958232@akhmatov.ru> Date: Tue, 24 Mar 2020 17:45:41 -0400 From: "Dan Langille" To: "Sergey Akhmatov" , "Pieper, Jeffrey E" , "freebsd-net@freebsd.org" Subject: Re: SFP+ on PRO/10GbE Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 48n4ZD6lpJz4d2v X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=langille.org header.s=fm3 header.b=oeqaGMov; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=Mxs7Pp5/; dmarc=pass (policy=none) header.from=langille.org; spf=pass (mx1.freebsd.org: domain of dan@langille.org designates 66.111.4.27 as permitted sender) smtp.mailfrom=dan@langille.org X-Spamd-Result: default: False [-5.58 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[langille.org:s=fm3,messagingengine.com:s=fm2]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.27:c]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; URIBL_BLOCKED(0.00)[messagingengine.com.multi.uribl.com,langille.org.multi.uribl.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[langille.org:+,messagingengine.com:+]; DMARC_POLICY_ALLOW(-0.50)[langille.org,none]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; IP_SCORE(-3.49)[ip: (-9.84), ipnet: 66.111.4.0/24(-4.89), asn: 11403(-2.69), country: US(-0.05)]; MID_RHS_WWW(0.50)[]; RCVD_IN_DNSWL_LOW(-0.10)[27.4.111.66.list.dnswl.org : 127.0.5.1] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 21:46:16 -0000 On Tue, Mar 24, 2020, at 4:58 PM, Sergey Akhmatov wrote: >=20 > On 2020-03-24 11:22 PM, Dan Langille wrote: > > > > media: Ethernet autoselect (1000baseSX ) > > status: active > > supported media: > > media autoselect > > media 10Gbase-SR > > nd6 options=3D23 > > plugged: SFP/SFP+/SFP28 10G Base-SR (LC) > > vendor: AVAGO PN: AFBR-703SDZ SN: AD0946A00YB DATE: 2009-11-10 > > module temperature: 38.09 C Voltage: 3.33 Volts > > RX: 0.31 mW (-4.95 dBm) TX: 0.56 mW (-2.48 dBm) > > [dan@r720-01:~] $ > > > > That looks right to me, but I have never inspected this before. >=20 > Not right actually. The supported media is "10Gbase-SR" which is corre= ct=20 > for this type of SFP,=C2=A0 but autodetected is "1000baseSX" and it's = not=20 > even listed as supported. >=20 > That doesn't make sense to me, maybe try to set media type manually? >=20 > # ifconfig ix0 media 10gbase-sr See my recent post. ifconfig down/up fixed the NIC. Thank you for your help. It is appreciated. --=20 Dan Langille dan@langille.org From owner-freebsd-net@freebsd.org Tue Mar 24 22:48:49 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B59DE26D03F for ; Tue, 24 Mar 2020 22:48:49 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48n5y50Sb2z430G for ; Tue, 24 Mar 2020 22:48:28 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm1-x332.google.com with SMTP id c81so410845wmd.4 for ; Tue, 24 Mar 2020 15:48:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=zaev03eGGrGmT7n653L1k36v+23UFXqrotDvhxOnsVQ=; b=pFpcyL1KUV1XCnPstf0FjWkJWbLZPQ/Sj7HtVkIcfTqxX4t2eEd8uNifG+PAUb3UPG FZMFHrzKlDVLMXrxrgsmnFEA1qDSv82yMPmhS/fkyF9LKjRtGud22ICe48cihdYP4Jj1 QWtz54Pj8GXdS+lyUEpwewysJ8jUYE0LU6ugl3P0JtM1/KMFRNOWEQI4hVGsYVqu26YW kbReYg2eId83AXJ3XY86ANSog50B9hWMP4vMxKYOH/2ALMYy7Elczc6ECubINT+kjYj+ KTzmaXxwy2WN+tVJToslJ1aL/ZCS/0UkPC0HTjrcP+yfGUDp+VmesOhzV5jRVs/fIF3W u1Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=zaev03eGGrGmT7n653L1k36v+23UFXqrotDvhxOnsVQ=; b=bKGput3rJAyfnnS8xvMHFfGfPjfxa7NZ5iwAUSxuWo79lX13IQQ3zKsYifvUvHLUM5 OthvqMA9XPSqgl4kXAt8QD15gg4ci6i0q9Lv+6LKupjkjDZkPnIQo2pPKYqRua7Sguza 08BYwF0gn8NL9ItniErYxZhBWRzQz1yZyTV3zaJ+f1HJp7eDknhmIWuGC2I/wyEPIQTY mvcZwQCQnXfydEfcSERsZF0Q9mOYM07R2RFfB1qO0B7nNFsTUqRrPDwtFHOAWfOz8IO2 DB6np+ArJ83gPe4YSYq7s1nl27+aAcO7EdCHGwie2rSnYeaipsi40D5nOCVJUWiFC499 UH8Q== X-Gm-Message-State: ANhLgQ32darCkqXmaxFlakUMAVqNVdOgORfeaHREX9/jhCcavVoH7ywC 7qzZlyfS0X2FhQlpA5eOJXUEvcuHyokrdw== X-Google-Smtp-Source: ADFU+vvgBqSAu/BSpqviDa4io9hqi9YBIyKYiowepZMsH+UiEY7E5bN8rLsoOu0Ux2lHQ2ZJlzw0Vg== X-Received: by 2002:a1c:9658:: with SMTP id y85mr293396wmd.63.1585090098389; Tue, 24 Mar 2020 15:48:18 -0700 (PDT) Received: from [10.44.128.75] ([193.117.175.106]) by smtp.gmail.com with ESMTPSA id m19sm6607875wml.48.2020.03.24.15.48.17 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 24 Mar 2020 15:48:17 -0700 (PDT) Subject: Re: SFP+ on PRO/10GbE To: freebsd-net@freebsd.org References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> <88679204-4c23-7fce-681d-ab633b326fef@akhmatov.ru> <3265c7ca-656c-9269-c889-fe5499958232@akhmatov.ru> <0afabbfe-ffd8-43f6-aeb8-647712232a0c@www.fastmail.com> From: Steven Hartland Message-ID: Date: Tue, 24 Mar 2020 22:48:17 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: <0afabbfe-ffd8-43f6-aeb8-647712232a0c@www.fastmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-Rspamd-Queue-Id: 48n5y50Sb2z430G X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=multiplay-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=pFpcyL1K; dmarc=pass (policy=none) header.from=multiplay.co.uk; spf=pass (mx1.freebsd.org: domain of killing@multiplay.co.uk designates 2a00:1450:4864:20::332 as permitted sender) smtp.mailfrom=killing@multiplay.co.uk X-Spamd-Result: default: False [-5.41 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[multiplay-co-uk.20150623.gappssmtp.com:s=20150623]; URIBL_BLOCKED(0.00)[gappssmtp.com.multi.uribl.com]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; DKIM_TRACE(0.00)[multiplay-co-uk.20150623.gappssmtp.com:+]; ZRD_FAIL(0.00)[query timed out]; RCVD_IN_DNSWL_NONE(0.00)[2.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[multiplay.co.uk,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-2.41)[ip: (-9.11), ipnet: 2a00:1450::/32(-2.38), asn: 15169(-0.48), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 22:48:49 -0000 Watch out for the fact some nics don't fully initialise until they are given an IP. On 24/03/2020 21:45, Dan Langille wrote: > On Tue, Mar 24, 2020, at 4:58 PM, Sergey Akhmatov wrote: >> On 2020-03-24 11:22 PM, Dan Langille wrote: >>> media: Ethernet autoselect (1000baseSX ) >>> status: active >>> supported media: >>> media autoselect >>> media 10Gbase-SR >>> nd6 options=23 >>> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) >>> vendor: AVAGO PN: AFBR-703SDZ SN: AD0946A00YB DATE: 2009-11-10 >>> module temperature: 38.09 C Voltage: 3.33 Volts >>> RX: 0.31 mW (-4.95 dBm) TX: 0.56 mW (-2.48 dBm) >>> [dan@r720-01:~] $ >>> >>> That looks right to me, but I have never inspected this before. >> Not right actually. The supported media is "10Gbase-SR" which is correct >> for this type of SFP,  but autodetected is "1000baseSX" and it's not >> even listed as supported. >> >> That doesn't make sense to me, maybe try to set media type manually? >> >> # ifconfig ix0 media 10gbase-sr > See my recent post. ifconfig down/up fixed the NIC. > > Thank you for your help. It is appreciated. > From owner-freebsd-net@freebsd.org Wed Mar 25 01:06:50 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3A6592707A7 for ; Wed, 25 Mar 2020 01:06:50 +0000 (UTC) (envelope-from dan@langille.org) Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48n91N647wz3QGc for ; Wed, 25 Mar 2020 01:06:32 +0000 (UTC) (envelope-from dan@langille.org) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.west.internal (Postfix) with ESMTP id 551637BE for ; Tue, 24 Mar 2020 20:47:49 -0400 (EDT) Received: from imap36 ([10.202.2.86]) by compute2.internal (MEProxy); Tue, 24 Mar 2020 20:47:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=langille.org; h= mime-version:message-id:in-reply-to:references:date:from:to :subject:content-type:content-transfer-encoding; s=fm3; bh=pEc0I 7ppUBolEH8Gf+dYaxJwwWJDmbB+c7S0IujpiGM=; b=YorONpzcfTKPJJOp97Afq AXPfo/Wc0jUCjjadmGkZ0aVF75oODKA8fSEZt0HKelnEANB4Ix9xOBrJuatJYBC6 m3ei1IIEdGrIc5vYns8KaOCANQB5dH6DDg+cQ7MgPivM+uZBsKsXdmK489FQs/DS x/j3RVwJdlJyR5ByHVsugUeVLf0Llezxr4kim+YFxnmXTL1Pyj7Jw4Li/+Vs3I5U yVB83Sb7btx+AM45elCh0fB0lBez82tHquK/1cTnsdF4tBSnSdFRtxvMnJf36Hom YFegxrkMYFpyghm8gYH4CSXNyoN0bVDZe0DH99IYSPqfRkA+d0sI4FuNHAC+680Z Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=pEc0I7ppUBolEH8Gf+dYaxJwwWJDmbB+c7S0Iujpi GM=; b=E1QA7a9Xjt+Dk/pVrTP6qcYqzj05jCUirp3LrrKbodgmr95dTZCdLsLia /+oEMjwjfF26jpw5MzzKt5NbslLZvqxtHSb9oowFf30+wmgiW3JTWTaaNLKkg/Bl +R+A5STPMTB4WQdB+IK1ay4ZwdQQ+O4mnMDzrTeG3TbOP6BfVmY/vVVpb1d8MvlT Saiybs/UMRLV7qjcp4GGxLgXSaFRR75MislrCf9YNSWIBrDvJG18nYysvwk7wJ1f LybDcCVuiF6HA/PCfLq43vSJCml/nr1V4MZvORDxQy+HrwH+LXOVc7JGhmqbZ1vR H1bM3+fk8/1pwTXoUDumYJFqirJfw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrudehvddgvdehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtgfesth hqredtreerjeenucfhrhhomhepfdffrghnucfnrghnghhilhhlvgdfuceouggrnheslhgr nhhgihhllhgvrdhorhhgqeenucffohhmrghinhepfhhrvggvsghsugdrohhrghenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegurghnsehlrghn ghhilhhlvgdrohhrgh X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 5F3F61880062; Tue, 24 Mar 2020 20:47:48 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.7-1021-g152deaf-fmstable-20200319v1 Mime-Version: 1.0 Message-Id: <3b2702bb-2186-47fd-a01e-683a6772173b@www.fastmail.com> In-Reply-To: References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> <88679204-4c23-7fce-681d-ab633b326fef@akhmatov.ru> <3265c7ca-656c-9269-c889-fe5499958232@akhmatov.ru> <0afabbfe-ffd8-43f6-aeb8-647712232a0c@www.fastmail.com> Date: Tue, 24 Mar 2020 20:47:28 -0400 From: "Dan Langille" To: freebsd-net@freebsd.org Subject: Re: SFP+ on PRO/10GbE Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 48n91N647wz3QGc X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=langille.org header.s=fm3 header.b=YorONpzc; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=E1QA7a9X; dmarc=pass (policy=none) header.from=langille.org; spf=pass (mx1.freebsd.org: domain of dan@langille.org designates 64.147.123.21 as permitted sender) smtp.mailfrom=dan@langille.org X-Spamd-Result: default: False [-5.60 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[langille.org:s=fm3,messagingengine.com:s=fm2]; XM_UA_NO_VERSION(0.01)[]; URIBL_BLOCKED(0.00)[langille.org.multi.uribl.com,messagingengine.com.multi.uribl.com]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.21]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; IP_SCORE(-3.51)[ip: (-9.87), ipnet: 64.147.123.0/24(-4.92), asn: 11403(-2.69), country: US(-0.05)]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[langille.org:+,messagingengine.com:+]; DMARC_POLICY_ALLOW(-0.50)[langille.org,none]; RCVD_IN_DNSWL_LOW(-0.10)[21.123.147.64.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; MID_RHS_WWW(0.50)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2020 01:06:50 -0000 They seem to come up on boot ok for me. This system has not been reboote= d since going to 10G.=20 Is that what you meant? On Tue, Mar 24, 2020, at 6:48 PM, Steven Hartland wrote: > Watch out for the fact some nics don't fully initialise until they are= =20 > given an IP. >=20 > On 24/03/2020 21:45, Dan Langille wrote: > > On Tue, Mar 24, 2020, at 4:58 PM, Sergey Akhmatov wrote: > >> On 2020-03-24 11:22 PM, Dan Langille wrote: > >>> media: Ethernet autoselect (1000baseSX ) > >>> status: active > >>> supported media: > >>> media autoselect > >>> media 10Gbase-SR > >>> nd6 options=3D23 > >>> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) > >>> vendor: AVAGO PN: AFBR-703SDZ SN: AD0946A00YB DATE: 2009-11-10 > >>> module temperature: 38.09 C Voltage: 3.33 Volts > >>> RX: 0.31 mW (-4.95 dBm) TX: 0.56 mW (-2.48 dBm) > >>> [dan@r720-01:~] $ > >>> > >>> That looks right to me, but I have never inspected this before. > >> Not right actually. The supported media is "10Gbase-SR" which is co= rrect > >> for this type of SFP,=C2=A0 but autodetected is "1000baseSX" and it= 's not > >> even listed as supported. > >> > >> That doesn't make sense to me, maybe try to set media type manually= ? > >> > >> # ifconfig ix0 media 10gbase-sr > > See my recent post. ifconfig down/up fixed the NIC. > > > > Thank you for your help. It is appreciated. > > >=20 > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"= > --=20 Dan Langille dan@langille.org From owner-freebsd-net@freebsd.org Wed Mar 25 08:06:16 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2D226279DD2 for ; Wed, 25 Mar 2020 08:06:16 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: from mail-io1-f49.google.com (mail-io1-f49.google.com [209.85.166.49]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48nLKC2304z4JLK for ; Wed, 25 Mar 2020 08:05:50 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: by mail-io1-f49.google.com with SMTP id n21so1290916ioo.10 for ; Wed, 25 Mar 2020 01:05:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=lMkYb3fQMJZSE6/AGwDVYk+VorZLcI7z5NSYScFJOpM=; b=zNdisJxNiLNVJdjgwGZ9D8P8ymJOkx5JKxvBQzmMccqd8hLf7O58rvAFHn2/6OUEvD fUEz4gJIKgHEHBPTXSQOCJ3qcEl8Wk6MFDOIXdQn2/s1s7AFaqcWIVxGzTkuShBUxlRc /nXpnjdsmsKCJfi4j4zdhk1b4D1I36o3JiiCt2S1HtaAYWFZn/HUDb6wjAxVlkC1xTvO 037DX8aQDETniKpWEY8Zlw9tvKrcFVHZYWK8lkthcA//hKz3kCNHxGKqOWuP4HiHVPKd Ob3oyjIRCGqE1csaC+Hn6bBf05Fn++xPPm/acZ2jiso0FwsIrDeq4PT4yHn6zxGj8AA2 q2SA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=lMkYb3fQMJZSE6/AGwDVYk+VorZLcI7z5NSYScFJOpM=; b=aU/krKlEFlaIouMQk4KUsi/JcOXlYN9lsx8yA82TZrnFwFqaRt4mfAiQWXLC4Mwct2 kngVmf5vPhlBgmoPup2Co5D3m5+SJHYJywS7g9CMJM3wb1Qc5TztWV1cNhnyuJjdTqal yrJwedVUbZPAszz5e5OF0qNcZv0DtFOVuz8WoG3mdN/XaN5P3s5/+4SjHEMvMOB7Z4Sz cIK1WPYF+6IpzTtkSulCvK4OpMRrt9tRNuGjIyekOTNgGP4FcdDTppk4FmkbhAaZdDxU g0alnli92HSBB0R2pSsobaiv9JKKzQbXYqug2KpgjGUgjvAa7IT+/0zF5BGIR3G2dr+H dvvg== X-Gm-Message-State: ANhLgQ28xCO84Dm1Zmb3LVSK25y00HG/qE+ssr1pDJ4dTKm6Rd73DQDq DO/Jf2yEKphk17vSTy8NptAzR219P+0AhEIiQqAZex05 X-Google-Smtp-Source: ADFU+vsnXHNrb7JBv8qtvzqPf9kQYOXB7TiEeZwGix2hzZ4RQrSmvCtjShHd//pkFApCKqWThhkOa0bXN0cEtTIg6e0= X-Received: by 2002:a6b:7c03:: with SMTP id m3mr1880296iok.36.1585123541383; Wed, 25 Mar 2020 01:05:41 -0700 (PDT) MIME-Version: 1.0 References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> <88679204-4c23-7fce-681d-ab633b326fef@akhmatov.ru> <3265c7ca-656c-9269-c889-fe5499958232@akhmatov.ru> <0afabbfe-ffd8-43f6-aeb8-647712232a0c@www.fastmail.com> <3b2702bb-2186-47fd-a01e-683a6772173b@www.fastmail.com> In-Reply-To: <3b2702bb-2186-47fd-a01e-683a6772173b@www.fastmail.com> From: Steven Hartland Date: Wed, 25 Mar 2020 08:05:30 +0000 Message-ID: Subject: Re: SFP+ on PRO/10GbE To: Dan Langille Cc: freebsd-net@freebsd.org X-Rspamd-Queue-Id: 48nLKC2304z4JLK X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=multiplay-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=zNdisJxN; dmarc=pass (policy=none) header.from=multiplay.co.uk; spf=pass (mx1.freebsd.org: domain of steven@multiplay.co.uk designates 209.85.166.49 as permitted sender) smtp.mailfrom=steven@multiplay.co.uk X-Spamd-Result: default: False [-4.43 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[multiplay-co-uk.20150623.gappssmtp.com:s=20150623]; URIBL_BLOCKED(0.00)[gappssmtp.com.multi.uribl.com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[multiplay-co-uk.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[49.166.85.209.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[multiplay.co.uk,none]; FORGED_SENDER(0.30)[killing@multiplay.co.uk,steven@multiplay.co.uk]; RWL_MAILSPIKE_POSSIBLE(0.00)[49.166.85.209.rep.mailspike.net : 127.0.0.17]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-1.73)[ip: (-6.26), ipnet: 209.85.128.0/17(-1.86), asn: 15169(-0.48), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[killing@multiplay.co.uk,steven@multiplay.co.uk]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2020 08:06:16 -0000 Just plugging the nic in isn=E2=80=99t always enough for it to link it need= s to be configured. If you have it configured in the os then this won=E2=80=99t be = a problem. Does that make sense? On Wed, 25 Mar 2020 at 01:10, Dan Langille wrote: > They seem to come up on boot ok for me. This system has not been rebooted > since going to 10G. > > Is that what you meant? > > On Tue, Mar 24, 2020, at 6:48 PM, Steven Hartland wrote: > > Watch out for the fact some nics don't fully initialise until they are > > given an IP. > > > > On 24/03/2020 21:45, Dan Langille wrote: > > > On Tue, Mar 24, 2020, at 4:58 PM, Sergey Akhmatov wrote: > > >> On 2020-03-24 11:22 PM, Dan Langille wrote: > > >>> media: Ethernet autoselect (1000baseSX > ) > > >>> status: active > > >>> supported media: > > >>> media autoselect > > >>> media 10Gbase-SR > > >>> nd6 options=3D23 > > >>> plugged: SFP/SFP+/SFP28 10G Base-SR (LC) > > >>> vendor: AVAGO PN: AFBR-703SDZ SN: AD0946A00YB DATE: 2009-11-10 > > >>> module temperature: 38.09 C Voltage: 3.33 Volts > > >>> RX: 0.31 mW (-4.95 dBm) TX: 0.56 mW (-2.48 dBm) > > >>> [dan@r720-01:~] $ > > >>> > > >>> That looks right to me, but I have never inspected this before. > > >> Not right actually. The supported media is "10Gbase-SR" which is > correct > > >> for this type of SFP, but autodetected is "1000baseSX" and it's not > > >> even listed as supported. > > >> > > >> That doesn't make sense to me, maybe try to set media type manually? > > >> > > >> # ifconfig ix0 media 10gbase-sr > > > See my recent post. ifconfig down/up fixed the NIC. > > > > > > Thank you for your help. It is appreciated. > > > > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > -- > Dan Langille > dan@langille.org > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Wed Mar 25 17:47:14 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 310772A7193 for ; Wed, 25 Mar 2020 17:47:14 +0000 (UTC) (envelope-from dan@langille.org) Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com [64.147.123.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48nbCj4W9Zz4301 for ; Wed, 25 Mar 2020 17:46:57 +0000 (UTC) (envelope-from dan@langille.org) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.west.internal (Postfix) with ESMTP id CEC27606 for ; Wed, 25 Mar 2020 13:46:46 -0400 (EDT) Received: from imap36 ([10.202.2.86]) by compute2.internal (MEProxy); Wed, 25 Mar 2020 13:46:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=langille.org; h= mime-version:message-id:in-reply-to:references:date:from:to :subject:content-type:content-transfer-encoding; s=fm3; bh=I532O MFVd1C6TNlz802oJN+i4fcyoY9b3LtblAjkj9o=; b=ETK9sfpNQ49azT02ZW3H1 GGzOg1WCHouPxMxfGmdInyvT0KslEwZhbJR4YIw71Rq9FeBrbSM0XSS+LsRc8NGm VckxvXucqLRrEzv2DBjezijodDcyTcV/I9mT/zlOHWTjcIkDkCfTAYIRw8/0AN+I POU0MIP2XNREw+GkDqt8rPmZ5OyHbeOivYPFQRp/vpL0+iAlnaXFBloMMJtYJbhI pFepVKaCR97aDOXhpd1ZCBKwnu3wyeKeaL1duDyoIjZGNidhLPs4negTQczLWe/M t0ckQs9IctWFCITvjtWnDPmFCcw9p0ziVzMTVvFX/6lj+ArPHZMRGghG7f3AWbtP g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=I532OMFVd1C6TNlz802oJN+i4fcyoY9b3LtblAjkj 9o=; b=jI8iiBRWVbEsqclyaGmV89fGFfXX8T/gLI5ldY5LUcwuzF0sC3JFcio/7 wAYBIUWirg/Yb/7h4QlhRxZy8wIVXWnapjaea9lBDucTTCyEULDLW3tMb/i+08FT 9acYp5m6Xh/BH2aMB51fn8cMUJjrCK+LUskQqp1A0rle7pIFel3KoIOt5Zh/VtBf r3XNYSWT9Z6PUZpxtKC7suOL6/oUMjCiGEVUIKOCHfA6mPy2skLh6AqAFtKydWND 2Jgnd60lFXxLVdSQDuTCHIemt/r0YfhPTtXlP3j7YYrJw/8v45j9lApMBsP7qRQo rwkKW4wmQWMQwyRot5Bw6QBcZzhrA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrudehgedgjeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtgfesth hqredtreerjeenucfhrhhomhepfdffrghnucfnrghnghhilhhlvgdfuceouggrnheslhgr nhhgihhllhgvrdhorhhgqeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmh grihhlfhhrohhmpegurghnsehlrghnghhilhhlvgdrohhrgh X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 319451880062; Wed, 25 Mar 2020 13:46:46 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.7-1021-g152deaf-fmstable-20200319v1 Mime-Version: 1.0 Message-Id: <3d6488bd-5f55-4903-bd16-8f0fb467f379@www.fastmail.com> In-Reply-To: References: <2A35EA60C3C77D438915767F458D6568B90935EC@ORSMSX111.amr.corp.intel.com> <88679204-4c23-7fce-681d-ab633b326fef@akhmatov.ru> <3265c7ca-656c-9269-c889-fe5499958232@akhmatov.ru> <0afabbfe-ffd8-43f6-aeb8-647712232a0c@www.fastmail.com> <3b2702bb-2186-47fd-a01e-683a6772173b@www.fastmail.com> Date: Wed, 25 Mar 2020 13:46:24 -0400 From: "Dan Langille" To: freebsd-net@freebsd.org Subject: Re: SFP+ on PRO/10GbE Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 48nbCj4W9Zz4301 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=langille.org header.s=fm3 header.b=ETK9sfpN; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=jI8iiBRW; dmarc=pass (policy=none) header.from=langille.org; spf=pass (mx1.freebsd.org: domain of dan@langille.org designates 64.147.123.25 as permitted sender) smtp.mailfrom=dan@langille.org X-Spamd-Result: default: False [-5.49 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[langille.org:s=fm3,messagingengine.com:s=fm2]; XM_UA_NO_VERSION(0.01)[]; URIBL_BLOCKED(0.00)[messagingengine.com.multi.uribl.com,langille.org.multi.uribl.com]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.25]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; IP_SCORE(-3.50)[ip: (-9.84), ipnet: 64.147.123.0/24(-4.92), asn: 11403(-2.69), country: US(-0.05)]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[langille.org:+,messagingengine.com:+]; DMARC_POLICY_ALLOW(-0.50)[langille.org,none]; RCVD_IN_DNSWL_FAIL(0.00)[25.123.147.64.list.dnswl.org:query timed out]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; MID_RHS_WWW(0.50)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2020 17:47:14 -0000 On Wed, Mar 25, 2020, at 4:05 AM, Steven Hartland wrote: > Just plugging the nic in isn=E2=80=99t always enough for it to link it= needs to be > configured. If you have it configured in the os then this won=E2=80=99= t be a > problem. >=20 > Does that make sense? I think you are saying that my entries in /etc/rc.conf are what helps. When I next reboot this server, I'll keep an eye on the link. Thank you. --=20 Dan Langille dan@langille.org From owner-freebsd-net@freebsd.org Wed Mar 25 22:31:51 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 38B4A26566B for ; Wed, 25 Mar 2020 22:31:51 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48njXQ18qSz3NFH for ; Wed, 25 Mar 2020 22:31:50 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from mail-qv1-f49.google.com (mail-qv1-f49.google.com [209.85.219.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: vmaffione) by smtp.freebsd.org (Postfix) with ESMTPSA id BE33918C1A for ; Wed, 25 Mar 2020 22:31:43 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: by mail-qv1-f49.google.com with SMTP id o7so1983141qvq.8 for ; Wed, 25 Mar 2020 15:31:43 -0700 (PDT) X-Gm-Message-State: ANhLgQ0ywjsJ4ddU1kXods8M0250MfBzbMlTy1TAIbeC9QpdcGLkrluW KpYijwNYTYxpdTCKEVAztW5HQCjXXtGENZEWJhs= X-Google-Smtp-Source: ADFU+vtnDmBTweNDTDnV/NRXDCWonXs8CEZTB7jP0SljpAqBfleaWcIB8Ybig2ZVV/huakj0tzwDoB4+QKY2HyufZPU= X-Received: by 2002:ad4:46e6:: with SMTP id h6mr5313194qvw.208.1585175502945; Wed, 25 Mar 2020 15:31:42 -0700 (PDT) MIME-Version: 1.0 References: <20200324123721.GA26248@staff.retn.net> <20200324140839.GB26248@staff.retn.net> In-Reply-To: <20200324140839.GB26248@staff.retn.net> From: Vincenzo Maffione Date: Wed, 25 Mar 2020 23:31:30 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: netmap/ixl and crc addition.. To: Alexandre Snarskii Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2020 22:31:51 -0000 Hi Alexandre, Thanks. Your patch looks good to me. I assume you have tested it? I will commit that to stable/11. The issue you report on stable/12 is more worrisome. The 'no space in TX ring' condition (head==cur==tail) is ok per-se: on a subsequent poll() wakeup (e.g. TX interrupt) or explicit ioctl(NIOCTXSYNC) you should see tail moving forward, therefore freeing some space to be used in the ring. However, the ring_reinit means that something is going wrong: either your application is using the TX ring incorrectly, or there is a bug in the netmap iflib code. Since FreeBSD 12, netmap support is provided by iflib, while before netmap support was provided directly by the ixl driver. In any case, it would probably help if you could provide some more detailed info (how to reproduce the problem). Cheers, Vincenzo Il giorno mar 24 mar 2020 alle ore 15:12 Alexandre Snarskii < snar@snar.spb.ru> ha scritto: > On Tue, Mar 24, 2020 at 03:37:36PM +0300, Alexandre Snarskii wrote: > > > > Hi! > > > > Long story short: looks like intel x722 does not by default add CRC to > > outbound frames, so with FreeBSD 11-stable netmap-generated traffic is > > dropped on the next port.. Fix is simple, attached. > > ... add missing attach :( > > > > > The same behaviour of 'unconditionally ask card to compute crc' can > > be found in both if_ixl: > > > https://svnweb.freebsd.org/base/stable/11/sys/dev/ixl/ixl_txrx.c?view=markup#l408 > > and in DPDK i40e driver: > > > https://github.com/DPDK/dpdk/blob/master/drivers/net/i40e/i40e_rxtx.c#L1105 > > so, I guess, it's safe. > > > > PS: of course, first idea was to upgrade to FreeBSD 12-stable, but while > > this upgrade solved the crc problem, this version shows 'stalled tx > queue' > > problem: after CTXSYNC tail == head == cur, 'no space in ring' > condition. > > Attempts to ignore this condition led to continuous ring resets in > txcsync: > > > > Mar 17 20:21:08 host kernel: 668.224836 [1679] nm_txsync_prologue > ixl1 TX3: fail 'head < kring->rhead || head > kring->rtail' h 136 c 136 t > 135 rh 135 rc 135 rt 135 hc 135 ht 135 > > Mar 17 20:21:08 host kernel: 668.238300 [1787] netmap_ring_reinit > called for ixl1 TX3 > > > > PPS: hardware details: Dell VEP4600, based on Xeon D-2100 with > > two onboard X722 ports (actually, four, but two of them are not > > wired). > > > > CPU: Intel(R) Xeon(R) D-2187NT CPU @ 2.00GHz (2000.06-MHz K8-class CPU) > > Origin="GenuineIntel" Id=0x50654 Family=0x6 Model=0x55 Stepping=4 > > > > ixl0: 1.11.9-k> mem 0xfa000000-0xfaffffff,0xfb018000-0xfb01ffff irq 11 at device > 0.0 numa-domain 0 on pci12 > > ixl0: using 1024 tx descriptors and 1024 rx descriptors > > ixl0: fw 3.1.57069 api 1.5 nvm 3.33 etid 80001007 oem 1.263.0 > > ixl0: PF-ID[0]: VFs 32, MSIX 129, VF MSIX 5, QPs 384, I2C > > ixl0: Using MSIX interrupts with 9 vectors > > ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active > > ixl0: Ethernet address: 3c:2c:30:30:59:85 > > ixl0: SR-IOV ready > > ixl0: netmap queues/slots: TX 8/1024, RX 8/1024 > > > > ixl0@pci0:184:0:0: class=0x020000 card=0x00008086 chip=0x37d38086 > rev=0x04 hdr=0x00 > > vendor = 'Intel Corporation' > > device = 'Ethernet Connection X722 for 10GbE SFP+' > > class = network > > subclass = ethernet > > > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Wed Mar 25 22:53:28 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7FF9A265E6C for ; Wed, 25 Mar 2020 22:53:28 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48nk1M3FQFz41KC; Wed, 25 Mar 2020 22:53:27 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1jHEtX-0004yH-JW; Thu, 26 Mar 2020 01:53:07 +0300 Date: Thu, 26 Mar 2020 01:53:07 +0300 From: Slawa Olhovchenkov To: Vincenzo Maffione Cc: Alexandre Snarskii , "freebsd-net@freebsd.org" Subject: Re: netmap/ixl and crc addition.. Message-ID: <20200325225307.GM8028@zxy.spb.ru> References: <20200324123721.GA26248@staff.retn.net> <20200324140839.GB26248@staff.retn.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-Rspamd-Queue-Id: 48nk1M3FQFz41KC X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.995,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2020 22:53:28 -0000 On Wed, Mar 25, 2020 at 11:31:30PM +0100, Vincenzo Maffione wrote: > Hi Alexandre, > Thanks. Your patch looks good to me. I assume you have tested it? > I will commit that to stable/11. > > The issue you report on stable/12 is more worrisome. The 'no space in TX > ring' condition (head==cur==tail) is ok per-se: on a subsequent poll() > wakeup (e.g. TX interrupt) or explicit ioctl(NIOCTXSYNC) you should see > tail moving forward, therefore freeing some space to be used in the ring. > However, the ring_reinit means that something is going wrong: either your > application is using the TX ring incorrectly, or there is a bug in the > netmap iflib code. Since FreeBSD 12, netmap support is provided by iflib, > while before netmap support was provided directly by the ixl driver. > In any case, it would probably help if you could provide some more detailed > info (how to reproduce the problem). Just run pkt-gen for transmit. The conditions is guaranteed to occur (for me, at least). From owner-freebsd-net@freebsd.org Thu Mar 26 10:39:00 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ADB4527B1D3 for ; Thu, 26 Mar 2020 10:39:00 +0000 (UTC) (envelope-from snar@snar.spb.ru) Received: from staff.retn.net (staff.retn.net [IPv6:2a02:2d8:0:20ff:232a::8]) by mx1.freebsd.org (Postfix) with ESMTP id 48p1g95WpVz4TrF; Thu, 26 Mar 2020 10:38:44 +0000 (UTC) (envelope-from snar@snar.spb.ru) Received: from staff.retn.net (staff.retn.net [IPv6:2a02:2d8:0:20ff:232a::8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: snarspb@) by staff.retn.net (Postfix) with ESMTPSA id 56C77CEB; Thu, 26 Mar 2020 13:20:14 +0300 (MSK) Date: Thu, 26 Mar 2020 13:20:08 +0300 From: Alexandre Snarskii To: Vincenzo Maffione Cc: Alexandre Snarskii , "freebsd-net@freebsd.org" Subject: Re: netmap/ixl and crc addition.. Message-ID: <20200326102008.GA54860@staff.retn.net> References: <20200324123721.GA26248@staff.retn.net> <20200324140839.GB26248@staff.retn.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 48p1g95WpVz4TrF X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of snar@snar.spb.ru has no SPF policy when checking 2a02:2d8:0:20ff:232a::8) smtp.mailfrom=snar@snar.spb.ru X-Spamd-Result: default: False [0.57 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.12)[-0.117,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[snar.spb.ru]; AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.31)[-0.312,0]; R_SPF_NA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:9002, ipnet:2a02:2d8::/32, country:EU]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(-0.00)[country: EU(-0.01)]; ONCE_RECEIVED(0.10)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2020 10:39:00 -0000 On Wed, Mar 25, 2020 at 11:31:30PM +0100, Vincenzo Maffione wrote: > Hi Alexandre, >   Thanks. Your patch looks good to me. I assume you have tested it? Sure, this patch is already handles production traffic. > I will commit that to stable/11. Thanks. > > The issue you report on stable/12 is more worrisome. The 'no space in TX ring' > condition (head==cur==tail) is ok per-se: on a subsequent poll() wakeup (e.g. > TX interrupt) or explicit ioctl(NIOCTXSYNC) you should see tail moving forward, > therefore freeing some space to be used in the ring. Unfortunately, further NIOCTXSYNC did not freed queue. Let me explain my observations in details: this issue was reproduceable even under really light load (say, hundreds pps per queue). As the load was ligth, I was able to insert debugging output before and after any calls to ioctl and poll, code is like: while(1) { if (nm_tx_pending(txring)) { printf("%s: pre-txsync cur: %d, head: %d, tail: %d\n", thread_name, txring->cur, txring->head, txring->tail); if (ioctl(pa->fd, NIOCTXSYNC) == -1) { log event }; printf("%s: post-txsync cur: %d, haad: %d, tail: %d\n", thread_name, txring->cur, txring->head, txring->tail); }; // setup poll descriptors with POLLIN only printf("%s: pre-poll cur: %d, head: %d, tail: %d\n", thread_name, txring->cur, txring->head, txring->tail); ret = poll(fds, 2, 1000); printf("%s: post-poll cur: %d, head: %d, tail: %d\n", thread_name, txring->cur, txring->head, txring->tail); Logs in normal situation (only single thread shown): Notice:nz(ixl0:2): pre-txsync cur: 649, head: 649, tail: 647 Notice:nz(ixl0:2): post-txsync cur: 649, head: 649, tail: 648 Notice:nz(ixl0:2): pre-poll cur: 649, head: 649, tail: 648 Notice:nz(ixl0:2): post-poll cur: 649, head: 649, tail: 648 in pre-txsync: cur = head = tail + 2 (sometimes tail + 3), queue is nearly empty. in post-txsync: cur = head = tail + 1, queue is flushed. however, after some time, txcsync call advanced tail not to head - 1, but to head: Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 652 Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654 and following txcsyncs was not able to free (already empty) queue: Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654 application entered busy wait.. (in my case most packets just zero-copied from rxring to txring, so if there are no space in txring, packets are not consumed from rx). > However, the ring_reinit means that something is going wrong: either your > application is using the TX ring incorrectly, or there is a bug in the netmap ring_reinit was indeed caused by my incorrect use of txring: I tried to ignore situation head == tail and inject packet anyway.. > iflib code. Since FreeBSD 12, netmap support is provided by iflib, while before > netmap support was provided directly by the ixl driver. > In any case, it would probably help if you could provide some more detailed > info (how to reproduce the problem). > > Cheers, >   Vincenzo > > Il giorno mar 24 mar 2020 alle ore 15:12 Alexandre Snarskii > ha scritto: > > On Tue, Mar 24, 2020 at 03:37:36PM +0300, Alexandre Snarskii wrote: > > > > Hi! > > > > Long story short: looks like intel x722 does not by default add CRC to > > outbound frames, so with FreeBSD 11-stable netmap-generated traffic is > > dropped on the next port.. Fix is simple, attached. > > ... add missing attach :( > > > > > The same behaviour of 'unconditionally ask card to compute crc' can > > be found in both if_ixl: > > https://svnweb.freebsd.org/base/stable/11/sys/dev/ixl/ixl_txrx.c?view= > markup#l408 > > and in DPDK i40e driver: > > https://github.com/DPDK/dpdk/blob/master/drivers/net/i40e/i40e_rxtx.c# > L1105 > > so, I guess, it's safe. > > > > PS: of course, first idea was to upgrade to FreeBSD 12-stable, but while > > this upgrade solved the crc problem, this version shows 'stalled tx > queue' > > problem: after CTXSYNC tail == head == cur, 'no space in ring' condition. > > Attempts to ignore this condition led to continuous ring resets in > txcsync: > > > > Mar 17 20:21:08 host kernel: 668.224836 [1679] nm_txsync_prologue        > ixl1 TX3: fail 'head < kring->rhead || head > kring->rtail' h 136 c 136 t > 135 rh 135 rc 135 rt 135 hc 135 ht 135 > > Mar 17 20:21:08 host kernel: 668.238300 [1787] netmap_ring_reinit        > called for ixl1 TX3 > > > > PPS: hardware details: Dell VEP4600, based on Xeon D-2100 with > > two onboard X722 ports (actually, four, but two of them are not > > wired). > > > > CPU: Intel(R) Xeon(R) D-2187NT CPU @ 2.00GHz (2000.06-MHz K8-class CPU) > >   Origin="GenuineIntel"  Id=0x50654  Family=0x6  Model=0x55  Stepping=4 > > > > ixl0: 1.11.9-k> mem 0xfa000000-0xfaffffff,0xfb018000-0xfb01ffff irq 11 at device > 0.0 numa-domain 0 on pci12 > > ixl0: using 1024 tx descriptors and 1024 rx descriptors > > ixl0: fw 3.1.57069 api 1.5 nvm 3.33 etid 80001007 oem 1.263.0 > > ixl0: PF-ID[0]: VFs 32, MSIX 129, VF MSIX 5, QPs 384, I2C > > ixl0: Using MSIX interrupts with 9 vectors > > ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active > > ixl0: Ethernet address: 3c:2c:30:30:59:85 > > ixl0: SR-IOV ready > > ixl0: netmap queues/slots: TX 8/1024, RX 8/1024 > > > > ixl0@pci0:184:0:0:    class=0x020000 card=0x00008086 chip=0x37d38086 rev= > 0x04 hdr=0x00 > >     vendor     = 'Intel Corporation' > >     device     = 'Ethernet Connection X722 for 10GbE SFP+' > >     class      = network > >     subclass   = ethernet > > > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Thu Mar 26 15:50:19 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 14C792A2FC8; Thu, 26 Mar 2020 15:50:19 +0000 (UTC) (envelope-from freebsd@qeng-ho.org) Received: from bede.qeng-ho.org (bede.qeng-ho.org [217.155.128.241]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48p8ZB5kPLz4VHL; Thu, 26 Mar 2020 15:49:54 +0000 (UTC) (envelope-from freebsd@qeng-ho.org) Received: from arthur.home.qeng-ho.org (arthur.home.qeng-ho.org [172.23.1.2]) by bede.qeng-ho.org (Postfix) with ESMTP id C6DA210671; Thu, 26 Mar 2020 15:49:42 +0000 (GMT) Subject: Re: IPv6 in jails To: Victor Sudakov , "Bjoern A. Zeeb" Cc: freebsd-net@freebsd.org, Miroslav Lachman <000.fbsd@quip.cz>, freebsd-questions@freebsd.org References: <20200318151556.GA64871@admin.sibptus.ru> <2dd539ed-0ee3-079b-27b2-28126056c69a@quip.cz> <20200318155046.GD65497@admin.sibptus.ru> <4CA69535-0F6C-40FC-83CF-5000FD728C2D@lists.zabbadoz.net> <20200319021432.GA80800@admin.sibptus.ru> <01EF7656-4F8A-4075-A0B4-27E8AB17B516@lists.zabbadoz.net> <20200319140132.GA93947@admin.sibptus.ru> From: Arthur Chance Message-ID: Date: Thu, 26 Mar 2020 15:49:42 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.3.1 MIME-Version: 1.0 In-Reply-To: <20200319140132.GA93947@admin.sibptus.ru> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 48p8ZB5kPLz4VHL X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd@qeng-ho.org designates 217.155.128.241 as permitted sender) smtp.mailfrom=freebsd@qeng-ho.org X-Spamd-Result: default: False [-4.83 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.155.128.240/29]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; SURBL_MULTI_FAIL(0.00)[query timed out]; DMARC_NA(0.00)[qeng-ho.org]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13037, ipnet:217.155.0.0/16, country:GB]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-2.53)[ip: (-7.85), ipnet: 217.155.0.0/16(-3.92), asn: 13037(-0.79), country: GB(-0.07)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2020 15:50:19 -0000 On 19/03/2020 14:01, Victor Sudakov wrote: > Bjoern A. Zeeb wrote: >> >>>> If it does, can you add a >>>> >>>> exec.start += "sleep 2 "; >>>> >>>> to your config >>> >>> OK, I've added it to the configs of 3 experimental jails. >>> >>>> and see if your problem goes away? >>> >>> It goes away partially (only for sshd in 2 of the 3 available jails), >>> and >>> not for syslogd in any of the 3 available jails. Restarting the daemons >>> from within the jail fixes the problem. An example from a problem jail: >>> >> .. >>> >>>> If it does, the reason is >>>> that you configure an IPv6 address to an interface and DUD has not >>>> yet >>>> completed by the time sshd or other daemons start. Giving it the 2 > > What is "DUD" BTW? > I suspect it's a typo for DAD - duplicate address detection which is part of IPv6. >>>> seconds >>>> avoids this problem and the address is usable at that time. >>> >>> There is obviously a race somewhere, but the 2 second sleep does not >>> eliminate it entirely. >> >> Well not so much of a race but than a “gap”. >> >> The point is you are configuring an address on the base system and the jail >> knows nothing about it so it’ll simply start the daemons. Normally the >> startup scripts would do the right thing. >> >> I don’t think “polluting” jail(8) with logic to check that the addresses >> become available or not is a good idea. However I agree that it should >> automatically do the right thing somehow .. >> >> >> >>> Thank you for the hint in the right direction, what would you suggest >>> further? >> >> If you make it 3 seconds, does it deterministically work then? > > Not quite: https://termbin.com/arvb > syslogd sometimes remains deprived of the IPv6 address. > -- Violets are red And roses are blue When metamaterials Alter their hue. From owner-freebsd-net@freebsd.org Thu Mar 26 20:02:34 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9D8E826051F for ; Thu, 26 Mar 2020 20:02:34 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48pG9j2YGKz3Q1c for ; Thu, 26 Mar 2020 20:02:33 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from mail-qk1-f172.google.com (mail-qk1-f172.google.com [209.85.222.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: vmaffione) by smtp.freebsd.org (Postfix) with ESMTPSA id 64B642ACF3 for ; Thu, 26 Mar 2020 20:02:26 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: by mail-qk1-f172.google.com with SMTP id d11so8300423qko.3 for ; Thu, 26 Mar 2020 13:02:26 -0700 (PDT) X-Gm-Message-State: ANhLgQ16rTJWJRMerSROLs12PPebXkhpV1B5QvT6KwlpuZd7WYmgIoRs Ei+vvNsKuru3j7oIdvIRTgJa+ZYY0ThJz3xnQDQ= X-Google-Smtp-Source: ADFU+vsX0dnX5L9rW7lwV1V07txYuzHWhwQbwsgSP2gwigMmcrB0DE3Hwmj02vCa9J2kpuqHiKfXmaPKejLFBy2yAw0= X-Received: by 2002:a37:a1cc:: with SMTP id k195mr10418899qke.169.1585252945605; Thu, 26 Mar 2020 13:02:25 -0700 (PDT) MIME-Version: 1.0 References: <20200324123721.GA26248@staff.retn.net> <20200324140839.GB26248@staff.retn.net> <20200326102008.GA54860@staff.retn.net> In-Reply-To: <20200326102008.GA54860@staff.retn.net> From: Vincenzo Maffione Date: Thu, 26 Mar 2020 21:02:13 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: netmap/ixl and crc addition.. To: Alexandre Snarskii Cc: FreeBSD Net Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2020 20:02:34 -0000 Il giorno gio 26 mar 2020 alle ore 11:39 Alexandre Snarskii < snar@snar.spb.ru> ha scritto: > On Wed, Mar 25, 2020 at 11:31:30PM +0100, Vincenzo Maffione wrote: > > Hi Alexandre, > > Thanks. Your patch looks good to me. I assume you have tested it? > > Sure, this patch is already handles production traffic. > > > I will commit that to stable/11. > > Thanks. > > > > > The issue you report on stable/12 is more worrisome. The 'no space in TX > ring' > > condition (head==cur==tail) is ok per-se: on a subsequent poll() wakeup > (e.g. > > TX interrupt) or explicit ioctl(NIOCTXSYNC) you should see tail moving > forward, > > therefore freeing some space to be used in the ring. > > Unfortunately, further NIOCTXSYNC did not freed queue. > > Let me explain my observations in details: this issue was reproduceable > even under really light load (say, hundreds pps per queue). As the load > was ligth, I was able to insert debugging output before and after any > calls > to ioctl and poll, code is like: > > while(1) { > if (nm_tx_pending(txring)) { > printf("%s: pre-txsync cur: %d, head: %d, tail: %d\n", > thread_name, txring->cur, txring->head, > txring->tail); > if (ioctl(pa->fd, NIOCTXSYNC) == -1) { > log event > }; > printf("%s: post-txsync cur: %d, haad: %d, tail: %d\n", > thread_name, txring->cur, txring->head, > txring->tail); > }; > // setup poll descriptors with POLLIN only > printf("%s: pre-poll cur: %d, head: %d, tail: %d\n", > thread_name, txring->cur, txring->head, txring->tail); > ret = poll(fds, 2, 1000); > printf("%s: post-poll cur: %d, head: %d, tail: %d\n", > thread_name, txring->cur, txring->head, txring->tail); > > Ok, but I have some preliminary questions and notes: - You show the code that syncs the tx ring to collect completed transmissions (and thus advance tail). However, I do not see the code that submits new packets to be transmitted (and thus advances head/tail). Where is it? If this is happening in a separate thread, that leads to undefined behaviour, e.g. ring resets (although you can't crash the kernel). > Logs in normal situation (only single thread shown): > > Notice:nz(ixl0:2): pre-txsync cur: 649, head: 649, tail: 647 > Notice:nz(ixl0:2): post-txsync cur: 649, head: 649, tail: 648 > Notice:nz(ixl0:2): pre-poll cur: 649, head: 649, tail: 648 > Notice:nz(ixl0:2): post-poll cur: 649, head: 649, tail: 648 > > in pre-txsync: cur = head = tail + 2 (sometimes tail + 3), queue > is nearly empty. > in post-txsync: cur = head = tail + 1, queue is flushed. > > however, after some time, txcsync call advanced tail not to head - 1, > but to head: > > Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 652 > Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654 > Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654 > Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654 > I see. This is wrong and must not happen. Once you get here, the tx ring becomes unusable, so we may as well ignore what happens afterwards. > > and following txcsyncs was not able to free (already empty) queue: > > Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654 > Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654 > Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654 > Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654 > Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654 > Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654 > Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654 > Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654 > Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654 > Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654 > > application entered busy wait.. (in my case most packets just zero-copied > from rxring to txring, so if there are no space in txring, packets are not > consumed from rx). > > > However, the ring_reinit means that something is going wrong: either your > > application is using the TX ring incorrectly, or there is a bug in the > netmap > > ring_reinit was indeed caused by my incorrect use of txring: I > tried to ignore situation head == tail and inject packet anyway.. > Yes, ring reinit actually resets the ring variables to a sane value. In case your issue is not caused by unsafe access to the tx ring from multiple threads (see above), it would help to know what value do you have in sysctl hw.ixl.enable_head_writeback, and also try if the problem is still there if you enable or disable that parameter (which is relevant for how tail is advanced). > > > iflib code. Since FreeBSD 12, netmap support is provided by iflib, while > before > > netmap support was provided directly by the ixl driver. > > In any case, it would probably help if you could provide some more > detailed > > info (how to reproduce the problem). > > > > Cheers, > > Vincenzo > > > > From owner-freebsd-net@freebsd.org Fri Mar 27 03:13:02 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6220F268C7B for ; Fri, 27 Mar 2020 03:13:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48pRk83v9Xz4dFV for ; Fri, 27 Mar 2020 03:12:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 6A6EE268A9E; Fri, 27 Mar 2020 03:05:33 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6A166268A9D for ; Fri, 27 Mar 2020 03:05:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48pRYl0LY6z4Zvt for ; Fri, 27 Mar 2020 03:05:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 985AA19A67 for ; Fri, 27 Mar 2020 02:55:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 02R2tioW007185 for ; Fri, 27 Mar 2020 02:55:44 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 02R2tis5007184 for net@FreeBSD.org; Fri, 27 Mar 2020 02:55:44 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 243463] ix0: Watchdog timeout Date: Fri, 27 Mar 2020 02:55:43 +0000 X-Bugzilla-Reason: CC AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.1-RELEASE X-Bugzilla-Keywords: IntelNetworking, needs-qa X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: denis@h3q.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: mfc-stable12? mfc-stable11? X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2020 03:13:02 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D243463 Denis Ahrens changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |denis@h3q.com --- Comment #6 from Denis Ahrens --- looks like 235524 for me. the igb interface will not survive iperf3 -t 300 for me. --=20 You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Fri Mar 27 07:49:06 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BF78F26D33E for ; Fri, 27 Mar 2020 07:49:06 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48pYrp5BDnz4Mdf for ; Fri, 27 Mar 2020 07:48:58 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: by mailman.nyi.freebsd.org (Postfix) id 5CCEA26D339; Fri, 27 Mar 2020 07:48:50 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5410926D336; Fri, 27 Mar 2020 07:48:50 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from forward103o.mail.yandex.net (forward103o.mail.yandex.net [37.140.190.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48pYrB1GLKz4MTh; Fri, 27 Mar 2020 07:48:24 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from mxback20o.mail.yandex.net (mxback20o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::71]) by forward103o.mail.yandex.net (Yandex) with ESMTP id 77ACF5F8017A; Fri, 27 Mar 2020 10:48:14 +0300 (MSK) Received: from localhost (localhost [::1]) by mxback20o.mail.yandex.net (mxback/Yandex) with ESMTP id lqfV92whFj-mD6GbFKv; Fri, 27 Mar 2020 10:48:13 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfw.ru; s=mail; t=1585295293; bh=7KDc+mlqaxPiIXhVAu1rq5V1k7I9L0IXpdzKTg2gg/Q=; h=Message-Id:Date:Subject:To:From; b=B8Dn/07UmJmzRB0gqrvatkHjqgDMcXJJZWqG935HqA4EmMKW4DZBgezg6ECPC6TPU S71B8e0nHiJ83z7oC8OelBPhTMOBhktPVc5XHm0IbA4KJGtS61fb0g2O1zYU2RC0ql ojDZ34uWyMecL531X8pxMe7GH6fNoV+AiOvrkAgw= Received: by myt6-4218ece6190d.qloud-c.yandex.net with HTTP; Fri, 27 Mar 2020 10:48:13 +0300 From: Alexander V. Chernikov To: "current@FreeBSD.org" , net Subject: CFT: Next-hop objects and scalable multipath routing MIME-Version: 1.0 X-Mailer: Yamail [ http://yandex.ru ] 5.0 Date: Fri, 27 Mar 2020 07:48:13 +0000 Message-Id: <1607511585294876@vla1-b2d94eaf2344.qloud-c.yandex.net> Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Rspamd-Queue-Id: 48pYrB1GLKz4MTh X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ipfw.ru header.s=mail header.b=B8Dn/07U; dmarc=none; spf=pass (mx1.freebsd.org: domain of melifaro@ipfw.ru designates 37.140.190.177 as permitted sender) smtp.mailfrom=melifaro@ipfw.ru X-Spamd-Result: default: False [-6.30 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[ipfw.ru:s=mail]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:37.140.128.0/18]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[ipfw.ru]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; IP_SCORE(-3.70)[ip: (-9.79), ipnet: 37.140.128.0/18(-4.89), asn: 13238(-3.85), country: RU(0.01)]; DKIM_TRACE(0.00)[ipfw.ru:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_LOW(-0.10)[177.190.140.37.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13238, ipnet:37.140.128.0/18, country:RU]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2020 07:49:06 -0000 I would like to introduce an implementation of scalable multipath routing. Previous implementation (RADIX_MPATH) focused on a simpler case like having 2 defaults, with performance falling linearly proportional to the number of paths. That implementation was also tightly coupled lookup algorithm details with the routing details, making it hard to hack both. The proposed one allows O(1) lookup and is more cache-efficient with the large amount of routes. Furthermore, multipath functionality is based on the number of internal changes, modernizing the old routing code. Most of the changes revolves around introducing the concept of _nexthops_. Nexthops are separate datastructures, containing all necessary information to perform packet forwarding such as gateway, interface and mtu. Nexthops are shared among the routes, providing more pre-computed cache-efficient data while requiring less memory. Multipath implementation adds _nexthop groups_ which are basically collection of nexthops weights, compiled into an array, to allow direct nexthop selection. More detailed technical description is available at [1]. Any comments/suggestions are welcome! Presentation of the similar functionality in the other OS: [2] Next-hop objects support was implemented in FRR in 2019 [3]. Next steps: As these changes decouples routing code details from algorithm details and abstracts callers, it is much easier to introduce a number of other relevant features. The most important proposed features are: nexthop-based route installation and custom per-address-family route lookup algorithms. The former targets improving convergence times for the large-fib boxes, while the latter may improve dataplane performance, especially for IPv6. How to test: fetch the patch from https://reviews.freebsd.org/D24141 rebuild kernel with ROUTE_MPATH option (already added to amd64 GENERIC) Optionally, rebuild world to get netstat nexthops/multipath groups reporting. Use route(8) to add multiple routes for the same destination, optionally specifying weight. Example: add 2:1 load balancing for the default route: route add -net default 192.168.53.1 -weight 100 route add -net default 192.168.53.2 -weight 200 netstat -4rnW .. Destination Gateway Flags Nhop# Mtu Netif Expire default 192.168.53.1 UGS 4 1500 em0 default 192.168.53.2 UGS 5 1500 em0 netstat -4onW Nexthop data Idx Type IFA Gateway Flags Use Mtu Netif Addrif Refcnt Prepend .. 4 v4/gw 192.168.53.128 192.168.53.1 GS 0 1500 em0 2 5 v4/gw 192.168.53.128 192.168.53.2 GS 0 1500 em0 1 Nexthop groups data MpIdx NHIdx Weigh Slots Gateway Netif Refcnt 1 ---- ---- ---- ---- ---- 1 4 100 1 192.168.53.1 em0 5 200 2 192.168.53.2 em0 [1] https://reviews.freebsd.org/D24141 [2] https://linuxplumbersconf.org/event/4/contributions/434/attachments/251/436/nexthop-objects-talk.pdf [3] https://github.com/FRRouting/frr/commit/d9f5b2f50f53d625986dbd47cd12778c9f841f0c From owner-freebsd-net@freebsd.org Fri Mar 27 09:35:44 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8ABFA26F94B for ; Fri, 27 Mar 2020 09:35:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48pcCy0Q9kz430r for ; Fri, 27 Mar 2020 09:35:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id E0D4E26F94A; Fri, 27 Mar 2020 09:35:33 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D946F26F949 for ; Fri, 27 Mar 2020 09:35:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48pcCl2RHtz42xD for ; Fri, 27 Mar 2020 09:35:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A2C001E71D for ; Fri, 27 Mar 2020 09:19:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 02R9J062011139 for ; Fri, 27 Mar 2020 09:19:00 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 02R9J0Mv011127 for net@FreeBSD.org; Fri, 27 Mar 2020 09:19:00 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 243463] ix0: Watchdog timeout Date: Fri, 27 Mar 2020 09:19:00 +0000 X-Bugzilla-Reason: AssignedTo CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.1-RELEASE X-Bugzilla-Keywords: IntelNetworking, needs-qa X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: silence@cnw.cz X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: mfc-stable12? mfc-stable11? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2020 09:35:44 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D243463 --- Comment #7 from Jiri --- Yes, agree. I have added the second X520 card, there is very low traffic (a= bout some MBits) and no problem observed here. This timeout behavior probably depends on traffic. I.E. high traffic =3D problem. --=20 You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.= From owner-freebsd-net@freebsd.org Fri Mar 27 11:37:58 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 11E712725C8 for ; Fri, 27 Mar 2020 11:37:58 +0000 (UTC) (envelope-from fernando@gont.com.ar) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48pfwy73Jsz3L1s for ; Fri, 27 Mar 2020 11:37:54 +0000 (UTC) (envelope-from fernando@gont.com.ar) Received: by mailman.nyi.freebsd.org (Postfix) id 6C33B2725C6; Fri, 27 Mar 2020 11:37:47 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6BA8C2725C5 for ; Fri, 27 Mar 2020 11:37:47 +0000 (UTC) (envelope-from fernando@gont.com.ar) Received: from fgont.go6lab.si (fgont.go6lab.si [91.239.96.14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48pfwV0Kcpz3Ktp; Fri, 27 Mar 2020 11:37:26 +0000 (UTC) (envelope-from fernando@gont.com.ar) Received: from [192.168.0.10] (unknown [181.45.84.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by fgont.go6lab.si (Postfix) with ESMTPSA id BCE9C83215; Fri, 27 Mar 2020 12:37:12 +0100 (CET) To: net@freebsd.org Cc: Hiroki Sato From: Fernando Gont Subject: PATCH: More appropriate vltime for IPv6 temp addrs (rfc4941bis) Message-ID: <975e56bb-a556-a1d7-937d-f08289c03911@gont.com.ar> Date: Fri, 27 Mar 2020 08:36:48 -0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 48pfwV0Kcpz3Ktp X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of fernando@gont.com.ar designates 91.239.96.14 as permitted sender) smtp.mailfrom=fernando@gont.com.ar X-Spamd-Result: default: False [-5.76 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[gont.com.ar]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-3.46)[ip: (-9.18), ipnet: 91.239.96.0/23(-4.59), asn: 198644(-3.54), country: SI(0.01)]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[14.96.239.91.list.dnswl.org : 127.0.10.0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:198644, ipnet:91.239.96.0/23, country:SI]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2020 11:37:59 -0000 Folks, This changes the default Valid Lifetime of IPv6 temporary addresses, as per draft-ietf-6man-rfc4941bis: ---- cut here ---- diff --git nd6.h nd6.h index 857657f6e20..5254224afc0 100644 --- nd6.h +++ nd6.h @@ -185,7 +185,7 @@ struct in6_ndifreq { #define RETRANS_TIMER 1000 /* msec */ #define MIN_RANDOM_FACTOR 512 /* 1024 * 0.5 */ #define MAX_RANDOM_FACTOR 1536 /* 1024 * 1.5 */ -#define DEF_TEMP_VALID_LIFETIME 604800 /* 1 week */ +#define DEF_TEMP_VALID_LIFETIME 172800 /* 2 days (draft-ietf-6man-rfc4941bis) */ #define DEF_TEMP_PREFERRED_LIFETIME 86400 /* 1 day */ #define TEMPADDR_REGEN_ADVANCE 5 /* sec */ #define MAX_TEMP_DESYNC_FACTOR 600 /* 10 min */ ---- cut here ---- P.S.: Patch also available at: https://www.gont.com.ar/code/patch-freebsd-ipv6-tempaddr-vltime.txt Thanks, -- Fernando Gont e-mail: fernando@gont.com.ar || fgont@si6networks.com PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1 From owner-freebsd-net@freebsd.org Sat Mar 28 22:03:27 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CF135267F77; Sat, 28 Mar 2020 22:03:27 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 48qXlx0lYcz4HQh; Sat, 28 Mar 2020 22:03:07 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id 02SLi80l017536 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 28 Mar 2020 21:44:12 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: abrancatelli@schema31.it Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id 02SLi686038651 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Sun, 29 Mar 2020 04:44:06 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: BSNMPD crashing To: Andrea Brancatelli , freebsd-stable@freebsd.org, "freebsd-net@freebsd.org" References: <3618d2f5680d312235ee77fb0c570ae2@schema31.it> From: Eugene Grosbein Message-ID: <597093de-b842-5396-e853-a97f1e693556@grosbein.net> Date: Sun, 29 Mar 2020 04:43:56 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <3618d2f5680d312235ee77fb0c570ae2@schema31.it> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 48qXlx0lYcz4HQh X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.96 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; IP_SCORE(-1.86)[ip: (-5.15), ipnet: 2a01:4f8::/29(-2.59), asn: 24940(-1.56), country: DE(-0.02)]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; RBL_SPAMHAUS_FAIL(0.00)[2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.d.6.2.c.2.c.0.8.f.4.0.1.0.a.2.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net:query timed out]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2020 22:03:28 -0000 29.03.2020 1:53, Andrea Brancatelli via freebsd-stable wrote: > If anyone is interested please feel free to contact me otherwise I'll > just delete it :-) What is most recent version of FreeBSD that has this problem with bsnmpd crashing? Please rebuild bsnmpd with debugging enabled, if you have sources installed: cd /usr/src/usr.sbin/bsnmpd make clean make cleandir make obj depend make WITH_DEBUG=yes install /usr/obj/usr/local/src/usr.sbin/bsnmpd/bsnmpd/bsnmpd /usr/sbin Restart bsnmpd and wait for next crash. Then you could obtain meaningfull backtrace. From owner-freebsd-net@freebsd.org Sat Mar 28 22:52:21 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1714D269815 for ; Sat, 28 Mar 2020 22:52:21 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qv1-xf43.google.com (mail-qv1-xf43.google.com [IPv6:2607:f8b0:4864:20::f43]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48qYrV3Nfzz4bFq; Sat, 28 Mar 2020 22:52:09 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qv1-xf43.google.com with SMTP id bu9so959792qvb.13; Sat, 28 Mar 2020 15:52:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=g+0aXElG6sVTMSAizXABgrSw3aAa04LoMo3XTdxHeZU=; b=aoyDArWkM8SZ8xXa3VKYAZ/WQ/njY9x4ixTxPq7AfiOWkCP9MoXKk5sTsNRFrMY9ir Ov7dMslJ+ruvbUAjYnSPUrI8F7iktGLPC1IGPSg3+KkylcRAZ+9NCRuOXrugOdezEgKM 1xvX+SZjM3b072LDplLHjA+qbI3rvhGqbENPhE3obSfpZ9toc3KJmb5+2vYze3UDpFyU oJ3LHHSSPyrMmrKsgBZUIYyAT7mFCJBEwLotiU48SqDNY6CBvxJv6fvLY3EZgzb6RSIA yLPFO4CnnedN9Vz76HL7T72f4dR8HwMZU6Cj6qXSS0Ffep50DC7ksjb7IFi3KRu0LPFA bejA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to; bh=g+0aXElG6sVTMSAizXABgrSw3aAa04LoMo3XTdxHeZU=; b=ExN62N0FHw+tsOIwpZWo8f/vFVrQ05HV2YDH6JJRqwyDTJW6iP2FH2mu9uQT6hjRxu HbOjdzuWfzcbFcAEJdcBhiyFqYujGtufvg31fZTqD+a//WuvbzkD6o+ChdWwSmZnCjGs 60tMBYa3DPL+7QMJLsAlJK1kf4tgMJDJUKwP5rDdZANC7IbJuSHiU1wEEpnzDNAKwA/8 vYm5u/rXfzrkJ5BWfy6RXLP0iB8+jY8Hz0FeOF7btx+k6EMJ/pPrGsEhUXP5UEhkmTzB aGjy3TDT7BQAblhmcQza2SAH3+bydvIuVLzyahxpOvI6Rai0jUqhRrXx65ZZIEG6s1Nw mNGg== X-Gm-Message-State: ANhLgQ3GoaiA2nrF9ew4JmHBVRsY+Ku0rkMR0/3mTTQxD3As+1v0qu3e PtKWWz0ZkZ3oyX6FVknV2sUvH6HOIi0= X-Google-Smtp-Source: ADFU+vsmr0mOLVlCjgEVr+sI936LbsFVm+eA5wpBOxC/+D1yi9NQcff9bo1fU1Sn2aTy+KIW6IRvcA== X-Received: by 2002:ad4:556e:: with SMTP id w14mr5758482qvy.144.1585435923133; Sat, 28 Mar 2020 15:52:03 -0700 (PDT) Received: from raichu (toroon0560w-lp130-10-174-94-17-182.dsl.bell.ca. [174.94.17.182]) by smtp.gmail.com with ESMTPSA id j20sm6973916qke.44.2020.03.28.15.52.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Mar 2020 15:52:02 -0700 (PDT) Sender: Mark Johnston Date: Sat, 28 Mar 2020 18:51:57 -0400 From: Mark Johnston To: Eric Joyner Cc: freebsd-net@freebsd.org, Hans Petter Selasky , John Baldwin , shurd , Drew Gallatin , Gleb Smirnoff Subject: Re: Issue with epoch_drain_callbacks and unloading iavf(4) [using iflib] Message-ID: <20200328225150.GA82767@raichu> References: <0e2e97f2-df75-3c6f-9bdd-e8c2ab7bf79e@selasky.org> <20200130030911.GA15281@spy> <20200212222219.GE83892@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 48qYrV3Nfzz4bFq X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=aoyDArWk; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::f43 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-1.87 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; RBL_SENDERSCORE_FAIL(0.00)[3.4.f.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.bl.score.senderscore.com:query timed out]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_IN_DNSWL_NONE(0.00)[3.4.f.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RBL_BLOCKLISTDE_FAIL(0.00)[3.4.f.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.bl.blocklist.de:query timed out]; IP_SCORE(-0.17)[ip: (0.01), ipnet: 2607:f8b0::/32(-0.36), asn: 15169(-0.46), country: US(-0.05)]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MIME_TRACE(0.00)[0:+]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2020 22:52:21 -0000 On Wed, Mar 11, 2020 at 04:32:40PM -0700, Eric Joyner wrote: > Mark, > > I did get some time to get back and retry this; however your second patch > still doesn't solve the problem. Looking into it a bit, it looks like the > kldunload process isn't hitting the code you've changed; it's hanging in > epoch_wait_preempt() in if_detach_internal(), which is immediately before > epoch_drain_callbacks(). > > I did a kernel dump while it was hanging, and this is the backtrace for the > kldunload process: I see. I think the callback can be made much simpler and avoid the problematic sched_bind() calls. I wrote a patch that allows waiting threads to lend scheduling priority to a preempted thread blocked in an epoch section, based on some code I wrote to implement preemptible SMR sections. If waiting for a running thread, the callback just spins. This might be enough to solve your problem, I posted the two lightly tested patches here: https://reviews.freebsd.org/D24214 https://reviews.freebsd.org/D24215 If we hit a situation where a reader is preempted and then its CPU is hogged by a high-priority kernel thread, this still won't be enough, but I suspect it'll solve your case. Would you be able to test?