From owner-freebsd-net@freebsd.org Tue Sep 29 01:11:09 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 15F2F42FAC2; Tue, 29 Sep 2020 01:11:09 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C0hCv4mhBz44hZ; Tue, 29 Sep 2020 01:11:07 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: by mail-wm1-f51.google.com with SMTP id s13so2945375wmh.4; Mon, 28 Sep 2020 18:11:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=JPCd8GkETgUfb7bzXVX45HGFbIDbSKt4xbmir/6Omj0=; b=ip4p/ifmduUij+Lg5bTHq0RJ4M4YMxUzyuLYsWrSFkdRIy3fd4FHD+wQeB6Ikz8Ft4 oCyOC1XG2z+kTy+qvBxcdpC0DjOj+TZE77ZjdmVBfQ0bUZ8CGUXOW4I4JLAXt6Gl1SH/ SPGhLWCbQt55zOLgEp+MJ48No2P+ugSbboJd86aGhfSFIVflnaZs3ad4jsm3NI+b7HfH gYiMqXQIB6iRNJ1mAlPNlipLhCDD3GM+xSh+GkDaP668CT6J2K7Tfs6+Pw2GRFsUQkJ0 LQZYUSwWy0GR0yjGJSBag1YFuKqe9H6wNVyiKEbY2U5AU8vX50814JJmkvB8VTE7jDOr a5dQ== X-Gm-Message-State: AOAM53077FADZH+yEyLV61MHWFefg+pdWXJEJGolReSVxNCxK7x85AXi I0t5Nhu8ZtXYEqadbRCFleWqcIN5qpNlAGL2Hduk8o5O X-Google-Smtp-Source: ABdhPJwYuhqZjetXi9NCwQO0rO9TqUCxzZrsDcZ2L+evR1QjkY63VOsnSkSNGdU6gtmUDb0ZhLi+ReRD6Kz2ZJ7ZMug= X-Received: by 2002:a1c:7918:: with SMTP id l24mr1791616wme.46.1601341865990; Mon, 28 Sep 2020 18:11:05 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Jason Tubnor Date: Tue, 29 Sep 2020 11:10:54 +1000 Message-ID: Subject: Re: Fragmentation of em(4) traffic (potentially others) when using vlanhwtag 12.2-BETA1 To: vmaffione@freebsd.org, mmacy@freebsd.org, freebsd-net@freebsd.org, erj@freebsd.org, freebsd-stable , re@freebsd.org, Peter Grehan X-Rspamd-Queue-Id: 4C0hCv4mhBz44hZ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of jtubnor@gmail.com designates 209.85.128.51 as permitted sender) smtp.mailfrom=jtubnor@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-0.94)[-0.945]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[tubnor.net]; NEURAL_HAM_LONG(-0.95)[-0.947]; SH_EMAIL_DBL_DONT_QUERY_IPS(0.00)[0.0.0.0:email,0.0.5.200:email]; DBL_PROHIBIT(0.00)[0.0.0.0:email,0.0.5.200:email]; NEURAL_HAM_SHORT(-1.10)[-1.104]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_IN_DNSWL_NONE(0.00)[209.85.128.51:from]; FORGED_SENDER(0.30)[jason@tubnor.net,jtubnor@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.128.51:from]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_NEQ_ENVFROM(0.00)[jason@tubnor.net,jtubnor@gmail.com]; MAILMAN_DEST(0.00)[freebsd-net,freebsd-stable] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2020 01:11:09 -0000 On Fri, 18 Sep 2020 at 19:52, Jason Tubnor wrote: > ---------- Forwarded message --------- > From: Jason Tubnor > Date: Thu, 17 Sep 2020 at 10:49 > Subject: Fragmentation of em(4) traffic (potentially others) when using > vlanhwtag 12.2-BETA1 > To: freebsd-stable > > > Hi, > > Doing some edge case testing with 12.2-BETA1, I have come across > fragmentation issues when vlanhwtag is enabled on the em(4) network device > (device listed below), causing IPSec traffic moving through a bhyve > 'router-on-a-stick' to tank on send. When you disable (-vlanhwtag), this > issue goes away and full speed send is possible. > > Here is a snip of the traffic leaving for the internet after traversing > several VLANs through a bhyve router with vlanhwtag enabled: > > 10:09:54.173412 Dest.443 > Source.60901: . ack 12384 win 492 2 {10944:12235} {15373:18253} > (DF) > 10:09:54.173936 Source.60901 > Dest.443: . 38413:39873(1460) ack 3837 win > 1027 (frag 20228:1480@0+) > 10:09:54.173937 Source > Dest: (frag 20228:1420@1480) > > pciconf -vlbc > > em0@pci0:0:25:0: class=0x020000 card=0x308617aa chip=0x15028086 > rev=0x04 hdr=0x00 > vendor = 'Intel Corporation' > device = '82579LM Gigabit Network Connection (Lewisville)' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xf7d00000, size 131072, > enabled > bar [14] = type Memory, range 32, base 0xf7d39000, size 4096, enabled > bar [18] = type I/O Port, range 32, base 0xf080, size 32, enabled > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message > cap 13[e0] = PCI Advanced Features: FLR TP > > > Further investigation with help from grehan@ this is a LRO issue and not a vlanhwtag issue. When a vlan interface is added to a bridge, LRO isn't automatically disabled. LRO behaviour changed between 11 and 12 and this is why we haven't seen this issue on our 11.x devices where bridges are extensively used with bhyve/tap. Can this be addressed prior to the 12.2 release please. Happy to test any patches. Thanks, Jason.