From owner-freebsd-net@freebsd.org Mon Oct 23 07:39:36 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9DB1E43875 for ; Mon, 23 Oct 2017 07:39:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B7F6F65D58 for ; Mon, 23 Oct 2017 07:39:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v9N7dZL3035902 for ; Mon, 23 Oct 2017 07:39:36 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 221122] Attaching interface to a bridge stops all traffic on uplink NIC for few seconds Date: Mon, 23 Oct 2017 07:39:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Not A Bug X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Oct 2017 07:39:36 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221122 --- Comment #17 from commit-hook@freebsd.org --- A commit references this bug: Author: mav Date: Mon Oct 23 07:39:05 UTC 2017 New revision: 324908 URL: https://svnweb.freebsd.org/changeset/base/324908 Log: MFC r324659: Update details of interface capabilities changed by bridge(4= ). PR: 221122 Changes: _U stable/11/ stable/11/share/man/man4/bridge.4 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Mon Oct 23 19:41:49 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9477E537A4 for ; Mon, 23 Oct 2017 19:41:49 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wm0-x22e.google.com (mail-wm0-x22e.google.com [IPv6:2a00:1450:400c:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B18481A00; Mon, 23 Oct 2017 19:41:49 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-wm0-x22e.google.com with SMTP id b189so11542393wmd.4; Mon, 23 Oct 2017 12:41:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=6hIWuQwW4ey5iWn3SCoeUlZDt88BNooWjzNH41rDtzA=; b=KrWD2Y62KSWLM91yrh8Dl4XNgHlNYR5SoSAMb+Z+9gLMN7oHZRjewABHpxXjYLDu+7 lcrtgsYZ6PkGNzBfJOgwI1BGRY6ggHIptb02e+GICBlpcDWfGhVjYCAkWOKi/oeanejn J6AI80DWzYWf6uVQPJqw5OpMeYSIgZZlfs+M63YYfrRp02pU8EAZaMfz4Jxx2U2gyH/g p/cu3Wv8IwvXKg2QD7RuxLm/llfCdDLVHaMOU4cFfIBRP7CbNvUIzfG41YsS4zLJaH/o udgL9RmXgqTay9M/QY2mC8ohhfHfC7o5Sys0o7F5aSBeDzJ0kszLjcsQyk6bvYh1EbgA 2oqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=6hIWuQwW4ey5iWn3SCoeUlZDt88BNooWjzNH41rDtzA=; b=eh+rg2CsrTz3A11+jDMstiSu3cxM8YIbZuZibmP7iv8Go91Gljwk8xIN/XUb4NsALN l4Ee4Bzq7+Q1Add3IkIkgrKGiBjIkS/hDjcCbWqdFy8uFjhdjdPUm+0Rev/QyWcnj/WJ jdzvGfGRhUK6POSBsEGXhHlKsbBLy+vjkPZMdDlSQFmgZeYYbM3SkZN6lo9xbah94MJ1 tYjeB8u1PtU1zE85zC9MKsxattL7n9ky981yM3zTvUiloy4LQvXQVM4H1RgTdeuRwKgh AZXL4ToPWgIJ/1GSQZjoCEtmftbf79R6yPcQcnx4yyuasHEqVntz+wtRW1KAdqG9XFs8 8y7Q== X-Gm-Message-State: AMCzsaW5o/9D/DIVorpwu3tH/+eLspB+Tt2GMBtVYCJw6EvhMiNh5v1l nu6Rx5UR9HPVw8p5qD/SJsVKcah94BwuXeVRPTv75A== X-Google-Smtp-Source: ABhQp+Sw/H6RfC5tn3hDQaenfI7EDvVKWbXFS3yH+k1vLqOnCFwWu4eVoUqwevgH/6MC5qsCUZWb6v4YLUwe2aZ7BLs= X-Received: by 10.28.175.73 with SMTP id y70mr5997706wme.21.1508787707595; Mon, 23 Oct 2017 12:41:47 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.86.70 with HTTP; Mon, 23 Oct 2017 12:41:46 -0700 (PDT) In-Reply-To: <20171016175738.GA1100@FreeBSD.org> References: <59567148.1020902@xiplink.com> <31535133-f95a-5db6-a04c-acc0175fa287@yandex.ru> <59DFD3CC.2000401@xiplink.com> <20171013211026.GB1055@FreeBSD.org> <59E4C40E.9060103@xiplink.com> <20171016175738.GA1100@FreeBSD.org> From: Adrian Chadd Date: Mon, 23 Oct 2017 12:41:46 -0700 Message-ID: Subject: Re: m_move_pkthdr leaves m_nextpkt 'dangling' To: Gleb Smirnoff Cc: Karim Fodil-Lemelin , FreeBSD Net , "Andrey V. Elsukov" Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Oct 2017 19:41:49 -0000 On 16 October 2017 at 10:57, Gleb Smirnoff wrote: > Karim, > > On Mon, Oct 16, 2017 at 10:37:02AM -0400, Karim Fodil-Lemelin wrote: > K> > Not only mbufs of M_PKTHDR may have m_nextpkt set. However, I tend to agree > K> > with the patch. But shouldn't we first copy the m_nextpkt to the new mbuf: > K> > > K> > + to->m_nextpkt = from->m_nextpkt; > K> > + from->m_nextpkt = NULL; > K> > > K> > Same way as we deal with tags. > K> > > K> > > K> > K> I think you are correct. If we look at the 'spirit' of m_move_pkthdr(); > K> In my mind, it is to deep copy all fields related to a packet header and > K> since m_nextpkt should only be carried by packet headers, it makes sense > K> to copy it within m_move_pkthdr(). > K> > K> This also raises the question (my apologies in advance from bringing > K> this up...) of weather or not m_nextpkt belongs in struct m_hdr and not > K> in struct pkthdr. > K> > K> In our case we are copying it explicitly outside the function as most of > K> users of m_move_pkthdr() do. > > Moving m_nextpkt from m_hdr to m_pkthdr would be much more intrusive > change, we can't handle that. > > I think an mbuf with m_nextpkt and no M_PKTRHDR is a valid one. In > a datagram socket buffer that could hold a record. (didn't check that, > just guessing). > > So, any objections on commiting this addition to m_move_pkthdr? > > + to->m_nextpkt = from->m_nextpkt; > + from->m_nextpkt = NULL; None from me. (I haven't checked to see if you've done it yet or not.) -adrian > -- > Gleb Smirnoff From owner-freebsd-net@freebsd.org Tue Oct 24 15:33:11 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13D27E50E13 for ; Tue, 24 Oct 2017 15:33:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A73B164701 for ; Tue, 24 Oct 2017 15:33:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v9OFX6xC005367 for ; Tue, 24 Oct 2017 15:33:07 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 205706] Watchdog timeout on em driver under heavy traffic on a bridge configuration Date: Tue, 24 Oct 2017 15:33:07 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: enigma.algo@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Oct 2017 15:33:11 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D205706 Peter changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |enigma.algo@gmail.com --- Comment #8 from Peter --- Same issue for me with the em0 driver. Kernel Version 11.0-RELEASE-p12 In the setup are two OpnSense firewalls (COTS PCs) back-2-back with an IPsec tunnel between. The error shows up immediately when traffic is generated on= the LAN side(s) at the same time. I feed 64Byte Ethernet frames with an IXIA te= st generator. Packet rate is 1448000 pps. I believe its always the WAN side (on both PCs), the em driver hangs with following error: em0: Watchdog timeout Queue[0]-- resetting Interface is RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw tdh =3D 818, hw tdt =3D 777 em0: Tx Queue Status =3D -2147483648 em0: TX descriptors avail =3D 40 em0: Tx Descriptors avail failure =3D 17012 em0: RX Queue 0 ------ em0: hw rdh =3D 139, hw rdt =3D 139 em0: RX discarded packets =3D 0 em0: RX Next to Check =3D 140 em0: RX Next to Refresh =3D 139 Nothing will cure this problem unless I reboot. I also tried to play with m= bufs but without success. pciconf -lvbc em0@pci0:1:0:0: class=3D0x020000 card=3D0xa01f8086 chip=3D0x10d38086 rev=3D= 0x00 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D '82574L Gigabit Network Connection' class =3D network subclass =3D ethernet bar [10] =3D type Memory, range 32, base 0xf7ec0000, size 131072, enabled bar [14] =3D type Memory, range 32, base 0xf7e00000, size 524288, enabled bar [18] =3D type I/O Port, range 32, base 0xe000, size 32, enabled bar [1c] =3D type Memory, range 32, base 0xf7ee0000, size 16384, enabled cap 01[c8] =3D powerspec 2 supports D0 D3 current D0 cap 05[d0] =3D MSI supports 1 message, 64 bit=20 cap 10[e0] =3D PCI-Express 1 endpoint max data 256(256) NS link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) cap 11[a0] =3D MSI-X supports 5 messages, enabled Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] ecap 0001[100] =3D AER 1 0 fatal 0 non-fatal 1 corrected ecap 0003[140] =3D Serial 1 6805caffff5fda26=20 Cheers, Peter --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Wed Oct 25 05:48:09 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DEB1CE3F965 for ; Wed, 25 Oct 2017 05:48:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CD2B28253C for ; Wed, 25 Oct 2017 05:48:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v9P5m8Ue031583 for ; Wed, 25 Oct 2017 05:48:09 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 221990] panic: Assertion reclaimable == delta failed at ../../../net/iflib.c:1947 Date: Wed, 25 Oct 2017 05:48:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Oct 2017 05:48:10 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221990 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |patch Assignee|freebsd-bugs@FreeBSD.org |freebsd-net@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Wed Oct 25 21:14:28 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 69ACFE548F8 for ; Wed, 25 Oct 2017 21:14:28 +0000 (UTC) (envelope-from aspam@cox.net) Received: from fed1rmwml107.cox.net (fed1rmwml107.cox.net [68.230.241.177]) by mx1.freebsd.org (Postfix) with ESMTP id 49A4C82852 for ; Wed, 25 Oct 2017 21:14:28 +0000 (UTC) (envelope-from aspam@cox.net) Received: from eastrmimpo110.cox.net ([68.230.241.223]) by eastrmfepo203.cox.net (InterMail vM.8.01.05.28 201-2260-151-171-20160122) with ESMTP id <20171025210241.GSDQ4081.eastrmfepo203.cox.net@eastrmimpo110.cox.net> for ; Wed, 25 Oct 2017 17:02:41 -0400 Received: from thunder.sweets ([68.100.138.62]) by eastrmimpo110.cox.net with cox id Rx2h1w00F1LxgH801x2hhz; Wed, 25 Oct 2017 17:02:41 -0400 X-CT-Class: Clean X-CT-Score: 0.00 X-CT-RefID: str=0001.0A020205.59F0FBF1.013B, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 X-CT-Spam: 0 X-Authority-Analysis: v=2.2 cv=J/fa1EvS c=1 sm=1 tr=0 a=3mkzfl4ircflX6G+lDqBYw==:117 a=3mkzfl4ircflX6G+lDqBYw==:17 a=8nJEP1OIZ-IA:10 a=x7bEGLp0ZPQA:10 a=02M-m0pO-4AA:10 a=e9ASbk4n0QUA:10 a=5bW0dUBaVq7C9jwtRgUA:9 a=wPNLvfGTeEIA:10 X-CM-Score: 0.00 Authentication-Results: cox.net; none Received: from [10.10.10.15] (thunder.sweets [10.10.10.15]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by thunder.sweets (Postfix) with ESMTP id 1B2EA113FC for ; Wed, 25 Oct 2017 17:02:41 -0400 (EDT) Message-ID: <59F0FBEE.6030008@cox.net> Date: Wed, 25 Oct 2017 17:02:38 -0400 From: Joe Buehler User-Agent: Thunderbird 1.5.0.12 (X11/20120201) MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: virtio_net / netmap RX dropping frames Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Oct 2017 21:14:28 -0000 I am running virtio_net (netmap-modified) on top of netmap (latest) in a KVM virtual machine. The host adapter is Intel 82599ES 10G and the VM is connected to it via macvtap. My test setup is a small program in the VM sending frames out to an external loopback device and watching what comes back. I am running at fairly low frame rates (200k frames / sec) and seeing RX frame drops and high latency (a few milliseconds). The TX frames are all making it to the external loopback device (based on device counters) but the macvtap device in the RX path is reporting dropped frames, the count agreeing with what the test program observes. I guess my first question has to do with ring sizes. The netmap API is reporting 255 buffers in the RX and TX rings. How do I increase this substantially? Joe From owner-freebsd-net@freebsd.org Thu Oct 26 07:39:35 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB6FAE40957 for ; Thu, 26 Oct 2017 07:39:35 +0000 (UTC) (envelope-from maurizio@metaljonica.it) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id BC4EC6FC5A for ; Thu, 26 Oct 2017 07:39:35 +0000 (UTC) (envelope-from maurizio@metaljonica.it) Received: by mailman.ysv.freebsd.org (Postfix) id BBA67E40954; Thu, 26 Oct 2017 07:39:35 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BB3D4E40953 for ; Thu, 26 Oct 2017 07:39:35 +0000 (UTC) (envelope-from maurizio@metaljonica.it) Received: from mail-out-01.it.net (mailprofe01-152.it.net [151.1.195.152]) by mx1.freebsd.org (Postfix) with ESMTP id 1F4AE6FC58 for ; Thu, 26 Oct 2017 07:39:33 +0000 (UTC) (envelope-from maurizio@metaljonica.it) Received: from mail-fe-avasout01.mail.int (10.213.52.33) by mail-out-01.it.net (8.6.060.05) id 5981871300771F48 for net@freebsd.org; Thu, 26 Oct 2017 09:39:09 +0200 X-Junkmail-Premium-Raw: score=26/50, refid=2.7.2:2017.10.26.44516:17:26.894, ip=, rules=__HAS_FROM, __FRAUD_SUBJ_ALLCAPS, __SUBJ_ALPHA_END, __TO_MALFORMED_2, __TO_NAME, __CT, __CTYPE_MULTIPART_ALT, __CTYPE_HAS_BOUNDARY, __CTYPE_MULTIPART, __MIME_VERSION, __UNUSABLE_MSGID, __MIME_TEXT_P2, __MIME_TEXT_H2, __ANY_URI, __URI_WITH_PATH, LINK_TO_IMAGE, URI_ENDS_IN_HTML, __FRAUD_CONTACT_NUM, __PHISH_PHRASE5, __CP_MEDIA_BODY, __FRAUD_CONTACT_ADDY, __C230066_P5, __FRAUD_MONEY_DENOMINATION, __LINES_OF_YELLING, SUPERLONG_LINE, __URI_IN_BODY, __URI_NOT_IMG, __HTTP_IMAGE_TAG, __HTML_BOLD, __HTML_AHREF_TAG, __URI_WITHOUT_PATH, __HTML_TAG_CENTER, __STYLE_TAG, __HAS_HTML, __HTML_TAG_TABLE, __HTML_TAG_DIV, BODY_SIZE_10000_PLUS, BODYTEXTP_SIZE_3000_LESS, BODYTEXTH_SIZE_3000_MORE, __MIME_TEXT_H1, __MIME_TEXT_P1, __MIME_HTML, __TAG_EXISTS_HTML, LINES_OF_YELLING_3, LEO_OBFU_FLOAT, __URI_NS, HTML_70_90, NO_MESSAGE_ID, __MIME_TEXT_H, __MIME_TEXT_P, NO_URI_HTTPS, STYLE_RATWARE_COMBINED Received: from host52-226-static.226-95-b.business.telecomitalia.it (95.226.226.52) by mail-fe-avasout01.mail.int (9.0.019.04) (authenticated as maurizio@metaljonica.it) id 59B3AA5501499D18 for net@freebsd.org; Thu, 26 Oct 2017 09:39:06 +0200 Message-ID: <59B3AA5501499D18@mail-fe-avasout01.mail.int> (added by postmaster@mail-fe-avasout01.mail.int) From: "Maurizio Pisano" Subject: NEW BEACH CLEANER 2017 - EVOLUTION 205 HD - WITH HIGH DUMP To: "net" MIME-Version: 1.0 Organization: Metaljonica snc Date: Thu, 26 Oct 2017 09:39:05 +0200 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2017 07:39:36 -0000 =EF=BB=BFNewsletter MetaljonicaIf you do not see this newsletter prope= rly, consult the online version. Add info@puliscispiaggia.com to your address book contacts and always = arrive in your inbox.=20 =20 EVOLUTION HD NEW MODEL 2017 - with high dump =20 =20 Evolution Series =20 Read MoreP Series =20 Read MoreS Series =20 Read More METALJONICA MACHINE CLEAN BEACH =20 Welcome to METALJONICA =E2=80=93 Italy=E2=80=99s Leader in the product= ion of Beach cleaner machines. METALJONICA has been producing Beach cl= eaner machines for about 20 years. To satisfy the market=E2=80=99s nee= ds, we produce several types of beach cleaner machine. They all work according to an exclusive system planned by our company = which makes them strong and efficient. Our machine do not fear stones,= where the other ones stop, we go ahead. To make the difference contri= butes our consolidated experience like producers of beach cleaner mach= ines. We are already located in different geographic areas of the worl= d, always looking for new partnerships. GO TO NEW WEB SITESHOW VIDEODOWNLOAD CATALOG BUY DIRECTLY IN FACTORY SALES MANAGER Maurizio Pisano=20 TEL: +39 335.16 35 168 EMAIL: Info@puliscispiaggia.com Call with Skype OFFICES AND PRODUCTION A14 exit (Zone Autoporto) 64026 Roseto degli Abruzzi (TE)=20 VAT: 01489870673=20 TEL: +39 +39 085.89 37 973=20 EMAIL: Info@puliscispiaggia.com =20 METALJONICA SNC Zona Artigianale Autoporto ZIP CODE 64026 - Roseto deg= li Abruzzi ITALY PHONE/FAX +39 085 8937973 www.puliscispiaggia.com info@puliscispiaggia= =2Ecom - Unsubscribe by email=20 This message, for the D.lgs n.196 / 30.06.03 (Privacy Code), may conta= in confidential and/or privileged information. If you are not the addr= essee or authorized to receive this for the addressee, you must not us= e, copy, disclose or take any action based on this message or any info= rmation herein. If you have received this message in error, please adv= ise the sender immediately by reply e-mail and delete this message. Th= ank you for your cooperation. Your e-mail has been retrieved from the = web or from public directories. Our intention is only to provide infor= mation. If no you like our communications, with simple e-mail communic= ation and nothing more will be delivered. Guarantee of confidentiality= Metaljonica s.n.c. guarantees the confidentiality of the data provide= d pursuant to Legislative Decree 196/2003 (Privacy Code). At any time = you can exercise your rights under art. 7 of Legislative Decree 196/20= 03., Including those to rectify and delete data or to oppose the treat= ment, writing to info@puliscispiaggia.com The data controller is Metal= jonica s.n.c.=20 From owner-freebsd-net@freebsd.org Thu Oct 26 07:41:25 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 503FCE40EBA for ; Thu, 26 Oct 2017 07:41:25 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: from mail-it0-x22d.google.com (mail-it0-x22d.google.com [IPv6:2607:f8b0:4001:c0b::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1355170016 for ; Thu, 26 Oct 2017 07:41:25 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: by mail-it0-x22d.google.com with SMTP id k70so12960707itk.0 for ; Thu, 26 Oct 2017 00:41:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=eTpK5F3pGtcZTv1kFQlls8hPmgoApASTu9my2eNuIac=; b=URL7mwl51QWwH6OlkKh3bCAPcaUK67shzSRGjXaNXnfWPhyDBdhzWDFEM2w/G4QnmF iGxbtOjPZuDJmserYXkfDX/EjKzy4sO3lBvaXXOGk3147ybvp58r9IuoEY/vtqSK5Uef 00U1wvvraM55RP1HTnpkHYu+R1TqBsIIfYt+YF+bHBR9mciOWs3FJ3snj1juo8gNhcBa xkgev96Db5XElZsYXUTYjAgehLxQCq/1Pvxv10evBK5oAWeBe/B8kO2OjabtGrBgSxR3 JZeF7/I7qQVWnZksrcxY9xOwDTiGm3y93ezi7Xf18PFGBlGKRTgd5Ze9VrKpA0WYqoup MIcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=eTpK5F3pGtcZTv1kFQlls8hPmgoApASTu9my2eNuIac=; b=Vq+T9JJVSmAI8VAABhD88yC0ZlsZVv1snEL1CXqU7MrdyBozSL0t+6lW32biKEvvSF CVvwIdQI67EZhawOcOAaF334/ONHu9Uelj4RNXuOSCmoVziE67RzrCfckpwAFaBUm+9f BzeL0D7J0v+TusSbnJhSaP0AIAvxci0NBb8NKNgHGE74+yKcm62spDDf8ts3Oq9nqwpB 2kcZtX7OJ/VvXM36lYYszkbUXKnKpOUF5kmDHlLX+jwATjiZ/lVzn14pedKTtk3eXWRS Me/6JG6aKBSERwvqpkiBJo7FYyYWrb3l4cQ4Ymecx5fg2up72mODTQtjVIewng8mXFjQ qObg== X-Gm-Message-State: AMCzsaXzTwEhKlQg+LtjBu8Z8TQX/Mnq+6yPfp7vUzqyr34zqCwld6Vv 01ipRHDjJbuTzN2mp6cq84W6cS+ZttrLncv9HnSiWQ== X-Google-Smtp-Source: ABhQp+RTHsxGtyjNOc7VT3WW3yoY+XC2dUHnsNqHmVn63fWQz6J3oG4ZMXbo1JsNb9Wb58mqZ1pCTzS+gJqBaigdIqc= X-Received: by 10.36.250.4 with SMTP id v4mr1305010ith.31.1509003684205; Thu, 26 Oct 2017 00:41:24 -0700 (PDT) MIME-Version: 1.0 Received: by 10.2.176.143 with HTTP; Thu, 26 Oct 2017 00:41:23 -0700 (PDT) In-Reply-To: <59F0FBEE.6030008@cox.net> References: <59F0FBEE.6030008@cox.net> From: Vincenzo Maffione Date: Thu, 26 Oct 2017 09:41:23 +0200 Message-ID: Subject: Re: virtio_net / netmap RX dropping frames To: Joe Buehler Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2017 07:41:25 -0000 I guess you are using a FreeBSD guest. Is this the case? If you have the chance, try a linux guest to check if virtio-net works better there (I've used netmap on the netmap-patched virtio-net in Linux guests, never tried on FreeBSD). The netmap ring size is just the NIC ring size. If you change the virtio-net NIC ring size (sysctl on FreeBSD, I guess). Anyway, for your specific use-case (VM accessing the physical 10G NIC) there is a way better solution, which is the netmap passthrough. Check out the virtualization.pdf in this tutorial https://github.com/vmaffione/netmap-tutorial. You basically need to run QEMU (with KVM enabled), saying that you want to pass through a netmap port (e.g. netmap:ethX in your case) to a VM. Then in the FreeBSD VM you will see a "ptnet0" interface, where you can use pkt-gen. You should get a 10x improvement if properly configured. Cheers, Vincenzo 2017-10-25 23:02 GMT+02:00 Joe Buehler : > I am running virtio_net (netmap-modified) on top of netmap (latest) in a > KVM virtual machine. The host adapter is Intel 82599ES 10G and the VM > is connected to it via macvtap. > > My test setup is a small program in the VM sending frames out to an > external loopback device and watching what comes back. > > I am running at fairly low frame rates (200k frames / sec) and seeing RX > frame drops and high latency (a few milliseconds). The TX frames are > all making it to the external loopback device (based on device counters) > but the macvtap device in the RX path is reporting dropped frames, the > count agreeing with what the test program observes. > > I guess my first question has to do with ring sizes. The netmap API is > reporting 255 buffers in the RX and TX rings. How do I increase this > substantially? > > Joe > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > -- Vincenzo Maffione From owner-freebsd-net@freebsd.org Thu Oct 26 10:56:03 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8C100E45B35 for ; Thu, 26 Oct 2017 10:56:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7490D75837 for ; Thu, 26 Oct 2017 10:56:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v9QAu16n016012 for ; Thu, 26 Oct 2017 10:56:03 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 219390] [ixgbe] [patch] ixgbe stripping vlan_tag by default Date: Thu, 26 Oct 2017 10:56:02 +0000 X-Bugzilla-Reason: CC AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: piotr.pietruszewski@intel.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2017 10:56:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219390 Piotr Pietruszewski changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |piotr.pietruszewski@intel.c | |om --- Comment #9 from Piotr Pietruszewski --- (In reply to Charles Goncalves from comment #8) Thank you for reproduction steps. We discovered that your patch doesn't cov= er all scenarios. Fixed patch is now under review ( https://reviews.freebsd.org/D12795 ). --=20 You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Thu Oct 26 11:44:09 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 04A3EE46C61 for ; Thu, 26 Oct 2017 11:44:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E6F7376FC5 for ; Thu, 26 Oct 2017 11:44:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v9QBi8hX036064 for ; Thu, 26 Oct 2017 11:44:08 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 219390] [ixgbe] [patch] ixgbe stripping vlan_tag by default Date: Thu, 26 Oct 2017 11:44:08 +0000 X-Bugzilla-Reason: CC AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: halfling@halfling.com.br X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2017 11:44:09 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219390 --- Comment #10 from Charles Goncalves --- (In reply to Piotr Pietruszewski from comment #9) Hello Piotr! Thank you for your answer. I have tested your patch in my test scenario and= it worked as expected.=20 Thank you very much! --=20 You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Thu Oct 26 14:31:42 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B210E4B13A for ; Thu, 26 Oct 2017 14:31:42 +0000 (UTC) (envelope-from aspam@cox.net) Received: from fed1rmfepi208.cox.net (fed1rmfepi208.cox.net [68.230.241.153]) by mx1.freebsd.org (Postfix) with ESMTP id 4667D80389 for ; Thu, 26 Oct 2017 14:31:42 +0000 (UTC) (envelope-from aspam@cox.net) Received: from eastrmimpo210.cox.net ([68.230.241.225]) by eastrmfepo102.cox.net (InterMail vM.8.01.05.28 201-2260-151-171-20160122) with ESMTP id <20171026143124.NGUX14605.eastrmfepo102.cox.net@eastrmimpo210.cox.net> for ; Thu, 26 Oct 2017 10:31:24 -0400 Received: from thunder.sweets ([68.100.138.62]) by eastrmimpo210.cox.net with cox id SEXQ1w0051LxgH801EXQ9X; Thu, 26 Oct 2017 10:31:24 -0400 X-CT-Class: Clean X-CT-Score: 0.00 X-CT-RefID: str=0001.0A020206.59F1F1BC.00E6, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 X-CT-Spam: 0 X-Authority-Analysis: v=2.2 cv=c+jbeV1l c=1 sm=1 tr=0 a=3mkzfl4ircflX6G+lDqBYw==:117 a=3mkzfl4ircflX6G+lDqBYw==:17 a=IkcTkHD0fZMA:10 a=x7bEGLp0ZPQA:10 a=02M-m0pO-4AA:10 a=e9ASbk4n0QUA:10 a=HxoghPbS-FUynUJKAIkA:9 a=QEXdDO2ut3YA:10 X-CM-Score: 0.00 Authentication-Results: cox.net; none Received: from [10.10.10.15] (thunder.sweets [10.10.10.15]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by thunder.sweets (Postfix) with ESMTP id CC031113FC; Thu, 26 Oct 2017 10:31:23 -0400 (EDT) Message-ID: <59F1F1BA.3020608@cox.net> Date: Thu, 26 Oct 2017 10:31:22 -0400 From: Joe Buehler User-Agent: Thunderbird 1.5.0.12 (X11/20120201) MIME-Version: 1.0 To: Vincenzo Maffione CC: "freebsd-net@freebsd.org" Subject: Re: virtio_net / netmap RX dropping frames References: <59F0FBEE.6030008@cox.net> In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2017 14:31:42 -0000 Vincenzo Maffione wrote: > I guess you are using a FreeBSD guest. Is this the case? If you have the Sorry, I am using LINUX, ubuntu 16.04 LTS for both host and VM. I am posting here at standing request of netmap driver author. The host has 24 CPUs @ 2.5 GHz and 128G of memory and is *idle* so I am a bit disappointed > chance, try a linux guest to check if virtio-net works better there > (I've used netmap on the netmap-patched virtio-net in Linux guests, > never tried on FreeBSD). > The netmap ring size is just the NIC ring size. If you change the > virtio-net NIC ring size (sysctl on FreeBSD, I guess). OK I'll look into that. I increased the ring size on the host ixgbe but that had no effect so I guess it must be virtio_net. > Anyway, for your specific use-case (VM accessing the physical 10G NIC) > there is a way better solution, which is the netmap passthrough. Unfortunately I don't have control of the host, just the VM, so pt netmap is not an option. My initial query regarded frame drops but the latency is also pretty bad. The LINUX ping utility inside the VM says 0.2 mS consistently without netmap in use. My app sees that value for almost all frames but it spikes (up to 0.6 mS!) for a few frames, which is not acceptable for this application -- was expecting much better due to network stack bypass. And this is just 100 frames/sec... Joe Buehler From owner-freebsd-net@freebsd.org Thu Oct 26 15:12:25 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A3DF8E4BE86 for ; Thu, 26 Oct 2017 15:12:25 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: from mail-io0-x232.google.com (mail-io0-x232.google.com [IPv6:2607:f8b0:4001:c06::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6A4D081739 for ; Thu, 26 Oct 2017 15:12:25 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: by mail-io0-x232.google.com with SMTP id 189so6094015iow.10 for ; Thu, 26 Oct 2017 08:12:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=gQ2N++ierR6cM+Xp9MZgrx0zH3f6v1/Cq+MpFn1x/9Y=; b=t29JMZstmElM99xZu50R1uhwswxyNT7q1HwV1HefLN18Nl3TQdLcd/CX4g1iIp4t3h z17zRPCN3d2pN2kbvw1iwOYqSj1gqt4+RSb+QHQDvsS00ilAZGorDKQpc1QHJF8Leo/z a8UvRs/S/LySWGbGiPynjbkaiEh4q/q0srM9RHncrJyp3zQZaEt9EmoLBsd6gbU8LxK8 28ZE5usCoIclkwCOPQy+lyC0b6lO4x7M4tCZ/U2K778/oz0qwuWmChv5OPr94Gb2fSNs NolaCzs90L4WQR7cik4zb6j+Q7wxQj4Pky21gma8OnYFPI/tMQ9lKsOz10/02d5XlxeS PidA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=gQ2N++ierR6cM+Xp9MZgrx0zH3f6v1/Cq+MpFn1x/9Y=; b=nv+lEmmIg6WNKkk9bA8Vt0HB5MPXeZUmHc4nmpn25mJjwWgpFUH+Bm/AIkBJ/Gh6S4 vNGv0eR+QaoqzqU2iwUhwMUKX7Mqeyg+GbKvsGInXPEmVX7FnVbz3/8aFfx5g2Ex7rBD eWikN5HwiADi+okU22jH6Qo8v0NTqziO5lmmNu1OaJsu4FJ8OagX58ud5PleBk7hK0Kt ld0fR59bGa8wNfUXl6gENd2x9+9/YiFWKP9zz2pSLYqvBB0xTjPR02/IKdBzkE4CdQRe xNkNfcVHyg7vjf23GeAex1jWnFric5J0FnaAhJpj3nHmChSwIhPn3YUCl0KBiYEkZ/eV dd/A== X-Gm-Message-State: AMCzsaUWaJYStKQYqm8RCg0mX3SkP0lrcqEvjWsuurOQQsVAU3R3NdXm mWUhga5KavBDVylln+noqBCB+sdm/RSgbXJc1nw= X-Google-Smtp-Source: ABhQp+Rm62tS8kDhZnvW3pkZdaMDvYJY5MPel/qYK8VNx3CTcUisHdGODI9vl7mKudqgA3FncEXbDU68f+ztJqLEf0E= X-Received: by 10.107.59.18 with SMTP id i18mr28723286ioa.97.1509030744664; Thu, 26 Oct 2017 08:12:24 -0700 (PDT) MIME-Version: 1.0 Received: by 10.2.176.143 with HTTP; Thu, 26 Oct 2017 08:12:24 -0700 (PDT) In-Reply-To: <59F1F1BA.3020608@cox.net> References: <59F0FBEE.6030008@cox.net> <59F1F1BA.3020608@cox.net> From: Vincenzo Maffione Date: Thu, 26 Oct 2017 17:12:24 +0200 Message-ID: Subject: Re: virtio_net / netmap RX dropping frames To: Joe Buehler Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2017 15:12:25 -0000 So you are using netmap only in the guest (and not in the host). And you are running a sender and a receiver inside the VM, both on the VM interface. Something like this # pkt-gen -i eth1 -f rx # pkt-gen -i eth1 -f tx ? What happens if you use pkt-gen rather than your application? 2017-10-26 16:31 GMT+02:00 Joe Buehler : > Vincenzo Maffione wrote: > > I guess you are using a FreeBSD guest. Is this the case? If you have the > > Sorry, I am using LINUX, ubuntu 16.04 LTS for both host and VM. I am > posting here at standing request of netmap driver author. > > The host has 24 CPUs @ 2.5 GHz and 128G of memory and is *idle* so I am > a bit disappointed > > > chance, try a linux guest to check if virtio-net works better there > > (I've used netmap on the netmap-patched virtio-net in Linux guests, > > never tried on FreeBSD). > > The netmap ring size is just the NIC ring size. If you change the > > virtio-net NIC ring size (sysctl on FreeBSD, I guess). > > OK I'll look into that. I increased the ring size on the host ixgbe but > that had no effect so I guess it must be virtio_net. > > > Anyway, for your specific use-case (VM accessing the physical 10G NIC) > > there is a way better solution, which is the netmap passthrough. > > Unfortunately I don't have control of the host, just the VM, so pt > netmap is not an option. > > My initial query regarded frame drops but the latency is also pretty > bad. The LINUX ping utility inside the VM says 0.2 mS consistently > without netmap in use. My app sees that value for almost all frames but > it spikes (up to 0.6 mS!) for a few frames, which is not acceptable for > this application -- was expecting much better due to network stack > bypass. And this is just 100 frames/sec... > > Joe Buehler > -- Vincenzo Maffione From owner-freebsd-net@freebsd.org Thu Oct 26 17:01:45 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 984CAE4E3FF for ; Thu, 26 Oct 2017 17:01:45 +0000 (UTC) (envelope-from aspam@cox.net) Received: from eastrmfepo102.cox.net (eastrmfepo102.cox.net [68.230.241.214]) by mx1.freebsd.org (Postfix) with ESMTP id 48D201627 for ; Thu, 26 Oct 2017 17:01:44 +0000 (UTC) (envelope-from aspam@cox.net) Received: from eastrmimpo109.cox.net ([68.230.241.222]) by eastrmfepo102.cox.net (InterMail vM.8.01.05.28 201-2260-151-171-20160122) with ESMTP id <20171026170144.TAXC14605.eastrmfepo102.cox.net@eastrmimpo109.cox.net> for ; Thu, 26 Oct 2017 13:01:44 -0400 Received: from thunder.sweets ([68.100.138.62]) by eastrmimpo109.cox.net with cox id SH1j1w00a1LxgH801H1jB6; Thu, 26 Oct 2017 13:01:44 -0400 X-CT-Class: Clean X-CT-Score: 0.00 X-CT-RefID: str=0001.0A020205.59F214F8.0042, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 X-CT-Spam: 0 X-Authority-Analysis: v=2.2 cv=OtA/823t c=1 sm=1 tr=0 a=3mkzfl4ircflX6G+lDqBYw==:117 a=3mkzfl4ircflX6G+lDqBYw==:17 a=IkcTkHD0fZMA:10 a=x7bEGLp0ZPQA:10 a=02M-m0pO-4AA:10 a=e9ASbk4n0QUA:10 a=OvaVfXy-ICjw5lLnD7EA:9 a=QEXdDO2ut3YA:10 X-CM-Score: 0.00 Authentication-Results: cox.net; none Received: from [10.10.10.15] (thunder.sweets [10.10.10.15]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by thunder.sweets (Postfix) with ESMTP id 483B9113FC; Thu, 26 Oct 2017 13:01:43 -0400 (EDT) Message-ID: <59F214F6.9060905@cox.net> Date: Thu, 26 Oct 2017 13:01:42 -0400 From: Joe Buehler User-Agent: Thunderbird 1.5.0.12 (X11/20120201) MIME-Version: 1.0 To: Vincenzo Maffione CC: "freebsd-net@freebsd.org" Subject: Re: virtio_net / netmap RX dropping frames References: <59F0FBEE.6030008@cox.net> <59F1F1BA.3020608@cox.net> In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2017 17:01:45 -0000 Vincenzo Maffione wrote: > So you are using netmap only in the guest (and not in the host). > And you are running a sender and a receiver inside the VM, both on the > VM interface. > Something like this > > # pkt-gen -i eth1 -f rx > # pkt-gen -i eth1 -f tx Yes that's the basic idea. > > ? > What happens if you use pkt-gen rather than your application? I was under the impression that I can't have two threads in the netmap kernel code at the same time so can't do that. Joe Buehler From owner-freebsd-net@freebsd.org Thu Oct 26 17:26:11 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C472FE4EEC6 for ; Thu, 26 Oct 2017 17:26:11 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: from mail-io0-x232.google.com (mail-io0-x232.google.com [IPv6:2607:f8b0:4001:c06::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 89B0A26E0 for ; Thu, 26 Oct 2017 17:26:11 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: by mail-io0-x232.google.com with SMTP id 134so7048004ioo.0 for ; Thu, 26 Oct 2017 10:26:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=L5jJq6Sbd0zluvDqZOGt9PLxs2RmRZKtRYxcff7ZH14=; b=qIftFGHOmk0ZcG07bPYv2ClGN2zXSBXzLd2M2Kn4yzohQFdkZp8A4ZTSTKG4mizY3X 5ma2b/k9OYcxvmTk9vrLm/Lk2+mCCsqWx0GtIra08SteP1UKmnzjrLlY4SHlzIw7xbLY DEEOQ3aaexZbDh+GmflSpCa0kY59dxec6c7pu++sCemSiH0O/RAt9XgfbKP+yM7nX3/u 9FJl2LMOBEXLouUZw6otzXT9plU6b6MXuZrKsRa+JLM8lZnhw53hPR6oD0EyNs7NVcyI rfpnQjN+/hpyr4SWKNtSteVO4lEBhiQfMcOIA1WGUq7UDu2s9Ke/oPtneY6BqtLLrzN6 lthA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=L5jJq6Sbd0zluvDqZOGt9PLxs2RmRZKtRYxcff7ZH14=; b=FZmtiIDiRzG9y528z8Iyzcx76xmyFSb8v/qMIewep6y3LPhDJIs112GUgIfLqSti2Q ZScuYva7PZuuL5ykhYFY/6FcjvkrlkoFgEecxiMB/vNInKsDMoEyfkr3boxr7IJ34Esn 0ZZ1qr3leZYolFFrL8egnuzguvZWeY3bO3La0rvc9pTvZ58u64nNaVbhpJz5GJMNGpn5 gD7m2ZbgjkXGQW7gvKS6RBVGU7BbS1vXl7taaksr4Nu7gMZKIHBT8/TqEHLZbTQ+lF5H XUftL494B8bJBH0cYiv27Z/fje0A0IAc5wiYmoHY2RzcG4cK7evPo1JJe53Dkl0IRsLc HSJA== X-Gm-Message-State: AMCzsaVX+SqHPPAPJuLOV3Zp6zuBl5cRTOYR09Q51aRMng5XbhSDDZvx Rz244zQUrhIeYuG511d1jRIuspzFQHg5pNfRaq8= X-Google-Smtp-Source: ABhQp+SypsGbh2plwekPy5y2eMnrCMLt+lRH5tYAmNNSAk67++gfYP5YXvVdCRVd24Pbph7T5dDwxgumq+niudeAO3E= X-Received: by 10.107.18.74 with SMTP id a71mr33330798ioj.58.1509038770717; Thu, 26 Oct 2017 10:26:10 -0700 (PDT) MIME-Version: 1.0 Received: by 10.2.176.143 with HTTP; Thu, 26 Oct 2017 10:26:09 -0700 (PDT) Received: by 10.2.176.143 with HTTP; Thu, 26 Oct 2017 10:26:09 -0700 (PDT) In-Reply-To: <59F214F6.9060905@cox.net> References: <59F0FBEE.6030008@cox.net> <59F1F1BA.3020608@cox.net> <59F214F6.9060905@cox.net> From: Vincenzo Maffione Date: Thu, 26 Oct 2017 19:26:09 +0200 Message-ID: Subject: Re: virtio_net / netmap RX dropping frames To: Joe Buehler Cc: FreeBSD Net Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2017 17:26:11 -0000 You can have how many threads and processes you want. The constraint is that there must not be two threads accessing the same ring at the same time. In this case each pktgen is accessing different rings. Il 26 ott 2017 7:01 PM, "Joe Buehler" ha scritto: > Vincenzo Maffione wrote: > > So you are using netmap only in the guest (and not in the host). > > And you are running a sender and a receiver inside the VM, both on the > > VM interface. > > Something like this > > > > # pkt-gen -i eth1 -f rx > > # pkt-gen -i eth1 -f tx > > Yes that's the basic idea. > > > > > ? > > What happens if you use pkt-gen rather than your application? > > I was under the impression that I can't have two threads in the netmap > kernel code at the same time so can't do that. > > Joe Buehler > From owner-freebsd-net@freebsd.org Thu Oct 26 17:49:00 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 40C8EE4F624 for ; Thu, 26 Oct 2017 17:49:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2CD733313 for ; Thu, 26 Oct 2017 17:49:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v9QHmx77016325 for ; Thu, 26 Oct 2017 17:49:00 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 221990] panic: Assertion reclaimable == delta failed at ../../../net/iflib.c:1947 Date: Thu, 26 Oct 2017 17:48:59 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: shurd@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2017 17:49:00 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221990 --- Comment #5 from Stephen Hurd --- Can you try with this: Index: sys/net/iflib.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sys/net/iflib.c (revision 324937) +++ sys/net/iflib.c (working copy) @@ -1931,6 +1931,7 @@ } done: + MPASS(n =3D=3D i =3D=3D 0); DBG_COUNTER_INC(rxd_flush); if (fl->ifl_pidx =3D=3D 0) pidx =3D fl->ifl_size - 1; It looks like ifl_credits could get out of sync in the error paths here, but I'm not sure you're hitting any of them. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Thu Oct 26 17:49:15 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 43F3BE4F656 for ; Thu, 26 Oct 2017 17:49:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 318B633B6 for ; Thu, 26 Oct 2017 17:49:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v9QHnEnK016735 for ; Thu, 26 Oct 2017 17:49:15 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 221990] panic: Assertion reclaimable == delta failed at ../../../net/iflib.c:1947 Date: Thu, 26 Oct 2017 17:49:15 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: shurd@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: shurd@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2017 17:49:15 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221990 Stephen Hurd changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-net@FreeBSD.org |shurd@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Thu Oct 26 18:05:07 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC367E4FFB1 for ; Thu, 26 Oct 2017 18:05:07 +0000 (UTC) (envelope-from aspam@cox.net) Received: from fed1rmwml205.cox.net (fed1rmwml205.cox.net [68.230.241.179]) by mx1.freebsd.org (Postfix) with ESMTP id A7C8663614 for ; Thu, 26 Oct 2017 18:05:07 +0000 (UTC) (envelope-from aspam@cox.net) Received: from eastrmimpo209.cox.net ([68.230.241.224]) by eastrmfepo101.cox.net (InterMail vM.8.01.05.28 201-2260-151-171-20160122) with ESMTP id <20171026180452.FOFI4075.eastrmfepo101.cox.net@eastrmimpo209.cox.net> for ; Thu, 26 Oct 2017 14:04:52 -0400 Received: from thunder.sweets ([68.100.138.62]) by eastrmimpo209.cox.net with cox id SJ4s1w00F1LxgH801J4s3d; Thu, 26 Oct 2017 14:04:52 -0400 X-CT-Class: Clean X-CT-Score: 0.00 X-CT-RefID: str=0001.0A020205.59F223C4.010D, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 X-CT-Spam: 0 X-Authority-Analysis: v=2.2 cv=G/psK5s5 c=1 sm=1 tr=0 a=3mkzfl4ircflX6G+lDqBYw==:117 a=3mkzfl4ircflX6G+lDqBYw==:17 a=IkcTkHD0fZMA:10 a=x7bEGLp0ZPQA:10 a=02M-m0pO-4AA:10 a=e9ASbk4n0QUA:10 a=YuaIYVqPCbt3uL0FN-4A:9 a=QEXdDO2ut3YA:10 X-CM-Score: 0.00 Authentication-Results: cox.net; none Received: from [10.10.10.15] (thunder.sweets [10.10.10.15]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by thunder.sweets (Postfix) with ESMTP id D03C5113FC; Thu, 26 Oct 2017 14:04:51 -0400 (EDT) Message-ID: <59F223C3.5040907@cox.net> Date: Thu, 26 Oct 2017 14:04:51 -0400 From: Joe Buehler User-Agent: Thunderbird 1.5.0.12 (X11/20120201) MIME-Version: 1.0 To: Vincenzo Maffione CC: FreeBSD Net Subject: Re: virtio_net / netmap RX dropping frames References: <59F0FBEE.6030008@cox.net> <59F1F1BA.3020608@cox.net> <59F214F6.9060905@cox.net> In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2017 18:05:07 -0000 Vincenzo Maffione wrote: > You can have how many threads and processes you want. The constraint is > that there must not be two threads accessing the same ring at the same > time. In this case each pktgen is accessing different rings. Thanks that was very useful info. I had run this before and got large frame drop so assumed it was a violation of threading constraints. So now I can remove a mutex from my app, which has an RX and TX thread. Running the two pkt-gen instances, I am getting a lot of RX frame drop. Based on counters, the TX frames are making it to the external loopback device, about 1 million frames/sec, which is looping them all back, but the macvtap interface on the host that feeds into the ixgbe/vhost/virtio_net/netmap interface in the VM shows about 80-90% of them as dropped. CPU usage is low in the VM, very roughly 25% for the TX thread and 5% for the RX thread. The frame rate displayed by pkt-gen and the CPU displayed by top is bouncing around. Joe Buehler From owner-freebsd-net@freebsd.org Fri Oct 27 01:06:43 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C61E5E57B1E for ; Fri, 27 Oct 2017 01:06:43 +0000 (UTC) (envelope-from sunxiaoye07@gmail.com) Received: from mail-oi0-x234.google.com (mail-oi0-x234.google.com [IPv6:2607:f8b0:4003:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8A2C870B4B for ; Fri, 27 Oct 2017 01:06:43 +0000 (UTC) (envelope-from sunxiaoye07@gmail.com) Received: by mail-oi0-x234.google.com with SMTP id j126so8639565oib.8 for ; Thu, 26 Oct 2017 18:06:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:from:date:message-id:subject:to; bh=kmEpKiS3lyJFkket+Wx9oq+3fay/HyVkEUcyfVCj7qA=; b=LhScPnYk7g45S6YRsqRedMz0bbLI6dk846dRqq1JbjE1l74xItGCMF+IIBlASiq5RX AF6E1c95ZvSg76o9XbyJ3XzIkJyt2hmWFFLLnAR2SCudBZymfz7zoLUpYdfMTlw2fvY5 6H9AnQWhJQj5JGNlDNvJIGEDXAWaBiGi8MYkAUw+vvSBrJiENtM7/Qpm3UaoXaTsK6FE l8GXmzp1dNWKUVthRtZrxEBGou23gqgap7XsE5LbLJFkiuZYNNvYO9jAysNdqHCaG44k fXYXadfIVIqnva0uRpHoCoVejh19/blKkXvGT+Y03LuQzNIcSCLAAeflIaSO+1d1XvI6 Uipg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:from:date:message-id:subject :to; bh=kmEpKiS3lyJFkket+Wx9oq+3fay/HyVkEUcyfVCj7qA=; b=Fe8hLEcUPXCdJ5q5GUE6A5sTamIQoXj4AxGGxhmu6hZ0AassPIHnDLQf8SRyRBzPZ+ CMmV2swobWBwpfSKSlwKmLKBpZLV5AKxI51TCKS2nNDgaAB0dUhdHFMo6Z/ZBB2swoVg B1Jf9rtr1/nu4/hV/BYNG/SKntppVnn+6859CWTS/TQtKoKE+scaEXOwXA45xOFXrRA4 rOF+J5GGGvYVt9ZbardVdQ0SE7eYFwltIbN3OGoizxyIaBbp5L6AEEJ+95dIclwFhwPB T1Q7g/suZy152fAYOx7kwEiG4LsVlZvIla7mQ6ZZc8ZptJZDsd0X6gU9SNG43aDlipJN l9Gg== X-Gm-Message-State: AMCzsaU0EFibI/aou5fhfwZuwjSvO1CFoL18CIu9lHa+s06O/E1VT15Z fvW1Z4qF8smdvK2veqSGb3+xTTigynPOVONSo3KAAA== X-Google-Smtp-Source: ABhQp+QMzNpdBUatFdLTuhpmF/r6nerDzgjtOdTBKPHF8YutYxc48UN/XH7Obmu8H70yon1MK9+mB04VfEDe36tAHcA= X-Received: by 10.202.77.82 with SMTP id a79mr3156617oib.297.1509066402653; Thu, 26 Oct 2017 18:06:42 -0700 (PDT) MIME-Version: 1.0 Sender: sunxiaoye07@gmail.com Received: by 10.157.72.241 with HTTP; Thu, 26 Oct 2017 18:06:42 -0700 (PDT) From: Xiaoye Sun Date: Thu, 26 Oct 2017 20:06:42 -0500 X-Google-Sender-Auth: UNIx-SHbmw6xbohQdsoR66Ngqh0 Message-ID: Subject: [netmap] when does a packet in the netmap ring send out exactly To: FreeBSD Net Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Oct 2017 01:06:43 -0000 Hi I write a netmap program that sends packets to the network. my program uses one netmap ring and fills the ring slots with packets. My program needs to do something (action A) after a particular packet (packet P) in the ring slot is sent to the network. so the program tracks the position of the tail point and checks if the tail point has moved across the slot I used to put that packet P. However, I found that the tail pointer may not move forward even seconds after the receiver side got packet P. Sometimes the tail pointer never moves forward until the TX ring is full. I try ioctl(NIOCTXSYNC), however, it cannot 100% solve the problem. My question is that is there a way to make the TX ring empty as early as possible so that I can know when my packet is sent out. or is there another way to know when the packet in the slot is sent to the network/NIC physical queue? I am using Linux 3.16.0-4-amd64. Thanks! Best, Xiaoye From owner-freebsd-net@freebsd.org Fri Oct 27 06:58:59 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1D020E5DB98 for ; Fri, 27 Oct 2017 06:58:59 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: from mail-io0-x230.google.com (mail-io0-x230.google.com [IPv6:2607:f8b0:4001:c06::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E5BA07E4EA for ; Fri, 27 Oct 2017 06:58:58 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: by mail-io0-x230.google.com with SMTP id m16so10650467iod.1 for ; Thu, 26 Oct 2017 23:58:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=lJVHEYE68FzbQYfDxjqTRWRXEL1JHGoAWbXTCKehHs0=; b=WgMYijgPIKWYuiPLVlPh34ZuR6Tc/r8lZvhIGnlf3ZgwoHe7dZHJ5DsmEtBVvki9s/ ZoTt+Q2hU1kqSAaT4J5kee0r6TMLLB4ASgSezcuLqi8KueIB7e5seQMv16/VUd5syw7c +d0TsFR9LTm0GunJbvKTFJ4p1H1BDd+bRYfzCam9fKFyglLSqUbWeDstsjmmsplj8ZCx JjElQhzVQ6o6J/uvdYiwp3NCGv1EpnujbHxveA7DbFstIzFU+YWf+Buzn/wFDvLlE/Qo sXFA29WTylc9Y+GwguRWfE+a/jP9h7DHiq+CyQhWtIgnpz/lmgJhh8q2aJQ7msvMu2OO VVJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=lJVHEYE68FzbQYfDxjqTRWRXEL1JHGoAWbXTCKehHs0=; b=UWcY//SI6ulbdBVVwj6FzLwNUkjxv1gSYVU/9yWvR7fE8Xbo6dSLrUuOccjYDHUFA7 A3VVPORa2O647ptWwwOExcnZ2wBJtBv1oG4sP2I1NMNrxnde8YbEVhRsrh2ffmuH5PU+ guEQDIERIxJZ8YpOTPkhK20rJ/eTs9MIIIwI9VYilZh5rz0ZtxeGh4Sp9KLy73cjmw6m phxGtGpUeKwVNJGSJcza7PZ/gLu/6FvwqUXM3QVbjDFiaR5OQ4xHvjR6LiNsAZCnMfNN jVCkITRbgDEnSEGqPcba7zUjVXSEY4khRhC+mvOdIPFwefxIY3ZmH4I5kqjm/7yp94B2 Pptw== X-Gm-Message-State: AMCzsaUv2Pm6Mj4Q2NtTExU74L9ZcoNvVVB7zg7bhfGT+NPxlmXYE3F+ oNuRGVy7EwYSO9I50dNrXqLZDVtZgRyebFbkDy0= X-Google-Smtp-Source: ABhQp+RLeNdqoRSIG7PRxOh7k464J2ZBkT5Ti5yq2JlUWAA8PUUDArCOgvrEziWSQz415G9TfAJ35PIaBqKUKfhGZAQ= X-Received: by 10.107.151.19 with SMTP id z19mr34281695iod.248.1509087537860; Thu, 26 Oct 2017 23:58:57 -0700 (PDT) MIME-Version: 1.0 Received: by 10.2.176.143 with HTTP; Thu, 26 Oct 2017 23:58:57 -0700 (PDT) In-Reply-To: <59F223C3.5040907@cox.net> References: <59F0FBEE.6030008@cox.net> <59F1F1BA.3020608@cox.net> <59F214F6.9060905@cox.net> <59F223C3.5040907@cox.net> From: Vincenzo Maffione Date: Fri, 27 Oct 2017 08:58:57 +0200 Message-ID: Subject: Re: virtio_net / netmap RX dropping frames To: Joe Buehler Cc: FreeBSD Net Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Oct 2017 06:58:59 -0000 2017-10-26 20:04 GMT+02:00 Joe Buehler : > Vincenzo Maffione wrote: > > > You can have how many threads and processes you want. The constraint is > > that there must not be two threads accessing the same ring at the same > > time. In this case each pktgen is accessing different rings. > > Thanks that was very useful info. I had run this before and got large > frame drop so assumed it was a violation of threading constraints. So > now I can remove a mutex from my app, which has an RX and TX thread. > > Running the two pkt-gen instances, I am getting a lot of RX frame drop. > Based on counters, the TX frames are making it to the external loopback > device, about 1 million frames/sec, which is looping them all back, but > the macvtap interface on the host that feeds into the > ixgbe/vhost/virtio_net/netmap interface in the VM shows about 80-90% of > them as dropped. CPU usage is low in the VM, very roughly 25% for the > TX thread and 5% for the RX thread. The frame rate displayed by pkt-gen > and the CPU displayed by top is bouncing around. > > Yes, mutexes are only needed for concurrent access to the same ring, which is not your case. >From this description it seems that your problem is not netmap or pkt-gen. Your TX pkt-gen transmits 1Mpps with 0.25 CPUs, which is ok giving the limitations of virtio-net. Your RX pkt-gen is not really doing that much work (0.05 CPU), which means that the virtio-net RX ring is almost always empty. You need to figure out who is dropping the RX packets and why. This seems to happen before the packets really make their way to the virtio-net RX ring. So it must be some queue overflow in the macvtap or your ixgbe device. I think the received packets are handled by this function https://elixir.free-electrons.com/linux/latest/source/drivers/net/tap.c#L317 which is registered here https://elixir.free-electrons.com/linux/latest/source/drivers/net/macvtap.c#L101 so if you see the macvtap interface drop counters increasing, it must be tap_handle_frame() dropping. If this is true, it means that your problem is macvtap. Cheers, Vincenzo > Joe Buehler > -- Vincenzo Maffione From owner-freebsd-net@freebsd.org Fri Oct 27 16:52:48 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4AEAFE4C059 for ; Fri, 27 Oct 2017 16:52:48 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: from mail-io0-x22a.google.com (mail-io0-x22a.google.com [IPv6:2607:f8b0:4001:c06::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 118DA6FE48 for ; Fri, 27 Oct 2017 16:52:48 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: by mail-io0-x22a.google.com with SMTP id n137so14045299iod.6 for ; Fri, 27 Oct 2017 09:52:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=L7oabb9zIJLF1H/zICGdiNCkFkgdBa6P4e2XhnPzhjE=; b=LoCH9NBG0rQwNxnp+KJXBKgv3f+h8TfvxyrtY4cjz07S+ThBhA55+aspg7Ym3DaTmc n7p4n9a/9lS7N8R2/bVZ77ebRh/C0FJaXbVUfF1vpHAT623mQz190sWGKiw7ECTWWOIT M26Z06/6OR9lTnxTc1kackUzo05y7GIwOt+FeIEWwigdAidodv77wIZWwqVdi3ewHZXe FAwwQhWRhRSUzNGQd2fqRd0yjqhhKPSJx2agOj1o6DIgJPqSV3q38/hOuEDTGNpZ98L8 cyepN0zRV4OObYxABj/6BfMvmbH8r/4gjxbJ1sgyqy3vk7T9lZcZ97+OCYmZbjTVl8lQ Y5nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=L7oabb9zIJLF1H/zICGdiNCkFkgdBa6P4e2XhnPzhjE=; b=P+Me0p6lyVkSAwpd+J0SOswULJMJW13bvyh3v1iqaWrm0UN0mj8HpBh5tDFhNlM4uL VO3hgpxGGItGO5BROmQ5SZmxhnCKTLE1oTcfMOWwxSTczebIGtrHljiZl23RdYVVmDm4 jAy8fPsHwFHKieyPBFjASjnefJVgQNg1qKRt8TvXYEvR5z6EOm8rGA/SBnCGfXqW6Sci m8gTEnKr15n3Vd0Sb+LarRKYSk4rv+W80cJKgJd7ArlRt8cE9DXbZnt0UVs6kmzX6Zf2 aNn7R8O2NprcZu9DUEOgUz67eitoUsNnvE+rRxOfTyMWMJCfw1371tt9Z73Flqxv5tNI DlEg== X-Gm-Message-State: AMCzsaWtFnmWx1NYlvlAZ1p9wGSz533/JP0U0oB6xXiaXN3+OkArOXjh zjlTReYwbno1Pitn7wX4QwCnXPIOSqbYb1ATvgs= X-Google-Smtp-Source: ABhQp+QjjIflu8oF0DrvwN/EAUcJL7ZEsRW2r4P0IXjO8ampGPRlP1DuCau6Ldjh7/5/jtE4Sk+Eefon5ViELuPGDRI= X-Received: by 10.107.141.206 with SMTP id p197mr1445434iod.92.1509123167119; Fri, 27 Oct 2017 09:52:47 -0700 (PDT) MIME-Version: 1.0 Received: by 10.2.176.143 with HTTP; Fri, 27 Oct 2017 09:52:46 -0700 (PDT) In-Reply-To: References: From: Vincenzo Maffione Date: Fri, 27 Oct 2017 18:52:46 +0200 Message-ID: Subject: Re: [netmap] when does a packet in the netmap ring send out exactly To: Xiaoye Sun Cc: FreeBSD Net Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Oct 2017 16:52:48 -0000 Hi, This is actually a limitation of the netmap API: ring->tail is exposed to the user so that it knows it can use the slots in the range "[ring->head..ring->tail[" for new transmissions (note that head is included, tail excluded, to prevent wraparound). However, there is no explicit indication of "up to what slots packets were transmitted". For hw NICs, however, ring->tail is an indication of where transmission was completed. Example: 1) at the beginning ring->tail = ring->head = ring->cur = 0 2) then your program moves head/cur forward: head = cur = 10 3) you call TXSYNC, to submit the packets to the NIC. 4) after the TXSYNC call, is very likely that tail is still 0, i.e. because no transmission has been completed by the NIC (and no interrupt generated). 5) say after 20 us you issue another TXSYNC, and in the meanwhile 6 packets had completed. In this case after TXSYNC you will find tail==5, meaning that packets in the slots 0,1,2,3,4 and 5 have been completed. Note that also the slot pointed by tail has been completed. But you are right that there is no way to receive completion notification if the queue is not full. You must use TXSYNC to check (by sleeping or busy wait) when tail moves forward. Cheers, Vincenzo 2017-10-27 3:06 GMT+02:00 Xiaoye Sun : > Hi > > I write a netmap program that sends packets to the network. my program > uses one netmap ring and fills the ring slots with packets. > My program needs to do something (action A) after a particular packet > (packet P) in the ring slot is sent to the network. so the program tracks > the position of the tail point and checks if the tail point has moved > across the slot I used to put that packet P. > However, I found that the tail pointer may not move forward even seconds > after the receiver side got packet P. > Sometimes the tail pointer never moves forward until the TX ring is full. > I try ioctl(NIOCTXSYNC), however, it cannot 100% solve the problem. > > My question is that is there a way to make the TX ring empty as early as > possible so that I can know when my packet is sent out. or is there another > way to know when the packet in the slot is sent to the network/NIC physical > queue? > > I am using Linux 3.16.0-4-amd64. > > Thanks! > > Best, > Xiaoye > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > -- Vincenzo Maffione From owner-freebsd-net@freebsd.org Fri Oct 27 20:34:36 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0958CE508A9 for ; Fri, 27 Oct 2017 20:34:36 +0000 (UTC) (envelope-from aspam@cox.net) Received: from fed1rmfepi106.cox.net (fed1rmfepi106.cox.net [68.230.241.137]) by mx1.freebsd.org (Postfix) with ESMTP id D834A767DA for ; Fri, 27 Oct 2017 20:34:35 +0000 (UTC) (envelope-from aspam@cox.net) Received: from eastrmimpo305.cox.net ([68.230.241.237]) by eastrmfepo103.cox.net (InterMail vM.8.01.05.28 201-2260-151-171-20160122) with ESMTP id <20171027203411.YOXV4084.eastrmfepo103.cox.net@eastrmimpo305.cox.net> for ; Fri, 27 Oct 2017 16:34:11 -0400 Received: from thunder.sweets ([68.100.138.62]) by eastrmimpo305.cox.net with cox id SkaB1w0091LxgH801kaB5X; Fri, 27 Oct 2017 16:34:11 -0400 X-CT-Class: Clean X-CT-Score: 0.00 X-CT-RefID: str=0001.0A020201.59F39843.00AA, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 X-CT-Spam: 0 X-Authority-Analysis: v=2.2 cv=ENB26xRC c=1 sm=1 tr=0 a=3mkzfl4ircflX6G+lDqBYw==:117 a=3mkzfl4ircflX6G+lDqBYw==:17 a=IkcTkHD0fZMA:10 a=x7bEGLp0ZPQA:10 a=02M-m0pO-4AA:10 a=kFlFZXso1ko05N4mdaoA:9 a=QEXdDO2ut3YA:10 X-CM-Score: 0.00 Authentication-Results: cox.net; none Received: from [10.10.10.15] (thunder.sweets [10.10.10.15]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by thunder.sweets (Postfix) with ESMTP id 92DAC113FC; Fri, 27 Oct 2017 16:34:10 -0400 (EDT) Message-ID: <59F39841.1070601@cox.net> Date: Fri, 27 Oct 2017 16:34:09 -0400 From: Joe Buehler User-Agent: Thunderbird 1.5.0.12 (X11/20120201) MIME-Version: 1.0 To: Vincenzo Maffione CC: FreeBSD Net Subject: Re: virtio_net / netmap RX dropping frames References: <59F0FBEE.6030008@cox.net> <59F1F1BA.3020608@cox.net> <59F214F6.9060905@cox.net> <59F223C3.5040907@cox.net> In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Oct 2017 20:34:36 -0000 Vincenzo Maffione wrote: > You need to figure out who is dropping the RX packets and why. This > seems to happen before the packets really make their way to the > virtio-net RX ring. So it must be some queue overflow in the macvtap or > your ixgbe device. I cranked up the qlen on macvtap1 (100,000) and that fixed the drops but the resulting latency is horrible, increasing with test length, as is the jitter. Given the low RX thread CPU it would appear that there is eomething wrong between macvtap and my code. I will have to chew on this a bit. Joe Buehler