From owner-freebsd-net@freebsd.org Fri Oct 27 06:58:59 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1D020E5DB98 for ; Fri, 27 Oct 2017 06:58:59 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: from mail-io0-x230.google.com (mail-io0-x230.google.com [IPv6:2607:f8b0:4001:c06::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E5BA07E4EA for ; Fri, 27 Oct 2017 06:58:58 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: by mail-io0-x230.google.com with SMTP id m16so10650467iod.1 for ; Thu, 26 Oct 2017 23:58:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=lJVHEYE68FzbQYfDxjqTRWRXEL1JHGoAWbXTCKehHs0=; b=WgMYijgPIKWYuiPLVlPh34ZuR6Tc/r8lZvhIGnlf3ZgwoHe7dZHJ5DsmEtBVvki9s/ ZoTt+Q2hU1kqSAaT4J5kee0r6TMLLB4ASgSezcuLqi8KueIB7e5seQMv16/VUd5syw7c +d0TsFR9LTm0GunJbvKTFJ4p1H1BDd+bRYfzCam9fKFyglLSqUbWeDstsjmmsplj8ZCx JjElQhzVQ6o6J/uvdYiwp3NCGv1EpnujbHxveA7DbFstIzFU+YWf+Buzn/wFDvLlE/Qo sXFA29WTylc9Y+GwguRWfE+a/jP9h7DHiq+CyQhWtIgnpz/lmgJhh8q2aJQ7msvMu2OO VVJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=lJVHEYE68FzbQYfDxjqTRWRXEL1JHGoAWbXTCKehHs0=; b=UWcY//SI6ulbdBVVwj6FzLwNUkjxv1gSYVU/9yWvR7fE8Xbo6dSLrUuOccjYDHUFA7 A3VVPORa2O647ptWwwOExcnZ2wBJtBv1oG4sP2I1NMNrxnde8YbEVhRsrh2ffmuH5PU+ guEQDIERIxJZ8YpOTPkhK20rJ/eTs9MIIIwI9VYilZh5rz0ZtxeGh4Sp9KLy73cjmw6m phxGtGpUeKwVNJGSJcza7PZ/gLu/6FvwqUXM3QVbjDFiaR5OQ4xHvjR6LiNsAZCnMfNN jVCkITRbgDEnSEGqPcba7zUjVXSEY4khRhC+mvOdIPFwefxIY3ZmH4I5kqjm/7yp94B2 Pptw== X-Gm-Message-State: AMCzsaUv2Pm6Mj4Q2NtTExU74L9ZcoNvVVB7zg7bhfGT+NPxlmXYE3F+ oNuRGVy7EwYSO9I50dNrXqLZDVtZgRyebFbkDy0= X-Google-Smtp-Source: ABhQp+RLeNdqoRSIG7PRxOh7k464J2ZBkT5Ti5yq2JlUWAA8PUUDArCOgvrEziWSQz415G9TfAJ35PIaBqKUKfhGZAQ= X-Received: by 10.107.151.19 with SMTP id z19mr34281695iod.248.1509087537860; Thu, 26 Oct 2017 23:58:57 -0700 (PDT) MIME-Version: 1.0 Received: by 10.2.176.143 with HTTP; Thu, 26 Oct 2017 23:58:57 -0700 (PDT) In-Reply-To: <59F223C3.5040907@cox.net> References: <59F0FBEE.6030008@cox.net> <59F1F1BA.3020608@cox.net> <59F214F6.9060905@cox.net> <59F223C3.5040907@cox.net> From: Vincenzo Maffione Date: Fri, 27 Oct 2017 08:58:57 +0200 Message-ID: Subject: Re: virtio_net / netmap RX dropping frames To: Joe Buehler Cc: FreeBSD Net Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Oct 2017 06:58:59 -0000 2017-10-26 20:04 GMT+02:00 Joe Buehler : > Vincenzo Maffione wrote: > > > You can have how many threads and processes you want. The constraint is > > that there must not be two threads accessing the same ring at the same > > time. In this case each pktgen is accessing different rings. > > Thanks that was very useful info. I had run this before and got large > frame drop so assumed it was a violation of threading constraints. So > now I can remove a mutex from my app, which has an RX and TX thread. > > Running the two pkt-gen instances, I am getting a lot of RX frame drop. > Based on counters, the TX frames are making it to the external loopback > device, about 1 million frames/sec, which is looping them all back, but > the macvtap interface on the host that feeds into the > ixgbe/vhost/virtio_net/netmap interface in the VM shows about 80-90% of > them as dropped. CPU usage is low in the VM, very roughly 25% for the > TX thread and 5% for the RX thread. The frame rate displayed by pkt-gen > and the CPU displayed by top is bouncing around. > > Yes, mutexes are only needed for concurrent access to the same ring, which is not your case. >From this description it seems that your problem is not netmap or pkt-gen. Your TX pkt-gen transmits 1Mpps with 0.25 CPUs, which is ok giving the limitations of virtio-net. Your RX pkt-gen is not really doing that much work (0.05 CPU), which means that the virtio-net RX ring is almost always empty. You need to figure out who is dropping the RX packets and why. This seems to happen before the packets really make their way to the virtio-net RX ring. So it must be some queue overflow in the macvtap or your ixgbe device. I think the received packets are handled by this function https://elixir.free-electrons.com/linux/latest/source/drivers/net/tap.c#L317 which is registered here https://elixir.free-electrons.com/linux/latest/source/drivers/net/macvtap.c#L101 so if you see the macvtap interface drop counters increasing, it must be tap_handle_frame() dropping. If this is true, it means that your problem is macvtap. Cheers, Vincenzo > Joe Buehler > -- Vincenzo Maffione