From owner-freebsd-net@FreeBSD.ORG Thu Oct 18 18:09:17 2012 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3C4B06DA; Thu, 18 Oct 2012 18:09:17 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id C52A18FC0A; Thu, 18 Oct 2012 18:09:16 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id v11so11593029vbm.13 for ; Thu, 18 Oct 2012 11:09:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=6xBmignxqv8e3WBsSItXa3RxiYoe1WRrJZ/N6CKnPh8=; b=K1V3I5qLhH+QphFD5pRElKzR4kE1ApJgsWEi6ap6vVIJtJbdh3AOAm1HrysxL65y9k Ya61cXCXaV+8GXg/W+1xRnrxyJeNOvv8st6kyynPZRVBZbXcRwXVnJnptYf14IxjuXbD ZiPPDUmmnXqhu4lZOHCCBQQBBRJ9tLYcQv8pGJPUgJ9pJaWk9keuhd3QWpdGRnw2lkMH g2R6409yOY9pQmFxjeK7nLmmx3vcUVwYDrFtY8K9BOL7nuesl9nrmrbCC9pMOHDIJHvW vAdzGew8OGbuMxTmbOmRtfpdU+SRJGHEoFYJmBmiF684fa87yLBj0QIQQqtoiXdiSQX2 +RVg== MIME-Version: 1.0 Received: by 10.52.34.37 with SMTP id w5mr13188544vdi.86.1350583756089; Thu, 18 Oct 2012 11:09:16 -0700 (PDT) Received: by 10.58.68.8 with HTTP; Thu, 18 Oct 2012 11:09:16 -0700 (PDT) In-Reply-To: <5080020E.1010603@networx.ch> References: <5079A9A1.4070403@FreeBSD.org> <20121013182223.GA73341@onelab2.iet.unipi.it> <5080020E.1010603@networx.ch> Date: Thu, 18 Oct 2012 11:09:16 -0700 Message-ID: Subject: Re: ixgbe & if_igb RX ring locking From: Jack Vogel To: Andre Oppermann Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "Alexander V. Chernikov" , Luigi Rizzo , net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 18:09:17 -0000 On Thu, Oct 18, 2012 at 6:20 AM, Andre Oppermann wrote: > On 13.10.2012 20:22, Luigi Rizzo wrote: > >> On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote: >> >>> Hello list! >>> >>> >>> Packets receiving code for both ixgbe and if_igb looks like the >>> following: >>> >>> >>> ixgbe_msix_que >>> >>> -- ixgbe_rxeof() >>> { >>> IXGBE_RX_LOCK(rxr); >>> while >>> { >>> get_packet; >>> >>> -- ixgbe_rx_input() >>> { >>> ++ IXGBE_RX_UNLOCK(rxr); >>> if_input(packet); >>> ++ IXGBE_RX_LOCK(rxr); >>> } >>> >>> } >>> IXGBE_RX_UNLOCK(rxr); >>> } >>> >>> Lines marked with ++ appeared in r209068(igb) and r217593(ixgbe). >>> >>> These lines probably do LORs masking (if any) well. >>> However, such change introduce quite significant performance drop: >>> >>> On my routing setup (nearly the same from previous -Intel 10G thread in >>> -net) adding lock/unlock causes 2.8MPPS decrease to 2.3MPPS which is >>> nearly 20%. >>> >> >> one option could be (same as it is done in the timer >> routine in dummynet) to build a list of all the packets >> that need to be sent to if_input(), and then call >> if_input with the entire list outside the lock. >> >> It would be even easier if we modify the various *_input() >> routines to handle a list of mbufs instead of just one. >> > > Not really. You'd just run into tons of layering complexity. > Somewhere the decomposition and serialization has to be done. > > Perhaps the right place is to dequeue a batch of packets from > the HW ring and then have a task/thread send it up the stack > one by one. > I was thinking about how to code this, something like what I did with the refresh routine, in any case I will experiment with it. Jack