From owner-freebsd-net@FreeBSD.ORG Sat Oct 13 19:24:22 2012 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E0AF9B4F; Sat, 13 Oct 2012 19:24:22 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 7A96A8FC08; Sat, 13 Oct 2012 19:24:22 +0000 (UTC) Received: by mail-vc0-f182.google.com with SMTP id fw7so5907763vcb.13 for ; Sat, 13 Oct 2012 12:24:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=CpnH+4N3bDvB9qaxqYHRCZav7EFQw3dU2IVNNVUC1BQ=; b=uzUocdkiB4kZDPQP57fNluMKKrl1opc9jf0HnQL2zO1abxf+tj9aY07xM1qIHKMkJC O9+Jiq7ZGULIRjd55L1czHOP9BNStojk7CfUfudRMyh2kCXF4nDOofKdiSPZjBpAUrZ9 7vls05AjEFpkrGDRfaJAO3X0qQPiRrzzMZeg8spZyhBz4hOeK8S2ljwE5optdZJzw/fv vADU5GzUxbjX9hqhr4emVxG9JrIfFGDv86tB/Ft+cIN04WyeAV+TkdX9x1aKYZKzAUB/ ZHaSmSqoiNZdT+5QHf7nde7Mj4TT3AAkv/8okuFwQ0n5XM8FxIJ2gThK0/RcsQv+F1RE ejIQ== MIME-Version: 1.0 Received: by 10.58.203.161 with SMTP id kr1mr1863785vec.35.1350156261714; Sat, 13 Oct 2012 12:24:21 -0700 (PDT) Received: by 10.58.68.8 with HTTP; Sat, 13 Oct 2012 12:24:21 -0700 (PDT) In-Reply-To: <20121013182223.GA73341@onelab2.iet.unipi.it> References: <5079A9A1.4070403@FreeBSD.org> <20121013182223.GA73341@onelab2.iet.unipi.it> Date: Sat, 13 Oct 2012 12:24:21 -0700 Message-ID: Subject: Re: ixgbe & if_igb RX ring locking From: Jack Vogel To: Luigi Rizzo Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "Alexander V. Chernikov" , net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Oct 2012 19:24:23 -0000 On Sat, Oct 13, 2012 at 11:22 AM, Luigi Rizzo wrote: > On Sat, Oct 13, 2012 at 09:49:21PM +0400, Alexander V. Chernikov wrote: > > Hello list! > > > > > > Packets receiving code for both ixgbe and if_igb looks like the > following: > > > > > > ixgbe_msix_que > > > > -- ixgbe_rxeof() > > { > > IXGBE_RX_LOCK(rxr); > > while > > { > > get_packet; > > > > -- ixgbe_rx_input() > > { > > ++ IXGBE_RX_UNLOCK(rxr); > > if_input(packet); > > ++ IXGBE_RX_LOCK(rxr); > > } > > > > } > > IXGBE_RX_UNLOCK(rxr); > > } > > > > Lines marked with ++ appeared in r209068(igb) and r217593(ixgbe). > > > > These lines probably do LORs masking (if any) well. > > However, such change introduce quite significant performance drop: > > > > On my routing setup (nearly the same from previous -Intel 10G thread in > > -net) adding lock/unlock causes 2.8MPPS decrease to 2.3MPPS which is > > nearly 20%. > > one option could be (same as it is done in the timer > routine in dummynet) to build a list of all the packets > that need to be sent to if_input(), and then call > if_input with the entire list outside the lock. > > It would be even easier if we modify the various *_input() > routines to handle a list of mbufs instead of just one. > > cheers > luigi > > Very interesting idea Luigi, will have to get that some thought. Jack > So my questions are: > > > > Can any real LORs happen in some complex setup? (I can't imagine any). > > If so: maybe we can somehow avoid/workaround such cases? (and consider > > removing those locks). > > > > > > > > -- > > WBR, Alexander > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >