From owner-freebsd-net@FreeBSD.ORG Tue Sep 25 20:40:59 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8F32110656B8; Tue, 25 Sep 2012 20:40:59 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-qc0-f182.google.com (mail-qc0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 069358FC17; Tue, 25 Sep 2012 20:40:58 +0000 (UTC) Received: by qcsl39 with SMTP id l39so1915916qcs.13 for ; Tue, 25 Sep 2012 13:40:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=bSP4+oPdI1Mz7gQO5mTlrY8RiS51xoi11z48b+zFnyw=; b=ZCoILS9ZLHn6pF6jXdqMhhJuhmvI6DgiyokTdduXa3UZKZQh2566NRqgBogxY3AqJI OP2c+KPbSKNw+j7JF1vZN9vILuI2rZBMya+OJBprKzzXw63cxdjjbiA1+tbSt5fPRDjk xXsxpYy1BSQYRCBlrZ5TQt+1Ru8TQCdIq0Y9Oac6r3rpQ0CDkRMGJE+Pj4cMc7gZ3+yu z/K/UszZxVKAT6cHrOx7D4HZwsCVk8IRBq+yccgu5ddHaTU+5owD3tlIh1ALqbT6uKYA um3HeCp2mpCfs/nj7X01AmcK+U+KeXLCqEyGRqdGTUXFz1iT3MvvpaIjNIOecVzLt23s iwEw== MIME-Version: 1.0 Received: by 10.224.58.134 with SMTP id g6mr31612093qah.40.1348605658193; Tue, 25 Sep 2012 13:40:58 -0700 (PDT) Received: by 10.49.108.193 with HTTP; Tue, 25 Sep 2012 13:40:58 -0700 (PDT) In-Reply-To: References: <201208161736.47250.jhb@freebsd.org> <201208170941.54482.jhb@freebsd.org> Date: Tue, 25 Sep 2012 13:40:58 -0700 Message-ID: From: Jack Vogel To: Vijay Singh Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org, John Baldwin Subject: Re: ixgbe rx & tx locks X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Sep 2012 20:40:59 -0000 Ah yes, at one time I was keeping the RX side lock when calling the stack, but then as I recall that had problems, so the code now releases and reaquires as you can see. It results in some contention but I'm not sure that's avoidable. I've seen some LRO related panics on the 1G driver that may be related to this lock release, or that's one theory I have.. Thanks for the testing Vijay! Jack On Tue, Sep 25, 2012 at 1:19 PM, Vijay Singh wrote: > > Vijay, can you test this to see if it helps with your test case? > > > >> Jack > > John, apologies for the delay. I have some data to share now. > > With your patch, the transmit side lock contention is all gone. > However I still see receive side contention. I have MSI/X enabled, > with one hw queue per-port. > > debug.lock.prof.stats: > lock held_at > contested_from instances > e1b:rx(0) ( sys/dev/ixgbe/ixgbe.c:4314) ( > sys/dev/ixgbe/ixgbe.c:4249) 6814 > e2b:rx(0) ( sys/dev/ixgbe/ixgbe.c:4314) ( > sys/dev/ixgbe/ixgbe.c:4249) 6962 > > These are from: > > 4297 static bool > 4298 ixgbe_rxeof(struct ix_queue *que, int count) > 4299 { > ..... > 4314 IXGBE_RX_LOCK(rxr); > > and > > 4220 static __inline void > 4221 ixgbe_rx_input(struct rx_ring *rxr, struct ifnet *ifp, struct > mbuf *m, u32 ptype) > 4222 { > .... > 4247 IXGBE_RX_UNLOCK(rxr); > 4248 (*ifp->if_input)(ifp, m); > 4249 IXGBE_RX_LOCK(rxr); > > So it seems like the interrupt handler is still contending with a > taskqueue handler on the receive side. > > -vijay > > PS: The interface names are custom. >