From owner-freebsd-net@FreeBSD.ORG Mon Aug 13 22:17:55 2012 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A23011065672 for ; Mon, 13 Aug 2012 22:17:55 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 506338FC0C for ; Mon, 13 Aug 2012 22:17:54 +0000 (UTC) Received: by vbmv11 with SMTP id v11so5263069vbm.13 for ; Mon, 13 Aug 2012 15:17:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=DyxgngiNSQ+qo8a4Fo1sDGYM+ZhYIm/uNOHWjA5o53E=; b=rQbQlaWSZisix2/tSONH4P9d643zD5bNmonp2REaCy7k5kzY3iSQMr7yDEVs8bj8Sj XcGgYZB0dYQDHUEaPpQBKf9fzpvHyubn3JYX2sye3Bf+IO4hrAX7Nh9PYi2MxrCGSA8d om5zJ5l/bvFR4TW9Lv+zHEayx4d0OQB+7ci2eYevwkmVhxsAq1hJK3gyP0V+vChUyFH4 CxuwnPTbLOBvc2MEyemddBtMqi4JUyP7qyI/WRFCGdd/1GqhrniDGq2y0bw1qxNwTbOu ixaEfoPXNvubxwcZyRkG85q3QkW1d4pjwuQTpfeEe0Ci99BzFj3qkr6OGLh12JTlB2M4 X4GA== MIME-Version: 1.0 Received: by 10.52.37.193 with SMTP id a1mr8011397vdk.23.1344896273994; Mon, 13 Aug 2012 15:17:53 -0700 (PDT) Received: by 10.58.187.134 with HTTP; Mon, 13 Aug 2012 15:17:53 -0700 (PDT) In-Reply-To: References: Date: Mon, 13 Aug 2012 15:17:53 -0700 Message-ID: From: Jack Vogel To: Vijay Singh Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: net@freebsd.org Subject: Re: ixgbe rx & tx locks X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Aug 2012 22:17:55 -0000 After looking at the code again I think this is just what's happening, mq_start will schedule the task when it can't get the lock (due to the interrupt already holding it presumeably), so you get the tasklet code in contention with the interrupt. Anyone with a clever notion of how to do things better? Jack On Mon, Aug 13, 2012 at 2:40 PM, Jack Vogel wrote: > Hmmm, this is odd, because the interrupt vector is not being re-enabled > unless > you are not scheduling the task, and when you are the interrupt isn't > enabled til > the end.... Something funny going on. > > You could get contention due to mq_start however, you sure about where its > coming from? > > Jack > > > > > On Mon, Aug 13, 2012 at 2:15 PM, Vijay Singh wrote: > >> Folks, I've been looking into lock contention on the ixgbe rx & tx locks. >> >> I have collected this data: >> >> debug.lock.prof.stats: >> max wait_max total wait_total count avg wait_avg >> cnt_hold cnt_lock name >> 263 304 1329357 858873 322010 4 2 >> 0 45210 sys/dev/ixgbe/ixgbe.c:1349 (sleep mutex:e2a:tx(0)) >> 415 189 14314889 62933 739895 19 0 >> 0 13694 sys/dev/ixgbe/ixgbe.c:1445 (sleep mutex:e2a:tx(0)) >> >> So the lock contention is coming from lock acquisitions in >> ixgbe_handle_que() and ixgbe_msix_que(). >> >> I am unable to understand why, since I would not expect the deferred >> task to execute in parallel with the msix interrupt handler. >> >> rx is similar: >> >> debug.lock.prof.stats: >> max wait_max total wait_total count avg wait_avg >> cnt_hold cnt_lock name >> 88 893 672058 269855 821847 0 0 >> 0 753 sys/dev/ixgbe/ixgbe.c:4211 (sleep mutex:e6b:rx(0)) >> 893 25 3994289 1937 1093238 3 0 >> 0 561 sys/dev/ixgbe/ixgbe.c:4276 (sleep mutex:e6b:rx(0)) >> >> Any thoughts? >> >> -vijay >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > >