From owner-freebsd-net@FreeBSD.ORG Thu Apr 26 18:14:17 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D572A106564A for ; Thu, 26 Apr 2012 18:14:17 +0000 (UTC) (envelope-from juli@clockworksquid.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 5D3A48FC14 for ; Thu, 26 Apr 2012 18:14:17 +0000 (UTC) Received: by weyt57 with SMTP id t57so1193366wey.13 for ; Thu, 26 Apr 2012 11:14:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :x-gm-message-state; bh=gP9b3XOZj/hvBYqrQv1cvCcxFb+mSlnqtmN3iMZC+Nc=; b=eI5f7WgXenSMVwK0YX5RbfNDlsSLswzSnk2Gl0ewtT4MCKBWolfmgQuugKv7pyiCMo y8lkwv1ZmH/r/yRm2twAS49dyNF8PcMAlB1r3gG3ApLlIO0nppuhqAHMNUyiBB/w2+TW M3NYhIDx4UkiUm1/T+4JS+t1QJs/cN59pDAo8W+D3tVnG1EjNa/Tu7IxfkXXo58HgXVD s0DtmD8IT4s+Oes7hV+neYBCMPjIb+kjBcGkTjXslGSaOgSk05Q5xEjGEM60Pybn+DEG Xf6qURgvBAZddygWK2Ce2guuOhIAxij/l3VmemOURcpt2+HBhRiK/Cn0j9IBjSUnRaQB iMsA== Received: by 10.180.94.161 with SMTP id dd1mr16137444wib.16.1335464056210; Thu, 26 Apr 2012 11:14:16 -0700 (PDT) MIME-Version: 1.0 Sender: juli@clockworksquid.com Received: by 10.216.110.67 with HTTP; Thu, 26 Apr 2012 11:13:55 -0700 (PDT) In-Reply-To: <1335463643.2727.10.camel@powernoodle-l7.corp.yahoo.com> References: <1335463643.2727.10.camel@powernoodle-l7.corp.yahoo.com> From: Juli Mallett Date: Thu, 26 Apr 2012 11:13:55 -0700 X-Google-Sender-Auth: FohBmk8mpRi9NlySaqOpmzmZvKQ Message-ID: To: Sean Bruno Content-Type: text/plain; charset=UTF-8 X-Gm-Message-State: ALoCoQnrTdCz1ZC7wNMdlYbG8176HLNd1ZPmZISmP9vZIxrAmwlITPGIobr3n+X12HnwXN2nTo7P Cc: "freebsd-net@freebsd.org" Subject: Re: igb(4) at peak in big purple X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2012 18:14:17 -0000 On Thu, Apr 26, 2012 at 11:07, Sean Bruno wrote: > I note form top that igb0 queue 0 is always "more busy" than any other > queue, there appears to be a second kernel igb0 "queue" process/thread > that appears to be moderately busy and 3 kernel igb0 "queue" > processes/threads that appear to be doing nothing in particular. Queue splitting in Intel cards is done using a hash of protocol headers, so this is expected behavior. This also helps with TCP and UDP performance, in terms of keeping packets for the same protocol control block on the same core, but for other applications it's not ideal. If your application does not require that kind of locality, there are things that can be done in the driver to make it easier to balance packets between all queues about-evenly. I've also seen it be the case that some cores will use more CPU time for reasons of unusually-slow memory access from that core, i.e. with non-uniform memory, or with some CPU threads, rather than because more packets are going to a queue. It would certainly be very nice at some point for netstat -I to grow awareness of multiple queues (which is a bit non-trivial because of how those statistics are gathered from struct ifnet) so that one could more easily watch in real time the packet rate to each queue. Probably someone with more patience than me could do this using the sysctl nodes the drivers export. Juli.