From owner-freebsd-net@FreeBSD.ORG Fri Apr 26 16:04:22 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 635E4BD2 for ; Fri, 26 Apr 2013 16:04:22 +0000 (UTC) (envelope-from weiler@soe.ucsc.edu) Received: from mail-pa0-f51.google.com (mail-pa0-f51.google.com [209.85.220.51]) by mx1.freebsd.org (Postfix) with ESMTP id 398D5107D for ; Fri, 26 Apr 2013 16:04:22 +0000 (UTC) Received: by mail-pa0-f51.google.com with SMTP id jh10so2570909pab.38 for ; Fri, 26 Apr 2013 09:04:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=x-received:message-id:date:from:user-agent:mime-version:to:cc :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=e2G2ymCvrRHMQBVbT1V7Ak8cR9j7oBLJfLnp+SGp7EQ=; b=hLKASNOxIq4tf0mZTphgCdKAD82LGoI4tXf7/L/nyR8XUZY5PeJ96g4rjfH98PHrJ+ zXm8PgpF293we66fRs7mlf3grfK3eZh1IPriPbzD48FF0PtGCLUNU7vRXG1Rd6RVWnJM rsfpDYJllNdaR3YZ+Rk4/iiNeCG2DUKOvSuhQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:message-id:date:from:user-agent:mime-version:to:cc :subject:references:in-reply-to:content-type :content-transfer-encoding:x-gm-message-state; bh=e2G2ymCvrRHMQBVbT1V7Ak8cR9j7oBLJfLnp+SGp7EQ=; b=QHH7YMQS48umRKskUl4fsEIFF9BGvETKZu31IaWKhANq/NMUvogMY+ZkioAr58RlJb HKq5nNX7CZ++0xqIHFnOBtj2zBoEUx5KqXQjITYQc8eJEMs0H1WXxi8mpJ7MXKg/SlP+ pkTL/NbIagvJlPTG8m5pYJ5hinOw2TwulVigoyeuBnc1la2AJ0cmnI8Jgy/2TYp2mFdh YpuQcNYrMLx+2XUndphVSOCXr+jGPoT18jMBGtbHz+mLXAyHOMVpzi9G6hAoKcvSOW9k g3qbl0bJD/uy5Zc+8cIUsIMwYejpHUrzUNZhV8JUn+Ef1juunpV1SP5vzeCChjVkCS0r h8Mg== X-Received: by 10.66.246.168 with SMTP id xx8mr35315912pac.107.1366992256731; Fri, 26 Apr 2013 09:04:16 -0700 (PDT) Received: from [192.168.0.146] (50-0-69-3.dsl.static.fusionbroadband.com. [50.0.69.3]) by mx.google.com with ESMTPSA id l4sm12245777pbo.6.2013.04.26.09.04.14 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 26 Apr 2013 09:04:15 -0700 (PDT) Message-ID: <517AA57F.2000501@soe.ucsc.edu> Date: Fri, 26 Apr 2013 09:04:15 -0700 From: Erich Weiler User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: Andre Oppermann Subject: Re: pf performance? References: <5176E5C1.9090601@soe.ucsc.edu> <20130426134224.GV76816@FreeBSD.org> <517A93FE.7020209@soe.ucsc.edu> <517AA337.8050505@freebsd.org> In-Reply-To: <517AA337.8050505@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Gm-Message-State: ALoCoQlY7SzOv/69Fdi+PU2OnV3amncRHcEy/8poyOK5K2CMo2xpzI5ETsI1GhcTcwwoAVomsMAX Cc: Paul Tatarsky , freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Apr 2013 16:04:22 -0000 >> But the work pf does would show up in 'system' on top right? So if I >> see all my CPUs tied up 100% >> in 'interrupts' and very little 'system', would it be a reasonable >> assumption to think that if I got >> more CPU cores to handle the interrupts that eventually I would see >> 'system' load increase as the >> interrupt load became faster to be handled? And thus increase my >> bandwidth? > > Having the work of pf show up in 'interrupts' or 'system' depends on the > network driver and how it handles sending packets up the stack. In most > cases drivers deliver packets from interrupt context. Ah, I see. Definitely appears for me in interrupts then. I've got the mxge driver doing the work here. So, given that I can spread out the interrupts to every core (like, pin an interrupt queue to each core), I can have all my cores work on the process. But seeing as though the pf bit is still serialized I'm not sure that I understand how it is serialized when many CPUs are handling interrupts, and hence doing the work of pf as well? Wouldn't that indicate that the work of pf is being handled by many cores, as many cores are handling the interrupts? Or are you saying that pf *is* being handled by many cores, but just in a serialized nature? Like, packet 1 is handled by core 0, then packet 2 is handled by core 1, packet 3 is handled by core 4, etc? Such that even though multiple cores are handling it, they are just doing so serially and not in parallel? And if so, maybe it still helps to have many cores? Thanks for all the excellent info! >> In other words, until I see like 100% system usage in one core, I >> would have room to grow? > > You have room to grow if 'idle' is more than 0% and the interrupts of > the networks cards are running on different cores. If one core gets > all the interrupts a second idle core doesn't get the chance to help > out. IIRC the interrupt allocation to cores is done at interrupt > registration time or driver attach time. It can be re-distributed > at run time on most architecture but I'm not sure we have an easily > accessible API for that. >