Date: Mon, 27 Sep 2010 19:35:55 +0400 From: Artemiev Igor <ai@kliksys.ru> To: freebsd-current@freebsd.org Subject: Re: netisr software flowid Message-ID: <20100927153555.GA9200@two.kliksys.ru> In-Reply-To: <alpine.BSF.2.00.1009270947110.82524@fledge.watson.org> References: <20100926235313.GA4848@two.kliksys.ru> <alpine.BSF.2.00.1009270947110.82524@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Sep 27, 2010 at 09:52:21AM +0100, Robert Watson wrote: > One reason I haven't merged the earlier patch is that many high-performance > 10gbps (and even 1gbps) cards now support multiple input queues in hardware, > meaning that they have already done the work distribution by the time the > packets get to the OS. This makes the work distribution choice quite a bit > harder: has a packet already been adequately balanced, or is further > rebalancing required -- and of so, an equal distribution as selected in that > patch might not generate well-balanced CPU load. > > Using just the RSS hash to distribute work, and single-queue input, I am able > to get doubled end-host TCP performance with highly concurrent connections at > 10gbps, which is a useful result. I have high on my todo list to get the > patch you referenced into the mix as well and see how much the software > distrbiution hurts/helps... Thanks for explanation. > Since you've done some measurement, what was the throughput on that system > without the patch applied, and how many cores? The server has four cores. Topology: <groups> <group level="1" cache-level="0"> <cpu count="4" mask="0xf">0, 1, 2, 3</cpu> <children> <group level="3" cache-level="2"> <cpu count="4" mask="0xf">0, 1, 2, 3</cpu> </group> </children> </group> </groups> Without patch i have only one netisr thread utilization with 100% cpu load and ~90% packets drop at max 80-90Kpps. The throughput oscillated from 2MB/s to 30MB/s. Cores 0,2,3 - netisr with cpu binding Core 1 - irq256 (bge0) bind via cpuset(1) P.S.: bge(4) patched for agressive interrupt moderation. Without this i have 11K+ int/sec and ~99% cpu usage only in the interrupt handling.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100927153555.GA9200>