From owner-freebsd-net@FreeBSD.ORG Tue Oct 28 18:10:48 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0D2A71065682 for ; Tue, 28 Oct 2008 18:10:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id 8AAA18FC0C for ; Tue, 28 Oct 2008 18:10:47 +0000 (UTC) (envelope-from mav@FreeBSD.org) X-Spam-Flag: SKIP X-Spam-Yversion: Spamooborona-2.1.0 Received: from [212.86.226.226] (account mav@alkar.net HELO mavbook.mavhome.dp.ua) by cmail.optima.ua (CommuniGate Pro SMTP 5.2.9) with ESMTPSA id 226355586; Tue, 28 Oct 2008 20:10:46 +0200 Message-ID: <490755A3.4050903@FreeBSD.org> Date: Tue, 28 Oct 2008 20:10:43 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.16 (X11/20080726) MIME-Version: 1.0 To: Bartosz Giza References: <1225203780.00029971.1225190402@10.7.7.3> <49072BC9.4010103@FreeBSD.org> <200810281613.13719.gizmen@blurp.pl> In-Reply-To: <200810281613.13719.gizmen@blurp.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: two NIC on 2 core system (scheduling problem) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Oct 2008 18:10:48 -0000 Bartosz Giza wrote: >> The CPU time you see there includes much more then just a card handling >> itself. It also includes CPU time of the most parts of network stack >> used to process received packet. So if you have NAT, big firewall, >> netgraph or any other CPU-hungry actions done with packets incoming via >> em0 you will see such results. >> Even more interesting is that if bge0 or fxp0 cards will require much >> CPU time to send packet, this time will also be accounted to em0 process. >> :) > > WOW that is weird. Yes i have quite big ipfw firewall and also couple of rules > from pf. So you are saying that whole overhead from firewall is counted to > this em taskq process. This is really strange for somebody who don't know > about this. > So what in case if i would have two em nic's. How would then overhead from > firewalls be counted ? Splited to two taskq processes ? > And another really weird thing is that if i have other card their processing > are counted to tasq process of different NIC. Why is this done in such a way ? There is no dedicated processes in system to handle routing, firewall, netgraph, etc. Many processes would lead to multiple context switches, bigger latencies and reduced performance. Instead most parts of network stack implemented in direct call fashion. So NIC receive interrupt initiates packet handling by doing stack function call, that function calls another and so on until packet will be completely processed and transmitted or queued. There is not so many exception from this rule, for example, dummynet which schedules part of activity to the timer events. So sometimes you still can see some activity from the swi:net, dummynet or some other threads. It also tells not so much about who really consumed the CPU, but more about who initiated that CPU consumption. In case of two em NICs em0 thread will mostly show load produced by em0->em1 traffic processing and em1 - load produced by em1->em0 traffic. -- Alexander Motin