From owner-freebsd-smp@FreeBSD.ORG Wed Nov 26 09:32:45 2008 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A595C1065676 for ; Wed, 26 Nov 2008 09:32:45 +0000 (UTC) (envelope-from archimedes.gaviola@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.176]) by mx1.freebsd.org (Postfix) with ESMTP id 7257D8FC20 for ; Wed, 26 Nov 2008 09:32:45 +0000 (UTC) (envelope-from archimedes.gaviola@gmail.com) Received: by wa-out-1112.google.com with SMTP id m34so187913wag.27 for ; Wed, 26 Nov 2008 01:32:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=L6o8jpKrHwsIenYEjlY2tQopcHJ7o7rSwk0EHHUjtiA=; b=AwTstYI7APeTMdgjUZbpbZLUJNLwdUgLydYBDhEv7HTnHZOy3D1C426IewB07/Qj/p cN2+DadZ0D4hCbwFQUbfzkVKK4V632IYkAIq5vtBdtNOadO9JybkfmX/6k7TdGg1ZTba NX/ztHzGr2BJYB5mzksUdX+opr/rk8l9QCbGg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=LkOz6fiZPpmidV0LzQ9Sdc1C2cvxw4TLaTZIUXcRpu0ZvBdUlQCgzKEjWKJrbiJLT3 AOkYIMt+bWU/nX3GyVQGxVqmMlg9bKZuF9aavXS85el+/ST6AtubNFre1OxKdy5AXiL8 X9nlYF2+ovk5zWJflVb9sxnhHguStpREfznVo= Received: by 10.114.169.2 with SMTP id r2mr3223857wae.161.1227691964491; Wed, 26 Nov 2008 01:32:44 -0800 (PST) Received: by 10.115.78.8 with HTTP; Wed, 26 Nov 2008 01:32:44 -0800 (PST) Message-ID: <42e3d810811260132l53027184s102e8a5e3b70dfb2@mail.gmail.com> Date: Wed, 26 Nov 2008 17:32:44 +0800 From: "Archimedes Gaviola" To: ivoras@freebsd.org, "John Baldwin" In-Reply-To: <200811171609.54527.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <42e3d810811100033w172e90dbl209ecbab640cc24f@mail.gmail.com> <42e3d810811170311uddc77daj176bc285722a0c8@mail.gmail.com> <42e3d810811170336rf0a0357sf32035e8bd1489e9@mail.gmail.com> <200811171609.54527.jhb@freebsd.org> Cc: freebsd-smp@freebsd.org Subject: Re: CPU affinity with ULE scheduler X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Nov 2008 09:32:45 -0000 > In both cases the code path a packet must go through is very similar: it > has to be received, then processed through firewalls and network stack > code, then delivered to application(s), so it's a serial process. There > are things that could be better parallelized in the stack and people are > working on them, but they will not be finished any time soon. Ah okay so the project is moving towards network stack parallelism. What is the benefit of parallelized network stack in comparison to the current serialized network stack? Is there any known issues with serialized network stack dealing with multiple CPUs? If it has, in what aspect, components or subsystem of the operating system? With network stack parallelism, what are the necessary changes of the operating system? How should be the network processing be optimized with parallelized network stack? I have gone through a technical paper in the Internet about evaluation on network stack parallelism strategies for modern operating system http://www.cs.rice.edu/CS/Architecture/docs/willmann-usenix06.pdf which described about approaches in implementing parallelized network stack in which also described FreeBSD were used as the prototype of the different approaches, from here I want to know what approach does FreeBSD is implementing, is it message-based parallelism or connection-based parallelism? Thanks, Archimedes From owner-freebsd-smp@FreeBSD.ORG Wed Nov 26 11:18:45 2008 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C61B106564A for ; Wed, 26 Nov 2008 11:18:45 +0000 (UTC) (envelope-from archimedes.gaviola@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.178]) by mx1.freebsd.org (Postfix) with ESMTP id C59998FC1B for ; Wed, 26 Nov 2008 11:18:44 +0000 (UTC) (envelope-from archimedes.gaviola@gmail.com) Received: by wa-out-1112.google.com with SMTP id m34so206349wag.27 for ; Wed, 26 Nov 2008 03:18:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=NlAg08kU/ALLkJNNl9Dj2HMduJeTPxpfNw2nb+u8nWs=; b=w4/4qhyCJIfEGaBc+/D7J/a0L/8lvfB7vbiYkVjAcI1IdnqUIRMLCkeemqSqUSocpE jVBHmvZh8ZaCOKUQu7gyuiXJlQNa7ui4mm28061HZMgNDWXzGLm03gHiebbCUEk4sOY5 hzt4tDzmdcC3rCvbcim1VrRrqmdatmG0VowyY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=vNOMdIH3SJnZOxN8ZhaI0Tjo1gDVceff9wcrF14i1n1CqmIVxttn44bvpaVaIybb+v aNflarCNCsfwgF2jIXaGaC3gHXqIzpOin0j9Ew/8aDFxPuWq1NUc9/3uEghiiRE6tejZ rYbpxzKQiavYgp+/TnMdEFlBlqeVwI+WmlW/g= Received: by 10.115.46.10 with SMTP id y10mr3277276waj.182.1227698324246; Wed, 26 Nov 2008 03:18:44 -0800 (PST) Received: by 10.115.78.8 with HTTP; Wed, 26 Nov 2008 03:18:43 -0800 (PST) Message-ID: <42e3d810811260318j2656ac57k465c56d1c2b0dcf2@mail.gmail.com> Date: Wed, 26 Nov 2008 19:18:43 +0800 From: "Archimedes Gaviola" To: ivoras@freebsd.org, "John Baldwin" In-Reply-To: <200811171609.54527.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_14844_13537311.1227698324243" References: <42e3d810811100033w172e90dbl209ecbab640cc24f@mail.gmail.com> <42e3d810811170311uddc77daj176bc285722a0c8@mail.gmail.com> <42e3d810811170336rf0a0357sf32035e8bd1489e9@mail.gmail.com> <200811171609.54527.jhb@freebsd.org> X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-smp@freebsd.org Subject: Re: CPU affinity with ULE scheduler X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Nov 2008 11:18:45 -0000 ------=_Part_14844_13537311.1227698324243 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline > Is there a tool that can we used to trace > this process just to be able to know which part of the kernel internal > is doing the bottleneck especially when net.isr.direct=1? By the way > with device polling enabled, the system experienced packet errors and > the interface throughput is worst, so I avoid using it though. > Since I was really looking for a tool to be able to know how packets are being processed from the interface and up to the network stack and applications, but I haven't found any tool for my concern. What I have found is the LOCK_PROFILING tool. Although I'm sure that this really not answer my concern but I just tried because I need to know something about locks which FreeBSD is using with. Some people consider that there's a lot of factors and variables with regards to network performance in FreeBSD, so I got a try on this tool. I also get valuable info from this link http://markmail.org/message/3uqxi4pipvvoy6jx#query:lock%20profiling%20freebsd+page:1+mid:ymqgrxqf4min54zd+state:results. Instead of the IBM machine with Broadcom NICs, I use another machine with 4 x Quad-Core AMD64 with still Broadcom NICs on FreeBSD-7.1 BETA2. I took data results with traffic and without traffic. With traffic, I use both TCP and UDP protocols in bombarding traffic. UDP for upload and TCP for download in a back-to-back setup. What I have found is that there's a high wait_total on some of the following when there's traffic: max total wait_total count avg wait_avg cnt_hold cnt_lock name 517 24761291 6165864 4460995 5 1 552124 1558183 net/route.c:293 (sleep mutex:radix node head) 277 1427082 140797 354220 4 0 14476 20674 amd64/amd64/io_apic.c:212 (spin mutex:icu) 33 25275 20744 5401 4 3 0 5400 amd64/amd64/mp_machdep.c:974 (spin mutex:sched lock 4) 17283 3346679 104214 107262 31 0 4545 4072 kern/kern_sysctl.c:1334 (sleep mutex:Giant) 257 28599 386 1302 21 0 35 30 vm/vm_fault.c:667 (sleep mutex:vm object) 282 2821743 2673 977635 2 0 926 552 net/if_ethersubr.c:405 (sleep mutex:bce1) 22 743637 157239 256274 2 0 5304 48357 dev/random/randomdev_soft.c:308 (spin mutex:entropy harvest mutex) 301 16301894 881827 1255534 12 0 241491 45973 dev/bce/if_bce.c:5016 (sleep mutex:bce0) 273 1228787 55458 103863 11 0 3733 4736 kern/subr_sleepqueue.c:232 (spin mutex:sleepq chain) 624 4682305 1339783 1251253 3 1 32664 254211 dev/bce/if_bce.c:4320 (sleep mutex:bce1) With lock profiling, how do we know that a certain kernel structure or function is causing a contention? I only have little knowledge about mutex, can someone elaborate on these especially sleep and spin mutex? Unfortunately due to the log result is too big for the mailing list then I only attached the complete log in compressed format. Thanks, Archimedes ------=_Part_14844_13537311.1227698324243-- From owner-freebsd-smp@FreeBSD.ORG Wed Nov 26 13:55:28 2008 Return-Path: Delivered-To: smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8A9E1106564A for ; Wed, 26 Nov 2008 13:55:28 +0000 (UTC) (envelope-from apache@vulcan.highspd.net) Received: from vulcan.highspd.net (vulcan.highspd.net [66.199.31.170]) by mx1.freebsd.org (Postfix) with ESMTP id 335F18FC1C for ; Wed, 26 Nov 2008 13:55:28 +0000 (UTC) (envelope-from apache@vulcan.highspd.net) Received: from vulcan.highspd.net (vulcan.highspd.net [127.0.0.1]) by vulcan.highspd.net (8.12.11/8.12.11) with ESMTP id mAQDtQvW017311 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 26 Nov 2008 07:55:27 -0600 Received: (from apache@localhost) by vulcan.highspd.net (8.12.11/8.12.11/Submit) id mAQDtQXk017310; Wed, 26 Nov 2008 07:55:26 -0600 Date: Wed, 26 Nov 2008 07:55:26 -0600 Message-Id: <200811261355.mAQDtQXk017310@vulcan.highspd.net> To: smp@freebsd.org From: "Col. Salva Taha" MIME-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: 8bit Cc: Subject: PROPERTIES. X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: salvataha1@live.co.za List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Nov 2008 13:55:28 -0000 Good Day, I wish to introduce myself to you.I am Col.Salva Taha a top Sudanese Goverment official who opposed the war in Dafur in my country Sudan.Due to my oppostion to the war,the goverment of my country has been persecuting me.Consequently my wife,children and I managed to enter a red cross airplane that was evacuating foreigners and we are presently in Cape Town,South Africa. We wish to invest in properties in your country with your assistance and cooperation.If you are in a good position to help my family, please send an email to the email address below indicating your desire to help my family invest the funds in your country and beyond. I await your email. best regards. God bless, Col. Salva Taha Email:salvataha1@live.co.za From owner-freebsd-smp@FreeBSD.ORG Wed Nov 26 14:41:48 2008 Return-Path: Delivered-To: smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0465C1065675 for ; Wed, 26 Nov 2008 14:41:48 +0000 (UTC) (envelope-from apache@vulcan.highspd.net) Received: from vulcan.highspd.net (vulcan.highspd.net [66.199.31.170]) by mx1.freebsd.org (Postfix) with ESMTP id A38E78FC0A for ; Wed, 26 Nov 2008 14:41:47 +0000 (UTC) (envelope-from apache@vulcan.highspd.net) Received: from vulcan.highspd.net (vulcan.highspd.net [127.0.0.1]) by vulcan.highspd.net (8.12.11/8.12.11) with ESMTP id mAQDeb0A013477 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 26 Nov 2008 07:40:37 -0600 Received: (from apache@localhost) by vulcan.highspd.net (8.12.11/8.12.11/Submit) id mAQDea6q013474; Wed, 26 Nov 2008 07:40:36 -0600 Date: Wed, 26 Nov 2008 07:40:36 -0600 Message-Id: <200811261340.mAQDea6q013474@vulcan.highspd.net> To: smp@freebsd.org From: "Col. Salva Taha" MIME-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: 8bit Cc: Subject: PROPERTIES. X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: salvataha1@live.co.za List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Nov 2008 14:41:48 -0000 Good Day, I wish to introduce myself to you.I am Col.Salva Taha a top Sudanese Goverment official who opposed the war in Dafur in my country Sudan.Due to my oppostion to the war,the goverment of my country has been persecuting me.Consequently my wife,children and I managed to enter a red cross airplane that was evacuating foreigners and we are presently in Cape Town,South Africa. We wish to invest in properties in your country with your assistance and cooperation.If you are in a good position to help my family, please send an email to the email address below indicating your desire to help my family invest the funds in your country and beyond. I await your email. best regards. God bless, Col. Salva Taha Email:salvataha1@live.co.za