From owner-freebsd-smp@FreeBSD.ORG Mon Oct 10 12:40:35 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E2AC416A41F for ; Mon, 10 Oct 2005 12:40:35 +0000 (GMT) (envelope-from rogers@netrunner.nu) Received: from netrunner.nu (netrunner.nu [80.232.32.150]) by mx1.FreeBSD.org (Postfix) with ESMTP id 18D0743D46 for ; Mon, 10 Oct 2005 12:40:34 +0000 (GMT) (envelope-from rogers@netrunner.nu) Received: from netrunner.nu (localhost [127.0.0.1]) by netrunner.nu (8.13.3/8.13.3) with ESMTP id j9ACdbMY082319 for ; Mon, 10 Oct 2005 14:39:37 +0200 (CEST) (envelope-from rogers@netrunner.nu) Received: from localhost (rogers@localhost) by netrunner.nu (8.13.3/8.13.3/Submit) with ESMTP id j9ACdaVY082316 for ; Mon, 10 Oct 2005 14:39:37 +0200 (CEST) (envelope-from rogers@netrunner.nu) Date: Mon, 10 Oct 2005 14:39:36 +0200 (CEST) From: Roger Skjetlein To: freebsd-smp@freebsd.org In-Reply-To: <20051008011047.T58661@netrunner.nu> Message-ID: <20051010143709.R82309@netrunner.nu> References: <20051008011047.T58661@netrunner.nu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Subject: Re: Poor network performance in 5.4 AMD64 Snap 8 with SMP support X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Oct 2005 12:40:36 -0000 Is this really not an aknowledged problem? I have seen several postings on internet regarding this, this should be considered as a severe bug. I have also done some testing on 5.4/Sparc64 and it actually I think I can replicate the same behaviour there with SMP kernel. On Sat, 8 Oct 2005, Roger Skjetlein wrote: > Hi, > > > I have a server with a Supermikro mainboard and two Xeon 3.0Ghz/2MB. When > running kernel with SMP support the network performance is very poor, this is > very visible on NFS traffic. When running UP kernel the performance is as > espected and very good. > > I have tried some different GB nics and the problems is static and > independent of modell/brand. > > I have also disabled as much as hardware possible, no usb, sio and etc. > > Typical indicator is also that the load on the box is skyrocketing when it > starts to serve nfs client, but the cpu load is very load. > > Is there any way around this problem? > > Regards, > RS > > From owner-freebsd-smp@FreeBSD.ORG Mon Oct 10 17:10:13 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 06B3B16A41F for ; Mon, 10 Oct 2005 17:10:12 +0000 (GMT) (envelope-from robert@orcafat.com) Received: from slay.orcafat.com (1-1-6-11a.mfb.mlm.bostream.se [82.182.195.254]) by mx1.FreeBSD.org (Postfix) with ESMTP id 743C343D46 for ; Mon, 10 Oct 2005 17:10:11 +0000 (GMT) (envelope-from robert@orcafat.com) Received: from localhost (localhost [127.0.0.1]) by slay.orcafat.com (Postfix) with ESMTP id 75E5336A for ; Mon, 10 Oct 2005 19:10:09 +0200 (CEST) Received: from slay.orcafat.com ([127.0.0.1]) by localhost (slay [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 22970-09 for ; Mon, 10 Oct 2005 19:10:08 +0200 (CEST) Received: from ht.orcafat.com (ht.orcafat.com [192.168.1.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by slay.orcafat.com (Postfix) with ESMTP id 1F9E5332 for ; Mon, 10 Oct 2005 19:10:08 +0200 (CEST) From: Robert Lindgren To: freebsd-smp@freebsd.org Date: Mon, 10 Oct 2005 19:10:07 +0200 User-Agent: KMail/1.8.2 References: <20051008011047.T58661@netrunner.nu> In-Reply-To: <20051008011047.T58661@netrunner.nu> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200510101910.07457.robert@orcafat.com> X-Virus-Scanned: by amavisd-maia-1.0.0-rc5 (Debian) at orcafat.com Subject: Re: Poor network performance in 5.4 AMD64 Snap 8 with SMP support X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Oct 2005 17:10:13 -0000 On Saturday 08 October 2005 01:11, Roger Skjetlein wrote: > Hi, > > > I have a server with a Supermikro mainboard and two Xeon 3.0Ghz/2MB. When > running kernel with SMP support the network performance is very poor, this > is very visible on NFS traffic. When running UP kernel the performance is > as espected and very good. > > I have tried some different GB nics and the problems is static and > independent of modell/brand. > > I have also disabled as much as hardware possible, no usb, sio and etc. > > Typical indicator is also that the load on the box is skyrocketing when it > starts to serve nfs client, but the cpu load is very load. > > Is there any way around this problem? I had the same problem a couple of weeks ago, but had to settle for UP, since it wasn't possible to get any decent performance with SMP in 5.4. One thing that you could try, which isn't a solution, is to run the interface in promiscuous mode, install trafshow and run it on the interface you have poor performance on. When I did that I was able to get better performance with SMP, in the real world the opposite would be true. BR Robert From owner-freebsd-smp@FreeBSD.ORG Thu Oct 13 09:05:29 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3C93316A41F for ; Thu, 13 Oct 2005 09:05:29 +0000 (GMT) (envelope-from asmrookie@gmail.com) Received: from zproxy.gmail.com (zproxy.gmail.com [64.233.162.192]) by mx1.FreeBSD.org (Postfix) with ESMTP id C790A43D49 for ; Thu, 13 Oct 2005 09:05:28 +0000 (GMT) (envelope-from asmrookie@gmail.com) Received: by zproxy.gmail.com with SMTP id 40so353043nzk for ; Thu, 13 Oct 2005 02:05:28 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=LT6xwGFtBcY2pM3I39+6mKey9HZxpzFRwsVbeAivwLE2NY5HuITLt5IhhMQ7U1dI2Sf0q/AV2JqcMCmXd2i6ODUPY3aKd0LrQ4IjIHBGtxJlUBx6LuVe46uN/D5D9xbnvJVTBB0nbeAvK3bBIWsdDbVGU2hobx/nWeCKO2LszyA= Received: by 10.36.252.29 with SMTP id z29mr629063nzh; Thu, 13 Oct 2005 02:05:27 -0700 (PDT) Received: by 10.36.46.7 with HTTP; Thu, 13 Oct 2005 02:05:27 -0700 (PDT) Message-ID: <3bbf2fe10510130205i2fdd6bfcl@mail.gmail.com> Date: Thu, 13 Oct 2005 11:05:27 +0200 From: rookie To: freebsd-smp@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Subject: struct turnstile and sleep mutex X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rookie@gufi.org List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Oct 2005 09:05:29 -0000 Hi, This would be a silly e-mail but I've not found too much material about tha= t. I was studing at struct turnstile code (kern/subr_turnstile.c, sys/turnstile.h) and there's something is not too clear for me. In particular why owner priority propagation is required in struct turnstile objects? What it does cause? Another question is about sleep mutex implementation; In the comments inside the code I've seen "turnstiles are used to implement not sleepable lock". So why sleep locks are sleeped through turnstiles? They might not use condition variables/sleepqueues? Greeting, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-smp@FreeBSD.ORG Thu Oct 13 17:02:16 2005 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C16B916A423 for ; Thu, 13 Oct 2005 17:02:16 +0000 (GMT) (envelope-from jhb@freebsd.org) Received: from mv.twc.weather.com (mv.twc.weather.com [65.212.71.225]) by mx1.FreeBSD.org (Postfix) with ESMTP id 041CC43D48 for ; Thu, 13 Oct 2005 17:02:15 +0000 (GMT) (envelope-from jhb@freebsd.org) Received: from [10.50.41.234] (Not Verified[10.50.41.234]) by mv.twc.weather.com with NetIQ MailMarshal (v6, 0, 3, 8) id ; Thu, 13 Oct 2005 13:18:37 -0400 From: John Baldwin To: freebsd-smp@freebsd.org, rookie@gufi.org Date: Thu, 13 Oct 2005 11:40:41 -0400 User-Agent: KMail/1.8.2 References: <3bbf2fe10510130205i2fdd6bfcl@mail.gmail.com> In-Reply-To: <3bbf2fe10510130205i2fdd6bfcl@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200510131140.42796.jhb@freebsd.org> Cc: Subject: Re: struct turnstile and sleep mutex X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Oct 2005 17:02:16 -0000 On Thursday 13 October 2005 05:05 am, rookie wrote: > Hi, > This would be a silly e-mail but I've not found too much material about > that. I was studing at struct turnstile code (kern/subr_turnstile.c, > sys/turnstile.h) and there's something is not too clear for me. In > particular why owner priority propagation is required in struct > turnstile objects? What it does cause? Let's say Thread A is a high priority thread and blocks on lock K. The K lock is owned by Thread B. Thread A now cannot run until Thread B runs and releases the lock. If Thread B has a low priority, then it may take a while before it runs, and other threads with priorities less than A but higher than B will run before B and thus effectively starve A. What priority propagation (AKA priority lending) does is lending A's priority to B until B releases K enabling A to run. > Another question is about sleep mutex implementation; In the comments > inside the code I've seen "turnstiles are used to implement not > sleepable lock". So why sleep locks are sleeped through turnstiles? > They might not use condition variables/sleepqueues? sleepqueues don't do priority propagation. sleep queues are generally used to wait for an async event that may or may not come, such as a top-half thread waiting in a driver for an interrupt to come in with data. Mutex waits are not indeterminate, however, as we don't let threads wait on async events (sleep queues) while holding mutexes, thus mutex waits will eventually be terminated barring someone getting stuck in an infinite loop while holding a mutex. For this reason, mutexes use a separate thread queue mechanism that does priority propagation. Priority propogation doesn't make sense for sleep queues since you can't identify an owner to lend priority to when you block. The async even could be triggered from anywhere. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org