From owner-freebsd-performance@FreeBSD.ORG Mon Oct 22 14:40:35 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B456F16A418 for ; Mon, 22 Oct 2007 14:40:35 +0000 (UTC) (envelope-from lex@sandy.ru) Received: from adm.sci-nnov.ru (adm.sci-nnov.ru [195.122.226.2]) by mx1.freebsd.org (Postfix) with ESMTP id 5F80613C48D for ; Mon, 22 Oct 2007 14:40:35 +0000 (UTC) (envelope-from lex@sandy.ru) Received: from vista (rb-vista.sci-nnov.ru [193.125.71.25]) by adm.sci-nnov.ru (Postfix) with ESMTP id B2DC619EA94 for ; Mon, 22 Oct 2007 18:18:44 +0400 (MSD) From: "Alex A. Pavlenko" To: Date: Mon, 22 Oct 2007 18:18:44 +0400 Message-ID: <096201c814b6$7476df50$7cf0f00a@corporate.sandy.ru> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 11 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 Thread-Index: AcgUtnQEdkJWWH7NT4uRMeraobWVUw== Subject: Packet loss problem. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2007 14:40:35 -0000 Hi, I'm not quite sure if it is write list for this question, so please forgive me if it is really so. The background is as follows. Hardware is HP Proliant DL380 G5 server with 2 3Ghz Xeon CPUs and 2 on-board Broadcom NICs(bge). Software - FreeBSD 6.2-STABLE, pf, ipfw+DUMMYNET. The only job for this server is to provide NAT(pf) and traffic-shape(dummynet) services for customers. It works fine while summary packet rate of incoming packets on both interfaces is no more than about 50 Kpps. With higher rates it starts to loose packets, 2% and more depending on incoming network traffic. At this time net.inet.ip.intr_queue_drops counter increases very fast, a few thousands per second. What I can not understand is that at the same time CPU load dedicated to interrupts is about 30%. The rest is dedicated to CPU idle state according to `top` command. I'm not the OS guru so I can interpret statistics above incorrectly, but my assumption is that OS uses only 1 of 4 CPUs cores to process incoming packets, other three cores perform a small amount of other system jobs. So what I would like to ask you is to give the right explanation for such behaviour. Also any suggestions to improve performance would be great since this service is very critical for our company. Thank you very much for any responses. -- Alex. From owner-freebsd-performance@FreeBSD.ORG Tue Oct 23 16:28:52 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B96616A417 for ; Tue, 23 Oct 2007 16:28:52 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.176]) by mx1.freebsd.org (Postfix) with ESMTP id 3607613C481 for ; Tue, 23 Oct 2007 16:28:52 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: by py-out-1112.google.com with SMTP id u77so3410102pyb for ; Tue, 23 Oct 2007 09:28:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:cc:mime-version:content-type:content-transfer-encoding:content-disposition; bh=ypgAIGJkXUYooeaUyvwQq0+jc4Wu2SQkxsOx86NwqlI=; b=J/RDxReKObQ+UD0nYYMZbKG0OfAupJMTk8J/ZlwTngrejJHB1NW+vgAiK3UEnt3+KRBWlhPwYtDRB79ExpMnmSPreZeTO8+/MeFHQlSDjL4H1MD3w9XPtoTIc6BuZ1U8WmQGdGGebgu1B//ZlBi+IBDuSO3ESbpq15nmHgFC7AQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:cc:mime-version:content-type:content-transfer-encoding:content-disposition; b=hFggIHZVEtfhrqtPxIr+jnFB44JJmKDWOspziuBzLVBDFIGi88MCgbqU74GF6+bHndhKaaZbItiNEH3uF4A0P9fq0hhA+ilqa5aH1qnopua4qjDgNjl2HYZeFOua0eoHHrsPQ3YV+Mr78TNH/Xh1Q4Acv0/XL5M9WI0ejCRu1tM= Received: by 10.35.131.13 with SMTP id i13mr7725881pyn.1193155361095; Tue, 23 Oct 2007 09:02:41 -0700 (PDT) Received: by 10.35.117.12 with HTTP; Tue, 23 Oct 2007 09:02:41 -0700 (PDT) Message-ID: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> Date: Tue, 23 Oct 2007 12:02:41 -0400 From: "Josh Carroll" To: freebsd-performance@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Cc: remy.nonnenmacher@activnetworks.com Subject: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: josh.carroll@gmail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2007 16:28:52 -0000 Hello, I posted this to the stable mailing list, as I thought it was pertinent there, but I think it will get better attention here. So I apologize in advance for cross-posting if this is a faux pas. :) Anyway, in summary, ULE is about 5-6 % slower than 4BSD for two workloads that I am sensitive to: building world with -j X, and ffmpeg -threads X. Other benchmarks seem to indicate relatively equal performance between the two. MySQL, on the other hand, is significantly faster in ULE. I'm trying to understand why ffmpeg and buildworld are slower in ULE than 4BSD, since it seems to me that ULE was supposed to be the better scaling scheduler. Here is a link to the original thread on the stable mailing list: http://lists.freebsd.org/pipermail/freebsd-stable/2007-October/037379.html Remy replied with some interesting results for building world between the two schedulers on an 8-way system. It seems that ULE suffers as more threads/processes are thrown at it, at least it appears that way from Remy's data. Does anyone have any additional performance tests I can run that might help indicate where the deficiency is in the ULE scheduler? MySQL performance is excellent, so I'm wondering if it was tuned to that particular workload? I'm not sure if Remy subscribes to this list, so I am CC'ing him. Hope you don't mind Remy :) Regards, Josh From owner-freebsd-performance@FreeBSD.ORG Tue Oct 23 17:49:48 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AAB8A16A421 for ; Tue, 23 Oct 2007 17:49:48 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (pointyhat.freebsd.org [IPv6:2001:4f8:fff6::2b]) by mx1.freebsd.org (Postfix) with ESMTP id D465213C480; Tue, 23 Oct 2007 17:49:47 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <471E343C.2040509@FreeBSD.org> Date: Tue, 23 Oct 2007 19:49:48 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: josh.carroll@gmail.com References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> In-Reply-To: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: remy.nonnenmacher@activnetworks.com, freebsd-performance@freebsd.org Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2007 17:49:48 -0000 Josh Carroll wrote: > Hello, > > I posted this to the stable mailing list, as I thought it was > pertinent there, but I think it will get better attention here. So I > apologize in advance for cross-posting if this is a faux pas. :) > > Anyway, in summary, ULE is about 5-6 % slower than 4BSD for two > workloads that I am sensitive to: building world with -j X, and ffmpeg > -threads X. Other benchmarks seem to indicate relatively equal > performance between the two. MySQL, on the other hand, is > significantly faster in ULE. > > I'm trying to understand why ffmpeg and buildworld are slower in ULE > than 4BSD, since it seems to me that ULE was supposed to be the better > scaling scheduler. > > Here is a link to the original thread on the stable mailing list: > > http://lists.freebsd.org/pipermail/freebsd-stable/2007-October/037379.html > > Remy replied with some interesting results for building world between > the two schedulers on an 8-way system. It seems that ULE suffers as > more threads/processes are thrown at it, at least it appears that way > from Remy's data. > > Does anyone have any additional performance tests I can run that might > help indicate where the deficiency is in the ULE scheduler? MySQL > performance is excellent, so I'm wondering if it was tuned to that > particular workload? > > I'm not sure if Remy subscribes to this list, so I am CC'ing him. Hope > you don't mind Remy :) One major difference is that your workload is 100% user. Also you were reporting ULE had more idle time, which looks like a bug since I would expect it be basically 0% idle on such a workload. Kris From owner-freebsd-performance@FreeBSD.ORG Tue Oct 23 18:13:10 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C5F1316A41B for ; Tue, 23 Oct 2007 18:13:10 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.237]) by mx1.freebsd.org (Postfix) with ESMTP id 975D513C4AA for ; Tue, 23 Oct 2007 18:13:10 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: by nz-out-0506.google.com with SMTP id l8so769009nzf for ; Tue, 23 Oct 2007 11:13:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=f640NNc0/t9bmNbj1uioFWfv/Q1R91bXR1yNn90FCEk=; b=lDHNCOTiAJnFXQPIUdrKrKsnFZZCF4D5TwYea6ZuqA83Ha3f94P6F6F3zRUACdtz6xlu7739Na2MiT3Ww8gO+eLPcQxUaaMPRnfw9YwdJ0K/lAIfHV0ukD/E+sx+nPWn2q6uuNbP04ExIoP+ufUkute4GMv8ElYVucvv7bflfcg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=emyAcrNhTjhkZRypmxGBhcpJ/dQRSzXnqb03emuzzMYTICGjITiczRn3H11+UH/WSUZDV8XT1YVAtNdzPbKRdQRWAMlDUUOkwsAd9ZV9SqL3YS3Jio14WXXdp4Yv/F1VBO9G6dhmMQvQJpHEFuOzNaJ4dpVgzr4zjxXqzYSTJdM= Received: by 10.114.106.1 with SMTP id e1mr3865405wac.1193161665480; Tue, 23 Oct 2007 10:47:45 -0700 (PDT) Received: by 10.114.13.15 with HTTP; Tue, 23 Oct 2007 10:47:45 -0700 (PDT) Message-ID: Date: Tue, 23 Oct 2007 10:47:45 -0700 From: "Kip Macy" To: josh.carroll@gmail.com In-Reply-To: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> Cc: remy.nonnenmacher@activnetworks.com, freebsd-performance@freebsd.org Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2007 18:13:10 -0000 On 10/23/07, Josh Carroll wrote: > Hello, > > I posted this to the stable mailing list, as I thought it was > pertinent there, but I think it will get better attention here. So I > apologize in advance for cross-posting if this is a faux pas. :) > > Anyway, in summary, ULE is about 5-6 % slower than 4BSD for two > workloads that I am sensitive to: building world with -j X, and ffmpeg > -threads X. Other benchmarks seem to indicate relatively equal > performance between the two. MySQL, on the other hand, is > significantly faster in ULE. > > I'm trying to understand why ffmpeg and buildworld are slower in ULE > than 4BSD, since it seems to me that ULE was supposed to be the better > scaling scheduler. > > Here is a link to the original thread on the stable mailing list: > > http://lists.freebsd.org/pipermail/freebsd-stable/2007-October/037379.html > > Remy replied with some interesting results for building world between > the two schedulers on an 8-way system. It seems that ULE suffers as > more threads/processes are thrown at it, at least it appears that way > from Remy's data. > > Does anyone have any additional performance tests I can run that might > help indicate where the deficiency is in the ULE scheduler? MySQL > performance is excellent, so I'm wondering if it was tuned to that > particular workload? > > I'm not sure if Remy subscribes to this list, so I am CC'ing him. Hope > you don't mind Remy :) ULE is tuned towards providing cpu affinity compilation and evidently encoding are workloads that do not benefit from affinity. Before we conclude that it is slower, try building with -j5, -j6, j7. -Kip From owner-freebsd-performance@FreeBSD.ORG Tue Oct 23 19:57:55 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 673EB16A41B for ; Tue, 23 Oct 2007 19:57:55 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.180]) by mx1.freebsd.org (Postfix) with ESMTP id 230F913C4D1 for ; Tue, 23 Oct 2007 19:57:54 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: by py-out-1112.google.com with SMTP id u77so3521389pyb for ; Tue, 23 Oct 2007 12:57:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=LEP/LmJvrIiN3+FsYr0R+4xOdnIsbeQ2oeHgh11fY68=; b=jgdMaYR+gCT+jQrRhKYL3KsngREeK1PU5NSOR1WlXnd/rXLtKTvwGktVrIIYOuRKh/QUlMpxRt7BHaIx11l1Z047eaMvdwPWbyn4GlkaZ0yucXAUaKSYJbCVE612XP5JmTHzUyboi/IYkVY6uMzi9hT3WinsD3z85BFiuuyxDHc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=YOeWxO60vWRCMR/y8qeXucyUh3e+nVxub0fbibuqtgSUdW2kQtvE19b6xxIhqd1ScQpalRELp9MLs3AAYmLcfVZCmhMYlvQj3PvDhCJwGUyTpf8Hfpvovy00MLV3hOO8OI4KdH+pxSRO2wu/4XTCV78qae4goaEljW6qxff2PQA= Received: by 10.35.33.15 with SMTP id l15mr7989556pyj.1193169465108; Tue, 23 Oct 2007 12:57:45 -0700 (PDT) Received: by 10.35.117.12 with HTTP; Tue, 23 Oct 2007 12:57:45 -0700 (PDT) Message-ID: <8cb6106e0710231257k154e9c6ev4b4ba8c3692206fb@mail.gmail.com> Date: Tue, 23 Oct 2007 15:57:45 -0400 From: "Josh Carroll" To: "Kip Macy" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> Cc: freebsd-performance@freebsd.org Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: josh.carroll@gmail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2007 19:57:55 -0000 > ULE is tuned towards providing cpu affinity compilation and evidently > encoding are workloads that do not benefit from affinity. Before we > conclude that it is slower, try building with -j5, -j6, j7. Here are the results of running ffmpeg with 4 through 8 threads on both schedulers: 4 threads 4bsd: 117.21 5 threads 4bsd: 95.75 6 threads 4bsd: 93.10 7 threads 4bsd: 92.19 8 threads 4bsd: 92.38 4 threads ule: 122.19 5 threads ule: 107.26 6 threads ule: 101.40 7 threads ule: 98.72 8 threads ule: 96.38 4 threads difference: 4.25 % 5 threads difference: 12.02 % 6 threads difference: 8.92 % 7 threads difference: 7.08 % 8 threads difference: 4.33 % I'm not sure why the performance differential is not consistent (probably something very technical a scheduler developer could explain) :) Do these results help at all? When running with 9 or more threads, ffmpeg spits out a lot of errors, so 8 was as high as I could go: Error while decoding stream #0.0 [h264 @ 0x264ae180]too many threads [h264 @ 0x264ae180]decode_slice_header error [h264 @ 0x264ae180]no frame! My next step is to run some transcodes with mencoder to see if it has similar performance between the two schedulers. When I have those results, I'll post them to this thread. Thanks for the attention, Josh From owner-freebsd-performance@FreeBSD.ORG Tue Oct 23 20:17:02 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F3D616A46D for ; Tue, 23 Oct 2007 20:17:02 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.225]) by mx1.freebsd.org (Postfix) with ESMTP id 4AFD113C48D for ; Tue, 23 Oct 2007 20:17:00 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: by nz-out-0506.google.com with SMTP id l8so798896nzf for ; Tue, 23 Oct 2007 13:16:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=Qr7yUp6ceCGSiKJR6eDF6KArYDvLf9dUEyDbIoqCHbo=; b=Cre2WSkO0B6Nhni1gcouh3sC2OmC0GAR525Y5PPzXQHg6emlfys3Dg+QZfx5NUxh5olc1tO5BGp8Wro8OGbTflOBeH+SHoFHR6xy8cibF7VIhreC8If6sKp3/TRz70qXeQuWOBLcntdI/cTg7KOe5kXM4tuLStlr7FyobEfV3cI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=HpmHYnhoav2bMh0IJ1fNJaYYAREbrKW/somG+GRSj8G9pySURRVDt/BNUubmgmSBPgLI6gw9OgZq+KltfYu0EX3Evbtl/OjeACFFAS0L3s7BprEIJP45ohxGkjcb7ExzSECuXx8LVE8mYtBW/dMwJ9ImHqvIIJ2R+NX+sudST6o= Received: by 10.35.40.10 with SMTP id s10mr8028698pyj.1193170614590; Tue, 23 Oct 2007 13:16:54 -0700 (PDT) Received: by 10.35.117.12 with HTTP; Tue, 23 Oct 2007 13:16:54 -0700 (PDT) Message-ID: <8cb6106e0710231316w48c2ce59w5df70103771642a1@mail.gmail.com> Date: Tue, 23 Oct 2007 16:16:54 -0400 From: "Josh Carroll" To: "Josh Paetzel" In-Reply-To: <200710231509.03771.josh@tcbug.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <8cb6106e0710231257k154e9c6ev4b4ba8c3692206fb@mail.gmail.com> <200710231509.03771.josh@tcbug.org> Cc: Kip Macy , freebsd-performance@freebsd.org Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: josh.carroll@gmail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2007 20:17:02 -0000 > Just curious, but are these results obtained while you are > overclocking your 2.4ghz CPU to 3.4ghz? That might be a useful > datapoint. Yes they are with the CPU overclocked. I have verified the results when not overclocked as well (running at stock). > It also might be useful to know what sort of disks you are using. > SATA is notoriously bad at parallel access, and compiling is of > course horribly disk bound to begin with. I'm sure disk I/O is a factor here. ULE is supposed to provide better interactiveness during high load (and I/O load), right? Perhaps the scheduler is being too liberal with time slices for I/O? Josh From owner-freebsd-performance@FreeBSD.ORG Tue Oct 23 21:02:42 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D80B016A419 for ; Tue, 23 Oct 2007 21:02:42 +0000 (UTC) (envelope-from josh@tcbug.org) Received: from conn-smtp.mc.mpls.visi.com (conn.mc.mpls.visi.com [208.42.156.2]) by mx1.freebsd.org (Postfix) with ESMTP id B7BA313C4A3 for ; Tue, 23 Oct 2007 21:02:42 +0000 (UTC) (envelope-from josh@tcbug.org) Received: from mail.tcbug.org (mail.tcbug.org [208.42.70.163]) by conn-smtp.mc.mpls.visi.com (Postfix) with ESMTP id 957DE78C4; Tue, 23 Oct 2007 15:09:04 -0500 (CDT) Received: from build64.tcbug.org (unknown [208.42.70.167]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.tcbug.org (Postfix) with ESMTP id 9AEDF10AA868; Tue, 23 Oct 2007 15:09:06 -0500 (CDT) From: Josh Paetzel To: freebsd-performance@freebsd.org, josh.carroll@gmail.com Date: Tue, 23 Oct 2007 15:09:00 -0500 User-Agent: KMail/1.9.7 References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <8cb6106e0710231257k154e9c6ev4b4ba8c3692206fb@mail.gmail.com> In-Reply-To: <8cb6106e0710231257k154e9c6ev4b4ba8c3692206fb@mail.gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1592137.Cs6eKOMjuQ"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200710231509.03771.josh@tcbug.org> Cc: Kip Macy Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2007 21:02:42 -0000 --nextPart1592137.Cs6eKOMjuQ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Tuesday 23 October 2007, Josh Carroll wrote: > > ULE is tuned towards providing cpu affinity compilation and > > evidently encoding are workloads that do not benefit from > > affinity. Before we conclude that it is slower, try building with > > -j5, -j6, j7. > > Here are the results of running ffmpeg with 4 through 8 threads on > both schedulers: > > 4 threads 4bsd: 117.21 > 5 threads 4bsd: 95.75 > 6 threads 4bsd: 93.10 > 7 threads 4bsd: 92.19 > 8 threads 4bsd: 92.38 > > 4 threads ule: 122.19 > 5 threads ule: 107.26 > 6 threads ule: 101.40 > 7 threads ule: 98.72 > 8 threads ule: 96.38 > > 4 threads difference: 4.25 % > 5 threads difference: 12.02 % > 6 threads difference: 8.92 % > 7 threads difference: 7.08 % > 8 threads difference: 4.33 % > > I'm not sure why the performance differential is not consistent > (probably something very technical a scheduler developer could > explain) :) > > Do these results help at all? When running with 9 or more threads, > ffmpeg spits out a lot of errors, so 8 was as high as I could go: > > Error while decoding stream #0.0 > [h264 @ 0x264ae180]too many threads > [h264 @ 0x264ae180]decode_slice_header error > [h264 @ 0x264ae180]no frame! > > My next step is to run some transcodes with mencoder to see if it > has similar performance between the two schedulers. When I have > those results, I'll post them to this thread. > > Thanks for the attention, > Josh Just curious, but are these results obtained while you are=20 overclocking your 2.4ghz CPU to 3.4ghz? That might be a useful=20 datapoint. It also might be useful to know what sort of disks you are using. =20 SATA is notoriously bad at parallel access, and compiling is of=20 course horribly disk bound to begin with. make buildworld also was never designed for massive parallelism at=20 all, and slows down considerably as you try to scale it up with more=20 cpus and increasing -j past a certain point. I don't know where the=20 break is, but it defintely has been hit at 16 cores. =2D-=20 Thanks, Josh Paetzel --nextPart1592137.Cs6eKOMjuQ Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQBHHlTfJvkB8SevrssRAnX3AJsF8CsT84wM2yLOf8A1hj2ljLqC2ACbBCuS Bgeem//qUy+TVAVJoMlwITo= =5Y55 -----END PGP SIGNATURE----- --nextPart1592137.Cs6eKOMjuQ-- From owner-freebsd-performance@FreeBSD.ORG Tue Oct 23 21:55:44 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A2D4916A41B for ; Tue, 23 Oct 2007 21:55:44 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.232]) by mx1.freebsd.org (Postfix) with ESMTP id 5F76013C4A5 for ; Tue, 23 Oct 2007 21:55:44 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: by nz-out-0506.google.com with SMTP id l8so822870nzf for ; Tue, 23 Oct 2007 14:55:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=IOlYIvnv8dp7hWihwSpMLnd6VB6f1DzR7hCfuiKpyY8=; b=pB8d+DfeC+v5WZmAW2dW1P2KXwj+449AkspJXEAgEy/5ZFDqhYXEJIw6J/KmL2QIR2QWPe3chWNRk1TLucoxKiuL51m6EZATtfi5kKmGNdmaFYocfZ6rs7AtN+aMiMWGHZblZU7daA/HWXL9r2fhKQZTpOWFrZFmWi/4RZNYGPg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=kFrM3KlkFLwgo46vjew5UPT5E1hotSwTKvQiHhhekwXPQaUYhkdXtiUqueOdLnfZch4zHmGav/x6ye24zJRnrt/5Lsy88G0Y67G6WS7bxb0QpTPricFoOIPZloqjCSLB/U6HgK4Y/se0qmGWcjIiLeJM6yqBRwmen205s30Hw38= Received: by 10.35.86.12 with SMTP id o12mr8135575pyl.1193176534408; Tue, 23 Oct 2007 14:55:34 -0700 (PDT) Received: by 10.35.117.12 with HTTP; Tue, 23 Oct 2007 14:55:34 -0700 (PDT) Message-ID: <8cb6106e0710231455j1f97c694l5e54578442bde123@mail.gmail.com> Date: Tue, 23 Oct 2007 17:55:34 -0400 From: "Josh Carroll" To: freebsd-performance@freebsd.org In-Reply-To: <8cb6106e0710231257k154e9c6ev4b4ba8c3692206fb@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <8cb6106e0710231257k154e9c6ev4b4ba8c3692206fb@mail.gmail.com> Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: josh.carroll@gmail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2007 21:55:44 -0000 > My next step is to run some transcodes with mencoder to see if it has > similar performance between the two schedulers. When I have those > results, I'll post them to this thread. mencoder is linked against the same libx264 library that ffmpeg uses for h.264 encoding, so I was expecting similar results as ffmpeg. However, the results are slightly different: 4BSD (threads=2): 93.82 real 182.82 user 0.30 sys 4BSD (threads=3): 64.79 real 184.27 user 0.41 sys 4BSD (threads=4): 51.36 real 185.76 user 0.31 sys 4BSD (threads=5): 49.88 real 186.11 user 0.24 sys 4BSD (threads=6): 49.53 real 186.28 user 0.32 sys 4BSD (threads=7): 49.45 real 186.32 user 0.33 sys 4BSD (threads=8): 49.36 real 186.39 user 0.34 sys ULE (threads=2): 92.81 real 182.41 user 0.36 sys ULE (threads=3): 64.28 real 184.57 user 0.39 sys ULE (threads=4): 56.83 real 185.83 user 0.32 sys ULE (threads=5): 55.30 real 185.95 user 0.42 sys ULE (threads=6): 55.38 real 186.12 user 0.45 sys ULE (threads=7): 55.24 real 186.14 user 0.60 sys ULE (threads=8): 55.08 real 186.28 user 0.52 sys What's interesting is that for threads=2 and threads=3, ULE and 4BSD are performing the same. After that, though, there's a 10% gap for the remaining data points. Also interesting is that they both reach a plateau at threads=5. I suppose this means mencoder is more efficient than ffmpeg? Anyway, ULE is still 10% slower with mencoder, which is "worse" than the 5% drop with ffmpeg. I decided to run pbzip2 also. The -p argument doesn't seem to necessarily create as many threads as you request (or it's completely I/O bound): 4BSD(-p 4): 30.91 real 117.32 user 4.67 sys 4BSD(-p 5): 31.45 real 119.49 user 5.02 sys 4BSD(-p 6): 31.85 real 120.42 user 5.49 sys 4BSD(-p 7): 31.55 real 119.16 user 5.59 sys 4BSD(-p 8): 31.92 real 120.29 user 5.81 sys ULE(-p 4): 33.73 real 114.60 user 4.51 sys ULE(-p 5): 31.57 real 116.80 user 5.18 sys ULE(-p 6): 31.74 real 118.00 user 5.21 sys ULE(-p 7): 32.04 real 118.32 user 5.39 sys ULE(-p 8): 32.35 real 120.22 user 6.05 sys ULE is slightly slower here with -p4 (9.12 %) and -p8 (1.35 %), but about the same for 5-7. Hope this helps, Josh From owner-freebsd-performance@FreeBSD.ORG Wed Oct 24 01:06:48 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4642E16A420 for ; Wed, 24 Oct 2007 01:06:48 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.226]) by mx1.freebsd.org (Postfix) with ESMTP id 17D5113C4C4 for ; Wed, 24 Oct 2007 01:06:47 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: by nz-out-0506.google.com with SMTP id l8so30685nzf for ; Tue, 23 Oct 2007 18:06:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=CDPZP3rIo+9CU5/IUiWNIMT/bqI87uC5LFqtBOr9+a4=; b=EYz6Ex7Xo7m22mhrDh8uO0ASXOqoGAmA9zxrytHS89x4AYJh2mxe8Wymnv6RmqgqUYA6fqydqpOdo1BoKwel8IOpDkvDUVnlz9lk3t8DBRSxXyMPGxGlJPrOLFKZ8B1eeMw58IjZiv5rlaljQjzQb1w2uyqi6AA124ZgvdtKB9U= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=CcIPnOkMqnK7ztHUE5Ywv4eN6ShjOGvIlIM6bLfeh8K5/Gj1WOu6VQNrc/MCNLjVFanMAwiOWCDxq7ZpcS5I5FZrq5YtIwPziSFEQv+IKCbxRpT5OpaIiGRuXCxL7mMikt7CcnpO3caPdZFnCEGolt4x3q0qHSPWgMbbeegGH1U= Received: by 10.35.92.18 with SMTP id u18mr22449pyl.1193187999070; Tue, 23 Oct 2007 18:06:39 -0700 (PDT) Received: by 10.35.117.12 with HTTP; Tue, 23 Oct 2007 18:06:39 -0700 (PDT) Message-ID: <8cb6106e0710231806g224fa219n9c6bc4900dcef9b7@mail.gmail.com> Date: Tue, 23 Oct 2007 21:06:39 -0400 From: "Josh Carroll" To: freebsd-performance@freebsd.org In-Reply-To: <8cb6106e0710231455j1f97c694l5e54578442bde123@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <8cb6106e0710231257k154e9c6ev4b4ba8c3692206fb@mail.gmail.com> <8cb6106e0710231455j1f97c694l5e54578442bde123@mail.gmail.com> Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: josh.carroll@gmail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2007 01:06:48 -0000 I decided to do some testing of concurrent processes (rather than a single process that's multi-threaded). Specifically, I ran 4 ffmpeg (without the -threads option) commands at the same time. The difference was less than a percent: 4bsd: 439.92 real 1755.91 user 1.08 sys ule: 442.10 real 1754.65 user 1.34 sys The difference in user/sys is slight, but there. Not sure if that's pertinent, though, given it is such a small percentage. I also ran the same scenario with mencoder, with similar results: 4bsd: 377.96 real 1501.58 user 2.04 sys ule: 377.50 real 1501.68 user 1.93 sys I think this is important, as it shows an N-process workload on an N-processor system is the same between ULE and 4BSD, while a single process (N-threads) workload on an N-processor system seems to favor 4BSD (at least for media encoding). I'm still unsure why MySQL is so much better with ULE, given these results. Again, hope this information is useful! Josh From owner-freebsd-performance@FreeBSD.ORG Wed Oct 24 01:25:05 2007 Return-Path: Delivered-To: freebsd-performance@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9557216A419; Wed, 24 Oct 2007 01:25:05 +0000 (UTC) (envelope-from davidxu@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7BDEB13C480; Wed, 24 Oct 2007 01:25:05 +0000 (UTC) (envelope-from davidxu@FreeBSD.org) Received: from [127.0.0.1] (root@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.1/8.14.1) with ESMTP id l9O1P0Ip007120; Wed, 24 Oct 2007 01:25:02 GMT (envelope-from davidxu@freebsd.org) Message-ID: <471E9F21.7090902@freebsd.org> Date: Wed, 24 Oct 2007 09:25:53 +0800 From: David Xu User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.13) Gecko/20070516 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Kris Kennaway References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <471E343C.2040509@FreeBSD.org> In-Reply-To: <471E343C.2040509@FreeBSD.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: josh.carroll@gmail.com, remy.nonnenmacher@activnetworks.com, freebsd-performance@FreeBSD.org Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2007 01:25:05 -0000 Kris Kennaway wrote: > One major difference is that your workload is 100% user. Also you were > reporting ULE had more idle time, which looks like a bug since I would > expect it be basically 0% idle on such a workload. > > Kris > We can not ignore this performance bug, also I had found that ULE is slower than 4BSD when testing super-smack's update benchmark on my dual-core machine. Regards, David Xu From owner-freebsd-performance@FreeBSD.ORG Wed Oct 24 02:10:36 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8871716A419 for ; Wed, 24 Oct 2007 02:10:36 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.234]) by mx1.freebsd.org (Postfix) with ESMTP id 5BFBD13C49D for ; Wed, 24 Oct 2007 02:10:36 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: by nz-out-0506.google.com with SMTP id l8so41130nzf for ; Tue, 23 Oct 2007 19:10:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=GErppc5Ovatpixno0NsdHtML0SMGVaRqszqOnCZlTEE=; b=jeLwGXzJHq20Un26pNXCo/rIHGdtbflJN53R56Mum95Smb4m79TQuRUu0r1kMSHIpLQy8842ykzsTCMEhFrW7I3dlkfa6qxPuOOVL2WQ/LC1PhBEEGSWIxkKKWIle83TUL9UgiBP4liPcfQ28HMb1hMohRMR4LKUkYX4GdjAHXY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=lP7OQSepS2fC94+IE2R+65ztxZH1P/DykAhUWov4vB8traYRGZkfqcD8etkN4VQpsy/gRMwRuHnDK+64iFJcx+LOuZkR2tMT6yylx2zkNJNAHNd7x+CAiNZj5u/p+qgDru7PO9Rc+zK8xmaLCHhiWfmcg+uIZIFbwNrCDrHAPW8= Received: by 10.35.10.13 with SMTP id n13mr88670pyi.1193191825834; Tue, 23 Oct 2007 19:10:25 -0700 (PDT) Received: by 10.35.117.12 with HTTP; Tue, 23 Oct 2007 19:10:25 -0700 (PDT) Message-ID: <8cb6106e0710231910n4605e776gb7af0025ff1d0d9c@mail.gmail.com> Date: Tue, 23 Oct 2007 22:10:25 -0400 From: "Josh Carroll" To: "David Xu" In-Reply-To: <471E9F21.7090902@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <471E343C.2040509@FreeBSD.org> <471E9F21.7090902@freebsd.org> Cc: remy.nonnenmacher@activnetworks.com, Kris Kennaway , freebsd-performance@freebsd.org Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: josh.carroll@gmail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2007 02:10:36 -0000 > We can not ignore this performance bug, also I had found that ULE is > slower than 4BSD when testing super-smack's update benchmark on my > dual-core machine. I actually saw improved performance with ULE over 4BSD for super-smack. What were the parameters you used for your testing? These were mine: super-smack ./select-key.smack 10 10000 super-smack ./update-select.smack 10 10000 I ran them again to confirm (10 runs each, averaged): 4BSD: super-smack ./select-key.smack 10 10000 : 55235.3 super-smack ./update-select.smack 10 10000 : 17029 ULE: super-smack ./select-key.smack 10 10000 : 65758.5 super-smack ./update-select.smack 10 10000 : 17366.7 So select-key is 19% faster! The numbers I had from 6.2 (4BSD, with libmap.conf set up to map libpthread to libthr): select-key: 50177.34 update-select: 14598.61 So either way, RELENG_7 is faster than 6.2 for super-smack, at least for me. And ULE here is quite a bit faster for select-key. Josh From owner-freebsd-performance@FreeBSD.ORG Wed Oct 24 13:39:39 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 98DC716A417 for ; Wed, 24 Oct 2007 13:39:39 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.178]) by mx1.freebsd.org (Postfix) with ESMTP id 5F3A113C4A8 for ; Wed, 24 Oct 2007 13:39:39 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: by py-out-1112.google.com with SMTP id u77so104214pyb for ; Wed, 24 Oct 2007 06:39:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=il8B0Ek4FeBJsoAUhQkgM594iMT3Ubs8aDa/E0tv2+o=; b=mSyLt665xk1lefYe0/FzWobfofH4iPYTxFBpViYoiN+bFXA2E0FjNd37efyZY0yOPR9FzC9y4EcMNMJmO08U97NwVoNFyIGiOpdcA1uTSr49e/My7zY9ks00Cc+c/Gd8ZFjz2dvStCyd5TPV/zYC/JyTT8TYcgt5tQDOspcrbfQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=JNVhxKuw+Iq6zjLMBx6jIKlVyYNHDCo4lW79Wd+Y2ZKpr6P0K8iBLMf1J5RWCR1QWCYBQ/lrU+/IJC/u5yyclEbHtMxEome2UhbKJK0Cs2XmcDM7uPCifDVmDTNJ67XWZuc1esNzQqyX+5N+68mSHqU4WpjMlgJemcA+QrzCUs0= Received: by 10.35.44.16 with SMTP id w16mr745614pyj.1193233169917; Wed, 24 Oct 2007 06:39:29 -0700 (PDT) Received: by 10.35.117.12 with HTTP; Wed, 24 Oct 2007 06:39:29 -0700 (PDT) Message-ID: <8cb6106e0710240639r20e03ce9w81ed3354338b7395@mail.gmail.com> Date: Wed, 24 Oct 2007 09:39:29 -0400 From: "Josh Carroll" To: "Bruce Evans" In-Reply-To: <20071024171915.E84143@delplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <471E343C.2040509@FreeBSD.org> <20071024171915.E84143@delplex.bde.org> Cc: remy.nonnenmacher@activnetworks.com, Kris Kennaway , freebsd-performance@freebsd.org Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: josh.carroll@gmail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2007 13:39:39 -0000 > 5-6% is a lot. ULE has some tuning for makeworld in -current, which > for me reduced it to less than 1% slower than 4BSD (down from 5-10% > slower), for the case of makeworld -j4 over nfs on a 2-CPU system with > the sources pre-cached on the server and objects on a local file system, > and extensive local tuning of makeworld, nfs and network drivers. I > think the tuning in ULE was mainly for a 2-CPU system, because makeworld > seemed to be very bad under ULE only with 2 CPUs. Apparently, it is also > very bad with more CPUs. There are sysctls to modify the ULE tuning. I found Kris' document here: http://people.freebsd.org/~kris/scaling/mysql.html Which also mentions some ULE tuning. The sched_ule man page doesn't mention what these are. Can someone point me to a document that explains the sysctl tuning I can try? I imagine I would want to tweak one or more of these: kern.sched.preemption: 1 kern.sched.topology: 0 kern.sched.steal_thresh: 2 kern.sched.steal_idle: 1 kern.sched.steal_htt: 1 kern.sched.balance_interval: 133 kern.sched.balance: 1 kern.sched.tryself: 1 kern.sched.affinity: 3 kern.sched.pick_pri: 1 kern.sched.preempt_thresh: 64 kern.sched.interact: 30 kern.sched.slice: 13 But I'm not sure which to try. I did try setting kern.sched.pick_pri to 0, but that hasn't helped the ffmpeg workload at least. Thanks, Josh From owner-freebsd-performance@FreeBSD.ORG Wed Oct 24 13:49:30 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 327ED16A46E for ; Wed, 24 Oct 2007 13:49:30 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from mrout2-b.corp.dcn.yahoo.com (mrout2-b.corp.dcn.yahoo.com [216.109.112.28]) by mx1.freebsd.org (Postfix) with ESMTP id 02AD513C4A7 for ; Wed, 24 Oct 2007 13:49:29 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from unknown-10-101-135-185.yahoo.com.neville-neil.com (proxy7.corp.yahoo.com [216.145.48.98]) by mrout2-b.corp.dcn.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id l9ODcsHJ029300; Wed, 24 Oct 2007 06:38:55 -0700 (PDT) Date: Wed, 24 Oct 2007 21:38:41 +0800 Message-ID: From: gnn@freebsd.org To: "Josh Carroll" In-Reply-To: <8cb6106e0710231806g224fa219n9c6bc4900dcef9b7@mail.gmail.com> References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <8cb6106e0710231257k154e9c6ev4b4ba8c3692206fb@mail.gmail.com> <8cb6106e0710231455j1f97c694l5e54578442bde123@mail.gmail.com> <8cb6106e0710231806g224fa219n9c6bc4900dcef9b7@mail.gmail.com> User-Agent: Wanderlust/2.15.5 (Almost Unreal) SEMI/1.14.6 (Maruoka) FLIM/1.14.8 (=?ISO-8859-4?Q?Shij=F2?=) APEL/10.7 Emacs/22.1 (i386-apple-darwin8.9.1) MULE/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Cc: freebsd-performance@freebsd.org Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2007 13:49:30 -0000 At Tue, 23 Oct 2007 21:06:39 -0400, Josh Carroll wrote: > > I decided to do some testing of concurrent processes (rather than a > single process that's multi-threaded). Specifically, I ran 4 ffmpeg > (without the -threads option) commands at the same time. The > difference was less than a percent: > > 4bsd: 439.92 real 1755.91 user 1.08 sys > ule: 442.10 real 1754.65 user 1.34 sys > > The difference in user/sys is slight, but there. Not sure if that's > pertinent, though, given it is such a small percentage. > > I also ran the same scenario with mencoder, with similar results: > > 4bsd: 377.96 real 1501.58 user 2.04 sys > ule: 377.50 real 1501.68 user 1.93 sys > > I think this is important, as it shows an N-process workload on an > N-processor system is the same between ULE and 4BSD, while a single > process (N-threads) workload on an N-processor system seems to favor > 4BSD (at least for media encoding). I'm still unsure why MySQL is so > much better with ULE, given these results. > > Again, hope this information is useful! > First of all, yes it is and thanks for doing all this. Second, the person who has been working on ULE is moving house so may not respond for a bit, so don't worry, you're work is not being ignored. Best, George From owner-freebsd-performance@FreeBSD.ORG Wed Oct 24 15:46:35 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DD94916A420 for ; Wed, 24 Oct 2007 15:46:35 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.176]) by mx1.freebsd.org (Postfix) with ESMTP id B32D313C4B2 for ; Wed, 24 Oct 2007 15:46:34 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: by py-out-1112.google.com with SMTP id u77so183952pyb for ; Wed, 24 Oct 2007 08:46:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=oxSs5n34RP4qbS9a+nSwV+s4h0R99Wakqs31AeGabcc=; b=WFEC1ae7C0CkPvxd9IE0s2XGgchqWnhGuTX4JWWhjd1K8kljKqqvXvLaQBmI18CecD46WezoEu5cH7KIsbDbcqjUuvop8RrjR1o0z+IUbbazhE/7ikw5uDorLMR27+6ptgSHWB3lITa2kkLR05S5fX3f0DWT93TvI5zS/65CipY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=AiSlNZyAEQDkD4pB2DT+tXjHoEf6n5O4ZoX5kHJnypFi5cy8F8Hg4JTJWiAfHPo5C4ucCYAe+LYhnw/QkNudERVVLAFaPf8ZnXGp0bH+4ZTvonZUVGVsO0ZrcIhUj43wqsnRHfq1OqD/8vg6rS0y8gYqcTBPHV3AbSR3N10Vab8= Received: by 10.35.44.16 with SMTP id w16mr867571pyj.1193240392953; Wed, 24 Oct 2007 08:39:52 -0700 (PDT) Received: by 10.35.117.12 with HTTP; Wed, 24 Oct 2007 08:39:52 -0700 (PDT) Message-ID: <8cb6106e0710240839h1a59f9f9y919e6b297c3efb8e@mail.gmail.com> Date: Wed, 24 Oct 2007 11:39:52 -0400 From: "Josh Carroll" To: "Nick Evans" In-Reply-To: <20071024113434.326c3749@pleiades.nextvenue.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <471E343C.2040509@FreeBSD.org> <20071024171915.E84143@delplex.bde.org> <8cb6106e0710240639r20e03ce9w81ed3354338b7395@mail.gmail.com> <20071024113434.326c3749@pleiades.nextvenue.com> Cc: remy.nonnenmacher@activnetworks.com, Kris Kennaway , freebsd-performance@freebsd.org, Bruce Evans Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: josh.carroll@gmail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2007 15:46:36 -0000 > kern.sched.steal_thresh is/was one of the more effective tuning sysctls. rev > 1.205 of sched_ule had a change that was supposed to automatically adjust it > based on the number of cores. Is this the same 8 core system as the > other thread? In that case the commit dictates steal_thresh should be set to > 3. Give that a try. This is a quad core (single cpu) system. Do these values look proper then? kern.sched.steal_thresh: 2 kern.sched.steal_idle: 1 kern.sched.steal_htt: 1 Thanks, Josh From owner-freebsd-performance@FreeBSD.ORG Wed Oct 24 15:50:46 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C72416A419; Wed, 24 Oct 2007 15:50:46 +0000 (UTC) (envelope-from nevans@talkpoint.com) Received: from mailbox.talkpoint.com (mailbox.talkpoint.com [204.141.15.162]) by mx1.freebsd.org (Postfix) with ESMTP id CD38013C465; Wed, 24 Oct 2007 15:50:45 +0000 (UTC) (envelope-from nevans@talkpoint.com) Received: from localhost (localhost [127.0.0.1]) by mailbox.talkpoint.com (Postfix) with ESMTP id A1BEF458009; Wed, 24 Oct 2007 11:50:38 -0400 (EDT) X-Virus-Scanned: amavisd-new at X-Spam-Flag: NO X-Spam-Score: -2.431 X-Spam-Level: X-Spam-Status: No, score=-2.431 tagged_above=-10 required=5 tests=[AWL=0.168, BAYES_00=-2.599] Received: from mailbox.talkpoint.com ([127.0.0.1]) by localhost (mailbox.talkpoint.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FfJFLgcZ+-df; Wed, 24 Oct 2007 11:50:35 -0400 (EDT) Received: from pleiades.nextvenue.com (pleiades.nextvenue.com [204.141.15.194]) by mailbox.talkpoint.com (Postfix) with ESMTP id 7252A458001; Wed, 24 Oct 2007 11:50:35 -0400 (EDT) Date: Wed, 24 Oct 2007 11:49:54 -0400 From: Nick Evans To: josh.carroll@gmail.com Message-ID: <20071024114954.5ea1b37b@pleiades.nextvenue.com> In-Reply-To: <8cb6106e0710240839h1a59f9f9y919e6b297c3efb8e@mail.gmail.com> References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <471E343C.2040509@FreeBSD.org> <20071024171915.E84143@delplex.bde.org> <8cb6106e0710240639r20e03ce9w81ed3354338b7395@mail.gmail.com> <20071024113434.326c3749@pleiades.nextvenue.com> <8cb6106e0710240839h1a59f9f9y919e6b297c3efb8e@mail.gmail.com> X-Mailer: Claws Mail 2.8.0 (GTK+ 2.10.9; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: remy.nonnenmacher@activnetworks.com, Kris Kennaway , freebsd-performance@freebsd.org, Bruce Evans Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2007 15:50:46 -0000 On Wed, 24 Oct 2007 11:39:52 -0400 "Josh Carroll" wrote: > > kern.sched.steal_thresh is/was one of the more effective tuning sysctls. > > rev 1.205 of sched_ule had a change that was supposed to automatically > > adjust it based on the number of cores. Is this the same 8 core system as > > the other thread? In that case the commit dictates steal_thresh should be > > set to 3. Give that a try. > > This is a quad core (single cpu) system. Do these values look proper then? > > kern.sched.steal_thresh: 2 > kern.sched.steal_idle: 1 > kern.sched.steal_htt: 1 > > Thanks, > Josh Yes, that's the proper default. You could try setting steal_thresh to 1. I noticed a problem with building ports on an 8 core Xeon system while 8 distributed.net crunchers were running. The port build would proceed incredibly slowly, steal_thresh=1 helped a little bit. It might not make up the 5% gap you're seeing though. During early ULE2/3 testing the other variables Jeff recommended trying were sched.pick_pri (which I never saw effect from), sched_tryself and sched.balance. They're all bools IIRC. Since this workload is a bit different from any of mine it would be worthwhile to try those variables. Nick From owner-freebsd-performance@FreeBSD.ORG Wed Oct 24 15:59:46 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD73016A46E; Wed, 24 Oct 2007 15:59:46 +0000 (UTC) (envelope-from nevans@talkpoint.com) Received: from mailbox.talkpoint.com (mailbox.talkpoint.com [204.141.15.162]) by mx1.freebsd.org (Postfix) with ESMTP id 03C8D13C48D; Wed, 24 Oct 2007 15:59:45 +0000 (UTC) (envelope-from nevans@talkpoint.com) Received: from localhost (localhost [127.0.0.1]) by mailbox.talkpoint.com (Postfix) with ESMTP id 73CE7458004; Wed, 24 Oct 2007 11:35:17 -0400 (EDT) X-Virus-Scanned: amavisd-new at X-Spam-Flag: NO X-Spam-Score: -2.43 X-Spam-Level: X-Spam-Status: No, score=-2.43 tagged_above=-10 required=5 tests=[AWL=0.169, BAYES_00=-2.599] Received: from mailbox.talkpoint.com ([127.0.0.1]) by localhost (mailbox.talkpoint.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KTwF4uFmCCrR; Wed, 24 Oct 2007 11:35:15 -0400 (EDT) Received: from pleiades.nextvenue.com (pleiades.nextvenue.com [204.141.15.194]) by mailbox.talkpoint.com (Postfix) with ESMTP id 1739C458001; Wed, 24 Oct 2007 11:35:15 -0400 (EDT) Date: Wed, 24 Oct 2007 11:34:34 -0400 From: Nick Evans To: josh.carroll@gmail.com Message-ID: <20071024113434.326c3749@pleiades.nextvenue.com> In-Reply-To: <8cb6106e0710240639r20e03ce9w81ed3354338b7395@mail.gmail.com> References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <471E343C.2040509@FreeBSD.org> <20071024171915.E84143@delplex.bde.org> <8cb6106e0710240639r20e03ce9w81ed3354338b7395@mail.gmail.com> X-Mailer: Claws Mail 2.8.0 (GTK+ 2.10.9; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: remy.nonnenmacher@activnetworks.com, Kris Kennaway , freebsd-performance@freebsd.org, Bruce Evans Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2007 15:59:46 -0000 On Wed, 24 Oct 2007 09:39:29 -0400 "Josh Carroll" wrote: > > 5-6% is a lot. ULE has some tuning for makeworld in -current, which > > for me reduced it to less than 1% slower than 4BSD (down from 5-10% > > slower), for the case of makeworld -j4 over nfs on a 2-CPU system with > > the sources pre-cached on the server and objects on a local file system, > > and extensive local tuning of makeworld, nfs and network drivers. I > > think the tuning in ULE was mainly for a 2-CPU system, because makeworld > > seemed to be very bad under ULE only with 2 CPUs. Apparently, it is also > > very bad with more CPUs. There are sysctls to modify the ULE tuning. > > I found Kris' document here: > > http://people.freebsd.org/~kris/scaling/mysql.html > > Which also mentions some ULE tuning. The sched_ule man page doesn't > mention what these are. Can someone point me to a document that > explains the sysctl tuning I can try? > > I imagine I would want to tweak one or more of these: > > kern.sched.preemption: 1 > kern.sched.topology: 0 > kern.sched.steal_thresh: 2 > kern.sched.steal_idle: 1 > kern.sched.steal_htt: 1 > kern.sched.balance_interval: 133 > kern.sched.balance: 1 > kern.sched.tryself: 1 > kern.sched.affinity: 3 > kern.sched.pick_pri: 1 > kern.sched.preempt_thresh: 64 > kern.sched.interact: 30 > kern.sched.slice: 13 > > But I'm not sure which to try. I did try setting kern.sched.pick_pri > to 0, but that hasn't helped the ffmpeg workload at least. > > Thanks, > Josh > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" kern.sched.steal_thresh is/was one of the more effective tuning sysctls. rev 1.205 of sched_ule had a change that was supposed to automatically adjust it based on the number of cores. Is this the same 8 core system as the other thread? In that case the commit dictates steal_thresh should be set to 3. Give that a try. Nick From owner-freebsd-performance@FreeBSD.ORG Wed Oct 24 16:08:04 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3075316A469 for ; Wed, 24 Oct 2007 16:08:04 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.187]) by mx1.freebsd.org (Postfix) with ESMTP id 0656C13C4B5 for ; Wed, 24 Oct 2007 16:08:03 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: by rv-out-0910.google.com with SMTP id l15so186759rvb for ; Wed, 24 Oct 2007 09:07:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=LAVZMicoR0gmEwGcfLE4LnDfPlq0rkiLJ6t22i+mZU8=; b=IRTEoDUK3e+8lMph17Cd2psUrPYRYtXsj2CtJqMOb69u4osJoD1QlqfI/z6M7amrzmsCEprFliiR+tUCvk94oJi2XrSpwcgXDIygLWmqaM1V/UfShW82AAhm0qu06YXChbaD5ne3TAegU37o5tmCqDIhjshc+qU5Vc+/RFslEoc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=ehC9vIkpceyay8ZeKTtF2eNWiP1tqwqn1wyHFiVEAR7mvFJC8gigGsLyrIKeXwbwOkikEL9MllRLeVYVKgWlZXIH8/VK2CiYsuAAAjNVg/pIAp4cy0MF1L6+QiCYNyMTv+Su8EIc+pYHj05sz05QDGqyLzkYpd9yyvbbASZKZY4= Received: by 10.141.206.13 with SMTP id i13mr358281rvq.1193242076389; Wed, 24 Oct 2007 09:07:56 -0700 (PDT) Received: by 10.35.117.12 with HTTP; Wed, 24 Oct 2007 09:07:56 -0700 (PDT) Message-ID: <8cb6106e0710240907p28f351f1w3d638e578a4aeb42@mail.gmail.com> Date: Wed, 24 Oct 2007 12:07:56 -0400 From: "Josh Carroll" To: "Nick Evans" In-Reply-To: <20071024114954.5ea1b37b@pleiades.nextvenue.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <471E343C.2040509@FreeBSD.org> <20071024171915.E84143@delplex.bde.org> <8cb6106e0710240639r20e03ce9w81ed3354338b7395@mail.gmail.com> <20071024113434.326c3749@pleiades.nextvenue.com> <8cb6106e0710240839h1a59f9f9y919e6b297c3efb8e@mail.gmail.com> <20071024114954.5ea1b37b@pleiades.nextvenue.com> Cc: remy.nonnenmacher@activnetworks.com, Kris Kennaway , freebsd-performance@freebsd.org, Bruce Evans Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: josh.carroll@gmail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2007 16:08:04 -0000 > Yes, that's the proper default. You could try setting steal_thresh to 1. I > noticed a problem with building ports on an 8 core Xeon system while 8 > distributed.net crunchers were running. The port build would proceed > incredibly slowly, steal_thresh=1 helped a little bit. It might not make up > the 5% gap you're seeing though. During early ULE2/3 testing the other > variables Jeff recommended trying were sched.pick_pri (which I never saw > effect from), sched_tryself and sched.balance. They're all bools IIRC. Since > this workload is a bit different from any of mine it would be worthwhile to > try those variables. Thanks for the information. Setting sched_tryself to 0 improved things slightly. sched.balance didn't seem to help. I'm trying to increase/decrease the balance_interval to see if that helps. Josh From owner-freebsd-performance@FreeBSD.ORG Wed Oct 24 16:15:29 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9749D16A421; Wed, 24 Oct 2007 16:15:29 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from fallbackmx01.syd.optusnet.com.au (fallbackmx01.syd.optusnet.com.au [211.29.132.93]) by mx1.freebsd.org (Postfix) with ESMTP id B209513C48D; Wed, 24 Oct 2007 16:15:28 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail11.syd.optusnet.com.au (mail11.syd.optusnet.com.au [211.29.132.192]) by fallbackmx01.syd.optusnet.com.au (8.12.11.20060308/8.12.11) with ESMTP id l9O7svR2000977; Wed, 24 Oct 2007 17:54:57 +1000 Received: from c220-239-235-248.carlnfd3.nsw.optusnet.com.au (c220-239-235-248.carlnfd3.nsw.optusnet.com.au [220.239.235.248]) by mail11.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id l9O7sVO5030677 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 24 Oct 2007 17:54:33 +1000 Date: Wed, 24 Oct 2007 17:54:31 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Kris Kennaway In-Reply-To: <471E343C.2040509@FreeBSD.org> Message-ID: <20071024171915.E84143@delplex.bde.org> References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <471E343C.2040509@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: josh.carroll@gmail.com, remy.nonnenmacher@activnetworks.com, freebsd-performance@freebsd.org Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2007 16:15:29 -0000 On Tue, 23 Oct 2007, Kris Kennaway wrote: > Josh Carroll wrote: >> Anyway, in summary, ULE is about 5-6 % slower than 4BSD for two >> workloads that I am sensitive to: building world with -j X, and ffmpeg >> -threads X. Other benchmarks seem to indicate relatively equal >> performance between the two. MySQL, on the other hand, is >> significantly faster in ULE. 5-6% is a lot. ULE has some tuning for makeworld in -current, which for me reduced it to less than 1% slower than 4BSD (down from 5-10% slower), for the case of makeworld -j4 over nfs on a 2-CPU system with the sources pre-cached on the server and objects on a local file system, and extensive local tuning of makeworld, nfs and network drivers. I think the tuning in ULE was mainly for a 2-CPU system, because makeworld seemed to be very bad under ULE only with 2 CPUs. Apparently, it is also very bad with more CPUs. There are sysctls to modify the ULE tuning. >> I'm trying to understand why ffmpeg and buildworld are slower in ULE >> than 4BSD, since it seems to me that ULE was supposed to be the better >> scaling scheduler. Makeworld is slower because any scheduling is bad for it. More context switches take longer and cost more by reducing affinity. >> Does anyone have any additional performance tests I can run that might >> help indicate where the deficiency is in the ULE scheduler? MySQL >> performance is excellent, so I'm wondering if it was tuned to that >> particular workload? I think it was. > One major difference is that your workload is 100% user. Also you were > reporting ULE had more idle time, which looks like a bug since I would expect > it be basically 0% idle on such a workload. No, at least buildworld, while being mainly user-CPU-bound by the gcc hog, does some disk accesses and a significant number of sycalls. I have to work very hard to reduce its idle time to about 5% for UP on local disks and to 11% for 2-way SMP over nfs. More idle time for ULE at least used to be a feature. ULE sometimes wants to avoid switching to another thread immediately, in the hope of finding a thread with with better affinity than the currently runnable ones. It waited far too long (in its idle threads) for makeworld with 2 CPUs. Waiting has a better chance of being best if there are many CPUs. Bruce From owner-freebsd-performance@FreeBSD.ORG Wed Oct 24 17:12:23 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2FF7C16A421 for ; Wed, 24 Oct 2007 17:12:23 +0000 (UTC) (envelope-from mgowda82@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.230]) by mx1.freebsd.org (Postfix) with ESMTP id E1A2C13C4AA for ; Wed, 24 Oct 2007 17:12:22 +0000 (UTC) (envelope-from mgowda82@gmail.com) Received: by nz-out-0506.google.com with SMTP id l8so228033nzf for ; Wed, 24 Oct 2007 10:12:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; bh=yJFXsusxMFwzlMuD2I0+fSP901/7X+Bj83EC97MfGNA=; b=b+Y4KG4iBj19sj8KvMBBoK+DsffRU0/79/MB4bpnJLX6A7vW8oL7W4+4b9Scly/AsEZX4gMu4aRv2GqrxnLCnxYHb7ohtFn9+yj9J/DhO2VUlpbgbPKzMPL/MB5L6VWlkzYkN4haKXvdB8kTj95PpeMdpAjvGiov5FVNrrdty9A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=NmB3jYw6/y0b5mFTxypbTJ5s4pCnKRQZKb1EBi3rm2d95YP1EbRZjb5DusTfvZSgHwfSxkmIt127PzgalH61DBZdJUyjnxVPg+Zl1ODkcqC6bh7gUFTgw2bYrqtSTxhUxhwbHo1rzCMmv+GeB+HH2ojVxHItAh5xj6gGMdqNshc= Received: by 10.142.174.8 with SMTP id w8mr241977wfe.1193244222149; Wed, 24 Oct 2007 09:43:42 -0700 (PDT) Received: by 10.142.192.12 with HTTP; Wed, 24 Oct 2007 09:43:42 -0700 (PDT) Message-ID: Date: Wed, 24 Oct 2007 09:43:42 -0700 From: "Manjunath R Gowda" To: josh.carroll@gmail.com In-Reply-To: <8cb6106e0710240907p28f351f1w3d638e578a4aeb42@mail.gmail.com> MIME-Version: 1.0 References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <471E343C.2040509@FreeBSD.org> <20071024171915.E84143@delplex.bde.org> <8cb6106e0710240639r20e03ce9w81ed3354338b7395@mail.gmail.com> <20071024113434.326c3749@pleiades.nextvenue.com> <8cb6106e0710240839h1a59f9f9y919e6b297c3efb8e@mail.gmail.com> <20071024114954.5ea1b37b@pleiades.nextvenue.com> <8cb6106e0710240907p28f351f1w3d638e578a4aeb42@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Nick Evans , remy.nonnenmacher@activnetworks.com, Kris Kennaway , freebsd-performance@freebsd.org, Bruce Evans Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2007 17:12:23 -0000 On 10/24/07, Josh Carroll wrote: > > > Yes, that's the proper default. You could try setting steal_thresh to 1. > I > > noticed a problem with building ports on an 8 core Xeon system while 8 > > distributed.net crunchers were running. The port build would proceed > > incredibly slowly, steal_thresh=1 helped a little bit. It might not make > up > > the 5% gap you're seeing though. During early ULE2/3 testing the other > > variables Jeff recommended trying were sched.pick_pri (which I never saw > > effect from), sched_tryself and sched.balance. They're all bools IIRC. > Since > > this workload is a bit different from any of mine it would be worthwhile > to > > try those variables. > > Thanks for the information. Setting sched_tryself to 0 improved things > slightly. sched.balance didn't seem to help. I'm trying to > increase/decrease the balance_interval to see if that helps. It's worth noting that there are kernel threads competing to run as well. For example ithread and taskqueue threads. How does ULE differentiate and schedule I/O bound threads? From owner-freebsd-performance@FreeBSD.ORG Wed Oct 24 18:19:28 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA7C516A417 for ; Wed, 24 Oct 2007 18:19:28 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id AACEC13C4B3 for ; Wed, 24 Oct 2007 18:19:28 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.104] (cpe-66-91-190-165.hawaii.res.rr.com [66.91.190.165]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id l9OIJCrK062023 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Wed, 24 Oct 2007 14:19:14 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Wed, 24 Oct 2007 11:21:30 -0700 (PDT) From: Jeff Roberson X-X-Sender: jroberson@10.0.0.1 To: Josh Carroll In-Reply-To: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> Message-ID: <20071024111105.M598@10.0.0.1> References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: remy.nonnenmacher@activnetworks.com, freebsd-performance@freebsd.org Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2007 18:19:29 -0000 On Tue, 23 Oct 2007, Josh Carroll wrote: > Hello, > > I posted this to the stable mailing list, as I thought it was > pertinent there, but I think it will get better attention here. So I > apologize in advance for cross-posting if this is a faux pas. :) > > Anyway, in summary, ULE is about 5-6 % slower than 4BSD for two > workloads that I am sensitive to: building world with -j X, and ffmpeg > -threads X. Other benchmarks seem to indicate relatively equal > performance between the two. MySQL, on the other hand, is > significantly faster in ULE. > > I'm trying to understand why ffmpeg and buildworld are slower in ULE > than 4BSD, since it seems to me that ULE was supposed to be the better > scaling scheduler. > > Here is a link to the original thread on the stable mailing list: > > http://lists.freebsd.org/pipermail/freebsd-stable/2007-October/037379.html > > Remy replied with some interesting results for building world between > the two schedulers on an 8-way system. It seems that ULE suffers as > more threads/processes are thrown at it, at least it appears that way > from Remy's data. > > Does anyone have any additional performance tests I can run that might > help indicate where the deficiency is in the ULE scheduler? MySQL > performance is excellent, so I'm wondering if it was tuned to that > particular workload? > > I'm not sure if Remy subscribes to this list, so I am CC'ing him. Hope > you don't mind Remy :) Josh, Thanks for your emails. First, as gnn mentioned, I'm without most of my things at the moment. I have some patches which might improve your workload but I need to test and tune them more myself before I give them out. I doubt any of the sysctls other than steal_thresh and balance_ticks will help your situation. ULE is tuned for workloads that benefit from improved affinity. Not necessarily mysql in particular. Tests with other workloads that benefit from improved affinity have verified that it's not really mysql specific tuning. The problem with buildworld for ULE is that 4BSD gets basically perfect affinity and perfect balancing because the compiler runs, typically uninterrupted, until it does a blocking disk transaction. At that point it doesn't matter which CPU it is resumed on. Your tests with ffmpeg threads vs processes probably is triggering more context switches due to lock contention in the kernel in the threads case. This is also likely the problem with some super-smack tests. On each context switch 4BSD has an opportunity to perfectly balance the CPUs. ULE does not because it's too costly and hinders other workloads. I don't doubt that we can improve things further. It will just have to wait for another few weeks before I'm able to do much about it. Thanks, Jeff > > Regards, > Josh > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Wed Oct 24 19:29:36 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 943FF16A417 for ; Wed, 24 Oct 2007 19:29:36 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.177]) by mx1.freebsd.org (Postfix) with ESMTP id 51BAF13C48A for ; Wed, 24 Oct 2007 19:29:36 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: by py-out-1112.google.com with SMTP id u77so318523pyb for ; Wed, 24 Oct 2007 12:29:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=mA9bNu/H0Dmg6317CEGakFLfGL8DrIvZ58xXlfSWJxE=; b=RdvpwG8ILT0FjgWsHMyNJCjsPCARi8FJXbPVhCA4aha05KC5Bl0KHG04s/v2lr1vQsNA7lCE5xTml7pDk3/jyUx3TNqHb8GgCNKuF/8C8fqzIyfqd+f388hfGFP1ehCKED4CY268Q/WdOjRNVsIfSPMH/w6UxWbhcih89QZKQkA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=okbE1h63dn6c0T/QgGvlbDvXqEloHBrwXn4u2VyvbW2AjWDJI4ShxOxGkm4XEm+uXQMjjCk+vSdmVeILzCPYByepGzhSih8OsaBewb/0eCIa3u+qWpB77Eps+t1gwnioFSTo2UpaqJcY0tE1xrrb6MwfS7OSRnuDhBb5RDiKC5g= Received: by 10.35.14.18 with SMTP id r18mr1050015pyi.1193254164329; Wed, 24 Oct 2007 12:29:24 -0700 (PDT) Received: by 10.35.117.12 with HTTP; Wed, 24 Oct 2007 12:29:24 -0700 (PDT) Message-ID: <8cb6106e0710241229i12852d8cq436f4c955ac62c56@mail.gmail.com> Date: Wed, 24 Oct 2007 15:29:24 -0400 From: "Josh Carroll" To: "Jeff Roberson" In-Reply-To: <20071024111105.M598@10.0.0.1> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <20071024111105.M598@10.0.0.1> Cc: remy.nonnenmacher@activnetworks.com, freebsd-performance@freebsd.org Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: josh.carroll@gmail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2007 19:29:36 -0000 > Your tests with ffmpeg threads vs processes probably is triggering more > context switches due to lock contention in the kernel in the threads case. > This is also likely the problem with some super-smack tests. On each > context switch 4BSD has an opportunity to perfectly balance the CPUs. ULE > does not because it's too costly and hinders other workloads. Thanks for the response Jeff, I appreciate it. Is this something that the scheduler can recognize and auto-tune itself for? E.g. have it recognize the scenario differences and do what 4BSD is doing for cases such as ffmpeg and what ULE otherwise does in other circumstances? I guess this would not come without its own overhead, which might defeat the purpose anyway. > I don't doubt that we can improve things further. It will just have to > wait for another few weeks before I'm able to do much about it. Thanks again, I do appreciate your work and help. So you're anticipating nothing can be done at all, or did you mean at the moment until you get your equipment back and get settled? Or is this something that would need to be looked at in -current rather than 7-STABLE? Regards, Josh From owner-freebsd-performance@FreeBSD.ORG Wed Oct 24 20:33:27 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 905E516A419 for ; Wed, 24 Oct 2007 20:33:27 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 3B74713C491 for ; Wed, 24 Oct 2007 20:33:27 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.104] (cpe-66-91-190-165.hawaii.res.rr.com [66.91.190.165]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id l9OKX8EM008989 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Wed, 24 Oct 2007 16:33:10 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Wed, 24 Oct 2007 13:35:26 -0700 (PDT) From: Jeff Roberson X-X-Sender: jroberson@10.0.0.1 To: Josh Carroll In-Reply-To: <8cb6106e0710241229i12852d8cq436f4c955ac62c56@mail.gmail.com> Message-ID: <20071024133240.X598@10.0.0.1> References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <20071024111105.M598@10.0.0.1> <8cb6106e0710241229i12852d8cq436f4c955ac62c56@mail.gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: remy.nonnenmacher@activnetworks.com, freebsd-performance@freebsd.org Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2007 20:33:27 -0000 On Wed, 24 Oct 2007, Josh Carroll wrote: >> Your tests with ffmpeg threads vs processes probably is triggering more >> context switches due to lock contention in the kernel in the threads case. >> This is also likely the problem with some super-smack tests. On each >> context switch 4BSD has an opportunity to perfectly balance the CPUs. ULE >> does not because it's too costly and hinders other workloads. > > Thanks for the response Jeff, I appreciate it. Is this something that > the scheduler can recognize and auto-tune itself for? E.g. have it > recognize the scenario differences and do what 4BSD is doing for cases > such as ffmpeg and what ULE otherwise does in other circumstances? I > guess this would not come without its own overhead, which might defeat > the purpose anyway. The problem is the two models are drastically incompatible. Rather than use a shared queue for some threads I'm just going to have to continue to refine the load balancing algorithm. The higher idle time with ULE is a symptom of assinging work to the wrong place. I think I have a good idea of how to improve that cpu selection. > >> I don't doubt that we can improve things further. It will just have to >> wait for another few weeks before I'm able to do much about it. > > Thanks again, I do appreciate your work and help. So you're > anticipating nothing can be done at all, or did you mean at the moment > until you get your equipment back and get settled? Or is this > something that would need to be looked at in -current rather than > 7-STABLE? I'm confident that we can improve things. It will probably not make the cut for 7.0 since it will be too disruptive. I'm sure it can be backported before 7.1 when ULE is likely to become the default. I hope that we can continue to work together to verify any fixes I may come up with. Thanks, Jeff > > Regards, > Josh > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Wed Oct 24 23:37:06 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 254F316A41A for ; Wed, 24 Oct 2007 23:37:06 +0000 (UTC) (envelope-from andrew.george.hammond@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.225]) by mx1.freebsd.org (Postfix) with ESMTP id D0ED613C4B2 for ; Wed, 24 Oct 2007 23:37:05 +0000 (UTC) (envelope-from andrew.george.hammond@gmail.com) Received: by nz-out-0506.google.com with SMTP id l8so312038nzf for ; Wed, 24 Oct 2007 16:36:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; bh=52YNvdtTjzchycCdyl1iDQmcW4I7w1DyJTtvyFHtDsM=; b=oyXTYaP3znBZGD2cI04WCcv3fgGtqkYmIqC6rXOPeaWgQKLIs3I7xRQEzny+VOdcpKu2MGtF4QlyC6E7rwJ6lVuqGGpsEnPYJ+RsMYvK6QXQl4vyUDPxPuZFb/O1fyHYnsdCmMNn6eWwwVnl34EuzARToxBxudiw5t0aU1cz7qk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type; b=Z4gkjH1avWBT8h9hD5TpjQOQjnwZjd/C09A/warIl2S6CAicgXDfw/JdQ7kIirGdJaMdmwuca3WEAGFQUh6n4E8mJpwt8TUvq+6frpaAfscThgJIIleDYAcjSn+9cZ2aEaNtHauyPCVC39QNSZVMqLb+XLYLruI+PiLam0lRiG4= Received: by 10.65.194.13 with SMTP id w13mr2377457qbp.1193267353483; Wed, 24 Oct 2007 16:09:13 -0700 (PDT) Received: by 10.64.203.6 with HTTP; Wed, 24 Oct 2007 16:09:13 -0700 (PDT) Message-ID: <5a0a9d6f0710241609q6e9abf82tfc836081afcad7dc@mail.gmail.com> Date: Wed, 24 Oct 2007 16:09:13 -0700 From: "Andrew Hammond" To: freebsd-performance@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: overhead for geom striping? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2007 23:37:06 -0000 Has anyone got numbers for overhead costs associated with using geom to strip across LUNs? I am particularly interested in latency increases, CPU costs, and any known throughput limitations. We're running FreeBSD 6.2. If there are any other issues associated with using it in production I'd really like to hear them too. Best Regards, Andrew Hammond From owner-freebsd-performance@FreeBSD.ORG Fri Oct 26 02:25:09 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 67AE216A421 for ; Fri, 26 Oct 2007 02:25:09 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.182]) by mx1.freebsd.org (Postfix) with ESMTP id 19CD813C4A8 for ; Fri, 26 Oct 2007 02:25:08 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: by py-out-1112.google.com with SMTP id u77so1171363pyb for ; Thu, 25 Oct 2007 19:25:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=pKyc60qp+/l+1sdcJHnvQQqpei9GFV2DQht/1wK0ca4=; b=rRB3GM7sTCXD+8cwsM+/+A3Y5URf3OAc0riyMf8qSxEeDrbsoAUqvWcDp/HTLaQz1SPLFwQLQHtyWlS4ZuwFbRBMKFqhUAj3IbdLeRIu14av/MshtbbgD3wGNauLlNWDCY4RzZdgkm2PnOBRhLfqHUi4n4MDtn1KyMlWAft829Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=SljTh5n7K4WIJj47XUiBb/qNc3aNe/MztYgmJ8J6JsMb/ZF2j6AHXWK9WIxs8basQY4R+Oq7fA1FLojQeBtLR6EXHh1ME9ynnGLdUg8FlF46ndm2Gxb5lmyYQwn/MNyFgzc2M3ORpTVtWRw/7qGokLvnZ0qBtQq3EFO1PJDAcxQ= Received: by 10.35.121.12 with SMTP id y12mr2829281pym.1193365508209; Thu, 25 Oct 2007 19:25:08 -0700 (PDT) Received: by 10.35.117.12 with HTTP; Thu, 25 Oct 2007 19:25:08 -0700 (PDT) Message-ID: <8cb6106e0710251925s2db0117cvcb67321b08d7b2a1@mail.gmail.com> Date: Thu, 25 Oct 2007 22:25:08 -0400 From: "Josh Carroll" To: "Jeff Roberson" In-Reply-To: <20071024133240.X598@10.0.0.1> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <20071024111105.M598@10.0.0.1> <8cb6106e0710241229i12852d8cq436f4c955ac62c56@mail.gmail.com> <20071024133240.X598@10.0.0.1> Cc: freebsd-performance@freebsd.org Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: josh.carroll@gmail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Oct 2007 02:25:09 -0000 > I'm confident that we can improve things. It will probably not make the > cut for 7.0 since it will be too disruptive. I'm sure it can be > backported before 7.1 when ULE is likely to become the default. That sounds great! I figured it was something that would have to wait until 7.0 released. I completely understand that. > I hope that we can continue to work together to verify any fixes I may > come up with. Absolutely! Just let me know how I can help. If you need a guinea pig...er...tester :) I'll be glad to help! Another thing I noticed between ULE and 4BSD is my core temperatures seem erratic on ULE. I use RRDtool with the new coretemp(4) feature and noticed the temperatures are spiking a lot with ULE, and generally slightly higher than when running a 4BSD kernel. I don't know if that's significant or not. Just something I noticed when I modified my RRD scripts to use coretemp. For a side-by-side comparison, see this page: http://pflog.net/~floyd/fbsd_sched.html Thanks again for all your help! Please let me know if/when I can do anything else to help out. Regards, Josh