From owner-freebsd-stable@FreeBSD.ORG Tue Nov 20 19:03:14 2007 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ADB0116A41A; Tue, 20 Nov 2007 19:03:14 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (pointyhat.freebsd.org [IPv6:2001:4f8:fff6::2b]) by mx1.freebsd.org (Postfix) with ESMTP id DD3FD13C469; Tue, 20 Nov 2007 19:03:13 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47432F77.3030606@FreeBSD.org> Date: Tue, 20 Nov 2007 20:03:19 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Alexey Popov References: <4741905E.8050300@chistydom.ru> <47419AB3.5030008@chistydom.ru> <4741A7DA.2050706@chistydom.ru> <4741DA15.9000308@FreeBSD.org> <47429DB8.7040504@chistydom.ru> <4742ADFE.40902@FreeBSD.org> <4742C46A.1060701@chistydom.ru> In-Reply-To: <4742C46A.1060701@chistydom.ru> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Attilio Rao , freebsd-stable@freebsd.org Subject: Re: 2 x quad-core system is slower that 2 x dual core on FreeBSD X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2007 19:03:14 -0000 Alexey Popov wrote: > Hi > > Kris Kennaway wrote: >>>>>>> CPU states: 9.5% user, 0.0% nice, 82.0% system, 0.5% >>>>>>> interrupt, 8.0% idle >>>>>> A wild idea that might not help: try reducing kern.hz in >>>>>> loader.conf to >>>>>> something like 100 and see if something significant changes. >>> Usually on PHP backends slow PHP code eats most of the CPU time. I >>> have %user much bigger than %system in CPU states. >>> But now %system is much bigger than %user and I can conclude that on >>> 8-core server FreeBSD consumes more CPU time than PHP. >> That is one possibility, but you still need to look at the actual >> throughput on these machines before making conclusions about which is >> performing better. Can you please provide those numbers for 6.x, 7.x >> with ULE and 4BSD on the 4-core and 8-core systems? > Ok, here's results of practical research. The following is approximate > maximum qps that backends can survive with my workload: > > 7-STABLE quad ULE 20 > 7-STABLE quad 4BSD 17 > 6-STABLE quad 14 > 6-STABLE dual 21 > Linux CentOS 5 quad >50 OK, so 7.x is an improvement compared to 6.x on the 8 core machine, and ULE is an improvement over 4BSD. This much is in line with expectations. Neither shows an improvement vs 4 cores. It is hard to say for certain without a direct profile comparison of the workload, but it is probably due to lockmgr contention. lockmgr is used for various locking operations to do with VFS data structures. It is known to have poor performance and scale very badly. It is interesting that you are running into this on a real workload though, so far I have only encountered it as a limiting factor in synthetic microbenchmarks. There was some work done over the summer on replacing lockmgr with something reasonable, but unfortunately it is not yet ready for testing. I am CC'ing the developer who was working on that (Attilio Rao). Depending on his availability it will probably be at least a couple of months before it is ready though. In the meantime there is unfortunately not a lot that can be done, AFAICT. There is one hack that I will send you later but it is not likely to help much. I will also think about how to track down the cause of the contention further (the profiling trace only shows that it comes mostly from vget/vput but doesn't show where these are called from). Kris > > With best regards, > Alexey Popov > >