From owner-freebsd-stable@FreeBSD.ORG Wed Nov 21 11:14:10 2007 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0E47316A418; Wed, 21 Nov 2007 11:14:10 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from comtv.ru (comtv.ru [217.10.32.17]) by mx1.freebsd.org (Postfix) with ESMTP id 543F013C467; Wed, 21 Nov 2007 11:14:08 +0000 (UTC) (envelope-from lol@chistydom.ru) X-UCL: actv Received: from yoda.org.ru ([83.167.98.162] verified) by comtv.ru (CommuniGate Pro SMTP 4.1.8) with ESMTP id 21325914; Wed, 21 Nov 2007 14:13:49 +0300 Received: from [80.68.244.40] (adm40.relax.ru [80.68.244.40]) (Authenticated sender: llp@soekris.ru) by yoda.org.ru (Postfix) with ESMTP id A95F828D6F; Wed, 21 Nov 2007 14:14:04 +0300 (MSK) Message-ID: <474412CF.5050200@chistydom.ru> Date: Wed, 21 Nov 2007 14:13:19 +0300 From: Alexey Popov User-Agent: Thunderbird 2.0.0.6 (X11/20070924) MIME-Version: 1.0 To: Kris Kennaway References: <4741905E.8050300@chistydom.ru> <47419AB3.5030008@chistydom.ru> <4741A7DA.2050706@chistydom.ru> <4741DA15.9000308@FreeBSD.org> <47429DB8.7040504@chistydom.ru> <4742ADFE.40902@FreeBSD.org> <4742C46A.1060701@chistydom.ru> <47432F77.3030606@FreeBSD.org> <474339E9.4080301@FreeBSD.org> In-Reply-To: <474339E9.4080301@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Attilio Rao , freebsd-stable@freebsd.org Subject: Re: 2 x quad-core system is slower that 2 x dual core on FreeBSD X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2007 11:14:10 -0000 Hi. Kris Kennaway wrote: >> In the meantime there is unfortunately not a lot that can be done, >> AFAICT. There is one hack that I will send you later but it is not >> likely to help much. I will also think about how to track down the >> cause of the contention further (the profiling trace only shows that >> it comes mostly from vget/vput but doesn't show where these are called >> from). > Actually this patch might help. It doesn't replace lockmgr but it does > fix a silly thundering herd behaviour. It probably needs some > adjustment to get it to apply cleanly (it is about 7 months old), and I > apparently stopped using it because I ran into deadlocks. It might be > stable enough to at least see how much it helps. Sorry, I didn't try you patch yet but I have other news. As mentioned in the description of your patch there is probably a scalability problem with stat() syscall on FreeBSD. The PHP code of our site consists of large amount of modules. I think this is true for many other large PHP sites. I reached out that PHP calls lstat() for every path element of each file it opens including modules. Truss output shows that PHP makes more than 2000 lstat's for one /index.php request. After investigation I found out that lstats() are called from realpath() libc function. It turned out that PHP has "realpath cache", but it's size by default is 16K which is not enough for my files. I set realpath_cache_size to 256K and now there is no that much lstat calls. Performance of 8-core machine growed in ~ 50% for me on 7-STABLE. Now it can handle 30 and more requests per seconds. I have the similiar results with 6-STABLE. Now I have not that big %sys values as it was before (see attached top output). Nevertheless, Linux with its >50 rps is still far away from FreeBSD. Linux makes that 2000+ lstat's without problem. There's still stat(), open(), gettimeofday(), close() syscalls for each include file in PHP that i can not switch off. And also it is unclear for me what to do with MySQL which happened to have the same problems for me. With best regards, Alexey Popov