From owner-freebsd-hackers@FreeBSD.ORG Wed Feb 9 02:31:01 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 12C38106567A; Wed, 9 Feb 2011 02:31:01 +0000 (UTC) (envelope-from davidxu@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D76C78FC08; Wed, 9 Feb 2011 02:31:00 +0000 (UTC) Received: from xyf.my.dom (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p192Uv7l096254; Wed, 9 Feb 2011 02:30:58 GMT (envelope-from davidxu@freebsd.org) Message-ID: <4D51FC60.8010005@freebsd.org> Date: Wed, 09 Feb 2011 10:30:56 +0800 From: David Xu User-Agent: Thunderbird 2.0.0.24 (X11/20100630) MIME-Version: 1.0 To: Ivan Voras References: <4D3CB2AF.9050003@yandex.ru> <4D4D9A4D.2070508@yandex.ru> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Ruslan Mahmatkhanov Subject: Re: Tracking down a problem with php on FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 02:31:01 -0000 Ivan Voras wrote: > On 5 February 2011 19:43, Ruslan Mahmatkhanov wrote: >> Hi, Ivan! >> >> Thank you much for response and sorry for late answer. We was able to >> collect some data about the issue to make discussion more objective. See >> below. > >>>> Simple php-fpm restart solves the problem, but i need to track it down >>>> to the cause of this situation and ask for your assistance and >>>> instructions on how to debug it. Some facts about this: >>> On one hand, FPM is said to be very experimental... >>> >>> Personally, I've been using apache22-worker or apache22-event + >>> mod_fcgid for years without trouble. >> We prefer to avoid using apache at all, because in this it's just adds yet >> another unneeded link and complexity. > > I guess it's about tradeoffs beween complexity and stability :) > >>>> - `top -mio` shows very high (80000-90000 for VCSW) VCSW/IVCSW values >>>> for php-fpm processes and LA is more than 120 > > I think this is significant, especially with this: > >> When attaching to any hanging php-fpm proccess with truss, than i see a lot >> of this calls: >> sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078) >> = 0 (0x0) >> sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078) >> = 0 (0x0) >> sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078) >> = 0 (0x0) >> sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078) >> = 0 (0x0) > > "Normal" processes of the type PHP is have no need to call > sched_yield() arbitrarily, unless they are implementing something they > shouldn't - like a synchronization primitive (semaphore/lock). If "a > lot" means "of the same order of magnitude as your VCSW rate", this is > the reason for it. > > I've analyzed my php-cgi binary and modules and they don't use sched_yield. > > And yes, grepping for it in the source finds it only in FPM: > > sapi/fpm/fpm/fpm_atomic.h:140: sched_yield(); > > It seems they are trying to implement a spinlock by hand, instead of > using what the OS provides. (on the other hand, the implementation > might be correct but they may be using it wrong). > > In any case, this points to bugs in FPM. if so, unfortunately I can't > help you further. > > If you really want to continue using FPM, I guess you should probably > replace this hand-made lock implementation by sem(4) or see if > "robust" pthreads mutexes can be committed and MFCed (maybe with David > Xu). > Yes, there is a p4 branch implemented pthread robust mutex, it requires ABI change. Document for the POSIX robust mutex is here: http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_mutexattr_getrobust.html