From owner-freebsd-hackers@FreeBSD.ORG Sat Feb 5 19:44:38 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6FE5D106564A for ; Sat, 5 Feb 2011 19:44:38 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 213C28FC13 for ; Sat, 5 Feb 2011 19:44:37 +0000 (UTC) Received: by qwj9 with SMTP id 9so2586778qwj.13 for ; Sat, 05 Feb 2011 11:44:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:from :date:x-google-sender-auth:message-id:subject:to:cc:content-type; bh=VZiDw3b56cm4XHBxggj5y3AfehGx2dph9OM78i7hX6w=; b=eRHliOu9kDQOYqputB5UJZJefEEzahH5hyq35bBuwgNZHdEn/RhNZ8Ry0FnL6lgLxi mxwAINFh0gMaxFMad9aEwWF1RAfLS8bssVcIikoXbplt0LDgQ0wXwLeWWznHTjTrIKyi ILyhTYh5Ho66GmJ6Yp1fRjP2UeeyFmvyGP5aI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type; b=tj1HMKJk9xfZ7wSxeSzAqEZjAbuK94k+tUsUOonmsKdK/9PCy9zVOrR3Af31Ikiol0 YtoBlibilW2Ak9lobJkNBgFP3ahgQtqTc0els6ftficeXYSWRdLG4SKe7MIa4WMvp+mb edeh7b8DwtlIlAARIPnfqKPi26r+tPdR1Y+88= Received: by 10.229.43.195 with SMTP id x3mr11349860qce.291.1296935076400; Sat, 05 Feb 2011 11:44:36 -0800 (PST) MIME-Version: 1.0 Sender: ivoras@gmail.com Received: by 10.229.67.2 with HTTP; Sat, 5 Feb 2011 11:43:56 -0800 (PST) In-Reply-To: <4D4D9A4D.2070508@yandex.ru> References: <4D3CB2AF.9050003@yandex.ru> <4D4D9A4D.2070508@yandex.ru> From: Ivan Voras Date: Sat, 5 Feb 2011 20:43:56 +0100 X-Google-Sender-Auth: ZkOzC9YleQpuHFcJ3zp9jScqif0 Message-ID: To: Ruslan Mahmatkhanov Content-Type: text/plain; charset=UTF-8 Cc: freebsd-hackers@freebsd.org, davidxu@freebsd.org Subject: Re: Tracking down a problem with php on FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Feb 2011 19:44:38 -0000 On 5 February 2011 19:43, Ruslan Mahmatkhanov wrote: > Hi, Ivan! > > Thank you much for response and sorry for late answer. We was able to > collect some data about the issue to make discussion more objective. See > below. >>> Simple php-fpm restart solves the problem, but i need to track it down >>> to the cause of this situation and ask for your assistance and >>> instructions on how to debug it. Some facts about this: >> >> On one hand, FPM is said to be very experimental... >> >> Personally, I've been using apache22-worker or apache22-event + >> mod_fcgid for years without trouble. > > We prefer to avoid using apache at all, because in this it's just adds yet > another unneeded link and complexity. I guess it's about tradeoffs beween complexity and stability :) >>> - `top -mio` shows very high (80000-90000 for VCSW) VCSW/IVCSW values >>> for php-fpm processes and LA is more than 120 I think this is significant, especially with this: > When attaching to any hanging php-fpm proccess with truss, than i see a lot > of this calls: > sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078) > = 0 (0x0) > sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078) > = 0 (0x0) > sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078) > = 0 (0x0) > sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078) > = 0 (0x0) "Normal" processes of the type PHP is have no need to call sched_yield() arbitrarily, unless they are implementing something they shouldn't - like a synchronization primitive (semaphore/lock). If "a lot" means "of the same order of magnitude as your VCSW rate", this is the reason for it. I've analyzed my php-cgi binary and modules and they don't use sched_yield. And yes, grepping for it in the source finds it only in FPM: sapi/fpm/fpm/fpm_atomic.h:140: sched_yield(); It seems they are trying to implement a spinlock by hand, instead of using what the OS provides. (on the other hand, the implementation might be correct but they may be using it wrong). In any case, this points to bugs in FPM. if so, unfortunately I can't help you further. If you really want to continue using FPM, I guess you should probably replace this hand-made lock implementation by sem(4) or see if "robust" pthreads mutexes can be committed and MFCed (maybe with David Xu). Here is the FPM file: http://svn.php.net/viewvc/php/php-src/branches/PHP_5_3/sapi/fpm/fpm/fpm_atomic.h?revision=305417&view=markup