Date: Sun, 23 Jan 2011 23:54:23 +0300 From: Ruslan Mahmatkhanov <cvs-src@yandex.ru> To: freebsd-hackers@freebsd.org Cc: cvs-src@yandex.ru Subject: Tracking down a problem with php on FreeBSD Message-ID: <4D3C957F.6080209@yandex.ru>
next in thread | raw e-mail | index | archive | help
Good day! We are using custom php application on FreeBSD 8.1R amd64. It is started with php-fpm 5.3.3 from ports as backend and nginx 0.8.54 as frontend. Several times per day this app is making self unavailable. Simple php-fpm restart solves the problem, but i need to track it down to the cause of this situation and ask for your assistance and instructions on how to debug it. Some facts about this: - I don't know how to manually reproduce this, but it happens several times every day - It happens on FreeBSD 7.x too - It happens with apache+mod_php instead php-fpm - It happens with lighthttpd instead nginx - It happens with both SHED_4BSD and SHED_ULE - It doesn't happen on php =< 5.2.12 - It happens with and w/o eaccelerator - `top -mio` shows very high (80000-90000 for VCSW) VCSW/IVCSW values for php-fpm processes and LA is more than 120 - user seeing http 502 error code in browser - php-fpm log has many of this lines in time of crash: Jan 23 17:56:58.176425 [WARNING] [pool world] server reached max_children setting (100), consider raising it Jan 23 17:56:59.528873 [WARNING] [pool world] child 686, script '/usr/local/www/world/public_html/script.php' execution timed out (10.074251 sec), terminating Jan 23 17:57:03.221677 [WARNING] [pool world] child 686 exited on signal 15 (SIGTERM) after 59.424804 seconds from start Jan 23 17:57:03.222580 [NOTICE] [pool world] child 4407 started Jan 23 17:57:03.222709 [NOTICE] Finishing ... Jan 23 17:57:03.260991 [WARNING] [pool site] child 3962, script '/usr/local/www/site/public_html/script.php' execution timed out (12.644470 sec), terminating - nginx log has many of this lines in time of crash: 2011/01/23 17:57:00 [error] 38018#0: *26006023 writev() failed (54: Connection reset by peer) while sending request to upstream, client: xx.xx.xx.xx, server: some.server.org, request: "POST /?ctrl=Chat& a=chatList&__path=chat_list&h=8093b9e1cf448762d5677e21bded97ae& h1=38ca8b747a46098c3b1a4f39e6658170 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9002", host: "some.server.org", referrer: "http://some.server.org/" 2011/01/23 17:57:00 [error] 38016#0: *26029878 kevent() reported about an closed connection (54: Connection reset by peer) while reading response header from upstream, client: xx.xx.xx.xx, server: some.server.org, request: "POST /?ctrl=Location&a=refresh& __path=refresh&h=276f591df26a65d9a1736f6e1006f4ab& h1=3c0916c16b1fc2e7015b71e90bbc3d23 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9002", host: "some.server.org", referrer: "http://world.mist-game.ru/" 2011/01/23 17:57:02 [crit] 38020#0: *26034390 open() "/tmp/nginx /client_temp/1/74/0000000741" failed (13: Permission denied) while sending request to upstream, client: xx.xx.xx.xx, server: some.server.org, request: "POST /?ctrl=Chat&a=send&__path=chat_send& h=4a27d8d382ba9b1059412323a451ef84& h1=b0a53c86e3c744a01356a5030559ed1a HTTP/1.1", upstream: "fastcgi://127.0.0.1:9002", host: "some.server.org", referrer: "http://world.mist-game.ru/" 2011/01/23 17:57:02 [alert] 38020#0: *26034390 http request count is zero while sending to client, client: xx.xx.xx.xx, server: some.server.org, request: "POST /?ctrl=Chat&a=send&__path=chat_send& h=4a27d8d382ba9b1059412323a451ef84& h1=b0a53c86e3c744a01356a5030559ed1a HTTP/1.1", upstream: "fastcgi://127.0.0.1:9002", host: "some.server.org", referrer: "http://some.server.org/" 2011/01/23 17:57:03 [error] 38014#0: *25997903 upstream prematurely closed connection while reading response header from upstream, client: 109.229.69.186, server: some.server.org, request: "POST /?ctrl=Chat&a=chatList&__path=chat_list& h=c8723de73c4f8ebb98f9bf746d75e965& h1=3ab289760a009b07b73c6d96cc94a509 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9002", host: "some.server.org", referrer: "http://some.server.org/" `top -mio` output in time of crash: http://pastebin.com/yrCwxnbr kernel config: http://pastebin.com/nGA0518A php port options: # cat /var/db/ports/php5/options WITH_CLI=true WITH_CGI=true WITH_FPM=true WITH_APACHE=true WITHOUT_AP2FILTER=true WITHOUT_DEBUG=true WITH_SUHOSIN=true WITH_MULTIBYTE=true WITHOUT_IPV6=true WITH_MAILHEAD=true WITHOUT_LINKTHR=true php-fpm pool configuration: [world] listen = 127.0.0.1:9002 listen.allowed_clients = 127.0.0.1 listen.owner = www listen.group = www listen.mode = 0666 user = www group = www pm = dynamic pm.max_children = 100 pm.start_servers = 20 pm.min_spare_servers = 5 pm.max_spare_servers = 35 pm.max_requests = 5000 pm.status_path = /JWorldStatus request_terminate_timeout = 10s Any help is highly appreciated. Thank you all in advance. PS. Please keep me in cc:. I'm not subscribed to freebsd-hackers@. -- Regards, Ruslan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4D3C957F.6080209>