From owner-freebsd-questions@FreeBSD.ORG Wed Apr 15 08:02:55 2015 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 70F5F60F for ; Wed, 15 Apr 2015 08:02:55 +0000 (UTC) Received: from mail-vn0-x22e.google.com (mail-vn0-x22e.google.com [IPv6:2607:f8b0:400c:c0f::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2801A10A for ; Wed, 15 Apr 2015 08:02:55 +0000 (UTC) Received: by vnbg190 with SMTP id g190so12091745vnb.12 for ; Wed, 15 Apr 2015 01:02:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=vz8aEfM7+zIO82k1C6+ohKOFhuRIa3PGfQDDrxbD/ks=; b=pKNcc7LlASk9jZ3yH91tALQ6GY7ZeRj/na3gH1nXTVg9G0bJpIL+zIgU6s/b/O+Nfc PdUfRHrCoYE04QDIyZ7xyzboWwaPoUd10aeFSnoNLuAAqNIl4KwKZYa/ZKoIiXQ15krj /V+qIG58EzxXa4kOqcVoD3L3Y1jcZF8+l4v/6jVpp6ws12+DuPGHP5LBPVAk8CQRoG4d lfQdZ3Db3F5SYN+eq1xHC7+cvJcpW0YMOgz8qOCzN0SeA7BU3D07/5CZb2gBsHn3eIWL VMUHthmxFCe2Q6SifhtVnk029bRD4Ox1mfLb3jIGyew6G1nJ3f+26KFV0hyAn4tlMIdW lHoQ== MIME-Version: 1.0 X-Received: by 10.60.37.166 with SMTP id z6mr19576900oej.63.1429084974226; Wed, 15 Apr 2015 01:02:54 -0700 (PDT) Received: by 10.76.72.66 with HTTP; Wed, 15 Apr 2015 01:02:54 -0700 (PDT) Date: Wed, 15 Apr 2015 12:02:54 +0400 Message-ID: Subject: High LA caused by php-fpm From: Dimitri Minaev To: freebsd-questions@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Apr 2015 08:02:55 -0000 Hello, Today, I tried to move our bunch of Drupal-powered web sites to a new server. The old server was running FreeBSD 7.2 and I wanted to install a newer version. In the meanwhile, I transferred the sites to the new server (FreeBSD 10.1 on ZFS mirror, 1CPU with 6 cores and hyperthreading). Everything looked fine, but as soon as the requests rate reached the usual 50-150 req/sec, the server shut up. Load average went up to 60, all requests timed out and the backlog on php-fpm socket was full (2048). All diagnostic tools showed that I/O was negligible, there was no swapping and the only problem was high CPU usage caused by php-fpm. Php-fpm was configured to start 100 children. When I lowered the number to 50, LA decreased to 30, but the requests were timing out as before. Things looked like high LA was caused by bad PHP code, but it was the same code that ran without problems on the old server. I tried to trace PHP scripts using `truss', but could not find any problems -- just normal operations. Slow PHP requests logged by php-fpm contained no regular patterns that might indicate problematic areas -- any kind of request ended up in the slow log. freebsd-questions@freebsd.org I moved the sites back to the old server. The LA on the old one stayed high until the queued requests had been processed. What's wrong with php-fpm? Some specs: FreeBSD 10.1-RELEASE-p9 running on Xeon(R) CPU E5-2420. Root partition is on ZFS. PHP 5.3.29 (5.3 required by developers). Nginx 1.6.2. MySQL runs on another server and doesn't look like a bottleneck. This is the upper part of `top' output: Mem: 317M Active, 5866M Inact, 6819M Wired, 2724M Free ARC: 5652M Total, 2382M MFU, 2554M MRU, 16K Anon, 32M Header, 684M Other Swap: 4096M Total, 4096M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 34803 www 81 0 310M 172M RUN 4 7:40 31.05% php-fpm{php-fpm} 34796 www 80 0 314M 175M CPU3 3 7:42 30.57% php-fpm{php-fpm} 34797 www 81 0 310M 171M RUN 7 7:35 30.57% php-fpm{php-fpm} 34785 www 80 0 306M 164M *Name 7 7:38 27.88% php-fpm{php-fpm} 34790 www 80 0 310M 172M RUN 5 6:37 27.88% php-fpm{php-fpm} 34783 www 80 0 310M 171M *Name 8 7:40 27.78% php-fpm{php-fpm} 34805 www 80 0 310M 170M CPU6 6 7:38 27.59% php-fpm{php-fpm} 34821 www 80 0 310M 171M RUN 11 7:39 26.95% php-fpm{php-fpm} 34815 www 80 0 318M 178M CPU8 8 7:36 26.37% php-fpm{php-fpm} 34822 www 80 0 310M 171M *Name 0 7:38 26.27% php-fpm{php-fpm} 34813 www 80 0 314M 175M RUN 10 7:40 25.98% php-fpm{php-fpm} 34816 www 79 0 314M 175M RUN 6 7:37 25.98% php-fpm{php-fpm} 34826 www 79 0 314M 175M CPU1 1 7:34 25.98% php-fpm{php-fpm} 34812 www 79 0 306M 167M RUN 6 7:33 25.88% php-fpm{php-fpm} 34823 www 79 0 306M 168M *Name 2 7:35 25.78% php-fpm{php-fpm} 34818 www 79 0 314M 174M *Name 0 7:42 25.68% php-fpm{php-fpm} 34829 www 79 0 310M 172M CPU11 11 7:37 25.68% php-fpm{php-fpm} 34806 www 79 0 310M 172M RUN 2 7:35 25.39% php-fpm{php-fpm} 34817 www 80 0 318M 177M CPU7 7 7:38 25.29% php-fpm{php-fpm} Thanks. -- With best regards, Dimitri Minaev