From owner-freebsd-fs@FreeBSD.ORG Tue May 31 15:40:46 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D83C21065674 for ; Tue, 31 May 2011 15:40:46 +0000 (UTC) (envelope-from patpro@patpro.net) Received: from rack.patpro.net (rack.patpro.net [193.30.227.216]) by mx1.freebsd.org (Postfix) with ESMTP id 3F3978FC0A for ; Tue, 31 May 2011 15:40:46 +0000 (UTC) Received: from rack.patpro.net (localhost [127.0.0.1]) by rack.patpro.net (Postfix) with ESMTP id 52FF01CC038 for ; Tue, 31 May 2011 17:21:30 +0200 (CEST) X-Virus-Scanned: amavisd-new at patpro.net Received: from amavis-at-patpro.net ([127.0.0.1]) by rack.patpro.net (rack.patpro.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3rkg-elBwE0V for ; Tue, 31 May 2011 17:21:27 +0200 (CEST) Received: from [127.0.0.1] (localhost [127.0.0.1]) by rack.patpro.net (Postfix) with ESMTP for ; Tue, 31 May 2011 17:21:27 +0200 (CEST) From: Patrick Proniewski Content-Type: multipart/signed; boundary=Apple-Mail-4--16922292; protocol="application/pkcs7-signature"; micalg=sha1 Date: Tue, 31 May 2011 17:21:15 +0200 Message-Id: <7DA2CB2F-FA87-427D-903E-514882EE6068@patpro.net> To: FreeBSD Filesystems Mime-Version: 1.0 (Apple Message framework v1084) X-Mailer: Apple Mail (2.1084) X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: No physical znode address X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2011 15:40:47 -0000 --Apple-Mail-4--16922292 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi all, I'm running a FreeBSD 8.2 server, with Apache 2.2 hosting around 260 web = sites. It's a virtual machine, running on top of ESXi and a SAN storage. The OS is installed on UFS, and a dedicated ZFS disk holds every web = sites. Each web site is a ZFS volume created from the zpool "tank". # zpool list =20 NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 149G 53.6G 95.4G 35% ONLINE - # zpool status =20 pool: tank state: ONLINE scrub: scrub completed after 0h19m with 0 errors on Fri May 13 22:57:10 = 2011 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 da1 ONLINE 0 0 0 errors: No known data errors Today, I've noticed an httpd process, stuck, using 100% CPU for hours. = It looks like the process has opened non-existing files. Here is a part = of the output of lsof: # lsof -p 10453 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME httpd 10453 www cwd No = physical znode address: 0xffffff0013019c60 httpd 10453 www rtd VDIR 0,87 512 2 / httpd 10453 www txt VREG 0,87 1321703 406618 = /usr/local/sbin/httpd httpd 10453 www txt VREG 0,87 246776 235521 = /libexec/ld-elf.so.1 httpd 10453 www txt VREG 0,87 154320 659461 = /lib/libm.so.5 ../.. httpd 10453 www 120r No = physical znode address: 0xffffff00132e2840 httpd 10453 www 121r No = physical znode address: 0xffffff0013019c60 httpd 10453 www 122r No = physical znode address: 0xffffff00132e2840 httpd 10453 www 123r No = physical znode address: 0xffffff0013019c60 httpd 10453 www 124r No = physical znode address: 0xffffff00132e2840 httpd 10453 www 125r No = physical znode address: 0xffffff0013019c60 httpd 10453 www 126r No = physical znode address: 0xffffff00132e2840 httpd 10453 www 127r No = physical znode address: 0xffffff0013019c60 httpd 10453 www 128r No = physical znode address: 0xffffff00132e2840 httpd 10453 www 129r No = physical znode address: 0xffffff0013019c60 httpd 10453 www 130r No = physical znode address: 0xffffff00132e2840 httpd 10453 www 131r No = physical znode address: 0xffffff0013019c60 ../.. Reading a part of lsof's source code, it seems to relate to ZFS = (dnode2.c - FreeBSD ZFS node functions for lsof). Using truss, I've discovered that the process is trying to stat a = non-existing file, with a way too long path. truss output is a infinite = repetition of: = stat("/Sites/sites//spip-core/sites/spip-core/sites/spip-core/sites/spip-c= ore/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/= spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/= sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip= -core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/site= s/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-cor= e/sites/spip-core/sites/spip-core/sites/spip-core/sites/spip-core/sites/sp= ip-core-vh/sites/edhum/bd/.Trashes////////////////////////////////////////= //////////////////////////////////////////////////////////////////////////= //////////////////////////////////////////////////////////////////////////= //////////////////////////////////////////////////////////////////////////= //////////////////////////////////////////////////////////////////////////= //////////////////////////////////////////////////////////////////////////= ////////////////////////////////////////////////////////////////////",0x7f= ffffffcc90) ERR#63 'File name too long' (I had to kill -9 truss process...) Obviously, there is something wrong with this particular web site. But = I'm afraid it could come from the file system, or impact the FS.=20 Any idea is welcome. patpro --Apple-Mail-4--16922292--